Korean POS tagging with Stanza

Stanza is a Python NLP toolkit developed by the Stanford NLP Group. It provides pretrained language-specific pipelines for tasks such as tokenization, POS tagging, morphological analysis, lemmatization, and dependency parsing. The language model and training package used by the pipeline can be specified when the pipeline is created.

Although spaCy also supports Korean, we use Stanza in this tutorial to gain experience with a different Korean NLP pipeline and annotation scheme. In my previous research, I have primarily used POS annotations based on the Sejong tag set for fine-grained morphosyntactic applications, so here we practice using a pipeline and annotation scheme—Stanza with Korean GSD model (which supports Sejong tag set).

Installing Stanza

Install Stanza from a terminal or command prompt:

pip install -U stanza

Import Stanza and download the Korean GSD models:

import stanza

stanza.download(
    "ko",
    package="gsd",
    processors="tokenize,pos,lemma"
)

Loading the Korean pipeline

Create a Korean pipeline using the GSD package:

nlp_ko = stanza.Pipeline(
    lang="ko",
    package="gsd",
    processors="tokenize,pos,lemma",
    use_gpu=False
)

Inspect the installed Stanza version and pipeline:

print("Stanza version:", stanza.__version__)
print("Processors:", nlp_ko.processors.keys())

The pipeline includes three processors:

tokenize: identifies sentences and tokens;
pos: assigns UPOS, XPOS, and morphological features;
lemma: assigns a lemma to each word.

Processing a Korean sentence

Create a Korean text:

korean_text = "학생들은 과제를 매우 꼼꼼하게 작성했습니다."

Process the text:

korean_doc = nlp_ko(korean_text)

The output is a Stanza Document object. A Document contains one or more Sentence objects, and each sentence contains Word objects.

print(type(korean_doc))
print(type(korean_doc.sentences[0]))
print(type(korean_doc.sentences[0].words[0]))

Print the words identified by the pipeline:

for sentence in korean_doc.sentences:
    for word in sentence.words:
        print(word.text)

Inspecting part-of-speech annotations

Each Stanza Word stores multiple linguistic annotations:

for sentence in korean_doc.sentences:
    for word in sentence.words:
        print(
            word.text,
            word.lemma,
            word.upos,
            word.xpos,
            word.feats,
            sep="\t"
        )

The attributes have the following meanings:

word.text: the word form represented by Stanza;
word.lemma: the lemma assigned to the word;
word.upos: the coarse-grained Universal POS category;
word.xpos: a more detailed Korean POS analysis;
word.feats: Universal Dependencies morphological features, when available.

For Korean, word.xpos may contain multiple tags joined with +. These tags represent the internal grammatical elements identified by the model.

For example, a form such as:

학생들은

may contain elements corresponding approximately to:

학생 + 들 + 은
student + plural + topic

However, the exact analysis depends on the model and its annotation conventions.

Creating and saving an analysis table

First, import pandas:

import pandas as pd

Create a DataFrame containing the word-level annotations:

korean_annotations = pd.DataFrame(
    [
        {
            "word": word.text,
            "lemma": word.lemma,
            "upos": word.upos,
            "xpos": word.xpos,
            "feats": word.feats,
        }
        for sentence in korean_doc.sentences
        for word in sentence.words
    ]
)

display(korean_annotations)

Save the DataFrame as a CSV file

korean_annotations.to_csv(
    "korean_annotations.csv",
    index=False,       # prevents row numbers from becoming a separate column
    encoding="utf-8-sig",  # helps display Korean characters correctly
)

Korean UPOS vs. XPOS tags

UPOS provides a broad grammatical category, while XPOS provides a more detailed Korean-specific analysis.

Process several Korean verb forms in context:

verb_text = (
    "학생이 과제를 작성한다. "
    "학생이 과제를 작성했다. "
    "학생이 과제를 작성하고 있다."
)

verb_doc = nlp_ko(verb_text)

for sentence in verb_doc.sentences:
    for word in sentence.words:
        if word.upos in {"VERB", "AUX"}:
            print(
                word.text,
                word.lemma,
                word.upos,
                word.xpos,
                word.feats,
                sep="\t"
            )

The words may share a broad UPOS category while receiving different XPOS analyses because of tense, aspect, connective, auxiliary, or sentence-ending elements.

Excluding punctuation

Unlike spaCy tokens, Stanza Word objects do not provide an is_punct attribute. We can exclude punctuation using the UPOS tag:

for sentence in korean_doc.sentences:
    for word in sentence.words:
        if word.upos != "PUNCT":
            print(word.text, word.upos)

Extracting nouns

Create another Korean sentence:

korean_text2 = "민수는 대학교 도서관에서 책을 읽었습니다."

korean_doc2 = nlp_ko(korean_text2)

Extract the lemmas of common nouns:

korean_nouns = [
    (word.lemma or word.text).lower()
    for sentence in korean_doc2.sentences
    for word in sentence.words
    if word.upos == "NOUN"
]

print(korean_nouns)

To include both common and proper nouns:

korean_nouns_and_names = [
    (word.lemma or word.text).lower()
    for sentence in korean_doc2.sentences
    for word in sentence.words
    if word.upos in {___, ___}
]

print(korean_nouns_and_names)

The expression:

word.lemma or word.text

uses the original form when no lemma is available.

Note. You can modify the filtering condition to extract verbs, auxiliaries, adjectives, or other word classes.

Counting part-of-speech categories

Use Counter to calculate the frequency of each UPOS category:

from collections import Counter

korean_pos_counts = Counter(
    word.upos
    for sentence in korean_doc.sentences
    for word in sentence.words
    if word.upos != "PUNCT"
)

print(korean_pos_counts)

Display the categories from most to least frequent:

for pos, frequency in korean_pos_counts.most_common():
    print(pos, frequency)

Processing multiple texts

Create a list of Korean texts:

korean_texts = [
    "첫 번째 학생은 글을 수정했습니다.",
    "두 번째 학생은 여러 예시를 추가했습니다.",
    "세 번째 학생은 자신의 주장을 명확하게 설명했습니다."
]

Process each text and extract its lemmas:

for text in korean_texts:
    doc = nlp_ko(text)

    lemmas = [
        (word.lemma or word.text).lower()
        for sentence in doc.sentences
        for word in sentence.words
        if word.upos != "PUNCT"
    ]

    print(lemmas)

Next: Exercises