Lemmatization

In NLP, lemmatization converts an inflected word form into a base or dictionary form called a lemma.

For example:

writes  → write
wrote   → write
writing → write

Tokenization and lemmatization are related but distinct processes:

Tokenization determines where token boundaries occur.
Lemmatization assigns a lemma to each token.

Loading the English and Korean pipelines

import spacy

nlp_en = spacy.load("en_core_web_trf")
nlp_ko = spacy.load("ko_core_news_lg")

print("English pipeline:")
print(nlp_en.pipe_names)

print("\nKorean pipeline:")
print(nlp_ko.pipe_names)

The English transformer model provides contextual representations that support components such as POS tagging and dependency parsing. Lemmatization is performed by a separate lemmatizer component in the pipeline.

The Korean pipeline includes both a morphologizer and a trainable lemmatizer. The morphologizer predicts grammatical information about each token, while the lemmatizer assigns its lemma.

English lemmatization

Process an English sentence using the English pipeline:

english_text = (
    "I am running because I have to eat "
    "chocolate ice cream."
)

english_doc = nlp_en(english_text)

Each Token object stores several linguistic annotations:

for token in english_doc:
    print(
        token.text,
        token.lemma_,
        token.pos_,
        token.tag_,
        token.morph
    )

The attributes used here are:

token.text: the original word form;
token.lemma_: the lemma assigned by spaCy;
token.pos_: the universal part-of-speech category;
token.tag_: a more language-specific part-of-speech tag;
token.morph: morphological features associated with the token.

For example, the surface form running should be associated with the lemma run.

To inspect only the word form and lemma:

for token in english_doc:
    print(token.text, "→", token.lemma_)

Extracting English lemmas

To extract the lemmas as a Python list:

english_lemmas = [
    token.lemma_
    for token in english_doc
]

print(english_lemmas)

To exclude spaces and punctuation:

english_lemmas = [
    token.lemma_.lower()
    for token in english_doc
    if not token.is_space and not token.is_punct
]

print(english_lemmas)

Lowercasing after processing allows forms such as I and i to be treated as the same string in later frequency calculations.

It is generally better to preserve the original capitalization while running the pipeline because capitalization may provide useful contextual information to the model.

Korean lemmatization and morphological analysis

Korean words may contain several grammatical elements within one space-delimited unit.

For example:

학생들은
학생 + 들 + 은
student + plural + topic

Similarly, a Korean verb form may include a lexical stem together with tense, aspect, honorific, and sentence-ending elements.

Process a Korean sentence using the Korean pipeline:

korean_text = "학생들은 과제를 작성하고 있었습니다."

korean_doc = nlp_ko(korean_text)

Inspect the analysis assigned to each token:

for token in korean_doc:
    print(
        token.text,
        token.lemma_,
        token.pos_,
        token.tag_,
        token.morph
    )

For Korean, the attributes can be interpreted as follows:

token.text: the token found in the original text;
token.lemma_: the lemma representation assigned by the lemmatizer;
token.pos_: a universal part-of-speech category such as NOUN or VERB;
token.tag_: a Korean-specific, fine-grained part-of-speech analysis;
token.morph: morphological information predicted for the token.

The value of token.tag_ may contain multiple tags joined with +. This indicates that the token contains more than one morphological element.

To make the output easier to read:

print(
    f"{'TOKEN':<15}"
    f"{'LEMMA':<20}"
    f"{'POS':<10}"
    f"{'TAG'}"
)

for token in korean_doc:
    print(
        f"{token.text:<15}"
        f"{token.lemma_:<20}"
        f"{token.pos_:<10}"
        f"{token.tag_}"
    )

Morphological analysis and lemmatization are not identical

A morphological analysis identifies grammatical components and categories within a word form.

A lemma is the base form selected to represent that word in lexical analysis.

For example, a Korean verb form may contain:

lexical stem + connective ending + auxiliary + tense + final ending

The morphological analysis describes these components, while the lemma provides a normalized representation of the token.

In the spaCy Korean pipeline, the morphologizer does not necessarily create a separate spaCy Token object for every individual morpheme. Instead, information about the internal analysis may be represented through attributes such as token.tag_, token.pos_, and token.morph.

Therefore, do not assume that:

one spaCy token = one Korean morpheme

The tokenization, fine-grained tag, and lemma should be inspected together.

Extracting Korean lemmas

To extract Korean lemmas:

korean_lemmas = [
    token.lemma_
    for token in korean_doc
    if not token.is_space and not token.is_punct
]

print(korean_lemmas)

Lowercasing does not affect Hangul, but using .lower() can still be useful when the corpus contains English words or Roman characters:

korean_lemmas = [
    token.lemma_.lower()
    for token in korean_doc
    if not token.is_space and not token.is_punct
]

print(korean_lemmas)

Comparing English and Korean analyses

The same attributes can be inspected for both languages:

def inspect_doc(doc):
    for token in doc:
        if not token.is_space:
            print(
                token.text,
                token.lemma_,
                token.pos_,
                token.tag_,
                token.morph,
                sep="\t"
            )

Apply the function to the English text:

print("English analysis:")
inspect_doc(english_doc)

Apply the same function to the Korean text:

print("\nKorean analysis:")
inspect_doc(korean_doc)

Although the same spaCy attributes are available, their interpretation may differ across languages.

For English, an inflected form such as running is usually represented as one token with the lemma run.

For Korean, one token may contain a lexical element together with one or more grammatical morphemes. Its fine-grained tag and lemma may therefore contain more complex information.

A reusable lemma-extraction function

The same basic function can be applied to either language pipeline:

def extract_lemmas(text, nlp):
    doc = nlp(text)

    return [
        token.lemma_.lower()
        for token in doc
        if not token.is_space and not token.is_punct
    ]

Extract English lemmas:

english_lemmas = extract_lemmas(
    english_text,
    nlp_en
)

print(english_lemmas)

Extract Korean lemmas:

korean_lemmas = extract_lemmas(
    korean_text,
    nlp_ko
)

print(korean_lemmas)

The function is the same, but the selected nlp pipeline determines how the text is tokenized, morphologically analyzed, and lemmatized.

Context-sensitive lemmatization in English

Lemmatization is not simply the removal of suffixes. The assigned lemma may depend on the token’s grammatical context and part of speech.

Consider the word saw:

text = "I saw a saw."

doc = nlp_en(text)

for token in doc:
    print(
        token.text,
        token.lemma_,
        token.pos_
    )

The two occurrences have the same surface form but different grammatical functions. The first is a verb and the second is a noun.

Inspecting Korean morphological structure

For Korean, examine how the pipeline analyzes nouns with particles and complex verb forms:

text = "학생들은 과제를 작성하고 있었습니다."

doc = nlp_ko(text)

for token in doc:
    if not token.is_space and not token.is_punct:
        print(
            "Token:", token.text,
            "\nLemma:", token.lemma_,
            "\nUniversal POS:", token.pos_,
            "\nKorean tag:", token.tag_,
            "\nMorphology:", token.morph,
            "\n"
        )

Consider the following questions:

Does a token correspond to an entire space-delimited unit?
Does token.tag_ contain more than one tag?
How are particles and endings represented?
Does the lemma correspond to a dictionary form or a sequence of analyzed elements?

Next: Subword tokenization