Tokenization

In this tutorial, we will first use spaCy for linguistic tokenization and lemmatization. spaCy is a well-known NLP library that provides language-specific processing pipelines.

Depending on the language and pipeline, spaCy can perform tasks such as:

tokenization;
POS tagging;
morphological analysis;
lemmatization;
dependency parsing;
named entity recognition.

Installing spaCy and language pipelines

First, install spaCy:

pip install spacy

In Google Colab, add ! before the command:

!pip install spacy

You must also install a trained pipeline for each language that you want to process.

In this tutorial, we will use:

Language	Pipeline
English	`en_core_web_trf`
Korean	`ko_core_news_lg`

Install both pipelines using:

python -m spacy download en_core_web_trf
python -m spacy download ko_core_news_lg

Why are we using these pipelines?

For English, we will use en_core_web_trf, a transformer-based pipeline. Transformer models produce contextualized representations of words, which can support more accurate contextual analyses. However, transformer pipelines require more memory and processing time than smaller pipelines.

For Korean, we will use ko_core_news_lg. spaCy does not currently provide an official transformer-based Korean pipeline, and ko_core_news_lg is the largest of the available official Korean pipelines. It includes Korean-specific tokenization and trained components for linguistic annotation.

Loading the English pipeline

The following code loads the English transformer pipeline:

import spacy

nlp_en = spacy.load("en_core_web_trf")

print("Language:", nlp_en.lang)
print("Pipeline components:", nlp_en.pipe_names)

The nlp_en object is a spaCy Language object. It contains the English tokenizer and the other processing components included in the pipeline.

Note: In Python, an object is a value that stores information and provides functions for working with that information. Here, nlp_en is an English text-processing object created by spaCy.

Loading the Korean pipeline

The following code loads the Korean large pipeline:

nlp_ko = spacy.load("ko_core_news_lg")

print("Language:", nlp_ko.lang)
print("Pipeline components:", nlp_ko.pipe_names)

The nlp_ko object contains the Korean tokenizer and the other components included in the Korean pipeline.

Use the pipeline that matches the language of the text:

english_text = "I am running because I have to eat chocolate ice cream."
korean_text = "나는 초콜릿 아이스크림을 먹어야 해서 달리고 있습니다."

english_doc = nlp_en(english_text)
korean_doc = nlp_ko(korean_text)

From a string to a `Doc`

When we apply a spaCy Language object to a raw string, spaCy returns a Doc object:

text = "The students were writing their assignments."

doc = nlp_en(text)

print(type(text))
print(type(doc))

The basic processing workflow is:

raw string → spaCy Language pipeline → Doc

A spaCy Doc stores the original text together with its tokens and linguistic annotations.

We can access the original text using:

print(doc.text)

A Doc also behaves like a sequence of Token objects:

for token in doc:
    print(token)

Each item produced by the loop is a spaCy Token object representing one token in the document.

Inspecting tokens

Each token in a Doc has several attributes:

for token in doc:
    print(
        token.text,
        token.idx,
        token.is_punct,
        token.is_space
    )

The attributes used here are:

token.text: the original token string;
token.idx: the starting character position of the token;
token.is_punct: whether the token is punctuation;
token.is_space: whether the token consists of whitespace.

To store the tokens in a regular Python list:

tokens = [token.text for token in doc]

print(tokens)
print(type(tokens))

English tokenization

Consider contractions and possessive forms in English:

english_text = "I can't attend today's meeting."

english_doc = nlp_en(english_text)

print([token.text for token in english_doc])

spaCy does not simply split the text at every space. It applies English-specific tokenization rules to identify token boundaries.

For example, a contraction such as can't may be divided into more than one token because it contains multiple grammatical elements.

Korean tokenization

Korean presents a different tokenization problem because an orthographic unit separated by spaces, often called an eojeol, may contain several grammatical elements.

korean_text = "학생들은 오늘 수업에 참여했습니다."

korean_doc = nlp_ko(korean_text)

print([token.text for token in korean_doc])

Inspect the token boundaries in more detail:

for token in korean_doc:
    print(
        token.text,
        token.idx,
        token.is_punct,
        token.is_space
    )

The resulting tokens may not correspond directly to units separated by spaces in the original sentence. This illustrates why tokenization must take the structure of each language into account.

Comparing English and Korean tokenization

english_text = "The students completed the assignment."
korean_text = "학생들은 과제를 완료했습니다."

english_doc = nlp_en(english_text)
korean_doc = nlp_ko(korean_text)

english_tokens = [
    token.text
    for token in english_doc
    if not token.is_space
]

korean_tokens = [
    token.text
    for token in korean_doc
    if not token.is_space
]

print("English tokens:")
print(english_tokens)

print("\nKorean tokens:")
print(korean_tokens)

Token boundaries should be interpreted in relation to the language-specific pipeline. An English token and a Korean token do not necessarily represent equivalent linguistic units.

Inspecting the pipeline components

We can inspect the components included in each pipeline:

print("English pipeline:")
print(nlp_en.pipe_names)

print("\nKorean pipeline:")
print(nlp_ko.pipe_names)

The exact components may differ across languages and models. Before conducting an analysis, check that the selected pipeline provides the annotations required for the research task.

You should also report the exact pipeline name in your research because tokenization and linguistic annotation may differ across languages, pipelines, and software versions.

Next: Lemmatization