Subword tokenization

spaCy tokenization creates tokens intended to support linguistic annotation and text analysis. Language models, however, often divide text into smaller computational units called subword tokens. Subword tokenization allows a language model to represent unfamiliar words, spelling variants, and morphologically complex forms using a fixed vocabulary.

A tokenizer belongs to a particular language model

A subword tokenizer is not a general-purpose linguistic tokenizer. It is constructed as part of a particular pretrained language model. Therefore, the tokenizer used to process a text should normally match the language model that will receive the text.

In this tutorial, we will use separate English and Korean models:

Language	Model and tokenizer
English	`google-bert/bert-base-cased`
Korean	`klue/bert-base`

Both are BERT-based models, but their vocabularies were constructed using different training data.

The English tokenizer was developed for an English pretrained model. The KLUE tokenizer was developed using Korean data and was designed to represent Korean text more appropriately.

Note: Another valid approach would be to use one multilingual tokenizer, such as multilingual BERT, for both languages. That approach is appropriate when investigating how the same multilingual model represents different languages.

Installing Transformers

In this tutorial, we will use tokenizers from the Hugging Face transformers library:

pip install transformers

Loading the English and Korean tokenizers

Use AutoTokenizer to load the tokenizer associated with each pretrained model:

from transformers import AutoTokenizer

english_subword_tokenizer = AutoTokenizer.from_pretrained(
    "google-bert/bert-base-cased"
)

korean_subword_tokenizer = AutoTokenizer.from_pretrained(
    "klue/bert-base"
)

AutoTokenizer.from_pretrained() reads the tokenizer configuration associated with the selected model and loads the appropriate tokenizer.

We can inspect the tokenizer classes:

print(type(english_subword_tokenizer))
print(type(korean_subword_tokenizer))

We can also inspect the vocabulary sizes:

print(
    "English vocabulary size:",
    english_subword_tokenizer.vocab_size
)

print(
    "Korean vocabulary size:",
    korean_subword_tokenizer.vocab_size
)

The two models do not share the same vocabulary. The same character sequence may therefore be divided differently by the two tokenizers.

English subword tokenization

Use the English tokenizer for the English sentence:

english_text = (
    "I am running because I have to eat "
    "chocolate ice cream."
)

english_subwords = english_subword_tokenizer.tokenize(
    english_text
)

print(english_subwords)
print("Number of English subwords:", len(english_subwords))

A frequent English word may remain one token, while a less frequent or morphologically complex word may be divided into multiple subword pieces.

BERT tokenizers may use ## to indicate that a subword continues the preceding piece. The symbol is part of the tokenizer’s representation and was not present in the original sentence.

Korean subword tokenization

Use the Korean tokenizer for the Korean sentence:

korean_text = "학생들은 과제를 작성하고 있었습니다."

korean_subwords = korean_subword_tokenizer.tokenize(
    korean_text
)

print(korean_subwords)
print("Number of Korean subwords:", len(korean_subwords))

The Korean tokenizer may divide a space-delimited unit into smaller pieces.

These pieces may sometimes resemble Korean morphemes, but they should not automatically be interpreted as linguistically defined morphemes. They are units selected for the vocabulary of the language model.

Subword token IDs

Language models process numerical IDs rather than the displayed token strings.

Convert the English text into token IDs:

english_ids = english_subword_tokenizer.encode(
    english_text,
    add_special_tokens=False
)

print(english_ids)

Convert the Korean text into token IDs:

korean_ids = korean_subword_tokenizer.encode(
    korean_text,
    add_special_tokens=False
)

print(korean_ids)

We can convert the IDs back into subword strings:

print(
    english_subword_tokenizer.convert_ids_to_tokens(
        english_ids
    )
)

print(
    korean_subword_tokenizer.convert_ids_to_tokens(
        korean_ids
    )
)

The numerical IDs are meaningful only in relation to the vocabulary of the corresponding tokenizer.

For example, English token ID 1000 and Korean token ID 1000 do not necessarily represent the same subword.

Special tokens

A tokenizer may add special tokens required by its language model.

english_encoded = english_subword_tokenizer(
    english_text
)

korean_encoded = korean_subword_tokenizer(
    korean_text
)

print(
    english_subword_tokenizer.convert_ids_to_tokens(
        english_encoded["input_ids"]
    )
)

print(
    korean_subword_tokenizer.convert_ids_to_tokens(
        korean_encoded["input_ids"]
    )
)

BERT-based models normally add special tokens marking the beginning and end of the input sequence.

Compare this with the earlier code using:

add_special_tokens=False

That option allows us to inspect only the subwords derived from the original text.

Comparing spaCy and subword tokenization in English

Load the English spaCy pipeline:

import spacy

nlp_en = spacy.load("en_core_web_trf")
english_doc = nlp_en(english_text)

Extract the spaCy tokens:

english_spacy_tokens = [
    token.text
    for token in english_doc
    if not token.is_space
]

print("English spaCy tokens:")
print(english_spacy_tokens)

print("\nEnglish BERT subwords:")
print(english_subwords)

The spaCy and BERT tokenizers serve different purposes:

spaCy tokens support linguistic annotation;
BERT subwords prepare the sentence for a particular language model.

A single spaCy token may correspond to one or more BERT subwords.

Comparing spaCy and subword tokenization in Korean

Load the Korean spaCy pipeline:

nlp_ko = spacy.load("ko_core_news_lg")
korean_doc = nlp_ko(korean_text)

Extract the spaCy tokens:

korean_spacy_tokens = [
    token.text
    for token in korean_doc
    if not token.is_space
]

print("Korean spaCy tokens:")
print(korean_spacy_tokens)

print("\nKLUE BERT subwords:")
print(korean_subwords)

A Korean spaCy token and a KLUE BERT subword are not necessarily equivalent units.

The spaCy pipeline produces units intended for linguistic analysis and annotation. The KLUE tokenizer divides the text according to the vocabulary and preprocessing system developed for the KLUE language model.

Comparing tokens, lemmas, and subwords

The three representations serve different purposes:

Representation	Main purpose
spaCy token	Identifying units for linguistic annotation
spaCy lemma	Grouping inflected or morphologically related forms
Subword token	Representing text as input to a particular language model

For example:

print("English spaCy analysis:")

for token in english_doc:
    if not token.is_space:
        print(
            token.text,
            token.lemma_,
            token.pos_,
            sep="\t"
        )

print("\nEnglish subwords:")
print(english_subwords)

For Korean:

print("Korean spaCy analysis:")

for token in korean_doc:
    if not token.is_space:
        print(
            token.text,
            token.lemma_,
            token.tag_,
            sep="\t"
        )

print("\nKorean subwords:")
print(korean_subwords)

Optional comparison: one multilingual tokenizer

To examine how one shared multilingual model tokenizes both languages, load multilingual BERT:

multilingual_tokenizer = AutoTokenizer.from_pretrained(
    "google-bert/bert-base-multilingual-cased"
)

Apply the same tokenizer to both sentences:

english_multilingual_subwords = (
    multilingual_tokenizer.tokenize(english_text)
)

korean_multilingual_subwords = (
    multilingual_tokenizer.tokenize(korean_text)
)

print("English with multilingual BERT:")
print(english_multilingual_subwords)

print("\nKorean with multilingual BERT:")
print(korean_multilingual_subwords)

Next: Frequency calculation