Exercises

Choose one text file containing at least 500 words.

You may use a text from a public source, your own writing, or another corpus. Save the file in .txt format and place it in the corpus folder.

In this exercise, you will:

select appropriate tools for the language of your text;
process the text with spaCy;
inspect tokens, lemmas, and grammatical annotations;
calculate token and lemma frequencies;
compare spaCy tokens with subword tokens.

Exercise 1. Select a text

Record the following information:

file name;
language;
source of the text;
approximate number of words.

Set the path to your file:

FILEPATH = "/your_file.txt"

Read the file:

with open(
    FILEPATH,
    "r",
    encoding="utf-8"
) as f:
    raw = f.read()

print(raw[:500])
print("Whitespace-separated words:", len(raw.split()))

Confirm that the file contains at least 500 words.

Exercise 2. Select and load the tools

Import the required libraries:

import spacy

from collections import Counter
from transformers import AutoTokenizer

Find a spaCy pipeline appropriate for the language of your text. Then enter its name below:

SPACY_MODEL = "your_spacy_pipeline"
nlp = spacy.load(SPACY_MODEL)

Find a pretrained language model appropriate for the language of your text. Load its tokenizer:

SUBWORD_MODEL = "your_pretrained_model"
subword_tokenizer = AutoTokenizer.from_pretrained(
    SUBWORD_MODEL
)

Inspect the selected tools:

print("spaCy language:", nlp.lang)
print("spaCy components:", nlp.pipe_names)
print("Subword tokenizer:", SUBWORD_MODEL)

Check that the spaCy pipeline includes the components needed for tokenization and lemmatization.

Exercise 3. Create and inspect a spaCy `Doc`

Process the text:

doc = nlp(raw)

print(type(raw))
print(type(doc))

Inspect the first 50 tokens:

for token in doc[:50]:
    if not token.is_space:
        print(
            token.text,
            token.lemma_,
            token.pos_,
            token.tag_,
            token.morph,
            sep="\t"
        )

Check whether the tokenization and lemmatization appear reasonable.

Exercise 4. Extract tokens and lemmas

Define functions that extract tokens and lemmas:

def extract_tokens(doc):
    return [
        token.text.lower()
        for token in doc
        if not token.is_space and not token.is_punct
    ]


def extract_lemmas(doc):
    return [
        token.lemma_.lower()
        for token in doc
        if not token.is_space and not token.is_punct
    ]

Apply the functions:

tokens = extract_tokens(doc)
lemmas = extract_lemmas(doc)

print("First 30 tokens:")
print(tokens[:30])

print("\nFirst 30 lemmas:")
print(lemmas[:30])

Count the instances and types:

print("Token instances:", len(tokens))
print("Token types:", len(set(tokens)))

print("Lemma instances:", len(lemmas))
print("Lemma types:", len(set(lemmas)))

Exercise 5. Calculate token and lemma frequencies

Calculate the frequencies:

token_frequency = Counter(tokens)
lemma_frequency = Counter(lemmas)

Print the most frequent units:

print("20 most common tokens:")
print(token_frequency.most_common(20))

print("\n20 most common lemmas:")
print(lemma_frequency.most_common(20))

Find units that occur only once:

token_hapax = [
    token
    for token, frequency in token_frequency.items()
    if frequency == 1
]

lemma_hapax = [
    lemma
    for lemma, frequency in lemma_frequency.items()
    if frequency == 1
]

print("Number of token hapax types:", len(token_hapax))
print("Number of lemma hapax types:", len(lemma_hapax))

print("\nSample token hapax types:")
print(token_hapax[:20])

print("\nSample lemma hapax types:")
print(lemma_hapax[:20])

Briefly compare the token-frequency and lemma-frequency lists.

Exercise 6. Apply a subword tokenizer

Select a short sample from the file:

sample = raw[:500]

Apply the tokenizer:

subwords = subword_tokenizer.tokenize(sample)

print("Subword tokens:")
print(subwords)

print("Number of subword tokens:", len(subwords))

Convert the sample into token IDs:

subword_ids = subword_tokenizer.encode(
    sample,
    add_special_tokens=False
)

print(subword_ids[:50])

Convert the IDs back into subword strings:

pieces = subword_tokenizer.convert_ids_to_tokens(
    subword_ids
)

print(pieces[:50])

Exercise 7. Compare spaCy and subword tokenization

Process the same sample with spaCy:

sample_doc = nlp(sample)

sample_spacy_tokens = [
    token.text
    for token in sample_doc
    if not token.is_space
]

Compare the results:

print("spaCy tokens:")
print(sample_spacy_tokens)

print("\nSubword tokens:")
print(subwords)

print("\nNumber of spaCy tokens:", len(sample_spacy_tokens))
print("Number of subword tokens:", len(subwords))

Identify two or three spaCy tokens that were divided into multiple subwords.

Exercise 8. Brief reflection

Answer the following questions briefly:

How did the token and lemma frequency lists differ?
Did you notice any tokenization or lemmatization errors?
How did spaCy tokens differ from subword tokens?
Which unit is more appropriate for lexical-frequency analysis of your text?

What to submit

Submit:

the text file or a link to its source;
the spaCy pipeline and subword tokenizer used;
the completed code and main outputs;
the brief reflection.

Back to Module 5