Frequency calculation

Corpus analysis often begins by counting how many times a particular linguistic unit occurs. We will compare:

token instances, which are individual occurrences of word forms;
token types, which are unique word forms;
lemma types, which group word forms assigned the same lemma.

Load the English pipeline

Import spaCy and Python’s Counter class:

import spacy

from collections import Counter

Load the English transformer pipeline:

nlp_en = spacy.load("en_core_web_trf")

Read the corpus file

Read the practice text from ./corpus/freq_test.txt:

with open(
    "./corpus/freq_test.txt",
    "r",
    encoding="utf-8"
) as f:
    raw = f.read()

print(raw[:500])

Process the text with spaCy:

doc = nlp_en(raw)

Extract tokens and lemmas

The following functions extract lowercase tokens and lemmas while excluding whitespace and punctuation:

def extract_tokens(doc):
    return [
        token.text.lower()
        for token in doc
        if not token.is_space and not token.is_punct
    ]


def extract_lemmas(doc):
    return [
        token.lemma_.lower()
        for token in doc
        if not token.is_space and not token.is_punct
    ]

Apply the functions:

tokens = extract_tokens(doc)
lemmas = extract_lemmas(doc)

print("First 10 tokens:")
print(tokens[:10])

print("\nFirst 10 lemmas:")
print(lemmas[:10])

The token and lemma lists normally contain the same number of instances because each retained token receives one lemma. However, the lemma list may contain fewer unique types because several word forms can share the same lemma.

Calculate token and lemma frequencies

Python’s Counter class counts how many times each item occurs:

token_frequency = Counter(tokens)
lemma_frequency = Counter(lemmas)

Use most_common() to display units in descending order of frequency:

print("10 most common tokens:")
print(token_frequency.most_common(10))

print("\n10 most common lemmas:")
print(lemma_frequency.most_common(10))

Count instances and types

A token instance is one occurrence of a word form, while a token type is a unique word form:

print("Number of token instances:", len(tokens))
print("Number of token types:", len(token_frequency))

A lemma instance is one occurrence of a lemma, while a lemma type is a unique lemma:

print("Number of lemma instances:", len(lemmas))
print("Number of lemma types:", len(lemma_frequency))

The number of lemma types may be smaller than the number of token types because different word forms can share the same lemma.

Inspect token–lemma relationships

Inspect how spaCy analyzed the first 20 tokens:

for token in doc[:20]:
    if not token.is_space and not token.is_punct:
        print(
            token.text,
            token.lemma_,
            token.pos_,
            sep="\t"
        )

Find low-frequency units

Find token types that occur only once:

token_hapax = [
    token
    for token, frequency in token_frequency.items()
    if frequency == 1
]

print("Number of token hapax types:", len(token_hapax))
print(token_hapax[:10])

Find lemma types that occur only once:

lemma_hapax = [
    lemma
    for lemma, frequency in lemma_frequency.items()
    if frequency == 1
]

print("Number of lemma hapax types:", len(lemma_hapax))
print(lemma_hapax[:10])

Save a frequency list as a CSV file

A frequency list can be saved as a .csv file for later analysis.

Import Python’s built-in csv module:

import csv

Define a function that saves a Counter object in descending order of frequency:

def save_frequency_csv(frequency, filepath, unit_name):
    with open(
        filepath,
        "w",
        encoding="utf-8",
        newline=""
    ) as f:
        writer = csv.writer(f)

        writer.writerow([unit_name, "frequency"])
        writer.writerows(frequency.most_common())

Save the token-frequency list:

save_frequency_csv(
    token_frequency,
    "token_frequency.csv",
    "token"
)

Save the lemma-frequency list:

save_frequency_csv(
    lemma_frequency,
    "lemma_frequency.csv",
    "lemma"
)

Next: Exercises