Back: English POS tagging

POS analysis of multiple files

Download sample.zip. In this task, you will:

  1. download the sample corpus to your local computer;
  2. process all .txt files using spaCy;
  3. count the UPOS and XPOS tags in each tex (both raw and normalized frequency);
  4. organize the results in pandas DataFrames;
  5. save the results as CSV files.

This would be a part of Assignment 3.


Download the sample corpus

After extracting the ZIP file, specify the local folder containing the .txt files:

from pathlib import Path 

sample_folder = Path("YOUR LOCAL PATH") 

text_files = sorted(sample_folder.glob("*.txt")) 

print(f"Number of text files: {len(text_files)}") 

for text_file in text_files: 
    print(text_file.name)

Expected output:

Number of text files: 5
sample1.txt
sample2.txt
sample3.txt
sample4.txt
sample5.txt

Process all text files

Read each .txt file and process the texts using nlp_en.pipe():

texts = [
    text_file.read_text(encoding="utf-8")
    for text_file in text_files
]

docs = list(nlp_en.pipe(texts))

Collect token-level POS annotations

For the frequency analysis, create another DataFrame that excludes spaces and punctuation:

token_annotations = []

for text_file, doc in zip(text_files, docs):
    for token in doc:
        if not token.is_space and not token.is_punct:
            token_annotations.append(
                {
                    "file": text_file.name,
                    "token": token.text,
                    "lemma": token.lemma_,
                    "upos": token.pos_,
                    "xpos": token.tag_,
                }
            )

print(f"Number of token annotations: {len(token_annotations)}")
print(token_annotations[:5]) # for preview

The resulting table contains the following columns:

  • file: the source filename;
  • token: the original word or punctuation mark;
  • lemma: the base form of the token;
  • upos: the universal POS tag;
  • xpos: the detailed English POS tag.
anno_df = pd.DataFrame(token_annotations) #save into dataframe
print(anno_df.head())

Count UPOS tags in each text

Group the data by filename and UPOS tag:

upos_counts = (
    anno_df
    .groupby(["file", "upos"])
    .size()
    .reset_index(name="count")
)

print(upos_counts.head())

Normalize UPOS frequencies

Now calculate the normalized frequency of each UPOS tag to account for differences in text length. As we practiced in an earlier module, we will normalize frequencies per 1,000 tokens.

The normalized frequency is calculated as:

Normalized frequency = (Raw frequency / Total number of tokens in the text) × 1,000

First, define a function that calculates normalized frequency:

def normalize_frequency(count, total_tokens, base=1000):
    return (count / total_tokens) * base

Next, calculate the total number of tokens in each text and apply the function to the UPOS counts:

token_totals = anno_df.groupby("file").size() # count the total number of tokens in a text

upos_counts["normalized_freq"] = upos_counts.apply(
    lambda row: normalize_frequency(
        row["count"],
        token_totals[row["file"]] ),
    axis=1
) # add normalized frequency counts

print(upos_counts.head()) # check

The resulting DataFrame contains both raw counts (count) and normalized frequencies (normalized_freq) for each UPOS tag in each text.

Save into a csv file

upos_counts.to_csv("upos_counts.csv", index=False)

Create a wide-format UPOS table

To show one text per row and one UPOS category per column:

upos_table = (
    upos_counts
    .pivot(
        index="file",
        columns="upos",
        values="count",
    )
    .fillna(0)
    .astype(int)
    .reset_index()
)
print(upos_table.head())

Compare two outputs from the previous upos_count and upos_table.

Save into a csv file

upos_table.to_csv("all_texts_upos_counts.csv", index=False)

Count XPOS tags in each text

Group the data by filename and XPOS tag, as you did for UPOS.

Normalize XPOS frequencies

Now try counting the normalized frequency of each XPOS tag (because the total number of tokens may vary across texts).

Create a wide-format XPOS table

Following the UPOS example, show one text per row and one XPOS tag per column.


Submission

Please submit the following files with the following naming conventions:

  • python code: en_pos.py
  • Token-level annotations: token_annotations.csv
  • UPOS (+ normalized) counts: upos_counts.csv
  • XPOS (+ normalized) counts: xpos_counts.csv
  • Wide-format UPOS table: all_texts_upos_counts.csv
  • Wide-format XPOS table: all_texts_xpos_counts.csv

Next: English dependency parsing