English POS tagging exercise
POS analysis of multiple files
Download sample.zip. In this task, you will:
- download the sample corpus to your local computer;
- process all
.txtfiles using spaCy; - count the UPOS and XPOS tags in each tex (both raw and normalized frequency);
- organize the results in pandas DataFrames;
- save the results as CSV files.
This would be a part of Assignment 3.
Download the sample corpus
After extracting the ZIP file, specify the local folder containing the .txt files:
from pathlib import Path
sample_folder = Path("YOUR LOCAL PATH")
text_files = sorted(sample_folder.glob("*.txt"))
print(f"Number of text files: {len(text_files)}")
for text_file in text_files:
print(text_file.name)
Expected output:
Number of text files: 5
sample1.txt
sample2.txt
sample3.txt
sample4.txt
sample5.txt
Process all text files
Read each .txt file and process the texts using nlp_en.pipe():
texts = [
text_file.read_text(encoding="utf-8")
for text_file in text_files
]
docs = list(nlp_en.pipe(texts))
Collect token-level POS annotations
For the frequency analysis, create another DataFrame that excludes spaces and punctuation:
token_annotations = []
for text_file, doc in zip(text_files, docs):
for token in doc:
if not token.is_space and not token.is_punct:
token_annotations.append(
{
"file": text_file.name,
"token": token.text,
"lemma": token.lemma_,
"upos": token.pos_,
"xpos": token.tag_,
}
)
print(f"Number of token annotations: {len(token_annotations)}")
print(token_annotations[:5]) # for preview
The resulting table contains the following columns:
file: the source filename;token: the original word or punctuation mark;lemma: the base form of the token;upos: the universal POS tag;xpos: the detailed English POS tag.
anno_df = pd.DataFrame(token_annotations) #save into dataframe
print(anno_df.head())
Count UPOS tags in each text
Group the data by filename and UPOS tag:
upos_counts = (
anno_df
.groupby(["file", "upos"])
.size()
.reset_index(name="count")
)
print(upos_counts.head())
Normalize UPOS frequencies
Now calculate the normalized frequency of each UPOS tag to account for differences in text length. As we practiced in an earlier module, we will normalize frequencies per 1,000 tokens.
The normalized frequency is calculated as:
Normalized frequency = (Raw frequency / Total number of tokens in the text) × 1,000
First, define a function that calculates normalized frequency:
def normalize_frequency(count, total_tokens, base=1000):
return (count / total_tokens) * base
Next, calculate the total number of tokens in each text and apply the function to the UPOS counts:
token_totals = anno_df.groupby("file").size() # count the total number of tokens in a text
upos_counts["normalized_freq"] = upos_counts.apply(
lambda row: normalize_frequency(
row["count"],
token_totals[row["file"]] ),
axis=1
) # add normalized frequency counts
print(upos_counts.head()) # check
The resulting DataFrame contains both raw counts (count) and normalized frequencies (normalized_freq) for each UPOS tag in each text.
Save into a csv file
upos_counts.to_csv("upos_counts.csv", index=False)
Create a wide-format UPOS table
To show one text per row and one UPOS category per column:
upos_table = (
upos_counts
.pivot(
index="file",
columns="upos",
values="count",
)
.fillna(0)
.astype(int)
.reset_index()
)
print(upos_table.head())
Compare two outputs from the previous upos_count and upos_table.
Save into a csv file
upos_table.to_csv("all_texts_upos_counts.csv", index=False)
Count XPOS tags in each text
Group the data by filename and XPOS tag, as you did for UPOS.
Normalize XPOS frequencies
Now try counting the normalized frequency of each XPOS tag (because the total number of tokens may vary across texts).
Create a wide-format XPOS table
Following the UPOS example, show one text per row and one XPOS tag per column.
Submission
Please submit the following files with the following naming conventions:
- python code: en_pos.py
- Token-level annotations: token_annotations.csv
- UPOS (+ normalized) counts: upos_counts.csv
- XPOS (+ normalized) counts: xpos_counts.csv
- Wide-format UPOS table: all_texts_upos_counts.csv
- Wide-format XPOS table: all_texts_xpos_counts.csv