POS analysis of multiple files

Suppose that you are working with multiple text files in the corpus/sample folder. The folder currently contains five sample texts.

In this task, you will:

download the sample corpus to your local computer;
process all .txt files using spaCy;
count the UPOS and XPOS tags in each text;
organize the results in pandas DataFrames;
save the results as CSV files.

Download the sample corpus

Download the five sample texts and save the ZIP file to your local computer: Download

After extracting the ZIP file, specify the local folder containing the .txt files:

from pathlib import Path 

sample_folder = Path( "YOUR LOCAL PATH") 

text_files = sorted(sample_folder.glob("*.txt")) 

print(f"Number of text files: {len(text_files)}") 

for text_file in text_files: 
    print(text_file.name)

Expected output:

Number of text files: 5
sample1.txt
sample2.txt
sample3.txt
sample4.txt
sample5.txt

Process all text files

Read each .txt file and process the texts using nlp_en.pipe():

texts = [
    text_file.read_text(encoding="utf-8")
    for text_file in text_files
]

docs = list(nlp_en.pipe(texts))

nlp_en.pipe() is useful when processing multiple texts because it processes them more efficiently than calling the model separately for every text.

Collect token-level POS annotations

Create one row for each token in the corpus:

token_annotations = []

for text_file, doc in zip(text_files, docs):
    for token in doc:
        if not token.is_space:
            token_annotations.append(
                {
                    "file": text_file.name,
                    "token": token.text,
                    "lemma": token.lemma_,
                    "upos": token.pos_,
                    "xpos": token.tag_,
                }
            )

Convert the results into a pandas DataFrame:

token_df = pd.DataFrame(token_annotations)

display(token_df)

The resulting table contains the following columns:

file: the source filename;
token: the original word or punctuation mark;
lemma: the base form of the token;
upos: the universal POS tag;
xpos: the detailed English POS tag.

Exclude punctuation from the POS counts

For the frequency analysis, create another DataFrame that excludes spaces and punctuation:

word_annotations = []

for text_file, doc in zip(text_files, docs):
    for token in doc:
        if not token.is_space and not token.is_punct:
            word_annotations.append(
                {
                    "file": text_file.name,
                    "token": token.text,
                    "lemma": token.lemma_,
                    "upos": token.pos_,
                    "xpos": token.tag_,
                }
            )

word_df = pd.DataFrame(word_annotations)

display(word_df)

This DataFrame includes only word tokens.

8. Count UPOS tags in each text

Group the data by filename and UPOS tag:

upos_counts = (
    word_df
    .groupby(["file", "upos"])
    .size()
    .reset_index(name="count")
)

display(upos_counts)

9. Create a wide-format UPOS table

To show one text per row and one UPOS category per column:

upos_table = (
    upos_counts
    .pivot(
        index="file",
        columns="upos",
        values="count",
    )
    .fillna(0)
    .astype(int)
    .reset_index()
)

display(upos_table)

Count XPOS tags in each text

Try it yourself!

Create a wide-format XPOS table

Try it yourself!

Saving the Results

Please submit the following files:

Token-level annotation files
UPOS frequency counts
XPOS frequency counts

Place all output files in a single folder and compress the folder as a .zip file.

File-Naming Conventions

Use the following naming conventions:

Token-level annotations: token_annotations.csv
UPOS counts: upos_counts.csv
XPOS counts: xpos_counts.csv
Wide-format UPOS table: all_texts_upos_counts.csv
Wide-format XPOS table: all_texts_xpos_counts.csv
Final compressed folder: lastname_firstname_pos_analysis.zip (Replace textname with the original text filename and lastname_firstname with your own name.)

Next: Korean POS tagging