POS tagging 2. English POS tagging
English POS tagging with spaCy
(Review) Installing spaCy and the English pipeline
Install spaCy from a terminal or command prompt:
pip install -U spacy
Then download the English transformer pipeline:
python -m spacy download en_core_web_trf
Loading the pipeline
Import spaCy and load the pipeline:
import spacy
nlp_en = spacy.load("en_core_web_trf")
Inspect the installed spaCy version and pipeline information (Recording versions is important because NLP predictions may differ across versions):
print("spaCy version:", spacy.__version__)
print("Pipeline name:", nlp_en.meta.get("name"))
print("Pipeline version:", nlp_en.meta.get("version"))
print("Components:", nlp_en.pipe_names)
Processing an English sentence
Create an English text:
english_text = (
"The students were carefully writing "
"their final research papers."
)
Process the text with the pipeline:
english_doc = nlp_en(english_text)
The output is a spaCy Doc object. The Doc contains a sequence of Token objects.
print(type(english_doc))
print(type(english_doc[0]))
You can print the tokenized text with a loop:
for token in english_doc:
print(token.text)
Inspecting part-of-speech annotations
Each Token stores multiple linguistic annotations:
for token in english_doc:
print(
token.text,
token.lemma_,
token.pos_,
token.tag_,
token.morph,
sep="\t"
)
The attributes have the following meanings:
token.text: the original token form;token.lemma_: the lemma assigned to the token;token.pos_: the coarse-grained Universal POS category;token.tag_: a more detailed English part-of-speech tag;token.morph: morphological features predicted for the token.
For example, an output may associate writing with:
text: writing
lemma: write
UPOS: VERB
tag: VBG
Creating and ssaving an analysis table
First, import pandas:
import pandas as pd
Pandas is a libary for organizing/analyzing tabular data. Here pd is a commonly used abbrevication for pandas.
Create a DataFrame containing the token-level annotations:
english_annotations = pd.DataFrame(
[ { "token": token.text,
"lemma": token.lemma_,
"upos": token.pos_,
"tag": token.tag_,
"morph": str(token.morph),
}
for token in english_doc
]
)
display(english_annotations)
Save the DataFrame as a CSV file
english_annotations.to_csv(
"english_annotations.csv",
index=False, # prevents from adding row numbers as a separate column
encoding="utf-8-sig", # helps presenting non-English characters
)
English: UPOS vs. XPOS tags
Consider the following forms:
write
writes
wrote
writing
written
They may all be assigned the universal category VERB, but their detailed English tags can distinguish base, present-tense, past-tense, and participial forms.
Process the forms in context:
verb_text = (
"I write every day. "
"She writes every day. "
"They wrote yesterday. "
"We are writing now. "
"The report was written yesterday."
)
verb_doc = nlp_en(verb_text)
for token in verb_doc:
if token.lemma_ == "write":
print(
token.text,
token.lemma_,
token.pos_,
token.tag_,
token.morph,
sep="\t"
)
The universal tag gives a broad grammatical category. The detailed tag and morphological features provide additional information about the form.
Context-sensitive tagging
POS tagging is contextual. Consider the word record:
context_text = "Researchers record speech and examine each record."
context_doc = nlp_en(context_text)
for token in context_doc:
if token.text.lower() == "record":
print(
token.text,
token.lemma_,
token.pos_,
token.tag_,
sep="\t"
)
The two occurrences have the same spelling but different grammatical functions. The first functions as a verb, while the second functions as a noun.
Try another ambiguous form:
saw_doc = nlp_en("I saw a saw.")
for token in saw_doc:
print(
token.text,
token.lemma_,
token.pos_,
token.tag_,
sep="\t"
)
Excluding punctuation and spaces
Many analyses exclude punctuation and space tokens:
for token in english_doc:
if not token.is_space and not token.is_punct:
print(token.text, token.pos_)
The conditions mean:
not token.is_space → keep tokens that are not spaces
not token.is_punct → keep tokens that are not punctuation
Extracting nouns
english_text2 = "Maria bought a book at the university library."
english_doc2 = english_nlp(english_text2)
Extract the lemmas of all common nouns:
english_nouns = [
token.lemma_.lower()
for token in english_doc2
if token.pos_ == "NOUN"
]
print(english_nouns)
This excludes proper nouns because spaCy normally assigns them the UPOS category PROPN.
To include both common and proper nouns:
english_nouns_and_names = [
token.lemma_.lower()
for token in english_doc2
if token.pos_ in [___, ___]
]
print(english_nouns_and_names)
Note. You can modify this code to extract verbs, auxiliary verbs, or other word classes by changing the POS tags in the filtering condition.
Counting part-of-speech categories
Use Counter to calculate the frequency of each UPOS category:
from collections import Counter
english_pos_counts = Counter(
token.pos_
for token in english_doc
if not token.is_space and not token.is_punct
)
print(english_pos_counts)
To display the categories from most to least frequent:
for pos, frequency in english_pos_counts.most_common():
print(pos, frequency)
Processing multiple texts efficiently
For a collection of texts, spaCy’s nlp.pipe() method is generally more efficient than calling the pipeline separately for every text.
english_texts = [
"The first student revised the essay.",
"The second student added several examples.",
"The third student explained the argument clearly."
]
for doc in nlp_en.pipe(english_texts):
lemmas = [
token.lemma_.lower()
for token in doc
if not token.is_space and not token.is_punct
]
print(lemmas)