Back: English dependency parsing

CoNLL-U format

It is useful to become familiar with the CoNLL-U format, a widely used format for representing token-level linguistic annotations in NLP. Each token is represented on a separate line, with columns for information such as the token form, lemma, POS tags, morphological features, syntactic head, and dependency relation.

In CoNLL-U, the syntactic head of each token is represented by its token ID.

For example:

1   Students   student   NOUN   NNS   Number=Plur   2   nsubj   _   _
2   analyzed   analyze   VERB   VBD   Tense=Past    0   ROOT    _   _
3   data       datum     NOUN   NNS   Number=Plur   2   dobj    _   _

Here:

  • token 1 (Students) depends on token 2 (analyzed);
  • token 3 (data) also depends on token 2;
  • HEAD = 0 indicates that token 2 is the root of the sentence.

Creating CoNLL-U output

Use the following function to convert a spaCy Doc into CoNLL-U-style output:

def doc_to_conllu(doc):

    lines = []

    for token in doc:

        if token.dep_ == "ROOT":
            head_id = 0
        else:
            head_id = token.head.i + 1

        feats = str(token.morph)
        if feats == "":
            feats = "_"

        columns = [
            str(token.i + 1),  # ID
            token.text,        # FORM
            token.lemma_,      # LEMMA
            token.pos_,        # UPOS
            token.tag_,        # XPOS
            feats,             # FEATS
            str(head_id),      # HEAD
            token.dep_,        # DEPREL
            "_",               # DEPS
            "_"                # MISC
        ]

        lines.append("\t".join(columns))

    return "\n".join(lines)

Try it:

doc = nlp_en(
    "The students carefully analyzed the new dataset."
)

print(doc_to_conllu(doc))

Generating CoNLL-U for multiple sentences

Create several sentences:

practice_sentences = [
    "The student read the article.",
    "The student read the difficult article.",
    "The student read the article carefully.",
    "The article was written by a graduate student."
]

Generate the annotations:

for doc in nlp_en.pipe(practice_sentences):
    print("# text =", doc.text)
    print(doc_to_conllu(doc))
    print()

Creating and saving a CoNLL-U file

So far, we have printed CoNLL-U-style annotations on the screen. We can also save the annotations as a .conllu file for later corpus analysis or annotation.

First, import Path:

from pathlib import Path

The following helper function ensures that missing values are represented with _:

def conllu_value(value):

    if value is None or value == "":
        return "_"

    return str(value).replace("\t", " ").replace("\n", " ")

Now create a function that processes multiple sentences and saves their annotations in a CoNLL-U file:

def save_conllu(raw_sentences, text_label, output_path):

    output_path = Path(output_path)
    conllu_lines = []

    for sentence_id, doc in enumerate(
        nlp_en.pipe(raw_sentences),
        start=1
    ):

        # Sentence-level metadata
        conllu_lines.append(
            f"# sent_id = {text_label}-{sentence_id}"
        )

        conllu_lines.append(
            f"# text = {doc.text}"
        )

        # Standard 10-column CoNLL-U format:
        # ID, FORM, LEMMA, UPOS, XPOS,
        # FEATS, HEAD, DEPREL, DEPS, MISC

        for token in doc:

            if token.dep_ == "ROOT":
                head_id = 0
            else:
                head_id = token.head.i + 1

            fields = [
                token.i + 1,       # ID
                token.text,        # FORM
                token.lemma_,      # LEMMA
                token.pos_,        # UPOS
                token.tag_,        # XPOS
                str(token.morph),  # FEATS
                head_id,           # HEAD
                token.dep_,        # DEPREL
                "_",               # DEPS
                "_"                # MISC
            ]

            conllu_lines.append(
                "\t".join(
                    conllu_value(field)
                    for field in fields
                )
            )

        # Blank line between sentences
        conllu_lines.append("")

    output_path.write_text(
        "\n".join(conllu_lines),
        encoding="utf-8"
    )

    print(f"Saved CoNLL-U file: {output_path}")

Save the practice sentences:

save_conllu(
    practice_sentences,
    text_label="practice",
    output_path="english_dependencies.conllu"
)

This creates a file named:

english_dependencies.conllu

The file contains sentence-level metadata followed by the standard ten CoNLL-U columns:

# sent_id = practice-1
# text = The student read the article.
1   The       the       DET    DT    Definite=Def|PronType=Art   2   det     _   _
2   student   student   NOUN   NN    Number=Sing                 3   nsubj   _   _
...

A blank line separates one sentence from the next.

Next: Exercises