Back: Evaluation

Build your own tagger

In 9-1 to 9-5, we trained a model with data that were already prepared. Now you will create your own data and train your own model. You do not need programming experience: you will write sentences, mark the words you care about, and run a few ready-made scripts.

The plan

  1. Choose 2–4 labels for a linguistic feature related to your research interest, and decide which language you will work in.
  2. Collect more than 100 sentences in that language. Choose a reasonable number to annotate!
  3. Mark the target words in each sentence.
  4. Run a script that splits the data 8:1:1 into training, development, and test sets (80/10/10 for 100 sentences).
  5. Train the model and evaluate it on the test set.
  6. Interpret the results and write up your submission.

Step 1. Choose your task

Your labels must be linguistic categories: grammatical or functional labels for words. Some ideas:

Idea Labels
Modality MODAL (can, must, should…), HEDGE (maybe, perhaps, seems…)
Verb type STATIVE, DYNAMIC
Parts of speech NOUN, VERB, ADJ, ADV
Discourse markers CONTRAST, CAUSE, ADDITION
Stance CERTAINTY, DOUBT

Mark the words that carry the label, for example the verb, not the whole sentence. Rules for labels:

  • Use 2–4 labels. Fewer labels are easier for a small dataset.
  • Write labels in capital letters, with no spaces (use _ if needed).
  • Do not use O as your own label. O is automatically given to all unmarked words.
  • Write down a one-line definition for each label before you start, and follow it every time (Recall the lecture on linguistic annotation).

Step 2. Collect 100 sentences

Choose 100 sentences that you want to use. You can write them yourself or copy them from sources you are allowed to use (for example, your own writing, news, or reviews). Tips:

  • Use one sentence per line, and keep them short to medium length.
  • Make sure each label appears (at least 15–20 times) across the 100 sentences. With too few examples, the model cannot learn the label.
  • Include different kinds of sentences, not 100 near-copies.
  • Do not use private information about real people.

Step 3. Mark the words

Create a plain text file named sentences.txt (use Notepad, TextEdit in plain-text mode, or a Colab text file). Put one sentence per line. Mark each target word like this:

[word|LABEL]

Everything you do not mark automatically becomes O (no label). Example with PAST and MODAL:

Maria [moved|PAST] to Seoul last year .
We [can|MODAL] meet tomorrow , but I [must|MODAL] leave early .
They [watched|PAST] a movie and [went|PAST] home .
The weather is nice today .

A phrase of several words can go in one bracket; each word inside gets the same label:

She [has to|MODAL] finish the report .

Rules:

  • Put spaces around punctuation, as in the examples above (today .), or let the script handle it; both work.
  • Do not use [, ], or | anywhere else in the sentence.
  • Do not leave blank lines in the middle.
  • Sentences that contain no labels are allowed, but most of your sentences should contain at least one.
  • Save the file with UTF-8 encoding.

Your file should contain about 100 lines (at least 50). The script splits them 8:1:1, so exactly 100 gives 80/10/10.

Step 4. Split the data with a script

Create a file named prepare_data.py in the same folder as sentences.txt (in Colab, put the code in a notebook cell and upload sentences.txt first). Paste in this code:

import random
import re
from pathlib import Path

TOKEN_PATTERN = re.compile(r"\w+(?:['’]\w+)?|[^\w\s]")
MARK_PATTERN = re.compile(r"\[([^\[\]|]+)\|([^\[\]|\s]+)\]")


def add_tokens(text, label, tokens, tags):
    for token in TOKEN_PATTERN.findall(text):
        tokens.append(token)
        tags.append(label)


def parse_line(line, line_number):
    tokens = []
    tags = []
    position = 0

    for match in MARK_PATTERN.finditer(line):
        add_tokens(line[position:match.start()], "O", tokens, tags)
        label = match.group(2).strip().upper()
        if label == "O":
            raise ValueError(f"Line {line_number}: do not use O as a label.")
        add_tokens(match.group(1), label, tokens, tags)
        position = match.end()

    rest = line[position:]
    if any(symbol in rest for symbol in "[]|"):
        raise ValueError(
            f"Line {line_number}: check your brackets and | marks."
        )
    add_tokens(rest, "O", tokens, tags)

    return " ".join(f"{t}|{g}" for t, g in zip(tokens, tags)), set(tags)


lines = [
    line.strip()
    for line in Path("sentences.txt").read_text(encoding="utf-8-sig").splitlines()
    if line.strip()
]

if len(lines) < 50:
    raise ValueError(f"Found only {len(lines)} sentences. Use at least 50.")

# 8:1:1 split. Dev and test each get 10% of the sentences.
n_dev = n_test = max(1, round(len(lines) * 0.1))
n_train = len(lines) - n_dev - n_test
print(f"Split: {n_train} train, {n_dev} dev, {n_test} test")

examples = [parse_line(line, number) for number, line in enumerate(lines, 1)]
all_labels = set().union(*(labels for _, labels in examples)) - {"O"}
print("Your labels:", sorted(all_labels))

if not all_labels:
    raise ValueError("No labels found. Mark words as [word|LABEL].")

# Try different shuffles until every label appears in all three files.
for seed in range(1000):
    shuffled = examples[:]
    random.Random(seed).shuffle(shuffled)
    splits = {
        "train": shuffled[:n_train],
        "dev": shuffled[n_train:n_train + n_dev],
        "test": shuffled[n_train + n_dev:],
    }
    if all(
        all_labels <= set().union(*(labels for _, labels in rows))
        for rows in splits.values()
    ):
        break
else:
    raise ValueError(
        "Could not give every label to train, dev, and test. "
        "Add more examples of your rarest label."
    )

for name, rows in splits.items():
    Path(f"{name}.txt").write_text(
        "\n".join(text for text, _ in rows) + "\n", encoding="utf-8"
    )
    print(name, len(rows), "sentences")

Run it:

python prepare_data.py

If something is wrong, the script tells you the line number. Fix sentences.txt and run it again.

When it works, you will have train.txt, dev.txt, and test.txt in the token|label format from 9-1. With 100 sentences, they contain 80, 10, and 10 sentences (an 8:1:1 split):

Maria|O moved|PAST to|O Seoul|O last|O year|O .|O

Open train.txt and check a few lines. Are the labels where you intended?

Note. Back up your original train.txt, dev.txt, and test.txt from 9-1 (or use a new folder) so they are not overwritten.

Step 5. Train and evaluate

Use the same scripts as before, with your new files in the folder:

  1. Follow 9-3 to read the data. The label list is created automatically from your train.txt. You do not need to type your labels. (Ignore the text saying the data have eight labels.)
  2. Follow 9-4 to train. With only 80 sentences, change two settings in TrainingArguments:

    num_train_epochs=10,
    per_device_train_batch_size=4,
    

    Also change the saved folder name from asc_tagger to a name that describes your tagger, for example tense_tagger or modal_tagger. Change it in both places in 9-4 (trainer.save_model(...) and tokenizer.save_pretrained(...)), and use the same name when loading the model in 9-5 (from_pretrained(...)). asc_tagger is the name for the ASC model, so do not reuse it for yours.

    Other languages. distilbert-base-cased is for English. If your sentences are in another language, change model_name in 9-4 to "distilbert-base-multilingual-cased", which supports many languages. Everything else stays the same. Note that the prepare_data.py tokenizer splits words on spaces and punctuation using \w, which works for most alphabetic languages (Korean, Spanish, French, etc.), but check train.txt to make sure the words were split the way you intended.

  3. Follow 9-5 to evaluate on test.txt.

Step 6. Understand your results

With only 10 test sentences, scores can change a lot from small differences. Do not worry if they are not high. Look at your report and answer these questions:

  1. Which label has the highest F1? Which has the lowest? Check the support column.
  2. Open test.txt and find a word the model probably got wrong. Why might it be hard?
  3. Were your own labels always consistent? Would a different person label the same words the same way?
  4. What would you change to improve the model (more sentences, clearer label definitions, more varied examples)?

Optional: try your model on a new sentence

Add this to the end of eval_tagger.py, or run it in a new cell:

from transformers import pipeline

tagger = pipeline(
    "token-classification",
    model="your_tagger_name",
    aggregation_strategy="simple",
)

for item in tagger("Write your own new sentence here."):
    print(item["word"], item["entity_group"], round(float(item["score"]), 2))

Replace "your_tagger_name" with your saved model folder name, and replace the example sentence with your own. Words not printed were labeled O.

Submission_note

For Assignment 4 (section Your tagger: Submission note), submit the following. Items 1–3 can be written in one document (in .txt, .md, .docx).

1. Labels

For each label you used (2–4 labels), give:

  • the label name;
  • a one-sentence definition (which words get this label, and which do not);
  • two example sentences from your data, with the labeled word(s) marked;
  • how many times it appears in train.txt, dev.txt, and test.txt. (Check the support column in test_report.txt for the test set.)

2. Sentence source

  • Where did your annotated sentences come from? (for example, your own writing, a news site, a novel, a class corpus). Give the title, author or website, and URL or date accessed.
  • How did you choose them? (for example, random, first 100 sentences, or selected because they contain the target words)
  • Did you edit or write any sentences yourself? If so, how many?

3. Tagger description

  • Tagger name (the name of your saved model folder).
  • What does it do? (one sentence, for example “It tags past-tense and modal verbs in English sentences.”)
  • Base model: distilbert-base-cased (English) or distilbert-base-multilingual-cased (other languages), and the language of your sentences.
  • Training settings you used (number of epochs and batch size).
  • Results: token accuracy and the precision, recall, and F1 for each of your labels (copy from test_report.txt).
  • Your answers to the five questions in Step 6.

Back to Module 9