Build your own tagger
Build your own tagger
In 9-1 to 9-5, we trained a model with data that were already prepared. Now you will create your own data and train your own model. You do not need programming experience: you will write sentences, mark the words you care about, and run a few ready-made scripts.
The plan
- Choose 2–4 labels for a linguistic feature related to your research interest, and decide which language you will work in.
- Collect more than 100 sentences in that language. Choose a reasonable number to annotate!
- Mark the target words in each sentence.
- Run a script that splits the data 8:1:1 into training, development, and test sets (80/10/10 for 100 sentences).
- Train the model and evaluate it on the test set.
- Interpret the results and write up your submission.
Step 1. Choose your task
Your labels must be linguistic categories: grammatical or functional labels for words. Some ideas:
| Idea | Labels |
|---|---|
| Modality | MODAL (can, must, should…), HEDGE (maybe, perhaps, seems…) |
| Verb type | STATIVE, DYNAMIC |
| Parts of speech | NOUN, VERB, ADJ, ADV |
| Discourse markers | CONTRAST, CAUSE, ADDITION |
| Stance | CERTAINTY, DOUBT |
Mark the words that carry the label, for example the verb, not the whole sentence. Rules for labels:
- Use 2–4 labels. Fewer labels are easier for a small dataset.
- Write labels in capital letters, with no spaces (use
_if needed). - Do not use
Oas your own label.Ois automatically given to all unmarked words. - Write down a one-line definition for each label before you start, and follow it every time (Recall the lecture on linguistic annotation).
Step 2. Collect 100 sentences
Choose 100 sentences that you want to use. You can write them yourself or copy them from sources you are allowed to use (for example, your own writing, news, or reviews). Tips:
- Use one sentence per line, and keep them short to medium length.
- Make sure each label appears (at least 15–20 times) across the 100 sentences. With too few examples, the model cannot learn the label.
- Include different kinds of sentences, not 100 near-copies.
- Do not use private information about real people.
Step 3. Mark the words
Create a plain text file named sentences.txt (use Notepad, TextEdit in plain-text mode, or a Colab text file). Put one sentence per line. Mark each target word like this:
[word|LABEL]
Everything you do not mark automatically becomes O (no label). Example with PAST and MODAL:
Maria [moved|PAST] to Seoul last year .
We [can|MODAL] meet tomorrow , but I [must|MODAL] leave early .
They [watched|PAST] a movie and [went|PAST] home .
The weather is nice today .
A phrase of several words can go in one bracket; each word inside gets the same label:
She [has to|MODAL] finish the report .
Rules:
- Put spaces around punctuation, as in the examples above (
today .), or let the script handle it; both work. - Do not use
[,], or|anywhere else in the sentence. - Do not leave blank lines in the middle.
- Sentences that contain no labels are allowed, but most of your sentences should contain at least one.
- Save the file with UTF-8 encoding.
Your file should contain about 100 lines (at least 50). The script splits them 8:1:1, so exactly 100 gives 80/10/10.
Step 4. Split the data with a script
Create a file named prepare_data.py in the same folder as sentences.txt (in Colab, put the code in a notebook cell and upload sentences.txt first). Paste in this code:
import random
import re
from pathlib import Path
TOKEN_PATTERN = re.compile(r"\w+(?:['’]\w+)?|[^\w\s]")
MARK_PATTERN = re.compile(r"\[([^\[\]|]+)\|([^\[\]|\s]+)\]")
def add_tokens(text, label, tokens, tags):
for token in TOKEN_PATTERN.findall(text):
tokens.append(token)
tags.append(label)
def parse_line(line, line_number):
tokens = []
tags = []
position = 0
for match in MARK_PATTERN.finditer(line):
add_tokens(line[position:match.start()], "O", tokens, tags)
label = match.group(2).strip().upper()
if label == "O":
raise ValueError(f"Line {line_number}: do not use O as a label.")
add_tokens(match.group(1), label, tokens, tags)
position = match.end()
rest = line[position:]
if any(symbol in rest for symbol in "[]|"):
raise ValueError(
f"Line {line_number}: check your brackets and | marks."
)
add_tokens(rest, "O", tokens, tags)
return " ".join(f"{t}|{g}" for t, g in zip(tokens, tags)), set(tags)
lines = [
line.strip()
for line in Path("sentences.txt").read_text(encoding="utf-8-sig").splitlines()
if line.strip()
]
if len(lines) < 50:
raise ValueError(f"Found only {len(lines)} sentences. Use at least 50.")
# 8:1:1 split. Dev and test each get 10% of the sentences.
n_dev = n_test = max(1, round(len(lines) * 0.1))
n_train = len(lines) - n_dev - n_test
print(f"Split: {n_train} train, {n_dev} dev, {n_test} test")
examples = [parse_line(line, number) for number, line in enumerate(lines, 1)]
all_labels = set().union(*(labels for _, labels in examples)) - {"O"}
print("Your labels:", sorted(all_labels))
if not all_labels:
raise ValueError("No labels found. Mark words as [word|LABEL].")
# Try different shuffles until every label appears in all three files.
for seed in range(1000):
shuffled = examples[:]
random.Random(seed).shuffle(shuffled)
splits = {
"train": shuffled[:n_train],
"dev": shuffled[n_train:n_train + n_dev],
"test": shuffled[n_train + n_dev:],
}
if all(
all_labels <= set().union(*(labels for _, labels in rows))
for rows in splits.values()
):
break
else:
raise ValueError(
"Could not give every label to train, dev, and test. "
"Add more examples of your rarest label."
)
for name, rows in splits.items():
Path(f"{name}.txt").write_text(
"\n".join(text for text, _ in rows) + "\n", encoding="utf-8"
)
print(name, len(rows), "sentences")
Run it:
python prepare_data.py
If something is wrong, the script tells you the line number. Fix sentences.txt and run it again.
When it works, you will have train.txt, dev.txt, and test.txt in the token|label format from 9-1. With 100 sentences, they contain 80, 10, and 10 sentences (an 8:1:1 split):
Maria|O moved|PAST to|O Seoul|O last|O year|O .|O
Open train.txt and check a few lines. Are the labels where you intended?
Note. Back up your original train.txt, dev.txt, and test.txt from 9-1 (or use a new folder) so they are not overwritten.
Step 5. Train and evaluate
Use the same scripts as before, with your new files in the folder:
- Follow 9-3 to read the data. The label list is created automatically from your
train.txt. You do not need to type your labels. (Ignore the text saying the data have eight labels.) -
Follow 9-4 to train. With only 80 sentences, change two settings in
TrainingArguments:num_train_epochs=10, per_device_train_batch_size=4,Also change the saved folder name from
asc_taggerto a name that describes your tagger, for exampletense_taggerormodal_tagger. Change it in both places in 9-4 (trainer.save_model(...)andtokenizer.save_pretrained(...)), and use the same name when loading the model in 9-5 (from_pretrained(...)).asc_taggeris the name for the ASC model, so do not reuse it for yours.Other languages.
distilbert-base-casedis for English. If your sentences are in another language, changemodel_namein 9-4 to"distilbert-base-multilingual-cased", which supports many languages. Everything else stays the same. Note that theprepare_data.pytokenizer splits words on spaces and punctuation using\w, which works for most alphabetic languages (Korean, Spanish, French, etc.), but checktrain.txtto make sure the words were split the way you intended. - Follow 9-5 to evaluate on
test.txt.
Step 6. Understand your results
With only 10 test sentences, scores can change a lot from small differences. Do not worry if they are not high. Look at your report and answer these questions:
- Which label has the highest F1? Which has the lowest? Check the
supportcolumn. - Open
test.txtand find a word the model probably got wrong. Why might it be hard? - Were your own labels always consistent? Would a different person label the same words the same way?
- What would you change to improve the model (more sentences, clearer label definitions, more varied examples)?
Optional: try your model on a new sentence
Add this to the end of eval_tagger.py, or run it in a new cell:
from transformers import pipeline
tagger = pipeline(
"token-classification",
model="your_tagger_name",
aggregation_strategy="simple",
)
for item in tagger("Write your own new sentence here."):
print(item["word"], item["entity_group"], round(float(item["score"]), 2))
Replace "your_tagger_name" with your saved model folder name, and replace the example sentence with your own. Words not printed were labeled O.
Submission_note
For Assignment 4 (section Your tagger: Submission note), submit the following. Items 1–3 can be written in one document (in .txt, .md, .docx).
1. Labels
For each label you used (2–4 labels), give:
- the label name;
- a one-sentence definition (which words get this label, and which do not);
- two example sentences from your data, with the labeled word(s) marked;
- how many times it appears in
train.txt,dev.txt, andtest.txt. (Check thesupportcolumn intest_report.txtfor the test set.)
2. Sentence source
- Where did your annotated sentences come from? (for example, your own writing, a news site, a novel, a class corpus). Give the title, author or website, and URL or date accessed.
- How did you choose them? (for example, random, first 100 sentences, or selected because they contain the target words)
- Did you edit or write any sentences yourself? If so, how many?
3. Tagger description
- Tagger name (the name of your saved model folder).
- What does it do? (one sentence, for example “It tags past-tense and modal verbs in English sentences.”)
- Base model:
distilbert-base-cased(English) ordistilbert-base-multilingual-cased(other languages), and the language of your sentences. - Training settings you used (number of epochs and batch size).
- Results: token accuracy and the precision, recall, and F1 for each of your labels (copy from
test_report.txt). - Your answers to the five questions in Step 6.