Back: Training

Evaluation

How well does our trained model label new sentences?

To find out, we will:

  1. Load the model saved during training.
  2. Predict labels for the test sentences.
  3. Compare those predictions with the correct labels.

We use test.txt for this final evaluation. These sentences were not used to train the model or select the best checkpoint.

Prepare the files

Create a file named eval_tagger.py in your working folder. Your folder should contain:

eval_tagger.py
test.txt
asc_tagger/
(train.txt)
(dev.txt)
(train_tagger.py)

asc_tagger is the folder containing the model and tokenizer saved during training.

Add the following Python blocks to evaluate_tagger.py in order. If you are using Google Colab, run them in code cells instead, with test.txt and asc_tagger available in the current working folder.

1. Load the trained model

Import the packages:

from pathlib import Path

import numpy as np
from datasets import Dataset
from sklearn.metrics import accuracy_score, classification_report
from transformers import (
    AutoTokenizer,
    AutoModelForTokenClassification,
    DataCollatorForTokenClassification,
    Trainer,
    TrainingArguments,
)

Load your saved model and tokenizer:

tokenizer = AutoTokenizer.from_pretrained("asc_tagger")

model = AutoModelForTokenClassification.from_pretrained(
    "asc_tagger"
)

label2id = model.config.label2id
id2label = model.config.id2label

We load asc_tagger, rather than distilbert-base-cased, because we want to evaluate the model we trained.

The saved model also contains the label mappings. Using these mappings ensures that each numeric ID has the same meaning as it did during training.

2. Read the test data

The test data use the same token|label format as the training data:

And|O we|O packed|INTRAN_RES up|O .|O

Define a function to read the sentences and their correct labels:

def read_dataset(file_path):
    rows = []
    text = Path(file_path).read_text(encoding="utf-8-sig")

    for line_number, line in enumerate(text.splitlines(), start=1):
        if not line.strip():
            continue

        tokens = []
        tags = []

        for item in line.split():
            if "|" not in item:
                raise ValueError(
                    f"Line {line_number}: missing | in {item!r}"
                )

            token, tag = item.rsplit("|", 1)

            if not token or tag not in label2id:
                raise ValueError(
                    f"Line {line_number}: invalid annotation {item!r}"
                )

            tokens.append(token)
            tags.append(tag)

        rows.append({"tokens": tokens, "tags": tags})

    if not rows:
        raise ValueError(f"{file_path} is empty.")

    return rows

Load the file:

test_rows = read_dataset("test.txt")
test_dataset = Dataset.from_list(test_rows)

print("Test sentences:", len(test_rows))

3. Prepare the sentences for the model

As in training, the tokenizer may split a word into several subtokens. We use the first subtoken to evaluate each word.

We assign -100 to special tokens and additional subtokens so that we can exclude them from scoring.

def tokenize_and_align_labels(examples):
    encoded = tokenizer(
        examples["tokens"],
        is_split_into_words=True,
        truncation=False,
    )

    max_length = model.config.max_position_embeddings

    for input_ids in encoded["input_ids"]:
        if len(input_ids) > max_length:
            raise ValueError(
                "A test sentence is too long for the model. "
                "Split it into shorter labeled sequences "
                "before evaluation."
            )

    all_labels = []

    for sentence_index, tags in enumerate(examples["tags"]):
        word_ids = encoded.word_ids(
            batch_index=sentence_index
        )

        label_ids = []
        previous_word_id = None

        for word_id in word_ids:
            if word_id is None or word_id == previous_word_id:
                label_ids.append(-100)
            else:
                label_ids.append(label2id[tags[word_id]])

            previous_word_id = word_id

        all_labels.append(label_ids)

    encoded["labels"] = all_labels
    return encoded

Unlike the training code, this function stops if a sentence exceeds the model’s length limit. This prevents us from silently leaving words out of the evaluation.

Apply the function:

tokenized_test = test_dataset.map(
    tokenize_and_align_labels,
    batched=True,
    remove_columns=["tokens", "tags"],
)

4. Generate predictions

Prepare the trainer:

data_collator = DataCollatorForTokenClassification(
    tokenizer=tokenizer,
)

evaluation_args = TrainingArguments(
    output_dir="evaluation_results",
    per_device_eval_batch_size=8,
    report_to="none",
    dataloader_num_workers=0,
)

trainer = Trainer(
    model=model,
    args=evaluation_args,
    processing_class=tokenizer,
    data_collator=data_collator,
)

Here, we use Trainer only to make predictions. We do not train the model again.

prediction_output = trainer.predict(tokenized_test)

For each position, the model returns a score for every possible label. Select the label with the highest score:

predicted_ids = np.argmax(
    prediction_output.predictions,
    axis=-1,
)

correct_ids = prediction_output.label_ids

Remove positions marked with -100, including padding:

mask = correct_ids != -100

correct_labels = correct_ids[mask]
predicted_labels = predicted_ids[mask]

We now have two matching lists of label IDs: the correct labels and the model’s predictions, with one entry per original word.

5. Compare predictions with the correct labels

First, calculate overall accuracy:

accuracy = accuracy_score(
    correct_labels,
    predicted_labels,
)

print(f"Token accuracy (including O): {accuracy:.4f}")

An accuracy of 0.95 means that 95% of the words received the correct label.

However, many words have the label O. A model can therefore achieve high overall accuracy while still making mistakes on ASC labels.

To examine ASC performance more closely, create a report for each category:

asc_label_ids = [
    label_id
    for label_id in sorted(id2label)
    if id2label[label_id] != "O"
]

asc_label_names = [
    id2label[label_id]
    for label_id in asc_label_ids
]

report = classification_report(
    correct_labels,
    predicted_labels,
    labels=asc_label_ids,
    target_names=asc_label_names,
    digits=4,
    zero_division=0,
)

print(report)

For each ASC category, the report shows:

Score Meaning
Precision Of the words predicted as this category, how many were correct?
Recall Of the words that truly belong to this category, how many did the model find?
F1-score A score that combines precision and recall.
Support The number of words with this correct label in the test data.

Precision, recall, and F1 range from 0 to 1. Higher scores are better.

The report does not display a separate row for O, but those words still count when measuring mistakes. For example, predicting TRAN_S for a word whose correct label is O lowers precision for TRAN_S.

6. Save the results

Save the accuracy and classification report to a text file:

results_text = (
    f"Token accuracy (including O): {accuracy:.4f}\n\n"
    f"ASC classification report:\n{report}"
)

output_dir = Path("evaluation_results")
output_dir.mkdir(parents=True, exist_ok=True)

report_path = output_dir / "test_report.txt"
report_path.write_text(results_text, encoding="utf-8")

print(f"Results saved to {report_path}")

Run the evaluation

Save eval_tagger.py. Run this command from your regular terminal, not Python’s >>> prompt:

python eval_tagger.py

If you are using Google Colab, run the Python code cells in order instead. The results will appear on screen and be saved in:

evaluation_results/test_report.txt

Understand the overall scores

Look at the scores for individual ASC categories as well as these summary rows:

  • Micro average: combines the counts of correct predictions and errors across ASC categories.
  • Macro average: gives every ASC category equal weight.
  • Weighted average: gives more weight to categories with more correct-label examples in the test set.

Back to Module 9