Back: Ready for training

Training

We now fine-tune distilbert-base-cased to predict ASC labels for individual tokens.

Note. DistilBERT (Distilled BERT) is a smaller, faster version of BERT (Bidirectional Encoder Representations from Transformers), a pretrained transformer-based language model.

Load the tokenizer and model

Import the tools:

from transformers import (
    AutoTokenizer,
    AutoModelForTokenClassification,
    DataCollatorForTokenClassification,
    Trainer,
    TrainingArguments,
    set_seed,
)

Load the pretrained tokenizer and model:

set_seed(42)

model_name = "distilbert-base-cased"

tokenizer = AutoTokenizer.from_pretrained(
    model_name,
    use_fast=True,
)

model = AutoModelForTokenClassification.from_pretrained(
    model_name,
    num_labels=len(label2id),
    id2label=id2label,
    label2id=label2id,
)

The tokenizer converts the input words into the format DistilBERT expects. The model combines pretrained DistilBERT with a token classification layer for our eight labels. You may see a message that some classification-layer weights are newly initialized. This is expected: this layer has not yet learned to predict our ASC labels.

Tokenize the data and align the labels

The tokenizer may split one word into several smaller units called subtokens. Our dataset provides one label per original word, so we need to align those labels with the tokenizer’s output.

We assign each word’s label to its first subtoken. Additional subtokens and special tokens receive -100, which tells the model to ignore those positions when calculating the loss. This value is not an ASC label.

def tokenize_and_align_labels(examples):
    tokenized_inputs = tokenizer(
        examples["tokens"],
        is_split_into_words=True,
        truncation=True,
        max_length=512,
    )

    aligned_labels = []

    for batch_index, tags in enumerate(examples["tags"]):
        word_ids = tokenized_inputs.word_ids(
            batch_index=batch_index
        )

        label_ids = []
        previous_word_id = None

        for word_id in word_ids:
            if word_id is None:
                label_ids.append(-100)
            elif word_id != previous_word_id:
                label_ids.append(label2id[tags[word_id]])
            else:
                label_ids.append(-100)

            previous_word_id = word_id

        aligned_labels.append(label_ids)

    tokenized_inputs["labels"] = aligned_labels
    return tokenized_inputs

Apply this function to each split:

tokenized_dataset = dataset.map(
    tokenize_and_align_labels,
    batched=True,
    remove_columns=dataset["train"].column_names,
)

print(tokenized_dataset)

Inputs longer than 512 subtokens, including special tokens, are truncated. Tokens beyond that limit are not used.

Configure training

Define the training settings:

training_args = TrainingArguments(
    output_dir="tagger_checkpoints",
    learning_rate=3e-5,
    per_device_train_batch_size=8,
    per_device_eval_batch_size=8,
    num_train_epochs=3,
    weight_decay=0.01,
    eval_strategy="epoch",
    save_strategy="epoch",
    load_best_model_at_end=True,
    metric_for_best_model="eval_loss",
    greater_is_better=False,
    save_total_limit=2,
    report_to="none",
    dataloader_num_workers=0,
    seed=42,
)

An epoch is one pass through the training data. We will train for three epochs. To check that the full workflow runs before starting a longer training run, you can first set num_train_epochs=1. At the end of each epoch, the trainer evaluates the model on the development set and saves a checkpoint. After training, it restores the checkpoint with the lowest development loss. Development loss measures prediction error across all labeled positions, including those labeled O. It is used here to select a checkpoint; we will examine classification metrics in the evaluation tutorial.

The test set is reserved for final evaluation and is not used to select the checkpoint.

Create a data collator:

data_collator = DataCollatorForTokenClassification(
    tokenizer=tokenizer,
)

The collator pads examples to the same length within each batch. It also pads the labels with -100 so that padding does not contribute to the loss.

Create the trainer:

trainer = Trainer(
    model=model,
    args=training_args,
    train_dataset=tokenized_dataset["train"],
    eval_dataset=tokenized_dataset["dev"],
    processing_class=tokenizer,
    data_collator=data_collator,
)

Train and save the model

Add this block at the end of train_tagger.py:

if __name__ == "__main__":
    trainer.train()

    trainer.save_model("asc_tagger")
    tokenizer.save_pretrained("asc_tagger")

Save the file. From a terminal or command prompt, run the following command in the folder containing your script and data files:

python train_tagger.py

Training checkpoints are saved in tagger_checkpoints. After training, the checkpoint selected using development loss and its tokenizer are saved in asc_tagger. We will load the saved model and tokenizer in the next tutorial to make predictions and evaluate performance on the test set.

Next: Evaluation