Ready for training
Ready for training
Check the dataset files
Extract asc-simple.zip and place these three files in your working folder:
train.txt: training data.dev.txt: development data for selecting the best checkpoint.test.txt: held-out data for final evaluation.
If you are using Colab, upload these three files to your notebook’s runtime using the Files panel.
Create a Python file
Now, we will create a Python file named train_tagger.py in the same folder. Add the following Python blocks to that file in order. If you are using Colab, run the blocks in notebook code cells in the same order instead.
Read the datasets
Import Path:
from pathlib import Path
Define a function that reads the token|label format:
def read_dataset(file_path):
rows = []
for line_number, line in enumerate(
Path(file_path).read_text(
encoding="utf-8-sig"
).splitlines(),
start=1,
):
line = line.strip()
if not line:
continue
tokens = []
tags = []
for item in line.split():
if "|" not in item:
raise ValueError(
f"{file_path}, line {line_number}: "
f"missing | in {item!r}"
)
token, tag = item.rsplit("|", 1)
if not token or not tag:
raise ValueError(
f"{file_path}, line {line_number}: "
f"invalid annotation {item!r}"
)
tokens.append(token)
tags.append(tag)
rows.append({
"tokens": tokens,
"tags": tags,
})
if not rows:
raise ValueError(f"{file_path} is empty.")
return rows
Load all three files:
train_rows = read_dataset("train.txt")
dev_rows = read_dataset("dev.txt")
test_rows = read_dataset("test.txt")
Check the number of sentences and inspect the first training example:
print("Training sentences:", len(train_rows))
print("Development sentences:", len(dev_rows))
print("Test sentences:", len(test_rows))
print(train_rows[0])
Each example contains two matching lists: tokens holds the words, and tags holds their labels.
Create the label mappings
The model uses numeric IDs to represent labels. First, collect the unique labels from the training data:
label_list = sorted({
tag
for row in train_rows
for tag in row["tags"]
})
print(label_list)
The simplified dataset contains eight labels:
['CAUS_MOT', 'DITRAN', 'INTRAN_MOT', 'INTRAN_RES', 'INTRAN_S', 'O', 'TRAN_RES', 'TRAN_S']
Create mappings between labels and numeric IDs:
label2id = {
label: index
for index, label in enumerate(label_list)
}
id2label = {
index: label
for label, index in label2id.items()
}
print(label2id)
label2id converts label names into numeric IDs for training. id2label converts the model’s predicted IDs back into label names.
Check that the development and test sets use labels included in the training label inventory:
for split_name, split_rows in [
("dev", dev_rows),
("test", test_rows),
]:
unknown_labels = {
tag
for row in split_rows
for tag in row["tags"]
if tag not in label2id
}
if unknown_labels:
raise ValueError(
f"{split_name} contains labels absent from training: "
f"{sorted(unknown_labels)}"
)
Note. If this check fails, review the labels for consistency across the three files. Do not move development or test examples into the training set to resolve the error.
Organize the existing splits
Import Dataset and DatasetDict:
from datasets import Dataset, DatasetDict
Store the three predefined splits in a DatasetDict:
dataset = DatasetDict({
"train": Dataset.from_list(train_rows),
"dev": Dataset.from_list(dev_rows),
"test": Dataset.from_list(test_rows),
})
print(dataset)
This organizes the data for the next step without changing which examples belong to each split.