HF 102 · Make a model yours

HF 101 called models other people had trained. This course trains one: the base model those were built on, distilbert-base-uncased, gets a new two output head and two passes over 640 labelled movie reviews, on a CPU, in about twenty seconds. Then the course does the thing most fine-tuning tutorials skip. It scores the result on 400 reviews the model never saw, puts the model nobody here trained beside it, and finds out that the fine-tune lost, 81.5% to 89.25%, by a margin the sample cannot explain away.

That verdict is the point of the course rather than its failure. Every lesson after the third is about earning it honestly: the two loss curves and where the held out one turns, a floor and a bar under every score, a set intersection that catches a leaked test set, a model card read for what the fine-tune inherited, and a final table in rows and seconds that says which model to ship and why. The recommendation this time is not to use the model you built, and knowing when that is the answer is what fine-tuning is for.

How this course teaches

Every lesson is the same twenty six cells, and nine of them are yours.

• A prediction you commit to before the cell runs. It is graded on the reasoning, not the guess, and being wrong here is the point.
• Warmups: a one line blank or a two to four line exercise under the theory it practices, each with a four rung hint ladder behind it, where the last rung explains and still does not hand over the code.
• An exercise that is broken when you open it.
• A diagnose cell: code that runs, prints a confident and plausible number, and is wrong. Something below it refuses the number by computing the same thing a second way, so nothing is taken on trust.
• A challenge that ends in a sentence you write. The tutor grades the sentence, which means a green tick you earned for the wrong reason can be taken back.

No cell in this course passes in the state it ships. That is deliberate, and it is checked mechanically before the course is published.

The particular danger of this subject is a number that flatters the model you trained. A dataset sorted by label reports a majority baseline of 100%. A training loss of 0.011 reads as an achievement. A score on the training rows beats the baseline by three points. A held out set that leaked into the training rows scores 95%. A card's 91.3% gets quoted as the bar on data it was never measured on. Every diagnose cell in this course is one of those, and every cross check is the second route that refuses it.

What you will be able to do

• Load a labelled dataset, read its rows and features, count its labels in every split, and cut it with seeds so that nothing in training ever touches the rows you score on.
• Tokenize a whole dataset in one call, choose a window by measuring rather than guessing, pad per batch instead of per dataset, and read back what a truncation dropped.
• Put a new head on a pretrained model, set up and run a fine-tune with Trainer on a CPU, read both curves out of its log, and save and reload what you trained.
• Score against a floor and a bar on the same rows, compare two models with a disagreement table, size the noise on a sample, and write a verdict that goes against the model you trained.
• Find the epoch where the held out loss turns, price the epochs past it, and choose the number of epochs from the right curve.
• Read a model card for the four facts that matter, list what a fine-tune inherits and prove it, and reproduce a documented bias in eight sentences.
• Put every model you trained in one table with the one you did not, and write the recommendation the numbers support.

The lessons

1. The datasets library. rotten_tomatoes: 8,530 training reviews, 1,066 validation, 1,066 test, labelled neg and pos, exactly balanced. The file is sorted by label, so the first 800 rows are all positive and a quick test set cut from the front reports a majority baseline of 100%. Shuffle with a seed, then select: 419 and 381. Cut with a seed: 640 and 160, and a different seed keeps only 34 of the 160. The course's TEST is 400 shuffled rows of the official test split, which shares no review with the training split.

2. Tokenizing a dataset. map(encode, batched=True) tokenizes 800 reviews in about a tenth of a second. The median review is 27 tokens, nine in ten are under 43, and one of the 800 is 78: a window of 64 cuts exactly one, and the fourteen tokens it drops are a sentence of Portuguese. Padding every row to 64 makes 56.9% of the tensor padding; a collator pads each batch to its own longest row, 37 then 38. The longest review by words is 55, which is why a word count says 64 is safe and the tokenizer disagrees.

3. Fine-tuning. A random head of 592,130 parameters on 66,362,880 trained ones scores 54.2% before training. Two epochs over 640 rows, 80 steps, about twenty seconds on a CPU: the training loss falls from 0.586 to 0.287, the held out loss falls to 0.497 and then rises to 0.573. The model scores 81.5% on the 400 test rows, 93.8% on its own 640, saves as four files and 269 MB, and reloads to the same logits. A learning rate a hundred times too large trains in the same time to 52.5%, which is the majority share: one label for everything.

4. Evaluating honestly. The floor is 50.25%; the bar is the off the shelf model at 89.25% on the same rows; the fine-tune is 81.5%. The two models disagree on 65 reviews and the off the shelf one wins 48 of them; the gap is about three standard errors. Scored on its own training rows the fine-tune wins, 93.75% to 90.6%, and a set intersection catches the rigged comparison. The verdict is written down as it came out.

5. Small-data overfitting. 160 rows, eight epochs. The training loss falls every epoch to 0.011 and the model scores 100% on its own rows. The held out loss bottoms out at 0.498 in the third epoch and reaches 0.895 in the eighth, eighty percent higher; the held out accuracy peaks at epoch 2 and ends seven points lower. Choosing the number of epochs by the training loss picks eight, every time, on any data; choosing by the held out loss picks three, and a three epoch run scores the same on the test rows as the eight epoch one.

6. Model cards and limits. The base card: 11,038 books and English Wikipedia, Apache 2.0, and an occupation example the lesson reproduces: carpenter for the man, nurse for the woman. The off the shelf card documents a country bias its own fine-tune added: This film was filmed in Italy. is positive at 0.974 and ... in Iraq. at 0.023. Your fine-tune inherits the tokenizer, the casing, the 512 token window and the license by construction, and swings by 0.1 across the same eight countries where the off the shelf model swings by 0.95: that bias was learned in the other fine-tune, not the base. The card's 91.3% is 89.25% on your reviews.

7. The verdict. Four models on the same 400 rows. The shelf model at 89.25% with no training; the base on 640 rows at 81.5% in twenty seconds; the base on 2,000 fresh rows at 82.75% in a minute, about 0.9 points per thousand rows; the shelf model fine-tuned one epoch on the 640 rows at 90.25% in ten seconds, a point above the shelf and inside the noise. Taking the 2,000 rows from the front of the same shuffle puts every held out review into the training set. The recommendation: ship the off the shelf model, start from the closest checkpoint if you fine-tune, and measure the shelf before training anything.

The models and the data

	Size	Used in
distilbert-base-uncased	268 MB	lessons 2 to 7 (the base being fine-tuned; lesson 6 also as fill-mask)
distilbert-base-uncased-finetuned-sst-2-english	268 MB	lessons 4, 6, 7 (the off the shelf bar)
cornell-movie-review-data/rotten_tomatoes	1 MB	every lesson

Every model and the dataset are pinned to one commit, so the numbers in the prose stay the numbers you see. The first run of each lesson downloads what it needs into ~/.cache/huggingface; every later run reads the cache. Every setup cell catches the download error and says so in a sentence.

Training runs on the CPU on purpose, use_cpu=True in every TrainingArguments, so that the same seed gives the same curves on every machine the course runs on. The heaviest lesson, the verdict, trains three models and takes about two minutes on a laptop CPU with eight threads, and several times longer on a slow one. Nothing here needs a GPU.

Where this course sits

Eleventh course of the ivyx AI series and the last of the deep learning wave. HF 101 is the prerequisite: pipelines, tokenizers and logits. The training loop, the two curves and what overfitting looks like are PYTORCH 102's; the held out split and the baseline are ML 101's.

The series ends here. What you trained lost to what was already there, and the course was built so that you could say so with the numbers in hand, which is the only way a fine-tune ever earns its place.

Requirements

Python 3 with torch, transformers, datasets and accelerate, and a network connection the first time each lesson runs. accelerate is what Trainer needs to run on PyTorch at all; without it lesson 3's TrainingArguments raises an ImportError naming it.

You install those packages yourself. The list above is a statement of what the lessons import, not an installer: nothing in this product installs a package or checks whether one is importable, so a learner opening this course against their own kernel or a bare virtualenv would otherwise meet an ImportError in the second cell of lesson 1. Every lesson's setup cell catches that and re-raises it carrying the command:

pip install torch --index-url https://download.pytorch.org/whl/cpu
pip install transformers datasets accelerate

The index url matters on Linux, where a plain pip install torch resolves to the CUDA build and drags in several gigabytes of nvidia-* wheels that a laptop downloads and cannot use. On the ivyx platform kernel torch, transformers and datasets are already there. On a Mac, Trainer would otherwise move the model to the Apple GPU and every forward pass you write by hand afterwards would fail with a device mismatch; use_cpu=True is what keeps the model where your tensors are.