PYTORCH 102 · Training real networks

PYTORCH 101 ended on an unfinished sentence. A linear model priced the SUV badge at 21,440 lira, the lot's own gap fell from 26,034 lira on the freshest cars to 17,242 on the tired ones, and the model had no way to say so, because a linear model holds one number per column and uses it for every car it will ever see.

This course lifts that ceiling and then spends most of its time on what the ceiling was hiding. A network that can express a trend can also express the noise, and every lesson after the third is about telling those two apart: two loss curves rather than one, three brakes measured against each other, a check that runs before the training does, and a final task where the honest answer is six cars out of eighty four.

How this course teaches

Every lesson is the same twelve cells, and five of them are yours.

• A prediction you commit to before the cell runs. It is graded on the reasoning, not the guess, and being wrong here is the point.
• A fill in the blank with one thing missing and a four rung hint ladder behind it, where the last rung explains and still does not hand over the code.
• An exercise that is broken when you open it.
• A diagnose cell: code that runs, prints a confident and plausible number, and is wrong. Something below it refuses the number by computing the same thing a second way, so nothing is taken on trust.
• A challenge that ends in a sentence you write. The tutor grades the sentence, which means a green tick you earned for the wrong reason can be taken back.

No cell in this course passes in the state it ships. That is deliberate, and it is checked mechanically before the course is published.

What you will be able to do

• Prove that two stacked Linear layers are one Linear layer, by multiplying their weight matrices and getting the single layer's weights back.
• Build an MLP, count its parameters exactly, and measure what its width is worth against the noise of reshuffling your data.
• Read the weights a network starts with, and recognise a hidden layer whose sixteen units are one unit in a sixteen unit coat.
• Train with a DataLoader, and say what a smaller batch buys and what it pays.
• Keep both loss curves for a run and find the epoch where the falling one stopped meaning anything.
• Apply dropout, weight decay and early stopping one at a time, and say what each one actually bought.
• Memorise twenty rows on purpose, in four seconds, before spending an hour on a real run.
• Save a model, load it into a network built from scratch, and know which line changes every number it produces.
• Report a classification result next to the model that has no parameters, and say what your network earned in cars rather than in percent.

The lessons

1. Why nonlinearity. nn.Sequential(nn.Linear(3, 16), nn.Linear(16, 1)) has 81 parameters where a single Linear has 4, trains the same way, and scores the same 0.0897 to four decimals. The proof is one line: W2 @ W1 comes out as the single layer's three weights, to seven decimals. Add nn.ReLU() between them and the held out loss goes to 0.0695, which is 3,620 lira of typical error down to 3,186, and the SUV badge finally costs 27,605 lira on a fresh car and 14,970 on a tired one, against the lot's own 25,901 and 16,322.

2. An MLP. A hidden layer of width units holds 5 * width + 1 numbers, which is the exact part. The measured part is stranger: on this lot width 2 beats width 8, the whole sweep from 2 to 256 spans 0.0079, and reshuffling the same data moves the same architecture by 0.1011. The sweep cannot order its own candidates, and knowing that is the result.

3. Initialization. Set every weight and bias to 0.10 and the model still trains, to 0.0799 held out, which beats the straight line from lesson 1. Sixteen identical units receive identical gradients and stay identical forever, so 81 parameters do the work of 6. Nothing in the loss says so. torch.unique on the weight and bias rows says so in one line, before training starts.

4. Batches. A step is an update, an epoch is a pass, and full batch training made them the same number. Batches of 32 reach the same training loss in 35 epochs rather than 366, using 385 updates rather than 366: the saving is in passes over the data, and per update the small batch is slightly worse. 336 cars at 32 per batch is 11 batches, the last holding 16, and averaging the eleven batch losses gives 0.0769 where the training loss is 0.0687.

5. Reading loss curves. One run, 300 epochs, two histories. The training loss falls the whole way and bottoms out at epoch 296. The validation loss turns at epoch 13 and is 36 percent worse by the end. The bottom is not a point but a region: 25 epochs within five percent of the best, scattered from 8 to 42. The cost of watching only the curve your loop prints is 554 lira per car.

6. Regularization. Dropout, weight decay and early stopping, one at a time. None of them finds a much better model: the best validation loss moves 0.0759 to 0.0745 to 0.0733. All of them change what the run is: the penalty for training to epoch 300 instead of stopping at the turn falls from 36 percent to 14 to 5. The decayed run's last epoch is within 23 lira per car of the plain run's best epoch, and nobody had to watch it.

7. Overfit on purpose. Twenty cars, 17,000 parameters, no regularization, and the loss goes to 3.77e-14 with the predictions matching the price list to the lira. That is the check to run first: it proves the data reaches the model, the gradients reach the weights and the optimizer moves. A frozen first layer stalls it at 2.14e-03 and p.grad is None names the two tensors responsible. A pass proves nothing about the labels, because the same network memorises scrambled prices just as happily.

8. Save, load, eval. state_dict holds six tensors for this model, named for the positions of its three Linear layers: 0, 3 and 6, stepping over the dropouts. Rebuild the architecture without those dropouts and the load fails on key names while every shape matches. Load it correctly, forget eval(), and the model quotes the same car 17,291 lira and then 14,043, because a module starts in training mode and the mode is not in the file.

9. A real small task. The gearbox, from five columns, as a label. Always saying automatic gets 59.5 percent of the held out cars right. A fitted logistic regression gets 59.5 percent, by saying automatic for 76 of 84 cars. A width 4 network gets 66.7 percent by calling 28 cars manual and being right about 17 of them. Six cars out of eighty four is what the network earned, and saying it that way is the last thing this course teaches.

The data

The same 420 cars as ML 101, ML 102 and PYTORCH 101, from the same seed, regenerated by the setup cell of every lesson. No file is downloaded and no csv is read: the arrays are built with numpy and crossed into torch, which is lesson 1 of PYTORCH 101.

From lesson 5 the lot is cut three ways rather than two. The 84 held out cars are the same 84 as in the first four lessons and no decision in the course is allowed to touch them. The validation set is carved out of what used to be training rows, which is the only honest place it can come from.

Where this course sits

Ninth course of the ivyx AI series and the second of the deep learning wave. PYTORCH 101 is the prerequisite and its loop is the loop this course puts layers inside. ML 101 supplies the held out discipline and the lot, and lesson 9's habit of comparing an accuracy against the commonest label is ML 102's.

HF 101 is next: models somebody else already trained, three lines to run one, and the tokenizer and the logits in the middle, which this course leaves you treating as numbers with a shape rather than as magic.

Requirements

Python 3 with torch, numpy and matplotlib. No GPU and no dataset. The heaviest cell in the course is 300 epochs on 252 rows, which takes about two seconds.

You install those packages yourself. The list above is a statement of what the lessons import, not an installer: nothing in this product installs a package or checks whether one is importable, so a learner opening this course against their own kernel or a bare virtualenv would otherwise meet an ImportError in the second cell of lesson 1. Every lesson's setup cell catches that and re-raises it carrying the command:

pip install torch --index-url https://download.pytorch.org/whl/cpu
pip install matplotlib

The index url matters on Linux, where a plain pip install torch resolves to the CUDA build and drags in several gigabytes of nvidia-* wheels that a laptop downloads and cannot use. The CPU wheel is 183 MB and 639 MB installed. On the ivyx platform kernel all three packages are already there.