PYTORCH 101 · Tensors and autograd

Deep learning is usually taught from the top down: import a library, copy a training loop, watch a number fall, believe it. This course goes the other way. Everything in it is small enough to check by hand, and every line of the loop at the end is a line you wrote and can account for.

Two things separate a torch tensor from the numpy array you already know. It can live on another device, and it remembers what was done to it. The second one is the whole subject: because a tensor remembers, torch can hand you the derivative of anything you computed, exactly, without you deriving a thing.

How this course teaches

Every lesson is the same twelve cells, and five of them are yours.

• A prediction you commit to before the cell runs. It is graded on the reasoning, not the guess, and being wrong here is the point.
• A fill in the blank with one thing missing and a four rung hint ladder behind it, where the last rung explains and still does not hand over the code.
• An exercise that is broken when you open it.
• A diagnose cell: code that runs, prints a confident and plausible number, and is wrong. Something below it refuses the number by computing the same thing a second way, so nothing is taken on trust.
• A challenge that ends in a sentence you write. The tutor grades the sentence, which means a green tick you earned for the wrong reason can be taken back.

No cell in this course passes in the state it ships. That is deliberate, and it is checked mechanically before the course is published.

What you will be able to do

• Move an array between numpy and torch, and say without running it which of the two ways shares memory with the original.
• Read a shape, a dtype and a device off a tensor, and fix the three errors that each of them causes.
• Get a gradient out of backward() and check it against a slope you computed by hand in AI MATH 102.
• Read the graph the forward pass left behind, and say why a gradient lands on some tensors and not others.
• Recognise a missing zero_grad() from the answer, because the loss curve will not tell you.
• Fit a model on raw data, watch it turn to infinity in two steps, and derive the fix from the gradient rather than guessing at learning rates.
• Rebuild the same fit with nn.Module and an optimizer, and name every line they replaced.

The lessons

1. Arrays with extras. torch.from_numpy shares memory with the array and torch.tensor copies it, and the difference is invisible until something changes underneath you. The diagnose cell claims a conversion shared memory, prints a number that agrees, and is refused by a mutation.

2. Shape, dtype, device. Three properties, three error messages. Numpy hands you float64 and torch works in float32, so the first matmul between them refuses. A dtype at the wrong width is where the money goes: above 16,777,216 a float32 counts in twos.

3. backward(). The promise AI MATH 102 made. The slope of x ** 2 at x = 3 is 6, the numerical recipe from that course gives 6.001, and autograd gives 6. The gap is the h you were told was a sane default.

4. The graph torch builds. Every operation on a tensor that requires a gradient leaves a grad_fn behind, and calling backward() walks that chain and then frees it. So a second backward() on the same graph raises, a gradient lands on leaves only, and the chain rule is visible as one number being three times another.

5. Descent, replayed. AI MATH 102's loop on the same parabola, with .backward() where the finite difference used to be. The old loop could never land on 3. It lands on 2.9995, and you can predict that number from h before you run anything.

6. The accumulation trap. Forget zero_grad() and the loss still falls, because a summed gradient still mostly points downhill. Over 40 steps the buggy run reaches a loss of 0.0319 and the correct one is still at 0.3727, so the bug scores ten times better than the fix. The answer is what convicts it.

7. Linear regression by hand. The lot from ML 101, fitted a third time. Raw kilometres blow the loss up to infinity by the second step. A learning rate small enough to survive lands on a slope with the wrong sign. Standardise the column and the loop finds a slope of minus 0.1409 lira per kilometre, which is the number scikit-learn returned in ML 101 to four decimals.

8. nn.Module and optim. The same fit again, now with nn.Linear, MSELoss and SGD. It reproduces lesson 7 to six decimal places, which is the point: the wrappers took over three lines and invented nothing. The diagnose cell is the loss function silently broadcasting a shape it should have refused.

9. Fit a new function. One more input, the whole loop written by you, and a loss that falls from 0.85 to 0.11. The coefficient says an SUV is worth 21,440 lira more than a hatchback at the same odometer reading, and the lot's own median prices say 21,712, which is how you know the model is not making it up.

The data

The 420 cars from ML 101, byte for byte, generated from the same seed in the setup cell of every lesson that needs them. Nothing is downloaded and nothing is written to disk. Lessons 1 and 2 use them as arrays to convert, and lessons 7 to 9 fit them.

That repetition is on purpose. AI MATH 101 multiplied matrices by hand, ML 101 handed the same lot to LinearRegression, and here you write the fit yourself and land on the same slope. Three passes at one problem, each one at a lower level than the last.

Where this course sits

Eighth course of the ivyx AI series and the first of the deep learning wave. AI MATH 102 is where its centre of gravity is: the gradient descent loop you wrote by hand there is the loop you rebuild in lesson 5, and autograd arrives as relief rather than magic because you already know what it replaced. ML 101 supplies the fitting vocabulary and the lot.

PYTORCH 102 is next and adds the layers: a nonlinearity, an MLP, batches, and loss curves read as a story rather than a number.

Requirements

Python 3 with torch and numpy. Nothing else, no GPU, and no dataset. The heaviest cell in the course is 400 gradient steps on 420 rows, which is a hundredth of a second.

You install torch yourself. The package list above is a statement of what the lessons import, not an installer, and nothing in the product will fetch it for you. If your kernel is one the platform provides then torch is already there and you can start. Otherwise:

pip install torch --index-url https://download.pytorch.org/whl/cpu

The index url matters on Linux. A plain pip install torch there resolves to the CUDA build and pulls several gigabytes of nvidia-* wheels that a machine without an NVIDIA card cannot use. The CPU wheel is 183 MB and everything in this course runs on it. On macOS and Windows a plain pip install torch already gives you the right thing. Verified against torch 2.13 and numpy 2.5, and the course avoids every API that has moved since torch 2.0.