RESEARCH 101 · Seeds, variance and ablation

ML 102 ended with a model that ranks which of 2,000 sold cars will come back with a warranty claim. This course runs that kind of model twice with nothing changed and reads two different scores, and then does what STATS 101 said to do with a number that moves: reports it as a mean with an error bar over seeds a reader can rerun. The model is a random forest on ML 102's service desk, regenerated from the same seed by every lesson's setup cell, and its score is the AUC. The two places randomness enters, the split and the forest, are found in lesson 1 and seeded on purpose in lesson 2; from lesson 3 on, a run is one integer.

Eight lessons take a number from one unseeded run to a report with two variants compared paired, an ablation with error bars, a log every run went into, and one sentence that carries the gap, the bar, the seeds and the p. Every number in the prose was produced by the cell above it.

How this course teaches

Every lesson is the same twenty four to twenty six cells, and nine of them are yours.

• A prediction you commit to before the cell runs. It is graded on the reasoning, not the guess, and being wrong here is the point.
• Warmups: a one line blank or a two to four line exercise under the theory it practices, each with a four rung hint ladder behind it, where the last rung explains and still does not hand over the code.
• An exercise that is broken when you open it.
• A diagnose cell: a report, a finding or a review that reads correctly and rests on one run, one seed, one selection or one unpaired bar. Something below it refuses the number by producing it again, or by running it on the seeds it should have run on.
• A challenge that ends in a sentence you write. The tutor grades the sentence, which means a green tick you earned for the wrong reason can be taken back.

No cell in this course passes in the state it ships. That is deliberate, and it is checked mechanically before the course is published.

The particular danger of this subject is a number that is true of one run and reported as true of the model. The same function printed 0.746 and then 0.680. A report gave an AUC to four decimals that could not be produced twice. A notebook set np.random.seed(0) at the top and called its run reproducible, and the same cell run twice printed two numbers. A variant beat the base by four hundredths on seed 5 and by three thousandths over twenty seeds. A best of twenty seeds beat one base run by eleven points and its own twin on the same seed by one and a half. A report with twenty runs per variant and error bars on both called a real improvement no better, because its runs were on different seeds. A feature ranked first by importance cost nothing when removed, and a feature ranked third cost nothing either. A record with params, a hash and a score had a seed of None and could not be reproduced. A sentence gave a gap with no bar, no seeds and no p. Every diagnose cell in the course is one of those, and every cross check is the second production of the number: the same report twice, twenty seeds against one, the paired gap against the claimed one, the ablation delta against the importance, the rerun against the record.

What you will be able to do

• Find the two places randomness enters a training run, measure how far ten identical runs spread, and refuse a number that cannot be produced again.
• Seed the split and the forest each on purpose, measure each source's share of the spread, and tell a seed that names a run from one that names a sequence.
• Report a score as a mean with a standard error over seeds, size the bar you can afford, and refuse an improvement that was a draw.
• Compare two variants on the same seeds with paired differences, a sign-flip permutation p and a win count, and refuse a best of twenty against a single run.
• Ablate a model one component at a time, table the deltas with error bars, and tell what the score needs from what the model happened to use.
• Log every run with its params, seed, data hash and score, reproduce the best run from the log to the last digit, and refuse a record that cannot be.
• Ask five questions of any claim, give it one of three verdicts, and rerun paired the one whose verdict the report got backwards.
• Write the report: the base line, the variants paired, the ablation with bars, the seeds and the hash, the log, and a sentence that can be refused against its table.

The lessons

1. The same code, twice. A forest with no seed scores 0.746, then 0.680. Ten unseeded runs spread by more than a tenth. A logistic regression on a split fixed at 0 repeats to the last digit and moves on an unseeded split. stratify fixes the 26 comebacks in the test set, not which. The diagnose is a report that cannot be produced twice.

2. What a seed fixes. Split fixed and forest loose moves the score; forest fixed and split loose moves it more; both fixed repeats. Over twenty seeds the split's contribution is 0.047 and the forest's 0.021. random.seed fixes nothing for scikit-learn; np.random.seed repeats the sequence of runs and not any one run. The diagnose is a notebook whose top cell seeds numpy and whose run does not repeat.

3. Twenty seeds. 0.734 ± 0.011 over twenty seeds, runs from 0.62 to 0.82. The balanced forest's 0.043 gain on seed 5 is 0.0035 over twenty. Eighty seeds halve the bar; a bar of 0.005 costs 97 seeds. The depth-six forest reaches the edge of the base's band. The diagnose is a finding read off one seed.

4. Comparing two variants. Paired by seed, depth six gains 0.0225 ± 0.0074, sixteen wins of twenty, sign-flip p = 0.006; unpaired, the same gap has twice the bar. The 300-tree forest's best seed is 0.8386, eleven points above the base on seed 0, a point and a half above its twin on seed 9, and 0.0035 above the base paired. The diagnose is that best-of-twenty claim.

5. Ablation. Removing inspection_score costs 0.114 ± 0.010, negative on ten of ten seeds; removing km, the forest's top feature by importance and a term of the rule that made the data, costs half a hundredth inside its bar, because year and price stand in for it. Importance ranks km first and the inspection fourth. The diagnose is a finding that seller_rating matters because it ranks third.

6. Logging a run. A record is params, seed, data hash and score, one JSON object per line. The best of ten logged runs is a depth-six run on seed 2, and score(seed, **params) reproduces it to the last digit. One changed number changes the hash entirely. The diagnose is a colleague's record with a seed of None. IVYX Studio's Experiments panel keeps the same record, experiments.log, for every pipeline run.

7. Reading a claim. Five questions: how many runs, same seeds, error bar, selection, data. A chart of single runs draws the seed's order. The unpaired depth-six report said no better at one bar; paired on the same seeds it is three bars. The diagnose is a review that accepted the unpaired reading.

8. A believable number. The base line, two variants paired, three components ablated, the seeds and the hash, sixty runs in the log, and the sentence. The depth-six gain survives the removal of the component the score needs, 0.027 ± 0.006, because the variant and the ablation answer different questions. The diagnose is a sentence with a gap and no bar, seeds or p. The last cell removes every file the course wrote.

What you need

• A Python kernel, 3.9 or later, with numpy, pandas and scikit-learn: pip install numpy pandas scikit-learn. No server, no network; a forest fits in a tenth of a second here, and the longest lesson runs a few hundred of them.
• About 450 minutes across the eight lessons, at 50 to 60 each.
• STATS 101, for the standard error, the confidence interval and the permutation test, and ML 102, for the service desk, the split and a metric that does not flatter a rare label.