ML 102: Classification and honest evaluation

Nine lessons on the part of classification that nobody shows you. The answer is a label now rather than a number, and the moment the thing you are predicting is rare, every metric on the screen starts flattering you. This course is about reading those numbers, pricing them, and catching the two ways a much better result can be worth nothing at all.

ML 101 installed the discipline: baseline first, never score on training rows. This one weaponises it.

How this course teaches

Same tutor as the rest of the series: predict before you run, and hints that climb a ladder ending in an explanation rather than pasted code.

What is different here is that every number in this course arrives clean and formatted, and the wrong ones arrive that way too. So every graded cell checks a result against a second route to the same question, and the second route is usually cheaper than the first:

assert their_flags.mean() < 0.25, "this sends 442 of 500 cars to the workshop"
assert miss_cost > alarm_cost, "which box is the problem when you count money?"
assert twins[came_back].mean() > 0.9, "how many of those did the model already own?"

Five of the nine diagnose cells catch a report whose numbers are all correct. One of them is a threshold sweep, done properly, against the wrong objective. Another is a pipeline, cross-validation and the right metric, all in place, over a frame that had already been ruined.

What you will be able to do

• Read a classifier's real output, which is a probability per row, and say why the library's 0.5 flags nothing at all on a rare event
• Train a tree shallow enough to read out loud, and explain why every one of its leaves votes no while the model has found something real
• Draw two decision boundaries and tell a shape apart from a score, when the two models rank within 0.03 of each other and quote one car a factor of two apart
• Refuse an accuracy that a model with no columns and no fitting also earns, and show that tuning a threshold for accuracy builds a system that inspects four cars a season
• Say which of precision and recall the problem needs, defend it with what a mistake costs, and report neither of them alone
• Put a price on all four boxes of a confusion matrix, take the threshold off an invoice instead of tuning it, and notice when counting mistakes and pricing them point in opposite directions
• Spot the two leaks that inflate an imbalanced result, and settle the argument by counting rows rather than by comparing metrics
• Put the preparation inside the model so folds refit it, get the effect of balancing without inventing a row, and report a score with its spread
• Take the finished thing to a population it has never seen, price it there, and write down whether it should ship and what it must not be used for

The lessons

Classifying

1. Logistic regression
2. Decision trees
3. Decision boundaries

Metrics 4. The accuracy trap 5. Precision and recall 6. The confusion matrix

Leakage 7. Data leakage 8. Pipelines

The whole judgment 9. An imbalanced problem

The data

Two thousand cars the forecourt sold last season, written to disk by each lesson's setup cell from a fixed seed, so any lesson opens on its own and rerunning is always safe. It is the PANDAS 101, DATAVIZ 101 and ML 101 lot again, now with its service desk attached: for every car, whether it came back with a warranty claim inside thirty days. One hundred and three of them did, which is 5.15 percent, and that number is the reason for eight of the nine lessons.

The risk is an interaction rather than a slope. A car past 105,000 km whose pre-sale inspection scored 5 or less comes back 22 percent of the time, against 2 percent everywhere else, and neither half of that rule is worth much alone. So a depth 2 tree finds it in two questions and labels every car as safe anyway, because 22 percent loses a vote. A logistic regression on twelve columns achieves an accuracy of 0.9480, which is exactly what a model with no columns gets. And the threshold that the dealership's own prices imply, 350 lira for an inspection against 4,200 for a repair it cannot charge for, is 0.083 rather than 0.5.

The table is generated, and the rule that decides a comeback is in plain sight in the setup cell. Real data never gives you that. Here it does, so when a model claims to have found something you can check whether it is true, and lessons 2 and 3 are built on that check.

Lesson 9 opens a second table nobody has seen: 700 cars sold by another dealer group in another city, from newer stock. The same model, the same prices and the same threshold save 20,650 lira per 500 cars down here and lose 2,800 per 700 up there, and the model is not miscalibrated when it happens. Lesson 9 removes both files.

Where this course sits

Seventh course of the ivyx AI series and the second half of the machine learning wave. ML 101 is the real prerequisite: every split, baseline and fold in this course assumes you have already met it and know why held out rows exist.

The evaluation arc that runs through the series arrives here at its sharpest. ML 101 asked you to beat a constant. This course asks you to beat a constant when the constant is right 95 percent of the time, and then twice more when the number in front of you is better than the truth. HF 102 closes the arc later on: a fine tuned model that loses to the zero shot baseline is a real and instructive outcome, and by then you will have met the shape three times.

PYTORCH 101 is next in the series and goes the other way, back under the library to where the fitting happens.

Requirements

Python 3, with numpy, pandas, scikit-learn and matplotlib. Verified against scikit-learn 1.9 with pandas 3.0 and against scikit-learn 1.6 with pandas 2.3. One habit the course insists on for a reason: every precision is computed with zero_division=0, because a threshold that flags nothing is a normal thing to meet here and the default is a warning rather than a number.