STATS 101 · Sampling, error and significance

A courier made 10,000 deliveries last month and somebody timed 25 of them. This course is about what that number is worth. Every lesson builds the same population from the same seed, so the true average, 33.91 minutes, is a number you can print, and every sample you draw can be held to it. A thousand couriers each timing 25 deliveries land within a minute of the truth fewer than half the time; one of them is six and a half minutes off, and every one of the thousand reports its number with the same confidence.

From there the course builds the instruments one at a time. The spread of those thousand estimates has a formula, the standard error, and the formula says why four times the data only halves the error. The formula gives an interval, and a thousand intervals against the truth show what its 95 percent promises and does not. Two depots, a shuffle, and a p-value; then a world where one proposal in ten is real, in which a p under 0.05 is real 37 percent of the time. Twenty tests on identical depots flag two. And the last lesson writes the sentence a manager can act on: a difference, its interval, its p-value, and a decision, in minutes and in hours.

Nothing here needs a library. The course runs on random, statistics and math from the standard library, writes no file, and every number in the prose was produced by the cell above it under a named seed.

How this course teaches

Every lesson is the same twenty six cells, and nine of them are yours.

• A prediction you commit to before the cell runs. It is graded on the reasoning, not the guess, and being wrong here is the point.
• Warmups: a one line blank or a two to four line exercise under the theory it practices, each with a four rung hint ladder behind it, where the last rung explains and still does not hand over the code.
• An exercise that is broken when you open it.
• A diagnose cell: code that runs, prints a confident and plausible number, and is wrong. Something below it refuses the number by computing the same thing a second way, so nothing is taken on trust.
• A challenge that ends in a sentence you write. The tutor grades the sentence, which means a green tick you earned for the wrong reason can be taken back.

No cell in this course passes in the state it ships. That is deliberate, and it is checked mechanically before the course is published.

The particular danger of this subject is a number that looks more precise than it is. A sample of 400 rated deliveries estimates the average to a decimal and is 3.4 minutes wrong, because customers rate fast deliveries. A standard error divided by n instead of the root of n is five times too small and reads as good news. An interval labelled 95 percent and built one standard error wide holds the truth 68 times in 100; the divide by n interval holds it 16. A positive gap is called confirmed when one shuffle in eight beats it. A p of 0.03 is read as a 97 percent chance of a real effect when the simulation says a third. A non-significant test on ten deliveries is read as proof of no difference at 14.5 percent power. And a pilot's 1.9 minute saving is reported as the size of the effect when its interval runs from a two minute loss to a six minute gain. Every diagnose cell is one of those, and every cross check is the second route that refuses it: the truth the lesson owns, a simulation of a thousand samples, the bootstrap beside the formula, the coverage of the procedure, the power table.

What you will be able to do

• Draw a sample from a population you own, measure its error, simulate a thousand of them to see what one is worth, and tell a small sample from a sample of the wrong list.
• Measure a standard error by simulation, compute it by formula, estimate it from one sample, and turn a precision target into a sample size and a cost.
• Build a 95 percent interval by formula and by bootstrap, extend the bootstrap to a median, keep the interval for a mean apart from the range for one delivery, and catch an interval too narrow to be true.
• Measure the coverage of an interval procedure, say what its percentage promises about one interval, and check a label against the procedure.
• Build a null hypothesis by shuffling, read a p-value as the share of shuffles that beat the data, choose the tail from the question, and let the test rather than the sign decide the sentence.
• State what a p-value is and is not, simulate a world with a base rate to find what a significant result is worth in it, and name the three things that change that worth.
• Say what a screen of many tests finds when there is nothing to find, correct the threshold for the count, measure the power of a test, and size a study so a real effect is not missed.
• Report a difference with its interval and its p-value, decide when they let you and refuse when they do not, and write the sentence a reader can act on.

The lessons

1. Population and sample. 10,000 delivery times from seed 101, true mean 33.91, true spread 8.95. Five samples of 25 scatter from 31.69 to 37.23. A thousand of them: 435 within a minute, 926 within three, the worst 6.45 off. A hundred per sample halves the spread. The diagnose is 400 rated deliveries estimating 30.51, precisely and wrongly.

2. The standard error. The spread of a thousand sample means at 4, 25, 100 and 400: 4.58, 1.83, 0.91, 0.43, and the formula sigma over root n within a few percent of each. One sample of 25 estimates its own standard error at 1.64 against a simulated 1.70. Dividing by n instead of root n claims 0.33 and is refused by a factor of five.

3. The confidence interval. Mean plus or minus 1.96 standard errors, 32.21 to 38.63 for the sample of 25; the bootstrap of the same sample, two thousand resamples with replacement, 32.44 to 38.50. The range for one delivery is 19 to 51 and holds 92 percent of them where the mean's interval holds 27. The bootstrap gives a median its only interval.

4. What 95 percent means. Seed 53's interval holds the truth and seed 52's misses, and the two look alike. A thousand intervals at n = 100 hold it 944 times; 686 at one standard error, 987 at 2.58; 933, 944 and 956 at 25, 100 and 400 per sample. An interval labelled 95 percent and built one standard error wide is refused by its coverage.

5. The null, by shuffling. North and South, 30 each, a gap of 2.63. Shuffle the labels two thousand times: 12.4 percent of shuffles beat it, so not shown, though South was built 3 minutes slower. Two depots with no difference give p = 0.457. Two tails double the p. Central, at 6.12 minutes and p = 0.0005, earns the word confirmed.

6. What a p-value is not. A thousand experiments in a world where one proposal in ten is real: 91 significant, 34 of them real, 37 percent. Half real: 88 percent. The experiments that read p = 0.03 were real a third of the time in the first world and nine tenths in the second. One minus p as the chance of an effect fails by a factor of three.

7. Twenty tests on nothing. Twenty depots from one population, two flagged, one at p = 0.0045. At least one false alarm in twenty tests 64 percent of the time; Bonferroni's 0.0025 clears all of them. Power for a 3-minute gap: 14.5 percent at 10 per depot, 38.5 at 30, 88.5 at 150. A large p at ten deliveries is inconclusive, never equally fast.

8. The verdict. A pilot of 40 per arm shows the new routing 1.91 minutes faster with an interval from minus 6.09 to plus 2.28 and p = 0.377: not decided. The rollout at 200 per arm: 2.09 faster, 0.41 to 3.77, p = 0.0175, decided, 348 hours a month between 69 and 628. The truth is 1.5 minutes, and the estimate alone is refused as a report.

What you need

• A Python kernel, 3.9 or later. No packages.
• About 450 minutes across the eight lessons, at 55 to 60 each. Lessons 6 and 7 run simulations that take up to half a minute per cell.
• AI MATH 102, for the mean, the standard deviation, and simulation with a seed.