DATA 203 · Session, State and Cache Layers

COST 201 ended with a cache that hit 85.8 percent of the time and a note that the number was too good. This course reopens that cache, and the first thing it does is replay a week of real traffic through a plausible version of it and watch a hit return one dealership's answer to another dealership's question. The cache was fast and it was wrong, and the hit rate never said so.

That is the subject: where the pieces of a running system's state live, how long each may live, and what a cache key has to contain before a hit can be trusted. The material is COST 201's own query log, 374 requests across eight days, with a second tenant's traffic spliced in so the log has two dealerships sharing one cache the way a real platform does. No model is involved. The log is replayed with pandas, the costs come from COST 201's price table, and every number this course quotes is one you recompute in a cell.

How this course teaches

Lesson 1 is a tour: it replays the day through a cache keyed on the bare question text and finds the hits that crossed between tenants and the hits that served a superseded answer. The eight lessons after it are graded work, each built the same way, and nine of their cells are yours.

• A prediction you commit to before the cell runs. It is graded on the reasoning, not the guess, and being wrong here is the point.
• Warmups: a one line blank or a two to four line exercise under the theory it practices, each with a four rung hint ladder behind it, where the last rung explains and still does not hand over the code.
• An exercise that is broken when you open it.
• A diagnose cell: code that runs, prints a confident and plausible answer, and is wrong. Something below it refuses the answer by computing the same thing a second way, so nothing is taken on trust.
• A challenge that ends in a record or a sentence you write. The tutor grades the sentence, which means a green tick you earned for the wrong reason can be taken back.

No cell in this course passes in the state it ships. That is deliberate, and it is checked mechanically before the course is published.

The particular danger of this subject is a cache that is fast and wrong. A key that leaves out the tenant serves one dealership's answer to another. A key that leaves out the corpus version serves the answer from last week's policy after the policy changed. A normalisation that lowers the case is safe and one that strips the last word is not, and both raise the hit rate. A higher hit rate reads like a better cache and is not, and the number that is, saved cost, moves the other way as the key is fixed. Every diagnose cell is one of those, and every cross check is the second computation that refuses it.

What you will be able to do

• Replay a log through a cache and separate a hit that helped from a hit that served the wrong answer.
• Say where conversation history, session state and a cached answer each live, how long each may live, and who may read it.
• Price what a process restart costs an in-memory cache, and weigh it against a store that survives the restart.
• Normalise a cache key, watch the hit rate rise, and tell a normalisation that keeps the meaning from one that removes it.
• Complete a key with the corpus version and the tenant, and explain why the hit rate falling is the fix rather than the cost.
• Compare a time to live against invalidation on a corpus that changes mid week, and find the stale answer nobody reported.
• Stack an exact, a near miss and a negative cache and measure what each adds over the one before it.
• Report a cache's worth as saved cost rather than hit rate, and design a two tenant key that a replay proves no answer crosses.

The lessons

1. The answer that belonged to someone else. One week of traffic, 374 requests, two dealerships, replayed through a cache keyed on the bare question. It hits 184 times, and 68 of those hits crossed between tenants and 119 served a superseded answer. The hit rate was 49 percent and it counted every one of those as a win.

2. Three lifetimes, not one. Conversation history, session state and cached answers are three kinds of state with three lifetimes and three readers, and a system that keeps them in one place cannot expire any of them correctly. Each is sorted by how long it may live and who may read it, and the cache is the only one of the three that another user may read.

3. What a restart costs. State kept in the process is gone when the process restarts, and this lesson puts it there, restarts it, and prices what was lost against what an external store costs to keep it. The in-memory cache is free until the restart and then it is a cold cache paying full price on every request.

4. What makes a key. Normalising the query text lifts the hit rate from 49 to 76 percent, and every point of that lift is a decision about whether two requests are the same question. Lowering the case is safe. Stripping punctuation is usually safe. Dropping the last word raises the rate too and changes the question, and the hit rate cannot tell the three apart.

5. The field the key left out. Adding the corpus version to the key drops the stale hits from 154 to zero, and adding the tenant drops the cross-tenant hits from 87 to zero. The hit rate falls both times, from 76 percent to 63 to 52, and both falls are the fix: every hit removed was an answer that was wrong.

6. Expiry. A time to live expires an answer on a clock and invalidation expires it when the thing it depends on changes. On a corpus that swaps editions mid week, a one hour time to live serves the old answer for the rest of the hour and a version in the key serves the new one on the next request. The stale answer nobody reported is the one the clock had not reached yet.

7. Three caches in a row. An exact cache, a near miss cache keyed on the retrieved document, and a negative cache for queries that return nothing, in one path. The exact cache alone hits 165 times. The near miss cache adds 77, reaching 242, and it carries an assumption the exact cache did not: that two questions reaching the same document want the same answer. The negative cache adds 28 more.

8. Measure it honestly. The four keys from lesson 5, each scored two ways: naive saved cost that counts every hit, and honest saved cost that counts only the hits that were correct. The naive number tracks the hit rate and the honest number moves against it, peaking not where the cache hits most but where it is right most.

9. Design the key for the whole system. The key for a two tenant system, built field by field, and a replay that runs the whole log through it and counts zero answers crossing between tenants. The proof is not the design, it is the count.

What you need

This course runs on Python with pandas and numpy. Install them into the kernel with:

%pip install pandas numpy

There is no model, no server and no network call. The query log, the price table and the second tenant's traffic are all built in the first cell of every lesson, so each lesson runs on its own.

What to read outside this course

COST 201 is the prerequisite: it builds the query log, the price table and the 85.8 percent cache this course reopens. DATA 202 builds the retriever the cache sits in front of. For the state lifetimes in lesson 2, the platform's own session and history contracts are the reference for how long each kind of state is kept in production.