RAG 202 · Reranking and Retrieval Evaluation

RAG 201 ended with a retriever that puts the gold chunk in the top five for 23 of 24 questions. This course asks the question that number does not answer: when the right chunk arrives, is the answer right? For the first time in the series a language model reads what the retriever found and writes an answer, and the answer is checked against a key. The two are different columns of one table, and they move apart as the retriever is given more to read: at k = 1 the gold chunk is in the context for 21 questions and 18 answers are right; at k = 8, 24 and 16. With the right chunk in front of it, a 1B model says the delivery is free, says the collector car is covered, and quotes the battery's 6 months from the 2023 edition that arrived beside the 2025 one.

The generator is llama3.2:1b through Ollama at seed 0 and temperature 0; the reranker is a 22.7 million parameter cross-encoder through transformers, pinned to one revision; the prices are an illustrative table you replace with your provider's. Everything runs on your machine, and every number in the prose was produced by the cell above it.

How this course teaches

Lesson 1 is a tour: it gets the generator answering, prints the two columns for k from 1 to 8, and shows the surprise without explaining it. The eight lessons after it are graded work, each built the same way, and nine of their cells are yours.

• A prediction you commit to before the cell runs. It is graded on the reasoning, not the guess, and being wrong here is the point.
• Warmups: a one line blank or a two to four line exercise under the theory it practices, each with a four rung hint ladder behind it, where the last rung explains and still does not hand over the code.
• An exercise that is broken when you open it.
• A diagnose cell: code that runs, prints a confident and plausible answer, and is wrong. Something below it refuses the answer by computing the same thing a second way, so nothing is taken on trust.
• A challenge that ends in a sentence you write. The tutor grades the sentence, which means a green tick you earned for the wrong reason can be taken back.

No cell in this course passes in the state it ships. That is deliberate, and it is checked mechanically before the course is published.

The particular danger of this subject is a good number on the wrong column. A retriever's 23 of 24 reported as accuracy. An MRR averaged over the questions a retriever returned, dropping the two it never did. A reranker added because the top five rose, while two questions fell from first. A generator's latency measured once, cold, with the model load in it. A cost split by the clock when the price table charges per call. The row with the most arrivals shipped over the row with the most right answers, and a gold set preference reversed by five questions it never saw. Every diagnose cell is one of those, and every cross check is the second route that refuses it: the answer column, the questions asked, the question that was first, the second call, the price table's own line, the held out five.

What you will be able to do

• Generate an answer from the top k chunks with a local model and check it against a key; measure arrival and correctness apart and name the questions where they disagree.
• Compute reciprocal rank and MRR by hand, cut it at the k a product reads, and admit a second acceptable document.
• Load a cross-encoder, read its scores for a pair, rerank a retriever's output and measure it on three first stages.
• Time every stage warm, read each as a share of the question, and refuse a cold start reported as a latency.
• Price a question by stage from a table you can replace, price a thousand, and find a reranker's break even.
• Build a trade off table with every column measured, find each column's knee by rule, keep the frontier and choose under a budget.
• Read a reranker's failures from its scores: a negation it prefers, a chunk about the topic with no answer, and a generator that scores everything the same.
• Choose k and a reranker under a latency budget and a price, hold the choice to questions it never saw, and name the question it costs.

The lessons

1. The two columns. One answer, deterministic on a second call. The key, and the two sentences it separates. The table: arrival 21, 22, 23, 23, 24 at k = 1, 2, 3, 5, 8; correctness 18, 15, 15, 18, 16. The answers that failed with the gold chunk present. No context at all: 8 of 24.

2. Recall against correctness. A number in the key must stand alone, so 12 is not 120 and 24 is not 24,000. Arrival and correctness as one function per k; the disagreements run one way, four or five questions wrong with the gold chunk at every k and at most one right without it. The model alone passes 8 keys, mostly guesses containing not; a stricter key passes 6. Retrieval buys eleven questions at k = 1. A retriever's 96 percent reported as accuracy is wrong by five.

3. MRR and the rank that matters. One over the rank, zero for never. Routed chunks 0.916, dense over whole documents 0.837, BM25 0.785, and BM25 over the 22 questions it returned reads 0.856, which is wrong. Cut at the product's k: 0.875 at k = 1, 0.910 at k = 3, and the 0.006 above that is a document nobody reads. The brake pads question: rank 3 with the gold document alone, rank 1 with the guide accepted.

4. A cross encoder over the top twenty. One score per pair, the same on every call. Over dense retrieval on whole documents: 17 to 20 first, MRR 0.837 to 0.910, the guarantee question from rank 4 to 1. Over BM25: 16 to 18. Over the routed chunk retriever: 21 to 19, MRR 0.916 to 0.889, top five 23 to 24. Depth 5 beats depth 20 because a reranker moves only what it is shown, in both directions.

5. Latency by stage. Warm: BM25 and the cosine under a millisecond, the embedding about 30, the cross-encoder 10 to 30 by depth, the generator about 250. The generator is four fifths of a question; the reranker's time grows with depth and the generator's barely with k. The generator as a reranker costs 170 milliseconds a pair. A cold start of a second reported as a latency.

6. Cents per question. A price table with shapes: per token in, per token out at four times, per call. One question at k = 3: 0.003 cents of generation, a millionth of embedding, 0.2 cents of reranker call; per thousand, three cents and two dollars. The whole corpus embeds once for less than one reranker call. A right answer worth a cent needs the reranker to buy 200 extra per thousand.

7. The trade off curve. Eight rows, four columns measured. Arrival's knee is k = 8 and correctness's is k = 1. The frontier is five rows; the reranked rows at k = 1, 3 and 8 are beaten outright. The row with the most arrivals has fewer right answers than the cheapest row.

8. What reranking makes worse. The battery is not covered scores 7.77 against 7.50 for is covered, and the negation wins about one in six corpus pairs. The breakdown FAQ scores 4.8 points above the policy with the number. The generator asked to score 240 pairs answers 2 for 232 and reorders nothing. The reranker turns the guarantee question right at the answer and agrees with the retriever on 22 of 24.

9. Choose k for a budget. The choice rebuilt as functions and held to RAG 101's five new questions. Under a hundredth of a cent, k = 1 plain; with a cent to spend, still a plain row. Every gold document arrives for all five under every setting and the answers still differ: 4 of 5 from one chunk, 3 of 5 from the reranked five. The row with the most gold arrivals is the worst on the questions it never saw.

Requirements

• Python 3.9 or later with ollama, numpy, pandas, torch and transformers. On Linux install torch from the CPU index:

pip install torch --index-url https://download.pytorch.org/whl/cpu
pip install transformers ollama numpy pandas

• A running Ollama (https://ollama.com) with two models pulled once:

ollama pull nomic-embed-text
ollama pull llama3.2:1b

• The reranker, cross-encoder/ms-marco-MiniLM-L6-v2 (91 MB), downloads from huggingface.co on the first run of lessons 4 to 9 and is read from the cache after that.
• RAG 201, for the retriever this course measures; RAG 101, for the corpus, the gold set and BM25; PANDAS 101 for the tables.

Every lesson's setup cell rebuilds the corpus and the index, checks for the server and the models it needs, and says what to run if any is missing. The generator's sentences and the timings are this machine's: the generator is deterministic here, another machine may word an answer differently, and every check in the course is a band or a relation rather than a typed number.

What to read

Passage Re-ranking with BERT by Nogueira and Cho (2019) is the cross-encoder as a reranker. Lost in the Middle by Liu and others (2023) measures what lesson 2 shows: more passages are not more right answers. The MS MARCO passage ranking task at microsoft.github.io/msmarco is what the reranker was trained on, and worth ten minutes so that you know what it was taught to prefer.