RAG 101 · Retrieval Basics

SQL 101 ended with one query over a lot of 24 used cars. This course starts from the paperwork around that lot: Northgate Motors' warranty and returns policies, a service schedule for every model, a long owner's guide, a page of frequently asked questions and a written description of every car. Fifty two documents, about two and a half thousand words, and 24 questions a customer might ask, each with the document that answers it written down. Somebody asks a question. Which document answers it?

That is retrieval, and it is the whole course. No language model is called in any lesson, and that is the point: a model that reads your documents has to be handed the right one, and handing it the right one is a counting problem that was solved decades ago and is still, for a large class of questions, solved best that way. You build the search by hand, from a tokeniser to BM25, and you measure it against the gold set at every step. RAG 201 adds embeddings to the same corpus, so that what they buy can be measured against what you already had.

How this course teaches

Lesson 1 is a tour: it puts the documents on disk, shows them from several angles, names the words the other lessons use, and shows the surprise without explaining it. The eight lessons after it are graded work, each built the same way, and nine of their cells are yours.

• A prediction you commit to before the cell runs. It is graded on the reasoning, not the guess, and being wrong here is the point.
• Warmups: a one line blank or a two to four line exercise under the theory it practices, each with a four rung hint ladder behind it, where the last rung explains and still does not hand over the code.
• An exercise that is broken when you open it.
• A diagnose cell: code that runs, prints a confident and plausible answer, and is wrong. Something below it refuses the answer by computing the same thing a second way, so nothing is taken on trust.
• A challenge that ends in a sentence you write. The tutor grades the sentence, which means a green tick you earned for the wrong reason can be taken back.

No cell in this course passes in the state it ships. That is deliberate, and it is checked mechanically before the course is published.

The particular danger of this subject is a ranking that looks right. The simplest search ranks the listing for A-117 tenth because the question's common words match a long document. Term frequency puts a 657 token guide first on 22 of 24 questions, including ones it does not answer. Counting occurrences instead of documents makes the idf of the negative and puts the right document first for the wrong reason. A sum where a mean belonged switches length normalisation off and reports a plausible search. The shorter of two warranty editions outranks the current one on every warranty question. A trailing s stemmer answers the part exchange question by matching takes. Every diagnose cell is one of those, and every cross check is the second route that refuses it: the model's name in the winning document, a document frequency that cannot exceed the corpus, an average document whose factor is 1, a sentence in the corpus that says an edition was replaced, the question's own subject word in the top hit.

What you will be able to do

• Turn a document into tokens with a rule you wrote, say what that rule does to an id, a number and a plural, and count a token across a corpus without being wrong by 22.
• Score a document by term frequency, measure the score over a gold set instead of admiring its top hit, and say why a count and a rate both lose to length.
• Build document frequency and idf by hand, weight a score with it, name the token that decided a document's score, and say from the gold set what the weight bought and what it did not.
• Write BM25 from its formula, say what k1 and b each change, measure both over a gold set, and defend a setting with the question that would move it.
• Cut a ranking at k, measure recall at several k, choose k from the curve and the token cost, and say what a recall figure does not tell you.
• Say what a stopword is and why the list is yours, predict what AND and OR return, and refuse a rank 1 hit that matched the wrong word.
• Sort a question into the kind keywords own, the kind they cannot answer and the kind the ranking decides, before running the search.
• Build the retriever from nothing, measure it on questions it has never seen, and write the sentence that says which one needs more than keywords.

The lessons

1. The documents. The tour. 52 documents by kind and length, two of them read in full, the 24 questions with their gold documents, and the simplest search there is: shared words, counted. It ranks the Nube schedule second for its own part number, the A-117 listing tenth, and finds the guarantee question at 17 and the part exchange question at 50. One prediction: which of five questions will it get right? None.

2. Words a search can see. split() counts warranty twice in a document that says it three times, because of a comma. The regex rule, lower case and runs of letters and digits, gives 99 tokens and 70 distinct against 98 and 79 on the warranty, and 2,715 against 2,560 over the corpus. A-117 becomes a and 117; 40,000 becomes two tokens; car's becomes car and s. Counting km with split() says 12 documents; the tokeniser says 34. 362 of the 620 tokens live in exactly one document.

3. Counting matches. Term frequency puts the guide first on 22 of 24 questions and the gold document first on 2. On the oil question the guide scores 112, 91 of it from the and a, and never says corsa; the Corsa schedule scores 9 at rank 11. Dividing by length moves the bias: the rate gets 12 of 24 right, never picks the guide, and ranks the Corsa schedule 14th. The diagnose hands over a document that cannot answer the question, and the cross check asks whether it mentions the car.

4. Rare words weigh more. idf = log(N / df) runs from 0.12 for a to 3.95 for a token in one document. Count times idf leaves rank one at 2 of 24 and the guide first on 22, because 52 times 0.49 is still 25 points, and lifts the top five from 17 to 21. The rarest token can belong to the wrong document: only the guide says kilometres and anything. The diagnose counts occurrences instead of documents, gives the a negative weight, and puts the Corsa schedule first for the wrong reason.

5. BM25 by hand. Saturation bends a count of nine oil to 2.14; the length factor makes the guide's single occurrences worth 0.16 where a listing's are worth 1.32. The Corsa schedule wins its question 7.09 to 4.93. Over the gold set: 13 first, 22 in the top five, guide first on none. b swept from 0 to 1 gives 7, 10, 14, 13, 13, and the one question between 0.5 and 0.75 is the winter tyres question whose answer is in the guide. The diagnose uses a sum as the average length and switches normalisation off.

6. Top k and recall. Recall at 1, 2, 3, 5, 10 is 13, 20, 21, 22, 22; two questions never arrive. The 2023 warranty outranks the 2025 edition that replaced it on all five warranty questions, for being 16 tokens shorter, and recall counts every one as a success. The diagnose reads 12 months off the top warranty policy; the cross check finds the sentence that says the edition was replaced.

7. Stopwords, AND and OR. 62 percent of the guide's score on the warranty question is stopwords. Removing them from the question takes gold first from 13 to 16 and leaves the top five at 22; the guarantee question goes from rank 22 to none. AND over a whole question returns nothing 20 times in 24 and the guide alone once. A trailing s stemmer puts the part exchange question at rank 1 by matching takes; the cross check asks whether the top document mentions part or exchange.

8. The case keywords win. A token in exactly one document is a key. Three questions carry an id key and all three are at rank 1; one question is made entirely of absent words and is never returned; sixteen ordinary questions are 11 at rank 1 and 15 in the top five, and the part exchange question hides among them. Adding one notice that says A-117 twice takes the listing's own question away from it. The diagnose picks the rarest token as the key and reads the guide.

9. Five questions. The retriever rebuilt from an empty namespace, checked against lesson 7's 16 and 22, and run on five questions the gold set does not contain. Four are first. Do you take my old car? has the trade-in policy at rank 3, because old and take are in no document and the policy says current car and takes. Adding trade-in to the question puts it first, and a person had to know to.

Requirements

• Python 3.9 or later with pandas. The course uses re, math and collections.Counter from the standard library and no model.
• PANDAS 101 for reading a frame and grouping it; NUMPY 101 for thinking in bulk; SQL 101 for asking a question of a table you did not write.

Every lesson's setup cell writes w1_docs.csv and w1_questions.csv from scratch, so any lesson can be opened on its own.

What to read

An Introduction to Information Retrieval by Manning, Raghavan and Schütze (free online): chapter 1 is lesson 1's count, chapter 6 is lessons 4 and 5. The Probabilistic Relevance Framework: BM25 and Beyond by Robertson and Zaragoza (2009) is the paper behind lesson 5.