RAG 201 · Chunking, Embedding, Hybrid Search

RAG 101 ended with a boundary drawn by measurement: a keyword search finds every word that is in the corpus and cannot supply a word that is not. Do you take part exchange? was never returned, because the policy says trade-in. This course runs the same 52 documents and the same 24 questions through a search that does not count words at all. A local model turns each text into a vector, the question into another, and ranks by the angle between them. Trade-in and part exchange point nearly the same way. The question is found at rank 1. And the part number question, which keywords answered first time, is lost to a schedule that looks the same.

Every number here is a difference from a number you already have, because the corpus, the gold set and the BM25 retriever are RAG 101's, unchanged. The models run on your own machine through Ollama; nothing leaves it and nothing needs a key. Two models are used, nomic-embed-text and all-minilm, and lesson 8 finds the smaller one's 256 token window in the vectors rather than in its documentation.

How this course teaches

Lesson 1 is a tour: it gets the model answering, embeds the corpus once, runs the race against BM25, and shows the result without explaining it. The eight lessons after it are graded work, each built the same way, and nine of their cells are yours.

• A prediction you commit to before the cell runs. It is graded on the reasoning, not the guess, and being wrong here is the point.
• Warmups: a one line blank or a two to four line exercise under the theory it practices, each with a four rung hint ladder behind it, where the last rung explains and still does not hand over the code.
• An exercise that is broken when you open it.
• A diagnose cell: code that runs, prints a confident and plausible answer, and is wrong. Something below it refuses the answer by computing the same thing a second way, so nothing is taken on trust.
• A challenge that ends in a sentence you write. The tutor grades the sentence, which means a green tick you earned for the wrong reason can be taken back.

No cell in this course passes in the state it ships. That is deliberate, and it is checked mechanically before the course is published.

The particular danger of this subject is a number that looks like meaning. A cosine of 0.977 between a sentence and its negation. A sentence splitter that turns 1.6 mm into two sentences and reports a plausible count. An overlapping chunker that passes the join test because it ignored the overlap. A fusion that adds a BM25 score of 16 to a cosine of 0.6 and calls it hybrid. A small model shipped alone on 18 of 24 with the listing id at rank 10. A chunk setting chosen on the gold set that misses the one held out question the model exists for. Every diagnose cell is one of those, and every cross check is the second route that refuses it: the token the two sentences do not share, the piece that begins with a digit, consecutive chunks that share a sentence, the question one search was surest about, the id question BM25 owns, five questions the gold set never saw.

What you will be able to do

• Cut a corpus by characters and by sentences, keep each chunk's owner, rank documents by their best chunk, and refuse a splitter that breaks a number in half.
• Add overlap, measure what it costs in chunks and text and buys at rank 1, and count the repeats a reader would pay for.
• Compute a cosine by hand and match the library, prove an embedding call is deterministic, say what a small edit does to a vector, and refuse to merge two sentences that say opposite things.
• Run two searches over one gold set, sort the questions by who owns them, and refuse a one number verdict that hides a kind of question.
• Fuse two rankings by rank and by normalised score, explain a loss by the votes that caused it, and choose a method by measurement.
• Sweep chunk size and overlap, read the table by column, find the settings nothing dominates, and choose one while naming the question it loses.
• Measure an embedding model's window from its vectors, race two models, and say which weakness each pipeline decision compensates for.
• Build a chunked, embedded, routed retriever from nothing and check the choice on questions it never saw.

The lessons

1. The race. The model answers, three texts become 768 numbers each, and the same text gives the same numbers twice. The cosine of the part exchange question with the trade-in policy is 0.56, with the opening hours 0.39, and every pair of documents in the corpus scores above 0.43. Dense retrieval against BM25 over 24 questions: 17 to 16 at rank 1, 24 to 22 in the top five, the part exchange question found first, the guarantee question at 4, and the part number lost to the Duna schedule by 0.007.

2. Chunking. A whole document's vector is an average: the guide scores 0.63 against the tyres question and its one relevant sentence 0.85. Two hundred characters gives 93 chunks and cuts a word at 28 of 41 boundaries, leaving a two character chunk. Sentences with a look behind give 242 and none mid word; the naive split on the full stop gives 263, 21 of them starting with a digit. One sentence per chunk is 18 first against 17.

3. Overlap. Two sentences sharing one: 190 chunks against 141, the corpus embedded 1.74 times over, and 21 first against 18. The three that move each have their answer across a sentence boundary: the part number, the deposit, the battery. For 22 of 24 questions the top five chunks repeat a sentence. A chunker with overlap must fail the join test.

4. Embeddings from a local model. 768 numbers at unit length; cosine by hand agrees with the library at 0.4662. car and vehicle 0.87, two part numbers 0.74, warranty and guarantee 0.47, oil and brake 0.42. Case 0.994, reordering 0.989, a changed number 0.874, a negation 0.977. The guide's vector leans to its first paragraph at 0.90. A dedupe rule on cosine merged a sentence with its opposite.

5. Dense against BM25. Fourteen questions both, two BM25 only, three dense only, five neither. By kind, BM25 owns ids 3 of 3 where dense has 2. Dense retrieval's bias is genre, not length: the test drive policy is in 8 of 24 top fives. The part number was lost by 0.007 because six schedules sit within 0.05; the fault question by 0.20 to a FAQ entry with no answer in it.

6. Hybrid search. Reciprocal rank fusion is 17 and 23 and loses the part exchange question from rank 1 to 7, because BM25 returned six schedules for the word part and each got votes from both lists. k moves that question and not the headline. Min max fusion at 0.3 on BM25 is 18 and 24; routing id questions to BM25 and the rest to the model is 18 and 24 in one line. Adding raw scores is BM25 in disguise.

7. The chunk size sweep. Thirteen settings run from 16 to 21 gold first. The best at rank 1 is not the best in the top five, and the two best by total tie. The guarantee question wants a whole policy and the part number question wants its two sentences in one chunk; no setting wins both, and no setting puts the guarantee question first. The frontier is the honest shortlist.

8. Which embedding model. The small model reads about 230 words of the guide's 651; its vector is unchanged by removing the last 300. Whole documents: 15 against 17, the listing id at rank 10. Chunks: 18. With the route: 21 and 24, what the large model reaches with three sentences sharing one. Shipping a model alone chooses on its weakest kind of question.

9. Choose for this corpus. The retriever rebuilt from the corpus up: sentence chunks two sharing one, the large model, ids routed to BM25. On the 24 gold questions, 21 and 23. On five questions it never saw, five of five, including Do you take my old car?, answered by We accept your current car as a trade-in at 0.69 with no shared word. The setting the gold set slightly preferred misses that question at rank 2.

Requirements

• Python 3.9 or later with ollama, numpy and pandas.
• A running Ollama (https://ollama.com) with two models pulled once:

ollama pull nomic-embed-text
ollama pull all-minilm

In IVYX Studio the LLMS panel installs Ollama and pulls models. Every lesson's setup cell checks for the server and the model it needs and says what to run if either is missing. Embedding calls are cached within a lesson; the first run of a lesson takes a few seconds.

• RAG 101, for the corpus, the gold set and the BM25 retriever this course races against; NUMPY 101 for arrays; AI MATH 101 for cosine.

Every lesson's setup cell writes w1_docs.csv and w1_questions.csv from scratch, so any lesson can be opened on its own. The cosines quoted in the prose were measured with nomic-embed-text and all-minilm on Ollama 0.32; another machine may differ in the last decimals, and every check in the course compares against a recomputation rather than a typed number.

What to read

Sentence-BERT by Reimers and Gurevych (2019) made a sentence into one vector compared by cosine. The Nomic Embed technical report (2024) describes the larger model this course runs. Chapter 6 of An Introduction to Information Retrieval by Manning, Raghavan and Schütze is the vector space model that dense retrieval is the learned version of.