RAG Score

Measures how good your search actually is. You write down which documents should come back for a question, and this counts how many of them did — no model is asked anything, so the number is repeatable and costs nothing.

What you can do

• Gate a pipeline on retrieval quality, the same way a test suite gates a build
• Compare two chunking or embedding choices on the same questions
• Catch a search that quietly stopped working when an index went stale
• See which expected document was missed, and where the first good one ranked

Requirements

• The Evals hub, which finds this engine and runs the suite
• Vectors, and a built index for the corpora your questions search

Getting started

Write a few questions with their answers in a .jsonl file — one line each, naming the documents that should come back. Point an eval suite at your search and list rag as its engine, then run it from the Evals panel. The first run tells you what your search is worth today; every run after that tells you whether a change helped.

Start with ten questions you already know the answers to. A small honest set beats a large guessed one.

The metrics

Metric	Meaning
rag.recall	Share of the declared documents that came back
rag.precision	Share of the returned hits that were declared
rag.mrr	Reciprocal rank of the first relevant hit; 0 when none came back
rag.hit@1	1 when the top hit was declared relevant
rag.retrieved	How many hits were considered. A counter — never threshold it
rag.mode.semantic	1 when the retrieval ran semantically

Threshold rag.mode.semantic in every suite. vectors.query falls back to keyword search when a corpus has no index, and on that path it echoes the corpora it was asked for — so a suite that does not check can report a healthy recall for a pipeline that never embedded anything.

The suite and its dataset

suites:
  - id: rag-code
    target:
      kind: capability
      id: vectors.query
    dataset: .punica/evals/rag-code.jsonl
    engines: [rag]
    thresholds:
      rag.recall: 0.8
      rag.mode.semantic: 1

Each dataset line is the retrieval's own input, plus what should come back:

{ "id": "gateway-approval",
  "input": { "query": "how does the gateway decide an approval",
             "corpora": ["code"], "topK": 10 },
  "expect": { "relevant": ["src/gateway.ts", "docs/approvals.md"] } }

Options

Option	Default	Meaning
match	path	Which field of a hit an expected value names — path, id or corpus
k	every hit	Score only the first k hits, so one dataset reads at several k without re-running

match is one field, never a chain. A resolver that tried id, then path, then a prefix would make a wrong match indistinguishable from a right one.

When the hits carry none of the named field, or a case declares nothing relevant, the engine says so instead of reporting a recall of zero: an absent field is a broken suite, not a bad retrieval.

Two things it does not do

It asks no model anything. Recall is counted against documents a person declared, so the number is reproducible, costs no tokens, and the dataset never leaves the machine.

It does not measure groundedness. Whether an answer is supported by what was retrieved needs a judge, and an LLM-as-judge metric belongs in its own engine rather than inside a deterministic gate.