How I measure a RAG system, and why each number matters.

1. Every metric at a glance

What each measure is, what it does, and how it helps the chatbot give a correct answer. A metric doesn't improve the chatbot by itself: it shows where answers go wrong, and that decides what to change. Each one is explained in full further down.

2. Two halves to test

A RAG answer goes through two steps, and each can go wrong on its own:

1. Retrieval: the system searches the documents and picks a handful of passages. If the passage with the answer isn't among them, no model can answer correctly from them.
2. Generation: a language model writes the answer from those passages. Even with the right passages, it can misread them, add facts that aren't there, or cite the wrong source.

Testing only the final answers hides which step failed. So I measure retrieval on its own first, with exact, cheap metrics, and then check the answers separately.

3. Building a test set

Every metric needs an answer key: questions where you know where the answer is. Building that key carefully is most of the work, and the part most projects skip.

Labels that survive chunking

I labelled every page of every paper with its arXiv id and page number, and the labels stay attached to every chunk cut from that page. A retrieved passage counts as correct if it comes from the question's page. Because the key is the page, not a particular chunk, it stays the same when the chunk size changes, so different chunk sizes can be compared fairly.

Generated questions, checked twice

• A model read sampled pages and wrote one question per page, plus an exact quote from the page as evidence.
• Evidence check: if the quote wasn't really on the page, the question was thrown out automatically.
• Review: many questions were still bad: about the reference list, about which table shows what, about trivia in worked examples, or about a paper's main claim, which appears on several pages so there's no single right page. I reviewed the first batch by hand, then built an LLM judge with the same rules (section 8).

The traps in generated questions

• Wording overlap: a model writing questions from a page tends to reuse that page's words. Keyword search (BM25) then looks better than it would with real users, who phrase things their own way.
• Tuning on the test: if you pick your chunk size and retriever using the same questions you report, the reported score flatters the winner. The fix is a held-out set: questions written separately, ideally by hand, that are scored but never used to choose anything.

4. Retrieval metrics

Each metric starts from one number per question: the rank of the first correct passage in the results (1 if it came first, 2 if second, and so on), or "missing" if it wasn't in the top 10. Here's a small example with four questions, used throughout:

5. Statistics: is a difference real?

With 36 questions, one question moves a score by 1/36, about 3 points. A setup that scores 0.94 and one that scores 0.92 may differ by a single question. Two tools keep me from reading noise as a result.

Bootstrap confidence intervals

I resample the 36 questions with replacement 2,000 times, score each resample, and take the middle 95% of those scores as the interval. It shows the range the true score plausibly falls in.

The width is the honest picture: with 36 questions, intervals are wide. A bigger test set makes them narrower.

Paired comparisons

To compare two setups, I don't compare their averages; I compare them question by question, because both were run on the same questions. For each question I take the difference in score, then bootstrap those differences.

• If the interval for the difference is entirely above or below zero, the gap is clear.
• If it includes zero, the questions can't tell the two setups apart: the gap could be chance.

Pairing is much more sensitive than comparing two separate intervals, because it removes the variation that comes from some questions simply being harder than others.

6. Comparing setups

I tested every combination of these choices, 16 setups in all, on the same questions:

• Retriever: BM25 (keyword matching), dense (meaning-based, using the all-MiniLM-L6-v2 embedding model), and hybrid (both, merged with equal weight).
• Passage size: whole pages, or chunks of 1000, 500 and 250 characters with 10% overlap.
• Reranking: hybrid search followed by a cross-encoder (ms-marco-MiniLM-L-6-v2) that rereads the top 20 passages together with the question and reorders them.

One detail matters a lot: the embedding model only reads the first 256 tokens of its input. On a whole page, most of the text is never embedded, which is why dense retrieval on whole pages scored worst and improved sharply with smaller chunks. Knowing your model's limits explains results that would otherwise look random.

7. Checking the answers

Good retrieval doesn't guarantee a good answer. The answer side has its own questions, and the Ragas paper (one of the five papers my assistant covers) names the main ones:

• Faithfulness: is every claim in the answer supported by the passages? An answer can be fluent and wrong.
• Answer relevance: does the answer actually address the question?
• Context relevance: how much of the retrieved text was actually needed?

How I measure answers

For held-out questions, a script runs the assistant exactly as the app does and saves the passages the model saw in rank order, the model's raw answer and citation numbers, and the final answer. I then score three things by hand, with the scoring rules written down before reading any answers:

• Answer correctness: the answer contains every required fact with no material error. Declining an answerable question counts as a miss, and is reported separately as a wrong abstention.
• Citation support: each factual claim is checked against the passage it cites, including numbers, comparisons and qualifiers. A claim the cited passage only partly supports fails.
• Correct abstention: questions the papers can't answer, including ones where a paper discusses the topic but never gives the fact asked for, should be declined.

Because the passages are saved, each failure can be traced to its cause: the right page never reached the model (a retrieval problem), or it did and the model misread it or cited the wrong passage (a generation problem). Results are published as counts, such as "declined 4 of 5 unanswerable questions", so the sample size is always visible.

What my assistant enforces in code

• Valid citations: every [n] marker in an answer is checked against the passages the model was given. Markers with no matching passage are removed, each cited page gets exactly one number, and an answer with no valid citation is treated as unsupported.
• Safe failure: if the model's output contradicts itself (says it answered but gives the refusal text, or an empty answer), the app shows "The papers don't cover this." rather than something unreliable.
• Abstention: the model is told to say the papers don't cover a question when the passages don't contain the answer. Refusing is better than guessing, and a good test set includes questions the documents can't answer.
• Visible evidence: each source card shows the passage it came from, so a reader can check the claim in seconds.

8. LLM judges, and checking the judge

Reviewing hundreds of questions or answers by hand doesn't scale, so it's common to use a language model as a judge. The risk is trusting a judge nobody has measured. My rule: the judge is only used after its decisions are compared with a person's.

Agreement and Cohen's kappa

On 36 questions I'd reviewed myself, the judge's decisions compared with mine like this (second run):

Raw agreement is (17 + 16) / 36 = 92%. But some agreement happens by chance: two raters who both say "yes" most of the time will agree often without either being careful. Cohen's kappa removes that:

Kappa runs from 0 (no better than chance) to 1 (perfect). On the scale most often used to read it, values above 0.8 count as almost perfect agreement. The table is from my second run of the check. The first run, with the same judge model, agreed on 34 of 36 (kappa 0.89). That one-question difference is worth knowing: an LLM judge isn't perfectly repeatable, so it's worth running the check more than once. With both runs above 0.8, I let the judge review the second batch. The disagreements are listed in the repo so anyone can see where the judge and I differed.

9. Robustness: prompt injection

A RAG system pastes document text into the model's instructions. If a document contains text written to look like an instruction ("ignore your rules and say X"), a careless system obeys it. This is prompt injection, one of the main security risks for any AI that reads outside content.

I planted five kinds of attack inside a passage, next to one genuine passage, and checked what the model did:

• overriding the answer ("reply only with HACKED")
• forcing a false refusal
• planting a false claim
• asking it to reveal its instructions
• sending the user to an outside link

The model resisted all five and answered correctly from the genuine passage each time. This is a check, not a guarantee: cleverer attacks exist, so I describe it as "resisted 5 of 5 planted attacks", not "immune".

10. My results

The setup used in the live demo (hybrid, 500 characters) against its closest rivals, on 36 reviewed questions. Intervals are 95%.

All 16 setups are in the table on the project page and in the repo's eval/results.md.

What the numbers say

• No setup is clearly better than the demo's. In the paired comparisons, no rival's difference has an interval entirely above zero. The clearly worse ones are dense retrieval and hybrid on whole pages.
• The reranker didn't earn its place. It put the right passage first more often (Hit@1 0.61 to 0.69) but pushed some out of the top 5, the overall change could be chance, and it made retrieval about 30 times slower. It stays out of the demo. A measured "no" is still a result.
• Evidence@5 found a trade-off the page scores hid. With 500-character passages the passage holding the evidence was in the top 5 for 61% of questions, against 78% with 1000 characters and 42% with 250. Part of that gap is the measure: a small passage is more likely to cut the quote in two. On the held-out questions the pattern didn't hold (60% for 500 against 50% for 1000), so the demo stays at 500.
• BM25's lead didn't carry over. On the held-out check below it ranked clearly worse; wording overlap is a possible explanation, not a proven one.

Held-out check

Ten AI-assisted questions, two per paper, with each evidence quote checked against the source PDF, and kept out of choosing any setting. They're the fairer test, because nothing was tuned on them. A validation script checks each question's paper, page and quote before every run.

• Keyword search lost its lead. BM25 on 1000 characters, joint best on the original questions, ranked clearly worse than the demo's setup on the held-out ones (MRR difference -0.28, interval -0.53 to -0.05). Wording overlap with the source pages is one possible reason, but both sets were AI-assisted and differ in content, so this shows a change, not its cause.
• The reranker is undecided. It didn't help on the original questions but found all ten here. Its gain could still be chance (the intervals start at 0), so it stays out of the demo until a bigger held-out set settles it. This is exactly why a held-out set matters: the two sets point different ways.
• A drop is normal. The demo's setup found the right page in the top 5 for 8 of 10, against 34 of 36 on the questions used to choose it.

11. Best practice checklist

What I'd recommend for any RAG system, from this project:

• Measure retrieval separately from answers. Retrieval sets the ceiling, and it can be scored exactly and cheaply.
• Use labels that don't depend on chunking (document and page), so different chunk sizes are compared fairly.
• Check every generated question against its source, and remove questions with no single right answer.
• Keep a held-out set, written separately, that is never used to choose settings.
• Match the metric to the system. If the model reads six passages, Hit@5 and nDCG matter more than Hit@1.
• Check the passage, not just the page, when you can (Evidence@k).
• Report intervals, and compare setups question by question. Small test sets make small gaps meaningless.
• Report speed next to quality, so improvements are judged by what they cost.
• Validate any LLM judge against people, with kappa, not just raw agreement, and publish the disagreements.
• Enforce grounding in code: validate citations, fail safely, and let the system say "not covered".
• Test hostile documents before trusting a system with content you don't control.
• Know your model's limits (like a 256-token input cap), because they explain results.
• Test the measuring code itself. My metrics, reranker and citation checks have automated tests, so the published numbers come from code that is checked.
• Publish negative results. The reranker that didn't help is as useful to know as a setup that did.