AI · Personal project · 2026
RAG Research Assistant
Answers questions about five well-known papers on retrieval and RAG, cites the paper and page for every claim, and says when the papers don't cover a question. My earlier telecare chatbot never measured whether it found the right passages, so this time I measured it.
How often the right page was in the top 5 results
36 reviewed questions. Each retriever was tried on whole pages and on passages of 1000, 500 and 250 characters.
| Retriever | Whole page | 1000 | 500 | 250 |
|---|---|---|---|---|
| Keyword (BM25) | 33 of 36 | 35 of 36 | 32 of 36 | 33 of 36 |
| Hybrid | 30 of 36 | 32 of 36 | 34 of 36 | 32 of 36 |
| Embeddings (dense) | 19 of 36 | 27 of 36 | 30 of 36 | 29 of 36 |
Used in the live demo Highest score
All sixteen results, with every measure
Hit@1 and @5: share of questions where the right page was first, or in the top 5. Evidence@5: stricter, the top-5 passage must also contain the question's whole evidence quote, matched on whole words. MRR and nDCG: how high the right page ranked (1 means always first; nDCG gives more credit to ranks 2 and 3). ms: retrieval time per question on my laptop.
| Retriever | Passage | Hit@1 | Hit@5 | Evid@5 | MRR | nDCG | ms |
|---|---|---|---|---|---|---|---|
| BM25 | page | 0.72 | 0.92 | 0.92 | 0.788 | 0.832 | 2 |
| BM25 | 1000 | 0.69 | 0.97 | 0.83 | 0.788 | 0.839 | 2 |
| Hybrid + rerank | 500 | 0.69 | 0.89 | 0.58 | 0.777 | 0.830 | 869 |
| Hybrid | 250 | 0.69 | 0.89 | 0.42 | 0.768 | 0.804 | 34 |
| BM25 | 500 | 0.67 | 0.89 | 0.61 | 0.760 | 0.811 | 4 |
| Hybrid + rerank | 250 | 0.67 | 0.83 | 0.39 | 0.754 | 0.793 | 515 |
| Hybrid | 500 | 0.61 | 0.94 | 0.61 | 0.750 | 0.806 | 30 |
| Hybrid + rerank | 1000 | 0.67 | 0.86 | 0.69 | 0.746 | 0.799 | 1514 |
| BM25 | 250 | 0.58 | 0.92 | 0.36 | 0.718 | 0.774 | 5 |
| Hybrid | 1000 | 0.56 | 0.89 | 0.78 | 0.701 | 0.773 | 26 |
| Dense | 250 | 0.61 | 0.81 | 0.39 | 0.688 | 0.730 | 26 |
| Hybrid + rerank | page | 0.47 | 0.72 | 0.72 | 0.598 | 0.668 | 2363 |
| Dense | 1000 | 0.47 | 0.75 | 0.56 | 0.594 | 0.658 | 22 |
| Dense | 500 | 0.39 | 0.83 | 0.50 | 0.573 | 0.657 | 24 |
| Hybrid | page | 0.31 | 0.83 | 0.83 | 0.522 | 0.619 | 23 |
| Dense | page | 0.22 | 0.53 | 0.53 | 0.359 | 0.432 | 21 |
What I found
No setup is clearly better than the one the demo uses. Compared question by question with 95% intervals, every rival's difference could be chance. Only embeddings alone and whole-page hybrid are clearly worse.
A cross-encoder reranker put the right passage first more often (Hit@1 0.61 to 0.69) but dropped others out of the top 5, and made retrieval about 30 times slower. I left it out of the demo. These Evidence figures use a corrected exact-match rule; an earlier version over-credited partial matches.
Checking the passage, not just the page, showed a trade-off: 1000-character passages held the evidence quote more often than 500 (78% against 61% in the top 5), partly because smaller passages are more likely to cut the quote in two. On the held-out questions the pattern didn't hold (60% for 500, 50% for 1000), so the demo stays at 500.
Keyword search did well partly because the test questions reuse the papers' wording. Embeddings on whole pages were poor because the model only reads the first 256 tokens.
Held-out check. On ten AI-assisted held-out questions, with evidence checked against the PDFs and kept out of choosing the demo's setup, keyword search on 1000 characters (joint best before) fell clearly behind the demo's hybrid setup in ranking (MRR 0.47 against 0.75), so its earlier lead didn't carry over. The reranker found all ten, but with so few questions that gain could still be chance.
What each metric means and how it helps the chatbot answer correctly →
Stack
Python · LangChain · BM25 · sentence-transformers · OpenAI · Pydantic · Gradio · pytest · GitHub Actions
