RAG evaluation matters because a retrieval-augmented generation system can sound completely convincing and still be wrong. The model writes in clean paragraphs, it cites something that looks like a source, and nobody can tell from the output alone whether the retriever pulled the right document or whether the generator invented the number in the third sentence.
So teams shipping RAG into legal review, claims processing, clinical support and internal knowledge search keep hitting the same wall. The demo passed, production did not, and the dashboard still shows a healthy score. Evidently the one number they tracked was never the one that broke.
This RAG evaluation guide stays on measurement: what to score, how to score it, and what each number is allowed to mean. If you are still deciding how evaluation, gating and ownership fit together across a programme, start with our RAG evaluation framework and treat this piece as the scoring layer underneath it.
Before the detail, here is the shape of the problem any RAG evaluation has to solve. A pipeline fails in three different places, and each place needs a measurement of its own.
The Same System, Two Very Different Numbers
One legal research assistant, three weeks after launch. Its generation scores look healthy, and one answer in six still misses a relevant statute.
Context recall. The retriever surfaced one of two required statutes, so the evidence for the second never reached the model at all.
Faithfulness on that same system. The answer stayed inside the evidence it was given, which is exactly why nobody noticed anything was missing.
01The short answer
RAG Evaluation Starts With Three Separate Zones
A RAG evaluation run answers three questions in order. Context recall tells you whether the evidence arrived. Faithfulness tells you whether the answer stayed inside it. Citation precision tells you whether the reference under each sentence actually supports it.
Evidence And Attribution
Zone 1 sits in the retriever. Zone 3 sits between the finished answer and the source it points at. Both are invisible in the text itself.
- The required chunk never reached the model
- The citation points near the answer, not at it
- Neither produces an error message
Faithfulness To The Evidence
The evidence arrived and the model wrote past it anyway. This is the zone most teams measure, and the one that stays green while the others fail.
- Claims the retrieved context does not entail
- Stylistically identical to grounded claims
- Moves on prompt and model version changes
A blended accuracy number is worse than useless in RAG evaluation, because it moves for reasons you cannot trace back to a component.
Splitting the retriever from the generator is not a new idea either. The original RAG paper treated them as two learned components rather than one black box, and evaluation has to follow that seam.
02Why one number fails
No Single RAG Evaluation Score Covers All Three Zones
This is the whole problem in one grid. Read across a row to see what that score can and cannot see.
| Score | Missed evidenceZone 1 | Fabricated claimsZone 2 | Wrong citationZone 3 | Off-topic answerEnd to end |
|---|---|---|---|---|
| Context recall | ||||
| Context precision | ||||
| Faithfulness | ||||
| Answer relevance | ||||
| Citation precision |
catches it reliably catches it sometimes blind to it
Every column needs at least one filled square, yet no single row provides them. That is why a dashboard built on one headline number moves late, and moves for reasons nobody can attribute.
Not Sure Where Your Retrieval Is Losing Evidence?
Send us your corpus profile, your question mix and your current numbers. Our engineers audit chunking, embeddings and reranking against a real golden set, then hand you the scores and the fix list.
Book a retrieval audit03Interactive
RAG Evaluation Calculator For Your Release Gates
Move the sliders to model a RAG evaluation run of your own. The calculator compares your three scores against the release floor for the risk tier you pick, then reports the weakest zone and whether the system would ship.
Check Your Scores Against A Release Floor
Planning thresholds set by the cost of a wrong answer in each deployment context.
The floors are the same ones in the threshold table further down. Notice how often a blended average clears a gate that at least one zone fails.
04Zone 1 of 3
Zone 1: Measuring Retrieval Quality
Retrieval quality is where RAG evaluation begins, and the retriever fills a fixed number of slots. Everything that matters shows up in which slots got filled with what.
Take one query against a contract corpus, a top-k of four, and three chunks that genuinely bear on the answer. The diagram below scores that single retrieval three ways, and the three numbers disagree sharply.
The highest of those three numbers is the one that lies. Rank-based scoring rewards a retriever for getting something useful into position one, while staying silent about the superseding document it never surfaced at all.
What Each Score Is Actually Counting
| Score | Formula | Drops when | Weight |
|---|---|---|---|
| Recall at k | Relevant chunks retrieved, over all relevant chunks that exist | The retriever cannot find the evidence at all | Watch first |
| Precision at k | Relevant chunks retrieved, over all chunks retrieved | Noise crowds the context window and raises cost | Softer failure |
| Mean reciprocal rank | One, over the rank of the first relevant chunk | The first hit lands low, and never otherwise | Reads high |
Two More Scores Worth Adding
NDCG at k accounts for graded relevance and rank order, which matters when chunks are partially useful. Context recall, the variant defined in the RAGAS paper, checks whether the retrieved context contains everything the reference answer needed. Overall it remains the best early-warning signal for retrieval regressions.
When recall drops, the fixes live in chunking strategy, embedding model choice, hybrid sparse-plus-dense retrieval, query rewriting and reranking. If you would rather label relevance automatically than by hand, eRAG feeds each retrieved document to the model individually and uses downstream performance as that document's relevance label.
05Zone 2 of 3
Zone 2: Measuring Faithfulness
The second stage of RAG evaluation scores faithfulness. Break the answer into claims and check each one against the retrieved evidence. Here is one answer, scored claim by claim.
The notice period is 30 days.
Supported, chunk 1Notice must be served in writing.
Supported, chunk 3Either party may waive the notice period by mutual consent.
No sourceNotice takes effect on receipt.
Supported, chunk 3Three of four claims hold up. The model invented the waiver clause. It reads plausibly, yet it appears nowhere in the retrieved context.
0.75Read that answer as a paragraph and nothing looks wrong. That is precisely the point. Fabricated claims are stylistically identical to grounded ones, so nobody catches them by reading. Someone has to score them.
Three Generation Scores, Three Different Jobs
What Each One Catches
- Faithfulness is the proportion of claims the context entails, so it catches outright fabrication
- Groundedness is a stricter sentence-level variant, so it catches the drift claim-level scoring smooths over
- Answer relevance checks whether the response addresses the question that was actually asked
Why All Three Belong In The Run
- Faithfulness alone will pass a hallucination-free answer to the wrong question
- Answer relevance alone will pass a beautifully targeted fabrication
- Groundedness alone is expensive to annotate at volume
- Together they close the generation stage without gaps
ARES shows how far this can be automated. It trains lightweight judges on synthetic data, then corrects their output against a small human-labelled set, which is a useful pattern once manual scoring stops scaling.
06Zone 3 of 3
Zone 3: Measuring Citation Accuracy
Citation accuracy is the part of RAG evaluation reviewers assume is already handled. A citation is correct only when the span it points to supports that exact sentence. Not the document. Not the topic. The sentence.
- Citation precision asks what share of the citations genuinely support the sentence they sit under.
- Citation recall asks what share of the sentences that required a citation actually got one.
- Attribution correctness asks whether each claim maps to the source it came from when an answer merges several. Systems that cite everything on every sentence score well on recall and badly here.
These definitions are not improvised. The ALCE benchmark formalised citation precision and recall at sentence level, and showed that models citing fluently still leave a large share of statements unsupported.
Most Of This Is Decided Before Evaluation Begins
Systems that cite the chunk identifiers they were actually shown, rather than reconstructing references from memory, score dramatically better.
So carry chunk metadata such as title, section, page and effective date through the prompt and back into the response, then validate the returned identifiers programmatically. Attribution then becomes lookup instead of generation.
07Symptom to fix
Your Score Dropped. Now What?
Find the number that moved, read what it means, then work the fixes in the order given.
| Score that dropped | What the drop means | Fixes in order |
|---|---|---|
| Context recall | Required evidence is not reaching the model at all, so no prompt change can rescue the answer. | 1. Review chunk size and overlap. 2. Add hybrid sparse and dense retrieval. 3. Raise top-k, then rerank. 4. Add query rewriting for vague inputs. |
| Context precision | Noise is crowding the window, which dilutes attention and raises cost before it hurts accuracy. | 1. Add a reranker. 2. Lower top-k. 3. Tighten chunk boundaries to whole clauses. 4. Filter by metadata such as document type and date. |
| Faithfulness | The evidence arrived and the model wrote past it, usually after a prompt, context length or model version change. | 1. Re-check the model version. 2. Tighten the grounding instruction. 3. Shorten the context. 4. Add an explicit abstention path. |
| Citation precision | References are being reconstructed rather than reported, so they point near the answer instead of at it. | 1. Pass explicit chunk identifiers. 2. Carry document metadata through the prompt. 3. Fix chunk boundaries. 4. Validate returned identifiers in code. |
| Answer relevance | The system is answering a nearby question, often because rewriting changed the intent of the query. | 1. Inspect rewritten queries. 2. Review the system prompt. 3. Segment scores by question type. 4. Read the failing traces individually. |
If two rows move together and nothing shipped, suspect the corpus rather than the code.
The usual cause is a re-index, a bulk document update, or a silent provider-side model change. Check all three before touching the retriever.
Shipping A RAG System You Need To Defend?
Our AI teams build retrieval pipelines with evaluation harnesses, citation grounding and release gates wired in from the first sprint, rather than bolted on after the first incident.
Talk to an AI solution architect08Ground truth
What A Good RAG Evaluation Set Is Made Of
Around 200 expert-annotated questions is enough, but only if the mix is right. Most sets sit entirely in the first band, which is why they report a system healthier than it is.
| Question band | Share | Why it belongs | Status |
|---|---|---|---|
| Single-hop, well-formed | 35% | One document answers it. Necessary, though it proves almost nothing on its own. | Baseline |
| Multi-hop | 25% | Two or more documents required. Where recall failures surface, and where most complaints originate. | Highest value |
| Post-update questions | 15% | The answer changed after a policy revision. Catches systems quoting superseded documents confidently. | Often missing |
| Adversarial phrasing | 15% | Ambiguous wording, unusual synonyms, and questions carrying a false premise inside them. | Often missing |
| Unanswerable | 10% | Nothing in the corpus answers these. Refusal is the only correct behaviour. | Rarely tested |
Annotate Both Stages, Or Retrieval Stays Unmeasurable
For every question, record the reference answer and the chunk identifiers that genuinely support it. Without that second field, nothing can compute a retrieval score at all. Consequently many teams end up with generation-only dashboards by accident rather than by choice.
Subject matter experts should do this work, because it is the highest-leverage hour anyone on the project will spend. Finally, freeze a holdout slice that nobody uses for prompt tuning, then version the whole set alongside the corpus.
09Tooling
Which Framework Runs Your RAG Evaluation
RAG evaluation tooling has matured. Pick by what you want it to do, because the harness around it is where the real work sits.
RAGAS And DeepEval
RAGAS is the reference implementation of faithfulness, answer relevance, context precision and context recall, with minimal setup.
- Pick RAGAS for a first credible baseline in an afternoon
- DeepEval adds pytest-style assertions and component scoring
- Pick DeepEval when evaluation should fail a build
TruLens, Phoenix And Opik
TruLens covers context relevance, groundedness and answer relevance with per-record tracing.
- Pick TruLens to debug one specific bad answer
- Phoenix and Opik capture traces over live traffic
- Pick those for monitoring and drift, not for gating
Whichever You Pick, Calibrate The Judge Behind It
An LLM produces almost every number above, since n-gram measures cannot recognise a correct answer phrased differently from the reference. An uncalibrated judge is therefore an unmeasured component sitting in the middle of your measurement system.
Have humans score 50 to 100 examples, then measure judge-to-human agreement. Below roughly 80 percent, fix the judge before anyone quotes its output.
Use binary or three-point scales. Asking a model for a score out of ten produces confident-looking noise clustered at 7 and 8.
Put the rubric and two or three worked examples in the judge prompt, because judges drift badly without anchors.
Watch for position and verbosity bias. Longer answers win, and options presented first win, so shuffle and re-run to detect it.
Pin the judge model version. A silent provider-side update moves every number you hold, and the regression will look like a system change.
10Release gates
What Good Enough Means Depends On The Cost Of Being Wrong
There is no universal RAG evaluation threshold. Find your column and read down it. These are the same floors the calculator uses.
| Score | Internal searcha wrong answer costs minutes | Customer facinga wrong answer costs trust | Regulateda wrong answer costs a filing |
|---|---|---|---|
| Context recall | 0.75 | 0.90 | 0.97 |
| Context precision | 0.60 | 0.75 | 0.85 |
| Faithfulness | 0.85 | 0.95 | 0.99 |
| Answer relevance | 0.80 | 0.85 | 0.92 |
| Citation precision | 0.80 | 0.95 | 0.99 |
Add one more gate beside these five: correct refusal on unanswerable questions, at roughly 0.60 internally, 0.80 for customer-facing systems and 0.95 in regulated work. It is the only gate that measures what the system does when the corpus has nothing to offer.
Two things follow. A threshold is a business decision rather than a technical one, so whoever owns the consequence of a wrong answer should set it. And prompt tuning alone will never reach the regulated column, because at those levels you need human review routing for anything scoring below the floor.
11Cadence
Where Each RAG Evaluation Check Runs
A RAG evaluation done once, a week before launch, has a shelf life of about a week. Documents change, indexes rebuild and providers update models, none of which touches your code.
Offline sets cannot anticipate what real users will ask. So in production, score a sample of live traces on the same measures, then segment by question type, by document source and by tenant. Aggregate numbers hide the failing segment, and the failing segment produces the complaints.
Meanwhile every user correction, thumbs-down and escalation is a labelled example waiting for collection. Feeding those back turns evaluation into a system that improves rather than a report that gets archived.
12Green boards
What The Dashboard Shows Versus What Is Happening
Five failures that never produce an error message. Instead, they produce a healthy-looking board.
| Dashboard says | Actually happening | Correction |
|---|---|---|
| Mean faithfulness 0.93, stable for six weeks | Scanned PDF answers score 0.41, while everything else carries the average | Report distributions and worst segments, not means |
| Faithfulness 0.95, citation precision 0.96, every gate green | The system is faithfully citing a policy that expired last quarter | Give corpus freshness a check of its own |
| Scores improved every sprint for two months | Prompts were tuned against the full eval set, so the numbers measure memorisation | Hold a slice back and touch it only at release |
| Citations appear on every answer and reviewers trust it | Nobody scores the citations, so they drift while still looking authoritative | Put citation precision on the release scorecard |
| Answer relevance 0.89, the system answers everything | It also answers questions the corpus cannot support | Add unanswerable cases and score refusal |
13Summary
The Short Version
Six RAG evaluation rules to carry into your next planning session.
Three failure zones need three separate measurements, since nothing covers more than one column of that grid.
Context recall warns you first about missing evidence, faithfulness about fabrication, citation precision about bad attribution.
A 200-question set only works if two thirds of it is deliberately hard: multi-hop, updated, adversarial and unanswerable.
Thresholds are a business decision, so set them against the cost of a wrong answer rather than today's numbers.
Calibrate the LLM judge against human labels, then pin its version.
Put the gates in CI and score sampled production traces, because the corpus and the model change without you.
14Getting started
Where To Start, And Where Our Teams Fit
First, pull thirty real questions from your logs, making sure a third are multi-hop and a few are unanswerable. Have a subject matter expert annotate the supporting chunks and reference answers, then run RAGAS or DeepEval over them. The first numbers will be uncomfortable, and they will point straight at whichever zone needs the work.
From there the loop is simple. Expand the set as production surfaces new failure shapes, calibrate the judge, set thresholds against the cost of a wrong answer, then put the whole thing in CI so RAG evaluation runs whether or not anyone remembers to.
How Our Teams Help
When a build is the right call, retrieval architecture, chunking and reranking are delivered alongside the RAG evaluation harness through our AI development services.
Citation-grounded answering over sensitive corpora runs as private large language model development, while risk tiers, release gates and review routing are agreed with the business through our AI consulting practice.
Production tracing, drift dashboards and feedback capture sit inside AI and ML implementation for document-heavy workflows.
Bring Your Corpus, We Will Bring The Scorecard
Share your question mix, your document sources and your compliance constraints. You will receive a mapped golden set, the measures that fit your pipeline, and release thresholds set against the cost of a wrong answer in your domain.
Request your evaluation plan15Common questions
Frequently Asked Questions
What Is The Difference Between Faithfulness And Answer Correctness?
Faithfulness checks whether the answer stays inside the retrieved context, whereas answer correctness checks whether the answer is true.
An answer can be perfectly faithful to a context that was itself the wrong document. Consequently you read retrieval scores alongside faithfulness rather than in place of it.
How Many Examples Does A RAG Evaluation Dataset Need?
Between 150 and 300 carefully annotated examples is enough for most production systems, provided the mix includes multi-hop questions, updated documents, unanswerable questions and adversarial phrasings.
Annotation quality matters far more than volume, since thousands of auto-generated questions will report a system healthier than it is.
Can BLEU Or ROUGE Be Used To Evaluate RAG Output?
Not on their own. Both compare surface n-grams against a reference, so a correct answer worded differently loses points while a fluent fabrication reusing reference vocabulary gains them.
LLM-as-a-judge scoring against a rubric handles semantic equivalence far better, as long as the judge itself matches human labels.
How Often Should A RAG System Be Re-Evaluated?
A fast smoke set should run on every pull request, the full golden set on every staging deploy, and sampled production traces continuously.
Any corpus re-index, embedding model change or provider model update should also trigger a full run, because all three shift behaviour without a line of your code changing.
What Causes Low Citation Precision Even When Faithfulness Is High?
Usually the model is reconstructing references rather than reporting the chunk identifiers it received. Chunk boundaries that cut through a clause, missing document metadata, and prompts that request citations without supplying stable identifiers all contribute.
Passing explicit chunk identifiers through the prompt, then validating the returned ones programmatically, resolves most of it.
Should Retrieval Be Evaluated Separately If End-To-End Answers Look Fine?
Yes. Answers frequently look fine while recall is degrading, because a capable generator writes coherently from partial evidence.
By the time end-to-end quality visibly drops, the retrieval problem has usually been present for weeks.
Which Score Should A Team Track First If They Can Only Track One?
Context recall. Since no downstream component can recover a missing chunk, recall failures are both the most damaging and the most invisible.
Faithfulness should follow immediately after, because together they cover the two stages of the pipeline.
Do RAG Evaluation Measures Work For Multilingual Or Multimodal Systems?
The retrieval measures transfer directly, since they only depend on relevance labels. The generation measures need a judge that performs reliably in the target language, so measure agreement with human labels separately per language.
Multimodal systems add a further failure point, because images converted to text by a vision model can introduce errors before the generator ever sees them.
