Evaluating a RAG assistant for grounding, not fluency
Can an automated eval reliably fail answers that contain uncited claims, before a human sees them?
A grounding score that checks every sentence against the retrieved sources catches more real errors than a human spot-check, and it does so consistently.
The failure mode of a retrieval-augmented assistant is not bad grammar. It is a fluent, confident sentence that no source supports. Optimising for tone hides this; optimising for grounding surfaces it.
The experiment
Take a fixed set of questions and sources. For each generated answer, split it into claims and check each claim against the retrieved passages. Score the answer by the fraction of claims that are actually supported, and fail any answer below a threshold.
What I am watching for
Whether the grounding score correlates with human judgement, and whether it stays stable across paraphrases of the same answer. If the score swings on wording, it is measuring fluency again, and the experiment has failed.
This is open. The interesting result would be a threshold that a team could trust enough to gate a release on.