Evaluating RAG Before Production: Accuracy, Citations and Freshness

Teams often discover after launch that a knowledge assistant sounds fluent while citing an outdated policy, mixing customer documents or providing no evidence. The issue is usually governance, evaluation, versions and permissions rather than a single vector database.

1. The short answer

Build a question set with a reference answer, allowed evidence and known unanswerable cases. Cover frequent tasks, boundaries and confusing document versions. Unanswerable questions belong in the set because a clear refusal is better than an invented answer.

2. The context

Do not judge retrieval only by the final response. Log retrieved documents, rank, chunk version, filtering reasons and citations. When an answer fails, the team should know whether the right evidence was not retrieved, an old version was retrieved or the model reasoned incorrectly despite evidence.

3. The real trade-offs

Enterprise documents have departments, tenants, validity periods and access levels. Enforce permissions at the data layer, not only through a prompt reminder. For bilingual data, check whether both versions are updated so one language does not keep an expired recommendation alive.

4. How to put it into practice

Freshness should be a computable field. Policies, prices, product docs and jobs expire at different rates. Mark affected chunks after an update and trigger regression tests for related questions. Without update time, a team cannot explain changing answers.

5. Common failure modes

A demo with a few questions hides stale documents, access leaks, refusal quality and evidence location. More retrieved passages are not always better; similar chunks can crowd out the current passage and make the answer less reliable.

6. FDE perspective

RAG is a knowledge workflow with versions, permissions, evaluation and update policies. The quality bar is not fluent language; it is finding the right evidence and stopping clearly when evidence is insufficient.

A practical checklist before shipping or buying

• Give every question a reference answer or an explicit unanswerable label.
• Log versions, retrieval rank, permission filters and citations.
• Trigger regression tests after document updates.
• Treat cross-tenant leakage and stale hits as release blockers.

What to do next

Start with one department, one document class and 20 high-value questions instead of ingesting the whole company.

This article is based on public documentation and research from RAGAS documentation. The FDE perspective is editorial analysis, not a quotation from the source.