A retrieval-augmented generation demo can look convincing after an afternoon. A dependable feature requires a clear job, controlled knowledge, measurable quality, and graceful failure.

Define the contract#

Write down what the feature may answer, which sources are authoritative, how fresh they must be, and what it should do when evidence is missing. Decide whether the product is offering discovery, summarisation, a draft, or a final answer. These are different risk levels.

Build retrieval as a system#

Clean and version source documents. Preserve headings, access controls, dates, and identifiers as metadata. Chunk around semantic boundaries rather than arbitrary character counts. Test keyword, vector, and hybrid search against real questions; reranking often improves a modest candidate set more efficiently than retrieving everything.

Access control must apply before context reaches the model. Filtering the final prose is too late. Citations should map to stable source locations so users can verify the answer.

Evaluate the complete behaviour#

Create a versioned set of representative questions, expected evidence, unacceptable claims, and cases that should be refused. Measure retrieval relevance separately from answer quality. Add human review for correctness, usefulness, citation support, tone, and safety.

Run evaluations when prompts, models, chunking, embeddings, or source processing changes. Model upgrades are product changes, not invisible infrastructure maintenance.

Operate for uncertainty#

Set latency and cost budgets. Cache safe repeated work, bound context, handle provider timeouts, and expose a non-AI fallback where possible. Log model and retrieval versions with privacy-conscious traces. Monitor empty retrievals, unsupported-answer rates, user corrections, latency, and spend.

The most trustworthy AI product is not the one that always produces text. It is the one that knows the limits of its evidence, makes those limits visible, and improves through a disciplined feedback loop.