RAG Explained: Retrieve the Right Evidence Before Generating an Answer

Harness Engineering · Module 5 of 16

Start here: basic Python for the optional lab; no model API or vector database needed.

Your goal: diagnose whether a wrong answer came from missing evidence, poor retrieval, or generation.

The assistant answers “yes, within 14 days” about opened coffee. The knowledge base contains the exception. Search retrieved only the general rule. Rewriting the final-answer prompt cannot reliably recover a passage the model never saw.

Retrieval-augmented generation, or RAG, supplies selected external evidence at inference time. It does not update model weights. Its useful promise is access to relevant maintained information; correctness still depends on extraction, retrieval, source interpretation, and answer verification.

Inspect the retrieval output before judging the generated answer.

1. Start with a document inventory

Our fictional source pack has three passages:

Preserve source ID, document version, location, effective date where supplied, and access metadata during ingestion. PDFs may lose reading order; tables may lose their headers; footnotes may qualify the main sentence. Inspect extracted text before building an index.

An index can be technically healthy and still omit the current revision. Record which source versions it contains and how updates, removals, and access changes propagate.

2. Choose chunks that retain meaning

A chunk is the unit retrieved for the model. Splitting at a fixed character boundary can separate a rule from its exception. Keeping the entire manual may preserve the exception but consume the context budget and dilute search results.

Try headings and semantic sections as initial boundaries. Use overlap only where it helps preserve context, then measure duplication and evidence recall. There is no universally correct chunk size.

For the coffee question, P2 may determine the answer, while P1 explains why being within seven days is insufficient. A retrieval result containing P1 alone gives the model an incomplete policy picture.

3. Understand what each search method measures

An embedding represents content as numerical features. Cosine similarity compares vector direction; a high score does not establish truth, currentness, or permission. Apply access restrictions before evidence enters model context.

4. Run a small retrieval failure

This complete standard-library example implements word overlap. It is neither BM25 nor embedding search, and its inputs are synthetic.

import re

docs = {
    "P1": "Unopened products returns 14 calendar days receipt",
    "P2": "Coffee consumable opened consumables excluded",
    "P3": "Orders dispatch three business days",
}


def terms(text):
    return set(re.findall(r"[a-z]+", text.lower()))


def retrieve(query, limit=2):
    q = terms(query)
    scored = [(key, len(q & terms(text))) for key, text in docs.items()]
    return sorted(scored, key=lambda item: (-item[1], item[0]))[:limit]


literal = retrieve("opened coffee return")
paraphrase = retrieve("Can I bring back a used drink product?")
assert literal[0][0] == "P2" and literal[0][1] == 2
assert all(score == 0 for _, score in paraphrase)
print("Literal:", literal)
print("Paraphrase:", paraphrase)

The block was executed locally. The literal query ranks P2 first with two overlapping terms. The paraphrase gives every document zero overlap. The alphabetical tie-break still returns IDs; those are not evidence of relevance. A real system needs a defined policy for weak or absent retrieval evidence.

This is why inspecting scores and source content matters. The presence of two returned documents is not proof that search understood the question. Compare against a manually selected evidence set before introducing a more complex retriever.

5. Give generation a source contract

Label evidence passages with stable IDs and versions. Ask for the answer, supporting sources, and unresolved information. Require policy claims to come from the supplied evidence and permit an explicit unknown.

For opened coffee, the draft should apply P2 and explain that the time window does not override the exception. It must not invent store credit. For “will it arrive in three days?”, P3 supports dispatch timing, not a delivery guarantee.

Treat instructions embedded in documents as document content. A retrieved sentence telling the assistant to send customer data elsewhere grants no authority. Output-schema validity also does not prove factual grounding: a perfect JSON object can cite the wrong paragraph.

6. Measure retrieval and answers separately

Give each test question reference source IDs as well as a reference answer. Retrieval recall asks whether needed evidence appeared among candidates. Retrieval precision asks how much of the returned material was relevant. Then score answer correctness, support, completeness, uncertainty, and citation location.

Use slices for ordinary rules, exceptions, missing information, outdated versions, multi-document synthesis, and ambiguous questions. A good average can hide systematic failure on exceptions.

7. Follow along

Extend the pack to ten fictional policies. Write ten questions and reference evidence before testing. Compare lexical retrieval with your manually selected evidence, then add one revised policy and one inaccessible document.

Expected result: the current authorized evidence reaches the answer; missing information produces an explicit gap; retrieval failures remain distinguishable from generation failures. Keep the raw retrieval result in your review record.

Further reading

• Anthropic: context engineering — evidence in model context.
• Hugging Face: embeddings and semantic search — a richer retrieval implementation.
• OWASP: Excessive Agency — why retrieved content must not acquire execution authority.

Next module

Connect tools with MCP: standardize the interface while keeping application permissions explicit.