Stop RAG Hallucinations: Force Your Bot to Cite Sources

Why RAG Still Hallucinates

RAG reduces hallucinations by grounding the model in retrieved text, but it does not eliminate them. The model can still produce statements not supported by the retrieved context, especially when the context is incomplete or when the model blends in knowledge from its pre-training. The problem is compounded because users trust a RAG system more than a base LLM, which makes uncited confabulations more dangerous in practice.

The solution is not to hope the model stays grounded. It is to engineer your prompt and output format so that every factual claim in the response is explicitly tied to a source. If the model cannot find support for a claim, it should say so instead of inventing one. You enforce this through structured output and a carefully written system prompt, not through trust.

Structured Output for Citation Enforcement

The cleanest pattern is to define a JSON schema for the response and instruct the model to return citations as a structured field. Each claim in the answer maps to a source index. Your application layer then renders the citations inline. This makes it trivial to verify every statement: if a claim has no citation, it was not in the retrieved context.

Design your schema to include an array of cited sources with their chunk index and the exact supporting text from the source. This forces the model to locate the evidence before writing the answer, which dramatically reduces the rate of unsupported statements. It also makes automated validation straightforward: you can check programmatically whether each cited chunk actually contains the supporting text the model returned.

Building the Citation Prompt

The system prompt below instructs the model to answer only from provided context and to format its response as JSON with inline citations. Pair this with your LLM provider's JSON mode or tool-calling feature to guarantee parseable output every time.

SYSTEM_PROMPT = """You are a precise research assistant.
Answer ONLY from the context chunks provided below.
Do not use knowledge from outside the provided context.
If the context does not contain enough information, respond:
  "The provided documents do not contain enough information to answer this question."

Return your answer as JSON with this exact structure:
{
  "answer": "Your answer here. Use [1], [2], etc. for inline citations.",
  "citations": [
    {
      "index": 1,
      "source_id": "the chunk id",
      "supporting_text": "The exact phrase from the source that backs this claim."
    }
  ],
  "confidence": "high | medium | low"
}"""

import anthropic
import json

def cited_rag_response(query, retrieved_chunks):
    context = "\n\n".join([
        "--- Chunk {} (id: {}) ---\n{}".format(i + 1, c["id"], c["text"])
        for i, c in enumerate(retrieved_chunks)
    ])
    user_message = "Context:\n{}\n\nQuestion: {}".format(context, query)

    client = anthropic.Anthropic()
    response = client.messages.create(
        model="claude-opus-4-5",
        max_tokens=1024,
        system=SYSTEM_PROMPT,
        messages=[{"role": "user", "content": user_message}],
    )

    result = json.loads(response.content[0].text)
    return result

answer = cited_rag_response("What is the refund policy?", chunks)
print("Answer: {}".format(answer["answer"]))
for cite in answer["citations"]:
    print("  [{}] {}".format(cite["index"], cite["supporting_text"]))

After you receive the structured response, add a validation step that checks whether each cited chunk actually contains the supporting text the model returned. A simple substring check is enough. If it does not match, flag the response for human review. This catches the rare case where a model confidently cites a chunk that does not actually contain the stated evidence.

Handling Gaps and Edge Cases

Your system will encounter queries the retrieved documents do not cover. Teach users to recognize the "not enough information" response as a correct and honest answer, not a failure. It means the system is working as designed. If you see it frequently for queries that should be answerable, that is a retrieval problem, not a citation problem: improve your chunking and embedding pipeline first.

Also watch for confidence inflation. A model returning "high" confidence on a thin context is a signal to inspect the citations. Add a post-processing check: if the number of citations is less than two for a multi-sentence answer, consider downgrading the confidence label automatically. Displaying confidence scores to users builds trust, but only if those scores are calibrated honestly.