Build naive RAG, and watch it fail

Live

Build naive RAG, and watch it fail

The fastest way to understand RAG is to build the simplest possible version and then break it on purpose. You will wire the naive loop: chunk your documents, embed the chunks, store the vectors, embed the user's question, retrieve the nearest chunks, and stuff them into the prompt. It works — and then it fails in specific, named ways. Those failures are the curriculum. Every technique in the rest of the module exists to repair one of them.

Why it matters

Most people meet RAG as a finished diagram and never see it break, so when their own system misbehaves they have no map of what went wrong. Building the broken version first gives you that map. When you later see irrelevant chunks, or a confidently wrong answer, you will recognise the failure by name and know which lever fixes it.

The three failures you will trigger, on the real corpus

Boundary cut

Fixed in Lesson 3

A naive fixed-size split slices the remote-work rule in half, so retrieval returns "up to three days per week" but loses "fully remote requires director approval." The answer is half-right — worse than wrong.

Keyword miss

Fixed in Lesson 4

Ask about policy code "POL-114" and vector search fumbles, because a rare code carries little semantic signal.

Confident and ungrounded

Fixed now

Ask about a policy that does not exist and the model invents one — until you add a grounding prompt.

The grounding prompt is the first real fix

Instructing the model to use only the provided context, to refuse when the context does not contain the answer, and to cite the source of what it used converts an eager fabricator into a careful assistant. Refusing to answer is correct behaviour, not a failure.

Grounding prompt pattern

Answer using ONLY the context below.
If the context does not contain the answer, say:
"I don't have enough information to answer that."
Cite the Document ID of each chunk you used.

Context:
{retrieved_chunks}

Question: {question}

How Claude Code changes this build

You will not hand-type the naive loop. You will have a coding agent scaffold it — faster and, more importantly, repeatable. The repeatable part comes from a Skill.

A Skill is a folder containing a `SKILL.md` file: short YAML metadata (`name` and `description`) followed by plain-language instructions, and optionally scripts and reference files the agent can use. Drop a `rag-scaffold` skill into `.claude/skills/`, and when you tell the agent to "scaffold the naive RAG loop," Claude loads that skill and builds the loop your team's way every time — same file layout, same grounding prompt, same refusal path.

Tip

Progressive disclosure

At startup Claude reads only each skill's name and description (~100 tokens), so many skills cost almost nothing until used. Full instructions load only when the description matches what you are doing. The description field is the trigger — a vague description is the most common reason a skill never fires.

What you will do

Create a `sample_docs/` folder, drop in the Northwind files, and have Claude Code build the naive loop from your `rag-scaffold` skill. Then break it three ways, name each failure, and add the grounding prompt.

Running example throughout this module: the Northwind Robotics internal docs (handbook, expenses, security, product spec) from the examples pack.

Watch out

Common mistakes

Shipping naive RAG without a refusal path.
Debugging generation before checking what retrieval returned.
Writing a skill description so vague the agent never loads it.

Go deeper (optional)