You cannot improve retrieval you only eyeball. This lesson teaches you to measure it, and to separate the two questions hiding inside any bad answer.

The two-question diagnosis

When an answer is wrong, ask in order: did retrieval return the right context, or did the model ignore the right context it was given? These are different bugs with different fixes. A retrieval miss means fix chunking, hybrid, or reranking. A generation miss on good context means fix the prompt or the model. Measuring them separately is the difference between guessing and engineering.

The smallest useful eval

A golden set: a handful of known questions, each paired with the source chunk that should be retrieved, plus at least one question whose honest answer is "not in the docs" so you test the refusal path. Run it, get a retrieval hit rate and an answer-correct rate, change one thing, run it again, and watch the number move.

Why it matters

Without evals, every change to your pipeline is a coin flip you cannot see the result of. With even a five-question golden set, retrieval becomes something you steer rather than something you pray over. This is the on-ramp to the full TRACE evals masterclass in Week 4.

The Claude Code method here

Evals are a perfect job for a skill paired with a subagent. Keep the golden set and scoring script in an `evals` skill, and run it inside a subagent — an isolated context — so the noisy work of loading documents, embedding, and scoring does not clutter your main conversation. The subagent returns just the scores; your main session stays clean.

Skills for the procedure, subagents for context isolation, hooks (later weeks) for deterministic enforcement. Choosing the right layer for each job is what separates a tidy agent setup from a tangled one.