Evaluation

Step 13 — The eval set

Story: As the person selling this, I have fifty questions with known correct answers, so that “did that change help?” has an answer instead of an opinion.

Mode: LEARN — write the acceptance test first. This step is the differentiator of the whole material.

Why now: After the system works and before you tune it. Building an eval set earlier means guessing at the failure modes; building it later means every tuning decision you already made was a guess.

Concepts:

You cannot tune what you cannot measure. Reading three answers tells you about three answers. Fluent wrong text is indistinguishable from fluent right text at a glance, which is exactly why this domain needs numbers more than most.
Where the questions come from, in order of value: real questions the business already receives (best — ask the client for their support inbox), questions you write from the corpus, and model-generated questions (weakest, and never on their own)
The circularity trap. An eval set generated by the same model you are evaluating measures agreement with itself. If you use generation to draft questions, a human must edit every one.
What one eval item holds: the question, the passage that should be retrieved, the facts a correct answer must contain, and the expected outcome — answered, refused, or escalated
Refusals and escalations belong in the set. A set of only answerable questions optimises straight into a system that answers everything, including what it should not.
Coverage on purpose: easy, hard, ambiguous, out-of-scope, adversarial, and per workspace — Ridgeline’s factual lookups, Fairview’s declined categories, Ledgerly’s version-dependent answers
Freezing the set, versioning it, and holding some questions back so you do not tune against everything you have
The cost of one run, measured before you run it fifty times

Libraries: none. This is a YAML or JSON file and a runner you write.

Expected outcome: At least 50 eval items across the three workspaces, hand-written or hand-edited, each with question, expected passage, required facts and expected outcome. At least 10 expecting refusal and 5 expecting escalation. A held-out split not used for tuning. A runner executing the set and writing results to a file. The measured cost and duration of one full run, written down.

Verification


L1 — Gating test	`ACC-13` — the runner executes the full set, produces one result per item with the retrieved passages and the outcome, and is reproducible: two runs over the same frozen set differ only in model output, never in which items ran or how they were scored.
L2 — Manual checks	(a) Read all 50 questions in one sitting. If you can predict the system’s answer to every one, the set is too easy and it will not catch a regression. (b) Check the origin of each question. Any generated by a model and not edited by you should be edited or removed.
L4 — Anti-patterns	`AP-13-a`, `AP-13-b`, `AP-13-c`, `AP-13-d` — full text
Done when	`ACC-13` green, and you know what one full run costs

Step 14 — Metrics, and judging an answer

Story: As the person selling this, I can say the assistant answers 84% of in-scope questions correctly, cites a supporting passage every time, and refuses when it should 91% of the time.

Mode: LEARN — the judge is the part everyone gets wrong, and a wrong judge is worse than no judge.

Why now: The set exists, so there is something to score. Separating the metrics is the step that turns a single vague number into a diagnosis.

Concepts:

Retrieval and answering fail separately and must be measured separately. One accuracy number cannot tell you whether the passage was missing or the model ignored it, and those have opposite fixes.
Retrieval metrics: recall at k (was the right passage there at all) and mean reciprocal rank (how high). Recall first — nothing downstream can recover a passage that was never retrieved.
Answer metrics: factual correctness against the required facts, citation validity, and groundedness — whether every claim is supported by the cited passage
Outcome metrics: refusal rate on out-of-scope questions, over-refusal on in-scope ones, escalation precision. Over-refusal is the metric that stops you gaming everything else by refusing more.
LLM-as-judge, and its failure modes: it prefers longer answers, it prefers answers that resemble its own style, it is inconsistent between runs, and it agrees with the system when both share a wrong assumption. Mitigations: a rubric rather than “is this good”, one criterion per call, forcing a citation from the answer as evidence.
Calibrating the judge against yourself. Grade 20 items by hand, compare with the judge, and measure agreement. A judge you have not calibrated is a number generator.
Structured outputs to make judge results parseable, and the constraint that the API’s citation feature cannot be combined with a structured output format — so the judge and the answerer are separate calls, which they should be anyway
Reporting: per workspace, per category, with the failures listed. The list of failures is more useful than the score.

Libraries: the anthropic SDK for the judge, your own scoring

Expected outcome: A scorer producing retrieval metrics, answer metrics and outcome metrics separately, per workspace and per category. An LLM judge with a written rubric, calibrated against 20 hand-graded items with the agreement rate recorded. A report — readable by a non-developer — with the headline numbers and every failing item listed with its actual answer.

Verification


L1 — Gating test	`ACC-14` — the scorer reports retrieval recall, citation validity and outcome accuracy as separate figures; deliberately breaking retrieval (return nothing) moves the retrieval metric and not the citation-validity metric; the judge agrees with your hand grades on at least 80% of the calibration items.
L2 — Manual checks	(a) Read every failing item and classify it: retrieval, generation, or a wrong expectation in the eval set. Fix the eval set errors — there will be some. (b) Run the judge twice on the same answers. If the scores move much, your rubric is too vague.
L4 — Anti-patterns	`AP-14-a`, `AP-14-b`, `AP-14-c`, `AP-14-d` — full text
Done when	`ACC-14` green, and you can say which of retrieval or generation is your weaker half

Step 15 — The regression gate, and tuning against it

Story: As a developer, a change that makes the assistant worse fails the build, so that six months of small improvements do not quietly average out to nothing.

Mode: LEARN — the discipline is the deliverable.

Why now: Metrics exist and are trustworthy. Now they become a gate, and the gate is what makes the numbers mean something over time.

Concepts:

A score with no gate decays. Without one, the number is checked when someone remembers, which is after the regression shipped.
Running an eval in CI: cost, duration, and non-determinism. A subset on every pull request, the full set nightly, is the usual shape.
Tolerance bands. The model is non-deterministic, so an exact threshold flaps. Set a band, and require a recorded decision to lower it.
Tuning against a held-out split. Tune on the training questions, report on the held-out ones. Tuning against the full set means measuring how well you have memorised it.
The tuning loop: change one thing, run, record, keep or revert. One variable at a time, because two changes that cancel out look like no change.
What is worth tuning, roughly in order of effect: chunking, retrieval mix and candidate count, the system prompt, the refusal threshold, the model and its effort level
When to stop. A number that has not moved in three changes means the ceiling is somewhere else — usually extraction quality from step 4.
Recording every run: what changed, what the scores were, what you decided. The history is what a client is actually buying.

Libraries: your CI, your runner from step 13

Expected outcome: An eval subset running in CI on every pull request and the full set nightly, with results published as an artefact. A recorded tolerance band and a documented override path. A tuning log with at least five recorded experiments, each with one change and its effect on the held-out split. A final report with headline numbers per workspace.

Verification


L1 — Gating test	`ACC-15` — a deliberately worse change (halve the retrieval candidate count) fails the gate; a neutral change passes; the gate reports which metric moved and by how much.
L2 — Manual checks	(a) Read your tuning log. Any entry that changed two things at once is an experiment you learned nothing from. (b) Compare your first full run with your latest. If the difference is not visible in the numbers, say so out loud — that is a real and useful finding.
L4 — Anti-patterns	`AP-15-a`, `AP-15-b`, `AP-15-c`, `AP-15-d` — full text
Done when	`ACC-15` green, the gate runs on every pull request, and you have a tuning log with five real experiments

Harness impact: AGENTS.md v4 — record the eval commands, the tolerance band, and the rule that no change to retrieval or prompting merges without an eval run.