The method
Measure before you tune.
A retrieval agent answers questions over public incident response guidance. Ten questions, each with a reference answer copied from the source, each run twice so variance is measured rather than assumed. One question is a control the corpus cannot answer, to see whether the agent declines or invents. A judge grades every answer and flags anything the reference does not support. Every run is appended to an audit log with the tool query, the passages it was given with their scores, the tokens and the latency.
Public guidance only
Judge vs verbatim reference
One unanswerable question
Per-run audit log
Seven runs
Every change, measured.
Twenty graded answers per run. Added is the share of answers asserting something the reference does not support. Agree is how often the same question got the same verdict on both runs. Trap is the control question.
| Run | Change | Correct | Partial | Wrong | Cites | Added | Agree | Trap |
|---|---|---|---|---|---|---|---|---|
| Baseline | Stock chunking, stock prompt | 5 | 11 | 4 | 0.80 | 0.85 | 0.50 | 1 / 2 |
| v1 | Section chunking, keyword channel, strict rules | 7 | 6 | 7 | 0.20 | 0.65 | 0.60 | 2 / 2 |
| v2 | Chunking regression repaired | 13 | 5 | 2 | 0.40 | 0.45 | 0.80 | 2 / 2 |
| v3 | Citation rule made mandatory | 9 | 7 | 4 | 0.85 | 0.50 | 0.60 | 0 / 2 |
| v4 | Not-covered case carved out of the citation rule | 13 | 4 | 3 | 0.85 | 0.40 | 0.50 | 2 / 2 |
| v4 repeat | Identical configuration, nothing changed | 10 | 7 | 3 | 0.75 | 0.50 | 0.80 | 2 / 2 |
| v5 | Reference answers corrected | 14 | 4 | 2 | 0.55 | 0.30 | 0.80 | 2 / 2 |
The correction
I reported that the agent invents things. It did not.
The most alarming example looked decisive. Asked how long incident evidence should be kept, the agent cited General Records Schedule 24 and a three year retention period. A specific, official sounding records requirement, apparently conjured.
Checking the source first turned that on its head. GRS 24 is stated in plain text in NIST SP 800-61r2, section 3.4.3. The agent read it correctly. The reference answer had left it out. Three of ten references were defective in the same direction, all too narrow.
Correcting them and changing nothing else moved the score further than any prompt or chunking change had: fourteen correct, unsupported additions down to 0.30, and both questions previously cited as failures correct on both runs.
In a harness judged by a model, everyone tunes the prompt and the chunking. Nobody audits the answer key. It was the largest single source of error here.
What broke
Four failures worth stealing.
The answer key, not the model
The dominant flag was answers adding material the reference did not contain, peaking at 85 percent. Three of ten reference answers turned out to be too narrow, and the agent was being marked down for being more complete than the key.
Fix. Audit the reference answers against the source before trusting a single verdict.
Chunking that erased a list
Section-aware chunking consumed each heading as a label and dropped it from the body. A four phase list in an unnumbered document fell under the minimum chunk length and vanished from the index. The agent then answered with phases from a different document.
Fix. Keep the heading in the body text and pack short sections up to the chunk window.
Two rules in conflict
Making citations mandatory raised the cite rate from 0.40 to 0.85 and simultaneously broke the control question. Told to always name a source, the agent stopped saying a question was uncovered and started naming the nearest document instead.
Fix. Exempt the not-covered case from the citation requirement, explicitly.
The noise floor
An identical configuration run twice scored 13 and then 10 out of 20. The spread of the instrument is the same size as most of the differences between versions, so small gains are not distinguishable from variance.
Fix. Run the eval twice with nothing changed. That gap is the smallest result you may claim.
Limits
Twenty graded answers cannot resolve a two answer difference, as the repeat run shows. The reference answers are written by hand and are the ceiling on grading quality. The corpus is public guidance, so the agent cannot answer questions about any particular organisation's environment. Numbers on this page are read back from the harness output, not estimated.