Why I threw away the fix that scored perfectly

Useful Brain is a question-answering assistant I built for company documents: handbooks, policies, plans. An employee asks a question and gets an answer made only of sentences that actually appear in those documents, each with a citation, or a plain “not enough evidence” when the documents do not say. After the repair work in August it scored 114 of 120 on its live test. By September the misses that kept coming back had one shape: the assistant found the right document and then quoted a lookalike sentence from the document next to it.

This post is about the day I fixed that twice. A fresh run on the morning of 6 September scored 116 of 120. The first fix I wrote repaired all four misses on replay, and I deleted it, because it worked by matching this corpus’s wording and would not have survived a new set of documents. The fix I kept is smaller and more general, and took the run to 118 of 120 (98.3%). The honest framing: runs of this system land anywhere between 116 and 120, so treat the two-point move as the category totals moving, not a pinned number.

One morning, two fixesNorthwind test: 120 fixed questions, answered by GLM 5.3 Flash

118/12098.3% with the general fix, the best run on this test

Fresh run, 6 September116/120Four misses, one pattern: it found the right documents, then quoted a near-identical sentence from the document next door

Fix 1, discarded4/4Code stitched in the missing sentence by matching words. It repaired every miss when replayed. Deleted: it worked because of this document set's wording, not because it solved the problem

Fix 2, kept118/120The code spots which document the evidence points to; the AI writes the answer. No question numbers and no words from these documents in the code

What was tested

The test, an eval in the jargon, is a fixed set of 120 questions with known right answers and known right documents, scored automatically: 70 factual questions, 17 trick questions (traps) where a plausible wrong document sits next to the right one, 13 permission cases where the answer lives in a document the asking person cannot read, 10 unanswerable questions and 10 multi-hop questions whose answer needs two documents. The corpus is 65 synthetic company documents with access rules. Every scoring rule is the same as in August and none was loosened: every sentence must be a verbatim span of a retrieved document, a multi-hop answer must cite both documents, and a permission or unanswerable question must come back as insufficient_evidence.

The model was GLM 5.3 Flash on Cloudflare Workers AI, the one chosen in the model bake-off. Retrieval, the step that finds candidate passages, was locked for the whole of this work. Only the answer layer, the code between retrieval and the validator that checks every sentence, was allowed to change.

Four misses, one shape

The fresh run’s four misses were one problem wearing four outfits. In each, retrieval delivered the gold documents, the ones the answer should cite. The draft then cited a neighbouring document instead: the handbook restating a rule the dedicated policy owns, or a page that mentions a programme it does not define.

Company documents do this constantly. A retention section says “request earlier deletion through the Customer Data Access Requests process”. A recruiting page says “the hiring bonus is a different program”. The sentence in front of the model names the document that owns the answer, and the model, having found a sentence that sounds right, stops there. I started calling these pointer cases: the evidence itself points somewhere else.

When one document points to anotherWhat the AI saw on each of the four misses

Quoted in its draft answerA neighbouring document"request earlier deletion through the Customer Data Access Requests process"

names it by title

Found, but not quotedCustomer Data Access RequestsThe document that holds the rule. It was in front of the AI the whole time.

The fix that worked and got deleted

My first fix was host-side sentence grafting. When a draft cited a neighbour and left an owner document uncited, the host itself picked a sentence from the owner, choosing by how many words it shared with the question, and grafted it into the answer before validation. I replayed the four failing questions and it repaired all four. On the scoreboard it was finished.

It was also wrong, and I knew it the moment I read the selection rule back. Word overlap picks the right sentence in this corpus because these 65 documents were generated with consistent phrasing: the question and the owning sentence tend to share vocabulary. Point the same code at a real company’s documents, where the question says “days off” and the policy says “annual leave entitlement”, and the graft picks a wrong sentence with the same confidence. Worse, a wrong graft would arrive already stitched into an answer that reads fluently. The whole reason this assistant exists is that a confident wrong answer about a leave policy is worse than no answer.

So the grafting went in the bin, with its 4 of 4. The report records it under “deliberately not done”. The lesson I keep relearning on this project is that a fix that fits the test is a fix that fits the test, and the number it produces is not evidence of anything except that.

The fix I kept

The kept design splits the job in two. The host does the part that can be done deterministically and without any knowledge of the corpus: noticing that the evidence points at an uncited document. The model does the part that needs judgement: deciding whether that document actually answers the question, and in which sentence. Three changes, all in the answer layer, shipped as prompt version grounded-answer.v9.

Pointer detection. A small module reads the question, the draft and the sentences the draft cited, and looks for uncited evidence documents named by their title words, plus documents whose text explicitly contrasts itself with a cited one (“is a different program”, “separate from”, “not interchangeable”, “does not replace”). It contains no question ids and no corpus vocabulary. It selects no sentences. Its only output is a hint: these documents were named, take a second look.

A wider coverage pass. The assistant already had a coverage pass, a second model call that asks, for multi-part questions, for the exact evidence sentence answering each still-open part. It now also runs for single-part drafts that carry a pointer hint, with a stronger instruction about telling similar programmes apart. That is one extra model call, only when the evidence itself points elsewhere. As before, the host keeps an addition only if it re-validates against the evidence ledger, so the model cannot smuggle in a sentence that is not there.

Two prompt rules, and a bug. The chat prompt now says that when the evidence calls two programmes different, each must be answered from its own document. And while wiring the hints I found that single-word titles (privacy-policy.md reduces to one distinctive word once “policy” is dropped as a stop word) could never trigger a coverage hint under the old rule, which demanded two matching title words. The threshold is now the smaller of two and the number of words the title has.

How an answer is madeWhere the September change went in

1. Document search and access checkUnchangedKeyword search, meaning search and re-ranking. Not touched in this work
2. AI draft, with a search toolInstructions v9Programmes the documents treat as different are each answered from their own document
3. Pointer checkNewSpots found documents that a quoted document names by title, or that set themselves against it. It only points; it picks no sentences
4. Completeness passWidenedUsed to run only on questions with several parts. Now also on single-part answers when a pointer is found. Anything added is checked again
5. Refusal check, then rescue with exact quotesUnchangedAs left after the August review
6. Sentence checkUnchangedEvery sentence must be an exact quote with a citation, or the answer is insufficient_evidence (not enough evidence)

116 to 118

The full battery ran again with the kept fix. Factual, trap and permission questions did not move, because they were already at or within one of their ceilings. Two categories moved: unanswerable from 9 to 10 of 10, and multi-hop from 8 to 9 of 10.

Before and after, by categoryHollow is the fresh 116 run; filled is the run with the kept fix

+2questions, both in the categories that held the pointer misses

Factual69 → 69 of 70

Trick questions17 → 17 of 17

Permission13 → 13 of 13

Unanswerable9 → 10 of 10

Needs two documents8 → 9 of 10

Two misses remain, and neither has the pointer shape. q093 is an identifier lookup where the assistant refused with the right document retrieved. q120 is an expanded multi-hop question whose answer cited the complaint escalation document and dropped the service-level credit policy. I re-asked both, and both passed. That is run-to-run variance in a reasoning model, the same churn the August post warned about, and it is why the honest claim is a band of 116 to 120 rather than 118.

The kept fix, one cell per question118 of 120 passed

2misses remain, and both pass when re-asked

Factual 69/70

Trick questions 17/17

Permission 13/13

Unanswerable 10/10

Needs two documents 5/5

Needs two documents, extra set 4/5

What stayed constant

Retrieval was locked, and the numbers say so. The deterministic retrieval suite, which runs beside every live pass as a regression guard, held at exactly its August values. On the live stack, retrieval delivered the gold documents for 99.5% of questions (97.4% in the August run), with zero forbidden-document retrievals and zero degraded turns.

Document search, unchangedA fixed set of checks, with the same results as in August

Found in the top three0.912How often the right document is in the top three

How high the right document ranks0.825How high the right document sits on average

Quality of the whole ranking0.837Ranking quality across the list

Forbidden documents retrieved0Documents the user must not see, found anyway

What this does not show

• The 116 is a fresh run, not a measured change from the August 114. Between the two, the answer layer and retrieval went through another round of changes, recorded in the repo’s execution tracker rather than in an eval report, and none of those runs is frozen. The 6 September morning run of that system scored 116; that is the number this fix was measured against. It has no frozen snapshot either; its category counts are recorded in the report’s table. Only the 118 run is frozen.
• Both residual misses pass on replay, which is evidence of variance, not of a fix. Expect 116 to 120 across runs and read category totals, not single questions.
• Complete-answer latency was p50 22.9s and p95 58.9s. The report records that as comparable to August (p50 15.6s, p95 56.3s, measured beside four other model runs; a solo August segment measured p50 about 19.6s). Still above the 15s p95 production target, still an open item.
• Citation broadening is still visible: 10 of 120 grounded answers cited a document outside the gold set, one more than August’s 9. Gold documents retrieved but uncited fell from 16 rows to 15.
• The grafting attempt’s 4 of 4 was a smoke replay of the four failing questions, not a full run, and it has no frozen snapshot. That is deliberate: it was never a candidate for the record.
• The pointer detector was written against one synthetic corpus. Its title matching and contrast phrases are general in design, but they have only been exercised on the 65 Northwind documents.

Evidence

The frozen findings for the 118 run are findings.glm-5.3-flash.json under evals/results/2026-09-06/. The source report is Pointer-triggered coverage: 116 to 118 without fitting the corpus, the detector is src/lib/agent/pointer-completion.ts, and the reproduce commands are in the evals index.

This is the third post about the same assistant. The first covers the repair from 72% to 95%: Teaching an assistant to answer only from its sources. The second ran five models through the same test and kept the cheapest of the two best: The cheaper AI model did the same job for a ninth of the price.