9 min read
Changing the AI model cut made-up claims in job applications
Asked about skills an applicant didn't have, the old model made up a claim in 24 of the 33 answers a strict checker read. Claude Haiku 5.5 did it once in 29.
Useful Apply helps people apply for jobs. It tailors their CV and cover letter to an ad, keeps track of their applications, and drafts answers to the questions application forms ask, such as “Have you used PyTorch in production?” or “When can you start?”. Those answers go out under the applicant’s name. If one claims a skill the applicant doesn’t have, it can carry them past the first screen and then sink them in the interview. A made-up claim is the worst mistake the app can make.
Until October 2026, those answers and six smaller jobs in the app ran on gpt-oss-120b, an open model (one anyone can download and run) hosted by Cloudflare. I tested it against Claude Haiku 5.5 on the same prompts, the written instructions the app sends a model, and moved everything to Haiku. On questions about skills the applicant lacks, a strict checker found a made-up claim in 24 of the 33 old answers it read, and in 1 of the 29 from Haiku. Haiku was faster too: a typical answer took 2.8 seconds instead of 8.8, at under half the cost per answer. The application tracker went from 48 to 139 correct out of 150, though most of the old model’s misses there broke a style rule rather than doing anything wrong.
Answers to questions about skills the applicant doesn't have8 questions, each asked 5 times. One square per answer: a row per question, a column per attempt
24 → 1answers with a made-up claim
gpt-oss-120bThe old model
24of 33 answers read
Claude Haiku 5.5The new model, medium thinking
1of 29 answers read
Made-up claimChecked, no made-up claimFailed a format check first, so not read
How I tested it
An eval is a fixed set of cases with known right answers that a system is scored against. I built one for each of the seven jobs moving to the new model: the chat assistant, the application tracker, screening answers (the replies to application-form questions), detecting the language of a CV or ad, ranking the daily job feed, planning job searches, and summarising a user’s career from their CV.
Every case ran through the code the app actually ships. Only login, the clock, the stored data and the limit on how often a user can call the app were fixed, so runs could repeat. The old model kept its old production settings. The test applicants are fictional: made-up people with made-up CVs, no real user’s data.
Haiku 5.5 can think before it answers, at a level you choose: off, low, medium or high. I ran all four alongside the old model, and repeated each case because a model can answer the same question differently: the 16 screening cases 5 times (80 answers), the 50 tracker messages 3 times (150), the 70 language cases 3 times (210).
Where the question was about facts, a judge read the answers: Claude Opus 5.5 at high effort. It saw a strict marking guide, the job ad, the applicant’s CVs, a list of what the CVs don’t support, the question and the answer, nothing else. Simple things, like the format or a word count, were checked by code first, and the judge only read answers that passed. I wrote down the rule for picking a winner before any paid run: the lowest thinking setting within one failure of the best one, with no more serious errors than the old model.
The answers that mattered most
Half of the 16 screening cases are about logistics: start date, notice period, salary. The other half are the hard ones. The job ad asks for a skill the applicant’s CV doesn’t have, and the question goes straight at it. A good answer says what the CV does support and points to related experience. A bad one claims the missing skill, or invents a number.
A made-up claim is anything the CVs don’t support: a missing skill claimed as held, an invented number, a result moved from one job to another. One answer said the applicant had added failure alerts to a forecasting pipeline; none of her CVs mention alerts. Another moved a time saving from one job on the CV to a different one. Another took five years in retail analytics and called it five years of forecasting.
The judge only reads an answer that first passes the format checks: the right shape and parts, a length within the range for that question type, no dash and no banned word. The rule is the same for both models. Of the 40 hard answers, the judge read 33 from the old model and found a made-up claim in 24. It read 29 from Haiku and found one. A second run on the final version of the prompt gave the same result: 1 in 29.
On the logistics half, the old model got the start date, notice period or salary wrong twice. Haiku at medium never did. Across all 80 answers, 47 passed every check on the old model and 68 on Haiku.
The tracker’s misses were mostly a dash
The application tracker lets you type things like “I’ve got an interview with them next Tuesday at 10” and updates the right application. Each of its 50 test messages has one correct outcome: the right action on the right application, with the right date. The reply must also be at most two sentences with no dash, because my style rules ban dashes in the app’s copy.
The old model passed 48 of 150. That number is unfair to it. Of its 102 failed messages, 92 had a dash in the reply, and 78 failed on style alone. Counting only real mistakes, it made 24: 10 status changes nobody asked for, 8 it missed, 5 wrong dates and 1 wrong action. Haiku at medium passed 139 of 150. Ten of its misses were replies longer than two sentences; one was a real mistake.
Why tracker messages failedOut of 150 messages per model
24 → 1real mistakes, once style rules are set aside
Those are the mistakes a user feels: a status changed behind their back, an interview on the wrong day.
Every job the old model ran
The old model ran as the comparison on four jobs that produced a score. Haiku came out ahead on all four.
Share of cases passed, old model to newHollow dot: gpt-oss-120b. Filled dot: Claude Haiku 5.5 at the setting I shipped. Scale 0 to 100%
On language detection, the old model fell back to English on 4 documents that weren’t English; Haiku’s one miss was an English ad from a German company.
The career summary is the honest exception. It reads a CV and writes a short profile the job feed uses. On the old prompt, both models invented things: the old model in 31 of 36 profiles, Haiku at medium in 16. The errors were wrong job-length arithmetic, programming languages listed as spoken ones, goals the CV never stated and loosely quoted figures. I moved the duration maths into code and tightened the prompt, and Haiku dropped to 3 of 36. That fix is a separate change, so the comparison above uses the old prompt for both, and I didn’t rerun the old model on the new one.
The other three jobs have no comparison. The search-planning judge failed while the old model was in the run (a typical old-model call took 25.1 seconds), and I dropped it before the chat and feed-ranking runs as the slowest option.
How hard to let it think
Thinking costs time and money, so I wanted the lowest setting that held up. On screening, below, “made up” counts flagged answers out of those the judge read, “logistics” counts wrong start dates, notice periods or salaries, and “time” is a typical answer. I shipped medium.
| Thinking | Passed | Made up | Logistics | Time |
|---|---|---|---|---|
| Off | 41/80 | 2/20 | 17 | 1.4s |
| Low | 63/80 | 5/28 | 0 | 2.0s |
| Medium | 68/80 | 1/29 | 0 | 2.8s |
| High | 69/80 | 2/31 | 0 | 3.2s |
| gpt-oss-120b | 47/80 | 24/33 | 2 | 8.8s |
Thinking off lost on every job that writes text or states facts, so nothing shipped with it.
Five jobs followed the rule I’d fixed in advance: chat, language detection and search planning at low, the tracker and screening at medium. I set two by hand. The career summary runs at medium, though low was within one failure, because made-up claims matter most and its extra 8 seconds happen in the background. Feed ranking runs at medium because it was best on pass rate and ranking quality; high hit its cap on how much it may write on 10 of 36 calls.
Faster and cheaper per answer
One screening answer, old model to newTypical call and slowest one in twenty, over 80 answers each. Cost at list prices
The tracker shows the same pattern: 2.1 seconds against 3.0 for a typical call, and $0.000166 against $0.0007. Models are billed per token, a chunk of text a few characters long, here at list prices on 22 October 2026: $0.10 per million read and $0.50 per million written for Haiku 5.5, $0.35 and $0.75 for gpt-oss-120b. Part of Haiku’s saving is prompt caching: repeated parts of a prompt, like the CVs, are read back from a cache at a tenth of the price. On screening, 47% of Haiku’s input came from the cache.
What stayed the same
The prompts, code paths, test cases, graders and decision rule were the same for both models. The CV and cover letter writer, which runs on a larger model, didn’t change.
What this does not show
- Some answers were never checked for made-up claims. The judge only read answers that passed the format checks. That left 11 of Haiku’s 40 hard answers unread, all too short, and 7 of the old model’s (3 too short, 3 missing a required part, 1 with a banned word). Haiku’s unread answers bunch up: none of its five answers to a German question was read, and one of five to a question asking for sales numbers, so its 1 in 29 leaves almost two of the eight questions out. The 3 old-model answers to the German question that were read were all flagged.
- The samples are small. The headline rests on 8 hard questions asked 5 times. One more made-up claim either way would move the numbers noticeably.
- The test data is constructed. The applicants are fictional. I’ll re-check these numbers after a week of real use.
- Two things changed at once in practice. The model changed, and so did where it runs: Cloudflare for the old one, Anthropic’s own service for the new one. Speed and cost reflect both.
- Not every job has a comparison. The old model was dropped partway through, so chat, feed ranking and the second runs are Haiku only. The search-planning judge broke on its first run and judged the wrong thing on its second; the run I decided on had no old-model arm.
- One prompt changed after its run. The tracker prompt changed afterwards, and the tracker wasn’t rerun. Language detection ran with an older, looser grader that only counted fallbacks to English on documents that weren’t English.
- The career-summary judge has known blind spots. It flags a Director filed under “manager” (a marking mismatch, not a model mistake), and the fictional applicants list programming languages as spoken ones, which the new rule rejects. Both touch the 16 and the 3 above.
- Known gaps remain. Chat sometimes replies in English to a German message (5 to 8 times per setting) and runs past three sentences in 24 to 27 of 108 replies. For one fictional applicant, a London junior who needs visa sponsorship, ranking still puts some jobs without sponsorship in the top ten.
- The eval had a cost. $26.86 in all, $21.2 of it on the Opus judge. The report’s cost table shows $26.78; it predates a $0.08 chat rerun. A stopped ranking run isn’t counted.
Evidence
The report is evals/results/2026-10-22-haiku-55-swap.md in the Useful Apply repository. Every call, answer and score per job and run is in evals/results/2026-10-22-haiku-55-swap/<job>/run-N.json, and the harness, fixtures and judge prompts are in evals/haiku-55-swap/. The per-token prices behind the costs are in its lib/core.ts. The repository is private.
For an earlier model choice made the same way, see The cheaper AI model did the same job for a ninth of the price.