Haiku vs Sonnet as a verdict writer: 8 cards, one gate, 36% fewer tokens
An A/B of two Claude models on the judgment layer of an SEC filing scanner. The cheaper model wrote more and was less calibrated. The config changed the same day.
Written up in October 2026 from the report committed on 2026-07-04. Numbers are from that run and have not been re-run.
The system
Opportunity Desk reads roughly 5,500 SEC filings a day so that a human reads about three. Deterministic detectors find the candidates: insider buy clusters on Form 4, odd-lot tender offers, spin-offs, activist 13D filings, and a short list of 8-K items. Nothing an LLM says can create or kill a candidate. That decision is made by code.
The LLM sits after the gate. It gets a card, a pre-filtered block of fields the detector already extracted, and returns one JSON object:
{
"headline": "<= 120 characters",
"thesis": "2-3 sentences",
"risk_flags": ["short phrases"],
"confidence": "low | medium | high",
"unverified": ["every assertion not grounded in the card"]
}The output is validated against that schema. An invalid response gets one stricter re-prompt. A transport failure gets two attempts, then a fallback backend. Anything still unproducible is stored as pending and retried in an hour. The pipeline never blocks on the model.
That design made the model a swappable part. So the question was: which part?
The constraint
The judge runs serially through a CLI on a subscription, not through the API. Each card is one process launch. A backfill produces about 170 cards in its first pass, and the poller must not starve while the judge works through them. So the costs that mattered were wall-clock seconds and output tokens, which on this transport is a proxy for how long the model spends thinking before it answers.
The rule going in, written in the harness docstring: Haiku ships unless Sonnet wins outright. Default to the small model. Make the bigger one earn it.
The run
One card per committed detector fixture, eight in total, parsed offline from real filings. Each card went to both models through the same prompt template, the same schema check, and the same transport. Model aliases were haiku and sonnet as the CLI resolved them that day.
| model | cards | schema-valid | avg output tokens | avg seconds |
|---|---|---|---|---|
| haiku | 8 | 8/8 | 3,284.6 | 31.52 |
| sonnet | 8 | 8/8 | 2,090.4 | 25.09 |
Both models produced valid JSON on every card. Both honored the one hard-wired instruction in the test: a microcap flagged microcap_pump_watch must carry that exact string in risk_flags. Both did.
Sonnet used 36% fewer output tokens on average and finished 20% faster. The surprise was the direction. The smaller model wrote more, not less.
Where they disagreed
Five of eight cards got the same confidence from both models. Three did not. In each of the three, I read the card and asked which answer I would have given.
Microcap insider cluster. Two insiders bought a combined $52.7K of a company with an estimated $180M market cap. Haiku said medium. Sonnet said low. Fifty-two thousand dollars across two people in a microcap is a thin signal. Low is right.
Spin-off, watch tier. A Form 10 with no distribution ratio and no record date. Haiku said high. Sonnet said low. There are no terms yet. High confidence in a watch-tier card with unset terms is a calibration miss.
8-K Item 1.05, cybersecurity incident. Haiku said low, citing unverified details. Sonnet said medium, with the reasoning that a company only files Item 1.05 once it has already determined the incident is material. That is how the rule works. Medium is defensible.
Sonnet’s headlines also carried more of the card’s numbers: the $52.7K figure, the full $36.00 to $40.00 Dutch auction range, the exact record date. Haiku’s headlines were shorter and dropped some of those.
What the averages hide
Eight cards is a small sample, and the per-card token counts swing. On the fixed-price tender card Haiku used 835 tokens and Sonnet used 2,404. On the watch-tier spin-off Haiku used 3,818 and Sonnet 1,246. The 36% is a mean over eight items with high variance, not a stable per-call ratio.
There were no gold labels. The quality comparison is my reading of eight cards against the fields in front of me. It is a judgment, not a measurement.
Fewer tokens also does not mean lower cost once you move to the API, where the two models have different per-token prices. On a subscription CLI the token count stands in for time. On metered billing you would redo this arithmetic.
The decision
By the stated rule, Haiku ships. The committed decision log says exactly that: Haiku ships, Sonnet is a viable and marginally leaner challenger.
By the end of the same day the production config said something else. The primary judge model became Sonnet, citing this report as the evidence. The next day, July 5, a rules-only pre-filter landed in front of the judge. It orders alert-capable cards first, caps each pass at 40 calls so a backfill cannot starve the poller, and sends digest-only cards to the challenger model. In production that means: cards that can push a notification to a phone go to Sonnet. Cards that only land in a daily markdown file go to Haiku.
That is not a reversal of the rule. It is a refinement of it. The rule assumed one model for every card. The run showed that the cards are not equal. A push alert is the only place a badly calibrated confidence costs a human attention. A digest line does not. So the better-calibrated model goes where calibration matters, and the cheaper model takes the rest. A week later, on July 11, a minimum-confidence gate on pushes made that dependency explicit: a card below medium confidence no longer alerts at all.
The routing split added no new model plumbing. The pre-filter had to exist anyway for ordering and the per-pass cap. Deciding which model reads a card became one more rule in it.
What I would do differently
Eight cards is enough to make a routing decision I can revisit. It is not enough to publish a quality ranking of two models, and I have not. The next step is cheap: a few dozen more fixture cards, two humans labeling confidence blind, and the same harness. The harness exists, writes its own report, and runs offline. That was the point of building it before choosing a model instead of after.