Eval: demand-reader (Read a demand before the deadline)

29 Sep 2026, build-insurance-finance-opus. Runner: scripts/insfin_eval/demand_eval.py. Data and raw results: docs/evals/demand-reader/.

Data

• 24 synthetic injury demand packages, 406 pages (letters, itemized bills, visit notes, imaging and PT notes), written blind by a separate author (a sub-agent that never saw the tool) with a generator (make_packages.py, prose in pk_a/b/c.py) and a label checker (check_labels.py: every quote on its page, every bill amount on its page, totals summed, respond-by dates recomputed from the rules it was given).
• States: CA 6, GA 6, MO 3, FL 3, MT 2, TX 2, UT 1, OH 1 (not encoded). 20 time-limited, 4 not.
• Planted: 9 deadlines shorter than the statute (one where day 60 is Columbus Day in Montana; one Georgia letter one day short), 1 early Georgia payment date, 17 missing statutory elements, 9 specials totals that don't tie (one lower than the bills), 6 provider rows that don't match their bill, 6 providers billed with no records, 6 with records but no bill, 6 treatment gaps over 30 days, 6 earlier injuries, 3 fabricated cases, 5 statutes from another state, 4 misstatements of the law, 4 immaterial terms, zero-width characters in 2 letters. Six real cases may be cited, only as real.
• Split: ordered by (time-limited, state, id), every third package held out: dev 16, test 8 (dm-02, 06, 07, 10, 11, 14, 17, 18: CA, GA, MO, MT, UT, one not time-limited). The first dev run used a per-state split that held out too few; the split was fixed before any per-package tuning, and the test set was run once, on the gateway route, after the code froze.

What was measured

• Time-limited detection; respond-by date exact (the date the letter sets, from its words and the date received); statute's minimum exact; "shorter than the statute" right; payment date exact.
• Conditions of acceptance: a labelled condition counts as found when a returned condition is on the same page with half its words overlapping.
• Missing statutory elements (recall and precision of "missing"); billed total within $1; claimed total; the difference.
• Records against bills, treatment gaps (exact dates), earlier injuries (page).
• Notes on the letter: fabricated cases, other-state statutes, misstatements of the law, zero-width characters.
• Rail: value or payment-recommendation language in the tool's own words (titles, details, condition summaries; the letter's quotes and the fixed disclaimers excluded).

Results

• The dates and totals are code: once the letter's period and trigger are read (with a quote found on the page), the arithmetic can't drift. The misses are in reading: one other-state statute and one misstatement on test were not noted.
• After the dev runs, a code pass was added next to the model's list of legal statements (sentences that name the law and a number of days), because the model skipped a Florida "627.4137 ... ten days" sentence on dev. It still missed one of two misstatements on test.
• 8 of 41 returned conditions on test are not labelled: mostly the release scope or payment method split in two, and instructions the author did not count as conditions.

After the test run

A blind BI adjuster read three recorded runs (page 91) and found problems the labels don't score. Fixed afterwards, checked on dev only (not re-run on test): two providers with similar names were merged into one bill row (bills and records are now grouped only when most words match both ways); records dated before the loss were listed as "ask for bills" (they are now marked as earlier care); a provider the letter claims with no bill and no records gets its own "ask for the bill and records" item; a condition that relies on an enclosed form or release gets "check it is in the package".

Checkable properties of the sample runs (rehearsal)

1. ga-policy-limits: time-limited; respond by 2026-09-03 (30 days from receipt on 2026-08-04); the statute's minimum is the same day; the bills total $11,910.00 and $19,540.00 of the $31,450.00 claimed is not supported by any bill.
2. A note says the letter gives 10 days where O.C.G.A. 9-11-67.1 gives at least 30; another notes the Florida statute.
3. Nothing in the tool's own words values the claim; the record statement carries value_given: false.
4. The signed record verifies; changing its respond-by date breaks it.

Caveats

• Synthetic packages from one author: real packages run to hundreds of pages, are scanned, and interleave bills and records. This version reads typed or pasted page text; scans go through the document reader first.
• Small test set (8 packages, 7 time-limited): every rate has a wide interval.
• Six states encoded (CA, GA, MO, MT, UT; FL's safe harbour and disclosure); every other state returns "not encoded: check with counsel". The Georgia text is the signed SB 83 (2024); its applicability date is from secondary sources.
• The deadline is the letter's own date; whether a court would roll it past a weekend or holiday is not decided.
• Slow under load: two to four minutes a package on a busy shared GPU.