73 · Legal · Healthcare · live
Medical chronology with page cites
Eval results
Scored on a held-out or test splitRun 27 Sep 2026Eval write-up (decosa-api, access required)
- Planted events found189 / 198test splitn = 1986 held-out synthetic packets, 96 pages; prompts and thresholds frozen after the dev split
- Date right, of events found189 / 189test splitn = 189
- Cite on the right page and box, of events found189 / 189test splitn = 189the cited box overlaps the planted line by at least half its height
- Lines that are not planted events2 / 224test splitn = 224both were therapy techniques in a PT plan listed as procedures
- Planted date conflicts flagged6 / 6test splitn = 60 false conflicts
- Planted gaps in treatment found6 / 6test splitn = 60 false gaps
- Pre-existing conditions flagged (related or not said right)12 / 12 (12 / 12)test splitn = 120 false flags
- Events on faxed copies cited on the copy too57 / 66test splitn = 66
- Seconds per 100 pages537.7 stest splitn = 96end to end on the shared hosted gateway, one packet at a time
- Planted events found, dev98 / 99dev (tuned on)n = 993 packets used while building; 1 of 114 lines not a planted event
Dataset
Nine synthetic record packets for fictional patients (3 dev, 6 held-out test), 16 pages each from five providers: typed PDFs with a text layer, scans with rotation, speckle and JPEG noise, handwritten forms and flow sheets, and faxed copies, with planted events, copies, one date conflict, one gap and two pre-existing conditions each (decosa_api/verticals/chronology/synth.py).
Caveats
- Same author wrote the generator, the answer key and the prompts; one template family, tidier than real records.
- Matching an entry to a planted event uses its type and key words; a few defensible extra lines (an operative note listed as a visit, orders as referrals) are counted as neither right nor wrong.
- No real records and no nurse reviewer's chronology to compare with.
- Timing is on a gateway shared with other workloads; a private GPU runs faster.
Nightly smoke check
Loading the nightly status…
- Result
- pass
- Run
- 27 Sep 2026
- Latency, this run
- n/a
- p50 over passed runs
- 25 s
- Receipts
- 8
- Model calls
- n/a
- Tokens
- n/a
- Cost per run
- $0.004
Self-host verification
Verified on 27 Sep 2026: fresh clone into a clean directory, api image from docker/api/Dockerfile, compose api with a named volume, direct route to the local Qwen3.8-27B and the running document reader and retrieval services, local signing; torn down after
The rehearsal bundle passed 12/12 in 68 s (the 16-page packet uploaded as files: conflict, gap, pre-existing, copies, cites in the index, reader receipts, record verified and failed when changed; every receipt attested) and the smoke module passed in 17.9 s. The reader and retrieval services were not built from the compose file here: the running ones were used.
Rehearsal bundle: medical-chronology.zip (943 KB, 12 checks). Mock inputs plus the expected results, so you can prove your own setup works before any real data touches it.
Known limits
- Hosted verification ran on the pre-release server (decosa-api the pre-release branch on our server, gateway route): the smoke test on one 4-page file, and the full 16-page sample from the console. The production API gets this tool when the branch merges.
- Measured on synthetic packets written by the same author as the prompts, one template family; not on real records or against a nurse reviewer's chronology.
- When the reader skips a line on a scan, the event on it is missed: 9 of 198 planted events were missed, mostly this way.
- Hosted runs are capped at 40 pages; a page takes about 5 s on the shared gateway.
- Billing records and charges are not extracted.
Models and licences
Standard tier. Licence posture: permissive (Apache, MIT or BSD).
- The event schema and page checks, copies, merging, conflicts, gaps, pre-existing, cites to chunks and bytes, the Markdown chronology and the signed record (no model; CPU)decosa-api medical chronology (decosa_api/verticals/chronology), importing the document reader, evidence retrieval (scans into retrieval), typed judgment and dates blocksAGPL-3.0-or-later
- One call per page (the page's elements, plus the page image for a scan) listing typed events with quote and date; typed judgments for duplicate and conflict pairs and for pre-existing conditions; re-reads of regions the parser was unsure ofQwen3.8-27B (NVFP4)Apache-2.0
- Finds and orders the regions of every page (text, headings, tables, checkboxes) with their boxesDocling 2.130 with the Heron layout model (document reader block)MIT (Docling) + Apache-2.0 (weights)
- Reads each region of a scanned page (text, handwriting, tables as cells)PaddleOCR-VL-1.6 (0.9B, document reader block)Apache-2.0
- Scans into retrieval: each page's elements as chunks with page and box, the index and layout hashes, every cite located to a chunk and byte span, and a reranked search per procedure for other pages that give another dateEvidence retrieval block: Qwen3-Embedding-0.6B + Qwen3-Reranker-4B (decosa-retrieval service)Apache-2.0
All quality evidence
Every sourced number on the tool’s Stack tab, by tier. Some are proxies from another task; their labels say so.
Lite · document reader and the model, no retrieval service (1)
- Planted events found, conflicts, gaps with BM25 only: not measured yetthe eval runner supports it (scripts/chronology_eval.py --bm25); not run on the test split
Standard · reader, retrieval service and the model (hosted demo) (5)
- Planted events found (held-out, 6 packets, 96 pages): 189 / 198 (95.5%)decosa-api docs/evals/medical-chronology.md, test split, 27 Sep 2026
- Date right / cite on the right page and box, of those found: dates 189 / 189, cites 189 / 189decosa-api docs/evals/medical-chronology.md, test split
- Lines that are not planted events: 2 / 224 (0.9%)decosa-api docs/evals/medical-chronology.md, test split
- Planted date conflicts / gaps / pre-existing flagged: 6 / 6, 6 / 6, 12 / 12; 0 false flags of each kinddecosa-api docs/evals/medical-chronology.md, test split
- Events on faxed copies carrying the copy's cite: 57 / 66decosa-api docs/evals/medical-chronology.md, test split