Chart reader (document reader block): eval

27 Sep 2026. On a pre-release build. Code decosa_api/docreader/charts.py; lab scripts/chart_lab/; scores docs/evals/chart-reader/*.json (every number below is in one of them).

The chart reader turns a figure back into numbers so they can be checked: Kaplan-Meier curves (survival at time t, medians, numbers at risk, a printed HR), forest plots (estimate and CI per row), and bar and line charts (a value per category). Five methods were measured on the same charts.

Data

All synthetic sets are drawn by scripts/chart_lab/synth.py (matplotlib) with exact truth kept alongside: fictional drugs, trials and companies, so the set is ours (AGPL-3.0-or-later with decosa-api). Styles vary per chart: 5 fonts, 4 sizes, 72 to 200 DPI, 5 palettes (including black and grey KM curves, and navy with light blue), line widths, grids, frames, censoring marks (|, +, none), risk tables (coloured or black), legends, printed HR annotations, printed forest values (half the forest plots), log and linear forest axes, value labels on bars, horizontal bars, rotated labels. Then a quarter are saved as low-quality JPEG (q 35 to 70) and a quarter as scans (rotation up to 0.8 degrees, grey paper, noise, specks, blur, JPEG q 40 to 65).

On the test set: 1,190 survival-at-t points and 137 medians on 60 KM plots, 912 numbers-at-risk cells, 436 forest rows (256 with printed values), 541 bars, 1,087 line points.

Metrics: KM survival at t, absolute error in probability (every x tick inside follow-up plus two off-tick times); medians, error as a percentage of the time axis ("not reached" must match); numbers at risk, exact per cell. Forest: estimate and both CI bounds within 3% relative, and exact to 2 decimals where printed. Bars and lines: relative error.

Results

Held-out test (240 charts, scored once)

Median relative errors (bars / lines): model 0.32% / 0.43%, geometry 0.22% / 0.19%, hybrid 0.08% / 0.17%.

By degradation (hybrid, test): KM S(t) within 0.02: clean 90.4%, JPEG 83.8%, scans 74.4%. Bars within 2%: clean 96.1%, JPEG 95.4%, scans 64.0%. Lines: clean 96.7%, JPEG 94.8%, scans 75.8%. Forest all three within 3%: clean 86.9%, JPEG 99.1%, scans 89.5%. Scans are where it is weakest.

A fix found on the test set (so the next row is not a clean held-out number): 9 of the 23 test charts where hybrid fell back to the model were forest plots with a log axis that the model called "linear"; calibration now tries the other scale when the named one does not fit. Re-scored once after the fix: forest all three within 3% 90.8% -> 99.8%, unprinted rows 78.3% -> 100%; the other kinds unchanged (fallbacks 23 -> 14). Dev was unchanged by the fix.

DePlot on a test subset (60 charts, same charts for every method)

DePlot is good on simple line charts and poor at everything this block needs most (KM, forest, grouped bars). It is not used.

Dev (80 charts; what the code was tuned on)

Dev to test drops by 3 to 13 points for hybrid: the geometry was tuned on 80 charts and meets new styles on test.

Real public-domain figures (5)

What the real figures show that the synthetic set did not: category axes without tick marks, tick marks drawn inside the axis, group header rows in a category list, and four curves in two colours told apart only by dash. The geometry fails safe on the first three (it falls back to the model's reading of the printed labels, which were exact on two of three bar charts) and fails on the fourth. Five figures are not a measurement; they are a list of the next things to build.

Choice

Hybrid, by measurement. The model reads text well (numbers at risk 97%, printed forest numbers 99%) and geometry badly (medians within 1% of the axis 13%); the code measures geometry well (medians 0.2% of the axis) but cannot name series or read labels without help. Together: medians within 1% of the axis for 80% (90% on clean charts), KM survival within 2 points for 85%, forest estimates and CIs within 3% for 91% (99.8% after the log-axis fix), bars and lines within 2% for 89% and 92%.

Numbers at risk: the page parser reads the table and the rows go to curves by name; when that fails, the model's reading is used. On test that gave 89.6% exact against the model's own 97.1%: the model reads the table better than the parser on dense or tightly spaced tables. Switching the risk table to the model's reading (with the parser as a cross-check) is the obvious next change; it was seen on test, so it is not made here.

Time and cost

Measured under the shared gateway's load, test set, 4 in parallel: the model call takes 9.3 s per chart (median), the page parser 0.8 s per call, the geometry about 0.6 s of CPU per chart. About 970 prompt and 440 generated tokens per chart: roughly $0.001 per chart at the gateway list price ($0.30 / $1.50 per million). The docreader routes add the page-parser calls (1 to 20 per chart; each 0.1 to 0.8 s).

Integration

• POST /docreader/chart: one image in, the table out with its provenance (plot box, method per value, calibration residuals, confidence, receipts) and a signed decosa.chart-receipt.v1.
• POST /docreader/read {"charts": true}: figure regions the layout model finds are cropped and read; the element carries chart, covered by the document receipt. On the figure-page sample (a born-digital CSR page) the figure is found and read: medians 14.45 and 8.50 months against 14.5 and 8.4 true, numbers at risk exact, printed HR read.
• CSR verifier figures: true: claims about KM medians, rates at a timepoint and HRs (overall, or a subgroup's row on a forest plot) are listed by one receipted Qwen call (it sees the sentences, never the figures' numbers) and compared in code within the drawing's resolution (median: the larger of 1% of the axis, 2 px, and the stated rounding; rate: 2 points plus rounding; printed HR: exact at its decimals). Numbers no table holds become traced_to_figure or figure_mismatch. On the two samples (fictional ZEN-301): both planted errors flagged (placebo median stated 9.7, the curve crosses 50% at 8.5; ECOG 1 HR stated 0.77, printed 0.71), all 7 true figure claims consistent; on the JSON sample the verifier's flags fell from 20 to 3 (the two planted errors, and a "65" in "younger than 65 years" that its context rules do not skip) because the figure-only numbers were traced to their figures. On the PDF sample the figure was located by the document reader and read from the page. This is a demo on our own samples, not a measured catch rate.
• Receipts: Qwen calls are gateway receipts; page-parser calls are decosa.model-call.v1 receipts (model paddleocr-vl-1.6, kind ocr); the geometry is code, recorded by the reader version in the chart receipt.

Licences (checked 27 Sep 2026)

Everything in the eval is our own synthetic set or US federal public-domain figures.

Where it fails

• Scans: KM survival within 2 points drops to 74%, bars to 64%. Speckle is taken for tick marks (the snap-fit helps), and thin lines lose their colour to JPEG chroma subsampling.
• Curves told apart only by line style (the FDA figure's dashed red and dashed black), and black-and-grey KM plots.
• Category axes without tick marks, ticks drawn inside the axis, and group header rows in category lists (all three from the real figures): the reader falls back to the model's reading.
• Dense risk tables whose numbers run together: the parser reads them as one number; the reader then reads cell by cell (slower) or falls back to the model.
• The KM trace reads the step curve; a median where the curve sits on 50% for a stretch is the first time it reaches 50%, which is the usual definition but not the only one.
• Confidence is a rule of thumb (calibration residuals, trace coverage, model-geometry agreement), not a calibrated probability.
• Only the four chart kinds. Pie charts, stacked bars, box plots, waterfall plots and spider plots are not read.

Own-model candidate

No open chart model with a clean licence reads KM plots or forest plots well enough (DePlot 28% of KM survival values within 2 points; PaddleOCR-VL's chart task 59%). The synthetic generator makes unlimited KM, forest, bar and line charts with exact truth, including the hard styles the eval exposed. A small model trained on it would replace the part the model does worst (geometry) and could also learn series-by-line-style. Proposal (not done here): fine-tune PaddleOCR-VL-1.6 (0.9B, Apache-2.0, already deployed) on its chart task with 50k synthetic charts rendered to our JSON schema, on the Mac or our server; keep the code geometry as the check. Released under Apache-2.0 once approved.

Checkable properties (chart samples, rehearsal)

1. chart-km: medians within 0.7 months of 14.5 and 8.4; numbers at risk exactly those in km.json; HR 0.62 (0.50 to 0.77) read from the annotation.
2. chart-forest: every row's estimate and CI equal to forest.json, source "printed", agreeing with the drawing.
3. chart-bar: all ten bars within 60 (1% of the axis) of bar.json.
4. Every chart result carries a decosa.chart-receipt.v1 that verifies and fails once the table is changed.
5. figure-page read with charts: true: one figure element with a KM chart; the document receipt verifies.
6. CSR zenavotide-301-figures with figures: true: exactly the placebo median and the ECOG 1 HR flagged.