39 · Music · live
Split-sheet and metadata checker
Eval results
Scored on a held-out or test splitRun 25 Sep 2026Eval write-up (decosa-api, access required)
- Planted errors found43 of 43test splitn = 43
- Precision, all catalogs64.8%test splitKnock-on issues counted separately (9).
- Precision, typed split sheets only19 of 19test splitn = 1915 catalogs; no false alarm.
- Precision, catalogs with a scanned split sheet27 of 52test splitn = 52All 25 false alarms trace to OCR.
- IPI extracted correctly from handwriting-style scans via OCR33.9%test splitn = 59100% on typed sheets (162 writers).
- Share sums and identifiers recomputed independently, disagreements0 on 151 share sums and 374 identifierstest split
Dataset
Synthetic, fictional catalogs from our own generator (2-3 song releases with split sheets in five styles including handwriting-font scans, DDEX ERN or distributor CSV, society registration export, contract excerpts), 0-3 planted errors each. Dev 8 catalogs; test 30 catalogs with 43 planted errors, run once after dev was frozen.
Caveats
- Synthetic data from our own generator; the generator and the checker were written by the same agent, so the structured formats are friendly to the parser.
- Handwriting fonts are not handwriting; real handwritten sheets will do worse through Tesseract.
- Scanned split sheets bring many OCR false alarms (25 of 52 issues).
- One run; output at temperature 0 on a shared gateway can still vary slightly.
Nightly smoke check
Loading the nightly status…
- Result
- pass
- Run
- 25 Sep 2026
- Latency, this run
- n/a
- p50 over passed runs
- 6.7 s
- Receipts
- 5
- Model calls
- n/a
- Tokens
- n/a
- Cost per run
- $0.003
Self-host verification
Verified on 25 Sep 2026: Fresh clone of the branch into a clean directory, docker build of docker/api/Dockerfile (with Tesseract), the api service with a named volume, pointed at the already-running local vLLM (Qwen3.8-27B NVFP4 on 127.0.0.1:8114) through host networking; then torn down.
The image builds with OCR on; the planted sample runs end to end on the direct route in 4.5 s (5 attested calls, the same 9 issues as hosted) and the scanned sample in 5.2 s (OCR in the container); the signed record verifies; no names or titles reach the logs. The model server's own startup was not re-verified (no new GPU load).
Rehearsal bundle: split-sheet-check.zip (124 KB, 12 checks). Mock inputs plus the expected results, so you can prove your own setup works before any real data touches it.
Known limits
- Measured on synthetic, fictional catalogs only; real exports from each society and distributor have their own layouts.
- Handwriting-style scans read through Tesseract lose identifiers (33.9% of IPIs read correctly); they are marked as OCR and not trusted.
- A valid check digit says an identifier is well formed, not that it is registered to the person named.
- One release at a time (12 documents hosted); no bulk catalog mode and no CWR filing.
- Latency depends on the shared gateway: 30-100 s per release was measured while it was saturated.
Models and licences
Standard tier. Licence posture: permissive (Apache, MIT or BSD).
- Checker: DDEX and CSV parsing, OCR, identifier check digits, exact share arithmetic, grounding of every model-read value, reconciliation, the proposed sheet and signed records (no model; CPU)decosa-api splits module (decosa_api/verticals/splits) with Tesseract OCRAGPL-3.0-or-later (decosa-api; Tesseract 5)
- Model: reads split sheets and contract excerpts (values copied as written, with line numbers) and answers typed same-writer and role questionsQwen3.8-27B (NVFP4)Apache-2.0
All quality evidence
Every sourced number on the tool’s Stack tab, by tier. Some are proxies from another task; their labels say so.
Lite · code only, any CPU (2)
- Identifiers and share sums in the reports rechecked by an independent implementation (disagreements): 0 of 374 identifiers, 0 of 151 sumsdecosa-api docs/evals/split-sheet-check.md, 2026-09-25; synthetic catalogs from scripts/splits_data.py (fictional), test seed held out
- Planted identifier errors found (ISWC, IPI, UPC check digits; malformed ISRC): ISWC check digit 3/3; IPI check digits 4/4; UPC check digit 5/5; malformed ISRC 6/6decosa-api docs/evals/split-sheet-check.md, 2026-09-25; synthetic catalogs from scripts/splits_data.py (fictional), test seed held out
Standard · one 96 GB card (measured; hosted demo) (8)
- Planted errors found, 30 held-out synthetic catalogs (43 planted): 43 of 43 (100.0%): IPI check digits 4/4; malformed ISRC 6/6; ISWC check digit 3/3; missing co-writer 6/6; missing publisher 1/1; society mismatch 5/5; composer/lyricist swap 8/8; 105% split 5/5; UPC check digit 5/5decosa-api docs/evals/split-sheet-check.md, 2026-09-25; synthetic catalogs from scripts/splits_data.py (fictional), test seed held out
- Issues that are a planted error, not a false alarm (knock-on effects of a planted error excluded): 64.8%: id_invalid 18/36; missing_writer 6/9; pro_mismatch 5/6; publisher_mismatch 0/2; publisher_missing 2/2; role_mismatch 8/8; share_total 7/7; share_unreadable 0/1decosa-api docs/evals/split-sheet-check.md, 2026-09-25; synthetic catalogs from scripts/splits_data.py (fictional), test seed held out
- Issues that are a planted error, split by whether the catalog has a scanned split sheet: typed-only catalogs (15): 19 of 19, no false alarm; catalogs with a scan (15): 27 of 52, every false alarm traces to an OCR misread (22 of the 25 are warnings marked as OCR)decosa-api docs/evals/split-sheet-check.md, 2026-09-25; synthetic catalogs from scripts/splits_data.py (fictional), test seed held out
- Unknown role words ("beat + chords", "wrote the lyrics") resolved by the typed role question: 10 of 10 correctdecosa-api docs/evals/split-sheet-check.md, 2026-09-25; synthetic catalogs from scripts/splits_data.py (fictional), test seed held out
- False alarms on the 7 clean catalogs: 3decosa-api docs/evals/split-sheet-check.md, 2026-09-25; synthetic catalogs from scripts/splits_data.py (fictional), test seed held out
- Split-sheet values read correctly, typed sheets (162 writers): name 100.0%, share 100.0%, role 93.8%, ipi 100.0%, pro 100.0%, publisher 100.0%decosa-api docs/evals/split-sheet-check.md, 2026-09-25; synthetic catalogs from scripts/splits_data.py (fictional), test seed held out
- Split-sheet values read correctly, handwriting-style scans through OCR (59 writers): name 98.3%, share 96.6%, role 98.3%, ipi 33.9%, pro 96.6%, publisher 94.9%decosa-api docs/evals/split-sheet-check.md, 2026-09-25; synthetic catalogs from scripts/splits_data.py (fictional), test seed held out; Tesseract 5 on synthetic handwriting fonts, not real handwriting
- Dev catalogs (8, used to fix the prompt and the OCR handling): planted found / precision: 10 of 10 / 83.3%decosa-api docs/evals/split-sheet-check.md, 2026-09-25; synthetic catalogs from scripts/splits_data.py (fictional), dev seed