72 · Compliance and trust · Sales and marketing · live
GPSR listing pack
Eval results
Scored on a held-out or test splitRun 27 Sep 2026Eval write-up (decosa-api, access required)
- Missing Article 19 elements found, frozen15 / 19test splitn = 190 false. The 4 misses were products with a model number but no batch or GTIN; the pack now says 'check' there (Art. 9(5) accepts a type).
- Missing elements found after that fix18 / 18test splitn = 18Same test products, so no longer held out; 2 of 24 hit a gateway outage.
- Label-vs-sheet mismatches found5 / 5test splitn = 50 false flags on the other 19 products.
- Extracted fields right87 / 88test splitn = 88
- Translations with no high flag (5 languages x 23 products)115 / 115test splitn = 115
- Planted translation errors caught (6 languages)1,314 / 1,328 (98.9%)test splitn = 1,328The language pack's held-out drift set; 1,320 after later fixes.
- Planted meaning errors caught by the meaning check (negation, dropped clause, roles, entity, hedge; 6 languages)556 / 562 (98.9%); 522 as errortest splitn = 562Language pack meaning check, synthetic sentences. It also flags 52 of 192 correct translations for a look (4 as error); grammar is not checked.
- Planted translation errors caught, 5 added languages (fi, hu, el, bg, ro)1,064 / 1,078 (98.7%)test splitn = 1,078The language pack's drift set through the Qwen3.8-27B route, checker frozen; 1,068 after post-test fixes. 20 of 300 clean translations flagged when frozen (1 real), 9 after.
- Added EU languages at or above the language pack's FLORES bar13 / 15test splitn = 15FLORES-200 devtest (1,012 sentences per language). COMET-22 89.1-91.7 on Qwen3.8-27B; Irish (73.4) and Maltese (69.0, and COMET cannot judge Maltese) are served as drafts that need a reviewer.
Dataset
32 synthetic products (8 dev, 24 test): a supplier sheet and a label each, with planted problems (a missing manufacturer e-mail or address, no EU responsible person for a non-EU maker, no model or identifier, a label number or model that differs from the sheet, no warnings).
Caveats
- The products, sheets, labels and planted problems are ours; no real supplier documents.
- One fix was made after the test run (type-only identifiers); the after-fix numbers are on the same products.
- The translations are checked in code for facts, and by a model for meaning (flipped negations, dropped clauses, swapped roles, wrong names); grammar is not checked and no native speaker has reviewed them.
- Label photos were tested on one rendered label, not on phone photos of real labels.
- The fifteen languages added on 27 Sep were checked on FLORES and on planted number errors in five of them, not on GPSR products; no native speaker has read them.
Nightly smoke check
Loading the nightly status…
- Result
- pass
- Run
- 30 Sep 2026
- Latency, this run
- n/a
- p50 over passed runs
- 12 s
- Receipts
- 37
- Model calls
- n/a
- Tokens
- n/a
- Cost per run
- $0.004
Self-host verification
Not yet verified on a fresh self-host setup.
Rehearsal bundle: gpsr-listing-pack.zip (2 KB, 6 checks). Mock inputs plus the expected results, so you can prove your own setup works before any real data touches it.
Known limits
- Synthetic products only in the eval; no real supplier sheets or label photos were measured.
- Harmonisation legislation (toys, electrical, cosmetics) adds label rules this pack does not check.
- Translations are checked for numbers, units, dates, codes, negations and locked terms, not for wording; a native speaker should read them.
- Scanned PDF sheets need the document reader block's service, which is not in the hosted demo; label photos are read by Qwen3.8.
Models and licences
Standard tier. Licence posture: permissive (Apache, MIT or BSD).
- Article 19 checklist, value lookup against the documents, label-vs-sheet comparison, listing export and signed record (no model; CPU)decosa-api GPSR pack (decosa_api/verticals/gpsr), on the language-pack block (decosa_api/lang) and the signed record (07)AGPL-3.0-or-later
- Label reading (image input) and field extraction as JSONQwen3.8-27B (NVFP4, vision tower on)Apache-2.0
- Translation of warnings and safety information (the language-pack block): German, French, Spanish, Italian, Dutch, Polish, Portuguese, CzechHy-MT2-7BApache-2.0
- Translation of warnings and safety information: the fifteen other EU languages (Swedish, Danish, Finnish, Greek, Romanian, Hungarian, Bulgarian, Croatian, Slovak, Slovenian, Lithuanian, Latvian, Estonian; Irish and Maltese as drafts)Qwen3.8-27B (the language-pack block's route for these languages)Apache-2.0
All quality evidence
Every sourced number on the tool’s Stack tab, by tier. Some are proxies from another task; their labels say so.
Standard · Qwen3.8-27B plus Hy-MT2-7B (hosted demo) (4)
- Missing Article 19 elements found (24 held-out synthetic products): 15 of 19 found, 0 false (frozen); 18 of 18 after one fix (22 products)docs/evals/gpsr-listing-pack.md, test split, 27 Sep 2026
- Label-vs-sheet mismatches found: 5 / 5, 0 false flagsdocs/evals/gpsr-listing-pack.md, test split
- Extracted fields right (model, GTIN, manufacturer name and e-mail): 87 / 88docs/evals/gpsr-listing-pack.md, test split
- Planted translation errors caught (numbers, units, negations, codes, dates; 6 languages): 1,314 / 1,328 (98.9%) frozen; 1,320 / 1,328 after fixesdocs/evals/language-pack.md, drift test split
Best · the 30B translation model (not measured) (1)
- FLORES-200 XCOMET-XXL (vendor): not measured yetHy-MT2 report, arXiv 2605.22064 (vendor figure 89.83)