157 · Finance and insurance · Compliance and trust · preview
Vendor bank-change check
Eval results
Scored on a held-out or test splitRun 29 Sep 2026Eval write-up (decosa-api, access required)
- Fraud flagged (score 3 or more)9 of 9test splitn = 9Wilson 95% CI 0.70-1.00; dev 21 of 21
- Genuine changes flagged0 of 8test splitn = 8dev 0 of 18
- Bank-change requests detected19 of 19test splitn = 19dev 45 of 45
- Content signs found: open model vs keyword rules43 of 47 vs 23 of 47syntheticn = 47dev and test together; precision 0.83 vs 0.88
- Verdicts calling an email safe0syntheticn = 64
Dataset
64 synthetic emails (30 fraud, 26 genuine, 8 with no bank change; 6 not in English) and a 26-vendor file, written blind by a separate author; stratified split, dev 45, test 19.
Caveats
- Synthetic emails from one author; real BEC mail is messier.
- Small test set (9 frauds, 8 genuine changes): wide confidence intervals.
- Code signs were tuned on the dev set.
- A hacked real mailbox writing calmly passes every check in the email; the call-back is the control.
Nightly smoke check
Loading the nightly status…
- Result
- pass
- Run
- 29 Sep 2026
- Latency, this run
- n/a
- p50 over passed runs
- 8.1 s
- Receipts
- 1
- Model calls
- n/a
- Tokens
- n/a
- Cost per run
- <$0.001
Self-host verification
Not yet verified on a fresh self-host setup.
Rehearsal bundle: bank-change-check.zip (4 KB, 8 checks). Mock inputs plus the expected results, so you can prove your own setup works before any real data touches it.
Known limits
- Hosted verification ran on our pre-release server through the production gateway, before these routes reached the production API.
- Measured on 64 synthetic emails written by one author; real business email compromise is messier.
- A calm email from a real vendor's hacked mailbox that keeps the same bank country passes every check in the email; only the call-back catches it.
- No domain age, ownership or account-validation look-ups (nothing leaves the box by design).
Models and licences
Standard tier. Licence posture: permissive (Apache, MIT or BSD).
- Reads the email's text and quotes the pressure, secrecy, 'don't call' and redirected-payment signs, and reads the new account's bank and holder; the header, domain and vendor-file checks are codeQwen3.8-27B (NVIDIA NVFP4)Apache-2.0
All quality evidence
Every sourced number on the tool’s Stack tab, by tier. Some are proxies from another task; their labels say so.
Lite · one 48 GB card (1)
- accuracy on this task: not measured yet
Standard · the hosted demo, one 96 GB card (2)
- fraud flagged / genuine changes flagged, held-out test (19 emails, gateway, run once): 9 of 9 / 0 of 8decosa-api docs/evals/bank-change-check.md, measured on our server 2026-09-29, gateway route
- content signs found, open model vs keyword rules (all 64 emails): 43 of 47 vs 23 of 47decosa-api docs/evals/bank-change-check.md, dev and test together
Best · DeepSeek-V4-Flash on two more cards (1)
- accuracy on this task: not measured yet
Wanted · two large judges from different families (1)
- accuracy on this task: not measured yet