46 · Sales and marketing · Finance and insurance · live
Auto F&I disclosure record
Eval results
Scored on a held-out or test splitRun 25 Sep 2026Eval write-up (decosa-api, access required)
- Per-check accuracy, test B (held out), run 131/32held outn = 32Repeat run 32/32.
- Planted answers found, test B (held out), run 18/8held outn = 8
- False alarms on clean calls, test B (held out), run 11 of 14held outn = 14Repeat run 0 of 14.
- Per-check accuracy, test, run 2 (after two fixes)77/78test splitn = 78Run 1, before the fixes: 76/78 with 2 of 30 false alarms.
- Per-check accuracy on ASR transcripts of TTS audio, run 145/45test splitn = 45Audio re-voiced 26 Sep 2026 from macOS voices to Decosa house voices (Kokoro-82M) and re-run: repeat 45/45; timestamps on the gold line 21/21 then 20/21 (first build: repeat 44/45, timestamps 21/21 then 19/21).
- Timestamps on the gold line, test B, run 115/15held outn = 15Repeat 15/16.
Dataset
17 synthetic F&I role-plays for a fictional dealer, written from SB 766 and Penal Code 632: dev 3, test 10 (4 clean, 6 planted), test B 4 written after test run 1 and never used to change anything; 5 also as TTS audio (Decosa house voices, Kokoro-82M, each allowed by the consent ledger) through the diarizer.
Caveats
- Synthetic only: 17 short scripted conversations with clean TTS audio; no crosstalk, speed-read menus, other languages or 30 to 60 minute sessions.
- The same author wrote the scripts, the expected answers and the prompts, so this is not an independent label set.
- Test run 1 led to two fixes before the test set was re-run; test B is the held-out number.
- Answer probabilities are not calibrated for F&I talk.
- Cannot tell whether a written disclosure was clear and conspicuous, in the right language, or given in time.
Nightly smoke check
Loading the nightly status…
- Result
- pass
- Run
- 25 Sep 2026
- Latency, this run
- n/a
- p50 over passed runs
- 3.3 s
- Receipts
- 10
- Model calls
- n/a
- Tokens
- n/a
- Cost per run
- $0.003
Self-host verification
Verified on 25 Sep 2026: fresh clone of decosa-api into a clean directory on our server, api image built from docker/api/Dockerfile, compose api service with a named data volume, direct route, local signing; torn down after
Both smoke tests in the assembly prompt passed against the already-running local Qwen3.8-27B vLLM (127.0.0.1:8114, network_mode host instead of the compose llm service): tb4 flagged the cooling-off denial and the add-on never discussed, 10 attested receipts, record verified, 3.8 s; the pasted transcript flagged the service contract as presented as required at 00:05; no consent gave 400. Model-server startup and the diarizer path were not re-run.
Rehearsal bundle: fi-disclosure-record.zip (3 KB, 9 checks). Mock inputs plus the expected results, so you can prove your own setup works before any real data touches it.
Known limits
- Hosted verification ran on the pre-release server (decosa-api the pre-release branch on our server, gateway route), driven from the branch site in headless Chromium, including at 390 px. The production API gets this vertical when the branch merges.
- Measured on 17 synthetic role-plays written by the building agent, with clean TTS audio. It has not been measured on real F&I recordings (long sessions, crosstalk, Spanish) or with a dealer compliance officer's labels.
- The Act's add-on and payment disclosures must be in writing. The tool reads them from the jacket as the dealer states them, and cannot judge whether they were clear, conspicuous or in the right language.
- One systematic miss: a hint at cancelling ('if you did cancel there's a restocking fee') is read as explaining the 3-day right. The misstatement check still flags that call.
- Audio intake (/fi/transcribe) is self-host only. The hosted demo's audio samples were transcribed on our server and are bundled.
- It does not check advertising or the first written price (1784.41(a)), GAP loan-to-value (1784.42(a)(3)), or add-on payment timing (1784.42(b)).
Models and licences
Standard tier. Licence posture: permissive (Apache, MIT or BSD).
- Extraction (what was said about each add-on, prices, payment) and one typed yes/no/unclear check per questionQwen3.8-27B (NVIDIA NVFP4)Apache-2.0
- Recording to a timed, speaker-labelled transcript (POST /fi/transcribe, self-host; the demo's audio samples were transcribed with it)MOSS-Transcribe-Diarize 0.9BApache-2.0
All quality evidence
Every sourced number on the tool’s Stack tab, by tier. Some are proxies from another task; their labels say so.
Lite · one 48 GB card (1)
- typed checks correct / planted problems found: not measured yet
Standard · the hosted demo, one 96 GB card (5)
- held-out test B, 4 synthetic conversations: typed checks correct / planted problems found / mismatches found: 31/32 / 8/8 / 3/3decosa-api docs/evals/fi-disclosure-record.md, measured on our server 2026-09-25, gateway route, run 1; data and prompts written by the building agent, prompts frozen on a separate 3-conversation dev split
- test set, 10 conversations: typed checks correct (run 1 before two fixes / after): 76/78 / 77/78; planted 12/13; mismatches 6/6decosa-api docs/evals/fi-disclosure-record.md, measured on our server 2026-09-25, gateway route
- false alarms on clean conversations (flag or review): test 0/30 after fixes (2/30 before); test B 1/14decosa-api docs/evals/fi-disclosure-record.md, measured on our server 2026-09-25
- 5 conversations on real speech-recognition transcripts of synthetic audio: checks / planted / cited time within 3 s: 45/45 / 10/10 / 21/21 (repeat 45/45, 10/10, 20/21)decosa-api docs/evals/fi-disclosure-record.md (asr runs), measured on our server 2026-09-26 on audio re-voiced 26 Sep 2026 from macOS voices to Decosa house voices (Kokoro-82M) (the first build: repeat 44/45, 9/10, 19/21), gateway route
- real F&I recordings reviewed by a dealer compliance officer: not measured yet
Best · DeepSeek-V4-Flash on two more cards (1)
- typed checks correct / planted problems found: not measured yet
Wanted · two large judges from different families (1)
- typed checks correct / planted problems found: not measured yet