{"schema_version":"1","site":"https://decosa.ai","id":"expert-to-sop","num":"76","name":"Expert-to-SOP","tool_name":"Turn an expert video into an SOP","short":"Expert-to-SOP","blurb":"Turns a screen or bench recording of an expert doing a procedure into a draft SOP. Every step cites a time range and a keyframe from the recording and is checked against what the expert said. Steps that were said but never shown are marked needs confirmation, steps shown but never said are marked, and a named reviewer signs each revision into a record anyone can verify. A draft for a person to approve, not a certified procedure.","status":"live","labels":{"industry":["field-trades","general"],"job":["draft","review"],"input":["files"],"deploy":["hosted","selfhost"],"status":"live","output":["text","record"],"data":["confidential"],"hardware":"gpu-96","licence":"permissive"},"industries":["field-trades","general"],"runs_in":["hosted","selfhost"],"part_of":[],"built_from":["video-understanding","diarize","grounding","signed-record"],"models":"Qwen3.8-27B (watches the recording, checks each step) · MOSS-Transcribe-Diarize (the narration)","where":"Hosted or self-host","hardware":"Qwen3.8-27B with video input and MOSS-Transcribe-Diarize: one 96 GB GPU (measured on two shared ones); ffmpeg, keyframes and the record on CPU","final_artifact":"A draft SOP in Markdown and JSON: numbered steps with time ranges, keyframes and statuses, and a signed revision record naming the reviewer.","self_host_first":false,"verification":{"receipt_coverage":"partial","summary":"Receipt per model call, with the video's sha256 and sampling in the request hash; signed ASR receipt; signed revision record with step, keyframe and receipt hashes","manual_qa":{"hosted":{"date":"2026-09-27","result":"pass","p50_ms":20200,"p95_ms":null,"runs":null,"receipts_per_run":16,"cost_per_run_usd":0.0156},"selfhost":{"date":"2026-09-27","result":"pass","method":"Fresh clone of the pre-release branch into a clean directory, docker build of the api image (32 s; ffmpeg 7.1.5 inside), the api with a named volume on host networking against the running local vLLM (Qwen3.8-27B with video input) and diarizer on the direct route; then torn down.","notes":"The rehearsal bundle passed 10 of 10 in 12.3 s; the labeldesk sample with speech to text inside the container drafted 12 steps (1 needs confirmation) in 18.6 s with 14 attested receipts and a model-call receipt for the ASR; the revision record verified; no step, narration or title text in the logs. The local vLLM was the production unit with the 32k video-token override, not the compose default (12,288); the model server's own startup was not re-verified (no new GPU load)."},"known_limits":["Hosted verification ran on the pre-release server (decosa-api the pre-release branch on our server, gateway route, live diarizer); production gets this tool when the branch merges.","Measured on six synthetic screen recordings made and labelled by the building agent; bench footage and real narration are not measured.","On 1 of 4 held-out recordings the model cited times in 2-second slots instead of reading them; the timing warning catches that case, the times themselves are not corrected.","The narration check can call a paraphrase or a speech-to-text error a difference (the printer name heard as \"DocB Thermal\")."],"nightly_covers":null},"nightly":"https://api.decosa.ai/verify/status"},"eval_summary":{"metrics":[{"name":"Planted on-screen steps found","value":"21 / 25","unit":null,"n":25,"split":"test","note":"All four misses in one recording whose times came back in 2-second slots. Dev: 14 / 14."},{"name":"Found steps in the right order","value":"100%","unit":null,"n":21,"split":"test","note":null},{"name":"Cited start within 3 s","value":"19 / 21","unit":null,"n":21,"split":"test","note":"Median error 0.7 s"},{"name":"Said-but-not-shown steps flagged needs confirmation","value":"4 / 5","unit":null,"n":5,"split":"test","note":null},{"name":"Shown-but-not-said steps marked not narrated","value":"3 / 4","unit":null,"n":4,"split":"test","note":null},{"name":"Seen steps wrongly flagged","value":"0","unit":null,"n":25,"split":"test","note":null},{"name":"Invented steps","value":"1","unit":null,"n":null,"split":"test","note":"A real click just outside the matcher's window; counted against us"}],"dataset":"6 synthetic narrated screen recordings (32-49 s) of scripted procedures in three fictional desktop apps, a stock synthetic voice, one said-only and one shown-only step planted in each; 2 dev (the demo samples), 4 held out.","held_out":true,"caveats":["The same author wrote the apps, the procedures, the ground truth and the prompts; the two dev recordings are the demo samples.","Screen recordings of simple desktop tasks with a clean synthetic voice; bench footage, long procedures and real speech are not measured.","One held-out recording's times came back in 2-second slots; a timing warning was added after the test run and changes no step or status.","A draft for a named reviewer; it does not judge whether a procedure is safe or correct."],"date":"2026-09-27","doc_url":"https://decosa.ai/metrics/evals/expert-to-sop"},"stack":{"summary":"An expert records the screen or the bench while doing a procedure and talking. Qwen3.8-27B watches the recording and lists each step it sees with its time range; a keyframe is cut at each cited moment; every step is checked against the narration. Steps said but never shown come back as needs confirmation, and steps shown but never said are marked. A named reviewer decides the flagged steps and signs the revision into a record anyone can verify. For CI, training and quality leads who keep work instructions current.","tagline":"A recording of an expert doing a procedure, turned into a draft SOP whose steps cite a time range and a keyframe, checked against what they said.","deployment":"hosted-or-self-host","regulatory_note":"Not legal or safety advice; a draft for a qualified person to review, not a validated or approved procedure. Checked against the primary text on 27 Sep 2026: where OSHA's control of hazardous energy standard applies, 29 CFR 1910.147(c)(4)(i) says procedures 'shall be developed, documented and utilized', (c)(4)(ii) that they 'shall clearly and specifically outline the scope, purpose, authorization, rules, and techniques', and (c)(6)(i) requires a periodic inspection of each procedure at least annually (https://www.osha.gov/laws-regs/regulations/standardnumber/1910/1910.147). A drafted SOP does not meet those duties by itself: the employer writes, authorizes and inspects the procedure. Recording people at work can be personal data (for example under the GDPR where workers are identifiable); record with the knowledge of the people in the recording and keep faces out where you can. The demo recordings are synthetic.","components":[{"id":"llm","role":"Watches the recording (one video part, sampled at 1 frame a second) and lists each step with its time range; judges each step against the narration near its time (the grounding block's judge); lists instructions said but not drafted, then checks them back against the video","name":"Qwen3.8-27B (NVIDIA NVFP4), video input","hf_repo":"nvidia/Qwen3.8-27B-NVFP4","license":"Apache-2.0","params":"27.8B","quant":"NVFP4 (MLP) + FP8 (attention/GDN), FP8 KV cache, MTP speculative decoding k=3","vram_gb":57,"memory_gb_estimate":null,"engine":"vLLM 0.29.0 with --limit-mm-per-prompt image=4,video=1 and --media-io-kwargs fps=1, 240 frames","receipt_coverage":"strong","in_hosted_demo":true,"tiers":["lite","standard"],"alternative_to":null},{"id":"asr","role":"The narration as timed lines, with a model-call receipt (audio hash in, transcript hash out) signed by the instance","name":"MOSS-Transcribe-Diarize 0.9B","hf_repo":"OpenMOSS-Team/MOSS-Transcribe-Diarize","license":"Apache-2.0","params":"0.9B","quant":"BF16","vram_gb":null,"memory_gb_estimate":null,"engine":"transformers (trust_remote_code) + moss_transcribe_diarize package (decosa-api services/diarize)","receipt_coverage":"partial","in_hosted_demo":true,"tiers":["standard"],"alternative_to":null},{"id":"sop","role":"Clip preparation (same timeline, no audio, at most 1280 px), keyframes at each cited time, statuses, sign-off and the signed revision record (no model; CPU)","name":"decosa-api video block and SOP module (decosa_api/video, decosa_api/verticals/sop)","hf_repo":null,"license":"AGPL-3.0-or-later","params":null,"quant":null,"vram_gb":0,"memory_gb_estimate":null,"engine":"Python 3.12; ffmpeg and ffprobe","receipt_coverage":"partial","in_hosted_demo":true,"tiers":["lite","standard"],"alternative_to":null}],"tiers":[{"id":"lite","label":"Lite · steps and keyframes, no narration check","summary":"The steps the video shows, each with a time range and a keyframe, and the sign-off record. No speech to text, so nothing is checked against what was said and said-only steps are not found.","components":["llm","sop"],"hardware":"1x RTX PRO 6000 96 GB (measured)","quality_evidence":[{"metric":"held-out test, 4 recordings: planted on-screen steps found / in order / start within 3 s","value":"21 of 25 / 100% / 19 of 21 (the draft is the same call as Standard)","source":"decosa-api docs/evals/expert-to-sop.md, measured on our server 2026-09-27, gateway route"}],"latency_note":"measured: the draft call alone takes seconds for a short recording","in_hosted_demo":false,"receipt_coverage":"strong","receipt_note":"Gateway receipt per call on the hosted route.","hosting":null},{"id":"standard","label":"Standard · steps checked against the narration (hosted demo)","summary":"Speech to text for the narration, the steps from the video, each step judged against what was said near its time, and instructions said but not shown checked back against the video and marked needs confirmation.","components":["llm","asr","sop"],"hardware":"1x RTX PRO 6000 96 GB (measured); the diarizer on a second GPU in the hosted setup","quality_evidence":[{"metric":"held-out test, 4 recordings: said-but-not-shown steps marked needs confirmation / shown-but-not-said marked not narrated","value":"4 of 5 / 3 of 4","source":"decosa-api docs/evals/expert-to-sop.md, measured on our server 2026-09-27, gateway route"},{"metric":"held-out test: said-and-shown steps confirmed / seen steps wrongly flagged / invented steps","value":"16 of 21 / 0 / 1","source":"decosa-api docs/evals/expert-to-sop.md, measured on our server 2026-09-27, gateway route"},{"metric":"held-out test: recordings where the model cited times in 2-second slots instead of reading them (a timing warning is shown)","value":"1 of 4","source":"decosa-api docs/evals/expert-to-sop.md, measured on our server 2026-09-27, gateway route"},{"metric":"demo samples (dev, used while writing the prompts): planted steps found / flags as planted","value":"14 of 14 / 4 of 4","source":"decosa-api docs/evals/expert-to-sop.md, measured on our server 2026-09-27, gateway route"}],"latency_note":"measured: under half a minute and a few cents or less per short recording at list price","in_hosted_demo":true,"receipt_coverage":"strong","receipt_note":"Gateway receipt per model call (video calls bind the video's sha256 and sampling); model-call receipt for the speech to text; signed revision record.","hosting":null}],"alternates":[],"services":[{"name":"decosa-api","port":8445,"image":"${DECOSA_REGISTRY}/decosa-api:0.1.0","purpose":"GET /sop/info, /sop/samples; POST /sop/draft (SSE or JSON), POST /sop/runs/{id}/signoff, GET /sop/runs/{id}/export?format=md|json|record. No GPU; ffmpeg inside. The upload is deleted when the run ends; reports are kept in memory for one hour."},{"name":"decosa-llm","port":8000,"image":"${DECOSA_REGISTRY}/decosa-llm:0.1.0","purpose":"vLLM OpenAI endpoint for Qwen3.8-27B with image and video input. Internal to the compose network."},{"name":"decosa-diarize","port":8092,"image":null,"purpose":"MOSS-Transcribe-Diarize for the narration (DECOSA_DIARIZE_URL). No published image yet; built from services/diarize. Without it, send the transcript with the recording."}],"tools":[{"name":"ffmpeg / ffprobe","url":"https://ffmpeg.org","license":"LGPL-2.1+ / GPL-2.0+ (run as separate programs)","purpose":"Probe the upload, make the clip the model sees, cut keyframes, extract the audio."}],"hardware":[{"tier":"1x RTX PRO 6000 Blackwell 96 GB","fits":true,"notes":"Measured: the hosted Qwen3.8-27B with video runs on one of these cards on our server (KV cache 60.17 GiB after video was turned on), the diarizer on the other."},{"tier":"1x RTX 5090 32 GB","fits":null,"notes":"Not measured. The weights are about 20 GB; a maximum video request is 32k prompt tokens of KV cache."},{"tier":"CPU only","fits":false,"notes":"The steps come from the video model. Clip preparation, keyframes, the record and verification run on CPU."}],"latency":[{"lane":"40 s narrated screen recording, full draft (speech to text, video, narration checks), hosted gateway route","typical_ms":20000,"source":"measured on our server 2026-09-27: 16-21 s on the pre-release server and in the eval (median 17.2 s dev, 18.5 s test) while other workloads used the gateway"},{"lane":"the video call alone, 120 s 1280x720 clip at the cap (32k tokens)","typical_ms":9800,"source":"measured on our server 2026-09-27, direct to vLLM; 17-19 s each with four at once"}],"benchmark":null,"notes":[]},"buyer_facts":[{"label":"What it does","value":"Turns one narrated recording of a procedure into numbered steps, each with a time range, a keyframe and a status: seen and said, seen but not narrated, narration differs, or said but not shown (needs confirmation). A named reviewer decides every flagged step and signs the revision; exports in Markdown and JSON."},{"label":"What it does not do","value":"It does not judge whether a procedure is safe, correct or compliant, and a draft is not an approved procedure. It does not hear sound effects or read audio cues other than speech, merge several recordings, or diff revisions. Recordings over 4 minutes are sampled more sparsely (one frame every 2.5 s at 10 minutes), so short actions can be missed."},{"label":"Data retention","value":"The recording and its clip are deleted when the run ends. The report, with its keyframes, is kept in memory for one hour for the token or key that made it. Logs carry run ids and counts, never titles, narration or step text. You keep the exports and the signed record, which holds hashes only."},{"label":"What leaves the box (hosted demo)","value":"The clip (no audio) and the narration text go to Qwen3.8-27B through our gateway, which Decosa operates; the audio is transcribed on Decosa's hosted service. The gateway's receipts hold hashes, not content. Self-hosted, nothing leaves."},{"label":"Accuracy","value":"On 4 held-out synthetic screen recordings: 21 of 25 planted on-screen steps found, all in order; 4 of 5 said-but-not-shown steps flagged; no seen step wrongly flagged. On 1 of the 4 the model cited times in 2-second slots instead of reading them; the draft then shows a timing warning."},{"label":"Cost per recording","value":"A few cents or less for a short recording at gateway list price (measured on the demo sample). Each run shows its own measured cost."},{"label":"Output","value":"A draft SOP with steps, time ranges, keyframes and statuses; after sign-off a decosa.record.v1 revision record with the recording hash, each step's hashes, the receipts and the reviewer (verify at /record/verify)."}],"data_handling":{"page":"/data#expert-to-sop","self_host":{"level":"confidential","leaves":"nothing","summary":"Runs on your machine; nothing is sent to Decosa or a third party by default."},"hosted":{"level":"operator-processed","demo_only":false,"summary":"TLS to Decosa's server, then decrypted and processed by Decosa's API server, with the open models run by NEAR AI through OpenRouter, with Reka AI as the only fallback under Decosa's account.","gpus":"operator-contracted","third_parties":[],"retention":"The recording and its clip are deleted when the run ends. The report, with its keyframes, is kept in memory for one hour for the token or key that made it. Logs carry run ids and counts, never titles, narration or step text. You keep the exports and the signed record, which holds hashes only.","used_for_training":false,"encrypted_while_processed":false},"sealed_tier":{"applies":false,"note":"The sealed tier (raw chat only, never use-case pipelines) is paused at launch (/docs/sealed-tier)."},"external_calls":[]},"console":{"href":"/tools/operations/expert-to-sop","input":"sop","lanes":[{"id":"draft","title":"Steps from the video","kind":"list"},{"id":"narration","title":"Checked against the narration","kind":"list"},{"id":"said_only","title":"Said, not shown","kind":"list"},{"id":"record","title":"Revision record","kind":"json"}],"samples":[{"n":1,"id":"labeldesk-add-printer","title":"Labeldesk add printer","deep_link":"/tools/operations/expert-to-sop?sample=1&autorun=0"},{"n":2,"id":"hmi-crimp-calibration","title":"Hmi crimp calibration","deep_link":"/tools/operations/expert-to-sop?sample=2&autorun=0"}],"deep_link_params":{"sample":"1-based index into samples, or a sample id","autorun":"1 = start the run once the sample is loaded; 0 (default) = only preselect","reduce-motion":"1 = turn off animations"}},"api":{"base":"https://api.decosa.ai","contract":"/api/contract.json","contract_markdown":"/api/contract.md","reference":"/docs/api","keys":"/account/keys"},"prompts":{"hosted":"/prompts/expert-to-sop-hosted.md","selfhost":"/prompts/expert-to-sop-selfhost.md","assemble":"/prompts/expert-to-sop-assemble.md","mac":null},"rehearsal":{"bundle":"/samples/expert-to-sop.zip","bundle_url":"https://decosa.ai/samples/expert-to-sop.zip","folder":"/samples/expert-to-sop/","expected":"/samples/expert-to-sop/expected.json","files":["/samples/expert-to-sop/expected.json","/samples/expert-to-sop/inputs/crimp-calibration.mp4","/samples/expert-to-sop/inputs/transcript.json"],"bytes":709777,"checks":["the gloves instruction (said, never shown) needs confirmation","the offset typed on screen but never said is marked not narrated","zeroing the height sensor is seen and said","every step has a keyframe","the cited times stay inside the recording (no timing warning)","at least six steps are drafted","an empty request is refused","the signed revision record verifies","the record fails once it is changed","every model call has a signed receipt"],"licence":"Synthetic: a fictional app written for Decosa, driven by a script and narrated by the Kokoro-82M stock voice am_michael (Apache-2.0). No real people, faces or voices. Recording CC0; part of decosa-api, AGPL-3.0-or-later.","about":"A 40-second screen recording of a fictional press operator panel: maintenance mode with a PIN, zero the height sensor, three test crimps, a height offset, save, and a log note, narrated by a stock synthetic voice. The narrator also says to wear cut-resistant gloves, which the recording never shows, and never mentions typing the offset, which it does show. The draft must mark the gloves instruction as needs confirmation, the offset step as seen but not narrated, the zeroing step as seen and said, cite a keyframe for every step, and keep the cited times inside the recording. A reviewer signs; the revision record must verify, and fail once a step count is changed.","run":{"containers":"docker compose exec api python scripts/rehearse.py expert-to-sop","checkout":"python scripts/rehearse.py expert-to-sop --bundle expert-to-sop.zip --base-url http://127.0.0.1:8445","mac":".venv/bin/python scripts/rehearse.py expert-to-sop"},"guidance":"Set up with a coding agent (we recommend Claude Code with Claude Opus 5.5; any capable coding agent works) on mock data only, run the rehearsal until every check passes, then run your own data locally yourself. Never give the agent real data during setup."},"hardware_fit":{"check":"/self-host/hardware?use=expert-to-sop","data":"/api/hardware.json","tiers":[{"id":"lite","gpu_gb":57.6,"basis":"stack","unknown":[]},{"id":"standard","gpu_gb":61.6,"basis":"estimate","unknown":[]}],"mac":null},"links":{"page":"/tools/operations/expert-to-sop","json":"/use-cases/expert-to-sop.json","metrics":"/metrics/expert-to-sop","console":"/tools/operations/expert-to-sop","console_sample":"/tools/operations/expert-to-sop?sample=1&autorun=0","stack":"/tools/operations/expert-to-sop#stack","try_live":"/tools/operations/expert-to-sop","watch":"/tools/operations/expert-to-sop","build":"/tools/operations/expert-to-sop#build","self_host":"/tools/operations/expert-to-sop#self-host","prompts":{"hosted":"/prompts/expert-to-sop-hosted.md","selfhost":"/prompts/expert-to-sop-selfhost.md","assemble":"/prompts/expert-to-sop-assemble.md","mac":null}}}