Run the visit with a copilot

# Assemble the Decosa visit copilot on this machine

You are setting up the self-hosted visit copilot on a Linux box with NVIDIA GPUs that the clinic controls.
- **During the visit:**
  - Voxtral Mini 4B Realtime gives live captions.
  - The MOSS-Transcribe-Diarize 0.9B service labels who said what, over rolling windows (the `turns` lane).
  - Qwen3.8-27B keeps a SOAP draft, ICD-10-CM and RxNorm codes, prior-auth prep and the E/M level current.
  - The `guidance` lane shows questions and history still worth covering, and considerations when a warning feature comes up. Each shows the transcript words and a cited public source; it is for the clinician and never written into the note.
- **After the visit:**
  - The LLM writes the note from the speaker turns, with `[[n]]` citations (reasoning off).
  - Check an AI note checks every sentence before the note is shown as final, optionally with our M17 detail checker on CPU. Failing sentences are held back and listed.
  - The paperwork drafts are filled from the visit and never signed or sent: patient instructions, a work or school note, and the FMLA WH-380-E provider sections.

In PriMock57 tests, pass 2 missed 8.4% of medical terms against 12.7% for Nemotron streaming ASR (scribe-bench RESULTS.md); the Voxtral live pass also missed 8.4%, at a higher WER (13.2 vs 10.3). Work step by step, show each command before running it, and stop to ask me before anything that needs sudo or changes the firewall.

**Hard rules**
- Real patient audio goes ONLY to the API you are installing here. The hosted Decosa API and demo site must never receive PHI.
- Keep `DECOSA_LLM_ROUTE=direct`. Clinic visits then get receipts with `status: "attested"`; signed by the clinic box's own key, since nothing is sent to a gateway.
- Keep the API on `127.0.0.1` unless I ask for LAN access. Never expose it to the internet.

## Step 0: set up with a coding agent, rehearse on mock data, then go private

This prompt is for a coding agent running on the machine that will host the service. We recommend Claude Code with
Claude Opus 5.5; any capable coding agent works. Work in this order:

1. Set up on mock data only. During the whole setup you (the agent) work with the synthetic sample bundle below and
   nothing else. Do not ask me for real data, and do not open, read, list or copy files that hold real data, even to
   "test with something realistic".
2. Rehearse. When the steps below are done and the service is healthy, fetch the mock-data bundle for this tool,
   https://decosa.ai/samples/clinical.zip (660 KB, 21 checks, synthetic or openly licensed: see `licence` in expected.json),
   show me what is in it, and run the rehearsal against the local API:
   `docker compose exec api python scripts/rehearse.py clinical` (the api image carries the same bundle under /app/rehearsal/clinical/;
   with no key set, the script asks the local API for a short demo token). From a decosa-api checkout instead:
   `python scripts/rehearse.py clinical --bundle clinical.zip --base-url http://127.0.0.1:<PORT>`.
   It sends the mock inputs to the local API and prints PASS or FAIL for each expected property (for example: "the session ends with a done event", "the session reports no errors", "the captions carry the complaint and the MRI order"). Show me
   the full output. Every check must pass. If one fails, fix the install and run it again; never edit `expected.json`
   to make a check pass.
3. Stop there. Once the rehearsal passes, tell me, and I will run my own data against the local API myself, on this
   machine.

For the person running this: a coding agent that runs in the cloud sees everything in its context, including files it
reads, command output and anything pasted into the chat. Keep real data out of the chat and out of anything the agent
can read. Switch to your own data only after the rehearsal has passed and the agent's work is done.

## 1. Check the machine
```bash
nvidia-smi --query-gpu=name,memory.total,driver_version,compute_cap --format=csv
df -h /var/lib/docker 2>/dev/null || df -h /     # need ~80 GB free for images and weights
docker --version && docker compose version
docker run --rm --gpus all ubuntu:24.04 nvidia-smi -L   # proves the NVIDIA container toolkit works
```
- Driver 580 or newer is required for the images as built. If older, tell me and stop.
- Pick the model build from the GPU:
  - Blackwell (compute_cap 10.x or 12.x, e.g. RTX PRO 6000 96 GB, B200): defaults below (NVFP4).
  - Hopper 80 GB+ (H100/H200): `LLM_MODEL=Qwen/Qwen3.8-27B-FP8`, `LLM_REVISION=017b9c7af6b5689d5dd426a76e0bc077eb5ca20a`, `LLM_GPU_UTIL=0.62`, `ASR_GPU_UTIL=0.25`.
  - 48 GB (L40S, RTX 6000 Ada): the FP8 settings plus `LLM_MAX_LEN=16384`, `LLM_GPU_UTIL=0.70`, `ASR_GPU_UTIL=0.22`, `DECOSA_LIVE_CAP=2`.
  - Under 48 GB: both models don't fit with useful KV cache. Stop and tell me.
  - Only the Blackwell setup has been measured; the others are starting points.
- Pick a quality tier with me:
  - **lite**: a 48 GB card, pass 1 only; skip step 4b.
  - **standard**: one 96 GB Blackwell card, both passes.
  - **best**: standard plus DeepSeek V4 Flash (`deepseek-ai/DeepSeek-V4-Flash`, MIT) as the writer, on two more 96 GB cards (see 4b).
- If Docker or the toolkit is missing (Ubuntu), install them, with my OK for sudo:
```bash
curl -fsSL https://get.docker.com | sudo sh
curl -fsSL https://nvidia.github.io/libnvidia-container/gpgkey | sudo gpg --dearmor -o /usr/share/keyrings/nvidia-container-toolkit-keyring.gpg
curl -fsSL https://nvidia.github.io/libnvidia-container/stable/deb/nvidia-container-toolkit.list \
  | sed 's#deb https://#deb [signed-by=/usr/share/keyrings/nvidia-container-toolkit-keyring.gpg] https://#' \
  | sudo tee /etc/apt/sources.list.d/nvidia-container-toolkit.list
sudo apt-get update && sudo apt-get install -y nvidia-container-toolkit
sudo nvidia-ctk runtime configure --runtime=docker && sudo systemctl restart docker
```

## 2. Images and models
Work in `~/decosa-clinical`. The images are `${DECOSA_REGISTRY}/decosa-{llm,asr,api}:0.1.0` (**publishing soon**). Try `docker pull` first. If a pull fails, build them:
- `decosa-llm`: `FROM vllm/vllm-openai@sha256:c2914767605584b6d8f45686b82de173ecc99e781897aa3d0a66dacd72c51ae1` (vLLM 0.29.0) with `ENTRYPOINT ["vllm","serve"]`.
- `decosa-asr`: `FROM vllm/vllm-openai:v0.27.1`, then `RUN pip install --no-cache-dir "mistral-common[audio]" soundfile librosa`, with `ENTRYPOINT ["vllm","serve"]`.
- `decosa-api`: needs the decosa-api source (`docker build -f docker/api/Dockerfile -t ${DECOSA_REGISTRY}/decosa-api:0.1.0 .` in the checkout). If I haven't given you the source, stop and ask me for it.

The weights (both Apache-2.0, about 25 GB) download from Hugging Face on first start into the `hf-cache` volume:
- `nvidia/Qwen3.8-27B-NVFP4` at revision `482ca0f3832238542f8f5295dde86b5f22711d80` (27.8B dense; NVFP4 MLP, FP8 attention);
- `mistralai/Voxtral-Mini-4B-Realtime-2602` (4.4B, BF16).

## 3. Write `.env` and `docker-compose.yml`
`.env` (adjust per step 1):
```bash
DECOSA_TAG=0.1.0
DECOSA_GPU=0
LLM_MODEL=nvidia/Qwen3.8-27B-NVFP4
LLM_REVISION=482ca0f3832238542f8f5295dde86b5f22711d80
LLM_MAX_LEN=65536
LLM_GPU_UTIL=0.60
ASR_GPU_UTIL=0.25
DECOSA_LIVE_CAP=4
DECOSA_BIND=127.0.0.1
DECOSA_PORT=8445
# the clinic web app's origin, e.g. https://scribe.clinic.local (keep comments on their own line: compose reads an
# inline comment after an empty value as the value)
DECOSA_CORS_ORIGINS=
# pass 2 (standard and best tiers, step 4b): the diarize service's URL as the api container sees it; empty = pass 1 only
DECOSA_DIARIZE_URL=
HF_TOKEN=
```
`docker-compose.yml`:
```yaml
name: decosa
x-gpu: &gpu
  deploy: { resources: { reservations: { devices: [ { driver: nvidia, device_ids: ["${DECOSA_GPU:-0}"], capabilities: [gpu] } ] } } }
x-health: &health
  test: ["CMD", "python3", "-c", "import urllib.request; urllib.request.urlopen('http://127.0.0.1:8000/health', timeout=4)"]
  interval: 15s
  timeout: 5s
  retries: 5
services:
  llm:
    image: ${DECOSA_REGISTRY}/decosa-llm:${DECOSA_TAG}
    <<: *gpu
    ipc: host
    restart: unless-stopped
    environment: { HF_TOKEN: "${HF_TOKEN:-}" }
    volumes: [hf-cache:/root/.cache/huggingface]
    command: ["${LLM_MODEL}", "--revision", "${LLM_REVISION}", "--served-model-name", "qwen3.8-27b",
      "--language-model-only", "--max-model-len", "${LLM_MAX_LEN}", "--gpu-memory-utilization", "${LLM_GPU_UTIL}",
      "--max-num-seqs", "16", "--kv-cache-dtype", "fp8_e4m3", "--speculative-config", '{"method":"mtp","num_speculative_tokens":3}',
      "--seed", "0", "--enable-force-include-usage", "--disable-uvicorn-access-log", "--host", "0.0.0.0", "--port", "8000"]
    healthcheck: { <<: *health, start_period: 900s }
  asr:
    image: ${DECOSA_REGISTRY}/decosa-asr:${DECOSA_TAG}
    <<: *gpu
    ipc: host
    restart: unless-stopped
    depends_on: { llm: { condition: service_healthy } }   # start after llm so the memory split is stable
    environment: { HF_TOKEN: "${HF_TOKEN:-}" }
    volumes: [hf-cache:/root/.cache/huggingface]
    command: ["--model", "mistralai/Voxtral-Mini-4B-Realtime-2602", "--tokenizer-mode", "mistral", "--config-format", "mistral",
      "--load-format", "mistral", "--compilation-config", '{"cudagraph_mode":"PIECEWISE"}', "--max-model-len", "45000",
      "--max-num-batched-tokens", "8192", "--max-num-seqs", "16", "--gpu-memory-utilization", "${ASR_GPU_UTIL}",
      "--served-model-name", "voxtral-realtime", "--host", "0.0.0.0", "--port", "8000"]
    healthcheck: { <<: *health, start_period: 600s }
  api:
    image: ${DECOSA_REGISTRY}/decosa-api:${DECOSA_TAG}
    restart: unless-stopped
    depends_on: { llm: { condition: service_healthy }, asr: { condition: service_healthy } }
    environment:
      DECOSA_ASR_WS: ws://asr:8000/v1/realtime
      DECOSA_LLM_ROUTE: direct            # never "gateway" with PHI
      DECOSA_LLM_URL: http://llm:8000/v1
      DECOSA_LLM_MODEL: qwen3.8-27b
      DECOSA_LIVE_CAP: ${DECOSA_LIVE_CAP}
      DECOSA_SESSIONS_PER_IP_HOUR: "1000"
      DECOSA_BUDGET_AUDIO_S: "3600"
      DECOSA_BUDGET_LLM_TOKENS: "200000"
      DECOSA_SESSION_TTL_S: "28800"
      DECOSA_CORS_ORIGINS: ${DECOSA_CORS_ORIGINS:-}
      DECOSA_CORS_ORIGIN_REGEX: '^https?://(localhost|127\.0\.0\.1)(:\d+)?$$'
      DECOSA_STUDIO_WORKER: none
      DECOSA_DIARIZE_URL: ${DECOSA_DIARIZE_URL:-}   # empty: pass 1 only (lite); see 4b
    ports: ["${DECOSA_BIND}:${DECOSA_PORT}:8445"]
    volumes: [decosa-data:/data]
    healthcheck:
      test: ["CMD", "python", "-c", "import urllib.request; urllib.request.urlopen('http://127.0.0.1:8445/healthz', timeout=4)"]
      interval: 30s
      timeout: 5s
      retries: 5
      start_period: 20s
volumes: { hf-cache: {}, decosa-data: {} }
```
Start it: `docker compose up -d`. The first start downloads weights; `llm` takes about 5–10 minutes to turn healthy, then `asr` 1–3 minutes more. Watch `docker compose ps` and `docker compose logs -f llm asr`.

## 4. Smoke test, pass 1 (synthetic visit, no microphone)
```bash
curl -s localhost:8445/healthz   # expect "ok": true, "asr": true, "llm": true, "llm_route": "direct"
TOKEN=$(curl -s localhost:8445/demo/session -H 'content-type: application/json' -d '{"vertical":"clinical"}' | jq -r .token)
curl -sN localhost:8445/demo/replay -H "authorization: Bearer $TOKEN" -H 'content-type: application/json' \
  -d '{"vertical":"clinical","script_id":"clinical-back-pain"}' | tee /tmp/clinical-smoke.sse | grep -o '"type": *"[a-z]*"' | sort | uniq -c
grep -o '"lane": *"[a-z_]*"' /tmp/clinical-smoke.sse | sort -u
```
The replay runs a synthetic 2-minute visit in real time through the real ASR and LLM. Pass criteria:
- the stream starts with `ready` (with `"receipts": "attested"` and the demo banner);
- `transcript` events arrive within a few seconds;
- `lane` events appear for `guidance`, `note`, `codes`, `priorauth` and `visit_level`;
- the `guidance` lane's data has `audience: "clinician"`, `inserted_into_note: false`, `alarm: false` and `lint.ok: true`. Every item has `basis.quote` and `basis.source.url`, and the `note` lane has no suggestions in it;
- `receipt` events carry `"status": "attested"`;
- it ends with `done`, whose `summary` has `note`, `note_markdown`, `codes`, `priorauth` and `visit_level` (and `"pass2": "unavailable"` while `DECOSA_DIARIZE_URL` is empty).

Also try `clinical-diabetes-followup`. On our server, with the LLM on its own GPU, lane updates took about 1.5–7.5 s (median), and the final note arrived 12–24 s after the audio ended. Report what you measure here.

## 4b. Speaker labels, the checked note and the paperwork (standard and best tiers)
decosa-api runs the rest itself, all with receipts:
- during the visit it sends rolling windows of the session audio (every 15 s of new audio) to a diarize service and labels each window's speakers as Clinician/Patient/Other;
- when the visit stops it commits the last window and writes the cited note from those turns;
- it self-checks the note, then drafts the paperwork.

The results arrive as lanes (`turns`, `final_note`, `selfcheck`, `paperwork`, `considerations`) and in the same `done` event. You only have to run the diarize service and set `DECOSA_DIARIZE_URL`.
- The service is `services/diarize` in the decosa-api source (MOSS-Transcribe-Diarize 0.9B, weights revision `704aa4a9c304e8520be88901e0d1960158ef5b15`). It has no published image yet. Install it from the source with the `uv` commands at the top of `services/diarize/requirements.txt` and run `DIARIZE_HOST=<address> DIARIZE_PORT=8092 DIARIZE_DEVICE=cuda:0 .venv-diarize/bin/python services/diarize/server.py` on this box (a systemd unit is fine). Bind it to the docker bridge address (`ip -4 addr show docker0`, usually `172.17.0.1`), which the `api` container can reach and the LAN cannot.
- Add `extra_hosts: ["host.docker.internal:host-gateway"]` to the `api` service, set `DECOSA_DIARIZE_URL=http://host.docker.internal:8092` in `.env`, and run `docker compose up -d api`. `curl -s localhost:8445/healthz` should then show `"diarize": true`.
- For standard, lower `LLM_GPU_UTIL` to 0.55 so the diarizer (about 4 GB) fits beside the live models. That's an estimate: the one-card fit has not been measured.
- Smoke test: rerun the step 4 replay with `"script_id":"clinical-visit-copilot"` (2:52, a back injury at work with a work note and FMLA). `done.summary` must now have:
  - `final_transcript.lines` with `Clinician` and `Patient` roles;
  - `final_note.status: "self-checked"`, with every kept sentence ending in `[[n]]`;
  - `selfcheck.summary`;
  - `paperwork.drafts` for `instructions`, `work_note` and `wh380e`, with `signatures_filled: 0`.

  On our server (2026-09-29, gateway route under shared load) everything arrived 34–72 s after the audio ended (median 60 s over 3 runs); on the direct route it took 20 s.
- **The M17 detail checker (optional, CPU):** it upgrades the self-check's "detail not in the visit" flags to changed-detail errors. Set it up as in the Check an AI note kit (`services/detail_checker`, weights `decosaai/decosa-note-detail-checker-modernbert-large`, `DETAIL_PORT=8495`), then give the api `DECOSA_DETAIL_URL=http://host.docker.internal:8495`. Without it, the self-check runs the sentence judge alone and says `detail_checker: "off"`.
- If the diarizer is down, the visit still finishes. `done` arrives with a note written from the live captions and marked not self-checked, and an `error` event says the speaker labels failed.
- Paperwork uses the clinic's own details. Set `DECOSA_CLINIC_PROFILE` in `.env` to one JSON line with `clinician`, `clinic`, `address`, `specialty`, `phone`, `fax` and `email`; `POST /copilot/paperwork` also takes them per draft. Without it, the drafts carry the built-in demo profile, which is labelled demo.

Alternative: a standalone worker for recordings saved to disk (not needed when the api runs pass 2 as above).
- Image `decosa-pass2` (local): `FROM vllm/vllm-openai:v0.27.1`, then `RUN pip install --no-cache-dir "git+https://github.com/OpenMOSS/MOSS-Transcribe-Diarize" soundfile openai` and `ENTRYPOINT []`. If transformers conflicts, pin what the MOSS repo asks for. Add it to compose as service `pass2`: GPU as `llm`, `hf-cache` mounted, `./visits:/visits`, `depends_on: llm`, `command: ["python","/app/after_visit.py","--watch","/visits"]`.
- Write `after_visit.py`. For each new `/visits/<id>.wav` (16 kHz mono, saved by the clinic app on this box):
  1. **Transcribe.** Load `OpenMOSS-Team/MOSS-Transcribe-Diarize` (revision `704aa4a9c304e8520be88901e0d1960158ef5b15`) with `AutoModelForCausalLM.from_pretrained(..., trust_remote_code=True, dtype=torch.bfloat16, attn_implementation="sdpa")` and `AutoProcessor`. Then call `moss_transcribe_diarize.inference_utils.build_transcription_messages(wav, prompt=DEFAULT_PROMPT)` and `generate_transcription(model, processor, messages, max_new_tokens=8192, do_sample=False, ...)`, and parse the result with `parse_transcript`. You get segments `{start, end, speaker, text}` labelled `S01`, `S02` and so on.
  2. **Role map.** Make one call to `http://llm:8000/v1` (model `qwen3.8-27b`, `max_tokens` 60, `chat_template_kwargs: {"enable_thinking": false}`) with this system prompt: "Decide which label is the CLINICIAN (asks questions, examines, advises, plans) and which is the PATIENT. Answer with one JSON object only, e.g. {"S01": "Doctor", "S02": "Patient"}; map extra labels to "Other"." Relabel the lines with the answer.
  3. **Cited note.** Number the transcript lines. Ask for a SOAP note with reasoning off and this rule: "Every sentence you write must end with a citation of the line number(s) that support it, formatted like [[12]] or [[12,15]]. A sentence with no supporting line must not be written." For the best tier, send this call to the DeepSeek endpoint instead.
  4. **Verify.** For each sentence, pass its cited lines ±1 line to the LLM as a judge. The judge first compares each fact (drug, dose, number, side, duration, negation, who it is about), then ends with the line `VERDICT: <SUPPORTED|PARTIAL|UNSUPPORTED> LEAK=<yes|no>`. Keep the verdict last.
  5. Write `/visits/<id>.note.json` and `.note.md`. Flag every sentence that isn't SUPPORTED, and every LEAK, for the clinician.
- Smoke test: `docker compose cp api:/app/demo_scripts/audio/clinical-back-pain.wav ./visits/test.wav` (a synthetic visit), wait for `test.note.md`, and check that most sentences carry `[[n]]` and every flagged one shows its verdict. For reference, a 9-minute visit took 39 s for pass 2 on our server. The role map, note and verifier have not been timed.
- Best tier only: serve DeepSeek V4 Flash with tensor parallel 2 on two other 96 GB cards. It takes about 83 GB of weights per GPU. Put it on its own OpenAI-compatible port on this box and point step 3 at it. It is not a decosa image, so ask me for the serving kit.

## 4c. Clinical considerations (all tiers, no extra service)
After each visit the api also emits a `considerations` lane (and `done.summary.considerations`): warning features stated in the visit and differential considerations from a bundled public-domain pack, each with transcript quotes, a cited source and the note's documentation gaps. For the clinician only; nothing is written into the note. Check it on its own with the synthetic chest-pain transcript in the image:
```bash
docker compose cp api:/app/rehearsal/clinical/inputs/chest-pain-transcript.txt /tmp/cp.txt
T2=$(curl -s localhost:8445/demo/session -H 'content-type: application/json' -d '{"vertical":"clinical"}' | jq -r .token)
jq -Rs '{transcript: .}' /tmp/cp.txt | curl -s localhost:8445/clinical/considerations -H "authorization: Bearer $T2" \
  -H 'content-type: application/json' -d @- | tee /tmp/cons.json | jq '[.items[] | {kind, topic, quotes: [.quotes[].text]}], .directive_lint.ok'
```
Pass: an item with `"topic": "rf_acs"` quoting the jaw, `true` for the lint, and every receipt `attested`. Then `jq '{statement, items, decisions: [{id: "c1", decision: "accepted"}]}' /tmp/cons.json | curl -s localhost:8445/clinical/considerations/decisions -H "authorization: Bearer $T2" -H 'content-type: application/json' -d @- | jq .record_check` must print `"ok": true`. On our server (26 Sep 2026, gateway route) this took 20-35 s and about 10 model calls.

## 5. Point the app at the local API
- Base URL: `http://localhost:8445` (for the Decosa web app: `NEXT_PUBLIC_DECOSA_API=http://localhost:8445`).
- Get a token with `POST /demo/session {"vertical":"clinical"}`, open `ws://localhost:8445/ws/live?vertical=clinical&token=<t>`, send 16 kHz mono PCM16 little-endian binary frames of about 100 ms, then `{"type":"stop"}`. Treat the latest `lane` event per lane id as the full state (replace, don't append). Show the `support` lane in its own panel with its `label` visible at all times, to the clinician only, and never copy its items into the note or the EHR automatically. Its red flags are considerations worded as "Possible red flag: consider whether urgent evaluation is needed"; the app must not turn them into alarms, pop-ups or sounds, and they are not for emergencies. The final note and codes arrive in `done`; your EHR integration decides where they go.
- The app's origin must be in `DECOSA_CORS_ORIGINS`, or the WebSocket closes with `4403`.
- LAN access: put a TLS reverse proxy (Caddy or nginx) in front, set `DECOSA_TRUSTED_PROXIES` to the proxy's address, and restrict it to the clinic network.
- Outbound calls: the codes lane queries NLM Clinical Tables, NLM RxNav and openFDA with only a condition name or a drug name. If policy forbids any egress, block it for the `api` container. The lanes then report no match, and the note is still written.

When done, print a short report: the tier, GPU and build chosen, image sources (pulled or built), health output, pass-1 and pass-2 smoke-test results and latencies, and anything that failed.