Score an oral exam against a rubric

# Assemble Decosa structured oral assessment on this machine

You are setting up a self-hosted oral-assessment assistant on this Linux machine, for vivas and structured interviews: live captions and an examiner prompt while the session runs, then (when it stops) examiner and candidate lines, words the two recognisers disagree on marked for a re-listen, draft rubric scores that cite the candidate's lines, and a signed draft record. The examiner then confirms or overrides every score with a reason, which is sealed in a review record chained to the draft. Work step by step, show me each command before running anything that needs sudo, and stop if a check fails.

**Before anything else, remind me:** the tool drafts, a person decides every score. Tell every candidate that the session is recorded and scored with AI help, and get consent where the law requires it. Oral exam recordings and scores are education records (FERPA); for hiring in New York City, Local Law 144 needs a bias audit and notice first. Repeat these points in your final summary.

## Step 0: set up with a coding agent, rehearse on mock data, then go private

This prompt is for a coding agent running on the machine that will host the service. We recommend Claude Code with
Claude Opus 5.5; any capable coding agent works. Work in this order:

1. Set up on mock data only. During the whole setup you (the agent) work with the synthetic sample bundle below and
   nothing else. Do not ask me for real data, and do not open, read, list or copy files that hold real data, even to
   "test with something realistic".
2. Rehearse. When the steps below are done and the service is healthy, fetch the mock-data bundle for this tool,
   https://decosa.ai/samples/oral-assessment.zip (4 KB, 11 checks, synthetic or openly licensed: see `licence` in expected.json),
   show me what is in it, and run the rehearsal against the local API:
   `docker compose exec api python scripts/rehearse.py oral-assessment` (the api image carries the same bundle under /app/rehearsal/oral-assessment/;
   with no key set, the script asks the local API for a short demo token). From a decosa-api checkout instead:
   `python scripts/rehearse.py oral-assessment --bundle oral-assessment.zip --base-url http://127.0.0.1:<PORT>`.
   It sends the mock inputs to the local API and prints PASS or FAIL for each expected property (for example: "all 6 rubric criteria get a draft score", "the wrong p-value definition scores 0 or 1", "agreeing with the misreading (the null is probably true) scores 0 or 1"). Show me
   the full output. Every check must pass. If one fails, fix the install and run it again; never edit `expected.json`
   to make a check pass.
3. Stop there. Once the rehearsal passes, tell me, and I will run my own data against the local API myself, on this
   machine.

For the person running this: a coding agent that runs in the cloud sees everything in its context, including files it
reads, command output and anything pasted into the chat. Keep real data out of the chat and out of anything the agent
can read. Switch to your own data only after the rehearsal has passed and the agent's work is done.

## What you are building

| service | image | model | port |
|---|---|---|---|
| `llm` | `${DECOSA_REGISTRY}/decosa-llm:0.1.0` (vLLM 0.29.0, `vllm/vllm-openai@sha256:c2914767605584b6d8f45686b82de173ecc99e781897aa3d0a66dacd72c51ae1`) | `nvidia/Qwen3.8-27B-NVFP4` @ `482ca0f3832238542f8f5295dde86b5f22711d80`, Apache-2.0 | internal 8000 |
| `asr` | `${DECOSA_REGISTRY}/decosa-asr:0.1.0` (vLLM 0.27.1 + `mistral-common[audio]`) | `mistralai/Voxtral-Mini-4B-Realtime-2602`, Apache-2.0 | internal 8000 |
| `api` | `${DECOSA_REGISTRY}/decosa-api:0.1.0` (no GPU) | none | `127.0.0.1:8445` |
| `diarize` (optional) | built from the decosa-api source (`services/diarize`) | `OpenMOSS-Team/MOSS-Transcribe-Diarize` @ `704aa4a9c304e8520be88901e0d1960158ef5b15`, Apache-2.0 | internal 8092 |

Audio, transcripts, scores and records stay on this machine. If I only want to score transcripts I already have (no audio), skip `asr` and `diarize`: the `/oral/*` routes need only `llm` and `api`. Without `diarize`, live sessions still work, but roles are guessed from question marks and nothing is marked uncertain.

## 1. Check the GPU, driver and Docker

1. Run `nvidia-smi`. I need one NVIDIA GPU with at least 48 GB and driver 580 or newer.
   - Blackwell (RTX PRO 6000, B200): use the defaults below (NVFP4).
   - Hopper (H100/H200): set `LLM_MODEL=Qwen/Qwen3.8-27B-FP8`, `LLM_REVISION=main`, `LLM_GPU_UTIL=0.62`.
   - 48 GB Ada/L40S: FP8 checkpoint as above, plus `LLM_MAX_LEN=16384`, `LLM_GPU_UTIL=0.70`, `ASR_GPU_UTIL=0.22`, `DECOSA_LIVE_CAP=2`.
   - Under 48 GB: stop and tell me it will not fit.
   Only the Blackwell defaults have been measured; the other rows are starting points.
2. Check `docker --version` and `docker compose version`. If Docker is missing, install Docker Engine from Docker's official apt/dnf repository for this distro.
3. Check `docker run --rm --gpus all ubuntu nvidia-smi`. If it fails, install the NVIDIA Container Toolkit (`nvidia-container-toolkit`) from NVIDIA's repository, run `sudo nvidia-ctk runtime configure --runtime=docker` and restart Docker.
4. Confirm about 80 GB of free disk for images and weights.

## 2. Get the images

The images are **on request** while self-host is in early access: ask at https://decosa.ai/contact?topic=self-host, and Decosa sends the registry (set it as `DECOSA_REGISTRY`), pull access and the compose file. Try `docker pull ${DECOSA_REGISTRY}/decosa-{llm,asr,api}:0.1.0`. If a pull fails (not published yet, or no access), build from source once the `decosa-api` source is published: clone it, then `docker compose build llm asr api` in the repo, which builds the same tags from `docker/`. If neither works, stop and tell me.

## 3. Write the compose file

Create `~/decosa-oral/.env`:

```bash
DECOSA_TAG=0.1.0
DECOSA_GPU=0
LLM_MODEL=nvidia/Qwen3.8-27B-NVFP4
LLM_REVISION=482ca0f3832238542f8f5295dde86b5f22711d80
LLM_MAX_LEN=65536
LLM_GPU_UTIL=0.60
ASR_GPU_UTIL=0.25
DECOSA_LLM_ROUTE=direct   # local model; receipts are signed by this box's own key ("attested")
DECOSA_SIGNER_NAME="<who signs these records, e.g. Riverbend College, Statistics department>"
DECOSA_LIVE_CAP=4
```

Create `~/decosa-oral/docker-compose.yml` with exactly these services:

```yaml
name: decosa-oral
x-gpu: &gpu
  deploy: { resources: { reservations: { devices: [ { driver: nvidia, device_ids: ["${DECOSA_GPU:-0}"], capabilities: [gpu] } ] } } }
x-health: &health
  interval: 15s
  timeout: 5s
  retries: 5
services:
  llm:
    image: ${DECOSA_REGISTRY}/decosa-llm:${DECOSA_TAG}
    <<: *gpu
    ipc: host
    restart: unless-stopped
    volumes: [hf-cache:/root/.cache/huggingface]
    command: ["${LLM_MODEL}", "--revision", "${LLM_REVISION}", "--served-model-name", "qwen3.8-27b",
              "--language-model-only", "--max-model-len", "${LLM_MAX_LEN}", "--gpu-memory-utilization", "${LLM_GPU_UTIL}",
              "--max-num-seqs", "16", "--kv-cache-dtype", "fp8_e4m3", "--speculative-config", '{"method":"mtp","num_speculative_tokens":3}',
              "--seed", "0", "--enable-force-include-usage", "--disable-uvicorn-access-log", "--host", "0.0.0.0", "--port", "8000"]
    healthcheck: { <<: *health, test: ["CMD", "python3", "-c", "import urllib.request; urllib.request.urlopen('http://127.0.0.1:8000/health', timeout=4)"], start_period: 900s }
  asr:
    image: ${DECOSA_REGISTRY}/decosa-asr:${DECOSA_TAG}
    <<: *gpu
    ipc: host
    restart: unless-stopped
    depends_on: { llm: { condition: service_healthy } }   # start after llm so the memory split is stable
    volumes: [hf-cache:/root/.cache/huggingface]
    command: ["--model", "mistralai/Voxtral-Mini-4B-Realtime-2602", "--tokenizer-mode", "mistral", "--config-format", "mistral",
              "--load-format", "mistral", "--compilation-config", '{"cudagraph_mode":"PIECEWISE"}', "--max-model-len", "45000",
              "--max-num-batched-tokens", "8192", "--max-num-seqs", "16", "--gpu-memory-utilization", "${ASR_GPU_UTIL}",
              "--served-model-name", "voxtral-realtime", "--host", "0.0.0.0", "--port", "8000"]
    healthcheck: { <<: *health, test: ["CMD", "python3", "-c", "import urllib.request; urllib.request.urlopen('http://127.0.0.1:8000/health', timeout=4)"], start_period: 600s }
  api:
    image: ${DECOSA_REGISTRY}/decosa-api:${DECOSA_TAG}
    restart: unless-stopped
    depends_on: { llm: { condition: service_healthy }, asr: { condition: service_healthy } }
    environment:
      DECOSA_ASR_WS: ws://asr:8000/v1/realtime
      DECOSA_LLM_ROUTE: ${DECOSA_LLM_ROUTE}
      DECOSA_LLM_URL: http://llm:8000/v1
      DECOSA_LLM_MODEL: qwen3.8-27b
      DECOSA_LIVE_CAP: ${DECOSA_LIVE_CAP}
      DECOSA_SESSIONS_PER_IP_HOUR: "1000"
      DECOSA_BUDGET_AUDIO_S: "3600"
      DECOSA_BUDGET_LLM_TOKENS: "200000"
      DECOSA_SESSION_TTL_S: "28800"
      DECOSA_LOCAL_SIGNING: "on"                      # Ed25519 key created at /data/attest/ed25519.pem on first start
      DECOSA_SIGNER_NAME: ${DECOSA_SIGNER_NAME}
      DECOSA_DIARIZE_URL: ""                          # step 4 sets this for speaker labels and uncertainty marks
      DECOSA_CORS_ORIGIN_REGEX: '^https?://(localhost|127\.0\.0\.1)(:\d+)?$$'
    ports: ["127.0.0.1:8445:8445"]
    volumes: [decosa-data:/data]
    healthcheck: { <<: *health, test: ["CMD", "python", "-c", "import urllib.request; urllib.request.urlopen('http://127.0.0.1:8445/healthz', timeout=4)"], start_period: 20s }
volumes: { hf-cache: {}, decosa-data: {} }
```

For transcripts only, delete the `asr` service and the `asr` line under `api.depends_on`. Run `docker compose up -d`, then poll `docker compose ps` until the services are healthy (the LLM takes 5–10 minutes the first time) and `curl -s localhost:8445/healthz` shows `"llm": true` (and `"asr": true` with audio). If `llm` runs out of memory, lower `LLM_GPU_UTIL` or `LLM_MAX_LEN`; the GPU shares must add up to less than about 0.9.

## 4. Optional: speaker labels and uncertainty marks (diarize service)

The `diarize` service (MOSS-Transcribe-Diarize 0.9B, Apache-2.0) has no published image yet. To add it, run `services/diarize` from the decosa-api source on this box (install with the `uv` commands at the top of `services/diarize/requirements.txt`, then `DIARIZE_HOST=172.17.0.1 DIARIZE_PORT=8092 DIARIZE_DEVICE=cuda:0 .venv-diarize/bin/python services/diarize/server.py`; `172.17.0.1` is the docker bridge address from `ip -4 addr show docker0`). Add `extra_hosts: ["host.docker.internal:host-gateway"]` to `api` and set `DECOSA_DIARIZE_URL: http://host.docker.internal:8092`; `/healthz` then shows `"diarize": true`. Lower `LLM_GPU_UTIL` to 0.55 so it fits (an estimate, not measured). If the source isn't available, skip this step.

## 5. The signing key

1. `curl -s localhost:8445/attest/signing-key` returns `{"scheme":"ed25519","pubkey":"<64 hex>",...}`. Show me the `pubkey`.
2. Tell me to **back up the `decosa-data` volume**: it holds the private key (`/data/attest/ed25519.pem`, mode 0600). An appeal months later needs the same key to show the records as issued here.
3. Receipts from this box say `status: "attested"`: signed by our own key. They prove nothing was changed after signing and who signed, not that a score is right.

## 6. Smoke test: score, review, verify, tamper

```bash
H='content-type: application/json'
TOKEN=$(curl -s localhost:8445/demo/session -H "$H" -d '{"vertical":"oral-assessment"}' | jq -r .token)
curl -s localhost:8445/oral/score -H "authorization: Bearer $TOKEN" -H "$H" -d '{"sample_id":"oral-stats-partial","stream":false}' > /tmp/oral.json
jq '{summary: .draft.summary, check: .draft.record_check, leading: [.ready.leading[].n], scores: [.draft.scores[] | {criterion, level, evidence: [.evidence[].n], flags}]}' /tmp/oral.json
jq '{draft: .draft.record, examiner: "Examiner 7", decisions: [.draft.scores[] | if .criterion == "q1-definition"
  then {criterion, action: "override", level: 0, reason: "The student then agreed with the misreading at line 4."}
  else {criterion, action: "confirm"} end]}' /tmp/oral.json > /tmp/review-req.json
curl -s localhost:8445/oral/review -H "authorization: Bearer $TOKEN" -H "$H" --data-binary @/tmp/review-req.json > /tmp/review.json
jq '{final_total, max_total, overrides, check}' /tmp/review.json
jq '{bundle: .bundle}' /tmp/review.json | curl -s localhost:8445/oral/verify -H "$H" --data-binary @- | jq '{ok, summary}'
jq '{bundle: (.bundle | (.review.entries[] | select(.kind=="decision" and .action=="override") | .text) |= "edited later")}' /tmp/review.json \
  | curl -s localhost:8445/oral/verify -H "$H" --data-binary @- | jq '{ok, summary}'
```

Pass if: six scores with evidence line numbers, `record_check.ok` true, line 3 in `leading`; the review prints `check.ok: true` with `overrides: 1`; the first verify prints `ok: true`; the second prints `ok: false` naming the edited decision. Levels can differ by one from the hosted demo's.

With `asr` running, also try the audio path: `curl -sN localhost:8445/demo/replay -H "authorization: Bearer $TOKEN" -H "$H" -d '{"vertical":"oral-assessment","script_id":"oral-stats-partial"}'` streams `transcript` events, the `prompt`, `guard` and `scores` lanes, and a `done` whose `summary.draft` and `summary.record` feed the same review (68 s of audio, paced in real time).

## 7. Point the app at the local API

- Base URL: `http://localhost:8445` (for the web app, `NEXT_PUBLIC_DECOSA_API=http://localhost:8445`). Add the app's origin to `DECOSA_CORS_ORIGINS` if it is not localhost.
- Live: `ws://localhost:8445/ws/live?vertical=oral-assessment&token=<t>&rubric=intro-stats-viva`, 16 kHz mono PCM16 frames of about 100 ms, then `{"type":"stop"}`.
- Your own rubric: `POST /oral/score` with `rubric` (the shape of `GET /oral/rubrics/intro-stats-viva`) and `lines` or `transcript`. Pass `names` to remove the candidate's and examiners' names; for students under 18 set `minor: true` with `consent_basis`, which keeps text out of the signed records.
- Keep the bundle from `/oral/review` with the grade. Anyone can check it: `POST /oral/verify`.
- Keep the API on `127.0.0.1`; put a TLS reverse proxy in front for the LAN and set `DECOSA_TRUSTED_PROXIES`.

Finish with a summary: what is running, the health output, the signing key's pubkey, both verify results, and the consent, FERPA, LL144 and human-decides reminders.