Build a model-risk evidence pack

# Decosa model-risk evidence pack: use the hosted API

You are wiring Decosa's model-risk evidence pack into this project. It re-runs a fixed test suite against an LLM system
we rely on (the hosted demo systems, or our own OpenAI-compatible endpoint): identity prompts, graded cases, stability
against a validated run, counterfactual fairness pairs. Flagged lanes are re-run before any alert. It returns a signed
pack, a Markdown binder and a hash-chained record our validators can recompute. Use only what is listed below. If you
need something else, stop and ask me.

- Base URL: `https://api.decosa.ai`
- Health check: `GET https://api.decosa.ai/healthz`.
- The pack is evidence, not a validation opinion and not legal advice. It detects only what its suite probes.
- The hosted demo is for synthetic cases. Real complaints or applications belong on a self-hosted box.

## Auth: API key (or a demo session)
1. Preferred: an API key (`dk_…`) from "Get an API key" on the tool page. Keep it in `DECOSA_API_KEY`, never in
   code. Send `Authorization: Bearer $DECOSA_API_KEY`.
2. Without a key: `POST https://api.decosa.ai/demo/session` with `{"vertical": "model-risk-pack"}` returns `{"token", "expires_at", "budget"}`.
   20,000 generated tokens per session: about three full packs.
3. A run needs at least 4,500 generated tokens left (402 otherwise). One run at a time per token.

## The run
`POST /mrm/runs` (token). Body, one of:
```json
{"target": "vendor-biased"}
{"endpoint": {"base_url": "https://api.example.com/v1", "model": "our-model", "api_key": "…", "black_box": true},
 "suite": "fernhill-v1", "baseline": { "…the first signed pack…": "" }, "history": [], "system": {"owner": "Model risk", "version": "2026-Q3"}}
```
- `target`: a demo system from `GET /mrm/targets` (`lender`, `vendor`, and the vendor after injected changes). Demo
  targets use their stored, signed baseline.
- `endpoint`: public https only; the key is used for this run and never logged or stored. `black_box: true` sends only
  the case text (the endpoint applies its own prompt); otherwise the suite's prompts are sent as the system message.
- `suite`: `fernhill-v1` (synthetic, `GET /mrm/suites/fernhill-v1`) or our own suite object in the same schema.
- `baseline`: our first pack for this system and suite. Without it the pack records the validated behaviour.
- JSON by default: `{pack, record, markdown, cost_usd, budget}`. With `Accept: text/event-stream` (or `"stream": true`):
  `ready`, receipt and `probe` events as each call lands, `lane` events, `recheck` (if a lane was flagged), `verdict`,
  `pack`, `record`, `markdown`, `budget`, `done`.
- `pack.verdict.status`: `pass`, `watch` (flagged once, not reproduced), `alert` (flagged and reproduced) or
  `inconclusive` (too many failed calls). `pack.flags` says which lane and why; the rules are `pack.policy`.

Other endpoints (no token): `GET /mrm/info`, `POST /mrm/verify` `{pack, record, suite?, baseline?}` →
`{ok, pack, record, recompute}`, `POST /mrm/render` `{pack}` → the Markdown binder, `POST /mrm/trend` `{packs}`,
`GET /attest/signing-key`, `GET /receipts/{id}`.

## Example: a weekly check of our vendor endpoint (Python, `pip install httpx`)
```python
import httpx, json, os, pathlib
API = "https://api.decosa.ai"
H = {"Authorization": f"Bearer {os.environ['DECOSA_API_KEY']}"}
d = pathlib.Path("mrm-packs"); d.mkdir(exist_ok=True)
body = {"endpoint": {"base_url": os.environ["VENDOR_URL"], "model": "triage-v3", "api_key": os.environ["VENDOR_KEY"], "black_box": True},
        "suite": json.load(open("our-suite.json"))}
if (d / "baseline.json").exists():
    body["baseline"] = json.load(open(d / "baseline.json"))
r = httpx.post(f"{API}/mrm/runs", json=body, headers=H, timeout=900)
r.raise_for_status()
res = r.json()
pack = res["pack"]
if "baseline" not in body:
    json.dump(pack, open(d / "baseline.json", "w"))
json.dump(pack, open(d / f"{pack['id']}.json", "w")); json.dump(res["record"], open(d / f"{pack['id']}.record.json", "w"))
open(d / f"{pack['id']}.md", "w").write(res["markdown"])
print(pack["verdict"]["status"], pack["verdict"]["reasons"])
```

## Keep and verify
Keep the pack, the record and the suite together. `POST https://api.decosa.ai/mrm/verify` checks the pack signature and every
record entry (text hashes, links, embedded receipts), then re-parses and re-grades every output from the record's texts
and recomputes each number and the verdict. Pin the server's key from `/attest/signing-key`.

## Honest limits
- A problem the suite does not probe (another product, another attribute, a ZIP code it does not use) is not detected.
- Drafts are graded by an open model (typed judgments); the grade is repeatable, not infallible.
- For a black box, identity cannot tell a model swap from a prompt or settings change; it shows that behaviour changed.
- Latency is measured on a shared GPU through the gateway.

On request. The container images and the compose file aren’t public yet. Ask for self-host access and Decosa sends the registry (DECOSA_REGISTRY) and the compose file’s URL (DECOSA_COMPOSE_URL) these steps use. They are the steps we tested end to end on a fresh machine.

# Decosa model-risk evidence pack: run it yourself (containers)

You are setting up the Decosa model-risk evidence pack on this machine, so our test cases and the systems we test stay
inside our network. It re-runs a signed test suite against an LLM system we use, re-checks anything it flags, and signs
a pack and a hash-chained record with this box's key. Nothing is sent to Decosa's hosted API.

Status: the container images (${DECOSA_REGISTRY}/decosa-*) and the compose file are on request while self-host is in early access (not on a public registry yet): ask at https://decosa.ai/contact?topic=self-host, and Decosa sends the registry as DECOSA_REGISTRY, the compose file URL as DECOSA_COMPOSE_URL, and pull access. If a pull fails with
"not found", "denied" or "unauthorized", stop and tell me. Do not substitute other images.

Ask me before any command that needs sudo, and show me the command first.

## Step 0: set up with a coding agent, rehearse on mock data, then go private

This prompt is for a coding agent running on the machine that will host the service. We recommend Claude Code with
Claude Opus 5.5; any capable coding agent works. Work in this order:

1. Set up on mock data only. During the whole setup you (the agent) work with the synthetic sample bundle below and
   nothing else. Do not ask me for real data, and do not open, read, list or copy files that hold real data, even to
   "test with something realistic".
2. Rehearse. When the steps below are done and the service is healthy, fetch the mock-data bundle for this tool,
   https://decosa.ai/samples/model-risk-pack.zip (3 KB, 10 checks, synthetic or openly licensed: see `licence` in expected.json),
   show me what is in it, and run the rehearsal against the local API:
   `docker compose exec api python scripts/rehearse.py model-risk-pack` (the api image carries the same bundle under /app/rehearsal/model-risk-pack/;
   with no key set, the script asks the local API for a short demo token). From a decosa-api checkout instead:
   `python scripts/rehearse.py model-risk-pack --bundle model-risk-pack.zip --base-url http://127.0.0.1:<PORT>`.
   It sends the mock inputs to the local API and prints PASS or FAIL for each expected property (for example: "the run finishes without errors", "all 14 calls were made (10 identity probes and 4 cases)", "no call failed"). Show me
   the full output. Every check must pass. If one fails, fix the install and run it again; never edit `expected.json`
   to make a check pass.
3. Stop there. Once the rehearsal passes, tell me, and I will run my own data against the local API myself, on this
   machine.

For the person running this: a coding agent that runs in the cloud sees everything in its context, including files it
reads, command output and anything pasted into the chat. Keep real data out of the chat and out of anything the agent
can read. Switch to your own data only after the rehearsal has passed and the agent's work is done.

## Steps
1. Docker: if `docker compose version` fails, install Docker Engine and the compose plugin using Docker's official
   instructions (docs.docker.com/engine/install), plus the NVIDIA container toolkit; check
   `docker run --rm --gpus all nvidia/cuda:12.8.0-base-ubuntu24.04 nvidia-smi`.
2. Fetch the compose file:
   `mkdir -p ~/decosa && cd ~/decosa && curl -fsSL "${DECOSA_COMPOSE_URL}" -o compose.yaml`
   Keep the `llm` service (Qwen3.8-27B on vLLM: the grader) and the `api` service. For `api` set
   `DECOSA_LLM_ROUTE=direct`, `DECOSA_LLM_URL=http://llm:8000/v1`, `DECOSA_LLM_MODEL=qwen3.8-27b`,
   `DECOSA_MRM_ALLOW_PRIVATE=1` (so we can test endpoints on our own network) and bind every port to 127.0.0.1.
3. Pull and start: `docker compose pull && docker compose up -d`. Wait for the `llm` health check (the first start
   downloads about 20 GB of weights).
4. Check: `curl -fsS http://127.0.0.1:<PORT>/mrm/info` lists the lanes, the policy and the grader;
   `GET /attest/signing-key` shows this box's public key. Show me the key: validators pin it.
5. Smoke test: get a token with `POST /demo/session {"vertical":"model-risk-pack"}`, then `POST /mrm/runs {"target": "lender"}`.
   Expect about 80 calls, triage 14-15 of 15, drafts 4-5 of 5 and a verdict. Then `POST /mrm/verify` with the pack
   and the record: `ok`, `record.ok` and `recompute.ok` must all be true.
6. Our own system: `POST /mrm/runs {"endpoint": {"base_url": "http://…/v1", "model": "…", "black_box": true}, "suite": <our suite>}`.
   Keep the first pack as the baseline and send it as `baseline` every time after. Schedule `scripts/mrm_monitor.py`
   weekly (cron or a systemd timer); it exits 2 on an alert.
7. Report back: the public key and key id, the smoke-test verdict, and how long the run took.

Off by default. Never on a box that holds confidential test cases. If I ask for it later, follow the Provide page.

No NVIDIA GPU? This tool also runs entirely on an Apple Silicon Mac (MLX, 32 GB of unified memory or more): use https://decosa.ai/prompts/model-risk-pack-mac.md instead.