Check if an open model can take over your prompt

# Decosa open-model migration check: use the hosted API

You are wiring Decosa's migration check into this project. It answers "can an open model take over this prompt?": it
runs a production prompt template on an open model (Qwen3.8-27B) over examples we already logged from our closed model,
scores each output against the one we logged, and returns agreement with a 95% interval, the failure clusters, latency,
cost per 1,000 requests and a go / no-go, sealed in a signed record. It never calls a closed API and never needs our
closed-model key: we send outputs we already have. Use only what is listed below. If you need something else, stop and ask me.

- Base URL: `https://api.decosa.ai`
- Health check: `GET https://api.decosa.ai/healthz`.
- Agreement with our current model is not correctness. Where we have human labels, send them too.

## Auth: API key (or a demo session)
1. Preferred: an API key (`dk_…`) from "Get an API key" on the tool page. Keep it in an environment variable,
   `DECOSA_API_KEY`, never in code. Send `Authorization: Bearer $DECOSA_API_KEY`.
2. Without a key: `POST https://api.decosa.ai/demo/session` with `{"vertical": "migration-check"}` returns `{"token", "expires_at", "budget"}`.
   a limited number of sessions per network per hour (the current limits are in `demo_sessions` of GET /healthz); each session has a token allowance (its `budget`); up to 40 examples per run (200 with a key).
3. The server estimates the generated tokens a run needs and answers 402 if the budget is short. One run at a time per token.

## The run
`POST /migration/runs` (token). Body:
```json
{
  "title": "Ticket triage",
  "task": {"type": "json", "schema": {"type": "object", "required": ["category"], "properties": {"category": {"enum": ["billing", "other"]}}}},
  "prompt": {"system": "You triage tickets...", "template": "Ticket:\n{{input}}"},
  "examples": [{"id": "t1", "input": "I was charged twice", "output": "{\"category\": \"billing\"}", "label": null}],
  "incumbent": {"model": "gpt-4o-mini"},
  "thresholds": {"pass_rate": 0.9, "schema_valid": 0.98},
  "max_output_tokens": 600
}
```
- `task.type`: `exact` (same answer after normalising whitespace, case, quotes, code fences, a trailing full stop),
  `label` (`labels` list, `extract`: `whole`, `first`, `last` or `double_bracket` for `[[X]]`), `json` (`schema`: a JSON
  Schema subset, `fields`: dotted paths to compare; default the schema's top-level properties), `freetext` (`rubric`:
  what matters, sent to the judge).
- `examples`: 10 to 200. `input` is a string (fills `{{input}}`) or an object of named fields for named placeholders.
  `output` is what our current model returned. Optional per example: `label` (the right answer), `latency_ms`,
  `prompt_tokens`, `completion_tokens` from our logs (then cost and latency use them instead of estimates).
- `incumbent.model`: matched against the dated price table (`GET /migration/info`); or send
  `incumbent.price: {"input_per_m": 0.15, "output_per_m": 0.60}` in dollars per million tokens.
- Verdict: `go` when the whole 95% interval of the pass rate clears `pass_rate`; `no-go` when it sits wholly below;
  `inconclusive` otherwise, or when more than 10% of examples could not be scored. Failed calls count as fails.
- JSON by default: `{report_id, headline, recommended, verdicts, candidates: {<id>: summary}, identity, examples, receipts, report, budget}`.
  Each summary: `pass {k, n, rate, ci95}`, `clusters [{cluster, count, share, examples}]`, `schema_valid`, `fields`,
  `labels {current, candidate}`, `confusion`, `judge_outcomes`, `latency_ms`, `cost.per_1k`, `verdict`, `reasons`,
  `stability.bootstrap_same_verdict`. Numbers are strings; there are no floats in a signed record.
- With `Accept: text/event-stream` (or `"stream": true`) it streams `ready`, then `receipt` and `example` events as each
  example is scored (not in input order), `identity`, `summary`, `record`, `report`, `budget`, `done`.

Other endpoints (no token): `GET /migration/info` (tasks, limits, the judge prompt and its sha256, the price table),
`GET /migration/samples`, `POST /migration/verify` `{"record": {...}}` → `{ok, record: {...}, recompute: {ok, diffs}}`,
`POST /record/verify`, `GET /attest/signing-key`, `GET /receipts/{id}`.

## Example: check a label prompt from our logs (Python, `pip install httpx`)
```python
import httpx, json, os
API = "https://api.decosa.ai"
H = {"Authorization": f"Bearer {os.environ['DECOSA_API_KEY']}"}
rows = [json.loads(l) for l in open("logs/intent-router.jsonl")][:200]
body = {"task": {"type": "label", "labels": ["billing", "tech", "sales", "other"], "extract": "whole"},
        "prompt": {"system": open("prompts/router.txt").read(), "template": "{{input}}"},
        "examples": [{"id": r["id"], "input": r["message"], "output": r["model_output"], "label": r.get("agent_label")} for r in rows],
        "incumbent": {"model": "gpt-4.1-mini"}, "thresholds": {"pass_rate": 0.95}, "max_output_tokens": 20}
r = httpx.post(f"{API}/migration/runs", json=body, headers=H, timeout=900)
r.raise_for_status()
rep = r.json()
print(rep["headline"])
for c in rep["candidates"].values():
    print(c["verdict"], c["pass"], c["cost"]["per_1k"])
    for cl in c["clusters"][:5]:
        print("  ", cl["count"], cl["cluster"], cl["examples"])
json.dump(rep["report"]["record"], open("migration-record.json", "w"))
```

## Keep and verify the record
`report.record` is a `decosa.record.v1` hash chain signed with the server's Ed25519 key: the prompt, every rendered input,
both outputs, every score and judgment (each model call with its gateway receipt) and the numbers. Strip the `text` field
from every entry to share it without our data; it still verifies. `POST https://api.decosa.ai/migration/verify` checks the chain,
the signature and the receipts, and recomputes every stated number from the per-example entries.

## Honest limits
- The free-text judge is a language model. Its agreement with human experts on MT-Bench is on the Stack tab; it is
  close to GPT-4's own, not perfect.
- Latency is measured on a shared GPU through the gateway; our own deployment will differ.
- Closed-model cost uses list prices checked on the date shown and estimated token counts unless we send ours.

On request. The container images and the compose file aren’t public yet. Ask for self-host access and Decosa sends the registry (DECOSA_REGISTRY) and the compose file’s URL (DECOSA_COMPOSE_URL) these steps use. They are the steps we tested end to end on a fresh machine.

# Decosa open-model migration check: run it yourself (containers)

You are setting up the Decosa migration check on this machine, so our logged production prompts and outputs never leave
it. It runs our prompt on an open model, scores it against the outputs we logged from our closed model, and signs a
go / no-go record. Nothing is sent to Decosa's hosted API, and no closed API is called.

Status: the container images (${DECOSA_REGISTRY}/decosa-*) and the compose file are on request while self-host is in early access (not on a public registry yet): ask at https://decosa.ai/contact?topic=self-host, and Decosa sends the registry as DECOSA_REGISTRY, the compose file URL as DECOSA_COMPOSE_URL, and pull access. If a pull fails with
"not found", "denied" or "unauthorized", stop and tell me. Do not substitute other images.

Ask me before any command that needs sudo, and show me the command first.

## Step 0: set up with a coding agent, rehearse on mock data, then go private

This prompt is for a coding agent running on the machine that will host the service. We recommend Claude Code with
Claude Opus 5.5; any capable coding agent works. Work in this order:

1. Set up on mock data only. During the whole setup you (the agent) work with the synthetic sample bundle below and
   nothing else. Do not ask me for real data, and do not open, read, list or copy files that hold real data, even to
   "test with something realistic".
2. Rehearse. When the steps below are done and the service is healthy, fetch the mock-data bundle for this tool,
   https://decosa.ai/samples/migration-check.zip (3 KB, 10 checks, synthetic or openly licensed: see `licence` in expected.json),
   show me what is in it, and run the rehearsal against the local API:
   `docker compose exec api python scripts/rehearse.py migration-check` (the api image carries the same bundle under /app/rehearsal/migration-check/;
   with no key set, the script asks the local API for a short demo token). From a decosa-api checkout instead:
   `python scripts/rehearse.py migration-check --bundle migration-check.zip --base-url http://127.0.0.1:<PORT>`.
   It sends the mock inputs to the local API and prints PASS or FAIL for each expected property (for example: "all 10 examples were scored", "at least 9 of 10 outputs parse against the schema", "no model call failed"). Show me
   the full output. Every check must pass. If one fails, fix the install and run it again; never edit `expected.json`
   to make a check pass.
3. Stop there. Once the rehearsal passes, tell me, and I will run my own data against the local API myself, on this
   machine.

For the person running this: a coding agent that runs in the cloud sees everything in its context, including files it
reads, command output and anything pasted into the chat. Keep real data out of the chat and out of anything the agent
can read. Switch to your own data only after the rehearsal has passed and the agent's work is done.

## Steps
1. Docker: if `docker compose version` fails, install Docker Engine and the compose plugin using Docker's official
   instructions for this distribution (docs.docker.com/engine/install). Also install the NVIDIA container toolkit and
   check `docker run --rm --gpus all nvidia/cuda:12.8.0-base-ubuntu24.04 nvidia-smi`.
2. Fetch the compose file:
   `mkdir -p ~/decosa && cd ~/decosa && curl -fsSL "${DECOSA_COMPOSE_URL}" -o compose.yaml`
   Read it. Keep the `llm` service (Qwen3.8-27B on vLLM) and the `api` service. For the `api` service set
   `DECOSA_LLM_ROUTE=direct`, `DECOSA_LLM_URL=http://llm:8000/v1`, `DECOSA_LLM_MODEL=qwen3.8-27b` and bind every port to
   127.0.0.1. To compare more open models, run them as extra OpenAI-compatible services and list them in
   `DECOSA_MIGRATION_CANDIDATES` (a JSON list of `{"id", "label", "base_url", "model", "license"}`); requests then pick
   them with `"candidates": [...]`.
3. Pull and start: `docker compose pull && docker compose up -d`. Wait for the `llm` health check (the first start
   downloads about 20 GB of weights).
4. Check: `curl -fsS http://127.0.0.1:<PORT>/migration/info` lists the task types, the candidates and the judge;
   `GET /attest/signing-key` shows this box's public key. Show me the key: it is what others pin to verify our records.
5. Smoke test: get a token with `POST /demo/session {"vertical":"migration-check"}`, fetch `GET /migration/samples`, and
   send the `tickets-json` sample's `request` to `POST /migration/runs`. Expect 20 examples scored, `schema_valid` 20 of
   20, a few `field priority differs` clusters and a verdict. Then `POST /migration/verify` with `report.record`: `ok`,
   `record.ok` and `recompute.ok` must all be true.
6. Report back: the public key and key id, the smoke-test verdict and clusters, and how long the run took.

Off by default. Joining as a provider serves other people's requests on this GPU; never do it on a box that holds
production logs. If I ask for it later, follow the Provide page instead of improvising.

No NVIDIA GPU? This tool also runs entirely on an Apple Silicon Mac (MLX, 32 GB of unified memory or more): use https://decosa.ai/prompts/migration-check-mac.md instead.