Check your provider serves the model you pay for

# Decosa Endpoint auditor: use the hosted API

You are wiring Decosa's endpoint auditor into this project. It checks whether an OpenAI-compatible endpoint really
serves the model it claims, at the quality it claims, and returns a report signed with the auditor's Ed25519 key.
Use only what is listed below. If you need something else, stop and ask me.

- Base URL: `https://api.decosa.ai`
- Health check: `GET https://api.decosa.ai/healthz`.

## Auth: API key (or a demo session)
1. Preferred: an API key (`dk_…`) from "Get an API key" on the tool page. Keep it in an environment variable,
   `DECOSA_API_KEY`, never in code. Send `Authorization: Bearer $DECOSA_API_KEY`.
2. Without a key: `POST https://api.decosa.ai/demo/session` with `{"vertical": "auditor"}` returns `{"token", "expires_at", "budget"}`.
   a limited number of sessions per network per hour (the current limits are in `demo_sessions` of GET /healthz). Over a limit: HTTP 429 with `Retry-After`.
3. One audit at a time per demo token. An audit needs at least 3,000 generated tokens of budget left (402 otherwise).

## Endpoints (no token needed unless marked)
- `GET /audit/signing-key` → `{"alg": "ed25519", "pubkey", "key_id", "domain", "canonical"}`. Pin this key.
- `GET /audit/targets` → our own demo endpoints `[{id, label, claimed_model, kind, note, expect, available}]`.
- `GET /audit/references` → reference fixtures `[{id, aliases, model, engine, recorded_at, fixture_sha256, fixture_sig, noise}]`.
- `POST /audit/runs` (token) → SSE. Body is either `{"target": "<demo id>"}` or
  `{"base_url": "https://…/v1", "model": "<name on that endpoint>", "claimed_model"?: "<reference alias>", "api_key"?: "<their key>", "context_tokens"?: 0-32000}`.
  - Hosted audits only reach public `https://` endpoints (400 for private, loopback or plain-http URLs).
  - `api_key` is the key for the audited provider. It is held in memory for this run only: never logged, stored or put in the report. The probes bill that provider (about 23-43 short requests plus the context probe).
- `GET /audit/reports/{id}` → a signed report. `GET /audit/reports` lists recent reports for our demo targets only.
- `POST /audit/verify` with `{"report": {...}}` → `{"valid_signature", "signed_by_this_auditor", "auditor_pubkey"}` (a convenience; verify locally too).

## SSE events from `/audit/runs`
- `{"type": "ready", "target", "claimed_model", "reference", "suite": {"id", "sha256", "goldens", "canaries"}}`
- `{"type": "lane", "lane": "identity" | "quality" | "performance", "data": {"checks": [{"id", "title", "status": "pass"|"warn"|"fail"|"skip", "value", "band", "detail"}], "progress"}}`. The latest event per lane is its full state: replace, don't append.
- `{"type": "receipt", ...}` for each probe that went through the Decosa API (hosted target only).
- `{"type": "lane", "lane": "verdict", "data": {"status", "reasons", "report", "report_url"}}`, then `{"type": "done", "summary": {"verdict", "report", "report_url"}}`, then a `budget` event.
- `{"type": "error", "message"}` if the run stops.

## Verdicts
`pass` (inside the reference stack's measured spread), `drift` (same family, different numerics or quality),
`fail` (not the claimed weights), `inconclusive` (too many probe errors, no reference, or a re-check that disagreed).
Drift and fail are only signed after a second pass agrees.

## Verify a report (Python, `pip install cryptography`)
```python
import json, urllib.request
from cryptography.hazmat.primitives.asymmetric.ed25519 import Ed25519PublicKey
API = "https://api.decosa.ai"
report = json.load(urllib.request.urlopen(f"{API}/audit/reports/<id>"))
pub = json.load(urllib.request.urlopen(f"{API}/audit/signing-key"))["pubkey"]
assert report["signer"]["pubkey"] == pub
body = {k: v for k, v in report.items() if k != "signature"}
msg = b"decosa.audit.report.v1\n" + json.dumps(body, sort_keys=True, separators=(",", ":"), ensure_ascii=False).encode()
Ed25519PublicKey.from_public_bytes(bytes.fromhex(pub)).verify(bytes.fromhex(report["signature"]), msg)
```
In a browser: rebuild the same bytes with a sorted-keys `JSON.stringify` and check them with WebCrypto `Ed25519`.

## Example: run an audit and print the verdict (Python, `pip install httpx`)
```python
import httpx, json, os
API = "https://api.decosa.ai"
H = {"Authorization": f"Bearer {os.environ['DECOSA_API_KEY']}"}
body = {"base_url": "https://api.example.com/v1", "model": "qwen3.8-27b", "api_key": os.environ["PROVIDER_KEY"], "context_tokens": 4000}
with httpx.stream("POST", f"{API}/audit/runs", json=body, headers=H, timeout=600) as r:
    if r.status_code != 200:
        r.read()
        raise SystemExit(f"{r.status_code}: {r.json().get('error') or r.json()}")   # e.g. a private URL or a host that does not resolve
    for line in r.iter_lines():
        if line.startswith("data: "):
            ev = json.loads(line[6:])
            if ev["type"] == "done":
                print(ev["summary"]["verdict"], ev["summary"]["report_url"])
```

On request. The container images and the compose file aren’t public yet. Ask for self-host access and Decosa sends the registry (DECOSA_REGISTRY) and the compose file’s URL (DECOSA_COMPOSE_URL) these steps use. They are the steps we tested end to end on a fresh machine.

# Decosa Endpoint auditor: run it yourself (containers)

You are setting up the Decosa endpoint auditor to run on this machine, so the API keys of the endpoints I audit, and
the endpoints themselves, never leave my network. It needs no GPU. Nothing is sent to Decosa's hosted API.

Status: the container images (${DECOSA_REGISTRY}/decosa-*) and the compose file are on request while self-host is in early access (not on a public registry yet): ask at https://decosa.ai/contact?topic=self-host, and Decosa sends the registry as DECOSA_REGISTRY, the compose file URL as DECOSA_COMPOSE_URL, and pull access. If a pull fails with
"not found", "denied" or "unauthorized", stop and tell me. Do not substitute other images.

Ask me before any command that needs sudo, and show me the command first.

## Step 0: set up with a coding agent, rehearse on mock data, then go private

This prompt is for a coding agent running on the machine that will host the service. We recommend Claude Code with
Claude Opus 5.5; any capable coding agent works. Work in this order:

1. Set up on mock data only. During the whole setup you (the agent) work with the synthetic sample bundle below and
   nothing else. Do not ask me for real data, and do not open, read, list or copy files that hold real data, even to
   "test with something realistic".
2. Rehearse. When the steps below are done and the service is healthy, fetch the mock-data bundle for this tool,
   https://decosa.ai/samples/auditor.zip (16 KB, 10 checks, synthetic or openly licensed: see `licence` in expected.json),
   show me what is in it, and run the rehearsal against the local API:
   `docker compose exec api python scripts/rehearse.py auditor` (the api image carries the same bundle under /app/rehearsal/auditor/;
   with no key set, the script asks the local API for a short demo token). From a decosa-api checkout instead:
   `python scripts/rehearse.py auditor --bundle auditor.zip --base-url http://127.0.0.1:<PORT>`.
   It sends the mock inputs to the local API and prints PASS or FAIL for each expected property (for example: "at least one audit target is up on this server", "the audit finishes without errors", "the audit compares against the shipped reference fixture"). Show me
   the full output. Every check must pass. If one fails, fix the install and run it again; never edit `expected.json`
   to make a check pass.
3. Stop there. Once the rehearsal passes, tell me, and I will run my own data against the local API myself, on this
   machine.

For the person running this: a coding agent that runs in the cloud sees everything in its context, including files it
reads, command output and anything pasted into the chat. Keep real data out of the chat and out of anything the agent
can read. Switch to your own data only after the rehearsal has passed and the agent's work is done.

## Steps
1. Docker: if `docker compose version` fails, install Docker Engine and the compose plugin using Docker's official
   instructions for this distribution (docs.docker.com/engine/install).
2. Fetch the compose file:
   `mkdir -p ~/decosa && cd ~/decosa && curl -fsSL "${DECOSA_COMPOSE_URL}" -o compose.yaml`
   Read it. Set these for the `api` service: `DECOSA_AUDITOR_KEY=/data/keys/auditor_ed25519.key`,
   `DECOSA_AUDITOR_CREATE_KEY=1` (the first start creates the key, mode 0600) and `DECOSA_AUDIT_ALLOW_PRIVATE=1`
   (so it may audit endpoints on this network). Keep `/data` on the named volume the compose file declares: the key
   is created inside it (a host `./keys` folder is not writable by the api's user, uid 10001). Back it up with
   `docker compose cp api:/data/keys ./auditor-keys-backup` and never commit it.
3. Pull and start: `docker compose pull && docker compose up -d`.
4. Check: `curl -fsS http://localhost:<PORT>/audit/signing-key` returns the auditor's public key. Show it to me: it is what
   others pin to verify my reports. `GET /audit/references` lists the signed reference fixtures.
5. Smoke test: get a token with `POST /demo/session {"vertical":"auditor"}`, then
   `POST /audit/runs` with `{"base_url": "<endpoint>/v1", "model": "<name>", "claimed_model": "qwen3.8-27b"}` and read the
   SSE stream until `done`. Pass a provider key as `api_key` from an environment variable; never print it.
6. Report back: the public key and key id, the references listed, and the smoke-test verdict and report id.

No NVIDIA GPU? This tool also runs entirely on an Apple Silicon Mac (MLX, 16 GB of unified memory or more): use https://decosa.ai/prompts/auditor-mac.md instead.