Run an end-to-end browser test

# Decosa verified test runs: add certified end-to-end tests to this project (hosted API)

You are adding Decosa verified test runs to this project. A test is a YAML spec of plain-language steps, each with
assertions. Decosa's browser agent carries out the steps; pass or fail comes only from the assertions, checked in code
on the live page. Every run returns a signed run certificate to keep with the release. Use only what is listed below.
If you need something else, stop and ask me.

- Base URL: `https://api.decosa.ai`
- What a run records and proves, the assertion kinds and the limits: `GET https://api.decosa.ai/testruns/info`.

## Auth: API key (or a demo session)
1. Preferred: an API key (`dk_…`) from "Get an API key" on the tool page, enabled for `test-runs`. Keep it in
   `DECOSA_API_KEY` (a CI secret), never in code. Send `Authorization: Bearer $DECOSA_API_KEY`.
2. Without a key: `POST https://api.decosa.ai/demo/session` with `{"vertical": "test-runs"}` returns `{"token", ...}`. A demo
   session can test only the built-in targets (`fixture`, `saucedemo`), not our own site.

## Try it first (no setup)
```bash
curl -s https://api.decosa.ai/testruns/samples | python3 -c "import json,sys; print(json.load(sys.stdin)['samples'][2]['spec'])" > promo.yml
curl -sN https://api.decosa.ai/testruns/runs -H "Authorization: Bearer $DECOSA_API_KEY" -H 'Content-Type: application/json' \
  -d "$(python3 -c "import json; print(json.dumps({'spec': open('promo.yml').read(), 'target': 'fixture', 'build': 'promo-wrong'}))")" \
  | python3 -c "import json,sys; r=json.load(sys.stdin); print(r['summary']); json.dump(r['certificate'], open('certificate.json','w'))"
```
Expect `FAIL: 2 of 3 steps passed. step 3: [data-test=discount] is "-$2.40" (read "-$4.80")`: that build has a seeded
bug. With `"build": "good"` it passes. Then `POST https://api.decosa.ai/testruns/verify` with `{"certificate": ...}` returns
`ok: true`; change any value in the file and it returns `ok: false` naming the entry.

## Our own site: prove we own the domain (once)
1. `POST /testruns/domains` `{"domain": "staging.example.com"}` → a token and two ways to publish it.
2. Either a DNS TXT record `_decosa-challenge.<domain>` = the token (covers the domain and its subdomains), or serve the
   token as the body of `https://<host>/.well-known/decosa-verify.txt` (covers that exact host). Ask me which.
3. `POST /testruns/domains/check` `{"domain": "..."}` → `verified: true` for 30 days. Runs against an unverified host
   are refused with 403. Only public https hosts on port 443.

## Write the specs
Put them in `tests/e2e/*.yml`. Format:
```yaml
name: Checkout total
start: /login                      # path on the target
approve: []                        # buttons the agent may press although they look final (Pay, Finish, Delete ...)
steps:
  - do: Log in as 'qa@example.com' with password '{{TEST_PASSWORD}}'.
    expect:
      - url_contains: /dashboard
  - goto: /cart                    # plain navigation, no model
    expect:
      - element_count: {selector: "[data-test=cart-line]", equals: 1}
      - value_equals: {selector: "[data-test=total]", equals: "$23.00"}
```
- Every step needs `expect`. Kinds: `text_present`, `text_absent` (`{text, selector?, ignore_case?}` or a string),
  `url_contains`, `url_equals`, `title_contains`, `element_count` (`{selector, equals | at_least | at_most}`),
  `value_equals`, `value_contains` (`{selector, equals | contains}`). Prefer `data-test` selectors.
- A value the agent types must be in quotes in the step. Passwords: `'{{NAME}}'` in the spec, the value in `secrets`
  (the model types the placeholder, the browser fills in the secret; neither the model nor the certificate sees it).
- One job per step, 1-6 actions. A step ends when its assertions pass after an action, when the agent says done, or
  after `max_actions` (default 6). The first failed step ends the run; later steps are marked not run.
- Check a spec without running it: `POST /testruns/spec/check` `{"spec": "<yaml>"}` (no key needed).
- Use test accounts and test data only.

## Run it in CI
`curl -fsSO https://api.decosa.ai/testruns/sdk/decosa_testrun.py` (standard library only), then
`python3 decosa_testrun.py tests/e2e/checkout.yml --target https://staging.example.com --build-id "$GITHUB_SHA" --secret TEST_PASSWORD --out evidence/`
with `DECOSA_API=https://api.decosa.ai` and `DECOSA_API_KEY` set. Exit 0 = pass and verified, 1 = an assertion failed, 2 = error
or refused. Upload `evidence/` as a build artifact (keep it as long as your audit window). GitHub Actions:
```yaml
      - run: |
          curl -fsSO "$DECOSA_API/testruns/sdk/decosa_testrun.py"
          python3 decosa_testrun.py tests/e2e/checkout.yml --target https://staging.example.com --build-id "$GITHUB_SHA" --secret TEST_PASSWORD --out evidence/
        env: { DECOSA_API: "https://api.decosa.ai", DECOSA_API_KEY: "${{ secrets.DECOSA_API_KEY }}", TEST_PASSWORD: "${{ secrets.STAGING_TEST_PASSWORD }}" }
      - uses: actions/upload-artifact@v4
        if: always()
        with: { name: run-certificate, path: evidence/ }
```
`--repeat 3` runs it three times and writes a signed flake report (steps that both passed and failed).

## Endpoints
- `POST /testruns/runs` (token) `{spec, target?, build?, build_id?, secrets?, repeat?}` → with `Accept: text/event-stream`
  an SSE stream (ready, entry, step, receipt, certificate, flake, done); otherwise JSON when the run ends:
  `{verdict, run_id, summary, certificate, check}` (with `repeat` > 1: `certificates`, `flake`, `report`).
- `GET /testruns/runs/{id}` (the same token) → the certificate, for 24 hours.
- `POST /testruns/verify` `{certificate}` or `{report}` (no token) → `{ok, summary, checks, bad, test_steps, verdict}`.
- 429 with `Retry-After` when the browsers are busy; 402 when the key's daily model budget is used up.

## Rules
- Never treat the model's "done" as a pass; only the verdict counts, and only a certificate that verifies.
- Never test a site we do not own or have written permission to test.
- Show me the failing step, its expected value and the value read from the page when a run fails.

On request. The container images and the compose file aren’t public yet. Ask for self-host access and Decosa sends the registry (DECOSA_REGISTRY) and the compose file’s URL (DECOSA_COMPOSE_URL) these steps use. They are the steps we tested end to end on a fresh machine.

# Decosa verified test runs: run it yourself (containers)

You are setting up Decosa's verified test runs on this machine, so the staging site, test accounts, screenshots and
certificates stay here. The runner (decosa-api with a headless Chromium) needs no GPU; Qwen3.8-27B chooses the
actions. Nothing is sent to Decosa's hosted API.

Status: the container images (${DECOSA_REGISTRY}/decosa-*) and the compose file are on request while self-host is in early access (not on a public registry yet): ask at https://decosa.ai/contact?topic=self-host, and Decosa sends the registry as DECOSA_REGISTRY, the compose file URL as DECOSA_COMPOSE_URL, and pull access. If a pull fails with
"not found", "denied" or "unauthorized", build decosa-api from source with
`docker build -f docker/api/Dockerfile --build-arg WITH_BROWSER=1 -t decosa-api:local .` (the browser is required),
and tell me. Do not substitute other images.

Hardware: any Linux x86_64 machine for the runner (2 GB RAM per concurrent browser); 1x RTX PRO 6000 96 GB (or a
32 GB card) for Qwen3.8-27B. Ask me before any command that needs sudo, and show me the command first.

## Step 0: set up with a coding agent, rehearse on mock data, then go private

This prompt is for a coding agent running on the machine that will host the service. We recommend Claude Code with
Claude Opus 5.5; any capable coding agent works. Work in this order:

1. Set up on mock data only. During the whole setup you (the agent) work with the synthetic sample bundle below and
   nothing else. Do not ask me for real data, and do not open, read, list or copy files that hold real data, even to
   "test with something realistic".
2. Rehearse. When the steps below are done and the service is healthy, fetch the mock-data bundle for this tool,
   https://decosa.ai/samples/test-runs.zip (2 KB, 8 checks, synthetic or openly licensed: see `licence` in expected.json),
   show me what is in it, and run the rehearsal against the local API:
   `docker compose exec api python scripts/rehearse.py test-runs` (the api image carries the same bundle under /app/rehearsal/test-runs/;
   with no key set, the script asks the local API for a short demo token). From a decosa-api checkout instead:
   `python scripts/rehearse.py test-runs --bundle test-runs.zip --base-url http://127.0.0.1:<PORT>`.
   It sends the mock inputs to the local API and prints PASS or FAIL for each expected property (for example: "the spec reads as valid, with 2 steps", "the good build passes", "every assertion on the good build passes"). Show me
   the full output. Every check must pass. If one fails, fix the install and run it again; never edit `expected.json`
   to make a check pass.
3. Stop there. Once the rehearsal passes, tell me, and I will run my own data against the local API myself, on this
   machine.

For the person running this: a coding agent that runs in the cloud sees everything in its context, including files it
reads, command output and anything pasted into the chat. Keep real data out of the chat and out of anything the agent
can read. Switch to your own data only after the rehearsal has passed and the agent's work is done.

## Rules you must keep
- Bind every port to 127.0.0.1. CI reaches it over a VPN or an authenticated proxy, never an open port.
- Only test sites I own or may test, with test accounts. List them in `DECOSA_TESTRUNS_TARGETS` (host names); add
  `DECOSA_TESTRUNS_ALLOW_PRIVATE=1` only if they are on a private network. Everything else is refused unless its domain
  is verified (DNS TXT or /.well-known file), exactly like the hosted service.
- The signing key is created on first start in the data volume (`attest/ed25519.pem`, 0600). Back it up; never print
  it; keep it away from the people whose releases the certificates vouch for.
- Passwords go in the run's `secrets`, never in a spec.

## Steps
1. Docker (and the NVIDIA Container Toolkit for the model): if `docker compose version` fails, install it from the
   official Docker instructions.
2. Fetch the compose file: `mkdir -p ~/decosa && cd ~/decosa && curl -fsSL "${DECOSA_COMPOSE_URL}" -o compose.yaml`. Read it.
   Keep the `api` and `llm` services. The `api` service needs `shm_size: 1gb` for Chromium.
3. In `.env`: `DECOSA_LLM_ROUTE=direct`, `DECOSA_TESTRUNS_TARGETS=<your staging hosts>`, `DECOSA_TESTRUNS_MAX=2`,
   `DECOSA_TESTRUNS_TTL_S=86400`, `DECOSA_ADMIN_SECRET=<long random>`.
4. Start: `docker compose up -d`. Wait for `curl -fsS http://localhost:<PORT>/testruns/info` to show `"available": true`.
5. Mint a key: `POST /v1/keys` with the admin secret and `{"label": "ci", "verticals": ["test-runs"]}`.
6. Smoke test: `curl -fsSO localhost:<PORT>/testruns/sdk/decosa_testrun.py`; save sample 3 from `/testruns/samples`
   as `promo.yml`; run `DECOSA_API=http://127.0.0.1:<PORT> DECOSA_API_KEY=dk_... python3 decosa_testrun.py promo.yml --target fixture --build good --out ev/`
   (expect exit 0) and again with `--build promo-wrong` (expect exit 1, step 3 read "-$4.80"). Flip one `"pass"` in a
   certificate and POST it to `/testruns/verify`: it must fail and name the assertion.
7. Report back: health, the signing key id from `/attest/signing-key`, the two verdicts, and the run times you saw.

No NVIDIA GPU? This tool also runs entirely on an Apple Silicon Mac (MLX, 32 GB of unified memory or more): use https://decosa.ai/prompts/test-runs-mac.md instead.