Run an open model behind the OpenAI API

You are setting up a **private code assistant** on this Linux machine: Qwen3.8-27B (Apache-2.0) on vLLM 0.29.0, with NVFP4 weights, its built-in MTP draft head (3 tokens) and an FP8 KV cache, behind an OpenAI-compatible `/v1` endpoint. Coding agents on this machine will use it, and source code must never leave the box. Work step by step, show me each command before running anything with `sudo`, and stop to ask if a check fails.

## Step 0: set up with a coding agent, rehearse on mock data, then go private

This prompt is for a coding agent running on the machine that will host the service. We recommend Claude Code with
Claude Opus 5.5; any capable coding agent works. Work in this order:

1. Set up on mock data only. During the whole setup you (the agent) work with the synthetic sample bundle below and
   nothing else. Do not ask me for real data, and do not open, read, list or copy files that hold real data, even to
   "test with something realistic".
2. Rehearse. When the steps below are done and the service is healthy, fetch the mock-data bundle for this tool,
   https://decosa.ai/samples/code.zip (1 KB, 9 checks, synthetic or openly licensed: see `licence` in expected.json),
   show me what is in it, and run the rehearsal against the local API:
   `docker compose exec api python scripts/rehearse.py code` (the api image carries the same bundle under /app/rehearsal/code/;
   with no key set, the script asks the local API for a short demo token). From a decosa-api checkout instead:
   `python scripts/rehearse.py code --bundle code.zip --base-url http://127.0.0.1:<PORT>`.
   It sends the mock inputs to the local API and prints PASS or FAIL for each expected property (for example: "the answer defines merge_intervals", "the answer carries doctests", "the answer finished within the token limit"). Show me
   the full output. Every check must pass. If one fails, fix the install and run it again; never edit `expected.json`
   to make a check pass.
3. Stop there. Once the rehearsal passes, tell me, and I will run my own data against the local API myself, on this
   machine.

For the person running this: a coding agent that runs in the cloud sees everything in its context, including files it
reads, command output and anything pasted into the chat. Keep real data out of the chat and out of anything the agent
can read. Switch to your own data only after the rehearsal has passed and the agent's work is done.

## 1. Check the GPU, driver and Docker
- Run `nvidia-smi`. The pinned stack needs a Blackwell GPU (RTX PRO 6000, B200, RTX 50-series) for NVFP4, and a driver of 580 or newer (the images were built against it). A 96 GB card is the measured target. 24–32 GB Blackwell cards can work with a shorter context, but that hasn't been measured.
- On Hopper or Ada (no NVFP4), use `LLM_MODEL=Qwen/Qwen3.8-27B-FP8` and `LLM_REVISION=main` below. That is the official FP8 checkpoint, Apache-2.0, 29 GB.
- If `docker` is missing, install Docker Engine from the official docs (https://docs.docker.com/engine/install/). If `docker run --rm --gpus all ubuntu nvidia-smi` fails, install the NVIDIA Container Toolkit (https://docs.nvidia.com/datacenter/cloud-native/container-toolkit/latest/install-guide.html), then run `sudo nvidia-ctk runtime configure --runtime=docker && sudo systemctl restart docker` and repeat the check.
- Make sure there are about 60 GB free for images and weights.

## 2. Images and weights (pinned)
- `${DECOSA_REGISTRY}/decosa-llm:0.1.0` is vLLM 0.29.0 serving the model. `${DECOSA_REGISTRY}/decosa-api:0.1.0` is an optional proxy that adds sessions, budgets and receipt ids. **Both are publishing soon.** Try `docker pull` first.
- Fallback for `decosa-llm`: it's a thin layer over the pinned vLLM image, so build it locally.
  ```bash
  mkdir -p ~/decosa-code/llm && cat > ~/decosa-code/llm/Dockerfile <<'EOF'
  FROM vllm/vllm-openai@sha256:c2914767605584b6d8f45686b82de173ecc99e781897aa3d0a66dacd72c51ae1
  ENTRYPOINT ["vllm", "serve"]
  EOF
  ```
- Fallback for `decosa-api`: build it from source (`docker build -f docker/api/Dockerfile .`) once its repo is published. Until then, skip the `api` service. Coding agents only need vLLM.
- Weights: `nvidia/Qwen3.8-27B-NVFP4` at revision `482ca0f3832238542f8f5295dde86b5f22711d80` (about 21 GB). They download into the `hf-cache` volume on first start.

## 3. Write `~/decosa-code/docker-compose.yml`
```yaml
name: decosa-code
services:
  llm:
    image: ${DECOSA_REGISTRY}/decosa-llm:0.1.0
    build: ./llm                      # used only if the image can't be pulled
    ipc: host
    restart: unless-stopped
    deploy:
      resources:
        reservations:
          devices: [{ driver: nvidia, device_ids: ["${DECOSA_GPU:-0}"], capabilities: [gpu] }]
    environment: { HF_TOKEN: "${HF_TOKEN:-}" }
    volumes: [hf-cache:/root/.cache/huggingface]
    ports: ["127.0.0.1:8000:8000"]    # localhost only: nothing on the network can reach the model
    command:
      - ${LLM_MODEL:-nvidia/Qwen3.8-27B-NVFP4}
      - --revision
      - ${LLM_REVISION:-482ca0f3832238542f8f5295dde86b5f22711d80}
      - --served-model-name
      - qwen3.8-27b
      - --language-model-only
      - --max-model-len
      - "${LLM_MAX_LEN:-262144}"
      - --gpu-memory-utilization
      - "${LLM_GPU_UTIL:-0.90}"
      - --max-num-seqs
      - "32"                          # MTP helps up to ~32 streams and hurts above (measured)
      - --kv-cache-dtype
      - fp8_e4m3
      - --speculative-config
      - '{"method":"mtp","num_speculative_tokens":3}'
      - --enable-auto-tool-choice     # agent tool calls (flags from the NVIDIA model card)
      - --tool-call-parser
      - qwen3_coder
      - --reasoning-parser
      - qwen3
      - --seed
      - "0"
      - --host
      - 0.0.0.0
      - --port
      - "8000"
    healthcheck:
      test: ["CMD", "python3", "-c", "import urllib.request; urllib.request.urlopen('http://127.0.0.1:8000/health', timeout=4)"]
      interval: 15s
      retries: 5
      start_period: 900s
  api:                                # optional: sessions, budgets, receipt ids
    image: ${DECOSA_REGISTRY}/decosa-api:0.1.0
    restart: unless-stopped
    depends_on: { llm: { condition: service_healthy } }
    environment:
      DECOSA_LLM_ROUTE: direct        # local model; receipts are signed by this box's key ("attested")
      DECOSA_LLM_URL: http://llm:8000/v1
      DECOSA_LLM_MODEL: qwen3.8-27b
      DECOSA_BUDGET_LLM_TOKENS: "2000000"
      DECOSA_SESSION_TTL_S: "28800"
      DECOSA_SESSIONS_PER_IP_HOUR: "1000"
      DECOSA_CODE_MAX_TOKENS: "8192"
      DECOSA_CODE_MAX_INPUT_CHARS: "400000"
    ports: ["127.0.0.1:8445:8445"]
    volumes: [decosa-data:/data]
volumes: { hf-cache: {}, decosa-data: {} }
```
Start it with `cd ~/decosa-code && docker compose up -d`. The first start downloads the weights, and `llm` takes about 5–10 minutes to become healthy. On a smaller card, lower `LLM_MAX_LEN` (for example 32768) and `LLM_GPU_UTIL`.

## 4. Smoke test
```bash
curl -s localhost:8000/v1/models | jq -r '.data[].id'          # expect qwen3.8-27b (and its older alias)
curl -s localhost:8000/v1/chat/completions -H 'content-type: application/json' -d '{
  "model":"qwen3.8-27b","max_tokens":400,"chat_template_kwargs":{"enable_thinking":false},
  "messages":[{"role":"user","content":"Write a Python function merge_intervals(intervals). Code only."}]}' \
  | jq -r '.choices[0].message.content'                        # expect a Python function
curl -s localhost:8000/v1/chat/completions -H 'content-type: application/json' -d '{
  "model":"qwen3.8-27b","messages":[{"role":"user","content":"Read the file README.md"}],
  "tools":[{"type":"function","function":{"name":"read_file","parameters":{"type":"object",
  "properties":{"path":{"type":"string"}},"required":["path"]}}}]}' \
  | jq '.choices[0].message.tool_calls'                        # expect a read_file call
```
If you started `api`, check it as well:
```bash
curl -s localhost:8445/healthz        # "llm": true ("asr" is false here because this stack has no speech model)
TOKEN=$(curl -s localhost:8445/demo/session -H 'content-type: application/json' -d '{"vertical":"code"}' | jq -r .token)
curl -si localhost:8445/v1/chat/completions -H "authorization: Bearer $TOKEN" -H 'content-type: application/json' \
  -d '{"model":"qwen3.8-27b","messages":[{"role":"user","content":"Say hi"}]}' | grep -i x-decosa-receipt
```
The response body carries `"receipt": {"status": "attested", ...}` in direct mode. The `api` route passes `tools` through: on the hosted service a request with `tools` returns `tool_calls` and a receipt (checked 2026-09-30). On your own box that needs the two tool-call flags above and has not been re-verified there, so test one call first, or point agents straight at vLLM.

## 5. Point your coding agents at the local endpoint
Base URL `http://127.0.0.1:8000/v1`, model `qwen3.8-27b`. vLLM started without `--api-key` accepts any key. If you add one, pass the same value below. Generic OpenAI-SDK tools: `export OPENAI_BASE_URL=http://127.0.0.1:8000/v1 OPENAI_API_KEY=local`.
- **OpenCode** (`opencode.json` in the project, or `~/.config/opencode/opencode.json`):
  ```json
  { "$schema": "https://opencode.ai/config.json",
    "provider": { "local": { "npm": "@ai-sdk/openai-compatible", "name": "Local Qwen",
      "options": { "baseURL": "http://127.0.0.1:8000/v1" },
      "models": { "qwen3.8-27b": { "name": "Qwen3.8-27B (local)" } } } },
    "model": "local/qwen3.8-27b" }
  ```
- **Continue** (`~/.continue/config.yaml`):
  ```yaml
  name: Local
  version: 1.0.0
  schema: v1
  models:
    - name: Qwen3.8-27B (local)
      provider: openai
      model: qwen3.8-27b
      apiBase: http://127.0.0.1:8000/v1
      apiKey: local
      roles: [chat, edit, apply]
      capabilities: [tool_use]
  ```
- **Codex-style CLIs** (`~/.codex/config.toml`; vLLM serves both `/v1/responses` and `/v1/chat/completions`, so choose the `wire_api` your version supports):
  ```toml
  model = "qwen3.8-27b"
  model_provider = "local"
  [model_providers.local]
  name = "Local vLLM"
  base_url = "http://127.0.0.1:8000/v1"
  env_key = "LOCAL_LLM_KEY"      # export LOCAL_LLM_KEY=local
  wire_api = "responses"
  ```
- vLLM 0.29.0 also serves an Anthropic-style `/v1/messages` route. We haven't tested agents against it.
- Tool calling via `qwen3_coder` follows the NVIDIA model card. It hasn't been soak-tested on this exact pin yet, so run your agent on a throwaway branch first.

## Notes
- Optional larger tier: DeepSeek-V4-Flash (MIT, 284B total / 13B active) on **two** RTX PRO 6000 cards, using the public MIT kit https://github.com/hikarioyama/dsv4-flash-nvfp4-sm120 (`serve_b12x_tp2.sh`, patched B12X vLLM build, MTP on). Stop `llm` first, because it takes both GPUs. It is served as `DeepSeek-V4-Flash` with the `deepseek_v4` tool parser. It isn't a hosted model, so it produces no receipts.
- Qwen3.8-Flash-Next is deliberately excluded. Its Qwen Community License requires a separate licence for any Model-as-a-Service business.