arizuko

arizuko › components › litellm

litellm

What it is

litellm is the model gateway: a third-party LLM proxy (BerriAI/litellm, image ghcr.io/berriai/litellm) that arizuko ships as a package. It holds the instance’s model credential and forwards every agent request to Anthropic with it. An agent container never sees that credential — it gets MODEL_GATEWAY_KEY, a virtual key that only the gateway accepts.

Add it with arizuko packages <instance> add litellm. The package is one compose fragment, template/services/litellm.yml, with the litellm config inline; there is no route into it from outside the docker network, so it ships no .yaml manifest.

Why it exists

Without it, the model credential is the one secret that enters the container by design: ANTHROPIC_API_KEY or CLAUDE_CODE_OAUTH_TOKEN lands in the spawn’s env because the Claude Code process itself spends it (see concepts/secrets). Anything running in that container can read it, and the only revocation is rotating the credential for everyone who shares it.

A gateway is configured, not intercepted. The client is told where to send its traffic (ANTHROPIC_BASE_URL) and what to send as its bearer (ANTHROPIC_AUTH_TOKEN); nothing is decrypted or spliced, and no CA goes into the container. That is why this works where a TLS-terminating proxy was declined.

The loop

  1. runed starts the agent container with MODEL_GATEWAY_URL and MODEL_GATEWAY_KEY. For the claude harness those become ANTHROPIC_BASE_URL=http://litellm:4000 and ANTHROPIC_AUTH_TOKEN=<key>; the real credential is not in the env.
  2. Claude Code posts to http://litellm:4000/v1/messages with Authorization: Bearer <key>. litellm checks it against its master key. No key, or the wrong one, is refused (401, or 400 no_db_connection for an unknown key — litellm phrases the miss as a database lookup it cannot do).
  3. litellm swaps in the real credential and calls api.anthropic.com. An API key goes out as x-api-key; an sk-ant-oat subscription token goes out as Authorization: Bearer plus the anthropic-beta: oauth-2025-04-20 header, which litellm adds on its own. The User-Agent upstream is litellm’s (litellm/1.101.0 measured); arizuko forges nothing.
  4. The reply streams back. litellm writes one JSON access line to stdout, and — when the instance exports OTLP — one span per request carrying gen_ai.usage.* token counts and gen_ai.cost.total_cost.

The config passes any model id through (model_name: "*"), so a new Claude release needs no edit. ANTHROPIC_API_KEY wins over CLAUDE_CODE_OAUTH_TOKEN when both are set in .env.

How it fits

 .env: ANTHROPIC_API_KEY | CLAUDE_CODE_OAUTH_TOKEN, MODEL_GATEWAY_KEY
   |                                              |
   | docker interpolates                          | runed hands to every spawn
   v                                              v
 litellm  <---- POST /v1/messages ----------  agent container
   |          Authorization: Bearer $MODEL_GATEWAY_KEY
   | real credential
   v
 api.anthropic.com
   |
   +--> stdout JSON access line  --> journalctl -u arizuko_<inst>
   +--> OTel span (tokens, cost) --> OTEL_EXPORTER_OTLP_ENDPOINT, if set

Inputs: HTTP from agent containers on the compose network. Outputs: HTTPS to the model provider. Hard deps: MODEL_GATEWAY_KEY in .env — arizuko packages <name> add litellm mints it there, and the fragment still interpolates it as required, so a hand-copied fragment with no key does not start at all rather than starting a gateway that accepts every caller.

With egress isolation on (EGRESS_IMAGE set), a spawn reaches the gateway through its own egress proxy, which sits on the compose network where litellm resolves. The proxy forwards plain HTTP to any host on the spawn’s allowlist, so litellm must be on it.

Which credential you give it

The gateway is credential-agnostic; what you put in .env decides the billing and who can use it.

Anthropic’s legal and compliance page documents provisioning an API key in a shared environment; the same page says developers may not intermediate Claude.ai session tokens. An operator pointing the gateway at their own subscription should read it and decide.

Why the package ships a database

litellm keeps virtual keys, budgets and its spend log in Postgres, so the package ships one (litellm_db) beside the proxy. Without it there are no per-turn keys, no budgets and no spend table — one MODEL_GATEWAY_KEY shared by every spawn, and GET /spend/logs answering 500. arizuko’s own stores stay SQLite; this Postgres is the gateway’s state, not arizuko’s, so it lives in a named volume rather than under the instance data dir.

The gateway is the ledger

arizuko stores no spend (spec 6/39). A second ledger can only agree, lag, or lie, so there is one number and litellm holds it.

The pre-spawn gate is a single GET /team/info doing three jobs at once: it creates the team a turn’s key needs to exist at all — a key naming a team litellm does not have fails every model call with 404 — it moves the team’s budget onto whatever cap the operator now holds, and it reads the spend its refusal quotes. Everything the operator sees, on the dashboard or over REST, reads back through the same meter.

Spend rows land on a batch timer (proxy_batch_write_at, 1 second here), so a row is readable a moment after the call, not instantly. That is the only lag, and no design removes it: what the decision buys is that nothing in arizuko can disagree with the meter, because nothing in arizuko holds a copy.

Logs and metering

Every compose service runs attached under the instance’s systemd unit, and the docker log driver is none, so a container’s stdout goes to journald and nowhere else. litellm joins that stream by the same mechanism, with the same arizuko_litellm_<flavor> | prefix every other service carries:

sudo journalctl -u arizuko_<instance> --no-pager | grep litellm_

The fragment sets json_logs: true, so each line is JSON like every Go daemon’s: one uvicorn.access record per request with method, path and status, plus an ERROR record for each rejected key. Nothing about the prompt is logged.

Token and cost accounting rides the instance’s existing OTLP declaration. litellm reads OTEL_EXPORTER_OTLP_ENDPOINT, OTEL_EXPORTER_OTLP_PROTOCOL and OTEL_EXPORTER_OTLP_HEADERS under the same names obs uses, and the fragment turns its otel callback on only when the endpoint is set. Spans go to <endpoint>/v1/traces with gen_ai.usage.input_tokens, gen_ai.usage.output_tokens, gen_ai.cost.total_cost, and the request and response model. Prompts and replies stay out: the fragment sets OTEL_INSTRUMENTATION_GENAI_CAPTURE_MESSAGE_CONTENT=false, and the exported payload was checked to carry no message text. Without a collector there is no per-request cost record from the gateway — the console span exporter would print about two hundred pretty-printed lines per request into the journal, prompt included, so it stays off.

Standalone usage

Yes. It is upstream litellm with a nine-line config; arizuko adds nothing to the image.

cat > config.yaml <<'EOF'
model_list:
  - model_name: "*"
    litellm_params:
      model: "*"
      api_key: os.environ/ANTHROPIC_API_KEY
litellm_settings:
  json_logs: true
EOF
docker run -d --name litellm -p 4000:4000 \
  -v "$PWD/config.yaml:/app/config.yaml:ro" \
  -e ANTHROPIC_API_KEY=sk-ant-... \
  -e LITELLM_MASTER_KEY=sk-my-virtual-key \
  ghcr.io/berriai/litellm:main-latest --config /app/config.yaml --port 4000

# probe
curl -s localhost:4000/health/liveliness
ANTHROPIC_BASE_URL=http://localhost:4000 ANTHROPIC_AUTH_TOKEN=sk-my-virtual-key claude -p 'hi'

Key env vars

Full list and defaults in reference/env.

Go deeper

Edit this page · updated