arizuko

arizuko › components › litellm

litellm

litellm is a model gateway that holds the model credential, so the guest of the agent never sees it, and it keeps the spend ledger.

litellm holds the model credential so the guest does not

litellm is the model gateway: a third-party LLM proxy, BerriAI/litellm with the image ghcr.io/berriai/litellm, that arizuko ships as a package. It holds the model credential of the instance and forwards every agent request to Anthropic with it. The guest of the agent never sees that credential. It gets MODEL_GATEWAY_KEY, a virtual key that only the gateway accepts.

Run arizuko package add -i <instance> catalog:litellm. Restart the instance to start the gateway. The package contains a compose fragment, template/services/litellm/litellm.yml, and an arizuko.yml manifest. The fragment contains the litellm configuration. No route reaches it from outside the docker network.

Without a gateway, the model credential enters the guest

Without the gateway, the model credential is the one secret that enters the guest by design. ANTHROPIC_API_KEY or CLAUDE_CODE_OAUTH_TOKEN is in the environment of the turn, because the Claude Code process itself uses it. See concepts/secrets. Anything that runs in that guest can read it, and the only revocation is a rotation of the credential for everyone who shares it.

The client is configured to use the gateway, and nothing intercepts its traffic. The client gets the target of its traffic in ANTHROPIC_BASE_URL and its bearer in ANTHROPIC_AUTH_TOKEN. Nothing is decrypted or spliced, and no CA goes into the guest. For this reason the gateway works, and a proxy that terminates TLS is not used.

Each request passes four steps

  1. runed starts the guest of the turn with MODEL_GATEWAY_URL and MODEL_GATEWAY_KEY. For the claude harness, these become ANTHROPIC_BASE_URL=http://litellm:4000 and ANTHROPIC_AUTH_TOKEN=<key>. The real credential is not in the environment.
  2. Claude Code posts to http://litellm:4000/v1/messages with Authorization: Bearer <key>. litellm checks the key against its master key. It refuses a missing or wrong key with 401, or an unknown key with 400 no_db_connection. litellm reports that miss as a database lookup it cannot do.
  3. litellm inserts the real credential and calls api.anthropic.com. An API key goes out as x-api-key. An sk-ant-oat subscription token goes out as Authorization: Bearer with the anthropic-beta: oauth-2025-04-20 header, which litellm adds itself. The upstream User-Agent is the one of litellm, measured as litellm/1.101.0, and arizuko forges nothing.
  4. The reply streams back. litellm writes one JSON access line to stdout. When the instance exports OTLP, it also sends one span per request with the gen_ai.usage.* token counts and gen_ai.cost.total_cost.

The configuration passes any model id through with model_name: "*", so a new Claude release needs no edit. When .env sets both, ANTHROPIC_API_KEY wins over CLAUDE_CODE_OAUTH_TOKEN.

The gateway sits between the guest and the provider

 .env: ANTHROPIC_API_KEY | CLAUDE_CODE_OAUTH_TOKEN, MODEL_GATEWAY_KEY
   │                                              │
   │ docker interpolates                          │ runed hands to every turn
   ▼                                              ▼
 litellm  ◄──── POST /v1/messages ──────────  the turn's guest
   │          Authorization: Bearer $MODEL_GATEWAY_KEY
   │ real credential
   ▼
 api.anthropic.com
   │
   ├──► stdout JSON access line  ──► journalctl -u arizuko_<inst>
   └──► OTel span (tokens, cost) ──► OTEL_EXPORTER_OTLP_ENDPOINT, if set
Inputs
HTTP from the guests of turns, through their egress proxies on the compose network.
Outputs
HTTPS to the model provider.
Dependencies
MODEL_GATEWAY_KEY in .env. arizuko package add -i <instance> catalog:litellm creates it there if absent. The fragment requires the key. So a fragment copied by hand with no key does not start, and no gateway runs that accepts every caller.

Egress isolation is not a setting. Every turn reaches the gateway through its own egress proxy, which runs in the container of runed on the compose network where litellm resolves. The proxy forwards plain HTTP to any host on the allowlist of the turn, so litellm must be on that list.

The credential decides the billing

The gateway works with either credential. What you put in .env decides the billing and which clients can use it.

API key, ANTHROPIC_API_KEY
Every harness works through the gateway in the same way: claude, codex and pi. Billing is per token.
Subscription OAuth token, CLAUDE_CODE_OAUTH_TOKEN
Only a first-party client uses the plan limits, and Anthropic checks which client asks, not which gateway forwarded the request. A measurement on 2026-09-04 through this gateway on plan credentials gave these results. The Claude Code CLI and codex completed turns on the plan quota. pi received HTTP 400, “Third-party apps now draw from your extra usage, not your plan limits.” So pi needs extra-usage credits or an API key. No gateway setting changes that. arizuko ships no mechanism that hides which client calls.

The legal and compliance page of Anthropic documents how to provision an API key in a shared environment. The same page says that developers may not intermediate Claude.ai session tokens. An operator who points the gateway at their own subscription must read it and decide.

The package ships its own database

litellm keeps virtual keys, budgets and its spend log in PostgreSQL, so the package ships one, litellm_db, next to the proxy. Without it, there are no keys per turn, no budgets and no spend table: one MODEL_GATEWAY_KEY for every spawn, and GET /spend/logs returns 500. This PostgreSQL holds the state of the gateway, not of arizuko. So it lives in a named volume and not under the data directory of the instance.

The gateway is the only spend ledger

arizuko stores no spend. A second ledger can only agree, lag or be wrong, so one number exists, and litellm holds it.

A folder is a team
The folder path is the litellm team_id as written. The cost_cap_cents_per_day of the folder is the max_budget of that team on a 1d window. litellm enforces the cap on each request: over the budget, it answers 429 Budget has been exceeded! Team=<folder>, and the call never reaches the model.
A person is an internal user
routd sets user_id on the key it creates, so the guest cannot change the person that a turn is charged to.
A turn is a key
One virtual key per turn, valid for the run ceiling, deleted when the turn ends.

The check before a spawn is one GET /team/info that does three things. It creates the team that the key of a turn needs. A key that names a team unknown to litellm fails every model call with 404. It moves the budget of the team to the cap that the operator holds now. It reads the spend that a refusal quotes. Everything the operator sees, on the dashboard or over REST, reads the same meter.

Spend rows land on a batch timer, proxy_batch_write_at, which is 1 second here, so a row is readable a moment after the call. That is the only lag. Because nothing in arizuko holds a copy of the spend, nothing in arizuko can disagree with the meter.

Logs go to journald and costs go to OTLP

Every compose service runs attached under the systemd unit of the instance, and the docker log driver is none. So the stdout of a container goes to journald and nowhere else. litellm joins that stream in the same way, with the same arizuko_litellm_<flavor> | prefix as every other service:

sudo journalctl -u arizuko_<instance> --no-pager | grep litellm_

The fragment sets json_logs: true, so each line is JSON like the lines of every Go daemon. Each request gives one uvicorn.access record with method, path and status. Each refused key gives an ERROR record. Nothing about the prompt is logged.

Token and cost accounting uses the existing OTLP settings of the instance. litellm reads OTEL_EXPORTER_OTLP_ENDPOINT, OTEL_EXPORTER_OTLP_PROTOCOL and OTEL_EXPORTER_OTLP_HEADERS under the same names as obs. The fragment turns its otel callback on only when the endpoint is set. Spans go to <endpoint>/v1/traces with gen_ai.usage.input_tokens, gen_ai.usage.output_tokens, gen_ai.cost.total_cost, and the request and response model. Prompts and replies stay out: the fragment sets OTEL_INSTRUMENTATION_GENAI_CAPTURE_MESSAGE_CONTENT=false, and a check of the exported payload found no message text. Without a collector, the gateway keeps no cost record per request. The console span exporter would print about two hundred lines per request into the journal, the prompt included, so it stays off.

litellm runs alone with a nine-line configuration

litellm runs alone. It is the upstream litellm with a nine-line configuration, and arizuko adds nothing to the image.

cat > config.yaml <<'EOF'
model_list:
  - model_name: "*"
    litellm_params:
      model: "*"
      api_key: os.environ/ANTHROPIC_API_KEY
litellm_settings:
  json_logs: true
EOF
docker run -d --name litellm -p 4000:4000 \
  -v "$PWD/config.yaml:/app/config.yaml:ro" \
  -e ANTHROPIC_API_KEY=<your api key> \
  -e LITELLM_MASTER_KEY=sk-my-virtual-key \
  ghcr.io/berriai/litellm:main-latest --config /app/config.yaml --port 4000

# probe
curl -s localhost:4000/health/liveliness
ANTHROPIC_BASE_URL=http://localhost:4000 ANTHROPIC_AUTH_TOKEN=sk-my-virtual-key claude -p 'hi'

Four settings configure the gateway

MODEL_GATEWAY_KEY
The litellm master key. arizuko package add -i <instance> catalog:litellm creates it in .env if absent. The gateway issues a separate virtual key for each turn. Delete the line to rotate it on the next generate.
MODEL_GATEWAY_URL
While the package is installed, every generate sets it to http://litellm:4000 by default. runed gives it to the spawn as its base URL. Set it in .env to use another gateway.
ANTHROPIC_API_KEY, CLAUDE_CODE_OAUTH_TOKEN
The real credential, which the gateway reads from .env. The API key wins.
OTEL_EXPORTER_OTLP_ENDPOINT, OTEL_EXPORTER_OTLP_PROTOCOL, OTEL_EXPORTER_OTLP_HEADERS
The OTLP settings of the instance, passed through unchanged.

reference/env has the full list and the defaults.

Three pages cover the details

Edit this page · updated