arizuko › howto › Kubernetes

deploy on Kubernetes

arizuko runs on a single Linux host under Docker Compose, and that is still the shortest path to a working instance. Kubernetes buys you two things: secrets that live in etcd instead of a mode-0600 .env, and the freedom to run the daemons that hold no file on more than one node. It does not buy you a scalable agent runtime — the turn path is still pinned to one machine, for reasons that have nothing to do with the database.

This page is a decision, not a recipe: what you can move, what you cannot, and the reason for each. Read it before you write a manifest.

what you can move: the rows

Every owner’s tables live on one DynamoDB-protocol server per instance. store.Served() is the branch, and it reads one variable: with DYNAMODB_URL set, a daemon opens its owner on that endpoint and opens no file at all (objstore/dynstore/env.go).

So the rows can leave the node entirely. The endpoint can be the extenddb pair the instance ships (template/services/extenddb.yml — ExtendDB over a PostgreSQL you already run), a PostgreSQL in another cluster, or Amazon DynamoDB itself: the daemons speak one protocol and never learn which is behind it.

A daemon needs five values to reach it, and FromEnv refuses a half-configured set rather than failing later at signing time:

The server’s own init run is yours to do once by hand — it prints an admin password and mints one key per daemon. Steps are in INSTALL.md § The store server. Nothing about that run changes under Kubernetes.

what stays pinned: routd and runed

routd and runed run one replica each and share a filesystem. Four reasons, none of them the database:

• The agent MCP socket. routd binds a Unix socket per turn at <data>/ipc/<folder>/gated.sock (routd/mcp.go ServeTurnMCP), and runed bind-mounts that directory into the agent container at /run/ipc (container/runner.go buildMounts). A Unix socket is an inode. The two daemons must see the same one.
• The container runtime socket. runed spawns each turn with docker run, so its pod mounts /var/run/docker.sock and joins the gid that owns it (compose/compose.go runedService, dockerGID). It is the only daemon wired that way, and it is wired to one node’s daemon.
• Bind sources are node paths. The spawned container is a sibling, not a child, so the node’s container runtime resolves every mount source. container.hp() rewrites each path from the data dir as runed sees it to the data dir as the node sees it, which is what HOST_DATA_DIR is for. Get that value wrong and every spawn mounts an empty directory.
• The group tree is files. routd reads PERSONA.md to build the prompt (routd/prompt.go personaBlock), CLAUDE.md for proactive mode (routd/proactive.go), and each folder’s .claude/skills/self/MIGRATION_VERSION to decide whether to enqueue a migration turn. runed mounts the same folder as the agent’s home.

what you can actually scale

Six daemons hold no file once the store is served. That makes them movable — not automatically replicable. Here is what each one still needs, which is what actually decides the replica count.

Only authd and proxyd take a second replica without an argument. onbod and dashd mount narrow subdirectories in compose, which reads as “nearly stateless” and is not — they write the group tree. proxyd and webd mount only routd’s store today, so once the store is served their mount carries nothing and can go.

adapters: one replica per account

Adapter dedup and delivery-claim tables moved to the store with everything else (chanlib.OpenAdapterStore). What did not move is the small per-account state each platform forces:

• teled writes its Telegram getUpdates offset to <DATA_DIR>/teled-offset-<name>. Lose it and Telegram re-delivers roughly a day of backlog, which the agent answers all over again. Two pollers on one bot token also fight for the same long-poll.
• whapd keeps the paired WhatsApp session in WHATSAPP_AUTH_DIR. Two processes writing it corrupt the pairing and you re-pair by hand.

So: one replica per adapter per account, with a small RWO volume. Two Telegram bots are two Deployments, not two replicas.

what is left on the volume

Once the rows are served, the shared volume is four directories, not the whole instance directory:

All four must be visible at a node path, because the node’s container runtime resolves the agent container’s bind mounts. A local or hostPath volume satisfies that; a network PVC whose node mount point you cannot name does not.

the shape that fits

Split Deployments fit the code better than one big pod, because proxyd’s backends are already rows naming http://<daemon>:8080 (see any template/services/*.yaml manifest). Give each daemon a Service named after it, keep the in-container port at :8080, and those rows resolve unchanged.

volume

One RWO volume for groups/, web/, ipc/ and app-src/, pinned to the node runed runs on. Size it for group files and media; the rows are on the store server now and do not grow it.

# pvc.yaml
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: arizuko-data
spec:
  accessModes:
    - ReadWriteOnce
  storageClassName: local-path   # a node-local class; see the note below
  resources:
    requests:
      storage: 20Gi

ConfigMap

Non-secret config. Visible in kubectl get configmap -o yaml.

# configmap.yaml
apiVersion: v1
kind: ConfigMap
metadata:
  name: arizuko-config
data:
  ASSISTANT_NAME: "Ari"
  ARIZUKO_INSTANCE: "demo"
  WEB_HOST: "https://demo.example.com"
  CONTAINER_IMAGE: "arizuko-ant:latest"
  DATA_DIR: "/srv/app/home"
  # the NODE path of the same volume — container.hp() rewrites every agent
  # bind source through this. Wrong value = every spawn mounts nothing.
  HOST_DATA_DIR: "/var/lib/rancher/k3s/storage/pvc-…_arizuko-data"
  AUTHD_URL: "http://authd:8080"
  ROUTER_URL: "http://routd:8080"
  RUNED_URL: "http://runed:8080"
  DYNAMODB_URL: "https://extenddb:18443"
  AWS_REGION: "us-east-1"
  LOG_LEVEL: "info"

Secret

Every daemon gets its own AUTHD_SERVICE_KEY and its own store access key, so in practice this is one Secret per daemon rather than one shared object. The shape:

# secret.yaml
apiVersion: v1
kind: Secret
metadata:
  name: arizuko-routd
type: Opaque
stringData:
  # routd owns the secrets table, so it holds the key that seals it.
  # runed deliberately does NOT get this one.
  SECRETS_KEY: "32-char-random"
  AUTHD_SERVICE_KEY: "per-daemon-service-key"
  AWS_ACCESS_KEY_ID: "routd-store-key"
  AWS_SECRET_ACCESS_KEY: "routd-store-secret"
  STORE_CA: "LS0tLS1CRUdJTiBDRVJU…"   # base64 of the endpoint cert

dashd also needs SECRETS_KEY — it writes user secrets directly and must seal with the same key routd reads with. authd, proxyd and dashd need AUTH_SECRET. Adapters need their platform token and nothing else from this list. The authoritative map is daemonKeys in compose/compose.go, which is what arizuko generate uses to scope each daemon’s env file; an unlisted key never reaches a daemon, and that is the point.

the pinned pair

routd and runed in one Deployment, one replica, sharing the volume. runed needs the node’s container socket and the gid that owns it.

# routd-runed.yaml
apiVersion: apps/v1
kind: Deployment
metadata:
  name: arizuko-turn
spec:
  replicas: 1
  strategy:
    type: Recreate          # never two runed holding one docker.sock
  selector:
    matchLabels: { app: arizuko, tier: turn }
  template:
    metadata:
      labels: { app: arizuko, tier: turn }
    spec:
      securityContext:
        runAsUser: 1000
        runAsGroup: 1000
        supplementalGroups: [999]   # the gid owning /var/run/docker.sock
      containers:
        - name: routd
          image: arizuko:latest
          command: ["routd"]
          envFrom:
            - configMapRef: { name: arizuko-config }
            - secretRef:    { name: arizuko-routd }
          volumeMounts:
            - { name: data, mountPath: /srv/app/home }
          readinessProbe:
            httpGet: { path: /health, port: 8080 }
            initialDelaySeconds: 5

        - name: runed
          image: arizuko:latest
          command: ["runed"]
          envFrom:
            - configMapRef: { name: arizuko-config }
            - secretRef:    { name: arizuko-runed }
          env:
            - { name: LISTEN_ADDR, value: ":8082" }   # routd already has :8080
          volumeMounts:
            - { name: data,   mountPath: /srv/app/home }
            - { name: docker, mountPath: /var/run/docker.sock }

      volumes:
        - name: data
          persistentVolumeClaim: { claimName: arizuko-data }
        - name: docker
          hostPath: { path: /var/run/docker.sock, type: Socket }

seeding the data dir

arizuko create writes the instance directory skeleton. Run it once as a Job against the same volume, or on the node before you apply anything. It is not a per-start step and does not belong in an init container that runs on every rollout.

The store server’s init run is separate and also one-time — it prints an admin password once (INSTALL.md § The store server). Do it before the first daemon starts, or every daemon boots onto tables that do not exist.

exposing it

proxyd is the only thing the outside world talks to. It serves plain HTTP on :8080 and terminates no TLS itself, so put an Ingress or a LoadBalancer in front of it.

# service.yaml
apiVersion: v1
kind: Service
metadata:
  name: proxyd
spec:
  selector: { app: arizuko, tier: proxyd }
  ports:
    - { name: http, port: 8080, targetPort: 8080 }
  type: ClusterIP

Give every other daemon the same shape — Service named after the daemon, port 8080 — and the proxyd_routes rows arizuko generate wrote keep resolving.

secret management with External Secrets

This is the reason most people come to Kubernetes with arizuko. The External Secrets Operator syncs a K8s Secret from Vault, AWS Secrets Manager or GCP Secret Manager on a schedule; rotation becomes a push to the store plus a rollout. With IRSA (AWS) or Workload Identity (GCP) the pod holds no long-lived credential at all — its identity is the proof.

helm repo add external-secrets https://charts.external-secrets.io
helm install external-secrets external-secrets/external-secrets \
  -n external-secrets --create-namespace

This example uses AWS Secrets Manager with IRSA (the pod’s ServiceAccount is annotated with an IAM role holding secretsmanager:GetSecretValue):

# secretstore.yaml
apiVersion: external-secrets.io/v1beta1
kind: SecretStore
metadata:
  name: aws-secrets
spec:
  provider:
    aws:
      service: SecretsManager
      region: us-east-1
      auth:
        jwt:
          serviceAccountRef:
            name: arizuko   # annotated with the IAM role ARN

# externalsecret.yaml
apiVersion: external-secrets.io/v1beta1
kind: ExternalSecret
metadata:
  name: arizuko-routd
spec:
  refreshInterval: 1h
  secretStoreRef:
    name: aws-secrets
    kind: SecretStore
  target:
    name: arizuko-routd      # creates the K8s Secret
  dataFrom:
    - extract:
        key: arizuko/demo/routd

For Vault, swap the provider block and point at your KV path; for GCP, use the GCP provider with Workload Identity. The ExternalSecret shape stays the same. Add Reloader if you want a rotation to trigger the rollout by itself — a synced Secret does not restart a running pod on its own.

apply and verify

kubectl apply -f pvc.yaml -f configmap.yaml -f secret.yaml
kubectl apply -f routd-runed.yaml -f service.yaml

kubectl get pods -w
kubectl logs deploy/arizuko-turn -c routd --tail=30

kubectl port-forward svc/proxyd 8080:8080 &
curl -s http://localhost:8080/health

Healthy daemons are not a working instance. Send a message and wait for the agent to answer before you call it deployed — every gate short of that passes on a fleet that replies to nothing.

troubleshooting

Pod stuck in Pending

Usually the PVC isn’t bound. kubectl get pvc; if it’s Pending, no provisioner matched the StorageClass. On a single node, local-path or a manual local PV.

Every daemon logs a store error on boot

A half-configured endpoint. FromEnv refuses an endpoint with no credential rather than failing later, so check all five values reached the container — kubectl exec … -- printenv | grep -E 'DYNAMODB|AWS_|STORE_CA'. An unset DYNAMODB_URL is worse than an error: the daemon quietly opens a file instead and looks healthy while it writes nowhere anyone reads.

Agent containers never spawn

Either runed can’t reach /var/run/docker.sock (check supplementalGroups matches the gid that owns it on the node) or HOST_DATA_DIR names a path the node does not have. The second one is quieter: the spawn succeeds and the agent finds an empty home.

The agent starts but has no tools

routd and runed landed on different filesystems, so the MCP socket routd bound is not the one runed mounted. They must share the volume and the node.

Telegram replays a day of messages after a rollout

teled lost its offset file. Give the adapter its own small persistent volume rather than emptyDir.

Secrets not applying after a Vault rotation

External Secrets updates the K8s Secret; the running pod keeps the env it started with. kubectl rollout restart, or install Reloader.