arizuko › howto › Kubernetes

Kubernetes

arizuko runs on one Linux host under Docker Compose, and Compose is the tested deploy path. The repository also has a Helm chart in deploy/helm/arizuko/ with the same daemons. No cluster has installed the chart yet. This page says what moves to Kubernetes, what stays on one node, and why.

The chart renders, and no cluster has run it

helm lint with ci/lint-values.yaml passes, and helm template renders every template: Deployments, Services, Secrets, the ConfigMap and the PVC. The image, entrypoint, env, mounts and privileges of each daemon come from compose/compose.go. No pod has started from the chart. [chart README] lists the values that you must set and the open questions.

# one-time, by hand, before the first start: extenddb init against your
# store, and arizuko create on the data volume
helm install arizuko deploy/helm/arizuko \
  --namespace arizuko --create-namespace \
  -f my-values.yaml

Kubernetes gives you two things. The secrets live in etcd or in an external secret store, not in a mode-0600 .env. And the daemons that hold no file can run on more than one node. Kubernetes does not give you an agent runtime over several nodes. The turn path stays on one machine.

The rows live on the store server, off the node

Every daemon keeps its tables on one DynamoDB-protocol server. With DYNAMODB_URL set, a daemon opens its tables at that endpoint and opens no file. The chart runs no store. The endpoint can be the extenddb pair from template/services/extenddb/extenddb.yml, a PostgreSQL that you run, or Amazon DynamoDB. The daemons speak one protocol and do not know which server answers.

A daemon needs five values to reach the store, and it refuses a partial set at start:

variable	meaning
`DYNAMODB_URL`	the endpoint, for example `https://extenddb:18443`
`AWS_REGION`	the signing region; default `us-east-1`
`AWS_ACCESS_KEY_ID`, `AWS_SECRET_ACCESS_KEY`	one pair for each daemon, so each daemon signs as itself and a policy limits its tables
`STORE_CA`	the endpoint certificate in base64; TLS is required, and the value is an env var because a daemon mounts no file for it

You run the init of the server once by hand. It prints an admin password and the account ID. [INSTALL.md] § The store server has the steps, and Kubernetes changes none of them.

routd and runed stay on one node

routd and runed run one replica each, on one node, with one shared volume. Four reasons hold them there, and none of them is the database:

The MCP socket of a turn is a file: routd binds one Unix socket for each turn at <data>/ipc/<folder>/gated.sock. runed runs a bridge for each turn that connects to that path and forwards it over the vsock of the guest. A Unix socket is an inode, so both daemons must see the same one.
runed needs the KVM device nodes: runed starts each turn as one qemu-system-x86_64 guest. In the chart, the runed pod mounts /dev/kvm, /dev/vhost-vsock and /dev/net/tun from the node as hostPath devices. It runs as root with NET_ADMIN and SYS_ADMIN, and its AppArmor profile is unconfined. Whether the hostPath mounts alone let runed open the devices under your container runtime is not measured. If they do not, runed logs /dev/kvm does not open and exits.
runed must open every path that a turn mounts: runed builds the staging tree of each turn with bind mounts in its own mount namespace and exports it with its child virtiofsd. No path is translated. So runed must open the group folder, the kernel and rootfs directories, and every root in MOUNT_ALLOWED_ROOTS. A path outside the data volume needs its own mount in the runed pod at the same absolute path. Otherwise the staging tree refuses it, and the turn fails.
The group tree is files: routd reads PERSONA.md for the prompt and CLAUDE.md for proactive mode. runed mounts the same folder as the home of the agent.

AppArmor, not seccomp, blocks the bind mounts. The default AppArmor profile denies mount(2) to a container, whatever its capabilities. Then the staging binds of runed fail with "permission denied", and every turn fails. Compose sets apparmor=unconfined on runed, and the chart sets appArmorProfile: Unconfined, which needs Kubernetes 1.30 or newer. The default seccomp profile already allows mount to a holder of CAP_SYS_ADMIN.

Two daemons take a second replica

Six daemons hold no file when the store is a server. That makes them movable, but not always replicable. The table shows what each one still needs:

daemon	still needs	replicas
`authd`	no disk; its own tables on the store, the OAuth env and the service keys	many
`proxyd`	no disk; its route table is the `proxyd_routes` rows, which it reads for each request	many
`webd`	no disk, but its SSE hub is a map in the process, `webd/hub.go`. `routd` posts each reply to `webd`. With two replicas, the post can reach the replica that the browser does not stream from.	one
`timed`	no disk and no data mount; it calls the HTTP API of `routd`. Two replicas start every schedule twice.	one
`onbod`	files: `container.SetupGroup` makes `groups/<folder>/` and the `web/pub` and `web/priv` folders of the group, so it needs the volume of `routd`, writable	one
`dashd`	files: it reads and writes skills, `CLAUDE.md` and uploads under `groups/`, and reads provider files from `surrogate/`	one

Only authd and proxyd take a second replica without a problem. onbod and dashd mount small subdirectories, but they write the group tree, so they are not stateless.

Each adapter account runs one replica

The adapter tables are on the store with all other tables. Each platform still needs some state for each account:

teled writes its Telegram getUpdates offset to <DATA_DIR>/teled-offset-<name>. Without the file, Telegram sends about one day of old messages again, and the agent answers them again. Two pollers on one bot token also compete for the same long poll.
whapd keeps the paired WhatsApp session in WHATSAPP_AUTH_DIR. Two processes that write it break the pairing, and you must pair again by hand.

So run one replica for each adapter account, with a small persistent volume. Two Telegram bots are two Deployments, not two replicas. The chart mounts no volume into an adapter, so the state of teled and slakd is lost with the pod. The chart does not include whapd.

The shared volume holds six directories

When the rows are on the store server, the shared volume holds only files:

path	holds	who uses it
`groups/`	one directory for each folder: the home of the agent, skills, `PERSONA.md`, `CLAUDE.md`, memory, media, logs	`routd` reads, `runed` mounts, `onbod` creates, `dashd` edits
`web/`	`pub/<folder>/` and `priv/<folder>/`, the web folders where an agent publishes	`runed` mounts, `onbod` creates
`ipc/`	one Unix socket for each folder during a turn	`routd` binds, the bridge of `runed` connects
`app-src/`	the `ant/` tree of the release, which `runed` copies out of its image at each start to share it into each guest	`runed` writes, the agent reads
`packages/`	installed package assets shared with guests	`runed` mounts, the agent reads
`images/`	the guest kernel and the rootfs from `make kernel` and `make rootfs`	`runed` boots them

All six are on one ReadWriteOnce volume that routd and runed mount at /srv/app/home. The chart puts every pod that mounts the volume on one node, through nodeName or nodeSelector. A node-local class such as local-path works. Whether a remote-attach CSI volume can back a virtiofsd export is not measured.

The proxyd route rows must name the chart Services

proxyd forwards by the proxyd_routes rows on the store. arizuko generate writes those rows with backends of the form http://<daemon>:8080, the Compose service names. The Services of the chart are named <release>-<daemon>, so those rows do not resolve on the cluster. The chart writes no rows, and you must write rows that name the chart Services. proxyd serves plain HTTP on :8080 and ends no TLS, so put an Ingress in front of it. The chart has an optional ingress.yaml.

External Secrets can own the Secrets

The chart makes one Secret for each daemon, named <release>-<daemon>, with exactly the keys that Compose gives that daemon. With secrets.external: true, the chart makes no Secret, and you create each one yourself. The External Secrets Operator can sync them from Vault, AWS Secrets Manager or GCP Secret Manager. A rotation is then a change in the secret store and a rollout. With IRSA on AWS or Workload Identity on GCP, the pod holds no long-lived credential.

helm repo add external-secrets https://charts.external-secrets.io
helm install external-secrets external-secrets/external-secrets \
  -n external-secrets --create-namespace

This example uses AWS Secrets Manager with IRSA. The ServiceAccount of the pod has an IAM role with secretsmanager:GetSecretValue:

# secretstore.yaml
apiVersion: external-secrets.io/v1beta1
kind: SecretStore
metadata:
  name: aws-secrets
spec:
  provider:
    aws:
      service: SecretsManager
      region: us-east-1
      auth:
        jwt:
          serviceAccountRef:
            name: arizuko   # annotated with the IAM role ARN

# externalsecret.yaml
apiVersion: external-secrets.io/v1beta1
kind: ExternalSecret
metadata:
  name: arizuko-routd
spec:
  refreshInterval: 1h
  secretStoreRef:
    name: aws-secrets
    kind: SecretStore
  target:
    name: arizuko-routd      # the Secret that the routd Deployment reads
  dataFrom:
    - extract:
        key: arizuko/demo/routd

For Vault or GCP, change the provider block. The ExternalSecret stays the same. A synced Secret does not restart a running pod. Add [Reloader] if a rotation must start the rollout.

A reply from the agent proves the deploy

kubectl -n arizuko get pods -l app.kubernetes.io/instance=arizuko
kubectl -n arizuko rollout status deploy/arizuko-routd
kubectl -n arizuko rollout status deploy/arizuko-runed

# runed names every unmet host precondition in its first log lines
kubectl -n arizuko logs deploy/arizuko-runed

Healthy pods are not a working instance. Send a message and wait for the answer of the agent before you call the instance deployed.

Six problems have a known cause

A pod stays Pending: Usually the PVC is not bound. Run kubectl get pvc. If it is Pending, no provisioner matches the StorageClass. On one node, use local-path or a local PV.
Every daemon logs a store error at start: The store values are incomplete. Check that all five values reach the container: kubectl exec <pod> -- printenv | grep -E 'DYNAMODB|AWS_|STORE_CA'.
No turn starts: runed logs every missing precondition at once and does not start. Read the log of the pod for the device, binary or directory that it names: /dev/kvm, /dev/vhost-vsock, /dev/net/tun, the egress binary, RUNED_KERNEL_DIR or RUNED_ROOTFS_DIR. reference/env § runed has the full set.
Every turn fails with staging tree: mount source: A turn mounts a path that the runed container cannot open. runed checks every source before it binds anything, so the message names the path. Mount that path into the runed pod at the same absolute path: the group volume and everything under MOUNT_ALLOWED_ROOTS or HOST_CODEX_DIR.
The agent starts and has no tools: routd and runed run on different filesystems. The bridge of runed then connects to a different socket than the one that routd bound. They must share the volume and the node.
The secrets do not change after a rotation: External Secrets updates the Secret, and the running pod keeps the env that it started with. Run kubectl rollout restart, or install Reloader.