arizuko › howto › Kubernetes

deploy on Kubernetes

arizuko runs fine on a single Linux host under Docker Compose. The main reason to move to Kubernetes is secret management: K8s Secrets live in etcd, which cloud providers encrypt at rest by default, and they integrate natively with Vault, AWS Secrets Manager, and GCP Secret Manager via the External Secrets Operator. No plaintext .env files on disk.

This guide covers a single-pod deployment — all daemons in one pod sharing a PersistentVolumeClaim. arizuko runs as three core daemons — authd (token authority), routd (conversation/router), runed (agent execution) — plus adapters and the web layer. They talk over HTTP (/v1/*); the per-turn agent MCP is a Unix socket routd hosts and runed mounts into the agent container. Each daemon owns its own SQLite DB on the shared PVC, and SQLite WAL has a single-writer constraint, so keep them in one pod (one replica). Split into separate Deployments only once you have a concrete reason.

why Kubernetes for secret management

arizuko needs a handful of long-lived secrets: SECRETS_KEY (the encryption key for the secrets table, read by routd), CHANNEL_SECRET, the per-daemon AUTHD_SERVICE_KEY each daemon uses to mint its service: token from authd, and one or more adapter tokens (TELEGRAM_BOT_TOKEN, SLACK_BOT_TOKEN, etc.). On a single host these live in a mode-0600 .env file — fine, but manual to rotate and invisible to audit logs.

In Kubernetes the same values become K8s Secret objects. Cloud-managed clusters encrypt etcd at rest by default; for tighter guarantees you can add envelope encryption backed by AWS KMS, GCP KMS, or HashiCorp Vault. IRSA (AWS) and Workload Identity (GCP) let pods fetch secrets without any long-lived credential at all — the node’s identity is the proof. The External Secrets Operator then syncs from Vault or Secrets Manager into K8s Secrets automatically on rotation.

Rolling deploys are also cleaner: update the Secret, trigger a rollout, old pods drain before new ones start.

architecture mapping

The Docker Compose stack maps to Kubernetes like this:

HOST_APP_DIR in Kubernetes

On a bare host, HOST_APP_DIR points to the checkout of the arizuko repo on the host filesystem. The agent container bind-mounts it to read skills from ant/skills/. In Kubernetes there’s no host checkout to bind-mount — skills are baked into the container image at build time.

Set HOST_APP_DIR to the path inside the container where skills land, typically /workspace or wherever the Dockerfile puts them. The agent is the container; the bind-mount becomes a no-op path reference to the image’s own filesystem. Check your arizuko-ant Dockerfile for the exact path.

HOST_DATA_DIR should still point to the PVC mount path (/srv/data/arizuko_<name>) — that part is unchanged.

step 1 — PersistentVolumeClaim

One RWO volume holds the entire data dir. Size it generously; SQLite and media attachments grow over time.

# pvc.yaml
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: arizuko-data
spec:
  accessModes:
    - ReadWriteOnce
  resources:
    requests:
      storage: 20Gi

step 2 — ConfigMap

Non-secret config. Values here are visible in kubectl get configmap.

# configmap.yaml
apiVersion: v1
kind: ConfigMap
metadata:
  name: arizuko-config
data:
  INSTANCE_NAME: "demo"
  ASSISTANT_NAME: "Ari"
  WEB_HOST: "https://demo.example.com"
  CONTAINER_IMAGE: "arizuko-ant:latest"
  HOST_DATA_DIR: "/srv/data/arizuko_demo"
  HOST_APP_DIR: "/workspace"
  LOG_LEVEL: "info"

step 3 — Secret

Sensitive vars. If you use the External Secrets Operator, this object is generated automatically from Vault or Secrets Manager — see secret management below. Otherwise create it manually:

# secret.yaml  (base64-encode every value)
apiVersion: v1
kind: Secret
metadata:
  name: arizuko-secrets
type: Opaque
stringData:
  SECRETS_KEY: "change-me-32-char-random-string"
  CHANNEL_SECRET: "change-me-another-32-char-string"
  AUTHD_SERVICE_KEY: "change-me-per-daemon-service-key"
  TELEGRAM_BOT_TOKEN: "123456:ABC..."   # if using teled
  SLACK_BOT_TOKEN: "xoxb-..."           # if using slakd
  SLACK_SIGNING_SECRET: "..."           # if using slakd

step 4 — Deployment

A single Deployment with one Pod. All daemon containers share the PVC and inherit both ConfigMap and Secret as environment variables. Adapt the containers list to match the adapters you’ve enabled.

# deployment.yaml
apiVersion: apps/v1
kind: Deployment
metadata:
  name: arizuko
spec:
  replicas: 1
  selector:
    matchLabels:
      app: arizuko
  template:
    metadata:
      labels:
        app: arizuko
    spec:
      initContainers:
        # seed the data dir on first run
        - name: init-data
          image: arizuko:latest
          command: ["arizuko", "create", "$(INSTANCE_NAME)"]
          envFrom:
            - configMapRef:
                name: arizuko-config
            - secretRef:
                name: arizuko-secrets
          volumeMounts:
            - name: data
              mountPath: /srv/data

      containers:
        - name: authd
          image: arizuko:latest
          command: ["/usr/local/bin/authd"]
          env:
            - name: LISTEN_ADDR
              value: ":8081"
          envFrom:
            - configMapRef:
                name: arizuko-config
            - secretRef:
                name: arizuko-secrets
          volumeMounts:
            - name: data
              mountPath: /srv/data
          readinessProbe:
            httpGet:
              path: /health
              port: 8081
            initialDelaySeconds: 5
            periodSeconds: 10

        - name: routd
          image: arizuko:latest
          command: ["/usr/local/bin/routd"]
          env:
            # authd's JWKS + token mint, and the runed dispatch target.
            # Containers in one pod share localhost; give each daemon its
            # own port (LISTEN_ADDR) so they don't collide.
            - name: LISTEN_ADDR
              value: ":8080"
            - name: AUTHD_URL
              value: "http://localhost:8081"
            - name: RUNED_URL
              value: "http://localhost:8082"
          envFrom:
            - configMapRef:
                name: arizuko-config
            - secretRef:
                name: arizuko-secrets
          volumeMounts:
            - name: data
              mountPath: /srv/data
          readinessProbe:
            httpGet:
              path: /health
              port: 8080
            initialDelaySeconds: 5
            periodSeconds: 10

        - name: runed
          image: arizuko:latest
          command: ["/usr/local/bin/runed"]
          env:
            - name: LISTEN_ADDR
              value: ":8082"
            - name: AUTHD_URL
              value: "http://localhost:8081"
          envFrom:
            - configMapRef:
                name: arizuko-config
            - secretRef:
                name: arizuko-secrets
          volumeMounts:
            - name: data
              mountPath: /srv/data

        - name: timed
          image: arizuko:latest
          command: ["/usr/local/bin/timed"]
          envFrom:
            - configMapRef:
                name: arizuko-config
            - secretRef:
                name: arizuko-secrets
          volumeMounts:
            - name: data
              mountPath: /srv/data

        - name: onbod
          image: arizuko:latest
          command: ["/usr/local/bin/onbod"]
          envFrom:
            - configMapRef:
                name: arizuko-config
            - secretRef:
                name: arizuko-secrets
          volumeMounts:
            - name: data
              mountPath: /srv/data

        - name: dashd
          image: arizuko:latest
          command: ["/usr/local/bin/dashd"]
          envFrom:
            - configMapRef:
                name: arizuko-config
            - secretRef:
                name: arizuko-secrets
          volumeMounts:
            - name: data
              mountPath: /srv/data

        - name: proxyd
          image: arizuko:latest
          command: ["/usr/local/bin/proxyd"]
          envFrom:
            - configMapRef:
                name: arizuko-config
            - secretRef:
                name: arizuko-secrets
          volumeMounts:
            - name: data
              mountPath: /srv/data
          ports:
            - containerPort: 8080
              name: http

        # add adapter containers here, e.g.:
        # - name: teled
        #   image: arizuko-teled:latest
        #   envFrom: [...]

      volumes:
        - name: data
          persistentVolumeClaim:
            claimName: arizuko-data

step 5 — Service

Expose proxyd’s port. Use a LoadBalancer for cloud clusters; an Ingress for shared clusters where TLS termination is already handled upstream.

# service.yaml
apiVersion: v1
kind: Service
metadata:
  name: arizuko
spec:
  selector:
    app: arizuko
  ports:
    - name: http
      port: 80
      targetPort: 8080
  type: LoadBalancer

If your cluster has an Ingress controller (nginx, Traefik, etc.), use type: ClusterIP here and add an Ingress resource with TLS. proxyd can also handle TLS itself — set PROXYD_TLS_CERT / PROXYD_TLS_KEY and expose port 443 instead.

step 6 — apply and verify

kubectl apply -f pvc.yaml
kubectl apply -f configmap.yaml
kubectl apply -f secret.yaml
kubectl apply -f deployment.yaml
kubectl apply -f service.yaml

# watch the pod come up
kubectl get pods -w

# check routd is ready
kubectl logs deploy/arizuko -c routd --tail=30

# hit the health endpoint
kubectl port-forward svc/arizuko 8080:80 &
curl -s http://localhost:8080/health

secret management with External Secrets

The External Secrets Operator syncs a K8s Secret from an external store on a schedule. When you rotate a secret in Vault or Secrets Manager, the operator updates the K8s Secret automatically, and the next pod restart picks it up.

Install the operator once per cluster:

helm repo add external-secrets https://charts.external-secrets.io
helm install external-secrets external-secrets/external-secrets -n external-secrets --create-namespace

Wire it to your store. This example uses AWS Secrets Manager with IRSA (the pod’s ServiceAccount is annotated with an IAM role that has secretsmanager:GetSecretValue):

# secretstore.yaml
apiVersion: external-secrets.io/v1beta1
kind: SecretStore
metadata:
  name: aws-secrets
spec:
  provider:
    aws:
      service: SecretsManager
      region: us-east-1
      auth:
        jwt:
          serviceAccountRef:
            name: arizuko   # annotated with IAM role ARN

# externalsecret.yaml
apiVersion: external-secrets.io/v1beta1
kind: ExternalSecret
metadata:
  name: arizuko-secrets
spec:
  refreshInterval: 1h
  secretStoreRef:
    name: aws-secrets
    kind: SecretStore
  target:
    name: arizuko-secrets   # creates the K8s Secret
  data:
    - secretKey: SECRETS_KEY
      remoteRef:
        key: arizuko/demo
        property: SECRETS_KEY
    - secretKey: CHANNEL_SECRET
      remoteRef:
        key: arizuko/demo
        property: CHANNEL_SECRET
    - secretKey: TELEGRAM_BOT_TOKEN
      remoteRef:
        key: arizuko/demo
        property: TELEGRAM_BOT_TOKEN

For HashiCorp Vault, swap the provider block for a Vault provider and point at your KV path. For GCP Secret Manager, use the GCP provider and Workload Identity instead of IRSA. The ExternalSecret shape stays the same.

Once this is running, you never create secret.yaml manually again. The operator manages the K8s Secret object; the Deployment reads it; rotation is a push to Vault + a pod restart.

agent containers (known gap)

runed spawns a per-group arizuko-ant container for each agent run. On bare Docker it calls docker run via the Docker socket. In Kubernetes the socket isn’t reliably available, and even if it is, containers spawned that way land outside the pod’s namespace.

A CONTAINER_BACKEND=kubernetes mode is planned that will create a Kubernetes Job per agent turn instead, inheriting the same Secret and ConfigMap. It isn’t shipped yet. Until then:

• Docker-in-Docker — add a dind sidecar container (image: docker:dind) and mount its socket into the runed container as /var/run/docker.sock. Works; not recommended for production because DinD requires privileged: true.
• KIND — replace DinD with a KIND node for dev and test environments. Same privilege requirement, but easier to snapshot.
• Skip agents — if you only need the message routing and channel adapter layer without the agent, the rest of the stack runs fine. Route messages to a #observe route or handle them via webhooks.

env vars reference

troubleshooting

Pod stuck in Pending

Usually the PVC isn’t bound. Check kubectl get pvc — if it’s Pending, the StorageClass may not support RWO or no provisioner is running. On a single-node cluster, use local-path provisioner (Rancher) or set a hostPath PV manually.

routd crashloops on startup

Usually the data dir isn’t seeded. Check that the initContainer ran and exited 0. If arizuko create failed, it logs why — missing env var, permission error, or existing corrupt DB. Check kubectl logs deploy/arizuko -c init-data.

Adapter can’t reach routd

All containers in the same Pod share localhost. Confirm ROUTER_URL is set to http://localhost:8080 (routd’s port in this manifest) in the adapter’s env, and that adapters wait for routd and authd to be ready.

Secrets not updating after Vault rotation

External Secrets syncs on refreshInterval. After sync, the K8s Secret is updated but the running pod still has the old env. Restart the pod: kubectl rollout restart deploy/arizuko. To automate this, add the Reloader controller, which triggers a rollout when a watched Secret changes.

Agent containers don’t spawn

See the agent containers section above. The CONTAINER_BACKEND=kubernetes path isn’t shipped yet; use DinD or skip agents until it is.