arizuko › howto › Operate

operate

A message that gets no reply stops at one of four daemons. Check each daemon in the order the message passes it. The first check that fails names the incident. Each incident below gives what to read, the cause, the fix and the source line.

The journal holds the log of every daemon

The systemd unit runs the generator from the release image, then runs docker compose up in the foreground. Compose writes the output of every container to its stdout, and journald keeps that output. One command reads all daemons:

sudo journalctl -u arizuko_<name> --since "15 min ago" --no-pager

Do not use docker logs. When the Docker daemon sets "log-driver": "none", docker logs writes an error to stderr and nothing to stdout. A | grep -c ERROR after it then prints 0, and the count is wrong.

A message passes four daemons before the reply

Each daemon has one condition that must be true and one place to read it:

platform message
      │
      ▼
┌───────────┐  healthy, "channel registered"    docker ps, journal
│  adapter  │
└─────┬─────┘
      ▼
┌───────────┐  a route answers, cursor moves    /dash/routes/, /dash/activity/
│   routd   │
└─────┬─────┘
      ▼
┌───────────┐  preconditions met, breaker shut  journal: "cannot run turns"
│   runed   │
└─────┬─────┘
      ▼
┌───────────┐  boots, calls back, submit_turn   journal: "turn did not start"
│ KVM guest │
└─────┬─────┘
      ▼
routd ──► adapter ──► platform reply

arizuko send writes a message into the queue of a folder as the operator and prints the reply. It does not use the platform:

./arizuko send -i <name> <folder> "ping" --wait

When this command gets a reply and the platform chat does not, the fault is in the adapter. When this command gets no reply, the fault is in routd, runed or the guest.

Each incident has one log line, one cause and one fix

runed refuses to start

Read

sudo journalctl -u arizuko_<name> | grep -A20 'cannot run turns'

Cause

runed checks every input before it starts a turn. It prints all missing inputs in one message: this host cannot run turns; N precondition(s) missing. The inputs are root, the device nodes /dev/kvm, /dev/vhost-vsock and /dev/net/tun, the qemu, egress, iptables and ip binaries, RUNED_KERNEL_DIR, RUNED_ROOTFS_DIR, the guest memory budget and the tap subnet.

Fix

Correct every line of the message, then restart. sudo make kernel DATA_DIR=/srv/data/arizuko_<name> and sudo make images DATA_DIR=/srv/data/arizuko_<name> build a missing artifact. Set RUNED_KERNEL_DIR and RUNED_ROOTFS_DIR in .env to the directories that the two commands print.

Source

runed/qemuhost.go:117 qemuPreconditions, the message at runed/qemuhost.go:218

runed refuses a guest size that the host memory cannot hold

Read

The precondition message contains RUNED_GUEST_MEM=… × MAX_CONCURRENT_CONTAINERS=… is … MiB, over this host's MemTotal.

Cause

Each concurrent turn is a guest with its own memory. The host must hold the guest size multiplied by the cap. runed refuses this pair at start, before a second turn fails.

Fix

Decrease RUNED_GUEST_MEM, default 2G, or MAX_CONCURRENT_CONTAINERS in .env. Then restart.

Source

runed/qemuhost.go:318 memoryFits

routd stops at start without SECRETS_KEY

Read

The journal shows SECRETS_KEY required: secrets are encrypted at rest, and routd exits with status 1.

Cause

routd encrypts every stored secret with a key that it derives from SECRETS_KEY. routd does not start without the key. arizuko create writes the key into .env. An .env that an operator writes again by hand does not contain it.

Fix

Copy the key of the instance back from the backup of .env. A new key starts routd, but it cannot decrypt a value that the old key encrypted. To change the key, set SECRETS_KEY=new,old. The first key encrypts. The other keys only decrypt.

Source

core/config.go:140, routd/cmd/routd/main.go:69, store/store.go:63

A daemon without a store key restarts the whole instance

Read

systemctl is-active arizuko_<name> prints activating. Two of fifteen containers run. The journal shows dynstore: endpoint "https://extenddb:18443" takes an access key and a secret from the enabled daemon. Before that, arizuko generate printed store enrolment is partial.

Cause

The daemon reads the store and has no key for it. The daemon exits at its first store access. Its exit stops docker compose up, and systemd starts the whole instance again. No arizuko command makes a store key. BUGS entries L319 and L406 record this gap.

Fix

Make the key as [INSTALL § The per-daemon store keys] describes. Then restart the service with sudo systemctl restart arizuko_<name>.

Source

objstore/dynstore/dynstore.go:69, the warning at compose/compose.go:1100

make images and make kernel stop without DATA_DIR

Read

kernel: set DATA_DIR=<instance data dir> or rootfs: set DATA_DIR=<instance data dir>, exit status 1.

Cause

Both artifacts go into DATA_DIR/images/. make kernel checks DATA_DIR first. make images builds every image before its rootfs step checks DATA_DIR, so it fails after the slow builds.

Fix

Run sudo make images DATA_DIR=/srv/data/arizuko_<name>. Docker keeps the image builds in its cache, so the second run starts at the export.

Source

Makefile:57, Makefile:308

Two instances on one host need two vsock windows

Read

The second instance logs vsock bind port 5555: address already in use at its first turn.

Cause

The vsock CIDs and ports belong to the host, not to one instance. Two instances with the default settings both use CID 3 and port 5555 and up. One instance uses MAX_CONCURRENT_CONTAINERS CIDs and two ports for each CID.

Fix

In the .env of the second instance, set RUNED_VSOCK_CID_BASE to the first base plus the cap or more. Set RUNED_VSOCK_PORT_BASE to the first base plus two times the cap or more. Also set a different RUNED_TAP_SUBNET, because runed derives the tap addresses from the subnet and the slot only. No check finds an overlap, so record the windows.

Source

runed/vmrecord.go:208 allocateCID, runed/vmrecord.go:238 vsockPorts, runed/vmctl.go:98, runed/vmnet.go:38

Old rootfs trees fill the data disk

Read

sudo arizuko images -i <name> lists every rootfs and kernel tree of the instance with its size, and marks each tree as in use, mounted or unused. One rootfs tree is about 7 GB.

Cause

A release with a changed agent image exports a new images/rootfs-<hash>/ tree. make images deletes no tree, because a running turn can use an older tree.

Fix

sudo arizuko images -i <name> --prune removes every unused tree. It keeps the trees that RUNED_ROOTFS_DIR and RUNED_KERNEL_DIR name, and it removes nothing while a tree has a mount. Instances on the same arizuko-ant image can use one DATA_DIR for the artifacts, so the export occurs once. docker builder prune -f removes the build cache. Do not run docker image prune -a: no container holds arizuko-ant between turns, so the prune deletes it, and the next rootfs export fails.

Source

cmd/arizuko/images.go runImages, compose/compose.go:1237 hostViewPaths

whapd stays up without a WhatsApp session

Read

The journal shows session invalidated; staying up unpaired — re-pair through /v1/pair, and the whapd /health endpoint returns 503.

Cause

WhatsApp ended the linked-device session. whapd does not exit, because the pairing endpoint is in the whapd process.

Fix

Open /dash/channels/whatsapp/pair and enter the phone number of the bot. On the phone, open Settings › Linked Devices › Link a Device › Link with phone number instead, and type the 8-character code. The code is valid for 60 seconds. whapd accepts 5 pairing starts in one hour.

Source

whapd/src/main.ts:320, dashd/main.go:578, [whapd README]

A second bot without its own token reads the first account

Read

Two services answer as one bot, or the second service sends each reply again.

Cause

A second account is its own fragment, <adapter>-<label>.yml, with its own CHANNEL_NAME. The fragment uses the keys of the base adapter. Only a <KEY>_<LABEL> line in .env replaces a key. Without that line, the second service reads the base account.

Fix

Set the override, for example TELEGRAM_BOT_TOKEN_JYNX for teled-jynx.yml. Run arizuko generate, then restart. Do not add a second token to the base fragment.

Source

compose/compose.go, [compose README]

A changed fragment does not get release fixes

Read

arizuko generate prints services/<name>.yml differs from the catalog and is FROZEN and lists the template lines that the file does not have.

Cause

arizuko generate replaces an installed fragment with a link to the template only when the two files are identical. After a change by hand, also to a comment, generate keeps the file. The file then does not get a new mount, env var or timeout from the template.

Fix

Move the difference into .env. .env holds STORE_HOST_PORT and the <KEY>_<LABEL> overrides. Copy the template over the changed file. The next arizuko generate replaces it with a link.

Source

compose/fragments.go:62 RelinkCatalog, cmd/arizuko/packages.go

An adapter that fails its healthcheck stays at "starting"

Read

sudo docker ps --format '{{.Names}}\t{{.Status}}' | grep -E 'starting|unhealthy', then sudo journalctl -u arizuko_<name> --since "10 min ago" | grep -E 'channel (registered|deregistered)'.

Cause

A container in a crash loop shows Restarting. An adapter that never passes its healthcheck shows health: starting with no restarts. The usual cause is a credential that the platform refuses. routd then removes and adds the channel again and again.

Fix

Find the refusal of the platform in the adapter lines of the journal. Put the new credential in .env, run arizuko generate, then restart. When the deregister count is near the register count, the adapter is not stable.

Source

chanlib/chanlib.go:180, chanreg/chanreg.go:203

A chat without a matching route gets no reply

Read

/dash/activity/ shows the newest message in and out of the chat. The unrouted section of /dash/routes/ lists every chat that routd drops. ./arizuko route -i <name> list prints the route table by seq.

Cause

When no route matches, routd marks the messages as seen and starts no turn. An #observe route above the answering route has the same result: the first match wins, and an observe route has no verb, so it takes every message, mentions included. BUGS entry L327 records this. The cursor moves, the breaker stays closed, and no error occurs.

Fix

Add the route, or give the answering route a lower seq than its #observe pair. A route row is operator data: change it with arizuko route.

Source

routd/loop.go:565 routeMiss, dashd/routes_admin.go:105

A backup comes before a large change

[INSTALL § A tar or a dump restores an instance] gives the tar of a stopped instance and the arizuko dump and arizuko apply commands.