arizuko › concepts › retries

retries

Every turn runs in its own throwaway container. Sometimes that container dies mid-thought — it runs out of memory, or it hits the run time limit — and nothing comes back. routd notices the empty-handed run and simply runs the turn again, up to three times, without telling you. You get one slightly slower reply instead of silence.

the problem it fixes

A killed container never reaches the point where it hands its answer back. From your side the bot just stops. Worse, routd has a guard that stops it re-feeding a turn it has already marked finished, so the message never gets another chance — the only way out was to @mention the bot again and hope. The incident that motivated this was a long research answer that got cut off and then stayed cut off.

what counts as failure

The test is “did any bot message land?”, not “did the agent finish cleanly?”. An agent that posted half an answer and then died counts as a success and is not retried — a partial reply you can read beats making you wait while the whole thing is redone. No bot message at all is the failure that triggers a retry.

what you actually see

Nothing, until it works. Each attempt waits 10 seconds and then spawns a fresh container. There is no exponential backoff: the old container is already dying and the new one starts from scratch, so there is nothing to wait for except the reaper, and a long ramp would only stretch out your silence.

If every attempt fails, one message finally arrives:

⚠ Agent couldn't complete this request after 3 attempts.

The chat is also flagged as errored so an operator can find it on the dashboard. A turn that failed without ever being retried — because the agent had already replied — gets the plainer notice instead: Failed: agent error on that message. Try rephrasing or send a different message.

the retry knows it is a retry

Each re-run gets a note prepended to the prompt before the agent sees it:

<system-note>This is retry attempt 2 of 3. The previous attempt was
killed before completing. Be conservative with resource
usage.</system-note>

That last sentence is doing the work. The most common cause of a killed container is the agent doing too much at once and running out of memory, and the most useful thing it can change is to do less. Telling it nothing would just get the same crash three times.

What carries over otherwise: the same message and attachments, the same conversation history, a brand-new container.

it survives a restart

The attempt count isn’t held in memory. Each retry bumps turn_context.retry_count and parks the turn in state='pending_retry', and routd’s crash-recovery sweep picks up pending_retry turns exactly as it picks up running ones. Restart routd mid-retry and the turn is still owed an attempt, with the right number of attempts already spent.

what is never retried

• Turns that replied. Any bot message at all, even a partial one, counts as done.
• Turns the agent failed on purpose. If the agent reports its own failure and the container then exits cleanly, that is a clean run — it never enters the retry branch at all.
• Runs an operator killed. A deliberate kill is recorded as killed, not as an error.

tuning it

One setting: MAX_TURN_RETRY, default 3. Set it to 0 to turn retries off entirely and get the failure notice on the first dead run.

It is deliberately instance-wide — there is no per-folder or per-skill override. How many times the runtime is willing to re-spawn a container is a property of the machine you’re running on, not of a tenant’s content. If one folder needs a different budget, the thing to change is that folder’s memory limit, not its retry count.

The give-up notice reports whatever you set: raise the ceiling to 5 and it says “after 5 attempts”.

go deeper

The decisions behind each of these — why success is a bot row and not a clean exit, why the backoff is flat: specs/5/12. The turn lifecycle this hooks into is specs/5/E; the container and its time limit belong to runed. What you see while a turn is still alive: live progress.