Hermes-agent has six layers of error recovery inside a single worker loop. Fuzzy tool-name matching. JSON schema repair. Context compression. Credential rotation. Token refresh. Stream reconnect. It's the best resilience engineering I've seen in an open-source agent. My own system, nanika, ships with zero of those inside the worker. Zero. A worker runs, produces output, dies. The retries live one layer up, in the coordinator. Both work. Both are wrong for the other's workload. Here's how to pick.
Six layers is one answer, not the answer
I spent a week reading hermes' recovery code and trying to reproduce its layers in nanika. It didn't fit. Every time I added a retry inside a worker, something else broke. The coordinator couldn't see which attempt the worker was on. Two workers fought over the same API credential. A worker that was supposed to die at the end of a phase started hoarding state because it needed to remember its own retry count.
None of that was hermes' fault. Hermes runs long-lived user conversations. Nanika runs short disposable phases. The resilience pattern that makes one survive for hours is the wrong pattern for a system that wants its workers replaceable every two minutes.
This is the binary decision every agent builder makes, usually without noticing. Resilience goes in the worker, or resilience goes in the coordinator. Pick wrong for your workload and you pay twice. Once in retry cost, once in debuggability. Below is the placement rule I settled on, the failure-mode table I commit to the repo, and the test that catches it when someone on the team puts a retry in the wrong layer six months from now.
Classify the workload before writing a single retry
Before touching code, write down two things about your agent.
Session shape. Long-lived and conversational, with a human typing into a running session for hours? Or short-lived and task-shaped, with one objective, one output file, minutes not hours?
Failure cost. Is a mid-session crash catastrophic, meaning the user loses their turns and their working context? Or is it cheap, meaning the coordinator re-dispatches the phase and no one notices?
Paste the answer above your retry code. Mine reads: short-lived phase, crash costs one re-run, worker can die and the coordinator picks up. A long-lived chat agent's note would read: mid-session crash costs the user 40 turns of context, worker must survive recoverable errors in-process. These two notes produce different architectures. Don't write retry code without one.
Skip cost. You'll default to worker-level retries because that's what every tutorial shows. Then your CI pipeline spends an hour retrying inside a crashed process that the orchestrator could have replaced in ten seconds. You won't know why until you trace it.
Draw the failure-mode table. Every row picks a side.
List every failure mode your agent can hit. Start from this and add yours.
- Invalid tool name
- Malformed JSON arguments
- Context window overflow
- Rate limit (429)
- Auth token expired
- Stream disconnected mid-response
- Tool call times out
- Dependency flaked (database, external API)
- Model returned gibberish
- Prompt injection detected
Two columns: WORKER and COORDINATOR. For every row, one X. No blanks, no both-columns. The placement rule is this:
If the failure is session-local, meaning fixable with information already inside the current conversation, it belongs in the worker. If the failure is environmental, meaning it needs resources outside this session (a different credential, a new phase, a human), it belongs in the coordinator.
Commit the table to your repo. Mine lives at docs/resilience-placement.md. When a pull request adds retry logic, the review asks one question: which row in the table is this, and is the code in the right layer for it?
Skip cost. You'll put credential rotation inside the worker because "it's a retry, retries go in the worker." Now your worker holds a credential pool. That means it needs shared state. That means your workers can't be disposable. You just lost the main payoff of a multi-agent system and you didn't notice.
Three placement rules
Use these to break ties when the table feels ambiguous.
Rule 1. Token-bounded repair belongs in the worker. JSON fixes, tool-name fuzzy match, compression of middle turns. The information needed to recover is already in the context. The worker has it. Fixing it costs one retry round-trip. Don't kick this to the coordinator.
Rule 2. Anything needing external state belongs in the coordinator. Credential rotation. Key cooldowns. Cost caps. Billing ceilings. The worker shouldn't know these exist. The coordinator should.
Rule 3. Anything that could loop forever belongs in the coordinator. If the recovery logic has a "retry N times then give up" clause, that N and the give-up path both belong one layer up. Workers shouldn't be allowed to decide "I'll try forever." That decision needs a grown-up in the room.
Point at any retry in your code. Name which rule put it there. If a reviewer asks "why is this in the worker?", your answer is rule 1, 2, or 3. Not "it felt right."
Skip cost. You let the worker own "retry on 429 with exponential backoff." Six months later you add a second agent that shares the API key. Now two workers race on the same credential. Both back off at different times. Both think they're the only one. The cooldown logic has to move to the coordinator anyway, and it's ten times harder to extract under production load than it would have been to place correctly on day one.
Observable retries, budgeted in the unit they pay in
Whichever layer owns a retry, that layer has to emit a structured event when the retry fires. Workers log retry.kind=json_repair, attempt=2, ok=true. Coordinators log phase.retry.kind=rate_limit, worker_died=true, new_credential=idx_3. Pipe both streams to the same sink. One grep tells you which layer caught the failure. A retry storm shows up as a row count, not as mysterious latency. You can compare retry cost per layer per week and make an honest call about which one is actually earning its complexity.
Budget each layer in the unit it pays in. Worker retries cost tokens. Coordinator retries cost phases. Each phase means a new worker spawn, fresh context, lost prefix cache. Set two numbers in your config:
worker.retry_token_budget: max tokens per retry times max retries per failure mode. If JSON repair can burn 3000 tokens across 2 retries, that's your ceiling. Hit it, stop, propagate the error up.coordinator.phase_redispatch_budget: max phase re-dispatches per mission. If a phase has re-dispatched three times, don't dispatch a fourth. Surface it as a blocker.
Skip cost. Unlimited retries in either layer burn money while looking like the system is working. A stuck worker retrying JSON repair in a loop consumes more tokens than the original task did. A coordinator re-dispatching a poisoned phase burns workspaces until the disk fills.
Test the placement by injecting the failure
For every row in your failure-mode table, write one integration test that injects the failure and asserts which layer catches it.
test: when a worker receives a malformed JSON tool arg,
the worker retries and recovers, no phase re-dispatch.
test: when the only API credential returns 429,
the coordinator rotates to a new credential and
re-dispatches the phase. Worker died cleanly.
No in-worker rotation happened.The test suite verifies placement, not just outcome. A refactor that accidentally moves retry logic to the wrong layer turns the corresponding test red. Someone adds "helpful" retry code in the worker six months from now, and the test catches it before the pull request merges.
Skip cost. Placement rot. The boundary between worker and coordinator blurs. You're back to the layer cake you were trying to avoid, and every new retry adds a bit more sediment.
The third layer nobody names
Both hermes and nanika put resilience in exactly one place. Worker or coordinator. But there's a third layer most agent systems don't name out loud: the user.
When you surface a failure to a human and say "retry?", you're delegating resilience to the person at the keyboard. Cheap for you. Expensive for them. Sometimes it's honest. The failure is genuinely unrecoverable by either machine layer, and pretending otherwise would just mean a worse retry done opaquely. Sometimes it's a copout. You didn't want to decide where the failure belonged, so you bubbled it up and made the user carry it.
I don't have a clean rule for when that's fine and when it's lazy. The honest answer probably depends on whether your agent is a tool or a product. A tool can ask its operator to fix things, and the operator signed up for that. A product that asks its users to fix things is one that hasn't finished building itself.
Here's where I land. Put worker-level retries behind token-bounded repair. Put coordinator-level retries behind anything that touches external state or anything that could loop forever. Log every retry in the same sink so you can see which layer actually catches production failures. Test the placement with injection so the boundary doesn't rot. And when you notice yourself surfacing a failure to the user, ask whether you're respecting their time or dodging the decision.
Six layers is an answer. Zero layers is an answer. The real question is whether you picked, or whether you inherited.