Seven verified gotchas in a pause/resume-native workflow framework's primitives (checkpoint, resume_inputs, compaction, disconnect)

A graph-based agent framework that can pause mid-run and resume later from persisted state (ADK 2.0's Workflow, as opposed to a plain LLM agent) exposes a small set of primitives — a rerun-on-resume decorator, a resume_inputs map, history compaction, a synchronous run-inside-request mode — and every one of them has a non-obvious edge that only shows up under real pause/resume traffic, never in a happy-path demo.

The signal that gives it away

You're looking at one of these seven, not a generic "the agent is flaky" bug, when:

1. A node that was supposed to come back to life after an external event (human answer, background job callback) never re-enters its own body — the interrupt was recorded, the answer arrived, but nothing downstream happened. 2. A pipeline restarts an expensive first step from scratch every time the user answers a gate, instead of continuing past it — as if the framework had no memory of what was already checkpointed. 3. A workflow that was paused waiting on a human hangs forever after a history-trimming feature was turned on, with a ValueError about a missing function call buried in the logs. 4. A step that runs synchronously inside the request/response cycle dies with a "was cancelled" log line the instant a user closes the tab or the connection blips — even though the work itself would have finished fine. 5. A loop that asks the same kind of question every round stops making progress after round one: it looks answered, but the node is actually replaying the very first round's answer forever. 6. A value pulled out of the resume state is not the string you expect — code that calls .strip() or does string math on it blows up two lines away from where the value actually originated, or (worse) fails silently inside a broad except and just degrades. 7. A "wrap it in try/except to be safe" instinct, applied at the wrong scope, turns off the framework's own ability to pause — the workflow behaves as if HITL were never wired in, with no error pointing at why.

1. The rerun-on-resume decorator is not optional metadata, it's the resume contract

Every node the framework may wake up from an interrupt must declare that it wants to be re-entered from the top on resume, not just "notified." In the installed google-adk==2.0.0 source, the flag defaults to False on the base node type (google/adk/workflow/_base_node.py:56), and the framework's own docs for the node with auth are explicit that skipping it is a configuration error, not a style choice ("FunctionNode with auth_config requires rerun_on_resume=True" — google/adk/workflow/_function_node.py:197-211). Field-verified consequence: without it, a resumed interrupt never re-enters the node body at all — the answer sits recorded, the node just never looks at it again. The corollary that bites people who *do* set the flag correctly: because the node body re-runs from line one every time, the body must be idempotent on re-run — anything with a side effect (a dispatch, a write) has to be guarded by checking state first, not assumed to run "only once" just because the code only appears once in the source.

2. A gate's answer is invisible to a node once a *fresh* invocation starts — and free text is what triggers one

The framework resolves whether an incoming message *resumes* an existing paused run or *starts a new one* by trying to match its function-response ids against function-call events already in history (google/adk/runners.py:670-706, _resolve_invocation_id_from_fr); a plain text message has no function-response id to match, so it always gets a brand-new `invocation_id`. That single fact cascades: the node-state rehydration step that a dynamic node relies on to recover "what have I already done" explicitly skips every event that doesn't belong to the current invocation_id (google/adk/workflow/utils/_rehydration_utils.py:203, if invocation_id and event.invocation_id != invocation_id: continue). So a stray free-text message — someone typing a comment instead of answering the pending gate — doesn't just fail to resume; it hands the node a *clean slate*, and any already-checkpointed step that was gated on resume_inputs.get(id) is None: dispatch sees None again and re-dispatches. Field-measured blast radius: 8 duplicate dispatches of the same expensive job within 50ms, on a single stray message, before a fix was in place. The framework's own checkpoint/rerun primitive is *not* the idempotency mechanism you need here — it protects against re-running mid-invocation, not against a fresh invocation with no memory of the last one. The actual fix is an external, durable guard keyed by session/step that survives invocation boundaries — see Idempotency guard with Redis SET NX — fire the dispatch once, never on retry-count alone for exactly how to build it, and don't try to solve this one at the framework-primitive layer, because there isn't a knob for it.

3. History compaction can quietly delete the very event a paused gate needs to find on resume

A feature that summarizes/trims old events to keep context small is, by itself, a reasonable default. The trap is specific: if compaction removes a function_call event that corresponds to a still-open interrupt, the resume path can no longer find it. The exact failure mode is verifiable in the installed source: resuming walks reversed(session.events) for a function-call whose id matches the incoming function-response id, and raises ValueError('Function call not found for function response ids: {fr_ids}.') if it never finds one (google/adk/runners.py:687-700) — that literal error text is what shows up in logs when compaction has already summarized the call away. Field-verified live: a workflow paused on a long-running gate hung after compaction removed its pending call from the DB, with exactly that error on resume. If your pipeline's event count per session is small (tens, not thousands), the honest fix is simpler than tuning the compactor around this edge case: turn compaction off for that app and revisit only if event volume actually becomes a cost problem. Compaction's intersection with session persistence is fragile in more than this one way: google/adk-python issue #3530, for instance, reports a *different* failure in the same neighborhood — compacted events failing to deserialize correctly on reload (an AttributeError from code expecting an object but getting a plain dict) — which is not the same bug as a still-open interrupt's function-call event being trimmed away, but is further evidence that compaction plus persisted, resumable state is a combination worth treating with suspicion rather than assuming well-tested.

4. A synchronous sub-node dies the instant the client disconnects — long-running work needs the external-resume path, not `run_node`

A node executed inline (ctx.run_node, no external pause) shares fate with the HTTP request/response task that's driving it. If the browser disconnects — tab closed, network blip, page refresh — most server frameworks cancel the underlying task, and that cancellation propagates straight up through the running node. The exact log line is reproducible from the installed source: the root-run loop catches asyncio.CancelledError and logs 'Root node %s was cancelled.' (google/adk/runners.py:801-802) — verified live, character-for-character, on a production pause point. The practical rule this earns: anything long enough that a user might plausibly navigate away during it does not belong on the synchronous `run_node` path, no matter how convenient inline execution is to write. It belongs on the external dispatch-then-RequestInput-then-resume path instead — the same one already proven for other long-running steps — which survives a disconnect because the pause is durable, not tied to a live request. A common fix: move any such step (for example, a slow analysis call) off run_node and onto this pattern; the fix is mechanical once the cause is clear, but invisible without the disconnect-specific log line to point at it.

5. Reusing the same interrupt id across loop rounds gets you the *previous* round's answer, not a fresh prompt

This is the subtlest one, and it inverts what the framework's own API doc seems to promise. RequestInput.interrupt_id's docstring in the installed source says explicitly: *"Reusing the same interrupt_id across loop iterations (e.g. a rejection/retry cycle) is supported — the framework matches function calls and responses by count"* (google/adk/events/request_input.py:44-50). That claim is true for what it covers — resolving *which* invocation a reused-id response belongs to. It does not cover a second, separate mechanism: a dynamic node's resume_inputs map is loaded from that node's own persisted per-run state on every resume (dict(run.state.resume_inputs) if run.state.resume_inputs else None, google/adk/workflow/_dynamic_node_scheduler.py:472-474) and written back after each run (run.state.resume_inputs = result.resume_inputs, same file, line 369) — i.e. it accumulates for the entire lifetime of that node's run, not just the current pause-point. So if a loop node emits RequestInput with the *same* id on round 2, and checks ctx.resume_inputs.get(GATE) is None to decide whether to wait, that check comes back False immediately — not because round 2 was answered, but because round 1's answer is still sitting in the accumulated dict under that same key. The loop reads the stale answer and either stalls or silently reprocesses old data instead of the new round's input. Field-verified fix, applied to a multi-round domain-selection loop: give the interrupt id a per-round suffix (f"{GATE}:{round}"), never the bare constant, for any gate that can legitimately fire more than once in a node's lifetime. The review-gate and product-gate patterns elsewhere in the same codebase already used this {base}:{n} shape for the unrelated reason of avoiding literal loops on re-ask — it turns out to be the same fix for a different bug.

6. What comes back out of resume state is whatever type your own bridge put in — not necessarily a string

function_response.response is an open-ended payload (google/adk/runners.py:651-652 just stores whatever object the response part carried, keyed by id) — the framework does not coerce it to any particular shape. If the code that constructs the resume message wraps a human's plain-text answer in a typed SDK object (for example, a google.genai.types.Content) instead of handing over a plain string or a small dict, then ctx.resume_inputs.get(interrupt_id) on the far side hands back that typed object. str(Content) produces its dataclass-style repr ("parts=[Part(text='…')] role='user'"), not the text — so a .strip() or any string operation either raises AttributeError outright, or, if it's caught by a broad except, silently degrades into a low-confidence/garbage path with no visible error. This bit every user-answered gate in one codebase at once (an item-selection gate, two review gates, a data-filter gate) because they all called the same downstream classifier on the raw resume value. Fix: one coercion helper, called at the single point where a resume value enters application code, that tries .parts[].text, then .text, then dict-key "text", then falls back to str() — never assume the shape, always normalize once at the boundary. This is the same family of bug as An LLM occasionally collapses a nested dict into a plain string — and the real shape it produces differs from the shape a downstream reader assumed and The LLM returns a string where you expected a number — the error explodes two lines later, not where it's born — a producer's actual output shape drifting from what the consumer assumed — just triggered by the resume bridge instead of an LLM response.

7. A broad `except` in the wrong place swallows the framework's own pause signal

The mechanism a dynamic node uses to tell its parent "I've hit an unresolved interrupt, please pause" is, deliberately, not a normal exception. In the installed source it's class NodeInterruptedError(BaseException) (google/adk/workflow/_errors.py:20) — subclassing BaseException, not Exception, specifically so that an ordinary except Exception: does not intercept it; only a bare except: or an explicit except BaseException: will. The framework itself catches it at exactly one deliberate point right after the node's run loop (google/adk/workflow/_node_runner.py:240-249) and treats it as "nothing more to do, the caller reads the interrupt ids off context" — a real control-flow signal, not an error. If application code between a ctx.run_node() call and that framework catch wraps things in except BaseException: (a pattern people reach for as "maximally safe" when hardening a migration or a bulk try/except cleanup), it intercepts the pause signal first and the framework never sees it — the workflow doesn't raise, doesn't log anything alarming, it simply behaves as if the interrupt had been silently resolved with nothing, and HITL for that path stops working with no error to point at. The rule this earns, verified against the installed source rather than assumed from folklore: never catch `BaseException` in code that sits on a path a dynamic node's `run_node` calls can traverse, and treat except Exception in the same neighborhood with the same suspicion, since it also disables the framework's own retry primitive for synchronous failures on that path.

Bonus, same family: task-mode delegation and external long-running resume do not compose (verified negative, not folklore)

Separate from the seven above, but the same "framework primitive has a documented gap, verify before you build around it" spirit: a mode='task' sub-agent (auto-delegation under a coordinator) that also contains a LongRunningFunctionTool paused and resumed via the same external /resume bridge does not work in this framework version — a live spike reproduced a deterministic crash on resume — a ValueError whose message indicates the named tool could not be found, because the resume path re-enters the task agent's own tool scope and can't re-resolve the delegation call that only exists in the coordinator's scope. This is not a private bug: the same class is a documented, open gap upstream — google/adk-python discussion #3348 shows the ADK team acknowledging that SequentialAgent *should* resume via a saved current_sub_agent marker but in practice re-executes, with the community's working fix being a custom orchestrator that checks completion-keys in session state before calling each sub-agent instead of relying on the built-in resume. The docs for the resume feature separately warn that tools run "at least once, maybe more" on resume (idempotency is your job) and that there is explicitly "no routing for nested AgentTool / task-mode delegation recovery." Practical takeaway: if you need both auto-delegation and external long-running resume, don't build it as task-mode-under-a-coordinator — implement the channels/verbs as gated graph nodes on the *proven* resume path instead (dispatch → RequestInput → external resume, matched by function_call_id == interrupt_id), and keep "which channel runs" as a plain state-gated routing decision, not a nested delegation.

What does NOT work

- Assuming a decorator/flag is "just for clarity" and skipping it under time pressure. rerun_on_resume looks optional because nothing crashes when it's missing — the node just quietly never comes back on resume. There's no exception to catch; the only signal is "this step never seems to finish." - Trusting the framework's own checkpoint/replay to protect against duplicate dispatch across a fresh invocation. It protects re-runs *within* an invocation; a fresh invocation (triggered by something as ordinary as a stray text message) has no memory of the old one by design, not by bug. - Leaving compaction on "because it's a sensible default" without checking whether any pending interrupt's function-call event could fall inside the trimmed window. The failure only shows up for a session that happens to have a gate open across the trim boundary — it will pass every test that doesn't leave a gate open that long. - Running an expensive/slow step on the synchronous `run_node` path because it's simpler to write than the dispatch/resume dance. It works in every manual test where you don't touch the browser tab, and fails exactly the first time a real user does what real users do (navigate away, refresh, lose network). - Trusting a framework API doc's blanket "X is supported" statement as covering every layer your own code touches. The interrupt_id reuse docstring is accurate about what it claims (function-call/response resolution) and silent about a different, adjacent persisted-state mechanism that also cares about that same id. Read the doc's claim narrowly; verify the actual behavior for *your* usage pattern before relying on it. - Adding a defensive broad `except` around code that might call into a dynamic node, "just to be safe." This is the single most counter-intuitive item on this page: hardening code this way is precisely what disables a native safety feature (the pause signal) rather than adding one. If you need a safety net around this kind of code, catch narrow, known exception types explicitly — never BaseException, and treat blanket except Exception in the same neighborhood as suspect too. - Debugging any of the above by adding more logging inside your own node body. Every one of these seven produces silence *before* your node code runs again (2, 3), *instead of* your node code raising (7), or *around* your node's synchronous call rather than inside it (4) — the fix in every case was reading the framework's own log line or source, not adding print statements to application code that never got a chance to execute.

Related

- Idempotency guard with Redis SET NX — fire the dispatch once, never on retry-count alone — the external, invocation-independent guard that closes gotcha #2's re-dispatch window. - Answering a gate with text + functionResponse together silently breaks Workflow resume — a distinct but adjacent resume-message trap: mixing a text part with a function-response in the *reply* to a gate, on the very same Workflow resume path these gotchas live on. - No long-running agent framework handles failure for you — you need an explicit error contract — why a paused long-running step also needs an explicit failure contract; this page is about the pause/resume mechanics working at all, that page is about what happens when the thing you were waiting for failed. - A long-running tool's follow-up turn is generated server-to-server and never streamed to the browser — the UI needs an F5 to show what already happened — a different silent failure in the same long-running family: resume succeeds mechanically but the result never reaches the live UI. - A resumed old session redoes expensive work from scratch — the idempotency guard was reading a short-TTL artifact — a companion failure where the idempotency guard from #2 exists but expires too early. - An LLM occasionally collapses a nested dict into a plain string — and the real shape it produces differs from the shape a downstream reader assumed and The LLM returns a string where you expected a number — the error explodes two lines later, not where it's born — the wider family that gotcha #6 belongs to: a producer's real output shape drifting from what the consumer assumed. - A blanket `except Exception` swallows an ImportError and degrades into a misleading business error — the sibling case of gotcha #7: a broad except hiding a different class of signal (a programming error) instead of a framework control-flow signal.