No long-running agent framework handles failure for you — you need an explicit error contract

A graph-based agent workflow with async long-running tools (dispatch → external job → resume) has no built-in primitive that turns "the external job failed" into "the run stays alive and the user can fix it" — you have to build that contract yourself, and every node has to remember to use it.

The signal that gives it away

You're looking at this exact gap when a step:

- shows "in progress" / "running" forever in the UI after an external job (queue worker, webhook callback, background task) has already failed — no exception was ever raised, the job just never called back into the paused node; - or completes with success: True and a happy final message even though the underlying job produced zero of whatever it was supposed to produce (zero rows enriched, zero renders started, zero items processed) — because nothing downstream ever explicitly checked the count, only the absence of an exception.

Both symptoms share one root cause: in a framework whose long-running mechanism is "dispatch now, resume later via an external callback carrying a FunctionResponse", failure is data ({success: False, error: "..."} arriving through the normal resume channel), not a Python exception. Anything built assuming "no exception ⇒ success" is blind to it.

How you verify the framework really has no native answer

Before building a manual contract, check the installed source, not just the marketing docs — framework behavior around retries/resume is exactly the kind of thing that changes between minor versions. On an installed google-adk==2.0.0, none of the built-in primitives cover the dispatch-to-external-worker / resume-via-FunctionResponse shape:

| Primitive | Why it doesn't apply | Where to check | |---|---|---| | RetryConfig(max_attempts) on a node | Retries synchronous, in-process exceptions. A failure arriving as an external FunctionResponse after a resume is not an exception, so it never triggers a retry. On top of that, the attempt counter lives in the execution loop's memory and is explicitly not persisted across a pause/resume round-trip — after a resume the counter restarts at 1, so a "3 attempts" config silently becomes "3 attempts per resume, forever." | node runner internals; independently confirmed on current docs (see provenance) | | A node that raises on failure | Kills the run (or at minimum forces heavy-handed session-level error handling) instead of leaving it alive for the user to fix the precondition and retry. This is the opposite of what you want for a recoverable failure like "external token expired." | node execution / error-event path | | The long-running tool wrapper itself | Typically has no error semantics at all — it is a thin adapter that forwards whatever your function returns as sequential responses. {success: False} and {success: True} are structurally identical to it; distinguishing them is 100% your own responsibility. | the tool wrapper's source, it's short — read the whole thing once | | A "reflect and retry" tool plugin | Scoped to tool calls issued by an LLM agent (it intercepts a tool's error response and feeds guidance back to the model to retry). It has no hook into a deterministic graph node's failure — inapplicable if your long-running step lives in a Workflow graph rather than being an LLM-driven tool call. | plugin docs | | A generic node-level error event | Signals "this run/node just died," which is a different thing from "recoverable, please retry" — it's the wrong shape of signal to build a recovery UI on. | node runner internals |

Practical rule this earns you: before you build a workaround around a fast-moving framework's gap, re-check the source of the version you actually have installed — an aspirational comment in your own code ("we'll add RetryConfig here") that was never wired up is not a safety net, it's a false sense of one. grep your codebase for the retry primitive's name before assuming it's active anywhere.

How you build the contract

Since nothing native fits, the fix is a single manual contract, applied at exactly two points per long-running step — dispatch and resume — never left to per-node discretion:

1. One guard function, not one per node. It takes whatever result came back (from the initial dispatch call *or* from the resume callback) and checks it explicitly for failure — never assume "no exception raised" means success. For a job that produces a *quantity* of something, the guard also has to check the count is non-zero, not just that the call returned 200/success — a call can return "success" with zero units of work actually done. 2. On failure, the guard emits a live, actionable pause — the native idiom to reuse is "ask for more input, but let the node re-run when resumed" (i.e., don't kill the run; surface a request for input carrying a payload the frontend can render as an action). The payload should carry: what's wrong, what fixes it, and how to signal "retry." Do not raise. 3. On success, the guard is a no-op — the design goal is that adding it costs nothing to the happy path, so there's no incentive to skip it under time pressure. 4. Give the retry a separate gate/state from the original in-flight marker. If "waiting for job" and "job needs a fix + retry" share one flag, a retry attempt can look like a duplicate in-flight job to your own idempotency guard and get silently dropped instead of re-dispatched. See Idempotency guard with Redis SET NX — fire the dispatch once, never on retry-count alone for why a naive re-dispatch guard bites here specifically. 5. Wrap every unguarded call that can raise inside the dispatch/resume path (an HTTP call with an unchecked raise_for_status(), for instance) so an external 401/5xx becomes the same actionable-failure shape instead of crashing the node outright — a crash here is strictly worse than a stuck spinner, because it can kill the whole run with no recovery path at all. 6. The frontend needs an explicit branch for the new payload kind. A contract that's only implemented on the backend and rendered by the UI as generic text is a downgrade, not a fix — the user sees an inert message instead of a button that resolves the block.

What doesn't work

- Retrofitting the framework's native retry mechanism. It's built for a different failure shape (synchronous in-process exceptions) and its state doesn't survive the exact round-trip (pause → external resume) that long-running failures live in. Spending effort trying to configure it for this case is time that doesn't pay off — verify with a source read, don't assume it "should" work because it exists. - Letting the node raise and relying on framework-level error handling to recover gracefully. Whatever exactly happens next (implementations differ on whether it's the single node or the whole run that's terminated), none of it is "stay alive, let the user fix the precondition, retry the same node" — which is the actual requirement. A raised exception is a dead end for a recoverable failure, full stop. - A single ad-hoc "if failed, do X" check sprinkled per node as you notice a gap. This was tried first and is the natural first instinct — it looks done because a few nodes have it. In practice, the failure contract needs to be applied at *every* dispatch site and *every* resume site for *every* long-running step, and any node someone adds later has no structural reason to remember it. Concretely, this shape of gap produces: a step stuck in "running" forever because the failure branch of the underlying job never routes back into the resume path (only the success branch does); a step that reports full success right after a resume because nothing after the resume re-checked the result; a return value silently discarded so the step just proceeds as if nothing happened; and swallowed network failures on the callback itself, which look identical to "still working" from the outside. Every one of these is invisible until someone reproduces the exact failure condition live — they do not show up in a happy-path test. - Trusting "HTTP 200 back" or "no exception raised" as your only success signal, especially for anything that's supposed to produce a *quantity* of output. A downstream call can return 200 with an application-level success: false buried in the body, or return "success" with zero items actually produced — if the only check is "did an exception happen," both of these get counted as a clean success and the caller proceeds to a "done" state with nothing to show for it. - A single broad `except` around the dispatch/resume path, intended to make things "safe." It's very easy for a defensive except here to also absorb the framework's own internal pause/interrupt signal, or a deliberately-raised precondition error — silently turning "the framework asked to pause" or "we intentionally rejected this" into "unexpected error, degrade to a generic fallback." See A blanket `except Exception` swallows an ImportError and degrades into a misleading business error and Some exceptions stringify to an empty string — logging only `str(e)` hides the real error for the same family of "broad catch destroys the actual signal" failure.

Related

- Seven verified gotchas in a pause/resume-native workflow framework's primitives (checkpoint, resume_inputs, compaction, disconnect) — the wider set of pause/resume gotchas this contract sits inside (invocation-scoped resume inputs, compaction eating paused gate calls, synchronous nodes dying on client disconnect). - Answering a gate with text + functionResponse together silently breaks Workflow resume — a second, unrelated way a resume can silently fail to land. - A long-running tool's follow-up turn is generated server-to-server and never streamed to the browser — the UI needs an F5 to show what already happened — a different silent failure in the same long-running-tool family: the result is correct but never reaches the live UI. - The tool stays 'running' for hours: a synchronous LLM call with no timeout blocks the event loop — the sibling failure mode where nothing ever comes back at all, versus this page's "something came back, but it was a failure nobody checked for." - Idempotency guard with Redis SET NX — fire the dispatch once, never on retry-count alone — why the retry path needs its own gate, not the original dispatch gate. - Coarse boolean completion gates hide partial completion — the general pattern of a boolean/success gate that can't distinguish "fully done" from "done in name only."