The tool stays 'running' for hours: a synchronous LLM call with no timeout blocks the event loop

An async tool that calls a synchronous, unbounded LLM-SDK method directly inside async def blocks the whole worker's event loop — and because a genuine hang never raises an exception, no failure event ever gets published; the job just sits at "running" forever.

The signal that gives it away

- A UI/sidebar step shows "running" for hours. No completed or failed event ever arrives. Refreshing the page changes nothing — there is genuinely nothing new to hydrate. - In the event/log stream for that specific job: you see the full started event, then several progress events ticking on a normal cadence (every 10-30s), then total silence — no completed, no failed, no exception logged anywhere. That exact shape — started + progress ×N + silence — is the hang fingerprint. It is distinct from a crash (which logs an exception) and from "never started" (no events at all). - In the persisted job/task table: a row stuck at status=running with started_at many minutes or hours in the past. Before calling this a hang, rule out three lookalikes that share the same UI symptom but have different causes and different fixes: a row already status=failed (the tool failed correctly — read its error, this isn't the bug); a row status=succeeded but the UI still shows running (a frontend hydration bug, not a backend hang); no row at all for the expected call (the caller never actually dispatched the tool — check the agent's own log for whether it emitted the call). - The cascade tell, which is what usually gets misreported as an unrelated bug: any downstream step that depends on this tool's output (e.g. it reads a cache/result key the hung tool was supposed to write) will itself stall or error out. From the outside this looks like "the agent went silent" or "the agent is stuck" — a completely different-sounding report — when the actual root cause is one specific upstream tool call that never returned. An agent waiting on a tool response has no way to distinguish "still legitimately working" from "hung and will never answer" — so don't diagnose "agent looks unresponsive" as an agent-behavior bug until you've checked whether an upstream tool is the one actually hung.

How to exploit it (diagnose, then fix)

Five-step diagnosis, in order:

1. Compare persisted job status against the live event stream for the same job. A running row with a stale started_at, paired with an event stream that also went silent past that same point in time, is the actual fingerprint — not just "status says running," which alone is ambiguous with the three lookalikes above. 2. Read the event lifecycle for that job specifically. Expected shape is started → progress(×N) → completed|failed. The hang pattern is started + a handful of progress events + nothing ever again — no terminal event of either kind, ever, no matter how long you wait. 3. Identify the actual blocking call. Grep the tool's source for the LLM SDK's synchronous generation call. Any call site not already wrapped in a thread-offload-plus-timeout pattern is a candidate for the hang. 4. Confirm the pattern in code, don't assume. await asyncio.wait_for(asyncio.to_thread(sync_call, ...), timeout=N) is safe. A bare response = sync_call(...) sitting directly inside an async def, with no wrapping at all, is the bug — visually easy to miss because it reads like ordinary, correct-looking code. 5. Trace what else depends on the hung tool's output. Anything reading a cache/result key the hung tool was supposed to write will also stall or error — and that dependency almost always gets reported as a separate, seemingly unrelated bug rather than traced back to this one call.

Once you've found the call site, the root cause is three compounding problems, and all three have to hold simultaneously for the hang to be both silent and permanent:

1. The call is synchronous but lives inside `async def`. Nothing ever hands control back to the event loop while it's in flight, so it doesn't just block its own task — it blocks the entire worker, including unrelated concurrent work on the same process. 2. No explicit timeout is set. LLM SDKs commonly apply no default timeout visible from the underlying HTTP client — the call can wait on a stalled or overloaded backend indefinitely. This isn't hypothetical: a filed, public issue against a major LLM provider's Python SDK describes exactly this — a socket that stays open without ever receiving data, with the calling thread blocked indefinitely (see provenance). 3. Error handling only fires on a raised exception. A try/except Exception around the call is structurally incapable of reaching its failure branch on a genuine hang, because a hang by definition never raises anything. Any finally-style guard that's only wired to run after a raise (rather than unconditionally) never fires either.

The fix is one canonical helper, reused at every call site instead of patched in ad hoc per file:

``python async def safe_generate_content(model, prompt, config=None, timeout: int = 120): return await asyncio.wait_for( asyncio.to_thread(model.generate_content, prompt, generation_config=config), timeout=timeout, ) ``

asyncio.to_thread moves the blocking call off the event loop into a worker thread; asyncio.wait_for gives that call a hard deadline and raises TimeoutError if it's crossed — these are exactly the primitives the standard library ships for this shape of problem (see provenance). Pick the timeout budget from the observed latency distribution plus margin for the SDK's own internal retries, not a round number pulled from nowhere — in one observed case typical calls ran 5-30s, so a 120s budget comfortably covered worst-case retry behavior without making a genuinely-stuck user wait an unreasonable time to *see* the failure.

Second, independent layer

every long-running tool needs a try/finally that guarantees exactly one outcome event is published, even on an abrupt exit. Track it with a local "already published" flag; in finally, if that flag is still false, publish a generic failure (e.g. error="abrupt_exit") rather than nothing. This is what turns "the call timed out" into "the caller receives a structured failure it can act on" instead of pushing the exact same silent-hang problem one layer up the call chain.

Acceptance check that the fix actually holds

feed the tool a condition that would previously have hung (simulate a rate-limited or unresponsive backend) and confirm — (a) it returns a structured {success: false, error: "timeout"} within the timeout window, not "eventually"; (b) the persisted job status flips to failed, never sits at running past the deadline; (c) whatever consumes that response (an orchestrating agent, a UI) receives the structured failure and can react — narrate a fallback, render a retry action — instead of waiting on nothing. One fixed call site, live-verified: a job that previously hung 12+ hours completed (correctly, with a real result) in 29 seconds after the wrap was applied — the fix doesn't just fail faster, it removes the hang for the happy path too, since the call was never actually the bottleneck; the missing bound was.

Immediate workarounds, for when you're staring at an already-stuck job in production and the fix isn't deployed yet — in increasing order of how drastic they are:

- A) Manually fail the stuck job. Flip its persisted status to failed, then manually publish the corresponding failed event on whatever channel the frontend listens to. The UI transitions out of "running" as soon as it receives that event — from the frontend's point of view this is indistinguishable from the code having done it. - B) Re-trigger the tool with an explicit bypass signal (a "force refresh" flag that skips whatever cache/dedup would otherwise make the retry a no-op). This unsticks the *user in front of you*; it does nothing for the underlying bug — the next hang-prone call still hangs exactly the same way. - C) Restart the worker process. Last resort, and drastic: it forcibly kills the blocked thread, which at least unwinds as an exception from the process's perspective (so a well-placed top-level try/except gets *some* signal), but any state that only lived in that process's memory is lost. Durable state in your job store or cache survives a restart; in-memory session state does not.

What doesn't work

- Treating "no exception raised" as proof the tool is fine. A hang produces zero exceptions by construction. Any failure path gated on catching one is waiting for something that will never happen. - Fixing the one call site from the bug report and assuming its siblings are safe. The exact unwrapped-sync-call pattern tends to be copy-pasted across every tool that talks to the same SDK, not just the one someone happened to notice hanging. In one observed audit, the pattern was present at roughly twenty call sites spread across seven tools — fixing the one that generated the ticket and stopping there would have left the rest waiting to hang the same way, on a schedule nobody could predict. - Deprioritizing call sites that aren't on the primary user-visible path ("it's just a background helper, nobody's watching it"). Two call sites deliberately left unwrapped on exactly that reasoning still ended up needing a follow-up sweep once they hung in practice — being off the main path reduces how fast you notice, not whether the bug exists. - Assuming a generic "stuck job" watchdog (auto-fail anything running past N minutes) is a substitute for the timeout fix. A watchdog catches the symptom minutes-to-hours late, and it can't distinguish "this specific class of hang" from "the worker process crashed mid-task" — it's a reasonable safety net for the latter, but not a fix for the former; see No long-running agent framework handles failure for you — you need an explicit error contract for why "something eventually happened" is not the same guarantee as "failure is reported promptly and correctly." - Chasing "the agent looks stuck / went silent" as an isolated bug. It's frequently the downstream symptom of exactly this hang, one layer removed — the agent is faithfully waiting on a tool response that will never arrive. Check for an upstream hung tool before spending time on agent-behavior instructions or prompt tuning; those fixes won't touch a problem that lives one layer below the agent entirely.

Related

- No long-running agent framework handles failure for you — you need an explicit error contract — the sibling failure mode: there, *something* comes back through the resume channel but nothing checks it for failure; here, *nothing* ever comes back at all. Different shape, same root instinct ("no exception ⇒ treat as fine") failing in two different places. - A per-call timeout doesn't kill the underlying thread — the process outlives its own deadline and blocks actor exit — the same "looks alive, isn't, and nothing raises to say so" shape in a different execution context: an external actor/process holding a connection open, instead of an in-process blocking call. - A blanket `except Exception` swallows an ImportError and degrades into a misleading business error — a related way a broad except gives false confidence: it only helps when something actually raises, and a hang is exactly the case where nothing does. - Idempotency guard with Redis SET NX — fire the dispatch once, never on retry-count alone — relevant the moment you re-trigger a previously-hung tool (workaround B above): a naive re-dispatch can collide with an idempotency guard that has no way to know the original attempt is dead, not just slow. - Static verification doesn't prove runtime behavior — the SEED / ACTIVATE / VERIFY / NEGATIVE checklist — this entire failure class is invisible to a syntax or type check; it only shows up under a live call against a real, or realistically slow, backend.