A per-call timeout doesn't kill the underlying thread — the process outlives its own deadline and blocks actor exit

A coroutine-level timeout (asyncio.wait_for around asyncio.to_thread) only cancels the *waiting*, not the OS thread doing the work — the orphaned thread can keep a network connection open long after the "timeout" fired, and if cleanup code later tries to close that same connection, cleanup itself hangs and the process never exits.

The signal that gives it away

- A platform-level job (Apify actor, container job, serverless task) reports a generic infrastructure timeout ("Actor timeout of N minutes exceeded", "run aborted") at a round number that is suspiciously larger than the actual work duration — e.g. the job's own logs show it printed its final "complete"/"exiting" line and even pushed its output well before the platform-level cutoff. - The run stays RUNNING for a long tail after the real work is done — no new log lines, no progress, just silence until an *external* cap (platform timeout, orchestrator poll-cap) fires and reports failure. - Two timeouts are stacked and only one of them is documented: a short per-call timeout your own code applies (e.g. 90s around one external API call) and a much larger caller-side or platform-side cap (e.g. a poller with max_polls=60 at 10s each = 10 minutes, or a platform hard limit at 1500s). When the inner timeout is the one that actually degrades, the caller's error message ("Actor timeout (10 minutes exceeded)") is its own cap, not the platform's, and is misleading about where time was actually lost. - If you strace/inspect the hung process, or just reason about the code: a bounded async call sits on top of a thread pool (asyncio.wait_for(asyncio.to_thread(blocking_call), timeout=T)), and a cleanup step later in the same run calls .close()/.aclose() on a client whose underlying connection is the one that orphaned thread still holds.

How it's exploited (i.e. how you fix it)

The root cause, confirmed against upstream: cancelling an asyncio.Task that wraps asyncio.to_thread (or loop.run_in_executor) cancels the *coroutine* waiting on the executor future — it does not stop the underlying OS thread. The thread keeps running the blocking call to completion in the background; only the *result* is discarded. This is a documented, still-open limitation of asyncio's executor model, not a bug in your code ([cpython#107505](https://github.com/python/cpython/issues/107505)).

A common chain, observed in production, generalized:

1. A blocking call to an external API (any SDK that isn't natively async) is wrapped as asyncio.wait_for(asyncio.to_thread(call), timeout=90) to keep one slow call from blocking the run. 2. When that call exceeds 90s, wait_for raises TimeoutError and the coroutine moves on — but the to_thread worker is still executing the blocking call, still holding its HTTP connection open. 3. The run finishes its actual work and reaches a cleanup step (await client.aclose() on the same HTTP client the orphaned thread is using). Closing a connection that another thread is actively using can itself block, sometimes until the socket's own OS-level timeout. 4. Because that cleanup call is await-ed before the process exits, the process never reaches its own exit call — it just hangs. The platform (Apify, a container scheduler, a job runner) has no visibility into "why": it only sees a process that hasn't terminated, so its own outer timeout eventually fires, much later, with a generic message. On the Apify platform specifically, run status is derived purely from whether the main process has terminated and its exit code — there is no partial-credit for "already pushed the output" ([Apify actor lifecycle docs](https://docs.apify.com/sdk/python/docs/concepts/actor-lifecycle)).

The fix that actually worked: don't try to make the cleanup safe — guarantee process exit independently of it. Start a hard watchdog in a separate daemon thread, timed generously above your expected cleanup duration, that force-exits the process no matter what the event loop or a hung cleanup call is doing:

```python import os, threading

watchdog = threading.Timer(20.0, lambda: os._exit(0)) watchdog.daemon = True watchdog.start() try: await cleanup_http_clients() # may itself hang — that's fine now except Exception: pass os._exit(0) ```

Why this specific shape matters: - os._exit() terminates the process immediately, bypassing atexit handlers, finally blocks and thread joins — which is exactly what you want when a child thread might otherwise keep the process alive forever. A normal sys.exit() would not help here: it only raises SystemExit in the *calling* thread and still waits for non-daemon threads. - The watchdog runs on a plain OS thread (threading.Timer), not on the asyncio event loop, precisely because the event loop itself may be the thing that's stuck waiting on the hung cleanup call. - The watchdog's deadline should sit comfortably below whatever *external* cap will otherwise fire (platform timeout, orchestrator poll-cap) — its job is to make sure the process reports its real, already-achieved outcome (output already persisted) instead of an unrelated infra timeout message overwriting it. - If the actual output was already durably written (e.g. pushed to a dataset/queue) before the watchdog fires, forcing exit at that point loses nothing — the work was already done and saved; only the process's own shutdown was stuck.

See also Actor exits success with errors buried in a summary dataset item — the signal that gets filtered away for a sibling Apify failure mode where the run *does* exit cleanly but the caller still misreads success/failure — the two are easy to confuse but are opposite ends of the same "process lifecycle vs actual outcome" mismatch. See also The tool stays 'running' for hours: a synchronous LLM call with no timeout blocks the event loop for the more basic version of this problem (a blocking call with *no* timeout at all blocking the event loop) — this page is what happens when you've already applied that fix and the call still doesn't actually stop.

What doesn't work

- Adding a timeout around the blocking call and assuming that's the fix. It bounds how long you *wait*, not how long the underlying work actually runs. If nothing downstream depends on that thread being truly dead, this is enough; if anything later touches the same resource (same HTTP client, same connection pool, same file handle), it isn't. - Retrying or increasing the per-call timeout. This treats the symptom (a slow call) when the actual failure is architectural (thread outlives cancellation). A longer timeout just delays the same hang. - Cleaning up "more carefully" instead of guaranteeing exit. Moving cleanup outside the actor's context manager and calling a plain process exit right after is a common attempt — that's necessary but *not sufficient*, because the cleanup call itself is exactly what can block. Any fix that still puts a hard exit *after* an unbounded await on cleanup inherits the same bug one line later. - Trusting the outer error message to diagnose the cause. A cap you set yourself (a poller's max_polls) and a cap the platform enforces independently can both fire with similarly-worded "timeout" messages; whichever fires first is the one the user sees, and it is rarely the true bottleneck. Trace actual log timestamps (when did the job print its own "done" line?) before trusting either timeout message. - Assuming a shorter watchdog is always safer. Too short, and it force-kills the process before legitimate slow-but-finishing cleanup completes, discarding information you might have wanted (e.g. cleanup-side error logs). The watchdog duration should be set from the *observed* worst-case cleanup time, not an arbitrary small number.