What doesn't work — stacking a custom retry layer over a library that already retries

Wrapping a network call in your own retry loop when the client library underneath already retries doesn't add resilience — it multiplies wasted attempts on top of the library's own, and if that extra sleep sits inside a concurrency gate, it stalls every other item queued behind it.

The signal that gives it away

- A run's duration doubles or triples versus its historical baseline with no other code change in the critical path (one observed case: ~12 min/chunk historically → 27-33 min, then one chunk stuck 2h32 with zero rows produced). - The error being retried is a time-boxed ban, not a transient blip: a Retry-After-style header, a fixed cooldown window, a "429/520 with a stated reset time." Retrying *inside* that window cannot dissolve it — RFC-defined Retry-After semantics exist precisely so a client stops hammering a service that has already told it when it will be available again (see provenance). If your error class carries that signal and you're still retrying immediately, that's the tell. - You already have a measurement that quietly disproves the retry's value, and nobody acted on it: in one observed case, 45 bans with the custom retry vs. 43 without — a ~5% difference, functionally noise — for a code path that was adding minutes per call. A retry layer that doesn't measurably reduce failures is pure overhead wearing the costume of resilience. - The "stuck" symptom shows up one layer removed from the actual cause: a whole batch of unrelated queued items stalls, not just the one making the failing call. That's the fingerprint of a sleep sitting inside a concurrency gate (semaphore, worker pool) rather than around a single isolated call — one slow slot doesn't just run slow, it holds the door shut for everyone behind it.

How it plays out (operational detail)

The compounding math. A helper (_zyte_get) wrapped self.client.get(payload) in 2 extra tries with sleep(35) / sleep(70) on 520/ban/429 — without checking what the underlying zyte_api client already did. Reading the library's own retry factory (and independently confirmed in its published docs) showed it was already far from naive:

| Error class | Library's own retry policy | |---|---| | download error / ban (520) | max_total=4, max_permanent=2, wait ~3-7s + exponential up to ~55s | | network error | keeps retrying for up to 15 minutes straight (stop_after_uninterrupted_delay) | | throttling / rate-limit (429/503) | never stops on its own (stop_never) — wait chain: 20-40s, 20-40s again, then 30-630s with exponential jitter |

So on a single banned query: the library already runs its ~4 attempts (2-4 min), raises, the custom wrapper sleeps 35s, the library runs 4 more attempts, the wrapper sleeps 70s, the library runs again. ~12 attempts and 10+ minutes spent on one call that fails anyway — because the two layers don't know about each other, so neither "gives up" at a point the other respects.

Where it stops being just slow and starts being a stall. The sleep lived inside the same semaphore that gated the whole per-item worker (enrich_company). With a concurrency limit of 20 and a phase that fired ~240 queries (8 per item), every slot doing a doomed retry-wrapped call held its place in the semaphore for the full sleep duration — other queued items got zero throughput while they waited, not reduced throughput. That's the difference between "one slow item" (fine) and "the whole batch stalls" (the 2h32 chunk).

The fix, three moves, applied together

1. Remove the custom wrapper. Call the library directly and trust its real retry behavior for genuinely transient errors (connection resets, momentary blips) — that's what it's actually good at. 2. Reconfigure the library's own retry policy for the error class that shouldn't get infinite patience. Most retry-capable clients expose a way to override defaults (here: an injectable retrying= policy) — override stop_never on throttling specifically, so a 429 can't hold a slot forever no matter what the vendor's default assumes about your workload. 3. Add one hard timeout around the whole call, outside and in addition to the library's internal retries (asyncio.wait_for(..., timeout=N)). This is the actual anti-stall backstop: no matter what the library does internally — 4 attempts, 15 minutes of network retry, an infinite throttling loop — no semaphore slot can be held past N seconds. It is a ceiling, not a substitute for the library's own retry logic underneath it.

For the specific error class that's a time-boxed ban (not a transient blip): the right call is don't retry it at all, fail fast, and reschedule it for later (e.g., at the end of the run) rather than holding a concurrency slot hoping the ban lifts mid-request. Losing one item to retry-later is cheaper than blocking twenty slots waiting on it.

Measured result (A/B, identical input, only the retry layer changed)

| Metric | With custom retry | Without (wait_for only) | Δ | |---|---|---|---| | Total duration | 1616s (27m) | 792s (13.3m) | -51% | | Slowest sub-step | 504.9s | 237.7s | -53% | | Bans on the retried error class | dozens | 4 | | | Hard-timeout cutoffs triggered | — | 0 | | | Downstream failures (unrelated calls further in the pipeline) | 69/518 | 0/524 | | | Output completeness (unrelated feature) | baseline | +32 (unchanged, within noise) | = |

Reading the numbers: bans dropped by an order of magnitude *because the retry was removed*, not despite it — the retry was re-triggering the same window-based ban by retrying inside it, not curing it. Zero hard-timeout cutoffs means the backstop never had to actually cut anything: the call graph, once unclogged, simply never ran long enough to need it — the timeout is insurance, not an active cap on the happy path. The unrelated downstream failures dropping to zero is a second-order effect: once the semaphore stopped clogging, pressure eased on everything sharing that worker pool, not just the retried call itself.

What doesn't work

- Widening the backoff instead of removing the wrapper, when you already have a measurement showing the retry doesn't reduce the failure rate (45 vs. 43 bans here). The instinct when a retry "isn't working" is usually to give it more patience (bigger sleep, more attempts) — but if the underlying error is a fixed-window ban, more patience just means more time spent re-triggering the same window, not a higher success rate. See the general shape of this in the Retry Storm antipattern (provenance): retrying more aggressively into a struggling or rate-limiting dependency doesn't help it recover, it makes recovery slower. - Treating every error the same regardless of whether it carries a stated reset window. A transient network blip and a time-boxed ban look identical in a bare except: block but need opposite handling — retry the first, don't the second. Not distinguishing them is what lets a wrapper "retry" something that was never going to succeed inside the window it retried in. - Fixing only the retry code and not the reason the errors were happening in the first place. In one observed case, a chunk of these bans traced back one layer further: a session/config mismatch fed the pipeline queries that were structurally implausible for the target data, which the upstream service flagged and banned far more than a well-formed query would have. No retry policy — however well-tuned — fixes a request that is fundamentally the wrong request; check whether you're retrying a genuinely transient failure or repeatedly re-sending a malformed one before tuning backoff at all. - Shipping the "add more retry" fix (or the "remove retry" fix) without an A/B measurement on identical input. Both directions feel intuitively safer than they are — more retries feels like more resilience, removing retries feels like less safety net. Only a controlled before/after on the same input told the truth here (-51% duration, bans down an order of magnitude, zero regressions elsewhere). A change that doesn't move a measured number is noise, not a fix, in either direction.

Related

- The tool stays 'running' for hours: a synchronous LLM call with no timeout blocks the event loop — the same "hard timeout as the only real anti-stall backstop" pattern, in a different failure shape: there a bare synchronous call with *no* timeout at all blocks an event loop; here a *layered* retry with no outer bound blocks a concurrency slot. Same fix family: one timeout, applied once, outside whatever the inner call does. - A per-call timeout doesn't kill the underlying thread — the process outlives its own deadline and blocks actor exit — a caution worth checking before trusting move 3 above completely: a coroutine-level wait_for timeout cancels the *waiting*, not necessarily the underlying OS-level call, so an orphaned connection can still linger past the deadline you thought was hard. - A flat HTTP timeout on an SSE consumer kills long turns — and hides the real error — the mirror-image mistake: a timeout set *too tight* for the real traffic shape kills legitimate long-running work. Here the risk runs the other way (no outer bound at all) — the fix is the same primitive, tuned from an observed latency distribution rather than picked from either extreme. - Static verification doesn't prove runtime behavior — the SEED / ACTIVATE / VERIFY / NEGATIVE checklist — the general principle behind the A/B requirement above: a retry policy "looking correct" in code review isn't proof it behaves correctly under real load; only measuring the same input before and after does.