A resumed old session redoes expensive work from scratch — the idempotency guard was reading a short-TTL artifact

An "already done? does the artifact exist?" idempotency guard checked working-state keys with a short TTL while the rest of the pipeline persists on a much longer TTL — once the short-TTL keys expire, the guard sees nothing and silently re-executes a step that was already completed.

The signal that gives it away

- A session/job resumed after a period of idle time re-runs a costly step (an external job dispatch, a paid API call, a long computation) even though everything *downstream* of that step is still present and intact. The re-run is asymmetric: some artifacts survived, one specific class did not. - The elapsed idle time roughly matches a suspiciously short, round TTL (a day, an hour) rather than the pipeline's real retention window (weeks or months). - Checking the actual keys shows the pattern: the *final* result key (if it was ever written) carries the long pipeline TTL, but the *in-progress / working / progress* keys the guard actually globbed for (prefix:*) carry a short "working data" TTL that has already lapsed. - The bug is invisible in tests run within the same session or the same hour — it only appears on genuinely old sessions, which is why it tends to surface late, in the field, not in CI. - A second, independent tell: the guard is implemented as "does at least one artifact of this pattern exist" (an existence check on a wildcard), not as "is the durable, final record of completion present." Existence-of-anything is a weaker predicate than existence-of-the-final-thing, and the gap between the two is exactly where a short-lived working artifact can masquerade as — or fail to substitute for — the real signal.

How to exploit / detect it

1. Identify what the idempotency guard actually queries (e.g. a key-pattern scan service:{id}:*, or a call like "does this artifact exist for this session"). Do not trust the docstring — read the literal pattern. 2. Diff the TTL of every key that pattern could match against the TTL the guard's author *assumes* it has. Concretely: TTL key on each key kind (:working:*, :progress, and the final key) and compare to the pipeline's standard retention (e.g. 180 days). Any keyed data written by a "cache working progress for observability" helper is a prime suspect — progress/telemetry helpers are routinely built to expire fast (correctly, for their own purpose) with zero regard for who else might be reading them as a completion signal. 3. Reproduce deterministically: force one of the short-TTL keys to expire early (EXPIRE key 1) or wait past its TTL, then resume the session/job. Compare pre- vs post-expiry behavior — before expiry, fast-forward with no re-dispatch; after, the expensive step re-triggers. 4. Confirm the root cause is TTL mismatch and not something else (e.g. a hashing/keying mismatch between the writer and the reader — check that both sides derive the same key from the same inputs; a sha256(x)[:12] vs sha256(y)[:16] split between two code paths can independently cause a guard miss and should be ruled out separately, see Idempotency guard with Redis SET NX — fire the dispatch once, never on retry-count alone). 5. Fix, KISS: align the TTL of every artifact the idempotency guard reads to the pipeline's real retention standard — not a separate, shorter TTL, and not a brand-new parallel "step done" marker. A new marker sounds appealing but now has to be kept in sync with cascade/rollback invalidation logic that already exists for the other keys (see Soft vs. hard cascade invalidation: when 'confirm, then delete downstream' destroys data you paid for) — doubling the invalidation surface instead of fixing the one line that sets the TTL. 6. After the fix, re-verify with the exact same repro: same idle window, same resume path, confirm zero re-dispatch and that the key's remaining TTL now matches the pipeline standard (e.g. ~15,552,000s for 180 days, not ~86,400s for 1 day).

This class of bug compounds with resume/invocation semantics in agentic workflow frameworks: if a free-text message resumes a paused workflow as a *new* invocation (rather than continuing the paused one), the framework's own resume-state is empty and the *only* thing telling the orchestrator "this step is already done" is exactly this kind of external existence check — see Seven verified gotchas in a pause/resume-native workflow framework's primitives (checkpoint, resume_inputs, compaction, disconnect) and Answering a gate with text + functionResponse together silently breaks Workflow resume. When that external check is itself built on a short-lived artifact, the two problems stack: the framework has already forgotten what happened, and the safety net it relies on has *also* expired.

What does NOT work

- Bumping the TTL "just for the working keys, a bit" (e.g. 1 day → 7 days). This only moves the failure window further out and makes it rarer, harder to notice, and harder to reproduce on demand — it is still wrong in principle. The correct fix is to make the guard's TTL match the pipeline's actual retention contract, not to guess a bigger arbitrary number. - Adding a separate "step completed" boolean/marker with its own long TTL, on top of the existing working-data keys. This looks like a quick patch but creates a second source of truth that must be invalidated in lockstep with the cascade logic already governing the other pipeline keys. When a user rolls back or edits an upstream step, whoever wrote the cascade invalidation now has to remember to also clear the new marker — miss it once and you get the opposite bug (a step marked "done" that was actually rolled back). The public idempotency-key precedent for payment APIs makes the same tradeoff explicit: keys are pruned after a fixed window and a reused key past that window is treated as a brand-new request — the window itself has to be a deliberate, shared parameter, not an incidental side effect of an unrelated helper's TTL choice. - Treating "artifact exists" as a strong completion signal without checking which artifact. A wildcard existence check (prefix:*) that happens to match a working/progress key is not the same guarantee as checking for the final, durable record — even when both currently return "found," they can drift apart in retention and silently stop agreeing after the short-lived one expires. Redis's own expiration semantics are unforgiving here: once a key's TTL lapses, EXPIRE/passive lookup on read simply returns nothing — there is no gray zone, no partial credit, the key is just gone as if it never existed. - Fixing only the symptom you found without auditing sibling keys. If one working-key helper had a mismatched short TTL, check every other helper on the same idempotency-sensitive code path for the same anti-pattern (progress-reporting helpers are frequently written independently of the completion-check logic that later comes to depend on them).

See also

- Seven verified gotchas in a pause/resume-native workflow framework's primitives (checkpoint, resume_inputs, compaction, disconnect) — the framework-level half of the same failure family: free-text resume opens a fresh invocation with empty resume-state, which is exactly what makes an external idempotency guard load-bearing in the first place. - Idempotency guard with Redis SET NX — fire the dispatch once, never on retry-count alone — the sibling guard pattern (fire-once dispatch lock) that must not be confused with this one: that guard prevents *concurrent re-dispatch within a run*, this one prevents *re-dispatch across a resumed, aged session*. Both fail if built on the wrong-lived artifact. - Answering a gate with text + functionResponse together silently breaks Workflow resume — another way a resume can silently restart a workflow from scratch. - 401 on a resumed session: a helper preferred the expired token snapshotted in state over the fresh one in Redis — a different "old session resumed" failure surfaced in the same investigation: stale data preferred over fresh, mirroring this bug's stale-signal shape from the opposite direction. - Soft vs. hard cascade invalidation: when 'confirm, then delete downstream' destroys data you paid for — why TTL and cascade-invalidation are two separate mechanisms that must not be conflated when deciding how a piece of state disappears. - Coarse boolean completion gates hide partial completion — a related family of bug where a completion gate's predicate is weaker than what the caller assumes it guarantees.