Idempotency guard with Redis SET NX — fire the dispatch once, never on retry-count alone

A durable external lock (SET key NX, TTL, released only on failure) that stops a re-run-on-resume workflow node from re-dispatching the same job every time a fresh invocation starts.

The signal that gives it away

You have a workflow engine that re-executes a node's body from the top on every wake-up (checkpoint/resume style), and the resume state it hands the node is scoped to a single invocation — not to the underlying job or session. The tell in production: the exact same downstream job fires multiple times within a tiny window (measured at 8 dispatches in ~50ms in one observed case) whenever the user sends anything that opens a *new* invocation while a previous dispatch is still in flight (a free-text message, a chip click, anything that isn't a direct reply to the pending gate). Symptom on the user side looks unrelated to duplication at all — it reads as "the pipeline stalls" or "messages don't show up," because the duplicate jobs are silently racing each other, not erroring visibly.

The root cause pattern, generalized: the framework's own resume/memoization primitives (checkpoint replay, per-invocation resume inputs) are scoped to the invocation, so on a fresh invocation the node has no memory that a previous invocation already dispatched the same job. Anything that only checks "did I already see the gate answer in *this* invocation's resume state?" will answer "no" on every new invocation and re-dispatch. If one node in your system already reads an external, invocation-independent store (e.g. it checks Redis for a result artifact before doing work) and a sibling node doesn't, you'll see the asymmetry directly: the guarded node is immune, the unguarded ones aren't.

How to exploit it (build the guard)

Two layers, combined — neither alone is sufficient (see below):

1. Artifact-guard (fast-forward on already-done). Before dispatching, check for the *result* artifact the job would produce, keyed by scope + job type (e.g. result:{scope_id}:{job_type}:*). If it's already there, skip dispatch entirely — the work is done, just re-read it. This covers the case where a fresh invocation starts *after* the job already completed.

2. Fire-once dispatch lock (`SET key NX`, TTL). Immediately before dispatch, SET inflight:{scope_id}:{step_id} 1 NX EX 900 (or your framework's async equivalent). Only the invocation that successfully sets the key gets to dispatch; every concurrent/later invocation that hits the same step while the key exists is a no-op. This covers the in-flight window: job dispatched, but its result artifact isn't written yet, so layer 1 alone wouldn't catch a second invocation racing in during that gap.

Release policy is the detail that actually matters. Release the inflight key only on the failure branch of the job, never on success. Releasing it as soon as the job resolved (success or failure) was tried, and it reopened the exact race being closed: a resume landing in the small window between "job resolved" and "result artifact fully persisted" would see the key gone and re-dispatch. Concretely, in one set of measurements: no guard → 8 duplicate dispatches in 50ms; release-on-any-resolution → down to 3-4; release-on-failure-only → 1. The lock's job is to protect the *entire* lifetime from dispatch to durable artifact write, and success already produces the artifact that layer 1 checks — so success doesn't need the lock released, the artifact-guard takes over.

Pick the TTL to match the realistic worst-case job duration plus margin, not a round default — too short and you get a second concrete failure mode (see A resumed old session redoes expensive work from scratch — the idempotency guard was reading a short-TTL artifact).

Related mechanics worth reading before you design this for your own framework: how the same engine's resume/checkpoint primitives are invocation-scoped in general (Seven verified gotchas in a pause/resume-native workflow framework's primitives (checkpoint, resume_inputs, compaction, disconnect)), and why you can't treat "job resolved" as "job succeeded" without an explicit failure contract (No long-running agent framework handles failure for you — you need an explicit error contract). If your guard reads any cached credential rather than a freshly-fetched one to talk to a sibling service, that's a separate stale-state trap (401 on a resumed session: a helper preferred the expired token snapshotted in state over the fresh one in Redis).

What doesn't work

- Wrapping the dispatch in the framework's own memoization/checkpoint primitive. If that primitive is *also* invocation-scoped (which is common — it's built for "don't redo work within this run," not "don't redo work across runs"), it gives you zero protection against the cross-invocation case that's actually causing the duplicates. This was tried first; it changed nothing because the failure mode was never about a node re-running mid-invocation — it was about a *new* invocation with no memory of the old one. - Artifact-guard alone, no dispatch lock. Looks sufficient on paper ("just check if the result exists before dispatching") but leaves the in-flight window completely open: the job has been dispatched but hasn't finished writing its result yet, so a second invocation arriving in that gap sees "no artifact" and dispatches again. This is exactly the race that produced the 8x duplication before the lock was added. - Dispatch lock alone, no artifact-guard. Handles the in-flight window but not the case where a completely fresh invocation shows up *after* the lock has already expired (TTL passed, job long since succeeded) — without an artifact check it has no way to know the work is already done and will dispatch again. The two layers cover two different time windows; you need both. - Assuming the failure branch is always reached. If the process crashes between dispatch and either success or the failure handler, the release-on-failure-only policy means the lock simply sits until TTL expiry — retries are blocked for that whole window. This is an accepted trade-off, not a bug, but it means TTL choice is also your worst-case retry-blocked duration; don't set it "generously long" without checking that number. - Diagnosing intermittent duplication from framework-level logs alone. The duplication only shows up as a spike in downstream job counts (e.g. count(*) group by job_type on the dispatch log) — the framework's own event stream looks perfectly normal for each individual invocation, because each one genuinely believes it's the first to see the gate. You have to correlate across invocations, not read one.