Coarse boolean completion gates hide partial completion

A gate that answers "has this already been done?" with a single EXISTS/boolean on an *aggregated* key can only ever mean "something was written under this key" — it has no memory of which sub-units that something covered, so anything added to the universe of expected sub-units after the gate first turned true is silently skipped forever.

The signal that betrays it

Look for these together — any one alone is common and harmless, the combination is the tell:

- A "done" predicate implemented as a bare existence check on one aggregate key (EXISTS key, key is not None, artifact_exists(prefix:*)), consumed by a branch literally named already_done / skip / fast_forward. - A companion field that *looks* content-aware but is actually just a count: written as len(items) and never compared against anything, never stored as the actual list of covered ids. It reads like tracking, but it tracks nothing — it's just a number nobody diffs. - Two or more independent call sites consuming the same coarse predicate (e.g. one path that decides "should I schedule this work" and a separate fast-forward/resume path that decides "can I skip this step") — when a gate this shallow gets a second consumer, the blast radius of the missing granularity doubles without anyone touching the gate itself. - The authoritative list of what *should* have been covered already exists somewhere else in state (a strategy object, a plan, a config with stable per-item ids) — the gate simply never reads it. - Symptom in the wild: you add a new segment/sub-scope/filter *after* the aggregate key was already written once, and that new addition is never processed again, with zero errors anywhere. It looks like the pipeline "forgot" the new item, when actually it never even tried.

This is the completion-tracking half of a well-known idempotency gap: naive idempotency implementations that check only "does a record for this key exist" cannot tell IN_PROGRESS from COMPLETED, let alone "completed for which subset" — the fix in that literature is to track explicit status per unit of work, not a single existence flag (see sources). The bug described here is the same shape one level up: the granularity of the "done" flag doesn't match the granularity of the actual work.

How it's exploited (i.e. how you fix it without breaking what already works)

1. Confirm the coarse predicate really is coarse. Grep for every consumer of the aggregate key/flag. If you find more than one independent site branching on it, treat the fix as touching all of them, not just the one you noticed the bug in. 2. Find the authoritative per-item list. In practice it's usually sitting right next to the coarse key already — a strategy/plan object with stable ids per sub-unit (segment_id, id, or name as fallback) that nothing currently reads for gating purposes. Don't invent a new source of truth if one already exists. 3. Add, don't replace. Write an additional field that is a *list of covered ids* (covered_ids, segments_searched_ids) next to the existing count — keep the count for backward compatibility, it costs nothing to keep writing it. 4. Build a per-subset predicate and AND it with the old boolean. search_covers_current_segments(expected_ids, covered_ids) — old aggregate blobs that predate the ids field simply return True (no ids recorded → assume full coverage), which is the deliberately conservative default: it means you never force a re-search of a pool you already paid for, you only add coverage for the *new* stuff that has no covered-id yet. 5. Route the gap to a single-item repair, never a full re-run. Once you know exactly which sub-unit is missing, the corrective action should be scoped to that one sub-unit (a single-segment search, a single-item enrichment) — reusing whatever "add one more of these" code path already exists — instead of re-triggering the whole original operation. If the underlying operation is paid/expensive, this distinction is the difference between a one-line fix and a five-figure re-billing mistake. 6. Ship it behind a flag, default OFF, until verified live. A gate this foundational sits underneath fast-forward/resume logic; a wrong per-subset predicate can make the pipeline think MORE is missing than it should (re-triggering paid work you already have) just as easily as it can miss things. Flip it on only after watching it decide correctly on real, already-populated state.

What doesn't work

- Tightening the boolean instead of adding a list. Making the EXISTS check "smarter" (TTL-based staleness, re-checking timestamps) doesn't help — the problem isn't *when* the key was written, it's *what it covers*. No amount of time-based logic recovers per-item information that was never recorded. - Trusting the count field as if it were the list. len(segments) was written at some point in the pipeline as a metric, not as a gate input — treating "the count changed" as a proxy for "coverage changed" breaks the moment two different-sized sets happen to produce the same count, or the count is written before the list it's counting is finalized. - Defaulting new-format blobs to "not covered" for backward compatibility. The tempting "safe" default when introducing the per-item ids field is to treat old blobs (which have no ids) as *not* covering anything — this inverts the actual risk: it forces a full, possibly paid, re-search of every pool that predates the fix, the exact kind of blast radius the fix was supposed to avoid. The conservative default has to point the other way: no ids recorded → assume already covered, only require the new field going forward. - Fixing only the consumer you found the bug in. When a shallow gate has more than one independent consumer, patching one and leaving the other on the old bare-boolean check just moves the starvation to whichever path you didn't touch — it will still fast-forward past newly added sub-units through the untouched path.

Related

- A sibling failure mode: a *different* mechanism (a batch cap combined with progress state written for the whole universe) produces the same symptom — permanent silent starvation of a backlog — without a boolean gate anywhere. - Check what you already own before paying again for the same external query — the other side of the same coin: before paying for a redo, check what you already covered; the normalization trap found there (mismatched filter formats hiding a real match) is the same "coverage looks incomplete but the check is just too shallow to see it" family. - A paid API response gets parsed into something useful, then never written anywhere that outlives the request — what happens when the paid result *is* correctly scoped but gets thrown away anyway; read together with this page, the two cover most of "we paid for it, now what." - Idempotency guard with Redis SET NX — fire the dispatch once, never on retry-count alone — a completion-tracking pattern that has the opposite problem solved: fire-once idempotency at the single-dispatch granularity, useful contrast for why granularity has to match the unit of work you actually care about. - Single-flight lock with Redis SET NX — reject or coalesce concurrent hits on the same mutation key — relevant if the fix above (read-list, then write an updated covered-ids list) is done under concurrent requests: the read-modify-write of the coverage list needs the same single-flight protection as any other multi-key mutation.