Soft vs. hard cascade invalidation: when 'confirm, then delete downstream' destroys data you paid for

Cascade invalidation ("user confirms step N → rename/expire every downstream derived key") is correct when the downstream value is cheap to recompute, and destructive when it isn't — the fix is to split the downstream key set explicitly by regeneration cost and give the expensive half a different, softer invalidation.

The signal that gives it away

- A single cascade mechanism walks *all* keys downstream of a "confirm/undo" action and treats them uniformly: rename to :stale:... + a short TTL, no matter what produced the value. If you can't point at a line of code that asks "was this cheap or expensive to make?", it's uniform, and it's a matter of time. - The TTL topology has a gap of multiple orders of magnitude between "how long we keep the live value" and "how long the user has to hit undo before it's gone forever." For example, a live working-state TTL of 180 days against an undo-window TTL of 180 seconds (later tightened to 30s once a lock closed the race that had justified the wider window) is a gap of roughly 5 orders of magnitude between the two numbers, applied by the *same* rename call to every key regardless of what it cost to build. - The persistent, canonical store already has the right table/columns for the expensive artifact (for example, a table shaped for exactly the enrichment record — the same fields the paid API returns), but the write path for that artifact never populates it — it only does a cache SETEX. That's the concrete tell: a durable table sitting unused next to an ephemeral-only write path *looks* like a safety net and isn't one. When the cascade eventually rewrites or expires the cache key, there is nothing underneath to recover from. - The artifact in question was the result of a paid external API call plus a multi-minute async pipeline (order of tens of minutes, real $ cost per unit, ~1,000+ records at tens of KB each merged incrementally in batches). Anything with that shape — paid lookup, long async job, human-reviewed output — is a candidate for the expensive bucket by definition; a value a single fast synthesis call can reproduce is not.

How to apply it

1. Enumerate the cascade map per stage: for every "confirm step N," which downstream keys get invalidated? Most pipelines already have this as a lookup table (step → [affected keys]) even if nobody has looked at it through a cost lens. 2. For every key in that map, ask one question: what does it cost in wall-clock time and money to regenerate this value from nothing? Split the set into two explicit, *named* buckets — don't infer cost implicitly from payload size or which service happens to write it; implicit rules rot the moment a new stage is added that writes something small-in-test but paid-in-production. - Cheap bucket → keep hard cascade. Rename the live key to :stale:{id}:{ts}, short TTL matched to what the UI actually promises for "undo" (for example, 30s). If the user doesn't undo in time, regenerating costs nothing, so losing it is fine. - Expensive bucket → switch to soft cascade. Instead of renaming the live key, copy it to a :backup key with a long TTL (for example, 30 days — an order of magnitude past any realistic "did the user notice" window) and leave the live key untouched. The step is still marked confirmed; the expensive artifact is not destroyed. Users keep working against slightly-stale-but-present data instead of nothing. 3. Give the expensive write path a durable destination, not just a longer cache fuse. A soft-cascade backup with a 30-day TTL is still ephemeral; if the only other copy is the live cache key, you've bought time, not durability. Dual-write the expensive artifact into the canonical store on the same code path that writes the cache (upsert on a stable external id so retries are idempotent). Roll this out behind a flag rather than flipping it for every session at once — see Canary rollout of a dual-write feature flag — phased percentages, health checks, non-destructive rollback. The failure mode of *not* doing this — a paid result computed once and never durably persisted — is its own pattern, see A paid API response gets parsed into something useful, then never written anywhere that outlives the request. 4. Build two independent recovery paths, and keep both: - a fast path keyed off cascade metadata — a pointer recorded at cascade time to exactly which keys were renamed and where they went. Short TTL (seconds), this is what a countdown "Undo" banner should show. - a slow fallback keyed off the backup copies directly, once the metadata pointer has expired. This path is naturally scoped to "primary" keys only unless you deliberately extend it — any auxiliary/side key the cascade also touches (dedup caches, generated-asset references, secondary indices) needs to be added to the fallback's key list explicitly, or recovery silently narrows every time the pipeline grows a new side-key. 5. Prefer not invalidating at all where the join is additive. If a downstream artifact is a filter/join over the expensive data by a stable category id (e.g., "which of these records belong to segment X") rather than a fresh recompute, re-running the cheap classification step re-derives the correct view without touching the expensive data. This is strictly safer than either hard or soft cascade, because nothing ever gets marked stale. Guard the cascade operation itself (and this reclassification) against concurrent triggers on the same session/step with a short-TTL SET NX lock — see Single-flight lock with Redis SET NX — reject or coalesce concurrent hits on the same mutation key.

What doesn't work

- Treating "invalidate" as one verb. A single rename+short-TTL cascade applied uniformly looks correct in dev because dev data is cheap. It only bites in production, against the one stage that has a real regeneration cost — and by the time it bites, the data is already gone. - "Fixing" it by lengthening the one global stale TTL. This moves the cliff, it doesn't remove it. You still eventually destroy the expensive artifact with no durable fallback — you've only bought more days before it happens — and now your genuinely-cheap keys also linger past the window the UI promises, so users see stale state from choices they've long since moved past. Two cost classes need two TTL policies, not one shared knob nudged upward. - Deciding cheap-vs-expensive implicitly. Inferring cost from payload size or which service writes the key rots the first time a new pipeline stage is added. Keep an explicit, named allow-list of expensive keys checked at cascade time. - Assuming the backup/fallback path is "good enough" without checking its coverage. If the slow fallback only restores primary keys and silently drops the auxiliary ones the cascade also renamed, "Undo" appears to succeed but returns the user to a partially-broken state — worse than an honest failure, because nothing signals the gap. - Trusting a "time remaining" UI without checking what its data source returns when there's nothing wrong. A common shape: the endpoint reporting whether the slow fallback is still available returns an empty/missing expiry when the backup is in fact fresh for weeks, and the frontend reads "no expiry value" as "not available," hiding the undo option entirely even though full recovery is possible. Missing-expiry-data and recovery-not-available are two different states — collapse them in the contract and a monitoring gap masquerades as a real deadline. - Trusting that a durable table existing in the schema means the data is protected. A table with the right columns sitting next to a cache-only write path is a schema, not a safety net. Check the write path, not the migration history.