Canary rollout of a dual-write feature flag — phased percentages, health checks, non-destructive rollback

A default-off flag that makes a write path also write to a new canonical store, rolled out in increasing session-hashed percentages with a log-grep-able signal at every phase, and rolled back by flipping the flag off — never by deleting what the new store already has.

The signal that gives away the technique

You're mid-migration from an old system of record (say, an in-memory/Redis cache) to a new canonical one (say, Postgres), and you cannot afford a big-bang cutover: the write path is hit by production traffic continuously, and a bad row shape or a saturated connection pool would be discovered only under real load. The tell in the code is a boolean gate wrapping a *second* write next to an existing one that's left untouched — "keep writing the old thing exactly as before, and also write the new thing behind a flag." If you see percentage-based branching keyed on a stable hash of an id (not random()), that's the canary bucket assignment, not A/B testing — the same entity must land in the same bucket on every call and across restarts, otherwise the health signal at each phase is noise.

How it's exploited (operationally)

Flag mechanics. A single env var (default false) gates the new write. The read of that env var is memoized (e.g. lru_cache), which has a direct operational consequence: flipping the env var alone does nothing until the writing service is restarted. Every step below that changes the flag or the percentage ends in "restart the writer," and that's also why rollback is fast — it's the same one action.

Canary bucketing. Hash the entity id (session id, user id, whatever the write is keyed on) with a stable hash (SHA-256 is enough), take it modulo 100, and compare against the target percentage: bucket < percentage → in canary. Properties that matter: - 0 = never, 100 = always, no special-casing needed. - Same id → same bucket on every call and across service restarts (no per-instance seed). - Distribution should be verified uniform over a real sample (order-of-thousands of ids landing within a few points of the target percentage), not assumed from "it's SHA-256 so it must be uniform."

Grep-able phase signal. Every call site inside the new write path logs a single fixed prefix (e.g. [DW]) at three moments: on first flag read (dual_write=True/False), on successful write, on conflict-resolved (idempotent upsert). This one prefix is the entire monitoring story — docker compose logs -f <writer> | grep "\[DW\]" during rollout, and its *absence* after a rollback is the confirmation the rollback worked. Don't design a dashboard before you have this; the log line is cheaper and catches issues the dashboard doesn't know to look for yet.

Phased rollout, in order

1. Phase 0 — flag off (baseline). Confirm the env var is unset/false and behavior is unchanged. This is the state you're proving you can always return to. 2. Phase 1 — single-session smoke test, manual trigger. Flip the flag on, restart the writer, confirm the startup log shows the flag as enabled. Then force exactly *one* known entity through the new path — a temporary hardcoded id check in the code (if entity_id == "<test-id>":) is acceptable here specifically *because* it's Phase 1 and gets removed in Phase 2; it lets you verify the new store got the row and the old store still has it (dual-write means old-path removal comes later, if ever) before any real traffic is exposed. 3. Phase 2 — canary 10%, driven by the real hash-bucket function, hardcode removed. Monitor for a window long enough to cover your real traffic pattern (e.g. a full day-of-week cycle as one illustrative baseline — tune to your own peak/off-peak shape, not this number). Health check = no increase in upstream call failure rate on the write's dependency, no connection-pool saturation on the new store. Expect roughly 10% of live entities to show the new-store row with the phase's log marker in that window — if it's off by more than a few points, the bucketing function is suspect before the feature is. 4. Phase 3 — 50%. Change the percentage constant, restart, same monitoring protocol and window as Phase 2. Nothing else changes — this is deliberately boring. 5. Phase 4 — 100% (or drop the canary check entirely and gate only on the flag). Monitor for longer than the earlier phases — as an illustrative starting point, some teams double the Phase 2–3 window — because at 100% every downstream consumer of the new store is now exposed, whereas Phase 2–3 only exposed a slice. Size the actual window to your own traffic cycle and blast radius, not to a fixed number. 6. Post-rollout integrity check. Compare a count from the new store against a count from the old store for the same population (e.g. distinct entities with rows in the new canonical table vs. keys matching a pattern in the old cache). Expect them to converge to roughly equal within the monitoring window, not instantly — writes are still trickling in from long-lived sessions that started before the flag flipped.

Rollback, target under 60 seconds, at any phase

1. Flip the env var back to false (a one-line sed/edit is fine — this is meant to be fast, not elegant). 2. Restart only the writing service. 3. Confirm via the startup log line that the flag reads as disabled.

The reason this is fast is exactly the reason the flag pattern was chosen in the first place: rollback is "stop writing to the new store now," not "undo what's already there." Rows already written to the new canonical store stay there — that data is presumed good and a later phase (or a separate initiative) will build on it. Treating rollback as if it must also clean up the new store turns a 60-second action into a data-deletion decision under incident pressure; don't conflate the two.

Test suite as a gate, not a formality. Before advancing *any* phase, run the dual-write unit tests and require all green — not "green enough." The set of properties worth having explicit tests for (this is the valuable part, more than the count): - round-trip: flag on + N inputs → correct subset written, correct subset skipped (mandatory-field rows skipped, not written malformed). - idempotency: re-processing the same entity doesn't duplicate rows, and a conflict-resolution upsert preserves the *original* value of a field that a later write might otherwise clobber (e.g. a provenance/source field) — this is the test that would have caught a silent-overwrite bug. - flag gate on/off via env override, including that the memoized read actually clears when the test forces it to (this is your proof that "restart clears the cache" isn't cargo-culted). - canary distribution uniform over a real sample, and canary hash stable across repeated calls with the same id. - the regression proof: simulate the exact failure mode the migration exists to fix (e.g. an operation on the old store that renames/expires an entity in a way that used to also lose it from the system of record) and assert the new canonical store's row is untouched. If you're doing this migration at all, this is usually the one test that encodes *why* — write it first.

What doesn't work

- Different percentages at different call sites during the same phase. If the write path has more than one entry point gated by the same canary, they must agree on the percentage constant. Drift here means you can't attribute a health-check signal to "10% of traffic" — you don't actually know what fraction is live. - Flipping the flag off in production without checking the log first. Disabling mid-flight without confirming the previous state produces orphaned partial-write log events that pollute the exact signal you'd need to diagnose *why* you rolled back. - Enabling the flag in `.env` straight to a percentage, skipping the single-entity smoke test. The smoke test phase exists to catch schema/shape mistakes (wrong types, missing constraints) cheaply, on one known entity, with an easy visual check on both stores. Skipping straight to 10% means the first sign of a shape bug is a wave of production failures instead of one manual query. - Clearing the memoized flag-read cache in production ad hoc instead of via a full service restart. If the read is behind a process-level memoization, only a restart is a verified-clean state; clearing the cache object at runtime without restarting the rest of the process's state is not something the design accounts for and isn't part of the tested path. - Treating "rollback" as "delete the new data." As above: conflating "stop the bleeding" with "undo the write" is the single costliest misunderstanding of this pattern, because it turns a safe, reversible action into a destructive one under time pressure.

Related: Idempotency guard with Redis SET NX — fire the dispatch once, never on retry-count alone (the fire-once idempotency discipline this pattern's upsert step depends on).