Static verification doesn't prove runtime behavior — the SEED / ACTIVATE / VERIFY / NEGATIVE checklist

Confirming that code *exists* and is *reachable* (it's in the bundle, the endpoint answers, the env var is set, the container is healthy) is necessary but not sufficient to confirm it *behaves* correctly at runtime — the two are different questions and passing the first tells you nothing about the second.

The signal that betrays it

You have a "green" checkpoint that is actually broken, and the tell is always the same shape: every check that passed was a check you could run without ever exercising the actual user-visible path.

Concretely, watch for a status report built entirely out of checks like these:

| Static check | What it proves | What it does NOT prove | |---|---|---| | grep -l "flag_marker" build/** — the marker string is in the compiled bundle | The code was included in the build | That the ON branch is what actually renders, or renders what you think | | curl -o /dev/null -w '%{http_code}' → 200 | The route/page responds over HTTP | That the DOM/response body contains the expected behavioral marker | | env \| grep FLAG → FLAG=true | The variable reached the process environment | That the running bundle actually *reads* that variable at runtime instead of a value baked in at build time | | Container Up (healthy), process listening | The service is up | That the specific feature path inside it works |

None of these four is wrong to run — they're good preconditions. The failure mode is treating "all four green" as equivalent to "feature confirmed working." It isn't: all four are orthogonal to the actual behavior. A commit can pass every single one of them and still have the flagged feature completely broken in the live UX. If a status report cites only these and never a captured screenshot, DOM query result, or store value, treat the checkpoint as unverified regardless of how confident the report sounds.

Two concrete shapes this takes, seen independently of each other:

- Patch-on-the-wrong-path: an additive code change (e.g. "return an extra field when a condition is false") lands in a function that *looks* like the one serving the live request, but the actual runtime dispatcher builds its response independently and never calls that function. The static diff is real, the deploy succeeds, the bundle contains the change — and production behavior is byte-identical to before, because the executed code path was never touched. This survives days in production because every static check (code merged, tests for the changed function pass, deploy healthy) stays green; the only thing that would catch it is inspecting the actual live response payload for the new field. - Subscribe-only store with no hydrate: a reactive store or listener that reacts correctly to *new* events but was never given a "fetch current state on mount" step. If the event of interest was published before the component mounted — the completely ordinary case of a user refreshing a page after something already happened — the store stays empty forever, with zero errors anywhere. All static checks (bundle contains the store, env flag is on, backend already has the data) pass; only "hard refresh, then look at what's rendered" reveals the gap. See Subscribe-only store — invisible to whoever mounts after the event already fired for the reactive-framework-specific version of this same bug.

How it's exploited (used as a diagnostic/verification technique)

Turn the anti-pattern into a checklist. To declare a feature flag (or any conditional runtime path) genuinely green, run all four steps — skipping any one of them is exactly where the false-green cases above slipped through:

``` 1. SEED — create the real initial condition in data ├── Not a mock: an actual persisted row / cache key / session state │ that matches what a real user's prior actions would produce └── Verify the seed took effect (read it back) BEFORE activating anything

2. ACTIVATE — flip the flag via the REAL runtime mechanism ├── If it's an env var, set it the way production actually sets it │ (container recreate, not just export in a shell you'll discard) ├── If it's a per-request/localStorage toggle, set it that way └── Re-read the flag from inside the running process to confirm it was actually picked up (not just present in the shell that launched it) — this is where build-time-baked vs runtime env divergence hides (see below)

3. VERIFY — a BEHAVIORAL marker, not a structural one ├── Headless browser: assert on a DOM element/class/attribute that only appears when the feature actually renders, not on HTTP status ├── OR: read the actual reactive store/state value at runtime ├── OR: query the persisted response payload directly (DB/API), not the code that theoretically produces it └── Compare ON vs OFF: the marker MUST differ between the two runs. If it doesn't, you're not testing the flag, you're testing noise.

4. NEGATIVE — flag OFF must not have moved either ├── Same seed, flag OFF (the default) └── Expected: legacy behavior is byte-identical to pre-change. This catches the "flag is dead code, nothing ever routes there either way" case, which SEED+ACTIVATE+VERIFY alone would miss if VERIFY was accidentally checking something both branches share. ```

Two multiplying factors that make this worse and are worth checking explicitly during step 2:

- Build-time vs runtime environment variables. A framework that supports both a build-time-baked env import and a genuinely-runtime one (SvelteKit's $env/static/public, injected and dead-code-eliminated at build time, vs $env/dynamic/public, read from the actual running platform) can have a flag that reads the *static* import: recreating the container with a new env var value changes nothing, because the value was already frozen into the bundle at the last build. docker exec ... env | grep FLAG shows the new value in the process environment — and the bundle still uses the old one. Check which import the code actually uses, not just whether the platform environment updated. - "Container healthy" is a liveness signal, not a correctness signal. An orchestrator's health check (the container equivalent of a Kubernetes liveness/readiness probe) confirms the process is up and accepting connections — it says nothing about whether a specific feature path inside that process is wired correctly. Treat "Up (healthy)" as a precondition to start testing, never as a substitute for it.

Testing methodology detail that compounds with this: if the system is multi-session/multi-tenant, run the SEED/ACTIVATE/VERIFY/NEGATIVE cycle against at least 2-3 parallel sessions on different tenants/domains, not one at a time — a single-session run can pass all four steps and still miss a cross-session data-isolation bug that only a concurrent, different-tenant run exposes. See Run 3 fresh sessions in parallel on different domains — the only test topology that catches cross-session pollution and concurrency bugs and, for the underlying storage bug shape that this specific test setup is designed to catch, A cache/lookup keyed only on domain (no session id) silently returns another session's data.

What doesn't work

- "It's in the bundle, ship it." Grepping the compiled output for a marker string proves inclusion, not execution. If the marker is inside a function that the live dispatcher doesn't call, this check is permanently green and permanently wrong. - Testing only the OFF path when OFF is the default. If a flag defaults to OFF and you only ever verify the OFF path (because that's what's already live and "known good"), you've verified nothing about the flag — it could be complete dead code, unreachable from any real trigger, and this test suite would never notice. - Trusting "HTTP 200" as a proxy for "renders correctly." Server-side rendering can return 200 with the *default/legacy* branch even when the flagged ON branch is broken client-side — the status code reflects that a response was produced, not which branch produced it or whether the client-side hydration of that branch succeeded. - Trusting the platform's reported environment over what the bundle actually reads. Confirming FLAG=true in the container's process environment is necessary but not sufficient if the framework has a build-time-baked variant of environment access that the code might be using instead — the two can silently diverge after any deploy that changes env without a full rebuild. - Assuming a listener that reacts to new events also covers events that already happened. A store or subscription that never does an initial state fetch on mount will look completely correct in every test where you activate the flag *and then* trigger the event in the same session — and will fail silently for the ordinary case of a user who refreshes after the event already fired. Only an explicit "arrive at an already-active state, then check what renders" scenario catches it — see Subscribe-only store — invisible to whoever mounts after the event already fired and, for the same root shape in a server-to-client streaming context, A long-running tool's follow-up turn is generated server-to-server and never streamed to the browser — the UI needs an F5 to show what already happened. - Declaring victory from a single-session test run. A feature that touches shared state (cache, lookup keyed only by a coarse identifier, session data) can pass every SEED/ACTIVATE/VERIFY/NEGATIVE check run in isolation and still be broken the moment two sessions run concurrently — isolation and concurrency bugs are, by construction, invisible to a test methodology that never runs more than one instance at a time.