401 on a resumed session: a helper preferred the expired token snapshotted in state over the fresh one in Redis

A server-to-server auth helper read a JWT from two possible sources — a snapshot persisted in durable session state, and a copy refreshed on every request in an external cache — and picked the durable-but-stale one first, so any resumed long-lived session sent an expired bearer token downstream.

The signal that gives it away

A 401 Unauthorized fires on an internal, service-to-service dispatch (not on a user-facing login or UI action), and only on resumed sessions that are old enough — a session created and then left alone past the token's lifetime, then picked back up. Freshly created sessions never show it, because their state-snapshotted token hasn't had time to expire yet. That age-gated pattern is the tell: if the same operation, invoked from a code path that reads credentials from a live, continuously-refreshed source rather than from session state, does *not* 401 on the exact same old session, you've confirmed the split — one code path is stale, the sibling path is fresh, and only one of them touches durable state.

The misleading part is that the failure *looks* like a user-auth problem (a 401 always reads that way at first glance) when it's actually purely internal: the human's login is fine, the UI is fine, and the bug is entirely inside a backend dispatch that talks to another backend service using its own bearer token, sourced independently of whatever the user is doing.

Concretely, decoding both candidate tokens side by side makes the bug undeniable: the one pulled from persisted session state has an exp in the past; the one pulled from the refreshed external store has an exp in the future ([RFC 7519 §4.1.4](https://datatracker.ietf.org/doc/html/rfc7519#section-4.1.4) requires that "the current date/time MUST be before the expiration date/time listed in the exp claim" — the same section also lets implementers allow "some small leeway, usually no more than a few minutes," for clock skew, so a token just past its nominal exp isn't automatically invalid. That leeway doesn't change the diagnosis here: the state-snapshotted token was stale by however long the session had sat idle — hours or days, not minutes — so it was well outside any reasonable skew tolerance, and there's no ambiguity once you see the two payloads next to each other).

How to exploit it (diagnose and fix)

1. Find every place the bearer token for a server-to-server call is sourced, and rank them by freshness guarantee, not by which one looks more "official." A common instance has two: (a) a value written into session state once, at session-creation time, that a helper function returned as first choice; (b) a per-request-refreshed cache entry (Redis key scoped to the session ID, TTL matched to the token's real lifetime, rewritten by a middleware on every inbound request) that the same helper checked only as a fallback if (a) was absent. The helper's own docstring already said the cache was the primary source — the code just didn't do what the docstring claimed. Read the code, not the comment, when auditing this kind of helper.

2. Reproduce deterministically before touching anything: take a session old enough that its state-snapshotted token's exp has passed, confirm the cache-refreshed copy for the same session is still valid (a request today refreshed it), then drive the code path to the point where it dispatches the server-to-server call. Pre-fix: 401, node crashes. Post-fix: call succeeds. This is a clean, deterministic repro — no timing flakiness — because it depends only on session age, not on race conditions.

3. Invert the priority, don't just add a fallback: make the freshness-guaranteed source (the one a middleware rewrites on every request, keyed by session ID, with a TTL that matches the credential's real lifetime) primary, and the durable-state snapshot the fallback *only* for the case where the fresh source is empty (e.g. cache eviction, cold start) — never the reverse. A snapshot in durable session state is, by construction, however old the session is; a credential with a shorter lifetime than "however old this session might get resumed" cannot safely live there as the preferred source. General framework confirmation: ADK's own docs are explicit that session state is meant to be durable and survive restarts (DatabaseSessionService / VertexAiSessionService persist it reliably) — durability is exactly the property you do *not* want for something that expires ([ADK session state docs](https://google.github.io/adk-docs/sessions/state/); ADK's dedicated authentication guidance likewise treats credential handling as its own concern, separate from general state — [ADK authentication docs](https://google.github.io/adk-docs/tools-custom/authentication/)). The general caching-literature framing is the same idea from the opposite direction: don't cache (or, here, "durably snapshot") data that must stay accurate in near-real-time, and match your refresh/expiration policy to the actual access pattern rather than to convenience ([cache-aside pattern, staleness guidance](https://learn.microsoft.com/en-us/azure/architecture/patterns/cache-aside)).

4. Sanity-check every other place in the same codebase that reads the "same" credential, because asymmetric behavior between sibling code paths is itself a diagnostic signal, not just a fix target. In one observed case, a second, older coordinator-style path that dispatched the equivalent operation as an LLM tool call never showed the bug — because it pulled its token from the live per-request tool context instead of from session state. That asymmetry was what first proved the state-snapshot path, not JWT validity in general, was the problem: same credential *type*, same downstream service, different source, different outcome.

Related: this is a narrower, credential-specific instance of a broader class — see A resumed old session redoes expensive work from scratch — the idempotency guard was reading a short-TTL artifact for the mirror-image failure (a *cache* TTL set too short, discarding still-good state) and Idempotency guard with Redis SET NX — fire the dispatch once, never on retry-count alone for another case where the right fix was "read the invocation-independent external store, not anything scoped to this run/session." Both are instances of the same underlying question this page answers in the negative direction for: which source of truth should win when a long-lived session state and a short-lived external store disagree.

What doesn't work

- Trusting the docstring/comment over the code. The helper's own documentation already stated the correct priority (fresh cache first); the implementation silently did the opposite. A code review or a quick grep confirming "does the code actually match what it says it does" would have caught this before it shipped — reading the prose next to a function is not the same as reading what the function returns. - Treating the 401 as a user-auth bug first. Because the error code is 401, the natural first instinct is to look at login, cookies, or the user's own session — none of which were involved. The dispatch was purely internal (backend calling backend), and time was lost checking the wrong layer before noticing the failing call target was another internal service, not the identity provider. - "It works for new sessions" as evidence of correctness. The bug is invisible on any session younger than the token's lifetime, which makes it easy to ship, pass casual QA, and only surface in production once real sessions age past that window. A regression check for this class of bug has to explicitly construct an *old* session and force a resume, not just exercise the happy path on a fresh one. - A single fallback without inverting priority. Simply adding the fresh source as a secondary check without making it primary reintroduces the exact bug for every session old enough to have both a populated (but stale) state snapshot and an available fresh source — the stale one still wins because it's checked first. The fix has to flip which source is authoritative, not just add another source to consult.