A paid API response gets parsed into something useful, then never written anywhere that outlives the request

You pay for an external call — search, scraping, a grounded LLM call, a registry lookup, an enrichment API — parse something real out of the response, and then that value dies before it reaches any store with retention longer than the request: a wrong dict key, a thing[0] on a list, a cache key with a TTL, a continue that also drops unrelated sibling data. The API bill is real; the asset it bought never exists anywhere durable.

The signal that betrays it

Look for these shapes; a mature data pipeline that pays for external calls usually has several at once, not just one:

- A destination column already exists and is already read by downstream code, but a population-rate check shows it empty or near-zero across a real sample, while the upstream code that should fill it is actively calling — and paying for — the exact API that would. A direct measurement showed this: a field meant to hold a verified email address stayed at zero across more than fourteen thousand populated records, while a sibling field on the same table, populated only through a different, unrelated path, was nonzero for a few thousand of them. The gap between "some path populates this" and "the expensive path populates this" is the number that matters — zero for the expensive path specifically, not zero overall, is what proves the paid call's own output is the thing being lost. - A field is written under one key name and read under a different one, with no exception anywhere: a dict .get() returns None, an ORM column defaults to null, everything "works" because the always-missing value fails soft. This exact shape has turned up twice in the same codebase, once where a merge step wrote a value under one key while the persistence step queried for a different, superficially-similar key, resulting in zero populated rows out of well over a hundred thousand despite the merge running successfully every single time. - Only the first element of a list survives. A response structurally returns several candidates — phone numbers, emails, snippets, search-result titles — and the code takes index [0] (or slices to a small N) into a variable that is the only one that ever gets serialized onward; the rest lived in a local variable for one line and is gone. The fix pattern that already exists nearby in a codebase that got this right once is the tell: a sibling field solved the same shape as a plural array (thing_all[]) right next to the still-broken singular one. - A cheaper-tier response already contains a field sufficient for the immediate need, but the code keeps only one identifier from it and re-pays a costlier call later for data the first response already had. Preview/summary fields from a paid search step are a common casualty — discarding everything but an internal id and then re-purchasing a full "enrich this id" call is paying twice for information you already received once. - The only durable-looking home for a paid result is a cache key with a TTL, while the schema already has a permanent row designed to hold it (the migration ran; nothing writes to it). Check the *actual remaining TTL* on live keys before deprioritizing this — a "30 days, plenty of runway" TTL sounds safe right up until you check a live key and find it is already 29 days into its 30-day life with no backfill job on the calendar. A cache is not a system of record: by design it should be safe to lose, and a durable acquisition sitting only in something safe-to-lose is already most of the way to lost. - A `continue`/`else: skip` written to handle one bad condition on the primary entity has the side effect of skipping persistence for unrelated data bundled in the same iteration. Measure the blast radius before deciding this is cosmetic: in one real batch, a discard keyed on "no stable id matched" for the primary entity also silently dropped every person record that had been correctly extracted alongside it — on the order of a hundred people lost through fewer than a thousand discarded primary records. The discard condition and the collateral loss are two different bugs that happen to share one continue. - A cap exists between "how much you're billed for" and "how much you keep," and the two are not the same cap. A call that classifies or scores every candidate and only afterward slices the result down to a small N still billed for classifying all of them — the cap saved nothing, it just decided what to throw away after paying for everything. Contrast with a cap that's scoped purely to what gets replayed into a follow-up prompt for token-budget reasons: that one is legitimate to keep on the *prompt*, but has no reason to also apply to what gets *stored*, since storage doesn't touch the billed call again. - A flag threaded end-to-end through the pipeline that is supposed to select between two code paths, where a runtime check across real entrypoints shows only one path is ever actually taken. The alternate branch and the flag both survive in the code, looking configurable, configuring nothing. - The single most reliable audit question for a mature pipeline: for every call that costs money, "which table/column holds the answer to this call two years from now?" If the honest answer is "nowhere" or "a cache key that already expired," you've found an instance.

How you exploit it (fix it, in the order that's actually safe)

1. Audit mechanically, call site by call site. For each paid call, write down three columns: what's requested, what's parsed out of the response into a variable, what of that reaches an actual write to a store with retention longer than the request. Anything present in column two but missing from column three is the leak, and it will almost always turn out to be one of: wrong key on read, first-of-array kept, or a destination column that exists but whose writer was never wired up. 2. Fix additively, never by renaming. The destination usually already exists — a migration ran ahead of the write code, or a sibling field of the same shape was already solved correctly nearby; copy its pattern rather than inventing a new one. If a field name was wrong on the read side, alias both the old (broken) and the new (correct) name in the output for at least one full deploy cycle — a downstream consumer that already reads the old, always-empty name as "normal" should not silently change contract on you the same day you fix it. 3. Turn single-item survivors into arrays, without breaking the field that already exists. If a response naturally returns several of something, add thing_all[] next to the existing single-value field instead of changing that field's type or meaning. 4. Move TTL-only data into the durable row on the same request that populates the cache, not as a background migration "for later" — "later" is bounded by the TTL, and it is shorter than it feels. Measure the actual remaining TTL on live keys before deciding priority. 5. Split the discard. Where a continue drops a batch because one condition on the primary entity failed, persist the primary entity through whatever fallback key is stable (a domain, a name-hash — anything short of the missing id) instead of dropping the record outright, and move persistence of sibling records (people, contacts, line items) out of the discard's blast radius entirely — they don't depend on the condition that triggered the skip and shouldn't die with it. 6. For a grounded/search-augmented paid LLM call specifically, persist the evidence trail, not just the answer. The response carries which queries were actually issued and which sources backed each claim at no extra cost over the call you already paid for; store it as a generic evidence[] (source, fetch date, snippet) rather than baking it directly into fixed category booleans. This matters independently of the waste problem: a boolean answers only the one question you thought of at write time and then discards the evidence that could answer a different question later; the generic evidence blob answers today's question and keeps every future one open, for the same money already spent. 7. Kill dead branches all the way, don't leave them half-plumbed. If a runtime check across real entrypoints confirms one branch of a threaded flag is never actually taken, remove the flag and the dead branch together, or wire the flag for real — a flag that looks configurable but isn't is worse than deleting it, because the next person trusts it. 8. Sequence fixes by risk, not by size of the number. Pure deletions (unused request fields, confirmed-dead branches) first — nothing downstream reads them, so nothing can break. Persistence gaps (writing what's already computed into a row that already exists for it) second — additive, and immediately verifiable by re-running the population-rate check. Recovery of currently-discarded data (truncated arrays, dropped sibling records) third — confirm there's a real downstream consumer before building the pipe, or you've only moved the waste one hop closer to storage. Cap and vertical-flag cleanup last — the lowest risk of further data loss, and closer to a modeling decision than a bug fix.

What does NOT work

- "It'll get backfilled later" for anything living only in a TTL'd cache. Later has a deadline you didn't put on your calendar — the cache's own expiry. Live keys checked near the end of their TTL window with no backfill job scheduled show what "later" actually means in practice: "never," just with an alarm nobody was watching. A cache's own contract is that losing its contents is safe — treating it as the record of a purchase inverts that contract silently. - Calling a key-mismatch bug fixed once you rename the read side to match the write side, without checking who already reads the old name. If a dashboard, export, or downstream consumer already treats the old, always-empty key as normal, a silent rename doesn't break anything today but changes the contract without telling anyone. Alias both names for a full deploy cycle. - Assuming every cap that looks the same in a diff has the same cost consequence. Two [:N] slices can read identically and mean opposite things: one bounds what you're billed for, the other only bounds what you keep from something you were billed for either way. Checking which is which requires reading the vendor's own billing unit, not the shape of your code — Google's Places API, for example, bills the *entire request* at whichever pricing tier its single highest-tier requested field belongs to, so trimming a field from a cheaper tier than the most expensive one you're still requesting reduces payload size, not your bill. Spend the optimization effort on the field that actually crosses a tier boundary, not the one that's easiest to spot in a diff. - Fixing the discard condition without checking what else shares its branch. A continue written to skip one bad case often shares a code path with unrelated writes for sibling records; narrowing the condition alone leaves the collateral loss exactly where it was — the unrelated writes have to move out of the discard's blast radius, not just get a narrower trigger to dodge. - Baking a paid, generic evidence signal directly into fixed category booleans "to save a step." It looks efficient — one flag instead of a stored blob — but the flag discards the sentence that justified it: nobody can re-derive a different, better question from a true/false later, while the same money, spent on storing the evidence generically, answers today's question and leaves every future question about the same data still answerable. - Treating "the request field is unused" and "the response field is discarded" as the same bug. They look alike in a diff (something present, unused) but have opposite fixes and opposite priority: an unused *request* field is pure waste to delete; a discarded *response* field is a purchase you already made that a persistence fix can still recover. Auditing them in the same pass without separating them tends to make people delete the wrong one or "fix" the cheap one and declare the audit done.

Related

Check what you already own before paying again for the same external query — the other half of the same coin: check what you already have before paying again. This page is about what to do with what you just paid for, so that check actually finds something the next time.

Coarse boolean completion gates hide partial completion — sibling failure mode: here the paid result is correctly scoped and simply thrown away; there, the completion-tracking itself is too coarse to notice a gap exists in the first place. Together they cover most of the "you paid for it, now what" problem.

Soft vs. hard cascade invalidation: when 'confirm, then delete downstream' destroys data you paid for — the eviction-side version of the same money: an aggressive TTL or cascade can destroy an expensive-to-recompute value even after it was correctly persisted once, which is the failure this page's fix #4 is trying to get ahead of.

The general shape here is "no errors, no warnings, quietly incomplete forever," produced by discard/mismatch/TTL causes — a sibling failure mode reaches the same symptom instead through a batch cap combined with whole-universe progress state.

Pydantic V2's default `extra='ignore'` drops unexpected fields silently — no error, no warning, the data just never arrives — a different mechanism with the same visible symptom: fields silently vanishing with zero exceptions, because a validation layer's default is to ignore what it doesn't recognize rather than to fail loud.