Hardcoded 'reasonable' defaults baked into an LLM prompt or pipeline become systemic bias or silently go stale

A value that looked like a safe, reasonable default when one engineer wrote it for one client — a numeric fallback, a worked example, a literal year — keeps being read by every prompt call after that, for every tenant, forever, until either it actively distorts output for anyone it wasn't written for (bias) or the world moves on and it becomes silently wrong (staleness). This root cause recurs: one team hit it four separate times in one multi-tenant LLM pipeline, at four different layers, and it was the single largest source of "the output looks plausible but is wrong" incidents.

The signal that gives it away

- A comment in the code that says, verbatim or in spirit, "applicable to all industries" / "common defaults" / "works for everyone" next to a block of numeric constants. That phrase is the whole bug in one sentence: whoever wrote it was reasoning from the one segment they had in mind when they wrote the block, and the label describes an intent ("generic"), not a verified property. Grep for it; it is a near-perfect predictor. - Branch-based domain/segment detection implemented as `if "keyword" in category.lower()` rather than an exact-match lookup. Substring detection silently fails for any input that doesn't contain the expected keyword and falls through to an else/default branch — and if that default branch is the "universal" block above, the failure is invisible: no exception, no log, just wrong numbers flowing downstream as if they were real. - The fallback path is exercised almost every call, not as a rare edge case. If the upstream data source (a discovery step, a user profile, an intake form) only rarely populates the specific fields a calculation needs, the "default" stops being a fallback and becomes the primary de facto input — nobody designed it to run 95% of the time, but that's what "reasonable default + spotty real data" adds up to in production. - Output that mixes vocabulary/framing from a domain the customer is not in — security-incident language for a connectivity reseller, hospitality occupancy metrics for a professional-services firm — while every individual number in the output looks internally consistent and well-formatted. The tell isn't a crash, it's domain mismatch that a domain expert catches on read but a schema validator never will, because the shape is fine — only the content is fabricated. - A single worked example (one industry, one persona, one pricing model) embedded in a prompt, docstring, or few-shot block that's shared across a multi-vertical product. Even one hardcoded example is enough to anchor the model toward that vertical's vocabulary and assumptions for every other customer that prompt serves — this isn't a hypothesis, it reproduces reliably enough that "3–4 examples spanning different verticals" is the standing fix, not "0 examples." - A literal year (or year range) as a string inside a search query, prompt instruction, or filter ("2024 2025"). It is correct exactly until the calendar turns over, then silently wrong for every call thereafter — no error, no crash, just queries that quietly stop matching what "current" or "recent" was supposed to mean. - A prescriptive psychological/behavioral rule baked into scoring or prompt logic as if it were universally true (e.g. "this persona always responds better to X framing", "always lead with the risk/loss angle") rather than as a hypothesis to be evidenced per case. If a scoring function gives a flat bonus for matching one framing/angle regardless of whether the underlying claim actually supports it, that's the numeric-default failure mode wearing a different (behavioral, not financial) costume — the audit signal is the same: one output style dominates the numbers when you tally a real batch (e.g. one framing shows up in nearly 100% of generated items instead of a spread).

How it's exploited (i.e., how you find, prove, and fix it)

1. Prove the bug with a real batch, not a code read. Code review alone won't convince anyone that a "reasonable default" is actually distorting output — the fix started from generating a real batch for a customer outside the anchor vertical and counting: a connectivity reseller's campaign came back with security-incident anchors (mean-time-to-detect improvements, breach-cost-avoidance framing, compliance-audit savings) attached to fiber and PBX products. The literal numbers looked professional — currency-formatted, plausible magnitudes — which is exactly why they were dangerous: nothing about their *shape* signaled "fabricated."

2. Trace root cause to the compounding chain, not a single line. In one observed case it was three failures stacking: - Segment/industry detection by substring match ("hospitality" in industry.lower()) that silently mis-routes any category string it wasn't written against. - One giant fallback block labeled as applicable to every segment, which is exactly what absorbs every substring-match miss. - Real per-tenant data being sparse enough that the fallback fires on the large majority of runs — turning a "just in case" default into the dominant input. Each layer alone looks defensible in isolation; together they guarantee fabricated domain-specific numbers reach the LLM prompt as if they were verified facts, and the LLM (correctly, from its point of view) cites them as given data.

3. Remove the magic values; replace with a grounded-only contract. The durable fix is architectural, not a patch to the substring match: no industry-specific numeric literal may live in application code at all. Every number that reaches the prompt must carry (value, unit, source_field, confidence) traceable to something the pipeline actually retrieved for *that* tenant — product pages, case studies, a discovery step's structured output — never a constant. Rank the sources so the prompt prefers the most grounded ones first, e.g.: 1. vendor's own stated product features (confidence ~1.0) 2. the vendor's own case-study results (confidence ~1.0) 3. industry benchmarks surfaced by a grounded search step for this tenant specifically (confidence ~0.85) 4. broader industry benchmarks from a less targeted discovery pass (confidence ~0.80) 5. value-proposition/differentiator claims the vendor makes about itself (confidence ~0.75) 6. competitive-positioning claims (confidence ~0.65)

Instruct the prompt explicitly: *use only numbers present in this sourced list; if a parameter has no source, omit that output entirely rather than inventing a plausible-looking number.* Grounding — tying every generated claim back to retrieved, sourced data — is exactly the mechanism mainstream LLM platforms ship to fight this class of fabrication; the pattern here is the same principle applied at the prompt-construction layer instead of relying on a platform feature.

4. When grounded data is genuinely insufficient, fail honest, don't fill the gap with scaffolding. Layer the recovery cheapest-first: (a) re-read data already fetched for this tenant, zero extra cost; (b) if still short, run a small number of additional targeted, cheap lookups (capped, e.g. at 2–3 per generation, to bound cost); (c) if still short of a minimum viable output count, return an explicit "insufficient grounded data" result rather than padding the output to hit a target volume. An honest "not enough real signal for this tenant" result beats a full-looking output built on invented numbers — this is the trade-off worth defending to stakeholders who want volume: a smaller, real batch is worth more than a full, fake one.

5. If a segment benchmark genuinely is useful (sparse-data tenants sometimes benefit from "typical uptime for your segment is X"), externalize it — don't re-embed it in code. Move segment benchmarks into an editable registry (e.g. a YAML file: segment_slug → {parameter: {value, unit, source}}) looked up by exact match against a normalized segment key, returning empty for anything not explicitly listed — never a substring in check, and never a catch-all branch. This keeps the number of "invented for you" values at zero for any segment not explicitly and deliberately curated, while still allowing product/marketing to add curated benchmarks without a code change.

6. Write an automatable lint check for the whole class, not just the one bug found. A single grep-able hit list of the banned parameter names (the specific numeric fields that must never appear as a hardcoded default or literal assignment in source) turns finding one instance into preventing all future instances of this shape: ``bash BANNED_PARAMS=(breach_probability avg_breach_cost mttd_days gdpr_avg_fine \ occupancy_rate adr_typical patient_record_count billable_rate_typical) for p in "${BANNED_PARAMS[@]}"; do grep -rnE "setdefault\(\"$p\"|\"$p\"\s*:\s*[0-9]" src/ && echo "BANNED default '$p' found" done `` Run it in CI or a periodic lint pass; any hit is a regression of the fix, not a new bug to re-diagnose from scratch.

7. Separately: neutralize single-vertical examples wherever prompts are shared across tenants. Grep prompts, tool docstrings, and few-shot blocks for the tell-tale shape of a narrative example (a specific competitor name, a specific pricing model like "monthly subscription", a specific persona like "20–50 employee firm"). Replace with either (a) a neutral placeholder (<PRODUCT_NAME>, <COMPETITOR_X>, {{segment}}, {{price}}) or (b) 3–4 examples spanning genuinely different verticals instead of one — a single example is a majority-label-style bias magnet even when nobody intended it as "the" example. Crucially: a numeric/string default is not the same thing as a narrative example, and only the latter needs neutralizing. Keeping a default like "target segment: SMB" is fine and doesn't need multi-vertical treatment as long as it's never dressed up as a worked example the model can pattern-match a whole scenario from. Measured effect of doing this de-anchoring for one non-anchor-vertical customer: qualitative output wording went from visibly anchored to the original vertical's vocabulary to neutral, and a downstream confidence score moved from 0.71 to 0.76, with run time unchanged — i.e. the fix cost nothing and only removed bias, it didn't trade off quality.

8. Separately: make every literal current-year/date reference in a query or instruction computed, never typed. Any query string with a year baked in ("2024 2025") is a bug that hasn't triggered yet. Different query intents need different offsets, which is itself worth encoding explicitly instead of using one variable everywhere: ``python from datetime import datetime now_year = datetime.now().year yr_fiscal = f"{now_year - 1} {now_year - 2}" # financial statements: this year's often isn't filed yet yr_recent = f"{now_year} {now_year - 1}" # news / awards / funding: want the freshest `` The reason two different offsets matter rather than one "current year" constant: a query for financial statements/revenue that uses the literal current year will typically retrieve nothing, because most jurisdictions' filings lag several months behind the calendar year — so "most recent" and "current" are not interchangeable even within the same fix.

9. Separately: if a psychological/behavioral framing rule is prescriptive rather than evidenced, replace the prescription with a diversity requirement plus mandatory evidence. Where a scoring or prompt rule hardcodes "persona X always gets framing Y", replace the hard mapping with (a) a required minimum diversity across framings/angles in any batch (e.g. "no single framing may exceed ~25–30% of output, and at least 2–3 distinct framings must appear"), and (b) a mandatory rationale field tying each chosen framing to something actually evidenced about *this* tenant, not to the persona-category lookup table. Measured before/after on a real batch: one framing went from effectively 100% of generated items to 14% (against a self-imposed ceiling of ≤30%), with all originally-intended framing categories represented at least once — proof that "diversity requirement + evidence field" is enough to break a systemic default without needing to ban the framing outright (it's still allowed, just no longer the unexamined default).

What doesn't work

- Patching the substring-match bug alone and declaring it fixed. Fixing "hospitality" in industry to something more precise still leaves the universal fallback block in place for every case the matcher doesn't handle — and a fallback block that's "applicable to all industries" by construction will always have some path that reaches it. The fallback block itself, not just the router in front of it, has to go. - Env-flag-gating the magic defaults instead of removing them. Keeping the hardcoded values behind a toggle ("disabled by default, but the code and the numbers are still there") preserves the tech debt indefinitely and it takes exactly one accidental flip, or one new call site that doesn't know about the flag, to reintroduce the bias. Removing the values from the codebase entirely is the only version of the fix that can't regress silently. - A generic environment-agnostic registry with substring/fuzzy matching "to be more flexible." Any lookup that isn't exact-match on a normalized key reintroduces exactly the original failure mode one layer down — a segment name that's close-but-not-identical to a registry key will either match the wrong entry or fall through to a default, and both outcomes are indistinguishable from the pre-fix state to whoever reads the output. - Disabling output generation entirely whenever grounded data is thin, as an overcorrection from "too much was being fabricated." Too aggressive for a system where partial-but-real output has legitimate value; the better balance is: retrieve what's real, generate less if that's all there is, and only hard-fail when the output would drop below a minimum viable count — not whenever any single parameter lacks a source. - Trusting that "the default is documented in a comment" prevents bias. A comment explaining that a number is a fallback doesn't stop the LLM from citing it as fact once it's in the prompt as a value — the model has no visibility into code comments, only into the text of the prompt it receives. Documentation is for the next engineer, not a runtime safeguard. - Treating "the one bad example found was removed" as done. Single-vertical anchoring tends to be scattered across many surfaces at once — tool docstrings, agent instructions, gap-filling templates, natural-language examples in a few different tools — because each was written independently by whoever built that surface, all reaching for "the example I have in mind." A one-file fix reliably misses siblings; treat it as a cross-cutting audit (grep for vertical-specific keywords across every prompt-adjacent file), not a single edit. - Assuming a numeric default and a narrative example need the same fix. Stripping out a reasonable numeric default (e.g. an assumed budget or target-segment string) "to be safe" alongside the narrative examples breaks working flows for no bias-reduction benefit — the two are different risks (numeric defaults barely influence the LLM's vocabulary/framing choices; narrative examples are what actually anchors them) and conflating them leads to over-deleting things that were fine. - Assuming a behavioral/framing fix is done once the numbers look better in one sample batch. A framing distribution that looks diverse in a single spot-check batch can regress the moment a new prompt path or a new content type is added upstream, because the diversity requirement has to be enforced structurally (in the scoring function and prompt rules), not just observed once. Re-check the distribution on a fresh batch after any change to the surfaces that feed that scoring/prompt path.

Related

- An LLM occasionally regurgitates a prompt's own XML wrapper tags into the value it was asked to produce — a different prompt-authoring failure from the same family: there the prompt's own scaffolding leaks into output post-hoc; here a stale or biased assumption baked into the prompt or its inputs leaks into output as if it were fact. Both look fine in prompt review and only degrade a specific way in production. - A gate classifier exceeds its mandate — and overwrites a score that was already computed — another case where an LLM's runtime behavior quietly diverges from what the prompt author assumed it would do, requiring a guard the prompt's wording alone can't provide. - An LLM occasionally collapses a nested dict into a plain string — and the real shape it produces differs from the shape a downstream reader assumed — sibling lesson from the same audit effort ("never trust the LLM/upstream data blindly"): there the danger is a shape assumption (nested dict collapsing to a string) breaking traversal; here it's a content assumption (a default value being true for every tenant) breaking correctness. Same discipline — assume less, verify more — applied to two different failure shapes.