An LLM occasionally collapses a nested dict into a plain string — and the real shape it produces differs from the shape a downstream reader assumed

An LLM's structured output cannot be trusted to keep the exact nesting, field names, or dict-vs-string shape a downstream reader assumes — sometimes it collapses a nested object into a bare string, sometimes the object is a dict but with different keys, nesting depth, or list-vs-dict shape than whoever wrote the reader expected; both variants render a consumer silently empty or crash it a few lines below where the real cause lives.

The signal that gives it away

Two distinct failure signatures, both traced back to the same root cause (trusting an assumed shape instead of the real one):

- The crash variant. A chained traversal like obj.get("a", {}).get("b", default) throws 'str' object has no attribute 'get'. The stack trace points at the traversal line, but the traversal line is not where the bug lives — the bug is upstream, wherever the LLM decided to return avg_breach_cost: "€180k" instead of the expected avg_breach_cost: {"value": "€180k", "unit": "EUR"}. Grep for the pattern across a service and every hit is a *candidate* crash site, not a confirmed one — it only fires the day the model happens to collapse that particular field: ``bash grep -rnE "\.get\([^()]*\)\.get\(" services/auth/ | grep -v __pycache__ | wc -l ` - **The silent-empty variant — no exception anywhere.** This is the more dangerous shape because nothing crashes. The generation step logs a clean success (competitors=5 counter_triggers=4 kw_sources=[...]), the write to storage succeeds, and yet the read path that's supposed to consume that data returns None or an empty block downstream (for example, an XML <product_facts> block injected into a prompt, always empty). Zero errors, zero warnings, at any layer. The only tell is a mismatch between "the producer logged success" and "the consumer has nothing" — which is easy to misread as "the producer didn't actually run" when it did, correctly, and the break is entirely in what the consumer assumed about where/how the producer's output was stored. - **The silent-empty variant is rarely a single bug — it's usually 2–3 independent wrong assumptions stacked, each one alone sufficient to produce the exact same "always empty" symptom:** 1. *Wrong key/path assumed.* A reader assumed a value lived at a standalone key (prospect_input:{session_id}), but the writer actually nested it three levels down inside a differently-keyed, timestamped record (company_profiler:{sid}:{dh}.full_profile.prospect_input.simple_product). The standalone key never existed — not "was empty," genuinely absent. 2. *Wrong nesting level assumed.* Even once you're reading the right root object, new fields can land as nested children of an existing sub-object (simple_product.enriched.competitive_analysis) rather than as siblings of it (simple_product.competitive_analysis) — a reader written against the sibling assumption finds nothing, silently, at every single call. 3. *Wrong field name or wrong list-vs-dict shape assumed for a field that does exist at the right path.* The value is there, it's even a dict — but the key someone assumed doesn't match the key the model's actual output schema uses, or a field assumed to be list[str] is actually list[dict], or a field assumed to be list[str] is actually a single dict with named sub-keys. Each mismatched field independently renders as "missing" with no error, because a .get() on a wrong key just returns None` and code carries on. - Because all three assumption-mismatches produce the identical downstream symptom ("this field is always empty/None"), fixing the first one you find and declaring the incident closed is a trap: the other two are still silently starving the same consumer, invisible until someone checks the *rest* of the payload too.

How it's exploited

For the crash variant, the fix is a single reusable safe-traversal helper, applied at every chained-.get() call site instead of patched ad hoc:

```python def _safe_get(obj, *keys, default=None): cur = obj for k in keys: if isinstance(cur, dict): cur = cur.get(k) else: return default return cur if cur is not None else default

# usage: _safe_get(profile, "team_structure", "total_employees", default=0) ```

The discipline that matters is checking isinstance(cur, dict) at every step of the walk, not just the first — a collapse can happen at any nesting level, and a helper that only guards the outermost .get() still crashes on the second one. In one real refactor this pattern was applied at roughly 70 call sites across a single file after an audit — the collapse isn't a one-off, it recurs anywhere an LLM populates a field that's *sometimes* structured and *sometimes* plain text (a price, a percentage, a short verdict are exactly the fields models like to "simplify" into a string when they're confident about the value and less confident about the wrapper).

For the silent-empty, multi-assumption variant, the fix is not "find and patch the one wrong .get()" — it's auditing the full path against the *real* payload, in this order:

1. Read path first. Don't assume the key you expect is the key that exists. Scan for the actual key pattern the writer uses (a wildcard scan like company_profiler:{sid}:*, explicitly skipping known non-data suffixes such as backup/working/progress/domain_index if the writer also persists those under similar prefixes), then extract the real nested path from there (full_profile.prospect_input.simple_product), not the standalone key a previous version of the code assumed. 2. Nesting level second. Once at the right root, re-check where new fields actually attached — as siblings of an existing sub-object, or nested one level deeper inside it (enriched.competitive_analysis vs a flat simple_product.competitive_analysis). Read the nested path, not the sibling path, if that's what the real payload shows. 3. Field name and shape third, field by field. Build an explicit table of "what I assumed" vs "what the payload actually has" by printing/logging the raw keys of a real production payload before writing a single line of reader code — never derive the expected shape from documentation or from a declared schema, only from an actual captured payload. Concretely, this table can look like:

| assumed | real | |---|---| | competitive_confidence | confidence_overall | | copy_angles_v4 | copy_angles | | our_voc = list[str] | our_voc = dict{pain_points, strengths, ...} | | counter_triggers = list[str] | list[dict{copy_angle, competitor_name, ...}] | | three_way_gaps = list[str] | list[dict{exploit_angle, competitor_name, ...}] | | competitors_detailed.positioning | competitors_detailed.positioning_summary |

For each mismatched field, write a small targeted normalizer (a lambda-style transform that extracts the real sub-key, e.g. pulling copy_angle/exploit_angle/positioning_summary out of the real dict shape) plus a dict→list (or list→dict) normalize wherever the cardinality itself was assumed wrong, rather than one generic "coerce everything" function — the mismatches are heterogeneous enough that a single generic coercer either over-fits one field or silently mishandles another. 4. Verify against a live, real session, not a unit test with a hand-built fixture. A fixture you wrote yourself will faithfully encode the same wrong assumption you're trying to catch. Confirm against a real end-to-end run: the consumer endpoint returns non-empty, correctly-shaped data with the confidence/metadata fields populated, and the final assembled artifact (for example, the injected prompt block) is a specific, non-trivial size with all expected sub-parts present — a size or shape close to zero after the "fix" means at least one of the three assumption layers is still wrong.

What doesn't work

- Assuming the declared/documented schema matches the real output. The whole class of bugs above exists *because* someone wrote reader code against what a schema or a prior version's documentation said the shape would be, without checking a real captured payload. A schema is a stated intent; an actual LLM call is what you have to code against. - Stopping at the first fix that removes the exception. Patching the one .get() chain that crashed does nothing for the silent-empty variant sitting one field over — that one never throws, so nothing tells you it's still broken. "No more crashes" is not the same claim as "the data now arrives," and only checking for the former leaves the latter unverified. - Treating "the producer logged success" as proof the consumer's assumptions are correct. These are independent claims. A producer can genuinely, correctly finish and log real counts while the specific path a *different* piece of code reads from is wrong in three unrelated ways — the producer's success log is evidence about the producer, not about the reader. - A single generic "coerce anything to the expected shape" function for all mismatched fields. The three example mismatches above are qualitatively different (wrong key name, wrong cardinality, wrong sub-key location) — a one-size-fits-all coercer tends to either silently mask a genuinely different fourth mismatch it wasn't written for, or produce a plausible-looking but wrong value instead of failing loudly. Small, explicit, per-field normalizers made the actual mismatches visible during development instead of papering over them. - A guard only at the outermost `.get()`. obj.get("a", {}) protects the first hop but does nothing for a collapse one or two levels deeper in the same chain — the walk has to check isinstance(..., dict) at every step, not just the entry point.

Related

- An LLM occasionally regurgitates a prompt's own XML wrapper tags into the value it was asked to produce — the same root discipline ("never trust LLM output shape before normalizing it") applied to a textual pollution (prompt tags echoed into a clean value) instead of a structural collapse (nested dict → plain string). - The LLM returns a string where you expected a number — the error explodes two lines later, not where it's born — a narrower, single-field version of the same family: a value typed as numeric downstream arrives as a string, and the resulting error surfaces lines away from where the bad value actually entered. - Pydantic V2's default `extra='ignore'` drops unexpected fields silently — no error, no warning, the data just never arrives — a sibling "silent, no-exception data loss" failure: there a validation library's own default drops unexpected fields; here a reader's wrong assumption about key/nesting/shape does the same, just without any library involved. - Hardcoded 'reasonable' defaults baked into an LLM prompt or pipeline become systemic bias or silently go stale — the wider family this belongs to: code written against an assumption (about output shape, about a "reasonable" default, about the calendar year) that was true once, in one observed case, and quietly stops being true elsewhere or over time. - A blanket `except Exception` swallows an ImportError and degrades into a misleading business error — a related but distinct trap: there, an overly broad except hides a structural error that *did* raise something; here, the harder silent-empty variant is exactly the case where nothing raises at all, so no except clause — broad or narrow — would have caught it in the first place.