The LLM returns a string where you expected a number — the error explodes two lines later, not where it's born

An LLM asked for an int-typed field can return a non-numeric string instead ("Unknown", "10-50", "5+") — and in a language where string * integer is a *legal* operation, that bad value survives every truthy guard and only crashes several statements downstream, far from where it actually entered the system.

The signal that gives it away

- Stack trace reads TypeError: can't multiply sequence by non-int of type 'float' (or the equivalent for /), raised inside a numeric computation, not at the point where the value was read or parsed. - The bug is intermittent with the same code path: one run of a pipeline computes fine, the next run on a differently-shaped input fails — because the failure depends on what the LLM *happened* to return this time for a field declared numeric in the schema, not on a code change. - A guard like if value: upstream did not catch it — the value passed the truthy check (a non-empty string is truthy), so whoever wrote the guard assumed it was doing type-safety work when it was only doing None/falsy-checking. - The field in question is a schema-typed numeric value populated by an LLM call (headcount, a count, a monetary amount used in arithmetic) rather than a value read directly from a database column or a strictly-validated form input. - Log context right before the crash shows a human-readable non-numeric string sitting in a variable whose name and prior usage strongly imply "this should always be a number" (e.g. a count field logged as employees=Unknown or employees=10-50 a few lines before the multiplication that blows up).

How it's exploited

Root cause, precisely stated: in a language where str * int returns a repeated string instead of raising (sequence repetition — confirmed in the [Python language reference](https://docs.python.org/3/reference/expressions.html#binary-arithmetic-operations): "one argument must be an integer and the other must be a sequence... sequence repetition is performed"), a partially-numeric expression chain doesn't fail at the first bad operand — it fails at the *first operand that can't participate in sequence repetition*, which is usually one or two operations later:

``python count = "Unknown" # LLM returned this for an int-typed field if count: # truthy check passes — "Unknown" is non-empty total = count * 45000 # LEGAL: "Unknown" repeated 45000 times (!), no error monthly = total * 1.4 # <-- TypeError explodes HERE, one line downstream ``

The if count: guard gives false confidence: it only rules out None/empty string, not "is this actually a number." The real defect — an LLM returning a non-conforming type for a schema-declared numeric field — is invisible until an operation appears that isn't legal for strings (multiplying by a float, dividing, calling round()). That gap between "bad value enters" and "crash happens" is exactly what makes the bug expensive to trace from the stack trace alone: the traceback line is never the root cause line.

Why this happens at the model layer: declaring a field int in a schema (including via a validation library like Pydantic) does not guarantee the model's actual output respects it. The [Gemini structured-output documentation](https://ai.google.dev/gemini-api/docs/structured-output) itself tells callers to validate values in the application even when the output is syntactically valid JSON against the schema — semantic/type conformance on individual fields is explicitly not guaranteed by the platform, only JSON-shape conformance is. Nested or deeply-generated fields are the most exposed, because validation enforcement tends to weaken the further a field is from the top-level schema.

Fix pattern: coerce defensively at the point of entry into the computation, not one layer above it.

``python def compute_metric(count: int = None, ...): # count may arrive as a non-numeric string from an upstream LLM call # ("Unknown", "10-50", "5+", "n/a") instead of respecting the int schema. if not isinstance(count, int): try: count = int(str(count).strip()) if count is not None else None except (ValueError, TypeError): count = None if isinstance(count, int) and count <= 0: count = None # ...rest of the function is now None-safe ``

Why coerce exactly here, and not further upstream (e.g. right after the LLM/JSON-parse step): the computation function is the *stable* boundary in the codebase — it doesn't change often. The upstream source (the shape of the LLM's JSON output) changes format far more often as prompts and models evolve. Centralizing the guard upstream means every new call site of the parsed payload has to remember the invariant holds; coercing at each numeric boundary means the guard travels with the computation that actually needs the invariant, and can't be silently bypassed by a new caller that skips the upstream step.

Where else to apply the same pattern: grep for every function doing arithmetic on a value that came from a dict produced upstream by an LLM call, not just the one that already crashed:

``bash grep -rnE "(\* [0-9]+\.[0-9]|/ [0-9]+\.[0-9])" <source dirs> | grep -v "test\|\.pyc" ``

Rule of thumb: any int * float or int / float where the left-hand operand originates from something.get(...) on an LLM-produced payload needs the same isinstance-then-coerce guard, even if it hasn't crashed yet — it just hasn't seen the right (wrong) input yet.

Verifying the fix end-to-end, not just re-reading the diff:

1. Re-trigger the same pipeline on the session/input that previously crashed. 2. Confirm the log line that prints the coerced value shows an int or None — never a raw string — at the point right before the computation. 3. When coercion produces None (value was genuinely non-numeric), the downstream block should *skip* cleanly (e.g. via the same if count: guard, now legitimately doing its job) rather than crash — and that sub-result should be marked with an explicit low/zero confidence rather than silently omitted, so a caller further out can tell "we didn't compute this" from "this computed to zero." A fix that turns a hard crash into a silent, unflagged skip has only moved the failure from loud to invisible. 4. Re-run against a *fresh* LLM call (not a cached response) at least once — the whole point of this bug class is that it depends on what the model happens to emit for that run, and a single passing re-run against a cached/replayed payload does not prove the coercion generalizes.

What doesn't work

- A bare truthy check (`if value:`) as a stand-in for a type check. It only filters None/empty-string, and a non-empty non-numeric string sails right through it — that's precisely the gap this bug exploits. Truthy and "is numeric" are different questions; conflating them is the root of the false confidence. - Trusting a Python type hint (`count: int = None`) as if it were enforced at runtime. Type hints in a dynamically-typed language document intent, they don't perform validation — a function signature that says int will happily accept and pass through a string with zero complaint until something incompatible is done with it. - Relying on schema-level validation (e.g. a Pydantic model, a declared JSON schema on the LLM call) as sufficient on its own, especially for nested or model-generated fields. A schema declares what the field *should* be; it does not guarantee the model's actual output — particularly several levels of nesting deep — will conform, and validation libraries don't always enforce constraints uniformly across nested, dynamically-populated structures. - Centralizing the coercion one layer upstream, right after the LLM/JSON-parse step, instead of at each computation's boundary. It looks cleaner (one guard instead of several) but means every new consumer of the parsed payload silently depends on that upstream guard having run, and a future refactor that adds a new call path around it reintroduces the exact same crash with no test catching it, because nothing at the computation site itself checks the type anymore. - Treating this as a one-off patch on the single function that crashed. The failure mode ("LLM-populated field, numeric-typed, used in arithmetic without a type check") recurs at every similarly-shaped call site in the codebase; fixing only the one that happened to be exercised first leaves the rest waiting for their own run of bad input.

Related

- An LLM occasionally collapses a nested dict into a plain string — and the real shape it produces differs from the shape a downstream reader assumed — same family, one level up in structure: instead of a scalar field arriving as the wrong type, a whole nested object collapses into a plain string or a different shape than the reader assumed. - Some exceptions stringify to an empty string — logging only `str(e)` hides the real error — a neighboring "the error surfaces far from its cause" trap: there the exception itself hides its own message; here the value that caused the exception was legal input two operations too early to trigger anything. - Pydantic V2's default `extra='ignore'` drops unexpected fields silently — no error, no warning, the data just never arrives — a sibling case of a validation library not doing the enforcement job its presence implies, silently rather than loudly. - A blanket `except Exception` swallows an ImportError and degrades into a misleading business error — a related but distinct trap: there a broad except swallows a structural error that *did* raise immediately; here the danger is the opposite — the bad value does *not* raise immediately, so nothing is there yet to catch. - Hardcoded 'reasonable' defaults baked into an LLM prompt or pipeline become systemic bias or silently go stale — the wider family this belongs to: code written against an assumption about LLM output (a type, a shape, a "reasonable" default) that holds most of the time and quietly stops holding on some inputs.