An LLM occasionally regurgitates a prompt's own XML wrapper tags into the value it was asked to produce

A prompt that wraps its inputs in XML-style tags (<current>, <context>) to give the model structure can, without any adversarial input at all, get echoed back — tags and all — instead of followed as an instruction; if the caller persists that output verbatim, the raw prompt scaffolding ends up stored and rendered to the end user.

The signal that gives it away

- A UI element (a card, a field, a summary) shows literal tag syntax where a clean value belongs — something in the shape of <current>Generated Value</current> followed by a <context> block and only then, at the very end, the narrative text that was actually supposed to be the whole answer. The tag names are recognizably *your own prompt's* wrapper names, not anything the model invented — that's the tell that this is prompt-echo, not a hallucination. - It happens despite an explicit "respond with ONLY the final value, no preamble, no labels" instruction in the prompt. That's the counterintuitive part worth internalizing: telling the model not to echo is not sufficient by itself. An instruction that works nearly every time still fails often enough that anything persisted straight from the model's response has to be treated as untrusted, unconditionally — not "trusted unless it looks weird." - It is intermittent, not deterministic — the same prompt shape against the same model call succeeds most of the time and echoes occasionally. That distinguishes it from a "broken since birth" bug: you won't catch it by testing once and seeing it work. It surfaces at volume, in production, across many calls. - When it does happen, the echoed structure is usually the entire prompt scaffolding, with the model's genuine new/updated value tacked on at the very end rather than in place of the wrapped placeholder — i.e., the model isn't ignoring the task, it's completing the task correctly and then *additionally* reproducing the input framing around it. - Direct evidence to search for: grep persisted values for the literal opening tag string used in your own prompt template (e.g. <current> if that's your wrapper name) — any hit outside of debug/test data is pollution, not a coincidence, since that string has no reason to appear in a clean answer.

How it's exploited (the two-step recovery)

1. Try structured extraction first, don't strip immediately. If the raw model output *starts* with your own wrapper tag, match and pull out its inner content rather than discarding the whole response: ``python m = re.match(r"\s*<current>([\s\S]*?)</current>", raw, flags=re.IGNORECASE) if m: raw = m.group(1).strip() ` This step matters because, empirically, when the model echoes the <current> wrapper it frequently echoes the value it was *given* verbatim inside it — so extracting between the tags recovers a usable answer far more often than you'd expect from "the model just repeated the prompt." Treat step 1 as recovery, not just cleanup. 2. **Fall back to a targeted strip of a closed, known tag list** — only if step 1 didn't match (i.e. the echo didn't take the clean single-wrapper shape): `python KNOWN_PROMPT_TAGS = ("current", "context", "user_feedback", "guidance", ...) for tag in KNOWN_PROMPT_TAGS: text = re.sub(rf"<{tag}>[\s\S]*?</{tag}>", "", text, flags=re.IGNORECASE) text = re.sub(rf"</?{tag}\b[^<>]*>", "", text, flags=re.IGNORECASE) ` The list is **closed and explicit** — it enumerates only the tag names your own prompt builder actually uses, not a wildcard. The two passes matter: one for well-formed <tag>…</tag> pairs, one for stray/unmatched tag fragments that can survive a partial echo. 3. **Keep the tag list in sync with the prompt builder, as one unit of change.** If a new XML wrapper name is added to the prompt (a new field, a new context section), the strip list must be updated in the same change — otherwise the newest wrapper is the one most likely to leak, and it's exactly the one your cleanup code doesn't know about yet. 4. **Diagnose in production with three checks, cheapest first:** - Grep persisted storage for the literal wrapper tag string across keys matching your data's shape — every hit is a confirmed pollution instance, not a false positive. - Grep application logs for whatever event your extraction/strip code emits (e.g. an "anti-echo extracted" / "anti-echo stripped" log line) to get a frequency estimate — this tells you if you're dealing with a rare edge case or a meaningful percentage of calls, which changes how urgently the structural fix (below) is worth doing. - Call the same prompt path manually with minimal input and assert the raw response doesn't contain the wrapper tag — a fast way to reproduce without needing production traffic. 5. **Structural alternative: force schema-constrained output instead of post-hoc parsing.** Where the underlying model API supports it, requesting structured output (e.g. Gemini's response_format object, with mime_type set to "application/json" and a schema` nested inside that same object) constrains generation so the model is obliged to return schema-conformant JSON — there is no room for a stray XML echo because the output isn't free text at all. This is the fix that removes the failure mode at the root instead of parsing around it. The trade-off is real: switching an existing text-in/text-out call site to structured output touches everything downstream that consumes the response, which can be a breaking change across a call chain wide enough to make it not worth doing immediately — a judgment call, not an automatic win. 6. Cleanup of already-persisted data is a separate step from the code fix. A forward-looking fix in the write path does nothing for values that were already saved polluted before the fix shipped — those still render with raw tags until something walks the existing data and re-applies the same extract-then-strip logic to it. Write that as a one-shot pass over every nested string field the pollution could have reached (recurse through dicts/lists, clean each string leaf), then invalidate any cache or derived view built on top of that data — otherwise the cache keeps serving the stale, polluted value even after the source record is clean. 7. Verify the fix in this order: run the cleanup pass and confirm the persisted value is clean → hard-refresh the view and confirm it renders clean (not just that storage is clean — caches can still be stale) → trigger a brand-new write through the fixed path and confirm the new save is clean from birth. All three matter; checking only the first gives false confidence if a cache layer sits between storage and the render.

What doesn't work

- Trusting the "respond with ONLY X" instruction as sufficient. It measurably reduces the echo rate but does not eliminate it. Any code that persists LLM output on the assumption that a strongly-worded instruction guarantees clean output will eventually show a user raw prompt scaffolding. Normalize before persisting, unconditionally — not as an exception-path safety net. - **A generic tag-stripping regex (<.*?> or similar).** This is the single most tempting shortcut and the most damaging one: it silently destroys any legitimate content that happens to contain angle brackets — an HTML/code snippet a user pasted, an emoji written as <3, a notation like <48h> or <v2>. A closed, named list of *your own prompt's* tags is the only version of this fix that's safe, because it can never match content it didn't itself introduce. - A single monolithic strip regex with no extraction step. Going straight to "delete anything matching a known tag" without first trying to extract from a well-formed <current>…</current> wrapper throws away a genuine, usable answer more often than it needs to — the two-step order (extract, then strip only as fallback) recovers more correct values than stripping alone. - Shipping the code fix and calling the incident closed. The fix only prevents *new* pollution; every record written before the fix shipped is still dirty and still renders with raw tags to whoever views it next. Treat "clean up existing data" as a required second half of the fix, not an optional follow-up. - Treating this as an adversarial prompt-injection problem and reaching for injection defenses. The failure here has no attacker and no malicious input — it's the model spontaneously reproducing its own scaffolding on ordinary, legitimate calls. Defenses built for system-prompt leakage under adversarial probing are a different (related) problem with different mitigations; the fix that actually applies here is output normalization at the point of persistence, not input filtering or injection detection.

Related

- An LLM occasionally collapses a nested dict into a plain string — and the real shape it produces differs from the shape a downstream reader assumed — the same root discipline ("never trust LLM output shape") applied to a structural collapse (nested dict → plain string) instead of a textual one (clean value → value-with-tags). - The LLM returns a string where you expected a number — the error explodes two lines later, not where it's born — another shape of "the model didn't return what the caller assumed," surfacing as a type error two lines downstream instead of as pollution visible immediately in the UI. - Pydantic V2's default `extra='ignore'` drops unexpected fields silently — no error, no warning, the data just never arrives — a sibling lesson on defaults that quietly let bad shape through: there a validation library's default, here a caller that persists without normalizing first. - Hardcoded 'reasonable' defaults baked into an LLM prompt or pipeline become systemic bias or silently go stale — a different prompt-authoring failure mode (baked-in defaults biasing or going stale over time) from the same family of "the prompt looked fine in review but degrades a specific way in production." - A gate classifier exceeds its mandate — and overwrites a score that was already computed — another case where an LLM's actual behavior at runtime quietly exceeds what its prompt asked for, requiring a guard the prompt's wording alone doesn't provide.