Effective Context vs Advertised Context: Which Number Actually Bounds Your Agent

A model's advertised context window is a capacity: how many tokens the API will accept before it returns an error. Effective context is a competence: how many tokens the model can still use correctly for a particular kind of work. The two are different quantities measured by different instruments, and every published attempt to measure the second finds it far smaller than the first.

This page is about which number to look at, and why "1M tokens" is never that number.

The short answer

There is no single effective-context number, and asking for one is the mistake. An effective length is a quadruple:

> effective length = f(model version, task category, pass threshold, harness), where the harness > also has a ceiling that can censor the answer.

Change any of the four and the number moves by more than an order of magnitude on the same model. The section "One model, two effective lengths" below shows exactly that, with both figures read off the primaries.

What you can do instead is pick the benchmark whose *task shape* matches your workload, read its *threshold definition*, check its *ceiling*, and then treat that number as your budget. The procedure is in "What number to look at" below.

Identifier

arXiv:2307.03172, v1 6 July 2023, v3 20 November 2023; published in TACL vol. 12 (2024), DOI 10.1162/tacl_a_00638. Liu, Lin, Hewitt, Paranjape, Bevilacqua, Petroni, Liang.

What it measures

two tasks, with the *position* of the relevant material varied while the total input is held fixed. (1) Multi-document QA over 10, 20 or 30 retrieved Wikipedia documents, exactly one of which contains the answer. (2) Synthetic key-value retrieval over 75, 140 or 300 JSON key-value pairs where "each of the keys and values are unique, randomly-generated UUIDs" (128-bit) - pure exact-match lookup, no reasoning at all. Models, by their own identifier strings: gpt-3.5-turbo-0613, gpt-3.5-turbo-16k-0613, claude-1.3, claude-1.3-100k, LongChat-13B (16K), MPT-30B-Instruct.

Result

a U-shaped curve. Performance is "often highest when relevant information occurs at the beginning or end of the input context, and significantly degrades when models must access relevant information in the middle of long contexts, even for explicitly long-context models."

The number that matters, and it is not the one usually quoted. For GPT-3.5-Turbo the paper reports closed-book accuracy - no documents at all, parametric memory only - of 56.1%, and oracle accuracy - the single correct document alone - of 88.3%. And then:

> "GPT-3.5-Turbo's multi-document QA performance can drop by more than 20%-in the worst case, > performance in 20- and 30-document settings is lower than performance without any input > documents (i.e., closed-book performance; 56.1%)."

Now put a token count on "20 and 30 documents", because this is the part that gets lost. The paper's own Table 3 gives, for the GPT-3.5-Turbo tokenizer, an average of 2,946.2 +/- 155.1 tokens for the 20-document setting (max 3,920) and 4,419.2 +/- 226.5 tokens for the 30-document setting (max 6,101).

So the model fell *below its own no-documents baseline* at roughly three to four and a half thousand tokens, inside a 16K window, on a model whose extended variant advertised 16K and whose Claude counterpart advertised 100K. Degradation is not a phenomenon of the last mile of a long window. It starts immediately.

The second finding, equally forgotten

"When the input context fits in the context window of both a model and its extended-context counterpart, we see that performance between them is nearly identical." The published pairs, closed-book / oracle: GPT-3.5-Turbo 56.1% / 88.3% versus GPT-3.5-Turbo (16K) 56.0% / 88.6%; Claude-1.3 48.3% / 76.1% versus Claude-1.3 (100K) 48.2% / 76.4%. Buying the long-context SKU bought capacity, not competence.

*Caveat with its date:* these are 2023 models. The mechanism replicated; the absolute numbers did not stay current, and nobody should cite 56.1% as a fact about anything shipping in 2026.

Identifier

arXiv:2404.06654, v1 9 April 2024, v3 6 August 2024; COLM 2024. Hsieh, Sun, Kriman, Acharya, Rekesh, Jia, Zhang, Ginsburg (NVIDIA). Harness and data generator: github.com/NVIDIA/RULER.

What it measures

13 synthetic tasks in 4 categories - retrieval (8 NIAH variants), multi-hop tracing (variable tracking), aggregation (common/frequent word extraction) and QA - generated fresh at each length so nothing can be memorised. Base of every score: 500 examples per task per length, at 4K, 8K, 16K, 32K, 64K and 128K. Seventeen models in the paper.

The definition, which is the paper's real contribution

a model's *effective length* is the longest tested length at which its 13-task average still exceeds a fixed threshold, and the threshold is the performance of Llama2-7B at 4K, namely 85.6%. The authors chose that number; it is a convention, not a law of nature, and it is why RULER's effective lengths are not comparable with anyone else's.

Result, from Table 3 (claimed -> effective), 13-task average

| Model | Claimed | Effective (RULER Table 3) | |---|---|---| | Gemini-1.5-Pro | 1M | >128K (ceiling of the test) | | GPT-4 | 128K | 64K | | Llama3.1 (70B) | 128K | 64K | | GLM4 (9B) | 1M | 64K | | Llama3.1 (8B) | 128K | 32K | | Yi (34B) | 200K | 32K | | GradientAI/Llama3 (70B) | 1M | 16K | | Mistral-v0.2 (7B) | 32K | 16K | | LWM (7B) | 1M | <4K |

The README on main at commit ab17b78 (last modified 2025-10-09) adds later entries in the same shape - InternLM2.5 (7B), claimed 1M, effective 4K; GradientAI/Llama3 (8B), claimed 1M, effective 16K - and states the finding flatly: "Almost all models fall below the threshold before reaching the claimed context lengths."

Three things about that table you must not misread, and all three are in the primary.

*First, ">128K" does not mean "1M works".* RULER's grid stops at 128K. ">128K" is a censored observation: the model had not yet failed when the ruler ran out. Nothing in RULER measures anything at 1M.

*Second, several of the strongest rows on the current README are vendor-submitted.* The README's own note block says the Jamba-1.5-large, FILM-7B, Qwen2.5-1M, Qwen3 and EXAONE 4.0 numbers are "reported by authors" of those models, from their own papers. They sit in the same table as NVIDIA's own runs, distinguished only by an asterisk and a footnote. That is not fraud and the maintainers disclose it - but a row measured by the model's vendor and a row measured by a third party are different evidence and should not be averaged in your head (How to Verify an Agent-Payment Protocol Claim Before Citing It — the four checks this wiki runs on every page).

*Third, RULER prints more than one effective length per model, and only Table 3 is the headline.* The appendix breaks the same runs down by category, and the effective lengths move. Llama3.1 (70B) is 64K on the 13-task average (Table 3), 64K on retrieval (Table 13), but 32K on aggregation (Table 15). GLM4 (9B) is 64K on Table 3 and 8K on aggregation. If you quote "RULER effective length" you must say which table, or you will quote a per-category number as if it were the headline. This is not hypothetical: it is one of the two documented ways the number gets mangled downstream, and the other one is below.

Identifier

arXiv:2410.02694, v1 3 October 2024, v3 6 March 2025; ICLR 2025. Yen, Gao, Hou, Ding, Fleischer, Izsak, Wasserblat, Chen (Princeton PLI + Intel).

What it measures

seven categories chosen to cover real applications rather than probes alone - synthetic recall, RAG, generation-with-citations (ALCE), passage re-ranking, long-document QA, summarisation and in-context learning. Base and units, both of which matter: lengths are L in {8K, 16K, 32K, 64K, 128K} where "L is the number of Llama-2 tokens" - not each model's own tokenizer, and not words - and each dataset is sampled at "100 to 600 examples". 59 long-context models studied; the correlation analysis below is computed across 35 instruction-tuned models at 128K input length (Figure 3 caption).

Result 1 - needle-in-a-haystack does not predict real work. Spearman rank correlation with InfiniteBench QA, n = 35, read from the Figure 4 panel annotations: original NIAH rho = 0.63 (p = 5.1e-05), RULER MK rho = 0.81 (p = 3.2e-09), HotpotQA rho = 0.88 (p = 2.9e-12). The paper's summary: "none of the synthetic tasks achieves an average correlation higher than 0.8", and "the popular RULER average score ... does not yield strong correlations (all < 0.85)".

Result 2 - and this is the one that makes a single effective-context number incoherent:

> "Performance degradation with longer inputs is category-dependent."

Concretely: "Most frontier models largely retain performance on recall and RAG with longer inputs; however, even the best models experience significant degradation as context length increases on tasks like re-ranking and generation with citations ... On generation with citations, open-source models completely collapse at 128K, while GPT-4o remains relatively stable." At 128K on citations and re-ranking, closed models are "30 to 40 absolute points better than the best open-source models".

Result 3 - the categories do not even track each other. Spearman correlation between HELMET categories at L = 128K, read off the Figure 5 matrix: Recall-RAG 0.88, RAG-LongQA 0.92, but ICL-Citations 0.34, ICL-Re-ranking 0.36, ICL-Summarisation 0.38. A model that holds up at 128K for one of your workloads tells you very little about the next one.

Identifier

arXiv:2412.15204, v1 19 December 2024, v2 3 January 2025. Bai, Tu, Zhang, Peng, Wang, Lv, Cao, Xu, Hou, Dong, Tang, Li (Tsinghua University + Zhipu.AI).

What it measures

503 four-choice multiple-choice questions (random baseline 25%) written by nearly 100 educated annotators over real material, in six categories - single-doc QA, multi-doc QA, long in-context learning, long-dialogue history, code-repository understanding, long structured data. Contexts run "from 8k to 2M words", with a median of 54k and a mean of 104k words. Note the unit: this benchmark is measured in words, so its bands are not interchangeable with the token grids above. Difficulty was enforced at annotation time: an item was accepted only if "all three LLMs cannot answer correctly at the same time, and the human reviewer cannot answer correctly within 3 minutes, even with searching tools within the document".

Result

human experts reach 53.7% under a 15-minute limit. The best model answering directly, GPT-4o-2024-08-06, reaches 50.1%; o1-preview-2024-09-12, which spends far more inference-time reasoning, reaches 57.7%. Chain-of-thought prompting is worth "an average 3.4% improvement for open-source models". Contamination control: with the question but *no* context, "most models achieve an overall accuracy ranging from 25% to 30%", i.e. at chance.

By length, from the paper's own text

on data under 32k the best model surpasses the human expert by 15.4 points; in the 32k-128k band the best model is still 5.6 points behind human accuracy.

The caveat the authors print themselves, which you must carry with the number

> "Models do not show lower scores on subsets with longer length ranges because the distribution of > tasks differs significantly across each length range."

So LongBench v2 is *not* a clean measurement of length degradation. It is a measurement of realistic long-context difficulty in which length and task type are confounded on purpose, because that is what real corpora look like. Use it to ask "can any model do this job at all", not "where does this model break".

Identifier

arXiv:2502.05167, v1 7 February 2025, v3 9 July 2025; ICML 2025, PMLR vol. 267. Modarressi, Deilamsalehy, Dernoncourt, Bui, Rossi, Yoon, Schuetze (LMU Munich/MCML + Adobe Research). Code and data: github.com/adobe-research/NoLiMa.

What it measures

needle-in-a-haystack with the lexical shortcut removed. The question and the needle are constructed to have minimal word overlap, so the model has to infer a latent association rather than string-match. The paper's own template (Table 2), quoted rather than paraphrased, is: question "Which character has been to *Wq*?" where *Wq* is a place - the worked example given is Dresden - and needle "Actually, [CHAR] lives next to the *Wn*", where *Wn* is a building or landmark - the worked example is the Semper Opera. Nothing lexical connects the question to the needle; the model must know the opera house is in that city. Haystacks are built by concatenating short snippets from ten open-licence books to defeat memorisation. Base of every cell in the main table: 7,540 tests per context length - 58 question-needle pairs x 5 haystacks x 26 needle placements.

Their threshold, explicitly different from RULER's

effective length is "the maximum length at which the score remains above a threshold, set at 85% of the model's base score". The base score is built from the 250, 500 and 1K runs: per question-needle example, average over the 5 haystacks, take the maximum of those three lengths, then average across examples. This is a *relative* threshold, so a weak model is not penalised for being weak - only for degrading.

Result (Table 3, claimed -> effective, with the 32K score for scale). The displayed grid runs 1K to 32K; 250 and 500 are the base-score runs and are not columns:

| Model | Claimed | Effective (NoLiMa) | Base | @32K | |---|---|---|---|---| | GPT-4o | 128K | 8K | 99.3 | 69.7 | | Gemini 2.0 Flash | 1M | 4K | 89.4 | 41.0 | | Claude 3.5 Sonnet | 200K | 4K | 87.5 | 29.8 | | Llama 3.3 70B | 128K | 2K | 97.3 | 42.7 | | Llama 3.1 70B | 128K | 2K | 94.5 | 43.2 | | Gemini 1.5 Pro | 2M | 2K | 92.6 | 48.2 | | Mistral Large 2 | 128K | 2K | 87.9 | 18.8 | | Command R+ | 128K | <1K | 90.9 | 7.4 | | Gemini 1.5 Flash | 1M | <1K | 84.7 | 28.6 | | GPT-4o mini | 128K | <1K | 84.8 | 13.7 |

Summary sentence, verbatim: "Out of the 13 models, 11 exhibit performance at 32K lengths that is half or less of their base scores."

The most recent primary figures we can stand behind, from the v3 appendix (Table 10; this run reduces needle placements from 26 to 11 per length for cost, so its base is 3,190 tests per length, not 7,540 - a smaller base than the main table, and we flag it wherever these numbers appear): GPT-4.1 claimed 1M, effective 16K, base 97.0, 128K score 64.7; Gemini 2.5 Flash claimed 1M, effective 2K; Llama 4 Maverick claimed 1M, effective 2K; Llama 4 Scout claimed 10M, effective 1K; GPT-4.1 Mini and GPT-4.1 Nano, both claimed 1M, effective <1K. The authors' own sentence: "GPT-4.1 shows clear improvements over prior models; however, its effective context length remains around 16K-well below the claimed 1M-and its performance drops below 65% on 128K context lengths."

Read NoLiMa as a floor for latent-association work, not as a general estimate. The benchmark is adversarial to lexical matching by construction. If your agent's retrieval keys genuinely appear verbatim in the haystack - exact identifiers, function names, SKUs - you are in the NIAH regime and RULER's numbers are the right ones. If the link between what you ask and what is written is semantic, NoLiMa's numbers are the right ones. Most agent memory is the second case, because the agent asks "what did we agree about pricing" and the record says "we told the buyer $4.20 a unit".

Do reasoning models fix it? Partly, and inconsistently

This is the obvious objection to everything above, so here is the evidence rather than a guess. NoLiMa-Hard is the 10 hardest of the 58 question-needle pairs, re-run against reasoning models. The paper (v3, 9 July 2025, Table 5) covers GPT-o1, GPT-o3 Mini and DeepSeek R1-Distill-Llama-70B and concludes: "Across all models, performance drops below the 50% mark at 32K context length."

That sentence has since been overtaken by the authors' own data. The NoLiMa repository README at commit cb14780 (last modified 2025-07-17), whose changelog reads "[2025-07-17]: Added evaluation results on GPT-o3 and GPT-o4 Mini on NoLiMa-Hard in the reasoning models section", prints:

| Model (NoLiMa-Hard) | Base | 8K | 16K | 32K | |---|---|---|---|---| | GPT-o3 | 100.0 | 86.2 | 74.9 | 58.5 | | Gemini 2.5 Pro | 99.1 | 63.0 | 58.6 | 58.6 | | GPT-o1 | 99.9 | 78.0 | 60.1 | 31.1 | | DeepSeek R1-Distill-Llama-70B | 99.9 | 75.5 | 49.4 | 20.7 | | GPT-o3 Mini | 98.8 | 36.9 | 25.5 | 18.9 | | GPT-o4 Mini | 99.6 | 30.8 | 20.2 | 11.7 | | GPT-4.1 (no CoT, for comparison) | 96.0 | 58.4 | 54.5 | 45.4 |

GPT-o3 and Gemini 2.5 Pro are the first entries to stay above half their base score at 32K, so the paper's "across all models" claim is now false for its own benchmark. This is a real improvement and we print it in full. It is also not a solution: at 32K both still sit around 58% of a base near 100%, and the two smaller reasoning models (o3 Mini, o4 Mini) are among the *worst* rows in the table - worse than GPT-4.1 with no reasoning at all. Inference-time reasoning is not a reliable substitute for putting the right tokens in front of the model, and it is not even monotone in model family. *Base to carry with these numbers:* NoLiMa-Hard is 10 question-needle pairs, not 58; the authors do not restate the placement count for this subset, so we do not print a test count for it.

One model, two effective lengths

Take meta-llama/Meta-Llama-3.1-70B-Instruct, one model, one version, no ambiguity:

- RULER effective length: 64K. (arXiv:2404.06654v3 Table 3, 13-task average; identical row on the NVIDIA/RULER README at commit ab17b78, which links that exact HuggingFace identifier. Threshold: absolute, Llama2-7B at 4K = 85.6%.) - NoLiMa effective length: 2K. (arXiv:2502.05167v3 Table 3. Threshold: relative, 85% of the model's own base score. Latent-association needle.)

Same model. Neither number is wrong and neither harness is broken. They measure different competences under different thresholds, and the gap between them *is* the answer to "how much context can I use": it depends entirely on what you are asking the context to do.

The base for the ratio, because this page would be a hypocrite without it. 64K / 2K = 32, and that arithmetic is ours on two published integers. But both integers are grid points on coarse doubling grids - RULER tests 4K, 8K, 16K, 32K, 64K, 128K; NoLiMa's main table tests 1K, 2K, 4K, 8K, 16K, 32K - and an "effective length" is the last grid point above threshold, not the crossing point. So the true RULER threshold-crossing for this model lies in [64K, 128K) and the true NoLiMa crossing in [2K, 4K). The honest statement of the gap is therefore between roughly 16x and 64x, and "32x" is the ratio of the two *reported* numbers, not a measured ratio of competences. We print the range. Anyone printing "32x" flat, including our own earlier draft, is quoting grid arithmetic as if it were a measurement.

This is why the question "what is Model X's effective context?" has no answer, and why any page, vendor deck or blog post that gives you one number for a model without naming the benchmark, the table, the task and the threshold has given you nothing you can act on.

Three checks that are ours

These are attestations, not citations: we are the primary source for each comparison, the method is two commands each, and each is falsifiable by opening the named documents at the named versions.

Check 1 - a peer-reviewed primary mis-states another peer-reviewed primary. arXiv:2502.05167v3 section 4.4 states: "in other benchmarks with similar settings, such as BABILong (QA1) ... and RULER ..., Llama 3.1 70B achieves effective lengths of 16K and 32K, respectively." RULER's headline table prints 64K for that model, in arXiv:2404.06654v3 Table 3 and on the NVIDIA/RULER README at commit ab17b78. (32K is RULER's Table 3 effective length for Llama3.1 8B, one row down - and also the *aggregation-category* effective length for the 70B in appendix Table 15, which is the likelier source of the slip.) The discrepancy is small and it happens to make NoLiMa's contrast *understated* rather than overstated - the real gap is 64K vs 2K, not 32K vs 2K, so the error runs against the authors' own argument. We report it because a reader who takes the NoLiMa sentence as their source for a RULER number will publish a wrong number with a correct-looking citation, which is exactly the failure mode this page exists to prevent.

Check 2 - the same NoLiMa sentence mis-states NoLiMa's own table. The continuation of that passage reads: "in NOLIMA, Llama 3.1 70B has an effective length of only 2K and shows a significant drop in performance at 32K lengths (42.7% vs. 94.3% base score)." NoLiMa's own Table 3 gives Llama 3.1 70B a base of 94.5 and a 32K score of 43.2; 42.7 is the 32K score of the row above, Llama 3.3 70B. The effective-length claim (2K) is correct; the two numbers in the parenthesis are not the ones in the table.

Check 3 - LongBench v2's conclusion contradicts its abstract. The abstract: "human experts achieving only 53.7% accuracy under a 15-minute time constraint ... the best-performing model, when directly answers the questions, achieves only 50.1% accuracy". The Conclusion: "human performance at 50.1% and the best LLM achieving 57.7% accuracy". Table 2 agrees with the abstract (GPT-4o-2024-08-06 = 50.1, o1-preview-2024-09-12 = 57.7). If you cite the conclusion sentence you will cite a model score as a human score.

Method for all three, stated so it can be redone in five minutes: download the PDFs from arxiv.org/pdf/&lt;id&gt; at the named version, extract text locally, and read the named table and the named section. Performed 2026-08-26.

What number to look at - the procedure

1. Name your task category. Recall, RAG over retrieved passages, re-ranking, generation with citations, summarisation, in-context learning, or free-form long-document QA. HELMET's Figure 5 is the evidence that this choice dominates everything else: at 128K, ICL and citation performance correlate at rho = 0.34. 2. Ask whether your lookups are lexical or latent. Exact identifiers -> RULER/NIAH regime. Semantic association -> NoLiMa regime. The same model is 64K in the first and 2K in the second. 3. Read the threshold before the number. RULER: absolute, Llama2-7B@4K = 85.6%, 13-task average. NoLiMa: relative, 85% of the model's own short-context base. An effective length quoted without its threshold is not a measurement. 4. Name the table, not just the benchmark. RULER publishes a headline effective length and four per-category ones that differ (Llama3.1 70B: 64K headline, 32K aggregation). "RULER says X" is under-specified. 5. Check the units and the ceiling. RULER's grid stops at 128K; HELMET's stops at 128K and is counted in Llama-2 tokens; NoLiMa's main table stops at 32K, with an appendix reaching 128K at a reduced base; LongBench v2 is counted in words. ">128K" means "did not fail before we ran out of ruler", and a "128K" from two of these harnesses is not the same 128K. 6. Check who ran it. A vendor-submitted row and a third-party row look identical in a markdown table; the RULER README distinguishes five of them with an asterisk. 7. Then measure your own. RULER and NoLiMa are both open source and both generate data from a tokenizer plus a seed. Your distractors are not their distractors. A one-afternoon run against your actual corpus beats every number on this page, including the ones we checked ourselves.

As a working budget while you do that

on latent-association retrieval, published effective lengths for models advertising 1M or more sit between <1K and 16K (arXiv:2502.05167v3, Tables 3 and 10, models released through mid-2025). On multi-task synthetic recall with an absolute threshold, they sit between <4K and >128K (arXiv:2404.06654v3 Table 3 plus README@ab17b78). Those are the ranges the primaries support. Anything narrower is someone's guess.

Why "1M tokens" is not the number: what a 1M claim is actually built on

The Qwen2.5-1M Technical Report (arXiv:2501.15383, 26 January 2025) is unusually clear about this, and we quote it because it is the *vendor's own* description of its own evidence. The 1M-token demonstration is the Passkey Retrieval task - find a hidden number in a wall of filler. For the harder benchmarks the same report states plainly: for RULER, "The maximum data length is 128K tokens"; for LV-Eval, "The maximum data length is 256K tokens"; for Longbench-Chat, "The maximum data length is 100K tokens".

So the structure of a typical 1M claim is: *passkey retrieval at 1M, everything else at 256K or less, and the threshold-defining benchmark at 128K.* RULER exists precisely because passkey and vanilla NIAH are saturated - the RULER paper's own words are that models achieve "nearly perfect performance on the passkey retrieval and the vanilla NIAH task" while degrading badly on everything else. A 1M context claim evidenced by passkey retrieval is a claim about the plumbing, not about comprehension. It is true, it is useful - the tokens really do go in and the attention really does reach them - and it is not the number that bounds your agent.

Anthropic, selling both the long-context model and the tooling to avoid needing it, describes the same phenomenon from the vendor side and names it (29 September 2025): "as the number of tokens in the context window increases, the model's ability to accurately recall information from that context decreases. While some models exhibit more gentle degradation than others, this characteristic emerges across all models." Treat that as a vendor's position with a date on it, not as measurement - but note that it agrees with the independent benchmarks against the vendor's own interest in selling window size.

What does NOT work

The exclusions, which are the part of this page most likely to save someone a week.

Printing the "200-400K effective tokens" band for models advertising 1M. This band circulates widely in 2026 write-ups. We went looking for a primary that measures it and did not find one. The precise form of the gap, because the loose form is wrong: benchmarks that *score* above 128K do exist - LV-Eval runs to 256K by the Qwen report's own description, and LongBench v2 has contexts to 2M words - but no benchmark we could reach publishes a threshold-defined effective length measured above 128K. RULER's grid stops at 128K; HELMET's stops at 128K; NoLiMa's main table stops at 32K and its appendix at 128K with a reduced placement count; LongBench v2 has the length but confounds it with task type by its authors' own note. We are therefore not printing the band as a fact. If you have the primary, it supersedes this paragraph and we want it. Until then: a number with no reachable base is not a measurement, it is a rumour with units (x402 'Tens of Millions of Transactions': Reconciling the Claim — a number without a base is not a fact).

Treating "effective context length" as a property of a model. It is a property of (model, task, threshold, harness, table). Llama 3.1 70B: 64K by RULER's headline, 32K by RULER's aggregation category, 2K by NoLiMa. Quote the whole thing or quote nothing.

Using needle-in-a-haystack - or a green NIAH heatmap - as evidence that a long window is usable. HELMET, n = 35 models at 128K: NIAH correlates with InfiniteBench QA at rho = 0.63, because "most models achieve either perfect or near-zero performance on the original NIAH, leaving few data points in the middle".

Reading ">128K" on a RULER-style board as "verified at 1M". It is a censored observation at the harness ceiling.

Comparing a "128K" from one harness with a "128K" from another without checking the unit. HELMET's lengths are Llama-2 tokens; RULER generates per model tokenizer; LongBench v2 counts words. A page that lines these up silently is committing the error it is warning about.

Averaging vendor-run and third-party-run rows in the same leaderboard. The NVIDIA/RULER README mixes them by design and discloses it in a footnote below the table.

Assuming degradation begins near the window limit. Lost in the Middle put GPT-3.5-Turbo below its own closed-book baseline at an average of 2,946 and 4,419 tokens, inside a 16K window.

Assuming the extended-context SKU uses context better. GPT-3.5-Turbo 56.1/88.3 vs GPT-3.5-Turbo (16K) 56.0/88.6; Claude-1.3 48.3/76.1 vs Claude-1.3 (100K) 48.2/76.4. Nearly identical, on the ranges where both fit.

Assuming a benchmark's long subset measures length. LongBench v2's own table note: task distribution differs across its length bands, so its length columns are not a degradation curve.

Assuming reasoning models have solved it. GPT-o3 on NoLiMa-Hard: base 100.0, 32K score 58.5. Better than everything before it, and still a 42-point fall. GPT-o4 Mini: base 99.6, 32K score 11.7.

Assuming "the context filled up" explains why your agent failed - and equally, assuming Vending-Bench proves it did not. The abstract of arXiv:2502.15840 (20 February 2025), over runs exceeding 20M tokens, says: "We find no clear correlation between failures and the point at which the model's context window becomes full, suggesting that these breakdowns do not stem from memory limits." That sentence is widely quoted, including by us elsewhere, without its base, and the base changes what it can support. Section 3.6 supplies it: the correlation is a Pearson r = 0.167 between "Days Until Sales Stop" and "Days Until Full Memory" across the 9 model rows of Table 9, each row a mean of five runs. And, decisively, the harness capped agent memory at 30k tokens against "allowed input lengths ranging from 100k to 2 million tokens". So Vending-Bench never let a context approach a model's limit; it cannot measure context-window degradation, and it does not claim to. What it does show is that agents derailed long after their memory stopped growing. Read it as evidence that long-horizon agent failure has causes beyond window size - which is real and important - not as evidence that window size is harmless (Vending-Bench 2: The Top Model Broke Eleven Truces It Did Not Need to Break).

When long context is enough, and external memory is not needed

The honest version, including the part that argues against the people selling memory.

Long context is enough when

- The whole job fits in the effective length for your task category, not the advertised one. If your agent reads one 30-page contract and answers questions about it, you are at roughly 20-30K tokens on a recall-shaped task, comfortably inside RULER-effective territory for any frontier model and inside NoLiMa-effective territory for GPT-4.1-class models. - The lookups are lexical. Exact identifiers, error codes, function names. This is the regime where models are near-saturated: passkey and vanilla NIAH are effectively solved to 1M. - The session is the unit of work. Nothing must survive to tomorrow, so there is nothing to store. Durable memory solves persistence; if you have no persistence requirement it solves a problem you do not have. - Retrieval would cost you more than it saves. LongBench v2's own experiment is the evidence. Setup, so you can judge it: contexts split into 512-token chunks with the GLM4-9B tokenizer, encoded with Zhipu Embedding3, top-N chunks retrieved for N in {4, 8, 16, 32, 64, 128, 256}. Result: GPT-4o's best RAG configuration was at the 128k retrieval length, and it still scored 0.6 points below its own no-RAG score at the full 128k window. For the strongest long-context model in that experiment, retrieval was a small net loss. That result is published by a group that co-develops a competing model, and it cuts against every vendor of external memory, including this wiki's own thesis. It stands. - More memory can actively hurt. Vending-Bench section 3.5.2 varied one agent's memory capacity (GPT-4o mini at 10k, 30k and 60k tokens, five runs each) and found that "agents with larger memory capacities performed worse than those with less memory". That is a commercial evals lab, with no memory product to sell, reporting that the bigger store lost. Anyone selling curation has to live with that finding, and we are printing it in our own shop window.

**External memory earns its place when the *opposite* conditions hold, and the same LongBench v2 experiment shows it: Qwen2.5-72B-Instruct scored 4.1 points higher with retrieval at a 32k retrieval length than with the entire 128k context and no retrieval**, and GLM-4-Plus improved too; the paper's sentence is "Both models perform better at a 32k retrieval context length compared to using the entire 128k context window without RAG, with Qwen2.5 showing a notable improvement of +4.1%." The pattern is legible: the *further a model's effective length is below the material*, the more retrieval buys. For a model whose effective length on your task is 2K and a corpus of 200K, curation is not an optimisation, it is the only way the relevant tokens get in front of the model at all.

Two more conditions that no context window can satisfy at any size, because they are not capacity problems: the material outlives the session (nothing in a context window survives it), and the material must be checkable - a context window preserves nothing about where a claim came from, when, or whether it held up last time, beyond what whoever assembled the window happened to paste in.

Our position, declared as an interest. This wiki is published by Sinapsi, which sells durable memory for agents, so read the previous paragraph as an interested party's argument and check the numbers, which are all from other people's primaries. This wiki's own thesis about long-horizon agent work is stated separately and is explicitly labelled a prediction, not a finding (The Binding Constraint Is Verified Context, Not the Model or the Rail (our prediction)). Nothing on *this* page tests it. Everything on this page is about a different, narrower question - what a context window can hold versus what it can use - and that question is answered by five external benchmarks that owe us nothing, two of which (LongBench v2's RAG result, Vending-Bench's memory-capacity result) return answers we would rather not have.

How to contradict this page

- Produce the primary for the "200-400K effective" band. A length-controlled, threshold-defined measurement at 256K or above, on a named model version, with a named harness. We looked and did not find it. If it exists, the "What does NOT work" entry above is wrong and should be replaced by the number. - Show a 2026-model measurement. The newest primaries here cover GPT-4.1, Gemini 2.5 Flash, Llama 4 (arXiv:2502.05167v3, 9 July 2025) and o3 / Gemini 2.5 Pro (NoLiMa README @ cb14780, 2025-07-17). If a current-generation model has an effective length near its advertised one under NoLiMa's threshold, the shape of this page changes and the tables should be rewritten rather than annotated. - Break check 1. Open arXiv:2404.06654v3 Table 3 and the NVIDIA/RULER README at commit ab17b78 and show a 32K in the Llama3.1-70B headline effective-length cell. Both are public. - Show that the grid caveat is wrong. If either harness reports a crossing point rather than a grid point, then "32x" is exact and our 16x-64x range is over-cautious. - Run RULER and NoLiMa on your own corpus and get a different answer. That would not refute the primaries, but it would refute step 7 of the procedure, which asserts that your distractors matter more than theirs.

Supersession

This page supersedes no page wholesale. The memoria/ cluster of this wiki already contains The Geometric Ceiling of Single-Vector Retrieval: What arXiv 2508.21038 Proves, and the Four Things It Does Not, which measures a different bound - the geometry of single-vector retrieval, not the usable span of a context window - and which this page neither corrects nor replaces. What this page does correct and qualify runs in four directions, all recorded here rather than left implicit:

1. It supersedes this wiki's own unbased use of the Vending-Bench sentence. The Binding Constraint Is Verified Context, Not the Model or the Rail (our prediction) quotes "We find no clear correlation between failures and the point at which the model's context window becomes full" from the abstract alone. The base is in section 3.6 - Pearson r = 0.167 over 9 rows, with agent memory capped at 30k tokens against models allowing 100k-2M - and it materially narrows what the sentence can support. Anyone citing that sentence from either page should cite the base with it. 2. It records that arXiv:2502.05167v3 mis-states RULER (check 1) and mis-states its own Table 3 (check 2). Neither error changes NoLiMa's conclusions; both would propagate into anyone quoting the sentence rather than the tables. 3. It records that arXiv:2502.05167v3's NoLiMa-Hard sentence "Across all models, performance drops below the 50% mark at 32K context length" is superseded by the authors' own repository at commit `cb14780` (2025-07-17), where GPT-o3 (58.5) and Gemini 2.5 Pro (58.6) sit above that mark. The paper is not wrong about the models it tested; it is out of date about its own benchmark. 4. **It qualifies The Binding Constraint Is Verified Context, Not the Model or the Rail (our prediction) in one direction and supports it in another.** That page's argument that capacity is not curation is supported here by five independent benchmarks. But the LongBench v2 RAG result and the Vending-Bench memory-capacity result are published evidence that adding retrieval, or adding memory, made things *worse* for specific models on specific tasks. Those belong on the record alongside the thesis, not in a footnote to it (What Is Actually Priced: The One-Cent Ceiling, and the Market That Escaped It).