Chunking, Measured: What the Two Published Numbers Actually Say, and What Their Bases Leave Blank

Chunking, Measured

Two chunking techniques have published numbers attached to them. One is a vendor engineering post, one is a preprint. Both numbers are real. Both are averages, one of them is not a benchmark score at all and the other is a benchmark score produced under a modified harness — and the two cannot be compared to each other, not because anyone was careless, but because they measure different quantities on different corpora with different metrics, and the only experiment that has ever put them side by side had one document in it.

This page states each number with the four things a retrieval figure needs to be citable — value, universe, metric, and baseline — and marks the blanks where a primary does not supply one. The blanks are the point. A relative improvement without its baseline is not a measurement, and both of the famous numbers here are relative improvements.

The two objects, with identifiers

| | Contextual Retrieval | Late chunking | |---|---|---| | Identifier | https://www.anthropic.com/engineering/contextual-retrieval | arXiv:2409.04701, DOI 10.48550/arXiv.2409.04701 | | Date of the fact | published 19 September 2024 | v1 7 Sep 2024, v2 2 Oct 2024, v3 7 Jul 2025 | | Publisher | Anthropic (Engineering); writing credited to Daniel Ford | Jina AI GmbH + Weaviate B.V.; Günther, Mohr, Williams, Wang, Xiao | | Peer review | none; engineering blog | none recorded on the arXiv listing as of 2026-08-26 | | Code | cookbook at platform.claude.com/cookbook/capabilities-contextual-embeddings-guide | github.com/jina-ai/late-chunking @ 1d3bb02, 2024-12-23 | | Harness | not published | MTEB 1.14.20, via a custom AbsTaskChunkedRetrieval subclass | | What it changes | inserts an LLM-written context string in front of each chunk before embedding | moves the chunk boundary to after the transformer and before mean pooling | | Marginal cost per chunk | one LLM call | no additional model call — but see the cost section |

Two URLs moved. https://www.anthropic.com/news/contextual-retrieval returns HTTP 307 and reaches /engineering/contextual-retrieval in two hops, via /research/contextual-retrieval (each hop 307; checked 2026-08-26). The Jina paper's bibliography cites the /news/ form, accessed 2024-09-29, and so do most secondary write-ups. Separately, the post's "our cookbook" link now resolves to platform.claude.com, not to the GitHub repository older citations point at. Both current targets return HTTP 200.

The two are not alternatives of the same kind, and the honest comparison has to say so before it says anything else. Contextual Retrieval is an *indexing-time expenditure*: you pay a generative model once per chunk to write 50–100 tokens of situating context. Late chunking is an *ordering change*: the same embedding model sees the same tokens, but the pooling boundary is applied to token embeddings instead of to the raw text. One buys quality with money; the other buys it with a longer forward pass and a model that supports one.

Contextual Retrieval: the three numbers and the four blanks

Regime: citational. Source: the Anthropic post of 19 September 2024, read from a local text extraction.

The post's headline is one sentence:

> This method can reduce the number of failed retrievals by 49% and, when combined with > reranking, by 67%.

The three arms it reports, verbatim:

| Arm | Failure rate | Relative reduction | |---|---|---| | Contextual Embeddings | 5.7% → 3.7% | 35% | | \+ Contextual BM25 | 5.7% → 2.9% | 49% | | \+ Cohere reranking | 5.7% → 1.9% | 67% |

Value — three relative reductions off a 5.7% failure rate.

Metric — stated exactly: "We use 1 minus recall@20 as our evaluation metric, which measures the percentage of relevant documents that fail to be retrieved within the top 20 chunks." So this is a *retrieval* number, not an answer-quality number. The post's own bridge — "which directly translates to better performance in downstream tasks" — is an assertion, not a measurement, and no downstream measurement appears in the post.

Universe — "the average performance across all knowledge domains with the top-performing embedding configuration (Gemini Text 004) and retrieving the top-20-chunks", where the domains are "codebases, fiction, ArXiv papers, Science Papers". Reranking runs over an initial retrieval of "the top 150", narrowed by "the Cohere reranker" (no version) to "the top 20".

Now the four blanks.

1. The baseline configuration is never stated. The word "baseline" does not occur in the post. This is the largest hole and it sits directly under the most-quoted number. Both the 35% arm and the 49% arm are quoted against the *same* 5.7%. *Inference, ours, not stated by the post:* since the 49% arm differs from the 35% arm by adding a contextualised BM25 index, a shared 5.7% baseline is consistent only with a baseline that has no BM25 in it at all — i.e. plain embeddings. If that reading is right, then the 49% bundles two changes: contextualising the chunks *and* adding lexical retrieval. The post's own conclusion says "Embeddings+BM25 is better than embeddings on their own", so some unstated share of the 49% is BM25 doing what BM25 does, not contextualisation. 2. The arm that would settle it was run, and is unreadable. This is the sharp version of blank 1. The post states, verbatim: "We ran a large number of tests, comparing different combinations of all the techniques described above (embedding model, use of BM25, use of contextual retrieval, use of a reranker, and total # of top-K results retrieved), all across a variety of different dataset types." *Use of BM25* and *use of contextual retrieval* are separate axes there, so a non-contextual embeddings+BM25 arm exists in the underlying data. Where does it live? Appendix I — "a breakdown of results across datasets, embedding providers, use of BM25 in addition to embeddings, use of contextual retrieval, and use of reranking" — which is delivered as <img> elements whose alt attributes are empty strings. Seven images, all alt="", no table, no CSV, no data file. The number that would decompose the 49% was measured and then published in a form no reader and no machine can read. 3. No query counts, no corpus sizes, no per-domain totals, no run counts, no intervals. Searched for; absent. A 5.7%→2.9% move is 2.8 percentage points; whether that is 28 queries out of 1,000 or 2.8 out of 100 is not knowable from the primary, and neither is its variance. The post's own summary bullets are qualitative where it matters most — "Adding context to chunks improves retrieval accuracy a lot" carries no number at all. 4. The 35% is not a separate, softer "cross-domain average". All three numbers are averages across the four domains; the configuration sentence governs all the graphs. The 35% is specifically the Contextual-Embeddings-only arm — the first row of the same table, not a fourth figure. We state this flatly because the briefing that produced this page carried the other reading, which is evidence that the reading is easy to fall into, not evidence about how widely it is held.

What the post does model well, and it is in the same post: the cost figure arrives with its assumptions attached. "Assuming 800 token chunks, 8k token documents, 50 token context instructions, and 100 tokens of context per chunk, the one-time cost to generate contextualized chunks is $1.02 per million document tokens." That is how a derived number should arrive. The quality figures in the same post do not arrive that way. See Context Engineering: Where the Term Came From, What It Denotes, and Which of Its Parts Have Actually Been Measured, which reads the same post from the angle of what the term means rather than what the numbers rest on, and which works out what the averaging over four unlike domains costs the reader.

Late chunking: benchmark, harness, and what Table 2 actually contains

Regime: citational for the printed values, attested for the recomputations.

The universe is small and the paper says so: "We restrict the evaluation on the smaller datasets, as splitting documents into smaller chunks increases the computational effort of the evaluation."

- Metric: nDCG@10, chunk rankings folded back to document rankings against BEIR QRels, with the fold-back rule stated in §4.1: "for documents occurring multiple times in the ranking, only the first occurrence is retained". - Benchmark: BEIR (Thakur, Reimers, Rücklé, Srivastava, Gurevych, *BEIR: A Heterogeneous Benchmark for Zero-Shot Evaluation of Information Retrieval Models*, NeurIPS 2021 Datasets & Benchmarks Track, Round 2). - Harness — not named in the paper, recoverable only from the repository. At jina-ai/late-chunking @ 1d3bb02, pyproject.toml pins mteb==1.14.20, transformers==4.43.4, torch==2.4.0, datasets==2.19.1, llama-index==0.11.10. The evaluation is not stock MTEB: chunked_pooling/mteb_chunked_eval.py defines AbsTaskChunkedRetrieval(AbsTask), a custom subclass that chunks the corpus, pools per chunk and folds chunk rankings back to documents, with truncate_max_length defaulting to 8192. Consequence, and it is the reason this bullet exists: the nDCG@10 values in Table 2 are *not* comparable to the published MTEB or BEIR leaderboard scores for the same model names. Anyone reading "jina-embeddings-v3 scores 71.8 on SciFact" here against a leaderboard row is comparing two different measurements that share a benchmark name. - Datasets (4): SciFact, NFCorpus, FiQA, TRECCOVID — the small end of BEIR. - Models (3): jina-embeddings-v2-small (arXiv:2310.19923), jina-embeddings-v3 (arXiv:2409.10173), nomic-embed-text-v1 (arXiv:2402.01613). - Chunking strategies (3): fixed-size 256 tokens; 5-sentence boundaries; semantic sentence boundaries via the llama-index implementation "with the default parameters", using jina-embeddings-v2-small-en for the similarity. - Base of each headline figure: 12 observations (3 models × 4 datasets) per strategy.

The paper's own summary, verbatim:

> Averaging results across three models and four datasets, we find a 3.63% relative improvement > (1.9% absolute) from naive chunking with sentence boundaries to late chunking using sentence > boundaries, a 3.46% improvement (1.8% absolute) from naive chunking to late chunking using > fixed-size boundaries, and a 2.70% improvement (1.5% absolute) from naive chunking to late > chunking when using semantic sentence boundaries.

Our recomputation from the printed cells (regime: attested; method: mean of the 12 cells per row, plain Python, inputs are the printed table):

| Strategy | naive AVG | late AVG | absolute | relative | paper prints | |---|---|---|---|---|---| | Fixed-size 256 | 52.25 | 54.04 | +1.79 pts | +3.43% | 1.8 / 3.46% | | Sentence (5) | 52.38 | 54.28 | +1.91 pts | +3.64% | 1.9 / 3.63% | | Semantic | 52.42 | 53.83 | +1.41 pts | +2.69% | 1.5 / 2.70% |

Two things fall out. First, the semantic row's "1.5% absolute" does not follow from Table 2; the printed cells give 1.408, and the paper's own relative figure (2.70%) is consistent with 1.41, not with 1.5. It is a rounding slip in the prose, not in the data, and it does not change the conclusion — but it is the kind of thing that gets quoted as "1.5 points" downstream. Second, note the unit: these "absolute %" figures are percentage points of nDCG@10, written with a percent sign. Points and percent are used interchangeably in that sentence.

How consistent is the win? The paper says "replacing naive methods with their late chunking counterparts almost always yields better performance". Quantified from the same table (regime: attested): across all 36 model × dataset × strategy comparisons, late chunking is higher in 33, tied in 2, and lower in 1 — the single loss being nomic-embed-text-v1 on SciFact with fixed-size chunks, 70.7 → 70.6. Both ties are nomic-embed-text-v1 on NFCorpus at 35.3, one under fixed-size and one under semantic boundaries. All of this is at the one decimal place the table prints; a tie at one decimal is not necessarily a tie underneath.

Magnitudes are not uniform, and here is the actual distribution. *Corrected on 2026-08-26: an earlier draft of this page said eight cells of the fixed-size row move by less than one point. Recomputed, three do. The full row, naive → late, is worth printing instead of summarising:*

| cell (fixed-size 256) | movement | |---|---| | NFCorpus / jina-embeddings-v2-small | 23.5 → 30.0, +6.5 | | TRECCOVID / jina-embeddings-v3 | 73.0 → 77.2, +4.2 | | TRECCOVID / nomic-embed-text-v1 | 72.9 → 75.0, +2.1 | | SciFact / jina-embeddings-v2-small | 64.2 → 66.1, +1.9 | | SciFact / jina-embeddings-v3 | 71.8 → 73.2, +1.4 | | FiQA / jina-embeddings-v3 | 46.3 → 47.6, +1.3 | | FiQA / nomic-embed-text-v1 | 37.0 → 38.3, +1.3 | | TRECCOVID / jina-embeddings-v2-small | 63.4 → 64.7, +1.3 | | NFCorpus / jina-embeddings-v3 | 35.6 → 36.7, +1.1 | | FiQA / jina-embeddings-v2-small | 33.3 → 33.8, +0.5 | | NFCorpus / nomic-embed-text-v1 | 35.3 → 35.3, 0.0 | | SciFact / nomic-embed-text-v1 | 70.7 → 70.6, −0.1 |

A single cell carries 6.5 of the 21.5 points of total movement in that row — 30% of the gain from one twelfth of the observations. The mean of +1.79 is a real average and is not a typical case.

The comparison at declared cost, which cannot be completed

This is where an honest page has to stop short of the conclusion people want.

Contextual Retrieval declares a cost. $1.02 per million document tokens, one-time, at indexing, with its four assumptions printed. The generating model is Claude 3 Haiku — the post names it in prose only ("We used the following Claude 3 Haiku prompt to generate context for each chunk"), and the Jina paper's reimplementation is what pins the exact checkpoint, claude-3-haiku-20240307. The post is explicit that the price depends on prompt caching: "Contextual Retrieval is uniquely possible at low cost with Claude, thanks to the special prompt caching feature." Without caching, the whole document is re-sent for every chunk.

That declared cost is no longer checkable. *Regime: citational.* Anthropic's deprecation page, read 2026-08-26, lists claude-3-haiku-20240307 as Retired, deprecated 19 February 2026 and retired 20 April 2026, replacement claude-haiku-4-5-20251001. Its per-token price is not on the current public pricing page: the string claude-3-haiku occurs zero times there, and "Haiku 3" survives only in two navigation entries for a release-notes page. This is not merely a lifecycle default — Claude Haiku 3.5 is also retired and is still carried as a priced row, annotated "retired, except on Bedrock and Google Cloud". Haiku 3 is not. So the $1.02 rests on a September-2024 price list for a model whose price the vendor no longer publishes. The figure is not wrong; it is undated in most quotations of it, which is the failure mode. Anyone quoting it in 2026 is quoting a two-year-old price for a retired model, and the same retirement makes the post's *quality* measurement unreproducible on its original configuration too.

Late chunking declares no cost at all. *Regime: citational, by recorded absence.* Searched across the full v3 text: "wall-clock", "FLOP", "latency" and "GPU hour" occur zero times, the word "cost" occurs zero times, and the only dollar sign in the paper is inside the fictional document of Table 4. What the paper offers instead is a comparative assertion about the other technique — "This is however computationally more expensive, as LLMs are typically much larger than embedding models or even require paid access to LLM APIs" — which is almost certainly true and is not measured.

And late chunking is not free, which the paper half-concedes without quantifying. Its own motivation for the "long late chunking" variant is that "the memory required for the encoding increases exponentially with an increasing number of tokens". Two notes on that sentence: the scaling of attention in sequence length is *quadratic*, not exponential, so the word is wrong; but the direction is right and it is the paper telling you that one forward pass over 8,192 tokens is not the same expenditure as thirty-two passes over 256 tokens. Nobody has published what that difference costs.

So the honest state of the comparison, as of 2026-08-26

one technique has a declared cost with stated assumptions resting on a price list the vendor has withdrawn, and the other has no declared cost of any kind. A comparison at declared cost cannot be completed, and any confident statement that late chunking is "the same quality for free" is filling in a blank that neither primary fills.

The only head-to-head that exists has n = 1

Regime: citational. This is the single most important structural fact on this page and it is routinely lost.

§4.5 of arXiv:2409.04701v3, "Comparison to Contextual Embedding", is described by its own authors as "a small-scale experiment". Its full extent:

- one fictional financial document about "ACME Corp", constructed for the purpose - five chunks - one query: "What is ACME Corp's revenue growth for Q2 2023?" - embedder jinaai/jina-embeddings-v2-small-en, contextualiser claude-3-haiku-20240307 - metric: raw cosine similarity, not a retrieval metric

The result, printed in Table 4: on the relevant chunk ("It highlighted a 3% revenue growth over the previous quarter"), late chunking scores 0.8516, contextual embedding 0.8590, naive chunking 0.6343. Both methods give the relevant chunk their highest score of the five; naive chunking ranks it below three others. The design is adversarial to naive chunking by construction, and the paper says so — the relevant chunk "misses the company's name, which is necessary to determine its relevancy". The authors' conclusion from it is careful and is about cost, not accuracy: "contextual embedding and late chunking produce similarity scores that are close to each other across all chunks, with late chunking having the advantage that it does not require using an additional large language model."

There is no benchmark comparison of the two techniques in either primary, and none in the sources this page reaches. The Anthropic post predates the paper's later versions and never mentions late chunking. Whoever runs BEIR or BEIR-adjacent tasks with both methods on the same embedder, same chunker and same queries will be the first. Note that the Jina repository at 1d3bb02 contains explanatory_contextual_retrieval.py, so the scaffolding for that experiment already exists in public.

What the versions changed, and why version pinning matters here

Regime: citational for both versions; attested for the correlation.

This preprint is a case where citing "Günther et al., 2024" without a version number points at a different set of claims than citing v3.

Dropped claim 1 — gains grow with document length. v1 (7 Sep 2024) states, verbatim: "Moreover, a higher average length of the documents correlates with a larger improvement achieved by late chunking over conventional chunking." Its base: five BEIR datasets, one model (jina-embeddings-v2-small), one chunking method (256 tokens), and "length" means the average number of *characters* per document, not tokens. Our recomputation from v1's Table 2: the rank correlation between average document length and the late−naive gap is Spearman 1.0 over those five points (Pearson 0.74) — but one of the five is Quora, where the paper itself says "the documents only contain one chunk", so the two methods are identical by construction and the gap is exactly 0.00. Excluding it leaves four points, still monotone (Spearman 1.0, Pearson 0.76). A monotone trend over four non-degenerate datasets with one model is a hypothesis, not a finding, and v3 does not contain the sentence. What v3 reports instead is a different axis: "late chunking performs better than naive chunking, specifically for small chunk sizes", with the explicit exception that "for some of the reading comprehension tasks, naive chunking works better when using large chunks." If you have seen "late chunking's advantage grows with document length" attributed to this paper, it is a v1 claim that the current version withdrew.

Dropped claim 2 — the no-chunking baseline. v1's Table 2 has a "Full" column, the score with no chunking at all. v3's §4.1 still says "and also report the score obtained without chunking" — and v3's Table 2 has no such column. The text promises a number the table does not deliver. That matters, because in v1 the missing column is where late chunking loses:

| dataset (v1, jina-embeddings-v2-small) | naive | late | no chunking | |---|---|---|---| | SciFact | 64.20 | 66.10 | 63.89 | | TRECCOVID | 63.36 | 64.70 | 65.18 | | FiQA2018 | 33.25 | 33.84 | 33.43 | | NFCorpus | 23.46 | 29.98 | 30.40 | | Quora | 87.19 | 87.19 | 87.19 |

On two of the four non-degenerate datasets, embedding the whole document with no chunking at all beat late chunking. The v1 numbers are the same runs as v3's jina-embeddings-v2-small fixed-size row, carried at one more decimal — all eight shared cells agree under round-half-up (64.20→64.2, 66.10→66.1, 63.36→63.4, 64.70→64.7, 33.25→33.3, 33.84→33.8, 23.46→23.5, 29.98→30.0; FiQA's 33.25 is the exact half-way case and is the only one where the rounding rule matters). So this is not a superseded experiment — it is the same experiment with a column removed.

An internal inconsistency to know before you cite Table 2

Regime: attested (cell-by-cell comparison of two printed tables in the same PDF).

Here is the whole sentence from §4.1 of v3, quoted entire because a truncated version of it makes this section look stronger than it is:

> In all experiments the chunks are non-overlapping, however, additional results demonstrated in > appendix A.2 show that overlapping the chunks generally neither improves nor harms the retrieval > performance.

Read plainly, that says the experiments of §4.1 — Table 2 among them — are non-overlapping, and that A.2 adds a separate overlapping comparison. Appendix A.2's Table 6 compares overlapping (16 tokens) against non-overlapping chunks for jina-embeddings-v2-small with fixed-size 256-token chunks: the same configuration as Table 2's first row.

If Table 2's row were non-overlapping, its cells should equal Table 6's "without overlap" columns. They equal the "with overlap" columns instead, all eight of them, and match "without overlap" in none:

| dataset | Table 2 naive | T6 w/ overlap | T6 w/o overlap | Table 2 late | T6 w/ overlap | T6 w/o overlap | |---|---|---|---|---|---|---| | SciFact | 64.2 | 64.2 | 61.7 | 66.1 | 66.1 | 65.9 | | NFCorpus | 23.5 | 23.5 | 22.8 | 30.0 | 30.0 | 30.5 | | FiQA | 33.3 | 33.3 | 32.8 | 33.8 | 33.8 | 34.0 | | TRECCOVID | 63.4 | 63.4 | 64.5 | 64.7 | 64.7 | 64.9 |

Either the sentence in §4.1 is wrong about its own headline row, or Table 6's two column labels are swapped. Which one does not change the paper's conclusion — with the "without overlap" naive baseline the gaps are, if anything, mostly larger — but it does mean the exact configuration behind the most-cited row is ambiguous in the primary, and it means one of the two overlap columns is carrying a number from the wrong condition. Reproduce in a minute from the PDF: transcribe both tables and compare. We report this to be checked, not as an established error in the data.

What does NOT work

The exclusions here are mostly the primaries' own, which is what makes them worth keeping.

Things the Anthropic post tried and rejected, verbatim: adding generic document summaries to chunks — "we experimented and saw very limited gains"; summary-based indexing — "we evaluated and saw low performance". A third technique, "hypothetical document embedding", is listed alongside them as a prior proposal with no result reported either way — do not read the post as evidence for or against HyDE. So *some* form of prepended text helps and *some* does not: the chunk-specific, document-conditioned context works, the generic summary does not. That distinction is the actual content of the technique.

Chunk overlap does not earn its keep. Table 6 of the late-chunking paper: 16-token overlap on 256-token chunks, four datasets, one model. "The results in Table 6 do not show a clear advantage of using overlaps. The nDCG@10 scores are relatively similar regardless." Overlap is the cheapest and most widely deployed context-preservation trick, and this is one of the few measurements of it; it neither helps nor hurts on this universe. Carry the base with it: 8 comparisons, one model.

Late chunking does nothing when the surrounding context is irrelevant. On the synthetic Needle-8192 and Passkey-8192 datasets from LongEmbed (Zhu, Wang, Yang, Song, Wu, Wei, Li, arXiv:2404.12096), the paper is blunt: "In this case, late chunking is not useful, as the additional context from the document is totally irrelevant." Both techniques are context-injection techniques. If the document around the answer does not help identify the answer, injecting it is a no-op — and on needle-style tasks it is worse than a no-op, because it dilutes the pooled vector with unrelated tokens.

Naive chunking wins at large chunk sizes on reading-comprehension retrieval. v3 §4.2, same family of tasks. The advantage of late chunking is concentrated at small chunk sizes.

Chunking itself is not always a win. Appendix A.1 measures naive chunking against no chunking on four non-synthetic LongEmbed tasks with jina-embeddings-v2-small, texts truncated at 8,192 tokens, 512-token chunks. The paper reports "+24.47%" average relative improvement. Our recomputation of the four per-dataset relative changes behind that average (regime: attested): NarrativeQA 32.73→47.63 = +45.5%, QMSum 36.81→48.34 = +31.3%, 2WikiMultiHopQA 70.32→86.3 = +22.7%, SummScreenFD 91.24→89.71 = −1.7%. Mean of the four: +24.4737%, reproducing the printed figure exactly. So the headline is an average over four observations spanning 47 percentage points of relative change, one of them with the opposite sign — and the paper says so: "Only on the SummScreenFD task did retrieval without chunking perform slightly better."

Neither technique moves the geometric ceiling. See The Geometric Ceiling of Single-Vector Retrieval: What arXiv 2508.21038 Proves, and the Four Things It Does Not. That result (arXiv:2508.21038) is about what a *fixed embedding dimension d* can realise as a top-k set, for any query. Contextual Retrieval changes the text that goes into a vector; late chunking changes when the pooling boundary is applied. Neither changes d, neither adds a second vector, and neither appears among the escape routes that paper names (cross-encoders, multi-vector/late-interaction, sparse retrieval, new single-vector architectures). *Inference, ours:* chunking also multiplies the number of independently retrievable units, so under Theorem 1 the required dimension d ≥ log C(n,k)/log(1+1/γ) goes *up*, not down — though the sibling page's own finding is that the bound is toothless at realistic k, so this is a direction, not a mechanism. Recorded absence: no source on this page tests either chunking technique on LIMIT, so "chunking fails on LIMIT" is not something we can assert — only that a technique which improves how well a vector represents its own context has no stated purchase on a failure mode that is not about context at all. The Needle-8192 result above is the closest measured analogue in this literature, and it points the same way.

Contextual Retrieval is out of scope below 200K tokens, by the post's own admission: "If your knowledge base is smaller than 200,000 tokens (about 500 pages of material), you can just include the entire knowledge base in the prompt that you give the model, with no need for RAG or similar methods." Whether that holds is a separate question with its own evidence — see Effective Context vs Advertised Context: Which Number Actually Bounds Your Agent — but the technique's own author draws the lower boundary of its usefulness there.

The late-chunking training method is a small effect on thin data. §4.4's span-pooling fine-tuning gives a consistent but tiny improvement (jina-embeddings-v3 on SciFact 72.59 → 72.61 at the same training set), and the authors state the limitation themselves: "the training dataset diversity is quite limited, as both training datasets are sourced from Wikipedia documents. The summed dataset encompasses only ∼470k pairs in total." Note also that the reference repository at 1d3bb02 contains no training code for this method — no path in its 20 files matches "train" — so §4.4 is the one part of the paper the published code does not let you reproduce. Do not quote the training numbers as part of the late-chunking result; they are a separate, weaker claim.

How to read a chunking number someone quotes at you

A procedure, usable against any of the numbers on this page or any that arrive next.

1. Is it relative or absolute? "49% fewer failed retrievals" is relative. Ask for both endpoints. 5.7% → 2.9% is a 2.8-point move; "49%" is the same fact dressed to sound larger. 2. What is the baseline configuration? Not the baseline *number* — the baseline *system*. If the improved arm adds two things at once, the number measures both. This is the blank under the most-cited chunking figure on this page. 3. What metric, and is it a retrieval metric or an answer metric? recall@20 and nDCG@10 are retrieval. Neither is answer accuracy, and the step from one to the other is an assumption. 4. Benchmark name is not enough — demand the harness and its version. "nDCG@10 on SciFact" was produced here by a custom MTEB subclass at mteb==1.14.20 that chunks the corpus and folds chunk rankings into document rankings. Those scores do not belong on the same axis as a leaderboard row for the same model and the same dataset name. 5. How many observations behind the average, and what is the spread? "3.46% averaged over three models and four datasets" is a base. "35% average across domains" without the domain count is not. When you can see the cells, look for the one carrying the mean: here, one cell of twelve carries 30% of the movement. 6. Which version, which model checkpoint, which commit? This literature moves claims between versions. claude-3-haiku-20240307 is retired; v1 and v3 of arXiv:2409.04701 make different claims about document length; the reference implementation is pinned at a 2024 commit. 7. What did it cost, and when was that price true? A quality gain with no cost attached is half a result, and a cost attached to a retired model is a dated one.

For why two numbers with the same benchmark name can still be incomparable, the same disease in a neighbouring field, see Three LoCoMo Scores Side by Side Are Not a Ranking: the denominator, the judge, and the one knob worth nine points.

Supersession and open questions

- Supersedes, within a source: arXiv:2409.04701v3 withdraws v1's document-length correlation and removes v1's no-chunking column. Anyone citing the length claim should cite v1 explicitly and carry its base (5 datasets, 1 model, length in characters, one dataset degenerate). - Supersedes, within this page: the "eight cells move by less than one point" figure in the first draft is withdrawn — the correct count is three, and the full distribution now replaces the summary. Recorded rather than deleted, because the page's whole argument is that unreproduced derived numbers are the failure mode, and it had one. - Open, and cheap to close: both techniques on the same embedder, same chunker, same query set, same harness, reported with cost. The code for both sides is public. Nobody has published it. - Open: the cost of late chunking in wall-clock, memory or FLOPs versus naive chunking at a fixed corpus size. Unmeasured in the primary. - Open: whether the Anthropic result reproduces on a current model. The measurement's contextualiser was retired 2026-04-20 and the post has not been re-run publicly. - Would supersede this page: any per-domain data release from Anthropic that resolves the 5.7% baseline configuration, or a v4 of arXiv:2409.04701 that restores the no-chunking column. Either would move claims here from "blank" to "measured", and this page should be edited, not appended. - Contradiction we are carrying, not resolving: §4.1's "non-overlapping" sentence against Table 6's column labels. Reported to be checked by a reader with the PDF, not asserted as an error in the data.