Three LoCoMo Scores Side by Side Are Not a Ranking: the denominator, the judge, and the one knob worth nine points

A buyer comparing agent-memory vendors in 2026 will be handed three numbers with the same label on them. 92.5 (Mem0). 75.14 (Zep). 74.0 (Letta). They look like a leaderboard.

They are not a leaderboard. They are three different measurements of three different question sets, scored by three different judges under three different rubrics, with three different models writing the answers — and in two of the three cases, the published source does not say which judge or which question set. This page shows what each number is made of, and measures how far the number moves when you turn a single dial and change nothing else.

The short version: on the same system, the same 1,540 questions and the same judge model, turning one retrieval dial moved the score 8.90 points — and that is bigger than most of the gaps the numbers are used to argue about.

The denominator: what "LoCoMo" refers to

Regime: citational (the two documents) + attested (our count).

There are two LoCoMo datasets, and almost every published score uses the smaller one while citing the paper that describes the larger one.

The paper — *Evaluating Very Long-Term Conversational Memory of LLM Agents*, arXiv 2402.17753, published as ACL Anthology 2024.acl-long.747, Maharana, Lee, Tulyakov, Bansal, Barbieri and Fang, single arXiv version v1 submitted 27 February 2024 — describes in its Table 5 a corpus of 50 conversations, averaging 19.3 sessions and 9,209.2 tokens each, with 7,512 questions in five categories: single-hop 2,705 (36%), multi-hop 1,104 (14.6%), temporal 1,547 (20.6%), open-domain 285 (3.9%), adversarial 1,871 (24.9%).

The public release is a different object, and the authors say so plainly in the repository README (github.com/snap-research/locomo, branch main, last push 2024-08-13):

> "This release is a subset of the conversations released previously with our first Arxiv > version in March 2024. The initial release contained 50 conversations. We sampled a subset of > the data to retain the longest conversations with high-quality annotations and for > cost-effective evaluation of closed-source LLMs."

We counted the released file ourselves on 2026-08-26. This is our own measurement, it takes ten seconds, and it is the cheapest thing in this page to falsify:

``python import json, collections d = json.load(open('locomo10.json')) # from snap-research/locomo, data/ print(len(d)) # 10 qs = [q for c in d for q in c['qa']] print(len(qs)) # 1986 print(collections.Counter(q['category'] for q in qs)) # {4: 841, 5: 446, 2: 321, 1: 282, 3: 96} print(sum(1 for q in qs if q['category'] != 5)) # 1540 print(sum(1 for q in qs if 'answer' not in q)) # 444 ``

| Category (id) | Count in the released file | Share | |---|---|---| | single-hop (4) | 841 | 42.35% | | temporal (2) | 321 | 16.16% | | multi-hop (1) | 282 | 14.20% | | open-domain (3) | 96 | 4.83% | | adversarial (5) | 446 | 22.46% | | Total | 1,986 | 100% | | Total excluding adversarial | 1,540 | 77.54% |

The category-id → name mapping is not our guess: it is read verbatim from a vendor's published harness (CATEGORY_NAMES in benchmarks/locomo/prompts.py, mem0ai/memory-benchmarks, commit 4b61c5d), and the per-category totals in that vendor's result files — 282 / 321 / 96 / 841 — match our independent count of the benchmark file exactly.

1,540 is the number to remember. It is what "LoCoMo" means in practice, and it is not 7,512.

The same file, described three ways, by three sellers

While we had the file open we checked the descriptions of it that appear in the published work. Same script, same file, same day. This is our measurement, not a quotation.

| Description | Whose | Our count of the released file | |---|---|---| | "approximately 600 dialogues ... on average" | Mem0, arXiv 2504.19413 §3 | 588.2 turns per conversation (range 369–689) — holds | | "200 questions on an average" | Mem0, arXiv 2504.19413 §3 | 198.6 per conversation — holds, but the range is 105–260, so conversations are not equally weighted | | "10 users, 35 conversation sessions each" | Zep, 09 Dec 2025 | 19, 19, 32, 29, 29, 28, 31, 30, 25, 30 — mean 27.2, max 32. No conversation has 35 sessions. | | — (no denominator stated at all) | Letta, 12 Aug 2025 | the post gives no question count and no category subset |

Zep's figure is wrong about a file anyone can download, in a post whose entire argument is that other people benchmark carelessly. We hold it to the standard it holds Mem0 to, and to the one we would want applied to us. It does not change Zep's accuracy numbers — the sweep was run on the real file — but it is the fourth different description of the same denominator in this page.

The category everybody drops, and why

Regime: citational + attested.

Of the 446 adversarial items in the released file, 444 carry no `answer` field at all — they carry an adversarial_answer instead, which is the *wrong* answer the question is designed to bait. Two items carry both. A standard answer-versus-gold judge has nothing to compare against.

Both vendors say the same thing about it, from opposite sides of a fight:

> Mem0, arXiv 2504.19413, §3: "this category was excluded from our evaluation because ground > truth answers were unavailable, and the expected behavior for this question type is that the > agent should recognize them as unanswerable."

> Zep, 06 May 2025: "Category 5 was unusable due to missing ground truth answers, forcing both > Mem0 and Zep to exclude it from their evaluations."

They are right, and the exclusion is defensible. But notice what leaves with it. In the original paper's own Table 2, the adversarial column is where the ranking inverts: gpt-3.5-turbo-16K with a 16K window scores the best overall F1 in that table (37.8) and 2.1 on adversarial, while GPT-4-turbo at a 4,096-token window scores 32.1 overall and 70.2 on adversarial. The authors' reading, verbatim: *"LLMs can be easily misled into generating hallucinations when they are subjected to long contexts."*

So the one category that measures *refusing to make something up* is the one that every published memory-vendor score omits. Every "overall" figure in this page is an overall over questions that all have answers. A memory system that confabulates fluently loses nothing on this benchmark.

Every published number, with the six fields it needs

Regime: citational. A LoCoMo score without answerer, judge, harness, retrieval depth, subset and date is a label, not a number. Below is every score we could locate, with the fields its own source actually supplies. "not stated" means we searched the source and it is absent — not that we did not look.

| Score | Whose system | Measured by | Fact date | Answerer | Judge | Subset | Depth | |---|---|---|---|---|---|---|---| | 66.88 ± 0.15 (J) | Mem0 | Mem0 | 2025-04-28 (arXiv v1) | gpt-4o-mini | not named in the paper | 1,540 (cat. 1–4) | n/a (extracted memories) | | 68.44 ± 0.17 (J) | Mem0 + graph | Mem0 | 2025-04-28 | gpt-4o-mini | not named | 1,540 | n/a | | 72.90 ± 0.19 (J) | *no memory system* (full context, ~26k tok) | Mem0 | 2025-04-28 | gpt-4o-mini | not named | 1,540 | n/a | | 65.99 ± 0.16 (J) | Zep | Mem0 | 2025-04-28 | gpt-4o-mini | not named | 1,540 | n/a | | 75.14 ± 0.17 (J) | Zep | Zep | correction undated; post 2025-05-06, upd. 2026-06-03 | not stated | not stated | not stated (cat. 5 excluded) | not stated | | 74.0 | Letta filesystem agent | Letta | 2025-08-12 | gpt-4o-mini | not stated | not stated | agent-chosen, iterative | | 69.62 ± 0.47 → 80.32 ± 0.43 | Zep | Zep | 2025-12-09 | gpt-4o-mini | gpt-4o-mini | not stated (10 conv.) | 5 configs, 10 runs each | | 91.56 (1410/1540) | Mem0 platform | Mem0 | run stamped 2026-04-06 | gpt-5 (azure) | gpt-5 (azure) | 1,540 (cat. 1–4) | top 200 memories | | 82.66 (1273/1540) | Mem0 platform | Mem0 | run stamped 2026-04-08 | gpt-5 (azure) | gpt-5 (azure) | 1,540 | top 50 memories | | 92.5 | Mem0 platform | Mem0 | published 2026-04-16 | not stated | not stated | "1,540 questions 5 categories" | not stated |

Read the two "not stated" columns before the numbers. Of the ten rows, three name their judge — two Mem0 artifacts (gpt-5) and one Zep sweep (gpt-4o-mini). The vendor most criticised in this page for marketing opacity, Mem0, is the only one that names the judge behind a headline figure — and it does so only inside a JSON artifact in a repository, never on the page where the figure is sold. Zep names its judge for the sweep it ran to show the dial's effect, and not for the 75.14 it wants quoted.

Two rows are the same product measured by two parties: Zep at 65.99 (Mem0's run) and Zep at 75.14 (Zep's run) — 9.15 points apart. Three rows are the same product across time and configuration: Zep at 65.99, 75.14 and 80.32 — a 14.33-point range, all of it under the label "Zep on LoCoMo". And Mem0's own product reads 66.88 in April 2025 and 92.5 in April 2026: 25.62 points, across an algorithm change, a model-class change (gpt-4o-mini → gpt-5), and a platform-versus-open-source change that the vendor discloses.

None of those spreads is a lie. All of them are the same benchmark name.

The measured swing: one dial, 8.90 points

Regime: citational for both values; the delta is our subtraction of exactly two published runs and no others.

This is the part that settles the question, because it is a controlled comparison published by one party about its own product, with per-question verdicts on disk.

In mem0ai/memory-benchmarks at commit 4b61c5d31b9c668a12b4f5e78064248a02c82d2b (committed 2026-05-13), two result artifacts sit next to each other:

| File | Run stamp | Answerer | Judge | Questions | Memories in prompt | Correct | Score | |---|---|---|---|---|---|---|---| | results/platform/locomo_results.json | 20260406_170942 | gpt-5 | gpt-5 | 1,540 | 200 | 1,410 | 91.56% | | results/platform/locomo_top50_results.json | 20260408_143630 | gpt-5 | gpt-5 | 1,540 | 50 | 1,273 | 82.66% |

Same system. Same 1,540 questions. Same answerer model. Same judge model. Same provider. Two days apart. 8.90 points, from changing how many retrieved memories are pasted into the prompt. Across all 1,540 evaluations in each file, both memories_evaluated and total_memories_retrieved take exactly one distinct value — 200 in the first, 50 in the second — so this is not an inference about the configuration. It is recorded per question, and the second run is a genuine 50-deep search, not a 50-item slice of a 200-deep one.

The drop is not concentrated in one question type. Per category, top-200 → top-50: multi-hop 93.26 → 82.27, temporal 92.83 → 86.29, open-domain 76.04 → 70.83, single-hop 92.27 → 82.76. Every category loses; multi-hop loses most.

Base of the derivation, and a limitation we did not see on first reading. Two runs, one per arm, 1,540 questions each, no repeats. That is one observation per condition, which is enough to establish that the dial matters and *not* enough to put a confidence interval on 8.90. Worse: both artifacts carry a non-null `merged_from_questions` field listing question ids — 156 of 1,540 in the top-200 file and 274 of 1,540 in the top-50 file. So roughly 10% and 18% of the per-question verdicts in these files were carried in from some other execution, whose configuration the file does not record. Neither artifact is a single clean run. The 8.90-point figure should be read as *the difference between two published result sets*, not as a clean two-arm experiment, and nobody should annualise, extrapolate or generalise it.

For the interval, use the other party's experiment. Zep swept the same dial ten times per setting (50 runs, gpt-4o-mini as both agent and grader, published 09 Dec 2025):

| Config | Accuracy | Context tokens | Retrieval p50 | |---|---|---|---| | 30/30 | 80.32 ± 0.43 | 1,997 | 189 ms | | 20/20 | 80.06 ± 0.33 | 1,378 | 241 ms | | 15/5 | 77.06 ± 0.41 | 756 | 199 ms | | 10/2 | 73.72 ± 0.41 | 504 | 161 ms | | 5/2 | 69.62 ± 0.47 | 347 | 149 ms |

10.70 points end to end over five published configurations, with standard deviations under half a point — the swing is two orders of magnitude larger than the run-to-run noise. Zep's own conclusion is the honest one: *"LoCoMo plateaus near 80% accuracy, but that's the benchmark's ceiling, not Zep's."*

Two independent vendors, two independent harnesses, two different model families, same finding: a retrieval-depth setting is worth roughly nine to eleven points on LoCoMo, which is more than the gap between any two vendors' headline claims. Note that these two experiments are *not* commensurable with each other — different models, different judges, different systems, and one of them is not a clean single run. They agree on a direction and an order of magnitude. Adding, averaging or interpolating between them would be the exact error this page exists to name.

The judge is not a measuring instrument, it is a policy

Regime: citational, verbatim from published prompt files.

The benchmark authors' own harness (task_eval/evaluate_qa.py) scores F1 on normalised exact match — it calls eval_question_answering(...) and writes the returned exact_matches as _f1. There is no LLM judge in it. Every vendor number in this page comes from a judge the vendor wrote.

Mem0's 2025 paper instructs its judge, twice, verbatim: *"you should be generous with your grading"*. Its answering prompt instructs the system under test: *"The answer should be less than 5-6 words."* Short answers help F1 and BLEU-1; a generous judge helps J. (An earlier draft of this page said that prompt was adapted from MemGPT's. We re-read the paper: it makes no such statement, and the claim is withdrawn.)

Mem0's 2026 harness rubric is published in full — genuinely to their credit — and it is a substantially more permissive policy than the 2025 one:

> "PARTIAL CREDIT: If the generated answer includes AT LEAST ONE correct item from the gold > answer's list, mark CORRECT. Getting 1 out of 2, 2 out of 4, etc. is always acceptable. Only > mark WRONG if NONE of the gold answer items appear."

> "DATE TOLERANCE: Dates within 14 days of each other are CORRECT. Durations within 50% are > CORRECT (e.g., '5 months' matches 'six months'; '19 days' matches 'two weeks')."

> "## ONLY mark WRONG if: — The generated answer contains ZERO correct items from the gold > answer — The answer addresses a completely different topic"

And an optional evidence mode that can only push scores up:

> "The gold answer may be wrong or oversimplified; if the generated answer provides a more > accurate or better-supported conclusion based on the evidence, that is acceptable. Use > evidence only to ACCEPT answers, never to reject them more strictly."

(That mode was off in both runs quoted above — with_evidence: false in both artifacts — so the 91.56 and 82.66 figures are not inflated by it.)

There is also a gold-answer transform: preprocess_answer truncates category-3 (open-domain) gold answers at the first semicolon before judging. Combined with partial credit, a two-part open-domain answer becomes a one-part target.

None of this is cheating. A temporal question whose gold answer is "7 May 2023" and whose generated answer is "May 2023" is arguably right, and *someone* has to decide. The point is that the deciding is a published policy, and no two numbers in the comparison table were decided by the same one. A 14-day date tolerance across 321 temporal questions is not a rounding convention; it is a scoring rule with a size.

Two numbers in one repository we could not reconcile

Regime: citational for both, and explicitly unresolved.

At the same commit 4b61c5d, README.md reports LoCoMo Top 200 = 92.5% (1425/1540) and Top 50 = 91.8% (1414/1540). The result files in that same commit report 91.56% (1410/1540) and 82.66% (1273/1540).

Both rows disagree, not just one: Top 200 by 0.97 points (15 questions) and Top 50 by 9.16 points (141 questions). We are not calling this an error and certainly not bad faith.

We can now close half of the mechanism and not the other half. run.py slices a cutoff as a prefix, sliced = formatted[:c], of one search performed at depth --top-k, and the CLI default is --top-k-cutoffs 10,20,50,200. So "Top 50" can mean *the top 50 of a 50-deep search* or *the top 50 of a 200-deep search*, and with a rank fusion those are different sets of 50 memories. The artifact side is settled: total_memories_retrieved is exactly 50 for all 1,540 questions, so the file's 82.66% is a real 50-deep search. The README side is not: it carries no run stamp and no linked artifact, and its own sub-score table is labelled "avg across top_10/20/50/200", which is consistent with a multi-cutoff run from a single 200-deep search but does not establish it. We could not close the gap from the published artifacts.

That is the entire thesis of this page in one repository: the most transparent vendor in the comparison publishes two numbers under one label, nine points apart, and a careful reader with the raw files cannot close it. A buyer holding a slide deck has no chance.

Related: the harness CLI defaults are --answerer-model gpt-4o, --judge-model gpt-4o, --top-k 200. The published runs used gpt-5 on azure. A user who clones the repo and runs it as documented is not reproducing the headline figure; they are producing a new one.

The dispute, with both interests on the table

Regime: citational.

Zep's charges against Mem0's 2025 measurement of Zep are specific and mechanical: the user model was wrong (*"assigned the user role to both participants"*), timestamps were passed by concatenation rather than through Zep's created_at field, and searches ran sequentially, which inflates a latency figure. Those are the kinds of errors that a vendor evaluating a competitor's product makes by default, not by malice, and they are checkable against the code Zep links.

The charges are well-founded in form. Mem0's paper concedes the shape of the problem in its own §4.5, where it reports that Zep's results improved when identical searches were re-run *"after a delay of several hours"* — that is a description of an asynchronous ingestion pipeline being queried before it had finished, which is precisely an integration mistake and not a property of the retrieval quality being scored.

And Zep is interested. It sells the product whose score it is revising upward, its corrected 75.14 was produced by its own team on its own infrastructure, it names neither the answerer nor the judge behind that number, and its post is titled to win. Its own correction notice records that the *first* version of its rebuttal miscalculated Zep's score too. Both parties got their own arithmetic wrong at least once in public. We would.

The generous reading, which we think is the right one: a vendor cannot competently benchmark a competitor's managed service. Not because of dishonesty, but because best-effort integration by a time-constrained team that has never run the product in anger is systematically worse than the vendor's own. Every cross-vendor table in this market inherits that bias, in the direction you would predict, and the size of it here — 9.15 points — is larger than most claimed wins.

The result that embarrasses the whole category

Regime: citational.

Letta put the raw LoCoMo conversations into a file, gave a gpt-4o-mini agent a filesystem toolset (grep, search_files, open, close — the post says the agent used semantic search, text matching and an answer_question terminator), and got 74.0% — *above* the 68.5% it attributes to Mem0's best graph variant. Published 12 August 2025. Letta's own reading, verbatim: *"current memory benchmarks may not be very meaningful"*, and *"memory is more about how agents manage context than the exact retrieval mechanism used."*

Set next to Mem0's 2025 Table 2, where full context with no memory system at all scored 72.90, beating every memory system in the table including Mem0's own 68.44, the pattern is hard to miss: on this benchmark, the specialised memory architectures were, in 2025, losing to *paste everything in* and to *let the agent grep*.

Held to the same standard as everyone else, Letta's number is the least documented in this page: the post states no question count, no category subset, no judge, and no number of runs — four of six fields blank — and its "68.5%" for Mem0 is the paper's 68.44 rounded the wrong way. Letta is a vendor too, competing in the same market, and it is also the party whose MemGPT was benchmarked by Mem0 — it reports that Mem0 *"did not respond to requests for clarification on how the benchmarking numbers were computed."* We have no way to verify a non-response, and we do not treat it as established. The finding is interesting because of its direction, not its precision.

What does NOT work

The exclusions, which are the most useful part of this page.

1. Comparing J scores across publications. Two J scores are comparable only if the judge model, the judge rubric, the answerer model and the retrieval depth all match. Across the ten rows in the table above, no two rows match on all four — and seven rows do not even name the judge. The comparison is not "noisy"; it is undefined. 2. Treating "LoCoMo" as one dataset. 7,512 questions is the paper; 1,986 is the release; 1,540 is what gets scored. A score cited against the paper's description is being cited against a corpus roughly five times larger than the one it ran on. 3. Reading a vendor's measurement of a competitor as that competitor's number. Measured size of this effect here: 9.15 points (Zep, 65.99 → 75.14). Discard cross-vendor rows or re-derive them. 4. Assuming a higher LoCoMo score means better memory. In Mem0's own 2025 table, *no memory system* scored 72.90 against Mem0-graph's 68.44. In Letta's 2025 run, *a filesystem* scored 74.0. A benchmark on which the null treatment wins does not rank the treatments. 5. Reporting a LoCoMo score without the retrieval depth. Measured: 8.90 points between top-200 and top-50 on one system with everything else held fixed; 10.70 points across Zep's five configurations. Depth belongs in the citation the way a model version does. 6. Assuming the published number reproduces from the published harness at its defaults. The defaults are gpt-4o / gpt-4o / top-200; the published runs are gpt-5 on azure. Both facts are documented in the same repository. 7. Using the paper's category proportions on the released file. Open-domain is 3.9% of the paper's 7,512 and 4.83% of the release's 1,986; single-hop is 36% versus 42.35%. Category weights changed when the subset was sampled for length. 8. Reading "overall" as covering the benchmark. Every vendor "overall" excludes the 446 adversarial questions, i.e. the entire test of whether the system declines to invent. A system optimised to always produce a confident five-word answer is optimised for exactly what is scored. 9. Averaging or splitting the difference between the three headline numbers. They are not three noisy estimates of one quantity. There is no quantity. 10. Trusting a scraped sub-score table. Mem0's research page renders its LoCoMo sub-scores in a layout where the labels break ("Open-domain Temporal / 82.3 / Temporal / 92.5"), and the tab reads "1,540 questions 5 categories" when 1,540 *is* the count with the fifth category removed. Read the repository, not the marketing page. 11. Assuming a published result file is one execution. Both Mem0 artifacts examined here carry merged_from_questions lists — 156 and 274 of 1,540 ids — so a tenth to a fifth of each file's verdicts came from elsewhere. If a vendor ships per-question artifacts, read the metadata block before the score; if it does not, you cannot check this at all. 12. Trusting a vendor's prose description of the dataset over the file. Zep's own post says LoCoMo has "35 conversation sessions each"; the released file has 19 to 32, mean 27.2. The file is 2.8 MB and free.

How to read a LoCoMo number in sixty seconds

For a buyer holding a slide. Six questions; if any three come back blank, the number is a label. This is the same discipline How to Verify an Agent-Payment Protocol Claim Before Citing It — the four checks this wiki runs on every page applies to payment-protocol claims.

1. How many questions? If the answer is not 1,540 (or the vendor cannot say), stop. 2. Which categories? If adversarial is included, ask how it was scored — 444 of its 446 items have no gold answer. 3. Which model wrote the answers? gpt-4o-mini and gpt-5 are not the same experiment. 4. Which model judged, under which rubric? Ask for the prompt file. If there is no prompt file, there is no metric. Seven of the ten rows above fail this one. 5. How many memories entered the prompt? Worth roughly nine points, measured above. 6. What is the date of the run — not of the blog post, and not of the paper it cites.

Then ask the only question that matters commercially, which LoCoMo does not answer for you: does the system handle a fact *changing*? Zep names this gap explicitly — *"The benchmark lacks questions designed to test knowledge updates—a critical function for agent memory where information changes over time"* — and points at LongMemEval instead. Mem0 points at BEAM and LongMemEval, both of which it now publishes alongside LoCoMo in the same README. Both vendors are pointing away from the benchmark their own headline number comes from, which tells you what the headline number is for.

Where this leaves our own wiki

The Binding Constraint Is Verified Context, Not the Model or the Rail (our prediction) proposes a falsifiable experiment: same model, with and without a verified external memory, five runs per arm on Vending-Bench 2, with the prediction that arm B beats arm A by at least 25%. This page is a signal against the design of that protocol, not against its thesis. If a single retrieval-depth setting is worth nine points on a memory benchmark, then "plus a verified external memory" is not one treatment — it is a family of treatments whose configuration space is larger than the effect being claimed. That protocol needs its retrieval depth, its consult-before-commit policy and its store contents pinned and published *before* the runs, and its per-run artifacts must record whether every run was executed under the pinned configuration — the merged_from_questions problem above is what happens when they do not. That correction belongs on the record and should propagate to that page.

Two sibling pages carry the market context this one sits in: what long-horizon agents actually fail at (Vending-Bench 2: The Top Model Broke Eleven Truces It Did Not Need to Break) and what the surrounding infrastructure explicitly does not promise (What None of These Protocols Currently Solve — the gaps common to AP2, AP4M, Visa, ACP and stablecoin rails).

Our interest, stated

Sinapsi sells a memory product for agents, in the same market as the vendors named here. Mem0, Zep, Letta and LangMem are competitors or adjacent sellers, and this page criticises the comparability of their published numbers.

Sinapsi has never run LoCoMo and has no LoCoMo score. We are not withholding a good one; we have not measured. If we publish one, it will carry all six fields from the checklist above, and it will still not be comparable to theirs — that is the entire point of the page, and it applies to us first. It applies to this page's own text too: an earlier draft of it put a paraphrase of Zep's words inside quotation marks and attributed Mem0's judge prompt to a source the paper never cites. Both were caught on re-fetch, both are corrected above, and both are recorded in the provenance rather than quietly fixed.

The three vendors here have each said a version of this in their own words: Mem0's harness README says *"Benchmark scores are not absolute numbers"* and names embedding model, LLM capability and retrieval depth as the reasons; Zep says *"A note on LoCoMo: we're not huge fans of this benchmark"* and *"Benchmarks tell you about benchmark performance"*; Letta says comparing memory tools is *"like comparing apples to oranges."* All three vendors agree the ranking does not exist. The ranking exists only in the comparison tables built by people who are not any of them. Including, if we are not careful, this one.

What would settle it

A single run, by a party selling none of these products, of all four systems (Mem0 platform, Zep, Letta filesystem, and a no-memory full-context control) over the same 1,540 questions, with one answerer model, one judge model, one published rubric, and each system integrated by its own vendor to its own satisfaction before scoring. Retrieval depth swept, not fixed. Adversarial questions scored separately as a refusal test rather than dropped. Every run executed end to end under one pinned configuration, with per-question verdicts published and no merged results.

Until that exists, the correct thing to do with three LoCoMo numbers side by side is not to rank them. It is to ask each vendor the six questions, and to weight the answers by how quickly they come back.

---

Supersession. This page contradicts no existing page in this wiki. It supplies a measurement caveat that The Binding Constraint Is Verified Context, Not the Model or the Rail (our prediction) should absorb, as noted above. If the reconciliation of Mem0's README and result-file figures becomes public, or the judge model behind the 2025 paper's J scores is named, or Zep publishes the answerer and judge behind 75.14, this page must be revised rather than reworded, and the affected rows in the score table retired.