Recall@5 Is Not a Number: Measuring LongMemEval Retrieval at Every k
What this measures
Section titled “What this measures”LongMemEval asks a question against a haystack of roughly 48 chat sessions per user — most of them near-identical distractors — and names the sessions that actually contain the evidence.
This is a retrieval measurement. No answerer, no LLM judge, nothing to disagree about: either the gold sessions came back in the top k or they did not. That matters, because a judge swap alone moves end-to-end agent-memory scores by around six points, which is larger than most published gaps between systems.
A query counts as every gold only if all of its evidence sessions appear in the top k. Queries average 1.7 golds, so missing one scores zero for that query.
Results
Section titled “Results”| top k | at least one gold | every gold |
|---|---|---|
| 5 | 96.0% | 85.0% |
| 10 | 98.5% | 92.9% |
| 20 | 99.6% | 98.3% |
| 40 | 99.6% | 98.7% |
479 of 500 queries scored, 0 errors. k=40 is the shipped default.
Why k=40 is the honest row
Section titled “Why k=40 is the honest row”k=40 is not a generous setting chosen to flatter the result — it is the configuration YantrikDB ships, and it is load-bearing. A paired 400-query run on BEAM, with the same reader and the same judge on both sides, measured k=40 → k=20 costing 3.4 rubric points across 8 of 10 categories. Retrieval depth is doing real work; reporting a narrower k would describe a system nobody runs.
Why the whole curve is published
Section titled “Why the whole curve is published”Two reasons, and the second is the interesting one.
First: we got this wrong ourselves. The initial measurement used k=5, because that is where a competing system’s headline sits. It returned 72.9% and read like a serious deficiency. It was measuring someone else’s operating point.
Second: “recall@5” is not well defined without naming the pool. On identical queries, with an identical metric, in the same run:
- retrieve 40, keep the best 5 documents → 85.0%
- retrieve 5 directly → 72.9%
Both are honestly describable as “recall@5”. The difference is that retrieval reranking depends on how large a pool it was given, and almost nobody states the pool. A single published number hides that; a curve cannot.
What is not claimed here
Section titled “What is not claimed here”No head-to-head result. Other systems publish at different k, with different scorers, on different splits. Lifting a figure out of its configuration and setting it beside ours is precisely the error that produced our own 72.9%.
What is offered instead is a stated method and a reproducible command:
python lme_recall_multik.py 500 24One retrieval per query at k=40, every prefix scored, parallel across queries — 500 queries in 5.2 minutes. Anyone can run it against their own system and put the curves side by side.