Whose Memory, Whose Model? Isolating Retrieval Quality on BEAM

The measurement problem

An end-to-end agent-memory benchmark score confounds at least three things:

1. the memory system’s retrieval quality,
2. whatever processing happens at ingest,
3. the model that reads the retrieved context and writes the answer.

This is not a hypothetical concern. The Agent Memory Benchmark’s own documentation notes that generation setup matters and that small changes in prompts or models move accuracy substantially. We measured it directly: swapping only the answerer, holding retrieval identical, moved our score by ~9 points.

So when two memory systems publish different end-to-end numbers under different answerers, the difference is not attributable to either system’s memory. It is a property of the whole pipeline.

This paper is about how large that effect turns out to be.

Removing the confound

Hindsight publishes per-query result files containing the exact context string injected for each query, for two configurations. All 400 BEAM-100K query IDs match ours.

That makes a controlled comparison possible:

A   yantrikdb context   (rag)          ─┐
R   hindsight context   (rag)          ─┼─→ same answerer → same judge → score
E   hindsight context   (single-query) ─┘

Answerer and judge are fixed at deepseek-v4-flash:0731, temperature 0, for all three. The only variable is which memory system produced the context.

A and R are mode-matched — both rag-mode runs — so A-vs-R is the real comparison. E is their weaker published configuration.

The headline result

Hindsight’s published BEAM-100K rag score is 0.862, read by gemini-3.1-pro-preview. Ours is 0.611, read by deepseek-v4-flash. On a leaderboard that is a decisive 25-point gap.

Read by the same model, the two systems’ retrieved contexts score:

condition	mode	binary	rubric	context tokens
YantrikDB	rag	72.2%	0.607	13,673
Hindsight	rag	71.5%	0.592	23,689
Hindsight	single-query	65.0%	0.563	17,655

A difference of 0.015 — not statistically significant by any test we applied. The 25-point published gap was almost entirely the reading model, not the memory system.

Testing it properly

Queries are nested inside 20 conversations, so they are not independent observations. All significance is computed at the conversation level, with sign-flip tests enumerated exactly over all 2²⁰ assignments.

comparison	binary	rubric	95% CI (rubric)	verdict
ours vs their rag	+3, p = 0.845	+0.015, p = 0.438	[−0.021, +0.052]	equivalent
ours vs their single-query	+29, p = 0.016	+0.044, p = 0.025	[+0.009, +0.077]	ours better
their rag vs their single-query	+26, p = 0.006	+0.029, p = 0.089	[−0.001, +0.061]	their rag better

Every robustness check on the primary comparison agrees with equivalence: per-conversation direction splits 7 to us, 5 tied, 8 to them; the confidence interval comfortably contains zero; and leave-one-conversation-out flips the sign, so the direction is not even stable.

Note the third row. Their own configuration choice moves their score more than the gap between their system and ours — a reminder of how much of a published number is setup rather than substance.

What parity costs

The comparison above is retrieval quality. The other half is what that quality costs to produce — same split, same statistic, from each system’s own published result files.

	ingest	retrieval (mean)	context	LLM at ingest
YantrikDB	241 s	80 ms	13,673 tok	none
Hindsight (rag)	404 s	2,565 ms	23,689 tok	fact extraction
Hindsight (single-query)	404 s	6,379 ms	17,655 tok	fact extraction

Our ingest is chunk → embed → index with a bundled 7 MB static embedder running in-process. No model server, no network, no API key. The only LLM anywhere in our pipeline is the benchmark’s own answerer, identical for every system it evaluates.

Statistically indistinguishable answer quality, on 42% fewer context tokens, at ~32× faster retrieval, with zero LLM calls at ingest.

That — not a win on the rubric — is the result.

A hypothesis we are withdrawing

An earlier version of this analysis argued that ingest-time LLM extraction discards evidence that later questions need. Against Hindsight’s single-query configuration, our advantages clustered neatly in exactly the categories that theory predicts: information extraction, temporal reasoning, summarization, multi-session reasoning.

Against their rag configuration, the pattern largely dissolves. Summarization flips from +0.099 to −0.054. Temporal reasoning falls from +0.106 to +0.044. They lead on event ordering, preference following and instruction following.

A category pattern observed against one competitor configuration is a property of that configuration, not evidence for a general mechanism. We are withdrawing the claim as unsupported by this experiment, and testing it directly instead — raw versus LLM-compressed context on the same substrate, where nothing but the compression differs.

We still hold the design bet: preserve evidence now, interpret it against the question later. This experiment does not prove it.

What this does not show

Stating the limits precisely matters more than the headline.

• We are not claiming our memory is better than theirs. On the mode-matched comparison the honest reading is equivalence, and the interval spans −0.021 to +0.052 — a small advantage in either direction remains possible.
• This is not a refutation of their published 0.862. We show only that their retrieved context, read by a mid-tier model, does not outperform ours. We did not run our context through their answerer, so the 2×2 is incomplete.
• Length is not controlled. Longer context can hurt through dilution, so we cannot rule out that their larger context cost them something a budget-matched test would recover.
• The contexts are not information-matched. Measured after publication: their rag contexts carry a median of 117 dates each; ours carry 5. The engine stores event time for every chunk and our provider discards it, so part of the 42% token saving is the cost of not carrying dates — a difference in what each memory layer emits, not a free efficiency. A dated, chronologically-ordered variant is being measured at full scale, and this page will be updated with the result either way.
• Their context was built for a frontier reader and may be disadvantaged by a mid-tier one in ways we have not characterised. This cuts against us, and is the most likely way the equivalence finding is wrong.
• Our weakest categories are real. Event ordering (0.298) and temporal reasoning (0.425) are where their rag configuration matches or beats us. Sequence reconstruction is our clearest improvement target.

A correction

Engine version, 2026-08-13. This paper briefly stated the engine as 0.14.0. That was wrong. The benchmark’s virtualenv holds yantrikdb-0.13.4, installed before the run — the version was assumed from the release timeline rather than read off the environment that produced the numbers. Every result on this page came from that same 0.13.4 build, so the comparisons remain internally consistent; only the label was wrong. A 0.14.1 run is pending and will be reported separately rather than folded into these figures.

The run files record no engine version, which is why the error was invisible from the outputs.

Reproducing

Everything needed ships in the public benchmark repository — including Hindsight’s per-query contexts for both configurations, so conditions R and E require no access to their system.

git clone https://github.com/vectorize-io/agent-memory-benchmark
cd agent-memory-benchmark && uv sync
python /path/to/yantrikdb/benchmarks/amb/install.py .

YDB_BENCH_TURN_AWARE=1 YDB_BENCH_TOPK=40 \
  uv run amb run --dataset beam --split 100k --memory yantrikdb -n ydb-final

uv run python frozen_context_eval.py \
  --contexts outputs/beam/ydb-final/rag/100k.json    --label A-yantrikdb-ctx
uv run python frozen_context_eval.py \
  --contexts outputs/beam/hindsight/rag/100k.json.gz --label R-hindsight-rag-ctx

uv run python frozen_stats.py A-yantrikdb-ctx R-hindsight-rag-ctx

Full method, statistics and caveats: docs/benchmarks/beam-frozen-context.md.

Next

Parity on 42% fewer tokens is a claim about a curve, and we have measured one point on it. The stronger form asks each engine for 2K, 4K, 8K, 12K and 16K and plots accuracy against tokens spent — which removes the length objection entirely and answers the question a memory system should actually be judged on: how much answer quality does it deliver per unit of context budget?

That work is in progress, alongside a matched-answerer run to complete the 2×2.