Memory benchmark
Measuring LLM memory on LOCOMO and LongMemEval.
A memory layer is only as good as the answers it makes possible. We ran TemporalStore — the open-source engine behind MatrixArk — against a leading memory layer on two standard conversational-memory suites, with the reader model and judge held fixed. TemporalStore scored 88% overall to the baseline's 78%, and reached the same quality on far fewer tokens.
LOCOMO and LongMemEval both test whether a system can answer questions about a long, multi-session conversation — but they stress different muscles. LOCOMO leans on recent, in-conversation facts. LongMemEval deliberately buries the evidence across many earlier sessions, so the system has to recall the right moment from far back and ignore everything that has since gone stale.
How the run was set up
Both memory layers ingested identical transcripts, answered the same gold questions, and were graded by the same LLM judge. The reader model, embedding model, and retrieval budget were fixed for both. The only variable is the memory layer: how each one extracts events, tracks freshness, and assembles the context pack the reader ultimately sees.
- Same datasets: full LOCOMO and LongMemEval question sets.
- Same reader and same LLM judge scoring against gold answers.
- Same embedding model and retrieval budget on both sides.
- Reported quality is LLM-judged correctness; retrieval is hit@k on the supporting evidence.
Scores
Per-dataset quality, retrieval recall, and token efficiency.
| Metric | TemporalStore | Leading memory layer | Read as |
|---|---|---|---|
| Overall quality (Claude judge) | 88% | 78% | +10 pts |
| LongMemEval | 98% | 66% | +32 pts on long horizons |
| LOCOMO | 83% | 84% | tied on recent recall |
| Retrieval hit@k (LongMemEval) | 1.00 | 0.88 | +0.12 recall |
| Tokens at equal quality | Large savings | Baseline | fewer tokens, same answers |
Why the gap lives in LongMemEval
The tie on LOCOMO is the tell. When the answer is recent, almost any competent retriever finds it, and both systems land around 83–84%. LongMemEval removes that shortcut: the evidence is old, and the conversation has moved on. There, TemporalStore reaches 98% against 66%.
The retrieval line is the mechanism. TemporalStore surfaced the supporting evidence at hit@k 1.00 versus 0.88. Every point of missing recall is an answer the reader could never have gotten right, no matter how strong the model — the ceiling is set before the prompt is even built. Time-aware extraction and freshness are what keep the right old fact reachable while the stale ones stay out.
The token result
Equal quality on fewer tokens matters as much as the quality gap itself. A filter-first context pack does not hedge by stuffing near-duplicate memories into the window; it spends the budget on facts that are still valid at request time. In production that is the difference between a context layer that pays for itself and one that inflates every prompt.
Q (turn 61): "What laptop did I say I switched to?"
Baseline pack: 3 memories, incl. the ORIGINAL laptop (now stale)
-> reader answers with the superseded machine
TemporalStore pack: 1 memory, the LATEST switch event (freshness-ranked)
-> reader answers correctly, on fewer tokens