Context benchmark
How much better is a real context layer?
We put TemporalStore, the open-source engine behind MatrixArk, head to head against a leading memory layer on a shared open-source memory harness with an LLM judge. Same questions, same reader model, same retrieval budget — the only variable is how each system decides what the model gets to remember. TemporalStore led overall 88% vs 78%, and won long-horizon memory outright.
Context management is easy to demo and hard to measure. Any system can return something relevant for a single-turn question. The real test is a long conversation where the answer depends on a fact stated forty turns ago, a preference that has since changed, or a commitment that has already been fulfilled. That is where a memory layer either earns its place or quietly injects stale, duplicated, or wrong-time context into the prompt.
What we actually compared
This is not a benchmark of TemporalStore against itself with favorable settings. Both systems ran on the same public harness, ingested the same transcripts, answered the same questions, and were graded by the same LLM judge against the same gold answers. The reader model and the embedding model were held fixed. What differs is the memory layer: extraction, freshness, and what each system chooses to put in front of the model at answer time.
- Overall quality — LLM-judged answer correctness across the full question set.
- LongMemEval — long-horizon recall where the evidence is buried across many sessions.
- LOCOMO — multi-session conversational memory with evolving facts.
- Retrieval hit@k — whether the supporting evidence was even surfaced to the reader.
- Token efficiency — tokens spent to reach that quality.
The shared harness
One reader, one judge, one retrieval budget. Swap only the memory layer.
Because everything downstream of the memory layer is identical, differences in score come from what each system remembered and chose to surface — not from a stronger reader.
Results
Three-way view: overall quality, per-dataset breakdown, and whether the evidence was retrieved at all.
| Metric | TemporalStore | Leading memory layer | Result |
|---|---|---|---|
| Overall Claude-judged quality | 88% | 78% | +10 pts overall |
| LongMemEval (long-horizon recall) | 98% | 66% | +32 pts on long horizons |
| LOCOMO (multi-session memory) | 83% | 84% | tied |
| Retrieval hit@k (LongMemEval) | 1.00 | 0.88 | evidence surfaced far more often |
| Tokens to reach equal quality | Large savings | Baseline | equal quality, fewer tokens |
Reading the numbers
The two systems tie on LOCOMO, where the relevant fact is usually recent and easy to reach. The gap opens on LongMemEval — 98% vs 66% — precisely the long-horizon setting where time-aware extraction and freshness matter. The retrieval line explains why: TemporalStore surfaced the supporting evidence 1.00 of the time versus 0.88. A reader cannot answer from context it never sees, so recall at the retrieval stage caps everything after it.
The token result is the one that changes budgets. Reaching the same answer quality on fewer tokens means a filter-first context pack is not padding the prompt with near-duplicate memories — it spends the window on facts that are still valid now.