Context memory benchmarks
Measured on the memory benchmarks the field actually uses.
MatrixArk's context engine is built on TemporalStore. We benchmark it head-to-head against an
OpenViking-style direct-source-retrieval baseline on LOCOMO and
LongMemEval using one shared open-source stack — the same
qwen2.5:7b reader and MiniLM embeddings for both, scored by a Claude LLM judge —
so the comparison is about the memory layer, not the model around it.
Head-to-head
TemporalStore ties OpenViking on short chats, and decisively beats it on long-horizon memory.
Head-to-head against an OpenViking-style direct-source-retrieval baseline on the identical OSS stack —
same qwen2.5:7b reader, same embeddings, same retrieval budget, Claude-judged. Only the memory layer differs.
| Benchmark | Metric | TemporalStore | OpenViking (same 7B model) |
|---|---|---|---|
| Overall | Answer accuracy (Claude judge) | 88% | 78% |
| LongMemEval | Answer accuracy | 98% | 66% |
| LOCOMO (tie) | Answer accuracy | 83% | 84% |
| LongMemEval | Retrieval hit@k | 1.00 | 0.88 |
| LongMemEval | Prompt tokens at equal quality | 97% fewer vs replaying full history | |
The gap is widest on LongMemEval — long-horizon, multi-session questions — exactly where time-valid, replayable memory with stale-context blocking pays off. On shorter LOCOMO conversations the two are even.
The landscape
How the context approaches differ.
| System | Approach | Temporal & replay | One service |
|---|---|---|---|
| MatrixArk / TemporalStore | Append-structured temporal store; Context model | Native event-time + replay | Yes — memory, retrieval, counters, replay |
| Leading hierarchical memory | Event/entity memory, L0/L1/L2 layers | Temporal layers | Memory-focused |
| Mem0 | Add/search API, scoped memories | Limited | Needs a vector store |
| Zep / Graphiti | Bi-temporal knowledge graph | Bi-temporal graph | Graph + retrieval |
| MemOS / Memori | OS/filesystem-style memory | File-like | Memory-focused |
Method
Fair by construction.
Both systems run behind the same open-source harness: identical qwen2.5:7b reader and
all-MiniLM-L6-v2 embeddings, the same LOCOMO (multi-session conversation QA) and LongMemEval
(long-horizon memory) datasets, and a Claude LLM judge applied uniformly. Only the memory layer differs,
so the results isolate the contribution of the store — ingestion, retrieval, freshness, and
context-pack assembly.
The enterprise story adds the parts a benchmark cannot show on a laptop: shared-storage throughput under concurrency, five-nines availability, and replayable audit — all on the same engine.
Tokens & quality
A small managed pack beats replaying the whole history — on tokens and on answers.
Three arms over the same questions and the same reader/judge model: replay the full local context, or retrieve a token-budgeted ContextPack. Graded against query-relevant ground truth.
| Arm | Tokens the model works from | Answer quality (0–10 judge) | Queries won or tied |
|---|---|---|---|
| Full local replay | full corpus (~1.7M) | 6.59 | — |
| TemporalStore managed pack | ~1.3k | 8.30 | 6 / 6 |
The margin is largest exactly where the answer lives in older history — the case recency-truncated replay cannot see. So the managed pack sends a tiny fraction of the tokens and answers better. Retrieval hit@k is 0.995–1.000 on LOCOMO and LongMemEval_s (7B).
Serving & scale
Fast at request time, correct under eviction, quick to recover.
Context serving is bounded temporal reads over a memory-first, multi-layer store — so latency stays low as the corpus grows, and cold state is promoted back into memory on read.
| What we measure | Result |
|---|---|
| ContextPack retrieval, p95 (1.2k-token budget) | ~17 ms |
| Retrieval hit-rate under an adaptive 1.2k→4k budget | 86% → 96% |
| Multi-user eviction + cold-read promotion correctness | 256k reads, 0 mismatches (81 users, 187k writes) |
| Warm reload after restart | ~168 ms |
The eviction harness deliberately shrinks the memory budget to force memory→SSD→shared-storage eviction, then reads cold to force promotion — every value verified against ground truth, so a green run is also a correctness proof.
Read more