Runtime reuse
TemporalStore makes prefix and KV-cache reuse context-aware.
LMCache-style systems and remote KV-cache layers reuse model-runtime state — they know tokens, prefixes, and cache placement. TemporalStore knows application context: what changed, what expired, what the user can access, which sources are valid, and which memories must be refreshed before a prefix is reused. Together they save compute and correctness.
The boundary matters. TemporalStore is not a transformer KV-cache engine. It does not own GPU attention blocks, token-position layout, paged attention, prefix matching inside the runtime, or vLLM/SGLang scheduler internals. Its job is one level up: decide which context sections are safe to reuse, which ones invalidate a prefix, and which metadata to pass to a remote cache layer.
That boundary is also the commercial one. Runtime cache vendors can cut prefill cost and latency, but an enterprise agent still needs a source of truth for whether reused context is correct, fresh, permissioned, and auditable. TemporalStore supplies that control plane.
LMCache -> runtime prefix reuse, KV-cache block movement, cache hits
TemporalStore -> time-aware context, source freshness, memory validity,
prompt-section stability, replay, invalidation reasons
MatrixDB (ent) -> hot cache metadata: keys, TTL state, retrieval lists at scale
MatrixKV (ent) -> low-volume truth: permissions, versions, leases, approvalsThe same tokens are not the same context
A support copilot reuses stable product instructions, but the customer timeline can change every minute.
S0_system_policy cache_policy = stable source_version = policy:v42
S1_tool_contract cache_policy = stable source_version = tools:v9
S2_customer_context cache_policy = time_sensitive
context_pack_id = ctxpack_acme_20260616_1015
valid_until = 2026-06-16T10:20:00Z
invalidates_on = [ticket_update, refund_approval, permission_change]
S3_latest_timeline cache_policy = volatile max_age_seconds = 30
LMCache can reuse the runtime state for the stable sections. TemporalStore decides whether
S2_customer_context and S3_latest_timeline must be regenerated
— and records why. A generic remote KV-cache sees similar tokens; only the
context layer knows the underlying facts moved.
How a request flows
The context layer sits above the cache and the runtime, deciding what is safe to reuse.
App asks for a pack → TemporalStore checks freshness and source versions → MatrixArk labels sections stable / time-sensitive / volatile → LMCache reuses only the safe prefix → the answer writes feedback and invalidation events back.
Better than a generic remote KV-cache alone
The hard part is not reuse. It is deciding safety.
| Generic remote KV-cache | TemporalStore + LMCache policy |
|---|---|
| Optimizes token / runtime reuse. | Reuses only after checking freshness, permissions, source versions, and open commitments. |
| Cache key is often token prefix, model, tenant, or request shape. | Cache key includes context-pack id, source versions, valid-time window, and section stability. |
| Invalidation is coarse: TTL, manual delete, prefix miss. | Invalidation is semantic: ticket changed, approval expired, document version changed, memory superseded. |
| Hard to explain why a prefix was reused. | Replay shows selected sections, blocked sections, freshness checks, and the decision note. |
| Risks mixing durable app memory with runtime mechanics. | Runtime cache stays runtime cache; TemporalStore stays durable application memory and policy state. |
CachePolicyEvent {
tenant_id: "acme", session_id: "support_session_77",
context_pack_id: "ctxpack_8f91", model: "llm-prod",
prompt_sections: [
{ id: "S0_system_policy", reuse: "allow", source_version: "policy:v42" },
{ id: "S1_tool_contract", reuse: "allow", source_version: "tools:v9" },
{ id: "S2_customer_context", reuse: "deny", reason: "ticket_update_after_pack" },
{ id: "S3_latest_timeline", reuse: "deny", reason: "volatile_window_expired" }
],
lmcache_hint: {
reusable_prefix_until_section: "S1_tool_contract",
cache_namespace: "tenant:acme:model:llm-prod"
}
}Where it helps most
- Support copilots that reuse product instructions but refresh customer timelines and promises.
- Legal copilots that reuse matter templates but invalidate source sections when a document version changes.
- Security agents that reuse playbook prefixes but refresh incident timelines and containment state.
- Enterprise RAG where stable policy text can be cached, but permissions and source freshness must be checked per request.
The short version: a remote KV-cache saves compute when prefixes repeat; TemporalStore saves correctness when context changes. LMCache answers "can this prefix be reused technically?" TemporalStore helps answer "should this context be reused for this user, at this time, with these sources and permissions?" The engine behind it is open source at temporalstore.ai.