Ingestion
Context extraction and ingestion should be a serving path, not glue code.
Around every model call, MatrixArk accepts raw text, extracts typed context, writes indexable TemporalStore records, retrieves bounded context packs, and ingests feedback after the final answer. That is how context becomes living memory instead of one-time retrieval.
The loop
Ingestion happens around the model call.
Before the model responds, MatrixArk ingests user intent, active workspace, selected entity, retrieved sources, and tool traces. After the model responds, it ingests the final answer, accepted corrections, rejected suggestions, tool outcomes, and new commitments. This is where MatrixArk differs from a simple memory API: it does not only “save a memory.” It extracts typed facts, chooses a temporal scope, writes indexable serving records, marks summaries dirty, and stores enough audit state to replay the model input later.
Before-LLM hook
Query, hints, active entity, and tool traces arrive as raw context.
Extraction
Entities, event type, time, filters, and source refs are pulled from raw text.
TemporalStore ingest
Nodes, events, indexes, and dirty summary markers are written for serving.
Retrieve
Summary vectors plus bounded temporal reads build a token-budgeted context pack.
Model answer
The harness assembles the prompt from the pack plus local context and calls the model.
Feedback hook
Final answer, corrections, and commitments are written back into the timeline.
A concrete conversation
One approval thread becomes serving state.
The example is intentionally small, but it mirrors a real product loop: an approval arrives before the model call, a tool result adds spend evidence, retrieval builds a compact context pack, and feedback writes the accepted answer back into memory.
Extraction decides what the event means; TemporalStore decides how that meaning becomes fresh, indexed, replayable, and safe to serve in the next prompt. Retrieval never sees a dump of every source — only compact, current context.
Before LLM:
text = "Alice approved the batch request
for Project 1. Budget is $42,000."
source = before_llm_hook
hints = { project: project_1, actor: alice }
Tool result:
text = "Team spent $39,000 last month
on Project 1 capacity."
source = tool_result_hook
Retrieve:
query = "Can we approve another batch
for Project 1?"
max_prompt_tokens = 120
Feedback:
final_answer = "Approve only if finance
confirms remaining budget..."
source = feedbackThe data contracts
A simple ingest API compiles into strict serving records.
Callers send raw text plus hints. MatrixArk extracts the record and writes the TemporalStore-facing contracts so the hot path never filters arbitrary JSON.
| Contract | Role | TemporalStore meaning |
|---|---|---|
| IngestRequest | Raw text plus hints from a query, document, tool, or final answer. | The customer-facing ingest API stays simple. |
| ExtractedEvent | Node path, event type, event time, filters, actor, confidence, importance. | Extraction output becomes bounded serving metadata. |
| ContextNode | Stable node with hash, parent, path, kind, summary, compact attrs. | Hierarchical context is compiled into hash-addressed serving state. |
| ContextEvent | Timestamped event with text, type, source, status, validity, source ref. | Append-only temporal memory row for prompt-time reads. |
| IndexRef | Secondary reference for event type, status, scope, actor, and flags. | Declared filters become indexable lookup paths. |
| SummaryDirtyMarker | Signals that a node summary needs refresh after ingest. | The write path stays lightweight while summaries refresh async. |
| ContextPackAudit | Selected event ids, query plan, returned tokens, and decision notes. | Every context pack becomes replayable. |
Two paths
Extraction on write, extraction again for query planning.
IngestRequest
-> extract_event()
-> stable_hash64(tenant + node_path)
-> upsert_node(ContextNode)
-> write_event(ContextEvent)
-> write_index_ref(IndexRef)
-> mark_summary_dirty(marker)
-> upsert(vector for node summary)RetrieveRequest
-> plan_query() // intent, window, filters
-> embed(query)
-> vector_search(summaries) // candidate nodes
-> + exact scoped node hash
-> query_events(nodes, window,
filters, limit)
-> to_pack(max_prompt_tokens)
-> write_pack_audit(audit)This gives two knobs: vector summaries help find the right node, while TemporalStore enforces time, scope, filters, limits, and replay. LLM extraction is flexible; serving storage is not. The output must compile into stable keys and declared indexes before it enters the low-latency path — otherwise every request turns into arbitrary JSON filtering. Large raw artifacts live in an object store; TemporalStore keeps source refs and serving metadata.
Production requirements
Who owns what in the ingest path.
- Customers send raw text, hints, and source refs — never TemporalStore keys.
- MatrixArk owns extraction, schema validation, hash computation, index selection, and token budgeting.
- TemporalStore owns append, secondary-index writes, bounded time-window reads, dirty markers, replay, and freshness guardrails.
- Large raw artifacts live in object storage; TemporalStore keeps source refs and serving metadata.
- Vector search is optional and summary-oriented; TemporalStore remains the authoritative temporal event store.
- Feedback ingestion writes final answers, corrections, commitments, and rejected suggestions back into the timeline.
Keep reading