TemporalStore.AI | Infrastructure deep dive

Hot path

Route once, execute at the shard, serve hot temporal state from memory, and refill colder blocks from PMem, SSD, or shared storage only when needed.

Durability

Every mutation is represented in the WAL; snapshots, page indexes, and replay bound recovery while replicas stay queryable.

Scale model

Open source runs local disk or MatrixRaft. Enterprise shared storage separates compute from durable state for elastic datanode capacity.

Architecture

All modules: serving core, dependencies, tiering, and storage backends.

Clients talk to a stateless proxy that routes by namespace and table. A metaserver owns placement; datanodes run the temporal models, back their state with an append-structured storage engine, and lean on two open-source libraries — MatrixCache for tiered caching and MatrixRaft for replicated durability. Hot context data is served memory-first, then automatically moves through PMem, SSD, and a shared store as it cools; the shared store is pluggable up to the enterprise backends.

Layer	Hot-path responsibility	Failure / scale boundary
Proxy	Accept SDK calls, batch writes, enforce backpressure, and route by namespace/table/bucket from a cached shard map.	Stateless; add replicas for connection fan-in and retry across proxy nodes.
Metaserver	Own placement, epochs, leases, shard ranges, and membership; proxies refresh maps only when placement changes.	Control plane; off the per-read hot path so metadata churn does not tax ordinary reads.
Datanode	Append WAL, apply model executors, maintain memory indexes, serve windows/tails/aggregates, and emit snapshots.	Shard owner; scale by adding shards/datanodes, or by disaggregating storage below the compute tier.
Storage engine	Pack values into append-only blocks and bands, update page/block/object indexes, checkpoint snapshots, and replay WAL.	No LSM compaction loop; garbage collection reclaims sparse regions off the hot write path.
Cache/tiering	Keep hot entity state in memory, demote warm blocks to PMem and SSD, and refill from PMem, SSD, or shared store on misses.	Admission protects hot context from cold replay scans; shared storage supports elastic read hydration while the serving path keeps extreme latency for hot reads.

Clients

AI agents & coding systemsSDK writes · online reads · stream consumers · replay jobs

↓ context ingest & retrieval ↓

TemporalStore serving core — open source

Proxy / gatewaynamespace + table routing, batching, backpressure

Metaservershards, bucket ranges, placement & membership

Datanodestemporal model executors, query planner, WAL

Dependency libraries — open source

MatrixCacheopen-source multi-layer cache — L1 memory + L2 SSD, admission & eviction

MatrixRaftRust Raft protocol for consensus, replicated WAL, and high availability

↓ context data moves hot → cold automatically ↓

Memory-first context data tiering

Memoryhot context nodes, live index, extreme serving latency

PMemnear-memory warm state, fast promotion path

SSDwarm blocks, bands, snapshot & page index

Shared storecold blocks, replay history, durable slabs

↓ pluggable storage backend ↓

Enterprise storage backends Enterprise only

MatrixObjectshared object store · compute/storage disaggregation MatrixKVtransactional metadata, coordination & backfill MatrixDBeventually consistent high-throughput KV

One request path, top to bottom: route → place → execute over cached, replicated, tiered temporal state. MatrixObject, MatrixKV, and MatrixDB are enterprise backends; local disk and MatrixRaft are open source.

Node layout

The actual cluster, and what lives inside one datanode.

Zooming in on the topology: a stateless proxy tier fields client connections and caches the routing map; a single logical metaserver owns placement and membership as a control plane off the hot path; and a fleet of datanodes each own a set of shards. Inside every datanode is a memory-to-PMem-to-SSD-to-durable hierarchy that keeps the hot working set closest to compute while nothing ever leaves the queryable surface.

Proxy tier — stateless

Proxy / gateway (N replicas)caches the shard map · routes by namespace + table + bucket · batching & backpressure · holds no durable state

↓ routing lookups (cached, refreshed on placement change) ↓

Control plane — off the hot path

Metaservershard map · bucket ranges & placement · membership · leases & epochs — consulted only when the map changes, never per read

↓ owns which datanode serves which shard ↓

Data plane — datanodes own shards

Datanode Ashards 0–3 · model executors + WAL

Datanode Bshards 4–7 · model executors + WAL

Datanode Cshards 8–11 · model executors + WAL

↓ inside one datanode: memory hierarchy & eviction ↓

Datanode internals — hot → cold with cache fill on miss

Memory (L1)hot entity working set · recent events · live page index — served with no I/O

PMemnear-memory tier for warm context and fast refill

SSD (MatrixCache L2)warm blocks & bands evicted from RAM · admission on read, LRU eviction under pressure

Shared storecold blocks/bands & durable slabs · source of truth for cache fill

Least-recently-used entity state is evicted from memory down to PMem, SSD, and eventually to the durable store — but it never leaves the queryable surface. A read that misses in RAM triggers a cache fill from PMem, L2 SSD, or the shared store, so cold history stays reachable while the hot path keeps extreme serving latency. Admission keeps a burst of cold replay traffic from flushing the hot set.

Request & replication flow

How a write moves through the system — and gets replicated.

Putting the pieces together: a single write travels client → proxy → the shard's primary datanode → WAL → replicas, and a read follows the same route to whichever datanode owns the shard.

1 · Route the request

Client / SDKkey + event

Proxyhash key → bucket → shard, from the cached shard map

Metaservershard → datanode placement & leases; consulted only on map change

↓ to the shard’s primary datanode ↓

2 · Persist the WAL first — before apply or replicate

Append to the WAL → fsyncevery write is made durable in the write-ahead log before it is applied or replicated (sync fsyncs here; async batches it in the background)

↓ only after the record is persisted ↓

3 · Apply to the in-memory model & index

Update in-memory modelthe executor mutates the hot entity state

Update in-memory indexbucket / page index maps key + time → where the value lives

↓ replicate the committed WAL record — by deployment mode ↓

4 · Replicate the WAL

Raft modeship the WAL to secondary datanodes; each applies it → a full replica; commit after quorum ack

Shared-store modewrite one durable copy to the shared store; stateless secondaries read it directly Ent

Local modeappend to the local shared file — one durable copy, no network replication

↓ ack the client — durable, and quorum-committed in Raft ↓

5 · Read — in-memory index lookup first

Client → Proxy → datanoderouted to the shard owner

In-memory index lookuphit → serve straight from memory, no I/O

↓ miss? the data was evicted or is cold ↓

6 · Cache fill on a miss — storage → cache → memory

SSD cache (MatrixCache L2)read the block here first

Shared store / local fileif not cached, read the durable copy

Promote into memoryadmit into cache → load into RAM → serve — cold data stays reachable

↓ and, under memory pressure, the reverse ↓

7 · Eviction — when memory fills, cold data ages down

Memory → SSD → shared storewhen RAM is full, least-recently-used entity state is evicted downward — demoted, never deleted. It stays queryable and is pulled back on the next read (step 6)

The WAL is the spine: every write is persisted to the WAL before it is applied or replicated, replicas stay consistent by applying it, and cold reads are reconstructed by replaying it. Reads hit the in-memory index first; a miss pulls the block up from SSD or the shared store into cache and memory; and when memory fills, cold entities are evicted downward but never leave the queryable surface. The two replication modes below show how step 4 differs between Raft and shared storage.

Deep dive

How each layer works.

Serving core

Metaserver + datanodes

The metaserver owns namespace, table, shard, bucket-range, placement, membership, lease, and epoch metadata. The proxy caches that routing map, so the control plane is not on the hot read path. Datanodes run model-aware executors over entity-local state, which lets a read ask for a window, filter, distinct set, or tail directly at the shard.

Storage engine

Append-structured blocks

Values are packed into blocks, grouped into append-only bands, and written into slab files. New state is appended instead of updated in place; obsolete state is reclaimed later by background garbage collection. Page, block, and object indexes keep reads direct even when entity history grows long.

Cache

MatrixCache tiering

Memory holds hot context nodes, recent events, and the live page index. PMem absorbs near-hot state. SSD holds warm blocks and bands. Admission and eviction protect the hot set so cold replay scans do not flatten serving latency under high concurrency.

Recovery

MatrixRaft + WAL replay

Durability starts with the WAL. Snapshots and checkpoints bound recovery; replicas load the last durable base and replay WAL after the checkpoint. MatrixRaft gives HA without shared storage, while leases and epochs fence stale writers after failover.

Tier movement

Memory to PMem to SSD to shared store

Context data ages automatically. Hot state stays closest to compute; colder blocks move down the hierarchy but remain queryable. Reads pull from the highest available tier, promote data back into cache, and keep long tails cheap and replayable.

Product boundary

Model-aware serving, not opaque KV

TemporalStore moves temporal math into the engine. Callers do not fetch whole histories and reprocess them in application code; the serving path understands context, long sequences, aggregates, control state, and retained windows.

Storage durability

Synchronous vs asynchronous storage — zero-loss or max throughput.

Every write is appended to the WAL, but you choose when that append is forced to disk. That choice is a single config flag, async_storage — the main durability-vs-throughput dial in TemporalStore.

Sync storage — async_storage = false (default)

Write arrivesappend the record to the WAL

↓ fsync this record to disk ↓

Durable on diskthe WAL record is fsync’d before anything is returned

↓ then apply & ack ↓

Ack the clientevery acknowledged write survives a crash

The record is on disk before the client hears “ok.” Zero data loss on power failure; the fsync sits on the write’s critical path.

Async storage — async_storage = true

Write arrivesappend to the WAL + apply in memory

↓ ack right away ↓

Ack the clientno fsync on the hot path

Background flushcoalesces many records into one fsync

The ack does not wait for fsync; a background flush batches many records into a single sync. Highest write throughput; a crash can lose only the last, not-yet-flushed window.

	Sync — `async_storage=false`	Async — `async_storage=true`
Durability	Every acked write is on disk — zero-loss	Last unflushed window can be lost on crash
Throughput	Bounded by per-write fsync	High — fsyncs are batched & coalesced
Write latency	Includes the fsync	Acked before the fsync
Best for	Counters and state that must never lose a write	High-volume context & feature-event ingest

How to use it

Sync is the default — zero-loss durability with no configuration. Turn on async when ingest throughput matters more than the last few milliseconds of writes: high-volume context and feature streams are the common case, and the coalesced control-state counter path is effective only with async on. Replication is orthogonal — both modes still append to the WAL that MatrixRaft or the shared store replicates.

Choosing the storage mode

# default is synchronous (per-write fsync) -- nothing to set for zero-loss durability

# opt into asynchronous (batched fsync) for maximum ingest throughput
export MATRIXARK_RUST_PROXY_ASYNC_STORAGE=true

# or per shard at runtime via the control plane
{ "kind": "set_config", "config": { "async_storage": true } }

WAL replication

How the write-ahead log moves between nodes.

A write for a shard lands on that shard's primary datanode, which appends it to the WAL and hands the record to MatrixRaft. Consensus replicates it to the secondary replicas; only once a quorum acknowledges does the write become commit-visible. Secondaries apply the committed WAL to rebuild their own queryable state, so any of them can serve reads or step up on failover.

Primary datanode — owns the shard

Primary datanodewrite lands here → appended to the WAL · not yet commit-visible

↓ replicate WAL record ↓

Consensus — MatrixRaft

MatrixRaftreplicates the WAL record · waits for quorum ack before commit-visible · lease/epoch fencing

↓ committed WAL fans out ↓

Secondary replicas — apply & stay hot

Secondary Aapplies committed WAL → rebuilds queryable state · failover candidate

Secondary Bapplies committed WAL → rebuilds queryable state · failover candidate

Quorum-acknowledged commit: a write is durable across the replica set before it is visible to readers. On primary loss, a replica holding the latest committed WAL is promoted; lease and epoch fencing reject a stale ex-primary that comes back, so it cannot corrupt state after failover.

1

Write

Request lands on the shard's primary datanode and is appended to the WAL.

2

Replicate

MatrixRaft ships the WAL record to the secondary replicas.

3

Quorum ack

A majority acknowledges; only then does the write become commit-visible.

4

Apply

Secondaries apply the committed WAL to rebuild queryable state, ready to serve or be promoted.

Shared-storage deployments read WAL directly

When the shared store is MatrixObject Enterprise, replication does not have to flow node-to-node. Because committed WAL and blocks are already durable in shared storage, secondaries can read them directly from the shared store instead of pulling from the primary. Compute and storage stay disaggregated: a replica catching up, or a freshly added datanode, hydrates from the object store rather than draining the primary's bandwidth, and the same committed-WAL contract still governs what is visible.

Two replication modes

Raft replication is not compute/storage disaggregation — shared storage is.

Both modes keep your context durable and highly available, but they get there in fundamentally different ways. It matters which one you pick, because only one lets compute and storage scale independently.

Raft replication — co-located

Datanode (primary) + local storagefull copy of the shard: WAL, blocks, bands

↓ MatrixRaft quorum ↓

Replica + local storagefull copy

Every replica is a whole datanode that stores its own complete copy. Compute and storage travel together — this is replication, not disaggregation.

Shared-store — disaggregated Enterprise

Datanodestateless compute

↓ read/write one durable copy ↓

MatrixObject — shared storeone durable, internally-replicated copy of blocks, bands, wal

Datanodes hold no durable data; the shared store owns durability. Compute and storage scale on independent axes — disaggregation.

	Raft replication (open source)	Shared-store replication Enterprise
Durable copies	One full copy per replica (N copies)	One copy in the shared store (it handles its own redundancy)
Compute & storage	Co-located on each node	Disaggregated — separate, independent tiers
Scale by	Adding full replicas (each stores everything)	Adding stateless datanodes (no data copied)
Recover a node by	Promoting a replica; new nodes stream a full copy	Reopening blocks from the shared store — no state transfer
Best for	HA without external storage; fixed-size clusters	Large scale, elastic capacity, concurrent read/write at five-nines
Disaggregation	No	Yes

Why no vector DB, why not RocksDB

Append-structured, model-aware, and temporal by design.

Why no separate vector database

Filter-first, not vector-first. Agent-memory retrieval is filter-first over a temporal tree. Scope hashes, time windows, and typed filters narrow the candidate set before any similarity math runs, which inverts the usual vector-database pipeline where an approximate-nearest-neighbor (ANN) index is the primary lookup and metadata filtering is a post-filter applied to whatever the index happened to return. That ordering matters: the questions agents actually ask — “what happened in this session,” “which tool calls in the last hour,” “what did I decide and when” — are answered exactly by scope and time, not by embedding proximity. Because the candidate set is already small and correct by construction, similarity ranking, when you want it, runs over tens of rows rather than the whole corpus.

Time-validity and supersession are things ANN cannot express. A fact carries a valid-time window, and a later write can supersede an earlier one, so retrieval returns what was true at the asked-about time and drops what has been overwritten. A pure ANN index has no notion of this — it returns whatever is geometrically nearest, which routinely means stale, superseded, or out-of-scope chunks that are merely “near” in embedding space. That is the failure mode behind wrong answers on knowledge-update and temporal-reasoning questions: the nearest chunk is often the outdated one, and a similarity index has no way to know it was replaced.

What skipping the vector DB actually saves. No standalone vector service to deploy, shard, back up, and pay for; no re-embedding and index-rebuild churn every time memory changes; no dual-write consistency problem to keep the store and the index in sync; and no cross-session leakage from chunks that are “near” in embedding space but belong to another session or another point in time. Semantic vector recall stays available as an optional add-on layered on top of the temporal filter — the right tool for genuinely fuzzy “find me something like this” recall — but it is not the primary index, so you run it only where it earns its cost, over an already-narrowed candidate set.

Why not RocksDB — much lower write amplification, not zero

What an LSM tree costs. RocksDB-style LSM trees rewrite SSTables during compaction: each logical write is physically re-written several times as data is merged down the levels — a write-amplification factor of many× — and background compaction periodically competes with foreground traffic for I/O, producing compaction stalls that surface as tail-latency spikes. High-write temporal workloads — event streams, tool traces, velocity counters, session logs — are exactly the access pattern that punishes that design.

What append-structured storage does instead. Values are packed into blocks, blocks are appended into bands, and bands are written into slab files; live data is never rewritten in place, and obsolete data is reclaimed by background garbage collection that drops whole regions once they fall below a liveness ratio — not by merging and re-sorting live rows. That removes the biggest LSM cost: the repeated re-write of data that has not changed, and the multi-level re-sort on the write path.

The honest bound: reduced a lot, not to zero. Append-structured storage greatly reduces write amplification — it does not eliminate it. Each write still pays for the WAL record, the packed block, and index updates, and background GC rewrites the surviving data when it reclaims a sparse region. The win is structural rather than absolute: there is no multiplicative LSM re-write of unchanged live data, and reclamation runs off the hot path — so you avoid both the many× write cost and the compaction-stall tail latency, while writes stay sequential and the WAL remains the single durable path. If you want the exact durability/throughput trade, that is the sync vs async storage dial above.

Model-aware, not opaque blobs

Because the executors run next to the data, callers do not re-implement temporal math on top of a KV value, and the store does not ship whole histories over the wire just to have the client throw most of it away. Windows, filters, distinct, and sequence logic run at the shard, so a read carries temporal semantics rather than returning an opaque blob for the client to re-process — one serving path replaces the usual stack of a stream job, a bespoke cache layout, and a repair pipeline per feature. See the data models — Context, Long Sequence Features, Aggregated Features, and Control State — for what each executor computes at read time.

Storage backends

Pick the durability tier for the deployment.

The shared store is pluggable. Two backends are open source; the disaggregated backend is enterprise.

Backend	Best for	Notes
Local disk open source	Single-node, dev, edge, self-hosted.	One owner per shard; simplest operations, no external dependencies.
MatrixRaft open source	Replicated HA without shared storage.	Consensus-replicated WAL across a fixed replica set; survives node loss. Compute and storage stay co-located — replication, not disaggregation.
MatrixObject Enterprise	Disaggregated, concurrent read/write at scale.	Shared durable object storage; compute and storage scale independently. Details

Deployment

Two modes: embedded (local) and distributed.

The same engine runs two ways. Local mode is embedded — one self-contained node with no proxy and no metaserver. Distributed mode adds a stateless proxy, a metaserver control plane, and a datanode fleet with replication. Bring your own OSS models for embeddings and reading — TemporalStore does not require a hosted API.

Local mode — embedded, single node

Your app / SDKin-process, or one local endpoint

↓ direct calls — no proxy hop ↓

Embedded TemporalStore engineone self-contained datanode · model executors + WAL + in-memory index · no proxy, no metaserver

↓ local durability ↓

Local disk / shared fileWAL, blocks, and bands in one data directory

TS_STANDALONE=1 / TS_META_ADDR=local. The engine runs embedded as a single node serving a local shard — ideal for dev, edge, single-node, and self-hosting. No control plane to run.

Distributed mode — proxy + metaserver + datanodes

Clients (many)SDKs, services, stream consumers

↓ through the gateway ↓

Proxyroutes by namespace / table / bucket

Metaservershard placement & membership

↓ route to the shard owner ↓

Datanodeshards + WAL

↓ replicate the WAL ↓

MatrixRaft replicasfull copies, quorum-committed

Shared storeone durable copy, disaggregated Ent

A real TS_META_ADDR (or TS_DISTRIBUTED=1) opts into the cluster: a stateless proxy tier, a metaserver control plane, and a datanode fleet with MatrixRaft or shared-store replication. Scale each tier independently.

Local (embedded): single self-contained node

In local mode TemporalStore runs embedded as a single self-contained datanode serving a local shard — no proxy and no metaserver to operate. Point it at a data directory and go.

Docker — single standalone node

# run one self-contained datanode on local disk
docker run -d --name temporalstore \
  -p 17102:17102 \
  -e TS_STANDALONE=1 \
  -e TS_META_ADDR=local \
  -e TS_STORAGE_BACKEND=local \
  -v $PWD/ts-data:/var/lib/temporalstore \
  ghcr.io/matrixarkai/temporalstore:latest

Distributed: proxy + metaserver + datanodes + replication

A cluster puts a stateless proxy tier in front of a metaserver control plane and a fleet of datanodes, with MatrixRaft (or a shared store) replicating each shard’s WAL.

docker-compose.yml — metaserver + datanodes

services:
  metaserver:
    image: ghcr.io/matrixarkai/temporalstore:latest
    command: metaserver --listen 0.0.0.0:17101
    ports: ["17101:17101"]

  datanode:
    image: ghcr.io/matrixarkai/temporalstore:latest
    command: datanode --meta metaserver:17101
    environment:
      TS_STORAGE_BACKEND: raft      # MatrixRaft replicated WAL
    deploy:
      replicas: 3
    depends_on: [metaserver]

Bring your own OSS models

Embeddings and any reader model are pluggable. A common self-hosted setup pairs a local embedding model for retrieval with an open-weights model served by Ollama.

Local models with Ollama

# pull an open-weights reader and an embedding model
ollama pull qwen2.5
ollama pull nomic-embed-text

# point TemporalStore at the local endpoints
export TS_EMBED_MODEL=nomic-embed-text
export TS_EMBED_ENDPOINT=http://localhost:11434
export TS_READER_MODEL=qwen2.5

A minimal client

Connect, append a few context records, then ask a temporal question at read time.

Python client — write then read

from temporalstore import Client

store = Client("localhost:17102")

# append time-indexed context events
store.append(session_id="sess-4817", kind="tool_trace",
             payload={"tool": "search", "ok": True})

# read time: build a budget-bounded ContextPack
pack = store.build_context(session_id="sess-4817",
                           window="24h", budget_tokens=2048)
print([(i.kind, i.ts) for i in pack.items])

Keep reading

Go deeper.

Deep essayServing engine internalsStorage layout, replication, and recovery. Data modelsWhat the engine servesContext, sequences, aggregates, control state. BenchmarksScale & quality evidenceLatency, parity, and context quality.

How TemporalStore serves temporal context at scale