Tech & infrastructure deep dive
How TemporalStore serves temporal context at scale.
TemporalStore is an append-structured, model-aware serving engine: a metaserver plus datanodes, a multi-layer cache, replicated WAL, and tiered storage from memory to SSD to shared store — with pluggable backends up to disaggregated, five-nines deployments.
Architecture
All modules: serving core, dependencies, tiering, and storage backends.
Clients talk to a stateless proxy that routes by namespace and table. A metaserver owns placement; datanodes run the temporal models, back their state with an append-structured storage engine, and lean on two open-source libraries — MatrixCache for tiered caching and MatrixRaft for replicated durability. Context data tiers from memory to SSD to a shared store, and the shared store is pluggable up to the enterprise backends.
One request path, top to bottom: route → place → execute over cached, replicated, tiered temporal state. MatrixObject, MatrixKV, and MatrixDB are enterprise backends; local disk and MatrixRaft are open source.
Node layout
The actual cluster, and what lives inside one datanode.
Zooming in on the topology: a stateless proxy tier fields client connections and caches the routing map; a single logical metaserver owns placement and membership as a control plane off the hot path; and a fleet of datanodes each own a set of shards. Inside every datanode is a memory-to-SSD-to-durable hierarchy that keeps the hot working set in RAM while nothing ever leaves the queryable surface.
Least-recently-used entity state is evicted from memory down to SSD and, eventually, to the durable store — but it never leaves the queryable surface. A read that misses in RAM triggers a cache fill from L2 or the shared store, so cold history stays reachable at a modest latency cost rather than being lost. Admission keeps a burst of cold replay traffic from flushing the hot set.
Deep dive
How each layer works.
Serving core: metaserver + datanodes
The metaserver is the control plane. It owns namespace, table, shard, and bucket-range metadata, decides which datanode owns which shard, and tracks membership and leases. It is not on the hot data path: the proxy caches routing and only consults the metaserver on placement changes. Datanodes are the data plane. Each runs model-aware executors — Context, Long Sequence Feature, Aggregated Feature, Control State — over entity-local state, so a read carries temporal semantics (window, filter, distinct, tail) instead of returning an opaque blob for the client to post-process.
Storage engine: append-structured blocks
Underneath every model is one storage engine. Values are packed into blocks; blocks are grouped into append-only bands; bands are written into slab files on disk. Nothing is updated in place — new state is always appended, and obsolete state is reclaimed later by background garbage collection rather than by rewriting live data. Three indexes make reads cheap: a logical page index maps entity/time keys to the pages that hold them, a physical block index locates blocks within slabs, and an object index tracks larger materialized objects. Because appends are sequential and indexes are small, tail reads and windowed slices stay fast even as an entity's history grows long.
Multi-layer cache: MatrixCache
MatrixCache fronts the engine with two tiers. L1 is in-memory and holds the hot working set — recent events and frequently touched Context nodes. L2 is on SSD and absorbs the warm set that no longer fits in memory. Admission decides what is worth caching and eviction reclaims space under pressure, so a burst of cold replay traffic does not flush the hot set. Under high concurrency this is what keeps p99 tail reads flat.
Replication & recovery: MatrixRaft + WAL replay
Durability starts with the WAL: every mutation is appended to the write-ahead log before it is acknowledged. Periodically the engine writes a snapshot and checkpoints the index, which bounds how far recovery has to go. On restart or failover a replica loads the last snapshot and durable pages, then replays the WAL after the checkpoint to reach the current state. MatrixRaft replicates that WAL by consensus, so a shard survives node loss without shared storage; primaries and secondaries fence with leases and epochs so a stale writer cannot corrupt state after a failover.
Data tiering: memory → SSD → shared store
Context data ages automatically. Recent events and hot nodes live in memory; warm blocks, bands, and the page-index snapshot settle onto SSD; cold blocks and replay history land in the shared store as durable slabs. Reads pull from the highest tier that has the data, and the boundaries move with access, so hot workloads stay in RAM while long tails remain cheap and fully replayable.
WAL replication
How the write-ahead log moves between nodes.
A write for a shard lands on that shard's primary datanode, which appends it to the WAL and hands the record to MatrixRaft. Consensus replicates it to the secondary replicas; only once a quorum acknowledges does the write become commit-visible. Secondaries apply the committed WAL to rebuild their own queryable state, so any of them can serve reads or step up on failover.
Quorum-acknowledged commit: a write is durable across the replica set before it is visible to readers. On primary loss, a replica holding the latest committed WAL is promoted; lease and epoch fencing reject a stale ex-primary that comes back, so it cannot corrupt state after failover.
Write
Request lands on the shard's primary datanode and is appended to the WAL.
Replicate
MatrixRaft ships the WAL record to the secondary replicas.
Quorum ack
A majority acknowledges; only then does the write become commit-visible.
Apply
Secondaries apply the committed WAL to rebuild queryable state, ready to serve or be promoted.
Shared-storage deployments read WAL directly
When the shared store is MatrixObject Enterprise, replication does not have to flow node-to-node. Because committed WAL and blocks are already durable in shared storage, secondaries can read them directly from the shared store instead of pulling from the primary. Compute and storage stay disaggregated: a replica catching up, or a freshly added datanode, hydrates from the object store rather than draining the primary's bandwidth, and the same committed-WAL contract still governs what is visible.
Two replication modes
Raft replication is not compute/storage disaggregation — shared storage is.
Both modes keep your context durable and highly available, but they get there in fundamentally different ways. It matters which one you pick, because only one lets compute and storage scale independently.
Every replica is a whole datanode that stores its own complete copy. Compute and storage travel together — this is replication, not disaggregation.
Datanodes hold no durable data; the shared store owns durability. Compute and storage scale on independent axes — disaggregation.
| Raft replication (open source) | Shared-store replication Enterprise | |
|---|---|---|
| Durable copies | One full copy per replica (N copies) | One copy in the shared store (it handles its own redundancy) |
| Compute & storage | Co-located on each node | Disaggregated — separate, independent tiers |
| Scale by | Adding full replicas (each stores everything) | Adding stateless datanodes (no data copied) |
| Recover a node by | Promoting a replica; new nodes stream a full copy | Reopening blocks from the shared store — no state transfer |
| Best for | HA without external storage; fixed-size clusters | Large scale, elastic capacity, concurrent read/write at five-nines |
| Disaggregation | No | Yes |
Why no vector DB, why not RocksDB
Append-structured, model-aware, and temporal by design.
No separate vector store for most queries
Agent-memory retrieval is filter-first over a temporal tree. Scope hashes, time windows, and typed filters narrow the candidate set before any similarity math runs, which inverts the usual vector-database pipeline where an approximate-nearest-neighbor (ANN) index is the primary lookup and metadata filtering is a post-filter applied to whatever the index happened to return. That ordering matters: the questions agents actually ask — “what happened in this session,” “which tool calls in the last hour,” “what did I decide and when” — are answered exactly by scope and time, not by embedding proximity. Because the candidate set is already small and correct by construction, similarity ranking, when you want it, runs over tens of rows rather than the whole corpus.
Crucially, time-validity and supersession are first-class in the temporal model: a fact carries a valid-time window, and a later write can supersede an earlier one, so retrieval returns what was true at the asked-about time and drops what has been overwritten. A pure ANN index has no notion of this — it returns whatever is geometrically nearest, which routinely means stale, superseded, or out-of-scope chunks that are merely “near” in embedding space. Semantic vector recall is therefore an optional add-on layered on top of the temporal filter for genuinely fuzzy recall, not the primary index. For most queries you deploy, sync, and pay for no standalone vector store, you skip re-embedding and index-rebuild churn, and you stop leaking chunks from other sessions or from a state that no longer holds.
No update-in-place write amplification
RocksDB-style LSM trees rewrite SSTables during compaction: every logical write is re-written several times as data is merged down the levels, so a single append can cost many physical writes, and background compaction periodically competes with foreground traffic for I/O, producing compaction stalls that show up as tail-latency spikes. TemporalStore is append-structured instead — values are packed into blocks, blocks are appended into bands, and bands are written into slab files; nothing is ever rewritten in place. Obsolete data is reclaimed by background garbage collection that drops whole obsolete regions once they fall below a liveness ratio, rather than by merging and re-sorting live data.
High-write temporal workloads — event streams, tool traces, velocity counters, session logs — are exactly the pattern that punishes LSM compaction, and exactly what append-structured storage absorbs cleanly: writes stay sequential, the WAL is the durable path, and reclamation happens off the write path. On top of that, the engine is model-aware: windows, filters, distinct, and sequence logic run at the shard, next to the data, so a read carries temporal semantics rather than returning an opaque KV blob for the client to re-process.
Model-aware, not opaque blobs
Because the executors run next to the data, callers do not re-implement temporal math on top of a KV value, and the store does not ship whole histories over the wire just to have the client throw most of it away. One serving path replaces the usual stack of a stream job, a bespoke cache layout, and a repair pipeline per feature. See the data models — Context, Long Sequence Features, Aggregated Features, and Control State — for what each executor computes at read time.
Storage backends
Pick the durability tier for the deployment.
The shared store is pluggable. Two backends are open source; the disaggregated backend is enterprise.
| Backend | Best for | Notes |
|---|---|---|
| Local disk open source | Single-node, dev, edge, self-hosted. | One owner per shard; simplest operations, no external dependencies. |
| MatrixRaft open source | Replicated HA without shared storage. | Consensus-replicated WAL across a fixed replica set; survives node loss. Compute and storage stay co-located — replication, not disaggregation. |
| MatrixObject Enterprise | Disaggregated, concurrent read/write at scale. | Shared durable object storage; compute and storage scale independently. Details |
Deployment
From one container to a distributed cluster.
Start local: a single datanode on local disk needs nothing else. Scale out by adding a metaserver, more datanodes, and MatrixRaft replication. Bring your own OSS models for embeddings and reading — TemporalStore does not require a hosted API.
Local: single node in a container
The standalone mode runs a datanode serving a local shard with no metaserver. Point it at a data directory and go.
# run one self-contained datanode on local disk
docker run -d --name temporalstore \
-p 17102:17102 \
-e TS_STANDALONE=1 \
-e TS_META_ADDR=local \
-e TS_STORAGE_BACKEND=local \
-v $PWD/ts-data:/var/lib/temporalstore \
ghcr.io/bjmeetsfo/temporalstore:latestDistributed: metaserver + datanodes + replication
A minimal cluster is one metaserver plus a few datanodes, with MatrixRaft replicating each shard's WAL.
services:
metaserver:
image: ghcr.io/bjmeetsfo/temporalstore:latest
command: metaserver --listen 0.0.0.0:17101
ports: ["17101:17101"]
datanode:
image: ghcr.io/bjmeetsfo/temporalstore:latest
command: datanode --meta metaserver:17101
environment:
TS_STORAGE_BACKEND: raft # MatrixRaft replicated WAL
deploy:
replicas: 3
depends_on: [metaserver]Bring your own OSS models
Embeddings and any reader model are pluggable. A common self-hosted setup pairs a local embedding model for retrieval with an open-weights model served by Ollama.
# pull an open-weights reader and an embedding model
ollama pull qwen2.5
ollama pull nomic-embed-text
# point TemporalStore at the local endpoints
export TS_EMBED_MODEL=nomic-embed-text
export TS_EMBED_ENDPOINT=http://localhost:11434
export TS_READER_MODEL=qwen2.5A minimal client
Connect, append a few context records, then ask a temporal question at read time.
from temporalstore import Client
store = Client("localhost:17102")
# append time-indexed context events
store.append(session_id="sess-4817", kind="tool_trace",
payload={"tool": "search", "ok": True})
# read time: build a budget-bounded ContextPack
pack = store.build_context(session_id="sess-4817",
window="24h", budget_tokens=2048)
print([(i.kind, i.ts) for i in pack.items])Keep reading