TemporalStore Deep Dive
A temporal serving and storage engine for temporal features, risk state, and online context.
Modern online decisions need more than latest-value lookups. They need fresh windows, filters, sequences, counters, distinct state, replay, and observability inside the request path.
Treat temporal context as a serving and storage problem, not a pile of custom cache keys, stream jobs, and repair scripts.
High-cardinality online decisions that need recent windows, sequences, distinct state, filters, and replayable context at request time.
Memory-first serving, model-aware objects, WAL replay, persisted pages, block cache, shared storage, and metaserver-managed placement.
The short version
TemporalStore is an open-source online temporal serving and storage engine. Its target is not the generic cache problem. Its target is the product gap between stream processing, feature stores, wide-column databases, and online caches.
Risk, fraud, ads, recommendation, marketplace trust, and LLM agent systems repeatedly ask questions such as: how many times did this device fail login in the last 30 minutes, which merchants did this card touch in the last 24 hours, which campaign impressions happened in the last hour, what did this user recently read, and what context should be placed into the next model call?
The product thesis: serve high-cardinality temporal features directly, without building a separate batch job, streaming job, cache layout, repair path, and custom serving service for every feature family.
Why cache plus pipelines breaks down
A common architecture starts cleanly: raw events go to a queue, a stream processor computes aggregates, Redis or another online store serves latest values, and offline jobs repair or backfill state. That is a strong architecture for stable feature sets.
It becomes expensive when temporal questions change every week. A risk team adds a new velocity check. An ads team changes a frequency cap. A recommender wants a different sequence slice. An agent product needs session state, tool-call history, policy counters, and user preferences in one context bundle. Each new question creates new stream logic, new keys, new TTL rules, and another place where serving behavior can drift from training behavior.
What TemporalStore owns
The central design choice is to make the storage engine understand the online model. Instead of storing an opaque blob and forcing every caller to implement window logic, the engine exposes model-aware commands over entity-local state.
Architecture
TemporalStore is organized as a metaserver plus data nodes. The metaserver owns namespace, table, partition, placement, and routing metadata. Clients or proxies use that metadata to route writes to primary partitions and route reads according to consistency and freshness policy.
A shard is the core serving unit. It owns a bucket range and contains the in-memory objects, model command executors, index, WAL, page store, block cache integration, background dump logic, and replica replay logic. Keys are hashed into slots; slots map to partition sets; partition sets contain a primary and optional secondaries.
Write workflow
A write is routed to the partition owner, applied to the model object in memory, logged as a replayable mutation, and later merged into persisted pages.
Resolve shard
Client computes bucket and resolves shard.
Receive command
Primary partition worker receives the command.
Select model
Command executor selects the data model.
Load object
Object manager loads or creates the object.
Mutate memory
Model mutates in-memory state.
WAL record
WAL records the mutation.
Mark dirty
Dirty bucket is marked for page dump.
Dump pages
Background storage merges and dumps pages.
This is why repeated updates to a hot entity can be efficient. The hot object is updated in memory while the storage layer can later dump merged object or bucket state. In an LSM design, every update enters the write path and later participates in compaction. In TemporalStore's target design, the storage engine can reduce write amplification for entity-local serving state by merging hot updates before page persistence.
write failed_login_count:
key = device_id
dimensions = { country: "US", method: "password" }
timestamp = now
bucket = 10 seconds
value += 1
query:
key = device_id
metric = failed_login_count
filter country == "US"
window = last 30 minutesRead workflow
Hot reads are served from memory. Warm reads use an in-memory index to locate persisted pages and can hit the block cache. Cold reads go to the shared durable store, decode the page or object, refill cache, and then execute the model-specific query.
The query is not a full-table scan. The routing layer sends the request to the entity's shard, the index identifies bucket and page metadata, and the model runs bounded logic over retained entity state. This is the right shape for high-cardinality online features, where there may be millions of sparse entity keys but each request usually asks about one entity or a small set of entities.
Storage layout: object, bucket, page, band, WAL
TemporalStore's storage vocabulary matters because it explains why the engine is different from a plain cache.
| Concept | Meaning | Why it matters |
|---|---|---|
| Object | The model-aware state for a key, such as a hash, sequence, control-state counter, or aggregate object. | The engine can run domain operations against the object instead of returning an opaque blob. |
| Bucket | A hash-space unit that groups keys for shard ownership and dump/load bookkeeping. | Dirty buckets can be tracked and persisted without rewriting unrelated data. |
| Page | A persisted unit containing encoded object or bucket state. | Cold or evicted state remains queryable through page reads and cache fills. |
| Band | An append-structured region used by page, index, or WAL storage. | Bands make append, freeze, reclaim, and garbage collection visible to the storage layer. |
| WAL | The mutation stream used for recovery and replica replay. | A replica can reconstruct recent updates by replaying mutations after a page/index checkpoint. |
| Index | In-memory and persisted metadata mapping buckets and objects to latest page locations. | Reads follow the index to the latest known state instead of hunting through storage. |
Replication and recovery
TemporalStore uses primary shards and secondary replicas. A secondary reconstructs queryable state from durable page/index state plus WAL replay. In a shared-store deployment, secondaries can read persisted streams directly. In a primary-pull design, secondaries can pull stream data from the current primary.
Important engineering guardrail: WAL alone is not enough forever. Once old updates have been merged into pages and the WAL checkpoint advances, recovery needs page streams, index metadata, and WAL after that checkpoint.
Primary writes
Primary writes mutation.
WAL append
WAL append records the update.
Dump pages
Dirty bucket is dumped into pages.
Index update
Index records latest page addresses.
Load base
Secondary loads base pages.
Replay WAL
Secondary replays WAL after checkpoint.
Queryable
Secondary becomes queryable when replay catches up.
The roadmap hardens this with explicit primary lease or epoch fencing, freshness gates before promotion, secondary lag metrics, and recovery tests that include historical pages. Split-brain writers and stale replicas must be rejected before a deployment can be trusted for correctness-sensitive workloads.
Data model examples
The right model depends on the product question.
| Use case | Entity key | Model | Example query |
|---|---|---|---|
| Purchase velocity | user_id | TemporalCounter | Count purchases in the last 5 minutes. |
| Failed login risk | device_id | TemporalAggregate with dimensions | Failed logins by country and method in the last 30 minutes. |
| Card testing | card_id | TemporalDistinct | Unique merchants touched in the last 24 hours. |
| Chargeback monitoring | merchant_id | TemporalAggregate | Chargebacks by channel in the last 7 days. |
| Frequency cap | campaign_id + user_id | Composite-key counter | Impressions in the last hour, day, or campaign window. |
| Ranking sequence | user_id | Sequence | Recent clicked items filtered by category and recency. |
| Agent context | session_id | Sequence plus counters | Recent tool calls, safety counters, and user preference deltas. |
Why not just Redis or RocksDB?
Redis-style systems are excellent for simple strings, hashes, latest profiles, leader boards, queues, and many cache workflows. MatrixDB is the enterprise product direction for eventually consistent KV workloads that need both low-latency serving and offline or nearline query access. TemporalStore is different: it tries to put temporal semantics inside the serving engine.
RocksDB is a powerful embedded LSM engine and often the right local persistence layer. The tradeoff is that repeated updates create LSM write-path work and later compaction. For a hot entity receiving many small counter or sequence updates, TemporalStore's model is to update in memory, append replayable mutations, then dump merged pages when the storage manager decides to persist dirty state.
| System | Good at | Where TemporalStore differs |
|---|---|---|
| Redis-compatible cache | Fast general data structures and latest-value serving. | TemporalStore adds model-aware windows, filters, replayable feature state, and persisted pages. |
| RocksDB-backed KV | Durable ordered local storage with mature LSM behavior. | TemporalStore avoids treating every hot temporal update as a generic KV rewrite. |
| Feature store | Registry, training sets, materialization, lineage, offline/online consistency. | TemporalStore can act as the online serving engine underneath the registry. |
| Stream processor | Known transformations, joins, durable event-time processing. | TemporalStore serves request-time entity windows when precomputing every window is too rigid. |
| Time-series database | Metric series, analytics queries, monitoring workloads. | TemporalStore is entity-serving-first, not dashboard-query-first. |
Where it fits with feature platforms
TemporalStore should not try to replace every feature platform capability on day one. Systems such as Feast, Chronon, Fennel, and Featureform are strong at registry, definitions, lineage, transformation orchestration, training sets, and offline/online consistency. TemporalStore is strongest as the online temporal serving engine underneath or beside those platforms.
LLM context is a related, not identical, problem
TemporalStore is not a GPU KV-cache manager. It does not replace the transformer KV cache used by vLLM, SGLang, TensorRT-LLM, or LMCache-style systems. The LLM runtime still needs tensor layout, prefix matching, GPU memory management, token-position lifecycle, and attention-cache APIs.
The overlap is structured context and state. Agent systems need recent conversation events, tool calls, retrieved-document metadata, memory freshness, safety counters, user preferences, and policy state. Vector databases are good at similarity search. TemporalStore is useful for temporal and structured context that should be filtered, counted, ordered, replayed, or expired with serving semantics.
Operational design
A serving engine is not a product until it is observable. The current TemporalStore.AI observability work exposes separate pages for TemporalStore, MatrixDB, MatrixKV, and a Prometheus-compatible metrics endpoint. For TemporalStore, the most important signals are shard health, primary placement, replica replay lag, WAL append latency, page dump progress, block-cache hit ratio, storage errors, and client retry visibility.
In a quick engineering environment, those pages can be served under the same HTTP port as the company site. That is useful for fast iteration, but it is not the production boundary. A production deployment should separate the public website from the authenticated operations console and keep raw metrics endpoints private.
Autoscale and rolling deployment design
Autoscaling should not mean that a new instance independently decides which partitions it owns. A new TemporalStore data node should boot from a versioned runtime package, discover its instance identity, register with the metaserver, and advertise capacity. The metaserver then assigns secondary replicas first, waits for catch-up, and only then moves read traffic or primary ownership.
Launch
ASG launches data node.
Start service
Runtime package starts service.
Register
Node registers capacity with metaserver.
Mark active
Metaserver marks node active.
Assign replicas
Planner assigns replicas.
Catch up
Replica catches up.
Update routing
Routing table updates.
Shift traffic
Reads or primaries move gradually.
Scale-down needs the reverse workflow. The node should enter draining state, stop receiving new primary assignments, move primary partitions away, wait for replacement replicas to become healthy, and only then complete the cloud lifecycle termination. This protects the serving path from raw instance termination and makes rolling upgrades safer.
Runtime package requirement: release artifacts should include data server, metaserver, proxy, client tools, dynamic libraries, systemd units, health checks, registration scripts, drain scripts, metrics configuration, and version metadata. Autoscale only works cleanly when a new node can become useful without manual copying or shell repair.
Engineering snapshot
The current AWS test cluster used one metaserver/client/UI node and two data nodes. A Prometheus bridge scraped the metaserver and both data nodes through their runtime variables endpoint.
| Signal | Latest observed value | Interpretation |
|---|---|---|
| Prometheus sources | 3 sources: metaserver, data01, data02 | Live scrape path is wired for TemporalStore. |
| Service smoke | 1,525 iterations over 30 minutes for core modules | STRING, COMMON, HASH, SET, FEATURE, IPS, and RISK stayed stable in that loop. |
| TemporalAggregate | Blocked by response-size check in the deployed artifact | This is not yet a clean aggregate scale pass. It is an explicit P0 follow-up. |
| Two-replica table path | Hit a condition-info load issue in the latest runtime | Secondary replication benchmarks should be rerun after table creation is fixed. |
These numbers are engineering snapshots, not final product benchmarks. The useful signal is the shape of the system: live metrics, module-level smoke coverage, and clear correctness gaps to fix before stronger claims.
The product boundary
TemporalStore should be honest about what it is. It is not a full warehouse, not a full feature platform, not a vector database, and not a transformer KV-cache runtime. It is an online state engine for temporal features and context. MatrixDB handles eventually consistent KV serving, profiles, and offline-queryable state. MatrixKV handles strongly consistent transactional metadata. Together they form the enterprise storage family.
- Use TemporalStore when the feature depends on recent events, windows, filters, distinct state, or sequences.
- Use MatrixDB when the workload is latest profile, large hash/profile KV, hot-key cache, tenant-scale service state, scans, exports, or offline/nearline query over persisted KV data and eventual consistency is acceptable.
- Use MatrixKV when the workload needs transactional KV, timestamp coordination, metadata correctness, or strong consistency.
- Use feature stores and warehouses for registry, training sets, lineage, offline truth, and backfills.