TemporalStore memory benchmark: LOCOMO & LongMemEval_s

639 genuine qwen2.5:7b open-source reader calls (200+200 across both arms on LOCOMO, 119+120 on LongMemEval_s), reader errors ≈ 0 — clearing the paper-comparable bar of reader_open_source_calls > 0 with no fallback.

LOCOMO ties; LongMemEval separates. On LOCOMO both backends retrieve the same source set within budget (retrieved-token ratio 1.00) so the shared reader returns identical answers — TemporalStore's edge is latency. The memory layer's value shows up at scale on LongMemEval_s.

The paper-comparable 120-question slice covers the single_session_user and multi_session types; retrieval is at ceiling (hit@k 1.000) on both, and the reader answers 84–100% of them from TemporalStore memory.

The leading hierarchical memory systems position themselves as the state of the art: each claims to beat the broader field of memory products, not just a full-context baseline. Zep bills itself as “the new state of the art in agent memory”; Mem0 reports double-digit accuracy gains over prior systems; MemOS reports leading OmniMemEval across 14 commercial memory products.

So we compare against the acknowledged leaders, not the long tail. Because these systems already claim to outrank the rest of the field, matching or beating them on the shared axes implies TemporalStore is competitive with the products they outrank — by construction. Refer to those systems' own pages and papers for their head-to-heads against other tools; that is why this report is TemporalStore vs the acknowledged leaders, not a sprawl of individual products.

Those systems all frame their own comparison the same way — their memory vs a full-context baseline, scored by a GPT-4o LLM judge. Their figures below are their own, cited to source — and because they use an LLM judge and stronger agents, they are not directly comparable to the open-source qwen2.5:7b numbers above; the comparable axes are token efficiency and latency.

Methodology honesty. A direct leaderboard claim against these needs the same official judge protocol (an LLM judge such as GPT-4o-mini + matched reader). Until then the leading-system paper scores remain the external target, and the fair result here is the identical-stack A/B above. Vendor scores are protocol-sensitive: Mem0 is 92.5 on its own run yet 57.68 under the independent study — different judge/prompt/models, very different numbers.

Sources: Mem0, Zep, MemOS, Memori, Beyond the Context Window. Leading-system paper: arXiv:2605.29640.

Confirms the deterministic result under a real LLM judge. LOCOMO ties (the 7B reader returns near-identical answers — OpenViking edges it by a single case); on LongMemEval_s TemporalStore wins by +32 points (98.0% vs 66.0%). OpenViking's misses are off-topic tangents and raw-context dumps that never state the asked fact — e.g. “what breed is my dog?” → gold Golden Retriever, TS “Golden Retriever”, OpenViking a paragraph about the Rover dog-walking app. Reader stayed OSS qwen2.5:7b for both arms; Claude is the evaluator, not the reader.

The failure mode, isolated. A bounded recency window answers everything recent and is a total blank on anything older or from a prior session (0.00 / 0.00) — it can only carry what fits. Adding the managed pack recovers every earlier-session fact and lifts cross-session recall from 0.0 to 1.0, at a fraction of full-replay tokens; the remote-only pack reconstructs at 0.80 with no local window at all. The recent dips (local+TS 0.80, remote-only 0.40) are the 7B reader mis-picking a very-recent value the raw window still holds — a reader effect, not a retrieval miss.

End-to-end hook: validated deterministic. These accuracy figures are the retrieval-algorithm probe. The production agent-hook serving path — which earlier collapsed above ~20 session events — is now fixed and validated: a retrieve-deadline correction plus a warm resident proxy give 14/14 cross-session coverage and ~7 ms warm retrieve (vs ~8.8 s cold) on the async production path. Reload (fresh restart) and promotion (disk→memory on normal restart) pass 4 restart cycles each; synchronous storage has a separate serving-record gap under fix, so production runs async.

AWS replication topology comparison

Important: this table is a correctness-first replication visibility run, not the 8,000-operation backend throughput run below. The MatrixObject/object-store row here uses a shared WAL/checkpoint path and forces each secondary to replay from its own cursor before reading the primary-written value.

Raft scale caveat: the first 1,000-op run hit append-pipeline backpressure with the default 32 KB memory replicate limit. Raising the Raft inflight and memory limits removed that rejection. Disabling persistence for successful read-safety probes produced a clean 250-op run with zero failures and secondary visibility p50 at 7.03 ms, but synchronous write p50 is still 3.55 s in this serial HTTP benchmark path.

Takeaway. MatrixObject shared sync is the disaggregated serving mode: keep durable state once, add or replace TemporalStore nodes without copying shards, and let secondaries catch up by replaying the shared log. Raft remains the right mode when replicated durability must not depend on a shared store, but the current AWS data-node path needs write-path optimization before it is fair to publish a same-scale Raft versus MatrixObject number.

The latest AWS validation compares both backends with the same Rust TemporalStore workload: 8,000 shared-store operations, 256-way concurrency, three worker samples per backend, commit 61ffcbe. Both backends finished healthy with replication_healthy=true and max_replica_lag=0.

Why the sync MatrixObject latency differs from the table above: this backend dataset reports 3.56 ms p50 / 6.25 ms p99 for MatrixObject sync storage writes under the EFS-vs-MatrixObject workload. The Raft-vs-Object topology dataset reports 8.92 ms p50 / 38.64 ms p99 because it measures the shared-sync replication path against Raft, not the backend-only EFS comparison. The numbers are both retained, but no longer treated as interchangeable.

Takeaway: MatrixObject removes the biggest EFS penalty in this topology: durable shared-store writes and WAL flushes. Under the same shared-storage backend workload it cuts sync write p99 from 29.45 ms to 6.25 ms, async flush p99 from 236.10 ms to 6.70 ms, and keeps replay consistent at zero lag. Read throughput also improves, but the object-store win is concentrated where EFS hurts TemporalStore most. MatrixObject — enterprise shared object store →

• Reader-hit is a deterministic lower bound; we also report a Claude LLM-judge pass above. Vendor 88–94 numbers are GPT-4o-judged with stronger agents — not directly comparable.
• 7b is a paper-comparable sample — 200 LOCOMO + 120 LongMemEval_s (CPU-only makes a full sweep ~1–2 days). Reader p50/p95 are inflated by the CPU-only, shared host and are not a serving-latency claim; retrieval latency is the storage-layer metric.
• The baseline is an OpenViking-style direct-source-retrieval reimplementation on the shared OSS stack, not OpenViking's shipping product.