TemporalStore memory benchmark: LOCOMO & LongMemEval_s

Shared-storage benchmark: MatrixObject vs EFS on AWS

TemporalStore shared-storage mode can use a shared filesystem such as EFS or the enterprise MatrixObject object store. The latest AWS validation compares both backends with the same Rust TemporalStore workload: 8,000 shared-store operations, 256-way concurrency, nine worker samples across three AWS validation runs, commit 79afda8. Both backends finished healthy with replication_healthy=true and max_replica_lag=0.

Why TemporalStore wins — even when the scores look similar

This benchmark isolates the memory layer on one shared OSS stack. Where a test fits inside a single retrieval budget (LOCOMO), both systems return the same context and the shared reader produces byte-identical answers — a tie. That convergence is a property of small inputs, not evidence the systems are equivalent. Separation shows up where production lives — long, multi-session histories:

Setup & fairness contract

Every pipeline — TemporalStore and the OpenViking-style baseline — runs through one shared OSS stack, so the only variable is the memory backend.

TemporalStore measured results

Retrieval hit@k, recall@k, token reduction and latency are storage-layer properties; reader-hit is the OSS reader answering from retrieved memory.

Head-to-head vs an OpenViking-style baseline

An OpenViking-style flat source-retrieval baseline, reimplemented on the identical OSS stack. M = TemporalStore, B = baseline.

Answer quality is reader-bound

Retrieval is near-ceiling at 1.5b; on the same TemporalStore memory, a bigger reader roughly doubles LOCOMO answer accuracy.

LongMemEval_s by question type (1.5b)

Why we benchmark against OpenViking / VikingMem — not the whole field

OpenViking and VikingMem position themselves as the state of the art: each claims to beat the broader field of memory products, not just a full-context baseline. Zep bills itself as “the new state of the art in agent memory”; Mem0 reports double-digit accuracy gains over prior systems; MemOS reports leading OmniMemEval across 14 commercial memory products.

Those systems all frame their own comparison the same way — their memory vs a full-context baseline, scored by a GPT-4o LLM judge. Their figures below are their own, cited to source — and because they use an LLM judge and stronger agents, they are not directly comparable to the deterministic, tiny-OSS-model numbers above; the comparable axes are token efficiency and latency.

Sources: Mem0, Zep, MemOS, Memori, Beyond the Context Window.

Claude-as-judge — quality under a real LLM judge

The results above use a deterministic answer-term scorer (a lower bound). The competitor papers score with an LLM judge (GPT-4o). We re-scored a paired sample with Claude as the judge (0–10 rubric, correct := score ≥ 6). The OSS reader is unchanged for both arms, so Claude is the evaluator, not the reader.

The three-arm memory-distance probe — where a bounded window goes blind

Complementary to the public benchmarks above: a controlled probe on a 3-session engineering conversation with 15 exact facts (numbers, commit hashes, config values) planted at three distances from the current turn — five each. The same reader (qwen2.5:7b, Claude-judged, embedding retrieval on) answers three ways. Accuracy = fraction answered with the exact value.

Caveats

• Reader-hit is a deterministic lower bound, not an LLM-judge score — not comparable to the 88–94 vendor numbers (GPT-4o-judged).
• 7b is a 200-question LOCOMO sample (CPU-only makes a full dual-model sweep ~1–2 days); 7b LongMemEval deferred.
• The baseline is OpenViking-style — a flat source-retrieval reimplementation on the shared stack, not any vendor's shipping product.