Provenance, not a press release

Don't trust the number. Recompute it.

Most memory benchmarks ask you to take a headline on faith. This one ships the grader, the per-question verdicts, and a one-command tool that re-derives the score from those verdicts — with no engine and no network. What stays private is the memory engine that produces the answers, never anything you need to check the scoring.

64.6%

31 / 48 · full-haystack

Baseline — retrieval only, no consolidation engine. ✓ recomputable

72.9%

35 / 48 · lower bound

Full engine, with memory consolidation. reported

50%

6 / 12 · on-device

Fully local, 3B model, zero cloud. ✓ recomputable

LongMemEval-M · full-haystack variant (~480 distractor sessions per question) · answers gemini-2.5-pro / Qwen2.5-3B · flexible LLM judge · e5 embeddings

The 30-second check

verify.js reads the published per-question ledgers, recomputes accuracy and the per-category breakdown, and asserts they equal the claimed headline. It proves the advertised number is the exact sum of the published rows — no hidden questions, no arithmetic massaging.

node · zero dependencies

# recompute every published score from its per-question rows
$ node verify.js

✓ Baseline (retrieval only, no consolidation engine)
  recomputed from 48 published rows:  31/48 = 64.6%
  claimed headline:                   31/48 = 64.6%
  match: total ok · hits ok · accuracy ok

✓ ALL LEDGERS CONSISTENT — headline = exact sum of published rows.

# and audit the grader itself against the public ground truth
$ node scoring.js --selftest
✓ self-test: 6/6 expected verdicts reproduced

Trace any row yourself. Every question id below is the official LongMemEval id — look it up in the public dataset to see the question, the haystack, and the ground-truth answer.

The auditable ledger — baseline, 48 questions

Every question, every verdict. This is the full record behind the 64.6%. Filter it, scan it, then run verify.js and watch the same 31/48 fall out.

Question id	Category	Verdict	Model answer → expected

Answer excerpts are trimmed for reading; the verdicts they carry are the real graded verdicts. Full untruncated transcripts are available on request.

Where the score comes from — baseline by category

What the memory engine actually does

The consolidation engine earns its keep exactly where memory gets hard: multi-session aggregation — questions whose answer is scattered across many conversations buried in hundreds of distractors.

1/8

baseline

→

5/8

full engine

Why 72.9% is reported as a lower bound. Only the multi-session category was re-run with the engine; the other categories' misses were never retried with consolidation, so a full engine re-run could only raise the number, not lower it. The individual full-engine transcripts are available on request rather than folded into the recomputable ledger — because the honest record notes that a couple of first-pass verdicts were LLM-judge noise, corrected on replay. We publish what replays.

How we avoid fooling ourselves

Replay before you cite. LLM answers and the judge are non-deterministic. A HIT counts only if it reproduces on a re-run with the same config. Several false verdicts were caught and discarded this way.
One config for the cited number. No per-question cherry-picking — a single configuration is applied uniformly across the whole set.
Question-blind memory. The engine never sees the benchmark questions while it organises memory. It consolidates topics, not answers.
Full-haystack, on purpose. ~480 distractor sessions per question — the hard, honest variant. Many quoted memory numbers use the easier oracle or -S sets; those are listed separately, never blended in.

Reference points — kept separate on purpose

~84%

Full engine on oracle-evidence vaults (no distractors) — the easy ceiling, not the headline.

~60%

Literature GPT-4o full-context — but on the easier -S variant, not full-haystack.

~50%

Fully local, 3B model, zero cloud — the price of sovereignty, reported plainly.

3 misses

The remaining full-engine misses are decomposed, not hand-waved: two are benchmark-structural (other personas' first-person text in the shared haystack), one a write/read coverage trade-off.