The First Unified Test of AI Memory Published Its Scores. Nothing Reached 60.
· CX Pulse
Fifteen commercial AI memory products were scored under one rulebook. The winner reached 58, and the weakest column was following the rules it was given.
The Agent Memory Leaderboard published its first verified results on August 12, and this week it climbed to the top of Hugging Face's trending list. It's the first attempt to score AI memory systems against each other under one fixed rulebook.
The setup is worth understanding, because it's what makes the numbers mean anything. A candidate system supplies only two operations, Add and Search. The platform controls everything else: the datasets, the model that writes the answer, the judges, the aggregation. So when one system beats another, the difference comes from how it stored and retrieved the information, not from a better answering model underneath.
Fifteen commercial products entered. Mem0, Cognee, Supermemory, MemOS, TencentDB and several NetEase systems are on the board.
The top score was 58.02 out of 100. The median was 41.4.
Recall is the part that already works
Scores are broken into seven capabilities, and reading across them is more useful than reading the rank.
MemoraX, the system in first place, scored 89.9 on explicit fact recall. Facts, attributes, sources, entities. Nearly nine out of ten.
The same system scored 51.2 on memory governance, which the benchmark defines as update, conflict, deletion and forgetting. Across the whole commercial field the median on that category was 32.1.
That gap is the finding. Storing what somebody told you is close to handled. Noticing that it stopped being true isn't.
What a 32 looks like from the customer's side
People change. They move. They replace the card. They cancel the second line. They tell you in March that email is fine, then call in June and ask to be phoned instead.
A memory layer scoring in the high eighties on recall and the low thirties on governance will hand back the March answer. It'll do it quickly, and it'll do it with a source attached, because retrieving the old fact is exactly the thing it's good at.
That is the version nobody catches. Nothing looks broken from the inside. The answer arrives fast, it arrives sourced, and it is simply out of date, which is why "our AI remembers your customers" is a claim that needs a second question after it.
The lowest column on the board is the one about following rules
Two other categories are worth pulling out.
Context learning and execution covers rules, procedures and constraints. In plain terms, it asks whether the system applies the policy it was given. The best score in the commercial field was 30.0. The median was 26.7. Two well known systems came in under ten.
Temporal and event reasoning covers dates, order and state changes. Median 21.5.
Those two are the ones a support operation runs on. Company policy, and the order things happened in. They're the weakest columns on a board where the leader is at 58.
Where this leaves a small company
None of this says the tools are bad. A first benchmark usually reads low, the field is young, and 58 today is a starting line rather than a ceiling. The useful part is knowing which half of the promise is real right now.
Any product demo of AI memory will show recall, because recall is the part that works. It looks like magic to watch a system quote something from four conversations ago.
The question that separates the products is what happens after the stored fact changes. Tell it something true, tell it later that the thing changed, then ask. Whether it returns the new answer, the old one, or both without picking is a thirty second answer, and it maps directly onto the column where every system on this board is weakest.
The industry spent this year teaching these systems to remember. The scores say the harder work, letting go of what's no longer true, is mostly still ahead.