Measuring memory beyond retrieval

Introduction

Enterprise AI systems are increasingly good at answering a well-formed question with the context they are given. The harder problem is handling more context effectively: more history, more users, more agents, and more data, without letting cost or latency make the system impractical.

That problem shows up everywhere enterprises want AI to do useful work. A support assistant needs to know what a customer already tried before escalating a case. A training system needs to connect course history, simulator performance, and field assessments. A pharma review assistant needs to trace how a claim changed across drafts, evidence packages, label updates, meeting notes, and prior objections.

In each case, the model cannot start from a blank prompt. It is operating inside a history: users, accounts, cases, decisions, preferences, documents, and events. A memory system is the part of the stack responsible for turning that history into usable context at a size and cost that can support real workflows.

As teams move from single assistants to fleets of agents, this becomes even more important. The promise of enterprise AI is not one chatbot answering one request; it’s many agents working across tools, teams, and processes without losing the operational context that makes their actions useful. In this benchmark, we evaluate Graphon as that memory system: storing, retrieving, and answering over long‑running conversational context.

We benchmarked Graphon on two academic conversational-memory evaluations: LOCOMO and LongMemEval-S. In our harness, Graphon achieved:

The headline is that Graphon handled long-running conversational memory with strong answer quality and $0 external LLM credits for graph/index build. On LOCOMO, Graphon answered at 82.4% QA accuracy, a +35.9 point lead over the in-harness BM25 anchor, with 5.60s average latency. On LongMemEval-S, Graphon answered at 62.5% QA accuracy across questions with approximately 115k-token chat histories, while retrieving gold evidence sessions at 0.827 recall@10, with 6.43s average latency.

For teams scaling agentic AI, that matters in practical terms: memory has to expand the amount of usable context without forcing every workflow into longer prompts, repeated intake, or expensive ingestion pipelines.

Why Memory Needs Its Own Benchmark

Most AI benchmarks test what a model can answer from a single prompt. Memory asks a different question: can the system preserve and use information across long histories, multiple sessions, updates, and time-sensitive facts?

LOCOMO evaluates long-term conversational memory across dialogues spanning up to 35 sessions, including question answering, event summarization, and dialogue generation. Its authors note that long-context models and RAG help, but still struggle with long-range temporal and causal dynamics in conversation1.

LongMemEval evaluates long-term interactive memory for chat assistants, including information extraction, multi-session reasoning, temporal reasoning, knowledge updates, and abstention2.

These are not edge cases for large organizations and growing startups. They are the shape of real enterprise grade work: account histories, support escalations, regulated review cycles, internal operations, training records, and project decisions.

What We Tested

We evaluated Graphon on:

• LOCOMO, using 300 sampled questions across ten multi-session conversations. Adversarial questions were excluded, matching the field-reporting convention in the benchmark report.
• LongMemEval-S, using 50 sampled questions, each with approximately 115k tokens of multi-session chat history.

We treat these as sampled evaluation runs, not full benchmark sweeps. The run stores question-level JSONL rows with answers, verdicts, quotes, and latencies so teams can audit individual results and variance. The evaluation pipeline was straightforward: render conversations to markdown, ingest and index them into Graphon groups or the BM25 anchor, retrieve the top 10 results, answer the question, and grade the answer by key-fact coverage.

For grading, gold answers were decomposed into atomic facts. The judge marked each fact as covered, partially covered, or missing, and had to cite a verbatim quote from the answer for every non-missing verdict. QA accuracy is the mean coverage score.

The same external model, gpt-4o at temperature 0, was used for the shared reader, key-fact decomposition, and judging roles. The report also records per-role token and cost ledgers, plus JSONL rows for answers, verdicts, quotes, and latency.

Both Graphon and the BM25 anchor share the same reader model, grading pipeline, and top-k retrieval constraint of 10, so differences reflect memory behavior rather than differences in the answering model.

Result 1: Graphon Connects Distributed Context to Improve Answer Quality

On LOCOMO, Graphon direct answer reached 82.4% QA accuracy and 0.751 recall@10, with 5.60s average latency.

In the same harness, the BM25 anchor with the shared reader reached 46.5% QA accuracy and 0.553 recall@10. That gives Graphon a +35.9 point QA accuracy lead over the in-harness BM25 anchor.

The direct-vs-shared-reader comparison is important. Graphon direct and Graphon + shared reader used the same retrieved evidence, with the same 0.751 recall@10. But Graphon's own answer path scored 17.7 points higher than the generic shared reader. The value is not only finding the right context; it is what happens after retrieval.

LOCOMO is built around long-running conversations where facts are distributed across sessions, user preferences evolve, and the relevant evidence may be far away from the current question. That makes it a useful proxy for enterprise workflows where the answer depends on more than the latest message.

The multi-hop result is especially important: 79.8% for Graphon direct vs. 16.4% for BM25. Multi-hop memory questions are closer to enterprise work than simple lookup because they require connecting information across a history rather than finding one isolated fact.

Recent public reports from other memory systems, including Memori and Mem0, also use LOCOMO to frame memory as an accuracy-cost problem. Our focus in this run was answer quality at controlled cost, not a leaderboard SOTA claim across non-identical harnesses.

In large industrial training workflows, an AI system needs to connect safety and SOP courses, simulator performance, incident reports, and field assessments into one longitudinal learning graph. Leaders could use that context to see where process breakdowns cluster and which cohorts are at highest operational risk.

In MLR review in pharma, an AI agent needs to traverse drafts, evidence packages, label changes, meeting notes, and prior objections to show how a claim evolved, which precedents apply, and what similar assets have already passed review.

Those are the kinds of workflows where memory becomes more than convenience. It becomes the system's ability to preserve institutional context and perform in mission critical situations.

Result 2: Graphon Maintains Answer Quality at Enterprise Context Scale

LongMemEval-S tests a different stress point: very large chat histories. In this run, each question came with approximately 115k tokens of multi-session history. Graphon direct answer reached 62.5% QA accuracy with 0.827 session-level recall@10 and 6.43s average latency.

Graphon retrieved the right evidence sessions at a high rate, but the BM25 anchor in this harness had higher session-level recall@10 at 0.923. Graphon's direct QA still led the BM25 anchor plus shared reader on answer accuracy: 62.5% vs. 54.5%.

The same pattern appears here: Graphon direct and Graphon + shared reader had the same 0.827 recall@10, but Graphon's own answer path scored 15.5 points higher. At the same time, the latency numbers matter. A 5.6-6.4 second average answer time is a real constraint for live support, coaching, and in-the-flow agent use cases, even if it may be acceptable for asynchronous research, review, or back-office workflows.

The benchmark report also lists published field numbers from graphify's benchmark page, where graphify and dense RAG report 76% QA accuracy on LongMemEval-S and BM25/mem0 report 70%. Those numbers are cross-referenceable but not identical-conditions because readers, judges, and harness details differ. The report recommends comparing system-minus-anchor deltas rather than treating every table cell as a same-lab apples-to-apples result.

Other recent public reports on LongMemEval reference 90%+ leaders under their own harnesses, so we treat our 62.5% as competitive but not SOTA. The focus of this run is the system-minus-anchor delta, direct-vs-reader behavior, latency, and cost profile rather than an absolute leaderboard rank.

Two patterns stand out:

• Graphon outperforms the BM25 anchor on multi-session and preference questions, where continuity and user modeling matter most.
• Temporal and knowledge-update questions remain challenging for both systems, consistent with LongMemEval's focus on time and updates as hard dimensions of long-term memory.

The takeaway for technical leaders is: Graphon can work over very large conversation histories, but the hardest cases are still the ones where the system must reason about what changed over time. That is the right next frontier for enterprise memory: not just remembering more, but knowing what is still current.

Limitations and Planned External Validation

These results come from Graphon's benchmark harness, using the datasets, sample sizes, pipeline, reader, and judge described above. The report includes in-harness BM25 anchors to make the comparisons easier to interpret, but published field numbers from other systems are not identical-condition comparisons because their harnesses, readers, and judges differ.

The next step is external validation: running the benchmark with an independent evaluator, such as a third-party research group or a customer-side MLOps team running our harness against their own LongMemEval and LOCOMO splits, and publishing the same core details that matter to technical buyers, including dataset split, retrieval setup, answer path, grading method, latency, and cost.

Until those external runs are complete, we explicitly do not market these results as benchmark SOTA, only as transparent, reproducible evidence of Graphon's memory behavior under the conditions described here.

Where Graphon Sits in the Memory Ecosystem

Graphon is not positioned here as a leaderboard-only research system. The goal is multimodal, multi-agent memory infrastructure with predictable indexing cost. Some highly optimized memory systems chase 90%+ benchmark results with more specialized pipelines, different harnesses, and in some cases higher token budgets. Our focus in this run is the enterprise tradeoff: answer quality, anchor-relative lift, latency, and $0 external LLM credits for graph/index build under a transparent harness.

What We Learned About Memory

Memory has to be measured at the answer level. Recall matters, but users experience the final answer. This is why the report includes both recall@10 and QA accuracy.

Synthesis matters after retrieval. On both benchmarks, Graphon's direct answer path used the same retrieved evidence as Graphon + shared reader but scored 15.5-17.7 points higher. That is the strongest evidence that memory quality is not just about finding chunks; it is also about turning remembered context into the right answer.

Multi-session memory is different from document search. Graphon's largest LOCOMO advantage appears on multi-hop questions, where the answer depends on relationships between facts rather than keyword overlap.

Time and updates remain hard. LongMemEval-S shows stronger Graphon performance on multi-session and single-session assistant questions, but lower performance on knowledge-update and temporal-reasoning questions. We are exploring time-aware indexing and update-tracking mechanisms to improve on these categories in the next iteration of the memory architecture.

Cost belongs in the benchmark. Enterprise memory systems ingest continuously: tickets, chats, meetings, notes, messages, documents, and workflow events. The report's $0 external LLM credits for graph/index build result matters because ingestion cost shapes how much context teams can afford to retain and use.

For a team ingesting hundreds of thousands of messages per day, even a modest $0.10 per 1K tokens ingestion pipeline can add up to thousands of dollars per month. Eliminating external LLM spend for indexing materially changes how much history teams can afford to keep.

Latency belongs in the benchmark too. Graphon's direct answer latency averaged 5.60s on LOCOMO and 6.43s on LongMemEval-S. That is part of the product tradeoff: long-memory accuracy is valuable, but live workflows need clear latency expectations.

Looking forward: What This Means For Teams Scaling Agentic AI

Agentic programs depend on more than model reasoning. They also need systems that can carry enough context forward across tools, sessions, users, and workflows.

Graphon's benchmark results point to a practical path: treat memory as shared infrastructure for agents, not as a longer prompt. The system needs to ingest more history, retrieve the right context, answer from it, and keep the economics workable as usage grows.

In practice, teams can plug Graphon in as the memory layer behind their existing orchestrators or agent frameworks, replacing ad hoc vector-store calls for multi-session context with a shared memory API.

That is the impact of these results: high answer quality on long-term memory benchmarks and $0 external LLM credits for graph/index build in this harness. For teams building agentic AI, memory is what lets agents operate over real enterprise history rather than isolated task windows.

If your team is building agentic systems over long-running customer, operational, training, or compliance histories, we can run this evaluation pattern on your data and show where memory improves answer quality, cost, and latency in your own workflow.