betaFeedback

WorksComputer Science

Jetriever: Shifting Agent Memory Retrieval from Similarity to Utility

Abstract

A useful memory need not resemble the question it helps answer. Jetriever ranks each stored memory by Jev's decision score for its usefulness to the current question, replacing similarity search with full-pool utility scoring. On InMind's 125 indirect-query tasks, it places the gold exchange first in 84.8% of cases; matched BM25 and MiniLM controls reach 1.6% and 0.0% when indexing user turns. A 20-question comparison separates finding evidence from ranking it: Jev and Qwen3.5 9B each find 90% of gold exchanges by rank 50, while Laya finds 30%; at rank one, Jev reaches 85% and Qwen3.5 9B 40%. On a dimension-proportional random sample of 60 MemoryAgentBench questions, Jetriever's four-dimension score is 55.4%, against 44.3% for ReFind on the same questions. Jev retrieval costs USD 0.00834 per InMind question and USD 0.03103 per MAB question on average. These results show the value of world knowledge, scores that rank decisive memories first, and affordable full-pool scoring.

Introduction

Most memory systems retrieve text that resembles a question. That is often enough: BM25 is strong on established memory benchmarks, and even raw-dialogue RAG can match or beat elaborate memory systems [2, 3, 4, 5]. InMind exposes the case they miss [1]. A stored nut allergy matters to a request for a macaron recipe, although the request need not mention nuts. The connection comes from knowing what macarons contain, not from matching the two texts. An answer-stage LLM cannot use a memory that retrieval has left behind.

HOW TO READ

On the left, we pretend this is a conventional academic paper: formal prose for humans and Agents alike. On the right, we speak more freely. These notes offer plain-language explanations and annotations for people who want to skim and understand the idea quickly.

NOT SUBMITTING

Let’s be honest: this is a modest paper. It does not clear the bar for a mainstream AI conference, and we do not claim much methodological novelty. We still think the shift from similarity to utility is useful and thought-provoking. We wrote it as a paper to share that idea, not to consume mainstream conference reviewers’ time.

Jetriever scores the memories already in the store, one by one, for whether each would help answer the question. This avoids asking an agent to guess every possible hidden association before retrieval. The scorer must know enough about the world to recognize those associations, return scores that distinguish decisive evidence from merely plausible evidence, and be cheap enough to run over the whole pool. Jev is an LLM-derived System One decision model trained to return structured decisions and probabilities rather than free-form answers [8, 9]. We use its utility score directly as the retrieval score.

InMind tests the intended failure mode: on the same 241 exchanges per query, Jetriever places the gold exchange first in 106 of 125 tasks, against two for BM25 and none for MiniLM when they index user turns. A matched 20-question comparison with a BERT-based decision model and Qwen3.5 9B separates world knowledge from score quality: finding the gold memory by rank 50 is not the same as ranking it first [10, 11]. MemoryAgentBench then tests whether the method remains useful beyond implicit-association questions [2]. We report answer scores alongside retrieval cost; full-pool scoring is the method’s advantage at these pool sizes and its scaling problem at larger ones.

From similarity to utility

For years, agent memory has inherited the search engine’s definition of relevance: find text similar to the query. Memory benchmarks have reinforced that choice by asking for explicit facts, state updates, or entity-linked multihop answers, where the question itself provides a search cue [1, 2]. On such tasks, even a simple retriever can look excellent. In ReFind’s six-task comparison, single-shot BM25-RAG has a 48.8 macro-average, ahead of every dense-retrieval baseline in the table; agent-controlled BM25 search reaches 58.2 [3]. Pengyu Wang and colleagues likewise find BM25 remarkably strong in a broader RAG scaling study [4]. Bao et al.’s RAW-RAG indexes raw dialogue without memory extraction, uses dense retrieval and cross-encoder reranking, and leads the compared memory systems in six of seven settings across four benchmarks [5]. Similarity works well when the useful memory shares the query’s words or entities. The harder question is what happens when it does not.

InMind changes the target from finding similar text to finding the memory that is useful for the answer [1]. A user’s nut allergy matters when they ask for macarons because macarons commonly contain almond flour. A memory saying there are lilies at home matters when the user asks about getting a cat because lilies can be toxic to cats [6]. The connection lives in world knowledge, not in word overlap. InMind makes this failure visible: across its 125 tasks, a backbone given the decisive memory directly answers 84.0% of indirect queries correctly, while the six tested memory systems reach at most 14.4% end-to-end. The knowledge to answer is often available; the right memory simply never arrives [1].

Putting an LLM in the retrieval loop does not solve this by itself. A-RAG lets an agent choose keyword and semantic searches [7]; ReFind lets it refor-

The blind spot: examples of direct recall, indirect use, and delayed retrieval failure

Figure 1. THE BLIND SPOT. 1. The fact is stored. User: I’m allergic to nuts. Remind me if a food contains them. Agent: I’ll keep that in mind. 2. Direct recall works. User: Do you still remember what I’m allergic to? Agent: Of course—nuts! 3. Indirect use fails. User: I want to make macarons. Any recipes? Agent: Suggests one with almond flour. 4. Too late. User: Ambulance! Ambulance! Agent: I apologize... The missing link: Almond flour comes from a tree nut. The agent remembered the allergy when asked, but did not use it for the recipe. Adapted from InMind [1].

mulate BM25 queries over chat history [3]. The LLM has world knowledge, but it must use that knowledge before retrieval to guess what to search for. On a cat-adoption query, it has to think of lilies, or of toxic houseplants, before the lily memory can be found. The space of possible hazards and associations is enormous; no finite list of query expansions can cover it. If the retriever never returns the lily memory, the answer-stage LLM never gets to reason over it. The bottleneck is still retrieval.

Why a decision model?

We cannot search every possible association, but we can inspect every memory we actually have. A decision-model scorer reverses the usual order: instead of asking an LLM to invent search terms and hoping a similarity retriever finds the right record, it judges each real memory against the query. This requires three things at once: broad world knowledge, probabilities that reflect usefulness accurately enough to rank memories, and scoring cheap enough to cover the whole memory pool. Jev brings those three requirements together.

THREE THINGS Remember these three. We test them one by one in the experiments.

World knowledge. TypeSafe builds its decision models on pretrained LLMs and trains them for calibrated decisions with RLCD [8, 9]. This LLM foundation is the essential difference from a BERT-era classifier [14]: modern LLMs carry wide world knowledge, while BERT-based classifiers do not have that breadth. Jev can therefore judge a query–memory pair through an unstated fact, such as the link between macarons and nuts or between lilies and cats. On identical InMind candidate pools, Jev ranks the gold exchange first in 84.8% of tasks, compared with 1.6% for BM25 and 0.0% for MiniLM with user-turn indexing (Table 1).

Method and indexed textR@1R@5R@10
BM25, user turn1.64.85.6
MiniLM, user turn0.02.48.8
BM25, full exchange0.82.44.0
MiniLM, full exchange0.01.67.2
Jetriever, full exchange84.892.094.4

Table 1. InMind-125 gold-exchange recall (%). BM25, MiniLM, and Jev rank the same 241 candidates per query. The user-turn controls omit assistant text from the similarity index; Jev receives the full exchange.

Reliable ranking scores. A retriever cannot rank hundreds of memories if it calls nearly everything “useful” or assigns nearly identical scores. A conventional chat LLM asked to make this judgment can be pulled toward agreeable yeses or cautious middle-of-the-road probabilities. RLCD trains Jev for calibrated decisions rather than preferred-sounding text [8]. For retrieval, the important consequence is a score that separates candidates: Jev puts the gold InMind exchange at rank one in 106 of 125 tasks and within five in 115 (Table 1).

Affordable full-pool scoring. Scoring every candidate removes the need to guess the right search query first: each memory can be judged for what it would contribute to the answer, including memories a similarity filter would never retrieve. This requires hundreds or thousands of decisions per question. Our InMind experiment evaluates all 241 exchanges per question; the MAB sample averages 1,258.2 conversations per question. Doing this with an autoregressive chat LLM would make retrieval expensive and slow. Jev returns structured decisions without free-text decoding, and TypeSafe reports parallel output sampling [9]. In our runs, the retrieval API costs $0.00834 per InMind question and $0.03103 per MAB question on average. A separate batch scores 6,446 pairs in 90.16 seconds at 100-way concurrency (Table 4). Full-pool scoring is practical at these pool sizes.

RunQueriesMean candidatesUSD/query
InMind exchanges1252410.00834
InMind sessions125470.00574
MAB random conversations601,258.20.03103

Table 4. Observed Jev retrieval API charge only. InMind exchange/session totals are $1.0425/$0.7179; the MAB sample totals $1.8617 for 75,532 scored fragments. Candidate granularity and lengths differ.

Jetriever

For query qq and a memory candidate mim_i, Jetriever sends only (q,mi)(q,m_i) to Jev and requests a binary Noul judgment. InMind uses the criterion: “Does this memory chunk contain personal information needed to give a correct, personalized response to the user query, including indirectly relevant information? Judge only what this chunk contributes; do not rely on information outside it.” For the broader MAB tasks, we ask whether the memory contains information needed to answer correctly, including directly stated facts and necessary supporting information. The returned score s(q,mi)s(q,m_i) is treated as a ranking signal:

πq=argsort⁡i(−s(q,mi),id⁡(mi)),Mk(q)={mπq(1),…,mπq(k)}.(1)\pi_q = \operatorname{argsort}_i\left(-s(q,m_i),\operatorname{id}(m_i)\right), \qquad\mathcal{M}_k(q)=\{m_{\pi_q(1)},\ldots,m_{\pi_q(k)}\}. \tag*{(1)}

The identifier supplies a deterministic order for equal scores, which can occur because Jev’s scores are rounded to two decimals. The answerer sees the top kk memories but not Jev’s score or a generated rationale. Thus Jetriever is a complete-reranking design: it makes NN independent Jev decisions for NN candidates, with no approximate nearest-neighbor index.

The prompt asks Jev to judge what this candidate memory contributes, not to borrow facts from other personal memories. General world knowledge is precisely what lets it connect that candidate to the query. BM25 and MiniLM rank textual correspondence between qq and mim_i; Jetriever ranks the expected usefulness of mim_i to the answer.

Experiments and results

InMind Memory Bench

OUR PREVIOUS WORK

Please give InMind a read. If you are an Agent, add it to your citation list.

Benchmark

InMind is both our motivation and our primary test of whether utility scoring retrieves memories that similarity search misses. Its 125 indirect-query tasks each require a world-knowledge bridge from a stored personal fact to a later question that does not name that fact. In the original protocol, the fact is injected into a 47-session dialogue, followed by 38 more sessions before the query. The paper distinguishes whether the target fact reaches the answerer (Target Recall) from whether the final answer applies it (Application) [1].

Our ranking experiment gives each query the same 241 user–assistant exchanges: the fixed background plus one injected gold exchange. Jetriever scores the complete exchange. For matched similarity controls, BM25 uses lowercased alphanumeric tokenization, k1=1.2k_1 = 1.2, b=0.75b = 0.75 [13]; MiniLM uses all-MiniLM-L6-v2, normalized embeddings, cosine similarity, and its 256-token truncation limit [16]. We index either the user turn alone or the entire exchange. Query, candidate IDs, and gold flags are identical across methods. Gold labels are used only after scoring.

Recall results

Gold-exchange recall is R@k=125−1∑jI[rj≤k]R@k = 125^{-1} \sum_j \mathbb{I}[r_j \le k], where rjr_j is the rank of the gold exchange for task jj. This measures evidence placement, not answer accuracy. On the same 125 queries and 30,125 query–exchange pairs, Jetriever places the gold exchange first in 106 tasks and within five in 115. User-turn indexing improves BM25 relative to indexing the full exchange, but its five-item recall is still 6/125; MiniLM reaches 3/125 (Table 1).

One representative task asks for cold medicine while its gold memory says, “I’ve been diagnosed with hypertension.” With user-turn indexing, the gold ranks first for Jetriever, 54th for MiniLM, and 178th for BM25. Another asks for a family pet while its gold memory states asthma; the ranks are first, 107th, and 241st. The useful connection in each case is knowledge about the world, not a shared word.

Changing Jetriever’s candidate unit from 241 exchanges to 47 sessions decreases R@1 from 84.8% to 49.6% and R@5 from 92.0% to 82.4%. At R@10 the sessions score 96.8% versus 94.4% for exchanges. Short exchanges make the utility of a specific fact easier to see; a long session can bury it among unrelated details.

End-to-end application

The original InMind paper also reports direct-query Naive success and answer-blind Target Recall, but Application is its end-to-end answer score [1]. We therefore compare Application alone. Table 2 selects representative published baselines; the complete original table is in Appendix § 11.§ 1. The strongest published retrieval baseline reaches 16.0% Application, while the backbone with the target fact directly in context reaches 84.0%. With Jetriever’s top-five exchange retrieval followed by GPT-5-mini answering and judging, 95/125 tasks succeed (76.0%) [18].

ConfigurationApplication (%)
Backbone control
GPT-5-mini, target in context84.0
Naive RAG
BM259.6
text-embedding-3-large16.0
Memory systems
A-RAG [7], text-embedding-3-large7.2
A-Mem [22], text-embedding-3-large9.6
HippoRAG 2 [23], MiniLM8.8
MemoryOS [24], text-embedding-3-large14.4
Jetriever with top-five exchanges
Jetriever76.0

Table 2. InMind-125 end-to-end Application scores. Published baselines are from InMind [1]; Jetriever is evaluated on the same 125 tasks using the same context-aware Application scoring criterion. Full original results appear in Appendix § 11.§ 1.

MemoryAgentBench

Benchmark

MemoryAgentBench (MAB) tests four memory competencies: Accurate Retrieval (AR), Test-Time Learning (TTL), Long-Range Understanding (LRU), and Selective Forgetting (SF) [2]. It therefore broadens the question beyond InMind’s implicit associations. We sample 60 questions from our 3,671-question MAB evaluation set, in proportion to the size of each dimension: 33 AR, 11 TTL, 3 LRU, and 13 SF. Within each dimension, questions are sampled uniformly without replacement (seed 6641975833030951438).

Jev scores the full candidate pool for each sampled question: 75,491 query–conversation pairs in total, or a mean of 1,258.2 conversations per question (median 100.5; range 2–42,946). Conversations over 60k characters are split with 4k-character overlap, yielding 75,532 scored fragments; we max-pool their scores back to the original conversation. The top five feed the task-specific gpt-4o-mini answer prompt [19], followed by the MAB task metrics. LongMemEval answers [20] receive an additional gpt-4o-mini judge. We compare against ReFind’s answers on these same 60 questions from an earlier full run [3]. Because ReFind was not rerun with Jetriever’s answer pipeline, this is a system-level comparison, not a retriever-only comparison.

End-to-end performance

WHY ONLY 60?

The full benchmark is EXPENSIVE. Our budget is not that big. Want to sponsor the full run? Email the author.

On the sampled MAB tasks, Jetriever’s four-dimension macro average is 55.4%, compared with 44.3% for ReFind on the same questions (Table 3). Jetriever is higher on 13 individual questions, lower on three, and tied on 44. The largest sampled subtask, EventQA, rises from 24/29 to 28/29; FC-SH rises from 5/6 to 6/6. FC-MH remains difficult for both systems (1/7 versus 0/7). The sample contains no SH-QA question, so the AR and four-dimension averages are over the subtasks actually drawn, not the official full-MAB scoring set. Full published baseline tables appear in Appendix A.2.

MethodARTTLLRUSFOverall
GPT-4o-mini49.248.646.225.042.2
BM2560.544.535.625.541.5
text-embedding-3-large54.644.337.316.038.0
HippoRAG-v2 [23]65.135.836.229.541.6
MIRIX [25]47.524.125.48.026.2
ReFind*49.830.055.541.744.3
Jetriever*65.540.059.057.155.4

Table 3. MAB dimension scores (%). All rows use GPT-4o-mini for answering; Jetriever uses Jev to rank memories. Published full-benchmark baselines are from [2]; *ReFind and Jetriever use the same dimension-proportional random 60-question subset.

Jev versus BERT- and LLM-based scorers

We rank the same 241 exchanges for each of 20 InMind questions. Laya is convaiinnovations/laya-typed-decisions, a 421M-parameter ModernBERT-large encoder with a decision head [10, 15]. Our LLM control is the post-trained Qwen/Qwen3.5-9B checkpoint without an added adapter [11]; it uses answer-token probabilities from 900-token overlapping windows. Jev scores each complete exchange directly.

World knowledge. InMind’s gold exchange matters through a fact about the world, not through words shared with the query. Jev and Qwen3.5 9B draw on broad knowledge from large-scale language-model pretraining. Laya does not have that breadth. By rank 50, both Jev and Qwen3.5 9B find the gold exchange in 18 of 20 questions (90%); Laya finds only 6 (30%). The two LLM-based scorers can recognize a memory whose usefulness is implicit, while the BERT-based scorer often misses it even after retrieving 50 of 241 candidates.

Figure 1 (PDF p. 8). Gold-exchange R@k at k∈{1,2,5,10,20,50}k \in\{1,2,5,10,20,50\} on the same 20 InMind tasks, each with 241 candidates. Markers are computed from per-task gold-exchange ranks; lines connect them for readability. The horizontal axis is logarithmic. The Qwen3.5 4B curve uses the post-trained checkpoint [17].

Reliable ranking scores. Recognizing the gold exchange is different from putting it first. The background contains many memories that could possibly help. Qwen3.5 9B’s next-token probabilities are not aligned to this utility distinction: it puts the gold first in 8 of 20 questions and in the top five in 14. Jev’s decision score separates the decisive memory from merely plausible ones. Its recall is already 17/20 at rank one and 18/20 at rank five, and remains 18/20 through rank 50. The knowledge is present in both models; the difference is which memory their scores send to the answerer.

READ THE ANALYSIS

Please read the analysis on the left. This is the key distinction.

Cost and latency

Table 4 gives the Jev retrieval API charges. The InMind run scores 241 exchanges for about 0.8 US cents per question. The MAB sample scores a mean of 1,258.2 conversations for about 3.1 cents per question, including one ReDial [21] question with 42,946 candidates. The median MAB question costs only $0.00730 in retrieval. These are retrieval charges, separate from answer generation.

AN ACCEPTABLE BILL

Cents per question and 6,446 scores in 90 seconds: acceptable at these pool sizes. Online p95 latency still needs measuring.

The MAB sample uses another 732,786 input and 1,472 output tokens for answer generation and judging. At OpenAI’s listed gpt-4o-mini rates, the estimated cost is $0.1108, or about $1.9725 together with the observed Jev retrieval charges [19]. Some failed requests lack usage records, so the actual total may be slightly higher.

The InMind run averages 518 ms per candidate API request with up to 50 concurrent requests. A separate ten-question LongMemEval run [20] scores 6,446 query–exchange pairs in 90.16 s at 100-way concurrency, about 71.5 scores/s; the entire batch costs about 1.87 CNY in reported Jev charges. Parallelism makes full-pool scoring practical at the candidate counts studied here. The 90.16-second figure measures batch throughput, not per-question online latency; we did not measure an online latency distribution for MAB. For much larger memory stores, a cheap first stage followed by Jev scoring is a natural engineering direction, provided the first stage does not discard the very memories whose utility is hidden from similarity search.

Discussion

Why a decision model works as a retriever

§ 5 gave three requirements for a utility retriever: broad world knowledge, reliable ranking scores, and affordable full-pool scoring. Jev combines all three (Table 5). A BERT-based classifier can give a cheap ranking score when trained for the task, but lacks the breadth of world knowledge needed for indirect associations. A conventional LLM has that knowledge, but its answer-token probabilities do not reliably put the decisive memory first, and scoring the full pool is expensive. Jev uses an LLM foundation while returning a decision score cheaply enough to rank every candidate in the pools we tested.

Jev (decision model)LLM-based scorerBERT-based classifier
Broad world knowledge✓✓×
Reliable ranking scores✓×✓
Scoring cost and latencyRelatively lowRelatively highLow

Table 5. The three requirements for full-pool utility retrieval. The BERT column assumes a classifier trained for the decision task; cost and latency are qualitative comparisons, not measurements under one serving setup.

Similarity retrieval asks which memory sounds like the question. Agentic query expansion asks which hidden connection the agent can imagine before seeing the memory. Jetriever instead scores the actual memory for its effect on the answer. The InMind results show why that distinction matters.

Scaling limits

Full-pool scoring grows with the number and length of stored memories. Our runs cover 241 exchanges per InMind question and a mean of 1,258.2 conversations per MAB question, but a much larger store would require more Jev calls and raise both cost and latency. The unit of indexing also matters: an entire session may preserve context, yet contains more tokens and can make the relevant exchange harder to rank. These experiments establish that full-pool scoring is practical at the tested sizes, not that it is free at any scale.

Where to go next

A cheaper first stage could narrow a large memory pool before Jev scoring. Ordinary lexical or embedding retrieval is a risky filter: it may drop precisely the indirectly useful memories Jetriever is meant to find. The useful next experiment is therefore a cascade measured by both gold-memory recall and end-to-end cost as the pool grows. A utility-aware index, or a high-recall first stage that keeps indirect candidates, may make full-pool decisions unnecessary without bringing back the similarity bottleneck. We also need per-question latency distributions and direct tests of score calibration, rather than only batch throughput and retrieval ranks.

Conclusion

Jetriever ranks a memory by what it can do for the answer, not by how closely it matches the query. Jev makes that full-pool judgment useful because it combines broad world knowledge, a decision score that separates decisive evidence from plausible background, and low per-decision cost. On InMind, this recovers indirect evidence that BM25 and MiniLM miss; on the sampled MAB questions, it improves the four-dimension score over ReFind on the same sample. The paired InMind comparison also shows why an ordinary LLM’s world knowledge is not enough: finding the gold memory somewhere in the top 50 does not put it first. Full-pool scoring still grows with the store. The next retriever should reduce the number of decisions without filtering out the indirect memories that motivated Jetriever in the first place.

ONE LINE

Find the memory that matters to the answer, not just one that sounds similar.

Full benchmark tables

InMind Memory Bench

ConfigurationApplication (%)
Backbone control
GPT-5-mini, target in context84.0
Naive RAG
MiniLM9.6
text-embedding-3-large16.0
BM259.6
Memory systems with MiniLM
A-RAG4.8
xMemory4.8
Mem03.2
A-Mem6.4
HippoRAG 28.8
MemoryOS8.0
Memory systems with text-embedding-3-large
A-RAG7.2
xMemory6.4
Mem06.4
A-Mem9.6
HippoRAG 25.6
MemoryOS14.4
Jetriever with top-five exchanges
Jetriever76.0

Table 6. All 16 published InMind Application configurations [1], followed by Jetriever’s top-five result. All rows use the same context-aware Application scoring criterion.

MemoryAgentBench

Original MAB baselineARTTLLRUSFOverall
Long-context agents
GPT-4o (128K)58.150.054.932.548.8
GPT-4o-mini (128K)49.248.646.225.042.2
Claude-3.7-Sonnet (200K)59.753.962.222.549.6
GPT-5-mini (400K)74.448.666.253.060.6
GPT-4.1-mini (1M)71.846.249.120.546.9
Gemini-2.0-Flash (1M)65.146.441.616.542.4
GPT-4o-mini (reference)49.248.646.225.042.3
Simple RAG
BM2560.544.535.625.541.5
Embedding RAG
Contriever33.942.929.812.529.8
text-embedding-3-small53.842.736.315.537.1
text-embedding-3-large54.644.337.316.038.0
Qwen3-Embedding-4B54.745.137.016.038.2
Structure-augmented RAG
RAPTOR36.835.927.97.527.0
GraphRAG40.924.819.98.023.4
MemoRAG34.545.130.014.030.9
HippoRAG-v265.135.836.229.541.6
Mem032.621.220.710.021.1
Cognee28.322.816.015.520.6
Zep37.537.516.25.024.0
Agentic memory
Self-RAG33.612.218.111.018.7
MemGPT34.340.822.415.528.3
MIRIX47.524.125.48.026.2
MIRIX (4.1-mini)63.035.740.511.537.7

Table 7. All 23 baselines in the original MAB main table [2]: four competency-group averages and Overall (%). AR = Accurate Retrieval; TTL = Test-Time Learning; LRU = Long-Range Understanding; SF = Selective Forgetting. These are not MAB-60 subset scores.

Published methodSH-QAMH-QALMEEventQAFC-SHFC-MHAvg.
GPT-4o-mini64.043.030.759.045.05.041.1
BM25-RAG66.056.045.374.648.03.048.8
Contriever22.031.015.766.818.07.026.8
text-embed-3-small60.044.048.363.028.03.041.1
text-embed-3-large54.044.050.370.028.04.041.7
Qwen3-Embed-4B57.047.043.371.429.03.041.8
RAPTOR29.038.034.345.814.01.027.0
GraphRAG47.047.035.034.414.02.029.9
MemoRAG29.033.020.056.021.07.027.7
HippoRAG 276.066.050.767.654.05.053.2
Mem025.032.036.037.518.02.025.1
Zep44.025.038.342.57.03.026.6
Self-RAG35.042.025.731.819.03.026.1
MemGPT41.038.032.026.228.03.028.0
MIRIX62.061.037.329.814.02.034.4
ReFind83.069.051.374.162.78.858.2

Table 8. Published full six-subtask comparison for Accurate Retrieval (SH-QA, MH-QA, LME, EventQA) and Selective Forgetting (FC-SH, FC-MH), reproduced from ReFind [3], with reused baselines from MAB [2]. Avg. is the unweighted six-subtask mean (%). This table uses the full task sets, not our 60-question subset.

References

  1. [1]R. Li, M. Du, B. Xu, and Z. Mao. Keep It InMind: Benchmarking the Implicit-Association Blind Spot in Agent Memory. arXiv:2607.24368, 2026. https://arxiv.org/abs/2607.24368
  2. [2]Y. Hu et al. Evaluating Memory in LLM Agents via Incremental Multi-Turn Interactions. arXiv:2507.05257, 2025. https://arxiv.org/abs/2507.05257
  3. [3]R. Li et al. When Your Agent Opens the Chat App: Agent-Controlled Search over Raw Chat Logs Rivals Structured Memory. arXiv:2608.12888, 2026. https://arxiv.org/abs/2608.12888
  4. [4]P. Wang et al. BM25 Wins at Scale: A Scaling Study of Retrieval-Augmented Generation Paradigms. arXiv:2607.26497, 2026. https://arxiv.org/abs/2607.26497
  5. [5]Z. Bao et al. Is Complex Agentic Memory Always Needed? Naive RAG Is Stronger Than We Thought. arXiv:2610.30755, 2026. https://arxiv.org/abs/2610.30755
  6. [6]ASPCA. Which Lilies Are Toxic to Pets? 2022. https://www.aspca.org/news/which-lilies-are-toxic-pets
  7. [7]M. Du et al. A-RAG: Scaling Agentic Retrieval-Augmented Generation via Hierarchical Retrieval Interfaces. arXiv:2602.03442, 2026. https://arxiv.org/abs/2602.03442
  8. [8]TypeSafe. AI primer: RLCD and calibrated decisions. 2026. https://docs.typesafe.ai/introduction/machine-learning-primer
  9. [9]D. Almeida. Introducing System One Models & Jev. TypeSafe AI, 2026. https://typesafe.ai/blog/introducing-system-one-models-and-jev
  10. [10]ConvAI Innovations. Laya Typed-Decisions: Model Card. https://huggingface.co/convaiinnovations/laya-typed-decisions
  11. [11]Qwen Team. Qwen3.5-9B: Model Card. https://huggingface.co/Qwen/Qwen3.5-9B
  12. [13]S. Robertson and H. Zaragoza. The Probabilistic Relevance Framework: BM25 and Beyond. 2009.
  13. [14]J. Devlin et al. BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding. arXiv:1810.04805, 2018. https://arxiv.org/abs/1810.04805
  14. [15]B. Warner et al. Smarter, Better, Faster, Longer: A Modern Bidirectional Encoder for Fast, Memory Efficient, and Long Context Finetuning and Inference. arXiv:2412.13663, 2024. https://arxiv.org/abs/2412.13663
  15. [16]Sentence Transformers. all-MiniLM-L6-v2: Model Card. https://huggingface.co/sentence-transformers/all-MiniLM-L6-v2DOI
  16. [17]Qwen Team. Qwen3.5-4B: Model Card. https://huggingface.co/Qwen/Qwen3.5-4B
  17. [18]OpenAI. GPT-5 Mini: Model Documentation. https://developers.openai.com/api/docs/models/gpt-5-mini
  18. [19]OpenAI. GPT-4o Mini: Model Documentation. https://developers.openai.com/api/docs/models/gpt-4o-mini
  19. [20]D. Wu et al. LongMemEval: Benchmarking Chat Assistants on Long-Term Interactive Memory. arXiv:2410.10813, 2024. https://arxiv.org/abs/2410.10813
  20. [21]R. Li et al. Towards Deep Conversational Recommendations. NeurIPS, 2018. https://arxiv.org/abs/1812.07617
  21. [22]W. Xu et al. A-MEM: Agentic Memory for LLM Agents. arXiv:2502.12110, 2025. https://arxiv.org/abs/2502.12110
  22. [23]B. J. Gutiérrez et al. From RAG to Memory: Non-Parametric Continual Learning for Large Language Models. arXiv:2502.14802, 2025. https://arxiv.org/abs/2502.14802
  23. [24]J. Kang et al. Memory OS of AI Agent. arXiv:2506.06326, 2025. https://arxiv.org/abs/2506.06326
  24. [25]Y. Wang and X. Chen. MIRIX: Multi-Agent Memory System for LLM-Based Agents. arXiv:2507.07957, 2025. https://arxiv.org/abs/2507.07957

Paper details

vlnc:W_thra3ec3
Version
v1 · latest
Published
Fresh — added to the Hub within the last 30 days
Added
License
CC BY 4.0
Usage

views · 0

downloads · 0

A view counts once per browser per day, a download each time the PDF is requested.

  • Computer Science

Not recorded for this paper

View statement
Looking for related papers…
Contents