WorksComputer Science
Where Did the Evidence Go? Diagnosing Proof-Set Loss in Long-Term Conversational Memory
Abstract
Long-term conversational question answering can require several separated dialogue turns that support an answer only together. We introduce EviCover, a retrieval approach designed to retain this complementary evidence. It converts questions into observable evidence requirements, checks requirement coverage using retrieved source identifiers, and selects turns jointly. We track annotated support sets - the benchmark-labeled dialogue turns associated with each question - through retrieval, reranking, and final answer-generation context. We evaluate the baseline memory pipeline on generated, human-verified LoCoMo conversations and human-written REALTALK conversations, and examine EviCover on REALTALK's 108 multi-hop questions. Compared with the hybrid lexical/dense baseline, support-turn recall among 60 candidate turns rises from 0.526 to 0.575, and recall in the final ten-turn context rises from 0.349 to 0.379. The evidence funnel identifies separate discovery and selection failures, while paired answer evaluation shows that greater evidence recall does not yet establish a correctness gain. This analysis locates where complementary evidence is lost and distinguishes evidence retrieval from its use in answer generation.
Introduction
To answer questions about earlier interactions, long-term assistants retrieve preferences, relationships, plans, and events from conversations distributed across sessions. A dialogue turn is an individual message from a participant. Recent benchmarks now test not only factual recall but also temporal reasoning, long-range understanding, and implicit constraints ([11, 8, 3, 9] (Wu et al. 2025)). These settings make the memory problem different from retrieving one independently relevant passage. A question such as “Does this person behave irresponsibly?” may be supported by one episode about health precautions and another about work obligations. Either episode is relevant, but neither alone justifies the conclusion.
Retrieval metrics such as mean reciprocal rank measure how early a supporting turn appears in a ranking. They do not show whether all complementary turns needed to answer a question survive reranking and enter the final context. Here, the reader is the language model that generates an answer from the selected dialogue turns. A turn can be retrieved initially and then discarded when candidates are scored independently or the context is truncated. Conversely, a retrieved turn may support an answer without being included in the benchmark annotations. We study these retrieval stages through an annotated proof set: all benchmark-labeled dialogue turns associated with a question. We evaluate a provenance-preserving memory pipeline on two types of conversation benchmarks: generated, human-verified LoCoMo and human-written REALTALK. The baseline configuration is selected using LoCoMo and held fixed for REALTALK evaluation. Multi-hop questions form the weakest REALTALK slice. For these questions, EviCover retrieves evidence for explicit requirements and jointly selects complementary turns. Figure 1 gives an overview of EviCover. We then compare retrieval trajectories, memory representations, and final-set selection using the same lexical and dense indexes, raw-turn source identifiers, and answer-generation model and prompts. We address two research questions:

Figure 1. EviCover overview: provenance-constrained proof-set retrieval. Requirement coverage is accepted only with a retrieved turn identifier. The pointwise and setwise top-10 contexts use the same discovered candidates and relevance scores; only final set selection changes. ER and PS are measured at every stored boundary.
RQ1: How does the baseline perform on generated and human-written conversations, and at which retrieval stages are annotated proof sets lost?
RQ2: Do requirement-guided discovery and setwise selection improve evidence delivery and answers enough to justify their cost?
We make two contributions. (1) We formalize an evidence funnel that reports support-turn recall and complete-proof-set survival at discovery, reranking, and reader-context boundaries, and augment it with turn-, session-, day-, and lexical-distance diagnostics. (2) We conduct a controlled evaluation of EviCover on REALTALK - observable requirement discovery, provenance-checked coverage, and set-level selection - alongside baseline results on LoCoMo, and jointly report evidence gains over flat hybrid retrieval and a pointwise selection control, answer evaluation, and inference cost.
Related Work
Prior work addresses how memories are organized, how supporting information is retrieved, and how memory is evaluated. We review these choices in that order.
Memory organization and recall. Structured memories aim to make related information easier to recover. A-MEM builds dynamic note networks (Xu et al., 2025); HippoRAG 2 integrates passages with graph-based associative retrieval (Jimenez Gutierrez et al., [4]); and Associa combines an event-centric graph with iterative recall (Zhang et al., 2025b). For document question answering, KG-augmented RAG combines reusable graph-community summaries with textual evidence (Karkout et al., [5]). APEX-MEM adds temporal structure and retrieval-time conflict resolution (Banerjee et al., [1]), while MemGAS associates and selects memories at multiple granularities (Xu et al., 2026). Memory-R1 learns memory operations and answer-side preselection (Yan et al., 2026), and MemCoRL alternates optimization of retrieval and utilization (Liu et al., [10]). These approaches motivate richer discovery and memory use. We examine whether complementary evidence found during discovery survives selection for the final context.
Multi-step retrieval and set selection. IRCoT interleaves retrieval with intermediate reasoning (Trivedi et al., [16]), while StepChain GraphRAG constructs query-local evidence paths (Ni et al., [12]). Both motivate searching beyond the initial question. ColBERT’s token-level late interaction provides finer relevance matching than one vector per turn (Khattab and Zaharia, [6]), but relevance scoring and joint coverage are distinct objectives. Setwise ranking compares document groups (Zhuang et al., 2024), and setwise insertion edits an existing ranking (Podolak et al., [14]). Most directly, SetR derives query information requirements and selects passages that cover them jointly (Lee et al., [7]). We apply this coverage objective to long-term conversations, where supporting messages may occur in different sessions. Coverage decisions must cite retrieved turn identifiers, and the evidence funnel tracks losses during discovery, independent scoring, and final context selection.
Evaluation of long-term memory. Benchmarks expose different demands on conversational memory. LoCoMo tests temporal and causal QA on generated, human-verified conversations (Maharana et al., [11]); REALTALK uses messages exchanged by real people over 21 days (Lee et al., [8]). This contrast motivates evaluating our baseline on both datasets. LongMemEval analyzes indexing, retrieval, and reading (Wu et al., 2025), while MemoryAgentBench tests retrieval, test-time learning, long-range understanding, and selective forgetting [3]. LoCoMo-Plus targets implicit constraints under cue–trigger semantic disconnect, which surface-form metrics can miss [9]. Our evidence funnel complements this evaluation by measuring whether the full annotated support set survives each retrieval stage.
Task and Evaluation
Problem setting
For user or conversation , let be the ordered raw dialogue turns. Each is a whole participant message with a source identifier. A benchmark question has reference answer and an annotated support set . Each item in is a whole turn labeled as supporting ; we call this collection the annotated proof set. A retrieval pipeline collects turns in across search rounds, keeping one copy per source identifier. It scores these turns independently against and retains a ranked candidate set . The final context , with , contains the selected turns supplied to the reader for answer generation. The reader returns answer and cited source identifiers . For the set operations below, turns are represented by their source identifiers.
At any stage , we report micro-averaged gold-turn evidence recall
and macro-averaged complete-proof-set rate
ER is the fraction of annotated support turns retained across questions; PS is the fraction of questions for which every annotated supporting turn is retained.
Retrieval and answer metrics
Recall@5 is the macro-average of , where contains the first five turns in the ranked final context. Mean reciprocal rank (MRR) averages the reciprocal rank of the first retrieved member of in the final context, with zero if none is retrieved. Under our reporting convention, a question with no annotated support contributes zero to both metrics. Unlike ER, Recall@5 is macro-averaged per question and truncated at five.
For answers, normalization lowercases text, collapses whitespace, and removes punctuation. Exact match (EM) is equality after this normalization. Token F1 is the harmonic mean of multiset-token precision and recall against . We use both as lexical agreement diagnostics, not sufficient measures of semantic correctness. LoCoMo and REALTALK report partial-match F1, while LongMemEval uses LLM-based evaluation to accommodate equivalent answer formulations [11, 8] (Wu et al. 2025). Our annotation-based abstention correctness is one when the system abstains exactly when ; missing annotations alone do not establish that a question is unanswerable. A citation is a dialogue-turn source identifier returned with the answer to indicate its supporting evidence. Citation validity requires and . Per-question citation precision and recall measure identifier overlap with the annotated support set:
with zero for an empty denominator. We macro-average these per-question scores and binary indicators over the evaluated questions. Our mechanically citation-grounded answer indicator is one only when is nonempty and not an abstention, all citations are valid, and . This is a provenance check, not a semantic entailment claim.
Lexical metrics penalize equivalents such as “twice” and “2 times.” We therefore complement them with LLM-as-a-judge evaluation using Qwen3-32B, which sees the question, reference, generated answer, retrieved context, and source identifiers. It assigns correct, partially correct, incorrect, correct abstention, or wrong abstention, and separately marks whether the answer is supported by the shown context. We report judge accepted as the combined rate of correct answers and judge-accepted abstentions, and report those labels separately when comparing reader prompts. An accepted abstention is a judge decision, not proof that the full conversation is unanswerable. Following known judge bias concerns (Zheng et al., 2023), these labels complement rather than replace the deterministic metrics. All principal intervals use 20,000 paired non-parametric bootstrap resamples of the ten conversations.
Method
Overview
EviCover changes the multi-hop branch of the frozen retrieval pipeline. It has three explicit stages. First, a planner converts the question into up to four observable evidence requirements and retrieves for each uncovered requirement. Second, a provenance audit may mark a requirement covered only by citing an identifier from the retrieved candidates. Third, the same pointwise relevance model used by the baseline supplies a prior, while a setwise selector builds ten complementary turns rather than returning the ten highest independent scores. Figure 1 shows the data passed between stages; gold support identifiers enter only after generation for evaluation.
Baseline retrieval pipeline
The memory store preserves each raw turn and its source identifier. BM25 is the lexical channel: it ranks turns by exact query-term evidence with document-length and term-frequency normalization [15]. The semantic channel embeds questions and turns into 4,096-dimensional vectors with E5-Mistral-7B-Instruct, a text-embedding model trained to place semantically related queries and passages near one another (Wang et al., 2024). We use cosine similarity between vectors for dense retrieval; “E5-Mistral” below denotes this concrete model.
At write time, each turn is linked to at most ten active turns that share an entity identifier or an identical nonempty subject; matching predicates, subjects, and entity-only matches yield ACCUMULATES, EXTENDS, and SPECIFIES edges, respectively. For factual and temporal questions, the ten strongest lexical/dense seeds start a depth-two walk over these edges (and temporal TRANSFORMS edges), with at most 12 returned neighbors per seed. For multi-hop questions, deterministic text rules produce at most six query facets and add dialogue neighbors within turns. The benchmark-provided query type chooses the branch. Weighted reciprocal-rank fusion (RRF) combines lexical, dense, and graph rankings as
with , , and when the graph channel is present [2]. The best 60 fused turns are then scored independently by the 0.6B Qwen3-Reranker model. It receives one pair at a time and converts its final yes/no logits to a relevance probability (Zhang et al., 2025a). Because no candidate is scored in the context of another candidate, we call this a pointwise relevance scorer; it is the only learned relevance model in the controlled baseline and requirement-guided comparisons.
Provenance-constrained requirement discovery
Requirement-guided discovery starts with a Qwen3-32B planner that maps to one to four observable, independently verifiable requirements . For example, an evaluative question about irresponsibility can yield one requirement for behavior affecting others and one for an unfulfilled obligation; it does not search for the abstract adjective alone. Each round retrieves up to 60 BM25 and 60 E5-Mistral results for one uncovered requirement and merges them by RRF. Candidates remain in across at most four rounds.
The planner then audits at most 16 shown candidates. A requirement may be marked covered only with a source identifier in ; code rejects absent or malformed identifiers. An uncovered requirement produces the next focused query. Retrieval stops when all requirements are covered, four rounds are exhausted, or a duplicate follow-up cannot be replaced. The same 0.6B Qwen3 pointwise model scores the accumulated candidates once. We retain its top 60 for a capacity-matched comparison with the baseline and expose up to its top 120 to the operational set selector. If checklist parsing fails, the system performs one search with the original question; if a later audit cannot be parsed, it stops expansion while retaining all candidates accumulated so far.
Requirement-seeded setwise selection
The pointwise control returns the ten highest relevance scores. Our selector instead seeds a ten-item set with the highest-ranked cited turn for each covered requirement, then fills empty slots by the pointwise prior. The first five positions of this seeded set are protected; they are not necessarily the original pointwise top five. Remaining candidates from the top 120 arrive in batches of eight. Qwen3-32B may replace at most one item per batch and three items overall, and only when the challenger directly adds or more strongly supports a listed requirement while the removed item is redundant. Invalid JSON, unknown identifiers, or an unsupported justification leave the set unchanged. Pointwise and setwise contexts therefore share , the same top-120 candidate set, pointwise scores, and ; only the final set utility differs.
Figure 1 shows the complete intervention. Gold support turns are used only after retrieval and generation for evaluation.
Experimental Setup
Data. The evaluated LoCoMo release contains ten generated, human-verified conversations: 5,882 turns (588.2 per conversation), up to 32 sessions, and 1,986 QA examples ([11]). REALTALK contains ten 21-day conversations between people with 8,944 turns and 728 memory-probing questions: 301 factual, 319 temporal, and 108 multi-hop ([8]). The multi-hop slice has 327 annotated support turns.
Models and budgets. Both controlled multi-hop systems use the same BM25 index, E5-Mistral-7B-Instruct embeddings, RRF weights, 0.6B Qwen3 pointwise scorer (maximum input length 4,096), and top-10 reader context. The flat baseline stores 60 pointwise-ranked candidates. Requirement-guided discovery reports a matched top-60 candidate set and an operational top-120 prefilter; the latter is shared by its pointwise and setwise top-10 variants. The requirement planner and setwise selector use Qwen3-32B with temperature zero and thinking disabled ((Yang et al., 2025)). The 0.6B pointwise scorer is non-generative and runs in evaluation mode without gradients. The frozen reader is gpt-oss-120b at temperature zero ([13]). The full baseline evaluation allows 6,000 context tokens; saved multi-hop contexts are replayed with a 4,000-token cap and no new retrieval. Qwen3-32B judge calls also use temperature zero.
Generation and citations. Query-type-specific prompts ask for concise factual, temporal, or multi-hop answers. Citation repair is gold-blind and can choose only a packed turn. It preserves an already valid citation; otherwise, for packed turn at rank , it computes
where removes common weak tokens, then
cites up to two source identifiers from the highest-scoring turn. It cannot introduce unretrieved evidence.
Protocol and reproducibility. Principal tables use the frozen baseline and EviCover runs; three additional full retrieval replications measure stability under the same configuration. Temperature zero removes intentional sampling, not all remote-serving nondeterminism. Principal multi-hop comparisons use all 108 questions. A repeatedly inspected fixed first-30 slice rejects expensive alternatives and is diagnostic, not held-out. The full-108 analysis includes these development questions; it is an in-corpus audit, not an independent held-out confirmation. Each run stores configuration, examples, retrieval traces, token use, packed contexts, metrics, and resumable state under a UUID. Paper tables and confidence intervals are regenerated from immutable JSONL artifacts.
Results
RQ1: Dataset evaluation and stage-wise proof-set loss
Table 1 shows that revising answer-generation settings on LoCoMo improves token F1 and citation-grounded answers while Recall@5 remains nearly constant. With the revised configuration held fixed, REALTALK results are lower on all reported answer and citation measures. The two datasets have different question distributions, so these scores do not establish their relative difficulty.
| Run | R@5 | F1 | CP | CR | Cit.-Gr. |
| LoCoMo initial | .772 | .316 | .463 | .448 | .493 |
| LoCoMo revised | .771 | .394 | .558 | .535 | .593 |
| REALTALK baseline | .579 | .291 | .477 | .374 | .511 |
Table 1. Baseline evaluation on both datasets. CP/CR are citation precision/recall; Cit.-Gr. is the mechanical citation-grounded answer rate defined in § 3.2.
The LLM judge assigns higher semantic scores than lexical metrics suggest: on all 728 questions, 55.9% are accepted (including accepted abstentions), 80.2% are accepted or partial, and 71.6% are supported by the shown context, compared with 6.7% EM. Table 2 nevertheless identifies multi-hop as the weakest slice for both retrieval and judge support. Temporal F1 is additionally affected by equivalent date formats and concise surface forms.
| Type () | R@5 | F1 | J-C/P | J-Sup. | Cit.-Gr. |
| Factual (301) | .391 | .381 | .847 | .787 | .475 |
| Temporal (319) | .814 | .239 | .777 | .702 | .611 |
| Multi-hop (108) | .409 | .191 | .750 | .556 | .315 |
Table 2. REALTALK baseline by query type. J-C/P is judge correct-or-partial; J-Sup. is judge-supported; Cit.-Gr. is the mechanical metric.
Temporal and semantic distance. For each annotated support identifier, we measure its distance tance from the final released turn of the same conversation. This is a lower bound on distance to the memory-probing question if that question was asked later. Table 3 confirms that the task is not merely short-range semantic retrieval: the median support turn is 12 days and 380 turn positions old, and 72.2% of all support turns are at least seven days old. At the question level, the oldest required support has a median age of 16.5 days.
| Distance unit | Median | P90 | Maximum |
| Turn positions | 380 | 989 | 1574 |
| Sessions | 12 | 19 | 24 |
| Days | 12 | 21 | 23 |
Table 3. Distance from each of 327 support turns to the final observed turn.
Temporal distance is not the only difficulty. Among the 294 support turns present as text in the release, 63.3% have zero content-word overlap with their question. We lowercase, retain alphanumeric tokens longer than one character, and remove the fixed English stopword list supplied with the diagnostic code. This is a surface-form proxy, not a semantic similarity measure. The remaining 33 annotated identifiers refer to media-only turns absent from the textual export; they are retained in temporal counts but excluded from lexical-overlap statistics. Recall by age is not monotonic: baseline final ER is .324 for 0–2 days, .407 for 3–6, .322 for 7–13, and .349 for 14+ days. The benchmark therefore combines genuine long-range memory with semantic mismatch rather than isolating either factor.
Table 4 and Figure 2 trace all 327 multi-hop support turns. The flat baseline retains 172 in its 60 pointwise-ranked candidates but only 114 in final context. Requirement-guided retrieval accumulates 217 and retains 188 in its matched top-60 candidate set: +0.049 ER with a 95% conversation-cluster bootstrap CI of [0.028, 0.067]. Allowing the operational prefilter to contain 120 candidates retains 207 turns; this number measures available evidence for set selection and is not a capacity-matched baseline comparison.
| System | Stage | Hits/327 | ER | Full sets |
| Flat hybrid | top-60 | 172 | .526 | 50/108 |
| Flat hybrid | final top-10 | 114 | .349 | 36/108 |
| Req. discovery | accumulated | 217 | .664 | 63/108 |
| Req. discovery | top-60 | 188 | .575 | 55/108 |
| Req. discovery | top-120 | 207 | .633 | 60/108 |
| Pointwise | final top-10 | 116 | .355 | 36/108 |
| Setwise | final top-10 | 124 | .379 | 38/108 |
Table 4. Evidence funnel on all 108 multi-hop questions. “Req. discovery” denotes requirement-guided retrieval. Top-60/120 rows report ranked candidates.

Figure 2. Annotated support-turn evidence recall. Candidate bars compare matched top-60 budgets; final-context bars compare top-10 outputs.
Setwise final context retains 124 turns, +0.031 over the complete flat pipeline (CI [0.007, 0.064]).
Setwise selection improves ER over the matched pointwise top-10 from .355 to .379 (+0.024, CI [0.009, 0.047]), but full proof sets increase only from 36 to 38; the +0.019 rate difference has CI [−0.018, 0.055]. An oracle top-10 from the same candidate set reaches .624 ER and preserves all 60 annotated support sets available in full. The 108 questions decompose into 45 whose complete set is never discovered, three lost during independent scoring, 22 complete among ranked candidates but incomplete after top-10 selection, and 38 complete in reader context.
Exploratory alternatives on the fixed-30 slice and additional memory representations are reported in Appendix § 9.
RQ2: Evidence gains do not yet establish answer gains
We replay answers over the 108 saved setwise contexts with the same reader, query-type prompt family, temperature, and citation policy. No retrieval is repeated. Table 5 shows that judge accepted-or-partial and supported rates each rise by .028, but both confidence intervals include zero. Mechanical F1 and citation-grounded rate decline slightly, also with intervals crossing zero. Correct labels change from 35 to 32, correct abstentions from 31 to 35, partial answers from 15 to 17, and wrong abstentions from 9 to 6. The new context therefore changes answer behavior without a statistically clear gain in full correctness.
| Metric | Flat | Req.+set | 95% CI | |
| Judge accepted | .611 | .620 | +.009 | [-.058,.074] |
| Accepted/partial | .750 | .778 | +.028 | [-.020,.083] |
| Judge supported | .556 | .583 | +.028 | [-.037,.080] |
| Judge score | 3.083 | 3.028 | -.056 | [-.180,.065] |
| Token F1 | .191 | .183 | -.009 | [-.023,.006] |
| Citation-grounded | .315 | .306 | -.009 | [-.063,.057] |
Table 5. Paired answer replay on 108 multi-hop questions. CIs are for .
Complete evidence remains associated with answer quality: when the setwise context contains the full annotated proof set, judge correct-or-partial is .895 and support is .737, versus .714 and .500 otherwise. This observational gap is not causal because easier questions may also be easier to retrieve, but it supports reporting complete-set survival rather than only the first relevant hit.
Reader-only intervention on frozen contexts
We next change only the multi-hop reader instruction on the same 108 saved EviCover contexts. The original prompt favors minimal answer spans and omits reasoning; the revised prompt permits one compact, explicitly supported reason, requires citations for its evidence, and prohibits unsupported elaboration. The model, temperature, 4,000-token context cap, and citation repair remain unchanged. This is a post-hoc reader ablation, not an independent test set or an additional retrieval improvement.
Table 6 reports the completed paired replay; all 108 judge responses parse successfully. Substantively correct labels increase from 32 to 36, but accepted abstentions decrease from 35 to 24 and wrong abstentions increase from 6 to 10. Partial and incorrect labels change from 17 to 21 and 18 to 17, respectively. Consequently, the overall accepted rate falls from .620 to .556 despite the higher mean score and citation-grounded rate. Judge-supported answers change only from 63 to 65. The accepted-rate decrease has a conversation-cluster CI of [−.131, −.009], while the citation-grounded increase has CI [.010, .126]; these exploratory in-
| Metric | Original | Revised | 95% CI | |
| Judge accepted | .620 | .556 | -.065 | [-.131,-.009] |
| Accepted/partial | .778 | .750 | -.028 | [-.095,.042] |
| Judge supported | .583 | .602 | +.019 | [-.029,.071] |
| Judge score | 3.028 | 3.157 | +.130 | [-.020,.284] |
| Token F1 | .183 | .147 | -.036 | [-.111,.031] |
| Citation-grounded | .306 | .370 | +.065 | [.010,.126] |
Table 6. Reader-only paired ablation on identical EviCover contexts (108 questions). Accepted includes judge-accepted abstentions.
tervals are not adjusted for multiple comparisons. Better citation overlap therefore does not establish better semantic correctness; we retain the original reader for the principal EviCover comparison.
Temperature-zero repeatability
Three independent replications use the same 108 questions, indexes, budgets, prompts, and decoding configuration. Across pairs, the final selected identifiers match exactly on 91.7–99.1% of questions; mean evidence-set Jaccard similarity is .971–.998. Recall@5 ranges from .4003 to .4020 (spread .0017), MRR from .4664 to .4667, and nDCG@5 from .3814 to .3838. Thus aggregate retrieval performance is stable in these replications, but the generated search trajectories and individual contexts are not identical. These are retrieval repetitions, not repeated answer generation or independent draws of new conversations.
Efficiency audit
Table 7 reports exact saved-run cost on the same 108 questions. Mean retrieval latency increases by 2.28×. EviCover adds 17.47 Qwen3-32B calls and 24.59K planner/selector tokens per question, for 2.66M tokens in total; 81.2% are spent by final set selection. Relative to the flat final context, this corresponds to 265,568 tokens per additional retained support turn and 1,327,838 per additional complete proof set. These are inference counts, not monetary or energy estimates, because both models ran on private endpoints without hardware accounting; the wall-clock ratio is observational rather than hardware-normalized.
| Measure | Flat hybrid | EviCover |
| Mean latency (s/query) | 18.28 | 41.62 |
| Pointwise candidates/query | 60 | 118.45 |
| Generative calls/query | 0 | 17.47 |
| Generative tokens/query | 0 | 24.59K |
| Final ER | 0.349 | 0.379 |
| Complete proof sets | 36 | 38 |
Table 7. Retrieval utility and cost on the same 108 multi-hop questions. LLM calls/tokens count additional generative Qwen3-32B retrieval work.
Discussion
Item relevance and set coverage are different objectives. The 22 questions with complete support among ranked candidates but incomplete support in the final context quantify the mismatch. A point-wise ranker can repeatedly select turns about the same aspect, whereas proof-set selection values the marginal coverage of a currently missing requirement. The oracle gap shows that a learned set utility remains a larger opportunity than further hand-tuning RRF weights.
Human dialogue changes the search target. LoCoMo multi-hop questions often combine explicit facts across time. REALTALK also asks for judgments and preferences inferred from behavior; the abstract term in the question may be absent from its support turns. Observable requirements improve discovery under this lexical mismatch, but Qwen3-generated requirements remain imperfect and expensive.
Reader utilization is a separate bottleneck. Better proof-set recall mostly changes abstention and partial-credit labels rather than full correctness. Future models should jointly score requirement coverage, provenance, redundancy, and whether the reader can compose an answer from the selected set.
The intervention is diagnostic, not yet operationally efficient. Most added computation is spent repeatedly deciding whether a candidate adds marginal coverage, yet only two additional complete proof sets reach the reader and answer intervals cross zero. EviCover is therefore evidence that pointwise selection is a real bottleneck, not evidence that this particular 32B implementation should be a universal production default. A practical successor would distill set utility into a smaller scorer or invoke it only when an inexpensive coverage test detects a likely incomplete proof set.
Conclusion
Stage-wise analysis reveals two separable failures in long-term conversational memory: complete proof sets are often never discovered, and complementary turns that are discovered are discarded before generation. Provenance-constrained requirement discovery improves the REALTALK candidate set at matched capacity, and setwise selection preserves a small portion of that gain, but at substantial inference cost. Complete-set and answer-level improvements remain modest and uncertain. Reporting temporal and lexical distance together with ER and PS through the retrieval funnel gives a more actionable account than a single top- or answer score: better memory requires complete evidence delivery, an efficient set objective, and a reader able to use the delivered evidence.
Limitations
The principal comparison has 108 questions nested in ten conversations. Conversation-cluster bootstrap respects that dependence, but ten clusters yield wide intervals. The fixed-30 slice was repeatedly inspected, and exploratory alternatives and the later reader-prompt change are descriptive rather than confirmatory. Intervals are conditional on the selected configurations, not adjusted for method selection or multiple exploratory comparisons. Retrieval replications measure inference stability on the same data, not generalization or repeatability of the answer evaluation. We use one reader model and one uncalibrated LLM judge. The judge sees mechanical scores and gold identifiers, is not blinded to those signals, and uses the same Qwen3-32B model family as discovery and selection. Its accepted-abstention labels must not be equated with corpus-level unanswerability; no human adjudication of those labels was performed.
LoCoMo and REALTALK have different question distributions, so cross-dataset scores do not rank intrinsic dataset difficulty. Temporal lag is measured to the final released turn rather than a question timestamp; it does not measure the number of turns given to the reader. Thirty-three media-only support identifiers lack released text. Annotated support may also be incomplete, especially for inferential questions; PS measures survival of the released annotation, not every valid proof. The controlled alternatives use implementations inspired by published approaches, not their trained checkpoints or full published training recipes, and do not establish leaderboard comparisons with those systems.
This study evaluates memory-question answering, not personalization: it does not test whether an assistant adapts recommendations, style, or actions to an individual user. Long-term memory can support personalization, but the present results should not be interpreted as evidence of personalized behavior. Hardware-normalized latency, monetary cost, and energy use were not measured.
Ethical considerations
REALTALK contains conversations between people. We use only the released benchmark, add no human-subject data, preserve source provenance, and do not quote source conversations in the manuscript. Illustrative examples are schematic rather than verbatim dialogue. A deployed memory system would require access control, deletion, retention limits, and user consent. We do not redistribute conversation text as part of the manuscript package; access remains governed by the source datasets’ terms.
Acknowledgment
This work was supported by the Ministry of Economic Development of the Russian Federation (IGK 000000C313925P4C0002), agreement No 139-152025-010.
Exploratory alternatives
Table 8 summarizes the fixed-30 diagnostic slice (82 support turns). Free iterative search, query-local graph chaining, breadth-then-depth expansion, and a restricted executable search program broaden discovery, but none exceeds requirement-guided support retention after the shared pointwise reranker. The repeated gap between accumulated and final ER motivated the setwise intervention rather than another query-expansion prompt.
| Discovery method | R@5 | Accum. ER | Ranked ER | Final ER |
| Flat lexical+dense | .452 | – | .524 | .378 |
| Free iterative | .397 | .646 | .549 | .366 |
| Requirement-guided | .424 | .683 | .573 | .378 |
| Query-local graph chain | .374 | .659 | .512 | .329 |
| Breadth-then-depth | .358 | .671 | .549 | .329 |
| Restricted search program | .386 | .634 | .537 | .341 |
Table 8. Exploratory fixed-30 multi-hop diagnostics (82 support turns).
Table 8: Exploratory fixed-30 multi-hop diagnostics (82 support turns).
Alternative representations expose the same boundary. A proposition index extracts 14,563 short facts from 8,925 processed turns and raises full multi-hop candidate hits from 172 to 188, but final hits change only from 114 to 115; a proposition-aware selector reduces them to 91. A provenance-linked index of 2,798 episodic events finds seven support turns absent from the baseline candidate set on a 30-question behavioral subset, but quota fusion does not retain them reliably. A ready-made ColBERT-style model matches baseline candidate ER (.524) yet lowers final ER (.341 versus .378). The proposition and event controls show that richer representations can surface complementary turns, but their gains are not automatically preserved by the shared itemwise objective.
References
- [1]Pratyay Banerjee, Masud Moshtaghi, Shivashankar Subramanian, Amita Misra, and Ankit Chadha. 2026. APEX-MEM: Agentic semi-structured memory with temporal reasoning for long-term conversational AI. In Proceedings of the 64th Annual Meeting of the Association for Computational Linguistics, pages 16470–16489.
- [2]Gordon V. Cormack, Charles L. A. Clarke, and Stefan Buettcher. 2009. Reciprocal rank fusion outperforms condorcet and individual rank learning methods. In Proceedings of the 32nd International ACM SIGIR Conference on Research and Development in Information Retrieval, pages 758–759.DOI
- [3]Yuanzhe Hu, Yu Wang, and Julian McAuley. 2026. Evaluating memory in LLM agents via incremental multiturn interactions. In International Conference on Learning Representations.arxiv.org/abs/2507.05257
- [4]Bernal Jimenez Gutierrez, Yiheng Shu, Weijian Qi, Sizhe Zhou, and Yu Su. 2025. From RAG to memory: Non-parametric continual learning for large language models. arXiv preprint arXiv:2502.14802.
- [5]Maha Karkout, Maria Andreevna Khodorchеnko, Nikolay Alekseevich Butakov, and Denis Nasonov. 2026. Efficient KG-augmented RAG with reusable graph community summaries. In Proceedings of the Knowledge Graphs and Large Language Models Workshop (KG-LLM) at LREC26, pages 110–119, Palma, Mallorca, Spain. European Language Resources Association (ELRA).DOI
- [6]Omar Khattab and Matei Zaharia. 2020. ColBERT: Efficient and effective passage search via contextualized late interaction over BERT. In Proceedings of the 43rd International ACM SIGIR Conference on Research and Development in Information Retrieval.arxiv.org/abs/2004.12832
- [7]Dahyun Lee, Yongrae Jo, Haeju Park, and Moontae Lee. 2025a. Shifting from ranking to set selection for retrieval augmented generation. In Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics, pages 17606–17619.
- [8]Dong-Ho Lee, Adyasha Maharana, Jay Pujara, Xiang Ren, and Francesco Barbieri. 2025b. REALTALK: A 21-day real-world dataset for long-term conversation. arXiv preprint arXiv:2502.13270.arxiv.org/abs/2502.13270
- [9]Yifei Li, Weidong Guo, Lingling Zhang, Rongman Xu, Muye Huang, Hui Liu, Lijiao Xu, Yu Xu, and Jun Liu. 2026. LoCoMo-Plus: Beyond-factual cognitive memory evaluation frameworks for LLM agents. In Proceedings of the 64th Annual Meeting of the Association for Computational Linguistics, pages 25085–25100.
- [10]Yuewen Liu, Peng Xu, Muxi Diao, Anyi Zhang, Yang Li, and Yutong Zhang. 2026. MemCorRL: Alternating co-optimization of memory retrieval and utilization via collaborative reinforcement learning. In Proceedings of the 64th Annual Meeting of the Association for Computational Linguistics, pages 38912–38924.
- [11]Adyasha Maharana, Dong-Ho Lee, Sergey Tulyakov, Mohit Bansal, Francesco Barbieri, and Yuwei Fang. 2024. Evaluating very long-term conversational memory of LLM agents. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 13851–13870.
- [12]Tengjun Ni, Xin Yuan, Shenghong Li, Kai Wu, Ren Ping Liu, Wei Ni, and Wenjie Zhang. 2025. StepChain GraphRAG: Reasoning over knowledge graphs for multi-hop question answering. arXiv preprint arXiv:2510.02827.arxiv.org/abs/2510.02827
- [13]OpenAI. 2025. gpt-oss-120b & gpt-oss-20b Model Card. https://openai.com/index/gpt-oss-model-card/. Accessed: 2026-08-02.
- [14]Jakub Podolak, Leon Peric, Mina Janicijevic, and Roxana Petcu. 2025. Beyond reproducibility: Advancing zero-shot LLM reranking efficiency with setwise insertion. arXiv preprint arXiv:2504.10509.arxiv.org/abs/2504.10509
- [15]Stephen Robertson and Hugo Zaragoza. 2009. The probabilistic relevance framework: BM25 and beyond. Foundations and Trends in Information Retrieval, 3(4):333–389.
- [16]Harsh Trivedi, Niranjan Balasubramanian, Tushar Khot, and Ashish Sabharwal. 2023. Interleaving retrieval with chain-of-thought reasoning for knowledge-intensive multi-step questions. In Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics.arxiv.org/abs/2212.10509