betaFeedback

WorksComputer Science

Where Did the Evidence Go? Diagnosing Proof-Set Loss in Long-Term Conversational Memory

Abstract

Long-term conversational question answering can require several separated dialogue turns that support an answer only together. We introduce EviCover, a retrieval approach designed to retain this complementary evidence. It converts questions into observable evidence requirements, checks requirement coverage using retrieved source identifiers, and selects turns jointly. We track annotated support sets - the benchmark-labeled dialogue turns associated with each question - through retrieval, reranking, and final answer-generation context. We evaluate the baseline memory pipeline on generated, human-verified LoCoMo conversations and human-written REALTALK conversations, and examine EviCover on REALTALK's 108 multi-hop questions. Compared with the hybrid lexical/dense baseline, support-turn recall among 60 candidate turns rises from 0.526 to 0.575, and recall in the final ten-turn context rises from 0.349 to 0.379. The evidence funnel identifies separate discovery and selection failures, while paired answer evaluation shows that greater evidence recall does not yet establish a correctness gain. This analysis locates where complementary evidence is lost and distinguishes evidence retrieval from its use in answer generation.

Introduction

To answer questions about earlier interactions, long-term assistants retrieve preferences, relationships, plans, and events from conversations distributed across sessions. A dialogue turn is an individual message from a participant. Recent benchmarks now test not only factual recall but also temporal reasoning, long-range understanding, and implicit constraints ([11, 8, 3, 9] (Wu et al. 2025)). These settings make the memory problem different from retrieving one independently relevant passage. A question such as “Does this person behave irresponsibly?” may be supported by one episode about health precautions and another about work obligations. Either episode is relevant, but neither alone justifies the conclusion.

Retrieval metrics such as mean reciprocal rank measure how early a supporting turn appears in a ranking. They do not show whether all complementary turns needed to answer a question survive reranking and enter the final context. Here, the reader is the language model that generates an answer from the selected dialogue turns. A turn can be retrieved initially and then discarded when candidates are scored independently or the context is truncated. Conversely, a retrieved turn may support an answer without being included in the benchmark annotations. We study these retrieval stages through an annotated proof set: all benchmark-labeled dialogue turns associated with a question. We evaluate a provenance-preserving memory pipeline on two types of conversation benchmarks: generated, human-verified LoCoMo and human-written REALTALK. The baseline configuration is selected using LoCoMo and held fixed for REALTALK evaluation. Multi-hop questions form the weakest REALTALK slice. For these questions, EviCover retrieves evidence for explicit requirements and jointly selects complementary turns. Figure 1 gives an overview of EviCover. We then compare retrieval trajectories, memory representations, and final-set selection using the same lexical and dense indexes, raw-turn source identifiers, and answer-generation model and prompts. We address two research questions:

EviCover overview diagram showing Question $q$ and Qwen3-32B observable requirements leading to BM25 + E5-Mistral search for uncovered requirements, Qwen3-32B coverage audit, unique discovered turns $A_q$, Qwen3-Reranker-0.6B scores, pointwise and requirement-seeded setwise top-10 contexts, and fixed reader

Figure 1. EviCover overview: provenance-constrained proof-set retrieval. Requirement coverage is accepted only with a retrieved turn identifier. The pointwise and setwise top-10 contexts use the same discovered candidates and relevance scores; only final set selection changes. ER and PS are measured at every stored boundary.

  • RQ1: How does the baseline perform on generated and human-written conversations, and at which retrieval stages are annotated proof sets lost?

  • RQ2: Do requirement-guided discovery and setwise selection improve evidence delivery and answers enough to justify their cost?

We make two contributions. (1) We formalize an evidence funnel that reports support-turn recall and complete-proof-set survival at discovery, reranking, and reader-context boundaries, and augment it with turn-, session-, day-, and lexical-distance diagnostics. (2) We conduct a controlled evaluation of EviCover on REALTALK - observable requirement discovery, provenance-checked coverage, and set-level selection - alongside baseline results on LoCoMo, and jointly report evidence gains over flat hybrid retrieval and a pointwise selection control, answer evaluation, and inference cost.

Related Work

Prior work addresses how memories are organized, how supporting information is retrieved, and how memory is evaluated. We review these choices in that order.

Memory organization and recall. Structured memories aim to make related information easier to recover. A-MEM builds dynamic note networks (Xu et al., 2025); HippoRAG 2 integrates passages with graph-based associative retrieval (Jimenez Gutierrez et al., [4]); and Associa combines an event-centric graph with iterative recall (Zhang et al., 2025b). For document question answering, KG-augmented RAG combines reusable graph-community summaries with textual evidence (Karkout et al., [5]). APEX-MEM adds temporal structure and retrieval-time conflict resolution (Banerjee et al., [1]), while MemGAS associates and selects memories at multiple granularities (Xu et al., 2026). Memory-R1 learns memory operations and answer-side preselection (Yan et al., 2026), and MemCoRL alternates optimization of retrieval and utilization (Liu et al., [10]). These approaches motivate richer discovery and memory use. We examine whether complementary evidence found during discovery survives selection for the final context.

Multi-step retrieval and set selection. IRCoT interleaves retrieval with intermediate reasoning (Trivedi et al., [16]), while StepChain GraphRAG constructs query-local evidence paths (Ni et al., [12]). Both motivate searching beyond the initial question. ColBERT’s token-level late interaction provides finer relevance matching than one vector per turn (Khattab and Zaharia, [6]), but relevance scoring and joint coverage are distinct objectives. Setwise ranking compares document groups (Zhuang et al., 2024), and setwise insertion edits an existing ranking (Podolak et al., [14]). Most directly, SetR derives query information requirements and selects passages that cover them jointly (Lee et al., [7]). We apply this coverage objective to long-term conversations, where supporting messages may occur in different sessions. Coverage decisions must cite retrieved turn identifiers, and the evidence funnel tracks losses during discovery, independent scoring, and final context selection.

Evaluation of long-term memory. Benchmarks expose different demands on conversational memory. LoCoMo tests temporal and causal QA on generated, human-verified conversations (Maharana et al., [11]); REALTALK uses messages exchanged by real people over 21 days (Lee et al., [8]). This contrast motivates evaluating our baseline on both datasets. LongMemEval analyzes indexing, retrieval, and reading (Wu et al., 2025), while MemoryAgentBench tests retrieval, test-time learning, long-range understanding, and selective forgetting [3]. LoCoMo-Plus targets implicit constraints under cue–trigger semantic disconnect, which surface-form metrics can miss [9]. Our evidence funnel complements this evaluation by measuring whether the full annotated support set survives each retrieval stage.

Task and Evaluation

Problem setting

For user or conversation uu, let Hu=(t1,…,tn)H_u = (t_1,\ldots,t_n) be the ordered raw dialogue turns. Each tit_i is a whole participant message with a source identifier. A benchmark question qq has reference answer aq∗a_q^* and an annotated support set Gq⊆HuG_q \subseteq H_u. Each item in GqG_q is a whole turn labeled as supporting qq; we call this collection the annotated proof set. A retrieval pipeline collects turns in AqA_q across search rounds, keeping one copy per source identifier. It scores these turns independently against qq and retains a ranked candidate set Pq⊆AqP_q \subseteq A_q. The final context Cq⊆PqC_q \subseteq P_q, with ∣Cq∣≤k|C_q| \leq k, contains the selected turns supplied to the reader for answer generation. The reader returns answer yqy_q and cited source identifiers ZqZ_q. For the set operations below, turns are represented by their source identifiers.

At any stage S∈{A,P,C}S \in\{A,P,C\}, we report micro-averaged gold-turn evidence recall

ER⁡(S)=∑q∣Gq∩Sq∣∑q∣Gq∣,(1)\operatorname{ER}(S)=\frac{\sum_q |G_q \cap S_q|}{\sum_q |G_q|}, \tag*{(1)}

and macro-averaged complete-proof-set rate

PS⁡(S)=1∣Q∣∑q1[Gq⊆Sq].(2)\operatorname{PS}(S)=\frac{1}{|Q|}\sum_q \mathbf{1}[G_q \subseteq S_q]. \tag*{(2)}

ER is the fraction of annotated support turns retained across questions; PS is the fraction of questions for which every annotated supporting turn is retained.

Retrieval and answer metrics

Recall@5 is the macro-average of ∣Gq∩Cq(5)∣/∣Gq∣|G_q \cap C_q^{(5)}|/|G_q|, where Cq(5)C_q^{(5)} contains the first five turns in the ranked final context. Mean reciprocal rank (MRR) averages the reciprocal rank of the first retrieved member of GqG_q in the final context, with zero if none is retrieved. Under our reporting convention, a question with no annotated support contributes zero to both metrics. Unlike ER, Recall@5 is macro-averaged per question and truncated at five.

For answers, normalization lowercases text, collapses whitespace, and removes punctuation. Exact match (EM) is equality after this normalization. Token F1 is the harmonic mean of multiset-token precision and recall against aq∗a_q^*. We use both as lexical agreement diagnostics, not sufficient measures of semantic correctness. LoCoMo and REALTALK report partial-match F1, while LongMemEval uses LLM-based evaluation to accommodate equivalent answer formulations [11, 8] (Wu et al. 2025). Our annotation-based abstention correctness is one when the system abstains exactly when Gq=∅G_q = \varnothing; missing annotations alone do not establish that a question is unanswerable. A citation is a dialogue-turn source identifier returned with the answer to indicate its supporting evidence. Citation validity requires Zq≠∅Z_q \neq\varnothing and Zq⊆CqZ_q \subseteq C_q. Per-question citation precision and recall measure identifier overlap with the annotated support set:

CP⁡q=∣Zq∩Gq∣∣Zq∣,CR⁡q=∣Zq∩Gq∣∣Gq∣,(3)\operatorname{CP}_q=\frac{|Z_q \cap G_q|}{|Z_q|}, \qquad\operatorname{CR}_q=\frac{|Z_q \cap G_q|}{|G_q|}, \tag*{(3)}

with zero for an empty denominator. We macro-average these per-question scores and binary indicators over the evaluated questions. Our mechanically citation-grounded answer indicator is one only when yqy_q is nonempty and not an abstention, all citations are valid, and Zq∩Gq≠∅Z_q \cap G_q \neq\varnothing. This is a provenance check, not a semantic entailment claim.

Lexical metrics penalize equivalents such as “twice” and “2 times.” We therefore complement them with LLM-as-a-judge evaluation using Qwen3-32B, which sees the question, reference, generated answer, retrieved context, and source identifiers. It assigns correct, partially correct, incorrect, correct abstention, or wrong abstention, and separately marks whether the answer is supported by the shown context. We report judge accepted as the combined rate of correct answers and judge-accepted abstentions, and report those labels separately when comparing reader prompts. An accepted abstention is a judge decision, not proof that the full conversation is unanswerable. Following known judge bias concerns (Zheng et al., 2023), these labels complement rather than replace the deterministic metrics. All principal intervals use 20,000 paired non-parametric bootstrap resamples of the ten conversations.

Method

Overview

EviCover changes the multi-hop branch of the frozen retrieval pipeline. It has three explicit stages. First, a planner converts the question into up to four observable evidence requirements and retrieves for each uncovered requirement. Second, a provenance audit may mark a requirement covered only by citing an identifier from the retrieved candidates. Third, the same pointwise relevance model used by the baseline supplies a prior, while a setwise selector builds ten complementary turns rather than returning the ten highest independent scores. Figure 1 shows the data passed between stages; gold support identifiers enter only after generation for evaluation.

Baseline retrieval pipeline

The memory store preserves each raw turn and its source identifier. BM25 is the lexical channel: it ranks turns by exact query-term evidence with document-length and term-frequency normalization [15]. The semantic channel embeds questions and turns into 4,096-dimensional vectors with E5-Mistral-7B-Instruct, a text-embedding model trained to place semantically related queries and passages near one another (Wang et al., 2024). We use cosine similarity between vectors for dense retrieval; “E5-Mistral” below denotes this concrete model.

At write time, each turn is linked to at most ten active turns that share an entity identifier or an identical nonempty subject; matching predicates, subjects, and entity-only matches yield ACCUMULATES, EXTENDS, and SPECIFIES edges, respectively. For factual and temporal questions, the ten strongest lexical/dense seeds start a depth-two walk over these edges (and temporal TRANSFORMS edges), with at most 12 returned neighbors per seed. For multi-hop questions, deterministic text rules produce at most six query facets and add dialogue neighbors within ±2\pm2 turns. The benchmark-provided query type chooses the branch. Weighted reciprocal-rank fusion (RRF) combines lexical, dense, and graph rankings as

sRRF(t)=∑rwr60+rank⁡r(t),(4)s_{\mathrm{RRF}}(t) = \sum_r \frac{w_r}{60 + \operatorname{rank}_r(t)}, \tag*{(4)}

with wBM25=0.25w_{\mathrm{BM25}} = 0.25, wE5=4.0w_{\mathrm{E5}} = 4.0, and wgraph=1.5w_{\mathrm{graph}} = 1.5 when the graph channel is present [2]. The best 60 fused turns are then scored independently by the 0.6B Qwen3-Reranker model. It receives one pair (q,t)(q,t) at a time and converts its final yes/no logits to a relevance probability (Zhang et al., 2025a). Because no candidate is scored in the context of another candidate, we call this a pointwise relevance scorer; it is the only learned relevance model in the controlled baseline and requirement-guided comparisons.

Provenance-constrained requirement discovery

Requirement-guided discovery starts with a Qwen3-32B planner that maps qq to one to four observable, independently verifiable requirements Rq={r1,…,rm}R_q = \{r_1,\ldots,r_m\}. For example, an evaluative question about irresponsibility can yield one requirement for behavior affecting others and one for an unfulfilled obligation; it does not search for the abstract adjective alone. Each round retrieves up to 60 BM25 and 60 E5-Mistral results for one uncovered requirement and merges them by RRF. Candidates remain in AqA_q across at most four rounds.

The planner then audits at most 16 shown candidates. A requirement may be marked covered only with a source identifier in AqA_q; code rejects absent or malformed identifiers. An uncovered requirement produces the next focused query. Retrieval stops when all requirements are covered, four rounds are exhausted, or a duplicate follow-up cannot be replaced. The same 0.6B Qwen3 pointwise model scores the accumulated candidates once. We retain its top 60 for a capacity-matched comparison with the baseline and expose up to its top 120 to the operational set selector. If checklist parsing fails, the system performs one search with the original question; if a later audit cannot be parsed, it stops expansion while retaining all candidates accumulated so far.

Requirement-seeded setwise selection

The pointwise control returns the ten highest relevance scores. Our selector instead seeds a ten-item set with the highest-ranked cited turn for each covered requirement, then fills empty slots by the pointwise prior. The first five positions of this seeded set are protected; they are not necessarily the original pointwise top five. Remaining candidates from the top 120 arrive in batches of eight. Qwen3-32B may replace at most one item per batch and three items overall, and only when the challenger directly adds or more strongly supports a listed requirement while the removed item is redundant. Invalid JSON, unknown identifiers, or an unsupported justification leave the set unchanged. Pointwise and setwise contexts therefore share AqA_q, the same top-120 candidate set, pointwise scores, and k=10k = 10; only the final set utility differs.

Figure 1 shows the complete intervention. Gold support turns are used only after retrieval and generation for evaluation.

Experimental Setup

Data. The evaluated LoCoMo release contains ten generated, human-verified conversations: 5,882 turns (588.2 per conversation), up to 32 sessions, and 1,986 QA examples ([11]). REALTALK contains ten 21-day conversations between people with 8,944 turns and 728 memory-probing questions: 301 factual, 319 temporal, and 108 multi-hop ([8]). The multi-hop slice has 327 annotated support turns.

Models and budgets. Both controlled multi-hop systems use the same BM25 index, E5-Mistral-7B-Instruct embeddings, RRF weights, 0.6B Qwen3 pointwise scorer (maximum input length 4,096), and top-10 reader context. The flat baseline stores 60 pointwise-ranked candidates. Requirement-guided discovery reports a matched top-60 candidate set and an operational top-120 prefilter; the latter is shared by its pointwise and setwise top-10 variants. The requirement planner and setwise selector use Qwen3-32B with temperature zero and thinking disabled ((Yang et al., 2025)). The 0.6B pointwise scorer is non-generative and runs in evaluation mode without gradients. The frozen reader is gpt-oss-120b at temperature zero ([13]). The full baseline evaluation allows 6,000 context tokens; saved multi-hop contexts are replayed with a 4,000-token cap and no new retrieval. Qwen3-32B judge calls also use temperature zero.

Generation and citations. Query-type-specific prompts ask for concise factual, temporal, or multi-hop answers. Citation repair is gold-blind and can choose only a packed turn. It preserves an already valid citation; otherwise, for packed turn tt at rank jj, it computes

3∣W(yq)∩W(t)∣+0.5∣W(q)∩W(t)∣+1/j(5)3|W(y_q)\cap W(t)| + 0.5|W(q)\cap W(t)| + 1/j \tag*{(5)}

where WW removes common weak tokens, then

cites up to two source identifiers from the highest-scoring turn. It cannot introduce unretrieved evidence.

Protocol and reproducibility. Principal tables use the frozen baseline and EviCover runs; three additional full retrieval replications measure stability under the same configuration. Temperature zero removes intentional sampling, not all remote-serving nondeterminism. Principal multi-hop comparisons use all 108 questions. A repeatedly inspected fixed first-30 slice rejects expensive alternatives and is diagnostic, not held-out. The full-108 analysis includes these development questions; it is an in-corpus audit, not an independent held-out confirmation. Each run stores configuration, examples, retrieval traces, token use, packed contexts, metrics, and resumable state under a UUID. Paper tables and confidence intervals are regenerated from immutable JSONL artifacts.

Results

RQ1: Dataset evaluation and stage-wise proof-set loss

Table 1 shows that revising answer-generation settings on LoCoMo improves token F1 and citation-grounded answers while Recall@5 remains nearly constant. With the revised configuration held fixed, REALTALK results are lower on all reported answer and citation measures. The two datasets have different question distributions, so these scores do not establish their relative difficulty.

RunR@5F1CPCRCit.-Gr.
LoCoMo initial.772.316.463.448.493
LoCoMo revised.771.394.558.535.593
REALTALK baseline.579.291.477.374.511

Table 1. Baseline evaluation on both datasets. CP/CR are citation precision/recall; Cit.-Gr. is the mechanical citation-grounded answer rate defined in § 3.2.

The LLM judge assigns higher semantic scores than lexical metrics suggest: on all 728 questions, 55.9% are accepted (including accepted abstentions), 80.2% are accepted or partial, and 71.6% are supported by the shown context, compared with 6.7% EM. Table 2 nevertheless identifies multi-hop as the weakest slice for both retrieval and judge support. Temporal F1 is additionally affected by equivalent date formats and concise surface forms.

Type (nn)R@5F1J-C/PJ-Sup.Cit.-Gr.
Factual (301).391.381.847.787.475
Temporal (319).814.239.777.702.611
Multi-hop (108).409.191.750.556.315

Table 2. REALTALK baseline by query type. J-C/P is judge correct-or-partial; J-Sup. is judge-supported; Cit.-Gr. is the mechanical metric.

Temporal and semantic distance. For each annotated support identifier, we measure its distance tance from the final released turn of the same conversation. This is a lower bound on distance to the memory-probing question if that question was asked later. Table 3 confirms that the task is not merely short-range semantic retrieval: the median support turn is 12 days and 380 turn positions old, and 72.2% of all support turns are at least seven days old. At the question level, the oldest required support has a median age of 16.5 days.

Distance unitMedianP90Maximum
Turn positions3809891574
Sessions121924
Days122123

Table 3. Distance from each of 327 support turns to the final observed turn.

Temporal distance is not the only difficulty. Among the 294 support turns present as text in the release, 63.3% have zero content-word overlap with their question. We lowercase, retain alphanumeric tokens longer than one character, and remove the fixed English stopword list supplied with the diagnostic code. This is a surface-form proxy, not a semantic similarity measure. The remaining 33 annotated identifiers refer to media-only turns absent from the textual export; they are retained in temporal counts but excluded from lexical-overlap statistics. Recall by age is not monotonic: baseline final ER is .324 for 0–2 days, .407 for 3–6, .322 for 7–13, and .349 for 14+ days. The benchmark therefore combines genuine long-range memory with semantic mismatch rather than isolating either factor.

Table 4 and Figure 2 trace all 327 multi-hop support turns. The flat baseline retains 172 in its 60 pointwise-ranked candidates but only 114 in final context. Requirement-guided retrieval accumulates 217 and retains 188 in its matched top-60 candidate set: +0.049 ER with a 95% conversation-cluster bootstrap CI of [0.028, 0.067]. Allowing the operational prefilter to contain 120 candidates retains 207 turns; this number measures available evidence for set selection and is not a capacity-matched baseline comparison.

SystemStageHits/327ERFull sets
Flat hybridtop-60172.52650/108
Flat hybridfinal top-10114.34936/108
Req. discoveryaccumulated217.66463/108
Req. discoverytop-60188.57555/108
Req. discoverytop-120207.63360/108
Pointwisefinal top-10116.35536/108
Setwisefinal top-10124.37938/108

Table 4. Evidence funnel on all 108 multi-hop questions. “Req. discovery” denotes requirement-guided retrieval. Top-60/120 rows report ranked candidates.

Candidate pool and reader context evidence recall bar charts

Figure 2. Annotated support-turn evidence recall. Candidate bars compare matched top-60 budgets; final-context bars compare top-10 outputs.

Setwise final context retains 124 turns, +0.031 over the complete flat pipeline (CI [0.007, 0.064]).

Setwise selection improves ER over the matched pointwise top-10 from .355 to .379 (+0.024, CI [0.009, 0.047]), but full proof sets increase only from 36 to 38; the +0.019 rate difference has CI [−0.018, 0.055]. An oracle top-10 from the same candidate set reaches .624 ER and preserves all 60 annotated support sets available in full. The 108 questions decompose into 45 whose complete set is never discovered, three lost during independent scoring, 22 complete among ranked candidates but incomplete after top-10 selection, and 38 complete in reader context.

Exploratory alternatives on the fixed-30 slice and additional memory representations are reported in Appendix § 9.

RQ2: Evidence gains do not yet establish answer gains

We replay answers over the 108 saved setwise contexts with the same reader, query-type prompt family, temperature, and citation policy. No retrieval is repeated. Table 5 shows that judge accepted-or-partial and supported rates each rise by .028, but both confidence intervals include zero. Mechanical F1 and citation-grounded rate decline slightly, also with intervals crossing zero. Correct labels change from 35 to 32, correct abstentions from 31 to 35, partial answers from 15 to 17, and wrong abstentions from 9 to 6. The new context therefore changes answer behavior without a statistically clear gain in full correctness.

MetricFlatReq.+setΔ\Delta95% CI
Judge accepted.611.620+.009[-.058,.074]
Accepted/partial.750.778+.028[-.020,.083]
Judge supported.556.583+.028[-.037,.080]
Judge score3.0833.028-.056[-.180,.065]
Token F1.191.183-.009[-.023,.006]
Citation-grounded.315.306-.009[-.063,.057]

Table 5. Paired answer replay on 108 multi-hop questions. CIs are for Δ\Delta.

Complete evidence remains associated with answer quality: when the setwise context contains the full annotated proof set, judge correct-or-partial is .895 and support is .737, versus .714 and .500 otherwise. This observational gap is not causal because easier questions may also be easier to retrieve, but it supports reporting complete-set survival rather than only the first relevant hit.

Reader-only intervention on frozen contexts

We next change only the multi-hop reader instruction on the same 108 saved EviCover contexts. The original prompt favors minimal answer spans and omits reasoning; the revised prompt permits one compact, explicitly supported reason, requires citations for its evidence, and prohibits unsupported elaboration. The model, temperature, 4,000-token context cap, and citation repair remain unchanged. This is a post-hoc reader ablation, not an independent test set or an additional retrieval improvement.

Table 6 reports the completed paired replay; all 108 judge responses parse successfully. Substantively correct labels increase from 32 to 36, but accepted abstentions decrease from 35 to 24 and wrong abstentions increase from 6 to 10. Partial and incorrect labels change from 17 to 21 and 18 to 17, respectively. Consequently, the overall accepted rate falls from .620 to .556 despite the higher mean score and citation-grounded rate. Judge-supported answers change only from 63 to 65. The accepted-rate decrease has a conversation-cluster CI of [−.131, −.009], while the citation-grounded increase has CI [.010, .126]; these exploratory in-

MetricOriginalRevisedΔ\Delta95% CI
Judge accepted.620.556-.065[-.131,-.009]
Accepted/partial.778.750-.028[-.095,.042]
Judge supported.583.602+.019[-.029,.071]
Judge score3.0283.157+.130[-.020,.284]
Token F1.183.147-.036[-.111,.031]
Citation-grounded.306.370+.065[.010,.126]

Table 6. Reader-only paired ablation on identical EviCover contexts (108 questions). Accepted includes judge-accepted abstentions.

tervals are not adjusted for multiple comparisons. Better citation overlap therefore does not establish better semantic correctness; we retain the original reader for the principal EviCover comparison.

Temperature-zero repeatability

Three independent replications use the same 108 questions, indexes, budgets, prompts, and decoding configuration. Across pairs, the final selected identifiers match exactly on 91.7–99.1% of questions; mean evidence-set Jaccard similarity is .971–.998. Recall@5 ranges from .4003 to .4020 (spread .0017), MRR from .4664 to .4667, and nDCG@5 from .3814 to .3838. Thus aggregate retrieval performance is stable in these replications, but the generated search trajectories and individual contexts are not identical. These are retrieval repetitions, not repeated answer generation or independent draws of new conversations.

Efficiency audit

Table 7 reports exact saved-run cost on the same 108 questions. Mean retrieval latency increases by 2.28×. EviCover adds 17.47 Qwen3-32B calls and 24.59K planner/selector tokens per question, for 2.66M tokens in total; 81.2% are spent by final set selection. Relative to the flat final context, this corresponds to 265,568 tokens per additional retained support turn and 1,327,838 per additional complete proof set. These are inference counts, not monetary or energy estimates, because both models ran on private endpoints without hardware accounting; the wall-clock ratio is observational rather than hardware-normalized.

MeasureFlat hybridEviCover
Mean latency (s/query)18.2841.62
Pointwise candidates/query60118.45
Generative calls/query017.47
Generative tokens/query024.59K
Final ER0.3490.379
Complete proof sets3638

Table 7. Retrieval utility and cost on the same 108 multi-hop questions. LLM calls/tokens count additional generative Qwen3-32B retrieval work.

Discussion

Item relevance and set coverage are different objectives. The 22 questions with complete support among ranked candidates but incomplete support in the final context quantify the mismatch. A point-wise ranker can repeatedly select turns about the same aspect, whereas proof-set selection values the marginal coverage of a currently missing requirement. The oracle gap shows that a learned set utility remains a larger opportunity than further hand-tuning RRF weights.

Human dialogue changes the search target. LoCoMo multi-hop questions often combine explicit facts across time. REALTALK also asks for judgments and preferences inferred from behavior; the abstract term in the question may be absent from its support turns. Observable requirements improve discovery under this lexical mismatch, but Qwen3-generated requirements remain imperfect and expensive.

Reader utilization is a separate bottleneck. Better proof-set recall mostly changes abstention and partial-credit labels rather than full correctness. Future models should jointly score requirement coverage, provenance, redundancy, and whether the reader can compose an answer from the selected set.

The intervention is diagnostic, not yet operationally efficient. Most added computation is spent repeatedly deciding whether a candidate adds marginal coverage, yet only two additional complete proof sets reach the reader and answer intervals cross zero. EviCover is therefore evidence that pointwise selection is a real bottleneck, not evidence that this particular 32B implementation should be a universal production default. A practical successor would distill set utility into a smaller scorer or invoke it only when an inexpensive coverage test detects a likely incomplete proof set.

Conclusion

Stage-wise analysis reveals two separable failures in long-term conversational memory: complete proof sets are often never discovered, and complementary turns that are discovered are discarded before generation. Provenance-constrained requirement discovery improves the REALTALK candidate set at matched capacity, and setwise selection preserves a small portion of that gain, but at substantial inference cost. Complete-set and answer-level improvements remain modest and uncertain. Reporting temporal and lexical distance together with ER and PS through the retrieval funnel gives a more actionable account than a single top-kk or answer score: better memory requires complete evidence delivery, an efficient set objective, and a reader able to use the delivered evidence.

Limitations

The principal comparison has 108 questions nested in ten conversations. Conversation-cluster bootstrap respects that dependence, but ten clusters yield wide intervals. The fixed-30 slice was repeatedly inspected, and exploratory alternatives and the later reader-prompt change are descriptive rather than confirmatory. Intervals are conditional on the selected configurations, not adjusted for method selection or multiple exploratory comparisons. Retrieval replications measure inference stability on the same data, not generalization or repeatability of the answer evaluation. We use one reader model and one uncalibrated LLM judge. The judge sees mechanical scores and gold identifiers, is not blinded to those signals, and uses the same Qwen3-32B model family as discovery and selection. Its accepted-abstention labels must not be equated with corpus-level unanswerability; no human adjudication of those labels was performed.

LoCoMo and REALTALK have different question distributions, so cross-dataset scores do not rank intrinsic dataset difficulty. Temporal lag is measured to the final released turn rather than a question timestamp; it does not measure the number of turns given to the reader. Thirty-three media-only support identifiers lack released text. Annotated support may also be incomplete, especially for inferential questions; PS measures survival of the released annotation, not every valid proof. The controlled alternatives use implementations inspired by published approaches, not their trained checkpoints or full published training recipes, and do not establish leaderboard comparisons with those systems.

This study evaluates memory-question answering, not personalization: it does not test whether an assistant adapts recommendations, style, or actions to an individual user. Long-term memory can support personalization, but the present results should not be interpreted as evidence of personalized behavior. Hardware-normalized latency, monetary cost, and energy use were not measured.

Ethical considerations

REALTALK contains conversations between people. We use only the released benchmark, add no human-subject data, preserve source provenance, and do not quote source conversations in the manuscript. Illustrative examples are schematic rather than verbatim dialogue. A deployed memory system would require access control, deletion, retention limits, and user consent. We do not redistribute conversation text as part of the manuscript package; access remains governed by the source datasets’ terms.

Acknowledgment

This work was supported by the Ministry of Economic Development of the Russian Federation (IGK 000000C313925P4C0002), agreement No 139-152025-010.

Exploratory alternatives

Table 8 summarizes the fixed-30 diagnostic slice (82 support turns). Free iterative search, query-local graph chaining, breadth-then-depth expansion, and a restricted executable search program broaden discovery, but none exceeds requirement-guided support retention after the shared pointwise reranker. The repeated gap between accumulated and final ER motivated the setwise intervention rather than another query-expansion prompt.

Discovery methodR@5Accum. ERRanked ERFinal ER
Flat lexical+dense.452–.524.378
Free iterative.397.646.549.366
Requirement-guided.424.683.573.378
Query-local graph chain.374.659.512.329
Breadth-then-depth.358.671.549.329
Restricted search program.386.634.537.341

Table 8. Exploratory fixed-30 multi-hop diagnostics (82 support turns).

Table 8: Exploratory fixed-30 multi-hop diagnostics (82 support turns).

Alternative representations expose the same boundary. A proposition index extracts 14,563 short facts from 8,925 processed turns and raises full multi-hop candidate hits from 172 to 188, but final hits change only from 114 to 115; a proposition-aware selector reduces them to 91. A provenance-linked index of 2,798 episodic events finds seven support turns absent from the baseline candidate set on a 30-question behavioral subset, but quota fusion does not retain them reliably. A ready-made ColBERT-style model matches baseline candidate ER (.524) yet lowers final ER (.341 versus .378). The proposition and event controls show that richer representations can surface complementary turns, but their gains are not automatically preserved by the shared itemwise objective.

References

  1. [1]Pratyay Banerjee, Masud Moshtaghi, Shivashankar Subramanian, Amita Misra, and Ankit Chadha. 2026. APEX-MEM: Agentic semi-structured memory with temporal reasoning for long-term conversational AI. In Proceedings of the 64th Annual Meeting of the Association for Computational Linguistics, pages 16470–16489.
  2. [2]Gordon V. Cormack, Charles L. A. Clarke, and Stefan Buettcher. 2009. Reciprocal rank fusion outperforms condorcet and individual rank learning methods. In Proceedings of the 32nd International ACM SIGIR Conference on Research and Development in Information Retrieval, pages 758–759.DOI
  3. [3]Yuanzhe Hu, Yu Wang, and Julian McAuley. 2026. Evaluating memory in LLM agents via incremental multiturn interactions. In International Conference on Learning Representations.arxiv.org/abs/2507.05257
  4. [4]Bernal Jimenez Gutierrez, Yiheng Shu, Weijian Qi, Sizhe Zhou, and Yu Su. 2025. From RAG to memory: Non-parametric continual learning for large language models. arXiv preprint arXiv:2502.14802.
  5. [5]Maha Karkout, Maria Andreevna Khodorchеnko, Nikolay Alekseevich Butakov, and Denis Nasonov. 2026. Efficient KG-augmented RAG with reusable graph community summaries. In Proceedings of the Knowledge Graphs and Large Language Models Workshop (KG-LLM) at LREC26, pages 110–119, Palma, Mallorca, Spain. European Language Resources Association (ELRA).DOI
  6. [6]Omar Khattab and Matei Zaharia. 2020. ColBERT: Efficient and effective passage search via contextualized late interaction over BERT. In Proceedings of the 43rd International ACM SIGIR Conference on Research and Development in Information Retrieval.arxiv.org/abs/2004.12832
  7. [7]Dahyun Lee, Yongrae Jo, Haeju Park, and Moontae Lee. 2025a. Shifting from ranking to set selection for retrieval augmented generation. In Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics, pages 17606–17619.
  8. [8]Dong-Ho Lee, Adyasha Maharana, Jay Pujara, Xiang Ren, and Francesco Barbieri. 2025b. REALTALK: A 21-day real-world dataset for long-term conversation. arXiv preprint arXiv:2502.13270.arxiv.org/abs/2502.13270
  9. [9]Yifei Li, Weidong Guo, Lingling Zhang, Rongman Xu, Muye Huang, Hui Liu, Lijiao Xu, Yu Xu, and Jun Liu. 2026. LoCoMo-Plus: Beyond-factual cognitive memory evaluation frameworks for LLM agents. In Proceedings of the 64th Annual Meeting of the Association for Computational Linguistics, pages 25085–25100.
  10. [10]Yuewen Liu, Peng Xu, Muxi Diao, Anyi Zhang, Yang Li, and Yutong Zhang. 2026. MemCorRL: Alternating co-optimization of memory retrieval and utilization via collaborative reinforcement learning. In Proceedings of the 64th Annual Meeting of the Association for Computational Linguistics, pages 38912–38924.
  11. [11]Adyasha Maharana, Dong-Ho Lee, Sergey Tulyakov, Mohit Bansal, Francesco Barbieri, and Yuwei Fang. 2024. Evaluating very long-term conversational memory of LLM agents. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 13851–13870.
  12. [12]Tengjun Ni, Xin Yuan, Shenghong Li, Kai Wu, Ren Ping Liu, Wei Ni, and Wenjie Zhang. 2025. StepChain GraphRAG: Reasoning over knowledge graphs for multi-hop question answering. arXiv preprint arXiv:2510.02827.arxiv.org/abs/2510.02827
  13. [13]OpenAI. 2025. gpt-oss-120b & gpt-oss-20b Model Card. https://openai.com/index/gpt-oss-model-card/. Accessed: 2026-08-02.
  14. [14]Jakub Podolak, Leon Peric, Mina Janicijevic, and Roxana Petcu. 2025. Beyond reproducibility: Advancing zero-shot LLM reranking efficiency with setwise insertion. arXiv preprint arXiv:2504.10509.arxiv.org/abs/2504.10509
  15. [15]Stephen Robertson and Hugo Zaragoza. 2009. The probabilistic relevance framework: BM25 and beyond. Foundations and Trends in Information Retrieval, 3(4):333–389.
  16. [16]Harsh Trivedi, Niranjan Balasubramanian, Tushar Khot, and Ashish Sabharwal. 2023. Interleaving retrieval with chain-of-thought reasoning for knowledge-intensive multi-step questions. In Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics.arxiv.org/abs/2212.10509

Paper details

vlnc:W_x9eas4ky
Version
v1 · latest
Published
Fresh — added to the Hub within the last 30 days
Added
License
CC BY 4.0
Usage

views · 0

downloads · 0

A view counts once per browser per day, a download each time the PDF is requested.

  • Computer Science

Not recorded for this paper

View statement
Looking for related papers…
Contents