WorksComputer Science
StaleBench: Isolating Stale Retrieval in Agent Memory, and Pricing the Fix
Abstract
An agent that stores facts over time eventually stores several versions of the same fact. Asked a value-agnostic question - "which team does X play for now?" - a similarity retriever cannot prefer the current version, because every version matches the entity the query names. StaleBench isolates this stale-retrieval failure on 892 real updating facts, with currency labels from the source (Wikidata validity years via TempLAMA), never from a retrieval score. Our central result is that retrieval metrics can rank memory policies backwards: a retention policy and its retain-everything control return the identical statement at rank 1 for every test query, yet differ by 29.5 and 65.6 points in reader accuracy on two readers when no date is shown. Contamination@5, the metric we add, ranks them correctly where current@1 and current@5 do not. Withdrawing the lexical anchor that statements carry costs deployed keying mechanisms tens of points of freshness; entity resolution on both the write path and the query recovers most of the loss. Three deployed systems (Mem0, Graphiti, LangMem) and a reimplemented ERASE each handle supersession differently, and no single scoring convention describes all four. We release the data, a frozen evaluator, and reference baselines.
Introduction
An agent that remembers what it is told will eventually remember several versions of one fact. Suppose it stored “Peter Sagan plays for Bora-Hansgrohe” and, later, “Peter Sagan plays for TotalEnergies”, and is then asked which team Sagan plays for now. Both statements name the entity the question names, so under any similarity function they match about equally well, and the retriever returns whichever one similarity noise favours: across the real facts of § 5.1 that is the stale version 0.530 of the time. The cheap fix — rerank the top hits by timestamp — returns the TotalEnergies statement first with the Bora statement beside it, and a reader handed both does worse than one handed the unfixed retrieval, for one of our two readers (§ 5.3). A retention policy that recognises the second statement as a new version of the first and overwrites it returns TotalEnergies alone — but only if it can tell that the two statements describe one fact, which is what most of this paper prices. Figure 1 lays out the benchmark that measures each step.

Figure 1. StaleBench in one picture. (a) Real updating facts become a chronological stream with one value-agnostic query per fact and a source-defined current label; two knobs set how hard same-fact detection is. (b) A policy is store-all retrieval, optionally time-aware, or retention = a keyer (is this a fact already stored?) × a mechanism (overwrite, or invalidate and keep), run on the write path, the read path, or both. (c) Each top- is scored by retrieval metrics, by what its text says, and by what a reader answers.
Existing evaluations do not isolate this failure: freshness benchmarks ask whether a model is current with the world, knowledge-conflict studies measure how a reader behaves once conflicting context has been retrieved, long-term-memory benchmarks measure broad recall, where multi-session gold dilutes staleness ([18] (Wu et al. 2025)), and production memory systems ship freshness mechanisms without a public instrument that says when the mechanism is needed (§ 2). Appendix K shows why citing such a mechanism is not the same as knowing it runs.
StaleBench is that instrument. It contributes:
An isolating task with source-defined labels. Facts stream into a store in time order, queries are value-agnostic, and the label “current” comes from the source (Wikidata validity years via TempLAMA ([5])), or a generator schedule), never from a retrieval score; metrics separate stale answers from misses and add contamination@k and footprint (§ 3).
A measured price for same-fact detection. Retention removes staleness at the source only if it can tell that two statements describe one fact. We make that decision’s difficulty a controlled variable: after the introducing statement, later mentions may use ordinary short forms (Ebrard, M. Ebrard, Los Angeles) instead of the full name, and along that axis § 5.2 prices a ladder of keyers, from cosine threshold to a true-key oracle, under two mechanisms.
A fix that is free. Those rungs, and the
systems they abstract, resolve entities when writing and answer queries by similarity. Running the same resolver on the query needs nothing the write path lacked, and the two paths fix different failures: the read path removes misses, the write path removes contamination and halves the footprint (§ 5.2).
(4) Evidence that the retrieval metric ranks these policies wrongly. We attach readers to the pipeline: current@1 inverts 13 of 42 policy comparisons against reader accuracy, contamination@5 inverts 2 (census in Appendix § 20), and a controlled pair — two keyers that return the same statement at rank 1 on every one of the 468 queries and differ only in what fills the rest of the context — is 65.6 points apart at the reader without surfaced dates (§ 5.3). Constructed contexts isolate the mechanism, and a closed-book control bounds what a reader knows without the store (0.088 and 0.244).
(5) Three deployed memory systems and one reimplemented mechanism placed on this surface, and the finding that placing one by source attribution does not work. Mem0, scored under the same harness, streams and encoder as the idealized rungs, is reported three ways — by source attribution, by what the memories say, by what a reader answers — and attribution disagrees with the text only on the deployed system; Graphiti’s invalidation fires on the write path and is ignored by its default read path; LangMem’s update fires and keeps what its model decides; ERASE’s edit reaches the stale entry and, under the smaller model, the neighbouring fact too (§ 5.4).
Related work
Table 6 (Appendix § 9) positions the benchmark. Four distinctions carry the novelty. Model vs. store freshness: TempLAMA (Dhingra et al., [5]) probes what a parametric LM believes at a time; FreshQA (Vu et al., 2024), RealTime QA (Kasai et al., [12]), StreamingQA (Liška et al., [17]), Dy-Know (Mousavi et al., [21]), and GrowOVER (Ko et al., [13]) test whether systems track a changing world or corpus. StaleBench reuses TempLAMA’s temporal triples but asks a different question one layer down: given that the agent already stored both the old and new value, does its retrieval surface the current one? Temporal QA and temporal-editing datasets — TimeQA (Chen et al., [3]), SituatedQA (Zhang and Choi, [34]), Temporal-Wiki (Jang et al., [8]), MQuAKE (Zhong et al., [35]) — likewise ask whether a model’s answers track time-scoped, evolving, or edited facts; none models an agent-owned store deciding what to retain when both values have already been written to it. Reader vs. retrieval: inter-context conflict studies (Chen et al., [2]; Jin et al., [10]) and prior-vs-context work (Wu et al., [29]) establish that readers resolve conflicting evidence unreliably. We take the complementary side — which versions reach the context — and then close the loop (§ 5.3), where the two literatures turn out to interact: a retrieval fix that looks strong in isolation can be harmful once a reader consumes its output. Keying vs. linking: the same-fact decision is an entity-resolution problem with a long literature — record linkage since Fellegi and Sunter [6], neural entity matching (Li et al., [16]), entity linking (Wu et al., [30]), coreference (Joshi et al., [11]), and, closest to our anchor-free axis, temporal entity linking over Wikipedia snapshots [33]. That work measures matching quality intrinsically; what is missing is its price in the decision it feeds — how many points of freshness a given quality of same-fact detection costs, and how that moves as the anchor is withdrawn — which is why our keying ladder is a cost curve against an oracle rather than another matcher. Mechanism vs. instrument: systems ship, and papers propose, freshness machinery — MemGPT’s self-editing [22], Mem0’s ADD/UPDATE/DELETE [4], Zep’s bi-temporal invalidation [24], Memory-R1’s RL-learned operations [32], ERASE’s rewrite-on-arrival [15], importance-and-recency scoring [23], graph-based indexing [9] — but no public benchmark isolates the failure these mechanisms exist to prevent or prices their hardest component. The policy ladder evaluates these mechanisms’ principles under one harness; § 5.4 runs Mem0, Graphiti and LangMem themselves and a reimplementation of ERASE.
| line of work | asks | does not model |
| TempLAMA; FreshQA; RealTime QA; StreamingQA; DyKnow; GrowOVER (Dhingra et al., [5]; Vu et al., 2024; Kasai et al., [12]; Liška et al., [17]; Mousavi et al., [21]; Ko et al., [13]) | is the model/corpus current with the world? | an agent-owned store; the retention decision; budgets |
| knowledge editing (Meng et al., [19, 20]) | can we rewrite weights? | the store entirely |
| knowledge conflict (Chen et al., [2]; Jin et al., [10]; Wu et al., [29]; Xu et al., [31]) | how does a reader behave given conflicting context? | which version reaches the context, and at what memory cost |
| entity linking, coreference, record linkage (Wu et al., [30]; Joshi et al., [11]; Zaporojets et al., [33]; Fellegi and Sunter, [6]; Li et al., [16]) | are these two mentions/records the same entity? | the price of getting it wrong in a down-stream retention decision |
| LoCoMo; LongMemEval (Maharana et al., [18]; Wu et al., 2025) | broad long-term memory ability | staleness in isolation (multi-session gold dilutes it) |
| memory systems (Packer et al., [22]; Chhikara et al., [4]; Rasmussen et al., [24]; Yan et al., [32]; Li et al., [15]; Jiménez Gutiérrez et al., [9]) | ship freshness mechanisms | an isolating measurement; oracle-priced keying |
| StaleBench | does the store surface its own most recent knowledge, what does each fix cost, and what does it cost the reader? | naturalistic dialogue (see Limitations); multi-valued facts only as set recall (Appendix M) |
Table 6. Positioning: what each line of work measures, and what it cannot see.
The benchmark
Task and metrics
A benchmark instance is a chronological stream of statements, each an (embeddable) natural-language assertion carrying a metadata timestamp, and a set of value-agnostic queries, one per fact, asking for the fact’s current value. A policy consumes the stream (deciding what to store, keep, or overwrite; optionally under a slot budget ) and answers each query with a ranking of stored items. With ground truth “the set of stored versions whose value is current at query time” (defined by the source; a value that recurs, A→B→A, is scored by value, not emission index), the top-1 is scored as current@1 (right fact, current value), stale (right fact, outdated value — the isolated failure), or miss (wrong fact). Over the top- we report current@k and contamination@k — whether the top- contains both a current and a stale version of the queried fact — and, after the stream, slots (store size). § 5.3 adds reader-side accuracy on the same queries, so contamination’s cost is measured rather than assumed.
contamination@k is not monotone in context quality — a policy can improve it by deleting the answer — so we never report it without current@k beside it (Appendix § 10). Table 1 collects the terms
| term | meaning |
| stale vs. miss | top-1 names the right fact with a superseded value, vs. the wrong fact |
| contamination@k | the top- holds both a current and a stale version of the queried fact |
| hedged | a context or answer names both the current and a superseded value |
| anchor | the subject’s full name on a statement; withdrawing it in (a) leaves a short form |
| keyer | decides whether an incoming statement describes a fact already stored |
| mechanism | what happens then: overwrite the slot, or invalidate and keep it |
| write / read path | where the keyer runs: on incoming statements, on the query, or both |
| controlled pair | two rungs identical at rank 1 on every query, differing only in contamination |
Table 1. Terms used throughout.
used from here on.
Real data: TempLAMA instantiation
TempLAMA [5] provides subject–relation–object triples with yearly validity (2010–2020) derived from Wikidata qualifiers. From 5,839 fact keys we keep those that are single-valued in every year (crisp top-1 scoring; 900 keys) and whose value changes at least once in the window: 892 facts over nine relations (teams, positions, employers, parties, coaches, chairs, heads of government, owners, schools), mean 2.1 (max 6) distinct values. Consecutive years with the same value collapse to one run, verbalized as a statement from relation-specific templates; the canonical template is the TempLAMA cloze with the value filled in, and with probability (paraphrase_p) a paraphrase template is used instead, so similarity-based same-key detection must survive surface variation. Statements carry the year only as metadata. The query is a value-agnostic current-value question naming the subject (“Which team does currently play for?”); gold is the value valid in the final year — a label from Wikidata, untouched by any system under test. Facts split by key hash into val (424, all tuning) and test (468, all reported numbers). Three larger configurations bound this one: a corpus with never-queried distractor facts (Appendix H), a 2,995-entry inventory (Appendix C), and a 2,904-fact scale track (Appendix M).
The anchor-free axis
Templated verbalization repeats the subject’s surface form on every statement, so matching that string is a nearly perfect same-fact detector (§ 5.1); real logs introduce an entity once and then refer to it in short form. We therefore make the anchor a knob: with probability (alias_p) a non-introducing statement names its subject by an alias derived mechanically from the full name — the head token (Ebrard), the leading token (Marcelo), an initialed form (M. Ebrard), or a drop-last shortening (Los Angeles for Los Angeles Rams).
Details of the controlled diagnostic generator, the distractor corpus and inventory configurations, and the tuning guardrails are in Appendix § 10.
Policies
Store-all victims. BM25 (Robertson and Zaragoza, [26]) and Cosine (dense retrieval over sentence embeddings; all-MiniLM-L6-v2 primary, all-mpnet-base-v2 robustness check (Reimers and Gurevych, [25])) store every statement and rank by similarity. Time-aware store-all (the cheap fixes). Time-rerank() retrieves the top- by similarity and answers with the most recent candidate; the candidate set is unchanged, so contamination@ for is identical to the victim’s by construction. Recency-weighted scores (Park et al., [23]). Both apply to BM25 as well.
Retention = a keyer a mechanism. Retention keeps one slot per fact; we factor it into the two choices it makes (Table 1). The keyer decides whether an incoming statement describes a fact already stored: threshold (nearest stored neighbour within cosine — what vector-store memories do), gazetteer (longest subject from a public inventory appearing verbatim), fuzzy linker (the canonical inventory and nothing else: candidate mentions are maximal runs of capitalized tokens, linked to the inventory entry with the highest character-trigram Jaccard similarity above a val-tuned threshold — deliberately ordinary machinery, no learning, no curated aliases), entity linker (the same inventory plus every alias derivable from it; its two ambiguity-resolution variants differ by at most 2.4 points), LLM (a model is shown the top-4 candidates and asked which, if any, is the same fact — it sees only statement text, never ids or timestamps), and oracle (the true fact key). The oracle bounds keying, not current@1: it still ranks by cosine, so it mis-picks a small fraction of facts, and a policy with a better ranking function can exceed it. The mechanism decides what happens then: overwrite replaces the slot destructively, while temporal-KG marks the superseded statement invalid and keeps it, so retrieval prefers valid edges and false merges stay recoverable — the Zep/Graphiti design (Rasmussen et al., [24]). Recency (FIFO) under budget is the degenerate retention policy.
Results
Real data: the failure and the cheap fixes
Table 2 is the headline. Store-all similarity retrieval fails on real updating facts exactly as the value-agnostic analysis predicts: cosine answers with a stale version on 0.530 of queries (BM25: 0.592), with no adversarial construction — only facts that changed and queries that name the entity rather than the answer. The cheap fixes then do what a practitioner would hope: reranking the top-2 similar hits by timestamp lifts current@1 to 0.925 (BM25 variant: 0.936), and recency-weighted scoring reaches 0.784. Four measured boundaries keep this from closing the problem: contamination@5 stays between 0.936 and 0.991 for every store-all policy, because reranking demotes stale versions within an unchanged candidate set rather than removing them; the repair is concentrated on easy facts (Appendix § 12); it assumes trustworthy timestamps, as retention assumes in-order arrival (Appendix § 13}); and its gain does not transfer to a reader (§5.3`).
| policy | current@1 ↑ | current@5 ↑ | stale@1 ↓ | miss@1 ↓ | contam@5 ↓ | slots ↓ |
| BM25 (store-all) | 0.400 [0.355,0.444] | 1.000 | 0.592 | 0.009 | 0.991 | 995 |
| BM25 + time-rerank | 0.936 [0.912,0.957] | 1.000 | 0.051 | 0.013 | 0.991 | 995 |
| Cosine (store-all) | 0.464 [0.419,0.506] | 0.985 | 0.530 | 0.006 | 0.974 | 995 |
| Cosine + time-rerank | 0.925 [0.900,0.947] | 0.985 | 0.049 | 0.026 | 0.974 | 995 |
| Cosine + recency-weight | 0.784 [0.746,0.821] | 0.991 | 0.203 | 0.013 | 0.936 | 995 |
| Reconsolidation () | 0.859 [0.827,0.889] | 0.929 | 0.073 | 0.068 | 0.137 | 513 |
| Lexical key (gazetteer) | 0.972 [0.957,0.985] | 0.985 | 0.004 | 0.024 | 0.000 | 467 |
| Keep-latest oracle | 0.979 [0.966,0.991] | 0.991 | 0.000 | 0.021 | 0.000 | 468 |
Table 2. TempLAMA test split (468 facts), paraphrase . Mean over queries; brackets = bootstrap 95% CIs; tuned hyperparameters from val ( for time-rerank, for the threshold keyer). Reconsolidation () is the threshold keyer and Lexical key the gazetteer keyer, both with overwrite. All four pre-specified paired comparisons (sign-flip permutation, Holm-corrected; the family is hard-coded before test evaluation and its store-all comparator is selected on val) are significant at . Two go against our own policies: time-rerank > cosine, and the best store-all fix > the threshold keyer on current@1; two go with them: the threshold keyer < the best store-all fix on contamination@5, and oracle > threshold keyer.
On the retention side, the gazetteer keyer reaches 0.972 with contamination 0.000 at 467 slots, against 0.979 for the true-key oracle (a difference we cannot reject: Holm-corrected sign-flip , overlapping CIs), while the threshold keyer manages 0.859. Read naively, same-fact detection is solved by string matching; § 5.2 shows that reading is an artifact of templated text.
One property of this configuration must be stated beside its table: anchored is saturated on current@1. A rule that recovers the subject from the canonical question, keeps every statement containing it verbatim, and answers with the most recent scores 0.996 — at or above the true-key oracle — while being maximally contaminated (contamination@5 = 1.0 at 995 slots), the arrangement § 5.3 prices as the worst measured at the reader. Saturation is a property of the retrieval metric, not of the task: reader accuracy across policies on this same split spans 0.327 to 0.985, which is why the reader track lives here, and why anchor_free (§ 5.2), where the strongest blind rule, a trigram mention linker, scores 0.630, is the configuration a leaderboard should rank on. Both blind rules ship as reference baselines (see Limitations).
Anchor-free: what same-fact detection actually costs
Table 3 (drawn as Figure 2, Appendix § 11) is the paper’s central measurement. Withdrawing the anchor separates rungs that are indistinguishable when it is present.
| policy | 0.25 | 0.5 | 0.75 | keying deficit at | ||
| BM25 (store-all) | 0.400 | 0.321 | 0.248 | 0.126 | 0.062 | — |
| BM25 + time-rerank | 0.936 | 0.891 | 0.840 | 0.776 | 0.726 | — |
| Cosine (store-all) | 0.464 | 0.391 | 0.282 | 0.177 | 0.111 | — |
| + time-rerank | 0.925 | 0.870 | 0.831 | 0.784 | 0.737 | — |
| + recency-weight | 0.784 | 0.679 | 0.585 | 0.513 | 0.419 | — |
| Overwrite: threshold | 0.859 | 0.765 | 0.671 | 0.605 | 0.513 | 31.4 |
| Overwrite: gazetteer (exact) | 0.972 | 0.752 | 0.575 | 0.333 | 0.169 | 65.8 |
| Overwrite: fuzzy linker (no alias table) | 0.968 | 0.908 | 0.846 | 0.767 | 0.714 | 11.3 |
| Overwrite: linker + alias table | 0.972 | 0.927 | 0.861 | 0.812 | 0.776 | 5.1 |
| Read-path key (no overwrite) | 0.989 | 0.953 | 0.915 | 0.865 | 0.831 | −0.4 |
| Symmetric key (read+write) | 0.989 | 0.953 | 0.915 | 0.865 | 0.831 | −0.4 |
| Symmetric key + alias table | 0.994 | 0.974 | 0.944 | 0.936 | 0.912 | −8.5 |
| Overwrite: true-key oracle | 0.979 | 0.942 | 0.895 | 0.859 | 0.827 | — |
Table 3. current@1 as the lexical anchor is withdrawn ( = alias_p: the probability a non-introducing statement uses a short form). Test split, , every policy re-tuned on val at every level. The last column is the keying deficit at : points of current@1 below the true-key oracle, which isolates the same-fact decision from the retrieval difficulty that aliasing also creates (and which the oracle pays too). The oracle is a write-path oracle — perfect same-fact detection, ordinary similarity retrieval — so it is an upper bound only within that family, and the rungs that also resolve the query post a negative deficit against it (§5.2).
When anchored, the top three rungs cannot be told apart — gazetteer vs. oracle raw , entity linker vs. oracle raw , the two keyers against each other (sign-flip permutation; we quote the raw here because Holm over this family inflates the first two to =1.00, which reads as a stronger statement than the data makes. These are failures to reject, not demonstrations of equivalence). When it is withdrawn they separate at , and they separate by very different amounts.
The gazetteer keyer falls from 0.972 to 0.169 (deficit 65.8); the retrieval-free audit (Appendix § 15) shows why: it can key only the introducing statement and scatters 89.3% of facts across slots, so its parity with the oracle in Table 2 was a property of templated text, which any benchmark with a repeated subject will report. The threshold keyer — what vector-store memories ship — degrades from 0.859 to 0.513 (deficit 31.4), because the similarity between “Marcelo Ebrard serves as X” and “Ebrard now occupies the office of Y” is carried by the surface form the alias removes. Both rungs are what deployed systems use, and both pay tens of points.
Above both sits the best a write-path keyer does without an alias table. The fuzzy linker gets the canonical subject inventory and no alias table, and must resolve short forms itself with character-trigram similarity: it reaches 0.714 at (deficit 11.3), recovering most of the gazetteer’s collapse without being told what any alias is. That is the number a future write-path keyer should be measured against.
The entity linker, handed the alias inventory for free — exactly the resource the other rungs lack — reaches 0.776 at , 5.1 points from the oracle, splitting 6.6% of facts. Anchor-free same-fact detection is therefore not intractable: it is an entity-resolution problem, solvable to within a few points by machinery the memory literature has not adopted, and catastrophic without it.
The ladder resolves entities on one path only. Every rung above keys on the write path and then answers queries by embedding similarity, exactly as deployed memory systems do. That is a choice, not a necessity: the query is text naming the same entity, and the same resolver can run on it. We add two rungs that do, using no resource the fuzzy linker lacks (same inventory, same , no alias table): read-path keying routes the query through the trigram matcher and ranks the matched key’s slots by recency while retaining everything; symmetric keying does that on top of the ordinary write-path merge.
The two paths buy different things. Read-path keying fixes top-1: at it cuts the fuzzy rung’s miss rate from 0.167 to 0.045 and lifts current@1 from 0.714 to 0.831, because it stops asking an embedding to match a short form it never saw. Write-path keying fixes the context: alone it holds contamination@5 at 0.032 in 467 slots, whereas read-path keying without it retains everything (995 slots, contamination@5 0.983: a store-all context with a good first hit). Only the symmetric rung gets both (0.831, contamination@5 0.034, 467 slots) — two rungs that tie on current@1 while differing by 0.983 against 0.034 in contamination@5 are not interchangeable (§5.3), and current@1 alone would call them equal.
Symmetric keying also beats the rung handed a complete alias table on the write path — 0.831 against 0.776 here (), and at the Appendix M scale, where the comparison is better powered, deficit 6.3 against 11.4. Ordinary resolution applied symmetrically beats a complete alias table applied asymmetrically — on mechanical short forms. With the natural short forms of Appendix § 17, nicknames and misspellings a trigram matcher cannot link, the alias table wins by two to three points, and the linker applied symmetrically is best of all.
Three conditions bound this result, each discussed in Appendix § 11: queries here always name their subject in full, so the read side is the easy direction for a resolver; the read-path rungs resolve the query over the same 468-entry inventory as the fuzzy rung, and a deployment whose inventory lacks the queried subject pays on both paths; and keep-latest-oracle is a write-path oracle, so the rungs that resolve the query are not bounded by it — which is why Table 3’s deficit column goes negative for them, and at the Appendix M scale that ordering reverts (6.3 points).
The reader track: what the retrieval metric does not tell you
Every result so far is a statement about which versions reach the context. Whether that matters depends on what a reader does with them — something the knowledge-conflict literature reports is unreliable [2, 10, 29] but which our metrics have so far only assumed. This section measures it.
Setup and controls. This track runs on the anchored configuration (, ; saturated on current@1, § 5.1), deliberately: retrieval has to be high across policies for a reader difference to be attributable to context composition rather than to retrieval failures. For each query a policy’s top-5 becomes a reader’s context, and the reader answers the same value-agnostic question; answers are scored by normalized containment against the source-defined current value and against the fact’s superseded values, giving the correct/stale decomposition retrieval already has. Two controls matter. TempLAMA facts are real Wikidata facts, so a reader might simply know them: the closed-book condition (no context) scores 0.088 for qwen2.5:14b and 0.244 for glm-4.6 — far enough below the context-conditioned numbers that those are not recall of pretraining, and different enough that readers should be compared within, not across. And every condition runs twice, with and without the stored year surfaced in the context — the reader-level form of the cheapest objection, why not show the model the dates?
Retrieval gains do not transfer (Table 4). Timestamp reranking improves retrieval current@1 by +46.2 points over store-all cosine (§ 5.1); at the reader it is worth -22.9 points for qwen2.5:14b and +6.6 for glm-4.6. For one of the two the sign flips: the policy that nearly doubles retrieval accuracy leaves the end task worse than the baseline it was introduced to repair. We do not claim to know why the readers differ — they differ in scale and in how much of this data they already know (0.088 against 0.244 closed book) — but for both the retrieval gain does not arrive. Retention behaves oppositely, worth +42.1 and +36.3 points, so the 4.7-point retrieval difference between reranking and gazetteer keying (0.925 vs 0.972) opens into a reader gap of tens of points. current@1 compresses a difference the reader amplifies, and a benchmark reporting current@1 alone misstates how these policies compare.
| policy | retrieval current@1 | retrieval cont.@5 | reader accuracy (no dates) qwen2.5:14b | reader accuracy (no dates) glm-4.6 | dates qwen2.5:14b | dates glm-4.6 |
| Cosine (store-all) | 0.464 | 0.974 | 0.558 | 0.611 | 0.959 | 0.957 |
| + time-rerank | 0.925 | 0.974 | 0.329 | 0.677 | 0.957 | 0.944 |
| Overwrite: threshold | 0.859 | 0.137 | 0.874 | 0.872 | 0.929 | 0.915 |
| Overwrite: gazetteer | 0.972 | 0.000 | 0.979 | 0.974 | 0.979 | 0.970 |
| Read-path key (no overwrite) | 0.989 | 0.998 | 0.327 | 0.684 | 0.983 | 0.957 |
| Symmetric key (read + write) | 0.989 | 0.000 | 0.983 | 0.979 | 0.983 | 0.972 |
| Overwrite: oracle | 0.979 | 0.000 | 0.985 | 0.981 | 0.985 | 0.974 |
Table 4. Reader accuracy when each policy’s own top-5 becomes the context (test split, anchored configuration , ). Retrieval columns are the same policies’ current@1 and contamination@5. Readers are qwen2.5:14b, glm-4.6; the last two columns repeat both readers with the stored year surfaced in the context. Retrieval hyperparameters are inherited from the val tuning of § 5.1 and are never tuned against reader accuracy. The two read-path rows are the controlled pair: identical retrieval columns except contamination@5, because they return the same statement at rank 1 for all 468 queries of this configuration.
A controlled pair: identical rank 1, 65.6 points of accuracy. The comparisons above run across policies that differ in many ways at once, which is the usual objection to reading a correlation causally. The read-path and symmetric keying rungs of § 5.2 remove that objection, because they are the same policy on the read path: both resolve the query through the same matcher and rank the same key’s slots by recency, so they return the same statement at rank 1 for all 468 queries — verified per query and pinned by a regression test, not inferred from matching aggregates. They are identical on current@1 (0.989 for both, exactly) and identical on stale@1 (exact in the anchored configuration this track uses; anchor-free the two agree on 467 of 468 queries, so the controlled claim is made only where it is exact). They differ only in whether the write path also merges, which changes only what fills ranks 2–5: contamination@5 0.998 against 0.000. The retain-everything member is a control, not a system we propose — its purpose is to hold rank 1 fixed while contamination varies, which no comparison between two independently designed policies can do.
current@1 cannot tell them apart, and current@5 prefers the contaminated one, which retains every version and so never lacks the current one. The reader is not indifferent: 0.983 against 0.327 on qwen2.5:14b (, sign-flip permutation), a gap of 65.6 points, and 0.979 against 0.684 on glm-4.6 (, 29.5 points); with rank 1 identical on every query, that difference is attributable to ranks 2–5 and to nothing else. The contaminated member posts the highest current@1 of any policy in the track — above the oracle’s 0.979, since resolving the query is a better way to rank one hit than perfect retention is — together with the worst reader accuracy of any keying policy on both readers, and for qwen2.5:14b below even store-all cosine (0.558), whose current@1 is less than half of it (for glm-4.6 it stays above store-all, 0.684 against 0.611). A benchmark reporting current@1 would rank these two as tied for first, and one reporting current@5 would put the pair’s loser alone at the top. The retrieval-side measure of context quality that separates them correctly is contamination@5 — the metric this paper adds, and this pair is the cleanest argument for it.
Surfacing dates closes the pair. With the stored year shown in the context, reranking’s reader moves to 0.957, the constructed-context penalty of Appendix L largely disappears (0.925 against 0.314), and the pair closes: the retain-everything member goes from 0.327 to 0.983 while the symmetric member stays at 0.983, 10 individual answers changing and cancelling; for glm-4.6 the dated pair reads 0.957 against 0.972, a gap of 1.5 points. Given a trustworthy year on every statement, the reader resolves the contradiction itself. The claim above is untouched — it is about what current@ can see, and the pair is identical at rank 1 on every query while 65.6 points apart without dates — but retention should be read as one of two ways to resolve the contradiction, in the store or in the prompt, and the second is nearly free. Retention is the one to prefer when the clock cannot be trusted (the dated prompt shares reranking’s assumption, and Appendix E shows what each family loses when its temporal signal is corrupted — retention reads arrival order, and none is exempt), when the store returns no validity timestamp per memory (Mem0 does not; Graphiti does, and its default read path discards it, §5.4), and under a slot budget, where the versions a retain-everything store needs are the first evicted (the pair holds 995 against 464 slots; Appendix O).
What this justifies. The census of every policy pair against every reader, without surfaced dates (Appendix L), says the same thing in aggregate: restricted to policies a practitioner might deploy, current@1 still inverts 3 of 24 comparisons and contamination@5 inverts 0. The inferential weight is carried by the controlled pair, at on 468 paired queries.
Reversing the presentation order of the identical top- (qwen2.5:14b only) moves reranking’s value against the store-all baseline from -22.9 to +13.9 points, while retention changes zero individual answers: one statement per fact leaves nothing to reorder. That ablation, the constructed-context probe that establishes the mechanism causally on both readers, and the full census are in Appendix L.
Deployed systems, scored three ways
We place four mechanisms on this surface (Table 5): three deployed systems and our reimplementation of a published fourth. Mem0 adjudicates the incoming memory, Graphiti invalidates the edge an arrival contradicts, LangMem updates in place, and ERASE edits the entries already stored. Each runs under the rungs’ harness, streams and encoder, and is scored three ways: by attributing returned memories to source statements, by what they say, and by what a reader answers from them.
Mem0. Attribution and text disagree only on the deployed system; on the rungs, which return the corpus verbatim, the two agree to within 2.0 points. Attribution overstates , the release whose ingestion adjudicates, by 34.0 points with qwen2.5:14b in the loop (an UPDATE relabels a memory without rewriting it, so a current id sits on superseded text; with glm-4.6 it rewrites the text and the two scorings agree, Appendix K) and breaks outright on (62% of memories carry an id naming a different entity than their text). What no scoring convention changes is that the ADD/UPDATE adjudication is absent from ’s ingestion path: its event histogram, a two-statement probe and the library source say so independently of any metric. Neither release returns a validity timestamp, so the dated-prompt alternative of §5.3 is unavailable to a deployment reading Mem0’s output; applying contribution (3) post hoc to either release’s contexts recovers most of what they lose at no model call, and keeping the oldest memory per entity instead of the freshest prices the write path itself at 0.353 for against 0.007 for store-all. Every (release, vector store) pair is hand-probed first (4/6 pass).
Graphiti is the mirror image: its invalidation fires on the write path (157 of 179 supersessions) and its default search ignores it, so it scores 0.347 as installed, below store-all, and 0.833 with its own validity flags respected; the reader moves from 0.593 to 0.893.
LangMem applies 375 updates and keeps 327 slots with qwen2.5:14b (current@1 0.773 by id and by text). With glm-4.6 it consolidates into chronologies: the memory a query lands on is current by id (0.800) and hedged by text (0.720), and the reader answers 0.613 from it against 0.833 from the smaller model’s store.
ERASE [15] edits the store rather than the arrival: the 5 stored facts nearest an incoming statement are retrieved, and one model call keeps, rewrites or deletes each before the statement is inserted. No release pins it, so we reimplemented the loop from the paper under the same probe gate. With glm-4.6 deciding it reaches current@1 0.953 by id and by text at contamination@5 0.060, and 0.980 at the reader: the best number any of the four posts here, within 3.3 points of the write-path oracle (5 of 150 queries) with no alias table and no identifiers. With qwen2.5:14b it reaches 0.853, and for 0.133 of facts no version survives anywhere in the returned depth: asked whether a neighbouring fact is still accurate, the model deleted facts about entities the arrival did not mention (147 deletions and 88 rewrites against 1395 keeps). Anchor-free, the larger model falls to 0.713: an arrival written with a short form does not retrieve the full-name version it supersedes among its 5 neighbours, so the edit never reaches the stale entry. The mechanism edits only what its retriever finds, which is the anchor dependence of § 5.2 on the write path, and the mechanism that can remove staleness is the one that can remove the fact.
Across the four the pattern is shared and the failure never is: Mem0 lost its adjudication between releases, Graphiti keeps its verdicts and discards them on read, LangMem applies them and either keeps everything or rewrites it into a chronology, depending on the model, and ERASE applies them to the store and, under qwen2.5:14b, to facts the arrival never mentioned. No single scoring convention describes all four.
Conclusion
StaleBench isolates a failure every long-lived agent memory will meet, measures it on real updating facts with labels no system under test can touch, and prices each repair. Metadata heuristics fix the top-1 symptom when timestamps are clean, except on the facts that change most, and leave a self-contradicting context whose cost at the reader is set by prompt conventions — presentation order, whether the year is shown — that they do not control; retention removes the contradiction wherever same-fact keying works and ingestion is in order, each family reading one temporal signal and blind to the other (Appendix § 13). Agent memory attempts same-fact detection, a problem entity resolution addressed decades ago, with weaker mechanisms and pays tens of points of freshness for it; running the resolver on the query too is free and, where the inventory covers the queried subjects, closes most of that gap (§ 5.2). The controlled pair — identical at rank 1 on every query, 29.5 and 65.6 points apart at the reader without surfaced dates — is the strongest evidence that current@k alone cannot measure freshness (§ 5.3).
Limitations
Saturation and headroom. anchored and full_corpus are saturated on current@1 (a blind rule scores 0.970 on the latter); anchor_free is not: the strongest blind rule reaches 0.630 against an oracle at 0.827. The entity-linker rung is handed the alias inventory the stream drew from, so its residual 5.1 points bound a real linker’s gap from below. The best write-path rung without an alias table is the fuzzy linker, whose deficit of 11.3 points roughly doubles on the realistic corpus (23.7, Table 14). The reader metric has a ceiling of 0.994, because three test facts nest a superseded value inside the gold value.
| policy | anchored open-world | anchored closed-world | anchor-free open-world | anchor-free closed-world |
| Cosine (store-all) | 0.462 | 0.462 | 0.109 | 0.109 |
| + time-rerank | 0.902 | 0.902 | 0.590 | 0.590 |
| Overwrite: threshold | 0.793 | 0.793 | 0.404 | 0.404 |
| Overwrite: gazetteer | 0.942 | 0.947 | 0.162 | 0.165 |
| Overwrite: fuzzy linker | 0.912 | 0.904 | 0.453 | 0.436 |
| Overwrite: linker + alias table | 0.902 | 0.947 | 0.585 | 0.600 |
| Symmetric key (read+write) | 0.947 | 0.932 | 0.624 | 0.577 |
| Overwrite: oracle | 0.966 | 0.966 | 0.690 | 0.690 |
Table 14. current@1 on the 7,078-statement corpus (6,083 distractor statements from fact keys that are never queried). Keyers hold either an open-world inventory (the 468 queried subjects) or a closed-world one (all 2,995 subjects in the stream). Store-all policies use no inventory, so their columns coincide.
Scope of the data. The headline track queries single-valued facts that change (892 of 5,839 keys). Appendix M lifts that restriction and reports set recall beside top-1, since a one-slot policy keeps 0.890 of simultaneously valid values against 0.940 for recency-weighted store-all. Validity is yearly, statements are templated, and the anchor-free axis uses mechanically derived short forms, so conversational reference — pronouns, nicknames, discourse coreference — is outside what this instrument measures; Appendix § 17 narrows the gap with real aliases for the 58% of test subjects that have one. We generate the axis because we need it controlled; TempEL [33] would make it naturalistic. English only; nine relations; 468 test facts is enough for tight CIs but not for per-relation claims.
Systems track. This is the narrowest evidence in the paper: four mechanisms, one of them our reimplementation, on 150 facts, with a single reader. For Mem0, 0.1.118 passes both halves of the probe on only one of three backends, so we offer a cross-backend check only for 2.0.11 (0.180–0.207 over 2 stores) and an independent repeat for 0.1.118, which reproduced it exactly. The second adjudicator, glm-4.6, moves Mem0’s numbers at the extraction stage, and that comparison holds only under an extraction prompt that admits third-person facts. That 62% of 2.0.11’s memories carry an id naming a different entity than their text is a measurement without a mechanism: a six-statement probe of the same release attached metadata correctly.
Temporal assumptions and the reader. Facts arrive as they become true and gold is the final value, so most-recent-equals-current holds by construction in the clean condition. Appendix § 13 measures corrupted clocks, corrupted arrival order and late stale reports. Ambiguous updates, where it is unclear whether a statement supersedes an earlier one, and corrections with their own semantics, such as a retraction that names an earlier statement as wrong, are out of scope. The reader track uses 2 models with one prompt format, so its claims are comparative across contexts under a fixed prompt, not absolute accuracy. Appendix § 31 gives the full discussion.
Ethical Considerations
Data and personal information. The facts are public biographical and organizational relations about public figures and organizations, taken from Wikidata via TempLAMA [5] and released under CC0. They name real people, which is intrinsic to the task: the failure concerns the current value of a fact about a named entity, so pseudonymizing subjects would remove the phenomenon under study. We add nothing beyond what Wikidata publishes. Party affiliation, a special category of personal data under some regimes, is among the nine relations; these are publicly recorded affiliations of office-holders and candidates, and we neither infer nor annotate any. Aliases are mechanical shortenings of source names or public short forms from Wikidata and Wikipedia. The release is a frozen 2010–2020 window. Statements are template-verbalized and no user-authored text is included, so the risk of offensive content is low.
Risks. A benchmark that makes freshness measurable can be over-read as certifying it; our central result argues against that reading. Retention policies delete superseded statements, and where the history is itself the record (audit, compliance, contested claims about a person) destructive overwrite is the wrong mechanism and a false merge is unrecoverable. We therefore evaluate invalidation alongside overwrite and report its footprint cost (Appendix § 15). We see no dual-use concern.
Compute. Every number outside the systems track reproduces on a laptop CPU in under an hour. The 30,818 model calls behind the LLM tracks are short completions, cached and shipped with the release (Appendix § 26). Appendix § 31 gives the full statement.
Acknowledgments
We thank the ARR reviewers and the area chair for comments that shaped this paper.
Positioning table
Table 6 positions the benchmark against each adjacent line of work.
Benchmark details
One property of contamination@k must be stated plainly because it constrains how the metric may be read: it is not monotone in context quality. A top- holding only stale versions scores 0, the same as a top- holding only the current one, because nothing in it contradicts anything else. A policy can therefore improve contamination by deleting the answer. We accordingly never report contamination@k without current@k beside it, and a policy that wins on contamination while losing on current@k is winning by omission, not by resolution — the two together separate current-only, contaminated and current-absent contexts.
Controlled generator
Each of entities carries a value that updates with per-step probability ; an event embedding mixes a unit key direction (the entity) with a value direction: , where (value_gap) sets how visibly an update moves the embedding, and jitters the key per event with (key_jitter) — the controlled analog of surface variation. Queries point at the clean key only (value-agnostic by construction). Labels come from the generator’s update schedule. The generator is a diagnostic for mapping preconditions; headline claims live on real data.
Alias inventory
Alias candidates that are bare organizational type words (FC, Line, Ministry) are rejected, since no writer uses them as standalone references and keeping them would make the track an unfair strawman rather than a measurement.
Three properties keep the axis interpretable. The first statement about each fact always carries the full name and every query always uses it, so no fact becomes unidentifiable in principle — what changes is the cost of identifying it. Aliases that collide across subjects (Harris) are deliberately kept (3.5% of 1,122 forms, touching 17% of subjects), because entity-resolution ambiguity is part of what we are pricing. And the inventory is public: it ships with the dataset, so a keyer can be handed it for free and the residual gap attributed to resolution rather than to missing information. 89% of subjects admit at least one alias; at , 472 of 995 test statements are anchor-free.
Corpus size and inventory
A store containing only the facts it will be asked about is an unrealistically easy store. The full_corpus configuration adds every TempLAMA fact key excluded from the queried set (those multi-valued in some year) as distractor statements: 6,083 extra statements, a 7,078-statement stream. Distractors are never queried and need no labels, but they populate the neighbourhood that same-fact detection must avoid merging across. This configuration also separates two claims that the small corpus conflates, by giving keyers either an open-world inventory (the 468 queried subjects only, so most of the corpus is out-of-inventory) or a closed-world one (all 2,995 subjects in the stream).
Guardrails
Four design rules keep the benchmark from favouring its own baselines: currency labels are source-defined, never derived from retrieval scores; every hyperparameter of every policy (, , , decay) is tuned on the val split only, by current@1, and re-tuned separately in every regime, so the store-all fixes are given their best shot everywhere; reader-track hyperparameters are inherited from that retrieval tuning and never tuned against reader accuracy; and results always report where the retention fixes fail.
Read-path and symmetric keying: additional analysis
Neither the entity linker’s 5.1-point deficit nor the fuzzy linker’s 11.3 (§ 5.2) should be read as the residual, because both are conditioned on a second variable the ladder holds fixed: how complete the inventory is. The alias rung’s is exhaustive, correct, free, and generated by the same function that inserted the aliases, so it has no out-of-vocabulary mass at all. The fuzzy rung is handed no alias table — but its inventory is still exactly the 468 queried subjects, which is to say the index of the answer key. Holding the corpus fixed and enlarging that inventory to 2,995 entries costs it 11.3 → 28.6 points, and on the realistic corpus its deficit is 23.7 (§H).
The object is therefore a surface, not a scalar. Same-fact detection costs roughly 65.8 points with exact matching, 31.4 with embedding thresholds, and 11.3–28.6 with ordinary fuzzy entity resolution depending on how much of the world the inventory covers — and a system that knows exactly which entities it will be asked about is at the optimistic corner of that surface. Inventory completeness is a priced axis of this benchmark, not a caveat on one number.
The comparison §5.2 draws — ordinary resolution on both paths against a complete alias table on the write path alone — changes one thing and confounds two, the path and the resolver, so we run the control that separates them: the same complete alias table, applied on both paths. It reaches 0.912, and the two terms come apart cleanly. Making resolution symmetric is worth 11.8 points to the fuzzy resolver and 13.7 to the alias-equipped one (both ), and resolver quality is still worth 8.1 points once both are symmetric (). So the asymmetry is not a substitute for better entity resolution and better entity resolution is not a substitute for fixing the asymmetry: they are separable, both large, and only one of them is free. Part of what this section prices as the cost of imperfect same-fact detection is the cost of resolving entities when writing and not when reading — recoverable at no cost in resolver quality — and the rest is the resolver, which the field should keep working on. None of this is new machinery. Linking a query to a knowledge-base entity and then retrieving by key
rather than by embedding is a long-standing pattern in entity-oriented and KB question answering; what is new here is not the mechanism but its price in this setting — that agent memory does not do it, that doing it costs nothing it does not already have, and that it is worth more than a complete alias table on the write path. This is also not query expansion. Read-side rewriting — HyDE and its descendants (Gao et al., [7]) — improves the query’s embedding so that similarity search retrieves better, and stays inside the similarity paradigm. The read-path rung does the opposite: it stops using similarity for this query, resolves the query to a store key, and orders that key’s contents by recency. The two are complementary — a better query embedding would help the similarity fallback these rungs still use when resolution fails — but the mechanism being priced here is resolution, not expansion, and it is cheap precisely because it does not involve a generative model.
One scope condition belongs here rather than in the limitations, because it cuts against the rungs we are advocating: queries in this benchmark always name their subject in full, so the read side is the easy direction for a resolver. That is what makes the comparison informative rather than flattering — even where resolution is easiest, no system we know of does it — but a deployment whose queries are themselves elliptical (“and who does he play for now?”) would pay on the read path too, and we do not measure that.
Figure 2. The keying ladder as the lexical anchor is withdrawn. The inventory-based keying rungs are indistinguishable at , and at the gazetteer falls 65.8 points below the oracle; the entity linker (handed the alias inventory) tracks the oracle, while the two mechanisms deployed memory systems use do not. The oracle keys perfectly but still retrieves by similarity, so the two rungs that resolve the query as well are not bounded by it — and the upper one, which resolves symmetrically with a complete alias table, is close to flat across the whole axis (0.994 to 0.912). Withdrawing the anchor costs almost nothing to a system that resolves entities on both paths.
These rungs do not escape the inventory axis, either: they resolve the query with the same matcher over the same 468-entry inventory as the fuzzy rung, so they sit at the same optimistic corner of the surface above, and a deployment whose inventory does not already contain the queried subject pays there on both paths rather than one. That is measured rather than assumed, and it is the one place where the paper’s headline surface and its headline fix would otherwise sit on different policies. Run along the same inventory axis, the symmetric rung’s anchor-free deficit goes -0.4 → 19.7 points as the inventory grows 468 → 2,995, against 11.3 → 28.6 for the write-path rung: the advantage is between 9.0 and 12.6 points of current@1 at every inventory size, and both rungs degrade at essentially the same rate. Symmetric keying is therefore uniformly better without being a way out of the inventory problem — it pays on two paths where the ladder paid on one, which is why its degradation is if anything slightly steeper. The same holds on the realistic corpus with an open-world inventory, the hardest configuration here: 0.624 anchor-free for a deficit of 6.6 points against the write-path rung’s 23.7 (§H).
What this does to our oracle. It also bounds our own metric, in our disfavour. keep-latest-oracle has perfect same-fact detection but still retrieves by similarity, so it is
a write-path oracle, not an end-to-end one: at it misses 0.173 of queries outright because the aliased query does not embed near its one retained slot. The symmetric rung, with no oracle knowledge, is statistically indistinguishable from it on current@1 (0.831 vs. 0.827, — a failure to reject, not a demonstration of equivalence) by trading misses for stale answers — its stale rate is 0.122 where the oracle’s is zero. So the keying deficit reported throughout this section is the distance to the best write-path policy, and is a valid comparison within that family, which is the family deployed systems belong to. It is not a floor on the task. At the §M scale the ordering reverts — the oracle leads by 6.3 points — which is the same trade resolving the other way, and is why we report the deficit against a write-path oracle rather than claiming the task is solved.
Finally, and against our own framing: in the anchor-free regime the cheap metadata fix is competitive on current@1 within the write-path family. Time-rerank reaches 0.737 against 0.776 for the entity linker, because reranking needs no keying at all — though it no longer holds against the read-path rungs of this section (0.831 and 0.912), which is one more reason the asymmetry was worth finding. Retention’s advantage there is not top-1 freshness but contamination (0.024 against 0.882) and footprint — and § 5.3 shows contamination is what the reader pays for when no date is shown.
Table 7 decomposes where the anchor-free er-
@page 15/39 starts_mid=1 ends_mid=1
Figure 2 (PDF p. 15). The keying ladder as the lexical anchor is withdrawn. The inventory-based keying rungs are indistinguishable at , and at the gazetteer falls 65.8 points below the oracle; the entity linker (handed the alias inventory) tracks the oracle, while the two mechanisms deployed memory systems use do not. The oracle keys perfectly but still retrieves by similarity, so the two rungs that resolve the query as well are not bounded by it — and the upper one, which resolves symmetrically with a complete alias table, is close to flat across the whole axis (0.994 to 0.912). Withdrawing the anchor costs almost nothing to a system that resolves entities on both paths.
for now?”) would pay on the read path too, and we do not measure that.
These rungs do not escape the inventory axis, either: they resolve the query with the same matcher over the same 468-entry inventory as the fuzzy rung, so they sit at the same optimistic corner of the surface above, and a deployment whose inventory does not already contain the queried subject pays there on both paths rather than one. That is measured rather than assumed, and it is the one place where the paper’s headline surface and its headline fix would otherwise sit on different policies. Run along the same inventory axis, the symmetric rung’s anchor-free deficit goes -0.4 → 19.7 points as the inventory grows 468 → 2,995, against 11.3 → 28.6 for the write-path rung: the advantage is between 9.0 and 12.6 points of current@1 at every inventory size, and both rungs degrade at essentially the same rate. Symmetric keying is therefore uniformly better without being a way out of the inventory problem — it pays on two paths where the ladder paid on one, which is why its degradation is if anything slightly steeper. The same holds on the realistic corpus with an open-world inventory, the hardest configuration here: 0.624 anchor-free for a deficit of 6.6 points against the write-path rung’s 23.7 (§H).
What this does to our oracle. It also bounds our own metric, in our disfavour. keep-latest-oracle has perfect same-fact detection but still retrieves by similarity, so it is
a write-path oracle, not an end-to-end one: at it misses 0.173 of queries outright because the aliased query does not embed near its one retained slot. The symmetric rung, with no oracle knowledge, is statistically indistinguishable from it on current@1 (0.831 vs. 0.827, — a failure to reject, not a demonstration of equivalence) by trading misses for stale answers — its stale rate is 0.122 where the oracle’s is zero. So the keying deficit reported throughout this section is the distance to the best write-path policy, and is a valid comparison within that family, which is the family deployed systems belong to. It is not a floor on the task. At the §M scale the ordering reverts — the oracle leads by 6.3 points — which is the same trade resolving the other way, and is why we report the deficit against a write-path oracle rather than claiming the task is solved.
Finally, and against our own framing: in the anchor-free regime the cheap metadata fix is competitive on current@1 within the write-path family. Time-rerank reaches 0.737 against 0.776 for the entity linker, because reranking needs no keying at all — though it no longer holds against the read-path rungs of this section (0.831 and 0.912), which is one more reason the asymmetry was worth finding. Retention’s advantage there is not top-1 freshness but contamination (0.024 against 0.882) and footprint — and § 5.3 shows contamination is what the reader pays for when no date is shown.
Table 7 decomposes where the anchor-free errors go, and it qualifies the previous paragraph in the benchmark’s own disfavour. The keying-deficit subtraction assumes the retrieval penalty of aliasing is separable from the keying penalty, and that assumption holds for the entity linker — its miss rate is close to the oracle’s, so nearly all of its deficit is stale answers, i.e. genuine keying failure. It does not hold for the gazetteer, whose loss is almost entirely stale, while the oracle’s is entirely miss: those are different failures and subtracting one from the other is informative only about magnitude. Relatedly, the entity linker’s low contamination at is not purely a win: its current@5 is below store-all’s, so part of the “no conflict in the context” is “no answer in the context”.
| policy () | cur@1 ↑ | stale@1 ↓ | miss@1 ↓ | cur@5 ↑ | cont.@5 ↓ | slots ↓ |
| BM25 (store-all) | 0.062 | 0.925 | 0.013 | 0.895 | 0.893 | 995 |
| BM25 + time-rerank | 0.726 | 0.113 | 0.160 | 0.895 | 0.893 | 995 |
| Cosine (store-all) | 0.111 | 0.868 | 0.021 | 0.891 | 0.882 | 995 |
| + time-rerank | 0.737 | 0.105 | 0.158 | 0.891 | 0.882 | 995 |
| + recency-weight | 0.419 | 0.199 | 0.382 | 0.650 | 0.404 | 995 |
| Overwrite: threshold | 0.513 | 0.297 | 0.190 | 0.771 | 0.261 | 561 |
| Overwrite: gazetteer (exact) | 0.169 | 0.808 | 0.024 | 0.893 | 0.784 | 936 |
| Overwrite: fuzzy linker (no alias table) | 0.714 | 0.120 | 0.167 | 0.801 | 0.032 | 467 |
| Overwrite: linker + alias table | 0.776 | 0.053 | 0.171 | 0.865 | 0.024 | 469 |
| Read-path key (no overwrite) | 0.831 | 0.124 | 0.045 | 0.987 | 0.983 | 995 |
| Symmetric key (read+write) | 0.831 | 0.122 | 0.047 | 0.868 | 0.034 | 467 |
| Symmetric key + alias table | 0.912 | 0.058 | 0.030 | 0.932 | 0.024 | 469 |
| Overwrite: true-key oracle | 0.827 | 0.000 | 0.173 | 0.925 | 0.000 | 468 |
Table 7. Where each policy’s errors go in the anchor-free regime (, test, ). contamination@5 is shown with current@5 because it is not monotone (§ 3): the entity linker’s near-zero contamination is partly the absence of any version of the fact, not only the absence of conflict.
The repair is concentrated where facts change least
Table 8: 423 of 468 test facts have exactly two versions; on these, time-rerank scores 0.967. On the 45 facts with versions — the population the introduction’s argument is about, where staleness pressure is highest — time-rerank falls to 0.533 while the threshold keyer (0.778) overtakes it and the oracle stays at 0.956. The mechanism is structural: rerank’s must cover all versions of the queried fact, so the optimal tracks the per-fact version count — knowledge that is itself the same-key problem. The headline 0.925 is a mixture dominated by the easy 90%; the benchmark reports both strata so the mixture cannot mislead.
Each fix reads one temporal signal
Table 10 corrupts the two signals a policy can use to tell a current version from a stale one. Permuting timestamps while keeping arrival order collapses every time-aware fix to store-all level (time-rerank 0.925 → 0.484; recency-weighted 0.784 → 0.451) while overwrite and invalidation retention are unchanged (threshold 0.859, gazetteer 0.972, oracle 0.979) — by construction, since a keyer never reads . It would be easy to stop there and conclude that retention “does not need the clock”. It does not; it needs the other signal. Overwrite keeps whichever version arrived last, so permuting arrival order while every timestamp stays intact — delayed or out-of-order ingestion — collapses the same rungs to the same floor (gazetteer 0.472, oracle 0.472, invalidation 0.472; store-all cosine is 0.464) while the time-aware fixes are untouched (time-rerank 0.925). The read-path rung, which retains everything and sorts the resolved key’s slots by , is the mirror image again: unchanged under disordered arrival (0.989) and collapsed under a permuted clock (0.491). A one-line variant that overwrites only when the incoming timestamp is newer swaps overwrite’s dependency (gazetteer 0.972 under disordered arrival, 0.488 under a permuted clock). With both signals corrupted nothing recovers, because the information is gone. Which rung is best is therefore a property of which signal a deployment can trust: with a clean clock and disordered arrival, the best policy in the table is the most contaminated one.
Full permutation is an extreme, so Table 9 does the realistic form: for a fraction of the 468 test facts the final version arrives before its predecessor — a stale report landing late. Overwrite retention loses its margin over chance in proportion to (gazetteer 0.972 → 0.875 / 0.729 / 0.486 at
| policy | ||||
| + time-rerank | 0.925 | 0.925 | 0.925 | 0.925 |
| Overwrite: gaz. | 0.972 | 0.875 | 0.729 | 0.486 |
| Overwrite: oracle | 0.979 | 0.880 | 0.734 | 0.491 |
| Symmetric key | 0.989 | 0.890 | 0.740 | 0.493 |
| If newer: gaz. | 0.972 | 0.972 | 0.972 | 0.972 |
Table 9. A late stale report: for a fraction of the 468 test facts the final version arrives before its predecessor, timestamps intact (; current@1, mean of 3 draws). Overwrite loses its margin in proportion to with contamination@5 fixed at 0.000: one confidently stale memory, not a contradiction.
| policy | clean | clock permuted | arrival permuted | both |
| Cosine (store-all) | 0.464 | 0.464 | 0.464 | 0.464 |
| + time-rerank | 0.925 | 0.484 | 0.925 | 0.449 |
| + recency-wt | 0.784 | 0.451 | 0.784 | 0.447 |
| Overwrite: threshold | 0.859 | 0.859 | 0.452 | 0.452 |
| Overwrite: gazetteer | 0.972 | 0.972 | 0.472 | 0.472 |
| Invalidate: gazetteer | 0.972 | 0.972 | 0.472 | 0.472 |
| Overwrite: oracle | 0.979 | 0.979 | 0.472 | 0.472 |
| Read-path key (retain all) | 0.989 | 0.491 | 0.989 | 0.452 |
| Symmetric key | 0.989 | 0.989 | 0.477 | 0.477 |
| Overwrite if newer: gazetteer | 0.972 | 0.488 | 0.972 | 0.450 |
| Overwrite if newer: oracle | 0.979 | 0.490 | 0.979 | 0.451 |
Table 10. The two temporal signals, corrupted separately and together (test, ; current@1, mean over 3 permutation seeds; hyperparameters are the val-tuned values of the clean regime, since a deployment cannot retune for a corruption it does not know it has). clock permuted: timestamps permuted across statements, arrival order intact — unreliable logging, imports, clock skew. arrival permuted: arrival order permuted, each statement keeping its own timestamp — delayed or out-of-order ingestion. Every time-aware or retention policy reads one signal and is blind to the other; with both corrupted nothing recovers. The per-seed spread is at most 5.8 points in any cell.
; oracle 0.880 / 0.734 / 0.491), preserves the subject’s surface form. Paraphrase alone therefore stresses only one rung of the ladder. § 5.2 removes the anchor itself.
while time-rerank (0.925 at ) and the newer-only overwrite (0.972) do not move. One detail matters for § 5.3: under this corruption overwrite’s contamination@5 stays at 0.000. It does not hand the reader a contradiction; it hands the reader a single, confidently stale memory, which the constructed-context probe scores at 0.000. Invalidation protects against that only if it invalidates by timestamp rather than by arrival. The division of labour is measured rather than assumed, and it is symmetric: metadata heuristics repair freshness exactly when time metadata is trustworthy; retention repairs it exactly when ingestion is in order.
Paraphrase: the same-key tax
Table 11: as statements shift from canonical to paraphrased templates, the threshold keyer loses 0.887 → 0.829 while the oracle holds (0.998 → 0.970), and the gap widens from 11.1 to 14.1 points. The gazetteer keyer tracks the oracle at every (0.991, 0.972, 0.964) because template-level paraphrase preserves the subject’s surface form. Paraphrase alone therefore stresses only one rung of the ladder. § 5.2 removes the anchor itself.
| policy | current@1 | cont.@5 | current@1 | cont.@5 | current@1 | cont.@5 |
| Cosine (store-all) | 0.447 [0.402,0.491] | 1.00 | 0.464 [0.419,0.506] | 0.97 | 0.468 [0.423,0.513] | 0.98 |
| Cosine + time-rerank | 0.959 [0.940,0.976] | 1.00 | 0.925 [0.900,0.947] | 0.97 | 0.942 [0.921,0.964] | 0.98 |
| Reconsolidation | 0.887 [0.857,0.915] | 0.09 | 0.859 [0.827,0.889] | 0.14 | 0.829 [0.795,0.863] | 0.16 |
| Keep-latest oracle | 0.998 [0.994,1.000] | 0.00 | 0.979 [0.966,0.991] | 0.00 | 0.970 [0.953,0.985] | 0.00 |
Table 11. Surface variation stresses similarity keying: current@1 [95% CI] and contamination@5 on the test split as the paraphrase probability grows. The oracle’s keying is unaffected by construction; the gazetteer keyer (values in the text) is unaffected while subjects stay verbatim.
Keying, measured without retrieval
Retrieval metrics mix keying quality with ranking quality. Table 12 separates them by scoring each keyer’s output directly as a clustering of statements against the true fact partition. The picture matches § 5.2 and sharpens it: withdrawing the anchor scatters 89.3% of facts across multiple slots for the gazetteer (from 0.4%) and 55.3% for the threshold keyer, against 13.2% for the fuzzy linker and 6.6% for the alias-equipped one. Merge errors move the other way — the entity linker creates 25 impure slots at against 2 at — which is the failure that destroys evidence irreversibly under overwrite. Split and merge trade against each other, and no rung but the oracle escapes both.
| keyer | anchored () merge err. | anchored () % split | anchor-free () merge err. | anchor-free () % split |
| Threshold | 40 | 22.6 | 98 | 55.3 |
| Gazetteer (exact) | 2 | 0.4 | 3 | 89.3 |
| Fuzzy linker (no alias table) | 5 | 0.4 | 35 | 13.2 |
| Linker + alias table | 2 | 0.4 | 25 | 6.6 |
| True-key oracle | 0 | 0.0 | 0 | 0.0 |
Table 12. Same-fact decisions scored directly as a clustering of statements against the true fact partition, independent of any retriever. Merge err. counts slots holding statements about more than one fact; % split is the fraction of facts scattered across several slots.
Two properties of this metric should be stated so it is not over-read. A keyer that refuses to link anything scores zero merge errors and near-total splitting by construction, so the gazetteer’s 89.3% at is partly definitional — it cannot match an absent string. And the two columns have different denominators (clusters and facts), so they order keyers rather than trading off on a common scale; a full merge/split operating curve over is left to the released harness.
Mechanism. Table 13 holds the keyer fixed and varies what retention does. With a good keyer, invalidation is free in quality terms: identical current@1 and contamination to overwrite, at 995 slots against 467 because nothing is deleted. With the threshold keyer it is slightly worse (0.821 vs 0.859), since a false merge invalidates a correct statement that then ranks below its own replacement. The design implication for production systems is concrete: tombstoning buys recoverability of false merges at roughly footprint and no accuracy cost — provided the keyer is good enough that invalidations are usually right.
| keyer | overwrite (destructive) current@1 | overwrite (destructive) cont.@5 | overwrite (destructive) slots | temporal KG (invalidate) current@1 | temporal KG (invalidate) cont.@5 | temporal KG (invalidate) slots |
| anchored, threshold | 0.859 | 0.137 | 513 | 0.821 | 0.126 | 995 |
| anchored, entity linker | 0.972 | 0.000 | 467 | 0.972 | 0.000 | 995 |
| anchored, true-key oracle | 0.979 | 0.000 | 468 | 0.979 | 0.000 | 995 |
| anchor-free, threshold | 0.513 | 0.261 | 561 | 0.451 | 0.237 | 995 |
| anchor-free, entity linker | 0.776 | 0.024 | 469 | 0.776 | 0.024 | 995 |
| anchor-free, true-key oracle | 0.827 | 0.000 | 468 | 0.827 | 0.000 | 995 |
Table 13. Retention mechanism at fixed keyer (): invalidation matches overwrite’s accuracy and contamination while keeping every superseded statement recoverable.
A realistic corpus, and the cost of an incomplete inventory
Scaling the store to 7,078 statements (Table 14) leaves the store-all picture intact (cosine 0.462, time-rerank 0.902) and makes the keying rungs harder in two ways.
First, keying is simply harder against 6,083 more neighbours: in the anchor-free configuration the entity linker’s deficit to the oracle roughly doubles
relative to the small corpus, which is why § 5.2’s 5.1 points are the optimistic end of the range. Second, inventory completeness becomes its own axis. An entity linker with an open-world inventory — one that knows the facts it is asked about but not the rest of the corpus — scores 0.902 against 0.947 closed-world, so incompleteness costs 4.5 points on its own, a cost any system deployed on open-domain logs pays by default and one invisible at the scale of § 5.2.
That second effect is modest, and it depends on the matcher. Matching inventory forms as bare substrings inflates it several-fold, because that links 42% of the 6,083 distractor statements into a queried fact through forms like “Democratic” or “John”; the token-boundary mention matching used here (§ 4) links 2.5%. Anything weaker measures the matcher rather than the inventory, and it does so in the direction that flatters this paper’s thesis. Completing the inventory helps both rungs, by 4.5 points for the linker and 0.4 for the gazetteer: with token-boundary matching a larger inventory does not manufacture cross-fact merges, which is precisely what a substring matcher does.
Natural short forms
The mechanical short forms of § 3.3 are what make the anchor settleable, and they are also the axis’s weakest point: every one is a substring of the full name. Table 15 reruns the ladder with the short forms people actually use — 1,176 forms harvested from Wikidata aliases and Wikipedia redirects for the test subjects (VR46, Vale Rossi, misspellings and transliterations included), 6.0% of which the mechanical generator also produces, and 0 of which name two subjects. Two things change and one does not.
Table 15: The anchor-free ladder at with short forms drawn from Wikidata aliases and English Wikipedia redirects instead of generated mechanically (test, ; current@1, and the keying deficit in points against the write-path oracle of the same configuration; every tunable re-tuned on val, whose own inventory covers 288 of its subjects). natural: a subject with no real short form stays anchored (270 of the 467 distinct subjects behind the 468 test facts have one; 59.6% of alias-eligible statements are aliased). natural+fallback: such a subject takes the mechanical forms instead (95.1%), the like-for-like comparison with Table 3.
| policy | mechanical current@1 | mechanical deficit | natural current@1 | natural deficit | natural+fallback current@1 | natural+fallback deficit |
| Overwrite: threshold | 0.513 | +31.4 | 0.656 | +16.5 | 0.506 | +22.0 |
| Overwrite: gazetteer | 0.169 | +65.8 | 0.498 | +32.3 | 0.162 | +56.4 |
| Overwrite: fuzzy linker | 0.714 | +11.3 | 0.733 | +8.8 | 0.573 | +15.4 |
| Read-path key (retain all) | 0.831 | −0.4 | 0.780 | +4.1 | 0.679 | +4.7 |
| Symmetric key | 0.831 | −0.4 | 0.780 | +4.1 | 0.679 | +4.7 |
| Overwrite: linker + alias table | 0.776 | +5.1 | 0.812 | +0.9 | 0.701 | +2.6 |
| Symmetric key + alias table | 0.912 | −8.5 | 0.985 | −16.5 | 0.959 | −23.3 |
| Overwrite: oracle | 0.827 | +0.0 | 0.821 | +0.0 | 0.726 | +0.0 |
Table 15.
Natural forms are harder for retrieval and easier for keying. A nickname shares no token with the full-name query, so even the write-path oracle drops (0.827 mechanical, 0.821 natural, 0.726 with fallback) and store-all cosine moves with it. But handed the natural inventory, the entity linker’s deficit shrinks from 5.1 to 0.9 points, because natural forms have no cross-subject collisions where the mechanical ones have 55. The rungs deployed systems use still pay tens of points: under fallback the threshold keyer’s deficit is 22.0 and the gazetteer’s 56.4 (31.4 and 65.8 mechanical); the milder natural column is a coverage effect, since 42% of subjects stay anchored there.
What does not survive is the headline of § 5.2 in its strong form. Ordinary resolution applied symmetrically no longer beats a complete alias table applied asymmetrically: the symmetric fuzzy rung scores 0.780 against the entity linker’s 0.812 on natural forms and 0.679 against 0.701 with fallback, where the trigram linker leaves 88 statements unlinked. The point estimates favour the table; neither difference is significant (raw and ), so what the natural forms establish is that the mechanical advantage is gone, not that it has reversed. Trigram similarity cannot link VR46 to Valentino Rossi; an alias table can. What survives is the weaker and more useful claim: resolving the query helps whatever resolver is used. The symmetric rung beats its write-path-only twin on both configurations ( and , Holm), and the entity linker applied symmetrically reaches 0.985 natural and 0.959 with fallback ( against its write-path-only form) — the best number on either configuration, 16.5 points above the write-path oracle. § 5.2’s statement is therefore scoped to the mechanical axis, and both configurations ship. One circularity carries over from the mechanical axis: the linker is handed the inventory the stream drew from, so its number is the ceiling of a deployment whose alias table is complete.
Can a language model do the keying?
The keying ladder so far is mechanical, and its cheap rungs fail in the anchor-free regime. The obvious next move — the one production memory systems now make, from Mem0’s LLM-decided UPDATE ([4]) to RL-learned memory operations ([32]) — is to let a language model adjudicate. This rung tests that directly (Table 16), and the answer is negative in a way worth reporting.
Table 16: A language model as the same-fact keyer, against the mechanical rungs (test, ). For each arriving statement the model is shown the top-4 stored candidates by cosine and asked which, if any, describes the same fact; it sees statement text only — never an id, a key, or a timestamp. % split is the retrieval-independent share of facts scattered over more than one slot (§G).
| keyer | current@1 anchored | current@1 anchor-free | % facts split anchored | % facts split anchor-free |
| threshold | 0.859 | 0.513 | 22.6 | 55.3 |
| gazetteer | 0.972 | 0.169 | 0.4 | 89.3 |
| linker + alias table | 0.972 | 0.776 | 0.4 | 6.6 |
| true-key oracle | 0.979 | 0.827 | 0.0 | 0.0 |
| LLM keyer (glm-4.6) | 0.846 | 0.459 | 27.6 | 52.4 |
Table 16.
The model underperforms every mechanical keyer when the anchor is present, and lands mid-ladder when it is not. Anchored, glm-4.6 reaches 0.846 against 0.972 for the gazetteer and 0.972 for the entity linker, leaving 13.2 points to the oracle — worse than every mechanical rung including the embedding threshold. Anchor-free it reaches 0.459, which beats the collapsed gazetteer (0.169) but sits below the embedding threshold (0.513) and far below the entity linker (0.776). The clustering audit shows the failure mode: it scatters 27.6% of facts across slots when anchored and 52.4% when not, against 6.6% for the linker, while creating very few impure slots (10 at ). The model is not confusing facts; it is declining to merge them.
The comparison is not resource-matched, and the direction of the mismatch matters. The mechanical rungs are handed a canonical subject inventory — for the entity linker, an alias table as well. The model is handed neither: it sees the arriving statement and the candidate statements as text, and must decide same-fact from that alone. So this is not a claim that language models are worse same-fact detectors than trigram matching under equal information; it is a measurement of the configuration people actually deploy, in which the LLM is asked to adjudicate without a gazetteer. An inventory-equipped LLM keyer is an obvious and unrun experiment, and the plausible result is that it closes some of the gap. What the measurement does establish is that a language model asked for a one-shot same-fact decision does not replace the inventory, which is the substitution the literature proposes.
This is not a retrieval failure. The obvious objection is that the right candidate was never shown. It usually was: for statements whose fact already occupies a slot, that slot is among the 4 offered candidates 0.970 of the time when anchored and 0.861 of the time when not. The distance between that ceiling and the model’s linking rate is judgment, not recall.
What this does and does not license. It is one prompt, one candidate set size, and 1 model, so it bounds LLM keying from below rather than above — a different prompt, more candidates, or a model fine-tuned for the task would plausibly do better, and the released harness makes that a one-line experiment. What it does establish is that delegating same-fact detection to a general-purpose model is not automatically an improvement over a gazetteer, and that it is not cheap: 994 model calls for 995 statements — one per arriving statement, against zero for every mechanical rung — to land below the entity linker in both regimes. On this evidence it is entity resolution machinery, not model scale, that closes the anchor-free gap of § 5.2.
Where a shipped system lands on the ladder
Everything above prices same-fact detection with rungs we built. The obvious question is where the systems people deploy actually sit, and § 2 names them — Mem0’s UPDATE operation, Zep’s edge invalidation, MemGPT’s self-editing memory — as evidence that the field already ships the fix. We ran Mem0 first, and the first thing we found was that the answer depends on which release you install.
The adjudication is not on the default ingestion path in the current release. Mem0’s documented loop is exactly the decision this paper prices: an LLM sees an incoming statement with the nearest stored memories and returns ADD, UPDATE, DELETE or NOOP. In the 0.1.x line that is what add() does — it calls get_update_memory_messages and can emit any of the four. In the 2.0.x line add() instead runs a phased extraction whose system prompt states that its sole operation is ADD, and the update-adjudication prompts have no call site anywhere under mem0/memory/; the ADD/UPDATE decision is reachable only through the separate update(memory_id, ...) API, which an ingestion pipeline does not call. So on 2.0.11 an UPDATE count of zero is fixed before any data is streamed, and reporting it as a property of our data would be a misattribution. We report it as what it is: a property of the release.
That distinction is easy to miss and expensive to get wrong, so the harness records lib_version and an adjudicates_on_add flag derived by inspecting the installed package, and we score both lines.
The consequence is visible in 2 statements. Stream “Peter Sagan plays for Bora-Hansgrohe.” then “Peter Sagan plays for TotalEnergies.”, ask which team he plays for now, and hold everything else fixed — same wrapper, same model, same encoder. 0.1.118 returns ADD 1, UPDATE 1, keeps 1 memory, and ranks the stale one nowhere: rank 1 is the current one. 2.0.11 returns ADD 2, keeps 2, and ranks the stale one first. That is the whole finding, before any benchmark is involved, and it is why the rest of this section reports a version rather than a system.
On 2.0.11, the system is a store-all policy. Its 329 ingestion calls over 329 statements produce 857 extraction results — more results than statements, because extraction splits a statement into several memories — and every one of them is an ADD (0 UPDATE). The store ends at 857 slots, and retrieval from it reaches current@1 0.180 at contamination@5 0.720: below plain cosine (0.460) over the identical statements with the identical encoder. It is not failing to retrieve the fact — current@5 is 0.820 — but failing to rank it first, because extraction rewrites each statement into derived memories and what the retriever scores is no longer the sentence whose currency the source defined. A system can lose freshness by paraphrasing its own inputs.
On 0.1.118, where the adjudication is on the add() path, it fires. The same streams, the same model, the same encoder, the same harness: 149 UPDATE and 31 DELETE decisions alongside 269 ADD, and a store that ends at 238 slots rather than 857. Withdraw the lexical anchor and it makes 114 UPDATE decisions instead of 149: a production adjudicator is not exempt from the axis § 5.2 varies, and its store grows accordingly. Those counts come from the system’s own report and are the least ambiguous thing in this section. We repeated the whole cell — same release, same backend, same model, fresh store — and it reproduced identically: the event histogram, the 238 slots and current@1 0.913/0.913 all landed on the same values, a difference of 0.0 points.
How well it works, though, depends entirely on how you score it — and that turned out to be the finding. Our convention attributes a returned memory to the statement that most recently wrote it. On 0.1.118 that convention reports current@1 0.913 at contamination@5 0.120, which reads as a retention policy whose keying works. Scoring the same contexts by what they say — does the surfaced memory name the current value of the queried fact, or a superseded one — reports 0.573 at 0.593. The two disagree by 34.0 points on rank 1 and 47.3 on contamination, in the direction that flatters the system.
The control says the disagreement is not the scorer. On the same stream, the 7 idealized rungs — which return the corpus verbatim and so must agree — differ by at most 2.0 points between the two scorings, with 0 ambiguous attributions between them. The deployed system differs by 34.0.
The mechanism is not the merge we expected, and it runs in both directions. The natural guess is self-contradicting merges — one memory naming both values — but no memory does: hedged@1 by text is 0.000. What happens on 0.1.118 is that an UPDATE relabels a memory without rewriting it, so a current version’s id comes to sit on superseded text. 81 of 150 queries surface two distinct memories collapsed onto one benchmark id, and 84 surface a memory the system marked as updated whose text still reads as the superseded value. Rank 1 and contamination@5 both call that clean, because both identify a version by an id and the ids now agree.
On 2.0.11 the same convention fails the other way, and harder. 421 of 681 surfaced memories — 62% — carry an id naming a different subject than their own text, against 1% on 0.1.118. The consequence is stark: the id-level miss rate is 0.533 while the text-level miss rate is 0.000. The system returns a memory about the queried fact essentially every time, and our wrapper cannot say which fact it is. Every id-level number for that release is a statement about our bookkeeping. We do not claim a mechanism: a six-statement probe of the same release attached metadata correctly, so whatever produces this does not appear at that size, and what we report is measured over the run’s own memories, where subject-named-in-text against subject-of-id needs no model call to check.
So the id-level headline does not survive. By id, current @1 between the two releases reads 0.913 → 0.180. By text it reads 0.573 → 0.480. A 73-point collapse becomes a single-digit one, and the gap was manufactured by a convention that is too kind to one release and broken on the other. This is the instrument catching a claim its own id-level numbers invite, the case § 5.3 makes in general.
The size of the disagreement is a property of the adjudicator model as well as of the convention: with glm-4.6 in the loop the two scorings agree (below, Table 21).
| system, model, prompt | anchored | anchor-free | |||||||||
| cur@1 | text | cont.@5 | slots | ops | cur@1 | text | cont.@5 | slots | ops | ||
| Mem0, qwen, default | 0.913 | 0.573 | 0.120 | 238 | 149+31 | 0.653 | 0.207 | 0.320 | 286 | 114+21 | |
| Mem0, glm, default | 0.067 | 0.060 | 0.000 | 20 | 2+1 | 0.153 | 0.173 | 0.007 | 36 | 7+1 | |
| Mem0, qwen, third-person | 0.960 | 0.587 | 0.100 | 251 | 155+23 | 0.627 | 0.200 | 0.393 | 303 | 110+12 | |
| Mem0, glm, third-person | 0.793 | 0.793 | 0.007 | 138 | 142+19 | 0.627 | 0.627 | 0.080 | 150 | 113+23 | |
| Graphiti, qwen, as returned | 0.347 | 0.347 | 0.940 | 164/321 | 157 | 0.180 | 0.180 | 0.820 | 169/306 | 137 | |
| Graphiti, qwen, valid first | 0.833 | 0.833 | 0.100 | 164/321 | 157 | 0.640 | 0.640 | 0.133 | 169/306 | 137 | |
| Graphiti, glm, as returned | 0.347 | 0.353 | 0.953 | 158/320 | 162 | — | — | — | — | — | |
| Graphiti, glm, valid first | 0.860 | 0.867 | 0.073 | 158/320 | 162 | — | — | — | — | — |
Table 21.
We verified that this is the system and not our backend, because that distinction decided the result in both directions. Every (release, vector store) pair is hand-probed before its numbers are used: one query, the embedder scored by hand, checked against what the store reports holding and against whether the system can record its own decisions at all. Of the pairs we tried, 4 of 6 pass (Table 18); we report no retrieval metrics from the rest, because a failing pair still produces plausible-looking numbers — one returns 2 of the 14 memories it holds and ranks none of them the one the embedder scores first; and one records no decision for any write while still persisting some of them — that describe our configuration rather than the system. A passing write gate shows only that the system recorded its own decisions, not that it made them well: on a pair whose read gate fails, the adjudicator is choosing against the candidates that backend returned, and its event counts differ from the same release on a passing backend accordingly. Both release lines above are quoted from chroma, so the comparison is same-backend and not an inference across backends. The measurement is also stable across backends: 2.0.11 spans 0.180–0.207 on current@1 over 2 of the stores that pass, a range of 2.7 points — far smaller than any difference this section turns on.
Table 18. Every (release, vector store) pair we probed, and whether its numbers are usable. writes: the system can record its own ADD/UPDATE decisions. retrieval: a hand-scored query returns the right fact first from what the store holds. Two pairs fail, in opposite halves, and both produce plausible-looking metrics that measure the backend rather than the system — which is why the selection of runs quoted in this section is made by this table and not by which result file was written last.
| mem0 release | vector store | writes | retrieval | returned | usable |
0.1.118 | chroma | ✓ | ✓ | 6 | ✓ |
0.1.118 | faiss | ✓ | × | 2 | × |
0.1.118 | qdrant | × | ✓ | 6 | × |
2.0.11 | chroma | ✓ | ✓ | 14 | ✓ |
2.0.11 | faiss | ✓ | ✓ | 14 | ✓ |
2.0.11 | qdrant | ✓ | ✓ | 14 | ✓ |
This does not overturn §J, and the difference is instructive. That section found a language model asked to adjudicate same-fact in one shot to be worse than every mechanical rung; this one finds a purpose-built adjudication loop, running a smaller model, doing rather well. Four things differ and we can separate none of them here: the loop (a system that maintains its own candidate set, retries and can DELETE, against a single classification call), the model, the query set (150 facts against 468), and the attribution convention above. The reading we can support is the one §J already states — that it bounds LLM keying from below rather than above — and §K is the empirical witness that the bound is loose. What survives both sections is narrower: a language model asked for a one-shot same-fact decision does not replace an entity inventory, and anchored, the cheapest mechanical rung that has one still wins.
Every comparison in this section is paired by query on a single stream, so the sign-flip test used throughout § 5.2 and § 5.3 applies unchanged; retrieval-side comparisons are Holm-corrected over one family of 62 covering both scorings, of which 46 survive at , on queries.
Neither scoring is the end-task number, so we asked the reader. Table 19 hands each policy’s own top-5 — for the system, its own rewritten memories in its own words — to qwen2.5:14b, and scores the answer against the source-defined current value. Dates are never surfaced in this track: the system returns no validity timestamp, only the store’s own created and updated times, and after an UPDATE a memory keeps its original writer’s metadata — so the with-dates arm that closes the controlled pair in § 5.3 is not available to a deployment reading this system’s output. On 0.1.118 anchored, the id metric says 0.913, the text metric says 0.573, and the reader answers 0.720 correctly: above rank-1 text accuracy because the current value is usually somewhere in the five (0.920), below the id metric because it is often not first. Both retrieval conventions are proxies, and they err in opposite directions.
| context source | as retrieved cur@1 | as retrieved reader | + query resolved cur@1 | + query resolved reader | + merged, freshest cur@1 | + merged, freshest reader | oldest reader |
| cosine-rag | 0.460 | 0.573 | 1.000 | 0.293 | 1.000 | 1.000 | 0.007 |
| time-rerank | 0.920 | 0.300 | 1.000 | 0.293 | 1.000 | 1.000 | 0.007 |
| reconsolidation | 0.893 | 0.920 | 0.973 | 0.893 | 0.973 | 0.973 | 0.820 |
| lexical-key | 0.973 | 0.987 | 0.987 | 0.987 | 0.987 | 0.987 | 0.987 |
| fuzzy-key | 0.987 | 1.000 | 1.000 | 1.000 | 1.000 | 1.000 | 1.000 |
| fuzzy-rw-key | 1.000 | 1.000 | 1.000 | 1.000 | 1.000 | 1.000 | 1.000 |
| keep-latest-oracle | 0.987 | 1.000 | 1.000 | 1.000 | 1.000 | 1.000 | 1.000 |
| Mem0 0.1.118 | 0.913 | 0.720 | 0.973 | 0.573 | 0.973 | 0.900 | 0.353 |
Table 19. Each policy’s own top-5 handed to qwen2.5:14b (anchored, ; rows carry the harness’s policy ids — reconsolidation is the threshold keyer, lexical-key the gazetteer, fuzzy-key the fuzzy linker, fuzzy-rw-key symmetric keying; the system’s row is its own rewritten memories in its own words, from the 0.1.118 run — 2.0.11 is in the text). as retrieved is what a deployment gets; the next two pairs apply this paper’s read-path fix post hoc to those same contexts, resolving the query through the keyer of § 5.2 and then keeping one memory per resolved entity, at no ingestion and no model call. The two middle cur@1 columns are identical wherever the readers beside them differ by up to 1.000 against 0.293: the retrieval metric cannot see the difference the reader is made of. The last column keeps the oldest memory per entity instead of the freshest. Because facts arrive as they become true, freshest-is-current is free here (150/150 facts); withdrawing it leaves only what each write path had already removed, so the column reads near zero for anything that stores every version and near one for anything that merges on write. It is a write-path purity measure that uses no identifiers and no timestamps in its favour, and it is where the deployed system is priced.
The releases separate far less at the reader than at any retrieval metric. As deployed, 0.1.118 answers 0.720 against 2.0.11’s 0.627 anchored — and anchor-free the order reverses, 0.500 against 0.580. The release without the adjudication is not the disaster the id metric reported.
And our own fix, applied to a shipped system, is worth more than the adjudication is. § 5.2 argues that resolving the query through the same keyer as the stream costs nothing and buys double digits. Nobody had applied it to a deployed system, so we did — post hoc, over the contexts the system already produced, with no change to its write path, no second ingestion and no model call. Resolving the query and then merging same-entity memories, freshest kept, lifts 0.1.118 from 0.720 to 0.900 anchored (+18.0 points, ) and from 0.500 to 0.853 anchor-free (+35.3). On 2.0.11 — the store-all release — it reaches 0.993, above what the superseding release achieves unaided, because a store that keeps everything still holds the current value and only needs to be read correctly.
That last number is not an accuracy claim, and this benchmark is why. Facts here arrive as they become true, so the final version of a fact is a current one for 150 of 150 facts in this subsample: on the clean condition a recency tie-break is correct for free, the metric ceiling is 1.000, and the keep-latest oracle of § 5.2 scores 0.987 on the same stream. The merged arm is fuzzy keying plus keep-latest applied to a system’s output, so it inherits that gift and lands where the oracle already sits. How much survives corrupted timestamps is §E’s regime, which we do not run this arm through.
Running the merge backwards prices the write path, and that is the number to keep. Hold the entity resolution fixed and keep the oldest memory per entity instead of the freshest. Everything recency was giving away is now withdrawn, and what remains is whatever the system’s own write path already did. Plain cosine collapses to 0.007, as it must: it stores every version, so the oldest is the first. 2.0.11 collapses with it, to 0.007 — indistinguishable from store-all, which is the reader-side confirmation that its ingestion path performs no supersession at all, arrived at without reference to any identifier. 0.1.118 does not collapse: it holds 0.353 anchored and 0.140 anchor-free, because its adjudicator has already merged a fact to one memory the oldest and the freshest are the same memory and it is the right one. That gap — 0.353 against 0.007 — is what the ADD/UPDATE loop is worth once the read path is held identical and the free recency signal is taken away. It is the cleanest positive measurement of a deployed supersession mechanism in this paper, it is much smaller than the id-level metric advertised, and it degrades on the paper’s own axis when the anchor is withdrawn.
That fix also decomposes into two halves a retrieval metric cannot tell apart, which is the cleanest statement of this paper’s thesis. Resolving the query alone raises the system’s current@1 to 0.973 and drops the reader to 0.573 (), because promoting every version of the queried fact fills the five slots with the fact contradicting itself. Adding the merge holds current@1 at 0.973 and takes the reader to 0.900. The store-all control is starker still: cosine’s contexts score current@1 1.000 under both arms and read 0.293 against 1.000. Timestamp reranking makes the same point unaided — current@1 0.920, reader 0.300. Reader-side comparisons are Holm-corrected over a family of 220, of which 92 survive.
The ladder still beats it, on the same stream, with no model at all. Table 17 scores our own rungs on the identical 150-fact stream, which matters because this subsample is easier for them than the full test split and a cross-table comparison would understate the gap. Symmetric keying — a canonical subject list, character-trigram matching, no alias table, no learning, no LLM call — reaches 1.000 anchored and 0.920 anchor-free at contamination@5 0.000/0.027 in 150 slots. Scored by text, the system reaches 0.573/0.207 at 0.593/0.747 in 238/286 slots, for 4.2 hours of local LLM inference across the two quoted runs (one per release, two configurations each). The expensive general mech- 恒一 anism is beaten by the cheap specific one, and the same cheap mechanism bolted onto the expensive one afterwards is what rescues it.
Table 17. Mem0 scored under the same harness as the idealized rungs, on the same streams, with the same encoder (test subsample, facts, ; anchored unless the column says otherwise; contamination@5 by id). The rightmost column is the system’s own decision histogram over its 329 ingestion calls. The library version is part of the result, not provenance: see the text.
| policy / system | version | cur@1 (anch.) | cur@1 (anch.-free) | cur@5 | cont.@5 | slots | ADD / UPDATE |
| Cosine (store-all) | -- | 0.460 | 0.140 | 0.987 | 0.973 | 329 | — |
| + time-rerank | -- | 0.920 | 0.787 | 0.987 | 0.973 | 329 | — |
| Overwrite: threshold | -- | 0.893 | 0.633 | 0.973 | 0.147 | 173 | — |
| Overwrite: gazetteer | -- | 0.973 | 0.207 | 0.987 | 0.000 | 150 | — |
| Overwrite: fuzzy linker | -- | 0.987 | 0.820 | 1.000 | 0.000 | 150 | — |
| Symmetric key (read+write) | -- | 1.000 | 0.920 | 1.000 | 0.000 | 150 | — |
| Mem0 (qwen2.5:14b) | 2.0.11 | 0.180 | 0.120 | 0.820 | 0.720 | 857 | 857 / 0 |
| Mem0 (qwen2.5:14b) | 0.1.118 | 0.913 | 0.653 | 0.987 | 0.120 | 238 | 269 / 149 |
| Overwrite: true-key oracle | -- | 0.987 | 0.887 | 1.000 | 0.000 | 150 | — |
So the mechanism the literature cites left the default path, and it matters less than the id-level numbers say. Between two releases of one library, with no change to the documented interface, the adjudication moved off add() and extraction began splitting one statement into several memories. What that costs depends on where you measure. The store grows 238 → 857 slots and contamination by text rises 0.593 → 0.947: those are consequences of storing everything and they are not in dispute. Rank-1 accuracy is a different story — 0.913 → 0.180 by id, 0.573 → 0.480 by text, and 0.720 → 0.627 at the reader, with the order reversing anchor-free. The first of those three is the number a leaderboard would report, and it is the least reliable.
That is the argument. A benchmark that scores rewriting memory systems by attributing returned memories to source records will misrank them, and the failure is not conservative: it flattered one release, broke on another, and inflated a nine-point difference into a 73-point one. Nothing in the id-level numbers announced this. It took scoring the same contexts by their content, and then putting a reader at the end, to see it — which is the case this paper makes for both, now made against itself.
The finding is not that Mem0 is poorly built; it is a widely used system and the 0.1.x behaviour is the one its documentation describes. Nor is it that the release without adjudication is fine: it holds 857 memories for 329 statements and hands a reader a context naming both a current and a superseded value 0.947 of the time. It is that the mechanism the literature cites as the answer to stale retrieval is absent from the path a practitioner installs, that its measured benefit where it does run is far smaller than a retrieval metric suggests, and that improvements of 18.0 and 36.7 points were available to the two releases from a wrapper with no model in it.
A second system, and the other mechanism. Graphiti ([24]), the open-source engine behind Zep, is the invalidation design our temporal-KG rung idealizes: a model extracts entities and relation edges from each statement, and an edge a later statement contradicts is marked invalid_at rather than deleted. We run graphiti-core 0.29.2 on FalkorDB with the same adjudicator model as Mem0 (qwen2.5:14b, through the library’s JSON-mode client) and the same MiniLM encoder as the dense baselines, on the identical 150-fact streams, under the same probe gate. On the two-statement probe invalidation fires and the current version ranks first. On a six-statement probe every write is stored and invalidation fires, and the current version is not first as returned: the library’s default search returns invalidated edges beside valid ones and breaks ties among versions of one fact by insertion order. The library does expose a validity filter on search; the default search does not apply it, and a deployment that calls search() as documented gets the un-filtered rank. The probe already contains the finding; Table 20 measures it.
| policy | anchored cur@1 | anchored by text | anchored cont.@5 | anchored reader | anchor-free cur@1 | anchor-free by text | anchor-free cont.@5 | anchor-free reader |
| Graphiti, as returned | 0.347 | 0.347 | 0.940 | 0.593 | 0.180 | 0.180 | 0.820 | 0.433 |
| Graphiti, valid edges first | 0.833 | 0.833 | 0.100 | 0.893 | 0.640 | 0.640 | 0.133 | 0.773 |
| Graphiti, valid edges only | 0.833 | 0.833 | 0.100 | — | 0.640 | 0.640 | 0.133 | — |
| Cosine (store-all) | 0.460 | 0.460 | 0.973 | 0.573 | 0.140 | 0.140 | 0.927 | 0.427 |
| + time-rerank | 0.920 | 0.920 | 0.973 | 0.300 | 0.787 | 0.787 | 0.927 | 0.220 |
| Symmetric key | 1.000 | 1.000 | 0.000 | 1.000 | 0.920 | 0.920 | 0.027 | 0.920 |
| Overwrite: oracle | 0.987 | 0.987 | 0.000 | 1.000 | 0.887 | 0.887 | 0.000 | 0.953 |
Table 20. The second deployed system. Graphiti (graphiti-core 0.29.2, FalkorDB, qwen2.5:14b adjudicator, MiniLM encoder) on the identical 150-fact streams, scored by id, by text and at the reader (qwen2.5:14b, own top-5), by three read paths over the same stored graph: the library’s default search (as returned), the same hits with non-invalidated edges ranked first (the temporal-KG rung’s read path), and invalidated edges dropped. Ladder rows are the same stream; contamination@5 is by id here and by text in Table 5. An empty reader cell means that arm was not run.
| system | anchored cur@1 | anchored by text | anchored cont.@5 | anchored reader | anchor-free cur@1 | anchor-free by text | anchor-free cont.@5 | anchor-free reader |
| Mem0 2.0.11 | 0.180 | 0.480 | 0.947 | 0.627 | 0.120 | 0.153 | 0.820 | 0.580 |
| Mem0 0.1.118 | 0.913 | 0.573 | 0.593 | 0.720 | 0.653 | 0.207 | 0.747 | 0.500 |
| Graphiti, as returned | 0.347 | 0.347 | 0.947 | 0.593 | 0.180 | 0.180 | 0.820 | 0.433 |
| Graphiti, valid edges first | 0.833 | 0.833 | 0.120 | 0.893 | 0.640 | 0.640 | 0.153 | 0.773 |
| LangMem, qwen2.5:14b | 0.773 | 0.773 | 0.173 | 0.833 | 0.340 | 0.413 | 0.213 | 0.600 |
| LangMem, glm-4.6 | 0.800 | 0.147 | 0.753 | 0.613 | 0.627 | 0.160 | 0.640 | 0.413 |
| ERASE, qwen2.5:14b | 0.853 | 0.860 | 0.100 | 0.867 | 0.733 | 0.707 | 0.160 | 0.807 |
| ERASE, glm-4.6 | 0.953 | 0.953 | 0.060 | 0.980 | 0.713 | 0.713 | 0.240 | 0.813 |
| Cosine (store-all) | 0.460 | 0.460 | 0.973 | 0.573 | 0.140 | 0.140 | 0.927 | 0.427 |
| + time-rerank | 0.920 | 0.920 | 0.973 | 0.300 | 0.787 | 0.787 | 0.927 | 0.220 |
| Symmetric key | 1.000 | 1.000 | 0.007 | 1.000 | 0.920 | 0.920 | 0.027 | 0.920 |
| Overwrite: oracle | 0.987 | 0.987 | 0.007 | 1.000 | 0.887 | 0.887 | 0.000 | 0.953 |
Table 5. Three deployed systems and ERASE, reimplemented, under the rungs’ harness, streams and encoder (test subsample, facts, ). cur@1 attributes the top memory to its source statement, by text scores what it says, cont.@5 is contamination@5 by text, and reader is qwen2.5:14b answering from the system’s own top-5. The Mem0 and Graphiti rows have qwen2.5:14b in the loop. The lower block is the ladder on the same streams. Per-system tables, gates and the second adjudicator are in Appendix § 19.
The write path works. Over the 329 anchored statements the system makes 157 invalidations against 179 supersessions in the stream and ends with 164 active edges of 321 (9 statements yield no edge), at 3.9 model calls per statement and 3.6 hours of local inference: the mechanism Mem0’s current release has lost is present here, on the default ingestion path. The read path ignores it. As returned, the system reaches current@1 0.347 by id and 0.347 by text — the two agree, because Graphiti’s edge text is a faithful rewrite of the statement — at contamination@5 0.940, below plain cosine over the same statements (0.460), and it ranks an invalidated edge above a valid one for 96 of 150 queries. Ranking the system’s own non-invalidated edges first — the read path of the temporal-KG rung, applied post hoc to what the system returned, with no model call — lifts it to 0.833 at contamination@5 0.100 (): most of the gap to the write-path rungs of Table 17 on current@1, and nearly all of it on contamination. Anchor-free the pattern holds and the axis reaches the system’s own entity resolution: 0.180 as returned against 0.640 valid-first, 137 invalidations against 179 supersessions, and 538 entity nodes created for the same 150 subjects against 445 anchored, because a short form the extractor fails to link becomes a new entity and 24 statements yield no edge at all (9 anchored). At the reader, qwen2.5:14b answers 0.593 from the system’s own top-5 and 0.893 from the valid-first top-5 (0.433 and 0.773 anchor-free), against 0.720 for Mem0’s adjudicating release as deployed; the paper’s read-path fix, resolving the query and merging same-entity memories, takes it to 0.967. The oldest-per-entity control that priced Mem0’s write path reads 0.033 here, as it must: invalidation deletes nothing, so withdrawing the recency signal leaves the oldest version of every fact, and what the mechanism is worth is read on the valid-first arm instead.
The finding is the mirror image of Mem0’s. There the adjudication had left the ingestion path of the current release; here it is present, fires, and is discarded by the default read path, so the system scored as installed sits below store-all and the system scored with its own verdicts respected closes most of the gap to the retention rungs on current@1 and nearly all of it on contamination. Both are the same lesson for a leaderboard: the number depends on which code path the practitioner gets, and a benchmark must say which one it scored.
A second adjudicator model. The runs above share one model, so “this system does not supersede” and “the model we gave it does not” were not separated. Table 21 separates them, and the first thing it separates is a stage before the adjudication. Mem0’s stock extraction prompt asks for facts about the user; glm-4.6 reads it literally and returns no facts for 307 of 329 third-person statements, so the store ends at 20 slots with 2 UPDATE decisions and current@1 0.067: as installed, the stronger model empties the system. qwen2.5:14b extracts from the same prompt (329 statements, 3 silent). The comparison the question needs therefore has to hold the extraction fixed. A replacement prompt that admits facts about any named entity — a documented configuration option, not a change to the library — leaves qwen2.5:14b where it was (0.960 by id and 0.587 by text against 0.913 and 0.573; 155 UPDATE decisions against 149), and under it glm-4.6 makes 142 UPDATE and 19 DELETE decisions, ends at 138 slots — nearly one per fact — and scores 0.793 by id and 0.793 by text at contamination@5 0.007 by id and 0.027 by text (anchor-free 0.627 / 0.627). Two things follow. The model matters where the adjudication runs: by text, the current value ranks first for 0.793 of queries against 0.587 with the smaller model, and the context contradicts itself 0.027 of the time against 0.653. And the attribution disagreement this appendix made a finding of is itself model-dependent: the larger model’s UPDATE rewrites the memory’s text to the current value, so id and text agree, where the smaller model’s UPDATE relabels a memory it does not rewrite. At the reader the two arms read 0.807 (glm-4.6) against 0.673 (qwen2.5:14b) from their own top-5, and 0.807 against 0.927 once the query is resolved and same-entity memories merged: the paper’s read-path fix recovers what the smaller adjudicator left undone and finds nothing left to recover on the larger one. Graphiti’s extraction prompts are entity-centric, so the model swap reaches its adjudication directly: with glm-4.6 it makes 162 invalidations against 157 with qwen2.5:14b, ends with 158 active edges of 320, and scores 0.347 as returned and 0.860 with valid edges first (0.347 and 0.833 with qwen2.5:14b), at 4.0 model calls per statement and 5.7 hours of hosted inference — the model changes almost nothing, because the read path that discards the invalidations is the same. The anchor-free configuration is not reported for this model: the library issues its extraction calls concurrently, the hosted endpoint admits one request in flight, and the run dropped statements on rate limits, which is a property of our account rather than of the system. What the swap establishes is narrower than a verdict on either model: the number a deployment gets moves with the model at the extraction stage before it moves at the adjudication, and a leaderboard must Table 21: A second adjudicator model. Mem0 0.1.118 (chroma) under its stock, user-centric extraction prompt and under a prompt that admits facts about any named entity, each driven by qwen2.5:14b and by glm-4.6; Graphiti with both models, as returned and with valid edges first. Same 150-fact streams as Table 17; contamination@5 by id. slots for Graphiti is active/total edges; the last column counts UPDATE+DELETE decisions (Mem0) or invalidations (Graphiti).
pin the model as it pins the version.
A third system, from the same mechanism family as Mem0. LangMem ([14]) is LangChain’s long-term memory: the model reads the incoming statement beside the stored memories that resemble it and returns, in one structured-output call, the memories to insert or patch — Mem0’s ADD/UPDATE loop from a different vendor, with deletes off by default. If the lesson of Mem0 is about the mechanism rather than the library, it should show here, and what shows is a third way for the same mechanism to leave a context stale (Table 22). On the probe every write is stored, the update fires, and the current version ranks first. Over the 329 anchored statements the system makes 327 inserts and 375 updates (16 statements change nothing) at 1.3 model calls per statement and 3.1 hours, and ends at 327 slots: it rewrites the memories an incoming statement resembles and inserts a memory for the statement, so a superseded value is overwritten in place while the footprint stays that of store-all, and the store also holds memories no statement wrote (“the user is likely interested in cycling”, a confidence annotation appended to a fact). The library logs and skips 61 patches the model emitted that could not be applied. By id it reaches current@1 0.773 at contamination@5 0.140; by text, 0.773 at 0.173 — the two agree, because an update here rewrites the text — with a miss rate of 0.147, since the memory a query lands on is sometimes the library’s own gloss on the fact rather than the fact; the reader answers 0.833 from its own top-5 and 0.920 once the query is resolved and same-entity memories merged. Anchor-free the mechanism stops helping where the read path cannot find the memory: cur-
Table 22: The third deployed system. LangMem 0.0.30 (LangChain’s long-term memory: one structured-output call per statement returns the memories to insert or patch; inserts and updates on, deletes off, as installed) on the identical 150-fact streams with the benchmark’s MiniLM encoder as its store index, scored by id, by text and at the reader (qwen2.5:14b, own top-5). Ladder rows are the same stream. contamination@5 is by text, since the library rewrites what it stores and adds memories no statement wrote.
| policy | anchored | anchor-free | ||||||
| cur@1 | by text | cont.@5 (text) | reader | cur@1 | by text | cont.@5 (text) | reader | |
| LangMem, qwen2.5:14b | 0.773 | 0.773 | 0.173 | 0.833 | 0.340 | 0.413 | 0.213 | 0.600 |
| LangMem, glm-4.6 | 0.800 | 0.147 | 0.753 | 0.613 | 0.627 | 0.160 | 0.640 | 0.413 |
| Cosine (store-all) | 0.460 | 0.460 | 0.973 | 0.573 | 0.140 | 0.140 | 0.927 | 0.427 |
| + time-rerank | 0.920 | 0.920 | 0.973 | 0.300 | 0.787 | 0.787 | 0.927 | 0.220 |
| Symmetric key | 1.000 | 1.000 | 0.007 | 1.000 | 0.920 | 0.920 | 0.027 | 0.920 |
| Overwrite: oracle | 0.987 | 0.987 | 0.007 | 1.000 | 0.887 | 0.887 | 0.000 | 0.953 |
Table 22.
rent@1 is 0.340 by id and 0.413 by text, and 0.553 of queries miss outright, because a memory written from a short form (“Ebrard now occupies the office of . . .”) is not what the full-name query retrieves; the reader answers 0.600 as returned and 0.700 with the read-path fix, which is where a resolver on the query pays. With glm-4.6 in the loop the same library does something else again. It consolidates: 209 slots anchored, and the memory a query lands on is no longer a statement but a biography — “Career: Honda (2000–2003), Yamaha (2004–2010), Ducati (2011–2012), Yamaha (2013–2020), Petronas Yamaha (2021)” — 332 characters long against 59 with the smaller model, naming 1.72 of the fact’s values on average against 0.89. By id it is current for 0.800 of queries; by text it is current for 0.147 and hedged for 0.720, the category this paper’s text-level scorer was built for and that neither Mem0 nor Graphiti populated. And 0.253 of those top memories name a year after 2020 — the model has written what it knows into the store, including, for Sagan, Team TotalEnergies (2022–2023), a team he did ride for after the window and one the stream never mentioned. The larger model turns the store into a chronology that is right about the world, hedged about the benchmark’s question, and partly not derived from the statements it was given. The reader pays for that: it answers 0.613 from these memories against 0.833 from the smaller model’s, and its errors are not stale (0.033) but neither value (0.367) — a chronology that runs past the window invites the answer that is true today rather than the one the source defines. The read-path fix, which lifts the smaller model’s store to 0.920, has nothing to merge here (0.627).
A fourth mechanism: editing the store rather
than the arrival. Mem0 and LangMem adjudicate the incoming memory; Graphiti invalidates the edge it contradicts. ERASE [15] edits the existing entries: the facts an arrival is related to are retrieved and a model keeps, rewrites or deletes each of them before the arrival is added. No release pins it, so we reimplemented its loop from the paper (Table 23) under the same gate: on the six-statement probe every write is stored, the edits fire, and the current version ranks first. Over the 329 anchored statements the model makes 88 rewrites and 147 deletions against 1395 keeps (0 decisions unparseable and treated as keep), at 1.0 model calls per statement and 0.5 hours, ending at 182 slots. By id it reaches current@1 0.853 at contamination@5 0.020; by text, 0.860 at 0.100, hedged 0.027. Its failure is the one editing the store invites: 0.147 of queries miss, and for 0.133 of facts no version survives anywhere in the returned depth — the model, asked whether a neighbour-ing fact is still accurate, deleted facts about other entities that the arrival did not touch; the reader answers 0.867 from its own top-5 (stale 0.013) and 0.873 with the read-path fix. Anchor-free: 0.733 by id, 0.707 by text, 0.173 missed, 194 slots, reader 0.807. With glm-4.6 deciding, the same loop makes 56 rewrites and 135 deletions, loses every version of only 0.013 of facts, and ends at 194 slots with current@1 0.953 by id and 0.953 by text at contamination@5 0.060; the reader answers 0.980 from its own top-5. That is the best number any of the four mechanisms posts in this paper — within 3.3 points of the write-path oracle on this stream (5 of 150 queries), with no alias table and no identifiers — and it comes from the one mechanism that edits the store rather than the arrival. The price is the deletions smaller model made wrongly: the mechanism that can remove staleness is the one that can remove the fact. Anchor-free the same model posts 0.713 with stale 0.220 at 250 slots (79 deletions against 135 anchored), reader 0.813: an arrival written with a short form does not retrieve the full-name version it supersedes among its five neighbours, so the edit never reaches the stale entry. The mechanism edits only what its retriever finds, which is the anchor-free dependence of § 5.2 again, now on the write path.
Table 23: The fourth mechanism: ERASE, rewrite-on-arrival ([15]), reimplemented from the paper because no release pins it. When a statement arrives, the 5 stored facts nearest to it are retrieved and one model call decides, for each, KEEP, REWRITE or DELETE; the statement is then inserted. Same 150-fact streams, MiniLM encoder, reader (qwen2.5:14b, own top-5). contamination@5 is by text, since a rewrite can keep the superseded value’s string in a past-tense clause.
| policy | anchored | anchor-free | ||||||
| cur@1 | by text | cont.@5 (text) | reader | cur@1 | by text | cont.@5 (text) | reader | |
| ERASE, qwen2.5:14b | 0.853 | 0.860 | 0.100 | 0.867 | 0.733 | 0.707 | 0.160 | 0.807 |
| ERASE, glm-4.6 | 0.953 | 0.953 | 0.060 | 0.980 | 0.713 | 0.713 | 0.240 | 0.813 |
| Cosine (store-all) | 0.460 | 0.460 | 0.973 | 0.573 | 0.140 | 0.140 | 0.927 | 0.427 |
| + time-rerank | 0.920 | 0.920 | 0.973 | 0.300 | 0.787 | 0.787 | 0.927 | 0.220 |
| Symmetric key | 1.000 | 1.000 | 0.007 | 1.000 | 0.920 | 0.920 | 0.027 | 0.920 |
| Overwrite: oracle | 0.987 | 0.987 | 0.007 | 1.000 | 0.887 | 0.887 | 0.000 | 0.953 |
Table 23.
Across the four the pattern is the same and the failure is never the same: Mem0 lost the adjudication between releases, Graphiti keeps its verdicts and discards them on read, LangMem applies them and either keeps everything or rewrites it into a chronology, depending on the model, and ERASE applies them to the store and, under qwen2.5:14b, to facts the arrival never mentioned. No single scoring convention describes all four.
We ran four systems rather than more, and the choice was depth over breadth: each is scored three ways, across releases or read paths, under a hand probe, for 4.2 hours of local inference (Mem0, one quoted run per release, two configurations each; five further runs separated the system from its backend), 7.5 hours (Graphiti), 3.1 hours per configuration (LangMem) and 0.5 hours per configuration (ERASE). The release ships the harness and the probe protocol so that a leaderboard, rather than this paper, is where systems accumulate.
We draw one methodological conclusion for this benchmark rather than for Mem0. A leaderboard for agent memory must pin and publish the version of every system it scores, because the mechanism under test can leave the default code path between minor releases without any change to the documented interface. Our release records it per run.
Reader track: constructed contexts, census details, and presentation order
Figure 3 plots the policy track (left) and the constructed contexts of Table 24 (right). The two readers fail differently, and the decomposition matters. For qwen2.5:14b the gap is almost entirely the reader asserting a superseded value: stale answers rise by 64.5 points and hedging by only 1.9. For glm-4.6 it splits 19.2 stale against 11.1 hedged — a meaningful part of its penalty is naming both values rather than the wrong one. Hedging is the better failure, and a benchmark that scored only “contains the current value” would miss the distinction entirely; we report it because it makes the primary reader’s result stronger and the secondary reader’s more qualified.
| constructed context | no dates: qwen2.5:14b | no dates: glm-4.6 | dates: qwen2.5:14b | dates: glm-4.6 | |
| no context (closed book) | 468 | 0.088 | 0.244 | 0.088 | 0.244 |
| current only | 468 | 0.991 | 0.987 | 0.953 | 0.985 |
| +1 stale, current first | 468 | 0.314 | 0.592 | 0.925 | 0.938 |
| +1 stale, current last | 468 | 0.850 | 0.637 | 0.970 | 0.972 |
| +2 stale, current middle | 42 | 0.262 | 0.405 | 0.952 | 0.976 |
| stale only | 468 | 0.000 | 0.000 | 0.002 | 0.000 |
Table 24. Constructed contexts: reader accuracy when contamination is manipulated rather than observed (test, facts per condition), with and without the stored year surfaced in the context.
Figure 3. Left: retrieval current@1 against reader accuracy for the primary reader; the dashed line is what a retrieval metric would predict. The arrow is the controlled pair — two keyers that return the same statement at rank 1 on every query of this configuration, so they share an exactly and the vertical distance is what contamination of ranks 2–5 costs. Right: constructed contexts, where contamination is manipulated rather than observed.
The obvious objection is that this is the position effect again: rank 1 is fixed, so the stale versions all sit below the current one, and the constructed-context probe below shows that arrangement is exactly the one qwen2.5:14b handles worst. The second reader answers it. glm-4.6’s positional penalty is 4.5 points with a [0.0, 9.2] interval and — we fail to detect a positional effect for it at all, which is not the same as establishing there is none, against 53.6 points at for qwen2.5:14b — and it still pays 29.5 points on the pair at . On a reader for which we cannot measure a position effect, contamination alone still costs almost thirty points.
The pair’s direction is a property of the design — rank 1 is held fixed, so only contamination can move the reader — but its magnitude is not context-size-free. We hold the context at top-5 throughout, matching the of contamination@5; a larger context admits more stale versions and a smaller one fewer, so the gap should be read as the cost of contamination at one context size we fixed in advance, not as a constant.
The mechanism is positional, and the probe is causal. In the policy track contamination is observed rather than assigned, so that comparison cannot separate “the context contains a stale version” from “the context is ordered differently”. Constructed contexts (Table 24) can, and they split the harm into two identified parts. From the current statement alone (0.991 correct), prepending one superseded version costs 14.1 points (95% CI [10.9, 17.3], , sign-flip permutation ) — the cheapest arrangement contamination admits, since the current statement still sits last. Putting the same two statements in the other order costs 53.6 points more (0.314 with the current version first against 0.850 with it last; 95% CI [48.7, 58.3], ), so the full cost of one stale neighbour ranges from 14.1 to 67.7 points depending only on where it sits. These are not two independent additive effects — ordering is a within-contamination effect and cannot exist without it — but conditional on contamination being present, arrangement accounts for roughly four times the best-case harm. The reader largely answers with whatever the context says last, which is exactly the configuration timestamp reranking creates. Reranking does not change which statements are retrieved, only their order: its top-5 is plain cosine’s top-5 for 100% of queries, and on 51% of them the order actually differs — promoting the current version to the front and leaving stale versions adjacent to the question. A context holding only a stale version yields 0.000. We do not report a dose effect beyond one stale version: the two-stale condition exists only for the 42 facts with two distinct superseded values, and comparing its 0.262 against a 468-fact condition would contrast populations rather than doses.
The census. contamination@5 was introduced in § 3 on the strength of prior work about readers, and the reader track supports the direction of that argument: the policies that drive it to zero are the ones whose gains transfer when no date is shown. Over every policy pair in each of the 2 reader policy tracks, current@1 orders 27 of 42 pairs the way reader accuracy does and inverts 13; contamination@5 agrees on 32, ties on 8, and inverts 2. That census includes the retain-everything control, which we built to be pathological, and pairs the reader itself separates by less than a point. Dropping both leaves 24 comparisons between policies a practitioner might deploy: current@1 still inverts 3 of them and contamination@5 inverts 0, at any gap threshold. An exact paired test over the restricted census gives — a description of where the metrics disagree rather than a test.
contamination@5’s census ties are real — cosine and time-rerank share a contamination@5 of 0.974 by construction (§ 4) while their reader accuracies are 0.558 and 0.329 — so 3 of its 8 ties are comparisons current@1 gets wrong; the rest are comparisons current@1 also gets right, and we do not claim it declines only the hard ones. A metric that refuses to rank is weaker than one that ranks correctly, and stronger than one that ranks backwards.
Three caveats. The 24 comparisons are 12 distinct policy pairs scored against 2 readers, not 24 independent observations. The current@1 inversions concentrate on two pairs, both involving timestamp reranking — which is the section’s thesis restated, not independent corroboration of it. What these results actually argue for is a position-aware successor — the rank of the last stale version in the top- — which the released evaluator makes a one-function addition and which we propose rather than claim. The caveat belongs here rather than in the limitations: a retention context differs from a store-all context in two ways at once — it holds one version of the queried fact instead of several, and therefore has room for more statements about other facts. That is what these policies genuinely produce, so the policy-track gap is a property of context composition as a whole rather than of contamination alone; the constructed-context probe is the isolated measurement, and it is the one the causal claim rests on. That probe runs on 2 readers (qwen2.5:14b, glm-4.6), so the mechanism is established causally on both rather than on the one that exhibits the sign flip. Both agree on what matters and disagree on what does not. A clean single-version context is answered almost perfectly by either (0.991 and 0.987), and adding a single stale version costs, in the worse order, 39.5–67.7 points depending on the reader — the range is the disagreement, the direction is not. Where they part company is position: for qwen2.5:14b the penalty depends violently on where the current version sits (0.314 first vs 0.850 last), while for glm-4.6 the two orders are close (0.592 vs 0.637). Contamination is what generalizes; the positional sensitivity to it is a property of the reader, which is the asymmetry § 5.3’s order ablation exploits. Surfacing the stored year removes most of the penalty on both: the worst ordering recovers from 0.314 to 0.925 for qwen2.5:14b and from 0.592 to 0.938 for glm-4.6. Two weaker forms of the same effect are established on both readers as well. Retrieval gains fail to transfer under either; and within the threshold keyer — the one policy that produces a substantial mix of clean and contaminated contexts — splitting its own queries by whether its top-5 was contaminated gives 0.625 against 0.913 for qwen2.5:14b ( vs ) and 0.672 against 0.903 for glm-4.6: a +28.8 and +23.2-point gap, same policy, same reader, same prompt. That is an association rather than a manipulation, but it replicates.
Reranking’s sign is set by a convention it does not control. The contexts above are presented in descending relevance — the common convention, which places the top-ranked statement farthest from the question and is therefore, by the position result established above, the arrangement least favourable to any policy whose mechanism is promoting the right statement to rank 1. Reversing that presentation over the identical retrieved top- is decisive, and the effect falls on exactly the policy that should show it:
| policy | desc. | asc. | shift |
| Cosine (store-all) | 0.558 | 0.564 | +0.6 () |
| + time-rerank | 0.329 | 0.703 | +37.4 () |
| Overwrite: gazetteer | 0.979 | 0.979 | +0.0 () |
| Overwrite: oracle | 0.985 | 0.985 | +0.0 () |
Table 30.
The aggregate shifts understate what is happening, and the per-answer counts are the informative statistic. Reordering changes 213 of 468 individual answers for store-all cosine — it is just as order-fragile as anything else — but the changes cancel, leaving +0.6 points (). For time-reranking 219 answers change and they do not cancel: +37.4 points. The retention policies change 0 answers. Not a null result at the aggregate — zero answers, on every query, because keeping one statement per fact leaves nothing about the queried fact to reorder. Store-all retrieval is order-fragile at the item level and retention is order-invariant at the item level, which is a property of the store, not of the reader.
This qualifies the earlier paragraph in one direction and sharpens it in another. The inversion is not a general property of timestamp reranking: relative to the store-all baseline it is worth −22.9 points under the common convention and +13.9 under the reversed one, so a practitioner can change its sign by changing how a prompt is assembled — itself worth knowing, and not something current@1 can express. What survives every convention is the gap it does not close: against a +46.2-point retrieval improvement its best case at the reader is +13.9, while retention is exactly invariant, its margin over the baseline moving from +42.1 to +41.5 only because the baseline itself moves. For this reader, the retrieval metric overstates reranking under both presentations tested and understates retention under neither — the order ablation ran on qwen2.5:14b only. It is the one experiment here we did not run on the hosted reader, and the reason to want it there is specific: glm-4.6 shows a much weaker positional penalty in the constructed-context probe (0.592 vs 0.637, against 0.314 vs 0.850), so the sign flip this ablation measures is plausibly smaller or absent for it. We report the invariance of retention as the general claim and the sign flip as a demonstration on one reader that it can occur, not that it always does.
Does it hold at scale, and with set-valued gold?
Every number so far comes from the 892 TempLAMA keys that are single-valued in every year and change at least once, which is 892 of 5,839 (the v1.0 release). That restriction buys crisp top-1 scoring and invites the most common objection a benchmark receives: the slice is small and possibly unrepresentative. Keeping every changing key and scoring a top-1 as current whenever its value is valid in the final year — so a fact with two simultaneously valid values simply has two current runs — raises the queried set to 2,904 test facts, of which 2,436 are genuinely multi-valued (the v1.1 scale track).
Table 25 repeats the ladder there, on 15,809 statements, and adds the metric the single-valued track could not define. A top-1 counts as current when its value is valid in the final year, so a fact with several simultaneously valid values is satisfied by any of them — the reading that favours retention, since a policy keeping one slot per fact can discard a still-valid value unseen. Set recall@5 charges that: the fraction of a fact’s distinct currently valid values whose statements appear in the top-5. It equals current@5 on single-valued facts and prices one slot per fact on the rest. Anchored, the oracle’s set recall is 0.890 against 0.940 for recency-weighted store-all, and on the 2,436 multi-valued facts 0.871 against 0.937; every retention rung sits below every store-all policy on this column while beating all of them on current@1 and contamination. One slot per fact is the right footprint for freshness and the wrong one for coverage, and the benchmark prices the two separately.
| policy | anchored current@1 | anchored cont.@5 | anchored set rec.@5 | anchor-free current@1 | anchor-free cont.@5 | anchor-free set rec.@5 |
| Cosine (store-all) | 0.470 | 0.837 | 0.896 | 0.232 | 0.681 | 0.742 |
| + time-rerank | 0.721 | 0.837 | 0.896 | 0.524 | 0.681 | 0.742 |
| + recency-wt | 0.699 | 0.808 | 0.940 | 0.385 | 0.579 | 0.706 |
| Overwrite: threshold | 0.777 | 0.167 | 0.791 | 0.493 | 0.266 | 0.666 |
| Overwrite: gazetteer | 0.916 | 0.009 | 0.851 | 0.293 | 0.543 | 0.709 |
| Overwrite: fuzzy linker | 0.900 | 0.013 | 0.838 | 0.512 | 0.051 | 0.556 |
| Overwrite: linker + alias table | 0.914 | 0.009 | 0.849 | 0.576 | 0.052 | 0.640 |
| Fuzzy linker, read+write | 0.932 | 0.013 | 0.846 | 0.627 | 0.054 | 0.603 |
| Overwrite: oracle | 0.955 | 0.000 | 0.890 | 0.690 | 0.000 | 0.785 |
Table 25. v1.1 scale track: the same ladder on every changing TempLAMA fact rather than only the single-valued ones. 2,904 test facts (2,436 of them multi-valued, mean 1.23 currently valid values each) against 468 in v1.0, scored identically — a top-1 is current when its value is valid in the final year — plus set recall@5: the fraction of a fact’s distinct currently valid values whose statements appear in the top-5.
The write-path keying ladder survives intact and one store-all comparison does not — a pre-specified comparison from Table 2, so it is stated plainly. Store-all similarity still fails (0.470 anchored, 0.232 anchor-free); timestamp reranking still repairs top-1 without touching contamination; and the keying ladder keeps its shape — anchored, the gazetteer and fuzzy rungs sit within a few points of the oracle, while anchor-free the gazetteer collapses to 0.293 and fuzzy resolution holds at 0.512. Multi-valued gold makes the task harder for every policy in the same direction, which is what it should do, and does not reorder the keying ladder. The v1.0 restriction bought scoring clarity, not a favourable result.
One ordering does reverse, and it is the cheap fix losing. Anchored, v1.0 has timestamp reranking ahead of the threshold keyer (0.925 against 0.859); at this scale the threshold keyer leads, 0.777 [0.762, 0.792] against 0.721 [0.705, 0.737] — a sign reversal with non-overlapping intervals. The mechanism is the one §D already identifies: reranking only repairs a query when its window spans every version of the fact, and multi-valued facts have more versions to span. So the statement is not that everything survives — it is that what survives is the keying result, and the metadata heuristic’s advantage over the cheapest keyer is the thing that does not.
The symmetric rung of § 5.2 holds at this scale, where the comparison is better powered: it reaches 0.627 anchor-free for a deficit of 6.3 points, against 11.4 for the rung handed a complete alias table on the write path alone, and 2.3 anchored — on mechanical short forms; Appendix § 17 shows where natural ones remove that advantage.
Data statement
Following Bender and Friedman [1], and covering the artifact questions of the Responsible NLP checklist.
Curation rationale. The queried set is not a sample of language use; it is a deliberately filtered set of facts that changed. From TempLAMA’s 5,839 fact keys we keep those single-valued in every year (so top-1 scoring is crisp) whose value changes at least once in 2010–2020 (892 facts). Facts that never change are excluded from the queried set precisely because they cannot exhibit the failure under study; they re-enter as distractors in the full_corpus configuration (Appendix § 10).
This filtering makes the resource an instrument for one phenomenon rather than a sample of world knowledge, and prevalence estimates should not be read off it.
Language variety. English only (en-US/en-GB undifferentiated), in two registers, both authored by us rather than found: declarative statement templates and interrogative query templates (Appendix V). Entity names appear as Wikidata’s English labels, which retain diacritics and non-English orthography for non-anglophone entities (e.g. Juan Román Riquelme). No text is drawn from naturally occurring corpora, so no dialect or idiolect is represented.
Speaker and annotator demographic. There are no speakers and no crowdworkers. Every statement is generated by template from a subject–relation–object triple, and every currency label is computed from Wikidata validity qualifiers by the released code (label_source: wikidata_validity_year in the key files). No human judgment enters the labels, which is a design guarantee of the benchmark rather than a resourcing decision: a label derived from human annotation could not be claimed to be independent of the systems under test.
Subject demographic. The entities named are public figures and organizations whose employment, office, party, team, or coaching relations Wikidata records with temporal qualifiers. The distribution is inherited from Wikidata’s coverage and is therefore skewed toward anglophone, male, and Global-North sports and politics; we did not rebalance it, and per-relation or per-region claims should not be made from this sample (see the Limitations section).
Speech situation. Synthetic and asynchronous: statements are presented to a memory system as a time-ordered stream with year metadata, and queries are posed after the stream ends. No interaction, no dialogue, no intended audience.
Text characteristics. Nine relations, one canonical statement template plus three paraphrase templates each, three query templates each (Appendix V). Statements are single sentences, typically 6–14 tokens. The anchor-free configurations replace the subject’s surface form in non-introducing statements with a mechanically derived short form, or on the natural configurations with a Wikidata alias or a Wikipedia redirect title (Appendix I); the redirect-derived forms are Wikipedia page titles, released with the inventory under CC BY-SA 4.0 rather than CC0.
Provenance appendix. Facts: Wikidata (CC0) via TempLAMA [5], which its authors distribute under the Apache License 2.0. Templates, the mechanical alias inventory, evaluator, and baselines: ours, released CC0 and Apache-2.0 respectively.
Controlled regime map
The generator (Table 26) varies three preconditions the real data cannot vary independently. Every policy is re-tuned on 10 held-out generator seeds at every setting, as on the real track; fixing , and across the whole controlled track would understate three of the four conclusions below. Update visibility: threshold keying stays perfect across the value_gap range (1.00 even at gap 0.8) because is retuned, from 0.45 at gap 0.3 down to 0.3 at gap 0.8. Held at a fixed , the same regime gives 0.60: the knob does not break threshold keying, it moves its operating point. That is the weaker claim, and the useful one — the required tracks a property of the data the keyer is trying to infer, and is not knowable in advance. Surface variation: key_jitter degrades similarity keying (1.00 → 0.80 at jitter 0.2, 0.54 at 0.4) faster than it degrades the oracle (0.99, 0.70), leaving keying deficits of 0.20, 0.16 and 0.05 at jitter 0.2/0.4/0.8. Threshold tuning matters here, and a grid that stops too high hides it: the optimal moves down as the key signal degrades (0.3 clean, 0.2 at jitter 0.4), and at jitter 0.4 that difference is worth 0.24 → 0.54 (Table 28). What survives is the weaker and more useful statement: a threshold keyer can be re-tuned to track a degrading key signal, but not back to the oracle, and the required is not knowable without the labels the keyer is trying to infer. At jitter 0.8 the knob degrades query–item matching itself (the oracle falls to 0.30), so readings there say little about keying. Budget (Table 27): time-rerank inherits FIFO eviction’s misses (0.47 at ) because it needs the stale copies it stores, while re-tention is budget-immune once one slot per fact fits. Note finally that in the clean regime the strongest store-all fix is recency-weighting (0.98 against 0.83 for reranking) while on its real data reranking wins: which cheap fix is best is itself regime-dependent. One generator property is worth stating because it shapes these numbers: with key_jitter = 0 every mention of the same (entity, version) has an identical embedding, so a reranker’s top- fills with duplicates and its tuned is correspondingly large (20).
| policy | value_gap 0.3 | value_gap 0.5 | value_gap 0.8 | key_jitter 0.0 | key_jitter 0.2 | key_jitter 0.4 | key_jitter 0.8 |
| Cosine (store-all) | 0.25 0.05 | 0.22 0.06 | 0.24 0.05 | 0.23 0.06 | 0.27 0.06 | 0.24 0.06 | 0.15 0.05 |
| + time-rerank | 0.84 0.03 | 0.82 0.04 | 0.80 0.04 | 0.83 0.03 | 0.92 0.04 | 0.62 0.06 | 0.30 0.07 |
| + recency-weight | 0.99 0.01 | 0.97 0.02 | 0.95 0.02 | 0.98 0.01 | 0.90 0.04 | 0.60 0.05 | 0.31 0.06 |
| Reconsolidation | 1.00 0.00 | 1.00 0.00 | 1.00 0.00 | 1.00 0.00 | 0.80 0.05 | 0.54 0.05 | 0.25 0.06 |
| Oracle | 1.00 0.00 | 1.00 0.00 | 1.00 0.00 | 1.00 0.00 | 0.99 0.01 | 0.70 0.05 | 0.30 0.05 |
Table 26. Controlled diagnostic, current@1 (mean 95% CI half-width over 20 seeds). Left: update visibility (value_gap). Right: surface variation (key_jitter) at gap 0.4. Every policy is re-tuned per column on 10 held-out generator seeds; the selected values vary across columns; their ranges are given in Appendix U.
| policy | |||
| Recency (FIFO) | 0.60 | 0.75 | 0.79 |
| Cosine + time-rerank | 0.47 | 0.65 | 0.82 |
| Reconsolidation | 1.00 | 1.00 | 1.00 |
| Keep-latest oracle | 1.00 | 1.00 | 1.00 |
Table 27. Slot budget (gap 0.4, entities): current@1. Store-all fixes need the store; retention keeps one slot per fact and is budget-immune once .
| clean | jitter 0.4 | |
| 0.1 | 0.73 | 0.50 |
| 0.2 | 0.95 | 0.54 |
| 0.3 | 1.00 | 0.42 |
| 0.45 | 1.00 | 0.25 |
| 0.6 | 1.00 | 0.24 |
| 0.75 | 1.00 | 0.24 |
| 0.9 | 0.24 | 0.24 |
Table 28. sensitivity (current@1). Clean: perfect across , false-merging below and false-splitting at . Under jitter 0.4 the optimum moves to and buys 0.24 0.54 — real, and still short of the oracle.
Robustness checks
Verbalization, crossed with the anchor. §F sweeps the paraphrase probability with subjects verbatim, and finds it stresses only the threshold rung. That leaves the paper’s central measurement untested along the same axis: § 5.2 fixes and varies the anchor, so “the anchor-free ladder is an artifact of how we phrase things” is a live objection that neither section alone answers. Here we cross the two, sweeping over 3 levels (, 0.5, 1; is every statement on one template, every statement paraphrased) at both ends of the anchor axis, re-tuning on val at each of the six cells. Anchor-free, the write-path ladder’s ordering is identical at every level, and the full ladder’s is identical except that the symmetric rung overtakes the write-path oracle once any paraphrasing is present — the same regime-dependent trade § 5.2 reports, appearing again as a function of verbalization rather than of scale. Magnitudes move as they should: store-all cosine spans 4.3 points across the range and the entity linker 5.1, since both depend on surface overlap. The exception is mechanistically informative — the symmetric rung moves 0.4
points across the entire range, because resolving the query text to a key and ranking that key by recency never asks how the sentence was phrased. Read-path keying is verbalization-invariant close to by construction, and the sweep shows it.
Encoder. With all-mpnet-base-v2 in place of MiniLM at (re-tuned on val), the ordering is unchanged: cosine 0.468, time-rerank 0.893 (contamination 0.949), recency-weighted 0.767, threshold 0.780 (contamination 0.160), gazetteer 0.940, oracle 0.944. One margin is fragile and worth naming: the threshold keyer’s lead over recency-weighting shrinks from 7.5 points (MiniLM) to 1.3 (mpnet). Critically, § 5.2’s measurement is not an artifact of the primary encoder either: under mpnet the gazetteer keyer’s deficit to the oracle still grows from 0.4 points when anchored to 52.8 when anchor-free, and the entity linker’s from 0.4 to 4.3 — the same collapse, the same surviving residue. Template realization. Across three template-assignment seeds per (hyperparameters fixed from the seed-0 tuning), current@1 varies by at most 0.02 (threshold keyer 0.823–0.859 at , time-rerank 0.925–0.940); no comparison in Table 2 is realization-dependent. Query wording. Sampling the query from all three question templates instead of the canonical one moves current@1 by < 0.01 for every policy (time-rerank 0.921, threshold 0.857); the failure is a property of value-agnosticity, not of one question phrasing.
Analysis
Why value-agnostic queries defeat similarity. The query embeds the entity; each version embeds entity + value. The query–version similarity is dominated by the shared entity component, so version order is decided by noise — and there are more stale versions than current ones whenever a fact has changed more than once. current@1 for store-all cosine on real data (0.464) is close to the reciprocal of the mean version count, as this account predicts, and the stratification (Table 8: 0.482 vs 0.289) matches it stratum by stratum.
| policy | 2 ver. () | ver. () |
| BM25 | 0.423 | 0.178 |
| BM25+rerank | 0.986 | 0.467 |
| Cosine | 0.482 | 0.289 |
| + time-rerank | 0.967 | 0.533 |
| + recency-wt | 0.794 | 0.689 |
| Reconsolid. | 0.868 | 0.778 |
| Lexical key | 0.974 | 0.956 |
| Oracle | 0.981 | 0.956 |
Table 8. current@1 by per-fact version count (test, ). The cheap fix collapses on the facts that changed most — exactly the motivating population.
What overwrite destroys. Auditing the threshold keyer’s overwrites at , : 482 of 995 events overwrite a slot; 73 of them (7.3% of the stream) overwrite a different fact — irreversible false merges that leave 26 facts with no surviving memory — while 71 residual duplicate slots are false splits that keep stale versions retrievable. This is why its current@5 (0.929) is below store-all’s (0.985): destroyed evidence cannot be re-ranked back. Table 13 prices the remedy: invalidation instead of deletion costs footprint and nothing else.
What the cheap fixes actually buy. Timestamp reranking is two lines of code for a +0.46 jump in current@1, and StaleBench prices its boundaries: the context still carries the contradiction, the repair concentrates on 2-version facts, the store keeps every version, eviction undercuts it, it is hostage to timestamp quality — and, most consequentially, for one of our two readers it leaves the end task worse than no fix at all (§ 5.3). A benchmark that reported only current@1 would have counted it a clear gain.
Reproducibility
All numbers are generated from result files produced by python -m stalebench.run (controlled; 20 seeds), run_real (TempLAMA), run_alias (anchor-free, corpus, mechanism), run_reader/run_llmkey (LLM tracks), run_scale (set-valued gold), run_paraphrase (verbalization sweep), run_systems (deployed systems), and diag_pair (the controlled pair’s rank-1 agreement), and injected into the manuscript by a script; no result is hand-transcribed. Reader calls that failed outright (provider content filter, unrecoverable error) are counted rather than dropped silently: 0 across the reported runs. The generator refuses to emit a macro twice with conflicting values, and asserts the premise the controlled pair depends on, so a stale or contradicted number fails the build rather than reaching the page. TempLAMA splits are the public GCS release of Dhingra et al. [5]; embeddings use publicly released sentence-transformers checkpoints (Reimers and Gurevych, [25]). Controlled-track intervals are bootstrapped over generator seeds rather than queries, and each seed contributes only 12 queries, so their per-seed granularity is coarse; real-data intervals resample the 468 test queries directly. Bootstrap CIs use resamples; paired tests are two-sided sign-flip permutation tests with permutations, Holm-corrected within a family pre-specified in code before test evaluation: four comparisons for Table 2, and 9 for the anchor-free track, which grew as reruns were added. A larger family is a harsher correction, so this is conservative, but the same raw is corrected differently in the two tracks and that is worth stating. Every LLM call is greedy-decoded and cached on disk keyed by (model, prompt, max_tokens); the cache ships with the release, so the reader and LLM-keyer numbers regenerate offline with no API access. It also contains judgments from runs that were started and abandoned — a third reader whose provider quota was exhausted mid-run, and a partial second LLM-keyer model — which are shipped as found rather than deleted, and are not reported anywhere in this paper. Answers are capped at 32 tokens, which suits the short-answer scoring but would penalise a verbose reader; that cap is a scope condition on the reader metric, not a property of the benchmark.
Compute. Everything reported here runs on one laptop CPU apart from the LLM tracks: the controlled sweeps, the TempLAMA tracks, and the alias and corpus tracks together take under an hour, dominated by sentence-transformer encoding, and are cached thereafter. The LLM tracks issue 30,818 model calls, 20,055 served by a locally hosted open-weights model and 10,763 by hosted models; the answers are a few tokens each and every call is cached. No GPU is required to reproduce any number in this paper.
The systems track (§K) is the exception to the cost profile above. Streaming 329 statements through a wrapped memory system costs 1.3 hours of local LLM inference for Mem0’s slowest configuration and 4.2 hours across its two quoted runs, and 7.5 hours for Graphiti, whose extraction makes 3.9 model calls per statement, because each system issues its own extraction and adjudication calls. Reproducing it additionally requires a pinned mem0ai version, one of the vector backends in Table 18, a pinned graphiti-core with a FalkorDB server, and a local Ollama serving the adjudicator model — which is why that track runs on a 150-fact subsample. On the anchored configuration the subsample reproduces store-all cosine, time-rerank and the oracle of the full test split to within 0.8 points; the keying rungs score higher on it than on the full split (Appendix § 19).
One reproducibility hazard is worth recording, since it is easy to introduce: ordering the gazetteer with sorted(set(...), key=len) resolves ties by set iteration order and therefore by the interpreter’s per-process string hash seed, and gazetteer results then drift by one query between runs. Keyers impose a total order.
Release
The release directory contains the 8 configurations (anchored, anchor_free, full_corpus, full_corpus_anchor_free, scale, scale_anchor_free, anchor_free_natural, anchor_free_natural_fallback) as statement and query JSONL for both splits, the alias inventories (the derivable one, and the natural one per split that Appendix § 17 draws from), a datasheet, a manifest with per-file row counts and MD5s, and evaluate.py — a frozen scorer that takes a submission of per-query rankings and reports the metrics above. Consuming only the released files and scoring with that evaluator reproduces the store-all rows of Table 2 exactly (baseline.py implements cosine and timestamp reranking, and trivial_baselines.py the two blind rules; the keying rungs need the subject inventory and the oracle needs the answer key, so neither is a baseline a submission could run). Facts derive from Wikidata (CC0) via TempLAMA, which its authors distribute under the Apache License 2.0; our derived splits, templates and mechanical alias inventory are released CC0, the natural alias inventory under CC BY-SA 4.0 (Appendix § 22), and the evaluator and baselines under Apache 2.0. Redistribution of the derived data is consistent with both upstream licences.
A leaderboard built on this release is meant to host production memory systems (Mem0, Graphiti/Zep, MemGPT-style agents) as policies under this harness. The full release directory, the harness and the result records are archived at https://doi.org/10.5281/zenodo.24237231; the repository is https://github.com/thenexuslab/stalebench-release.
Verbalization crossed with the anchor
Table 29 crosses the paraphrase probability with anchor availability , to check that the ladder’s ordering does not depend on how statements are verbalized.
| policy | anchored () | anchor-free () | ||||
| Cosine (store-all) | 0.447 | 0.464 | 0.468 | 0.085 | 0.111 | 0.128 |
| + time-rerank | 0.959 | 0.925 | 0.942 | 0.801 | 0.737 | 0.744 |
| Overwrite: threshold | 0.887 | 0.859 | 0.829 | 0.549 | 0.513 | 0.526 |
| Overwrite: gazetteer | 0.991 | 0.972 | 0.964 | 0.143 | 0.169 | 0.177 |
| Overwrite: fuzzy linker | 0.987 | 0.968 | 0.959 | 0.754 | 0.714 | 0.690 |
| Overwrite: linker + aliases | 0.991 | 0.972 | 0.964 | 0.825 | 0.776 | 0.774 |
| Symmetric key (read+write) | 0.989 | 0.989 | 0.987 | 0.833 | 0.831 | 0.835 |
| Overwrite: true-key oracle | 0.998 | 0.979 | 0.970 | 0.880 | 0.827 | 0.808 |
Table 29. current@1 as verbalization diversity and anchor availability are varied together (test split, re-tuned on val in every cell). is the probability a statement uses a paraphrased template rather than the relation’s base one. The write-path ladder’s ordering is identical at every level; including the symmetric rung it is identical except that the symmetric rung overtakes the write-path oracle once any paraphrasing is present. Magnitudes move — store-all cosine by 4.3 points across the range, the entity linker by 5.1 — while the symmetric rung moves 0.4, because resolving the query to a key never asks how the sentence was phrased.
Tuned hyperparameters
Every policy is re-tuned in every regime, on the val split (real tracks) or on 10 held-out generator seeds disjoint from the reported ones (controlled track). The selected values vary substantially — time-rerank’s ranges over and the threshold keyer’s over — which is the reason a single fixed setting misrepresents several policies at once. The full table ships in results/ stalebench_results.json under tuned and in each real-track cell under tuned.
Relation templates
Nine TempLAMA relations are verbalized with one canonical statement template (the TempLAMA cloze with the value filled in) plus three paraphrase templates each, and carry three value-agnostic question templates; the headline uses the canonical question, and the query-wording check of Appendix § 24 samples all three. The full sets ship in stalebench/templama.py. Example (P286): canonical “{s} is the head coach of {s}.”; paraphrases “{s} is coached by {v}.”, “{v} runs the bench for {s}.”, “The head-coaching job at {s} belongs to {s}.”; questions “Who is currently the head coach of {s}?”, “Who coaches {s} now?”, “What is the name of {s}’s current head coach?”.
Limitations and ethical considerations in full
The Limitations and Ethical Considerations sections of the main paper are condensed from the text below.
Limitations
Three properties of this release bound what its numbers mean. First, saturation, which § 5.1 states once: anchored (and full_corpus, where the blind rule scores 0.970) is saturated on current@1 and anchor_free is not. The strongest blind rule there — trigram mention linking on the read path over an inventory recovered from the public question templates — reaches 0.630 against an oracle at 0.827, at contamination@5 0.641; it differs from the read-path rung of § 5.2 (0.831 on the same statements) only in requiring no token-boundary-aligned mention span, so careful mention detection is worth most of that gap and the released number is a floor rather than a restatement of our own result. Both rules ship as reference baselines. Second, the entity-linker rung is handed the alias inventory the stream drew from — the generator’s on the mechanical configurations, the harvested one on the natural configurations of Appendix § 17 — so it has no out-of-vocabulary mass at all; its residual 5.1 points are ambiguity within a known alias space, and the gap a real linker pays is bounded below, not estimated, by that number. The best write-path rung without an alias table is the fuzzy linker: it reaches 0.714 for a deficit of 11.3 points, and that — not 5.1 — is this paper’s estimate of the residue. On the realistic corpus it roughly doubles (23.7 points, Table 14, so the benchmark is not saturated where it asks to be ranked. Third, the reader metric has a ceiling below one: three test facts have a superseded value nested inside the gold value (Democratic Party inside Democratic Party of Korea), so answering exactly correctly scores as hedged; the attainable maximum is 0.994, which is where the oracle reader’s 0.985 should be read against.
The headline track restricts the queried set to single-valued facts that change (892 of 5,839 keys) for crisp top-1 scoring; Appendix M lifts that restriction to 2,904 test facts, 2,436 of them multi-valued, and the write-path keying ladder holds. A top-1 there counts as current if its value is valid in the final year, so a fact with several simultaneously valid values is satisfied by any one of them; the set recall@5 of Appendix M charges what that hides — a one-slot policy keeps 0.890 of the valid values against 0.940 for recency-weighted store-all — and we report the two metrics side by side rather than claiming either alone. Validity is yearly and statements are templated: the anchor-free track models aliasing with mechanically derived short forms, which covers the common case in logs but not nicknames, cross-lingual forms, or pronominal coreference resolvable only from discourse. Conversational reference — pronouns, nicknames, discourse coreference — is therefore outside what this instrument measures; Appendix § 17 narrows the gap with real aliases for the 58% of test subjects that have one, without closing it. TempEL ([33]) supplies real temporal mention drift over Wikipedia snapshots and would make that axis naturalistic rather than synthetic; we generate the axis instead because we need it controlled — the point is to read a policy’s cost as a function of anchor availability, which requires setting that availability rather than observing it — and a naturalistic version of the same axis is the obvious next instrument. The alias inventory we hand the entity linker is also cleaner than one a deployed system would build. The systems track is the narrowest evidence in the paper and should be read that way. Four mechanisms, one of them our reimplementation, on 150 facts, and for Mem0 two releases with, for 0.1.118, a single backend — the only (release, backend) pair of the three we tried that passes both halves of the probe. We report a cross-backend stability check for 2.0.11 (0.180–0.207 over 2 stores) and cannot offer one for 0.1.118; we do report an independent repeat of that cell on chroma, which reproduced its event histogram, slot count and current@1 exactly (0.0 points apart), so within one backend the pipeline is reproducible at temperature zero even though its extraction is model-driven. A second adjudicator model, glm-4.6, is measured in Appendix § 19 (Table 21); it moves the numbers at the extraction stage before the adjudication, and the comparison holds only under an extraction prompt that admits third-person facts. Reader-side numbers use a single reader, unlike § 5.3, which uses two.
The attribution finding itself has a bound. That 62% of 2.0.11’s memories carry an id naming a different entity than their text is measured over that run’s own memories, and a six-statement probe of the same release attached metadata correctly — so the effect does not appear at small scale and we offer no mechanism for it, only the measurement. Nothing here is load-bearing for the version finding, which rests on a dispatch path anyone can read, an event histogram and a two-statement probe.
Because facts arrive as they become true and gold is the final value, most-recent-equals-current holds by construction in the clean condition; Appendix § 13 measures the corrupted-clock and corrupted-arrival regimes and a late-stale-report dose, but ambiguous updates — a statement that may or may not supersede an earlier one — and corrections with their own semantics — a retraction that names a previous statement as wrong — are out of scope. The reader track uses 2 models with one prompt format; prompt sensitivity is a known confound in reader studies, and our claim is comparative across contexts under a fixed prompt, not absolute accuracy. English only; nine relations; 468 test facts is enough for tight CIs but not for per-relation claims.
Ethical considerations
Data provenance and personal information. The benchmark’s facts are public biographical and organizational relations — team membership, employer, office held, head coach, chair, party, head of government, owner, school — about public figures and organizations, taken from Wikidata via TempLAMPAA ([5]) and released under CC0. They name real people, and that is intrinsic to the task rather than incidental: the failure we measure is a retriever’s inability to prefer the current value of a fact about a named entity, so pseudonymizing the subjects would remove the phenomenon under study. We add no information beyond what Wikidata already publishes, collect nothing from private sources, and include no contact details, identifiers, images, or free text written by the individuals concerned. One category deserves explicit mention: political party affiliation is among the nine relations, and party membership is treated as a special category of personal data under some regimes. These are the publicly recorded affiliations of office-holders and candidates, already published under CC0 by Wikidata, and we neither infer nor annotate any affiliation ourselves. The aliases on the anchor-free track are mechanical shortenings (head token, leading token, initialed form) of names already present in the source, or — on the two natural configurations — public short forms taken from Wikidata’s alias field and English Wikipedia redirect titles; neither adds identifying information beyond what those sources publish. The release is a frozen 2010–2020 window and is not updated, so it does not track any individual going forward.
Offensive content. Statements are template-verbalized subject–relation–object triples over nine factual relations; no free text is model-generated in the released data and no user-authored text is included, so the release carries little risk of offensive, harmful, or deceptive content. The reader-track model outputs that ship in the call cache are short factual answers to these same questions.
Risks of the work. The clearest risk is that a benchmark which makes freshness measurable is over-read as certifying it. Our central result argues against exactly that reading — a single retrieval number can rank memory policies backwards — and we report where each fix fails for the same reason. A second risk is specific to the remedy we price: retention policies delete superseded statements, and in deployments where the history is itself the record (audit, compliance, provenance, contested claims about a person), destructive overwrite is the wrong mechanism and a false merge is unrecoverable. That is why we evaluate Zep-style invalidation alongside overwrite and report its footprint cost (Appendix § 15): a system that must not lose history has a measured alternative here. We see no dual-use concern; the beneficiary of fresher agent memory is the user who would otherwise receive a confidently outdated answer.
Environmental impact. Every number outside the systems track reproduces on a single laptop CPU in under an hour, and no GPU is required for any result in this paper. The LLM tracks issue 30,818 short completions, 20,055 served by a locally hosted open-weights model and 10,763 by hosted models, cached on disk and shipped with the release so that reproduction requires no further inference (Appendix § 26).
References
- [1]Emily M. Bender and Batya Friedman. 2018. Data statements for natural language processing: Toward mitigating system bias and enabling better science. Transactions of the Association for Computational Linguistics, 6:587–604.DOI
- [2]Hung-Ting Chen, Michael J.Q. Zhang, and Eunsol Choi. 2022. Rich knowledge sources bring complex knowledge conflicts: Recalibrating models to reflect conflicting evidence. In Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 2292–2307.arxiv.org/abs/2210.13701
- [3]Wenhu Chen, Xinyi Wang, and William Yang Wang. 2021. A dataset for answering time-sensitive questions. In Proceedings of the Neural Information Processing Systems Track on Datasets and Benchmarks (NeurIPS).arxiv.org/abs/2108.06314
- [4]Prateek Chhikara, Dev Khant, Saket Aryan, Taranjeet Singh, and Deshraj Yadav. 2025. Mem0: Building production-ready AI agents with scalable long-term memory. In ECAI 2025, Frontiers in Artificial Intelligence and Applications. IOS Press.arxiv.org/abs/2504.19413
- [5]Bhuwan Dhingra, Jeremy R. Cole, Julian Martin Eischenslos, Daniel Gillick, Jacob Eisenstein, and William W. Cohen. 2022. Time-aware language models as temporal knowledge bases. Transactions of the Association for Computational Linguistics, 10:257–273.
- [6]Ivan P. Fellegi and Alan B. Sunter. 1969. A theory for record linkage. Journal of the American Statistical Association, 64(328):1183–1210.DOI
- [7]Luyu Gao, Xueguang Ma, Jimmy Lin, and Jamie Callan. 2023. Precise zero-shot dense retrieval without relevance labels. In Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 1762–1777, Toronto, Canada. Association for Computational Linguistics.arxiv.org/abs/2212.10496
- [8]Joel Jang, Seonghyeon Ye, Changho Lee, Sohee Yang, Joongbo Shin, Janghoon Han, Gyeonghun Kim, and Minjoon Seo. 2022. TemporalWiki: A lifelong benchmark for training and evaluating ever-evolving language models. In Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 6237–6250.
- [9]Bernal Jiménez Gutiérrez, Yiheng Shu, Yu Gu, Michihiro Yasunaga, and Yu Su. 2024. HippoRAG: Neurobiologically inspired long-term memory for large language models. In Advances in Neural Information Processing Systems (NeurIPS).
- [10]Zhuoran Jin, Pengfei Cao, Yubo Chen, Kang Liu, Xiao-Jian Jiang, Jiexin Xu, Li Qiuxia, and Jun Zhao. 2024. Tug-of-war between knowledge: Exploring and resolving knowledge conflicts in retrieval-augmented language models. In Proceedings of the 2024 Joint International Conference on Computational Linguistics, Language Resources and Evaluation (LREC-COLING), pages 16867–16878.arxiv.org/abs/2402.14409
- [11]Mandar Joshi, Omer Levy, Luke Zettlemoyer, and Daniel S. Weld. 2019. BERT for coreference resolution: Baselines and analysis. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), pages 5803–5808.arxiv.org/abs/1908.09091
- [12]Jungo Kasai, Keisuke Sakaguchi, Yoichi Takahashi, Ronan Le Bras, Akari Asai, Xinyan Yu, Dragomir Radev, Noah A. Smith, Yejin Choi, and Kentaro Inui. 2023. RealTime QA: What’s the answer right now? In Advances in Neural Information Processing Systems (NeurIPS), Datasets and Benchmarks Track.arxiv.org/abs/2207.13332
- [13]Dayoon Ko, Jinyoung Kim, Hayeon Choi, and Gunhee Kim. 2024. GrowOVER: How can LLMs adapt to growing real-world knowledge? In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (ACL), pages 3282–3308.
- [14]LangChain. 2025. LangMem: long-term memory for agents. https://github.com/langchain-ai/langmem. Version 0.0.30.
- [15]Belinda Z. Li, Emmy Liu, Alexis Ross, Abbas Zeitoun, Graham Neubig, and Jacob Andreas. 2025. Language modeling with editable external knowledge. In Findings of the Association for Computational Linguistics: NAACL 2025, pages 3070–3090.arxiv.org/abs/2406.11830
- [16]Yuliang Li, Jinfeng Li, Yoshihiko Suhara, AnHai Doan, and Wang-Chiew Tan. 2020. Deep entity matching with pre-trained language models. Proceedings of the VLDB Endowment, 14(1):50–60.DOI
- [17]Adam Liška, Tomáš Kočiský, Elena Gribovskaya, Tayfun Terzi, Eren Sezener, Devang Agrawal, Cyprien de Masson d’Autume, Tim Scholtes, Manzil Zaheer, Susannah Young, Ellen Gilsenan-McMahon, Sophia Austin, Phil Blunsom, and Angeliki Lazaridou. 2022. StreamingQA: A benchmark for adaptation to new knowledge over time in question answering models. In Proceedings of the 39th International Conference on Machine Learning (ICML), pages 13604–13622.arxiv.org/abs/2205.11388
- [18]Adyasha Maharana, Dong-Ho Lee, Sergey Tulyakov, Mohit Bansal, Francesco Barbieri, and Yuwei Fang. 2024. Evaluating very long-term conversational memory of LLM agents. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (ACL), pages 13851–13870.
- [19]Kevin Meng, David Bau, Alex Andonian, and Yonatan Belinkov. 2022. Locating and editing factual associations in GPT. In Advances in Neural Information Processing Systems (NeurIPS).arxiv.org/abs/2202.05262
- [20]Kevin Meng, Arnab Sen Sharma, Alex Andonian, Yonatan Belinkov, and David Bau. 2023. Mass-editing memory in a transformer. In International Conference on Learning Representations (ICLR).arxiv.org/abs/2210.07229
- [21]Seyed Mahed Mousavi, Simone Alghisi, and Giuseppe Riccardi. 2024. DyKnow: Dynamically verifying time-sensitive factual knowledge in LLMs. In Findings of the Association for Computational Linguistics: EMNLP 2024, pages 8014–8029.arxiv.org/abs/2404.08700
- [22]Charles Packer, Sarah Wooders, Kevin Lin, Vivian Fang, Shishir G. Patil, Ion Stoica, and Joseph E. Gonzalez. 2023. MemGPT: Towards LLMs as operating systems. arXiv preprint arXiv:2310.08560.arxiv.org/abs/2310.08560
- [23]Joon Sung Park, Joseph C. O’Brien, Carrie J. Cai, Meredith Ringel Morris, Percy Liang, and Michael S. Bernstein. 2023. Generative agents: Interactive simulacra of human behavior. In Proceedings of the 36th Annual ACM Symposium on User Interface Software and Technology (UIST).arxiv.org/abs/2304.03442
- [24]Preston Rasmussen, Pavlo Paliyчuk, Travis Beauvais, Jack Ryan, and Daniel Chalef. 2025. Zep: A temporal knowledge graph architecture for agent memory. arXiv preprint arXiv:2501.13956.arxiv.org/abs/2501.13956
- [25]Nils Reimers and Iryna Gurevych. 2019. Sentence-BERT: Sentence embeddings using siamese BERT-networks. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), pages 3982–3992.arxiv.org/abs/1908.10084
- [26]Stephen Robertson and Hugo Zaragoza. 2009. The probabilistic relevance framework: BM25 and beyond. Foundations and Trends in Information Retrieval, 3(4):333–389.
- [29]Kevin Wu, Eric Wu, and James Zou. 2024. ClashEval: Quantifying the tug-of-war between an LLM’s internal prior and external evidence. In Advances in Neural Information Processing Systems (NeurIPS), Datasets and Benchmarks Track.arxiv.org/abs/2404.10198
- [30]Ledell Wu, Fabio Petroni, Martin Josifoski, Sebastian Riedel, and Luke Zettlemoyer. 2020. Scalable zero-shot entity linking with dense entity retrieval. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 6397–6407.arxiv.org/abs/1911.03814
- [31]Rongwu Xu, Zehan Qi, Zhijiang Guo, Cunxiang Wang, Hongru Wang, Yue Zhang, and Wei Xu. 2024. Knowledge conflicts for LLMs: A survey. In Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 8541–8565.
- [32]Sikuan Yan, Xiufeng Yang, Zuchao Huang, Ercong Nie, Zifeng Ding, Zonggen Li, Xiaowen Ma, Jinhe Bi, Kristian Kersting, Jeff Z. Pan, Hinrich Schuetze, Volker Tresp, and Yunpu Ma. 2026. Memory-R1: Enhancing large language model agents to manage and utilize memories via reinforcement learning. In Proceedings of the 64th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 12805–12825.
- [33]Klim Zaporojeцs, Lucie-Aimée Kaffee, Johannes Deleu, Thomas Demeester, Chris Develder, and Isabelle Augenstein. 2022. TempEL: Linking dynamically evolving and newly emerging entities. In Advances in Neural Information Processing Systems 35 (NeurIPS), Datasets and Benchmarks Track.
- [34]Michael J.Q. Zhang and Eunsol Choi. 2021. SituatedQA: Incorporating extra-linguistic contexts into QA. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 7371–7387.arxiv.org/abs/2109.06157
- [35]Zexuan Zhong, Zhengxuan Wu, Christopher D. Manning, Christopher Potts, and Danqi Chen. 2023. MQaKE: Assessing knowledge editing in language models via multi-hop questions. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 15686–15702.

