WorksComputer Science
Experience Rot: Experienced Agents Fail Where Novices Succeed
Abstract
Agents are increasingly built to learn from experience: they keep conversation histories, memory files, retrieved notes and skills, and reuse what worked before. We show that this experience turns against them when their tools change silently. We test a range of models with tools defined as in the Model Context Protocol (MCP). After a silent change, experienced agents fail on most tasks, while agents new to the task mostly succeed. We call this failure Experience Rot. Experienced agents copy their earlier call instead of checking the current interface, even when the tool definition shows the change. Then they report success. When the documentation is incomplete, experience is what makes agents succeed. After a change, the same experience leaves them worse off than no experience at all. The failure appears in every form of memory we test and grows with the amount of experience. Simple remedies fall short: announcing the update barely helps, and pointing the agent to the documentation fails whenever the documentation does not spell out the answer.
Introduction
Agents are increasingly built to learn from experience. They keep conversation histories, write notes to memory files, retrieve records of past tasks, and save scripts as reusable skills [8] (Packer et al. 2023; Shinn et al. 2023; Zhao et al. 2024; Wang et al. 2024; Wang et al. 2023; Zhou et al. 2026). The idea is simple. An agent that has called a tool successfully once does not need to look up how to call it again.
This only works if the tool still works the way it did. Tools change. A field is renamed, a set of allowed values is replaced, or the meaning of a number is reversed. Interfaces in use today already disagree on such details: in Todoist’s API reference and in two MCP servers for Todoist, task priority 1 is the highest in some places and 4 in others (Figure 1a). Many changes are also silent. An MCP server can replace its tool definitions during a session, and some services accept a request in the old form and ignore the fields they do not know (Li et al., 2026b). A survey of agent memory names such changes as an open problem [11], and adaptation to them has not been tested (Qian et al., 2026).

Figure 1. Experience rot. (a) Deployed interfaces in which one field has opposite meanings, and in which changes need not produce errors: Todoist’s API reference (Doist, [10]) and two Todoist MCP servers (A: greirson, 2025, B: mehularora8, 2025); Python, JavaScript and POSIX conventions (Python Software Foundation, 2026; MDN Web Docs, 2026; The Open Group, 2018); the MCP specification (Model Context Protocol, 2025); SilentProbe (Li et al., 2026b). (b) Two runs of DeepSeek-V4-Flash on the same final task with the same tool definition. Without experience, the agent uses the field the definition lists. With experience from version A, it repeats its earlier call; the service accepts the call, ignores the unknown field, and the agent reports success. (c) Success on the final task without experience, with valid experience, and with experience from before the change, when the documentation gives the range of a value but not its meaning (incomplete documentation; six models, four changes; § 4). Error bars are one standard error.
We find that experienced agents fail where novices succeed. Figure 1b shows one case. An agent schedules an event and puts the participants in attendees, as the service requires. The service then renames the field to guests. With the next request, the agent receives a tool definition that lists guests. It writes attendees anyway. The service accepts the call and stores an event without participants, and the agent tells the user that the event is scheduled. An agent without the earlier task uses guests and succeeds. The pattern holds across models and changes: after a silent change, experienced agents fail on most tasks, while agents new to the task mostly succeed. Figure 1c shows this when the documentation is incomplete. Valid experience lifts success from 63% to 100%. Out-of-date experience drops it to 23%, far below agents without experience.
We call this failure experience rot: an agent’s own earlier tool use, correct when it was made, becomes wrong after a silent change, and the agent reuses it anyway. The underlying behaviour is known. Agents reproduce the outputs of similar past tasks, which Xiong et al. (2026) call experience-following. We ask what this behaviour costs when tools change, why agents do not notice, and what helps.
We study these questions in a controlled setting. Each service has two versions that differ in one interface detail. An agent first completes a task on one version. The final task then runs on the same version (valid experience), on the other version (out-of-date experience), or without the earlier task (no experience). The current documentation is always one search away. With six models, we vary what the documentation says, where the experience is stored, how much of it there is, what is added to the final task, and who wrote the earlier task. Most conditions were preregistered. We report five findings.
Out-of-date experience is worse than no experience (§ 4). When the documentation is incomplete, experienced agents fall far below novices (Figure 1c). When it is complete, novices nearly always succeed and experienced agents still fail on most tasks. Nearly every failed run ends with a report that the task is done.
Experienced agents stop checking (§ 5). They copy their earlier call instead of reading the documentation, even when the tool definition in front of them shows the change. Lookups before the first write drop to at most 6% of runs, and agents talk about valid and out-of-date experience in the same way.
Every form of memory is affected, and more experience makes it worse (§ 6). The failure appears with conversation history, memory files, retrieved notes and skills, even when the task cannot be done without the memory. It grows with the number of earlier tasks.
Simple remedies fall short (§ 7). Announcing the update barely helps. An instruction to read the documentation helps only when it sits next to the request and the documentation contains the answer; otherwise it does nothing or makes things worse. Checking the result after writing works, at more than twice the model calls.
Hand-written histories hide the problem (§ 8). Studies often give agents a history written by the researchers. One check step in such a
history removes most of the cost, because agents copy the check along with the call.
Our contributions are a controlled design that isolates the effect of an agent’s own experience under silent tool changes, the evidence above, and its consequences for memory systems, tool design and the evaluation of agent memory (§ 9).
Related Work
Following one’s own experience. Xiong et al. (2026) show that agents with a memory of past executions produce outputs similar to those of the retrieved records when the task inputs are similar, and call this experience-following; errors in stored experience then propagate to new tasks. In multi-turn settings, models imitate their own earlier responses (conversational inertia; Wan et al. (2026)), keep to earlier reasoning when later turns bring corrections (contextual inertia; Chen et al. [5]), stay with early assumptions (Laban et al. (2025)), and make more errors after seeing their own errors (Sinha et al. (2025)). Older information in context also interferes with updated values (Wang and Sun (2025)). These works vary the experience or the conversation while the environment stays fixed. We hold the agent’s experience fixed in content and change the environment underneath it, and we measure whether the agent checks the environment before it reuses the experience.
Changing tools. Most tool-use benchmarks keep the tools fixed (Yao et al., 2024; Lu et al., 2024; Trivedi et al., 2024). MCPEvol-Bench applies mutation operators to 123 real MCP servers and finds lower success on the evolved servers; adding reflection, planning or memory modules reduces the drop (Liu et al. (2026)). ProEvolve generates linked versions of an environment for the same tasks; traces or summaries from earlier versions reduce the number of tool calls but do not raise success, and lower it in several settings (Li et al. (2026a)). From an analysis of failures, its authors suggest that agents with summaries of earlier versions hold outdated beliefs about which tools exist instead of consulting the current tool list. We test this kind of explanation with controlled manipulations and measure the lookups directly. ToolEVO trains models to adapt to API changes from explicit errors (Chen et al. [4]). In SKILL.nb, workflow-memory baselines lose 11–14 points when the web interface changes under the same tasks (El Hattami et al. [12]). The set-shifting test asks whether agents keep using a tool group that has stopped being reliable (Ye (2026)), and ScrambleTool
keep using a tool group that has stopped being reliable (Ye (2026)), and ScrambleToolBench finds belief inertia when tool identifiers are permuted (Toh et al. (2026)). MemDrift shows that preferences stored in memory shift tool-call parameters in contexts where they do not apply (Dabas et al. [9]). In code generation, models use deprecated APIs (Wang et al. (2025)), have difficulty writing code for a specified library version (Islah et al. (2024)), and are hard to update with knowledge of changed API functions (Liu et al. (2024)).
Stale memory. STALE asks whether language models and memory systems recognise that a stored memory is no longer valid (Chao et al. [3]). Fan et al. [14] describe state drift, where an agent acts on a stale version of a fact that is present in its context. Nagayashiki (2026) give agents inherited memories with a superseded constraint: under a budget of two verification actions out of six, agents rarely check the source record, while with a budget of four they check it in 88.7% of cases. Chen et al. [6] study how agents consume conflicting memory, and Wang et al. (2026) study reasoning fixation and belief distortion caused by memories (both papers call their benchmark Mem-TrapBench). Hu et al. (2026) find that detailed trajectories in memory can hinder adaptation to new tasks, while abstract procedural memories transfer more reliably. Most of these studies write the memories themselves. We use the agent’s own experience and show that hand-written histories change the measured cost (§ 8).
Information in view and silent failures. Englander et al. [13] find that agents rarely act on useful information they encounter in the environment, but use it when it is placed in the task prompt (pass@ 1 rises from 55.88 to 76.25). We find a related pattern within the context: documentation attached to the task message is used, documentation in the system prompt or the tool description much less. Cheng et al. [7] study when agents call a tool again instead of reusing earlier results. Agents over-trust tool outputs that look valid (Sun et al. (2024)). In production APIs, 44 of 61 constraints that are stated only in prose fail silently, against none of 111 machine-checkable constraints, and four of five endpoints ignore an undeclared parameter (Li et al. (2026b)). Implicit failures lower recovery by 37% relative to explicit errors (Zhu et al. (2026)), and agents often report success on tasks they did not complete (Advani [1]). Our silent changes follow these observations: unknown fields are ignored, and the invisible changes are stated only in prose.
A human parallel. In the Einstellung effect, people who have solved problems with one method keep using it when a simpler method is available (Luchins, 1942), and chess experts who know a familiar solution keep attending to it while they report looking for a better one [2]. Studies of language models have used this framing for reasoning benchmarks (Haq et al., 2025; Kim et al., 2025). We note the resemblance in behaviour only and make no claim about a shared mechanism.
Setup
The design compares agents that differ in one respect: whether their experience comes from the current version of a tool or from an earlier one.
Services and changes
We built three small services: a calendar, a task list, and a mail client. Each service has two versions, A and B, that differ in one interface detail (Table 1). The agent reaches a service through tools with typed definitions, in the form used by function calling and by the Model Context Protocol: each tool has a name, a description, and a JSON schema for its arguments. Each service also offers read tools that return the stored state (for example calendar_get_event) and a docs_search tool that returns the reference documentation of the running version (Appendix § 11).
| Change | Version A | Version B | In definition |
| Field name | attendees | guests | yes |
| Structure | end_time | duration_minutes | yes |
| Value | urgent, . . . , low | p0, . . . , p3 | yes |
| Priority direction | 1 is highest | 4 is highest | no |
| Weekday numbering | 0 = Monday | 0 = Sunday | no |
| Month numbering* | 1 = January | 0 = January | no |
| Duration unit* | minutes | seconds | no |
Table 1. The changes. In the first three, a field or value changes, and a typed tool definition lists the fields or values of the running version. In the others, a number keeps its range but changes its meaning, and the tool definition is the same in both versions (for example an integer from 1 to 4). *Used only with incomplete documentation.
The tool definitions always follow the running version: an earlier task on version A shows the definitions of A, and a final task on version B shows those of B. Changes are silent. A request that is valid only in the other version is accepted with the same response as a correct request, {"ok": true, "result": {"accepted": true, . . . }}. An unknown field is ignored, an unknown value leaves the stored state unchanged, and an integer takes the meaning of the running version. In Figure 1b, the event is stored without participants, and the response does not say so. The read tools show the stored state in plain terms (“Highest”, “Tuesday”, the end time of an event), so an agent can always check what its write did.
What the documentation says. With complete documentation, the documentation of each version gives the fields, the values and what the values mean. The agent can then learn of a change in one of three ways. For a changed field or value, the write tool either takes an untyped payload, so that only the documentation gives the fields, or has a typed definition that lists the fields or values of the running version. For a changed meaning, the definition is the same in both versions, and only the documentation says what the number means. With incomplete documentation, the documentation gives only the range of a value, for example “priority is an integer from 1 to 4”, and not what it means. Real documentation is often like this (Figure 1a).
Episodes
An episode has an earlier task and a final task of the same kind, for example scheduling two different events. The agent does the earlier task live, with the tools and tool definitions of the version it runs on. An attempt counts only if the stored state matches the request; failed attempts are discarded and repeated, up to ten times. The final task then follows as the next message in the same conversation. There are three arms. With valid experience, the earlier task ran on the same version as the final task. With out-of-date experience, it ran on the other version. With no experience, there is no earlier task. Each change is run in both directions, A to B and B to A, and the same task instances are used in all arms. The valid and out-of-date arms therefore differ only in the version on which the agent gained its experience. With incomplete documentation, the earlier request says what the values mean on its version, for example “For this service, priority 4 is the highest and 1 is the lowest.”; the final request does not.
Later sections change one part of this design at a time: where the experience is stored and how much of it there is (§ 6), what is added to the final task (§ 7), and who wrote the earlier task (§ 8). Appendix § 13 lists all conditions.
Measures. A program checks the stored state after the final task. Success means that the state matches the request. The cost of out-of-date experience is the success rate with valid experience minus the success rate with out-of-date experience. We also record whether the agent searched the documentation before its first write (lookup), whether its first write uses the usage of the other version (old usage; for example attendees on version B, or 1 for the highest priority when 4 is the highest), whether it called a read tool after its first write (read-back), the number of model calls, and whether the final reply reports the task as done (Appendix § 16.§ 9).
Models and statistics
We test six models: GLM-5.3-Flash, Muse-Glimmer-30B, DeepSeek-V4-Flash, MiniMax-M3, Qwen3.8-Flash, and Inkling (Appendix § 14). Each model runs the same simple agent loop with temperature 0.7 and at most 16 model calls per task. Of 12,624 episodes, 136 are excluded: 9 because a model request failed and 127 because the earlier task never succeeded, most of them on the weekday change (Appendix § 19).
Estimates are pooled over models, changes and directions. We report each estimate one standard error from a cluster bootstrap over task instances: instances are resampled within each change and direction, jointly for all models and conditions, with 10,000 replicates. We call a difference positive or negative when its 95% percentile interval excludes zero. For most conditions with complete documentation, the design, the predictions, the exclusion rules and the analysis code were preregistered before the run, and all preregistered predictions held (Appendix § 15). The conditions with incomplete documentation were not preregistered.
Experienced Agents Fail Where Novices Succeed
Figure 2 compares the three arms on the same final tasks.

Figure 2. (a) Success on the same final tasks with no, valid and out-of-date experience, ± one standard error. “Field”: a renamed field, a changed structure or a changed set of values, with an untyped write tool or with a typed definition that lists the current fields. “Meaning of a number”: a number keeps its range and changes its meaning. (b) Share of failed runs with out-of-date experience whose final reply tells the user that the task is done. Six models; numbers in Table 4.
With incomplete documentation, experience is what makes agents succeed. Without experience, agents succeed in 63% of tasks. They follow their own default and succeed when the running version happens to match it, for example in every run where January is month 1 and in 40% where it is month 0 (Appendix § 18). With valid experience, success is 100%.
After a change, the same experience is worse than none. With out-of-date experience, success falls to 23%, 0.40 0.01 below agents without experience. Every first write uses the meaning from the earlier version. The drop appears for each of the four changes, with success between 15% and 31% (Table 17).
| Change | No experience | Valid | Out-of-date | Out-of-date, instruction to read the docs | Out-of-date, instruction to verify |
| Priority direction | |||||
| Weekday numbering | |||||
| Month numbering | |||||
| Duration unit |
Table 17. Incomplete documentation by change: success one standard error.
Experienced agents fail even when the answer is in front of them. With complete documentation, agents without experience succeed in 88–100% of tasks, and agents with valid experience in all of them. With out-of-date experience, success falls to 26–49%. It is 49% when the tool definition in front of the agent lists the new field: the definition says guests, and the agent writes attendees (Figure 1b).
Agents report the failures as successes. The service accepts the outdated call, and the agent tells the user that the task is done (Figure 2b). This happens in 347 of 348 failed runs with complete documentation, for example “The event ‘Security Review’ has been scheduled . . . with Paul and Maya as attendees” (MiniMax-M3) for an event stored without participants. With incomplete documentation the share is 92% (Appendix § 16.§ 9).
Experienced Agents Stop Checking
The current interface is one search away, and sometimes already in the tool definition. Experienced agents still fail because they copy their earlier call without checking it.
They stop looking things up. Agents without experience search the documentation when they need it: in every run when only the documentation gives the fields, and in 70% of runs when a number changes its meaning (Figure 3a). With experience, valid or out of date, at most 6% do. The earlier call alone is enough for this. In 135 of 144 earlier tasks with a renamed field or a changed structure, the agent made no lookup, so no documentation of the old version was in the conversation; the first write still repeated the old field in 61% of these runs. Agents also copy details that the task does not fix, such as the format of the start time (Appendix § 16.§ 6).

Figure 3. (a) Share of runs that search the documentation before the first write. (b) Runs whose reasoning or reply text before the first write plans to check the documentation or refers to the earlier task (complete documentation). (c) Per model, read-back after the first write against success with out-of-date experience; filled: complete documentation with typed definitions, open: incomplete documentation. The line marks equal values.
They do not tell valid from out-of-date experience. Before the first write, agents without experience often say that they will check the documentation. Experienced agents instead refer to the earlier task, and they do so as often when their experience is out of date as when it is valid (Figure 3b). Typical statements are “I already know from the previous request that the highest priority is ‘urgent’” (MiniMax-M3) and “I could search docs again, but I already know the format from the previous search” (GLM-5.3-Flash). One GLM run notices that the current schema no longer lists duration_minutes and keeps it anyway: “The tool accepted duration_minutes before. I’ll do the same.”
Agents that read back their result recover. Most failures follow one path: no lookup, old usage, no read-back. Runs that write the old usage and then read back the stored state almost always see the error and correct it (94 of 97; Appendix § 16.§ 8). Across models, success with out-of-date experience follows the read-back rate in both documentation settings (Figure 3c). Qwen3.8-Flash reads back after every write and nearly always recovers; with complete documentation, MiniMax-M3, DeepSeek-V4-Flash and Inkling never read back.
Every Form of Memory Is Affected, and More Experience Makes It Worse
So far the earlier task stayed in the conversation, and the agent could do the final task without it. Agents in practice keep experience in other forms, and they keep it because they need it. We therefore make the experience necessary. Each earlier request contains information that the final request refers to, such as “These two people are my review panel” and later “with my review panel as participants”. Without the earlier task, agents succeed in 3–10% of these final tasks. We store the experience in four ways: in the conversation; in a memory file that the agent writes after the earlier task and that appears in the system prompt of a new session; as retrieved memory, the three entries of that note that best match the final request; and as a skill, a reusable Python script that the agent wrote and ran, together with the note.
The cost appears in every design. In every design, valid experience makes the final task possible, and out-of-date experience loses much of that gain (Figure 4a). The cost is 0.53 in the conversation and 0.20–0.30 in the other designs. The content of the experience is carried over: with out-of-date experience, the first write has the right content, such as the right participants, in 95–100% of runs. What fails is the interface. The agents write the old usage into their notes: 60–73% of the memory files record the field or value of the earlier version, and 92% of the notes that come with a skill. The note-based designs lose less than the conversation, and agents with a note search the documentation more often (24–34% against 2%). We did not test why; Hu et al. (2026) likewise find that abstract memories transfer more reliably than detailed trajectories.

Figure 4. (a) Success on final tasks that need the earlier task, by where the experience is stored. (b) Success with one, three or ten earlier tasks in the conversation. Five changes, six models, one standard error; numbers in Table 5 and 7.
More earlier tasks, higher cost. With one, three or ten earlier tasks of the same kind in the conversation, success with out-of-date experience falls from 46% to 33% and 26%, while success with valid experience stays at 97–100% (Figure 4b). The lookup rate falls from 4% to 0%. An agent that has used a tool more often relies on its experience more.
Simple Remedies Fall Short
If agents copy their experience only because they have no reason to doubt it, a simple addition to the context should be enough. We tested three kinds of additions to the final task: telling the agent that the service has changed, pointing it to the documentation, and making it check the result (Figure 5; Appendix § 16.§ 2). In each condition the agent has out-of-date experience in the conversation. With nothing added, success is 46% with complete documentation and 23% with incomplete documentation.

Figure 5. Success with out-of-date experience and model calls per task, by what is added to the final task. Upper block: complete documentation (five changes; †run separately on the same task instances, where nothing added gives 0.45). Lower block: incomplete documentation (four changes). Six models; lines mark one standard error. The dashed line marks success with valid experience (2.9 and 3.1 calls), the dotted line success without experience with incomplete documentation.
Telling the agent that the service changed barely helps. A version number in every tool description and an update notice in the final task (“The service was updated on 2026-03-01; see the reference documentation for details.”) raise success only to 50%. Recording the service version in a memory file lowers the cost only from 0.27 to 0.20 (Table 5).
| Where the experience is stored | Success | Lookup before | Old usage in | Content right in | |||
| No experience | Valid | Out-of-date | Cost | first write | first write | first write | |
| Conversation | |||||||
| Memory file | |||||||
| Retrieved memory (top 3) | |||||||
| Skill (script and note) | |||||||
| Memory file + version tag | – | ||||||
| Memory file + current docs | – |
Table 5. Experience stored in different forms, with final requests that refer to the earlier task. Success and cost one standard error; lookup, old usage and content of the first write for out-of-date experience. “Content right” means that the first write carries the information from the earlier task (for example the right participants), whatever the field names. Five changes, six models. The last two rows add the service version or the current documentation to the memory file (§ 7); they were run with valid and out-of-date experience only.
Pointing to the documentation works only next to the request, and only if the documentation has the answer. An instruction in the system prompt to check the documentation before writing lowers success to 29%: the agents follow it in the earlier task (98%) but not in the final task (4%). Appended to the final task message, the same instruction makes every agent search, and success rises to 99%. The documentation itself behaves the same way: in the task message it gives 99%, in the system prompt or the tool description 54–55%. Placed next to a memory file, it lowers the cost from 0.27 to 0.10. With incomplete documentation, the instruction in the task message again makes every agent search, but the documentation does not contain the answer, and success stays at 25%. Even when the experience is still valid, the instruction raises the model calls per task from 2.9 to 3.6 and the input tokens by 40% (Appendix § 18).
Checking the result works, at more than twice the model calls. An instruction to read back the stored state before reporting completion, and to correct any mismatch, raises success to 91% with complete documentation and to 69% with incomplete documentation. The agents still write the old usage first, then read back and correct it, at 6.5 and 10.6 model calls per task against 2.9 and 3.1 with valid experience. A service can also show the result without being asked. When every write returns the stored object, success rises from 45% to 79% at 4.6 calls per task. In 59 of the 77 remaining failures, the agent receives a stored object that shows the wrong state and does not write again.
Hand-Written Histories Hide the Problem
Studies of agent memory often give the agent histories written by the authors instead of letting it gain the experience itself. We compare the two on the same tasks. The hand-written earlier task makes the correct call for its version and receives the acknowledgement. In one form it then reads
back the stored state and confirms it; in the other it ends with “Done. The event is scheduled.” With a hand-written history that contains the read-back step, success with out-of-date experience is 88%, against 46% after the agent’s own earlier task (Figure 6). The agents copy the procedure of the history: 84% read back after their first write, against 30% after their own earlier task. Without the read-back step, the read-back rate falls to 17% and success to 58%. An evaluation with hand-written histories that contain a check step would measure about a fifth of the cost that agents incur with their own experience (0.11 against 0.54). This may be one reason why the failure has received little attention.

Figure 6. Success when the agent did the earlier task itself and when we wrote it, with and without a read-back step. Above each group: the share of runs with out-of-date experience that read back after the first write. Five changes, six models, ± one standard error.
Discussion
An agent’s own successful experience changes how it uses a tool: it stops checking. After a silent change, earlier successes become failures that the agent reports as successes, and with incomplete documentation the experienced agent does worse than an agent without experience. Storing what worked is therefore not enough; a memory system also needs a way to notice that a stored usage no longer holds. In our tests, checking the stored result did this reliably, at a cost in model calls; version labels and pointers to the documentation did not. Evaluations of memory under change should use the agent’s own experience and incomplete documentation, because hand-written histories and complete documentation both hide the problem (Appendix § 10).
Limitations
The services are synthetic and small, and each version pair differs in one interface detail. Real changes often come in groups, and some produce partial errors; we chose single silent changes to isolate the effect of experience. The tasks are short, and the experience consists of one to ten earlier tasks of the same kind. With complete documentation, the documentation is correct and always one call away; in practice documentation can lag behind the service. With incomplete documentation, the earlier request states what the values mean, so the experience contains an explicit rule; we did not test experience that the agent had to find out by trial. All six models run in the same minimal agent loop; agent frameworks with their own planning or verification steps may behave differently, as the effect of the instruction to check the result suggests. Most remedies were tested only with the experience in the conversation.
Most excluded episodes come from three models whose earlier task on the weekday version in which 0 is Monday never succeeded (Appendix § 19). Their weekday estimates therefore condition on runs in which the earlier task succeeded. The standard errors resample task instances and treat the models and changes as fixed; they do not cover variation across services or models beyond those tested and may be optimistic. The conditions with incomplete documentation were not preregistered. With incomplete documentation, the success of agents without experience depends on whether the running version matches what the agent assumes by default; both directions are run for every change. The measures of reasoning text and of completion reports rely on keyword patterns (Appendices § 16.§ 7 and § 16.§ 9).
Further Discussion
Memory systems and agent harnesses. Most memory designs store what worked and assume that it stays true. Our results show that stored experience also changes how an agent works: it stops checking. A memory system therefore needs a way to find out that a stored usage no longer holds. Labelling memories with the service version does little, because agents do not act on the label. Two measures worked in our tests. A harness can attach the current documentation, or a short instruction to read it, to the request itself rather than to the system prompt; this helps only when the documentation contains the answer, and it adds model calls even when the experience is still valid. A harness can also make the agent read back the result of each write; this works whether or not the documentation is complete, at more than twice the model calls. Where the documentation may be incomplete, checking the result is the safer choice. We tested these measures with the experience in the conversation.
Tool design. Changes that the tool definition shows were copied less often than changes that only the documentation describes (old usage 69% against 99%). Meanings such as which end of a priority scale is highest or which day is day 0 can be put into the schema, for example as enum names or field descriptions, so that an agent that does not read the documentation still sees them. Write responses that return the stored object let many agents see and correct their error without being asked. Rejecting unknown fields would turn a silent change into an error; we did not test it.
Evaluation. How the experience is produced changes what an evaluation measures. Hand-written histories carry procedures that the agent copies, and one read-back step in such a history hides most of the cost. Evaluations of agent memory under change should use the agent’s own trajectories, or report the procedures that their hand-written histories contain. They should also include tasks with incomplete documentation. When the documentation spells out everything, agents without experience almost always succeed, so neither the value of experience nor the harm of out-of-date experience relative to none becomes visible.
Mechanism. We have not established why agents stop checking. The data are consistent with experience-following (Xiong et al., 2026): the more similar earlier tasks the agent has done, the more closely its first write follows them. Where text is placed also matters. The same instruction and the same documentation were acted on when they were attached to the request, and had little or a negative effect in the system prompt or a tool description.
Services, Tool Definitions and Documentation
Tools. The calendar service has calendar_create_event, calendar_update_event, calendar_delete_event, calendar_get_event and calendar_list_events. The task service has tasks_list_tasks, tasks_read_task and either tasks_update_priority or tasks_set_review_day. The mail service, used only for the reply-default change, has mail_find_thread, mail_create_reply_draft, mail_get_draft, mail_update_draft and mail_send_draft. Every service has docs_search (“Search the reference documentation for this service.”), which returns the reference documentation of the running version. Read tools return the stored state in neutral terms (for example participants, or a priority shown as Highest, High, Medium or Low), so a read-back shows whether the request took effect without revealing the field names of either version.
Typed definitions. Below is the create tool of the calendar service in version B of the field-name change; version A is identical except that guests is attendees.
{"name": "calendar_create_event", "description": "Create a calendar event.",
"parameters": {"type": "object", "properties": {"payload":
{"type": "object", "properties": {
"title": {"type": "string", "description": "Event title
."},
"start_time": {"type": "string", "description": "ISO 8601
start time."},
"duration_minutes": {"type": "integer", "description": "
Length of the event in minutes."},
"guests": {"type": "array", "items": {"type": "string"},
"description": "Email addresses of the people
taking part."}},
"required": ["title", "start_time", "duration_minutes"]}},
"required": ["payload"], "additionalProperties": false}}
In the structure change, the payload has end_time (A) or duration_minutes (B), and an event without a length lasts 30 minutes. In the value change, priority has the enum urgent, high, normal, low (A) or p0, p1, p2, p3 (B). In the priority-direction change, priority is an integer from 1 to 4 in both versions, and in the weekday change day is an integer from 0 to 6 in both versions; the descriptions are “Priority level.” and “Day of the week.”. In the documentation-only case, the calendar write tools take an untyped payload (“Request fields. Field names and values follow the service reference.”) and the priority tool an untyped string.
Documentation. The documentation of each version has a section per tool with a field table or a value list and an example. The weekday change, version A:
# Task reference
## tasks_set_review_day
Sets the day of the week on which an existing task is
reviewed. Arguments: `task_id` and
`day` is an integer from 0 to 6: 0 = Monday, 1 = Tuesday, 2
= Wednesday, 3 = Thursday,
4 = Friday, 5 = Saturday, 6 = Sunday.
Example arguments:
{"task_id": "tsk_100", "day": 5}
## tasks_list_tasks
Lists tasks whose title contains the given text, with their
review day.
## tasks_read_task
Returns the stored task, including its review day.
Version B reads “0 = Sunday, 1 = Monday, . . . , 6 = Saturday”. The priority-direction documentation reads “1 is the highest and 4 is the lowest” (A) or “4 is the highest and 1 is the lowest” (B).
Tasks. An earlier and a final task of the field-name change: “Schedule a 45-minute event titled “Hiring Debrief” on March 27, 2026 at 10:00 AM America/Los_Angeles time, with Noah ([email protected]) and Alice ([email protected]) as participants.” Priority tasks ask the agent to “set it to the highest priority level that the task service offers”; weekday tasks to “set its weekly review day to Wednesday”. Each task instance draws titles, times, people and days from fixed pools with a seed determined by the change, the direction and the repeat index.
Prompts
System prompt. “You are an agent that completes the user’s requests by calling the available tools. Carry out each request with the tools, then reply with a brief final message.”
Memory request (after the earlier task). “Write a short note for yourself to save as memory.md. It will be shown to you at the start of future sessions with this service. Record anything from this session that would help with similar requests. Reply with the note text only.”
Memory header. “Notes you saved in earlier sessions (memory.md):”; with a version tag: “Notes you saved in earlier sessions (memory.md), written when the service was at version 2.2:” (2.3 for valid experience).
Retrieved-memory header. “Notes from earlier sessions that may be relevant (retrieved from memory.md):”
Documentation header. “Reference documentation for this service:”, followed by the documentation of the running version.
Version and notice. Appended to each service tool description: “Service version 2.2.” in the earlier task and “Service version 2.3. The service was updated on 2026-03-01; see the reference documentation for details.” in the final task.
Instruction to read the documentation first. “Before your first tool call that changes data in a request, check the reference documentation for the exact fields and values to use.”
Same, in the task message. “Before your first tool call that changes data in this request, check the reference documentation for the exact fields and values to use.”
Instruction to verify and correct. “Before you reply that a request is complete, read back the stored state and confirm that every requested detail is in place. If something does not match, check the reference documentation and correct it.”
Skill. The system prompt adds: “You also have a workspace whose files persist across sessions. Python scripts in the workspace can call any of the service tools with import service and service.call(tool_name, arguments), which returns the same result as calling the tool directly.” The earlier request adds: “Do this by writing a reusable Python script in your workspace that you can run for similar requests, and run it.”
Hand-written history without read-back. The final reply is “Done. The event is scheduled.” (calendar) or “Done. The task priority is set.” (task service). In the weekday change the latter reply is also used, although that task sets a review day.
Conditions and Episode Counts
Table 2 lists the conditions. When the experience is stored in different forms, the five changes are the field-name, structure, value, priority-direction and weekday changes, with final requests that refer to the earlier task. With three or ten earlier tasks, the additional earlier tasks have titles that do not occur in the main task pools, so no two tasks in an episode share a title. The step limit is 16 model calls for each earlier and final task, and the memory request is retried once if the note is empty.
| Setting | Conditions | Changes | Arms | Episodes per model |
| Complete documentation (§ 4) | untyped tool; typed tool listing the fields; meaning of a number | 3 + 3 + 3 | none, valid, out-of-date | 324 |
| Incomplete documentation (§ 4){sup}‡` | earlier request states the meaning | 4 | none, valid, out-of-date | 288 |
| Where the experience is stored (§ 6) | conversation, memory file, retrieved memory, skill | 5 | memory-needed | 280 |
| How much experience (§ 6) | 3 or 10 earlier tasks in the conversation | 5 | typed | 120 |
| Additions to the final task (§ 7) | documentation in tool description / system prompt / task message; version and notice; instruction to read first; instruction to verify | 5 | typed | 360 |
| nothing added (rerun); instruction in the task message; write returns the stored object{sup}†` | 5 | typed | 240 | |
| 5 | typed | 60 | ||
| memory file with version tag; memory file with current documentation | 5 | memory-needed | 120 | |
| incomplete documentation; instruction to read the docs / to verify{sup}‡` | 4 | 192 | ||
| Who wrote the earlier task (§ 8) | hand-written earlier task with / without read-back step | 5 | typed | 120 |
| Total per model | 2,104 |
Table 2. Conditions. Each change runs in both directions. Instances per change, direction and arm: six with complete documentation and for the additions to the final task, three for the storage, amount and authorship conditions, and twelve with incomplete documentation. The meaning-of-a-number case with complete documentation includes the reply-default change, which is reported separately (Appendix § 16.§ 5). The nothing-added and valid arms of the additions, and the single earlier task in the amount-of-experience comparison, come from the complete-documentation conditions. {sup}†Run after the other conditions on the same task instances, with its own preregistration ({xref}Appendix H). {sup}‡Not preregistered. Six models, 12,624 episodes.
Models
We use GLM-5.3-Flash, Muse-Glimmer-30B, DeepSeek-V4-Flash (version 0731), MiniMax-M3, Qwen3.8-Flash and Inkling. All models are used with native tool calling, temperature 0.7 and at most 4,096 output tokens per call. Requests that failed were retried; an episode with a request that still failed is excluded.
Preregistered Predictions
The preregistration fixed the models, the conditions, the number of instances, the seeds, the exclusion rules, the predictions and the analysis script, and recorded the hashes of the code before the run. Table 3 lists all preregistered contrasts. Every directional prediction held. The bootstrap resamples task instances (change, direction, repeat index) within strata of change and direction. Instances with the same change share seeds across conditions, so contrasts between conditions resample the same instances jointly.
| Contrast | Prediction | Estimate | Result |
| P1 Cost, interface only in the documentation | holds | ||
| P1 Cost, change visible in the tool definition | holds | ||
| P1 Cost, change not visible in the tool definition | holds | ||
| P2 Cost, visible change − documentation only | none | negative | |
| P3 Documentation in the task message | holds | ||
| P3 Documentation in the tool description | holds | ||
| P3 Documentation in the system prompt | none | positive | |
| P3 Version number and update notice | none | positive | |
| P3 Instruction: read the documentation first | none | negative | |
| P3 Instruction: verify and correct | holds | ||
| P4 Success without experience, memory-needed tasks | holds | ||
| P4 Cost, earlier conversation | holds | ||
| P4 Cost, memory file | holds | ||
| P4 Cost, retrieved memory | holds | ||
| P4 Cost, skill | holds | ||
| P4 Cost, memory file with current documentation − memory file | holds | ||
| P4 Cost, memory file with version tag − memory file | none | negative | |
| P5 Cost, hand-written without − with read-back step | holds | ||
| P5 Cost, own earlier task − hand-written with read-back step | holds | ||
| P6 Cost, three earlier tasks − one | holds | ||
| P6 Cost, ten earlier tasks − three | none | positive |
Table 3. Preregistered contrasts (estimate ± one standard error). P1–P2: complete documentation; P3: additions to the final task, success with out-of-date experience under each addition minus success without it; P4: where the experience is stored; P5: who wrote the earlier task; P6: how much experience. A prediction holds when the 95% percentile interval lies on the predicted side; for contrasts without a prediction, the sign is reported when the interval excludes zero.
Additional Results
Main results in numbers
Table 4 gives the numbers behind Figure 2 and Figure 3a, and Table 5 those behind Figure 4a. The lower cost of retrieved memory is not due to missing information: the retrieved entries contain the needed information in 90% of runs, and in those runs the cost is 0.18. Without experience, agents with complete documentation fail mainly on the weekday change (78% success), where all 16 failures use the other numbering without consulting the documentation (Appendix § 16.§ 5).
| Change | Success | Lookup before | Old usage in | Read-back after | Failures reported | |||
| No experience | Valid | Out-of-date | Cost | first write | first write | first write | as done | |
| Incomplete documentation (four changes) | ||||||||
| Meaning of a number | ||||||||
| Complete documentation | ||||||||
| Field or value, untyped tool | ||||||||
| Field or value, in the definition | ||||||||
| Meaning of a number |
Table 4. Success on the final task with no, valid and out-of-date experience, and the cost (valid minus out-of-date), each one standard error. Lookup is given for no experience / out-of-date experience; old usage, read-back and the share of failed runs whose final reply reports the task as done are given for out-of-date experience. “Field or value”: the field-name, structure and value changes, with an untyped write tool or with a typed definition that lists the current fields. “Meaning of a number”: the priority-direction and weekday changes, and with incomplete documentation also the month and duration changes. Six models.
Additions to the final task
Table 6 gives the full results. In the earlier task, the share of agents who search the documentation is 98% under the instruction to read it first and 46–64% under the other conditions.
Who wrote the earlier task, and how much experience
Table 7 gives the numbers behind Figure 4b and Figure 6.
Per model
Table 8 gives per-model results. Every model succeeds with valid experience and fails more often with out-of-date experience, except Qwen3.8-Flash, which reads back after every write and corrects its errors. Every model succeeds in at least 97% of runs when the documentation is in the task message.
Per change
Table 9 gives the results with complete documentation by change. When the typed definition lists the current fields, agents without experience rarely search the documentation for the two calendar changes (7%), whose typed definitions are self-explanatory, but often for the value change (62%). The reply-default change in the mail service (the reply goes to the sender only or to all participants when the reply scope is omitted) shows no cost: with out-of-date experience the agents set the scope explicitly and succeed in every run. As preregistered, it is not pooled with the other changes of meaning.
Copying details
Table 10 compares the first write with the earlier call. When the interface is described only in the documentation, experienced agents reproduce the format of the start time in every run, agents without experience in 77%. With typed definitions, nearly all agents use the same format, so this measure does not separate the arms there. Field order does not separate the arms in either case.
Reasoning text
We concatenate the reasoning and reply text of all turns before the first write. A run plans a lookup if the text contains a verb such as check, look up, consult, search, read, review, verify or confirm followed within 40 characters by docs, documentation, reference, API, schema or field name, or mentions docs_search. It refers to an earlier task if it contains already, previous(ly), earlier, before, last time or same followed within 50 characters by docs, documentation, reference, field, format, structure, payload, call or API, or one of the phrases “as before”, “same as before”, “like before” and “I al-
| Added to the final task | All | Field or value | Meaning | Lookup before first write | Old usage in first write | Read-back after first write | Model calls |
| Nothing added | 0.46 ± 0.01 | 0.49 ± 0.02 | 0.42 ± 0.02 | 0.04 | 0.80 | 0.30 | 4.0 |
| Documentation in the task message | 0.99 ± 0.00 | 0.99 ± 0.01 | 0.99 ± 0.01 | 0.09 | 0.01 | 0.30 | 2.9 |
| Documentation in the tool description | 0.55 ± 0.01 | 0.61 ± 0.02 | 0.47 ± 0.02 | 0.08 | 0.62 | 0.30 | 3.6 |
| Documentation in the system prompt | 0.54 ± 0.01 | 0.57 ± 0.02 | 0.48 ± 0.02 | 0.07 | 0.63 | 0.29 | 3.5 |
| Version number and update notice | 0.50 ± 0.01 | 0.55 ± 0.02 | 0.44 ± 0.02 | 0.07 | 0.76 | 0.30 | 3.9 |
| Instruction: read the documentation first | 0.29 ± 0.01 | 0.26 ± 0.01 | 0.35 ± 0.02 | 0.04 | 0.96 | 0.29 | 4.0 |
| Instruction: verify and correct | 0.91 ± 0.01 | 0.93 ± 0.01 | 0.87 ± 0.03 | 0.03 | 0.84 | 0.92 | 6.5 |
caption: Additions to the final task, complete documentation, out-of-date experience. “Nothing added” is the out-of-date arm without additions (five changes). With valid experience the same changes give 1.00 success and 2.9 model calls.
:::
| Earlier task(s) | Valid | Out-of-date | Cost |
| Own earlier task | 1.00 ± 0.00 | 0.46 ± 0.01 | 0.54 ± 0.01 |
| Hand-written, with read-back | 0.99 ± 0.00 | 0.88 ± 0.01 | 0.11 ± 0.01 |
| Hand-written, read-back removed | 0.93 ± 0.01 | 0.58 ± 0.02 | 0.36 ± 0.02 |
| One earlier task | 1.00 ± 0.00 | 0.46 ± 0.01 | 0.54 ± 0.01 |
| Three earlier tasks | 0.99 ± 0.01 | 0.33 ± 0.02 | 0.66 ± 0.02 |
| Ten earlier tasks | 0.97 ± 0.01 | 0.26 ± 0.02 | 0.71 ± 0.02 |
caption: Who wrote the earlier task (top) and how many earlier tasks there are (bottom): success and cost ± one standard error. The first row of each block is the agent’s own single earlier task.
:::
| Model | None | Valid | Out-of-date | Old usage | Read-back | Docs in task | Verify |
| GLM | 0.98 | 1.00 | 0.70 | 0.80 | 0.53 | 1.00 | 0.98 |
| Muse | 1.00 | 1.00 | 0.25 | 0.92 | 0.23 | 0.97 | 0.82 |
| DeepSeek | 0.90 | 1.00 | 0.35 | 0.61 | 0.00 | 1.00 | 0.83 |
| MiniMax | 0.92 | 1.00 | 0.05 | 0.95 | 0.00 | 0.98 | 0.81 |
| Qwen3.8 | 1.00 | 1.00 | 1.00 | 0.88 | 1.00 | 1.00 | 1.00 |
| Inkling | 0.88 | 1.00 | 0.38 | 0.64 | 0.00 | 1.00 | 0.98 |
caption: Per model, complete documentation and typed tool definitions: success by arm, and old usage and read-back with out-of-date experience; success with out-of-date experience when the documentation is added to the task message, and under the instruction to verify and correct.
:::
| Change | None | Valid | Out-of-date | Lookup (none) | Old usage (out-of-date) |
| Field name (docs only) | 1.00 | 1.00 | 0.24 | 1.00 | 1.00 |
| Structure (docs only) | 1.00 | 1.00 | 0.21 | 1.00 | 1.00 |
| Value (docs only) | 1.00 | 1.00 | 0.33 | 1.00 | 0.97 |
| Field name | 0.99 | 1.00 | 0.49 | 0.07 | 0.65 |
| Structure | 1.00 | 1.00 | 0.46 | 0.07 | 0.61 |
| Value | 1.00 | 1.00 | 0.51 | 0.62 | 0.79 |
| Priority direction | 0.97 | 1.00 | 0.36 | 0.92 | 1.00 |
| Numbering | 0.78 | 1.00 | 0.48 | 0.49 | 0.98 |
| Reply default | 0.93 | 1.00 | 1.00 | 0.08 | 0.00 |
caption: Complete documentation by change. Top block: untyped tool; middle: typed tool that lists the current fields or values; bottom: meaning of a number.
:::
| Case | Valid | Out-of-date | None | Valid | Out-of-date | None |
| Untyped tool | 1.00 | 1.00 | 0.77 | 0.76 | 0.76 | 0.82 |
| Typed tool | 1.00 | 1.00 | 0.99 | 0.83 | 0.64 | 0.74 |
caption: Calendar changes, complete documentation: share of first writes whose start time has the same format (seconds, offset) and whose payload has the same field order as the earlier call. For agents without experience the comparison is with the earlier call of the out-of-date run on the same task instance.
:::
| Arm | With text | Plans lookup | Refers to earlier |
| No experience | 517 / 574 | 0.69 | 0.02 |
| Valid experience | 390 / 566 | 0.06 | 0.24 |
| Out-of-date experience | 407 / 565 | 0.09 | 0.25 |
caption: Complete documentation, all changes except the reply default: runs with reasoning or reply text before the first write, and the share of those that mention a plan to check the documentation or refer to an earlier task.
:::
ready know”. Table 11 gives the rates. Further examples from failed runs with out-of-date experience: “Same pattern as before - find the task and update priority to p0.” (MiniMax-M3); “I already know from the docs that priority 1 is the highest.” (Inkling); “I’ll create this event using the same field structure.” (DeepSeek-V4-Flash).
Paths to success and failure
Table 12 lists the most frequent paths. Of the 184 runs that wrote the old usage and did not read back, 183 failed; of the 97 runs that wrote the old usage and read back, 94 succeeded.
Completion reports
We classify the final reply of a run as reporting the task as done if it contains one of the words scheduled, created, set, updated, done, completed, successfully, moved, changed, added or saved, or
| Path | Runs | Share |
| no lookup, old usage, no read-back, failure | 183 | 0.52 |
| no lookup, old usage, read-back, success | 94 | 0.27 |
| no lookup, current usage, no read-back, success | 45 | 0.13 |
| looked up, current usage, no read-back, success | 12 | 0.03 |
| no lookup, current usage, read-back, success | 7 | 0.02 |
| no lookup, current usage, no read-back, failure | 3 | 0.01 |
caption: Complete documentation, typed tool definitions, out-of-date experience (350 runs): the six most frequent combinations of lookup before the first write, old usage in the first write, read-back after it, and outcome.
:::
the phrases “has been”, “is now” or “now shows”, and none of unable, could not, cannot, failed, unfortunately, sorry, error, did not or “was not”. Runs that stop at the step limit have no final reply and count as not reporting. With complete documentation, 347 of the 348 failed runs with out-of-date experience end with such a reply. With incomplete documentation, 405 of 438 do; 12 tell the user that the write did not take effect, 13 stop at the step limit, and 8 end with a reply that matches neither pattern.
Instruction in the Task Message and Stored-Object Responses
These conditions were run after the others, with their own preregistration (models, conditions, instances, predictions, analysis script and code hashes). They use the six models, the five typed changes, both directions and six task instances per cell, with the same seeds as before, so the task instances equal those of the other additions to the final task. There are three conditions, all with the earlier task live in the same conversation:
Nothing added. The out-of-date condition with complete documentation, run again.
Instruction in the task message. The final task message ends with “Before your first tool call that changes data in this request, check the reference documentation for the exact fields and values to use.” The earlier task and the system prompt are unchanged.
Write returns the stored object. In both phases, every accepted write response also contains stored, the object as the read tools return it, for example {"accepted": true, "event_id": "evt_600", "stored": {. . . , "participants": []}}. Run with valid and out-of-date experience.
The run has 1,440 episodes; 11 are excluded because the earlier task never succeeded (Inkling 7, DeepSeek-V4-Flash 3, MiniMax-M3 1, all on the weekday change), and none for failed model requests. The preregistered prediction (higher success with out-of-date experience when writes return the stored object) held: +0.34 ± 0.02. The instruction in the task message had no directional prediction; its difference from the plain condition is +0.54 ± 0.02. The cost under write responses that return the stored object is 0.21 ± 0.01. The rerun of the nothing-added condition differs from the original by −0.01 ± 0.02. Table 13 gives the process measures, and Table 14 and Table 15 give the results by kind of change and by model.
| Condition | Success | Lookup | Old usage | Read-back | Second write | Calls |
| Nothing added | 0.45 ± 0.02 | 0.03 | 0.81 | 0.30 | 0.27 | 4.0 |
| Instruction in the task message | 0.99 ± 0.00 | 1.00 | 0.02 | 0.31 | 0.02 | 3.7 |
| Instruction in the system prompt* | 0.29 ± 0.01 | 0.04 | 0.96 | 0.29 | 0.26 | 4.0 |
| Write returns the stored object | 0.79 ± 0.01 | 0.02 | 0.83 | 0.48 | 0.70 | 4.6 |
| Write returns the stored object, valid experience | 1.00 ± 0.00 | 0.01 | 0.00 | 0.11 | 0.00 | 2.7 |
Table 13. Instruction in the task message and stored-object responses, out-of-date experience unless marked (five changes, six models). Success ± one standard error; lookup before, old usage in, and read-back after the first write; share of runs with a second write; model calls. *From Table 6, same task instances.
| Condition | All | Field or value | Meaning | Runs |
| Nothing added | 0.45 ± 0.02 | 0.48 ± 0.02 | 0.39 ± 0.02 | 353 |
| Instruction in the task message | 0.99 ± 0.00 | 0.99 ± 0.01 | 0.99 ± 0.01 | 356 |
| Instruction in the system prompt* | 0.29 ± 0.01 | 0.26 ± 0.01 | 0.35 ± 0.02 | 360 |
| Write returns the stored object | 0.79 ± 0.01 | 0.81 ± 0.02 | 0.74 ± 0.02 | 360 |
| Write returns the stored object, valid experience | 1.00 ± 0.00 | 1.00 ± 0.00 | 1.00 ± 0.00 | 360 |
Table 14. Instruction in the task message and stored-object responses: success ± one standard error, overall and for the changes of a field or value (three) and of meaning (two). *From Table 6.
| Model | Nothing added | Instruction in task msg. | Instruction in system* | Write returns out-of-date | Write returns stored valid |
| GLM | 0.62 | 1.00 | 0.58 | 0.97 | 1.00 |
| Muse | 0.30 | 1.00 | 0.13 | 0.57 | 1.00 |
| DeepSeek | 0.33 | 0.98 | 0.02 | 0.98 | 1.00 |
| MiniMax | 0.07 | 0.98 | 0.00 | 0.43 | 1.00 |
| Qwen3.8 | 1.00 | 0.98 | 1.00 | 1.00 | 1.00 |
| Inkling | 0.36 | 1.00 | 0.03 | 0.77 | 1.00 |
Table 15. Instruction in the task message and stored-object responses: success by model. *From Table 6.
Incomplete Documentation
Documentation. In both versions the tool definitions and the documentation give only the type or range. The priority tool documents “priority is an integer from 1 to 4.”, the review-day tool “day is an integer from 0 to 6, one value for each day of the week.”, the review-month tool “month is an integer, one value for each month of the year.”, and the calendar documents duration as “Length of the event (required)”. The examples in the documentation are the same in both versions. The read tools return the priority as Highest, High, Medium or Low, the review day and month by name, and an event with its start, end and duration in minutes. The month change (version A: 1 is January; version B: 0 is January) and the duration change (minutes or seconds) are used only with incomplete documentation; a value outside the range of the running version leaves the stored state unchanged.
Earlier request. The earlier request ends with a sentence that says what the values mean on its version: “(For this service, priority 1 is the highest and 4 is the lowest.)” or “. . . priority 4 is the highest and 1 is the lowest.”; “(For this service, review days are numbered 0 = Monday to 6 = Sunday.)” or “. . . 0 = Sunday to 6 = Saturday.”; “(For this service, months are numbered 1 = January to 12 = December.)” or “. . . 0 = January to 11 = December.”; “(For this service, the event length is given in minutes.)” or “. . . in seconds.”. The final request has no such sentence.
Instructions. The instruction to read the documentation appended to the final task message is the one in Appendix § 17; the instruction to verify has the wording of the system-prompt instruction (Appendix § 12) and is appended to the final task message.
Runs and exclusions. Twelve task instances per change, direction and arm; task titles are drawn from 24 titles. Of 2,880 episodes, 29 are excluded because the earlier task never succeeded in ten attempts: Inkling 27, all on the weekday version in which 0 is Monday, and DeepSeek-V4-Flash 2, on the month version in which 0 is January. There were no failed model requests. Without experience, success is 49% and 71% on the two priority versions, 43% and 86% on the two weekday versions, 100% and 40% on the two month versions (1 or 0 is January), and 100% and 17% on the two duration versions (minutes or seconds). Table 16 gives the process measures, and Table 17 and Table 18 give the results by change and by model.
| Condition | Success | Lookup before first write | Old usage in first write | Read-back after first write | Second write | Model calls |
| No experience | 0.35 | 0.47 | 0.33 | 0.17 | 5.1 | |
| Valid experience | 0.02 | 0.00 | 0.32 | 0.00 | 3.1 | |
| Out-of-date experience | 0.01 | 1.00 | 0.30 | 0.28 | 4.8 | |
| Out-of-date, instruction to read the docs | 1.00 | 0.99 | 0.32 | 0.28 | 5.1 | |
| Out-of-date, instruction to verify | 0.07 | 1.00 | 1.00 | 0.90 | 10.6 |
Table 16. Incomplete documentation: the tool definition and the documentation give the range of each value but not its meaning, and the earlier request says what the values mean on its version. Success one standard error; process measures as in Table 4; second write: share of runs with more than one write. The two instructions are appended to the final task message (§ 7). Four changes, six models.
| Model | None | Valid | Out-of-date | + instruction docs | + instruction verify | Read-back (out-of-date) |
| GLM | 0.85 | 1.00 | 0.40 | 0.46 | 0.82 | 0.55 |
| Muse | 0.43 | 1.00 | 0.02 | 0.01 | 0.21 | 0.20 |
| DeepSeek | 0.49 | 1.00 | 0.00 | 0.04 | 0.74 | 0.00 |
| MiniMax | 0.52 | 0.98 | 0.01 | 0.01 | 0.75 | 0.01 |
| Qwen3.8 | 1.00 | 1.00 | 0.93 | 0.93 | 0.96 | 1.00 |
| Inkling | 0.50 | 1.00 | 0.00 | 0.03 | 0.67 | 0.00 |
Table 18. Incomplete documentation by model: success, and the read-back rate with out-of-date experience.
Cost of the instruction with valid experience. With complete documentation and the five typed changes, we also appended the instruction to read the documentation to the final task message when the experience was valid (360 episodes, the same task instances as Table 6). Success stayed at 100% (100% without the instruction); the share of runs that search the documentation before the first write rose from 1% to 100%, model calls per task from 2.92 to 3.55, input tokens from 3,839 to 5,361 and output tokens from 224 to 346. Six episodes were excluded because the earlier task never succeeded.
Calibration and Exclusions
Before the run, a calibration run with all six models, one instance per cell and one direction (1,062 episodes) checked four rules fixed in advance: C1, failed model requests in more than 3% of a model’s episodes; C2, success with valid experience below 70% in any model and condition; C3, more than 10% of valid-experience runs reaching the step limit; C4, the earlier task never succeeding in more than 10% of a cell. Only C4 was triggered, for Inkling in two of the additions to the final task, all on the weekday change; as the rule specified, the number of attempts for the earlier task was raised from seven to ten. The calibration episodes are not used.
Across all conditions, 136 of 12,624 episodes are excluded: 9 because a model request failed (MiniMax-M3 8, Muse-Glimmer-30B 1) and 127 because the earlier task never succeeded in ten attempts (Inkling 76, DeepSeek-V4-Flash 36, MiniMax-M3 15). Of the latter, 124 are on the weekday change, two on the month change and one on another change. Of all earlier tasks in the included episodes with complete documentation, 97% succeeded at the first attempt. Valid-experience runs reached the step limit in fewer than 1% of cases.
References
- [1]Laksh Advani. 2026. From Confident Closing to Silent Failure: Characterizing False Success in LLM Agents. arXiv:2606.09863. Preprint, arXiv:2606.09863.arxiv.org/abs/2606.09863
- [2]Merim Bilalić, Peter McLeod, and Fernand Gobet. 2008. Why Good Thoughts Block Better Ones: The Mechanism of the Pernicious Einstellung (Set) Effect. Cognition, 108(3):652–661.DOI
- [3]Hanxiang Chao, Yihan Bai, Rui Sheng, Tianle Li, and Yushi Sun. 2026. STALE: Can LLM Agents Know When Their Memories Are No Longer Valid? arXiv:2605.06527. Preprint, arXiv:2605.06527.arxiv.org/abs/2605.06527
- [4]Guoxin Chen, Zhong Zhang, Xin Cong, Fangda Guo, Yesai Wu, Yankai Lin, Wenzhen Feng, and Yasheng Wang. 2025. Learning Evolving Tools for Large Language Models. In International Conference on Learning Representations (ICLR). ArXiv:2410.06617.arxiv.org/abs/2410.06617
- [5]Xingwu Chen, Zhanqiu Zhang, Yiwen Guo, and Difan Zou. 2026a. Breaking Contextual Inertia: Reinforcement Learning with Single-Turn Anchors for Stable Multi-Turn Interaction. arXiv:2603.04783. Preprint, arXiv:2603.04783.arxiv.org/abs/2603.04783
- [6]Yixiong Chen, Xinyi Bai, and Alan Yuille. 2026b. The Compliance Trap: Diagnosing How AI Agents Consume Conflicting Memory. arXiv:2607.10608. Preprint, arXiv:2607.10608.arxiv.org/abs/2607.10608
- [7]Yize Cheng, Arshia Soltani Moakhar, Chenrui Fan, Parsa Hosseini, Kazem Faghih, Zahra Sodagar, Wenxiao Wang, and Soheil Feizi. 2025. Your LLM Agents are Temporally Blind: The Misalignment Between Tool Use Decisions and Human Time Perception. arXiv:2510.23853. Preprint, arXiv:2510.23853.arxiv.org/abs/2510.23853
- [8]Prateek Chhikara, Dev Khant, Saket Aryan, Taranjeet Singh, and Deshraj Yadav. 2025. Memo: Building Production-Ready AI Agents with Scalable Long-Term Memory. arXiv:2504.19413. Preprint, arXiv:2504.19413.
- [9]Mahavir Dabas, Jihyun Jeong, Ming Jin, and Ruoxi Jia. 2026. Memory-Induced Tool-Drift in LLM Agents. arXiv:2605.24941. Preprint, arXiv:2605.24941.arxiv.org/abs/2605.24941
- [10]Doist. 2026. Todoist API v1 Reference. https://developer.todoist.com/api/v1/. Sections “Sync / Tasks / Add a task” and “Create Task”. Accessed 2026-10-12.
- [11]Pengfei Du. 2026. Memory for Autonomous LLM Agents: Mechanisms, Evaluation, and Emerging Frontiers. arXiv:2603.07670. Preprint, arXiv:2603.07670.arxiv.org/abs/2603.07670
- [12]Amine El Hattami, Nicolas Chapados, and Christopher Pal. 2026. SKILL.nb: Selective Formalization and Gated Execution for Durable Agent Workflows. arXiv:2606.08049. Preprint, arXiv:2606.08049.
- [13]Leon Engländer, Sophia Althammer, Ahmet Üstün, Matthias Gallé, and Tom Sherborne. 2026. Agents Explore but Agents Ignore: LLMs Lack Environmental Curiosity. arXiv:2604.17609. Preprint, arXiv:2604.17609.arxiv.org/abs/2604.17609
- [14]Xinyi Fan, Miri Liu, Ruozhen Yang, Siru Ouyang, and Jiawei Han. 2026. Can Agent Memory Systems Track Evolving State? arXiv:2608.19652. Preprint, arXiv:2608.19652.arxiv.org/abs/2608.19652