Learning What to Remember: Resource-Efficient Self-Improvement for Lifelong Agents

Introduction

Self-improving agents continually accumulate and reuse experience from evolving task streams to improve their behavior after deployment [35]. Such experience-driven adaptation is particularly valuable for long-running agents facing evolving user needs, environments, and tasks [4, 6, 16]. However, accumulating more experience does not necessarily translate into better decisions. In practice, generation is constrained by latency, computation, or inference cost; interaction opportunities are finite; and only a bounded amount of historical experience can be stored or reused [8, 11, 29]. This raises a fundamental question: how can agents effectively improve from experience when generation, interaction, and memory resources are jointly constrained?

Our evaluation reveals that existing memory mechanisms often struggle under such constraints. As shown in Figure 1(a), several sophisticated lifelong-memory approaches provide limited improvement over simple retrieval or even no-memory agents, and can substantially underperform in some environments. These results suggest that accumulating and reusing experience alone is insufficient for reliable adaptation under stringent resource constraints. Importantly, this limitation cannot be

40th Conference on Neural Information Processing Systems (NeurIPS 2026) Workshop: Towards Test-Time Continual Learning Agents (TTCL). attributed simply to insufficient memory capacity: under the same resource budget, different memory policies can lead to markedly different downstream task performance.

To study this challenge systematically, we formalize resource-constrained lifelong agents: agents that must continually adapt from experience while operating under bounded generation, interaction, and memory budgets. As illustrated in Figure 1(b), we jointly constrain the resources available for task execution and experience reuse. Specifically, the agent operates with a bounded output budget and limited interaction rounds, while its external memory is constrained in capacity, retrieval count, and injection length. The backbone remains frozen throughout the task stream, allowing us to study experience-driven adaptation under a controlled, stringent resource budget.

Why does experience-based adaptation become fragile under such constraints? When only a small amount of experience can be retained or reused, and opportunities for reasoning and recovery are limited, each memory decision becomes more consequential. Semantic relevance alone may be insufficient: an experience can appear relevant to the current query yet provide little downstream utility. Conversely, useful experience that is not retained or selected cannot be recovered by simply injecting more context or extending the interaction. Resource scarcity therefore exposes a fundamental distinction between accumulating experience and effectively learning from it. The central challenge is not merely to retrieve relevant experience, but to determine which experiences are actually worth retaining and reusing to improve future decisions.

To address this challenge, we propose TokenWorth, a lightweight feedback-driven memory adaptation framework that learns query-dependent memory worth from sparse execution feedback. TokenWorth combines an interpretable online base scorer with a residual Recursive Feature Machine (RFM), allowing stable global patterns of memory usefulness to be complemented by nonlinear, query-specific corrections. The learned worth guides both memory retrieval and bounded memory-bank management, enabling the agent to continually adapt its experience-reuse policy without backbone updates or additional language-model calls.

We evaluate TokenWorth on three LifelongAgentBench environments covering database interaction, operating-system tasks, and knowledge-graph reasoning, under both local and API-based deployments. Across all six environment–deployment combinations, TokenWorth achieves the highest task success rate among the evaluated methods while using the fewest output tokens. These results suggest that resource-efficient adaptation depends not on accumulating more experience, but on learning what is worth remembering.

Our contributions are threefold:

• We formulate resource-constrained lifelong agents and investigate experience-driven adaptation under stringent joint constraints on generation, interaction, and memory.

• We propose TokenWorth, a lightweight online framework that learns query-dependent memory worth from execution feedback and uses it consistently for memory retrieval and bounded memory management.

• We evaluate TokenWorth across three LifelongAgentBench environments under both local and API-based deployments, demonstrating improved task success and generation efficiency under stringent resource constraints.

Related Work

Experience-Driven Self-Improvement in Language Agents. Self-improving language agents aim to continually improve from accumulated interaction experience, enabling adaptation after deployment without necessarily updating backbone parameters [6, 35]. Early agent systems demonstrated that past trajectories, reflections, or reusable skills can improve future decisions, including Reflexion [19], ExpeL [33], Voyager [24], and SocioDojo [4]. More recent work has developed explicit memory and skill mechanisms for continual adaptation, such as MemoryBank [37], Mem0 [5], MemRL [32], MemSkill [30], and AutoSkill [27]. Meanwhile, LifelongAgentBench [34], AgentCL [21], SkillLearnBench [36], and XSkill [10] provide systematic evaluations of experience-driven adaptation. While these studies explore how agents accumulate and exploit experience, we investigate how agents can learn which experiences are worth reusing to support effective adaptation under stringent resource constraints.

Resource-Constrained Agent Adaptation. Resource efficiency has been extensively studied for LLM inference and serving, including memory-efficient execution [11] and collaborative inference under computation and communication constraints [29]. Recent work has extended resource considerations to agent adaptation. Hajimiri et al. [9] study whether online memory and skill modules remain beneficial under fixed inference-token budgets, while Wu et al. [25] investigate continual learning with hard-bounded on-device memory. BudgetMem [31] considers query-aware routing across runtime memory budgets, and Chen and Cheng [3] model agentic memory value using multiple factors for retention and retrieval. These studies highlight the importance of resource-aware experience reuse, but primarily focus on individual constraints, runtime budgeting, or memory valuation in isolation. In contrast, we study experience-driven self-improvement under stringent joint constraints on generation, interaction, and memory. We propose TokenWorth, which learns query-dependent memory worth from online execution feedback to guide both retrieval and bounded memory management, enabling continual adaptation without backbone updates.

Resource-Constrained Lifelong Agents

We consider a lifelong agent that sequentially processes an online task stream $\mathcal{Q} = \{q_1, q_2, \ldots, q_T\}$. At step $t$, the agent receives task $q_t$ and may retrieve a set of past experiences $\mathcal{R}_t$ from its current memory bank $\mathcal{B}_t = \{m_1, \ldots, m_{N_t}\}$. Conditioned on $q_t$ and $\mathcal{R}_t$, the backbone model interacts with the environment for $J_t$ rounds, producing a trajectory

where $o_{t,j}$ and $a_{t,j}$ denote the observation and model output at interaction round $j$, respectively. The environment then returns a task-level reward $r_t \in\{0,1\}$. After observing this outcome, the memory policy may update the bank from $\mathcal{B}_t$ to $\mathcal{B}_{t+1}$. We focus on a frozen-backbone setting, where experience-driven adaptation occurs through online updates to external memory and its management policy, rather than model parameters.

We impose resource constraints on both task execution and experience reuse. For execution, the total amount of model generation associated with each task, including both environment interaction and any method-specific auxiliary language-model calls, is bounded by

where $A_t^{\mathrm{aux}}$ denotes the output tokens generated by auxiliary language-model calls such as memory construction, reflection, or skill extraction. For memory, the persistent bank is bounded by

while the memories supplied to the agent satisfy

We summarize the overall resource budget as

which jointly constrains generation, interaction, memory storage, and memory usage.

Let $\pi_{\mathrm{mem}}$ denote an online memory policy that determines both which experiences to retrieve for the current task and how the memory bank is updated after observing the task outcome. Through execution feedback, this policy can adapt how accumulated experience influences future decisions without modifying the backbone. For a fixed resource budget $\mathcal{C}$, we define its lifelong task performance as

The objective is therefore

where $\Pi(\mathcal{C})$ denotes the class of online memory policies satisfying the resource constraints above.

Our focus is on experience-driven adaptation under a stringent joint resource budget, where generation, interaction, and memory resources are simultaneously limited. Under such constraints, ineffective memory decisions cannot easily be compensated for by retrieving additional experience, generating longer reasoning traces, or interacting for more rounds. Each retrieval and retention decision therefore becomes more consequential. The central challenge for resource-efficient self-improvement is not merely to accumulate or retrieve relevant experience, but to learn which experiences are actually worth retaining and reusing to improve future decisions.

TokenWorth: Feedback-Driven Memory Adaptation

Overview

Figure 2 illustrates the feedback-driven adaptation process of TokenWorth. Given a query $q_t$, the agent first recalls a small candidate set from its bounded memory bank. TokenWorth estimates the query-dependent worth of each candidate using a learnable linear base scorer and an RFM-based residual correction, selecting the highest-valued experience for task execution. After receiving the environment reward, the worth model is updated online, and the learned worth further guides memory-bank management. Through this feedback loop, the agent continually adapts how past experience influences future decisions without updating its backbone.

TokenWorth consists of three components. First, an interpretable linear scorer captures stable memory-utility patterns, while a residual Recursive Feature Machine (RFM) models nonlinear, query-specific corrections [2, 18]. Second, fast value updates and slower AGOP-based feature adaptation balance stability and adaptivity under sparse feedback. Third, reward guides both retrieval and bounded memory management.

Learning Query-Dependent Memory Worth

Effective experience-driven adaptation requires identifying which past experiences are worth reusing for future decisions. Unlike a fixed memory-utility score, the usefulness of an experience depends on the current query and must be inferred from sparse execution feedback. Each task is executed only once, feedback is observed only for the memory actually used, and execution failure may arise from factors beyond the selected memory itself.

To balance stability and adaptivity under such feedback, TokenWorth decomposes memory worth into two complementary components. A lightweight linear base scorer captures stable and interpretable patterns across the task stream, while an RFM learns nonlinear residual structure not captured by the base scorer. The linear component is updated conservatively around an informative prior, whereas the RFM adapts more flexibly to local query–memory interactions.

Candidate recall and representation. Each memory $m_i \in\mathcal{B}_t$ contains a compact record of a previously successful interaction together with lightweight statistics describing its historical use. Given the current query $q_t$, we first recall a small candidate set using a lightweight textual relevance function:

where $K_c$ controls only the internal candidate pool and is independent of the number of memories ultimately supplied to the agent.

For each candidate $m_i \in\mathcal{C}_t$, we construct four interpretable features

capturing lightweight textual similarity, historical utility, reliability, and recency.

For residual estimation, we additionally construct a richer query–memory representation

This representation combines hashed textual similarity features with the empirical history of each memory. The text representation is deterministic and lightweight rather than a learned semantic embedding; detailed definitions are provided in Appendix B.

Prior-regularized base scorer. The base scorer captures stable global relationships between memory characteristics and downstream usefulness:

Rather than learning $\mathbf{w}_t$ from scratch, we initialize it from an informative prior $\mathbf{w}_0$ and regularize subsequent online updates toward this prior. Given observed execution feedback $\mathcal{D}_t^{\mathrm{base}}=\{(\boldsymbol{\phi}_j^*,y_j)\}$, we optimize

The prior anchors the scorer when feedback is scarce, while online updates allow it to capture evolving patterns of memory usefulness. Conservative periodic updates, smoothing, and bounded weights further reduce unstable drift; implementation details are provided in Appendix B.

Residual worth estimation. The linear base scorer captures stable global regularities but cannot model all nonlinear or query-specific interactions among memory signals. TokenWorth therefore augments it with an RFM-based residual correction:

where $f_t$ predicts the residual memory worth not explained by the base scorer and $\eta_t$ controls its contribution. The selected memory is

The linear scorer captures stable patterns of memory usefulness, while the RFM models nonlinear, query-specific deviations.

After task execution, the observed reward provides execution-derived supervision for the selected query–memory pair:

The base scorer is updated directly from $(\phi_t^*,y_t)$. For the residual learner, we first compute the prediction implied by the base scorer,

and define the residual target

The RFM is then updated from $(z_t^*,\delta_t)$. This feedback-driven update allows subsequent memory decisions to incorporate observed execution outcomes. Importantly, the supervision reflects downstream task success rather than the counterfactual causal contribution of $m_t^*$: TokenWorth does not re-execute the same task with alternative memories.

Two-Timescale RFM Adaptation

Continual adaptation from sparse feedback requires balancing responsiveness to recent outcomes with stability across the task stream. Within the residual learner, prediction and feature geometry need not adapt at the same rate. Recent outcomes provide immediate evidence about local residual structure, whereas the geometry governing query–memory features should evolve more conservatively. TokenWorth therefore separates fast residual adaptation from slow geometry adaptation.

Fast residual adaptation. Given a short feedback buffer $D_t^{\mathrm{fast}} = \{(z_j,\delta_j)\}_{j=1}^{N_f}$, we fit a kernel ridge regressor

where $M_t$ is an adaptive diagonal feature metric. The KRR coefficients are updated frequently, allowing the residual learner to rapidly capture deviations from the global base scorer.

Slow geometry adaptation. A fixed kernel metric would assume that different query–memory features retain the same relative importance throughout the task stream. Instead, RFM adapts the feature geometry using the Average Gradient Outer Product (AGOP) [18, 13]. Given a longer feedback window $D_t^{\mathrm{slow}} = \{(z_j,\delta_j)\}_{j=1}^{N_s}$, we compute the diagonal AGOP

and update the metric as

After each metric update, the KRR predictor is refitted under the updated geometry.

These adaptation mechanisms play complementary roles in online memory-worth learning. The prior-regularized linear scorer captures slowly evolving global regularities, the fast KRR update tracks recent residual structure, and the slower AGOP update adapts the feature geometry more conservatively. Together, they aim to balance stability and responsiveness when learning from limited execution feedback.

Worth-Guided Memory Lifecycle

Learning which memory to use for the current query addresses only part of experience-driven adaptation. Under a bounded persistent capacity $B_{\max}$, the agent must also determine which experiences deserve to remain available for future tasks. TokenWorth therefore extends the same query-dependent notion of worth from immediate retrieval to long-term memory management.

Only successful, externally verified interactions are considered as new memory candidates. For a candidate memory $m$, we estimate its broader worth over a small set of recent probe queries $\mathcal{P}_t$:

While $s_t(q,m)$ measures the query-dependent worth of $m$ for a particular query, $V_t(m)$ aggregates this signal over recent queries to estimate whether the memory deserves persistent capacity under the current task distribution.

TokenWorth follows a coverage–quality principle for memory admission. When capacity remains available, successful experiences are admitted to build initial coverage. Once the bank is full, a candidate is admitted if it is sufficiently novel, or if its probe-averaged worth exceeds that of the best existing memory and satisfies a positive-score ratio criterion. Candidates failing this admission gate may still replace a low-utility nearest neighbor when they provide a sufficiently large worth improvement. Detailed admission and replacement criteria are provided in Appendix § 9.

For candidates passing the admission gate, TokenWorth enforces the capacity bound by evicting the memory with the lowest probe-averaged worth from the enlarged bank:

TokenWorth thus applies a unified notion of memory worth at two complementary timescales: query-dependent worth determines which experience should be used now, while probe-averaged worth determines which experience should be retained for later. This feedback-driven memory lifecycle continually reallocates limited capacity toward experiences estimated to be useful for future tasks, allowing the agent to adapt its experience-reuse policy over time. All adaptation occurs within the lightweight memory controller; TokenWorth requires no backbone updates, task replay, counterfactual execution, or additional language-model calls.

Experiments

Experimental Setup

Benchmarks. We conduct our main evaluation on three environments from LifelongAgentBench [34]: database interaction (DB), operating-system interaction (OS), and knowledge-graph interaction (KG). These environments cover heterogeneous forms of interactive decision making under an online lifelong protocol, where agents can continually adapt their experience-reuse policies based on preceding interactions. In all environments, the memory bank is initialized empty, each task is executed once, and memory decisions depend only on experience observed from previous tasks.

We additionally evaluate generalization beyond LifelongAgentBench using online lifelong variants of BigCodeBench [38] and ALFWorld [20]. These results are reported in Appendix § 10.

Backbone and deployment. We evaluate TokenWorth under two complementary deployment settings. In the local setting, all methods use Qwen2.5-14B-Instruct [17] as the frozen backbone, where generation and interaction are constrained by computation and latency. In the API-based setting, all methods use DeepSeek-V4-Flash [26] through an external API, where model generation and repeated interaction directly incur inference cost. The backbone remains frozen throughout evaluation, so adaptation occurs solely through external memory mechanisms.

Resource-constrained protocol. Following § 3, we evaluate all methods under a stringent joint resource budget that constrains generation, interaction, persistent memory capacity, and memory injection. Within each environment and deployment setting, all methods process the same task stream under the same resource budget. All language-model generation incurred by a method, including auxiliary calls for memory construction, reflection, or skill extraction, is counted toward the same per-task generation budget $O_{\max}$. This controlled protocol allows us to examine the effectiveness of experience-driven adaptation when both task execution and experience reuse are severely constrained. Across DB, OS, and KG, memory capacity is 64 and at most one memory is retrieved per task. The respective output-token budgets are 128, 512, and 128, with interaction limits of 3, 5, and 15 rounds. These results characterize effectiveness under this fixed stringent budget; we do not claim robustness across a range of budget levels.

Baselines. We compare TokenWorth with a no-memory agent (Standard), vanilla retrieval-augmented generation (RAG) [15], and representative lifelong-memory methods, including MemRL [32], Mem0 [5], MemP [7], MemSkill [30], Reflexion [19], EvoSC [28], and EvoSkill [1]. All methods share the same frozen backbone, task stream, and resource budget within each setting.

Metrics. We evaluate task effectiveness using task success rate (SR $\uparrow$). To measure generation efficiency, we report the average number of output tokens per task (OutputTok $\downarrow$), including all language-model generation incurred during task execution and method-specific auxiliary operations such as memory construction, reflection, or skill extraction. We further report the relative change in output-token usage with respect to the no-memory Standard agent:

where a negative value indicates lower generation consumption than the no-memory baseline.

Results and Analysis

RQ1: Can TokenWorth support effective experience-driven adaptation under stringent resource constraints? Table 1 reports the main results on LifelongAgentBench. TokenWorth achieves the highest task success rate in all six environment–deployment combinations, demonstrating the effectiveness of its feedback-driven memory adaptation framework across both local and API-based backbones.

Method	Local: Qwen2.5-14B-Instruct DB	Local: Qwen2.5-14B-Instruct OS	Local: Qwen2.5-14B-Instruct KG	API: DeepSeek-V4-Flash DB	API: DeepSeek-V4-Flash OS	API: DeepSeek-V4-Flash KG
Standard	25.2	50.2	<u>30.7</u>	26.2	67.2	12.7
RAG	<u>33.0</u>	<u>54.6</u>	11.3	29.4	66.6	31.3
MemRL	28.0	52.2	4.7	30.4	66.2	<u>32.0</u>
Mem0	21.2	50.6	<u>30.7</u>	<u>32.2</u>	68.8	15.3
MemP	23.4	46.8	27.3	28.0	69.8	21.3
MemSkill	24.4	50.8	26.0	27.4	66.0	20.7
Reflexion	22.0	48.8	30.0	30.6	67.2	14.7
EvoSC	19.2	50.2	27.3	30.6	<u>70.2</u>	18.7
EvoSkill	24.6	49.6	30.0	25.6	68.4	14.7
TokenWorth	65.0	59.4	36.7	62.2	73.2	36.7

Under the local deployment, the improvement is particularly pronounced on DB, where TokenWorth reaches 65.0%, outperforming the strongest baseline RAG (33.0%) by 32.0 percentage points. It also improves the strongest baseline by 4.8 points on OS and 6.0 points on KG. The same pattern holds under the API-based deployment: TokenWorth improves the best competing method by 30.0 points on DB, 3.0 points on OS, and 4.7 points on KG. Notably, several memory-based methods provide limited gains over the no-memory Standard agent and can even substantially underperform it, particularly on KG. These results suggest that accumulating and reusing experience alone does not guarantee effective adaptation. Learning which experiences are worth reusing is particularly important when generation, interaction, and memory resources are jointly constrained.

RQ2: Can effective experience reuse improve generation efficiency? Figure 3(a) compares TokenWorth’s output-token usage across the six LifelongAgentBench environment–deployment combinations. We report the relative change with respect to the no-memory Standard agent, while the diamonds indicate the most token-efficient competing method. TokenWorth achieves the lowest output-token usage in all six settings.

Under the local Qwen2.5-14B-Instruct deployment, TokenWorth reduces output generation by 9.3%, 51.5%, and 18.7% on DB, OS, and KG, respectively. Under DeepSeek-V4-Flash, the corresponding reductions are 31.6%, 58.3%, and 21.3%. The reduction is particularly substantial on OS, where TokenWorth uses less than half the output tokens of the Standard agent under both deployments.

These savings accompany the highest success rate in all six settings (Table 1). TokenWorth also uses fewer output tokens than the most generation-efficient baseline in each setting, with clear margins on DB and OS. Although the margin over MemRL on API-based KG is smaller, TokenWorth still improves both success and token usage. These results suggest that learning to reuse higher-value experience helps the frozen backbone solve more tasks with less generation, supporting resource-efficient experience-driven adaptation.

RQ3: How do the adaptation components affect memory-worth learning? Figure 3(b) reports a core ablation on DB with Qwen2.5-14B-Instruct. We compare the base scorer alone (Base Only), the base scorer augmented with an RFM residual learner using a fixed feature metric (Base + RFM), and the full framework with AGOP-based feature adaptation (Full).

The base scorer achieves 64.2%, demonstrating the effectiveness of stable, interpretable memory signals. Adding the RFM residual learner with a fixed feature metric reduces performance to 63.0%, indicating that nonlinear residual modeling alone does not necessarily improve memory-worth estimation. With AGOP-based feature adaptation, the full framework reaches 65.0%, recovering this degradation and modestly outperforming the base-only variant. These results suggest that adaptive feature geometry improves residual corrections under sparse execution feedback, highlighting the importance of balancing stability and adaptivity in online memory-worth learning.

Conclusion

We study experience-driven adaptation in lifelong agents under joint constraints on generation, interaction, and memory. We propose TokenWorth, a lightweight feedback-driven memory adaptation framework that learns query-dependent memory worth from online execution feedback and uses it for retrieval and bounded memory management. Across three LifelongAgentBench environments and local and API-based backbones, TokenWorth achieves the highest task success rates among evaluated methods while using the fewest output tokens. Additional experiments show competitive performance on coding and embodied-agent tasks. These results suggest that resource-efficient self-improvement depends not on accumulating more experience, but on learning what is worth remembering. Since feedback is observed only for selected memories, their causal contributions remain unidentified. Developing budget-efficient credit assignment remains an important direction for future work.

Acknowledgments and Disclosure of Funding

This work was supported by the National Key R&D Program of China under Grant No. 2024YFE0202800 and the National Natural Science Foundation of China under Grant Nos. 62522605 and 62376118.

Additional Experimental Details

Benchmarks

Our main experiments use three environments from LifelongAgentBench [34]: database interaction (DB), operating-system interaction (OS), and knowledge-graph interaction (KG). Tasks are processed sequentially, and only experience from preceding tasks may be reused. DB requires structured database operations and answers grounded in retrieved observations. OS requires multi-step system management through Bash commands. KG follows an AgentBench-style protocol in which the agent interacts with a knowledge graph through structured tool calls, while SPARQL queries are executed by the backend. The environments thus cover heterogeneous forms of interactive decision making under a shared online lifelong-learning protocol.

We additionally evaluate online lifelong variants of BigCodeBench [38] and ALFWorld [20]; their results are reported in Appendix § 10.

Beyond language agents, class-incremental learning has examined the impact of task ordering on continual adaptation [12] and the variability of performance across evaluation sequences [14]. These studies highlight the importance of evaluation protocols in continual learning, while our experiments focus on online memory adaptation under fixed resource budgets.

Baselines

We compare TokenWorth with a no-memory agent (Standard), retrieval-augmented generation (RAG), and representative memory- and skill-learning methods.

Standard. The frozen backbone solves every task independently, without persistent cross-task memory.

RAG. RAG [15, 22, 23] stores successful experiences and retrieves them by lightweight textual relevance, without online value learning.

MemRL. MemRL [32] uses execution feedback to learn utility signals for episodic memories, serving as a value-aware baseline.

Mem0. Mem0 [5] supports memory extraction, storage, retrieval, and update in long-running agents.

MemP. MemP [7] retains procedural task experience for later reuse.

MemSkill. MemSkill [30] converts experience into reusable memory skills and evolves them online.

Reflexion. Reflexion [19] stores verbal reflections derived from task feedback for reuse in subsequent tasks.

EvoSC. EvoSC [28] consolidates and evolves agent experience.

EvoSkill. EvoSkill [1] extracts reusable skills from interaction trajectories and updates its skill repository.

Within each environment and deployment setting, methods share the same task sequence, backbone, environment interface, and resource budget. We preserve baseline-specific auxiliary operations such as reflection, memory construction, and skill extraction; their generated output tokens are included in both the per-task budget and the reported OutputTok. Baseline implementations are adapted to satisfy the common resource limits.

Backbones and Deployment Settings

Local deployment. All methods use the frozen Qwen2.5-14B-Instruct [17] backbone on NVIDIA A800 GPUs.

API-based deployment. All methods use the frozen DeepSeek-V4-Flash [26] backbone through an external inference API.

In both deployments, the memory bank starts empty, task order is fixed within an environment, and no backbone parameters are updated. TokenWorth uses neither offline pretraining nor task replay or counterfactual re-execution.

Evaluation Protocol

We measure task effectiveness with task success rate (SR ↑). A task succeeds only if the environment evaluator marks its final answer or state as correct. Generation efficiency is measured by average output tokens per task (OutputTok ↓), including tokens from task execution and all method-specific auxiliary language-model calls. Relative to Standard,

Negative values indicate fewer generated tokens. OutputTok does not include input tokens or controller-side computation and should not be interpreted as total inference cost.

Implementation Details of TokenWorth

Common Online Implementation

Online feedback protocol. At the beginning of task $t$, retrieval uses the bank $B_t$ and feedback available through task $t-1$. Task $t$ is executed once. Following its completion, the observed reward updates the worth model and the selected memory’s statistics, and memory admission or eviction constructs $B_{t+1}$. These updates may affect task $t+1$ and later tasks; there is no same-task counterfactual execution or backbone fine-tuning.

Lightweight text representation. We construct deterministic 512-dimensional text vectors by token hashing and character 3–5-gram hashing, followed by $\ell_2$ normalization. This representation requires no learned encoder and captures lightweight textual similarity rather than learned semantic embeddings. For a query $q$ and memory $m$, the residual input is

Here $u_m$ is historical utility. With $h_m$ and $d_m$ denoting counts of successful and unsuccessful executions, respectively, conditioned on selecting $m$, the other statistics are

Here $\Delta t_m$ is the number of steps since last access, $c_m$ is the access count, and $\tau_m$ is the creation step. We set $\kappa=0.02$. The success and failure counts are execution-derived statistics, not causal estimates of a memory’s effect.

Candidate recall and base scorer. The bank is screened by lightweight textual relevance to form

For each candidate, define $\boldsymbol{\phi}(q,m)=[\phi_{\mathrm{sim}},\phi_{\mathrm{util}},\phi_{\mathrm{rel}},\phi_{\mathrm{recent}}]$ and $s_{\mathrm{base},t}(q,m)=\mathbf{w}_t^{\mathsf{T}}\boldsymbol{\phi}(q,m)$. We initialize $\mathbf{w}_0=[1.0,0.10,0.10,0.02]$ and fit

The FIFO training window contains at most 256 selected query–memory pairs, with $y_j \in\{-1,+1\}$. We use $\lambda_w = 0.1$ and refit after every 50 new feedback examples using deterministic Newton optimization with backtracking line search. Updated weights are smoothed with coefficient $\beta_w = 0.5$ and clipped coordinate-wise between $[0,0,0,0]$ and $[2.0,1.0,1.0,1.0]$.

Residual worth estimation. The combined score and selected memory are

To avoid abrupt residual corrections during early adaptation, we linearly increase the residual coefficient over the first three KRR fits:

where $n_{\mathrm{KRR}}(t)$ denotes the number of completed KRR fits and $\eta_{\max}$ is the environment-specific coefficient given in Table A1. For a task reward $r_t \in\{0,1\}$, the execution label and residual target are

Setting	DB	OS	KG
Generation budget $O_{\max}$	128	512	128
Interaction budget $J_{\max}$	3	5	15
Memory capacity $B_{\max}$	64	64	64
Retrieved memories $K_{\max}$	1	1	1
Injection budget $L_{\max}$	900 chars	800 chars	4096 chars
Candidate pool size $K_c$	12	12	12
KRR update interval	50	25	25
AGOP update interval	200	100	100
Residual influence $\eta$	0.15	0.05	0.10
RFM warm-up feedback	4	4	24

Only the selected pair $(q_t,m_t^\star)$ receives feedback. These labels reflect downstream task outcomes, not same-task comparisons among memories.

Kernel ridge regression and AGOP. The residual predictor fits kernel ridge regression (KRR) to a FIFO window of at most 128 feedback pairs $\{(z_j,\delta_j)\}$, using the Laplacian kernel

with $\lambda_{\mathrm{KRR}} = 0.01$ and $\gamma= 1.0$. The metric $M_t$ is diagonal. Fast KRR coefficients are periodically refitted; slower feature-geometry updates use the diagonal Average Gradient Outer Product (AGOP):

The slow window contains at most 256 feedback examples. We set $\beta_M = 0.25$, clip diagonal weights to $[0.05,8.0]$, and normalize them to mean 1. KRR is refitted after every AGOP update; update intervals are listed in Table A1.

Execution-derived memory statistics. A newly created memory starts with utility $u(m)=1.0$. After a task with selected memory $m_t^\star$, its utility is updated without an EMA:

Successful (unsuccessful) task execution with the selected memory increments $h_m$ ($d_m$). Access count and recency are updated after each use.

Memory admission, replacement, and eviction. Only externally verified successful tasks produce candidate memories. While $|\mathcal{B}_t| < B_{\max}$, each eligible candidate is admitted to establish coverage. For a full bank, let $m_{\mathrm{near}}$ denote the nearest existing memory in the same hashed textual representation. Define

The probe set $\mathcal{P}_t$ includes the current query and up to eight previous queries on which memory retrieval occurred (FIFO), hence $|\mathcal{P}_t| \le9$. The positive-score ratio $\rho_t^{+}$ measures the fraction of probe queries for which the candidate receives a positive worth score, rather than the fraction on which it outperforms the nearest memory.

For a full bank, a candidate passes the admission gate if

where $\tau_{\mathrm{nov}} = 0.35$, $\delta_{\mathrm{write}} = 0.02$, and $\tau_{\mathrm{pos}} = 0.45$. Thus, admission requires either sufficient novelty or a worth improvement over the best existing memory together with a sufficiently high positive-score ratio.

Candidates passing this admission gate are added to the bank. The capacity bound is then enforced by evicting the memory with the lowest probe-averaged worth from the enlarged bank:

The newly admitted candidate may itself be evicted if it has the lowest probe-averaged worth.

If the candidate fails the admission gate, the controller instead considers direct replacement of its nearest existing memory. Replacement occurs only when the nearest memory has low utility and the candidate exceeds its probe-averaged worth by a larger margin:

When both conditions hold, $m_{\mathrm{new}}$ directly replaces $m_{\mathrm{near}}$ without invoking the lowest-worth eviction rule. Candidates satisfying neither the admission gate nor the fallback replacement conditions are rejected.

Common Hyperparameters

The main common parameters are $\mathbf{w}_0 = [1.0, 0.10, 0.10, 0.02]$, $\lambda_w = 0.1$, $\beta_w = 0.5$, $\lambda_{\mathrm{KRR}} = 0.01$, $\gamma= 1.0$, $\beta_M = 0.25$, and $\kappa= 0.02$. The base, fast KRR, and slow AGOP feedback windows contain at most 256, 128, and 256 examples, respectively. The shared memory-management thresholds are $\tau_{\mathrm{nov}} = 0.35$, $\delta_{\mathrm{write}} = 0.02$, $\tau_{\mathrm{pos}} = 0.45$, and $\delta_{\mathrm{rep}} = 0.10$. Further bounds and update rules are specified in Appendix B above, and the environment-dependent settings in Table A1.

Environment-Specific Configuration

TokenWorth uses the same worth-learning method across DB, OS, and KG, with resource limits and online-update intervals adapted to the environment.

Environment-Specific Memory Construction

DB. Successful tasks are stored as compact demonstrations containing the question, valid SQL actions, truncated database observations, and final answer. Invalid actions and unnecessary output are omitted.

OS. Complete shell trajectories contain instance-specific paths, user names, and process identifiers that transfer poorly. We retain concise patterns from successful Bash actions, including permission management, batch file processing, text editing, symbolic links, and process management. A task is successful when the environment evaluator reports correct, including when the interaction limit is reached after a correct final state.

KG. Memories contain successful knowledge-graph tool calls and truncated knowledge-graph observations, rather than SPARQL queries executed by the backend. KG allows up to 15 interaction rounds and 4096 characters of injected memory; residual adaptation starts after at least 24 query–memory feedback pairs. In all three environments, TokenWorth first screens up to $K_c = 12$ memories and injects at most one selected experience.

Generalization Beyond LifelongAgentBench

We evaluate online lifelong variants of BigCodeBench [38] (code generation) and ALFWorld [20] (text-based embodied decision making). Each run starts without stored experience from benchmark tasks. Tasks are processed sequentially, with retrieval using only preceding experience. Memory is updated after the current task has been evaluated, without access to future tasks. The language-model backbone remains frozen throughout each run.

BigCodeBench. We use BigCodeBench Full v0.1.4 in the Instruct format, comprising 1,140 coding problems. Each method makes a single pass through the same deterministic sequence: the 992 problems in Full excluding Hard, followed by the 148 problems in the Hard subset. Within these two groups, we preserve the order of the pinned Full and Hard dataset files, respectively; no random shuffling is applied. The memory bank is carried across the two groups without resetting. For each problem, the agent generates one solution, which is sanitized and evaluated using BigCodeBench’s official execution-based evaluator inside a Docker worker. A problem succeeds only when the evaluator returns pass; generated-code failures and execution timeouts count as unsuccessful outcomes. Evaluator infrastructure errors invalidate the run. We report pass@1 over all 1,140 problems, counting each problem once. Memory is updated after evaluation, and previously evaluated problems are not revisited. The local deployment uses Qwen2.5-14B-Instruct. The API deployment uses DeepSeek-V4-Flash with thinking explicitly disabled. Both deployments use temperature zero. Reference solutions and unit tests are excluded from the model’s memory and input prompts.

ALFWorld. We use Qwen2.5-14B-Instruct as the frozen backbone, with temperature zero and top-$p$ equal to one. A complete run contains 574 episodes: 300 training episodes selected by task-type-stratified sampling, all 140 validation-seen episodes, and all 134 validation-unseen episodes. The phases run in this order. The base seed is 42; the sampling and ordering routines use seeds 42, 43, and 44 for the three phases, respectively. Within each phase, game paths are sorted, shuffled within task-type groups using the phase seed, and selected by round-robin traversal of the groups. The resulting ordered episode manifest is fixed and shared across methods, and the runner checks that the environment follows this order. Each selected episode is executed once. An episode succeeds when TextWorld’s external won flag indicates task completion. The success rate reported here is the total number of successful episodes divided by 574, including all three phases; it is not an unweighted average of phase success rates. Memory persists across phase boundaries and is updated after every completed episode, including validation-seen and validation-unseen episodes. Thus, this is an online lifelong protocol rather than evaluation with memory frozen after training. Expert trajectories are excluded from memory.

Resource accounting. We follow the resource-constrained evaluation principle in § 3, using benchmark-specific limits. Within each benchmark and deployment, methods share the backbone, ordered task stream, solver-call output limit, and, for ALFWorld, interaction controller. Memory contexts are bounded by the complete injected block, including its headings and separators. Injection lengths below are measured in characters. The two adapters enforce output limits per model call, rather than pooling all calls into a single per-task output allowance. Method-specific auxiliary calls, including memory construction, reflection, and skill extraction, have a separate limit of 512 output tokens per call. Their actual usage is recorded in addition to solver usage, including action repairs in ALFWorld. TokenWorth makes no auxiliary language-model calls. Consequently, the methods share per-call limits but need not consume equal total output tokens per task or episode.

BigCodeBench configuration. The solver output limit is $O_{\max} = 1280$ tokens per call for both the local and API deployments. Auxiliary API calls have thinking disabled. TokenWorth uses a memory capacity of $B_{\max} = 50$ records, injects at most $K_{\max} = 1$ record, and caps the complete injection block at $L_{\max} = 4096$ characters. Each stored experience is also capped at 4096 characters. BM25 recalls at most $K_c = 12$ candidates for learned worth ranking. All TokenWorth representation, learning, and memory-management parameters are identical across the local and API deployments; the solver output budget and thinking setting follow the selected deployment.

ALFWorld configuration. The interaction limit is $J_{\max} = 50$ environment steps per episode. Each action-generation call has an output limit of $O_{\max} = 128$ tokens. An inadmissible action permits at most one repair call for that step, also capped at 128 output tokens. These are per-call limits, not a 128-token allowance for the entire episode. The same action controller is used for all methods, with memory-dependent loop removal disabled. TokenWorth uses $B_{\max} = 128$ records, $K_{\max} = 1$, a complete injection budget of $L_{\max} = 4200$ characters, a per-experience limit of 4096 characters, and a BM25 candidate pool of at most $K_c = 12$.

Online learning configuration. Both benchmarks use the same deterministic 512-dimensional hashing representation of tokens and character 3–5-grams. Only a memory actually injected into the agent receives outcome supervision. KRR and AGOP updates are scheduled every 75 and 225 accumulated feedback examples, respectively, with FIFO buffers of 128 and 256 examples. These counters track feedback examples, not elapsed tasks or episodes. Each scheduled residual fit requires at least four feedback examples; this minimum does not trigger an immediate fit at the fourth example. Under these settings, the first scheduled KRR fit occurs at feedback example 75. The residual coefficient is $\eta_t = 0.10 \min(1, u_t/3)$, where $u_t$ is the number of completed KRR fits: its influence ramps linearly over the first three fits. KRR uses ridge regularization 0.01 and Laplacian-kernel scale 1. AGOP uses smoothing 0.25, clips the updated diagonal metric to $[0.05, 8]$, normalizes it to have mean one, and immediately refits KRR. The base scorer is refitted every 50 feedback examples, using a FIFO of 256, regularization 0.1, and smoothing 0.5. Successful interactions can create new experiences; unsuccessful interactions provide negative feedback to injected memories without creating successful-experience records.

Table A2 shows that TokenWorth achieves 43.1% on local BigCodeBench and 73.5% on local ALFWorld, exceeding the best competitors by 0.2 and 0.3 percentage points, respectively. On API-based BigCodeBench, TokenWorth reaches 50.6%, below RAG’s 50.9%. These small margins support competitive transfer rather than consistent dominance beyond the three main interactive environments.

Method	Local: Qwen2.5-14B-Instruct — BigCode	Local: Qwen2.5-14B-Instruct — ALFWorld	API: DeepSeek-V4-Flash — BigCode
Standard	41.7	59.1	49.9
RAG	42.6	<u>73.2</u>	50.9
MemRL	42.9	69.7	50.5
Mem0	41.7	64.1	50.2
MemP	42.3	66.4	49.6
MemSkill	42.9	63.6	49.1
Reflexion	40.8	69.2	49.0
EvoSC	42.6	65.0	49.6
EvoSkill	41.8	64.8	48.8
TokenWorth	43.1	73.5	<u>50.6</u>

Hyperparameter Sensitivity

We evaluate the sensitivity of TokenWorth to its key hyperparameters on the same 500-task DB stream using Qwen2.5-14B-Instruct. Starting from the default configuration, which achieves a 65.0% task success rate, we vary one parameter at a time while keeping all others fixed. Table A3 summarizes the results.

Parameter	Default	Tested value	SR (%)	$\Delta$ (pp)
RFM influence $\eta$	0.15	0.05	62.4	$-2.6$
RFM influence $\eta$	0.15	0.10	63.8	$-1.2$
RFM influence $\eta$	0.15	0.20	64.2	$-0.8$
RFM influence $\eta$	0.15	0.30	64.0	$-1.0$
Base regularization $\lambda_w$	0.10	0.03	64.8	$-0.2$
Base regularization $\lambda_w$	0.10	0.30	62.6	$-2.4$
Base regularization $\lambda_w$	0.10	1.00	63.0	$-2.0$
Base smoothing	0.50	0.25	64.2	$-0.8$
Base smoothing	0.50	0.75	62.8	$-2.2$
Base smoothing	0.50	1.00	62.6	$-2.4$
Base update interval	50	25	65.2	$+0.2$
Base update interval	50	100	62.4	$-2.6$
AGOP smoothing	0.25	0.10	64.4	$-0.6$
AGOP smoothing	0.25	0.50	64.6	$-0.4$
AGOP smoothing	0.25	0.75	62.2	$-2.8$
KRR update interval	50	25	62.2	$-2.8$
KRR update interval	50	100	63.4	$-1.6$
AGOP update interval	200	100	62.6	$-2.4$
AGOP update interval	200	400	62.0	$-3.0$
Kernel $\gamma$	1.0	0.5	64.0	$-1.0$
Kernel $\gamma$	1.0	2.0	63.0	$-2.0$

TokenWorth maintains relatively stable performance under moderate changes to several hyperparameters. Varying the RFM influence $\eta$ between 0.10 and 0.30 results in performance differences of at most 1.2 percentage points, while adjusting AGOP smoothing between 0.10 and 0.50 leads to differences within 0.6 points. Reducing the base update interval from 50 to 25 yields a comparable success rate of 65.2%.

Larger deviations in adaptation settings lead to more pronounced performance degradation. Reducing $\eta$ to 0.05 decreases success by 2.6 points, while increasing AGOP smoothing to 0.75 causes a 2.8-point drop. The update schedules also play an important role: changing the KRR update interval from 50 to 25 reduces success by 2.8 points, while changing the AGOP interval from 200 to 100 or 400 decreases success by 2.4 and 3.0 points, respectively. These results highlight the importance of balancing adaptation speed and stability, supporting the conservative, two-timescale design of TokenWorth.