betaFeedback

WorksComputer Science

MARS: Memory-Aware Routing for Session-Level LLM Assistants

Abstract

Long-term LLM assistants must repeatedly decide what prior context to expose and which model should answer as an interaction evolves. Existing memory systems and model routers usually make these decisions separately, although memory construction changes the evidence, distractors, and difficulty of the resulting input. We introduce MARS, a memory-aware routing framework that uses session state as a shared control interface between memory construction and model selection. A Query and Topic Manager (QTM) estimates compact decision-relevant state that guides a fixed state-conditioned memory policy and an online router, enabling each turn to receive an appropriate memory view and a suitable model under a quality-memory-cost trade-off. We evaluate MARS on RealMem and MultiChallenge with 16 system configurations each, covering long-history memory-driven interaction and bounded sessions with structured context interference. On RealMem, MARS preserves 98.2% of full-context GPT-4o's QA while using only 14.5% of its generation API cost, including both generation and controller calls for MARS, and achieves the strongest measured memory-recall and retrieval scores among compared systems. On MultiChallenge, MARS trails the strongest static system overall but approaches its performance at lower recorded cost on a mechanism-focused subset. Together, these results support a favorable quality-memory-cost trade-off for long-history, memory-dependent interaction.

Introduction

Modern LLM assistants are no longer used only for isolated questions; they increasingly support ongoing work across many turns and sessions. In long-term use, interactions can span many turns and recur across tasks, a setting highlighted by recent work on long-term conversational and memory-driven agents [13, 22, 20, 3]. Users may resume unfinished artifacts, request revisions, refer to earlier constraints, or ask for final answers that depend on decisions made several turns or sessions earlier. Before answering such turns, an assistant must decide both what prior context to include and which model should answer. This creates a quality–memory–cost trade-off: richer memory can provide necessary evidence but also increases context length and distractors, while stronger models can better interpret complex memory views but are more expensive.

Existing systems usually handle these decisions separately. Memory systems store, update, and retrieve prior information [10, 20, 30] (Chikhara et al. 2025), while model routers select a generator from a candidate pool to trade off quality and cost [4, 8, 7, 17, 16, 23, 27]. This separation overlooks a key dependency: memory construction determines what evidence, constraints, and distractors enter the prompt, while routing determines whether the selected model can use that context reliably and cost-effectively.

This dependency makes memory construction and model routing inherently coupled. As Figure 1 illustrates, finalizing a launch brief requires using the updated launch date (May 19, superseding May 12) while preserving an earlier constraint of exactly three paragraphs. Omitting the constraint or exposing the obsolete date can cause errors, while an unsuitable model may fail even when the relevant memory is present. Exposing all history can add distractors and increase context difficulty [12, 2], and always invoking a stronger model can incur unnecessary cost. The appropriate model depends on the selected memory view rather than the raw query alone. The key challenge is therefore to coordinate what history becomes input and which model can use that context reliably and cost-effectively.

Figure 1. Illustrative motivation for memory-aware routing in a multi-turn session. MARS uses a shared QTM-derived session state to coordinate task-relevant memory construction and online model routing. The surrounding panels illustrate missing constraints, stale memory, model mismatch, and costly overuse. A–C denote candidate models; outputs are schematic and costs qualitative, with “lower” relative to the overuse configuration.

Motivated by this coupling, we study session-level adaptive memory routing: coordinating memory exposure and model selection throughout a session under a quality–memory–cost trade-off. We formulate this as a sequential decision problem: the system estimates session state, constructs a candidate memory view, selects a model, packs the view for that model, generates a response, and updates its routing preference from turn-level feedback. Because the memory view changes the evidence, distractors, context length, and reasoning burden faced by the selected model, the system needs a shared session-state interface for both memory construction and model routing.

We introduce MARS, a memory-aware routing framework for session-level LLM assistants. MARS uses a Query and Topic Manager (QTM) to infer structured session state from the query, active thread, recent turns, and memory pool. The memory policy uses this state to build a task-relevant candidate memory view. The online router uses the same state and pre-packing view statistics to select a model; the selected model’s budget then determines the final context packing. After generation, turn-level feedback updates the router, enabling lightweight adaptation without retraining the underlying LLMs. The QTM prompt, schema, guardrails, and state-to-memory mapping are fixed; the memory view varies by turn, while the router learns online. MARS achieves coordination through a shared structured session-state interface.

We evaluate MARS on two representative multi-turn benchmarks, REALMEM ([3]) and MULTICHALLENGE ([25]), using 16 system configurations on each. The experiments test reliability across bounded session interference and long-history memory-driven interaction. On REALMEM, MARS retains 98.2% of full-context GPT-4o’s persona-macro QA with generation-controller API cost equal to 14.5% of GPT-4o’s generation cost while obtaining the strongest measured memory-recall and retrieval scores. On full MULTICHALLENGE, full-context Qwen2.5 72B is strongest (67/245), followed by GPT-4o (64/245), while MARS reaches 58/245. On the two axes most aligned with state recovery and version editing, MARS reaches 44/137 versus 45/137 for Qwen2.5 72B at lower recorded cost. Together, these results support our central claim: coordinating memory exposure and model choice through shared session state provides a useful quality–memory–cost operating point in long-history, memory-dependent interaction.

Our contributions are:

  • We formulate session-level adaptive memory routing as a coordinated sequential decision problem over memory exposure and model selection under a quality–memory–cost trade-off.

  • We introduce MARS, which uses QTM-derived state as a shared control interface for fixed-policy memory construction, online model routing, and feedback updates.

  • We provide a cross-regime evaluation on two representative multi-turn benchmarks, showing that MARS achieves a strong quality–memory–cost trade-off in long-history interaction.

Related Work

LLM routing and online model selection. Model routing reduces inference cost by selecting among candidate LLMs instead of using a single expensive model for every request. Prior work studies cascades, benchmark-trained routers, quality-aware routing, preference-trained routers, and dynamic model selection for quality–cost trade-offs [4, 24, 7, 8, 17, 16, 27]. Recent work builds on contextual-bandit foundations [11, 6, 1] and frames multi-LLM choice as a sequential decision problem with feedback, preferences, or budget constraints [23, 21, 28]. These methods primarily route from the current query, prompt, or learned query–model affinity. MARS differs in the decision interface: the router operates over structured session state, and this same state also controls memory construction.

Long-term memory, adaptive retrieval, and session evaluation. Long-term memory systems help agents store, update, and retrieve information across interactions, from retrieval-augmented generation [10] and generative agents [22] to memory-management systems such as MemGPT, Mem0, and A-MEM [20, 5, 30]. Long-context studies show that simply increasing context length does not guarantee reliable evidence use: agents still struggle with very long conversational history [13], and models can fail depending on where evidence appears in the context [12, 2]. Closest to adaptive retrieval, MBA-RAG uses a multi-armed bandit to select retrieval strategies based on question complexity and retrieval cost [26]; however, it does not jointly route the generator model or reason over evolving session state. Benchmarks such as REALMEM and MULTICHALLENGE evaluate complementary aspects of memory-driven and path-dependent multi-turn interaction [3, 25]. We use them to ask a different question: whether memory exposure and model routing remain reliable when the operating regime shifts from bounded session interference to longer histories, stronger memory pressure, and more difficult retrieval.

Method: Session-Level Adaptive Memory Routing

We study session-level adaptive memory routing: coordinating what memory to expose and which model to invoke during long-term assistant interaction. At turn tt, the system observes the current query qtq_t, previous interaction history H<tH_{<t}, and a persistent memory store Bt−1B_{t-1} containing active-task records, recent-turn summaries, and longer-term memories from earlier sessions. Let A\mathcal{A} be the candidate model pool. Before generating a response, the system builds a candidate memory view c~t\tilde{c}_t, chooses a model at∈Aa_t \in\mathcal{A}, and packs the final context ct(at)c_t(a_t) for that model.

This formulation makes explicit how simpler systems restrict the decision space. A fixed configuration uses both a fixed memory policy c~t=πmem0(qt,H<t,Bt−1)\tilde{c}_t = \pi_{\mathrm{mem}}^0(q_t, H_{<t}, B_{t-1}) and a fixed generator a0a^0. A routing-only system chooses ata_t under a memory policy that does not share the router’s session-state interface, while a memory-only system adapts c~t\tilde{c}_t under a fixed generator. These abstractions are incomplete for session-level interaction because memory construction and ata_t interact: the constructed context changes the input difficulty and evidence structure, while the selected model determines how reliably that context is interpreted.

Session-State Interface

MARS introduces an explicit state estimation step. A Query and Topic Manager (QTM) maps the current turn into a structured session state:

zt=fQTM(qt,H<t,Bt−1).(1)z_t = f_{\mathrm{QTM}}(q_t, H_{<t}, B_{t-1}). \tag*{(1)}

Rather than producing a free-form summary, QTM emits a constrained control state designed around the two downstream decisions: constructing the candidate memory view c~t\tilde{c}_t and selecting the model ata_t.

We organize ztz_t into three functional groups. First, session-continuity fields identify whether the turn continues the active thread, shifts topic, or enters a revision, verification, or synthesis phase. Second, memory-control fields indicate whether prior state is needed, which memory scope should be considered, whether recent history should be preferred, and how much evidence should be exposed. Third, routing-difficulty fields describe the expected burden placed on the generator, such as memory dependence, ambiguity, and interaction phase; the router later combines these state signals with pre-packing load statistics from the candidate memory view. These fields are control signals rather than topic labels: the memory policy uses them to determine what parts of Bt−1B_{t-1} enter c~t\tilde{c}_t, while the router uses them, together with the candidate-view statistics, to estimate which model capacity is appropriate.

The full QTM schema and allowed values are provided in Appendix § 8.§ 1, and the QTM prompt template is provided in Appendix § 8.§ 2. The corresponding memory-view and router-feature mappings are detailed in Appendices § 8.§ 3 and § 8.§ 4. The key design is to use ztz_t as the shared interface between memory construction and model routing, rather than treating retrieval as a preprocessing step and routing as a separate decision over the raw query. The QTM prompt, schema, guardrails, and mapping ψmem\psi_{\mathrm{mem}} are fixed before evaluation, while the routing policy updates from online feedback. The two decisions are coordinated through a shared structured session-state interface.

Coupled Memory Construction and Model Routing

Given the session state, MARS first derives memory-view controls:

mt=ψmem(zt),(2)m_t = \psi_{\mathrm{mem}}(z_t), \tag*{(2)}

where mtm_t summarizes the QTM-induced memory lens, retrieval scope, exposure budget, and recency preference. The memory policy first constructs a candidate memory view, before model-specific packing:

c~t=πmem(mt,qt,H<t,Bt−1).(3)\tilde{c}_t = \pi_{\mathrm{mem}}(m_t, q_t, H_{<t}, B_{t-1}). \tag*{(3)}

In our implementation, the candidate view is assembled from three ordered memory blocks:

c~t=Select⁡mt(stactive,htrecent,rthist),(4)\tilde{c}_t = \operatorname{Select}_{m_t}\left(s_t^{\mathrm{active}}, h_t^{\mathrm{recent}}, r_t^{\mathrm{hist}}\right), \tag*{(4)}

where stactives_t^{\mathrm{active}} stores the active thread or artifact state, htrecenth_t^{\mathrm{recent}} contains recent turns, and rthistr_t^{\mathrm{hist}} contains selected historical evidence. The operator Select⁡mt\operatorname{Select}_{m_t} denotes candidate block selection and ordering under the controls in mtm_t, before model-specific prompt-budget packing. For example, QTM-derived memory need and lens fields decide whether memory is included in the candidate view, retrieval scope selects local active-thread evidence or broader historical memory, top-kk hints set the retrieval budget, recency preference biases ranking, and action/phase fields preserve active state for continuation, revision, or verification turns. Thus, MARS separates access to memory from exposure of memory: a system may store a large history, but the generator receives the selected evidence after packing for its input budget. Appendix § 8.§ 3 gives the concrete memory-view policy and prompt assembly.

The router uses the candidate view to select a generator; final packing follows that selection:

xt=ϕ(zt,c~t,qt),(5)x_t = \phi(z_t,\tilde{c}_t,q_t), \tag*{(5)}
at=πt(xt),at∈A,(6)a_t = \pi_t(x_t),\quad a_t \in\mathcal{A}, \tag*{(6)}
ct(at)=Pack⁡at(c~t,qt),(7)c_t(a_t) = \operatorname{Pack}_{a_t}(\tilde{c}_t,q_t), \tag*{(7)}
yt=gat(qt,ct(at)).(8)y_t = g_{a_t}(q_t,c_t(a_t)). \tag*{(8)}

Here xtx_t is the routing feature vector, πt\pi_t is the online routing policy, and Pack⁡at\operatorname{Pack}_{a_t} clips and assembles the context under the selected model’s input budget. The router features describe the pre-packing candidate view, including estimated memory-token load, similarity statistics, and QTM fields; they do not describe the final window-clipped prompt. If generation falls back to another model, the candidate view is repacked for that model. Appendix § 8.§ 3 details budget estimation and prompt assembly. The router does not memorize raw topic identifiers; instead, it learns state-level patterns such as when memory-heavy revision, temporal recall, or final synthesis requires a stronger operating point. Appendix § 8.§ 4 gives the full feature mapping.

We instantiate πt\pi_t with a LinUCB-style contextual bandit (Abbasi-Yadkori et al., [1]). For each model, Aa=γIA_a = \gamma I, ba=0b_a = 0, and θ^a=Aa−1ba\hat{\theta}_a = A_a^{-1}b_a initially; selection is

at=arg⁡max⁡a∈A(θ^aTxt+αxtTAa−1xt),(9)a_t = \underset{a\in\mathcal{A}}{\arg\max}\left(\hat{\theta}_a^{\mathsf{T}}x_t+\alpha\sqrt{x_t^{\mathsf{T}}A_a^{-1}x_t}\right), \tag*{(9)}

where AaA_a tracks uncertainty, γ>0\gamma> 0 is the ridge parameter, and α\alpha controls exploration. LinUCB is a design choice rather than the central contribution. The central contribution is the decision interface: the router learns over the same structured state that controls memory exposure.

Turn-Level Feedback

After the selected model produces yty_t, MARS estimates a local feedback signal for the routing decision. Following cost-aware LLM routing and online multi-LLM selection ([4, 23]), we convert turn-level quality into a cost-aware scalar utility:

r^t=Jturn(yt,qt,zt,ct(at))−λCtgen(at,ct(at)),(10)\hat{r}_t = J_{\mathrm{turn}}(y_t,q_t,z_t,c_t(a_t))-\lambda C_t^{\mathrm{gen}}(a_t,c_t(a_t)), \tag*{(10)}

where Ctgen(at,ct(at))C_t^{\mathrm{gen}}(a_t,c_t(a_t)) denotes the selected generator’s inference cost, including model-specific pricing and token usage of the final model-specific prompt, and λ\lambda controls the quality–cost trade-off. The quality term JturnJ_{\mathrm{turn}} is computed by a rubric-based turn judge rather than by a single unstructured score. The rubric checks whether the response answers the current query, uses the exposed memory consistently, and avoids hard failures such as topic drift or missing required state. These diagnostics are mapped to a bounded quality score by a fixed weighting and penalty rule, before subtracting generation cost; the rubric, prompt structure, and reward form are provided in Appendix § 8.§ 5.

Cost affects model choice through the observed reward rather than through direct price subtraction at decision time. Expensive model calls are penalized after generation, so future routing decisions gradually favor models that provide better quality–cost utility under similar session states. Only the selected arm is updated:

Aat←Aat+xtxtT,bat←bat+r^txt,(11)A_{a_t} \leftarrow A_{a_t}+x_tx_t^{\mathsf{T}},\quad b_{a_t}\leftarrow b_{a_t}+\hat{r}_t x_t, \tag*{(11)}

then θ^at=Aat−1bat\hat{\theta}_{a_t}=A_{a_t}^{-1}b_{a_t} is recomputed. The reward is computed only from information available at turn tt and does not use future turns, final labels, or benchmark-level pass decisions.

Turn-Level Workflow

Algorithm 1 summarizes the turn-level workflow of MARS. We write Lt=(qt,H<t,Bt−1)\mathcal{L}_t=(q_t,H_{<t},B_{t-1}) for the turn input. At each turn, QTM first estimates a structured session state. The memory policy converts this state Algorithm 1 MARS turn-level workflow

Require: turn input It=(qt,H<t,Bt−1)\mathcal{I}_t = (q_t, H_{<t}, B_{t-1}), model pool A\mathcal{A}, router policy πt\pi_t

  1. STATE: zt←fQTM(It)z_t \leftarrow f_{\mathrm{QTM}}(\mathcal{I}_t)

  2. MEMORY CONTROLS: mt←ψmem(zt)m_t \leftarrow\psi_{\mathrm{mem}}(z_t)

  3. CANDIDATE VIEW: c~t←πmem(mt,It)\tilde{c}_t \leftarrow\pi_{\mathrm{mem}}(m_t, \mathcal{I}_t)

  4. FEATURES: xt←ϕ(zt,c~t,qt)x_t \leftarrow\phi(z_t, \tilde{c}_t, q_t)

  5. ROUTE: at←πt(xt)a_t \leftarrow\pi_t(x_t)

  6. PACK: ct(at)←Pack⁡at(c~t,qt)c_t(a_t) \leftarrow\operatorname{Pack}_{a_t}(\tilde{c}_t, q_t)

  7. GENERATE: yt←gat(qt,ct(at))y_t \leftarrow g_{a_t}(q_t, c_t(a_t))

  8. UTILITY: u^t←Jturn(yt,qt,zt,ct(at))\hat{u}_t \leftarrow J_{\mathrm{turn}}(y_t, q_t, z_t, c_t(a_t))

  9. REWARD: r^t←u^t−λCgen(at,ct(at))\hat{r}_t \leftarrow\hat{u}_t - \lambda C_{\mathrm{gen}}(a_t, c_t(a_t))

  10. ROUTER UPDATE: πt+1←Update⁡(πt,xt,at,r^t)\pi_{t+1} \leftarrow\operatorname{Update}(\pi_t, x_t, a_t, \hat{r}_t)

  11. MEMORY UPDATE: (H≤t,Bt)←MemoryUpdate⁡(It,yt,zt)(H_{\leq t}, B_t) \leftarrow\operatorname{MemoryUpdate}(\mathcal{I}_t, y_t, z_t)

into memory-view controls and constructs the candidate view. The router selects a generator using the shared state and pre-packing candidate statistics; the view is then packed for that model before generation. Turn-level feedback updates the routing policy. The underlying LLMs are not retrained.

Experimental Setup

Our controlled experiments evaluate MARS across two complementary regimes. REALMEM tests long-history memory-driven interaction; MULTICHALLENGE tests bounded structured sessions with interference and path-dependent evaluation. The same 16-system matrix is used on both, enabling consistent comparison across the two regimes.

Benchmarks

REALMEM. REALMEM is an external benchmark for real-world memory-driven interaction [3]. We evaluate its 10-persona evaluation slice: 10 personas and 1,415 queries. We report normalized QA, Mem Recall, Recall@20 (R@20), normalized discounted cumulative gain at 20 (NDCG@20), and API cost. Main quality and memory metrics are macro-averaged over personas; query-weighted aggregates and scored coverage are reported in the appendix.

MULTICHALLENGE. MULTICHALLENGE evaluates state maintenance in bounded but structured multi-turn conversations [25]. Our converted split contains 245 scenarios and 1,480 turns. Each scenario contains intermediate turns followed by a verification turn that uses the official binary rubric question as the final user query. The expected answer is YES when the supplied pass criterion is satisfied, testing whether the system preserved the required topic state and constraints (Appendix § 9.§ 2).

The four axes are INFERENCE MEMORY (102 scenarios), INSTRUCTION RETENTION (62), SELF COHERENCE (46), and RELIABLE VERSION EDITING (35). We report the full aggregate first and analyze the first and fourth axes (137 scenarios) separately because they directly test state recovery and local revision. Chunk size 3 bundles trajectories into context blocks with distractor sessions (Appendix § 9.§ 2). For the MARS run, we use λcost=0.002\lambda_{\mathrm{cost}} = 0.002, matching the main REALMEM routing objective.

Baselines

For both benchmarks, we evaluate 16 systems formed from memory settings and model/router choices. Full-context baselines use one fixed generator for all turns: GPT-4o ([19]), Qwen2.5 72B ([31]), Mixtral 8×7B Instruct ([9]), or Llama3B ([14]). Memory baselines pair MEM0 ([5]) or A-MEM ([30]) with each fixed generator. Routing baselines include a METALLM-style router ([16]) and hybrids that combine MEM0 or A-MEM with METALLM-style routing. MARS is the full system, using QTM-guided memory construction and online model routing. We use MEM0 and A-MEM as representative plug-in memory substrates because they can be consistently crossed with fixed generators and routers across both benchmarks.

QTM uses Qwen2.5-7B-Instruct ([31]) with a fixed prompt, schema, guardrails, and state-to-memory mapping. A GPT-4o-mini ([18]) turn judge supplies local reward after generation; it never sees future turns, final labels, gold final answers, or benchmark pass decisions (Appendix § 8.§ 5).

Ablation Protocol

We run diagnostic ablations on a fixed three-persona REALMEM evaluation slice containing 473 turns. All variants use the same candidate model pool, deterministic generation, and a matched top-10 historical retrieval budget. The slice uses λcost=0.5\lambda_{\mathrm{cost}} = 0.5, compared with λcost=0.002\lambda_{\mathrm{cost}} = 0.002 in the main REALMEM run; its retrieval budget also differs from the main policy’s top-kk hints. Comparisons are therefore made within this diagnostic configuration. The ablations examine QTM-guided context construction versus raw-query semantic retrieval, QTM state/header exposure in the generator prompt, QTM-derived routing versus query-embedding routing with semantic or recency-based memory, and online adaptation via a no-update diagnostic. The no-update variant freezes the router at initialization, so we treat it as a stress test of online adaptation rather than as a tuned static-router baseline. Ablation metrics follow the official REALMEM evaluation pipeline, with QA normalized to the same scale as the main results. Full settings and diagnostics are provided in Appendix § 11.

Evaluation, Cost, and Reporting

For REALMEM, we use the official retrieval and generation evaluation pipeline, with GPT-4o as the final judge when judging is required. This final evaluator is separate from the GPT-4o-mini turn-level judge used for online routing updates. For MULTICHALLENGE, we use a GPT-4o path-aware final judge following the benchmark style: it checks trajectory continuity and final-answer correctness. A scenario passes only when both checks are accepted and the model’s parseable final YES/NO answer matches the expected criterion (Appendix § 9.§ 4).

For REALMEM, main tables report generator and QTM/controller API cost, excluding online feedback; Results separately reports reconstructed feedback-inclusive totals. For MULTICHALLENGE, the archived runs did not itemize QTM/controller cost separately, so tables retain the original run-level recorded-cost convention. Final benchmark evaluators and local memory compute and storage are excluded from these API cost views. The online reward uses only selected-generator cost (Appendix § 8.§ 6). We bootstrap REALMEM by persona and MULTICHALLENGE by scenario, and separately test eight chunk orders on 30 fixed MULTICHALLENGE scenarios; Appendix § 8.7 gives these diagnostics.

Results

We evaluate the quality–memory–cost trade-off across long-history interaction on REALMEM and bounded session interference on MULTICHALLENGE.

RealMem: QA–Cost Trade-off and Memory Metrics

Table 1 and Figure 2 report persona-macro quality and generation-controller API cost over 10 personas. Appendix § 10.§ 1 gives all 16 systems.

Memory SettingModel / RouterQA ↑\uparrowMem Rec. ↑\uparrowR@20 ↑\uparrowNDCG@20 ↑\uparrowGen.-Ctrl. ($) ↓\downarrow
MARS0.74060.67940.80150.616826.6480
Full contextGPT-4o0.75400.16400.51880.3216184.3967
Full contextMETALLM0.73310.16030.51880.3216180.0189
MEM0GPT-4o0.69090.59970.63050.538841.3834
MEM0METALLM0.66520.60480.63050.538824.3499
A-MEMGPT-4o0.71720.64790.68170.584643.1311
A-MEMMETALLM0.70440.64710.68170.584620.2875

Table 1. REALMEM key systems (10 personas; 1,415-query pool). Metrics are persona-macro averages. Cost records generation and controller API calls; online feedback and final evaluation are excluded. Overflow-dominated full-context rows remain in Appendix § 10.§ 1.

Figure 2. REALMEM persona-macro QA versus generation-controller API cost for key systems. Online feedback and final evaluation are excluded.

Full-context GPT-4o obtains 0.7540 QA at $184.40. MARS reaches 0.7406 (98.2%) at $26.65 (14.5%) under the generation-controller view and has the best memory-recall and retrieval scores. A-MEM+MetaLLM is cheaper ($20.29) but has lower QA (0.7044) and memory metrics, giving a different operating point. Adding the estimated GPT-4o-mini feedback cost gives MARS a feedback-inclusive API total of approximately $32.53 (17.6% of full-context GPT-4o), compared with approximately $23.49 for A-MEM+MetaLLM under the same accounting. These estimates use logged input-token counts and estimated output usage; final benchmark evaluation and local compute are excluded (Appendix § 8.§ 6).

The persona-bootstrap 95% interval for QA is [0.7183, 0.7641]; other intervals are in Appendix § 8.7. Archived provider errors show context-limit failures on 1,344/1,415 queries for both Qwen2.5-72B and Mixtral, and 1,247/1,415 for Llama3B; affected rows receive zero score and zero recorded cost. Their low QA therefore reflects the evaluated full-history configuration’s context limits as well as response quality.

MultiChallenge: Full Aggregate and Mechanism-Focused Subset

Table 2 reports the full aggregate first and the mechanism-focused subset second; complete matrices are in Appendices § 10.§ 2–§ 10.§ 3.

SystemFull pass ↑\uparrowSubset pass ↑\uparrowRecorded subset cost ($) ↓\downarrow
MARS58/24544/1371.5670
Full Qwen2.5 72B67/24545/1371.8528
Full GPT-4o64/24536/1379.1044
MEM0+Mixtral57/24539/1370.4620

Table 2. MultiChallenge full aggregate and mechanism-focused subset. Costs follow the original run-level accounting convention; QTM/controller cost was not separately itemized in the archived logs.

On full MultiChallenge, Qwen2.5 72B (67/245) and GPT-4o (64/245) outperform MARS (58/245). Qwen’s full-set recorded cost is also slightly lower ($3.1665 versus $3.2504). On the focused subset, MARS reaches 44/137 versus 45/137 for Qwen at lower recorded cost. Per-axis, MARS remains close on Inference Memory (33/102 versus 36/102) and is higher on Reliable Version Editing (11/35 versus 9/35), while larger gaps remain on the other two axes (Appendix § 10.§ 4). This supports using the subset as a mechanism-focused view of state recovery and local revision rather than a replacement for the full aggregate. MARS’s full-set pass rate is 0.2367 with a scenario-bootstrap 95% interval of [0.1837, 0.2898]. Across eight randomized chunk orders of the same fixed 30-scenario subset, with a freshly initialized bandit for each replay, the diagnostic pass rate averages 18.3% with a sample standard deviation of 3.6 percentage points (4/30–7/30); Appendix § 8.7 gives the construction and per-order results.

RealMem Diagnostic Ablations

We run diagnostic ablations on a fixed three-persona RealMem evaluation slice (473 turns), with matched top-10 retrieval and the same model pool.

VariantQA ↑\uparrowMem Rec. ↑\uparrowNDCG@10 ↑\uparrowGen.-Ctrl. ($) ↓\downarrow
Full MARS0.74440.68040.59745.643
Embed. semantic0.72310.64290.51625.911
No state/header0.70990.63970.51626.338
Recent-only0.62790.10700.22995.638
No-QTM semantic0.73870.62870.51624.570
No-QTM recent0.62440.11180.22994.918
No update†^\dagger0.42210.68820.59830.439

Table 3. RealMem diagnostic ablations (473 turns; λcost=0.5\lambda_{\mathrm{cost}} = 0.5). Bold marks the best online variant. †^\daggerNo-update freezes the untuned cold-start router. Full metrics: Appendix § 11.

Full MARS has the highest QA among online variants. Under the matched top-10 budget, embedding-only semantic retrieval lowers QA from 0.7444 to 0.7231 and Mem Recall from 0.6804 to 0.6429, while recent-only memory reduces Mem Recall to 0.1070. Conversely, the no-update row retains strong retrieval metrics but drops QA to 0.4221. These diagnostics support complementary roles for state-guided memory construction and online adaptation.

QTM Prompt and Controller Sensitivity

We probe local sensitivity of the QTM interface under two types of controlled variation on the same 150 frozen session snapshots from seven RealMem evaluation personas. With Qwen2.5-7B-Instruct fixed, we compare the original prompt with a semantics-preserving paraphrase and an instruction-reordered variant. We then keep the original prompt and schema fixed and replace the QTM controller with Mistral-7B-Instruct-v0.3 ([15]). Across all comparisons, the session history, memory pool, retrieval policy, and deterministic decoding are held fixed. We measure agreement of the effective MATCH/NEW_TOPIC decision and Jaccard overlap of the pre-packing retrieved memory-source IDs.

Controlled variationAction agr. ↑Retrieval Jaccard ↑
Prompt paraphrase90.0%0.920
Instruction reordering96.0%0.973
Mistral-7B controller82.7%0.875

Table 4. QTM interface-sensitivity probe over 150 fixed session snapshots. Prompt variants use the default Qwen2.5-7B-Instruct controller; the controller-swap row uses the original prompt/schema with Mistral-7B-Instruct-v0.3. Each row is compared with the Qwen2.5-7B-Instruct original-prompt reference.

The prompt perturbations preserve most effective thread-boundary decisions and retrieved memory sources; on the 60 inputs with active thread history—where a NEW_TOPIC decision can trigger archival—agreement rises to 95.0%/98.3% and retrieval Jaccard to 0.984/1.000 for paraphrasing/reordering, respectively. Under controller substitution, the effective action agrees on 124/150 inputs (82.7%) with retrieval Jaccard 0.875; on the same 60 active-history inputs, agreement is 53/60 (88.3%) and retrieval Jaccard is 0.960. Together, these probes provide complementary evidence that the key thread-boundary and retrieval decisions remain largely preserved under the tested prompt changes and one same-scale controller substitution. Appendices E.5–E.6 provide detailed diagnostics.

Cross-Regime Analysis

Static Qwen2.5 72B leads on full MULTICHALLENGE, while full-context Qwen2.5-72B and Mixtral exceed their context limits on 1,344/1,415 REALMEM queries. This contrast suggests a greater role for selective memory exposure under long-history pressure; strong static models remain effective in bounded contexts.

Conclusion

We introduced MARS, which coordinates a fixed state-conditioned memory policy and an online model router through a shared session-state interface. It achieves a strong quality–memory–cost trade-off on REALMEM and remains competitive on the mechanism-focused MULTICHALLENGE subset, while trailing the two strongest static systems on the full aggregate.

Appendix Overview

This appendix provides implementation details, benchmark construction, full result tables, and diagnostic analyses for MARS. Appendix § 8 specifies the QTM state interface, memory-view construction, router features, turn-level reward, and cost accounting. Appendix § 9 describes the REALMEM and MULTICHALLENGE protocols, including metric normalization and the chunked-context construction used for MULTICHALLENGE. Appendix § 10 reports full system matrices. Appendix § 11 reports the three-persona REALMEM component ablations and cost-weight sensitivity, together with QTM prompt- and controller-sensitivity probes. Appendix § 12 provides anonymized qualitative examples and routing-behavior diagnostics from fixed evaluation outputs. Appendix § 13 discusses deployment feedback and reproducibility information.

Implementation Details

The main text formulates MARS as coordinating memory exposure and model routing through shared state. Its QTM prompt, schema, guardrails, and state-to-memory mapping are fixed; only the router updates online.

Optimization scope. In the evaluated implementation, the QTM schema, guardrails, and state-to-memory policy are fixed before evaluation, while the LinUCB router adapts online. Coordination is realized through the shared session state and the turn-dependent memory view induced by that state.

QTM Schema and State Representation

The Query and Topic Manager (QTM) maps the current turn into a structured session state. In our implementation, QTM is instantiated with qwen/qwen-2.5-7b-instruct. We use this smaller instruction model because QTM does not generate final answers; it produces a compact control state used by both the memory policy and the online router.

The schema describes task state rather than topic identity. Omitting raw topic labels directs the router toward state-level patterns, such as the model utility associated with memory-heavy revision, temporal recall, or final synthesis. Table 5 summarizes the fields used by the current implementation.

FieldAllowed values / representationMemoryRouterRole in the decision interface
actionMATCH, NEW_TOPICyesyesSegments the turn as continuation of the active thread or a topic switch.
query_typeSTATIC, DYNAMIC, TEMPORAL, PROACTIVEyesyesDistinguishes self-contained, evolving, temporal, and proactive turns.
phaseEXPLORE, CONSTRAIN, REVISE, SYNTHESIZE, VERIFYyesyesEncodes interaction phase for memory exposure and model selection.
taxonomy.*need_memory, memory_span, ambiguity, and risk metadatayesyesRecords decision-relevant memory and ambiguity state without raw topic identity.
retrieval_scopeLOCAL, GLOBALyesyesSelects active-thread evidence or broader historical retrieval.
top_k_hint3, 5, 8yesyesGives the memory constructor a retrieval-budget hint.
prefer_recent0, 1yesyesBiases memory construction toward recent history when relevant.
lens_recommendation0 or 3yesyesControls minimal or full memory exposure.
lens_confidence, lens_reasonconfidence and short reasonyesoptionalProvides a traceable rationale for the selected memory lens.
qtm_llm_called, qtm_usagelogged diagnosticsnonoUsed for cost tracing and QTM-overhead accounting.

Table 5. QTM state schema used in the REALMEM implementation. The main-text abstract fields map to this concrete schema: for example, memory need is represented primarily through taxonomy.need_memory; memory view is represented through lens_recommendation, retrieval_scope, and top_k_hint.

QTM structured output shape used in REALMEM mode.

Inputs: ACTIVE CONTEXT, USER INPUT, and SYSTEM RULES. The QTM emits controller state only and does not generate the final answer.

Required JSON fields:

action: MATCH or NEW_TOPIC
query_type: STATIC, DYNAMIC, TEMPORAL, or PROACTIVE
phase: EXPLORE, CONSTRAIN, REVISE, SYNTHESIZE, or VERIFY
taxonomy: memory and risk metadata, including need_memory, memory_span, and ambiguity
lens_recommendation: 0 or 3
top_k_hint: 3, 5, or 8
prefer_recent: 0 or 1
retrieval_scope: LOCAL or GLOBAL
lens_confidence: confidence score for the selected lens
lens_reason: concise reason for the selected lens

Anonymized example output:
{
  "action": "MATCH",
  "query_type": "DYNAMIC",
  "phase": "REVISE",
  "taxonomy": {
    "need_memory": 1,
    "memory_span": "ACTIVE_THREAD",
    "ambiguity": "LOW"
  },
  "lens_recommendation": 3,
  "top_k_hint": 5,
  "prefer_recent": 1,
  "retrieval_scope": "LOCAL",
  "lens_confidence": 0.82,
  "lens_reason": "Revision of the active artifact."
}

Implementation note: invalid fields are clamped during post-processing, and revision-like continuation turns are conservatively mapped to MATCH with memory enabled.
QTM structured output contract used in the REALMEM-mode implementation

Figure 3. QTM structured output contract used by the REALMEM-mode implementation, with an anonymized example output.

QTM Prompt Template

The QTM prompt asks the controller to infer decision-relevant session state without generating a final answer. In REALMEM mode, the prompt has three input blocks: PREVIOUS ACTIVE CONTEXT, NEW USER INPUT, and SYSTEM RULES. It requires strict JSON output matching Table 5. Controller outputs are post-processed using the fixed schema and conservative continuation rules. Invalid fields are clamped to allowed values, and parsing failures use conservative fallback state choices, including continuation with memory enabled for revision-like turns.

Compact QTM prompt excerpt used in the implementation

Figure 4. Compact QTM prompt excerpt used by the implementation.

Memory Construction and Prompt Assembly

Given the QTM state, MARS first derives memory-view controls mt=ψmem(zt)m_t = \psi_{\mathrm{mem}}(z_t) and then constructs a pre-packing candidate view from active-thread state, recent turns, and retrieved historical memory:

c~t=Select⁡mt(stactive,htrecent,rthist).(12)\widetilde{c}_t = \operatorname{Select}_{m_t}\left(s_t^{\mathrm{active}}, h_t^{\mathrm{recent}}, r_t^{\mathrm{hist}}\right). \tag*{(12)}

Here mtm_t controls which blocks are included, how much historical evidence is retrieved, and their ordering in the candidate view. Model-specific packing is performed after routing. This implements the distinction between access and exposure: a large memory store may be available, but only the decision-relevant memory view should be placed in the generator prompt.

After the bandit selects ata_t, the candidate view c~t\widetilde{c}_t is packed into ct(at)c_t(a_t) using that model’s context window, output reservation, and safety margin. The final generator prompt has four ordered blocks: SYSTEM RULES, MEMORY GUIDANCE, MEMORY, and USER QUERY. The guidance contains the QTM-derived state header but omits internal controller metadata. The total input budget is

Bin(at)=Wctx(at)−Rout(at)−Msafe(at),(13)B_{\mathrm{in}}(a_t) = W_{\mathrm{ctx}}(a_t) - R_{\mathrm{out}}(a_t) - M_{\mathrm{safe}}(a_t), \tag*{(13)}

where WctxW_{\mathrm{ctx}} is the model context window, RoutR_{\mathrm{out}} the reserved output budget, and MsafeM_{\mathrm{safe}} a safety margin. The memory blocks are clipped to fit the input budget together with the system rules, guidance, and query. Token lengths are estimated using the release implementation’s model-specific budget rules rather than computed exactly with each model’s tokenizer. A safety fallback to another model triggers fresh packing of the candidate view under the fallback model’s budget. Both benchmark runners follow this candidate-view, routing, and model-specific packing sequence.

QTM conditionActive stateRecent turnsHistorical retrievalIntended behavior
taxonomy.need_memory=0 or lens_recommendation=0omitted or minimaloptional short windownoneAvoid unnecessary context expansion for self-contained turns.
action=MATCH, retrieval_scope=LOCALincludedincludednone or active-thread artifactsPreserve the active thread for continuation, local edit, or verification turns.
phase=REVISE or phase=VERIFYincludedincludedoptional local evidenceSupport revision and verification without drifting from the current artifact.
retrieval_scope=GLOBALoptional active summaryshort recency windowranked global candidatesRecover cross-session evidence for temporal or resumed-topic queries.
prefer_recent=1included if activerecency-biasedrecent candidates prioritizedBias memory construction toward recent history when recency is decision-relevant.
lens_recommendation=3 or top_k_hint=8includedincludedexpanded ranked evidenceProvide a richer memory view for synthesis, finalization, or high-difficulty turns.

Table 6. Memory-view construction policy. QTM emits top-kk hints in {3,5,8}\{3,5,8\} for the main MARS policy. The three-persona ablation slice uses a matched top-10 historical retrieval budget across the main memory-construction variants to control exposure budget.

Generator prompt assembly used by MARS.

Figure 5. Generator prompt assembly used by MARS.

Online Router Features and Update

The online router receives a feature vector

xt=ϕ(zt,c~t,qt),(14)x_t = \phi(z_t,\tilde{c}_t,q_t), \tag*{(14)}

where ϕ\phi encodes QTM fields, memory-view indicators, retrieval-scope signals, interaction phase, turn depth, and pre-packing candidate-load statistics. Features are extracted before selecting the generator and do not use the final model-specific packed context. We do not include raw topic identity as a categorical feature. We instantiate the online router with a LinUCB-style contextual bandit:

at=arg⁡max⁡a∈A(θ^aTxt+αxtTAa−1xt).(15)a_t = \underset{a\in\mathcal{A}}{\arg\max}\left(\hat{\theta}_a^{\mathsf{T}}x_t+\alpha\sqrt{x_t^{\mathsf{T}}A_a^{-1}x_t}\right). \tag*{(15)}

The router is updated using the turn-level reward:

πt+1=Update⁡(πt,xt,at,r^t).(16)\pi_{t+1}=\operatorname{Update}(\pi_t,x_t,a_t,\hat{r}_t). \tag*{(16)}

LinUCB is used because it supports lightweight online updates over structured features. Other contextual bandit policies could be substituted without changing the MARS interface.

Feature groupExamplesEncodingPurpose
QTM categorical statetopic relation, query type, memory need, retrieval scope, memory view, interaction phaseone-hot / categoricalMain session-state signal for model selection.
Memory-load featurescandidate memory blocks, retrieved-history flag, estimated candidate-token countnormalized numericEstimates the load of the pre-packing candidate view.
Session-position featuresturn depth, recent revision flag, finalization flagnumeric / binaryDistinguishes early continuation from later synthesis or verification.
Retrieval featuresretrieval scope, number of candidates, retrieval score summariesnumeric / categoricalIndicates whether the response depends on recovered evidence.
Cost featuresmodel-specific cost enters through reward rather than direct decision-time subtractionreward-side scalarLets the router learn quality–cost utility from observed outcomes.

Table 7. Router feature groups. The implementation uses structured state and pre-packing candidate statistics rather than raw topic identifiers.

Hyperparameter / settingFinal value
Candidate model pool A\mathcal{A}Main REALMEM runs: Llama-3.2-3B, Mixtral-8x7B, Qwen-72B, GPT-4o; three-persona ablation pool: Llama3B, Qwen2.5-72B, GPT-4o.
Feature dimension30
LinUCB exploration coefficient α\alpha0.8
Regularization / initializationper-arm Aa=λregIA_a = \lambda_{\mathrm{reg}} I with λreg=1.0\lambda_{\mathrm{reg}} = 1.0 and ba=0b_a = 0
Feature normalizationcategorical one-hot plus normalized numeric features, including retrieval-budget, similarity, retrieval-count, and binary state flags
Warm-start policycold-start online policy for MARS; the no-update ablation freezes this cold-start policy without online updates
Reward cost weightλcost=0.002\lambda_{\mathrm{cost}} = 0.002 in the main REALMEM full-slice router run; λcost=0.5\lambda_{\mathrm{cost}} = 0.5 in the three-persona REALMEM ablation

Table 8. Router hyperparameters and settings.

The main REALMEM run and the three-persona ablation use different cost weights, candidate pools, and retrieval budgets. They are separate experimental configurations rather than a single hyperparameter sweep. All component-ablation variants use λcost=0.5\lambda_{\mathrm{cost}} = 0.5, so their comparisons remain within the diagnostic configuration.

Turn-Level Judge and Reward

During online adaptation, MARS uses a turn-level judge to estimate local reward for the selected model response. In our experiments, this online feedback judge is implemented with GPT-4o-mini and is used only to update the routing policy; it is not used for benchmark reporting.

The turn-level judge receives the current query, final packed memory context ct(at)c_t(a_t), QTM state, generated answer, and non-final current-turn diagnostics when available. It does not receive future turns, final labels, gold final answers, or benchmark-level pass decisions. This separation prevents online updates from using information that would be unavailable at decision time.

Reward aggregation. The six rubric scores in Table 9 are normalized by dividing by 5. Benchmark-specific mappings combine these scores with coverage signals and failure indicators into three components: memory use MtM_t, answer correctness AtA_t, and answer quality QtQ_t.

Judge outputDefinitionRangeScoring focus
answer_helpfulnessHelpfulness and directness of the candidate answer for the current query.0–5helpfulness
state_alignmentConsistency with active topic, session state, constraints, and memory context.0–5fidelity diagnostic
revision_applicationWhether revision instructions are applied without restarting, overwriting, or drifting.0–5revision diagnostic
intermediate_productQuality of intermediate artifacts or state updates when the turn is not final.0–5intermediate quality
routing_adequacyWhether the selected model is adequate for the state and context.0–5routing diagnostic
final_integrationWhether the response integrates prior state and evidence into the final answer.0–5fidelity diagnostic
Hard-failure flagsTopic drift, restart instead of revision, hallucinated context use, or severe missing-history use.0/1penalty

Table 9. Turn-level judge outputs. Normalized rubric scores, coverage signals, and failure indicators form the three composite components in Eq. (17), weighted by 0.45, 0.40, and 0.15. A benchmark-specific failure penalty is applied before clipping; generation cost is then subtracted.

Both benchmarks use

u^t=clip⁡(0.45Mt+0.40At+0.15Qt−Pt, 0, 1),(17)\begin{aligned} \hat{u}_t &= \operatorname{clip}(0.45M_t + 0.40A_t \\ &\quad+ 0.15Q_t - P_t,\,0,\,1), \tag*{(17)} \end{aligned}
r^t=u^t−λCtgen(at,ct(at)).\hat{r}_t = \hat{u}_t - \lambda C_t^{\mathrm{gen}}(a_t,c_t(a_t)).

Here u^t=Jturn(yt,qt,zt,ct(at))\hat{u}_t = J_{\mathrm{turn}}(y_t,q_t,z_t,c_t(a_t)) is the pre-cost utility defined in the main text. The weights 0.45, 0.40, and 0.15 apply to the three composite components, not directly to the six raw rubric scores.

For REALMEM, MtM_t averages retrieval coverage, normalized state alignment, and normalized routing adequacy. The answer component AtA_t averages normalized helpfulness, normalized state alignment, and one minus the hallucination flag. The quality component QtQ_t averages normalized intermediate-product and final-integration scores, using the intermediate-product score alone on non-final steps. For MULTICHALLENGE, the memory component additionally includes normalized revision application; coverage is set to 1 when explicit memory supervision is unavailable.

Failure penalties. With binary flags ordered as topic drift, restart, hallucination, and severe missing history, PtP_t is their weighted sum with coefficients (0.10, 0.06, 0.12, 0.12) for REALMEM and (0.10, 0.12, 0.10, 0.10) for MULTICHALLENGE. The routing reward subtracts only the selected generator’s cost; QTM charges remain part of the reported API cost. The released implementation will include the prompts, benchmark-specific component mappings, and reward-processing configurations used in the experiments.

Compact turn-level judge prompt excerpt

Figure 6. Compact turn-level judge prompt excerpt used for online feedback.

Cost Accounting

For REALMEM, the main-table API cost is

Cgen+ctrl=∑t(Ctgen+Ctctrl).(18)C_{\mathrm{gen+ctrl}} = \sum_t \left(C_t^{\mathrm{gen}} + C_t^{\mathrm{ctrl}}\right). \tag*{(18)}

Online feedback and final benchmark evaluators are excluded from this main-table view. Local memory operations have no external API charge in our setup, but their compute, storage, latency, and engineering costs are unpriced. The online reward subtracts only generator cost because the router chooses the generator. On REALMEM, generation-controller API costs are $26.6480 for MARS, $184.3967 for full-context GPT-4o, and $20.2875 for A-MEM+MetaLLM. The MARS/GPT-4o cost ratio is 14.5%.

The feedback-inclusive totals in Section 5.1 are estimated from logged online-judge input-token counts and conservatively estimated output usage, priced at $0.15 and $0.60 per million tokens, respectively. This adds approximately $5.88 for MARS and $3.20 for A-MEM+MetaLLM, giving approximately $32.53 and $23.49. These estimates exclude final evaluators and local compute. For MULTICHALLENGE, costs follow the original run-level recording convention, without a separate QTM/controller breakdown.

ComponentRole / modelInput priceOutput price
QTM/controllerqwen/qwen-2.5-7b-instruct$0.04 / 1M tokens | $0.10 / 1M tokens
Turn-level judgeGPT-4o-mini online feedback judge$0.15 / 1M tokens | $0.60 / 1M tokens
Final REALMEM evaluatorGPT-4o final benchmark judge where requiredexcluded from both viewsexcluded
Final MULTICHALLENGE evaluatorGPT-4o path-aware final judgeexcluded from both viewsexcluded
Generatorscandidate pool A\mathcal{A}Llama3B: $0.05; Mixtral: $0.20; Qwen72B: $0.35; GPT-4o: $2.50 / 1M tokensLlama3B: $0.10; Mixtral: $0.20; Qwen72B: $0.40; GPT-4o: $10.00 / 1M tokens
Memory search/addMEM0/A-MEM where applicableno external embedding API; local compute unpricedsame

Table 10. Cost-accounting components and the unit prices used for the reported runs. Prices are per 1M tokens where applicable.

Uncertainty and Interaction-Order Diagnostics

Empirical bootstrap resampling over the 10 REALMEM personas gives 95% intervals of [0.7183, 0.7641] for QA, [0.6571, 0.7024] for Mem Recall, and [2.4004,2.4004,2.9394] for per-persona generation-controller cost. The corresponding persona-macro centers are 0.7406, 0.6794, and $2.6648, matching the aggregation used in the main table. As an aggregation check, direct query-weighted MARS values are 0.7404 QA over 1,411 scored queries and 0.6815 Mem Recall over 1,393 scored queries; full-context GPT-4o obtains 0.7529 and 0.1655, respectively, over 1,414 scored queries. The substantive comparison is unchanged. Scenario-level empirical bootstrap gives a full-MULTICHALLENGE pass rate of 0.2367 with a 95% interval of [0.1837, 0.2898] and a focused-subset rate of 0.3212 with a 95% interval of [0.2409, 0.4015], conditional on the evaluated order.

In REALMEM, each persona has an independently initialized router. Reordering personas therefore does not alter their router updates, while within-persona chronology defines the causal history and is preserved. The bootstrap intervals quantify sample variation across personas or scenarios, conditional on the evaluated histories and ordering.

For a separate interaction-order diagnostic, we replay the same fixed 30-scenario MULTICHALLENGE subset under eight randomized chunk orders while holding the evaluated cases and all non-order configurations fixed. The original scenario sequence is partitioned into ten chunks of three scenarios; each seed shuffles only the chunk order, with within-chunk order preserved. Each replay starts from a freshly initialized bandit in a separate process, and online updates are confined to that replay. We retain the main MULTICHALLENGE routing configuration, including chunk size 3, chunk-level memory scope, Qwen2.5-7B as QTM, λcost=0.002\lambda_{\mathrm{cost}} = 0.002, the same four-model router pool, and unchanged router features and decoding settings.

To keep the order comparison internally consistent, all eight replays are evaluated with the same fixed GPT-4o-mini path-aware evaluator. The primary MULTICHALLENGE benchmark results use the GPT-4o final evaluator described in Appendix § 9.§ 4.

SeedFinal passPass rate
135/3016.7%
294/3013.3%
477/3023.3%
616/3020.0%
797/3023.3%
975/3016.7%
1135/3016.7%
1315/3016.7%
Order mean5.5/3018.3%

Table 11. Interaction-order diagnostic on the same fixed set of 30 MULTICHALLENGE scenarios across eight randomized chunk orders. The sample standard deviation of the order-level pass rates is 3.6 percentage points.

Across the eight sampled orders, pass counts range from 4/30 to 7/30, with a mean pass rate of 18.3% and a sample standard deviation of 3.6 percentage points. This diagnostic characterizes variation across the tested processing orders rather than absolute benchmark performance.

Benchmark Details

RealMem Protocol and Metric Normalization

REALMEM evaluates long-term memory-driven interaction ([3]). We use the evaluation pool of 10 personas and 1,415 queries and report normalized QA, Mem Recall, Recall@20, NDCG@20, and API cost. Main-table quality and memory metrics are macro-averaged over the 10 personas; Section B.7 reports direct query-weighted aggregates and their scored coverage separately.

The original REALMEM QA judge score is reported on a 0–3 scale. Following the normalized metric used in the main text, we report

QAnorm=QAraw3.(19)\mathrm{QA}_{\mathrm{norm}} = \frac{\mathrm{QA}_{\mathrm{raw}}}{3}. \tag*{(19)}

All REALMEM QA values in Appendix § 10.§ 1 use this normalized score, so they align with Table 1 in the main text. Mem Recall is the REALMEM memory judge score. Recall@20 and NDCG@20 measure retrieval coverage and ranking quality.

Recall@20 and NDCG@20 are taken directly from the official retrieval evaluator, without redefining the metrics for full-context baselines. They assess the retrieval output scored by that pipeline; the full-context label describes the generator input and does not imply a retrieval score of one. Fixed generators paired with the same retrieval output therefore share retrieval scores even when their QA differs.

MultiChallenge Conversion and Axes

Of the 273 official MULTICHALLENGE scenarios, we reserved 28 (approximately 10%) for QTM prompt development and preliminary testing and excluded them from the reported evaluation split. The remaining 245 scenarios, comprising 1,480 turns, form the evaluation split; the split identifiers will be released with the implementation. We convert each scenario into a session-level memory-routing trajectory: intermediate turns provide the path over which systems must maintain state. The final verification turn uses the scenario’s official binary rubric question as the last user query. Under the supplied pass criterion, a trajectory satisfying the required memory and constraints supports the expected answer YES. This question originates from the official evaluation rubric; placing it in the assistant’s input trajectory is part of our conversion.

In the converted evaluation format, intermediate turns are marked as eval_type=0; the final official rubric question is marked as eval_type=1 with its supplied PASS_CRITERIA; exactly one eval_type=1 appears per scenario and it is always the final turn. The benchmark contains four disjoint axes:

  • INFERENCE MEMORY: 102 scenarios.

  • INSTRUCTION RETENTION: 62 scenarios.

  • SELF COHERENCE: 46 scenarios.

  • RELIABLE VERSION EDITING: 35 scenarios.

The main text reports the full 245-scenario aggregate first, then analyzes INFERENCE MEMORY and RELIABLE VERSION EDITING (137 scenarios) as a mechanism-focused subset. The per-axis table is in Appendix § 10.§ 4.

Chunked Context Construction

The current MULTICHALLENGE construction uses chunk size 3. A chunk bundles several session trajectories into a longer context block, creating topic interference. The system must identify the active trajectory, select relevant evidence, and ignore distractor sessions in the same chunk.

The construction procedure is:

  1. Group scenarios into chunks of size 3.

  2. For each chunk, concatenate the trajectories into a shared context block.

  3. Mark one scenario trajectory as the active trajectory for the current query.

  4. Present the system with the chunked context and the active final-turn request.

  5. Evaluate whether the final answer follows the active trajectory rather than distractor trajectories.

This setting stresses both retrieval selectivity and model capacity because the correct response requires maintaining the active state under structured interference.

Final Evaluation Judges

For REALMEM, we use the official retrieval and generation evaluation pipeline. The final reporting protocol uses GPT-4o as the generation/memory judge model where required, with retrieval-memory helpfulness evaluated using top-k=5k = 5 evidence. This final evaluator is used only for benchmark reporting and is distinct from the step-level feedback path used for online routing updates.

For MULTICHALLENGE, we use a path-aware final judge following the benchmark’s evaluation style. The judge first considers the trajectory through turns 1 to tmax⁡−1t_{\max} - 1: whether the system maintained the correct topic memory, preserved constraints, and produced intermediate responses that support the final answer. It then evaluates the final turn at tmax⁡t_{\max} by asking whether the maintained memory supports the response and whether the response itself is correct. A scenario is counted as passed under strict joint logic: the continuity/path verdict, final-answer judge verdict, and parseable model final-token answer must all satisfy the supplied pass criterion, with YES as the expected answer to the final official rubric question.

Path-aware MULTICHALLENGE final judge protocol.

Inputs:

  • Full trajectory through turns 1 to tmax⁡−1t_{\max} - 1.

  • The scenario’s official binary rubric question, used as the final user request at tmax⁡t_{\max} with “Please answer with only YES or NO.”

  • System final answer.

  • Scenario-specific PASS_CRITERIA or expected behavior.

Strict joint pass logic: a scenario passes only when the judge accepts memory-continuity/path consistency, accepts final-answer correctness, and the model’s final-token answer is parseable and matches the expected YES under the supplied pass criterion.

Evaluator setting: GPT-4o with deterministic judge decoding, temperature 0, and a short max-token budget.

Path-aware MULTICHALLENGE final judging protocol

Figure 7. Path-aware MULTICHALLENGE final judging protocol used for the fixed evaluation matrix.

Full Results

This section reports the full benchmark matrices. The three-persona component ablations are presented separately in Appendix § 11.

RealMem Full Matrix

Table 12 reports the full matrix. Context-limit failures are identified from provider error payloads: 1,344 Qwen2.5-72B queries, 1,344 Mixtral queries, and 1,247 Llama3B queries. Affected queries receive zero score and zero recorded cost.

Memory SettingModel / RouterGen.-Ctrl. ($) ↓ | Norm. QA ↑ | Mem Rec. ↑ | R@20 ↑ | NDCG@20 ↑
MARS26.64800.74060.67940.80150.6168
Full contextGPT-4o184.39670.75400.16400.51880.3216
Full contextMETALLM180.01890.73310.16030.51880.3216
Full contextQwen2.5 72B†0.47370.04220.20410.51880.3216
Full contextMixtral†0.29700.03360.15770.51880.3216
Full contextLlama3B†0.29110.02140.16050.51880.3216
MEM0GPT-4o41.38340.69090.59970.63050.5388
MEM0METALLM24.34990.66520.60480.63050.5388
MEM0Qwen2.5 72B3.83250.51370.59610.63050.5388
MEM0Mixtral3.57890.56750.60350.63050.5388
MEM0Llama3B0.93830.26890.60590.63050.5388
A-MEMGPT-4o43.13110.71720.64790.68170.5846
A-MEMMETALLM20.28750.70440.64710.68170.5846
A-MEMQwen2.5 72B3.51060.46120.64470.68170.5846
A-MEMMixtral3.75630.57960.64450.68170.5846
A-MEMLlama3B1.01750.28910.64750.68170.5846

Table 12. Full REALMEM matrix. Quality and memory metrics are persona-macro averages; cost records generation and controller API calls. †Context-limit failures affect 1,344/1,415 Qwen queries, 1,344/1,415 Mixtral queries, and 1,247/1,415 Llama3B queries, so scores conflate executability and response quality.

MultiChallenge Full Aggregate

Memory SettingModel / RouterPass Rate ↑Pass ↑Recorded Cost ()↓∣Cost/Pass() \downarrow| Cost/Pass () ↓
MARS0.236758/2453.25040.0560
Full contextQwen2.5 72B0.273567/2453.16650.0473
Full contextGPT-4o0.261264/24515.34910.2398
MEM0Mixtral0.232757/2450.77120.0135
MEM0GPT-4o0.228656/24513.97690.2496
MEM0METALLM0.212252/2454.78250.0920
Full contextMETALLM0.208251/2456.49450.1273
Full contextMixtral0.191847/2451.23270.0262
MEM0Qwen2.5 72B0.191847/2452.59460.0552
A-MEMMETALLM0.183745/2454.76640.1059
A-MEMGPT-4o0.179644/24511.33330.2576
A-MEMQwen2.5 72B0.179644/2451.41520.0322
A-MEMMixtral0.167341/2450.78230.0191
Full contextLlama3B0.077619/2450.60860.0320
MEM0Llama3B0.069417/2450.54480.0320
A-MEMLlama3B0.057114/2450.26170.0187

Table 13. MULTICHALLENGE full aggregate over 245 scenarios, the primary benchmark view. Costs follow the original run-level accounting convention; QTM/controller cost was not separately itemized in the archived logs.

MultiChallenge Mechanism-Focused Subset

Table 14 reports all 16 systems on the mechanism-focused subset. It supplements rather than replaces the full aggregate.

Memory SettingModel / RouterPass Rate ↑Pass ↑Recorded Cost ($) ↓ | Cost/Pass ($) ↓
MARS0.321244/1371.56700.0356
Full contextQwen2.5 72B0.328545/1371.85280.0412
MEM0Mixtral0.284739/1370.46200.0118
Full contextGPT-4o0.262836/1379.10440.2529
Full contextMixtral0.255535/1370.75340.0215
MEM0GPT-4o0.255535/1378.43750.2411
MEM0METALLM0.255535/1372.80920.0803
MEM0Qwen2.5 72B0.240933/1371.55970.0473
Full contextMETALLM0.226331/1373.68140.1188
A-MEMGPT-4o0.204428/1376.86310.2451
A-MEMMETALLM0.189826/1372.79600.1075
A-MEMMixtral0.189826/1370.46750.0180
A-MEMQwen2.5 72B0.175224/1370.84360.0352
Full contextLlama3B0.102214/1370.32710.0234
MEM0Llama3B0.080311/1370.31730.0288
A-MEMLlama3B0.05117/1370.14340.0205

Table 14. MULTICHALLENGE mechanism-focused subset over INFERENCE MEMORY and RELIABLE VERSION EDITING; not the primary leaderboard. Costs follow the original run-level accounting convention, without a separate historical QTM/controller itemization.

MultiChallenge Per-Axis Breakdown

Table 15 reports the category-level breakdown needed to contextualize the target subset and reduce ambiguity about axis-level behavior.

Memory SettingModel / RouterInf. Mem. /102Instr. Ret. /62Self Coh. /46Version Edit /35Total /245
MARS33861158
Full contextQwen2.5 72B36148967
Full contextGPT-4o281810864
Full contextMETALLM26128551
Full contextMixtral2866747
Full contextLlama3B1050419
MEM0GPT-4o271110856
MEM0METALLM2798852
MEM0Qwen2.5 72B2586847
MEM0Mixtral3199857
MEM0Llama3B1042117
A-MEMGPT-4o25115344
A-MEMMETALLM23136345
A-MEMQwen2.5 72B22128244
A-MEMMixtral2278441
A-MEMLlama3B652114

Table 15. MULTICHALLENGE per-axis pass-count breakdown. Bold and underline mark best and second-best pass counts within each axis.

Ablations and Sensitivity Analysis

This section provides component-level evidence for the mechanisms used by MARS. The main benchmark tables compare complete systems on the full evaluation slices; the analyses below examine QTM-guided memory construction, QTM-derived prompt state, no-QTM routing, recency-only memory selection, online updates, cost weighting, and local QTM prompt sensitivity. The main component ablations use a three-persona subset of the REALMEM evaluation pool, while the prompt-sensitivity probe uses frozen snapshots from seven evaluation personas. The main result aggregates all 10 personas and 1,415 queries. The diagnostic samples are drawn from this evaluation pool.

RealMem Three-Persona Ablation Setup

Table 16 summarizes the ablation setting. The three personas are randomly sampled with seed 28. All main memory-construction variants use a matched top-10 retrieval budget so that differences are not explained by exposing more historical memory. The component ablation slice uses λcost=0.5\lambda_{\mathrm{cost}} = 0.5 for all variants in Table 18. This differs from the main full-slice REALMEM run because the ablation is designed as a controlled within-slice diagnostic with stronger cost pressure; the resulting scores are compared only within the ablation slice.

SettingValue
Evaluation subsetThree personas from the 10-persona evaluation pool
Persona samplingrandom sample with seed 28
PersonasEthan_Hunt, Kenta_Tanaka, Sarah_Miller
Evaluated turns473
Historical retrieval budgettop-10
Model poolLlama3B, Qwen2.5 72B, GPT-4o
Reward cost weight for component ablationsλcost=0.5\lambda_{\mathrm{cost}} = 0.5
Generation temperature0.0
Provider fallbackenabled
Generation error rate0.0 for all variants

Table 16. REALMEM three-persona ablation setup. The slice is used to test component behavior under a controlled retrieval budget.

VariantQA raw ↑QA ↑Mem Rec. ↑Mem Help. ↑R@10 ↑NDCG@10 ↑Cost ↓MemFail ↓L/Q/G %
Full MARS2.23330.74440.68041.69670.67440.59745.6436.349.5/61.7/28.8
Embedding-only semantic2.16920.72310.64291.64610.60260.51625.9119.309.1/58.6/32.3
No QTM state/header2.12980.70990.63971.65750.60260.51626.33810.159.1/55.6/35.3
Recent-only memory1.88370.62790.10700.24100.29810.22995.63827.278.0/61.1/30.9
No-QTM semantic2.21620.73870.62871.61740.60260.51624.57010.993.0/75.9/21.1
No-QTM recent1.87320.62440.11180.23470.29810.22994.91834.678.0/66.6/25.4
No online update11.26640.42210.68821.70980.67440.59830.43918.6099.6/0.0/0.4

Table 18. Official REALMEM evaluation results on the three-persona ablation slice, with diagnostic failure and routing statistics. QA is official raw QA divided by 3; cost is generation-controller API cost in USD. Bold marks the best online component variant. 1The no-update row is a frozen-router diagnostic, excluded from best-online markings.

Metrics. We report the official REALMEM evaluation metrics on the ablation slice. Official QA is produced on the original 0–3 scale and normalized as QAnorm=QAraw/3\mathrm{QA}_{\mathrm{norm}} = \mathrm{QA}_{\mathrm{raw}}/3, matching the main text. Mem Recall and Mem Helpful are official memory-evaluation metrics. Recall@10 all and NDCG@10 measure retrieval coverage and ranking quality on the ablation slice. Total cost includes generator and QTM/controller calls where applicable, following the cost convention in Appendix § 8.§ 6. MemFail is the percentage of evaluated turns with memory-related hard-failure flags from the turn-level diagnostic judge. The L/Q/G column reports the percentage of turns routed to Llama3B, Qwen2.5 72B, and GPT-4o, respectively.

Variant Definitions

Table 17 defines the evaluated component variants. The ablations are organized by dependency: QTM-guided context construction, QTM state/header exposure in the generator prompt, QTM-based router features, recency-only memory selection, and online updates. The no-update variant keeps the QTM-guided memory view but freezes the router at initialization. We treat this as a diagnostic stress test of online adaptation rather than as an optimized static-router baseline.

VariantAblated dependencyPurpose
Full MARS (mars_full)noneFull memory-aware routing system.
Embedding-only semantic (mars_embedctx_bm)Replace QTM-guided memory construction with raw-query semantic top-10 retrievalTests whether QTM-guided context improves over embedding-only memory.
No QTM state/header (mars_semctx_noactive)Start from embedding-only semantic retrieval and remove the QTM state/header from the generator promptIsolates prompt-state exposure relative to the embedding-only variant; its retrieval policy is unchanged.
Recent-only memory (mars_recentctx_bm)Replace QTM/semantic retrieval with recent-only top-10 memoryTests whether recency alone is sufficient.
No-QTM semantic (mars_noqtm_semctx)Disable QTM; use semantic memory and query-embedding router featuresStress-tests the framework without QTM state.
No-QTM recent (mars_noqtm_recentctx)Disable QTM; use recent-only memory and query-embedding router featuresTests the weakest no-QTM recency-based setting.
No online update (mars_full_no_update)Keep QTM-guided memory construction but freeze the cold-start routerDiagnoses adaptation from this initialization.

Table 17. Ablation variants on the three-persona REALMEM evaluation slice. All variants use deterministic generation and the same model pool; historical retrieval uses a matched top-10 budget unless otherwise noted.

Official Evaluation Results

Table 18 reports the official REALMEM evaluation metrics and diagnostic routing statistics for the component ablation variants. Full MARS obtains the strongest overall profile among the online ablation variants: it has the highest normalized QA, the strongest Mem Recall, and the best retrieval metrics under the matched top-10 budget. The no-QTM semantic variant is close in QA, showing that semantic retrieval is a strong baseline, but it has lower memory scores than full MARS. The recent-only variants degrade sharply on memory and retrieval metrics, indicating that simple recency is not a reliable long-session memory strategy in this setting.

The no-update diagnostic has high memory and retrieval metrics because it retains QTM-guided context construction. However, its QA is much lower: freezing the cold-start router leaves the policy concentrated on the cheapest model, which preserves retrieval quality but substantially reduces answer quality. This row should therefore be interpreted as a cold-start online-adaptation diagnostic, not as a tuned static-router baseline.

Interpretation. The ablations support three conservative conclusions. First, QTM-guided memory construction provides the strongest quality–memory operating point among the main memory-construction variants under a matched top-10 retrieval budget. Full MARS improves over embedding-only semantic retrieval in QA, Mem Recall, Mem Helpful, Recall@10, NDCG@10, and MemFail. Second, semantic retrieval is a strong baseline: the no-QTM semantic variant approaches full MARS in normalized QA, but its lower Mem Recall and Mem Helpful scores indicate weaker memory reliability. Third, recency-only memory selection is insufficient for REALMEM-style long-session use, as both recent-only variants collapse on Mem Recall, Mem Helpful, Recall@10, and NDCG@10. The no-update result further shows that good memory construction alone is insufficient when the frozen router selects an underpowered generator.

The no-QTM semantic variant is cheaper and remains competitive in QA, while the no-update diagnostic is much cheaper but has substantially lower QA. These variants occupy different points on the quality–memory–cost trade-off.

Cost-Weight Sensitivity

Table 19 reports a cost-weight sensitivity analysis over the same three-persona ablation slice. These rows keep the full MARS architecture and vary only the cost coefficient in the online reward. We report the three additional settings, λcost∈{0,0.25,1.0}\lambda_{\mathrm{cost}} \in\{0, 0.25, 1.0\}; the main component-ablation setting uses λcost=0.5\lambda_{\mathrm{cost}} = 0.5 and is reported in Table 18. The results show that the cost coefficient shifts the quality–memory–cost operating point rather than producing a uniformly dominant setting. Removing the cost penalty slightly improves Mem Recall but increases cost and does not improve QA over the main setting. Stronger cost pressure reduces cost while maintaining comparable retrieval scores. We therefore treat λcost\lambda_{\mathrm{cost}} as a deployment knob rather than as a universally optimal scalar.

VariantQA ↑\uparrowMem Rec. ↑\uparrowR@10 ↑\uparrowNDCG@10 ↑\uparrowCost ↓\downarrowMemFail ↓\downarrowL/Q/G %
Full MARS, λ=0\lambda= 00.73290.68700.67020.59525.9648.257.8/60.7/31.5
Full MARS, λ=0.25\lambda= 0.250.72090.68320.67020.59335.5088.2510.4/61.7/27.9
Full MARS, λ=1.0\lambda= 1.00.72250.68590.67650.59995.3296.559.9/63.8/26.2

Table 19. Cost-weight sensitivity on the same three-persona REALMEM ablation slice. These rows keep the full MARS architecture and vary only the reward cost coefficient. The main component-ablation setting uses λcost=0.5\lambda_{\mathrm{cost}} = 0.5 and is reported in Table 18.

QTM Prompt Sensitivity

We additionally test the local stability of the QTM interface under two controlled prompt changes using 150 frozen turn snapshots from seven REALMEM evaluation personas. For each snapshot, the user query, active context, recent history, system rules, historical memory pool, and embeddings from BAAI/bge-small-en-v1.5 ([29]) are fixed. We compare the original QTM prompt with (i) a paraphrase that preserves the decision definitions and JSON schema and (ii) an instruction-reordered variant that reverses the five Important semantics items while leaving the remaining text unchanged. All variants use Qwen2.5-7B-Instruct with deterministic decoding. This is a controller-level diagnostic rather than an end-to-end rerun of the main benchmark.

We report two quantities with direct downstream effects. Action agreement is the fraction of inputs whose effective MATCH/NEW_TOPIC decision is unchanged after the same production parser and deterministic release logic. Retrieval-source Jaccard is computed per input over the selected historical memory-chunk IDs (up to k=20k = 20 before final prompt-budget packing) and then averaged; double-empty pairs receive 1 and one-sided empty pairs receive 0. The original prompt is a reference, not a gold annotation, so disagreement is not an error rate.

The full-sample results show high local agreement under both tested changes, with still higher agreement on the 60 active-history inputs. We additionally report repeat-run, parsing, and candidate-pool checks. On a preselected 30-input repeat set, same-template reruns yield 96.67%/0.9667 action agreement/Jaccard for the original prompt and 100%/1.000 for each variant. These repeated runs provide a descriptive reference for within-template variability. All 450 primary calls complete without request errors, production parsing failures, or conservative action fallbacks; three invalid auxiliary query_type values under instruction reordering are mapped to the standard STATIC default. Effective-action agreement remains close to raw-action agreement (90.0% versus 88.7% for paraphrasing and 96.0% for reordering), while auxiliary schema fields are not perfectly invariant.

High retrieval overlap also persists with larger candidate pools. Historical pools range from 2 to 224 chunks (median 94.5); among the 118 inputs with at least 2k=402k = 40 candidates, mean Jaccard is 0.916 for paraphrasing and 0.975 for instruction reordering. The original prompt produces 127 MATCH and 23 NEW_TOPIC decisions; paraphrasing changes 15 NEW_TOPIC decisions to MATCH and none in the opposite direction, while reordering changes five in the former direction and one in the latter. Because the original prompt is not ground truth, these transitions limit claims about subclass invariance but do not establish a quality degradation.

Overall, the probe supports a narrow conclusion: under the tested paraphrase and instruction reordering, QTM largely preserves the effective thread-boundary decision and retrieved memory sources. It does not establish invariance of the complete structured state, accumulated online trajectories, or end-to-end answer quality.

QTM Controller Substitution

We complement the prompt perturbations with a same-scale controller substitution on the same 150 frozen, stratified reconstructed snapshots. The default Qwen2.5-7B-Instruct controller is replaced by Mistral-7B-Instruct-v0.3 while the original QTM prompt and schema, session histories, historical memory pools, BGE embeddings, production parser, deterministic release logic, and retrieval policy are held fixed. The comparison therefore isolates the controller model within this frozen-snapshot diagnostic; it does not regenerate assistant answers or rerun the online routing trajectory.

We use the same two downstream quantities as in Appendix § 11.§ 5: the effective MATCH/NEW_TOPIC decision and Jaccard overlap of the pre-packing top-20 historical memory-source IDs. All controller outputs used in this probe are parseable JSON, and the runs complete without request errors, runtime errors, or parser fallback.

MetricAll 150 inputs60 with active history
Action agreement124/150 (82.7%)53/60 (88.3%)
Wilson 95% interval75.8–87.9%77.8–94.2%
Mean retrieval Jaccard0.8750.960
Median retrieval Jaccard1.0001.000
Identical retrieval set115/150 (76.7%)52/60 (86.7%)
Jaccard ≥ 0.9125/150 (83.3%)55/60 (91.7%)

Table 21. QTM controller-substitution diagnostic. Mistral-7B-Instruct-v0.3 is compared with the Qwen2.5-7B-Instruct original-prompt reference on the same fixed snapshots. The active-history slice is the same 60-input slice used in Table 20.

Prompt variantAll 150 inputs60 with active history
Action agr. ↑Mean Jaccard ↑Action agr. ↑Mean Jaccard ↑
Paraphrase90.0%0.92095.0%0.984
Instruction reordering96.0%0.97398.3%1.000

Table 20. QTM prompt-sensitivity results. The active-history slice is post hoc and includes inputs where NEW_TOPIC can trigger archival.

The two controllers agree on most effective thread-boundary decisions, and the induced retrieval-source sets overlap strongly. The agreement is higher on inputs with active thread history, where a NEW_TOPIC decision can change archival behavior: action agreement reaches 88.3%, mean Jaccard reaches 0.960, and 86.7% of retrieval sets are identical. These results provide controller-substitution evidence at the decision-interface level for one alternative 7B controller, complementing the prompt-sensitivity probe above.

Qualitative and Behavioral Analysis

Context and Memory Failure Cases

This section provides qualitative examples showing how systems fail under long histories, topic interference, or version-state pressure. The goal is to make the mechanism behind the aggregate metrics visible. The examples below are anonymized from fixed MULTICHALLENGE final evaluation outputs where MARS passes and multiple fixed full-context baselines fail.

Query / settingRequired memoryBaseline patternMARS patternError type
Present-tense continuation constraintPreserve a previously imposed grammar/style rule across turns.Several full-context baselines lose the active rule.Maintains the constraint and returns a passing final verdict.Instruction retention
School memory recall under distractorsRecall that a schoolyard feature was located near a sanitation hazard.Baselines deny the specific detail or lose path continuity.Recovers the relevant detail and answers consistently.Inference memory
Three-paragraph revision limitPreserve a local document-format constraint while editing a later version.Baselines drift on the paragraph-count requirement.Keeps the active artifact state and satisfies the editing constraint.Reliable version editing

Table 22. Representative anonymized MULTICHALLENGE patterns summarized from fixed final-evaluation outputs. They are illustrative rather than additional quantitative evidence.

Static-Configuration Error Examples on REALMEM

The following paired examples illustrate missing, ignored, or stale memory but do not establish cause. Aggregate cheap full-context scores are dominated by the overflows in Table 12, so they are not clean evidence of inferior model reasoning.

Failure caseRelevant memory evidenceStatic-system behaviorMARS behaviorDiagnosis
Travel-plan follow-up after frustrationJuly travel windows, five adult travelers, and comfort-focused flight constraints.Gives generic or conflicting travel guidance.Recovers the travel constraints and responds in the correct planning frame.Relevant-memory miss
Broken workout-video linkA recurring 9 PM back-care routine and selected exercise playlist.Gives generic troubleshooting or ignores the routine context.Uses the routine and exercise context to provide a targeted next step.Context ignored
Story-development continuationThe “Golden Hook” is locked in and the next component is character development.Pivots to a generic plot-structure step.Continues from the locked story state and focuses on the intended component.Version-state drift

Table 23. Representative anonymized REALMEM static-configuration error patterns; diagnoses are illustrative and descriptive, not causal.

Routing Behavior by Session State

This diagnostic summarizes whether the router’s behavior changes with QTM state. We summarize model choice and turn cost by realized retrieval scope, interaction phase, and memory-token exposure. Memoryscope slices use retrieval_scope; phase slices use qtm_phase. Cheap models group Llama-3.2-3B and Mixtral-8x7B; the mid model is Qwen-72B; the strong model is GPT-4o. High context difficulty is the top third of turns by mem_tokens. These slices are descriptive diagnostics rather than causal routing rules: model choice depends on the full feature vector and online reward history, so a single QTM field such as retrieval_scope need not monotonically increase strong-model usage.

QTM state sliceCheap model %Mid model %Strong model %Avg. cost
retrieval_scope=LOCAL12.962.924.30.0199
retrieval_scope=GLOBAL16.665.617.80.0155
qtm_phase=REVISE7.668.424.00.0202
qtm_phase=SYNTHESIZE/VERIFY22.151.626.20.0207
High context difficulty14.462.922.70.0206

Table 24. Routing-behavior diagnostic over 1,415 REALMEM turns. Percentages are within each slice; average cost includes generator and QTM/controller cost when logged.

Deployment Feedback and Reproducibility

Replacing Turn-Level Judges with Deployment Feedback

The experiments use a turn-level LLM judge to provide scalar feedback for online routing. This choice makes benchmark evaluation reproducible, but the MARS interface does not require the reward to come from an LLM judge. The online router only requires a scalar reward associated with the current state, selected model, response, and cost.

In a deployed assistant, this reward could be replaced or supplemented by human or task-derived feedback. Examples include explicit user ratings, edit acceptance, task completion signals, human preference comparisons, workflow success checks, or organization-specific quality audits. Over repeated interactions, such feedback could train a routing policy that reflects user-specific or domain-specific preferences. For example, one user may prefer cheaper models unless quality drops sharply, while another may prefer stronger models for revision or finalization tasks.

This interface allows the routing policy to incorporate application-specific feedback without changing the shared-state memory construction and model-selection pipeline.

Reproducibility Checklist

Table 25 summarizes the experimental setup. Following submission, we will release an implementation package with code, configurations, prompt templates, and evaluation scripts to support reproduction.

Artifact license and intended use. Our release is intended to support reproduction of the reported experiments. It will include code, prompt templates, configurations, and evaluation scripts, but it will not redistribute benchmark data or private intermediate logs. Users should obtain REALMEM and MULTI-CHALLENGE from their original sources and follow their respective licenses and terms. We will release our implementation artifacts under the Apache License 2.0.

ItemReproducibility information
Code and configsAn implementation package will be released with system configurations, prompt templates, and evaluation scripts.
REALMEM splitREALMEM evaluation pool with 10 personas and 1,415 queries; QA is normalized as official raw QA divided by 3.
MULTICHALLENGE splitConverted MULTICHALLENGE split with 245 scenarios and 1,480 turns; chunk size 3; target subset consists of INFERENCE MEMORY and RELIABLE VERSION EDITING.
Ablation sliceThree-persona REALMEM evaluation slice with 473 turns (472 valid QA evaluations for the full variant); personas sampled with seed 28; matched top-10 historical retrieval budget.
Candidate model poolsMain REALMEM runs use the configured MARS candidate pool reported in Appendix § 8.§ 4; ablations use Llama3B, Qwen2.5 72B, and GPT-4o.
QTM and router settingsMain QTM uses qwen/qwen-2.5-7b-instruct; its prompt, schema, guardrails, and state-to-memory mapping are fixed. The controller-swap diagnostic additionally evaluates Mistral-7B-Instruct-v0.3 with the same original prompt/schema. Only the router updates online in the main system.
Feedback and evaluatorsGPT-4o-mini supplies online routing feedback; primary benchmark final evaluators are described in Appendix § 9.§ 4. The interaction-order diagnostic also uses a fixed GPT-4o-mini path-aware evaluator across all replays.
Reward and costReward aggregation is specified in Appendix § 8.§ 5; rubric dimensions appear in Table 9. Main REALMEM uses λcost=0.002\lambda_{\mathrm{cost}} = 0.002; component ablations use λcost=0.5\lambda_{\mathrm{cost}} = 0.5. Cost accounting follows Appendix § 8.§ 6.
Cost reportingMain REALMEM tables include generator and controller API costs (Appendix § 8.§ 6); historical MULTICHALLENGE costs use the original run-level convention. Final evaluators and local compute are excluded.

Table 25. Reproducibility checklist. The release package will provide the implementation, configurations, prompt templates, and evaluation scripts.

References

  1. [1]Yasin Abbasi-Yadkori, Dávid Pál, and Csaba Szepesvári. 2011. Improved algorithms for linear stochastic bandits. In Advances in Neural Information Processing Systems 24.
  2. [2]Yushi Bai, Xin Lv, Jiazheng Zhang, Hongchang Lyu, Jiankai Tang, Zhidian Huang, Zhengxiao Du, Xiao Liu, Aohan Zeng, Lei Hou, Yuxiao Dong, Jie Tang, and Juanzi Li. 2023. LongBench: A bilingual, multitask benchmark for long context understanding. arXiv preprint arXiv:2308.14508.arxiv.org/abs/2308.14508
  3. [3]Haonan Bian, Zhiyuan Yao, Sen Hu, Zishan Xu, Shaolei Zhang, Yifu Guo, Ziliang Yang, Xueran Han, Huacan Wang, and Rongchao Chen. 2026. Realmem: Benchmarking llms in real-world memory-driven interaction. arXiv preprint arXiv:2601.06966.arxiv.org/abs/2601.06966
  4. [4]Lingjiao Chen, Matei Zaharia, and James Zou. 2023. FrugalGPT: How to use large language models while reducing cost and improving performance. arXiv preprint arXiv:2305.05176.arxiv.org/abs/2305.05176
  5. [5]Prateek Chhikara, Dev Khant, Saket Aryan, Taranjeet Singh, and Deshraj Yadav. 2025. MemO: Building production-ready ai agents with scalable long-term memory. arXiv preprint arXiv:2504.19413.
  6. [6]Wei Chu, Lihong Li, Lev Reyzin, and Robert E. Schapire. 2011. Contextual bandits with linear payoff functions. In Proceedings of the Fourteenth International Conference on Artificial Intelligence and Statistics, volume 15 of Proceedings of Machine Learning Research, pages 208–214. PMLR.
  7. [7]Dujian Ding, Ankur Mallick, Chi Wang, Robert Sim, Subharata Mukherjee, Victor Ruhle, Laks V. S. Lakshmanan, and Ahmed Hassan Awadallah. 2024. Hybrid LLM: Cost-efficient and quality-aware query routing. arXiv preprint arXiv:2404.14618.arxiv.org/abs/2404.14618
  8. [8]Qitian Jason Hu, Jacob Bieker, Xiuyui Li, Nan Jiang, Benjamin Keigwin, Gaurav Ranganath, Kurt Keutzer, and Shriyaas Kaustubh Upadhyay. 2024. RouterBench: A benchmark for multi-LLM routing system. arXiv preprint arXiv:2403.12031.arxiv.org/abs/2403.12031
  9. [9]Albert Q. Jiang, Alexandre Sablayrolles, Antoine Roux, Arthur Mensch, Blanche Savary, Chris Bamford, Devendra Singh Chaplot, Diego de las Casas, Emma Bou Hanna, Florian Bressand, Gianna Lengyel, Guillaume Bour, Guillaume Lam ple, Lélio Renard Lavaud, Lucile Saulnier, Marie-Anne Lachaux, Pierre Stock, Sandeep Subramanian, Sophia Yang, and 7 others. 2024. Mixtral of experts. arXiv preprint arXiv:2401.04088.arxiv.org/abs/2401.04088
  10. [10]Patrick Lewis, Ethan Perez, Aleksandra Piktus, Fabio Petroni, Vladimir Karpukhin, Naman Goyal, Heinrich Küttler, Mike Lewis, Wen-tau Yih, Tim Rocktäschel, Sebastian Riedel, and Douwe Kiela. 2020. Retrieval-augmented generation for knowledge-intensive NLP tasks. In Advances in Neural Information Processing Systems, volume 33, pages 9459–9474.arxiv.org/abs/2005.11401
  11. [11]Lihong Li, Wei Chu, John Langford, and Robert E. Schapire. 2010. A contextual-bandit approach to personalized news article recommendation. In Proceedings of the 19th International Conference on World Wide Web, WWW ’10, pages 661–670. Association for Computing Machinery.DOI
  12. [12]Nelson F. Liu, Kevin Lin, John Hewitt, Ashwin Paranjape, Michele Bevilacqua, Fabio Petroni, and Percy Liang. 2024. Lost in the middle: How language models use long contexts. Transactions of the Association for Computational Linguistics, 12:157–173.arxiv.org/abs/2307.03172
  13. [13]Adyasha Maharana, Dong-Ho Lee, Sergey Tulyakov, Mohit Bansal, Francesco Barbieri, and Yuwei Fang. 2024. Evaluating very long-term conversational memory of LLM agents. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 13851–13870, Bangkok, Thailand. Association for Computational Linguistics.
  14. [14]Meta. 2024. Llama 3.2 model card. Accessed October 9, 2026.
  15. [15]Mistral AI. 2024. Model card for Mistral-7B-Instruct-v0.3. Accessed October 9, 2026.
  16. [16]Quang H. Nguyen, Duy C. Hoang, Juliette Decugis, Saurav Manchanda, Nitesh V. Chawla, and Khoa D. Doan. 2024. Metallm: A high-performant and cost-efficient dynamic framework for wrapping llms. arXiv preprint arXiv:2407.10834.arxiv.org/abs/2407.10834
  17. [17]Isaac Ong, Amjad Almahairi, Vincent Wu, Wei-Lin Chiang, Tianhao Wu, Joseph E. Gonzalez, M. Waleed Kaddous, and Jon Stoica. 2024. Routellm: Learning to route llms with preference data. arXiv preprint arXiv:2406.18665.arxiv.org/abs/2406.18665
  18. [18]OpenAI. 2024a. GPT-4o mini: advancing cost-efficient intelligence. Accessed October 9, 2026.
  19. [19]OpenAI. 2024b. Hello GPT-4o. Accessed October 9, 2026.
  20. [20]Charles Packer, Sarah Wooders, Kevin Lin, Vivian Fang, Shishir G. Patil, Ion Stoica, and Joseph E. Gonzalez. 2023. MemGPT: Towards LLMs as operating systems. arXiv preprint arXiv:2310.08560.arxiv.org/abs/2310.08560
  21. [21]Pranoy Panda, Raghav Magazine, Chaitanya Devaguptapu, Sho Takemori, and Vishal Sharma. 2025. Adaptive llm routing under budget constraints. arXiv preprint arXiv:2508.21141.arxiv.org/abs/2508.21141
  22. [22]Joon Sung Park, Joseph C. O’Brien, Carrie J. Cai, Meredith Ringel Morris, Percy Liang, and Michael S. Bernstein. 2023. Generative agents: Interactive simulacra of human behavior. In Proceedings of the 36th Annual ACM Symposium on User Interface Software and Technology, UIST ’23. Association for Computing Machinery.arxiv.org/abs/2304.03442
  23. [23]Manhin Poon, XiangXiang Dai, Xutong Liu, Fang Kong, John C. S. Lui, and Jinhang Zuo. 2025. Online multi-llm selection via contextual bandits under unstructured context evolution. arXiv preprint arXiv:2506.17670.arxiv.org/abs/2506.17670
  24. [24]Tal Shnitzer, Anthony Ou, Mírian Silva, Kate Soule, Yuekai Sun, Justin Solomon, Neil Thompson, and Mikhail Yurochkin. 2023. Large language model routing with benchmark datasets. arXiv preprint arXiv:2309.15789.arxiv.org/abs/2309.15789
  25. [25]Ved Sirdeshmukh, Kaustubh Deshpande, Johannes Mols, Lifeng Jin, Ed-Yeremai Cardona, Dean Lee, Jeremy Kritz, Willow Primack, Summer Yue, and Chen Xing. 2025. Multichallenge: A realistic multi-turn conversation evaluation benchmark challenging to frontier LLMs. arXiv preprint arXiv:2501.17399.arxiv.org/abs/2501.17399
  26. [26]Xiaqiang Tang, Qiang Gao, Jian Li, Nan Du, Qi Li, and Sihong Xie. 2024. MBA-RAG: a bandit approach for adaptive retrieval-augmented generation through question complexity. arXiv preprint arXiv:2412.01572.arxiv.org/abs/2412.01572
  27. [27]Xinyuan Wang, Yanchi Liu, Wei Cheng, Xujiang Zhao, Zhengzhang Chen, Wenchao Yu, Yanjie Fu, and Haifeng Chen. 2025. Mixllm: Dynamic routing in mixed large language models. arXiv preprint arXiv:2502.18482.arxiv.org/abs/2502.18482
  28. [28]Wang Wei, Tiankai Yang, Hongjie Chen, Yue Zhao, Franck Dernoncourt, Ryan A. Rossi, and Hoda Eldardiry. 2025. Learning to route LLMs from bandit feedback: One policy, many trade-offs. Preprint, arXiv:2510.07429.arxiv.org/abs/2510.07429
  29. [29]Shitao Xiao, Zheng Liu, Peitian Zhang, Niklas Muennighoff, Defu Lian, and Jian-Yun Nie. 2024. C-Pack: Packed resources for general Chinese embeddings. In Proceedings of the 47th International ACM SIGIR Conference on Research and Development in Information Retrieval, pages 641–649. Association for Computing Machinery.arxiv.org/abs/2309.07597
  30. [30]Wujiang Xu, Zujie Liang, Kai Mei, Hang Gao, Juntao Tan, and Yongfeng Zhang. 2025. A-mem: Agentic memory for llm agents. arXiv preprint arXiv:2502.12110.arxiv.org/abs/2502.12110
  31. [31]An Yang, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chengyuan Li, Dayiheng Liu, Fei Huang, Haoran Wei, Huan Lin, Jian Yang, Jianhong Tu, Jianwei Zhang, Jianxi Yang, Jiaxi Yang, Jingren Zhou, Junyang Lin, Kai Dang, and 22 others. 2024. Qwen2.5 technical report. arXiv preprint arXiv:2412.15115.arxiv.org/abs/2412.15115

Paper details

vlnc:W_885d6qyq
Version
v1 · latest
Published
Fresh — added to the Hub within the last 30 days
Added
License
CC BY 4.0
Usage

views · 0

downloads · 0

A view counts once per browser per day, a download each time the PDF is requested.

  • Computer Science

Not recorded for this paper

View statement
Looking for related papers…
Contents