betaFeedback

WorksComputer Science

Peer Memory Changes Who LLM Agents Consult Without a Detected Gain in Answer Accuracy

Abstract

Multi-agent LLM systems often reuse the same agents across tasks, creating an opportunity to draw on past interactions when deciding whom to consult. We study this form of memory with PeerMem, which saves MACE's LinUCB state and reuses it when the same agents return. Using six agents, a fixed set of 80 HotpotQA questions, and 12 paired lineages per model, we compare cold, warm, and shuffled initialization in an answer-exchange protocol with reference-answer feedback. Under stable evidence access, warm memory lowers pooled hindsight regret and increases consultation of other evidence holders on Qwen3-32B and Gemma-4-31B-it, while final token F1 differences remain uncertain. Qwen interaction traces further show that warm agents more often encounter peer answers that score above their own prior answers. We extend the evaluation to Qwen3.5-9B using freshly learned memory to compare warm and cold initialization under reassigned and paired stable roles. The primary changed-minus-stable final-F1 interaction is -1.38 points, with a 95% interval of [-3.50,+0.73]. Together, these experiments trace the effects of peer memory through consultation, answer revision, and final task performance.

Introduction

Multi-agent large language model (LLM) systems often reuse the same agents across tasks, allowing past interactions to inform later decisions about whom to consult [21, 8] (Li et al. 2023). Remembering which peers were useful can reduce repeated exploration, provided that their past performance remains informative. The value of this experience also depends on what a selected peer communicates and how the reader uses that information to revise its answer.

We study this problem in the HotpotQA setting used by MACE, where six agents each receive one passage: two receive evidence and four receive distractors, with these roles hidden from the agents [3, 22]. In the stable-role experiment, the same IDs retain their roles across two sets of 40 questions within a paired replicate, which we call a lineage. This makes past experience relevant to later consultation, since peer usefulness primarily reflects stable access to evidence and the selector receives reference-answer feedback after each revision. Our extension, PeerMem, saves the LinUCB statistics learned for each selector-peer pair and reuses them when the same IDs return [10]. On the second set of questions, we compare empty state, correctly matched saved state, and the same state shuffled across peer IDs.

Our contribution is a controlled empirical study of how persistent peer memory shapes collaboration. The cold, warm, and shuffled conditions measure changes in consultation under stable evidence access, while interaction traces connect the quality of consulted answers to the revisions that follow. We then

40th Conference on Neural Information Processing Systems (NeurIPS 2026). Workshop: Personalized, Aligned, Long-Term Memory for AI Systems (PALM). move evidence to former distractor holders in a further study, pairing the reassigned conditions with stable controls to examine memory under changing information access. This design evaluates peer memory across three stages of collaboration: choosing whom to consult, revising an answer, and completing the task.

Multi-agent coordination and peer selection

Multi-agent LLM systems support collaborative reasoning by exchanging answers, critiques, and intermediate results [21, 5] (Li et al. 2023). Effective collaboration depends on both coordination and the use of the information received: debate does not consistently improve over simpler prompting [18], and communication or verification failures can undermine its benefits [2]. These findings motivate evaluating peer selection together with the answers agents produce after consultation.

MACE learns pair-specific LinUCB statistics for peer selection; its July 2026 version evaluates reuse with frozen parameters on a later task block and transfer from HotpotQA to 2WikiMultihopQA [3, 10]. PeerMem builds on this selector by comparing cold, correctly indexed, and shuffled initialization while learning continues online. Across stable and reassigned evidence access, we track how this initialization affects both consultation and subsequent answer revision.

Persistent memory, partner models, and reputation

Agent memory retains reflections, skills, and workflows for later tasks [16, 25, 20]. G-Memory organizes collaboration experience into a hierarchy of insight, query, and interaction graphs that agents can reuse across trials [24]. Explicit Trait Inference models partners’ warmth and competence from interaction histories, reuses frozen profiles in later interactions, and tests profile transfer between Stag Hunt and Prisoner’s Dilemma [1].

Several methods focus on peer reliability. Σ-Mem updates peer-competence and peer-relationship memories from correctness feedback and evaluates counterfactual reliability shifts, with a direct routing readout that returns the selected peer’s answer [6]. Our protocol follows the additional step in which the reader revises its own answer after consultation. Related approaches include DRF, which combines an interactive rating network, reputation scores, and UCB-based agent selection [11], and CADMAS-CTX, which maintains context-conditioned capability posteriors for uncertainty-aware delegation and evaluates warm starts followed by task-mixture shifts [12]. Together, these studies motivate examining how a selector’s accumulated experience affects whom agents consult and the answers they subsequently produce.

This question also builds on classical reputation systems and bandits that learn from historical observations [9, 17, 23] (Teacy et al. 2006). By holding the selector fixed and varying its initialization, we study the value of past experience in a setting where access to evidence is the main source of differences in peer usefulness.

Method

Task and interaction protocol

We use the HotpotQA distractor setting, in which each question contains two evidence passages and eight distractors [22]. For each question, we distribute the two evidence passages and four sampled distractors among six agents, giving each agent one passage. The stable-role experiment keeps evidence with the same two IDs throughout both task blocks, so the passage contents change across questions while the hidden roles remain fixed.

Each question begins at revision round t=0t = 0, when every agent answers from its own passage, and continues through three rounds of consultation and revision. In round tt, each agent selects an ID, reads that peer’s answer from round t−1t - 1, and revises its own answer; all agents complete a round before the next begins. The exchange contains only answers, while each agent’s passage remains private and evidence-role labels remain hidden.

MACE, UCB1, in-context, and random selection can choose any of the six IDs, including self, in which case the agent receives its own previous answer. Uniform random selection therefore chooses self with probability 1/61/6, and the fixed-cycle baseline always selects the next ID. Figure 1 summarizes this protocol and the memory intervention between task blocks.

Stable-role protocol with six agents, hidden evidence holders, Layer 2 initialization, and synchronous revisions

Figure 1. Stable-role protocol with evidence assigned to the same two IDs across task blocks and roles hidden from the agents. Each question consists of an initial answer and three synchronous revisions per agent. The MACE conditions differ in initialization and all update online using reference-answer rewards; random selection provides an additional baseline.

Peer selection and persistent memory

MACE represents each possible read with a normalized nine-feature context vector computed from the agents’ responses and interaction history [3]. The vector combines three nn-gram diversity features, three response-graph centrality features, the selector-peer pair’s past mean reward, the normalized revision round, and a bias term. Appendix § 13 gives the full definition.

For each ordered pair of selecting agent ii and candidate peer jj, LinUCB maintains a design matrix AijA_{ij} and reward vector bijb_{ij}, along with the pair’s selection count and mean reward [10]. MACE selects the candidate with the largest predicted reward plus an uncertainty bonus, then updates that pair’s statistics using the reward from the reader’s revised answer. PeerMem saves these pair-indexed statistics so that they can initialize a later run.

We vary this initialization through three conditions: a cold start begins with empty state, a warm start loads the saved state under the original peer IDs, and a shuffled start assigns each of the five non-self records to a different peer ID while leaving the self record in place. All three conditions continue updating their state after initialization, allowing us to compare their behavior under the same feedback rule.

Reward and evaluation measures

We evaluate answers using exact match (EM) and bag-of-token F1F_1 under the SQuAD convention, which normalizes text by lowercasing, removing punctuation and articles, and collapsing whitespace [15]. Final accuracy averages the six agents’ individual scores after revision round T=3T = 3, giving 240 responses from 40 questions for each lineage and condition. Comparisons then use the 12 paired lineage means.

Let sq,i,ts_{q,i,t} be the selecting agent’s token F1F_1 after revision round tt. We reward a selection using the reader’s resulting answer:

rq,i,t=12[(sq,i,t−sq,i,t−1)+sq,i,t].(1)r_{q,i,t} = \frac{1}{2}\left[(s_{q,i,t} - s_{q,i,t-1}) + s_{q,i,t}\right]. \tag*{(1)}

The reward lies in [−0.5,1][-0.5, 1] and credits both improvement and retention of a high-scoring answer. In both layers, the evaluator computes it from the reference answer and uses it to update the selector, while keeping the reference answer hidden from the answering agents.

Evidence-holder reads. We identify evidence holders from the seeded passage assignment and measure consultation in two ways. The first is the fraction of all selections directed to either evidence-holder ID, including self-selection, and has a uniform expectation of 2/6=1/32/6 = 1/3; this is the measure shown in Figure 2. The external evidence-holder read rate counts only selections of an evidence holder other than the reader, using all selections as its denominator. With a uniform expectation of 5/185/18, this rate measures how often the reader consults another evidence holder.

Figure 2. Warm minus cold difference in the probability of selecting an evidence-holder ID across Layer 2, including self-selection. Each point summarizes a five-question bin across 12 lineages, with shaded bands showing marginal, exploratory 95% paired t11t_{11} intervals and dashed lines showing half-run means. Qwen3-32B retains a 4–5 percentage point advantage, while the Gemma-4-31B-it advantage falls from 9.6 points to 2.2 points.

Selection entropy and realized reward. If pi,jp_{i,j} is agent ii’s share of selections directed to ID jj, its selection entropy is −∑jpi,jlog⁡pi,j-\sum_j p_{i,j} \log p_{i,j}, which we average across the six selecting agents. This quantity ranges from zero for one repeated choice to log⁡6\log6 for uniform selection, while mean realized reward averages Equation (1) over the reads that occurred.

Pooled hindsight selection regret. For a lineage ℓ\ell, let μ^ℓ(i,j)\hat{\mu}_{\ell}(i,j) denote the mean reward observed when agent ii selected peer jj, pooled across conditions within the same layer. Taking bℓ(i)=max⁡jμ^ℓ(i,j)b_{\ell}(i) = \max_j \hat{\mu}_{\ell}(i,j) as the hindsight benchmark, we define regret for condition aa as

Rℓa=∑(q,i,t)∈Da,ℓ[bℓ(i)−μ^ℓ(i,jq,i,ta)].(2)R_{\ell}^{a} = \sum_{(q,i,t)\in D^{a,\ell}} \left[b_{\ell}(i) - \widehat{\mu}_{\ell}\left(i,j_{q,i,t}^{a}\right)\right]. \tag*{(2)}

This metric compares each choice with the peer that has the highest pooled observed mean after the run, using peer values estimated from the histories each policy produces. Lower regret indicates greater concentration on peers with high pooled rewards, which we assess alongside the realized reward and final accuracy of those choices. We make these in-sample comparisons within each model and layer, using a shared question set, condition pool, and hindsight benchmark.

Interaction diagnostics. To examine the step from consultation to revision, we compare the revised answer after each external read with the reader’s and selected peer’s answers from round t−1t-1. We measure the selected answer’s F1F_1, whether it scores above the reader’s prior answer, and the reader’s subsequent change in F1F_1. When the prior answers differ after normalization, we also measure answer agreement: whether the revision matches the peer’s prior answer. Recovery and damage track transitions to and from token-perfect F1=1F_1 = 1, which can differ from exact match. These exploratory diagnostics summarize the interactions produced by each policy, with eligible denominators reported explicitly and all comparisons aggregated within lineages.

Experimental setup

Models, replication, and comparison conditions

The stable-role experiment evaluates Qwen3-32B [13] and Gemma-4-31B-it [7] over 12 seeded lineages per model, with all six agents in a run sharing the same deployment and prompts. Every model, lineage, and condition receives the same ordered set of 80 questions: the first 80 usable entries in the HotpotQA distractor validation split after filtering for hard questions with at least 10 passages. Layer 1 uses the first 40 questions and Layer 2 the next 40, with the lineage seed determining passage assignments and selector behavior.

Layer 1: selector check. Layer 1 compares five policies on the first 40 questions. Alongside the LinUCB policy mace, we evaluate ucb1, which uses empirical reward and a count-based UCB bonus with coefficient 2\sqrt{2} without the nine-feature context, and pre_defined, which follows a fixed cycle in which each agent selects the next ID, producing zero selection entropy. The in_context policy asks the language model whose response to read after showing each candidate’s selection count, mean reward, and nine-feature context; invalid outputs fall back to the fixed cycle. Finally, random samples uniformly from all six IDs. The state learned by mace in each lineage initializes the memory conditions in Layer 2.

Layer 2: memory transfer. Layer 2 evaluates memory transfer on 40 disjoint questions using four conditions. Both mace_warm and mace_shuffled reuse the lineage’s Layer 1 mace state: warm preserves the original peer IDs, and shuffled permutes them as described in § 3.2. The comparison conditions are mace_cold, which initializes new MACE state, and random, which samples uniformly and independently of learned state.

The shuffle moves every non-self record to a different ID while allowing transfers between the two evidence holders. Across the 12 lineages, 47 of 144 records learned about evidence holders (32.6%) remain assigned to evidence-holder IDs at initialization, so this control tests a partial disruption of identity and role alignment. Appendix § 17 describes the retained alignment, and Appendix § 13 gives implementation details, decoding settings, and response statistics.

Evidence-role reassignment

We test evidence-role reassignment in an additional study on Qwen3.5-9B [14], served by Together AI with advertised FP8 quantization. Following the same fixed 40-question learning and 40-question evaluation blocks, the study uses six agents and three synchronous revisions. Each of 12 lineages learns fresh MACE state from empty initialization, with all priors completed before evaluation begins.

Evaluation crosses cold or warm initialization with stable or changed evidence-holder roles. Stable conditions retain the original passage assignment, while changed conditions swap both evidence holders’ passages with two seeded former distractor holders so that old and current evidence-holder IDs are disjoint. The resulting assignment stays fixed over the 40 evaluation questions. Each warm condition loads a separate copy of the lineage’s prior under the original IDs, and each cold condition starts empty; all conditions continue the reference-answer updates in Equation (1). The intervention changes both readers’ and peers’ information ownership, while agents receive neither role labels nor peer passages.

The primary endpoint, fixed before inspecting this study’s outcomes, is the paired final-F1F_1 interaction, (warm−cold)changed−(warm−cold)stable(\mathrm{warm}-\mathrm{cold})_{\mathrm{changed}}-(\mathrm{warm}-\mathrm{cold})_{\mathrm{stable}}. Each condition contributes an average over 240 final agent responses per lineage. We report the mean interaction with a two-sided 95% t11t_{11} interval, conditional on the fixed questions and assuming independent, approximately normal lineage interactions. The remaining analyses—within-regime contrasts, reward, consultation rates, and five-question windows—are exploratory. Appendix § 14 gives the decoding and evaluation settings.

Statistical analysis of the stable-role experiment

We use the lineage as the unit of replication and pair all Layer 2 contrasts within lineages. The primary comparisons are warm minus cold for regret and final F1F_1, evaluated separately for each model; warm minus shuffled and all other metrics are exploratory. We report mean paired differences, two-sided 95% t11t_{11} intervals under an assumption of independent, approximately normal paired differences, and the number of lineages, out of 12, that favor warm. For regret, we express the mean difference and its interval as percentages of the model’s mean cold regret across the 12 lineages.

The intervals are marginal and unadjusted for multiple comparisons, describing variation across the 12 seeds and passage assignments on the fixed question set. Separate API requests may contribute additional generation variability. Although the interaction diagnostics summarize many reads, these reads are selected by their respective policies and the sample size for inference remains 12 lineages.

Across all stable-role conditions, at least 99.98% of responses yielded a valid answer, and all selector replies were parseable. Fewer than 0.4% of responses reached the output limit in each model and layer (Appendix § 13).

Results

We first report the stable-role experiment.

Selector check

In Layer 1, MACE has the lowest pooled hindsight regret on both models (Table 5, Appendix § 11). UCB1 has higher exact match on Qwen3-32B and higher F1F_1, exact match, and realized reward on Gemma-4-31B-it, illustrating the complementary information provided by selection and answer measures. We therefore track both consultation and final answers when evaluating memory transfer in Layer 2.

conditionregretF1F_1EMrewardentropy
Qwen3-32B
mace41.68250.29320.20900.15061.6603
ucb149.37330.28900.21220.14421.7706
pre_defined50.67890.27490.20140.13870.0000
in_context54.82870.25200.18090.12750.4720
random54.36430.27330.19760.13981.7715
Gemma-4-31B-it
mace22.74510.13840.09720.06761.7260
ucb125.45550.14020.09860.06781.7827
pre_defined26.28400.12810.09270.06260.0000
in_context23.47270.13140.09510.06490.0257
random27.14680.12700.08990.06291.7715

Table 5.

Persistent state changes consultation

Warm memory lowers regret relative to cold on both models, with a paired difference of −7.4% of cold regret on Qwen3-32B (95% CI [−12.8, −2.1]; 10 of 12 lineages favor warm) and −6.0% on Gemma-4-31B-it (CI [−11.5, −0.5]; 11 of 12 favor warm). Selection also becomes more concentrated: warm initialization has lower entropy in 11 of 12 lineages for each model. These results show that saved LinUCB state carries experience from the first task block into later choices.

The evidence-holder read trajectories show how these choices evolve over the evaluation block (Figure 2). Counting all selections of evidence-holder IDs, including self-selection, warm Qwen3-32B agents exceed cold agents by 4.3 percentage points in the first half and 5.4 points in the second. For Gemma-4-31B-it, the corresponding gap narrows from 9.6 to 2.2 points.

Warm initialization also increases consultation of other evidence holders across the full run. Measured as a share of all choices, external evidence-holder reads rise by 4.88 percentage points on Qwen3-32B (95% CI [+0.77, +9.00]; 8 of 12 lineages favor warm) and 4.09 points on Gemma-4-31B-it (CI [+1.80, +6.37]; 10 of 12 favor warm). Round-0 answers confirm that evidence holders begin with more useful information than distractor holders (Appendix § 13), giving these consultation changes a direct interpretation in terms of information access.

Final accuracy and realized reward

The Layer 2 means and paired contrasts in Table 1 and Table 2 leave the effect of warm initialization on final accuracy uncertain. With one F1F_1 point equal to 0.01 on the reported scale, the warm-minus-cold estimate is −1.26 points on Qwen3-32B (95% interval [−3.15, +0.62]; 4 of 12 lineages favor warm) and +0.23 points on Gemma-4-31B-it (interval [−0.57, +1.04]; 7 of 12 lineages favor warm). Both intervals include zero, even as consultation shifts consistently toward evidence holders.

conditionregretF1F_1EMrewardentropy
Qwen3-32B
mace_cold53.30620.34880.28680.18161.6203
mace_warm49.35220.33620.27400.17621.5551
mace_shuffled53.29640.33590.28190.17631.5888
random73.52890.31640.25560.16111.7715
Gemma-4-31B-it
mace_cold27.08130.14890.12260.07151.7181
mace_warm25.46460.15130.12120.07361.6741
mace_shuffled26.72940.14410.11490.07021.6886
random32.36490.13820.11280.06681.7715

Table 1. Layer 2 means over 12 lineages. Regret totals are unnormalized and should be compared only within a model and layer. Entropy is descriptive. Bold marks the lowest pooled hindsight regret and the highest F1F_1, EM, and realized reward within each model.

metric and modelcontrastΔ\Delta95% CIwarm favored
Regret, percentage of cold
Qwen3-32Bwarm − cold−7.4[−12.8, −2.1]10/12
Qwen3-32Bwarm − shuffled−7.4[−12.3, −2.5]10/12
Gemma-4-31B-itwarm − cold−6.0[−11.5, −0.5]11/12
Gemma-4-31B-itwarm − shuffled−4.7[−11.2, +1.8]8/12
Final token F1F_1
Qwen3-32Bwarm − cold−0.0126[−0.0315, +0.0062]4/12
Qwen3-32Bwarm − shuffled+0.0003[−0.0181, +0.0187]5/12
Gemma-4-31B-itwarm − cold+0.0023[−0.0057, +0.0104]7/12
Gemma-4-31B-itwarm − shuffled+0.0072[−0.0012, +0.0156]8/12

Table 2. Paired Layer 2 contrasts. Regret is expressed as a percentage of the same model’s cold regret. Intervals are two-sided 95% t11t_{11} intervals. The last column counts lineages in which warm has lower regret or higher F1F_1. Warm minus cold is the primary contrast; warm minus shuffled is exploratory.

Realized reward follows a similar pattern on Qwen3-32B, where the warm-minus-cold difference is −0.0053, with an exploratory 95% interval of [−0.0157, +0.0050]. Together with the accuracy estimates, this result shows how a measurable shift in consultation can coexist with uncertainty in task outcomes.

The shuffled-state comparison further shows a regret reduction on Qwen3-32B: warm minus shuffled is −7.4%, with a CI of [−12.3, −2.5]. Gemma’s regret contrast and both models’ F1F_1 contrasts have intervals that include zero. These contrasts evaluate the partial disruption of role alignment described in Section 4.1.

What happens after an external read?

The interaction traces connect the answers agents consult to the revisions that follow (Table 3). On Qwen3-32B, external reads that offer an answer scoring above the reader’s prior answer increase from 18.69% under cold initialization to 20.92% under warm initialization, a paired difference of +2.24 percentage points with an exploratory interval of [+0.40, +4.08]. Because this opportunity measure compares the two prior answers, it reflects the quality of both reader and peer. The mean consulted-answer F1F_1 difference also has a positive estimate, with an interval that includes zero.

DiagnosticColdWarmΔ\Delta95% CI
Qwen3-32B
Consulted prior F1F_10.37750.3910+0.0135[-0.0139, +0.0409]
Better-answer opportunity (%)18.6920.92+2.24[+0.40, +4.08]
Answer agreement (%)27.3625.57-1.79[-3.58, +0.01]
Revision ΔF1\Delta F_10.02390.0214-0.0026[-0.0084, +0.0033]
Gemma-4-31B-it
Consulted prior F1F_10.16820.1746+0.0064[-0.0089, +0.0217]
Better-answer opportunity (%)15.1814.87-0.31[-1.50, +0.88]
Answer agreement (%)15.7415.67-0.07[-1.24, +1.10]
Revision ΔF1\Delta F_1-0.0027-0.0022+0.0006[-0.0025, +0.0036]

Table 3. Exploratory external-read diagnostics: means of 12 lineage estimates and paired warm-minus-cold 95% t11t_{11} intervals. Opportunity means the peer’s prior F1F_1 exceeds the reader’s. Agreement means the revision equals the peer’s prior answer, conditional on different normalized prior answers. Eligible counts are in Appendix E. Rates use percentages, rate contrasts use percentage points, and F1F_1 uses the 0–1 scale.

Among Qwen reads where reader and peer initially disagree, the revision matches the peer’s answer in 27.36% of eligible cold reads and 25.57% of warm reads, giving a paired difference of −1.79 percentage points with an interval of [−3.58, +0.01]. Such agreement can accompany either a useful correction or adoption of an incorrect answer, so we also measure the change in the reader’s score. Mean F1F_1 improvement per external read is 0.0239 under cold initialization and 0.0214 under warm initialization, with an uncertain paired difference. On Gemma, intervals include zero for changes in better-answer opportunities, answer agreement, and per-read improvement, and the mean revision change is slightly negative in both conditions.

Appendix § 15 complements these averages with token-perfect recovery and damage rates and examples of useful and harmful agreement. Taken together, the diagnostics allow us to follow consultation and revision as separate steps in the collaboration process.

Persistence after evidence reassignment

The primary final-F1F_1 interaction in the Qwen3.5-9B study is −1.38 points, with a 95% t11t_{11} interval of [−3.50, +0.73], leaving the effect of reassignment on warm initialization uncertain (Table 4). Within regimes, the warm-minus-cold estimate is −1.50 points under changed roles, with an interval of [−2.79, −0.20], and −0.11 points under stable roles, with an interval of [−1.92, +1.70]. These within-regime contrasts are exploratory and unadjusted for multiplicity.

Evidence rolesColdWarmWarm–cold [95% CI]
Stable0.2470.246-0.001 [-0.019, +0.017]
Changed0.2530.238-0.015 [-0.028, -0.002]
Changed–stable interaction-0.014 [-0.035, +0.007]

Table 4. Fresh Qwen3.5-9B study: final token F1F_1 means and paired differences across 12 lineages. Intervals use t11t_{11}, conditional on the fixed questions. The changed-minus-stable interaction is the primary target; within-regime contrasts are exploratory. Reassignment changes both readers’ and peers’ information ownership.

Using all choices as the denominator, warm initialization increases external current-evidence consultations by 3.45 percentage points under stable roles, with an exploratory interval of [+0.75, +6.15]. Under changed roles, the corresponding contrast is −1.83 points, with an interval of [−6.02, +2.36], while warm agents self-select 5.00 points more often than cold agents, with an interval of [+3.18, +6.82]. Appendix § 14 extends these analyses with reward, both external-read denominators, all eight planned windows, and response validity.

Discussion

Our results show that remembering peers changes whom agents consult: under stable evidence access, warm initialization lowers pooled hindsight regret and increases consultation of other evidence holders on both models. This consistency indicates that experience from earlier tasks carries forward into later choices. Assessing final answers alongside those choices gives a fuller account of memory’s value, since the final-F1F_1 differences remain uncertain in this workflow.

The interaction traces help connect the consultation results to the answer-only communication protocol, in which each agent holds one passage and receives the selected peer’s extracted answer. Warm Qwen agents more often encounter peer answers that score above their own prior answers, while changes in answer agreement and revision gain remain uncertain. This motivates controlled experiments that vary the information peers share and examine how readers use it.

The reassignment study broadens this evaluation to changing evidence access, using freshly learned memory and paired stable controls to examine what happens when different agents hold useful information. Under reassignment, warm agents self-select more often than cold agents, while the primary interaction leaves uncertain how reassignment changes the accuracy effect. Together with the shuffled-state comparison, this experiment provides a way to assess peer memory as the relationship between agent identity and information access changes.

Limitations

The experiments use six agents that share a model and prompts, exchange answers, and learn from reference-answer feedback after every revision. The reassignment study introduces a different, smaller model and provider and changes each reader’s own evidence access along with peer roles. Evaluating heterogeneous agents, gradual or repeated changes, richer messages, and rewards available without reference answers would establish how broadly the findings extend beyond these conditions.

The observed accuracy, especially on Gemma, reflects the answer-only workflow and may change with stronger reasoning or communication protocols. HotpotQA answers may also appear in pretraining, allowing both passage-based reasoning and memorized knowledge to contribute. Because each study uses the same fixed 80 questions and 12 lineages, its intervals describe variation on those questions; newly sampled questions and additional benchmarks would broaden the evidence.

The interaction diagnostics are observational, with message content, the reader’s prior answer, and generation behavior all potentially affecting a revision. Hindsight regret likewise depends on the interactions each policy selects and cannot estimate how an unchosen peer would have helped in the same situation. Interpretation also depends on generation settings: the stable-role output cap is unknown, and the reassignment study uses a 192-token cap reached by 3.2% of responses. Longer outputs and changes to hosted model versions may affect performance.

Broader impacts

Persistent peer records can preserve useful experience as well as mistaken beliefs, making it important to communicate where a record came from, which peer it describes, and when it should be reevaluated. Deployment also requires attention to privacy and data governance as interaction histories accumulate. Future evaluations should test these practices while tracking answer quality as information access changes.

Conclusion

PeerMem shows how experience with peers carries forward into later collaboration: reusing MACE’s selector state lowers pooled hindsight regret and increases consultation of other evidence holders when roles stay fixed, while the effect on final accuracy remains uncertain. Interaction traces connect these choices to answer revisions, and the reassignment study extends the evaluation to changing evidence access. Together, the findings support evaluating peer memory through the full sequence of consultation, revision, and final task performance.

Acknowledgments and Disclosure of Funding

We thank the Algoverse AI Research program for its mentorship. This work received no funding. The authors declare no relevant competing interests.

Layer 1 selector check

Layer 1 compares selectors on the first 40 questions, before persistence is introduced. These descriptive means characterize the selector used in the memory-transfer experiment. Regret totals use a shared hindsight reference within each layer and model.

Table 5: Layer 1 means over 12 lineages. Bold marks the lowest pooled hindsight regret and the highest F1F_1, EM, and realized reward within each model. Entropy is descriptive.

Additional stable-role contrast visualization

Figure 3. Paired differences with 95% t11t_{11} intervals. Negative regret and positive final F1F_1 favor warm. Three regret intervals exclude zero, while all four final-F1F_1 intervals include zero.

Implementation and reproducibility details

Implementation and code availability. We implement the MACE selector described in its July 2026 paper [3]. We will release the experiment code, configurations, prompt templates, and analysis scripts in a public GitHub repository, including the question-selection and seeded role-assignment procedures. Our selection analysis uses pooled hindsight regret as defined in § 3.3; MACE’s October revision reports an empirical realized-reward gap against Random [4].

Context features. For selecting agent ii, candidate peer jj, and revision round tt, the MACE context is

xi,j,t=[di,j(1),di,j(2),di,j(3),cj(1),cj(2),cj(3),p^i,j,τt,1]T.(3)x_{i,j,t} = \left[d_{i,j}^{(1)}, d_{i,j}^{(2)}, d_{i,j}^{(3)}, c_{j}^{(1)}, c_{j}^{(2)}, c_{j}^{(3)}, \hat{p}_{i,j}, \tau_t, 1\right]^{\mathsf{T}}. \tag*{(3)}

The diversity feature di,j(n)d_{i,j}^{(n)} is one minus the Jaccard similarity between the agents’ nn-gram sets, and cj(n)c_{j}^{(n)} gives peer jj’s normalized share of total response-graph divergence at order nn. Historical reward enters as pi,j=(μi,j+0.5)/1.5p_{i,j} = (\mu_{i,j} + 0.5)/1.5, where μi,j\mu_{i,j} is the pair’s mean reward; the remaining entries are the normalized revision round τt=t/T\tau_t = t/T and a bias. Before scoring, we scale the vector as x←x/max⁡(1,∥x∥2)x \leftarrow x/\max(1,\lVert x\rVert_2). LinUCB initializes Aij=I9A_{ij} = I_9 and bij=0b_{ij} = 0, uses exploration coefficient α=1\alpha= 1, and updates the selected ordered pair after each reward. For NN agents and dd features, these matrices and vectors require O(N2d2)O(N^2d^2) numeric storage, and selection requires no additional language-model call.

Prompt information flow. At revision round 0, each agent receives the question and its assigned passage and returns a concise response inside an answer tag. Subsequent rounds add the agent’s previous answer and the selected peer’s extracted answer to this input. Extraction uses the last closed answer tag, falling back to the trailing text of an unclosed final tag or the last nonempty line when no tag is present. For in_context selection, the model receives each candidate’s selection count, mean reward, and nine-feature context before choosing whose answer to read. Appendix § 16 provides the templates used for these calls.

Models and decoding. We serve Qwen3-32B and Gemma-4-31B-it through Azure Foundry with temperature 0 and top_p left unspecified, keeping the deployment and prompts fixed across conditions within each model. Qwen receives the /no_think prompt suffix, and Gemma receives no suffix. The client defaults to 192 output tokens, although the effective limit used in these runs was not recorded. Qwen runs on a dedicated two-H100 deployment; Gemma’s serving hardware is unspecified.

Experiment size and response length. Each model is evaluated on 12 lineages with six agents and 40 questions per layer, with each agent producing an initial response and three revisions per question. The five Layer 1 conditions require 66,240 model calls per model, including in-context selection, and the four Layer 2 conditions require 46,080, for a total of 224,640 calls across both models. Depending on layer and condition, extracted answers average 2.91–3.14 whitespace-delimited words for Qwen and 4.08–4.48 for Gemma.

Responses reached the output limit in 0.11% and 0.23% of Qwen calls in Layers 1 and 2, respectively, and in 0.39% and 0.28% of Gemma calls. These responses remain in the analysis and pass through the same parser as all other responses.

Question selection and evidence-holder roles. The data loader scans the HotpotQA distractor validation split in its stored order and takes the first 80 hard questions with at least 10 passages, preserving the same question IDs and order across models, lineages, and conditions. Each lineage seed determines which two IDs receive evidence, and this assignment is shared across the lineage’s stable-role conditions.

Evidence-role validation. Round-0 responses confirm that agents assigned evidence begin with substantially more useful information. Across Layer 2 conditions, their F1F_1 ranges from 0.520 to 0.532 on Qwen3-32B and from 0.367 to 0.369 on Gemma-4-31B-it, compared with distractor-holder ranges of 0.151 to 0.157 and 0.051 to 0.052, respectively. These role labels come directly from the passage assignment.

Existing assets and licenses. HotpotQA is distributed under CC BY-SA 4.0 [22].1 The public Qwen3-32B and Gemma-4-31B-it releases carry Apache 2.0 licenses and are cited through their technical reports [13, 7].2 We do not train or release a new model or dataset.

Evidence-role reassignment study

Model and decoding. Together AI serves Qwen/Qwen3.5-9B with advertised FP8 quantization and a 262,144-token total context limit. We set temperature to 0, max_tokens to 192, and enable_thinking=false in chat_template_kwargs, with no suffix added to the prompts. Answering and revision prompts, extraction, self-selection, and MACE updates follow Section 3, and peer selection requires no language-model calls. Every response used in the analysis reports the requested model ID and zero reasoning tokens, although the provider did not supply an immutable served-weight identifier or hardware details. The public model release carries an Apache 2.0 license.3

Protocol and reproducibility. The learning phase uses 12 lineages, 40 questions, six agents, and four generations per agent, yielding 11,520 answer calls; the four evaluation conditions add 46,080 calls, bringing the total to 57,600. Passage assignments, reassignment, and scheduling use fixed seeds, with cold/warm order and stable/changed order balanced across lineages. Conditions run sequentially, with six concurrent agent requests in each synchronous round, and each warm condition receives its own copy of the completed prior. Although API generations may vary even at temperature 0, the configurations, question order, responses, and selector states support replay of scoring and policy updates.

Consultation denominators. Old-holder and current-holder choice rates divide by all revisions and include self-selection. For external current-evidence consultations, we report rates using both all external reads and all choices as denominators, with the latter matching the denominator in the stable-role experiment. Each evaluation condition contains 720 revisions per lineage, and each five-question window contains 30 final answers and 90 revisions; external-read counts depend on the policy’s choices. Across these analyses, the unit of replication remains the 12 lineages.

Response validity and length limits. All 57,600 raw responses used in the analysis are nonempty and none is content-filtered, but the parser returns an empty answer for 25 generations, which remain in the analysis with F1=0F_1 = 0. Invalid-extraction counts for learning, stable cold, stable warm, changed cold, and changed warm are 2, 9, 8, 5, and 1, respectively, out of 11,520 responses per condition. Among the 11,520 evaluation final answers, three are invalid—two in changed cold and one in stable cold—with none in either warm cell. Of all responses, 1,838 (3.19%) reach the length limit, and all receive normal scoring.

Table 6 reports length-limit counts across all generation rounds, showing 19 fewer capped responses in changed warm than in changed cold. Because round-specific and final-answer cap counts are unavailable, the influence of truncation on final scores remains uncertain.

CellLength-limitedGenerationsRate (%)
Learning34011,5202.95
Stable cold34011,5202.95
Stable warm37311,5203.24
Changed cold40211,5203.49
Changed warm38311,5203.32
Total1,83857,6003.19

Table 6.

Table 6: Qwen3.5-9B responses that reached the output limit across 12 lineages and all four generation rounds. Each condition contains 11,520 responses, and all responses remain in the analysis.

Exploratory consultation and reward. Table 7 reports consultation and reward contrasts with marginal, unadjusted t11t_{11} intervals. The warm-minus-cold difference in old-holder choices under changed roles is −1.12-1.12 percentage points, with an interval of [−4.03,+1.78][-4.03,+1.78], and the whole-run reward interaction is −0.0065-0.0065, with an interval of [−0.0185,+0.0055][-0.0185,+0.0055]. Figure 5 traces the consultation contrasts across all eight planned five-question windows.

MetricStable warm–coldChanged warm–cold
Reward-0.000 [-0.010, +0.010]-0.006 [-0.015, +0.002]
External current-E (external reads)+4.587 [+1.941, +7.232]+0.424 [-4.616, +5.463]
External current-E (all choices)+3.449 [+0.745, +6.153]-1.829 [-6.018, +2.361]
Old-holder choices (includes self)+5.370 [+2.790, +7.950]-1.123 [-4.028, +1.782]
Current-holder choices (includes self)+5.370 [+2.790, +7.950]+2.176 [-2.727, +7.079]
Self choices+1.400 [-0.145, +2.946]+5.000 [+3.176, +6.824]

Table 7. Exploratory whole-run paired contrasts with 95% Student intervals. Rate differences are percentage points. External current-E among external reads divides by external reads; its companion divides by all choices. Inclusive holder and self rates also divide by all choices. Retained CSV analysis records contain every numerator, denominator, and eligible lineage.

Figure 5. Exploratory Qwen3.5-9B warm-minus-cold consultation differences over all eight five-question bins, with marginal paired 95% t11t_{11} bands. The top panel divides external current-evidence reads by external reads; the lower panels divide old/current-holder selections by all choices and include self-selection. Stable old and current holders coincide; changed old and current holders are disjoint.

Figure 4. Qwen3.5-9B final-F1F_1 interaction by lineage. Each point is the changed-minus-stable difference in the warm-minus-cold contrast. The summary gives the mean and 95% t11t_{11} interval across 12 lineages, conditional on the fixed evaluation questions.

Interaction diagnostics and illustrative traces

For each Layer 2 revision, we compare the reader’s new answer with its own and the selected peer’s answers from the preceding round, using evidence roles from the seeded passage assignment. Scores are computed against the reference answer, and rewards follow Equation (1).

For consulted-answer quality, better-answer opportunities, and revision change, the external-read denominators are 8,036 cold and 8,007 warm reads on Qwen, and 7,674 cold and 7,564 warm reads on Gemma. Agreement excludes identical normalized prior answers, leaving 4,608 cold and 4,836 warm reads on Qwen, and 4,334 cold and 4,172 warm reads on Gemma. Token-perfect recovery is conditional on prior F1<1F_1 < 1; damage is conditional on prior F1=1F_1 = 1. All comparisons use 12 lineage-level averages.

Illustrative transitions. The examples below show the first recovery and first damage event in each model’s warm condition, ordered by lineage, question, round, and reader. Both examples per model occur in lineage 0; aggregate rates appear in the preceding tables.

The questions are shortened for presentation, and the examples illustrate how agreement can accompany either a correction or an error depending on the answer a peer supplies.

Table 8: External-read token-perfect transitions. Recovery conditions on previous reader F1<1F_1 < 1 and requires revised F1=1F_1 = 1; damage conditions on previous F1=1F_1 = 1 and requires revised F1<1F_1 < 1. NcN_c, NwN_w give eligible cold/warm reads across all lineages. Values are lineage-mean percentages; contrasts and paired 95% t11t_{11} intervals use percentage points. Transitions are defined using token F1F_1.

ModelTransitionNcN_cNwN_wColdWarmΔ\Delta [95% CI]
QwenRecovery588459367.016.76-0.25 [-1.38, +0.88]
QwenDamage2152207110.5411.58+1.04 [-1.64, +3.72]
GemmaRecovery679967231.501.38-0.11 [-0.62, +0.40]
GemmaDamage8758419.379.55+0.18 [-1.90, +2.25]

Table 8.

Table 9: External reads by receiver and selected-peer passage role. NcN_c, NwN_w count eligible reads for revision change; F1F_1 change is on the 0–1 scale. Estimates first average within each lineage. Contrasts pair the 12 lineage estimates with 95% t11t_{11} intervals. Role labels follow the seeded passage assignment.

Receiver →\to peerNc/NwN_c/N_wCold ΔF1\Delta F_1Warm ΔF1\Delta F_1Difference95% CI
Qwen3-32B
Evidence →\to evidence939/11390.03560.0251-0.0105[-0.0261, +0.0052]
Evidence →\to distractor1702/1502-0.0190-0.0219-0.0030[-0.0167, +0.0108]
Distractor →\to evidence3056/32780.05500.0582+0.0033[-0.0099, +0.0165]
Distractor →\to distractor2339/20880.0054-0.0072-0.0126[-0.0282, +0.0030]
Gemma-4-31B-it
Evidence →\to evidence686/7730.00150.0007-0.0007[-0.0098, +0.0084]
Evidence →\to distractor1774/1531-0.0316-0.0293+0.0023[-0.0055, +0.0101]
Distractor →\to evidence2302/25680.02260.0213-0.0013[-0.0081, +0.0055]
Distractor →\to distractor2912/2692-0.0065-0.0108-0.0043[-0.0099, +0.0012]

Table 9.

Table 10: Deterministically selected warm-condition examples. The first arrow links the reader’s previous answer to its revision; the peer column gives the selected previous-round answer. E and D indicate actual evidence and distractor roles. Each recovery changes F1F_1 from 0 to 1; each damage example changes it from 1 to 0.

ModelQuestion and reader transitionPeer answer and rolesReference
QwenWhat coastal area borders Dirleton’s medieval fortress?<br>North Sea →\to YellowcraigYellowcraig<br>reader D, peer EYellowcraig
QwenWhere is Sachin Warrier’s former software employer headquartered?<br>Mumbai →\to UnknownUnknown<br>reader E, peer DMumbai
GemmaWhich Midwest track hosts a 500-mile race each May?<br>I do not have enough information to answer this question. →\to Indianapolis Motor SpeedwayIndianapolis Motor Speedway<br>reader D, peer EIndianapolis Motor Speedway
GemmaAre Ferocactus and Silene both types of plant?<br>Yes →\to Insufficient informationInsufficient information<br>reader D, peer Eyes

Table 10.

Prompt templates

The stable-role experiments use the following templates, with one system message and one user message in each model call. Qwen appends /no_think to each user message, while Gemma uses no suffix. For each call, the braced fields are filled with the corresponding question, passage, responses, or candidate information.

Discussion system message.

You are one of several agents answering a question. Each agent has been given a different
piece of context, and no single piece is guaranteed to be relevant.

Initial user message.

Context:
{context}
Question: {question}
First, briefly reason over the provided context. Then, place your final answer between
<answer> and </answer> tags. Keep the answer concise - a word, phrase, or short sentence.
Example: <answer>Paris</answer>

Revision user message.

Context:
{context}
Question: {question}
YOUR PREVIOUS RESPONSE:
{own_previous}
RESPONSE FROM {peer_name}:
{peer_previous}
INSTRUCTIONS:
Review your previous response and the peer's response carefully. You may agree, disagree, or
selectively incorporate the peer's reasoning. Then provide your updated, complete
response to the QUESTION.
First, briefly reason over the provided context. Then, place your final answer between
<answer> and </answer> tags. Keep the answer concise - a word, phrase, or short sentence.
Example: <answer>Paris</answer>

The context contains the title and text of the agent’s assigned passage, and both previous-response fields contain extracted answers. Thus, although the revision instructions invite agents to incorporate a peer’s reasoning, the information supplied excludes the peer’s generated reasoning and source passage.

In-context selector system message.

You are selecting which peer's response to read.

In-context selector user message. The user message starts with the following lines:

QUESTION: {question}

HISTORY:

One row follows for each of the six candidates, including the selecting agent itself:

Agent {k} [chosen {n_k} times, mean reward: {mu_k} | features: [Diversity (unigram): {x_1},
Diversity (bigram): {x_2}, Diversity (trigram): {x_3}, Centrality (unigram): {x_4},
Centrality (bigram): {x_5}, Centrality (trigram): {x_6}, Hist. Reward: {x_7}, Norm. Round:
{x_8}, Bias: {x_9}]]

Each row summarizes the agent’s previous selections of that candidate through a selection count and mean reward. The nine features describe the candidate’s answer diversity relative to the selecting agent and its centrality in the response graph, each measured with unigrams, bigrams, and trigrams. They also include the pair’s historical reward, the revision round, and a bias term. We rescale the historical-reward feature and normalize the feature vector, while leaving the separately displayed mean reward unscaled. After the candidate rows, the message ends with: Based on the historical information, whose full response would you like to read to help refine your answer? You may also choose your own. → Reply with only the agent number (1 to 6).

This LLM selector is used in Layer 1, where the selected candidate’s extracted answer is passed to the revision.

Shuffled-state control

For each selecting agent, the shuffled condition keeps the self record in place and assigns each of the other five records to a different peer ID, using a permutation determined by the lineage seed and selecting-agent ID. All learned statistics move together, after which online updates continue under the new assignment.

The shuffle changes peer identities while preserving some alignment with evidence access: across 12 lineages, 23 of 120 non-self records learned about evidence holders move to another evidence-holder ID. Together with the 24 fixed self records, this leaves 47 of 144 evidence-origin records (32.6%) assigned to evidence holders, defining the degree of role disruption tested by the control.

A follow-up test could isolate non-self attribution by resetting self records in both conditions and comparing correctly indexed state with a role-aware permutation that moves every evidence-origin record to a distractor ID and gives every evidence-holder ID a distractor-origin record. With two evidence holders and four distractors, the remaining transfers would occur between distractors.

NeurIPS Paper Checklist

Claims

Question: Do the main claims made in the abstract and introduction accurately reflect the paper’s contributions and scope?

Answer: [Yes]

Justification: The Abstract and Introduction distinguish the stable-role study from the fresh-provider reassignment study, describe consultation separately from answer quality, and retain uncertainty in the primary interaction.

Guidelines:

  • The answer [N/A] means that the abstract and introduction do not include the claims made in the paper.

  • The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A [No] or [N/A] answer to this question will not be perceived well by the reviewers.

  • The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.

  • It is fine to include aspirational goals as motivation as long as it is clear that these goals are not attained by the paper.

Limitations

Question: Does the paper discuss the limitations of the work performed by the authors?

Answer: [Yes]

Justification: Section 7 discusses the stable and reassigned evidence settings, answer-only exchange, low absolute accuracy, possible benchmark memorization, supervised reference-answer feedback, limited replication, observational interaction diagnostics, exploratory analyses, and the limits of hindsight regret.

Guidelines:

  • The answer [N/A] means that the paper has no limitation while the answer [No] means that the paper has limitations, but those are not discussed in the paper.

  • The authors are encouraged to create a separate “Limitations” section in their paper.

  • The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.

  • The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with a few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.

  • The authors should reflect on the factors that influence the performance of the approach. For example, a facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or a speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.

  • The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.

  • If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.

  • While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren’t acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an important role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.

Theory assumptions and proofs

Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?

Answer: [N/A]

Justification: The paper contains no theoretical results, theorems, or proof claims.

Guidelines:

  • The answer [N/A] means that the paper does not include theoretical results.

  • All the theorems, formulas, and proofs in the paper should be numbered and cross-referenced.

  • All assumptions should be clearly stated or referenced in the statement of any theorems.

  • The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch to provide intuition.

  • Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.

  • Theorems and Lemmas that the proof relies upon should be properly referenced.

Experimental result reproducibility

Question: Does the paper fully disclose all the information needed to reproduce the main experimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?

Answer: [No]

Justification: The paper describes the experimental protocol, prompt templates, and analyses, although the effective output-token limit for the stable-role runs and the exact hosted model weights are unavailable.

Guidelines:

  • The answer [N/A] means that the paper does not include experiments.

  • If the paper includes experiments, a [No] answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.

  • If the contribution is a dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.

  • Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.

  • While NeurIPS does not require releasing code, the conference does require all submissions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example (a) If the contribution is primarily a new algorithm, the paper should make it clear how to reproduce that algorithm. (b) If the contribution is primarily a new model architecture, the paper should describe the architecture clearly and fully. (c) If the contribution is a new model (e.g., a large language model), then there should either be a way to access this model for reproducing the results or a way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset). (d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reproducibility.

In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.

Open access to data and code

Question: Does the paper provide open access to the data and code, with sufficient instructions to faithfully reproduce the main experimental results, as described in supplemental material?

Answer: [No]

Justification: HotpotQA is publicly available and cited. We plan to release the experiment code, configurations, and analysis scripts in a public GitHub repository; public code access is pending that release.

Guidelines:

  • The answer [N/A] means that paper does not include experiments requiring code.

  • Please see the NeurIPS code and data submission guidelines (https://neurips.cc/public/guides/CodeSubmissionPolicy) for more details.

  • While we encourage the release of code and data, we understand that this might not be possible, so [No] is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for a new open-source benchmark).

  • The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines (https://neurips.cc/public/guides/CodeSubmissionPolicy) for more details.

  • The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.

  • The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only a subset of experiments are reproducible, they should state which ones are omitted from the script and why.

  • At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).

  • Providing as much information as possible in supplemental material (appended to the paper) is recommended, but including URLs to data and code is permitted.

Experimental setting/details

Question: Does the paper specify all the training and test details (e.g., data splits, hyperparameters, how they were chosen, type of optimizer) necessary to understand the results?

Answer: [Yes]

Justification: The experiments involve no model training. Sections 3 and 4, together with Appendices C and D, specify the data split, agents, conditions, parameters, prompt information flow, model identifiers, decoding, and analysis needed to understand the results.

Guidelines:

  • The answer [N/A] means that the paper does not include experiments.

  • The experimental setting should be presented in the core of the paper to a level of detail that is necessary to appreciate the results and make sense of them.

  • The full details can be provided either with the code, in appendix, or as supplemental material.

Experiment statistical significance

Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?

Answer: [Yes]

Justification: Section 4 defines the stable-role and role-reassignment lineage-level analyses and marginal two-sided 95% t11t_{11} intervals, and Section 5 reports the corresponding estimates and replication counts.

Guidelines:

  • The answer [N/A] means that the paper does not include experiments.

  • The authors should answer [Yes] if the results are accompanied by error bars, confidence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.

  • The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).

  • The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)

  • The assumptions made should be given (e.g., Normally distributed errors).

  • It should be clear whether the error bar is the standard deviation or the standard error of the mean.

  • It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have a 96% CI, if the hypothesis of Normality of errors is not verified.

  • For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out of range (e.g., negative error rates).

  • If error bars are reported in tables or plots, the authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.

Experiments compute resources

Question: For each experiment, does the paper provide sufficient information on the computer resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?

Answer: [No]

Justification: Appendix C reports experiment size and the two-H100 Qwen deployment. Execution time and the serving hardware for Gemma and Together are unspecified.

Guidelines:

  • The answer [N/A] means that the paper does not include experiments.

  • The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.

  • The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.

  • The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn’t make it into the paper).

Code of ethics

Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https://neurips.cc/public/EthicsGuidelines?

Answer: [Yes]

Justification: Sections 3 and 7, together with the Broader Impacts section, document the public-benchmark study, its limitations, and its risks. No human participants or private interaction histories were collected.

Guidelines:

  • The answer [N/A] means that the authors have not reviewed the NeurIPS Code of Ethics.

  • If the authors answer [No], they should explain the special circumstances that require a deviation from the Code of Ethics.

  • The authors should make sure to preserve anonymity (e.g., if there is a special consideration due to laws or regulations in their jurisdiction).

Broader impacts

Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?

Answer: [Yes]

Justification: Section 8 discusses mistaken peer records, record provenance and reevaluation, and privacy and data governance when interaction histories persist. Future work can evaluate these deployment practices.

Guidelines:

  • The answer [N/A] means that there is no societal impact of the work performed.

  • If the authors answer [N/A] or [No], they should explain why their work has no societal impact or why the paper does not address societal impact.

  • Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.

  • The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate Deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.

  • The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.

  • If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML).

Safeguards

Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pre-trained language models, image generators, or scraped datasets)?

Answer: [N/A]

Justification: The work releases no new dataset or model and does not introduce an artifact with a high risk of misuse.

Guidelines:

  • The answer [N/A] means that the paper poses no such risks.

  • Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.

  • Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.

  • We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make a best faith effort.

Licenses for existing assets

Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?

Answer: [Yes] Justification: The paper cites HotpotQA and all three model releases. Appendices C and D state the CC BY-SA 4.0 dataset license and public Apache 2.0 model licenses.

Guidelines:

  • The answer [N/A] means that the paper does not use existing assets.

  • The authors should cite the original paper that produced the code package or dataset.

  • The authors should state which version of the asset is used and, if possible, include a URL.

  • The name of the license (e.g., CC-BY 4.0) should be included for each asset.

  • For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.

  • If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswithcode.com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of a dataset.

  • For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.

  • If this information is not available online, the authors are encouraged to reach out to the asset’s creators.

New assets

Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?

Answer: [N/A]

Justification: A public code release is planned, but no new assets are publicly released with the current manuscript.

Guidelines:

  • The answer [N/A] means that the paper does not release new assets.

  • Researchers should communicate the details of the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc.

  • The paper should discuss whether and how consent was obtained from people whose asset is used.

  • At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.

Crowdsourcing and research with human subjects

Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?

Answer: [N/A]

Justification: The paper involves neither crowdsourcing nor research with human subjects.

Guidelines:

  • The answer [N/A] means that the paper does not involve crowdsourcing nor research with human subjects.

  • Including this information in the supplemental material is fine, but if the main contribution of the paper involves human subjects, then as much detail as possible should be included in the main paper.

  • According to the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.

15. Institutional review board (IRB) approvals or equivalent for research with human subjects

Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?

Answer: [N/A]

Justification: The paper involves no research with human subjects, so IRB or equivalent review is not applicable.

Guidelines:

  • The answer [N/A] means that the paper does not involve crowdsourcing nor research with human subjects.

  • Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.

  • We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.

  • For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.

16. Declaration of LLM usage

Question: Does the paper describe the usage of LLMs if it is an important, original, or non-standard component of the core methods in this research? Note that if the LLM is used only for writing, editing, or formatting purposes and does not impact the core methodology, scientific rigor, or originality of the research, declaration is not required.

Answer: [Yes]

Justification: Sections 3 and 4, with Appendices C and D, describe Qwen3-32B, Gemma-4-31B-it and Qwen3.5-9B as answering agents and the LLM-based in-context selector.

Guidelines:

  • The answer [N/A] means that the core method development in this research does not involve LLMs as any important, original, or non-standard components.

  • Please refer to our LLM policy in the NeurIPS handbook for what should or should not be described.

References

  1. [1]Suhaib Abdurahman, Etsuko Ishii, Katerina Margatina, Divya Bhargavi, Monica Sunkara, and Yi Zhang. Explicit trait inference for multi-agent coordination. In Proceedings of the 64th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 1670–1704. Association for Computational Linguistics, July 2026. doi: 10.18653/v1/2026.acl-long.77. URL https://aclanthology.org/2026.acl-long.77/.DOI
  2. [2]Mert Cemri, Melissa Z. Pan, Shuyi Yang, Lakshya A. Agrawal, Bhavya Chopra, Rishabh Tiwari, Kurt Keutzer, Aditya Parameswaran, Dan Klein, Kannan Ramchandran, Matei Zaharia, Joseph E. Gonzalez, and Ion Stoica. Why do multi-agent LLM systems fail? arXiv preprint arXiv:2503.13657, 2025. URL https://arxiv.org/abs/2503.13657.
  3. [3]Hyeong Kyu Choi, Jiatong Li, Wendi Li, Xin Eric Wang, and Sharon Li. Multi-agent LLMs fail to explore each other. arXiv preprint arXiv:2607.11250v1, July 2026a. URL https://arxiv.org/abs/2607.11250v1.
  4. [4]Hyeong Kyu Choi, Jiatong Li, Wendi Li, Xin Eric Wang, and Sharon Li. Do multi-agent LLMs explore each other? arXiv preprint arXiv:2607.11250v2, October 2026b. URL https://arxiv.org/abs/2607.11250v2.
  5. [5]Yilun Du, Shuang Li, Antonio Torralba, Joshua B. Tenenbaum, and Igor Mordatch. Improving factuality and reasoning in language models through multiagent debate. In International Conference on Machine Learning (ICML), 2024. URL https://arxiv.org/abs/2305.14325.
  6. [6]Peilin Feng, Suorong Yang, and Soujanya Poria. Σ-Mem: An online reliability memory for LLM-based multi-agent systems. arXiv preprint arXiv:2607.27958v1, July 2026. URL https://arxiv.org/abs/2607.27958v1.
  7. [7]Gemma Team. Gemma 4 technical report, 2026. URL https://arxiv.org/abs/2607.02770.
  8. [8]Taicheng Guo, Xiuying Chen, Yaqi Wang, Ruidi Chang, Shichao Pei, Nitesh V. Chawla, Olaf Wiest, and Xiangliang Zhang. Large language model based multi-agents: A survey of progress and challenges. arXiv preprint arXiv:2402.01680, 2024. URL https://arxiv.org/abs/2402.01680.
  9. [9]Audun Jøsang and Roslan Ismail. The beta reputation system. In Proceedings of the 15th Bled Electronic Commerce Conference, 2002.
  10. [10]Lihong Li, Wei Chu, John Langford, and Robert E. Schapire. A contextual-bandit approach to personalized news article recommendation. In Proceedings of the 19th International Conference on World Wide Web (WWW), pages 661–670, 2010. doi: 10.1145/1772690.1772758. URL https://doi.org/10.1145/1772690.1772758.
  11. [11]Yuwei Lou, Hao Hu, Shaocong Ma, Zongfei Zhang, Liang Wang, Jidong Ge, and Xianping Tao. DRF: LLM-AGENT dynamic reputation filtering framework. In Neural Information Processing (ICONIP 2025), volume 16312 of Lecture Notes in Computer Science, pages 127–141. Springer, 2026. doi: 10.1007/978-981-95-4384-7_10. URL https://doi.org/10.1007/978-981-95-4384-7_10.
  12. [12]Chuhan Qiao. CADMAS-CTX: Contextual capability calibration for multi-agent delegation. arXiv preprint arXiv:2604.17950v1, April 2026. URL https://arxiv.org/abs/2604.17950v1.
  13. [13]Qwen Team. Qwen3 technical report, 2025. URL https://arxiv.org/abs/2505.09388.
  14. [14]Qwen Team. Qwen3.5: Towards native multimodal agents, February 2026. URL https://qwen.ai/blog?id=qwen3.5.
  15. [15]Pranav Rajpurkar, Jian Zhang, Konstantin Lopyrev, and Percy Liang. SQuAD: 100,000+ questions for machine comprehension of text. In Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 2383–2392. Association for Computational Linguistics, 2016. doi: 10.18653/v1/D16-1264. URL https://aclanthology.org/D16-1264/.DOI
  16. [16]Noah Shinn, Federico Cassano, Ashwin Gopinath, Karthik Narasimhan, and Shunyu Yao. Reflection: Language agents with verbal reinforcement learning. In Advances in Neural Information Processing Systems (NeurIPS), 2023. URL https://proceedings.neurips.cc/paper_files/paper/2023/hash/1b44b878bb782e6954cd888628510e90-Abstract-Conference.html.arxiv.org/abs/2303.11366
  17. [17]Pannagadatta Shivaswamy and Thorsten Joachims. Multi-armed bandit problems with history. In Proceedings of the Fifteenth International Conference on Artificial Intelligence and Statistics (AISTATS), volume 22 of Proceedings of Machine Learning Research, pages 1046–1054, 2012.
  18. [18]Andries Petrus Smit, Nathan Grinsztajn, Paul Duckworth, Thomas D. Barrett, and Arnu Pretorius. Should we be going MAD? A look at multi-agent debate strategies for LLMs. In International Conference on Machine Learning (ICML), pages 45883–45905, 2024. URL https://proceedings.mlr.press/v235/smit24a.html.arxiv.org/abs/2311.17371
  19. [20]Zora Zhiruo Wang, Jiayuan Mao, Daniel Fried, and Graham Neubig. Agent workflow memory. arXiv preprint arXiv:2409.07429, 2024. URL https://arxiv.org/abs/2409.07429.
  20. [21]Qingyun Wu, Gagan Bansal, Jieyu Zhang, Yiran Wu, Beibin Li, Erkang Zhu, Li Jiang, Xiaoyun Zhang, Shaokun Zhang, Jiale Liu, Ahmed Hassan Awadallah, Ryen W White, Doug Burger, and Chi Wang. AutoGen: Enabling next-gen LLM applications via multi-agent conversation. arXiv preprint arXiv:2308.08155, 2023. URL https://arxiv.org/abs/2308.08155.
  21. [22]Zhilin Yang, Peng Qi, Saizheng Zhang, Yoshua Bengio, William W. Cohen, Ruslan Salakhutdinov, and Christopher D. Manning. HotpotQA: A dataset for diverse, explainable multi-hop question answering. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 2369–2380, 2018. doi: 10.18653/v1/D18-1259. URL https://aclanthology.org/D18-1259/.DOI
  22. [23]Chicheng Zhang, Alekh Agarwal, Hal Daumé, III, John Langford, and Sahand Negahban. Warm-starting contextual bandits: Robustly combining supervised and bandit feedback. In Proceedings of the 36th International Conference on Machine Learning, volume 97 of Proceedings of Machine Learning Research, pages 7335–7344. PMLR, 2019. URL https://proceedings.mlr.press/v97/zhang19b.html.arxiv.org/abs/1901.00301
  23. [24]Guibin Zhang, Muxin Fu, Kun Wang, Frank Wan, Miao Yu, and Shuicheng Yan. G-Memory: Tracing hierarchical memory for multi-agent systems. In Advances in Neural Information Processing Systems, volume 38, 2025. doi: 10.52202/085713-0439. URL https://papers.neurips.cc/paper_files/paper/2025/hash/136a45cd9b841bf785625709a19c6508-Abstract-Conference.html.DOI
  24. [25]Andrew Zhao, Daniel Huang, Quentin Xu, Matthieu Lin, Yong-Jin Liu, and Gao Huang. ExpeL: LLM agents are experiential learners. In Proceedings of the AAAI Conference on Artificial Intelligence, 2024. URL https://arxiv.org/abs/2308.10144.

Paper details

vlnc:W_fy23y4p7
Version
v1 · latest
Published
Fresh — added to the Hub within the last 30 days
Added
License
CC BY 4.0
Usage

views · 0

downloads · 0

A view counts once per browser per day, a download each time the PDF is requested.

  • Computer Science

Not recorded for this paper

View statement
Looking for related papers…
Contents