betaFeedback

WorksComputer Science

Turning Multi-Agent Communication into Token-Level Credit with On Policy Self Distillation

Abstract

Reinforcement learning improves problem solving in large language model multi-agent systems. Existing approaches overlook communication knowledge or compress it into scalar rewards, leaving coarse credit with little guidance on which decisions each agent should reinforce or revise. OPSD4MAS converts future communication into fine-grained token-level credit through On Policy Self Distillation (OPSD). An external model extracts facts and assesses their trajectory relations for coarse-grained filtering, followed by fine-grained leave-one-out filtering of target-token influence against agent-wise outcome advantages. The fact-conditioned policy supplies token-level credit, combined with GRPO through a linearly annealed joint objective. Our method improves mathematical pass@4 over GRPO by up to 20.00 percentage points. Ablations support complete filtering, while frozen-history injection diagnostics support fact utility. Together, these results support combining agent-level outcome guidance with communication-derived token-level credit. Code is available at https://github.com/Camel-Prince/OPSD4MAS.

Introduction

Reinforcement learning (RL) improves large language model (LLM) reasoning by updating policies from feedback on the quality of their generated answers (Shao et al., 2024; Yu et al., 2025). Extending RL to LLM-based multi-agent systems (MAS) trains interacting agents to reason and communicate for joint success ([3] (Liu et al. 2025)). Most approaches derive scalar rewards from task outcomes or agent communication, assign credit to participating agents, and update

their policies to improve multi-turn reasoning and coordination within each task. Many recent MAS training methods follow scalar-reward RL. Methods such as MAGRPO, M-GRPO, and MARTI turn task outcomes, intermediate evaluations, or judgments of agent interactions into scalar rewards for policy updates ([6] (Liu et al. 2025; Zhang et al. 2026a)). Figure 1(a) contrasts two common routes. Outcome-derived rewards assign task-success signals to responses, as in the agent-wise advantage normalization of DrMAS ([3]). Communication-informed rewards additionally use signals from discussions, as in MAPoRL and CoMAS, while retaining scalar feedback (Park et al., 2025; Xue et al., 2026). These scalar-reward approaches either leave communication-derived knowledge unused in reward construction or compress it into scores, losing a valuable source of learning signal. For an intermediate response in a MAS, future communication can explain an error, validate a claim, or suggest a repair. Scalar rewards also yield coarse credit assignment: a response-level advantage indicates whether to reinforce a response, but cannot itself distinguish which tokens a correction supports or challenges, even when token gradients differ.

From scalar to token-level credit and test performance

Figure 1. (a) Two scalar-reward RL routes versus our RLVR–OPSD combination. (b) Test pass@1/pass@4 for CoMAS, DrMAS (GRPO/DAPO), and OPSD4MAS. FW2/FW4 denote OPSD4MAS with future windows of two/four responses, both trained with OPSD and GRPO.

To address this limitation, we propose OPSD4MAS, which converts future communication into token-level credit using On Policy Self Distillation (OPSD), in which the current policy conditioned on selected future information acts as a privileged teacher for the same policy under its ordinary context. Comparing their probabilities on student-generated tokens supplies token-level credit, as shown in Figure 1(a), incorporating communication-derived knowledge into agent parameters without future information at inference time.

The key to OPSD4MAS is constructing useful privileged context. Unfiltered communication mixes useful corrections with errors and irrelevant remarks. We therefore introduce Fact Extraction and Hierarchical Filtering (FEHF). For each target response, an external model extracts facts from its future communication window and labels their relations to same-query trajectories. Association filtering ranks candidate facts by their relevance to solving the current query, assessed from whether following or violating each fact aligns with the outcome advantages of trajectories for that query. Facts followed in higher-advantage trajectories or violated in lower-advantage trajectories receive higher scores. Leave-one-out (LOO) filtering then checks whether each candidate fact changes the target response’s token probabilities in the direction favored by that response’s GRPO outcome advantage. Retained facts provide privileged context without external teacher logits. We combine OPSD with reinforcement learning with verifiable rewards (RLVR), using GRPO and linearly decreasing the OPSD loss weight.

Across six benchmarks in Table 1, OPSD4MAS with a future window of four responses (FW4) improves mean pass@1 by 8.87 and 6.14 percentage points over GRPO and DAPO, respectively. Mean pass@4 gains are 9.54 and 8.48 percentage points. Figure 1(b) shows three representative datasets. We also inject selected facts into frozen histories and regenerate the final Solver answer. With Relaxed filtering, accuracy exceeds no injection by 1.98–6.13 percentage points across four checkpoints in Figure 4. The training ablations in Table 2 compare raw communication, extracted facts, and the two filtering stages. Extraction, Association, and LOO improve mean pass@1 over their preceding ablated settings by 9.33, 0.42, and 21.92 percentage points, respectively, with the largest gain from target-specific LOO.

MethodAMC23 p1p_1AMC23 p4p_4AIME24 p1p_1AIME24 p4p_4AIME25 p1p_1AIME25 p4p_4MATH-500 p1p_1MATH-500 p4p_4Olympiad p1p_1Olympiad p4p_4Minerva p1p_1Minerva p4p_4
Base Model60.2474.7025.8343.3319.1723.3382.4090.4041.0453.0424.7231.99
Frozen-weight MAS: no LLM parameter updates
Solver–Verifier MAS56.6371.0822.5036.6716.6726.6783.5091.8046.8159.2633.2741.18
AFlow61.4477.1127.5046.6722.5026.6779.1588.6045.4458.0730.6139.34
GPTSwarm61.4474.7020.8336.6720.8330.0080.7589.6044.3354.2224.7234.19
Scalar-reward RL: sparse credit assignment
DrMAS-GRPO68.0783.1348.3363.3332.5053.3390.9594.6056.1168.3036.7642.65
DrMAS-DAPO78.0185.5446.6766.6738.3353.3390.3594.4058.9368.0036.8643.75
CoMAS71.2981.9343.3360.0036.6750.0090.9094.2056.5968.4436.4041.18
Ours: token-level credit assignment
OPSD4MAS (FW2)74.1087.9550.8370.0041.6773.3389.0095.2055.5967.1138.4247.06
OPSD4MAS (FW4)82.2391.5762.5083.3350.0073.3391.5096.6060.3069.9339.4347.79

Table 1. Test performance (%): p1/p4p_1/p_4 denote pass@1/pass@4. Bold underlined and italic underlined values rank first and second in each column. All nine methods are evaluated on six datasets. Olympiad denotes English text-only OlympiadBench mathematics. Evaluation protocols and scoring qualifications appear in § 5.1 and Appendix § 10.§ 1.

ConditionEEAALLp1p_1p4p_4
No OPSD–––49.6366.60
Raw Commun.–––33.2450.46
Unfiltered Facts✓––42.5763.97
Coarse-filtered Facts✓✓–42.9965.18
Full✓✓✓64.9182.74

Table 2. Training ablations: mean pass@1/pass@4 (p1/p4p_1/p_4, %) on AMC23, AIME24, and AIME25, with equal dataset weights. Raw Commun. abbreviates Raw future communication. E/A/LE/A/L: extraction/Association/LOO. No OPSD and Full reuse GRPO and FW4 from Table 1.

Figure 4. Filtering-strength trend (60–90% axis): Relaxed is best at all checkpoints. None is the no-injection reference, and Strict through All relax LOO. Category spacing does not encode fact quantity.

Overall, our contributions are threefold:

  • We turn communication into token-level credit. We identify knowledge lost or compressed by scalar-reward MAS training and use future communication as privileged context for token-level credit.

  • We develop fact-guided joint OPSD–RLVR training. FEHF combines fact extraction with coarse- and fine-grained filtering. The fact-guided teacher supplies OPSD credit alongside RLVR, instantiated with GRPO, under a linearly annealed joint objective.

  • We show performance gains and downstream fact utility. OPSD4MAS (FW4) improves both metrics across six benchmarks over all three trained baselines, with gains up to 20.00 percentage points. Ablations and frozen-history injection diagnostics support the filtering design and the downstream utility of selected facts.

Related Work

Multi-agent systems. LLM-based MAS coordinate specialized agents through role-based communication, debate, and shared task execution ([10, 2] (Wu et al. 2023)). Beyond manually specified interactions, AFlow and ADAS search for effective agent organizations ([7] (Zhang et al. 2025)). These approaches address how agents collaborate. A complementary question is how post-training can improve the policies participating in that collaboration.

Reinforcement learning for multi-agent systems. RL post-training improves LLM policies using feedback on generated answers. GRPO, RLOO, and DAPO offer policy-gradient approaches with different advantage estimators and optimization designs [1] (Shao et al. 2024; Yu et al. 2025). Extending reward-based learning to interacting agents introduces the problem of assigning credit across roles and turns. MAGRPO optimizes cooperative policies using shared rewards, while DrMAS normalizes outcome rewards with agent-specific statistics [3] (Liu et al. 2025). To account for collaboration, MA-PoRL shapes verifier rewards using changes in answer correctness and influence across discussions, while CoMAS derives scalar action rewards from interaction-based judgments (Park et al., 2025; Xue et al., 2026). However, these methods reduce task feedback to scalar rewards rather than use communication content directly for task-specific token-level credit. OPSD4MAS addresses this gap by conditioning a privileged teacher on selected communication while retaining outcome credit.

On Policy Self Distillation. On Policy Distillation (OPD) is an emerging LLM post-training paradigm that queries a teacher on student-generated trajectories, providing token-level learning signals on states visited by the student (Song and Zheng, 2026). In On Policy Self Distillation (OPSD), the student model or a copy acts as teacher with additional information unavailable to the ordinary-context student. Related methods differ in their teachers and privileged information. OPCD studies frozen and shared-weight context teachers (Ye et al., 2026). MOPD uses successful and failed same-question rollouts with self or external teachers (Yu et al., 2026). GEAR uses a ground-truth-conditioned self-model to reweight outcome advantages, not a direct distillation loss [11]. TurnOPD uses a frozen external teacher on student-visited multi-turn histories (Zhou et al., 2026). EDGE-OPD masks updates by privileged-context evidence, whereas AR-OPD controls full-view residual guidance around a partial-privilege anchor [8] (Zhang 2026). Our focus is selecting reliable, target-specific facts from mixed-quality future communication, rather than directly adopting complete answers or alternative rollouts as privileged context.

OPD for MAS. Recent work applies OPD to multi-agent learning. MAS-OPD supplies a frozen external teacher with verifier-produced attributions of past-turn conflicts (Zhong et al., 2026). π\pi-Play conditions an EMA self-teacher on an examiner’s question-construction paths (Zhang et al., 2026b). OPSD4MAS instead extracts facts from future inter-agent communication during task solving. Cross-trajectory association and target-specific LOO filtering select these facts as privileged context for each role’s privileged teacher.

Problem Formulation

A language-model multi-agent system (MAS) coordinates communicating agents through a predefined workflow, graph, or autonomous decisions. In the role-specific setting, each role r∈Rr \in\mathcal{R} has parameters θr\theta_r, and Θ={θr:r∈R}\Theta= \{\theta_r : r \in\mathcal{R}\} collects all trainable parameters. We write πθr=πθr\pi_\theta^r = \pi_{\theta_r} for its policy. For query qq, index sampled trajectories by jj. Stochastic execution gives

τj=MAS⁡Θ(q),τj=(yj,1,…,yj,Tj).(1)\tau_j = \operatorname{MAS}_{\Theta}(q), \qquad\tau_j = (y_{j,1}, \ldots, y_{j,T_j}). \tag*{(1)}

Response yj,iy_{j,i} is generated by role r(i)r(i), which can act repeatedly. Token tt has context (q,yj,<i,yj,i,<t)(q, y_{j,<i}, y_{j,i,<t}) under shared history. Restricted communication exposes only the permitted prefix. MAS RL updates all participating agents’ parameters, not only the final answering agent. With outcome reward R(T)R(\mathcal{T}), the objective is

max⁡Θ Eq,τ∼MAS⁡Θ(q)[R(τ)].(2)\max_{\Theta}\ \mathbb{E}_{q,\tau\sim\operatorname{MAS}_{\Theta}(q)}[R(\tau)]. \tag*{(2)}

DrMAS [3] normalizes outcome rewards by agent within each query group to obtain

Aj,ir(i)=sj,i−μq,r(i)σq,r(i)+ϵ.(3)A_{j,i}^{r(i)} = \frac{s_{j,i} - \mu_{q,r(i)}}{\sigma_{q,r(i)} + \epsilon}. \tag*{(3)}

Here sj,is_{j,i} is the outcome reward assigned to response yj,iy_{j,i} in GRPO training. Mean μ\mu and standard deviation σ\sigma pool all same-agent responses across trajectories for query qq, counting repeated turns separately. This agent-wise advantage weights every valid response token in GRPO’s clipped surrogate. Communication-informed CoMAS instead optimizes interaction-derived action rewards with REINFORCE++ and a clipped surrogate (Xue et al., 2026). Its parameters index agents that can perform different interaction roles. Appendix A details their normalization and KL terms.

Method

Motivation

Scalar-reward MAS training discards or compresses communication knowledge useful for collaboration. We use future communication as privileged context for token-level OPSD credit. Mixing corrections with errors and irrelevant remarks motivates fact extraction and two filtering stages. Communication also lacks verified outcome feedback, motivating annealed joint OPSD–GRPO training. Following Figure 2, we describe privileged-context construction, then joint training.

Figure 2. OPSD4MAS combines agent-wise GRPO credit with token-level OPSD credit. Future communication is converted into facts, ranked by association with query-normalized trajectory advantage A(τj)A(\tau_j), and filtered by LOO against DrMAS agent-wise advantage AirA_i^r, also used by GRPO. The selected set FiF_i conditions each role’s privileged teacher. Probability displays and numerical examples are illustrative.

Fact Extraction and Hierarchical Filtering

Fact extraction. Given trajectory τj\tau_j, consider its ii-th response yiy_i, omitting jj below and writing r=r(i)r=r(i). An external model examines communication in yi+1,…,ymin⁡(i+W,T)y_{i+1},\ldots,y_{\min(i+W,T)} and extracts atomic corrections, warnings, validated claims, and strategies (Figure 2, panels 1–2). Facts retain their source and target identities. The terminal response can supply facts to earlier steps but has no future source itself. Appendix § 11 gives the Extractor and Relationer prompts. Extraction converts communication into individual facts, and two filters then select the target’s privileged information.

Association filtering. Association filtering (Figure 2, panel 3) uses fact–trajectory relations for the same query, labeled by the external model: rj(f)∈{+1,−1,0}r_j(f)\in\{+1,-1,0\} for followed, violated, and unrelated. For fact ranking, we separately compute A(τj)=(R(τj)−μq)/(σq+ϵ)A(\tau_j)=(R(\tau_j)-\mu_q)/(\sigma_q+\epsilon), counting each trajectory once in the query group’s reward statistics. Unlike the response-occurrence normalization in Equation (3), this does not weight repeated agent turns. We compute

assoc⁡(f)=∑jA(τj)rj(f)∑j∣A(τj)∣∣rj(f)∣+ϵ.(4)\operatorname{assoc}(f)=\frac{\sum_j A(\tau_j)r_j(f)}{\sum_j \lvert A(\tau_j)\rvert\lvert r_j(f)\rvert+\epsilon}. \tag*{(4)}

Following a fact at positive advantage or violating it at negative advantage contributes positively, whereas the reverse contributes negatively. Unrelated labels contribute neither term. The score ranks facts by outcome association on the query rather than weighting the actor loss. Retaining a limited number forms CiC_i, without a positive-score threshold. Appendix § 9 details normalization, exclusions, and tie-breaking.

Fact-Conditioned OPSD and Annealed Joint Training

Target-specific LOO filtering. From the Association-ranked candidates, LOO retains facts with sufficient influence on the target’s tokens, predominantly aligned with its outcome advantage (Figure 2, panel 4). It compares the full candidate set with one fact removed, using cached pre-update policy scores. Write

gi,t(S)=log⁡πθr(yi,t∣q,y<i,yi,<t,S).g_{i,t}(S)=\log\pi_{\theta}^{r}(y_{i,t}\mid q,y_{<i},y_{i,<t},S).

We define the removal effect and its outcome-directed value as

δf,i,tLOO=gi,t(Ci)−gi,t(Ci∖{f}),\delta_{f,i,t}^{\mathrm{LOO}}=g_{i,t}(C_i)-g_{i,t}(C_i\setminus\{f\}),
zf,i,t=Airδf,i,tLOO.(5)z_{f,i,t}=A_i^r\delta_{f,i,t}^{\mathrm{LOO}}. \tag*{(5)}

Here AirA_i^r is the target’s DrMAS agent-wise advantage used by GRPO (Equation (3), jj omitted), not Association’s A(τj)A(\tau_j). Let m+=Et[max⁡(zf,i,t,0)]m_+=\mathbb{E}_t[\max(z_{f,i,t},0)] and m−=Et[max⁡(−zf,i,t,0)]m_-=\mathbb{E}_t[\max(-z_{f,i,t},0)], where expectations include only active response tokens. We measure effect, directional precision, and net alignment:

E(f)=Et[(δf,i,tLOO)2],(6)E(f)=\sqrt{\mathbb{E}_t\left[(\delta_{f,i,t}^{\mathrm{LOO}})^2\right]}, \tag*{(6)}
P(f)=m+m++m−+ϵ,(7)P(f)=\frac{m_+}{m_++m_-+\epsilon}, \tag*{(7)}
N(f)=m+−λcm−.(8)N(f)=m_+-\lambda_c m_-. \tag*{(8)}

EE measures influence on token log probabilities, while m+m_+ and m−m_- measure its advantage-aligned and opposing mass. PP is the positive influence share, not a token fraction, and NN penalizes opposing mass. Useful facts should have sufficient influence, a high positive share, and a large net effect. For this target, a fact passes when E≥Emin⁡E\ge E_{\min}, P≥Pmin⁡P\ge P_{\min}, and N>Nmin⁡N>N_{\min}. Top-KK ranking gives the final set Fi⊆CiF_i\subseteq C_i. We use λc=1\lambda_c=1 by default and rescore FiF_i for OPSD. Context-matching and budget details appear in Appendix § 9.

Fact-Conditioned OPSD and Annealed Joint Training

Fact-conditioned token-level credit. The privileged teacher and ordinary-context student are two views of the same role policy. Using the selected set FiF_i, we compute

δi,t=log⁡πθr(yi,t∣q,y<i,yi,<t,Fi)−log⁡πθr(yi,t∣q,y<i,yi,<t).(9)\delta_{i,t}=\log\pi_{\theta}^{r}(y_{i,t}\mid q,y_{<i},y_{i,<t},F_i) -\log\pi_{\theta}^{r}(y_{i,t}\mid q,y_{<i},y_{i,<t}). \tag*{(9)}

Scores are cached before actor updates. The token-dependent coefficient is Ai,tOPSD=sg⁡(λOδi,tA_{i,t}^{\mathrm{OPSD}}=\operatorname{sg}(\lambda_O\delta_{i,t}, where sg⁡\operatorname{sg} stops gradients and λO=1\lambda_O = 1 by default. It is zero when no facts are selected. Unlike either outcome advantage, this coefficient comes from token log-probability differences.

Role-wise OPSD objective. Within one role-homogeneous loss-reduction batch, let IrI_r contain role rr’s responses, mi,tm_{i,t} be the active-token mask, and Zr=∑i∈Ir∑tmi,tZ_r = \sum_{i\in I_r}\sum_t m_{i,t}. For Zr>0Z_r > 0,

LOPSD(r)=−1Zr∑i∈Ir∑tmi,tAi,tOPSD⋅log⁡πθr(yi,t∣q,y<i,yi,<t).(10)\mathcal{L}_{\mathrm{OPSD}}^{(r)} = -\frac{1}{Z_r}\sum_{i\in I_r}\sum_t m_{i,t}A_{i,t}^{\mathrm{OPSD}}\cdot\log\pi_{\theta}^{r}(y_{i,t}\mid q,y_{<i},y_{i,<t}). \tag*{(10)}

Only ordinary-context probabilities carry gradients to θr\theta_r. OPSD uses neither ratios nor clipping. Appendix A gives the caching and reduction details.

Joint training and linear annealing. Our experiments combine GRPO’s clipped policy-gradient loss with OPSD, as illustrated in Figure 2’s upper panel:

L(r)(s)=LGRPO(r)+wO(s)LOPSD(r),\mathcal{L}^{(r)}(s) = \mathcal{L}_{\mathrm{GRPO}}^{(r)} + w^{O}(s)\mathcal{L}_{\mathrm{OPSD}}^{(r)},
wO(s)=max⁡(w0−sρ,wmin⁡).(11)w^{O}(s) = \max(w_0-s\rho,w_{\min}). \tag*{(11)}

Here ss is the training step, while w0w_0, ρ\rho, and wmin⁡w_{\min} set the initial weight, decay, and floor. Both terms update θr\theta_r: GRPO and LOO share DrMAS agent-wise advantage AirA_i^r, whereas OPSD supplies fact-conditioned token-varying credit. Annealing reduces the OPSD contribution while keeping verified outcome feedback active. Appendix A gives implementation details and loss aggregations. Inference uses ordinary contexts, without future communication or the external model.

Experiments

Setup

Models and data. We initialize trained agents from Qwen3-4B (Yang et al., 2025), and all settings use non-thinking mode. AFlow (Zhang et al., 2025) and GPTSwarm (Zhuge et al., 2024) also use Qwen3-4B executors. AFlow’s search optimizer is Qwen3-235B-A22B-Instruct-2507-Pro (Qwen Team, 2025). For policy training, we use DAPO-MATH-17k (Yu et al., 2025). We test on six other datasets: AMC23, AIME24, AIME25 (Mathematical Association of America, n.d.), MATH-500 (Lightman et al., 2023), the English text-only mathematics subset of OlympiadBench (He et al., [4]), and Minerva (Lewkowycz et al., [9]).

Compared methods. (1) Base Model: Qwen3-4B answers directly without multi-agent interaction. (2) Frozen-weight MAS: Solver–Verifier MAS uses the autonomous interaction loop shared by DrMAS and OPSD4MAS, with model-decided termination and no LLM parameter updates. AFlow and GPTSwarm optimize collaboration structures or prompts while retaining fixed LLM weights. (3) Scalar-reward RL: DrMAS (Feng et al., [3]) with GRPO (Shao et al., 2024) or DAPO (Yu et al., 2025) uses outcome rewards. CoMAS (Xue et al., 2026) derives scalar rewards from interactions. (4) OPSD4MAS: FW2 and FW4 combine OPSD with GRPO, using future windows of two and four responses. Table 1 reports pass@1 and pass@4 as percentages. Appendix § 10.§ 1 gives protocols for evaluation and selection.

Main Results

Gains over scalar-reward RL. FW4 leads all six datasets in Table 1. With equal dataset weights, mean pass@1 exceeds GRPO, DAPO, and CoMAS by 8.87, 6.14, and 8.46 percentage points. Mean pass@4 gains are 9.54, 8.48, and 11.13 percentage points, respectively. The gains over CoMAS extend this comparison to communication-informed rewards. Across both scalar-reward routes, these results support combining outcome credit with communication-derived token-level credit.

Gains over frozen-weight MAS. FW4 also exceeds every reported frozen-weight result, including both the shared Solver–Verifier loop and optimized collaboration structures. On MATH-500, it reaches 91.50% pass@1 and 96.60% pass@4, improving on Solver–Verifier MAS by 8.00 and 4.80 percentage points. On Olympiad, the respective gains are 13.49 and 10.67 percentage points. With Qwen3-4B executors throughout, these results favor learning agent policies over relying solely on fixed-weight collaboration.

Benefits of a larger future window. FW2 is already competitive with scalar-reward RL. Across six datasets, its mean pass@1 is close to DAPO’s, while its mean pass@4 is 4.83 percentage points higher. Expanding the window to four responses raises mean pass@1 and pass@4 by 6.06 and 3.65 percentage points. The observed gains support a broader span of future communication as a valuable source of information for fact selection.

Ablating Future Communication Processing

Table 2 compares five FEHF training variants. No OPSD uses GRPO alone. Raw future communication supplies unprocessed text as privileged context. Unfiltered Facts extracts facts without filtering. Coarse-filtered Facts adds Association but omits LOO, while Full applies both filters.

Only Full exceeds No OPSD in both equal-dataset means, by 15.28 and 16.15 percentage points. Every tested incomplete variant falls below scalar-reward RL alone: access to future communication can therefore hurt training when the privileged context is inadequately processed. Extraction improves mean pass@1/pass@4 over raw communication by 9.33/13.52 percentage points, and Association adds 0.42/1.21 percentage points. Adding LOO produces the largest further gains, 21.92/17.56 percentage points. These results support the complete pipeline: extraction makes knowledge individually assessable, Association prioritizes outcome-related facts, and LOO selects facts with favorable target-specific influence.

Fact Injection: Utility and Task Matching

We vary LOO filtering strength and inject the resulting facts as privileged context to assess their downstream utility. At four FW4 checkpoints, we freeze saved histories and regenerate only the final active Solver on training questions. Facts enter historical and final-input slots. None supplies no facts, while Strict and Relaxed apply different LOO thresholds. All bypasses LOO on the post-association candidate pool.

Task matching under quantity controls. For each filtering profile, Shuffled replaces the original facts with facts from other questions. The paired design keeps injection slots and visible fact counts the same, with approximate length matching, to test whether appropriate content helps beyond adding more text. As shown in Figure 3, Relaxed yields the largest matched–shuffled advantage at every checkpoint, 2.86–8.12 percentage points. This contrast supports task-appropriate selection, rather than extra text alone, as a source of benefit.

Figure 3. Quantity-matched control design: Relaxed has the largest matched–shuffled gains. Nonzero y-axis ranges differ across panels, so compare numerical values across checkpoints. Protocol: Appendix § 10.§ 3.

An intermediate filtering strength works best. Figure 4 compares the ordered settings None, Strict, Relaxed, and All. Relative to the no-injection baseline, Relaxed improves accuracy by 6.13/4.27/1.98/2.10 percentage points at checkpoints 100/110/120/130. Accuracy rises from Strict to Relaxed and falls from Relaxed to All at every checkpoint. Neither very restrictive selection nor removing the LOO gate gives the best result: useful privileged context requires an appropriate filtering strength, not simply a larger fact pool.

Case Study: Understanding Fact Selection

We examine representative facts to explain what our filters remove and retain. Figure 5 connects each decision to the fact’s content, source response, and proposed target.

Figure 5. Selected cases connect filtering decisions to fact content: (a) no actionable repair, (b) a falsifiable error, (c) a concrete correction. Queries and facts are abridged. Boxed facts are proposed privileged contexts for the named responses. DROP/REJECT denotes non-retention, not injection. A/E/N/P denote association, RMS effect, net alignment, and directional precision. Content checks are independent of logged selection, with sources and scores in Appendix § 12.

Low association: no actionable repair. Panel (a) asks for a tetrahedron’s maximum volume. The fact extracted from Verifier y2y_2 tells Solver y1y_1 only that the optimization is unfinished. It supplies no missing constraint, bound, or corrected calculation. With A=−1A=-1, this progress description is dropped before LOO scoring.

High association, unfavorable local direction. Panel (b) specifies a1=a2=a3=1a_1=a_2=a_3=1 and geometric consecutive triple sums with ratio 3. Solver y3y_3 supplies Verifier y2y_2 with the claim an=(3n−10)/13a_n=(3^n-10)/13 for n≡1(mod3)n\equiv1\pmod{3}. At n=1n=1, it gives −7/13-7/13, contradicting the initial condition. Despite A=1A=1 and substantial effect E=0.295E=0.295, negative alignment N=−0.141N=-0.141 and low directional precision P=0.320P=0.320 lead to rejection. Strong teacher influence alone is insufficient, so the local screen also tests its direction relative to outcome credit.

High association, favorable local direction. Panel (c) factors (a−10)(b−10)=2123(a-10)(b-10)=2123 to minimize a rectangle’s perimeter. The fact from Solver y5y_5 for Verifier y4y_4 corrects 13×163=211913\times163=2119 and supplies 2123=11×1932123=11\times193, yielding sides 21 and 203 and perimeter 448. The repair is retained with A=1A=1, E=0.522E=0.522, N=0.246N=0.246, and P=0.795P=0.795.

These cases illustrate how FEHF turns future communication into privileged context. In particular, LOO selects facts by their token-level influence relative to the target’s agent-wise GRPO advantage. The annealed OPSD–RLVR objective then combines agent-wise outcome credit with fact-conditioned token-level credit.

Conclusion

We observe that scalar-reward MAS training compresses or ignores communication and assigns coarse credit. To address this limitation, we propose OPSD4MAS, using Fact Extraction and Hierarchical Filtering (FEHF) to turn future communication into privileged context. Its linearly annealed joint objective combines OPSD token-level credit with GRPO outcome guidance. Supporting this design, OPSD4MAS with a future window of four responses (FW4) leads all baselines across six mathematical benchmarks. Ablations support complete FEHF, while injection diagnostics highlight task matching and filtering strength. Together, FEHF converts future communication into privileged context, enabling annealed OPSD–RLVR joint training to combine agent-wise outcome credit with fact-conditioned token-level credit.

Limitations

Our evaluation focuses on mathematical reasoning with Qwen3-4B agents. Extending the approach to other tasks, model scales, and interaction structures remains future work. Fact extraction and relation labeling rely on an external model, so the quality of privileged context can depend on that model and on filtering thresholds, context budgets, and annealing.

The injection diagnostic evaluates selected facts at fixed weights and frozen histories. Its four checkpoints belong to one training run rather than independent training seeds, so the observed checkpoint differences do not measure variation across training runs.

Ethical Considerations

This work studies reinforcement learning on competition-mathematics data and does not use personal or sensitive information. External model calls receive problem-solving traces, so deployments should ensure that traces contain no confidential data and comply with the provider’s data policy. More broadly, multi-agent self-improvement can amplify incorrect feedback shared between agents. The proposed filters are intended to mitigate this risk, but a reduction has not been demonstrated. Selected facts should not be interpreted as verified truth outside the evaluated domain.

OpenAI Codex was used to assist with code development and debugging, language revision, figure prototyping, and consistency checking. All AI-assisted material included in this submission was reviewed by the authors, who remain fully responsible for the content, claims, citations, code, and reported results.

Implementation and Reproducibility

Computing environment. The project used a computing environment with 24 NVIDIA A100 GPUs, each with 80 GB of memory. This describes the available project resources, not the allocation of every individual run.

Baseline objectives and parameter indexing. DrMAS uses a clipped importance-ratio objective for each agent, with outcome credit normalized by agent action statistics [3]. We follow its query-and-agent response grouping in Equation (3). CoMAS instead collects agent outputs and interaction-derived rewards into a buffer and optimizes a REINFORCE++ clipped surrogate (Xue et al., 2026). For output oo and input pp, its pre-normalization advantage includes a cumulative KL term,

At=r(o)−β∑u=t∣o∣log⁡πθk(ou∣p,o<u)πref(ou∣p,o<u).(12)A_t = r(o) - \beta\sum_{u=t}^{|o|} \log\frac{\pi_{\theta_k}(o_u \mid p, o_{<u})}{\pi_{\mathrm{ref}}(o_u \mid p, o_{<u})}. \tag*{(12)}

After batch normalization, the objective sums min⁡(ρtA^t,clip⁡(ρt,1−ϵ,1+ϵ)A^t)\min(\rho_t\hat{A}_t, \operatorname{clip}(\rho_t, 1-\epsilon, 1+\epsilon)\hat{A}_t) over tokens, with current/old-policy ratio ρt\rho_t. The reported zero-KL-weight configuration has a token-constant reward component. CoMAS agents can take different interaction roles, whereas θr\theta_r in our formulation indexes a fixed role policy.

Cached scores and loss reduction. Teacher and ordinary-context token scores are cached before actor updates, and their difference is computed without gradient tracking. The OPSD coefficient is detached, so gradients flow only through ordinary-context token log probabilities in the actor loss. Extraction and filtering do not receive gradients. The default OPSD loss averages over active response tokens in each role-homogeneous worker microbatch before gradient accumulation. Prompt and inactive tokens are excluded. GRPO instead averages each response’s masked token mean over nonempty responses of that role and uses the clipped ordinary-policy likelihood ratio. OPSD uses neither ratios nor clipping. These distinct reductions are retained in the joint objective.

Two outcome advantages. Association computes A(τj)A(\tau_j) from one episode reward per trajectory within query qq. Its statistics use trajectories identified by the query’s relation maps, with population standard deviation and ϵ=10−6\epsilon= 10^{-6}. Zero-variance groups, including singletons, receive zero. These values weight fact–trajectory relations rather than the actor loss.

GRPO computes the DrMAS agent-wise advantage from response rewards sj,i=∑trj,i,ts_{j,i} = \sum_t r_{j,i,t}, grouping responses by query and agent ID. Repeated turns count separately. For at least two responses, normalization uses the sample standard deviation with denominator n−1n - 1.

with denominator n−1n - 1. A singleton uses mean zero and standard deviation one. Equal-reward groups of at least two responses receive zero. Association uses episode rewards, while GRPO uses the trainer’s response reward tensor, including any configured reward processing. LOO reads the same cached advantages_grpo as the actor loss rather than recomputing an Association advantage.

Candidate selection and ranking. A source fact can reach earlier active non-judger responses in the same trajectory within WW positions. Validated claims from entirely unsuccessful groups are excluded, and duplicate normalized lesson text is collapsed. Candidate ranking uses descending Association, relation coverage, and type priority (correction, warning, validated claim, strategy), followed by shorter future distance and ascending fact ID. Duplicate ties retain the larger ID. Facts passing LOO are ranked by descending net alignment, precision, and RMS effect, then shorter distance and ascending ID before top-KK truncation. All LOO token averages use active response masks. The final selected set receives a fresh teacher-forced scoring pass for OPSD.

Teacher-context construction. The serializer appends [Privileged Future Lessons] and numbered facts to the last user message, then applies the chat template with a generation marker. To meet the context budget, it first removes lessons from the end of their supplied order. If necessary, it removes older unprotected history messages while retaining the first message, first user message, and latest message, then removes further lessons. Full-candidate and leave-one-out prompts are serialized separately. Their difference isolates a single fact when the retained history and all other facts match. Under truncation, it reflects the resulting serialized contexts. For a single candidate, an unavailable removal score is replaced by the cached ordinary score. A missing full score or a missing removal score for multiple candidates causes rejection. An unavailable selected teacher score leaves the OPSD coefficient zero.

Supplementary Results and Evaluation Protocols

Benchmark and Baseline Protocols

Agents initialize from Qwen3-4B and train on DAPO-MATH-17k. All settings use non-thinking mode. FW2 and FW4 denote future windows of two and four responses. Trained checkpoints are selected by validation and then evaluated on the test sets.

Dataset coverage and aggregation. Table 1 reports all nine methods on AMC23, AIME24, AIME25, MATH-500, Olympiad, and Minerva. Main-result means weight the six datasets equally, whereas ablation means use the three named competition datasets. MATH-500 is the held-out subset used by Lightman et al. (2023), not the complete MATH test set Hendrycks et al., [5]. Olympiad denotes OlympiadBench’s English text-only mathematics subset He et al., [4]. Minerva follows the mathematical evaluation setting of Lewkowycz et al. [9]. The reported Base evaluation covers 83, 30, 30, 500, 675, and 272 questions, respectively. Accuracy is reported in percent, and differences are percentage points. A pass@4 sample is a complete trajectory producing a final answer, not an individual agent response within a shared interaction.

Fixed-weight comparisons. Base Model answers directly. Solver–Verifier MAS retains the initial weights in the model-terminated interaction loop shared by DrMAS and OPSD4MAS. AFlow and GPTSwarm use Qwen3-4B executors and optimize collaboration structures or prompts without LLM parameter updates. AFlow’s search optimizer uses the service identifier Qwen3-235B-A22B-Instruct-2507-Pro, with the underlying published model documented by the Qwen team (Qwen Team, 2025). The service suffix is retained for reproducibility. All baseline scores are evaluations in this study rather than numbers copied from the original method papers.

Dataset-Level Ablation Results

Table 3 gives the dataset-level results underlying Table 2. Each metric averages AMC23, AIME24, and AIME25 equally, using full supplied precision before rounding. No OPSD and Full reuse the DrMAS-GRPO and OPSD4MAS (FW4) results from Table 1, respectively. The intermediate settings compare raw future communication, extraction alone, and extraction with Association.

ConditionAMC23AIME24AIME25
p1p_1p4p_4p1p_1p4p_4p1p_1p4p_4
No OPSD68.0783.1348.3363.3332.5053.33
Raw future communication57.2274.7025.0046.6717.5030.00
Unfiltered Facts63.5581.9240.8366.6723.3343.33
Coarse-filtered Facts65.6385.5434.1763.3329.1746.67
Full82.2391.5762.5083.3350.0073.33

Table 3. Per-dataset ablation results (%). Displayed values are rounded, whereas Table 2 averages the original supplied precision, not these rounded cells.

Four-Checkpoint Injection Diagnostic

Evaluation setting. Figure 3 and Figure 4 use FW4 checkpoints 100, 110, 120, and 130 with their saved trajectories and facts. The evaluator freezes the MAS history, inserts facts at targeted historical positions and the final Solver input, and regenerates only the final active Solver on training questions. Model weights remain fixed during this diagnostic.

Filtering profiles. Strict requires RMS effect E≥0.10E \ge0.10, directional precision P≥0.65P \ge0.65, and net alignment N>0.001N > 0.001. Relaxed uses E≥0.005E \ge0.005, P≥0.50P \ge0.50, and N>−0.001N > -0.001, with configuration name loose. All use the post-association candidate pool without a LOO gate, including candidates with missing LOO scores. It does not restore Association-rejected facts. None is the no-injection reference.

Matched controls and statistical unit. For each profile, Shuffled uses facts from other questions at corresponding injection slots, matching visible fact counts with approximate type and length matching. It does not sample random facts from the current question. Queries are the independent units for statistical inference, with trajectories and repeated answers nested within them. The four checkpoints are from one training run. Comparisons are descriptive, without significance tests.

Figure scales. Figure 3 uses checkpoint-specific nonzero ranges of 70–82%, 56–68%, 84–90%, and 82–88%. Numerical values, rather than bar heights across panels, should be compared. Figure 4 uses 60–90%. Its category spacing does not represent equal fact-count increments.

Extractor and Relationer Prompts

These literal templates are exported from the implementation. Braced uppercase slots denote runtime substitutions, not example data. The accompanying source bundle records the builder path and SHA256 hash.

Extractor input. RECORDS_JSON is an array of records containing record_id, problem, evaluated_response, and comment. MAX_FACTS and EXTRACTION_VERSION come from configuration.

You are a mathematical Fact Card extractor. Analyze every record independently and output only the most decision-relevant mathematical lessons. The comment is fallible evidence, not an authority: verify the evaluated_response against the problem yourself, even when the comment approves it or fails to mention an error. When comment is empty, analyze evaluated_response directly. Identify the decisive mathematical pivot, in both directions: the earliest invalid step that can make the final answer wrong, and the key successful step or result that materially makes the correct answer possible. A decisive success factor may be the right modeling reduction, theorem, invariant, formula, substitution, construction, bound, or feasibility constraint; prefer the smallest concrete step without which the conclusion would not follow, rather than routine algebra or a summary of the whole solution. Check formulas, theorem hypotheses, algebraic substitutions, signs, coefficients, arithmetic, domains, quantifiers, integrality or positivity constraints, and whether the conclusion is necessary and sufficient. Preserve exact equations and explicitly contrast an incorrect formula or substitution with its correct replacement. Do not merely restate the final answer. Order facts by priority: (1) decisive correction of the earliest root-cause error and decisive validated success factor or key step; (2) supporting verified formula, theorem, or derivation; (3) specific warning or validation check; (4) general strategy only when no more concrete mathematical lesson is available. Use fact_type=correction for a specific mathematical error and its fix; validated_claim only for a claim independently verified as correct, especially a decisive success-producing step; warning for a concrete failure mode; strategy for a specific reusable method. Each lesson must be atomic, actionable, self-contained, concise, and usable without the original response. Reject bare correctness verdicts, vague advice such as ’be rigorous’, duplicated lessons, and unsupported claims. If no reliable mathematical lesson exists, return an empty facts list. Return at most {MAX_FACTS} facts per record in priority order. Return strict JSON only with schema {"records":[{"record_id":"…","facts":[{"lesson_text":"…","fact_type":"correction—warning—validated_claim—strategy"}]}]}. Version: {EXTRACTION_VERSION}

Records:

{RECORDS_JSON}

Relationer input. FACTS_JSON contains fact_id and lesson_text. TRAJECTORIES_JSON contains trajectory identifiers and ordered responses. RELATION_VERSION comes from configuration.

For each lesson, classify each trajectory’s observable behavior. followed means clear consistent behavior without a clear opposite; violated means clear opposite behavior not clearly corrected; unrelated means absent, insufficient, or mixed behavior. Do not make causal claims. Return strict JSON only with schema {“facts”:[{\“fact_id\”:“...”,“relations”:{“trajectory_uid”:“followed—violated—unrelated”}}]}. Version: { RELATION_VERSION}

Facts:

{FACTS_JSON}

Trajectories:

{TRAJECTORIES_JSON}

Fact-Selection Case Sources

The three cases in Figure 5 come from the FW4 checkpoint-130 export. Table 4 identifies the source records and decisions. Question text comes from saved student prompts. The accompanying records preserve full fact text, response identifiers, target scores, and input hashes.

Panel / IDStepFact IDFact / traj. lineSource / targetDecision
(a) / B31303a722f49767d87d137954ec5500 / 2302 / 1Association rejected
(b) / C11304c0f1ebe82fad73ec9e52a59652 / 2483 / 2LOO rejected
(c) / A11303fa5b26b4d135ef12d49acb0558 / 155 / 4LOO aligned

Table 4. Sources of the cases in Figure 5. Line numbers index the fact-card and trajectory exports from one. Source/target gives the response positions within each trajectory.

Content checks and recorded scores. In (a), the fact describes unfinished optimization without giving a missing constraint or calculation. Its Association is approximately −1-1, and the target is rejected before LOO scoring. In (b), the proposed recurrence yields a1=−7/13a_1=-7/13 rather than the required a1=1a_1=1. Its recorded scores are (A,E,N,P)≈(1,0.294626,−0.140895,0.320246)(A,E,N,P)\approx(1,0.294626,-0.140895,0.320246). In (c), 11×193=212311\times193=2123 gives sides 21 and 203 with perimeter 448, and direct substitution verifies 21×203−5×448=202321\times203-5\times448=2023. The recorded scores for its fourth-response target are (A,E,N,P)≈(1,0.521616,0.245840,0.794624)(A,E,N,P)\approx(1,0.521616,0.245840,0.794624). These mathematical checks accompany the logged filtering decisions, not additional regenerated answers.

References

  1. [1]Arash Ahmadian, Chris Cremer, Matthias Gallé, Marzieh Fadaee, Julia Kreutzer, Olivier Pietquin, Ahmet Üstün, and Sara Hooker. 2024. Back to basics: Revisiting REINFORCE style optimization for learning from human feedback in LLMs. arXiv preprint arXiv:2402.14740.arxiv.org/abs/2402.14740
  2. [2]Yilun Du, Shuang Li, Antonio Torralba, Joshua B. Tenenbaum, and Igor Mordatch. 2023. Improving factuality and reasoning in language models through multiagent debate. arXiv preprint arXiv:2305.14325.arxiv.org/abs/2305.14325
  3. [3]Lang Feng, Longtao Zheng, Shuo He, Fuxiang Zhang, and Bo An. 2026. Dr. MAS: Stable reinforcement learning for multi-agent LLM systems. arXiv preprint arXiv:2602.08847.arxiv.org/abs/2602.08847
  4. [4]Chaoqun He, Renjie Luo, Yuzhuo Bai, Shengding Hu, Zhen Thai, Junhao Shen, Jinyi Hu, Xu Han, Yujie Huang, Yuxiang Zhang, Jie Liu, Lei Qi, Zhiyuan Liu, and Maosong Sun. 2024. OlympiadBench: A challenging benchmark for promoting AGI with olympiad-level bilingual multimodal scientific problems. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 3828–3850.
  5. [5]Dan Hendrycks, Collin Burns, Saurav Kadavath, Akul Arora, Steven Basart, Eric Tang, Dawn Song, and Jacob Steinhardt. 2021. Measuring mathematical problem solving with the MATH dataset. arXiv preprint arXiv:2103.03874.
  6. [6]Haoyang Hong, Jiajun Yin, Yuan Wang, Jingnan Liu, Zhe Chen, Ailing Yu, Ji Li, Zhiling Ye, Hansong Xiao, Yefei Chen, Hualei Zhou, Yun Yue, Minghui Yang, Chunxiao Guo, Junwei Liu, Peng Wei, and Jinjie Gu. 2025. Multi-agent deep research: Training multi-agent systems with M-GRPO. Preprint, arXiv:2511.13288.arxiv.org/abs/2511.13288
  7. [7]Shengran Hu, Cong Lu, and Jeff Clune. 2025. Automated design of agentic systems. In International Conference on Learning Representations.arxiv.org/abs/2408.08435
  8. [8]Aristotelis Lazaridis, Dylan Bates, Aman Sharma, Brian King, Vincent Lu, and Jack FitzGerald. 2026. EDGE-OPD: Internalizing privileged context with evidence guided on-policy distillation. arXiv preprint arXiv:2605.23493.arxiv.org/abs/2605.23493
  9. [9]Aitor Lewkowycz, Anders Andreassen, David Dohan, Ethan Dyer, Henryk Michalewski, Vinay Ramasesh, Ambrose Slone, Cem Anil, Imanol Schlag, Theo Gutman-Solo, Yuhuai Wu, Behnam Neyshabur, Guy Gur-Ari, and Vedant Misra. 2022. Solving quantitative reasoning problems with language models. In Advances in Neural Information Processing Systems, volume 35.arxiv.org/abs/2206.14858
  10. [10]Guohao Li, Hasan Abed Al Kader Hammoud, Hani Itani, Dmitrii Khizbullin, and Bernard Ghanem. 2023. CAMEL: Communicative agents for “mind” exploration of large language model society. In Advances in Neural Information Processing Systems.arxiv.org/abs/2303.17760
  11. [11]Sijia Li, Yuchen Huang, Zifan Liu, Yanping Li, Jingjing Fu, Li Zhao, Jiang Bian, Ling Zhang, Jun Zhang, and Rui Wang. 2026. GEAR: Granularity-adaptive advantage reweighting for LLM agents via self-distillation. arXiv preprint arXiv:2605.11853.arxiv.org/abs/2605.11853

Paper details

vlnc:W_pf5cvxcz
Version
v1 · latest
Published
Fresh — added to the Hub within the last 30 days
Added
License
CC BY 4.0
Usage

views · 0

downloads · 0

A view counts once per browser per day, a download each time the PDF is requested.

  • Computer Science

Not recorded for this paper

View statement
Looking for related papers…
Contents