WorksComputer Science
Agentic On-Policy Distillation Implicitly Reshapes LLMs' ReAct Patterns
Abstract
On-policy distillation (OPD) transfers specialized capabilities to language-model agents, but its cross-domain effects on interaction remain unclear. We investigate how OPD reshapes ReAct interaction patterns across mathematics, ALFWorld, and -bench, focusing on reasoning, actions, and observation acquisition. General-reasoning supervision reduces mathematical tool attempts from 70.4% to 14.4% while increasing accuracy from 16.7% to 38.0%; code-interpreter supervision improves household actions and retail queries. In contrast, ALFWorld supervision reduces -bench accuracy from 41.5% to 6.6%, accompanied by frequent tool calls that violate user-information dependencies. To address negative transfer using only source-domain training trajectories, we explore selective OPD guided by teacher preference changes before and after domain-specific RL. Retaining the top 95% of positions raises accuracy from 11.5% to 29.9% with a source-learning trade-off.
Introduction
Large Language Model (LLM) agents’ competence depends on how it connects reasoning, action, and observation. Mathematical assistants may call python interpreter (Xue et al., [45]) to calculate or directly derive an answer; Service agents may query an account or request a missing identifier. Reasoning and Acting (ReAct) (Yao et al., [47]), which interleaves reasoning traces with task-specific actions, allowing an LLM to plan, track its progress, and handle exceptions through thought while gathering information from external environments such as search APIs or tools, decouples reasoning with actions and observations and becomes the most widely-used paradigm ([24, 23]) which LLMs use to solve agentic tasks ([12, 39]). On-Policy Distillation (OPD), which transfers teacher capabilities by supervising trajectories sampled by the student ([1, 30]), has also become an important step in LLM post-training pipeline ([34] (Ma et al. 2026; Xu et al. 2026)). OPD training can affect models’ choices, especially in the multi-turn agentic tasks, including how to organize reasoning and tool-use, how to obtain new information, interact with the environment, and so on.
We study these ReAct interaction patterns across four domains: (1) pure natural-language general reasoning (Hendrycks et al., [9]) (GR), (2) general reasoning with code-interpreter tools ([8, 22]) (CI), (3) ALFWorld (Shridhar et al., [29]), and (4) -bench (Barres et al., [4]). We find two observable ReAct pattern shifts in trajectories of the above agentic tasks. First, OPD can shift the route to a mathematical answer between tool-assisted computation and direct natural-language reasoning (§ 4.1): GR-domain OPD training starkly reduces models’ tool attempt rate from 70.4% to 14.4% in the CI tasks. Second, it can change how the student chooses actions, adjusts them after environment feedback, and waits for information or authorization before acting (§ 4.2): CI supervision improves both ALFWorld and tasks, accompanied by fewer repeated actions and more complete queries. Meanwhile, we find that ALFWorld-domain OPD makes student’s performance drastically drop from 41.5% to 6.6%. The student frequently generates request-like text alongside tool calls without yielding the turn to the user. More importantly, semantic review identifies calls that depend on unavailable user information or authorization in 55.8% of trajectories. These findings connect cross-domain outcomes to reasoning routes, feedback-driven actions, and the acquisition of observations.
We analyze that severely overfitting to task-specific ReAct patterns is the main cause of the observed negative transfer. Hence, we design a selective policy (§ 5.1), which keeps the generic agentic abilities (mostly derived from the agentic RL training Yu et al. (2025)) and discards the plain transfer of surface-level task-specific ReAct patterns, as illustrated in Figure 1. Given a student’s rollout, we use both of the teacher’s checkpoints before and after domain-specific RL to compute token log-probabilities for sampled tokens. Naturally, the difference () of log-likelihood between two checkpoints (post-/pre-RL) indicates how agentic RL changes teacher models’ preference on the target token. We find that excluding a small fraction of tokens (i.e., with the least 5% RL-induced ).) in the ALFWorld-domain OPD training can effectively mitigate the catastrophic negative transfer on the tasks, while maintaining most of the performance gain on ALFWorld tasks. We also extend the analysis to a RL-trained GR teacher: selective supervision preserves more CI tool use at the same mean GR accuracy, with a reduction in CI accuracy.

Figure 1. Agentic OPD can change the ReAct patterns of LLM agents. Our proposed mitigation approach helps alleviate the overfitting to task-specific patterns.
Our key contributions are two-fold: (1) We systematically examine how OPD training, particularly from domain-specialized teachers, changes the reasoning, action, and observation-acquisition patterns of LLM agents across math reasoning (with and without Python code interpreter), ALFWorld, and -bench. We identifies both positive cross-domain transfer and negative transfer, notably ALF-domain OPD training will drastically make student’s performance drop down; and (2) To mitigate this failure using only ALF-source training trajectories, we propose selecting OPD token positions according to the log-probability change between a teacher checkpoint before and after domain-specific RL, excluding the lowest-tail, which largely alleviates the degradation of accuracy while maintaining most of the performance gain on the ALFWorld. GR-to-CI extension examines whether the method also preserves mathematical tool-use behavior.
Related Work
On-Policy Distillation. Knowledge distillation transfers teacher behavior to students [11]. OPD supervises student-generated contexts [1, 30], while multi-teacher OPD integrates domain experts (Ma et al., 2026). Recent work explores entropy-aware KL objectives [13], OPD optimization geometry [27], and efficient offline distillation (Wu et al., 2026b). For agentic OPD, SEED distills hindsight skills (Wu et al., 2026a), ATOD introduces turn-aware weighting [33], and ActFirst-OPD separates action and reasoning [53]. Studies further examine teacher–student mismatch in multi-turn agents [36, 54, 56] and cross-domain transfer [35] (Li et al. 2026). Delta-based distillation leverages teacher-predecessor preference differences [10]. We examine these effects in agentic trajectories [26], tracing solution routes, responses to feedback, and the timing of information requests. This behavioral focus complements work on forgetting [14, 20] and multi-task interference (Yu et al., 2020).
LLM Agents’ ReAct Patterns. ReAct couples reasoning with actions and observations [47]. Toolformer and Tool-LLM develop tool-use capabilities [23, 24], and API-Bank evaluates tool-augmented interaction (Li et al., 2023). Reasoning methods influence the route to an answer through intermediate derivations and sampled solutions [38, 15, 37] (Xu et al. 2025). Dynamic reasoning strategies [2] and feedback-based revision [28] likewise make process behavior important. Our comparisons ask how such observable patterns change after domain distillation, including when improved answer accuracy accompanies less tool use and when frequent actions bypass necessary user feedback. Tool uncertainty [18], procedurally generated environments [32], and prompt-injection evaluation [6] address complementary aspects of reliable interaction.
Preliminaries and Experiment Settings
Agent interaction. An agent receives a goal and interacts with external participants and systems. At decision turn , its history contains the instructions, previous assistant outputs, tool results, environment observations, and user replies. The policy generates an output . An output may communicate with a user, invoke a tool, or complete the task. The interaction process then supplies feedback , producing . For an assistant-generated token at decision turn , we denote its complete autoregressive context by .
We examine ReAct interaction patterns by analyzing how agents organize reasoning, acting, and obtaining observation throughout an interaction [47]. Reasoning can derive an answer. Acting selects an operation and its arguments. Obtaining observations includes requesting information and yielding the turn so another participant can provide it. They provide three observable perspectives on a trajectory.
OPD and MOPD. Let denote a prompt sampled from the training dataset , and the student and teacher policies, a student-generated response, and the context at step . On-Policy Distillation (OPD) samples on-policy trajectories from the student policy and obtains dense token-level supervision from the teacher policy . The student is optimized by minimizing , the reverse KL divergence between the student and teacher (Gu et al., 2024):
In practice [42, 51, 34], the reverse KL is estimated by a sampled-token approximation [25], so that can be written as a reinforcement-learning objective (i.e., Policy-Gradient (PG) style OPD):
where denotes the stop-gradient. We adopt PG-style OPD throughout: a token is reinforced when the teacher assigns it higher probability than the student, and suppressed otherwise. Multi-Teacher On-Policy Distillation (MOPD) (Ma et al., 2026) is a natural extension of OPD and now a standard post-training paradigm [42, 34] for integrating capabilities from multiple domains: the student samples from its own rollouts, each prompt is routed to its corresponding domain teacher, and the optimization procedure is identical to single-teacher OPD.
Experimental setup. In this work, we study four representative task domains with distinct ReAct patterns: (1) single-turn General Reasoning (GR) task, which uses natural language to solve math problems without calling tools; (2) Code-Interpretation (CI) [45], using Python code-interpreter to solve complex math problems; (3) ALFWorld (ALF) [29], which simulates real-world embodied agentic tasks and supplies text-based household action feedback; and (4) -bench () [4] adds simulated users and account-side tools in airline, retail, and telecom tasks. Student models for all of the OPD experiments are Qwen3-4B-Instruct-2507 [46].
For evaluation, we use AIME2026 problems [52] sampled 16 times and BeyondAIME problems [5] sampled five times in each mathematical setting. ALF-World includes 140 in-distribution and 134 out-of-distribution tasks sampled four times. -bench includes 100 tasks across airline, retail, and telecom, each sampled eight times. The OPD teacher models are derived through applying domain-specific RL training on Qwen3-4B-Instruct-2507 and Qwen3-30B-A3B-Instruct-2507. The experimental settings are concluded in Table 7. Training prompts and tool interfaces are provided in Appendix § 15.
| Domain | Interaction approach |
| GR | Text-based mathematical reasoning without an external tool-feedback loop. |
| CI | Reasoning interleaved with Python execution for calculation, symbolic manipulation, and verification. |
| ALF | Reasoning followed by an admissible environment action, which returns the next observation. |
| Customer communication and account-side tool use, with actions conditioned on available information and required authorization. |
Table 7. Interaction approaches in the four training domains. Their feedback sources and turn requirements are described below.

Figure 2. Cross-domain accuracy changes relative to Base. Rows denote student training sources; columns denote evaluation domains. Outlined cells indicate domains included in training. Absolute accuracies are reported in Appendix § 12.
We compare single-source students, four pairwise compositions, and a CI/ALF/ composition. Model labels indicate training sources, with each model evaluated at a fixed checkpoint selected during source-domain validation. Large rollout batches support substantial learning within 50–60 updates for comparisons; the learning curves and checkpoint identities are given in Appendix § 10.§ 5. Evaluation protocols are specified in Appendix § 11.
Agentic OPD Reshapes ReAct Patterns
We train students with OPD on individual task domains and with MOPD on mixtures of two or three domains, then evaluate each student across tool-enabled and no-tool mathematics, ALFWorld, and -bench. Figure 2 shows that training on one task can substantially change performance on other tasks. For example, CI-only OPD observably raises ALFWorld accuracy from 0.129 to 0.402 (+0.273) and accuracy from 0.415 to 0.510 (+0.095). ALF-only OPD instead drastically lowers accuracy from 0.415 to 0.066 (−0.349). The effects of domain-specific supervision therefore extend beyond the task that supplies its training trajectories, with different outcomes across source–target pairs.
Task scores establish the scope of this cross-domain effect but do not show what changed in the student’s interaction policy. We examine two observable ReAct patterns in the resulting trajectories. First, OPD can shift the route to a mathematical answer between tool-assisted computation and direct natural-language reasoning (§ 4.1). Second, it can change how the student chooses actions, adjusts them after environment feedback, and waits for information or authorization before acting (§ 4.2). These analyses connect the performance differences in Figure 2 to specific decisions in the agent’s reasoning and interaction process.
Reasoning Patterns Shift
Table 1 partitions correct mathematical outputs by whether they include a tool attempt. Following GR training, correct outputs without tools account for 30.1% of all trajectories, compared with 2.4% for Base. Correct outputs containing a tool attempt decrease from 14.3% to 7.9%. The gain in total accuracy therefore accompanies a substantial change in the route by which correct answers are produced.
| Model | Accuracy | Tool Call Rate | Correct no tool | Correct with tool |
| Base | 16.70 | 70.40 | 2.40 | 14.30 |
| ALF-OPD | 38.30 | 11.50 | 30.60 | 7.70 |
| CI-OPD | 52.40 | 73.00 | 4.90 | 47.60 |
| GR-OPD | 38.00 | 14.40 | 30.10 | 7.90 |
Table 1. The statistics on CI math tasks.
Code features help locate this change. Trajectories containing loop-like code decrease from 304 to 97, and those containing solver calls decrease from 75 to eight. In a matched-input urn problem, Base increments the total number of marbles and enumerates the red count until two hypergeometric probabilities agree. The GR student instead cancels the binomial coefficients, derives , , and , and applies the lower bounds on both colors. Both obtain the correct answer, 190. Further cases appear
in Appendix H.
ALF training exhibits a related shift in mathematics. Its accuracy is 38.3%, tool-attempt rate 11.5%, and correct no-tool share 30.6%. In a rotation-geometry case, Base constructs the wrong triangle and uses Python to calculate the resulting expression. The ALF student derives the rotated point in coordinates and obtains the correct answer without a tool. By contrast, the CI student retains a 73.0% tool-attempt rate, and correct tool-containing outputs account for 47.6% of its 52.4% accuracy.
| Model | All | Retail | Tool | Calls | Error | Query |
| Base | 41.50 | 48.10 | 95.90 | 5.66 | 32.60 | 54.50 |
| CI | 51.00 | 65.30 | 95.80 | 6.50 | 23.50 | 67.50 |
Table 2b. Positive transfer from CI. CI →
Observation 1: OPD shifts reasoning routes. In tool-enabled mathematics, GR and ALF supervision shift the student models’ reasoning route toward natural-language derivations (e.g., leveraging sophisticated algorithms to avoid large computational loads), while CI supervision favors solutions containing Python code tool-calls (e.g., using more loop code for brute-force search). OPD therefore changes the student’s route to an answer across domains.
Action and Observation Patterns Shift
Action: CI-OPD enables more productive environment interactions in ALFWorld and tasks. In ALFWorld, Table 2a shows that accuracy rises to 0.402 while adjacent action repetitions fall from 4.5% to 0.7%. After a “Nothing happens” observation, the fraction of subsequent actions that change increases from 90.8% to 94.3%. The mean number of calls rises from 12.97 to 18.94, so the improvement combines longer interaction with fewer consecutive repetitions. For an instance, base student model stops after searching empty drawers in a bowl-on-coffee-table task. The CI-trained student continues to the dining tables and finds the bowl. It first tries placing it without moving, then responds to no-op feedback by moving to the coffee table and completing the task.
| Model | Accuracy | Calls | Repeat | Change |
| Base | 12.90 | 12.97 | 4.50 | 90.80 |
| CI | 40.20 | 18.94 | 0.70 | 94.30 |
Table 2. Positive transfer from CI. (a) Calls are complete generated blocks per rollout; Repeat is exact repetition among adjacent actions; Change is a changed action after no-op feedback with a following action. (b) All/Retail are overall and retail task accuracy; Tool is rollout-level attempt coverage; Error is the fraction of rollouts with error feedback; Query is complete reference-query coverage on the 200 eligible retail rollouts. Calls again use a per-rollout denominator. Detailed metric definitions and denominators are provided in Appendix C. (a) CI → ALFWorld
CI training also raises task accuracy to 0.510. As shown in Table 2b, the overall fraction of rollouts containing a tool attempt is nearly unchanged, 95.9% versus 95.8%. Retail contributes 55 of the 76 additional successful samples, with accuracy increasing from 0.481 to 0.653. On retail tasks with reference lookup actions, the share of trajectories that generate all reference queries increases from 54.5% to 67.5%. The fraction of all rollouts with error feedback falls from 32.6% to 23.5%. These gains accompany more complete information retrieval and fewer error-containing trajectories. For a payment-method change, Base assumes the order already uses the requested card without retrieving it. The CI-trained student queries the order, observes a different card, checks the gift-card balance, obtains confirmation, and successfully changes payment. CI also produces more mixed communication–tool turns: improved query sequences and turn protocol compliance are different properties.
Action: OPD changes how agents obtain information. In contrast to CI’s positive transfer, ALF-only OPD reduces task accuracy from 41.5% to 6.6%. We discover that this decline accompanies a change in how the student obtains information from users.
We measure ask–call co-occurrence as the fraction of assistant outputs containing detected request-like text that also contain a generated tool call. As shown in Table 3, this rate reaches 94.3% for the ALF student, indicating that asking and calling frequently occur within the same turn. Such text is not delivered to the user, and co-occurrence may indicate a problem when the call requires information not yet provided by the user.
| Model | Accuracy | Ask–call | Dependency |
| Base | 41.50 | 0.04 | 0.50 |
| Teacher | 46.00 | 0.67 | 0.38 |
| ALF | 6.60 | 94.30 | 55.80 |
| ALF+ | 68.90 | 0.24 | 0.00 |
Table 3. Information acquisition in interactions. Accuracy measures task success.

Figure 3. Excerpts from trajectories for the same annotated task. The ALF student generates a user-ID request alongside a tool call using a schema-example ID. ALF+ waits for the user’s actual ID.
To distinguish these cases, we additionally report the dependency-violation rate: the fraction of trajectories containing at least one call that depends on user information or authorization requested in the same output but not yet obtained. These violations are identified through LLM-based semantic review of the interaction history available before each call (Appendix § 11.§ 3). As shown in Table 3, dependency violations occur in 55.8% of ALF student trajectories, compared with 0.50% for Base. These results indicate that the ALF-trained student frequently proceeds with tool calls before obtaining necessary user information or authorization. Figure 3 illustrates this pattern. The ALF student generates a request for a user ID but immediately calls the lookup tool using an example ID from the tool schema, without waiting for a reply. The ALF+ student instead waits for the user’s ID before making the corresponding call. This contrast is consistent with its much lower ask–call co-occurrence and absence of detected dependency violations.
Notably, the ALF teacher rarely exhibits ask–call co-occurrence on , so the student’s pattern does not simply mirror the teacher’s target-domain behavior. ALFWorld typically supplies observations through environment actions, whereas also requires waiting for user-provided information or approval. The observed transfer is therefore consistent with a shift toward tool-mediated observation acquisition, even when the required feedback must come from the user.
Observation 2: OPD changes Action Patterns Domain-specific OPD supervision (e.g., from CI) can be associated with more effective and productive action patterns after environment feedback, which can also generalize to other held-out agentic tasks. However, domain-specific OPD supervision (e.g., from ALF) can instead starkly change the action habit of obtaining information by LLM agents (e.g., producing frequent tool-calls rather than requesting feedback from the user-side in the tasks).
Task-matched supervision in MOPD
The pronounced shifts above arise after single-source training; MOPD tests whether adding supervision from the evaluated task domain changes these observations. Table 4 compares four pairwise compositions and the three-teacher student through scores and conversational behavior.
Table 4 Teacher composition. (a) Accuracy on CI/GR, ALFWorld, and . (b) Ask–call uses request turns; error repetition uses post-error turns. Definitions and counts appear in Appendix § 11.
| Model | CI | ALF | GR | |
| CI+GR | 51.30 | 38.60 | 45.30 | 44.50 |
| ALF+CI | 49.80 | 83.20 | 34.00 | 46.70 |
| CI+ | 50.20 | 28.40 | 67.80 | 45.80 |
| ALF+ | 33.80 | 83.40 | 68.90 | 36.70 |
| ALF+CI+ | 51.10 | 86.90 | 64.40 | 45.00 |
| Model | score | Ask–call | Error repeat |
| ALF+CI | 34.00 | 25.60 | 8.10 |
| CI+ | 67.80 | 0.66 | 9.70 |
| ALF+ | 68.90 | 0.24 | 10.60 |
| ALF+CI+ | 64.40 | 0.10 | 3.00 |
Table 4. Teacher composition. (a) Accuracy on CI/GR, ALFWorld, and . (b) Ask–call uses request turns; error repetition uses post-error turns. Definitions and counts appear in Appendix § 11.
(1) In mathematics, CI+GR reaches 51.3% accuracy with 74.4% tool attempts when tools are available, while achieving 44.5% accuracy in the no-tool setting. This joint behavior contrasts with GR-only training, which reaches 38.0% accuracy and 14.4% tool attempts in tool-enabled mathematics. The route shift observed after GR-only supervision is thus not fixed when CI supervision is also present. (2) Adding prompts and their specialist teacher to ALF training raises accuracy from the ALF-only student’s 0.066 to 0.689. Same-turn ask–call co-occurrence falls from 94.3% to 0.24%, and the reviewed ALF+ rollouts contain no current-request dependency violations. At the same time, ALFWorld accuracy remains high (0.834 versus 0.855 for ALF-only). Within these evaluated tasks, task-matched supervision largely removes the specific harmful timing pattern identified in §4.2.
However, coverage by task-matched teachers is nevertheless limited to the training sources. ALF+CI has no supervision: its performance still drops by 7%, and meanwhile ask–call co-occurrence remains 25.6%. This held-out result shows that combining teachers can still leave ReAct pattern (negative) transfer in an unseen task domain. Training cannot enumerate every agentic task or feedback protocol. The observed recovery therefore motivates more principled methods for preserving feedback dependencies and execution roles across domains, beyond adding a specialist teacher for each known task.
Exploring Mitigations for Negative Transfer in Agentic OPD
In this section, we mainly explore how to mitigate the potential cross-task negative transfer in agentic OPD training, using ALF-to- transfer (§ 4.2) as the main case and extending the analysis to GR-to-CI transfer.
Methodology: Selective OPD Supervision on Tokens with RL-Induced Preference
We summarize our core insight derived from the previous observations is agentic OPD training can teach student two parts of knowledge: (1) generic agentic abilities (e.g., how to reason and act based on observations) and (2) task-specific ReAct patterns (e.g., action habit of obtaining information in § 4.2). Severely overfitting to part (2) is the main cause of the negative transfer. Hence, the key idea of our proposed approach is to design a selective policy, which keeps the generic agentic abilities (mostly derived from the agentic RL training (Yu et al. (2025)) and discards the plain transfer of surface-level task-specific ReAct patterns.
Given a student’s rollout, we use both of the teacher’s checkpoints before and after domain-specific RL to calculate log-likelihood on every token. Naturally, the difference of log-likelihood between two checkpoints (post-/pre-RL) indicates how agentic RL changes teacher models’ preference on the target token. Let and denote the two teacher checkpoints. For student-generated token under complete history , define
Positive indicates increased preference after RL; a strongly negative value indicates suppression. We rank positions from the highest to the lowest by signed and gradually retain more high-rank tokens (i.e., initially retain top 10%, then top 20%, 30%, …), while monitoring how student’s performance change on ALFWorld and -bench. Specifically, for a rollout with valid assistant positions , retained fraction selects :
More detailed description and settings of our experiments can be found in Appendix § 10.§ 4.
Results The results are shown in Figure 4. We evaluate preference-guided selection using retained fractions of 10%, 20%, 50%, 95%, 99%, and 100%, where full retention corresponds to ordinary OPD. All runs use only ALFWorld training trajectories. We use evaluation to analyze the source–target trade-off across retention ratios. The key findings are concluded as following three phases: (Phase 1) By only retaining 10% tokens, OPD can achieve about 80% performance gain (14.3% → 69.4%) on the training agentic task (i.e., ALFWorld). Meanwhile, the student model’s performance (41.5% → 31.1%) does not drop that drastically; (Phase 2) Gradually increasing the retained fraction (10% → 95%), we observe that model’s ALF accuracy slightly improves (69.4% → 76.6%) while performance remains stable; (Phase 3) However, retaining the final 5% tokens (with least 5% ) starkly decreases the model’s performance (29.9% → 11.5%). Besides, we trace the variation of ask–call co-occurrence (as defined in § 4.2, measuring to what extent the models’ ReAct mode is interfered): Phase 3 has seen the steepest ascent of this metric (24.3% → 97.1%). We also conduct a comparison experiment (“Random95%”) by randomly retaining 95% tokens and excluding the rest 5% tokens. Dashed lines in Figure 4 show the results: unlike retaining Top95% tokens, Random95% still affects student’s performance.
Figure 4. Effect of retained fraction after 20 updates on ALF accuracy, accuracy, and ask–call co-occurrence. The shaded leftmost point is an evaluation of the shared initialization from the Top95 run, before training. Trained points use their numerical retained fractions on the x-axis. Labels are offset for the closely spaced 95/99/100% points. Horizontal dashed lines show the Random95 update-20 evaluation.
Insight: Excluding Smallest Tokens Can Rescue.
Excluding a small fraction of tokens (i.e., with the least 5% RL-induced ) in the ALFWorld-domain OPD training can effectively mitigate the catastrophic negative transfer on the tasks, while maintaining most of the performance gain on ALFWorld tasks.
More Analysis and Explorations
Section 5.1 shows that excluding a small fraction of positions helps alleviate large-scale performance degradation on bench. We present more analysis and explorations on this result. Detailed training dynamics and a separate GR prefix-supervision experiment are provided in Appendices § 14.
Which part of tokens are excluded? As shown in Figure 5, reasoning content (text outside tool calls) tokens account for 98.19% of excluded positions and 90.08% of all valid positions, giving modest enrichment (1.09×), while Tool-call onsets (the <tool_call> special tokens) are more selectively affected: 19.13% are excluded, or 3.84× the overall exclusion rate. The specific tool-call content (tokens inside the <tool_call> and </tool_call>) are rarely excluded, with an exclusion rate of only 0.32%. This pattern connects supervision to our ReAct observations. ALFWorld often obtains observations through environment actions, whereas tasks often require a user reply before acting. Suppressive updates to planning, state descriptions, and action onsets can change how the shared agent organizes its next step to obtain information.

Figure 5. Exclusion rates and enrichment on Top95 training contexts. Reasoning content denotes assistant text outside tool-call blocks. <tool_call> is the opening special token; payload denotes tokens inside a call. Parentheses show enrichment relative to the overall exclusion rate (dashed line).
Additional Explorations We first test three alternatives: delimiter masking removes supervision on the opening and closing tool-call tokens; full-block masking additionally removes the enclosed payload; and negative masking removes all negative ordinary OPD advantages while retaining the full-token denominator. These controls examine structural positions and feedback direction.
We further compare ranking rules using ordinary OPD feedback and the same selected-token normalization. Advantage Top95 and Advantage Top98 exclude the lowest 5% and 2% of signed ordinary advantages, respectively. The former matches the nominal exclusion fraction of Top95. On fixed diagnostic contexts, the latter removes 43.56% of absolute coefficient mass, compared with 41.02% for Top95; the retained mass after normalization is 57.54% and 62.02%, respectively. This provides an approximately coefficient-comparable control, rather than a matched online gradient update. Absolute- Top95 instead excludes the 5% of positions with the smallest , testing preference-change magnitude rather than direction.
Advantage Top95 partially preserves accuracy (23.50%), but ask–call co-occurrence remains high (82.68%). Advantage Top98 and Absolute- Top95 reach 5.88% and 17.13% accuracy, with co-occurrence of 99.16% and 83.94%, respectively. All three achieve higher ALF accuracy (81.66–85.86%) than Top95 (76.64%), while Top95 retains higher accuracy (29.88%) and substantially lower co-occurrence (24.26%). Thus, neither filtering the strongest negative advantages nor selecting by change magnitude reproduces the same source–target balance in these comparisons. Appendix § 14.§ 4 provides definitions, coefficient diagnostics and learning curves for these three ranking controls.
Extending Selection to GR-to-CI Transfer
We next test whether selective supervision also changes the mathematical tool-use shift observed in § 4.1. For this extension, we train a Qwen3-30B-A3B teacher with tool-free GR reinforcement learning and use its pre-RL checkpoint as the reference. This teacher is distinct from the GR teacher in the earlier cross-domain matrix. Starting from the same 4B student initialization, Full OPD and signed- Top95 use the same tool-free training data and are evaluated after 20 updates. The paired CI and GR evaluation sets contain the same mathematical questions, with a code interpreter available only in CI.
Table 6 shows similar GR accuracy (46.02% versus 46.33%), while Top95 increases CI tool attempts from 10.41% to 29.49%. The proportion of CI samples that use a tool and answer correctly increases from 6.02% to 11.63%. Overall CI accuracy is lower, however (36.53% versus 37.65%), with fewer correct answers on the no-tool route (24.90% versus 31.63%). Selection therefore preserves more tool-assisted behavior without improving aggregate CI accuracy in this setting. Together with ALF-to-, this extension shows why task accuracy and interaction behavior should be assessed jointly. Appendix § 14.§ 5 reports the recipe and learning curves.
| Method | GR acc. | CI acc. | CI tool attempts | No tool & correct | Tool & correct |
| Full OPD | 46.02 | 37.65 | 10.41 | 31.63 | 6.02 |
| Signed- Top95 | 46.33 | 36.53 | 29.49 | 24.90 | 11.63 |
Table 6. Results of GR-to-CI transfer.
Conclusion
Agentic OPD reshapes how LLM agents reason, act, and obtain observations across domains, producing positive and negative transfer. Task-matched MOPD mitigates the conversational failures induced by ALF supervision on . As a source-only training exploration, we select ordinary OPD supervision positions using pre/post-domain-RL preference changes. Excluding the low- tail substantially alleviates negative transfer while sacrificing source-domain performance. Token-level analysis reveals concentrated context-dependent suppressive feedback, including reasoning and tool-call onsets. Structural masks, negative masking, and alternative ranking rules yield different source–target trade-offs; none of the reported alternatives reproduces the same target-interaction preservation as signed- Top95 in the ALF comparison. An extension with a RL-trained GR teacher preserves more CI tool use at equal mean GR accuracy. These findings highlight the importance of controlling where specialization feedback is applied in agentic OPD.
Limitations
Our experiments focus on the Qwen3 model family with 4B students and mathematical and interactive tasks. Extending the analysis to larger students, other model families, and a broader range of agent environments would help establish the generality of the observed transfer patterns. The effects of supervision selection may also depend on the training recipe and interaction protocol, motivating further study across additional configurations.
Ethical Considerations
This research helps identify potential risks arising when specialized training changes an agent’s interaction behavior. In particular, changes in information gathering and user confirmation can lead to actions based on unsupported assumptions or actions taken before authorization is obtained. Our findings can inform evaluations of tool-using agents and encourage attention to feedback dependencies, confirmation requirements, and execution permissions in practical applications.
Discussion
What should agentic training evaluate? Task accuracy, action content, and feedback dependencies capture different properties of agent behavior. The mathematical results pair improved accuracy with fewer tool attempts; CI transfer pairs better outcomes with action adjustment and query coverage. Conversely, ALF transfer produces request-like text alongside tool calls, some of which depend on user information or authorization not yet obtained. Evaluation should therefore measure how an agent prepares an action, acquires its prerequisites, and responds to the resulting observation, alongside task utility [3] and benchmark breadth [19, 55]. Importantly, ask–call co-occurrence captures generated output patterns rather than actual user requests, and should be interpreted alongside dependency violations and task accuracy.
How should supervision be composed? MOPD provides complementary training coverage: CI supports mathematical tool use and feedback-driven actions, while task-matched supervision mitigates failures to wait for required user information. These findings connect teacher composition to behavioral coverage and task grouping [31] (Yu et al. 2020). Source-only -guided selection offers another approach when target-domain training data are unavailable. Excluding a small low- tail reduces concentrated suppressive feedback at context-dependent positions, with a source-learning trade-off. Delimiter masking, full-block masking, and negative masking fail to reproduce the same source–target balance. These results highlight the importance of feedback location and direction, while further controlled interventions are needed to isolate the mechanisms underlying cross-domain transfer.
Experimental Setup and Training
Our main experimental settings are concluded in Table 7.
Domain Interaction and Tool-Use Instructions
The domains differ in how reasoning leads to an action and how the agent obtains the feedback needed for its next decision.
General reasoning (GR). GR training uses mathematical problems solved through natural-language reasoning. The model develops a solution and produces its answer without an external tool-feedback loop. This source therefore supervises text-based derivation rather than interaction with an execution environment.
Code interpretation (CI). CI allows the agent to invoke code_interpreter when calculation, symbolic manipulation, or verification is useful. A tool call supplies Python code, and its execution result is appended to the interaction history. The agent can then use that result to continue reasoning, revise a calculation, or produce its answer.
ALFWorld (ALF). At each decision step, the agent receives an environment observation and a list of admissible actions. The prompt asks it to reason about the current situation and execute exactly one listed action through alfworld_step. The action argument must match an entry in the admissible action list. Execution returns the next observation, so information gathering and task progress are both mediated by environment actions.
Conversational tool use (). The agent alternates between communicating with a customer and invoking an available account-side tool. The prompt specifies one interaction branch per turn: a customer-facing message or a tool call. Missing information must be obtained from the customer, and database-changing actions require explicit confirmation under the corresponding domain policy. Customer-side device procedures are communicated as instructions for the customer to perform. Airline, retail, and telecom share these interaction requirements while using different policies and tools.
Tool calls in CI, ALF, and use JSON objects with name and arguments fields enclosed by <tool_call> and </tool_call> markers. This serialization is shared across the tool-using domains; the surrounding reasoning, feedback source, and turn requirements depend on the task.
Model Initialization and Teacher Construction
All students initialize from Qwen3-4B-Instruct-2507. The CI teacher is a domain-specialized model from the same 4B family. The ALF and teachers initialize from Qwen3-30B-A3B-Instruct-2507, with approximately 30B total and 3B active parameters, and undergo domain-specific agentic RL. GR uses the released general 30B-A3B model. Teachers remain frozen during OPD. Figure 6 shows teacher learning during specialization and marks the checkpoints used for distillation. Table 8 summarize the optimization settings.
Figure 6. Teacher RL learning curves. Solid curves show validation task outcomes, and dashed curves show sampled training reward. CI validation measures answer accuracy, with separate update axes for its two training stages. Squares mark the selected teacher checkpoints. Training and validation use their respective prompt pools and decoding settings.
| Parameter | Teacher RL | OPD / selected-position runs |
| Optimizer | Megatron Adam | Megatron Adam |
| Learning rate | ||
| Warmup / schedule | none / constant | none / constant |
| Weight decay | 0.01 | 0.01 |
| Adam momentum | backend (0.9, 0.999) | backend (0.9, 0.999) |
| Adam epsilon | backend default; no override | backend default; no override |
| Gradient-norm clip | 1.0 | 1.0 |
| Policy-ratio clip / dual clip | [0.8, 1.28] / 10 | [0.8, 1.28] / 10 |
| OPD advantage clamp | not applicable | [−10, 10] |
| Entropy coefficient | 0 | 0 |
| Explicit KL loss / KL in reward | both disabled | both disabled |
| Task-reward mixing | GRPO reward advantage | 0 (monitoring only) |
| Distillation coefficient | not applicable | 1 |
| Precision | bfloat16 | bfloat16 |
| Dynamic token microbatching | enabled | enabled |
| Teacher gradients | trainable RL policy | frozen scoring only |
Table 8. Optimization settings separated from sampling and task rewards. Adam momentum and epsilon are inherited by the backend when no explicit optimizer override is provided. Configured minibatch sizes and rollout counts are listed separately because the worker expands prompt-unit minibatches by the rollout count.
For preference-guided supervision, the ALF teacher’s checkpoint immediately before domain-specific RL serves as the pre-domain-RL reference, and its specialized checkpoint serves as the post-domain-RL teacher.
CI teacher. CI specialization uses two GRPO stages on a 13,616-row English subset of DAPO-Math17k (Yu et al. (2026)) with Python execution. Stage 1 combines answer verification with tool-use shaping and a repetition penalty. Stage 2 initializes from the stage-1 weights at update 30 with a fresh optimizer and retains answer verification and the repetition penalty. The selected CI teacher is the stage-2 checkpoint at update 106.
Let be the answer-verifier reward, the number of tool responses passing the response-error detector, and an indicator. The stage rewards are
The repetition detector checks whether the final 512-character block appears earlier in an output of at least 1,024 characters. indicates that tools are called but no response passes the error detector. requires at least 500 characters before the first call. requires a clean execution before the first boxed answer, or a clean execution when no boxed answer is produced.
ALF and teachers. ALF specialization uses 3,553 prompts and binary environment task-completion reward. The training collection contains 2,160 rows: 1,178 airline, 637 retail, and 345 telecom. Its reward follows the benchmark’s task checks, including database and communication requirements where applicable. The selected teachers are the ALF checkpoint at update 60 and the checkpoint at update 160.
On-Policy Distillation Implementation
The student generates trajectories using the interaction approach of each training domain. A frozen teacher scores the student-generated assistant tokens under the same complete histories, including previous assistant outputs, tool observations, and user replies. In MOPD, each prompt is scored by its assigned source-domain teacher.
Let denote that teacher and the rollout student. The ordinary token advantage is
The implementation clips this advantage and detaches the resulting coefficient . For , the clipped surrogate is
Dual clipping gives
The advantage clamp, ratio bounds, and dual-clip coefficient are listed in Table 8. The log policy ratio is bounded to before exponentiation.
Let identify assistant positions with teacher scores that are retained for supervision. The loss averages retained-token means across rollouts:
Tool observations, user replies, and padding remain available as context but receive no distillation loss. Sequences with no retained positions contribute zero. Task outcomes are recorded for monitoring; the update uses the distillation advantage.
Source mixtures. Pairwise CI+ALF and CI+GR training use 3,000 prompts from each source. CI+ and ALF+ combine 3,000 prompts from the first source with 2,160 prompts. Three-teacher training combines 3,000 CI, 3,000 ALF, and 2,160 prompts. The collections are concatenated and shuffled, with teacher routing determined by each prompt’s source. These counts determine prompt-level sampling proportions; trajectory lengths determine the corresponding token exposure.
Preference-Guided Supervision
Preference-guided experiments use only the 3,000-prompt ALF training subset. The frozen pre-domain-RL reference and post-domain-RL teacher score the same student-generated positions.
For each multi-turn rollout, selection ranks valid assistant positions across all turns by signed , as defined in § 5.1. A retained fraction selects of the valid positions. Ties prioritize the earlier response position. Retained positions receive the ordinary OPD advantage, including both positive and negative feedback, and their loss is normalized by the selected count.
Random95 draws a uniform subset without replacement using the same retained count and normalization. Top100 retains all valid positions. The remaining training settings are given in Table 9. Definitions of the structural and feedback controls appear in Appendix § 14.
| Student/condition | Rows | Batch | Samples | Traj./batch | Prompt/response caps |
| CI | 17,917 | 128 | 4 | 512 | 1,024/31,740 |
| ALF | 3,553 | 128 | 4 | 512 | 1,024/31,740 |
| 2,160 | 32 | 8 | 256 | 8,192/32,768 | |
| GR | 17,917 | 32 | 4 | 128 | 8,192/32,768 |
| CI+GR | 6,000 | 128 | 4 | 512 | 8,192/32,768 |
| ALF+CI | 6,000 | 128 | 4 | 512 | 1,024/31,740 |
| CI+ | 5,160 | 128 | 4 | 512 | 8,192/32,768 |
| ALF+ | 5,160 | 128 | 4 | 512 | 8,192/32,768 |
| ALF+CI+ | 8,160 | 128 | 4 | 512 | 8,192/32,768 |
| Top95/Random95/Top100 | 3,000 | 32 | 4 | 128 | 8,192/32,768 |
Table 9. Main OPD and selected-position launch recipes. Batch and configured PPO minibatch are prompt counts; “Samples” is rollouts per prompt. Learning rate is and per-GPU microbatch is one. Prompt/response caps are measured in tokens.
Checkpoint Selection and Learning Dynamics
Each named student uses a fixed representative checkpoint selected from its validation stage. The same checkpoint is used for its task outcomes and behavioral analyses. Figure 7 and Figure 8 show the learning curves and mark these checkpoints.
Figure 7. Single-teacher OPD learning curves. Solid curves show source-validation outcomes, and dashed curves show sampled training outcomes. Squares mark the checkpoints used in the main analyses. CI is displayed through update 130 and GR through update 50. The validation curve contains measurements at updates 0, 25, and 50. Mathematical monitoring pools differ from the 980-trajectory behavioral evaluation subset.
Figure 8. Source-validation learning curves for teacher compositions. Each curve reports a separate training domain; squares mark the checkpoints used in the main analyses. ALF+CI and CI+ are displayed through update 80, and CI+GR through update 50. Other panels show their respective recorded ranges.
The curves document source learning for single-teacher and multi-teacher training. For compositions, each domain is monitored separately, so progress in one source is visible alongside progress in the others. Preference-guided comparisons instead use the common update-20 checkpoint specified in § 5.1. Evaluation datasets, decoding settings, and behavioral metrics are described in Appendix § 11.
Evaluation Protocols and Behavioral Metrics
Evaluation Data and Execution
Mathematical evaluation contains 30 AIME2026 [52] questions sampled 16 times and 100 BeyondAIME [5] questions sampled five times, giving 980 trajectories per model. The tool-enabled setting provides code_interpreter; the no-tool setting evaluates text-based solutions. ALFWorld evaluation contains 140 in-distribution and 134 out-of-distribution tasks, each evaluated four times, yielding 1,096 trajectories per model. The evaluation contains 100 tasks sampled eight times: 160 airline, 320 retail, and 320 telecom trajectories. We use the same sampling settings for all training rollouts and evaluations: temperature 1.0, top-p 1.0, and unrestricted top-k. Teacher token scoring computes probabilities for student-generated tokens at temperature 1.0, without generating additional responses.
Agent execution uses the Qwen3 chat template and structured tool calls. The loop parses the complete assistant output before choosing the next interaction branch. In , exactly one parsed call with object-valued arguments is submitted to the environment. If prose accompanies the call, it remains in the agent history but is not sent to the customer. Text-only outputs are sent to the customer. Multiple parsed calls, or a tool marker with no parseable call, produce local error feedback.
The user simulator uses DeepSeek-V4-Flash (Xu et al., 2026) at temperature 0.7. Natural-language assertions use GPT-5.5 at temperature 1.0.
Metric Definitions
Table 10 defines the outcome, generation, and interaction metrics. Their denominators follow the event being measured: task outcomes use trajectories, request–call co-occurrence uses request turns, and feedback-response metrics use eligible action transitions.
| Metric | Unit or denominator | Definition |
| Task accuracy | Evaluation trajectories | Mean answer correctness for mathematics or binary task completion for ALFWorld and . |
| Tool-attempt rate | Evaluation trajectories | Fraction containing at least one generated tool-call marker. |
| Calls per trajectory | Evaluation trajectories | Mean number of generated spans enclosed by opening and closing tool-call markers. |
| Call JSON validity | Complete generated call blocks | Whether a block parses to an object with a string-valued name and dictionary-valued arguments. |
| Ask–call cooccurrence | Assistant outputs containing detected request-like text | Fraction of assistant outputs containing detected request-like text that also contain a generated tool call. |
| Dependency-violation rate | Evaluation trajectories | Fraction of trajectories containing a tool call that depends on user information or authorization not yet obtained, as identified through semantic review. |
| Affected-task coverage | Benchmark tasks | Fraction with at least one sampled trajectory containing a dependency violation. |
| Adjacent-action repetition | Consecutive ALF action pairs | Fraction with identical normalized action strings. |
| Post-no-op adjustment | ALF no-op observations followed by another action | Fraction whose next action differs from the action that produced the no-op observation. |
| Post-error repetition | Tool-error observations followed by an assistant turn | Fraction whose next parsed call repeats the preceding call’s tool name and normalized arguments. |
| Error-bearing trajectory rate | Evaluation trajectories | Fraction containing at least one tool-error observation. |
| Reference-query coverage | Eligible retail trajectories () | Fraction containing every annotated reference query, matched by tool name and normalized arguments. |
Table 10. Definitions of the evaluation metrics. Calls are reported as a mean count; rates are reported as percentages. Generated calls and JSON validity describe model outputs, while task accuracy measures final outcomes.
Mathematical solution routes. Correct outputs are partitioned by the presence of a tool attempt. Both shares use all 980 trajectories as their denominator, so correct tool-containing and correct no-tool outputs sum to overall accuracy.
Semantic Review of Dependency Violations
Information requests and dependencies. The request detector identifies request-like expressions in generated assistant text after removing complete tool-call blocks. These include requests to provide, confirm, share, or supply information, as well as expressions such as “I need your” followed by an identifier or phone number. Ask–call co-occurrence measures the fraction of outputs containing detected request-like text that also contain a generated tool call. Importantly, in our execution protocol, text accompanying a tool call is not delivered to the user. Thus, this metric captures the co-occurrence of request-like text and tool calls in model outputs, rather than requests actually communicated to users. The dependency-violation metric further identifies calls that require information or authorization not yet obtained from the user.
Responses to feedback. ALF action comparisons use normalized action strings. For post-error repetition, eligible feedback contains Error:; the first parsed call in the following assistant turn is compared with the last parsed call in the preceding turn. These metrics measure how the agent changes its actions after an environment response.
Reference-query coverage. The retail subset contains trajectories with annotated get_* or find_* reference actions. Coverage requires matching all reference queries somewhere in the generated calls. It captures whether the agent retrieves the required information, independently of query order.
Semantic Review of Dependency Violations
We review 800 natural trajectories each from Base, the ALF teacher, the ALF student, and the ALF+ student. Qwen3.8-Max assesses tool-call attempts using the domain policy, tool schemas, preceding history, and current assistant output. Model identity, final task outcome, and future feedback are withheld. An independent human annotator additionally reviews 120 stratified tool-call events, including predicted violations, non-violations, and ambiguous cases, without access to model identities or task outcomes. Human judgments agree with the LLM reviewer on 118 of the 120 examined events.
A current-request dependency violation occurs when a call requires the information or authorization requested in that same turn. Values appearing only in schema examples or assistant assertions do not establish that the user has supplied them. Calls independent of the current request are assessed separately. Table 11 reports the turn-level counts and their coverage across trajectories and tasks.
Table 11 Natural-trajectory semantic review. The first two columns report turn-level numerator/denominator pairs. Rollout coverage counts trajectories containing at least one dependency violation. Task coverage counts tasks with at least one affected trajectory among their eight samples.
| Model | Ask+call / Ask | Dependent / Ask+call | Affected rollouts / 800 | Affected tasks / 100 |
| Base | 1/2,278 | 0/1 | 4 | 4 |
| ALF teacher | 14/2,081 | 4/14 | 3 | 3 |
| ALF | 5,022/5,328 | 4,411/5,022 | 446 | 98 |
| ALF+ | 5/2,055 | 0/5 | 0 | 0 |
Table 11.
For ALF, 87.8% of detected ask–call turns contain a call dependent on the unanswered request. Violations affect 98 of the 100 tasks, showing that the pattern extends across the evaluation set. The ALF+ student has no detected current-request dependency violations in these reviewed trajectories. Semantic review also covers attempts outside the lexical request detector. This accounts for Base’s four affected trajectories despite no violation in its single detected ask–call turn.
Complete Evaluation Results
Table 12 gives the absolute accuracies underlying the cross-domain heatmap and teacher-composition analysis. Each named student uses the checkpoint specified in Appendix § 10.§ 5. Table 13 separates performance into airline, retail, and telecom.
Table 12 Complete accuracies for the selected students.
| Model | CI | ALF | GR | |
| Base | 0.167 | 0.129 | 0.415 | 0.424 |
| GR | 0.380 | 0.214 | 0.425 | 0.413 |
| CI+GR | 0.513 | 0.386 | 0.453 | 0.445 |
| CI | 0.524 | 0.402 | 0.510 | 0.444 |
| ALF | 0.383 | 0.855 | 0.066 | 0.417 |
| 0.262 | 0.229 | 0.611 | 0.367 | |
| ALF+CI | 0.498 | 0.832 | 0.340 | 0.467 |
| CI+ | 0.502 | 0.284 | 0.678 | 0.458 |
| ALF+ | 0.338 | 0.834 | 0.689 | 0.367 |
| ALF+CI+ | 0.511 | 0.869 | 0.644 | 0.450 |
Table 12.
| Model | Airline | Retail | Telecom |
| Base | 0.325 | 0.481 | 0.394 |
| ALF teacher | 0.319 | 0.634 | 0.356 |
| teacher | 0.469 | 0.684 | 0.822 |
| CI | 0.425 | 0.653 | 0.409 |
| ALF | 0.019 | 0.119 | 0.037 |
| 0.369 | 0.528 | 0.816 | |
| ALF+CI | 0.125 | 0.400 | 0.388 |
| CI+ | 0.463 | 0.672 | 0.791 |
| ALF+ | 0.369 | 0.678 | 0.859 |
| ALF+CI+ | 0.275 | 0.662 | 0.809 |
Table 13. Per-domain task accuracy. Teacher rows provide reference performance; the remaining rows are 4B students.
CI’s largest gain occurs in retail. The ALF teacher also performs substantially better than the ALFtrained student on , consistent with the distinction between teacher competence and its transfer through source-domain supervision.
Token-Level Analysis of Preference-Guided Selection
We analyze a batch of rollouts sampled during Top95 training. All comparisons below use the same token sequences and model scores. For upper-tail comparisons, each rollout contributes as many highest- positions as excluded positions.
Feedback Concentration and Direction
We measure feedback using the clipped ordinary advantage at unit policy ratio:
where is the number of valid assistant positions. The excluded share of absolute coefficient mass is
We average this ratio equally across rollouts. The statistic measures frozen feedback coefficients before importance weighting and optimizer transformations.
Top95 excludes 41.02% of ordinary absolute coefficient mass, compared with 5.56% for Random95. Of the mass excluded by Top95, 39.59 percentage points come from negative advantages and 1.43 from positive advantages. Selection therefore primarily removes feedback that suppresses student-generated continuations.
Negative advantages account for 88.49% of ordinary mass on these contexts. Top95 retains 48.90 percentage points as negative feedback, preserving correction at other positions. After selected-token normalization, retained absolute mass is 62.02% of ordinary OPD for Top95 and 99.33% for Random95.
RL-induced preference changes and OPD feedback. Figure 9 relates the selection signal to ordinary OPD feedback. Positions are ranked by signed within each rollout, and the curves average the corresponding bin means equally across rollouts.
Figure 9. RL-induced log-probability changes and OPD advantages on Top95 training contexts. Curves show mean ordinary advantage, pre-domain-RL reference advantage, and signed across within-rollout preference percentiles. Shading marks the excluded tail.
In the lowest- tail, both the pre-domain-RL reference and the specialist assign negative advantages on average, with the specialist providing stronger suppression. Domain-specific RL therefore strengthens existing corrective feedback at many excluded positions. For 16.05% of excluded positions, it also reverses the feedback direction: while . Together with the coefficient-mass analysis above, this pattern shows that the excluded tail concentrates feedback intensified by specialization.
At the opposite tail, is positive while the mean ordinary advantage remains negative. Specialization increases the teacher’s relative preference for these continuations, but the student can still assign them higher probability than the specialist. Their corrective feedback remains supervised. The middle percentiles have mean preference changes and advantages close to zero. Thus, the pre/post-RL comparison determines where supervision is applied, while the teacher–student comparison determines the direction of the retained update. The occurrence-level distributions in Figure 10 further illustrate this distinction.
Figure 10. Token-occurrence densities of signed against ordinary OPD advantage and pre-domain-RL reference advantage. Axes use symmetric-log scaling, and colors indicate occurrence counts.
Token Frequency and Context Dependence
For token type or structural category , let count all occurrences and excluded occurrences. We distinguish its share of all positions, , its share of exclusions, , and its exclusion rate, . Enrichment is the exclusion rate divided by the overall rate . Token identities preserve case and leading whitespace.
Figure 11 compares the frequency of token types in all and excluded positions. For example, the leading-space token check is excluded in 99/425 occurrences, whereas need is excluded in 104/2,967. The opening tool-call delimiter is excluded in 128/669 occurrences. These counts distinguish common vocabulary from positions preferentially removed by selection.
Figure 11. Frequent excluded token types. Bars show their shares of all valid positions and excluded positions. A centered dot denotes leading whitespace.
Preference changes also vary across occurrences of the same token. Of the 547 token types in the excluded tail, 366 also occur in the matched highest- tail. Figure 12 shows this variation for eight token types. Selection responds to a token’s use in its interaction history, rather than giving each token type a fixed selection status.
Figure 12. Signed for eight token types across their observed histories, ordered by retained fraction. Red occurrences are excluded and blue occurrences retained. Vertical jitter separates occurrences. A centered dot denotes leading whitespace.
Illustrative Training Excerpts
The following excerpts show assistant tokens from Top95 training. Background color represents signed : red indicates decreased preference after domain RL, white indicates little change, and green indicates increased preference. Color saturation is clipped at . Underlining marks positions excluded by the within-rollout Top95 mask.
-6 −3 0 +3 +6
1. Tool-call initiation (unsuccessful rollout)
Assistant
<tool_call>
{"name": "alfworld_step", "arguments": {"action": "inventory"}}
</tool_call><|im_end|>
[Unscored feedback omitted]
Assistant
<tool_call>
{"name": "alfworld_step", "arguments": {"action": "look"}}
</tool_call><|im_end|>
[Unscored feedback omitted]
Assistant
<tool_call>
{"name": "alfworld_step", "arguments": {"action": "go to cabinet 1"}}<|im_end|>
[End of displayed excerpt.]
Selected scored positions (, ordinary ; mask status):
<tool_call>: −5.37, −12.43; excluded.
inventory: +6.00, −7.45; retained.
cabinet: −1.48, −4.68; retained.
<|im_end|>: −1.40, −1.37; retained.
The first excerpt locates selection at entry into a tool call. The opening delimiter is excluded, while much of the enclosed action content remains supervised. The retained token inventory has positive but negative ordinary advantage, illustrating how a retained position can still receive suppressive OPD feedback.
2. Planning and action preparation (successful rollout)
Assistant
I need to find a remotecontrol and put it in the sofa. First, I should check where the remotecontrol might be. Since remotecontrols are often on tables, in drawers, or on shelves, I’ll start by going to the diningtable to check if the remotecontrol is there.
<tool_call>
{"name": "alfworld_step", "arguments": {"action": "go to diningtable 1"}}
</tool_call><|im_end|>
[Unscored feedback omitted]
Assistant
I found the remotecontrol on the diningtable. Now, I need to pick it up and put it in the sofa. First, I’ll take the remotecontrol from the diningtable.
<tool_call>
{"name": "alfworld_step", "arguments": {"action": "take remotecontrol 1 from diningtable 1"}}
</tool_call><|im_end|>
[Unscored feedback omitted]
Assistant
I now have the remotecontrol in my inventory. The next step is to go to the sofa and place the remote control inside it.
2. Planning and action preparation (successful rollout) (continued)
[End of displayed excerpt.]
Selected scored positions (, ordinary ; mask status):
put: +1.12, +0.56; retained. check: −1.77, −1.94; excluded. remote: +1.12, −0.51; retained. on: +2.48, +0.00; retained. in: −1.38, −10.56; excluded.
The second excerpt connects selection to planning and action preparation. The agent describes a search-and-placement sequence, with different preference changes within the same plan. The retained token put receives positive ordinary feedback, while excluded occurrences such as check and in receive negative feedback. These examples connect the coefficient analysis to concrete uses of language and tool-call boundaries.
Additional Supervision Experiments
We examine how supervision coverage, output structure, and feedback direction affect source learning and cross-domain interaction. The ALF-source controls supplement the update-20 comparison in Table 5. A separate GR-prefix experiment examines whether supervising the beginning of a response changes subsequent mathematical tool use.
| Approach | ALF acc. | acc. | Ask–call rate |
| Signed- Top95 | 76.64 | 29.88 | 24.26 |
| Delimiter masking | 76.28 | 4.88 | 99.37 |
| Full-block masking | 79.74 | 3.00 | 99.72 |
| Negative masking | 80.02 | 3.63 | 83.54 |
| Advantage Top95 | 81.66 | 23.50 | 82.68 |
| Advantage Top98 | 83.03 | 5.88 | 99.16 |
| Absolute- Top95 | 85.86 | 17.13 | 83.94 |
Table 5. Results of alternative masking approaches.
Learning Dynamics across Retained Fractions
Figure 13 shows source learning and target performance throughout the selection experiments. Evaluation accuracy is measured at initialization and updates 10 and 20. Training curves report mean ALF task reward on each sampled rollout batch.
Figure 13. Learning dynamics across retained fractions. Evaluation curves show ALF and accuracy at initialization and updates 10 and 20. Dashed curves show sampled ALF training reward at updates 1–20. All values are percentages.
The trajectories complement the final-checkpoint comparison by showing how source learning and target degradation develop under different supervision coverage.
Structural Masking
Structural masking tests whether removing supervision at fixed tool-call positions preserves target interaction.
Delimiter masking removes loss on <tool_call> and </tool_call>, while retaining the payload and surrounding text. Full-block masking additionally removes the enclosed payload. Its span detector tracks opening-minus-closing counts across the serialized response, so an unmatched opening can extend masking into subsequent assistant positions. Both masks use retained-token normalization (Equation (6)).
Table 5 reports their update-20 task outcomes and ask–call co-occurrence. Call construction differs substantially between the two masks: Delimiter produces 4,424 invalid-JSON spans among 15,441 complete calls, compared with 33 among 11,905 for Full-block. Fixed structural masking therefore affects call construction as well as the organization of interaction.
Negative Masking
Negative masking tests whether removing all suppressive feedback reproduces the effect of selective supervision. It uses
where counts all valid assistant positions. The full-token denominator is retained when negative advantages are removed.
Negative masking learns the source task but develops unsupported tool selections on . At update 20, 7,531 of 8,573 structurally valid, JSON-parseable calls use names absent from the supplied tool schemas. These calls occur in 726 of 800 trajectories and include wrapper names such as call_tool and action.
On the fixed Top95 diagnostic contexts, negative masking retains 11.51% of ordinary absolute coefficient mass. The intervention therefore changes both feedback direction and its retained strength. Figure 14 compares its learning dynamics with the structural and random-selection controls.
Figure 14. Learning dynamics of Random95, delimiter masking, full-block masking, and negative masking. Evaluation curves report ALF and accuracy; training curves report sampled ALF task reward.
Ranking Controls and Feedback Strength
Selection rules. All ranking controls select positions separately within each multi-turn rollout. Among its valid assistant positions, receive ordinary OPD loss, normalized by . The advantage-ranked controls retain the highest reward-signed values, with or 0.98 in the main comparison. Absolute- Top95 retains the highest values at , including large changes of either sign. All retain the ordinary signed OPD advantage at selected positions. They do not replace it with the ranking score.
Coefficient diagnostics. Table 14 applies each rule to the same saved Top95 diagnostic contexts and frozen model scores. Excluded mass is the share of absolute ordinary OPD coefficients on excluded positions, computed with full-token normalization. Retained mass additionally accounts for selected-token normalization: within each rollout, its ratio to ordinary OPD is . Both statistics are then averaged over rollouts. They are coefficient diagnostics, not parameter gradient norms; gradient directions and optimizer state are not matched by these calculations.
| Rule | Excluded mass (%) | Retained mass (%) |
| Signed- Top95 | 41.02 | 62.02 |
| Advantage Top95 | 66.88 | 34.83 |
| Advantage Top98 | 43.56 | 57.54 |
| Absolute- Top95 | 0.021 | 105.15 |
Table 14. Fixed-context feedback diagnostics. Excluded mass uses full-token normalization; retained mass includes the rescaling. Percentages are rollout-level averages.
Advantage Top98 removes a comparable amount of coefficient mass on these contexts but selects different positions: its excluded set covers only 19.16% of the signed- Top95 tail. The resemblance in mass is not maintained by construction during training. On eight saved Advantage Top98 contexts at update 20, the actual mask retains 68.55% of ordinary coefficient mass, whereas applying a virtual signed- Top95 mask to the same contexts retains 78.36%. Absolute- Top95 removes only 0.021% of coefficient mass on the fixed Top95 contexts. Selected-token normalization then raises retained mass to 105.15% of ordinary OPD. These observations explain why matching the number of retained positions does not match feedback strength, and why the ranking results do not fully separate selection content from optimization strength.
Learning dynamics. Figure 15 shows the three added main-text controls over the first 20 updates. Each evaluation contains 1,096 ALF episodes and 800 rollouts. The latter cover 100 tasks with eight samples per task. Ask–call co-occurrence uses the number of generated assistant turns containing matched request language as its denominator.
Figure 15. Ranking-control dynamics over 20 updates. Solid curves show ALF and evaluation accuracy at initialization and updates 10 and 20. Light dashed curves show mean ALF training reward on each sampled batch. All values are percentages.
GR-to-CI Selective-Supervision Extension
Teacher, data, and training. The extension uses a self-trained Qwen3-30B-A3B GR teacher at its 60-update RL checkpoint and the original pre-RL Qwen3-30B-A3B-Instruct-2507 reference. Both students start from Qwen3-4B-Instruct-2507 and train on 17,916 tool-free GR prompt rows, containing 17,254 unique questions after removal of an exact held-out question overlap. Both conditions use a prompt batch size of 32, four sampled rollouts per prompt, a PPO minibatch size of 32, and an actor learning rate of . Prompt and response limits are 8,192 and 32,768 tokens. They share the dual-scorer implementation, with retained fractions of 1.0 and 0.95, respectively, and complete 20 student updates.
Paired interfaces. CI and GR use identical mathematical questions, answers, and repeat identifiers, with their task-specific system prompts; only CI exposes the code interpreter. Each interface contains 980 expanded samples: 30 AIME2026 questions sampled 16 times and 100 BeyondAIME questions sampled five times. Evaluation uses temperature 1.0 and top-p 1.0. Accuracy is the fraction of correct expanded samples, rather than a uniform average over the 130 unique questions. Tool attempts, executed calls, and correct answer routes are recorded separately.
Figure 16. GR-to-CI learning dynamics. Lines connect the scheduled evaluations at initialization and update 10 with the update-20 companion evaluation reported in Table 6.
The update-20 checkpoints are evaluated once more under fixed-checkpoint sampling, with no further training. Table 6 reports this companion evaluation.
The CI response-length-limit rate is 3.78% under Full OPD and 0.10% under Top95; corresponding GR rates are 4.90% and 0.10%. These measures describe output-length behavior and are not direct estimates of repetition.
GR Prefix Supervision
The beginning of a mathematical response often introduces the problem representation and solution plan before carrying out detailed derivations or external computation. Inspired by Ziheng et al. [57], we therefore ask whether concentrating OPD supervision on this early prefix can influence subsequent tool use. The motivation is to emphasize strategy formation while reducing supervision on the remainder of the solution trajectory.
The GR-prefix experiment retains valid supervision within the first 1,000 positions of the serialized response tensor:
where identifies assistant-generated positions with teacher scores. The position budget is shared across assistant turns. Interleaved observations occupy serialized positions but receive no loss. Selection is based on position, with ordinary OPD feedback applied to the retained tokens.
Figure 17 compares prefix supervision with full GR supervision through 60 updates. Under prefix supervision, the tool-enabled mathematical tool-attempt rate falls from 71.0% to 21.6%, while accuracy rises from 17.6% to 33.1%. Full GR reaches 38.1% accuracy with an 18.0% tool-attempt rate at the same update. Source GR accuracy is 36.6% for the prefix student and 39.3% for full supervision.
Figure 17. GR-prefix and full-supervision learning dynamics through 60 updates. Panels report source GR accuracy, tool-enabled mathematical accuracy, and tool-attempt rate.
Restricting feedback to the response prefix therefore still produces a substantial shift away from tool use. The higher accuracy under full supervision indicates that feedback on later solution content contributes additional learning.
Delta-Based Feedback Exploration
Inspired by On-Policy Delta Distillation [10], we also explore directly using pre/post-domain-RL preference differences as distillation feedback. Specifically, we center the delta feedback under the student distribution and retain it when its sign agrees with centered ordinary OPD feedback. Under our multi-turn training setup, this variant exhibits generation degeneration: after 20 updates, both ALFWorld and accuracies drop to 0%, and all 800 rollouts reach the response-length limit without generating complete tool calls. This result motivates using for position selection while retaining the ordinary OPD advantage.
Training Prompts and Tool Interfaces
We reproduce the tool-use instructions and interfaces used in our experiments. The common call format is shown once, followed by the domain-specific prompts. Task instances are omitted; the policy excerpts retain their explicit omission markers.
Common Tool-Call Format
CI, ALFWorld, and use the following call format.
Shared tool-call serialization
For each function call, return a json object with function name and arguments within <tool_call></tool_call> XML tags:
<tool_call>
{"name": <function-name>, "arguments": <args-json-object>}
</tool_call>
Code Interpretation
The CI prompt allows Python execution for calculation, symbolic manipulation, and verification.

Figure 1 (PDF p. 30).
You are a math expert. You are given a question and you need to solve it step by step. You should use the ‘code_interpreter‘ tool to write and execute Python code for calculations, symbolic math, or result verification when needed. Put your final answer in \boxed{}.
{
"type": "function",
"function": {
"name": "code_interpreter",
"description": "Execute Python source code and return its output. Use this tool for exact calculations, symbolic math, enumeration, simulation, and verification.",
"parameters": {
"type": "object",
"properties": {
"code": {
"type": "string",
"description": "The Python code to execute."
}
},
"required": [
"code"
]
}
}
}
ALFWorld
The ALFWorld prompt supplies an observation and admissible actions, then asks the agent to reason and execute one listed action.

Figure 2 (PDF p. 30).
You are an expert agent operating in the ALFRED Embodied Environment. At each step you are given your current observation and the list of admissible actions. Reason step-by-step about the current situation, then take exactly one admissible action by calling the ‘alfworld_step‘ tool with that action. The action you pass MUST be copied verbatim from the admissible action list.
{
"type": "function",
"function": {
"name": "alfworld_step",
"description": "Take one admissible action in the ALFWorld environment. Choose exactly one action from the current admissible action list.",
"parameters": {
"type": "object",
"properties": {
"action": {
"type": "string",
"description": "The admissible action to take, copied verbatim from the admissible action list."
}
},
"required": [
"action"
]
}
}
}
Conversational Tool Use
The three domains share turn-level instructions and use their respective domain policies and tool interfaces.

Figure 3 (PDF p. 30).
You are a customer service agent. Follow the domain operating rules exactly. On each turn, do exactly one of:
send a message to the customer; or
call exactly one available tool. Do not combine customer-facing text and a tool call in the same turn. You may call only tools listed in the tool interface supplied to you. Actions described as customer-side or device-side are performed by the customer, not by you. When such an action is needed, explain it to the customer and ask the customer to perform it. Use the done tool only when the task is complete or cannot be completed.
Airline: policy excerpt and tool fields
As an airline agent, you can help users **book**, **modify**, or **cancel** flight reservations. You also handle **refunds and compensation**.
[… other policy clauses omitted …]
Before taking any actions that update the booking database (booking, modifying flights, editing baggage, changing cabin class, or updating passenger information), you must list the action details and obtain explicit user confirmation (yes) to proceed.
[… other policy clauses omitted …]
You should only use tools provided in the agent tool interface. Do not attempt to call procedures that are merely described in the policy or in the customer’s request. You should only make one tool call at a time, and if you make a tool call, you should not respond to the user simultaneously. If you respond to the user, you should not make a tool call at the same time.
Example tool-interface fields (other fields omitted):
{
"name": "get_reservation_details",
"description": "Get the details of a reservation.",
"required": [
"reservation_id"
]
}
Retail: policy excerpt and tool fields
At the beginning of the conversation, you have to authenticate the user identity by locating their user id via email, or via name + zip code. This has to be done even when the user already provides the user id.
[… other policy clauses omitted …]
Before taking any action that updates the database (cancel, modify, return, exchange), you must list the action details and obtain explicit user confirmation (yes) to proceed.
[… other policy clauses omitted …]
You should use only tools provided in the agent tool interface. Do not attempt to call procedures that are merely described in the policy or in the customer’s request. You should at most make one tool call at a time, and if you take a tool call, you should not respond to the user at the same time. If you respond to the user, you should not make a tool call at the same time.
Example tool-interface fields (other fields omitted):
{
"name": "modify_pending_order_payment",
"description": "Modify the payment method of a pending order. The agent needs to explain the modification detail and ask for explicit user confirmation (yes/no) to proceed.",
"required": [
"order_id",
"payment_method_id"
]
}
Telecom: policy excerpt and tool fields
As a telecom agent, you can help users with **technical support**, **overdue bill payment**, **line suspension**, and **plan options**.
[… other policy clauses omitted …]
You must first identify the customer.
[… other policy clauses omitted …]
Customer-side device procedures
The following procedures are performed by the customer on their own device. They are not callable tools for the customer service agent. When one of these procedures is needed, explain the steps to the customer and ask the customer to perform them. Do not emit a tool call using any of these procedure names. You must understand these procedures because technical support may require you to guide the customer through them.
[… other policy clauses omitted …]
Troubleshooting Service Problems
Airplane Mode
Airplane Mode is a feature that disables all wireless radios, including cellular. If it is enabled, it will prevent any
/ Telecom: policy excerpt and tool fields (continued)
cellular connection. Ask the customer to check whether Airplane Mode is ON using the status bar or network settings. If it is ON, ask the customer to turn it OFF.
Example tool-interface fields (other fields omitted):
{
"name": "enable_roaming",
"description": "Enables international roaming on a line.",
"required": [
"customer_id",
"line_id"
]
}
Case Studies
We present two complete comparisons: an urn problem illustrating mathematical solution routes, and a retail task illustrating information retrieval and confirmation before action. Each comparison shows the input, assistant outputs, and returned feedback in their original order. Retail initial user messages are shown separately because their wording differs.
Enumeration and Algebraic Derivation
The two models solve the same urn problem. Base uses Python enumeration, while the GR student derives an integer parametrization of the feasible color counts. Both obtain the answer 190.
Urn problem: Base, correct
User (question)
An urn contains marbles. Each marble is either red or blue, and there are at least 7 marbles of each color. When 7 marbles are drawn randomly from the urn without replacement, the probability that exactly 4 of them are red equals the probability that exactly 5 of them are red. Find the sum of the five least values of for which this is possible.
Assistant
The probability that exactly of the 7 marbles drawn are red is given by the hypergeometric distribution. In this distribution, the probability is:
where is the number of red marbles in the urn, is the number of blue marbles, and is the total number of marbles.
We are told that the probability of drawing exactly 4 red marbles is equal to the probability of drawing exactly 5 red marbles:
The denominators are the same, so they cancel out. This simplifies to:
We can use properties of binomial coefficients to simplify this equation. Let’s expand the terms:
Substituting these into the equation:
We can cancel out the common terms ( and ) from both sides:
This simplifies to:
Now we can directly compute the values for small and , ensuring that both and . Since the problem asks for the sum of the five least values of , we will iterate over possible , and for each , check if there exists an such that the equation holds and both and .
Let’s write the code to compute these values.
Tool call: code_interpreter (decoded code argument)
from math import comb
def check_probability_equation(n):
Urn problem: Base, correct (continued)
for r in range(7, n - 6): # r >= 7, n - r >= 7 implies r <= n - 7
if comb(r, 4) * comb(n - r, 3) == comb(r, 5) * comb(n - r, 2):
return True
return False
# Find the five smallest values of n for which the probability condition holds
valid_n_values = []
n = 14 # Starting from n = 14 to satisfy both conditions on r and n - r
while len(valid_n_values) < 5:
if check_probability_equation(n):
valid_n_values.append(n)
n += 1
sum_of_valid_n_values = sum(valid_n_values)
valid_n_values, sum_of_valid_n_values
Tool response
([22, 30, 38, 46, 54], 190)
Assistant
The five smallest values of for which the probability condition holds are 22, 30, 38, 46, and 54. The sum of these values is:
Thus, the final answer is 190.
Urn problem: GR student, correct
User (question)
An urn contains marbles. Each marble is either red or blue, and there are at least 7 marbles of each color. When 7 marbles are drawn randomly from the urn without replacement, the probability that exactly 4 of them are red equals the probability that exactly 5 of them are red. Find the sum of the five least values of for which this is possible.
Assistant
We are told:
An urn contains marbles, with at least 7 red and 7 blue marbles.
When 7 marbles are drawn without replacement, the probability of getting exactly 4 red is equal to the probability of getting exactly 5 red.
We are to find the sum of the five least values of for which this is possible.
Step 1: Define variables
Let:
= number of red marbles in the urn
= number of blue marbles in the urn
We are told , , and .
We are drawing 7 marbles without replacement.
The probability of drawing exactly red marbles is given by the hypergeometric distribution:
We are told:
Since the denominator is the same and nonzero, we can cancel it:
Step 2: Use identities
Recall:
Urn problem: GR student, correct (continued)
But we can use the identity:
Similarly,
Let’s plug into the equation:
Substitute:
Now divide both sides by (assuming they are nonzero, which they are for , ):
Now solve for in terms of :
We want and to be integers, , , and . So must be an integer. Therefore,
But , so:
So
Let’s write , for integer
Then:
So:
We now need:
(since integer)
So
Let’s compute the values of for , and find the five smallest such .
:
:
:
:
Urn problem: GR student, correct (continued)
:
So the five least values of are:
Sum them:
[check mark] Final Answer:
The contrast is in the solution procedure: Base searches over candidate totals and red counts, whereas GR reduces the probability equality to a linear relation and enumerates the resulting integer parameter. The final answer agrees across the two routes.
Retrieval and Confirmation in Retail
Both trajectories concern the same annotated payment-change task. Base retrieves user information but does not query the order, then assumes that the requested card is already in use. The CI-trained student retrieves the order state, checks the gift-card balance, obtains confirmation of an alternative card, and modifies the pending order.
Retail payment change: Base, unsuccessful
User (initial message)
Hi, I need some quick help. Can you tell me how much balance is left on my gift card? Also, I have a recent order and I’m trying to remember which credit card I used for it—was it Visa, Mastercard, or Amex?
Assistant
To assist you with your gift card balance and the credit card used in your recent order, I’ll first need to authenticate your identity. Could you please provide your email address?
User
Sure, my email is [email protected]. I’m kind of in a rush, so I’d really appreciate it if we could get to the gift card balance and my order details quickly.
Assistant
<tool_call>
{
"name": "find_user_id_by_email",
"arguments": {
"email": "[email protected]"
}
}
</tool_call>
Tool response
isabella_lopez_6490
Assistant
<tool_call>
{
"name": "get_user_details",
"arguments": {
"user_id": "isabella_lopez_6490"
}
}
</tool_call>
Tool response
{"user_id": "isabella_lopez_6490", "name": {"first_name": "Isabella", "last_name": "Lopez"}, "address": {"address1": "710
Sunset Drive", "address2": "Suite 176", "city": "Phoenix", "country": "USA", "state": "AZ", "zip": "85034"}, "email":
"[email protected]", "payment_methods": {"credit_card_8897086": {"source": "credit_card", "id": "
Retail payment change: Base, unsuccessful (continued)
credit_card_8897086, "brand": "visa", "last_four": "8902"}, "gift_card_8245350": {"source": "gift_card", "id": "gift_card_8245350", "balance": 60.0}, "credit_card_8554680": {"source": "credit_card", "id": "credit_card_8554680", "brand": "mastercard", "last_four": "4336"}, "paypal_1621947": {"source": "paypal", "id": "paypal_1621947"}, "orders": ["\#W4923227"]}
Assistant Your gift card balance is $60.00. For your recent order (#W4923227), you used a Visa card ending in 8902. Let me know if you need any further assistance!
User Okay, thank you for that. So I want to confirm—if my order already went through on the Visa, can I just leave it that way, because I prefer not to use my other cards? And more importantly, can I apply that $60 gift card balance to my order now?
Assistant Your order is already processed, so no changes to the payment method are needed. Since the order has already been fulfilled, you cannot apply the gift card balance to it directly. However, if you’d like to use the $60 gift card balance for a future purchase, you can do so when placing a new order. Let me know if you need help with anything else!
User Hmm, that’s a bit of a bummer. Since I really don’t want to use my Mastercard or Amex because of their balances, is there any way you can still put the gift card toward this order—maybe as a partial payment adjustment or a store credit for that amount? Or would I have to return something and then rebuy it to use the gift card?
Assistant Unfortunately, gift card balances cannot be applied as partial payments or store credits to existing orders once they have been fulfilled. The gift card can only be used for future purchases when placing new orders. To use the balance, you would need to place a new order and apply the gift card during checkout. If you’d like, I can help you with any other questions or assist with future orders. Let me know how I can support you further!
Retail payment change: CI student, successful
User (initial message) Hi, I need some help quickly—I’m trying to check the balance on my gift card, and I also want to see which credit card I used for my recent order. I might want to use the gift card on that order instead, but if I can’t, I’d like to switch the payment to my Visa. Can you look into that for me?
Assistant I can help you check your gift card balance and see which credit card was used for your recent order. First, I’ll need to authenticate your identity to access your profile and order details. Could you please provide your email address or your first name, last name, and zip code?
User Sure, my email is [email protected]. Can you check everything with that?
Assistant
<tool_call>
{
"name": "find_user_id_by_email",
"arguments": {
"email": "[email protected]"
}
}
</tool_call>
Tool response
isabella_lopez_6490
Assistant Thank you, Isabella. I have successfully authenticated your identity. Now, I will retrieve your gift card balance and the payment method used for your recent order. Let me check that information for you. First, I’ll get your user details to see your gift card balance and payment methods. Then, I’ll look up your recent order to identify which credit card was used.
Retail payment change: CI student, successful (continued)
Let me proceed with getting your user details.
<tool_call>
{
"name": "get_user_details",
"arguments": {
"user_id": "isabella_lopez_6490"
}
}
</tool_call>
Tool response
{"user_id": "isabella_lopez_6490", "name": {"first_name": "Isabella", "last_name": "Lopez"}, "address": {"address1": "710 Sunset Drive", "address2": "Suite 176", "city": "Phoenix", "country": "USA", "state": "AZ", "zip": "85034"}, "email": "[email protected]", "payment_methods": {"credit_card_8897086": {"source": "credit_card", "id": "credit_card_8897086", "brand": "visa", "last_four": "8902"}, "gift_card_8245350": {"source": "gift_card", "id": "gift_card_8245350", "balance": 60.0}, "credit_card_8554680": {"source": "credit_card", "id": "credit_card_8554680", "brand": "mastercard", "last_four": "4336"}, "paypal_1621947": {"source": "paypal", "id": "paypal_1621947"}}, "orders": ["#W4923227"]}
Assistant
I now have your user details. Here’s what I found:
Gift Card Balance: $60.00 (ID: gift_card_8245350)
Payment Methods: - Visa ending in 8902 (ID: credit_card_8897086) - Mastercard ending in 4336 (ID: credit_card_8554680) - PayPal account (ID: paypal_1621947)
Next, I’ll check the details of your recent order to see which credit card was used. Your recent order ID is #W4923227. Let me retrieve that order information.
<tool_call>
{
"name": "get_order_details",
"arguments": {
"order_id": "#W4923227"
}
}
</tool_call>
Tool response
{"order_id": "#W4923227", "user_id": "isabella_lopez_6490", "address": {"address1": "710 Sunset Drive", "address2": "Suite 176", "city": "Phoenix", "country": "USA", "state": "AZ", "zip": "85034"}, "items": [{"name": "Bluetooth Speaker", "product_id": "4768869376", "item_id": "7751905257", "price": 321.18, "options": {"color": "red", "battery life": "10 hours", "water resistance": "yes"}}], "status": "pending", "fulfillments": [], "payment_history": [{"transaction_type": "payment", "amount": 321.18, "payment_method_id": "credit_card_8554680"}], "cancel_reason": null, "exchange_items": null, "exchange_new_items": null, "exchange_payment_method_id": null, "exchange_price_difference": null, "return_items": null, "return_payment_method_id": null}
Assistant
Here are the details of your recent order:
Order ID: #W4923227
Status: Pending
Item: Bluetooth Speaker (Color: Red, Battery Life: 10 hours, Water Resistance: Yes)
Price: $321.18
Payment Method Used: Mastercard ending in 4336 (ID: credit_card_8554680)
You have a gift card balance of $60.00, which is not sufficient to cover the full order amount of $321.18. Therefore, you cannot use the gift card to pay for this order.
However, you can switch the payment method to your Visa ending in 8902 (ID: credit_card_8897086), which is a valid option for this pending order.
Would you like me to proceed with modifying the payment method of your order to your Visa card? Please confirm with a "yes" if you’d like to proceed.
Note: This change will keep the order in "pending" status, and the original Mastercard payment will be refunded within 5–7 business days.
User
Yes, please switch it to my Visa. Can you confirm once it’s done? I’m trying to wrap this up quickly.
Assistant
I will now modify your order’s payment method to your Visa card (ending in 8902). Let me proceed with that action.
<tool_call>
{
"name": "modify_pending_order_payment",
"arguments": {
"order_id": "#W4923227",
"payment_method_id": "credit_card_8897086"
``
Retail payment change: CI student, successful (continued)
} } </tool_call>
Tool response
{"order_id": "#W4923227", "user_id": "isabella_lopez_6490", "address": {"address1": "710 Sunset Drive", "address2": "Suite 176", "city": "Phoenix", "country": "USA", "state": "AZ", "zip": "85034"}, "items": [{"name": "Bluetooth Speaker", "product_id": "4768869376", "item_id": "7751905257", "price": 321.18, "options": {"color": "red", "battery life": "10 hours", "water resistance": "yes"}}], "status": "pending", "fulfillments": [], "payment_history": [{"transaction_type": "payment", "amount": 321.18, "payment_method_id": "credit_card_8554680"}, {"transaction_type": "payment", "amount": 321.18, "payment_method_id": "credit_card_8897086"}, {"transaction_type": "refund", "amount": 321.18, "payment_method_id": "credit_card_8554680"}], "cancel_reason": null, "exchange_items": null, "exchange_new_items": null, "exchange_payment_method_id": null, "exchange_price_difference": null, "return_items": null, "return_payment_method_id": null}
Assistant
The payment method for your order #W4923227 has been successfully updated to your Visa card ending in 8902. Here are the updated details:
Payment Method: Visa (ending in 8902)
Original Mastercard (ending in 4336) has been refunded within 5–7 business days. All actions are complete. If you need any further assistance, feel free to ask! You are all set. done
The complete interaction locates the difference in the information used before modification. CI checks the pending order and its payment history, then waits for confirmation before changing the payment method. Its mixed message–tool turns coexist with a successful query-and-confirmation sequence.
References
- [1]R. Agarwal, N. Vieillard, Y. Zhou, P. Stanczyk, S. Ramos Garea, M. Geist, and O. Bachem. On-policy distillation of language models: Learning from self-generated mistakes. In International Conference on Learning Representations, volume 2024, pages 21246–21263, 2024.arxiv.org/abs/2306.13649
- [2]Y. Aoki, K. Kudo, T. Kuribayashi, S. Sone, M. Taniguchi, K. Sakaguchi, and K. Inui. First heuristic then rational: Dynamic use of heuristics in language model reasoning. In Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, pages 14255–14271, 2024.
- [3]N. Arabzadeh, S. Huo, N. Mehta, Q. Wu, C. Wang, A. Hassan, C. L. Clarke, and J. Kiseleva. Assessing and verifying task utility in llm-powered applications. In Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, pages 21868–21888, 2024.arxiv.org/abs/2405.02178
- [4]V. Barres, H. Dong, S. Ray, X. Si, and K. Narasimhan. τ²-Bench: Evaluating Conversational Agents in a Dual-Control Environment. arXiv preprint arXiv:2506.07982, 2025.arxiv.org/abs/2506.07982
- [5]ByteDance-Seed. BeyondAIME: Advancing math reasoning evaluation beyond high school olympiads. Hugging Face Dataset, 2025. URL https://huggingface.co/datasets/ByteDance-Seed/BeyondAIME.
- [6]E. Debenedetti, J. Zhang, M. Balunovic, L. Beurer-Kellner, M. Fischer, and F. Tramèr. Agentdojo: A dynamic environment to evaluate prompt injection attacks and defenses for llm agents. Advances in neural information processing systems, 37:82895–82920, 2024.arxiv.org/abs/2406.13352
- [8]J. He-Yueya, G. Poesia, R. E. Wang, and N. D. Goodman. Solving math word problems by combining language models with symbolic solvers, 2023. URL https://arxiv.org/abs/2304.09102.
- [9]D. Hendrycks, C. Burns, S. Kadavath, A. Arora, S. Basart, E. Tang, D. Song, and J. Steinhardt. Measuring mathematical problem solving with the MATH dataset. In Thirty-fifth Conference on Neural Information Processing Systems Datasets and Benchmarks Track (Round 2), 2021. URL https://openreview.net/forum?id=7Bywt2m0sCe.
- [10]B. Heo, J. Hwang, S. Yun, and D. Han. On-policy delta distillation. arXiv preprint arXiv:2607.15161, 2026.arxiv.org/abs/2607.15161
- [11]G. Hinton, O. Vinyals, and J. Dean. Distilling the knowledge in a neural network. arXiv preprint arXiv:1503.02531, 2015.arxiv.org/abs/1503.02531
- [12]C. E. Jimenez, J. Yang, A. Wettig, S. Yao, K. Pei, O. Press, and K. Narasimhan. Swe-bench: Can language models resolve real-world github issues?, 2024. URL https://arxiv.org/abs/2310.06770.
- [13]W. Jin, T. Min, Y. Yang, D. Wei, Y. Zhou, S. R. Kadhe, N. Baracaldo, and K. Lee. Entropy-aware on-policy distillation of language models. arXiv preprint arXiv:2603.07079, 2026.arxiv.org/abs/2603.07079
- [14]J. Kirkpatrick, R. Pascanu, N. Rabinowitz, J. Veness, G. Desjardins, A. A. Rusu, K. Milan, J. Quan, T. Ramalho, A. Grabska-Barwinska, et al. Overcoming catastrophic forgetting in neural networks. Proceedings of the national academy of sciences, 114(13):3521–3526, 2017.DOI
- [15]T. Kojima, S. S. Gu, M. Reid, Y. Matsuo, and Y. Iwasawa. Large language models are zero-shot reasoners. Advances in neural information processing systems, 35:22199–22213, 2022.arxiv.org/abs/2205.11916
- [18]H. Liu, Z.-Y. Dou, Y. Wang, N. Peng, and Y. Yue. Uncertainty calibration for tool-using language agents. In Findings of the Association for Computational Linguistics: EMNLP 2024, pages 16781–16805, 2024a.DOI
- [19]X. Liu, H. Yu, H. Zhang, Y. Xu, X. Lei, H. Lai, Y. Gu, H. Ding, K. Men, K. Yang, et al. Agentbench: Evaluating llms as agents. In International Conference on Learning Representations, volume 2024, pages 52989–53046, 2024b.arxiv.org/abs/2308.03688
- [20]D. Lopez-Paz and M. Ranzato. Gradient episodic memory for continual learning. Advances in neural information processing systems, 30, 2017.
- [22]C. Qian, E. C. Acikgoz, Q. He, H. Wang, X. Chen, D. Hakkani-Tür, G. Tur, and H. Ji. Toolrl: Reward is all tool learning needs, 2025. URL https://arxiv.org/abs/2504.13958.
- [23]Y. Qin, S. Liang, Y. Ye, K. Zhu, L. Yan, Y. Lu, Y. Lin, X. Cong, X. Tang, B. Qian, et al. Toolllm: Facilitating large language models to master 16000+ real-world apis. In International Conference on Learning Representations, volume 2024, pages 9695–9717, 2024.arxiv.org/abs/2307.16789
- [24]T. Schick, J. Dwivedi-Yu, R. Dessì, R. Raileanu, M. Lomeli, E. Hambro, L. Zettlemoyer, N. Cancedda, and T. Scialom. Toolformer: Language models can teach themselves to use tools. Advances in neural information processing systems, 36:68539–68551, 2023.arxiv.org/abs/2302.04761
- [25]J. Schulman. Approximating KL divergence. http://joschu.net/blog/kl-approx.html, Mar. 2020. Blog post.
- [26]J. Shen, G. Chen, and C. Mao. When top-k misses the decision: Tool-call drift in multi-teacher on-policy distillation. arXiv e-prints, pages arXiv–2607, 2026a.arxiv.org/abs/2607.07050
- [27]Z. Shen, Y. Li, Q. Yin, C. T. Leong, Z. Wang, Y. Chen, R. Han, S. Lee, and Y. R. Fung. On the geometry of on-policy distillation. arXiv preprint arXiv:2606.07082, 2026b.arxiv.org/abs/2606.07082
- [28]N. Shinn, F. Cassano, A. Gopinath, K. Narasimhan, and S. Yao. Reflexion: Language agents with verbal reinforcement learning. Advances in neural information processing systems, 36:8634–8652, 2023.arxiv.org/abs/2303.11366
- [29]M. Shridhar, X. Yuan, M.-A. Côté, Y. Bisk, A. Trischler, and M. Hausknecht. ALFWorld: Aligning Text and Embodied Environments for Interactive Learning. In Proceedings of the International Conference on Learning Representations (ICLR), 2021. URL https://arxiv.org/abs/2010.03768.
- [30]M. Song and M. Zheng. A survey of on-policy distillation for large language models. arXiv preprint arXiv:2604.00626, 2026.arxiv.org/abs/2604.00626
- [31]T. Standley, A. Zamir, D. Chen, L. Guibas, J. Malik, and S. Savarese. Which tasks should be learned together in multi-task learning? In International conference on machine learning, pages 9120–9132. PMLR, 2020.arxiv.org/abs/1905.07553
- [32]M. Sullivan, M. Hartmann, and A. Koller. Procedural environment generation for tool-use agents. In Proceedings of the 2025 Conference on Empirical Methods in Natural Language Processing, pages 18555–18573, 2025.
- [33]Q. Tan, Z. Zong, Y. Li, and P. Chen. Atod: Annealed turn-aware on-policy distillation for multi-turn autonomous agents. arXiv preprint arXiv:2606.27814, 2026.arxiv.org/abs/2606.27814
- [34]K. Team, T. Bai, Y. Bai, Y. Bao, J. Cai, X. Cai, P. Cao, Y. Cao, Z. Chai, Y. Charles, et al. Kimi k3: Open frontier intelligence. arXiv preprint arXiv:2607.24653, 2026.
- [35]J. Wang, J. Lyu, Y. Cai, S. Wu, P. Wang, J. Liu, X. Cheng, K. Yang, Y. Chen, S. Yang, et al. Beyond prompt count: How data shapes transfer in on-policy distillation. arXiv preprint arXiv:2609.37377, 2026a.arxiv.org/abs/2609.37377
- [36]J. Wang, W. Zhang, W. Shi, Y. Li, and J. Cheng. Tcod: Exploring temporal curriculum in on-policy distillation for multi-turn autonomous agents. arXiv preprint arXiv:2604.24005, 2026b.arxiv.org/abs/2604.24005
- [37]X. Wang, J. Wei, D. Schuurmans, Q. Le, E. Chi, S. Narang, A. Chowdhery, and D. Zhou. Self-consistency improves chain of thought reasoning in language models. arXiv preprint arXiv:2203.11171, 2022.arxiv.org/abs/2203.11171
- [38]J. Wei, X. Wang, D. Schuurmans, M. Bosma, F. Xia, E. Chi, Q. V. Le, D. Zhou, et al. Chain-of-thought prompting elicits reasoning in large language models. Advances in neural information processing systems, 35:24824–24837, 2022.arxiv.org/abs/2201.11903
- [39]J. Wei, Z. Sun, S. Papay, S. McKinney, J. Han, I. Fulford, H. W. Chung, A. T. Passos, W. Fedus, and A. Glaese. Browsecomp: A simple yet challenging benchmark for browsing agents, 2025. URL https://arxiv.org/abs/2504.12516.
- [42]B. Xiao, B. Xia, B. Yang, B. Gao, B. Shen, C. Zhang, C. He, C. Lou, F. Luo, G. Wang, et al. Mimo-v2-flash technical report. arXiv preprint arXiv:2601.02780, 2026.arxiv.org/abs/2601.02780
- [45]Z. Xue, L. Zheng, Q. Liu, Y. Li, X. Zheng, Z. Ma, and B. An. Simpletir: End-to-end reinforcement learning for multi-turn tool-integrated reasoning. In International Conference on Learning Representations, volume 2026, pages 8424–8449, 2026.arxiv.org/abs/2509.02479
- [46]A. Yang, A. Li, B. Yang, B. Zhang, B. Hui, B. Zheng, B. Yu, C. Gao, C. Huang, C. Lv, et al. Qwen3 technical report. arXiv preprint arXiv:2505.09388, 2025.arxiv.org/abs/2505.09388
- [47]S. Yao, J. Zhao, D. Yu, N. Du, I. Shafran, K. Narasimhan, and Y. Cao. React: Synergizing reasoning and acting in language models. arXiv preprint arXiv:2210.03629, 2022.arxiv.org/abs/2210.03629
- [51]A. Zeng, X. Lv, Z. Hou, Z. Du, Q. Zheng, B. Chen, D. Yin, C. Ge, C. Huang, C. Xie, et al. Glm-5: from vibe coding to agentic engineering. arXiv preprint arXiv:2602.15763, 2026.arxiv.org/abs/2602.15763
- [52]Y. Zhang and Math-AI Team. American invitational mathematics examination (AIME) 2026, 2026. URL https://huggingface.co/datasets/math-ai/aime26.
- [53]Z. Zheng, J. Wu, S. Zhang, Z. Zhang, Y.-S. Ong, and S. Liu. Act first, reason later: Accelerating on-policy distillation for multi-turn agents via reference-conditioned inverse dynamics. arXiv preprint arXiv:2609.36608, 2026.arxiv.org/abs/2609.36608
- [54]Q. Zhong, M. Zheng, M. Song, X. Lin, J. Sun, H. Jiang, X. Wang, and J. Fang. Sod: Step-wise on-policy distillation for small language model agents. arXiv preprint arXiv:2605.07725, 2026.arxiv.org/abs/2605.07725
- [55]S. Zhou, F. F. Xu, H. Zhu, X. Zhou, R. Lo, A. Sridhar, X. Cheng, T. Ou, Y. Bisk, D. Fried, et al. Webarena: A realistic web environment for building autonomous agents. In International Conference on Learning Representations, volume 2024, pages 15585–15606, 2024.arxiv.org/abs/2307.13854
- [56]Y. Zhou, L. Zhang, Y. Wu, M. Wang, B. Peng, J. Liu, X. Fan, and Z. Zhao. Sage-opd: Selective agent-guided intervention for multi-turn on-policy distillation. arXiv preprint arXiv:2606.19659, 2026.arxiv.org/abs/2606.19659
- [57]Z. Ziheng, J. Li, H. Tang, Y. N. Wu, and D. Terzopoulos. Less is more: Early stopping rollout for on-policy distillation. arXiv preprint arXiv:2605.27028, 2026.












