betaFeedback

WorksComputer Science

SteerDistill: Quantizing LLM Agents via Closed-Loop Teacher Steering

Abstract

Low-bit quantization reduces the inference cost of LLM agents but can compromise task success. Examining failed rollouts, we find that quantized agents overlook task requirements or environmental conditions, producing environment transitions that diverge from successful full-precision references. Correcting these behavioral divergences remains difficult: offline data may miss the student's failure contexts, while its own rollouts may provide few correct responses in those contexts. We introduce SteerDistill, a quantization-aware distillation method in which a full-precision teacher guides the student to revise failed responses and iteratively refines its guidance using feedback from student execution. This closed-loop teacher steering produces locally verified behavioral corrections. We internalize these corrections by training the student without guidance to match the steered student's predictions at corrected turns and the full-precision teacher's at other turns, enabling unassisted inference. Across five agentic benchmarks and three models, SteerDistill exceeds the strongest quantized baseline by 1.67–2.18 points in average score under W4A16 and stays within one point of BF16. On τ2τ^2-Bench, it outperforms QeRL by 1.84 percentage points in success rate using 67% fewer training GPU hours and produces shorter successful trajectories than the quantized baselines.

Project: https://air-embodied-brain.github.io/SteerDistill

Code: https://github.com/air-embodied-brain/SteerDistill

Introduction

Large language model (LLM) agents call tools and interact with their environments over multiple turns to complete tasks [1, 3, 4, 5] [2]. Repeated model calls make inference cost a central deployment concern [6, 7, 8], and low-bit quantization offers a practical way to reduce it [9, 10, 11, 12, 13, 14]. However, quantized agents often fall short of their full-precision counterparts in task success.

Examining failed rollouts, we find that quantized agents overlook task requirements or environmental conditions, producing environment transitions that differ from those of successful full-precision references (Sections D.1 and D.2). We refer to this mismatch as behavioral divergence. For example, an agent tries to read a file before entering its directory, although the available reading tool only accesses files in the current directory (Figure 1 and Figure 11). In another task, an agent creates the requested files but leaves some empty instead of writing the specified contents (Figure 6).

Figure 1. SteerDistill constructs and internalizes behavioral corrections. In failure contexts where unguided rollouts provide few correct responses, closed-loop teacher steering helps the student revise its decisions by making overlooked task requirements and environmental conditions explicit. The teacher refines its guidance using execution feedback; locally verified corrections are then distilled for inference without teacher guidance. For Qwen3-8B under W4A16, SteerDistill approaches BF16 in five-benchmark mean score (a) and achieves higher τ2\tau^{2}-Bench success than QeRL with 67% fewer training GPU hours (b). Shading in (b) shows ±1\pm1 standard deviation across four evaluation runs at each checkpoint.

Figure 11. Entering the workspace before reading the file. The reading tool only accesses files in the current directory, so the unguided attempt to read the workspace file from /alex fails. Under teacher steering, the student checks its location, enters workspace, and retrieves the requested file contents.

However, identifying and correcting these behavioral divergences remain challenging. Offline training and distillation [15, 16, 17, 18] rely on data collected independently of the student’s interactions and may therefore miss the contexts in which it diverges. On-policy rollouts [19, 20, 21] address this coverage gap, but may provide few correct responses in those contexts. In our BFCL V4 multi-turn diagnostic, the student’s unguided retries yield a correct action at only 9.6% of divergent turns.1

Contribution. We introduce SteerDistill, a quantization-aware distillation (QAD) method for LLM agents (Figure 1). The key idea is to construct and internalize teacher-guided behavioral corrections. During training, a full-precision teacher identifies the task requirements and environmental conditions overlooked by the student and provides targeted guidance for choosing an appropriate action. The student generates and executes a revised response, and the teacher refines its guidance using execution feedback. We call this process closed-loop teacher steering. By distilling these teacher-guided behavioral corrections, we train the student to take appropriate actions without teacher guidance at inference.

Two designs implement this approach. (i) Teacher-steered trajectory revision constructs behavioral corrections by revising failed student rollouts through closed-loop teacher steering. We use successful teacher references to locate behavioral divergences. At each divergence, the student revises its response from the restored pre-error context, and locally verified corrections are retained for distillation. In the same diagnostic, teacher steering raises turn-level Pass@4 from 9.6% to 82.3%. (ii) Internalization of steered behavior distills the locally verified corrections while retaining full-precision teacher supervision at other turns. At corrected turns, the student learns from the frozen steered student’s predictions without receiving the guidance.

Without teacher guidance at inference, SteerDistill achieves the strongest quantized performance in our comparisons, with lower training cost and shorter successful trajectories. (i) Accuracy. Across five agentic benchmarks and three models under W4A162, it exceeds the strongest quantized baseline by 1.67–2.18 points in average score and stays within one point of BF16; on Qwen3-8B under W2A16, the margin grows to 3.39 points (Table 1). (ii) Training cost. On τ\tau-2-Bench, SteerDistill outperforms QeRL by 1.84 percentage points in success rate using 67% fewer training GPU hours (Figure 3a). (iii) Inference cost. On τ\tau-2-Bench, its successful tasks use 4.0K output tokens on average, close to BF16’s 3.9K, versus 4.1K–5.2K for the quantized baselines (Figure 3b). (iv) Design insights. Ablations support intervening at behavioral divergences, guiding the student to generate its own corrections, retaining locally verified corrections, and using different distillation objectives at corrected and other turns (Table 2).

ModelPrecisionMethodBFCL V4 Overallτ2\tau^2-Bench Pass@1WebShop SRALFWorld SRSearch QA EMAvg.
Qwen3 8BBF16–46.8932.6468.5879.9239.6553.54
W4A16GPTQ40.7326.8559.2471.3233.6446.36
EfficientQAT41.7728.0061.3473.1835.0847.87
BitDistiller42.4628.3162.0773.9135.6948.49
On-Policy QAD44.4328.7764.7276.5337.4750.38
QeRL44.9129.8465.2977.0437.9451.00
Ours46.8431.7968.4179.7539.1253.18
W4A8GPTQ35.9022.9054.7066.8030.2042.10
EfficientQAT40.2027.1060.0071.8034.0046.62
BitDistiller41.2027.8060.8072.9034.8047.50
On-Policy QAD43.1527.8563.3575.1036.2549.14
QeRL42.8529.0564.1075.5536.9049.69
Ours45.2030.1066.7078.2537.8551.62
W2A16GPTQ13.846.7230.2124.6318.9118.86
EfficientQAT30.9218.4449.1353.8731.4636.76
BitDistiller33.7119.8651.8457.3634.2239.40
On-Policy QAD32.8820.5753.0858.4132.9639.58
QeRL31.6420.9150.7455.6231.5838.10
Ours38.1223.6855.7163.2734.0542.97
Qwen3.5 4BBF16–50.4062.8069.8078.6046.8061.68
W4A16GPTQ43.9156.1860.6569.7840.5854.22
EfficientQAT45.3657.6862.7671.8442.0855.94
BitDistiller46.1758.3363.6172.5442.7756.68
On-Policy QAD48.0259.4866.0374.9244.4958.59
QeRL48.5360.3166.7375.6345.0159.24
Ours49.8261.9169.0877.8246.1160.95
Gemma 4 12B-itBF16–64.1861.4672.4184.6342.9665.13
W4A16GPTQ57.8254.3663.4875.7236.9157.66
EfficientQAT59.4656.1265.7277.6838.2459.44
BitDistiller60.0856.7466.4378.3138.7960.07
On-Policy QAD61.8758.3668.6180.7640.5262.02
QeRL62.4159.0269.3381.3541.0762.64
Ours63.5460.7271.3883.6242.3164.31

Table 1. Agentic benchmark results across models and quantization settings. Scores are percentages; higher is better. Avg. is the unweighted mean of the five displayed benchmark scores. Bold marks the best quantized result for each model and precision.

Figure 3. Efficiency analysis of Qwen3-8B under W4A16. (a) Training efficiency on τ2\tau^{2}-Bench. GPU hours are computed as end-to-end wall-clock time for low-bit training multiplied by the number of GPUs. For SteerDistill, this includes teacher steering and verification. All runs use 8 NVIDIA A100 80GB GPUs. At each checkpoint, curves show the mean Pass@1 over four independent evaluation runs, with shaded regions indicating ±1\pm1 standard deviation across those runs. (b) Inference efficiency on τ2\tau^{2}-Bench: average output tokens per successful task versus Pass@1.

Related work

Capability recovery for low-bit LLM quantization. To preserve model capabilities under low-bit quantization, one line of work improves the fidelity of quantized representations through weight reconstruction, scaling, and rotation [9, 10, 11, 22, 23, 24]. Complementary training-based approaches recover task performance using hard token targets [15, 16], soft distributions from full-precision teachers [17, 18, 20], or verifiable rewards [19, 20]. For quantized LLM agents, however, the available training data (including offline data and on-policy rollouts) may lack effective corrections in contexts where the student overlooks task requirements or environmental conditions. To address this, we extend quantization-aware distillation with closed-loop teacher steering to construct and internalize behavioral corrections (§ 3.1 and § 3.2.1).

Teacher guidance and context distillation for LLM agents. Teacher-generated responses [25, 26, 27] and rollouts conditioned on privileged context [28, 29] give students access to improved training trajectories. To assess the value of such assistance, FutureBridge-OPD [30] evaluates teacher interventions through the distillation signals in subsequent student rollouts. OVCSD [31] instead uses task success to verify privileged continuations from student failure states, providing outcome-grounded corrective supervision. Our closed-loop teacher steering instead evaluates student corrections at the divergent turn, checking their environment transitions against the successful reference and using feedback from failed attempts to refine guidance. For quantized agents, this local loop targets overlooked task requirements and environmental conditions. We internalize these corrections through context distillation, following prior work on transferring the benefits of privileged information to unguided learners [32, 33].

Method

SteerDistill constructs and internalizes behavioral corrections in each training cycle (Figure 2). Teacher-steered trajectory revision (§ 3.1) revises the current student’s failed rollouts through closed-loop teacher steering and retains locally verified corrections. Internalization of steered behavior (§ 3.2.1) then trains the student on the retained trajectories without guidance, learning from the steered student at corrected turns and the full-precision teacher at other turns. The updated student generates the next batch of rollouts.

Figure 2. One training cycle of SteerDistill. (i) Closed-loop teacher steering guides the frozen student to revise divergent turns and uses execution feedback to refine the guidance. Local verification compares the resulting environment transition with that of a successful teacher reference. On failure, the student retries with refined guidance from the same pre-error history and state. Verified corrections are retained, and rollout continues without guidance. (ii) To internalize these corrections, the learner receives no guidance and learns from the steered snapshot’s predictions at corrected turns and the full-precision teacher’s predictions elsewhere. Context cc contains the retained history and response prefix without steering prompts; only the steered target additionally receives GG. The updated student generates the next cycle’s rollouts.

Setup. Let πT\pi_T be the frozen full-precision teacher and πθ≡πQ(θ)\pi_\theta\equiv\pi_{Q(\theta)} the trainable quantized student initialized from the same checkpoint, where θ\theta denotes the master weights and QQ is a fixed quantization operator. At turn tt, the student generates a response yty_t given history hth_t, comprising the task, previous responses, and environment observations. A response may include reasoning followed by an action or answer. During cycle kk, a frozen snapshot πθk\pi_{\theta_k} generates the initial rollouts and steered revisions. Distillation updates the learner πθ\pi_\theta on the retained trajectories to obtain θk+1\theta_{k+1} for the next cycle.

Teacher-steered trajectory revision

For each failed student rollout, we seek a successful teacher reference for the same task and initial environment state. This reference remains fixed throughout revision. Successful student rollouts and failed rollouts without a successful reference are retained unchanged for distillation.

Localizing an intervention turn. We align the student and reference trajectories by their environment transitions, using only environment states to determine a match. Actions that produce matching transitions are treated as equivalent. We intervene at the first behavioral divergence, denoting the student turn by t∗t^* (Table 2, Q1). To revise this decision in its original context, we restore the history and environment state before t∗t^*.

Closed-loop teacher steering. The teacher uses the failed response and successful reference to identify task requirements or environmental conditions overlooked by the student (Figure 2, i). It provides a steering prompt Gt∗G_{t^*} that explains the discrepancy, the conditions under which the guidance applies, and a corrective rule (Table 2, Q3). Conditioned on Gt∗G_{t^*}, the frozen student generates a revised response yt∗∼πθk(⋅∣ht∗,Gt∗)y_{t^*} \sim\pi_{\theta_k}(\cdot\mid h_{t^*}, G_{t^*}) and executes its action.

We then locally verify the revised response by comparing the resulting environment states and returned information with those of the reference. If verification fails, the teacher refines its guidance using feedback from the attempt, and the student retries from the same pre-error history and state. The loop continues until a correction is locally verified or the per-turn attempt budget is exhausted. We compare teacher steering with unguided retries and teacher takeover in Table 2 (Q2).

Internalization of steered behavior

Retaining verified corrections. Once a revised response passes local verification, we retain it and its steering prompt for distillation. The response replaces the failed turn, and the student generates a new continuation without guidance. If the task still fails, we realign the revised rollout with the fixed reference to locate the next behavioral divergence. Revision stops when the task succeeds, no further divergence can be located, or an intervention exhausts its attempt budget. The current trajectory and its verified corrections are retained for distillation even if the task remains unsolved. Table 2 (Q4) evaluates both local verification and retaining corrections without requiring terminal success. Algorithm 1 and implementation details appear in Section A.

Internalization of steered behavior

We train the student on the retained trajectories using two sources of supervision: the steered student at corrected turns and the full-precision teacher at all other turns (Figure 2, ii). The learner receives no steering prompts in either case. Following prior work on turn-specific distillation [26], we use forward KL at corrected turns and reverse KL elsewhere (Table 2, Q5).

Learning from the steered student. At a corrected turn, we retrieve the steering prompt GG that elicited the accepted response. Let cc denote the interaction history and the prefix of that response, with all steering prompts removed. The frozen student snapshot πθk\pi_{\theta_k} predicts each token from (c,G)(c, G), while the learner πθ\pi_\theta predicts from cc alone. We minimize forward KL to match the steered student’s predictions:

ℓFKL(c,G;θ)=DKL(πθk(⋅∣c,G)∥πθ(⋅∣c)).(1)\ell_{\mathrm{FKL}}(c,G;\theta)=D_{\mathrm{KL}}\left(\pi_{\theta_k}(\cdot\mid c,G)\mathbin\Vert\pi_\theta(\cdot\mid c)\right). \tag*{(1)}

Retaining teacher supervision. At all other turns, including continuation turns that receive no further correction, we retain full-precision teacher supervision on the student-generated responses [21]. The teacher and learner receive the same context cc without guidance. We minimize reverse KL:

ℓRKL(c;θ)=DKL(πθ(⋅∣c)∥πT(⋅∣c)).(2)\ell_{\mathrm{RKL}}(c;\theta)=D_{\mathrm{KL}}\left(\pi_\theta(\cdot\mid c)\mathbin\Vert\pi_T(\cdot\mid c)\right). \tag*{(2)}

Training objective. We assign the loss by turn and average over all retained assistant-response tokens. For trajectory ii, let Ii\mathcal{I}_i denote its retained corrected turns, each with its steering prompt Gi,tG_{i,t}. For turn tt and token jj, write ci,t,j=(hi,t,yi,t,<j)c_{i,t,j}=(h_{i,t},y_{i,t,<j}). Let Ak\mathcal{A}_k contain all retained assistant-token positions (i,t,j)(i,t,j) in cycle kk. The objective is

L(k)(θ)=1∣Ak∣∑(i,t,j)∈Ak{ℓFKL(ci,t,j,Gi,t;θ),t∈Ii,ℓRKL(ci,t,j;θ),t∉Ii.(3)\mathcal{L}^{(k)}(\theta)=\frac{1}{\lvert\mathcal{A}_k\rvert}\sum_{(i,t,j)\in\mathcal{A}_k} \begin{cases} \ell_{\mathrm{FKL}}(c_{i,t,j},G_{i,t};\theta), & t\in\mathcal{I}_i,\\ \ell_{\mathrm{RKL}}(c_{i,t,j};\theta), & t\notin\mathcal{I}_i. \end{cases} \tag*{(3)}

Experiments

Experimental setup

Benchmarks and metrics. We evaluate on five agentic benchmarks: BFCL V4 [34], τ2\tau^2-Bench [35], WebShop [36], ALFWorld [37], and Search QA. We report the official Overall score for BFCL V4. For τ2\tau^2-Bench, we report success rate (Pass@1) in the main table and Pass@4 in Section C.2. We likewise report success rate (SR) for WebShop and ALFWorld and include WebShop Task Score in the appendix. For Search QA, we follow Search-R1 [38] and compute Exact Match (EM) scores on seven question-answering datasets.3 We report their unweighted mean in the main table and per-dataset EM in the appendix.

Models and training data. We evaluate SteerDistill and the baselines on three models: Qwen3-8B [46], Qwen3.5-4B [47], and Gemma 4 12B-it [48]. For each model, we obtain task-specific BF16 checkpoints through GRPO post-training [49].4 Each checkpoint serves as the full-precision reference and frozen teacher, and initializes all quantized methods. We use the same task-specific data or environments for BF16 post-training and subsequent low-bit training.

For BFCL V4 and τ2\tau^2-Bench, we train on executable tool-use tasks from AgentWorldModel [50] and evaluate the same tool-use checkpoint on both benchmarks. For WebShop and ALFWorld, we use their respective training environments. For Search QA, we follow Search-R1 [38] and use the training sets of NQ [39] and HotpotQA [42]. We evaluate on all seven datasets, treating NQ and HotpotQA as in-domain and the remaining five as out-of-domain.

Baselines. We compare with five baselines for quantization and capability recovery: (i) GPTQ [9] for post-training quantization (PTQ); (ii) EfficientQAT [15] for quantization-aware training (QAT); (iii) BitDistiller [17] for offline quantization-aware distillation (QAD); (iv) On-Policy QAD, implemented following NVIDIA NeMo-RL’s quantization-aware distillation workflow [51]; and (v) QeRL [19] for reinforcement learning with verifiable rewards.5 All training-based methods converge within the two-epoch training budget. Detailed training settings appear in Section B.

Quantization and evaluation protocol. We evaluate all three models with INT4 weights and BF16 activations (W4A16). For Qwen3-8B, we additionally evaluate INT4 weights with INT8 activations (W4A8) and INT2 weights with BF16 activations (W2A16). At inference, SteerDistill uses only the trained quantized student, without the teacher or steering prompts.

Main results

Gains across all five benchmarks. On Qwen3-8B under W4A16, SteerDistill outperforms all quantized baselines on each of the five benchmarks (Table 1). Its average score of 53.18 exceeds the strongest baseline, QeRL, by 2.18 points. With a remaining gap of only 0.36 points to the BF16 reference, SteerDistill recovers nearly all of the performance lost to quantization.

Near-BF16 performance across models. On Qwen3.5-4B and Gemma 4 12B-it under the same W4A16 setting, SteerDistill improves the average score over the strongest quantized baseline, QeRL, by 1.71 and 1.67 points, respectively. Across all three models, its average score remains within one point of the corresponding BF16 reference.

Gains persist at lower precisions. On Qwen3-8B, SteerDistill retains the highest average score under both W4A8 and W2A16. It outperforms the strongest quantized baseline in each setting: QeRL by 1.93 points under W4A8 and On-Policy QAD by 3.39 points under W2A16. Across these two settings, it also outperforms the quantized baselines on every benchmark except Search QA under W2A16, where BitDistiller scores 34.22 versus 34.05 for SteerDistill.

Efficiency

Higher success rates with less training. On Qwen3-8B, SteerDistill achieves higher Pass@1 on τ2\tau^{2}-Bench with fewer GPU hours than On-Policy QAD and QeRL (Figure 3a). Its intermediate checkpoint at 32 GPU hours reaches 31.68%, exceeding On-Policy QAD’s converged score at about 40 GPU hours by 2.91 percentage points. It also outperforms QeRL at 96 GPU hours by 1.84 percentage points while using 67% fewer GPU hours. This faster recovery is consistent with the targeted supervision provided by locally verified behavioral corrections.

Higher success rates with fewer output tokens. On Qwen3-8B under W4A16, SteerDistill achieves higher Pass@1 on τ2\tau^{2}-Bench with fewer output tokens per successful task than the quantized baselines (Figure 3b). Averaging over each method’s own successful tasks, we obtain 4.0K output tokens for SteerDistill, compared with 4.1K–5.2K for the quantized baselines and 3.9K for BF16. Its successful trajectories also involve fewer interaction steps and tool calls than those of the quantized baselines (Table 7). The W4A8 and W2A16 results show the same pattern (Section C.3).

MethodInput TokensCached InputOutput TokensInteraction StepsTool Calls
BF16185K148K3.9K11.44.2
GPTQ270K223K5.2K13.85.25
EfficientQAT243K199K4.7K13.14.92
BitDistiller235K192K4.5K12.94.78
On-Policy QAD214K173K4.2K12.34.55
QeRL206K166K4.1K12.14.43
Ours192K153K4.0K11.64.24

Table 7. Detailed trajectory and inference statistics for Qwen3-8B on τ\tau-Bench. Quantized methods use W4A16, with BF16 as the full-precision reference. All entries are averages over each method’s own successful tasks. Cached-input token counts are reported descriptively; K denotes thousands of tokens.

Ablation studies and analysis

ConfigurationOverallNon-LiveLiveMulti-TurnWeb SearchMemoryIrrel.
BF1646.8987.1481.7648.5318.5719.3278.63
On-Policy QAD44.4385.7280.1945.1416.8617.1674.88
Full SteerDistill (Ours)46.8488.2981.7148.4718.4418.7978.56
Q1: Intervention localization (default: Behavioral divergence)
Random eligible turn43.5285.2079.7244.0016.1016.3073.48
First action mismatch45.0186.2080.5545.9517.2017.4576.20
Q2: Intervention strategy (default: Teacher steering)
Unguided student retry44.4685.8180.0845.2616.9317.0574.97
Teacher takeover45.0686.3180.6946.0217.3117.5575.81
Q3: Steering format and source (default: Teacher / Structured)
Teacher / Free-form44.7887.1080.8545.3517.1017.0575.50
Student / Structured41.7483.7279.0241.4814.6614.9171.08
GPT-5.6-Sol / Structured45.2787.0580.9046.1517.5517.6575.90
Q4: Revision retention (default: Local verification)
No local verification42.1884.1078.9542.3014.9515.2071.55
Final success only46.3087.9181.0947.7218.1618.2778.02
Q5: Distillation routing (default: FKL at corrected turns; RKL elsewhere)
Reverse KL throughout45.2986.4580.9346.3317.4817.7876.02
Forward KL throughout44.5885.9880.4745.2217.2816.8975.32
Reversed routing (RKL / FKL)44.2885.6480.2844.7516.9616.8974.93

Table 2. Ablation study on BFCL V4 using Qwen3-8B under W4A16. BF16 and On-Policy QAD provide reference scores. Each Q1–Q5 group lists alternatives to its stated default; all defaults together form the full-method row. Bold marks the best quantized result in each column. The Final success only variant adds terminal-success filtering after local verification. Section D.3 shows a case study for Q3.

We examine the design choices for constructing and internalizing behavioral corrections on BFCL V4 using Qwen3-8B under W4A16 (Table 2). Q1–Q4 examine correction construction: where to intervene, how to generate corrections, and how to verify and retain them. Q5 then tests the distillation objectives used for internalization at corrected and other turns. All scores discussed below are BFCL V4 Overall scores.

Q1 (Intervention localization): Intervene at the behavioral divergence, not action mismatch. We use environment states to intervene at the first behavioral divergence rather than the first action mismatch. This improves the score by 1.83 points. Different actions can produce the same environment transition, so intervening on action mismatches can introduce unnecessary revisions into the distillation data. Intervening at a random eligible turn yields 43.52, even below On-Policy QAD’s 44.43.

Q2 (Intervention strategy): Steer the student instead of directly taking over. From the same restored history and environment state, we compare student retry without guidance, direct response replacement by the teacher, and student revision under teacher guidance. The student continues the rollout in all three cases. Steering improves the score by 2.38 points over retry and 1.78 points over takeover.

Q3 (Steering format and source): Use structured steering prompts generated by the teacher. We compare two formats for the teacher’s steering prompts: structured prompts that separately state the discrepancy, applicable conditions, and corrective rule, and free-form prompts without a prescribed structure. With the intervention turn, reference, and verification fixed, the structured format gains 2.06 points. This result is consistent with prior findings on the benefits of specific, actionable feedback and feedback organized around explicit requirements [52, 53]. Keeping the structured format and reference fixed, we obtain gains of 5.10 points over the student and 1.57 points over GPT-5.6-Sol [54] by using the full-precision teacher as the prompt source. Section D.3 compares the resulting prompts and student continuations in a ticket-creation task.

Q4 (Revision retention): Retain locally verified corrections even when the task fails. Removing local verification substantially lowers the score from 46.84 to 42.18. Once corrections are locally verified, additionally requiring terminal task success yields a slightly lower score of 46.30. This filter discards verified corrections from unsuccessful trajectories without improving performance.

Q5 (Distillation routing): Use forward KL at corrected turns and reverse KL elsewhere. We keep the retained trajectories, target models, and input contexts fixed and vary only the KL direction. Using forward KL (FKL) at corrected turns and reverse KL (RKL) elsewhere gains 1.55 points over RKL throughout and 2.26 points over FKL throughout. Reversing this assignment lowers the score by 2.56 points.

Conclusion

SteerDistill recovers the agentic capabilities of quantized LLMs through closed-loop teacher steering and distillation. By targeting the task requirements and environmental conditions overlooked at behavioral divergences, the teacher guides the student to generate its own corrections and refines its guidance using execution feedback. Distillation internalizes the locally verified behavioral corrections for unassisted inference. Across five benchmarks and three models, SteerDistill improves over quantized baselines under W4A16 and stays within one point of BF16 in average score. On τ2\tau^{2}-Bench, it outperforms QeRL by 1.84 percentage points in success rate using 67% fewer training GPU hours and produces shorter successful trajectories than the quantized baselines. We discuss limitations in Section E.

Algorithm 1 Teacher-steered trajectory revision for one rollout.

Require: student rollout τ\tau, frozen student πθk\pi_{\theta_k}, teacher πT\pi_T, successful reference τT\tau^T if available, attempt budget R≥1R \ge1

Ensure: retained trajectory τ\tau, corrected turns I\mathcal{I}, and prompt map G\mathcal{G}

  1. I←∅\mathcal{I} \leftarrow\varnothing; G←∅\mathcal{G} \leftarrow\varnothing; M←∅\mathcal{M} \leftarrow\varnothing

  2. if τ\tau succeeds or no successful reference is available then

  3. return (τ,I,G)(\tau, \mathcal{I}, \mathcal{G})

  4. end if

  5. while τ\tau fails do

  6. Align τ\tau with fixed τT\tau^T to find the first divergent turn t⋆t^\star and reference transition ρT\rho^T

  7. if no such pair is available then return (τ,I,G)(\tau, \mathcal{I}, \mathcal{G})

  8. F←∅\mathcal{F} \leftarrow\varnothing

  9. for r=1,…,Rr = 1, \ldots, R do

  10. Teacher πT\pi_T forms G\mathcal{G} from the aligned discrepancy, memory M\mathcal{M}, and feedback F\mathcal{F}

  11. Restore history ht⋆h_{t^\star} and the environment state before turn t⋆t^\star

  12. Sample y∼πθk(⋅∣ht⋆,G)y \sim\pi_{\theta_k}(\cdot\mid h_{t^\star}, \mathcal{G}); execute its action to obtain transition ρ\rho

  13. (v,f)←LocalVerify⁡(ρ,ρT)(v, f) \leftarrow\operatorname{LocalVerify}(\rho, \rho^T)

  14. if vv is true then break

  15. Append the prompt G\mathcal{G}, candidate response evidence, and report ff to F\mathcal{F}

  16. end for

  17. if vv is false then return (τ,I,G)(\tau, \mathcal{I}, \mathcal{G})

  18. Replace turn t⋆t^\star and its suffix with yy and a continuation without guidance from πθk\pi_{\theta_k}

  19. Discard correction records at or after t⋆t^\star from (I,G)(\mathcal{I}, \mathcal{G})

  20. I←I∪{t⋆}\mathcal{I} \leftarrow\mathcal{I} \cup\{t^\star\}; G[t⋆]←G\mathcal{G}[t^\star] \leftarrow\mathcal{G}; M←G\mathcal{M} \leftarrow\mathcal{G}

  21. end while

  22. return (τ,I,G)(\tau, \mathcal{I}, \mathcal{G})

Implementation details of SteerDistill

We provide implementation details for trajectory revision and distillation, following the two stages in § 3. Table 3 summarizes the notation.

SymbolMeaning
Policies and conditioning
πT\pi_TFrozen full-precision teacher.
θ\theta, πθ\pi_\thetaStudent master weights and trainable quantized policy.
kk, θk\theta_k, πθk\pi_{\theta_k}Training cycle, frozen student weights, and corresponding policy.
ii, tt, jjTrajectory, turn, and response-token indices.
τ\tau, τT\tau^TStudent trajectory and its fixed successful teacher reference.
hth_t, yty_tHistory before turn tt and the response generated at that turn.
ci,t,jc_{i,t,j}Token context (hi,t,yi,t,<j)(h_{i,t}, y_{i,t,<j}), with all steering prompts removed.
Revision state and retained supervision
t∗t^*Intervention turn selected by alignment at the current revision step.
ρ\rho, ρT\rho^TCandidate transition and its aligned reference transition.
RR, vv, ff, FFAttempt budget, acceptance flag, verification report, and accumulated feedback.
GG, Gi,tG_{i,t}Current steering prompt and the prompt retained for turn tt in trajectory ii.
M\mathcal{M}Memory holding the latest accepted steering prompt within the current trajectory.
G\mathcal{G}Stored prompt map: G[t]=Gi,t\mathcal{G}[t] = G_{i,t} for trajectory ii.
Ii\mathcal{I}_iRetained corrected turns in trajectory ii; these receive FKL supervision.
Training objective
Ak\mathcal{A}_kRetained assistant-token positions (i,t,j)(i,t,j) in cycle kk.
ℓFKL\ell_{\mathrm{FKL}}, ℓRKL\ell_{\mathrm{RKL}}Per-token losses at corrected and other turns (Equations (1) to (2)).
L(k)(θ)\mathcal{L}^{(k)}(\theta)Mean token loss over Ak\mathcal{A}_k, assigned by turn (Equation (3)).

Table 3. Key notation for SteerDistill. The revision algorithm handles one trajectory; training adds index ii.

Teacher-steered trajectory revision

Algorithm 1 specifies how a student rollout is revised and retained for distillation.

Environment states and alignment. We align student and reference trajectories using environment states to match their transitions. In AgentWorldModel, state is represented by task-relevant database contents. ALFWorld uses world facts and available actions. WebShop uses product selections, page type, and available choices. Search QA uses retrieved evidence as its information state.

Matching state transitions can result from different actions and occur at different turn indices. We select the first divergent student turn t⋆t^\star for intervention and use the corresponding reference transition ρT\rho^T to verify candidate revisions.

Local verification. Local verification determines whether a candidate response can replace the divergent turn. In AgentWorldModel, we require the database states before and after execution to match those of the reference and reject responses that cause environment errors. Read operations must also return matching information. For a terminal response, we require the reference to terminate as well and the candidate to pass the task verifier.

In ALFWorld, WebShop, and Search QA, we accept a valid candidate if its resulting environment state matches the reference or if both executions achieve task success. This allows successful executions to end in different states. Verification returns an acceptance flag vv and a report ff with the result and, on rejection, the failure category used to refine steering.

Steering prompts and feedback. The teacher forms a steering prompt GG from the aligned discrepancy and memory M\mathcal{M}. This memory stores the latest accepted steering prompt within the current trajectory, allowing the teacher to incorporate earlier guidance that remains applicable. The new prompt explains the discrepancy, the conditions under which its corrective rule applies, the expected transition, and how to verify it.

We allow up to RR attempts at each intervention. Before each attempt, we restore the same pre-error history and environment state. The frozen student then generates a response conditioned on GG and executes its action. If local verification rejects the attempt, we append its prompt, candidate assistant content and tool calls, termination status, and verification report ff to the feedback history F\mathcal{F}. The teacher uses summaries of F\mathcal{F}, together with the original failed rollout, fixed reference, and memory M\mathcal{M}, to refine GG for another attempt. Feedback history F\mathcal{F} is reset at each new intervention, while memory M\mathcal{M} carries accepted guidance across interventions.

Continuation and retained data. Once a response passes local verification, it replaces the response at t∗t^*, and the old suffix is discarded. We remove correction records at or after t∗t^* from I\mathcal{I} and G\mathcal{G}, then record the corrected turn in I\mathcal{I} and its steering prompt in G\mathcal{G}. This prompt also becomes the new memory M\mathcal{M}. From the resulting environment state and observation, the student generates a new continuation without guidance. If the task still fails, we realign the revised trajectory with the fixed reference to locate the next behavioral divergence.

Revision stops when the task succeeds, all RR attempts at an intervention fail local verification, or no further divergence can be aligned with the reference. In all cases, we retain the current trajectory together with I\mathcal{I} and G\mathcal{G} for distillation, including accepted corrections in trajectories that remain unsuccessful. If no correction was accepted, we retain the original student rollout.

Internalization of steered behavior

The retained trajectories provide the histories and response tokens used for distillation. At turns in I\mathcal{I}, we use the corresponding prompts in G\mathcal{G} to compute targets from the steered student. At all other turns, targets come from the full-precision teacher. The learner receives no steering prompts in either case.

Quantized generation and optimization. The teacher and student start from the same BF16 checkpoint. The teacher stays frozen, while the student maintains trainable BF16 master weights. In cycle kk, we use a frozen snapshot θk\theta_k in the low-bit runtime for rollout generation and revision.

During distillation, we recompute the FKL target πθk(⋅∣c,G)\pi_{\theta_k}(\cdot\mid c,G) at corrected turns using a frozen fake-quantized copy of the same snapshot and the stored steering prompt GG. At all other turns, the full-precision teacher supplies the RKL target πT(⋅∣c)\pi_T(\cdot\mid c). For both targets, cc is the same retained history and response prefix used by the learner.

The learner πθ(⋅∣c)\pi_\theta(\cdot\mid c) uses fake quantization in its forward pass, with gradients propagated to its BF16 master weights through the straight-through estimator. The updated master weights form snapshot θk+1\theta_{k+1}, which is quantized for rollout generation and revision in the next cycle.

Loss computation and masking. For each retained assistant turn, we compute loss only on its response tokens. These include assistant text and serialized tool calls, as well as reasoning and termination tokens when present. System and user messages, environment observations, and steering prompts are excluded from the loss. Discarded responses and suffixes are excluded from training.

We reuse the token IDs recorded during generation when available. Otherwise, we tokenize the retained response and append an EOS token unless generation was truncated by length.

At each supervised response-token position, we compute KL over the full vocabulary, using FKL for corrected turns and RKL for all other turns. This assignment also applies to regenerated continuations. The objective in Equation (3) averages over all supervised response tokens, so corrected and other turns are weighted by their respective token counts.

Evaluation details

Training settings. Table 4 lists the hyperparameters used for the low-bit training stage. All training-based methods use the same learning rate, trajectory batch size, and total number of epochs. EfficientQAT divides its two epochs evenly between Block-AP and E2E-QP. All training-based methods converge within this two-epoch budget.

MethodEpochsLearning RateBatch Size (trajectories)
EfficientQAT1+11 + 18×10−68 \times10^{-6}16
BitDistiller28×10−68 \times10^{-6}16
On-Policy QAD28×10−68 \times10^{-6}16
QeRL28×10−68 \times10^{-6}16
Ours28×10−68 \times10^{-6}16

Table 4. Hyperparameters for the low-bit training stage. Batch size counts complete trajectories. For EfficientQAT, 1+11 + 1 denotes one epoch of Block-AP followed by one epoch of E2E-QP.

The methods retain their respective optimization settings. BitDistiller uses AdamW with a constant learning rate and zero weight decay. On-Policy QAD follows NVIDIA NeMo-RL’s quantization-aware distillation workflow, using a rollout temperature of 1.0 and top-64 teacher logits. QeRL uses GRPO with temperature 1.0 and clipping bounds (0.2, 0.28), together with AdamW-8bit and its official adaptive quantization-noise configuration. For SteerDistill, we set the attempt budget to R=4R = 4 at each intervention in teacher-steered trajectory revision (Section A).

ModelPrecisionMethodBFCL V4 Overallτ2\tau^2-Bench Pass@1
Qwen3 8BBF16–40.4030.00
Qwen3 8BW4A16BitDistiller35.3424.29
Qwen3 8BW4A16On-Policy QAD36.1624.58
Qwen3 8BW4A16QeRL38.8528.92
Qwen3 8BW4A16Ours39.6729.05

Table 5. Results without initial GRPO post-training on Qwen3-8B. Quantized methods use W4A16. Scores are percentages; bold marks the best quantized result.

For SteerDistill, we follow the asymmetric group-wise quantization implementation in BitDistiller [17], including its calibration and clipping settings. We use a weight group size of 64 across all quantized methods. All experiments run on 8 NVIDIA A100 GPUs.

Inference settings. We follow the benchmark evaluation protocols and use the same inference settings for all quantized methods within each benchmark. We use temperatures of 0.001 for BFCL V4, 0 for τ2\tau^2-Bench, and 0.7 for WebShop, ALFWorld, and Search QA. Each response is limited to 32,768 output tokens, except in WebShop and ALFWorld, where the limit is 4,096. Interaction limits are 200 steps for τ2\tau^2-Bench, 15 for WebShop, 50 for ALFWorld, and 4 for Search QA; BFCL V4 follows the official evaluator’s limits. The τ2\tau^2-Bench user simulator uses GPT-5.6 at temperature 0. For Search QA, we retrieve three passages per query.

Additional results

Results without initial GRPO

We initialize SteerDistill and three quantized baselines directly from the original Qwen3-8B checkpoint, without initial task-specific GRPO post-training. This checkpoint also serves as the BF16 reference and frozen teacher. Apart from the initial checkpoint, all training settings match the main Qwen3-8B W4A16 experiments (§ 4.1 and Table 4).

Table 5 shows that SteerDistill outperforms the three quantized baselines on both benchmarks and remains within one point of BF16. All methods score lower than their post-GRPO counterparts in Table 1, while their relative ordering remains unchanged.

Additional evaluation metrics

Table 6 supplements the main results for Qwen3-8B under W4A16. For τ2\tau^2-Bench, we supplement Pass@1 with Pass@4, the fraction of tasks solved at least once in four runs. WebShop Task Score complements the success rate by measuring how well the selected product satisfies the user’s requirements. For Search QA, per-dataset EM provides a breakdown of the seven-dataset average reported in the main table. SteerDistill outperforms the quantized baselines on all reported metrics. On Search QA, this advantage holds for both in-domain datasets (NQ and HotpotQA) and all five out-of-domain datasets.

Methodτ2\tau^{2}-Bench Pass@4WebShop ScoreNQ†^{\dagger}TriviaQA⋆^{\star}PopQA⋆^{\star}HotpotQA†^{\dagger}2Wiki⋆^{\star}MuSiQue⋆^{\star}Bamboogle⋆^{\star}
BF1654.2078.6943.3662.4241.9636.5835.1215.9242.18
GPTQ45.5070.8637.6256.0436.0530.4428.3912.2334.71
EfficientQAT47.1072.7339.0857.6937.4832.0230.1712.8836.25
BitDistiller47.5573.4139.7158.3138.1132.7130.8613.2136.92
On-Policy QAD48.2075.6141.1860.2239.6934.4732.8614.4639.44
QeRL50.1076.0841.6460.7340.1335.0333.3614.7939.92
Ours53.0578.5742.8461.8841.4136.0134.5515.5341.61

Table 6. Additional evaluation metrics for Qwen3-8B. Quantized methods use W4A16, with BF16 as the full-precision reference. On τ2\tau^{2}-Bench, Pass@4 is the fraction of tasks solved at least once in four runs. We report Task Score for WebShop and per-dataset Exact Match (EM) for Search QA. †\dagger and ⋆\star denote in-domain and out-of-domain Search QA datasets, respectively. Higher is better for all metrics; bold marks the best quantized result in each column.

Inference efficiency

For Qwen3-8B on τ2\tau^{2}-Bench, we supplement the main analysis with W2A16 and W4A8 results and detailed trajectory statistics under W4A16. Token counts, interaction steps, and tool calls are averaged over each method’s own successful tasks, so the averages cover different task subsets.

SteerDistill achieves higher Pass@1 with fewer output tokens per successful task than the quantized baselines under W2A16 and W4A8 (Figure 4), consistent with the W4A16 results in Figure 3b. The detailed W4A16 statistics in Table 7 show that SteerDistill’s successful trajectories also use fewer input tokens and involve fewer interaction steps and tool calls than those of the quantized baselines.

Figure 4. Additional performance–token efficiency results of Qwen3-8B on τ2\tau^{2}-Bench under (a) W2A16 and (b) W4A8. The x-axis reports the average output tokens per successful task, while the y-axis reports Pass@1 performance.

Additional case studies

Inference cases

We compare all seven methods on three task segments to illustrate how overlooked task requirements lead to behavioral divergence. In the train and watchlist tasks, some agents act before obtaining the missing route or company name, requesting tickets for an assumed route or removing a stock the user has not identified (Figures 5 and Figure 7). The file-writing task is fully specified, yet some agents leave files empty or write the requested contents to the wrong files (Figure 6).

Identifying the company before removing a stock

Task: Remove a company from a watchlist containing NVDA and ZETA. The user initially leaves the company unnamed, then clarifies “Zeta Corp”.

BF16<br>Asks, then removes ZETAremove_stock_from_watchlist(symbol="ZETA")<br>get_watchlist() → ["NVDA"]
SteerDistill<br>Asks, then removes ZETAremove_stock_from_watchlist(symbol="ZETA")<br>get_watchlist() → ["NVDA"]
On-Policy QAD<br>No removal callReply: "attempts to remove it from the watchlist failed due to its invalidity."<br>get_watchlist() → ["NVDA", "ZETA"]
QeRL<br>Removes both before clarificationremove_stock_from_watchlist(symbol="NVDA")<br>remove_stock_from_watchlist(symbol="ZETA")<br>get_watchlist() → []
BitDistiller<br>Asks, then removes ZETAremove_stock_from_watchlist(symbol="ZETA")<br>get_watchlist() → ["NVDA"]
EfficientQAT<br>Price lookup fails; removes after clarificationremove_stock_from_watchlist(symbol="ZETA")<br>get_watchlist() → ["NVDA"]
GPTQ<br>Removes NVDA before clarificationremove_stock_from_watchlist(symbol="NVDA")<br>After the user identifies Zeta Corp:<br>remove_stock_from_watchlist(symbol="ZETA")<br>get_watchlist() → []

Figure 7. Deleting only the stock identified by the user. BF16, SteerDistill, BitDistiller, and EfficientQAT remove ZETA after clarification and retain NVDA. QeRL and GPTQ remove NVDA before clarification and end with an empty watchlist. On-Policy QAD reports a failed removal but makes no removal call, so both stocks remain.

Clarifying the route before booking

Task: Book train tickets for March 8, 2023, with no fare-class preference. The user has not supplied the departure or destination city.

BF16<br>Asks for the routeReply: "Please provide the starting city and the city you wish to travel to."
SteerDistill<br>Asks for the routeReply: "To proceed with booking the train tickets, I need the departure city and destination city for your journey."
On-Policy QAD<br>Invents a routeTrains_1_FindTrains(_from="Boston, MA", to="New York, NY")<br>Then calls GetTrainTickets for the same invented route.
QeRL<br>Searches blank endpoints, then asksTrains_1_FindTrains(_from="", to="")<br>Reply: "Could you please clarify these details?"
BitDistiller<br>Requests tickets for an invented routeFirst searches with no route, then:<br>Trains_1_GetTrainTickets(_from="Boston, MA", to="New York, NY")
EfficientQAT<br>Claims booking without a routeTrains_1_FindTrains(date_of_journey="03/08/2023")<br>Reply: "Your train ticket(s) for March 8th, 2023 has been booked!"
GPTQ<br>Invents a route, then asksTrains_1_FindTrains(_from="San Francisco, CA", to="Los Angeles, CA")<br>Then asks the user to confirm the departure and destination.

Table 5 (PDF p. 18). Clarifying the route before acting. BF16 and SteerDistill ask for the missing departure and destination cities before calling tools. QeRL and GPTQ ask only after searching with missing or invented endpoints. On-Policy QAD and BitDistiller instead request tickets for an invented route, while EfficientQAT claims a booking without obtaining the route.

Completing all three requested writes

Task: In WebDevProjects, create styles.css with “Hello World!”, index.html with “Hi World!”, and script.js with “Halo World!”.

The write tool requires an existing file; rejected writes must be retried after file creation.

BF16Writes all three requested strings to their respective files.
SteerDistillWrites all three requested strings to their respective files.
On-Policy QADWrites styles.css correctly. Creates index.html and script.js, but leaves both unwritten.
QeRLWrites all three requested strings to their respective files.
BitDistillerWrites styles.css and script.js correctly. Creates index.html, but never writes to it.
EfficientQATCreates styles.css and index.html after rejected writes, but does not retry either write. Writes script.js correctly.
GPTQecho(content="Hi World!", file_name="styles.css")<br>echo(content="Hello World!", file_name="index.html")<br>Writes script.js correctly.

Table 6 (PDF p. 18). Writing the specified contents to each file. BF16, SteerDistill, and QeRL complete all three writes correctly. On-Policy QAD, BitDistiller, and EfficientQAT create the files but omit some writes. GPTQ writes to all three files but swaps the contents of styles.css and index.html.

Training cases

The training cases show how teacher steering helps the student correct a failed decision from the pre-error history and state. In the file-writing and directory-navigation cases, the guidance makes an unmet prerequisite explicit: create the file before writing or enter the workspace before reading (Figure 8 and Figure 11; see also Figure 1). In the tool-list and card-control cases, the guidance clarifies which tools and parameters to use. The student calls the advertised tool names without their list numbers or sets the flag that enforces the country restriction (Figure 9 and Figure 10).

Figure 8. Creating the file before writing the requested sentence. The original rollout creates the file after a failed write but then writes only a fragment. Under teacher steering, the student creates the file first, and local verification accepts the new empty file. It then continues without the steering prompt and writes the full second sentence containing “server error.”

Figure 9. Using tool names without their list numbers. The original rollout includes the list numbers in both tool names, so both calls are rejected. Under teacher steering, the student calls the advertised door-control tool; local verification confirms that all four doors are unlocked. The student then turns on the headlights without the steering prompt.

Figure 10. Enforcing the requested country restriction. Both calls specify US,CA, but the original call leaves international transactions enabled. Under teacher steering, the student sets international_allowed=0, which enforces the country restriction in this API. The resulting country controls match the reference.

In the file-writing and tool-list cases, local verification checks the result of the corrected action: the newly created empty file or the unlocked doors match the reference state (Figure 8 and Figure 9). The student then continues without the steering prompt, writing the requested sentence or turning on the headlights.

Creating the file and selecting the full sentence

Task: Create 1.txt and append the second full sentence containing “server error” from dev_summary.txt.

The first matching sentence is “No server error occurred during the initial phase.” The second is “However, a server error was detected in the final testing phase.”

Original rolloutRevised rollout
Creates after failure; writes a fragmentCreates first; writes the full sentence
Original turn echo(content="resolving the server error.", file_name="1.txt") → echo: cannot write to '1.txt': No such file Continuation touch(file_name="1.txt") → null echo(content="resolving the server error.", file_name="1.txt") → nullCorrected turn touch(file_name="1.txt") → null Continuation echo(content="However, a server error was detected in the final testing phase.", file_name="1.txt") → null

Teacher steering prompt

PreconditionEnsure the file '1.txt' exists and is properly initialized.
Decision ruleUse mcp_tool_touch to create the file if it doesn't exist, then use mcp_tool_echo to append the second 'server error' sentence.
Expected transitionThe file '1.txt' should contain the second sentence mentioning 'server error' appended successfully.
Recovery checkVerify the file '1.txt' exists and contains the correct sentence. If not, reattempt creation and appending.

↓

Local verification and continuation

Local verificationThe new, empty 1.txt matches the reference state.
ContinuationWith the steering prompt removed, the student writes the full second sentence.

Enforcing the country restriction

Task: Restrict active virtual card 5 to the US and Canada, retaining its USD 2,000 monthly limit.

In this API, international_allowed=0 enforces allowed_countries; setting it to 1 permits international transactions.

Original rolloutRevised rollout
Country list does not restrict usageCountry list is enforced
update_card_controls(create_card_controls(
allowed_countries="US,CA", card_id=5,allowed_countries="US,CA", card_id=5,
international_allowed=1)international_allowed=0)

Teacher steering prompt

PreconditionThe virtual card (ID: 5) has been created, its spending limit updated to 2000.00, and its status set to 'active'.
Decision ruleTo lock international transactions outside 'US' and 'CA', use 'mcp_server_create_card_controls' with 'international_allowed' set to 0 (to restrict) and 'allowed_countries' set to 'US,CA'.
Expected transitionA card_controls record is created for card_id 5 with international_allowed=0 and allowed_countries='US,CA'.
Recovery checkIf the tool returns an error or the status is not updated, verify the card_id and ensure 'international_allowed' is 0 to correctly enforce the country whitelist.

Saved post-action state

Original rolloutinternational_allowed = true; allowed_countries = US,CA
Revised rolloutinternational_allowed = false; allowed_countries = US,CA
Reference comparisonThe saved configuration matches the reference on task-relevant fields.

Finding the directory before reading the file

Task: Show the contents of project_analysis.txt in the workspace folder so the user can review it. The agent starts in /alex, and cat reads files only from the current directory.

Original rolloutRevised rollout
Reads from the current directoryChecks the directory, then reads
cat(file_name="project_analysis.txt") → cat: 'project_analysis.txt': No such file or directorypwd() → /alex<br>cd(folder="workspace") → workspace<br>cat(file_name="project_analysis.txt") → Initial analysis content.

Teacher steering prompt

PreconditionFile 'project_analysis.txt' not found in current directory('/alex').
Decision ruleNavigate to the 'workspace' directory (identified via 'ls' and 'pwd') before attempting to access the file.
Expected transitionCurrent working directory changes to '/alex/workspace', making 'project_analysis.txt' accessible.
Recovery checkCall 'mcp_tool_cat' with 'project_analysis.txt' after confirming the directory transition.

Execution check

Directory/alex → /alex/workspace
File contentsThe returned text matches project_analysis.txt; the files are unchanged.

Steering prompt comparison

The ticket-creation case complements Q3 by showing how prompt format and source affect the guidance the student receives (Table 2 and Figure 12). Although the trading session is already authenticated, creating a ticket requires a separate login. Structured prompts from both the teacher and the student identify this prerequisite, and both continuations create the ticket after login. The free-form teacher prompt instead claims that the ticket already exists; the student logs in but stops without creating one.

The GPT-generated structured prompt also gives the correct login order and instructs the student to keep credentials out of the ticket description. The student logs in before creating the ticket but still includes the credentials in its description. This partial adherence illustrates why closed-loop teacher steering uses execution feedback to assess whether the intended behavioral correction has occurred.

Steering prompts and the login prerequisite

Task: Create a priority-3 support ticket titled “Urgent: Transaction Issue” using the supplied login credentials. The trading session is authenticated; the ticketing session is not.

MethodSteering prompt (excerpt)Student continuation and result
Teacher StructuredIf the user is not authenticated, call mcp_tool_ticket_login with provided credentials before creating the ticket.Logs in, then creates the ticket. One ticket; description contains the issue only.
Teacher Free-formA priority-level 3 support ticket titled "Urgent: Transaction Issue" has been successfully created in your name! Here are the key details:Ticket creation fails without a login. Logs in, then claims completion without creating a ticket. Saved ticket queue: empty.
Student StructuredAlways use the ticket login tool first to authenticate the user prior to creating a support ticket.Logs in, then creates the ticket. One ticket; description contains the issue only.
GPT StructuredAuthenticate with the ticketing system using the supplied username and password before attempting ticket creation; do not include the password or other login credentials in the ticket description.Logs in, then creates the ticket. One ticket; description includes login credentials.
Unguided Before steeringNo steering prompt.Ticket creation fails without a login. Logs in and retries successfully. One ticket; description includes login credentials.

Table 2 (PDF p. 25).

Figure 12. Guidance content and student execution in ticket creation. The four guided continuations start from the same history and environment state. Structured teacher and student prompts lead to tickets without credentials in their descriptions, while the free-form teacher prompt leads to no ticket. Both the GPT-guided continuation and the original unguided rollout create a ticket containing credentials; the latter first recovers from a login error.

Recovery cases

These cases illustrate an emergent behavior beyond the corrections explicitly targeted during training. Our trajectory revision constructs corrections from the restored pre-error history and state; at inference, the distilled student also recovers from execution errors within its ongoing rollout, without teacher guidance. In the file-writing trajectory from Figure 6, it creates the missing file after a rejected write and retries (Figure 13). In the messaging task, it successively resolves an unsupported argument and a missing login session before sending the message (Figure 14).

Figure 13. Recovering from a missing-file error. SteerDistill creates styles.css and retries the original write. A later read returns the requested contents.

Figure 14. Resolving successive errors before sending a message. SteerDistill removes the unsupported sender_id argument and retries. When the revised call reports that no user is logged in, the agent logs in as the sender and sends the message successfully.

Limitations

Teacher-steered trajectory revision requires successful teacher references, restoration of the pre-error history and environment state, and reliable local verification. Without a successful reference, a failed rollout is retained for distillation but receives no teacher-steered corrections. Applicability to environments without state restoration or reliable local verification remains untested. Moreover, local verification checks a revised response’s effect in the current context; it does not guarantee eventual task success or establish the correctness of every reasoning step in the response.

References

  1. [1]Shunyu Yao, Jeffrey Zhao, Dian Yu, Nan Du, Izhak Shafran, Karthik Narasimhan, and Yuan Cao. ReAct: Synergizing reasoning and acting in language models. In International Conference on Learning Representations, 2023. URL https://arxiv.org/abs/2210.03629v3.
  2. [3]Qingyun Wu, Gagan Bansal, Jieyu Zhang, Yiran Wu, Beibin Li, Erkang Zhu, Li Jiang, Xiaoyun Zhang, Shaokun Zhang, Ahmed Awadallah, et al. AutoGen: Enabling next-gen LLM applications via multi-agent conversation. In Conference on Language Modeling, 2024. URL https://www.microsoft.com/en-us/research/publication/autogen-enabling-next-gen-llm-applications-via-multi-agent-conversation-framework/.microsoft.com/en-us/research/publication/autogen-enabling-next-gen-llm-applications-via-multi-agent-conversation-framework
  3. [4]Yuxuan Yan, Shiqi Jiang, Ting Cao, Yifan Yang, Qianqian Yang, Yuanchao Shu, Yuqing Yang, and Lili Qiu. AVA: Towards agentic video analytics with vision language models. In 23rd USENIX Symposium on Networked Systems Design and Implementation (NSDI 26), pages 1939–1957, Renton, WA, may 2026. USENIX Association. ISBN 978-1-939133-54-0. URL https://www.usenix.org/conference/nsdi26/presentation/yan.usenix.org/conference/nsdi26/presentation/yan
  4. [5]Guohong Liu, Jialei Ye, Pengzhi Gao, Wei Liu, Jian Luan, Yunxin Liu, and Yuanchun Li. ScaleWoB: Guiding GUI agents with coding agents via large-scale environmental synthesis. arXiv:2605.25160, 2026.arxiv.org/abs/2605.25160
  5. [6]Lei Wang, Chen Ma, Xueyang Feng, Zeyu Zhang, Hao Yang, Jingsen Zhang, Zhiyuan Chen, Jiakai Tang, Xu Chen, Yankai Lin, et al. A survey on large language model based autonomous agents. Frontiers of Computer Science, 18(6): 186345, 2024. doi: 10.1007/s11704-024-40231-1.DOI
  6. [7]Zheng, L., Yin, L., Xie, Z., Sun, C., Huang, J., Yu, C. H., Cao, S., Kozyrakis, C., Stoica, I., Gonzalez, J. E., & al., . et . (2024). SGLang: Efficient execution of structured language model programs. Advances in Neural Information Processing Systems, 37, 62557–62583. https://doi.org/10.52202/079017-2000
  7. [8]Li, Y., Wen, H., Wang, W., Li, X., Yuan, Y., Liu, G., Liu, J., Xu, W., Wang, X., Sun, Y., & al., . et . (2024). Personal LLM agents: Insights and survey about the capability, efficiency and security. https://arxiv.org/abs/2401.05459
  8. [9]Frantar, E., Ashkboos, S., Hoefler, T., & Alistarh, D. (2022). GPTQ: Accurate post-training quantization for generative pre-trained transformers. https://arxiv.org/abs/2210.17323
  9. [10]Guangxuan Xiao, Ji Lin, Mickael Seznec, Hao Wu, Julien Demouth, and Song Han. SmoothQuant: Accurate and efficient post-training quantization for large language models. In Proceedings of the 40th International Conference on Machine Learning, volume 202 of Proceedings of Machine Learning Research, pages 38087–38099. PMLR, 2023. URL https://proceedings.mlr.press/v202/xiao23c.html.proceedings.mlr.press/v202/xiao23c.html
  10. [11]Ji Lin, Jiaming Tang, Haotian Tang, Shang Yang, Wei-Ming Chen, Wei-Chen Wang, Guangxuan Xiao, Xingyu Dang, Chuang Gan, and Song Han. AWQ: Activation-aware weight quantization for on-device LLM compression and acceleration. In Proceedings of Machine Learning and Systems, volume 6, pages 87–100, 2024. URL https://proceedings.mlsys.org/paper_files/paper/2024/hash/42a452cbafa9dd64e9ba4aa95cc1ef21-Abstract-Conference.html.DOI
  11. [12]Xiangyu Li, Chengyu Yin, Weijun Wang, Jianyu Wei, Ting Cao, and Yunxin Liu. Vec-LUT: Vector table lookup for parallel ultra-low-bit LLM inference on edge devices. In Proceedings of the 24th Annual International Conference on Mobile Systems, Applications and Services, pages 216–230. ACM, 2026. doi: 10.1145/3745756.3809200.DOI
  12. [13]Zhirui Huang, Rui Ma, Shijie Cao, Ran Shu, Ian Wang, Ting Cao, Chixiao Chen, and Yongqiang Xiong. TENET: An efficient sparsity-aware LUT-centric architecture for ternary LLM inference on edge. arXiv:2509.13765, 2025.arxiv.org/abs/2509.13765
  13. [14]Du, D., Cao, S., Cheng, J., Mai, L., Cao, T., & Yang, M. (2026). BitDecoding: Unlocking tensor cores for long-context LLMs with low-bit KV cache. IEEE International Symposium on High Performance Computer Architecture (HPCA), 1–13. https://doi.org/10.1109/hpca68181.2026.11408481
  14. [15]Mengzhao Chen, Wenqi Shao, Peng Xu, Jiahao Wang, Peng Gao, Kaipeng Zhang, and Ping Luo. EfficientQAT: Efficient quantization-aware training for large language models. arXiv:2407.11062, 2024.arxiv.org/abs/2407.11062
  15. [16]Janghwan Lee, Sihwa Lee, Jinseok Kim, Yongjik Kim, Jieun Lim, Jinwook Oh, and Jungwook Choi. ReQAT: Achieving full-precision reasoning accuracy with 4-bit floating-point quantization-aware training. arXiv:2606.15682, 2026.arxiv.org/abs/2606.15682
  16. [17]DaYou Du, Yijia Zhang, Shijie Cao, Jiaqi Guo, Ting Cao, Xiaowen Chu, and Ningyi Xu. BitDistiller: Unleashing the potential of sub-4-bit LLMs via self-distillation. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 102–116. Association for Computational Linguistics, 2024. doi: 10.18653/v1/2024.acl-long.7.DOI
  17. [18]Xin, M., Priyadarshi, S., Xin, J., Kartal, B., Vavre, A., Thekkumpate, A. K., Chen, Z., Mahabaleshwarkar, A. S., Shafah, I., Bercovich, A., & al., . et . (2026). Quantization-aware distillation for NVPFP4 inference accuracy recovery. https://arxiv.org/abs/2601.20088
  18. [19]Huang, W., Ge, Y., Yang, S., Xiao, Y., Mao, H., Lin, Y., Ye, H., Liu, S., Cheung, K. C., Yin, H., & al., . et . (2025). QeRL: Beyond efficiency – quantization-enhanced reinforcement learning for LLMs. https://arxiv.org/abs/2510.11696
  19. [20]NVIDIA. (2026). Quantization-aware RL (QARL). NeMo-RL Documentation. https://docs.nvidia.com/nemo/rl/latest/guides/quantization-aware-rl.htmldocs.nvidia.com/nemo/rl/latest/guides/quantization-aware-rl.html
  20. [21]Agarwal, R., Vieillard, N., Zhou, Y., Stanczyk, P., Ramos Garea, S., Geist, M., & Bachem, O. (2024). On-policy distillation of language models: Learning from self-generated mistakes. International Conference on Learning Representations. https://proceedings.iclr.cc/paper_files/paper/2024/hash/5be69a584901a26c521c2b51e40a4c20-Abstract-Conference.htmlproceedings.iclr.cc/paper_files/paper/2024/hash/5be69a584901a26c521c2b51e40a4c20-Abstract-Conference.html
  21. [22]Wenqi Shao, Mengzhao Chen, Zhaoyang Zhang, Peng Xu, Lirui Zhao, Zhiqian Li, Kaipeng Zhang, Peng Gao, Yu Qiao, and Ping Luo. OmniQuant: Omnidirectionally calibrated quantization for large language models. arXiv:2308.13137, 2023.arxiv.org/abs/2308.13137
  22. [23]Ashkboos, S., Mohtashami, A., Croci, M. L., Li, B., Cameron, P., Jaggi, M., Alistarh, D., Hoefler, T., & Hensman, J. (2024). QuaRot: Outlier-free 4-bit inference in rotated LLMs. 2404.00456
  23. [24]Zechun Liu, Changsheng Zhao, Igor Fedorov, Bilge Soran, Dhruv Choudhary, Raghuraman Krishnamoorthi, Vikas Chandra, Yuandong Tian, and Tijmen Blankevoort. SpinQuant: LLM quantization with learned rotations. arXiv:2405.16406, 2024.arXiv
  24. [25]Jiaqi Wang, Wenhao Zhang, Weijie Shi, Yaliang Li, and James Cheng. TCOD: Exploring temporal curriculum in on-policy distillation for multi-turn autonomous agents. arXiv:2604.24005, 2026.arXiv
  25. [26]Li, G., Zheng, M., Song, M., Liu, R., Yang, T., Sun, J., Zhong, Q., Guo, H., Fang, J., Zhang, D., & al., . et . (2026). On-policy distillation with curriculum turn-level guidance for multi-turn agents. 2606.15912
  26. [27]Yuanjie Lyu, Chengyu Wang, Jun Huang, and Tong Xu. Student-centered distillation narrows the agentic gap between small and large LLMs. In Proceedings of the 43rd International Conference on Machine Learning, volume 306 of Proceedings of Machine Learning Research, pages 84006–84029. PMLR, 2026. URL https://proceedings.mlr.press/v306/lyu26j.html.proceedings.mlr.press/v306/lyu26j.html
  27. [28]Aristotelis Lazaridis, Dylan Bates, Aman Sharma, Brian King, Vincent Lu, and Jack FitzGerald. EDGE-OPD: Internalizing privileged context with evidence guided on-policy distillation. arXiv:2605.23493, 2026.arXiv
  28. [29]Can Xie, Yuyi Zhou, Wen Yang, Ziyi Zhang, Siyao Song, Yingzhuo Deng, Shuo Ren, and Jiajun Zhang. EDGE: Experience-distillation for guided exploration in agentic reinforcement learning. arXiv:2608.21946, 2026.arXiv
  29. [30]Chen, C., Fan, Y., Sun, T., Yang, Y., Sun, C., Mao, D., Qiao, H., Zhang, Z., Wang, J., Sun, C., & al., . et . (2026). Look ahead before you distill: Future trajectory validation of teacher guidance for agentic on-policy distillation. 2608.01953
  30. [31]Xu Xia, Jinghua Piao, Min Yang, Xiaochong Lan, Jiaju Chen, and Yong Li. From scoring to acting: Outcome-verified comparative self-distillation for LLM agents. arXiv:2607.27937, 2026.arXiv
  31. [32]Tianzhu Ye, Li Dong, Xun Wu, Shaohan Huang, and Furu Wei. On-policy context distillation for language models. arXiv:2602.12275, 2026.arXiv
  32. [33]Chenhui Gou, Haoqin Tu, Yunhao Fang, Jianfei Cai, and Hamid Rezatofighi. Sample-efficient learning from agent experience. arXiv:2607.21051, 2026.arXiv
  33. [34]Patil, S. G., Mao, H., Yan, F., Ji, C. C.-J., Suresh, V., Stoica, I., & Gonzalez, J. E. (2025). The berkeley function calling leaderboard (BFCL): From tool use to agentic evaluation of large language models. Proceedings of the 42nd International Conference on Machine Learning, 267, 48371–48392. https://proceedings.mlr.press/v267/patiil25a.htmlproceedings.mlr.press/v267/patiil25a.html
  34. [35]Victor Barres, Honghua Dong, Soham Ray, Xujie Si, and Karthik Narasimhan. τ²-bench: Evaluating conversational agents in a dual-control environment, 2025. arXiv:2506.07982.arXiv
  35. [36]Shunyu Yao, Howard Chen, John Yang, and Karthik Narasimhan. WebShop: Towards scalable real-world web interaction with grounded language agents. In Advances in Neural Information Processing Systems, volume 35, pages 20744–20757, 2022. doi: 10.52202/068431-1508.DOI
  36. [37]Mohit Shridhar, Xingdi Yuan, Marc-Alexandre Côté, Yonatan Bisk, Adam Trischler, and Matthew Hausknecht. ALF-World: Aligning text and embodied environments for interactive learning. In International Conference on Learning Representations, 2021. URL https://arxiv.org/abs/2010.03768.
  37. [38]Bowen Jin, Hansi Zeng, Zhenrui Yue, Jinsung Yoon, Sercan Arik, Dong Wang, Hamed Zamani, and Jiawei Han. Search-R1: Training LLMs to reason and leverage search engines with reinforcement learning. arXiv:2503.09516, 2025.arXiv
  38. [39]Kwiatkowski, T., Palomaki, J., Redfield, O., Collins, M., Parikh, A., Alberti, C., Epstein, D., Polosukhin, I., Devlin, J., Lee, K., & al., . et . (2019). Natural questions: A benchmark for question answering research. Transactions of the Association for Computational Linguistics, 7, 452–466. https://doi.org/10.1162/tacl_a_00276
  39. [40]Mandar Joshi, Eunsol Choi, Daniel S. Weld, and Luke Zettlemoyer. TriviaQA: A large scale distantly supervised challenge dataset for reading comprehension. In Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 1601–1611. Association for Computational Linguistics, 2017. doi: 10.18653/v1/P17-1147.DOI
  40. [41]Alex Mallen, Akari Asai, Victor Zhong, Rajarshi Das, Daniel Khashabi, and Hannaneh Hajishirzi. When not to trust language models: Investigating effectiveness of parametric and non-parametric memories. In Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 9802–9822. Association for Computational Linguistics, 2023. doi: 10.18653/v1/2023.acl-long.546.DOI
  41. [42]Zhilin Yang, Peng Qi, Saizheng Zhang, Yoshua Bengio, William Cohen, Ruslan Salakhutdinov, and Christopher D. Manning. HotpotQA: A dataset for diverse, explainable multi-hop question answering. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pages 2369–2380. Association for Computational Linguistics, 2018. doi: 10.18653/v1/D18-1259.DOI
  42. [43]Xanh Ho, Anh-Khoa Duong Nguyen, Saku Sugawara, and Akiko Aizawa. Constructing a multi-hop QA dataset for comprehensive evaluation of reasoning steps. In Proceedings of the 28th International Conference on Computational Linguistics, pages 6609–6625. International Committee on Computational Linguistics, 2020. doi: 10.18653/v1/2020.coling-main.580.DOI
  43. [44]Harsh Trivedi, Niranjan Balasubramanian, Tushar Khot, and Ashish Sabharwal. ♫ MuSiQue: Multihop questions via single-hop question composition. Transactions of the Association for Computational Linguistics, 10:539–554, 2022. doi: 10.1162/tacl_a_00475.DOI
  44. [45]Ofir Press, Muru Zhang, Sewon Min, Ludwig Schmidt, Noah A. Smith, and Mike Lewis. Measuring and narrowing the compositionality gap in language models. In Findings of the Association for Computational Linguistics: EMNLP 2023, pages 5687–5711. Association for Computational Linguistics, 2023. doi: 10.18653/v1/2023.findings-emnlp.378.DOI
  45. [46]Yang, A., Li, A., Yang, B., Zhang, B., Hui, B., Zheng, B., Yu, B., Gao, C., Huang, C., Lv, C., & al., . et . (2025). Qwen3 technical report.arXiv
  46. [47]Team, Q. (2026). Qwen3.5: Towards native multimodal agents. Qwen Blog. https://qwen.ai/blog?id=qwen3.5qwen.ai/blog?id=qwen3.5
  47. [48]Team, G. (2026). Gemma 4 technical report.arXiv
  48. [49]Shao, Z., Wang, P., Zhu, Q., Xu, R., Song, J., Bi, X., Zhang, H., Zhang, M., Li, Y. K., Wu, Y., & al., . et . (2024). DeepSeekMath: Pushing the limits of mathematical reasoning in open language models.arXiv
  49. [50]Zhaoyang Wang, Canwen Xu, Boyi Liu, Yite Wang, Siwei Han, Zhewei Yao, Huaxiu Yao, and Yuxiong He. Agent world model: Infinity synthetic environments for agentic reinforcement learning. arXiv:2602.10090, 2026.arxiv.org/abs/2602.10090
  50. [51]NVIDIA. (2025). NeMo RL: A scalable and efficient post-training library. https://github.com/NVIDIA-NeMo/RLgithub.com/NVIDIA-NeMo/RL
  51. [52]Madaan, A., Tandon, N., Gupta, P., Hallinan, S., Gao, L., Wiegreffe, S., Alon, U., Dziri, N., Prabhumoye, S., Yang, Y., & al., . et . (2023). Self-Refine: Iterative refinement with self-feedback. Advances in Neural Information Processing Systems, 36, 46534–46594. https://doi.org/10.52202/075280-2019
  52. [53]Jonathan Cook, Tim Rocktäschel, Jakob Nicolaus Foerster, Dennis Aumiller, and Alex Wang. Check your work: Structured checklist feedback for improving large language models. In Maria Liakata, Viviane P. Moreira, Jiajun Zhang, and David Jurgens, editors, Proceedings of the 64th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 16649–16688, San Diego, California, United States, July 2026. Association for Computational Linguistics. doi: 10.18653/v1/2026.acl-long.759.DOI
  53. [54]OpenAI. (2026). GPT-5.6 system card. https://deploymentsafety.openai.com/gpt-5-6deploymentsafety.openai.com/gpt-5-6

Paper details

vlnc:W_55umdjq2
Version
v1 · latest
Published
Fresh — added to the Hub within the last 30 days
Added
License
CC BY 4.0
Usage

views · 8

downloads · 0

A view counts once per browser per day, a download each time the PDF is requested.

  • Computer Science

Not recorded for this paper

View statement
Looking for related papers…
Contents