betaFeedback

WorksComputer Science

Teaching Code Agents to Ask Without Leaking Private Code

Abstract

Local coding agents can extend their reach by consulting a stronger cloud model. Such consultations, however, often expose the repository's private identifiers, file paths, and source code. Host-side masking hides recognized identifiers but leaves the surrounding code and the agent's phrasing unchanged, while training for less disclosure risks suppressing consultation or producing questions too vague to answer. We propose utility-verified abstraction (UVA), which teaches the agent to describe private entities by their role and type while retaining the technical details an expert needs. From solved repair trajectories, we rewrite each question into this form and keep the rewrite only if it discloses fewer private entities and the repair, replayed from the same execution state with the repair trajectories. Across two benchmarks, the trained agent consults more often than before training. On repository-level issue repair, it solves 37.6% of issues, against 25.3% for the same model working alone, a gain from the training as a whole rather than from consultation alone. Its questions disclose 3.7-fold fewer private entities and 1.4-fold less verbatim code than the untrained agent's questions to the same expert.

Introduction

Large language models (LLMs) have demonstrated strong capabilities in code generation and program repair ((Jiang et al., 2026; Zhang et al., 2024b)). Coding agents, which integrate LLMs with development tools, can iteratively inspect files, modify code, and execute tests to produce a patch given an issue description and a repository ((Jimenez et al., 2024; Yang et al., 2024a)). Their growing capabilities have led to increasing adoption in real-world software development ((Robbes et al., 2026)). However, many coding agents rely on proprietary LLMs hosted by external providers, which raises privacy concerns when repository content is sent outside the organization. Running a smaller model locally can keep repository operations within the organization, but may limit the agent’s ability to resolve complex issues. To address this trade-off, local execution can be combined with remote consultation. A small student model runs on the machine that holds the repository and performs all editing and testing, while a larger expert model provides advice remotely without directly accessing the repository. Prior work has explored how to allocate computation across models through cascades and routing (Chen et al., 2024b; Ong et al., 2025), and SWE-Protégé shows that a local student can learn when to consult a stronger expert (Kon et al., 2026).

However, keeping repository operations local does not fully address these privacy concerns. The student may still include private identifiers, file paths, or source code in its consultations, making this information visible to the external provider. Existing approaches can limit such disclosure through prompt-based redaction and abstraction (Siyan et al., 2025; Dou et al., 2024), which instruct the model to remove or generalize private information before the question is sent to an external model. The student must nevertheless determine which details can be removed without obscuring the problem, and a privacy instruction alone provides no task-level feedback on whether the resulting question retains the information needed for useful advice. Such an instruction may also lower the consultation rate rather than improve the questions, in which case a reduction in total disclosure reflects silence rather than safer consultation. Alternatively, a host-side sanitizer can replace recognized private strings with placeholders before transmitting the question. This hides the identifiers that the sanitizer recognizes, but the surrounding source code is still transmitted. String replacement alone also does not check whether the sanitized question preserves the context the expert needs to give useful repair advice. The central challenge is therefore to reduce the private information disclosed in a consultation while preserving the technical content needed for useful advice. Removing too little leaves repository content exposed, while removing too much may omit the conditions that determine the correct repair. Training the student to make these choices requires examples of questions that satisfy both requirements. Whether a rewritten question remains useful, however, can be judged only after the expert has answered it and the student has completed the repair, so such examples cannot be labeled from the question text alone.

In this paper, we present utility-verified abstraction (UVA), a training procedure that teaches local coding agents to formulate useful questions with less private information. UVA constructs training examples from the consultations of successful repair trajectories, retaining an abstract rewrite of a question only if it discloses fewer private entities and the replayed repair still succeeds. An offline teacher rewrites each original question given its interaction history, describing private entities by their roles and types while preserving the symptom, the relevant technical constraints, and the decision on which advice is sought (Figure 1). Each rewrite then undergoes two checks. First, we count the private entities in the original and rewritten questions, namely the identifiers, file and module names, and paths that the student has read locally and that are not standard-library or public names. Second, we restore the repository snapshot and interaction history at the original consultation, obtain the expert’s answer to the rewritten question, and resume the student’s repair with that answer under the same remaining budget. A rewrite enters training only if it discloses strictly fewer private entities and the replayed repair passes the task’s tests. We then use LoRA-based supervised fine-tuning to train the student to generate each accepted rewrite from its original interaction history, mixing these examples with full repair trajectories to help preserve its repair capability. At deployment, the student decides when to consult and generates its own questions, without the offline teacher or any host-side rewriting.

Figure 1. One consultation, asked two ways. A local coding agent works through a repair and, at one step, consults a third-party cloud expert. (a) The untrained agent quotes three private identifiers of the repository in its question. (b) Our trained agent describes the same entities by role and type. The expert gives both the same advice and both repairs succeed, so the abstract question gets the same help while the identifiers stay on the device.

We conduct experiments on SWE-bench Verified and Terminal-Bench using Qwen3.5-9B (Qwen Team, 2026) as the student. With DeepSeek-V3.2 (DeepSeek-AI et al., 2025) as the expert for both UVA and the untrained consulting baseline, UVA improves pass@1 on SWE-bench Verified from 24.6% to 37.6%, while reducing mean disclosure per consultation from 4.42 to 1.20 private entities and from 8.4 to 5.9 verbatim source-code 5-grams. These reductions are not obtained by consulting less, as the consultation rate rises from 4.8% to 7.4%. The gain in solve rate is attributable to the training as a whole rather than to consultation alone, as the trained student reaches 36.2% without an expert. Compared with the host sanitizer, UVA achieves a higher solve rate (37.6% versus 25.0%), although its private-entity disclosure remains higher (1.20 versus 0.97). With Claude Opus 4.8 (Anthropic, [5]) as the expert, the trained student achieves 40.6% pass@1 on SWE-bench Verified and 43.8% accuracy on Terminal-Bench. In ablations with Claude Opus 4.8 as the expert, removing the repair replay lowers pass@1 from 40.6% to 22.4%, below the same model working alone. We also observe privacy by silence on SWE-bench Verified, where a privacy prompt reduces the consultation rate without reducing mean disclosure per consultation. This finding indicates that evaluations of private consultation should report the consultation rate together with the disclosure per consultation. Our privacy claims concern literal disclosure and do not establish protection against information inferred from the transmitted content.

Our contributions are summarized as follows:

  • We propose UVA, a training procedure that builds supervision for consulting a cloud expert with less private information, keeping only role-and-type rewrites that pass a disclosure check and a repair replay.

  • We identify privacy by silence on SWE-bench Verified, showing that a lower consultation rate can accompany an apparent privacy improvement without lower mean disclosure per consultation.

  • We evaluate UVA on two benchmarks against no consultation, unrestricted consultation, a privacy prompt, and a deterministic host sanitizer, and release the source code.

Related Work

Small and large models are usually combined by choosing which model answers, either by cascading on a confidence signal ((Chen et al., 2024b; Aggarwal et al., 2024)) or by routing before generation ((Ong et al., 2025; Ding et al., 2024)). A second line checks a message before it is sent, with guard models that classify the message ((Inan et al., 2023; Rebedea et al., 2023)) or sanitizers that substitute its sensitive spans ((Chen et al., 2023; Pilán et al., 2022)). Both lines leave unchanged the policy that writes the message. SWE-Protégé trains an on-device repair agent, which alone acts on the repository, to decide when to escalate to an expert (Kon et al., 2026), but it does not control what the question discloses. PAPILLON optimises the prompt sent to the remote model (Siyan et al., 2025), Huang et al. (2026) train a rewriter on spans annotated as task-essential, Dou et al. (2024) rephrase private content into a less specific form, and Zeng et al. (2026) concurrently train an on-device model to decide when to escalate and how to phrase the request. Scrouting verifies its outbound handoff by checking whether the claims in it are true (Bhola et al., [10]). None of these methods tests a rewritten request by re-running the task. On the training side, tool calls are learned by imitating a teacher’s traces (Qin et al., 2024; Patil et al., 2024) or by rewarding executed calls (Qiao et al., 2024; Jin et al., 2025). Rejection-sampling fine-tuning instead keeps a sampled candidate only when it passes an acceptance criterion, a likelihood margin in Toolformer (Schick et al., 2023) and task success in TALM and STaR (Parisi et al., 2022; Zelikman et al., 2022); this filter has since been applied to unit-tested code and repository trajectories (Haluptzok et al., 2023; Pan et al., 2025) and, closest to our setting, to query rewriters (Ma et al., 2023) and to models that write the question they send ([4, 2]). These criteria measure utility alone and discard the rejected candidates or use them as negatives for preference learning (Rafailov et al., 2023), whereas ours keeps a rewrite only if the replayed repair succeeds and the rewrite discloses strictly fewer private entities than the original question (Appendix § 7).

Problem Setting

We consider a repair task consisting of an issue description, a repository, and an evaluation test suite. A small student model runs locally with policy πθ\pi_{\theta}, selecting each action from its interaction history hh. The student may also consult a remote expert EE, which receives only a question qq and a fixed system prompt, while all repository operations remain local. A complete run is a rollout τ\tau, with S(τ)=1S(\tau)=1 if its submitted patch passes the required tests and S(τ)=0S(\tau)=0 otherwise.

Threat model. The local host, the student, and the offline teacher that rewrites training questions are trusted. The teacher reads the original questions, so on a private repository it must run inside the trust boundary. The expert’s provider is honest but curious and sees every transmitted question. Inference from the transmitted content lies outside this scope.

Disclosure measures. We measure the literal disclosure of private repository content, which we call P1 to separate it from hard secrets such as credentials (P0). Both classes are determined automatically (Appendix § 9). Let the reference rhr_h be the file contents, tracebacks, and command outputs that the student has observed up to hh. An entity is an identifier, a module or file name, or a path. Let I(rh)I(r_h) be the set of private entities, those that appear in rhr_h and are not on a versioned allowlist of standard-library and public names. Let W5(rh)W_5(r_h) be the set of token 5-grams on the lines of rhr_h that look like source code, with comments removed. For a transmitted question qq, we define the leaked private-entity count LP1L_{\mathrm{P1}} and the outbound code overlap OCO as

LP1(q;rh)=∣{u∈I(rh):u occurs verbatim in q}∣,(1)L_{\mathrm{P1}}(q;r_h) = \left|\left\{u \in I(r_h) : u\ \text{occurs verbatim in } q\right\}\right|, \tag*{(1)}
OCO⁡(q;rh)=∣W5(rh)∩w5(q)∣,(2)\operatorname{OCO}(q;r_h) = \left|W_5(r_h) \cap w_5(q)\right|, \tag*{(2)}

where w5(q)w_5(q) is the set of token 5-grams over all lines of qq. Both are computed on the actual transmitted text, including any host-side transformation. Appendix § 9 specifies the reference construction and filtering rules. We omit rhr_h when clear from context.

Our objective is to improve repair success while reducing disclosure per consultation. For a fixed expert and execution budget, we report the solve rate together with the mean disclosure per transmitted question, the total disclosure over all transmitted questions divided by their number. This mean is undefined for a policy that never consults, so Table 1 marks it n/a for Solo. Because a policy can also lower its total disclosure per task by consulting less, we report the consultation rate, the share of rollouts that consult at least once, to separate lower disclosure per question from less frequent consultation. Appendix § 9 states these quantities formally.

PolicySWE-bench VerifiedTerminal-Bench
pass@1 ↑pass@3 ↑Ask%LP1L_{P1} ↓OCO ↓pass@1 ↑Ask%LP1L_{P1} ↓OCO ↓
Solo (no ask)25.331.40n/an/a20.20n/an/a
Natural24.632.44.84.428.424.77.94.184.0
Privacy prompt23.831.21.44.649.520.27.93.172.3
Host sanitizer25.028.64.60.976.022.55.60.713.6
Ours, by cloud expert
DeepSeek-V3.237.645.67.41.205.929.29.02.351.4
GPT-5.6 Sol38.348.88.22.146.236.07.91.972.0
Claude Opus 4.840.659.27.62.314.843.811.21.362.1

Table 1. Table 1. Solve rate and outbound disclosure of each consultation policy. The first four rows are the baselines, all run with the untrained student, and the Ours rows are the student trained by UVA. The baselines solve about as many tasks with an expert as without one. With the same expert, the trained student solves more tasks and consults more often. Each of its questions also discloses fewer private entities and less verbatim code. Its private-entity count stays above the host sanitizer’s.

Method

Utility-Verified Abstraction (UVA) trains the student to formulate consultations that disclose less private information while retaining the technical content needed for repair. As shown in Figure 2, the procedure collects consultation candidates, constructs abstract rewrites verified by disclosure checks and repair replay, and fine-tunes the student on the accepted examples.

Figure 2. Overview of UVA. (a) Consultations are collected from successful student rollouts. (b) An offline teacher describes private entities by their roles and types. A rewrite is retained only if it strictly reduces measured disclosure and the repair passes the task’s tests when replayed from the saved state. (c) The student is fine-tuned with LoRA on the verified pairs together with repair trajectories. At inference, it generates consultations directly without a rewriter. The example and disclosure counts are illustrative.

Collecting unconstrained consultations

We collect consultations from student rollouts on training tasks with executable tests, without imposing privacy constraints on the generated questions. The student consults the expert by issuing a command, in the same format as its local tool calls. To collect consultations from different stages of a repair, we prompt the student to consult at scheduled steps, starting at a fixed step and repeating at a fixed interval up to a per-rollout cap (Appendix § 10). At each scheduled step, we append an instruction to consult to a temporary copy of the interaction history and sample a question from the student on this copy. We send the question to the expert and add the expert’s response to the running trajectory, which never contains the appended instruction. We use this schedule only for data collection; during evaluation, the student decides when to consult.

We keep consultations only from successful rollouts, those with S(τ)=1S(\tau)=1, so that each one comes from a repair that succeeded. This filter does not establish that the consultation caused the success. We call the kept consultations candidates and denote their set by C\mathcal{C}. We keep one candidate per identical interaction history, cap the number of candidates per task, and keep each candidate and its replay within the data split of its originating task. Each candidate records the interaction history hh, the original question q−q^{-}, the expert response, the repository snapshot, the random seed, and the remaining execution budgets, which together allow replay from the saved consultation state.

Verifying rewrites by repair replay

For each candidate (h,q−)∈C(h,q^{-}) \in\mathcal{C}, an offline teacher RR generates an abstract rewrite q+=R(h,q−)q^{+}=R(h,q^{-}). The teacher is distinct from the expert EE and is used only during data construction. Its instruction is to replace private entities with descriptions of their roles and types while preserving the observed symptom, relevant technical constraints, and decision requiring advice. Simply deleting details can remove conditions that determine the correct repair, while replacing names with opaque placeholders such as IDENT_3 does not tell the expert what role each entity plays. For example, an abstract question can describe a user-scoped cache key that collides across tenants and ask whether the key should include a tenant-scoped component. This retains the technical condition and proposed decision without exposing repository-specific names. Appendix § 8.§ 2 provides the teacher prompt.

Before replay, a rewrite must pass a fixed privacy predicate AA, disclose strictly fewer private entities than the original question, and pass a validity check VV. The privacy predicate is

A(q)⟺LP0(q)=0∧L^P1(q)≤κP1,(3)A(q) \Longleftrightarrow L_{P0}(q)=0 \land\widehat{L}_{P1}(q)\leq\kappa_{P1}, \tag*{(3)}

where ∧\land denotes logical and and LP0(q)L_{P0}(q) counts hard secrets found by a pattern scan, such as credentials, internal URLs, and private absolute paths. The detector-only count L^P1(q)\widehat{L}_{P1}(q) counts every entity span that the entity recognizer flags in qq, whether or not the span appears in rhr_h, and κP1=2\kappa_{P1}=2. The predicate AA therefore depends on the question alone, while the strict reduction compares both questions against rhr_h. Accepted training rewrites contain no pattern-detected hard secrets and at most two flagged P1P1 spans. The validity check V(q+)V(q^{+}) requires a nonempty question of bounded length in plain prose, with no code fence, shell command, or opaque placeholder, whose closing sentence poses the question. The allowed abstraction structure is specified in Appendix § 8.§ 2.

For candidates that pass these checks, we restore the repository snapshot, interaction history, and remaining budget at the original consultation. The expert answers q+q^{+}, and the student resumes under the original decoding and retry rules. Let τ+\tau^{+} denote the resulting rollout. During collection and replay, we send each question to the expert unmodified, so the scored question is exactly the text that the student or teacher wrote. A candidate is accepted only when

G(h,q+,q−)⟺A(q+)∧LP1(q+)<LP1(q−)∧V(q+)∧S(τ+)=1.(4)\begin{aligned} G(h,q^{+},q^{-}) &\Longleftrightarrow A(q^{+}) \\ &\quad\land L_{P1}(q^{+}) < L_{P1}(q^{-}) \land V(q^{+}) \\ &\quad\land S(\tau^{+})=1. \tag*{(4)} \end{aligned}

Both questions are scored against the same reference rhr_h. The strict reduction applies to LP1L_{P1}, and OCO is recorded for analysis only. We call the last condition, S(τ+)=1S(\tau^{+})=1, the replay gate.

The procedure thus keeps a rewrite only if it discloses fewer private entities than the original question and the repair that continues from the expert’s answer to it succeeds. Replaying from the saved state keeps the history before the consultation identical, but a passing replay does not establish that the consultation was necessary or that the rewrite preserves the probability of success. Rejected candidates are recorded by rejection stage (Appendix § 8.§ 3).

Training strategy

The accepted pairs form the supervision set D={(h,q+):(h,q−)∈C,G(h,q+,q−)}\mathcal{D} = \{(h, q^{+}) : (h, q^{-}) \in\mathcal{C}, G(h, q^{+}, q^{-})\}. We use LoRA-based supervised fine-tuning to train the student to generate each accepted question q+q^{+} from its original interaction history hh, keeping the backbone parameters frozen. Because the same adapter is active throughout the agent loop, question supervision can also affect ordinary repair actions. We therefore combine the consultation examples with full repair trajectories, minimizing

L(θ)=Lquestion(θ;D)+λLsolve(θ;Dsolve).(5)\mathcal{L}(\theta) = \mathcal{L}_{\mathrm{question}}(\theta; \mathcal{D}) + \lambda\mathcal{L}_{\mathrm{solve}}(\theta; \mathcal{D}_{\mathrm{solve}}). \tag*{(5)}

Here, Lquestion\mathcal{L}_{\mathrm{question}} is the negative log-likelihood of the rewritten question conditioned on hh, and Lsolve\mathcal{L}_{\mathrm{solve}} is the negative log-likelihood of the student’s actions in the repair trajectories Dsolve\mathcal{D}_{\mathrm{solve}}. The trajectories in Dsolve\mathcal{D}_{\mathrm{solve}} are action steps sampled from the same successful collection rollouts, with the consultation turns excluded from the targets, so an original question q−q^{-} appears only as context. The repair trajectories preserve the student’s repair capability while the student learns to write abstract questions. Because their targets are the student’s own successful repair actions, some taken after an expert reply had entered the context, training on them can also raise the solve rate of rollouts that never consult (Appendix § 11.§ 2).

Experiments

We evaluate UVA on two benchmarks in terms of task success and outbound disclosure, and examine the effects of its training components and the amount of supervision.

Setup

Benchmarks. Our primary benchmark is SWE-bench Verified (Jimenez et al., 2024), which contains 500 human-validated GitHub issues with per-instance execution environments. We score patches with a local scorer that follows the official rule, under which a patch resolves an issue when all fail-to-pass tests pass and no pass-to-pass test fails. To evaluate beyond program repair, we additionally use all 89 tasks in Terminal-Bench 2.0 (Merrill et al., 2026), with success determined by its task-specific verifiers.

Baselines. We compare UVA with four baselines, all of which use the same untrained student. Solo runs the untrained student without consultation and shows how many tasks the student solves on its own. Natural lets the untrained student consult the expert and sends its questions unmodified. Privacy prompt adds the offline teacher’s abstraction instruction verbatim to the untrained student’s system prompt (Appendix § 8.§ 2), without updating model parameters. The instruction asks the student to describe private entities by their roles and types while preserving the technical conditions and repair decision. Host sanitizer applies a deterministic transform TT that replaces the identifiers, module names, paths, URLs, and secrets it recognizes with placeholders. The expert receives T(q)T(q), and we map the placeholders in its response back to the original strings locally. Ours uses the student trained by UVA. Natural, Privacy prompt, and Ours transmit their generated questions without host-side sanitization.

Metrics. We report pass@kk on SWE-bench Verified using the standard unbiased estimator from n≥kn \ge k samples per instance (Chen et al., 2021), so that pass@1 estimates the solve probability, and pass@1 on Terminal-Bench, its native single-attempt accuracy. Ask% is the percentage of rollouts with at least one transmitted consultation. LP1L_{P1} counts the distinct private entities in a transmitted question and OCO counts the distinct source-code 5-grams that the question copies verbatim, both averaged over all transmitted questions (Appendix § 9). Metric definitions, statistical procedures, and consultation accounting are provided in Appendices § 9, § 13, and § 11.

Implementation details. We build the consultation training pairs from synthetic repair tasks in SWE-smith (Yang et al., 2025). The student is Qwen3.5-9B (Qwen Team, 2026), adapted using rank-16 LoRA on its linear projections. All consulting policies use DeepSeek-V3.2 (DeepSeek-AI et al., 2025) as the expert unless otherwise specified; we additionally evaluate UVA with GPT-5.6 Sol (OpenAI, 2026) and Claude Opus 4.8 [5]. Each task allows at most three consultations. Model versions, prompts, consultation limits, execution budgets, privacy rules, sample counts, and random seeds are fixed before the final evaluation (Appendix § 10). All benchmark instances remain in the evaluation denominator, with unscorable instances counted as failures. The agent’s shell activates the per-instance test environment that the grader uses.

Results

Task solving. The untrained student solves about as many tasks with an expert as without one. On Verified, its pass@1 is 25.3 under Solo, 24.6 under Natural, 23.8 under Privacy prompt, and 25.0 under Host sanitizer, within 1.5 points of one another. On Terminal-Bench, no consulting policy differs significantly from Solo under the exact McNemar test on paired tasks, with p=0.34p = 0.34 for Natural, 1.00 for Privacy prompt, and 0.75 for Host sanitizer. With the same expert, the student trained by UVA solves more tasks. With DeepSeek-V3.2 as the expert, its pass@1 on Verified is 37.6, against 24.6 for Natural and 25.3 for Solo, and its pass@3 is 45.6, against 32.4 for Natural. On Terminal-Bench it solves 26 of the 89 tasks, against 22 for Natural and 18 for Solo. A 4-task difference cannot reach significance under this test (its smallest possible pp is 0.125), so Terminal-Bench alone does not separate UVA from Natural with this expert. With GPT-5.6 Sol as the expert, the trained student reaches 38.3 on Verified and 36.0 on Terminal-Bench, and with Claude Opus 4.8 it reaches 40.6 and 43.8. We did not run the untrained student with these experts, so their gain over Natural mixes the effect of the questions with that of a stronger expert.

Collaboration quality. A private consultation policy is effective only if it keeps consulting and sends less in each consultation, and each baseline gives up one of the two. The privacy prompt lowers the total disclosure per task by consulting less. On Verified it consults in 7 of 500 rollouts, against 72 of 1,500 for Natural, a gap significant under Fisher’s exact test at p=3×10−4p = 3 \times10^{-4}, while its 14 questions disclose slightly more on average, 4.64 private entities against 4.42 and 9.5 source-code 5-grams against 8.4. We call this privacy by silence, an apparent privacy gain that comes from asking less rather than from disclosing less in each question. Its Verified Ask% and disclosure, like those of Host sanitizer, come from one of three runs. On Terminal-Bench the privacy prompt consults as often as Natural, in 7.9% of tasks, and lowers LP1L_{P1} from 4.18 to 3.17, so privacy by silence appears only on Verified. The host sanitizer keeps consulting and replaces the names it recognizes, but it forwards the surrounding code, 6.0 source-code 5-grams per question on Verified against 8.4 for Natural. UVA keeps consulting and sends less in each consultation. With DeepSeek-V3.2 as the expert, the trained student consults more often than the untrained student, in 7.4% of Verified rollouts against 4.8% and in 9.0% of Terminal-Bench tasks against 7.9%. Each of its questions discloses 3.7-fold fewer private entities, 1.20 against 4.42, and 1.4-fold less verbatim code, 5.9 against 8.4, on Verified, and 1.8-fold fewer entities and 2.9-fold less code on Terminal-Bench. Without any host transform, its verbatim-code overlap matches or undercuts the host sanitizer’s on both benchmarks, at 5.9 against 6.0 and 1.4 against 3.6 (Appendix § 11.§ 1), while its private-entity count stays above the sanitizer’s, at 1.20 against 0.97 on Verified. With the other experts, the trained student’s disclosure per question also stays below that of Natural with DeepSeek-V3.2. Training changes how often the student consults but not when, since the trained student consults at a median step of 39 against 38 for Natural (Appendix § 11.§ 3).

Ablation on the training strategy. We remove or replace one component of UVA at a time, retrain each variant on the same candidate pool, initialization, and update budget as the 300-pair student of Table 1, and evaluate it on Verified, as in every panel of Figure 3 (panel (a)). The four variants drop the repair-trajectory mixture, drop the replay gate, replace supervised fine-tuning on q+q^{+} with DPO on the contrast between q+q^{+} and q−q^{-}, and train toward the deterministic sanitizer’s output instead of the abstraction, which we call Host-SFT. All trained students in panels (a) and (b) consult Claude Opus 4.8 [5], while the marks taken from Table 1 come from the untrained student with DeepSeek-V3.2 (DeepSeek-AI et al., 2025) or with no expert, so a comparison with those marks also changes the expert.

Figure 3. Ablation on training strategy and training data, and the overall costs. In (a) and (b), the trained students consult Claude Opus 4.8, while the marks taken from Table 1 and the zero-pair point are the untrained student with DeepSeek-V3.2. (a) Disclosure is LP1L_{P1}; the triangles mark Natural and Host sanitizer. (b) Nested budgets of 100–400 accepted pairs; disclosure is OCO. (c) Cloud consultation cost per 1,000 tasks against the solve rate, each point labelled with its price.

Every trained variant discloses between 1.60 and 2.67 private entities per question, below the 4.42 of Natural, and the variants differ mainly in how many tasks they solve. Without the repair-trajectory mixture, the student still discloses only 2.67 entities, but pass@1 falls from 40.6 to 32.2, so the question supervision alone lowers disclosure and the repair trajectories are needed to keep the solve rate. Without the replay gate, disclosure falls furthest, to 1.60 entities, and pass@1 falls to 22.4, below the 25.3 of Solo. This is consistent with the replay gate removing rewrites that omit details the repair depends on, which the student would otherwise learn to omit. DPO reaches the disclosure of UVA, 2.40 against 2.31, but solves only 25.0, so training on the contrast with q−q^{-} lowers disclosure without the solve gain of supervised fine-tuning on q+q^{+}. Host-SFT solves 18.4 and discloses 1.70, more than the 0.97 of the sanitizer it imitates, so imitating placeholders reaches neither the sanitizer’s disclosure nor the solve rate of abstraction. The full procedure gives the highest solve rate of the five at a disclosure within the range of the variants, so removing or replacing any component lowers the solve rate.

Ablation on the training data. To measure how much training data the abstraction needs, we retrain the student on the first 100, 200, 300, and 400 accepted pairs, each subset containing the previous one, under the recipe of Eq. (5) with the mixture ratio and epoch count held fixed (Figure 3(b)). The point at zero pairs, marked base, is Natural. Because the mixture ratio is fixed, ∣Dsolve∣|\mathcal{D}_{\mathrm{solve}}| grows with the pair budget, so the curve varies both terms of Eq. (5) together rather than isolating the consultation pairs. Table 1’s Ours rows use the 300-pair student (Appendix § 10). This panel reads disclosure as OCO for two reasons. Renaming symbols leaves the surrounding code in place, so OCO separates an abstract question from a pseudonymized one, and OCO needs no entity recognizer. Up to 300 pairs, pass@1 rises to 40.6, while OCO stays near the 8.4 of Natural at 100 pairs and then falls to 4.8, below the host sanitizer’s 6.0. This is consistent with the abstraction being learned from the pairs rather than from the instruction alone, since the privacy prompt gives the untrained student the same instruction and does not lower OCO (9.5 against 8.4 in Table 1). At 400 pairs, pass@1 is 39.4 and OCO is 5.5, so collecting more than 300 pairs brings no further gain under this recipe.

Cost analysis. Figure 3(c) plots, for each expert, the cost of cloud consultation per 1,000 tasks against the solve rate, with the student’s own inference counted separately. Consultation costs $0.012 per 1,000 tasks with DeepSeek-V3.2, $0.21 with GPT-5.6 Sol, and $0.48 with Claude Opus 4.8, under 0.2% of the about $268 that the student’s own inference costs. Because the student runs on local hardware, we price its inference at OpenRouter’s published per-token rate for Qwen3.5-9B. Appendix § 11.§ 4 gives the measured length and the pricing of a consultation. Training is a one-off cost that amortizes as the deployed student is used, so we do not account for it here.

Conclusion

This paper shows that a local code agent can learn to consult a cloud expert while exposing less of its repository. Trained only on rewrites that disclose less and still lead to a successful replayed repair, the student solves 9 to 24 points more tasks than the same model working alone on both benchmarks, consults at least as often as before training, and discloses fewer private entities and less verbatim code per question. Removing any training component lowers the solve rate.

Extended Related Work

Organised by mechanism rather than topic, because the distinction our method turns on is mechanical. Almost every established answer acts on the traffic from outside the policy, whereas ours acts on the policy that writes it.

Small–large model collaboration in coding agents

Allocation between a cheap and an expensive model is decided after the fact by cascades, which escalate on a verification or consistency signal (Chen et al., 2024b; Aggarwal et al., 2024; Yue et al., 2024) and whose optimality is unified with routing by Dekoninck et al. (2025), or in advance by learned routers, including on-device versus cloud (Ong et al., 2025; Ding et al., 2024) and RL-trained multi-round variants (Zhang et al., 2025a). The decision can sit inside the acting model, calibrated for coverage (Ren et al., 2023) or post-trained jointly with solving (Fang et al., 2026), or be made architectural, with the cloud writing a sketch that the device holding the private context specialises (Zhang et al., 2024a; Niu et al., 2025). In coding agents the same question is posed temporally (Son et al., 2026) and empirically (Gandhi et al., 2025). Throughout, the learned object is a destination or a trigger. The exception composes an outbound handoff and verifies it, but for the truthfulness of its reproduction claims rather than by re-running the task, and its frontier model writes the patch rather than advising [10]. SWE-Protégé fixes the configuration we inherit (Kon et al., 2026).

Guards and sanitizers at the trust boundary

Host-side redaction detects and substitutes sensitive spans, with reverse mapping (Chen et al., 2023), importance weighting (Shen et al., 2024), or inference-time search over a privacy-ordered space (Zhou et al., 2025), and is scored by recall of masked spans (Pilán et al., 2022). Our Host sanitizer belongs to this family, but we decline its metric, because scoring a sanitizer’s output with the detector that produced it drives the measured leak to near zero by construction. Learned guards classify what the policy already wrote (Inan et al., 2023; Rebedea et al., 2023); stronger guarantees restrict what the writer may see or send [7] (Hong et al. 2024), and contextual integrity supplies the predicate that licenses a role-and-type target, appropriate flow rather than absence of personal data (Ghalebikesabi et al., 2024). Agent benchmarks establish that stated norms do not predict behaviour under action (Shao et al., 2024; Mireshghallah et al., 2024; Zharmagambetov et al., 2025) and that the leak lives in intermediate calls (Hu et al., 2026); our provenance inventory descends from canaries and verbatim extraction [12] (Carlini et al. 2021; Carlini et al. 2023), and bounds literal disclosure only, not inference (Staab et al., 2024). Prompt injection is the adjacent threat model we disclaim (Debenedetti et al., 2024; Zhan et al., 2024; Debenedetti et al., 2025), though SecAlign is a structural precedent for moving a security property into the weights (Chen et al., 2025). Methods that change what the model writes are the axis we position against (Siyan et al., 2025; Huang et al., 2026; Hui et al., 2026; Kim et al., 2025; Staab et al., 2025; Dou et al., 2024; Zeng et al., 2026); one head-to-head study reports the best tradeoff for pseudonymization, judged conversationally rather than by re-execution (Xiao et al., 2026).

Tool use in LLM agents

Our action channel follows the interleaved reason-and-act loop (Yao et al., 2023), interface-dominated agent design (Yang et al., 2024a), and the single executable channel (Wang et al., 2024). Tool calls are learned by imitation on teacher corpora, graded against a reference (Qin et al., 2024; Patil et al., 2024; Li et al., 2023; by RL on executed calls, spent on the decision (Qiao et al., 2024; Qian et al., 2025a) or on what a query retrieves (Jin et al., 2025); or by filter-then-fine-tune, which divides on the filter signal. Toolformer keeps a call when its returned result lowers a distance-weighted loss by a margin, measured against the better of not calling and calling without the result, so a call whose mere presence helps is rejected (Schick et al., 2023); TALM and STaR keep a trace only on task outcome (Parisi et al., 2022; Zelikman et al., 2022), a rule carried to interpreter-filtered program synthesis (Haluptzok et al., 2023) and repository trajectories (Pan et al., 2025). Applying an outcome filter to the content of the call is nearest to us (Nogueira and Cho, 2017; Ma et al., 2023; Andukuri et al., [4]; Ambati et al., [2]). Learning when to call is a distinct axis we do not train (Asai et al., [6]; Qian et al., 2025b; Zhang et al., 2025b; Wang et al., 2025).

Selection-based post-training

Selection keeps the correct samples (Yuan et al., 2023) or a reward-ranked fraction (Dong et al., 2023) and can iterate (Zelikman et al., 2022; Gulcehre et al., 2023), though the loop saturates and then overfits (Singh et al., 2024), which is why we take a single pass. DPO’s reference-anchored objective (Rafailov et al., 2023) constrains the gap within a pair and not the level. The preferred likelihood can fall while the margin widens (Pal et al., 2024; Rafailov et al., 2024; Xiao et al., 2024), displaced mass lands outside the pair (Razin et al., 2025; Yan et al., 2025), and the effect is worst for near-duplicates, which our pairs are. Two caveats cut the other way. Ranking accuracy does track win rate near the reference (Chen et al., 2024a), and off-policy targets move mass slowly (Tajwar et al., 2024), an objection that applies to our own teacher-authored supervision. Training a verifier on the discarded negatives is the principled counterargument (Hosseini et al., 2024), and adding a likelihood term to the pairwise loss reaches a related conclusion (Pang et al., 2024); the former addresses correctness negatives, not ones differing by a handful of private tokens. Our teacher rewrite descends from critique-and-revise (Bai et al., [8]), contrastive construction (Yang et al., 2024b), and mistake rewriting (An et al., [3]), none of which checks that the revision still executes. Mixing a second objective against narrow-tuning regression is established practice (Ouyang et al., 2022; Scialom et al., 2022) and the risk it guards is documented (Qi et al., 2024; Betley et al., [9]). Safe RLHF keeps two criteria separate rather than collapsing them (Dai et al., 2024), and safety tuning past a point yields exaggerated safety (Bianchi et al., [11]), the published name for the failure our replay gate exists to prevent. We are not aware of prior work that admits a training candidate only when a task-success predicate and a disclosure predicate hold simultaneously.

Method Details

Algorithm

Algorithm 1 states the full pipeline of § 4 in one place. It collects consultation states from solved rollouts, rewrites each question with the offline teacher, checks the rewrite’s disclosure and form, replays it from the identical state, and fine-tunes the student on the survivors together with the repair trajectories.

Teacher prompt and the allowed abstraction structure

The teacher is an offline rewriting model, distinct from the deployed expert EE; because it sees the private q−q^{-}, pair construction runs where the data may live, here on corpora whose “private” content is synthetic (SWE-smith), and locally for a genuinely private repository. Its instruction is reproduced

Algorithm 1 Pair construction and training in utility-verified abstraction (UVA)

Require: training tasks X\mathcal{X} with executable tests; student πθ\pi_{\theta}; teacher RR; expert EE; privacy predicate AA (Eq. (3)); validity check VV; mixing weight λ\lambda

1: C←∅\mathcal{C} \leftarrow\varnothing; Dsolve←∅\mathcal{D}_{\mathrm{solve}} \leftarrow\varnothing ▷ collect (no host rewrite at any point) 2: for task x∈Xx \in\mathcal{X} do 3: roll out πθ\pi_{\theta} on xx as τ\tau, injecting consultations on a fixed schedule; at each injected consultation, record hh, q−q^{-}, the expert’s reply, the repository snapshot, the seed, and the remaining budgets 4: if S(τ)=1S(\tau) = 1 then 5: add the recorded consultation states of τ\tau to C\mathcal{C} 6: add the action steps of τ\tau, with consultation turns excluded as targets, to Dsolve\mathcal{D}_{\mathrm{solve}} 7: end if 8: end for 9: keep one candidate per identical history in C\mathcal{C}; cap candidates per task 10: D←∅\mathcal{D} \leftarrow\varnothing 11: for (h,q−)∈C(h, q^{-}) \in\mathcal{C} do ▷ rewrite, replay, verify 12: q+←R(h,q−)q^{+} \leftarrow R(h, q^{-}) ▷ offline teacher: roles and types replace private entities 13: if A(q+)∧LP1(q+)<LP1(q−)∧V(q+)A(q^{+}) \land L_{\mathrm{P1}}(q^{+}) < L_{\mathrm{P1}}(q^{-}) \land V(q^{+}) then 14: restore the snapshot at hh; send q+q^{+}; obtain EE’s reply; resume πθ\pi_{\theta} to termination as τ+\tau^{+} 15: if S(τ+)=1S(\tau^{+}) = 1 then 16: admit (h,q+)(h, q^{+}) into D\mathcal{D} ▷ all four conditions of GG hold 17: end if 18: end if 19: record the rejection stage of every non-admitted candidate 20: end for 21: train: fine-tune θ\theta on Lquestion(θ;D)+λLsolve(θ;Dsolve)\mathcal{L}_{\mathrm{question}}(\theta;\mathcal{D}) + \lambda\mathcal{L}_{\mathrm{solve}}(\theta;\mathcal{D}_{\mathrm{solve}}) (Eq. (5))

in full below (the Privacy prompt baseline of § 5.1 prompts the base student with this same instruction, verbatim).

Prompt B.1: Teacher rewriter (_REWRITER_SYSTEM), verbatim

You rewrite a software engineer’s help request so it can be sent to an outside expert
WITHOUT leaking any private code detail, while staying just as useful to answer.
RULES:
– Remove EVERY private identifier: file paths, module/package names, class names,
function/method names, variable/parameter names, and any verbatim code or literal
values.
– Do NOT replace them with opaque placeholders (no IDENT_1, FUNC_2, etc.).
Instead describe each by its ROLE and TYPE, e.g. “the constructor of a shared
auth-token object”, “a token-generator callable”, “a boolean formatting option”, “a
context-manager’s await hook”.
– KEEP all engineering substance: the observed symptom, what was expected, the
STRUCTURAL pattern of the bug (e.g. “two adjacent same-type constructor arguments are
passed in swapped order”), what the engineer already ruled out, and the exact decision
they need judged.
– The result MUST be answerable by an expert who cannot see the code: convey enough
structure to reason about, without naming anything private.
– Keep it concise (2–5 sentences). Output ONLY the rewritten question, no preamble.

One asymmetry between this instruction and the measurement of Appendix § 9 is deliberate. The instruction removes every identifier, including public library and standard-library names that the detector never charges (its allowlist). At rewrite time neither the teacher nor, later, the student can reliably decide provenance, whether a name is public API or a repository symbol shadowing one, so the target errs toward removal, and the utility cost of over-abstraction is policed by the replay gate rather than by trusting the rewriter’s judgment, since a rewrite that abstracts away a load-bearing name, public or private, fails its replay and is rejected. The measurement side is unaffected, because LP1L_{\mathrm{P1}} charges only provenance-verified private entities either way.

The validity check VV in Eq. (4) enforces the same structure mechanically; the residual private identifiers are bounded by AA (Eq. (3)), not by VV. It requires no opaque placeholder token (a pattern check for IDENT_n-style aliases), a nonempty question of bounded length in plain prose with no code fence or shell command, whose closing sentence is a question or a request.

Admission rules

The target question is the exact one used in the successful replay. A consultation carries no attachment, so the question is the whole outbound message and question-level acceptance is full-message acceptance. A pair is admitted only with a measurable disclosure contrast, in which q+q^{+} passes the fixed privacy predicate AA and discloses strictly fewer private entities than q−q^{-} under the sanitizer-independent measure of Appendix § 9; we do not additionally require q−q^{-} to fail AA. Identical pairs, pairs with no disclosure reduction, malformed asks, missing snapshots, incomplete logs, and replays with modified tests are rejected. Neither source success nor replay success licenses a claim that an ask was necessary. The training-data grid of § 5 uses the first 100 to 400 accepted pairs in collection order, and the student of Table 1 is trained on the first 300.

Privacy Measurement

Definitions and audit scope

The reference rhr_h is the raw machine-side view at prefix hh, the file contents, tracebacks, and command outputs the agent has observed, and it is never transmitted. Each transmitted question is scored against the reference of the prefix at which it was sent. Let I(r)I(r) be the allowlist-filtered private-entity set of reference rr, constructed below, let W5(r)W_5(r) be the set of token 5-grams on the lines of rr that contain code syntax (an equals sign, a bracket, a colon, or a keyword such as def or return), with comments removed, and let W5(q)W_5(q) be the token 5-grams over all the lines of the message qq. For every policy the counts are taken on the message it transmits, which for Host sanitizer is T(q)T(q). The P1 measure used throughout the paper is LP1(q;r)L_{\mathrm{P1}}(q;r) of Eq. (1), the number of distinct private entities disclosed, and the outbound code overlap is OCO⁡(q;r)\operatorname{OCO}(q;r) of Eq. (2), the number of leaked 5-grams. We report counts rather than ratios, because a ratio over a small reference misranks a short abstract question by exposure; these are unique-span counts, not transmitted-token counts.

Per-consultation mean and consultation rate. For a policy πθ\pi_\theta under a fixed expert and execution budget, let U(πθ)=Eτ∼πθ[S(τ)]U(\pi_\theta)=\mathbb{E}_{\tau\sim\pi_\theta}[S(\tau)] be the solve probability, with expectations over tasks and rollout randomness, and let N(τ)N(\tau) be the number of transmitted consultations in rollout τ\tau. The mean disclosure per consultation under a measure ℓ∈{LP1,OCO⁡}\ell\in\{L_{\mathrm{P1}},\operatorname{OCO}\} is

Dℓ(πθ)=Eπθ[∑j=1N(τ)ℓ(qj;rhj)]Eπθ[N(τ)](6)D_\ell(\pi_\theta)=\frac{\mathbb{E}_{\pi_\theta}\left[\sum_{j=1}^{N(\tau)}\ell(q_j;r_{h_j})\right]}{\mathbb{E}_{\pi_\theta}[N(\tau)]} \tag*{(6)}

where qjq_j is the message transmitted at consultation history hjh_j; it is undefined when Eπθ[N(τ)]=0\mathbb{E}_{\pi_\theta}[N(\tau)] = 0. The expected total disclosure per task is Eπθ[N(τ)]Dℓ(πθ)\mathbb{E}_{\pi_\theta}[N(\tau)]D_\ell(\pi_\theta), and the consultation rate is Ask⁡(πθ)=Pr⁡πθ(N(τ)>0)\operatorname{Ask}(\pi_\theta) = \Pr_{\pi_\theta}(N(\tau) > 0). Table 1 reports pass@1 as the estimate of UU, the two means DLP1D_{\mathrm{LP1}} and DOCOD_{\mathrm{OCO}} pooled over all transmitted consultations, and Ask as Ask%.

Construction of I(r)I(r) and the detector-only count. Both classes are determined automatically. P0 spans are matched by fixed patterns for credentials, URLs, absolute paths, and high-entropy strings, and P1 membership is decided by provenance, that is by membership in I(r)I(r). The inventory I(r)I(r) is assembled from the machine side, never from a sanitizer’s output. The raw machine-side view at prefix hh, the file contents, tracebacks, and command output the agent has actually observed, is tokenized for candidate entities (identifiers, module and file names, and absolute paths), and the versioned allowlist removes standard-library, builtin, and protocol names together with tokens too short or too generic to identify anything. A span in the payload is charged only if it is in I(r)I(r), so a public API name the model could have produced from its own parameters is not counted, and a private name the model paraphrased instead of writing is correctly not counted either. The entity recognizer is the pseudonymizer distributed with SWE-agent (Yang et al., 2024a), used here only to enumerate candidates; the provenance test against I(r)I(r) is what makes the measure independent of any arm’s transform, and it is the reason the Host sanitizer arm, whose transform is that same pseudonymizer, is not scored by its own verdict, which would put its leak near zero by construction.

The gate of Eq. (3) uses a looser detector-only count, which charges every span the recognizer flags in the question, with no provenance test. It is typically larger than LP1L_{\mathrm{P1}}, because it also charges public names that happen to look private. Every disclosure number we report is the provenance-verified count.

For Host sanitizer, the reverse map stays local and is never transmitted. Both W5(r)W_5(r) and w5(q)w_5(q) are taken over code lexemes, namely identifiers, numbers, multi-character operators, and single punctuation marks, after comment stripping, and W5(r)W_5(r) uses only the source-like lines of the observed context. The overlap scorer identifies source-like lines heuristically, so OCO is a retention proxy rather than a precise measurement of proprietary source alone.

Terminal-Bench release

Choice of a frozen release. Terminal-Bench is a continuously evolving benchmark. Tasks are added, modified, and retired between tagged releases, and the public leaderboard tracks the newest one. We evaluate on the frozen 2.0 release, all 89 of its tasks (the 2.1 task set is identical), for two reasons. First, a frozen release keeps results reproducible, since it stays byte-identical, whereas a number reported against a moving target cannot be re-derived once tasks change. Second, the comparison a newer release would buy us is not available to us anyway, because the leaderboard entries are frontier agents driving frontier models through their own harnesses, while our measurements are of a 9B local student under the harness and step budget of Appendix § 10. Our Terminal-Bench numbers are therefore for comparison between the policies in this paper, all of which run under identical conditions, and are not leaderboard-comparable in either direction. Every task in the release is assigned; tasks whose verifier cannot be scored are reported as such rather than dropped. The private-entity inventory is built as on repair tasks, from what the agent observed on the machine before each consultation (Appendix § 9).

Implementation and Hyperparameters

Table 2 records the implementation choices and hyperparameters behind all results.

ItemValue
StudentQwen3.5-9B (Qwen Team, 2026), sampled at temperature 0.8 with thinking disabled
ExpertsDeepSeek-V3.2 for every consulting policy; GPT-5.6 Sol and Claude Opus 4.8 for UVA only
TeacherDeepSeek-V3.2, prompted as in Appendix § 8.§ 2, temperature 0.3, at most 600 output tokens
Training tasksSWE-smith (Yang et al., 2025), whose source repositories are disjoint from the 12 of SWE-bench Verified
CollectionA consultation is forced at step 10 and every 12 steps after it, at most five per rollout, within 75 steps; the reply enters the trajectory and the forcing instruction does not
GateEq. (3) with κP1=2\kappa_{\mathrm{P1}} = 2 on the detector-only count, the strict reduction, VV of Eq. (4), and the replay from the snapshot at hh
Training (UVA)Supervised fine-tuning on q+q^{+} with two repair-step examples per question, which realizes λ\lambda of Eq. (5) as an example ratio; 339 optimizer updates at batch 8, three epochs; learning rate 7×10−57 \times10^{-5}; sequence length 3,072 tokens; prompt cap 9,000 characters
DPO (ablation)The same pairs as preferences against a frozen reference, β=0.1\beta= 0.1, learning rate 5×10−65 \times10^{-6}, the same 339 updates
AdaptationLoRA of rank 16, α=32\alpha= 32, dropout 0.05, on all linear projections of the 4-bit base, merged for serving; seed 42
HardwareTraining on 8 H100 GPUs; student inference on 16 L40S GPUs
Training-data gridNested budgets of 100/200/300/400 accepted pairs, 300 for the student of Table 1; mixture ratio and epochs fixed
EvaluationAt most three consultations per task; step limits of 200 on Verified and 100 on Terminal-Bench; SWE-bench Verified, all 500 instances, three runs with a fixed seed each; Terminal-Bench, all 89 tasks, one run (Appendix § 13)

Table 2. Implementation choices and hyperparameters.

Harness note. The agent’s shell activates the per-instance environment used by the grader. A non-interactive shell that never activates the test environment would silently develop and self-test against the wrong interpreter, which would invalidate the solve numbers; ensuring this activation is a prerequisite for reporting them.

Consultation affordance on both benchmarks. The agent has a single action channel, one fenced command block per turn, and the consultation is offered inside it rather than as a separate tool schema. The system prompt states that a block may carry either a shell command or ask_expert "...", and the harness intercepts the latter, sends the question, and returns the expert’s reply as that command’s output. Asking is therefore the same kind of act as running a command, costs one step, and is timed by the policy itself; nothing in the prompt instructs the model to ask, or when. The budget (at most three consultations per task) is stated in the prompt and enforced by the harness.

Prompt D.1: Consultation affordance, as it appears in every consulting policy’s system prompt

TWO KINDS OF COMMAND go in that block:
1.  Any shell command, run on this machine:
    grep -rn "def resolve" /testbed/src

2.  ask_expert, which sends one self-contained question to the cloud engineer instead
of running anything locally. Their reply comes back as the output of that command, so
you can read it and act on it in your next step. You have three questions for this
``
task:
  ask_expert "The failing test raises KeyError inside resolve(), but the only
  assignment to that key happens in the branch the traceback says was taken.
  What could make that key missing at that point?"
They cannot see your machine, so the question has to carry what you observed, what you
expected instead, and what you are uncertain about.

Prompt D.1 gives the wording for repair tasks. The two benchmarks receive the same affordance in the same position, directly after the response-format rules, as the second of “two kinds of command,” with one worked example whose surface form matches the benchmark’s own work (a failing test on repair tasks, a build or service error on terminal tasks) and with the same closing sentence, that the expert cannot see the machine and the question must therefore carry the observed behavior, the expected behavior, and the open uncertainty. The rest of the two system prompts differs only where the benchmarks already differ, in the working directory, the grading contract, and the submit token.

This symmetry is a measurement requirement, not a stylistic one, because the ask rate (Ask% in Table 1) is one of the reported quantities, and a difference in where the affordance sits in the prompt would be indistinguishable from a difference between the task domains. We verified it by inspection of the rendered prompts, and we report the ask rate as measured under this shared affordance; the consultation policies differ from each other only in what is done to the question after the model writes it (nothing, host pseudonymization, or our trained abstraction), never in how the option is presented.

Loss masks. The question loss covers the tokens of q+q^{+} only, and the ask_expert command that opens the consultation is masked, so no target supervises the decision to consult. The solve loss covers the repair-action tokens; padding, role delimiters, and tool scaffolding enter neither. The DPO baseline additionally uses frozen-reference likelihoods and a rejected-question pass.

Baseline fairness. All learners consume the same accepted questions and the same number of optimizer updates. DPO also trains on q−q^{-} and scores a frozen reference, so it spends more compute per update than Eq. (5), which uses q−q^{-} only to select pairs. The no-replay ablation trains on every rewrite of the same candidate pool that passed the checks before replay, the question-only ablation sets λ=0\lambda= 0 at the same update budget, and Host-SFT replaces q+q^{+} with the sanitizer’s output T(q−)T(q^{-}) under the same recipe and gate, except for the placeholder clause of VV, which that output necessarily violates.

Additional Results

This appendix extends the results of § 5.2. It compares the disclosure of UVA with that of the Host sanitizer (Appendix § 11.§ 1), gives the solve rate of the trained student without an expert and the solve rate split by whether a rollout consulted (Appendix § 11.§ 2), locates the consultations of each policy along the agent steps (Appendix § 11.§ 3), and gives the measured length and the pricing of a consultation behind the cost analysis (Appendix § 11.§ 4). Appendix F analyzes why the untrained consulting baselines fail and follows one of their trajectories.

Disclosure against the host sanitizer

A stricter reading of Table 1 asks whether UVA matches the Host sanitizer’s disclosure without host help, at a solve rate no worse than Natural’s. With the same expert, UVA meets this for verbatim code, with OCO at 5.9 against 6.0 on Verified and 1.4 against 3.6 on Terminal-Bench, and falls short for private identifiers, with LP1L_{P1} at 1.20 against 0.97 and 2.35 against 0.71. The gate of § 4 admits training questions with at most two detector-flagged P1 spans, which bounds the detector count and leaves the provenance count free. A deployment can therefore keep the host transform as a fail-closed floor behind the trained policy. The two are complementary, since the policy removes the surrounding code, which renaming cannot, and the transform replaces the residual names it recognizes.

Solve rate by consultation and expected disclosure per task

Table 3 runs the trained student of Table 1 with consultation disabled, so that every rollout works alone, and sets it against the untrained student working alone and against the trained student with each expert. The gap between the first two rows is what the training changes in the student’s own repair, and the gap between the second row and the expert rows is what consultation adds on top of it. Training alone raises pass@1 from 25.3 to 36.2 on Verified and from 20.2 to 24.7 on Terminal-Bench. Consultation then adds 1.4 and 4.5 points with DeepSeek-V3.2, 2.1 and 11.3 with GPT-5.6 Sol, and 4.4 and 19.1 with Claude Opus 4.8. On Verified, most of the gain over Solo is therefore the training, while on Terminal-Bench the two stronger experts add more than the training does.

ExpertVerifiedTerminal-Bench
None* (Solo)25.320.2
None36.224.7
DeepSeek37.629.2
GPT-5.638.336.0
Opus 4.840.643.8

Table 3. Solve rate of the trained student with consultation disabled, against the untrained student working alone and the trained student with each expert. An asterisk marks the untrained student; every other row is the student trained by UVA, the 300-pair student of Table 1. Both columns are pass@1, over the 500 Verified instances with 3 samples each and a single attempt on the 89 Terminal-Bench tasks.

Table 4 splits pass@1 on Verified by whether a rollout transmitted at least one consultation, and gives the expected number of private entities a policy discloses per task, its consultations per rollout times its LP1L_{P1} (§ 3). The split is descriptive. A rollout consults on its own initiative, so the consulting rollouts are selected by the state the student reached, and under both untrained policies they solve less often than the rest. Host sanitizer is split over the one run its disclosure columns come from (Appendix § 13).

NaturalHost sanitizerOurs
Ask%4.84.67.4
Calls per asking rollout1.401.351.39
Consulting rollouts13/722/2352/111
Non-consulting rollouts356/1,428123/477512/1,389
pass@1, consulting18.18.746.8
pass@1, non-consulting24.925.836.9
LP1L_{\mathrm{P1}} per task0.300.060.12

Table 4. Solve rate by whether a rollout consulted, and expected disclosure per task, on SWE-bench Verified. Natural is over its 1,500 rollouts and Host sanitizer over the one run of 500 behind its disclosure columns in Table 1. Ours consults DeepSeek-V3.2. The rollout rows count solved rollouts over all rollouts in the group, and disclosure per task is Ask% times the calls per asking rollout times LP1L_{\mathrm{P1}} per question.

The expert can change the outcome only of the rollouts that consult, so consulting rollouts contribute at most Ask% points to pass@1, and the measured split places the gain. With DeepSeek-V3.2 as the expert of both, the non-consulting rollouts of UVA solve 512 of 1,389 against 356 of 1,428 for Natural, which accounts for 10.4 of the 13.0-point gain, and its consulting rollouts solve 52 of 111 against 13 of 72, the remaining 2.6 points. The consulting rollouts of the trained student also solve more often than its non-consulting ones, 46.8 against 36.9, whereas under both untrained policies they solve less often than the rest. The split is not measured for Claude Opus 4.8, so at least 15.3 − 7.6 = 7.7 of its 15.3-point gain over Solo arises in rollouts that never consulted.

The timing of consultations

UVA has no training target for when to consult (§ 4), yet its Ask% rises from 4.8 to 7.4 (Table 1). Ask% counts how many rollouts consult but not at which step, so Figure 4 plots the steps at which each policy consults and the step at which each rollout submits its patch, over the rollouts behind Table 1; the Ours row uses DeepSeek-V3.2 as the expert. We analyze only the first 150 steps, which contain every measured consultation and over 91% of submissions. Training changes how often the student consults but not the typical step at which it consults. The trained student consults at a median step of 39, against 38 for Natural, and submits later, at a median step of 92 against 81.

Figure 4. Consultation and submission timing per policy. The upper half of each violin is the distribution of the steps at which the policy consulted, and the mirrored lower half that of the step at which each submitting rollout handed in its patch. Darker shading marks the interquartile range, and each median line carries its step. All halves share one kernel width and peak height and span only the steps their data cover, so the violins compare the shapes of the distributions, and Ask% in Table 1 gives how often each policy consults.

Consultation length and pricing

The cloud cost of Figure 3(c) multiplies, for each expert, its Ask% from Table 1 by the consultations per asking rollout, by the tokens of a consultation, and by the provider’s published price per token. The consultations per asking rollout are 1.40, measured on Natural as 101 answered consultations over 72 consulting rollouts, and applied to every expert. The length is measured on the same 101 consultations, none of them truncated. The fixed expert system prompt is 368 characters, and the question and the reply average 487 and 542 characters. We convert characters to tokens at four characters per token, since the experts tokenize differently, which moves every point by the same factor. The prices are OpenRouter’s rates per million input and output tokens, retrieved on 2026-09-23, $0.269 and $0.40 for DeepSeek-V3.2, $2 and $10 for GPT-5.6 Sol, and $5 and $25 for Claude Opus 4.8. The student’s own inference is priced at the same provider’s $0.10 and $0.15 for Qwen3.5-9B.

Baseline Failure Analysis

This appendix examines why the two untrained consulting baselines fall short, over the SWE-bench Verified rollouts behind Table 1. Natural consults in 72 of its 1,500 rollouts, with 101 consultations, and Privacy prompt in 7 of its 500, with 14. Disclosure is scored with the provenance inventory of Appendix § 9, resolution with the strict rule of Table 1, and a file counts as a gold file when the reference patch edits it. Table 5 closes the appendix with a trajectory from Privacy prompt.

Tasksympy__sympy-20438, “is_subset gives wrong results”. One consultation, at step 21 of 116, after the student has reproduced the failure.
QuestionThe failing test shows that ProductSet and FiniteSet with the same elements are not considered equal (b == c is False but set(b) == set(c) is True). When is_subset is called, it falls back to comparing self.intersect(other) == self, which fails for ProductSet because the intersection returns a FiniteSet. Additionally, the simplify() of Eq(b, c) fails with AttributeError: ’Complement’ object has no attribute ’equals’. What is the structural cause of this issue and how should is_subset be fixed?
DisclosureLP1L_{\mathrm{P1}} = 4, all four entities also named by the issue text; OCO = 19.
Expert replyThe structural cause is that ProductSet does not implement __eq__ or equals to compare element-wise with FiniteSet [. . . ] Immediate step: Override is_subset in ProductSet to compare element containment without relying on intersect.
OutcomeThe student adds that override to sympy/sets/sets.py and guards the equals call in sympy/core/relational.py, and the tests fail. The reference patch edits sympy/core/relational.py and the set-comparison handlers under sympy/sets/handlers/, and leaves sets.py untouched.

Table 5. A Privacy prompt consultation that discloses four entities and misdirects the repair. Private entities counted by LP1L_{\mathrm{P1}} are in bold; the expert reply is truncated.

Natural consults and still fails. 59 of the 72 consulting rollouts fail, and consulting rollouts solve less often than the rest, 18.1% against 24.9% (Table 4). The failures end close to the fix. 40 of the 59 submit a patch that edits a gold file and still fails the tests, 13 edit only other files, and 6 submit no patch, so in most failures the student has already found where to change the code and lacks the right change. The questions it sends leave out the evidence it holds. Before 72 of the 85 failing consultations the student has seen a traceback or a failing test, yet none of the 85 questions carries a traceback and 24 name an error type. The questions typically restate the issue together with the student’s own hypothesis, so the expert answers from the description alone, and its advice names a file in 11 of the 85. Two further patterns recur. 14 of the 101 consultations ask the expert to decide whether an existing test or the issue is right, a judgement that needs the test the expert never sees, and 7 reproduce the worked example of Prompt D.1 verbatim, a question about a KeyError in resolve() that belongs to none of the tasks. 91 of the 101 consultations disclose at least one private entity.

The privacy prompt still discloses. All 14 consultations of Privacy prompt disclose at least one private entity, at a mean of 4.64 against the 4.42 of Natural, and 12 of them carry verbatim source 5-grams. On reading all 14, none replaces a name with a description of its role and type, which is the core of the instruction. The instruction changes where the disclosed names come from rather than how many there are. Per consultation, Privacy prompt discloses 1.2 entities that appear only in what the student read on the machine, against 2.4 for Natural, and 3.4 that the issue text also uses, against 2.0. The student drops some names it found in files and restates the issue in the issue’s own names. Identifiers make up 54% of its disclosed entities, and file and module names 42%. Its lower disclosure per task comes from asking in 7 of 500 rollouts (§ 5.2), and its 6 failing rollouts end as those of Natural do, 4 of them with a patch on a gold file.

A trajectory. Table 5 follows the one consultation of a Privacy prompt rollout that shows both failures. The question names four repository entities, every one of which the issue text also uses, and quotes the failing expressions, although the system prompt asks for role-and-type descriptions. The expert, answering from the description alone, proposes a mechanism, and the student implements it in a file that the reference patch leaves untouched.

Statistical Details

On SWE-bench Verified, every policy is run three times over all 500 instances with a fixed seed per run, so each row of Table 1 rests on 1,500 rollouts. pass@1 is the resolved fraction of those rollouts, that is the mean over instances of the per-instance solve rate, so it is a count over 1,500 rather than 500. pass@3 is the fraction of the 500 instances resolved by at least one of the three runs. On Terminal-Bench, every policy is run once over all 89 tasks. Ask% on either benchmark is the fraction of rollouts with at least one transmitted consultation. On Verified, the Ask% and disclosure columns of Privacy prompt and Host sanitizer come from one of their three runs, 500 rollouts each, and those of the other consulting rows from all 1,500.

Two exact tests are used. Paired comparisons of two policies on the same tasks use the exact McNemar test with discordant counts (b,cb,c), as for the Terminal-Bench policies in § 5.2. Consultation rates of two policies are compared by Fisher’s exact test on the counts of consulting rollouts, as in the collaboration-quality paragraph of § 5.2. A non-significant difference is reported as inconclusive rather than as equivalence, and the conditional solve rates of Appendix § 11.§ 2 are descriptive.

Artifacts and Licenses

We use every artifact for research evaluation, consistent with its intended use, and we release our own code for research use under the MIT license. SWE-bench Verified (Jimenez et al., 2024) is distributed through the MIT-licensed SWE-bench code base; its dataset card states no separate license, and each task repository is a public open-source project under its own license. Terminal-Bench 2.0 (Merrill et al., 2026) is released under Apache-2.0, and the SWE-smith code and dataset (Yang et al., 2025) under MIT. The student, Qwen3.5-9B (Qwen Team, 2026), is released under Apache-2.0, and the weights of DeepSeek-V3.2 (DeepSeek-AI et al., 2025) under MIT. The experts and the teacher were accessed through commercial API providers under their terms of service; GPT-5.6 Sol (OpenAI, 2026) and Claude Opus 4.8 (Anthropic, [5]) are proprietary models subject to their providers’ usage policies. The mini-SWE-agent harness and the SWE-agent pseudonymizer (Yang et al., 2024a) are MIT-licensed. Issue texts and task instructions are in English.

References

  1. [2]Rajeev Bhatt Ambati, Tianyi Niu, Aashu Singh, Shlok Mishra, Snigdha Chaturvedi, and Shashank Srivastava. Socratic students: Teaching language models to learn by asking questions, 2026. https://arxiv.org/abs/2512.13102.
  2. [3]Shengnan An, Zexiong Ma, Zeqi Lin, Nanning Zheng, Jian-Guang Lou, and Weizhu Chen. Learning from mistakes makes llm better reasoner, 2024. https://arxiv.org/abs/2310.20689.
  3. [4]Chinmaya Andukuri, Jan-Philipp Fränken, Tobias Gerstenberg, and Noah D. Goodman. Star-gate: Teaching language models to ask clarifying questions, 2024. https://arxiv.org/abs/2403.19154.
  4. [5]Anthropic. System card: Claude Opus 4.8, May 2026. https://www.anthropic.com/claude-opus-4-8-system-card.
  5. [6]Akari Asai, Zeqiu Wu, Yizhong Wang, Avirup Sil, and Hanneah Hajishirzi. Self-rag: Learning to retrieve, generate, and critique through self-reflection. In The Twelfth International Conference on Learning Representations, ICLR 2024, Vienna, Austria, May 7-11, 2024. OpenReview.net, 2024. https://openreview.net/forum?id=hSyW5go0v8.arxiv.org/abs/2310.11511
  6. [7]Eugene Bagdasarian, Ren Yi, Sahra Ghalebikesabi, Peter Kairouz, Marco Gruteser, Sewoong Oh, Borja Balle, and Daniel Ramage. Airgapagent: Protecting privacy-conscious conversational agents. In Proceedings of the 2024 on ACM SIGSAC Conference on Computer and Communications Security, CCS ’24, page 3868–3882, New York, NY, USA, 2024. Association for Computing Machinery. ISBN 9798400706363. doi: 10.1145/3658644.3690350. https://doi.org/10.1145/3658644.3690350.
  7. [8]Yuntao Bai, Saurav Kadavath, Sandipan Kundu, Amanda Askell, Jackson Kernion, Andy Jones, Anna Chen, Anna Goldie, Azalia Mirhoseini, Cameron McKinnon, Carol Chen, Catherine Olsson, Christopher Olah, Danny Hernandez, Dawn Drain, Deep Ganguli, Dustin Li, Eli Tran-Johnson, Ethan Perez, Jamie Kerr, Jared Mueller, Jeffrey Ladish, Joshua Landau, Kamal Ndousse, Kamile Lukošuite, Liane Lovitt, Michael Sellitto, Nelson Elhage, Nicholas Schiefer, Noemi Mercado, Nova DasSarma, Robert Lasenby, Robin Larson, Sam Ringer, Scott Johnston, Shauna Kravec, Sheer El Showk, Stanislav Fort, Tamera Lanham, Timothy Telleen-Lawton, Tom Conerly, Tom Henighan, Tristan Hume, Samuel R. Bowman, Zac Hatfield-Dodds, Ben Mann, Dario Amodei, Nicholas Joseph, Sam McCandlish, Tom Brown, and Jared Kaplan. Constitutional ai: Harmlessness from ai feedback, 2022. https://arxiv.org/abs/2212.08073.
  8. [9]Jan Betley, Niels Warncke, Anna Sztyber-Betley, Daniel Tan, Xuchan Bao, Martín Soto, Megha Srivastava, Nathan Labenz, and Owain Evans. Training large language models on narrow tasks can lead to broad misalignment. Nature, 649(8097):584–589, January 2026. ISSN 1476-4687. doi: 10.1038/s41586-025-09937-5. http://dx.doi.org/10.1038/s41586-025-09937-5.
  9. [10]Ishaan Bhola, Adithyan Krishnan, and Mukunda NS. Scrouting: Cost-aware routing of coding agents by scouting the repository first, 2026. https://arxiv.org/abs/2608.04804.
  10. [11]Federico Bianchi, Mirac Suzgun, Giuseppe Attanasio, Paul Röttger, Dan Jurafsky, Tatsunori Hashimoto, and James Zou. Safety-tuned llamas: Lessons from improving the safety of large language models that follow instructions. In The Twelfth International Conference on Learning Representations, ICLR 2024, Vienna, Austria, May 7-11, 2024. OpenReview.net, 2024. https://openreview.net/forum?id=gT5hALch9z.
  11. [12]Nicholas Carlini, Chang Liu, Úlfar Erlingsson, Jernej Kos, and Dawn Song. The secret sharer: evaluating and testing unintended memorization in neural networks. In Proceedings of the 28th USENIX Conference on Security Symposium, SEC’19, page 267–284, USA, 2019. USENIX Association. ISBN 9781939133069.arxiv.org/abs/1802.08232

Paper details

vlnc:W_vqh6dvcc
Version
v1 · latest
Published
Fresh — added to the Hub within the last 30 days
Added
License
CC0-1.0
Usage

views · 9

downloads · 0

A view counts once per browser per day, a download each time the PDF is requested.

  • Computer Science

Not recorded for this paper

View statement
Looking for related papers…
Contents