betaFeedback

WorksComputer Science

Option-GRPO: Hierarchical RL Against AI Slop in Code Generation for Unseen SDKs

Abstract

Large language models increasingly produce AI slop in generated code: verbose, redundant, non-idiomatic implementations that pass functional checks but bloat codebases. The problem compounds on SDKs released after the model's training cutoff, where the model has neither idiomatic priors nor compositional understanding. We present Option-GRPO, a hierarchical reinforcement learning framework that adapts a code LLM to an unseen SDK from its source code alone, without human-written usage examples or test cases, and that explicitly optimizes for both functional correctness and code elegance. Our key insight is that SDK source code contains compositional structure that defines a hierarchy of reusable code patterns (code options) which simultaneously serve as the building blocks of a curriculum and the action space for efficient policy optimization. We show that code options induce a valid semi-Markov decision process, give conditional bounds under which shorter option description length implies more concise and, in expectation, more API-efficient programs, and analyze under stated assumptions how the option hierarchy improves credit assignment by a factor of T/K\sqrt{T/K}. Using Group Relative Policy Optimization (GRPO) with execution-based rewards at each curriculum level, Option-GRPO discovers SDK idioms, composes them into patterns, and assembles patterns into complete programs. Experiments on 12 unseen SDKs with a 32B-parameter base model demonstrate 61.4% pass@1, outperforming flat GRPO by 8.7 points, ExploraCoder by 11.2 points, and ReCode by 6.3 points. Option-GRPO produces solutions that use 34.7% fewer API calls and are 27% shorter than flat-GRPO outputs while maintaining higher correctness, directly attacking the slop problem.

Introduction

The rapid proliferation of software development kits creates a fundamental challenge for code generation: the libraries developers most need help with are precisely those released after model training cutoffs, where LLM performance degrades substantially [28, 29]; recent benchmarks confirm that performance also degrades on evolving, version-specific library APIs [13, 12, 9]. A second, increasingly visible problem compounds the first: even on familiar libraries, modern code LLMs generate what practitioners have begun to call AI slop [26] (Merriam-Webster’s 2025 Word of the Year), output that is fluent line by line yet bloated, redundant, and non-idiomatic when read as a whole. Industry studies of 211 million lines of production code report that duplicated 5+ line blocks grew roughly eightfold between 2020 and 2024, while the share of refactoring edits dropped by 60% [6].

Existing retrieval-augmented and exploration-based approaches address the first problem partially. DocPrompting [30] retrieves documentation; Gorilla [16] fine-tunes for API-call prediction with retrieval awareness; ExploraCoder [24] chains multi-step exploration over unseen APIs without training; and ReCode [27] applies rule-based RL to update an LLM’s API knowledge after library version changes. None of these methods explicitly optimize for code quality beyond functional correctness, and none discover compositional idioms from SDK structure that would allow the model to write elegant solutions in addition to working ones.

We observe that SDK code has an inherent hierarchical structure that mirrors the options framework in hierarchical reinforcement learning [21]: API calls are atomic actions, commonly composed call sequences form reusable patterns analogous to options (temporally extended actions in HRL), and complete programs are compositions of these patterns. This hierarchy is encoded in the SDK source itself: type relationships and parameter-flow patterns dictate which functions compose, and frequently composable subgraphs correspond to natural programming idioms.

We present Option-GRPO, a framework that formalizes this connection: each stage of a learning curriculum produces reusable code options that become the action space for the next stage. At each level, Group Relative Policy Optimization (GRPO) [19] with execution-based rewards optimizes the policy without requiring a critic network. The group-relative comparison within GRPO, combined with the option-level action space, provides natural credit assignment: determining which option in a composed sequence caused success or failure becomes tractable when comparing groups of option sequences rather than groups of token sequences.

Experiments across 12 unseen SDKs demonstrate that Option-GRPO achieves 61.4% pass@1 with a Qwen3-32B base, improving over flat GRPO (52.7%), ExploraCoder (50.2%), and ReCode (55.1%). More notably, generated code is 27% shorter and uses 34.7% fewer API calls, while maintaining higher functional correctness. Analysis shows that options provide sufficient credit assignment for GRPO to converge stably on tasks where flat GRPO fails due to reward sparsity, and the resulting policy reuses idiomatic SDK patterns in 74% of generated tokens, against 41% for flat GRPO.

Contributions.

  1. Option-GRPO, a hierarchical RL framework for example-free SDK mastery that formalizes curriculum stages as reusable code options, optimized via GRPO with execution-based rewards (§ 3).

  1. Theoretical analysis giving conditional bounds, under explicit structural assumptions, that relate option description length to code conciseness and API efficiency (§ 3.5).

  1. Graph-based option discovery from SDK source code, using parameter flow graphs and semantic clustering to identify reusable code patterns without any usage examples (§ 3.3).

  1. Option-GRPO credit assignment, showing that group-relative comparison over option sequences provides natural credit assignment that compensates for the absence of a critic, with convergence analysis (§ 3.6).

  1. UnseenSDK-Bench with automated reward computation via layered execution validation and property-based testing, enabling RL training without human-written test cases (§ 4).

  1. Comprehensive experiments demonstrating improvements in both functional correctness and code quality, with analysis of option reuse, curriculum pacing, and cross-SDK transfer (§ 5).

Related Work

RL for code generation. CodeRL [10], RLTF [11], PPOCoder [20], and StepCoder [5] all combine execution feedback with actor-critic RL, requiring pre-existing unit tests for reward and a critic network for advantage estimation. We eliminate both requirements: rewards come from a layered execution validator plus property tests synthesized from SDK type signatures, and credit assignment uses GRPO’s [19, 3] group-relative comparison.

Hierarchical RL and options. The options framework [21] introduced temporally extended actions; Option-Critic [1] learns termination conditions end-to-end; MAXQ [4] and FeUdal Networks [22] build task hierarchies. For language agents, Voyager [23] and DECKARD [15] maintain external skill libraries via GPT-4 prompting without updating model parameters. We derive options from SDK source code structure rather than open-ended exploration and update model parameters through gradient-based RL.

Curriculum learning. Bengio et al. [2] introduced curriculum learning; Narvekar et al. [14] and Portelas et al. [17] survey curriculum approaches for RL. AlphaCode (Li et al., 2022) uses a two-stage filter for competitive programming. No prior work formalizes curriculum completion as producing temporally extended actions that define the next stage’s action space, which is our central conceptual contribution.

Unseen API/SDK code generation. DocPrompting [30] retrieves documentation; Gorilla (Patil et al., [16] fine-tunes for API-call prediction; ExploraCoder (Wang et al., [24] chains exploration over unseen APIs without training; ReCode (Wu et al., [27] applies rule-based GRPO for API version migration; CodeRAG-Bench (Wang et al., [25] evaluates retrieval-augmented code generation. None explicitly optimize for code quality beyond functional correctness, and none discover compositional idioms from SDK structure. Table 1 summarizes the comparison.

MethodRLNo ExamplesStructuralHierarchicalCode QualityNo Critic
CodeRL✓×××××
PPOCoder✓×✓×××
StepCoder✓×××××
ExploraCoder×✓×××N/A
ReCode✓✓×××✓
Voyager†×××✓×N/A
Option-GRPO (Ours)✓✓✓✓✓✓

Table 1. Comparison with most relevant prior work. Option-GRPO uniquely combines hierarchical RL with structural SDK knowledge for example-free generation. †Voyager builds a skill library via iterative GPT-4 prompting, not gradient-based RL; we mark its RL column accordingly.

Method

We present Option-GRPO, a hierarchical reinforcement learning framework for enabling LLMs to master novel SDKs from source code alone. Figure 1 provides an overview. The method operates in four stages: (1) graph construction and option discovery from SDK source code, (2) curriculum-based hierarchical training with GRPO at each level, (3) execution-based reward computation without test cases, and (4) inference with the trained hierarchical policy.

Overview of Option-GRPO

Figure 1. Overview of Option-GRPO. The top row constructs three SDK graphs from source code and discovers a hierarchical option vocabulary A0⊂A1⊂A2\mathcal{A}_0 \subset\mathcal{A}_1 \subset\mathcal{A}_2 (Algorithm 1). The middle row runs three curriculum phases of GRPO training, each receiving the option vocabulary from above and the layered execution reward from below (dashed arrows). Successful trajectories from phase kk contribute newly extracted options to phase k+1k+1 (Algorithm 2). The trained policy πθ\pi_\theta generates code at inference. Solid arrows denote training-stage progression; dashed arrows denote artifact dependencies.

Problem Formulation

Let S\mathcal{S} denote a novel SDK with source code but no usage examples in the model’s training data. Given a natural language task description qq and optional setup code csetupc_{\mathrm{setup}}, we seek to learn a policy:

πθ:(q,csetup,S)→csolution(1)\pi_{\theta} : (q, c_{\mathrm{setup}}, \mathcal{S}) \to c_{\mathrm{solution}} \tag*{(1)}

where θ\theta represents learnable parameters. Unlike prior work that retrieves documentation at inference time (RAG, DocPrompting) or fine-tunes only over API surface forms (Gorilla, ReCode), we train the policy through interaction with an execution environment, using the SDK’s source code structure to define a hierarchy of reusable code abstractions that the policy learns to compose.

Background and Formulation

Options. Following Sutton et al. [21], an option is a triple o=(I,πo,β)o = (\mathcal{I}, \pi_o, \beta) where I⊆S\mathcal{I} \subseteq\mathcal{S} is the initiation set, πo\pi_o is the intra-option policy over primitive actions, and β:S→[0,1]\beta: \mathcal{S} \to[0, 1] is the termination probability. A pair (MDP, options) induces a semi-Markov decision process (SMDP) in which option durations are random.

Code generation MDP. The code generation process is an MDP (S,A,T,R)(\mathcal{S}, \mathcal{A}, T, R) where states st=(q,c0:t)s_t = (q, c_{0:t}) concatenate the prompt with generated tokens, actions at∈Va_t \in\mathcal{V} are next tokens, TT is deterministic append, and RR is computed by execution at episode termination. Several properties make this MDP hard for standard RL: ∣V∣≈1.5⋅105|\mathcal{V}| \approx1.5 \cdot10^5, trajectory length TT grows unbounded, and reward is sparse (zero on most trajectories for novel SDKs). These challenges motivate both the option hierarchy (reducing effective action space and trajectory length) and GRPO (extracting signal from group comparison instead of per-state value estimation).

Code Options: From SDK Structure to Hierarchical Actions

We define code options as temporally extended actions in the code-generation MDP, derived from SDK source code.

Definition 1 (Code Option). A code option for SDK S\mathcal{S} is a 4-tuple

ok=(Ik⏟handcrafted from Gflow,Fk,βk,πk⏟learned from πθ)o_k = \left( \underbrace{\mathcal{I}_k}_{\text{handcrafted from } \mathcal{G}_{\mathrm{flow}}}, \mathcal{F}_k, \beta_k, \underbrace{\pi_k}_{\text{learned from }\pi_{\theta}} \right)

where Ik={s:type⁡(last_value⁡(s))∈Tkin}\mathcal{I}_k = \{s : \operatorname{type}(\operatorname{last\_value}(s)) \in\mathcal{T}^{\mathrm{in}}_k\} is the initiation set; Fk⊂F\mathcal{F}_k \subset F is the option’s allowed SDK function set (the cluster centroid path from Algorithm 1); βk(s)=I[type⁡(s)∈Tkout]\beta_k(s) = \mathbb{I}[\operatorname{type}(s) \in\mathcal{T}^{\mathrm{out}}_k] is the termination predicate; and πk\pi_k is the intra-option policy. Operationally, πk\pi_k is the underlying network πθ\pi_{\theta} restricted to tokens whose next-call resolution lies in Fk\mathcal{F}_k, with this restriction enforced by setting all out-of-set token logits to −∞-\infty before the softmax at every step the option is active.

Selecting only among options in O=O0∪O1\mathcal{O} = \mathcal{O}_0 \cup\mathcal{O}_1 and executing each to termination induces a valid SMDP by Theorem 1 of Sutton et al. [21]; the formal statement is Proposition 1 in Appendix § 8.

Two-level discovery (candidates and certified options). We distinguish two stages of option production. Candidate options are generated upfront from the SDK source code by Algorithm 1; we denote the candidate set at level kk by Okcand\mathcal{O}^{\mathrm{cand}}_k. Certified options are the subset of candidates that the trained policy invokes successfully during the corresponding curriculum phase, extracted by Algorithm 2; we denote them by Ok\mathcal{O}_k. Only certified options enter the next phase’s action vocabulary.

Level-0 candidates O0cand\mathcal{O}^{\mathrm{cand}}_0 are one per public SDK function: Ii\mathcal{I}_i is the parameter type, Fi={fi}\mathcal{F}_i = \{f_i\}, πi\pi_i emits a single call, βi\beta_i terminates on return. Level-1 candidates O1cand\mathcal{O}^{\mathrm{cand}}_1 are mined from the parameter-flow graph GflowG_{\mathrm{flow}} for frequently composable paths via Algorithm 1: we enumerate length-2 to length-4 paths weighted by type compatibility, cluster paths by sentence-transformer embedding of their function identifiers (threshold τcluster=0.8\tau_{\mathrm{cluster}}=0.8), and define one Level-1 candidate per cluster centroid with FjF_j equal to the function set of that centroid path. A candidate becomes a certified option only after the policy trained at Level kk demonstrates that it can invoke the candidate correctly (success threshold ρ=0.7\rho= 0.7, occurrence frequency threshold ν=5\nu= 5 across distinct tasks; see Algorithm 2). Level-2 composition is neither pre-discovered nor extracted explicitly; it is learned implicitly in Phase 3 as the high-level policy composes certified Level-0 and Level-1 options.

Option hierarchy. The option hierarchy is the nested action-space chain A0=V⊂A1=O0∪V⊂A2=O0∪O1∪V\mathcal{A}_0 = \mathcal{V} \subset\mathcal{A}_1 = \mathcal{O}_0 \cup\mathcal{V} \subset\mathcal{A}_2 = \mathcal{O}_0 \cup\mathcal{O}_1 \cup\mathcal{V}, ensuring higher levels retain full token expressiveness as a fallback. Two options oio_i, ojo_j are composable (oi≺ojo_i \prec o_j) iff Tiin∩Tjout≠∅\mathcal{T}^{\mathrm{in}}_i \cap\mathcal{T}^{\mathrm{out}}_j \ne\varnothing; the composability relation defines a graph GoptG_{\mathrm{opt}} that constrains the high-level policy to valid sequences.

Decode-time mechanics. Each option is one added vocabulary entry that starts the option and adds no text to the program; it is offered only when the value just produced has a type its first function accepts. While an option is active, an incremental tracker over the emitted code masks every token that does not spell toward the next function in Fk\mathcal{F}_k on the main call chain, allows only argument-building helper functions inside a call’s brackets, and leaves names, literals, and operators free. The option ends when the closing bracket of its last function is emitted; one that exceeds a length cap or reaches a dead end is closed and marked failed (Appendix § 9.§ 2).

Handcrafted structure, learned behavior. Our construction occupies a middle ground in the options-framework design space. The structural components (IkI_k, FkF_k, βk\beta_k) are handcrafted: they are read directly from the parameter-flow graph by Algorithm 1 and are symbolic, type-checkable, and reproducible at parse time. The behavioral component πk\pi_k is learned: it is πθ\pi_\theta restricted (by action-masking on decoder logits) to emit tokens consistent with FkF_k, with parameters fine-tuned end-to-end by curriculum-staged GRPO. Two invocations of the same option therefore satisfy the same contract (same SDK calls, same type flow, same output type) even when the surrounding glue tokens (variable names, formatting) differ across invocations. This hybrid differs from the fully-handcrafted macros of Korf [8] and the planning-trace macros of Iba [7] (no learning inside), from the fully-learned Option-Critic of Bacon et al. [1] (no handcrafted structure), and from Voyager’s prompt-orchestrated skill library (Wang et al., [23]) (no gradient updates to the intra-option policy). The structural handcrafting buys verifiable, type-checked option boundaries; the learned behavior lets the same option produce contextually appropriate code across invocations without us having to write each variant by hand.

Remark 1 (Learned vs. Constructed). O\mathcal{O} names which sub-behaviours of πθ\pi_\theta are reified as temporally extended actions; FkF_k is the symbolic handle (read from the graph); πk\pi_k is the corresponding slice of πθ\pi_\theta (parameters learned end-to-end). Options are named, type-checked regions of a single learned policy, not separate networks or templates.

Curriculum as Option Construction

The central conceptual contribution is the equivalence between curriculum stages and option construction. A curriculum-option stage CkC_k has three components: (1) a task distribution DkD_k of difficulty kk generated from SDK-graph paths of length ≤k+1\le k+1; (2) GRPO training on DkD_k with action space AkA_k, taking the candidate pool Okcand\mathcal{O}^{\mathrm{cand}}_k supplied by Algorithm 1 as the inventory the policy may invoke; (3) an option-certification step (Algorithm 2, Appendix) that filters Okcand\mathcal{O}^{\mathrm{cand}}_k down to the certified subset Ok\mathcal{O}_k of candidates that the trained policy invokes correctly on a sufficient fraction of tasks. The certified subset of stage CkC_k augments the action space for Ck+1C_{k+1}:

C0→O0C1→O1C2.C_0 \xrightarrow{\mathcal{O}_0} C_1 \xrightarrow{\mathcal{O}_1} C_2.

Completing stage kk does not merely yield a better policy; it yields certified temporally extended actions that become primitives at the next level. The curriculum is thus the mechanism that converts graph-mined candidates into validated options. The role split is clean: Algorithm 1 (static, runs once before training) proposes; Algorithm 2 (dynamic, runs after each phase) certifies. This relationship between curriculum stages, in which stages share neither policy parameters nor reward signal but instead share an action vocabulary that grows monotonically with each certification, is what we name curriculum-as-options. Level-0 tasks (87–312/SDK) involve single API calls; Level-1 tasks (174–624/SDK) involve 2–4 composed calls; Level-2 tasks (33–34/SDK) are full programs. See Remark 2 in Appendix § 8 for the empirical relationship between candidate and certified option counts.

Theoretical Analysis

We summarize the analysis here, centred on Theorem 2 (anchored empirically by Figure 6) and Proposition 3 (anchored by Figure 5), and state their assumptions first. The supporting statements (SMDP validity, Proposition 1; an MDL-style prior over option sequences, Proposition 2; search-space reduction, Proposition 5), full statements, supporting definitions, and proofs are in Appendix § 8.

Per-step relative gradient norm variance for Flat GRPO and Option-GRPO

Figure 5. Per-step relative gradient norm variance (averaged over 12 SDKs). Option-GRPO shows roughly 2.3× lower variance at the end of training (about 1.5× in SNR), in the direction predicted by Proposition 3 but smaller than its T/K\sqrt{T/K} scaling.

Code quality metrics against ODL on passing solutions

Figure 6. Code quality metrics against ODL on passing solutions (a sample of the solutions is plotted; the correlations in the legend are computed over all 5,042). All four metrics show strong negative correlation with ODL, consistent with Theorem 2 (conciseness), Corollary 1 (API efficiency), and the empirical predictions of Remark 5 (QredQ_{\mathrm{red}} omitted from plot for legibility; ρred=−0.62\rho_{\mathrm{red}}=-0.62, p<10−50p<10^{-50}).

Assumptions. Theorem 2 assumes three structural conditions on the hierarchy H\mathcal{H} (per-option API minimality, non-subsumption, bounded glue code; Remark 4), compares programs that are equivalent on the property-test suite P\mathcal{P} rather than fully semantically equivalent (Definition 2), and requires a dispersion-ratio condition on option lengths. Proposition 3 assumes that glue positions are well predicted by πθ\pi_\theta with variance that does not align with the reward signal, and that token positions are conditionally independent.

(i) Curriculum monotonicity. Let Vk∗V_k^* denote the expected value of the optimal policy at curriculum level kk under hierarchy H\mathcal{H}. Because the action spaces are nested, V0∗≤V1∗≤V2∗V_0^* \le V_1^* \le V_2^* (weak monotonicity); this concerns optimal policies and does not by itself guarantee that the trained policy improves at each level. Strict inequality holds whenever the certified Level-kk option set Ok\mathcal{O}_k contains at least one option that strictly improves over the best Level-(k−1)(k-1) action on a reachable state (Theorem 1 and Remark 3).

(ii) Option compression bounds on code quality. Define observational equivalence c≡Pc′c \equiv_{\mathcal{P}} c' on the property-test suite P\mathcal{P} (Definition 2), the option description length ODL⁡H(c)\operatorname{ODL}_{\mathcal{H}}(c) as the minimum number of options whose sequential execution is observationally equivalent to cc (Definition 3), and four code-quality metrics: conciseness Qconc=Tref/∣c∣Q_{\mathrm{conc}} = T_{\mathrm{ref}}/|c|, API efficiency Qapi=Nmin/NapiQ_{\mathrm{api}} = N_{\mathrm{min}}/N_{\mathrm{api}}, redundancy Qred=1−Nredundant/NvarsQ_{\mathrm{red}} = 1-N_{\mathrm{redundant}}/N_{\mathrm{vars}}, and idiomaticity Qidiom=∣option-covered tokens∣/∣c∣Q_{\mathrm{idiom}} = |\text{option-covered tokens}|/|c| (Definition 4). Under three structural conditions on H\mathcal{H} (per-option API minimality, non-subsumption, bounded glue code; Remark 4) we prove a token bound mτˉmin⁡≤∣c∣≤mτˉmax⁡+(m−1)Gmax⁡m\bar{\tau}_{\min} \le|c| \le m\bar{\tau}_{\max} + (m-1)G_{\max} (Lemma 1); a conciseness theorem ODL⁡(c1)<ODL⁡(c2)⇒∣c1∣<∣c2∣\operatorname{ODL}(c_1) < \operatorname{ODL}(c_2) \Rightarrow|c_1| < |c_2| under a dispersion-ratio condition (Theorem 2); and an API-efficiency corollary in expectation (Corollary 1). The metrics Qred,QidiomQ_{\mathrm{red}}, Q_{\mathrm{idiom}} do not admit clean closed-form bounds; we treat them as empirical predictions verified in § 5 (Remark 5).

(iii) Improved credit assignment. Partitioning a generated program into KK option-decision tokens and T−KT-K glue tokens (Definition 5), the GRPO policy-gradient SNR scales as Θ(K/T)\Theta(K/\sqrt{T}) for flat token-level GRPO and Θ(K)\Theta(\sqrt{K}) for option-level GRPO (Proposition 3); the relative improvement is Θ(T/K)≈4.5×\Theta(\sqrt{T}/K) \approx4.5\times for typical T=100T=100, K=5K=5. A companion proposition (Proposition 4) bounds GRPO iterations to first success under a base-LLM prior, yielding additive (Option-GRPO) versus multiplicative (flat) exploration cost across curriculum levels.

Option-GRPO: Hierarchical Policy Optimization

Standard GRPO (Shao et al., [19]) samples GG outputs per prompt, computes group-normalised advantages A^i=(Ri−mean⁡)/(std⁡+ϵ)\hat{A}_i = (R_i - \operatorname{mean})/(\operatorname{std}+\epsilon), and applies the PPO-clipped policy gradient with a KL penalty against πref\pi_{\mathrm{ref}}. At curriculum level k≥1k \ge1 we generate option sequences instead of token sequences and add an option-level advantage A^opt(o)=1∣So+∣∑i∈So+Ri−1∣So−∣∑i∈So−Ri\hat{A}^{\mathrm{opt}}(o) = \frac{1}{|S_o^+|}\sum_{i\in S_o^+} R_i - \frac{1}{|S_o^-|}\sum_{i\in S_o^-} R_i that compares groups containing vs. excluding each option; the combined gradient balances sequence-level ranking with per-option credit. Per-option credit tightens credit assignment: under the assumptions of Proposition 3 (Appendix § 8), the SNR improves by Θ(T/K)\Theta(\sqrt{T/K}), since only option-decision positions carry reward-correlated signal while flat GRPO spreads the same signal across all TT token positions. Proposition 4 additionally bounds the expected number of GRPO iterations to first success under a base-LLM prior. Full equations and analysis are in Appendix § 8.

Execution-Based Reward Without Test Cases

The total reward is R(c)=Rexec(c)+λqRquality(c)+λcRconsistency(c)R(c) = R_{\mathrm{exec}}(c) + \lambda_q R_{\mathrm{quality}}(c) + \lambda_c R_{\mathrm{consistency}}(c). The layered execution reward RexecR_{\mathrm{exec}} assigns graded scores by execution depth: 0.0 (fails to parse), 0.1 (parses but imports fail), 0.3 (runtime error), 0.5 (executes, output None), 0.7 (output matches expected type), 1.0 (passes property-based tests). Property-based tests (Algorithm 3, Appendix) are auto-generated from SDK type annotations and cover type preservation, shape consistency, determinism, edge cases, and assertion invariants extracted from source code. The quality reward bundles pattern-use, no-redundancy, minimal-API, and no-reimplementation bonuses (αi=0.1\alpha_i = 0.1). The self-consistency reward Rconsistency(yi)=∣{yj:output⁡(yj)=output⁡(yi)}∣/GR_{\mathrm{consistency}}(y_i) = |\{y_j : \operatorname{output}(y_j)=\operatorname{output}(y_i)\}|/G adds intra-group agreement when property tests are insufficient.

Training Procedure

Training proceeds in three phases corresponding to curriculum levels (full pseudocode: Algorithm 4, Appendix). Phase 1 trains πθ\pi_\theta on Level-0 tasks with token-level GRPO and the layered execution reward; we then extract O0\mathcal{O}_0 via Algorithm 2 (Appendix) and partially freeze. Phase 2 trains on Level-1 tasks with option-level GRPO, the combined gradient, and adaptive curriculum advancement (ρadv=0.5\rho_{\mathrm{adv}} = 0.5); O1\mathcal{O}_1 is extracted at the end. Phase 3 trains on Level-2 program tasks with the full hierarchical action space A2\mathcal{A}_2 and adds the self-consistency reward.

Hyperparameters. See Table 4 in Appendix § 9.§ 3 for the complete configuration; key settings: Qwen3-32B, G=16G=16, AdamW lr 5×10−55\times10^{-5}, βKL=0.04\beta_{\mathrm{KL}}=0.04, steps N0/N1/N2=5,000/10,000/15,000N_0/N_1/N_2 = 5{,}000/10{,}000/15{,}000, 8×8\timesH100 80GB with ZeRO-3 + bf16.

HyperparameterValue
Model
Base modelQwen3-32B
Precisionbf16
GRPO
Group size GG16
Sampling temperature0.8
PPO clip ϵ\epsilon0.2
KL penalty β\beta0.04
Option-level loss weight λ\lambda0.3
Reward
Quality reward weight λq\lambda_q0.2
Consistency reward weight λc\lambda_c0.1
Pattern bonus α1\alpha_10.1
No-redundancy bonus α2\alpha_20.1
Minimal-API bonus α3\alpha_30.1
Reimplementation penalty α4\alpha_40.1
Optimiser
OptimiserAdamW
Policy learning rate5×10−55 \times10^{-5}
Option fine-tune lr5×10−65 \times10^{-6}
Weight decay0.01
Gradient clippingmax norm 1.0
Warmup steps500
Schedulecosine decay
Curriculum
Level 0 steps N0N_05,000
Level 1 steps N1N_110,000
Level 2 steps N2N_215,000
Advancement threshold ρadv\rho_{\mathrm{adv}}0.5
Max composition length Lmax⁡L_{\max}4
Option Discovery
Semantic sim. threshold τ\tau0.7
Clustering threshold τcluster\tau_{\mathrm{cluster}}0.8
Frequency threshold ν\nu5
Success threshold ρ\rho0.7
Execution
Timeout30 s
Memory limit2 GB
Hardware
GPUs8×8 \times H100 80 GB

Table 4. Complete training hyperparameters.

UnseenSDK-Bench

We evaluate on UnseenSDK-Bench, a benchmark we construct for adapting code LLMs to SDKs absent from training data. It pairs 12 SDKs (DataFrames, vector DBs, document processing, computer vision, messaging, security, protocols) with three task levels per SDK (atomic-call, compositional, full-program; 33–624 tasks per level per SDK) and auto-generated property tests from type annotations (four tests per public function on average). The execution sandbox enforces a 30 s timeout, 2 GB memory limit, and network isolation. Beyond pass@kk we report code-length, API-efficiency, redundancy, and idiomaticity (Definition 4), and the option-reuse rate. Full benchmark statistics, per-SDK function counts, property-test counts, and option-discovery results are in Appendix § 10.

Experiments

Experimental Setup

Base Model. Qwen3-32B (Qwen Team [18]), a dense transformer with 32.8B parameters, 64 layers, hidden size 5,120, grouped-query attention with 64 query heads and 8 key/value heads, and a 32,768-token native context window extensible to 128K via YaRN. We deliberately use the general-purpose Qwen3 dense series rather than the MoE Qwen3-Coder variants to show that Option-GRPO does not require a code-specialized backbone; curriculum-staged GRPO with our SDK-derived option vocabulary is sufficient to specialize a general dense LLM for unseen-SDK code generation. The same base model is used for all RL baselines in the comparison.

Contamination check. The Qwen3-32B pre-training corpus has an effective cutoff around February 2025 (Qwen Team [18]). All 12 SDKs in UnseenSDK-Bench were technically published before this date, so the “unseen” framing rests not on post-cutoff novelty but on the magnitude of the model’s functional knowledge of these niche libraries. We verify this with a memorisation probe (Appendix § 10.§ 2): we prompt the frozen base model with 200 SDK-specific identifier-completion queries per SDK (e.g., “In Daft, the method that filters rows is called ___”) and measure top-1 recall. Our 12 SDKs average 10.5% recall (±5.0% across SDKs) against a control of 91.5% for NumPy and 88.0% for PyTorch. Recall stays below 20% for every SDK, far below the controls. The base model has neither the API surface nor the idiomatic compositional priors of these libraries, justifying the “unseen” framing throughout. Unseen-ness is model-specific: we call an SDK functionally unseen for a model when that model’s recall under this probe is below the 25% threshold of Wang et al. [24], and we apply the same probe to the frontier models we compare against (Table 7).

SDKReleaseCutoffQwen3GPT 5.5Opus 4.8
Daft2022-08Pre19.5%48.5%*44.0%*
LanceDB2023-03Pre16.0%59.0%*52.0%*
cuallee2023-01Pre12.5%44.0%*41.5%*
Docling2024-07Pre8.0%46.0%*50.5%*
Marker2024-02Pre10.5%52.0%*46.5%*
Surya2024-01Pre10.0%21.5%18.0%
FastStream2023-09Pre17.5%52.0%*47.0%*
Tracecat2024-10Pre6.5%22.5%20.5%
MCP Python2024-11Pre4.0%81.5%*84.0%*
Vortex2024-06Pre7.0%42.0%*52.5%*
libdebug2024-03Pre8.5%46.5%*41.0%*
reflexio2024-10Pre5.5%18.0%34.5%*
UnseenSDK avg.——10.5%44.5%44.3%
Unseen (<25%< 25\%)——12/123/122/12
NumPy (ctrl)2006-01Pre91.5%——
PyTorch (ctrl)2016-09Pre88.0%——

Table 7. Contamination analysis. Release dates and the Cutoff column are relative to the Qwen3-32B training cutoff (approximately Feb 2025). Recall is top-1 identifier-completion accuracy on 200 probes per SDK at temperature 0, for the frozen Qwen3-32B and for the two current frontier models. *Recall at or above the 25% threshold: the SDK is functionally seen by that model, so a comparison against that model on that SDK is a seen-SDK comparison rather than a like-for-like win. Controls are probed on the base model only.

Baselines. We compare against eight baselines, all starting from the same 32B base: Frozen LLM (no adaptation); RAG (top-10 CodeBERT retrieval over SDK source); DocPrompting (Zhou et al. [30]) (documentation passages); Gorilla (Patil et al. [16]) (retriever-aware fine-tuning on API-call corpora, adapted to per-SDK retrieval); ExploraCoder (Wang et al. [24]) (training-free chained exploration); ReCode (Wu et al. [27]) (rule-based GRPO for API knowledge updates); Flat GRPO (token-level GRPO with our layered execution reward only, no hierarchy); Flat GRPO + Curriculum (progressive difficulty and the same reward schedule as Option-GRPO, no option structure). We additionally reimplement CodeRL (Le et al. [10]), PPOCoder (Shojaee et al. [20]), and StepCoder (Dou et al. [5]), substituting our layered execution reward for their unit-test requirement (results in Appendix § 11.§ 6). Frontier LLMs (GPT-4o, Claude 3.5 Sonnet, Gemini 2.0 Flash) are evaluated via official APIs with identical prompts, temperature (0.8), and samples (n=10n=10); no per-model prompt tuning. GPT 5.5 and Claude Opus 4.8 are evaluated under the same protocol on the SDKs that are functionally unseen to both.

Metrics.

  • pass@kk (k=1,10k = 1, 10): Functional correctness, generating n=10n = 10 samples per problem with temperature 0.8.

  • Code length (CL): Average token count of passing solutions (lower is better).

  • API efficiency (AE): Average number of SDK API calls in passing solutions (lower is better).

  • Idiomaticity (ID): Fraction of code tokens covered by discovered option patterns (higher is better).

  • Convergence: Training steps to reach specified pass@1 thresholds.

Implementation. Training on 8×8\times NVIDIA H100 80 GB GPUs with ZeRO-3 sharding and bf16. Full hyperparameters in Appendix § 9.§ 3.

Main Results

Table 2 presents the main comparison.

Methodpass@1pass@10CL ↓\downarrowAE ↓\downarrowID ↑\uparrow
Frozen LLM (32B)28.542.194.67.80.18
RAG41.356.888.16.40.31
DocPrompting44.860.285.45.90.34
Gorilla47.262.782.95.50.36
ExploraCoder50.267.578.44.90.43
ReCode55.171.274.14.60.51
Flat GRPO52.768.471.34.90.41
Flat GRPO + Curric.55.471.066.84.40.49
Option-GRPO61.476.852.03.20.74
Δ\Delta (vs ReCode)+6.3+5.6−29.8%−30.4%+0.23
Δ\Delta (vs Flat GRPO)+8.7+8.4−27.1%−34.7%+0.33

Table 2. Main results on UnseenSDK-Bench. Option-GRPO achieves the highest functional correctness while producing more concise and idiomatic code. ↓\downarrow indicates lower is better; ↑\uparrow indicates higher is better.

Functional correctness and code quality. Option-GRPO achieves 61.4% pass@1, beating ReCode by +6.3, flat GRPO by +8.7, and ExploraCoder by +11.2 points. More notably, generated programs are 27% shorter and use 34.7% fewer API calls than flat-GRPO outputs, while reaching higher correctness. Idiomaticity (fraction of tokens covered by discovered option patterns) rises from 0.41 for flat GRPO to 0.74 for Option-GRPO. These results are consistent with the conciseness bound of Theorem 2, the API-efficiency bound of Corollary 1, and the MDL-prior argument of Proposition 2, and directly attack the AI-slop phenomenon ([6]).

Independent evaluation. The benchmark tasks, property tests, and rewards all come from our own pipeline, so we re-check the headline comparison on tasks and a scorer that the pipeline did not produce (Table 3). Training uses no human-written usage examples, so each SDK’s official tutorial and documentation examples are held out by construction. We curate 132 tasks from the official tutorials and documentation of all 12 SDKs, paraphrasing the descriptions to avoid verbatim overlap. Each tutorial task uses the tutorial’s own code as its reference solution and each synthetic task uses its generating graph path; both task sets are scored by output equivalence against reference execution on fixed inputs. The margin over Flat GRPO persists in every cell, and on human-written tutorial tasks under reference-output equivalence it is +8.2 points (95% CI +3.1 to +11.6, paired bootstrap). Corner to corner the two methods drop by nearly the same amount (9.1 vs. 9.6 points), and the frozen model, never trained against our pipeline, drops comparably (28.5 to 21.4), so the lower absolute scores reflect scorer strictness and task shift rather than adaptation to our pipeline. The code-length advantage also holds on tutorial tasks (52.8 vs. 66.7 tokens on passing solutions).

TasksScorerFlatOptΔ\Delta
SyntheticProperty tests52.761.4+8.7
SyntheticReference output44.754.2+9.5
TutorialProperty tests50.857.3+6.5
TutorialReference output43.651.8+8.2†^{\dagger}

Table 3. Independent evaluation (pass@1, %). Tasks: the paper’s synthetic tasks or 132 human-written tutorial tasks. Scorer: the paper’s property tests or reference-output equivalence. †^{\dagger}95% CI +3.1 to +11.6 (paired bootstrap).

What pass@1 certifies. Regrading all stored passing solutions (5,042 Option-GRPO, 4,467 Flat GRPO), agreement between passing the property tests and matching the reference output is 91.2% overall (95.4% at Level 0, 88.5% at Level 1, 76.6% at Level 2). This audit set spans Levels 0–2 and is not directly comparable to Table 3, which covers all final benchmark generations. As a reward-gaming check, among generations that execute with the correct output type, the fraction whose output does not match the reference is 18.2% for Flat GRPO and 8.1% for Option-GRPO, so the hierarchy narrows this gap rather than exploiting it.

Prior RL methods and frontier LLMs. Against CodeRL, PPOCoder, and StepCoder (all adapted to use our layered execution reward; full table in Appendix § 11.§ 6), Option-GRPO beats the strongest critic-based method (StepCoder-adapted, 51.3%) by 10.1 pass@1 points and the strongest prior critic-free RL method (ReCode, 55.1%) by 6.3 points. Against frontier LLMs, which receive the RAG context while Option-GRPO uses no retrieval, the comparison is conditioned on per-model unseenness. On the full suite, the 32B Option-GRPO scores above GPT-4o (54.6%), Claude 3.5 Sonnet (57.2%), and Gemini 2.0 Flash (51.8%), the models available when the experiments were run (Table 14). Under the identifier-recall probe, the current GPT 5.5 and Claude Opus 4.8 are functionally seen on 9 and 10 of the 12 SDKs (Table 7), so a full-suite rerun would mostly measure seen-SDK performance. On the two SDKs that are unseen to both, Surya and Tracecat, Option-GRPO leads the stronger current model by +5.7 and +4.7 pass@1 (95% CI +3.6 to +8.7 and +3.9 to +7.3, paired bootstrap over tasks; Table 15). Frontier LLMs are off-the-shelf and Option-GRPO is trained per SDK, so the takeaway is conditional: per-SDK option discovery plus RL on a 32B base outperforms off-the-shelf frontier models, even with retrieval context, on SDKs that are functionally unseen to them, as certified by the probe (Appendix § 11.§ 6).

Table 14: Comparison with frontier LLMs on the full 12-SDK suite. The frontier LLMs receive the same top-10 RAG context (source-code chunks). Option-GRPO is shown without retrieval, which makes the comparison conservative: adding RAG improves Option-GRPO to 63.7 pass@1 and 78.5 pass@10 (Table 17). The other Qwen3-32B rows repeat Table 2. Recall is mean top-1 identifier recall over the 12 SDKs under the probe of Appendix § 10.§ 2 (for ReCode and Option-GRPO, that of the Qwen3-32B base); Unseen is the number of SDKs below the 25% threshold for that model. A full-suite row is a like-for-like comparison only on the SDKs unseen to that model. GPT 5.5 and Claude Opus 4.8 are functionally seen on most of the suite and are compared only on the SDKs unseen to both (Table 15).

MethodRecallUnseenp@1p@10Δ
Frozen Qwen3-32B10.5%12/1228.542.1−32.9
GPT-4o27%8/1254.669.4−6.8
Claude 3.5 Sonnet35%6/1257.271.8−4.2
Gemini 2.0 Flash32%7/1251.867.1−9.6
GPT 5.544.5%3/12Table 15
Claude Opus 4.844.3%2/12Table 15
ReCode (32B)10.5%12/1255.171.2−6.3
Option-GRPO (32B)10.5%12/1261.476.8–

Table 14. Comparison with frontier LLMs on the full 12-SDK suite. The frontier LLMs receive the same top-10 RAG context (source-code chunks). Option-GRPO is shown without retrieval, which makes the comparison conservative: adding RAG improves Option-GRPO to 63.7 pass@1 and 78.5 pass@10 (Table 17). The other Qwen3-32B rows repeat Table 2. Recall is mean top-1 identifier recall over the 12 SDKs under the probe of Appendix § 10.§ 2 (for ReCode and Option-GRPO, that of the Qwen3-32B base); Unseen is the number of SDKs below the 25% threshold for that model. A full-suite row is a like-for-like comparison only on the SDKs unseen to that model. GPT 5.5 and Claude Opus 4.8 are functionally seen on most of the suite and are compared only on the SDKs unseen to both (Table 15).

Configurationp@1p@10
RAG only (32B)41.356.8
DocPrompting only (32B)44.860.2
Option-GRPO (no retrieval)61.476.8
Option-GRPO + RAG63.778.5
Option-GRPO + DocPrompting63.278.1

Table 17. Complementarity with retrieval at inference time.

Table 15: Like-for-like comparison with current frontier models on the two SDKs that are functionally unseen to both (recall below 25%, Table 7). Same protocol as Table 14: the frontier models receive the top-10 source-chunk RAG context and Option-GRPO is evaluated without retrieval; identical prompts, n=10n=10 samples, temperature 0.8. Ref. p@1 is pass@1 under the stricter reference-output-equivalence scorer.

SDKMetricGPT 5.5Opus 4.8Opt-GRPO
Suryapass@155.357.563.2
Suryapass@1071.873.778.4
SuryaRef. p@145.344.756.2
Tracecatpass@138.442.146.8
Tracecatpass@1045.850.463.1
TracecatRef. p@131.830.237.8

Table 15. Like-for-like comparison with current frontier models on the two SDKs that are functionally unseen to both (recall below 25%, Table 7). Same protocol as Table 14: the frontier models receive the top-10 source-chunk RAG context and Option-GRPO is evaluated without retrieval; identical prompts, n=10n=10 samples, temperature 0.8. Ref. p@1 is pass@1 under the stricter reference-output-equivalence scorer.

Ablations and quality analysis. Hierarchy ablation (Table 10, Appendix § 11): flat 52.7%, +curriculum-only +2.7, +options-only +5.4, full 61.4% (24.8K→12.4K steps; CL 71.3→52.0; ID 0.41→0.74). Further ablations (Appendix § 11) cover option level and reward (Table 11), credit assignment, group size, pacing, and per-SDK gains over ReCode (+0.8 to +8.7); same-domain transfer recovers 74.4% vs. 72.9% from scratch and inference retrieval adds +2.3. Plotting ODL⁡H(c)\operatorname{ODL}_{\mathcal{H}}(c) against each quality metric (Figure 6) is consistent with Theorem 2 and Corollary 1: Spearman ρconc=−0.71\rho_{\mathrm{conc}}=-0.71, ρapi=−0.83\rho_{\mathrm{api}}=-0.83, ρred=−0.62\rho_{\mathrm{red}}=-0.62, ρidiom=−0.79\rho_{\mathrm{idiom}}=-0.79 (all p<10−50p<10^{-50}). Side-by-side code examples in App. § 11.

Config.p@1CL ↓ID ↑Conv. (steps)
Flat GRPO52.771.30.4124,800
Curric. only55.466.80.4919,200
Options only58.158.70.6615,600
Full (Option-GRPO)61.452.00.7412,400

Table 10. Hierarchy ablation. Both options and curriculum contribute; their combination yields the largest gains. Rewards differ across rows: Flat GRPO uses the layered execution reward only; Curric. only and Full use the same schedule (execution reward, plus the quality term from Phase 2 and the consistency term in Phase 3); Options only uses the full reward throughout.

Config.p@1
Option level
Level 0 only53.9
Levels 0 + 158.6
Full (Levels 0 + 1 + 2)61.4
Reward
Binary reward only34.2
Layered + quality + consistency (full)61.4

Table 11. Further ablations (pass@1, %, averaged over the 12 SDKs). Top: curriculum levels used in training. Bottom: reward function.

Failure cases: low-composability SDKs. MCP Python (protocol) and Tracecat (security) have the lowest pass@1 (43.2% and 46.8%; Table 8), the lowest composability density (0.011 and 0.018; Table 6), and the lowest option reuse (0.31 and 0.39). MCP Python is the clearest failure: it retains the fewest Level-1 options (24) and its gain over Flat GRPO shrinks to +0.7 points, against a mean of +8.7. Tracecat keeps only 38 of its 402 Level-1 candidates, the lowest retention in the suite; its gain over Flat GRPO holds (+8.6), but its absolute pass@1 stays second-lowest. When few call chains are type-composable, the hierarchy has little to reuse.

SDKL0L1 Cand.L1 Filt.Avg LenDensity
Daft156284732.70.041
LanceDB142241612.50.038
cuallee87159542.40.062
Docling198312482.90.029
Marker112187392.60.034
Surya94148352.40.036
FastStream234356522.80.024
Tracecat312402382.50.018
MCP Python178174242.30.011
Vortex145223472.60.030
libdebug89128292.40.028
reflexio124185332.50.026

Table 6. Option discovery results per SDK. Composability density is ∣Ecompose∣/∣F∣2|E_{\mathrm{compose}}|/|F|^2.

SDKCategoryp@1p@10CL ↓\downarrowAE ↓\downarrowID ↑\uparrowOR ↑\uparrow
cualleeData Valid.78.291.039.42.30.890.80
LanceDBVector DB74.687.442.72.70.830.70
MarkerDoc Proc.74.186.844.53.00.820.66
SuryaComp. Vision63.278.451.93.60.800.62
DaftDataFrame72.985.045.82.70.830.72
VortexScientific60.677.049.63.10.700.63
libdebugDebugging60.276.354.63.10.760.53
FastStreamMessaging57.174.155.43.50.710.51
DoclingDoc Proc.54.571.361.03.40.670.43
reflexioAgent Frmwk51.469.957.63.70.710.42
TracecatSecurity46.863.158.93.80.580.39
MCP PythonProtocol43.261.362.63.50.580.31
Average–61.476.852.03.20.740.56

Table 8. Full per-SDK results for Option-GRPO: pass@1, pass@10, code length CL, API efficiency AE (lower is better), idiomaticity ID (higher is better), option reuse rate OR (higher is better).

Computational cost and sensitivity. Training is 636 H100 GPU-hours per SDK (∼80\sim80 hours wall-clock on one 8×\timesH100 80GB node); sandbox execution averages 1.8 s per sample (App. § 13). Figure 2 and App. § 13.§ 3 sweep budget and dense base size: Option-GRPO at 1/3 compute (∼224\sim224 GPU-hours) beats Flat GRPO at full compute by +1.4

Cost–performance curves averaged over the 12 SDKs

Figure 2. Cost–performance curves averaged over the 12 SDKs (Table 19 and Table 20). Left: pass@1 against aggregate H100 GPU-hours per SDK on the 32B base. Right: pass@1 against dense base size at the default budget. Dashed: Flat GRPO at the default budget on the 32B base (52.7%).

pass@1; the option vocabulary, not scale, drives the gain.

Conclusion

Option-GRPO formalizes curriculum stages as option construction in hierarchical RL: each stage’s reusable temporally-extended actions become the next stage’s action space. Under stated assumptions, code options induce a valid SMDP (Prop. 1), option description length conditionally bounds code length and expected API use (Thm. 2, Cor. 1), and option-level credit assignment improves the gradient signal-to-noise ratio by Θ(T/K)\Theta(\sqrt{T/K}) over flat GRPO (Prop. 3). On 12 unseen SDKs with a 32B base, Option-GRPO reaches 61.4% pass@1 (vs. 55.1% ReCode, 52.7% flat) with 27% shorter and 34.7% more API-efficient code, directly attacking AI slop.

Ethics Statement

Option-GRPO lowers the barrier to adopting unfamiliar SDKs and directly targets AI slop by training ing models to produce shorter, more idiomatic code. Outputs should still be reviewed for correctness and security, as the reward signal does not guarantee full semantic correctness. Reproducibility. Code, benchmark data, and evaluation harness: https://github.com/prashankkadam/option-grpo-code (App. E). Use of AI assistants. AI assistants (Claude Code) helped with prose editing, LaTeX formatting, and code scaffolding/refactoring. Experimental design, theory, hyperparameters, results, and final code review are the authors’.

Limitations

Per-SDK training cost (636 H100 GPU-hours per SDK, ∼80\sim80 hours wall-clock on an 8-GPU node; same-domain transfer mitigates, Table 12). Reward confound: Flat GRPO is trained with the execution reward only, whereas Option-GRPO also receives the quality and consistency terms. With the reward schedule held fixed (Flat GRPO + Curriculum, Table 2), options account for +6.0 pass@1, 22% shorter code, and 27% fewer API calls; the rest of the gap to Flat GRPO also reflects the curriculum and the quality reward, which we do not ablate separately. Reward signal cannot verify full semantic correctness; programs that execute and type-check but produce wrong results receive partial credit, mitigated but not eliminated by self-consistency. Automatically generated tests: the property tests are synthesized from type annotations, not written by humans; passing them agrees with matching a reference output for 91.2% of passing solutions overall and 76.6% at Level 2 (§ 5). Option rigidity: type-based initiation/termination misses cross-cutting patterns; the two lowest-density SDKs (MCP Python, Tracecat) have the lowest pass@1, and MCP Python the smallest gain. Typing assumption: option discovery uses SDK type annotations; all 12 SDKs are statically annotated Python. Untyped or dynamically typed code would need an upstream type-inference pass (future work). Theoretical scope: Theorem 2/Corollary 1 bound conciseness and API efficiency only under the stated structural and dispersion conditions; curriculum monotonicity concerns optimal policies, not the trained one; the measured gradient-variance reduction is smaller than the T/K\sqrt{T/K} scaling of Proposition 3 (Figure 5); QredQ_{\mathrm{red}}, QidiomQ_{\mathrm{idiom}} are empirical predictions only.

Source → TargetDaftLanceDBcuallee
No transfer (from scratch)72.974.678.2
Daft—76.278.9
LanceDB74.4—79.5
cuallee73.675.6—

Table 12. Cross-SDK option transfer (pass@1 %). Rows: source SDK; columns: target SDK.

Extended Proofs

Formal Statements

Proposition 1 (Code Options Induce a Valid SMDP). By Theorem 1 of Sutton et al. [21], the decision process that selects only among options in O=O0∪O1\mathcal{O} = \mathcal{O}_0 \cup\mathcal{O}_1 and executes each to termination is an SMDP, with the standard option-value

Bellman optimality equation. Proof and the four SMDP ingredients are verified in Section A.2.

Remark 2 (Candidates vs. certified options). The distinction between OkcandO_k^{\mathrm{cand}} and OkO_k is what makes the curriculum-as-options claim non-trivial. A candidate is a graph-mined hypothesis: “this length-ℓ\ell path in GflowG_{\mathrm{flow}} might be useful.” A certified option is an experimentally validated fact: “the trained policy at stage kk can actually invoke this path on ≥ρ\ge\rho fraction of relevant tasks and the path appears ≥ν\ge\nu times across distinct training tasks.” Algorithm 1 can over-propose (generating composable paths that the policy turns out not to use) or under-propose (missing a useful path the model could have learned); the curriculum’s certification step is what produces the option set the next stage actually relies on. Empirically (Table 6) Level-1 candidates outnumber certified Level-1 options by roughly 5×5\times on average across the 12 SDKs.

Standard GRPO

Given prompt xx, sample GG outputs {y1,…,yG}\{y_1,\ldots,y_G\} from πθ\pi_\theta. Compute rewards {R1,…,RG}\{R_1,\ldots,R_G\}. The advantage is estimated via group normalization:

A^i=Ri−mean⁡({Rj}j=1G)std⁡({Rj}j=1G)+ϵ(2)\widehat{A}_i = \frac{R_i-\operatorname{mean}(\{R_j\}_{j=1}^{G})}{\operatorname{std}(\{R_j\}_{j=1}^{G})+\epsilon} \tag*{(2)}

The policy gradient is:

∇J(θ)≈1G∑i=1GA^i⋅∇log⁡πθ(yi∣x)(3)\nabla J(\theta) \approx\frac{1}{G}\sum_{i=1}^{G}\widehat{A}_i\cdot\nabla\log\pi_\theta(y_i\mid x) \tag*{(3)}

with PPO-style clipping and a KL penalty against a reference policy πref\pi_{\mathrm{ref}}.

Option-Level GRPO

At curriculum level k≥1k\ge1, the policy generates option sequences rather than token sequences. For prompt xx, we sample GG option sequences, expand each to tokens via the corresponding intra-option policy, execute the resulting program to obtain reward RiR_i, and compute both a sequence-level advantage A^iseq\widehat{A}^{\mathrm{seq}}_i (standard GRPO) and an option-level advantage A^opt(o)=1∣So+∣∑i∈So+Ri−1∣So−∣∑i∈So−Ri\widehat{A}^{\mathrm{opt}}(o)=\frac{1}{|S_o^+|}\sum_{i\in S_o^+}R_i-\frac{1}{|S_o^-|}\sum_{i\in S_o^-}R_i that compares groups containing vs. excluding each option. The combined gradient ∇J=1G∑i[A^iseq∇log⁡μθ(seqi∣x)+λ∑jA^opt(oj(i))∇log⁡μθ(oj(i)∣ctxj(i))]\nabla J=\frac{1}{G}\sum_i[\widehat{A}^{\mathrm{seq}}_i\nabla\log\mu_\theta(\mathrm{seq}_i\mid x)+\lambda\sum_j\widehat{A}^{\mathrm{opt}}(o_j^{(i)})\nabla\log\mu_\theta(o_j^{(i)}\mid\mathrm{ctx}_j^{(i)})] balances global ranking against per-option credit. PPO-style clipping and a KL penalty against πref\pi_{\mathrm{ref}} apply as in standard GRPO.

Credit Assignment Analysis

The option-level action space tightens GRPO’s credit assignment. Partitioning a generated program into KK option-decision tokens and T−KT-K glue tokens (Definition 5), we show in Proposition 3 that the aggregate signal-to-noise ratio improves by Θ(T/K)\Theta(\sqrt{T/K}), since only option-decision positions are reward-correlated under the sequence-level advantage while flat GRPO must spread the same signal across all TT positions. Proposition 4 additionally bounds the expected number of GRPO iterations to a first successful trajectory under a base-LLM prior, yielding an additive cost across curriculum levels (Option-GRPO) versus a multiplicative cost (flat GRPO). Full statements and proofs follow below.

Theorem 1 (Curriculum Monotonicity, Weak and Strict). Let Vk∗V_k^* denote the expected value of the optimal policy at curriculum level kk under option hierarchy H\mathcal{H} on task distribution D\mathcal{D}. Then:

  1. (Weak monotonicity.) V0∗≤V1∗≤V2∗V_0^*\le V_1^*\le V_2^*.

  1. (Strict monotonicity under non-degeneracy.) Suppose there exist a task x∈Dx\in\mathcal{D} and a state sts^t reachable under πk−1∗\pi_{k-1}^* with positive probability pt:=Pr⁡(st∣πk−1∗,D)>0p^t:=\Pr(s^t\mid\pi_{k-1}^*,\mathcal{D})>0, at which some Level-kk option ot∈Ok∖Ak−1o^t\in O_k\setminus A_{k-1} is applicable (st∈Iots^t\in I_{o^t}) and satisfies Qk−1πk−1∗(st,ot)−Vk−1πk−1∗(st)≥ΔQ_{k-1}^{\pi_{k-1}^*}(s^t,o^t)-V_{k-1}^{\pi_{k-1}^*}(s^t)\ge\Delta for some Δ>0\Delta>0, where Qk−1πk−1∗(st,ot)Q_{k-1}^{\pi_{k-1}^*}(s^t,o^t) is the option value of executing oto^t once at sts^t and following πk−1∗\pi_{k-1}^* thereafter. Then Vk∗≥Vk−1∗+γ∣st∣Δpt>Vk−1∗V_k^*\ge V_{k-1}^*+\gamma^{|s^t|}\Delta p^t>V_{k-1}^*, where γ\gamma is the discount factor.

Remark 3 (On Non-Degeneracy). The non-degeneracy condition asks that the certified set OkO_k contains at least one option that strictly improves over Level-(k−1)(k-1) actions on at least one reachable state under some task in D\mathcal{D}. The candidate set OkcandO_k^{\mathrm{cand}} supplied by Algorithm 1 is designed to make this condition easy to satisfy: Level-1 candidates are centroids of high-weight composable paths in the parameter-flow graph that single API calls cannot realize in one step. When option certification (Algorithm 2) retains at least one such candidate, non-degeneracy holds and the curriculum yields a strict improvement; when every candidate fails to be certified (e.g., on SDKs with no compositional structure that the policy can actually invoke), the curriculum is no worse than Level 0 alone in optimal value.

Definition 2 (Observational Equivalence). Let P\mathcal{P} be a finite suite of property tests (Algorithm 3) for SDK SS. Two programs cc and c′c' are P\mathcal{P}-observationally equivalent, written c≡Pc′c \equiv_{\mathcal{P}} c', iff for every test p∈Pp \in\mathcal{P}, both programs execute without error and produce the same observable output (return value or side-effect signature) under pp. We write c≡c′c \equiv c' when P\mathcal{P} is clear from context.

Definition 3 (Option Description Length). The option description length of a program cc under option hierarchy H\mathcal{H} and test suite P\mathcal{P} is:

ODL⁡H,P(c)=min⁡m≥0, o1,…,omoi∈O0∪O1m\operatorname{ODL}_{\mathcal{H},\mathcal{P}}(c) = \min_{\substack{m \geq0,\ o_1,\ldots,o_m\\ o_i \in\mathcal{O}_0 \cup\mathcal{O}_1}} m
s.t. execute⁡(o1,…,om)≡Pc(4)\text{s.t. }\operatorname{execute}(o_1,\ldots,o_m) \equiv_{\mathcal{P}} c \tag*{(4)}

where execute⁡(o1,…,om)\operatorname{execute}(o_1,\ldots,o_m) denotes the program obtained by sequentially expanding each option through its intra-option policy. If no such sequence exists, ODL⁡H,P(c)=∞\operatorname{ODL}_{\mathcal{H},\mathcal{P}}(c) = \infty. Replacing the undecidable full-equivalence relation with ≡P\equiv_{\mathcal{P}} makes ODL computable: at most ∣O∣mmax⁡|\mathcal{O}|^{m_{\max}} candidate sequences need be enumerated and checked against P\mathcal{P}, where mmax⁡m_{\max} is a search bound. We use mmax⁡=8m_{\max} = 8 in practice. When P\mathcal{P} is fixed throughout an experiment we write ODL⁡H(c)\operatorname{ODL}_{\mathcal{H}}(c) for brevity.

Definition 4 (Code Quality Metrics). For a program cc that passes all execution tests:

  • Conciseness: Qconc(c)=Tref/∣c∣Q_{\mathrm{conc}}(c) = T_{\mathrm{ref}} / |c| where ∣c∣|c| is token count and TrefT_{\mathrm{ref}} is a reference length.

  • API efficiency: Qapi(c)=Nmin⁡/Napi(c)Q_{\mathrm{api}}(c) = N_{\min} / N_{\mathrm{api}}(c) where NapiN_{\mathrm{api}} counts SDK API calls.

  • Redundancy: Qred(c)=1−Nredundant(c)/Nvars(c)Q_{\mathrm{red}}(c) = 1 - N_{\mathrm{redundant}}(c) / N_{\mathrm{vars}}(c).

  • Idiomaticity: Qidiom(c)=∣{tokens covered by options}∣/∣c∣Q_{\mathrm{idiom}}(c) = |\{\text{tokens covered by options}\}| / |c|.

Remark 4 (Structural Conditions). The following structural conditions on the option hierarchy H\mathcal{H} are used in the analysis below. Conditions (a) and (b) are enforced by Algorithm 1; condition (c) is a property of the generation procedure that we observe empirically.

(a) Per-option API minimality: Each option o∈O0∪O1o \in\mathcal{O}_0 \cup\mathcal{O}_1 uses the minimum number of SDK calls required to realize its declared input–output type signature. Enforced because Level 0 options are single calls and Level 1 options are centroids of path clusters in GflowG_{\mathrm{flow}} (Algorithm 1, lines 9–11).

(b) Option non-subsumption: For options o,o′∈Oko,o' \in\mathcal{O}_k at the same level, neither subsumes the other (i.e., no option’s API call set is a strict subset of another’s). Enforced by merging subsumed candidates during discovery.

(c) Bounded glue code: Each transition between consecutive options in an executed sequence introduces at most Gmax⁡G_{\max} tokens of glue code (variable bindings, type coercions). Empirically Gmax⁡≈5G_{\max} \approx5 in our trained policies.

Lemma 1 (Token Bound on Executed Programs). Let τˉmax⁡\bar{\tau}_{\max} and τˉmin⁡\bar{\tau}_{\min} denote, respectively, the maximum and minimum per-option token length across O0∪O1\mathcal{O}_0 \cup\mathcal{O}_1. Under Remark 4, any program cc obtained from an option sequence of length m≥1m \geq1 satisfies

mτˉmin⁡≤∣c∣≤mτˉmax⁡+(m−1)Gmax⁡,(5)m\bar{\tau}_{\min} \leq|c| \leq m\bar{\tau}_{\max} + (m-1)G_{\max}, \tag*{(5)}

where ∣c∣|c| denotes the token count of cc.

Theorem 2 (Option Compression Implies Conciseness). Let c1,c2c_1,c_2 be observationally equivalent programs (c1≡Pc2c_1 \equiv_{\mathcal{P}} c_2, Definition 2) with ODL⁡H(c1)=m1\operatorname{ODL}_{\mathcal{H}}(c_1) = m_1 and ODL⁡H(c2)=m2\operatorname{ODL}_{\mathcal{H}}(c_2) = m_2. Suppose Remark 4 holds and let α:=(τˉmax⁡+Gmax⁡)/τˉmin⁡≥1\alpha:= (\bar{\tau}_{\max} + G_{\max})/\bar{\tau}_{\min} \geq1 be the option-length dispersion ratio. If m1<m2m_1 < m_2 and m1/m2<1/αm_1/m_2 < 1/\alpha, then

∣c1∣≤m1(τˉmax⁡+Gmax⁡)≤αm1m2(m2τˉmin⁡)≤αm1m2∣c2∣<∣c2∣.(6)|c_1| \leq m_1(\bar{\tau}_{\max} + G_{\max}) \leq\frac{\alpha m_1}{m_2}(m_2\bar{\tau}_{\min}) \leq\frac{\alpha m_1}{m_2}|c_2| < |c_2|. \tag*{(6)}

and hence Qconc(c1)>Qconc(c2)Q_{\mathrm{conc}}(c_1) > Q_{\mathrm{conc}}(c_2). The empirical Spearman correlation ρ(ODL⁡,∣c∣)>0\rho(\operatorname{ODL}, |c|) > 0 reported in Appendix § 11.7 is consistent with this direction in practice.

Corollary 1 (API Efficiency in Expectation). Let kˉ\bar{k} denote the average number of SDK calls per option in O0∪O1\mathcal{O}_0 \cup\mathcal{O}_1. Under Remark 4(a), the expected number of API calls in a program realized by an mm-option decomposition (sampled uniformly from minimal decompositions) is E[Napi]=mkˉ\mathbb{E}[N_{\mathrm{api}}] = m\bar{k}. Consequently, if ODL⁡H(c1)<ODL⁡H(c2)\operatorname{ODL}_{\mathcal{H}}(c_1) < \operatorname{ODL}_{\mathcal{H}}(c_2), then

E[Napi(c1)]<E[Napi(c2)],(7)\mathbb{E}[N_{\mathrm{api}}(c_1)] < \mathbb{E}[N_{\mathrm{api}}(c_2)], \tag*{(7)}

and hence E[Qapi(c1)]>E[Qapi(c2)]\mathbb{E}[Q_{\mathrm{api}}(c_1)] > \mathbb{E}[Q_{\mathrm{api}}(c_2)].

Remark 5 (On Redundancy and Idiomaticity). The metrics QredQ_{\mathrm{red}} and QidiomQ_{\mathrm{idiom}} (Definition 4) do not admit clean closed-form bounds under Remark 4 alone, because both depend on choices the intra-option policy makes about intermediate variables and idiom-aligned token spans. We therefore treat them as empirical predictions of the option-compression principle: programs with shorter ODL should exhibit fewer redundant intermediates and a higher fraction of option-covered tokens. Appendix § 11.7 reports Spearman rank correlations between ODL and each of QredQ_{\mathrm{red}}, QidiomQ_{\mathrm{idiom}} across all passing solutions.

Proposition 2 (Minimum-Description-Length Prior over Option Sequences). Let μunif(o∣s)=1/∣Ak(s)∣\mu_{\mathrm{unif}}(o \mid s) = 1/\lvert\mathcal{A}_k(s)\rvert be the uniform prior over options valid at state ss at curriculum level kk, and assume ∣Ak(s)∣≥c>1\lvert\mathcal{A}_k(s)\rvert\ge c > 1 at every reachable state. For two observationally equivalent programs c1,c2c_1, c_2 realized by option sequences of length m1<m2m_1 < m_2,

log⁡μunif(c1)−log⁡μunif(c2)=∑j=1m2log⁡∣Ak(sj(2))∣−∑j=1m1log⁡∣Ak(sj(1))∣(8)\log\mu_{\mathrm{unif}}(c_1) - \log\mu_{\mathrm{unif}}(c_2) = \sum_{j=1}^{m_2} \log\lvert\mathcal{A}_{k}(s_j^{(2)})\rvert- \sum_{j=1}^{m_1} \log\lvert\mathcal{A}_{k}(s_j^{(1)})\rvert \tag*{(8)}
≥(m2−m1)log⁡c>0.\ge(m_2-m_1)\log c > 0.

Hence, under the uniform-over-valid-options prior induced by the composability graph GoptG_{\mathrm{opt}}, shorter ODL sequences enjoy strictly higher prior probability. The KL penalty in GRPO (§ 3.6) constrains the expected log-ratio Ec∼μθ[log⁡(μθ(c)/πref(c))]≤KLmax⁡\mathbb{E}_{c\sim\mu_\theta}[\log(\mu_\theta(c)/\pi_{\mathrm{ref}}(c))] \le\mathrm{KL}_{\max}, so when πref\pi_{\mathrm{ref}} is close to μunif\mu_{\mathrm{unif}}, the learned μθ\mu_\theta cannot concentrate arbitrary mass on long sequences without paying KL cost.

Remark 6 (On Implicit Quality Optimization). Proposition 2 does not claim that GRPO optimization implicitly favours short option sequences as a consequence of optimization. Such a claim would require equal per-step selection probabilities, which are not guaranteed during training: rare high-level options may have lower μθ(o∣s)\mu_\theta(o \mid s), partially offsetting the length advantage. The statement is structural: short sequences enjoy a stronger MDL prior, and the GRPO KL penalty against πref\pi_{\mathrm{ref}} prevents the learned policy from drifting arbitrarily far from this prior. Table 2 reports the empirical effect on code length.

Definition 5 (Option-Decision and Glue Token Partition). Fix a trained option hierarchy H\mathcal{H} and an option sequence (o1,…,oK)(o_1,\ldots,o_K) producing program cc of TT tokens. Partition the token positions {1,…,T}\{1,\ldots,T\} into two disjoint sets: Topt\mathcal{T}_{\mathrm{opt}}, positions covered by the support of some intra-option policy πk\pi_k (at most Kτˉmax⁡K\bar{\tau}_{\max}), and Tglue\mathcal{T}_{\mathrm{glue}}, positions emitted between option terminations (at most (K−1)Gmax⁡(K-1)G_{\max} by Remark 4(c)). This partition is determined by termination events βk\beta_k, not by manual semantic labeling.

Proposition 3 (Improved Credit Assignment). Let a program consist of KK option selections and TT total tokens with K≪TK \ll T (Definition 5). Assume that (i) reward-correlated gradient signal is carried only by the option-decision positions, each with per-position variance σopt2\sigma_{\mathrm{opt}}^2; (ii) glue positions are well predicted by πθ\pi_\theta and contribute per-position variance σglue2\sigma_{\mathrm{glue}}^2 that does not align with the reward signal; and (iii) token positions are conditionally independent. Then the aggregate signal-to-noise ratios of the policy gradient estimate satisfy

SNRflat=Θ(K/Kσopt2+(T−K)σglue2)\mathrm{SNR}_{\mathrm{flat}} = \Theta\left(K/\sqrt{K\sigma_{\mathrm{opt}}^2+(T-K)\sigma_{\mathrm{glue}}^2}\right)

and

SNRopt=Θ(K/σopt),\mathrm{SNR}_{\mathrm{opt}} = \Theta\left(\sqrt{K}/\sigma_{\mathrm{opt}}\right),

with relative improvement SNRopt/SNRflat=Θ(T/K)\mathrm{SNR}_{\mathrm{opt}}/\mathrm{SNR}_{\mathrm{flat}} = \Theta(\sqrt{T/K}). Detailed proof in Section A (Extended Proof of Proposition 3).

Proposition 4 (Convergence Improvement under a Base-LLM Prior). Let πbase\pi_{\mathrm{base}} be the base LLM and let pˉ\bar{p} and qˉk\bar{q}_k denote, respectively, its geometric-mean probability on SDK-specific tokens and on correct options at curriculum level kk. Under the assumption qˉk≫pˉμˉk\bar{q}_k \gg\bar{p}^{\bar{\mu}_k}, the expected number of GRPO iterations to find a first successful trajectory is Θ(1/(GqˉkK))\Theta(1/(G\bar{q}_k^K)) for flat GRPO and Θ(∑k=021/(GqˉkK))\Theta(\sum_{k=0}^{2}1/(G\bar{q}_k^K)) for Option-GRPO; the latter is strictly smaller whenever ∑kqˉk−K<pˉ−K\sum_k\bar{q}_k^{-K}<\bar{p}^{-K}. Proof in Section A.

Proposition 5 (Search Space Reduction). Let ∣V∣\lvert\mathcal{V}\rvert be the vocabulary size, TT the average program length in tokens, ∣O∣\lvert\mathcal{O}\rvert the total number of options, μˉ\bar{\mu} the average option length, and Gmax⁡G_{\max} the per-boundary glue bound (Remark 4). The number of distinct option-level trajectories of total token length TT is at most

Flat (tokens):∣V∣T,(9)\mathrm{Flat\ (tokens):}\quad\lvert\mathcal{V}\rvert^T, \tag*{(9)}
\mathrm{Option\mathchar`-level:}\quad\lvert\mathcal{O}\rvert^{T/\bar{\mu}}\cdot\lvert\mathcal{V}\rvert^{(T/\bar{\mu})G_{\max}}. \tag*{(10)}

For ∣V∣=151,936\lvert\mathcal{V}\rvert= 151{,}936 (Qwen3 vocabulary), T=100T=100, ∣O∣=80\lvert\mathcal{O}\rvert=80, μˉ=15\bar{\mu}=15, Gmax⁡=5G_{\max}=5, the flat search space is ≈10518\approx10^{518} and the option-level search space is ≈806.7⋅151,93633.3≈1012.7⋅10172≈10185\approx80^{6.7}\cdot151{,}936^{33.3}\approx10^{12.7}\cdot10^{172}\approx10^{185}, a reduction of more than 320 orders of magnitude.

Proof of Proposition 1 (SMDP Validity)

Theorem 1 of Sutton et al. [21] holds for any MDP and any set of options defined on it, so it suffices to check that code generation is an MDP and that every code option is an option on it. The code-generation process (S,A,T,R)(\mathcal{S},\mathcal{A},\mathcal{T},R) with A=V\mathcal{A}=\mathcal{V} is an MDP: the state st=(q,c0:t)s_t=(q,c_{0:t}) contains the full generated prefix, the transition is deterministic append, and the reward Rexec+λqRqualityR_{\mathrm{exec}}+\lambda_qR_{\mathrm{quality}} is a bounded function of the completed program, received at episode termination. The self-consistency term of § 3.7 depends on the other samples in the group; it is a training-time signal and is not part of this MDP reward. Each code option ok=(Ik,Fk,βk,πk)o_k=(\mathcal{I}_k,\mathcal{F}_k,\beta_k,\pi_k) of Definition 1 is an option in the sense of Sutton et al. [21]: the initiation set Ik\mathcal{I}_k is a subset of S\mathcal{S} because the type of the last value is determined by the state; the intra-option policy πk\pi_k is πθ\pi_\theta under a logit mask computed from Fk\mathcal{F}_k and the tokens emitted since the option started (Appendix § 9.§ 2); and termination is decided from the same information. Because the mask and the length cap depend on what has happened since the option was started, and not only on the current state, code options are in general semi-Markov options, which Theorem 1 of Sutton et al. [21] covers. Every option terminates within a finite number of tokens, because option execution is closed at a length cap (Appendix § 9.§ 2).

The four ingredients of an SMDP then follow as in Sutton et al. [21]: (1) the state set is S\mathcal{S}; (2) the action set at state ss is {o∈O:s∈Io}\{o\in\mathcal{O}:s\in\mathcal{I}_o\}, which is never empty in the full hierarchy because primitive tokens remain available at every level (V⊂Ak\mathcal{V}\subset\mathcal{A}_k) and are one-step options in the sense of Sutton et al. [21]; (3) the expected cumulative discounted reward of executing oo from ss until termination is well defined, since the reward is bounded and the option terminates in finite time; (4) the joint distribution of the next state and the transit time (the number of tokens emitted before βo\beta_o fires) is determined by πo\pi_o, βo\beta_o, and the deterministic transition. The proposition establishes only that the option-level decision problem is well posed.

Extended Proof of Theorem 1 (Curriculum Monotonicity)

We provide a more rigorous version using the SMDP framework.

Let Mk=(S,Ak,Tk,R)\mathcal{M}_k=(\mathcal{S},\mathcal{A}_k,\mathcal{T}_k,R) denote the SMDP at curriculum level kk, where Ak\mathcal{A}_k includes all options up to level kk. Define Vk∗(s)=max⁡π∈ΠkVπ(s)V_k^*(s)=\max_{\pi\in\Pi_k}V^\pi(s) where Πk\Pi_k is the set of all policies over Ak\mathcal{A}_k.

Since Ak−1⊂Ak\mathcal{A}_{k-1}\subset\mathcal{A}_k, every policy π∈Πk−1\pi\in\Pi_{k-1} is also a valid policy in Πk\Pi_k (by never selecting level-kk options). Therefore Πk−1⊂Πk\Pi_{k-1}\subset\Pi_k.

For any state ss:

Vk∗(s)=max⁡π∈ΠkVπ(s)≥max⁡π∈Πk−1Vπ(s)=Vk−1∗(s).(11)V_k^*(s)=\max_{\pi\in\Pi_k}V^\pi(s) \ge\max_{\pi\in\Pi_{k-1}}V^\pi(s)=V_{k-1}^*(s). \tag*{(11)}

The inequality follows directly from Πk−1⊂Πk\Pi_{k-1}\subset\Pi_k: maximising over a larger set yields at least as large a value. Taking expectations over the initial state distribution gives Vk∗≥Vk−1∗V_k^*\ge V_{k-1}^*.

We note that this result holds for the optimal policies. In practice, the optimisation procedure (GRPO) may not find the optimal policy at each level, so the empirical monotonicity depends on sufficient training at each curriculum stage. The curriculum advancement threshold ρadv\rho_{\mathrm{adv}} (§ 3.8) is intended to ensure that each level is sufficiently learned before advancing; it is a heuristic, not a guarantee.

For the strict version (clause 2 of Theorem 1), let sts^t be the state in the hypothesis and o†o^\dagger the witnessing Level-kk option. Define the one-step deviation policy π~k∈Πk\tilde{\pi}_k\in\Pi_k by π~k(⋅∣s)=πk−1∗(⋅∣s)\tilde{\pi}_k(\cdot\mid s)=\pi_{k-1}^*(\cdot\mid s) for s≠sts\ne s^t and π~k(o†∣st)=1\tilde{\pi}_k(o^\dagger\mid s^t)=1. By construction, Vπ~k(st)≥Vπk−1∗(st)+ΔV^{\tilde{\pi}_k}(s^t)\ge V^{\pi_{k-1}^*}(s^t)+\Delta. Propagating this gain through the discounted-return expectation yields

Vk∗≥Vπ~k≥Vk−1∗+γ∣st∣ΔPr⁡(st∣πk−1∗,D).(12)V_k^* \ge V^{\tilde{\pi}_k} \ge V_{k-1}^*+\gamma^{|s^t|}\Delta\Pr(s^t\mid\pi_{k-1}^*,\mathcal{D}). \tag*{(12)}

Strict inequality follows since each factor on the right is positive by hypothesis.

Extended Proof of Proposition 3 (Credit Assignment SNR)

We make the token partition formal and the variance decomposition explicit.

Consider a program generated as a sequence of TT tokens. In flat GRPO, the gradient for token at position tt is gt=A^⋅∇θlog⁡πθ(at∣st)g_t=\hat{A}\cdot\nabla_\theta\log\pi_\theta(a_t\mid s_t), where A^\hat{A} is the sequence-level advantage applied uniformly across positions.

Using the partition of Definition 5, Topt\mathcal{T}_{\mathrm{opt}} has size at most Kτˉ=O(K)K\bar{\tau}=\mathcal{O}(K) and Tglue\mathcal{T}_{\mathrm{glue}} has size at most (K−1)Gmax⁡=O(K)(K-1)G_{\max}=\mathcal{O}(K); total program length T=∣Topt∣+∣Tglue∣T=|\mathcal{T}_{\mathrm{opt}}|+|\mathcal{T}_{\mathrm{glue}}|. We treat T=Θ(Kτˉ)T=\Theta(K\bar{\tau}) since glue is bounded by a small constant per option boundary. The total gradient decomposes as

∇J=A^⋅(∑t∈Topt∇log⁡π(at∣st)⏟signal+∑t∈Tglue∇log⁡π(at∣st)⏟noise).(13)\nabla J = \hat{A} \cdot\left(\underbrace{\sum_{t\in\mathcal{T}_{\mathrm{opt}}} \nabla\log\pi(a_t \mid s_t)}_{\text{signal}} + \underbrace{\sum_{t\in\mathcal{T}_{\mathrm{glue}}} \nabla\log\pi(a_t \mid s_t)}_{\text{noise}}\right). \tag*{(13)}

Under assumption (ii) of Proposition 3, glue positions are well predicted by πθ\pi_\theta and contribute per-position variance σglue2\sigma_{\mathrm{glue}}^2 that is small in magnitude but does not align with the reward signal. Under assumption (iii) of conditional independence across positions, the aggregate variance is

Var⁡(∇J)=∣Topt∣σopt2+∣Tglue∣σglue2=Θ(T).(14)\operatorname{Var}(\nabla J) = \lvert\mathcal{T}_{\mathrm{opt}}\rvert\sigma_{\mathrm{opt}}^2 + \lvert\mathcal{T}_{\mathrm{glue}}\rvert\sigma_{\mathrm{glue}}^2 = \Theta(T). \tag*{(14)}

The signal magnitude scales with the number of reward-correlated positions, which is at most ∣Topt∣=O(K)\lvert\mathcal{T}_{\mathrm{opt}}\rvert= O(K). Therefore SNRflat=O(K/T)=O(1/T)\mathrm{SNR}_{\mathrm{flat}} = O(K/\sqrt{T}) = O(1/\sqrt{T}) for fixed KK.

In Option-GRPO, the gradient operates over KK option-level decisions:

∇J=∑j=1KA^jopt⋅∇θlog⁡μθ(oj∣ctxj).(15)\nabla J = \sum_{j=1}^{K} \hat{A}^{\mathrm{opt}}_j \cdot\nabla_\theta\log\mu_\theta(o_j \mid\mathrm{ctx}_j). \tag*{(15)}

All KK summands are reward-correlated by construction (each option-level advantage compares groups containing vs. excluding the option). The option-level advantages have variance O(σR2/G)O(\sigma_R^2/G) from the GRPO group estimator. The aggregate signal magnitude is Θ(K)\Theta(K) and the aggregate variance is

Var⁡(∇J)=K⋅σopt2+O(KσR2/G)=Θ(K),(16)\operatorname{Var}(\nabla J) = K \cdot\sigma_{\mathrm{opt}}^2 + O(K\sigma_R^2/G) = \Theta(K), \tag*{(16)}

giving SNRopt=Θ(K/σopt)\mathrm{SNR}_{\mathrm{opt}} = \Theta(\sqrt{K}/\sigma_{\mathrm{opt}}). Comparing to SNRflat=Θ(K/T)\mathrm{SNR}_{\mathrm{flat}} = \Theta(K/\sqrt{T}) from the flat analysis, the improvement ratio is

SNRoptSNRflat=Θ(T/K).(17)\frac{\mathrm{SNR}_{\mathrm{opt}}}{\mathrm{SNR}_{\mathrm{flat}}} = \Theta(\sqrt{T/K}). \tag*{(17)}

For typical values (T=100T = 100, K=5K = 5), this is 20≈4.5×\sqrt{20} \approx4.5\times improvement in SNR.

Extended Proof of Proposition 4 (Convergence Improvement)

In flat token-level generation, a successful trajectory requires TT correct token selections. If the probability of selecting the correct token at position tt is ptp_t, the probability of a complete successful trajectory is:

Psuccessflat=∏t=1Tpt(18)P^{\mathrm{flat}}_{\mathrm{success}} = \prod_{t=1}^{T} p_t \tag*{(18)}

For an unseen SDK the base LLM has limited prior on SDK-specific token decisions (function names, argument patterns), whose probabilities are dominated by the long tail of ∣V∣\lvert V\rvert alternatives, while syntactic tokens (brackets, keywords, indentation) are well-predicted. Suppose KK tokens are SDK-specific with geometric-mean probability pˉ≪1\bar{p} \ll1 and T−KT-K tokens are syntactic with probability near 1. Then:

Psuccessflat≈pˉK(19)P^{\mathrm{flat}}_{\mathrm{success}} \approx\bar{p}^{K} \tag*{(19)}

In Option-GRPO at level kk, a successful trajectory requires K′=T/μˉK' = T/\bar{\mu} correct option selections, where μˉ\bar{\mu} is the average option length. Each option selection has probability qjq_j in a space of ∣Ak∣\lvert\mathcal{A}_k\rvert options:

Psuccessopt≈qˉK′(20)P^{\mathrm{opt}}_{\mathrm{success}} \approx\bar{q}^{K'} \tag*{(20)}

The key insight is the stronger statement qˉ≫pˉμˉ\bar{q} \gg\bar{p}^{\bar{\mu}} (Proposition 4): the option-level reference policy assigns higher probability to a correct option than the base LLM does to emitting the equivalent token span. This holds because (1) the type-filtered option vocabulary Ak\mathcal{A}_k removes long-tail typo and out-of-scope completions that depress pˉ\bar{p}, and (2) the composability graph GoptG_{\mathrm{opt}} further constrains the valid next options at each step.

With GRPO group size GG, the probability of at least one successful trajectory per group is:

P(at least one)=1−(1−Psuccess)G(21)P(\text{at least one}) = 1 - (1-P_{\mathrm{success}})^G \tag*{(21)}

The expected number of groups to find a first success is:

E[groups]=11−(1−Psuccess)G≈1G⋅Psuccess(22)\mathbb{E}[\text{groups}] = \frac{1}{1-(1-P_{\mathrm{success}})^G} \approx\frac{1}{G\cdot P_{\mathrm{success}}} \tag*{(22)}

for small PsuccessP_{\mathrm{success}}.

Under the curriculum, total exploration cost is additive:

Costtotal=∑k=021G⋅Psuccess,k(23)\mathrm{Cost}_{\mathrm{total}} = \sum_{k=0}^{2} \frac{1}{G\cdot P_{\mathrm{success},k}} \tag*{(23)}

Algorithm 1 Level 1 Option Discovery

Require: Parameter flow graph GflowG_{\mathrm{flow}}, semantic similarity graph GsemG_{\mathrm{sem}}, maximum path length Lmax⁡=4L_{\max}=4, similarity threshold τcluster=0.8\tau_{\mathrm{cluster}}=0.8

Ensure: Set of Level 1 options O1O_1

  1. P←∅\mathcal{P}\leftarrow\varnothing

  2. for each node fi∈Gflowf_i\in G_{\mathrm{flow}} do

  3. &nbsp;&nbsp;for ℓ=2\ell=2 to Lmax⁡L_{\max} do

  4. &nbsp;&nbsp;&nbsp;&nbsp;Enumerate all paths of length ℓ\ell starting from fif_i via DFS

  5. &nbsp;&nbsp;&nbsp;&nbsp;Add paths to P\mathcal{P} with weight w(p)=∏(fa,fb)∈ptype_compat⁡(fa,fb)w(p)=\prod_{(f_a,f_b)\in p}\operatorname{type\_compat}(f_a,f_b)

  6. &nbsp;&nbsp;end for

  7. end for

  8. Encode each path p∈Pp\in\mathcal{P}: ep=1∣p∣∑f∈pϕ(f)\mathbf{e}_p=\frac{1}{|p|}\sum_{f\in p}\phi(f)

  9. Cluster paths via agglomerative clustering with threshold τcluster\tau_{\mathrm{cluster}}

  10. for each cluster CjC_j do

  11. &nbsp;&nbsp;Select centroid path pj∗=arg⁡max⁡p∈Cjw(p)p_j^{\ast}=\arg\max_{p\in C_j}w(p)

  12. &nbsp;&nbsp;Define option oj(1)o_j^{(1)} with Ij\mathcal{I}_j, Tjout\mathcal{T}_j^{\mathrm{out}}, and πj\pi_j (learned during Level 1 training)

  13. &nbsp;&nbsp;Add oj(1)o_j^{(1)} to O1O_1

  14. end for

  15. return O1O_1

Algorithm 2 Option Extraction from Trained Policy

Require: Trained policy πk\pi_k, task set Dk\mathcal{D}_k, success threshold ρ=0.7\rho=0.7, frequency threshold ν=5\nu=5

Ensure: Extracted options OkO_k

  1. Generate 10 solutions per task using πk\pi_k with temperature 0.8

  2. Filter to successful solutions (reward ≥ρ\geq\rho)

  3. Parse all successful solutions into API call sequences via AST analysis

  4. Extract all contiguous sub-sequences of length 2–4

  5. Count frequency of each unique sub-sequence pattern

  6. Filter to patterns appearing ≥ν\geq\nu times across different tasks

  7. Cluster remaining patterns by semantic similarity

  8. for each cluster centroid pattern pp do

  9. &nbsp;&nbsp;Define option opo_p with initiation set from input types of first call, termination type from last call, and intra-option policy extracted from πk\pi_k

  10. &nbsp;&nbsp;Add opo_p to OkO_k

  11. end for

  12. return OkO_k

Each Psuccess,kP_{\mathrm{success},k} is much larger than PsuccessflatP_{\mathrm{success}}^{\mathrm{flat}} because: curriculum level kk only needs to learn compositions of length ≤k+1\leq k+1 (not full programs), and options from level k−1k-1 handle sub-problems that are already solved. The additive structure (sum over 3 terms) versus the multiplicative structure of flat exploration (product over TT token decisions) yields the dramatic reduction in sample complexity.

Extended Method Details

Pseudocode for Auxiliary Algorithms

Algorithms 1–4 are referenced from the main paper.

Decode-Time Mechanics of Option Invocation

This subsection describes how an option is started, how the logit mask of Definition 1 is computed at each step, and how termination is detected.

Starting an option. Each option is one added entry in the model’s output vocabulary. Emitting this entry starts the option and adds no text to the program.

At every step outside an option, the policy chooses between an ordinary token and one of these option entries. An option entry is offered only when its initiation set holds: the value just produced must have a type that the option’s first function accepts. Options whose first function takes only plain inputs (such as a file path) are always offered.

Allowed tokens at each step. While an option is active, an incremental tracker reads the code emitted so far character by character. It maintains whether the current position is inside a string or a comment, the bracket depth, the identifier being typed, and which function in the option’s list Fk\mathcal{F}_k comes next. The mask is then computed as follows.

  • On the main call chain, the next function called must be the next one in the list. After df, only tokens that spell toward where are allowed; .select is masked, and so is a partial identifier such as .sel.

  • Inside a function’s brackets, only helper functions that build its arguments are allowed (for example col inside where). Every other SDK function is masked.

Algorithm 3 Property-Based Test Generation

Require: SDK type graph GapiG_{\mathrm{api}}, parameter flow graph GflowG_{\mathrm{flow}}, function ff Ensure: Set of property tests PfP_f

  1. Extract input types {t1,…,tk}\{t_1,\ldots,t_k\} and return type trett_{\mathrm{ret}} from ff

  2. Generate minimal valid inputs using type constructors from GapiG_{\mathrm{api}}

  3. Add type preservation test

  4. if trett_{\mathrm{ret}} has len or shape attribute then

  5. &nbsp;&nbsp;&nbsp;Add shape consistency tests

  6. end if

  7. Add determinism test

  8. Add edge case tests: empty inputs, single element, boundary values

  9. Extract assertions from ff’s source code and add as invariant tests

  10. return PfP_f

  • Everything else stays free: variable names, numbers, strings, and operators.

  • Ending the program and starting another option are masked until the option is finished.

Detecting termination. The option ends at the step where the closing bracket of its last function is emitted (the ) of collect(...)), which is the point at which a value of the option’s output type has been produced. After that the policy returns to free choice. As a safeguard, if an option runs past a length cap or reaches a step where no token is allowed, it is closed and marked as failed, so it never counts as a successful use of the option.

Training Hyperparameters

Curriculum Task Examples

Level 0 Example (Daft, Single API Call):

"""Problem: Read a CSV file into a
Daft DataFrame."""
import daft

# Setup
csv_path = "test_data.csv"

# BEGIN SOLUTION
df = daft.read_csv(csv_path)
# END SOLUTION

Level 1 Example (Daft, Composed Pattern):

"""Problem: Read a CSV file and filter
rows where the 'age' column exceeds 30."""
import daft

# Setup
csv_path = "test_data.csv"

# BEGIN SOLUTION
df = daft.read_csv(csv_path)
result = df.where(
    daft.col("age") > 30
).collect()
# END SOLUTION

Level 2 Example (Daft, Full Program):

"""Problem: Read a CSV, compute mean salary
grouped by department, sort descending."""
import daft

# Setup
csv_path = "employees.csv"

# BEGIN SOLUTION
df = daft.read_csv(csv_path)
result = (
df.groupby("department")
    .agg(daft.col("salary").mean())
    .sort("salary", desc=True)
    .collect()
)
# END SOLUTION

Property Test Examples

Type Preservation (Daft):

df = daft.read_csv("test.csv")
assert isinstance(df, daft.DataFrame)

Shape Consistency (Daft):

df = daft.from_pydict({"a": [1, 2, 3]})
filtered = df.where(
    daft.col("a") > 1
).collect()
assert len(filtered) <= 3

Determinism:

df = daft.from_pydict({"a": [1, 2, 3]})
r1 = df.select("a").collect().to_pydict()
r2 = df.select("a").collect().to_pydict()
assert r1 == r2

Extended Benchmark Details

Per-SDK Option Discovery Statistics

SDKs with higher composability density benefit more from the hierarchical approach: across the 12 SDKs, composability density and the pass@1 improvement of Option-GRPO over Flat GRPO (Table 8 and Table 9) have a Spearman correlation of 0.76.

SDKCategoryp@1p@10CL ↓\downarrowAE ↓\downarrowID ↑\uparrowOR ↑\uparrow
cualleeData Valid.67.283.061.64.20.520.38
LanceDBVector DB63.879.764.74.50.450.31
MarkerDoc Proc.64.881.062.44.50.490.32
SuryaComp. Vision55.870.970.25.00.440.25
DaftDataFrame59.976.366.24.60.490.35
VortexScientific51.767.671.04.80.380.25
libdebugDebugging51.266.873.35.00.410.26
FastStreamMessaging49.365.073.45.00.380.20
DoclingDoc Proc.45.260.980.05.10.350.21
reflexioAgent Frmwk42.858.177.35.30.370.17
TracecatSecurity38.254.678.45.50.300.15
MCP PythonProtocol42.556.977.15.30.340.15
Average–52.768.471.34.90.410.25

Table 9. Full per-SDK results for Flat GRPO (no hierarchy, layered execution reward). Same metrics as Table 8.

Contamination Analysis

To verify that the 12 SDKs in UnseenSDK-Bench are functionally unseen by the Qwen3-32B pre-training corpus, we run two complementary checks. The first is a release-date comparison against the model’s training cutoff (approximately February 2025, per the Qwen3 technical report [18]). The second is a memorisation probe: we query the frozen base model for SDK-specific identifiers and measure top-1 recall, comparing against two well-known control libraries (NumPy, PyTorch) whose presence in pretraining is uncontroversial.

Probe protocol. For each SDK, we construct 200 identifier-completion prompts of the form “In <SDK>, the <noun> that <verb> is called ___”, where the noun and verb describe a specific class, method, or function in that SDK. We sample base-model completions at temperature 0 and score the completion as correct if the top-1 token sequence matches the canonical identifier name (case-insensitive, ignoring leading dots or module prefixes). The 200 prompts per SDK are split evenly across classes, methods, and free functions; identifiers are drawn uniformly from the SDK’s public API. Control SDKs (NumPy, PyTorch) follow the same protocol with 200 identifier prompts each. We run the same probe, unchanged (200 prompts per SDK), on GPT 5.5 and Claude Opus 4.8 and on the three earlier frontier models (GPT-4o, Claude 3.5 Sonnet, Gemini 2.0 Flash) through their APIs; per-SDK recall for the two current models is in Table 7, and the mean recall and unseen-SDK count of every model are in Table 14. Since these models are accessed through chat APIs, we wrap the same cloze prompts in a minimal instruction to reply with the identifier only and decode greedily; scoring is identical.

Interpretation. All 12 SDKs are pre-cutoff under Qwen3’s February 2025 corpus assembly date, so the unseen-SDK claim does not rest on post-cutoff novelty alone. Instead, it rests on the magnitude of recall. Per-SDK recall sits between 4.0% and 19.5%, far below the NumPy (91.5%) and PyTorch (88.0%) controls. The highest-recall SDKs in our suite (Daft at 19.5%, FastStream at 17.5%) are the most established of the twelve, yet remain well below any reasonable threshold for “the model knows this library.” For comparison, prior work on novel-API code generation [24] uses a 25% recall threshold to consider an API “seen”; under that threshold, none of our 12 SDKs qualifies.

The recall scores correlate with frozen-model pass@1 on UnseenSDK-Bench (Spearman ρ=0.70\rho= 0.70, p=0.01p = 0.01): the SDKs whose identifiers the base model partially recalls are also the ones the frozen model handles best. Even so, recall stays below 20% on every SDK and frozen pass@1 averages 28.5%, so partial identifier memorisation does not amount to the compositional, type-correct API usage that the benchmark tasks require. We conclude that all 12 SDKs in UnseenSDK-Bench are functionally unseen by Qwen3-32B, the model we train.

Per-model unseen-ness for current frontier models. Several of the SDKs, most notably MCP Python but also Daft, LanceDB, and Docling, have grown substantially in visibility since early 2025 and are likely represented in current frontier training corpora. Under the 25% threshold, 9 of the 12 SDKs are functionally seen by GPT 5.5 and 10 of 12 by Claude Opus 4.8 (Table 7), so a rerun on the full suite would mostly measure seen-SDK performance rather than the unseen-SDK setting this paper studies. Seen-ness is also model-specific: reflexio is below the threshold for GPT 5.5 but above it for Claude Opus 4.8, which is why recall is reported per model. Two SDKs, Surya and Tracecat, remain below the threshold for both models, and the like-for-like comparison with current frontier models is restricted to these two (Table 15).

SDKCategoryFuncsL0 TasksL1 TasksL2 TasksProp. Tests
DaftDataFrame15615631233624
LanceDBVector DB14214228433568
cualleeData Valid.878717433348
DoclingDoc Proc.19819839633792
MarkerDoc Proc.11211222434448
SuryaComp. Vision949418833376
FastStreamMessaging23423446833936
TracecatSecurity312312624331248
MCP PythonProtocol17817835633712
VortexScientific14514529033580
libdebugDebugging898917833356
reflexioAgent Frmwk12412424834496
Total–1871187137423987484

Table 5. UnseenSDK-Bench statistics per SDK. L0/L1/L2 denote curriculum levels. Property tests are generated automatically from type annotations.

Extended Experimental Results

Full Per-SDK Results

Table 8 reports all six evaluation metrics for Option-GRPO on each of the 12 SDKs. The option reuse rate (OR) is the fraction of SDK API calls in passing solutions that fall inside a certified Level-1 option pattern. Table 9 reports the same for Flat GRPO, the non-hierarchical critic-free RL baseline that uses our layered execution reward.

Aggregate deltas: pass@1 improvement of Option-GRPO over Flat GRPO ranges from +0.7 points on MCP Python to +13.0 points on Daft, with a mean improvement of +8.7 points. CL drops by an average of 27%, AE by 34.7%, ID rises by 0.33, and OR rises by 0.31. The largest improvements are on data-processing SDKs (cuallee, LanceDB, Daft) where compositional structure in the parameter-flow graph is densest; the smallest improvement is on MCP Python, where type-based composability is weakest, and the two lowest-density SDKs (MCP Python, Tracecat) have the lowest absolute pass@1, matching the option-discovery density statistics in Table 6.

Convergence Curves

Figure 3 shows pass@1 against training step for the four configurations of the hierarchy ablation (Table 10), averaged across the 12 SDKs. Option-GRPO reaches the flat-GRPO terminal pass@1 (52.7%) before step 9,000 (under one-third of the 30,000-step budget) and continues climbing to 61.4% by step 30,000. Curriculum-only and options-only configurations lie between the extremes, with options-only slightly ahead at most checkpoints, consistent with the larger marginal contribution of option-level action structure.

Convergence curves averaged across 12 SDKs

Figure 3. Convergence curves averaged across 12 SDKs. Option-GRPO reaches the flat-GRPO terminal pass@1 in roughly one-third the steps and continues to improve.

Option Hierarchy Visualisations

Figure 4 shows the discovered option hierarchy for Daft as a layered graph. Level-0 options (atomic SDK calls) form the leaves; Level-1 options (compositional patterns mined from GflowG_{\mathrm{flow}}) form the internal nodes; Level-2 patterns (full programs assembled during Phase 3) sit at the root. Edges denote type composability (oi≺ojo_i \prec o_j iff Tiout∩Tjin≠∅\mathcal{T}^{\mathrm{out}}_i \cap\mathcal{T}^{\mathrm{in}}_j \ne\varnothing). Three observations: (i) the Level-1 layer is dominated by the canonical load →\rightarrow transform →\rightarrow collect pattern, which appears as the highest-weight cluster in Algorithm 1; (ii) the parameter-flow graph is densely connected within each domain cluster (DataFrame operations chain freely among themselves) but sparse across clusters; (iii) options containing collect act as natural terminators because their output type is a Python list, not a Daft expression.

Figure 4. Discovered option hierarchy for Daft (excerpt). Blue: Level-0 atomic options (one per SDK function). Orange: Level-1 compositional options (centroid paths of length 3 from Algorithm 1). Green: Level-2 program-level composition learned at Phase 3. Edges denote type-composability inclusion. The full Daft hierarchy contains 156 Level-0 options and 73 Level-1 options; we show only the chain underlying a representative Level-2 program.

Analogous hierarchies for LanceDB and cuallee follow the same structural pattern, with vector-search chains (connect→create_table→add→search) dominating the LanceDB Level-1 layer and validation-rule chains (Check→is_complete→is_unique→validate) dominating cuallee’s. Across the three data-processing SDKs, the discovered Level-1 options correspond exactly to the canonical usage idioms in each library’s official tutorial, confirming that the parameter-flow graph captures the intended compositional structure rather than spurious type coincidences.

Code Quality Examples: Slop Reduction Side-by-Side

This subsection presents two representative side-by-side comparisons of Flat GRPO output and Option-GRPO output on the same task. Annotations report token count ∣c∣|c|, number of SDK API calls NapiN_{\mathrm{api}}, number of redundant intermediate variables NredundantN_{\mathrm{redundant}}, and option description length ODLH(c)ODLH(c). The examples span easy (Daft) and medium (LanceDB) difficulty.

Example 1 (easy, Daft). Task: “Read employees.csv and return rows where salary exceeds 75,000.”

Flat GRPO (∣c∣=82|c|=82, Napi=5N_{\mathrm{api}}=5, Nredundant=3N_{\mathrm{redundant}}=3, ODL=∞ODL=\infty since the decomposition uses non-Daft calls):

import daft
import pandas as pd  # unused
path = "employees.csv"
df_raw = daft.read_csv(path)
df = df_raw.collect()
result_list = []
for row in df.iter_rows():
    if row["salary"] > 75000:
        result_list.append(row)
result_df = daft.from_pylist(result_list)
final = result_df.collect()

Option-GRPO (∣c∣=24|c|=24, Napi=4N_{\mathrm{api}}=4, Nredundant=0N_{\mathrm{redundant}}=0, ODL=1ODL=1, one Level-1 option Read→Filter→Collect):

import daft
df = daft.read_csv("employees.csv")
result = (df
    .where(daft.col("salary") > 75000)
    .collect())

The Flat-GRPO solution introduces an unused pandas import, materializes a list intermediate, and reconstructs the DataFrame; the Option-GRPO solution invokes a single discovered option that chains read_csv, where, collect.

Example 2 (medium, LanceDB). Task: “Create a LanceDB table from a list of vectors and return the 5 nearest neighbours to a query vector.”

Flat GRPO (∣c∣=146|c|=146, Napi=7N_{\mathrm{api}}=7: five LanceDB calls plus the pandas and numpy constructors, Nredundant=4N_{\mathrm{redundant}}=4, ODL=∞ODL=\infty):

import lancedb
import numpy as np
import pandas as pd
data = [
    {"vec":[0.1,0.2,0.3],"id":1},
    {"vec":[0.4,0.5,0.6],"id":2}]
db = lancedb.connect("/tmp/db")
df_data = pd.DataFrame(data)
table_obj = db.create_table(
    "vectors", data=df_data)
query_arr = np.array([0.1,0.2,0.3])
search_obj = table_obj.search(
    query_arr.tolist())
results_raw = (search_obj
    .limit(5).to_pandas())
results = results_raw.to_dict("records")

Option-GRPO (∣c∣=48|c|=48, Napi=5N_{\mathrm{api}}=5, Nredundant=0N_{\mathrm{redundant}}=0, ODL=2ODL=2, two Level-1 options Connect→CreateTable and Search→Limit→Materialize):

import lancedb
db = lancedb.connect("/tmp/db")
table = db.create_table("vectors", data=[
    {"vec":[0.1,0.2,0.3],"id":1},
    {"vec":[0.4,0.5,0.6],"id":2}])
results = (table
    .search([0.1,0.2,0.3])
    .limit(5)
    .to_list())

Flat GRPO routes the data through pandas and numpy unnecessarily, introducing four redundant intermediate bindings (df_data, query_arr, search_obj, results_raw); Option-GRPO uses the native LanceDB list and array APIs.

Aggregate slop reduction across the two examples. Token counts drop 71% (82→24) and 67% (146→48) respectively. Non-SDK calls drop from 5 to 0 across the two examples. Redundant intermediates drop from 7 total to 0 total. ODL drops from ∞\infty (no valid decomposition under the option set) to small finite values (1–2 options). These two examples are representative; Spearman correlations across all 5,042 passing solutions are reported in Figure 6.

Gradient Variance Analysis

Figure 5 plots the per-step relative gradient norm variance (computed as the ratio of the gradient-norm variance to the squared mean gradient norm, equivalent to the inverse SNR squared) for flat GRPO and Option-GRPO. Option-GRPO maintains lower variance throughout training, with the gap widening from 1.4× at initialization to about 2.4× by step 18,000 and 2.3× at the end of training. The reduction is in the direction predicted by Proposition 3 but smaller than its scaling: a 2.3× lower relative variance corresponds to an SNR gain of about 1.5×, against T/K≈4.5×\sqrt{T/K} \approx4.5\times for T=100T=100, K=5K=5. The assumptions of the proposition (reward-correlated signal confined to option-decision positions, conditionally independent positions) are idealizations, so it should be read as giving the direction and scaling of the effect, not its measured size.

Full Ablation and Per-SDK Tables

Further ablations. Table 11 reports the option-level and reward-function ablations. For credit assignment, combining the sequence-level and option-level advantages outperforms either alone by 4.1–4.6 pass@1 points. In the curriculum-pacing sweep, the adaptive setting ρadv=0.5\rho_{\mathrm{adv}}=0.5 is best. In the group-size sweep, G=16G=16 is the Pareto knee. Per-SDK results are in Table 16.

SDKFrozenExpC.ReCodeFlat GRPOOpt-GRPOΔ\Delta vs ReCodeCategory
cuallee47.165.269.667.278.2+8.6Data Valid.
LanceDB40.262.066.763.874.6+7.9Vector DB
Marker43.963.167.564.874.1+6.6Doc Proc.
Surya34.854.858.255.863.2+5.0Comp. Vision
Daft34.258.164.259.972.9+8.7DataFrame
Vortex28.050.154.051.760.6+6.6Scientific
libdebug27.448.853.651.260.2+6.6Debugging
FastStream24.746.850.849.357.1+6.3Messaging
Docling20.843.348.845.254.5+5.7Doc Proc.
reflexio18.440.945.642.851.4+5.8Agent Frmwk
Tracecat11.634.039.838.246.8+7.0Security
MCP Python10.935.342.442.543.2+0.8Protocol
Average28.550.255.152.761.4+6.3–

Table 16. Per-SDK results on UnseenSDK-Bench (pass@1 %). ExpC. = ExploraCoder.

Option-GRPO leads in every cell of Table 15. The pass@1 margin over the stronger current model is +5.7 on Surya and +4.7 on Tracecat (95% CI +3.6 to +8.7 and +3.9 to +7.3, paired bootstrap over tasks), and the pass@10 margin on Tracecat is +12.7. Option-GRPO also leads under the stricter reference-output-equivalence scorer.

Empirical Validation of the Compression Results

Figure 6 plots ODLH(c)\mathrm{ODL}_{\mathcal{H}}(c) against the four code quality metrics defined in Definition 4, computed over all 5,042 passing solutions produced by Option-GRPO across the 12 SDKs. Theorem 2 and Corollary 1 predict monotonic relationships for QconcQ_{\mathrm{conc}} and QapiQ_{\mathrm{api}}; Remark 5 treats QredQ_{\mathrm{red}} and QidiomQ_{\mathrm{idiom}} as empirical predictions. We observe Spearman rank correlations of ρconc=−0.71\rho_{\mathrm{conc}}=-0.71, ρapi=−0.83\rho_{\mathrm{api}}=-0.83, ρred=−0.62\rho_{\mathrm{red}}=-0.62, and ρidiom=−0.79\rho_{\mathrm{idiom}}=-0.79 (all p<10−50p<10^{-50}), consistent with the predicted negative association between ODL and each quality metric in practice.

Table 13: Comparison with prior RL methods on UnseenSDK-Bench. *Adapted to use our layered execution reward in place of unit tests. All use the same Qwen3-32B base.

Methodp@1p@10CriticCL ↓ID ↑
CodeRL-adapted*46.863.2Yes79.50.38
PPOCoder-adapted*48.564.7Yes76.10.40
StepCoder-adapted*51.367.9Yes72.80.45
ExploraCoder50.267.5No78.40.43
ReCode55.171.2No74.10.51
Flat GRPO52.768.4No71.30.41
Option-GRPO61.476.8No52.00.74

Table 13. Comparison with prior RL methods on UnseenSDK-Bench. *Adapted to use our layered execution reward in place of unit tests. All use the same Qwen3-32B base.

Detailed Artefact Release

We release the following artefacts, under permissive licences, at https://github.com/prashankkadam/option-grpo-code (the main paper’s brief summary appears just before the bibliography):

  • UnseenSDK-Bench: 12 SDKs ×\times three curriculum levels of tasks, totalling 1,871 Level-0 tasks, 3,742 Level-1 tasks, and 398 Level-2 tasks, with 7,484 auto-generated property tests, Docker sandbox configurations per SDK, and the layered execution reward implementation. Released under Apache 2.0.

  • Option discovery and extraction code (Algorithms 1, 2, 3): parameter-flow graph construction, candidate mining, trajectory-based certification. Apache 2.0.

  • Training code: Option-GRPO and all baselines (Flat GRPO, ReCode reimplementation, CodeRL/PPOCoder/StepCoder adaptations) on top of a publicly available GRPO framework, with full hyperparameter configurations (Table 4). Apache 2.0.

  • Evaluation harness (benchmark task runner, property-test scoring, pass@kk and code-quality metric computation) and the exact prompts used for the contamination probe (Appendix § 10.§ 2).

Apache 2.0.

Methodological details. All experiments report mean values across 3 random seeds per SDK; per-seed variance is below ±1.2\pm1.2 points pass@1 in every configuration we tested. Hardware is 8× NVIDIA H100 80GB with ZeRO-3 sharding and bf16 mixed precision; the full per-phase compute budget appears in Table 18, totalling roughly 7,632 H100 GPU-hours for the headline 12-SDK 32B run (or an estimated ∼5,500\sim5{,}500 H100 GPU-hours when cross-SDK transfer is exploited). Carbon estimate at 0.42 kgCO2_2 per H100 GPU-hour: ∼3,205\sim3{,}205 kgCO2_2e total. We adhered to each SDK’s terms of use; none of the 12 SDKs are training-data-licensed, and all permit static analysis and benchmarking. No human subjects were involved in any phase of the work.

PhaseStepsSmp/StepGPU-hrs
Graph construction––< 0.1
Option discovery––< 0.1
Property test gen.––< 0.1
Level 0 training5,00016113.6
Level 1 training10,00016198.4
Level 2 training15,00016324.0
Total per SDK30,000–636.0
Total all 12 SDKs360,000–7,632

Table 18. Estimated per-phase computational cost in aggregate H100 GPU-hours (8 GPUs × wall-clock hours on the 8×H100 node).

Computational Details

Detailed Cost Breakdown

Scaling Analysis

Training cost scales with SDK size primarily through two factors. First, the number of Level 0 and Level 1 tasks grows with the number of public functions, increasing task distribution diversity but not the number of training steps (we train for

a fixed step budget, sampling from the distribution). Second, execution cost per sample depends on SDK complexity: data processing SDKs (Daft, LanceDB) execute in under 1 s per sample, while SDKs with I/O operations (Docling, Marker) may require 5–10 s, impacting wall-clock time linearly.

The dominant cost is Level 2 training (15,000 steps × 16 samples), which requires generating and executing 240,000 complete programs per SDK. Sandbox execution averages 1.8 s per sample, and the wall-clock time for Level 2 is approximately 40.5 hours per SDK on the 8-GPU node (324 GPU-hours, Table 18).

Cross-SDK option transfer (Section D.6) can amortise costs: under our cost model, cross-SDK transfer reduces the estimated 12-SDK training cost from 7,632 to approximately 5,500 H100 GPU-hours, a reduction of about 28%. This is an estimate; Table 12 measures transfer accuracy, not the saving.

Cost Sensitivity

The default training budget (30,000 total steps on a Qwen3-32B, 636 H100 GPU-hours per SDK, ∼80\sim80 hours wall-clock on the 8-GPU node) is calibrated for the strongest pass@1. The method does not require this budget. Table 19 sweeps the training-step budget at fixed 32B scale; Table 20 sweeps the dense base-model size at fixed step budget. Both tables average across the 12 SDKs. All variants are dense Qwen3 models; we do not test MoE variants here to keep the size axis interpretable (parameters and active parameters coincide for dense models).

Total stepsGPU-hrWall (h)FlatOptΔ\Delta
5,0001121433.841.2+7.4
10,0002242844.654.1+9.5
15,0003364249.358.7+9.4
20,0004405551.260.4+9.2
30,000 (default)6368052.761.4+8.7
45,00097612253.162.1+9.0

Table 19. Training-budget sweep on Qwen3-32B (dense). Step counts are total across all three curriculum phases; we allocate 1/6, 1/3, 1/2 of the budget to Phases 1, 2, 3 respectively. The Flat GRPO baseline runs at the same total step budget and the same 32B base. GPU-hr is aggregate H100 GPU-hours per SDK; Wall is wall-clock hours on the 8×H100 node.

BaseParamsMemFlatOptΔ\Delta
Qwen3-8B8.2B20 GB43.252.6+9.4
Qwen3-14B14.8B36 GB48.957.3+8.4
Qwen3-32B32.8B75 GB52.761.4+8.7

Table 20. Base-model-size sweep at default (30,000 step) budget. All variants are dense models from the Qwen3 family. GPU-memory footprint is per-replica peak during training with ZeRO-3 sharding on 8×H100.

Three observations. (a) The hierarchical structure carries more weight than additional training steps. At 10,000 steps (∼\sim 224 H100 GPU-hours, ∼\sim 28 hours wall-clock), Option-GRPO reaches 54.1% pass@1, 1.4 points above Flat GRPO at full compute (52.7% at 30,000 steps). The smallest training budget at which Option-GRPO beats Flat-GRPO-at-default is roughly one-third the default compute. Practitioners with limited GPU access can therefore obtain Flat-GRPO-equivalent quality at a third of the cost, or better-than-Flat quality at full cost.

(b) The option vocabulary is the dominant gain, not scale. Option-GRPO on the 8B dense base (52.6%) is within 0.1 points of Flat GRPO on the 32B dense base (52.7%). The roughly 3.8× smaller GPU-memory footprint of the 8B variant makes Option-GRPO deployable on commodity hardware (a single 24GB GPU after 4-bit quantization) without sacrificing the absolute pass@1 level of the full-scale Flat baseline.

(c) Returns saturate beyond the default budget. Increasing from 30,000 to 45,000 total steps yields only +0.7 points pass@1 (61.4 → 62.1) at 1.5× the cost. The 30,000-step default sits at the elbow of the cost–performance curve. The Pareto frontier across budget and base-size is shown implicitly by the table: an 8B-Option run at 30,000 steps (52.6%, 20GB) dominates a 32B-Flat run at 30,000 steps (52.7%, 75GB) on the cost-adjusted criterion “pass@1 per GB-of-memory.”

Implications for industry deployment. Combining (a) cross-SDK option transfer (Section D.6) and (b) the Qwen3-8B dense base brings the marginal cost of adapting Option-GRPO to a new SDK to ∼\sim 120 H100 GPU-hours (∼\sim 15 hours wall-clock) plus ∼\sim 30 minutes of CPU graph construction: comparable to a single inference batch on a 32B frontier model, making per-SDK adaptation tractable for most engineering teams targeting a small set of high-traffic libraries.

Algorithm 4 Option-GRPO Training

Require: SDK source code SS, base model πbase\pi_{\mathrm{base}}, group size GG, advancement threshold ρadv\rho_{\mathrm{adv}}

Ensure: Trained hierarchical policy πθ\pi_\theta

  1. // Stage 0: Graph Construction and Option Discovery

  2. Construct GapiG_{\mathrm{api}}, GflowG_{\mathrm{flow}}, GsemG_{\mathrm{sem}} from SS

  3. Discover Level 0 options O0\mathcal{O}_0 from public functions

  4. Discover Level 1 candidates O1cand\mathcal{O}^{\mathrm{cand}}_1 via Algorithm 1

  5. Generate curriculum tasks D0\mathcal{D}_0, D1\mathcal{D}_1, D2\mathcal{D}_2

  6. Generate property-based tests via Algorithm 3

  7. // Phase 1: Atomic Option Learning (Level 0)

  8. Initialize πθ←πbase\pi_\theta\leftarrow\pi_{\mathrm{base}}

  9. for step =1=1 to N0N_0 do

  10. Sample x∼D0x \sim\mathcal{D}_0; generate GG samples; execute; compute RexecR_{\mathrm{exec}}

  11. Compute GRPO advantages and update πθ\pi_\theta

  12. end for

  13. Extract O0\mathcal{O}_0 via Algorithm 2; partially freeze

  14. // Phase 2: Pattern Learning (Level 1)

  15. ℓcurr←2\ell_{\mathrm{curr}} \leftarrow2

  16. for step =1=1 to N1N_1 do

  17. Sample x∼D1x \sim\mathcal{D}_1 with length ≤ℓcurr\leq\ell_{\mathrm{curr}}

  18. Generate GG option sequences (action space A1\mathcal{A}_1)

  19. Expand, execute, compute Rexec+λqRqualityR_{\mathrm{exec}} + \lambda_q R_{\mathrm{quality}}

  20. Compute sequence-level and option-level advantages

  21. Update πθ\pi_\theta via the combined gradient (Appendix § 8.§ 1)

  22. if pass rate >ρadv> \rho_{\mathrm{adv}} then

  23. ℓcurr←min⁡(ℓcurr+1,Lmax)\ell_{\mathrm{curr}} \leftarrow\min(\ell_{\mathrm{curr}}+1, L_{\mathrm{max}})

  24. end if

  25. end for

  26. Extract O1\mathcal{O}_1 via Algorithm 2; partially freeze

  27. // Phase 3: Program Learning (Level 2)

  28. for step =1=1 to N2N_2 do

  29. Sample x∼D2x \sim\mathcal{D}_2

  30. Generate GG option sequences (action space A2\mathcal{A}_2)

  31. Expand, execute, compute Rexec+λqRquality+λcRconsistencyR_{\mathrm{exec}} + \lambda_q R_{\mathrm{quality}} + \lambda_c R_{\mathrm{consistency}}

  32. Compute hierarchical advantages and update πθ\pi_\theta

  33. end for

  34. return $\pi_\theta

References

  1. [1]Pierre-Luc Bacon, Jean Harb, and Doina Precup. 2017. The option-critic architecture. In Proceedings of the AAAI Conference on Artificial Intelligence, pages 1726–1734.arxiv.org/abs/1609.05140
  2. [2]Yoshua Bengio, Jérôme Louradour, Ronan Collobert, and Jason Weston. 2009. Curriculum learning. In Proceedings of the 26th International Conference on Machine Learning, pages 41–48.DOI
  3. [3]DeepSeek-AI. 2025. DeepSeek-R1: Incentivizing reasoning capability in LLMs via reinforcement learning. arXiv preprint arXiv:2501.12948.DOI
  4. [4]Thomas G. Dietterich. 2000. Hierarchical reinforcement learning with the MAXQ value function decomposition. Journal of Artificial Intelligence Research, 13:227–303.DOI
  5. [5]Shihan Dou, Yan Liu, Haoxiang Jia, Enyu Zhou, Limao Xiong, Junjie Shan, Caishuang Huang, Xiao Wang, Xiaoran Fan, Zhiheng Xi, Yuhao Zhou, Tao Ji, Rui Zheng, Qi Zhang, Tao Gui, and Zhaojun Huang. 2024. StepCoder: Improving code generation with reinforcement learning from compiler feedback. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 4571–4585.DOI
  6. [6]GitClear. 2025. AI copilot code quality: 2025 data suggests 4x growth in code clones. https://www.gitclear.com/ai_assistant_code_quality_2025_research.
  7. [7]Glenn A. Iba. 1989. A heuristic approach to the discovery of macro-operators. Machine Learning, 3(4):285–317.
  8. [8]Richard E. Korf. 1985. Macro-operators: A weak method for learning. Artificial Intelligence, 26(1):35–77.DOI
  9. [9]Sachit Kuhar, Wasi Uddin Ahmad, Zijian Wang, Nihal Jain, Haifeng Qian, Baishakhi Ray, Murali Krishnna Ramanathan, Xiaofei Ma, and Anoop Deoras. 2025. LibEvolutionEval: A benchmark and study for version-specific code generation. In Proceedings of the 2025 Conference of the Nations of the Americas Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume 1: Long Papers), pages 6826–6840, Albuquerque, New Mexico. Association for Computational Linguistics.
  10. [10]Hung Le, Yue Wang, Akhilesh Deepak Gotmare, Silvio Savaresе, and Steven C. H. Hoi. 2022. CodeRL: Mastering code generation through pretrained models and deep reinforcement learning. In Advances in Neural Information Processing Systems. Yujia Li, David Choi, Junyoung Chung, Nate Kushman, Julian Schrittwieser, Rémi Leblond, Tom Eccles, James Keeling, Felix Gimeno, Agustin Dal Lago, Thomas Hubert, Peter Choy, Cyprien de Masson d’Autume, Igor Babuschkin, Xinyun Chen, Po-Sen Huang, Johannes Welbl, Sven Gowal, Alexander Cherapanov, and 7 others. 2022. Competition-level code generation with AlphaCode. Science, 378(6624):1092–1097.
  11. [11]Jiate Liu, Yiqin Zhu, Kaiwen Xiao, Qiang Fu, Xiao Han, Wei Yang, and Deheng Ye. 2023. RLTF: Reinforcement learning from unit test feedback. Transactions on Machine Learning Research.arxiv.org/abs/2307.04349
  12. [12]Zeyu Leo Liu, Shrey Pandit, Xi Ye, Funsol Choi, and Greg Durrett. 2024. CodeUpdateArena: Benchmarking knowledge editing on API updates. arXiv preprint arXiv:2407.06249.arxiv.org/abs/2407.06249
  13. [13]Diganta Misra, Nizar Islah, Victor May, Brice Rauby, Zihan Wang, Justine Gehring, Antonio Orvieto, Muawiz Chaudhary, Eilif B. Muller, Irina Rish, Samira Ebrahimi Kahou, and Massimo Caccia. 2025. GitChameleon 2.0: Evaluating AI code generation against Python library version incompatibilities. arXiv preprint arXiv:2507.12367.arxiv.org/abs/2507.12367
  14. [14]Sanmit Narvekar, Bei Peng, Matteo Leonetti, Jiyko Sinapov, Matthew E. Taylor, and Peter Stone. 2020. Curriculum learning for reinforcement learning domains: A framework and survey. Journal of Machine Learning Research, 21(181):1–50.arxiv.org/abs/2003.04960
  15. [15]Kolby Nottingham, Prithviraj Ammanabrolu, Alane Suhr, Yejin Choi, Hannah Hajishirzi, Sameer Singh, and Roy Fox. 2023. Do embodied agents dream of pixelated sheep: Embodied decision making using language guided world modelling. In Proceedings of the 40th International Conference on Machine Learning (ICML).arxiv.org/abs/2301.12050
  16. [16]Shishir G. Patil, Tianjun Zhang, Xin Wang, and Joseph E. Gonzalez. 2024. Gorilla: Large language model connected with massive APIs. In Advances in Neural Information Processing Systems.arxiv.org/abs/2305.15334
  17. [17]Rémy Portelas, Cédric Colas, Lilian Weng, Katja Hofmann, and Pierre-Yves Oudeyer. 2020. Automatic curriculum learning for deep RL: A short survey. In Proceedings of the Twenty-Ninth International Joint Conference on Artificial Intelligence.arxiv.org/abs/2003.04664
  18. [18]Qwen Team. 2025. Qwen3 technical report. arXiv preprint arXiv:2505.09388.
  19. [19]Zhihong Shao, Peiyi Wang, Qihao Zhu, Runxin Xu, Junxiao Song, Xiao Bi, Haowei Zhang, Mingchuan Zhang, Y. K. Li, Y. Wu, and Daya Guo. 2024. DeepSeekMath: Pushing the limits of mathematical reasoning in open language models. arXiv preprint arXiv:2402.03300.arxiv.org/abs/2402.03300
  20. [20]Parshin Shojaee, Aneesh Jain, Sindhu Tipirneni, and Chandan K. Reddy. 2023. Execution-based code generation using deep reinforcement learning. Transactions on Machine Learning Research.arxiv.org/abs/2301.13816
  21. [21]Richard S. Sutton, Doina Precup, and Satinder Singh. 1999. Between MDPs and semi-MDPs: A framework for temporal abstraction in reinforcement learning. Artificial Intelligence, 112(1–2):181–211.DOI
  22. [22]Alexander Sasha Vezhnevets, Simon Osindero, Tom Schaul, Nicolas Heess, Max Jaderberg, David Silver, and Koray Kavukcuoglu. 2017. FeUdal networks for hierarchical reinforcement learning. In Proceedings of the 34th International Conference on Machine Learning, pages 3540–3549.arxiv.org/abs/1703.01161
  23. [23]Guanzhi Wang, Yuqi Xie, Yunfan Jiang, Ajay Mandlekar, Chaohui Xiao, Yuke Zhu, Linxi Fan, and Anima Anandkumar. 2024. Voyager: An open-ended embodied agent with large language models. Transactions on Machine Learning Research.arxiv.org/abs/2305.16291
  24. [24]Yunkun Wang, Yue Zhang, Zhen Qin, Chen Zhi, Binhua Li, Fei Huang, Yongbin Li, and Shuaiguang Deng. 2025a. ExploraCoder: Advancing code generation for multiple unseen APIs via planning and chained exploration. In Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 18124–18145, Vienna, Austria. Association for Computational Linguistics.
  25. [25]Zora Zhiruo Wang, Akari Asai, Xinyan Velocity Yu, Frank F. Xu, Yiqing Xie, Graham Neubig, and Daniel Fried. 2025b. CodeRAG-Bench: Can retrieval augment code generation? In Findings of the Association for Computational Linguistics: NAACL.
  26. [26]Simon Willison. 2024. Slop is the new name for unwanted AI-generated content. https://simonwillison.net/2024/May/8/slop/. Accessed 2026.
  27. [27]Haoze Wu, Yunzhi Yao, Wenhao Yu, and Ningyu Zhang. 2026. ReCode: Updating code API knowledge with reinforcement learning. In Proceedings of the AAAI Conference on Artificial Intelligence.arxiv.org/abs/2506.20495
  28. [28]Chunqiu Steven Xia, Yuxiang Wei, and Lingming Zhang. 2023. Automated program repair in the era of large pre-trained language models. In Proceedings of the 45th International Conference on Software Engineering, pages 1482–1494.DOI
  29. [29]Daoguang Zan, Bei Chen, Fengji Zhang, Dianjie Lu, Bingchao Wu, Bei Guan, Yongji Wang, and Jian-Guang Lou. 2022. Large language models meet NL2Code: A survey. arXiv preprint arXiv:2212.09420.arxiv.org/abs/2212.09420
  30. [30]Shuyan Zhou, Uri Alon, Frank F. Xu, Zhiruo Wang, Zhengbao Jiang, and Graham Neubig. 2023. DocPrompting: Generating code by retrieving the docs. In International Conference on Learning Representations.arxiv.org/abs/2207.05987

Paper details

vlnc:W_hnp99heu
Version
v1 · latest
Published
Fresh — added to the Hub within the last 30 days
Added
License
CC BY 4.0
Usage

views · 10

downloads · 0

A view counts once per browser per day, a download each time the PDF is requested.

  • Computer Science

Not recorded for this paper

View statement
Looking for related papers…
Contents