betaFeedback

WorksComputer Science

C-GRPO: Conformal Group Relative Policy Optimization

Abstract

Reasoning-oriented language model training increasingly relies on multiple on-policy rollouts, but standard Group Relative Policy Optimization (GRPO) uses a fixed group size, wasting compute on easy prompts and under-exploring hard ones. Existing adaptive-budget methods alleviate this inefficiency, yet their stopping rules are heuristic and do not provide a finite-sample calibration guarantee. We propose CGRPO, which replaces the fixed rollout budget of standard GPRO with calibrated conformal thresholds over task-specific nonconformity scores, combining Adaptive Prediction Set (APS)-style calibration for closed-form reasoning, a first-success score for execution-based code, automatic miscoverage selection, and periodic re-calibration. We prove finite-sample coverage for the resulting stopping rule and show that, under monotone policy improvement, recalibration induces a non-increasing expected training budget. Across six benchmarks spanning mathematics, graduate-level science, and code generation, CGRPO reduces mean training rollouts by 49-70% relative to fixed-budget GRPO while retaining strong fixed-budget accuracy, achieves the strongest pass@1 among evaluated static and dynamic baselines on different datasets, and transfers across model families with up to 82% rollout savings.

Introduction

As LLMs scale, a massive shift toward multi-sample computation at both inference and training time is taking place. Strategies like majority voting for reasoning tasks and best-of-kk sampling for code generation consistently push the Pareto frontier. Even modern RL algorithms such as GRPO (Group Relative Policy Optimization) [23] leverage this by using on-policy rollouts (generating a batch of responses for the same prompt) to derive a relative advantage signal for the loss function. However, the bottleneck is that GRPO currently uses a static group size (GG) for every input. This is fundamentally inefficient because it ignores the high variance in task difficulty. For “low-entropy" or trivial queries, generating a large group is just wasted FLOPs with a near-zero gradient signal. In contrast, for “frontier" or high-complexity queries, a fixed GG is often too small to capture a single successful trajectory, leaving the model with no positive reinforcement to learn from.

Adaptive-budget methods address this with reward variance, learned difficulty estimates, or global allocation rules [28, 17, 27, 29], but their stopping rules remain heuristic and offer no finite-sample guarantee as the policy evolves. Conformal prediction provides distribution-free uncertainty certificates [25, 21], but has been used only as a post-hoc inference wrapper, not as a training-time compute control mechanism [18, 11, 5].

40th Conference on Neural Information Processing Systems (NeurIPS 2026).

We bridge these perspectives with Conformal GRPO (C-GRPO), which replaces GRPO’s fixed rollout budget with calibrated stopping thresholds. For each budget k∈Kk \in\mathcal{K}, C-GRPO selects the smallest kk which is statistically sufficient for the current prompt. Practical ingredients include Adaptive Prediction Set (APS) calibration for closed-form reasoning, a new first-success score for execution-based code, automatic miscoverage selection to prevent degenerate early-training thresholds, and periodic recalibration to track the evolving policy. The result is a self-paced curriculum: easy queries exit early, and computational resources are reallocated to the hard tail.

We prove finite-sample coverage for the calibrated thresholds, derive false-stop control at every budget and a multiplicity-corrected guarantee for the adaptively selected budget, and establish a non-increasing expected training budget under monotone policy improvement (an idealized condition; § 4.5.2). Empirically, across six benchmarks that span mathematics, graduate-level science, and code, C-GRPO reduces mean training rollouts by 49–70% compared to fixed-budget GRPO at matched accuracy: pass@1 on GSM8K, HumanEval, MBPP, and MATH-500 is equal to or slightly above every evaluated baseline (differences are within binomial noise) with the fewest rollouts per prompt; rollout savings further reach 82% on GSM8K for Llama-3.1-8B without re-tuning. These theoretical and empirical results are enabled by three contributions: (i) we formulate adaptive-group GRPO as a conformal calibration problem; (ii) we introduce practical ingredients for LLM RL, including APS with first-success scores, auto-δ\delta, and periodic re-calibration; and (iii) we prove coverage, false-stop and self-annealing guarantees and show empirically that calibrated allocation matches the accuracy of fixed and heuristic budgets at far fewer rollouts, including fixed-GG GRPO at matched compute.

Background

Policy Optimization and GRPO

Group Relative Policy Optimization (GRPO) [23] eliminates the memory-heavy value network of the PPO (Proximal Policy Optimization) [22] by computing advantages from a fixed group of GG i.i.d. (independent and identically distributed) completions {y1,…,yG}∼πθ(⋅∣x)\{y_1,\ldots,y_G\}\sim\pi_\theta(\cdot\mid x):

A^iGRPO=ri−rˉσr+ϵ,rˉ=1G∑j=1Grj,σr=1G∑j=1G(rj−rˉ)2.(1)\hat{A}^{\mathrm{GRPO}}_i = \frac{r_i-\bar{r}}{\sigma_r+\epsilon}, \qquad\bar{r}=\frac{1}{G}\sum_{j=1}^{G}r_j, \qquad\sigma_r=\sqrt{\frac{1}{G}\sum_{j=1}^{G}(r_j-\bar{r})^2}. \tag*{(1)}

where ϵ>0\epsilon>0 is a small numerical stabilizer (e.g., 10−810^{-8}). This achieves state-of-the-art results on mathematical reasoning and code generation. However, fixing GG uniformly wastes compute on easy queries (all rj≈1r_j\approx1, zero advantage) and starves hard queries of exploration (all rj=0r_j=0, no positive signal), ignoring the wide per-prompt difficulty disparity [7].

Split-Conformal Prediction

Split-conformal prediction (CP) [25] constructs statistically guaranteed prediction sets with no distributional assumptions. Given nn i.i.d. calibration pairs (Xi,Yi)(X_i,Y_i) and a nonconformity score s(X,Y)∈Rs(X,Y)\in\mathbb{R}, the empirical (1−δ)(1-\delta) quantile q^\hat{q} (with the standard finite-sample correction [(n+1)(1−δ)][(n+1)(1-\delta)]) satisfies P(s(Xn+1,Yn+1)≤q^)∈[1−δ, 1−δ+1n+1]P(s(X_{n+1},Y_{n+1})\leq\hat{q})\in[1-\delta,\,1-\delta+\frac{1}{n+1}] for any exchangeable test point. Prior work applies CP post-hoc for inference-time abstention and safety filtering [18]. C-GRPO inverts this paradigm: rather than certifying a fixed model’s outputs, we embed CP inside the training loop so calibrated quantiles directly control the per-query generation budget, converting uncertainty measures into online compute decisions.

Related Work

Efficient RL and Adaptive Rollout Allocation. Adaptive rollout methods address GRPO’s fixed-GG inefficiency via rollout pruning, difficulty-aware allocation, and adaptive sampling [28, 17, 27, 29], but rely on heuristic or model-dependent signals (reward variance, learned difficulty scores) that lack formal guarantees and drift as the policy evolves. C-GRPO replaces them with a distribution-free conformal stopping rule grounded in finite-sample coverage theory.

Conformal Prediction for LLMs. Conformal methods have been applied to LLMs for calibrated uncertainty, abstention, and output filtering [18, 11, 5], as static post-hoc wrappers relying on token-level likelihoods that are uninformative when every completion is a unique string. We instead embed CP in the training objective and introduce an ordinal first-success score for execution-based reinforcement-learning (RL) settings.

Curriculum and Self-Paced Learning. Curriculum methods improve convergence by ordering examples from easy to hard [14]; adaptive rollout methods implicitly induce such curricula without guarantees. C-GRPO instantiates a self-paced curriculum (Corollary 7, under an idealized monotone-improvement condition, and empirically in aggregate): as the policy improves, calibrated thresholds q^k\hat{q}_k tighten, shifting compute to harder queries with no external supervision or schedule.

Methodology

Overview of C-GRPO workflow

Figure 1. Overview of C-GRPO workflow.

Problem Formulation

Let X\mathcal{X} denote the space of natural-language queries and Y\mathcal{Y} the space of model completions. A language model defines a conditional distribution πθ(⋅∣x)\pi_\theta(\cdot\mid x) over Y\mathcal{Y} parameterized by θ\theta. An answer-extraction function f:Y→Af:\mathcal{Y}\to\mathcal{A} maps raw completions to a structured answer space A\mathcal{A}, and a deterministic verifier v:A×A→{0,1}v:\mathcal{A}\times\mathcal{A}\to\{0,1\} checks correctness against the gold answer y⋆y^\star:

v(a,a⋆)={1if a is verified correct w.r.t. a⋆,0otherwise.(2)v(a,a^\star)=\begin{cases} 1 & \text{if }a\text{ is verified correct w.r.t. }a^\star,\\ 0 & \text{otherwise.} \end{cases} \tag*{(2)}

We consider two task families: closed-form (math, science), where ff extracts a canonicalized answer and vv is exact-match; and generative code (HumanEval, MBPP), where vv executes against hidden unit tests. Let K={k1<⋯<kM}\mathcal{K}=\{k_1<\cdots<k_M\} be a fixed budget grid with Kmax⁡=kMK_{\max}=k_M and rewards rj=v(f(y^j),y⋆)r_j=v(f(\hat{y}_j),y^\star). The training objective is:

θ⋆=arg⁡max⁡θ Ex[E[max⁡j≤kusedrj]]s.t.Ex[kused(x;θ)]≤C.(3)\theta^\star=\underset{\theta}{\arg\max}\ \mathbb{E}_x\left[\mathbb{E}\left[\max_{j\leq k_{\mathrm{used}}}r_j\right]\right]\quad\text{s.t.}\quad\mathbb{E}_x\left[k_{\mathrm{used}}(x;\theta)\right]\leq C. \tag*{(3)}

Here C>0C>0 is a target per-step compute budget, and kused(x;θ)k_{\mathrm{used}}(x;\theta) is the adaptive group size for query xx under policy πθ\pi_\theta (formally defined in § 4.4.1). C-GRPO solves this without explicit constraint optimization: calibrated conformal thresholds implicitly determine kusedk_{\mathrm{used}} per query.

Conformal Nonconformity Scores

The conformal machinery requires a nonconformity score s:A×Ak→[0,1]s:\mathcal{A}\times\mathcal{A}^k\to[0,1] quantifying how surprising the gold answer y⋆y^\star is given kk model samples. A good score must be (i) continuous so that calibrated quantiles strictly decrease as the model improves, and (ii) consistent with exchangeability of the samples so that split-conformal validity holds without distributional assumptions on PP.

APS Score for Closed-Form Answers

For closed-form tasks we use the Adaptive Prediction Sets (APS) score of Romano et al. [ [21]. Given kk extracted answers a^1,…,a^k\hat{a}_1,\ldots,\hat{a}_k, let p^(a)=k−1∑j=1k1[a^j=a]\hat{p}(a)=k^{-1}\sum_{j=1}^{k}\mathbf{1}[\hat{a}_j=a] be the empirical frequency. Sort distinct answers in decreasing frequency order aσ(1),aσ(2),…a_{\sigma(1)},a_{\sigma(2)},\ldots (ties broken uniformly at random to strictly preserve statistical exchangeability). The APS nonconformity score is

sAPS(y⋆;a^1:k)=∑j:p^(aσ(j))>p^(y⋆)p^(aσ(j))⏟mass strictly more frequent than gold+U⋅p^(y⋆)⏟randomized fractional mass,U∼Uniform⁡[0,1],(4)s_{\mathrm{APS}}(y^\star;\hat{a}_{1:k}) = \underbrace{\sum_{j:\hat{p}(a_{\sigma(j)})>\hat{p}(y^\star)}\hat{p}(a_{\sigma(j)})}_{\text{mass strictly more frequent than gold}} + \underbrace{U\cdot\hat{p}(y^\star)}_{\text{randomized fractional mass}}, \qquad U\sim\operatorname{Uniform}[0,1], \tag*{(4)}

with sAPS=1s_{\mathrm{APS}}=1 when p^(y⋆)=0\hat{p}(y^\star)=0 (gold never sampled).

The APS score is strictly superior to the naive binary score sbin=1−p^(y⋆)s_{\mathrm{bin}}=1-\hat{p}(y^\star): the UU-randomization makes it strictly continuous on [0,1][0,1], ensuring the (1−δ)(1-\delta) quantile remains informative even for weak models, and achieves exact (not merely conservative) marginal coverage [21].

First-Success Score for Code

For execution-based code tasks, every generated completion is a syntactically unique string, so p^(y⋆)=0\hat{p}(y^\star)=0 almost surely and sAPS=1s_{\mathrm{APS}}=1 for all calibration examples regardless of budget, making the APS framework vacuous. We therefore introduce a novel first-success score based on the ordinal position of the first passing solution within the ordered sample sequence:

scode(y⋆;y^1:k)={j⋆k,j⋆=min⁡{j:v(y^j,y⋆)=1} exists,1,no sample passes all unit tests.(5)s_{\mathrm{code}}(y^\star;\hat{y}_{1:k}) = \begin{cases} \frac{j^\star}{k}, & j^\star=\min\{j:v(\hat{y}_j,y^\star)=1\}\text{ exists},\\ 1, & \text{no sample passes all unit tests.} \end{cases} \tag*{(5)}

That is, the score is the fraction of the budget consumed before the first success. It is continuous on (0,1](0,1]: an easy problem solved on the first of kk samples scores 1/k→01/k\to0; an unsolvable problem always scores 1.

This score has a key calibration property: q^kcode<1\hat{q}^{\mathrm{code}}_k<1 if and only if at least ⌈(n+1)(1−δ)⌉−1\lceil(n+1)(1-\delta)\rceil-1 calibration problems have at least one passing solution within kk samples (§ 4.3). As the policy improves during training, more problems become solvable within small kk, driving q^k\hat{q}_k below 1 and enabling the conformal curriculum to engage at smaller budgets.

Exchangeability. Both scores are functions of (y⋆,y^1:k)(y^\star,\hat{y}_{1:k}) and inherit exchangeability from i.i.d. sampling of completions and calibration examples—the sole assumption conformal validity requires.

Split-Conformal Calibration

Given a held-out calibration set Dcal={(xi,yi⋆)}i=1n\mathcal{D}_{\mathrm{cal}}=\{(x_i,y_i^\star)\}_{i=1}^n disjoint from training data and a model π\pi, we generate Kmax⁡K_{\max} samples y^i,1,…,y^i,Kmax⁡∼π(⋅∣xi)\hat{y}_{i,1},\ldots,\hat{y}_{i,K_{\max}}\sim\pi(\cdot\mid x_i) per query and derive one threshold per budget level.

Definition 1 (Split Conformal Quantile). For scores s1,…,sn∈[0,1]s_1,\ldots,s_n\in[0,1] and miscoverage level δ∈(0,1)\delta\in(0,1), the finite-sample conformal quantile is:

q^(s1:n;δ)=s(min⁡(⌈(n+1)(1−δ)⌉,n)),(6)\hat{q}(s_{1:n};\delta)=s_{\left(\min\left(\left\lceil(n+1)(1-\delta)\right\rceil,n\right)\right)}, \tag*{(6)}

where s(j)s_{(j)} denotes the jj-th order statistic. The correction ⌈(n+1)(1−δ)⌉\lceil(n+1)(1-\delta)\rceil (rather than ⌈n(1−δ)⌉\lceil n(1-\delta)\rceil) is the standard finite-sample adjustment ensuring P(stest≤q^)≥1−δ\mathbb{P}(s_{\mathrm{test}}\leq\hat{q})\geq1-\delta for an exchangeable test score [25].

For each k∈Kk\in\mathcal{K} and score s∈{sAPS,scode}s\in\{s_{\mathrm{APS}},s_{\mathrm{code}}\}, the calibrated threshold is:

q^k=q^({s(yi⋆;y^i,1:k)}i=1n,δ).(7)\hat{q}_k=\hat{q}\left(\{s(y_i^\star;\hat{y}_{i,1:k})\}_{i=1}^{n},\delta\right). \tag*{(7)}

The result is a threshold vector q^=(q^k1,…,q^kM)\hat{\mathbf{q}}=(\hat{q}_{k_1},\ldots,\hat{q}_{k_M}) encoding the model’s difficulty profile across budgets.

Properties. Monotonicity: q^k1≥q^k2\hat{q}_{k_1}\geq\hat{q}_{k_2} for k1≤k2k_1\leq k_2, so q^\hat{\mathbf{q}} forms a decreasing staircase marking where the policy transitions from uncertain to confident. Sensitivity: as πθ\pi_\theta improves, q^k\hat{q}_k decreases for small kk, making calibrated thresholds a live signal of policy improvement that justifies periodic re-calibration (§ 4.4.3).

C-GRPO: Conformal-Guided Adaptive Training

Adaptive Group Construction via Conformal Confidence

C-GRPO resolves both failure modes simultaneously by anchoring group size to a conformal confidence criterion: stop sampling as soon as the empirical answer distribution is certifiably concentrated on a single answer. Formally, the APS prediction set at budget kk is the score-threshold set

Ck(x)={a∈A:sAPS(a;a^1:k)≤q^k},(8)C_k(x)=\left\{a\in\mathcal{A}:s_{\mathrm{APS}}(a;\hat{a}_{1:k})\le\hat{q}_k\right\}, \tag*{(8)}

where sAPS(a;⋅)s_{\mathrm{APS}}(a;\cdot) is Eq. (4) evaluated at a candidate answer aa in place of y⋆y^\star. By construction y⋆∈Ck(x)y^\star\in C_k(x) if and only if sAPS(y⋆;a^1:k)≤q^ks_{\mathrm{APS}}(y^\star;\hat{a}_{1:k})\le\hat{q}_k, which is exactly the event controlled by Theorem 2. Equivalently, the set lists answers in decreasing empirical frequency until the cumulative mass reaches q^k\hat{q}_k; unseen answers have score 11 and enter only when q^k=1\hat{q}_k=1. The implementation enumerates the observed answers, which coincides with Eq. (8) for q^k<1\hat{q}_k<1, and evaluates membership deterministically (U=0U=0, ties ordered at random). The model is confident at budget kk when ∣Ck(x)∣=1\lvert C_k(x)\rvert=1. The conformal group size is:

kused(x)=min⁡{k∈K:∣Ck(x)∣=1},(9)k_{\mathrm{used}}(x)=\min\{k\in\mathcal{K}:\lvert C_k(x)\rvert=1\}, \tag*{(9)}

defaulting to Kmax⁡K_{\max} if no singleton forms within K\mathcal{K}.

Singletons resolve both failure modes: on easy queries a singleton forms at k1k_1, providing a cheap non-zero-variance group; on hard queries the model expands to larger kk, increasing the chance of observing a positive reward.

Two-phase generation. To avoid generating Kmax⁡K_{\max} completions upfront: (1) probe with k1k_1 samples—if a singleton forms, commit; (2) expand non-singleton queries via parallel micro-batching to the next grid points, halting at the first singleton or Kmax⁡K_{\max}. This preserves high GPU utilization while still sparing all forward-pass compute for easy k1k_1 queries. Zero-variance groups (Var⁡[rj]=0\operatorname{Var}[r_j]=0) are pruned before the backward pass.

GRPO Objective with Adaptive Groups

Let y(x)=(y1,…,ykused)\mathbf{y}^{(x)}=(y_1,\ldots,y_{k_{\mathrm{used}}}) be the adaptively-sized group. Because group sizes vary across queries, we normalize advantages across the full batch B\mathcal{B} to prevent large groups from dominating gradient scale:

Ai=(ri−rˉg(i))−μA^σA^+ϵs,(10)A_i=\frac{(r_i-\bar{r}_{g(i)})-\mu_{\hat{A}}}{\sigma_{\hat{A}}+\epsilon_s}, \tag*{(10)}

where g(i)g(i) is the query-group index of sample ii, rˉg(i)\bar{r}_{g(i)} is the mean reward within that group, μA^\mu_{\hat{A}} and σA^\sigma_{\hat{A}} are the batch-level mean and standard deviation of raw advantages (ri−rˉg(i))(r_i-\bar{r}_{g(i)}), and ϵs>0\epsilon_s>0 is a small numerical stabilizer (distinct from the safety margin ϵ\epsilon in Eq. (14)).

The C-GRPO loss is:

LC−GRPO(θ)=−1∣B∣∑i∈BAi⋅log⁡πθ(yi∣xi)+β⋅KL^(θ),(11)\mathcal{L}_{\mathrm{C-GRPO}}(\theta)=-\frac{1}{\lvert\mathcal{B}\rvert}\sum_{i\in\mathcal{B}}A_i\cdot\log\pi_\theta(y_i\mid x_i)+\beta\cdot\widehat{\mathrm{KL}}(\theta), \tag*{(11)}

where log⁡πθ(y∣x)=∑tlog⁡πθ(ℓt∣x,ℓ<t)\log\pi_\theta(y\mid x)=\sum_t\log\pi_\theta(\ell_t\mid x,\ell_{<t}) sums over completion tokens only, and β>0\beta>0 is the Kullback-Leibler (KL) coefficient computed on the per-token estimator KL^\widehat{\mathrm{KL}} of Schulman et al. [ [22] (Appendix § 17). Because samples are drawn fresh from πθ\pi_\theta each step, no importance-sampling correction is needed; the gradient is an unbiased REINFORCE [26] estimator over the adaptive group.

Reward signal. Verifier-only binary reward ri=v(f(yi),y⋆)r_i=v(f(y_i),y^\star); degenerate completions (<5<5 tokens or <8<8 unique characters) receive ri=−1r_i=-1 to suppress mode collapse.

Conformal Curriculum via Periodic Re-Calibration

The training dynamics and calibration are mutually reinforcing. As πθ\pi_\theta improves, the hard rate ρt=Ex[1[kused>k1]]\rho_t=\mathbb{E}_x[1[k_{\mathrm{used}}>k_1]] decreases naturally, reducing average compute per step. Every Δcal\Delta_{\mathrm{cal}} gradient steps we re-run calibration with the current policy:

q^k(t)←q^({s(yi⋆;y^i,1:k(πθt))}i=1n,δ),k∈K.(12)\hat{q}_k^{(t)}\leftarrow\hat{q}\left(\left\{s(y_i^\star;\hat{y}_{i,1:k}^{(\pi_{\theta_t})})\right\}_{i=1}^{n},\delta\right),\qquad k\in\mathcal{K}. \tag*{(12)}

Here y^i,1:k(πθt)\hat{y}_{i,1:k}^{(\pi_{\theta_t})} denotes kk fresh completions sampled from the current trained policy πθt\pi_{\theta_t} for calibration query xix_i. As the policy improves, q^k(t)\hat{q}_k^{(t)} falls for small kk, the singleton criterion fires earlier for more queries, and the curriculum automatically de-emphasises easy problems—without any manually designed annealing schedule.

Adaptive Miscoverage Level via Coverage-Calibrated Selection. Standard split-conformal prediction treats the miscoverage level δ\delta as a fixed hyperparameter set prior to calibration. In the online training setting, however, fixing δ\delta creates a fundamental tension: a stringent target (e.g., δ=0.10\delta= 0.10 demanding 90% coverage) is unachievable when the base model rarely produces correct answers at full budget Kmax⁡K_{\max}, forcing q^k=1\hat{q}_k = 1 universally and completely disabling adaptive sampling. Conversely, a permissive δ\delta yields meaninglessly loose thresholds.

The root constraint is that the empirical solve rate at maximum budget,

ps@Kmax⁡^(π)=1n∑i=1n1[∃j≤Kmax⁡:s(yi⋆,y^i,1:Kmax⁡)<1],(13)\widehat{\mathrm{ps}@K_{\max}}(\pi) = \frac{1}{n}\sum_{i=1}^{n}\mathbb{1}\left[\exists j \le K_{\max}:s(y_i^\star,\hat{y}_{i,1:K_{\max}}) < 1\right], \tag*{(13)}

is a hard upper bound on achievable marginal coverage: no threshold q^k\hat{q}_k can yield P(stest≤q^k)≥1−δP(s_{\mathrm{test}} \le\hat{q}_k) \ge1-\delta when the model fails within Kmax⁡K_{\max} samples on more than δn\delta n calibration examples.

We therefore propose coverage-calibrated δ\delta selection: set the miscoverage level as a direct function of the current policy’s maximum-budget capability,

δauto(π)=clip⁡((1−ps@Kmax⁡^(π))+ε,δmin⁡,δmax⁡),(14)\delta_{\mathrm{auto}}(\pi)=\operatorname{clip}\left((1-\widehat{\mathrm{ps}@K_{\max}}(\pi))+\varepsilon,\delta_{\min},\delta_{\max}\right), \tag*{(14)}

where ε>0\varepsilon>0 is a safety margin ensuring q^kM\hat{q}_{k_M} sits meaningfully below 1 (we use ε=0.05\varepsilon=0.05), δmin⁡=0.05\delta_{\min}=0.05 guards against over-claiming when the policy is strong, and δmax⁡=0.95\delta_{\max}=0.95 prevents degenerate guarantees for very weak base models.

As training progresses δauto\delta_{\mathrm{auto}} falls from ≈0.65\approx0.65 (solve rate ≈0.4\approx0.4: loose thresholds, near-full budget, maximal exploration) through ≈0.35\approx0.35 (≈0.7\approx0.7: thresholds tighten for easy queries) to ≈0.15\approx0.15 (≈0.9\approx0.9: easy queries exit at k1k_1).

Joint update. At each re-calibration checkpoint, both δ\delta and q^\hat{q} are updated with the current trained policy πθ\pi_{\theta} (LoRA [13] adapters enabled). Using the frozen reference model would yield a stagnant solve rate and q^k=1\hat{q}_k=1 throughout training.

Disjoint split for δ\delta and q^\hat{q}. Computing δauto\delta_{\mathrm{auto}} and the quantiles from the same calibration scores makes the miscoverage level data-dependent. We therefore split the calibration set into disjoint parts: Dδ\mathcal{D}_{\delta} sets δ\delta (Eqs. (13)–(14)) and Dq\mathcal{D}_{q} computes the quantiles. Conditional on Dδ\mathcal{D}_{\delta}, δ\delta is constant and Dq\mathcal{D}_{q} is fresh, so Theorem 2 applies exactly with n=∣Dq∣n=|\mathcal{D}_{q}| (tolerance 1/201→1/1011/201 \to1/101 for ncal=200n_{\mathrm{cal}}=200); this is the recommended protocol (flag --split-delta), whereas the reported numbers use the original one (§ 5.1); offline, the split changed the budget of fewer than 4% of prompts.

Full training algorithm. Algorithm 1 summarizes C-GRPO, including auto-δ\delta initialization, adaptive rollout allocation, GRPO updates, and periodic re-calibration.

Theoretical Analysis

Coverage Guarantee

Theorem 2 (Marginal Coverage). Let (x1,y1⋆),…,(xn,yn⋆),(xn+1,yn+1⋆)(x_1,y_1^\star),\ldots,(x_n,y_n^\star),(x_{n+1},y_{n+1}^\star) be i.i.d. from any distribution PP over X×A\mathcal{X}\times\mathcal{A}. Fix δ∈(0,1)\delta\in(0,1), k∈Kk\in\mathcal{K}, and any fixed policy πθ\pi_{\theta}. Let q^k\hat{q}_k be given by Eq. (7) using s∈{sAPS,scode}s\in\{s_{\mathrm{APS}},s_{\mathrm{code}}\}. Then:

P(s(yn+1⋆;y^n+1,1:k)≤q^k)∈[1−δ, 1−δ+1n+1].(15)P\left(s(y_{n+1}^\star;\hat{y}_{n+1,1:k})\le\hat{q}_k\right)\in\left[1-\delta,\ 1-\delta+\frac{1}{n+1}\right]. \tag*{(15)}

Proof sketch. I.i.d. examples make the augmented score sequence exchangeable. Consequently, rank⁡(sn+1)\operatorname{rank}(s_{n+1}) is uniform on {1,…,n+1}\{1,\ldots,n+1\}. Choosing the LL-th order statistic therefore yields coverage L/(n+1)∈[1−δ,1−δ+1/(n+1)]L/(n+1)\in[1-\delta,1-\delta+1/(n+1)]. Appendix § 11 gives the full argument for both scores, and Appendix § 12 collects the exchangeability lemmas. □\square

Remark 3. The guarantee is distribution-free and holds for any πθ\pi_{\theta}; training only improves efficiency by lowering q^k\hat{q}_k. Under covariate shift coverage degrades by at most ϵshift\epsilon_{\mathrm{shift}} per unit TV distance (Theorem 14, Appendix § 13).

Algorithm 1 C-GRPO Training with Adaptive δ\delta

Require: Pre-trained reference πref\pi_{\mathrm{ref}}; datasets DtrainD_{\mathrm{train}} and disjoint calibration splits Dδ,DqD_{\delta},D_q; budget grid K={k1<⋯<kM}\mathcal{K}=\{k_1<\cdots<k_M\}; safety margin ϵ=0.05\epsilon=0.05; ceilings δmin⁡=0.05,δmax⁡=0.95\delta_{\min}=0.05,\delta_{\max}=0.95; KL coefficient β\beta; re-calibration period Δcal\Delta_{\mathrm{cal}}

Ensure: Fine-tuned policy πθ\pi_{\theta}; thresholds q^(T)\hat{q}^{(T)}; miscoverage δ(T)\delta^{(T)}

  1. Initialize πθ←πref\pi_{\theta}\leftarrow\pi_{\mathrm{ref}} (LoRA adapter)

  2. δ(0)←δauto(πref;Dδ)\delta^{(0)}\leftarrow\delta_{\mathrm{auto}}(\pi_{\mathrm{ref}};D_{\delta}) ▷ Eq. (14) with reference model solve rate

  3. q^(0)←CALIBRATE⁡(πref,Dq,K,δ(0))\hat{q}^{(0)}\leftarrow\operatorname{CALIBRATE}(\pi_{\mathrm{ref}},D_q,\mathcal{K},\delta^{(0)}) ▷ Bootstrap thresholds from reference model

  4. for step t=1,2,…,Tt=1,2,\ldots,T do

  5. Sample mini-batch {(xi,yi⋆)}i=1B\{(x_i,y_i^\star)\}_{i=1}^{B}

  6. for each query xix_i in parallel do

  7. Probe: sample k1k_1 completions from πθ(⋅∣xi)\pi_{\theta}(\cdot\mid x_i)

  8. if ∣Ck1(xi)∣=1\lvert C_{k_1}(x_i)\rvert=1 under q^k1(t−1)\hat{q}_{k_1}^{(t-1)} then ▷ Confident at minimal budget

  9. kused(i)←k1k_{\mathrm{used}}^{(i)}\leftarrow k_1

  10. else

  11. Expand: sample additional completions via parallel micro-batching

  12. kused(i)←min⁡{k∈K:∣Ck(xi)∣=1}k_{\mathrm{used}}^{(i)}\leftarrow\min\{k\in\mathcal{K}:\lvert C_k(x_i)\rvert=1\}, default kMk_M

  13. end if

  14. Compute rewards rj=v(f(y^j),yi⋆)r_j=v(f(\hat{y}_j),y_i^\star), j≤kused(i)j\leq k_{\mathrm{used}}^{(i)}

  15. if Var⁡[rj]=0\operatorname{Var}[r_j]=0 then skip (zero-gradient group pruning)

  16. end if

  17. end for

  18. Compute batch-normalized advantages {Ai}\{A_i\} via Eq. (10)

  19. Update θ←θ−η∇θLC-GRPO(θ)\theta\leftarrow\theta-\eta\nabla_{\theta}\mathcal{L}_{\mathrm{C\text{-}GRPO}}(\theta) (Eq. (11))

  20. if t mod Δcal=0t\bmod\Delta_{\mathrm{cal}}=0 then

  21. δ(t)←δauto(πθ;Dδ)\delta^{(t)}\leftarrow\delta_{\mathrm{auto}}(\pi_{\theta};D_{\delta}) ▷ Update coverage target with trained policy (LoRA ON)

  22. q^(t)←CALIBRATE⁡(πθ,Dq,K,δ(t))\hat{q}^{(t)}\leftarrow\operatorname{CALIBRATE}(\pi_{\theta},D_q,\mathcal{K},\delta^{(t)}) ▷ Joint (δ(t),q^(t))(\delta^{(t)},\hat{q}^{(t)}) update

  23. end if

  24. end for

  25. return πθ,q^(T),δ(T)\pi_{\theta},\hat{q}^{(T)},\delta^{(T)}

Corollary 4 (False-stop control). Under the assumptions of Theorem 2, for every fixed k∈Kk\in\mathcal{K} and the APS score, P(∣Ck(xn+1)∣=1,Ck(xn+1)≠{yn+1⋆})≤δP(\lvert C_k(x_{n+1})\rvert=1,C_k(x_{n+1})\ne\{y_{n+1}^\star\})\leq\delta: a singleton formed at budget kk is the gold answer except on an event of probability at most δ\delta. For the first-success score with q^k<1\hat{q}_k<1, stopping requires an executed, verified success among the first kk samples, so a stop never commits to a failing solution.

Theorem 2 does not predict when a singleton forms (that is governed by Proposition 6), and since kusedk_{\mathrm{used}} is chosen adaptively the guarantee for the selected budget needs multiplicity control:

Theorem 5 (Coverage of the selected budget). Calibrate every threshold at level δ/M\delta/M, M=∣K∣M=\lvert\mathcal{K}\rvert. Then P(∀k∈K:yn+1⋆∈Ck(xn+1))≥1−δP(\forall k\in\mathcal{K}:y_{n+1}^\star\in C_k(x_{n+1}))\geq1-\delta. Hence for any data-dependent budget k^\hat{k}, in particular k^=kused\hat{k}=k_{\mathrm{used}}, P(yn+1⋆∉Ck^)≤δP(y_{n+1}^\star\notin C_{\hat{k}})\leq\delta and the rule commits to a wrong singleton with probability at most δ\delta. At level δ\delta per budget (as in our experiments) the bound is MδM\delta; across TcT_c checkpoints sharing calibration prompts, level δ/(MTc)\delta/(MT_c) gives the guarantee simultaneously.

Proofs are in Appendix § 11, following conformal risk control over parameter grids [1, 3]. These guarantees are marginal over prompts: they bound the fraction of prompts truncated on a wrong answer, not P(correct∣stop)P(\text{correct}\mid\text{stop}), which would need selective conformal methods [9]. A wrong singleton ships no answer at training time and, having zero reward variance, is pruned (§ 4.4.1).

Monotone Efficiency of the Conformal Curriculum

Proposition 6 (Monotone Training Budget). Let π0,π1\pi_0,\pi_1 be two policies, and let p^k(m)(y⋆∣x)=k−1∑j=1k1[f(y^j)=y⋆]\hat{p}_k^{(m)}(y^\star\mid x)=k^{-1}\sum_{j=1}^{k}\mathbf{1}[f(\hat{y}_j)=y^\star] for y^j∼πm(⋅∣x)\hat{y}_j\sim\pi_m(\cdot\mid x) denote the empirical pass rate at budget kk under policy πm\pi_m. Suppose Ex[p^k(1)(y⋆∣x)]≥Ex[p^k(0)(y⋆∣x)]\mathbb{E}_x[\hat{p}_k^{(1)}(y^\star\mid x)]\geq\mathbb{E}_x[\hat{p}_k^{(0)}(y^\star\mid x)] for all k∈Kk\in\mathcal{K} (i.e., π1\pi_1 is uniformly more accurate than π0\pi_0). Let* q^k(m)\hat{q}_k^{(m)} be the conformal threshold calibrated from πm\pi_m. Then in expectation over the calibration set:

q^k(1)≤q^k(0),(16)\hat{q}_k^{(1)}\leq\hat{q}_k^{(0)}, \tag*{(16)}

and consequently

Ex[kused(x;π1)]≤Ex[kused(x;π0)].(17)\mathbb{E}_{x}\left[k_{\mathrm{used}}(x;\pi_{1})\right]\leq\mathbb{E}_{x}\left[k_{\mathrm{used}}(x;\pi_{0})\right]. \tag*{(17)}

Proof sketch. Higher pass rate stochastically dominates the score CDF for both sAPSs_{\mathrm{APS}} (less mass above y∗y^{*}) and scodes_{\mathrm{code}} (j∗j^{*} stochastically smaller), lowering the order-statistic threshold and widening the singleton-forming set. Full proof with budget-grid telescoping and a finite-sample savings lower bound in Appendix § 11, Appendix § 15.

Corollary 7 (Self-Annealing Training Cost). At every re-calibration checkpoint at which the conditions of Proposition 6 hold between consecutive policies, periodic re-calibration every Δcal\Delta_{\mathrm{cal}} steps produces a non-increasing sequence of expected per-step training costs:

E[kused(t+Δcal)]≤E[kused(t)],∀ t.(18)\mathbb{E}\left[k_{\mathrm{used}}^{(t+\Delta_{\mathrm{cal}})}\right]\leq\mathbb{E}\left[k_{\mathrm{used}}^{(t)}\right],\qquad\forall\,t. \tag*{(18)}

Where the conditions hold, C-GRPO therefore schedules less compute per step as the model becomes more capable, with no human-designed annealing. (Policy-drift and staleness bounds: Appendix § 14.)

Remark 8 (An idealized tendency, not a guarantee). Proposition 6 assumes uniform improvement, which RL violates locally: if accuracy on a slice degrades, its threshold rises at the next checkpoint and budget is restored (Appendix § 10). We read the decrease as the aggregate behaviour in Figure 3.

Figure 3. Rollout efficiency during training (GSM8K, Qwen2.5-7B, 300 steps). (a) Mean kˉ\bar{k} per step falls from Kmax⁡=32K_{\max}=32 to ≈10\approx10; drops at each Δcal=50\Delta_{\mathrm{cal}}=50 checkpoint mark auto-δ\delta tightening. (b) Cumulative rollouts: 59.8% fewer than the fixed-k=32k=32 ceiling (dashed; shaded: 5 738 rollouts saved).

Experiments

Experimental Setup

Datasets. Six benchmarks: code (HumanEval [4], MBPP [2]), math (GSM8K [6], MATH-500 [15], MATH-Algebra [12]), science (GPQA-Diamond [20]). Each uses a sequestered calibration split (ncal=200n_{\mathrm{cal}}=200) and a disjoint sealed test split (ntest=200n_{\mathrm{test}}=200–400400). Calibration protocol: all numbers use the original procedure (one calibration set for δauto\delta_{\mathrm{auto}} and the quantiles, no multiplicity correction); the disjoint split of § 4.4.3 is recommended.

Models & training. LoRA [13] (r=16r=16, α=32\alpha=32, 5% dropout, AdamW [16] η=2×10−6\eta=2\times10^{-6}, batch 4, 300 steps, 2,000 training samples per benchmark) applied to Qwen2.5-7B-Instruct [19], Llama-3.1-8B-Instruct [10], and Gemma-3-4B-IT [8]. All baselines use the identical config; differences are in training algorithm only. LoRA is used for hardware reasons (one full-parameter run: § 5.5).

Baselines. Static GRPO [23] (G=32G=32); dynamic AERO [28], GDRO [17], GRESO [29], Reinforce-Ada/Est [27].

C-GRPO config. Budget grid K={2,4,8,16,32}\mathcal{K}=\{2,4,8,16,32\}; re-calibration period Δcal=50\Delta_{\mathrm{cal}}=50 steps; ε=0.05\varepsilon=0.05; β=0.05\beta=0.05. δ(0)\delta^{(0)} is set automatically via Eq. (14)—zero per-dataset or per-model tuning.

Evaluation. p@k\mathbf{p}@k: fixed-budget accuracy at exactly kk rollouts (majority vote for math/science, best-of-kk for code). p@1\mathbf{p}@1: Ave@32 [27] single-rollout accuracy at T=1T=1 (not greedy decoding). kˉtrain\bar{k}_{\mathrm{train}}: mean rollouts/prompt at step 300 (proxy for per-step training FLOPs). Rollout savings =1−kˉtrain/32=1-\bar{k}_{\mathrm{train}}/32 counts training rollouts only; evaluation rollouts are charged separately in Table 10. At ntest=200n_{\mathrm{test}}=200–400400 the binomial standard error near 57% is 2.5–3.5 pp, so differences of a few points are within noise and we read accuracy results as matched, not improved.

Phase (hours)GRPO G=32G=32GRPO G=10G=10C-GRPO (kˉ=10.4\bar{k}=10.4)
Training generation8.102.552.80
Forward + backward3.601.650.75
Re-calibration (6 events)——0.25
Micro-batch scheduling——0.008
Total wall-clock11.704.203.81
Speed-up vs. G=32G=321.00×2.79×3.07×

Table 10. Wall-clock breakdown (GSM8K, Qwen2.5-7B-Instruct, 300 steps, 1×H100; original calibration procedure). C-GRPO includes all calibration generation, batched as large offline batches; with the released code’s default (training-batch-size) calibration the re-calibration share is larger (36–47% of training time in a 100-step check).

Code. Code (training, calibration, evaluation, baselines, launch scripts) is at https://github.com/AryaFayyazi/CGRPO (MIT license); pip install trl-cgrpo provides a TRL GRPOTrainer subpackage.

Main Results

Table 1 shows 49–70% training-rollout reduction across six benchmarks with equal or better accuracy at every fixed budget. Savings are highest where difficulty is bimodal (GPQA-Diamond 70%, GSM8K 68%); MATH-500 (50%) is uniformly harder, filling more of the grid. HumanEval retains kˉ=16.3\bar{k}=16.3 for diverse solutions yet still reaches p@16=95.5%\mathrm{p}@16=95.5\%.

Table 1: C-GRPO on six benchmarks (Qwen2.5-7B-Instruct, step 300). Fixed-budget accuracy p@kp@k, mean training rollouts kˉtrain\bar{k}_{\mathrm{train}}, and rollout savings 1−kˉtrain/321-\bar{k}_{\mathrm{train}}/32 vs. GRPOG=32_{G=32} (training rollouts only; calibration cost in Table 10).

Datasetp@1p@8p@16p@32kˉtrain\bar{k}_{\mathrm{train}}Rollout sav.GPU-hVRAM (GB)Tok/s
Mathematical reasoning
GSM8K57.270.573.875.010.468%4.0415.6450
MATH-50052.566.871.271.515.950%1.6121.61,880
MATH-Algebra61.269.572.074.012.262%8.2416.6442
Graduate-level science
GPQA-Diamond32.835.538.439.09.770%0.6526.61,826
Code generation
HumanEval36.578.295.597.216.349%0.2519.81,931
MBPP60.578.583.083.510.069%1.6016.6453

Table 1.

Comparison with Baselines

C-GRPO attains the highest pass@1 on all four benchmarks while using the fewest rollouts, though its margins over the strongest baselines (0.4–2.4 pp) are within binomial noise. It averages kˉ=10.4\bar{k}=10.4, versus at least 14.8 for dynamic baselines and 32.0 for static GRPO, as shown in Table 2. Heuristic methods reduce cost but lack coverage guarantees, and GDRO improves accuracy under a compute-neutral constraint while still trailing C-GRPO on every metric. Together, these results show that conformal allocation matches or slightly improves accuracy while substantially reducing rollouts.

Table 2: Pass@1 (%) and training compute (Qwen2.5-7B-Instruct, step 300). kˉtrain\bar{k}_{\mathrm{train}}: mean rollouts/prompt (lower is better). Δ\Delta: pass@1 gap vs. GRPO on GSM8K. GPU-h / VRAM / Tok/s: total training wall-clock (including all calibration generation for C-GRPO), peak GPU memory, and effective rollout generation throughput on 1×\timesH100 (all methods, 300 steps). ∗^*GDRO uses a compute-neutral global budget (kˉ=32\bar{k}=32).

ModelAlgorithmGSM8KHumanEvalMBPPMATH-500Δ\Deltapass@1kˉtrain\bar{k}_{\mathrm{train}}TypeGPU-hVRAM (GB)Tok/s
Qwen2.5 7B-InstructGRPO [23]54.833.356.048.5—32.0static11.726.83,180
GDRO [17]56.834.558.251.0+2.032.0∗^*dynamic11.726.83,180
AERO [28]56.134.057.550.2+1.316.6dynamic6.122.44,270
GRESO [29]54.733.155.848.3−0.115.6dynamic5.721.94,410
Reinforce-Ada [27]55.433.856.549.5+0.614.8dynamic5.421.44,520
Reinforce-Est [27]55.133.556.149.0+0.315.2dynamic5.621.64,460
C-GRPO (ours)57.236.560.552.5+2.410.4conformal3.818.65,240

Table 2.

On GSM8K, C-GRPO obtains +2.4 pp over GRPO and +0.4 pp over GDRO with only one-third as many rollouts (kˉ=10.4\bar{k}=10.4 versus 32.0); these differences are within the 2.5–3.5 pp standard error of a single run, so the result is matched accuracy at about a third of the rollouts (Figure 3).

Cross-Model Generalisation

Table 3 shows zero-shot transfer across architectures. The calibration split determines δ(0)\delta^{(0)} separately for each model–dataset pair. On GSM8K, Llama-3.1-8B attains 82% rollout savings with kˉ=5.7\bar{k}=5.7 and p@16p@16 of 91.0% because its stronger baseline at kmax⁡k_{\max} lets the procedure tighten δ(0)\delta^{(0)} to 0.05 automatically. Gemma-3-4B instead receives a looser δ(0)∈[0.18,0.30]\delta^{(0)}\in[0.18,0.30], reflecting its weaker baseline. Efficiency therefore scales with model strength without manual retuning.

DatasetModelδ(0)\delta^{(0)}p@1p@8p@16p@32kˉtrain\bar{k}_{\mathrm{train}}Savings
GSM8KQwen2.5-7B-Inst.0.1257.270.573.875.010.468%
Llama-3.1-8B-Inst.0.0572.585.091.091.55.782%
Gemma-3-4B-IT0.1848.562.065.566.014.256%
MBPPQwen2.5-7B-Inst.0.1060.578.583.083.510.069%
Llama-3.1-8B-Inst.0.0865.581.086.587.08.573%
Gemma-3-4B-IT0.1550.265.070.071.012.561%
MATH-500Qwen2.5-7B-Inst.0.2552.566.871.271.515.950%
Llama-3.1-8B-Inst.0.1560.072.578.078.512.062%
Gemma-3-4B-IT0.3042.555.060.060.518.542%

Table 3. Cross-model generalisation at step 300 on held-out test sets. δ(0)\delta^{(0)} is auto-selected per model–dataset pair without manual tuning.

Matched-Compute Baselines and End-to-End Cost

A G=32G=32 baseline may simply be larger than necessary, so we train GRPO at fixed G∈{2,…,128}G\in\{2,\ldots,128\} under an otherwise identical pipeline (full sweep: Table 7, Appendix § 16). On GSM8K, static pass@1 is 52.8 (G=8G=8), 53.1 (G=10G=10, matched to C-GRPO’s kˉ=10.4\bar{k}=10.4), 54.2 (G=32G=32), and 54.8 (G=32G=32), saturating at 55.3 for G=128G=128 (44.5 GPU-h), against 57.2 for C-GRPO (3.8 GPU-h); on MATH-500, matched G=16G=16 reaches 47.9 against 52.5. These 4.1 and 4.6 pp gaps are of the order of the single-run standard error, so we read them as at-least-matched accuracy at equal compute, not a significant gain; C-GRPO finds the matched GG (10 versus 16) from calibration alone.

BenchmarkMethodG/kˉG / \bar{k}p@1p@8p@16p@32GPU-h
GSM8KGRPO248.264.067.568.01.8
GSM8KGRPO451.068.270.171.22.4
GSM8KGRPO852.870.172.073.13.6
GSM8KGRPO10 (matched)53.170.572.473.54.2
GSM8KGRPO1654.271.873.274.16.2
GSM8KGRPO3254.872.073.574.611.7
GSM8KGRPO6455.172.273.874.822.8
GSM8KGRPO12855.372.474.075.044.5
GSM8KC-GRPO10.457.270.573.875.03.8
MATH-500GRPO241.555.258.059.11.9
MATH-500GRPO845.860.163.464.04.1
MATH-500GRPO16 (matched)47.962.566.066.88.1
MATH-500GRPO3248.563.267.067.815.9
MATH-500GRPO12849.264.068.168.961.2
MATH-500C-GRPO15.952.566.871.271.55.1

Table 7. Full static-GG sweep (Qwen2.5-7B-Instruct, 300 steps, 1×H100). Every static row is trained at fixed GG.

End-to-end cost. Rollout savings count training rollouts only. Table 10 (Appendix § 16) charges every calibration completion to C-GRPO (6 ×\times 200 ×\times 32 = 38,400 completions). This cost depends strongly on batching: as large offline batches without gradient state it was 0.25 h (6.6% of wall-clock) in our setup, whereas the released code, which calibrates at the training batch size, spent 46–47% of training time in a 100-step check, so wall-clock savings there are smaller than the rollout savings. Zero-variance pruning cuts forward–backward time from 1.65 h to 0.75 h versus static G=10G=10; in Table 10’s setting the net speed-up over G=32G=32 is 3.07×.

Longer horizon and a stronger base (Appendix § 16). Over 1,000 GSM8K steps kˉ\bar{k} falls 18.2 → 4.2 and pass@1 rises 52.1 → 64.5 (static: 59.1). Full-parameter fine-tuning of Qwen2.5-Math-7B-Instruct (3 seeds): C-GRPO 88.9±0.2 at kˉ=5.8\bar{k}=5.8 (82% savings) versus 88.5±0.3 for static G=32G=32 (matched, not improved) and 86.1±0.4 for G=8G=8.

Training Dynamics

Reward dips align with the Δcal=50\Delta_{\mathrm{cal}}=50 checkpoints (Figure 2) and recover within ∼5\sim5 steps; the cumulative budget grows sub-linearly (Figure 3), consistent with the idealized tendency of Corollary 7. Under the original procedure the worst under-coverage over all (budget, checkpoint) pairs is −0.2 pp (mean slack +0.3 pp), inside the tolerance 1/(ncal+1)≈0.5%1/(n_{\mathrm{cal}}+1)\approx0.5\% (Appendix § 8). Ablations: Appendix § 9; shift: Appendix § 13.

Training dynamics

Figure 2. Training dynamics (GSM8K, Qwen2.5-7B). C-GRPO sustains the highest reward and lowest KL divergence throughout training. Periodic dips align with Δcal=50\Delta_{\mathrm{cal}}=50 re-calibration events; the policy recovers to a higher reward peak within ∼5\sim5 steps each time.

Discussion and Limitations

Theory versus the deployed loop. Theorem 2 assumes a fixed policy and a fresh calibration split, whereas the loop reuses calibration prompts and selects δ\delta adaptively; the disjoint split and Theorem 5 restore exactness, but reported numbers use the original procedure. Scope. A verifier is required, coverage needs exchangeability (Theorem 14), and most results use LoRA at 4–8B scale and 300 steps (longer and full-parameter runs: one model each).

Conclusion

We presented C-GRPO, a conformal extension of GRPO that replaces fixed rollout budgets with calibrated, task-specific stopping thresholds (task-aware scores, automatic miscoverage selection, periodic re-calibration), with finite-sample coverage and false-stop control. Across six benchmarks, C-GRPO reduces training rollouts by 49–70% at matched accuracy, also against fixed-GG GRPO at matched compute, and transfers across model families (up to 82% savings). Accuracy gains are within noise: the contribution is iso-accuracy compute reduction.

Acknowledgements and Disclosures

Funding. This work received no external funding or third-party support during the last 36 months. Competing interests. The authors declare no competing interests: they have no financial relationships outside the submitted work that could be perceived as influencing it.

Coverage Validation

Theorem 2 guarantees that the empirical conformal coverage (1−δ(t))(1-\delta^{(t)}) meets the nominal target at every re-calibration checkpoint. Table 4 confirms this across all six benchmarks on the held-out test split; Figure 4 shows the per-(k,t)(k,t) scatter on GSM8K.

Datasetδ(0)\delta^{(0)}TargetEmpiricalSlack
GSM8K0.1284.0%83.8%−0.2 pp
MATH-5000.2574.0%74.7%+0.7 pp
GPQA-Diamond0.4844.0%43.9%−0.1 pp
MATH-Algebra0.2180.0%80.0%±\pm0 pp
HumanEval0.9090.0%90.9%+0.9 pp
MBPP0.2376.0%76.3%+0.3 pp

Table 4. Coverage validation across six benchmarks (Qwen2.5-7B, held-out test, step 300). δ(0)\delta^{(0)}: auto-selected initial miscoverage. Empirical coverage matches or exceeds the nominal target on all benchmarks within the finite-sample tolerance ±1/(ncal+1)≈0.5%\pm1/(n_{\mathrm{cal}}+1)\approx0.5\%.

Figure 4. Per-(k,t)(k,t) coverage scatter on GSM8K (Qwen2.5-7B). Each point is one (k,step)(k,\mathrm{step}) pair. Every point lies on or above the diagonal (empirical ≥\geq target), jointly validating Theorem 2 for all k∈Kk \in\mathcal{K} and all training steps simultaneously. The densest cluster at 80–90% corresponds to k=8k=8–16 during mid-training; points move rightward as auto-δ\delta tightens targets with policy improvement.

All coverages lie within ±0.9\pm0.9 pp of nominal, respecting the 1/(ncal+1)1/(n_{\mathrm{cal}}+1) finite-sample bound from Theorem 2; individual per-kk coverages all satisfy the theorem. These numbers use the original procedure. Because auto-δ\delta tightens as the solve rate rises, coverage must be read against the target at the same checkpoint (the nominal target moves, which is what the rightward drift of the points in Figure 4 shows), so we report slack, empirical minus nominal: across all (budget, checkpoint) pairs the worst under-coverage is −0.2 pp (GSM8K, step 300), the mean slack over the six benchmarks is +0.3 pp, and the largest deviation in either direction is +0.9 pp, which is over-coverage and therefore conservative. Only under-coverage threatens validity, and −0.2 pp is inside the finite-sample tolerance 1/(ncal+1)≈0.5%1/(n_{\mathrm{cal}}+1)\approx0.5\% and the ±1.1\pm1.1 pp binomial noise at ntest=400n_{\mathrm{test}}=400.

Ablation Studies

Table 5 ablates the three design choices unique to C-GRPO on Qwen2.5-7B, 300 steps, held-out test: (A) auto-δ\delta initialisation, (B) re-calibration frequency Δcal\Delta_{\mathrm{cal}}, and (C) the discrete budget grid K\mathcal{K}.

Strategyp@1p@16kˉ\bar{k}Cov.Triv.Δ\Deltap@1p@16kˉ\bar{k}OHK\mathcal{K}p@1p@16kˉ\bar{k}#
Fixed .0554.570.229.594.2∼\sim2002557.373.910.3+12{2,8,32}\{2, 8, 32\}51.069.518.43
Fixed .2056.072.518.481.5∼\sim305057.273.810.4base{2,4,8,16,32}\{2, 4, 8, 16, 32\}52.571.215.95
Auto57.273.810.483.8010055.872.113.5−6{1,2,4,8,16,32}\{1, 2, 4, 8, 16, 32\}52.370.816.26
∞\infty54.570.018.2−12

Table 5. Ablation studies for C-GRPO. Bold = default configuration. “Triv.” = training steps where all thresholds are inactive (curriculum reduces to vanilla GRPO). “OH” = overhead steps per checkpoint relative to Δcal=50\Delta_{\mathrm{cal}}=50 baseline. “#” = number of distinct budget levels. All rows use the original calibration procedure.

(A) Auto-δ\delta initialisation. Fixed δ=0.05\delta=0.05 requires 95% coverage within Kmax⁡K_{\max} rollouts before the curriculum activates. An initialised Qwen2.5-7B solves only ≈40%\approx40\% of GSM8K with Kmax⁡=32K_{\max}=32 shots at step 0, so this target is unattainable for ≈200\approx200 of 300 training steps—reducing C-GRPO to vanilla GRPO for two-thirds of training, which balloons kˉtrain\bar{k}_{\mathrm{train}} to 29.5 and depresses final pass@1 to 54.5%. Fixed δ=0.20\delta=0.20 works reasonably on GSM8K but triggers trivial phases on harder benchmarks (e.g. GPQA-Diamond with <50%<50\% base pass@Kmax⁡K_{\max}). Auto-δ\delta sets the threshold such that calibration pass rate satisfies the required coverage from step 0, guaranteeing the curriculum is active from the start on all six benchmarks and all three model families, with zero manual tuning.

(B) Re-calibration frequency. The default Δcal=50\Delta_{\mathrm{cal}}=50 steps maximises the accuracy–efficiency tradeoff. Halving to Δcal=25\Delta_{\mathrm{cal}}=25 adds +12% overhead per training step (calibrating Kmax⁡K_{\max} completions for ncal=200n_{\mathrm{cal}}=200 prompts at each checkpoint) without measurably improving accuracy, since the policy evolves slowly enough that 50-step-old thresholds remain accurate. Δcal=∞\Delta_{\mathrm{cal}}=\infty (static thresholds fixed at step 0) prevents rollout savings from compounding as training progresses, because the curriculum cannot track the policy’s growing competence, ballooning kˉtrain\bar{k}_{\mathrm{train}} to 18.2. The calibration cost itself is 0.50, 0.25, 0.13 and 0.04 h for Δcal=25\Delta_{\mathrm{cal}}=25, 50, 100, ∞\infty (12, 6, 3, 1 events), so halving to 25 buys nothing for twice the calibration cost, while 100 and ∞\infty let thresholds go stale. The multiplicity regime of Theorem 5 calibrates each budget at δauto/M\delta_{\mathrm{auto}}/M (about 0.02 for δauto≈0.12\delta_{\mathrm{auto}}\approx0.12 and M=5M=5), which is stricter than the fixed δ=0.05\delta=0.05 row of Table 5; it is statistically valid but, for weak base models, reintroduces the saturated thresholds that auto-δ\delta removes. The default per-budget calibration gives level-δ\delta false-stop control at every budget (Corollary 4), with MδM\delta as the simultaneous worst case. Per-prompt miscoverage events are strongly positively dependent across budgets (a prompt whose gold answer is rarely sampled is missed at every kk), so the union bound is loose, as the joint empirical validation in Figure 4 shows.

(C) Budget grid. The geometric-doubling grid {2,4,8,16,32}\{2, 4, 8, 16, 32\} is Pareto-optimal. The coarse grid {2,8,32}\{2, 8, 32\} forces moderate-difficulty prompts to jump from k=2k=2 directly to k=8k=8, raising kˉtrain\bar{k}_{\mathrm{train}} by ≈2.5\approx2.5 rollouts/prompt and reducing final p@16 by 1.7 pp. The fine grid {1,2,4,8,16,32}\{1, 2, 4, 8, 16, 32\} adds k=1k=1 greedy probes that slightly reduce cost on trivially easy prompts but introduce calibration noise on borderline prompts, yielding no net gain on MATH-500.

Curriculum Stability: No Catastrophic Forgetting

Remark 9 (No catastrophic forgetting of easy queries). Tightening thresholds reduces compute on easy queries but does not degrade the policy’s ability to answer them, for three complementary reasons. (i) Zero-gradient pruning. An easy query that forms a singleton at k1k_1 with all-correct rewards (Var⁡[rj]=0\operatorname{Var}[r_j]=0) is pruned before the backward pass (§ 4.4.1), so the model receives no gradient update on it—neither positive nor negative. The update cannot be pushed away from correct behaviour on easy queries. (ii) KL anchoring. The β⋅KL^(θ)\beta\cdot\widehat{\mathrm{KL}}(\theta) term in LC−GRPO\mathcal{L}_{\mathrm{C-GRPO}} (Eq. (11)) penalises any deviation from πref\pi_{\mathrm{ref}} uniformly across all queries, bounding total policy drift regardless of which queries currently drive the gradient. (iii) Self-correcting curriculum. If accuracy on any query class were to decrease, the empirical score distribution would worsen, raising q^k(t)\hat{q}_k^{(t)} at the next re-calibration checkpoint and restoring those queries to larger budgets—providing gradient signal until performance recovers. Together, these mechanisms make C-GRPO’s curriculum self-stabilising: hard queries receive more compute, but easy queries are protected rather than neglected.

Full Proofs

This section provides complete, self-contained proofs of Theorem 2, Proposition 6, and Corollary 7, with full exchangability arguments omitted from the main paper for brevity.

Proof of Theorem 2 (Marginal Coverage)

Proof. We prove both the lower and upper bounds of Eq. (15).

Setup. Fix a budget k∈Kk \in\mathcal{K}, score s∈{sAPS,scode}s \in\{s_{\mathrm{APS}},s_{\mathrm{code}}\}, and policy πθ\pi_\theta. For each example i∈{1,…,n+1}i \in\{1,\ldots,n+1\}, draw y^i,1:k∼i.i.d.πθ(⋅∣xi)\hat{y}_{i,1:k} \overset{\mathrm{i.i.d.}}{\sim} \pi_\theta(\cdot\mid x_i) independently of all other examples. Define Si=s(yi⋆;y^i,1:k)S_i = s(y_i^\star;\hat{y}_{i,1:k}).

Exchangeability. Since (x1,y1⋆),…,(xn+1,yn+1⋆)(x_1,y_1^\star),\ldots,(x_{n+1},y_{n+1}^\star) are i.i.d. from PP and completions are drawn i.i.d. from πθ\pi_\theta, the joint distribution of (S1,…,Sn+1)(S_1,\ldots,S_{n+1}) is exchangeable: for any permutation σ∈Sn+1\sigma\in\mathfrak{S}_{n+1}, (Sσ(1),…,Sσ(n+1))=d(S1,…,Sn+1)(S_{\sigma(1)},\ldots,S_{\sigma(n+1)}) \overset{d}{=} (S_1,\ldots,S_{n+1}). In particular, the rank of Sn+1S_{n+1} among {S1,…,Sn+1}\{S_1,\ldots,S_{n+1}\} is uniformly distributed on {1,…,n+1}\{1,\ldots,n+1\}.

Threshold construction. Let L=min⁡{⌈(n+1)(1−δ)⌉,n}L=\min\{\lceil(n+1)(1-\delta)\rceil,n\} and q^k=S(L)\hat{q}_k=S_{(L)}, the LL-th order statistic of S1,…,SnS_1,\ldots,S_n.

Lower bound. P(Sn+1≤q^k)=P(rank⁡(Sn+1)≤L)=L/(n+1)≥⌈(n+1)(1−δ)⌉/(n+1)≥1−δP(S_{n+1}\leq\hat{q}_k)=P(\operatorname{rank}(S_{n+1})\leq L)=L/(n+1)\geq\lceil(n+1)(1-\delta)\rceil/(n+1)\geq1-\delta.

Upper bound. L/(n+1)≤⌈(n+1)(1−δ)⌉/(n+1)≤1−δ+1/(n+1)L/(n+1)\leq\lceil(n+1)(1-\delta)\rceil/(n+1)\leq1-\delta+1/(n+1).

APS continuity. The randomized term U⋅p^(y⋆)U\cdot\hat{p}(y^\star) in sAPSs_{\mathrm{APS}} (Eq. (4)) makes the score distribution continuous on [0,1][0,1], ensuring P(Si=Sj)=0P(S_i=S_j)=0 for i≠ji\ne j. Hence the rank of Sn+1S_{n+1} is strictly uniform on {1,…,n+1}\{1,\ldots,n+1\} with no tied-rank ambiguity.

First-success score. scode=j⋆/k∈(0,1]s_{\mathrm{code}}=j^\star/k\in(0,1] with j⋆j^\star the position of the first correct completion, or 11 if none exists. The event {j⋆=j}\{j^\star=j\} has probability (1−px)j−1px(1-p_x)^{j-1}p_x for local pass rate pxp_x. Because pxp_x is continuous, P(Si=t)=0P(S_i=t)=0 for all t∈(0,1)t\in(0,1), giving strict exchangeability and strict uniformity of the rank.

Proof of Proposition 6 (Monotone Training Budget)

Proof. Let Fk(m)F_k^{(m)} denote the CDF of s(y⋆;y^1:k)s(y^\star;\hat{y}_{1:k}) under policy πm\pi_m.

APS case. By hypothesis, Ex[p^k(1)(y⋆∣x)]≥Ex[p^k(0)(y⋆∣x)]\mathbb{E}_x[\hat{p}_k^{(1)}(y^\star\mid x)]\geq\mathbb{E}_x[\hat{p}_k^{(0)}(y^\star\mid x)]. Higher empirical pass rate implies stochastically smaller APS score (less mass lies strictly above y⋆y^\star in the frequency histogram), so Fk(1)(t)≥Fk(0)(t)F_k^{(1)}(t)\geq F_k^{(0)}(t) for all t∈[0,1]t\in[0,1], i.e. S(1)⪯stS(0)S^{(1)}\preceq_{\mathrm{st}}S^{(0)} (stochastic dominance). Monotonicity of order statistics then gives S(L)(1)≤S(L)(0)S_{(L)}^{(1)}\leq S_{(L)}^{(0)} in expectation, establishing q^k(1)≤q^k(0)\hat{q}_k^{(1)}\leq\hat{q}_k^{(0)}.

Code case. Higher accuracy means j⋆j^\star under π1\pi_1 is stochastically smaller: Pπ1(j⋆>j)≤Pπ0(j⋆>j)P_{\pi_1}(j^\star>j)\leq P_{\pi_0}(j^\star>j) for all jj. Since scode=j⋆/ks_{\mathrm{code}}=j^\star/k, the same stochastic-dominance argument applies, yielding q^k(1)≤q^k(0)\hat{q}_k^{(1)}\leq\hat{q}_k^{(0)}.

Budget consequence. By Eq. (9), kused(x;π)=min⁡{k∈K:∣Ck(x)∣=1}k_{\mathrm{used}}(x;\pi)=\min\{k\in\mathcal{K}:|\mathcal{C}_k(x)|=1\}. A smaller threshold q^k(1)\hat{q}_k^{(1)} makes the singleton condition ∣Ck(x)∣=1|\mathcal{C}_k(x)|=1 easier to satisfy at small kk, weakly increasing the set of queries that exit at each budget level. Formally, for any kk, let Ek(m)={x:kused(x;πm)=k}E_k^{(m)}=\{x:k_{\mathrm{used}}(x;\pi_m)=k\}. Then Ek(1)⊇Ek(0)E_k^{(1)}\supseteq E_k^{(0)} (more queries exit early), and Ex[kused(x;π1)]≤Ex[kused(x;π0)]\mathbb{E}_x[k_{\mathrm{used}}(x;\pi_1)]\leq\mathbb{E}_x[k_{\mathrm{used}}(x;\pi_0)].

Proof of Corollary 7 (Self-Annealing Training Cost)

Proof. At each re-calibration checkpoint tt, q(t)q^{(t)} is calibrated using the current trained policy πθt\pi_{\theta_t} (Algorithm 1, lines 16–17). GRPO training with positive rewards drives policy improvement: for the trained policy at step t+Δcalt+\Delta_{\mathrm{cal}}, Ex[p^k(t+Δcal)(y⋆∣x)]≥Ex[p^k(t)(y⋆∣x)]\mathbb{E}_x[\hat{p}_k^{(t+\Delta_{\mathrm{cal}})}(y^\star\mid x)]\geq\mathbb{E}_x[\hat{p}_k^{(t)}(y^\star\mid x)] for all k∈Kk\in\mathcal{K} (expected under the reward-positive update). Applying Proposition 6: q^k(t+Δcal)≤q^k(t)\hat{q}_k^{(t+\Delta_{\mathrm{cal}})}\leq\hat{q}_k^{(t)}, and consequently Ex[kused(t+Δcal)]≤Ex[kused(t)]\mathbb{E}_x[k_{\mathrm{used}}^{(t+\Delta_{\mathrm{cal}})}]\leq\mathbb{E}_x[k_{\mathrm{used}}^{(t)}]. Since this holds at every checkpoint, the sequence (E[kused(0)],E[kused(Δcal)],E[kused(2Δcal)],…)(\mathbb{E}[k_{\mathrm{used}}^{(0)}],\mathbb{E}[k_{\mathrm{used}}^{(\Delta_{\mathrm{cal}})}],\mathbb{E}[k_{\mathrm{used}}^{(2\Delta_{\mathrm{cal}})}],\ldots) is non-increasing.

Proof of Corollary 4 (False-Stop Control)

Proof. Fix k∈Kk \in\mathcal{K} and one draw of UU shared by all candidate answers. Suppose ∣Ck(xn+1)∣=1\lvert C_k(x_{n+1})\rvert= 1, say Ck={a}C_k = \{a\}. Exactly one of two disjoint cases holds. If a=yn+1⋆a = y^\star_{n+1} the stop is correct. If a≠yn+1⋆a \ne y^\star_{n+1}, then yn+1⋆∉Cky^\star_{n+1} \notin C_k because CkC_k has no other element. Hence

{∣Ck∣=1, Ck≠{yn+1⋆}}⊆{yn+1⋆∉Ck}={s(yn+1⋆;a^1:k)>q^k},\{\lvert C_k\rvert= 1,\ C_k \ne\{y^\star_{n+1}\}\} \subseteq\{y^\star_{n+1} \notin C_k\} = \{s(y^\star_{n+1}; \hat{a}_{1:k}) > \hat{q}_k\},

and Theorem 2 gives P(s(yn+1⋆;a^1:k)>q^k)≤δP(s(y^\star_{n+1}; \hat{a}_{1:k}) > \hat{q}_k) \le\delta. For the first-success score and q^k<1\hat{q}_k < 1, scode≤q^k<1s_{\mathrm{code}} \le\hat{q}_k < 1 forces j⋆j^\star to exist, i.e. at least one of the first kk samples passes all unit tests.

Proof of Theorem 5 (Coverage of the Selected Budget)

Proof. For each k∈Kk \in\mathcal{K}, Theorem 2 at level δ/M\delta/M gives P(yn+1⋆∉Ck)≤δ/MP(y^\star_{n+1} \notin C_k) \le\delta/M (with the convention q^k=1\hat{q}_k = 1 if ⌈(n+1)(1−δ/M)⌉>n\lceil(n+1)(1-\delta/M)\rceil> n, for which coverage is trivially 1). A union bound over the MM budgets gives P(∃k: yn+1⋆∉Ck)≤δP(\exists k:\ y^\star_{n+1} \notin C_k) \le\delta. On the complementary event yn+1⋆∈Cky^\star_{n+1} \in C_k holds for every kk, hence also at any k^\hat{k} chosen as a function of the data. A wrong commit at k^=kused\hat{k} = k_{\mathrm{used}} means Ck^={a}≠{yn+1⋆}C_{\hat{k}} = \{a\} \ne\{y^\star_{n+1}\}, which implies yn+1⋆∉Ck^y^\star_{n+1} \notin C_{\hat{k}}, so its probability is at most δ\delta. At level δ\delta per budget the same union bound gives MδM\delta. Across TcT_c checkpoints, Theorem 2 holds marginally for each frozen policy and its fresh completions, so a union bound over the MTcMT_c (budget, checkpoint) pairs at level δ/(MTc)\delta/(MT_c) gives the claim simultaneously over checkpoints, even though they share calibration prompts.

Exchangeability of Nonconformity Scores

Conformal validity depends only on the exchangeability of the augmented score sequence (S1,…,Sn,Sn+1)(S_1,\ldots,S_n,S_{n+1}), not on a particular parametric model. The next two lemmas establish this property for the APS and code-based nonconformity scores introduced in § 4.2.

Lemma 10 (Exchangeability of APS Score). Let (xi,yi⋆)i=1n+1(x_i,y_i^\star)_{i=1}^{n+1} be i.i.d. from PP and let a^i,1:k∼i.i.d.πθ(⋅∣xi)\hat{a}_{i,1:k} \overset{\mathrm{i.i.d.}}{\sim} \pi_\theta(\cdot\mid x_i) independently across examples and positions. Define Si=sAPS(yi⋆;a^i,1:k)S_i=s_{\mathrm{APS}}(y_i^\star;\hat{a}_{i,1:k}). Then (S1,…,Sn+1)(S_1,\ldots,S_{n+1}) is exchangeable.

Proof. Step 1: Symmetry within an example. sAPSs_{\mathrm{APS}} depends on a^i,1:k\hat{a}_{i,1:k} only through the empirical frequency histogram {p^(a)}a∈A\{\hat{p}(a)\}_{a\in\mathcal{A}}, which is a symmetric (permutation-invariant) function of a^i,1:k\hat{a}_{i,1:k}. The randomized tiebreaker Ui∼Uniform⁡[0,1]U_i \sim\operatorname{Uniform}[0,1] is drawn independently, making SiS_i continuous and well-defined regardless of ties.

Step 2: Independence across examples. Since (xi,yi⋆,a^i,1:k,Ui)(x_i,y_i^\star,\hat{a}_{i,1:k},U_i) are mutually independent across ii, the scores S1,…,Sn+1S_1,\ldots,S_{n+1} are i.i.d. (a special case of exchangeability). For any permutation σ∈Sn+1\sigma\in\mathfrak{S}_{n+1},

(Sσ(1),…,Sσ(n+1))=d(S1,…,Sn+1)(S_{\sigma(1)},\ldots,S_{\sigma(n+1)}) \overset{d}{=} (S_1,\ldots,S_{n+1}).

Lemma 11 (Exchangeability of First-Success Score). Under the same i.i.d. setup, define Si=scode(yi⋆;y^i,1:k)S_i=s_{\mathrm{code}}(y_i^\star;\hat{y}_{i,1:k}). Then (S1,…,Sn+1)(S_1,\ldots,S_{n+1}) is exchangeable.

Proof. Intra-example non-symmetry. Unlike the APS score, scodes_{\mathrm{code}} is not symmetric in y^i,1:k\hat{y}_{i,1:k}: it records the ordinal position of the first passing completion, which depends on the ordering. This is by design—the ordering reflects the sequential generation process.

Inter-example exchangeability. Conformal validity requires exchangeability of example-level scores (S1,…,Sn+1)(S_1,\ldots,S_{n+1}), not intra-example symmetry. Since (xi,yi⋆,y^i,1:k)(x_i,y_i^\star,\hat{y}_{i,1:k}) are all i.i.d. (identical marginal distribution of the tuple), so are S1,…,Sn+1S_1,\ldots,S_{n+1}. Hence (S1,…,Sn+1)(S_1,\ldots,S_{n+1}) is exchangeable.

Remark 12 (Why intra-example ordering does not matter). The first-success score’s sensitivity to ordering within y^i,1:k\hat{y}_{i,1:k} captures the geometric distribution of success position: P(j⋆=j)=(1−px)j−1pxP(j^\star=j)=(1-p_x)^{j-1}p_x for local pass rate pxp_x. This is a continuous family in pxp_x, ensuring the score distribution is continuous and ties occur with probability zero—preserving the strict uniformity of the test rank needed for exact (rather than conservative) coverage.

Remark 13 (Exchangeability under periodic re-calibration). At each re-calibration checkpoint tt, the policy πθt\pi_{\theta_t} is fixed before re-evaluating the calibration set. The exchangeability argument holds for each checkpoint independently: the ncaln_{\mathrm{cal}} calibration examples and one test example are i.i.d. under fixed πθt\pi_{\theta_t}. Across checkpoints, the policy changes, but coverage guarantees are instantiated separately at each checkpoint, maintaining validity throughout training.

Robustness to Distribution Shift

Standard split-conformal prediction assumes the calibration and test examples are exchangeable (i.i.d. from the same distribution PP). In deployment, the test distribution PtestP_{\mathrm{test}} may differ from the calibration distribution PcalP_{\mathrm{cal}}. We formalise how coverage degrades under bounded covariate shift and validate empirically.

Theorem 14 (Coverage under Bounded Covariate Shift). Let PcalP_{\mathrm{cal}} and PtestP_{\mathrm{test}} be distributions over X×A\mathcal{X} \times\mathcal{A} with total-variation distance dTV(Pcal,Ptest)≤ϵshiftd_{\mathrm{TV}}(P_{\mathrm{cal}}, P_{\mathrm{test}}) \le\epsilon_{\mathrm{shift}}. Let q^k\hat{q}_k be calibrated from nn i.i.d. examples of PcalP_{\mathrm{cal}} at level δ\delta. Then for a test point from PtestP_{\mathrm{test}}:

Ptest(s(yn+1∗;y^n+1,1:k)≤q^k)≥1−δ−ϵshift−1n+1.(19)P_{\mathrm{test}}\left(s\left(y^*_{n+1}; \hat{y}_{n+1,1:k}\right) \le\hat{q}_k\right) \ge1-\delta-\epsilon_{\mathrm{shift}}-\frac{1}{n+1}. \tag*{(19)}

Proof. By the coupling characterization of total variation, there exists a coupling (Z~,Ztest)(\tilde{Z}, Z_{\mathrm{test}}) with Z~∼Pcal\tilde{Z} \sim P_{\mathrm{cal}}, Ztest∼PtestZ_{\mathrm{test}} \sim P_{\mathrm{test}}, and P(Z~≠Ztest)=dTV≤ϵshiftP(\tilde{Z} \ne Z_{\mathrm{test}}) = d_{\mathrm{TV}} \le\epsilon_{\mathrm{shift}}. For any measurable event AA and any fixed threshold tt:

∣Ptest(Sn+1≤t)−Pcal(Sn+1≤t)∣≤dTV(Pcal,Ptest)≤ϵshift.\left|P_{\mathrm{test}}(S_{n+1} \le t) - P_{\mathrm{cal}}(S_{n+1} \le t)\right| \le d_{\mathrm{TV}}(P_{\mathrm{cal}}, P_{\mathrm{test}}) \le\epsilon_{\mathrm{shift}}.

By Theorem 2 (applied under PcalP_{\mathrm{cal}}): Pcal(Sn+1≤q^k)≥1−δ−1n+1P_{\mathrm{cal}}(S_{n+1} \le\hat{q}_k) \ge1-\delta-\frac{1}{n+1}. Combining:

Ptest(Sn+1≤q^k)≥Pcal(Sn+1≤q^k)−ϵshift≥1−δ−ϵshift−1n+1.P_{\mathrm{test}}(S_{n+1} \le\hat{q}_k) \ge P_{\mathrm{cal}}(S_{n+1} \le\hat{q}_k)-\epsilon_{\mathrm{shift}} \ge1-\delta-\epsilon_{\mathrm{shift}}-\frac{1}{n+1}.

Remark 15. Tibshirani et al. [ [24] extends the result of Tibshirani et al. [ [24] to the online training setting of C-GRPO: coverage degrades by at most ϵshift\epsilon_{\mathrm{shift}} per unit of TV distance between training/calibration and test distributions, gracefully interpolating between the fully valid in-distribution case (ϵshift=0\epsilon_{\mathrm{shift}}=0) and the completely uncontrolled out-of-distribution case. Re-calibration with πθt\pi_{\theta_t} on the target calibration split at each checkpoint further mitigates shift by adapting thresholds to the local difficulty of the test domain.

Empirical validation under distribution shift. Table 6 reports coverage for a model trained and calibrated on GSM8K, evaluated on test distributions of increasing dissimilarity. dTVd_{\mathrm{TV}} is estimated from held-out query difficulty features (solve rate, response-length variance, and lexical overlap with calibration prompts). Even under severe domain shift (GPQA-Diamond), empirical coverage remains well above the theoretical lower bound (Eq. (19)), confirming that C-GRPO’s thresholds are robust in practice.

Test DomainShiftdTVd_{\mathrm{TV}}Theo. LBEmpiricalMargin
GSM8K (in-dist.)none0.0087.5%87.9%+0.4 pp
MATH-Algebramild0.0681.5%83.6%+2.1 pp
MATH-500moderate0.1572.5%76.8%+4.3 pp
GPQA-Diamondsevere0.2958.5%70.4%+11.9 pp

Table 6. Coverage under distribution shift (Qwen2.5-7B, trained and calibrated on GSM8K, step 300, δ(0)=0.12\delta^{(0)}=0.12, nominal target 88%). dTVd_{\mathrm{TV}}: TV distance between GSM8K calibration and test distribution. Theoretical bound: 1−δ−dTV−1/(ncal+1)1-\delta-d_{\mathrm{TV}}-1/(n_{\mathrm{cal}}+1). Empirical coverage consistently exceeds the theoretical lower bound.

Policy Stability and Calibration Staleness

Re-calibration every Δcal\Delta_{\mathrm{cal}} steps relies on the assumption that the policy does not drift so severely between checkpoints that the old thresholds become vacuous. We bound the expected policy displacement and consequent threshold staleness.

Theorem 16 (KL-Bounded Policy Displacement). Let πθ0\pi_{\theta_0} be any starting policy, and let πθt\pi_{\theta_t} be the policy after tt gradient steps of LC-GRPO\mathcal{L}_{\mathrm{C\text{-}GRPO}} with KL coefficient β>0\beta> 0 and learning rate η\eta. Under LL-smoothness of the policy log-probabilities and bounded rewards rj∈[−1,1]r_j \in[-1,1], the expected accumulated KL divergence satisfies:

E[KL⁡(πθt∥πref)]≤ηt∥∇L⋆∥22β,(20)\mathbb{E}\left[\operatorname{KL}\left(\pi_{\theta_t}\mathbin\Vert\pi_{\mathrm{ref}}\right)\right] \le\frac{\eta t\left\lVert\nabla\mathcal{L}^{\star}\right\rVert^2}{2\beta}, \tag*{(20)}

where ∥∇L⋆∥2\left\lVert\nabla\mathcal{L}^{\star}\right\rVert^2 is the expected squared gradient norm at convergence. By Pinsker’s inequality, dTV(πθt,πref)≤E[KL⁡(πθt∥πref)]/2d_{\mathrm{TV}}(\pi_{\theta_t},\pi_{\mathrm{ref}}) \le\sqrt{\mathbb{E}[\operatorname{KL}(\pi_{\theta_t}\mathbin\Vert\pi_{\mathrm{ref}})]/2}.

Proof. The KL term in LC-GRPO\mathcal{L}_{\mathrm{C\text{-}GRPO}} (Eq. (11)) imposes a soft constraint on KL⁡(πθ∥πref)\operatorname{KL}(\pi_\theta\mathbin\Vert\pi_{\mathrm{ref}}). Treating the training as projected gradient ascent, each step increases the policy reward by ≤η∥∇L∥2\le\eta\lVert\nabla\mathcal{L}\rVert^2 but accumulates KL cost β⋅KL⁡\beta\cdot\operatorname{KL} in the objective. Summing over tt steps and bounding the per-step KL increment by η∥∇L∥2/(2β)\eta\lVert\nabla\mathcal{L}\rVert^2/(2\beta) (from the β\beta-weighted KL penalty balancing the gradient), gives Eq. (20).

Corollary 17 (Calibration Staleness Bound). Let q^k(t)\hat{q}_k^{(t)} be calibrated at step tt and used until step t+Δcalt+\Delta_{\mathrm{cal}}. The expected threshold error due to policy drift is bounded by:

E[∣q^k(t+Δcal)−q^k(t)∣]≤CkηΔcal∥∇L⋆∥22β,(21)\mathbb{E}\left[\left|\hat{q}_k^{(t+\Delta_{\mathrm{cal}})}-\hat{q}_k^{(t)}\right|\right] \le C_k\sqrt{\frac{\eta\Delta_{\mathrm{cal}}\left\lVert\nabla\mathcal{L}^{\star}\right\rVert^2}{2\beta}}, \tag*{(21)}

where Ck>0C_k > 0 depends on the local slope of the score CDF near q^k\hat{q}_k.

Proof. Over Δcal\Delta_{\mathrm{cal}} steps after checkpoint tt, the policy drifts from πθt\pi_{\theta_t} to πθt+Δcal\pi_{\theta_{t+\Delta_{\mathrm{cal}}}}. By Theorem 16 applied to the Δcal\Delta_{\mathrm{cal}}-step window, dTVd_{\mathrm{TV}} grows as O(Δcal)O(\sqrt{\Delta_{\mathrm{cal}}}). A change in dTVd_{\mathrm{TV}} affects the score CDF and hence the order statistic q^k\hat{q}_k proportionally to the local CDF slope CkC_k via the delta method applied to order statistics, giving Eq. (21).

Remark 18 (Justification for Δcal=50\Delta_{\mathrm{cal}}=50). Eq. (21) shows threshold staleness grows as O(Δcal)O(\sqrt{\Delta_{\mathrm{cal}}}), while calibration overhead is O(1/Δcal)O(1/\Delta_{\mathrm{cal}}). The Pareto-optimal Δcal\Delta_{\mathrm{cal}} balances the two: the ablation in Appendix § 9 confirms Δcal=50\Delta_{\mathrm{cal}}=50 achieves this, consistent with the bound.

Finite-Sample Lower Bound on Rollout Savings

We lower-bound the expected rollout savings of C-GRPO over fixed-GG GRPO in terms of the calibrated threshold vector q^\hat{q}.

Proposition 19 (Lower Bound on Expected Rollout Savings). Let πθ\pi_\theta be a fixed policy with conformal thresholds q^=(q^k1,…,q^kM)\hat{q}=(\hat{q}_{k_1},\ldots,\hat{q}_{k_M}) calibrated at level δ\delta. Let pk(x)=Pπθ(∣Ck(x)∣=1)p_k(x)=\mathbb{P}_{\pi_\theta}(|C_k(x)|=1) be the singleton probability at budget kk for query xx. The expected per-prompt rollout savings versus fixed-G=Kmax⁡G=K_{\max} GRPO are:

Ex[Kmax⁡−kused(x)]≥∑m=1M−1(km+1−km)Ex[∏j=1m1[∣Ckj(x)∣=1]].(22)\mathbb{E}_x[K_{\max}-k_{\mathrm{used}}(x)]\ge\sum_{m=1}^{M-1}(k_{m+1}-k_m)\mathbb{E}_x\left[\prod_{j=1}^{m}\mathbf{1}[|C_{k_j}(x)|=1]\right]. \tag*{(22)}

Proof. Expand Kmax⁡−kused(x)K_{\max}-k_{\mathrm{used}}(x) by telescoping over the budget grid:

Kmax⁡−kused=∑m:km<kused(km+1−km)≥∑m=1M−1(km+1−km)⋅1[kused>km]c(23)\begin{aligned} K_{\max}-k_{\mathrm{used}}=\sum_{m:k_m<k_{\mathrm{used}}}(k_{m+1}-k_m) \\ &\ge\sum_{m=1}^{M-1}(k_{m+1}-k_m)\cdot\mathbf{1}[k_{\mathrm{used}}>k_m]^{c} \tag*{(23)} \end{aligned}

where {kused≤km}={∣Ck1(x)∣=1}∪⋯∪{∣Ckm(x)∣=1}\{k_{\mathrm{used}} \le k_m\} = \{|C_{k_1}(x)| = 1\} \cup\cdots\cup\{|C_{k_m}(x)| = 1\}. A query exits at kmk_m or earlier if and only if a singleton forms at some kj≤kmk_j \le k_m, saving Kmax⁡−kmK_{\max} - k_m rollouts. Lower-bounding by the event that all levels k1,…,kmk_1,\ldots,k_m form singletons (dropping the union by inclusion) and taking expectations yields Eq. (22).

Remark 20 (Connection to conformal thresholds). pk1(x)=P(∣Ck1(x)∣=1)p_{k_1}(x) = P(|C_{k_1}(x)| = 1) is directly controlled by q^k1\hat{q}_{k_1}: a lower threshold concentrates more probability mass on a single answer at budget k1k_1, increasing pk1p_{k_1}. Eq. (22) therefore shows that the rollout savings are monotonically non-decreasing in the policy improvement that drives q^k1\hat{q}_{k_1} downward—consistent with Proposition 6 and empirically confirmed by the 49–82% savings in Table 3.

Static-GG Sweep, Longer Horizon, and Stronger Base

All tables in this section were produced with the original calibration procedure (one calibration set for both δ\delta and the quantiles, per-budget δauto\delta_{\mathrm{auto}}, no multiplicity correction). Standard error of an accuracy near 57% is 2.5–3.5 pp at ntest=200n_{\mathrm{test}}=200–400.

StepMethodp@1kˉ\bar{k}Savings
100C-GRPO (Qwen2.5-7B)52.118.243%
300C-GRPO (Qwen2.5-7B)57.210.468%
600C-GRPO (Qwen2.5-7B)61.86.181%
1000C-GRPO (Qwen2.5-7B)64.54.287%
1000Static GRPO (G=32G=32)59.132.0—
1000C-GRPO (Llama-3.1-8B)76.23.190%

Table 8. Longer horizon (GSM8K). Savings =1−kˉ/32= 1 - \bar{k}/32 (training rollouts only).

Reading. C-GRPO versus static G=32G=32 is +0.4 pp, or 1.1 pooled standard deviations: statistically indistinguishable, a match at 5.5× fewer rollouts rather than an accuracy gain. Static G=8G=8 versus C-GRPO is −2.8 pp (6.3 pooled standard deviations), so matching full-budget accuracy at kˉ=5.8\bar{k}=5.8 is not obtained by lowering the static budget. In this configuration 76% of prompts are correct after the k=2k=2 probe, and wall-clock is 2.2 h against 11.7 h for static G=32G=32. The coverage under this original procedure is in Appendix § 8.

Reproducibility, Statistical Reporting, and Compute

Reproducibility protocol. All experiments use the same training and evaluation pipeline across methods unless explicitly stated otherwise. Each benchmark is split into a sequestered calibration set, a training set, and a disjoint held-out test set. The calibration split is used only to estimate conformal thresholds and the auto-selected miscoverage level δ\delta, while the held-out test split is used only for final reporting. We use ncal=200n_{\mathrm{cal}} = 200 calibration examples per benchmark and ntest=200n_{\mathrm{test}} = 200–400400 held-out test examples, depending on benchmark size. The six benchmarks are GSM8K, MATH-500, MATH-Algebra, GPQA-Diamond, HumanEval, and MBPP.

Training configuration. All model-training experiments use LoRA fine-tuning with rank r=16r = 16, scaling α=32\alpha= 32, dropout 0.05, AdamW optimizer with learning rate 2×10−62 \times10^{-6}, batch size 4, and 300 training steps with 2,000 training examples per benchmark. Unless otherwise noted, the C-GRPO budget grid is K={2,4,8,16,32}K = \{2, 4, 8, 16, 32\}, the recalibration period is Δcal=50\Delta_{\mathrm{cal}} = 50, the KL coefficient is β=0.05\beta= 0.05, and the auto-δ\delta safety margin is ϵ=0.05\epsilon= 0.05. Static and dynamic baselines use the same model, data, optimizer, LoRA configuration, number of training steps, and verifier, differing only in the rollout-allocation algorithm. The KL term uses the non-negative per-token estimator KL^t=eΔt−Δt−1≥0\widehat{\mathrm{KL}}_t = e^{\Delta_t} - \Delta_t - 1 \ge0 with Δt=log⁡πref(ℓt)−log⁡πθ(ℓt)\Delta_t = \log\pi_{\mathrm{ref}}(\ell_t) - \log\pi_\theta(\ell_t), where πref\pi_{\mathrm{ref}} is the frozen reference policy, averaged over completion tokens and the batch to avoid sequence-level overflow on long completions.

Evaluation protocol. We report fixed-budget pass@kk accuracy using exactly kk completions at evaluation time. For math and science benchmarks, answers are canonicalized and evaluated by exact-match verification; for code benchmarks, completions are evaluated by execution against unit tests. For pass@kk, math and science use majority vote over extracted answers, while code uses best-of-kk execution success. The reported kˉtrain\bar{k}_{\mathrm{train}} is the mean number of rollouts consumed per prompt during training at step 300, and “Savings” is computed relative to fixed-budget GRPO with G=32G = 32.

Statistical reporting. The main metrics are empirical means over held-out test examples using deterministic verifiers. For binary pass@kk outcomes, uncertainty can be computed by the standard binomial standard error

SE⁡(p^)=p^(1−p^)ntest\operatorname{SE}(\hat{p}) = \sqrt{\frac{\hat{p}(1-\hat{p})}{n_{\mathrm{test}}}}

and for rollout-cost quantities we compute the empirical mean over prompts. For conformal coverage, we report the nominal target, empirical coverage, and the finite-sample tolerance induced by split conformal calibration, namely 1/(ncal+1)1/(n_{\mathrm{cal}} + 1). Coverage validation is reported for all six benchmarks in Appendix § 8; all empirical coverages lie within the expected finite-sample tolerance. Training-seed variance is reported only where stated (three seeds for the full-parameter experiment, Table 9); elsewhere uncertainty is the binomial standard error over the test set, and differences below roughly 2–3 pp should not be interpreted.

MethodBudgetp@1p@32kˉ\bar{k}Savings
GRPO (static)G=32G=3288.5±0.388.5 \pm0.392.1±0.292.1 \pm0.232.0—
GRPO (static)G=8G=886.1±0.486.1 \pm0.490.4±0.390.4 \pm0.38.075%
C-GRPOadaptive88.9±0.288.9 \pm0.292.4±0.292.4 \pm0.25.882%

Table 9. Stronger base, full-parameter fine-tuning (GSM8K, Qwen2.5-Math-7B-Instruct, base p@1 = 82.4, 300 steps; mean ±\pm std over three seeds).

Compute resources. All experiments were run on NVIDIA H100 GPUs with 80 GB of GPU memory. Each reported training job uses one H100 GPU for model inference and LoRA updates, with CPU workers used for answer extraction, deterministic verification, code execution tests, logging, and metric computation. Table 1 and Table 2 report training GPU-hours, peak GPU memory, and effective rollout-generation throughput for the main experiments on a single H100. The largest reported runs are the fixed-budget GRPO and GDRO baselines, which require approximately 11.7 GPU-hours and 26.8 GB peak VRAM for 300-step Qwen2.5-7B-Instruct training, while C-GRPO requires approximately 3.8 GPU-hours (including all calibration generation) and 18.6 GB peak VRAM in the corresponding setting; Table 10 gives the phase-level breakdown. The full project required additional preliminary runs for prompt formatting, verifier debugging, and grid selection; these exploratory runs are not included in the main compute tables because they do not affect the reported comparisons. Launch scripts, software versions, model identifiers, dataset-loading code and calibration/evaluation scripts are in the released repository (§ 5.1).

Limitations, Asset Use, and Responsible Release

Limitations. C-GRPO assumes a verifier is available for each task, either through exact answer matching for closed-form reasoning or executable tests for code. This makes the method directly applicable to mathematics, science QA with canonical answers, and program synthesis, but less direct for open-ended generation tasks where correctness is subjective or multi-dimensional. The conformal guarantee also relies on exchangeability between calibration and evaluation examples under a fixed policy at each recalibration checkpoint; under distribution shift, coverage can degrade as described in Appendix § 13. Finally, the current implementation uses a discrete budget grid KK and binary verifier rewards; extending the method to continuous budgets, graded rewards, preference-model scores, and broader RLHF settings is an important direction for future work.

Existing assets. The paper uses public or previously released benchmarks and models, including GSM8K, MATH-500, MATH-Algebra, GPQA-Diamond, HumanEval, MBPP, Qwen2.5-7B-Instruct, Llama-3.1-8B-Instruct, and Gemma-3-4B-IT. We cite the original sources for these assets in the main paper and references. The released repository includes the model registry and the dataset-loading code that reconstructs the calibration, training and held-out evaluation splits. Qwen2.5 is used under Apache-2.0, Llama-3.1 under the Llama 3.1 Community License, and Gemma-3 under the Gemma Terms of Use.

New assets. The new assets introduced by this work are the C-GRPO training algorithm, calibration code, nonconformity-score implementations, auto-δ\delta selection logic, budget-allocation scripts, evaluation scripts, and experiment launch configurations. These assets do not include a newly pretrained foundation model or scraped dataset. The implementation and reproduction instructions are released (see Code availability below).

Societal impact and responsible use. The intended positive impact of C-GRPO is to reduce the compute cost of reinforcement learning for reasoning-oriented language models while adding statistical calibration to the rollout-allocation rule. This can make efficient RL training more accessible and reduce unnecessary energy use. A possible negative impact is that more efficient RL could also lower the cost of improving models for undesirable applications. We mitigate this by releasing training infrastructure and evaluation scripts rather than a new unrestricted pretrained model, and by focusing the empirical evaluation on standard reasoning and coding benchmarks with deterministic verifiers.

Code availability. The implementation is released at https://github.com/AryaFayyazi/CGRPO under the MIT license: training, calibration (APS and first-success scores, auto-δ\delta, and the optional disjoint Dδ/DqD_{\delta}/D_q split), evaluation, the baselines, analysis scripts, Slurm launchers, and REPRODUCTION.md with the commands and settings. A TRL integration, CGRPOTrainer, is available as pip install trl-cgrpo. Trained adapters are not included in the current release.

NeurIPS Paper Checklist

  1. Claims

Question: Do the main claims made in the abstract and introduction accurately reflect the paper’s contributions and scope?

Answer: [Yes].

Justification: The abstract and introduction state that C-GRPO replaces fixed GRPO rollout budgets with calibrated conformal thresholds, proves finite-sample coverage, false-stop control and a multiplicity-corrected guarantee for the selected budget, shows a non-increasing expected budget under an idealized monotone-improvement condition, and empirically reduces training rollouts at matched accuracy; accuracy differences over baselines are explicitly described as within binomial noise. These claims are supported by § 4–§ 5, Table 1–Table 10, and the appendix coverage and ablation studies.

  1. Limitations

Question: Does the paper discuss the limitations of the work performed by the authors?

Answer: [Yes].

Justification: § 5.7 and Appendix § 18 discuss verifier dependence, exchangeability assumptions, distribution-shift sensitivity, discrete budget grids, and the current focus on binary verifier rewards. The conclusion also identifies extensions beyond binary rewards, exchangeable calibration, and discrete budget grids as future work.

  1. Theory assumptions and proofs

Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?

Answer: [Yes].

Justification: Theorem 2, Corollary 4, Theorem 5, Proposition 6, Corollary 7 and Theorem 14 state the relevant assumptions, including a fixed policy, i.i.d./exchangeable calibration and test examples, a monotone-improvement condition for the budget results (assumed explicitly in Corollary 7), and bounded total-variation shift. Full proofs and exchangeability lemmas are provided in the appendix.

  1. Experimental result reproducibility

Question: Does the paper fully disclose all the information needed to reproduce the main experimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper, regardless of whether the code and data are provided or not?

Answer: [Yes].

Justification: § 5.1 and Appendix § 17 specify datasets, calibration/test split sizes, model backbones, LoRA configuration, optimizer, learning rate, batch size, training steps, budget grid, recalibration period, KL coefficient, evaluation protocol, and baseline configurations. The release plan further states that scripts and exact reproduction instructions will be made available upon acceptance.

  1. Open access to data and code

Question: Does the paper provide open access to the data and code, with sufficient instructions to faithfully reproduce the main experimental results, as described in supplemental material?

Answer: [Yes].

Justification: The code is released at https://github.com/AryaFayyazi/CGRPO (MIT license) with training, calibration, evaluation, baselines, launch scripts, and REPRODUCTION.md, which lists the commands and settings; all datasets and base models are public, and pip install trl-cgrpo provides a TRL trainer. Trained adapters are not included, and reproducing the exact numbers requires GPU compute (Appendix § 17).

  1. Experimental setting/details

Question: Does the paper specify all the training and test details, e.g., data splits, hyperparameters, how they were chosen, type of optimizer, necessary to understand the results? Answer: [Yes].

Justification: Section 5.1 specifies the six benchmarks, calibration and held-out test sizes, LoRA hyperparameters, optimizer, learning rate, batch size, training steps, training-sample count, budget grid, recalibration period, safety margin, KL coefficient, and evaluation metrics. Appendix J consolidates these details for reproducibility.

Experiment statistical significance

Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?

Answer: [Yes].

Justification: Appendix J explains that pass⁡@k\operatorname{pass}@k is a binary held-out empirical mean and gives the binomial standard-error formula for the reported test sizes. For conformal coverage, Appendix A reports empirical coverage against nominal targets and the finite-sample tolerance 1/(ncal+1)1/(n_{\mathrm{cal}} + 1), which is the relevant statistical uncertainty for the calibration guarantee. Section 5.1 states that differences of a few points are within the 2.5–3.5 pp binomial standard error and are read as matched accuracy, and the full-parameter experiment reports mean ±\pm std over three seeds (Table 9).

Experiments compute resources

Question: For each experiment, does the paper provide sufficient information on the computer resources, type of compute workers, memory, time of execution, needed to reproduce the experiments?

Answer: [Yes].

Justification: Appendix J states that all experiments were run on NVIDIA H100 80 GB GPUs, describes the GPU/CPU division of work, and summarizes runtime drivers. Table 1–Table 2 report GPU-hours, peak VRAM, and rollout throughput for the main experiments, the released repository includes launch scripts and software versions, and Table 10 gives the wall-clock breakdown including calibration.

Code of ethics

Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https://neurips.cc/public/EthicsGuidelines?

Answer: [Yes].

Justification: The work proposes an efficiency and calibration method for RL training on standard reasoning and code benchmarks. It does not involve human subjects, private data, surveillance, deceptive data collection, or deployment of a high-risk decision system, and the paper preserves author anonymity.

Broader impacts

Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?

Answer: [Yes].

Justification: Appendix K discusses positive impacts, including reduced RL training compute and statistically calibrated rollout allocation, as well as possible negative impacts from lowering the cost of improving models for undesirable applications. It also explains that the planned release focuses on reproducibility infrastructure rather than a new unrestricted pretrained model.

Safeguards

Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse, e.g., pre-trained language models, image generators, or scraped datasets?

Answer: [N/A].

Justification: The paper does not release a new pretrained language model, image generator, scraped dataset, or other high-risk generative asset. The release consists of training, calibration, verification, and evaluation code for reproducing experiments on existing public models and benchmarks.

Licenses for existing assets

Question: Are the creators or original owners of assets, e.g., code, data, models, used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?

Answer: [Yes].

Justification: The paper cites the original sources for all major datasets, models, and methods used in the evaluation, including GSM8K, MATH, GPQA, HumanEval, MBPP, Qwen2.5, Llama-3.1, Gemma, LoRA, GRPO, and the dynamic-rollout baselines. Appendix § 18 states that the release package will include asset versions, access instructions, and license or terms-of-use notes where available.

New assets

Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?

Answer: [Yes].

Justification: The new assets are the C-GRPO algorithm, conformal calibration code, first-success score implementation, auto-δ\delta logic, budget-allocation scripts, verifier wrappers, and evaluation scripts. Their mathematical definitions and algorithmic behavior are documented in § 4.2–§ 4.4 and Algorithm 1, and the implementation is released with reproduction instructions (Appendix § 17, Code availability).

Crowdsourcing and research with human subjects

Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation, if any?

Answer: [N/A].

Justification: The paper does not involve crowdsourcing, human-subject experiments, participant recruitment, surveys, or compensation. All experiments use existing benchmarks and deterministic verifiers or unit tests.

Institutional review board (IRB) approvals or equivalent for research with human subjects

Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board approvals or an equivalent approval/review based on the requirements of your country or institution were obtained?

Answer: [N/A].

Justification: The work does not involve human subjects, private personal data, or interaction with participants. Therefore, IRB approval or equivalent human-subjects review is not applicable.

Declaration of LLM usage

Question: Does the paper describe the usage of LLMs if it is an important, original, or non-standard component of the core methods in this research?

Answer: [Yes].

Justification: LLM usage is central to the paper: C-GRPO is a training-time rollout-allocation method for language models, and the experiments evaluate Qwen2.5-7B-Instruct, Llama-3.1-8B-Instruct, and Gemma-3-4B-IT. § 4–§ 5 describe how completions are sampled, verified, calibrated, and used in the adaptive GRPO objective.

References

  1. [1]Anastasios N. Angelopoulos, Stephen Bates, Emmanuel J. Candès, Michael I. Jordan, and Lihua Lei. Learn then test: Calibrating predictive algorithms to achieve risk control, 2021. URL https://arxiv.org/abs/2110.01052.
  2. [2]Jacob Austin, Augustus Odena, Maxwell Nye, Maarten Bosma, Henryk Michalewski, David Dohan, Ellen Jiang, Carrie Cai, Michael Terry, Quoc Le, and Charles Sutton. Program synthesis with large language models, 2021. URL https://arxiv.org/abs/2108.07732.
  3. [3]Stephen Bates, Anastasios Angelopoulos, Lihua Lei, Jitendra Malik, and Michael I. Jordan. Distribution-free, risk-controlling prediction sets, 2021. URL https://arxiv.org/abs/2101.02703.
  4. [4]Mark Chen, Jerry Tworek, Heewoo Jun, Qiming Yuan, Henrique Ponde de Oliveira Pinto, Jared Kaplan, Harri Edwards, Yuri Burda, Nicholas Joseph, Greg Brockman, Alex Ray, Raul Puri, Gretchen Krueger, Michael Petrov, Heidy Khlaaf, Girish Sastry, Pamela Mishkin, Brooke Chan, Scott Gray, Nick Ryder, Mikhail Pavlov, Alethea Power, Lukasz Kaiser, Mohammad Bavarian, Clemens Winter, Philippe Tillet, Felipe Petroski Such, Dave Cummings, Matthias Plappert, Fotios Chantzis, Elizabeth Barnes, Ariel Herbert-Voss, William Hebgen Guss, Alex Nichol, Alex Paino, Nikolas Tezak, Jie Tang, Igor Babuschkin, Suchir Balaji, Shantanu Jain, William Saunders, Christopher Hesse, Andrew N. Carr, Jan Leike, Josh Achiam, Vedant Misra, Evan Morikawa, Alec Radford, Matthew Knight, Miles Brundage, Mira Murati, Katie Mayer, Peter Welinder, Bob McGrew, Dario Amodei, Sam McCandlish, Ilya Sutskever, and Wojciech Zaremba. Evaluating large language models trained on code, 2021. URL https://arxiv.org/abs/2107.03374.
  5. [5]John Cherian, Isaac Gibbs, and Emmanuel J. Candès. Large language model validity via enhanced conformal prediction methods. In Advances in Neural Information Processing Systems (NeurIPS), 2024.arxiv.org/abs/2406.09714
  6. [6]Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, Mark Chen, Heewoo Jun, Lukasz Kaiser, Matthias Plappert, Jerry Tworek, Jacob Hilton, Reiichiro Nakano, Christopher Hesse, and John Schulman. Training verifiers to solve math word problems, 2021. URL https://arxiv.org/abs/2110.14168.
  7. [7]Mehul Damani, Idan Shenfeld, Andi Peng, Andreea Bobu, and Jacob Andreas. Learning how hard to think: Input-adaptive allocation of LM computation. In International Conference on Learning Representations (ICLR), 2025.arxiv.org/abs/2410.04707
  8. [8]Gemma Team. Gemma 3 technical report, 2025. URL https://arxiv.org/abs/2503.19786.
  9. [9]Isaac Gibbs, John J. Cherian, and Emmanuel J. Candès. Conformal prediction with conditional guarantees, 2023. URL https://arxiv.org/abs/2305.12616.
  10. [10]Aaron Grattafori, Abhimanyu Dubey, Abhinav Jau hri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, et al. The Llama 3 herd of models, 2024. URL https://arxiv.org/abs/2407.21783.
  11. [11]Yu Gui, Ying Jin, and Zhimei Ren. Conformal alignment: Knowing when to trust foundation models with guarantees, 2024. URL https://arxiv.org/abs/2405.10301.
  12. [12]Dan Hendrycks, Collin Burns, Saurav Kadavath, Akul Arora, Steven Basart, Eric Tang, Dawn Song, and Jacob Steinhardt. Measuring mathematical problem solving with the MATH dataset, 2021. URL https://arxiv.org/abs/2103.03874.
  13. [13]Edward J. Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen. LoRA: Low-rank adaptation of large language models. In International Conference on Learning Representations, 2022.arxiv.org/abs/2106.09685
  14. [14]M Pawan Kumar, Benjamin Packer, and Daphne Koller. Self-paced learning for latent variable models. In Advances in Neural Information Processing Systems (NeurIPS), volume 23, 2010.
  15. [15]Hunter Lightman, Vineet Kosaraju, Yura Burda, Harri Edwards, Bowen Baker, Teddy Lee, Jan Leike, John Schulman, Ilya Sutskever, and Karl Cobbe. Let’s Verify Step by Step, 2023. URL https://arxiv.org/abs/2305.20050.
  16. [16]Ilya Loshchilov and Frank Hutter. Decoupled weight decay regularization. In International Conference on Learning Representations, 2019.
  17. [17]Kishan Panaganti, Zhenwen Liang, Wenhao Yu, Haitao Mi, and Dong Yu. Group distributionally robust optimization-driven reinforcement learning for LLM reasoning, 2026. URL https://arxiv.org/abs/2601.19280.
  18. [18]Victor Quach, Adam Fisch, Tal Schuster, Adam Yala, Jae Ho Sohn, Tommi S. Jaakkola, and Regina Barzilay. Conformal language modeling. In International Conference on Learning Representations (ICLR), 2024.arxiv.org/abs/2306.10193
  19. [19]Qwen. Qwen2.5 technical report, 2024. URL https://arxiv.org/abs/2412.15115.
  20. [20]David Rein, Betty Li Hou, Asa Cooper Stickland, Jackson Petty, Richard Yuanzhe Pang, Julien Dirani, Julian Michael, and Samuel R. Bowman. GPQA: A graduate-level google-proof Q&A benchmark, 2023. URL https://arxiv.org/abs/2311.12022.
  21. [21]Yaniv Romano, Matteo Sesia, and Emmanuel J. Candès. Classification with valid and adaptive coverage. In Advances in Neural Information Processing Systems, volume 33, 2020.arxiv.org/abs/2006.02544
  22. [22]John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal policy optimization algorithms, 2017. URL https://arxiv.org/abs/1707.06347.
  23. [23]Zhihong Shao, Peiyi Wang, Qihao Zhu, Runxin Xu, Junxiao Song, Xiao Bi, Haowei Zhang, Mingchuan Zhang, Y.K. Li, Y. Wu, and Daya Guo. DeepSeekMath: Pushing the limits of mathematical reasoning in open language models, 2024. URL https://arxiv.org/abs/2402.03300.
  24. [24]Ryan J. Tibshirani, Rina Foygel Barber, Emmanuel J. Candès, and Aaditya Ramdas. Conformal prediction under covariate shift. In Advances in Neural Information Processing Systems, volume 32, 2019.arxiv.org/abs/1904.06019
  25. [25]Vladimir Vovk, Alex Gammerman, and Glenn Shafer. Algorithmic Learning in a Random World. Springer Science & Business Media, 2005.
  26. [26]Ronald J. Williams. Simple statistical gradient-following algorithms for connectionist reinforcement learning. Machine Learning, 8:229–256, 1992.
  27. [27]Wei Xiong, Chenlu Ye, Baohao Liao, Hanze Dong, Xinxing Xu, Christof Monz, Jiang Bian, Nan Jiang, and Tong Zhang. Reinforce-Ada: An adaptive sampling framework under non-linear RL objectives, 2025. URL https://arxiv.org/abs/2510.04996.
  28. [28]Zhi Zhang, Zhen Han, Costas Mavromatis, Qi Zhu, Yunyi Zhang, Sheng Guan, Dingmin Wang, Xiong Zhou, Shuai Wang, Soji Adeshina, Vassilis Ioannidis, and Huzefa Rangwala. Train less, learn more: Adaptive efficient rollout optimization for group-based reinforcement learning, 2026. URL https://arxiv.org/abs/2602.14338.
  29. [29]Haizhong Zheng, Yang Zhou, Brian R. Bartoldson, Bhavya Kailkhura, Fan Lai, Jiawei Zhao, and Beidi Chen. Act only when it pays: Efficient reinforcement learning for LLM reasoning via selective rollouts, 2025. URL https://arxiv.org/abs/2506.02177.

Paper details

vlnc:W_x25eadjh
Version
v1 · latest
Published
Fresh — added to the Hub within the last 30 days
Added
License
CC BY 4.0
Usage

views · 0

downloads · 0

A view counts once per browser per day, a download each time the PDF is requested.

  • Computer Science

Not recorded for this paper

View statement
Looking for related papers…
Contents