WorksComputer Science
C-GRPO: Conformal Group Relative Policy Optimization
Abstract
Reasoning-oriented language model training increasingly relies on multiple on-policy rollouts, but standard Group Relative Policy Optimization (GRPO) uses a fixed group size, wasting compute on easy prompts and under-exploring hard ones. Existing adaptive-budget methods alleviate this inefficiency, yet their stopping rules are heuristic and do not provide a finite-sample calibration guarantee. We propose CGRPO, which replaces the fixed rollout budget of standard GPRO with calibrated conformal thresholds over task-specific nonconformity scores, combining Adaptive Prediction Set (APS)-style calibration for closed-form reasoning, a first-success score for execution-based code, automatic miscoverage selection, and periodic re-calibration. We prove finite-sample coverage for the resulting stopping rule and show that, under monotone policy improvement, recalibration induces a non-increasing expected training budget. Across six benchmarks spanning mathematics, graduate-level science, and code generation, CGRPO reduces mean training rollouts by 49-70% relative to fixed-budget GRPO while retaining strong fixed-budget accuracy, achieves the strongest pass@1 among evaluated static and dynamic baselines on different datasets, and transfers across model families with up to 82% rollout savings.
Introduction
As LLMs scale, a massive shift toward multi-sample computation at both inference and training time is taking place. Strategies like majority voting for reasoning tasks and best-of- sampling for code generation consistently push the Pareto frontier. Even modern RL algorithms such as GRPO (Group Relative Policy Optimization) [23] leverage this by using on-policy rollouts (generating a batch of responses for the same prompt) to derive a relative advantage signal for the loss function. However, the bottleneck is that GRPO currently uses a static group size () for every input. This is fundamentally inefficient because it ignores the high variance in task difficulty. For “low-entropy" or trivial queries, generating a large group is just wasted FLOPs with a near-zero gradient signal. In contrast, for “frontier" or high-complexity queries, a fixed is often too small to capture a single successful trajectory, leaving the model with no positive reinforcement to learn from.
Adaptive-budget methods address this with reward variance, learned difficulty estimates, or global allocation rules [28, 17, 27, 29], but their stopping rules remain heuristic and offer no finite-sample guarantee as the policy evolves. Conformal prediction provides distribution-free uncertainty certificates [25, 21], but has been used only as a post-hoc inference wrapper, not as a training-time compute control mechanism [18, 11, 5].
40th Conference on Neural Information Processing Systems (NeurIPS 2026).
We bridge these perspectives with Conformal GRPO (C-GRPO), which replaces GRPO’s fixed rollout budget with calibrated stopping thresholds. For each budget , C-GRPO selects the smallest which is statistically sufficient for the current prompt. Practical ingredients include Adaptive Prediction Set (APS) calibration for closed-form reasoning, a new first-success score for execution-based code, automatic miscoverage selection to prevent degenerate early-training thresholds, and periodic recalibration to track the evolving policy. The result is a self-paced curriculum: easy queries exit early, and computational resources are reallocated to the hard tail.
We prove finite-sample coverage for the calibrated thresholds, derive false-stop control at every budget and a multiplicity-corrected guarantee for the adaptively selected budget, and establish a non-increasing expected training budget under monotone policy improvement (an idealized condition; § 4.5.2). Empirically, across six benchmarks that span mathematics, graduate-level science, and code, C-GRPO reduces mean training rollouts by 49–70% compared to fixed-budget GRPO at matched accuracy: pass@1 on GSM8K, HumanEval, MBPP, and MATH-500 is equal to or slightly above every evaluated baseline (differences are within binomial noise) with the fewest rollouts per prompt; rollout savings further reach 82% on GSM8K for Llama-3.1-8B without re-tuning. These theoretical and empirical results are enabled by three contributions: (i) we formulate adaptive-group GRPO as a conformal calibration problem; (ii) we introduce practical ingredients for LLM RL, including APS with first-success scores, auto-, and periodic re-calibration; and (iii) we prove coverage, false-stop and self-annealing guarantees and show empirically that calibrated allocation matches the accuracy of fixed and heuristic budgets at far fewer rollouts, including fixed- GRPO at matched compute.
Background
Policy Optimization and GRPO
Group Relative Policy Optimization (GRPO) [23] eliminates the memory-heavy value network of the PPO (Proximal Policy Optimization) [22] by computing advantages from a fixed group of i.i.d. (independent and identically distributed) completions :
where is a small numerical stabilizer (e.g., ). This achieves state-of-the-art results on mathematical reasoning and code generation. However, fixing uniformly wastes compute on easy queries (all , zero advantage) and starves hard queries of exploration (all , no positive signal), ignoring the wide per-prompt difficulty disparity [7].
Split-Conformal Prediction
Split-conformal prediction (CP) [25] constructs statistically guaranteed prediction sets with no distributional assumptions. Given i.i.d. calibration pairs and a nonconformity score , the empirical quantile (with the standard finite-sample correction ) satisfies for any exchangeable test point. Prior work applies CP post-hoc for inference-time abstention and safety filtering [18]. C-GRPO inverts this paradigm: rather than certifying a fixed model’s outputs, we embed CP inside the training loop so calibrated quantiles directly control the per-query generation budget, converting uncertainty measures into online compute decisions.
Related Work
Efficient RL and Adaptive Rollout Allocation. Adaptive rollout methods address GRPO’s fixed- inefficiency via rollout pruning, difficulty-aware allocation, and adaptive sampling [28, 17, 27, 29], but rely on heuristic or model-dependent signals (reward variance, learned difficulty scores) that lack formal guarantees and drift as the policy evolves. C-GRPO replaces them with a distribution-free conformal stopping rule grounded in finite-sample coverage theory.
Conformal Prediction for LLMs. Conformal methods have been applied to LLMs for calibrated uncertainty, abstention, and output filtering [18, 11, 5], as static post-hoc wrappers relying on token-level likelihoods that are uninformative when every completion is a unique string. We instead embed CP in the training objective and introduce an ordinal first-success score for execution-based reinforcement-learning (RL) settings.
Curriculum and Self-Paced Learning. Curriculum methods improve convergence by ordering examples from easy to hard [14]; adaptive rollout methods implicitly induce such curricula without guarantees. C-GRPO instantiates a self-paced curriculum (Corollary 7, under an idealized monotone-improvement condition, and empirically in aggregate): as the policy improves, calibrated thresholds tighten, shifting compute to harder queries with no external supervision or schedule.
Methodology

Figure 1. Overview of C-GRPO workflow.
Problem Formulation
Let denote the space of natural-language queries and the space of model completions. A language model defines a conditional distribution over parameterized by . An answer-extraction function maps raw completions to a structured answer space , and a deterministic verifier checks correctness against the gold answer :
We consider two task families: closed-form (math, science), where extracts a canonicalized answer and is exact-match; and generative code (HumanEval, MBPP), where executes against hidden unit tests. Let be a fixed budget grid with and rewards . The training objective is:
Here is a target per-step compute budget, and is the adaptive group size for query under policy (formally defined in § 4.4.1). C-GRPO solves this without explicit constraint optimization: calibrated conformal thresholds implicitly determine per query.
Conformal Nonconformity Scores
The conformal machinery requires a nonconformity score quantifying how surprising the gold answer is given model samples. A good score must be (i) continuous so that calibrated quantiles strictly decrease as the model improves, and (ii) consistent with exchangeability of the samples so that split-conformal validity holds without distributional assumptions on .
APS Score for Closed-Form Answers
For closed-form tasks we use the Adaptive Prediction Sets (APS) score of Romano et al. [ [21]. Given extracted answers , let be the empirical frequency. Sort distinct answers in decreasing frequency order (ties broken uniformly at random to strictly preserve statistical exchangeability). The APS nonconformity score is
with when (gold never sampled).
The APS score is strictly superior to the naive binary score : the -randomization makes it strictly continuous on , ensuring the quantile remains informative even for weak models, and achieves exact (not merely conservative) marginal coverage [21].
First-Success Score for Code
For execution-based code tasks, every generated completion is a syntactically unique string, so almost surely and for all calibration examples regardless of budget, making the APS framework vacuous. We therefore introduce a novel first-success score based on the ordinal position of the first passing solution within the ordered sample sequence:
That is, the score is the fraction of the budget consumed before the first success. It is continuous on : an easy problem solved on the first of samples scores ; an unsolvable problem always scores 1.
This score has a key calibration property: if and only if at least calibration problems have at least one passing solution within samples (§ 4.3). As the policy improves during training, more problems become solvable within small , driving below 1 and enabling the conformal curriculum to engage at smaller budgets.
Exchangeability. Both scores are functions of and inherit exchangeability from i.i.d. sampling of completions and calibration examples—the sole assumption conformal validity requires.
Split-Conformal Calibration
Given a held-out calibration set disjoint from training data and a model , we generate samples per query and derive one threshold per budget level.
Definition 1 (Split Conformal Quantile). For scores and miscoverage level , the finite-sample conformal quantile is:
where denotes the -th order statistic. The correction (rather than ) is the standard finite-sample adjustment ensuring for an exchangeable test score [25].
For each and score , the calibrated threshold is:
The result is a threshold vector encoding the model’s difficulty profile across budgets.
Properties. Monotonicity: for , so forms a decreasing staircase marking where the policy transitions from uncertain to confident. Sensitivity: as improves, decreases for small , making calibrated thresholds a live signal of policy improvement that justifies periodic re-calibration (§ 4.4.3).
C-GRPO: Conformal-Guided Adaptive Training
Adaptive Group Construction via Conformal Confidence
C-GRPO resolves both failure modes simultaneously by anchoring group size to a conformal confidence criterion: stop sampling as soon as the empirical answer distribution is certifiably concentrated on a single answer. Formally, the APS prediction set at budget is the score-threshold set
where is Eq. (4) evaluated at a candidate answer in place of . By construction if and only if , which is exactly the event controlled by Theorem 2. Equivalently, the set lists answers in decreasing empirical frequency until the cumulative mass reaches ; unseen answers have score and enter only when . The implementation enumerates the observed answers, which coincides with Eq. (8) for , and evaluates membership deterministically (, ties ordered at random). The model is confident at budget when . The conformal group size is:
defaulting to if no singleton forms within .
Singletons resolve both failure modes: on easy queries a singleton forms at , providing a cheap non-zero-variance group; on hard queries the model expands to larger , increasing the chance of observing a positive reward.
Two-phase generation. To avoid generating completions upfront: (1) probe with samples—if a singleton forms, commit; (2) expand non-singleton queries via parallel micro-batching to the next grid points, halting at the first singleton or . This preserves high GPU utilization while still sparing all forward-pass compute for easy queries. Zero-variance groups () are pruned before the backward pass.
GRPO Objective with Adaptive Groups
Let be the adaptively-sized group. Because group sizes vary across queries, we normalize advantages across the full batch to prevent large groups from dominating gradient scale:
where is the query-group index of sample , is the mean reward within that group, and are the batch-level mean and standard deviation of raw advantages , and is a small numerical stabilizer (distinct from the safety margin in Eq. (14)).
The C-GRPO loss is:
where sums over completion tokens only, and is the Kullback-Leibler (KL) coefficient computed on the per-token estimator of Schulman et al. [ [22] (Appendix § 17). Because samples are drawn fresh from each step, no importance-sampling correction is needed; the gradient is an unbiased REINFORCE [26] estimator over the adaptive group.
Reward signal. Verifier-only binary reward ; degenerate completions ( tokens or unique characters) receive to suppress mode collapse.
Conformal Curriculum via Periodic Re-Calibration
The training dynamics and calibration are mutually reinforcing. As improves, the hard rate decreases naturally, reducing average compute per step. Every gradient steps we re-run calibration with the current policy:
Here denotes fresh completions sampled from the current trained policy for calibration query . As the policy improves, falls for small , the singleton criterion fires earlier for more queries, and the curriculum automatically de-emphasises easy problems—without any manually designed annealing schedule.
Adaptive Miscoverage Level via Coverage-Calibrated Selection. Standard split-conformal prediction treats the miscoverage level as a fixed hyperparameter set prior to calibration. In the online training setting, however, fixing creates a fundamental tension: a stringent target (e.g., demanding 90% coverage) is unachievable when the base model rarely produces correct answers at full budget , forcing universally and completely disabling adaptive sampling. Conversely, a permissive yields meaninglessly loose thresholds.
The root constraint is that the empirical solve rate at maximum budget,
is a hard upper bound on achievable marginal coverage: no threshold can yield when the model fails within samples on more than calibration examples.
We therefore propose coverage-calibrated selection: set the miscoverage level as a direct function of the current policy’s maximum-budget capability,
where is a safety margin ensuring sits meaningfully below 1 (we use ), guards against over-claiming when the policy is strong, and prevents degenerate guarantees for very weak base models.
As training progresses falls from (solve rate : loose thresholds, near-full budget, maximal exploration) through (: thresholds tighten for easy queries) to (: easy queries exit at ).
Joint update. At each re-calibration checkpoint, both and are updated with the current trained policy (LoRA [13] adapters enabled). Using the frozen reference model would yield a stagnant solve rate and throughout training.
Disjoint split for and . Computing and the quantiles from the same calibration scores makes the miscoverage level data-dependent. We therefore split the calibration set into disjoint parts: sets (Eqs. (13)–(14)) and computes the quantiles. Conditional on , is constant and is fresh, so Theorem 2 applies exactly with (tolerance for ); this is the recommended protocol (flag --split-delta), whereas the reported numbers use the original one (§ 5.1); offline, the split changed the budget of fewer than 4% of prompts.
Full training algorithm. Algorithm 1 summarizes C-GRPO, including auto- initialization, adaptive rollout allocation, GRPO updates, and periodic re-calibration.
Theoretical Analysis
Coverage Guarantee
Theorem 2 (Marginal Coverage). Let be i.i.d. from any distribution over . Fix , , and any fixed policy . Let be given by Eq. (7) using . Then:
Proof sketch. I.i.d. examples make the augmented score sequence exchangeable. Consequently, is uniform on . Choosing the -th order statistic therefore yields coverage . Appendix § 11 gives the full argument for both scores, and Appendix § 12 collects the exchangeability lemmas.
Remark 3. The guarantee is distribution-free and holds for any ; training only improves efficiency by lowering . Under covariate shift coverage degrades by at most per unit TV distance (Theorem 14, Appendix § 13).
Algorithm 1 C-GRPO Training with Adaptive
Require: Pre-trained reference ; datasets and disjoint calibration splits ; budget grid ; safety margin ; ceilings ; KL coefficient ; re-calibration period
Ensure: Fine-tuned policy ; thresholds ; miscoverage
Initialize (LoRA adapter)
▷ Eq. (14) with reference model solve rate
▷ Bootstrap thresholds from reference model
for step do
Sample mini-batch
for each query in parallel do
Probe: sample completions from
if under then ▷ Confident at minimal budget
else
Expand: sample additional completions via parallel micro-batching
, default
end if
Compute rewards ,
if then skip (zero-gradient group pruning)
end if
end for
Compute batch-normalized advantages via Eq. (10)
Update (Eq. (11))
if then
▷ Update coverage target with trained policy (LoRA ON)
▷ Joint update
end if
end for
return
Corollary 4 (False-stop control). Under the assumptions of Theorem 2, for every fixed and the APS score, : a singleton formed at budget is the gold answer except on an event of probability at most . For the first-success score with , stopping requires an executed, verified success among the first samples, so a stop never commits to a failing solution.
Theorem 2 does not predict when a singleton forms (that is governed by Proposition 6), and since is chosen adaptively the guarantee for the selected budget needs multiplicity control:
Theorem 5 (Coverage of the selected budget). Calibrate every threshold at level , . Then . Hence for any data-dependent budget , in particular , and the rule commits to a wrong singleton with probability at most . At level per budget (as in our experiments) the bound is ; across checkpoints sharing calibration prompts, level gives the guarantee simultaneously.
Proofs are in Appendix § 11, following conformal risk control over parameter grids [1, 3]. These guarantees are marginal over prompts: they bound the fraction of prompts truncated on a wrong answer, not , which would need selective conformal methods [9]. A wrong singleton ships no answer at training time and, having zero reward variance, is pruned (§ 4.4.1).
Monotone Efficiency of the Conformal Curriculum
Proposition 6 (Monotone Training Budget). Let be two policies, and let for denote the empirical pass rate at budget under policy . Suppose for all (i.e., is uniformly more accurate than ). Let* be the conformal threshold calibrated from . Then in expectation over the calibration set:
and consequently
Proof sketch. Higher pass rate stochastically dominates the score CDF for both (less mass above ) and ( stochastically smaller), lowering the order-statistic threshold and widening the singleton-forming set. Full proof with budget-grid telescoping and a finite-sample savings lower bound in Appendix § 11, Appendix § 15.
Corollary 7 (Self-Annealing Training Cost). At every re-calibration checkpoint at which the conditions of Proposition 6 hold between consecutive policies, periodic re-calibration every steps produces a non-increasing sequence of expected per-step training costs:
Where the conditions hold, C-GRPO therefore schedules less compute per step as the model becomes more capable, with no human-designed annealing. (Policy-drift and staleness bounds: Appendix § 14.)
Remark 8 (An idealized tendency, not a guarantee). Proposition 6 assumes uniform improvement, which RL violates locally: if accuracy on a slice degrades, its threshold rises at the next checkpoint and budget is restored (Appendix § 10). We read the decrease as the aggregate behaviour in Figure 3.
Figure 3. Rollout efficiency during training (GSM8K, Qwen2.5-7B, 300 steps). (a) Mean per step falls from to ; drops at each checkpoint mark auto- tightening. (b) Cumulative rollouts: 59.8% fewer than the fixed- ceiling (dashed; shaded: 5 738 rollouts saved).
Experiments
Experimental Setup
Datasets. Six benchmarks: code (HumanEval [4], MBPP [2]), math (GSM8K [6], MATH-500 [15], MATH-Algebra [12]), science (GPQA-Diamond [20]). Each uses a sequestered calibration split () and a disjoint sealed test split (–). Calibration protocol: all numbers use the original procedure (one calibration set for and the quantiles, no multiplicity correction); the disjoint split of § 4.4.3 is recommended.
Models & training. LoRA [13] (, , 5% dropout, AdamW [16] , batch 4, 300 steps, 2,000 training samples per benchmark) applied to Qwen2.5-7B-Instruct [19], Llama-3.1-8B-Instruct [10], and Gemma-3-4B-IT [8]. All baselines use the identical config; differences are in training algorithm only. LoRA is used for hardware reasons (one full-parameter run: § 5.5).
Baselines. Static GRPO [23] (); dynamic AERO [28], GDRO [17], GRESO [29], Reinforce-Ada/Est [27].
C-GRPO config. Budget grid ; re-calibration period steps; ; . is set automatically via Eq. (14)—zero per-dataset or per-model tuning.
Evaluation. : fixed-budget accuracy at exactly rollouts (majority vote for math/science, best-of- for code). : Ave@32 [27] single-rollout accuracy at (not greedy decoding). : mean rollouts/prompt at step 300 (proxy for per-step training FLOPs). Rollout savings counts training rollouts only; evaluation rollouts are charged separately in Table 10. At – the binomial standard error near 57% is 2.5–3.5 pp, so differences of a few points are within noise and we read accuracy results as matched, not improved.
| Phase (hours) | GRPO | GRPO | C-GRPO () |
| Training generation | 8.10 | 2.55 | 2.80 |
| Forward + backward | 3.60 | 1.65 | 0.75 |
| Re-calibration (6 events) | — | — | 0.25 |
| Micro-batch scheduling | — | — | 0.008 |
| Total wall-clock | 11.70 | 4.20 | 3.81 |
| Speed-up vs. | 1.00× | 2.79× | 3.07× |
Table 10. Wall-clock breakdown (GSM8K, Qwen2.5-7B-Instruct, 300 steps, 1×H100; original calibration procedure). C-GRPO includes all calibration generation, batched as large offline batches; with the released code’s default (training-batch-size) calibration the re-calibration share is larger (36–47% of training time in a 100-step check).
Code. Code (training, calibration, evaluation, baselines, launch scripts) is at https://github.com/AryaFayyazi/CGRPO (MIT license); pip install trl-cgrpo provides a TRL GRPOTrainer subpackage.
Main Results
Table 1 shows 49–70% training-rollout reduction across six benchmarks with equal or better accuracy at every fixed budget. Savings are highest where difficulty is bimodal (GPQA-Diamond 70%, GSM8K 68%); MATH-500 (50%) is uniformly harder, filling more of the grid. HumanEval retains for diverse solutions yet still reaches .
Table 1: C-GRPO on six benchmarks (Qwen2.5-7B-Instruct, step 300). Fixed-budget accuracy , mean training rollouts , and rollout savings vs. GRPO (training rollouts only; calibration cost in Table 10).
| Dataset | p@1 | p@8 | p@16 | p@32 | Rollout sav. | GPU-h | VRAM (GB) | Tok/s | |
| Mathematical reasoning | |||||||||
| GSM8K | 57.2 | 70.5 | 73.8 | 75.0 | 10.4 | 68% | 4.04 | 15.6 | 450 |
| MATH-500 | 52.5 | 66.8 | 71.2 | 71.5 | 15.9 | 50% | 1.61 | 21.6 | 1,880 |
| MATH-Algebra | 61.2 | 69.5 | 72.0 | 74.0 | 12.2 | 62% | 8.24 | 16.6 | 442 |
| Graduate-level science | |||||||||
| GPQA-Diamond | 32.8 | 35.5 | 38.4 | 39.0 | 9.7 | 70% | 0.65 | 26.6 | 1,826 |
| Code generation | |||||||||
| HumanEval | 36.5 | 78.2 | 95.5 | 97.2 | 16.3 | 49% | 0.25 | 19.8 | 1,931 |
| MBPP | 60.5 | 78.5 | 83.0 | 83.5 | 10.0 | 69% | 1.60 | 16.6 | 453 |
Table 1.
Comparison with Baselines
C-GRPO attains the highest pass@1 on all four benchmarks while using the fewest rollouts, though its margins over the strongest baselines (0.4–2.4 pp) are within binomial noise. It averages , versus at least 14.8 for dynamic baselines and 32.0 for static GRPO, as shown in Table 2. Heuristic methods reduce cost but lack coverage guarantees, and GDRO improves accuracy under a compute-neutral constraint while still trailing C-GRPO on every metric. Together, these results show that conformal allocation matches or slightly improves accuracy while substantially reducing rollouts.
Table 2: Pass@1 (%) and training compute (Qwen2.5-7B-Instruct, step 300). : mean rollouts/prompt (lower is better). : pass@1 gap vs. GRPO on GSM8K. GPU-h / VRAM / Tok/s: total training wall-clock (including all calibration generation for C-GRPO), peak GPU memory, and effective rollout generation throughput on 1H100 (all methods, 300 steps). GDRO uses a compute-neutral global budget ().
| Model | Algorithm | GSM8K | HumanEval | MBPP | MATH-500 | pass@1 | Type | GPU-h | VRAM (GB) | Tok/s | |
| Qwen2.5 7B-Instruct | GRPO [23] | 54.8 | 33.3 | 56.0 | 48.5 | — | 32.0 | static | 11.7 | 26.8 | 3,180 |
| GDRO [17] | 56.8 | 34.5 | 58.2 | 51.0 | +2.0 | 32.0 | dynamic | 11.7 | 26.8 | 3,180 | |
| AERO [28] | 56.1 | 34.0 | 57.5 | 50.2 | +1.3 | 16.6 | dynamic | 6.1 | 22.4 | 4,270 | |
| GRESO [29] | 54.7 | 33.1 | 55.8 | 48.3 | −0.1 | 15.6 | dynamic | 5.7 | 21.9 | 4,410 | |
| Reinforce-Ada [27] | 55.4 | 33.8 | 56.5 | 49.5 | +0.6 | 14.8 | dynamic | 5.4 | 21.4 | 4,520 | |
| Reinforce-Est [27] | 55.1 | 33.5 | 56.1 | 49.0 | +0.3 | 15.2 | dynamic | 5.6 | 21.6 | 4,460 | |
| C-GRPO (ours) | 57.2 | 36.5 | 60.5 | 52.5 | +2.4 | 10.4 | conformal | 3.8 | 18.6 | 5,240 |
Table 2.
On GSM8K, C-GRPO obtains +2.4 pp over GRPO and +0.4 pp over GDRO with only one-third as many rollouts ( versus 32.0); these differences are within the 2.5–3.5 pp standard error of a single run, so the result is matched accuracy at about a third of the rollouts (Figure 3).
Cross-Model Generalisation
Table 3 shows zero-shot transfer across architectures. The calibration split determines separately for each model–dataset pair. On GSM8K, Llama-3.1-8B attains 82% rollout savings with and of 91.0% because its stronger baseline at lets the procedure tighten to 0.05 automatically. Gemma-3-4B instead receives a looser , reflecting its weaker baseline. Efficiency therefore scales with model strength without manual retuning.
| Dataset | Model | p@1 | p@8 | p@16 | p@32 | Savings | ||
| GSM8K | Qwen2.5-7B-Inst. | 0.12 | 57.2 | 70.5 | 73.8 | 75.0 | 10.4 | 68% |
| Llama-3.1-8B-Inst. | 0.05 | 72.5 | 85.0 | 91.0 | 91.5 | 5.7 | 82% | |
| Gemma-3-4B-IT | 0.18 | 48.5 | 62.0 | 65.5 | 66.0 | 14.2 | 56% | |
| MBPP | Qwen2.5-7B-Inst. | 0.10 | 60.5 | 78.5 | 83.0 | 83.5 | 10.0 | 69% |
| Llama-3.1-8B-Inst. | 0.08 | 65.5 | 81.0 | 86.5 | 87.0 | 8.5 | 73% | |
| Gemma-3-4B-IT | 0.15 | 50.2 | 65.0 | 70.0 | 71.0 | 12.5 | 61% | |
| MATH-500 | Qwen2.5-7B-Inst. | 0.25 | 52.5 | 66.8 | 71.2 | 71.5 | 15.9 | 50% |
| Llama-3.1-8B-Inst. | 0.15 | 60.0 | 72.5 | 78.0 | 78.5 | 12.0 | 62% | |
| Gemma-3-4B-IT | 0.30 | 42.5 | 55.0 | 60.0 | 60.5 | 18.5 | 42% |
Table 3. Cross-model generalisation at step 300 on held-out test sets. is auto-selected per model–dataset pair without manual tuning.
Matched-Compute Baselines and End-to-End Cost
A baseline may simply be larger than necessary, so we train GRPO at fixed under an otherwise identical pipeline (full sweep: Table 7, Appendix § 16). On GSM8K, static pass@1 is 52.8 (), 53.1 (, matched to C-GRPO’s ), 54.2 (), and 54.8 (), saturating at 55.3 for (44.5 GPU-h), against 57.2 for C-GRPO (3.8 GPU-h); on MATH-500, matched reaches 47.9 against 52.5. These 4.1 and 4.6 pp gaps are of the order of the single-run standard error, so we read them as at-least-matched accuracy at equal compute, not a significant gain; C-GRPO finds the matched (10 versus 16) from calibration alone.
| Benchmark | Method | p@1 | p@8 | p@16 | p@32 | GPU-h | |
| GSM8K | GRPO | 2 | 48.2 | 64.0 | 67.5 | 68.0 | 1.8 |
| GSM8K | GRPO | 4 | 51.0 | 68.2 | 70.1 | 71.2 | 2.4 |
| GSM8K | GRPO | 8 | 52.8 | 70.1 | 72.0 | 73.1 | 3.6 |
| GSM8K | GRPO | 10 (matched) | 53.1 | 70.5 | 72.4 | 73.5 | 4.2 |
| GSM8K | GRPO | 16 | 54.2 | 71.8 | 73.2 | 74.1 | 6.2 |
| GSM8K | GRPO | 32 | 54.8 | 72.0 | 73.5 | 74.6 | 11.7 |
| GSM8K | GRPO | 64 | 55.1 | 72.2 | 73.8 | 74.8 | 22.8 |
| GSM8K | GRPO | 128 | 55.3 | 72.4 | 74.0 | 75.0 | 44.5 |
| GSM8K | C-GRPO | 10.4 | 57.2 | 70.5 | 73.8 | 75.0 | 3.8 |
| MATH-500 | GRPO | 2 | 41.5 | 55.2 | 58.0 | 59.1 | 1.9 |
| MATH-500 | GRPO | 8 | 45.8 | 60.1 | 63.4 | 64.0 | 4.1 |
| MATH-500 | GRPO | 16 (matched) | 47.9 | 62.5 | 66.0 | 66.8 | 8.1 |
| MATH-500 | GRPO | 32 | 48.5 | 63.2 | 67.0 | 67.8 | 15.9 |
| MATH-500 | GRPO | 128 | 49.2 | 64.0 | 68.1 | 68.9 | 61.2 |
| MATH-500 | C-GRPO | 15.9 | 52.5 | 66.8 | 71.2 | 71.5 | 5.1 |
Table 7. Full static- sweep (Qwen2.5-7B-Instruct, 300 steps, 1×H100). Every static row is trained at fixed .
End-to-end cost. Rollout savings count training rollouts only. Table 10 (Appendix § 16) charges every calibration completion to C-GRPO (6 200 32 = 38,400 completions). This cost depends strongly on batching: as large offline batches without gradient state it was 0.25 h (6.6% of wall-clock) in our setup, whereas the released code, which calibrates at the training batch size, spent 46–47% of training time in a 100-step check, so wall-clock savings there are smaller than the rollout savings. Zero-variance pruning cuts forward–backward time from 1.65 h to 0.75 h versus static ; in Table 10’s setting the net speed-up over is 3.07×.
Longer horizon and a stronger base (Appendix § 16). Over 1,000 GSM8K steps falls 18.2 → 4.2 and pass@1 rises 52.1 → 64.5 (static: 59.1). Full-parameter fine-tuning of Qwen2.5-Math-7B-Instruct (3 seeds): C-GRPO 88.9±0.2 at (82% savings) versus 88.5±0.3 for static (matched, not improved) and 86.1±0.4 for .
Training Dynamics
Reward dips align with the checkpoints (Figure 2) and recover within steps; the cumulative budget grows sub-linearly (Figure 3), consistent with the idealized tendency of Corollary 7. Under the original procedure the worst under-coverage over all (budget, checkpoint) pairs is −0.2 pp (mean slack +0.3 pp), inside the tolerance (Appendix § 8). Ablations: Appendix § 9; shift: Appendix § 13.

Figure 2. Training dynamics (GSM8K, Qwen2.5-7B). C-GRPO sustains the highest reward and lowest KL divergence throughout training. Periodic dips align with re-calibration events; the policy recovers to a higher reward peak within steps each time.
Discussion and Limitations
Theory versus the deployed loop. Theorem 2 assumes a fixed policy and a fresh calibration split, whereas the loop reuses calibration prompts and selects adaptively; the disjoint split and Theorem 5 restore exactness, but reported numbers use the original procedure. Scope. A verifier is required, coverage needs exchangeability (Theorem 14), and most results use LoRA at 4–8B scale and 300 steps (longer and full-parameter runs: one model each).
Conclusion
We presented C-GRPO, a conformal extension of GRPO that replaces fixed rollout budgets with calibrated, task-specific stopping thresholds (task-aware scores, automatic miscoverage selection, periodic re-calibration), with finite-sample coverage and false-stop control. Across six benchmarks, C-GRPO reduces training rollouts by 49–70% at matched accuracy, also against fixed- GRPO at matched compute, and transfers across model families (up to 82% savings). Accuracy gains are within noise: the contribution is iso-accuracy compute reduction.
Acknowledgements and Disclosures
Funding. This work received no external funding or third-party support during the last 36 months. Competing interests. The authors declare no competing interests: they have no financial relationships outside the submitted work that could be perceived as influencing it.
Coverage Validation
Theorem 2 guarantees that the empirical conformal coverage meets the nominal target at every re-calibration checkpoint. Table 4 confirms this across all six benchmarks on the held-out test split; Figure 4 shows the per- scatter on GSM8K.
| Dataset | Target | Empirical | Slack | |
| GSM8K | 0.12 | 84.0% | 83.8% | −0.2 pp |
| MATH-500 | 0.25 | 74.0% | 74.7% | +0.7 pp |
| GPQA-Diamond | 0.48 | 44.0% | 43.9% | −0.1 pp |
| MATH-Algebra | 0.21 | 80.0% | 80.0% | 0 pp |
| HumanEval | 0.90 | 90.0% | 90.9% | +0.9 pp |
| MBPP | 0.23 | 76.0% | 76.3% | +0.3 pp |
Table 4. Coverage validation across six benchmarks (Qwen2.5-7B, held-out test, step 300). : auto-selected initial miscoverage. Empirical coverage matches or exceeds the nominal target on all benchmarks within the finite-sample tolerance .
Figure 4. Per- coverage scatter on GSM8K (Qwen2.5-7B). Each point is one pair. Every point lies on or above the diagonal (empirical target), jointly validating Theorem 2 for all and all training steps simultaneously. The densest cluster at 80–90% corresponds to –16 during mid-training; points move rightward as auto- tightens targets with policy improvement.
All coverages lie within pp of nominal, respecting the finite-sample bound from Theorem 2; individual per- coverages all satisfy the theorem. These numbers use the original procedure. Because auto- tightens as the solve rate rises, coverage must be read against the target at the same checkpoint (the nominal target moves, which is what the rightward drift of the points in Figure 4 shows), so we report slack, empirical minus nominal: across all (budget, checkpoint) pairs the worst under-coverage is −0.2 pp (GSM8K, step 300), the mean slack over the six benchmarks is +0.3 pp, and the largest deviation in either direction is +0.9 pp, which is over-coverage and therefore conservative. Only under-coverage threatens validity, and −0.2 pp is inside the finite-sample tolerance and the pp binomial noise at .
Ablation Studies
Table 5 ablates the three design choices unique to C-GRPO on Qwen2.5-7B, 300 steps, held-out test: (A) auto- initialisation, (B) re-calibration frequency , and (C) the discrete budget grid .
| Strategy | p@1 | p@16 | Cov. | Triv. | p@1 | p@16 | OH | p@1 | p@16 | # | |||||
| Fixed .05 | 54.5 | 70.2 | 29.5 | 94.2 | 200 | 25 | 57.3 | 73.9 | 10.3 | +12 | 51.0 | 69.5 | 18.4 | 3 | |
| Fixed .20 | 56.0 | 72.5 | 18.4 | 81.5 | 30 | 50 | 57.2 | 73.8 | 10.4 | base | 52.5 | 71.2 | 15.9 | 5 | |
| Auto | 57.2 | 73.8 | 10.4 | 83.8 | 0 | 100 | 55.8 | 72.1 | 13.5 | −6 | 52.3 | 70.8 | 16.2 | 6 | |
| 54.5 | 70.0 | 18.2 | −12 |
Table 5. Ablation studies for C-GRPO. Bold = default configuration. “Triv.” = training steps where all thresholds are inactive (curriculum reduces to vanilla GRPO). “OH” = overhead steps per checkpoint relative to baseline. “#” = number of distinct budget levels. All rows use the original calibration procedure.
(A) Auto- initialisation. Fixed requires 95% coverage within rollouts before the curriculum activates. An initialised Qwen2.5-7B solves only of GSM8K with shots at step 0, so this target is unattainable for of 300 training steps—reducing C-GRPO to vanilla GRPO for two-thirds of training, which balloons to 29.5 and depresses final pass@1 to 54.5%. Fixed works reasonably on GSM8K but triggers trivial phases on harder benchmarks (e.g. GPQA-Diamond with base pass@). Auto- sets the threshold such that calibration pass rate satisfies the required coverage from step 0, guaranteeing the curriculum is active from the start on all six benchmarks and all three model families, with zero manual tuning.
(B) Re-calibration frequency. The default steps maximises the accuracy–efficiency tradeoff. Halving to adds +12% overhead per training step (calibrating completions for prompts at each checkpoint) without measurably improving accuracy, since the policy evolves slowly enough that 50-step-old thresholds remain accurate. (static thresholds fixed at step 0) prevents rollout savings from compounding as training progresses, because the curriculum cannot track the policy’s growing competence, ballooning to 18.2. The calibration cost itself is 0.50, 0.25, 0.13 and 0.04 h for , 50, 100, (12, 6, 3, 1 events), so halving to 25 buys nothing for twice the calibration cost, while 100 and let thresholds go stale. The multiplicity regime of Theorem 5 calibrates each budget at (about 0.02 for and ), which is stricter than the fixed row of Table 5; it is statistically valid but, for weak base models, reintroduces the saturated thresholds that auto- removes. The default per-budget calibration gives level- false-stop control at every budget (Corollary 4), with as the simultaneous worst case. Per-prompt miscoverage events are strongly positively dependent across budgets (a prompt whose gold answer is rarely sampled is missed at every ), so the union bound is loose, as the joint empirical validation in Figure 4 shows.
(C) Budget grid. The geometric-doubling grid is Pareto-optimal. The coarse grid forces moderate-difficulty prompts to jump from directly to , raising by rollouts/prompt and reducing final p@16 by 1.7 pp. The fine grid adds greedy probes that slightly reduce cost on trivially easy prompts but introduce calibration noise on borderline prompts, yielding no net gain on MATH-500.
Curriculum Stability: No Catastrophic Forgetting
Remark 9 (No catastrophic forgetting of easy queries). Tightening thresholds reduces compute on easy queries but does not degrade the policy’s ability to answer them, for three complementary reasons. (i) Zero-gradient pruning. An easy query that forms a singleton at with all-correct rewards () is pruned before the backward pass (§ 4.4.1), so the model receives no gradient update on it—neither positive nor negative. The update cannot be pushed away from correct behaviour on easy queries. (ii) KL anchoring. The term in (Eq. (11)) penalises any deviation from uniformly across all queries, bounding total policy drift regardless of which queries currently drive the gradient. (iii) Self-correcting curriculum. If accuracy on any query class were to decrease, the empirical score distribution would worsen, raising at the next re-calibration checkpoint and restoring those queries to larger budgets—providing gradient signal until performance recovers. Together, these mechanisms make C-GRPO’s curriculum self-stabilising: hard queries receive more compute, but easy queries are protected rather than neglected.
Full Proofs
This section provides complete, self-contained proofs of Theorem 2, Proposition 6, and Corollary 7, with full exchangability arguments omitted from the main paper for brevity.
Proof of Theorem 2 (Marginal Coverage)
Proof. We prove both the lower and upper bounds of Eq. (15).
Setup. Fix a budget , score , and policy . For each example , draw independently of all other examples. Define .
Exchangeability. Since are i.i.d. from and completions are drawn i.i.d. from , the joint distribution of is exchangeable: for any permutation , . In particular, the rank of among is uniformly distributed on .
Threshold construction. Let and , the -th order statistic of .
Lower bound. .
Upper bound. .
APS continuity. The randomized term in (Eq. (4)) makes the score distribution continuous on , ensuring for . Hence the rank of is strictly uniform on with no tied-rank ambiguity.
First-success score. with the position of the first correct completion, or if none exists. The event has probability for local pass rate . Because is continuous, for all , giving strict exchangeability and strict uniformity of the rank.
Proof of Proposition 6 (Monotone Training Budget)
Proof. Let denote the CDF of under policy .
APS case. By hypothesis, . Higher empirical pass rate implies stochastically smaller APS score (less mass lies strictly above in the frequency histogram), so for all , i.e. (stochastic dominance). Monotonicity of order statistics then gives in expectation, establishing .
Code case. Higher accuracy means under is stochastically smaller: for all . Since , the same stochastic-dominance argument applies, yielding .
Budget consequence. By Eq. (9), . A smaller threshold makes the singleton condition easier to satisfy at small , weakly increasing the set of queries that exit at each budget level. Formally, for any , let . Then (more queries exit early), and .
Proof of Corollary 7 (Self-Annealing Training Cost)
Proof. At each re-calibration checkpoint , is calibrated using the current trained policy (Algorithm 1, lines 16–17). GRPO training with positive rewards drives policy improvement: for the trained policy at step , for all (expected under the reward-positive update). Applying Proposition 6: , and consequently . Since this holds at every checkpoint, the sequence is non-increasing.
Proof of Corollary 4 (False-Stop Control)
Proof. Fix and one draw of shared by all candidate answers. Suppose , say . Exactly one of two disjoint cases holds. If the stop is correct. If , then because has no other element. Hence
and Theorem 2 gives . For the first-success score and , forces to exist, i.e. at least one of the first samples passes all unit tests.
Proof of Theorem 5 (Coverage of the Selected Budget)
Proof. For each , Theorem 2 at level gives (with the convention if , for which coverage is trivially 1). A union bound over the budgets gives . On the complementary event holds for every , hence also at any chosen as a function of the data. A wrong commit at means , which implies , so its probability is at most . At level per budget the same union bound gives . Across checkpoints, Theorem 2 holds marginally for each frozen policy and its fresh completions, so a union bound over the (budget, checkpoint) pairs at level gives the claim simultaneously over checkpoints, even though they share calibration prompts.
Exchangeability of Nonconformity Scores
Conformal validity depends only on the exchangeability of the augmented score sequence , not on a particular parametric model. The next two lemmas establish this property for the APS and code-based nonconformity scores introduced in § 4.2.
Lemma 10 (Exchangeability of APS Score). Let be i.i.d. from and let independently across examples and positions. Define . Then is exchangeable.
Proof. Step 1: Symmetry within an example. depends on only through the empirical frequency histogram , which is a symmetric (permutation-invariant) function of . The randomized tiebreaker is drawn independently, making continuous and well-defined regardless of ties.
Step 2: Independence across examples. Since are mutually independent across , the scores are i.i.d. (a special case of exchangeability). For any permutation ,
.
Lemma 11 (Exchangeability of First-Success Score). Under the same i.i.d. setup, define . Then is exchangeable.
Proof. Intra-example non-symmetry. Unlike the APS score, is not symmetric in : it records the ordinal position of the first passing completion, which depends on the ordering. This is by design—the ordering reflects the sequential generation process.
Inter-example exchangeability. Conformal validity requires exchangeability of example-level scores , not intra-example symmetry. Since are all i.i.d. (identical marginal distribution of the tuple), so are . Hence is exchangeable.
Remark 12 (Why intra-example ordering does not matter). The first-success score’s sensitivity to ordering within captures the geometric distribution of success position: for local pass rate . This is a continuous family in , ensuring the score distribution is continuous and ties occur with probability zero—preserving the strict uniformity of the test rank needed for exact (rather than conservative) coverage.
Remark 13 (Exchangeability under periodic re-calibration). At each re-calibration checkpoint , the policy is fixed before re-evaluating the calibration set. The exchangeability argument holds for each checkpoint independently: the calibration examples and one test example are i.i.d. under fixed . Across checkpoints, the policy changes, but coverage guarantees are instantiated separately at each checkpoint, maintaining validity throughout training.
Robustness to Distribution Shift
Standard split-conformal prediction assumes the calibration and test examples are exchangeable (i.i.d. from the same distribution ). In deployment, the test distribution may differ from the calibration distribution . We formalise how coverage degrades under bounded covariate shift and validate empirically.
Theorem 14 (Coverage under Bounded Covariate Shift). Let and be distributions over with total-variation distance . Let be calibrated from i.i.d. examples of at level . Then for a test point from :
Proof. By the coupling characterization of total variation, there exists a coupling with , , and . For any measurable event and any fixed threshold :
By Theorem 2 (applied under ): . Combining:
Remark 15. Tibshirani et al. [ [24] extends the result of Tibshirani et al. [ [24] to the online training setting of C-GRPO: coverage degrades by at most per unit of TV distance between training/calibration and test distributions, gracefully interpolating between the fully valid in-distribution case () and the completely uncontrolled out-of-distribution case. Re-calibration with on the target calibration split at each checkpoint further mitigates shift by adapting thresholds to the local difficulty of the test domain.
Empirical validation under distribution shift. Table 6 reports coverage for a model trained and calibrated on GSM8K, evaluated on test distributions of increasing dissimilarity. is estimated from held-out query difficulty features (solve rate, response-length variance, and lexical overlap with calibration prompts). Even under severe domain shift (GPQA-Diamond), empirical coverage remains well above the theoretical lower bound (Eq. (19)), confirming that C-GRPO’s thresholds are robust in practice.
| Test Domain | Shift | Theo. LB | Empirical | Margin | |
| GSM8K (in-dist.) | none | 0.00 | 87.5% | 87.9% | +0.4 pp |
| MATH-Algebra | mild | 0.06 | 81.5% | 83.6% | +2.1 pp |
| MATH-500 | moderate | 0.15 | 72.5% | 76.8% | +4.3 pp |
| GPQA-Diamond | severe | 0.29 | 58.5% | 70.4% | +11.9 pp |
Table 6. Coverage under distribution shift (Qwen2.5-7B, trained and calibrated on GSM8K, step 300, , nominal target 88%). : TV distance between GSM8K calibration and test distribution. Theoretical bound: . Empirical coverage consistently exceeds the theoretical lower bound.
Policy Stability and Calibration Staleness
Re-calibration every steps relies on the assumption that the policy does not drift so severely between checkpoints that the old thresholds become vacuous. We bound the expected policy displacement and consequent threshold staleness.
Theorem 16 (KL-Bounded Policy Displacement). Let be any starting policy, and let be the policy after gradient steps of with KL coefficient and learning rate . Under -smoothness of the policy log-probabilities and bounded rewards , the expected accumulated KL divergence satisfies:
where is the expected squared gradient norm at convergence. By Pinsker’s inequality, .
Proof. The KL term in (Eq. (11)) imposes a soft constraint on . Treating the training as projected gradient ascent, each step increases the policy reward by but accumulates KL cost in the objective. Summing over steps and bounding the per-step KL increment by (from the -weighted KL penalty balancing the gradient), gives Eq. (20).
Corollary 17 (Calibration Staleness Bound). Let be calibrated at step and used until step . The expected threshold error due to policy drift is bounded by:
where depends on the local slope of the score CDF near .
Proof. Over steps after checkpoint , the policy drifts from to . By Theorem 16 applied to the -step window, grows as . A change in affects the score CDF and hence the order statistic proportionally to the local CDF slope via the delta method applied to order statistics, giving Eq. (21).
Remark 18 (Justification for ). Eq. (21) shows threshold staleness grows as , while calibration overhead is . The Pareto-optimal balances the two: the ablation in Appendix § 9 confirms achieves this, consistent with the bound.
Finite-Sample Lower Bound on Rollout Savings
We lower-bound the expected rollout savings of C-GRPO over fixed- GRPO in terms of the calibrated threshold vector .
Proposition 19 (Lower Bound on Expected Rollout Savings). Let be a fixed policy with conformal thresholds calibrated at level . Let be the singleton probability at budget for query . The expected per-prompt rollout savings versus fixed- GRPO are:
Proof. Expand by telescoping over the budget grid:
where . A query exits at or earlier if and only if a singleton forms at some , saving rollouts. Lower-bounding by the event that all levels form singletons (dropping the union by inclusion) and taking expectations yields Eq. (22).
Remark 20 (Connection to conformal thresholds). is directly controlled by : a lower threshold concentrates more probability mass on a single answer at budget , increasing . Eq. (22) therefore shows that the rollout savings are monotonically non-decreasing in the policy improvement that drives downward—consistent with Proposition 6 and empirically confirmed by the 49–82% savings in Table 3.
Static- Sweep, Longer Horizon, and Stronger Base
All tables in this section were produced with the original calibration procedure (one calibration set for both and the quantiles, per-budget , no multiplicity correction). Standard error of an accuracy near 57% is 2.5–3.5 pp at –400.
| Step | Method | p@1 | Savings | |
| 100 | C-GRPO (Qwen2.5-7B) | 52.1 | 18.2 | 43% |
| 300 | C-GRPO (Qwen2.5-7B) | 57.2 | 10.4 | 68% |
| 600 | C-GRPO (Qwen2.5-7B) | 61.8 | 6.1 | 81% |
| 1000 | C-GRPO (Qwen2.5-7B) | 64.5 | 4.2 | 87% |
| 1000 | Static GRPO () | 59.1 | 32.0 | — |
| 1000 | C-GRPO (Llama-3.1-8B) | 76.2 | 3.1 | 90% |
Table 8. Longer horizon (GSM8K). Savings (training rollouts only).
Reading. C-GRPO versus static is +0.4 pp, or 1.1 pooled standard deviations: statistically indistinguishable, a match at 5.5× fewer rollouts rather than an accuracy gain. Static versus C-GRPO is −2.8 pp (6.3 pooled standard deviations), so matching full-budget accuracy at is not obtained by lowering the static budget. In this configuration 76% of prompts are correct after the probe, and wall-clock is 2.2 h against 11.7 h for static . The coverage under this original procedure is in Appendix § 8.
Reproducibility, Statistical Reporting, and Compute
Reproducibility protocol. All experiments use the same training and evaluation pipeline across methods unless explicitly stated otherwise. Each benchmark is split into a sequestered calibration set, a training set, and a disjoint held-out test set. The calibration split is used only to estimate conformal thresholds and the auto-selected miscoverage level , while the held-out test split is used only for final reporting. We use calibration examples per benchmark and – held-out test examples, depending on benchmark size. The six benchmarks are GSM8K, MATH-500, MATH-Algebra, GPQA-Diamond, HumanEval, and MBPP.
Training configuration. All model-training experiments use LoRA fine-tuning with rank , scaling , dropout 0.05, AdamW optimizer with learning rate , batch size 4, and 300 training steps with 2,000 training examples per benchmark. Unless otherwise noted, the C-GRPO budget grid is , the recalibration period is , the KL coefficient is , and the auto- safety margin is . Static and dynamic baselines use the same model, data, optimizer, LoRA configuration, number of training steps, and verifier, differing only in the rollout-allocation algorithm. The KL term uses the non-negative per-token estimator with , where is the frozen reference policy, averaged over completion tokens and the batch to avoid sequence-level overflow on long completions.
Evaluation protocol. We report fixed-budget pass@ accuracy using exactly completions at evaluation time. For math and science benchmarks, answers are canonicalized and evaluated by exact-match verification; for code benchmarks, completions are evaluated by execution against unit tests. For pass@, math and science use majority vote over extracted answers, while code uses best-of- execution success. The reported is the mean number of rollouts consumed per prompt during training at step 300, and “Savings” is computed relative to fixed-budget GRPO with .
Statistical reporting. The main metrics are empirical means over held-out test examples using deterministic verifiers. For binary pass@ outcomes, uncertainty can be computed by the standard binomial standard error
and for rollout-cost quantities we compute the empirical mean over prompts. For conformal coverage, we report the nominal target, empirical coverage, and the finite-sample tolerance induced by split conformal calibration, namely . Coverage validation is reported for all six benchmarks in Appendix § 8; all empirical coverages lie within the expected finite-sample tolerance. Training-seed variance is reported only where stated (three seeds for the full-parameter experiment, Table 9); elsewhere uncertainty is the binomial standard error over the test set, and differences below roughly 2–3 pp should not be interpreted.
| Method | Budget | p@1 | p@32 | Savings | |
| GRPO (static) | 32.0 | — | |||
| GRPO (static) | 8.0 | 75% | |||
| C-GRPO | adaptive | 5.8 | 82% |
Table 9. Stronger base, full-parameter fine-tuning (GSM8K, Qwen2.5-Math-7B-Instruct, base p@1 = 82.4, 300 steps; mean std over three seeds).
Compute resources. All experiments were run on NVIDIA H100 GPUs with 80 GB of GPU memory. Each reported training job uses one H100 GPU for model inference and LoRA updates, with CPU workers used for answer extraction, deterministic verification, code execution tests, logging, and metric computation. Table 1 and Table 2 report training GPU-hours, peak GPU memory, and effective rollout-generation throughput for the main experiments on a single H100. The largest reported runs are the fixed-budget GRPO and GDRO baselines, which require approximately 11.7 GPU-hours and 26.8 GB peak VRAM for 300-step Qwen2.5-7B-Instruct training, while C-GRPO requires approximately 3.8 GPU-hours (including all calibration generation) and 18.6 GB peak VRAM in the corresponding setting; Table 10 gives the phase-level breakdown. The full project required additional preliminary runs for prompt formatting, verifier debugging, and grid selection; these exploratory runs are not included in the main compute tables because they do not affect the reported comparisons. Launch scripts, software versions, model identifiers, dataset-loading code and calibration/evaluation scripts are in the released repository (§ 5.1).
Limitations, Asset Use, and Responsible Release
Limitations. C-GRPO assumes a verifier is available for each task, either through exact answer matching for closed-form reasoning or executable tests for code. This makes the method directly applicable to mathematics, science QA with canonical answers, and program synthesis, but less direct for open-ended generation tasks where correctness is subjective or multi-dimensional. The conformal guarantee also relies on exchangeability between calibration and evaluation examples under a fixed policy at each recalibration checkpoint; under distribution shift, coverage can degrade as described in Appendix § 13. Finally, the current implementation uses a discrete budget grid and binary verifier rewards; extending the method to continuous budgets, graded rewards, preference-model scores, and broader RLHF settings is an important direction for future work.
Existing assets. The paper uses public or previously released benchmarks and models, including GSM8K, MATH-500, MATH-Algebra, GPQA-Diamond, HumanEval, MBPP, Qwen2.5-7B-Instruct, Llama-3.1-8B-Instruct, and Gemma-3-4B-IT. We cite the original sources for these assets in the main paper and references. The released repository includes the model registry and the dataset-loading code that reconstructs the calibration, training and held-out evaluation splits. Qwen2.5 is used under Apache-2.0, Llama-3.1 under the Llama 3.1 Community License, and Gemma-3 under the Gemma Terms of Use.
New assets. The new assets introduced by this work are the C-GRPO training algorithm, calibration code, nonconformity-score implementations, auto- selection logic, budget-allocation scripts, evaluation scripts, and experiment launch configurations. These assets do not include a newly pretrained foundation model or scraped dataset. The implementation and reproduction instructions are released (see Code availability below).
Societal impact and responsible use. The intended positive impact of C-GRPO is to reduce the compute cost of reinforcement learning for reasoning-oriented language models while adding statistical calibration to the rollout-allocation rule. This can make efficient RL training more accessible and reduce unnecessary energy use. A possible negative impact is that more efficient RL could also lower the cost of improving models for undesirable applications. We mitigate this by releasing training infrastructure and evaluation scripts rather than a new unrestricted pretrained model, and by focusing the empirical evaluation on standard reasoning and coding benchmarks with deterministic verifiers.
Code availability. The implementation is released at https://github.com/AryaFayyazi/CGRPO under the MIT license: training, calibration (APS and first-success scores, auto-, and the optional disjoint split), evaluation, the baselines, analysis scripts, Slurm launchers, and REPRODUCTION.md with the commands and settings. A TRL integration, CGRPOTrainer, is available as pip install trl-cgrpo. Trained adapters are not included in the current release.
NeurIPS Paper Checklist
Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper’s contributions and scope?
Answer: [Yes].
Justification: The abstract and introduction state that C-GRPO replaces fixed GRPO rollout budgets with calibrated conformal thresholds, proves finite-sample coverage, false-stop control and a multiplicity-corrected guarantee for the selected budget, shows a non-increasing expected budget under an idealized monotone-improvement condition, and empirically reduces training rollouts at matched accuracy; accuracy differences over baselines are explicitly described as within binomial noise. These claims are supported by § 4–§ 5, Table 1–Table 10, and the appendix coverage and ablation studies.
Limitations
Question: Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes].
Justification: § 5.7 and Appendix § 18 discuss verifier dependence, exchangeability assumptions, distribution-shift sensitivity, discrete budget grids, and the current focus on binary verifier rewards. The conclusion also identifies extensions beyond binary rewards, exchangeable calibration, and discrete budget grids as future work.
Theory assumptions and proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?
Answer: [Yes].
Justification: Theorem 2, Corollary 4, Theorem 5, Proposition 6, Corollary 7 and Theorem 14 state the relevant assumptions, including a fixed policy, i.i.d./exchangeable calibration and test examples, a monotone-improvement condition for the budget results (assumed explicitly in Corollary 7), and bounded total-variation shift. Full proofs and exchangeability lemmas are provided in the appendix.
Experimental result reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main experimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper, regardless of whether the code and data are provided or not?
Answer: [Yes].
Justification: § 5.1 and Appendix § 17 specify datasets, calibration/test split sizes, model backbones, LoRA configuration, optimizer, learning rate, batch size, training steps, budget grid, recalibration period, KL coefficient, evaluation protocol, and baseline configurations. The release plan further states that scripts and exact reproduction instructions will be made available upon acceptance.
Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instructions to faithfully reproduce the main experimental results, as described in supplemental material?
Answer: [Yes].
Justification: The code is released at https://github.com/AryaFayyazi/CGRPO (MIT license) with training, calibration, evaluation, baselines, launch scripts, and REPRODUCTION.md, which lists the commands and settings; all datasets and base models are public, and pip install trl-cgrpo provides a TRL trainer. Trained adapters are not included, and reproducing the exact numbers requires GPU compute (Appendix § 17).
Experimental setting/details
Question: Does the paper specify all the training and test details, e.g., data splits, hyperparameters, how they were chosen, type of optimizer, necessary to understand the results? Answer: [Yes].
Justification: Section 5.1 specifies the six benchmarks, calibration and held-out test sizes, LoRA hyperparameters, optimizer, learning rate, batch size, training steps, training-sample count, budget grid, recalibration period, safety margin, KL coefficient, and evaluation metrics. Appendix J consolidates these details for reproducibility.
Experiment statistical significance
Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Answer: [Yes].
Justification: Appendix J explains that is a binary held-out empirical mean and gives the binomial standard-error formula for the reported test sizes. For conformal coverage, Appendix A reports empirical coverage against nominal targets and the finite-sample tolerance , which is the relevant statistical uncertainty for the calibration guarantee. Section 5.1 states that differences of a few points are within the 2.5–3.5 pp binomial standard error and are read as matched accuracy, and the full-parameter experiment reports mean std over three seeds (Table 9).
Experiments compute resources
Question: For each experiment, does the paper provide sufficient information on the computer resources, type of compute workers, memory, time of execution, needed to reproduce the experiments?
Answer: [Yes].
Justification: Appendix J states that all experiments were run on NVIDIA H100 80 GB GPUs, describes the GPU/CPU division of work, and summarizes runtime drivers. Table 1–Table 2 report GPU-hours, peak VRAM, and rollout throughput for the main experiments, the released repository includes launch scripts and software versions, and Table 10 gives the wall-clock breakdown including calibration.
Code of ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https://neurips.cc/public/EthicsGuidelines?
Answer: [Yes].
Justification: The work proposes an efficiency and calibration method for RL training on standard reasoning and code benchmarks. It does not involve human subjects, private data, surveillance, deceptive data collection, or deployment of a high-risk decision system, and the paper preserves author anonymity.
Broader impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes].
Justification: Appendix K discusses positive impacts, including reduced RL training compute and statistically calibrated rollout allocation, as well as possible negative impacts from lowering the cost of improving models for undesirable applications. It also explains that the planned release focuses on reproducibility infrastructure rather than a new unrestricted pretrained model.
Safeguards
Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse, e.g., pre-trained language models, image generators, or scraped datasets?
Answer: [N/A].
Justification: The paper does not release a new pretrained language model, image generator, scraped dataset, or other high-risk generative asset. The release consists of training, calibration, verification, and evaluation code for reproducing experiments on existing public models and benchmarks.
Licenses for existing assets
Question: Are the creators or original owners of assets, e.g., code, data, models, used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Answer: [Yes].
Justification: The paper cites the original sources for all major datasets, models, and methods used in the evaluation, including GSM8K, MATH, GPQA, HumanEval, MBPP, Qwen2.5, Llama-3.1, Gemma, LoRA, GRPO, and the dynamic-rollout baselines. Appendix § 18 states that the release package will include asset versions, access instructions, and license or terms-of-use notes where available.
New assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [Yes].
Justification: The new assets are the C-GRPO algorithm, conformal calibration code, first-success score implementation, auto- logic, budget-allocation scripts, verifier wrappers, and evaluation scripts. Their mathematical definitions and algorithmic behavior are documented in § 4.2–§ 4.4 and Algorithm 1, and the implementation is released with reproduction instructions (Appendix § 17, Code availability).
Crowdsourcing and research with human subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation, if any?
Answer: [N/A].
Justification: The paper does not involve crowdsourcing, human-subject experiments, participant recruitment, surveys, or compensation. All experiments use existing benchmarks and deterministic verifiers or unit tests.
Institutional review board (IRB) approvals or equivalent for research with human subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board approvals or an equivalent approval/review based on the requirements of your country or institution were obtained?
Answer: [N/A].
Justification: The work does not involve human subjects, private personal data, or interaction with participants. Therefore, IRB approval or equivalent human-subjects review is not applicable.
Declaration of LLM usage
Question: Does the paper describe the usage of LLMs if it is an important, original, or non-standard component of the core methods in this research?
Answer: [Yes].
Justification: LLM usage is central to the paper: C-GRPO is a training-time rollout-allocation method for language models, and the experiments evaluate Qwen2.5-7B-Instruct, Llama-3.1-8B-Instruct, and Gemma-3-4B-IT. § 4–§ 5 describe how completions are sampled, verified, calibrated, and used in the adaptive GRPO objective.
References
- [1]Anastasios N. Angelopoulos, Stephen Bates, Emmanuel J. Candès, Michael I. Jordan, and Lihua Lei. Learn then test: Calibrating predictive algorithms to achieve risk control, 2021. URL https://arxiv.org/abs/2110.01052.
- [2]Jacob Austin, Augustus Odena, Maxwell Nye, Maarten Bosma, Henryk Michalewski, David Dohan, Ellen Jiang, Carrie Cai, Michael Terry, Quoc Le, and Charles Sutton. Program synthesis with large language models, 2021. URL https://arxiv.org/abs/2108.07732.
- [3]Stephen Bates, Anastasios Angelopoulos, Lihua Lei, Jitendra Malik, and Michael I. Jordan. Distribution-free, risk-controlling prediction sets, 2021. URL https://arxiv.org/abs/2101.02703.
- [4]Mark Chen, Jerry Tworek, Heewoo Jun, Qiming Yuan, Henrique Ponde de Oliveira Pinto, Jared Kaplan, Harri Edwards, Yuri Burda, Nicholas Joseph, Greg Brockman, Alex Ray, Raul Puri, Gretchen Krueger, Michael Petrov, Heidy Khlaaf, Girish Sastry, Pamela Mishkin, Brooke Chan, Scott Gray, Nick Ryder, Mikhail Pavlov, Alethea Power, Lukasz Kaiser, Mohammad Bavarian, Clemens Winter, Philippe Tillet, Felipe Petroski Such, Dave Cummings, Matthias Plappert, Fotios Chantzis, Elizabeth Barnes, Ariel Herbert-Voss, William Hebgen Guss, Alex Nichol, Alex Paino, Nikolas Tezak, Jie Tang, Igor Babuschkin, Suchir Balaji, Shantanu Jain, William Saunders, Christopher Hesse, Andrew N. Carr, Jan Leike, Josh Achiam, Vedant Misra, Evan Morikawa, Alec Radford, Matthew Knight, Miles Brundage, Mira Murati, Katie Mayer, Peter Welinder, Bob McGrew, Dario Amodei, Sam McCandlish, Ilya Sutskever, and Wojciech Zaremba. Evaluating large language models trained on code, 2021. URL https://arxiv.org/abs/2107.03374.
- [5]John Cherian, Isaac Gibbs, and Emmanuel J. Candès. Large language model validity via enhanced conformal prediction methods. In Advances in Neural Information Processing Systems (NeurIPS), 2024.arxiv.org/abs/2406.09714
- [6]Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, Mark Chen, Heewoo Jun, Lukasz Kaiser, Matthias Plappert, Jerry Tworek, Jacob Hilton, Reiichiro Nakano, Christopher Hesse, and John Schulman. Training verifiers to solve math word problems, 2021. URL https://arxiv.org/abs/2110.14168.
- [7]Mehul Damani, Idan Shenfeld, Andi Peng, Andreea Bobu, and Jacob Andreas. Learning how hard to think: Input-adaptive allocation of LM computation. In International Conference on Learning Representations (ICLR), 2025.arxiv.org/abs/2410.04707
- [8]Gemma Team. Gemma 3 technical report, 2025. URL https://arxiv.org/abs/2503.19786.
- [9]Isaac Gibbs, John J. Cherian, and Emmanuel J. Candès. Conformal prediction with conditional guarantees, 2023. URL https://arxiv.org/abs/2305.12616.
- [10]Aaron Grattafori, Abhimanyu Dubey, Abhinav Jau hri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, et al. The Llama 3 herd of models, 2024. URL https://arxiv.org/abs/2407.21783.
- [11]Yu Gui, Ying Jin, and Zhimei Ren. Conformal alignment: Knowing when to trust foundation models with guarantees, 2024. URL https://arxiv.org/abs/2405.10301.
- [12]Dan Hendrycks, Collin Burns, Saurav Kadavath, Akul Arora, Steven Basart, Eric Tang, Dawn Song, and Jacob Steinhardt. Measuring mathematical problem solving with the MATH dataset, 2021. URL https://arxiv.org/abs/2103.03874.
- [13]Edward J. Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen. LoRA: Low-rank adaptation of large language models. In International Conference on Learning Representations, 2022.arxiv.org/abs/2106.09685
- [14]M Pawan Kumar, Benjamin Packer, and Daphne Koller. Self-paced learning for latent variable models. In Advances in Neural Information Processing Systems (NeurIPS), volume 23, 2010.
- [15]Hunter Lightman, Vineet Kosaraju, Yura Burda, Harri Edwards, Bowen Baker, Teddy Lee, Jan Leike, John Schulman, Ilya Sutskever, and Karl Cobbe. Let’s Verify Step by Step, 2023. URL https://arxiv.org/abs/2305.20050.
- [16]Ilya Loshchilov and Frank Hutter. Decoupled weight decay regularization. In International Conference on Learning Representations, 2019.
- [17]Kishan Panaganti, Zhenwen Liang, Wenhao Yu, Haitao Mi, and Dong Yu. Group distributionally robust optimization-driven reinforcement learning for LLM reasoning, 2026. URL https://arxiv.org/abs/2601.19280.
- [18]Victor Quach, Adam Fisch, Tal Schuster, Adam Yala, Jae Ho Sohn, Tommi S. Jaakkola, and Regina Barzilay. Conformal language modeling. In International Conference on Learning Representations (ICLR), 2024.arxiv.org/abs/2306.10193
- [19]Qwen. Qwen2.5 technical report, 2024. URL https://arxiv.org/abs/2412.15115.
- [20]David Rein, Betty Li Hou, Asa Cooper Stickland, Jackson Petty, Richard Yuanzhe Pang, Julien Dirani, Julian Michael, and Samuel R. Bowman. GPQA: A graduate-level google-proof Q&A benchmark, 2023. URL https://arxiv.org/abs/2311.12022.
- [21]Yaniv Romano, Matteo Sesia, and Emmanuel J. Candès. Classification with valid and adaptive coverage. In Advances in Neural Information Processing Systems, volume 33, 2020.arxiv.org/abs/2006.02544
- [22]John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal policy optimization algorithms, 2017. URL https://arxiv.org/abs/1707.06347.
- [23]Zhihong Shao, Peiyi Wang, Qihao Zhu, Runxin Xu, Junxiao Song, Xiao Bi, Haowei Zhang, Mingchuan Zhang, Y.K. Li, Y. Wu, and Daya Guo. DeepSeekMath: Pushing the limits of mathematical reasoning in open language models, 2024. URL https://arxiv.org/abs/2402.03300.
- [24]Ryan J. Tibshirani, Rina Foygel Barber, Emmanuel J. Candès, and Aaditya Ramdas. Conformal prediction under covariate shift. In Advances in Neural Information Processing Systems, volume 32, 2019.arxiv.org/abs/1904.06019
- [25]Vladimir Vovk, Alex Gammerman, and Glenn Shafer. Algorithmic Learning in a Random World. Springer Science & Business Media, 2005.
- [26]Ronald J. Williams. Simple statistical gradient-following algorithms for connectionist reinforcement learning. Machine Learning, 8:229–256, 1992.
- [27]Wei Xiong, Chenlu Ye, Baohao Liao, Hanze Dong, Xinxing Xu, Christof Monz, Jiang Bian, Nan Jiang, and Tong Zhang. Reinforce-Ada: An adaptive sampling framework under non-linear RL objectives, 2025. URL https://arxiv.org/abs/2510.04996.
- [28]Zhi Zhang, Zhen Han, Costas Mavromatis, Qi Zhu, Yunyi Zhang, Sheng Guan, Dingmin Wang, Xiong Zhou, Shuai Wang, Soji Adeshina, Vassilis Ioannidis, and Huzefa Rangwala. Train less, learn more: Adaptive efficient rollout optimization for group-based reinforcement learning, 2026. URL https://arxiv.org/abs/2602.14338.
- [29]Haizhong Zheng, Yang Zhou, Brian R. Bartoldson, Bhavya Kailkhura, Fan Lai, Jiawei Zhao, and Beidi Chen. Act only when it pays: Efficient reinforcement learning for LLM reasoning via selective rollouts, 2025. URL https://arxiv.org/abs/2506.02177.

