WorksComputer Science
GIST: Gibbs-Targeted Hierarchical Data Selection for Offline RLVR
Abstract
Offline reinforcement learning with verifiable rewards (RLVR) trains on a fixed pool of sampled responses and typically supervises every token of every response, although responses and tokens are not equally informative. We study which ones to supervise through the Gibbs-optimal policy of KL-regularized RLVR, which offline training targets, and measure the mismatch between a policy and this target with a bidirectional KL divergence. At the reference policy, this mismatch factorizes exactly and separates the two outcomes: correct responses contribute mainly through the forward term, concentrated on their low-probability tokens, and incorrect responses through the reverse term, concentrated on the responses the policy generates often and on their high-probability tokens. Based on this analysis, we propose GIST (Gibbs-Targeted Hierarchical Data Selection), a two-level selection method that approximates these principles with signals available before training. GIST uses the sampling temperature to select correct responses on-policy and incorrect responses near the policy's mode, and it uses the initial policy's token probabilities to supervise only the tokens where the mismatch lies. Applied to two offline training objectives, GIST raises the average accuracy over GSM8K, MATH-500, and MBPP for all three models under both objectives. In a controlled comparison on Qwen3-0.6B/GSM8K, it outperforms seven selection baselines that choose from the same candidates while supervising 66.6% fewer tokens than training on all candidates of the same prompts.
Introduction
Offline reinforcement learning with verifiable rewards (RLVR) trains a language model on a fixed pool of responses whose outcomes are checked automatically ([20, 44, 7, 49]). Common approaches supervise every token of the responses they retain, although responses and tokens are not equally informative. Deciding which responses to keep and which of their tokens to supervise is therefore a central design choice in offline RLVR.
Existing methods address parts of this choice: rejection sampling keeps verified correct responses [58, 56], reward-weighted objectives such as BOLT weight responses by their reward [36, 45], and other methods subsample rollouts [22, 53], or select tokens by entropy or probability [50, 27]. They leave open how to combine response selection with outcome-dependent token supervision, although whether a token deserves supervision depends on both the outcome and how the response was sampled. A low-probability token in a high-temperature sample may be noise, but in a verified correct response it is more likely a valid step that the policy under-weights, while failures sampled at low temperature reveal errors near the policy’s mode.
We study selection through the Gibbs-optimal policy of KL-regularized RLVR, the target that offline training fits, and measure the mismatch between the policy and this target with a bidirectional KL divergence and a KL-curvature score that bounds it from above and below. The forward term governs correct responses, whose informative instances are under-represented and whose informative tokens have low probability. The reverse term governs incorrect responses, whose informative instances are over-represented and whose informative tokens have high probability. At the reference policy, where offline selection takes place, the mismatch factorizes exactly.
We then propose GIST (Gibbs-Targeted Hierarchical Data Selection), a two-level selection method that approximates these principles with three quantities available at data-construction time: the sampling temperature, the verifier outcome, and the initial policy’s token probabilities. At the response level, GIST takes correct responses from on-policy samples and incorrect responses from near-mode samples, and it keeps the prompts that supply both. At the token level, it supervises the low-probability tokens of correct responses and the high-probability tokens of incorrect ones.
Paired with two offline objectives, BOLT and SFT, GIST improves accuracy on GSM8K, MATH-500, and MBPP in nearly all settings (29 of 30) across three models from two families, and it raises the average accuracy in every case. On Qwen3-0.6B/GSM8K, where all methods choose from the same candidates, GIST outperforms seven selection baselines while supervising 66.6% fewer tokens and taking 49.9% fewer optimizer steps than training on all candidates of the same prompts (Figure 1).
Our contributions are as follows. (i) We derive selection principles for offline RLVR from the bidirectional mismatch to the Gibbs-optimal target, separating the roles of correct and incorrect responses at the response and token levels (Section 3). (ii) We propose GIST, a two-level selection method that approximates these principles from signals observed during data construction (Section 4). (iii) Across three models and two offline objectives, GIST improves nearly all benchmark results, and in a controlled comparison it outperforms seven selectors while supervising 66.6% fewer tokens (Section 5).
Preliminaries
KL-regularized RLVR. For a prompt and response , a verifier assigns a binary reward , with for correct and for incorrect responses. Throughout, the reference policy is the initial policy , and KL-regularized RLVR [34, 20] maximizes
where controls the strength of regularization.
Proposition 2.1 (Gibbs target). For a fixed prompt , the unique maximizer of Eq. (1) is
and for any policy with finite KL divergence to ,
The target reweights the reference policy by the exponentiated reward [38, 19, 40], and Eq. (3) identifies the objective gap with the reverse KL to .
Offline fitting of the Gibbs target. Offline training fits using a fixed pool of responses drawn from a rollout distribution , with wherever . Rewriting the forward KL as an expectation under gives
where does not depend on . When , the weight reduces to , and (4) becomes the reference-sampled weighted-likelihood objective of BOLT [45]. This objective fits the target in the forward direction, whereas the objective gap in (3) is a reverse KL. Its weights also separate the two outcome classes: an incorrect response receives times the weight of a correct response to the same prompt. Section 3 uses both directions to locate the mismatch, and Section 4.3 describes how the weights are computed in training.
Token-level view. Let denote the prefix at step , and let be the expectation of over reference continuations of , so that is the normalizer in (2) and for the soft value [60, 16, 41]. The Gibbs target then factorizes over tokens as
so a token is more likely under the target than under the reference policy exactly when it raises the soft value. For two distributions and over responses with conditionals and , the chain rule for KL divergence [4] gives
Throughout, , so that .
A KL-Curvature View of Data Selection
To decide which responses and tokens to supervise, we ask where the policy differs most from the Gibbs target of Section 2, first for whole responses and then for individual tokens (proofs in Appendix A).
Bidirectional Mismatch and Curvature Score
Fix a prompt , and write and . For , define
The forward term is the target-fitting loss of Section 2; it measures how much the policy underrepresents target behavior [32]. The reverse term, scaled by , is the RLVR objective gap (Proposition 2.1); it measures how much policy mass lies on responses that the target down-weights. The score weights the squared probability gap by the local curvature of both KL terms [5], so a gap of a given size counts more when it occurs at small probabilities.
Theorem 3.1 (KL-curvature bounds). Let be positive on a common support with , where . Then , where , , and . As the density ratios converge uniformly to one, .
Up to constants, the score thus bounds the mismatch from above and below, and it is exact to second order near the target. The curvature factor also explains why a verifier label alone does not determine a response’s value: the same probability gap counts more where the policy or the target places little mass. The coefficient weights the two directions and is used only in the analysis.
Budgeted Response Selection
To select under a budget, we attribute the mismatch to individual responses through the contribution , which is nonnegative and satisfies (§ 2).
Corollary 3.2 (Budgeted selection guarantee). Let contain the candidates with the largest , and let maximize over subsets of size . Under the conditions of Theorem 3.1,
Selecting by the curvature score therefore captures a constant fraction of the largest attainable mismatch, similar to coreset guarantees [43, 33], and this fraction approaches one as the policy approaches the target. Note that the guarantee is about the mismatch captured at a fixed policy; it does not concern the outcome of training on the selected set.
With binary rewards, the target is , and the score behaves differently on the two classes.
Corollary 3.3 (Informative correct and incorrect responses). For fixed : (i) if and , then and the forward term dominates; (ii) if and , then and the reverse term dominates. At comparable reference probabilities, (i) favors under-represented correct responses and (ii) favors over-represented incorrect responses.
Under-represented correct responses drive the forward (coverage) error, and over-represented incorrect ones drive the reverse error, which Proposition 2.1 identifies with the RLVR objective gap. The two directions describe the same mismatch: mass moved onto under-represented correct responses must come from other responses, and the reverse term measures how much of it sits on over-represented errors. Offline selection evaluates the mismatch at , where both classes have an exact form.
Proposition 3.4 (Exact mismatch at the reference policy). At , and , where . With the reference success probability, and .
Hence every correct response is under-represented and every incorrect response is over-represented, as in Corollary 3.3. Within a class, , so frequently generated incorrect responses contribute most. For correct responses, the under-representation is uniform at the response level and localized at the token level (§ 3.3).
Remark 3.5 (The curvature score at the reference policy). At , Eq. (7) gives
so is constant within a class and Corollary 3.2 is exact there; across classes, Proposition 3.4 gives the exact weights (Appendix § 11.§ 3).
Token-Level Mismatch
Applying the chain rule of Eq. (6) to both terms of decomposes the mismatch over prefixes, with the forward part weighted by and the reverse part by . For an observed token, we define its conditional score and contribution as in Eq. (7) and § 3.2, with evaluated at that token. Theorem 3.1 and Corollary 3.2 also hold for these conditional quantities at each prefix, and the two regimes again separate.
Corollary 3.6 (Informative tokens). For fixed and comparable target conditionals across positions: (i) in a correct response, low-probability tokens () carry the forward mismatch, with ; (ii) in an incorrect response, high-probability tokens () carry the reverse mismatch, with .
Proposition 3.7 (Conditional token-level factorization). At , with and , we have , , and .
By Eq. (5), is the token’s soft-value ratio; GIST ranks positions by these conditional quantities and does not estimate the prefix weights (Appendix § 11). The bound supports Corollary 3.6(i) at the reference policy even without the comparability condition: a large soft-value gain requires a low , although a low probability alone does not imply a large gain. For an incorrect response, decreases by in total, and for a given decrease, grows with , as in (ii). Because is increasing on , the largest conditional contribution attainable at a position with is
which decreases in . The high-probability tokens of a correct response therefore account for little of its mismatch.
Remark 3.8 (Where enters selection). The within-class ranking by , the bound , and the signs of the soft-value totals hold for every , so the four principles in Table 1 do not depend on . Only the balance between the two classes depends on and , through the ratio of their total contributions,
At and , is 2.6 for and 13.4 for ; at , it is 1.8 and 2.5. Since the analysis does not fix the class balance, GIST sets it through the number of retained responses in each class.
| Class | Dominant term | Response level | Token level |
| Correct | forward (coverage) | under-represented | low-probability tokens |
| Incorrect | reverse (over-commitment) | over-represented | high-probability tokens |
Table 1. Selection principles implied by the bidirectional mismatch.
Table 1 summarizes the four principles, each of which follows from the curvature of the two KL terms.
GIST: Gibbs-Targeted Hierarchical Data Selection
The principles in Table 1 depend on target probabilities and soft values, which are not available during data construction. Instead of estimating these quantities, GIST uses three observable signals: the sampling temperature, the verifier outcome, and the initial-policy token probability. With these signals, it selects responses and then tokens within them, and it keeps the prompts that supply both outcomes (Figure 2). All selections are made once, before training.
Figure 2. Overview of GIST (schematic). (a) Rollouts on-policy () and near the mode (). (b) Correct responses are taken from and incorrect ones from . (c) The frozen initial policy scores tokens to mask both classes. (d) Selected responses and masks are passed to an offline training objective.
Prompt and Response Selection
For each prompt , the initial policy generates responses at temperature and at , without top- ([18]) or top- truncation. A verifier assigns to correct and to incorrect responses. GIST takes correct responses from the on-policy pool and incorrect responses from the near-mode pool; we call this assignment temperature pairing:
Within each class, the contribution is proportional to (Proposition 3.4), so sampling at concentrates on the incorrect responses that contribute most. For correct responses, we choose with the token level in mind: on-policy responses keep the low-probability steps that the mask selects (§ 5.4), and they need no temperature correction in Eq. (4). The subsequent selection and masking, however, make the training loss an approximation to that objective. A prompt is eligible if and . The quotas are a practical choice and do not follow from the analysis. They guarantee samples of both classes and bound the weights of correct responses; the cost is that the hardest prompts are excluded. For each eligible prompt, GIST samples correct and incorrect responses uniformly without replacement, treating duplicates as distinct instances. The analysis does not fix the balance between the classes (Remark 3.8), and we use equal quotas.
Token Selection
For a retained response , let and let be its mean over the response positions, excluding EOS. The masks are
These masks implement the token-level principles, with the response mean as the threshold between low and high probability. Correct responses keep their low-probability tokens, where the under-represented steps lie (Corollary 3.6(i), Proposition 3.7), and incorrect responses keep their high-probability tokens, the confident errors of Corollary 3.6(ii). Both masks are applied in every run that uses GIST’s token selection. They depend only on the verifier outcome and the initial policy, not on the training objective, and are computed once with the frozen initial policy and stay fixed during training.
Training with the Selected Data
GIST outputs the selected responses, their rewards, and the token masks, so it can be combined with any offline objective that trains on weighted responses; we use two. BOLT (Shu et al., [45]), the weighted-likelihood objective of § 2, weights each response by , rescaled so that the weights of the prompt’s 32 candidates average to one. The weights are computed on the full candidate pool before selection and are not renormalized afterward. A correct response therefore receives a weight of about , where is the number of correct candidates, so prompts that the policy solves less often get more, and an incorrect response receives times that weight. SFT instead gives every retained response unit weight. Both objectives average, over the minibatch, each response’s weight times its mean negative log-likelihood on the masked positions; unmasked positions remain in the context but receive no loss.
Experiments
Main Results
Setup. We train Qwen3-0.6B, Qwen3.5-2B, and Gemma-4-E2B-it (Yang et al., [54]; Qwen Team, [39]; Gemma Team, [12]) and evaluate on GSM8K, MATH-500, and MBPP-full (Cobbe et al., [3]; Hendrycks et al., [17]; Lightman et al., [24]; Austin et al., [1]). For math, we report Strict accuracy, which requires the prescribed answer format, and Flexible accuracy (Flex.), which also credits correctly extracted answers; for code, we report pass@1. We pair GIST with two objectives: BOLT () and SFT, which imitates every training response with unit weight. All trained methods share 32 candidates per prompt (16 per temperature) and validation-based checkpoint selection; each objective is trained either on all candidates or on GIST’s selected subset (+ GIST; details in Appendix B).
Results. Averaged over the three benchmarks (Avg. in Table 2), GIST improves both objectives on every model, by 1.26–2.55 points over BOLT and 1.11–4.41 points over SFT, and BOLT + GIST has the highest average for all three models. GIST improves over BOLT on both mathematical benchmarks for all three models under both metrics (Table 2). On GSM8K, the gains reach 2.50 points on Strict and 3.26 on Flexible. On Flexible, BOLT falls below the initial checkpoint for two models, whereas BOLT + GIST improves on it. For Qwen3-0.6B, paired tests over the 1,319 GSM8K questions support both gains (Appendix § 12.§ 5). BOLT also trains on prompts whose candidates are all incorrect (Appendix § 12), so this compares the two pipelines end to end; § 5.2 and § 5.3 isolate the selection components. Gains on MATH-500 are smaller (0.40–1.80 points). On MBPP-full, GIST gains 4.80 points on Qwen3-0.6B and 0.60 on Gemma-4-E2B-it, and trails BOLT by 0.60 on Qwen3.5-2B.
Table 2: Test results with BOLT and SFT (%). Avg.: mean of GSM8K Flex., MATH-500 Flex., and MBPP-full pass@1. Bold marks the best score within each model and column. Parentheses: change vs. Initial.
| Model | Method | GSM8K Strict | GSM8K Flex. | MATH-500 Strict | MATH-500 Flex. | MBPP-full pass@1 | Avg. |
| Qwen3-0.6B | Initial | 40.86 | 46.10 | 44.20 | 49.80 | 22.80 | 39.57 |
| SFT | 42.08 (+1.22) | 47.38 (+1.28) | 44.40 (+0.20) | 50.00 (+0.20) | 24.20 (+1.40) | 40.53 (+0.96) | |
| + GIST | 43.82 (+2.96) | 48.14 (+2.04) | 44.80 (+0.60) | 50.40 (+0.60) | 28.80 (+6.00) | 42.45 (+2.88) | |
| BOLT | 43.29 (+2.43) | 47.46 (+1.36) | 44.60 (+0.40) | 50.40 (+0.60) | 26.60 (+3.80) | 41.49 (+1.92) | |
| + GIST | 45.79 (+4.93) | 49.73 (+3.63) | 45.40 (+1.20) | 51.00 (+1.20) | 31.40 (+8.60) | 44.04 (+4.47) | |
| Qwen3.5-2B | Initial | 54.51 | 55.19 | 46.00 | 55.00 | 37.40 | 49.20 |
| SFT | 55.04 (+0.53) | 55.80 (+0.61) | 46.20 (+0.20) | 55.00 (+0.00) | 39.60 (+2.20) | 50.13 (+0.93) | |
| + GIST | 56.18 (+1.67) | 56.71 (+1.52) | 46.80 (+0.80) | 55.80 (+0.80) | 41.20 (+3.80) | 51.24 (+2.04) | |
| BOLT | 54.13 (−0.38) | 54.51 (−0.68) | 46.40 (+0.40) | 54.40 (−0.60) | 42.40 (+5.00) | 50.44 (+1.24) | |
| + GIST | 56.63 (+2.12) | 57.09 (+1.90) | 47.40 (+1.40) | 56.20 (+1.20) | 41.80 (+4.40) | 51.70 (+2.50) | |
| Gemma-4-E2B-it | Initial | 12.28 | 26.99 | 2.60 | 38.60 | 55.00 | 40.20 |
| SFT | 12.81 (+0.53) | 27.82 (+0.83) | 12.40 (+9.80) | 42.00 (+3.40) | 55.80 (+0.80) | 41.87 (+1.67) | |
| + GIST | 13.87 (+1.59) | 29.04 (+2.05) | 41.80 (+39.20) | 53.00 (+14.40) | 56.80 (+1.80) | 46.28 (+6.08) | |
| BOLT | 12.89 (+0.61) | 26.31 (−0.68) | 40.80 (+38.20) | 52.80 (+14.20) | 56.40 (+1.40) | 45.17 (+4.97) | |
| + GIST | 14.10 (+1.82) | 29.57 (+2.58) | 42.20 (+39.60) | 53.20 (+14.60) | 57.00 (+2.00) | 46.59 (+6.39) |
Table 2.
GIST also improves over SFT for every model, benchmark, and metric (Table 2). The gains are 0.76–1.74 points on GSM8K, 0.40–0.80 on MATH-500 for the two Qwen models, and 1.00–4.60 on MBPP-full. On Gemma-4-E2B-it, MATH-500 Strict rises from 12.40% to 41.80% while Flexible rises by 11.00 points, so most of this gain comes from following the prescribed answer format. On GSM8K, the gains from GIST are similar under Flexible and Strict accuracy for both objectives, which suggests that they do not come only from answer formatting.
Comparison with Selection Baselines
All selectors share the candidate pool, the 1,593 eligible prompts, and the BOLT objective (Table 3). Random samples four correct and four incorrect responses uniformly and supervises all tokens, RFT-style keeps all correct responses, and the others adapt HES, PODS, ProFit, ETS, and RSI-S ([22, 53, 27, 50, 30]) (Appendix § 13).
Table 3: Selection methods with BOLT on a shared candidate pool (Qwen3-0.6B/GSM8K, %).
| Random | PODS | RFT-style | RSI-S | ETS | ProFit | HES | GIST | |
| Strict | 43.14 | 43.52 | 44.05 | 43.21 | 43.75 | 42.46 | 44.20 | 45.79 |
| Flex. | 48.60 | 48.22 | 48.14 | 47.84 | 47.54 | 46.47 | 47.31 | 49.73 |
Table 3.
GIST achieves the highest accuracy on both metrics, 1.59–3.33 points above all seven baselines on Strict and 1.13–3.26 points on Flexible. It exceeds Random by 2.65 points on Strict and 1.13 points on Flexible while supervising 31.5% fewer tokens (§ 5.4), and it exceeds training on all candidates by 1.97 and 3.03 points. Appendix § 12.§ 5 reports paired significance tests for these comparisons.
Ablation Study
We ablate GIST on the shared pool, changing one design choice at a time (Table 4 and Table 5).
| Sub. | Pair. | Mask | Wt. | Strict ↑ | Flex. ↑ | |
| (a) | × | × | × | ✓ | 43.82 (−1.97) | 46.70 (−3.03) |
| (b) | ✓ | × | × | ✓ | 43.14 (−2.65) | 48.60 (−1.13) |
| (c) | ✓ | × | ✓ | ✓ | 43.59 (−2.20) | 48.75 (−0.98) |
| (d) | ✓ | ✓ | × | ✓ | 43.75 (−2.04) | 47.61 (−2.12) |
| (e) | ✓ | ✓ | ✓ | × | 43.82 (−1.97) | 48.14 (−1.59) |
| (f) | ✓ | ✓ | ✓ | ✓ | 45.79 | 49.73 |
Table 4. Component ablation, Qwen3-0.6B on GSM8K (%). Sub.: 4+4 responses instead of all 32; Pair.: correct from , incorrect from ; Mask: of Eq. (12); Wt.: Boltzmann vs. unit. Parentheses: change vs. (f).
| Positions | Kept | Strict ↑ | Flex. ↑ |
| All tokens | 100 | 43.75 (−2.04) | 47.61 (−2.12) |
| Random† | 31 | 44.81 (−0.98) | 45.64 (−4.09) |
| Highest-prob.† | 31 | 44.05 (−1.74) | 47.23 (−2.50) |
| Below-mean (GIST) | 31 | 45.79 | 49.73 |
Table 5. Token rule for correct responses, Qwen3-0.6B on GSM8K (%). Kept: approximate share of positions supervised per correct response (%). †Same number of positions per response as GIST. Parentheses: change vs. GIST.
Removing pairing (row c), the mask (row d), or the Boltzmann weights (row e, the SFT + GIST entry of Table 2) from GIST lowers Strict/Flexible accuracy by 2.20/0.98, 2.04/2.12, and 1.97/1.59 points, respectively, and removing all selection (row a) costs 1.97/3.03 (Table 4). Replacing GIST’s token rule also lowers both metrics (Table 5): supervising all tokens costs 2.04 and 2.12 points, a random subset of the same size costs 0.98 and 4.09, and keeping the highest-probability tokens of correct responses costs 1.74 and 2.50.
Rows (b), (c), (d), and (f) of Table 4 form a design of pairing and masking (Figure 3). The mask adds 2.04 and 2.12 points with pairing but only 0.45 and 0.15 without it, an interaction of 1.59 and 1.97 points.
Figure 3. Pairing and masking on Qwen3-0.6B/GSM8K. Unpaired responses are the Random selector; the mask helps mainly with paired responses.
Discussion
Token and response selection. The analysis in Section 3 holds at the policy that selects the data, so we compare its consequences with ablations that change one part of the data at a time. The token-rule ablation keeps GIST’s responses fixed and changes only which positions are supervised, which tests Corollary 3.6(i). By Proposition 3.7, a token can gain much soft value only if is low (), so the high-probability tokens of a correct response should carry little of its mismatch. Keeping them instead of the low-probability ones loses most of the mask’s benefit (Table 5), and the ProFit adaptation, which also keeps high-probability tokens, is the weakest selector (Table 3). Below-mean tokens make up about 31% of correct responses at both temperatures, but their mean is 0.41 at and 0.58 at (Appendix § 12.§ 5); since the attainable contribution in Eq. (9) decreases in , the mask should find more informative tokens in on-policy responses. Without the mask, a whole correct response contributes in proportion to (Proposition 3.4), which favors near-mode responses. Together, these results predict that pairing and masking help mainly in combination, which matches the interaction in Section 5.3.
Supervision and training cost. On the shared pool, GIST keeps 12,744 of the 50,976 candidates and supervises 66.6% fewer tokens than training on all candidates and 31.5% fewer than Random, while scoring higher than both Table 6, Figure 1, Appendix § 12.§ 6. Masked positions still enter the forward and backward passes, so the compute saving over training on all candidates comes from keeping a quarter of the responses. GIST reaches its selected checkpoint in 2,394 optimizer steps, about half of the 4,779 needed when training on all candidates, although its selected checkpoint comes after twice as many epochs. Selection reuses BOLT’s candidate pool and adds one forward pass of the frozen initial policy to compute .
Figure 1. Test Strict accuracy versus supervised tokens on Qwen3-0.6B/GSM8K. All methods use BOLT on the same candidate pool. GIST reaches the highest accuracy while supervising 66.6% fewer tokens than training on all candidates and 31.5% fewer than random selection.
| All | Random | GIST | |
| Tokens | 13.60M | 6.64M | 4.55M |
| Reduction | – | 51.2% | 66.6% |
| Strict ↑ | 43.82 | 43.14 | 45.79 |
| Flex. ↑ | 46.70 | 48.60 | 49.73 |
Table 6. Supervised tokens up to each method’s selected checkpoint (Qwen3-0.6B/GSM8K). Reduction is relative to all candidates.
Generality. GIST selects data using only the sampling temperature, the verifier outcome, and the initial policy’s token probabilities, with the same settings for every model and benchmark (Appendix § 12). Because the selection does not depend on the training objective, it can be paired with other offline methods. SFT imitates every retained response with unit weight, including the incorrect ones, and SFT + GIST still matches or exceeds BOLT trained on all candidates in 14 of the 15 comparisons. The benefit of the selection therefore does not come only from BOLT’s reward weighting, although with GIST’s data the weights add a further 0.20–2.60 points in every comparison.
Related Work
Offline RLVR and policy optimization. Reinforcement learning with verifiable rewards (RLVR) improves reasoning using automatically checkable outcomes [20, 44, 7, 55]. Most RLVR methods optimize online with PPO-style updates [42, 28, 57], whereas offline approaches [21] such as ILQL [47] and OREO [49] learn from previously collected rollouts. BOLT [45] shows that Boltzmann-weighted likelihood, in the spirit of reward-weighted regression [37, 36], on reference-policy samples fits the Gibbs-optimal target of KL-regularized RLVR. These methods define the training objective; GIST instead decides which responses and tokens of a finite rollout pool to train on, and can be paired with BOLT or other offline objectives. Work on alignment and distillation chooses which KL direction or -divergence the model should minimize [13, 48, 14], whereas we use both directions to select the training data.
Data selection for reasoning post-training. Selection for post-training [59, 52] has been studied at two granularities. At the response level, rejection-sampling fine-tuning and self-training [58, 56, 8, 15, 46] keep verified correct trajectories, HES [22] ranks trajectories by the entropy of a few high-entropy tokens, PODS [53] subsamples rollouts for policy updates, and dynamic sampling in DAPO [55] discards prompts whose rollouts are all correct or all incorrect; other methods filter RL prompts by difficulty or learning impact [2, 23, 51]. At the token level, Rho-1 [25] and Token Cleaning [35] select tokens by excess loss or update influence, and critical-token estimation [26] contrasts models trained on correct and incorrect trajectories. In RLVR, policy entropy governs exploration [6] and high-entropy forking tokens concentrate effective updates, which motivates entropy-based token selection (ETS) [50]; RSI-S [30] and GMTS [29] use surprisal- and gradient-based token scores, and ProFit [27] keeps high-probability tokens during SFT. Each of these methods selects at a single level with its own criterion. GIST derives both from the same bidirectional mismatch and applies them hierarchically, keeping, unlike ProFit, the low-probability tokens of correct responses.
Conclusion
We studied data selection for offline RLVR through the bidirectional mismatch between the policy and the Gibbs-optimal target. In the regimes we identify, the forward term governs correct responses and the reverse term governs incorrect ones. GIST turns this view into response- and token-level selection rules that use only the sampling temperature, the verifier outcome, and the initial policy’s token probabilities. Across three models and two offline objectives, it improves nearly all benchmark results, and it outperforms seven selectors on a shared candidate pool while supervising far fewer tokens than training on all candidates. More broadly, the analysis suggests that which responses and which tokens to train on can be read from the policy itself, and we expect this view to apply to other offline objectives and to settings in which the data are re-selected as the policy changes.
Ethics Statement
This work uses publicly available models and benchmarks and involves no human subjects or personal data. We do not foresee ethical concerns beyond those of the underlying models.
Reproducibility Statement
Proofs of all theoretical results are given in Appendix § 11. Appendix § 12 specifies the data construction and eligibility rule (Appendix § 12.§ 1), the token masks (Appendix § 12.§ 2), the training objectives and hyperparameters (Appendix § 12.§ 3), the evaluation protocol (Appendix § 12.§ 4), and the paired tests (Appendix § 12.§ 5); Appendix § 13 describes every selection baseline and ablation control. All models and benchmarks are publicly available.
AI Use Statement
In this work, we used generative AI tools to edit the paper for readability, to polish the formatting of tables and figures, and to check that the numbers reported in the text match the tables. We have not used generative AI tools to develop the theoretical framework, formulate or prove the mathematical claims, design or implement the method and experiments, generate or process data, or interpret the results. We have reviewed all AI-assisted edits, checked every reported number against our experimental logs, and take full responsibility for the content of the paper.
Proofs
Throughout, , which satisfies , , and .
Gibbs Target and Offline Fitting
Proof of Proposition 2.1. Fix and write . Since , for any with finite KL divergence to ,
Nonnegativity of KL shows that is the unique maximizer with , and subtracting gives Eq. (3).
**Derivation of Eq. (4). . The first term does not depend on . If wherever , the second term equals , and by Eq. (2).
KL-Curvature Bounds and Budgeted Selection
Proof of Theorem 3.1. Because , we have and . For in an interval with , Taylor’s theorem around gives . With ,
and with ,
Multiplying the second pair by and adding gives for each , and summing over gives the theorem. For the local statement, applied to both ratios gives with .
Proof of Corollary 3.2. Using the per-response bounds and the optimality of for , .
Proof of Corollary 3.3. Write . (i) With , the first term is and the second is ; their ratio is , so . (ii) With , the second term is and the first is ; their ratio is , so . At fixed target probability, the first expression decreases and the second increases in .
Selection at the Reference Policy
Proof of Proposition 3.4. By Eq. (2), when . Substituting into gives , and yields . With , , so and .
Between-class constants (Remark 3.5). At , the density ratios range from to , so , below at for . For and , Eq. (8) gives an incorrect-to-correct score ratio between about and for responses of equal probability, whereas the exact ratio is between 2 and 101.
Token-Level Results
Chain rule and target conditional. With a finite horizon and terminated sequences padded by an absorbing state, . Taking expectations under and conditioning each term on its prefix gives the forward part of Eq. (6); exchanging and gives the reverse part. For the conditional, summing the Gibbs target over completions of a prefix gives , and the ratio of to is Eq. (5). Applying the proof of Theorem 3.1 to gives , and the proofs of Corollaries 3.2 and 3.3 apply verbatim to tokens, which proves Corollary 3.6.
Proof of Proposition 3.7. At , Eq. (5) gives , and the computation in the proof of Proposition 3.4 gives . Since and , the product telescopes to . Finally, , so . For the prefix-weighted share, the forward part uses , and the reverse part uses with .
Eq. (9). for , so whenever . Expanding gives the stated expression, whose derivative in is on . Along a correct response , and along an incorrect response .
Eq. (10). By Proposition 3.4, the total contribution of class is , which equals for and for . The values quoted in Remark 3.8 follow by direct evaluation.
Implementation Details
Data Construction
For each training prompt, the initial policy generates responses at and at , without top- or top- truncation. Rollouts use the chat template with thinking disabled, seed 42, and a stop at the closing </answer> tag, with at most 1,024 new tokens for GSM8K and MATH and 2,048 for MBPP. Validation sets are 500 held-out training questions for GSM8K and for MATH, and MBPP task IDs 511–600. The verifier assigns to correct and to incorrect responses: final-answer matching for GSM8K and MATH-500, and the unit tests for MBPP-full. Malformed or truncated responses are kept and verified normally; a response from which no answer can be extracted or verified receives . Duplicate responses are kept as separate instances. A prompt is eligible if it has at least correct responses and at least incorrect responses, and GIST samples four of each uniformly without replacement. Table 7 reports the eligible prompts; on Qwen3-0.6B/GSM8K, GIST retains 12,744 of the 50,976 candidates of the 1,593 eligible prompts. In Table 2, BOLT and SFT train on all 32 candidates of every training prompt, and each objective with + GIST trains on GIST’s selected subset. The controlled comparisons of Sections 5.2–5.4 use only the eligible prompts, and “all candidates” there refers to the 32 candidates of each eligible prompt.
| Model | GSM8K | MATH | MBPP |
| Qwen3-0.6B | 1,593 / 6,973 | 1,619 / 7,000 | 66 / 374 |
| Qwen3-5.2B | 2,574 / 6,973 | 1,100 / 7,000 | 65 / 374 |
| Gemma-4-2B-it | 1,373 / 6,973 | 1,646 / 7,000 | 58 / 374 |
Table 7. Eligible prompts over training prompts.
Token Masks
Token probabilities are computed once with the frozen initial policy and without temperature scaling, and the masks stay fixed during training. The mean is taken over the eligible response positions, which exclude prompt, padding, and EOS positions, so EOS never enters or the mask. Comparisons are strict, so positions equal to the mean are not selected. Whenever GIST’s token selection is applied, correct responses use and incorrect responses use ; the baselines keep their own token rules (Appendix § 13). Unselected positions remain in the context but receive no loss.
Training Objective
BOLT weights are computed within each prompt’s full pool of 32 candidates before selection, as described in § 4.3, with and , and are not renormalized after selection. An incorrect response therefore has weight times that of a correct response to the same prompt. No off-policy correction is applied to responses from the pool, so estimates the inverse success rate of the mixed-temperature pool rather than ; the BOLT baseline uses the same mixed pool. For a prompt whose 32 candidates are all incorrect, every weight equals one, so BOLT imitates these responses; GIST excludes such prompts through eligibility. The loss multiplies each response’s mean masked token negative log-likelihood by its weight and averages over the minibatch. SFT uses the same loss with for every response. SFT + GIST, which is also row (e) of Table 4, applies GIST’s selection and masks, so the positions of incorrect responses that selects are imitated with the same weight as the positions of correct ones. All other settings match BOLT. We fine-tune LoRA adapters (rank 8, , dropout 0.05) with fused AdamW (, , weight decay 0.1), a constant schedule with 10% warmup, BF16, and gradient checkpointing. Learning rates are for GSM8K and for MATH and MBPP; the global batch size is 32 for GSM8K and 128 for MATH and MBPP, and the maximum training length is 1,024 and 4,096 tokens, respectively. Training runs for up to 12 epochs with early stopping (patience 2) after at least 5 epochs, and the checkpoint with the best validation Strict accuracy (pass@1 for MBPP) is evaluated on the test set.
Evaluation
GSM8K is evaluated with lm-evaluation-harness (Gao et al., [11]) using 8-shot Q:/A: prompts without the chat template and at most 256 new tokens. GSM8K Strict requires “The answer is . . . ” followed by an exact match, and MATH Strict requires an exact match of the normalized last boxed answer; Flexible accuracy extracts the final answer from other formats. MBPP-full reports pass@1, executing the benchmark’s test cases in a bubblewrap sandbox with a 5-second timeout and CPU, memory, and file-size limits. Each entry is a single training run. The binomial standard error of a single test accuracy is about 1.4 points on GSM8K () and 2.2 points on MATH-500 and MBPP-full (); Appendix § 12.§ 5 reports paired tests for the main GSM8K comparisons.
Paired Tests and Token Statistics
Table 8 reports paired tests on the 1,319 GSM8K test questions (Qwen3-0.6B), using per-question correctness of the two compared checkpoints: a paired bootstrap 95% interval for the accuracy difference ([10]) and McNemar’s test [31], following Dror et al. [9].
| Comparison | Metric | Difference (pp) | Bootstrap 95% CI | McNemar |
| GIST − BOLT | Strict | +2.50 | [+0.30, +4.62] | 0.027 |
| GIST − BOLT | Flex. | +2.27 | [+0.15, +4.40] | 0.044 |
| GIST − Random | Strict | +2.65 | [+0.61, +4.70] | 0.014 |
| GIST − Random | Flex. | +1.14 | [−0.83, +3.11] | 0.293 |
| GIST − All candidates | Strict | +1.97 | [−0.08, +4.02] | 0.072 |
| GIST − All candidates | Flex. | +3.03 | [+0.99, +5.08] | 0.005 |
Table 8. Paired tests on GSM8K (Qwen3-0.6B, 1,319 questions). Differences are computed from per-question counts and may differ by 0.01 from differences of the rounded accuracies in the main text.
Below-mean tokens by sampling temperature. We measure the fraction of positions with in correct responses, excluding EOS. Pooled over tokens, it is 31.57% at and 31.93% at over all training prompts, and 31.38% and 31.75% over the 1,593 eligible prompts. With each response weighted equally, it is 31.39% and 30.69% over all prompts, and 31.22% and 30.56% over eligible prompts. Comparing correct responses within the same prompt, the high-temperature fraction is lower by 0.36 points over all prompts (95% CI [, ]) and by 0.57 points over eligible prompts ([, ]). Both temperatures thus contain a substantial fraction of below-mean tokens, and the sign of the small difference depends on aggregation, reflecting differences in response length. The probability of the selected tokens differs more: on the eligible prompts, below-mean tokens of correct responses have mean (median) of 0.583 (0.603) at and 0.411 (0.415) at . The above-mean tokens that selects in incorrect responses have mean (median) of 0.974 (0.998) at and 0.956 (0.993) at .
Retained Data and Supervision
Table 9 extends Table 6 with the retained responses, selected epochs, and optimizer steps. Supervised tokens count active loss positions through the selected checkpoint, including repeated use across epochs and the positions that selects in incorrect responses. Relative to training on all candidates, Random supervises 51.2% fewer tokens and GIST 66.6% fewer; GIST supervises 31.5% fewer than Random. Random and GIST retain the same number of responses but not the same responses, so the per-epoch reduction from masking also reflects response-length differences.
| Quantity | All candidates | Random | GIST |
| Retained responses | 50,976 | 12,744 | 12,744 |
| Selected epoch | 3 | 5 | 6 |
| Supervised tokens (M) | 13.596 | 6.641 | 4.546 |
| Supervised tokens per epoch (M) | 4.53 | 1.33 | 0.76 |
| Optimizer steps | 4,779 | 1,995 | 2,394 |
| Test Strict / Flex. (%) | 43.82 / 46.70 | 43.14 / 48.60 | 45.79 / 49.73 |
Table 9. Retained data and supervision on the shared 1,593-prompt pool (Qwen3-0.6B/GSM8K, BOLT).
Baselines and Ablation Controls
All selectors and controls use the shared candidate pool, the 1,593 eligible prompts, and the BOLT objective of Appendix § 12, unless stated otherwise.
Selection baselines. Random samples four correct and four incorrect responses per prompt uniformly from the combined temperature pools and supervises all eligible tokens. RFT-style retains all verifier-correct responses. The HES, PODS, ProFit, ETS, and RSI-S adaptations apply each method’s selection rule within this pipeline instead of reproducing its full training algorithm, and each retains eight responses per prompt. HES scores a response by the summed predictive entropy of its top 0.5% highest-entropy positions and keeps the four highest-scoring correct responses and four uniformly chosen incorrect ones. PODS keeps the eight responses that maximize the variance of their binary rewards, which gives four of each class. Both supervise all eligible tokens. ProFit, ETS, and RSI-S keep GIST’s responses and select tokens in both classes with one shared rule. ProFit keeps tokens whose frozen-policy probability is at least 0.1 (96.9% of tokens), ETS keeps the 20% of tokens with the highest predictive entropy, and RSI-S keeps tokens whose score lies in [] (99.3% of tokens).
Ablation controls (Tables 4 and 5). The Mask column of Table 4 and the token rules of Table 5 vary the treatment of correct responses, on which the BOLT weight concentrates. In Table 4, row (a) trains on all 32 candidates of each eligible prompt with all eligible tokens, and row (b) is the Random selector. Row (c) applies GIST’s masks to the same responses as Random. Row (d) keeps GIST’s responses, supervises all eligible tokens of correct responses, and uses for incorrect ones; it is the All tokens entry of Table 5. Row (e) keeps GIST’s responses and masks and sets for every response, which is SFT + GIST. In the token-rule ablation, Random keeps, in each correct response, as many uniformly chosen positions as GIST selects, and Highest-probability keeps the same number of highest-probability positions; incorrect responses use in all variants.
References
- [1]Jacob Austin, Augustus Odena, Maxwell Nye, Maarten Bosma, Henryk Michalewski, David Dohan, Ellen Jiang, Carrie Cai, Michael Terry, Quoc Le, and Charles Sutton. Program synthesis with large language models, 2021. URL https://arxiv.org/abs/2108.07732.
- [2]Sanghwan Bae et al. Online difficulty filtering for reasoning oriented reinforcement learning, 2025. URL https://arxiv.org/abs/2504.03380.
- [3]Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, Mark Chen, Heewoo Jun, Lukasz Kaiser, Matthias Plappert, Jerry Tworek, Jacob Hilton, Reiichiro Nakano, Christopher Hesse, and John Schulman. Training verifiers to solve math word problems, 2021. URL https://arxiv.org/abs/2110.14168.
- [4]Thomas M. Cover and Joy A. Thomas. Elements of Information Theory. Wiley, second edition, 2006.
- [5]Imre Csiszár. Information-type measures of difference of probability distributions and indirect observations. Studia Scientiarum Mathematicarum Hungarica, 2:299–318, 1967.
- [6]Ganqu Cui et al. The entropy mechanism of reinforcement learning for reasoning language models, 2025. URL https://arxiv.org/abs/2505.22617.
- [7]DeepSeek-AI. DeepSeek-R1: Incentivizing reasoning capability in LLMs via reinforcement learning, 2025. URL https://arxiv.org/abs/2501.12948.
- [8]Hanze Dong, Wei Xiong, Deepanshu Goyal, Yihan Zhang, Winnie Chow, Rui Pan, Shizhe Diao, Jipeng Zhang, Kashun Shum, and Tong Zhang. RAFT: Reward ranked finetuning for generative foundation model alignment. Transactions on Machine Learning Research, 2023. URL https://arxiv.org/abs/2304.06767.
- [9]Rotem Dror, Gili Baumer, Segev Shlomov, and Roi Reichart. The hitchhiker’s guide to testing statistical significance in natural language processing. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics, 2018. URL https://aclanthology.org/P18-1128/.DOI
- [10]Bradley Efron and Robert J. Tibshirani. An Introduction to the Bootstrap. Chapman & Hall, 1993.
- [11]Leo Gao et al. A framework for few-shot language model evaluation. https://github.com/ EleutherAI/lm-evaluation-harness, 2024.
- [12]Gemma Team. Gemma 4 technical report, 2026. URL https://arxiv.org/abs/2607. 02770.
- [13]Dongyoung Go, Tomasz Korbak, Germán Kruszewski, Jos Rozen, Nahyeon Ryu, and Marc Dymet- man. Aligning language models with preferences through f-divergence minimization. In Inter- national Conference on Machine Learning, 2023. URL https://arxiv.org/abs/2302. 08215.
- [14]Yuxian Gu, Li Dong, Furu Wei, and Minlie Huang. MiniLLM: Knowledge distillation of large language models. In International Conference on Learning Representations, 2024. URL https: //arxiv.org/abs/2306.08543.arxiv.org/abs/2306.08543
- [15]Caglar Gulcehre et al. Reinforced self-training (ReST) for language modeling, 2023. URL https: //arxiv.org/abs/2308.08998.arxiv.org/abs/2308.08998
- [16]Tuomas Haarnoja, Haoran Tang, Pieter Abbeel, and Sergey Levine. Reinforcement learning with deep energy-based policies. In International Conference on Machine Learning, 2017. URL https://arxiv.org/abs/1702.08165.
- [17]Dan Hendrycks, Collin Burns, Saurav Kadavath, Akul Arora, Steven Basart, Eric Tang, Dawn Song, and Jacob Steinhardt. Measuring mathematical problem solving with the MATH dataset. In Advances in Neural Information Processing Systems Datasets and Benchmarks Track, 2021. URL https://arxiv.org/abs/2103.03874.
- [18]Ari Holtzman, Jan Buys, Li Du, Maxwell Forbes, and Yejin Choi. The curious case of neural text degeneration. In International Conference on Learning Representations, 2020. URL https: //arxiv.org/abs/1904.09751.arxiv.org/abs/1904.09751
- [19]Tomasz Korbak, Ethan Perez, and Christopher L. Buckley. RL with KL penalties is better viewed as Bayesian inference. In Findings of the Association for Computational Linguistics: EMNLP 2022, 2022. URL https://arxiv.org/abs/2205.11275.
- [20]Nathan Lambert et al. Tülu 3: Pushing frontiers in open language model post-training, 2024. URL https: //arxiv.org/abs/2411.15124.arxiv.org/abs/2411.15124
- [21]Sergey Levine, Aviral Kumar, George Tucker, and Justin Fu. Offline reinforcement learning: Tutorial, review, and perspectives on open problems, 2020. URL https://arxiv.org/abs/2005. 01643.
- [22]Xiaoyuan Li, Yubo Ma, Chengpeng Li, Fengbin Zhu, YiYao Yu, Keqin Bao, Wenjie Wang, Fuli Feng, and Daiheng Liu. Unified data selection for LLM reasoning, 2026. URL https: //arxiv.org/abs/2605.22389.arxiv.org/abs/2605.22389
- [23]Xuefeng Li, Haoyang Zou, and Pengfei Liu. LIMR: Less is more for RL scaling, 2025. URL https://arxiv.org/abs/2502.11886.
- [24]Hunter Lightman, Vineet Kosaraju, Yura Burda, Harri Edwards, Bowen Baker, Teddy Lee, Jan Leike, John Schulman, Ilya Sutskever, and Karl Cobbe. Let’s verify step by step. In International Conference on Learning Representations, 2024. URL https://arxiv.org/abs/2305. 20050.
- [25]Zhenghao Lin, Zhibin Gou, Yeyun Gong, Xiao Liu, Yelong Shen, Ruochen Xu, Chen Lin, Yujiu Yang, Jian Jiao, Nan Duan, and Weizhu Chen. Rho-1: Not all tokens are what you need, 2024a. URL https://arxiv.org/abs/2404.07965.
- [26]Zicheng Lin et al. Critical tokens matter: Token-level contrastive estimation enhances LLM’s reasoning capability, 2024b. URL https://arxiv.org/abs/2411.19943.
- [27]Tao Liu, Taijiang Wu, Rumming Yang, Shaoning Sun, Junjie Wang, and Yujiu Yang. ProFit: Leveraging high-value signals in SFT via probability-guided token selection. In Findings of the Association for Computational Linguistics: ACL 2026, pp. 15383–15401, San Diego, California, United States, July 2026. Association for Computational Linguistics. doi: 10.18653/v1/2026.findings-acl.755. URL https://aclanthology.org/2026.findings-acl.755/.DOI
- [28]Zichen Liu, Changyu Chen, Wenjun Li, Penghui Qi, Tianyu Pang, Chao Du, Wee Sun Lee, and Min Lin. Understanding R1-Zero-like training: A critical perspective, 2025. URL https://arxiv.org/abs/2503.20783.
- [29]Outongyi Lv, Yuanwei Zhang, and Xiaoqun Zhang. GMTS: Gradient magnitude-based token selection improves RLVR training for LLM reasoning, 2026a. URL https://arxiv.org/abs/2608.30632.
- [30]Outongyi Lv, Yanzhao Zheng, Yuanwei Zhang, Zhenghao Huang, Xingjun Wang, Baohua Dong, Hangcheng Zhu, and Yingda Chen. Which tokens matter? Adaptive token selection for RLVR with the relative surprisal index, 2026b. URL https://arxiv.org/abs/2606.31575.
- [31]Quinn McNemar. Note on the sampling error of the difference between correlated proportions or percentages. Psychometrika, 12(2):153–157, 1947.pubmed.ncbi.nlm.nih.gov/20254758
- [32]Tom Minka. Divergence measures and message passing. Technical Report MSR-TR-2005-173, Microsoft Research, 2005.
- [33]Baharan Mirzasoleiman, Jeff Bilmes, and Jure Leskovec. Coresets for data-efficient training of machine learning models. International Conference on Machine Learning, 2020. URL https://arxiv.org/abs/1906.01827.
- [34]Long Ouyang et al. Training language models to follow instructions with human feedback. In Advances in Neural Information Processing Systems, 2022. URL https://arxiv.org/abs/2203.02155.
- [35]Jinlong Pang, Na Di, Zhaowei Zhu, Jiaheng Wei, Hao Cheng, Chen Qian, and Yang Liu. Token cleaning: Fine-grained data selection for LLM supervised fine-tuning. In Proceedings of the 42nd International Conference on Machine Learning, volume 267 of Proceedings of Machine Learning Research, pp. 47837–47858. PMLR, 2025. URL https://proceedings.mlr.press/v267/pang25a.html.arxiv.org/abs/2502.01968
- [36]Xue Bin Peng, Aviral Kumar, Grace Zhang, and Sergey Levine. Advantage-weighted regression: Simple and scalable off-policy reinforcement learning, 2019. URL https://arxiv.org/abs/1910.00177.
- [37]Jan Peters and Stefan Schaal. Reinforcement learning by reward-weighted regression for operational space control. In International Conference on Machine Learning, 2007.DOI
- [38]Jan Peters, Katharina Mülling, and Yasemin Altun. Relative entropy policy search. In Proceedings of the AAAI Conference on Artificial Intelligence, 2010.DOI
- [39]Qwen Team. Qwen3.5: Towards native multimodal agents. https://qwen.ai/blog?id=qwen3.5, 2026.
- [40]Rafael Rafailov, Archit Sharma, Eric Mitchell, Christopher D. Manning, Stefano Ermon, and Chelsea Finn. Direct preference optimization: Your language model is secretly a reward model. In Advances in Neural Information Processing Systems, volume 36, pp. 53728–53741, 2023. URL https://arxiv.org/abs/2305.18290.
- [41]Rafael Rafailov, Joey Hejna, Ryan Park, and Chelsea Finn. From r to q*: Your language model is secretly a Q-function. In Conference on Language Modeling, 2024. URL https://arxiv.org/abs/2404.12358.
- [42]John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal policy optimization algorithms, 2017. URL https://arxiv.org/abs/1707.06347.
- [43]Ozan Sener and Silvio Savarese. Active learning for convolutional neural networks: A core-set approach. In International Conference on Learning Representations, 2018. URL https://arxiv.org/abs/1708.00489.
- [44]Zhihong Shao, Peiyi Wang, Qihao Zhu, Runxin Xu, Junxiao Song, Xiao Bi, Haowei Zhang, Mingchuan Zhang, Y. K. Li, Y. Wu, and Daya Guo. DeepSeekMath: Pushing the limits of mathematical reasoning in open language models, 2024. URL https://arxiv.org/abs/2402.03300.
- [45]Yao Shu, Chenxing Wei, Hongbin Lin, Shuang Qiu, and Hui Xiong. Reference-sampled Boltzmann projection for KL-Regularized RLVR: Target-matched weighted SFT, finite one-shot gaps, and policy mirror descent, 2026. URL https://arxiv.org/abs/2605.02469.
- [46]Avi Singh et al. Beyond human data: Scaling self-training for problem-solving with language models. Transactions on Machine Learning Research, 2024. URL https://arxiv.org/abs/2312.06585.
- [47]Charlie Victor Snell, Ilya Kostrikov, Yi Su, Sherry Yang, and Sergey Levine. Offline RL for natural language generation with implicit language Q learning. In International Conference on Learning Representations, 2023. URL https://arxiv.org/abs/2206.11871.
- [48]Chaoqi Wang, Yibo Jiang, Chenghao Yang, Han Liu, and Yuxin Chen. Beyond reverse KL: Generalizing direct preference optimization with diverse divergence constraints. In International Conference on Learning Representations, 2024a. URL https://arxiv.org/abs/2309.16240.
- [49]Huaijie Wang, Shibo Hao, Hanze Dong, Shenao Zhang, Yilin Bao, Ziran Yang, and Yi Wu. Offline reinforcement learning for LLM multi-step reasoning, 2024b. URL https://arxiv.org/abs/2412.16145.
- [50]Shenzhi Wang, Le Yu, Chang Gao, Chujie Zheng, Shixuan Liu, Rui Lu, Kai Dang, Xionghui Chen, Jianxin Yang, Zhenru Zhang, Yuqiong Liu, An Yang, Andrew Zhao, Yang Yue, Shiji Song, Bowen Yu, Gao Huang, and Junyang Lin. Beyond the 80/20 rule: High-entropy minority tokens drive effective reinforcement learning for LLM reasoning. In Advances in Neural Information Processing Systems, 2025a. URL https://arxiv.org/abs/2506.01939.
- [51]Yiping Wang et al. Reinforcement learning for reasoning in large language models with one training example, 2025b. URL https://arxiv.org/abs/2504.20571.
- [52]Mengzhou Xia, Sadhika Malladi, Suchin Gururangan, Sanjeev Arora, and Danqi Chen. LESS: Selecting influential data for targeted instruction tuning. In International Conference on Machine Learning, 2024. URL https://arxiv.org/abs/2402.04333.
- [53]Yixuan Even Xu, Yash Savani, Fei Fang, and J. Zico Kolter. Not all rollouts are useful: Down-sampling rollouts in LLM reinforcement learning, 2025. URL https://arxiv.org/abs/2504.13818.
- [54]An Yang et al. Qwen3 technical report, 2025. URL https://arxiv.org/abs/2505.09388.
- [55]Qiying Yu, Zheng Zhang, Ruofei Zhu, Yufeng Yuan, Xiaochен Zuo, Yu Yue, Weinan Dai, Tiantian Fan, Gaohong Liu, Lingjun Liu, Xin Liu, Haibin Lin, Zhiji Lin, Bole Ma, Guangming Sheng, Yuxuan Tong, Chi Zhang, Mofan Zhang, Wang Zhang, Hang Zhu, Jinhua Zhu, Jiaze Chen, Jiangjie Chen, Chengyi Wang, Hongli Yu, Yuxuan Song, Xiangpeng Wei, Hao Zhou, Jingjing Liu, Wei-Ying Ma, Ya-Qin Zhang, Lin Yan, Mu Qiao, Yonghui Wu, and Mingxuan Wang. DAPO: An open-source LLM reinforcement learning system at scale. In Advances in Neural Information Processing Systems, 2025. URL https://arxiv.org/abs/2503.14476.
- [56]Zheng Yuan, Hongyi Yuan, Chengpeng Li, Guanting Dong, Keming Lu, Chuanqi Tan, Chang Zhou, and Jingren Zhou. Scaling relationship on learning mathematical reasoning with large language models, 2023. URL https://arxiv.org/abs/2308.01825.
- [57]Yang Yue et al. Does reinforcement learning really incentivize reasoning capacity in LLMs beyond the base model? In Advances in Neural Information Processing Systems, 2025. URL https://arxiv.org/abs/2504.13837.
- [58]Eric Zelikman, Yuhuai Wu, Jesse Mu, and Noah D. Goodman. STaR: Bootstrapping reasoning with reasoning. In Advances in Neural Information Processing Systems, 2022. URL https://arxiv.org/abs/2203.14465.
- [59]Chunting Zhou et al. LIMA: Less is more for alignment. In Advances in Neural Information Processing Systems, 2023. URL https://arxiv.org/abs/2305.11206.
- [60]Brian D. Ziebart. Modeling Purposeful Adaptive Behavior with the Principle of Maximum Causal Entropy. PhD thesis, Carnegie Mellon University, 2010.DOI


