WorksComputer Science
SpeCal: Spectral Calibration for Unsupervised RLVR
Abstract
Reinforcement learning with verifiable rewards (RLVR) has substantially improved the reasoning abilities of large language models, but its dependence on ground-truth answers or external verifiers limits its ability to exploit abundant unlabeled reasoning data. Unsupervised RLVR alleviates this bottleneck by deriving scalar rewards from the model's own rollouts. However, the reliability of such self-generated supervision varies substantially across rollout groups and individual responses, while existing methods construct or refine their rewards mainly from decoded outcomes and reward statistics. These endpoint signals reveal what answers the model produces, but provide limited information about whether the underlying rollout representations form a reliable structure for learning. Our key observation is that hidden-state spectra offer a complementary view of supervision reliability by characterizing both the organization of a rollout group and the support received by each rollout within that group. Based on this observation, we propose SpeCal, a bi-level spectral calibration framework for unsupervised RLVR. At the group level, SpeCal combines spectral mass with a Gaussian band-pass gate over normalized entropy-based effective rank, favoring rollout groups with substantial representation variation whose effective dimensionality lies in a preferred regime. At the rollout level, the same group effective rank determines how many dominant directions are retained in a leave-one-out spectral subspace formed by the remaining rollouts, within which the conditional support of each held-out rollout is measured. The resulting signals calibrate the original reward-induced advantages at the group and rollout levels without redefining the underlying unlabeled reward.
Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than the author(s) must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected].
KDD ’27, San Jose, CA, USA
© 2027 Copyright held by the owner/author(s). Publication rights licensed to ACM. ACM ISBN XXX-XXXX https://doi.org/XXXX
CCS Concepts
• Computing methodologies → Natural language processing; Unsupervised learning.
Keywords
reinforcement learning, large language models, unsupervised learning, spectral methods
ACM Reference Format:
Kai Tang, Xing Zheng, Haobo Wang, Guangcheng Zhu, Shenzhi Yang, Zhengqing Zang, Yingfan Ma, Cheng Peng, Bowen Song, and Weiqiang Wang. 2027. SpeCal: Spectral Calibration for Unsupervised RLVR. In KDD ’27: KDD 2027, August 2027, San Jose, CA, USA. ACM, New York, NY, USA, 13 pages. https://doi.org/XXXX
Introduction
Reinforcement learning with verifiable rewards (RLVR) has become an effective paradigm for improving the reasoning ability of large language models (LLMs) [7, 11, 22, 38], especially in domains where final outcomes can be checked by rule-based verifiers or executable tests [28, 35]. However, the scalability of standard RLVR is constrained by the availability of externally verifiable training questions, including ground-truth answers, executable tests, and reliable domain-specific verifiers [23, 24, 30]. This requirement limits its ability to exploit the large amount of unlabeled reasoning data. Unsupervised RLVR aims to relax this bottleneck by deriving training signals from the model’s own rollouts on unlabeled questions, preserving the reward-based optimization paradigm of RLVR while reducing the dependence on external supervision. Most unsupervised RLVR methods derive pseudo-rewards from output-derived signals computed over a group of sampled rollouts, including majority voting [46], predictive confidence [1], and inter-response agreement [43]. Despite their different reward constructions, these approaches generally aggregate output evidence using predefined rules, without explicitly modeling how supervision reliability varies at different granularities. Yet the reliability of self-generated supervision can differ substantially across questions, answer candidates, and individual rollouts. To this end, recent methods refine these signals at multiple granularities using richer statistics derived from decoded outputs. DARE retains the full empirical answer distribution rather than reducing it to a single majority outcome, while RESTRAIN modulates candidate- and question-level contributions using answer frequency and group consistency [8, 41].
Although these refinements preserve more output-side information, their reliability estimates remain functions of compressed summaries of decoded outcomes or predictive confidence. However, compressing a rollout group into these output-derived summaries is inherently information-losing: two groups can have identical final-answer histograms and confidence statistics while differing substantially in their internal generation processes. Once this information has been discarded by the compression, no downstream estimator that observes only the resulting summaries can recover it when assessing the reliability of self-generated supervision.
Fortunately, the model’s hidden states offer a complementary view of these differences. Studies of individual hidden states have shown that internal states distinguish truthful from hallucinated generations [3, 5, 9], while Chain-of-Embedding characterizes how hidden states evolve across layers and uses the resulting latent trajectory for label-free response evaluation [34]. Together, these studies demonstrate that hidden states encode rich signals about the model’s internal generation dynamics and the reliability of its responses. However, they predominantly evaluate responses in isolation, whereas unsupervised RLVR needs to derive supervision through relative comparisons among multiple rollouts. The relevant object is therefore not an isolated hidden state, but the joint geometry formed by the rollout representations. For a set of centered representations, the eigenspectrum of their Gram matrix provides a compact and permutation-invariant description of this geometry: spectral mass measures the total inter-rollout variation around the group mean [4], while effective rank characterizes how this variation is distributed across principal directions [10].
Building on this spectral view, we propose SpeCal, a two-level spectral calibration framework for unsupervised RLVR. Rather than constructing another pseudo-reward, SpeCal uses hidden-state geometry as complementary structural evidence for calibrating the advantages induced by an existing unlabeled reward scheme. At the group level, it combines spectral mass with an effective-rank-based band-pass gate to modulate the contribution of the entire rollout group according to the magnitude and directional organization of its representational variation. This allows SpeCal to distinguish informative rollout structures from collapsed or fragmented representation patterns, even when their endpoint statistics appear similar. At the rollout level, SpeCal reuses the group effective rank to determine the number of dominant directions retained in a leave-one-out spectral subspace. For each rollout, the reference subspace is constructed from the remaining responses sampled for the same prompt, and the rollout is evaluated according to its conditional support within this cross-rollout structure. This design prevents a trajectory from being evaluated solely by the directions it creates itself, while measuring whether its representation is supported by the shared structure of the group.
We conduct extensive experiments on two 8B model backbones across six in-domain mathematical-reasoning benchmarks and three out-of-domain general-reasoning benchmarks. Averaged across the two backbones, SpeCal improves over the strongest unsupervised RLVR baselines by 2.7 points in-domain and 1.1 points out-of-domain, while matching or surpassing fully supervised RLVR in several settings. SpeCal consistently improves multiple unlabeled reward schemes, demonstrating that its gains are not tied to a particular pseudo-reward construction. Further ablations verify the complementary contributions of group-level spectral calibration and rollout-level conditional support. These results show that hidden-state spectral geometry provides an effective and general reliability signal for learning from self-generated supervision.
Preliminary
Standard RLVR Paradigm
Reinforcement learning with verifiable rewards (RLVR) optimizes language models using outcome rewards provided by rule-based verifiers. Formally, given a labeled dataset of question–answer pairs , the policy generates a response for each question . Let denote the final answer extracted from , the verifier assigns a binary reward , where is the indicator function. We adopt the widely used Group Relative Policy Optimization (GRPO) [12] as our base algorithm. GRPO eliminates the need for a learned value model by estimating advantages from the rewards of multiple responses to the same question. Specifically, for each question , we sample responses from the old policy and compute their rewards . The group-normalized advantage is given by
Then, the GRPO objective is defined as
Here, is the importance weight, is the clipped surrogate objective, and denotes the KL divergence.
Unsupervised RLVR
Unlike standard RLVR, unsupervised RLVR operates on an unlabeled question set and derives rewards from the model’s own rollouts. For each question , the policy samples responses and extracts their final answers . Given a pseudo-label inferred from these responses, the unlabeled reward is commonly defined as . A representative approach is majority voting [46], which constructs the pseudo-label as . The resulting rewards are normalized as in Eq. (1) and optimized using the GRPO objective in Eq. (2). More generally, SpeCal is agnostic to how the pseudo-label or unlabeled reward is constructed. Since these rewards are inferred from self-generated outcomes rather than external verification, their induced advantages may have heterogeneous reliability, which SpeCal calibrates without modifying the underlying reward.
Methods
In this section, we introduce SpeCal, a fine-grained bi-level calibration framework for unsupervised RLVR. As shown in Figure 1, SpeCal evaluates the balanced spectral geometry across rollouts at the group level, and measures each trajectory’s conditional spectral support against the leave-one-out subspace at the rollout level.
Figure 1. Overview of SpeCal. Given a rollout group, SpeCal constructs a centered Gram matrix from the hidden-state representations. Group-level calibration combines spectral mass with a band-pass gate over normalized effective rank , while rollout-level calibration scores each rollout against the leave-one-out denoised subspace formed by the remaining rollouts. The resulting weights and calibrate the original unsupervised advantages without modifying the reward.
Group Calibration via Spectral Geometry
Response Representations and Group Spectrum
For the -th rollout with generated tokens, let denote the hidden state of token at layer . We aggregate information across a selected set of layers and then mean-pool over all generated tokens:
Mean pooling is motivated by the observation that semantic information is distributed throughout the generated sequence, making the aggregated representation more informative than relying on any single token state [32]. We then obtain the normalized response representation and stack all representations into the group matrix . To isolate variation across rollouts, we center the group representations and construct their Gram matrix :
Since is positive semidefinite and is centered, its rank is at most . We denote the group spectral dimension as , and write the corresponding eigenvalues as , including zero-valued directions when the effective rank is smaller than . These spectral quantities are used below to define the group-level calibration score.
Group-Level Spectral Calibration
A rollout group provides a useful learning signal only when it offers nontrivial yet coherent semantic variation. If the responses collapse to a small region in representation space, the group lacks semantic contrast for relative comparison; if the responses disperse into too many weakly related modes, the variation may instead reflect outliers or fragmented reasoning paths. This motivates a calibration score that considers both the amount of inter-rollout variation and the effective dimensionality of this variation. Given the group spectrum , we first compute the spectral mass , which measures the overall dispersion of the rollout representations. We then normalize the eigenspectrum as and compute the entropy effective rank [26]:
The normalized effective rank estimates the fraction of available spectral directions that are effectively used by the group [10].
We do not assume that a larger effective rank is always preferable. Very small indicates collapse or single-axis variation, whereas overly large may correspond to fragmented groups with weak shared structure. We therefore use a Gaussian band-pass gate over and modulate the spectral mass by this gate:
Here, denotes the target normalized effective rank and controls the tolerance of effect rank. Thus, becomes large only when the rollout group has sufficient spectral mass and its effective dimensionality lies in the desired range. The unnormalized effective rank also provides the spectral bandwidth used in trajectory-level calibration. We set , which estimates how many dominant semantic directions should be retained when constructing leave-one-out subspaces below.
Rollout Calibration via Spectral Support
Denoised Leave-One-Out Subspace
The group-level weight calibrates the overall usefulness of a rollout group, but it does not specify how the learning signal should be distributed among individual rollouts. For rollout-level calibration, a response should not certify itself by the spectral directions it alone creates: unusual hidden variation may arise from noise, verbosity, or idiosyncratic reasoning rather than reliable semantic structure [2, 18]. We therefore evaluate each trajectory against a reference geometry independently formed by the remaining rollouts. Specifically, for the -th trajectory, we remove from the group and write . Let be the leave-one-out mean, and define the centered leave-one-out matrix as .
To denoise this reference geometry, we do not use the full span of the remaining rollouts, since weak spectral directions may correspond to accidental or unstable variation. Instead, we retain only the dominant spectral directions that capture the effective semantic structure of the rollout group. Specifically, we compute the compact SVD of the centered leave-one-out matrix:
Here, , and the columns of represent the spectral directions supported by the remaining rollouts. The group-level effective rank provides an adaptive spectral bandwidth for this leave-one-out subspace: we retain the top available directions from and denote the truncated basis by . The resulting denoised leave-one-out projector is
This construction preserves the dominant directions supported by the remaining rollouts while discarding weaker directions beyond the group-level effective dimensionality. In this way, leave-one-out prevents self-certification, and effective-rank truncation prevents the trajectory from being evaluated against noisy or accidental spectral directions.
Conditional Spectral Support
Given the effective-rank truncated leave-one-out projector , we measure how much of the held-out rollout is supported by the semantic structure established by the remaining rollouts. Specifically, using the centered rollout representation , we define the conditional spectral support score as:
The numerator measures the projected energy of onto the effective leave-one-out spectral subspace [21], while the denominator measures its total energy relative to the same reference mean. Therefore, measures conditional support rather than intrinsic trajectory complexity: a trajectory receives a high score only when its group-relative variation is explained by dominant directions independently supported by the remaining rollouts.
We then normalize the support scores within each rollout group and apply clipping to obtain the trajectory-level calibration weight:
Here, and are clipping bounds. The group-wise normalization keeps the average trajectory weight close to one before clipping, so redistributes learning emphasis among rollouts within the same group rather than changing the overall strength of the prompt-level update.
Bi-Level Calibrated Objective
We finally combine the two calibration levels by reweighting the original unsupervised advantage.
Let denote the group-level calibration coefficient and denote the trajectory-level calibration coefficient. The calibrated advantage is defined as
Here, denotes stop-gradient, and denotes RMS normalization [42] within the batch and rollout group respectively, which preserves bi-level advantages scale after reweighting. Thus, controls how strongly the whole rollout group contributes to learning, while only redistributes learning emphasis among trajectories within the group. The policy is then optimized with the same GRPO objective as Eq. (2), replacing with :
Theoretical Analysis
We analyze SpeCal from two complementary perspectives. First, we characterize the information limitation of endpoint-based unsupervised signals. Second, we show that the spectral quantities used by SpeCal estimate a well-defined marginal geometry of the rollout process. Formal assumptions and proofs are deferred to Appendix A.
Information Loss of Endpoint Signals
Let denote the complete rollout evidence for question , including both decoded outcomes and response representations . Existing unsupervised methods typically construct their learning signals from an endpoint summary . The following proposition formalizes the information loss induced by this compression.
Proposition 4.1 (Endpoint Information Loss). For any random variable describing the reliability of the induced supervision
signal,
The gap is strict whenever endpoint-equivalent rollout groups can have different conditional supervision reliability.
Proposition 4.1 shows that the limitation of endpoint-based methods is not restricted to a particular reward design: once the rollout group is compressed into decoded statistics, reliability-relevant representational information may become irrecoverable.
Marginal Spectral Consistency
For a fixed question and policy , let and be two i.i.d. response representations sampled from . We define the marginal inter-rollout geometry as , and estimate it using the centered empirical matrix . To distinguish finite-group estimates from their marginal counterparts, we use and to denote the group weight and conditional support computed by SpeCal from the sampled rollout group; these correspond to and in Sec. 3. We denote by and the same spectral constructions evaluated on the marginal geometry.
Proposition 4.2 (Marginal Spectral Consistency). Under the regularity and rank-stability conditions stated in Appendix A,
Proposition 4.2 shows that the finite-group calibration signals used by SpeCal converge to well-defined properties of the marginal rollout geometry, rather than depending on arbitrary spectral fluctuations of a particular sampled group.
Experiments
Setup
Implementation Details
Our implementation is built upon the verl framework [29] and follows the standard GRPO training recipe [12]. We conduct training on the challenging subset of DeepMath-103K [28] with difficulty scores . To evaluate the generality of SpeCal across different model families and initial reasoning capabilities, we experiment with backbones from both the Qwen and Llama families, including Qwen3-8B-Base and DeepSeek-R1-Distill-Llama-8B. All experiments are performed on 8 NVIDIA A100 GPUs with a total batch size of 128, a micro-batch size of 32, and a learning rate of . For each prompt, we sample rollouts during training. To construct hidden-state representations, we uniformly select eight layers across the depth of the policy model. For group-level calibration, we set the target normalized effective rank to and its tolerance to . For rollout-level calibration, the conditional support weights are clipped to . We use the same SpeCal hyperparameters across all backbones and settings. Additionally, unless otherwise specified, all analyses in this section are conducted on Qwen3-8B-Base using the default training and evaluation configurations. Additional implementation details are provided in Appendix B.
Evaluation
Following the evaluation protocols adopted in prior work [37, 39, 45], we conduct comprehensive evaluations on a diverse collection of mathematical and general reasoning benchmarks. For mathematical reasoning, we consider six competition-level datasets: AIME 2024, AIME 2025, MATH-500 [16], Minerva [19], AMC [20], and OlympiadBench [15]. To evaluate out-of-domain generalization, we further include four general reasoning benchmarks: ARC-c [6], GPQA-Diamond [25] (denoted as GPQA), and MMLU-Pro [33]. To account for differences in test-set size, we report avg@32 for AIME 2024/2025 and AMC, pass@1 for MMLU-Pro, and avg@4 for the remaining benchmarks. All evaluations use a temperature of 0.6 and top- of 1.0 under a consistent decoding configuration.
Baselines
We compare SpeCal with six representative unsupervised RLVR baselines spanning consensus-based pseudo-labeling, entropy and confidence based intrinsic rewards, and teacher-guided supervision. (1) TTRL [46] takes the majority-voted answer within each rollout group as the pseudo-label and assigns positive rewards to responses that match it. (2) Tok-entropy [1] uses the average token entropy as an intrinsic confidence signal, favoring rollouts with more concentrated token-level predictions. (3) Seq-entropy [1] measures confidence at the sequence level and rewards responses with higher overall generation likelihood. (4) Self-certainty [44] quantifies intrinsic certainty through the KL divergence between the predicted token distributions and the uniform distribution, assigning larger rewards to more decisive rollouts. (5) SCRL [36] improves the robustness of consensus-based supervision through selective-complementary reward construction: it applies strict consensus criteria to retain reliable positive pseudo-labels and introduces entropy-gated negative pseudo-labeling to suppress likely incorrect trajectories when the answer distribution is dispersed. (6) Co-rewarding [43] maintains a slowly updated reference teacher and constructs rewards according to the agreement between sampled responses and teacher-generated pseudo-labels. For a fair comparison, all baselines are implemented using the same model backbones, training data, rollout budget, and GRPO optimization configuration. About other related work mentioned above like RESTRAIN [41] and DARE [8], we did not make a comparison because the code has not yet been open-sourced.
Main Results
Table 1 compares SpeCal with six representative unsupervised RLVR baselines on two 8B model backbones. We report mathematical reasoning tasks and out-of-domain generalization tasks separately, while including the before-RL and fully supervised results as reference points.
Table 1: Main results on Qwen3-8B-Base and DeepSeek-R1-Distill-Llama-8B. Each entry reports the mean accuracy over three independent runs with different random seeds. ID Avg. is computed over six mathematical-reasoning benchmarks, while OOD Avg. is computed over three general-reasoning benchmarks. Bold marks the highest mean among unsupervised methods, while gray rows provide the before-RL and fully supervised references.
| Methods | In-Domain Performance | Out-of-Domain Performance | ||||||||
| AIME 24/25 | AMC | MATH-500 | Minerva | Olympiad | Avg. | ARC-c | GPQA | MMLU-Pro | Avg. | |
| Qwen3-8B-Base | ||||||||||
| Before RL | 9.4/7.3 | 38.0 | 64.4 | 25.6 | 32.5 | 29.5 | 29.7 | 11.5 | 46.6 | 29.3 |
| Fully Supervised | 23.1/19.8 | 55.1 | 85.0 | 46.5 | 50.9 | 46.7 | 93.9 | 50.5 | 65.5 | 70.0 |
| Tok-entropy | 12.3/11.1 | 44.8 | 78.5 | 34.6 | 41.8 | 37.2 | 43.1 | 10.2 | 56.7 | 36.7 |
| Seq-entropy | 14.2/15.8 | 51.9 | 78.3 | 38.0 | 43.1 | 40.2 | 81.3 | 20.8 | 59.5 | 53.9 |
| TTRL | 17.7/12.5 | 54.4 | 81.7 | 43.8 | 46.0 | 42.7 | 91.3 | 43.6 | 63.0 | 66.0 |
| Self-certainty | 9.1/7.7 | 45.2 | 77.1 | 41.1 | 39.7 | 36.6 | 64.4 | 18.9 | 59.2 | 47.5 |
| SCRL | 16.7/14.2 | 49.4 | 76.2 | 40.4 | 40.2 | 39.5 | 90.2 | 41.6 | 62.1 | 64.6 |
| Co-rewarding | 14.4/12.2 | 52.3 | 79.2 | 43.1 | 44.4 | 40.9 | 91.0 | 27.8 | 62.7 | 60.5 |
| SpeCal (ours) | 20.9/16.3 | 58.1 | 83.9 | 44.7 | 49.5 | 45.6 | 91.7 | 46.7 | 63.4 | 67.3 |
| DeepSeek-R1-Distill-Llama-8B | ||||||||||
| Before RL | 13.3/23.3 | 50.6 | 72.6 | 26.1 | 40.4 | 37.7 | 5.3 | 9.3 | 42.1 | 18.9 |
| Fully Supervised | 43.3/23.3 | 66.3 | 82.2 | 36.8 | 54.7 | 51.1 | 46.9 | 37.3 | 51.2 | 45.1 |
| Tok-entropy | 3.3/3.3 | 36.1 | 48.6 | 15.4 | 27.6 | 22.4 | 34.3 | 7.6 | 25.9 | 22.6 |
| Seq-entropy | 6.7/3.3 | 28.9 | 48.6 | 15.8 | 29.4 | 22.1 | 40.3 | 14.3 | 27.4 | 27.3 |
| TTRL | 33.3/16.7 | 66.3 | 82.2 | 34.6 | 54.5 | 47.9 | 23.8 | 36.9 | 48.7 | 36.5 |
| Self-certainty | 16.7/16.7 | 51.8 | 73.0 | 26.1 | 43.8 | 38.0 | 25.3 | 16.2 | 44.1 | 28.5 |
| SCRL | 21.7/20.5 | 61.4 | 77.8 | 31.3 | 43.1 | 42.6 | 58.7 | 40.1 | 51.2 | 50.0 |
| Co-rewarding | 30.0/20.0 | 69.9 | 79.6 | 28.7 | 48.1 | 46.1 | 31.3 | 29.9 | 51.2 | 37.5 |
| SpeCal (ours) | 33.3/23.3 | 69.9 | 83.6 | 35.5 | 55.9 | 50.3 | 58.1 | 40.7 | 53.9 | 50.9 |
Table 1. Main results on Qwen3-8B-Base and DeepSeek-R1-Distill-Llama-8B.
SpeCal delivers consistent in-domain improvements across both model families. On Qwen3-8B-Base, it raises the average accuracy from 42.7 with TTRL to 45.6, corresponding to a 2.9-point absolute gain and a 6.8% relative improvement. The gain is broad across tasks: SpeCal improves all six mathematical-reasoning benchmarks, with increases ranging from 0.9 points on Minerva to 3.8 points on AIME 2025. A similar pattern is observed on DeepSeek-R1-Distill-Llama-8B, where SpeCal improves the strongest unsupervised average from 47.9 to 50.3. It improves five benchmarks over TTRL and ties it on AIME 2024. Averaged across the two backbones, SpeCal improves the strongest unsupervised baseline by 2.7 points in-domain. In comparison, the entropy-based methods vary considerably across model families and reduce the DeepSeek model below its before-RL performance, highlighting the difficulty of transferring a fixed intrinsic reward heuristic across different policy models. SpeCal also achieves the best unsupervised OOD average on both model backbones. On Qwen3-8B-Base, it reaches an OOD average of 67.3, outperforming TTRL at 66.0 by 1.3 points. SpeCal improves over TTRL on all three OOD benchmarks, with gains of 0.4 points on ARC-c, 3.1 points on GPQA, and 0.4 points on MMLU-Pro. On DeepSeek-R1-Distill-Llama-8B, SpeCal obtains an OOD average of 50.9, exceeding the strongest baseline, SCRL at 50.0, by 0.9 points. At the task level, it outperforms the strongest baseline by 0.6 points on GPQA and 2.7 points on MMLU-Pro, while trailing SCRL by 0.6 points on ARC-c. Averaged across the two backbones, SpeCal improves the strongest unsupervised OOD baseline by 1.1 points. These results demonstrate consistent average-level gains across model families, while also showing that the task-level advantage depends on the target distribution.
Notably, SpeCal approaches or even surpasses fully supervised RLVR in several settings. Its in-domain average is only 1.1 points below the fully supervised reference on Qwen3-8B-Base and 0.8 points below it on DeepSeek-R1-Distill-Llama-8B. SpeCal outperforms the supervised reference on AMC with Qwen3-8B-Base. On DeepSeek-R1-Distill-Llama-8B, it exceeds the supervised reference on MATH-500 and OlympiadBench and matches it on AIME 2025 and AMC. SpeCal also surpasses fully supervised RLVR on all three OOD benchmarks with the Llama backbone. These results highlight the strong competitiveness of SpeCal and the potential of unsupervised RLVR as a scalable alternative when manually annotated rewards are unavailable.
Further Analysis
Ablation Study
We ablate SpeCal according to its natural bi-level decomposition: group-level calibration contains two spectral geometry factors, while trajectory-level calibration controls within-group reweighting. At the group level, measures the spectral mass, i.e., how much the rollout representations spread from their center, and measures the effective-rank structure, i.e., how this variation is distributed across spectral directions. Accordingly, SpeCal w/o Spread removes the spectral mass component , and SpeCal w/o ER Gate removes the effective-rank gate over . At the trajectory level, SpeCal w/o Traj. Support removes , disabling conditional spectral support while retaining group-level calibration. Majority Vote RLVR removes all spectral calibration, and SpeCal enables all three components.
As shown in Table 2, every partial SpeCal variant improves the majority-vote baseline, raising the average score from 42.7 to 44.2–44.5. However, removing , , or decreases the full SpeCal performance by 1.1, 1.4, and 1.3 points, respectively, indicating that the three components provide complementary information. Notably, removing spectral mass and removing effective rank produce similar averages, but lead to different task-wise behaviors: the former performs better on AIME25 and Olympiad, whereas the latter is stronger on AMC and Minerva. This contrast suggests that the magnitude and directional organization of group variation capture distinct properties rather than acting as interchangeable measures. Trajectory-level support yields the largest average contribution. Compared with the group-only variant, adding improves AIME25 and AMC by 2.7 and 4.2 points, respectively, showing the benefit of distinguishing trajectories that are supported by the shared spectral structure from those dominated by idiosyncratic directions. Although individual ablations obtain slightly better results on Minerva or Olympiad, none is consistently superior across tasks: the complete SpeCal achieves the best average performance and the best result on four of the six benchmarks. Overall, the results show that spectral mass determines whether sufficient group-level variation exists, effective rank characterizes how this variation is organized, and conditional support further allocates learning emphasis within the group.
| Method | Calibration Components | Benchmarks | ||||||||
| AIME24 | AIME25 | AMC | MATH-500 | Minerva | Olympiad | Avg. | ||||
| Majority Vote RLVR | 17.7 | 12.5 | 54.4 | 81.7 | 43.8 | 46.0 | 42.7 | |||
| SpeCal w/o Spread | 19.2 | 15.7 | 56.0 | 82.0 | 44.3 | 49.7 | 44.5 | |||
| SpeCal w/o ER Gate | 19.2 | 13.8 | 57.6 | 81.6 | 44.5 | 48.7 | 44.2 | |||
| SpeCal w/o Traj. Support | 20.4 | 13.6 | 53.9 | 82.5 | 46.9 | 48.8 | 44.3 | |||
| SpeCal | 20.9 | 16.3 | 58.1 | 83.9 | 44.7 | 49.5 | 45.6 |
Table 2. Ablation study of SpeCal components. denotes the group-level spectral mass component, denotes the normalized effective rank used by the group-level Gaussian gate, and denotes trajectory-level conditional spectral support.
Compatibility with Different Unlabeled Reward Schemes
We investigate whether SpeCal generalizes across different constructions of unlabeled rewards by applying it to Majority Vote, sequence-level entropy (Seq-entropy), token-level entropy (Tok-entropy), and
Self-certainty. As shown in Figure 2, SpeCal improves the ID average under all four schemes, yielding gains of 2.1–3.9 percentage points. The OOD results reveal clearer differences: Majority Vote improves from 67.0 to 67.3, Seq-entropy from 53.9 to 57.4, Self-certainty from 47.5 to 58.6, and Tok-entropy from 36.7 to 56.0. A clear trend emerges from these results: reward schemes with lower original OOD performance obtain larger gains from SpeCal. Because each comparison keeps the underlying reward unchanged and differs only in whether its induced advantages are spectrally calibrated, the improvements demonstrate that SpeCal provides information complementary to the original reward signal. The modest gain on Majority Vote about OOD suggests that there is less room for improvement when the reward already transfers well, whereas the substantially larger gains on Tok-entropy and Self-certainty show that spectral calibration is especially useful when the original supervision transfers poorly. Overall, these results demonstrate that SpeCal is compatible with diverse unlabeled reward schemes and improves their robustness under distribution shift without redesigning the rewards themselves.

Figure 2. Compatibility of SpeCal with different unlabeled reward schemes. Gray bars denote the original schemes, while blue bars denote their SpeCal-enhanced variants.
Sensitivity Analysis
Figure 3 evaluates the sensitivity of SpeCal to the target normalized effective rank , its tolerance , and the trajectory-weight clipping bounds and . Across all four one-dimensional sweeps, the in-domain average ranges from 44.1 to 45.6, remaining at least 1.4 points above the Majority Vote RLVR baseline of 42.7. The default values , , , and achieve the best observed score in their respective sweeps, while several neighboring settings remain competitive. At the group level, performance varies only moderately across , suggesting that useful rollout groups occupy a relatively broad effective-dimensionality regime rather than a sharply localized target. The tolerance has a more visible effect: an overly narrow tolerance assigns substantial weights to only a small range of effective-rank structures, reducing the effective utilization of otherwise informative rollout groups. Conversely, increasing beyond the default gradually weakens the selectivity of the Gaussian gate and makes spectrally distinct groups receive more similar weights. At the trajectory level, the clipping bounds determine the dynamic range available for within-group calibration. Setting too low allows weakly supported trajectories to be strongly suppressed, whereas raising it toward 1 compresses the weight distribution and limits the ability of SpeCal to distinguish trajectories with different support levels. The upper bound exhibits a complementary trade-off: a restrictive value leaves little room to emphasize highly supported trajectories, while an overly large value may concentrate the update on a small subset of rollouts. Although the upper-bound sweep is mildly non-monotonic, the best performance is obtained at the moderate setting , with values around 1.5–2.0 generally outperforming more restrictive or permissive choices. Overall, SpeCal remains effective across a broad range of configurations, while the selected defaults balance group-level selectivity, sample utilization, and sufficient trajectory-level weight contrast.

Figure 3. Hyperparameter sensitivity of SpeCal to the target normalized effective rank , the effective-rank tolerance , and the lower and upper trajectory-weight clipping bounds and . The blue star marks the default setting in each sweep, while the dashed horizontal line denotes the Majority Vote RLVR baseline.
Related Work
Unsupervised RLVR. Unsupervised RLVR derives supervision from model-generated responses without external labels. TTRL [46] constructs pseudo-labels through majority voting, whereas entropy-based methods [1] and Intuitor [44] use intrinsic confidence signals. Co-rewarding stabilizes self-generated supervision through cross-question agreement or a slowly updated teacher [43]. More recent approaches retain richer endpoint information: distribution-aware reward estimation models answer-distribution statistics [8], while RESTRAIN combines vote-frequency weighting, negative advantage offsets, and question-level self-consistency [41]. Nevertheless, because these methods optimize rewards derived from the model’s own outputs, errors in the proxy signals can be repeatedly reinforced during training. Agreement-based signals may stabilize an incorrect answer as the group consensus, producing spurious self-consistency, while confidence-based objectives may continue sharpening a distribution whose confidence is misaligned with correctness. Under prolonged optimization, the policy can exploit these proxy signals without improving task correctness, resulting in reward hacking and eventual training collapse [14, 27]. These findings highlight the fragility of relying exclusively on self-generated endpoint signals, whose reliability estimates remain derived from decoded frequencies, confidence, or teacher agreement rather than rollout representation geometry. SpeCal is complementary to these reward-construction strategies: it leaves the underlying reward unchanged and calibrates the induced advantages through group spectral geometry and rollout-level cross-rollout support.
Spectral Methods. Spectral methods reveal global structure in high-dimensional representations. Spectral Contrastive Learning connects self-supervised objectives to augmentation-graph eigendecomposition [13], singular-spectrum analyses diagnose dimensional collapse [17], and RankMe uses effective rank as a label-free representation-quality measure [10]. For LLM generations, INSIDE detects hallucinations using the covariance spectrum of hidden-state response embeddings [5], Spectral Uncertainty decomposes predictive uncertainty through von Neumann entropy [31], and Generation Space Size uses EigenScore and its leave-one-out variant to characterize generation diversity [40]. These studies establish the value of spectra for representation quality, uncertainty, and diversity, but do not use them to calibrate RLVR updates. To the best of our knowledge, SpeCal is the first to introduce hidden-state spectral calibration into unsupervised RLVR.
Conclusion
In this work, we presented SpeCal, a bi-level spectral calibration framework that improves the reliability of unsupervised RLVR without altering its underlying pseudo-reward. At the group level, SpeCal combines spectral mass with an effective-rank band-pass gate to identify rollout groups exhibiting sufficient and well-organized semantic variation, while at the rollout level, it evaluates each response against an effective-rank-adaptive leave-one-out subspace, thereby emphasizing trajectories supported by spectral directions independently established by the remaining rollouts. Experiments across Qwen- and Llama-based models demonstrate consistent improvements on both in-domain mathematical reasoning and out-of-domain general reasoning benchmarks. The compatibility experiments further show that SpeCal complements multiple unlabeled reward schemes, while the ablation results confirm that spectral spread, effective-rank calibration, and conditional rollout support provide complementary information. These findings suggest that hidden-state geometry offers a useful intermediate-granularity signal between coarse endpoint rewards and costly process supervision. Although the current evaluation focuses primarily on 8B-scale models and requires access to policy hidden states during training, future work could explore efficient spectral approximations, larger model scales, broader reasoning domains, and other forms of self-generated or weak supervision.
References
Detailed Theoretical Analysis
This appendix provides the assumptions and proofs for Propositions 4.1 and 4.2. The first result characterizes the information discarded by endpoint summarization, while the second identifies the marginal geometric quantities estimated by the two calibration levels of SpeCal.
Proof of Endpoint Information Loss
Recall that contains the decoded outcomes and response representations of a rollout group, while is a deterministic endpoint summary. The reliability variable may represent, for example, whether the pseudo-label is correct or whether the induced group-level supervision agrees with an oracle signal.
Proof of Proposition 4.1. Since is a deterministic function of , conditioning on fully determines and therefore
Applying the chain rule for mutual information in two orders gives
Hence,
By the conditional mutual-information identity,
Because is determined by , we have , yielding (13). Non-negativity follows from that of the KL divergence.
The gap is zero if and only if the endpoint summary is sufficient for predicting , namely, , or equivalently, . Conversely, if endpoint-equivalent rollout groups occur with positive probability and two such groups and satisfy but , then the endpoint summary cannot distinguish their reliability, and the information gap is strictly positive.
Proof of Marginal Spectral Consistency
Setup. Fix a question and a policy . Let be i.i.d. normalized response representations drawn from , where . Define the population mean and covariance as
For two independent draws and from the same distribution, the marginal inter-rollout geometry is
The equality follows by expanding the outer product and using . Thus, the marginal inter-rollout geometry coincides with the covariance of the prompt-conditional representation distribution.
Given sampled representations, let , and let stack row-wise. We define the centered empirical spectral matrix as
Empirical and marginal calibration quantities. Let be the eigenvalues of , including zero-valued directions when necessary. When the empirical spectral mass is nonzero, define
where and is defined as zero. The finite-group calibration coefficient is then
This is the empirical quantity denoted by in the main text. For a spectrally collapsed group with zero empirical mass, we set .
Let denote the eigenvalues of . We define the corresponding marginal quantities by
The marginal group-level target is
For rollout-level calibration, let , and let contain the corresponding top eigenvectors of . The marginal projector and conditional support are
For a finite group, denotes the support computed by the same leave-one-out construction as in the main method, using the top available directions.
Assumption 1 (Spectral regularity). For the fixed question and policy , the following conditions hold:
(A1) The representations are i.i.d. and satisfy , with .
(A2) The marginal effective rank is separated from integer boundaries, i.e., , so the retained dimension is locally stable.
(A3) The marginal covariance has a positive eigengap at , namely , and with probability one. The eigengap condition is unnecessary when $k_q^\star = D.
Proof of Proposition 4.2. We first establish the finite-sample unbiasedness of the empirical geometry. The pairwise-scatter identity gives
Each pair has the same distribution as the independent pair , and the sum contains terms. Therefore,
We next consider the large-group limit. By the strong law of large numbers,
It follows that
Because is fixed, the convergence also holds in operator norm. The ordered empirical eigenvalues therefore converge to the corresponding marginal eigenvalues. For all sufficiently large , we also have . Under Assumption 1, spectral mass, entropy effective rank, normalized effective rank, and the Gaussian gate are continuous at the marginal geometry. The continuous-mapping theorem consequently yields
For rollout-level calibration, removing a fixed observation does not change the limiting mean or covariance. Hence, the leave-one-out spectral matrix converges to . Assumption 1 ensures that for all sufficiently large and that the retained eigenspace is uniquely defined. Standard eigenspace perturbation results then imply
Together with the convergence of the leave-one-out mean, applying the continuous-mapping theorem to the normalized quadratic form gives
Equations (27) and (29) complete the proof.
Connection to the Gram matrix used by SpeCal. The method computes , whereas the analysis uses . These matrices share the same nonzero singular spectrum up to the deterministic factor . Therefore, they induce the same normalized eigenspectrum and entropy effective rank, while their spectral masses satisfy
The consistency result thus applies directly to the spectral quantities used in SpeCal.
Scope of the result. Proposition 4.2 distinguishes finite-sample unbiasedness from nonlinear plug-in consistency. The empirical geometry matrix is an unbiased estimator of the marginal inter-rollout geometry, whereas the group weight and conditional support converge to their marginal targets as the rollout group grows. The proposition does not claim that these nonlinear quantities are unbiased at finite , nor that marginal spectral geometry is identical to ground-truth correctness.
Additional Experimental Setups
We limit input prompts to 1,024 tokens and generated responses to 4,096 tokens to accommodate long-form reasoning. For rollout generation, we use the following system prompt: “Let’s think step by step and output the final answer within \boxed{}.” Rollouts are generated using vLLM¹ with responses per prompt, a temperature of 1.0, and top- of 1.0. By default, all models and baselines are trained for a total of 500 global steps.
We employ Math-Verify3 as the sole outcome reward function, deliberately avoiding auxiliary rewards for formatting constraints or response lengths to prevent reward hacking. Finally, rollout generation is accelerated via the vLLM framework3 with 8 rollouts per prompt, using a fixed temperature of 1.0 and a top-p value of 1.0. For hidden-state extraction, we select eight layers evenly spaced across the network depth for every model backbone by default. Additional ablations on this layer-selection strategy are provided in Appendix § 11.§ 1.
Additional Experimental Results
Effect of Representation Construction
To investigate how layer selection affects the hidden-state representation used by SpeCal, we compare four construction strategies under otherwise identical training and evaluation settings. Uniform- evenly selects layers across the model depth, whereas Top-8 uses only the top eight layers. For all variants, the selected layer representations are concatenated, followed by mean pooling over the response tokens. As shown in Figure 4, all four variants outperform the Majority Vote baseline of 42.7 on the ID evaluation. Uniform-8 achieves the best ID average of 45.6, followed closely by Uniform-4 at 45.3, while Uniform-16 and Top-8 both obtain 44.9. At the same layer count, Uniform-8 exceeds Top-8 by 0.7 points on ID and 0.1 points on OOD, suggesting that covering different model depths is preferable to relying exclusively on upper-layer features. The OOD results exhibit a different trend: Uniform-4 achieves the highest average of 69.2, outperforming the OOD baseline by 2.2 points, whereas Uniform-8 and Top-8 provide smaller improvements of 0.3 and 0.2 points, respectively. Increasing the number of uniformly selected layers to 16 reduces the OOD average to 65.9. Overall, these results show that incorporating more hidden states does not monotonically improve performance; a compact selection distributed across the model depth can retain sufficient task information while avoiding potentially redundant representations.
Figure 4. Comparison of different hidden-state representation constructions on ID and OOD evaluations. Uniform- selects layers uniformly across the model, whereas Top-8 selects the top eight layers. All variants otherwise use layer concatenation and response-token mean pooling. The dark hatched bar marks the default Uniform-8 configuration, while the gray dashed line denotes the Majority Vote baseline. Values above the bars report average performance. Panel-specific y-axis ranges with explicit breaks are used for improved readability.
Training Efficiency
We compare the end-to-end training efficiency of different methods by measuring their average wall-clock time per optimization step under the same hardware and training configuration. As shown in Table 3, SpeCal requires 198 seconds per step on Qwen3-8B-Base and 251 seconds on DeepSeek-R1-Distill-Llama-8B. Compared with TTRL, SpeCal introduces only a 3.1% runtime overhead on Qwen3-8B-Base and is numerically 1.2% faster on DeepSeek-R1-Distill-Llama-8B, indicating that the additional spectral calibration incurs limited computational overhead. SpeCal is also substantially more efficient than Co-rewarding, reducing the per-step training time by 19.2% and 41.2% on the two base models, respectively. SCRL remains the fastest method, particularly on Qwen3-8B-Base, where SpeCal requires 76.8% more time; however, this gap decreases to only 2.0% on DeepSeek-R1-Distill-Llama-8B. Overall, although SpeCal is not the fastest method in every setting, it maintains runtime comparable to TTRL and avoids the considerably larger overhead of Co-rewarding across both model families.
| Method | Qwen3-8B-Base | DeepSeek-R1-Distill-Llama-8B |
| TTRL | 192 | 254 |
| SCRL | 112 | 246 |
| Co-rewarding | 245 | 427 |
| SpeCal | 198 | 251 |
Table 3. Average training time per optimization step under two base models. All methods are evaluated using the same hardware and training configuration.
Pseudocode
The complete training procedure of SpeCal is summarized in Algorithm 1.
Algorithm 1: SpeCal
Input: Unlabeled dataset , policy , unlabeled reward scheme , training steps , rollout size , selected layers , calibration parameters
Output: Optimized policy
for step to do
Sample an unlabeled mini-batch ;
foreach do
Generate rollouts ;
Construct unlabeled rewards using and compute the original advantages ;
// Group-Level Spectral Calibration
Extract response representations and construct the centered group spectrum via Eqs. (3)–(4);
Compute the group weight and adaptive spectral bandwidth via Eqs. (5) and (6);
// Rollout-Level Spectral Calibration
foreach in the rollout group do
Construct the effective-rank-truncated leave-one-out subspace via Eqs. (7)–(8);
Compute the rollout weight via Eqs. (9) and (10);
Calibrate the original advantages using and via Eq. (11);
Update with the calibrated GRPO objective in Eq. (12);
return Optimized policy ;
References
- [1]Shivam Agarwal, Zimin Zhang, Lifian Yuan, Jiawei Han, and Hao Peng. 2026. The unreasonable effectiveness of entropy minimization in llm reasoning. Advances in Neural Information Processing Systems 38 (2026), 107150–107180.arxiv.org/abs/2505.15134
- [2]Arash Ahmadian, Chris Cremer, Matthias Gallé, Marzieh Fadaee, Julia Kreutzer, Olivier Pietquin, Ahmet Üstün, and Sara Hooker. 2024. Back to basics: Revisiting REINFORCE-style optimization for learning from human feedback in LLMs. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers). 12248–12267.arxiv.org/abs/2402.14740
- [3]Amos Azaria and Tom Mitchell. 2023. The Internal State of an LLM Knows When It’s Lying. In Findings of the Association for Computational Linguistics: EMNLP 2023. Association for Computational Linguistics, Singapore, 967–976. doi:10.18653/v1/2023.findings-emnlp.68
- [4]Adrien Bardes, Jean Ponce, and Yann LeCun. 2021. Vicreg: Variance-invariance-covariance regularization for self-supervised learning. arXiv preprint arXiv:2105.04906 (2021).arxiv.org/abs/2105.04906
- [5]Chao Chen, Kai Liu, Le Chen, Yi Gu, Yue Wu, Mingyuan Tao, Zhihang Fu, and Jieping Ye. [n. d.]. INSIDE: LLMs’ Internal States Retain the Power of Hallucination Detection. In The Twelfth International Conference on Learning Representations.
- [6]Peter Clark, Isaac Cowhey, Oren Etzioni, Tushar Khot, Ashish Sabharwal, Carissa Schoenick, and Oyvind Tafjord. 2018. Think you have solved question answering? try arc, the ai2 reasoning challenge. arXiv preprint arXiv:1803.05457 (2018).arxiv.org/abs/1803.05457
- [7]Gheorghe Comanici, Eric Bieber, Mike Schaekermann, Ice Pasput, Noveen Sachdeva, Inderjit Dhillon, Marcel Blistein, Ori Ram, Dan Zhang, Evan Rosen, et al. 2025. Gemini 2.5: Pushing the frontier with advanced reasoning, multimodality, long context, and next generation agentic capabilities. arXiv preprint arXiv:2507.06261 (2025).arxiv.org/abs/2507.06261
- [8]Bodong Du, Xuanqi Huang, and Xiaomeng Li. 2026. Distribution-Aware Reward Estimation for Test-Time Reinforcement Learning. arXiv:2601.21804 https://arxiv.org/abs/2601.21804
- [9]Hanyu Duan, Yi Yang, and Kar Yan Tam. 2024. Do LLMs Know about Hallucination? An Empirical Investigation of LLM’s Hidden States. arXiv preprint arXiv:2402.09733 (2024). arXiv:2402.09733 [cs.CL] doi:10.48550/arXiv.2402.09733DOI
- [10]Quentin Garrido, Randall Balestriero, Laurent Najman, and Yann LeCun. 2023. Rankme: Assessing the downstream performance of pretrained self-supervised representations by their rank. In International conference on machine learning. PMLR, 10929–10974.arxiv.org/abs/2210.02885
- [11]Daya Guo, Dejian Yang, Haowei Zhang, Junxiao Song, Peiyi Wang, Qihao Zhu, Runxin Xu, Ruoyu Zhang, Shirong Ma, Xiao Bi, et al. 2025. DeepSeek-R1 incentivizes reasoning in LLMs through reinforcement learning. Nature 645, 8081 (2025), 633–638.DOI
- [12]Daya Guo, Dejian Yang, Haowei Zhang, Junxiao Song, Peiyi Wang, Qihao Zhu, Runxin Xu, Ruoyu Zhang, Shirong Ma, Xiao Bi, et al. 2025. Deepseek-r1: Incentivizing reasoning capability in llms via reinforcement learning. arXiv preprint arXiv:2501.12948 (2025).DOI
- [13]Jeff Z HaoChen, Colin Wei, Adrien Gaidon, and Tengyu Ma. 2021. Provable guarantees for self-supervised deep learning with spectral contrastive loss. Advances in neural information processing systems 34 (2021), 5000–5011.
- [14]Bingxiang He, Yuxin Zou, Zeyuan Liu, Shangziqi Zhao, Zixuan Fu, Junlin Yang, Cheng Qian, Kaiyan Zhang, Yuchen Fan, Ganqu Cui, et al. 2026. How Far Can Unsupervised RLVR Scale LLM Training? arXiv preprint arXiv:2603.08660 (2026).arxiv.org/abs/2603.08660
- [15]Chaoqun He, Renjie Luo, Yuzhuo Bai, Shengding Hu, Zhen Thai, Junhao Shen, Jinyi Hu, Han, Yujie Huang, Yuanxiang Zhang, et al. 2024. Onlymipadbench: A challenging benchmark for promoting agi with olympiad-level bilingual multimodal scientific problems. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers). 3828–3850.
- [16]Dan Hendrycks, Collin Burns, Saurav Kadavath, Akul Arora, Steven Basart, Eric Tang, Dawn Song, and Jacob Steinhardt. 2021. Measuring mathematical problem solving with the math dataset. arXiv preprint arXiv:2103.03874 (2021).
- [17]Li Jing, Pascal Vincent, Yann LeCun, and Younang Tian. 2021. Understanding dimensional collapse in contrastive self-supervised learning. arXiv preprint arXiv:2110.09348 (2021).
- [18]Wouter Kool, Herke van Hoof, and Max Welling. 2019. Buy 4 REINFORCE Samples, Get a Baseline for Free! https://openreview.net/forum?id=r1gTGL5DE
- [19]Aitor Lewkowycz, Anders Andreassen, David Dohan, Ethan Dyer, Henryk Michalewski, Vinay Ramas esh, Ambrose Slone, Cem Anil, Imanol Schlag, Theo Gutman-Solo, et al. 2022. Solving quantitative reasoning problems with language models. Advances in neural information processing systems 35 (2022), 3843–3857.
- [20]Jia Li, Edward Beeching, Lewis Tunstall, Ben Lipkin, Roman Soletsky, Shengyi Huang, Kashif Rasul, Longhui Yu, Albert Q Jiang, Ziju Shen, et al. 2024. Numinamath: The largest public dataset in ai4maths with 860k pairs of competition math problems and solutions. Hugging Face repository 13, 9 (2024), 9.
- [21]Ibrahima J Ndiour, Nilesh A Ahuja, and Omesh Tickoo. 2022. Subspace modeling for fast out-of-distribution and anomaly detection. In 2022 IEEE International Conference on Image Processing (ICIP). IEEE, 3041–3045.
- [22]OpenAI. 2024. OpenAI o1 system card. CoRR abs/2412.16720 (2024). arXiv:2412.16720 doi:10.48550/ARXIV.2412.16720
- [23]Miao Peng, Nuo Chen, Zongrui Suo, and Jia Li. 2025. Rewarding graph reasoning process makes llms more generalized reasoners. In Proceedings of the 31st ACM SIGKDD Conference on Knowledge Discovery and Data Mining V. 2. 2257–2268.
- [24]Yun Qu, Qi Wang, Yixiu Mao, Vincent Tao Hu, Björn Ommer, and Xiangyang Ji. 2026. Can Prompt Difficulty be Online Predicted for Accelerating RL Finetuning of Reasoning Models? In Proceedings of the 32nd ACM SIGKDD Conference on Knowledge Discovery and Data Mining V.1, KDD 2026, Jeju Island, Korea, August 9-13, 2026.arxiv.org/abs/2507.04632
- [25]David Rein, Betty Li Hou, Asa Cooper Stickland, Jackson Petty, Richard Yuanzhe Pang, Julien Dirani, Julian Michael, and Samuel R Bowman. 2023. Gpqa: A graduate-level google-proof q&a benchmark. arXiv preprint arXiv:2311.12022 (2023).arxiv.org/abs/2311.12022
- [26]Olivier Roy and Martin Vetterli. 2007. The effective rank: A measure of effective dimensionality. In 2007 15th European signal processing conference. IEEE, 606–610.
- [27]Sheikh Shafayat, Fahim Tajwar, Ruslan Salakhutdinov, Jeff Schneider, and Andrea Zanette. 2025. Can Large Reasoning Models Self-Train? arXiv preprint arXiv:2505.21444 (2025).
- [28]Zhihang Shao, Peiyi Wang, Qihao Zhu, Runxin Xu, Junxiao Song, Xiao Bi, Haowei Zhang, Mingchuan Zhang, YK Li, Yang Wu, et al. 2024. Deepseekmath: Pushing the limits of mathematical reasoning in open language models. arXiv preprint arXiv:2402.03300 (2024).arxiv.org/abs/2402.03300
- [29]Guangming Sheng, Chi Zhang, Zilingfeng Ye, Xibin Wu, Wang Zhang, Rui Zhang, Yanghua Peng, Haibin Lin, and Chuan Wu. 2025. Hybridflow: A flexible and efficient rlhf framework. In Proceedings of the Twentieth European Conference on Computer Systems. 1279–1297.DOI
- [30]Yi Su, Dian Yu, Lifeng Song, Juntao Li, Haitao Mi, Zhaopeng Tu, Min Zhang, and Dong Yu. 2025. Expanding rl with verifiable rewards across diverse domains. arXiv preprint arXiv:2503.23829 (2025).arxiv.org/abs/2503.23829
- [31]Nassim Walha, Sebastian G Gruber, Thomas Decker, Yinchong Yang, Alireza Jamvanmardi, Eyke Hüllermeier, and Florian Buettner. 2026. Fine-grained uncertainty decomposition in large language models: A spectral approach. In Proceedings of the AAAI Conference on Artificial Intelligence, Vol. 40. 26090–26098.
- [32]Sophie L Wang, Phillip Isola, and Brian Cheung. 2026. The truth lies somewhere in the middle (of the generated tokens). arXiv preprint arXiv:2605.09969 (2026).arxiv.org/abs/2605.09969
- [33]Yubo Wang, Xueguang Ma, Ge Zhang, Yuansheng Ni, Abhranil Chandra, Shengyao Guo, Weiming Ren, Aaran Alrulaj, Xuan He, Ziyan Jiang, et al. 2024. Mmlu-pro: A more robust and challenging multi-task language understanding benchmark. Advances in Neural Information Processing Systems 37 (2024), 95266–95290.arxiv.org/abs/2406.01574
- [34]Yiming Wang, Pei Zhang, Baosong Yang, Derek F. Wong, and Rui Wang. 2025. Latent Space Chain-of-Embedding Enables Output-free LLM Self-Evaluation. In The Thirteenth International Conference on Learning Representations. https://openreview.net/forum?id=jxo70B9fQoarxiv.org/abs/2410.13640
- [35]Shengyu Wang, Hanqin Liu, Shun Zhang, Shengyu Ye, Zhirong Wu, Yang Wang, Zhijian Xu, Xiao Liang, Junjie Li, Ziming Miao, et al. 2025. Reinforcement learning with verifiable rewards implicitly incentivizes correct reasoning in base llms. arXiv preprint arXiv:2506.14245 (2025).
- [36]Dong Yan, Jian Liang, Yanbo Wang, Shuo Lu, Ran He, and Tieniu Tan. 2026. What dissonances lies? selective-complementary reinforcement learning at test time. In Proceedings of the 64th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers). 28957–28970.
- [37]Jianhao Yan, Yafu Li, Zican Hu, Zhi Wang, Ganqu Cui, Xiaoye Qu, Yu Cheng, and Yue Zhang. 2026. Learning to reason under off-policy guidance. Advances in Neural Information Processing Systems 38 (2026), 117157–117186.arxiv.org/abs/2504.14945
- [38]An Yang, Anfeng Li, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chang Gao, Chenghen Huang, Chenxu Lv, et al. 2025. Qwen3 technical report. arXiv preprint arXiv:2505.09388 (2025).arxiv.org/abs/2505.09388
- [39]Shenzhi Yang, Guangcheng Zhu, Xing Zheng, Yingfan MA, Zhongqi Chen, Bowen Song, Weiqiang Wang, Junbo Zhao, Gang Chen, and Haobo Wang. 2025. TraPO: A Semi-Supervised Reinforcement Learning Framework for Boosting LLM Reasoning. arXiv preprint arXiv:2512.13106 (2025).arxiv.org/abs/2512.13106
- [40]Sunny Yu, Ahmad Jabbar, Robert Hawkins, Dan Jurafsky, and Myra Cheng. 2025. Generation Space Size: Understanding and Calibrating Open-Endedness of LLM Generations. arXiv preprint arXiv:2510.12699 (2025).arxiv.org/abs/2510.12699
- [41]Jianjun Guo, Zhaolun Su, Leitian Tao, Haoxhu Wang, Aashu Singh, Hanchao Yu, Jianyu Wang, Hongyang Gao, Weizhe Yuan, Jason E Weston, et al. 2026. RE-STRAIN: From spurious votes to signals—self-training RL with self-penalization. In The Fourteenth International Conference on Learning Representations.
- [42]Biao Zhang and Rico Sennrich. 2019. Root mean square layer normalization. Advances in neural information processing systems 32 (2019).arxiv.org/abs/1910.07467
- [43]Zizhuo Zhang, Jianing Zhu, Xinmu Ge, Zihua Zhao, Zhanke Zhou, Xuan Li, Xiao Feng, Jiangchao Yao, and Bo Han. 2025. Co-reward: Self-supervised reinforcement learning for large language model reasoning via contrastive agreement. arXiv e-prints (2025), arXiv–2508.
- [44]Xuandong Zhao, Zhewei Kang, Aosong Feng, Sergey Levine, and Dawn Song. 2025. Learning to reason without external rewards. arXiv preprint arXiv:2505.19590 (2025).arxiv.org/abs/2505.19590
- [45]Guangcheng Zhu, Shenzhi Yang, Haobo Wang, Xing Zheng, Yingfan MA, Xuening Feng, Zhongqi Chen, Kai Tang, Zhengqing Zang, Bowen Song, et al. 2026. GeoMin: Data-Efficient Semi-Supervised RLVR via Geometric Distribution Modeling. arXiv preprint arXiv:2606.04516 (2026).arxiv.org/abs/2606.04516
- [46]Yuxin Zuo, Kaiyan Zhang, Li Sheng, Shang Qu, Ganqu Cui, Xuekai Zhu, Haozhan Li, Xinwei Long, Ermo Hua, Biqing Qi, et al. 2026. Ttrl: Test-time reinforcement learning. Advances in Neural Information Processing Systems 38 (2026), 131459–131483.arxiv.org/abs/2504.16084

