On the Primacy Bias in RLVR Training

Code: https://github.com/minhphucng30-UND/Primacy-bias-RLVR

Introduction

Large language models (LLMs) fine-tuned for reasoning have recently achieved remarkable gains on complex reasoning benchmarks. A key driver is Reinforcement Learning with Verifiable Reward (RLVR) [10, 20], which iteratively optimizes LLMs using simple, verifiable rewards, thereby incentivizing complex reasoning behaviors such as backtracking and self-verification within the reasoning traces of these models.

Despite this progress, the answer to the question “does RLVR expand toward new reasoning capabilities?” still remains under debate. Some studies have suggested that RL fine-tuning only improves sample efficiency on problems already solvable within the base model, rather than enabling it to solve new problems [41, 23, 36]. On the other hand, some other works [31, 22, 35, 40] have provided evidence that RL-trained models can still induce novel reasoning strategies on out-of-distribution (OOD) domains. These works, however, suggest that the relationship between the base model and the evaluated task—specifically the extent to which the base model can already solve the task problems—may play an important role in explaining whether RLVR leads to reasoning expansion or sharpening.

In this paper, we seek a principled explanation for these seemingly contradictory outcomes through the lens of primacy bias—the tendency to overfit to initial experiences (i.e., the pre-trained knowledge) that can damage the rest of the learning process [17, 2]. This bias not only limits exploration but also impairs the learnability of on-policy data during RL training. It arises because the model becomes increasingly rigid and remains confined near its pre-trained initialization, rather than allowing meaningful adaptation of its internal representations towards new downstream tasks—a behavior commonly referred to as the lazy training regime [24, 8, 24]. To study this hypothesis, we first investigate primacy bias in RLVR using experiments that fully control both pre-training and post-training pipeline (Section 4). We compare coverage dynamics during RL fine-tuning induced by pre-training against random initialization. Our results show that primacy bias leads to coverage shrinkage, regardless of the fine-tuning distribution. Furthermore, we observe that this primacy bias induces sparser parameter updates during training which hinders the network’s ability to learn rare or more challenging problems (i.e., those that the pre-trained or base model struggles to handle). These observations suggest that the RLVR model mostly operates in the lazy regime [8, 15], staying close to its pre-trained initialization and requiring prolonged training to meaningfully adapt to harder examples [27, 18, 24].

We then extend our analysis to LLM reasoning and investigate the existence of lazy training under RLVR (Section 5). Our results show that RLVR-trained models could often be well approximated by the linearization around their pre-trained base model, while exhibiting minimal feature drift. This result suggests that RLVR in this more general setting also operates in the lazy regime, providing a possible explanation for the sparse parameter updates observed during RLVR training, as reported in [26]. We believe that these findings place RLVR within a theoretically tractable regime, which can provide a foundation for more principled, scalable, and analyzable studies of learning dynamics in reasoning post-training [27, 18, 6].

Motivated by our findings, we identify an unlearnable phenomenon that emerges from the interaction between primacy bias and the lazy training regime in RLVR-based LLM training (§ 15). To study this, we first perform fine-tuning across a mixture of easy and hard reasoning problems and observe that mixed training disproportionately improves rewards on easy examples while limiting learning on harder ones. Our results show that although hard problems lead to only modest overall gains during training, they are actually the ones that help most in expanding the model’s reasoning capabilities. At the same time, focused training on hard problems introduces a subtle trade-off that must be carefully taken into account. We find that prolonged training on these difficult examples can also hurt and lead to catastrophic forgetting on the easier problems, likely due to substantial deviations from the initial base model representations. This explains why, as observed in [22], prolonged RLVR training can expand the reasoning boundary in less familiar or out-of-distribution domains, where the pre-trained model initially struggles. In such settings, extended training enables the model to gradually escape the lazy regime and meaningfully adapt to the new target tasks [18, 35, 40], ultimately leading to broader reasoning coverage. In contrast, in domains where the pre-trained model already performs well—such as mathematical or code reasoning, particularly on easier problems—prolonged training may provide limited benefit and can even lead to adverse effects such as coverage shrinkage or catastrophic forgetting. In these cases, the gains obtained from adapting to harder examples may be outweighed by degradation on the easier ones due to substantial deviations from the base model representations and the effects of catastrophic forgetting.

We hope that our study provides a clearer understanding of the conflicting observations surrounding RLVR post-training and motivates more principled approaches for studying and controlling the trade-off between adaptation and forgetting.

In summary, our main contributions are as follows:

• We unveil the primacy bias and lazy learning as the underlying mechanism to reconcile conflicting findings of whether RLVR expands or sharpens the reasoning boundary, where models tend to rely on base model’s features when adapting to post-training data, limiting their ability to acquire new capabilities during RL fine-tuning.

• Through our controlled and large-scale empirical analysis, we demonstrate that RLVR operates in a near-linearized regime around the base model, with the learned representation remaining largely unchanged throughout the RL fine-tuning process. As a result, this leads models to disproportionately improve on easy problems while limiting their ability to effectively learn on hard ones.

• Despite limited overall improvement, learning on hard problems can expand the reasoning boundary, but comes at the cost of forgetting on easy problems. This explains why, on OOD tasks,

where forgetting is minimal, improvements on unsolved problems can increase Pass@$k$, whereas on over-trained domains, severe forgetting masks out improvement on hard problems.

Related Works

RL Generalization of Reasoning LMs. The role of reinforcement learning in driving generalization beyond the base model has been a central topic of study. Early works argue that RL-trained models may not acquire fundamentally new reasoning capabilities beyond those already present in their base models [39, 23, 36]. In particular, [41] demonstrates that RL training can even reduce the number of solvable problems, as measured by the Pass@$k$ metric, suggesting that performance is largely constrained by the base model’s capacity. Additionally, [36] proposes a theoretical framework indicating that RL is unlikely to surpass the inherent limitations of the base model. By contrast, [22] shows that extended RL training can enhance coverage on synthesized reasoning tasks. [35] further shows that RL can exhibit grokking and develop new reasoning strategies on extremely hard tasks. In contrast, we argue that these conflicting findings may arise from primacy bias: early learned features can hinder learning on hard mathematical tasks, while in OOD settings, where such biases are misaligned with target tasks, RL can promote new reasoning strategies.

Primacy Bias in Neural Networks. Similar to biological systems, neural networks also exhibit critical periods—early phases of training during which representations remain highly plastic and are disproportionately shaped by initial data [2]. This phenomenon has later been explored in reinforcement learning, where it is often referred to as primacy bias [17]. Primacy bias occurs when the RL learning process is disrupted by overfitting to earlier experiences. One effect induced by primacy bias is the reduction of update/gradient norm, causing parameters to stop changing and hindering learnability during RL training [1, 17]. Recently, [24] attributed primacy bias to limited feature learning, indicating a model’s confinement to the lazy regime in which it fits the data without substantially updating its representations. This effectively behaves as a linear model, and consequently generalizes poorly. We build on this line by connecting coverage shrinkage to primacy bias, showing that primacy bias also arises in RLVR training, hindering learnability on hard problems.

Neural Tangent Kernel. The infinite-width limit is a well-studied model for deep network optimization [15]. In this regime, models remain close to initialization and behave like linear models governed by the initial Neural Tangent Kernel (NTK), which remains constant during training—known as the lazy training regime [8]. Several studies have investigated the inductive biases in lazy training[27, 6, 5], revealing that the NTKs preferentially fit data using the top eigenmodes of their spectrum, a phenomenon termed spectral bias. While this spectral bias can enable good generalization on some learning problems, target functions $y(x)$ that are not in the top eigenspaces of the kernel (misaligned), which can be measured by centered-kernel alignment (CKA) [9], require a potentially much larger amount of data. In contrast, in a rich (feature-learning) regime, the model can deviate substantially from its initialization, enabling meaningful changes to its internal representations, improving sample efficiency on target functions that are challenging for the lazy regime.[12, 18, 4] Furthermore, [18] characterizes the effect of delayed generalization (grokking) as a phase transition from lazy to rich (feature-learning) learning dynamics. In this paper, we show that RLVR models operate in the lazy regime, where RL-trained models remain close to the pre-trained model, inducing a bias toward solutions near the initial representation. As a result, optimization tends to prioritize improvements on easy problems while making little to no progress on harder ones, leading to a form of delayed generalization.

Preliminaries

Language Model (LM). We consider a language model $\pi_{\theta}$ parameterized by $\theta$, where the models learn to map an input prompt $\mathbf{x}$ to predictions $\mathbf{y} = \{y_1, \cdots, y_L\} \in\mathcal{V}^L$, where $\mathcal{V}$ is the vocabulary size, in an auto-regressive manner:

where $y_k$ is the token at timestep $k$, and $\mathbf{y}_{<k}$ denotes the first $k-1$ tokens of $\mathbf{y}$, and $z_{\theta}(\mathbf{x},\mathbf{y}_{<k}) \in\mathbb{R}^{\nu}$ denotes the logits at the $k$-th token, and $\lvert\mathbf{y}\rvert$ is the response length.

Coverage. Coverage is the primary metric to quantify the reasoning capacity of LLMs. Let $r(x,y) \in\{0,1\}$ be a binary verifiable reward, indicating whether the response $y$ is correct for a given problem $x$. To measure the coverage of a language model $\pi_\theta$, the Pass@$k$ metric has been often used. This metric quantifies the probability that $\pi_\theta$ correctly solves a problem $x$ within $k$ attempts, and is defined as:

where $\operatorname{Pass}@1(x,\pi_\theta) = \mathbb{E}_{y\sim\pi_\theta(\cdot\mid x)}[r(x,y)=1]$. In practice, Pass@$k$ is evaluated over a problem set $\mathcal{D}$ and measures the fraction of problems solved at least once by $\pi_\theta$ within $k$ attempts.

Primacy bias and Lazy learning

We study how pre-training–induced bias affects coverage expansion (or shrinkage) during RL training through the lens of primacy bias [17, 2], a phenomenon in which early learning experiences hinder later learning phases. For example, pre-trained models tend to learn representations dominated by the majority classes, leaving rare classes underrepresented. Thus, RL finetuning is "initially restricted" by these "suboptimal" representations. Furthermore, we’ll show that, under some conditions, the feature updates during RL stage are limited, a phenomenon known as lazy learning [24, 8]. As a result, due to the primacy bias, hard (or rare) examples remain underexplored during the RL phase. We hypothesize that this phenomenon consequently affects the reasoning boundary (coverage) in LLM post-training.

Primacy bias leads to coverage shrinkage

In this section, we study the impact of primacy bias in RLVR in a toy image classification setting where we can control the full pre-training and post-training pipeline, as well as input difficulty. Using a standard image classification task like MNIST [21], we first pre-train a neural network on a labeled dataset and then fine-tune it with GRPO [33], where the reward is 1 if the predicted class matches the ground-truth label and 0 otherwise. In this setting, we control the input’s difficulty based on how often its class appears in the pre-training dataset. We model this effect using Zipf’s law distribution. So, the probability of observing the $i$-th most frequent class is given by:

where $C$ is the number of classes, and $\beta> 0$ is a parameter to control the skewness of the data distribution. As $\beta$ increases, the long tail becomes more pronounced, making the model focus on learning features from dominant classes, which can be suboptimal for low-frequency classes [11, 19] (as shown in Figure 1 (left)). To study the role of data distribution in RLVR, we also construct three fine-tuning datasets: $\mathcal{D}_{\mathrm{easy}}$, $\mathcal{D}_{\mathrm{hard}}$, and $\mathcal{D}_{\mathrm{uniform}}$, corresponding to data distributions that emphasize dominant classes, rare classes, and a uniform distribution over all classes, as shown in Figure 1 (right). Our first observation is that RLVR training from a pre-trained model (using $\beta= 1.2$) leads to coverage shrinkage, whereas RLVR training from a randomly initialized model does not (as shown in Figure 2, left). Notably, this observation holds across different finetuning datasets.

Secondly, when fine-tuning from a pre-trained model, using easy problems leads to a shrinkage in coverage, where the model primarily sharpens its performance on already well-learned classes, similar to observations in post-trained LLMs [41]. On the other hand, training on hard problems can initially improve $\mathrm{Pass}@k$ but eventually degrades with prolonged training. As shown in Figure 2 (right), the loss on easy classes steadily increases over the course of training. This indicates that focusing on difficult examples induces forgetting of easier ones, ultimately leading to a decline in $\mathrm{Pass}@k$.

Additionally, we examine the effect of skewness $\beta$ from pre-training to RL training. We find that larger skewness leads to: (i) Delayed improvements on hard problems, requiring substantially more training steps to reach perfect training accuracy on the hard examples (Figure 3, left); and (ii) Sparser parameter updates, with only a small subset of parameters changing during RL training (Figure 3, right).

Taken together, these findings provide evidence for a form of primacy bias in RLVR, whereby early training experiences hinder the model’s ability to learn rare classes or hard problems. Because the model relies heavily on pre-trained features when adapting to new data, it leads to limited parameter updates during RL, and delayed performance gains on hard examples.

Lazy learning regime

Recently, Lyle et al. [24] have proposed a framework to understand primacy bias in deep learning. In this framework, primacy bias occurs because the network fails to enter a feature-learning regime after early training. Instead, the fine-tuning stage operates in a lazy regime where the network becomes increasingly rigid and does not produce substantial changes in the learned representation. When the data distribution or task changes, e.g., as in post-training, gradient updates are insufficient to significantly modify the previously learned features. As a result, the model continues to rely on representations optimized for the initial data rather than adapting to the new distribution, leading to degraded performance relative to a freshly initialized model.

Based on our observation and the above discussion, we hypothesize that RLVR is operated in the lazy regime. In this regime, the training dynamics stay close to the pre-trained initialization and a neural network $f_\theta(\mathbf{x})$ can be well approximated by its first-order linearization around the initialization:

where $\theta_0$ denotes the pre-trained initialization of the model parameters. Furthermore, the learning dynamics are dictated by the initial neural tangent kernel (NTK) [15]:

The learnability of different problem subsets. To identify which tasks the network can learn well in the lazy regime, several studies have proposed using centered kernel alignment, $\operatorname{CKA}(k_{\mathrm{NTK}},\mathbf{y}\mathbf{y}^{\mathrm{T}})$ [9, 32], as a measure of similarity between the initial NTK and the downstream target labels $\mathbf{y}$:

A high CKA value indicates that the initial kernel is well-aligned with the target task, enabling good generalization with minimal feature learning, and generalizes poorly when the target labels $\mathbf{y}$ are misaligned with the initial NTK. As shown in the third panel of Figure 3, greater skewness leads to lower CKA alignment on $\mathcal{D}_{\mathrm{hard}}$, indicating that delayed generalization can be understood as the misalignment of features. In this transition, the model must adapt its internal representations to improve sample efficiency on functions that are poorly captured under the lazy regime [12, 18, 4].

Fine-Tuning Reasoning Models with RLVR Occurs in the Lazy Regime

The above analysis suggests that primacy bias induced by pre-trained initialization drives the model into a lazy regime, limiting its ability to learn from hard problems. In this section, we investigate whether the lazy regime persists during RLVR fine-tuning in reasoning models. To do so, we evaluate two criteria: (i) whether the linearly approximated model closely matches the behavior of the fine-tuned model, and (ii) whether RLVR training induces significant feature drift from the pre-trained initialization.

RLVR models are well-approximated by linearization

We compare the model and its corresponding linearized policy. Specifically, we linearly approximate the fine-tuned model logits, denoted as $z_{\mathrm{lin}}$, from the pre-trained model logits $z_{\theta_0}$ as follows:

where $z_{\theta_0}$ denotes the logits of the pre-trained policy, to quantify the linearized approximation between the RL and SFT stages, as recent evidence suggests that SFT already operates in a near-lazy regime [25, 30, 3]. We utilize the KL divergence between the fine-tuned models and their corresponding linearized approximations:

We analyze publicly released checkpoints, as shown in Table 1. The suite spans different RL algorithms (GRPO, DAPO) and domains (math, coding, instruction following).

Base Model	fine-tuned (FT) Model	Algorithm	Data	KL ($\pi_\theta\Vert\pi_{\mathrm{lin}}$)	Sparsity Update (%)
Qwen2.5-Math-1.5B	DS-R1-Distill-Qwen-1.5B	SFT	Mixed	12.73	2.8%
DS-R1-Distill-Qwen-1.5B	DeepScaler-1.5B-Preview	GRPO	Math	0.06	53.8%
DS-R1-Distill-Qwen-1.5B	DeepCoder-1.5B-Preview	GRPO	Code	0.125	45.5%
Qwen3-8B-Base	Klear-Reasoner-8B-SFT	SFT	Math+Code	17.946	69.5%
Qwen3-8B-Base	GT-Qwen3-8B-Base	GRPO	Math	0.042	79.9%
Qwen3-4B	Polaris-4B-Preview	DAPO	Math	0.052	79.3%

Results. As shown in Table 1, interestingly, for all fine-tuned RL models, the KL divergence is extremely small, indicating that the linearized approximation closely matches the fully fine-tuned models. In contrast, supervised fine-tuning (SFT) yields much larger KL divergence, reflecting greater nonlinear changes. Moreover, RL-trained models exhibit a substantially higher update sparsity ratio than SFT, suggesting that they remain closer to their base model. We also quantify the per-token log-probability correlation between the linearized approximation and the final fine-tuned models. As shown in Figure 4, the linearized approximation closely matches the RL fine-tuned models, whereas the SFT models show little to no correlation.

RLVR models show minimal feature drifts compared to SFT

Another characteristic of the lazy–rich dichotomy in supervised learning [16, 38] is that the lazy regime yields minimal representational change. To quantify feature similarity between pre-trained and fine-tuned models, we measure subspace rotation via the mean principal angle (MPA) [7, 43] and spectral changes via spectrum drift (NSS). Specifically, given centered activation matrices $A_b^l$ and $A_{\mathrm{ft}}^l$ at layer $l$, we compute MPA and NSS between them.

where $V_{b,k},V_{\mathrm{ft},k}\in\mathbb{R}^{k\times d}$ denotes the top-$k$ right singular vectors of $A_b^l$, $A_{\mathrm{ft}}^l$, and $\sigma_i$ denotes the singular value.

The feature similarities between the pre-trained and their corresponding fine-tuned (SFT or RLVR) models across different layers are shown in Figure 5. Across models, RL fine-tuning yields consistently small principal angles, indicating stronger preservation of the base model’s features. In contrast, SFT exhibits substantially larger deviations from initial features.

To further support that LLM RL fine-tuning operates in the lazy regime, where the Neural Tangent Kernel (NTK) matrix mostly stays constant throughout the training process, we first calculate the empirical NTK (eNTK) on 256 unique samples across MLP layers using Qwen2.5-3B-Instruct, following previous work [25]:

where $f_\theta$ denotes the model predictions, we calculate the relative distance computed before and after fine-tuning:

where $\mathcal{K}^{\mathrm{NTK}}_{\mathrm{base}}$ denotes the eNTK of the base model and $\mathcal{K}^{\mathrm{NTK}}_{\mathrm{FT}}$ denotes the eNTK of the RL fine-tuned model.

As shown in Figure 6, RL remains tightly concentrated near zero with significantly less variance, indicating that the kernel changes only marginally during training. In contrast, SFT exhibits a much broader distribution that deviates substantially from zero, suggesting significantly larger kernel changes. These results suggest that RL fine-tuning operates in a lazy regime, where the fine-tuned model is well-approximated by its linearization, and the learned representations remain largely unchanged throughout fine-tuning.

The =-1 Unlearnable Phenomenon in RLVR Training.

In this section, we show that, due to primacy bias, LLMs primarily improve rewards on already known and easy problems throughout the RLVR training. As a result, they achieve only marginal gains on harder problems. We also observe that, while training on these hard problems yields little improvement, it can improve Pass@k performance; however, prolonged training comes at the expense of reduced Pass@1 due to forgetting on easy problems.

The harder the examples are, the harder it is for RLVR models to learn

Our hypothesis is that, in the lazy regime, RLVR models rely on their initial representations to optimize rewards, which may be insufficient for harder examples and thus lead to degraded learning efficiency.

Setup. We fine-tune the Qwen2.5-3B-Instruct model on the MATH dataset and filter out unsolved problems under the final policy to focus on solvable instances. The remaining data is partitioned into disjoint subsets of increasing difficulty based on pass@1 accuracy, and separate policies are trained on each subset to analyze how learning dynamics vary with difficulty level. Additional experimental details are provided in Appendix § 9. In short, difficulty is closely related to CKA: well-aligned examples are easier to learn, whereas poorly aligned examples remain difficult. This not only supports our analysis above, but also sheds further light on the learning dynamics of RLVR on these examples. For evaluation, we utilize test sets from Olympiad-Bench [13], MNR [37], and MATH-500 [14] datasets.

Remark. Throughout this paper, we define example difficulty by how challenging an example is for the model to solve, as measured by its accuracy. In the controlled toy experiments (Sec. 4), we have full control over the pretraining data distribution, allowing us to manipulate difficulty by increasing distribution skew. In contrast, in the large-scale LLM experiments, where the pretraining data distribution is inaccessible, we cannot directly manipulate difficulty. Instead, we estimate it empirically using the model’s accuracy on the MATH benchmark, with lower accuracy indicating greater difficulty. Despite these differences in how difficulty is manipulated or measured, both settings share the same underlying notion of difficulty.

Training exclusively on hard examples yields little improvement. Our first result is shown in Figure 7. We observe that RLVR yields meaningful improvements only on easy subsets, whereas on harder subsets, it shows little to no improvement, even when training is focused on them. Our results show that during training, most improvements arise from easier examples, while harder examples contribute minimally to overall reward gains. This provides further evidence to the phenomenon we observed earlier in Sec. 4, where rare (hard) problems exhibit delayed improvement and require substantially more training steps due to a lack of feature learning, and prolonged training is needed to adapt their representations.

Training on hard examples improves coverage (Pass@k), while easy examples degrade it. Interestingly, our evaluation results in Table 2 show that despite limited improvement from training in hard subsets, the model still exhibits improved Pass@k, while learning on easy problems exhibits degradation in Pass@k and coverage shrinkage. This suggests that these small gains can still transfer positively to other hard problems.

Table 2: Comparison of Pass@128 performance when learning on hard/easy and full training dataset, where we train for 512 steps.

Model	OlympiadBench	Minerva	MATH-500
Base	68.77 (+0.0)	53.91(+0.0)	94.31 (+0.0)
GRPO (full dataset)	67.87 (-0.9)	48.83 (-5.08)	93.24 (-1.07)
GRPO (Hard)	69.87 (+1.1)	52.98 (-0.93)	94.51 (+0.2)
GRPO (Easy)	66.38 (-2.39)	49.18 (-4.73)	92.47 (-1.84)

Impact of prolonged training. We further examine whether increasing gradient steps improves Pass@k, expecting the model to escape the lazy regime. Figure 8 shows its evolution over post-training steps. In contrast to [28], prolonged training actually degrades performance on the Pass@k metric. We hypothesize that this discrepancy stems from forgetting easier problems. As shown in Figure 9, additional training steps tend to reduce average accuracy, suggesting that RLVR models gradually forget easy examples during training.

Pass@$k$ has been a standard metric for assessing the reasoning capabilities of LLMs. However, its value is inherently dependent on the specific evaluation set on which it is computed. In mathematical reasoning, Pass@$k$ is usually calculated over all test sets, including both already solvable and unsolvable problems. As a result, the decline in performance on already solvable problems might mask the improvement of RL-trained models on hard, unsolvable problems. To quantify this phenomenon, we also compare with Pass@$k$ specifically computed on unsolvable problems (i.e., zero-accuracy problems) under the base model. As shown in Figure 10, training on hard problems can significantly help models solve new problems and expand their coverage (Pass@$k$), in contrast to GRPO on the full dataset. This result further suggests that RLVR models can still acquire new capabilities under the prolonged training, but this gain comes at the cost of forgetting easier problems which might be hurtful in the simple well-trained tasks but mostly helpful if tasks are hard enough and new to models.

Discussion. Our analysis suggests that just exclusively learning on hard problems (without any algorithmic modification) could be sufficient to improve models’ coverage (Pass@$k$). However, prolonged training can also degrade Pass@$k$ due to forgetting solvable/easy problems. This explains why, in mathematical reasoning — where Pass@$k$ is computed over a mixture of easy and hard problems — such forgetting can significantly hurt overall performance and hinder progress on hard problems. In contrast, on OOD and more difficult tasks [22, 35], this forgetting effect can actually be beneficial, since the base model performs poorly with initial representations and improvements on previously unsolved problems dominate the evaluation.

Conclusion

We investigate the ongoing debate over whether RLVR truly expands the reasoning boundary beyond what is acquired during pre-training, through the lens of primacy bias and the lazy training regime. Through extensive, controlled, and large-scale empirical analysis, we confirm that this bias emerges during RLVR training, causing models to remain confined around their pre-trained initialization and preventing meaningful adaptation to hard examples. We then uncover an unlearnable phenomenon as a consequence of primacy bias, where RL favors improvement on easy problems, hindering the ability to learn harder ones. Consequently, attempting to escape the lazy regime can help models learn previously unsolvable problems, but at the cost of forgetting easier ones. This helps explain why, in OOD domains—where forgetting is less harmful—RL can expand the reasoning boundary, in contrast to mathematical reasoning.

Limitations.

This work primarily contributes to the ongoing debate on whether RLVR truly expands reasoning boundaries beyond those acquired during pre-training, attributing these conflicting views to the primacy bias phenomenon. It may help practitioners detect and mitigate the trade-off between adaptability and forgetting in post-training, thereby improving reliability in verifiable domains such as math and code. However, stronger RLVR training may also enhance general reasoning capabilities that could be misused. Moreover, methods relying on automated verifiers are susceptible to reward hacking and over-optimization, while diversity-preserving interventions may increase the likelihood of rare undesirable outputs. Our empirical analysis is limited to finite evaluation sets and task-specific reward signals. Extending our analysis to coding and other verifiable domains would be valuable. However, due to computational constraints, particularly the high cost of sandbox-based program execution for coding RLVR, we leave these extensions for future work.

Additional Experimental Details.

Models. We utilize Qwen2.5-3B-Instruct [29] for RLVR training.

Training configurations. We utilize the Verl framework [34] for RLVR training. We train each model on 4 GPUs with a constant learning rate of $10^{-6}$, the maximum response length is 3072, with a total batch size of 256. For each prompt, we collect 16 rollouts to compute advantages for the GRPO update. We use a sampling temperature $\tau= 1$. We do not apply Reverse KL divergence or entropy loss in our training. We train each model on 4 Nvidia H100 GPUs.

Details constructing difficulty subset of training data. We fine-tune the Qwen2.5-3B-Instruct model [29] on the MATH training dataset $\mathcal{D}$ [14], which consists of 7500 training examples, to identify difficult examples after RL fine-tuning. Full experimental details are provided in Appendix B. After fine-tuning, we discard all problems with zero accuracy under the final policy $\pi_\theta$, retaining only solvable instances. We then partition $\mathcal{D}$ into $N$ disjoint subsets $\{\mathcal{D}_k\}_{k=1}^{N}$ according to their estimated accuracy under $\pi_\theta$, ordered by increasing difficulty:

where $\{\tau_k\}_{k=1}^{N}$ is an increasing sequence of thresholds defining the difficulty levels. We then fine-tune a separate policy on each subset $\mathcal{D}_k$ to study how learning dynamics vary with problem difficulty. To quantify improvements, we define the accuracy improvement metric before and after fine-tuning on these subsets:

To estimate the average accuracy of the RL-trained policy, we sample $G = 64$ responses per problem with a temperature of 1.0. Finally, we construct $N$ disjoint subsets, where each subset contains problems whose scores fall within the interval $[\tau_k, \tau_{k+1}]$, for $k \in[N]$.

Gradient norm in RL-training.

In this section, we investigate the gradient norm when learning under easy/hard problems. We argue that the gradient is dominated by the easy problems, and, as a result is biased towards making progress on easy problems at the expense of hard problems. Figure 11 demonstrates our results. Harder examples tend to have smaller gradient norms, suggesting that RLVR makes faster progress on easy problems compared to harder ones.

Additional experiments on Llama-3.2-3B-Instruct.

To strengthen the empirical evidence, we conducted additional experiments on Llama-3.2-3B-Instruct, using the same training setup as in our Qwen2.5-3B experiments. Specifically, we compare training on the full dataset against training exclusively on hard examples.

We provide our results in Figure 12, showing that training exclusively on hard examples leads to substantially slower optimization. Training on the full dataset rapidly improves training accuracy, Furthermore, Table 3 shows that, on the new model backbone, hard-only training consistently achieves higher Pass@128 on problems unsolvable by the base model and maintains better Pass@$k$ performance than GRPO training on the full dataset, consistent with our earlier observations on Qwen2.5-3B-Instruct.

Model	OlympiadBench	Minerva	MATH-500
Base	61.01	51.37	91.30
GRPO (full dataset)	50.36	51.04	83.4
GRPO (Hard)	57.8	51.61	89.1

NeurIPS Paper Checklist

1. Claims

Question: Do the main claims made in the abstract and introduction accurately reflect the paper’s contributions and scope?

Answer: [Yes]

Justification: We uncover the primacy bias and lazy training regime in Large Language Models (Sections 4 and 5), we further uncover the unlearable phenomenon during RLVR training, addressing the conflicting debate on whether RL can truly expands reasoning boundary (§ 15).

Guidelines:

• The answer [N/A] means that the abstract and introduction do not include the claims made in the paper.

• The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A [No] or [N/A] answer to this question will not be perceived well by the reviewers.

• The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.

• It is fine to include aspirational goals as motivation as long as it is clear that these goals are not attained by the paper.

2. Limitations

Question: Does the paper discuss the limitations of the work performed by the authors?

Answer: [Yes]

Justification: We discuss the limitations of the work in Appendix § 8

Guidelines:

• The answer [N/A] means that the paper has no limitation while the answer [No] means that the paper has limitations, but those are not discussed in the paper.

• The authors are encouraged to create a separate “Limitations” section in their paper.

• The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.

• The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with a few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.

• The authors should reflect on the factors that influence the performance of the approach. For example, a facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or a speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.

• The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.

• If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.

• While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren’t acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an important role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.

3. Theory assumptions and proofs

Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?

Answer: [Yes]

Justification: We present our lazy regime and primacy bias assumption in § 4.2.

Guidelines:

• The answer [N/A] means that the paper does not include theoretical results.

• All the theorems, formulas, and proofs in the paper should be numbered and cross-referenced.

• All assumptions should be clearly stated or referenced in the statement of any theorems.

• The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch to provide intuition.

• Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.

• Theorems and Lemmas that the proof relies upon should be properly referenced.

Experimental result reproducibility

Question: Does the paper fully disclose all the information needed to reproduce the main experimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?

Answer: [Yes]

Justification: All implementation details are in B.

Guidelines:

• The answer [N/A] means that the paper does not include experiments.

• If the paper includes experiments, a [No] answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.

• If the contribution is a dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.

• Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.

• While NeurIPS does not require releasing code, the conference does require all submissions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example (a) If the contribution is primarily a new algorithm, the paper should make it clear how to reproduce that algorithm. (b) If the contribution is primarily a new model architecture, the paper should describe the architecture clearly and fully. (c) If the contribution is a new model (e.g., a large language model), then there should either be a way to access this model for reproducing the results or a way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset). (d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.

Open access to data and code

Question: Does the paper provide open access to the data and code, with sufficient instructions to faithfully reproduce the main experimental results, as described in supplemental material?

Answer: [Yes]

Justification: The data and models are publicly available, and the code is available at https://anonymous.4open.science/r/neurips26_sub-ODE8/.

Guidelines:

• The answer [N/A] means that paper does not include experiments requiring code.

• Please see the NeurIPS code and data submission guidelines (https://neurips.cc/public/guides/CodeSubmissionPolicy) for more details.

• While we encourage the release of code and data, we understand that this might not be possible, so [No] is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for a new open-source benchmark).

• The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines (https://neurips.cc/public/guides/CodeSubmissionPolicy) for more details.

• The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.

• The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only a subset of experiments are reproducible, they should state which ones are omitted from the script and why.

• At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).

• Providing as much information as possible in supplemental material (appended to the paper) is recommended, but including URLs to data and code is permitted.

Experimental setting/details

Question: Does the paper specify all the training and test details (e.g., data splits, hyperparameters, how they were chosen, type of optimizer) necessary to understand the results?

Answer: [Yes]

Justification: All implementation details are in the Appendix B.

Guidelines:

• The answer [N/A] means that the paper does not include experiments.

• The experimental setting should be presented in the core of the paper to a level of detail that is necessary to appreciate the results and make sense of them.

• The full details can be provided either with the code, in appendix, or as supplemental material.

Experiment statistical significance

Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?

Answer: [No]

Justification: Training Large Language models is time-consuming and compute-intensive. Our experiments do not run multiple seeds on one configuration due to limited computing and financial budget. Our evaluation protocol is similar to prior influential works in LLM Reasoning [42, 35, 40].

Guidelines:

• The answer [N/A] means that the paper does not include experiments.

• The authors should answer [Yes] if the results are accompanied by error bars, confidence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.

• The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).

• The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)

• The assumptions made should be given (e.g., Normally distributed errors).

• It should be clear whether the error bar is the standard deviation or the standard error of the mean.

• It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have a 96% CI, if the hypothesis of Normality of errors is not verified.

• For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out of range (e.g., negative error rates).

• If error bars are reported in tables or plots, the authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.

Experiments compute resources

Question: For each experiment, does the paper provide sufficient information on the computer resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?

Answer: [Yes]

Justification: They are mentioned in Appendix B.

Guidelines:

• The answer [N/A] means that the paper does not include experiments.

• The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.

• The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.

• The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn’t make it into the paper).

Code of ethics

Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https://neurips.cc/public/EthicsGuidelines?

Answer: [Yes]

Justification: This work solely focuses on understanding the ongoing debate of coverage expansion/shrinkage problem in RLVR training. No human subject or private data is involved in this work, and no experiment is conducted for harmful purposes.

Guidelines:

• The answer [N/A] means that the authors have not reviewed the NeurIPS Code of Ethics.

• If the authors answer [No], they should explain the special circumstances that require a deviation from the Code of Ethics.

• The authors should make sure to preserve anonymity (e.g., if there is a special consideration due to laws or regulations in their jurisdiction).

Broader impacts

Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?

Answer: [Yes]

Justification: We mentioned in the Appendix A.

Guidelines:

• The answer [N/A] means that there is no societal impact of the work performed.

• If the authors answer [N/A] or [No], they should explain why their work has no societal impact or why the paper does not address societal impact.

• Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.

• The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate Deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.

• The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.

• If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML).

Safeguards

Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pre-trained language models, image generators, or scraped datasets)?

Answer: [No]

Justification: This work solely focuses on understanding the ongoing debate between the coverage expansion/shrinkage problem in RLVR training. No new data or pretrained model is released in this work.

Guidelines:

• The answer [N/A] means that the paper poses no such risks.

• Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.

• Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.

• We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make a best faith effort.

Licenses for existing assets

Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?

Answer: [Yes]

Justification: All data used in this work are publicly available for research purposes and are cited accordingly.

Guidelines:

• The answer [N/A] means that the paper does not use existing assets.

• The authors should cite the original paper that produced the code package or dataset.

• The authors should state which version of the asset is used and, if possible, include a URL.

• The name of the license (e.g., CC-BY 4.0) should be included for each asset.

• For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.

• If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswithcode.com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of a dataset.

• For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.

• If this information is not available online, the authors are encouraged to reach out to the asset’s creators.

New assets

Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?

Answer: [N/A]

Justification: This work solely focuses on understanding the ongoing debate between the coverage expansion/shrinkage problem in RLVR training. No new data or pretrained model is released in this work.

Guidelines:

• The answer [N/A] means that the paper does not release new assets.

• Researchers should communicate the details of the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc.

• The paper should discuss whether and how consent was obtained from people whose asset is used.

• At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.

Crowdsourcing and research with human subjects

Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?

Answer: [N/A]

Justification: This work solely focuses on understanding the ongoing debate between the coverage expansion/shrinkage problem in RLVR training. No human subject is involved in this work.

Guidelines:

• The answer [N/A] means that the paper does not involve crowdsourcing nor research with human subjects.

• Including this information in the supplemental material is fine, but if the main contribution of the paper involves human subjects, then as much detail as possible should be included in the main paper.

• According to the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.

Institutional review board (IRB) approvals or equivalent for research with human subjects

Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?

Answer: [N/A]

Justification: This work solely focuses on understanding the ongoing debate between the coverage expansion/shrinkage problem in RLVR training. No human subject is involved in this work.

Guidelines:

• The answer [N/A] means that the paper does not involve crowdsourcing nor research with human subjects.

• Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.

• We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.

• For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.

Declaration of LLM usage

Question: Does the paper describe the usage of LLMs if it is an important, original, or non-standard component of the core methods in this research? Note that if the LLM is used only for writing, editing, or formatting purposes and does not impact the core methodology, scientific rigor, or originality of the research, declaration is not required.

Answer: [N/A]

Justification: We solely use LLM to help with writing, all core methods and experiments 1170 were proposed without LLM involvement.

Guidelines:

• The answer [N/A] means that the core method development in this research does not involve LLMs as any important, original, or non-standard components.

• Please refer to our LLM policy in the NeurIPS handbook for what should or should not be described.