WorksComputer Science
RLVR Sharpens the Pencil, Not the Mind
Abstract
Models like DeepSeek-R1 made Reinforcement Learning with Verifiable Rewards (RLVR) famous: reward a model for correct answers and it gets dramatically better at reasoning without being taught how. The leading explanation is sharpening—that the base model already knows how to solve these problems and RLVR just makes it more likely to reach for its best solution paths. This has only ever been checked by comparing before and after training, never by watching what changes inside across it. We tracked OLMo-2-1B across all 13 training checkpoints on GSM8K. First-try accuracy climbs from 58.0% to 61.3%, but ask the model 256 times and solution coverage actually rises from 94.5% to 96.9%, proving the knowledge was already there. We then planted reasoning mistakes into problems and trained simple probes to detect them. Those probes held between AUROC 0.798 and 0.840 across all 13 checkpoints and 3 error types, a spread smaller than the confidence interval; the model never builds a sharper internal picture of what a reasoning mistake looks like. The one place we find real internal change is at the final answer token, where a probe distinguishing early from late checkpoints hits an AUROC of 0.93, while the same probe on the actual reasoning steps lands near chance. RLVR makes the model lead with its best thinking. It does not change what that thinking is.
Introduction
Models like DeepSeek-R1 and OpenAI o1 are trained by rewarding correct answers with no supervision over how they get there, and show large gains on math and reasoning benchmarks ([4, 9]). The field has largely settled on an explanation: sharpening. The base model already knows how to solve these problems; RLVR just makes it more likely to reach for its best solution paths ([14, 6]). This is supported both by behavioral evidence, showing that solutions are reachable from the base model given enough attempts ([14]), and mechanistic evidence, showing that reasoning directions pre-exist in base model activations before any RLVR training ([10]). But the behavioral work tracks outputs across training checkpoints while the mechanistic work only compares a base model to a trained model at a single endpoint. Nobody has asked whether internal representations of reasoning actually change as training progresses.
We track OLMo-2-1B across all 13 RLVR checkpoints on grade-school arithmetic, measuring both outputs and internal states at each step. Unlike the boundary-narrowing Yue et al. find in larger models, pass@256 does not shrink here, giving us a clean window into whether representational change accompanies behavioral change. It does not. Internal representations of reasoning errors hold flat across all 13 checkpoints, while the only detectable change concentrates at the final answer token rather than at the reasoning steps where errors occur. We report a single-model, single-task (arithmetic) result and scope claims accordingly.
Related Work
Behavioral evidence for sharpening. Yue et al. track pass@ up to across training checkpoints and find that RLVR improves pass@1 while narrowing the set of problems the model can eventually solve, concluding that the capability was already there in the base model [14]. Karan and Du reach the same conclusion without any training at all, recovering RLVR-level performance through likelihood-guided resampling from the base model [6]. Dang et al. offer a mode-collapse account of why the boundary narrows [3]. These works map how outputs change across training; what they leave open is whether internal representations shift in parallel.
Mechanistic evidence for sharpening. Venhoff et al. use sparse autoencoders and steering vectors to show that reasoning directions already exist in base model activations, with post-training teaching the model when to use them rather than creating them [10]. This is an endpoint comparison between a base model and a trained model. We ask the same question but across all training checkpoints, and separate it into two parts: do outputs change, and do internal representations of reasoning change.
What changes during RLVR training. Recent work finds that RLVR drives near-linear drift in weights and outputs [11] and that distributional shifts are sparse, close to zero at most token positions and large at a few [7]. Yan et al. show that representational change, where it exists, can be localized causally [12]. We test whether arithmetic error representations fall in the invariant majority or the shifting minority.
Error representations in activations. Linear probes can detect truthfulness and error states in model activations [1], but such signals can be diagnostic rather than causal [13] and can reflect surface features rather than genuine reasoning [5]. We control for this directly in § 4.2.
Setup
Model. OLMo-2-0425-1B-Instruct [8]; 16 layers (); 13 checkpoints at steps 200–2600 (interval 200).
Task. GSM8K [2] test split; 1319 problems; step-by-step generation; final numeric answer parsed.
We measure how well the model solves problems at each checkpoint and whether its internal states encode reasoning errors differently as training progresses, via two procedures (full pseudocode in Appendix § 7): a behavioral sweep that samples completions per checkpoint to compute pass@ at both (wide, all checkpoints) and (deep, 5-checkpoint subset); and a representational probe that injects one of three controlled arithmetic errors into correctly solved problems and trains logistic-regression probes on PCA-reduced hidden states at six readout sites to test whether checkpoints differ in how those errors are encoded.
Code. All code, per-checkpoint probe outputs, and figure-generation scripts are released at https://github.com/Leviathan2006/rlvr-pencil.
Results
Behavior Becomes More Precise Without Losing Breadth
pass@1 rises from 0.580 to 0.613 across training. The apparent drop in pass@32 is an artifact of small : pass@256 actually rises from 0.945 to 0.969 (Figure 1), so the set of problems the model can eventually solve grows, not shrinks. This differs from the boundary-narrowing Yue et al. and Dang et al. find in larger models [14, 3], and gives us a clean window into whether internal representations shift alongside behavior.

Figure 1. First-try accuracy and deep coverage across training. (a) pass@k curves from the deep sweep. (b) pass@1 rises by +3.3 pts while pass@256 does not narrow; the model’s solution coverage stays intact across training.
Internal Error Representations Remain Unchanged
For the operand error, AUROC spans 0.798–0.840 across all 13 checkpoints—a spread of 0.042, smaller than the CI half-width of . The other two types hold the same pattern: operator-swap (spread 0.032, ) and result-scale (spread 0.033, ); Figure 2(a). Drift is statistically indistinguishable from zero across all three.

Figure 2. (a) Error-detection AUROC at the dependent-step token (layer 11) across the RLVR trajectory; shaded band is the bootstrap 95 % CI for the +3-operand condition. (b) Checkpoint-identity AUROC by readout site; the L0 baseline (×) is near chance throughout.
The signal is computed, not surface. Probes can pick up shallow cues like token length or activation norm rather than anything about the reasoning itself [5]. A norm proxy rules this out here, sitting at chance throughout (0.50–0.52). The probe signal rises from chance at layer 0 (AUROC 0.498) to a mid-layer peak of 0.833, with near-identical layer profiles across all 13 checkpoints. The error information the model encodes is a mid-layer computation, not a surface cue.
Positive control. Using the identical pipeline, we probe whether clean activations can distinguish early from late checkpoints. At the output token, the probe reaches AUROC 0.93 for both operator-swap and result-scale, with the layer-0 baseline at chance. At the intermediate reasoning tokens, it lands near chance (Figure 2(b)). The probe finds real training-induced change at the output and finds none at the reasoning steps. The flat trajectory on error representations is meaningful precisely because the method demonstrably detects change where change exists.
The Invariance Is Robust, Not an Artifact of Analysis Choices
The result holds across both pre-registered choices. Probing all 17 layers, the cross-checkpoint spread stays within noise at 16 of 17; the per-checkpoint layer profiles are near-identical in shape across training. Probing all six readout sites, every site is invariant (spread ). Full tables and figures are in Sections C and D.
Training Alters the Final Output, Not the Intermediate Reasoning
Bringing the two findings together: error-detection decodability is flat across RLVR training (constant, three error types, and robust to layer and site choice; § 4.3), while first-try accuracy improves and RLVR’s representational footprint is large at the output token. In this setting, RLVR acts on the output distribution rather than on the internal arithmetic reasoning-error representation. A practical consequence is that probes or verifiers trained to detect reasoning errors in the base model should transfer across RLVR checkpoints, amortizing verification compute.
Computational Budget and Reproducibility
All experiments are inference-only: we evaluate publicly released OLMo-2-1B checkpoints ([8]) without updating any weights, using the standard -FLOPs-per-token approximation (, prompt ≈150 tokens, generated chain ≈180 tokens). Total cost across all reported experiments—behavioral pass@ sweeps, representational probing across three error types, and the layer- and site-robustness ablations—is ≈ FLOPs, roughly below the -FLOP single-run training cap; the behavioral sweeps dominate this total since they generate up to 256 samples per problem. We release the injection-and-probing pipeline, per-checkpoint activation tensors and probe outputs (AUROC values with bootstrap CIs) for every layer, site, and error type, and the pass@ generation logs; the only external dependencies are the public OLMo-2-1B checkpoints and the GSM8K test split ([2]).
Conclusion
We establish that RLVR drives behavioral improvements by altering the final output distribution rather than reshaping internal arithmetic reasoning-error representations. A detailed discussion of our findings, along with the study’s limitations and broader impacts, is provided in Appendix § 8.
Detailed Procedures
The two procedures summarized in § 4.2 are given here in full.
Algorithm 1: Behavioral Evaluation
Input: Checkpoints , problems , temperature
for each checkpoint do
Sample completions per problem at temperature ;
Compute pass@ via unbiased estimator [2];
// Wide sweep: , , all 13 checkpoints
// Deep sweep: , , 5-checkpoint subset
Algorithm 2: Representational Probe
Input: Correctly solved problems , error types : operand (), operator-swap, result-scale
for each , each do
Inject into an early reasoning step; regenerate remainder;
Label chain REPAIR if answer correct, CARRY if answer flips;
Extract hidden states at 6 readout sites: corrupted step, dependent step, answer token;
for each readout site, each layer do
Fit PCA-256 on concatenated clean corrupt activations;
Train logistic regression; evaluate via 5-fold CV AUROC with bootstrap confidence intervals;
Pre-registered site: layer 11, token L_dep_num;
Discussion and Limitations
These results carry clear implications, and understanding their scope is important.
(i) Complementary levels of analysis. pass@ captures whole-problem solution-finding. The representational metric is narrower: at one specific step, how do activations encode a single injected arithmetic error? Behavioral evidence shows the output distribution tightening; representational evidence shows the substrate encoding arithmetic errors is not what changes. The two scales divide the question rather than compete over it—and their combination localises where RLVR acts. (ii) Depth through focus. Arithmetic on OLMo-2-1B earns its specificity. Arithmetic gives unambiguous ground truth for what counts as a reasoning error. Thirteen checkpoints provide trajectory-level resolution across training. The constrained domain keeps representational changes isolable—task variation would bury the within-training dynamics we are trying to characterise. Scaling to diverse tasks and architectures trades that granularity for coverage, and the signal goes with it.
(iii) A clean window into RLVR. pass@1 rises 3.3 points; coverage does not narrow. That modesty is useful. Without severe mode collapse, the main confound drops away—so the question becomes clean: does representational change track behavioural change even when the behavioural shift is real but small? Here, it does not. A noisier setting would have buried that.
(iv) A localized, not merely correlational, null. Linear probes establish that information is present in activations but not that the model uses it causally. We take this concern seriously. Our positive control, however, directly addresses it: the identical probing pipeline successfully and precisely detects where RLVR does leave a representational footprint—at the output token, with AUROC 0.93, rising monotonically along the reasoning chain. The method demonstrably finds change where change exists. Where it returns a flat trajectory, that null is meaningful.
The trajectory-level design fills a concrete gap left by representational endpoint comparisons [10], and situates one interpretable arithmetic-error representation within the invariant majority that recent sparse-shift work predicts [7].
Layer-Sweep Invariance
Motivation. The main analysis pre-registers layer 11 as the reporting layer (median of best layers across dry runs). A reviewer might ask whether the invariance result is specific to that choice—whether other layers show drift that layer 11 happens to miss. We address this by probing all 17 layers (embedding through layer 16) at five spanning checkpoints (steps 200, 800, 1400, 2000, 2600) using the same injection pipeline and identical probe configuration (, PCA-256, 5-fold CV AUROC, bootstrap 95% CIs).
Note on absolute AUROC. These ablation runs used a chain-collection target () rather than the main analysis’s fixed-attempt limit (450 problems), yielding vs. in the main analysis. The larger probe training set raises absolute AUROC by approximately 0.10 relative to the main figures. The spread across checkpoints—our invariance measure—is unaffected by this, and is in fact tighter here (mean spread 0.022 vs. 0.042 in the main analysis), providing a stronger invariance signal.
Results. Figure 3 shows the AUROC profile across all 17 layers for each of the five checkpoints. Three observations stand out.

Figure 3. Error-detection AUROC at every layer of OLMo-2-1B, for five RLVR checkpoints. The five profiles are near-identical in shape: chance at layer 0, rising to a peak around layer 6, declining in late layers. The vertical dotted line marks the pre-registered reporting layer (L11). Near-identical profiles across all five checkpoints confirm that the computational structure of the error representation—not just its endpoint decodability—is invariant across RLVR training. Note: absolute AUROC is higher than in the main analysis owing to a larger chain sample ( vs. ); see text.
First, the layer-0 AUROC is at chance (0.500–0.540 across all checkpoints, mean 0.517), confirming the signal is a computed mid-layer representation, not a surface artefact from token identity—consistent with the norm-proxy control in the main paper.
Second, the five per-checkpoint profiles are near-identical in shape: all rise steeply from layer 0, peak around layer 6 (mean AUROC 0.983), and decline through the final layers. This shape-stability across training is stronger evidence for representational invariance than a single-layer AUROC comparison, since it shows the entire computational trajectory of the representation is preserved.
Third, the cross-checkpoint spread is within noise at 16 of 17 layers. At the peak (layer 6) the spread is only 0.008, a fraction of the CI half-width (0.009). The one marginal exception is layer 13 (spread 0.073, CI half-width 0.031), a late layer where absolute AUROC is already declining (mean 0.864) and signal is weaker. No systematic drift direction is visible there across checkpoints. Table 1 reports the spread and CI half-width for every layer.
Table 1: Cross-checkpoint spread and mean CI half-width at each layer. Spread CI half-width indicates invariance within noise; 16 of 17 layers satisfy this. Layer 13 is the marginal exception (spread 0.073, half-width 0.031) in a late layer where absolute signal is already declining.
| Layer | Mean AUROC | Spread | CI half-width | Invariant |
| 0 | 0.517 | 0.039 | 0.050 | ✓ |
| 1 | 0.868 | 0.020 | 0.031 | ✓ |
| 2 | 0.952 | 0.007 | 0.019 | ✓ |
| 3 | 0.949 | 0.014 | 0.018 | ✓ |
| 4 | 0.966 | 0.014 | 0.013 | ✓ |
| 5 | 0.971 | 0.009 | 0.013 | ✓ |
| 6 | 0.983 | 0.008 | 0.009 | ✓ |
| 7 | 0.971 | 0.008 | 0.013 | ✓ |
| 8 | 0.975 | 0.009 | 0.011 | ✓ |
| 9 | 0.957 | 0.016 | 0.015 | ✓ |
| 10 | 0.959 | 0.010 | 0.015 | ✓ |
| 11 | 0.933 | 0.034 | 0.020 | ✓ |
| 12 | 0.922 | 0.033 | 0.023 | ✓ |
| 13 | 0.864 | 0.073 | 0.031 | × |
| 14 | 0.849 | 0.058 | 0.035 | ✓ |
| 15 | 0.826 | 0.048 | 0.036 | ✓ |
| 16 | 0.811 | 0.054 | 0.036 | ✓ |
Table 1.
Readout-Site Invariance
Motivation. The main analysis pre-registers one readout site (L_dep_num: the number token in the step that depends on the corrupted step). A reviewer might ask whether the invariance result is specific to that site, and whether other readout positions—especially those closer to the injection point—show drift that the pre-registered site misses. We probe all six readout sites at layer 11 across the same five checkpoints.
Results. Table 2 and Figure 4 report the results. Every site is invariant: spreads range from 0.000 (L_inj_step) to 0.034 (L_dep_num), all within the mean CI half-width. This confirms that the pre-registered site is not a cherry-picked location where the invariance result happens to appear; it holds across the entire chain, from the injection point to the final answer token.
| Site | Mean AUROC | Spread | CI half-width | Invariant |
| L_inj_num | 0.998 | 0.002 | 0.002 | ✓ |
| L_inj_step | 1.000 | 0.000 | 0.000 | ✓ |
| L_dep_num | 0.933 | 0.034 | 0.020 | ✓ |
| L_dep_last | 0.998 | 0.003 | 0.003 | ✓ |
| L_dep_mean | 0.995 | 0.006 | 0.005 | ✓ |
| L_answer | 0.960 | 0.021 | 0.014 | ✓ |
Table 2. Readout-site invariance check. All six sites show cross-checkpoint spread within noise at layer 11. The spatial gradient (highest near injection, attenuating toward the answer) is preserved across checkpoints.

Figure 4. Error-detection AUROC at each of six readout sites (layer 11) for five RLVR checkpoints. Bar groups at each site show near-identical heights across checkpoints (spread ≤ 0.034 at every site, all within noise). The spatial gradient—highest at the injection site, attenuating toward the answer—is preserved across training, showing that both the error signal and its downstream propagation pattern are invariant to RLVR. Note: absolute AUROC is higher than in the main analysis owing to a larger chain sample; see Appendix § 9.
The absolute AUROC pattern replicates the spatial gradient seen in the main paper: decodability is highest at the injection site (≈ 1.000, where the corrupted number is present in the text) and attenuates downstream. This attenuation is also invariant—the gradient shape is preserved across checkpoints—indicating that not just the error signal but its propagation pattern is stable across RLVR training.
Robustness to Evaluation Protocol
We report pass@1 at temperature 0.8 under a zero-shot chat prompt, parsing the final number after “The answer is”. Because absolute GSM8K accuracy is sensitive to the evaluation protocol, we verify that our central object—the trajectory of accuracy across checkpoints—does not depend on these choices. We re-evaluate all five spanning checkpoints under a grid crossing decoding (greedy vs. sampled at ) with parser (strict “answer is” vs. a lenient last-number parser), on a fixed 200-problem subset of the test split.
Table 3 reports the results. Three points follow. First, the parser is immaterial: strict and lenient agree to within 0.002 at every checkpoint. Second, decoding sets … level: greedy decoding runs points above sampled pass@1 (mean 0.646 vs. 0.615), since sampling occasionally selects a sub-optimal token. This accounts for the difference between our reported figures and higher numbers obtained under greedy, full-test-set evaluation; the residual is attributable to our zero-shot prompt and evaluation subset. Third, and most important for our claims, the +3.3-point upward trend across training is present under both decoding schemes—the trajectory we study is a property of the checkpoints, not of the decoding protocol, so every trajectory-level conclusion in the paper is unaffected by this choice.
| Checkpoint | Greedy/strict | Greedy/lenient | Sampled/strict | Sampled/lenient |
| step_200 | 0.625 | 0.625 | 0.595 | 0.590 |
| step_800 | 0.645 | 0.645 | 0.610 | 0.610 |
| step_1400 | 0.655 | 0.650 | 0.580 | 0.580 |
| step_2000 | 0.660 | 0.655 | 0.640 | 0.640 |
| step_2600 | 0.645 | 0.645 | 0.650 | 0.645 |
| mean | 0.646 | 0.644 | 0.615 | 0.613 |
Table 3. GSM8K accuracy across five checkpoints under a decoding parser grid (200-problem subset). Parser choice is immaterial (); decoding sets the absolute level ( pts); the upward trend across training holds under both decoding schemes.
References
- [1]Azaria, A. and Mitchell, T. The internal state of an LLM knows when it’s lying. In Findings of the Association for Computational Linguistics: EMNLP 2023, pp. 967–976, 2023.arxiv.org/abs/2304.13734
- [2]Cobbe, K., Kosaraju, V., Bavarian, M., Chen, M., Jun, H., Kaiser, L., Plappert, M., Tworek, J., Hilton, J., Nakano, R., Hesse, C., and Schulman, J. Training verifiers to solve math word problems. arXiv preprint arXiv:2110.14168, pp. 1–22, 2021.arxiv.org/abs/2110.14168
- [3]Dang, X., Baek, C., Kolter, J. Z., and Raghunathan, A. Assessing diversity collapse in reasoning. In Workshop on Scaling Self-Improving Foundation Models without Human Supervision, ICLR, 2025.
- [4]DeepSeek-AI. DeepSeek-R1: Incentivizing reasoning capability in LLMs via reinforcement learning. arXiv preprint arXiv:2501.12948, pp. 1–85, 2025.DOI
- [5]Haller, P., Ibrahim, M., Kirichenko, P., Sagun, L., and Bell, S. J. LLM knowledge is brittle: Truthfulness representations rely on superficial resemblance. arXiv preprint arXiv:2510.11905, pp. 1–16, 2025.arxiv.org/abs/2510.11905
- [6]Karan, A. and Du, Y. Reasoning with sampling: Your base model is smarter than you think. arXiv preprint arXiv:2510.14901, pp. 1–28, 2025.arxiv.org/abs/2510.14901
- [7]Meng, H., Huang, K., Wei, S., Ma, C., Yang, S., Wang, X., Wang, G., Ding, B., and Zhou, J. Sparse but critical: A token-level analysis of distributional shifts in RLVR fine-tuning of LLMs. arXiv preprint arXiv:2603.22446, pp. 1–20, 2026.arxiv.org/abs/2603.22446
- [8]OLMo Team. 2 OLMo 2 furious. arXiv preprint arXiv:2501.00656, pp. 1–40, 2025.arxiv.org/abs/2501.00656
- [9]OpenAI. OpenAI o1 system card. Technical report, OpenAI, 2024. URL https://openai.com/index/openai-o1-system-card/.
- [10]Venhoff, C., Arcuschin, I., Torr, P., Conmy, A., and Nanda, N. Base models know how to reason, thinking models learn when. arXiv preprint arXiv:2510.07364, pp. 1–30, 2025.arxiv.org/abs/2510.07364
- [11]Wang, T., Liu, J., Wu, Z., Jin, S., Chen, W., Xu, H., and Miao, N. Not all steps are informative: On the linearity of LLMs’ RLVR training. arXiv preprint arXiv:2601.04537, pp. 1–24, 2026.
- [12]Yan, L., Li, R., Chen, G., Li, Q., Geng, J., Li, W., Wang, V., and Lee, C. Spurious rewards paradox: Mechanistically understanding how RLVR activates memorization shortcuts in LLMs. arXiv preprint arXiv:2601.11061, pp. 1–26, 2026.arxiv.org/abs/2601.11061
- [14]Yue, Y., Chen, Z., Lu, R., Zhao, A., Wang, Z., Yue, Y., Song, S., and Huang, G. Does reinforcement learning really incentivize reasoning capacity in LLMs beyond the base model? arXiv preprint arXiv:2504.13837, pp. 1–25, 2025.arxiv.org/abs/2504.13837