betaFeedback

WorksComputer Science

Primitives and Paths: Predicting What Post-Training Can Do from What the Base Model Already Has

Abstract

Does reinforcement learning teach language models new capabilities, or merely amplify ones acquired during pre-training? The literature offers evidence for both: pass@kk studies suggest RL reweights latent behaviours while some controlled composition studies demonstrate gains beyond simple amplification. We argue that this question has no answer independent of the capability state of the base model, and propose a two-dimensional state space for post-trainability. The first axis, effective full-solution support, measures whether successful trajectories can be sampled under a feasible exploration budget. The second axis, atomic primitive coverage, measures whether the constituent skills of a solution can be executed when the burden of composition is removed. Together, these dimensions motivate three regimes – amplification, composition-limited, and acquisition-limited – each predicting a distinct empirical signature and calling for a distinct intervention. We propose a prospective measurement protocol and investigate the framework using synthetic composition tasks across three small language models. We find that primitive mastery alone does not guarantee that RL can learn a composition. Checkpoints that execute every primitive reliably can fail to improve when complete solutions are never sampled during training, whereas modest composition supervision that establishes nonzero support enables substantial RL gains. The framework reframes mid-training as preparation for post-training: its role is to move models into states from which RL can operate effectively.

Introduction

Reinforcement learning with verifiable rewards (RLVR) has become a central component of post-training language models for mathematical, coding, and other reasoning-intensive tasks. Yet the reasons behind its empirical success remain poorly understood and there is substantial disagreement over what RL contributes beyond capabilities already acquired during pre-training. Does RL teach models genuinely new ways to reason, or does it primarily make existing but low-probability behaviours more likely?

Recent evidence supports both views. Yue et al. [ [12] show that although RL-trained models substantially outperform their base models at pass@1, this advantage often disappears at sufficiently large sampling budgets: given enough attempts, the base model recovers many of the same successful reasoning trajectories as its RL-trained counterpart. Zhao et al. [ [14] similarly find that RL strongly amplifies behaviours established by the pre-training distribution. Together, these results suggest a view of RL as primarily reweighting latent capabilities rather than expanding them.

Other results complicate this position. Yuan et al. [ [11] construct tasks in which constituent skills are available to the base model while their composition is withheld, and show that RL can learn new, transferable compositions of those skills. Chu et al. [ [4] find that RL can generalise beyond training examples in settings where SFT is prone to memorisation, while Bansal et al. [ [1] find that direct RL can expand high-kk performance when applied to pre-training checkpoints, even in settings where the conventional SFT-then-RL pipeline instead sharpens the output distribution. Taken together, these findings suggest that “does RL create new capabilities?” may not have a single answer independent of the model’s state before post-training.

We propose that the effect of post-training should instead be understood relative to the capability state of the base model. In particular, we distinguish two properties of a base model on a task:

  1. Full-solution support: the extent to which complete successful trajectories are already represented in the model’s output distribution

  2. Atomic primitive coverage: the extent to which the constituent skills required for a successful solution are individually available, even when the model cannot reliably compose them

These dimensions distinguish qualitatively different post-training regimes. When both are high, RL can amplify already successful behaviour. When atomic primitive coverage is high but complete solutions are rare, post-training faces a composition problem. When the required atomic primitives themselves are absent, the task is instead acquisition-limited and may require additional supervised or mid-training data before RL can be effective.

This perspective is motivated by a growing body of work showing that standard pre-training metrics are insufficient for predicting downstream post-training outcomes. Kang et al. [ [8], for example, find that high pre-RL SFT performance can be negatively associated with final RL performance and that pass@large-kk and held-out generalisation loss provide substantially stronger predictors. Theoretically, Chen et al. [ [2] formalise a related notion of coverage: the probability mass a pre-trained model assigns to high-quality responses, and show that coverage plays a fundamental role in determining the effectiveness of subsequent post-training and inference-time search. At the same time, decomposing overall task accuracy into constituent skills can reveal important capability gaps obscured by aggregate metrics ([10]).

Our goal is therefore to develop a diagnostic framework for post-trainability. Given a base model and a target problem distribution, what can we measure before post-training that tells us which learning regime we are in and how effective subsequent RL is likely to be? We operationalise full-solution support and atomic primitive coverage, study whether these measurements predict post-training outcomes, and use the resulting state space to propose implications for mid-training. More broadly, we argue that mid-training should not be viewed solely as a means of maximising base-model performance. Its role may instead be to move models into capability states from which subsequent post-training can operate effectively.

Background

RL as capability amplification

One prominent view of reasoning post-training is that RL primarily increases the probability of successful behaviours that are already latent in the base model. Yue et al. [ [12] motivate this view through pass@kk. While post-RL models outperform their corresponding base models when only a small number of samples are drawn, base models often catch up to the post-RL models at large kk. Under this interpretation, pre-training has already placed some probability mass on successful reasoning trajectories. RL changes their relative likelihood and converts rare successes into reliable behaviour.

Zhao et al. [ [14] provide complementary evidence by controlling pre-training data mixtures. Across different RL algorithms and model scales, they find that RL tends to concentrate generation around behavioural patterns already induced by pre-training. Changes to the pre-training distribution consequently alter the behaviours that post-training ultimately amplifies. This establishes an important dependence between the two stages: the same post-training objective can produce different outcomes depending on what pre-training has made available.

Chen et al. [ [2] define the coverage of a pre-trained model in terms of the probability mass assigned to high-quality responses. They show that coverage, rather than cross-entropy loss alone, influences the ability of several downstream procedures to recover good outputs. Foster et al. [ [5] similarly establish a computational role for base-model coverage in efficient exploration. These results motivate measuring not only what a base model produces on its first attempt, but also the structure of the wider distribution from which post-training begins.

Large-kk pass rates are a natural empirical proxy for this property but should not be interpreted as a perfect measure of reasoning support. In particular, large sampling budgets can eventually produce accidental correct answers on tasks with small or discrete answer spaces. We therefore treat pass@kk as one operational measure of full-solution support rather than identifying it with support itself.

Atomic skills, composition, and skill acquisition

A pure reweighting account of RL cannot explain all observed RL gains. Yuan et al. [ [11] explicitly separate possession of atomic skills from the ability to compose them. Their models are first trained on individual functions such as ff and gg while the target composition f(g(x))f(g(x)) is withheld. Subsequent RL can help the model acquire the unseen composition and transfer the learned compositional behaviour to new settings. This suggests that the absence of complete successful trajectories does not imply that a task lies outside the model’s post-trainable capability boundary because the relevant ingredients may already exist in decomposed form.

Qin et al. [ [10] similarly decompose mathematical problem solving into planning, execution, and verification capabilities, and find that RL can affect these components differently. In their experiments, RL predominantly improves execution robustness on problems the model already knows how to solve, while weaknesses in planning impose a “coverage wall” on more novel problems. Whether post-training succeeds may thus depend not only on whether a complete solution is ever sampled, but on which constituent skills are available and which bottleneck prevents them from being assembled into a successful trajectory.

Pre-existing behaviours also shape a model’s capacity to self-improve. Gandhi et al. [ [6] identify verification, backtracking, subgoal setting, and backward chaining as behaviours associated with successful self-improvement under RL. Models initially lacking these behaviours improve less under identical post-training, whereas exposure to data rich in these reasoning patterns – including through continued pre-training – substantially enhances subsequent gains. Two pretrained models with similar headline accuracy may therefore present very different substrates for post-training.

The interaction between SFT, RL, and model state

The appropriate post-training objective may itself depend on the capability state of the base model. Chu et al. [ [4] find that SFT and RL play different roles: SFT is useful for establishing output conventions and initial behaviours while outcome-based RL generalises more strongly in controlled tasks. Conversely, when the model cannot obtain useful rewards on difficult examples, SFT can provide capabilities that RL struggles to discover. This motivates methods such as ReLIFT Ma et al. [2026], which selectively introduces supervised solutions for questions outside the model’s current capability range.

Difficulty therefore matters relative to the current policy rather than solely as an intrinsic property of a problem. Hu et al. [2026] study how examples of varying difficulty should be allocated across SFT and RL, showing that the allocation materially affects final reasoning performance. Kang et al. [ [8] further show that maximising SFT performance is not equivalent to finding the best initialisation point for RL: checkpoints with stronger SFT scores (higher accuracy) can produce weaker final models after identical RL training. Together, these findings motivate evaluating a checkpoint by its post-training potential, rather than its immediate accuracy alone.

Bansal et al. [ [1] push this observation further by applying RL directly to intermediate pre-training checkpoints. They find that RL can improve reasoning substantially even relatively early in pre-training, and that the qualitative effect of RL depends on the preceding training pipeline. In their experiments, SFT followed by RL tends to sharpen the output distribution whereas direct RL can improve both pass@1 and higher-kk performance. “What RL does” thus cannot be separated cleanly from the state from which it begins.

Mid-training as preparation for post-training

If successful post-training depends on particular skills or behaviours already being available, then mid-training offers an opportunity to deliberately establish the prerequisites that RL can subsequently exploit. Recent results support this interpretation. Zhang et al. [ [13] jointly study pre-training, mid-training, and RL in controlled reasoning environments. They find that RL gains depend strongly on whether target tasks lie near the model’s existing “competence boundary” and that allocating part of a fixed compute budget to mid-training can substantially improve subsequent RL generalisation. Gandhi et al. [ [6] similarly show that continued pre-training on data enriched for particular reasoning behaviours can transform a model that plateaus under RL into one that self-improves effectively.

This suggests a different objective for mid-training. Rather than asking only which additional data maximise pre-RL benchmark accuracy, we may ask which data establish the primitives and behavioural support required for successful post-training later. Under this view, reasoning-rich mid-training and RL have complementary roles: mid-training can supply or strengthen missing atomic skills, while RL can exploit, compose, and amplify those skills through on-policy experience.

We introduce a state-space framework in the following section that makes this distinction explicit and provides an operational basis for deciding which intervention is appropriate for a given model-task pair.

The State Space

Full-solution support. The first dimension captures the extent to which a base model can already generate complete successful solutions before post-training. This is closely related to the notion of coverage studied by Chen et al. [ [2], which measures the probability mass a pretrained model places on correct or high-quality responses. Large-kk sampling provides one way to empirically probe this property because a problem with low pass⁡@1\operatorname{pass}@1 but high pass⁡@k\operatorname{pass}@k reveals successful trajectories that the base policy rarely selects, whereas a problem that remains unsolved at large kk provides little evidence that a complete solution is accessible to the model at all Yue et al. [ [12].

We use “support” here in an operational sense rather than a strictly measure-theoretic sense. Finite sampling cannot distinguish a trajectory with zero probability from one with very small probability. Our quantity of interest is therefore effective full-solution support: whether successful trajectories occur with sufficient probability to be recoverable under a feasible sampling and exploration budget. Crucially, this quantity depends on the decoding policy and computational budget, so failure to observe a successful trajectory does not imply that its probability is zero.

Primitive coverage. The second dimension asks whether the model possesses the constituent skills required by the task when the burden of composing those skills is removed. Consider a solution decomposed into primitives P(x)=(s1,…,sm)\mathcal{P}(x) = (s_1,\ldots,s_m). A model may fail to generate the complete solution while nevertheless executing each sjs_j reliably in isolation or when given the appropriate intermediate state or scaffold. We refer to the extent to which these constituent skills are available as atomic primitive coverage. The distinction is motivated by evidence that aggregate accuracy can obscure substantial differences between capabilities such as planning, execution, and verification Qin et al. [ [10], and that RL can learn unseen compositions of atomic skills that were already acquired before RL began Yuan et al. [ [11].

Relationship between the axes. Although the two dimensions capture distinct properties, they are not necessarily independent. A complete successful solution must, by construction, involve some valid sequence of constituent operations. We would therefore generally expect high full-solution support to be accompanied by the availability of the relevant primitives, while high primitive coverage need not imply high full-solution support since the model may fail to select or compose those primitives autonomously.

This suggests an asymmetry in the state space, with model-task pairs plausibly concentrated away from the lower-right region of the plane. We therefore interpret the state space as a diagnostic framework for identifying potential bottlenecks, rather than as four independent or sharply bounded regions.

Together, these two dimensions produce the state space shown in Figure 1.

A state space for post-trainability showing four regimes: Composition-limited, Amplification, Acquisition-limited, and Diagnostic mismatch, plotted by Effective full-solution support and Atomic primitive coverage

Figure 1. A state space for post-trainability. Full-solution support measures whether complete successful trajectories are accessible under the base policy, while atomic primitive coverage measures whether the constituent skills required for a solution are individually available.

Post-training regimes

Amplification. When both primitive coverage and full-solution support are high, successful reasoning trajectories are already accessible under the base policy and rest on capabilities the model can reliably execute. Post-training does not need to construct a qualitatively new solution procedure and can instead increase the probability assigned to successful trajectories. This regime corresponds most directly to the reweighting account of RLVR proposed by Yue et al. [2025] and Zhao et al. [2025]. This region can be thought of as “easy RL” although the difficulty of RL within this region still depends on the density of successful trajectories and different sampling budgets: a solution sampled once in 10510^{5} attempts and one sampled once in five attempts are both cases of amplification but present very different exploration problems. When successful trajectories are sufficiently accessible, standard outcome-based RL can amplify them, typically producing larger improvements at small sampling budgets than at large ones. A characteristic empirical signature of amplification is therefore a substantial increase in low-kk performance after RL with comparatively little expansion at large kk.

Composition-limited. When primitive coverage is high but full-solution support is low, the model appears to possess the ingredients of a solution without reliably being able to assemble them into an end-to-end trajectory. We call this regime composition-limited. Yuan et al. [2025] provide a controlled example of this setting: models that had acquired atomic functions ff and gg could subsequently learn the withheld composition f(g(x))f(g(x)) through RL.

However, possessing the required primitives does not guarantee that vanilla outcome-based RL can learn their composition. If successful end-to-end trajectories are too rare under the training-time sampling distribution, the model may receive little or no informative reward signal even when it can execute every primitive individually. Successful post-training would constitute more than simple probability sharpening here. If complete solutions that were absent or extremely rare before RL become recoverable at large sampling budgets afterwards, while the underlying primitives were already mastered, this is evidence of compositional learning. When compositional learning succeeds, we expect improvements in both pass@1 and effective full-solution support. Interventions in this regime should therefore target the accessibility and execution of complete compositions, potentially through outcome-based RL when rewards are reachable, or through composition-focused supervision, curricula, scaffolding, or intermediate rewards when they are not.

Acquisition-limited. When both full-solution support and primitive coverage are low, at least part of the substrate required for the target task appears to be missing or unreliable. We refer to this regime as acquisition-limited. Low measured primitive coverage does not necessarily establish that a skill is absent. It may instead reflect an inappropriate decomposition, insufficient scaffolding, or a mismatch between the probing context and the context in which the skill is actually available. Vanilla outcome-based RL faces two difficulties here: successful trajectories provide little reward signal, and effective exploration may still fail if the model cannot execute one or more necessary constituent skills. This regime motivates interventions before or alongside RL. Mid-training, SFT, or distillation can strengthen missing or unreliable primitives, potentially increasing primitive coverage and making subsequent composition learning more feasible.

Diagnostic mismatch. High measured full-solution support combined with low measured primitive coverage is qualitatively different from the other regions. If a model reliably solves a task while failing probes for the skills assumed to be necessary, the most immediate conclusion would be to call the decomposition or measurement procedure into question. The model may be exploiting an alternative solution strategy, relying on memorised or contaminated examples, using a shortcut not captured by the proposed primitives, or exposing weaknesses in the primitive probes themselves. We therefore interpret this region as a prompt for further diagnosis rather than a distinct learning regime or a mathematically impossible state.

A continuous state space. The three learning regimes and one diagnostic region are intended as conceptual regimes rather than hard categories. Both axes are continuous, and the relevant transition points depend on the post-training algorithm, rollout budget, verifier, and task. Our central claim is therefore not that every model-task pair belongs unambiguously to one quadrant, but that these two measurements expose different bottlenecks that aggregate base-model accuracy conflates.

Measurement Protocol

We propose a protocol for locating a model-task pair in the state space before post-training. The protocol consists of obtaining independent reference decompositions, measuring effective full-solution support and atomic primitive coverage, diagnosing compositional bottlenecks, and assigning provisional regimes. All measurements and classifications should be made before observing post-training outcomes. Throughout, let π0\pi_0 denote the base policy and π1\pi_1 a post-trained policy derived from it.

The central obstacle in measuring primitive coverage is obtaining a valid decomposition P(x)\mathcal{P}(x) without the synthetic control of Yuan et al. [2025]. Some ways that reference solutions can be obtained independently of π1\pi_1 are using human annotations, stronger external models, or programmatically generated solutions where available. Each reference trajectory is verified for correctness and decomposed into an ordered sequence of primitives P(x)=(s1,…,sm)\mathcal{P}(x) = (s_1,\ldots,s_m). Where multiple valid decompositions exist, these should all be considered when assessing the robustness of primitive coverage.

Estimating full-solution support. For each problem xx, we draw nn independent samples from π0\pi_0 under a specified decoding policy and estimate pass@kk for k≤nk \le n using the unbiased estimator of Chen et al. [2021]. We report both the per-rollout success rate, which estimates the probability of a successful individual trajectory, and pass@kk, which measures success under a sampling budget of kk attempts. The problem’s xx-coordinate is p^0(x)=pass⁡@K(x)\hat{p}_0(x) = \operatorname{pass}@K(x) at the largest feasible budget KK. Since final-answer matching can overestimate support on tasks where answers can be guessed, we follow Yue et al. [2025] and validate a random subsample of successful base-model trajectories for reasoning validity, reporting the estimated false-success rate.

Estimating primitive coverage. For each primitive sjs_j in the reference decomposition, we can construct an isolated execution probe which supplies the appropriate intermediate state and explicitly specifies the operation to be performed, removing the burden of selecting the next primitive. Let qjexecq_j^{\mathrm{exec}} denote the probability that π0\pi_0 executes this operation correctly.

We define atomic primitive coverage using the isolated execution probabilities:

c^(x)=min⁡jqjexec(x),\hat{c}(x) = \min_j q_j^{\mathrm{exec}}(x),

since a single unreliable required primitive can create an execution bottleneck. We additionally report aggregate primitive success rates as the minimum is sensitive to probe noise and decomposition granularity.

Scaffolding and composition diagnostics. To investigate this distinction, we compare autonomous solution success with completion under oracle assistance. A complementary instrument is the handoff curve: given a verified reference trajectory, we supply the first tt steps to π0\pi_0 and measure its probability of correctly completing the remaining trajectory.

Let Hx(t)H_x(t) denote this oracle-assisted completion probability, with Hx(0)H_x(0) corresponding to autonomous full-solution success. We define an empirical composition gap at handoff point tt as

Gcomp(x;t)=Hx(t)−Hx(0).G_{\mathrm{comp}}(x;t) = H_x(t) - H_x(0).

A large positive gap indicates that supplying part of the solution substantially improves completion which is consistent with a bottleneck in autonomous composition. Comparing different scaffolds, such as supplying the next operation explicitly or providing only the current intermediate state, can further help distinguish operation selection from primitive execution and multi-step chaining.

Classifying model-task pairs. We assign provisional regimes using the pre-training-time coordinates (p^0(x),c^(x))(\hat{p}_0(x), \hat{c}(x)). A pair is classified as amplification-oriented when both coordinates exceed thresholds τp\tau_p and τc\tau_c, composition-limited when support is below τp\tau_p but primitive coverage exceeds τc\tau_c, and potentially acquisition-limited when both are below threshold.

We should treat τp\tau_p and τc\tau_c as reporting parameters rather than universal constants. They should be specified before post-training and varied in sensitivity analyses. In particular, the support threshold should be interpreted relative to the rollout budget and sampling distribution of the post-training algorithm.

Triage for measurement mismatch. High measured support with low measured coverage is handled by inspecting the successful trajectories themselves, in decreasing order of parsimony. If they follow a decomposition different from P(x)P(x), the primitive set was wrong and should be re-probed against the observed decomposition. If they follow P(x)P(x) while the corresponding scaffolded probes fail, the probes are at fault and should be repaired. If the trajectories are themselves invalid (e.g. answer-first rationalisation, contamination artifacts, or reasoning that does not support the answer), the observation lends itself to a shortcut interpretation, predicting poor transfer. Each exit either relocates the problem or repairs the instrument; none defines a training regime.

Testing the State-Space Framework

We test our proposed framework by constructing a synthetic composition task in which supervised training controls exposure to individual primitives and their compositions. We then measure each checkpoint’s pre-RL primitive coverage and effective full-solution support and test whether these measurements predict its response to identical outcome-based RL. To distinguish effects of training history from model architecture, we repeat the experiment across three small language models: Qwen3-1.7B, Gemma-3-1B, and Qwen2.5-1.5B-Instruct.

Task and setup. Following the setting of Yuan et al. [ [11], we define two operations on five-digit strings: FF rotates the string left by one position, and GG increments its first digit by three modulo ten. Each problem requires a sequence of 2-4 operations containing both FF and GG. A programmatic verifier checks every intermediate state and the final answer.

For each architecture, we construct checkpoints with different supervised training histories:

  1. M1M_1 is trained on FF only.

  1. M1+C(n)M_1+\mathrm{C}^{(n)} starts from M1M_1 and receives nn examples of mixed compositions, so it sees GG only inside compositions.

  1. M2M_2 is trained on both FF and GG, but never on mixed compositions.

  1. M2+C(n)M_2+\mathrm{C}^{(n)} starts from M2M_2 and receives the same nn composition examples as M1+C(n)M_1+\mathrm{C}^{(n)}.

In both composition conditions, n∈{100,300}n \in\{100, 300\} and the models are trained for one epoch.

These checkpoints are designed to represent different regions of our state space. M1M_1 represents an acquisition-limited state since one of the required primitives is unavailable. The comparison between M1M_1 and M2M_2 represents a primitive-acquisition intervention. In M2M_2, both primitives are now available, but their autonomous composition remains unreliable, which places M2M_2 in the composition-limited regime. Adding composition supervision (M2→M2+C(n)M_2 \to M_2+\mathrm{C}^{(n)}) makes complete trajectories more accessible to RL. The model thus moves towards the amplification regime in the M2+C(n)M_2+\mathrm{C}^{(n)} checkpoints.

We measure pre-RL success on 150 held-out problems using 64 samples per problem. We then apply the same outcome-based GRPO procedure to each checkpoint (150 updates, three seeds). Rewards require a fully correct trajectory.

ModelCheckpointPre-RL pass@1Pre-RL pass@64Pre-RL FF / GGPost-RL pass@1Post-RL pass@64Change Δ\Deltapass@1Change Δ\Deltapass@64
Qwen3-1.7BM1M_10.0000.0001.00 / 0.000.0000.0000.0000.000
Qwen3-1.7BM1+C(100)M_1+\mathrm{C}^{(100)}0.0000.0001.00 / 0.000.0000.0000.0000.000
Qwen3-1.7BM1+C(300)M_1+\mathrm{C}^{(300)}0.0030.0330.97 / 0.000.1720.376+0.169+0.342
Qwen3-1.7BM2M_20.0000.0001.00 / 1.000.0000.0000.0000.000
Qwen3-1.7BM2+C(100)M_2+\mathrm{C}^{(100)}0.0640.1801.00 / 1.000.4520.482+0.388+0.302
Qwen3-1.7BM2+C(300)M_2+\mathrm{C}^{(300)}0.7510.8931.00 / 1.000.9881.000+0.237+0.107
Gemma-3-1BM1M_10.0000.0001.00 / 0.000.0000.0000.0000.000
Gemma-3-1BM1+C(100)M_1+\mathrm{C}^{(100)}0.0000.0000.99 / 0.000.0000.0000.0000.000
Gemma-3-1BM1+C(300)M_1+\mathrm{C}^{(300)}0.0010.0201.00 / 0.000.2340.242+0.232+0.222
Gemma-3-1BM2M_20.0000.0001.00 / 1.000.4760.500+0.476+0.500
Gemma-3-1BM2+C(100)M_2+\mathrm{C}^{(100)}0.5480.6331.00 / 1.000.9790.991+0.431+0.358
Gemma-3-1BM2+C(300)M_2+\mathrm{C}^{(300)}0.9600.9871.00 / 1.000.9890.998+0.030+0.011
Qwen2.5-1.5BM1M_10.0000.0001.00 / 0.000.0000.0000.0000.000
Qwen2.5-1.5BM1+C(100)M_1+\mathrm{C}^{(100)}0.0000.0001.00 / 0.000.0000.0000.0000.000
Qwen2.5-1.5BM1+C(300)M_1+\mathrm{C}^{(300)}0.0050.0930.95 / 0.010.3990.551+0.393+0.458
Qwen2.5-1.5BM2M_20.0000.0001.00 / 1.000.0000.0000.0000.000
Qwen2.5-1.5BM2+C(100)M_2+\mathrm{C}^{(100)}0.0460.0731.00 / 1.000.3110.322+0.266+0.249
Qwen2.5-1.5BM2+C(300)M_2+\mathrm{C}^{(300)}0.1920.5331.00 / 1.000.8680.971+0.676+0.438

Table 1. Pre- and post-RL performance on 150 held-out compositions (64 samples per problem). FF / GG is pre-RL execution accuracy on isolated single-operation prompts with held-out strings. Post-RL values and changes are averaged over three GRPO seeds.

Results. All three M2M_2 checkpoints execute the individual primitives with near-perfect accuracy yet have no observed successful trajectories on the held-out compositions before RL. Qwen3-1.7B and Qwen2.5-1.5B do not improve after RL with GRPO, while Gemma does. A post-hoc examination of the Gemma training logs showed that it had rare successful trajectories on the RL training prompts (6 out of 2,000 prompts solved at least once in 64 pre-RL samples), despite zero observed held-out pass@64. This highlights that effective support depends on the training distribution and sampling policy, rather than held-out benchmark performance alone.

The M1M_1 checkpoints, trained only on FF, have zero success before and after RL across all three architectures. Composition supervision changes this outcome, but its effect depends on the amount of data provided. With 100 composition examples, M1+C(100)M_1+\mathrm{C}^{(100)} still shows no improvement. With 300 examples, the checkpoints acquire nonzero full-solution support, and GRPO raises pass@1 to 0.17-0.40. Interestingly, these checkpoints retain near-zero accuracy on isolated GG probes, despite successfully using GG within some complete trajectories.

More generally, composition-focused supervision increases effective full-solution support and enables larger subsequent RL gains. When initial support is limited, RL can increase both pass@1 and pass@64, making previously rarely sampled solutions more accessible at finite sampling budgets. At high initial support, as with M2+C(300)M_2+C^{(300)} for Qwen3-1.7B, improvements are concentrated more heavily at pass@1 which is consistent with amplification.

Limitations and Future Work

Primitive coverage and composition. Our experiments show that isolated primitive mastery does not necessarily translate into successful composition. Models may execute individual primitives reliably while struggling to select and transition between them, and isolated probes may fail to reveal skills available in compositional contexts. Consequently, we do not yet establish that measured primitive coverage adds predictive power beyond full-solution support. Moreover, coverage estimates depend on how solutions are decomposed into primitives and the scaffolding used to probe them. A promising direction is to better characterise compositional connectivity which can be thought of as the model’s ability to select, sequence, and transition between available skills. Such orchestration capabilities may help explain variation within the composition-limited regime.

Support measurement and threshold selection. Effective full-solution support depends on the task distribution, sampling policy, and exploration budget. Failure to observe a successful trajectory at finite kk does not establish that it is absent, and our experiments show that held-out pass@kk can underestimate the reward accessibility encountered during RL training. Our classification also depends on thresholds τp\tau_p and τc\tau_c, which we do not yet offer a principled selection procedure for. Future work could calibrate these thresholds to the training-time exploration budget or replace discrete regimes with continuous models of post-training outcomes.

Generalisability. Our experiments use small language models and synthetic composition tasks with simple and known primitives and deterministic verification. While this allows us to isolate different capability bottlenecks in a toy example, natural reasoning tasks admit multiple valid decompositions, less clearly defined primitives, and more complex forms of composition. Extending the framework to mathematical reasoning, coding, and larger models will be important for establishing whether these diagnostics generalise beyond controlled settings.

Conclusion

We have argued that the question “does RL create new capabilities?” is ill-posed when asked in the abstract. It is better replaced by a conditional question: given a base model’s capability state on a task, what can post-training do from there? The state space we propose factors the state into two measurable properties – effective full-solution support and atomic primitive coverage – that aggregate model accuracy conflates. Apparent contradictions in the RLVR literature can then be read as observations about different regions of this space: pass@k convergence results characterise amplification, controlled composition studies characterise the composition-limited regime, and failures of vanilla RL on hard tasks may reflect acquisition-limited states or composition-limited states in which complete solutions are too rare to be rewarded. Our synthetic experiments support the central role of effective support. Outcome-based RL improved only checkpoints with some reachable successful trajectories and composition supervision that created such trajectories subsequently enabled large gains. They also show that primitive mastery is not sufficient on its own and that measured support depends on the distribution and budget under which it is probed. The two axes are not independent because any complete solution must be built from a valid sequence of operations, and the space carries internal structure that can be used to validate measurements rather than merely classify outcomes.

References

  1. [1]R. Bansal, C. Mohri, T. Qin, D. Alvarez-Melis, and S. Kakade. RL excursions during pre-training: Re-examining policy optimization for llm training, 2026. URL https://arxiv.org/abs/2606.04272.
  2. [2]F. Chen, A. Huang, N. Golowich, S. Malladi, A. Block, J. T. Ash, A. Krishnamurthy, and D. J. Foster. The Coverage Principle: How Pre-Training Enables Post-Training, Oct. 2025. URL http://arxiv.org/abs/2510.15020. arXiv:2510.15020 [stat.ML].
  3. [4]T. Chu, Y. Zhai, J. Yang, S. Tong, S. Xie, D. Schuurmans, Q. V. Le, S. Levine, and Y. Ma. SFT Memorizes, RL Generalizes: A Comparative Study of Foundation Model Post-training, May 2025. URL http://arxiv.org/abs/2501.17161. arXiv:2501.17161 [cs.AI].
  4. [5]D. J. Foster, Z. Mhammedi, and D. Rohatgi. Is a good foundation necessary for efficient reinforcement learning? the computational role of the base model in exploration, 2025. URL https://arxiv.org/abs/2503.07453.
  5. [6]K. Gandhi, A. Chakravarthy, A. Singh, N. Lile, and N. D. Goodman. Cognitive Behaviors that Enable Self-Improving Reasoners, or, Four Habits of Highly Effective STaRs, Aug. 2025. URL http://arxiv.org/abs/2503.01307. arXiv:2503.01307 [cs.CL].
  6. [8]F. Kang, M. Kuchnik, K. Padt he, M. Vlastelica, R. Jia, C.-J. Wu, and N. Ardalani. Quagmires in SFT-RL Post-Training: When High SFT Scores Mislead and What to Use Instead, Oct. 2025. URL http://arxiv.org/abs/2510.01624. arXiv:2510.01624 [cs.LG].
  7. [10]T. Qin, C. F. Park, M. Kwun, A. Walsman, E. Malach, N. Anand, H. Tanaka, and D. Alvarez-Melis. Decomposing Elements of Problem Solving: What "Math" Does RL Teach?, May 2025. URL http://arxiv.org/abs/2505.22756. arXiv:2505.22756 [cs.AI].
  8. [11]L. Yuan, W. Chen, Y. Zhang, G. Cui, H. Wang, Z. You, N. Ding, Z. Liu, M. Sun, and H. Peng. From $f(x)$ and $g(x)$ to $f(g(x))$: LLMs Learn New Skills in RL by Composing Old Ones, Dec. 2025. URL http://arxiv.org/abs/2509.25123. arXiv:2509.25123 [cs.AI].
  9. [12]Y. Yue, Z. Chen, R. Lu, A. Zhao, Z. Wang, Y. Yue, S. Song, and G. Huang. Does Reinforcement Learning Really Incentivize Reasoning Capacity in LLMs Beyond the Base Model?, Nov. 2025. URL http://arxiv.org/abs/2504.13837. arXiv:2504.13837 [cs.AI].
  10. [13]C. Zhang, G. Neubig, and X. Yue. On the Interplay of Pre-Training, Mid-Training, and RL on Reasoning Language Models, Dec. 2025. URL http://arxiv.org/abs/2512.07783. arXiv:2512.07783 [cs.CL].
  11. [14]R. Zhao, A. Meterez, S. Kakade, C. Pehlevan, S. Jelassi, and E. Malach. Echo Chamber: RL Post-training Amplifies Behaviors Learned in Pretraining, Aug. 2025. URL http://arxiv.org/abs/2504.07912. arXiv:2504.07912 [cs.LG].

Paper details

vlnc:W_ecvvx2my
Version
v1 · latest
Published
Fresh — added to the Hub within the last 30 days
Added
License
CC BY 4.0
Usage

views · 10

downloads · 0

A view counts once per browser per day, a download each time the PDF is requested.

  • Computer Science

Not recorded for this paper

View statement
Looking for related papers…
Contents