betaFeedback

WorksComputer Science

Policy Improvement-Guided Replay for Continual Reinforcement Learning

Abstract

Continual reinforcement learning (CRL) requires models to acquire new capabilities while retaining performance on previous tasks. A stability-oriented perspective common in conventional continual learning emphasizes knowledge preservation and interference mitigation. Complementing this perspective, we revisit CRL for vision-language models through policy improvement, a fundamental principle of RL, to preserve acquired performance gains. Using the performance difference lemma, we characterize historical performance changes through the current policy's expected advantages on historical task distributions, motivating replay of historical prompts with the current policy. To allocate limited replay memory, we propose Contrastive Likelihood Replay (CoLiR), which contrasts likelihood gains on correct responses from the final task policy with likelihood reductions on incorrect responses from the initial policy. Our analysis links correct-response likelihood gains to reference-policy advantages under an idealized KL-regularized optimal update and provides an endogenous-reward interpretation for general policy updates. CoLiR estimates scores from finite training rollouts and cached log-likelihoods, selects prompts at task boundaries, and replays them uniformly during subsequent training, with KL regularization restricted to replayed prompts. Memory construction only requires a small amount of additional cross-policy scoring but no additional response generation. Across three Qwen3-VL model sizes on MRCL, CoLiR achieves the highest mean final accuracy and mean task accuracy among the compared methods while maintaining competitive new-task performance.

Introduction

Continual post-training enables large language and vision-language models (VLMs) to incrementally acquire diverse downstream capabilities in response to evolving application requirements [8, 12]. However, adaptation to new tasks

Figure 1. Task retention in CRL: CoLiR outperforms all compared methods in both final and stage-averaged accuracy.

can degrade performance on previously learned ones, a phenomenon known as catastrophic forgetting [10]. The central challenge lies in enabling cumulative capability acquisition across successive stages of post-training while preserving previously learned capabilities.

Prior efforts to mitigate forgetting in continual supervised fine-tuning have primarily relied on explicit knowledge-preservation mechanisms, such as rehearsal and parameter regularization [10, 17]. Recent studies further suggest that the post-training paradigm itself influences knowledge retention [14]. Compared with supervised fine-tuning, reinforcement learning (RL) can exhibit stronger retention, attributed to on-policy data and mode-seeking updates [4], as well as an implicit preference for solutions with smaller policy divergence [31]. These findings highlight the inherent potential of RL to mitigate forgetting, motivating further research into continual reinforcement learning (CRL) for large models. Nevertheless, CPO reveals that RL remains susceptible to catastrophic forgetting in continual post-training of VLMs and introduces explicit regularization to constrain historical policy drift during sequential RL [20].

These studies highlight the retention properties of RL and the effectiveness of constraining historical policy drift. However, such constraints do not directly characterize how policy updates affect historical task performance. This reflects a stability-oriented perspective commonly adopted in conventional continual learning, motivating a complementary perspective grounded in policy improvement, a fundamental principle of RL. Although RL pursues policy improvement by maximizing the expected return on an individual task, optimizing this objective independently at each learning stage does not guarantee the preservation of performance gains achieved on preceding tasks. This raises an important question: how can policy improvement guide the selection and retention of historical experiences during continual RL, particularly under limited memory capacity?

To address this question, we revisit CRL through the performance difference lemma, a foundational tool for analyzing policy improvement [28]. Under a single-step formulation that treats a complete response as an action, the performance change on a historical task equals the current policy’s expected advantage over a reference policy under that task’s prompt distribution. Preserving previously acquired performance gains thus requires maintaining nonnegative expected advantages relative to the corresponding historical reference policies. Current-task-only RL directly optimizes expected returns over the current task distribution, providing no guarantee of preserving performance on previous tasks. Historical prompt replay addresses this limitation by allowing the current policy to respond to retained prompts and receive task-specific verifier feedback, thereby incorporating historical reward objectives into subsequent policy optimization. Under limited memory capacity, however, the effectiveness of replay also depends on which historical prompts are retained.

From a policy improvement perspective, we aim to retain historical prompts associated with substantial performance gains during task learning, rather than relying solely on their final correctness or difficulty. However, directly estimating prompt-specific advantages requires evaluating the reference policy’s expected reward, which can be computationally expensive. To address this, we investigate policy likelihood changes as a tractable proxy for advantage-guided selection. Under an idealized KL-regularized update, we establish a monotonic relationship between likelihood gains on correct responses and their reference-policy advantages. For general policy updates, we further characterize how likelihood gains reflect changes in success probability and probability redistribution among correct responses, providing an endogenous-reward interpretation [16]. Together, these results provide theoretical motivation for using likelihood changes to guide historical prompt selection.

Building on this analysis, we propose CoLiR (Contrastive Likelihood Replay), a policy improvement-guided replay method for CRL. CoLiR combines likelihood gains on correct responses sampled from the final task policy with likelihood reductions on incorrect responses sampled from the initial policy, capturing both the reinforcement of correct responses and the suppression of incorrect ones. We derive sufficient conditions under which the population contrastive score indicates positive policy improvement. In practice, we estimate this score from finite training rollouts and use the resulting ranking to construct a capacity-limited replay buffer at the end of each task. During subsequent training, retained prompts are uniformly sampled for joint optimization with current-task data, with KL regularization applied to replayed prompts. By reusing training rollouts and cached log-likelihoods, CoLiR requires only a single round of additional cross-policy scoring, without generating new responses for memory construction. Across three model sizes on MRCL, CoLiR achieves the highest mean final task and mean task accuracy among the compared methods, while maintaining competitive performance on new tasks.

Related Work

Continual Reinforcement Learning and Post-Training CRL aims to adapt policies to sequential tasks while preserving capabilities and enabling knowledge transfer [1, 13, 36]. Existing methods mitigated forgetting by constraining changes to important parameters (EWC [17]), consolidating knowledge through distillation (P&C [29]), or combining replay with on-policy learning to balance retention and adaptation (CLEAR [26]). LAMOL and SSR explored generated training examples for knowledge retention in language-model continual learning [10, 32]. In continual post-training, prior studies found that RL could retain knowledge better than supervised fine-tuning [4, 14], which RL’s Razor attributed to a preference for smaller policy divergence [31]. Yet CPO showed that RL can still suffer substantial forgetting in multimodal reasoning and language-model-free regularization to constrain historical policy drift [20]. While prior work emphasized knowledge retention and interference mitigation, we revisit CRL for VLMs through the performance difference lemma and develop likelihood-based selective replay to retain previously acquired policy improvements.

Replay Selection In conventional RL, PER assigned higher sampling probabilities to transitions with larger temporal-difference errors [27], while PLR prioritized environments by estimated learning potential [11]. For efficient language-model RL, Prompt Replay prioritized intermediate-difficulty prompts for repeated on-policy optimization [2]; Prompt Curriculum Learning and RIF-RFT selected prompts with effective learning signals [7, 14]; and DPS and GPS used historical training dynamics to predict informative prompts and reduce sampling overhead [21, 24]. CPR retains historical prompts and dynamically prioritized their reuse by recent pass rates and cooldown constraints [19]. These prompt-selection methods primarily targeted difficulty or future learning potential. To retain prior policy improvements in CRL for VLMs, we draw on connections between policy likelihood ratios and implicit rewards [16, 25] to select historical prompts with pronounced likelihood changes during task learning. Unlike dynamic replay prioritization, our memory selection occurs at task boundaries and is decoupled from subsequent uniform replay.

Preliminaries

Reinforcement Learning with Verifiable Rewards RLVR optimizes language models with programmatically checkable outcomes [30, 38]. For a prompt x∼Dx \sim D, the policy generates y=(o,a)y = (o, a), comprising a reasoning trace oo and final answer aa; a task-specific verifier assigns the binary correctness reward R(x,y)∈{0,1}R(x,y) \in\{0,1\}. The rollout policy πold≡πθold\pi_{\mathrm{old}} \equiv\pi_{\theta_{\mathrm{old}}} samples GG responses Y={yj}j=1GY = \{y^j\}_{j=1}^{G} per prompt. Group Relative Policy Optimization (GRPO) [30] estimates group-relative advantages:

A^j=R(x,yj)−μRσR+ξ,\widehat{A}_j = \frac{R(x,y^j)-\mu_R}{\sigma_R+\xi},

where μR\mu_R and σR\sigma_R are the group reward mean and standard deviation, and ξ>0\xi> 0 ensures numerical stability. All tokens in a response share this advantage under outcome supervision. Define the clipped surrogate

Cϵ(r,A)=min⁡{rA,clip⁡(r,1−ϵ,1+ϵ)A},C_{\epsilon}(r,A)=\min\{rA,\operatorname{clip}(r,1-\epsilon,1+\epsilon)A\},

with clipping parameter ϵ\epsilon. GRPO applies token-level clipping, whereas CoLiR uses Group Sequence Policy Optimization (GSPO) [41] as the optimization algorithm, which applies sequence-level clipping for training stability, with length-normalized importance ratios and objective

J(πθ)=1GEx,Y[∑j=1GCϵ([πθ(yj∣x)πold(yj∣x)]1/∣yj∣,A^j)].J(\pi_\theta)=\frac{1}{G}\mathbb{E}_{x,Y}\left[\sum_{j=1}^{G}C_{\epsilon}\left(\left[\frac{\pi_\theta(y^j\mid x)}{\pi_{\mathrm{old}}(y^j\mid x)}\right]^{1/\lvert y^j\rvert},\widehat{A}_j\right)\right].

Continual Reinforcement Learning CRL trains a shared policy on sequential tasks T1,…,TKT_1,\ldots,T_K. Each VLM task ii provides prompts x=(v,u)∼Dix=(v,u)\sim D_i, with visual input vv and textual query uu. From an RL perspective, we model response generation as a single-step decision: xx is the initial state, and the complete response y∼πθ(⋅∣x)y\sim\pi_\theta(\cdot\mid x) is the action, scored by a task verifier with Ri(x,y)∈{0,1}R_i(x,y)\in\{0,1\}. Let πˉiπ(x)=Ey∼π(⋅∣x)[Ri(x,y)]\bar{\pi}_i^\pi(x)=\mathbb{E}_{y\sim\pi(\cdot\mid x)}[R_i(x,y)] denote the prompt-level success probability. The expected task return is

Ji(πθ)=Ex∼Di, y∼πθ(⋅∣x)[Ri(x,y)],J_i(\pi_\theta)=\mathbb{E}_{x\sim D_i,\ y\sim\pi_\theta(\cdot\mid x)}[R_i(x,y)],

which equals expected answer accuracy under binary rewards. At stage kk, the goal is to improve performance on DkD_k while mitigating forgetting on tasks i<ki<k. Historical performance is Jold(πθ)=∑i<kwiJi(πθ)J_{\mathrm{old}}(\pi_\theta)=\sum_{i<k}w_iJ_i(\pi_\theta), with fixed evaluation weights wi≥0w_i\geq0 and ∑i<kwi=1\sum_{i<k}w_i=1. We use a streaming setting: training data arrive task by task, and direct access to historical data is restricted after each task.

Methodology

Revisiting CRL through Policy Improvement

We revisit CRL through the lens of policy improvement, a fundamental principle of RL. For a historical task ii and a fixed reference policy πˉi\bar{\pi}_i, the response-level advantage is

Aiπˉi(x,y)=Ri(x,y)−πˉi(x).(1)A_i^{\bar{\pi}_i}(x,y)=R_i(x,y)-\bar{\pi}_i(x). \tag*{(1)}

Under the single-step formulation, the performance difference lemma [28] yields

Ji(πθ)−Ji(πˉi)=Ex∼Di, y∼πθ(⋅∣x)[Aiπˉi(x,y)].(2)J_i(\pi_\theta)-J_i(\bar{\pi}_i)=\mathbb{E}_{x\sim D_i,\ y\sim\pi_\theta(\cdot\mid x)}\left[A_i^{\bar{\pi}_i}(x,y)\right]. \tag*{(2)}

Cross-task perspective. Eq. (2) indicates preserving or improving historical performance requires nonnegative expected advantage relative to the reference policy, evaluated over historical inputs x∼Dix\sim D_i and responses generated by the current policy y∼πθ(⋅∣x)y\sim\pi_\theta(\cdot\mid x). Since the reference baseline πˉi(x)\bar{\pi}_i(x) is independent of θ\theta, maximizing the expected advantage is equivalent to maximizing the expected reward on task ii. Optimizing only the current distribution DkD_k, which generally differs from DiD_i, does not guarantee historical performance. This motivates historical task replay, which reintroduces past task inputs and enables the current policy to be optimized against historical task objectives.

Within-task perspective. Limited memory capacity of historical replay raises a further question: which historical prompts should be retained? From a policy improvement perspective, advantage measures the relative value of a response, and differences in response advantages reflect the varying contributions of prompts. This motivates prioritizing prompts that contribute most to advantage maximization, with the aim of preserving the key improvements acquired during previous tasks. However, binary verification rewards indicate correctness without directly quantifying the advantage relative to the reference policy, making such contributions difficult to assess directly. This motivates exploring more accessible signals that can reflect advantage-related learning contributions and guide replay selection.

Policy Likelihood Ratios as an Advantage Proxy

To identify historical prompts associated with substantial policy improvement, we investigate whether policy likelihood changes of correct responses can serve as a tractable proxy for advantage-based replay selection. Let πi−\pi_i^- and πi+\pi_i^+ denote the policies before and after learning task ii, respectively. For each prompt xx, let qT+(⋅∣x)q_T^+(\cdot\mid x) denote the final policy’s conditional distribution over correct responses. We consider the average log-likelihood improvement of correct responses:

P1(x)=Ey∼qT+(⋅∣x)[log⁡πiT(y∣x)πi0(y∣x)].(3)P_1(x)=\mathbb{E}_{y\sim q_T^+(\cdot\mid x)}\left[\log\frac{\pi_i^T(y\mid x)}{\pi_i^0(y\mid x)}\right]. \tag*{(3)}

We establish its theoretical motivation through advantage monotonicity and endogenous reward interpretation.

Connection to Policy Advantage. We first consider an idealized KL-regularized policy improvement objective:

πiT=arg⁡max⁡π{Ey∼π[Ri(x,y)]−βDKL(π∥πi0)},(4)\pi_i^T=\underset{\pi}{\arg\max}\left\{\mathbb{E}_{y\sim\pi}[R_i(x,y)]-\beta D_{\mathrm{KL}}(\pi\mathbin\Vert\pi_i^0)\right\}, \tag*{(4)}

where β>0\beta>0 is a fixed regularization coefficient.

Proposition 1 (Advantage Monotonicity). Under the KL-regularized optimal policy, P1(x)P_1(x) is strictly increasing in the reference-policy advantage of a correct response:

P1(x)=−log⁡[1−Aiπi0(x,y+)(1−e−1/β)].(5)P_1(x)=-\log\left[1-A_i^{\pi_i^0}(x,y^+)\left(1-e^{-1/\beta}\right)\right]. \tag*{(5)}

The proof is provided in Appendix § 7.§ 1. For sufficiently strong KL regularization, P1(x)≈Aiπi0(x,y+)/βP_1(x)\approx A_i^{\pi_i^0}(x,y^+)/\beta. This idealized result motivates likelihood-based selection, although GSPO does not require KL regularization. Proposition 2 provides a complementary endogenous-reward interpretation for general policy updates.

Endogenous Reward Interpretation. We next provide an endogenous-reward interpretation of P1P_1 that applies to general policy updates. Recent work on endogenous rewards [16] connects language model likelihoods to implicit rewards through inverse reinforcement learning. Following its outcome-reward formulation, we consider

r^π(x,y)=αlog⁡π(y∣x)+Vπ(x),(6)\hat r_{\pi}(x,y)=\alpha\log\pi(y\mid x)+V_{\pi}(x), \tag*{(6)}

where α>0\alpha>0 and Vπ(x)V_{\pi}(x) is a prompt-dependent value term.

Proposition 2 (Endogenous Reward and Performance Decomposition). Let pt(x)p_t(x) and qt+(⋅∣x)q_t^+(\cdot\mid x) denote the success probability and conditional distribution over correct responses under πit\pi_i^t, for t∈{0,T}t\in\{0,T\}. Assuming 0<p0(x),pT(x)<10<p_0(x),p_T(x)<1 and common support on correct responses, we have

P1(x)=EqT+[Δr^i(x,y)]−ΔVi(x)α,(7)P_1(x)=\frac{\mathbb{E}_{q_T^+}[\Delta\hat r_i(x,y)]-\Delta V_i(x)}{\alpha}, \tag*{(7)}

and

P1(x)=log⁡pT(x)p0(x)+DKL(qT+∥q0+).(8)P_1(x)=\log\frac{p_T(x)}{p_0(x)}+D_{\mathrm{KL}}(q_T^+\mathbin\Vert q_0^+). \tag*{(8)}

The proof is provided in Appendix § 7.§ 2. The first identity interprets P1P_1 as the response-dependent component of endogenous reward changes. The second decomposes P1P_1 into the log-relative change in success probability and the distributional shift among correct responses. Under the KL-regularized update in Proposition 1, the conditional distribution over correct responses is unchanged, making the KL term vanish. For general policy updates, this term captures additional changes in the distribution of correct responses.

Combining Eq. (8) with the performance difference lemma gives

EπT[A^iπi0(x,y)]=p0(x)[eP1(x)−DKL(qT+∥q0+)−1].(9)\mathbb{E}_{\pi_T}[\hat A_i^{\pi_i^0}(x,y)]=p_0(x)\left[e^{P_1(x)-D_{\mathrm{KL}}(q_T^+\mathbin\Vert q_0^+)}-1\right]. \tag*{(9)}

Thus, when the KL divergence term is small, P1P_1 closely approximates the log-relative performance improvement. Together, these results justify P1P_1 as a tractable proxy for advantage-guided replay selection. Under the KL-regularized policy improvement, P1P_1 preserves the ordering of reference-policy advantages. Under general policy updates, it captures the response-dependent component of endogenous reward changes and is explicitly related to the realized expected advantage through Eq. (9).

CoLiR: Contrastive Likelihood Replay

While the preceding analysis establishes the theoretical motivation for P1P_1, policy improvement under binary rewards involves both reinforcing correct responses and suppressing incorrect ones. Specifically, relative to the initial policy πi0\pi_i^0, incorrect responses have negative advantages, i.e., Aiπi0(x,y−)=−πi0(x)A_i^{\pi_i^0}(x,y^-)=-\pi_i^0(x). To capture this complementary aspect, we define

P2(x)=Ey∼q0−(⋅∣x)[log⁡πi0(y∣x)πiT(y∣x)],(10)P_2(x)=\mathbb{E}_{y\sim q_0^-(\cdot\mid x)}\left[\log\frac{\pi_i^0(y\mid x)}{\pi_i^T(y\mid x)}\right], \tag*{(10)}

where q0−q_0^- denotes the initial policy's conditional distribution over incorrect responses. Thus, P1P_1 measures the reinforcement of correct responses generated by the final policy, whereas P2P_2 measures the suppression of incorrect responses generated by the initial policy.

We combine these complementary signals into a contrastive score:

Si(x)=12(P1(x)+P2(x)).(11)S_i(x)=\frac{1}{2}\left(P_1(x)+P_2(x)\right). \tag*{(11)}

Following Eq. (6), we have

2αSi(x)=EqT+[Δr^i(x,y)]−Eq0−[Δr^i(x,y)].(12)2\alpha S_i(x)=\mathbb{E}_{q_T^+}[\Delta\hat r_i(x,y)]-\mathbb{E}_{q_0^-}[\Delta\hat r_i(x,y)]. \tag*{(12)}

This contrast cancels the prompt-dependent value change, allowing Si(x)S_i(x) to capture shifts in endogenous reward preference for correct over incorrect responses. Furthermore, the contrastive score admits a direct connection to realized policy improvement.

Figure 2. Overview of CoLiR. CoLiR builds a replay buffer based on contrastive scores and samples from it during training on new tasks.

Proposition 3 (Contrastive Likelihood and Policy Improvement). Let pt(x)p_t(x) and qt±(⋅∣x)q_t^{\pm}(\cdot\mid x) denote the success probability and conditional response distributions under πit\pi_i^t, for t∈{0,T}t \in\{0,T\}. Assume 0<p0(x),pT(x)<10 < p_0(x),p_T(x) < 1 and finite conditional KL divergences. Then

2Si(x)=logit⁡pT(x)−logit⁡p0(x)+εi(x),(13)2S_i(x) = \operatorname{logit} p_T(x) - \operatorname{logit} p_0(x) + \varepsilon_i(x), \tag*{(13)}

where

εi(x)=DKL(qT+∥q0+)+DKL(q0−∥qT−)≥0.\varepsilon_i(x) = D_{\mathrm{KL}}(q_T^+ \| q_0^+) + D_{\mathrm{KL}}(q_0^- \| q_T^-) \ge0.

If εi(x)≤δ\varepsilon_i(x) \le\delta, then

Si(x)>δ2⟹Ey∼πiT(⋅∣x)[Ai0(x,y)]>0.(14)S_i(x) > \frac{\delta}{2} \Longrightarrow\mathbb{E}_{y\sim\pi_i^T(\cdot\mid x)}[A_i^0(x,y)] > 0. \tag*{(14)}

The proof is provided in Appendix § 7.§ 3. Proposition 3 relates the contrastive score to realized policy improvement through changes in success log-odds. Under bounded distributional discrepancies, larger scores provide higher lower bounds on policy improvement, motivating the selection of high-scoring prompts for replay.

In practical implementation, we leverage the group-based sampling mechanism, which generates GG responses per prompt and records their log-likelihoods during policy optimization. To reduce computational cost, for each task ii, we reuse the incorrect responses from the initial training iteration and the correct responses from the final iteration, together with their corresponding log-likelihoods. Let Y0,x−\mathcal{Y}_{0,x}^{-} and YT,x+\mathcal{Y}_{T,x}^{+} denote these two response sets, respectively. The empirical contrastive score is computed as

S^i(x)=12∣YT,x+∣∑y∈YT,x+log⁡πiT(y∣x)πi0(y∣x)+12∣Y0,x−∣∑y∈Y0,x−log⁡πi0(y∣x)πiT(y∣x).(15)\widehat{S}_i(x) = \frac{1}{2|\mathcal{Y}_{T,x}^{+}|} \sum_{y\in\mathcal{Y}_{T,x}^{+}} \log\frac{\pi_i^T(y\mid x)}{\pi_i^0(y\mid x)} + \frac{1}{2|\mathcal{Y}_{0,x}^{-}|} \sum_{y\in\mathcal{Y}_{0,x}^{-}} \log\frac{\pi_i^0(y\mid x)}{\pi_i^T(y\mid x)}. \tag*{(15)}

Since the log-likelihoods under the generating policies are already available, only two additional cross-policy likelihood evaluations on partial responses are required, without additional response generation.

After completing task ii, we consider prompts whose final rollout groups have mixed correctness outcomes, and whose initial rollout groups contain at least one incorrect response. We rank these eligible prompts by S^i(x)\widehat{S}_i(x) and retain the top-BB prompts in the replay buffer, where BB denotes the per-task memory budget. During subsequent task learning, we allocate a replay budget and randomly sample historical prompts from the replay buffer for joint optimization with current-task data. To align with the KL-regularized policy improvement, we further impose a KL term on replayed prompts to mitigate forgetting.

Experiments

Experimental Setup

Datasets. We conduct CRL experiments on the MRCL benchmark [20], following the task sequence Med-BookVQA, Navigation, We-Math2, Puzzle, and FinMME. These tasks cover medical visual question answering, spatial navigation, mathematical reasoning, visual puzzles, and financial understanding. Due to answer leakage in the dataset provided by CPO [20], we correct formatting inconsistencies, answer annotations, and scoring errors, and remove invalid or duplicate examples from the original benchmark. We evaluate final performance, performance throughout training, and new-task learning using Mean Final Accuracy (MFN), Mean Task Accuracy (MTA), and Mean Finetune Accuracy (MFT), respectively. We also assess general capability retention on out-of-distribution (OOD) benchmarks covering STEM reasoning, general visual question answering, hallucination, chart and document understanding, and counting.

Experimental settings. We conduct experiments on Qwen3-VL at three scales: 2B, 4B, and 8B. CoLiR uses GSPO [41] with historical replay. By ranking prompts using the contrastive score, CoLiR retains at most B=80B = 80 prompts per historical task, with BB much smaller than the size of each task’s training set.

We jointly optimize performance on current and historical tasks using responses generated by the current policy, with a training batch size of 32 prompts; each batch includes 4 replay prompts at T2 and 8 at T3–T5, sampled uniformly across historical tasks. The replay KL loss coefficient is β=0.01\beta= 0.01, with the reference policy frozen at the start of the current task. To evaluate CoLiR’s effectiveness, we compare CoLiR with supervised baselines, including FullSFT, LoRA [9], O-LoRA [34], and RegLoRA [5], and RL baselines, including GRPO [30], GRPO* [30] (with a KL coefficient of 0.04), GSPO [41], and CPO [20]. CPO employs parameter regularization, a common technique in continual learning, to limit changes to parameters associated with historical tasks. After correcting the MRCL data and scoring issues, we reproduce all baselines using their official code. Further baseline details are provided in Appendix § 8.§ 2.

Main Results

CoLiR consistently improves task retention across model scales. CoLiR achieves the highest MFN and MTA at all three model scales (Table 1). Consistent with CPO’s findings [20], the MFT–MFN gap for GRPO and GSPO indicates forgetting during subsequent task learning. CoLiR substantially improves retention relative to these baselines, and its MFN advantage over CPO increases with model size, reaching 9.99 percentage points at 8B. Despite allocating part of the training budget to replay, CoLiR achieves MFT comparable to that of GSPO across all three model sizes, and outperforms CPO at 2B and 4B. CoLiR also maintains high Navigation accuracy after subsequent tasks and shows improvements across tasks with different reasoning requirements at 4B. Together, these results demonstrate that CoLiR effectively mitigates forgetting while preserving new-task learning ability.

SizeMethodMedBookNavig.We-Math2PuzzleFinMMEMFN↑MTA↑MFT↑
2BFullSFT45.6719.1133.9084.0032.6943.0751.9764.33
2BLoRA59.0849.1735.7588.0032.8052.9656.8762.34
2BO-LoRA57.7222.8235.2275.5035.1745.2951.3257.66
2BRegLoRA61.6343.0435.5181.2533.0050.8954.4157.80
2BGRPO50.9319.1145.1469.5041.7845.2949.8855.17
2BGRPO*52.4619.4848.4455.2541.2743.3844.9145.87
2BGSPO47.0313.3640.2476.7543.1444.1053.1963.43
2BCPO56.5433.5847.3475.7542.3351.1154.1761.31
2BCoLiR (Ours)53.1452.6945.8082.5042.4855.3259.8463.68
4BFullSFT53.4824.8637.7879.7535.4246.2657.1466.35
4BLoRA66.8945.6442.2635.0041.6246.2844.9351.94
4BO-LoRA75.0449.5443.6793.7541.3260.6663.2266.97
4BRegLoRA76.9174.9544.7037.7541.0255.0756.4759.09
4BGRPO60.9518.7451.5476.0043.1450.0755.6062.48
4BGRPO*48.902.0453.1828.2541.1734.7135.4436.54
4BGSPO58.5735.6251.6091.5045.5656.5764.8773.45
4BCPO60.2760.1155.3083.0049.2961.5966.9570.78
4BCoLiR (Ours)62.8291.2857.9689.5045.2169.3570.8672.27
8BFullSFT58.5725.6043.2035.7537.7940.1849.2658.24
8BLoRA65.8747.1244.8995.2542.2859.0861.2370.85
8BO-LoRA72.5052.3244.7835.2542.7949.5350.3753.75
8BRegLoRA72.8472.3646.3955.5044.7058.3660.6762.93
8BGRPO62.143.9056.3567.0045.1646.9149.9052.98
8BGRPO*67.0627.2762.1058.5048.5952.7055.9259.86
8BGSPO58.743.1556.3790.2545.4650.7960.8572.71
8BCPO64.5233.5856.4389.2547.1258.1865.4972.51
8BCoLiR (Ours)60.6194.8156.3784.0045.0668.1770.3672.44

Table 1. Table 1. Performance comparison of different methods on the MRCL benchmark at three model scales of Qwen3-VL. The best results within each scale are shown in bold. The second-best results are underlined.

General Capability Retention

CoLiR preserves general capabilities while learning new tasks. Although CPO and GRPO* achieve average OOD scores similar to CoLiR (Table 2), they rely on explicit parameter regularization and a stronger KL constraint applied to all training prompts, respectively, both of which restrict model updates. CoLiR instead emphasizes RL optimization on current-task prompts and selected historical prompts, with a smaller KL coefficient applied only to replayed prompts. Taken together with the results in Table 1, these findings show that CoLiR achieves comparable general capability preservation while attaining stronger task acquisition and retention than both methods at 4B. Using the same GSPO optimizer, CoLiR retains substantially more of the original model’s general capabilities than vanilla GSPO across multiple evaluation dimensions. This contrast supports the contribution of valuable historical replay, together with its local KL constraint, to capability preservation. Overall, CoLiR improves performance on the task sequence while limiting degradation in general capabilities.

DimensionBenchmarkBaseFullSFTGRPOGRPO*GSPOCPOCoLiR
STEMMMMU-Profull_{\mathrm{full}} [39]46.0923.4135.1041.9533.5137.1341.50
MathVistamini_{\mathrm{mini}} [18]55.8048.4058.5064.2061.5066.5067.00
MathVisionTEST_{\mathrm{TEST}} [33]21.2216.7826.2823.8823.5528.5529.70
MathVersemini_{\mathrm{mini}} [40]36.9027.1340.4640.5642.9448.1545.13
General VQARealWorldQA [37]68.2450.3365.2356.4761.8360.3957.52
MMStar [6]59.1353.0058.8057.1360.5362.2762.40
HallucinationPOPE [15]87.2684.0382.8885.4685.0984.7183.89
Chart/Doc UnderstandingDocVQAVAL_{\mathrm{VAL}} [22]65.3462.4655.7769.7157.8469.1360.01
CharXivDQ_{\mathrm{DQ}} [35]71.8642.4143.9873.0049.1054.9353.62
CharXivRQ_{\mathrm{RQ}} [35]33.9024.9028.0025.8030.1033.3031.30
CountingCountBenchQA [3, 23]84.3269.2560.6971.0867.6258.6678.62
Average57.2845.6550.5255.3952.1554.8855.52

Table 2. Performance comparison on out-of-distribution benchmarks after sequential training. The best results among trained models are shown in bold. The second-best results are _underlined_.

Ablation Studies

Component Ablations. Filtering, contrastive selection, and replay KL jointly improve performance. As shown in Table 3, the full method performs best across all three metrics. Removing candidate filtering produces the largest decrease in final accuracy, while only random sampling also weakens retention. These results support two-stage selection: filtering defines the eligible pool, and contrastive scoring prioritizes prompts whose likelihood changes indicate substantial policy improvement. In addition, removing replay KL lowers all three metrics, affecting both task acquisition and retention. These results support the empirical benefit of KL regularization on replayed prompts.

MethodMFN↑MTA↑MFT↑
CoLiR69.3570.8672.27
w/o accuracy filtering60.7863.5170.60
w/o contrastive selection66.5968.7771.10
w/o replay KL66.9968.1670.21

Table 3. Ablation results for different components.

Comparison of Selection Methods. Joint contrastive scoring improves replay selection. As shown in Table 4, joint contrastive selection consistently outperforms accuracy-gain selection, selection based on either P1P_1 or P2P_2 alone. An intuitive alternative is to replay prompts on which the model has improved most, as measured by the gain in sampled accuracy. CoLiR’s advantage over this alternative suggests that changes in the response distribution provide useful selection information beyond the gain in sampled accuracy, supporting the use of log-likelihood changes as a proxy for advantage. Joint scoring also outperforms scoring based on either P1P_1 or P2P_2 alone, indicating the benefit of considering how the model both reinforces correct responses and suppresses incorrect ones when selecting prompts for replay.

Selection methodMFN↑MTA↑MFT↑
Contrastive (CoLiR)69.3570.8672.27
Accuracy gain63.0865.4470.06
P1P_1 only66.8068.5070.64
P2P_2 only64.0067.2569.80
Random66.5968.7771.10

Table 4. Performance comparison under different replay selection methods.

Further Analysis

Likelihood Components of Selected Prompts. What likelihood changes characterize the prompts selected by CoLiR? Selected prompts generally have high P1P_1 and P2P_2, indicating increased likelihood of correct responses and decreased likelihood of incorrect ones (Figure 3). Meanwhile, the joint score also captures prompts with high P1P_1 but lower P2P_2, and vice versa. Thus, contrastive selection captures learning progress reflected in either or both components. These patterns are consistent with the contrastive score capturing a shift in relative preference toward correct responses, supporting the benefit of combining both components observed in Table 4.

Figure 3. Joint distributions of correct-response log-likelihood increase and incorrect-response log-likelihood decrease across T2–T5.

Correctness Patterns of Selected Prompts. How does likelihood-based selection relate to changes in correctness? As shown in Figure 4, selection rates vary across prompts grouped by early and late correctness, estimated from eight responses per prompt. On Navigation, selected prompts are concentrated in groups with high late correctness, while on Puzzle, selection favors prompts that move from low early correctness to higher late correctness and also show an accuracy increase. Selected prompts in We-Math2 and FinMME span a wider range of early-to-late correctness transitions. These observations suggest a correlation between likelihood changes and accuracy gains, while potentially capturing additional information about the response distribution beyond sampled accuracy.

Figure 4. Replay selection rates across different combinations of early and late correct-response counts for T2–T5.

Conclusions

We revisit CRL for VLMs through policy improvement, using the performance difference lemma to motivate historical prompt replay. CoLiR selects prompts by contrasting likelihood gains on correct final-policy responses with reductions on incorrect initial-policy responses. Our analysis connects correct-response likelihood gains to reference-policy advantages under idealized KL-regularized optimization and provides an endogenous-reward interpretation for general updates. CoLiR estimates scores from cached training rollouts and likelihoods, requiring no additional response generation for selection. The selected prompts are uniformly replayed during subsequent RL training. Experiments across three model scales show improved retention while maintaining new-task learning.

LLM Usage Statement

We used OpenAI’s GPT-6 and GPT-6.1 to assist with text polishing, related-work searches and citation checks, experimental analysis, and checking theorem proofs. For figures, these models only drafted initial layouts, which the authors manually refined. The authors take full responsibility for all submitted content. We provided major portions of the manuscript to OpenAI for reviewer-style feedback. The authors take full responsibility for all submitted content.

Theoretical Proofs

Proof of Proposition 1

Proof. The optimal policy associated with the KL-regularized objective has the Boltzmann form

πiT(y∣x)=πi0(y∣x)eRi(x,y)/βZi(x),\pi_i^T(y \mid x) = \frac{\pi_i^0(y \mid x)e^{R_i(x,y)/\beta}}{Z_i(x)},

where the normalization term is

Zi(x)=1−πi0(x)+πi0(x)e1/β.Z_i(x) = 1-\pi_i^0(x)+\pi_i^0(x)e^{1/\beta}.

For any correct response y+y^+ under the binary reward formulation, Ri(x,y+)=1R_i(x,y^+) = 1. Its advantage under the reference policy is therefore

Aiπi0(x,y+)=Ri(x,y+)−Ey∼πi0(⋅∣x)[Ri(x,y)]=1−πi0(x).A_i^{\pi_i^0}(x,y^+) = R_i(x,y^+)-\mathbb{E}_{y\sim\pi_i^0(\cdot\mid x)}[R_i(x,y)] = 1-\pi_i^0(x).

Moreover, substituting the optimal-policy expression into Eq. (3) gives

P1(x)=log⁡e1/β1−πi0(x)+πi0(x)e1/β.P_1(x)=\log\frac{e^{1/\beta}}{1-\pi_i^0(x)+\pi_i^0(x)e^{1/\beta}}.

Using πi0(x)=1−Aiπi0(x,y+)\pi_i^0(x)=1-A_i^{\pi_i^0}(x,y^+), we obtain

P1(x)=−log⁡[1−Aiπi0(x,y+)(1−e−1/β)],P_1(x)=-\log\left[1-A_i^{\pi_i^0}(x,y^+)\left(1-e^{-1/\beta}\right)\right],

which establishes Eq. (5).

Furthermore, differentiating this expression with respect to the reference-policy advantage yields

∂P1(x)∂Aiπi0(x,y+)=1−e−1/β1−Aiπi0(x,y+)(1−e−1/β)>0.\frac{\partial P_1(x)}{\partial A_i^{\pi_i^0}(x,y^+)} =\frac{1-e^{-1/\beta}}{1-A_i^{\pi_i^0}(x,y^+)(1-e^{-1/\beta})}>0.

Hence, P1(x)P_1(x) is strictly increasing in the positive-response advantage.

When β\beta is sufficiently large, 1/β1/\beta is small. Applying the first-order Taylor expansion e−z=1−z+O(z2)e^{-z}=1-z+O(z^2) around z=0z=0 gives

1−e−1/β=1β+O(1β2)≈1β.1-e^{-1/\beta}=\frac{1}{\beta}+O\left(\frac{1}{\beta^2}\right)\approx\frac{1}{\beta}.

Substituting this approximation into Eq. (5) and applying −log⁡(1−z)=z+O(z2)-\log(1-z)=z+O(z^2) for small zz yields

P1(x)≈Aiπi0(x,y+)β.P_1(x)\approx\frac{A_i^{\pi_i^0}(x,y^+)}{\beta}.

Thus, under strong KL regularization, the likelihood improvement of correct responses is approximately proportional to the reference-policy advantage. □

Proof of Proposition 2

Proof. We first prove the endogenous reward characterization. By the definition of the endogenous reward,

r^πiT(x,y)=αlog⁡πiT(y∣x)+VπiT(x),\hat{r}_{\pi_i^T}(x,y)=\alpha\log\pi_i^T(y\mid x)+V_{\pi_i^T}(x),

and

r^πi0(x,y)=αlog⁡πi0(y∣x)+Vπi0(x).\hat{r}_{\pi_i^0}(x,y)=\alpha\log\pi_i^0(y\mid x)+V_{\pi_i^0}(x).

Taking their difference gives

Δr^i(x,y)=αlog⁡πiT(y∣x)πi0(y∣x)+ΔVi(x).\Delta\hat{r}_i(x,y)=\alpha\log\frac{\pi_i^T(y\mid x)}{\pi_i^0(y\mid x)}+\Delta V_i(x).

Since ΔVi(x)\Delta V_i(x) depends only on the prompt and is independent of the response yy, taking the expectation under the final policy’s correct-response distribution qT+(⋅∣x)q_T^+(\cdot\mid x) yields

EqT+[Δr^i(x,y)]=αP1(x)+ΔVi(x).\mathbb{E}_{q_T^+}[\Delta\hat{r}_i(x,y)]=\alpha P_1(x)+\Delta V_i(x).

Rearranging this equality proves Eq. (7).

We next derive the performance decomposition. For any correct response y∈Yx+y\in\mathcal{Y}_x^+, the policy probability can be factorized as

πit(y∣x)=pt(x)qt+(y∣x),t∈{0,T},\pi_i^t(y\mid x)=p_t(x)q_t^+(y\mid x),\qquad t\in\{0,T\},

because pt(x)p_t(x) is the total probability assigned to correct responses and qt+(⋅∣x)q_t^+(\cdot\mid x) is the corresponding conditional distribution. Therefore,

πiT(y∣x)πi0(y∣x)=pT(x)p0(x)qT+(y∣x)q0+(y∣x).\frac{\pi_i^T(y\mid x)}{\pi_i^0(y\mid x)} =\frac{p_T(x)}{p_0(x)}\frac{q_T^+(y\mid x)}{q_0^+(y\mid x)}.

Taking logarithms gives

log⁡πiT(y∣x)πi0(y∣x)=log⁡pT(x)p0(x)+log⁡qT+(y∣x)q0+(y∣x).\log\frac{\pi_i^T(y\mid x)}{\pi_i^0(y\mid x)} =\log\frac{p_T(x)}{p_0(x)}+\log\frac{q_T^+(y\mid x)}{q_0^+(y\mid x)}.

Taking the expectation with respect to qT+(⋅∣x)q_T^+(\cdot\mid x), the first term remains unchanged because it is independent of yy, while the second term becomes

EqT+[log⁡qT+(y∣x)q0+(y∣x)]=DKL(qT+(⋅∣x)∥q0+(⋅∣x)).\mathbb{E}_{q_T^+}\left[\log\frac{q_T^+(y\mid x)}{q_0^+(y\mid x)}\right] =D_{\mathrm{KL}}\left(q_T^+(\cdot\mid x)\|q_0^+(\cdot\mid x)\right).

Combining the two terms proves Eq. (8). □

Proof of Proposition 3

Proof. For a given prompt xx, let Yx+\mathcal{Y}_x^+ and Yx−\mathcal{Y}_x^- denote the sets of correct and incorrect responses, respectively. By the definition of conditional distributions, for t∈{0,T}t \in\{0,T\}, we have

πit(y∣x)={pt(x)qt+(y∣x),y∈Y+,(1−pt(x))qt−(y∣x),y∈Y−.(16)\pi_i^t(y \mid x) = \begin{cases} p_t(x)q_t^+(y \mid x), & y \in\mathcal{Y}^+,\\ (1-p_t(x))q_t^-(y \mid x), & y \in\mathcal{Y}^-. \end{cases} \tag*{(16)}

For correct responses sampled from the final policy, the likelihood improvement can be expressed as

P1(x)=EqT+[log⁡πiT(y∣x)πi0(y∣x)]=log⁡pT(x)p0(x)+EqT+[log⁡qT+(y∣x)q0+(y∣x)]=log⁡pT(x)p0(x)+DKL(qT+∥q0+).(17)P_1(x) = \mathbb{E}_{q_T^+}\left[\log\frac{\pi_i^T(y \mid x)}{\pi_i^0(y \mid x)}\right] = \log\frac{p_T(x)}{p_0(x)} + \mathbb{E}_{q_T^+}\left[\log\frac{q_T^+(y \mid x)}{q_0^+(y \mid x)}\right] = \log\frac{p_T(x)}{p_0(x)} + D_{\mathrm{KL}}(q_T^+ \mathbin{\|} q_0^+). \tag*{(17)}

Similarly, for incorrect responses sampled from the initial policy, we obtain

P2(x)=Eq0−[log⁡πi0(y∣x)πiT(y∣x)]=log⁡1−p0(x)1−pT(x)+DKL(q0−∥qT−).(18)P_2(x) = \mathbb{E}_{q_0^-}\left[\log\frac{\pi_i^0(y \mid x)}{\pi_i^T(y \mid x)}\right] = \log\frac{1-p_0(x)}{1-p_T(x)} + D_{\mathrm{KL}}(q_0^- \mathbin{\|} q_T^-). \tag*{(18)}

Combining the two expressions yields

2Si(x)=P1(x)+P2(x)=log⁡pT(x)(1−p0(x))p0(x)(1−pT(x))+εi(x)=logit⁡pT(x)−logit⁡p0(x)+εi(x),(19)2S_i(x) = P_1(x) + P_2(x) = \log\frac{p_T(x)(1-p_0(x))}{p_0(x)(1-p_T(x))} + \varepsilon_i(x) = \operatorname{logit} p_T(x) - \operatorname{logit} p_0(x) + \varepsilon_i(x), \tag*{(19)}

where

εi(x)=DKL(qT+∥q0+)+DKL(q0−∥qT−)≥0.\varepsilon_i(x) = D_{\mathrm{KL}}(q_T^+ \mathbin{\|} q_0^+) + D_{\mathrm{KL}}(q_0^- \mathbin{\|} q_T^-) \ge0.

This establishes Eq. (13).

Since εi(x)\varepsilon_i(x) is nonnegative, the decomposition immediately implies

pT(x)>p0(x)⟹Si(x)>0,Si(x)<0⟹pT(x)<p0(x).(20)p_T(x) > p_0(x) \Longrightarrow S_i(x) > 0, \qquad S_i(x) < 0 \Longrightarrow p_T(x) < p_0(x). \tag*{(20)}

Thus, positive policy improvement necessarily yields a positive score, whereas a negative score indicates performance degradation.

Furthermore, suppose εi(x)≤δ\varepsilon_i(x) \le\delta. Rearranging the decomposition gives

logit⁡pT(x)−logit⁡p0(x)=2Si(x)−εi(x)≥2Si(x)−δ.(21)\operatorname{logit} p_T(x) - \operatorname{logit} p_0(x) = 2S_i(x) - \varepsilon_i(x) \ge2S_i(x) - \delta. \tag*{(21)}

Therefore, if Si(x)>δ/2S_i(x) > \delta/2, then logit⁡pT(x)>logit⁡p0(x)\operatorname{logit} p_T(x) > \operatorname{logit} p_0(x). Since the logit function is strictly increasing, this implies pT(x)>p0(x)p_T(x) > p_0(x).

Experimental Details

Datasets and Corrections

Figure 6 shows one test example from each MRCL task in training order, illustrating the different response formats used across tasks. For MedBookVQA, an extra space in correct options leaks the answers, allowing all evaluated models to easily exceed 90% accuracy. We standardize option spacing to remove this shortcut. For Navigation, we correct an answer annotation and extend scoring to accept equivalent shortest paths. For We-Math2, we repair malformed text and incorrect answers, remove duplicate records, and exclude invalid examples. For Puzzle, we remove duplicate examples. For FinMME, we correct answer annotations, remove empty unit hints and duplicate records, and exclude invalid examples. The scoring implementation supports accepted-answer aliases so that equivalent valid answers receive the same reward. Table 5 summarizes the four-task correction patch; answer corrections are counted before duplicate removal.

TaskCorrectedExcludedDuplicates
Navigation100
We-Math21913138
Puzzle002
FinMME14584

Table 5. Summary of the data correction patch for T2–T5.

Figure 6. Examples from the five MRCL [20] tasks in training order. Each panel shows a test image, its question, and the reference answer. Task instructions are abbreviated for display.

Baseline Methods

Supervised baselines. (1) FullSFT: updates all model parameters through supervised fine-tuning; (2) LoRA [9]: learns low-rank weight updates while freezing the pre-trained parameters; (3) O-LoRA [34]: uses orthogonal low-rank subspaces to reduce interference between tasks; (4) RegLoRA [5]: regularizes important elements of LoRA weight updates to preserve historical knowledge.

Reinforcement learning baselines. (1) GRPO [30]: optimizes the policy using group-relative advantages; (2) GRPO* [30]: adds KL regularization to GRPO; (3) GSPO [41]: uses sequence-level importance ratios for policy optimization; (4) CPO [20]: regularizes changes to important parameters to mitigate forgetting during sequential RL training.

Implementation Details

We train the supervised baselines for 2, 3, 1, 1, and 1 epochs on T1–T5, respectively, while RL training runs for 600 steps per task. For all model sizes, GSPO, CPO, and CoLiR share the same checkpoint after training on MedBookVQA with GSPO, since the first stage involves only a single task. Their continual-learning updates begin at the second stage.

Table 6 summarizes the training and replay settings of CoLiR. The early and late response sets used for selection come from the first and last distinct complete generation events recorded for each prompt during task training. When fewer than KK prompts satisfy the filtering criteria, all eligible prompts are retained.

Parameter NameValue
Learning rate5×10−65 \times10^{-6}
Training steps (per task)600
Training batch size32
Group size GG8
Replay batch size (T2)4
Replay batch size (T3–T5)8
Replay buffer size KK (per task)80
Replay KL coefficient β\beta0.01
Clipping parameter ϵlow\epsilon_{\mathrm{low}}3×10−43 \times10^{-4}
Clipping parameter ϵhigh\epsilon_{\mathrm{high}}4×10−44 \times10^{-4}

Table 6. Training and replay settings of CoLiR.

Evaluation Metrics

Let at,ia_{t,i} denote accuracy on task ii after training on task tt, with $1 \leq i \leq t \leq T$ and T=5T = 5. The metrics introduced in the main text are computed as

MFN=1T∑i=1TaT,i,(22)\mathrm{MFN} = \frac{1}{T}\sum_{i=1}^{T} a_{T,i}, \tag*{(22)}
MTA=1T∑i=1T(1T−i+1∑t=iTat,i),(23)\mathrm{MTA} = \frac{1}{T}\sum_{i=1}^{T}\left(\frac{1}{T-i+1}\sum_{t=i}^{T}a_{t,i}\right), \tag*{(23)}
MFT=1T∑i=1Tai,i.(24)\mathrm{MFT} = \frac{1}{T}\sum_{i=1}^{T}a_{i,i}. \tag*{(24)}

Additional Experiments

Replay Memory Capacity

How does the number of retained prompts affect performance? As shown in Table 7, increasing the capacity from 40 to 80 prompts per historical task improves MFN, MTA, and MFT, with a larger gain in MFN than in MFT. Increasing the capacity further to 160 does not yield additional gains, and K=80K = 80 achieves the strongest results in this comparison. This pattern indicates that replay benefits do not increase monotonically with memory capacity. In CoLiR, capacity determines how many prompts are retained from the contrastive ranking, and expanding the retained set also includes lower-ranked candidates. The results are consistent with prioritizing prompts associated with substantial policy improvement, rather than relying solely on a larger replay memory.

KKMFN↑\uparrowMTA↑\uparrowMFT↑\uparrow
4058.2363.2170.04
8069.3570.8672.27
16067.9569.3471.07

Table 7. Performance under different replay memory capacities. KK is the maximum number of retained prompts per historical task. Best results are shown in bold.

Contrastive Score Ablation

Do higher-scoring prompts provide greater replay value? We initialize three branches from the same 4B checkpoint saved at the end of Navigation training (T2), so all three inherit the same MedBookVQA replay set. We rank the Navigation candidate prompts by contrastive score. One branch adds the top 80 prompts to its replay buffer, another adds the bottom 80, and the third adds 80 sampled at random from the same pool. All three branches then train on We-Math2 (T3) under otherwise identical settings.

As shown in Table 8, the three branches achieve similar accuracy on We-Math2, while high-score replay yields the highest accuracy on both prior tasks, with the clearest advantage on Navigation. On Navigation, low-score replay also outperforms random replay. Figure 4 in the main paper suggests that higher contrastive scores tend to accompany larger gains in prompt-level accuracy during the original task, while lower-scoring prompts tend to show smaller gains. The low-score set may therefore contain more prompts for which accuracy remains relatively low late in Navigation training, including some with success rates near 50%. At this level, sampling several responses to the same prompt is more likely to yield both correct and incorrect answers within a rollout group. The resulting differences in reward provide a learning signal during replay and may explain the low-score branch’s advantage over random selection. The further gain from high-score replay, together with the comparison of selection methods in Table 4 of the main paper, suggests that selecting prompts with high contrastive scores helps direct a limited replay budget toward prompts on which the policy made larger gains during the original task.

SelectionHistorical tasksCurrent task
MedBookVQANavigationWe-Math2
High-score69.2791.2863.26
Low-score68.5986.8362.17
Random64.8684.6062.53

Table 8. Task accuracy (%) after T3 with different replay selections from a shared checkpoint. Best results are shown in bold.

Composition of Selected Prompts

How does contrastive selection change the composition of the replay set? As shown in Figure 5, the selected prompts differ from the eligible pool in both knowledge category and path length. On We-Math2, geometry accounts for a larger share of the selected set, increasing from 63.9% to 79.2%, while the share of algebra decreases and that of probability and statistics remains similar. Selection therefore changes the relative emphasis across knowledge categories while retaining prompts from all three categories. On Navigation, the selected set is more concentrated on paths of 2–4 steps, especially 3-step paths, whereas paths of 6 steps or more become less frequent. Thus, contrastive selection does not simply favor longer paths. These shifts arise after candidate filtering, when eligible prompts are ranked by their contrastive scores. Under our policy-improvement formulation, likelihood changes serve as a proxy for advantage; selection therefore prioritizes the improvement reflected in response distributions rather than preserving category proportions or ranking prompts by path length. The resulting replay set reflects where the policy has improved within each task, connecting the observed composition shifts to the selection criterion.

Figure 5. Composition of the eligible pool and selected replay prompts: (a) knowledge categories on We-Math2; (b) path lengths on Navigation. Shares are normalized within each pool.

Detailed Continual Learning Results

Tables 9–11 present the task-by-stage accuracy matrices. Rows indicate the most recently trained task, and columns indicate the evaluated task. Bold entries mark the diagonal; blank cells correspond to tasks not yet learned.

MethodRowMedBookNavigationWe-Math2PuzzleFinMME
(a) FullSFTMedBook75.55
(a) FullSFTNavigation71.3177.92
(a) FullSFTWe-Math271.1456.0341.23
(a) FullSFTPuzzle50.087.0530.6494.25
(a) FullSFTFinMME45.6719.1133.9084.0032.69
(b) LoRAMedBook74.36
(b) LoRANavigation74.5372.54
(b) LoRAWe-Math270.9763.8239.76
(b) LoRAPuzzle63.6737.1136.2792.25
(b) LoRAFinMME59.0849.1735.7588.0032.80
(c) O-LoRAMedBook73.85
(c) O-LoRANavigation72.6756.22
(c) O-LoRAWe-Math267.9151.7638.06
(c) O-LoRAPuzzle63.3320.4135.5685.00
(c) O-LoRAFinMME57.7222.8235.2275.5035.17
(d) RegLoRAMedBook73.34
(d) RegLoRANavigation72.1658.63
(d) RegLoRAWe-Math271.3156.5937.77
(d) RegLoRAPuzzle66.0441.9335.8486.25
(d) RegLoRAFinMME61.6343.0435.5181.2533.00
(e) GRPOMedBook64.35
(e) GRPONavigation63.5039.52
(e) GRPOWe-Math261.9726.1652.43
(e) GRPOPuzzle59.9316.1448.3377.75
(e) GRPOFinMME50.9319.1145.1469.5041.78
(f) GRPO*MedBook57.89
(f) GRPO*Navigation57.8914.47
(f) GRPO*We-Math258.0616.1451.45
(f) GRPO*Puzzle56.3718.1849.9664.25
(f) GRPO*FinMME52.4619.4848.4455.2541.27
(g) GSPOMedBook67.57
(g) GSPONavigation65.0366.05
(g) GSPOWe-Math265.2048.9855.41
(g) GSPOPuzzle58.7413.7341.4685.00
(g) GSPOFinMME47.0313.3640.2476.7543.14
(h) CPOMedBook67.57
(h) CPONavigation66.2163.64
(h) CPOWe-Math263.8446.5753.78
(h) CPOPuzzle62.1437.6626.0779.25
(h) CPOFinMME56.5433.5847.3475.7542.33
(i) CoLiR (Ours)MedBook67.57
(i) CoLiR (Ours)Navigation61.8066.60
(i) CoLiR (Ours)We-Math261.4664.0153.02
(i) CoLiR (Ours)Puzzle60.6159.5549.6588.75
(i) CoLiR (Ours)FinMME53.1452.6945.8082.5042.48

Table 9. Task accuracy (%) after each training stage on Qwen3-VL-2B. Each panel shows one method.

PanelTaskMedBookNavigationWe-Math2PuzzleFinMME
(a) FullSFTMedBook77.76
(a) FullSFTNavigation76.0683.49
(a) FullSFTWe-Math273.8566.0549.06
(a) FullSFTPuzzle53.3158.2640.2686.00
(a) FullSFTFinMME53.4824.8637.7879.7535.42
(b) LoRAMedBook77.25
(b) LoRANavigation74.8760.11
(b) LoRAWe-Math274.5348.6143.97
(b) LoRAPuzzle44.315.9432.3236.75
(b) LoRAFinMME66.8945.6442.2635.0041.62
(c) O-LoRAMedBook81.32
(c) O-LoRANavigation75.5573.10
(c) O-LoRAWe-Math278.4459.3746.12
(c) O-LoRAPuzzle76.0654.3645.2493.00
(c) O-LoRAFinMME75.0449.5443.6793.7541.32
(d) RegLoRAMedBook80.81
(d) RegLoRANavigation71.6582.93
(d) RegLoRAWe-Math279.4675.5145.43
(d) RegLoRAPuzzle80.3174.5844.8745.25
(d) RegLoRAFinMME76.9174.9544.7037.7541.02
(e) GRPOMedBook71.65
(e) GRPONavigation68.0862.52
(e) GRPOWe-Math268.5943.7858.08
(e) GRPOPuzzle61.4632.8448.6277.00
(e) GRPOFinMME60.9518.7451.5476.0043.14
(f) GRPO*MedBook56.37
(f) GRPO*Navigation50.251.48
(f) GRPO*We-Math250.252.9752.70
(f) GRPO*Puzzle51.102.4152.5731.00
(f) GRPO*FinMME48.902.0453.1828.2541.17
(g) GSPOMedBook73.85
(g) GSPONavigation73.8591.84
(g) GSPOWe-Math271.3184.9764.02
(g) GSPOPuzzle68.0832.8454.1692.00
(g) GSPOFinMME58.5735.6251.6091.5045.56
(h) CPOMedBook73.85
(h) CPONavigation72.1682.00
(h) CPOWe-Math271.9979.2265.24
(h) CPOPuzzle69.4469.7659.0883.50
(h) CPOFinMME60.2760.1155.3083.0049.29
(i) CoLiR (Ours)MedBook73.85
(i) CoLiR (Ours)Navigation68.9387.01
(i) CoLiR (Ours)We-Math269.2791.2863.26
(i) CoLiR (Ours)Puzzle67.4090.1758.6992.00
(i) CoLiR (Ours)FinMME62.8291.2857.9689.5045.21

Table 10. Task accuracy (%) after each training stage on Qwen3-VL-4B. Each panel shows one method.

(a) FullSFTMedBookNavigationWe-Math2PuzzleFinMME
MedBook79.63
Navigation79.1286.46
We-Math276.5780.1552.31
Puzzle34.3055.6641.0235.00
FinMME58.5725.6043.2035.7537.79
(b) LoRAMedBookNavigationWe-Math2PuzzleFinMME
MedBook82.34
Navigation80.9884.97
We-Math279.1275.7049.91
Puzzle47.3721.8926.1594.75
FinMME65.8747.1244.8995.2542.28
(c) O-LoRAMedBookNavigationWe-Math2PuzzleFinMME
MedBook81.32
Navigation79.9762.34
We-Math279.6350.6547.06
Puzzle78.9534.1444.6235.25
FinMME72.5052.3244.7835.2542.79
(d) RegLoRAMedBookNavigationWe-Math2PuzzleFinMME
MedBook82.68
Navigation81.3277.37
We-Math278.7874.0347.88
Puzzle77.0874.0346.5362.00
FinMME72.8472.3646.3955.5044.70
(e) GRPOMedBookNavigationWe-Math2PuzzleFinMME
MedBook69.78
Navigation71.9912.62
We-Math268.592.2361.09
Puzzle68.253.9059.3476.25
FinMME62.143.9056.3567.0045.16
(f) GRPO*MedBookNavigationWe-Math2PuzzleFinMME
MedBook71.31
Navigation73.1752.88
We-Math271.9934.6964.28
Puzzle70.9731.3563.2062.25
FinMME67.0627.2762.1058.5048.59
(g) GSPOMedBookNavigationWe-Math2PuzzleFinMME
MedBook74.19
Navigation72.6788.68
We-Math273.3461.6062.95
Puzzle66.897.6155.0592.25
FinMME58.743.1556.3790.2545.46
(h) CPOMedBookNavigationWe-Math2PuzzleFinMME
MedBook74.19
Navigation73.3485.34
We-Math270.1262.1565.40
Puzzle71.3152.3262.4490.50
FinMME64.5233.5856.4389.2547.12
(i) CoLiR (Ours)MedBookNavigationWe-Math2PuzzleFinMME
MedBook74.19
Navigation71.6594.25
We-Math270.8094.0661.44
Puzzle67.0691.4758.0387.25
FinMME60.6194.8156.3784.0045.06

Table 11. Task accuracy (%) after each training stage on Qwen3-VL-8B. Each panel shows one method.

References

  1. [1]David Abel, André Barretto, Benjamin Van Roy, Doina Precup, Hado Philip van Hasselt, and Santosh Singh. A definition of continual reinforcement learning. In Advances in Neural Information Processing Systems 36: Annual Conference on Neural Information Processing Systems 2023, NeurIPS 2023, New Orleans, LA, USA, December 10 - 16, 2023, 2023. 2arxiv.org/abs/2307.11046
  2. [2]Andrei Baroian and Rutger Berger. Prompt replay: speeding up gro with on-policy reuse of high-signal prompts. CoRR, abs/2603.21177, 2026. 2
  3. [3]Lucas Beyer, Andreas Steiner, André Susano Pinto, Alexander Kolesnikov, Xiao Wang, Daniel Salz, Maxim Neumann, Ibrahim Alabdulmohsin, Michael Tschannen, Emanuele Bugliarello, Thomas Unterthiner, Daniel Keysers, Skanda Koppula, Fangyu Liu, Adam Gryncer, Alexey Gritsenko, Neil Houlsby, Manoj Kumar, Keran Rong, Julian Eisenschlos, Rishabh Kabra, Matthias Bauer, Matko Bošnjak, Xi Chen, Matthias Minderer, Paul Voigtlaender, Ioana Bica, Ivana Balazevic, Joan Puigcerver, Pinelopi Papalampidi, Olivier Henaff, Xi Xiang, Radu Soricut, Jeremiah Harmsen, and Xiaohua Zhai. PaliGemma: A versatile 3B VLM for transfer. arXiv preprint arXiv:2407.07726, 2024. 7arxiv.org/abs/2407.07726
  4. [4]Howard Chen, Noam Razin, Karthik Narasimhan, and Danqi Chen. Retaining by doing: The role of on-policy data in mitigating forgetting. CoRR, abs/2510.18874, 2025. 1, 2arxiv.org/abs/2510.18874
  5. [5]Jinpeng Chen, Runmin Cong, Yuzhi Zhao, Hongzheng Yang, Guangneng Hu, Horace H. S. Ip, and Sam Kwong. SEFE: superficial and essential forgetting eliminator for multimodal continual instruction tuning. In Forty-second International Conference on Machine Learning, ICML 2025, Vancouver, BC, Canada, July 13-19, 2025. PMLR / OpenReview.net, 2025. 6, 2arxiv.org/abs/2505.02486
  6. [6]Lin Chen, Jinsong Li, Xiaoyi Dong, Pan Zhang, Yuhang Zang, Zehui Chen, Haodong Duan, Jiaqi Wang, Yu Qiao, Dahua Lin, and Feng Zhao. Are We on the Right Way for Evaluating Large Vision-Language Models? arXiv preprint arXiv:2403.20330, 2024. 7arxiv.org/abs/2403.20330
  7. [7]Zhaolin Gao, Joongwon Kim, Wen Sun, Thorsten Joachims, Sid Wang, Richard Yuanzhe Pang, and Liang Tan. Prompt curriculum learning for efficient LLM post-training. In The Fourteenth International Conference on Learning Representations, ICLR 2026, Rio de Janeiro, Brazil, April 23-27, 2026. proceedings.iclr.cc, 2026. 2arxiv.org/abs/2510.01135
  8. [8]Haiyang Guo, Fanhu Zeng, Ziwei Xiang, Fei Zhu, Da-Han Wang, Xu-Yao Zhang, and Cheng-Lin Liu. Hide-lava: Hierarchical decoupling for continual instruction tuning of multimodal large language model. In Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), ACL 2025, Vienna, Austria, July 27 - August 1, 2025, pages 13572–13586. Association for Computational Linguistics, 2025. 1arxiv.org/abs/2503.12941
  9. [9]Edward J. Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen. Lora: Low-rank adaptation of large language models. In The Tenth International Conference on Learning Representations, ICLR 2022, Virtual Event, April 25-29, 2022. OpenReview.net, 2022. 6, 2arxiv.org/abs/2106.09685
  10. [10]Jianheng Huang, Leyang Cui, Ante Wang, Chengyi Yang, Xinting Liao, Linfeng Song, Junfeng Yao, and Jinsong Su. Mitigating catastrophic forgetting in large language models with self-synthesized rehearsal. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), ACL 2024, Bangkok, Thailand, August 11-16, 2024, pages 1416–1428. Association for Computational Linguistics, 2024. 1, 2arxiv.org/abs/2403.01244
  11. [11]Minqi Jiang, Edward Grefenstette, and Tim Rocktäschel. Prioritized level replay. In Proceedings of the 38th International Conference on Machine Learning, ICML 2021, 18-24 July 2021, Virtual Event, pages 4940–4950. PMLR, 2021. 2arxiv.org/abs/2010.03934
  12. [12]Zixuan Ke, Haowei Lin, Yijia Shao, Hu Xu, Lei Shu, and Bing Liu. Continual training of language models for few-shot learning. In Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, EMNLP 2022, Abu Dhabi, United Arab Emirates, December 7-11, 2022, pages 10205–10216. Association for Computational Linguistics, 2022. 1
  13. [13]Khimiya Khetarpal, Matthew Riemer, Irina Rish, and Doina Precup. Towards continual reinforcement learning: A review and perspectives. J. Artif. Intell. Res., 75:1401–1476, 2022. 2arxiv.org/abs/2012.13490
  14. [14]Song Lai, Haohan Zhao, Rong Feng, Changyi Ma, Wenzhuo Liu, Hongbo Zhao, Xi Lin, Dong Yi, Min Xie, Qingfu Zhang, Hongbin Liu, Gaofeng Meng, and Fei Zhu. Reinforcement fine-tuning naturally mitigates forgetting in continual post-training. CoRR, abs/2507.05386, 2025. 1, 2arxiv.org/abs/2507.05386
  15. [15]Yifan Li, Yifan Du, Kun Zhou, Jinpeng Wang, Wayne Xin Zhao, and Ji-Rong Wen. Evaluating object hallucination in large vision-language models. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, 2023. 7arxiv.org/abs/2305.10355
  16. [16]Yi-Chen Li, Tian Xu, Yang Yu, Xuqin Zhang, Xiong-Hui Chen, Zhongxiang Ling, Ningjing Chao, Lei Yuan, and Zhi-Hua Zhou. Generalist reward models: Found inside large language models. CoRR, abs/2506.23235, 2025. 2, 3, 4arxiv.org/abs/2506.23235
  17. [17]Brandon Shuen Yi Loke, Filippo Quadri, Gabriel Vivanco, Maximilian Casagrande, and Saúl Fenollosa. Overcoming catastrophic forgetting in neural networks. CoRR, abs/2507.10485, 2025. 1, 2arxiv.org/abs/2507.10485
  18. [18]Pan Lu, Hritik Bansal, Tony Xia, Jiacheng Liu, Chunyuan Li, Hannah Hajishirzi, Hao Cheng, Kai-Wei Chang, Michel Galley, and Jianfeng Gao. MathVista: Evaluating Mathematical Reasoning in Visual Contexts. arXiv preprint arXiv:2310.02255, 2023. 7arxiv.org/abs/2310.02255
  19. [19]Lirui Luo, Guoxi Zhang, Hongming Xu, Rongqing Li, Cong Fang, and Lifeng Fan. Continual reasoning gym: Diagnosing and harnessing shared reasoning in continual RLVR. CoRR, abs/2608.18574, 2026. 2arxiv.org/abs/2608.18574
  20. [20]Mao-Lin Luo, Zhe-Xu Wang, Zi-Hao Zhou, Bo Ye, Jian Zhao, Min-Ling Zhang, and Tong Wei. RL forgets! towards continual policy optimization. CoRR, abs/2607.04364, 2026. 1, 2, 5, 6, 7, 3, 4arxiv.org/abs/2607.04364
  21. [21]Yixiu Mao, Yun Qu, Qi Wang, Heming Zou, and Xiangyang Ji. Dynamics-predictive sampling for active RL finetuning of large reasoning models. In The Fourteenth International Conference on Learning Representations, ICLR 2026, Rio de Janeiro, Brazil, April 23-27, 2026. proceedings.iclr.cc, 2026. 2arxiv.org/abs/2603.10887
  22. [22]Minesh Mathew, Dimosthenis Karatzas, and C. V. Jawahar. DocVQA: A dataset for VQA on document images. In Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision, pages 2200–2209, 2021. 7arxiv.org/abs/2007.00398
  23. [23]Roni Paiss, Ariel Ephrat, Omer Tov, Shiran Zada, Inbar Mosseri, Michal Irani, and Tali Dekel. Teaching CLIP to Count to Ten. arXiv preprint arXiv:2302.12066, 2023. 7arxiv.org/abs/2302.12066
  24. [24]Yun Qu, Qi Wang, Yixiu Mao, Heming Zou, Yuhang Jiang, Weijie Liu, Clive Bai, Kai Yang, Yangkun Chen, Saiyong Yang, and Xiangyang Ji. Small generalizable prompt predictive models can steer efficient RL post-training of large reasoning models. CoRR, abs/2602.01970, 2026. 2arxiv.org/abs/2602.01970
  25. [25]Rafael Rafailov, Archit Sharma, Eric Mitchell, Christopher D. Manning, Stefano Ermon, and Chelsea Finn. Direct preference optimization: Your language model is secretly a reward model. In Advances in Neural Information Processing Systems 36: Annual Conference on Neural Information Processing Systems 2023, NeurIPS 2023, New Orleans, LA, USA, December 10 - 16, 2023, 2023. 3arxiv.org/abs/2305.18290
  26. [26]David Rolnick, Arun Ahuja, Jonathan Schwarz, Timothy P. Lillicrap, and Gregory Wayne. Experience replay for continual learning. In Advances in Neural Information Processing Systems 32: Annual Conference on Neural Information Processing Systems 2019, NeurIPS 2019, December 8-14, 2019, Vancouver, BC, Canada, pages 348–358, 2019. 2
  27. [27]Tom Schaul, John Quan, Ioannis Antonoglou, and David Silver. Prioritized experience replay. In 4th International Conference on Learning Representations, ICLR 2016, San Juan, Puerto Rico, May 2-4, 2016, Conference Track Proceedings, 2016. 2arxiv.org/abs/1511.05952
  28. [28]John Schulman, Sergey Levine, Pieter Abbeel, Michael I. Jordan, and Philipp Moritz. Trust region policy optimization. In Proceedings of the 32nd International Conference on Machine Learning, ICML 2015, Lille, France, 6-11 July 2015, pages 1889–1897. JMLR.org, 2015. 2, 3arxiv.org/abs/1502.05477
  29. [29]Jonathan Schwarz, Wojciech Czarnecki, Jelena Luketina, Agnieszka Grabska-Barwinska, Yee Whye Teh, Razvan Pascanu, and Raia Hadsell. Progress & compress: A scalable framework for continual learning. In Proceedings of the 35th International Conference on Machine Learning, ICML 2018, Stockholmsmässan, Stockholm, Sweden, July 10-15, 2018, pages 4535–4544. PMLR, 2018. 2
  30. [30]Zhibang Shao, Peiyi Wang, Qihao Zhu, Runxin Xu, Junxiao Song, Mingchuan Zhang, Y. K. Li, Y. Wu, and Daya Guo. Deepseekmath: Pushing the limits of mathematical reasoning in open language models. CoRR, abs/2402.03300, 2024. 3, 6, 2arxiv.org/abs/2402.03300
  31. [31]Idan Shenfeld, Jyothish Pari, and Pulkit Agrawal. R1’s razor: Why online reinforcement learning forgets less. In The Fourteenth International Conference on Learning Representations, ICLR 2026, Rio de Janeiro, Brazil, April 23-27, 2026. proceedings.iclr.cc, 2026. 1, 2
  32. [32]Fan-Keng Sun, Cheng-Hao Ho, and Hung-Yi Lee. LAMOL: language modeling for lifelong language learning. In 8th International Conference on Learning Representations, ICLR 2020, Addis Ababa, Ethiopia, April 26-30, 2020. OpenReview.net, 2020. 2arxiv.org/abs/1909.03329
  33. [33]Ke Wang, Junting Pan, Weikang Shi, Zimu Lu, Mingjie Zhan, and Hongsheng Li. Measuring Multimodal Mathematical Reasoning with MATH-Vision Dataset. arXiv preprint arXiv:2402.14804, 2024. 7arxiv.org/abs/2402.14804
  34. [34]Xiao Wang, Tianze Chen, Qiming Ge, Han Xia, Rong Bao, Rui Zheng, Qi Zhang, Tao Gui, and Xuanjing Huang. Orthogonal subspace learning for language model continual learning. In Findings of the Association for Computational Linguistics: EMNLP 2023, Singapore, December 6-10, 2023, pages 10658–10671. Association for Computational Linguistics, 2023. 6, 2arxiv.org/abs/2310.14152
  35. [35]Zirui Wang, Mengzhou Xia, Luxi He, Howard Chen, Yi-tao Liu, Richard Zhu, Kaiqu Liang, Xindi Wu, Haotian Liu, Sadhika Malladi, Alexis Chevalier, Sanjeev Arora, and Danqi Chen. CharXiv: Charting Gaps in Realistic Chart Understanding in Multimodal LLMs. arXiv preprint arXiv:2406.18521, 2024. 7arxiv.org/abs/2406.18521
  36. [36]Maciej Wolczyk, Michal Zajac, Razvan Pascanu, Lukasz Kucinski, and Piotr Milos. Continual world: A robotic benchmark for continual reinforcement learning. In Advances in Neural Information Processing Systems 34: Annual Conference on Neural Information Processing Systems 2021, NeurIPS 2021, December 6-14, 2021, virtual, pages 28496–28510, 2021. 2
  37. [37]xAI. Grok-1.5 Vision Preview. https://x.ai/news/grok-1.5v, 2024. 7
  38. [38]Qiying Yu, Zheng Zhang, Ruofei Zhu, Yufeng Yuan, Xiaochen Zuo, Yu Yue, Weinan Dai, Tiantian Fan, Gaohong Liu, Juncai Liu, Lingjun Liu, Xin Liu, Haibin Lin, Zhiqı Lin, Bole Ma, Guangming Sheng, Yuxuan Tong, Chi Zhang, Mofan Zhang, Ru Zhang, Wang Zhang, Hang Zhu, Jinhua Zhu, Jiaze Chen, Jiangjie Chen, Chengyi Wang, Hongli Yu, Yuxuan Song, Xiangpeng Wei, Hao Zhou, Jingjing Liu, Wei-Ying Ma, Ya-Qin Zhang, Lin Yan, Yonghui Wu, and Mingxuan Wang. DAPO: an open-source LLM reinforcement learning system. In Advances in Neural Information Processing Systems 38: Annual Conference on Neural Information Processing Systems 2025, NeurIPS 2025, San Diego, CA, USA, December 2-7, 2025, 2025 / Mexico City, Mexico, November 30 - December 5, 2025, 2025. 3arxiv.org/abs/2503.14476
  39. [39]Xiang Yue, Tianyu Zheng, Yuansheng Ni, Yubo Wang, Kai Zhang, Shengbang Tong, Yuxuan Sun, Botao Yu, Ge Zhang, Huan Sun, Yu Su, Wenhu Chen, and Graham Neubig. MMMU-Pro: A More Robust Multi-discipline Multimodal Understanding Benchmark. arXiv preprint arXiv:2409.02813, 2024. 7arxiv.org/abs/2409.02813
  40. [40]Renrui Zhang, Dongzhi Jiang, Yichi Zhang, Haokun Lin, Ziyu Guo, Pengshuo Qiu, Aojun Zhou, Pan Lu, Kai-Wei Chang, Peng Gao, and Hongsheng Li. MathVerse: Does Your Multi-modal LLM Truly See the Diagrams in Visual Math Problems? arXiv preprint arXiv:2403.14624, 2024. 7arxiv.org/abs/2403.14624
  41. [41]Chujie Zheng, Shixuan Liu, Mingze Li, Xiong-Hui Chen, Bowen Yu, Chang Gao, Kai Dang, Yuqiong Liu, Rui Men, An Yang, Jingren Zhou, and Junyang Lin. Group sequence policy optimization. CoRR, abs/2507.18071, 2025. 3, 6arxiv.org/abs/2507.18071

Paper details

vlnc:W_nsemw88a
Version
v1 · latest
Published
Fresh — added to the Hub within the last 30 days
Added
License
CC BY 4.0
Usage

views · 5

downloads · 0

A view counts once per browser per day, a download each time the PDF is requested.

  • Computer Science

GPT-6, GPT-6.1

View statement
Looking for related papers…
Contents