betaFeedback

WorksComputer Science

From Visual Affinity to Learning Credit in Multimodal Reinforcement Learning

Abstract

Reinforcement learning with verifiable rewards (RLVR) improves multimodal reasoning, but uniformly assigning a single response-level advantage to all tokens overlooks the distinct roles of visual observations and subsequent reasoning. Visual affinity alone is insufficient: predictive states may align with both image and prompt-context representations, and upweighting valid visual observations can amplify their negative-advantage updates when later reasoning fails. To address this issue, we propose Outcome-Conditioned Visual Credit (OVC), a plug-in framework that separates visual-association measurement from token-level credit assignment and is adaptable to policy-optimization methods with uniform response-level advantage allocation. Relative Visual Encoding constructs a context-relative visual coordinate by contrasting each predictive state's affinity to image and prompt-context representations. Outcome-conditioned Evidence Calibration combines this coordinate with uncertainty-scaled predictive features and calibrates their contributions to outcome-discriminative evidence through mean contrasts between positive- and negative-advantage responses, scaled by regularized within-side variation. Visual association thus informs credit through observed outcome differences rather than directly determining update priority. Normalized Credit Allocation conditions the calibrated evidence on the response's advantage sign to produce positive token weights with unit global mean. These detached weights preserve advantage signs while modulating individual token-gradient contributions. OVC reuses existing rollouts without auxiliary reward models, process annotations, or inference-time modules. Instantiated through GSPO-token, OVC retains sequence-level clipping and consistently improves six-benchmark mean accuracy over GSPO across three models, with gains of 1.86-2.39 percentage points.

https://kawhiiiileo.github.io/OVC_PAGE/

https://github.com/kawhiiiileo/OVC

Introduction

Reinforcement learning with verifiable rewards (RLVR) [13, 29, 49, 55] has become a key approach to improving reasoning in large vision language models (LVLMs) [1, 27, 41]. It provides verifiable outcome supervision without dense intermediate-step annotations. Recent efforts have advanced multimodal reasoning via improved training data [3, 12, 16, 17, 19, 25, 46], policy optimization [9, 30, 33, 36, 42, 44, 48], and perceptual supervision [39] [37]. Yet outcome feedback remains coarser than the computations it guides. A single response may involve visual observations, symbolic transformations, and intermediate inferences, while its reward evaluates only the complete solution. This motivates finer-grained token credit assignment using visual signals.

Existing approaches [8, 11, 13, 40] derive token-level signals from visual dependence, representational similarity, predictive uncertainty, or region alignment. Although these signals capture different aspects of visual involvement, their conversion into update coefficients still requires deciding how the measured association should affect learning. With DelTA’s predictive-state representation [52], a state may align with both image and prompt representations, so absolute image affinity alone does not distinguish these associations. Moreover, a valid visual observation may occur in a negative-advantage response when later reasoning fails. Upweighting it would strengthen its negative update even if the error occurs elsewhere in the response. The central issue is therefore not merely how to measure visual association, but how to interpret it under outcome feedback.

Figure 1 illustrates this distinction. In (a), the highest image-affinity group does not attain the highest observed mean accuracy, suggesting a possible mismatch between affinity ranking and response quality. In (b), responses approximately matched on image affinity exhibit different outcome statistics across context-affinity levels. These observations motivate considering contextual association alongside image affinity, rather than treating the latter alone as sufficient for credit allocation. Together with the advantage-sign dependence discussed above, they motivate a two-stage principle: measuring visual relevance relative to context and calibrating it against outcome differences.

Image not extracted

Figure 1. Affinity and response correctness. We analyze 8 valid responses per randomly sampled question from a frozen Qwen3.5-9B trained with GSPO on Math77K, without correctness filtering. Affinity is siu=∣Ri∣−1∑t∈Ricos⁡(hit,giu)s_i^u = \lvert R_i\rvert^{-1}\sum_{t\in R_i}\cos(h_{it},g_i^u), where hith_{it} predicts token tt, giug_i^u is the mean-pooled image (u=imgu=\mathrm{img}) or non-image prompt (u=ctxu=\mathrm{ctx}) prototype from the same intermediate layer, and RiR_i indexes reasoning tokens excluding special tokens and the final answer. No coordinate rescaling or image–context subtraction is applied. (a) Mean accuracy over four within-question image-affinity rank groups (two responses each). Each faint point is a group mean from a question-level bootstrap resample, with shared indices across groups. Violins show the distributions; dark markers are original-sample means. (b) Joint correctness of within-question pairs roughly matched on image affinity. Rows and columns represent lower- and higher-context-affinity response outcomes, respectively; cells report percentages of all matched pairs.

To instantiate this principle, we propose Outcome-Conditioned Visual Credit (OVC). Its first component, Relative Visual Encoding (RVE), contrasts a predictive state’s cosine affinity to image and prompt-background representations under a shared diagonal scaling. The resulting coordinate captures relative visual association without directly prescribing an update magnitude. Outcome-conditioned Evidence Calibration (OEC) then combines this coordinate with predictive features and determines their evidential contributions from contrasts between positive- and negative-advantage responses. A regularized estimate of within-side diagonal variation calibrates the relative contribution of each coordinate. Consequently, with the statistics held fixed, the visual contribution depends on the outcome contrast and the variance scale, not on affinity alone, and equals zero when the visual reference means coincide.

This calibrated evidence is converted into policy updates through Normalized Credit Allocation (NCA). NCA produces positive, globally normalized token coefficients that modulate the original sequence advantage while preserving its sign. Applied through GSPO-token [55], these detached coefficients retain sequence-level importance ratios and clipping while adjusting the contribution of individual token-gradient terms. OVC introduces no auxiliary reward model, additional process or region supervision, or inference-time module. To clarify this reweighting, we characterize the resulting gradient and separate response-level weighting from within-response allocation, enabling controlled evaluation of token-coefficient correspondence.

Our contributions are threefold.

  • We introduce a context-relative visual coordinate that contrasts image affinity with prompt-background affinity, providing an explicit visual measure for credit estimation.

  • We develop an outcome-conditioned calibration method combining visual and predictive features via advantage-side contrasts and regularized within-side variation, with no fixed positive update preference for visual affinity.

  • We characterize how calibrated evidence enters policy updates and distinguish response-level reweighting from allocation within a response, establishing an analytical basis for controlled evaluation of token credit assignment.

Related Work

Policy Optimization and Perceptual Supervision for Multimodal Reasoning

RLVR improves reasoning through outcome verification. GRPO [29] employs group-relative advantage estimation, while GSPO [55] introduces sequence-level importance ratios for policy optimization. In multimodal settings, MM-Eureka [25] investigates large-scale training with rule-based rewards, and VL-ReThinker [34] improves training through selective sample replay and forced reflection. Another line of work refines supervision. PAPO [39] constructs an implicit perceptual objective through image perturbations, whereas VisualPRM [37] trains a separate process reward model to evaluate intermediate steps and support candidate solution selection. These studies primarily advance optimization mechanisms, training procedures, or supervision sources. Complementing these efforts, our work investigates how visual representations can inform fine-grained credit estimation under outcome feedback, rather than redesigning sampling strategies or introducing an independent process evaluator.

Visually Guided Fine-Grained Credit Assignment

VPPO [11] and PGPO [47] exploit changes in predictive distributions across visual conditions to filter token updates or redistribute advantages. PRPO [15] further combines strong and weak perturbations to distinguish visual dependence from prediction instability. VGPO [40] derives attention-related scores from visual prototypes and combines late-stage compensation with advantage reweighting within and across trajectories. KAWHI [8] connects structured visual evidence to RLVR through geometry-guided key-region selection, query-key alignment in vision-critical heads, and paragraph-level credit redistribution. These methods explicitly connect visual signals to policy updates. However, visual sensitivity, representational affinity, and region alignment do not, by themselves, determine a token’s contribution to outcome improvement. Our approach instead treats visual association as a feature requiring calibration. We construct a context-relative visual coordinate and determine its evidential contribution through statistical contrasts between positive- and negative-advantage responses and within-side variance, thereby separating the measurement of visual relevance from the allocation of learning credit.

Method

Visual relevance describes an association with the image, whereas learning credit determines the strength of a policy update. We treat the former as a measurement to be calibrated against outcome feedback, rather than as a quantity to maximize. We first characterize the calibration rule, then specify its estimator and policy objective.

Setting and notation. For each multimodal input x=(I,q)x = (I, q), a behavior policy πold\pi_{\mathrm{old}} samples GG responses. Response yi=(yit)t=1Tiy_i = (y_{it})_{t=1}^{T_i} receives a verifiable reward RiR_i and a group-relative advantage [29]

Ai=Ri−R‾g(i)sg(i)+ϵA,ci=sign⁡(Ai),(1)A_i = \frac{R_i - \overline{R}_{g(i)}}{s_{g(i)} + \epsilon_A}, \qquad c_i = \operatorname{sign}(A_i), \tag*{(1)}

where g(i)g(i) contains responses to the same input, R‾g(i)\overline{R}_{g(i)} and sg(i)s_{g(i)} are the corresponding reward mean and standard deviation, and ϵA>0\epsilon_A > 0. The sign cic_i indicates relative reward, not token correctness. An actor mini-batch contains BB nonempty responses and N=∑iTiN = \sum_i T_i valid tokens. Advantages are computed within each rollout group, whereas the reference statistics and coefficient normalization are aggregated over the actor mini-batch. All credit statistics are detached and fixed during its optimizer step. Sums below exclude padding.

A Calibration Principle for Visual Credit

Let zit=[vit;ρit]z_{it} = [v_{it}; \rho_{it}] combine a predictive feature vitv_{it} with a scalar visual measurement ρit\rho_{it}. For c∈{−1,+1}c \in\{-1, +1\}, let Tc={(i,t)∣ci=c, 1≤t≤Ti}T_c = \{(i,t) \mid c_i = c,\ 1 \leq t \leq T_i\} contain the valid response tokens on the corresponding advantage side. We initialize the reference means using advantage-magnitude weights and assume Mc>0M_c > 0 for both advantage sides

μc(0)=1Mc∑(i,t)∈Tc∣Ai∣zit,\mu_c^{(0)} = \frac{1}{M_c} \sum_{(i,t)\in T_c} |A_i|z_{it},
Mc=∑(i,t)∈Tc∣Ai∣.(2)M_c = \sum_{(i,t)\in T_c} |A_i|. \tag*{(2)}

These statistics are aggregated across the actor mini-batch, rather than estimated separately for each input. The reference-based calibration below assumes M+>0M_+ > 0 and M−>0M_- > 0. The two sides are defined by the sign of the sequence advantage, not by token-level correctness labels.

We write μ+\mu_+, μ−\mu_- generically for the reference means before or after refinement. Their refinement and the construction of a positive diagonal matrix DD encoding regularized within-side variation are specified in § 3.2. We calibrate a token feature against both advantage sides by comparing its squared distances to the two references under the shared metric D−1D^{-1}:

e(z)=∥z−μ−∥D−12−∥z−μ+∥D−12,(3)e(z) = \lVert z - \mu_-\rVert_{D^{-1}}^2 - \lVert z - \mu_+\rVert_{D^{-1}}^2, \tag*{(3)}

where ∥a∥D−12=aTD−1a\lVert a\rVert_{D^{-1}}^2 = a^\mathsf{T}D^{-1}a. Positive evidence indicates that zz is closer to the positive-advantage reference, while negative evidence indicates the opposite.

For fixed reference statistics, define the variance-calibrated direction

b∗=D−1Δμ,Δμ=μ+−μ−.b^* = D^{-1}\Delta\mu, \qquad\Delta\mu= \mu_+ - \mu_-.

Direct expansion of the distance contrast in Eq. (3) gives

e(z)=2(b∗)T(z−μ‾),μ‾=μ++μ−2.(4)e(z) = 2(b^*)^\mathsf{T}(z-\overline{\mu}), \qquad\overline{\mu} = \frac{\mu_+ + \mu_-}{2}. \tag*{(4)}

Each coordinate therefore contributes through its advantage-side mean contrast relative to its regularized within-side scale, rather than through its magnitude alone. For Δμ≠0\Delta\mu\ne0, Appendix § 8 further shows that b∗b^* maximizes the squared reference separation normalized by a regularized diagonal variation surrogate, up to nonzero rescaling. This provides a fixed-statistic characterization of the calibration rule, not an auxiliary objective optimized by the actor.

When does visual relevance influence credit? With D=diag⁡(Dh,dρ)D = \operatorname{diag}(D_h, d_\rho), Eq. (3) decomposes as

e(z)=eh(v)+2Δμρdρ(ρ−μ‾ρ).(5)e(z) = e_h(v) + 2\frac{\Delta\mu_\rho}{d_\rho}(\rho-\overline{\mu}_\rho). \tag*{(5)}

For fixed statistics, the direct visual term vanishes when Δμρ=0\Delta\mu_\rho= 0 and reverses its dependence on ρ\rho when the contrast changes sign. We use the signed evidence cie(zit)c_i e(z_{it}) to measure agreement with the response’s update direction. Thus, larger relative visual affinity receive no unconditional positive credit bonus. Appendix § 8.§ 3 derives the conditional sensitivities of the evidence and raw coefficients and distinguishes them from causal attribution. This motivates separating visual-relevance measurement from credit assignment.

Image not extracted

Figure 2. Overview of Outcome-Conditioned Visual Credit (OVC). An image–question pair produces a group of responses whose rewards yield group-relative advantages. For each response token, OVC combines relative image–context affinity with a prediction-aware representation, calibrates this evidence across advantage groups, and assigns normalized token weights to the policy update.

Estimating Relative Visual Evidence

Relative Visual Encoding. Let hit∈RHh_{it} \in\mathbb{R}^{H} be the state at a fixed intermediate layer aligned with the prediction of yity_{it}. Let HisH_{is} denote states at the same layer over the full input sequence. Image positions ViV_i and non-image prompt positions CiC_i define

giimg=1∣Vi∣∑s∈ViHis,gictx=1∣Ci∣∑s∈CiHis.(6)g_i^{\mathrm{img}} = \frac{1}{\lvert V_i\rvert}\sum_{s\in V_i} H_{is}, \qquad g_i^{\mathrm{ctx}} = \frac{1}{\lvert C_i\rvert}\sum_{s\in C_i} H_{is}. \tag*{(6)}

The context prototype is obtained from the original multimodal input, not a separate text-only forward. Let Si=diag⁡(σi)−1S_i = \operatorname{diag}(\sigma_i)^{-1}, where σi\sigma_i contains floored response-wise population standard deviations of hith_{it}. Applying the same scaling to the predictive state and both prototypes, RVE computes

ρit=cos⁡(Sihit,Sigiimg)−cos⁡(Sihit,Sigictx).(7)\rho_{it} = \cos(S_i h_{it}, S_i g_i^{\mathrm{img}}) - \cos(S_i h_{it}, S_i g_i^{\mathrm{ctx}}). \tag*{(7)}

This measures image affinity relative to prompt context. The mean used to estimate σi\sigma_i is not subtracted in the cosine calculation. Appendix § 7 gives the estimator and its metric interpretation.

We retain sampled-token prediction information with an uncertainty-scaled feature

vit=(1−pit)hit,zit=[vit;ρit],(8)v_{it} = (1-p_{it})h_{it}, \qquad z_{it} = [v_{it};\rho_{it}], \tag*{(8)}

where pit=πθ(yit∣xi,yi,<t)p_{it} = \pi_\theta(y_{it}\mid x_i,y_{i,<t}) is evaluated by the current actor. Only vitv_{it} is multiplied by 1−pit1-p_{it}. Using a state from the selected intermediate layer makes vitv_{it} a gradient-inspired predictive feature rather than an exact parameter gradient (Appendix § 7.§ 2).

Outcome-conditioned Evidence Calibration. Given μc(0)\mu_c^{(0)} and McM_c from Eq. (2), OEC estimates advantage-weighted within-side variation over the actor mini-batch:

qk=∑c∈{−1,+1}1Mc∑(i,t)∈Tc∣Ai∣(zitk−μc,k(0))2.(9)q_k = \sum_{c\in\{-1,+1\}}\frac{1}{M_c}\sum_{(i,t)\in T_c}\lvert A_i\rvert\left(z_{itk}-\mu_{c,k}^{(0)}\right)^2. \tag*{(9)}

For all H+1H+1 coordinates, let qˉ=H−1∑k=1Hqk\bar{q}=H^{-1}\sum_{k=1}^{H}q_k. We construct the diagonal scale matrix D=diag⁡(d)D=\operatorname{diag}(d) using shrinkage and a variance floor:

dk=ϵD+max⁡{(1−η)qk+ηqˉ, rmin⁡qˉ}.(10)d_k=\epsilon_D+\max\left\{(1-\eta)q_k+\eta\bar{q},\ r_{\min}\bar{q}\right\}. \tag*{(10)}

Here 0≤η≤10\le\eta\le1, rmin⁡≥0r_{\min}\ge0, and ϵD>0\epsilon_D>0, ensuring dk≥ϵDd_k\ge\epsilon_D and D≻0D\succ0. The visual coordinate is excluded from the shrinkage target qˉ\bar{q} but receives the same regularization. This is a diagonal scale estimate, not a full covariance matrix.

We perform one soft reference refinement. Let e(0)e^{(0)} use the initial references in Eq. (4), and define τc\tau_c as the unweighted population standard deviation of e(0)(zit)e^{(0)}(z_{it}) over TcT_c, with a strictly positive floor. We then compute

uit(0)=sigmoid⁡(cie(0)(zit)τci),(11)u_{it}^{(0)} = \operatorname{sigmoid}\left(\frac{c_i e^{(0)}(z_{it})}{\tau_{c_i}}\right), \tag*{(11)}
μc∗=∑τc∣Ai∣uit(0)zit∑τc∣Ai∣uit(0).\mu_c^* = \frac{\sum_{\tau_c} |A_i| u_{it}^{(0)} z_{it}}{\sum_{\tau_c} |A_i| u_{it}^{(0)}}.

The final score e∗e^* uses μ+∗,μ−∗\mu_+^*, \mu_-^* and the same DD; the final coefficient mapping also retains the pre-refinement τc\tau_c. This makes the visual coefficient in Eq. (5) outcome-dependent after refinement as well. Appendix D specifies the ordering and derives the soft assignment and reference update.

Credit Allocation and Policy Optimization

Normalized Credit Allocation. For 0<λmin⁡≤λmax⁡0 < \lambda_{\min} \le\lambda_{\max}, NCA maps signed evidence to positive raw coefficients and applies token-mean normalization over the actor mini-batch:

w^it=λmin⁡+(λmax⁡−λmin⁡)⋅sigmoid⁡(cie∗(zit)τci),\widehat{w}_{it} = \lambda_{\min} + (\lambda_{\max} - \lambda_{\min}) \cdot\operatorname{sigmoid}\left(\frac{c_i e^*(z_{it})}{\tau_{c_i}}\right),
Z=1N∑i,tw^it,wit=w^itZ,Z = \frac{1}{N}\sum_{i,t}\widehat{w}_{it}, \qquad w_{it} = \frac{\widehat{w}_{it}}{Z},
Ait′=sg⁡(wit)Ai.(12)A'_{it} = \operatorname{sg}(w_{it}) A_i. \tag*{(12)}

For Ai=0A_i = 0, we instead set w^it=λmin⁡\widehat{w}_{it} = \lambda_{\min} and include it in ZZ. Positive coefficients preserve advantage signs. Appendix E.1 gives their final bounds. Normalization is global rather than per response, so the estimator can adjust both relative response strength and allocation within a response. It fixes the global mean coefficient, but does not constrain the norm of the resulting policy gradient.

We apply these coefficients through GSPO-token. With Lit(θ)=log⁡πθ(yit∣xi,yi,<t)L_{it}(\theta) = \log\pi_\theta(y_{it} \mid x_i, y_{i,<t}), define

si(θ)=exp⁡[1Ti∑t(Lit(θ)−Lit(θold))],s_i(\theta) = \exp\left[\frac{1}{T_i}\sum_t \left(L_{it}(\theta) - L_{it}(\theta_{\mathrm{old}})\right)\right],
s~it(θ)=sg⁡(si(θ))exp⁡(Lit(θ)−sg⁡(Lit(θ))).(13)\widetilde{s}_{it}(\theta) = \operatorname{sg}(s_i(\theta)) \exp\left(L_{it}(\theta) - \operatorname{sg}(L_{it}(\theta))\right). \tag*{(13)}

Here sg⁡\operatorname{sg} stops gradients. Let s~itc=clip⁡(s~it,1−ϵl,1+ϵh)\widetilde{s}_{it}^{c} = \operatorname{clip}(\widetilde{s}_{it}, 1-\epsilon_l, 1+\epsilon_h). The surrogate to maximize is

JOVC(θ)=1B∑i1Ti∑tmin⁡{s~itcAit′,s~itcAit′}.(14)J_{\mathrm{OVC}}(\theta) = \frac{1}{B}\sum_i \frac{1}{T_i}\sum_t \min\left\{\widetilde{s}_{it}^{c} A'_{it}, \widetilde{s}_{it}^{c} A'_{it}\right\}. \tag*{(14)}

Although s~it=si\widetilde{s}_{it} = s_i numerically, its derivative remains token-specific. Appendices E.2 and E.3 derive this gradient and separate response-level weighting from token allocation. No gradients pass through the credit estimator. The policy loss can still update the entire actor, including its visual encoder. OVC reuses the sampled responses and adds no learnable head or inference-time module. Algorithm 1 and Appendix F describe the actual multi-pass training computation.

Experiment

Experimental Setup

Models and training data. We evaluate Qwen3-VL-4B-Instruct, Qwen3-VL-8B-Instruct [1], and Qwen3.5-9B [27]. For main results, the Qwen3-VL models are trained on ViRL39K [34], while Qwen3.5-9B uses Math77K, the 77,135 mathematical subset of DeepVision-103K [31]. Within each main-comparison setting, all methods share the same instruction-tuned initialization.

Benchmarks. For main results, we report MMMU [50], MathVerse [53], MathVision [35], We-Math [26], MathVista [22], and MMK12-val [25]. The first five use LMMs-Eval [51] with matched decoding and scoring within each model size. MMK12-val follows the fixed training-time validation protocol. Full settings are given in Appendix A.

Experimental Results

Main Results

We compare with the instruction-tuned Base model, GSPO [55], DeLTA [52], VPPO [11], and AT-RL [13]. OVC applies its detached weights through GSPO-token; uniform weights recover the GSPO gradient (Appendix E.2).

Consistent gains across model scales. Table 1 shows that OVC improves the six-task average over GSPO by +2.04, +2.39, and +1.86 points on 4B, 8B, and 9B, respectively, and exceeds Base by +4.80, +4.75, and +3.32 points. OVC also achieves the highest average among all compared RL methods on both Qwen3-VL scales, with the gain extending to Qwen3.5-9B trained on the larger-scale Math77K corpus.

MethodMMMUMathVisionWe-MathMathVistaMMK12-valMathVerseAvg.
Qwen3-VL-4B
Base46.5621.7175.5273.8354.7363.1255.91
GSPO [55]44.6724.5876.5274.8367.1464.2958.67+2.76
DelTA [52]44.3325.0076.0074.0074.5864.6759.76+3.85
VPPO [11]45.2123.1276.4374.1969.2364.2358.74+2.83
AT-RL [13]43.5222.7876.3374.2270.2263.4558.42+2.51
GSPO+OVC (Ours)46.8326.6477.0574.3774.5864.8060.71+4.80
Qwen3-VL-8B
Base48.7824.6780.2374.4064.4662.8459.23
GSPO [55]44.5627.3083.3376.1074.7363.5361.59+2.36
DelTA [52]49.2124.0185.4077.0377.0664.8262.92+3.69
VPPO [11]49.4327.6984.3676.2175.2964.0962.85+3.62
AT-RL [13]OOMOOMOOMOOMOOMOOMOOM
GSPO+OVC (Ours)49.5629.9383.4576.9778.9465.0563.98+4.75
Qwen3.5-9B (Larger-scale training: Math77K)
Base45.6736.1887.2479.2079.0977.3667.46
GSPO [55]46.7834.8787.1879.2087.7477.7468.92+1.46
GSPO+OVC (Ours)50.3340.4688.5180.3088.2876.8070.78+3.32

Table 1. Main results on six multimodal benchmarks. Scores (%) are averaged over four evaluation sampling seeds at fixed checkpoints. Bold and underlined values indicate the best and second-best distinct scores within each model group. Avg. is the six-benchmark mean; red superscripts denote gains over Base. Qwen3.5-9B uses larger-scale Math77K training data. OOM denotes out of memory.

Performance across multimodal benchmarks. The largest gains over GSPO occur on MMK12-val, MathVision, and MMMU: +7.44/+4.21 points on MMK12-val for 4B/8B, +5.59 on MathVision for 9B, and +2.16/+5.00/+3.55 on MMMU. OVC also reverses the Base-to-GSPO drop on MMMU for 4B/8B, reaching 46.83/49.56 versus Base scores of 46.56/48.78. On 9B, its only regression relative to GSPO is MathVerse (−0.94 points).

Comparison with non-visual discriminative credit. OVC exceeds DeLTA’s six-task average by +0.95 points on 4B and +1.06 on 8B. It leads DeLTA on five benchmarks and ties on MMK12-val at 4B. At 8B, its largest gains are MathVision (+5.92) and MMK12-val (+1.88), while DeLTA remains stronger on We-Math and nearly tied on MathVista.

Gains from strong initializations. These gains are measured on strong, already post-trained models. Prior exposure to related visual-mathematical problems may reduce the marginal gains from additional RL, although sample-level overlap with the original training corpora remains unverified. Under matched initializations, RL data, and rollout configurations, OVC’s gains are consistent with more effective use of existing representations and outcome feedback through finer-grained credit allocation. These results suggest that improved credit assignment remains beneficial even with substantial task-relevant competence.

Ablation Studies

Table 2 examines OVC on Qwen3-VL-8B across relative visual encoding, within-side variance calibration, and token-specific weight allocation. Full OVC achieves a four-task mean of 62.72, exceeding the uniform-credit GSPO reference (60.42) by 2.30 percentage points. Configuration definitions are provided in Appendix § 6.§ 2.

ConfigurationMathVisionMathVistaMathVerseMMK12-valAvg.Δ\Delta
Base24.6774.4062.8464.4656.59−6.13
GSPO (uniform credit)27.3076.1063.5374.7360.42−2.30
OVC (full)29.9376.9765.0578.9462.72—
(a) Relative visual encoding
No visual coordinate28.2174.9763.4675.4360.52−2.20
Absolute image affinity28.0375.6363.9876.3160.99−1.73
Separate affinity coordinates28.8975.5963.8374.2060.63−2.09
No response-wise scaling (Si=IS_i = I)28.8974.9963.7975.3260.75−1.97
(b) Within-side variance calibration
Identity metric (D=ID = I)28.1275.3464.2576.2160.98−1.74
Total-variance calibration28.0975.9164.1274.3360.61−2.11
No visual coordinate + D=ID = I28.6475.2363.6775.4160.74−1.98
(c) Token-specific weight allocation
Response-mean weights28.0675.3964.8876.3261.16−1.56
Within-response permutation27.5475.1462.4374.2859.85−2.87
Affinity-ranked weights28.5776.8664.3976.3461.54−1.18

Table 2. Ablation study of OVC on Qwen3-VL-8B. Base and GSPO are from Table 1. Scores are percentages; Avg. denotes the four-benchmark mean. Δ\Delta indicates the change from full OVC in percentage points. Best scores are bold. See Appendix § 6.§ 2 for details.

Relative visual encoding. Removing the visual coordinate, using absolute image affinity, or using separate image/context coordinates lowers the four-task mean by 2.20, 1.73, and 2.09 points, respectively. Separate coordinates retain both affinity measurements yet underperform their explicit difference, supporting the use of a context-relative visual representation. Removing response-wise scaling (Si=IS_i = I) further reduces the mean by 1.97 points, indicating that pre-cosine geometry contributes to evidence estimation.

Within-side variance calibration. The identity metric and total-variance calibration lower the mean by 1.74 and 2.11 points, respectively. Setting D=ID = I retains the advantage-side reference contrast but removes variance scaling. Adding the relative visual coordinate increases the mean from 60.52 to 62.72 with within-side calibration, compared with a smaller increase from 60.74 to 60.98 under the identity metric. Conversely, within-side calibration slightly reduces performance without the visual coordinate (60.74 to 60.52), but substantially improves it with the coordinate (60.98 to 62.72). These results yield a positive interaction of 1.96 points, suggesting that relative visual encoding and variance calibration are complementary rather than independently sufficient.

Token-specific weight allocation. Response-mean, within-response permutation, and affinity-ranked weights reduce the mean by 1.56, 2.87, and 1.18 points, respectively. Response averaging preserves each response’s mean coefficient, while permutation additionally preserves its complete weight multiset but disrupts token–weight correspondence. Their deficits support the importance of token-specific allocation beyond response-level reweighting or nonuniform coefficients alone. Affinity-ranked weights also underperform full OVC, suggesting that directly favoring higher visual affinity cannot reproduce outcome-conditioned allocation. Although 9/10 ablations remain numerically above GSPO, none matches full OVC, with within-response permutation even falling below the uniform-credit reference.

Chart-centric Reasoning

We additionally evaluate OVC on chart-centric reasoning with Qwen3-VL-4B-Instruct. After SFT on 100K ChartVerse examples [20], the resulting model serves as Base. GSPO and GSPO+OVC start from Base and train for 72 rollout steps on the same 18,280-question subset of ChartVerse-RL-40K. Training and evaluation details are provided in Appendices F.2.1 and A.6.

As shown in Table 3, OVC achieves the best performance on 5/6 benchmarks [10, 24, 32, 38, 43, 45], improving the average score from 54.13 to 55.36 (+1.23 points) over GSPO and exceeding Base by 3.64 points. The largest gains over GSPO occur on ChartQA-Pro (+3.62) and ChartMuseum (+1.85), while ChartX-QA shows a slight decrease (-0.25).

MethodChart QA-ProChar XiVChart MuseumChartX-QAEvo Chart-QAChart BenchAvg.
Base37.6962.6636.8363.6947.3962.0851.72
GSPO40.5663.4741.5766.5547.6065.0354.13 +2.41^{+2.41}
OVC (Ours)44.1864.7743.4266.3048.2865.1855.36 +3.64^{+3.64}

Table 3. Chart-centric reasoning on Qwen3-VL-4B. Scores (%) are averaged over three evaluation sampling seeds. Avg. is the six-benchmark mean. Bold and underlined values denote the best and second-best results. Red superscripts indicate gains over Base.

These results support the applicability of OVC to chart-focused reinforcement learning under matched initialization, training data, and rollout budgets.

Extension to Other Policy Optimizers

To examine whether OVC is applicable beyond GSPO, we extend it to DAPO and GRPO, which also use shared response-level advantages for token updates. Experiments are conducted on Qwen2.5-VL-7B [2] trained with ViRL39K.

As shown in Table 4, OVC increases the average of 6 tasks by 1.73 and 0.76 points for DAPO and GRPO, respectively. Improvements are observed on 5/6 benchmarks for DAPO and all six for GRPO. These results illustrate the applicability of OVC to different policy optimization objectives, rather than restricting token-level credit reweighting to GSPO.

MethodMMMUMath VisionWe-MathMath VistaMMK12-valMath VerseAvg.
DAPO50.0014.8069.5470.6776.8845.5854.58
DAPO+OVC52.7820.3970.9271.3076.5845.8956.31
GRPO51.1114.8069.3770.2074.8845.0054.23
GRPO+OVC52.2215.1370.6370.2775.5046.1754.99

Table 4. Extending OVC to alternative policy optimizers. Results on Qwen2.5-VL-7B trained on ViRL39K. Avg. is the mean across six benchmarks. Bold marks the better result in each pair.

Zero-shot Generalization

To assess transfer beyond the main evaluation suite, we evaluate two groups of tasks: chart understanding and VQA, and scientific and spatial reasoning. These transfer evaluations use the ViRL39K- and Math77K-trained checkpoints from the main comparisons and their corresponding instruction-tuned Base checkpoints.

General-purpose VQA. We evaluate ChartQA [23], CharXiv [38], A-OKVQA [28], and SimpleVQA [4] across three model scales. As shown in Figure 3, OVC improves the four-task mean over GSPO by 0.14, 0.72, and 0.31 points on 4B, 8B, and 9B, respectively, while consistently exceeding the corresponding Base. The largest improvement over

Image not extracted

Figure 3. Zero-shot transfer gains over Base. Bars show score changes relative to the corresponding instruction-tuned Base model. Avg. is the mean over four VQA benchmarks.

GSPO occurs on 8B, where OVC achieves a 0.69-point gain over Base, compared with a 0.03-point decrease for GSPO. Benchmark descriptions, score aggregation, and absolute results are provided in Appendix § 6.5.1.

Scientific and spatial reasoning. We further examine cross-domain transfer using Qwen3.5-9B trained on Math77K, evaluating five benchmarks [6, 7, 14, 21, 54] covering scientific knowledge and spatial reasoning. As shown in Table 5, OVC outperforms GSPO on all five benchmarks, raising the average score from 62.95 to 63.91. The largest gains are observed on EmbSpatial (+1.65 points) and SuperChem (+1.10 points). OVC also exceeds the Base model by 2.34 points on average. Together, these results suggest that the benefits of outcome-conditioned token reweighting extend beyond the visual-mathematical evaluation domain. Benchmark descriptions and score aggregation are provided in Appendix § 6.5.2.

MethodScienceQARxnBenchSUPER​ChemEmbSpatialERQAAvg.
Qwen3.5-9B92.8887.0810.2073.1944.5061.57
GSPO93.1588.3212.3075.8045.2062.95
OVC (Ours)93.8989.0513.4077.4545.7563.91

Table 5. Zero-shot transfer to scientific and spatial benchmarks. GSPO and OVC are trained on Math77K using Qwen3.5-9B. Avg. denotes the mean score across five benchmarks. The best results are highlighted in bold.

Visualizations

Distribution of token credit weights. Figure 4 provides a qualitative visualization of the token-credit patterns produced by OVC and DelTA. OVC correctly converts the bearing to the 65∘65^\circ interior angle and obtains d=11.8d = 11.8 km. Its decisive reasoning span receives relatively high credit. In contrast, DelTA uses 25∘25^\circ as the triangle angle and obtains 25.325.3 km, while assigning relatively low credit to the first erroneous reasoning span.

Image not extracted

Figure 4. Qualitative visualization of within-response token-credit patterns. OVC correctly converts the bearing to the 65∘65^\circ triangle angle and obtains d=11.8d = 11.8 km, whereas DelTA first errs by using 25∘25^\circ as the triangle angle and obtains 25.325.3 km. Colors indicate the final normalized token-credit weights witw_{it}, recomputed offline from the corresponding checkpoints. Red boxes mark the key angle conversions and final answers. The OVC and DelTA responses have positive and negative group-relative advantages, respectively.

This example illustrates how OVC can better align token-level credit with visually grounded reasoning steps.

Training dynamics. Figure 5 compares OVC and GSPO during Qwen3.5-9B training on Math77K. OVC generally achieves higher training reward and MMK12-val performance over the training trajectory. The consistent advantage in both optimization reward and validation performance provides further evidence for the effectiveness of OVC beyond the final-checkpoint comparison.

Image not extracted

Figure 5. Training dynamics on Qwen3.5-9B.

Efficiency

Figure 6 shows the training-time cost breakdown. The OVC-specific credit-estimation passes account for 189 s, or 5.16% of the 3663 s total step time. OVC adds no learnable parameters or inference-time computation; its additional training cost comes from four no-gradient statistics passes. Implementation details are given in Appendix § 11.

Image not extracted

Figure 6. Training-time cost breakdown of OVC.

Conclusion

We introduced OVC to address a central distinction in multimodal RLVR: visual association does not, by itself, determine learning credit under response-level feedback. OVC measures visual affinity relative to context, combines it with predictive features, and calibrates their contributions through advantage-side contrasts and regularized within-side variation. The resulting evidence modulates token-specific gradient contributions through positive, normalized weights, without changing the original advantage signs or sequence-level clipping. Our analysis further distinguishes response-level reweighting from within-response allocation, clarifying how calibrated coefficients affect individual token gradients. Experiments across three models and six benchmarks show improved average accuracy over GSPO without additional process supervision or inference-time modules. Overall, OVC connects visual representations with fine-grained policy optimization by treating visual association as outcome-conditioned evidence rather than direct update priority.

Evaluation protocol

Benchmarks and evaluation settings

For the main comparisons, we use the official MMK12 test split as a fixed validation set and report the final scheduled validation score. We evaluate MMMU, MathVerse, MathVision, We-Math, and MathVista after training using LMMs-Eval [51]. All compared checkpoints within each main-comparison setting use the same benchmark splits and evaluation procedures. Table 6 summarizes this protocol. The chart-centric evaluation is described in Appendix § 6.6.

BenchmarkSplitSamplesPrompting protocolScoring protocol
MMMUvalidation900multiple-choice promptingdeterministic accuracy
MathVersetestmini3,940zero-shot direct queryDeepSeek V4 correctness judge
MathVisiontestmini304original questionDeepSeek V4 correctness judge
We-Mathtestmini1,740reasoning with explicit final answerdeterministic Loose accuracy
MathVistatestmini1,000three zero-shot prompting settingsanswer extraction + deterministic scoring; mean over settings
MMK12-valtest2,000training validation promptvalidation accuracy

Table 6. Benchmark protocol used for external evaluation.

MMMU [50]. We evaluate the 900-example validation split. Following the public Qwen protocol, all images are placed before the textual query. The prompt contains the available hint, question, and answer choices when present, and asks the model to select the correct option for multiple-choice items. Responses are scored by the benchmark’s deterministic accuracy evaluator.

MathVerse [53]. We evaluate the 3,940-example testmini split with the benchmark’s zero-shot query that does not add an explicit chain-of-thought instruction or few-shot demonstrations. Benchmark-defined termination strings are preserved. Responses are judged with DeepSeek V4 as described in Sec. A.4.

MathVision [35]. We evaluate the 304-example testmini split. The textual input is the dataset question itself, matching the public Qwen evaluation convention, with no additional answer-format instruction. Responses are judged by the same DeepSeek V4 correctness protocol used for MathVerse.

We-Math [26]. We evaluate the 1,740-example testmini split. The prompt presents the image, question, and answer options and requests an explicit reasoning trace followed by a clearly delimited final answer. We report accuracy from the benchmark’s deterministic reasoning evaluator.

MathVista [22]. We evaluate the 1,000-example testmini split under three zero-shot prompting settings. The CoT setting asks for a reasoned answer followed by the required final value or option. The Solution setting explicitly prompts the model to produce a solution before the final answer. The Format setting requests only an answer in the format required by the target type, such as an option letter, integer, floating-point value with the prescribed precision, or list. Image captions and OCR annotations are not provided. We report the arithmetic mean over the three settings as the MathVista score.

MMK12-val [25]. MMK12 is an independently collected multimodal K–12 reasoning benchmark covering mathematics, physics, chemistry, and biology. We use its official 2,000-example test split as a fixed validation set during RL training. The same split and evaluation procedure are used for all compared methods within each model size. MMK12 does not participate in policy optimization, checkpoint selection, or early stopping. In Table 1, we report the score from the final scheduled validation event rather than selecting the best score observed during training.

Ablation configurations

Each variant modifies the indicated representation, metric, or allocation rule while retaining the remaining OVC procedure. Statistics are computed from the current actor and rollouts of the corresponding run, rather than copied from a separately trained full-OVC model. Representation and metric variants re-estimate their reference means, margin temperatures, refined references, and normalization factor. Within each update, the selected metric and pre-refinement temperatures remain fixed during the single reference refinement. All operations use valid response tokens and exclude padding.

Visual representations. Define the scaled affinities

aitu=cos⁡(Sihit,Sigu),u∈{img,ctx}.(15)a_{it}^{u} = \cos(S_i h_{it}, S_i g^{u}), \qquad u \in\{\mathrm{img}, \mathrm{ctx}\}. \tag*{(15)}

Full OVC uses ρit=aitimg−aitctx\rho_{it} = a_{it}^{\mathrm{img}} - a_{it}^{\mathrm{ctx}}. The representation variants replace the joint feature by

No visual coordinate:zit=vit,Absolute image affinity:zit=[vit;aitimg],Separate affinity coordinates:zit=[vit;aitimg;aitctx].(16)\begin{aligned} \text{No visual coordinate:}\quad&z_{it} = v_{it},\\ \text{Absolute image affinity:}\quad&z_{it} = [v_{it}; a_{it}^{\mathrm{img}}],\\ \text{Separate affinity coordinates:}\quad&z_{it} = [v_{it}; a_{it}^{\mathrm{img}}; a_{it}^{\mathrm{ctx}}]. \tag*{(16)} \end{aligned}

The separate-coordinate variant retains the same response-wise SiS_i for both affinities and uses an (H+2)(H+2)-dimensional diagonal metric. Its shrinkage target remains the mean variation of the first HH predictive coordinates; both appended affinity coordinates are excluded from that target but receive the same regularization in Eq. (10). “Absolute” means without context subtraction, not an absolute value. The Si=IS_i = I variant removes scaling only from the two cosine measurements and retains their difference. In every representation variant, vit=(1−pit)hitv_{it} = (1-p_{it})h_{it} is unchanged, and appended affinity coordinates are not multiplied by 1−pit1-p_{it}.

Metric variants. The identity variant sets D=ID = I over the entire selected feature space while retaining advantage-side reference estimation, soft refinement, and sign-conditioned allocation. The no-visual identity variant additionally sets zit=vitz_{it} = v_{it}. For total-variance calibration, the side-normalized weights in Eq. (9) are retained, but both sides are centered at their common midpoint:

μˉ(0)=μ+(0)+μ−(0)2,\bar{\mu}^{(0)} = \frac{\mu_{+}^{(0)} + \mu_{-}^{(0)}}{2},
qktot=∑c∈{−1,+1}1Mc∑(i,t)∈Tc∣Ai∣(zitk−μk(0))2=qk+12(μ+,k(0)−μ−,k(0))2.(17)\begin{aligned} q_{k}^{\mathrm{tot}} = \sum_{c \in\{-1,+1\}} \frac{1}{M_c} \sum_{(i,t) \in\mathcal{T}_c} \lvert A_i \rvert\left(z_{itk} - \mu_k^{(0)}\right)^2 \\ &= q_k + \frac{1}{2}\left(\mu_{+,k}^{(0)} - \mu_{-,k}^{(0)}\right)^2. \tag*{(17)} \end{aligned}

This is twice the total variance of an equally weighted mixture of the two advantage-side distributions, matching the summed-side scale of Eq. (9). It changes the centering without changing the relative side weights. The resulting qktotq_k^{\mathrm{tot}} and qˉtot=H−1∑k=1Hqktot\bar{q}^{\mathrm{tot}} = H^{-1}\sum_{k=1}^{H}q_k^{\mathrm{tot}} replace qkq_k and qˉ\bar{q} in Eq. (10), with the same η\eta, rmin⁡r_{\min}, and ϵD\epsilon_D.

Allocation variants. At each actor update, these controls first compute the standard OVC normalized weights witw_{it} for that run, then modify their assignment before evaluating Eq. (14). Response-mean allocation replaces every coefficient in response ii by

witmean=wˉi=1Ti∑s=1Tiwis.(18)w_{it}^{\mathrm{mean}} = \bar{w}_i = \frac{1}{T_i}\sum_{s=1}^{T_i}w_{is}. \tag*{(18)}

Within-response permutation uses

witperm=wi,Πi(t).(19)w_{it}^{\mathrm{perm}} = w_{i,\Pi_i(t)}. \tag*{(19)}

where Πi\Pi_i is sampled uniformly over permutations of valid token positions, independently for each response at each actor update, and held fixed throughout that update.

Affinity-ranked allocation reassigns the same normalized weight multiset according to relative affinity ρit\rho_{it}. Let ti,1,…,ti,Tit_{i,1}, \ldots, t_{i,T_i} order the token positions by nondecreasing ρit\rho_{it}, with ties resolved by token position, and let wi,(1)≤⋯≤wi,(Ti)w_{i,(1)} \le\cdots\le w_{i,(T_i)} be the sorted weights. Then

wi,ti,kaff=wi,(k),k=1,…,Ti.(20)w_{i,t_{i,k}}^{\mathrm{aff}} = w_{i,(k)}, \qquad k = 1,\ldots,T_i. \tag*{(20)}

The ordering is independent of cic_i: higher relative affinity receives a no-smaller coefficient on either advantage side. Only the assignment is changed; coefficient values remain those generated by OVC.

All three transformations preserve each response’s total weight and hence the global token mean. Permutation and affinity ranking additionally preserve the complete within-response weight multiset. No second normalization is applied, and the selected coefficients remain detached in Ait′,abl=sg⁡(witabl)AiA_{it}^{\prime,\mathrm{abl}} = \operatorname{sg}(w_{it}^{\mathrm{abl}})A_i. Zero-advantage responses retain their constant normalized coefficients. These preservation properties hold within a fixed update, not across the evolving trajectories of separately trained variants (Appendix § 10.§ 3).

Generation protocol

The following decoding configuration applies to the main benchmark evaluations. Responses are sampled with temperature T=0.7T = 0.7, nucleus-sampling threshold p=0.8p = 0.8, and top-kk truncation with k=20k = 20. The presence penalty is 1.5 and the repetition penalty is 1.0. The default evaluation seed is 3407; additional replication seeds are reported with the corresponding results. The model’s native thinking mode is disabled, although a benchmark prompt may still explicitly request a written reasoning trace.

The context window is 32,768 tokens, with at most 16,384 generated tokens per response. MathVision uses an image-area range of 602,112–4,014,080 pixels, and MMMU uses 1,003,520–4,014,080 pixels. The remaining benchmarks use the standard model processor settings. Benchmark-defined stop sequences are retained; otherwise generation terminates at the model’s end-of-sequence token or the generation limit.

Benchmark-authored termination strings are retained, but the evaluator does not add an implicit double-newline stopping rule to tasks that do not define one. This is important for long-form mathematical responses, where stopping at the first blank line can truncate a valid multi-paragraph solution.

Scoring protocol

For the main benchmarks, we use DeepSeek V4 [5] where a language-model judge is required. The judging instructions and parsing rules are held fixed across all compared checkpoints.

MathVerse and MathVision. For each response, DeepSeek V4 receives the textual question, reference answer, and model response and returns a binary correctness decision. The judge assesses answer equivalence while allowing superficial differences in formatting, LaTeX notation, capitalization, and spacing. Numerical values, units, option identity, and required answer components must agree with the reference answer.

MathVista. DeepSeek V4 extracts the final answer from the model response using the question as context. The extracted answer is normalized and compared with the reference by the deterministic MathVista evaluator according to the target answer type, including multiple-choice and numerical answers. The reported MathVista score is the arithmetic mean over the three prompting settings.

MMMU and We-Math. MMMU is scored by its native deterministic accuracy evaluator. We-Math is scored using the benchmark’s deterministic reasoning scorer.

Zero-shot generalization

We assess transfer beyond the main evaluation suite in two groups: chart understanding and visual question answering (VQA), and scientific and spatial reasoning. We compare the corresponding instruction-tuned Base models with GSPO and OVC checkpoints without additional target-benchmark adaptation. Within each model–benchmark comparison, task-specific prompting, decoding, and scoring settings are held fixed across methods.

General-purpose VQA

We evaluate ChartQA, CharXiv, A-OKVQA, and SimpleVQA on Qwen3-VL-4B, Qwen3-VL-8B, and Qwen3.5-9B, using the final GSPO and OVC checkpoints for each model. The first two benchmarks focus on interpreting charts, whereas the latter two assess visual question answering that involves commonsense or factual knowledge.

ChartQA [23]. ChartQA evaluates question answering over charts, combining the identification of visual elements and numerical values with arithmetic and logical operations. It probes whether a model can use chart content to answer questions that go beyond recognizing single labels or marks.

CharXiv [38]. CharXiv evaluates understanding of realistic charts collected from scientific papers. Its descriptive questions examine basic chart elements, while its reasoning questions require synthesizing information across visual elements. We report the arithmetic mean of the descriptive and reasoning scores as the CharXiv score.

A-OKVQA [28]. A-OKVQA evaluates visual question answering that requires commonsense and world knowledge in addition to image understanding. Its questions concern depicted scenes and require linking visual observations with knowledge beyond what is explicitly visible in the image.

SimpleVQA [4]. SimpleVQA evaluates multimodal factuality through short-answer visual questions with verifiable reference answers. It covers diverse topics and tests the combination of visual recognition and factual knowledge needed to answer these questions.

Reporting. Table 7 reports the absolute scores. The four-task Avg. is the unweighted arithmetic mean of ChartQA, CharXiv, A-OKVQA, and SimpleVQA, with CharXiv counted once after averaging its two subtask scores. Figure 3 shows the corresponding score changes relative to the Base model at each scale.

MethodChartQACharXivA-OKVQASimpleVQAAvg.
Qwen3-VL-4B
Base83.8458.0484.8925.9363.18
GSPO [55]84.1259.6885.4125.9863.80
GSPO+OVC (Ours)84.1660.1286.0325.4363.94
Qwen3-VL-8B
Base85.2465.6887.9527.7066.64
GSPO [55]84.6866.4488.3027.0166.61
GSPO+OVC (Ours)85.6566.9888.6128.0667.33
Qwen3.5-9B
Base85.3675.0987.2527.7568.86
GSPO [55]85.5674.9687.6927.7568.99
GSPO+OVC (Ours)86.4375.5487.4227.8069.30

Table 7. Per-task zero-shot transfer results. CharXiv is the mean of its reasoning and descriptive scores. The corresponding gains over Base are visualized in Figure 3.

Scientific and spatial reasoning

For this group, we compare the Qwen3.5-9B Base model with GSPO and OVC checkpoints trained on Math7K. The evaluation covers ScienceQA, RxnBench, SUPERChem, EmbSpatial, and ERQA, with results reported in Table 5.

ScienceQA [21]. ScienceQA contains multiple-choice questions from elementary and high-school curricula across natural, social, and language science. Questions may include image or textual context. It assesses curriculum-level knowledge and reasoning from the question and its available context.

RxnBench [14]. RxnBench evaluates chemical reaction understanding using material from scientific literature. Its questions probe the interpretation of reaction schemes, including molecular structures, reaction conditions, reagent roles, and chemical transformations, connecting domain-specific visual notation with chemical knowledge and reasoning.

SUPERChem [54]. SUPERChem is an expert-curated chemistry benchmark emphasizing multi-step problem solving across chemical subfields. The benchmark provides text-only and multimodal problem formats, as well as expert-authored solution paths that support analysis of reasoning quality in addition to final-answer performance. It complements RxnBench with a broader test of challenging chemistry problem solving.

EmbSpatial-Bench [6]. EmbSpatial-Bench, abbreviated as EmbSpatial in the result table, evaluates spatial understanding from egocentric views of three-dimensional environments. Its multiple-choice questions cover above, below, left, right, close, and far relationships, probing relative object positions and depth ordering from visual observations.

ERQA [7]. ERQA evaluates embodied reasoning through image-based multiple-choice questions. It covers spatial, action, and trajectory reasoning, state estimation, pointing, multi-view reasoning, and task reasoning. Together with EmbSpatial-Bench, it tests embodied-relevant visual reasoning through question answering rather than closed-loop robot control.

Reporting. The Avg. in Table 5 is the unweighted arithmetic mean of the five benchmark scores, rather than an accuracy pooled over their individual examples.

Chart-centric evaluation

We evaluate six chart benchmarks using the tasks and scoring rules summarized in Table 8.

Benchmark / taskSplit or setQuestionsScoring rule
ChartQA-Pro [24]Evaluation set1,948Native scorer, including numerical tolerance and textual ANLS where applicable.
CharXiv-RQ [38]Validation1,000Benchmark-specific reasoning rubric.
CharXiv-DQ [38]Validation4,000Benchmark-specific descriptive rubric.
ChartMuseum [32]Test1,000Benchmark-specific answer-comparison rubric.
ChartX-QA [43]Test1,152QA scoring rubric, including 5% numerical tolerance.
EvoChart-QA [10]Evaluation set1,250Exact matching or 5% numerical tolerance, depending on question type.
ChartBench [45]Evaluated subset2,100Final-answer extraction followed by numerical tolerance or Yes/No matching.

Table 8. Chart-centric evaluation protocol. The six benchmarks comprise seven evaluated tasks.

Generation. For each checkpoint, we generate one answer per question with temperature 1.0 and top-pp 1.0. The maximum response length is 16,384 tokens within a 32,768-token context window. Image areas range from 802,816 to 3,211,264 pixels. We use benchmark-specific prompts with native thinking mode disabled.

Relative Visual Features

Response-wise scaling and geometric interpretation

For coordinate kk, the response-wise scale in Eq. (7) is

hˉik=1Ti∑thitk,σik=max⁡{1Ti∑t(hitk−hˉik)2,ϵσ2}(21)\bar{h}_{ik} = \frac{1}{T_i}\sum_t h_{itk}, \qquad \sigma_{ik} = \sqrt{\max\left\{\frac{1}{T_i}\sum_t\left(h_{itk}-\bar{h}_{ik}\right)^2,\epsilon_\sigma^2\right\}} \tag*{(21)}

Here ϵσ>0\epsilon_\sigma> 0, and the variance is the population variance over valid response tokens. The same Si=diag⁡(σi)−1S_i = \operatorname{diag}(\sigma_i)^{-1} transforms hith_{it} and both prototypes, without subtracting hˉi\bar{h}_i in the cosine. Only ρit\rho_{it} uses this transform. The predictive feature remains (1−ρit)hit(1-\rho_{it})h_{it}.

Let Wi=SiTSiW_i = S_i^{\mathsf{T}}S_i. For nonzero transformed vectors and inactive numerical cosine floors,

cos⁡(Sia,Sib)=aTWibaTWiabTWib.(22)\cos(S_i a,S_i b) = \frac{a^{\mathsf{T}}W_i b}{\sqrt{a^{\mathsf{T}}W_i a}\sqrt{b^{\mathsf{T}}W_i b}}. \tag*{(22)}

Thus both affinities use the same response-dependent diagonal metric. Defining a^=Sia/∥Sia∥2\hat{a}=S_i a/\lVert S_i a\rVert_2 gives

ρit=h^itT(g^iimg−g^ictx),∣ρit∣≤2.(23)\rho_{it} = \hat{h}_{it}^{\mathsf{T}}\left(\hat{g}_i^{\mathrm{img}}-\hat{g}_i^{\mathrm{ctx}}\right), \qquad \lvert\rho_{it}\rvert\le2. \tag*{(23)}

The bound follows from Cauchy–Schwarz and the triangle inequality. Identical normalized prototypes yield zero relative affinity. Standard norm clamping also retains the bound since each clamped cosine remains in [−1,1][-1,1].

This contrast does not remove all contextual information or isolate a causal image effect. Prompt states can already depend on the image. Image-position states follow causal visibility and need not encode question tokens appearing later in the prompt. The prototypes share a layer basis, not necessarily an identical information set. RVE provides one scalar per predictive state and does not localize image regions.

Predictive feature and gradient interpretation

Let the actual output-head input be ftf_t, with logits aj=WjTfta_j = W_j^\mathsf{T} f_t and probabilities pj=softmax⁡(a)jp_j = \operatorname{softmax}(a)_j. Holding ftf_t fixed,

log⁡py=WyTft−log⁡∑jexp⁡(WjTft),(24)\log p_y = W_y^\mathsf{T}f_t - \log\sum_j \exp(W_j^\mathsf{T}f_t), \tag*{(24)}
∂log⁡py∂Wy=(1−py)ft.\frac{\partial\log p_y}{\partial W_y} = (1-p_y)f_t.

This output-row identity motivates the uncertainty-scaled feature. In Eq. (8), replacing ftf_t with a state from the selected intermediate layer retains the factor 1−py1-p_y but not the exact gradient identity. The final probabilities and intermediate states correspond to different stages of the actor. Accordingly, vitv_{it} is a gradient-inspired predictive feature, not a full-parameter score vector. The selected intermediate layer is fixed.

Properties of Outcome-conditioned Calibration

All results in this section condition on fixed reference means and a positive diagonal matrix DD. They characterize the estimator at these statistics, rather than the evolution of the complete RL procedure. The reference estimators require M+>0M_+ > 0 and M−>0M_- > 0. Under 0≤η≤10 \le\eta\le1, rmin⁡≥0r_{\min} \ge0, and ϵD>0\epsilon_D > 0, Eq. (10) ensures dk≥ϵD>0d_k \ge\epsilon_D > 0 because qk≥0q_k \ge0, and hence guarantees D≻0D \succ0. A missing advantage side is outside the domain of this two-reference estimator; a denominator floor alone does not define the missing empirical reference.

Optimal coefficients and normalized contrast

Proposition 1 (Regularized contrast). Let D≻0D \succ0 and Δμ=μ+−μ−\Delta\mu= \mu_+ - \mu_-. The unique maximizer of

FD(b)=bTΔμ−12bTDb(25)F_D(b) = b^\mathsf{T}\Delta\mu- \frac{1}{2}b^\mathsf{T}Db \tag*{(25)}

is b∗=D−1Δμb^* = D^{-1}\Delta\mu. If Δμ≠0\Delta\mu\ne0, the nonzero scalar multiples of b∗b^* are exactly the maximizers of

RD(b)=(bTΔμ)2bTDb,b≠0,(26)R_D(b) = \frac{(b^\mathsf{T}\Delta\mu)^2}{b^\mathsf{T}Db}, \qquad b \ne0, \tag*{(26)}
max⁡bRD(b)=ΔμTD−1Δμ.\max_b R_D(b) = \Delta\mu^\mathsf{T}D^{-1}\Delta\mu.

Equivalence to a distance contrast

With μˉ=(μ++μ−)/2\bar{\mu} = (\mu_+ + \mu_-)/2 and ∥a∥D−12=aTD−1a\lVert a\rVert_{D^{-1}}^2 = a^\mathsf{T}D^{-1}a,

∥z−μ−∥D−12−∥z−μ+∥D−12=2zTD−1(μ+−μ−)+μ+TD−1μ+−μ−TD−1μ−=2ΔμTD−1(z−μˉ)=e(z).(27)\begin{aligned} \lVert z-\mu_-\rVert_{D^{-1}}^2-\lVert z-\mu_+\rVert_{D^{-1}}^2 &= 2z^\mathsf{T}D^{-1}(\mu_+-\mu_-) + \mu_+^\mathsf{T}D^{-1}\mu_+ - \mu_-^\mathsf{T}D^{-1}\mu_- \\ &= 2\Delta\mu^\mathsf{T}D^{-1}(z-\bar{\mu}) = e(z). \tag*{(27)} \end{aligned}

This identity follows directly from cancellation of the common quadratic term and does not require the variational characterization in Proposition 1. A large coordinate magnitude alone therefore does not determine the distance contrast. The calibration rescales a nonzero outcome difference by dk−1d_k^{-1}, while coordinates with equal reference means contribute zero directly. With D=ID=I, this score becomes an ordinary squared-distance contrast.

Direct contribution of the visual coordinate

For z=[v;ρ]z=[v;\rho] and D=diag⁡(Dh,dρ)D=\operatorname{diag}(D_h,d_\rho), Eq. (27) yields

e(z)=2ΔμhTDh−1(v−μˉh)+2Δμρdρ(ρ−μˉρ).(28)e(z)=2\Delta\mu_h^\mathsf{T}D_h^{-1}(v-\bar{\mu}_h)+2\frac{\Delta\mu_\rho}{d_\rho}(\rho-\bar{\mu}_\rho). \tag*{(28)}

Proof. Differentiating the quadratic criterion gives

∇bFD(b)=Δμ−Db,∇b2FD(b)=−D≺0.(29)\nabla_bF_D(b)=\Delta\mu-Db,\qquad\nabla_b^2F_D(b)=-D\prec0. \tag*{(29)}

Thus FDF_D is strictly concave, and its unique stationary point b∗=D−1Δμb^*=D^{-1}\Delta\mu is the unique maximizer. For the normalized criterion, write u=D1/2bu=D^{1/2}b and a=D−1/2Δμa=D^{-1/2}\Delta\mu. Cauchy–Schwarz gives

RD(b)=(uTa)2uTu≤aTa=ΔμTD−1Δμ.(30)R_D(b)=\frac{(u^\mathsf{T}a)^2}{u^\mathsf{T}u}\le a^\mathsf{T}a=\Delta\mu^\mathsf{T}D^{-1}\Delta\mu. \tag*{(30)}

Equality holds when bb is a nonzero scalar multiple of D−1ΔμD^{-1}\Delta\mu.

When Δμ=0\Delta\mu=0, FDF_D is uniquely maximized at b=0b=0, whereas RD(b)=0R_D(b)=0 for every nonzero bb. The denominator in RDR_D is a regularized diagonal variation surrogate, not generally the full empirical variance of bTzb^\mathsf{T}z, since cross-coordinate covariances are not modeled. The result requires neither a Gaussian assumption nor a fitted probability model.

Comparison with an uncalibrated direction. For b0=Δμ≠0b_0=\Delta\mu\ne0, Proposition 1 yields

RD(b∗)=ΔμTD−1Δμ≥∥Δμ∥24ΔμTDΔμ=RD(b0).(31)R_D(b^*)=\Delta\mu^\mathsf{T}D^{-1}\Delta\mu\ge\frac{\lVert\Delta\mu\rVert_2^4}{\Delta\mu^\mathsf{T}D\Delta\mu}=R_D(b_0). \tag*{(31)}

Equality holds exactly when Δμ\Delta\mu is an eigenvector of DD. This comparison characterizes coordinate calibration under the stated fixed-statistic criterion. It does not establish a corresponding ordering of rewards, token-label accuracy, or generalization after reference refinement and nonlinear weight mapping.

which proves Eq. (5). The maximum criterion in Eq. (26) also decomposes as

RDmax⁡=ΔμhTDh−1Δμh+(Δμρ)2dρ.(32)\mathcal{R}^{\max}_{D} = \Delta\mu_h^{\mathsf{T}} D_h^{-1} \Delta\mu_h + \frac{(\Delta\mu_{\rho})^2}{d_{\rho}}. \tag*{(32)}

The last term quantifies the visual coordinate’s contribution to this fixed-statistic diagonal criterion. It is not a measure of information unique to the image, since redundancy with hidden coordinates is not removed. Removing RVE and refitting the estimator can change the hidden references as well.

Holding the final references, DD, τc>0\tau_c > 0, the predictive feature vv, and the advantage side c∈{−1,+1}c \in\{-1,+1\} fixed, let uc=sigmoid⁡(ce∗(z)/τc)u_c = \operatorname{sigmoid}(ce^*(z)/\tau_c). Then

∂e∗∂ρ=2cΔμρ∗dρ,∂uc∂ρ=uc(1−uc)2cΔμρ∗τcdρ.(33)\frac{\partial e^*}{\partial\rho} = \frac{2c\Delta\mu_{\rho}^*}{d_{\rho}}, \qquad\frac{\partial u_c}{\partial\rho} = u_c(1-u_c)\frac{2c\Delta\mu_{\rho}^*}{\tau_c d_{\rho}}. \tag*{(33)}

The raw weight has the same sensitivity multiplied by λmax⁡−λmin⁡\lambda_{\max}-\lambda_{\min}. These derivatives show that the response’s advantage side and the empirical visual contrast jointly determine the dependence on visual affinity. These are conditional sensitivities of the evidence and raw coefficient map, not gradients propagated through the credit estimator. The final normalized coefficients additionally depend on the shared factor ZZ, and re-estimating the reference statistics introduces further dependencies. Here calibration refers to variance scaling of observed advantage-side contrasts, not correctness-probability calibration or causal attribution.

Soft Reference Refinement

The default estimator performs one refinement. Let mit(0)=cie(0)(zit)m_{it}^{(0)} = c_i e^{(0)}(z_{it}), and compute

m‾c=1∣Tc∣∑Tcmit(0),τc=max⁡{1∣Tc∣∑Tc(mit(0)−m‾c)2,ϵτ}.(34)\overline{m}_c = \frac{1}{|\mathcal{T}_c|}\sum_{\mathcal{T}_c} m_{it}^{(0)}, \qquad \tau_c = \max\left\{\sqrt{\frac{1}{|\mathcal{T}_c|}\sum_{\mathcal{T}_c}(m_{it}^{(0)}-\overline{m}_c)^2},\epsilon_{\tau}\right\}. \tag*{(34)}

Here ϵτ>0\epsilon_{\tau} > 0. Unlike the reference moments, this population standard deviation is not weighted by ∣Ai∣|A_i|. The temperatures are computed from the initial references before Eq. (11) updates those references once. Final weights use the refined references while retaining both the initial DD and the pre-refinement τc\tau_c.

Entropy-regularized assignment. The soft assignment has the following standard variational form [52]. For fixed mm and τ>0\tau> 0, consider

max⁡u∈[0,1]{um+τH(u)},H(u)=−ulog⁡u−(1−u)log⁡(1−u).(35)\max_{u \in[0,1]} \{um+\tau\mathcal{H}(u)\}, \qquad \mathcal{H}(u)=-u\log u-(1-u)\log(1-u). \tag*{(35)}

With the convention 0log⁡0=00\log0=0, the objective is continuous on [0,1][0,1]. On (0,1)(0,1), its first derivative is m+τlog⁡((1−u)/u)m+\tau\log((1-u)/u) and its second derivative is −τ/[u(1−u)]<0-\tau/[u(1-u)]<0. For finite mm, the unique stationary point lies in (0,1)(0,1) and is the unique maximizer:

u=sigmoid⁡(m/τ).(36)u=\operatorname{sigmoid}(m/\tau). \tag*{(36)}

Reference update. Fix ritc=∣Ai∣uit(0)r_{it}^c=|A_i|u_{it}^{(0)} for (i,t)∈Tc(i,t)\in\mathcal{T}_c. For finite margins, 0<uit(0)<10<u_{it}^{(0)}<1, so Mc>0M_c>0 implies ∑Tcritc>0\sum_{\mathcal{T}_c}r_{it}^c>0 in exact arithmetic. Consider the weighted least-squares problem

min⁡μ∑Tcritc∥zit−μ∥D−12.(37)\min_{\mu}\sum_{\mathcal{T}_c}r_{it}^c\lVert z_{it}-\mu\rVert_{D^{-1}}^2. \tag*{(37)}

Its Hessian is 2(∑Tcritc)D−1≻02(\sum_{\mathcal{T}_c}r_{it}^c)D^{-1}\succ0. Setting its gradient to zero therefore gives the unique minimizer:

2D−1∑Tcritc(μ−zit)=0,μ=∑Tcritczit∑Tcritc,(38)2D^{-1}\sum_{\mathcal{T}_c}r_{it}^c(\mu-z_{it})=0,\qquad \mu=\frac{\sum_{\mathcal{T}_c}r_{it}^c z_{it}}{\sum_{\mathcal{T}_c}r_{it}^c}, \tag*{(38)}

recovering Eq. (11). These variational characterizations apply separately to the assignment and reference-update subproblems. They do not identify the complete procedure as EM or as coordinate ascent on a common objective. The estimator performs one refinement rather than iterating to convergence.

Credit Coefficients and Policy Updates

Normalization, bounds, and signs

Assume N>0N>0 and 0<λmin⁡≤λmax⁡0<\lambda_{\min}\le\lambda_{\max}. Every raw coefficient, including that of a zero-advantage token, lies in [λmin⁡,λmax⁡][\lambda_{\min},\lambda_{\max}]. Its global token mean ZZ therefore lies in the same positive interval. Consequently,

1N∑i,twit=1,λmin⁡λmax⁡≤wit≤λmax⁡λmin⁡,sign⁡(witAi)=sign⁡(Ai).(39)\frac{1}{N}\sum_{i,t}w_{it}=1,\qquad \frac{\lambda_{\min}}{\lambda_{\max}}\le w_{it}\le\frac{\lambda_{\max}}{\lambda_{\min}},\qquad \operatorname{sign}(w_{it}A_i)=\operatorname{sign}(A_i). \tag*{(39)}

The raw interval [0.8,1.2][0.8,1.2] thus gives the conservative final bounds [2/3,3/2][2/3,3/2]. Zero-advantage tokens affect ZZ but make no direct policy-gradient contribution. Global token-mean normalization does not preserve each response’s mean coefficient, the advantage mass of either side, or the policy-gradient norm.

Token-specific gradients under sequence clipping

The stop-gradient construction in Eq. (13) gives

s~it=sinumerically,∇θs~it=si∇θLit(θ).(40)\widetilde{s}_{it}=s_i\quad\text{numerically},\qquad \nabla_{\theta}\widetilde{s}_{it}=s_i\nabla_{\theta}L_{it}(\theta). \tag*{(40)}

This is the GSPO-token derivative. For nonzero advantages and away from clipping boundaries, define

κi={I{si<1+ϵh},Ai>0,I{si>1−ϵl},Ai<0.(41)\kappa_i = \begin{cases} \mathbb{I}\{s_i < 1+\epsilon_h\}, & A_i > 0,\\ \mathbb{I}\{s_i > 1-\epsilon_l\}, & A_i < 0. \end{cases} \tag*{(41)}

All valid tokens in a response have the same numerical ratio and advantage sign. The active clipping branch is therefore shared across that response. Treating AiA_i and witw_{it} as constants during differentiation,

∇θJOVC=1B∑i∣Ai≠0κisiAiTi∑twit∇θLit(θ).(42)\nabla_\theta J_{\mathrm{OVC}} = \frac{1}{B}\sum_{i\mid A_i\ne0} \frac{\kappa_i s_i A_i}{T_i} \sum_t w_{it}\nabla_\theta L_{it}(\theta). \tag*{(42)}

Sequence clipping therefore selects the active response-level surrogate terms, while each retained token-gradient term is multiplied by its own coefficient. Setting all wit=1w_{it}=1 recovers the gradient of the corresponding GSPO surrogate with the same response averaging and clipping rule.

Why a shared differentiable ratio is insufficient. If one directly reuses differentiable sis_i in every token term, the unclipped objective becomes

Jshared=1B∑iAisiwˉi,wˉi=1Ti∑twit.(43)J_{\mathrm{shared}} = \frac{1}{B}\sum_i A_i s_i\bar{w}_i,\qquad \bar{w}_i = \frac{1}{T_i}\sum_t w_{it}. \tag*{(43)}

Its gradient is

∇θJshared=1B∑isiAiwˉiTi∑t∇θLit(θ).(44)\nabla_\theta J_{\mathrm{shared}} = \frac{1}{B}\sum_i \frac{s_i A_i\bar{w}_i}{T_i} \sum_t \nabla_\theta L_{it}(\theta). \tag*{(44)}

Only the response-average coefficient survives. The same reduction holds under clipping, with the corresponding factor κi\kappa_i. The stop-gradient construction instead preserves the correspondence between individual tokens and their weights, despite an identical forward ratio value.

Separating response weighting from token allocation

For fixed rollouts and a fixed actor parameter value θ\theta, let git=∇θLit(θ)g_{it}=\nabla_\theta L_{it}(\theta) and gˉi=Ti−1∑tgit\bar{g}_i=T_i^{-1}\sum_t g_{it}. Define the vector

Ci=1Ti∑t(wit−wˉi)(git−gˉi).(45)C_i = \frac{1}{T_i}\sum_t (w_{it}-\bar{w}_i)(g_{it}-\bar{g}_i). \tag*{(45)}

Expanding and using the zero sums of centered quantities yields

1Ti∑twitgit=wˉigˉi+Ci,∇θJOVC=1B∑i∣Ai≠0κisiAi(wˉigˉi+Ci).(46)\begin{aligned} \frac{1}{T_i}\sum_t w_{it}g_{it} &= \bar{w}_i\bar{g}_i+C_i,\\ \nabla_\theta J_{\mathrm{OVC}} &= \frac{1}{B}\sum_{i\mid A_i\ne0} \kappa_i s_i A_i(\bar{w}_i\bar{g}_i+C_i). \tag*{(46)} \end{aligned}

The first term is response-level weighting. The second depends on the correspondence between weights and token gradients within that response. Replacing every weight by wˉi\bar{w}_i removes CiC_i exactly while preserving the first term.

For a uniformly random permutation Πi\Pi_i of a response’s weights, with token gradients and the weight multiset held fixed,

EΠi[CiΠi]=0,(47)\mathbb{E}_{\Pi_i}[C_i^{\Pi_i}] = 0, \tag*{(47)}

because EΠi[wi,Πi(t)]=wˉi\mathbb{E}_{\Pi_i}[w_{i,\Pi_i(t)}]=\bar{w}_i at every position. This conditional expectation concerns a fixed update; an individual permutation need not eliminate the allocation term. It motivates permutation and response-mean controls without implying identical training trajectories or long-run performance.

The preceding identities characterize the detached surrogate in Eq. (14). They do not make the coefficients ground-truth process advantages, establish an unbiased gradient of the original expected reward, or guarantee monotonic improvement. Shared model parameters also prevent advantage-sign preservation from implying an independent probability increase or decrease for every token.

Algorithm and Implementation

Complete algorithm and implementation details

Statistical scope and alignment. The described recipe uses one image per input, nonempty image and context position sets, and a fixed intermediate layer. The state hith_{it} is taken at the position predicting yity_{it}, rather than at the position after uitu_{it} has been consumed. Probabilities come from the current actor rather than being reused from the behavior policy. In the default micro-batch of one response, the RVE scale is response-wise even though G=8G=8.

Reference moments, margin scales, and normalization are aggregated over the actor mini-batch across micro-batches and data-parallel ranks. Initial sufficient statistics consist of each side’s mass, weighted feature sum, and weighted squared-feature sum. Margin temperatures use unweighted token counts and margin moments. RVE scales are computed before these aggregations and are not global rollout-group statistics.

Nonzero and degenerate batches. The mathematical reference formulas assume positive side masses. For complete rollout groups, any group with nonconstant rewards supplies both advantage signs. If all groups have constant rewards, every AiA_i and the policy gradient are zero. Numerical denominators and variances use floors in the implementation. Group reward standard deviations use the sample estimator with denominator G−1G-1, whereas the feature and margin variances use population estimators. The all-image formulation does not describe mixed image-free data.

ModelTraining dataPrompt batchRollouts GGLearning rateMax promptMax responseCheckpoint
Qwen2.5-VL-7BViRL39K512161×10−61 \times10^{-6}2,04816384step 76
Qwen3-VL-4BViRL39K512161×10−61 \times10^{-6}2,04820,480step 76
Qwen3-VL-8BViRL39K512161×10−61 \times10^{-6}2,0488,192step 76
Qwen3.5-9BMath77K25681×10−61 \times10^{-6}4,09616,384step 300
Qwen3-VL-4B (Chart)ChartVerse (18k)256161×10−61 \times10^{-6}16,38416,384step 72

Table 9. Training recipes and evaluated checkpoints.

Algorithm 1 OVC actor update with one reference refinement

Require: Rollouts, masks, rollout-group rewards, behavior log probabilities, and actor θ\theta.

Require: N>0N > 0 and M+,M−>0M_{+}, M_{-} > 0.

  1. Compute AiA_i and cic_i using Eq. (1).

  2. Keep θ\theta fixed until the final optimizer step.

All aggregates span valid actor-mini-batch tokens across micro-batches and data-parallel ranks.

Passes 1–4 recompute zitz_{it} using response-wise RVE and Eq. (8), without gradients.

Pass 1: Reference moments

  1. Aggregate, for c∈{−1,+1}c \in\{-1,+1\} and k=1,…,H+1k = 1,\ldots,H+1,

(Mc,Sc,k,Qc,k)←∑(i,t)∈Tc∣Ai∣(1,zitk,zitk2).(M_c,S_{c,k},Q_{c,k}) \leftarrow\sum_{(i,t)\in\mathcal{T}_c}|A_i|(1,z_{itk},z_{itk}^{2}).
  1. μc,k(0)←Sc,k/Mc\mu_{c,k}^{(0)} \leftarrow S_{c,k}/M_c, qk←∑c∈{−1,+1}[Qc,k/Mc−(μc,k(0))2]q_k \leftarrow\sum_{c\in\{-1,+1\}}[Q_{c,k}/M_c-(\mu_{c,k}^{(0)})^2].

  2. Construct DD using Eq. (10).

Pass 2: Margin scales

  1. Compute cie(0)(zit)c_{i}e^{(0)}(z_{it}) using μ±(0)\mu^{(0)}_{\pm} and DD.

  2. Estimate τc\tau_c from unweighted margin moments using Eq. (34).

Pass 3: Reference refinement

  1. Compute uit(0)u_{it}^{(0)} and aggregate μc∗\mu_c^{*} using Eq. (11).

  2. Form e∗e^{*} with μ±∗\mu_{\pm}^{*}; retain DD and the pre-refinement τc\tau_c.

Pass 4: Weight normalization

  1. Compute w^it\widehat{w}_{it} using Eq. (12); set w^it=λmin⁡\widehat{w}_{it}=\lambda_{\min} when Ai=0A_i=0.

  2. Z←N−1∑i,tw^itZ \leftarrow N^{-1}\sum_{i,t}\widehat{w}_{it}, including zero-advantage tokens.

Pass 5: Policy optimization

  1. Recompute differentiable Lit(θ)L_{it}(\theta) and detached credit features.

  2. Recompute wit=w^it/Zw_{it}=\widehat{w}_{it}/Z with μ±∗\mu_{\pm}^{*}, DD, τc\tau_c, ZZ fixed.

  3. Set Ait′=sg⁡(wit)AiA'_{it}=\operatorname{sg}(w_{it})A_i and construct GSPO-token ratios using Eq. (13).

  4. Accumulate gradients of −JOVC-J_{\mathrm{OVC}} using Eq. (14); update θ\theta once.

Normalization domains. The reference estimator uses ∣Ai∣|A_i| for each token rather than ∣Ai∣/Ti|A_i|/T_i. Longer responses therefore have greater total weight in the reference statistics. In contrast, Eq. (14) averages tokens within a response and then averages responses. Global coefficient normalization does not remove this difference.

Computation. For one refinement, the current implementation uses four no-gradient statistics forwards and one training forward per micro-batch. The actor parameters do not change between these passes. Sufficient-statistic aggregation agrees algebraically with direct global aggregation when the recomputed features are identical, up to floating-point order. The calibration arithmetic is O(NH)O(NH) per pass with O(H)O(H) accumulated statistics, excluding model forwards, activation storage, and communication. These counts are not measured runtime or speed claims. Standard inference does not require the credit estimator.

OVC uses the scalar sequence reward supplied by each training recipe. For the accuracy-plus-format recipe, this reward is Ri=0.9Riacc+0.1RifmtR_i=0.9R_i^{\mathrm{acc}}+0.1R_i^{\mathrm{fmt}}. The chart-centric recipe instead uses binary correctness rewards from CompassVerifier [18], without a format-reward component (Appendix F.2.1). Across these recipes, advantage signs describe relative sequence rewards within a rollout group, rather than token-level correctness. The stated GSPO objective has no dual-clipping term. Neither adaptive layer selection, region supervision, a full covariance estimator, nor an extra process reward model is part of this method.

Training recipes and checkpoints

Table 9 summarizes the RL training recipes and evaluated checkpoints for the main comparisons, optimizer extensions, and chart-centric experiment. Within each setting, compared methods share the training data and rollout configuration unless otherwise noted. The chart-centric runs are initialized from the shared Chart SFT checkpoint at optimizer step 782. Their final checkpoint is taken after 72 rollout steps. Complete chart-specific settings are given in Appendix F.2.1.

Additional OVC-specific implementation constants for the Qwen3.5-9B recipe are listed in Table 10. These settings concern the credit estimator itself and are separate from the model-level training recipe summarized above.

Chart-centric training

Supervised initialization. We fine-tune Qwen3-VL-4B-Instruct on 100,000 examples from ChartVerse-SFT-600K [20], using image–question inputs and complete chain-of-thought solutions as targets. We reserve 512 examples for validation, with their images held out from both SFT and RL training. SFT runs for one epoch with a global batch size of 128, a learning rate of 10−510^{-5}, cosine decay with 10% warmup, and a maximum sequence length of 22,000 tokens. The final checkpoint at optimizer step 782 serves as Base and initializes both RL methods.

QuantitySetting
Predictive layer and hidden widthℓ=16\ell= 16, H=4096H = 4096
Responses per inputG=8G = 8
Global actor mini-batch512 responses
Forward micro-batch per GPU1 response
Reference refinementsK=1K = 1
Raw coefficient interval[0.8,1.2][0.8, 1.2]
Shrinkage and relative variance floorη=0.1\eta= 0.1, rmin⁡=10−3r_{\min} = 10^{-3}
Response scale floorϵσ=10−6\epsilon_{\sigma} = 10^{-6}
Margin temperature floorϵτ=10−8\epsilon_{\tau} = 10^{-8}
Lower and upper ratio clipping3×10−43 \times10^{-4}, 4×10−44 \times10^{-4}
Auxiliary KL and entropy lossesDisabled

Table 10. Defaults of the supplied Qwen3.5-9B recipe.

RL data. For each question in the full 40,000-question ChartVerse-RL-40K pool, the SFT model samples eight responses at temperature 1. CompassVerifier-7B [18] judges their correctness, and we retain questions with one to seven correct responses. Filtering and deduplication yield 18,280 training questions shared by GSPO and GSPO+OVC.

References

  1. [1]Shuai Bai, Yuxuan Cai, Ruizhe Chen, Keqin Chen, Xionghui Chen, Zesen Cheng, Lianghao Deng, Wei Ding, Chang Gao, Chunjian Ge, et al. Qwen3-VL technical report. arXiv preprint arXiv:2511.21631, 2025. 1, 5arxiv.org/abs/2511.21631
  2. [2]Shuai Bai, Keqin Chen, Xuejing Liu, Jialin Wang, Wenbin Ge, Sibo Song, Kai Dang, Peng Wang, Shijie Wang, Jun Tang, Humen Zhong, Yuanzhi Zhu, Mingkun Yang, Zhaohai Li, Jianqiang Wang, Pengfei Wang, Wei Ding, Zheren Fu, Yiheng Xu, Jiabo Ye, Xi Zhang, Tianbao Xie, Zesen Cheng, Hang Zhang, Zhibo Yang, Haiyang Xu, and Junyang Lin. Qwen2.5-VL technical report. arXiv preprint arXiv:2502.13923, 2025. 7arxiv.org/abs/2502.13923
  3. [3]Liang Chen, Hongcheng Gao, Tianyu Liu, Zhiqi Huang, Flood Sung, Xinyu Zhou, Yuxin Wu, and Baobao Chang. G1: Bootstrapping perception and reasoning abilities of vision-language model via reinforcement learning. arXiv preprint arXiv:2505.13426, 2025. 1arxiv.org/abs/2505.13426
  4. [4]Xianfu Cheng, Wei Zhang, Shiwei Zhang, Jian Yang, Xiangyuan Guan, Xianjie Wu, Xiang Li, Ge Zhang, Jiaheng Liu, Yuying Mai, et al. SimpleVQA: Multimodal factuality evaluation for multimodal large language models. In 2025 IEEE/CVF International Conference on Computer Vision (ICCV), pages 4637–4666. IEEE, 2025. 7, S3arxiv.org/abs/2502.13059
  5. [5]DeepSeek-AI. DeepSeek-V4: Towards highly efficient million-token context intelligence. arXiv preprint arXiv:2606.19348, 2026. S2, S4arxiv.org/abs/2606.19348
  6. [6]Mengfei Du, Binhao Wu, Zejun Li, Xuanjing Huang, and Zhongyu Wei. EmbSpatial-Bench: Benchmarking spatial understanding for embodied tasks with large vision-language models. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers), pages 346–355. Association for Computational Linguistics, 2024. 8, S4arxiv.org/abs/2406.05756
  7. [7]Gemini Robotics Team. Gemini Robotics: Bringing AI into the physical world. arXiv preprint arXiv:2503.20020, 2025. 8, S4arxiv.org/abs/2503.20020
  8. [8]Yuhang Han, Yuyang Wu, Zhengbo Jiao, Yiyu Wang, Xuyang Liu, Shaobo Wang, Hanlin Xu, Xuming Hu, and Linfeng Zhang. Bridging visual representation and reinforcement learning from verifiable rewards in large vision-language models. In European Conference on Computer Vision, pages 293–311. Springer, 2026. 1, 3
  9. [9]Zefeng He, Xiaoye Qu, Yafu Li, Siyuan Huang, Daizong Liu, and Yu Cheng. FrameThinker: Learning to think with long videos via multi-turn frame spotlighting. In International Conference on Learning Representations, 2026. 1arxiv.org/abs/2509.24304
  10. [10]M uye Huang, Han Lai, Xinyu Zhang, Wenjun Wu, Jie Ma, Lingling Zhang, and Jun Liu. EvoChart: A benchmark and a self-training approach towards real-world chart understanding. In Proceedings of the AAAI Conference on Artificial Intelligence, pages 3680–3688, 2025. 7, S4arxiv.org/abs/2409.01577
  11. [11]Siyuan Huang, Xiaoye Qu, Yafu Li, Yun Luo, Zefeng He, Daizong Liu, and Yu Cheng. Spotlight on token perception for multimodal reinforcement learning. In International Conference on Learning Representations, pages 103291–103325, 2026. 1, 2, 5, 6arxiv.org/abs/2510.09285
  12. [12]Wenxuan Huang, Bohan Jia, Shaosheng Cao, Zheyu Ye, Fei zhao, Zhe Xu, Yao Hu, and Shaohui Lin. Vision-r1: Incentivizing reasoning capability in multimodal large language models. In International Conference on Learning Representations, pages 63794–63812, 2026. 1arxiv.org/abs/2503.06749
  13. [13]Zhengbo Jiao, Shaobo Wang, Zifan Zhang, Wei Wang, Bing Zhao, Hu Wei, and Linfeng Zhang. Credit where it is due: Cross-modality connectivity drives precise reinforcement learning for mllm reasoning. arXiv preprint arXiv:2602.11455, 2026. 1, 5, 6arxiv.org/abs/2602.11455
  14. [14]Hanzheng Li, Xi Fang, Yixuan Li, Chaozheng Huang, Junjie Wang, Xi Wang, Hongzhe Bai, Bojun Hao, Shenyu Lin, Huiqi Liang, Linfeng Zhang, and Guolin Ke. RxnBench: A multimodal benchmark for evaluating large language models on chemical reaction understanding from the scientific literature. Journal of Chemical Information and Modeling, 66(10): 5747–5756, 2026. 8, S3
  15. [15]Qiming Li, Tianlun Li, Xiaolong Cheng, Hangyu Li, Ruiyan Gong, Kangning Niu, Kaitao Jiang, and Mu Xu. PRPO: Perception-reinforced policy optimization via token-level dynamic advantage reshaping. arXiv preprint arXiv:2606.08708, 2026. 3arxiv.org/abs/2606.08708
  16. [16]Shenshen Li, Xing Xu, Kaiyuan Deng, Lei Wang, Heng Tao Shen, and Fumin Shen. Truth in the few: High-value data selection for efficient multi-modal reasoning. arXiv preprint arXiv:2506.04755v2, 2026. 1arxiv.org/abs/2506.04755
  17. [17]Yiqing Liang, Jielin Qiu, Wenhao Ding, Zuxin Liu, James Tompkin, Mengdi Yu, Mengzhou Xia, Zhengzhong Tu, Laixi Shi, and Jiacheng Zhu. MoDoMoDo: Multi-domain data mixtures for multimodal llm reinforcement learning. arXiv preprint arXiv:2505.24871, 2025. 1arxiv.org/abs/2505.24871
  18. [18]Shudong Liu, Hongwei Liu, Junnan Liu, Linchen Xiao, Songyang Gao, Chengqi Lyu, Yuzhe Gu, Wenwei Zhang, Derek F. Wong, Songyang Zhang, and Kai Chen. CompassVerifier: A unified and robust verifier for LLMs evaluation and outcome reward. In Proceedings of the 2025 Conference on Empirical Methods in Natural Language Processing, pages 33466–33494. Association for Computational Linguistics, 2025. S8, S9
  19. [19]Xiangyan Liu, Jinjie Ni, Zijian Wu, Chao Du, Longxu Dou, Haonan Wang, Tianyu Pang, and Michael Qizhe Shieh. Noisy-Rollout: Reinforcing visual reasoning with data augmentation. In Advances in Neural Information Processing Systems, 2025. 1DOI
  20. [20]Zheng Liu, Honglin Lin, Xiaoyang Wang, Xin Gao, Yu Li, Mengzhang Cai, Yun Zhu, Zhanping Zhong, Qizhi Pei, Zhuoshi Pan, Xiaoran Shang, Conghui He, Bin Cui, Wentao Zhang, and Lijun Wu. ChartVerse: Scaling chart reasoning via reliable programmatic synthesis from scratch. In Proceedings of the 64th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 7551–7577. Association for Computational Linguistics, 2026. 7, S8
  21. [21]Pan Lu, Swaroop Mishra, Tanglin Xia, Liang Qiu, Kai-Wei Chang, Song-Chun Zhu, Oyvind Tafjord, Peter Clark, and Ashwin Kalyan. Learn to explain: Multimodal reasoning via thought chains for science question answering. In Advances in Neural Information Processing Systems, pages 2507–2521, 2022. 8, S3arxiv.org/abs/2209.09513
  22. [22]Pan Lu, Hritik Bansal, Tony Xia, Jiacheng Liu, Chunyuан Li, Hannah Hajishirzi, Hao Cheng, Kai-Wei Chang, Michel Galley, and Jianfeng Gao. MathVista: Evaluating mathematical reasoning of foundation models in visual contexts. In International Conference on Learning Representations, 2024. 5, S1arxiv.org/abs/2310.02255
  23. [23]Ahmed Masry, Do Xuan Long, Jia Qing Tan, Shafiq Joty, and Enamul Hoque. ChartQA: A benchmark for question answering about charts with visual and logical reasoning. In Findings of the Association for Computational Linguistics: ACL 2022, pages 2263–2279, Dublin, Ireland, 2022. Association for Computational Linguistics. 7, S3arxiv.org/abs/2203.10244
  24. [24]Ahmed Masry, Mohammed Saidul Islam, Mahir Ahmed, Aayush Bajaj, Firoz Kabir, Aaryaman Kartha, Md Tahmid Rahman Laskar, Mizanur Rahman, Shadikuр Rahman, Mehrad Shahmohammadi, Megh Thakkar, Md Rizwan Parvez, Enamul Hoque, and Shafiq Joty. ChartQAPro: A more diverse and challenging benchmark for chart question answering. In Findings of the Association for Computational Linguistics: ACL 2025, pages 19123–19151. Association for Computational Linguistics, 2025. 7, S4arxiv.org/abs/2504.05506
  25. [25]Fanqing Meng, Lingxiao Du, Zongkai Liu, Zhixiang Zhou, Quanfeng Lu, Daocheng Fu, Tiancheng Han, Botian Shi, Wenhai Wang, Junjun He, Kaipeng Zhang, Ping Luo, Yu Qiao, Qiaosheng Zhang, and Wenqi Shao. MM-Eureka: Exploring the frontiers of multimodal reasoning with rule-based reinforcement learning. arXiv preprint arXiv:2503.07365v2, 2025. 1, 2, 5, S1arxiv.org/abs/2503.07365
  26. [26]Runqi Qiao, Qiuna Tan, Guanting Dong, Minhui Wu, Chong Sun, Xiaoshuai Song, Jiapeng Wang, Zhuoma GongQue, Shanglin Lei, YiFan Zhang, Zhe Wei, Miaoxuan Zhang, Runfeng Qiao, Xiao Zong, Yida Xu, Peiqing Yang, Zhimin Bao, Muxi Diao, Chen Li, and Honggang Zhang. We-Math: Does your large multimodal model achieve human-like mathematical reasoning? In Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 20023–20070, Vienna, Austria, 2025. Association for Computational Linguistics. 5, S1arxiv.org/abs/2407.01284
  27. [27]Qwen Team. Qwen3.5: Towards native multimodal agents. Qwen Blog, 2026. https://qwen.ai/blog?id=qwen3.5. 1, 5
  28. [28]Dustin Schwenk, Apoorv Khandelwal, Christopher Clark, Kenneth Marino, and Roozbeh Mottaghi. A-OKVQA: A benchmark for visual question answering using world knowledge. In European conference on computer vision, pages 146–162. Springer, 2022. 7, S3arxiv.org/abs/2206.01718
  29. [29]Zhihong Shao, Peiyi Wang, Qihao Zhu, Runxin Xu, Junxiao Song, Xiao Bi, Haowei Zhang, Mingchuan Zhang, Y. K. Li, Y. Wu, and Daya Guo. Deepseekmath: Pushing the limits of mathematical reasoning in open language models. arXiv preprint arXiv:2402.03300, 2024. 1, 2, 3arxiv.org/abs/2402.03300
  30. [30]Haozhan Shen, Peng Liu, Jingcheng Li, Chunxin Fang, Yibo Ma, Jiajia Liao, Qiaoli Shen, Zilun Zhang, Kangjia Zhao, Qianqian Zhang, Ruochen Xu, and Tiancheng Zhao. Vlm-rl: A stable and generalizable rl-style large vision-language model. arXiv preprint arXiv:2504.07615, 2025. 1
  31. [31]Haoxiang Sun, Lizhen Xu, Bing Zhao, Wotao Wei, Wei Wang, Boyu Yang, Rui Wang, and Hu Wei. DeepVision-103K: A visually diverse, broad-coverage, and verifiable mathematical dataset for multimodal reasoning. arXiv preprint arXiv:2602.16742, 2026. 5arxiv.org/abs/2602.16742
  32. [32]Liyan Tang, Grace Kim, Xinyu Zhao, Thom Lake, Wenxuan Ding, Fangcong Yin, Prasann Singhal, Manya Wadhwa, Zeyu Liu, Zayne Sprague, Ramya Namuduri, Bodun Hu, Juan Rodriguez, Puyuan Peng, and Greg Durrett. ChartMuseum: Testing visual reasoning capabilities of large vision-language models. In Advances in Neural Information Processing Systems, 2025. 7, S4arxiv.org/abs/2505.13444
  33. [33]Zhongwei Wan, Zihao Dou, Che Liu, Yu Zhang, Dongfei Cui, Qinjian Zhao, Hui Shen, Jing Xiong, Yi Xin, Yifan Jiang, Chaofan Tao, Yangfan He, Mi Zhang, and Shen Yan. SRPO: Enhancing multimodal LLM reasoning via reflection-aware reinforcement learning. In Advances in Neural Information Processing Systems, 2025. 1arxiv.org/abs/2506.01713
  34. [34]Haozhe Wang, Chao Qu, Zuming Huang, Wei Chu, Fangzhen Lin, and Wenhu Chen. VL-Rethinker: Incentivizing self-reflection of vision-language models with reinforcement learning. In Advances in Neural Information Processing Systems, pages 30865–30891. Curran Associates, Inc., 2025. 2, 5arxiv.org/abs/2504.08837
  35. [35]Ke Wang, Junting Pan, Weikang Shi, Zimu Lu, Houxing Ren, Aojun Zhou, Mingjie Zhan, and Hongsheng Li. Measuring multimodal mathematical reasoning with MATH-Vision dataset. In Advances in Neural Information Processing Systems, pages 95095–95169. Curran Associates, Inc., 2024. 5, S1arxiv.org/abs/2402.14804
  36. [36]Peiyu Wang, Yichen Wei, Yi Peng, Xiaokun Wang, Weijie Qiu, Wei Shen, Tianyidan Xie, Jiangbo Pei, Jianhao Zhang, Yunzhuo Hao, Xuchen Song, Yang Liu, and Yahui Zhou. Skywork r1v2: Multimodal hybrid reinforcement learning for reasoning. arXiv preprint arXiv:2504.16656, 2025. 1 Weiyun Wang, Zhangwei Gao, Lianjie Chen, Zhe Chen, Jing-guo Zhu, Xiangyu Zhao, Yangzhou Liu, Yue Cao, Sheng-long Ye, Xizhou Zhu, Lewei Lu, Haodong Duan, Yu Qiao, Jifeng Dai, and Wenhai Wang. VisualPRM400K: An effective dataset for training multimodal process reward models. In International Conference on Learning Representations, 2026. 1, 2
  37. [38]Zirui Wang, Mengzhou Xia, Luxi He, Howard Chen, Yitao Liu, Richard Zhu, Kaiqiu Liang, Xindi Wu, Haotian Liu, Sadhika Malladi, Alexis Chevalier, Sanjeev Arora, and Danqi Chen. CharXiv: Charting gaps in realistic chart understanding in multimodal LLMs. In Advances in Neural Information Processing Systems, pages 113569–113697, 2024. 7, S3, S4arxiv.org/abs/2406.18521
  38. [39]Zhenhailong Wang, Xuehai Sui, Sofia Stoica, Haiyang Xu, Hongru WANG, Hyeonjeong Ha, Xiusi Chen, Yangyi Chen, Ming Yan, Fei Huang, and Heng Ji. Perception-aware policy optimization for multimodal reasoning. In International Conference on Learning Representations, pages 80502–80530, 2026. 1, 2arxiv.org/abs/2507.06448
  39. [40]Zengbin Wang, Feng Xiong, Liang Lin, Xuecai Hu, Yong Wang, Yanlin Wang, Man Zhang, and Xiangxiang Chu. Visually-guided policy optimization for multimodal reasoning. In Proceedings of the 64th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 6646–6664, 2026. 1, 3
  40. [41]Zichen Wen, Boxue Yang, Shuang Chen, Yaojie Zhang, Yuhang Han, Junlong Ke, Cong Wang, Yicheng Fu, Jiawang Zhao, Jiangchao Yao, et al. Innovator-VL: A multimodal large language model for scientific discovery. arXiv preprint arXiv:2601.19325, 2026. 1arxiv.org/abs/2601.19325
  41. [42]Jiaer Xia, Yuhang Zang, Peng Gao, Sharon Li, and Kaiyang Zhou. Visionary-R1: Mitigating shortcuts in visual reasoning with reinforcement learning. Transactions on Machine Learning Research, 2026. 1arxiv.org/abs/2505.14677
  42. [43]Renqiu Xia, Hancheng Ye, Xiangchao Yan, Qi Liu, Hongbin Zhou, Zijun Chen, Botian Shi, Junchi Yan, and Bo Zhang. ChartX and ChartVLM: A versatile benchmark and foundation model for complicated chart reasoning. IEEE Transactions on Image Processing, 34:7436–7447, 2025. 7, S4
  43. [44]Tong Xiao, Xin Xu, Zhenya Huang, Hongyu Gao, Quan Liu, Qi Liu, and Enhong Chen. Perception-R1: Advancing multimodal reasoning capabilities of MLLMs via visual perception reward. In International Conference on Learning Representations, 2026. 1arxiv.org/abs/2506.07218
  44. [45]Zhenzhao Xu, Sinan Du, Yiyan Qi, Chengjin Xu, Chun Yuan, and Jian Guo. ChartBench: A benchmark for complex visual reasoning in charts. arXiv preprint arXiv:2312.15915, 2023. 7, S4arxiv.org/abs/2312.15915
  45. [46]Huanjin Yao, Qixiang Yin, Jingyi Zhang, Min Yang, Yibo Wang, Wenhao Wu, Fei Su, Li Shen, Minghui Qiu, Dacheng Tao, and Jiaxing Huang. R1-ShareVL: Incentivizing reasoning capabilities of multimodal large language models via Share-GRPO. In Advances in Neural Information Processing Systems, 2025. 1DOI
  46. [47]Zekai Ye, Qiming Li, Xiaocheng Feng, Ruihan Chen, Ziming Li, Haoyu Ren, Kun Chen, Dandan Tu, and Bing Qin. Not all tokens see equally: Perception-grounded policy optimization for large vision-language models. arXiv preprint arXiv:2604.01840, 2026. 2
  47. [48]En Yu, Kangheng Lin, Liang Zhao, Jisheng Yin, Yana Wei, Yuang Peng, Haoran Wei, Jianjian Sun, Chunrui Han, Zheng Ge, Xiangyu Zhang, Daxin Jiang, Jingyu Wang, and Wenbing Tao. Perception-R1: Pioneering perception policy with reinforcement learning. In Advances in Neural Information Processing Systems, 2025. 1arxiv.org/abs/2504.07954
  48. [49]Qiying Yu, Zheng Zhang, Ruofei Zhu, Yufeng Yuan, Xiaochen Zuo, Yu Yue, Weinan Dai, Tiantian Fan, Gaohong Liu, Juncai liu, LingJun Liu, Xin Liu, Haibin Lin, Zhiqi Lin, Bole Ma, Guangming Sheng, Yuxuan Tong, Chi Zhang, Mofan Zhang, Ru Zhang, Wang Zhang, Hang Zhu, Jinhua Zhu, Jiaze Chen, Jiangjie Chen, Chengyi Wang, Hongli Yu, Yuxuan Song, Xiangpeng Wei, Hao Zhou, Jingjing Liu, Wei-Ying Ma, Ya-Qin Zhang, Lin Yan, Yonghui Wu, and Mingxuan Wang. DAPO: An open-source llm reinforcement learning system at scale. In Advances in Neural Information Processing Systems, pages 113222–113244. Curran Associates, Inc., 2025. 1arxiv.org/abs/2503.14476
  49. [50]Xiang Yue, Yuansheng Ni, Kai Zhang, Tianyu Zheng, Ruoqi Liu, Ge Zhang, Samuel Stevens, Dongfu Jiang, Weiming Ren, Yuxuan Sun, et al. MMMU: A massive multi-discipline multimodal understanding and reasoning benchmark for expert agi. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 9556–9567, 2024. 5, S1arxiv.org/abs/2311.16502
  50. [51]Kaichen Zhang, Bo Li, Peiyuan Zhang, Fanyi Pu, Joshua Adrian Cahyono, Kairui Hu, Shuai Liu, Yuanhan Zhang, Jingkang Yang, Chunyuan Li, et al. LMMs-Eval: Reality check on the evaluation of large multimodal models. In Findings of the Association for Computational Linguistics: NAACL 2025, pages 881–916, 2025. 5, S1
  51. [52]Kaiyi Zhang, Wei Wu, and Yankai Lin. DelTA: Discriminative token credit assignment for reinforcement learning from verifiable rewards. arXiv preprint arXiv:2605.21467, 2026. 1, 5, 6, S6arxiv.org/abs/2605.21467
  52. [53]Renrui Zhang, Dongzhi Jiang, Yichi Zhang, Haokun Lin, Ziyu Guo, Pengshuo Qiu, Aojun Zhou, Pan Lu, Kai-Wei Chang, Yu Qiao, et al. MathVerse: Does your multi-modal llm truly see the diagrams in visual math problems? In European Conference on Computer Vision, pages 169–186. Springer, 2024. 5, S1arxiv.org/abs/2403.14624
  53. [54]Zehua Zhao, Zhixian Huang, Junren Li, Siyu Lin, Junting Zhou, Fengqi Cao, Kun Zhou, Rui Ge, Tingting Long, Yuexiang Zhu, Yan Liu, Jie Zheng, Junnian Wei, Rong Zhu, Peng Zou, Wenyu Li, Zekai Cheng, Tian Ding, Yaxuan Wang, Yizhao Yan, Tingru Wei, Haowei Ming, Weijie Mao, Chen Sun, Yiming Liu, Zichen Wang, Zuo Zhang, Tong Yang, Hao Ma, Zhen Gao, and Jian Pei. SUPERChem: A multimodal reasoning benchmark in chemistry. arXiv preprint arXiv:2512.01274, 2025. 8, S4arxiv.org/abs/2512.01274
  54. [55]Chujie Zheng, Shixuan Liu, Mingze Li, Xiong-Hui Chen, Bowen Yu, Chang Gao, Kai Dang, Yuqiong Liu, Rui Men, An Yang, Jingren Zhou, and Junyang Lin. Group sequence policy optimization. arXiv preprint arXiv:2507.18071, 2025. 1, 2, 5, 6, S3arxiv.org/abs/2507.18071

Paper details

vlnc:W_acqjxgnh
Version
v1 · latest
Published
Fresh — added to the Hub within the last 30 days
Added
License
CC BY 4.0
Usage

views · 0

downloads · 0

A view counts once per browser per day, a download each time the PDF is requested.

  • Computer Science

Not recorded for this paper

View statement
Looking for related papers…
Contents