WorksComputer Science
Stochastic Self-Distillation for Flow-Based Vision-Language-Action Models
Abstract
Flow-based vision-language-action (VLA) models achieve strong performance in robotics through imitation learning, yet fine-tuning them with on-policy reinforcement learning (RL) is limited. In group-based RL methods like Group Relative Policy Optimization (GRPO), every rollout step requires multi-step ODE integration, and the sampler's limited stochasticity leads to rollout groups with low variance. We propose VLA-SSD, which addresses both high inference latency and limited stochasticity. In particular, our method self-distills a pretrained flow-matching action head into a stochastic one-step policy by conditioning a single-jump generator on a Brownian path. We derive a conditional action density via our stochastic one-step policy, enabling efficient post-training via likelihood-ratio RL algorithms. We show that distilling VLA-SSD from the VLA reduces the action head's forward-pass time by a factor of 7.7, at the cost of a 5.5% relative decrease in the success rate on the LIBERO benchmark. However, by leveraging the increased sample diversity of VLA-SSD during group generation, post-training a VLA using GRPO yields a 20.6% relative improvement.
Introduction
Vision–language–action (VLA) models have achieved significant advances by leveraging knowledge from pre-trained vision–language models in robotics ([6, 13]). However, state-of-the-art VLAs have yet to achieve comparable generalization to other foundation models due to their reliance on imitation learning from expert demonstrations for adaptation to embodiments and environments ([3, 14] (Physical Intelligence et al. 2025; NVIDIA et al. 2025; Shukor et al. 2025)). Frontier flow-based VLAs generate continuous action chunks with a flow-matching action head ([3] (Physical Intelligence et al. 2025)). Yet, enabling them to improve via on-policy reinforcement learning (RL) is still limited by the inference time of multi-step ODE integration and a lack of stochasticity.
To improve efficiency, one-step flow-based policies have been introduced for continuous control ((Zhan et al., 2026; Ziakas et al., 2026)), replacing multi-step ODE integration with a single transport step learned via self-distillation ([4, 12, 5] (Song et al. 2023)). However, in RL methods using multi-sample advantage estimation, such as Group Relative Policy Optimization (GRPO), the baseline is derived directly from a group of trajectories given the current state. Because flow-matching rely entirely on the initial random noise draw for stochasticity, they restrict the action diversity within the sampled group, resulting in a degraded training signal ((Shao et al., 2024)). To enable exploration, existing approaches increase sample diversity by introducing an external noise network or reverting to multi-step samplers that inject noise at each integration step ([7, 16] (Zhang et al. 2025)).
In this work, we address both the efficiency and stochasticity limitations of current VLAs by learning a stochastic one-step policy via self-distillation. We propose VLA-SSD, a method that utilizes Itô flow maps ((Pan et al., 2026)) to distill a multi-step flow-matching VLA head into a one-step stochastic policy. In contrast to prior work, we introduce stochasticity directly through self-distillation rather than relying on an external noise network or a multi-step sampler. To enable likelihood-ratio policy optimization such as GRPO, we derive a conditional action density that requires only a single additional Jacobian-vector product to approximate. In our experiments, we first distill VLA-SSD from , achieving a 7.7× faster action head with a 0.3-point drop in success rate, while reducing zero-variance groups from 63% to 35% relative to the original sampler. Leveraging the stochasticity and efficiency of VLA-SSD, we post-train it with GRPO, reaching 93.0% on LIBERO, a 15.9-point gain over the SFT checkpoint, and matching four-step GRPO with (91.5%) while generating each action chunk 3.2× faster during training rollouts.
Figure 1. Left: an ODE sampler and Flow-SDE spend action-expert evaluations per chunk; VLA-SSD spends one and keeps the spread of the SDE. Right: for a fixed and , the Itô flow map conditions on a Brownian path , so paths give different one-step actions with different returns, and the GRPO group carries gradient; at all coincide.
In summary, our main contributions are as follows:
One-Step Stochastic VLA Policies: We introduce VLA-SSD, a novel self-distillation framework that converts multi-step flow-matching VLA heads into efficient, one-step stochastic policies using Itô flow maps.
Density Evaluation for Likelihood-Ratio Optimization: To overcome the lack of tractable likelihoods in flow-based models, we derive a closed-form conditional action density that evaluates efficiently via a single Jacobian-vector product.
Efficient Policy Distillation: We demonstrate that distilling VLA-SSD from yields a 7.7× faster action head with a minimal 0.3-point performance drop, while mitigating the exploration bottleneck by reducing zero-variance groups from 63% to 35%.
RL Post-Training: When post-trained with GRPO, VLA-SSD reaches a 93.0% success rate on LIBERO, a 15.9-point gain over the SFT checkpoint, matching the standard four-step baseline while accelerating training rollouts by 3.2×.
Preliminaries
Markov decision process. We consider a Markov decision process (MDP) (Sutton & Barto, 1998) with state space , action space , reward function , transition kernel , and discount factor . The state consists of the visual observations, the proprioceptive state, and the language instruction; the action is a chunk of consecutive actions. A policy is evaluated by its expected return , where indexes decision steps and is the episode horizon. We consider episodic manipulation tasks with a sparse binary reward, at the decision step at which the task is first completed and 0 otherwise, and take , so that the return of an episode is . Throughout, denotes MDP time; the flow time introduced next indexes the generation of a single action chunk and is distinct from .
Flow-matching policies. A generative policy can be formulated as a transport from a reference measure to a target action distribution . The transport is defined by a state-conditioned flow that solves the following ODE:
where is a time-dependent velocity field. The terminal map induces the policy ; it is deterministic for fixed and , while sampling induces a distribution over actions. To train the flow policy, we define a conditional probability path using the endpoints and the linear interpolation . The velocity field is learned by regressing against the target conditional velocity using the conditional flow-matching objective [17, 1]:
where is a coupling over the endpoints —e.g., independent coupling —and in the last equality we substitute the analytic conditional velocity for the linear interpolant.
Flow map policies. Instead of solving the ODE, we parameterize and learn its two-time solution operator, referred to as the flow map [11, 12, 4, 1] (Song et al. 2023; Ziakas et al. 2026). For , let denote the flow map associated with the ODE in eq. (1). It transports an action from time to and satisfies the jump condition
The one-step policy draws and returns , inducing the action distribution
Following Boffi et al. [4], we parameterize the flow map using the average action velocity :
Thus, represents a finite jump of size along the ODE trajectory. On the diagonal, the average velocity recovers the instantaneous velocity,
This diagonal relation connects the finite-time flow map to the instantaneous flow-matching velocity. We use it to construct the training objective for the stochastic map in § 3.1.
Group relative policy optimization. GRPO (Shao et al., 2024) is an on-policy reinforcement learning algorithm that replaces the learned value baseline of actor–critic methods with a Monte Carlo baseline computed within a group of rollouts. At each decision step , the policy samples an action chunk , and the objective of § 2 reduces to with the binary return . GRPO samples a group of trajectories from the same task and initial state, and assigns to every decision step of trajectory an advantage obtained by normalizing its return within the group:
If all returns in a group coincide, for all members; we term such groups degenerate, as they consume rollout computation but contribute no gradient. Using these advantages, GRPO updates the policy by minimizing a PPO-style (Schulman et al., 2017) clipped surrogate:
Method
Stochastic Flow Map Policies
Introducing stochasticity via Itô maps. While standard flow maps enable one-step generation, their stochasticity is limited to the initial random noise . To this end, we extend the flow map in eq. (4) to an Itô map, which takes an intermediate action and a realization of a standard Brownian motion over as inputs and predicts a future action on the same stochastic path Pan et al. (2026). Evaluated from to , it defines a stochastic one-step policy. In practice, we condition the network on a finite-dimensional representation of .
Following the Itô-map construction of Pan et al. (2026), its martingale component captures the Brownian noise accumulated from time 0 to , weighted by the diffusion profile and scaled by the noise level :
where is the diffusion profile and is the noise scale. The Itô map then combines a path-dependent average drift with the martingale increment :
where approximates the path-conditioned average finite-variation term in the Itô-map decomposition. At , the martingale vanishes and the stochastic construction reduces to the deterministic flow map in eq. (4). For each action coordinate, we use the truncated Karhunen–Loève expansion , where . The increment is then linear in the coefficients,
so holds pathwise for , and the same conditions on every subinterval. In practice, we discretize the time-changed Brownian representation rather than the driving Brownian motion directly, using for ; see section D. All finite-dimensional density calculations use these coefficients. For , the stochastic policy is
Self-distillation. Starting from the pretrained velocity field, we train the Itô map without an external teacher [5] (Ziakas et al. 2026). On the diagonal , the drift is independent of the Brownian path, and we evaluate it using the zero path . Compared with standard flow matching, its regression target includes the score correction of the marginal-preserving SDE. Under , this target becomes ; the derivation is given in section B. Following Boffi et al. [4], we consider Lagrangian (LSD), Eulerian (ESD), and progressive (PSD) self-distillation. These enforce consistency at the transported endpoint, at the starting point, and between direct and composed jumps, respectively (see section A).
In this work, we use Lagrangian self-distillation (LSD) [4] (Pan et al. 2026). At fixed and Brownian coefficients , let
denote the transported action. LSD matches the endpoint derivative of the finite-variation component to the diagonal drift evaluated at . The diagonal loss is
The LSD loss is
Here, denotes a stop-gradient. The derivative is taken at fixed and on the finite-variation component
so the martingale increment itself is not differentiated.
At , the score correction and martingale component vanish, and eq. (14) reduces to deterministic flow-map self-distillation. We use LSD because the Eulerian objective does not identify the Itô map (Pan et al., 2026, Prop. 2), while the progressive objective requires two sequential forward passes. The full objective is ; implementation details are provided in section D.
On-Policy Optimization of Stochastic Flow Maps
Intractable action likelihood. To optimize our policy via likelihood-ratio RL, we need a tractable closed-form evaluation for the conditional action density . Since continuous flow-based models lack this, prior work converts the flow model into a discrete-time Markov process and computes likelihood ratios over intermediate Gaussian denoising steps rather than the final action [7] (Zhang et al. 2025). Our approach bypasses this limitation by deriving an auxiliary-variable conditional density. To do so, we isolate the Gaussian component of the Brownian path that generates the martingale increment in eq. (11), enabling us to treat the one-step action as an invertible transformation of a single Gaussian variable, conditioned on the remaining path randomness.
To isolate the martingale increment from the rest of the Brownian path, we project the path’s random coefficients into two independent components. Let be the full-interval coefficient from eq. (10), and define its unit direction . For each action coordinate , the increment is . We decompose the underlying standard Gaussian into a component parallel to (which perfectly determines ) and an orthogonal remainder :
Let and . We group all the randomness that we want to condition on into a single auxiliary variable . Because orthogonal projections of a standard Gaussian are independent, the additive noise follows with variance , completely independent of . Since the initial latent is also drawn independently, is independent of the entire auxiliary variable .
Conditioned on , the Brownian path is fully determined by , so the action in eq. (11) is a function of alone,
where is the full-jump drift evaluated on the reconstructed path and is its Jacobian. Given a stored rollout , the following proposition gives the density of under updated parameters , using the change of variables of invertible residual networks [2, 8].
Proposition 1 (Conditional action density). Assume and . Then independently of , with , the map is invertible, and
where .
Proof sketch. The action is generated by . The randomness comes from the initial latent and the Brownian coefficients . As shown above, splitting isolates the additive noise , which is entirely independent of the auxiliary variable . The full proof is in section C.
To evaluate eq. (17) in practice, we approximate the updated preimage with one fixed-point iteration initialized at the stored rollout increment, and retain the first-order term . We estimate this trace with a single Hutchinson probe, requiring one additional Jacobian-vector product, and reuse the same probe for the old and updated policies to reduce variance in their score difference. The resulting score approximates the conditional log-density; the inverse update and scoring formula are provided in section C and eq. (30).
Likelihood ratios with auxiliary variables. Theorem 1 gives the action density conditioned on the auxiliary variable , rather than the marginal density . Let denote the distribution of , which is independent of . The marginal policy is
We equivalently define the augmented policy
The augmented policy samples the pair , while the environment receives only . Marginalizing out therefore recovers eq. (18), so and induce the same trajectory distribution and expected return .
Because is independent of ,
Consequently, before the approximations in section C, the policy-gradient estimator constructed from the conditional density is unbiased for . The augmented and marginal formulations optimize the same expected-return objective, although their likelihood-ratio variances and clipped surrogate objectives can differ.
The auxiliary variable sampled during rollout is stored and reused when rescoring the recorded action; the Brownian path is never resampled. Using the approximate score from eq. (30), the likelihood ratio is
where indexes trajectories and indexes decision steps.
GRPO. For each update, we collect groups of trajectories from the same task and initial environment state. Group members use independent initial latents and Brownian coefficients , and the auxiliary variables sampled at every decision step are stored for policy rescoring. Episode returns are converted into the group-normalized advantages in eq. (6). The Brownian path increases action diversity beyond that provided by alone. Without this additional stochasticity, the policy more frequently produces zero-variance groups, for which all advantages vanish; section 4.3 measures this effect. We minimize the clipped GRPO surrogate in eq. (7) using the approximate likelihood ratio from eq. (21). The stored auxiliary variable is reused when evaluating both the old and updated policies. Training details are provided in section D.
Experiments
The experiments evaluate the end-to-end performance of VLA-SSD and analyze its underlying mechanics. We first evaluate the final on-policy reinforcement learning success rates and computational efficiency against multi-step baselines (section 4.2). We then analyze the source of the one-step policy’s sample diversity and its impact on the GRPO signal (section 4.3). Finally, we ablate the initial self-distillation phase to isolate the impact of different consistency objectives on the base policy (section 4.4).
Experimental Setup
Model and checkpoints. We use (Physical Intelligence et al., 2025), a 3.3B-parameter VLA model with a 300M-parameter action expert serving as the flow-matching head described in section 3. The vision–language backbone is kept frozen throughout training. All runs start from the SFT checkpoints released with ([7]), so differences from its published numbers come from the post-SFT procedure alone. On LIBERO, reinforcement learning starts from the few-shot checkpoint (77.1%), as in ; on MetaWorld the released checkpoint scores 51.8% under our protocol.
Benchmarks and protocol. LIBERO ([18]) is evaluated on its four ten-task suites (spatial, object, goal, long) with 500 episodes per suite; a four-suite mean therefore averages 2,000 episodes. and has a standard error of 0.5–0.7 points. MetaWorld MT50 (Yu et al., 2020) is evaluated on 50 tasks 10 fixed reset states at horizon 100, following . Unless stated otherwise, success rates are deterministic evaluations.
| Policy | Sampler | Passes | Forward (ms) | LIBERO | MetaWorld |
| SFT | ODE, | 10 | 58.4 | 77.1 | 51.8 |
| , GRPO | SDE, | 4 | 24.5 | 91.5 | — |
| , PPO | SDE, | 4 | 24.5 | 98.3 | 70.7 |
| Itô student (ours) | Itô, | 1 | 7.6 | 68.6 | 49.0 |
| Itô + GRPO (ours) | Itô, | 1 | 7.6 | 93.0 | 72.0 |
Table 1. Main results. Success on the LIBERO four-suite mean and MetaWorld MT50 (ours at ); passes is action-expert evaluations per chunk, forward its time per chunk on one A100. rows are its published numbers from the same SFT checkpoints. Bold is best per column.
Baselines. We compare against ’s GRPO with Flow-SDE and PPO with Flow-Noise at their operating depth , using the results that work reports on the same checkpoints and protocol; against the deterministic flow map distilled with the same recipe at ; and against the released ODE sampler at and . We also evaluate a four-step ReinFlow-style PPO baseline (Appendix § 13).
Training. Distillation uses the Lagrangian objective, with the sampling interval gradually expanded over the first quarter of training, Karhunen–Loève modes, and . We use the profile , for which the diagonal correction in eq. (13) equals the constant . GRPO runs in RLinf (Yu et al., 2025) on four A100 GPUs with groups of rollouts sharing a task and initial state. Section D provides all settings.
Main Results: On-Policy RL Performance
On both benchmarks in table 1, the performance improvement from GRPO far exceeds the initial drop caused by one-step distillation. The final policy improves over the released SFT checkpoint by 15.9 percentage points on LIBERO and 20.2 points on MetaWorld. Under GRPO, our one-step policy achieves 93.0% on LIBERO, outperforming ’s four-step GRPO policy (91.5%). While four-step PPO achieves a higher 98.3% on LIBERO, it requires the substantial overhead of training an additional vision–language critic. On MetaWorld, our one-step GRPO policy achieves 72.0%, outperforming the four-step PPO baseline (70.7%).
Analysis: Stochasticity and Sample Diversity
Source of sample diversity. Table 2 distinguishes variation induced by the initial latent from variation introduced during generation by fixing (Figure 2). Both the ODE sampler and the deterministic map are deterministic conditional on , and their effective rank drops to zero when is fixed. With independently sampled latents, the ODE sampler has an effective rank of 3.2. The Itô map retains a similar rank with fixed and free latents (18.5 and 18.2), showing that substantial diversity can be generated through the Brownian path alone. Its fixed-latent rank exceeds that of four-step Flow-SDE (14.3) and matches that of isotropic Gaussian output noise. Their effects on return variation nevertheless differ, as examined next.

Figure 2. End-effector trajectories with (a) independent and (b) shared initial latents . Fixing collapses the deterministic samples, while the stochastic samplers retain diversity. The distilled models shown use an earlier student checkpoint.
Zero-variance groups. The group-relative reward term in GRPO provides no learning signal when all members of a group obtain identical returns. Success rates are similar across samplers (83.1–84.6%), whereas the fraction of zero-variance groups varies. The ODE sampler and deterministic map produce zero return variance in 62.5% and 56.2% of groups, respectively. Isotropic output noise reduces this fraction to 47.9%, and four-step Flow-SDE to 45.8%. The Itô map yields the lowest observed fraction, 35.4%, despite having a similar effective rank to the isotropic control. This suggests that its sample diversity translates more effectively into within-group return variation. Under binary rewards, a lower zero-variance fraction corresponds to a larger proportion of groups providing a group-relative learning signal
Retention under training noise. Exploration noise is useful only if the noisy policy still succeeds often enough for groups to contain both outcomes. On MetaWorld the Itô student attains 36.8% under the training noise against 49.0% deterministically, a retention of 75%; after GRPO the figures are 55.4% and 72.0%, a retention of 77%. For comparison, reports that Flow-SDE at retains 14.7% of its deterministic success and 86% at , which is why that method is trained at four steps.
| Sampler | Effective rank | GRPO groups | ||
| Free | Fixed | SR (%) | Zero-var. | |
| ODE, | 3.2 | 0.0 | 84.4 | 0.625 |
| Flow-SDE, | 13.7 | 14.3 | 84.6 | 0.458 |
| Deterministic map, | 10.3 | 0.0 | 84.4 | 0.562 |
| + noise | 18.5 | 18.5 | 84.4 | 0.479 |
| Itô map, , | 18.2 | 18.5 | 83.1 | 0.354 |
Table 2. Sample diversity and GRPO signal. Effective rank measures action diversity with free or fixed initial latent ; zero-variance is the fraction of rollout groups with identical returns. SR denotes success rate. Multi-step samplers use the SFT checkpoint; one-step samplers use its distilled student.
Exploration during GRPO. All runs on each benchmark started from the same distilled checkpoint. We evaluated success after 100 optimizer updates with exploration noise turned off. First, we varied only the rollout noise scale. The default gave the highest success rate on both benchmarks among the three scales tested. Second, we trained a one-step control with the same GRPO settings, disabling the Brownian-path input and increment and adding independent isotropic Gaussian noise to the output. The Itô policy led by 2.0 percentage points on LIBERO and 0.4 on MT50.
| Training condition | LIBERO success (%) | MT50 success (%) |
| (a) Rollout noise scale for the Itô policy | ||
| 69.10 | 58.80 | |
| (default) | 73.50 | 59.40 |
| 65.15 | 55.20 | |
| (b) One-step GRPO policies | ||
| Itô map | 73.50 | 59.40 |
| Isotropic output noise | 71.50 | 59.00 |
Table 3. Effect of rollout noise scale and noise source on task success after 100 GRPO optimizer updates.
Ablation: Impact of Self-Distillation
Distillation loss. Distilled from the full-demonstration checkpoint, the one-step Itô student attains 96.55% on the LIBERO four-suite mean against the ten-step teacher’s 96.85%, with the residual concentrated on the long-horizon suite. Evaluated at and , the same checkpoint varies by less than points, indicating that the student approximates the solution operator over the interval rather than a single endpoint. The loss is larger when the teacher is weaker: 8.5 points from the few-shot LIBERO checkpoint (77.1 → 68.6) and 2.8 points on MetaWorld (51.8 → 49.0), the two starting points of § 4.2.
The LIBERO comparison in table 1 uses ten ODE steps for the SFT teacher (77.1%) and one step for the distilled student (68.6%). Holding sampling depth fixed at one step, self-distillation raises pre-RL success from 40.55% to 68.60% (section E).
The order of training also matters: distilling from the converged SFT checkpoint rather than jointly from the base raised a matched-budget student from 12.2% to 42.7% at .
Consistency objective and map. Table 4 compares deterministic and Itô maps under three consistency objectives using the teacher. The Lagrangian and progressive objectives yield one-step performance reductions of 2.9–6.2 percentage points, with the largest reductions on the long-horizon suite. The Eulerian objective exhibits substantially larger reductions, consistent with reduced accuracy of first-order extrapolation from the diagonal as increases. LSD achieves the highest mean success for both maps and is used throughout. Under LSD, the Itô map achieves 88.95%, compared with 91.30% for the deterministic map, indicating a 2.35-percentage-point performance trade-off associated with the stochastic formulation. Its exploration properties are analyzed in § 4.3.
| Map | Objective | Forward (ms) | spatial | object | goal | long | mean |
| teacher, | — | 58.4 | 96.8 | 98.8 | 95.8 | 85.2 | 94.15 |
| Deterministic flow map | LSD | 7.6 | 96.2 | 96.0 | 94.0 | 79.0 | 91.30 |
| PSD | 7.6 | 96.4 | 95.4 | 93.6 | 77.8 | 90.80 | |
| ESD | 7.6 | 90.2 | 76.8 | 77.0 | 26.0 | 67.50 | |
| Itô flow map, | LSD | 7.6 | 94.2 | 93.6 | 91.0 | 77.0 | 88.95 |
| PSD | 7.6 | 96.6 | 91.2 | 90.8 | 73.2 | 87.95 | |
| ESD | 7.6 | 85.4 | 76.8 | 62.6 | 37.2 | 65.50 |
Table 4. One-step distillation from on LIBERO, 500 episodes per suite, all students evaluated at . The teacher is ’s full-dataset SFT checkpoint at ; the deterministic map under ESD is MeanFlow [12]. Forward is the action head’s time per chunk on one A100. Bold marks the best objective for each map; the standard error on each mean is about 0.7 points.
Related Work
Reinforcement fine-tuning of generative policies
Building on pretrained VLAs [6, 13, 3] (Physical Intelligence et al. 2025), online RL has been explored for autoregressive action generation [20, 15]. For diffusion and flow policies, DPPO optimizes the denoising chain with PPO (Ren et al., 2025), while ReinFlow and introduce stochastic denoising transitions for likelihood-based fine-tuning, retaining multi-step generation [7] (Zhang et al. 2025). In image generation, Flow-GRPO and DanceGRPO apply GRPO through ODE-to-SDE conversion [19] (Xue et al. 2025); Flow-Map GRPO extends this approach to few-step generators through anchored stochastic composition [16]. Complementary critic-based approaches optimize a fixed policy’s latent-noise distribution (DSRL) (Wagenmaker et al., 2025), update flow maps through critic-guided trust regions (Flow Map -Guidance) (Ziakas et al., 2026), or combine value calibration with action filtering (FORCE) (Zhang et al., 2026). Our work enables GRPO fine-tuning of a one-step stochastic VLA through its conditional action density, avoiding a multi-step denoising chain.
One-step generation and stochastic flow maps
Diffusion and flow policies model multimodal robot actions through iterative sampling [10, 3]. Consistency models, shortcut models, MeanFlow, and flow-map methods replace iterative integration with learned finite-time transport [11, 12, 4, 1] (Song et al. 2023), including self-distillation from the map’s own diagonal [5]. In robotics, Consistency Policy and SnapFlow distill policies into few-step or one-step samplers (Prasad et al., 2024; Luan et al., 2026), while Let It Be Simple enables one-step decoding through high-noise training [9]. These maps are deterministic given the initial noise. Stochastic extensions include Meta Flow Maps for reward-conditioned posterior sampling (Potaptchik et al., 2026) and Itô maps for Brownian-path-conditioned SDE transitions (Pan et al., 2026). We build on Itô maps to combine one-step VLA generation with explicit exploration and conditional-density-based on-policy optimization.
Conclusion
We presented VLA-SSD, which distills a flow-matching VLA into a one-step stochastic policy based on Itô flow maps and fine-tunes it with GRPO. The Brownian path provides exploration with an explicit noise scale, and the derived conditional density makes the policy usable in likelihood-based on-policy updates. On LIBERO and MetaWorld MT50, the result matches multi-step GRPO fine-tuning with using just one action-expert evaluation per chunk instead of four. Regarding limitations, our approximate likelihood scoring relies on a first-order truncation of the log-determinant. In addition, while GRPO avoids the heavy memory overhead of a vision–language critic, it does not match the peak absolute sample efficiency of critic-based PPO on LIBERO. However, because physical robot rollouts operate in real-time and constitute the primary bottleneck of on-policy RL, the faster rollout generation provided by VLA-SSD is particularly valuable for real-world deployment. Translating this framework to real-robot fine-tuning is the natural next step. Finally, investigating adaptive step sizing—where the model defaults to the one-step Itô map but allocates additional sampling steps for complex, high-precision maneuvers—presents an exciting direction for bridging one-step and multi-step continuous control.
Acknowledgments
This work was supported by UKRI (EP/Y037111/1) as part of the ProSafe project (EU Horizon 2020, MSCA, grant no. 101119358).
Consistency Objectives for Flow-Map Distillation
This appendix states the three self-distillation objectives of section 3.1 in full, in the notation of section 2, and records the properties that motivate the choice made in the main text. Throughout, the flow map is parameterized as for , the diagonal is trained by in eq. (13), and sg denotes a stop-gradient; conditioning on and the superscript are suppressed where unambiguous.
Origin of the conditions. The exact solution operator of the ODE satisfies the jump condition and the semigroup property for . Differentiating the jump condition with respect to the endpoint gives ; differentiating with respect to the start time and using gives . Each of these identities, together with the semigroup property, yields a residual that vanishes for the exact map and can be minimized for the parameterized map, with the diagonal serving as the teacher [4, 5] (Ziakas et al. 2026).
Lagrangian self-distillation (LSD). Writing for the transported state,
The condition matches the endpoint derivative of the map to the instantaneous velocity evaluated at the transported state, so the map is supervised along the trajectory it generates. The time derivative is obtained by a Jacobian-vector product with respect to the scalar , and the target requires one additional evaluation of the diagonal at .
Eulerian self-distillation (ESD).
The condition transports the map along the characteristics of the diagonal field at the start point, so both the residual and its target are evaluated at rather than at the transported state. The spatial term is a Jacobian-vector product in the action argument. For the average-velocity parameterization, removing the stop-gradient and expanding shows that the ESD gradient coincides with that of MeanFlow (Geng et al. [12]) up to the choice of teacher for the instantaneous velocity; see Ziakas et al. (2026, App. B).
Progressive self-distillation (PSD).
The condition enforces the semigroup property directly: a jump from to must agree with the composition of two shorter jumps through an intermediate time . It requires no derivative of the network but two sequential forward passes on the left-hand side, and it constrains the off-diagonal map only through other off-diagonal evaluations, so the diagonal enters indirectly.
Diagonal Target of the Itô Map
Throughout this derivation, we fix the conditioning state and assume independent endpoint coupling, , with . We suppress the dependence on below.
The diagonal loss in eq. (13) regresses the zero-path diagonal drift onto the drift of the SDE that shares marginals with the flow-matching ODE. For the interpolant , the marginal path of the ODE is also the marginal path of the SDE
for any diffusion profile , since the added diffusion and the score correction cancel in the Fokker–Planck equation. Conditioned on the endpoint , the interpolant is Gaussian with mean and covariance , so the conditional score is . Substituting the conditional velocity and the conditional score gives the regression target
which is the target in eq. (13); as for conditional flow matching, regressing onto the conditional target recovers the marginal drift in expectation. The coefficient remains bounded as provided that remains bounded. With the profile used in our experiments, the coefficient reduces to the constant , and the Brownian variance accumulated over is in the continuum; section D gives the truncated value used in practice.
Density diagnostics. At the end of distillation the Brownian pathway’s output projection is of order ; consequently the variance of the single-probe trace estimator and the truncation error of the first-order log-determinant in eq. (30), under , are both below nats per chunk. At the first minibatch of every run the ratio equals one to floating-point precision and the mean Jacobian correction is indistinguishable from zero. These values are measured at the start of training; the pathway norm was not logged during reinforcement learning.
Proof of the Conditional-Density Proposition
Proof of theorem 1. For each action coordinate, and are independent Gaussian projections. Since , the coordinates of are independent and , where . Because is sampled independently of the Brownian coefficients, is independent of .
Fix and an action . A preimage of under satisfies
The right-hand side is a contraction under the assumption of the theorem, so the Banach fixed-point theorem guarantees a unique solution . Therefore, is invertible. Its Jacobian is
which is nonsingular because . Applying the change-of-variables formula to gives eq. (17).
The proposition provides the exact density conditioned on the auxiliary variable ; the marginal action density remains intractable. We next approximate the inverse and log-determinant required to evaluate the conditional density.
Approximating policy scoring. After a policy update, the stored increment need not generate the recorded action under the updated parameters. The updated preimage satisfies the fixed-point equation
Starting from the stored increment , we approximate the updated preimage using one fixed-point iteration:
Under the contraction assumption of theorem 1, the Jacobian correction to the conditional log-density admits the convergent trace expansion
We retain its first-order term and estimate the trace using a Hutchinson probe :
The final term requires one Jacobian-vector product. At , the fixed-point update recovers the stored increment exactly, . We use the same probe for the old and updated policies to reduce variance in the likelihood ratio. Thus, eq. (30) is used as an approximate policy score rather than an exact likelihood.
Scoring accuracy across RL checkpoints. At early, middle, and late stages of GRPO training, we replayed eight LIBERO action chunks for each pair of pre- and post-update policies. We recomputed the approximate log-ratios in FP32 and compared them with reference values obtained using a converged inverse and an explicit Jacobian. On these samples, the mean absolute error was below 0.0002 at all three checkpoints.
| RL checkpoint | Mean absolute log-ratio error (FP32) |
| Early | 0.000181 |
| Middle | 0.000070 |
| Late | 0.000061 |
Table 5. FP32 log-ratio error across GRPO checkpoints.
Implementation Details
Model. The action expert of is extended with two additive pathways, each a two-layer MLP of width 1024 with swish activation and a zero-initialised output layer, so that the extended model reproduces the pretrained velocity at initialisation. The interval pathway embeds with a sinusoidal positional encoding (periods to ); the path pathway projects the martingale increment through the action input projection, mean-pools it over the horizon, adds the interval encoding, and passes the sum through the MLP. Both outputs enter the expert through its adaptive RMS-norm conditioning, not as additional tokens. The vision–language backbone (SigLIP and Gemma-2B) is frozen; the action expert, its projections, and the two pathways are trained, all in bfloat16 under FSDP without parameter sharding. The expert predicts a chunk of 10 steps in a 32-dimensional padded action space; the first 5 steps of the first 7 dimensions are executed, and the conditional density is evaluated on these coordinates only.
Brownian path. Each of the coordinates carries an independent standard Brownian motion on , represented by its first Karhunen–Loève modes, with , , and . The diffusion profile is implemented as the time change , so that the increment of eq. (10) is the closed-form difference with ; no quadrature is used. For the full jump, , so the realized variance in eq. (17) is rather than its continuum limit ; the same truncation is used at training and at scoring, since a mismatch would bias every importance ratio.
Distillation. The student is initialised from the SFT checkpoint and trained for 1,000 optimizer steps on 4 A100 GPUs with global batch 128 (micro-batch 4), AdamW with , , weight decay , and gradient clipping at 1.0. The learning rate is warmed up linearly to over the run. The objective is with , where the consistency term is switched on after the first 5% of steps; the consistency residual is not normalised and is typically one to two orders of magnitude larger than the diagonal residual, so does not correspond to an equal weighting of the two terms. Interval endpoints are drawn as , , where mid and are the midpoint and length of the interval between two independent draws, and increases linearly from 0 at 5% of the run to 1 at 25%; a quarter of each batch is forced to the full interval . The noise scale is ramped linearly from 0 to over the first 30% of the run and held fixed thereafter. The checkpoint at step 600 is used.
Reinforcement learning. GRPO runs in RLinf on 4 A100 GPUs. Each iteration collects 8 groups of rollouts on LIBERO (32 parallel environments, two rollout epochs) and 32 groups on MetaWorld, with the same task and initial state within a group, , and applied in full from the first iteration. Episodes run for at most 500 environment steps on LIBERO and 100 on MetaWorld; the return is 1 at the step of first success and 0 otherwise. Groups whose mean return lies outside are discarded, which removes degenerate groups before the update. Advantages use and are normalised across the batch; the clipped objective uses with a dual clip at 3.0, token-mean aggregation, no entropy bonus, and no KL penalty. Each iteration performs one pass over the retained rollout chunks in minibatches of 256 with micro-batch 64, using AdamW with learning rate , , weight decay , and gradient clipping at 1.0, without warmup or decay. The Jacobian-vector products in eqs. (14) and (30) are computed by forward-mode automatic differentiation.
Training budget and wall-clock time. The original Itô + GRPO runs collected 64 rollouts per iteration on LIBERO and 256 on MT50. We report the full training budgets below.
| Benchmark | RL iterations | Optimizer updates | Rollouts collected |
| LIBERO | 600 | 15,000 | 38,400 |
| MT50 | 300 | 7,500 | 76,800 |
Table 6. Training budgets for the original Itô + GRPO runs.
End-to-end RL time includes policy scoring. The Itô times are estimates for the original 600-iteration LIBERO run and 300-iteration MT50 run.
| Training method | LIBERO (hours) | MT50 (hours) |
| Itô + GRPO (estimated) | 33.3 | 49.3 |
| Isotropic output noise + GRPO | 74.8 | 72.1 |
Table 7. End-to-end GRPO training time, including policy scoring.
Evaluation. LIBERO is evaluated on the four ten-task suites with 500 episodes per suite in 500 parallel environments; MetaWorld MT50 on 50 tasks with 10 fixed reset states each, following . Forward times in table 1 are wall-clock times of the action head per chunk on one A100.
Successful Rollouts. We visualize successful rollouts of the one-step VLA-SSD policy (Itô map, , at evaluation) on the four LIBERO suites and on MetaWorld MT50. Each figure shows uniformly spaced frames from a single successful episode, from the initial state to the step at which success is registered.

Figure 3. LIBERO-Spatial, task 4. Pick up the black bowl in the top drawer of the wooden cabinet and place it on the plate.

Figure 4. LIBERO-Spatial, task 7. Pick up the black bowl on the stove and place it on the plate.

Figure 5. LIBERO-Object, task 0. Pick up the alphabet soup and place it in the basket.

Figure 6. LIBERO-Object, task 7. Pick up the milk and place it in the basket.

Figure 7. LIBERO-Goal, task 0. Open the middle drawer of the cabinet.

Figure 8. LIBERO–Goal, task 5. Push the plate to the front of the stove.

Figure 9. LIBERO–Long, task 2. Turn on the stove and put the moka pot on it.

Figure 10. LIBERO–Long, task 4. Put the white mug on the left plate and put the yellow and white mug on the right plate.

Figure 11. assembly-v3 (MetaWorld MT50). Pick up a nut and place it onto a peg.

Figure 12. basketball-v3 (MetaWorld MT50). Dunk the basketball into the basket.

Figure 13. button-press-topdown-v3 (MetaWorld MT50). Press a button from the top.

Figure 14. drawer-open-v3 (MetaWorld MT50). Open a drawer.

Figure 15. shelf-place-v3 (MetaWorld MT50). Pick and place a puck onto a shelf.

Figure 16. window-open-v3 (MetaWorld MT50). Push and open a window.
Effect of Self-Distillation Before GRPO
Both LIBERO runs started from the same SFT checkpoint. We applied self-distillation before GRPO in one run and trained the other directly with one-step Itô + GRPO. We kept the GRPO settings fixed and evaluated both policies with one-step sampling and exploration noise turned off. The distilled policy started from a higher success rate and retained a 21.45-percentage-point lead after 100 optimizer updates.
| Training pipeline | Before RL | After 100 updates |
| Self-distillation followed by one-step GRPO | 68.60 | 73.50 |
| Direct one-step GRPO | 40.55 | 52.05 |
Table 1 (PDF p. 19). LIBERO success (%).
Comparison with Four-Step ReinFlow-Style PPO
On each benchmark, both methods started from the same SFT checkpoint before method-specific training. We evaluated both policies on the same initial states after 100 optimizer updates. At this checkpoint, VLA-SSD led by 32.95 percentage points on LIBERO and 4.00 points on MT50.
| Benchmark | Method | Passes | Success (%) | Head time (ms/chunk) |
| LIBERO | VLA-SSD + GRPO | 1 | 73.50 | 7.6 |
| ReinFlow-style PPO | 4 | 40.55 | 68.5 | |
| MT50 | VLA-SSD + GRPO | 1 | 59.40 | 7.6 |
| ReinFlow-style PPO | 4 | 55.40 | 80.0 |
Table 2 (PDF p. 19).
References
- [1]Michael S. Albergo, Nicholas M. Boffi, and Eric Vanden-Eijnden. Stochastic Interpolants: A Unifying Framework for Flows and Diffusions. arXiv preprint arXiv:2303.08797, 2023. URL https://arxiv.org/abs/2303.08797. 3, 9
- [2]Jens Behrmann, Will Grathwohl, Ricky T. Q. Chen, David Duvenaud, and Joern-Henrik Jacobsen. Invertible residual networks. In Proceedings of the 36th International Conference on Machine Learning, volume 97, pp. 573–582. PMLR, 2019. 5arxiv.org/abs/1811.00995
- [3]Kevin Black, Noah Brown, Danny Driess, Adnan Esmail, Michael Equi, Chelsea Finn, Niccolo Fusai, Lachy Groom, Karol Hausman, Brian Ichter, Szymon Jakubczak, Tim Jones, Liyiming Ke, Sergey Levine, Adrian Li-Bell, Mohith Mothukuri, Suraj Nair, Karl Pertsch, Lucy Xiaoyang Shi, James Tanner, Quan Vuong, Anna Walling, Haohuan Wang, and Ury Zhilinsky. π₀: A Vision-Language-Action Flow Model for General Robot Control. arXiv preprint arXiv:2410.24164, 2024. URL https://arxiv.org/abs/2410.24164. 1, 9
- [4]Nicholas M. Boffi, Michael S. Albergo, and Eric Vanden-Eijnden. Flow map matching with stochastic interpolants: A mathematical framework for consistency models. arXiv preprint arXiv:2406.07507, 2024. URL https://arxiv.org/abs/2406.07507. 1, 3, 4, 9, 13
- [5]Nicholas M. Boffi, Michael S. Albergo, and Eric Vanden-Eijnden. How to build a consistency model: Learning flow maps via self-distillation. arXiv preprint arXiv:2505.18825, 2025. URL https://arxiv.org/abs/2505.18825. 1, 4, 9, 13
- [6]Anthony Brohan, Noah Brown, Justice Carbajal, Yevgen Chebotar, Xi Chen, Krzysztof Choro-manski, Tianli Ding, Danny Driess, Avinava Dubey, Chelsea Finn, Pete Florence, Chuyuan Fu, Montse Gonzalez Arenas, Keerthana Gopalakrishnan, Kehang Han, Karol Hausman, Alexander Herzog, Jasmine Hsu, Brian Ichter, Alex Irpan, Nikhil Joshi, Ryan Julian, Dmitry Kalashnikov, Yuheng Kuang, Isabel Leal, Lisa Lee, Tsang-Wei Edward Lee, Sergey Levine, Yao Lu, Henryk Michalewski, Igor Mordatch, Karl Pertsch, Kanishka Rao, Krista Reymann, Michael Ryoo, Grecia Salazar, Pannag Sanketi, Pierre Sermanet, Jaspiar Singh, Anikaith Singh, Radford Socratic, Huong Tran, Vincent Vanhoucke, Quan Vuong, Ayzaan Wahid, Stefan Welker, Paul Wohlhart, Jialin Wu, Fei Xia, Ted Xiao, Peng Xu, Sichun Xu, Tianhe Yu, and Brianna Zitkovich. RT-2: Vision-Language-Action Models Transfer Web Knowledge to Robotic Control. arXiv preprint arXiv:2307.15818, 2023. URL https://arxiv.org/abs/2307.15818. 1, 9
- [7]Kang Chen, Zhihao Liu, Tonghe Zhang, Zhen Guo, Si Xu, Hao Lin, Hongzhi Zang, Xiang Li, Quanlu Zhang, Zhaofei Yu, Guoliang Fan, Tiejun Huang, Yu Wang, and Chao Yu. π_RL: Online RL Fine-tuning for Flow-based Vision-Language-Action Models. arXiv preprint arXiv:2510.25889, 2025. URL https://arxiv.org/abs/2510.25889. 1, 5, 6, 9
- [8]Ricky T. Q. Chen, Jens Behrmann, David K. Duvenaud, and Joern-Henrik Jacobsen. Residual flows for invertible generative modeling. In Advances in Neural Information Processing Systems, volume 32, 2019. 5arxiv.org/abs/1906.02735
- [9]Yitong Chen, Shiduo Zhang, Jingjing Gong, and Xipeng Qiu. Let It Be Simple: One-Step Action Generation for Vision-Language-Action Models. arXiv preprint arXiv:2606.05737, 2026. URL https://arxiv.org/abs/2606.05737. 9
- [10]Cheng Chi, Siyuan Feng, Yilun Du, Zhenjia Xu, Eric Cousineau, Benjamin CM Burchfiel, and Shuran Song. Diffusion Policy: Visuomotor Policy Learning via Action Diffusion. In Proceedings of Robotics: Science and Systems, 2023. doi: 10.15607/RSS.2023.XIX.026. URL https://www.roboticsproceedings.org/rss19/p026.html. 9DOI
- [11]Kevin Frans, Danijar Hafner, Sergey Levine, and Pieter Abbeel. One Step Diffusion via Shortcut Models. In The Thirteenth International Conference on Learning Representations, 2025. URL https://openreview.net/forum?id=OlzB6LnXcS. 3, 9arxiv.org/abs/2410.12557
- [12]Zhengyang Geng, Mingyang Deng, Xingjian Bai, J. Zico Kolter, and Kaiming He. Mean Flows for One-step Generative Modeling. arXiv preprint arXiv:2505.13447, 2025. URL https://arxiv.org/abs/2505.13447. 1, 3, 9, 14
- [13]Moo Jin Kim, Karl Pertsch, Siddharth Karamcheti, Ted Xiao, Ashwin Balakrishna, Suraj Nair, Rafael Rafailov, Ethan Foster, Grace Lam, Pannag Sanketi, Quan Vuong, Thomas Kollar, Benjamin Burchfiel, Russ Tedrake, Dorsa Sadigh, Sergey Levine, Percy Liang, and Chelsea Finn. OpenVLA: An Open-Source Vision-Language-Action Model. arXiv preprint arXiv:2406.09246, 2024. URL https://arxiv.org/abs/2406.09246. 1, 9
- [14]Moo Jin Kim, Chelsea Finn, and Percy Liang. Fine-Tuning Vision-Language-Action Models: Optimizing Speed and Success. arXiv preprint arXiv:2502.19645, 2025. URL https://arxiv.org/abs/2502.19645. 1
- [15]Haozhan Li, Yuxin Zuo, Jiale Yu, Yuhao Zhang, Zhaohui Yang, Kaiyan Zhang, Xuekai Zhu, Yuchen Zhang, Tianxing Chen, Gangu Cui, Dehui Wang, Dingxiang Luo, Yuchen Fan, Youbang Sun, Jia Zeng, Jiangmiao Pang, Shanghang Zhang, Yu Wang, Yao Mu, Bowen Zhou, and Ning Ding. SimpleVLA-RL: Scaling VLA Training via Reinforcement Learning. arXiv preprint arXiv:2509.09674, 2025. URL https://arxiv.org/abs/2509.09674. 9
- [16]Zhiqi Li, Wen Zhang, and Bo Zhu. Flow-Map GRPO: Reinforcement Learning for Few-Step Flow-Map Generators via Anchored Stochastic Composition. arXiv preprint arXiv:2607.00535, 2026. URL https://arxiv.org/abs/2607.00535. 1, 9
- [17]Yaron Lipman, Ricky T. Q. Chen, Heli Ben-Hamu, Maximilian Nickel, and Matt Le. Flow Matching for Generative Modeling. In The Eleventh International Conference on Learning Representations, 2023. URL https://openreview.net/forum?id=PqvMRDCJ9t9. 3arxiv.org/abs/2210.02747
- [18]Bo Liu, Yifeng Zhu, Chonkai Gao, Yihao Feng, Qiang Liu, Yuke Zhu, and Peter Stone. LIBERO: Benchmarking Knowledge Transfer for Lifelong Robot Learning. In Advances in Neural Information Processing Systems 36, Datasets and Benchmarks Track, 2023. URL https://arxiv.org/abs/2306.03310. 6
- [19]Jie Liu, Gongye Liu, Jiajun Liang, Yangguang Li, Jiaheng Liu, Xintao Wang, Pengfei Wan, Di Zhang, and Wanli Ouyang. Flow-GRPO: Training Flow Matching Models via Online RL. arXiv preprint arXiv:2505.05470, 2025. URL https://arxiv.org/abs/2505.05470. 9
- [20]Guanxing Lu, Wenkai Guo, Chubin Zhang, Yuheng Zhou, Haonan Jiang, Zifeng Gao, Yansong Tang, and Ziwei Wang. VLA-RL: Towards Masterful and General Robotic Manipulation with Scalable Reinforcement Learning. arXiv preprint arXiv:2505.18719, 2025. URL https://arxiv.org/abs/2505.18719. 9
