betaFeedback

WorksComputer Science

POKE: Probe Offline, Keep Efficiently for Vision-Language-Action Models

Abstract

Visual token pruning reduces the inference cost of Vision-Language-Action (VLA) models, but attention-based scores do not directly measure a token's effect on the predicted action. We define action sensitivity as the magnitude of the first-order action change when a token is replaced by its camera's mean token and estimate it for all tokens with a single backward pass. Computing it online is costly: scoring on the full (unpruned) graph needs an extra forward pass at every control step. Scoring on the pruned forward pass is cheaper but causes scoring-graph lock-in: removed tokens lose their direct paths through later layers and tend to receive lower scores, making those locations less likely to return. On replayed observations, even with a random conditioning mask, pruned-graph scoring re-selects 59% of the mask's tokens, versus 6% expected by chance; attention histories show a similar bias. POKE (Probe Offline, Keep Efficiently) computes full-graph action sensitivities offline and distills them into a student with under 0.3M parameters. At inference, the student scores the current observation without backpropagation or previous masks, and score-guided diversity selection retains the tokens. Keeping 32 of 512 visual tokens on LIBERO, POKE achieves 95.2% success on OpenVLA-OFT and 90.8% on π0.5π_{0.5} at 1.85×1.85\times and 1.26×1.26\times speedups, outperforming attention-, diversity-, and distillation-based pruners, with larger margins on LIBERO-Plus.

Introduction

Vision-language-action (VLA) models convert camera images and language instructions into robot actions. Their visual inputs often contain hundreds of tokens, making visual token pruning a practical way to reduce inference cost. Removing tokens in early layers reduces computation in every subsequent layer, so a pruner has to decide which tokens to keep while preserving the policy’s actions. Existing methods, however, use attention as a proxy for this decision, including attention received by visual tokens [3], temporal aggregation of action-to-visual attention [15], and attention combined with inter-frame similarity [26]. These signals describe how tokens are attended to or reused. Yet the tokens a model attends to most need not be the ones that determine its actions, and none of these signals directly measures how changing a token changes the resulting action.

We therefore define token importance by action sensitivity: the magnitude of the first-order change in the action when a visual token is replaced with its camera-wise mean representation. Since importance is defined at the policy output, it applies to both parallel-regression and flow-based action heads, using the final action chunk for the latter. In Figure 1a, attention-based masks keep only the bowl and gripper, whereas action sensitivity also keeps the target plate.

Motivation on OpenVLA-OFT showing retained visual tokens and online probe scores

Figure 1. Motivation on OpenVLA-OFT. (a) LIBERO-Goal, 64/512 tokens; rows: third-person/wrist; clear patches: retained. Attention misses the target plate kept by action sensitivity. (b) LIBERO-Long, B=32B = 32, 9 closed-loop episodes (mean, IQR). Left: cumulative number of distinct tokens ever retained; pruned-graph scoring keeps returning to the same tokens. Right: share of score assigned to tokens outside the previous mask; 19% on the pruned graph vs. 77% on the full graph (dotted: uniform scores).

Computing exact sensitivities requires one backward pass per action dimension: 56 for a chunk of eight 7-D actions in OpenVLA-OFT. Projecting the action onto one shared random direction reduces this to a single backward pass for all tokens. Even so, scoring online on the full graph needs an extra unpruned forward and backward pass per control step, running 2.4× and 3.1× slower than the unpruned OpenVLA-OFT and π0.5\pi_{0.5} policies, which defeats the purpose of pruning.

A cheaper alternative scores tokens on the pruned forward pass and uses the scores to select the next frame’s tokens. Each frame enters with all visual tokens, but the previous mask removes some before the later layers, so those tokens lose their direct paths to the action and tend to score lower. When these scores set the next mask, removed locations are less likely to return. We call this scoring-graph lock-in (Figure 1b). Even when the previous mask is drawn at random, pruned-graph scores re-select 59% of its tokens, against 6% expected by chance, and switching to full-graph scoring raises closed-loop success (§4.5). The same lock-in appears in the attention histories that VLA pruners reuse from earlier control steps [15, 24]. Scores therefore need a path independent of prior pruning decisions.

We propose POKE (Probe Offline, Keep Efficiently), which separates importance estimation from inference. POKE computes full-graph action-sensitivity labels offline on observations from the policy’s fine-tuning demonstrations and distills them into a lightweight student with fewer than 0.3M parameters, keeping the policy frozen. Labeling takes under one GPU-hour per suite. At inference, the student predicts scores from the current visual tokens, language, and state context in a single forward pass, removing online backpropagation and any conditioning on previous masks. Tokens are then selected either by top-kk ranking or by diversity selection within a high-scoring candidate set; the latter reduces redundancy while keeping sensitivity-based candidate filtering. The full policy thus provides the training signal, and the deployed selector adds only a small scoring and selection cost.

On four LIBERO suites [13], keeping 32 of 512 visual tokens, POKE reaches 95.15% success at 1.85× on OpenVLA-OFT [8] and 90.80% at 1.26× on π0.5\pi_{0.5} [5], exceeding VLA-Pruner by 5.90 and 15.75 pp, and the margin grows under the observation shifts of LIBERO-Plus (§ 4.3). In contrast, distilling attention labels instead of action sensitivity lowers success in all eight tested combinations (§ 4.4).

Our contributions are threefold:

  • Action sensitivity for visual token pruning. An action-grounded, first-order measure of token importance estimated with one shared-probe backward pass, together with an exact pairwise analysis of its ranking behavior.

  • Scoring-graph lock-in. Controlled interventions and fixed-observation replays show that pruned-graph scores favor retained tokens even without feedback, for both gradients and attention histories.

  • POKE via full-graph sensitivity distillation. A compact selector trained offline that prunes without backpropagation or previous masks, improving the success-latency trade-off over attention-, diversity-, and distillation-based baselines on parallel-regression, flow-based, and autoregressive policies on LIBERO and SimplerEnv.

Related Work

Visual Token Pruning

Visual token reduction selects or aggregates tokens depending on the input. DynamicViT [19] and EViT [12] prune or fuse less informative tokens. In vision-language models (e.g., LLaVA [14]), FastV [3] prunes visual tokens by early-layer attention, VisionZip [27] combines encoder attention with similarity-based aggregation, and Vis-Pruner [29], DART [25], and CDPruner [30] add distinctive, non-redundant, or instruction-conditioned diverse tokens. [31] argue that concentrating the budget on highly attended tokens discards holistic visual context. These works establish importance and diversity as complementary criteria; we study them for preserving actions in closed-loop.

Distillation-based Token Pruning

Distillation transfers expensive importance signals to efficient token selectors. ToP [9] distills final-layer token rankings into earlier layers, SDTP [22] trains in-network pruning modules on gradient-times-input saliency of hidden states with regression and ranking losses, and MAP [21] distills middle-layer text-to-vision attention into an input-level predictor and combines it with diversity; we compare against an adaptation of MAP in § 4. First-order expansions also underlie Taylor pruning criteria [16], which estimate the change in loss. POKE instead distills the first-order change in the predicted robot action chunk under a camera-mean token replacement, scores tokens before the policy’s Transformer layers, and examines how the computation graph used to obtain these scores affects selection over successive control steps.

Efficient Visual Token Processing in VLAs

VLA-specific methods exploit task context and temporal structure during control. Many select tokens from attention or task reused from earlier control steps: VLA-Pruner [15] fuses semantic importance with temporally aggregated action-to-visual attention, SpecPrune-VLA [24] combines previous- and current-frame cues, and ADP [17] adapts retention to manipulation stages. Others reduce computation more broadly: VLA-Cache [26] reuses key–value states of visually stable tokens, SP-VLA [11] and EfficientVLA [28] pair pruning with model scheduling or layer reduction and caching, and LightVLA [6] learns differentiable selection during policy fine-tuning. POKE keeps the policy frozen and trains a separate selector on full-graph action-sensitivity labels; at deployment it uses only the current observation, without backpropagation or previous masks.

Method

Let X=[x1,…,xN]∈RN×dX = [x_1, \ldots, x_N] \in\mathbb{R}^{N \times d} denote the visual tokens at the projector output, ℓ\ell the language input, and pp the proprioceptive state. The tokens come from CC cameras, with index set Ic\mathcal{I}_c for camera cc and camera assignment c(i)c(i) for token ii; we retain BB of them in total. The frozen policy outputs an action chunk aθ(X,ℓ,p)∈RAa_\theta(X, \ell, p) \in\mathbb{R}^{A}, which for flow-based policies is the final chunk integrated from fixed initial noise; we omit ℓ\ell and pp when unambiguous. A control step tt is one policy call on a new camera frame, with visual tokens XtX_t and state ptp_t. Figure 2 gives an overview.

Figure 2. Method overview. (a) Offline training: a frozen full-graph VLA produces single-probe action-sensitivity labels, which are distilled into a lightweight student. (b) Inference: student predicts token scores, and MMDP selects diverse tokens from scoring candidates.

Action Sensitivity

The most direct measure of a token’s importance is how much the action changes when the token is removed. Actual deletion, however, changes the sequence length and the subsequent computation graph, so it admits no local derivative. We instead replace each token with the mean representation of its camera, which keeps the length fixed while suppressing token-specific information relative to the camera mean.

bc=1∣Ic∣∑j∈Icxj,δi=stopgrad⁡(xi−bc(i)).(1)b_c = \frac{1}{\lvert\mathcal{I}_c \rvert} \sum_{j \in\mathcal{I}_c} x_j, \qquad\delta_i = \operatorname{stopgrad}(x_i - b_{c(i)}). \tag*{(1)}

The baseline is computed from the original observation and held fixed along the replacement path xi(α)=xi−αδix_i(\alpha) = x_i - \alpha\delta_i; camera-wise means avoid mixing representations from different views. A first-order expansion at α=0\alpha= 0 gives

aθ(Xi←b)−aθ(X)≈−ui,a_\theta(X^{i \leftarrow b}) - a_\theta(X) \approx-u_i,
ui:=Jiδi∈RA,Ji=∂aθ∂xi.(2)u_i := J_i \delta_i \in\mathbb{R}^{A}, \qquad J_i = \frac{\partial a_\theta}{\partial x_i}. \tag*{(2)}

We define action sensitivity as si=∥ui∥2s_i = \lVert u_i \rVert_2. Unlike a Jacobian norm, which reflects sensitivity to arbitrary input perturbations, sis_i measures the action change along the direction that pruning removes: the token’s information beyond its camera mean. Because deployment deletes tokens after several layers rather than replacing them at the projector, sis_i remains a surrogate for deletion damage (§ 4.6).

Single-Probe Estimation

Exact uiu_i for all tokens needs AA backward passes (A=56A = 56 for OpenVLA-OFT, A=70A = 70 for π0.5\pi_{0.5}). Instead, we draw a shared direction vv uniformly from the unit sphere in RA\mathbb{R}^{A}; one backward pass through v⊤aθv^{\top}a_{\theta} gives gi=Ji⊤vg_i = J_i^{\top}v for every token, and

s^i=A∣gi⊤δi∣=A∣v⊤ui∣.(3)\hat{s}_i = \sqrt{A}\lvert g_i^{\top}\delta_i\rvert= \sqrt{A}\lvert v^{\top}u_i\rvert. \tag*{(3)}

Since E[vv⊤]=I/A\mathbb{E}[vv^{\top}] = I/A, s^i2\hat{s}_i^2 is unbiased for si2s_i^2, but its relative variance 2(A−1)/(A+2)2(A-1)/(A+2) approaches two (Appendix § 16), so individual magnitudes are noisy.

Selection, however, depends only on within-camera rankings. Because vv is shared, the probability of reversing a pair depends on their sensitivity ratio and the alignment of their action effects. For si>sjs_i > s_j, let r=si/sjr = s_i/s_j. The flip probability is 2πarctan⁡(1/r)\frac{2}{\pi}\arctan(1/r) for orthogonal effects (0.20 at r=3r = 3, 0.06 at r=10r = 10), and zero for collinear effects. Empirically, single-probe scores reach a within-camera Spearman correlation of 0.80 with exact sensitivities, and the 16 highest-sensitivity tokens per camera carry 84% of the sensitivity energy (Appendix § 16).

Full-Graph Teacher

Online scoring and lock-in. Action sensitivity needs a forward pass to aθa_\theta followed by backpropagation. Scoring each new frame before pruning it would therefore require an extra unpruned forward pass at every control step, which defeats the purpose of pruning. A practical online probe instead scores within the pruned forward pass it already runs. At control step tt, the policy receives a new frame XtX_t with all visual tokens, prunes it with the mask Mt−1M_{t-1} chosen at the previous control step, and outputs the action; backpropagating through this same pruned forward gives the scores that select MtM_t for the next frame:

s^t,ipruned=Si(Xt,ℓ,pt;Mt−1),(4)\hat{s}_{t,i}^{\mathrm{pruned}} = S_i(X_t,\ell,p_t;M_{t-1}), \tag*{(4)}

where SiS_i denotes Eq. (3), with probe randomness suppressed. Every frame arrives with all visual tokens, but tokens outside Mt−1M_{t-1} leave the scoring graph after the cut layer: they keep indirect gradient paths through earlier-layer attention and lose their direct paths through later layers. Scores read on the pruned graph therefore favor retained tokens, and because MtM_t then prunes frame t+1t+1, the bias carries over to the next control step. We call this scoring-graph lock-in. Path deletion need not reduce gradient norms in general, so §4.5 directly quantifies the bias and isolates it from the feedback loop. Attention histories are mask-dependent in the same way, since excluded positions receive no new attention after the cut.

Offline full-graph labels. The teacher retains all visual tokens and computes

st,iT=Si(Xt,ℓ,pt;1).(5)s_{t,i}^{T} = S_i(X_t,\ell,p_t;1). \tag*{(5)}

For a fixed observation and fixed randomness, these labels do not depend on a previous pruning mask. We compute and cache them offline on fine-tuning demonstration observations, keeping the policy frozen.

For flow-based π0.5\pi_{0.5}, the action expert starts from noise ε\varepsilon and updates an action chunk over KK Euler steps (τ:1→0\tau: 1 \to0), conditioned on the visual prefix. We fix ε\varepsilon and differentiate the final chunk:

aθ(X,ℓ,p)=Φε(X,ℓ,p),Ji=∂Φε∂xi.(6)a_\theta(X,\ell,p) = \Phi_\varepsilon(X,\ell,p), \qquad J_i = \frac{\partial\Phi_\varepsilon}{\partial x_i}. \tag*{(6)}

Backpropagation traverses the full integration, accounting for repeated use of the visual prefix and propagation through intermediate action states, so the label reflects the final action rather than an isolated velocity prediction. Offline labeling uses one noise sample per observation, and the student, which does not receive ε\varepsilon, learns an observation-conditioned approximation; rankings vary far less across noise samples than across probes (Appendix § 16). Appendix § 8.1 gives the autoregressive-action relaxation.

Lightweight Student

The POKE student predicts within-camera importance from the current observation. For each token, we combine a low-dimensional projection hih_i with camera and scene means hˉc(i)\bar{h}_{c(i)}, hˉ\bar{h}, language features hℓh_\ell, state pp, spatial and camera coordinates qiq_i, and magnitude vi=log⁡RMS⁡(xi)v_i = \log\operatorname{RMS}(x_i):

zi=[hi;hˉc(i);hˉ;hℓ;p;qi;vi],r^i=gϕ(zi).(7)z_i = [h_i;\bar{h}_{c(i)};\bar{h};h_\ell;p;q_i;v_i], \qquad\hat{r}_i = g_\phi(z_i). \tag*{(7)}

The scoring head is a two-layer MLP with hidden width 64. Including the input projections, the student has approximately 271K parameters for OpenVLA-OFT and 140K for π0.5\pi_{0.5}. Its inputs are available from projector outputs and input embeddings, before the policy’s Transformer layers.

Teacher scores are log-transformed and standardized within each camera to form targets ycy_c. With Qc=softmax⁡(yc)Q_c = \operatorname{softmax}(y_c) and Pc=softmax⁡(r^c)P_c = \operatorname{softmax}(\hat{r}_c), we minimize

L=Ec[DKL(Qc∥Pc)]+λregLreg+λrankLrank.(8)\mathcal{L} = \mathbb{E}_c[D_{\mathrm{KL}}(Q_c \mathbin{\|} P_c)] + \lambda_{\mathrm{reg}}\mathcal{L}_{\mathrm{reg}} + \lambda_{\mathrm{rank}}\mathcal{L}_{\mathrm{rank}}. \tag*{(8)}

The KL, regression, and pairwise ranking terms supervise the importance distribution, standardized scores, and relative ordering, respectively; the latter two follow SDTP [22]. Loss and training details are provided in Appendix § 9. At deployment, the student produces all scores in one forward pass, without teacher evaluation, policy backpropagation, or previous-mask inputs, which removes the mask-conditioning path of Eq. (4).

Score-Guided Selection

We allocate b=B/Cb = B/C tokens to each camera. Top-kk retains the bb highest student scores. To reduce redundancy, **MMDP (max–min diversity pruning) applies the max–min selection of DivPrune [1], which operates there on all visual tokens, only to a candidate set Cc\mathcal{C}_c of the top min⁡(2b,∣Ic∣)\min(2b, |\mathcal{I}_c|) scores. With cosine distance d(i,j)d(i,j) between projector embeddings xix_i and xjx_j, the first token is the candidate farthest from its nearest neighbor, and we then repeatedly add

i∗=arg⁡max⁡i∈Cc∖Scmin⁡j∈Scd(i,j)(9)i^* = \arg\max_{i \in\mathcal{C}_c \setminus\mathcal{S}_c} \min_{j \in\mathcal{S}_c} d(i,j) \tag*{(9)}

until ∣Sc∣=b|\mathcal{S}_c| = b. Scores thus set relevance and embedding distance sets redundancy [18]. Selection costs O(b2d)O(b^2d) per camera and is included in all reported latencies; unselected tokens are removed at the cut layer (§ 4.1).

Experiments

Experimental Setup

We evaluate OpenVLA-OFT [8] (parallel regression, suite-specific checkpoints) and π0.5\pi_{0.5} [5] (flow matching, one checkpoint for all suites) on the four LIBERO suites, with 500 episodes per suite and initial states matched across methods. Both policies receive 512 visual tokens from two cameras. We retain B∈{128,64,32}B \in\{128,64,32\}, split equally between cameras, and delete the rest at the input to layer 4 (OFT) or 3 (π0.5\pi_{0.5}); the first three control steps of each

episode are unpruned for all methods, following VLA-Pruner’s history warmup, although POKE needs none. Speedups are paired median latencies per pruned control step against Vanilla, including scoring and selection; π0.5\pi_{0.5} timings use CUDA graphs except for online probes, which are timed against eager Vanilla (Appendix § 8.5). Autoregressive OpenVLA on LIBERO and a π0\pi_0 re-implementation on SimplerEnv [10] are reported in Table 2 (Appendices § 11 and 11). Robustness to observation shift is evaluated on LIBERO-Plus (§ 4.3).

MethodSpatialObjectGoalLongAvg.(↑)
OpenVLA
Vanilla84.684.074.051.473.50
FastV61.274.258.836.457.65
VLA-Cache67.471.060.234.258.20
VLA-Pruner75.676.261.442.063.80
POKE (top-kk)72.881.270.044.267.05
POKE (MMDP)78.882.272.453.671.75
MethodCarrotEggplantSpoonStackAvg.(↑)
π0\pi_0
Vanilla57.184.682.149.268.23
FastV50.87.561.239.639.79
VLA-Pruner52.920.467.935.844.27
POKE (top-kk)52.120.061.743.344.27
POKE (MMDP)52.130.872.547.150.63

Table 2. Additional policies and benchmark at B=32B = 32 of 256 visual tokens (success, %). Top: autoregressive OpenVLA on LIBERO. Bottom: open-pi-zero π0\pi_0 (Bridge checkpoint) on SimplerEnv WidowX visual matching, 240 episodes per task; its student is trained on real BridgeData V2 frames. Details and full results appear in Appendices 10 and 11.

All token-deletion methods share the budgets and cut. The cut and attention layers follow the released VLA-Pruner code [15] on OFT and its π0\pi_0 setting on π0.5\pi_{0.5}; B=128B = 128 and 64 match its 25% and 12.5% retention, and B=32B = 32 extends it. We compare against Random, attention-based FastV [3], diversity-based DivPrune [1], VLA-Pruner (VP) with temporal action-to-visual attention, and, on OFT, the key–value reuse of VLA-Cache [26]. SpecPrune-VLA† [24] keeps the static stage of SpecPrune-VLA, which joins the previous control step’s attention-selected tokens with early-layer attention and frame-difference tokens, at a fixed budget (Appendix § 8.4). MAP† adapts the attention-distillation pruner MAP [21], keeping its teacher, predictor, and selection rule but training with our objective (Appendix 7.3). Training it with MAP’s original objective instead changes success by at most 2.0 pp. POKE students are trained once per policy–suite pair on full-graph P=1P = 1 labels from fine-tuning demonstrations (Appendix 8). As diagnostics for lock-in, online probes score the mask for the next control step on the pruned graph or with an extra full-graph forward pass (§ 3.3). Implementation details appear in Appendix 7.2.

Main Results: Success and Runtime

At B=32B = 32, POKE (MMDP) achieves 95.15% average success at 1.85× speedup on OpenVLA-OFT and 90.80% at 1.26× on π0.5\pi_{0.5} (Table 1), exceeding VP by 5.90 and 15.75 pp at comparable speeds. The learned selector adds no measurable cost: POKE (top-kk) is within 0.2 ms of FastV on π0.5\pi_{0.5} and no slower on OpenVLA-OFT, and MMDP adds 1.6–5.1 ms depending on the budget (Appendix 13). The smaller speedup on π0.5\pi_{0.5} comes from the policy’s latency profile: pruning cuts VLM-backbone FLOPs by 4.4× and prefix latency by 2.46×, but the prefix is only 35% of Vanilla latency (28.5 of 81.4 ms; Table 25), so no token pruner can exceed 1.54× here.

Method\multicolumn{6}{c}{B=128B = 128}\multicolumn{6}{c}{B=64B = 64}\multicolumn{6}{c}{B=32B = 32}
Spa.Obj.GoalLongAvg.(↑)×(↑)Spa.Obj.GoalLongAvg.(↑)×(↑)Spa.Obj.GoalLongAvg.(↑)×(↑)
OpenVLA-OFT
Vanilla98.697.297.693.496.701.0098.697.297.693.496.701.0098.697.297.693.496.701.00
Random95.694.695.089.893.751.4372.480.852.433.859.851.7843.815.68.810.019.551.89
FastV97.497.295.691.895.501.4095.894.693.684.292.051.7391.680.892.643.277.051.88
DivPrune98.096.493.690.294.551.3795.893.288.872.487.551.6776.490.480.840.872.101.84
VLA-Cache97.098.097.490.895.801.2896.092.494.280.290.701.4490.488.890.470.485.001.47
VLA-Pruner97.498.098.491.096.201.3396.097.897.688.695.001.6694.093.491.078.689.251.82
SpecPrune-VLA†97.697.096.289.495.051.3598.097.095.886.494.301.6596.692.486.872.687.101.80
MAP†98.697.296.492.096.051.3698.496.898.089.095.551.6897.495.694.486.493.451.83
Probe top-kk (pruned)97.096.898.090.895.650.8495.490.894.685.491.550.9792.470.890.059.678.201.05
Probe MMDP (pruned)97.498.498.290.696.150.8196.498.098.090.295.650.9796.296.491.663.486.901.04
Probe top-kk (full)98.498.498.091.296.500.2896.898.498.090.095.800.3498.497.096.281.893.350.41
POKE (top-kk)99.298.098.091.096.551.4298.298.898.093.497.101.7498.296.897.482.893.801.89
POKE (MMDP)98.698.096.091.896.101.3599.098.096.092.896.451.6897.897.697.288.095.151.85
π0.5\pi_{0.5}
Vanilla98.697.897.292.496.501.0098.697.897.292.496.501.0098.697.897.292.496.501.00
Random73.482.280.268.476.051.2250.448.845.830.643.901.3011.620.433.64.817.601.31
FastV95.897.695.690.494.851.2286.093.482.875.284.351.2740.674.054.220.847.401.29
DivPrune98.097.296.889.095.251.1694.297.494.084.692.551.2283.284.480.063.877.851.28
VLA-Pruner99.098.297.490.696.301.1291.894.292.085.290.801.2068.892.077.462.075.051.24
SpecPrune-VLA†89.495.284.873.885.801.1565.871.273.243.863.501.2226.842.051.89.432.501.26
MAP†98.898.697.490.096.201.1496.294.895.481.892.051.2391.094.891.268.486.351.26
Probe top-kk (pruned)97.897.896.090.495.500.4893.892.091.681.089.600.4761.876.878.046.665.800.48
Probe MMDP (pruned)98.098.296.690.095.700.4797.095.696.687.494.150.4783.289.688.672.283.400.48
Probe top-kk (full)98.098.697.491.296.300.3297.296.895.887.494.300.3289.493.493.279.488.850.32
POKE (top-kk)97.898.096.689.695.501.2296.697.893.689.694.401.2784.686.277.068.479.051.29
POKE (MMDP)98.898.497.690.496.301.1496.898.097.689.095.351.2297.096.293.676.490.801.26

Table 1. LIBERO success rates (%) and speedups over Vanilla (×) with BB retained visual tokens. †: our adaptation. Gray: online diagnostic probes (§4.5), excluded from bold along with Vanilla; on π0.5\pi_{0.5} their backward passes run in eager mode, so their speedups are against eager Vanilla (Table 26). POKE uses its B=32B = 32 checkpoints at all budgets; timing details in Appendix § 14.

Execution PathImagePrefixDenoiseFull callPrefix GFLOPs
Eager, Full27.429.7232.0289.02,101
Eager, B=32B = 3227.418.0232.4278.1482
CUDA graph, Full9.728.542.981.42,101
CUDA graph, B=32B = 329.711.641.562.3482

Table 25. Per-stage π0.5\pi_{0.5} latency (ms, B=32B = 32; median of 20 repetitions per entry). Image encoding, the VLM prefix, the 10-step denoising loop, and the full sampling call are each timed with CUDA events and device synchronization around the call, on a synthetic observation with a fixed keep set (no token selection). All entries come from one session; in every row the stage sum matches the full call within 0.5 ms. The full call excludes input/output transforms, token selection, and the device-to-host copy, which the end-to-end timings in Table 27 include (83.2 vs. 81.4 ms for Vanilla under CUDA graphs in the same session). Prefix GFLOPs count the VLM backbone only.

Against MAP†. The strongest baseline at this budget is MAP†, which also combines distilled importance with diversity selection; POKE (MMDP) is higher on both policies (+1.7 and +4.45 pp). Without diversity, the comparison depends on label density. On OFT Long, POKE (top-kk) reaches 82.8% against 74.8% for MAP with its original objective and top-kk selection (Appendix 7.3). On π0.5\pi_{0.5}, POKE (top-kk) with P=1P = 1 trails MAP†, whose rule includes diversity, while P=8P = 8 lifts top-kk above it on Long and Goal (75.4 vs. 68.4% and 92.8 vs. 91.2%; Table 6).

TeacherLong GPU-hLong Top-kkLong MMDPGoal GPU-hGoal Top-kkGoal MMDP
P=1P = 10.768.476.40.377.093.6
P=3P = 32.070.280.00.986.894.0
P=5P = 53.372.480.41.185.293.4
P=8P = 85.475.483.02.392.894.2
Online probe (full, top-kk)–79.4––93.2–

Table 6. Teacher probe count on π0.5\pi_{0.5} Long and Goal (B=32B = 32; success %, 500 episodes). One checkpoint per PP; P=1P = 1 is from Table 1. Label costs are GPU-hours; Long P=3,5P = 3, 5 are scaled from measured P=8P = 8. Inference cost is unchanged.

Budgets and other policies. At B=128B = 128 the non-random methods are within 2 pp on OFT, where even Random retains 93.75%. On π0.5\pi_{0.5}, B=64B = 64 is the more practical operating point: POKE (MMDP) stays within 1.15 pp of Vanilla at 1.22×. POKE also outperforms VP on autoregressive OpenVLA and SimplerEnv (Table 2). On SimplerEnv, the π0\pi_0 student is trained on real BridgeData V2 frames and evaluated in simulation; despite this real-to-sim gap, POKE (MMDP) retains 74% of Vanilla success versus 65% for VP.

Robustness to Observation Shift

POKE students are trained only on unperturbed demonstrations, so we test whether their rankings transfer to LIBERO-Plus [4], which perturbs each LIBERO task along seven dimensions: camera viewpoint, robot initial state, language, lighting, background, sensor noise, and object layout. All methods and both policies run the same stratified subset with the settings of § 4.1. Because Vanilla itself degrades, Table 3 reports retention, success relative to Vanilla, for each perturbation category.

MethodCameraRobotLang.LightBackgr.NoiseLayoutTotal
Vanilla (%)66.474.880.791.989.674.483.774.4
OpenVLA-OFT
FastV0.560.510.740.630.730.580.660.64
DivPrune0.460.350.540.470.560.380.540.48
VLA-Pruner0.660.690.890.860.830.780.810.79
MAP†0.720.870.850.890.950.690.850.82
POKE (top-kk)0.820.890.970.980.960.930.920.92
POKE (MMDP)0.840.770.950.970.950.870.940.90
π0.5\pi_{0.5}
Vanilla (%)71.474.884.394.093.787.186.183.6
FastV0.330.280.520.500.520.340.520.43
DivPrune0.380.330.560.660.660.380.690.52
VLA-Pruner0.300.330.540.730.630.330.610.49
MAP†0.350.350.570.550.580.360.550.47
POKE (top-kk)0.520.480.670.780.700.480.770.63
POKE (MMDP)0.720.720.900.970.910.640.890.82

Table 3. LIBERO-Plus retention by perturbation category at B=32B = 32 on a stratified 25% subset (2,508 perturbed tasks, one episode each), pooled over the four suites. Vanilla: success (%); other rows: success relative to Vanilla. Policies and students are unchanged (trained on unperturbed demonstrations). Per-suite success appears in Appendix 12.

A POKE variant has the highest retention in every category on both policies, although on OpenVLA-OFT the margins over MAP† are small for robot initial state (0.89 vs. 0.87) and background (0.96 vs. 0.95). Camera shifts move

objects to different patch positions, so they test whether the student relies on a positional prior: POKE retains 82–84% of Vanilla success on OpenVLA-OFT against 66% for VP and 72% for MAP†, and POKE (MMDP) retains 72% on π0.5\pi_{0.5} against at most 38% for the baselines. Over all categories, POKE (MMDP) retains 82% on π0.5\pi_{0.5} against 43–52%, and its 27.4 pp gap to VP exceeds the 15.75 pp gap on unperturbed LIBERO (Table 22). MAP†, which exceeds VP by 11.3 pp on LIBERO with π0.5\pi_{0.5}, falls below it here (39.3 vs. 41.0%).

MethodSpatialObjectGoalLongAvg.(↑)Ret.(↑)
Vanilla82.783.864.067.874.61.00
FastV77.033.244.436.047.70.64
DivPrune53.327.842.619.435.80.48
VLA-Pruner71.763.350.650.559.00.79
MAP†71.863.259.350.861.30.82
POKE (top-kk)81.375.759.658.668.80.92
POKE (MMDP)79.773.359.157.367.40.90
π0.5\pi_{0.5} Vanilla91.087.878.277.983.71.00
π0.5\pi_{0.5} FastV45.742.538.017.135.80.43
π0.5\pi_{0.5} DivPrune54.751.344.324.143.60.52
π0.5\pi_{0.5} VLA-Pruner44.551.143.524.941.00.49
π0.5\pi_{0.5} MAP†42.549.843.421.639.30.47
π0.5\pi_{0.5} POKE (top-kk)68.360.850.531.352.70.63
π0.5\pi_{0.5} POKE (MMDP)83.372.564.053.768.40.82

Table 22. LIBERO-Plus success (%) by suite at B=32B = 32. Avg.: unweighted mean over suites; Ret.: Avg. relative to Vanilla. The Total column of Table 3 pools tasks across suites and can differ slightly.

Effect of Action-Sensitivity Supervision

Changing only the teacher. With the student, objective, and selection rule fixed, replacing action-sensitivity labels with full-graph action-to-visual attention, the signal behind VP’s temporal score, lowers success in all eight policy–suite–rule combinations (Table 4). Under top-kk, which compares the two score pools without any diversity correction, the drop is 3.2–6.0 pp; under MMDP it remains 1.6–4.6 pp. With denser P=8P = 8 labels, the top-kk gap on π0.5\pi_{0.5} Long widens to 12.6 pp (75.4 vs. 62.8; Table 6).

SuiteTeacherOpenVLA-OFTπ0.5\pi_{0.5}
Top-kkMMDPTop-kkMMDP
ObjectAttention93.496.082.8±1.6294.4±2.02
Action sens.96.897.686.0±1.4496.4±2.84
LongAttention80.40±2.3883.20±1.2862.8±0.4871.8±2.64
Action sens.83.93±0.8187.53±1.8668.8±0.9876.4±1.12
MethodCurrent-frame signalSuccess rate prunedSuccess rate fullOverlap prunedOverlap full
VP: action attention only–3.469.40.9950.664
VP: combinedSemantic attention78.682.40.5150.364
SpecPrune†: history only–4.884.41.0000.431
SpecPrune†: combinedLocal attention + frame difference72.680.60.5450.343
Action-sensitivity probe (top-kk)–59.681.80.850.41

Table 4. Teacher supervision at B=32B = 32 (success, %). Only the teacher labels differ (Appendix 8). ±: mean±SD over three training seeds, separate from the Table 1 checkpoint; other entries use the Table 1 checkpoint.

Action fidelity at shared observations. Applying each method’s masks to the same Vanilla-rollout observations (OFT Long, 625 frames), both POKE variants yield lower action deviations than VP and FastV on all ten tasks (p=0.002p = 0.002; Table 36); Figure 3 shows a closed-loop example.

Qualitative rollouts on OpenVLA-OFT Long task 6

Figure 3. Qualitative rollouts on OpenVLA-OFT Long task 6 (B=32B = 32) from the same initial state, showing frames and the applied third-person mask. VLA-Pruner times out at step 520 (red); POKE (top-kk) succeeds at step 220 (green). Figure 4 adds more results.

MethodDev./random ↓Below random ↑Tasks < VP
FastV0.3710.9142/10
VLA-Pruner0.3320.938–
POKE (top-kk)0.2520.98110/10
POKE (MMDP)0.2440.97310/10

Table 36. Mask fidelity at shared observations (OFT Long, B=32B = 32, 625 frames from Vanilla rollouts). Deviation is normalized by a random mask (lower is better); below random: fraction of frames with a smaller deviation than the random mask. VP’s temporal signal is updated along the shared trajectory. This measures mask quality on common reference states, not on each method’s own deployment states.

Is diversity alone sufficient? No. DivPrune applies the same max–min rule to all tokens without scores. It is the weakest non-random baseline on OFT and trails POKE (MMDP) by 12.95 pp on π0.5\pi_{0.5} (Table 1). Scores alone are not always sufficient either: MMDP improves POKE over top-kk by 11.75 pp on π0.5\pi_{0.5}, 4.70 on OpenVLA, and 1.35 on OFT. Top-kk can waste budget on redundant high-sensitivity patches, while unfiltered diversity selects background.

Why Distill a Full-Graph Teacher?

Full-graph. At B=32B = 32, switching the online top-kk probe from pruned to full-graph scoring raises OFT success from 90.0% to 96.2% on Goal and from 59.6% to 81.8% on Long. On π0.5\pi_{0.5}, average success rises from 65.80% to 88.85% (Table 1). Full-graph scoring is costly, however, running at 0.41× and 0.32× the unpruned speed, respectively. POKE amortizes this computation offline through distillation.

Attention histories. Lock-in also appears in attention histories (Table 5). With history as the only signal, pruned-graph history freezes the mask (overlap ≥0.995\ge0.995) and success collapses to 3.4% (VP action attention) and 4.8% (SpecPrune-VLA†); full-graph history restores 69.4% and 84.4%. On π0.5\pi_{0.5} Long, full history raises the complete SpecPrune-VLA† from 9.4% to 50.4% (Appendix § 10.5). The complete VP changes less (78.6% to 82.4%) because its current-frame semantic route re-admits tokens: under pruned history, 69% of re-entering tokens come from that route alone.

Is the feedback loop necessary? No; the pruned graph biases scores even without feedback. We replay probe scoring on OFT Long demonstration frames (B=32B = 32, 3,157 frames), using the same probe on full and pruned graphs. When the conditioning mask is drawn randomly at each frame, a scorer that ignored the mask would re-select 6% of it by chance; pruned-graph scoring re-selects 59%. On these frames, tokens outside the mask hold 94% of the full-graph score mass but only 28% of the pruned-graph mass. In closed loop, where the mask is the probe’s own previous selection, the shares are 77% and 19% (Figure 1b).

Student Scope and Approximations

Probe count and label cost. With P=1P = 1, caching teacher labels costs 0.3–0.7 GPU-hours per suite (Appendix § 9), whereas exact labels would need AA backward passes per frame, about 47 GPU-hours for π0.5\pi_{0.5} Long alone. We train students on multi-probe labels for π0.5\pi_{0.5} Long and Goal (Table 6). On Long, P=8P = 8 improves success by 7.0 pp (top-kk) and 6.6 pp (MMDP) and narrows the top-kk gap to the online full-graph probe from 11.0 to 4.0 pp. On Goal, it raises top-kk by 15.8 pp, to within 0.4 pp of the online probe. The MMDP margin over top-kk shrinks from 16.6 to 1.4 pp on Goal but stays near 8 pp on Long, so denser labels replace the diversity gain only on some suites. The cost (5.4 and 2.3 GPU-hours) is paid once offline.

Student scope. The student is not a fixed positional prior: it keeps most of Vanilla success under camera shifts (§ 4.3), and zero-shot cross-suite students still outperform VP (Spatial→Long 84.4 vs. 78.6%, Long→Spatial 99.4 vs. 94.0% with MMDP; Table 18). For π0.5\pi_{0.5}, a single student trained jointly on all four suites matches per-suite students (90.55 vs. 90.80% at B=32B = 32 with MMDP; Appendix 9.7).

EvaluateTrain\multicolumn{2}{c}{B=32B = 32}\multicolumn{2}{c}{B=64B = 64}
Top-kkMMDPTop-kkMMDP
LongSpatial80.2 (−2.6)84.4 (−3.6)89.0 (−4.4)91.0 (−1.8)
VPVP78.688.6
SpatialLong97.4 (−0.8)99.4 (+1.6)97.8 (−0.4)97.2 (−1.8)
VPVP94.096.0

Table 18. Cross-suite transfer on OpenVLA-OFT. A POKE student trained using one suite’s teacher labels is evaluated on another without retraining. B=64B = 64 changes both suite and budget. Parentheses give differences (pp) from target-suite training; italic VP values are target-suite baselines.

Surrogate fidelity. Replacement and deletion rank mask damage similarly on OFT Spatial (median within-frame Spearman 0.964). On OFT Goal demonstration frames, POKE (top-kk) masks, with no backward pass, move the action about one quarter as far as random masks of the same budget (median ratio 0.254; Table 34). Exact-sensitivity masks, which need 56 backward passes per frame, reach 0.250, and the single-probe teacher masks reach 0.285. The student’s modest top-32 recall against its teacher (0.60; Appendix § 9) therefore does not turn into action error. The ranking is also meaningful in reverse: keeping the 32 tokens with the lowest exact sensitivity moves the action 1.84 times as far as a random mask (Table 34).

MaskDev./randomBelow random
Exact top-kk (56 backward passes)0.2500.988
POKE (top-kk) (0 backward passes)0.2540.983
Probe(P=3P=3) top-kk0.2670.988
Probe(P=1P=1) top-kk (teacher)0.2850.986
POKE (MMDP)0.3430.977
Exact MMDP0.3640.977
Probe(P=1P=1) MMDP0.4020.973
DivPrune0.7580.734
Random1.0000.500
Probe(P=1P=1) bottom-kk1.5990.044
Exact bottom-kk1.8440.014

Table 34.

Discussion

Lock-in is a property of the scoring graph. Our interventions (§4.5) suggest that methods reusing temporal signals need a mask-independent entry path or full-graph scoring to avoid lock-in. Reusing a mask over several control steps causes a separate problem: the policy cannot attend to newly relevant regions until the mask is refreshed, and success decreases at longer reuse intervals in our tests (Table 24). Lock-in explains why online pruned-graph probing fails and why history-based pruners degrade at small budgets. It does not explain POKE’s margin over per-frame selectors FastV and MAP†, which have no mask feedback; that margin comes from the supervision signal (§ 4.4).

SuiteR=1R = 1R=2R = 2R=4R = 4R=8R = 8
Spatial78.077.077.071.4
Object80.881.880.677.0
Goal72.272.670.867.4

Table 24. Success rates (%) when reusing masks for RR control steps: OpenVLA, B=32B = 32, online full-graph probe MMDP. Scoring once every RR control steps reduces its amortized frequency to 1/R1/R, while masks become stale.

Why can the student match or exceed online probes? At B=32B = 32, the student matches or exceeds the online full-graph probe in average success under top-kk selection on OpenVLA-OFT and OpenVLA. We hypothesize two factors: first, the student scores the current observation directly rather than applying a mask chosen at the previous control step; second, training across frames averages the part of the single-probe error that is not predictable from the observation (Appendix § 16).

On π0.5\pi_{0.5}, the student instead trails the online probe under top-kk (79.05% vs. 88.85% on average), and MMDP closes most of this gap (90.80%). Single-probe variance is nearly identical for OpenVLA-OFT and π0.5\pi_{0.5} (1.90 vs. 1.92; Appendix § 16) and the online probe also uses P=1P = 1, so the shortfall is a distillation gap rather than a limit of the signal: P=8P = 8 labels largely close it (§ 4.6). Why π0.5\pi_{0.5} students are more sensitive to label noise is untested; one candidate is that the action expert re-reads the same pruned prefix at all ten denoising steps.

How many probes for the teacher? Our main results use P=1P = 1, the cheapest setting and the probe count of the online diagnostic; larger PP is an offline trade-off (§ 4.6).

Limitations. First, unlike training-free pruners, POKE requires offline teacher labeling and student training on demonstrations. Second, action sensitivity is a first-order, token-wise surrogate that does not explicitly model joint multi-token interactions, though candidate-restricted diversity selection helps mitigate redundancy in practice. Third, on π0.5\pi_{0.5} the ten-step denoising loop is unaffected by token pruning and caps the end-to-end gain (§ 4.2). Finally, all closed-loop evaluations are in simulation; as partial evidence beyond it, the SimplerEnv student is trained on real BridgeData V2 frames, and POKE has the highest retention under the LIBERO-Plus camera, lighting, and noise shifts.

Conclusion

Action sensitivity measures which visual tokens a VLA policy needs to keep its actions, but computing it online on the pruned graph causes scoring-graph lock-in, a bias also observed in the pruned-graph attention histories of VLA-Pruner and SpecPrune-VLA. POKE computes it offline on the full graph and distills it into a small student that scores the current observation alone. Keeping 32 of 512 tokens, it reaches 95.2% on OpenVLA-OFT, 90.8% on π0.5\pi_{0.5} at 1.85×1.85 \times and 1.26×1.26 \times speedups, ahead of attention-, diversity-, and distillation-based selectors and degrading less than VLA-Pruner on LIBERO-Plus; one student suffices for the jointly trained π0.5\pi_{0.5}. Real-robot evaluation and combining token selection with action-expert acceleration remain open.

LLM Usage Statement

We used Claude (Anthropic) and ChatGPT (OpenAI) for writing and coding assistance. For writing, the tools were used throughout the main paper and the supplementary material to proofread English grammar and style, and to condense text to fit the page limit. They also gave feedback on clarity and organization, such as which paragraphs to shorten or move to the appendix, and proposed revised wording for paragraphs and figure captions, which the authors reviewed and edited. For code, the tools were used to write plotting code for the figures, to refactor our research code, and to port the public VLA-Cache and VLA-Pruner implementations from the VLA-Pruner repository into our codebase as baselines. The authors designed POKE, ran and verified all experiments, and checked and cited all references themselves; the tools did not generate experimental results or data. The authors take full responsibility for the content of this paper.

Implementation and Reproducibility

Policies and Pruning Paths

All policies use public checkpoints with frozen parameters. OpenVLA-OFT uses a 7B backbone with suite-specific checkpoints, 256 visual tokens from each of the third-person and wrist cameras, an additional token for the 8-dimensional proprioceptive state, and parallel prediction of eight 7-dimensional actions (A=56A = 56). Autoregressive OpenVLA uses 256 tokens from one third-person camera and predicts seven action tokens (A=7A = 7). For OFT, sensitivities differentiate the action as the checkpoint returns it: the arm dimensions are unnormalized to execution units with the checkpoint’s q01–q99 statistics, and the gripper dimension, which the checkpoint leaves unnormalized, stays in its [−1,1][-1,1] range.

π0.5\pi_{0.5}. We convert openpi’s pi05_libero checkpoint, jointly fine-tuned on all four LIBERO suites, to PyTorch bf16 (openpi commit 215abfb2). The backbone is PaliGemma (Gemma 2B, 18 layers); layer jj of the 18-layer Gemma 300M action expert attends to the keys and values of prefix layer jj. LIBERO uses agentview and wrist images from two of the three camera slots. We omit the masked, empty right-wrist slot and verify matching outputs in a self-test. SigLIP at 224-pixel resolution produces a 16×1616 \times16 token grid per camera, for 512 visual tokens in total.

In this configuration, tokens are deleted at the input to layer 3: layers 0–2 process all tokens, and prefix/expert attention from layer 3 onward uses the retained visual tokens with their original position indices. Actions are generated over a horizon of ten by ten Euler steps, with replanning every five environment steps. Sensitivities use the seven physical action dimensions, excluding 25 padding dimensions (A=70A = 70). We multiply normalized actions by their dimension-wise scales (quantile half-ranges), so the derivatives are expressed in execution units. The constant unormalization offsets do not affect the Jacobian.

Episode limits are 220, 280, 300, and 520 environment steps for Spatial, Object, Goal, and Long. Reproduced Vanilla success rates are 98.6, 97.8, 97.2, and 92.4% (average 96.50%), compared with 98.8, 98.2, 98.0, and 92.4% (96.85%) reported for this checkpoint in the openpi repository. Each suite differs by at most 0.8 pp, within the sampling error of 500 episodes (about 0.8 pp at 97% success). Rollout evaluation uses eager attention without compilation for deterministic execution; the separate CUDA-graph timing path is described in Appendix 13.1.

Discrete-action policies. For autoregressive OpenVLA, the executed argmax action is nondifferentiable. We fix the greedily decoded action-token sequence and use a teacher-forced forward pass to define a continuous surrogate,

a~k=∑msoftmax⁡(zk)mβm,(10)\tilde{a}_k = \sum_m \operatorname{softmax}(z_k)_m \beta_m, \tag*{(10)}

where βm\beta_m is the action-bin center at position kk. Sensitivity is computed for this surrogate, rather than for the executed argmax mapping. Bin centres βm\beta_m lie in the normalized range [−1,1][-1,1]; as in predict_action, arm dimensions are then unnormalized with the checkpoint’s q01–q99 statistics and the gripper stays in [−1,1][-1,1], the same convention as for OFT.

Control steps and pruning location. Throughout, a control step (“call”) is one policy inference on a new camera frame, which executes an action chunk of several environment steps. Layer indices are 0-based. OFT deletes visual tokens at the input to layer 4; π0.5\pi_{0.5} and OpenVLA use layer 3. Retained tokens keep their original position indices and thus their assigned positional phases. All nonvisual tokens, including BOS, proprioception, prompt, action slots, and stop tokens, are retained. OFT enforces a quota of B/2B/2 per camera. The first three control steps of every episode are unpruned, so shorter episodes have a smaller fraction of pruned control steps (e.g., 91% on Long and 77% on Spatial in the recorded runs).

In OpenVLA, the mask chosen during prefill is held fixed throughout the seven action-token decoding steps. At each decoding step, the implementation checks that post-cut key–value caches have the expected shortened lengths and aborts otherwise. All reported rollouts pass this check. OFT runs use the bidirectional-attention fork required by its checkpoints. A first-frame check verifies nonzero upper-triangular attention at the cut layer, guarding against accidental use of the causal fork.

Baseline Implementations

Random samples BB tokens at each control step subject to camera quotas. Our VLA-Pruner implementation combines the semantic signal at the cut with temporally smoothed action-to-visual attention from a deeper layer, and selects the mask within the current forward pass. Per camera, it takes the union of the top-bb tokens under each signal and applies the same greedy max–min routine as MMDP to this union. We port the selection logic while retaining OFT’s bidirectional policy implementation, since replacing it with the repository’s causal Transformers fork would change the policy. FastV ranks visual tokens by attention averaged across heads and queries at the same cut, retaining the top-B/CB/C per camera. It omits the temporal signal and diversity filtering, providing a comparison at a matched pruning location.

DivPrune [1] is run with our own implementation of its greedy max–min algorithm, the same routine that MMDP uses (Eq. (9)), with the candidate set enlarged to all 256 tokens of a camera and no importance scores. As in DivPrune, distances are cosine distances between projector outputs, and the first token is the one farthest from its nearest neighbor. Two settings differ from the original. Selection is applied per camera with a quota of B/2B/2, whereas DivPrune selects from the visual tokens of a single image. Tokens are deleted at the same cut as the other baselines (input to layer 4 for OFT and 3 for π0.5\pi_{0.5}), whereas DivPrune by default prunes before the language model; it also allows pruning at a decoder layer, and the later cut retains more information for the first layers (Appendix § 16).

For π0.5\pi_{0.5}, FastV uses attention from prefix layer k−1k-1. VP’s SvlS_{\mathrm{vl}} uses that layer’s attention, while SactS_{\mathrm{act}} aggregates action-to-visual attention from the latter half of expert layers and flow steps, with history window 3 and γ=0.8\gamma= 0.8. These settings follow the original π0\pi_0 configuration without modification.

VLA-Cache reuses key–value states across frames and is evaluated on OFT. We port its implementation from the public VLA-Pruner code base, which includes it as a baseline, without changing its reuse criterion. We did not implement it for π0.5\pi_{0.5}; caching may still apply to flow policies.

Online probes compute P=1P = 1 action sensitivities to select the mask for the next control step, whereas the POKE student selects from the current observation. The pruned condition backpropagates through the current masked forward; the full condition adds a separate all-token forward for scoring.

MAP Adaptation

MAP [21] was proposed for MLLMs. We reimplement MAP’s Eqs. (1)–(4) for OpenVLA-OFT from the paper, without an official release, and make the unspecified widths explicit below.

Teacher (QCTS). MAP labels use exactly the frames, episode split, and full-graph setting of the action-sensitivity cache (Appendix § 9), so both teachers are computed on identical observations. For each frame, the unpruned policy runs with SDPA attention, and we record attention at all 32 language model layers. Query rows are the question tokens of the OFT prompt, “What action should the robot take to instruction?”, excluding the In:/Out: markers, BOS, and action slots. Following MAP’s Eq. (1), attention to each visual token is averaged over heads and question tokens and then ℓ1\ell_1-normalized jointly over all 512 visual tokens of both cameras, giving plp^l. The reference distribution preflp^l_{\mathrm{ref}} uses the same images and proprioceptive state with the question replaced by “What is shown in this image?”. The teacher is pl∗p^{l*} with l∗=arg⁡max⁡lDJS(pl,prefl)l^* = \arg\max_l D_{\mathrm{JS}}(p^l,p^l_{\mathrm{ref}}) (MAP’s Eq. (2)), selected once per frame and shared by both cameras. Labels are computed from single frames, without temporal aggregation.

Predictor. Following MAP’s Eq. (3), question-token input embeddings and projector outputs pass through modality-specific two-layer MLPs (4096 →\rightarrow 256 →\rightarrow 256, GELU), followed by bias-free projections WQ,WK\mathbf{W}_Q,\mathbf{W}_K to 64 dimensions. RoPE with the backbone’s base frequency is applied at the original sequence positions of the question and visual tokens. The predicted distribution p^\hat p is the soft-max of the scaled query–key products over the 512 visual tokens, averaged over question tokens. MAP does not specify these widths; our choice gives 2,261,504 parameters, about eight times the POKE student.

Objective. MAP minimizes DKL(pl∗∥p^)D_{\mathrm{KL}}(p^{l*}\Vert\hat p) over all visual tokens. MAP†^\dagger instead uses our objective, so that the training loss matches POKE: the logarithms of p^\hat p and pl∗p^{l*} are standardized within each camera as in Eq. (14), and Eq. (15) is applied with the same weights, pair sampling, and margin. Optimization settings and the episode split also match POKE, and the checkpoint with the lowest validation KL is selected.

Using MAP’s original objective instead changes success at B=32B = 32 by at most 2.0 pp: 86.40 vs. 86.35% (LIBERO) and 38.3 vs. 39.3% (LIBERO-Plus) with π0.5\pi_{0.5}, and 84.4 vs. 86.4% (LIBERO-Long) and 61.4 vs. 61.3% (LIBERO-Plus) with OpenVLA-OFT. This is far below the gap to POKE, so the choice of objective does not explain the difference.

Selection and pruning. Selection is applied per camera with b=B/2b = B/2. The top-kk variant retains the bb highest predicted scores. The diversity variant follows MAP’s Eq. (4): it initializes the retained set with the top K0=⌈fb⌉K_0 = \lceil fb\rceil predicted tokens and then greedily adds arg⁡max⁡ip^i∥vi−BBTvi∥2\arg\max_i \hat p_i\lVert\mathbf{v}_i-\mathbf{B}\mathbf{B}^{\mathsf{T}}\mathbf{v}_i\rVert_2 over all remaining tokens of the camera, where B\mathbf{B} is an orthonormal basis of the retained projector outputs, maintained by Gram–Schmidt with reorthogonalization. We use f=0.5f = 0.5. Unlike MMDP, candidates are not restricted to the top-2b2b scores, and importance multiplies the residual rather than gating eligibility. MAP†^\dagger in Table 1 uses this rule, the stronger of the two: with MAP’s original objective on OpenVLA-OFT LIBERO-Long at B=32B = 32, it reaches 84.4% versus 74.8% for top-kk. MAP prunes before the first language-model layer; to match the other baselines, we instead delete tokens at the same cut (input to layer 4), with the same three unpruned warmup control steps. The predictor uses only projector outputs and prompt embeddings, so scoring still precedes the language model. MAP† latency is measured with the same protocol as the other methods.

Adaptation to π0.5\pi_{0.5}. The π0.5\pi_{0.5} variant follows the same procedure with three changes. The teacher compares all 18 PaliGemma prefix layers and selects, per frame, the layer that maximizes the JS divergence between attention under the task instruction and under the reference question. Query rows are the tokens of the task instruction itself, since the π0.5\pi_{0.5} prompt has no surrounding question template, and keys are the 512 visual tokens of the two real cameras. The predictor takes 2048-dimensional projector outputs and instruction embeddings, with the same hidden width (256) and query–key width (64) as for OFT, giving about 1.2M parameters. Training uses our objective, selection uses MAP’s Eq. (4) with f=0.5f = 0.5, and tokens are deleted at the input to layer 3, as for the other π0.5\pi_{0.5} baselines.

SpecPrune-VLA Adaptation

SpecPrune-VLA [24] combines action-level static pruning, layer-level dynamic pruning inside the language model, and an action-aware controller that switches between pruning profiles. The latter two change the number of retained tokens from call to call and from layer to layer. SpecPrune-VLA† keeps only the static stage and fixes its output to BB tokens, deleted at the same cut as the other baselines (input to layer 4 for OFT and 3 for π0.5\pi_{0.5}) after the same three unpruned warmup calls. We port the selection from the public OpenVLA-OFT code.

Token sets. For each camera, b=B/2b = B/2 is split as Kglobal:Klocal:Kdyn=30:24:20K_{\mathrm{global}} : K_{\mathrm{local}} : K_{\mathrm{dyn}} = 30 : 24 : 20, the base values of SpecPrune-VLA, by largest remainder (7:5:47 : 5 : 4 at B=32B = 32). The token score is the attention from instruction tokens to each visual token, averaged over heads and instruction tokens, as in the released code; the fixed prompt prefix “In: What action should the robot take to” is excluded. The paper writes the score as visual-to-instruction attention; both exist under OFT’s bidirectional attention, and we follow the code.

  • VglobalV_{\mathrm{global}}: the highest-scoring tokens at the 15th and the last layer (0-based 14 and 31) of the previous call. The two layers alternate until KglobalK_{\mathrm{global}} distinct tokens are chosen, which replaces the original union of per-layer top-K/2K/2 sets, whose size varies.

  • VlocalV_{\mathrm{local}}: the same score at layers 0 and 1 of the current call, before the cut, alternating in the same way.

  • VdynV_{\mathrm{dyn}}: the patches least similar to the frame six environment steps earlier, among those with cosine similarity below τ\tau (0.986 third-person, 0.98 wrist). Similarity is computed on the 14×1414 \times14 pixel patches aligned with the visual tokens, as in the released code. Each environment step is the code’s comparison interval when the controller is off.

The retained set is Vglobal∪Vlocal∪VdynV_{\mathrm{global}} \cup V_{\mathrm{local}} \cup V_{\mathrm{dyn}}. When the sets overlap, the remaining slots are filled in VglobalV_{\mathrm{global}} order; ties among tokens without a global score are broken by the current local score.

History graph and path conditions. As published, the previous call’s attention comes from its pruned forward, so tokens deleted at that call receive no global score. The full-history condition of Table 5 instead reads it from a separate unpruned forward on the same observation and discards that forward’s actions. Path conditions disable VlocalV_{\mathrm{local}} and VdynV_{\mathrm{dyn}}; their share of bb is then filled in VglobalV_{\mathrm{global}} order, so the budget is unchanged.

Adaptation to π0.5\pi_{0.5}. The π0.5\pi_{0.5} variant differs in three settings. VglobalV_{\mathrm{global}} uses prefix layers 8 and 17 of the 18 PaliGemma layers, the relative depth of the 15th and 32nd of 32. The query rows are all valid language tokens, since the π0.5\pi_{0.5} prompt has no fixed template and also carries the discretized state. The client sends the frame six environment steps earlier with each call; π0.5\pi_{0.5} replans every five environment steps and OFT every eight. VlocalV_{\mathrm{local}} uses prefix layers 0 and 1, before the cut at layer 3.

Latency. Reading attention at the local, cut, and global layers is part of the method and is included in the measured latency.

Determinism and Timing Protocol

Repeated Vanilla evaluations on the same machine and kernels produced identical episode outcomes on OFT Long and OpenVLA Goal/Object. For π0.5\pi_{0.5}, initial noise is seeded by (suite, task, episode, control step); all 30 rerun episodes matched. Online probes use the same noise as the executed action, including the additional full-graph scoring pass. These checks cover only the tested configuration and hardware. All runs and timings use a single NVIDIA RTX A6000 (48 GB) with driver 535.309.01 and CUDA 12.2.

On OFT Long task 5, all four online conditions encountered a simulator QACC instability warning at the same simulation time, t=0.5480t = 0.5480. The warning recurred deterministically from the same initial state; we did not check whether it affected the subsequent trajectories identically.

Timing starts from a prepared observation and ends when CPU actions are returned, excluding environment stepping and image preparation. It includes student scoring, candidate construction, and diversity selection. Methods are executed on the same observation in randomized order within each repetition, and speedups are paired ratios against Vanilla. We report results from five independent processes with 50 repetitions each, with no concurrent workloads on the machine. Each process discards GPU warmup iterations before timing; this is separate from the three unpruned control steps at the start of each episode, which are not timed. Reported latencies are medians over pruned calls, pooled across processes, so speedups are per pruned control step rather than amortized over episodes.

POKE Student Architecture and Training

Teacher Labels

Teacher labels use each policy’s fine-tuning demonstrations. OFT uses the LIBERO RLDS data, with preprocessing and state normalization matched to the evaluation pipeline and checkpoint statistics. Demonstrations are subsampled every eight environment steps. Training and validation episodes are disjoint, with splits stratified by instruction. Validation labels are used only for checkpoint selection.

For π0.5\pi_{0.5}, we split the openpi fine-tuning dataset (physical-intelligence/libero) by suite, use ten demonstrations per task, subsample every eight environment steps, and reserve 20% for validation. Each frame receives one initial-noise sample, P>1P>1 labels use v0,…,vP−1v_0,\ldots,v_{P-1}. Labels differentiate the full-graph action in Eq. (6), without averaging over multiple noise samples.

For OFT, the selected checkpoints occur at epochs 5–8, well before the end of training. Their validation Recall@32 values are modest (Table 7), yet action deviations under POKE-selected masks can approach those of teacher-selected masks (Appendix § 16). One possible explanation is that sensitivity energy is concentrated in a few tokens, but these measurements do not isolate the effect of missed tokens.

BackboneSuiteTrainValBest epochVal KLRecall@32Labels (GPU-h)
OFTSpatial1,23835280.14670.5560.3
OFTObject1,49837550.14580.5600.4
OFTGoal1,33735050.15090.5970.3
OFTLong2,76669170.14450.5680.7
π0.5\pi_{0.5}Spatial1,297312180.13410.5530.3
π0.5\pi_{0.5}Object1,525376170.12810.5850.4
π0.5\pi_{0.5}Goal1,266317200.13670.5400.3
π0.5\pi_{0.5}Long2,750678190.13270.5360.7
π0.5\pi_{0.5}All (joint)6,8381,683180.13230.554–
OpenVLASpatial1,23835060.15130.5480.3
OpenVLAObject1,498375120.14190.5620.4
OpenVLAGoal1,29935180.14320.5580.3
OpenVLALong2,765689100.14860.5570.7

Table 7. POKE student training records. Frame counts follow subsampling every eight environment steps. Recall measures overlap between the student’s and teacher’s top-BB sets on validation observations. Label-generation cost is GPU time spent building the teacher cache.

Attention-teacher control. Table 4 changes only the teacher labels. The attention teacher relabels exactly the frames of the action-sensitivity cache; visual features and episode splits are copied unchanged, and only the score is replaced. For each frame, the unpruned policy runs with SDPA attention, and we read layer-15 attention from the 56 action-slot query rows to all 512 visual tokens, averaged over heads and action queries. This is the action-to-visual signal underlying VP’s SactS_{\mathrm{act}} (Appendix § 10.5), computed on the full graph from a single frame, without temporal history. For π0.5\pi_{0.5}, the attention teacher relabels the frames of its action-sensitivity cache in the same way. It reuses the initial noise of each frame’s action-sensitivity label, integrates all ten flow steps on the unpruned graph, and reads attention from the action-horizon query rows of action-expert layer 8 to the 512 visual tokens in the prefix keys of the same layer. The scores are averaged over heads, horizon queries, and all ten flow steps. Capturing attention leaves the sampled actions unchanged. The scores pass through the same per-camera normalization (Eq. (14)), optimization (Eq. (15)), and the POKE student, objective (Eq. (15)), optimization, and selection rules are unchanged. The π0.5\pi_{0.5} Object/Long comparisons and OFT Long report means and SDs over three additional training seeds (2, 5, 10), drawn at random; OFT Object uses the seed-0 checkpoint of Table 1, which reports seed-0 results for all trained selectors.

Architecture

Input features. A visual token xi∈Rdx_i\in\mathbb{R}^{d} (d=4096d=4096, or 2048 for π0.5\pi_{0.5}) is projected to 32 dimensions as

hi=GELU⁡(WvLN⁡(xi)+bv),(11)h_i=\operatorname{GELU}(W_v\operatorname{LN}(x_i)+b_v), \tag*{(11)}

with non-affine LayerNorm. Camera and scene context are

hˉc=1∣Ic∣∑j∈Ichj,hˉ=1C∑chˉc.(12)\bar{h}_c=\frac{1}{|\mathcal{I}_c|}\sum_{j\in\mathcal{I}_c}h_j,\qquad\bar{h}=\frac{1}{C}\sum_c\bar{h}_c. \tag*{(12)}

Language features hℓh_\ell are formed by averaging valid prompt input embeddings and applying a separate LayerNorm–linear–GELU projection to 32 dimensions; no deep contextual language states are required.

With qiq_i containing two normalized patch coordinates and a camera identifier, and νi=log⁡RMS⁡(xi)\nu_i=\log\operatorname{RMS}(x_i), the scorer is

zi=[hi;hˉc(i);hˉ;hℓ;p;qi;νi],z_i=[h_i;\bar{h}_{c(i)};\bar{h};h_\ell;p;q_i;\nu_i],
r^i=W2GELU⁡(W1zi+b1)+b2.(13)\hat{r}_i=W_2\operatorname{GELU}(W_1z_i+b_1)+b_2. \tag*{(13)}

The MLP hidden width is 64. Including both input projections, the parameter counts are 271,297 for OFT (∣p∣=8|p|=8), 140,225 for π0.5\pi_{0.5} (d=2048d=2048, ∣p∣=8|p|=8), and 270,785 for OpenVLA (∣p∣=0|p|=0). Scores are unconstrained since selection uses within-camera rankings.

Objective and Optimization

Teacher normalization. For each observation, scores are normalized independently within each camera:

ℓ~i=log⁡max⁡(s^i,10−12),\tilde{\ell}_i=\log\max(\hat{s}_i,10^{-12}),
yi=ℓ~i−μc(i)max⁡(σc(i),0.1).(14)y_i=\frac{\tilde{\ell}_i-\mu_{c(i)}}{\max(\sigma_{c(i)},0.1)}. \tag*{(14)}

Here μc\mu_c and σc\sigma_c are the population mean and standard deviation across the camera’s log scores. This transformation reduces scale variation and preserves ordering above the floor; scores clipped at the floor can tie.

Objective. With T=1T=1, Qc=softmax⁡(yc/T)Q_c=\operatorname{softmax}(y_c/T), and Pc=softmax⁡(r^c/T)P_c=\operatorname{softmax}(\hat{r}_c/T), the loss is

L=T2Ec[DKL(Qc∥Pc)]+0.1LSmoothL1(r^,y)+0.2Lrank.(15)\mathcal{L}=T^2\mathbb{E}_c[D_{\mathrm{KL}}(Q_c\mathbin\Vert P_c)] +0.1\mathcal{L}_{\mathrm{SmoothL1}}(\hat{r},y)+0.2\mathcal{L}_{\mathrm{rank}}. \tag*{(15)}

The ranking term is

Lrank=E(i,j)[softplus⁡(−dij)],dij=sign⁡(yi−yj)(ri−rj).(16)\begin{aligned} \mathcal{L}_{\mathrm{rank}} &= \mathbb{E}_{(i,j)}[\operatorname{softplus}(-d_{ij})],\\ d_{ij} &= \operatorname{sign}(y_i-y_j)(r_i-r_j). \tag*{(16)} \end{aligned}

Each iteration samples 256 within-camera index pairs (i,j)(i,j), shared across the batch and applied separately in each camera; pairs with ∣yi−yj∣≤0.1|y_i-y_j| \le0.1 are excluded.

Optimization. We train for 20 epochs with AdamW (learning rate 3×10−43 \times10^{-4}, weight decay 10−410^{-4}), batch size 4, and gradient clipping at 1.0. The checkpoint with the lowest validation KL is selected; the policy remains frozen.

Probe Count for 0.5 Teacher Labels

Table 6 evaluates students trained with different teacher probe counts on π0.5\pi_{0.5} Long. A single run computes eight probes per frame (seeds 0–7) using the frames, episode split, and initial noise of the P=1P = 1 cache. The label for probe count PP uses the first PP probes:

s^i(P)=(1P∑k=1Ps^i,k2)1/2.(17)\hat{s}_i^{(P)} = \left(\frac{1}{P}\sum_{k=1}^{P}\hat{s}_{i,k}^{2}\right)^{1/2}. \tag*{(17)}

Its square averages PP unbiased estimates of si2s_i^2 (Appendix § 16). Probe 0 reproduces the cached P=1P = 1 labels, with zero maximum relative difference on the 50 checked frames.

The student architecture, objective, 20 training epochs, and checkpoint selection by validation KL follow Appendix § 9. Rollouts use the 500 initial states and initial-noise seeds of Table 1. The eight-probe run takes 5.62 s per frame on one RTX A6000 (5.4 GPU-hours for 3,428 frames), against 0.7 GPU-hours for the P=1P = 1 cache (Table 7).

For P=3P = 3 and P=5P = 5, validation KL is lowest at the final epoch, whereas for P=8P = 8 it is lowest at epoch 17 and remains within 0.0886–0.0897 thereafter. The P=3P = 3 and P=5P = 5 students may therefore benefit from longer training, which we did not test. Because Table 8 uses a different validation target for each probe count, our downstream comparison uses closed-loop success (Table 6).

TeacherBest epochVal KLRecall@32
P=1P = 1190.13270.536
P=3P = 3200.10320.583
P=5P = 5200.09630.595
P=8P = 8170.08860.611

Table 8. Training records of π0.5\pi_{0.5} Long students for teacher probe count PP. Val KL and Recall@32 are measured against each row’s own validation labels. Because the targets differ with PP, these values do not directly compare student quality across rows. The P=1P = 1 row is from Table 7.

Additional Success-Rate Results

Task-Level Success

On Long, full-graph top-k improves over pruned-graph top-k on nine of ten tasks. The largest gains occur on tasks 6 and 8 (+42 and +46 pp), where pruned scoring reaches only 30% and 28%. Task 7, already at 82%, gains 14 pp. The graph effect is therefore broadly distributed across tasks, unlike the concentrated diversity gains below.

Method0123456789Overall
Goal
Probe top-k (pruned)96989672968870100949090.0
Probe MMDP (pruned)989294809680821001009491.6
Probe top-k (full)1009496949892901009810096.2
DivPrune92929048981682100989280.8
Graph effect+4−40+22+2+4+200+4+10+6.2
Long
Probe top-k (pruned)5478808236583082286859.6
Probe MMDP (pruned)7894847042542496306263.4
Probe top-k (full)8092789666787296748681.8
DivPrune7092662861288024440.8
Graph effect+26+14−2+14+30+20+42+14+46+18+22.2

Table 9. Task-level success rates (%) for online methods and DivPrune on OpenVLA-OFT (B=32B = 32). The four methods are evaluated in one session per suite. Graph effect is full minus pruned scoring with top-k selection, in percentage points.

Task0123456789Overall
Seed 2+2+8+28+4+8−2+6+6−8−4+4.8
Seed 50+4+30−6+8−4+2+2−80+2.8
Seed 100+4+24−8+20−10+20−2+2+3.2
Mean+0.7+5.3+27.3−3.3+12.0−5.3+3.3+2.7−6.0−0.7+3.6

Table 10. MMDP minus top-k success rates (pp) for POKE on OpenVLA-OFT Long (B=32B = 32), shown separately for three training seeds.

Task-Level Contribution of Diversity

Task 2 accounts for much of the gain (+28 / +30 / +24 pp across seeds), and task 4 improves consistently. Tasks 5 and 8 regress for every seed. Excluding task 2, average gains fall to +2.22 / −0.22 / +0.89 pp. Thus, diversity repeatedly recovers particular weak tasks rather than uniformly improving the suite.

Table 11 reports single-checkpoint results on π0.5\pi_{0.5}. MMDP improves success on 32 of 40 tasks, ties on three, and reduces success on five, yielding an average gain of 11.75 percentage points across the four suites.

TaskSpatial Top-kkSpatial MMDPObject Top-kkObject MMDPGoal Top-kkGoal MMDPLong Top-kkLong MMDP
088100929492926672
186987410088986688
21001008610094987484
392100889450848490
46088889096986058
57698989648827686
610098889880986664
7789688941001006698
8961009610048885044
97092649674987680
Mean84.697.086.296.277.093.668.476.4
Δ\Delta (pp)+12.4+10.0+16.6+8.0

Table 11. Task-level POKE success rates (%) on π0.5\pi_{0.5} at B=32B = 32. Each task is evaluated over 50 episodes. Top-kk and MMDP denote the two POKE selection rules. Entries report single-checkpoint results, rather than averages over training seeds; task IDs are suite-specific.

Full Qualitative Rollouts

Figure 4 extends Figure 3 to all four methods on the same episode. FastV and VLA-Pruner reach the limit of 520 environment steps, while POKE (top-k) and POKE (MMDP) succeed at environment steps 220 and 278, respectively.

All methods on OpenVLA-OFT Long task 6

Figure 4. All methods on OpenVLA-OFT Long task 6 (B=32B = 32), from the same initial state as Figure 3. Layout and borders follow Figure 3.

Scoring Graphs and Mask Overlap

Table 1 reports the corresponding success and runtime; Table 12 and Table 13 add consecutive-mask overlap. Lower overlap alone does not explain higher success: on OFT, DivPrune has the lowest overlap and the lowest success.

MethodGoal SuccessGoal overlapLong SuccessLong overlap
Probe top-k (pruned)90.00.8159.60.85
Probe MMDP (pruned)91.60.6263.40.65
Probe top-k (full)96.20.3681.80.41
DivPrune80.80.2540.80.32
POKE (top-k)97.40.5383.93±0.810.62
POKE (MMDP)97.20.3987.53±1.860.44

Table 12. Scoring graphs and mask overlap on OpenVLA-OFT (B=32B = 32). Overlap is consecutive-mask Jaccard, averaged first within episodes and then across episodes. Online methods are evaluated in one session per suite. POKE Long success is mean±SD over three training seeds; POKE overlap uses one checkpoint (seed 0). These seed averages differ from the single-checkpoint results in Table 1.

MethodSpatial SuccessSpatial overlapObject SuccessObject overlap
Probe top-k (pruned)61.80.6876.80.69
Probe MMDP (pruned)83.20.5489.60.54
Probe top-k (full)89.40.4293.40.47
POKE (top-k)84.60.7486.20.78
POKE (MMDP)97.00.5596.20.54

Table 13. Scoring graphs and mask overlap on π₀.₅ (B=32B = 32), using the definition in Table 12. Masks were recorded for Spatial and Object. All values use a single run, matching the success rates in Table 1.

History-Graph Interventions

The attention rows of Table 5 vary the history graph and selection rule on OFT Long. All conditions use B=32B = 32

(16 tokens per camera), deletion at the input to layer 4, SactS_{\mathrm{act}} from layer 15, history window 3, γ=0.8\gamma= 0.8, and three warmup control steps. We evaluate 500 matched initial states once per condition. Executed actions always come from the masked forward; SvlS_{\mathrm{vl}} is read from that forward in all conditions.

History graph. Only the source of new SactS_{\mathrm{act}} history entries changes. The pruned condition averages layer-15 attention across heads and action-slot queries, scatters it to retained token positions, and fills excluded positions with zero. The full condition adds an unpruned forward on the same observation to obtain all 512 entries and discards its actions. Full history is a mechanistic intervention, not an acceleration method.

Selection. VP forms the union of the top-16 SvlS_{\mathrm{vl}} and top-16 SactS_{\mathrm{act}} candidates per camera, then selects 16 by max–min diversity. The SactS_{\mathrm{act}}-only condition selects its top-16 per camera with index-based tie breaking, matching the implementation’s fallback selection.

Metrics. We count only pruned control steps and exclude overlap/re-entry transitions whose preceding control step is warmup. With retained sets KtK_t, Kt−1K_{t-1} of size BB,

Jt=∣Kt∩Kt−1∣∣Kt∪Kt−1∣.(18)J_t = \frac{\lvert K_t \cap K_{t-1}\rvert}{\lvert K_t \cup K_{t-1}\rvert}. \tag*{(18)}

We average JtJ_t first within episodes and then across episodes, as in Table 12. Re-entry tokens are Kt∖Kt−1K_t \setminus K_{t-1}.

Using the actual decision scores, we reconstruct the per-camera candidate sets CvlC_{\mathrm{vl}} and CactC_{\mathrm{act}}. A re-entry belongs to the SvlS_{\mathrm{vl}}-only category if it lies in Cvl∖CactC_{\mathrm{vl}} \setminus C_{\mathrm{act}}. The reported fraction pools such token counts over all eligible control steps, then divides by the pooled number of re-entry tokens. It is therefore weighted by re-entry counts, unlike the episode-averaged Jaccard statistic. For SactS_{\mathrm{act}}-only selection, Kt=CactK_t = C_{\mathrm{act}}, so this fraction is zero by definition and displayed as a dash. Act. support is the number of visual tokens with nonzero history-aggregated SactS_{\mathrm{act}} at decision time. We discard each episode’s first three pruned control steps and average support over all remaining control steps pooled across the 500 episodes, rather than averaging episode means. This removes the full-width warmup entries from the three-step history.

After omitting the first three pruned control steps, SactS_{\mathrm{act}}-only with pruned history has Jaccard 1.000 at the reported precision and mean temporal-score support 32.13, nearly equal to the retention budget (B=32B = 32). These observations are consistent with an effectively fixed mask. Pruned-history VP has a larger mean support of 49.02; full history gives support 512 at every retained measurement for both selection rules. Under pruned history, the temporal-score support is contained in the union of masks represented in the window. Outside that union, scores are zero: such tokens cannot outrank positive-score candidates, although zero-score ties can permit selection if too few positive candidates remain.

For pruned-history VP, the remaining 31% of re-entries

also belong to CactC_{\mathrm{act}}. After warmup history expires, nonzero temporal scores can recover tokens absent from the immediately preceding mask but present earlier in the window. The SvlS_{\mathrm{vl}} route provides candidates outside that support; membership in either candidate set is not a causal attribution of the final diversity selection.

Reproduction checks. Pruned-history VP matches the 78.6% Long success rate in Table 1. The runner rejects mismatched cut, attention layer, history window, or budget. Matching aggregate success does not show that the masks or action trajectories are identical.

Comparisons across scoring methods. On OFT Long at B=32B = 32, the full-graph probe reaches 81.8%, compared with 69.4% for full-history SactS_{\mathrm{act}} alone. Both use per-camera top-k and scores delayed by one control step, but the attention condition additionally averages a three-step history, and the probe runs were collected separately. Full-history VP reaches 82.4%, compared with 88.0% for POKE (MMDP). These comparisons do not isolate the scoring signal: temporal aggregation, selection rules, and mask timing differ.

SpecPrune-VLA†^{\dagger}. The SpecPrune-VLA†^{\dagger} rows of Table 5 use the settings of Appendix § 8.4 on the same 500 OFT Long initial states, once per condition. Only the graph of the global attention changes between pruned and full history; VlocalV_{\mathrm{local}} and VdynV_{\mathrm{dyn}} are always computed at the current control step before the cut. Overlap and re-entry exclude transitions from warmup control steps, without further trimming. A re-entering token is attributed to the set or sets that selected it at control step tt; tokens added by the fill follow VglobalV_{\mathrm{global}} order and are counted with VglobalV_{\mathrm{global}}. Shares pool token counts over all eligible control steps, as for VP.

With VglobalV_{\mathrm{global}} alone and pruned history, the mask cannot change after warmup: tokens deleted at the previous control step have no global score, and the previously retained tokens fill all bb slots, so the overlap is 1.000 by construction. The same signal with full history reaches 84.4%, above the original combination with either history. Under pruned history, all re-entries come from the current-frame sets, mostly VdynV_{\mathrm{dyn}}; under full history, VglobalV_{\mathrm{global}} order supplies 39% of them, and adding the current-frame sets lowers success from 84.4% to 80.6%. The current-frame paths therefore compensate for the pruned history rather than adding information that the full-graph history lacks.

On π₀.₅ the same pattern is stronger. Pruned-history SpecPrune-VLA†^{\dagger} is more persistent than on OFT (overlap 0.763 versus 0.545); its re-entries come mostly from VdynV_{\mathrm{dyn}} (84%) and rarely from VlocalV_{\mathrm{local}} (16%, against 44% on OFT), and its success (9.4%) is close to Random (4.8%). With VglobalV_{\mathrm{global}} alone the mask again freezes (5.0%). Full history raises VglobalV_{\mathrm{global}} alone to 68.2% and the original combination to 50.4%, with layers, query tokens, budget, and initial noise unchanged, so the low Table 1 result reflects the pruned history rather than the layer mapping of Appendix § 8.4. With full history, adding the current-frame sets lowers success by 17.8 pp, against 3.8 pp on OFT.

Table 14. Detailed history-graph interventions (OpenVLA-OFT Long, B=32B = 32, one run). Success is in percent. Trimmed overlap excludes each episode’s first three pruned control steps, during which full-width warmup entries can remain in the history. Re-entry is the mean number of entering tokens per control step; SvlS_{\mathrm{vl}}-only is the pooled re-entry fraction defined in the text. Act. support is the mean number of tokens with nonzero temporal scores over the trimmed interval.

SelectionHistorySuccessOverlapTrimmed overlapRe-entrySvlS_{\mathrm{vl}}-onlyAct. support
VLA-Prunerpruned78.60.5150.51810.00.6949.02
VLA-Prunerfull82.40.3640.36814.60.48512.00
SactS_{\mathrm{act}}pruned3.40.9951.0000.08–32.13
SactS_{\mathrm{act}}full69.40.6640.6716.2–512.00

Table 14.

Table 15. SpecPrune-VLA† history-graph and path conditions (LIBERO-Long, B=32B = 32, 500 episodes, one run per condition). Re-entry is the mean number of entering tokens per pruned control step. The last four columns split re-entering tokens by the set that selected them: VglobalV_{\mathrm{global}} order (including the fill), VlocalV_{\mathrm{local}} only, VdynV_{\mathrm{dyn}} only, or both current-frame sets. Current-frame only is the sum of the last three, the analogue of the SvlS_{\mathrm{vl}}-only fraction of Table 14.

PathsHistorySuccessOverlapRe-entryCurrent-frame onlyVglobalV_{\mathrm{global}} orderVlocalV_{\mathrm{local}}VdynV_{\mathrm{dyn}} / both
OpenVLA-OFT
All (original)pruned72.60.5458.891.000.000.440.54 / 0.02
Allfull80.60.34314.930.610.390.260.34 / 0.01
VglobalV_{\mathrm{global}}pruned4.81.0000.00––––
VglobalV_{\mathrm{global}}full84.40.43112.250.001.00––
π0.5\pi_{0.5}
All (original)pruned9.40.7634.401.000.000.160.84 / 0.00
Allfull50.40.49410.810.560.440.120.44 / 0.00
VglobalV_{\mathrm{global}}pruned5.01.0000.00––––
VglobalV_{\mathrm{global}}full68.20.5399.630.001.00––

Table 15.

Shadow Scoring on a Shared Trajectory

Closed-loop comparisons of scoring graphs change the observations as well as the masks. To hold observations fixed, we replay P=1P = 1 probe scoring sequentially along the OFT Long fine-tuning demonstrations used for student training (B=32B = 32, 3,157 frames). No action is executed, so the observations do not depend on any condition. At each frame tt, the same probe vv is applied to four graphs. Full keeps all tokens; its per-camera top-16, FtF_t, is the reference selection. The reference is therefore the P=1P = 1 full-graph score with the same probe, not the exact Jacobian. Pruned-self deletes tokens outside the mask MM that this condition selected at the previous frame, reproducing the online pruned probe without its effect on the trajectory. Pruned-indep uses a mask redrawn at every frame, independently of scores and previous selections, by sampling 16 of the 256 tokens of each camera uniformly without replacement. The graph is reduced, but no selection feeds back. Pruned-good uses the full-graph probe’s selection at the previous frame, a good mask that this condition did not choose. Each condition then selects its per-camera top-16, KtK_t. Frames are spaced every eight demonstration steps, matching the eight environment steps executed between OFT control steps. As in deployment, the first three frames of each episode serve as an unpruned warmup and are excluded from all metrics, leaving 3,157 frames.

With conditioning mask MM, top-BB recall is ∣Kt∩Ft∣/B|K_t \cap F_t|/B and mask overlap is ∣Kt∩M∣/∣Kt∣|K_t \cap M|/|K_t|, with ∣Kt∣=32|K_t| = 32. Excluded mass is the share of the condition’s scores ss on tokens outside MM; full-graph mass is the share of full-graph scores on the same tokens. Re-entry recall is ∣Kt∩(Ft∖M)∣/∣Ft∖M∣|K_t \cap(F_t \setminus M)|/|F_t \setminus M|, averaged over frames with a nonzero denominator; the pooled ratio of total hits to total denominators gives nearly the same values (0.083, 0.078, and 0.168 for self, good, and indep). Spearman correlations with full-graph scores use all tokens. Values are means over frames.

Three observations follow. First, every pruned graph strongly favors the tokens it retains: excluded tokens receive 19–28% of the pruned-graph score mass, compared with 72–94% of the full-graph mass on the same tokens. Second, this bias does not require feedback. Because the independent mask is drawn uniformly, a selection that ignored it would overlap it by 16/256 ≈ 6% in expectation, yet 59% of the pruned-indep selection stays inside it. Third, @page 19/27 starts_mid=1 ends_mid=1

Table 14. Detailed history-graph interventions (OpenVLA-OFT Long, B=32B = 32, one run). Success is in percent. Trimmed overlap excludes each episode’s first three pruned control steps, during which full-width warmup entries can remain in the history. Re-entry is the mean number of entering tokens per control step; SvlS_{\mathrm{vl}}-only is the pooled re-entry fraction defined in the text. Act. support is the mean number of tokens with nonzero temporal scores over the trimmed interval.

SelectionHistorySuccessOverlapTrimmed overlapRe-entrySvlS_{\mathrm{vl}}-onlyAct. support
VLA-Prunerpruned78.60.5150.51810.00.6949.02
VLA-Prunerfull82.40.3640.36814.60.48512.00
SactS_{\mathrm{act}}pruned3.40.9951.0000.08–32.13
SactS_{\mathrm{act}}full69.40.6640.6716.2–512.00

Table 14 (PDF p. 19).

Table 15. SpecPrune-VLA† history-graph and path conditions (LIBERO-Long, B=32B = 32, 500 episodes, one run per condition). Re-entry is the mean number of entering tokens per pruned control step. The last four columns split re-entering tokens by the set that selected them: VglobalV_{\mathrm{global}} order (including the fill), VlocalV_{\mathrm{local}} only, VdynV_{\mathrm{dyn}} only, or both current-frame sets. Current-frame only is the sum of the last three, the analogue of the SvlS_{\mathrm{vl}}-only fraction of Table 14.

PathsHistorySuccessOverlapRe-entryCurrent-frame onlyVglobalV_{\mathrm{global}} orderVlocalV_{\mathrm{local}}VdynV_{\mathrm{dyn}} / both
OpenVLA-OFT
All (original)pruned72.60.5458.891.000.000.440.54 / 0.02
Allfull80.60.34314.930.610.390.260.34 / 0.01
VglobalV_{\mathrm{global}}pruned4.81.0000.00––––
VglobalV_{\mathrm{global}}full84.40.43112.250.001.00––
π0.5\pi_{0.5}
All (original)pruned9.40.7634.401.000.000.160.84 / 0.00
Allfull50.40.49410.810.560.440.120.44 / 0.00
VglobalV_{\mathrm{global}}pruned5.01.0000.00––––
VglobalV_{\mathrm{global}}full68.20.5399.630.001.00––

Table 15 (PDF p. 19).

bination to 50.4%, with layers, query tokens, budget, and initial noise unchanged, so the low Table 1 result reflects the pruned history rather than the layer mapping of Appendix § 8.4. With full history, adding the current-frame sets lowers success by 17.8 pp, against 3.8 pp on OFT.

Shadow Scoring on a Shared Trajectory

Closed-loop comparisons of scoring graphs change the observations as well as the masks. To hold observations fixed, we replay P=1P = 1 probe scoring sequentially along the OFT Long fine-tuning demonstrations used for student training (B=32B = 32, 3,157 frames). No action is executed, so the observations do not depend on any condition. At each frame tt, the same probe vv is applied to four graphs. Full keeps all tokens; its per-camera top-16, FtF_t, is the reference selection. The reference is therefore the P=1P = 1 full-graph score with the same probe, not the exact Jacobian. Pruned-self deletes tokens outside the mask MM that this condition selected at the previous frame, reproducing the online pruned probe without its effect on the trajectory. Pruned-indep uses a mask redrawn at every frame, independently of scores and previous selections, by sampling 16 of the 256 tokens of each camera uniformly without replacement. The graph is reduced, but no selection feeds back. Pruned-good uses the full-graph probe’s selection at the previous frame, a good mask that this condition did not choose. Each condition then selects its per-camera top-16, KtK_t. Frames are spaced every eight demonstration steps, matching the eight environment steps executed between OFT control steps. As in deployment, the first three frames of each episode serve as an unpruned warmup and are excluded from all metrics, leaving 3,157 frames.

With conditioning mask MM, top-BB recall is ∣Kt∩Ft∣/B|K_t \cap F_t|/B and mask overlap is ∣Kt∩M∣/∣Kt∣|K_t \cap M|/|K_t|, with ∣Kt∣=32|K_t| = 32. Excluded mass is the share of the condition’s scores ss on tokens outside MM; full-graph mass is the share of full-graph scores on the same tokens. Re-entry recall is ∣Kt∩(Ft∖M)∣/∣Ft∖M∣|K_t \cap(F_t \setminus M)|/|F_t \setminus M|, averaged over frames with a nonzero denominator; the pooled ratio of total hits to total denominators gives nearly the same values (0.083, 0.078, and 0.168 for self, good, and indep). Spearman correlations with full-graph scores use all tokens. Values are means over frames.

Three observations follow. First, every pruned graph strongly favors the tokens it retains: excluded tokens receive 19–28% of the pruned-graph score mass, compared with 72–94% of the full-graph mass on the same tokens. Second, this bias does not require feedback. Because the independent mask is drawn uniformly, a selection that ignored it would overlap it by 16/256 ≈ 6% in expectation, yet 59% of the pruned-indep selection stays inside it. Third, the conditioning mask sets how much of the full-graph ranking survives. Starting from the full probe’s previous mask, pruned scoring recovers 40% of the full-graph top-BB, compared with 32% from its own previous mask and 22% from an independent mask, while overlap remains above 80% for both previous-frame masks. Feedback thus selects which tokens are locked in, and the mask-conditioned graph keeps them. Re-entry recall is low in all pruned conditions. These are open-loop replays on demonstrations, so they account for the persistence in Table 12 but do not directly predict closed-loop success.

GraphTop-BB rec. ↑\uparrowOverlapExcl. massFull massRe-entry rec. ↑\uparrowSpearman
Full1.000––––1.000
Pruned-self0.3180.8290.1980.7840.0830.498
Pruned-good0.3950.8570.1860.7170.0770.516
Pruned-indep0.2180.5940.2840.9370.1680.447

Table 16. Shadow scoring on OFT Long demonstration frames (B=32B = 32, 3,157 frames). All conditions use the same probe at each frame and differ only in the graph; the reference is the P=1P = 1 full-graph selection. Excl. mass and Full mass are the shares of the condition’s and the full-graph scores on tokens outside the conditioning mask.

Joint Student for π0.5\pi_{0.5}

The joint student uses the same architecture (140,225 parameters), objective, and optimization as the per-suite students, trained on the union of the four suites’ cached labels with their original episode splits (6,838 training and 1,683 validation frames). No labels are recomputed, so it adds no teacher cost. Frames are not reweighted, so Long contributes about 40% of training data. The checkpoint with the lowest validation KL on the pooled validation set is selected (epoch 18). Table 17 compares it with the per-suite students on each suite’s validation frames and in closed loop.

StudentSpatialObjectGoalLongAvg.
Val KL ↓\downarrowPer-suite0.13410.12810.13670.1327–
Joint0.13350.12750.13640.1324–
Recall@32 ↑\uparrowPer-suite0.5530.5850.5400.536–
Joint0.5550.5860.5450.541–
Success (%)Per-suite97.096.293.676.490.80
Joint96.497.091.077.890.55

Table 17. Per-suite and joint students on π0.5\pi_{0.5}. Validation metrics use each suite’s validation frames; success uses MMDP at B=32B = 32 with 500 episodes per suite. Per-suite success values are from Table 1.

The joint student matches or slightly improves offline label fit on every suite. Closed-loop differences range from −2.6 pp (Goal) to +1.4 pp (Long), with an average of −0.25 pp. With 500 episodes, the standard error of a single suite’s success rate is about 1.2 pp near 92%, and per-suite

checkpoints vary by 1–3 pp across training seeds (Table 4), so we do not interpret these differences as systematic. Both students are single checkpoints.

Transfer Across Suites and Budgets

Across the four rule/budget combinations, Spatial-to-Long transfer loses 3.1 pp on average relative to target-suite training, whereas the reverse direction loses approximately 0.4 pp. Scene diversity may explain the asymmetry, but we did not separate it from other suite differences.

Autoregressive Policy: OpenVLA

We apply POKE to OpenVLA [7], retaining BB of 256 third-person visual tokens at the input to layer 3. Sensitivities use the continuous relaxation in Appendix § 8.1 (A=7A = 7). We evaluate B=32B = 32.

At B=32B = 32 (Table 19), POKE (MMDP) reaches 71.75% average success, 97.6% of Vanilla (73.50%), and exceeds VP (63.80%) by 7.95 pp, with gains on all four suites (+3.2 to +11.6 pp, largest on Goal and Long). POKE (top-kk) reaches 67.05%, so diversity selection adds 4.70 pp, between the gains on OpenVLA-OFT (1.35 pp) and π0.5\pi_{0.5} (11.75 pp). Full-graph probe MMDP reaches 71.15%, Table 20. Latency (ms/call) and speedup over Vanilla on Open-VLA Goal (B=32B = 32), using the protocol in Appendix § 8.5.

MethodSpatialObjectGoalLongAvg.
Vanilla84.684.074.051.473.50
Random12.241.231.28.823.35
FastV61.274.258.836.457.65
VLA-Cache67.471.060.234.258.20
VLA-Pruner75.676.261.442.063.80
Probe top-kk (full)72.476.266.443.464.60
Probe MMDP (full)78.080.872.253.671.15
POKE (top-kk)72.881.270.044.267.05
POKE (MMDP)78.882.272.453.671.75

Table 19. LIBERO success rates (%) for autoregressive OpenVLA at B=32B = 32 of 256 visual tokens, with 500 episodes per suite. Parentheses in online-probe names indicate the scoring graph. Bold marks the highest non-Vanilla value in each column, including ties.

MethodmsSpeedup
Vanilla125.81.00×
VLA-Pruner94.21.34×
VLA-Cache101.11.24×
Probe top-kk (full)197.80.64×
POKE (top-kk)88.21.43×
POKE (MMDP)93.11.35×

Table 20.

within 0.6 pp of POKE (MMDP), which needs no online backpropagation. On Goal (Table 20), POKE (MMDP) runs at 1.35×1.35\times, comparable to VP (1.34×1.34\times), whereas full-graph probe top-kk is slower than Vanilla (0.64×0.64\times).

The squared-sensitivity estimator has lower relative variance at A=7A = 7 than at A=56A = 56 (1.33 vs. 1.90), although single-probe noise remains substantial (Appendix § 16). For a fixed token pair, Eq. (21) depends on the norm ratio rr and alignment ρ\rho, not directly on AA. Both MMDP variants exceed Vanilla on Long (53.6 vs. 51.4%); we do not read this as evidence that pruning improves the underlying policy.

SimplerEnv with π0\pi_0

Policy and environment. We use open-pi-zero [20], a PyTorch re-implementation of π0\pi_0 [2], with its public Bridge checkpoint (Beta time sampling), trained on BridgeData V2 [23]. It takes one 224×224224 \times224 third-person image (256 SigLIP tokens), at most 20 text tokens, and one proprioceptive token, and predicts a chunk of four 7-D actions with ten Euler steps of its action expert against the cached prefix. We evaluate the four WidowX visual-matching tasks of SimplerEnv [10] with the authors’ protocol: 24 object configurations per task, 10 trials each (240 episodes), all four predicted actions executed, and success read at the time limit. With open-pi-zero’s own evaluation code in float32, Vanilla reaches 57.5/85.0/82.9/45.4% (Carrot, Eggplant, Spoon, Stack; 67.7% on average) against the reported 55.8/85.4/84.6/47.9% (68.4%).

Pruning. We delete visual tokens at the input to layer 3 of the 18-layer Gemma backbone, the same cut as for π0.5\pi_{0.5}, keeping B=32B = 32 of 256 tokens; retained tokens keep their original position ids, and the action expert reads the reduced cache at every flow step. As on LIBERO, the first three control steps of each episode are unpruned, and the initial flow noise is fixed per (task, episode, control step), so all methods share the noise at matching control steps and outcomes are paired by episode. Our pruned forward pass re-implements the layer loop with the policy’s own modules; without pruning it reproduces open-pi-zero’s action output exactly under the same noise, and the resulting

Vanilla (68.2%) agrees with the reproduction above within 0.5 pp. FastV reads attention at layer 2; VLA-Pruner follows its π0\pi_0 setting (Appendix § 8.2), with action-to-visual attention averaged over the latter half of the expert layers and flow steps and a window of three control steps (γ=0.8\gamma= 0.8).

Teacher and student. Teacher labels follow the π0.5\pi_{0.5} protocol: one probe through all ten flow steps of the full graph, camera-mean baseline, and the final (unclipped) chunk of A=28A = 28 dimensions in executed units. Labels come from the policy’s training data: eight shards spread evenly over the 1,024 training shards of BridgeData V2, skipping episodes without instructions, which gives 300 episodes and 2,758 frames at a stride of four environment steps (the control interval). This takes 0.82 s per frame (38 minutes) on one GPU. The student has the architecture of § 3.4 with one camera and a 7-D proprioceptive input (140,161 parameters) and is trained as in Appendix § 9 on an 80/20 episode split (2,242/516 frames); the selected checkpoint reaches a validation top-32 recall of 0.61 (0.125 by chance). The student is thus trained on real images and deployed on simulated ones without adaptation.

Results. Table 21 pairs each method with VLA-Pruner by episode. POKE (MMDP) succeeds on 152 episodes where VLA-Pruner fails and fails on 91 where it succeeds (exact McNemar p=1.1×10−4p = 1.1 \times10^{-4} over 960 episodes), with significant gains on Eggplant and Stack. POKE (top-kk) ties VLA-Pruner (44.27% each), so on this flow policy, as on π0.5\pi_{0.5}, score filtering alone does not separate POKE from the attention baselines and diversity selection over the candidates does. Eggplant, whose sink scene places the target far from the gripper, drops most for every method; Vanilla keeps 84.6%, POKE (MMDP) 30.8%, VLA-Pruner 20.4%, and FastV 7.5%.

MethodCarrotEggplantSpoonStackPooled
FastV19/24 (0.54)9/40 (9.3×10−6\times10^{-6})21/37 (0.048)40/31 (0.34)89/132 (0.0046)
POKE (top-kk)22/24 (0.88)37/38 (1.00)30/45 (0.11)39/21 (0.027)128/128 (1.00)
POKE (MMDP)21/24 (0.87)52/27 (0.0066)25/54 (0.19)46/19 (0.0011)152/91 (1.1×10−4\times10^{-4})

Table 21.

Table 21. SimplerEnv paired comparison with VLA-Pruner (π0\pi_0, B=32B = 32). Entries: episodes where only the method succeeds / only VLA-Pruner succeeds (two-sided exact McNemar pp), 240 episodes per task.

LIBERO-Plus Evaluation

Protocol. LIBERO-Plus [4] provides 2,402, 2,518, 2,591, and 2,519 perturbed variants of the Spatial, Object, Goal, and Long tasks. Within each suite we group tasks by perturbation category and difficulty level, fix a random order within each group, and take tasks in proportion to group size, giving 600, 630, 648, and 630 tasks (25%). Every method and both policies evaluate exactly these tasks with the initial state of the official evaluation and one episode per task. Episode limits, the settle steps, action chunking, the cut layer, the per-camera budget, and the three unpruned warmup control steps are those of the main experiments. For π0.5\pi_{0.5}, initial noise is seeded by (suite, task, episode, control step) as in Appendix § 8.5. The policies, POKE students, and baselines are those of Table 1; no component is retrained or re-tuned on perturbed data. Retention is a method’s success divided by Vanilla success on the same tasks.

Table 22 reports per-suite success for the same runs. On OpenVLA-OFT, both POKE variants exceed VP on every suite and MAP† on every suite but Goal, where all three are within 0.5 pp.

Detailed Runtime Results

POKE (top-kk) is faster than POKE (MMDP) in every reported suite/budget condition, consistent with MMDP’s additional candidate-distance computation. The difference is approximately 1.6 ms at B=32B = 32, 2.7 ms at B=64B = 64, and 4.8 ms at B=128B = 128. POKE (MMDP) beats VP in 83% of timing repetitions at B=128B = 128, versus 93–98% in the other settings; top-kk beats VP in at least 98% throughout.

Pruned probes exhibit greater timing variation: B=32B = 32 speedups span 1.04–1.16× across the recorded runs, with Spatial near the upper end. At B=64,128B = 64, 128, the reported ranges narrow to less than 0.01×, but speedups fall below one. Other methods vary within approximately ±0.02× across the four suites.

Success changes little at R=2R = 2, but drops on all three suites at R=8R = 8 (−6.6, −3.8, and −4.8 pp relative to R=1R = 1). Mask reuse thus trades scoring cost for stale selections, whereas the POKE student scores every observation.

Timing

The eager PyTorch path spends most of its measured time in the ten-step action-expert loop (Table 25). We use CUDA graphs for the image encoder, VLM prefix, and denoising loop to reduce launch overhead. The prefix is split around selection into layers 00–k−1k - 1 and kk–17. FastV selection, VP’s combine-then-filter operation, student scoring, and MMDP run outside the graphs and remain within the timing boundary. For the tested identical inputs, graph-path actions and VP action attention match the eager outputs with maximum absolute error zero, including FastV/VP selection. Online-probe backward passes are not captured. All methods, including Vanilla, online probes, and POKE, share one re-implemented path that omits the masked camera slot and prompt padding, so ratios are unaffected. This Vanilla is faster than openpi’s reference path (334 ms per call), so using it as the denominator is conservative. Timings use an RTX A6000 and five processes with 50 repetitions each. Online-probe backward passes are not captured, so all probe timings on π0.5\pi_{0.5} use the eager path and are compared with eager Vanilla (Table 26).

MethodB=32B = 32 msSpeedupB=64B = 64 msSpeedupB=128B = 128 msSpeedup
Vanilla300.01.00×306.01.00×299.91.00×
FastV290.31.03× [1.03–1.04]296.71.03× [1.03–1.04]290.51.03× [1.03–1.03]
VLA-Pruner296.91.01× [1.01–1.01]304.61.00× [1.00–1.00]301.01.00× [0.99–1.00]
POKE (top-kk)290.71.03× [1.03–1.03]297.21.03× [1.03–1.03]290.91.03× [1.03–1.03]
POKE (MMDP)292.51.03× [1.02–1.03]300.41.02× [1.01–1.02]296.61.01× [1.01–1.01]
Probe top-kk (pruned)636.30.48× [0.46–0.49]646.30.47× [0.46–0.49]638.70.48× [0.46–0.49]
Probe MMDP (pruned)638.10.48× [0.46–0.49]650.70.47× [0.46–0.48]645.60.47× [0.46–0.48]
Probe top-kk (full)925.90.32× [0.32–0.33]951.00.32× [0.32–0.33]939.40.32× [0.31–0.33]

Table 26. π0.5\pi_{0.5} latency (ms) and speedups in eager mode, where every method, including online-probe backward passes, runs on the same execution path. Speedups are medians of paired ratios over five processes (ranges in brackets); Vanilla omits masked camera/prompt padding as in Table 27.

In eager mode the denoising loop dominates (Table 25), so pruning yields at most 1.03×, whereas the full-graph probe runs at 0.32×; CUDA graphs remove launch overhead and expose the prefix savings.

Under CUDA graphs, pruning to B=32B = 32 makes the prefix 2.46× faster (28.5 to 11.6 ms), consistent with the 4.4× reduction in prefix FLOPs, but leaves the denoising loop nearly unchanged (42.9 to 41.5 ms). The loop runs the action expert ten times over a short action sequence, so shortening the cached prefix barely changes its cost, and it accounts for two thirds of the pruned call. The full call therefore speeds up by 1.31×, and the end-to-end 1.26× in Table 27 additionally includes transforms and token selection. The same structure explains the small latency change between B=128B = 128 and B=32B = 32 (Table 28).

MethodOpenVLA-OFT msOpenVLA-OFT Speedupπ0.5\pi_{0.5} msπ0.5\pi_{0.5} Speedup
Vanilla130.11.00×84.71.00×
FastV69.21.88× [1.86–1.88]65.61.29× [1.29–1.29]
VLA-Pruner71.51.82× [1.80–1.82]68.51.24× [1.24–1.24]
VLA-Cache88.61.47× [1.47–1.48]––
Probe top-k (pruned)123.01.05× [1.04–1.15]630.20.13× [0.13–0.14]
Probe MMDP (pruned)124.81.04× [1.03–1.14]632.40.13× [0.13–0.14]
Probe top-k (full)321.00.41× [0.41–0.41]700.40.12× [0.12–0.13]
POKE (top-kk)68.81.89× [1.88–1.89]65.81.29× [1.29–1.29]
POKE (MMDP)70.41.85× [1.84–1.85]67.41.26× [1.26–1.26]

Table 27. Latency (ms/call) and speedups at B=32B = 32. Speedups are medians of paired ratios with randomized method order; brackets give ranges over five independent processes. For π0.5\pi_{0.5}, the image encoder, prefix, and denoising loop use CUDA graphs; selection and probe backward passes run outside them. Vanilla uses the same path with masked camera/prompt padding omitted. See Appendix § 8.5.

MethodB=32B = 32 msB=32B = 32 SpeedupB=64B = 64 msB=64B = 64 SpeedupB=128B = 128 msB=128B = 128 Speedup
Vanilla84.71.00×84.61.00×84.81.00×
FastV65.61.29×66.61.27×69.51.22×
VLA-Pruner68.51.24×70.51.20×75.41.12×
Probe top-kk (pruned)630.20.13×625.00.14×629.90.13×
Probe MMDP (pruned)632.40.13×628.80.13×635.70.13×
Probe top-kk (full)700.40.12×697.20.12×704.40.12×
POKE (top-kk)65.81.29×66.81.27×69.61.22×
POKE (MMDP)67.41.26×69.51.22×74.71.14×
Backbone FLOP22.9%28.1%38.3%

Table 28. Budget-dependent π0.5\pi_{0.5} latency (ms) and speedups with CUDA graphs. FLOPs are percentages of the unpruned VLM backbone; values for B=64,128B = 64, 128 are estimates from layer-wise token counts.

VLA-Pruner reports 1.95× on π0\pi_0 at 12.5% retention (RTX 4090; Vanilla 104.5 ms). That result uses a different policy, implementation, and hardware, and we did not reproduce its pipeline. In ours, the 35% prefix share bounds any token pruner at 1.54×.

VP costs 2.9–5.9 ms more than FastV, while POKE (MMDP) costs 1.6–5.1 ms more than POKE (top-kk) across budgets. The latter differs from FastV by at most 0.2 ms. These are differences between whole pipelines, not direct measurements of the student scorer’s cost. Selection overhead grows with budget in these measurements, but we did not characterize how it scales.

Statistical Analyses

Episode-paired tests match initial states. Bootstrap intervals resample paired episodes within fixed tasks. Sign tests use tasks as units. We do not treat similar point estimates as evidence of equivalence or of improvement unless a test supports it.

SuiteBBVanillaVLA-PrunerFastVPOKE (top-kk)POKE (MMDP)Probe (pruned)
Spatial32129.970.7 / 1.84×68.6 / 1.90×68.1 / 1.91×69.7 / 1.87×1.16×
Object32129.070.7 / 1.82×68.3 / 1.88×68.0 / 1.90×69.6 / 1.85×1.05×
Goal32130.171.5 / 1.82×69.2 / 1.88×68.8 / 1.89×70.4 / 1.85×1.05×
Long32130.772.0 / 1.82×69.2 / 1.89×69.2 / 1.89×70.9 / 1.85×1.06×
Spatial64130.779.2 / 1.65×75.9 / 1.72×75.5 / 1.73×78.1 / 1.68×0.97×
Object64130.478.6 / 1.66×75.5 / 1.72×75.0 / 1.73×77.6 / 1.68×0.97×
Goal64130.378.6 / 1.66×75.5 / 1.73×74.9 / 1.74×77.8 / 1.68×0.97×
Long64130.478.1 / 1.66×75.2 / 1.73×74.8 / 1.74×77.3 / 1.68×0.97×
Goal128129.997.5 / 1.33×92.9 / 1.40×91.9 / 1.42×96.7 / 1.35×0.84×

Table 23. OpenVLA-OFT latency (ms/call) and paired speedups over Vanilla, using Appendix § 8.5. Only Goal is timed at B=128B = 128. The final column reports speedups only.

The Goal graph effect remains significant after correction, and full-graph top-kk also exceeds pruned-graph MMDP. The same-session selection-rule comparison under pruned scoring is not significant.

For Long’s +22.2 pp graph effect, nine of ten tasks improve (two-sided sign test p=0.0215p = 0.0215). An unpaired approximation gives z=7.95z = 7.95, but it is not an episode-paired analysis, so the main text reports effect size and task-wise direction instead.

Surrogate Fidelity and Offline Mask Quality

Three evaluation sets. All analyses in this section use OFT, B=32B = 32, and deletion at layer 4.

  • Set 1: mask quality. We apply alternative masks to 120 frames from 43 Goal validation episodes, sampled at frame indices 3/8/13/18/23 after warmup. Deviations are normalized by the policy’s dimension-wise output ranges. An initial filename-ordered sample mostly contained training episodes; its POKE results were discarded. All reported Set 1 results were rerun on validation episodes.

  • Set 2: replacement and probe analysis. On 100 validation frames from Spatial, we compute and store exact ui=Jiδiu_i = J_i\delta_i at the configured model precision. We resample probes across ten seeds without recomputing the Jacobian. Replacement/deletion comparisons use seven masks per frame: exact top-kk, three probe top-kk masks, and three random masks (700 frame–mask pairs).

  • Set 3: shared-observation comparisons. Vanilla closed-loop rollouts on Long use ten tasks and two prespecified initial states per task (indices 0 and 1). After warmup,

FastV, VP, both POKE variants, and three random masks are evaluated on each of the 625 shared observations, one per control step. These evaluation forwards do not control the environment. VP updates its temporal signal along the entire Vanilla trajectory, including warmup. Deviation is the chunk-wide L2 norm after normalization by dimension-wise q01–q99 ranges, divided by the mean deviation of three random masks on the same frame. Frames and masks are repeated measurements. We report distributions for Set 2 without tests that assume independent frame–mask pairs. For Set 3, we aggregate paired frame-wise deviation differences within each task by the median and apply sign tests at the task level.

Single-probe variance. For a uniform spherical v∈RAv \in\mathbb{R}^{A} and unit u^\hat{u}, E[(vTu^)2]=1/A\mathbb{E}[(v^{\mathsf{T}}\hat{u})^{2}] = 1/A and E[(vTu^)4]=3/[A(A+2)]\mathbb{E}[(v^{\mathsf{T}}\hat{u})^{4}] = 3/[A(A + 2)]. Thus s^i2=A(vTui)2\hat{s}_{i}^{2} = A(v^{\mathsf{T}}u_{i})^{2} is unbiased for si2s_{i}^{2} and, for si>0s_{i} > 0,

Var⁡[s^i2](E[s^i2])2=3AA+2−1=2(A−1)A+2.(19)\frac{\operatorname{Var}[\hat{s}_{i}^{2}]}{(\mathbb{E}[\hat{s}_{i}^{2}])^{2}} = \frac{3A}{A + 2} - 1 = \frac{2(A - 1)}{A + 2}. \tag*{(19)}

This ratio increases with AA toward two (1.33 at A=7A = 7, 1.90 at A=56A = 56, and 1.92 at $A = 70). Averaging PP independent squared-score probes divides the relative variance by PP; the relative standard deviation decreases as $1/\sqrt{P}$. The ranking argument in § 3.2 instead relies on shared multiplicative factors.

Deriving the reversal probability. With a shared probe, s^i=Asiκi\hat{s}_{i} = \sqrt{A}s_{i}\kappa_{i} with κi=∣vTu^i∣\kappa_{i} = |v^{\mathsf{T}}\hat{u}_{i}|. For si>sj>0s_{i} > s_{j} > 0, the probe reverses the pair exactly when κj/κi>r\kappa_{j}/\kappa_{i} > r, apart from the zero-probability event κi=0\kappa_{i} = 0. Fix nonzero effects with si>sjs_{i} > s_{j}, r=si/sj>1r = s_{i}/s_{j} > 1, ρ=u^iTu^j\rho= \hat{u}_{i}^{\mathsf{T}}\hat{u}_{j}, and A≥2A \ge2. For ∣ρ∣<1|\rho| < 1, write v=z/∥z∥v = z/\|z\|, z∼N(0,IA)z \sim\mathcal{N}(0,I_{A}). Normalization cancels from the ratio of projections. Then X=zTu^iX = z^{\mathsf{T}}\hat{u}_{i} and Y=zTu^jY = z^{\mathsf{T}}\hat{u}_{j} are standard normal with correlation ρ\rho. With independent standard-normal WW, Y=ρX+1−ρ2WY = \rho X + \sqrt{1-\rho^{2}}W. Since W/XW/X is standard Cauchy, Y/XY/X is Cauchy with location ρ\rho and scale σ=1−ρ2\sigma= \sqrt{1-\rho^{2}}. Its two tails give

Pr⁡[∣Y/X∣>r]=1−1π[arctan⁡r−ρσ+arctan⁡r+ρσ].(20)\Pr[|Y/X| > r] = 1 - \frac{1}{\pi}\left[\arctan\frac{r-\rho}{\sigma} + \arctan\frac{r+\rho}{\sigma}\right]. \tag*{(20)}

For r>1r > 1, the two positive arctangent arguments have product greater than one. The arctangent addition identity therefore yields

Pr⁡v[s^j>s^i]=1πarctan⁡(2r1−ρ2r2−1).(21)\Pr_{v}[\hat{s}_{j} > \hat{s}_{i}] = \frac{1}{\pi}\arctan\left(\frac{2r\sqrt{1-\rho^{2}}}{r^{2}-1}\right). \tag*{(21)}

This expression is independent of AA conditional on (r,ρ)(r,\rho). At ρ=0\rho= 0 it is exactly 2πarctan⁡(1/r)\frac{2}{\pi}\arctan(1/r); at ∣ρ∣=1|\rho| = 1, the common absolute projection factor gives zero reversal probability for a strict norm ordering. The reported Monte Carlo check uses $2 \times 10^{5}probespercombinationofprobes per combination ofA \in {7, 56, 70}, r∈{1.5,3,10}r \in\{1.5, 3, 10\}, and ρ∈{0,0.5,−0.9}\rho\in\{0, 0.5, -0.9\}, with absolute errors below 0.003.

Comparison (A − B)A onlyB onlyDifference (pp)Paired 95% CIExact ppHolm pp
Full − Pruned, top-kk409+6.2[3.6, 8.8]9.3×10−69.3 \times10^{-6}2.8×10−52.8 \times10^{-5}
Full top-kk − Pruned MMDP3512+4.6[2.0, 7.4]1.1×10−31.1 \times10^{-3}2.2×10−32.2 \times10^{-3}
Pruned MMDP − Pruned top-kk3426+1.6[−1.4, 4.6]0.3660.366

Table 29. Paired-episode analyses of the scoring graph on OpenVLA-OFT Goal (B=32B = 32). A only counts episodes where A succeeds and B fails. Exact pp values use a two-sided binomial test on discordant pairs. Holm correction treats the three comparisons as one family.

ResampledPairwise SpearmanPairwise Recall@32Against mean SpearmanAgainst mean Recall@32
ε\varepsilon only0.930 (0.030)0.854 (0.063)0.957 (0.022)0.882 (0.071)
Probe vv only0.768 (0.031)0.566 (0.058)0.864 (0.019)0.659 (0.056)
ε\varepsilon and vv0.763 (0.029)0.560 (0.056)0.860 (0.019)0.652 (0.060)

Table 30. Resampling stability of π0.5\pi_{0.5} teacher labels: 200 Long validation frames and eight samples per condition. Values are means across frames, with between-frame SDs in parentheses. Recall uses 16 tokens per camera (32 in total).

MasknnMedian ratioIQR
Overall7001.076[0.996, 1.200]
Exact top-kk1001.117[0.986, 1.333]
Probe top-kk (seed 0/1/2)3001.115–1.139–
Random (seed 0/1/2)3001.034–1.040–
Within-frame Spearman(Drepl,DdeleteD_{\mathrm{repl}}, D_{\mathrm{delete}}): median 0.964, Mean 0.933

Table 31. Replacement versus deletion deviations (Set 2). The ratio is Drepl/DdeleteD_{\mathrm{repl}}/D_{\mathrm{delete}}. Within-frame Spearman compares the seven masks evaluated at one observation.

If effect directions are themselves sampled independently and uniformly, one additionally averages over their alignment distribution. Here E[ρ2]=1/AE[\rho^2] = 1/A, and each conditional reversal probability is at most its orthogonal value. The reported direction-averaged values at r=3r = 3 are 0.192 for A=7A = 7 and 0.203 for A=56A = 56, versus 0.205 for ρ=0\rho= 0. This dimension-dependent mixture is distinct from the fixed-pair result; actual policy alignment is measured in Table 33.

signed cos∣cos⁡∣\lvert\cos\rvertcos⁡2\cos^2
All token pairs0.0570.3240.155
Top-32 pairs0.0510.3860.211
Isotropic(A=56A=56)00.1070.018

Table 33.

Noise dependence of flow-policy labels. Each π0.5\pi_{0.5} teacher label uses one initial noise ε\varepsilon and one probe vv. On 200 Long validation frames, we resample (i) only ε\varepsilon with vv fixed to its teacher value, (ii) only vv with ε\varepsilon fixed, or (iii) both, using eight full-graph samples per condition. Pairwise metrics average comparisons between sample pairs from the same frame. Against-mean metrics compare each sample to the mean of its eight score vectors, including that sample. Spearman is computed within cameras; recall uses the top-16 per camera, for 32 tokens total. Recomputing cached labels with identical (ε,v)(\varepsilon, v) gives maximum relative difference zero in the reported check.

Rankings are more stable under noise-only resampling than under probe-only resampling, while resampling both gives results close to probe-only. This compares ranking stability only; it is not a variance decomposition and does not show that noise effects are negligible. The student, which does not receive ε\varepsilon, learns an observation-conditioned approximation, and the mixed training objective need not average out its noise dependence exactly. The analysis covers only Long and does not test whether single-noise and noise-averaged labels lead to different closed-loop success. Against-mean metrics use a reference that includes the sample itself, not an independent estimate of the expected score.

Replacement versus deletion. Set 2 compares projector-output replacement (replace) and deletion at the cut (delete_k) for identical retained sets.

Replacement and deletion have strongly associated within-frame damage rankings, with a median deviation ratio of 1.076 (about 8% higher for replacement). The ratio is somewhat larger for top-kk masks. The high correlation supports the surrogate empirically, although the two can still select different tokens. The pooled correlation across 700 frame-mask pairs is 0.965; because it mixes frames and mask types, we report within-frame correlations in the main text. In Set 1, cut-layer deletion produces 17–22% less deviation than projector deletion on the same retained sets (Ddelete−k/Ddelete_0=0.78D_{\mathrm{delete-k}}/D_{\mathrm{delete\_0}} = 0.78–0.83), supporting later deletion for this one-step fidelity measure.

Single-probe ranking. Set 2 compares exact sensitivities with P=1P = 1 estimates.

Within-camera rank correlation is approximately 0.80, with small variation across the ten sampled probe seeds. Retained sensitivity energy is 83%, compared with 0.67 set recall, consistent with unequal score contributions among tokens.

Absolute-cosine and squared-cosine means exceed isotropic references by approximately three and nine times overall, and 3.6 and twelve times among top-32 tokens, while signed cosines are close to zero. Effect directions are therefore more aligned up to sign than isotropic vectors, સ

Table 32. Single-probe ranking on 100 Set 2 frames. The first four rows report mean±\pmSD (sample SD) across ten probe seeds (0–9), each summarized by its frame mean. Recall and retained energy compare the probe and exact masks, both formed from the top 16 tokens per camera (32 in total); retained energy is the ratio of the exact ∑isi2\sum_i s_i^2 over the two masks. Energy@32 is the fraction of a frame’s total exact ∑isi2\sum_i s_i^2 over all 512 tokens captured by the exact top-32 tokens, selected globally or as the top 16 per camera.

MetricValue
Spearman(s^P=1\hat{s}^{P=1}, ss), All tokens0.835 ±\pm 0.006
Spearman(s^P=1\hat{s}^{P=1}, ss), Within-camera mean0.797 ±\pm 0.007
Top-32 recall0.667 ±\pm 0.014
Retained energy (vs. exact)0.834 ±\pm 0.012
Energy@32, Global top-32 (median, IQR)0.858 [0.827, 0.898]
Energy@32, Top-16 per camera (median, IQR)0.836 [0.809, 0.875]

Table 32.

Table 33. Alignment of action-effect directions (Set 2). Pairwise cosine statistics are averaged within each frame and then across frames. The isotropic reference uses independent uniform unit vectors in R56\mathbb{R}^{56}.

without sharing a single common axis; absolute projections depend only on this unsigned alignment. Neither the alignment nor the energy concentration implies large norm gaps at the selection boundary or recovery of the whole set, and we do not separate their contributions here.

What preserves single-probe rankings? By Eq. (21), the pairwise reversal probability decreases with both the sensitivity ratio and the alignment of action effects. At r=3r = 3, substituting the measured mean ∣cos⁡∣\lvert\cos\rvert (0.32–0.39) for ∣ρ∣\lvert\rho\rvert lowers this probability by only about 4–6% relative to orthogonal effects. In practice, rankings are preserved mainly because sensitivity energy is concentrated in few tokens (about 84% in the top 16 per camera). This view predicts weaker discrimination when a larger budget moves the selection boundary into regions where energy is spread more evenly. The small differences between methods at B=128B = 128 are consistent with this, although we do not test the link directly.

Mask quality. Set 1 compares action deviations against random masks with the same budget and camera quotas. Median range-normalized deviations, before random-reference ratios, are 0.336 for exact top-kk, 0.360 for probe top-kk, 0.363 for POKE (top-kk), 0.973 for DivPrune, and 1.314 for random. These medians need not match the medians of frame-wise ratios. DivPrune lies between random and top-kk in this offline measure, as it does in closed loop,

Table 34. Action deviations relative to random masks with the same B=32B = 32 and camera quotas (Set 1, deletek\text{delete}_{k}; lower is better). Dev./random: on each frame, every mask’s deviation is divided by the mean deviation of the n=32n = 32 random masks on that frame, and the table reports the median of this ratio over the 120 frames; all masks from a frame share the denominator, so within-frame orderings and the paired tests in Table 35 do not depend on it. Below random: the fraction of the 120 ×\times 32 frame–random-mask pairs in which the mask’s deviation is strictly lower (ties count against the mask). The Random row is the reference by definition (1.000 and 0.500), not a measured value.

A − BMedian differenceA win fractionpp
POKE (top-kk) − Probe(P=1P=1) top-kk−0.0110.5330.44
POKE (top-kk) − Probe(P=3P=3) top-kk+0.0100.4580.13
POKE (top-kk) − Exact top-kk+0.0130.3589.8×10−49.8 \times10^{-4}
Probe(P=1P=1) top-kk − Exact top-kk+0.0210.3922.2×10−32.2 \times10^{-3}
POKE (MMDP) − Probe(P=1P=1) MMDP−0.0560.6928.1×10−68.1 \times10^{-6}

Table 35. Frame-paired Wilcoxon signed-rank tests (Set 1, n=120n = 120, delete_k). Negative differences favor A. These exploratory tests do not account for within-episode dependence and are not used as confirmatory evidence of significance or equivalence.

which supports the proxy for these rules without validating it for every selection rule. Deviations under POKE (top-kk) and probe top-kk are of similar magnitude, but p=0.44p = 0.44 does not demonstrate equivalence. Modest student–teacher Recall@32 (0.597) can coexist with relatively low action deviation when scores are unevenly concentrated. Training across observations may also reduce probe-specific noise; we did not test this directly.

Comparisons on shared observations. Table 36 summarizes the Set 3 comparison reported in the main paper; task-wise results are below. Both POKE variants have lower separate task medians than both attention baselines on all ten tasks. A frame-wise Wilcoxon comparison of FastV and VP gives p=p = 4.1×10−54.1 \times10^{-5}, whereas the task-level sign test gives p=0.109p = 0.109 (eight tasks favor VP). Treating correlated frames as independent overstates the evidence, so the main text uses task-level tests. The top-kk/MMDP task-level comparison (p=0.754p = 0.754) does not separate the two variants, which is not

TaskFastVVLA-PrunerPOKE (top-kk)POKE (MMDP)
00.3600.3230.2800.246
10.2930.2650.2160.240
20.4320.3140.3110.265
30.2960.3120.2430.222
40.2790.2910.2080.200
50.4830.3890.2570.249
60.3440.3430.2250.229
70.4120.3870.3760.285
80.4450.4100.2680.247
90.3070.3040.2220.216
All (frame median)0.3710.3320.2520.244
Comparison (a − b)TasksSign-test pp
FastV − VLA-Pruner2/80.109
FastV − POKE (top-kk)0/100.002
FastV − POKE (MMDP)0/100.002
VLA-Pruner − POKE (top-kk)0/100.002
VLA-Pruner − POKE (MMDP)0/100.002
POKE (top-kk) − POKE (MMDP)4/60.754

Table 37. Task-level median normalized deviations on shared Long observations (Set 3, B=32B = 32; lower is better). The lower block reports two-sided task-level sign tests, with counts of tasks favoring a/b based on the median paired frame-wise difference. This statistic need not equal the difference of the separate task medians in the upper block.

the same as showing they are equivalent.

Where offline fidelity and closed-loop success diverge.

On Goal demonstration frames (Set 1), exact top-kk has lower one-step deviation than exact MMDP (median paired difference −0.115-0.115, win fraction 80%, exploratory frame-level p=9.8×10−13p = 9.8 \times10^{-13}). On Long reference rollouts (Set 3), the reported task-level test does not separate the two POKE variants. Yet Long closed-loop success favors POKE (MMDP) for all three training seeds. Multi-step success also depends on recovery after deviations, which one-step fidelity does not capture, so the closed-loop gain of MMDP cannot be explained by one-step fidelity alone. We therefore use the offline proxy as a diagnostic next to rollouts, not in place of them.

References

  1. [1]Saeed Ranjbar Alvar, Gursimran Singh, Mohammad Akbari, and Yong Zhang. Diviprune: Diversity-based visual token pruning for large multimodal models. In 2025 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 9392–9401. IEEE, 2025. 5, 2arxiv.org/abs/2503.02175
  2. [2]Kevin Black, Noah Brown, Danny Driess, Adnan Esmail, Michael Equi, Chelsea Finn, Niccolo Fusai, Lachy Groom, Karol Hausman, Brian Ichter, et al. π₀: A vision-language-action flow model for general robot control. arXiv preprint arXiv:2410.24164, 2024. 11arxiv.org/abs/2410.24164
  3. [3]Liang Chen, Haozhe Zhao, Tianyu Liu, Shuai Bai, Junyang Lin, Chang Zhou, and Baobao Chang. An image is worth 1/2 tokens after layer 2: Plug-and-play inference acceleration for large vision-language models. In European Conference on Computer Vision, pages 19–35. Springer, 2024. 1, 2, 5
  4. [4]Senyu Fei, Siyin Wang, Junhao Shi, Zihao Dai, Jikun Cai, Pengfang Qian, Li Ji, Xin zhe He, Shiduo Zhang, Zhaoye Fei, Jinlan Fu, Jingjing Gong, and Xipeng Qiu. Libero-plus: In-depth robustness analysis of vision-language-action models. arXiv preprint arXiv:2510.13626, 2025. 6, 11arxiv.org/abs/2510.13626
  5. [5]Physical Intelligence, Kevin Black, Noah Brown, James Darpinian, Karan Dhabalia, Danny Driess, Adnan Esmail, Michael Equi, Chelsea Finn, Niccolo Fusai, et al. π₀.₅: a vision-language-action model with open-world generalization. arXiv preprint arXiv:2504.16054, 2025. 2, 5arxiv.org/abs/2504.16054
  6. [6]Titong Jiang, Xuefeng Jiang, Yuan Ma, Xin Wen, Bailin Li, Kun Zhan, Peng Jia, Yahui Liu, Sheng Sun, and Xianpeng Lang. The better you learn, the smarter you prune: Towards efficient vision-language-action models via differentiable token pruning. arXiv preprint arXiv:2509.12594, 2025. 3arxiv.org/abs/2509.12594
  7. [7]Moo Jin Kim, Karl Pertsch, Siddharth Karamcheti, Ted Xiao, Ashwin Balakrishna, Suraj Nair, Rafael Rafailov, Ethan Foster, Grace Lam, Pannag Sanketi, et al. Openvla: An open-source vision-language-action model. arXiv preprint arXiv:2406.09246, 2024. 10arxiv.org/abs/2406.09246
  8. [8]Moo Jin Kim, Chelsea Finn, and Percy Liang. Fine-tuning vision-language-action models: Optimizing speed and success. arXiv preprint arXiv:2502.19645, 2025. 2, 5arxiv.org/abs/2502.19645
  9. [9]Juny an Li, Li Lyna Zhang, Jiah ang Xu, Yujing Wang, Shaoguang Yan, Yunqing Xia, Yuqing Yang, Ting Cao, Hao Sun, Weiwei Deng, et al. Constraint-aware and ranking-distilled token pruning for efficient transformer inference. In Proceedings of the 29th ACM SIGKDD Conference on knowledge discovery and data mining, pages 1280–1290, 2023. 2DOI
  10. [10]Xuanlin Li, Kyle Hsu, Jiayuan Gu, Karl Pertsch, Oier Mees, Homer Rich Walke, Chuyuan Fu, Ishikaa Lunawat, Isabel Sieh, Sean Kirmani, Sergey Levine, Jiajian Wu, Chelsea Finn, Hao Su, Quan Vuong, and Ted Xiao. Evaluating real-world robot manipulation policies in simulation. arXiv preprint arXiv:2405.05941, 2024. 5, 11arxiv.org/abs/2405.05941
  11. [11]Ye Li, Yuan Meng, Zewen Sun, Kangye Ji, Chen Tang, Jia-jun Fan, Xinzhu Ma, Shu-Tao Xia, Zhi Wang, and Wenwu Zhu. Sp-vla: A joint model scheduling and token pruning approach for vla model acceleration. In The Fourteenth International Conference on Learning Representations, 2025. 3arxiv.org/abs/2506.12723
  12. [12]Youwei Liang, Chongjian Ge, Zhan Tong, Yibing Song, Jue Wang, and Pengtao Xie. Not all patches are what you need: Expediting vision transformers via token reorganizations. arXiv preprint arXiv:2202.07800, 2022. 2arxiv.org/abs/2202.07800
  13. [13]Bo Liu, Yifeng Zhu, Changkai Gao, Yihao Feng, Qiang Liu, Yuke Zhu, and Peter Stone. Libero: Benchmarking knowledge transfer for lifelong robot learning. Advances in Neural Information Processing Systems, 36:44776–44791, 2023. 2arxiv.org/abs/2306.03310
  14. [14]Haotian Liu, Chunyuan Li, Yuheng Li, and Yong Jae Lee. Improved baselines with visual instruction tuning. In 2024 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 26286–26296. IEEE, 2024. 2arxiv.org/abs/2310.03744
  15. [15]Ziyan Liu, Yeqiu Chen, Hongyi Cai, Tao Lin, Shuo Yang, Zheng Liu, and Bo Zhao. Bridging the semantic-action gap in visual token pruning for efficient vla inference. arXiv preprint arXiv:2511.16449, 2025. 1, 3, 5arxiv.org/abs/2511.16449
  16. [16]Pavlo Molchanov, Stephen Tyree, Tero Karras, Timo Aila, and Jan Kautz. Pruning convolutional neural networks for resource efficient inference. In International Conference on Learning Representations, 2017. 2arxiv.org/abs/1611.06440
  17. [17]Xiaohuan Pei, Yuxing Chen, Siyu Xu, Yunke Wang, Yuheng Shi, and Chang Xu. Action-aware dynamic pruning for efficient vision-language-action manipulation. In International Conference on Learning Representations, pages 10832–10851, 2026. 3
  18. [18]Hanchuan Peng, Fuhui Long, and Chris Ding. Feature selection based on mutual information criteria of max-dependency, max-relevance, and min-redundancy. IEEE Transactions on pattern analysis and machine intelligence, 27(8):1226–1238, 2005. 5pubmed.ncbi.nlm.nih.gov/16119262
  19. [19]Yongming Rao, Wenliang Zhao, Benlin Liu, Jiwen Lu, Jie Zhou, and Cho-Jui Hsieh. Dynamicvit: Efficient vision transformers with dynamic token sparsification. Advances in neural information processing systems, 34:13937–13949, 2021. 2arxiv.org/abs/2106.02034
  20. [20]Allen Z. Ren. open-pi-zero: Re-implementation of pio vision-language-action (vla) model from physical intelligence. https://github.com/allenzren/open-pi-zero, 2024. GitHub repository. 11
  21. [21]Yuyao Sun, Tao Deng, Shuang Li, Deqing Wang, Hao Geng, and Minjun Yu. Learning to predict middle-layer attention in mllms for visual token pruning. arXiv preprint arXiv:2608.06411, 2026. 2, 5arxiv.org/abs/2608.06411
  22. [22]Yao Tao, Yehui Tang, Yun Wang, Mingjian Zhu, Hailin Hu, and Yunhe Wang. Selectively-driven dynamic token pruning for large language models. arXiv preprint arXiv:2504.04514, 2025. 2, 4arxiv.org/abs/2504.04514
  23. [23]Homer Rich Walke, Kevin Black, Tony Z Zhao, Quan Vuong, Chongyi Zheng, Philippe Hansen-Estruch, Andre Wang He, Vivek Myers, Moo Jin Kim, Max Du, et al. Bridgedata v2: A dataset for robot learning at scale. In Conference on robot learning, pages 1723–1736. PMLR, 2023. 11arxiv.org/abs/2308.12952
  24. [24]Hanzhen Wang, Jiaming Xu, Yushun Xiang, Jiayi Pan, Yongkang Zhou, Yong-Lu Li, and Guohao Dai. Specprune-vla: Accelerating vision-language-action models via action-aware self-speculative pruning. arXiv preprint arXiv:2509.05614, 2025. 1, 3, 5arxiv.org/abs/2509.05614
  25. [25]Zichen Wen, Yifeng Gao, Shaobo Wang, Junyuan Zhang, Qintong Zhang, Weijia Li, Conghui He, and Linfeng Zhang. Stop locking for “important tokens” in multimodal language models: Duplication matters more. In Proceedings of the 2025 Conference on Empirical Methods in Natural Language Processing, pages 9972–9991, 2025. 2arxiv.org/abs/2502.11494
  26. [26]Siyu Xu, Yunke Wang, Chenghao Xia, Dihao Zhu, Tao Huang, and Chang Xu. Vla-cache: Efficient vision-language-action manipulation via adaptive token caching. Advances in Neural Information Processing Systems, 38:164448–164473, 2026. 1, 3, 5arxiv.org/abs/2502.02175
  27. [27]Senqiao Yang, Yukang Chen, Zhuotao Tian, Chengyao Wang, Jingyao Li, Bei Yu, and Jiaya Jia. Visionzip: Longer is better but not necessary in vision language models. In 2025 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 1972–19802. IEEE, 2025. 2arxiv.org/abs/2412.04467
  28. [28]Yantai Yang, Yuhao Wang, Zichen Wen, Luo Zhongwei, Chang Zou, Zhipeng Zhang, Chuan Wen, and Linfeng Zhang. Efficientvla: Training-free acceleration and compression for vision-language-action models. Advances in Neural Information Processing Systems, 38:40891–40914, 2026. 3arxiv.org/abs/2506.10100
  29. [29]Qizhe Zhang, Aosong Cheng, Ming Lu, Renrui Zhang, Zhiyong Zhuo, Jiajun Cao, Shaobo Guo, Qi She, and Shanghang Zhang. Beyond text-visual attention: Exploiting visual cues for effective token pruning in vlms. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 20572–20867, 2025. 2arxiv.org/abs/2412.01818
  30. [30]Qizhe Zhang, Mengzhen Liu, Lichen Li, Ming Lu, Yuan Zhang, Junwen Pan, Qi She, and Shanghang Zhang. Beyond attention or similarity: Maximizing conditional diversity for token pruning in mllms. Advances in Neural Information Processing Systems, 38:25438–25468, 2026. 2arxiv.org/abs/2506.10967
  31. [31]Xin Zou, Di Lu, Yizhou Wang, Yibo Yan, Yuanhuiyi Lyu, Xu Zheng, Linfeng Zhang, and Xuming Hu. Don’t just chase “highlighted tokens” in mllms: Revisiting visual holistic context retention. Advances in Neural Information Processing Systems, 38:39800–39832, 2026. 2arxiv.org/abs/2510.02912

Paper details

vlnc:W_6fmfjdup
Version
v1 · latest
Published
Fresh — added to the Hub within the last 30 days
Added
License
CC BY 4.0
Usage

views · 0

downloads · 0

A view counts once per browser per day, a download each time the PDF is requested.

  • Computer Science

Claude, ChatGPT

View statement
Looking for related papers…
Contents