betaFeedback

WorksComputer Science

ACiT: Proprioceptive-Prior Action Chunking with Inverted Transformers

Abstract

Visuomotor policies condition action prediction on visual and proprioceptive observations. While historical proprioceptive sequences carry rich dynamic priors, current methods treat them as equal to high-dimensional vision, causing the proprioceptive signal to be overwhelmed. To identify what is lost in equal-conditioning fusion, we analyze proprioceptive data and find that strong inter-joint coupling is the key action-determining structure. This finding aligns with the neural sensorimotor loop, in which proprioception first forms a body-centered prior and vision is later calibrated via parietal remapping. Inspired by this, we propose ACiT, Proprioceptive-Prior Action Chunking with Inverted Transformers, which reinforces historical joint-coupling as the primary condition with vision as a later calibration. This design is instantiated through three modules: Inverted State Encoding builds a proprioceptive prior via per-joint tokens and inter-joint self-attention. State-Visual Grounding lets joint tokens query visual evidence, making vision goal- and calibration-only. Inverted Action Chunking decodes per-joint trajectories independently. ACiT improves success rates on most LIBERO, RoboTwin 2.0, and real-world tasks, with the largest gains on those that require dynamics and coordination. The same conditioning design transferred to the VLA backbone pi_0-ACiT yields significant gains with better execution efficiency.

Introduction

Visuomotor policies predict actions from multi-view observations and proprioception. Early work conditioned on a single current state, where ACT maps the current joint state and images to an action chunk [41]. Diffusion Policy conditions on a short observation history instead [8]. A single frame proved insufficient to resolve action ambiguity, while history carries dynamics and hidden-state information, so proprioceptive history was explicitly incorporated. Recent models such as A2A and Chronos explicitly model proprioceptive history [12, 42]. Another line discretizes historical states or actions into tokens to better integrate with LLM backbones, at the cost of losing continuous coupling structure [14, 22, 28]. Reconstruction-only tokenization can also erode control-relevant information [16]. These works validate the value of using history from both continuous and discrete directions, and state history is becoming a consensus for action prediction.

These advances have not converged at the conditioning interface. First, mainstream methods map historical proprioceptive states as an independent modality, either discretely or continuously, and place them alongside a pooled global visual vector as conditions for action generation. This naive fusion lets high-dimensional, low-SNR visual features entangle and dominate the conditioning, leading to degraded robustness or spurious shortcuts [15, 38]. Second, we analyze the state history and the action itself: the effective dimensionality of the action distribution is far below the number of joints, and joints are strongly coupled, as shown in Figure 1(a). However, existing methods model temporal dependencies and pack all joints into one vector, so the inter-joint coupling, a structure critical to action, can only be implicit inside the token.

Figure 1. Motivation. (a) Embodiment: a dual-arm robot has 14 joints, each with its own state history, while the action space has a much lower effective dimensionality (PR 1.44–4.42) and strong inter-joint coupling, the state therefore should not be compressed into one shared vector. (b) Biological inspiration: proprioception forms a body-centered prior in S1, vision reaches PPC via V1 only to remap and calibrate the goal, and M1 issues the command. The two modalities are not symmetric, and their order is fixed. (c) Conditioning interface: ACT concatenates a single-frame state with vision into one condition shared by all joints; ACiT keeps the per-joint prior as the primary condition, retrieves visual evidence per joint, and decodes one trajectory per joint.

Therefore, how to encode historical state information, how to fuse modalities, and how to feed the result into action decoding is an open problem. Inspired by neuroscience, the human can chain from peripheral proprioception to body-centered action generation as is follows (Figure 1(b)): muscle and joint receptors continuously encode joint angle, velocity, and tension and project through large-diameter myelinated afferents [29, 36]. The somatosensory cortex and posterior parietal cortex aggregate proprioceptive history into a body-state representation, forming a body-centered prior [1, 2]. The posterior parietal cortex then remaps visual goals from retinal coordinates into body-centered coordinates, with proprioceptive context modulating this remapping [2, 10]. Finally, the parietal-motor pathway sends the calibrated body-centered representation to action generation [23, 37]. The key computational asymmetry is that proprioception provides a continuous, low-latency body-state stream, whereas visual goals are remapped and integrated into the body-centered representation afterward.

We map this sensorimotor loop to Bayesian cue integration: the fast, continuous proprioceptive stream forms a prior over the body state, visual goals provide goal evidence (a likelihood-like term), and calibrating the prior yields a posterior-like body-centered representation for action generation. Based on this, we propose ACiT on top of ACT, the smallest architecture that still carries the full interface (multi-view images, proprioception, and a chunked action decoder), so that the encoder and the head stay fixed while only the conditioning changes. It has three modules. First, Inverted State Encoding compresses each joint’s entire history into a token and directly models inter-joint coupling with self-attention. Second, State-Visual Grounding treats vision as a per-joint goal-calibration signal and does not let it directly enter the encoder. Third, Inverted Action Chunking treats each joint as a token on the action side, and each joint independently generates its complete trajectory chunk. The decoder does not need to translate between “visual coordinates” and “joint coordinates”. The historical proprioceptive sequence and action generation share a more consistent feature space. To separate the interface from the backbone, we port the same conditioning to a pretrained VLA and obtain π0\pi_0-ACiT, which keeps π0\pi_0’s architecture and changes only how the state enters it. Experiments on LIBERO, RoboTwin 2.0, and real-world manipulation tasks show that ACiT and π0\pi_0-ACiT improve success on most tasks, with improved robustness under visual perturbations, especially on tasks that depend on dynamics and joint coordination.

Our contributions are as follows:

  • We formulate the conditioning interface as Bayesian sensorimotor cue integration, with history as the prior and vision as the calibration evidence, and realize it as an explicit asymmetric fusion.

  • We propose ACiT, which uses three inverted Transformer modules to explicitly build and preserve the inter-joint coupling dynamics in the state history instead of flattening them.

  • We validate ACiT on LIBERO, RoboTwin 2.0, and real-world tasks, where it improves the average success rate by 5.7 to 10.8 points over the strongest baseline. π0\pi_0-ACiT improves the average success rate by 26.9 points at 20.2% lower inference latency than π0\pi_0.

Related Work

History in the conditioning interface. Early visuomotor policies condition on a single frame: ACT maps the current joint state and images to an action chunk [41], and Diffusion Policy instead uses a short observation window [8]. Later work makes the state history explicit in the conditioning input [12, 42]. A parallel line discretizes states or actions so that autoregressive backbones can consume them [14, 21, 22, 28], at the cost of losing continuous structure, and reconstruction-only tokenization can erode control-relevant information [16]. Under both interfaces proprioception enters as one flat vector or a one-dimensional token sequence, so the coupling among joints stays implicit.

Vision and proprioception under fusion. Pooling both modalities into one condition leaves nothing to stop the high-dimensional visual stream from dominating and introducing spurious shortcuts [15, 38]. One line responds by letting the robot state steer vision, through robot-centric pooling [43] or active visual attention [39], yet the asymmetry is applied to perception rather than to the conditioning itself. Large VLA backbones inherit the same symmetric interface, concatenating one proprioceptive token with visual and language tokens [4, 5, 25].

Structure as an inductive bias. Topology-aware models inject the kinematic graph as a structural bias for cross-embodiment transfer [24, 27]. ACiT shares that motivation but acts on the conditioning interface: it carries history per joint, keeps the resulting prior primary, and lets each joint retrieve its own visual calibration in a single generation pipeline. Appendix § 10 expands the comparison.

Method

Preliminaries

Let the robot have DD joints, with state st∈RD\mathbf{s}_t \in\mathbb{R}^{D} and action at∈RD\mathbf{a}_t \in\mathbb{R}^{D} at time tt. An action-chunking policy predicts

At=[at,…,at+T−1]∈RT×D,(1)\mathbf{A}_t = [\mathbf{a}_t,\ldots,\mathbf{a}_{t+T-1}] \in\mathbb{R}^{T \times D}, \tag*{(1)}

and executes its first few steps. The state history is St=[st−H+1,…,st]∈RH×D\mathbf{S}_t = [\mathbf{s}_{t-H+1},\ldots,\mathbf{s}_t] \in\mathbb{R}^{H \times D}, where HH is the history length (number of past frames). V∈RN×dv\mathbf{V} \in\mathbb{R}^{N \times d_v} denotes the visual token matrix, whose rows are the NN visual tokens.

Most chunk-based policies share one skeleton [8, 41]: a Transformer encoder [34] maps the observations Ot\mathbf{O}_t to a condition, and a decoder emits the chunk, p(At∣Ot)=Dec⁡θ(Enc⁡ϕ(Ot))p(\mathbf{A}_t \mid\mathbf{O}_t) = \operatorname{Dec}_{\theta}(\operatorname{Enc}_{\phi}(\mathbf{O}_t)). What differs across methods is how the proprioceptive history is represented inside that condition, which is the axis ACiT changes. The encoder and the decoder are those of the host policy (Sec. D.1). The equal-conditioning baseline we depart from places the current state and vision side by side as conditions:

πθ(At∣st,V).(2)\pi_{\theta}(\mathbf{A}_t \mid\mathbf{s}_t,\mathbf{V}). \tag*{(2)}

Bayesian Sensorimotor Formulation. Let bt\mathbf{b}_t be the latent body-centered state and gg the task goal (vision, or vision and language in a VLA). The ideal visuomotor computation factorizes into two stages,

p(bt∣St,V)∝p(bt∣St) p(V∣bt,g),(3)p(\mathbf{b}_t \mid\mathbf{S}_t,\mathbf{V}) \propto p(\mathbf{b}_t \mid\mathbf{S}_t)\,p(\mathbf{V} \mid\mathbf{b}_t,g), \tag*{(3)}
p(At∣St,V)=∫p(At∣bt) p(bt∣St,V) dbt.(4)p(\mathbf{A}_t \mid\mathbf{S}_t,\mathbf{V}) = \int p(\mathbf{A}_t \mid\mathbf{b}_t)\,p(\mathbf{b}_t \mid\mathbf{S}_t,\mathbf{V})\,d\mathbf{b}_t. \tag*{(4)}

Equation (3) states that visual evidence calibrates a proprioceptive prior into a posterior body-state estimate, and Equation (4) marginalizes over this posterior to produce the action distribution. This formulation mirrors Bayesian cue integration in sensorimotor control [13, 33].

ACiT Instantiation. Rather than modeling explicit probability distributions, ACiT uses deterministic representations to instantiate this computation:

πθ(At∣Gt),Gt=fθ(St,V),(5)\pi_{\theta}(\mathbf{A}_t \mid\mathbf{G}_t), \qquad\mathbf{G}_t = f_{\theta}(\mathbf{S}_t,\mathbf{V}), \tag*{(5)}

where Gt\mathbf{G}_t is a deterministic representation of the posterior-like body state in Eq. (3), and the action head πθ(At∣Gt)\pi_{\theta}(\mathbf{A}_t \mid\mathbf{G}_t) replaces the integration in Eq. (4). The function fθf_{\theta} is factorized into per-joint history encoding with a body-centered prior (§ 3.2) followed by state-visual grounding (§ 3.3). The action head consumes Gt\mathbf{G}_t in § 3.4. This is a computational analogy, not a literal probabilistic model. bt\mathbf{b}_t is distinct from the CVAE latent zz used for training.

Inverted State Encoding

For each joint jj, ACiT applies a depthwise 1D convolution along time with kernel size κ\kappa, followed by a shared temporal projection:

ui(j)=∑τ=−⌊κ/2⌋⌊κ/2⌋kτ(j)sˉi+τ(j),i=1,…,H,(6)u_i^{(j)} = \sum_{\tau=-\lfloor\kappa/2 \rfloor}^{\lfloor\kappa/2 \rfloor} k_{\tau}^{(j)}\bar{s}_{i+\tau}^{(j)}, \qquad i=1,\ldots,H, \tag*{(6)}

where κ\kappa is the kernel size, sˉi(j)=st−H+i(j)\bar{s}_i^{(j)} = s_{t-H+i}^{(j)} is the ii-th frame in the history window, kτ(j)k_{\tau}^{(j)} is the joint-specific convolution weight at offset τ\tau, and zero padding is applied at both ends. We use κ=3\kappa=3 by default, giving weights k(j)=[k−1(j),k0(j),k1(j)]k^{(j)}=[k_{-1}^{(j)},k_0^{(j)},k_1^{(j)}]. The convolved features are stacked into u(j)=[u1(j),…,uH(j)]T∈RH\mathbf{u}^{(j)}=[u_1^{(j)},\ldots,u_H^{(j)}]^{\mathsf{T}} \in\mathbb{R}^{H}.

A shared linear layer and a joint-identity embedding then project the temporal feature to a dd-dimensional token:

h(j)=Wtimeu(j)+e(j)∈Rd,(7)\mathbf{h}^{(j)} = W_{\mathrm{time}}\mathbf{u}^{(j)}+\mathbf{e}^{(j)} \in\mathbb{R}^{d}, \tag*{(7)}

where Wtime∈Rd×HW_{\mathrm{time}} \in\mathbb{R}^{d \times H} is shared across joints and e(j)∈Rd\mathbf{e}^{(j)} \in\mathbb{R}^{d} is a learnable joint-identity embedding. Stacking all joint tokens gives Xt=[h(1);…;h(D)]∈RD×d\mathbf{X}_t=[\mathbf{h}^{(1)};\ldots;\mathbf{h}^{(D)}] \in\mathbb{R}^{D \times d}. The depthwise convolution captures local temporal patterns such as velocity and acceleration without mixing joints. The history is in effect a multivariate time series with one channel per joint, and the coupling we care about is a dependency between channels rather than between time steps. Treating each channel as a token and letting attention operate across channels is the inverted formulation that has been shown to capture variate-wise structure in forecasting [19]. Here the variates are joints, and their dependency is the coupling that determines the action.

Proprioceptive prior. The joint tokens then attend to each other, so that the coupling structure is represented explicitly rather than hidden inside a flattened vector:

Q=XtWQ,K=XtWK,Zx=XtWV,(8)\mathbf{Q}=\mathbf{X}_tW_Q,\quad\mathbf{K}=\mathbf{X}_tW_K,\quad\mathbf{Z}_x=\mathbf{X}_tW_V, \tag*{(8)}
Pt=softmax⁡(QKTdk)Zx∈RD×d,(9)\mathbf{P}_t=\operatorname{softmax}\left(\frac{\mathbf{Q}\mathbf{K}^{\mathsf{T}}}{\sqrt{d_k}}\right)\mathbf{Z}_x \in\mathbb{R}^{D \times d}, \tag*{(9)}

where Q,K,Zx\mathbf{Q},\mathbf{K},\mathbf{Z}_x are the queries, keys and values, dkd_k is the key dimension, and Pt=[h~(1),…,h~(D)]\mathbf{P}_t=[\tilde{\mathbf{h}}^{(1)},\ldots,\tilde{\mathbf{h}}^{(D)}] is a deterministic representation of the prior p(bt∣St)p(\mathbf{b}_t \mid\mathbf{S}_t) in Eq. (3), i.e., the body-centered prior from which the action is eventually generated.

State-Visual Grounding

Given the prior Pt\mathbf{P}_t, vision now enters only as evidence. Each prior token serves as a query to the visual tokens:

Qc=PtWQ′,Kc=VWK′,Zc=VWV′,(10)\mathbf{Q}_c=\mathbf{P}_tW'_Q,\quad\mathbf{K}_c=\mathbf{V}W'_K,\quad\mathbf{Z}_c=\mathbf{V}W'_V, \tag*{(10)}
Gt=softmax⁡(QcKcTdk)Zc∈RD×d.(11)\mathbf{G}_t=\operatorname{softmax}\left(\frac{\mathbf{Q}_c\mathbf{K}_c^{\mathsf{T}}}{\sqrt{d_k}}\right)\mathbf{Z}_c \in\mathbb{R}^{D \times d}. \tag*{(11)}

Figure 2. ACiT architecture, organized by its three modules. (a) Inverted State Encoding: per-joint depthwise 1D convolutions turn each joint’s history into one token, and self-attention among the tokens gives the proprioceptive prior PtP_t. (b) State-Visual Grounding: the joint tokens query the visual tokens of the multi-view observations and produce grounded tokens GtG_t. (c) Inverted Action Chunking: GtG_t is the memory of a transformer decoder whose learnable joint queries output one TT-step trajectory per joint.

Equation (11) is the concrete instantiation of the mapping fθf_\theta in Eq. (5), where Gt=[c(1);…;c(D)]G_t = [c^{(1)};\ldots;c^{(D)}] is the grounded condition and a deterministic representation of the posterior-like body state in Eq. (3). All blocks use pre-LN residual connections followed by a feed-forward network. Visual evidence is thus retrieved into a body-centered state rather than fused symmetrically with proprioception. This is consistent with visual targets being remapped into body-centered reference frames [2, 10] and with vision modulating, rather than re-placing, the body schema [1].

Inverted Action Chunking

The action head is interchangeable. With the ACT transformer decoder, DD learned joint queries Qa∈RD×dQ_a \in\mathbb{R}^{D \times d} attend to the memory M=[Gt;m]M = [G_t;m]. As in the self- and cross-attention blocks above, each head computes

O=softmax⁡(QaKm⊤dk)Zm∈RD×d,(12)O = \operatorname{softmax}\left(\frac{Q_a K_m^\top}{\sqrt{d_k}}\right) Z_m \in\mathbb{R}^{D \times d}, \tag*{(12)}

where Km=MWakK_m = M W_a^k, Zm=MWavZ_m = M W_a^v, and m=Wmzm = W_m z is a learned token embedding of the CVAE latent zz defined below. The CVAE latent is used only here. State-visual grounding is independent of zz. With a conditional 1D U-Net, the denoiser is conditioned on GtG_t and mm and produces the same per-joint output layout. In both cases, a shared linear projection maps every joint output to a full trajectory:

a^(j)=Woo(j),Wo∈RT×d,(13)\hat{a}^{(j)} = W_o o^{(j)}, \qquad W_o \in\mathbb{R}^{T \times d}, \tag*{(13)}

and concatenation gives A^t=[a^(1),…,a^(D)]∈RT×D\hat{A}_t = [\hat{a}^{(1)},\ldots,\hat{a}^{(D)}] \in\mathbb{R}^{T \times D}. The action is thus modeled as DD joint trajectories rather than TT time-step vectors.

Training Objective

Following ACT [41], we use a CVAE [31] latent z∼qϕ(z∣St,At)=N(μϕ,diag⁡(σϕ2))z \sim q_\phi(z \mid S_t,A_t) = \mathcal{N}(\mu_\phi,\operatorname{diag}(\sigma_\phi^2)), where z=μϕ+σϕ⊙ϵz = \mu_\phi+ \sigma_\phi\odot\epsilon with ϵ∼N(0,I)\epsilon\sim\mathcal{N}(0,I), and z=0z = 0 at inference. The objective is the standard CVAE loss [31]:

L=ED[1TD∥A^t−At∥1+λ KL⁡(qϕ∥p)],(14)\mathcal{L} = \mathbb{E}_{\mathcal{D}}\left[\frac{1}{TD}\lVert\hat{A}_t-A_t\rVert_1 + \lambda\,\operatorname{KL}(q_\phi\Vert p)\right], \tag*{(14)}

where ED\mathbb{E}_{\mathcal{D}} is the expectation over demonstrations, qϕq_\phi abbreviates qϕ(z∣St,At)q_\phi(z \mid S_t,A_t), pp abbreviates p(z)=N(0,I)p(z)=\mathcal{N}(0,I), KL denotes the Kullback-Leibler divergence, and λ\lambda sets the weight of this term.

Experiments

Experimental Setup

Benchmarks and Tasks. We evaluate on two simulation benchmarks and a real-world dual-arm platform. RoboTwin 2.0 [7] is a dual-arm manipulation benchmark with 14-DoF joint states and actions and three cameras (a top camera and two wrist cameras). We use six tasks: Lift Pot, Pick Dual Bottles, Handover Block, Move Can Pot, Place Dual Shoes and Shake Bottle, abbreviated as Lift, Pick Dual, Handover, Move Can, Shoes and Shake. Each task has an easy and a randomized hard split, the latter varying object pose, color and background (Appendix § 7.§ 1). LIBERO [18] is a single-arm benchmark with 7-DoF actions, 8-DoF states, and two cameras. We evaluate on Spatial, Object, and LIBERO-10. We exclude Goal, which requires language instructions.

Real-World Platform. We build a dual-arm platform from two reBot Arm B601-DM arms [30]. Each arm has six DoF, a

1.5 kg payload, and ±0.2\pm0.2 mm repeatability. A global camera and two wrist cameras provide observations. We evaluate four tasks: pick cube and stack cube with one arm, and lift pot and handover cube with two arms. Hyperparameters and per-task protocols are in Appendix § 7.§ 2.

Baselines and Comparisons. For small action policies we compare three baselines. ACT [41] uses a CVAE and a transformer decoder on the ACT backbone and conditions on the current frame only. Diffusion Policy [8] and the flow-matching baseline [17] share the same observation encoder and the same 1-D U-Net, and both condition on a proprioceptive history. They differ only in the generative objective (DDPM [11] versus flow matching [17]) and in the sampler. The task-specific policies are trained from scratch with LeRobot [6] in PyTorch [26], on a shared ResNet18 [9] backbone and with AdamW [20]. The VLA keeps its PaliGemma backbone. Simulation runs in MuJoCo [32] through RoboTwin [7].

Main Results on Simulation Benchmarks

Figure 5 compares ACiT with ACT, DP, and FM on LIBERO and RoboTwin 2.0. On LIBERO the gain appears on Spatial, where success depends on precise placement and therefore needs a different visual reference for each joint. On Object the strong baselines are near saturation and ACiT matches them, while on LIBERO-10 the four methods are

Figure 5. Main results on the simulation benchmarks. Left: LIBERO. Right: RoboTwin 2.0. Pale bars denote easy and solid bars denote randomized hard. A number above a pale bar is its easy rate, and a number on a solid bar is its hard rate. ACiT is first in each group.

close and the LIBERO mean is carried by Spatial. On RoboTwin 2.0 the pattern is sharper: ACiT leads on the tasks limited by inter-arm coordination (Pick Dual Bottles, Place Dual Shoes), and its margin over the strongest baseline grows under randomization on the tasks limited by velocity and phase (Shake Bottle, Move Can Pot), where a flat condition cannot express the motion that the scene excites.

What the Attention Does. Figure 4 shows what the two attention mechanisms do. In the top row the joint self-attention is sparse: a few pairs carry most of it, and they are not limited to neighboring joints, including pairs that span the two arms. Joint-level tokens carry this structure, which a flattened state vector cannot. In the bottom row a joint’s query does not spread over the frame but lands on the objects its motion depends on, the bottle a gripper takes in Pick Dual Bottles and, in Place Dual Shoes, both the shoe to lift and the box to place it in. Vision enters as evidence addressed to a joint, not as one summary of the scene.

Figure 4. Joint attention of ACiT. Top: joint self-attention inside the Inverted State Encoding on two tasks, averaged over five episodes and ten frames, with the left arm (joints 1–7) and the right arm (joints 8–14) marked. Bottom: joint-to-vision cross attention inside State-Visual Grounding, for the gripper of the right arm and for the elbow of the left arm.

Real-World Experiments

Figure 3 shows the four real-world tasks, and one of the most direct pieces of evidence of the gain appears in failure recovery. Figure 6 is a closed-loop trace of ACiT: the grasp fails at Step 2, the arm falls back to the previous state at Step 3 and re-attempts at Step 4, completing the pick at Step 6. In the rollouts we inspected, the baselines do not back off at the failure point and instead keep searching in place.

Real-world platform and tasks

Figure 3. Real-world platform and tasks. The Dual reBotArm-B601-DM performs four tasks: (a) pick cube, (b) stack cube, (c) lift pot, and (d) handover cube.

Closed-loop recovery on the real robot

Figure 6. Closed-loop recovery on the real robot. Top: global view. Bottom: gripper view. After a grasp failure (Step 2), ACiT falls back to the previous state (Step 3) and re-attempts (Step 4), completing the pick (Step 6).

A failed grasp is a proprioceptive event: the gripper closes to its limit without an object and the arm does not take the load it expects, while vision stays misleading because the cube is still in the workspace, only displaced. Under equal conditioning the visual evidence can dominate the proprioceptive one, which leaves the policy holding the pose or closing again. ACiT lets proprioception carry the state and reduces vision to a per-joint calibration, so the condition reflects a closed and empty gripper, and the decoded chunk is the back-off-and-retry segment that the demonstrations contain.

Figure 7 reports the success rates under in-distribution, out-of-distribution (unseen object colors and initial poses) and dark-background conditions. ACiT is the strongest or tied-strongest method in every condition and task, and its margin grows under the two shifts: with a dark background it still picks cube at 83%, against less than 40% for the baselines.

Figure 7. Real-world results. Success rates for ACiT, ACT, DP, and FM on four tasks under in-distribution, unseen-OOD, and dark-background-OOD conditions. ACiT is shown first in each group.

Transfer to a Large VLA

We apply ACiT’s conditioning to the post-training of the mainstream π0\pi_0 model [4], which yields π0\pi_0-ACiT. Both models are fine-tuned end-to-end per task, with the PaliGemma VLM [3] and the flow-matching expert keeping their architecture and objective. Only π0\pi_0’s single-frame proprioceptive token is replaced by GtG_t from the Inverted State Encoding and State-Visual Grounding of ACiT.

Simulation Benchmarks

Figure 8 compares the two models on six RoboTwin tasks. The gains land on the same group of tasks as the ACT-family results of Figure 5: both backbones improve where one frame cannot determine each joint’s action on its own (Pick Dual Bottles, Handover Block, Place Dual Shoes) and stay flat where it can (Lift Pot, Move Can Pot, Shake Bottle). Two backbones that differ in size, visual prior and action head agree on which tasks benefit, so the gain comes from the conditioning interface rather than from a property of a backbone. On the randomized hard split it moves to Shake Bottle and Lift Pot, whose actions depend on velocity and on the phase relation between joints, a relation that exists only in the history. Randomization removes the same fraction for both models, so the gain is not a tolerance to visual shift.

Figure 8. Transferring the conditioning design to a large VLA on RoboTwin 2.0. Success rate of π0\pi_0-ACiT (left bar of each task) and π0\pi_0 (right bar). The pale bar is the easy rate and the solid bar drawn on top of it is the randomized-hard rate, so the pale cap is the drop caused by randomization. Numbers above the pale bars are easy rates, numbers on the solid bars hard rates.

Real-World Transfer

Figure 9 repeats the comparison on the dual-arm B601 platform. π0\pi_0-ACiT improves over π0\pi_0 on average across the four tasks and three conditions (57.0 against 39.4). The gain is largest under the dark-background shift, where π0\pi_0 nearly fails on the two dual-arm tasks and π0\pi_0-ACiT still completes them. In distribution and under the unseen shift π0\pi_0-ACiT matches the ACT-based ACiT, so the interface transfers to a large pretrained backbone at a lower cost per action (Table 3). Averaged over the 12 RoboTwin settings (6 tasks, 2 splits) and the 12 real-world settings (4 tasks, 3 conditions), π0\pi_0-ACiT raises the success rate over π0\pi_0 by 26.9 points.

Figure 9. Real-world π0\pi_0/π0\pi_0-ACiT transfer on the B601 platform. Each task has six bars: π0\pi_0-ACiT under in-distribution, unseen-object/background, and dark-background conditions (light to dark orange), followed by π0\pi_0 under the same conditions (light to dark blue).

MethodParamsStepsms/act.Control (Hz)
ACT84.0 M1 (1)0.362794
DP293.2 M10 (10)2.12471
FM293.2 M10 (20)3.74267
ACiT92.6 M1 (1)0.482064
π0\pi_04.03 B10 (10)5.93169
π0\pi_0-ACiT4.05 B10 (10)4.73211

Table 3. Inference cost on one RTX 3090, at the 240 × 320 observation resolution used throughout the RoboTwin experiments. Steps counts integrator steps (network evaluations in brackets), and the last two columns are per executed action.

Ablation

Table 1 removes one module at a time, keeping the history length, the encoder, the protocol and the action head fixed. Equal fusion self-attends visual and joint tokens together. w/o joint attn keeps the per-joint 1D convolution

VariantLIBERORoboTwinReal world
easyhardIDunseendark
ACiT (full)79.062.714.060.034.144.1
equal fusion58.3<br>−20.752.3<br>−10.45.0<br>−9.048.4<br>−11.617.5<br>−16.613.3<br>−30.8
w/o joint attn69.3<br>−9.757.0<br>−5.79.8<br>−4.252.5<br>−7.523.3<br>−10.828.3<br>−15.8
flat state65.3<br>−13.753.3<br>−9.47.0<br>−7.049.2<br>−10.815.0<br>−19.121.6<br>−22.5
flat decode75.0<br>−4.060.0<br>−2.711.8<br>−2.254.2<br>−5.825.8<br>−8.341.6<br>−2.5

Table 1. Module ablation (mean success rate %). Small numbers give the change relative to the full model, per-task rates in Appendix § 8.§ 1.

but drops the attention among joint tokens, and flat state drops both. Flat decode replaces per-joint decoding with one shared query.

Removing any one module hurts on every benchmark, so the three are complementary rather than redundant. The conditioning relation costs most, the state-side variants next and the action-side factorization least, in five of the six settings. The exception is the unseen condition, where flat state falls further than equal fusion, so under unreliable vision the per-joint state representation matters more than the fusion relation.

Robustness

Training Seeds. Two further seeds leave the task ranking unchanged and move each task by at most eight points. Averaged over the tasks the success rate is 62.7±0.662.7 \pm0.6, so the aggregate in Figure 5 does not depend on the seed, while a single task carries the few points of spread that 100 episodes leave.

History Length. Figure 10 shows a flat optimum: performance peaks at eight frames, changes little between four and sixteen, and is clearly worse with a single frame, whose cost is largest under the dark-background shift. We keep eight frames throughout.

Figure 10. History length of the conditioning input. (a) Simulation: RoboTwin easy and hard, and LIBERO. (b) Real world: in-distribution, unseen and dark-background. All curves peak at H=8H=8 or tie it at 16; the per-task rates are in Appendix § 8.

Efficiency

Table 3 reports inference cost and model size. ACiT adds only 8.6 M parameters over ACT (92.6 M against 84.0 M) and keeps its one-step sampler, at 0.48 ms per action, while DP and FM pay ten integrator steps at a 3.2× larger model. The same holds after transfer: π0\pi_0-ACiT needs 4.73 ms per action against 5.93 ms for π0\pi_0, at the same ten steps and essentially the same size (4.05 B against 4.03 B), so the reduction comes from the interface, not from a cheaper sampler.

Conclusion

We presented ACiT, which treats historical joint coupling as the primary condition and vision as a later calibration: each joint carries its own history, retrieves its own visual evidence and decodes its own trajectory. It improves most LIBERO, RoboTwin 2.0 and real-world tasks, and transfers to a pretrained VLA at lower cost.

Limitations. ACiT is instantiated only with a continuous action head, so it does not cover the discrete action tokens that autoregressive VLAs rely on. Extending it there needs demonstration data beyond what our dual-arm platform provides, which we leave to future work.

Acknowledgments

We thank Shenzhen Zhibo Data Technology Co., Ltd. for the professional collection of the real-robot data used in this work.

Experimental Setup

Simulation Benchmarks

RoboTwin 2.0 is a bimanual platform with two 6-DoF arms and parallel grippers, simulated for table-top manipulation. We use six tasks, shown in Figure 11. Lift Pot lifts a pot by both handles and holds it; Pick Dual Bottles grasps and lifts two bottles at once; Handover Block passes a block from one arm to the other; Move Can Pot lays the can against the pot; Place Dual Shoes places two shoes into a box; Shake Bottle lifts a bottle and shakes it. Each task comes with an easy split (demo_clean) and a randomized split (demo_randomized) in which object poses, object colors and the scene background are randomized. All numbers are 100 episodes per task, and the success criterion is the one shipped with the benchmark, listed in Table 4.

TaskSuccess condition
Lift Potpot above 0.82 m and upright, both TCPs within 3 cm of its handles
Pick Dual Bottlesboth bottles within 10 cm of their targets and above 0.89 m
Handover Blockblock within 3 cm of the target pose and right gripper open
Move Can Potcan tipped on its side beside the pot (<20 cm, 3.5 cm, 15°), at table height, both grippers open
Place Dual Shoesboth shoes at the target pose (5 cm, 0.07 quaternion) and both grippers open
Shake Bottlebottle raised above 0.8 m

Table 4. Success conditions of the six RoboTwin tasks, as implemented by the benchmark. A rollout counts as a success only when every condition of its task holds at the same step.

Six RoboTwin 2.0 tasks: Lift Pot, Pick Dual Bottles, Handover Block, Move Can Pot, Place Dual Shoes, and Shake Bottle

Figure 11. The six RoboTwin 2.0 tasks used in § 4.2, viewed from the head camera. Both arms are visible in each panel; the randomized split varies object poses, colors and the background of the same scenes.

LIBERO evaluates single-arm table-top manipulation with language instructions. We use three suites: LIBERO-10 (ten long-horizon tasks), LIBERO-Object (objects unseen in training) and LIBERO-Spatial (spatial relations), with 100 episodes per suite.

Implementation Details

The ACT-family models share one protocol: 50,000 training steps, batch size 16, action chunk 50 with 50 executed steps, seed 1000, and AdamW [20]. The visual backbone is a ResNet-18 [9]. All models are implemented in PyTorch [26] with LeRobot [6], and the simulator is MuJoCo [32]. For run-to-run variation we follow the protocol of ACT [41], which reports three training seeds on simulation and a single trained policy on hardware, where every additional seed needs a full physical re-evaluation. Table 2 of the main paper reports the same sweep. ACiT uses a state history of H=8H=8 frames. ACT conditions on the current state only; DP and FM take H=2H=2 frames and flatten them into the global condition, which is the setting their original papers use. DP and FM execute 32 steps in the 50k_final runs and 10 in the 5task runs.

Taskseed 1000seed 1001seed 1002mean ± std
Lift Pot55565455.0 ± 1.0
Pick Dual61635961.0 ± 2.0
Handover69696969.0 ± 0.0
Move Can64606864.0 ± 4.0
Place Dual Shoes41353838.0 ± 3.0
Shake Bottle88899089.0 ± 1.0

Table 2. Training-seed variance of ACiT (easy split, success rate over 100 episodes, hence integer).

Architecturally, ACiT keeps the ACT transformer (d=512d=512, eight heads, FFN width 3200, four encoder and seven decoder layers, 32-d CVAE latent) and adds a two-layer state encoder followed by a two-layer cross-attention block. The π0\pi_0 backbone is left intact; π0\pi_0-ACiT adds a two-layer state encoder and a single grounding layer to it and keeps the flow-matching action expert unchanged.

For the VLA transfer (§ 4.4), we initialize π0\pi_0-ACiT from pi0_base. The PaliGemma [3] backbone and the flow-matching action expert keep their original architecture, and π0\pi_0’s single-frame proprioceptive token is replaced by the two conditioning modules of ACiT. The Inverted State Encoding encodes each joint’s eight-frame history with a 1D convolution and a linear projection into a joint token, and forms the proprioceptive prior by inter-joint self-attention. State-Visual Grounding then applies a joint-level cross-attention over the visual tokens, producing GtG_t as the proprioceptive condition of the action expert. The Inverted Action Chunking is not transferred, and the flow-matching objective is unchanged, so the action head keeps its architecture and capacity and only the conditioning side changes. Both models are trained per task with full fine-tuning, batch size 32, for 50k steps on two A100 GPUs.

Evaluation follows the benchmark defaults: 240 × 320 on RoboTwin for every model, with 100 episodes per task, 256 × 256 on LIBERO with 100 episodes per suite, and 30 rollouts per task and condition on the real-world platform, where training uses 50 demonstrations per task, the same recipe as above, and all rollouts run on a single RTX 3090.

Ablations, Sensitivity and Cost

Module Ablation

Table 5 and Table 6 give the per-task rates behind Table 1. The three removals do not fail in the same places. Equal fusion costs the most on Handover Block and Place Dual Shoes, the two tasks whose success turns on coupling between joints, and it is almost free on Move Can Pot. Flat state costs a nearly constant amount on every task, which is what one expects when the model loses the per-joint state representation rather than a specific coupling. Flat decode is the opposite: it loses on Pick Dual Bottles and Place Dual Shoes, where each joint needs its own trajectory, and it gains on Handover Block and Move Can Pot, where one shared trajectory keeps the joints in phase.

VariantRoboTwin (easy) LiftPickHand.MoveShoesShakeMeanRoboTwin (hard) LiftPickHand.MoveShoesShakeMeanLIBERO 10Obj.Spa.Mean
ACiT55616964388962.741401904714.050998879.0
equal fusion56503068228852.312020255.030737258.3
w/o joint attn39606962308257.0360120389.838908069.3
flat state48505856288053.323070307.035857665.3
flat decode54457974238560.05811604111.845958575.0

Table 5. Per-task module ablation on the simulation benchmarks (success rate, %, 100 episodes per task).

VariantIn-distribution PickStackPotHand.MeanUnseen PickStackPotHand.MeanDark background PickStackPotHand.Mean
ACiT93.363.333.350.060.053.323.316.743.334.183.343.316.733.344.1
equal fusion76.756.723.336.748.440.06.76.716.717.530.013.30.010.013.3
w/o joint attn80.056.730.043.352.543.313.313.323.323.353.323.313.323.328.3
flat state70.056.723.346.749.233.36.76.713.315.043.323.36.713.321.6
flat decode83.346.730.056.754.243.313.313.333.325.873.330.020.043.341.6

Table 6. Per-task module ablation on the real-world platform (success rate, %; 30 rollouts per cell).

Sensitivity to History Length

History length is a hyperparameter of the conditioning interface rather than a module, so we report it separately. Table 7 and Table 8 vary the number of proprioceptive frames and keep everything else fixed. The same pattern appears everywhere: one frame is the weakest, four recovers most of the gap, eight is best on RoboTwin and on the real platform and matches four on LIBERO, and sixteen is slightly worse, which is what one expects when extra frames add redundancy but no new dynamics.

HHRoboTwin (easy) LiftPickHand.MoveShoesShakeMeanRoboTwin (hard) LiftPickHand.MoveShoesShakeMeanLIBERO 10Obj.Spa.Mean
152456058348555.7250120359.046918273.0
454566662368760.231001604211.851998879.3
855616964388962.741401904714.050998879.0
1654606863378861.741301804513.349988778.0

Table 7. History length on the simulation benchmarks (success rate, %, 100 episodes per task). HH is the history length, in frames.

HH\multicolumn{5}{c}{In-distribution}\multicolumn{5}{c}{Unseen}\multicolumn{5}{c}{Dark background}
PickStackPotHand.MeanPickStackPotHand.MeanPickStackPotHand.Mean
176.746.723.340.046.743.316.713.333.326.670.033.310.020.033.3
486.756.730.046.755.046.720.013.336.729.276.736.713.326.738.4
893.363.333.350.060.053.323.316.743.334.183.343.316.733.344.1
1690.060.030.046.756.750.020.016.740.031.783.343.316.733.344.1

Table 8. History length on the real-world platform (success rate, %; 30 rollouts per cell).

Inference Cost

Cost is measured on a single RTX 3090 (24 GB) with batch size 1 and fp16, and each entry of Table 3 averages 100 calls

after 20 warm-up calls on the RoboTwin Lift Pot checkpoints at 50k steps. The table reports the cost per executed action, which divides one inference by the number of actions run without re-planning, i.e., what a closed-loop controller pays. The figures below are per inference, so that samplers can be compared directly. Step counts alone do not fix the cost, because integrators differ in how many network evaluations (NFE) they spend per step. DP and FM share the same U-Net and the same parameter count but not the same sampler: DP takes one evaluation per DDIM step, whereas FM integrates with a midpoint rule that evaluates the network twice per step, so its ten steps cost twenty evaluations (119.8 ms per inference) and match DP at twenty steps (120.5 ms); at an equal budget of ten evaluations the two coincide (67.5 against 68.1 ms). We therefore report both steps and NFE, and read cost differences across families as sampler differences unless the NFE budget is held fixed.

Seed Variance

Table 2 of the main paper repeats the training of ACiT and ACT on RoboTwin with two further seeds. The evaluation episodes are shared across runs, so the only source of variation is the training itself, and the spread across seeds stays small relative to the gap between the two models.

Analysis

Failure Cases

Figure 12 and Figure 13 also show how the methods fail once the scene is shifted. The baselines fail at the grasp itself: under a changed background, ACT and FM approach the cube with an offset and close on empty space, and DP pushes it away before closing; under a changed object color the same offset appears, which is the behavior expected when a single frame has to carry both the pose and the appearance. ACiT keeps the grasp in the same frames and fails later, when placing the object. Its remaining failures are therefore concentrated in placement precision and in the phase of a handover, not in grasp stability, which is consistent with Figure 7: the gap to the baselines is smallest on the two tasks whose success is dominated by the grasp and largest under the dark shift.

Figure 12. Out-of-distribution dark background. ACT, DP, FM and ACiT on the four tasks under the background shift.

Figure 13. Out-of-distribution unseen initial state and color, on Pick cube and Stack cube.

Figure 14–Figure 19 replay one episode of each task under all four policies, with the checker outcome and the number of steps used marked on the last frame; the success conditions are listed in Table 4; every row of a figure shares the same initial state. On Place Dual Shoes success requires both shoes inside the box with the target orientation and both grippers open, which only ACiT satisfies. On Pick Dual Bottles ACiT lifts the bottle within a fraction of the step budget while the baselines never leave the table. On Lift Pot ACiT and DP both succeed, with ACiT finishing in fewer steps, while ACT and FM fail. On Move Can Pot only ACiT brings the can to the pot. On Shake Bottle DP never lifts the bottle, and on LIBERO-Spatial all four policies succeed with ACiT taking the fewest steps.

Qualitative comparison on Place Dual Shoes under four policies

Figure 14. Qualitative comparison on Place Dual Shoes, same task and episode under four policies. The outcome and the number of steps are marked on the last frame. Only ACiT leaves both shoes in the box and releases them; ACT keeps a shoe in its closed gripper, and DP and FM stop with one shoe outside the box.

LIBERO-Spatial qualitative comparison showing four rows labeled ACiT, ACT, DP, and FM, with success · 71 steps, success · 75 steps, success · 73 steps, and success · 78 steps, respectively; time arrow below

Figure 19. Qualitative comparison on LIBERO-Spatial. All four policies solve the episode, and ACiT finishes in the fewest steps.

Action-Space Dependency Metrics

The motivation for the architecture rests on two properties of the action distribution: its effective dimensionality is far below the number of joints, and the joints are strongly coupled, with part of that coupling delayed in time. We measure

Attention Maps and Head Matching

Figure 20 compares the two matrices. For each dual-arm task we compute the mean 14×1414 \times14 self-attention map of each head over 5 episodes and 10 frames, symmetrize it, and compare it with the absolute zero-lag action-correlation matrix using Spearman correlation over off-diagonal entries. The best head per task is only weakly aligned: Lift Pot 0.05 (head 5), Pick Dual 0.12 (head 6), Handover 0.18 (head 2), and Shoes 0.26 (head 5). Thus head-level attention is not a reliable proxy for the action-correlation structure. We therefore report it as a qualitative supplement to the quantitative statistics of §C.2.

Figure 20. Action correlation matrix (left) versus the best-matching self-attention head (right) for each dual-arm task. The Spearman correlation over off-diagonal entries is shown above each attention map.

Extended Related Work

Proprioceptive History: Representation and Conditioning

Action policies are moving from single-frame proprioceptive conditions to explicit history modeling. In the continuous route, ACT [41] maps the current joint state and visual observations to an action chunk with a CVAE. Diffusion Policy [8] conditions on the last HH observations, proprioception included, concatenated into one global vector before conditional denoising; its own ablation reports that a vision-based policy degrades as HH grows, which suggests that a flattened history does not scale. A2A [12] embeds historical proprioceptive sequences into the latent space of action generation, and Chronos [42] treats observation history as the latent state of the policy. Meanwhile, another route discretizes historical actions or state-action trajectories into tokens for autoregressive VLAs at the cost of losing continuous coupling: VQ-BeT [14] uses a residual VQ-VAE to encode continuous action chunks into discrete latent codes and adds a continuous offset for precision. FAST [28] compresses action chunks in the frequency domain with DCT and BPE, substantially reducing autoregressive training cost. OmniSAT [21] further uses B-spline encoding and multi-stage residual quantization to quantize position, rotation, and gripper subspaces separately. VITA [22] learns a shared discrete latent space for vision and action and internalizes visual dynamics as an implicit chain of thought. Both continuous and discrete routes validate the value of using history [35, 40]. However, under either interface, proprioception is still represented as a flat vector or a one-dimensional token sequence, and the continuous coupling among joints is not explicitly modeled.

At the fusion level, mainstream methods symmetrically concatenate vision and proprioception into the same attention stack. A few works instead let the robot state actively guide vision: RcP [43] uses proprioceptive-state correspondence to guide robot-centric pooling of image features, extracting robot-centric visual representations robust to self-distractors. AVA-VLA [39] uses a recurrent history belief to reweight current visual tokens. These explorations show that proprioception should not be only a passive condition, but they stop at perceptual filtering and do not carry structured joint priors through to action generation.

TaskPRwithin-Lwithin-Rcross(0)cross(best)
Lift Pot1.440.6020.5790.8020.956
Pick Dual Bottles2.710.3410.3240.5790.881
Handover Block1.920.4220.5960.4050.716
Place Dual Shoes4.420.3410.3010.3520.670

Table 9. Action-space statistics on the four dual-arm RoboTwin tasks. Move Can Pot and Shake Bottle are single-arm and are not listed, because their cross-arm quantities are undefined.

Topology-Aware Morphological Bias

Standard VLA policies are usually embodiment-agnostic and must infer kinematic structure from data. GET-Zero [27] proposes a Graph Embodiment Transformer that uses embodiment graph connectivity as a learned structural bias in attention, enabling zero-shot adaptation to new joint configurations and link lengths. Human2LocoMan [24] further proposes a Modularized Cross-embodiment Transformer (MXT) that uses embodiment-specific tokenizers and detokenizers on a shared Transformer trunk to handle heterogeneous bodies. These works establish that joints are not independent dimensions and that the kinematic graph should be an explicit inductive bias. However, existing topology methods mainly target cross-embodiment transfer and action-interface design. They do not treat continuous per-joint proprioceptive history as a generative prior, nor let each joint retrieve its own visual calibration signal. ACiT unifies continuous per-joint history, kinematic topology, and per-joint visual grounding in a single inverted generation pipeline.

Qualitative comparison on Pick Dual Bottles

Figure 15. Qualitative comparison on Pick Dual Bottles. ACiT lifts the bottle clear of the table and finishes well inside the step budget, while ACT, DP and FM leave both bottles in place.

Qualitative comparison on Lift Pot

Figure 16. Qualitative comparison on Lift Pot. ACiT and DP both raise the pot, with ACiT finishing in fewer steps, while ACT and FM fail to lift it.

Qualitative comparison on Move Can Pot

Figure 17. Qualitative comparison on Move Can Pot. ACiT places the can against the pot and finishes in 158 steps; ACT never moves it into place, DP does not approach it, and FM knocks it away from its start position.

Qualitative comparison on Shake Bottle

Figure 18. Qualitative comparison on Shake Bottle. ACiT, ACT and FM shake the bottle, while DP never lifts it and is marked failure.

References

  1. [1]Minoru Asada. Proprioception and body schema. In The Sense of Self: Perspectives from Science and Zen Buddhism. Oxford University Press, 2018. doi: 10.1093/oso/9780199674923.003.0018.DOI
  2. [2]Marie Avillac, Sophie Denève, Etienne Olivier, Alexandre Pouget, and Jean-René Duhamel. Reference frames for representing visual and tactile locations in parietal cortex. Nature Neuroscience, 8(7):941–949, 2005. doi: 10.1038/nn1480.DOI
  3. [3]Lucas Beyer, Andreas Steiner, André Susano Pinto, Alexander Kolesnikov, Xiao Wang, Daniel Salz, Maxim Neumann, Ibrahim Alabdulmohsin, Michael Tschannen, Emanuele Bugliarello, et al. Paligemma: A versatile 3B VLM for transfer. arXiv preprint arXiv:2407.07726, 2024.arxiv.org/abs/2407.07726
  4. [4]Kevin Black, Noah Brown, Danny Driess, et al. π₀: A vision-language-action flow model for general robot control. arXiv preprint, 2024.DOI
  5. [5]Kevin Black et al. π₀.₅: A vision-language-action model with open-world generalization. https://www. physicalintelligence.company/blog/pi05, 2025.
  6. [6]Remi Cadene, Simon Alibert, Alexander Soare, Quentin Gallouedec, Adil Zouitine, and Thomas Wolf. LeRobot: State-of-the-art machine learning for real-world robotics in PyTorch. https://github.com/huggingface/lerobot, 2024.
  7. [7]Tianxing Chen et al. Robotwin 2.0: A scalable data generator and benchmark with strong domain randomization for robust bimanual robotic manipulation. arXiv preprint arXiv:2506.18088, 2025.arxiv.org/abs/2506.18088
  8. [8]Cheng Chi, Zhenjia Xu, Siyuan Feng, Eric Cousineau, Yilun Du, Benjamin Burchfiel, Russ Tedrake, and Shuran Song. Diffusion policy: Visuomotor policy learning via action diffusion. In Robotics: Science and Systems (RSS), 2023.
  9. [9]Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2016.arxiv.org/abs/1512.03385
  10. [10]Tobias Heed, Verena N. Buchholz, Andreas K. Engel, and Brigitte Röder. Tactile remapping: From coordinate transformation to integration in sensorimotor processing. Trends in Cognitive Sciences, 19(5):251–258, 2015. doi: 10.1016/j.tics.2015.03.001.DOI
  11. [11]Jonathan Ho, Ajay Jain, and Pieter Abbeel. Denoising diffusion probabilistic models. In Advances in Neural Information Processing Systems (NeurIPS), 2020.arxiv.org/abs/2006.11239
  12. [12]Jindou Jia, Gen Li, Xiangyu Chen, Tuo An, Yuxuan Hu, Jingliang Li, Xinying Guo, and Jianfei Yang. Action-to-action flow matching. arXiv preprint arXiv:2602.07322, 2026.arxiv.org/abs/2602.07322
  13. [13]Konrad P. Körding and Daniel M. Wolpert. Bayesian integration in sensorimotor learning. Nature, 427(6971):244–247, 2004. doi: 10.1038/nature02169.DOI
  14. [14]Seungjae Lee, Yibin Wang, Haritheja Etukuru, H. Jin Kim, Nur Muhammad Mahi Shafiullah, and Lerrel Pinto. Behavior generation with latent actions. In International Conference on Machine Learning (ICML), 2024.arxiv.org/abs/2403.03181
  15. [15]Jijie Li, Xu Yang, Junhong Zou, Chunhai Zhao, Chaoyang Zhao, Zhen Lei, and Xiangyu Zhu. Object-centric conditioning for visuomotor flow matching. In Conference on Robot Learning (CoRL), 2026.arxiv.org/abs/2609.24155
  16. [16]Wenjie Li, Yash Jangir, Ignacy Stepka, Yash Agarwal, Marion Kipsang, and Yonatan Bisk. Lost in reconstruction: Aligning action representations with language in vision-language-action models. arXiv preprint arXiv:2608.10484, 2026.arxiv.org/abs/2608.10484
  17. [17]Yaron Lipman, Ricky T. Q. Chen, Heli Ben-Hamu, Maximilian Nickel, and Matt Le. Flow matching for generative modeling. In International Conference on Learning Representations (ICLR), 2023.arxiv.org/abs/2210.02747
  18. [18]Bo Liu, Yifan Zhu, Chongkai Gao, Yuxuan Feng, Qiang Liu, Yuke Zhu, and Peter Stone. Libero: Benchmarking knowledge transfer for lifelong robot learning. In Advances in Neural Information Processing Systems (NeurIPS), 2023.arxiv.org/abs/2306.03310
  19. [19]Yong Liu, Tengge Hu, Haoran Zhang, Haixu Wu, Shiyu Wang, Lintao Ma, and Mingsheng Long. iTransformer: Inverted transformers are effective for time series forecasting. In International Conference on Learning Representations (ICLR), 2024.arxiv.org/abs/2310.06625
  20. [20]Ilya Loshchilov and Frank Hutter. Decoupled weight decay regularization. In International Conference on Learning Representations (ICLR), 2019.
  21. [21]Huaihai Lyu, Chaofan Chen, Senwei Xie, Pengwei Wang, Xiansheng Chen, Shanghang Zhang, and Changsheng Xu. Omnisat: Compact action token, faster auto regression. arXiv preprint arXiv:2510.09667, 2025.arxiv.org/abs/2510.09667
  22. [22]Xiangkai Ma, Lekai Xing, Han Zhang, Wenzhong Li, and Sanglu Lu. Unifying perception and action: A hybrid-modality pipeline with implicit visual chain-of-thought for robotic action generation. arXiv preprint arXiv:2511.19859, 2025.arxiv.org/abs/2511.19859
  23. [23]R. Chris Miall and Daniel M. Wolpert. Forward models for physiological motor control. Neural Networks, 9(8):1265–1279, 1996. doi: 10.1016/S0893-6080(96)00035-4.DOI
  24. [24]Yaru Niu, Yunzhe Zhang, Mingyang Yu, Changyi Lin, Chenhao Li, Yikai Wang, Yuxiang Yang, Wenhao Yu, Tingnan Zhang, Zhenzhen Li, Jonathan Francis, Bingqing Chen, Jie Tan, and Ding Zhao. Human2locoman: Learning versatile quadrupedal manipulation with human pretraining. In Robotics: Science and Systems (RSS), 2025.arxiv.org/abs/2506.16475
  25. [25]NVIDIA. Gr00t n1: An open foundation model for generalist humanoid robots. arXiv preprint arXiv:2503.14734, 2025.
  26. [26]Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, et al. PyTorch: An imperative style, high-performance deep learning library. In Advances in Neural Information Processing Systems (NeurIPS), 2019.arxiv.org/abs/1912.01703
  27. [27]Austin Patel and Shuran Song. Get-zero: Graph embodiment transformer for zero-shot embodiment generalization. In IEEE International Conference on Robotics and Automation (ICRA), 2025.arxiv.org/abs/2407.15002
  28. [28]Karl Pertsch, Kyle Stachowicz, Brian Ichter, et al. Fast: Efficient action tokenization for vision-language-action models. In Robotics: Science and Systems (RSS), 2025.arxiv.org/abs/2501.09747
  29. [29]Uwe Proske and Simon C. Gandevia. The proprioceptive senses: Their roles in signaling body shape, body position and movement, and muscle force. Physiological Reviews, 92(4):1651–1697, 2012. doi: 10.1152/physrev.00048.2011.DOI
  30. [30]Seeed Studio. rebot arm b601-dm: Fully open-source 6-dof robotic arm. https://github.com/Seeed-Projects/reBot-DevArm, 2026.
  31. [31]Kihyuk Sohn, Honglak Lee, and Xinchen Yan. Learning structured output representation using deep conditional generative models. In Advances in Neural Information Processing Systems (NeurIPS), 2015.
  32. [32]Emanuel Todorov, Tom Erez, and Yuval Tassa. MuJoCo: A physics engine for model-based control. In IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), 2012.DOI
  33. [33]Robert J. van Beers, Anne C. Sittig, and Jan J. Denier van der Gon. Integration of proprioceptive and visual position-information: An experimentally supported model. Journal of Neurophysiology, 81(3):1355–1364, 1999. doi: 10.1152/jn.1999.81.3.1355.DOI
  34. [34]Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Lilian Jones, Aidan N. Gomez, Łukasz Kaiser, and Illia Polosukhin. Attention is all you need. In Advances in Neural Information Processing Systems (NeurIPS), 2017.arxiv.org/abs/1706.03762
  35. [35]Fangyuan Wang, Peng Zhou, Jiaming Qi, Shipeng Lyu, Chengyang He, David Navarro-Alarcon, and Guodong Guo. Think proprioceptively: State-grounded visual token selection for vla policies. arXiv preprint arXiv:2602.06575, 2026.arxiv.org/abs/2602.06575
  36. [36]Uwe Windhorst. Muscle proprioceptive feedback and spinal networks. Brain Research Bulletin, 73(4-6):155–202, 2007. doi: 10.1016/j.brainresbull.2007.03.010.DOI
  37. [37]Daniel M. Wolpert, Zoubin Ghahramani, and Michael I. Jordan. An internal model for sensorimotor integration. Science, 269(5232):1880–1882, 1995. doi: 10.1126/science.7569931.DOI
  38. [38]Yilin Wu, Rutav Shah, Shuaiyi Huang, Sumedh A. Sontakke, Homanga Bharadhwaj, Andrea Bajcsy, Max Simchowitz, and Guanya Shi. Memory without curation: Learning multi-stream latent memory for robot policies. arXiv preprint arXiv:2610.14406, 2026.arxiv.org/abs/2610.14406
  39. [39]Lei Xiao, Jifeng Li, Juntao Gao, Feiyang Ye, Yan Jin, Jingjing Qian, Jing Zhang, Yong Wu, and Xiaoyuan Yu. Ava-vla: Improving vision-language-action models with active visual attention. In IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2026.arxiv.org/abs/2511.18960
  40. [40]Senlan Yao, Chenyu Yang, Jaehoon Kim, Aristotelis Sympetheros, and Robert K. Katschmann. Learning robust dexterous in-hand manipulation from joint sensors with proprioceptive transformer. arXiv preprint arXiv:2605.21330, 2026.arxiv.org/abs/2605.21330
  41. [41]Tony Z. Zhao, Vikash Kumar, Sergey Levine, and Chelsea Finn. Learning fine-grained bimanual manipulation with low-cost hardware. In Robotics: Science and Systems (RSS), 2023.arxiv.org/abs/2304.13705
  42. [42]Yulin Zhou, Yimeng Wang, Nengyu Wang, Shaojia Xing, Shiyun Tu, Xiang Li, Jingkai Zhang, Ningbo Jiang, Yuankai Lin, Hua Yang, Xiangrui Zeng, and Zhouping Yin. Chronos: A physics-informed full-history framework for non-markovian long-horizon manipulation. arXiv preprint arXiv:2606.30318, 2026.arxiv.org/abs/2606.30318
  43. [43]Zheyu Zhuang, Ville Kyrki, and Danica Kragic. Raising body ownership in end-to-end visuomotor policy learning via robot-centric pooling. In IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), 2024.arxiv.org/abs/2411.04331

Paper details

vlnc:W_e575bng5
Version
v1 · latest
Published
Fresh — added to the Hub within the last 30 days
Added
License
CC BY 4.0
Usage

views · 2

downloads · 0

A view counts once per browser per day, a download each time the PDF is requested.

  • Computer Science

Not recorded for this paper

View statement
Looking for related papers…
Contents