WorksComputer Science
PAIR: Proprioception-Aligned Image-Ray Representations for Viewpoint-Robust Manipulation
Abstract
While imitation learning has made substantial progress toward generalizable robot manipulation through scaling, viewpoint robustness remains a major bottleneck. Visuomotor policies rely heavily on visual inputs, yet images provide an inherently ambiguous signal under viewpoint change: similar scenes may appear markedly different across viewpoints, while distinct states may share certain visual overlaps. A policy must therefore infer not only the task-relevant state, but also how it is visually projected under a particular camera configuration. This entanglement of viewpoint-dependent appearance and task-relevant state can make learning inefficient and limit generalization to unseen viewpoints. Recent work has begun to address this challenge through camera-ray conditioning and representation learning using geometric cues to induce invariance. However, providing camera information as additional context does not by itself ground visual representations in how the task scene is visually projected, while geometric alignment without camera context can introduce spurious correspondences by ignoring viewpoint-dependent projection. We propose PAIR, a contrastive learning framework that aligns joint image-ray representations using robot proprioception. PAIR encourages observations with similar robot states to remain close in representation space across viewpoints, providing a structured grounding signal for learning viewpoint-robust features. PAIR is a highly adaptable framework: it can be used to train Diffusion Policy (DP) or post-train vision-language-action (VLA) models. In simulated and real-world manipulation benchmarks with camera variation, PAIR consistently outperforms the behavior cloning baseline without ray conditioning by 12.7% and 26.7% points, respectively, demonstrating stronger generalization to unseen viewpoints. Project website: PAIRrobot.github.io.
Introduction
Robot imitation learning has made significant progress toward general-purpose manipulation by scaling training data, model capacity, and compute [1]. Recent visuomotor policies can learn increasingly more general manipulation behaviors directly from image observations, and can transfer across different objects, tasks, and scenes. However, robustness to viewpoint variation remains a key challenge for generalizable manipulation [2].
This challenge stems from a fundamental property of visual observations: images do not directly encode the physical state of the world, but instead capture a viewpoint-dependent projection of that state. As the camera viewpoint changes, a fixed underlying task state can give rise to substantially different visual observations through changes in perspective and visibility. Conversely, distinct physical states may appear deceptively similar when they share similar camera-relative geometry or when the task-relevant context is only partially captured within the camera’s field of view (see Figure 1A). This ambiguity is further exacerbated by typical camera-independent action spaces, in which visual observations vary with camera viewpoint while the corresponding action targets remain unchanged. Consequently, to infer which actions are appropriate for the given task, a visuomotor policy must not only capture the task-relevant information in the image, but also reason about how exactly they are visually projected.

Figure 1. Resolving visual ambiguity with camera geometry. (A) Robot datasets often contain visually confounding observations, where similar robot appearances correspond to substantially different robot states. (B) Camera-ray information provides explicit camera-to-robot geometry to help disambiguate such observations.
Consider a robot approaching a mug for grasping. From one view, the gripper may appear to the left of the mug, while from another view, the same physical configuration may make the gripper appear to the right of the mug. Yet this change in image-space geometry does not imply a corresponding change in what the robot should physically do. The policy must therefore determine how the observed visual relationship between the robot and the mug corresponds to their underlying spatial relationship. This implicitly demands that the policy look for additional visual cues such as the robot base to establish the robot’s spatial frame or fuse the vision features with the internal robot state for viewpoint-invariant action prediction. However, such visual context is not always guaranteed to be available in full, nor reliably utilized when learning from demonstrations. In addition, existing visuomotor policies typically rely on shallow, nonprincipled fusion between vision and robot state, where images are processed by a vision encoder and robot-state features are incorporated later through simple fusion mechanisms [3, 4].
Recent studies have explored improving viewpoint invariance for visuomotor policies through camera-ray conditioning [5] to use the camera state governing the visual projection as input, or use geometric signals such as proprioception to construct positive pairs for representation alignment [6, 7]. However, while camera-ray conditioning provides useful viewpoint context, it does not by itself ground visual representations in the projected task state. Similarly, while alignment using physically grounded signals can encourage invariance across physically similar states, this can produce conflicting supervision by ignoring viewpoint-dependent projection without camera context. In light of this, we propose PAIR, a contrastive learning framework for viewpoint-robust manipulation that jointly learns image–ray representations using Plücker embeddings with supervised proprioceptive alignment. PAIR is a highly adaptable framework: it can be used to train lightweight policy classes such as diffusion policy (DP) or fine-tune pre-trained Vision-Language-Action (VLA) models. Specifically, PAIR builds on the pre-trained SigLIP-family encoders and incorporates the proposed alignment objective as an auxiliary loss alongside the standard behavior cloning objective.
We make the following contributions. First, we introduce PAIR, a general training framework that unlocks the benefit of proprioceptive alignment through joint image–ray representation learning. Second, we show that PAIR consistently improves the performance of both diffusion policies and VLA models when trained on the same demonstration data. Third, we conduct extensive evaluations across multiple simulated and real-world benchmarks under systematic viewpoint variations, demonstrating strong robustness to changes in camera viewpoint both quantitatively and qualitatively.
Related Work
Vision Representations for Robotic Manipulation
A dominant paradigm in visual representation learning is vision-language alignment, which has enabled VLA models
with strong semantic understanding and instruction-following capabilities. Accordingly, many VLM and VLA architectures adopt pre-trained vision encoders such as CLIP [8] and SigLIP [9], whose representations are explicitly aligned with language semantics through large-scale image-text pre-training. While this semantic alignment provides strong high-level visual understanding, such representations often lack the fine-grained spatial and geometric awareness required for robotic manipulation [1].
Several methods study viewpoint robustness directly from RGB observations with data augmentation techniques including systematic data collection [10] or novel-view synthesis [11], while other methods improve the learning signal through cross-view supervision [12] and contrastive learning [6, 7]. Recent efforts attempt to directly use camera extrinsic information to account for viewpoint changes. For example, Zhang et al. [13] express robot actions in the camera frame to unify action representations across heterogeneous robot datasets, while Jiang et al. [5] condition the policy on Plücker ray maps derived from the camera pose. Rather than using camera geometry solely as an input signal or as a reference frame for expressing actions, PAIR grounds the image-ray representation in the underlying robot proprioceptive state using contrastive learning.
Rays as Positional Embeddings
Recent vision models increasingly incorporate camera rays as an explicit geometric signal for vision tasks including video generation [14], novel-view synthesis [15], and pose estimation [16]. Similar lines of work have also explored ray injection as positional encoding. For example, RayPE [17] incorporates Plücker ray geometry directly into Transformer attention for video generation. In robotics, Xu et al. [18] builds on RayPE for modality-balanced action generation in VLAs, while RayViT [19] injects patch-aligned Plücker ray features into a pre-trained DINOv3 ViT backbone as geometry-aware positional embeddings. PAIR similarly introduces Plücker rays as positional embeddings, but applies this during post-training of SigLIP/SigLIP2 [9, 20] encoders to ground the camera-ray-aware visual representations using robot proprioception.
Method
In this section, we formalize the problem in Sec. III-A, analyze the visual ambiguity that motivates our method in Sec. III-B, introduce the image-ray fusion mechanism in Sec. III-C, and describe the training procedure in Sec. III-D.
Problem Statement
We consider visuomotor policy learning from visual observations , robot proprioceptive state , camera rays , and a task specification (e.g., language), where we augment imitation learning with a spatially grounded visual representation learning problem. Specifically, a vision encoder maps the visual observation and optionally its associated camera geometry to a latent representation , where the ray information provides a geometry-aware positional encoding for the visual tokens. We assume that camera positions change across episodes, but remained fixed within each episode.
Our goal is to learn such that is grounded in the underlying robot state . In particular, observations of the same physical robot configuration from different viewpoints should induce consistent latent representations. The resulting representation is then used by a visuomotor policy , which predicts a sequence of future robot actions over an action horizon .
Visual Ambiguity of Proprioceptive Alignment
While proprioceptive alignment of visual representations provides a potential mechanism for inducing viewpoint invariance, aligning image representations according to proprioception alone is inherently ambiguous: proprioception specifies a state in a fixed physical frame, whereas the corresponding visual observation is largely affected by the camera–robot geometry.
Let and denote the end-effector pose and the camera pose in the robot-base frame, respectively. The visual appearance of the robot end-effector is governed in large part by their relative transform
Consequently, two observations and may satisfy
so that the robot appears similar despite occupying different physical states, as illustrated in Figure 1.A. Conversely, the same physical state can appear substantially different under varied camera poses. This leads to conflicting signals for the vision encoder: visually similar observations may be pushed apart, while visually dissimilar observations are pulled together, without specifying the camera transformations responsible.
Ray Positional Encoding
PAIR resolves this ambiguity by explicitly providing camera ray information, such that the encoder receives both the visual input and the camera geometry under which it was projected, and using proprioception to align the joint image–ray representation. Specifically, we motivate camera rays as a geometry-aware positional signal for visual tokens. As observed in RayPE [17], Plücker ray geometry is compatible with attention mechanisms because its canonical pairwise relation is bilinear. Let the ray associated with token be
where and denote its origin and direction in 3D space. The Plücker reciprocal product between two rays is
a pairwise measure of their geometric incidence. When projected ray features enter the query and key vectors, the geometry–geometry component of an attention score takes the bilinear form
where, for example, exactly recovers Eq. (4). Thus, attention between tokens from different views can directly capture pairwise geometric relations between their 3D rays, while naturally representing the Plücker reciprocal product. This structure is particularly relevant in robot demonstrations, where the robot embodiment provides a persistent physical reference with consistent correspondences across viewpoints for similar proprioceptive states. We use this compatibility to motivate exposing ray geometry to the attention layers of the vision encoder. See Table VI for quantitative analysis of attention activations and Figure 6 for qualitative visualizations of ray-conditioned attention maps.
| Task | BC | BC | PAIR | PAIR |
| w/o Ray | w/ Ray | w/o Ray | w/ Ray | |
| LIBERO | 1.28 | 2.91 | 1.64 | 3.52 |
| Real-World | 1.17 | 2.16 | 2.25 | 2.67 |
Table VI. Normalized average attention assigned to the visual token containing the ground-truth tool center point. The raw attention score is normalized by the expected attention under a uniform distribution, where a value of 1 denotes the expected value.
Figure 6. Visualization of the rollout attention maps of SmolVLA models trained on the LIBERO dataset. The average attention activations are overlaid on the corresponding input images. To ensure the consistency of the rollout, we sample a single trajectory from the training dataset. Without ray input, attention is concentrated on spatially irrelevant regions, even with proprioceptive alignment. In contrast, ray-conditioned leads to stronger attention to the end effector, with proprioceptive alignment further concentrating activation around it and its nearby workspace.
For each visual token and each of the robot arms, we construct the corresponding Plücker rays and encode them using lightweight convolutional networks. Specifically, we obtain two projected camera ray embeddings: , expressed relative to the end-effector of robot , and , expressed relative to the fixed robot base, as shown in Figure 2.A. The end-effector-relative rays encode a moving target that changes with the robot’s movement, whereas base-relative rays provide a persistent physical anchor across observations. These references jointly encode dynamic and static geometric structure, providing the geometric cues necessary for proprioceptive alignment across changes in visual appearance. For empirical results showing the benefits of using both ray modalities, see Table V.
Figure 2. Overview of PAIR. (A) PAIR introduces two ray representations: camera rays expressed relative to the moving end-effector frame and the robot-base frame, both encoded using Plücker coordinates. Red indicates zero Plücker moment, corresponding to rays that pass through the respective origin. The ray features are injected into the image features through an identity-initialized linear projection, which acts as a selective positional embedding for each vision token. (B) The joint image-ray representations are aligned according to proprioception similarity using supervised contrastive learning, which induces invariance against viewpoint variation.
| Task (View) | PAIR | PAIR w/o | PAIR w/o |
| Square D1 (Multi) | 38.5 | 28.8 (-9.7) | 35.3 (-3.2) |
| ALOHA Transfer (Multi) | 74.0 | 55.2 (-18.8) | 67.0 (-7.0) |
| LIBERO | 88.9 | 87.2 (-1.7) | 86.8 (-2.1) |
Table V. Ablation of end-effector-relative () and robot-base-relative () camera rays in PAIR. Average success rates (%) are reported, with the performance drop shown in parentheses.
We concatenate all representations and project them back to the original visual dimension,
where we initialize this projection as
The resulting image-ray patches is then added to the original position embedding of the pre-trained SigLIP model. As a result, ray injection introduces minimal architectural change: a single identity-initialized linear layer is added to the original model architecture.
Supervised Contrastive Learning
For proprioception-aligned image-ray representation learning, we adopt a supervised contrastive learning [22] objective based on geometric similarity between robot states. Specifically, let denote the -normalized latent representation of observation , and let the corresponding proprioceptive state be , where , , and denote the end-effector position, orientation, and gripper width, respectively. Given a training batch of size , we define the pairwise similarity matrix between observations and as
where is the sharpness temperature. In the single-view setting, , while in the multi-view setting, , which is a cross-view instance of .
For spatial grounding, we define the ground-truth pairwise distance as
where controls the quaternion distance term
For multi-robot settings, the pairwise distances between corresponding arms are averaged. We convert the geometric distance in Eq. (9) into a percentile-based soft positive weight. Let denote the global percentile rank of and the percentile cutoff. We define
such that the closest pairs receive weight 1 (hard positive), decreasing linearly to 0 at the cutoff. We optimize the weighted InfoNCE objective
where, in the single-view setting, diagonal pairs () are excluded from the loss. This pulls image-ray representations with similar robot states closer, while pushing apart those with dissimilar robot states, as shown in Figure 2.B. The final training objective combines the behavior cloning loss with the proprioception alignment loss :
where scales the proprioception alignment objective.
Experiments
In this section, we test the effectiveness of PAIR for viewpoint generalization of visuomotor policies under various settings. To understand whether the benefits arise specifically from proprioceptive alignment, we compare against temporal alignment [23], which defines correspondence based on the temporal distance between observations. We then examine how these alignment signals interact with camera-ray conditioning and affect downstream policy learning. Specifically, we address the following research questions:
RQ1 Does PAIR improve visual robustness across robotic embodiments (single-arm vs bi-arm), learning settings (single-task vs multi-task), and policy classes (Diffusion Policy vs VLA)?
RQ2 Is proprioception a better alignment signal than time when learning from expert demonstrations?
RQ3 Is ray conditioning necessary to unlock the benefits of proprioceptive alignment for visual representations?
Benchmarks
We evaluate PAIR on three simulated robot manipulation benchmarks spanning different embodiments, tasks, and learning settings.
a) MimicGen: [24] is a benchmark for imitation learning with synthetically augmented demonstrations. We use MimicGen as a standard single-task learning benchmark to study viewpoint robustness. We include the Square, Three Stack, and Three Piece Assembly tasks using the D1 distributions. Each task includes 1,000 demonstrations.
b) AlohaGym: [25] provides a canonical bimanual manipulation task on the ALOHA platform. We evaluate PAIR on the cube transfer task, where the dataset consists of 50 human and 50 motion-planned demonstrations, totaling 100 episodes.
c) LIBERO: [26] is a manipulation benchmark comprising four task suites: Object, Goal, Spatial, and Long, which emphasize object variation, spatial reasoning, goal-conditioned behavior, and long-horizon task execution, respectively. LIBERO evaluation uses language conditioning to distinguish among tasks. Policies are trained on the full LIBERO dataset spanning 6,500 episodes in 130 tasks.
For all three benchmarks, we re-collect the original demonstrations under diverse camera viewpoints by varying the camera extrinsics while keeping the underlying task definitions unchanged. This procedure introduces controlled variation in the visual observations for a direct evaluation of visual robustness. For single-task settings, we define the camera look-at point as the midpoint between the target objects and the robot end-effector, and sample camera distances for which the end-effector falls within the camera's field of view. For LIBERO, we pre-sample 50 camera poses for each task, all distinct from those used during data collection, and apply the same sampling criterion to ensure that the robot end-effector remains visible throughout each rollout. In addition, we evaluate both single-view and multi-view scenarios, where multiple viewpoints are provided for each training demonstration in the multi-view setting. A comparison of the camera settings is provided in Table II, and the distribution of camera poses is shown in Figure 3.
Figure 3. Simulated benchmarks and real-world experiments. PAIR is evaluated across both simulated and real-world manipulation tasks. In simulation, MimicGen and LIBERO use a Franka Panda arm with two side views and one front view, respectively, while AlohaGym uses a bimanual Aloha platform with two front views. In the real world, we evaluate three tabletop tasks using a Franka Research 3 (FR3) arm with two front view cameras, where the camera poses are randomized and estimated using AprilTags [21] at the start of each rollout.
Policy Architectures
To test whether the benefits of PAIR extend beyond a particular policy type, we evaluate it with two distinct policy classes: vision-based Diffusion Policies (DP) for single task learning and pre-trained Vision-Language-Action models for multi-task learning.
a) Diffusion Policy (DP): We use a U-Net Diffusion Policy [3] with a pre-trained SigLIP2 [20] visual encoder for single-task evaluation on MimicGen and AlohaGym. We apply PAIR ray conditioning to the visual tokens as described in Sec. III-C. We retain SigLIP2’s multi-head attention pooling (MAP) projection both for proprioceptive alignment and for aggregating the token representations into a global visual feature passed to the policy head.
b) SmolVLA: For multi-task learning, we fine-tune SmolVLA [27], a pre-trained vision-language-action model. Since its SigLIP vision encoder does not retain the original MAP projection head, we initialize it from scratch. SmolVLA is trained as a language-conditioned multi-task policy and evaluated across all four LIBERO task suites.
| Name | Value | Name | Value |
| Training | Training | ||
| Train steps | 200K | Train steps | 160K |
| Learning Rate | Learning Rate | ||
| BC batch size | 64 | BC batch size | 64 |
| PAIR batch size | 192 | PAIR batch size | 192 |
| 0.05 | 0.05 | ||
| 0.02 | 0.05 | ||
| 0.025 | 0.025 | ||
| 0.05 | 0.05 | ||
| Pre-trained MAP | True | Pre-trained MAP | False |
| 1 | 1 | ||
| 32 | 50 | ||
| (a) Diffusion Policy. | (b) SmolVLA. |
Table 1. Hyperparameters for Diffusion Policy and SmolVLA.
Training Objectives and View Settings
All methods share the same backbone, optimizer, and schedule, and differ only in the auxiliary objective added to the BC loss. The policy always consumes a single camera view; single-view and multi-view settings differ only in how the remaining views are used during training. Critically, proprioception features used to align the vision features in PAIR is provided as extra input modality for all cases.
a) BC: Standard behavior cloning without auxiliary loss. In the multi-view setting, the input view is randomly pooled from the candidate views.
b) TCN: The TCN [23] objective as the auxiliary loss. We follow the original implementation for both the single-view and multi-view settings.
c) PAIR: Proprioceptive alignment loss in Eq. (12) as the auxiliary loss. In the multi-view setting, the target view is sampled from the set of candidate camera views to provide cross-view learning signals.
Training/Evaluation Protocol
Diffusion Policies are trained for 200k steps, with five checkpoints saved at 10k-step intervals from 160k to 200k steps. SmolVLA models are trained for 160k steps, with five
| Env. | # Robots | # Cams | View | Multi-task | Horizon | ||||
| MimicGen | 1 | 2 | Side | No | 800/800/1000 | ||||
| ALOHAGym | 2 | 2 | Front | No | 800 | ||||
| LIBERO | 1 | 1 | Front | Yes | 600 | ||||
| Real-World | 1 | 2 | Front | Yes | 1000 | ||||
| Views | Auxiliary Objective | Square D1 w/o Ray | Square D1 w/ Ray | 3-Stack D1 w/o Ray | 3-Stack D1 w/ Ray | 3-Pc Assembly D1 w/o Ray | 3-Pc Assembly D1 w/ Ray | ALOHA Transfer w/o Ray | ALOHA Transfer w/ Ray |
| Single | None (BC) | 11.7 ± 1.2 | 13.4 ± 2.1 | 20.1 ± 1.1 | 34.1 ± 0.9 | 10.2 ± 1.3 | 18.1 ± 2.1 | 25.3 ± 1.6 | 35.2 ± 2.9 |
| Single | Temporal alignment (TCN) | 11.7 ± 1.5 | 16.3 ± 4.5 | 19.0 ± 2.0 | 39.0 ± 2.6 | 12.7 ± 2.1 | 18.3 ± 2.5 | 27.3 ± 1.5 | 35.3 ± 3.5 |
| Single | Proprio alignment (PAIR) | 20.7 ± 2.8 | 24.4 ± 2.9 | 36.3 ± 2.5 | 45.1 ± 3.7 | 20.1 ± 3.5 | 29.1 ± 1.8 | 30.0 ± 0.7 | 42.3 ± 2.1 |
| Multi | None (BC) | 26.9 ± 4.6 | 24.1 ± 1.1 | 44.1 ± 1.8 | 49.6 ± 2.6 | 17.3 ± 1.5 | 27.1 ± 1.8 | 55.7 ± 2.1 | 66.3 ± 2.9 |
| Multi | Temporal alignment (TCN) | 28.0 ± 2.0 | 32.3 ± 2.5 | 47.3 ± 2.1 | 50.0 ± 3.0 | 17.7 ± 2.1 | 25.3 ± 2.1 | 58.7 ± 3.2 | 70.0 ± 5.3 |
| Multi | Proprio alignment (PAIR) | 34.7 ± 3.0 | 38.5 ± 3.4 | 48.9 ± 3.7 | 53.9 ± 2.8 | 23.2 ± 3.4 | 30.5 ± 3.4 | 51.0 ± 2.0 | 74.0 ± 2.6 |
Table 2. Comparison of environments, including robot and camera counts, viewpoint, multi-task setting, and task horizon. MimicGen uses separate task horizons for three tasks.
| Auxiliary Objective | Object w/o Ray | Object w/ Ray | Goal w/o Ray | Goal w/ Ray | Spatial w/o Ray | Spatial w/ Ray | Long w/o Ray | Long w/ Ray | Average w/o Ray | Average w/ Ray |
| None (BC) | 94.6 ± 0.3 | 92.9 ± 0.6 | 90.4 ± 0.6 | 89.2 ± 0.5 | 82.8 ± 0.9 | 84.3 ± 1.1 | 57.9 ± 1.1 | 66.9 ± 0.9 | 81.4 | 83.3 |
| Time-contrastive (TCN) | 93.3 ± 0.4 | 94.1 ± 0.3 | 88.5 ± 0.6 | 87.3 ± 0.9 | 79.5 ± 0.5 | 84.3 ± 1.1 | 61.1 ± 1.1 | 70.4 ± 0.6 | 80.6 | 84.0 |
| Proprio-contrastive (PAIR) | 92.8 ± 0.5 | 95.7 ± 0.3 | 84.7 ± 0.5 | 90.8 ± 0.6 | 84.5 ± 0.6 | 86.8 ± 0.5 | 73.4 ± 3.1 | 82.1 ± 0.7 | 83.9 | 88.9 |
Table IV. Success rate (%) of SmolVLA on the four LIBERO task suites across 3 seeds. For each seed, the maximum success rates from 5 checkpoints is reported.
checkpoints saved at 10k-step intervals from 120k to 160k steps. For image augmentation, we randomly crop 256 × 256 images and their corresponding Plücker embeddings to 224 × 224 during training and apply a center crop during inference. For evaluation, single-task policies are rolled out 200 times for each model checkpoint. For LIBERO, each task is evaluated over 50 rollouts; with 10 tasks per task suite, this totals 500 rollouts per suite. The action horizon for open-loop rollout is 12 for all cases except AlohaGym, which uses 16. Table I summarizes the training and evaluation hyperparameters for both policy classes.
Simulation Results
Tables III and IV summarize the performance of PAIR across single-task and multi-task learning settings, respectively. In single-task learning, PAIR with ray conditioning improves the maximum success rate on average by 16.2%, 9.1%, and 9.5% points over BC without ray conditioning, ray-conditioned BC, and PAIR without ray conditioning, respectively. On LIBERO, PAIR with ray conditioning achieves the highest average success rate of 88.9%, outperforming ray-conditioned BC by 5.6% points. Compared with TCN, PAIR further improves the maximum success rate by 4.9% and 3.3% points with and without ray conditioning, respectively.
Real-World Experiments
We design three real-world tasks using a Franka Research 3 robot and two RealSense D435 cameras capturing images at 30 FPS, as illustrated in Figure 3. For each of the three tasks, 100 demonstrations are collected using VR-controlled teleoperation. For each rollout, cameras are re-positioned, and their poses are estimated using AprilTags [21]. Similar to LIBERO, we fine-tune the pre-trained SmolVLA on all three tasks jointly with language conditioning. We compare BC and PAIR with and without ray conditioning, using 30 rollouts per task in each setting. To ensure that evaluation is conducted from genuinely unseen viewpoints, after each camera calibration, we resample a camera pose if both its camera position and ground-plane look-at point are within 1
cm of those of any viewpoint sampled during data collection.

Figure 4. Success rates (%) of BC (red) and PAIR (blue), with and without ray conditioning, in three real-world tasks. Both subtask completion rates and final task success rates are reported.
Ablation Studies and Analysis
Importance of Ray Conditioning for Proprioceptive Alignment
We ablate the two camera-ray references used in PAIR to assess their individual contributions to proprioceptive alignment. Table V shows that removing either the end-effector-relative or base-relative ray modalities consistently lead to performance drop, which suggests that the both ray modalities provide complementary geometric information.
Effect of and
In Figure 5, we examine the effects of the quaternion-distance coefficient and the

Figure 5. Policy performance with varied and values for Diffusion Policy trained on Square D1 (blue) and SmolVLA trained on LIBERO (red) with the PAIR objective. The baselines report the performance of BC with ray conditioning.
alignment-loss coefficient on policy performance of DP trained on the multi-view Square D1 task and SmolVLA trained on LIBERO. PAIR improves performance across different choices of , with the best performances achieved when for both cases. For the alignment loss coefficient , 0.02 and 0.05 yield the best performance for DP and SmolVLA, respectively. Notably, DP is more sensitive to larger , where the alignment loss dominates the imitation learning loss.
Attention Activation Analysis
To assess whether PAIR improves attention to the robot end-effector and nearby task region, we measure the attention assigned to the visual token containing the ground-truth tool center point across SigLIP layers in Table VI. Ray conditioning consistently increases attention to the tool tip, with PAIR yielding the highest activation— that of BC without ray. The corresponding attention maps are visualized in Figure 6.
Computational Cost
Table VII compares the training cost of BC, TCN, and PAIR. While both TCN and PAIR introduce additional overhead relative to BC due to their auxiliary alignment objectives, PAIR is substantially more efficient than TCN, requiring 58.8% fewer FLOPs and 60% fewer decoded frames. In addition, ray input incurs only a small computational overhead, increasing FLOPs by 3.0% for BC and 3.8% for TCN and PAIR.
| Metric | BC | TCN | PAIR |
| FLOPs (, ) | 1.68 / 1.73 | 14.1 / 14.7 | 5.83 / 6.05 |
| Decoded Frames () | 64 | 640 | 256 |
Table VII. Computational and dataloading cost of training SmolVLA with BC, TCN, and PAIR. FLOPs are reported per forward pass for models without and with ray conditioning, respectively. Decoded Frames denotes the number of camera frames decoded during dataloading for each training batch. Training settings are provided in Table I.B.
Discussion
The empirical results show that proprioceptive alignment is not uniformly beneficial on its own, with mixed gains in the absence of ray conditioning. In contrast, when combined with rays, PAIR consistently outperforms the baseline methods across all evaluated settings, which indicates that camera-ray conditioning is a computationally cheap lever for unlocking the benefits of proprioceptive alignment.
In addition, the alignment with respect to proprioceptive signal consistently outperforms the temporal signal, while ray-conditioning benefits both methods. Notably, these gains hold across different robotic embodiments, single- and multi-task learning settings, and both Diffusion Policy and VLA policy classes, indicating that the benefits of PAIR generalize across diverse robotic learning settings.
In the multi-view learning setting, PAIR improves upon its single-view counterpart by leveraging observations from multiple viewpoints. The improvement reflects PAIR’s ability to exploit cross-view signals from observations of the same task scene for proprioceptive alignment. By effectively extracting these shared cross-view signals, PAIR shows the potential to scale to more diverse viewpoint settings and support cross-view representation learning.
Conclusion
In this work, we presented PAIR, a framework for aligning image-ray representations with robot proprioception to improve viewpoint robustness. While PAIR significantly improves generalization to unseen camera poses, there are several limitations. First, existing robotic datasets often contain demonstrations collected from a limited or skewed distribution of camera viewpoints, and the effectiveness of PAIR on such datasets may depend on the degree of viewpoint diversity present in the data. Second, collecting data under systematically diverse viewpoints with accurate camera calibration requires additional human effort and increases data-collection time. Finally, as shown in Table VII, the proprioceptive alignment objective incurs extra computational overhead during training. Future directions include building large-scale real-world datasets with systematically controlled viewpoint diversity, leveraging camera pose estimation for scalable image–ray pre-training on existing robotic datasets, and extending PAIR to cross-embodiment learning.
References
- [1]F. Lin, Y. Hu, P. Sheng, C. Wen, J. You, and Y. Gao, “Data scaling laws in imitation learning for robotic manipulation,” in International Conference on Learning Representations, Y. Yue, A. Garg, N. Peng, F. Sha, and R. Yu, Eds., vol. 2025, 2025, pp. 54 877–54 910.arxiv.org/abs/2410.18647
- [2]J.-C. Pang, N. Tang, K. Li, Y. Tang, X.-Q. Cai, Z.-Y. Zhang, G. Niu, M. Sugiyama, and Y. Yu, “Learning view-invariant world models for visual robotic manipulation,” in International Conference on Learning Representations, 2025.
- [3]C. Chi, S. Feng, Y. Du, Z. Xu, E. Cousineau, B. Burchfield, and S. Song, “Diffusion policy: Visuomotor policy learning via action diffusion,” in Proceedings of Robotics: Science and Systems (RSS), 2023.
- [4]K. Black, N. Brown, others, and Physical Intelligence, “π0.5: a vision-language-action model with open-world generalization,” in Conference on Robot Learning (CoRL), 2025.arxiv.org/abs/2504.16054
- [5]T. Jiang, J. Ji, X. Tan, J. Fang, A. Bhattad, V. Guzilini, and M. R. Walter, “Do you know where your camera is? View-invariant policy learning with camera conditioning,” in IEEE International Conference on Robotics and Automation (ICRA), 2026.
- [6]S.-W. Lee, X. Kang, B. Yang, and Y.-L. Kuo, “Class-Contrastive learning via action sequence supervision for robot manipulation,” in Conference on Robot Learning, ser. Proceedings of Machine Learning Research, vol. 305. PMLR, 2025, pp. 4743–4766.arxiv.org/abs/2508.01600
- [7]T. Kim, J. Lee, M. Koo, D. Kim, K. Lee, C. Kim, Y. Seo, and J. Shin, “Contrastive representation regularization for vision-language-action models,” in International Conference on Machine Learning (ICML), 2026.arxiv.org/abs/2510.01711
- [8]A. Radford, J. W. Kim, C. Hallacy, A. Ramesh, G. Goh, S. Agarwal, G. Sastry, A. Askell, P. Mishkin, J. Clark, G. Krueger, and I. Sutskever, “Learning transferable visual models from natural language supervision,” in International Conference on Machine Learning. PMLR, 2021, pp. 8748–8763.arxiv.org/abs/2103.00020
- [9]X. Zhai, B. Mustafa, A. Kolesnikov, and L. Beyer, “Sigmoid loss for language image pre-training,” in Proceedings of the IEEE/CVF International Conference on Computer Vision, 2023, pp. 11 975–11 986.arxiv.org/abs/2303.15343
- [10]Y. Xie, Y. Wang, S. Zhao, C.-E. Wu, M. Tomizuka, J. Xie, and H.-S. Fang, “Multi-camera view scaling for data-efficient robot imitation learning,” arXiv preprint arXiv:2604.00557, 2026.arxiv.org/abs/2604.00557
- [11]L. Y. Chen, C. Xu, K. Dharmarajan, M. Z. Irshad, R. Cheng, K. Keutzer, M. Tomizuka, Q. Vuong, and K. Goldberg, “Rovi-aug: Robot and viewpoint augmentation for cross-embodiment robot learning,” in Conference on Robot Learning, 2024.arxiv.org/abs/2409.03403
- [12]Y. Seo, J. Kim, S. James, K. Lee, J. Shin, and P. Abbeel, “Multi-view masked world models for visual robotic manipulation,” in Proceedings of the 40th International Conference on Machine Learning, ser. Proceedings of Machine Learning Research, vol. 202. PMLR, 2023, pp. 30 613–30 632.arxiv.org/abs/2302.02408
- [13]T. Zhang, H. Duan, H. Hao, Y. Qiao, J. Dai, and Z. Hou, “Grounding actions in camera space: Observation-centric vision-language-action policy,” in Proceedings of the AAAI Conference on Artificial Intelligence, vol. 40, no. 22, 2026, pp. 18 782–18 790.
- [14]H. He, X. Xu, Y. Guo, G. Wetzstein, B. Dai, H. Li, and C. Yang, “Cameractrl: Enabling camera control for video diffusion models,” in International Conference on Learning Representations (ICLR), 2025.arxiv.org/abs/2503.10592
- [15]X. Chen, W. Zhou, and Z. Cheng, “Wildrayzer: Self-supervised large view synthesis in dynamic environments,” 2026.arxiv.org/abs/2601.10716
- [16]J. Y. Zhang, A. Lin, M. Kumar, T.-H. Yang, D. Ramanan, and S. Tulsiani, “Cameras as rays: Pose estimation via ray diffusion,” in International Conference on Learning Representations (ICLR), 2024.arxiv.org/abs/2402.14817
- [17]M. Yin, J. Lu, W. Hu, W. Zhao, S. Ying, and K. Han, “Raype: Ray-space positional encoding for 3d-aware video generation,” arXiv preprint arXiv:2606.27345, 2026.
- [18]K. Xu, Z. Zhu, A. Chen, S. Zhao, Q. Huang, Y. Yang, H. Lu, R. Xiong, M. Tomizuka, and Y. Wang, “Seeing to act, prompting to specify: A bayesian factorization of vision language action policy,” arXiv preprint arXiv:2512.11218, 2025.arxiv.org/abs/2512.11218
- [19]Q. Wang, L. Chen, P. Sun, A. Taranovic, N. Freymuth, G. Li, W. Liao, C. Nagy, Y. Tan, T. Chen et al., “Rayvit: Ray-conditioned visual representations for viewpoint-robust imitation learning,” arXiv preprint arXiv:2607.29622, 2026.arxiv.org/abs/2607.29622
- [20]M. Tschannen, A. Gritsenko, X. Wang, M. F. Naeem, I. Al-abdulmohsin, N. Parthasarathy, T. Evans, L. Beyer, Y. Xia, B. Mustafa et al., “Siglip 2: Multilingual vision-language encoders with improved semantic understanding, localization, and dense features,” arXiv preprint arXiv:2502.14786, 2025.arxiv.org/abs/2502.14786
- [21]E. Olson, “Apriltag: A robust and flexible visual fiducial system,” in 2011 IEEE international conference on robotics and automation. IEEE, 2011, pp. 3400–3407.DOI
- [22]P. Khosla, P. Teterwak, C. Wang, A. Sarna, Y. Tian, P. Isola, A. Maschinot, C. Liu, and D. Krishnan, “Supervised contrastive learning,” Advances in neural information processing systems, vol. 33, pp. 18 661–18 673, 2020.arxiv.org/abs/2004.11362
- [23]P. Sermanet, C. Lynch, Y. T. J. Chen, J. Hsu, E. Jang, S. Schaal, S. Levine, and G. Brain, “Time-contrastive networks: Self-supervised learning from video,” 2018 IEEE International Conference on Robotics and Automation (ICRA), pp. 1134–1141, 2017.arxiv.org/abs/1704.06888
- [24]A. Mandlekar, S. Nasiriany, B. Wen, I. Akinola, Y. Narang, L. Fan, Y. Zhu, and D. Fox, “Mimicgen: A data generation system for scalable robot learning using human demonstrations,” arXiv preprint arXiv:2310.17596, 2023.
- [25]T. Z. Zhao, V. Kumar, S. Levine, and C. Finn, “Learning fine-grained bimanual manipulation with low-cost hardware,” arXiv preprint arXiv:2304.13705, 2023.arxiv.org/abs/2304.13705
- [26]B. Liu, Y. Zhu, C. Gao, Y. Feng, Q. Liu, Y. Zhu, and P. Stone, “Libero: Benchmarking knowledge transfer for lifelong robot learning,” Advances in Neural Information Processing Systems, vol. 36, pp. 44 776–44 791, 2023.arxiv.org/abs/2306.03310
- [27]M. Shukor, D. Aubakirova, F. Capuano, P. Kooijmans, S. Palma, A. Zouitine, M. Aractingi, C. Pascal, M. Russi, A. Marafioti et al., “Smolvla: A vision-language-action model for affordable and efficient robotics,” arXiv preprint arXiv:2506.01844, 2025.arxiv.org/abs/2506.01844


