WorksComputer Science
TacMind: Benchmarking and Modeling Tactile Interaction States for Robotic Manipulation
Abstract
Tactile sensing provides robots with direct observations of contact physics, local geometry, and interaction dynamics, making it a vital source of feedback for contact-rich manipulation. However, existing tactile manipulation studies are often developed and evaluated within specific task settings, making it difficult to systematically compare how tactile information contributes under different interaction conditions. In this work, we develop a benchmark in simulation comprising 128 tasks organized around three complementary evaluation regimes: Precise, Recognition, and Dynamic. The benchmark further includes controlled static and dynamic task variants and 12 perturbation settings for evaluating robustness under changes in visual and physical conditions. These settings motivate studying tactile information as a temporally evolving interaction representation rather than only as an instantaneous observation. We therefore propose an action-conditioned latent predictive world model that integrates tactile history and learns a tactile state through prediction of future tactile latents. Building on this representation, we design a model that conditions action generation on the learned tactile state and uses predicted future tactile information to refine candidate actions through predictive tactile reflection. Experiments on the proposed benchmark show consistent gains from incorporating tactile information across the evaluated regimes. Project Page: https://tacmind-bench.github.io/
Introduction
Vision provides rich global information for robotic manipulation, while tactile sensing provides direct measurements of local physical interaction, including contact geometry, deformation, slip, and object-dependent physical cues. Such information is useful not only when vision is ambiguous, but also for regulating contact and tracking interaction dynamics during manipulation Howe [16]; Hogan et al. [15]; Wu et al. [38]; Shi et al. [30]; Huang et al. [17]. Tactile sensing complements vision by directly observing local physical interactions, and recent visuo-tactile methods have shown benefits in contact-rich manipulation when visual observations are ambiguous or insufficient Luu et al. [24]; Chen et al. [4]; Zheng et al. [49]. However, tactile feedback is not only useful at the current timestep. During manipulation, it provides information about what is being contacted, how the interaction evolves, and how subsequent actions may affect that interaction. This motivates representing touch as an evolving interaction state rather than only as an instantaneous auxiliary observation.
Existing evaluation settings do not yet systematically characterize this broader role of touch. Many visuo-tactile methods are evaluated in real-world setups that are difficult to reproduce across laboratories, while recent simulation benchmarks have mainly studied capabilities such as high-precision manipulation and tactile object recognition Li et al. [21]; Luu et al. [24]; Chen et al. [4]. Precision tasks are often initialized from favorable contact states, such as with an object already grasped or placed near the target, and many contact-rich tasks use configurations that remain fixed during execution. These settings are useful for evaluating individual tactile skills, but provide limited evidence about how tactile information contributes as an interaction changes over time. In particular, it remains unclear how touch supports precise control, resolves visual ambiguity, adapts to changing interactions, and provides complementary cues under variations in visual observations or object properties.
To study these settings systematically, we introduce a capability-oriented tactile benchmark TACMIND-BENCH with 28 standard tasks and 100 extra tasks. The benchmark evaluates tactile sensing under three complementary conditions: (i) precise manipulation, where small contact errors affect task success; (ii) recognition under ambiguity, where task-relevant information is difficult to recover from vision alone; (iii) dynamic interaction, where the task state changes during execution together with controlled perturbations in lighting, camera configuration, and object properties. These settings place different demands on tactile representations. Recognition under visual ambiguity requires preserving local appearance, geometry, and object-dependent cues. Precise and dynamic manipulations additionally require information about contact deformation and its evolution over time. For control, the representation should also retain information that is useful for predicting how actions may change the physical interaction. We therefore consider three properties important for a general tactile state: semantically informative, temporally grounded, and action-relevant.
Learning such a state introduces several challenges. Vision-based tactile sensors provide complementary signals: tactile RGB observations capture local appearance and geometry, while marker deformation reflects interaction patterns such as shear, torsion, and sliding Luu et al. [24]. Tactile observations are also history-dependent, since similar instantaneous measurements may arise from different interaction trajectories. Finally, a representation useful for control should encode not only the current interaction but also information predictive of its subsequent evolution under action.
We address these challenges with TacMind-0, a vision-tactile-language-action framework that maintains a streaming tactile state throughout manipulation. Its tactile representation is learned by Tac-LeWM, a predictive latent model that integrates tactile appearance, deformation, and interaction history. Given a history of tactile observations and robot actions, Tac-LeWM predicts the latent representation of a future tactile chunk rather than reconstructing future tactile images. This joint-embedding objective encourages the current state to retain information predictive of subsequent interaction dynamics without requiring explicit reconstruction of potentially ambiguous sensory details. Our formulation is inspired by predictive latent world models Maes et al. [25] and differs from approaches that primarily use tactile prediction for explicit future rollout or observation forecasting Higuera et al. [14]; Zang et al. [45]; here, future touch is used as supervision for learning the tactile representation.
Tac-LeWM further incorporates spatial and temporal structure specific to vision-based tactile sensing. A canonical tactile rotary positional encoding maps mirrored fingertip sensors into a shared local frame and encodes spatial relations along multiple directions, helping represent deformation patterns that are not aligned with the image axes. A streaming causal tactile encoder incrementally updates the tactile state while reusing cached historical features, allowing long tactile histories to be incorporated efficiently during online control.
After the tactile representation is obtained, our TacMind-0 uses the resulting state in two ways. First, the tactile state directly conditions action generation, providing physical information that may be ambiguous or unavailable from vision. Second, we introduce tactile reflection: after generating an initial action proposal, the model predicts the tactile latent state associated with that proposal and uses the predicted interaction state to refine the action. This allows the policy to use both tactile evidence accumulated from previous interaction and information about the possible tactile consequence of a candidate action.
Our contributions are summarized as follows:
We introduce a capability-oriented tactile benchmark with 28 standard tasks and 100 extra tasks covering precise manipulation, tactile recognition under visual ambiguity, dynamic interaction, and robustness under 12 types of controlled perturbations.
We propose TacMind-0, a vision-tactile-language-action framework that maintains a streaming tactile state for action generation and refines candidate actions through predictive tactile reflection.
We develop Tac-LeWM, a predictive tactile representation model that integrates tactile appearance, marker deformation, and long interaction histories through latent predictive learning, canonical spatial encoding, and streaming causal modeling.
Related Work
Tactile Manipulation Benchmarks. Recent tactile simulators and benchmarks have enabled increasingly systematic evaluation of tactile perception and contact-rich manipulation Wang et al. [37]; Si & Yuan [33]; Akinola et al. [1]; Nguyen et al. [26]; Li et al. [21]; Luu et al. [24]; Chen et al. [4]. Existing suites cover precision manipulation, tactile perception, representation comparison, and deformable-object interaction, with recent work further considering robustness and transferable tactile features Jing et al. [20]; Higuera et al. [12]. These benchmarks provide important evaluation infrastructure, but are typically organized around particular task families, sensing settings, or interaction regimes. Our benchmark instead organizes tasks by the role of touch in manipulation, covering precise control, visual ambiguity, dynamic interaction, and controlled perturbations within a common framework.
Tactile-Aware Manipulation and Predictive Modeling. Tactile feedback has been incorporated into manipulation through learned representations, reactive control, and predictive modeling. Prior work has studied transferable tactile representations Guzey et al. [10]; Xu et al. [39]; Higuera et al. [12]; Zhao et al. [47]; Higuera et al. [13], visuo-tactile manipulation Yin et al. [42]; Yuan et al. [44]; Yu et al. [43]; Huang et al. [17], and high-frequency tactile correction for contact-rich policies Xue et al. [41]; Bi et al. [2]; Zhang et al. [46]; Zheng et al. [49]. A closely related line models future tactile interaction for planning or control: VT-WM learns action-conditioned visuo-tactile dynamics, TacForeSight uses predicted tactile dynamics as an anticipatory control signal, and HiTac-WAM predicts structured contact consequences for candidate-action evaluation Higuera et al. [14]; Zang et al. [45]; Xue et al. [40]. Recent approaches such as TouchWorld and VitaCWorld further use predicted visuo-tactile trajectories for closed-loop refinement or rollout-based reasoning Zhou et al. [50]; Huang et al. [18]. Our Tac-LeWM instead uses future tactile latents as a learning signal for the current tactile state, which is then used by TacMind-0 for tactile-conditioned action generation and predictive tactile reflection.
TacMind-Bench Design
TacMind-Bench organizes manipulation tasks according to the dominant role of tactile information in task completion, rather than by object category or manipulation primitive. It consists of a Standard suite with 28 tasks and an Extra suite with 100 tasks. Both suites cover three evaluation regimes: precise contact, recognition under ambiguity, and dynamic interaction. TacMind-Bench Standard contains a curated set of representative tasks used for both training and evaluation. The Extra suite extends these tasks with greater diversity in textures and language descriptions, together with additional manipulation tasks involving new objects. Figure 2 provides an overview of TacMind-Bench and its contact diversity. Complementing the three evaluation regimes, we annotate tasks with 11 contact behaviors using a multi-label scheme, since a single task may involve multiple behaviors throughout execution. Representative examples pair manipulation scenes with tactile visualizations from both fingers, illustrating the diverse contact patterns encountered across the benchmark. Table 1 compares TacMind-Bench with representative tactile manipulation benchmarks. Unlike prior benchmarks that primarily focus on particular manipulation skills or tactile objectives, TacMind-Bench explicitly organizes evaluation around different roles of tactile information throughout manipulation.
Figure 2. Overview of TacMind-Bench. The benchmark comprises 128 tasks (28 Standard and 100 Extra), 287 object assets, two tactile sensors (GSmini and Daimon), and 12 controlled perturbation settings covering illumination, camera viewpoint, and object scale. (a) Task composition and contact-behavior coverage. The inner ring shows the three evaluation regimes within Standard and the Extra suite; the outer ring and accompanying table summarize contact-behavior annotations. These annotations are non-exclusive: a task may involve multiple contact behaviors. (b)–(l) Representative tasks illustrating the 11 contact behaviors, each shown alongside tactile visualizations from the left (L) and right (R) fingers.
| Benchmark | Benchmark Objective | Tasks | E2E | SD. | Dyn. | Perturb. |
| ManiSkill-ViTac | Precision contact-rich manipulation | 3 | – | – | – | – |
| ManiFeel | Tactile-aware exploration and contact-rich manipulation | 9 | △ | △ | – | – |
| UniVTAC | Unified visuo-tactile manipulation | 8 | △ | – | – | – |
| Tabero | Gentle manipulation with tactile force regulation | 40 | ✓ | – | – | – |
| TacMind-Bench | Roles of touch throughout manipulation | 128 | ✓ | ✓ | ✓ | ✓ |
Table 1. Comparison with representative tactile manipulation benchmarks. ✓ denotes an explicit benchmark design axis, △ denotes partial coverage, and – denotes that the dimension is not explicitly evaluated. E2E denotes manipulation that includes object acquisition rather than only the terminal contact stage. SD. denotes State Disambiguation. Dyn. denotes externally driven source or target motion during execution. Perturb. denotes an orthogonal controlled robustness axis.
Precise Contact. Precise contact tasks evaluate insertion and mating without externally driven motion of the source object or target. Vision provides coarse spatial configuration, but local contact states and corrective actions can be difficult to infer visually. These tasks therefore assess how tactile feedback supports fine contact regulation. Many existing precision-oriented evaluations focus on terminal contact, often initializing the object in-hand or near the target. These settings isolate fine contact regulation but exclude acquisition, transport, and the transition into contact. TacMind-Bench retains these settings while adding tasks spanning pickup, transport, contact establishment, and final insertion or placement, enabling evaluation across a broader manipulation process.
Recognition. Recognition tasks require recovering physical information that is occluded, visually ambiguous, or difficult to observe, including object pose, size, surface texture, texture orientation, and boundary geometry. For example, a box partially occludes the rectangular peg, while the wrist view provides limited information about its in-hand pose. Hexagonal prisms with different side-surface textures are difficult to distinguish from distant front or top-down wrist views, even without explicit occlusion. While prior benchmarks include individual tactile-discrimination tasks, TacMind-Bench makes state disambiguation an explicit evaluation regime, covering partial occlusion, limited viewpoints, low visual contrast, and clutter. Some tactile-search tasks further require accumulating and retaining evidence across multiple contacts before identifying the target. These settings evaluate tactile cues beyond contact force or marker deformation alone.
Dynamic Interaction. Dynamic-interaction tasks extend precision manipulation with externally driven motion in three settings. In moving-target alignment, the object is already grasped while the target moves, isolating online alignment from acquisition. Static-source to moving-target tasks retain the complete pick-and-insert procedure while the target moves continuously. Moving-source to static-target tasks require acquiring an object from a moving source before manipulating it toward a stationary target. These settings distinguish motion affecting target alignment from motion affecting object acquisition. Where possible, dynamic tasks are paired with static counterparts using the same object, target, and manipulation primitive, enabling controlled comparisons between stationary and time-varying interactions.
Controlled Perturbations. Following common evaluation protocols, we vary object positions across trials. TacMind-Bench further introduces controlled visual and physical perturbations as an evaluation axis orthogonal to the three task regimes. Changes in illumination and camera viewpoint alter visual observations while leaving tactile sensing conditions unchanged, enabling evaluation of the complementary role of touch under visual variation. We also perturb object size. Compared with changes in object position or appearance, moderate scale variations introduce subtle visual differences while substantially altering contact geometry. These variations are particularly relevant to precision manipulation and tasks that use tactile feedback to regulate grasp force or contact.
Extra Suite. Details and additional benchmark settings are provided in the Appendix.
TacMind-0
Overview
TacMind-0 is a vision-tactile-language-action model that maintains a predictive tactile interaction state throughout manipulation. At time , the robot receives visual observations , a language instruction , and a tactile observation . We denote the tactile history as , where is the tactile context length. The policy predicts an action chunk , where is the action horizon. Tac-LeWM encodes into a streaming tactile state , learned through action-conditioned prediction of future tactile latents. TacMind-0 combines this state with vision and language to generate , optionally refined through predictive tactile reflection:
where is the Tac-LeWM predictor, and denotes the predicted future tactile latent.
Learning a Predictive Tactile State
Tac-LeWM adapts the joint-embedding predictive formulation of LeWM Maes et al. [25] to vision-based tactile sensing, learning from future embeddings rather than reconstructing pixels. We modify the input representation and spatio-temporal encoding to account for tactile structure.
Tactile Input Representation. For each finger , we overlay the marker deformation representation onto the tactile RGB image to obtain , where denotes the overlay operation. This combines local appearance and geometry with deformation cues such as shear, torsion, and sliding. The left and right fingertip observations are arranged into and patchified into tactile encoder tokens.
Canonical Tactile Rotary (CTR) Positional Embedding. Image positional encodings typically associate tokens with their locations on the image raster, while axial RoPE encodes horizontal and vertical offsets separately Heo et al. [11]; ); Liu et al. [23]. For tactile observations, however, the spatial structure of contact-induced deformation is not necessarily aligned with these axes. As illustrated in Figure 3(a), shear, normal loading, and partial slip produce different marker-displacement fields whose directions can vary across the contact region. For example, normal loading can produce outward displacement, whereas partial slip produces nonuniform local motion. Interpreting these patterns involves relating deformation cues across spatial locations, rather than considering each marker displacement independently. This motivates a positional representation that explicitly encodes spatial relationships along multiple directions. The tactile input introduces an additional geometric consideration. Global raster coordinates depend on how the fingertip images are arranged in the input mosaic, while the left and right sensors are mounted in mirrored orientations, as shown in Figure 3(b). We therefore use local fingertip coordinates rather than global mosaic positions. For a token from finger at time and local patch coordinate , we use and retain sensor identity through a separate learnable finger embedding. The right fingertip is mapped into a shared canonical frame using , where is the local patch-grid width. This mapping changes only positional coordinates, leaving the input image unchanged. Within this canonical frame, we project local coordinates along eight directions as , where and . We encode these projections and the temporal index through rotary transformations of the query and key features in tactile causal attention. For two tokens separated by , the corresponding spatial rotary phase differences depend on . Each projection thus provides a relative-position cue along a particular orientation, including directions oblique to the image axes. The tactile features carry the observed deformation information, while CTR supplies a shared, direction-aware positional structure for learning spatial relationships among these features.
Figure 3. Overview of Tac-LeWM. (a–b) Marker-motion directions vary with contact conditions, while the left and right fingertip sensors have mirrored spatial layouts. These properties motivate tactile-specific spatial encoding that accounts for directional deformation and cross-fingertip symmetry. (c) Tac-LeWM learns a predictive tactile state by combining a streaming tactile encoder with an action-conditioned latent predictor trained to predict future tactile embeddings. (d) The streaming encoder reuses historical features through cached keys and values, enabling efficient online updates. (e) Causal attention integrates preceding tactile context while preventing access to future observations.
Streaming Causal Encoding. Tactile observations are history-dependent: similar instantaneous contact patterns may result from different preceding interactions. We therefore maintain the tactile state causally over time. Let denote the tactile token features at time , where contains cached keys and values from preceding tactile observations. Current tactile tokens may attend to current and historical observations, while future tactile observations are excluded from the online state. We use the encoder’s global tactile token as the interaction state, denoted by . At inference, cached historical keys and values are reused when a new tactile observation arrives, allowing the tactile state to be updated incrementally without repeatedly encoding the complete history. The KV cache therefore serves as an efficient implementation of temporal context, while is the latent state consumed by the predictor and action policy.
Action-Conditioned Latent Prediction. For each time , we define a future tactile chunk as , where is the prediction offset and is the prediction horizon. The corresponding latent targets are . As in LeWM, the tactile encoder and predictor are optimized jointly, and the future tactile embeddings are produced by the same tactile encoder rather than by a separate target network. Given the intervening actions , the predictor estimates , where is an auxiliary visual feature used during Tac-LeWM pretraining and fine-tuning.
A frozen DINOv3 Siméoni et al. [34] encoder extracts from the current scene observation to help disambiguate future tactile outcomes. Visual conditioning is randomly dropped during training and omitted at inference, while is computed only from tactile observations up to time .
Predictive Learning Objective. Tac-LeWM predicts the future tactile embeddings rather than reconstructing future tactile images. The prediction objective is
Following LeWM Maes et al. [25], we additionally apply SIGReg to reduce representation collapse. The complete Tac-LeWM objective is
where controls the regularization strength. Both and are optimized under this objective.
Tactile-Conditioned Action Generation
VTLA Architecture. TacMind-0 builds on a pretrained vision-language-action policy with a Gemma 3 vision-language backbone and a -style flow-matching action expert. Tac-LeWM provides the streaming tactile state used by the policy, as shown in Figure 1 (c). Given the visual observation , language instruction , and tactile state , we form the multimodal conditioning . The tactile features are projected to the hidden dimension of the action expert and inserted as tactile conditioning tokens, while temporal tactile aggregation remains within the Tac-LeWM encoder. The policy then generates an initial action chunk as . Each action contains a 7-DoF arm command and one continuous gripper command. The action expert follows the standard conditional flow-matching formulation. Given a demonstrated action chunk and Gaussian noise , we interpolate them at flow time as , with target velocity . The action expert is trained using
Figure 1. Overview of TacMind-Bench and TacMind-0. (a) TacMind-Bench comprises 28 standard and 100 additional tasks spanning precise contact, recognition under ambiguity, and dynamic interaction, together with controlled perturbations for robustness evaluation. (c) TacMind-0 is our VTLA model that maintains a streaming tactile state for action generation. (d) Predictive tactile reflection leverages Tac-LeWM to estimate future tactile states conditioned on candidate actions and uses them to refine action generation. (e) Tac-LeWM is pretrained on large-scale multi-sensor tactile manipulation demonstrations collected from both simulation and the real world. (b) TacMind-0 achieves competitive performance on TacMind-Bench Standard.
At inference, integrating the velocity field from noise produces the initial action proposal .
Predictive Tactile Reflection. The initial action policy uses tactile evidence accumulated up to the current time, but it does not explicitly condition action generation on the tactile state predicted under its own candidate action. We therefore introduce predictive tactile reflection. Given the initial proposal , Tac-LeWM predicts the corresponding future tactile latent chunk as . The auxiliary visual input used during Tac-LeWM pretraining is omitted here. We summarize the predicted tactile chunk as and use it directly as an additional action-conditioning signal rather than decoding it into tactile images. We implement action refinement through flow reversal steering. Let denote the flow transformation from latent noise to an action chunk. Starting from the initial action , we reverse the base flow to obtain . We then augment the policy conditioning with the predicted tactile state, , where injects the predicted tactile feature into the flow-matching layers through adaptive normalization. The refined action is generated by . The reverse pass uses the original conditioning , whereas the second forward pass additionally receives the predicted tactile state. The reverse–forward procedure therefore changes the conditioning under which the action is generated rather than simply reconstructing the initial proposal. The two tactile signals play distinct roles: (i) summarizes the observed interaction history and (ii) represents the tactile state predicted under the candidate action and conditions action refinement.
Training and Online Execution
Tac-LeWM Pretraining and Fine-tuning. For pretraining and fine-tuning, we jointly optimize the tactile encoder and predictor using . We also use action representation regularization in pretraining. The auxiliary DINOv3 Siméoni et al. [34] feature is used only during this stage and is randomly dropped during training. Since the training datasets differ in their action representations, we do not expect a unified preprocessing pipeline across all datasets. Instead, we use lightweight MLP projections to map actions from different datasets into a shared representation space, followed by a shared action encoder that extracts action features used to condition the predictor. More details about pretraining are in the Appendix.
VLA Post-Training. We perform two-stage post-training. First, the fine-tuned Tac-LeWM’s encoder is frozen and used to provide to TacMind-0. The action expert is optimized using . Second, we freeze the tactile encoder, VLM backbone, and action expert; only the predictor of Tac-LeWM and the AdaRMSNorm layer are trainable using action loss and Tac-LeWM training loss.
Inference. At inference, future and history tactile observations are not required. The tactile encoder incrementally updates its KV cache as new tactile observations arrive. TacMind-0 first generates conditioned on . When tactile reflection is enabled, Tac-LeWM predicts the future tactile latent from , and a second conditioned flow pass produces the refined action .
Experiments
Setup
We conduct experiments to evaluate our method and baseline models on the 28 standard tasks of TacMind-Bench using the GelSight Mini tactile sensor. Prior works primarily evaluate policies under in-domain settings, since tactile observations are invariant to external ambient visual variations and depend directly on local object-sensor interactions. To provide a systematic evaluation of visuotactile robustness, each task is evaluated over episodes, covering one in-domain (InD) setting and 12 out-of-domain (OOD) perturbation settings across lighting conditions, camera viewpoints, and object scales. Because the 28 tasks are distributed across the three regimes as 6 Precise, 11 Recognition, and 11 Dynamic tasks, we report the regime-level success rates alongside the task-weighted overall average (Avg), separately for the InD and OOD settings.
Implementation Details. TacMind-0 is initialized from the pretrained OpenDM0.5 model, which adopts Gemma 3 as the VLM backbone and a -style action expert. Tac-LeWM is pretrained on multiple tactile manipulation datasets collected from diverse sources; further details of the pre-training procedure are provided in the Appendix. All models are trained using 100 demonstrations per task on 8 NVIDIA A100 GPUs with an action chunk size of 32. For ACT+Tactile and Diffusion Policy, we train a separate specialist model for each individual task, whereas the VLA and VTLA generalist models are jointly trained across all 28 tasks with a single set of weights. For ACT+Tactile, we follow the model architecture of UniVTAC Chen et al. [4]. For Diffusion Policy Chi et al. [5], we adopt the implementation provided by LeRobot Cadene et al. [3]. For OpenPI0.5 Intelligence et al. [19], OpenDM0.5 Dexmal Team [36], and N0-VTLA Team et al. [36], we use their respective official codebases. Because optimizing a single-task specialist policy avoids cross-task interference and multi-task capacity bottlenecks, single-task and multi-task policies are grouped separately in Table 2 for fair comparison.
| Model | Precision InD | Precision Light | Precision Cam | Precision Size | Recognition InD | Recognition Light | Recognition Cam | Recognition Size | Dynamic InD | Dynamic Light | Dynamic Cam | Dynamic Size | Avg InD | Avg OOD |
| Single-Task Specialist Policies (trained separately per task) | ||||||||||||||
| ACT+Tactile | 30.0 | 26.0 | 2.5 | 2.5 | 69.1 | 48.9 | 31.7 | 13.9 | 15.9 | 10.5 | 1.8 | 1.6 | 39.8 | 16.4 |
| DP | 23.3 | 23.3 | 0.0 | 2.9 | 27.7 | 23.2 | 6.1 | 5.0 | 13.2 | 9.8 | 2.1 | 1.1 | 21.1 | 8.1 |
| Multi-Task Generalist Policies (jointly trained across all 28 tasks) | ||||||||||||||
| OpenPI0.5 | 5.0 | 0.0 | 0.0 | 0.0 | 30.0 | 24.2 | 1.2 | 15.9 | 0.5 | 0.2 | 0.0 | 0.2 | 13.0 | 5.5 |
| OpenDM0.5 | 20.0 | 17.3 | 2.5 | 6.3 | 38.6 | 28.1 | 20.1 | 16.9 | 8.2 | 4.8 | 1.5 | 1.8 | 22.7 | 11.4 |
| N0-VTLA | 25.8 | 15.0 | 6.8 | 4.2 | 45.9 | 18.4 | 8.9 | 12.0 | 5.9 | 3.1 | 1.7 | 1.4 | 25.9 | 7.8 |
| TacMind-0 (Ours) | 40.0 | 35.0 | 14.4 | 17.9 | 50.0 | 43.6 | 35.0 | 28.1 | 10.5 | 7.4 | 3.3 | 4.1 | 32.3 | 20.7 |
Table 2. Performance on TacMind-Bench Standard (GelSight Mini). OOD performance is categorized into lighting, camera, and size perturbations.
Evaluation Results
TacMind-Bench Results. The quantitative results across both InD and OOD settings of TacMind-Bench are reported in Table 2. Among multi-task generalist policies, TacMind-0 consistently achieves state-of-the-art performance across all three evaluation regimes, reaching 32.3% InD and 20.7% OOD overall success rates. Notably, TacMind-0 also surpasses the single-task ACT+Tactile specialist on the Precise regime and overall OOD robustness. Qualitative comparisons are illustrated in Figure 4, with additional visual rollouts provided in the Appendix.

Figure 4. Comparison between N0-VTLA and TacMind-0 on the Dynamic suite of TacMind-Bench.
Ablation Study
In this section, we conduct ablation studies on our proposed VTLA model, TacMind-0, and our streaming tactile encoder, Tac-LeWM. Additional component-wise ablations of Tac-LeWM are provided in the Appendix.
TacMind-0. We ablate the key components of TacMind-0 in Table 3. “Tactile” denotes the incorporation of streaming tactile state representations, while “Reflection” refers to our proposed predictive tactile reflection mechanism. Incorporating tactile features substantially improves performance on both the Precise and Recognition regimes, demonstrating the benefit of explicitly modeling tactile interaction states. Further introducing tactile reflection yields consistent gains across all regimes, with particularly notable improvements under OOD perturbations, highlighting its effectiveness in enhancing robustness and generalization.
| Tactile | Reflection | Precise | Recog. | Dynamic | Avg | ||||
| InD | OOD | InD | OOD | InD | OOD | InD | OOD | ||
| 20.0 | 8.7 | 38.6 | 21.7 | 8.2 | 2.7 | 22.7 | 11.4 | ||
| 35.8 | 12.4 | 47.3 | 30.2 | 9.1 | 3.6 | 29.8 | 15.9 | ||
| 40.0 | 22.4 | 50.0 | 35.6 | 10.5 | 4.9 | 32.3 | 20.7 |
Table 3. Ablation study of TacMind-0. We evaluate the contributions of streaming tactile features and predictive tactile reflection across different evaluation regimes and distributions.
Tactile Representation Quality. We assess Tac-LeWM through tactile reconstruction and linear probing in Figure 5(a–Figure 5c). Pretraining improves current-frame reconstruction, increasing PSNR from 31.83 to 34.42 dB and SSIM from 0.875 to 0.914. Linear probes achieve Pearson correlations of 0.8981, 0.9053, 0.8976, and 0.7968 for proprioception, gripper state, actions, and tactile pad energy, respectively. These results indicate that the tactile representations retain linearly decodable information about robot states, actions, and tactile responses.

Figure 5. Analysis of Tac-LeWM. (a) Effect of pretraining on current-frame tactile reconstruction. (b–c) Linear probing of proprioception, gripper state, actions, and tactile pad energy, evaluated using MSE () and Pearson correlation (). (d) Tactile encoder inference latency across batch sizes, with and without streaming. (e) Future tactile reconstruction quality with different cache lengths. (f) Future tactile reconstruction quality under clean, shuffled, and polluted memory.
Temporal Context and Streaming. We further examine the role of tactile history and the efficiency of streaming inference in Figure 5(d–Figure 5f). Increasing the cache length from to improves future tactile reconstruction PSNR from 27.34 to 28.98 dB. Shuffling the memory reduces PSNR to 27.90 dB, while memory pollution lowers it to 24.18 dB, suggesting that prediction benefits from both the temporal ordering and relevance of historical context. Streaming reduces the cost of maintaining this context by reusing cached historical features. Encoder inference latency decreases from 18.36 to 10.33 ms at batch size 1 and from 129.79 to 29.74 ms at batch size 256.
Conclusion
In this work, we introduce TacMind-Bench, covering precise contact, recognition under ambiguity, dynamic interaction, and controlled perturbations. We further propose Tac-LeWM and TacMind-0 for predictive tactile representation learning and tactile-conditioned action generation. Experiments demonstrate improved robustness and manipulation performance, while dynamic interaction remains challenging.
AI Use Statement
In this work, we used generative AI tools to assist with polishing and improving the clarity of the manuscript, debugging code, and supporting data collection for our benchmark. We did not use generative AI tools to independently formulate the main research claims or conclusions. All AI-assisted manuscript edits were reviewed by the authors. AI-assisted code debugging was followed by manual inspection and testing to verify correctness. For benchmark data collection, AI-generated or AI-assisted outputs were reviewed and filtered by the authors according to our data collection protocol and quality-control criteria. We take full responsibility for the final content of this work, including all text, claims, code, and benchmark data produced with the aid of generative AI.
Ethics Statement
Our benchmark is built in the simulation. The collected data will be released, but there is no human subjects data recorded.
Reproducibility Statement
The design of our benchmark is introduced in Section 3. The framework and training/inference setup of our model are introduced in Section 4. The details of the benchmark and experiment setup are stated in Section 5 and the appendix. The data and code of the benchmark and the model will be made public.
Benchmark Data Collection
Agentic manipulation data generation.
Motivation. Agentic data generation is particularly suitable for data collection in our tasks, especially for the Precise suite. We initially followed prior work Liu et al. [22] and used a SpaceMouse to collect data. However, due to our strict success criteria and the limited precision of manipulation teleoperation, this approach proved difficult and inefficient, often resulting in low-quality action trajectories. With our carefully designed agentic data-generation pipeline, data collection becomes substantially more efficient, produces higher-quality trajectories, and scales more easily.
Settings. To collect scalable, high-quality manipulation data, we develop an agentic harness based on Codex to generate and debug agentic action experts on the Isaac Sim platform. We formulate agentic manipulation as a controller program with an explicit closed-loop structure, which makes decisions based on observations, robot states, and tactile information. The control process separates free-space motion from contact-rich or tactile-aware motion. Motion planning handles approaching and transporting objects in free space, while local closed-loop control executes grasping, alignment, insertion, pressing, fitting, and fastening. Tactile or geometric conditions, rather than fixed timing, trigger transitions between stages.
Environment & Sensors. We use Isaac Sim 5.1 NVIDIA as the backend for realistic simulation and CuRobo Sundaralingam et al. [35] for manipulation trajectory planning. We use two recently popular visual-tactile sensors, i.e., GelSight Mini and DM-Tac. GelSight Mini and DM-Tac are vision-based optical tactile sensors that infer contact information from camera observations of elastomer deformation. GelSight Mini provides high-resolution sensing of contact geometry and texture, while DM-Tac emphasizes dense and dynamic tactile perception, including contact shape, pressure, tangential forces, and slip. The robot is a Franka arm with two grippers that are equipped with these two sensors. Because the two grippers have different geometries, we tuned the trajectories and data collection expert separately.
Success criteria. A task is considered successful only when the object reaches a stable seated state, such as full insertion, complete press fitting, or completion of the required amount of rotation. In discrimination and search tasks, decisions must be based on tactile observations collected through interaction. To ensure high-quality data, the success criteria used for data collection are stricter than those used for evaluation.
For agentic data collection, we find that trajectories judged successful during collection do not always succeed when replayed under the same scene configuration. This is because the collection process effectively performs closed-loop manipulation: rather than relying entirely on a predefined motion pattern, the agent dynamically replans and adjusts its actions in response to environmental changes. Meanwhile, the simulation environment includes some nondeterminism, and discrepancies can occur even when the same random seed is used. This issue is particularly pronounced for fine-grained manipulation tasks, where even small deviations can lead to task failure. In contrast, replay simply re-executes the recorded trajectory and cannot perform such dynamic corrections. Therefore, we consider only trajectories that also succeed during replay to be valid training trajectories.
Skills. Skills can extend the ability of the data collection agent, thus leading to a more accurate, efficient data collection pipeline. Given the pipeline stages and the need to construct a benchmark, we develop skills for simulation assets, task generation, environment building, and data generation, as shown in Table 4.
| Skill | Functional Scope |
expert-trajectory | Development and refinement of expert policies for our benchmark, including tactile grasping, cuRobo-based free-space motion planning, insertion and placement, success-condition specification, and trajectory-smoothness optimization. |
dynamic-collect | Collection and same-environment replay-gated evaluation of dynamic conveyor-belt tasks tac_dynamic, together with systematic diagnosis of Kit execution hangs and zero-episode dataset failures. |
reset-randomization | Design and validation of reset-time perturbations for the benchmark suite, including pose_range, object permutation, and layout_xy, with consistency verification between JSON specifications and runtime perturbation bounds. |
generate-tactile-task | Generation and validation of collectible tactile manipulation environments from natural-language task specifications. |
object-expert-repair | Diagnosis and remediation of geometry and collision issues, insertion failures, false-positive success conditions, and expert trajectory collection consistent with teleoperated behaviour. |
Table 4. Summary of data collection skills.
Built-in CLI. The agent needs to interact with the simulation dynamically and understand the physical problems during rollout. To improve the efficiency of data collection and help the agents generate trajectories and repair problems when generating the environment and rolling it out, we develop a set of built-in functions to let the agent read the relative simulation debug logs and the vision of the simulation, as well as automatically process the collected data, as shown in Table 5.
| Function | Role in the Data Processing Pipeline |
collect | Performs data collection, validation, and persistence while exposing diagnostic information and runtime task traces to support agent-driven analysis of collection failures and their underlying patterns. |
expert | Dispatches the task-specific expert procedure to generate demonstrations for the target manipulation task. |
replay_gate | Replays collected trajectories in the corresponding environment and evaluates execution correctness as a quality-control gate for dataset admission. |
capture | Captures images from arbitrary tasks in Isaac Sim for visual inspection, debugging, and downstream multimodal processing. |
validate_dataset | Inspects dataset metadata and verifies individual episodes against the predefined episode schema to ensure structural consistency. |
convert_dataset | Converts datasets between formats, processes action sequences, and removes redundant or unnecessary frames to produce training-ready datasets. |
Table 5. Built-in Functions and Their Roles in the Data Processing Pipeline.
Human in the loop. We found that constructing a simulation environment, performing tactile-oriented tasks, and creating corresponding trajectory-generation experts solely through the agent are very difficult. Because of limitations in 3D spatial reasoning, physical information understanding, and rollout video understanding, agents often cannot identify failure reasons and automatically repair problems, especially for tasks that require precise, contact-rich manipulation. To ensure the quality of simulation assets, environments, and training data, as well as fair evaluation, we use a human-in-the-loop approach across the entire data collection pipeline.
Touch in the collection. In addition to synchronously recording tactile RGB images, marker information, and other tactile signals during data collection, tactile sensing is also actively used throughout the agentic data collection process. During grasping or contact verification, we use the indentation depth measured by the tactile sensors as input to an adaptive gripper controller, which adjusts the gripper opening accordingly to ensure that the applied grasping force remains within the desired range. For sorting tasks, we leverage privileged information while using rule-based procedures to mimic the stochastic process by which humans identify objects through touch. For search tasks, during data collection, we determine whether the target object has been found based on whether the tactile sensor registers a contact response.
Randomness in data collection. During data collection, especially for the Recognition suite, we not only randomize the position in a small set of object positions, but we also change the object’s position and randomize the order of the searching process to simulate the behavior, e.g., a human uses the head to search and recognize the object he needs by touch.
Action Representation. We use 8D absolute joint actions with a continuous gripper. During data collection, we used EE for replay; the success rate is very low because IK errors reduce data-collection efficiency and data quality.
Other Modalities. Since our benchmark is built on IsaacSim, it can export depth, semantic masks, object tracking, object pose, and other information. We have also reserved the export interface.
Data Collection Experience & Discussion
Length of the trajectory. For demonstrations of the same task, trajectories should minimize redundant motion and unnecessary corrections. Excessive trajectory length increases the effective task horizon and may exacerbate compounding errors in imitation learning Ross et al. [28]; Shi et al. [31]; Zhao et al. [48]. Prior work also identifies path length and trajectory consistency as relevant indicators of demonstration quality Sakr et al. [29]. Therefore, operators should execute tasks efficiently while preserving motions necessary for safety and robustness.
Speed of the trajectory. Demonstrations should exhibit smooth and temporally consistent motion rather than strictly constant velocity. Abrupt acceleration, frequent stops, and oscillatory corrections introduce unnecessary action variability and may hinder policy learning. Motion smoothness, often characterized by jerk, is associated with natural, high-quality motion generation Flash & Hogan [9]; Sakr et al. [29]. Operators should therefore maintain stable motion in free space and smoothly decelerate during contact or precision-sensitive phases.
Consistency of the trajectories. Demonstrations of the same task should follow consistent motion strategies and execution patterns. Large inter-demonstration variations in trajectory geometry and motion characteristics can increase ambiguity in the learned action distribution. Prior work shows that consistency across demonstrations is strongly associated with both task success and generalization performance Sakr et al. [29].
Predictability of the actions. Operators should avoid arbitrary variations in action selection under similar task states. Consistent state-action mappings reduce unnecessary action ambiguity and improve the predictability of demonstrations, which has been identified as an important indicator of robot data Shi et al. [32].
Task Suites and Benchmark Details
The Standard suite comprises 28 tasks spanning the three evaluation regimes (Precision, Recognition, and Dynamic). All Standard tasks share the same observation interface, instantiated with two gripper embodiments (GelSight Mini and Daimon), and each provides 100 expert demonstrations collected with the agentic pipeline described above. Figure 6 shows typical agent-view and tactile observations at a contact-rich moment, and the natural-language instruction of every task is listed in Table 6; both group the tasks by evaluation regime.

Figure 6. The 28 standard tasks of TacMind-Bench (GelSight Mini).
| Regime | Task | Language Instruction |
| Precise | 00 | Pick up the tube and insert it into the black rubber hole base. |
| 01 | Pick up the key and insert it into the blue plastic lock. | |
| 02 | Pick up the connector and insert it into the orange port. | |
| 03 | Pick up the gear and insert it into the white painted base. | |
| 04 | Insert the three-prong power plug into the power socket. | |
| 05 | Insert the bulb into the bulb socket. | |
| Recog | 00 | Pick the larger M16 nut instead of the M8 nut from the covered tray, judging size by touch. |
| 01 | Retrieve the dimpled golf ball rather than the smooth ball from the covered bin using touch. | |
| 02 | Explore the covered compartment tray by touch, find the service token with the sawtooth edge, and place it on the rubber service mat. | |
| 03 | Among the caps in the covered recycling tray, find the cap with horizontal ribs—not vertical ribs—and place it in the recycle cup. | |
| 04 | Explore the two balls by touch, identify the dimpled golf ball rather than the smooth table-tennis ball, and place it in the left cup. | |
| 05 | Among the six equal-size cylinders in the open parts box, find the one with the most circumferential threads and place it on the inspection tray. | |
| 06 | Among the tokens in the covered compartment tray, find the one with the wave-shaped edge and place it on the rubber service mat. | |
| 07 | Among the three caps on the table, use touch to find the cap with horizontal ribs and place it in the bowl. | |
| 08 | Pick up the horizontal rectangular peg from the box bottom, lift it vertically, and insert it. | |
| 09 | Find and retrieve the target object from the box using tactile feedback. | |
| 10 | Distinguish the golf ball from the table tennis ball by touch and lift the golf ball. | |
| Dynamic | 00 | Insert the three-prong power plug already held by the gripper into the power socket on the moving conveyor platform. |
| 01 | Insert the bulb already held by the gripper into the bulb socket on the moving conveyor platform. | |
| 02 | Insert the USB plug already held by the gripper into the USB socket on the moving conveyor platform. | |
| 03 | Pick the test tube from the table and insert it into the angled hole base mounted on the conveyor platform. | |
| 04 | Pick the test tube from the moving source base and insert it into the angled hole base on the table. | |
| 05 | Pick the key from the table and insert it into the target lock mounted on the conveyor platform. | |
| 06 | Pick the key from the moving source lock and insert it into the target lock on the table. | |
| 07 | Pick the HDMI connector from the table and insert it into the HDMI slot mounted on the conveyor platform. | |
| 08 | Pick the HDMI connector and insert it into the HDMI slot on the table. | |
| 09 | Pick the gear from the table and press it into the gear base mounted on the conveyor platform. | |
| 10 | Pick the gear and press it into the gear base on the table. |
Table 6. All 28 TacMind-Bench Standard tasks, grouped by evaluation regime. For the Dynamic regime, the fixture is mounted on the moving conveyor platform unless stated otherwise.
Precision regime (6 tasks). Tabletop contact regulation with static scenes: pick-and-insert of a tube, key, connector, or gear into a hole base, lock, or port; insertion of a power plug and a bulb.
Recognition regime (11 tasks). Pick-and-place in which the target is visually ambiguous or occluded and must be identified by touch (texture, ribs, edges, size): in seven tasks the objects are hidden in boxes or under covered trays, and in the remaining four they are visible but nearly identical.
Dynamic regime (11 tasks). Insertion and press-fitting under externally driven motion: either the target fixture (moving-target variants) or the source base (moving-source variants) is mounted on a translating conveyor platform, requiring precise, tactile-guided alignment under relative motion.
Extra Suite. The Extra suite contains 100 tasks covering a broader range of interactions, including opening drawers and doors, pressing, sliding, rolling, cutting, wiping, and assembly. Additional variants introduce changes in objects, materials, spatial configurations, and language instructions. The complete catalog lists task IDs and natural-language instructions. The main comparative evaluation focuses on the 28 Standard tasks.
Evaluation Settings. Each Standard task is evaluated under one in-domain setting and 12 controlled perturbations: four lighting conditions, six camera configurations, and two object scales. We conduct 20 episodes per setting, yielding 260 evaluation episodes per task. These perturbations provide a common robustness evaluation across the three task regimes.

Figure 7. The 28 standard tasks of TacMind-Bench (Daimon).
| Regime | Task | Language | Instruction |
| Extra | 00 | Pull the drawer open by at least 12 cm. | |
| Extra | 01 | Grasp the handle and open the hinged door by at least 60 degrees. | |
| Extra | 02 | Turn the valve counterclockwise by 90 degrees relative to its initial angle. | |
| Extra | 03 | Press the spring button at least 8 mm for 0.3 seconds, then release it. | |
| Extra | 04 | Move the bistable lever from the negative well to the positive well, then let go. | |
| Extra | 05 | Pull the cap completely out of its spring-loaded retaining shoes and move it away by at least 3 cm. | |
| Extra | 06 | Follow the thin sinusoidal rail from its left end to its right end while maintaining tactile contact. | |
| Extra | 07 | Pivot the block upright by 90 degrees while keeping it supported by the table and allowing rotation within the gripper. | |
| Extra | 08 | Use the knife held in the gripper to cut the banana at its middle scored plane, and lift the blade clear while leaving both portions on the board. | |
| Extra | 09 | Use the knife held in the gripper to cut the cucumber at its middle scored plane, and lift the blade clear while leaving both portions on the board. | |
| Extra | 10 | Push the block along the complete S-shaped path while maintaining tactile and tabletop contact. | |
| Extra | 11 | Roll the can through half a turn along the table without sliding or lifting it. | |
| Extra | 12 | Turn the medium gear by ninety degrees so that both passive output gears rotate through their required angles. | |
| Extra | 13 | Lift the mug, thread its handle over the hook, lower it, and release it so that the hook supports its weight. | |
| Extra | 14 | Lift the rod, hold the hand still, allow a twenty-millimeter downward slip, then tighten the fingers to arrest it without regrasping. | |
| Extra | 15 | Pick up the horizontal rod near its center, orient it upright, place its flat end beside the cradle, then release and retreat while it stands unsupported. | |
| Extra | 16 | Grasp the raised latch grip, depress its spring-loaded lug below the keeper, slide it clear along the rail, then release and retreat while the plunger returns. | |
| Extra | 17 | Briefly touch each of the three inspection coupons, lifting clear between probes. Then indicate the raised defect with a steady final gel contact. | |
| Extra | 18 | Withdraw the RAM module at least 2 cm, reinsert its keyed edge, then release it and retreat. | |
| Extra | 19 | Grasp the cloth-wrapped sponge, wipe the dirt from the floral porcelain vase secured in its cradle, then lift the sponge clear. | |
| Extra | 20 | Grasp the felt eraser, erase the TacMind lettering from the rectangular whiteboard, then lift the eraser clear. | |
| Extra | 21 | Grasp the light bulb and screw it fully into the socket. | |
| Extra | 22 | Grasp the nut and thread it fully onto the bolt. | |
| Extra | 23 | Grasp the knife and dice the tomato at every scored plane. | |
| Extra | 24 | Grasp the knife and dice the potato at every scored plane. | |
| Extra | 25 | Grasp the knife and cut the carrot at every scored plane. | |
| Extra | 26 | Grasp the knife and cut the baguette at every scored plane. | |
| Extra | 27 | Grasp the knife and cut the baguette at every scored plane. | |
| Extra | 28 | Grasp the knife and cut the banana at every scored plane. | |
| Extra | 29 | Grasp the knife and cut the banana at every scored plane. | |
| Extra | 30 | Grasp the knife and cut the banana at every scored plane. | |
| Extra | 31 | Grasp the knife and cut the bread loaf at every scored plane. | |
| Extra | 32 | Grasp the knife and cut the carrot at every scored plane. | |
| Extra | 33 | Grasp the knife and cut the purple carrot at every scored plane. | |
| Extra | 34 | Grasp the knife and cut the chili at every scored plane. | |
| Extra | 35 | Grasp the knife and cut the corn at every scored plane. | |
| Extra | 36 | Grasp the knife and cut the cucumber at every scored plane. | |
| Extra | 37 | Grasp the knife and cut the cucumber at every scored plane. | |
| Extra | 38 | Grasp the knife and cut the cucumber at every scored plane. | |
| Extra | 39 | Grasp the knife and cut the cucumber at every scored plane. | |
| Extra | 40 | Grasp the knife and cut the cucumber at every scored plane. | |
| Regime | Task | Language Instruction | |
| Extra | 41 | Grasp the knife and cut the pickling cucumber at every scored plane. | |
| Extra | 42 | Grasp the knife and cut the eggplant at every scored plane. | |
| Extra | 43 | Grasp the knife and cut the garlic bread at every scored plane. | |
| Extra | 44 | Grasp the knife and cut the lotus root at every scored plane. | |
| Extra | 45 | Grasp the knife and cut the okra at every scored plane. | |
| Extra | 46 | Grasp the knife and cut the parsnip at every scored plane. | |
| Extra | 47 | Grasp the knife and cut the sweet potato at every scored plane. | |
| Extra | 48 | Grasp the knife and cut the zucchini at every scored plane. | |
| Extra | 49 | Grasp the knife and cut the zucchini at every scored plane. | |
| Extra | 50 | Grasp the knife and cut the yellow zucchini at every scored plane. | |
| Extra | 51 | Sort all blocks into the matching color zones. | |
| Extra | 52 | Load the coffee machine, place the cup under the spout, and press the button to brew coffee into the cup. | |
| Extra | 53 | Push each shape into its matching recess without lifting it. | |
| Extra | 54 | Stack every block into one aligned, stable tower on the target pad. | |
| Extra | 55 | pick up the tube and insert it into the orange plastic hole base | |
| Extra | 56 | pick up the tube and insert it into the yellow plastic hole base | |
| Extra | 57 | pick up the tube and insert it into the right hole base | |
| Extra | 58 | pick up the tube and insert it into the middle hole base | |
| Extra | 59 | pick up the tube and insert it into the left hole base | |
| Extra | 60 | pick up the black rubber tube and insert it into the hole base | |
| Extra | 61 | pick up the orange plastic tube and insert it into the hole base | |
| Extra | 62 | pick up the yellow plastic tube and insert it into the hole base | |
| Extra | 63 | pick up the right tube and insert it into the hole base | |
| Extra | 64 | pick up the middle tube and insert it into the hole base | |
| Extra | 65 | pick up the left tube and insert it into the hole base | |
| Extra | 66 | pick up the key and insert it into the red plastic lock | |
| Extra | 67 | pick up the key and insert it into the brass lock | |
| Extra | 68 | pick up the key and insert it into the right lock | |
| Extra | 69 | pick up the key and insert it into the middle lock | |
| Extra | 70 | pick up the key and insert it into the left lock | |
| Extra | 71 | pick up the key from the blue plastic source lock and insert it into the target lock | |
| Extra | 72 | pick up the key from the red plastic source lock and insert it into the target lock | |
| Extra | 73 | pick up the key from the brass source lock and insert it into the target lock | |
| Extra | 74 | pick up the right key and insert it into the lock | |
| Extra | 75 | pick up the middle key and insert it into the lock | |
| Extra | 76 | pick up the left key and insert it into the lock | |
| Extra | 77 | pick up the connector and insert it into the blue port | |
| Extra | 78 | pick up the connector and insert it into the black port | |
| Extra | 79 | pick up the connector and insert it into the right port | |
| Extra | 80 | pick up the connector and insert it into the middle port | |
| Extra | 81 | pick up the connector and insert it into the left port | |
| Extra | 82 | pick up the orange connector and insert it into the port | |
| Extra | 83 | pick up the blue connector and insert it into the port | |
| Extra | 84 | pick up the black connector and insert it into the port | |
| Extra | 85 | pick up the right connector and insert it into the port | |
| Extra | 86 | pick up the middle connector and insert it into the port | |
| Extra | 87 | pick up the left connector and insert it into the port | |
| Extra | 88 | pick up the gear and insert it into the painted green base | |
| Extra | 89 | pick up the gear and insert it into the steel base | |
| Extra | 90 | pick up the gear and insert it into the right base | |
| Extra | 91 | pick up the gear and insert it into the middle base | |
| Extra | 92 | pick up the gear and insert it into the left base | |
| Extra | 93 | pick up the white painted gear and insert it into the base | |
| Extra | 94 | pick up the painted green gear and insert it into the base | |
| Extra | 95 | pick up the steel gear and insert it into the base | |
| Extra | 96 | pick up the right gear and insert it into the base | |
| Extra | 97 | pick up the middle gear and insert it into the base | |
| Extra | 98 | pick up the left gear and insert it into the base | |
| Extra | 99 | pick up the medium gear and insert it into the matching shaft of the gear base |
Table 7. All 100 TacMind-Bench Extra tasks. Task IDs correspond to the zero-based IDs in the Extra task catalog.
Figure 8. Visualization of tactile information of Gelsight Mini.
Tac-LeWM Pretraining
Heterogeneous pretraining data. To improve the generality of Tac-LeWM beyond a single sensor and visual domain, we construct a heterogeneous visuo-tactile corpus from TacMind-Bench, ToucHD Feng et al. [8], Daimon-Infinity Daimon Robotics (2026), RDP Xue et al. [41], VLA-Touch Bi et al. [2], and VisuoTactile-QINGLOONG OpenLoong Community (2026), comprising approximately 3.0M frames and 42 hours of interaction. These datasets span different tactile sensors, camera configurations, robot platforms, and teleoperation interfaces, providing substantially broader contact and visual variation than TacMind-Bench alone. We map all observations to a common visual–tactile interface, with visual inputs resized to a shared spatial resolution and tactile observations represented using a consistent temporal layout. Actions require a different treatment because the datasets expose fundamentally different control parameterizations, including end-effector poses, wrist motions, joint-space commands, and gripper states. Rather than forcing these commands into a single kinematic representation, we retain their source-specific semantics and use a dedicated action embedding for each source before the shared dynamics predictor. Actions are additionally normalized within each source using statistics computed only from valid action samples. This prevents differences in physical scale or control parameterization from dominating the shared predictor.

Figure 9. Visualization of tactile information of Daimon.
Action-aware pretraining objective. A central challenge of heterogeneous pretraining is that visual and tactile context alone can often explain a large fraction of short-horizon state variation. Consequently, minimizing prediction error does not guarantee that the learned dynamics remain sensitive to the commanded action. This problem becomes more pronounced when the corpus contains samples without valid action supervision. We therefore separate representation learning from action-conditioned dynamics learning. Samples without valid actions may contribute to the representation objective, but they are excluded from the predictive and action-reliance terms rather than being interpreted as zero actions. For action-bearing samples, we explicitly encourage the predictor to distinguish the true action sequence from a shuffled alternative. The complete pretraining objective is
where denotes the visual–tactile representation objective, indicates the availability of valid action supervision, and and denote predictions conditioned on the true and shuffled actions, respectively. Actions are shuffled only among samples from the same source, preventing the predictor from satisfying the objective through dataset identification rather than action sensitivity. The target representation and shuffled conditioning branch are detached when evaluating the action-reliance term, so that this constraint primarily shapes the dynamics predictor instead of distorting the representation or action embedding spaces. We also randomly suppress visual observations during pretraining, encouraging the tactile pathway to retain predictive information rather than becoming redundant with vision.

Figure 10. Visualization of the extra suite task (0-63) of TacMind-Bench.
Pretraining protocol. Although the unified corpus contains multiple sources, the final training mixture emphasizes TacMind-Bench together with RDP, VLA-Touch, and VisuoTactile-QINGLOONG, ToucHD, and Daimon Infinity. TacMind-Bench receives the largest sampling mass because it defines the downstream task distribution, while the additional sources provide complementary contact dynamics and sensor diversity. Combined with source-specific action embeddings, per-source normalization, masked action supervision, and the action-reliance objective, this formulation allows Tac-LeWM to exploit heterogeneous visuo-tactile data without sacrificing actionconditioned dynamics. The resulting pretrained model is used as the initialization for subsequent task-specific adaptation and control learning.
Figure 11. Visualization of the extra suite task (64-99) of TacMind-Bench.
Additional Ablation Study
Tac-LeWM Components. We ablate three components of Tac-LeWM. Camera removes the auxiliary visual conditioning from the predictor, Causal Attn replaces causal attention with standard self-attention, and CTR replaces Canonical Tactile RoPE with the original ViT positional embedding. As shown in Table 8, the variants perform similarly on most linear probes, while removing causal attention or CTR produces a clearer degradation in tactile pad prediction. This suggests that these designs have a larger effect on deformation-related tactile information than on linearly decoding proprioceptive or action variables. Table 9 shows a more pronounced difference in future reconstruction. While current reconstruction remains relatively similar across variants, future prediction degrades substantially without causal attention or CTR. These results suggest that causal temporal modeling and tactile-specific positional encoding are particularly useful for predicting future tactile representations, while visual conditioning provides additional predictive context.
| Variant | Proprio (MSE ↓ / r ↑) | Gripper (MSE ↓ / r ↑) | Action (MSE ↓ / r ↑) | Pad (MSE ↓ / r ↑) |
| Full | 0.1816 / 0.8981 | 0.1958 / 0.9053 | 0.1841 / 0.8976 | 0.2771 / 0.7968 |
| −Camera | 0.1748 / 0.9027 | 0.1958 / 0.9053 | 0.1770 / 0.9022 | 0.2905 / 0.7855 |
| −Causal Attn | 0.1733 / 0.9035 | 0.1958 / 0.9053 | 0.1756 / 0.9030 | 0.3557 / 0.7282 |
| −CTR | 0.1831 / 0.8971 | 0.1958 / 0.9053 | 0.1851 / 0.8968 | 0.3366 / 0.7476 |
Table 8. Ablation study on Tac-LeWM: linear probing results.
| Variant | Current Reconstruction | Future Reconstruction | ||||||||
| PSNR | MSE | RMSE | MAE | SSIM | PSNR | MSE | RMSE | MAE | SSIM | |
| Full | 31.83 | 0.00157 | 0.0341 | 0.0223 | 0.8748 | 28.98 | 0.00175 | 0.0391 | 0.0263 | 0.8593 |
| −Camera | 32.66 | 0.00177 | 0.0348 | 0.0230 | 0.8649 | 26.52 | 0.00255 | 0.0489 | 0.0342 | 0.8034 |
| −Causal Attn | 31.45 | 0.00208 | 0.0385 | 0.0252 | 0.8451 | 23.07 | 0.00558 | 0.0724 | 0.0494 | 0.7116 |
| −CTR | 30.85 | 0.00190 | 0.0372 | 0.0242 | 0.8620 | 23.42 | 0.00567 | 0.0714 | 0.0518 | 0.7088 |
Table 9. Ablation study on Tac-LeWM on TacMind-Bench Standard 28 tasks: current and future reconstruction results.
Inference Acceleration
To reduce the computational latency of TacMind-0, we optimize the repeated action integration performed by the Action Expert. We compile the Action Expert with TensorRT and employ CUDA Graphs to capture and replay fixed-shape computations, reducing GPU kernel launch overhead. These optimizations are applied to both standard action generation and the three rounds of action integration in Tac-FRS.
We evaluate the Action Expert on an NVIDIA RTX 5090 using a synthetic 8-layer model with a hidden dimension of 512 and randomly initialized weights. Table 10 reports the median latency over 30 measurements. Combining TensorRT and CUDA Graph reduces the latency from 115.4 ms to 4.9 ms for a single integration round and from 300.9 ms to 14.0 ms for the three rounds in Tac-
| Inference backend | Single round (ms) | Tac-FRS three rounds (ms) |
| PyTorch | 115.4 | 300.9 |
| CUDA Graph | 16.5 | 57.0 |
| TensorRT | 11.9 | 32.4 |
| TensorRT + CUDA Graph | 4.9 | 14.0 |
Table 10. Action integration latency on an NVIDIA RTX 5090. Values are median latencies over 30 measurements.
FRS. The reported latencies cover action integration only; they exclude the VLM, tactile encoder, world predictor, and simulation. Evaluation with trained checkpoints and closed-loop task success rates remains future work.
Real-World Experiment
Experimental Setup
We conduct real-world experiments on a RealMan-ISF robotic arm equipped with a DM-Tac W tactile sensor to evaluate our method on fine manipulation and tactile discrimination tasks. Our benchmark comprises six subtasks across three categories: Insert, Assemble, and Discriminate. These categories evaluate precise alignment, contact-guided assembly, and object weight discrimination, respectively. The subtasks are defined as follows:
Insert–USB: Align a USB connector with its corresponding port and insert it.
Insert–Plug: Align a plug with its corresponding socket and insert it.
Assemble–Large Gear (GearL): Align a large gear with the target assembly position and complete assembly through positional and rotational adjustments.
Assemble–Small Gear (GearS): Align a small gear with the target assembly position and complete assembly through positional and rotational adjustments.
Discriminate–Heavy Bottle: Identify the bottle containing an added weight through physical interaction.
Discriminate–Light Bottle: Identify the light bottle without an added weight through physical interaction.
Representative trajectories for all six subtasks are shown in Figure 12. Each row presents eight frames in chronological order, illustrating the progression from the initial scene to task completion.

Figure 12. Representative trajectories for the six real-world tasks. From top to bottom, the rows show USB insertion, plug insertion, small-gear assembly, large-gear assembly, heavy-bottle discrimination, and light-bottle discrimination. Each row contains eight frames arranged in chronological order from left to right.
The dataset contains 50 trajectories per subtask, yielding 300 trajectories in total. We use task success rate as the primary evaluation metric. Each method is evaluated over 20 trials per subtask, and success rate is computed as the fraction of successful trials.
Implementation Details
We compare our method with two baselines: , N0-VTLA. All methods are trained for 40,000 steps using eight NVIDIA A100 GPUs with a global batch size of 64. For each method, we evaluate checkpoints at 20,000, 30,000, and 40,000 training steps and report the best-performing result among these checkpoints.
Task Analysis
Figure 13 reports the success rates of the three methods across six real-world tasks. TacMind-0 achieves an average success rate of 77.5%, compared with 60.0% for N0-VTLA and 19.2% for .
Figure 13. Success rates on six real-world manipulation tasks. We compare , N0-VTLA, and TacMind-0 on two insertion tasks (USB and Plug), two assembly tasks (GearS and GearL), and two weight discrimination tasks (Heavy and Light). Each method is evaluated over 20 trials per task. The numbers above the bars denote success rates in percent, and Avg is the mean across the six tasks.
Insert. Insertion requires precise visual alignment and timely correction when contact reveals a positioning error. On USB and Plug insertion, succeeds in only 5% of trials for each task. Without tactile feedback, it has difficulty detecting an obstructed insertion and correcting its position. N0-VTLA improves the success rates to 25% and 30%, respectively, indicating that tactile feedback helps the policy recover from misalignment. TacMind-0 further raises the success rates to 45% on USB and 55% on Plug. These results suggest that its combination of manipulation precision and tactile feedback supports both initial alignment and corrective actions after contact.
Assemble. Gear assembly requires the policy to locate a visually occluded target position and respond to contact between the gears. After reaching the insertion position, the policy must detect whether the gears have meshed and adjust their relative rotation when necessary. achieves success rates of 30% on GearS and 25% on GearL, reflecting the difficulty of these operations without tactile feedback. N0-VTLA reaches 75% and 55%, respectively, but its lower performance on GearL indicates that tactile localization alone does not always lead to successful assembly. TacMind-0 achieves 85% on both tasks. Its consistent performance across the two gear sizes is consistent with more reliable tactile-guided positioning and contact-dependent rotational adjustment.
Discriminate. Bottle discrimination requires the policy to determine whether a bottle contains an added weight through physical interaction. succeeds in 20% of Heavy trials and 30% of Light trials. Both rates are below the 50% chance level for a binary choice, as task success also requires the correct choice to be executed successfully. With tactile feedback, N0-VTLA reaches 85% on Heavy and 90% on Light. TacMind-0 obtains the highest success rates, achieving 100% and 95%, respectively. These results show that TacMind-0 can reliably use interaction feedback to distinguish the bottles and complete the corresponding actions.
References
- [1]Iretiayo Akinola, Jie Xu, Jan Carius, Dieter Fox, and Yashraj Narang. Tacsl: A library for visuotactile sensor simulation and learning. IEEE Transactions on Robotics, 41:2645–2661, 2025.DOI
- [2]Jianxin Bi, Kevin Yuchen Ma, Ce Hao, Mike Shou Zheng, and Harold Soh. Vla-touch: Enhancing vision-language-action model with dual-level tactile feedback. IEEE Robotics and Automation Letters, 2026.DOI
- [3]Remi Cadene, Simon Alibert, Francesco Capuano, Michel Aractingi, Adil Zouitine, Pepijn Kooijmans, Jade Choghari, Martino Russi, Caroline Pascal, Steven Palma, et al. Lerobot: An open-source library for end-to-end robot learning. In International Conference on Learning Representations, volume 2026, pp. 122398–122417, 2026.arxiv.org/abs/2602.22818
- [4]Baijun Chen, Weijie Wan, Tianxing Chen, Xianda Guo, Congsheng Xu, Yuanyang Qi, Haojie Zhang, Longyan Wu, Tianling Xu, Zixuan Li, et al. Univtac: A unified simulation platform for visuo-tactile manipulation data generation, learning, and benchmarking. arXiv preprint arXiv:2602.10093, 2026.arxiv.org/abs/2602.10093
- [5]Cheng Chi, Zhenjia Xu, Siyuan Feng, Eric Cousineau, Yilun Du, Benjamin Burchfiel, Russ Tedrake, and Shuran Song. Diffusion policy: Visuomotor policy learning via action diffusion. The International Journal of Robotics Research, 44(10-11):1684–1704, 2025.
- [8]Ruoxuan Feng, Yuxuan Zhou, Siyu Mei, Dongzhan Zhou, Pengwei Wang, Shaowei Cui, Bin Fang, Guocai Yao, and Di Hu. Anytouch 2: General optical tactile representation learning for dynamic tactile perception. In The Fourteenth International Conference on Learning Representations, 2026. URL https://openreview.net/forum?id=ndIoNnABZ.arxiv.org/abs/2602.09617
- [9]Tamar Flash and Neville Hogan. The coordination of arm movements: an experimentally confirmed mathematical model. The Journal of Neuroscience, 5(7):1688–1703, 1985.pubmed.ncbi.nlm.nih.gov/4020415
- [10]Irmak Guzey, Ben Evans, Soumith Chintala, and Lerrel Pinto. Dexterity from touch: Self-supervised pre-training of tactile representations with robotic play. arXiv preprint arXiv:2303.12076, 2023.
- [11]Byeongho Heo, Song Park, Dongyoon Han, and Sangdoo Yun. Rotary position embedding for vision transformer. In European Conference on Computer Vision, pp. 289–305. Springer, 2024.arxiv.org/abs/2403.13298
- [12]Carolina Higuera, Akash Sharma, Chaithanya Krishna Bodduluri, Taosha Fan, Patrick Lancaster, Mrinal Kalakrishnan, Michael Kaess, Byron Boots, Mike Lambeta, Tingfan Wu, and Mustafa Mukadam. Sparsh: Self-Supervised Touch Representations for Vision-Based Tactile Sensing. In Proceedings of The 8th Conference on Robot Learning, volume 270, pp. 885–915, 2024. URL https://mlanthology.org/corl/2024/higuera2024corl-sparsh/.arxiv.org/abs/2410.24090
- [13]Carolina Higuera, Akash Sharma, Taosha Fan, Chaithanya Krishna Bodduluri, Byron Boots, Michael Kaess, Mike Lambeta, Tingfan Wu, Zixi Liu, Francois Robert Hogan, et al. Tactile beyond pixels: Multisensory touch representations for robot manipulation. In Conference on Robot Learning, pp. 105–123. PMLR, 2025.arxiv.org/abs/2506.14754
- [14]Carolina Higuera, Sergio Arnaud, Byron Boots, Mustafa Mukadam, Francois Robert Hogan, and Franziska Meier. Visuo-tactile world models. 2026.arxiv.org/abs/2602.06001
- [15]Francois R Hogan, Jose Ballester, Siyuan Dong, and Alberto Rodriguez. Tactile dexterity: Manipulation primitives with tactile feedback. In 2020 IEEE international conference on robotics and automation (ICRA), pp. 8863–8869. IEEE, 2020.arxiv.org/abs/2002.03236
- [16]Robert D Howe. Tactile sensing and control of robotic manipulation. Advanced Robotics, 8(3): 245–261, 1993.DOI
- [17]Binghao Huang, Yixuan Wang, Xinyi Yang, Yiyue Luo, and Yunzhu Li. 3d-vitac: Learning fine-grained manipulation with visuo-tactile sensing. arXiv preprint arXiv:2410.24091, 2024.arxiv.org/abs/2410.24091
- [18]Yunao Huang, Shiyu Sang, Haotao Lu, Suting Ni, Shijie Wu, Ziyang Guo, Ye Shi, and Jingya Wang. Vitacworld: Scaling visuo-tactile world models for contact-rich robot manipulation. arXiv preprint arXiv:2607.22530, 2026.arxiv.org/abs/2607.22530
- [19]Physical Intelligence, Kevin Black, Noah Brown, James Darpinian, Karan Dhabalia, Danny Driess, Adnan Esmail, Michael Equi, Chelsea Finn, Niccolo Fusai, et al. \pi_{0.5}: a vision-language-action model with open-world generalization. arXiv preprint arXiv:2504.16054, 2025.arxiv.org/abs/2504.16054
- [20]Bowen Jing, Mingxin Wang, Ruiyang Hao, Chenchen Ge, Hanwen Shen, Junjie He, Yang Cui, Yiming Hou, Weitao Zhou, Jiawei Wang, et al. Softbench: A safety-aware visuo-tactile benchmark for physically constrained robotic manipulation of deformable objects. arXiv preprint arXiv:2607.04234, 2026.arxiv.org/abs/2607.04234
- [21]Chuanyu Li, Renjun Dang, Xiang Li, Zhiyuan Wu, Jing Xu, Hamidreza Kasaei, Roberto Calandra, Nathan Lepora, Shan Luo, Hao Su, et al. Maniskill-vitac 2025: challenge on manipulation skill learning with vision and tactile sensing. arXiv preprint arXiv:2411.12503, 2024.arxiv.org/abs/2411.12503
- [22]Bo Liu, Yifeng Zhu, Chongkai Gao, Yihao Feng, Qiang Liu, Yuke Zhu, and Peter Stone. Libero: Benchmarking knowledge transfer for lifelong robot learning. arXiv preprint arXiv:2306.03310, 2023.arxiv.org/abs/2306.03310
- [23]Haoyu Liu, Sucheng Ren, Tingyu Zhu, Peng Wang, Cihang Xie, Alan Yuille, Zeyu Zheng, and Feng Wang. Spiral rope: Rotate your rotary positional embeddings in the 2d plane. arXiv preprint arXiv:2602.03227, 2026.arxiv.org/abs/2602.03227
- [24]Quan Khanh Luu, Pokuang Zhou, Zhengtong Xu, Zhiyuan Zhang, Qiang Qiu, and Yu She. Mani-feel: Benchmarking and understanding visuotactile manipulation policy learning. arXiv preprint arXiv:2505.18472, 2025.arxiv.org/abs/2505.18472
- [25]Lucas Maes, Quentin Le Lidec, Damien Scieur, Yann LeCun, and Randall Balestriero. Leworld-model: Stable end-to-end joint-embedding predictive architecture from pixels. arXiv preprint arXiv:2603.19312, 2026.arxiv.org/abs/2603.19312
- [26]Duc Huy Nguyen, Tim Schneider, Guillaume Duret, Alap Kshirsagar, Boris Belousov, and Jan Peters. Tacex: Gelsight tactile simulation in isaac sim—combining soft-body and visuotactile simulators. arXiv preprint arXiv:2411.04776, 2024. NVIDIA. Isaac Sim. URL https://github.com/isaac-sim/IsaacSim.
- [28]Stéphane Ross, Geoffrey Gordon, and Drew Bagnell. A reduction of imitation learning and structured prediction to no-regret online learning. In Proceedings of the fourteenth international conference on artificial intelligence and statistics, pp. 627–635. JMLR Workshop and Conference Proceedings, 2011.arxiv.org/abs/1011.0686
- [29]Maram Sakr, Juyan Zhang, HF Machiel Van der Loos, Dana Kulić, and Elizabeth Croft. Consistency matters: Defining demonstration data quality metrics in robot learning from demonstration. ACM Transactions on Human-Robot Interaction, 15(2):1–31, 2025.
- [30]Jiaxin Shi, Xidong Zhang, Fucai Zhu, Zhe Li, Siyu Zhu, and Weihao Yuan. 3dthinkvla: Endowing vision-language-action models with latent 3d priors via 3d-thinking-guided co-training. arXiv preprint arXiv:2606.04436, 2026a.arxiv.org/abs/2606.04436
- [31]Lucy Xiaoyang Shi, Archit Sharma, Tony Z. Zhao, and Chelsea Finn. Waypoint-based imitation learning for robotic manipulation. In Proceedings of The 7th Conference on Robot Learning, pp. 2195–2209, 2023.arxiv.org/abs/2307.14326
- [32]Modi Shi, Li Chen, Jin Chen, Yuxiang Lu, Chiming Liu, Guanghui Ren, Ping Luo, Di Huang, Maoqing Yao, and Hongyang Li. Is diversity all you need for scalable robotic manipulation? IEEE Transactions on Robotics, 2026b.arxiv.org/abs/2507.06219
- [33]Zilin Si and Wenzhen Yuan. Taxim: An example-based simulation model for gelsight tactile sensors. IEEE Robotics and Automation Letters, 7(2):2361–2368, 2022.arxiv.org/abs/2109.04027
- [34]Oriane Siméoni, Huy V. Vo, Maximilian Seitzer, Federico Baldassarre, Maxime Oquab, Cijo Jose, Vasil Khalidov, Marc Szafraniec, Seungeun Yi, Michaël Ramamonjisoa, Francisco Massa, Daniel Haziza, Luca Wehrstedt, Jianyuan Wang, Timothée Darcet, Théo Moutakanni, Leonel S entana, Claire Roberts, Andrea Vedaldi, Jamie Tolan, John Brandt, Camille Couprie, Julien Mairal, Hervé Jégou, Patrick Labatut, and Piotr Bojanowski. Dinov3, 2025. URL https://arxiv.org/abs/2508.10104.
- [35]Balakumar Sundaralingam, Siva Kumar Sastry Hari, Adam Fishman, Caelan Garrett, Karl Van Wyk, Valts Blukis, Alexander Millane, Helen Oleynikova, Ankur Handa, Fabio Ramos, Nathan Ratliff, and Dieter Fox. curobo: Parallelized collision-free minimum-jerk robot motion generation, 2023. URL https://arxiv.org/abs/2310.17274.
- [36]NeoteAI Team, Fudan TEAI Team, et al. n_0-vtla: Scaling vision-tactile-language-action model with latent tactile tokens. arXiv preprint arXiv:2607.23782, 2026.arxiv.org/abs/2607.23782
- [37]Shaoxiong Wang, Mike Lambeta, Po-Wei Chou, and Roberto Calandra. Tacto: A fast, flexible, and open-source simulator for high-resolution vision-based tactile sensors. IEEE Robotics and Automation Letters, 7(2):3930–3937, 2022.DOI
- [38]Yansong Wu, Zongxie Chen, Fan Wu, Lingyun Chen, Liding Zhang, Zhenshan Bing, Abdalla Swikir, Sami Haddadin, and Alois Knoll. Tactifusion: Force-domain diffusion policy for precise tactile manipulation. In 2025 IEEE international conference on robotics and automation (ICRA), pp. 11831–11837. IEEE, 2025.arxiv.org/abs/2409.11047
- [39]Zhengtong Xu, Raghava Uppuluri, Xinwei Zhang, Cael Fitch, Philip Glen Crandall, Wan Shou, Dongyi Wang, and Yu She. Unit: Data efficient tactile representation with generalization to unseen objects. IEEE Robotics and Automation Letters, 10(6):5481–5488, 2025.arxiv.org/abs/2408.06481
- [40]Chao Xue, Chaofan Zhang, Wenxuan Ma, Guocai Yao, Shaowei Cui, and Shuo Wang. Hitac-wam: A hierarchical tactile world action model for contact-rich robot manipulation. arXiv preprint arXiv:2608.19574, 2026.arxiv.org/abs/2608.19574
- [41]Han Xue, Jieji Ren, Wendi Chen, Gu Zhang, Yuan Fang, Guoying Gu, Huazhe Xu, and Cewu Lu. Reactive diffusion policy: Slow-fast visual-tactile policy learning for contact-rich manipulation. arXiv preprint arXiv:2503.02881, 2025.arxiv.org/abs/2503.02881
- [42]Zhao-Heng Yin, Binghao Huang, Yuzhe Qin, Qifeng Chen, and Xiaolong Wang. Rotating without seeing: Towards in-hand dexterity through touch. arXiv preprint arXiv:2303.10880, 2023.arxiv.org/abs/2303.10880
- [43]Kelin Yu, Yunhai Han, Qixian Wang, Vaibhav Saxena, Danfei Xu, and Ye Zhao. Mimictouch: Leveraging multi-modal human tactile demonstrations for contact-rich manipulation. arXiv preprint arXiv:2310.16917, 2023.arxiv.org/abs/2310.16917
- [44]Ying Yuan, Haichuan Che, Yuzhe Qin, Binghao Huang, Zhao-Heng Yin, Kang-Won Lee, Yi Wu, Soo-Chul Lim, and Xiaolong Wang. Robot synesthesia: In-hand manipulation with visuotactile sensing. In 2024 IEEE International Conference on Robotics and Automation (ICRA), pp. 6558–6565. IEEE, 2024.arxiv.org/abs/2312.01853
- [45]Yujie Zang, Yuhang Zheng, Xian Nie, Yupeng Zheng, Shuai Tian, Songen Gu, Chen Gao, Zining Wang, Shuicheng Yan, and Wenchao Ding. Tacforesight: Force-guided tactile world model for contact-rich manipulation. arXiv preprint arXiv:2606.11184, 2026.
- [46]Xidong Zhang, Yichi Zhang, Jiaxin Shi, Fucai Zhu, Siyu Zhu, Michael Yu Wang, Xiaojun Wu, and Weihao Yuan. Unitactvla: Unified tactile understanding and prediction in vision language action models. arXiv preprint arXiv:2606.31723, 2026.arxiv.org/abs/2606.31723
- [47]Jialiang Zhao, Yuxiang Ma, Lirui Wang, and Edward H. Adelson. Transferable tactile transformers for representation learning across diverse sensors and tasks. arXiv preprint arXiv:2406.13640, 2024.arxiv.org/abs/2406.13640
- [48]Tony Z Zhao, Vikash Kumar, Sergey Levine, and Chelsea Finn. Learning fine-grained bimanual manipulation with low-cost hardware. arXiv preprint arXiv:2304.13705, 2023.arxiv.org/abs/2304.13705
- [49]Yuhang Zheng, Songen Gu, Weize Li, Yupeng Zheng, Yujie Zang, Shuai Tian, Xiang Li, Ce Hao, Chen Gao, Si Liu, et al. Omnivta: Visuo-tactile world modeling for contact-rich robotic manipulation. arXiv preprint arXiv:2603.19201, 2026.arxiv.org/abs/2603.19201
- [50]Jianyi Zhou, Feiyang Hong, Yunhao Li, Yicheng Zhao, Yongjue Cen, Zirui Liu, Jiakang Huang, Zirui Chen, Ruiyang Zhang, Weizhuo Zhu, et al. Touchworld: A predictive and reactive tactile foundation model for dexterous manipulation. arXiv preprint arXiv:2607.07287, 2026.arxiv.org/abs/2607.07287





