WorksComputer Science
TactileLab: Sim-to-Real Dexterous Robot Manipulation with Efficient Shear-Sensitive Tactile Simulation
Abstract
Robotic simulators that represent shear within tactile sensing are becoming increasingly important for contact-rich manipulation. However, simulating contact shear with high-fidelity deformable or soft-body simulation is computationally demanding, limiting scalability to large-scale reinforcement learning. Here we introduce TactileLab, a scalable multimodal tactile simulation framework built on IsaacLab, with a particular focus on a new optical-flow-based tactile representation termed simulated tactile flow. Instead of relying on soft-body physics to model shear deformation, TactileLab uses rigid-body simulation to efficiently estimate dense tangential motion at the contact interface, producing tactile flow fields that encode shear-relevant contact dynamics. This flow field complements geometric features from a contact depth field. We further introduce a novel policy architecture for jointly processing long-horizon proprioceptive signals and rich tactile observations. Across multiple contact-rich tasks, policies trained using simulated tactile flow learn efficiently at scale and exhibit emergent contact-aware behaviours. To facilitate sim-to-real policy deployment, we then propose a novel real-to-sim tactile translation method that maps real tactile image streams to simulated tactile flow and depth fields, together with a new tactile data collection method, and validate it across diverse tactile sensor designs. Using this pipeline, we demonstrate real-world deployment on tasks encompassing dual-arm and dexterous in-hand manipulation. Overall, TactileLab provides an end-to-end sim-to-real framework for scalable tactile policy learning, enabling diverse shear-sensitive manipulation across heterogeneous tactile robotic systems.
YL and NFL were supported by ARIA Robot Dexterity award SMRB-PR01-P19. YL conducted this work at the University of Bristol.
Project webpage: https://yijionglin.github.io/tactilelab/
Introduction
Tactile sensing provides local, contact-grounded feedback that is essential for fine manipulation tasks. This is the case when a robot must finely regulate interaction forces such as to maintain contact or respond to small changes at the contact interface. For such tasks, vision-based tactile sensors are attractive because they provide high-resolution and highly-sensitive spatial observations. However, scalable simulation methods still struggle to represent tangential interaction efficiently, which is needed for tasks that depend on shear forces.
Most existing tactile simulators fall into two categories: 1) soft-body simulators can model compliant contact with high fidelity [1, 2, 3], but can be too slow for large-scale reinforcement learning; 2) rigid-body simulators are fast and scalable, but usually provide only depth-like signals [4, 5, 6, 7, 8]. As a result, they are well suited for normal-contact reasoning but less effective for tasks in which shear, slip, or tangential motion matters. This limitation is important for manipulation problems such as in-contact trajectory following, insertion, and in-hand object manipulation in the presence of external perturbations. Depth-only observations cannot fully capture contact dynamics over time: similar depth signals may correspond to different contact states and future outcomes, making the task partially observable and harder for policies to learn. Because of this, recent tactile simulators have modelled distributed normal and shear feedback by relying on geometry-dependent contact formulations and object-specific preprocessing [9], [10]. Here we introduce a new way to instead calculate dense shear cues directly from rigid-body simulation without those assumptions, using a novel optical-flow-based tactile representation that we term simulated tactile flow.
This new flow representation allows us to address the gap of efficient shear-sensitive tactile simulation and robot learning, by introducing TactileLab: a complete sim-to-real framework that augments rigid-body tactile sensing with a dense tangential displacement field. Rather than explicitly simulating elastomer deformation, the framework represents contact using two complementary components: normal-contact depth and a tangential-displacement vector field. This combination gives a compact and sim-to-real transferable tactile observation that is efficient to compute, maps onto various tactile sensor designs, and is suitable for large-scale reinforcement learning.
Our main contributions are as follows:
A scalable multimodal tactile simulation framework for contact-rich manipulation: TactileLab is built on top of IsaacLab [11] and supports GPU-parallel simulation for efficient reinforcement learning. It provides multiple tactile sensing modalities, including contact forces, contact positions, contact depth images, and simulated tactile flow, allowing researchers to study both low-dimensional and high-dimensional tactile feedback within a unified framework.
A novel simulated tactile modality for shear-relevant contact interaction: We introduce simulated tactile flow, an optical-flow-inspired tactile modality that represents dense tangential motion at the contact interface. Together with contact depth, simulated tactile flow provides complementary high-dimensional tactile observations of normal contact geometry and tangential contact motion, enabling policies to exploit shear-relevant cues without explicit soft-body simulation.
A diverse suite of tactile robotic manipulation environments: TactileLab provides a set of contact-rich robotic environments for single-arm, dual-arm, and dexterous in-hand manipulation. These show the scalability and generality of TactileLab for policy learning with high-dimensional contact-depth and tactile flow observations. We conduct detailed simulation studies on two representative tasks: dual-arm object following and dexterous in-hand object manipulation.
A new sample-efficient policy architecture for fusing proprioceptive history and tactile observations: We propose PETS-Net (Positional-Encoding Temporal-Spatial Network) for encoding long-horizon low-dimensional proprioceptive histories and fusing them with high-dimensional tactile observations. Experimental results show that PETS-Net learns more efficiently than baseline policy architectures, demonstrating the benefit of jointly encoding temporal robot-state histories and spatial tactile features for contact-rich manipulation.
A new real-to-sim tactile transfer method for multimodal tactile observations: Building on a previous static transfer method for single tactile images, we propose separate real-to-sim translators for contact depth and tactile flow. Specifically, the tactile-flow translator takes real tactile image streams as input and predicts simulated tactile flow, extending real-to-sim transfer from static contact-depth reconstruction to dynamic tangential-motion representations. We further validate the real-to-sim pipeline across diverse tactile sensor designs, demonstrating its applicability beyond a single sensor configuration.
Real-world deployment of TactileLab-trained policies: Using the proposed multimodal tactile simulation and real-to-sim transfer pipeline, we demonstrate successful real-world deployment of policies trained in TactileLab on two representative contact-rich manipulation tasks: dual-arm object following and dexterous in-hand object manipulation. These results demonstrate the applicability of the TactileLab framework across distinct robotic embodiments and manipulation settings.
TactileLab will be released as an open-source framework to lower the barrier to entry to tactile robot learning, enable reproducible research, and accelerate progress in tactile robotics.
Related Work
Tactile Simulation
Existing tactile simulators broadly separate into rigid-body and deformable-body approaches. Rigid-body methods, including Tactile Gym [4], [5], TACTO [7], and other large-scale frameworks [6], [8], generate tactile signals from contact geometry, penetration depth, or contact forces available in conventional physics engines. They support efficient contact simulation but generally represent instantaneous normal contact without explicitly modelling shear deformation. In contrast, soft-body simulators model marker or elastomer deformation using analytical or calibrated approximations, FEM, or material-point methods [1]–[3], [12]–[17]. Although they capture richer contact information, they require sensor-specific deformation models and calibrated material parameters. Full soft-body solvers are also computationally intensive and generally offer lower throughput than rigid-body simulation [2], [17], limiting large-scale parallel data generation for data-hungry methods such as RL. Recent geometry-aware approaches additionally model shear-sensitive signals. TacSL [9] computes dense normal and shear force fields, while HydroShear [10] models path-dependent hydroelastic shear. However, both require prior access to the contacting object shape through a signed-distance-function (SDF) representation and produce modelled force or marker-displacement fields. TactileLab instead derives tactile flow directly from rigid-body contact kinematics without object-specific SDF preprocessing, retaining scalable simulation while providing dense tangential-motion observations for shear-sensitive tasks.
A central design goal of TactileLab is to support both traditional low-dimensional tactile signals and rich high-dimensional tactile observations. Low-dimensional signals, such as contact positions, contact forces, and shear vectors, provide compact information about contact events and interaction dynamics [1], [13], [18]–[21]. In contrast, high-dimensional tactile observations, such as contact-depth images [4]–[7], [9] and optical-flow-based tactile flow [22]–[24], provide dense spatial information over the tactile sensing surface. This allows researchers to study how different levels of tactile feedback affect policy learning, from compact contact-state representations to image-like tactile observations suitable for end-to-end learning.
Tactile Sim-to-Real Policy for Robotic Manipulation
Early tactile sim-to-real policies were mainly developed for robotic arms equipped with vision-based tactile end-effectors. For example, the Tactile Gym series trains RL policies using simulated contact-depth images and maps real tactile images into the same observation space through paired real-to-sim translation [4], [5], [19], [25, 26, 27]. These methods bridge the tactile observation gap effectively, but represent instantaneous contact geometry rather than tangential contact motion.
With the development of more advanced tactile simulators, recent methods [1], [9] have further demonstrated sim-to-real transfer for contact-rich manipulation using richer or more physically grounded tactile observations. More closely related to tangential contact sensing, approaches [3], [10], [15] incorporate shear-related tactile observations into policy learning and demonstrate sim-to-real transfer on manipulation tasks. However, such tactile sim-to-real approaches have so far been primarily demonstrated on robotic arms or grippers, while dexterous-hand manipulation has generally relied on proprioception or lower-dimensional contact representations.
At the dexterous-hand level, rapid motor adaptation first enabled zero-shot in-hand rotation by inferring latent object properties from proprioceptive history [28]. RotateIt extends this proprioception-only framework by incorporating visual observations and discretized tactile information [29]. Touch Dexterity instead transfers in-hand rotation policies using binary contact taxels distributed over the fingers and palm [30]. More recent methods use compact contact representations: AnyRotate estimates contact pose and force from real tactile images during tactile sim-to-real policy deployment [20], while Pan et al. [31] introduce a physics-grounded centre-of-pressure representation that captures contact force and location for in-hand manipulation.
Collectively, these studies demonstrate effective tactile sim-to-real transfer using depth images, binary or discretized contact signals, compact contact properties, and shear-related representations. However, a common dense temporal tactile representation that can support sim-to-real transfer across different types of vision-based tactile sensors and robotic embodiments remains underexplored. TactileLab introduces a real-to-sim transfer method that maps consecutive real tactile images into the dense tactile-flow space used during simulation, supporting policy transfer across different sensor designs, robotic embodiments, and manipulation tasks.
Multimodal Tactile Robotic Simulator
Overview of TactileLab
TactileLab is a scalable multimodal tactile simulation framework built on top of IsaacLab [11]. Rather than pursuing high-fidelity soft-body simulation of tactile elastomers, TactileLab focuses on the efficient generation of tactile observations within rigid-body simulation, enabling large-scale reinforcement learning for contact-rich robotic manipulation. In particular, TactileLab introduces simulated tactile flow, which provides a dense representation of tangential contact motion, complementing contact depth without requiring computationally expensive soft-body simulation. While instantiated here using rigid-body contact kinematics, tactile flow itself is not inherently tied to rigid-body physics and can naturally be extended to deformable-body simulation by tracking the motion of points on the tactile elastomer surface.
TactileLab follows the design desiderata of efficient vision-based tactile simulation, including high throughput, flexibility, and ease of use. Similar to prior efficient tactile simulation approaches [4], TactileLab does not aim to reproduce the full physical deformation or optical transduction process of a specific tactile sensor. Instead, it simulates task-relevant tactile quantities that can be generated efficiently at scale and later aligned with real tactile observations through real-to-sim transfer. It prioritizes scalable data generation, compatibility with large-scale reinforcement learning, and flexibility across sensor morphologies and manipulation tasks.
Multimodal Tactile Simulation
TactileLab provides multiple tactile sensing modalities within the same simulation framework. At each simulation step, a rigid-body simulator can provide the robot state, object state, contact information, and relative motion between the tactile sensor and contacting objects. Based on this information, TactileLab generates observations at different levels of abstraction over low- and high-dimensional tactile information.
Low-dimensional tactile information includes compact contact descriptors such as contact force, position, and shear over time (Figure 2 A3, left). These signals summarize contact interaction through physically meaningful quantities that are useful for analysis and privileged learning. In contrast, high-dimensional tactile observations such as contact depth and tactile flow are represented as dense fields over the tactile sensing surface, which are closer to the image-like observations produced by vision-based tactile sensors.

Figure 2. Overview of the TactileLab simulation framework and simulated tactile-flow computation. The rigid-body simulation pipeline computes contact geometry and generates low-dimensional contact descriptors, contact-depth images, and dense tactile-flow fields for policy learning. Simulated tactile flow is obtained by projecting the three-dimensional contact patch onto the tactile image plane and calculating the image-space displacement induced by the relative motion between the tactile sensor and the contacted object.
High-dimensional tactile information includes the contact-depth image at time , where each pixel value in the -dimensional array represents the local normal contact depth at a corresponding point on the tactile sensing surface (Figure 2 A3, right). This provides a dense representation of normal contact geometry, indicating where contact occurs and how deeply the object indents into the sensor surface. Likewise, the tactile-flow field is denoted by , where each pixel has a two-dimensional tangential motion vector at the contact interface. Together, these constitute a multimodal tactile observation
where denotes the contact-depth image, the tactile-flow field, and denotes low-dimensional contact information, including contact force, position, and shear vectors.
In practice, different tasks and experiments may use different subsets of these modalities. For example, contact-depth images are used across the broader task suite to demonstrate scalable tactile policy learning, while contact depth and tactile flow are used in representative shear-relevant tasks to study multimodal tactile feedback and real-world deployment.
Simulated Tactile Flow
Vision-based tactile sensors capture both spatial contact geometry and temporal cues from in-contact motion, such as slip, stick–slip transitions, and tangential displacement. In real sensors, these cues are observed through optical flow or marker motion in tactile image sequences [22, 23, 24]. Simulating them remains challenging: soft-body methods can reproduce elastomer deformation and realistic image motion but are computationally expensive, whereas rigid-body simulators scale efficiently to parallel reinforcement learning but generally lack a dense representation of tangential contact motion.
To bridge this gap, we introduce simulated tactile flow: an optical-flow-inspired tactile modality that represents dense in-plane motion at the contact interface. To the best of our knowledge, this is the first work to formulate dense contact motion as a high-dimensional tactile observation for policy learning in scalable rigid-body simulation. The simulated tactile flow is constructed directly from contact kinematics while providing an efficient, object-agnostic representation of shear-relevant contact cues. The details are described below.
Tactile Flow Representation
At time , the simulated tactile flow is defined as a 2D vector field over a tactile image
where and . Here and denote the horizontal and vertical image-space displacements, respectively, between times and . Each tactile pixel is associated with a two-dimensional motion vector. The resulting dense tactile-flow field is compatible with convolutional policy networks and can be spatially aligned with contact-depth images for multimodal fusion.
Although simulated tactile flow is represented in image coordinates, it is induced by relative tangential motion between the contacting object and the tactile sensor. It thus provides an optical-flow-inspired representation of shear-relevant contact cues without explicitly simulating elastomer deformation. Importantly, the proposed tactile-flow representation is independent of the underlying contact dynamics formulation. Although TactileLab computes it from rigid-body contact kinematics for efficiency and scalability, the same representation can also be derived from deformable-body simulations by tracking the motion of surface points at the sensor interface.
Rigid-Body Computation for Tactile Flow
In TactileLab, simulated tactile flow is computed as the dense image-space displacement induced by relative rigid-body motion between the contacting object and the tactile sensor. The tactile sensor is modeled as an embedded camera observing the contacting object surface through the tactile skin, where the resulting contact geometry is approximated by the geometric penetration of the object surface into the nominal tactile-elastomer surface. Given a contact-depth map over pixels , we define a binary contact mask
where is a small contact-depth threshold. For each contact pixel satisfying , let denote the corresponding point on the contacting object surface that penetrates the nominal tactile-elastomer surface, expressed in the tactile-sensor frame. Its image coordinate is then given by , where denotes the ray projection onto the tactile image plane.
The simulated tactile-flow field over the complete tactile-image domain is then defined as
over points . Here the raw image-space displacement is
where denotes the same object-surface point after being transformed by the relative rigid-body motion over one simulation timestep, expressed in the tactile-sensor frame. The resulting is the dense tactile-flow representation defined in part a), with zero flow outside the contact region. It is computed directly from rigid-body motion, contact geometry, and camera projection, without explicitly simulating tactile-elastomer deformation.
c) Relation to Contact Depth: Contact-depth images provide a clean and physically interpretable representation of normal contact geometry. They indicate where contact occurs and how deeply the object presses into the tactile surface, making them effective for many tactile manipulation tasks. However, contact depth primarily captures normal interaction and does not explicitly encode tangential contact motion.
This limitation becomes critical for tasks where shear, slip, or persistent in-contact motion play a central role. Two contact states may produce similar depth images for distinct tangential motion patterns. Simulated tactile flow complements contact depth by providing dense in-plane motion cues at the contact interface. While contact depth represents where contact occurs, tactile flow represents how the contact is moving tangentially.
Together, contact depth and tactile flow provide complementary high-dimensional tactile observations of normal contact geometry and tangential contact motion. This enables TactileLab to represent shear-relevant interaction cues without explicitly simulating soft-body deformation, while remaining efficient enough for large-scale reinforcement learning.
Multimodal Tactile Policy Learning
Task Setting and Multimodal Tactile Observation
We study multimodal tactile policy learning on two representative contact-rich tasks: dual-arm object-following and dexterous in-hand object manipulation (as shown in Figure 1). Both tasks require the policy to reason about contact evolution over time and to react to normal and tangential interaction at the tactile interface. The two tasks share a similar teacher–student learning structure (Figure 3). The task descriptions are as follows while more details are given in § 6-B and VI-C:

Figure 1. TactileLab is an end-to-end framework for efficient, shear-sensitive tactile robot learning. Built on IsaacLab, it integrates GPU-accelerated tactile simulation with a novel sensing modality, simulated tactile flow, for diverse contact-rich tasks and heterogeneous robotic systems. The framework includes modules for tactile policy and representation learning, a real-to-sim transfer pipeline supporting several real tactile sensors, and sim-to-real deployment of tactile reinforcement learning policies for various robots. This unified design enables scalable policy training, efficient collection of simulated tactile data, and effective transfer from simulation to real-world contact-rich manipulation.

Figure 3. (a) Overview of multimodal tactile policy learning for dexterous in-hand manipulation. To evaluate the benefit of our rich tactile observations, unknown random force and torque perturbations are applied to the manipulated object, making the task more challenging and requiring robust and informative tactile feedback. (b) The structure of the proposed PETS-Net for high-dimensional tactile policy learning.
Dual-arm object-following in a 6D workspace is a shear-dependent task that has previously been studied using tactile sensing and a traditional PID controller tuned in the real world [32]. The goal is for a tactile follower arm to maintain stable contact with an object moved by a leader arm along a predefined trajectory. Here, we investigate whether simulated tactile flow enables this task to instead be learned with deep RL in the TactileLab simulation. The action space consists of 6D pose changes of the tactile follower arm.
The dexterous in-hand object manipulation task follows settings similar to those of prior works [20, 28], in which a policy controls a dexterous hand to rotate an object held by the fingers without palm support. The action space consists of incremental commands for the 16 finger-joint angles. To further assess the benefits of richer tactile modalities, the manipulation environment is made more challenging by applying unknown random force and torque perturbations to the object.
The simulator dynamics define an underlying finite-horizon Markov decision process (MDP) with state , action , reward , and discount factor . However, the deployable policy receives only partial observations , which appear as a partially observable Markov decision process (POMDP). Consequently, the base policy uses a short proprio-action history to provide local velocity- and acceleration-related information through recent proprioceptive states and action changes
where denotes the proprioceptive states (such as the hand joint configuration or the robot arm end-of-effector’s pose), is the previous action, and is the short history length used by the base policy.
In the experiments, the student observation is selectable across different modality combinations. The set of available tactile modalities includes some image-like modalities
Then denotes the subset of tactile modalities used in a particular experiment, with being image-like. Following the notation from Sec. III-B, the corresponding tactile observation is written as , where , , denote the observation components. The deployable student system therefore uses the short base-policy input together with a longer history for latent adaptation, where is the adaptation history length.
Two-stage Multimodal Tactile Policy Learning
Here we adopt a two-stage teacher-student learning framework [28] (see, Figure 3). In Stage 1, a teacher policy is trained in simulation using information unavailable on the real robot: denotes the privileged extrinsic state, including the object’s mass, pose, twist, friction, scale; the external perturbation forces and torques applied to the object; and the hand orientation. The base policy then predicts actions with the short proprio-action history , using a privileged latent vector that is a compact representation mapped using an extrinsic encoder . The teacher policy is optimized using RL to maximize the expected return . This stage allows the policy to learn strong contact-rich behaviours with access to privileged simulation information. During student learning, the privileged extrinsic encoder is replaced by an adaptation module that estimates the extrinsic latent vector from real-world-available proprioceptive, action, and tactile-observation histories.
In Stage 2, the base policy is copied from Stage 1 and frozen, while the extrinsic encoder estimates the latent vector from histories of proprioceptive states, actions, and selected tactile observations available in the real world. The encoder is trained to match the privileged latent representation using a standard least-squares loss
The student policy then acts using the same base-policy input as the teacher policy but with extrinsic vectors predicted from the adaptation module,
Here, denotes the short proprio-action history length used by the frozen base policy. Because is inferred only from real-world-available proprioceptive, action, and tactile histories, the resulting policy can be deployed on the real robot without access to privileged simulation states.
Proposed Positional-Encoding Temporal-Spatial Network
Neural networks are known to exhibit a bias toward learning low-frequency functions, which can limit their ability to represent rapidly varying input-output relationships [33], [34]. This issue is particularly relevant in contact-rich dexterous manipulation, where high-dimensional state–action spaces, discontinuous contact patterns, and contact-mode transitions can cause small changes in joint configuration, previous action, or contact state to require different corrections [35], [36].
Here we propose a novel Positional-Encoding Temporal-Spatial Network (PETS-Net) and use it as the adaptation module , as shown in Figure 3(b). Although a standard adaptation module [28] can also replace the privileged extrinsic encoder during student learning, its representational capacity could be limited when the input consists of long-horizon proprioceptive histories, action histories, and high-dimensional tactile-observation histories. PETS-Net is designed to improve this adaptation process by learning a richer tactile-proprioceptive representation from real-world-available observations. Specifically, PETS-Net estimates the privileged latent vector from real-world-available temporal information by combining positional-encoded joint-angle and action histories with spatially encoded tactile-observation histories. The positional-encoding component is applied inside the adaptation module to robotic proprioceptive states (e.g., joint angles) and action sequences, enabling proprioceptive and action histories to be represented more expressively. The spatial component encodes selected high-dimensional tactile observations, such as contact-depth images and tactile-flow fields, and the temporal component aggregates the encoded tactile features together with the positional-encoded proprio-action histories over time.
For compactness, we denote the positional-encoded proprio-action feature used by PETS-Net as
where denotes or . Here, the positional encoding is defined by frequency-band encoding over components
concatenating the encodings of the individual components .
For high-dimensional tactile inputs, the selected image-like modalities are encoded by modality-specific spatial tactile encoders: for contact-depth map, and for tactile flow. Their latent embeddings are concatenated with the low-dimensional contact descriptors , when used, to form the tactile feature
Here, and denote the latent vectors encoding contact-depth and tactile-flow information, respectively. A temporal encoder then aggregates tactile features and positional-encoded proprio-action features
over an adaptation history window of length . Finally, PETS-Net predicts the latent interaction representation
through a latent head . This design allows the adaptation module to reason over both the spatial structure of selected tactile images and their temporal evolution, while also using a richer positional-encoded representation of proprioceptive and action histories.
Real-to-Sim Tactile Transfer
Real-to-Sim Tactile-Flow Translation Formulation
To bridge the observation gap between real vision-based tactile sensing and the high-dimensional simulated tactile modalities used in our policy-learning framework, we formulate real-to-sim tactile-flow prediction as a pixel-level image-pair-to-image translation problem. Given two consecutive real tactile images, the model will predict a pixel-aligned simulated tactile-flow field that is directly compatible with the tactile modality available in TactileLab.
In this work, the high-dimensional real-to-sim tactile translation includes contact-depth maps and tactile-flow fields (see Sec. III-C). Since contact-depth prediction from vision-based tactile images has been explored in prior work [4], [5], we use it as a supporting modality and focus on the more challenging problem of learning pixel-level simulated tactile flow from real tactile image pairs. Unlike single-frame contact-depth prediction [4], [5], which mainly estimates normal contact geometry from a single tactile image, our formulation predicts a dense temporal tactile-flow field from two consecutive real tactile images, capturing local in-plane motion caused by tangential interaction. It also differs from low-dimensional shear-vector estimation [21], [32], as the output preserves pixel-level spatial structure across the tactile image.
Given two consecutive real tactile images , we define the real-to-sim tactile-flow mapping
as a function that maps a pair of real tactile images to the corresponding simulated tactile-flow field:
Our aim is to learn a real-to-sim tactile transfer that approximates , as described in the next subsection.
TacFlowNet: Learning Pixel-Level Tactile-Flow Prediction
Having defined the desired real-to-sim tactile-flow mapping in § 5.4-A, we now introduce Real-to-Sim-TacFlowNet (TacFlowNet), a learnable model (as shown in Figure 4) that approximates this mapping from real tactile-image pairs to simulated tactile-flow fields. During training, each real-to-sim sample consists of a real start–final tactile image pair and its paired simulated tactile-flow target. Because the instantaneous real robot velocity during this short motion is unavailable and may be non-constant, we treat the start–final image pair as an effective one-step tactile transition corresponding to one tactile-sensor sampling interval. The detailed sensor-frequency normalization used to construct the paired simulated target is described in § 6-A.

Figure 4. Overview of Real-to-Sim-TacFlowNet. A pair of real tactile images is concatenated and passed to a pix2pix-style generator , which predicts a pixel-level simulated tactile-flow field. During training, the predicted flow is supervised by a paired simulated tactile-flow target through a reconstruction loss and by a discriminator through an adversarial loss. The HSV flow images are used only for visualization; the network output is a two-channel flow field .
We concatenate the two real tactile images along the channel dimension to form a single sample so that the model can learn motion-dependent deformation cues from the tactile appearance change between the two contact states. The mapping to a simulated tactile-flow target is implemented by a convolutional generator , denoted as TacFlowNet: following a pix2pix-style conditional image-to-image translation formulation [37], the network predicts
where the corresponding tactile flow target is defined in Sec. III-C. The dense pixel-level supervision allows the model to learn a direct real-to-sim mapping from real tactile image pairs to the tactile-flow modality.
To encourage both pixel-level accuracy and realistic spatial flow structure, we train the mapping using a conditional adversarial objective together with a supervised flow reconstruction loss [37]. The discriminator receives the real tactile image pair together with either the paired simulated tactile-flow target or the generated tactile-flow prediction. An adversarial loss
is combined with a masked pixel-level flow reconstruction loss
where denotes element-wise multiplication, denotes the binary contact mask associated with the -th simulated tactile-flow target, and prevents division by zero. This loss enforces dense agreement between the predicted and target simulated tactile-flow fields within the contact region. The overall training objective is parametrised by a coefficient ,
that balances adversarial realism vs pixel-level flow accuracy.
The key contribution of TacFlowNet is the temporal, pixel-level prediction of simulated tactile flow from real tactile image pairs. Unlike analytical optical-flow pipelines that explicitly track printed markers, dense color patterns, or local image patches, our method learns a direct mapping from real tactile appearance changes to the simulated tactile-flow modality used for policy learning. The predicted output is therefore a simulated tactile-flow representation, rather than the true optical flow of the real tactile images.
During real-world deployment, TacFlowNet receives consecutive tactile frames, , captured at the tactile-sensor sampling frequency. The trained generator converts each consecutive image pair into pixel-level simulated tactile-flow observations, enabling policies trained with simulated tactile flow to operate from real tactile inputs.
Experiments
Real-to-Sim Tactile Data Collection
To train the real-to-sim tactile translators, we collect paired real and simulated tactile observations using a shared contact-and-shear protocol. Each sample is specified by a 6D contact pose and a prescribed relative motion , where denotes the contact position and denotes roll, pitch, and yaw. The sampling ranges for contact poses and relative motions are reported in the Appendix.
For each sample, the real tactile sensor is first moved to a clearance pose and then brought into contact with the object at . After contact is established, we record the initial tactile image . The sensor then executes the prescribed motion to , while maintaining contact, and the final tactile image is recorded. This yields a real tactile image pair
capturing the change in elastomer deformation induced by the contact motion.
Corresponding simulation labels are generated using the tactile sensor model in TactileLab, based on the sampled contact poses and relative motions. We record contact-depth maps at the initial and final states, denoted by and , together with the associated dense tactile-flow field . The resulting paired dataset is
These labels supervise the translation of real tactile observations into contact-depth and tactile-flow representations. During deployment, the trained translators process consecutive real tactile frames.
As shown in Figure 9, the collected dataset covers representative contact motions, including pressing, sliding, tilting, and twisting. For visual reference, sparse marker flow is computed from the real tactile images using a classical optical-flow method, illustrating the observed shear deformation. The simulated tactile-flow fields provide dense pixel-level supervision for learning motion-related tactile features.

Figure 9. Examples of paired real–sim tactile data collection for representative contact motions, including press, slide, tilt, and twist. Each example shows the real robot collection setup, corresponding final tactile image from the real image pair, real vs. generated sparse flow fields, simulated contact depth, and dense simulated tactile flow.
Dual-Arm Object-Following Task Setup
To evaluate whether the proposed simulated tactile flow provides useful shear information in rigid-body simulation, we consider a dual-arm object-following task adapted from [21], [32]. A leader arm moves an object along a randomly generated 6-DoF trajectory, while a tactile follower arm must maintain a target contact pose relative to the moving object. Unlike prior approaches based on hand-designed pose-and-shear tactile servoing controllers, we learn the follower behaviour using a two-stage teacher–student RL pipeline. The task depends critically on shear because lateral and rotational object motions induce tangential motion at the contact interface that cannot be observed from contact depth alone.
State, Observation, and Action Spaces: In Stage 1, the teacher policy receives privileged state information comprising the object pose and twist, the follower TCP pose and twist, and low-dimensional contact information, including the contact position and shear offset. These privileged states are used exclusively for teacher-policy training.
In Stage 2, the student policy receives the follower TCP pose together with the contact-depth map and simulated tactile-flow field. These high-dimensional tactile observations are used exclusively during student-policy training. The student policy is trained to imitate the teacher through action distillation. Since this task has smaller observation and action spaces than the dexterous in-hand manipulation task, plain action distillation is sufficient for training the student policy.
2) Reward Function and Evaluation Metrics: The reward encourages the follower TCP to maintain the target contact pose relative to the moving object while producing stable control behaviour. The per-step reward at time is
where the distance term penalizes the Euclidean distance between the current and target relative contact positions, the orientation term penalizes the cosine difference between the current and target relative orientations of the object and the follower TCP, and the action term penalizes excessive control actions to encourage stable follower behaviour. The coefficients , , and determine the relative weights of the corresponding terms.
For evaluation, we report the training curves of the position-distance reward and orientation-distance reward in simulation. For real-world experiments, we report the mean Euclidean position error and mean geodesic orientation error between the current and target relative contact poses over each episode.
In-hand Manipulation Task Setup
Next, we evaluate the contributions of the rich tactile representation and PETS-Net on the dexterous in-hand manipulation task using two 16-DoF dexterous-hand embodiments: the Allegro Hand and the LEAP Hand [38]. In simulation, both hands are equipped with four tactile fingertip models and use the same task formulation, tactile observation modalities, reward design, and teacher–student learning framework. Policies are trained separately for each embodiment using embodiment-specific kinematics, joint limits, action scaling, and initial grasp configurations. The task requires the hand to maintain a stable grasp on an object while inducing continuous rotation about a desired axis.
Our task setting is related to prior dexterous in-hand manipulation formulations [20, 28, 29], but differs in both task randomization and tactile observations. Following AnyRotate [20], we randomize the hand orientation during training and evaluation, rather than considering only a single palm-up hand orientation as in prior in-hand manipulation settings [28, 29]. This makes the task more challenging because the policy must learn gravity-invariant and contact-aware manipulation behaviours across different hand poses. To validate the benefit of rich tactile feedback under more difficult contact conditions, we further apply unknown random external forces and torques to the object during training and evaluation. These perturbations make the manipulation problem more sensitive to contact changes, object motion, and slip-like behaviours.
1) State, Observation and Action Spaces: We primarily investigate the proposed high-dimensional tactile representation, where each fingertip provides simulated contact-depth and tactile-flow observations. The four fingertip observations are tiled into image-like inputs for the student policy, and we evaluate different combinations of proprioception, contact depth, tactile flow, and low-dimensional contact information.
For comparison, we also consider the following observation settings, with detailed dimensions provided in the Appendix:
(a) Privileged State. Used only for teacher-policy training, including object pose and motion, hand orientation, physical properties, and external disturbances.
(b) Proprioception. Following HORA [28], this includes hand joint positions and current joint targets, with short-term inputs for control and longer history for student adaptation.
(c) Low-dimensional Contact Information. Following AnyRotate [20], each fingertip provides local contact position, force magnitude, and a binary contact flag, optionally over a temporal history.
Having defined the policy formulation above, we next introduce the experimental comparisons used to evaluate the proposed tactile modalities and policy architecture. These settings are included purely for ablation and comparison rather than as alternative components of our method.
These observation settings are used only to define the comparisons in our modality ablation experiments, where all policies use PETS-Net and differ only in their observation inputs. We compare proprioception-only policies [28] and existing low-dimensional contact representations [20] against our contact-depth and tactile-flow modalities, evaluated individually and in combination. Separately, we assess the contribution of PETS-Net by replacing it with a standard adaptation module under representative observation settings.
2) Reward Function and Evaluation Metrics: The reward encourages continuous object rotation about the desired axis while penalizing unstable object motion, deviation from the initial grasp posture, joint-target tracking error, and excessive control effort. The per-step reward at time is
where rewards the object angular velocity along the desired rotation axis, while , , , and penalize object translation, deviation from initial grasp posture, joint-target tracking error, and control effort, respectively. The coefficients , , , and determine the relative weights of the corresponding penalty terms.
For evaluation, we report metrics that measure object retention and rotation performance [20, 28]. Specifically, Rotation Count (Rot.) measures the accumulated object rotation about the commanded target axis, expressed as the equivalent number of complete revolutions. For simulation experiments, we report Rot. together with Time-to-Fail (TTF) and Fall Rate. TTF measures the elapsed simulation time before the object is dropped, while Fall Rate denotes the percentage of evaluation episodes in which the object is dropped before the maximum episode duration. For real-world experiments, each trial lasts for at most 600 control steps, corresponding to 30 s of execution. We report Rot. together with Time to Termination (TTT), defined as the elapsed time before the object is dropped, becomes stuck, or the trial otherwise becomes invalid. A trial that remains valid for the full evaluation duration achieves the maximum TTT of 30 s.
Results and Analysis
Simulation Results: Ablation Study of Tactile Modalities
1) Training Performance across Tactile Modalities: We compare our rich, high-dimensional tactile modalities with two baselines: a proprioception-only setting, corresponding to the HORA-style baseline [28], as well as proprioception plus low-dimensional contact information, corresponding to the AnyRotate-style tactile baseline [20]. Experiments show that richer tactile observations substantially improve both convergence speed and final policy performance (Figure 5a).
Figure 5. Ablation studies of tactile observation richness and temporal history in the in-hand manipulation task. (a) Policy input modalities compared with proprioception-only HORA [28] and LD contact AnyRotate [20]. (b) Comparison of contact depth, tactile flow, and their combination. (c) Final reward across modalities and history lengths, with percentages showing gains over the preceding shorter history. Dense tactile observations, particularly tactile flow, approach saturation with shorter histories. Solid curves show the mean over three random seeds, with shaded regions denoting ±1 standard deviation. Values in (c) report mean ± standard deviation across three random seeds.
The proprioception-only policy [28] learns slowly and converges to a much lower reward, indicating that proprioceptive history alone provides limited information about the local fingertip-object interaction. Adding low-dimensional contact information (Anyrotate [20]) improves training, as compact contact position and force cues provide useful information about contact state, which is consistent with the results reported in [20]. However, this representation still loses detailed spatial contact geometry and tangential motion information. In contrast, our rich, high-dimensional tactile observations, including contact depth and tactile flow, lead to faster convergence and higher final reward. Notably, adding low-dimensional contact information on top of the high-dimensional tactile modalities gives only marginal additional improvement, suggesting that the proposed dense tactile observations already capture the key contact information needed for policy learning.
2) Contribution of Contact Depth and Tactile Flow: To assess the contributions of contact depth and tactile flow, we first consider the shear-sensitive dual-arm object-following task.
Figure 6 shows that tactile flow outperforms depth alone. For approximately flat contacts, depth maps mainly capture normal deformation and provide limited information about tangential motion. Tactile flow directly encodes shear displacement and implicitly reflects contact area, improving both translational and rotational tracking. Combining depth and flow accelerates learning but achieves similar final performance to flow alone, indicating that flow is the dominant modality.
Figure 6. Ablation studies of contact depth and tactile flow for the shear-sensitive dual-arm object-following task. Solid curves show the mean over three random seeds, with shaded regions denoting ±1 standard deviation.
We next examine whether this trend extends to the more complex in-hand manipulation task. As shown in Figure 5b, tactile flow outperforms contact depth alone, highlighting the importance of tangential-motion cues for object rotation. Combining both modalities achieves the fastest convergence and highest final reward: depth captures contact geometry, complementing flow that describes fingertip–object shear motion. This advantage is further reflected in the simulated task evaluation (Table I), where combining depth and flow achieves the longest time-to-fail and lowest fall rate both within the training distribution (27.01, s, 11.76%) and on OOD objects (25.08, s, 18.69%), outperforming either modality alone.
| Observations | Within Training Distribution | Out-of-Distribution | ||||
| Time-to-Fail (s) | Rot. Counts | Fall Rate (%) | Time-to-Fail (s) | Rot. Counts | Fall Rate (%) | |
| Oracle Teacher | 27.45 ± 2.13 | 1.88 ± 0.66 | 9.37 | 25.11 ± 3.51 | 1.74 ± 0.81 | 18.09 |
| Prop. | 19.87 ± 4.04 | 0.97 ± 0.88 | 47.66 | 16.64 ± 4.24 | 0.88 ± 0.85 | 57.85 |
| Prop. + LD Contact | 25.18 ± 3.34 | 1.21 ± 0.90 | 21.33 | 22.00 ± 4.09 | 1.19 ± 0.88 | 31.21 |
| Prop. + Flow (Ours) | 25.57 ± 3.30 | 1.24 ± 0.84 | 17.45 | 22.80 ± 3.89 | 1.19 ± 0.91 | 29.82 |
| Prop. + Depth (Ours) | 26.19 ± 3.14 | 1.21 ± 0.89 | 14.26 | 22.27 ± 4.08 | 1.20 ± 0.91 | 29.91 |
| Prop. + Depth + Flow (Ours) | 27.01 ± 2.82 | 1.28 ± 0.85 | 11.76 | 25.08 ± 3.50 | 1.27 ± 0.94 | 18.69 |
Table I. In-hand manipulation performance across tactile observation modalities on within-distribution and out-of-distribution objects in simulation. Results are reported as mean ± standard deviation.
Overall, both tasks demonstrate that tactile flow provides the primary shear-sensitive information needed for contact-rich control, consistently outperforming contact depth alone. Contact depth remains complementary by providing normal contact geometry and distribution cues.
3) Rich Tactile Observations Reduce the Required Temporal Context: We now compare the history lengths within each modality and modalities under fixed history lengths (Figure 5c).
For proprioception-only and low-dimensional contact inputs, longer histories substantially improve learning speed, final reward, and convergence stability. Instantaneous observations provide insufficient information about contact transitions, object motion, and previous action effects, while longer histories allow the policy to reconstruct these interaction dynamics. Performance generally improves up to a history length of approximately 10–15 steps before approaching saturation.
In contrast, policies using contact-depth or tactile-flow observations reach strong performance for only 5–10 history steps. These modalities make the interaction state more directly observable: contact depth describes contact depth and geometry, while tactile flow explicitly captures local shear motion. Their combination therefore provides complementary normal and tangential contact information.
Overall, the results reveal a trade-off between observation richness and temporal context. Sparse or indirect observations require longer histories, whereas dense tactile feedback reduces the memory burden of the temporal adaptor while improving policy performance.
4) Emergent Contact-Aware Pinch-Gaiting Behaviour: As shown in Figure 7, policies trained with rich tactile observations develop a new emergent pinch-gaiting strategy. Two opposing fingers stabilize the object, while the remaining fingers alternately make and break contact to generate rotation. The depth maps capture these contact transitions, whereas the coherent tactile-flow patterns indicate consistent tangential motion during active contact. This structured behaviour was not observed with proprioception-only or low-dimensional contact inputs, suggesting that dense tactile feedback supports coordinated grasp stabilization and fingertip gaiting.
Figure 7. Emergent Contact-Aware Pinch-Gaiting Behaviour with Dense Contact-Depth and Tactile-Flow Feedback. Dotted circles denote in-contact while solid ones denote no contact.
Simulation Results: Ablation Study of PETS-Net
1) Positional Encoding Applied to the Proprioceptive-Action History: We first study the effect of applying positional encoding (PE) to the proprioceptive-action history. As shown in Figure 8a, all PE-based variants outperform the model without PE. The no-PE variant converges more slowly and reaches a lower final reward, indicating that raw proprioceptive-action histories alone are less effective for representing the temporal evolution of hand-object interaction. In contrast, PE-based variants achieve faster reward growth and higher final performance, with dependence on the number of frequency components: is best with competitive. Increasing further does not provide additional benefit, which because higher-frequency encodings increase the feature dimensionality and make the adaptor harder to optimize. These results suggest that a moderate-frequency positional encoding provides useful temporal structure for the adaptor.

Figure 8. Ablation studies of PETS-Net. (a) Effect of positional encoding frequency in the adaptor. (b) Effect of temporal-history encoder structure. (c) Effect of spatial and temporal components. Solid curves show the mean over three random seeds, with shaded regions denoting ±1 standard deviation.
2) Temporal Encoder Ablation: Second, we compare several temporal encoders, including a Temporal Convolutional Network (TCN) [39], a Gated Recurrent Unit (GRU) [40], a Long Short-Term Memory (LSTM) network [41], and a Transformer [42], while keeping the remaining PETS-Net components unchanged. As shown in Figure 8b, the TCN achieves the highest final reward and stable convergence, while the GRU provides comparable but slightly lower performance.
The TCN also provides a favorable memory–performance trade-off in our setting. Policy performance largely saturates with a 10-frame history, for which lightweight temporal convolutions are sufficient to aggregate the short-term contact dynamics encoded by the CNN. The Transformer does not improve performance under this history horizon and requires substantially more VRAM, preventing evaluation with 15- or 20-frame histories on our hardware (specified in Appendix). These results support the use of the TCN as an efficient temporal adaptor for the current tactile representation and training setting, rather than indicating a general advantage over attention-based or recurrent architectures.
3) Ablation on Spatial and Temporal Components: Third, we ablate the spatial and temporal components of PETS-Net using contact-depth and tactile-flow observations. The full model combines CNN-based spatial encoding with temporal-history modeling. We compare it against three reduced variants: temporal history without the spatial encoder, single-step input with spatial encoding, and a single-step baseline without either component. As shown in Figure 8c, the full spatio-temporal model achieves the highest reward. Spatial encoding alone improves over the single-step baseline, confirming that CNN features capture useful contact geometry and deformation. However, its lower performance indicates that a single tactile observation cannot fully represent contact evolution. Conversely, the temporal-history-only variant performs poorly, suggesting that temporal modeling is ineffective without first extracting meaningful spatial features from the high-dimensional tactile maps. Overall, the results support the PETS-Net design. CNN-based encoding extracts spatial contact information, while the TCN captures temporal changes in contact and object motion. Combining both components enables more effective policy distillation and achieves faster convergence and higher final reward than the reduced variants.
Real-to-Sim Tactile Grounding Evaluation
We now conduct real-to-sim experiments to validate whether the proposed simulated tactile representations can be grounded in real-world sensor observations through the proposed tactile transfer pipeline. Specifically, we evaluate whether the real-to-sim translators can recover simulated tactile-flow representations from real observations of various sensors while preserving shear information for downstream motion estimation. The evaluation covers various camera-based tactile sensors: a markerless GelSight Mini [43], DIGIT [44], 9DTact [45], a GelSight Mini with markers, a flat TacTip [4], and a hemispherical TacTip [5] (Figure 1; contact and shear pose ranges given in Appendix C).
For each sensor, we train FlowMotionNet , a CNN regressor that maps simulated tactile flow to planar motion (, , ). The regressor is trained exclusively on simulated data and then frozen. It is evaluated on 1) held-out simulated tactile flow; and 2) on unseen real tactile-image pairs translated by the corresponding real-to-sim model:
The simulated results provide an in-domain reference, while evaluation on translated real inputs measures the performance of the full real-to-sim transfer pipeline. For comparison, we also train and evaluate a separate regressor directly on real tactile image pairs to estimate the same planar motion components. Comparing direct image-based estimation with estimation from the generated simulated tactile flow allows us to assess how introducing the intermediate flow representation affects shear-estimation accuracy.
Table II reports the mean absolute error (MAE) and coefficient of determination () for simulated, real, and real-to-sim inputs, averaged over eight independently trained regressors. Translational results are averaged across the two planar axes. All sensor configurations achieve similar performance on simulated inputs, with translational MAEs of 0.29–0.30 mm/s, rotational MAEs of 1.29–1.42°/s, and .
| Sensor | Press depth | (mm/s) | (°/s) | ||||
| Sim | Real | Real-to-Sim | Sim | Real | Real-to-Sim | ||
| GelSight Mini, markerless | 1 mm | 0.30/0.99 | 1.79/-0.02 | 0.34/0.98 | 1.33/0.99 | 4.50/0.89 | 1.18/0.99 |
| DIGIT, markerless | 1 mm | 0.29/0.99 | 0.64/0.94 | 0.48/0.96 | 1.32/0.99 | 3.61/0.92 | 3.33/0.92 |
| 9DTact, markerless | 1 mm | 0.30/0.99 | 0.45/0.97 | 0.35/0.98 | 1.29/0.99 | 2.02/0.97 | 1.29/0.99 |
| GelSight Mini, markers | 1 mm | 0.30/0.99 | 0.35/0.98 | 0.27/0.99 | 1.30/0.99 | 0.90/1.00 | 0.97/1.00 |
| Flat TacTip | 1 mm | 0.30/0.99 | 0.26/0.99 | 0.24/0.99 | 1.30/0.99 | 1.02/0.99 | 1.00/0.99 |
| Hemispherical TacTip | 1 mm | 0.30/0.99 | 0.27/0.99 | 0.29/0.99 | 1.42/0.99 | 2.09/0.98 | 2.22/0.97 |
| Hemispherical TacTip | 5 mm | 0.29/0.99 | 0.25/0.99 | 0.27/0.99 | 1.32/0.99 | 0.91/1.00 | 1.25/0.99 |
Table II. Tactile-flow-based shear estimation performance for simulated, real, and real-to-sim inputs. Each entry reports MAE/, averaged over 8 independently trained regressors (identical data, stochastic weight initialization/batch order). Sim denotes the in-domain result obtained directly from simulated test flow, Real denotes the result obtained directly from real tactile image pairs, and Real-to-Sim denotes the result obtained from real tactile image pairs translated into simulated tactile flow before shear estimation.
Sensors with explicit markers provide accurate shear estimates directly from real inputs, with relatively small changes following real-to-sim translation. At a 1 mm press depth, the GelSight Mini with markers and flat TacTip achieve real-to-sim translational MAEs of 0.27 and 0.24 mm/s and rotational MAEs of 0.97°/s and 1.00°/s, respectively, with .
In contrast, markerless sensors benefit more substantially from real-to-sim translation. For the markerless GelSight Mini, translation reduces the translational MAE from 1.79 to 0.34 mm/s and the rotational MAE from 4.50°/s to 1.18°/s, while increasing the corresponding values from −0.02 to 0.98 and from 0.89 to 0.99. Similarly, the markerless 9DTact improves from 0.45 to 0.35 mm/s and from 2.02°/s to 1.29°/s. The markerless DIGIT also improves, although its
rotational MAE remains higher at 3.33°/s, with . These results indicate that real-to-sim translation can mitigate the challenges of shear estimation from markerless tactile images, although its effectiveness varies across sensors.
The hemispherical TacTip benefits from deeper contact. Increasing the press depth from 1 to 5 mm maintains similar real-to-sim translational accuracy, with MAEs of 0.29 and 0.27 mm/s, while reducing the rotational MAE from 2.22°/s to 1.25°/s and increasing rotational from 0.97 to 0.99. The same trend is observed for direct real inputs, where the rotational MAE decreases from 2.09°/s to 0.91°/s. This improvement is consistent with deeper contact making rotational shear more observable through larger marker displacements, exploiting the TacTip’s soft hemispherical elastomer and large deformation range.
Figure 10 shows qualitatively similar real-to-sim tactile-flow transfer across the flat TacTip, 9DTact, and GelSight Mini with markers. Although the GelSight Mini achieves slightly better transfer accuracy because its higher surface friction suppresses slip during data collection, we select the hemispherical TacTip

Figure 10. Test-time demonstration of tactile real-to-sim transfer for representative sliding and twisting motions using flat tactile sensors: Flat TacTip, 9DTact, and GelSight Mini with markers. In each example, the large image on the left shows the real-world sensor setup, while the panels on the right show the real tactile images, real vs. generated sparse flow fields, the generated pixel-level dense flow field, and simulated vs. generated sparse flow fields. The results demonstrate real-to-sim tactile-flow transfer across sensors with different sensing principles.
for downstream in-hand manipulation and object-following tasks. Its greater durability and larger elastic deformation range make it better suited to sustained contact-rich interaction, whereas the GelSight Mini is more fragile and constrained in deformation range [5].
Real-world Evaluation
Dual-arm Object-following
We evaluate the learned policy on the real dual-arm object-following task using three objects (Figure 11 top row) and two motion patterns including straight-line and circular motions. Notably, the policy is trained only in simulation using a rigid cube under randomized straight-line motions, while the real-world evaluation includes unseen circular motions and objects with different geometry, compliance, and surface properties. As shown in Table III and Figure 11 (second and third rows), the follower arm consistently tracks the leader motion and maintains stable tactile contact across all tested settings. The best performance is achieved on the rigid cube, which is closest to the simulated training condition (mean position/orientation errors 0.91 mm/0.019 rad and 1.42 mm/0.021 rad for straight-line and circular motion). For the soft ball, the errors increase moderately (1.75 mm/0.036 rad and 2.87 mm/0.054 rad), reflecting the additional difficulty introduced by object compliance. The foam brain is the most challenging object due to its irregular surface texture and more complex contact geometry, resulting in larger errors (4.20 mm/0.041 rad and 6.43 mm/0.057 rad). Nevertheless, the policy remains effective under these out-of-distribution conditions, with all mean position errors below 6.5 mm and orientation errors below 0.06 rad. These results demonstrate that the tactile-flow-based policy trained in simulation can transfer to the real robot and generalize beyond the training distribution in both motion type and object properties.
Figure 11. Real-world object-following trajectories. Top row: rigid cube, soft ball, and foam brain in the experiments. Orange and purple curves denote the leader and follower paths, respectively; red, green, and blue arrows denote the -, -, and -axes of both TCPs’ frames. Second and third rows: straight-line and circular motions.
| Object | Trajectory | Pos. Error (mm) | Orient. Error (rad) |
| Rigid cube | Straight | 0.91 | 0.019 |
| Rigid cube | Circle | 1.42 | 0.021 |
| Soft ball | Straight | 1.75 | 0.036 |
| Soft ball | Circle | 2.87 | 0.054 |
| Foam brain | Straight | 4.20 | 0.041 |
| Foam brain | Circle | 6.43 | 0.057 |
Table III. Mean position and orientation errors in the real-world dual-arm object-following task.
2) In-hand Manipulation: The sim-to-real in-hand rotation policies are evaluated on three sets of out-of-distribution (OOD) objects, covering cylindrical, round, and cube-shaped geometries (Figure 12). Within each category, the objects vary in size, mass, center of mass, surface friction, texture, and compliance. Each method is evaluated from 10 distinct initial grasp configurations, with the maximum episode duration limited to 30 s. We report the number of completed rotations (Rot.) and the time to termination (TTT). The real-world in-hand manipulation experiments are conducted using a symmetric LEAP Hand equipped with four vision-based tactile fingertips [46].

Figure 12. Three sets of OOD objects used for real-world in-hand manipulation evaluation. Cylinder-shape objects (blue box): blue cylinder toy, cylindrical can. Round-shape objects: apple, sweetsop, peach, and brain-shaped foam toy. Cube-shape objects: small paper box, sauce container, chewing-gum container, and supplement bottle. The objects exhibit substantial variations in geometry, size, mass distribution, surface texture, friction, and compliance.
The overall results demonstrate the benefit of high-dimensional tactile observations (Table IV). The policy using contact depth and tactile flow achieves the best rotation performance, completing rotations on average, compared with for contact depth alone, for low-dimensional contact, and for proprioception. Both rich-tactile policies also maintain the objects for the full 30-s episode horizon, whereas the LD-contact and proprioception baselines terminate earlier. These results indicate that contact depth substantially improves grasp maintenance, while tactile flow further improves rotation efficiency by capturing contact-induced tangential motion and shear-related deformation cues.
Table IV. Real-world in-hand rotation performance. Results are reported as mean standard deviation over evaluation trials.
The performance further diversifies across object-shape categories (Table V). The combined depth-and-flow policy achieves the best performance for all three geometries, obtaining , , and rotations on cylindrical, round, and cube-shaped objects, respectively. The advantage of tactile flow is most pronounced for cube-shaped objects, for which the performance increases from with contact depth alone to with depth and flow. This result suggests that tactile flow becomes particularly valuable when the contact geometry changes abruptly and smooth finger-object motion is difficult to maintain.
Table V. Real-world in-hand rotation performance (Rot. ) across different object geometries. Results are reported as mean standard deviation over evaluation trials.
Notably, all policies in this evaluation are trained in simulation exclusively with capsule-shaped objects using the LEAP Hand. Nevertheless, the rich-tactile policies successfully generalize to real cube-shaped objects, despite the substantial geometric difference. Unlike capsules, which provide smooth and continuously curved surfaces, cubes contain flat faces, sharp edges, and corners that can abruptly change the contact normal and intermittently obstruct finger motion. Contact depth provides detailed local geometric information, while tactile flow captures the evolution of contact-induced deformation and relative surface motion. Together, these observations allow the policy to adapt its finger motions to contact conditions that are not encountered during capsule-only training.
In contrast, AnyRotate [20] represents tactile feedback using only a low-dimensional 2D contact pose and force magnitude. In its original setting, both capsule- and cube-shaped objects were included during simulation training to obtain reliable transfer to real cube-shaped objects. Under our more restrictive capsule-only training protocol, the low-dimensional-contact baseline performs worse than the proprioception-only baseline on cube-shaped objects, achieving rotations compared with . The contact poses and force patterns generated near cube edges and corners can differ substantially from those observed during capsule training, causing the low-dimensional policy to encounter out-of-distribution tactile states. This can result in unstable finger motions, contact jamming, and object drops. These results indicate that low-dimensional contact measurements alone are insufficient for robust geometric generalization, whereas high-dimensional contact depth and tactile flow provide a richer representation of previously unseen contact interactions.
As qualitatively illustrated in Figure 13, rotation performance varies substantially with object geometry. The round objects are the easiest to rotate because their shapes are most similar to the capsule-shaped objects used for policy training in simulation. Over the same time interval, the peach and apple rotate by nearly , whereas the cylindrical objects rotate by approximately to . The box-shaped objects are the most challenging, achieving only about to of rotation over the same interval. Their flat faces and sharp edges interrupt smooth rolling and can obstruct finger motion during rotation. Nevertheless, our rich-tactile policy can continuously rotate these box-shaped objects, whereas the evaluated state-of-the-art baselines fail to complete stable rotation. This result highlights the benefit of high-dimensional contact-depth and tactile-flow observations for generalizing to object geometries that differ substantially from those used during simulation training. Videos showing the corresponding manipulation sequences together with the multimodal tactile streams are provided in the supplementary material.

Figure 13. Snapshots of real-world in-hand rotation for example objects with different geometries. From top to bottom, the rows show cylindrical objects in cyan-blue frames (toy cylinder and can), round objects in purple frames (peach and apple), and box-shaped objects in orange frames (pot box and gum container).
Discussion
Tactile Flow for Shear-sensitive Manipulation Tasks
We evaluated simulated tactile flow on a dual-arm object-following task and a substantially more demanding in-hand manipulation task to demonstrate that the proposed representation is structurally necessary for complex contact dynamics.
In the dual-arm object-following task, although tangential interaction is important, the contact geometry remains relatively constrained and low-dimensional. With sufficient engineering effort, it is still possible to design controller rules (e.g., PID controllers) based on explicitly modeled shear quantities or estimated contact states [21], [32]. In such settings, tactile information can be abstracted into interpretable variables, and the control problem remains close to being fully observable under carefully designed representations.
Dexterous in-hand manipulation, however, fundamentally breaks this paradigm. The contact state becomes distributed, multi-point, and time-varying across a high-DoF hand [20], [28], [29], with continual perturbations and subtle object motions. In this regime, explicitly parameterizing contact locations, shear poses, or force vectors becomes combinatorially complex and practically infeasible. More importantly, from a dynamical systems perspective, the task becomes severely partially observable when conditioned on depth-only or instantaneous contact measurements. By incorporating dense, high-resolution tactile flow with temporal integration, the observation more closely approximates a sufficient statistic of the contact dynamics, effectively restoring observability in tasks with distributed tangential interaction. Demonstrating success in in-hand manipulation therefore serves as a decisive validation: tactile flow is not simply an auxiliary cue for structured tracking, but a representation that enables learning in regimes where explicit modeling fails and memoryless policies cannot succeed. This establishes tactile flow as a scalable foundation for shear-dependent, high-complexity manipulation beyond what rigid-body depth-based tactile simulation [4, 5, 6, 7] can support.
Role of Multimodal Tactile Policy Learning
The proposed PETS-Net policy architecture combines positional-encoded proprioceptive histories, high-dimensional tactile observations, and distillation latent adaptation. Compared with prior work that uses compact tactile feedback or analytically designed tactile controllers, PETS-Net learns directly from multimodal tactile observations within a teacher–student reinforcement learning framework. The tactile encoder extracts spatial contact features from contact depth and tactile flow, while the adaptation module converts temporal tactile-proprioceptive histories into a latent representation compatible with the privileged teacher policy. This enables efficient learning and deployment of tactile policies for shear-sensitive tasks such as object following, shear recovery, and dexterous in-hand manipulation.
An important benefit of the proposed tactile representation is the complementarity between contact depth and tactile flow. Contact depth provides dense information about where and how deeply contact occurs, while tactile flow provides motion cues about how the contact evolves tangentially. This is particularly useful in tasks where different contact states may have similar instantaneous depth observations but different future contact evolution. Together this allows our policy learning pipeline to learn control directly from the rich multimodal tactile observations generated by TactileLab.
Real-to-Sim Tactile Transfer for Sim-to-Real Policy Deployment
A key design principle of TactileLab is to use the simulated tactile modalities as a common intermediate representation between real tactile sensing and policy learning. Rather than reproducing the full appearance or deformation mechanics of each physical tactile sensor in simulation, contact depth and tactile flow provide physically interpretable representations of normal contact geometry and tangential contact motion. Policies can therefore be trained directly from these representations at scale in simulation, while sensor-specific real-to-sim translators map real tactile observations into the same observation space during deployment. The cross-sensor real-to-sim validation results show that tactile flow can be recovered from tactile sensors with substantially different sensing geometries and surface features, including marker-based TacTip and GelSight sensors as well as 9DTact. The poor performance of the markerless GelSight Mini also highlights an important practical requirement: tangential deformation must remain observable in the real tactile images for reliable tactile-flow recovery.
Together with the real-world dual-arm object-following and dexterous in-hand manipulation experiments, these results demonstrate the complete transfer pathway from real tactile observations to simulated tactile representations and finally to policies trained entirely in simulation and deployed on physical robots. This separation between sensor-specific perception and policy learning reduces the dependence of the control policy on the appearance and sensing mechanism of a particular tactile sensor. In principle, adapting the framework to a new sensor would primarily require learning a new real-to-sim observation mapping into the same contact-depth and tactile-flow space, rather than redesigning the simulated tactile representation or the downstream policy around a sensor-specific tactile signal.
Limitations and Future Work
1) No-slip Assumption: In the tasks considered in this work, the real-to-sim tactile transfer assumes that the elastomer remains in sticking contact with the object surface, such that contact deformation is dominated by shear rather than slip. While this assumption is generally satisfied once the policy is sufficiently trained to maintain stable contact, external disturbances or abrupt contact changes may violate this condition, potentially leading to unrecoverable states. Closing the sim-to-real gap on slip would require introducing slip-detection capability, which the sensor used in this work has been demonstrated to be effective at [47], although it is an open problem how to include this in tactile simulation.
2) Extension to Soft-body Tactile Simulation: In this work, we investigate simulated tactile flow within a rigid-body simulation framework to obtain an efficient and scalable approximation of shear-relevant contact cues. However, the tactile-flow representation itself is not restricted to rigid-body simulation. The same formulation can be extended to soft-body tactile simulators, where the flow field could be computed from simulated elastomer deformation or rendered tactile image motion. Thus, rigid-body tactile flow provides an efficient abstraction for large-scale policy learning, while soft-body simulators could provide a higher-fidelity source of the same modality when computational cost is less restrictive.
Overall, extending the method to more diverse object shapes will require larger high-dimensional real-to-sim datasets and geometry-aware transfer models. In particular, the model must account for local contact curvature, contact orientation, object shape, and the resulting image-space deformation patterns. This is especially important for tactile flow, since the same tangential contact motion may appear differently depending on the local geometry of the contacting surface. Addressing these challenges could enable a more general tactile interface that maps diverse real sensors and contact geometries into shared physically grounded representations. Such an interface would provide a common basis for learning tactile manipulation policies across sensors, tasks, and robotic embodiments.
References
- [1]Zilin Si, Gu Zhang, Qingwei Ben, Branden Romero, Zhou Xian, Chao Liu, and Chuang Gan. DIFFERTACTILE: A physics-based differentiable tactile simulator for contact-rich robotic manipulation. In The Twelfth International Conference on Learning Representations, 2024.arxiv.org/abs/2403.08716
- [2]Xuyang Zhang, Jiaqi Jiang, Zhuo Chen, Yongqiang Zhao, Tianqi Yang, Daniel Fernandes Gomes, Jianan Wang, and Shan Luo. Simtac: A physics-based simulator for vision-based tactile sensing with biomorphic structures. Cyborg and Bionic Systems, 7:0510, 2026.DOI
- [3]Yuyang Li, Wenxin Du, Chang Yu, Puhao Li, Zhihang Zhao, Tengyu Liu, Chenfanu Jiang, Yixin Zhu, and Siyuan Huang. Taccel: Scaling up vision-based tactile robotics via high-performance gpu simulation. Advances in Neural Information Processing Systems, 38:94577–94604, 2025.arxiv.org/abs/2504.12908
- [4]Alex Church, John Lloyd, Raia Hadsell, and Nathan F Lepora. Tactile sim-to-real policy transfer via real-to-sim image translation. In Conference on Robot Learning, pages 1645–1654, 2022.
- [5]Yijiong Lin, John Lloyd, Alex Church, and Nathan F Lepora. Tactile gym 2.0: Sim-to-real deep reinforcement learning for comparing low-cost high-resolution robot touch. IEEE Robotics and Automation Letters, 7(4):10754–10761, 2022.
- [6]Daniel Fernandes Gomes, Paolo Paoletti, and Shan Luo. Generation of gelsight tactile images for sim2real learning. IEEE Robotics and Automation Letters, 6(2):4177–4184, April 2021.
- [7]Shaoxiong Wang, Mike Lambeta, Po-Wei Chou, and Roberto Calandra. TACTO: A Fast, Flexible, and Open-Source Simulator for High-Resolution Vision-Based Tactile Sensors. IEEE Robotics and Automation Letters, pages 3930–3937, 2022.DOI
- [8]Quan Khanh Luu, Nhan Huu Nguyen, et al. Simulation, learning, and application of vision-based tactile sensing at large scale. IEEE Transactions on Robotics, 39(3):2003–2019, 2023.DOI
- [9]Iretiayo Akinola, Jie Xu, Jan Carius, Dieter Fox, and Yashraj Narang. TacsL: A library for visuotactile sensor simulation and learning. IEEE Transactions on Robotics, 41:2645–2661, 2025.DOI
- [10]An Dang, Jayjun Lee, Mustafa Mukadam, X Alice Wu, Bernadette Bucher, Manikantan Nambi, and Nima Fazeli. Hydroshear: Hydroelastic shear simulation for tactile sim-to-real reinforcement learning. arXiv preprint arXiv:2603.00446, 2026.arxiv.org/abs/2603.00446
- [11]Mayank Mittal et al. Isaac Lab: A GPU-accelerated simulation framework for multi-modal robot learning. arXiv preprint arXiv:2511.04831, 2025.
- [12]Zilin Si and Wenzhe Yuan. Taxim: An example-based simulation model for gelsight tactile sensors. IEEE Robotics and Automation Letters, pages 2361–2368, 2022.arxiv.org/abs/2109.04027
- [13]Zixi Chen, Shixin Zhang, Shan Luo, Fuchun Sun, and Bin Fang. Tacchi: A pluggable and low computational cost elastomer deformation simulator for optical tactile sensors. IEEE Robotics and Automation Letters, 8(3):1239–1246, 2023.DOI
- [15]Yongqiang Zhao, Kun Qian, Boyi Duan, and Shan Luo. Fots: A fast optical tactile simulator for sim2real learning of tactile-motor robot manipulation skills. IEEE Robotics and Automation Letters, 9(6):5647–5654, 2024.arxiv.org/abs/2404.19217
- [17]Baijun Chen, Weijie Wan, Tianxing Chen, Xianda Guo, Congsheng Xu, Yuanyang Qi, Haojie Zhang, Longyan Wu, Tianling Xu, Zixuan Li, Yizhe Wu, Rui Li, Xiaokang Yang, Ping Luo, Wei Sui, and Yao Mu. Univtac: A unified simulation platform for visuo-tactile manipulation data generation, learning, and benchmarking, 2026.arxiv.org/abs/2602.10093
- [18]Jie Xu, Sangwoon Kim, Tao Chen, Alberto Rodriguez Garcia, Pulkit Agrawal, Wojciech Matusik, and Shinjiro Sueda. Efficient tactile simulation with differentiability for robotic manipulation. In 6th Annual Conference on Robot Learning, 2022.
- [19]Max Yang, Yijiong Lin, Alex Church, John Lloyd, Dandan Zhang, David A.W. Barton, and Nathan F. Lepora. Sim-to-real model-based and model-free deep reinforcement learning for tactile pushing. IEEE Robotics and Automation Letters, 8(9):5480–5487, 2023.arxiv.org/abs/2307.14272
- [20]Max Yang, Chenghua Lu, Alex Church, Yijiong Lin, Chris Ford, Haoran Li, Efi Psomopoulou, David AW Barton, and Nathan F Lepora. Annotate: Gravity-invariant in-hand object rotation with sim-to-real touch. In Proceedings of The 8th Conference on Robot Learning, volume 270 of Proceedings of Machine Learning Research, pages 4727–4747. PMLR, 06–09 Nov 2025.arxiv.org/abs/2405.07391
- [21]Kipp McAdam Freud, Yijong Lin, and Nathan F Lepora. Sim-to-real shear-based tactile servoing. In Joseph Lim, Shuran Song, and Hae-Won Park, editors, Proceedings of The 9th Conference on Robot Learning, volume 305 of Proceedings of Machine Learning Research, pages 3401–3412. PMLR, 27–30 Sep 2025.
- [22]Yipai Du, Guanlan Zhang, Yazhan Zhang, and Michael Yu Wang. High-resolution 3-dimensional contact deformation tracking for fingervision sensor with dense random color pattern. IEEE Robotics and Automation Letters, 6(2):2147–2154, 2021.DOI
- [23]Guanlan Zhang, Yipai Du, Hongyu Yu, and Michael Yu Wang. Deltact: A vision-based tactile sensor using a dense color pattern. IEEE Robotics and Automation Letters, 7(4):10778–10785, 2022.
- [24]Niklas Funk, Erik Helmut, Georgia Chaltzavaki, Roberto Calandra, and Jan Peters. Evatec: An event-based optical tactile sensor for robotic manipulation. IEEE Transactions on Robotics, 40:3812–3832, 2024.arxiv.org/abs/2312.01236
- [25]Mauro Comi, Yijong Lin, Alex Church, Alessio Tonioni, Laurence Aitchison, and Nathan F Lepora. Touchsdf: A deepsdf approach for 3d shape reconstruction using vision-based tactile sensing. IEEE Robotics and Automation Letters, 9(6):5719–5726, 2024.arxiv.org/abs/2311.12602
- [26]Zhiyuan Wu, Yijiong Lin, Yongqiang Zhao, Xuyang Zhang, Zhuo Chen, Nathan Lepora, and Shan Luo. Vitaggen: Robotic pushing with vision-to-touch generation. IEEE Robotics and Automation Letters, 10(11):12269–12276, 2025.
- [27]Yijiong Lin, Bowen Deng, Keju Pu, Chenghua Lu, Max Yang, Efi Psomopoulou, and Nathan F Lepora. Neuraltouch: Neural descriptors for precise sim-to-real tactile robot control. IEEE/ASME Transactions on Mechatronics, 2026.DOI
- [28]Haozhi Qi, Ashish Kumar, Roberto Calandra, Yi Ma, and Jitendra Malik. In-hand object rotation via rapid motor adaptation. In Proceedings of The 6th Conference on Robot Learning, pages 1722–1732, 2023.arxiv.org/abs/2210.04887
- [29]Haozhi Qi, Brent Yi, Sudharsan Suresh, Mike Lambeta, Yi Ma, Roberto Calandra, and Jitendra Malik. General in-hand object rotation with vision and touch. In Jie Tan, Marc Toussaint, and Kourosh Darvish, editors, Proceedings of The 7th Conference on Robot Learning, volume 229 of Proceedings of Machine Learning Research, pages 2549–2564. PMLR, 06–09 Nov 2023.arxiv.org/abs/2309.09979
- [30]Zhao-Heng Yin, Binghao Huang, Yuzhe Qin, Qifeng Chen, and Xiaolong Wang. Rotating without seeing: Towards in-hand dexterity through touch. Robotics: Science and Systems, 2023.arxiv.org/abs/2303.10880
- [31]Jiahe Pan, Stelian Coros, Jitendra Malik, and Toru Lin. Beyond binary: Sim-to-real dexterous manipulation with physics-grounded contact representation. arXiv preprint arXiv:2605.28812, 2026.arxiv.org/abs/2605.28812
- [32]John Lloyd and Nathan F Lepora. Pose-and-shear-based tactile servoing. The International Journal of Robotics Research, 43(7):1024–1055, 2024.
- [33]Nasim Rahaman, Aristide Baratin, Devansh Arpit, Felix Draxl er, Min Lin, Fred Hamprecht, Yoshua Bengio, and Aaron Courville. On the spectral bias of neural networks. In International conference on machine learning, pages 5301–5310. PMLR, 2019.
- [34]Ben Mildenhall, Pratul P Srinivasan, Matthew Tancik, Jonathan T Barron, Ravi Ramamoorthi, and Ren Ng. Nerf: Representing scenes as neural radiance fields for view synthesis. Communications of the ACM, 65(1):99–106, 2021.arxiv.org/abs/2003.08934
- [35]Aravind Rajeswaran, Vikash Kumar, Abhishek Gupta, Giulia Vezzani, John Schulman, Emanuel Todorov, and Sergey Levine. Learning complex dexterous manipulation with deep reinforcement learning and demonstrations. In Robotics: Science and Systems, 2018.arxiv.org/abs/1709.10087
- [36]Boyuan Liang, Kei Ota, Masayoshi Tomizuka, and Devesh K Jha. Robust in-hand manipulation with extrinsic contacts. In 2024 IEEE International Conference on Robotics and Automation (ICRA), pages 6544–6550. IEEE, 2024.arxiv.org/abs/2403.18960
- [37]Phillip Isola, Jun-Yan Zhu, Tinghui Zhou, and Alexei A Efros. Image-to-image translation with conditional adversarial networks. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pages 1125–1134, July 2017.arxiv.org/abs/1611.07004
- [38]Kenneth Shaw, Ananye Agarwal, and Deepak Pathak. Leap hand: Low-cost, efficient, and anthropomorphic hand for robot learning. In Robotics: Science and Systems, 2023.arxiv.org/abs/2309.06440
- [39]Shao jie Bai, J. Zico Kolter, and Vladlen Koltun. An empirical evaluation of generic and recurrent networks for sequence modeling. arXiv preprint arXiv:1803.01271, 2018.arxiv.org/abs/1803.01271
- [40]Kyunghyun Cho, Bart van Merriënboer, Çağlar Gülçehre, Dzmitry Bahdanau, Fethi Bougares, Holger Schwenk, and Yoshua Bengio. Learning phrase representations using RNN encoder–decoder for statistical machine translation. In Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing, pages 1724–1734, 2014.arxiv.org/abs/1406.1078
- [41]Sepp Hochreiter and Jürgen Schmidhuber. Long short-term memory. Neural Computation, 9(8):1735–1780, 1997.
- [42]Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin. Attention is all you need. Advances in neural information processing systems, 30, 2017.arxiv.org/abs/1706.03762
- [43]Wenzhen Yuan, Siyuan Dong, and Edward H Adelson. Gelsight: High-resolution robot tactile sensors for estimating geometry and force. Sensors, 17(12):2762, November 2017.DOI
- [44]Mike Lambeta, Po-Wei Chou, Stephen Tian, Brian Yang, Benjamin Maloon, Victoria Rose Most, Dave Stroud, Raymond Santos, Ahmad Byagowi, Gregg Kammerer, et al. DIGIT: A Novel Design for a Low-Cost Compact High-Resolution Tactile Sensor With Application to In-Hand Manipulation. IEEE Robotics and Automation Letters, 5(3):3838–3845, July 2020.DOI
- [45]Changyi Lin, Han Zhang, Jikai Xu, Lei Wu, and Huazhe Xu. 9dtact: A compact vision-based tactile sensor for accurate 3d shape reconstruction and generalizable 6d force estimation. IEEE Robotics and Automation Letters, 9(2):923–930, 2024.arxiv.org/abs/2308.14277
- [46]Wenhao He, Max Yang, Chenghua Lu, Yijiong Lin, David A. W. Barton, and Nathan F. Lepora. Tactile in-hand object rotation: A comparative study of hand morphology in simulation and hardware, 2026.arxiv.org/abs/2610.26759
- [47]Jasper Wollaston James and Nathan F. Lepora. Slip detection for grasp stabilization with a multifingered tactile robot hand. IEEE Transactions on Robotics, 37(2):506–519, 2021.



