betaFeedback

WorksComputer Science

Learning Heterogeneous Tactile Representation for General Dexterous Manipulation

Abstract

Tactile sensing is an indispensable modality for contact-rich dexterous manipulation, a capability that hints at the ultimate frontiers of embodied intelligence. However, commercially available tactile sensors are still at a relatively early stage of development, with each type of sensor offering irreplaceable strengths alongside its own drawbacks. A key challenge is thus to free tactile understanding from sensor-specific physics, so that embodied models can effectively leverage heterogeneous tactile data. To this end, we develop LingBot-Touch, a tactile foundation model characterized by the following properties. (i) Our model yields a unified representation across different tactile sensors (i.e., piezoresistive, magnetic, and visuotactile sensors), and is easily extensible to new sensor types. (ii) The resulting tactile tokens are plug-and-play: they can be seamlessly integrated into policy models across paradigms, from diffusion policy to vision-language-action (VLA) models such as π0.5π_{0.5} and LingBot-VLA, consistently enhancing manipulation dexterity. (iii) Extensive experiments, spanning simulation and real-robot deployments across multiple hardware platforms, demonstrate that our tactile representation substantially benefits complex manipulation tasks while maintaining strong computational efficiency. We believe the findings in this work demonstrate the potential of leveraging cross-sensor data to scale tactile learning, and offer encouraging insights for the design of future hardware and embodied models.

Website: https://technology.robbyant.com/lingbot-touch

Github: https://github.com/robbyant/lingbot-touch

Checkpoints: https://huggingface.co/collections/robbyant/lingbot-touch

Introduction

Dexterous manipulation with multi-fingered hands is among the most remarkable abilities of humans, and giving robots the same ability is a central goal of physical intelligence. Much of that ability rests on touch: with fingertip sensation blocked, people struggle with tasks as simple as striking a match or buttoning a shirt, even with vision intact [16]. Robots, by contrast, still manipulate objects almost entirely by sight [8, 10, 31], and sight cannot tell them how hard they are pressing, whether a grasp is slipping, or what is happening under an occluding finger.

Why not touch? Not for lack of sensors. Piezoresistive, magnetic, and visuotactile sensors are all commercially available (Figure 1, top), but touch remains an early-stage modality that lacks the three ingredients that let vision and language scale. There is no common input format: each sensor type defines its own channels, layout, resolution, and sampling rate [7, 37], so a reading from one sensor means nothing to a model trained on another. There is no large curated corpus, because data from one sensor cannot be pooled with data from the next and every hour requires a sensorized hand touching the world. And there is no established recipe: evidence that touch helps robot learning [14, 37] is scattered across sensor-specific studies and individual tasks, with no shared representation to compare them on.

Overview of LingBot-Touch

Figure 1. Overview of LingBot-Touch. Top: what tactile data look like at scale. Our 5,000-hour corpus is multi-source, pooling egocentric recordings from human tactile gloves with teleoperated robot hands across piezoresistive, magnetic, and optical sensing (left); heterogeneous in taxel geometry, resolution, and channels, which we reconcile in a canonical fingertip space (middle); and sparse, with most taxels silent most of the time (right). Bottom: what one encoder trained on this corpus enables. Frozen features recover contact force, location, extent, and dynamics and align with vision (left); the same tokens plug into imitation-learning and VLA policies on real dual-arm hands (middle) and into reinforcement learning in simulation (right). LingBot-Touch transfers to a sensor model unseen in pretraining without retraining, and its 5.46M-parameter Tiny variant adds under 0.2% to the parameters of the policies it augments.

The three gaps share a root: tactile data are tied to the physics of the sensor that produced them. Contact itself is not. A fingertip pressing on an edge is the same event whether a magnetometer, a resistive grid, or a camera behind a gel records it, so a representation of contact can be shared once sensor-specific physics is removed from the input.

In this paper, we present LingBot-Touch, a tactile foundation model that learns such a representation from the three dominant sensing principles, piezoresistive, magnetic, and optical, with one encoder, and extends to new sensors without retraining. Three design decisions make this possible. We map every sensor’s taxels into a canonical fingertip space, so that a model sees positions on a finger rather than indices in a sensor-specific array (§ 3.2). We train with a joint-embedding predictive architecture [1, 4] that predicts representations of masked taxels in latent space, which suits signals that are sparse and carry no labels (§ 3.2). And we assemble a 5,000-hour corpus from human tactile gloves and teleoperated robot hands, the first at a scale where representation learning for touch becomes possible (§ 3.1).

Trained once on this corpus, LingBot-Touch behaves the way a foundation model should (Figure 1, bottom). Its frozen features recover contact force, location, extent, and short-term dynamics from fingertip signals and retrieve the matching video frame from touch alone, on an optical hand and seen in pretraining and on a magnetic hand whose sensor model was not; on the latter it outperforms FTP-1 [34] fitted with a sensor-specific adapter on all five probing tasks (§ 5.2, § 5.3). Its tokens plug into existing policies without changing them: on 26 real-world dual-arm tasks, adding the frozen 5.46M-parameter encoder raises the success rate of Diffusion Policy [10], π0.5\pi_{0.5} [8], LingBot-VLA [31], and T-Rex [21] by 14 to 26 percentage points, while feeding the same policies raw tactile signals yields at most about half that gain (§ 7). In simulation, reinforcement learning on LingBot-Touch features succeeds on 69% of attempts across twelve dexterous tasks, against 46% with features from a task-specific CNN and 21% without touch, and with far lower variance across seeds (§ 8). The representation also scales: the Small model improves monotonically from 1K to 5K hours of pretraining on every probing task and both hands (§ 5.3), data from one sensor improves the representation of another (§ 5.3), and latent prediction outperforms signal reconstruction by a wide margin, with a VAE objective even degrading as data grow (§ 5.4).

Our contributions are as follows.

  • A tactile corpus at scale. 5,000 hours of recordings, 464K episodes, and 1.33 billion frames across piezoresistive, magnetic, and optical sensing, collected from human tactile gloves and teleoperated robot hands over 24 task categories.

  • A tactile foundation model. LingBot-Touch combines a canonical fingertip space, modality-specific tokenizers, and a shared Haptic Transformer, trained with Tactile-JEPA, a latent-prediction objective adapted to sparse, unlabeled tactile signals through a mask warm-up and layout augmentation. One encoder serves all three sensing principles and transfers to an unseen sensor model without retraining.

  • Evidence across the stack. Frozen-encoder probing of contact physics on two hands, 26 real-world manipulation tasks with four policy families, and twelve reinforcement-learning tasks in simulation, together with scaling, multi-source, and objective analyses that explain where the gains come from.

  • An open release. The inference code, a small set of sample tactile recordings, and checkpoints for the Tiny and Small variants pretrained on approximately 5,000 hours of tactile data will be released. These resources will help researchers integrate the tactile encoder into downstream robot learning systems.

Related Work

Tactile Sensing

Tactile sensors capture contact geometry, forces, and interaction dynamics through diverse transduction mechanisms. Magnetic sensors, such as ReSkin [5] and AnySkin [6], sense contact through deformation-induced changes in magnetic fields; PaXini’s commercial systems also employ Hall-effect sensing [23]. Optical tactile sensors, including GelSight [35], DIGIT [17], and the fingertip sensors integrated into Sharpa Wave [26], use internal imaging to capture contact-induced deformation. Resistive and capacitive sensors encode contact through changes in electrical resistance or capacitance, exemplified by the piezoresistive Scalable Tactile Glove (STAG) [28] and the conformable capacitive electronic skin DexSkin [30].

These technologies observe shared properties of physical contact through different measurement mechanisms, producing signals that vary in channel semantics, spatial resolution, sampling rate, and sensing layout. Similar interactions can therefore yield substantially different observations across sensors. LingBot-Touch addresses this heterogeneity through a common interface that accommodates diverse measurements while preserving contact geometry and temporal structure, enabling unified spatiotemporal contact representations across sensing modalities.

Tactile Representation Learning

Self-supervised representation learning has driven recent advances in large language and multimodal models, contributing to the emergence and development of world models, vision-language-action (VLA) models, and world-action models. Relevant approaches can be broadly grouped into joint-embedding architectures, such as DINO [9], CLIP [24], SigLIP [36], and VideoCLIP [32]; generative architectures, such as MAE [13] and VideoMAE [29]; and joint-embedding predictive architectures, including I-JEPA [1], V-JEPA [4], V-JEPA 2 [2], LeJEPA [3], and V-JEPA 2.1 [20].

Tactile representation learning adapts these approaches to tactile observations. Sparsh [14] investigates reconstruction, self-distillation, and joint-embedding prediction for optical tactile sensing. T3 [37] combines sensor-specific encoders with a shared Transformer backbone to enable transfer across optical tactile sensors and tasks. UniTouch [33] connects tactile representations to pretrained multimodal embeddings, while AnyTouch [11] combines masked modeling, multimodal alignment, and cross-sensor matching to learn static and dynamic tactile features. AnyTouch 2 [12] further emphasizes force perception and temporal dynamics across optical tactile sensors. Sparsh-skin [25] learns contextual representations from magnetic tactile histories and sensor geometry, whereas Sparsh-X [15] integrates image, audio, motion, and pressure signals from Digit 360 [18]. HTT [7] addresses heterogeneous optical and array-based sensors through modality-specific reconstruction and representation alignment between paired sensors. FTP-1 [34] maps heterogeneous observations into morphology-aware tokens for foundation-policy pretraining and supports adaptation to new hardware through sensor-specific encoders.

To address the high sparsity, low signal-to-noise ratios, and multi-source heterogeneity of tactile observations, we introduce LingBot-Touch, a tactile foundation model that combines a multi-branch tokenizer with a shared Haptic Transformer to learn unified, reusable representations of contact physics across sensors and heterogeneous layouts.

LingBot-Touch

LingBot-Touch learns actionable tactile representations for dexterous manipulation policies from data. Two challenges stand in the way. First, tactile data are scarce. Unlike vision and language, no tactile corpus exists today at the scale that representation learning requires. We address this directly by collecting 5,000 hours of tactile recordings (§ 3.1). Because this effort spans human tactile gloves and robot hands with different sensors, the resulting data are heterogeneous in transduction principle, taxel layout, resolution, and sampling rate, which we handle through a canonical geometric interface and modality-specific tokenizers that feed a shared Haptic Transformer (§ 3.2). Second, these recordings come with at most coarse task-level descriptions, and nothing that labels the contact physics inside a clip, such as the forces exerted, where and how widely contact occurs, or when it begins and ends. The model must therefore discover the structure of contact on its own. We address this with a self-supervised objective that predicts contact in latent space, together with a mask warm-up that keeps training stable on sparse signals (§ 3.2), and with layout augmentation that lets the learned representation transfer across sensors (§ 4).

Problem formulation. We represent a normalized tactile clip as X∈RT×N×3\mathbf{X} \in\mathbb{R}^{T \times N \times3}, where TT is the number of frames and NN is the total number of valid taxels across fingers. Each taxel is associated with a three-dimensional position and a finger identity. Sensor layouts may differ in taxel count and arrangement, while single-channel observations are zero-padded to the common three-channel interface. We seek a foundational tactile encoder that maps these observations into a shared representation space, Z=Eθ(g)(X;A)Z = E_{\theta}^{(g)}(\mathbf{X}; \mathbf{A}), where gg selects the sensing-modality branch and A\mathbf{A} denotes the canonical five-dimensional token addresses.

Touch at Scale: 5,000 Hours from Tactile Gloves and Robot Hands

As summarized in Figure 2, our multi-source tactile dataset comprises 4,983 hours of recordings, 464,431 interaction episodes, and approximately 1.33 billion frames from piezoresistive, magnetic, and visuotactile tactile sensors. Reaching this scale is hard because every hour requires a sensorized hand physically interacting with the world. We do so by combining two ways of producing such data. Human operators wearing tactile gloves generate the bulk of the corpus. A glove needs no robot, so one operator can record hours of everyday manipulation per day in ordinary environments, and the resulting egocentric recordings cover a breadth of objects and behaviors that would be slow to script for a robot. We use piezoresistive and magnetic gloves for these recordings. Teleoperation contributes the remainder: a dual-arm system with Sharpa Wave dexterous hands, whose fingertips carry visuotactile sensors, records the same families of tasks from the hand we later deploy on. Gloves account for two thirds of the corpus and teleoperation for the remaining third.

Composition and diversity of the LingBot-Touch pretraining dataset

Figure 2. Composition and diversity of the LingBot-Touch pretraining dataset. Our dataset comprises 4,983 hours of tactile recordings, spanning 464K episodes and 1.33 billion frames across piezoresistive, magnetic, and visuotactile sensing modalities. To support representation learning for downstream understanding of contact physics and robotic dexterous manipulation, it covers diverse real-world human and robot manipulation tasks, encompassing varied sensing principles, spatial resolutions, sampling frequencies, and physical taxel layouts.

The two sources are matched on behavior, not on hardware. Every episode is assigned to one of 24 task categories, from opening, closing, and assembly to wiping, pouring, and tool use (Figure 2), and most categories are covered by more than one source. The sensors, by contrast, differ in transduction principle, taxels per finger, sampling rate, and fingertip geometry (Table 4). Comparable interactions observed through incomparable sensors are what allow a shared encoder to learn regularities of contact that belong to no single sensor, which § 5.3 tests directly.

SensingSensor / vendorPlatformCollectionHoursEpisodesTaxels/fingerRate (Hz)
PiezoresistiveJuqiaoTactile glove (human)Egocentric2,19786,9069100
Xspark AI1660
TACHIN124 / 61†60
MagneticPaXiniTactile glove (human)Egocentric1,136294,871116/96 †41.7-83.3
VisuotactileSharpa Wave fingertipDual-arm robotTeleoperation1,65082,654240×240240 \times24030
Total4,983464,431

Table 4. Composition of the pretraining corpus. Each source is listed with its sensing principle, platform, collection mode, and scale. †TACHIN uses 124 valid taxels on the thumb and 61 on each remaining finger. PaXini uses 116 valid taxels on the thumb and 96 on each remaining finger.

Annotations stop at the episode-level category. No clip carries labels of force, contact location, or contact timing. During pretraining, we sample clips from the piezoresistive, magnetic, and visuotactile sources with equal sampling probabilities, and all supervision comes from the latent prediction objective in § 3.2.

Tactile-JEPA: Predicting Contact in Latent Space

Tactile-JEPA trains the encoder Eθ(g)E_{\theta}^{(g)} by predicting, in latent space, the representation of a complete tactile clip from a partially visible one (Figure 3). Given a clip X\mathbf{X}, we partition its valid positions into a masked set M\mathcal{M} and a visible set C\mathcal{C}, and present only XC\mathbf{X}_{\mathcal{C}} to a context encoder. A target encoder processes the complete clip, and a predictor must map the context encoder’s output to the target encoder’s output at every position, masked or visible. Neither branch reconstructs the signal itself, so the objective is not dominated by the near-zero background that fills most of a tactile clip; § 5.4 compares this choice against signal-space reconstruction. Making the scheme work across sensors requires a token address that is meaningful for any sensor, a backbone shared across sensors except at its boundaries, and masking and loss choices suited to sparse signals. We describe each in turn.

Overview of LingBot-Touch

Figure 3. Overview of LingBot-Touch. LingBot-Touch learns unified tactile representations through self-supervised prediction in latent space. Heterogeneous spatiotemporal signals from magnetic, piezoresistive, and visuotactile sensors are mapped to a canonical spherical space and incorporated into pretraining through weighted sampling. Modality-specific tokenizers convert these signals into tokens for a shared Transformer backbone equipped with 5D rotary positional embeddings (RoPE). The context encoder processes only visible tokens, and its outputs are concatenated with mask tokens before being passed to the corresponding modality-specific predictor. The target encoder processes the unmasked input, providing latent targets for the context loss on visible positions and the prediction loss on masked positions. Its parameters are updated through an Exponential Moving Average (EMA) of the context encoder, with gradients stopped through the target branch.

Canonical fingertip space and token addresses.

Taxel indices describe sensor-local ordering but do not identify corresponding physical locations across sensors. We therefore describe where a measurement was taken by a continuous 3D coordinate and a finger identity, and place the coordinates of every sensor in one canonical fingertip space. Starting from the geometric assets of each sensor, we align the fingertip with a canonical orientation, then scale and translate it into a shared finger-local coordinate frame with a unit-radius reference geometry (Figure 3, bottom left):

pi=αp~i+b,(1)\mathbf{p}_i = \alpha\tilde{\mathbf{p}}_i + \mathbf{b}, \tag*{(1)}

where p~i\tilde{\mathbf{p}}_i is the orientation-aligned position of the taxel associated with token ii, and α>0\alpha> 0 and b∈R3\mathbf{b} \in\mathbb{R}^3 are fixed offline for each sensor–finger template. For PaXini and XHand1, we set α=1/r\alpha= 1/r and b=−o/r\mathbf{b} = -\mathbf{o}/r, where c\mathbf{c} and rr are the center and radius of the minimum enclosing sphere of the taxel positions and reference fingertip geometry. The coordinate origin is o=(ux,cy,uz)\mathbf{o} = (u_x, c_y, u_z), where uu is the designated contact-center anchor and yy denotes the finger direction. Sharpa Wave follows the same height-aligned normalization using its dense surface points, with an additional depth translation matching the PaXini reference sphere. Juqiao and Wujie use fixed PaXini reference radii and contact-region anchors. Tachin uses the maximum isotropic fit within the reference sphere subject to pulp-depth alignment, followed by a scale factor of 0.999. These parameters are reused across temporal windows.

Each valid token then receives a five-dimensional address:

ai=(ti,fi,pi)∈R5,A=(ai)i=1L,(2)\mathbf{a}_i = (t_i, f_i, \mathbf{p}_i) \in\mathbb{R}^5,\qquad\mathbf{A} = (\mathbf{a}_i)_{i=1}^{L}, \tag*{(2)}

where tit_i indexes the token’s temporal window, fif_i identifies the finger, and LL is the number of valid tokens; a taxel keeps the same pi\mathbf{p}_i across windows. The address enters the model only through rotary position embeddings (RoPE) applied to attention queries and keys, with each of the five axes acting on its own slice of the head dimension. This positional encoding does not require a fixed token count or a sensor-specific positional table, allowing a shared backbone to process different sensor layouts and supporting transfer to unseen sensors.

Finally, signal amplitudes are normalized by fixed channel response bounds associated with each sensor’s measurement range: nonnegative normal responses are mapped to [0,1][0,1] and signed tangential responses to [−1,1][-1,1]. Together with the canonical geometry, these dimensionless amplitudes form the common input interface for multi-source learning.

Modality-specific tokenizers and a shared Haptic Transformer.

Sensor-specific parameters are confined to shallow projections at the boundaries of the network. We use three tokenizers, one each for magnetic, piezoresistive, and visuotactile sensing. Each is a two-layer MLP that embeds two consecutive frames of a single valid taxel, six numbers in all, into one token, so tokenization preserves the spatial granularity of the sensor and a clip of TT frames yields L=NT/2L = NT/2 tokens. The tokens then enter the Haptic Transformer (HaT), a Transformer encoder shared across all sources that incorporates time, finger identity, and canonical position through the five-axis RoPE above. The tokenizers extract shallow features from heterogeneous measurements, while successive HaT layers learn shared tactile representations at increasing levels of abstraction. For sensing group gg, we denote the tokenizer together with HaT by Eθ(g)E_{\theta}^{(g)}. The predictor Pϕ(g)P_{\phi}^{(g)} follows the same pattern: group-specific input and output projections around a shared prediction Transformer, with the same five-dimensional positional encoding. Predictions and targets are

z^=Pϕ(g)(Eθ(g)(XC;AC),ΔM;A),zˉ=Eˉθ(g)(X;A),(3)\begin{aligned} \hat{\mathbf{z}} &= P_{\phi}^{(g)}\left(E_{\theta}^{(g)}(\mathbf{X}_{C}; \mathbf{A}_{C}), \Delta\mathcal{M}; \mathbf{A}\right), \\ \bar{\mathbf{z}} &= \bar{E}_{\theta}^{(g)}(\mathbf{X}; \mathbf{A}), \tag*{(3)} \end{aligned}

where arguments after the semicolon specify positional conditioning through RoPE, and ΔM\Delta\mathcal{M} denotes learnable mask tokens placed at the masked addresses. The context encoder sees visible tokens only; the predictor processes visible-token representations and mask tokens jointly and outputs a prediction for both sets. Eˉθ(g)\bar{E}_{\theta}^{(g)} is the target encoder, with the multi-level readout described below. Predictions and targets are aligned by their original token indices.

Targets and objective. Following the multi-level supervision of V-JEPA 2.1 [20], the target encoder processes the complete clip, and we take its features from four depths, normalize each individually, and concatenate them to form zˉ\bar{z}. The context encoder concatenates features from the corresponding layers so that the predictor receives multi-level context. We supervise masked and visible positions alike by minimizing

L=1∣M∣D∑i∈M∥z^i−sg⁡(zˉi)∥1+λ∣C∣D∑i∈C∥z^i−sg⁡(zˉi)∥1,(4)\mathcal{L} = \frac{1}{|\mathcal{M}|D}\sum_{i\in\mathcal{M}}\lVert\hat{z}_i-\operatorname{sg}(\bar{z}_i)\rVert_1 + \frac{\lambda}{|\mathcal{C}|D}\sum_{i\in\mathcal{C}}\lVert\hat{z}_i-\operatorname{sg}(\bar{z}_i)\rVert_1, \tag*{(4)}

where DD is the concatenated feature dimension, sg denotes stop-gradient, and λ=0.5\lambda= 0.5. The prediction loss supervises representations at masked positions, encouraging the model to infer missing contact information from the visible context. The context loss aligns predictions at visible positions with targets computed from the complete clip, encouraging the preservation of observed contact information. The loss is averaged over sampled clips. Gradients update the online tokenizers, HaT, and predictor, including the learnable mask tokens parameterized by ϕ\phi. The target tokenizers and encoder are updated exclusively through an EMA of their online counterparts. For downstream tasks, we freeze the pretrained EMA target tokenizers and encoder. Each input is routed to the appropriate tokenizer branch according to its sensor type and processed by the shared HaT backbone. Token features from four selected HaT layers are individually layer-normalized and concatenated along the feature dimension, providing representations at multiple depths for task-specific probes and control policies.

Masking and mask warm-up. We randomly select valid taxels for masking and extend the same spatial mask throughout the clip, so that M\mathcal{M} contains every token of a masked taxel. The context encoder therefore cannot read a masked taxel’s response at another time step and must infer it from the surrounding taxels and their temporal evolution. The final masking ratio is 80% of valid taxels. Tactile signals are sparse, however, and this ratio cannot be applied from the first step. If contact involves kk taxels and masking is uniform, all kk are hidden with probability about 0.8k0.8^k, so for the few taxels a fingertip typically presses on, a large share of clips present the predictor with a context that contains no contact at all, while its targets, computed from the full clip, do. The loss is then dominated by targets that are unpredictable from the input, and in practice we observe gradient explosions and unstable optimization. We therefore warm up the ratio (Figure 4): it increases linearly from 40% to 80% over the first 300 optimizer steps and stays fixed thereafter. The warm-up spans a negligible fraction of training (we pretrain LingBot-Touch for a total of 200,000 optimizer updates), lowers the chance of erasing all contact evidence while the encoder is still untrained, and then raises prediction difficulty to the level at which the rest of pretraining runs.

Figure 4. Spatial mask-ratio warm-up. The masking ratio increases linearly from 40% to 80% over the first 300 training steps, then remains fixed. Red and green denote masked and visible taxels.

Layout-Agnostic Augmentation

The canonical fingertip space of § 3.2 gives every sensor’s taxels comparable coordinates, but it does not by itself make the encoder indifferent to layout. Pretraining presents exactly three layouts, one per sensor, and RoPE hands the backbone their exact coordinates, so nothing prevents it from memorizing three point sets. A hand it has never seen is a fourth. We therefore treat the layout as a nuisance variable and randomize it during pretraining in two ways: the density at which a fingertip is sampled, and the precise position of each taxel. § 5.3 tests the result on a sensor absent from pretraining, and § 6 ablates the augmentation.

Multiple sampling densities. We expose the model to the same interaction at multiple spatial resolutions. For magnetic sensing, we construct views with 24, 32, 48, 64, 76, and 96 taxels per finger from native layouts containing 116 taxels on the thumb and 96 on each remaining finger. The 96-taxel view uses a fixed subset of the thumb taxels and retains all taxels on the remaining fingers; lower-resolution views are obtained by resampling the native measurements. For visuotactile sensing, we aggregate deformation-map responses into 32, 96, or 200 virtual taxels per finger. Resampling uses local Gaussian weights based on the 3D distances between source and target positions and operates on the observed response channels. For each training clip, we uniformly sample one view from the resolutions available for its source.

We retain the native layouts of the piezoresistive sources. In particular, the Juqiao and Xspark fabric gloves provide coarse fingertip arrays of 3×33 \times3 and 4×44 \times4 taxels, respectively, making further downsampling undesirable because it would discard already limited spatial detail. Across the multi-resolution views, the underlying interaction remains shared while spatial sampling varies, encouraging representations that generalize across resolutions.

Taxel position perturbation. To discourage shortcuts based on exact coordinate templates, we perturb canonical taxel positions by offsets bounded by local spacing:

pi′=pi+δi,∥δi∥2≤0.5 di,(5)\mathbf{p}'_i = \mathbf{p}_i + \boldsymbol{\delta}_i,\qquad\lVert\boldsymbol{\delta}_i\rVert_2 \le0.5\,d_i, \tag*{(5)}

where pi\mathbf{p}_i is the canonical coordinate of Eq. (1) and did_i is the distance from taxel ii to its nearest other valid taxel on the same finger. The bound scales with local sampling density and keeps every displacement inside the taxel’s own neighborhood. Offsets are resampled for each training clip and shared by a taxel across all temporal windows and across the context and target branches, so both branches see the same geometry and the target stays consistent with the context. The perturbed coordinates replace pi\mathbf{p}_i in the address, giving ai′=(ti,fi,pi′)\mathbf{a}'_i = (t_i, f_i, \mathbf{p}'_i). The response sequence and its temporal correspondence are untouched; only the positional encoding sees a nearby but different layout. We report the data augmentation ablations in Table 1. These results show that data augmentation substantially improves LingBot-Touch’s performance on downstream tactile understanding tasks.

AblationSettingForce recovery MAE (N) ↓Center error Mean dist. (mm) ↓Contact coverage MAE (pp) ↓Future force MAE (N) ↓Retrieval Recall@5 (%) ↑
Spatial masking (%)202.0741.1682.1731.52223.875
402.1890.9452.3281.52625.375
602.5741.7623.6711.56724.500
801.6880.2230.8731.45235.375
902.2842.3243.4291.62414.375
Temporal masking (% of clip)[50,50]2.5760.9092.2071.55220.000
[75,75]2.2842.3243.4291.62414.375
[75,100]2.1550.7282.0171.55622.750
[100,100]1.8860.9041.3051.55126.500
Data augmentationValue + position1.5900.2390.9311.47837.125
Value only2.2842.3243.4291.62414.375
Position only1.7020.3050.9881.45330.625
None1.8080.6391.4281.50825.625
Clip length (frames)42.2842.3243.4291.62414.375
82.3832.4074.5191.6559.500
161.8060.6891.3321.52929.250
321.5380.2070.7651.48033.250
Tubelet size (frames/token)12.2842.3243.4291.62414.375
21.6530.2120.8891.40033.375
41.4960.1821.0271.44635.125
82.9312.1934.2181.64813.375
EMA momentum0.998 → 1.07.2213.4018.2211.7281.375
0.999 → 1.01.5290.2711.3611.46334.625
0.99925 (constant)2.2842.3243.4291.62414.375
Tokenizer / predictor interfacesShared / shared2.2842.3243.4291.62414.375
Grouped / group-routed1.6820.7021.1471.56221.750
Training objectiveTarget + context2.2842.3243.4291.62414.375
Target only2.3032.3803.1371.61014.000

Table 1. Table 1. LingBot-Touch ablations on Sharpa Wave. We evaluate frozen Tiny encoders pretrained on a 1K-hour data subset for 100 epochs. Unless varied within a block, the default configuration uses 90% spatial masking, a temporal masking range of [75, 75]%, value augmentation only, 4-frame clips, 1-frame tubelets, constant EMA momentum of 0.99925, shared tokenizer and predictor interfaces, and joint target and context losses. Bold numerical entries indicate the best result within each block. The default configuration serves as a common baseline for fair comparison and does not represent our final, best-performing configuration.

Experiments: Probing the Tactile Representation

This section asks whether LingBot-Touch learns what it was built to learn, and why. Throughout, the pretrained encoder is frozen and read out by lightweight probes, so every number measures the representation rather than a task-specific model. We ask three questions. Does the representation expose the physics of contact, and does it do so on a sensor the model never saw (§ 5.2)? How does it change with more hours, more sensor types, and a larger model (§ 5.3)? Which design decisions matter, from the latent-prediction objective to masking and augmentation (§ 5.4 and § 6)? We close by looking at the latent space itself (§ 6.1). § 7 then asks whether the same frozen representation helps robots act.

Setup

Platforms. All probing experiments are conducted on two dexterous hands. Sharpa Wave uses the visuotactile fingertip sensors that provided the robotic portion of the pretraining dataset, so its sensor is seen during pretraining. Each fingertip produces a deformation map at a native resolution of 240×240240 \times240 and a sampling rate of 30 Hz, which we spatially downsample to 96 taxels for model input. XHand1 uses magnetic fingertip sensors of a model absent from the pretraining dataset, differing from the pretraining gloves in spatial resolution, sampling rate, and taxel placement. Each fingertip has 120 native taxels sampled at 30 Hz. In our experiments, we spatially downsample these observations to 96 taxels per finger, although the model can also directly process all 120 native taxels. XHand1 inputs follow the same processing pipeline as other magnetic sources: taxel positions are mapped into the canonical fingertip space using the sensor geometry (§ 3.2), and the readings are processed by the magnetic tokenizer. No tokenizer or encoder parameters are updated to accommodate the new sensor.

Models. We pretrain two sizes of the Haptic Transformer: LingBot-Touch-Tiny and LingBot-Touch-Small, with 5.46M and 21.76M parameters, respectively. Both use 12 Transformer layers, with embedding dimensions of 192 and 384, respectively. Parameter counts include the tokenizers and shared HaT backbone, excluding the pretraining predictor and downstream task heads. Unless stated otherwise, results in this section use LingBot-Touch-Small pretrained on the full dataset.

Protocol. Following the V-JEPA evaluation protocol [4], we freeze the EMA target tokenizer and target encoder, read out the encoder’s concatenated multi-level features, and train one attentive probe per task: a cross-attention pooling layer followed by a linear head. The tokenizer and encoder receive no task supervision. Table 5 in Appendix lists dataset sizes, inputs, outputs, and metrics for every task, while Table 6 in Appendix provides the detailed probe training budgets and hyperparameters.

TaskInputOutputTrain (h)Test (h)Metric
Fingertip force recoveryTactile clip 16×5×M×316 \times5 \times M \times3Force magnitudes 8×5×18 \times5 \times12.4920.361Force magnitude MAE (N)
Contact center estimationTactile clip 16×5×M×316 \times5 \times M \times33D contact centers 8×5×38 \times5 \times32.6160.334Distance error (mm)
Contact area estimationTactile clip 16×5×M×316 \times5 \times M \times3Coverage ratios 8×5×18 \times5 \times12.6160.334Coverage MAE (%)
Future force predictionTactile clip + joint positions 16×5×M×316 \times5 \times M \times3; 5×J5 \times JFuture force magnitudes 4×5×14 \times5 \times11.8430.484Force magnitude MAE (N)
Tactile-to-video retrievalTactile clip 16×5×M×316 \times5 \times M \times3Similarity scores1.7980.237Top-5 recall (%)

Table 5. Contact-physics probing tasks. Training and test durations, input and output specifications, and evaluation metrics for frozen-encoder probing.

Shared settings
Tactile encoderFrozen pretrained target encoder
Tactile input16 frames; 5 fingers of one hand
Readout blocks / width / heads1 / 192 / 3
MLP hidden width / dropout768 / 0
Optimizer / learning rate / weight decayAdamW / 10−410^{-4} / 0.01
Training epochs / GPUs per run100 / 8
Learning-rate schedule / warm-upConstant / none
Task-specific settings
TaskLoss
Global batchMetric
Additional settings
Force recoveryMagnitude L1
128Magnitude MAE (N)
Attentive readout.
Contact centerEuclidean distance
128Mean distance (mm)
Per-finger training-set z-score coordinates.
Contact coverageCoverage L1
128Coverage MAE (pp)
Sigmoid output; targets in [0, 1].
Future force predictionMagnitude L1
1284-frame magnitude MAE (N)
Measured current/future joint positions and last force magnitude; state/fusion widths 64/64.
Tactile–video retrievalSymmetric InfoNCE
64 pairsRecall@5 (%)
Frozen V-JEPA 2 ViT-L; 16 video frames at 256 × 256; projection dimension 1024; τ=0.07\tau= 0.07.

Table 6. Shared training hyperparameters and task-specific settings for frozen-encoder probing. Sharpa Wave and XHand1 probes are trained separately. Note: Center and coverage share one probe with equal loss weights. Retrieval batches use one pair per recording and share negatives across GPUs. pp: percentage points.

Tasks. Five tasks test four aspects of contact (Figure 5). For contact intensity, fingertip force recovery regresses the resultant contact force in newtons from fingertip signals. For spatial structure, contact center estimation localizes the center of pressure, scored by distance error in millimeters, and contact area estimation predicts the fraction of the fingertip in contact, scored by coverage error in percentage points. For temporal dynamics, future force prediction forecasts the resultant force over the next four frames from the tactile history and the future finger joint states. For cross-modal correspondence, tactile-to-video retrieval finds the video frame recorded at the same instant as a tactile clip, scored by top-5 recall. This retrieval task tests whether touch alone identifies the visual context of an interaction.

Contact-physics probing tasks across fingertip force recovery, contact center estimation, contact area estimation, future force prediction, and tactile-to-video retrieval

Figure 5. Contact-physics probing tasks. Five frozen-encoder probing tasks on two dexterous hands, Sharpa Wave and XHand1: fingertip force recovery, contact center and coverage estimation, future force prediction, and tactile-to-video retrieval. They test contact intensity, spatial structure, short-term dynamics, and cross-modal correspondence. Each task trains its own attentive probe on the multi-level features of the frozen EMA target encoder. For fingertip force recovery, labels are obtained by averaging the per-finger resultant force vectors over each pair of consecutive frames and then computing their magnitudes. Contact centers are computed as contact-strength-weighted centroids of valid taxels above the activation threshold, while contact coverage is the fraction of valid taxels that are active. Future force labels represent per-finger changes in force magnitude at each of the next four frames relative to the final history frame. Contact centers are supervised only when contact is present.

Contact Physics from a Frozen Encoder

Figure 6 reports all five tasks on both hands and compares against FTP-1 [34], a foundation tactile policy that maps heterogeneous sensors into a morphology-aware token space through sensor-specific encoders. FTP-1 is the closest prior encoder in scope, since it is built for heterogeneous sensors and includes a magnetic branch. For FTP-1, we freeze the released shared tactile backbone and train a separate input adapter for each sensor and probe. The Sharpa Wave adapter encodes 96 deformation values per finger, replacing the released image tokenizer, while the XHand1 adapter encodes 120 three-channel taxel measurements per finger. Each adapter is jointly trained with its probe head for 100 epochs on the corresponding downstream training split. For LingBot-Touch, we freeze the target encoder and train only the probe head, without an additional input adapter.

Comparison of FTP-1 and LingBot-Touch across five tactile tasks on Sharpa Wave and XHand1

Figure 6. Comparison with FTP-1 using frozen pretrained backbones. Results on Sharpa Wave (top) and XHand1 (bottom). Gray bars show FTP-1 with sensor-specific adapters trained for each platform. Colored bars show LingBot-Touch with the HaT-Small backbone, pretrained on 5K hours of tactile data and evaluated without sensor-specific encoder adaptation. Arrows indicate whether lower or higher values are better.

On the sensor seen in pretraining. On Sharpa Wave, LingBot-Touch recovers fingertip force to within 0.81 N on sensors with a 0 to 30 N range, locates the contact center to within 0.12 mm, and estimates coverage to within 0.66 percentage points, against 2.31 N, 2.19 mm, and 2.90 points for FTP-1. Future force error falls from 1.67 to 1.04 N. The widest gap is in retrieval, where top-5 recall rises from 16.5% to 72.6%: the representation carries information about the interaction as a whole, not only about the local contact patch. These margins do not depend on the full corpus. With 1,000 hours of pretraining, force and future-force errors are already below 2 N, contact-center error is below 1 mm, and coverage error is below 2 points (Figure 7).

Scaling with pretraining data and model size across tactile tasks

Figure 7. Scaling with pretraining data and model size. Frozen-encoder probing on Sharpa Wave (top) and XHand1 (bottom) for Tiny and Small encoders pretrained on 1K to 5K hours with fixed source proportions. Small improves or holds on every task with every increment of data on both hands; Tiny saturates at about 4K hours and loses retrieval recall at 5K.

On a sensor never seen. On XHand1, with no parameter updated for the sensor, the same encoder reaches 1.32 N, 0.16 mm, 0.43 points, and 1.45 N on the four physical tasks and 47.1% retrieval recall. FTP-1 fitted with an XHand1-specific adapter reaches 3.72 N, 5.89 mm, 4.39 points, 2.00 N, and 6.0%. The spatial tasks show the largest margins: contact-center error on the unseen sensor is more than an order of magnitude below FTP-1’s error on the seen one. Errors on XHand1 are higher than on Sharpa Wave on four of the five tasks, as expected for a sensor with no pretraining exposure, but the ranking of tasks and the size of the gap to FTP-1 are the same on both hands. This is the behavior the canonical fingertip space and the layout augmentation of § 4 were designed to produce, and it separates LingBot-Touch from encoders that need a dedicated adapter or tokenizer for each sensor model.

Scaling the Corpus: Hours, Sensors, and Model Size

The corpus is the lever behind every result in this paper, and it can be turned in three ways: how many hours are used, which sensors they come from, and how large a model absorbs them. We vary each in turn and probe the frozen encoder on both hands.

More hours. We pretrain Tiny and Small on five subsets of the corpus, from 1K to 5K hours with the source proportions fixed (Figure 7). Small improves or holds on every task with every additional thousand hours, on both hands; on Sharpa Wave its force error falls from 1.68 to 0.81 N and its retrieval recall rises from 42.5% to 72.6%. Tiny tracks the same curve to about 4K hours and then flattens, and at 5K hours its retrieval recall drops on both hands. At that point the data are not the limit; the model is, so capacity has to grow with the corpus. Real-world manipulation improves with pretraining hours in the same way, as § 7 reports.

Which hours, at fixed per-sensor data. The corpus mixes three sensing principles, and we ask whether they help one another (Figure 8). Probing uses the four physical tasks; single-source visuotactile models are probed on Sharpa Wave and single-source magnetic models on XHand1. We first hold a sensor’s own data fixed and add the others: a single-source visuotactile model sees 331 hours and a single-source magnetic model 228 hours, while the multi-source model sees those same hours plus 441 hours of piezoresistive data, 1,000 hours in all. The multi-source model improves on seven of the eight sensor-task pairs (Figure 8, top). On the magnetic side every task improves, force error from 4.34 to 2.52 N and future-force error from 3.77 to 1.84 N; on the visuotactile side force, coverage, and future force improve, while contact-center error rises from 0.41 to 0.68 mm.

Single-source versus multi-source pretraining results

Figure 8. Single-source versus multi-source pretraining. Blue: probed on Sharpa Wave. Orange: probed on XHand1. Top: adding 441 hours of piezoresistive data and the other sensor’s data to a fixed 331 or 228 hours of a sensor’s own data improves seven of eight sensor-task pairs. Bottom: at a fixed 1,000-hour budget, a single-source model beats the three-source mixture on its own sensor on seven of eight pairs, by the widest margins on the spatial tasks. Bold marks the better of each pair. LingBot-Touch uses the HaT-Tiny backbone.

Which hours, at fixed total budget. We then give each single-source model 1,000 hours of its own sensor and compare against the same 1,000-hour mixture, in which visuotactile and magnetic data make up 33% and 23% (Figure 8, bottom). Now the single-source model wins on its own sensor on seven of eight pairs: by a wide margin on the spatial tasks (contact center 0.34 versus 0.68 mm for visuotactile, 0.33 versus 1.12 mm for magnetic) and by little on temporal prediction, where the multi-source model is in fact better for visuotactile (1.49 versus 1.67 N). The two settings say the same thing from opposite sides. Force and dynamics transfer across transducers, so hours from another sensor substitute for a sensor’s own when those are scarce. Spatial precision depends on seeing a sensor’s own layout at its own resolution, so at a fixed budget pooling costs some of it, in exchange for one encoder that serves three sensor families.

The sensor that contributed nothing. XHand1 appears in none of these corpora, yet it benefits from every knob. Its probing curves follow the Sharpa Wave curves in shape as hours increase, for both model sizes (Figure 7, bottom), and the magnetic bars of Figure 8 are XHand1 results: a magnetic model given other sensors’ hours gets better on a magnetic sensor it never saw. What improves with more and more varied data is a representation of contact, not of any particular transducer.

Takeaway. Pool every sensor, and scale the model with the data. No tactile sensor today has 5,000 hours of its own, and these experiments show it does not need them: hours from other transducers carry most of what matters for intensity and dynamics, and the remaining gap in spatial precision is the price of a single encoder that transfers to hands outside the corpus. Further, this reframes what matters when scaling tactile data. Because hours transfer across transducers, the sensor to scale with is not the most capable one but the one at the sweet spot between capability and ease of capture, which is why two thirds of our corpus come from gloves a person can wear all day rather than from the fingertips we later deploy on.

Latent Prediction versus Signal Reconstruction

Sec. 3.2 motivates latent-space prediction for sparse tactile signals. We evaluate this choice by pretraining three models on identical multi-source tactile data with matched hyperparameters: a variational autoencoder (VAE) and a masked autoencoder (MAE), both of which reconstruct signals, and LingBot-Touch. Each model is pretrained at two data scales, 1K and 4K hours, and evaluated on the five probing tasks (Figure 9). To ensure comparable model capacity, VAE and MAE each use 5.46M parameters, matching LingBot-Touch with the HaT-Tiny backbone. MAE and LingBot-Touch also use the same masking ratio of 80% in this comparison.

Pre-training losses, robot experiment performance, and probe task frozen evaluation

Figure 9. Latent prediction versus signal reconstruction. VAE, MAE, and LingBot-Touch are pretrained on identical data with matched hyperparameters. Top left: pretraining losses. Bottom: frozen-encoder probing with 1K and 4K hours of pretraining data. Top right: real-world manipulation success rates with the pretrained tactile encoders integrated into LingBot-VLA.

Training dynamics. The VAE loss drops to a low plateau within a few hundred steps and stays there; the MAE loss converges to a small reconstruction error. Neither optimization signal corresponds to a good representation. LingBot-Touch shows a different trajectory: the loss falls, then rises toward a nonzero plateau as the EMA targets evolve, the expected signature of a target that keeps moving. Because the three losses live in different spaces, we compare the representations only through probing.

Probing. LingBot-Touch is best or tied-best on every task at both scales. At 1K hours it cuts the contact-center error of either reconstruction model to less than a third (0.90 mm against 3.03 for the VAE and 3.07 for the MAE) and reaches 32.1% retrieval recall against 4.4% and 5.6%. At 4K hours it improves on all five tasks, to 0.32 mm and 49.3% recall. The reconstruction objectives do not scale the same way. The VAE gets worse with more data on four of five tasks, its contact-center error rising from 3.03 to 4.08 mm and its recall falling from 4.4% to 3.1%; the MAE improves on four tasks but its recall stays below 9%. Reconstructing a signal that is mostly zeros rewards modeling the background, and more background does not help. Predicting in latent space never sees the background as a target.

Ablation Study

Table 1 varies one design choice at a time and probes the frozen encoder on the five Sharpa Wave tasks. All variants use a Tiny encoder pretrained for 100 epochs on the 1K-hour subset, so absolute numbers sit below those of the full models and only comparisons within a block are meaningful.

Masking. An 80% spatial mask is best on all five metrics; against 90% it lowers force error from 2.28 to 1.69 N and raises retrieval recall from 14.4% to 35.4%. V-JEPA reports that higher ratios yield more abstract features; for sparse tactile signals the ratio must instead be balanced against how much contact evidence survives in the context, the same tension that motivates the mask warm-up of § 3.2. Extending the mask across the whole clip beats shorter temporal spans on four of five metrics, confirming that the predictor should not be allowed to copy a masked taxel from a neighboring time step.

Clip length and tubelet size. For visuotactile signals at 30 Hz, 32-frame clips (about 1.07 s) with two or four frames per token (67 or 133 ms) are best; single-frame tokens and 8-frame tubelets are markedly worse. These values are tied to the sampling rate and do not transfer as frame counts to sensors running at other rates.

Augmentation. Value and position augmentation together are best on four of five metrics, raising retrieval recall from 25.6% without augmentation to 37.1%. Position augmentation alone is marginally better on future force (1.45 versus 1.48 N).

EMA momentum. A schedule from 0.999 to 1.0 beats both a schedule from 0.998 and a constant 0.99925 on all five metrics. The 0.998 schedule collapses, with force error above 7 N and recall near 1%, which shows how sensitive latent-target training on sparse signals is to how fast the target moves.

Interfaces and objective. Grouped tokenizers with group-routed predictor interfaces beat fully shared interfaces on all five metrics, cutting contact-center error from 2.32 to 0.70 mm; the sensor-specific boundary layers of § 3.2 earn their place. Adding the context loss to the target loss is close to neutral: three metrics improve slightly and two degrade slightly.

What the Latent Space Encodes

We close by looking at how LingBot-Touch organizes short tactile histories, using the phases of robot manipulation as the lens.

Protocol. We use the Tiny variant of LingBot-Touch pretrained on the full pretraining dataset. Features are extracted from the frozen target encoder using 16-frame windows with a stride of two frames. We average token features within each finger over time and taxels, then concatenate the five 768-dimensional finger features into a 3,840-dimensional representation per window. These representations are projected into two dimensions using UMAP. Each point is colored according to the annotated interaction phase of its corresponding window: non-contact, contact onset, increasing contact force, decreasing contact force, or contact release.

Observations. In every example of Figure 10, windows without contact collapse into one tight cluster away from everything else. That part is expected, since their inputs are near zero; what matters is the structure within contact. Contact windows form smooth, extended curves ordered in time: as finger responses strengthen, weaken, and redistribute across fingers, consecutive windows occupy neighboring positions along the curve, and distinct phases of the same interaction occupy distinct segments of it. The representation therefore preserves continuous variation within a contact episode rather than only the contact versus no-contact distinction, which is what a policy needs in order to time its actions.

Contact separation and continuity in the latent space

Figure 10. Contact separation and continuity in the latent space. Each example pairs finger-wise tactile responses with phase annotations (left) and a UMAP projection of frozen LingBot-Touch features (right). Each point is a 16-frame window of five-finger observations; point colors follow the phase annotations and curve colors identify fingers. Non-contact windows form one cluster, and contact windows form continuous structures that track the strengthening and weakening of the responses.

Experiments: Robot Learning with Frozen Tactile Tokens

§ 5 showed that the representation exposes contact physics to a probe. This section asks whether a robot can use it. Throughout, LingBot-Touch is a drop-in module: the encoder stays frozen, its tokens are appended to whatever the policy already observes, and nothing else about the policy or its training changes. We ask three questions. Does adding the tokens improve imitation-learning and VLA policies on real hardware, and by how much more than adding the raw tactile signal (§ 7.2)? Does it improve reinforcement learning in simulation, in final performance and in the stability of training (§ 8)? And does representation quality, as measured by probing, carry over to the policy when we vary the pretraining data or swap in another tactile encoder (§ 8.1)?

Setup

Platforms. The same two hands as in § 5, each mounted on a dual-arm system: Sharpa Wave, whose visuotactile sensors are seen in pretraining, and XHand1, whose magnetic sensors are not. Both hands are also simulated for the reinforcement-learning experiments, with tactile signals rendered as deformation maps from geometric penetration depth (Sec. C).

Integration. On the real robots a frozen LingBot-Touch-Tiny encoder, 5.46M parameters, encodes 16-frame tactile windows sampled at 30 Hz for each active hand. Trainable cross-attention readouts aggregate its outputs into 40 tokens per hand, which are linearly projected to each policy’s conditioning dimension and enter the policy as an additional conditioning input beside vision and proprioception. The encoder adds 0.16%, 0.15%, and 0.13% to the parameter counts of T-Rex, π0.5\pi_{0.5}, and LingBot-VLA. In simulation, the same frozen encoder processes eight-frame tactile windows, and trainable cross-attention readouts compress its tokens into features concatenated with non-tactile observations as policy inputs (Sec. C.3).

Real-world protocol. We post-train four policies, Diffusion Policy as an imitation-learning baseline and π0.5\pi_{0.5}, LingBot-VLA, and T-Rex as VLA baselines [8, 10, 21, 31], in three configurations each: the original policy without tactile input, the policy with the raw tactile signal added, and the policy with LingBot-Touch tokens added. All three see the same demonstrations, 200 episodes per task and 500 for tasks that need both hands, and each is evaluated over 20 consecutive trials per task. Sec. B describes every task and its success criterion.

Simulation protocol. Twelve dexterous tasks, eight on Sharpa Wave and four on XHand1, follow prior tactile reinforcement-learning suites [19, 22]: Baoding rotation, in-hand rotation, nut assembly, plate spinning, pen spinning, blind grasping, grasp stabilization, and ball balancing on Sharpa Wave, and Baoding rotation, ball balancing, blind grasping, and grasp stabilization on XHand1. Policies are trained with PPO over three seeds and compared on success rate and return under five tactile observations: none, a binary contact indicator per taxel, the raw signal, features from a CNN trained with the policy, and frozen LingBot-Touch features.

Real-World Manipulation with Imitation Learning and VLA Policies

Tasks. The 26 tasks, 20 on Sharpa Wave and 6 on XHand1, are chosen so that touch matters. They include fine manipulation (SD card ejection, pen uncapping, straw insertion), contact-rich actions (cap twisting, spray trigger actuation, zipper closing), fragile and deformable objects (egg transfer, deformable cup transport, bag extraction), tactile discrimination (stiffness sorting, mahjong sorting), and bimanual coordination (box handover, book extraction). Table 2 lists every task and Figure 11 the means.

Table 2. Real-robot success rates on Sharpa Wave and XHand1. Success rates are reported in percent over 20 trials per task. For each policy, we compare the original model, the model augmented with raw tactile signals, and the model augmented with tokens from a frozen LingBot-Touch encoder (Ours). We use the HaT-Tiny variant of LingBot-Touch, pretrained on 4K hours of data. Means weight all 20 Sharpa Wave tasks or all 6 XHand1 tasks equally; bold indicates the highest mean within each policy for each hand.

HandTaskDiffusion Policy BaselineDiffusion Policy + TactileDiffusion Policy + Oursπ0.5\pi_{0.5} Baselineπ0.5\pi_{0.5} + Tactileπ0.5\pi_{0.5} + OursLingBot-VLA BaselineLingBot-VLA + TactileLingBot-VLA + OursT-Rex BaselineT-Rex + TactileT-Rex + Ours
Sharpa WaveSingle-page turning3.7521.2530.0030.0076.2586.2523.7560.0067.5052.5055.0067.50
Sharpa WavePlate wiping24.0044.0047.0068.0074.0076.0074.0081.0083.0050.0075.0075.00
Sharpa WaveStacked cup separation50.0045.0070.0061.2567.5072.5095.0070.0097.5042.5065.0080.00
Sharpa WaveMahjong sorting22.5040.0028.7535.0040.0070.0028.755.0056.2535.0047.5058.75
Sharpa WavePillbox refilling8.7512.5012.5022.5022.5032.5022.5025.0031.2526.2523.7541.25
Sharpa WaveBusiness card extraction20.0032.5056.2550.0050.0075.0042.5061.2575.0065.0057.5075.00
Sharpa WaveCard dealing46.2560.0075.0056.2575.0078.7558.7565.0075.0073.7578.7588.75
Sharpa WaveDeformable cup transport46.2542.5063.7572.5081.2578.7593.3093.7596.7095.0097.50100.00
Sharpa WaveSD card ejection6.0010.0016.0047.0069.0064.0034.0072.0064.0032.0084.0092.00
Sharpa WaveBlock flipping9.008.0010.0067.0084.0090.0034.0048.0075.0029.0073.0083.00
Sharpa WavePen uncapping0.000.000.0036.0052.0059.0033.0048.0065.0022.0033.0061.00
Sharpa WaveCap twisting56.2570.0076.2578.7585.0091.2575.0075.0087.5056.2562.5090.00
Sharpa WaveBag extraction11.0020.0016.0047.5040.0053.7546.2560.5082.5040.0046.2562.50
Sharpa WaveStraw insertion14.0018.0021.0030.0030.0041.0035.0056.0059.0029.0044.0044.00
Sharpa WaveFan opening17.5027.5041.2525.0045.0052.2531.2550.0070.0058.7551.2566.25
Sharpa WaveZipper closing14.009.0022.0039.0015.0059.0011.0016.0055.0015.0017.0042.00
Sharpa WaveSpray trigger actuation30.0032.5028.7551.2560.0071.2542.5060.0087.5028.7565.0076.25
Sharpa WaveStiffness sorting48.0052.5028.7541.2557.5081.2537.5056.2572.5032.5063.7568.75
Sharpa WaveBook extraction11.0038.0041.0036.0062.0076.0036.0064.0081.0050.0069.0059.00
Sharpa WaveBox handover8.0018.0026.0057.0048.0063.0058.0059.0069.0048.0051.0070.00
Sharpa WaveMean success rate (%)22.3130.0635.5147.5656.7068.5845.6056.2972.5144.0657.9970.05
XHand1Single-page turning11.2535.0048.7553.7550.0091.2565.0067.5068.7547.5050.0062.50
XHand1Egg transfer32.5025.0022.5067.5065.0083.7565.0028.7566.2555.0033.7568.75
XHand1Stacked cup separation27.5032.5041.2532.5032.5072.5058.7585.0086.2541.2546.2555.00
XHand1Deformable cup transport11.0023.0026.0052.0048.0054.0037.0058.0090.0044.0053.0063.00
XHand1Stiffness sorting30.0022.5047.5032.5062.5082.5067.5050.0065.0050.0067.5072.50
XHand1Box handover4.006.0028.0060.0030.0061.0050.0054.0074.0062.0061.0061.00
XHand1Mean success rate (%)19.3824.0035.6749.7148.0074.1757.2157.2175.0449.9651.9263.79

Table 2.

Bar chart showing mean success rate over 26 real-world tasks for Diffusion Policy, T-Rex, $\pi_{0.5}$, and LingBot-VLA, comparing Original model, + Tactile, and + Tactile (Ours)

Figure 11. Mean success over 26 real-world tasks for four policies. LingBot-Touch enters only at post-training, as frozen tokens conditioning the policy; it plays no part in any baseline’s pretraining. The raw-tactile configuration is the control.

Across policies. Averaged over the 26 tasks, LingBot-Touch tokens raise the success rate of Diffusion Policy from 21.6% to 35.6%, of π0.5\pi_{0.5} from 48.1% to 69.9%, of LingBot-VLA from 48.3% to 73.1%, and of T-Rex from 45.4% to 68.6%: gains of 13.9, 21.8, 24.8, and 23.2 percentage points for well under a fifth of a percent in parameters. The gain is not confined to a few tasks. LingBot-Touch is the best of the three configurations in 89 of the 104 task-policy pairs and at or above the original policy in 99 of them.

Representation, not modality. The raw-tactile configuration is the control that separates having touch from understanding it. Adding the raw signal helps about half as much at best, 7.0, 6.6, 8.2, and 11.2 points for the four policies, and it hurts in 22 of the 104 pairs, in some cases badly: LingBot-VLA falls from 28.8% to 5.0% on mahjong sorting and from 65.0% to 28.8% on egg transfer, and π0.5\pi_{0.5} from 39.0% to 15.0% on zipper closing. Given 200 demonstrations, a policy cannot learn to read a sparse, high-dimensional sensor from scratch, and the attempt can distract it from the signals it already reads. The frozen tokens arrive already meaning something.

On the hand the encoder never saw. XHand1 makes the point sharply. On its six tasks the raw signal is nearly useless to the policies, changing the mean success of Diffusion Policy, π0.5\pi_{0.5}, LingBot-VLA, and T-Rex by +4.6, −1.7, 0.0, and +2.0 points, while LingBot-Touch tokens computed from the same signal, with no sensor-specific training, add 16.3, 24.5, 17.8, and 13.8 points. Whatever makes an unfamiliar sensor hard for a policy to read, the canonical representation of § 3.2 removes it.

Where the gains come from. The largest per-task gains sit at the step of a task that vision cannot resolve: whether the zipper slider is engaged (LingBot-VLA, 11% to 55%), whether the trigger has been pulled far enough (LingBot-VLA, 42.5% to 87.5%), whether the card has released (T-Rex, 32% to 92%), whether the cap has broken loose (T-Rex, 56% to 90%), and whether the top cup has separated from the stack (π0.5\pi_{0.5} on XHand1, 32.5% to 72.5%). In the recorded runs the tactile-conditioned policies adjust grip and timing at exactly those moments instead of replaying the demonstrated trajectory. LingBot-Touch is not uniformly better: it falls below the raw-signal configuration in 9 of the 104 pairs, five of them for Diffusion Policy, and below the original policy in 5. Sec. B analyzes the behaviors task by task.

Reinforcement Learning in Simulation

Final performance. LingBot-Touch features give the highest success rate and the highest return on all twelve tasks (Table 3). Averaged equally over tasks, success reaches 69.0%, against 45.6% for CNN features trained with the policy, 38.5% for the raw signal, 30.3% for binary contact, and 21.4% with no touch. The ordering is itself informative: a learned representation beats the raw signal, and a pretrained one beats a representation learned inside the task. The margin is widest where contact changes continuously and the policy must keep up. Plate spinning reaches 97.4% compared with 82.9% for the best alternative, while pen spinning reaches 97.1% compared with 64.2%. For ball balancing, the success rate reaches 67.7% on Sharpa Wave versus 32.4% for the best alternative, and 35.9% on XHand1 versus 10.0%. Binary contact and no touch score zero on both hands.

Hand modelTaskMeasureNo TactileBinary TactileRaw TactileTactile CNNOurs
Baoding RotationSuccess (%) ↑34.9 ± 6.838.2 ± 11.435.5 ± 8.744.8 ± 9.355.5 ± 1.8
Reward ↑175.8 ± 5.4162.9 ± 22.9169.3 ± 8.2195.7 ± 32.6212.7 ± 0.3
Sharpa WaveIn-Hand RotationSuccess (%) ↑11.5 ± 13.518.3 ± 31.733.4 ± 19.526.6 ± 23.156.4 ± 7.5
Reward ↑187.7 ± 75.9208.3 ± 75.7188.1 ± 17.0127.9 ± 36.8265.1 ± 28.0
Nut AssemblySuccess (%) ↑28.3 ± 19.921.9 ± 13.555.8 ± 7.540.8 ± 17.386.5 ± 4.2
Reward ↑2269.6 ± 88.01972.4 ± 300.52504.0 ± 234.92383.0 ± 140.82680.4 ± 176.1
Plate SpinningSuccess (%) ↑0.5 ± 0.856.5 ± 40.582.9 ± 14.759.9 ± 36.797.4 ± 2.0
Reward ↑21.2 ± 78.1180.3 ± 218.3286.7 ± 46.6248.8 ± 151.3394.4 ± 17.7
Pen SpinningSuccess (%) ↑31.8 ± 55.164.2 ± 55.630.5 ± 52.862.1 ± 54.097.1 ± 1.1
Reward ↑220.5 ± 342.8417.7 ± 356.2191.3 ± 326.0396.1 ± 339.5623.3 ± 3.0
Blind GraspingSuccess (%) ↑33.7 ± 11.130.8 ± 18.338.6 ± 18.367.6 ± 12.485.2 ± 9.2
Reward ↑−22.6 ± 17.9−23.2 ± 37.8−15.0 ± 41.634.2 ± 29.578.5 ± 18.3
Grasp StabilizationSuccess (%) ↑35.2 ± 14.427.8 ± 9.863.3 ± 9.074.3 ± 15.980.3 ± 4.5
Reward ↑23.9 ± 3.323.3 ± 2.330.6 ± 2.234.0 ± 3.236.6 ± 0.6
Ball BalancingSuccess (%) ↑0.0 ± 0.00.0 ± 0.032.4 ± 4.031.1 ± 11.367.7 ± 1.8
Reward ↑−190.4 ± 2.3−185.1 ± 0.4−71.0 ± 16.2−71.6 ± 51.0127.2 ± 21.1
XHand1Baoding RotationSuccess (%) ↑0.1 ± 0.124.2 ± 2.62.5 ± 1.830.4 ± 6.641.0 ± 2.5
Reward ↑93.6 ± 13.6151.0 ± 5.7115.2 ± 7.0185.2 ± 6.4192.9 ± 14.2
Ball BalancingSuccess (%) ↑0.0 ± 0.00.0 ± 0.010.0 ± 5.90.7 ± 0.635.9 ± 4.9
Reward ↑−184.9 ± 0.3−183.7 ± 0.4−128.8 ± 27.9−171.9 ± 6.1−2.8 ± 17.0
Blind GraspingSuccess (%) ↑75.0 ± 8.772.4 ± 4.055.1 ± 7.277.0 ± 4.987.8 ± 4.2
Reward ↑46.0 ± 25.045.1 ± 10.4−17.7 ± 15.953.1 ± 10.685.5 ± 8.5
Grasp StabilizationSuccess (%) ↑6.0 ± 3.29.7 ± 5.921.7 ± 12.731.8 ± 2.136.6 ± 3.4
Reward ↑10.9 ± 3.111.5 ± 2.316.6 ± 4.818.9 ± 0.720.5 ± 0.7

Table 3. Reinforcement learning on contact-rich dexterous manipulation tasks in simulation. Integrating LingBot-Touch substantially improves training success rates and total rewards, consistently outperforming the compared tactile and non-tactile baselines across tasks on Sharpa Wave and XHand1.

Stability. The standard deviation across seeds tells the other half of the story. For LingBot-Touch it never exceeds 9.2 points and averages 3.9; for the alternatives it reaches 55 on pen spinning and 40 on plate spinning, which means some seeds solve the task and others never do. The learning curves in Figure 12 show the same effect over time: with LingBot-Touch the return rises earlier on pen spinning, on ball balancing on both hands, and on XHand1 blind grasping, and the shaded bands stay narrow. Nut assembly is the exception, with a slower start before the highest final return. A frozen representation that already separates contact states lowers the burden on exploration, and policies that start from it converge both faster and more reliably.

Figure 12. Reinforcement-learning curves on twelve dexterous tasks, Sharpa Wave (a–h) and XHand1 (i–l). Lines show the mean episode return over three seeds; bands show one standard deviation across the smoothed per-seed curves.

Which Representation Helps a Policy

The probing results of § 5 rank representations; this subsection examines whether these rankings translate into policy performance. Using the same LingBot-VLA policy backbone, we post-train each configuration with 200 episodes per task and evaluate it over 20 trials on each of five Sharpa Wave tasks: block flipping, SD card ejection, spray trigger actuation, stacked cup separation, and plate wiping. The policy alone achieves an average success rate of 55.9%, which increases to 82.2% when augmented with tactile representations from the HaT-Tiny of LingBot-Touch (Figure 13).

Bar charts showing mean success rates by pretraining hours and tactile encoder across five tasks

Figure 13. Which representation helps a policy? We evaluate LingBot-VLA post-trained on five Sharpa Wave tasks, with 20 trials per task. (a) Mean success rate as the LingBot-Touch encoder’s pretraining data increases from 1K to 5K hours. (b) Per-task success rates with the FTP-1 tactile encoder versus LingBot-Touch in the same policy. LingBot-Touch uses the HaT-Tiny configuration and is pretrained on the full 5K-hour dataset.

Pretraining hours. Encoders pretrained on 1K to 5K hours lift the policy to 69.8%, 74.3%, 77.4%, 81.4%, and 82.2% (Figure 13, left). Every additional thousand hours of tactile data, none of it from these tasks or this robot’s demonstrations, makes the downstream policy better, and the curve has the same shape as the probing curves of § 5.3.

Another encoder. Replacing LingBot-Touch with the tactile encoder of FTP-1 in the same policy yields 67.5%, less than half the gain (Figure 13, right). FTP-1 helps on three tasks but lowers success on stacked cup separation, 95% to 85%, and plate wiping, 74% to 67%; LingBot-Touch is at or above the original policy on all five. The probing gaps of § 5.2, where FTP-1’s errors are several times larger, show up as policy gaps.

Conclusion and Discussion

We presented LingBot-Touch, a tactile foundation model that learns a shared representation of contact from piezoresistive, magnetic, and visuotactile sensors. A canonical fingertip space gives heterogeneous sensors one input interface, and Tactile-JEPA trains a shared Haptic Transformer by predicting contact in latent space. Pretrained once on 5,000 hours of recordings from tactile gloves and teleoperated robot hands, the encoder is frozen and used as is.

Three findings follow. The representation belongs to contact rather than to a transducer: without any parameter update, the encoder recovers contact physics on a magnetic hand absent from pretraining and outperforms FTP-1 fitted with a sensor-specific adapter. The gain for robot learning comes from the representation, not the modality: frozen tokens improve imitation learning, VLA, and reinforcement learning policies across 26 real-world and twelve simulated tasks, while the raw signal helps at most half as much and often hurts. And the representation keeps improving with hours, sensor families, and model size, where signal reconstruction stalls.

These results suggest how to scale touch. No single sensor has thousands of hours, and it does not need them: intensity and dynamics transfer across sensing principles, so data can be collected with whatever sensor is easiest to record and spent on whatever hand is deployed, at some cost in spatial precision. We hope this informs the design of future tactile hardware and of embodied models that treat touch as a first-class input.

Limitations. First, real-world and simulation experiments need to be conducted using a wider variety of hand embodiments. Second, sensor coverage can be extended further. Capacitive sensors are currently absent from pretraining because the capacitive recordings available to us did not meet our quality requirements. Finally, the canonical geometry and pretraining layouts focus on the fingertips, leaving contacts on the palm and elsewhere along the fingers underrepresented. Extending layout coverage from the fingertips to the whole hand is an important next step.

Appendix

Implementation Details

This section provides additional details on the pretraining corpus and the frozen-encoder probing setup. Table 4 summarizes the data sources, including their sensing principles, collection platforms and modes, dataset sizes, spatial resolutions, and sampling rates. Table 5 specifies the five contact-physics probing tasks, together with their input and output representations, training and test data durations, and evaluation metrics. Table 6 reports the shared probe architecture and optimization settings, as well as task-specific losses, batch sizes, and additional configurations. Throughout probing, the pretrained target encoder remains frozen, and probes are trained separately for Sharpa Wave and XHand1.

Real-Robot Evaluation

Table 7 summarizes task descriptions, hand coverage, and scoring criteria. Check marks indicate evaluated hand configurations. Numbers in parentheses denote points; allocations are additive unless an ordinal scale is stated, in which case the highest level reached determines the score. Figure 14 and Figure 15 illustrate two of the evaluated tasks. Each image contains five representative frames ordered from left to right.

Five representative frames showing stacked cup separation

Figure 14. Stacked cup separation (XHand1). The robot stabilizes the nested stack, grasps and separates the uppermost cup, and holds the separated cup independently.

Five representative frames showing spray trigger actuation

Figure 15. Spray trigger actuation (Sharpa Wave). The robot grasps the spray bottle, aims the nozzle toward the target, and actuates the trigger to discharge water.

TaskSharpa WaveXHand1DescriptionSuccess criterion
Single-page turning✓✓Turn one page right to left.1 point each: approach edge, move page by contact, cross spine with one page, complete turn. No tearing. Total 4.
Plate wiping✓—Wipe a stain from a stabilized plate.1 point each: lift cloth; stabilize plate; touch stain; clean; return cloth and withdraw hands. Plate intact and stable. Total 5.
Stacked cup separation✓✓Separate the uppermost nested cup.1 point each: stabilize stack; grasp top cup; separate; hold 2 s. Exactly one cup, no damage or stack disturbance. Total 4.
Mahjong sorting✓—Sort a mahjong tile by touch.Grasp (1); correct tray (+3) or wrong tray (+1); maximum 4. Face exposure to the policy camera during identification invalidates the trial.
Pillbox refilling✓—Refill a compartment and close its lid.1 point each: approach latch; release; grasp surrogate; deposit; close lid. Surrogate retained, lid latched. Total 5.
Business card extraction✓—Extract and retain a business card.1 point each: align; pinch; extract; hold stably after extraction. Total 4.
Card dealing✓—Separate and lift one playing card.Ordinal: approach (1), contact (2), slide out (3), grasp and lift (4).
Deformable cup transport✓✓Transport a thin-walled cup without damage.Credit is based on contact, grasp stability, transport, deformation control, placement. Full credit requires a grasp of the outer sidewalls, no excessive deformation, and upright release in the target plate.
SD card ejection✓—Eject and remove an SD card shell.Approach (1); press and withdraw finger to unlock (2); grasp and remove shell completely (2). Total 5.
Block flipping✓—Reorient a cube to the target face.Grasp (1); finger reorientation (2); container placement (1); target face up (1). Wrist rotation alone earns no reorientation credit. Total 5.
Pen uncapping✓—Remove a pen cap bimanually.1 point each: hold barrel; lift between hands; approach cap; grasp cap stably; remove and retain cap. Control both parts. Total 5.
Cap twisting✓—Unscrew and remove a bottle cap.Ordinal: approach (1), grasp and rotate effectively (2), fully unscrew (3), lift clear (4).
Bag extraction✓—Pull a bag from its dispenser.1 point each: stabilize box; approach bag; stable pinch; lift visibly out. Total 4.
Straw insertion✓—Insert a straw through a sealed lid.Credit is based on prescribed grasp, film contact and depression, puncture, prescribed depth. No buckling, tipping, or cup-bottom contact.
Fan opening✓—Activate and orient a fan.1 point each: stabilize fan; touch button; press to activate; orient airflow to move flag. Total 4.
Zipper closing✓—Close a pouch zipper.1 point each: lift and stabilize pouch; approach; grasp slider; close zipper; release into container. Zipper stays closed. Total 5.
Spray trigger actuation✓—Aim and operate a spray bottle.Grasp (1); aim (1); trigger stage: actuation (1) or visible water discharge (2). Maximum 4.
Stiffness sorting✓✓Probe and sort sponges by stiffness.Credit is based on press–release, correct category, placement in the matching tray. Full credit requires agreement with the predefined sample category.
Book extraction✓—Extract a tightly shelved book.Edge contact (1); partial pull (2); thumb–finger grasp (1); fully lift clear (1). Total 5.
Box handover✓✓Transfer a box between hands.Credit is based on independent lift, shared grasp, full support transfer, target placement. Full credit requires an independent receiving-hand hold and stable placement without dropping.
Egg transfer—✓Transfer an egg replica between holders.1 point each: grasp; lift clear; position above target; release. Egg stably seated. Total 4.

Table 7. Task descriptions, hand coverage, and scoring criteria for the real-robot evaluation.

Simulation-Based Evaluation of Tactile Representations

Simulation Probing

Since LingBot-Touch is pretrained on multi-source real-world tactile data, its ability to generalize to simulated tactile observations remains to be evaluated. Before conducting reinforcement learning experiments, we perform simulation-based probing to assess whether the pretrained representations preserve contact-related physical information in the unseen simulation domain. We collect 7,200 seconds of simulated single-hand grasping data on the Sharpa Wave platform, covering tactile interactions with 47 objects across diverse grasp poses. We conduct four probing tasks: fingertip force recovery, contact center estimation, contact area estimation, and future force prediction. We use the same training hyperparameters and evaluation metrics as in the real-world probing experiments.

As shown in Figure 16, the contact center and contact area prediction results indicate that the pretrained LingBot-Touch representations retain decodable spatial contact information in simulation. With the encoder frozen, the probe heads recover contact locations, spatial distributions, and coverage. Across the complete test set, contact-positive regions achieve a mean Dice score of 0.708 and an area MAE of 1.089 percentage points. These results suggest that LingBot-Touch representations support the recovery of simulated contact geometry despite the distribution shift between simulated tactile observations and real-world sensor measurements.

Figure 16. Simulation-based contact physics probing. We conduct the same probing tasks as in the real-world evaluation on the Sharpa Wave platform to assess the generalization of LingBot-Touch representations to simulated tactile data. The pretrained encoder remains frozen, and task-specific attentive probes are trained independently.

The fingertip force recovery and future force prediction probes further assess force-magnitude and temporal information. The current-force estimates capture loading, holding, and unloading trends, although deviations in absolute magnitude remain, with an overall MAE of 3.264 N. Future force-change prediction achieves an MAE of 0.459 N, indicating that historical tactile features contain information useful for predicting short-term force changes; the absolute future forces shown in the figure are reconstructed by adding the predicted changes to the observed current force. Differences in response mechanisms and contact dynamics between Isaac Sim’s deformation-based tactile observations and real-world tactile measurements contribute to the absolute-force estimation errors in simulation. Overall, the four probing experiments indicate that frozen LingBot-Touch representations retain useful spatial contact, force-magnitude, and short-term temporal information in the simulation domain, providing a basis for using LingBot-Touch representations in subsequent simulation-based reinforcement learning experiments.

Simulated Dexterous Manipulation Tasks

We construct eight types of dexterous manipulation tasks and conduct reinforcement learning experiments in simulation using two dexterous hands, Sharpa Wave and XHand1. Table 8 summarizes the hand-model coverage, description, and success criterion for each task.

TaskSharpa WaveXHand1DescriptionSuccess criterion
Baoding Rotation✓✓Coordinate two objects of varying shapes and sizes within the palm to repeatedly exchange their target positions.At least 10 target exchanges.
In-Hand Rotation✓—Coordinate finger motions to rotate objects spanning nine shapes and three scales about a prescribed axis.At least 1 net turn about the prescribed axis.
Nut Assembly✓—Turn hexagonal nuts of four sizes through finger contacts under the thread constraint of a fixed bolt.At least 5 net turns from the initial angle at episode end.
Plate Spinning✓—Support a square plate and spin it about the vertical axis while controlling tilt and maintaining supporting contacts.Complete 3 valid net turns within 40 s.
Pen Spinning✓—Coordinate finger contacts to spin a slender capsule-shaped pen continuously while maintaining support and preventing drops.Complete 3 valid net turns within 12 s.
Blind Grasping✓✓Without vision, locate, grasp, lift, and hold cubes and cylinders with varied sizes and randomized poses.Maintain a lift of ≥6\geq6 cm with opposing contact for 2 s.
Grasp Stabilization✓✓Stabilize a grasped cuboid against external forces and torques while reducing contact load and preventing drops.Retain for 12 s without failure; mean total normal load ≤10\leq10 N.
Ball Balancing✓✓Control tray orientation with the hand to balance a rolling ball and prevent it from falling.Retain the ball for 10 s without task failure.

Table 8. Eight simulated task types, hand-model coverage, and episode-level success criteria.

Reinforcement Learning Experimental Settings

Simulation-based Tactile Generation. We generate simulated tactile observations following the geometry-based penetration depth method of TacMap [27]. Specifically, sensing points and their corresponding surface normals are predefined on the curved fingertip surfaces. During simulation, GPU-parallelized ray queries are performed against the object meshes to estimate local penetration depths along the surface normals. These measurements are then assembled into spatial deformation maps that represent the contact distribution and local fingertip deformation.

Tactile Observation Processing. To accommodate different fingertip geometries, we construct tactile observations according to the sensing layouts of Sharpa Wave and XHand1. Specifically, Sharpa Wave uses 96 sensing points per finger, while XHand1 uses 120 sensing points per finger. The simulated deformation measurements are normalized, and 8 consecutive frames are stacked to capture temporal contact dynamics.

Tactile Observation Configurations. To evaluate the contribution of different tactile representations to reinforcement learning, we consider five observation configurations: No Tactile, Binary Tactile, Raw Tactile History, Tactile CNN, and the pretrained LingBot-Touch representation. The No Tactile baseline uses only non-tactile observations, including proprioceptive information. Binary Tactile provides thresholded contact indicators derived from the deformation measurements, while Raw Tactile History directly uses the temporal sequence of deformation observations. Tactile CNN employs a trainable tactile encoder optimized jointly with the policy. For LingBot-Touch, the simulated deformation observations are processed by the frozen pretrained tactile encoder, followed by a trainable readout module that produces features for downstream control. For the pretrained encoder, the scalar deformation measurements are placed in the third channel of the three-dimensional tactile input, while the first two channels are set to zero. The resulting tactile features are concatenated with the corresponding non-tactile observations and provided to the policy network.

Reinforcement Learning Protocol. Within each manipulation task, all observation configurations are evaluated under the same reinforcement learning framework, with consistent action spaces, reward functions, environment randomization, and task-specific training settings. The primary difference between configurations lies in how tactile information is represented and processed before being incorporated into the policy. This controlled comparison allows us to assess the influence of tactile representations on policy learning efficiency, training stability, and final manipulation performance.

References

  1. [1]Mahmoud Assran, Quentin Duval, Ishan Misra, Piotr Bojanowski, Pascal Vincent, Michael Rabbat, Yann LeCun, and Nicolas Ballas. Self-Supervised Learning From Images With a Joint-Embedding Predictive Architecture. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 15619–15629, 2023.arxiv.org/abs/2301.08243
  2. [2]Mido Assran, Adrien Bardes, David Fan, Quentin Garrido, Russell Howes, Mojtaba Komeili, Matthew Muckley, Ammar Rizvi, Claire Roberts, Koustuv Sinha, Artem Zholus, Sergio Arnaud, Abha Gejji, Ada Martin, Francois Robert Hogan, Daniel Dugas, Piotr Bojanowski, Vasil Khalidov, Patrick Labatut, Francisco Massa, Marc Szafraniec, Kapil Krishnakumar, Yong Li, Xiaodong Ma, Sarath Chandar, Franziska Meier, Yann LeCun, Michael Rabbat, and Nicolas Ballas. V-JEPA 2: Self-supervised video models enable understanding, prediction and planning. arXiv preprint arXiv:2506.09985, 2025.arxiv.org/abs/2506.09985
  3. [3]Randall Balestriero and Yann LeCun. LeJEPA: Provable and Scalable Self-Supervised Learning Without the Heuristics. arXiv preprint arXiv:2511.08544, 2025.arxiv.org/abs/2511.08544
  4. [4]Adrien Bardes, Quentin Garrido, Jean Ponce, Xinlei Chen, Michael Rabbat, Yann LeCun, Mahmoud Assran, and Nicolas Ballas. Revisiting feature prediction for learning visual representations from video. Transactions on Machine Learning Research, 2024. arXiv:2404.08471.arxiv.org/abs/2404.08471
  5. [5]Raunaq Bhirangi, Tess Hellebrekers, Carmel Majidi, and Abhinav Gupta. ReSkin: versatile, replaceable, lasting tactile skins. In Proceedings of the 5th Conference on Robot Learning, volume 164 of Proceedings of Machine Learning Research, pages 587–597. PMLR, 2022.
  6. [6]Raunaq Bhirangi, Venkatesh Pattabiraman, Enes Erciyes, Yifeng Cao, Tess Hellebrekers, and Lerrel Pinto. AnySkin: Plug-and-Play Skin Sensing for Robotic Touch. In 2025 IEEE International Conference on Robotics and Automation (ICRA), pages 16563–16570, 2025.
  7. [7]Jianxin Bi, Qiang Wang, Jayaram Reddy, Kelvin Lin, Soibkhon Khajikhahnov, Ruihan Gao, and Harold Soh. Heterogeneous Tactile Transformer. In Conference on Robot Learning (CoRL), 2026. Accepted to CoRL 2026.arxiv.org/abs/2606.29948
  8. [8]Kevin Black, Noah Brown, James Darpinian, Karan Dhabalia, Danny Driess, Adnan Esmail, Michael Robert Equi, Chelsea Finn, Niccolo Fusai, Manuel Y. Galliker, Dibya Ghosh, Lachy Groom, Karol Hausman, brian ichter, Szymon Jakubczak, Tim Jones, Liyming Ke, Devin LeBlanc, Sergey Levine, Adrian Li-Bell, Mohith Mothukuri, Suraj Nair, Karl Pertsch, Allen Z. Ren, Lucy Xiaoyang Shi, Laura Smith, Jost Tobias Springenberg, Kyle Stachowicz, James Tanner, Quan Vuong, Homer Walke, Anna Walling, Haohuan Wang, Lili Yu, and Ury Zhilinsky. π₀.₅: a vision-language-action model with open-world generalization. In Joseph Lim, Shuran Song, and Hae-Won Park, editors, Proceedings of The 9th Conference on Robot Learning, volume 305 of Proceedings of Machine Learning Research, pages 17–40. PMLR, 27–30 Sep 2025.arxiv.org/abs/2504.16054
  9. [9]Mathilde Caron, Hugo Touvron, Ishan Misra, Hervé Jégou, Julien Mairal, Piotr Bojanowski, and Armand Joulin. Emerging Properties in Self-Supervised Vision Transformers. In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), pages 9650–9660, 2021.arxiv.org/abs/2104.14294
  10. [10]Cheng Chi, Siyuan Feng, Yilun Du, Zhenjia Xu, Eric Cousineau, Benjamin Burchfiel, and Shuran Song. Diffusion policy: Visuomotor policy learning via action diffusion. In Proceedings of Robotics: Science and Systems (RSS), 2023.
  11. [11]Ruoxuan Feng, Jiangyu Hu, Wenke Xia, Tianci Gao, Ao Shen, Yuhao Sun, Bin Fang, and Di Hu. Learning Unified Static-Dynamic Representation across Multiple Visuo-tactile Sensors. In The Thirteenth International Conference on Learning Representations, 2025.arxiv.org/abs/2502.12191
  12. [12]Ruoxuan Feng, Yuxuan Zhou, Siyu Mei, Dongzhan Zhou, Pengwei Wang, Shaowei Cui, Bin Fang, Guocai Yao, and Di Hu. AnyTouch 2: General Optical Tactile Representation Learning For Dynamic Tactile Perception. In The Fourteenth International Conference on Learning Representations, 2026.arxiv.org/abs/2602.09617
  13. [13]Kaiming He, Xinlei Chen, Saining Xie, Yanghao Li, Piotr Dollár, and Ross Girshick. Masked Autoencoders Are Scalable Vision Learners. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 16000–16009, 2022.arxiv.org/abs/2111.06377
  14. [14]Carolina Higuera, Akash Sharma, Chaithanya Krishna Bodduluri, Taosha Fan, Patrick Lancaster, Mrinal Kalakrishnan, Michael Kaess, Byron Boots, Mike Lambeta, Tingfan Wu, and Mustafa Mukadam. Sparsh: Self-supervised touch representations for vision-based tactile sensing. In Proceedings of The 8th Conference on Robot Learning, volume 270 of Proceedings of Machine Learning Research, pages 885–915. PMLR, 2025.arxiv.org/abs/2410.24090
  15. [15]Carolina Higuera, Akash Sharma, Taosha Fan, Chaithanya Krishna Bodduluri, Byron Boots, Michael Kaess, Mike Lambeta, Tingfan Wu, Zixi Liu, Francois Robert Hogan, and Mustafa Mukadam. Tactile Beyond Pixels: Multisensory Touch Representations for Robot Manipulation. In Proceedings of The 9th Conference on Robot Learning, volume 305 of Proceedings of Machine Learning Research, pages 105–123. PMLR, 2025.
  16. [16]Roland S. Johansson and J. Randall Flanagan. Coding and use of tactile signals from the fingertips in object manipulation tasks. Nature Reviews Neuroscience, 10(5):345–359, 2009.DOI
  17. [17]Mike Lambeta, Po-Wei Chou, Stephen Tian, Brian Yang, Benjamin Maloοn, Victoria Rose Most, Dave Stroud, Raymond Santos, Ahmad Byagowi, Gregg Kammerer, Dinesh Jayaraman, and Roberto Calandra. DIGIT: A Novel Design for a Low-Cost Compact High-Resolution Tactile Sensor With Application to In-Hand Manipulation. IEEE Robotics and Automation Letters, 5(3):3838–3845, 2020.DOI
  18. [18]Mike Lambeta, Tingfan Wu, Ali Sengul, Victoria Rose Most, Nolan Black, Kevin Sawyer, Romeo Mercado, Haozhi Qi, Alexander Sohn, Byron Taylor, Norb Tydingco, Gregg Kammerer, Dave Stroud, Jake Khatha, Kurt Jenkins, Kyle Most, Neal Stein, Ricardo Chavira, Thomas Craven-Bartle, Eric Sanchez, Yitian Ding, Jitendra Malik, and Roberto Calandra. Digitizing Touch with an Artificial Multimodal Fingertip. arXiv preprint arXiv:2411.02479, 2024.arxiv.org/abs/2411.02479
  19. [19]Elle Miller, Trevor McInroe, David Abel, Oisin Mac Aodha, and Sethu Vijayakumar. Enhancing tactile-based reinforcement learning for robotic control. In NeurIPS, 2025.
  20. [20]Lorenzo Mur-Labadia, Matthew Muckley, Amir Bar, Mido Assran, Koustuv Sinha, Mike Rabbat, Yann LeCun, Nicolas Ballas, and Adrien Bardes. V-JEPA 2.1: Unlocking dense features in video self-supervised learning. arXiv preprint arXiv:2603.14482, 2026.
  21. [21]Dantong Niu, Zhuoyang Liu, Zekai Wang, Boning Shao, Zhao-Heng Yin, Anirudh Pai, Yuvan Sharma, Stefano Saravalle, Ruijie Zheng, Jing Wang, Ryan Punamiya, Mengda Xu, Yuqi Xie, Yunfan Jiang, Letian Fu, Konstantinos Kallidromitis, Matteo Gioia, Junyi Zhang, Jiaxin Ge, Haiwen Feng, Fabio Galasso, Wei Zhan, David M. Chan, Yutong Bai, Roei Herzig, Jiahui Lei, Fei-Fei Li, Ken Goldberg, Jitendra Malik, Pieter Abbeel, Yuke Zhu, Danfei Xu, Jim, Fan, and Trevor Darrell. T-Rex: Tactile-reactive dexterous manipulation, 2026.arxiv.org/abs/2606.17055
  22. [22]Jiahe Pan, Stelian Coros, Jitendra Malik, and Toru Lin. Beyond binary: Sim-to-real dexterous manipulation with physics-grounded contact representation, 2026.arxiv.org/abs/2605.28812
  23. [23]PaXini AI. PXCap III: Tri-Finger Data Collection Glove. Official product documentation. Product specifications identify a PaXini 6D Hall array multidimensional tactile sensor. Accessed 2026-11-08.
  24. [24]Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, Gretchen Krueger, and Ilya Sutskever. Learning Transferable Visual Models From Natural Language Supervision. In Proceedings of the 38th International Conference on Machine Learning, volume 139 of Proceedings of Machine Learning Research, pages 8748–8763. PMLR, 2021.arxiv.org/abs/2103.00020
  25. [25]Akash Sharma, Carolina Higuera, Chaithanya Krishna Bodduluri, Zixi Liu, Taosha Fan, Tess Hellebrekers, Mike Lambeta, Byron Boots, Michael Kaess, Tingfan Wu, Francois Robert Hogan, and Mustafa Mukadam. Self-supervised perception for tactile skin covered dexterous hands. In Proceedings of The 9th Conference on Robot Learning, volume 305 of Proceedings of Machine Learning Research, pages 2311–2328. PMLR, 2025.arxiv.org/abs/2505.11420
  26. [26]Sharpa. Sharpa Wins CES 2026 Innovation Award for Groundbreaking AI Robotic Hand SharpaWave. Official manufacturer announcement, December 16, 2025. Describes the miniature cameras used in SharpaWave fingertip tactile sensing. Accessed 2026-11-08.
  27. [27]Lei Su, Zhijie Peng, Renyuan Ren, Shengping Mao, Juan Du, Kaifeng Zhang, and Xuezhou Zhu. Tacmap: Bridging the tactile sim-to-real gap via geometry-consistent penetration depth map, 2026.arxiv.org/abs/2602.21625
  28. [28]Subramanian Sundaram, Petr Kellnhofer, Yunzhu Li, Jun-Yan Zhu, Antonio Torralba, and Wojciech Matusik. Learning the signatures of the human grasp using a scalable tactile glove. Nature, 569(7758):698–702, 2019.DOI
  29. [29]Zhan Tong, Yibing Song, Jue Wang, and Limin Wang. VideoMAE: Masked Autoencoders are Data-Efficient Learners for Self-Supervised Video Pre-Training. In Advances in Neural Information Processing Systems, volume 35, pages 10078–10093. Curran Associates, Inc., 2022.arxiv.org/abs/2203.12602
  30. [30]Suzannah Wistreich, Baiyu Shi, Stephen Tian, Samuel Clarke, Michael Nath, Chengyi Xu, Zhenan Bao, and Jiajun Wu. DexSkin: High-Coverage Conformable Robotic Skin for Learning Contact-Rich Manipulation. In Proceedings of The 9th Conference on Robot Learning, volume 305 of Proceedings of Machine Learning Research, pages 769–793. PMLR, 2025.arxiv.org/abs/2509.18830
  31. [31]Wei Wu, Fangjing Wang, Fan Lu, He Sun, Shi Liu, Yunnan Wang, Yibin Yan, Yong Wang, Shuaiwei Ma, Xinyang Wang, et al. From foundation to application: Improving VLA models in practice. arXiv preprint arXiv:2607.06403, 2026.arxiv.org/abs/2607.06403
  32. [32]Hu Xu, Gargi Ghosh, Po-Yao Huang, Dmytro Okhonko, Armen Aghajanyan, Florian Metze, Luke Zettlemoyer, and Christoph Feichtenhofer. VideoCLIP: Contrastive Pre-training for Zero-shot Video-Text Understanding. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pages 6787–6800. Association for Computational Linguistics, 2021.arxiv.org/abs/2109.14084
  33. [33]Fengyu Yang, Chao Feng, Ziyang Chen, Hyoungseob Park, Daniel Wang, Yiming Dou, Ziyao Zeng, Xien Chen, Rit Gangopadhyay, Andrew Owens, and Alex Wong. Binding Touch to Everything: Learning Unified Multimodal Tactile Representations. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 26340–26353, 2024.arxiv.org/abs/2401.18084
  34. [34]Chengbo Yuan, Zicheng Zhang, Mingjie Zhou, Wendi Chen, Yi Wang, Zhuoyang Liu, Dantong Niu, Shuo Wang, Hui Zhang, Wenkang Zhang, Yingdong Hu, Yuanqing Gong, Wanli Xing, Chuan Wen, Cewu Lu, Kaifeng Zhang, and Yang Gao. FTP-1: A generalist foundation tactile policy across tactile sensors for contact-rich manipulation. arXiv preprint arXiv:2606.13102, 2026.arxiv.org/abs/2606.13102
  35. [35]Wenzhen Yuan, Siyuan Dong, and Edward Adelson. GelSight: High-Resolution Robot Tactile Sensors for Estimating Geometry and Force. Sensors, 17(12):2762, 2017.DOI
  36. [36]Xiaohua Zhai, Basil Mustafa, Alexander Kolesnikov, and Lucas Beyer. Sigmoid Loss for Language Image Pre-Training. In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), pages 11975–11986, 2023.arxiv.org/abs/2303.15343
  37. [37]Jialiang Zhao, Yuxiang Ma, Lirui Wang, and Edward Adelson. Transferable Tactile Transformers for Representation Learning Across Diverse Sensors and Tasks. In Proceedings of The 8th Conference on Robot Learning, volume 270 of Proceedings of Machine Learning Research, pages 3766–3779. PMLR, 2025.arxiv.org/abs/2406.13640

Paper details

vlnc:W_4kezxs5c
Version
v1 · latest
Published
Fresh — added to the Hub within the last 30 days
Added
License
CC BY 4.0
Usage

views · 0

downloads · 0

A view counts once per browser per day, a download each time the PDF is requested.

  • Computer Science

Not recorded for this paper

View statement
Looking for related papers…
Contents