betaFeedback

WorksComputer Science

Language-Grounded Force/Torque Pretraining for Extrinsic Contact Perception

Abstract

Fine manipulation requires robots to recognize when a grasped object makes or changes contact with its environment, yet such extrinsic contact is not always directly observable from vision or local contact geometry. Force and torque signals carry this information, but also vary with object weight, grasp configuration, and robot motion, leaving it unclear whether they can provide a reusable representation of physical state. Here we show that bilateral six-axis force/torque histories can be pretrained as a semantically structured physical representation by aligning them with a frozen language space using supervision generated from simulated pick-and-place interactions. On unseen objects in simulation, the learned representation supports text-prototype readout of robot action, object weight, and grasp interaction. In real-world placement, unlabeled calibration enables contact and support-state recognition without updating the encoder, outperforming a vertical-force sign-reversal rule baseline. For connector insertion, an event absent from pretraining, low-rank adaptation improves data-efficient learning over training from scratch. These results show that force/torque histories can support both direct readout of known physical states and efficient adaptation to new extrinsic-contact events.

Introduction

When should a robot stop descending while placing an object on a table? How can it tell when a connector is fully seated? Both require recognizing changes in extrinsic contact: contact between a grasped object and the environment away from the fingertips. Reliable perception of such contacts is critical for fine manipulation, yet their effects are not always directly observable from vision or local contact geometry. They are transmitted as force and torque (F/T) through the grasped object, but the resulting signals also vary with object weight, grasp configuration, and robot motion. This raises a central question: Can a force representation capture the underlying physical state well enough to be reused across objects, conditions, and contact events?

Here, we address this question through pretraining on bilateral F/T histories. Each sensor measures a resultant force and torque combining gravity, inertia, grasping, and environmental contact. We seek a shared representation from which action, load, grasp interaction, and support conditions can be read out. The input consists solely of low-dimensional F/T histories, without contact-surface images or point clouds. The intended reuse includes recognizing known states under new conditions and learning events absent from pretraining.

Unlike our approach, semantic contact representations have been studied using optical tactile images and 3D point clouds. CLTP and FG-CLTP align tactile 3D representations with contact descriptions [1, 2], while Tejwani et al. align human-applied force with language [3] and FM-VLA pretrains F/T histories for downstream action generation [4]. These studies establish the value of semantic or pretrained contact representations, but the physical semantics and reusability of six-axis F/T histories themselves remain unclear. Our focus is the independent evaluation of F/T histories as a physical-state representation: whether they support multiple semantic readouts, transfer to real extrinsic contact, and adaptation to an unseen event.

We ground F/T representations in a pretrained language space to make them semantically structured and directly queryable. Language provides both semantic supervision during pretraining and a means to specify physical states at inference. We contrastively align F/T encoder outputs with a frozen language space, allowing defined physical states to be read out through text-prototype similarity without separate classification heads. Our contribution concerns the physical information and reuse supported by this formulation, rather than a new encoder architecture or loss.

A major obstacle to scaling this semantic learning is the lack of semantic supervision for F/T histories. Collecting real interactions across diverse objects and temporally labeling contact and slip intervals is costly. We therefore use simulations to generate both F/T observations and window-level semantic descriptions from privileged physical states, enabling large-scale F/T–language pretraining without manual sequence annotation.

In evaluation, we first probe robot action, object weight, and grasp-interaction recognition on unseen objects, comparing encoder architectures and pretraining objectives. We then evaluate two forms of reuse. In real-object placement, the simulation-pretrained encoder is transferred to real F/T measurements after unlabeled calibration, enabling contact and support-state readout without encoder updates. For full connector seating, a new downstream event not represented in pretraining, we use separately collected labeled data and find that frozen linear readout offers limited benefit, whereas LoRA [5] adaptation improves low-data performance.

Our contributions are:

  • We demonstrate the semantic utility of bilateral F/T histories: a common language-aligned representation

Overview of automated dataset construction, contrastive learning, and evaluation and reuse

Figure 1. Overview. (a) Automated pick-and-place simulation generates bilateral F/T histories and semantic annotations from simulator states. (b) F/T embeddings are contrastively aligned with a frozen language space. (c) The learned representation is evaluated through physical-state readout on unseen objects, real-world transfer with a frozen encoder, and few-shot adaptation to an unseen insertion-completion event with LoRA.

supports multiple physical-state readouts through text prototypes.

  • We build an automated pipeline that derives semantic supervision from simulator states, enabling large-scale F/T–language pretraining.

  • We characterize reuse and its limits through real-world contact recognition without encoder updates and data-efficient adaptation to an unseen insertion-completion event.

We release the code and dataset in our GitHub repository.

Related Work

Force and Tactile Perception of Extrinsic Contact

Extrinsic contact can be estimated and controlled using tactile and kinematic observations at the grasp, including contact location and transmitted wrench [6]. Such work shows that contact away from the fingertips can be inferred through the grasped object. Learned representations incorporating force have also supported manipulation: Lee et al. use self-supervised visual, force, and robot-state representations for insertion [7]. More recently, ForceVLA, ForceVLA2 and FAVLA integrate force into action generation [8, 9, 10], and FM-VLA uses pretrained F/T histories as contact-related memory [4].

We share the interest in extrinsic contact but study F/T-only perception before contact-geometry estimation or policy learning. We evaluate whether a common representation supports language-based identification of several physical states and adaptation to an unseen contact event.

Tactile Pretraining and Language Alignment

Sparsh, Sparsh-X, and AnyTouch 2 learn tactile representations of physical attributes and contact dynamics through self-supervision [11, 12, 13]. UniTouch and TVL connect touch with vision and language [14], [15], while CLTP and FG-CLTP align tactile 3D point clouds with contact descriptions [1], [2]. CLTP and FG-CLTP are particularly related in their use of simulated contacts, attribute-based descriptions, and transfer to real sensing. OmniVTLA semantically aligns tactile information including spatial contact-force fields [16]; TaF-VLA uses force information to supervise an encoder of tactile histories [17].

Our input is the history of resultant forces and torques at each finger, rather than contact images, point clouds, or spatial force distributions. We share the use of semantic alignment but evaluate recognition on unseen objects and transfer to real extrinsic contact.

Sensor–Language Representations

IMU2CLIP aligns inertial time series with video and language [18], and ImageBind connects heterogeneous modalities, including IMU signals, in a shared space [19]. SenorLM generates statistical, structural, and semantic descriptions for large-scale sensor–language pretraining [20]. For force signals, Tejwani et al. combine contrastive and reconstruction/translation learning to relate human-applied force profiles to language [3].

Our target is the physical state arising from robot–object–environment interaction. We construct window-level descriptions from simulator states and evaluate both languageprototype recognition and supervised adaptation to a new event.

Language-Grounded F/T Pretraining

Input and Shared Representation

We learn a representation of F/T histories for physical-state readout and reuse across objects and contact events. The history up to time tt is

Xt=[xt−T+1,…,xt]∈RT×Dsensor,(1)X_t = [x_{t-T+1}, \ldots, x_t] \in\mathbb{R}^{T \times D_{\mathrm{sensor}}}, \tag*{(1)}

where Dsensor=12D_{\mathrm{sensor}} = 12 concatenates six-axis measurements from both fingers. The F/T encoder maps this history to ztF/T=fθF/T(Xt)∈Rdz_t^{\mathrm{F/T}} = f_\theta^{\mathrm{F/T}}(X_t) \in\mathbb{R}^d. Sentence-BERT [21] maps the paired description ltl_t to ztLang=fLang(lt)∈Rdz_t^{\mathrm{Lang}} = f^{\mathrm{Lang}}(l_t) \in\mathbb{R}^d. We freeze the language encoder and optimize only the F/T encoder.

Contrastive Alignment and Readout

For a batch of BB pairs, we define

sij=cos⁡(ziF/T,zjLang)τ,(2)s_{ij} = \frac{\cos(z_i^{\mathrm{F/T}}, z_j^{\mathrm{Lang}})}{\tau}, \tag*{(2)}

where the temperature is fixed at τ=0.07\tau= 0.07, and cosine similarity uses normalized embeddings. The symmetric InfoNCE objective [22], [23] is

LF/T→Lang=−1B∑i=1Blog⁡exp⁡(sii)∑j=1Bexp⁡(sij),(3)\mathcal{L}_{\mathrm{F/T}\to\mathrm{Lang}} = -\frac{1}{B}\sum_{i=1}^{B}\log\frac{\exp(s_{ii})}{\sum_{j=1}^{B}\exp(s_{ij})}, \tag*{(3)}
LLang→F/T=−1B∑i=1Blog⁡exp⁡(sii)∑j=1Bexp⁡(sji),(4)\mathcal{L}_{\mathrm{Lang}\to\mathrm{F/T}} = -\frac{1}{B}\sum_{i=1}^{B}\log\frac{\exp(s_{ii})}{\sum_{j=1}^{B}\exp(s_{ji})}, \tag*{(4)}
Lalign=12(LF/T→Lang+LLang→F/T).(5)\mathcal{L}_{\mathrm{align}} = \frac{1}{2}\left(\mathcal{L}_{\mathrm{F/T}\to\mathrm{Lang}} + \mathcal{L}_{\mathrm{Lang}\to\mathrm{F/T}}\right). \tag*{(5)}

At inference, candidate descriptions are encoded as language prototypes, and the state with maximum cosine similarity is selected. Comparing a history with “Holding an object.” and “Placing an object.” thus provides a state readout without a trained classification head. For an event absent from pretraining, we instead attach a supervised head and evaluate frozen linear readout and LoRA adaptation. These protocols distinguish querying known semantics from learning a new event.

Generating F/T–Language Pairs from Physics

Interactions and Observations

We automate pick-and-place with a Franka Emika Panda in Genesis [24], varying objects, density, motion conditions, and supports. Episodes include grasping, lifting, holding, descent, placement, pressing, and release, with finger–object slip and object–support contact.

Two virtual six-axis fingertip sensors record 12-dimensional F/T at 100 Hz. We extract causal 0.8 s histories (80 samples) at a 0.1 s stride. Synchronized wrist-view and fixed third-person RGB images are stored but are not inputs to training or evaluation.

Semantic Descriptions

Simulator states determine four attributes, composed into descriptions according to the window’s manipulation and contact conditions:

  • Action: Command logs and contact events define the state. Lift contains liftoff, place contains initial support contact, and press denotes subsequent sustained pressing. Hold includes transporting an object in the air.

  • Weight: Light or heavy based on mass, omitted when the environment fully supports the object.

  • Interaction: Stable, slip, or accidental drop, based on relative finger–object motion and contact. Stable/slip is omitted when grasp contact is insufficiently sustained.

  • Support Surface: Flat, step, slope, or wobbling support during placement or pressing, based on the surface actually contacted and its motion.

Windows with missing attributes are retained if other attributes remain valid. Descriptions include “Lifting a heavy object while it is slipping.” and “Placing an object on a wobbling support.”; transitions appear as, for example, “place then release.” Simulator states supply supervision without inferring labels from F/T alone; we do not assume that every internal attribute is uniquely recoverable from F/T.

Scale and Splits

We retain accidental drops as physical failure events but exclude episodes outside common force/position quality criteria, including forces of at least 100 N or displacements of at least 5 cm within 0.01 s. After filtering, the collection contains 1,531,185 windows from 40,878 episodes. Distribution-aware subsampling without replacement reduces dominance by long-lasting states while retaining rare events and transitions. The pretraining set contains 437,244 windows from 40,851 episodes involving 892 Google Scanned Objects (GSO) [25].

Evaluation uses 111,355 windows from 2,173 episodes involving 75 YCB objects [26] absent from training. Attribute-specific subsets yield 19,683 window evaluations in total. Encoder checkpoints are selected using a window-level validation split from GSO; the object-disjoint YCB set is used only for final evaluation.

Experiments

We evaluate three research questions:

  • RQ1: Can language prototypes read out robot action, object weight, and grasp interaction on unseen objects?

  • RQ2: Can the encoder recognize real contact and support conditions without parameter updates?

  • RQ3: Does pretraining help learn insertion completion, an event absent from its annotations, with few labeled training episodes?

RQ1 probes load and grasp information; RQ2 and RQ3 evaluate reuse for extrinsic contact.

ModelActionWeightInteraction
PatchTST94.02 ± 0.1193.00 ± 0.5078.14 ± 0.73
CNN93.28 ± 0.1993.97 ± 0.3881.24 ± 0.46
MLP94.12 ± 0.3193.38 ± 0.5082.57 ± 0.34
Supervised MLP89.31 ± 0.3491.12 ± 0.5075.34 ± 0.82

Table I. State recognition for unseen objects. Macro F1 scores are means±sample standard deviations across 3 seeds, reported as percentages. The first three models use sentence-template text prototypes; the last is a supervised classifier trained on the same attributes, including support surface.

MethodActionWeightInteraction
BYOL76.8883.2173.84
Masked reconstruction85.2486.8178.30
VAE82.4183.7177.19
VICReg80.1473.9573.49

Table II. Linear probing after self-supervised pretraining. Macro F1 scores are reported as percentages. Results are from a single seed per method.

a) Encoder implementation: PatchTST [27] divides time series into patches and independently encodes each channel with a shared Transformer for forecasting and representation learning. We divide each 80-sample channel into 8 patches and aggregate all 96 channel–patch features using a Token-Fusion head. Learnable channel and patch-position embeddings are added before a one-layer Transformer with a CLS token; the CLS output is projected to 768 dimensions for language alignment. PatchTST below denotes this Token-Fusion variant. The CNN processes the 12 input channels with four 1D convolutional blocks (kernel size 5), followed by global average pooling and a linear projection to 768 dimensions. The MLP receives the 80 × 12 input flattened into a vector while preserving temporal and channel ordering. All three encoders have approximately 1.11M parameters.

b) Input standardization: We standardize each channel using training-set means and standard deviations shared across all windows. We do not center and scale each window independently, thereby preserving between-window differences in steady loads and amplitudes. Calibration statistics and offset correction for real data are specified below.

RQ1: Physical-State Recognition on Unseen Objects

Protocol

We first test whether the pretrained representation preserves multiple physical attributes on objects unseen during training. Using cosine similarity to language prototypes, we recognize six actions (grasp, hold, lift, place, press, release), two weight classes (light, heavy), and three grasp-interaction classes (stable, slip, accidental drop). For action evaluation, we exclude entire episodes containing accidental drops and transition windows such as “grasp then lift”. Evaluation sets are balanced independently for each attribute: 1,668 windows per action, 3,000 per weight class, and 1,225 per interaction class. Candidate classes are expressed using templates such as “{Action} on an object.” and “The object is {Weight}”, and their embeddings are compared with the F/T embedding using cosine similarity.

We compare PatchTST, CNN, and MLP under the same contrastive conditions. As a non-language semantic-supervision reference, we train a multi-task MLP on simulator-derived action, weight, grasp-interaction, and support-surface labels using the same training data. Models use the same training data, batch size 512, learning rate 2.0×10−32.0 \times10^{-3}, and at most 400 epochs, with warmup, cosine decay, early stopping, and validation-based checkpoint selection. Input normalization uses training data only.

Language-Prototype Readout

All three contrastively trained encoders achieve strong Macro F1 scores on unseen objects (Table I), indicating that the language-aligned representation preserves action, weight, and grasp-interaction information beyond the training objects. With the same MLP backbone, language-prototype readout exceeds the supervised reference by 4.81, 2.26, and 7.23 percentage points in action, weight, and interaction Macro F1, respectively, even when the supervised targets include support surface. One possible explanation is that fixed sentence embeddings provide a shared semantic structure for joint descriptions of physical attributes, acting as an inductive bias for generalization to unseen objects. Thus, in this setting, language-based readout does not require sacrificing recognition performance relative to dedicated classification heads.

No architecture is consistently best: MLP performs best on action and interaction, whereas CNN is best on weight. These results indicate that the observed recognition capability is not restricted to a particular encoder architecture. The evaluation uses finite canonical descriptions and does not test generalization to unrestricted language queries.

Self-Supervised Linear Readout

We next ask how much physical-state information can be learned from the F/T signal alone, without semantic supervision. Using the same PatchTST backbone, we compare masked reconstruction, BYOL [28], VAE [29], and VICReg [30], and evaluate each frozen encoder with a supervised linear head.

Masked reconstruction predicts masked samples; BYOL matches two augmented views; VAE learns a reconstruction-based latent variable model; and VICReg regularizes invariance, variance, and covariance across augmented views.

Masked reconstruction gives the strongest linear readout across all three attributes, with VAE next (Table II). This shows that signal-centric pretraining alone can recover substantial physical-state structure from F/T histories and provides a strong baseline for semantic pretraining. Figure 2 qualitatively illustrates that different physical states can produce distinct F/T dynamics.

Example force histories under contrasting physical conditions in simulation and real-world experiments

Figure 2. Example 0.8 s force histories under contrasting physical conditions in simulation and real-world experiments: (a) larger force magnitudes for a heavier object, (b) stronger force fluctuations during slip, (c) slower load transfer on a wobbling support, and (d) slower load transfer for a compliant object (ball).

However, Tables I and II differ in both readout and supervision. Their numerical differences alone do not establish general superiority of language-contrastive pretraining. RQ3 compares these objectives under common downstream adaptation protocols.

RQ2: Real Extrinsic Contact without Encoder Updates

We next test whether physical states learned in simulation can be read out from real F/T measurements without updating the encoder. We evaluate two aspects of extrinsic contact: detecting support contact during placement and identifying the contacted support condition. We use the language-pretrained PatchTST as the real-robot backbone, based on an auxiliary hold-versus-place prototype readout. CNN and MLP were evaluated afterwards as an additional architecture check on the fixed-support wooden-cuboid task, achieving 11/20 and 17/20 detections, respectively, compared with 20/20 for PatchTST. These results were consistent with the choice of PatchTST for the main real-robot evaluation.

Calibration

To account for the simulation-to-real difference in sensor coordinates, scale, and offset, we perform calibration using unlabeled real data. A UR3 with a Robotiq 2F gripper and bilateral MMS101 six-axis fingertip sensors (Figure 3a) records 12-dimensional F/T at 100 Hz. Sensor coordinates are mapped to the simulated fingertip frames. Ten unlabeled real pick-and-place trials provide normalization statistics. To compensate for sensor drift, we correct offsets using 1 s of unloaded measurements before each grasp. Encoder parameters remain fixed.

Robot hardware, objects, and insertion setup

Figure 3. Hardware used in RQ2, RQ3. (a) UR3 with a Robotiq 2F gripper and bilateral MMS101 six-axis fingertip sensors, (b) Objects used in RQ2, (c) Connector and socket used in RQ3.

Contact Detection

During descent, the preceding 0.8 s of F/T history is processed every 50 ms and compared with the text prototypes “Holding an object.” and “Placing an object.” After an initial 2 s gate, descent is stopped when the similarity to “Placing an object.” exceeds that of “Holding an object.” for three consecutive evaluations.

As a tuning-free rule-based reference, we use the vertical-force transition caused by load transfer after support contact. The detector stops when the sum of the bilateral offset-corrected FzF_z signals remains below 0 N for three consecutive evaluations. This corresponds to the transition from supporting the object’s weight to receiving a reaction force from the support.

Reference contact time is estimated from the operator’s visual timestamp after subtracting a separately measured 300 ms reaction time. We evaluate 20 trials per condition: three objects on a fixed support (wooden cuboid, cork

ConditionMethodSuccessDelay [s]
Fixed, wooden cuboidProposed20/200.348
Fixed, wooden cuboidFzF_z20/200.492
Fixed, cork cylinderProposed19/200.430
Fixed, cork cylinderFzF_z20/200.547
Fixed, compliant ballProposed11/200.631
Fixed, compliant ballFzF_z0/20—
Wobbling, wooden cuboidProposed12/200.437
Wobbling, wooden cuboidFzF_z0/20—

Table III. Real-world contact detection. Delay is measured from reaction-time-corrected contact to issuing a halt command and averaged over successful trials only.

cylinder, and compliant ball) and the wooden cuboid on a wobbling support (Figure 3b). Success requires issuing a halt command within 0.8 s after reference contact, matching the input-history length; beyond this interval, the initial contact transition no longer appears in the causal input window. Halt command latency is reported over successful trials.

Across the three fixed-support objects, the proposed method succeeds in 50/60 trials (83.3%), compared with 40/60 (66.7%) for the FzF_z rule (Table III). The two methods are comparable on the rigid wooden and cork objects, but their behavior diverges on the compliant ball and wobbling support: the FzF_z rule never triggers within 0.8 s, whereas the proposed method succeeds in 11/20 and 12/20 trials, respectively. On the wooden and cork objects, the proposed method also yields shorter mean halt-command delays over successful trials.

The failure of the FzF_z rule under compliant and moving-support conditions is consistent with delayed load reversal after initial contact. On the wobbling support, extending its observation horizon to 5 s yields 15/20 eventual detections, but with a mean delay of 3.890 s. The pretrained model may therefore use F/T changes during load transfer before sign reversal occurs.

Support-Condition Readout

We next test whether the same embedding used for contact detection also retains information about the contacted support. We compare the prototypes “Placing an object on a fixed flat surface.” and “Placing an object on a wobbling surface.” on successfully detected wooden-cuboid contacts: 20 fixed-support and 20 wobbling-support trials. The latter comprise the initial 12 successes plus 8 successes obtained from 13 additional attempts. The support is predicted from whichever prototype has higher cosine similarity, with no trained classification head or tuned threshold.

The prototype comparison correctly identifies 31/40 trials (77.5%, Table IV). Thus, the contact-detection embedding retains information about support condition. This result is conditional on successful contact detection, and wobbling supports remain less reliable (13/20).

True supportfixed cosinewobble cosineAccuracy
Fixed0.2630.14518/20 (90.0%)
Wobbling0.0610.08113/20 (65.0%)
Overall––31/40 (77.5%)

Table IV. Support recognition conditional on successful contact detection. Similarities are mean cosine similarities to the two language prototypes.

RQ3: Adapting to Unseen Insertion Completion

We finally test whether pretraining helps learn a new extrinsic-contact event that is not represented in the pretraining annotations. The downstream task is to recognize when a connector becomes fully seated, which can inform whether insertion should continue or stop (Figure 3c).

Data and Protocol

We formulate insertion completion as binary recognition of whether the connector is fully seated, rather than insertion-motion generation. Using separately collected real-robot data, we record bilateral F/T at 100 Hz and extract causal 0.8 s histories every 0.1 s. The reference seating time tseatt_{\mathrm{seat}} is obtained by subtracting a measured 300 ms reaction time from the operator timestamp, and each window is labeled by its endpoint:

yt={0,t<tseat,1,t≥tseat.(6)y_t = \begin{cases} 0, & t < t_{\mathrm{seat}}, \\ 1, & t \ge t_{\mathrm{seat}}. \end{cases} \tag*{(6)}

Thus, the positive class includes full seating and all subsequent observations.

We split 125 successful episodes into 100 training, 5 validation, and 20 held-out test episodes. We vary the number of labeled training episodes N∈{1,2,5,10,25,50,100}N \in\{1, 2, 5, 10, 25, 50, 100\} over five seeds, using identical nested subsets across methods. Normalization statistics are fixed from the full 100-episode training split; checkpoints are selected on the validation set, and evaluation uses window-level Macro F1 on the test episodes.

Adaptation Methods

Using the same PatchTST backbone, we compare language-contrastive, masked-reconstruction, VAE, VICReg, and BYOL pretraining. Frozen readout trains only a 769-parameter linear head. LoRA [5] updates rank-4 adapters in the attention query/value projections together with the head, for 8,961 trainable parameters. Scratch End-to-End instead trains all 1,111,937 parameters from random initialization on the downstream data.

Limits of Frozen Readout

Frozen pretrained representations help only in the very low-data regime (Figure 4). For example, at N=1N = 1, VICReg and VAE reach (83.62±8.08)%(83.62 \pm8.08)\% and (82.40±7.27)%(82.40 \pm7.27)\%, compared with (77.52±2.15)%(77.52 \pm2.15)\% for Scratch. By N=10N = 10, however, Scratch reaches (94.51±1.08)%(94.51 \pm1.08)\% and exceeds the best frozen representation, and this gap persists at larger NN. At N=100N = 100, Scratch reaches (98.34±0.19)%(98.34 \pm0.19)\%, compared with (91.90±1.02)%(91.90 \pm1.02)\% for frozen language-contrastive features.

Figure 4. Insertion-completion recognition with frozen linear readout and Scratch End-to-End. Curves show five-seed means; bands and error bars indicate ±1\pm1 standard deviation.

Thus, representations that support known semantic readout do not necessarily make a new event linearly separable. This motivates adapting the representation rather than keeping it fixed.

Parameter-Efficient Adaptation

Adapting the pretrained representation changes this trend (Figure 5). With

Figure 5. Insertion-completion recognition with LoRA adaptation. Evaluation conditions and the Scratch baseline are shared with Figure 4. Curves show five-seed means; error bars and bands indicate ±1\pm1 sample standard deviation.

LoRA, language-contrastive pretraining reaches (91.66±2.77)%(91.66 \pm2.77)\% at N=1N = 1, (95.29±1.33)%(95.29 \pm1.33)\% at N=5N = 5, and (96.93±1.17)%(96.93 \pm1.17)\% at N=10N = 10, compared with Scratch means of (77.52±2.15)%(77.52 \pm2.15)\%, (84.72±5.61)%(84.72 \pm5.61)\%, and (94.51±1.08)%(94.51 \pm1.08)\%, respectively. The advantage narrows as more labeled data become available, with both models approaching 99%99\% at N=100N = 100.

The benefit is not unique to language supervision: masked reconstruction has the highest mean at N=1N = 1, and other pretrained objectives also benefit from adaptation. These results indicate that pretraining is more useful for this new event as an adaptable initialization than as a fixed feature extractor. Together, RQ2 and RQ3 characterize two forms of reuse: querying known semantics under new conditions and adapting a pretrained representation to a new contact event.

Limitations and Future Work

The scope of pretraining remains limited. Our simulated data center on pick-and-place and do not cover the full diversity of contact-rich manipulation, including sustained sliding, rotation, deformation, and broader variations in object and support properties. Real-robot deployment also requires calibration from unlabeled data, and contact-detection failures remain for compliant objects and moving supports. Future work should combine broader simulation with automatically collected real interactions to reduce this domain gap.

The evaluation also limits the conclusions that can be drawn. Language-based recognition uses a finite set of pre-defined prototypes and does not test paraphrase robustness or unseen concepts. Support-condition recognition is evaluated only after successful contact detection, and the insertion dataset contains successful episodes only; failure discrimination and recovery are therefore not established.

Conclusion

We studied language-grounded pretraining of bilateral F/T histories for reusable physical-state perception. Simulator states generate supervision for 437,244 paired windows from 40,851 episodes and 892 objects. Language prototypes read out multiple attributes on unseen objects and recognize real contact and support conditions after unlabeled calibration, without encoder updates. For unseen insertion completion, frozen features have limitations, but LoRA improves performance with few labeled training episodes. These findings show that low-dimensional F/T histories transmitted through a grasped object can support both known-state readout and adaptation to new extrinsic contact events.

Appendix

a) Wrench and annotation conventions: For each simulated finger, the recorded wrench sums all contact forces on the finger link in its local frame, with torque computed about the link origin. Real measurements use the same channel order and an axis mapping Rz(π)R_z(\pi), but no torque-origin translation; thus, exact geometric equivalence of simulated and physical torque reference points is not assumed. Contact-state transitions are confirmed over five consecutive samples and assigned to the first sample of the confirmed run. Weight is split at 0.2 kg; slip is defined from tangential finger–object relative motion with a 5 mm/s5\ \mathrm{mm}/\mathrm{s} threshold.

b) Language training and splits: The retained corpus contains a finite template vocabulary and uses the frozen all-mpnet-base-v2 Sentence-Transformer. InfoNCE treats only the paired diagonal example as positive; windows with identical descriptions may remain in the denominator. Historical checkpoints use a window-level GSO validation split. The final YCB evaluation uses disjoint objects and is not used for checkpoint selection. In an additional three-seed MLP sensitivity study, episode-disjoint validation yielded similar YCB Macro F1 scores to the historical window-level split (Action: −0.20 pp, Weight: −0.03 pp, Interaction: +0.59 pp).

c) Computational resources and training time: Training was ran on a workstation with an NVIDIA RTX A6000 GPU. 400-epoch contrastive training with batch size 512 took 7.71–7.89 h for PatchTST, 0.98–1.03 h for CNN, and 0.66–0.70 h for MLP.

d) Inference latency: An additional offline benchmark of the deployed PatchTST readout on one RTX A6000 (batch size one, 80×1280 \times12 input) measured 1.83 ms mean and 1.89 ms 95th-percentile latency over 1,000 calls after 200 warm-up calls. In the online inference, the interval averaged 11.18 ms (95th percentile: 15.61 ms) over all placement trials.

Acknowledgment

This work was supported by JSPS KAKENHI Grant Number JP25H01135. We gratefully acknowledge this support.

References

  1. [1]W. Ma, et al., “Cltp: Contrastive language-tactile pre-training for 3d contact geometry understanding,” 2025.
  2. [2]W. Ma, C. Zhang, Y. Cai, G. Yao, S. Cui, and S. Wang, “Fg-cltp: Fine-grained contrastive language tactile pretraining for robotic manipulation,” 2026.
  3. [3]R. Tejwani, K. A. Velazquez, J. Payne, P. Bonato, and H. Asada, “Demonstrating Shared Force-Language Embeddings for Natural Human-Robot Communication,” in Proceedings of Robotics: Science and Systems, June 2025.DOI
  4. [4]R. Li, et al., “Fm-vla: Force-based memory for vision-language-action models in contact-rich manipulation,” 2026.arxiv.org/abs/2607.18231
  5. [5]E. J. Hu, et al., “LoRA: Low-rank adaptation of large language models,” in International Conference on Learning Representations, 2022.arxiv.org/abs/2106.09685
  6. [6]S. Kim, D. K. Jha, D. Romeres, P. Patre, and A. Rodriguez, “Simultaneous tactile estimation and control of extrinsic contact,” in 2023 IEEE International Conference on Robotics and Automation (ICRA), 2023, pp. 12 563–12 569.arxiv.org/abs/2303.03385
  7. [7]M. A. Lee, et al., “Making sense of vision and touch: Self-supervised learning of multimodal representations for contact-rich tasks,” in 2019 International Conference on Robotics and Automation (ICRA), 2019, pp. 8943–8950.arxiv.org/abs/1810.10191
  8. [8]J. Yu, et al., “Forcevla: Enhancing vla models with a force-aware moe for contact-rich manipulation,” in Advances in Neural Information Processing Systems, D. Belgrave, et al., Eds., vol. 38, Main Conference. Curran Associates, Inc., 2025, pp. 93 409–93 439.arxiv.org/abs/2505.22159
  9. [9]Y. Li, et al., “Forcevla2: Unleashing hybrid force-position control with force awareness for contact-rich manipulation,” in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), June 2026, pp. 8911–8920.arxiv.org/abs/2603.15169
  10. [10]L. Yao, et al., “Favla: A force-adaptive fast-slow vla model for contact-rich robotic manipulation,” 2026.arxiv.org/abs/2602.23648
  11. [11]C. Higuera, et al., “Sparsh: Self-supervised touch representations for vision-based tactile sensing,” in 8th Annual Conference on Robot Learning, 2024.arxiv.org/abs/2410.24090
  12. [12]C. Higuera et al., “Tactile beyond pixels: Multisensory touch representations for robot manipulation,” in 9th Annual Conference on Robot Learning, 2025.arxiv.org/abs/2506.14754
  13. [13]R. Feng, et al., “Anytouch 2: General optical tactile representation learning for dynamic tactile perception,” in International Conference on Learning Representations, C. Vondrick, B. Hariharan, C. Raffel, L. Pinto, D. Yang, and A. Faust, Eds., vol. 2026, 2026, pp. 3543–3577.arxiv.org/abs/2602.09617
  14. [14]F. Yang, et al., “Binding touch to everything: Learning unified multimodal tactile representations,” 2024.arxiv.org/abs/2401.18084
  15. [15]L. Fu, et al., “A touch, vision, and language dataset for multimodal alignment,” in Proceedings of the 41st International Conference on Machine Learning, ser. ICML’24. JMLR.org, 2024.arxiv.org/abs/2402.13232
  16. [16]Z. Cheng, et al., “Omnivla: Vision-tactile-language-action models with semantic-aligned tactile sensing,” IEEE Robotics and Automation Letters, vol. 11, no. 9, pp. 10 146–10 153, 2026.
  17. [17]Y. Huang, et al., “Tactile-force alignment in vision-language-action models for force-aware manipulation,” 2026.arxiv.org/abs/2601.20321
  18. [18]S. Moon, A. Madotto, Z. Lin, A. Saraf, A. Bearman, and B. Damavandi, “IMU2CLIP: Language-guided motion sensor translation with multimodal contrastive learning,” in Findings of the Association for Computational Linguistics: EMNLP 2023, H. Bouamor, J. Pino, and K. Bali, Eds. Singapore: Association for Computational Linguistics, Dec 2023, pp. 13 246–13 253.DOI
  19. [19]R. Girdhar, et al., “Imgbind: One embedding space to bind them all,” in Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, 2023, pp. 15 180–15 190.
  20. [20]Y. Zhang, et al., “SensorLM: Learning the language of wearable sensors,” in Advances in Neural Information Processing Systems, vol. 38. Neural Information Processing Systems Foundation, Inc., 2025, pp. 52 274–52 310.arxiv.org/abs/2506.09108
  21. [21]N. Reimers and I. Gurevych, “Sentence-bert: Sentence embeddings using siamese bert-networks,” in Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing. Association for Computational Linguistics, 11 2019.arxiv.org/abs/1908.10084
  22. [22]A. van den Oord, Y. Li, and O. Vinyals, “Representation learning with contrastive predictive coding,” 2019.arxiv.org/abs/1807.03748
  23. [23]A. Radford, et al., “Learning transferable visual models from natural language supervision,” in International conference on machine learning. PMLR, 2021, pp. 8748–8763.arxiv.org/abs/2103.00020
  24. [24]Genesis Authors, “Genesis: A universal and generative physics engine for robotics and beyond,” December 2024. [Online]. Available: https://github.com/Genesis-Embodied-AI/Genesis
  25. [25]L. Downs, et al., “Google scanned objects: A high-quality dataset of 3d scanned household items,” in 2022 International Conference on Robotics and Automation (ICRA), 2022, pp. 2553–2560.arxiv.org/abs/2204.11918
  26. [26]B. Calli, A. Singh, A. Walsman, S. Srinivasa, P. Abbeel, and A. M. Dollar, “The ycb object and model set: Towards common benchmarks for manipulation research,” in 2015 international conference on advanced robotics (ICAR). IEEE, 2015, pp. 510–517.DOI
  27. [27]Y. Nie, N. H. Nguyen, P. Sinthong, and J. Kalagnanam, “A time series is worth 64 words: Long-term forecasting with transformers,” in International Conference on Learning Representations, 2023.
  28. [28]J.-B. Grill, et al., “Bootstrap your own latent: a new approach to self-supervised learning,” in Proceedings of the 34th International Conference on Neural Information Processing Systems, ser. NIPS ’20. Red Hook, NY, USA: Curran Associates Inc., 2020.arxiv.org/abs/2006.07733
  29. [29]D. P. Kingma and M. Welling, “Auto-Encoding Variational Bayes,” in 2nd International Conference on Learning Representations, ICLR 2014, Banff, AB, Canada, April 14-16, 2014, Conference Track Proceedings, 2014.arxiv.org/abs/1312.6114
  30. [30]A. Bardes, J. Ponce, and Y. LeCun, “VICReg: Variance-invariance-covariance regularization for self-supervised learning,” in International Conference on Learning Representations, 2022.arxiv.org/abs/2105.04906

Paper details

vlnc:W_x6x6pge6
Version
v1 · latest
Published
Fresh — added to the Hub within the last 30 days
Added
License
CC BY 4.0
Usage

views · 0

downloads · 0

A view counts once per browser per day, a download each time the PDF is requested.

  • Computer Science

Not recorded for this paper

View statement
Looking for related papers…
Contents