betaFeedback

WorksComputer Science

Adaptive Hybrid Force-Vision Control with EMG-Derived References for Deformable Bottle Manipulation

Abstract

Robotic automation of laboratory tasks has become a growing interest to remove researchers from hazardous environments and repetitive procedures. However, even equipment as simple as squeeze bottles poses a challenging manipulation problem due to its deformability. This paper presents an adaptive hybrid vision-force controller for robotic grippers to effectively manipulate laboratory squeeze bottles to dispense liquid. This approach integrates force and vision feedback according to the interaction state, enabling effective squeezing while accounting for bottle deformation and recovery. The proposed controller is validated by comparing its performance with a pure force controller on 12 combinations of conditions consisting of 2 target dispense volumes, 2 plastic squeeze bottles, and 1 to 3 squeezes used to dispense. Human demonstrations of these operating conditions were collected using electromyography (EMG) sensing mapped to a desired grip force, which was then used as the force reference for the controllers to squeeze the bottles. Results demonstrated that the proposed adaptive hybrid vision-force controller achieved more accurate liquid dispensing with a mean absolute error (MAE) of 6.54 mL compared to a MAE of 15.92 mL from the pure force controller, while also replicating the human action recorded from the EMG signal more closely.

Introduction

There is an increasing interest in using robotic manipulators for laboratory automation, as they can automate repetitive tasks and limit contact between researchers and hazardous materials through automation and remote operation [1]. Laboratory settings demand that robots perform experiments that require precise, accurate, and repeatable manipulation of deformable objects such as squeeze bottles, used for dispensing solutions and cleaning equipment. However, current robotic systems struggle to reliably perform contact-rich manipulation on deformable objects in dynamic environments that require high repeatability and accuracy [2] [3]. Squeeze-bottle manipulation is particularly challenging as it requires coordinated regulation of grip force and object deformation, especially when multiple squeezes are required, and the bottle must recover between successive interactions.

Tactile sensors directly measure the grip force being applied by a robot manipulator, providing key feedback regarding the state of deformable objects [4]. However, experiments revealed that force feedback alone was insufficient to represent the state of a wash bottle due to similar force measurement readings at different levels of deformation. In particular, during the “release phase”, after water has been dispensed and the bottle is returning to its original form, commanding a low grip force was not sufficient to allow the bottle to return to its original shape. Therefore, machine vision can provide complementary information about the object state by directly estimating levels of deformation [5], motivating the integration of force and vision for squeeze bottle manipulation.

In addition to sensing the state of the object, force-based manipulation requires a reference specifying the desired interaction force. Rather than manually defining this reference, human demonstrations can provide reference force profiles to replicate desired squeezing behaviours. This serves to both create desirable squeezing profiles, and allow for potential direct teleoperation of the system. Direct operator control allows for increased safety and adaptability, which is critical in laboratory settings to modify experiments and operate in human-centered lab environments [6]. Operator inputs can be captured using EMG sensors, which provide a non-invasive method of measuring grip force through muscle activation [7] and do not interfere with manipulation contact, allowing for more intuitive demonstrations to be collected. In this work, EMG signals recorded during human demonstrations are mapped to desired robot grip-force profiles, providing human-derived references for reproducing the demonstrated squeezing behaviour.

Department of Mechanical and Mechatronics Engineering, University of Waterloo, Canada [email protected], [email protected], [email protected], [email protected]

The authors acknowledge the support of the Natural Sciences and Engineering Research Council of Canada (NSERC) through the CREATE ADVENTOR program under grant 584985 and NSERC PGS-D under grant 600034-2025.

The multiple sensing modalities allow both the intended measurement readings at different levels of deformation. In particular, during the “release phase”, after water has been dispensed and the bottle is returning to its original form, commanding a low grip force was not sufficient to allow the bottle to return to its original shape. Therefore, machine vision can provide complementary information about the object state by directly estimating levels of deformation [5], motivating the integration of force and vision for squeeze bottle manipulation.

In addition to sensing the state of the object, force-based manipulation requires a reference specifying the desired interaction force. Rather than manually defining this reference, human demonstrations can provide reference force profiles to replicate desired squeezing behaviours. This serves to both create desirable squeezing profiles, and allow for potential direct teleoperation of the system. Direct operator control allows for increased safety and adaptability, which is critical in laboratory settings to modify experiments and operate in human-centered lab environments [6]. Operator inputs can be captured using EMG sensors, which provide a non-invasive method of measuring grip force through muscle activation [7] and do not interfere with manipulation contact, allowing for more intuitive demonstrations to be collected. In this work, EMG signals recorded during human demonstrations are mapped to desired robot grip-force profiles, providing human-derived references for reproducing the demonstrated squeezing behaviour.

The multiple sensing modalities allow both the intended interaction force and the changing deformation state of the object to be incorporated into the manipulation task. Building on this multimodal information, this paper presents two main contributions: 1) an EMG-driven adaptive hybrid force-vision controller for deformable squeeze-bottle manipulation in laboratory automation, combining reference force profiles from human demonstrations with grip force measured using force sensitive resistors (FSRs) and visually estimated object deformation, as shown in Figure 1. The controller adaptively integrates force and deformation feedback to regulate squeezing while allowing the bottle to recover between successive interactions; and 2) a comprehensive set of experimental validations on the TIAGo Pro robot (PAL Robotics, S.L., Spain), demonstrating that the proposed controller can more accurately replicate human demonstrations and achieve improved dispensing accuracy when compared to a force-only controller, across different bottle sizes, target amounts, and number of squeezes.

The adaptive hybrid force-vision controller framework

Figure 1. The adaptive hybrid force-vision controller framework detailing the EMG to grip force mapping, EMG signal processing, and bottle deformation estimation using vision inputs into the adaptive hybrid controller.

Background

Electromyography Usage in Robotics

Electromyography (EMG) measures the electrical signals generated from neuromuscular activity, and can be recorded non-invasively using surface electrodes (sEMG) [8]. Since processed EMG amplitude is correlated with force generated by the muscle, EMG signals can be used to estimate applied force during isometric tasks [9]. This enables the collection of human grasping demonstrations without interfering with the manipulated object or restricting natural hand motion.

EMGs are widely used to estimate human force through muscle activity for human-robot interaction. Xue et al. and Gowtham et al. used EMGs to measure muscle activity in the forearm and predicted grip force for position control of a robotic gripper [7], [10]. Geetha et al. used EMGs for arm gesture recognition to control a mobile manipulator [11]. Cheon et al. used EMGs to actuate a robotic glove to augment the user’s grasping force [12]. Dosen et al. used EMGs for feedback control to augment the gripping force of a myoelectric prosthesis [13]. While these works demonstrate the potential of EMGs for human-in-the-loop control, they primarily focus on rigid-object manipulation or force augmentation. EMG signals provide information about human force but do not directly capture changes in the manipulated object’s state during interaction. Therefore, their application to contact-rich deformable object manipulation remains underexplored.

Deformable Object Manipulation in Robotics

Deformable objects present a unique challenge for robot manipulation tasks since the object’s state changes as it is manipulated. As Zhu et al. discuss in a review, deformable object manipulation remains an active area of research needing development across hardware design, sensing, deformation modeling, planning, and control [14]. Some works in literature rely on model-based estimation for a soft object’s physical properties [15], [16] [2]. However, these models often require extensive calibration and are difficult to generate, limiting their scalability when manipulating objects with large variations in material properties.

Some approaches avoid manually creating a model by using force feedback and learning. Lee et al. learned force-based manipulation from human demonstrations of end-effector poses and the applied forces at each pose [17]. Sanchez et al. used measured contact forces while deforming objects to continuously update a deformation model [5]. Zhong et al. proposed a regressor-based approach in which using force feedback, the robot can continuously improve its model of a linear deformable object’s behaviour instead of using a fixed model [18]. However, force feedback alone provides limited information about the overall state of the object outside of the force sensor contact points.

Other approaches adopt model-free methods by using computer vision to track an object’s deformations. Maitin-Shepard et al. used vision-based grasp point detection to detect the corners of cloth to autonomously fold a towel [19]. Sun et al. used RGB-D images of clothing on an active stereo robot head to interpret surface wrinkles in clothing for dual-arm manipulation [20]. Lee et al. demonstrated in two studies that using RGB-D images and point clouds to track soft-object deformations for adaptation of impedance control parameters improved both the accuracy and precision of the manipulation of doughs and soft tissues [21], [22].

While model-based methods have demonstrated effective deformable object manipulation, scaling such approaches to laboratory automation remains challenging. In laboratory environments, robots may interact with a wide variety of deformable tools and containers, making it impractical to manually construct or learn object-specific physical models for each object. Therefore, this work adopts a model-free vision-based approach to estimate deformation during manipulation, providing object-state information that complements force feedback.

Hybrid Force-Vision Controllers

A central theme in deformable object manipulation is that both force feedback and object deformation information is necessary for the task. Hybrid force-vision control leverages both sensing modalities to fuse contact-rich data with overall object state. Leite et al. and Antonio et al. used hybrid vision-force controllers to interact with and estimate unknown surface geometry using a fixed camera and wrist-mounted force sensors [23], [24]. Sheng et al. extended this work to apply hybrid vision force control on deformable surfaces using variable impedance control to handle the non-linear and dynamic stiffness [25].

These works on hybrid force-vision control primarily focus on autonomous manipulation and do not consider the integration of human manipulation strategies into deformable object interaction. Humans naturally combine muscle activation, contact force, and visual feedback to adapt their actions during manipulation. However, the integration of these sensing modalities for reproducing human manipulation behaviours in deformable object interaction remains underexplored. Therefore, this work investigates the integration of

EMG-derived human force references with force and vision feedback for human-like deformable object manipulation.

Methodology

Framework

The framework proposed in this paper consists of four main components, as shown in Figure 1. First, an EMG to grip force mapping using FSRs must be established to translate EMG signals to force signals to be used as the force reference for the controller. Second, EMG signal processing must also be performed to extract usable EMG envelopes. Third, a point cloud vision system is used to track the deformation of the squeeze bottles for adaptation in the hybrid controller. Lastly, the adaptive hybrid controller combines these signals to replicate the bottle squeezing action from the EMG-derived human demonstration.

EMG Placements and Signal Processing

The Shimmer3 EXG sensor units (Shimmer Research Ltd., Ireland) were used for this work. The EMG signal was collected at 1024 Hz with a gain of 12 on the sensor unit. The signal was rectified and filtered using a 4th-order Butterworth bandpass filter at 20-500 Hz to remove noise and movement artifacts to extract the EMG envelope [26]. The placement of sEMG to measure muscle activity to predict grip force is an extensively researched topic. The flexor digitorum superficialis (FDS) is a muscle in the forearm that flexes the middle phalanges (long bones in fingers) of the medial 4 digits, and reinforces flexion against resistance (squeezing) [27]. Literature has seen success in estimating grip force using sEMGs on the FDS [7], [12], [28]. From literature and experimental trials of bottle squeezing, sEMG electrodes were placed as seen in Figure 2. For this work, a ground terminal was placed on the distal radius bone, and the positive and negative terminals were placed approximately on the centre of the FDS.

FSRs and sEMG electrode placements on human operator’s arm

Figure 2. FSRs and sEMG electrode placements on human operator’s arm for FSR to EMG mapping data collection.

Gripper Design

The two-finger gripper used in this paper has standard flat-surface fingers. Therefore, the concave deformation of the bottle when squeezed causes sensors on flat grippers to lose contact with the bottle’s surface. As such, custom gripper fingers had to be designed to ensure the force sensor’s contact and accurate reading throughout the squeezing task.

Leveraging the flexible FSRs, curved surface fingers were designed to ensure reliable contact between the sensor and bottle, even during deformation. This rounded finger design and placement of the FSRs on the gripper’s fingers can be seen in Figure 3. Furthermore, the FSR sensor has a 4x4 grid of tactile pressure sensing nodes. With a FSR sensor placed on each finger of the gripper, the average of the 32 node readings were used as the measured gripping force.

End-effector with Intel RealSense D405 camera and custom rounded-finger gripper

Figure 3. End-effector with the Intel RealSense D405 camera at the wrist and custom rounded-finger gripper equipped with force sensitive resistors.

EMG to FSR Mapping

To transfer human squeezing force via muscle activations, a map between EMG signals to grip force must be established. To collect the data for this mapping, EMG signals of the FDS muscle activity and FSR signals of the squeezing force were collected simultaneously during squeezing of a foam block. The sEMG electrodes were placed as described in sec. III-B and FSRs were fixed onto rigid 3D-printed caps on the user’s thumb and middle finger (see Figure 2) to best replicate the gripping of the designed rigid two-fingered robotic gripper.

Bottle Deformation Estimation Using Computer Vision

The designed controller performs its adaptation based on a visual estimation of the squeeze bottle’s deformation relative to its undeformed size. This allows deformation estimation to be independent of bottle size. A depth camera placed on the manipulator’s final link, angled downward at 35∘35^\circ, is used to capture the point cloud of the bottle during the squeezing task. This places the centre of the captured image at the centre of the gripper, as seen in Figure 3.

The point cloud processing to estimate the bottle deformation was done using the Point Cloud Library (PCL) with the following steps [29]:

  1. Extract bottle points from the captured point cloud using color-based clustering and the known gripper location in the camera frame.

  2. Transform the point cloud to align the bottle’s cross-section with the xx-zz plane.

  3. Estimate the undeformed bottle width by averaging the widest xx-zz slice over a 1 cm height range (green region in Figure 4).

Vision deformation estimation system with point cloud measurement points overlaid

Figure 4. Vision deformation estimation system with point cloud measurement points overlaid. Green points represent those used for the undeformed bottle size reference and red points represents the deformed measurement.

  1. Estimate the deformed bottle width by averaging the thinnest xx-zz slice over a 1 cm height range (red region in Figure 4).

  1. Calculate deformation as the percentage reduction from the undeformed to deformed width, enabling size-independent measurement.

Adaptive Hybrid Control

To replicate human bottle-squeezing behaviour using a position-controlled robotic gripper, EMG signals recorded during human demonstrations were mapped to desired FSR readings. This desired squeezing force profile was then used as the force reference for the position based force controller. However, force control alone was found to reproduce the squeezing phase more effectively than the release phase. In particular, the gripper could close in response to increasing EMG-derived force commands, but often failed to reopen when the EMG signal decreased, even when the mapped desired force indicated relaxation. This limitation motivated the introduction of a hybrid adaptive controller that combines EMG-derived force feedback with vision-based estimation of bottle deformation, as illustrated in Figure 1.

The controller is designed such that force feedback dominates during active squeezing, when the EMG signal is high, while bottle-deformation feedback becomes dominant when the EMG signal is low. This allows the gripper to respond not only to the desired squeezing force but also to the deformation state of the bottle, particularly during grip relaxation and reopening. The desired gripper position is given by:

xd=x−σKp,f,adapt(f−fd)−(1−σ)Kp,deformBdeform−Kdx˙.(1)x_d = x - \sigma K_{p,f,\mathrm{adapt}}(f - f_d) - (1-\sigma)K_{p,\mathrm{deform}}B_{\mathrm{deform}} - K_d\dot{x}. \tag*{(1)}

where xdx_d is the desired gripper position, xx is the measured gripper position, and x˙\dot{x} is the measured gripper velocity. The desired force fdf_d is obtained from the EMG-to-FSR mapping, while ff is the measured FSR force at the gripper. The term BdeformB_{\mathrm{deform}} represents the bottle deformation estimated from vision. The modulation factor σ\sigma varies with the EMG signal and determines the relative contribution of force and deformation feedback: larger values of σ\sigma place greater emphasis on force control, whereas smaller values increase the contribution of deformation feedback. When using pure force control to evaluate the impact of the machine vision, σ\sigma was set to 1 to remove the vision based compensation and only utilize force feedback. The gain Kp,f,adaptK_{p,f,\mathrm{adapt}} is the adaptive proportional gain applied to the force error, Kp,deformK_{p,\mathrm{deform}} is the proportional gain associated with bottle deformation, and KdK_d is the damping gain applied to the gripper velocity.

Additional experiments showed that greater squeezing force was required as the deformation of the bottle increased. To account for this behaviour, the proportional gain applied to the force error was made adaptive on the estimated bottle deformation. The adaptive force gain is defined as:

Kp,f,adapt=Kp,f+Kp,f,deformmax⁡(Bdeform−Bnom,0)(2)K_{p,f,\mathrm{adapt}} = K_{p,f} + K_{p,f,\mathrm{deform}}\max(B_{\mathrm{deform}} - B_{\mathrm{nom}},0) \tag*{(2)}

where Kp,fK_{p,f} is the nominal proportional gain on the force error, and Kp,f,deformK_{p,f,\mathrm{deform}} determines how strongly the force gain increases with bottle deformation. The parameter BnomB_{\mathrm{nom}} defines the deformation threshold above which the additional gain becomes active. Consequently, the controller increases its force-feedback gain only when the estimated bottle deformation exceeds the nominal deformation level, enabling stronger squeezing action when larger forces are required.

Experiments and Results

Experimental Setup

The experimental setup for the adaptive hybrid controlled robotic bottle squeezing task is depicted in Figure 5. The TIAGo Pro’s gripper (position-controlled from 0 (opened) to 1 (closed)) was equipped with the custom designed fingers using RX-M0404S FSR sensors (Roxi Electronic Technology Co., Ltd., China) and an Intel RealSense D405 depth camera (Intel Corporation, USA) was attached at the end-effector’s final link for the vision-based bottle deformation estimation. Commercial plastic wash bottles were used as the squeeze bottles, and a digital scale with ±\pm1 mL accuracy was used to measure the water dispensed from the experiments.

Experimental setup of the robotic bottle squeezing task

Figure 5. Experimental setup of the robotic bottle squeezing task using the TIAGo Pro, an Intel RealSense D405 camera, and an FSR-equipped gripper used to squeeze a plastic bottle to dispense water into a cup on a digital scale.

The experimental setup to collect human demonstrations of bottle squeezing is depicted in Figure 6. For the purpose of this paper, data from a single user (one of the authors) was collected. The user was equipped with FSR caps on their thumb and middle finger and sEMGs were placed on the FDS and distal radius bone to measure the FDS muscle activity. The user was instructed to squeeze the bottle with only their thumb and middle finger to dispense water into the cup on the digital scale.

Experimental setup to collect human demonstrations of bottle squeezing

Figure 6. Experimental setup to collect human demonstrations of bottle squeezing. FSR finger-caps are placed on the thumb and middle finger and sEMG electrodes are placed on the FDS and distal radius bone while the user dispenses water using the plastic squeeze bottle onto the digital scale.

This experiment compared the dispensed volume of water using the squeeze bottle with a tuned force controller and a tuned adaptive hybrid force-vision controller presented in this work. The tuned values used were Kp,f=0.0002K_{p,f} = 0.0002, Kp,f,deform=0.001K_{p,f,\mathrm{deform}} = 0.001, Kp,deform=0.003K_{p,\mathrm{deform}} = 0.003, Kd=0.1K_d = 0.1, and Bnom=20%B_{\mathrm{nom}} = 20\%. Based on the transition zone of 0-0.03mV in the EMG to FSR mapping seen in Figure 7, the hybrid modulation factor was chosen to have the relation σ=max⁡(0.03−EMG0.03,0)\sigma= \max\left(\frac{0.03-\mathrm{EMG}}{0.03}, 0\right). During the force controller trials, σ\sigma was set to 1 to remove any visual components and rely only on grip force feedback.

Trained regression of the mean force sensitive resistor values as a function of flexor digitorum superficialis electromyography values during squeezing

Figure 7. Trained regression of the mean force sensitive resistor (FSR) values as a function of flexor digitorum superficialis (FDS) electromyography (EMG) values during squeezing.

To test the dispensing accuracy and human demonstration replicability of the controller on various bottles, the experiment in this study tested 12 variations of squeeze bottle dispensing cases where the amount of liquid dispensed, bottle size, and number of squeezes per trial were varied. The variations combined dispensing approximately 40mL or 80mL of water, using small (250 mL) or large (500 mL) squeeze bottles, and using 1, 2, or 3 squeezes to dispense the liquid at a rate of approximately 1 mL/s. Two human demonstration of each case were collected and recorded using ROS2 bags. The recorded EMG signals for each of the 24 human demonstrations were then replayed and used as inputs for the force reference of the adaptive hybrid controller and the force controller. Five trials of each case for each controller were collected.

Results

EMG to FSR Mapping

The data collected to create the mapping between the mean FSR values as a function of FDS EMG values was recorded at 100 Hz with 1 operator performing this squeezing task for 50 trials. For each trial, the operator squeezed the block from the minimum contraction required to hold the block to their maximum voluntary contraction over approximately 3 seconds with 1 second breaks in between. The regression relationship was obtained

by training a multilayer perceptron network consisting of two hidden layers with four units each and a single-unit output layer. Sigmoid activation functions were applied to the hidden layers, while the network weights were constrained to be positive using a Softplus transformation. The model was trained using L1 loss, with early stopping applied when the validation loss failed to improve by more than 10−610^{-6} for 2000 consecutive epochs. The collected data points and the resulting regression can be seen in Figure 7. The resultant regression has an L1-loss of 18.9442. This regression is used as the mapping from EMG to desired FSR force values in the controllers of this experiment.

Bottle Deformation Estimation Using Computer Vision

The performance of the bottle deformation estimation vision system is evaluated in Figure 8. The vision system accurately estimates bottle deformation within the range of 0–65% deformation, achieving mean absolute errors (MAEs) of 5.26% and 6.96% for the small and large bottles, respectively. However, the estimation accuracy decreases significantly beyond 70% deformation. At high deformation levels, the visible shape of the bottle changes substantially, causing the deformation estimation to use less deformed areas of the bottle as it cannot detect the fully deformed area between the gripper. As a result, inaccurate deformation estimates are produced in this highly deformed region. Therefore, to ensure reliable deformation feedback during control, the maximum gripper closing position was limited to 0.8, corresponding to approximately 70% bottle deformation, where the vision estimator maintains sufficient accuracy.

Vision estimated compared to measured bottle deformation from 0-90% deformation

Figure 8. Vision estimated compared to measured bottle deformation from 0-90% deformation.

Adaptive Hybrid Control vs. Force Control

Figure 9 presents the dispensed-volume distributions for the adaptive hybrid and force controllers across all bottle, target-volume, squeeze-count, and human demonstration trial conditions. For each condition, the candlestick plots summarize the five repeated robot trials, while the red horizontal lines indicate the corresponding volume dispensed during the human demonstration. Table I complements this figure by summarizing the dispensing performance for each condition and human demonstration. The table reports the standard deviation of the five robot dispensing trials, together with the MAE and root mean squared error (RMSE) relative to

Volume dispensed from adaptive hybrid control and force control

Figure 9. Volume dispensed using adaptive hybrid and force control under different bottle and squeezing conditions. Each candlestick pair represents the five repeated robot trials corresponding to human demonstrations 1 (left) and 2 (right), with red lines indicating the human dispensed volumes.

BottleTargetSqueezesHuman DemonstrationAdaptive Hybrid ControlForce Control
SDMAERMSESDMAERMSE
Small40111.102.202.417.8914.6016.22
Small40122.515.606.032.6116.4016.57
Small40213.837.207.971.3029.8029.82
Small40223.397.007.636.7625.2025.92
Small40312.412.002.198.3528.8029.75
Small40322.617.407.764.2829.6029.85
Small80113.278.208.711.529.409.50
Small80125.3211.6012.542.0011.0011.14
Small80216.966.006.235.074.206.18
Small80221.305.205.334.154.404.90
Small80316.025.606.071.146.606.68
Small80321.309.809.871.105.205.29
Large40112.704.605.202.0029.0029.06
Large40121.794.204.492.8824.6024.73
Large40211.952.802.972.7925.4025.52
Large40224.219.209.942.8825.6025.73
Large40313.052.403.163.4430.6030.75
Large40321.954.604.922.8825.6025.73
Large80114.604.804.982.055.806.08
Large80124.396.807.515.768.209.69
Large80212.3012.4012.574.305.406.31
Large80222.419.609.841.524.604.80
Large80312.689.8010.094.024.004.56
Large80323.328.008.538.048.009.90

Table I. Per-condition dispensing performance of the adaptive hybrid vision-force and force controllers. SD denotes the standard deviation of the five robot dispensing trials per-condition. MAE and RMSE are computed relative to the corresponding human-demonstration dispensed volume. All values are in mL.

the corresponding human-demonstration dispensed volume. Figure 10 shows the time-series responses for both controllers for the first trial of each experimental condition. The figure includes the EMG command, the commanded FSR signal from the EMG to FSR mapping, the measured FSR signal during the bottle squeezing task, and the resultant gripper position from each controller, over time. Only one trial has been presented here to provide a clear visual, but these results represents the general response across all trials.

Time-series comparison of trial 1 across all experimental conditions

Figure 10. Time-series comparison between the adaptive force-vision and force control methods for trial 1 across all bottle, target volume, and squeezing conditions. The solid black line represents the commanded FSR/EMG value, while the blue dotted lines represent measured signals.

Discussion

The adaptive hybrid controller generally maintained the dispensed volume closer to the human-demonstration target, particularly with the 40 mL conditions. Averaged across the 24 condition–human demonstration combinations, the adaptive hybrid controller achieved an MAE of 6.54 mL and an RMSE of 6.96 mL, compared with 15.92 mL and 16.45 mL, respectively, for force control. The difference was most evident for the 40 mL target, for which the mean MAE across conditions was 4.93 mL for adaptive hybrid control and 25.43 mL for force control. Table I further shows that the variability across the five repeated robot trials was comparable between the two approaches, with mean per-condition standard deviations of 3.14 mL and 3.70 mL for adaptive hybrid and force control, respectively. Overall, these results indicate that the adaptive hybrid controller reproduced the human-demonstration dispensed volume with greater accuracy while maintaining comparable repeatability.

Further differences between the two controllers can be observed in the time-series responses shown in Figure 10. Bottle squeezing presents a unique deformable-object manipulation task in which the relationship between applied force and dispensed volume depends on the residual deformation of the bottle. Specifically, the same squeezing force can produce different dispensed volumes depending on the bottle’s deformation state. Partially releasing the bottle between successive squeezes allows air to re-enter and the bottle to recover towards its undeformed state, altering the force and deformation required for subsequent squeezes. Consequently, reproducing the grip relaxation between squeezes is an important component of the human demonstration.

In the two- and three-squeeze conditions, the adaptive hybrid controller reproduced this relaxation behaviour. Decreases in the EMG command between squeezes were accompanied by reopening of the gripper, allowing the bottle to partially recover before the subsequent squeeze. In contrast, this behaviour was not reproduced by the force controller. The force-controlled gripper generally remained closed after the initial squeeze rather than reopening in response to the reduction in the human-derived EMG command. This suggests that force tracking alone was insufficient to reproduce the temporal grip-relaxation behaviour required for repeated manipulation of the deformable bottle.

A similar trend can be observed across the gripper-position responses of the force controller. Across all tested conditions, the gripper typically closed to approximately 0.75 and remained near this position for the remainder of the trial. This provides a possible explanation for the clustering of force-controlled dispensed volumes in the approximately 60–80 mL range observed in Figure 9. Because the force controller produced similar gripper trajectories across different conditions, the resulting bottle deformation and dispensed volume also tended to be similar despite differences in the demonstrated squeezing behaviour. These results indicate that reproducing the force response alone does not provide sufficient control to capture the condition-dependent manipulation behaviour of the deformable bottle.

Overall, the two controllers produced substantially different squeezing behaviours despite being driven by the same human-derived EMG command profile. The adaptive hybrid controller reproduced the grip-relaxation pattern associated with repeated human squeezing while also achieving lower dispensing errors overall. This demonstrates the benefit of incorporating object deformation information alongside force feedback when reproducing human manipulation of deformable objects.

Conclusion and Future Work

This work presented an adaptive hybrid force-vision controller that integrates EMG-derived human force references, force feedback from FSRs, and vision-based bottle deformation estimation for deformable bottle manipulation. The proposed controller was experimentally validated across 12 squeezing conditions with varying bottle sizes, target dispensed volumes, and squeezing patterns. Compared with a pure force controller, the proposed approach achieved significantly closer accuracy with human-demonstrated dispensed volumes by adapting the gripper behaviour based on both interaction forces and object deformation. These results demonstrate the importance of incorporating deformation information alongside force feedback for deformable object manipulation. Furthermore, the successful reproduction of EMG-driven human squeezing behaviours highlights the potential of EMGs as a non-invasive modality for collecting natural human demonstrations in contact-rich manipulation tasks, enabling future research in learning from demonstration and behaviour cloning.

Future work will focus on improving the generalizability of the proposed approach for broader laboratory automation scenarios. Variations in bottle materials, liquid viscosity, and object geometry may affect the relationship between applied force and resulting deformation, requiring additional sensing and adaptive mechanisms. Furthermore, extending the approach to other laboratory equipment, including fragile glass containers, will require control strategies that can balance secure grasping with safe contact forces to prevent damage while maintaining reliable manipulation. Ultimately, this research aims to enable robots to acquire human-like manipulation skills for complex laboratory environments through intuitive and non-invasive human demonstrations.

References

  1. [1]K. Thurow et al., “Editorial: Robotics in laboratory automation,” SLAS Technology, vol. 36, p. 100373, 2026.DOI
  2. [2]V. E. Arriola-Rios et al., “Modeling of deformable objects for robotic manipulation- A tutorial and review,” Frontiers in Robotics and AI, vol. 7, 2020.
  3. [3]A. I. Cooper et al., “Accelerating discovery in natural science laboratories with ai and robotics: Perspectives and challenges,” Science Robotics, vol. 10, no. 106, p. eadv7932, 2025.DOI
  4. [4]Q. Cong, W. Fan, and D. Zhang, “Tactf-gripper: A reconfigurable fin-ray-based gripper with tactile skin for in-hand manipulation,” in Actuators, vol. 13, no. 12. MDPI, 2024, p. 521.DOI
  5. [5]I. Sanchez et al., “Blind manipulation of deformable objects based on force sensing and finite element modeling,” Frontiers in Robotics and AI, vol. 7, 2020.DOI
  6. [6]C. Gentile et al., “Manipulation tasks in hazardous environments using a teleoperated robot: A case study at cern,” Sensors, vol. 23, no. 4, 2023.DOI
  7. [7]J. Xue and K. W. C. Lai, “Dynamic gripping force estimation and reconstruction in emg-based human-machine interaction,” Biomedical Signal Processing and Control, vol. 80, p. 104216, 2023.DOI
  8. [8]V. Gohel and N. Mehendale, “Review on electromyography signal acquisition and processing,” Biophysical reviews, vol. 12, no. 6, pp. 1361–1367, 2020.DOI
  9. [9]J. C. Coggshall and G. A. Bekey, “EMG-force dynamics in human skeletal muscle,” Medical and biological engineering, vol. 8, no. 3, pp. 265–270, may 1970.pubmed.ncbi.nlm.nih.gov/5491714
  10. [10]S. Gowtham et al., “Emg-based control of a 5 dof robotic manipulator,” in International Conference on Wireless Communications Signal Processing and Networking (WiSPNET), 2020, pp. 52–57.DOI
  11. [11]K. Geetha et al., “Emg controlled mobile robot equipped with gripper mechanism for fine motor skills training in rehabilitation,” Results in Engineering, vol. 26, p. 104563, 2025.
  12. [12]S. Cheon et al., “Single emg sensor-driven robotic glove control for reliable augmentation of power grasping,” IEEE Transactions on Medical Robotics and Bionics, vol. 3, no. 1, pp. 179–189, 2021.DOI
  13. [13]S. Dosen et al., “Emg biofeedback for online predictive control of grasping force in a myoelectric prosthesis,” Journal of NeuroEngineering and Rehabilitation, vol. 12, no. 1, p. 55, 2015.DOI
  14. [14]J. Zhu et al., “Challenges and outlook in robotic manipulation of deformable objects,” IEEE Robotics & Automation Magazine, vol. 29, no. 3, pp. 67–77, 2022.arxiv.org/abs/2105.01767
  15. [15]T. Wada et al., “Robust manipulation of deformable objects by a simple pid feedback,” in Proceedings. IEEE International Conference on Robotics and Automation, vol. 1, 2001, pp. 85–90.DOI
  16. [16]T. Wada, Indirect Simultaneous Positioning of Extensible Deformable Objects. London: Springer London, 2000, pp. 73–89.DOI
  17. [17]A. X. Lee et al., “Learning force-based manipulation of deformable objects from multiple demonstrations,” in IEEE International Conference on Robotics and Automation (ICRA), 2015, pp. 177–184.DOI
  18. [18]H. Zhong et al., “Regressor-based model adaptation for shaping deformable linear objects with force control,” in IEEE International Conference on Robotics and Biomimetics (ROBİO), 2023, pp. 1–7.DOI
  19. [19]J. Mautin-Shepard et al., “Cloth grasp point detection based on multiple-view geometric cues with application to robotic towel folding,” in IEEE International Conference on Robotics and Automation, 2010, pp. 2308–2315.
  20. [20]L. Sun et al., “Accurate garment surface analysis using an active stereo robot head with application to dual-arm flattening,” in IEEE International Conference on Robotics and Automation, 2015, pp. 185–192.DOI
  21. [21]J. Lee et al., “VAISI: Vision-based adaptive impedance-control for surgical incisions,” in IEEE-RAS 24th International Conference on Humanoid Robots (Humanoids), 2025, pp. 1–8.
  22. [22]——, “VAIRO: A vision-based adaptive impedance-control robotic framework,” in 34th IEEE International Conference on Robot and Human Interactive Communication (RO-MAN), 2025, pp. 462–468.DOI
  23. [23]A. C. Leite et al., “Hybrid vision-force robot control for tasks on unknown smooth surfaces,” in Proceedings IEEE International Conference on Robotics and Automation, 2006, pp. 2244–2249.DOI
  24. [24]——, “Hybrid adaptive vision—force control for robot manipulators interacting with unknown surfaces,” The International Journal of Robotics Research, vol. 28, no. 7, pp. 911–926, 2009.DOI
  25. [25]Q. Sheng et al., “Hybrid vision-force robot force control for tasks on soft tissues,” in 27th International Conference on Mechatronics and Machine Vision in Practice (M2VIP), 2021, pp. 705–710.DOI
  26. [26]P. Konrad, “The abc of emg,” A practical introduction to kinesiological electromyography, vol. 1, 01, 2005.
  27. [27]L. Okafor and M. A. Varacallo, “Anatomy, shoulder and upper limb, hand flexor digitorum superficialis muscle,” in StatPearls [Internet]. StatPearls Publishing, 2022.
  28. [28]T. Bazina et al., “Koopman-driven grip force prediction through emg sensing,” IEEE Transactions on Neural Systems and Rehabilitation Engineering, vol. 33, pp. 2192–2202, 2025.DOI
  29. [29]R. B. Rusu and S. Cousins, “3D is here: Point Cloud Library (PCL),” in IEEE International Conference on Robotics and Automation, Shanghai, China, 2011.DOI

Paper details

vlnc:W_5uu7mdyh
Version
v1 · latest
Published
Fresh — added to the Hub within the last 30 days
Added
License
CC BY 4.0
Usage

views · 8

downloads · 0

A view counts once per browser per day, a download each time the PDF is requested.

  • Computer Science

Not recorded for this paper

View statement
Looking for related papers…
Contents