betaFeedback

WorksComputer Science

Accelerated Reinforcement Learning for Soft Robotics: Hours Instead of Years

Abstract

Reinforcement learning (RL) requires large amounts of interaction data or a reliable simulator, both hard to obtain for nonlinear soft robotic hardware. Recent work shows that a time-scale separation inherent to soft robot dynamics justifies their reduction to low-dimensional invariant manifolds, known as adiabatic spectral submanifolds (aSSMs). We demonstrate that an aSSM-reduced model, a simple polynomial ordinary differential equation learned purely from pre-collected data, can serve as an RL training environment. On a pneumatically actuated soft arm, we train a trajectory-tracking policy in 3 hours on a six-dimensional aSSM-reduced model and deploy it without fine-tuning, whereas reproducing this policy training extrapolates to 3 years of hardware interaction. Our policy tracks dynamic targets with 40% lower error than otherwise identical policies trained on a 4.8-million-parameter recurrent neural network (RNN) or a 1.7-million-parameter transformer, which take 13 and 100 hours to identify, against 5 minutes for our mathematically justified 6,924-parameter model.

Learning control policies for robots to excel at human- or animal-oriented skills, such as walking, swimming, flying, and playing physical or cognitive games, is an ambitious open problem of current interest in the machine learning and robotics communities (see Kober et al. [24] and Ibarz et al. [19] for a review). Since biological morphologies and their associated mechanics heavily inspire robot designs, the development of general-purpose controllers that emulate the trial-and-error learning processes of biological organisms to master complex skills has received broad interest.

Reinforcement learning (RL) formalizes control by trial and error (see Sutton and Barto [38]). An RL agent learns the control policy that maps an observation of the robot and the current target to a distribution over control actions. Control actions are sampled from the policy and applied to the robot. The resulting state is observed, and the policy parameters are updated to maximize a reward that quantifies the proximity to the target associated with the skill.

Methods that learn from rewards evaluated on interaction data, the model-free family, require significantly more data than a physical robot can usually provide, since every environment step costs wall-clock time and mechanical life of the hardware. Developments in rigid robotics alleviated this by moving the exploration off the hardware into high-fidelity, parallelizable simulators (see Makoviychuk et al. [27]), an approach that produced the locomotion (see Miki et al. [30]) and manipulation (see Kroemer et al. [25]) results now state of the art on a variety of rigid robotic platforms such as ANYmal and the Franka arm.

The same pipeline is generally not available for soft robots. For certain types of soft robotic hands (see Toshimitsu et al. [42]) and continuum arms (see Satheeshbabu et al. [36]), rigid-body physics engines such as MuJoCo, or elastic rod-based simulators (see Naughton et al. [33]) that approximate their kinematics already aid in training model-free RL policies. But they suffer from the simulation-to-reality gap for dexterous manipulation tasks. On the other hand, current soft robotic simulators that are able to capture the nonlinear dynamics (see Faure et al. [13], Mekkattu et al. [29], and Parthasarathy et al. [35]) are computationally costly to be integrated in the policy training loop. These practical limitations motivate the development of learning RL policies directly on hardware soft robot data.

Attempts to control soft robots through an RL policy trained solely on hardware data are not new. The earliest work by Braganza et al. [7] learned a model-free policy to control a soft robotic manipulator. The method, however, does not appear scalable to complex soft-robot geometries. Indeed, even with advances in RL algorithms and computational power, no successful model-free policies for complex soft robots have emerged to date. The reasons are not surprising, as model-free policies require extensive hardware exploration to converge, which, for delicate soft robots, is nearly impossible because their softness, coupled with repeated use, leads to memory effects and plasticity that tend to cause hardware failure.

At the crossroads of RL implementation difficulties: dealing with delicate hardware and no access to a reliable, fast simulator. Large black-box neural network-based models trained on dynamic hardware data to model the dynamics of soft robots offer a solution. The underlying hope is that the neural network’s large parameter count provides a reasonable estimate of the soft robot’s high fidelity, and the more appealing aspect is that neural-network-based control policies are largely compatible with training on such models, ensuring fast, parallelizable training that explores the model’s state space. These are termed model-based RL methods and are further sub-classified as online when policy training happens directly by interacting with hardware, or offline when policy training happens using the model as a training environment.

On soft robotic manipulators, Thuruth et al. [40] learns a nonlinear autoregressive model of the dynamics using a one-step memory window. They then employ this model inside a classical model predictive controller to generate target trajectories, on which an RL agent learns a control policy while interacting with the hardware. Recent work by Zheng et al. [44] and Nan et al. [32] uses a larger neural network with one-step memory information and trains simultaneously with a control policy on hardware data. These online model-based methods have seen success, but, as their authors point out, it is unclear whether the errors can be attributed to the policy or to model learning (see Janner et al. [20]). All in all, these approaches, though data-efficient, still converge to RL policies that are task-specific and need additional interaction with the hardware to be generally applicable.

To mitigate these issues, Nazeer et al. [34] and Zhang et al. [43] focus on training a long short-term memory (LSTM) and a multilayer perceptron (MLP), respectively, both with larger memory windows, and then train control policies on these models. However, they report accuracy gaps when applied to the hardware and hence use online model-free RL methods to fine-tune their predictions. For purely offline model-based methods for soft robots, Berdica et al. [3] and Centurelli et al. [12] use LSTMs to learn the dynamics of soft arms and train policies on them. Notably, the latter tracks dynamic targets using an LSTM-based RL policy, achieving accuracy on planar trajectories. Their precollected data for training the LSTM is modest. Still, they offer no systematic guidelines for choosing the network architecture, hyperparameters, and memory windows, which are essential for LSTMs, which are prone to long-term prediction inaccuracies and poor generalization to unseen test data (see Kaundinya et al. [23]).

As an alternative to data-driven black-box modeling, the mathematical theory of spectral submanifolds (SSMs), which are low-dimensional attracting invariant manifolds (see Haller [15] for a review), provides a rigorous geometric understanding of the dominant dynamics of nonlinear dynamical systems. Specifically, restricting dynamics to slow SSMs allows for justified model reduction of nonlinear continuum mechanical systems described by ordinary differential equations (ODEs) (see Haller and Ponsioen [17] and Haller et al. [18]) or partial differential equations (PDEs) (see Buza and Haller [8]). The resulting SSM-reduced model is a low-dimensional ODE, which can be extracted purely from data as a simple polynomial model (see Cenedese et al. [9] and Axås et al. [2]), and has demonstrated predictive accuracy in a range of mechanical experiments subject to previously unseen external forcing (see Cenedese et al. [10], Cenedese et al. [11], Bettini et al. [4], Bettini et al. [5], and Morsy et al. [31]).

Dominant dynamics of nonlinear soft robots captured by slow SSMs have already been locally extracted near equilibria and used for controls in Alora et al. [1] for a hardware Diamond robot. Only recently, in Kaundinya et al. [21], a time-scale separation phenomenon was identified as common to all soft robotics control operations, enabling the global extraction of the dominant dynamics onto adiabatic spectral submanifolds (aSSMs) via extensions to the local theory of SSMs (see Haller and Kaundinya [16]). They show, across a variety of soft robotic simulators, that aSSMs extracted from data can be used in a model predictive control (MPC) scheme successfully for accurate trajectory tracking.

This methodology has very recently been demonstrated on hardware, by further reducing aSSMs onto slow manifolds to control artificial muscles (see Bettini et al. [6]) and to precisely control helicoid soft robots along 3D target tracks (see Kaundinya et al. [22]). Existing aSSM-based controllers pair the reduced model with classical numerical control techniques, which linearize the dynamics at each solve step. As a consequence, the nonlinear structure of the aSSM model is only partially exploited. Control methods, such as an RL agent optimizing policies directly on nonlinear model rollouts, without repeated linearization, have therefore remained unexplored for aSSM-reduced models. We will fill this gap with the present work.

Contribution: We develop a method to train RL control policies offline on a data-driven aSSM-reduced model (see Figure 1c), without the RL training interacting with the hardware robot. Specifically:

Overview of learning an aSSM RL policy on a soft robot

Figure 1. (a) Snapshot of the SoPrA arm. (b) Geometry of the adiabatic spectral submanifold (aSSM) in the state- and control action- space. (c) General overview of training an RL agent on an aSSM-reduced model.

  • [aSSM for a new hardware platform] We extract and learn a six-dimensional aSSM-reduced model for a pneumatically actuated soft robot, a soft continuum proprioceptive arm (SoPrA) (see Extended Data Figure 1a and § 1.1), purely from hardware data. Previously, the data-driven aSSM methodology has been tested on tendon-driven trunk robots (see Kaundinya et al. [22]). This demonstrates the generality of our method, which is agnostic to the physics of the actuating mechanics.

  • [Pareto dominant aSSM-RL policies] Combining traditional RL policy optimization (see Schulman et al. [37]) with our aSSM-reduced model (see § 1.3) leads to Pareto dominant aSSM-RL control policies across model complexity, model learning time, RL policy training time, and closed-loop accuracy (see § 1.4). We use baselines that are state-of-the-art autoregressive neural network-based models from Li et al. [26], which they term robotic world models (RWMs).

  • [Faster control] Our aSSM-RL policy operates at a controller frequency of 100 [Hz], which provides a 100x speed-up over neural-network-based MPC controllers in Zheng et al. [44] and a 2x speed-up over aSSM-based MPC controllers in Kaundinya et al. [22].

  • [Improved closed-loop accuracy compared to prior work] Our aSSM-RL policy reduces closed-loop tracking error by 40% relative to the best-performing autoregressive model-based baseline, across all trajectory tracking tasks (§ 1.5), and by 20% relative to our own aSSM-MPC baseline. Furthermore, aSSM-RL reduces tracking error by 55% relative to the LSTM-based policy of Centurelli et al. [12] and by 76% relative to the online model-based policies of Nan et al. [32].

Results

We present a detailed case study on the performance of model-based RL policies for controlling a soft continuum arm. Specifically, our experimental platform is the SoPrA arm (see Toshimitsu et al. [41] for details), a pneumatically actuated soft continuum arm. The SoPrA arm is mounted vertically and hangs against gravity (see Extended Data Figure 1a). The cross-sectional radius of the arm at the free end is 2 [cm] and 3 [cm] at the base. It has a longitudinal length of 30 [cm]. The arm is assembled by putting together two modules. Each module is cast in silicone material and comprises three expandable chambers. The chambers are connected to a pressure-driven actuation system that sets the pressure in each chamber to control the extent of expansion. We refer to the Supplementary Information for further hardware details.

An external motion capture system monitors the arm’s movements. The camera system records the positions of the free end, which we refer to as the tip, and the mid-section, which we refer to as the mid, where the modules meet. Our objective is to learn a model of the soft arm’s dynamics using data collected by the camera system and the control actions applied to the robot to generate it. We then use this learned model to explore the soft arm’s control action space and learn an RL control policy for target tracking tasks.

On the modeling side, in § 1.1, we provide a detailed discussion of learning a low-dimensional 6D aSSM-reduced model for the arm. We also learn various, routinely used neural-network-based dynamics models for the arm’s end-effector: multi-layer perceptrons (MLPs), recurrent neural networks (RNNs), and Transformers. Specifically, the RNN and Transformer baselines are the current state-of-the-art in recent work by Li et al. [26], which they denote as robotic world models (RWMs). We systematically compare the predictive power of all these models, including our aSSM-reduced model, for previously unseen random control actions in § 1.2. The exact training details for all the neural network models used for comparisons are provided in the Supplementary Information.

We then use our learned aSSM model in § 1.3 to train a model-based RL policy for closed-loop control of the soft arm and follow this up in § 1.4 by outlining the Pareto dominance of using an aSSM-based RL policy compared to other neural-network-based policies. We finally deploy our aSSM-based RL policy on the hardware arm in § 1.5 to perform closed-loop tracking of 2D targets.

Data-driven modeling of the soft robotic arm via aSSMs

As a continuum, the SoPrA soft arm is an infinite-dimensional dynamical system. Under soft-material constitutive laws and elastic continuum assumptions, the system’s continuum equations of motion can be discretized to construct high-fidelity, high-dimensional dynamical systems. However, there are no robust high-fidelity simulators that faithfully model the hardware. Thus, we must rely on data from a partially observable state space to infer a dynamical model for the SoPrA arm.

Under normal operating condition, the SoPrA arm exhibits an underlying time-scale separation: its intrinsic vibration rates die out significantly faster than the rate of change of control actions generated by the actuation system. Evidence of this time-scale separation has recently been documented in soft robotic hardware: the artificial muscle joint system of Bettini et al. [6] and the soft helicoid trunk robot of Kaundinya et al. [22].

Relevant to our setting, the theoretical work by Haller and Kaundinya [16] extended to the control action setting by Kaundinya et al. [21] provides proofs for the existence of a family of low-dimensional invariant manifolds called adiabatic spectral submanifolds (aSSMs), in the phase space of dynamical systems satisfying the time-scale separation assumption outlined above. Specifically, for target-tracking control, normally attracting aSSMs, Aϵ\mathcal{A}_{\epsilon}, arise as the nonlinear continuations of the collection of dominant slow eigenspaces of the linearization of the robotic system about the frozen-time (i.e., ϵ=0\epsilon= 0) limit of a target trajectory Γ(ϵt)\Gamma(\epsilon t) (see Extended Data Figure 1b). An arbitrary initial condition in the phase space of the robot will converge quickly onto the aSSM and then later converge to the target trajectory. Hence, restricting the dynamics to Aϵ\mathcal{A}_{\epsilon} provides an accurate reduced model of the soft robot that can be used for controls.

Kaundinya et al. [21] and Kaundinya et al. [22] provide rigorous, data-driven methods for extracting low-dimensional, normally attracting aSSMs from partially observable state space coordinates of soft robots. We will closely follow their setup but use notation consistent with standard conventions in the Reinforcement Learning (RL) literature (see Sutton and Barto [38]). We define the partially observable state space of the arm as

s∈S⊆Rns,(1)s \in\mathcal{S} \subseteq\mathbb{R}^{n_s}, \tag*{(1)}

and the finite-dimensional control action,

a∈C⊆Rna,(2)a \in\mathcal{C} \subseteq\mathbb{R}^{n_a}, \tag*{(2)}

where S\mathcal{S} and C\mathcal{C} denote the physically admissible state and control action subsets, respectively. In our current hardware setup for the SoPrA arm, we can only observe the tip and mid positions. Hence we have ns=6n_s = 6, and the observable state vector takes the form

s={x,y,z}={xtip,xmid,ytip,ymid,ztip,zmid}∈R6.(3)s = \{x,y,z\} = \{x_{\mathrm{tip}},x_{\mathrm{mid}},y_{\mathrm{tip}},y_{\mathrm{mid}},z_{\mathrm{tip}},z_{\mathrm{mid}}\} \in\mathbb{R}^{6}. \tag*{(3)}

The pressure-driven actuation system of SoPrA has 6 air chambers, but due to current hardware limitations, only 5 are active and available to the user, which sets na=5n_a = 5.

We investigate the presence of normally attracting aSSMs in the observable space ss of the SoPrA arm by commanding random step control action sweeps to it. These sweeps suffice to probe the leading-order aSSM geometry and its internal dynamics. As shown in Figure 2a, based on data collected from a 2.2-hour step sweep of 400 randomly sampled unique static control actions held for 20 seconds each, the SoPrA arm undergoes underdamped decays that eventually converge to unique static configurations in the observable space. Both of these characteristics support the assumption that normally attracting aSSMs exist in the phase space of the SoPrA arm (see § 3).

Figure 2. [Modeling SoPrA via an aSSM-reduced model]. (a) Response of the SoPrA arm plotted in its state-space subjected to random step control actions. (b) Time series plots of the aSSM model predictions (green) and the SoPrA arm’s true response (black) for unseen step control actions in the workspace of the SoPrA arm. (c) State-space plots of the aSSM model’s predictions (green) and the SoPrA arm’s true response (black) for a periodic control action.

Using a spectrogram analysis of each decaying signal, we infer from the collected data, that the minimal dimension of the normally attracting fibers of the aSSM is d=6d = 6. This can also be deduced from the geometry of the SoPrA arm, which is symmetric in the x−yx-y plane neat the rest state. This implies that the modes of vibration along these axes are 1-1 resonant, which calls for the inclusion of at least 2 modes in the aSSM construction. The arm also exhibits longitudinal vibrations in the zz direction, which constitute another mode of vibration, thus explaining the 3 dominant frequencies observed in the spectrogram analysis. All these minimal sets of modes are underdamped for the SoPrA arm, which is qualitatively different from the overdamped dynamics of tendon-driven trunk helicoid robots (see Kaundinya et al. [22]).

Our next step is to use the collected step control data and inferred dimension d=6d = 6 of the aSSM fibers to learn the aSSM-reduced model for the SoPrA arm. That consists of three learnable functions: the critical manifold parametrization, which maps static control actions to static states; the aSSM parametrization; and the aSSM-reduced dynamics. The critical manifold parametrization is learned from the static states and control actions obtained by postprocessing the step control data in the observable space ss. The aSSM parametrization and dynamics cannot be learned directly in ss, as velocity information is required to model the dynamics, and an ambient space is needed to guarantee the embedding of the dd-dimensional fibers of the aSSM geometry. To address these requirements, we employ delay embedding. Specifically, we construct a delay-embedded observable space by stacking the current state with its two time-delayed copies, in which a 6D fiber of the aSSM geometry can be embedded (see Takens [39]). We then use the collected step control data to learn all the aSSM-related functions in this delay-embedded observable space, following the procedure outlined in § 3.1.

The resulting aSSM-reduced model, which we will term the aSSM model, is of the form

r=VT(a(t)) μ(s−sˉ(a(t))),r˙=R(r,sˉ(a(t))),s=W(r,sˉ(a(t)))+sˉ(a(t)),(4)\begin{aligned} \mathbf{r} &= \mathbf{V}^{\mathsf{T}}(a(t))\,\boldsymbol{\mu}\left(\mathbf{s}-\bar{\mathbf{s}}(a(t))\right),\\ \dot{\mathbf{r}} &= \mathbf{R}\left(\mathbf{r},\bar{\mathbf{s}}(a(t))\right),\\ \mathbf{s} &= \mathbf{W}\left(\mathbf{r},\bar{\mathbf{s}}(a(t))\right)+\bar{\mathbf{s}}(a(t)), \tag*{(4)} \end{aligned}

where r∈R6\mathbf{r} \in\mathbb{R}^{6} is reduced coordinates on the aSSM; V(⋅)\mathbf{V}(\cdot) is the chart map of the aSSM; sˉ(⋅)\bar{\mathbf{s}}(\cdot) is the critical manifold parametrization; μ\boldsymbol{\mu} is a function performing delay embedding; R(⋅)\mathbf{R}(\cdot) is the aSSM reduced dynamics; and W(⋅)\mathbf{W}(\cdot) is the aSSM parametrization. We have found that quadratic polynomial approximations for W(⋅)\mathbf{W}(\cdot), and R(⋅)\mathbf{R}(\cdot), and cubic polynomial approximations for sˉ(⋅)\bar{\mathbf{s}}(\cdot) suffice to model the dynamics of the SoPrA arm accurately.

We plot the aSSM model’s predictions against the SoPrA arm’s true response on previously unseen step control actions in Figure 2b and on a previously unseen periodic control action sequence in Figure 2c. We observe that aSSM predictions qualitatively capture the decay profiles of all observable states. The normalized mean trajectory error achieved by the aSSM model on these test data was 9% for step control data and 3% on the periodic control action data. We note that we do not have exact matches for any of the cases, and we attribute this to a hardware inaccuracy in the pressure actuation system. To illustrate this inaccuracy, in Extended Data Figure 1, we plot the commanded pressure action variable a2a_{2} and compare it with the actual measured pressure in the chamber. We observe that, at all times, the pressure valves keep fluctuating and exhibit lags that lead to minor fluctuations in the arm’s response (see Extended Data Figure 1). This hardware issue is common in pressure control systems that are operated in a more dynamic regime. A mismatch between the commanded and measured pressure will result in accuracy issues for modeling and control in the subsequent sections. We aim to report results despite these limitations and test closed-loop control performance.

Commanded versus measured chamber pressure

(a)

Predictions on an unseen periodic control action

(b)

Extended Data Figure 1. Mismatch between commanded and measured pressure in SoPrA. a, Time series of the control action a2a_{2} for the periodic case, with the pressure commanded by the user (black) and the pressure measured in the chamber (orange). b, The mismatch in a makes the measured response of the SoPrA arm (black) fluctuate about the periodic response that the aSSM model predicts (green).

Predictive power of models on unseen control actions across the full action space

As a comparison, we also learn the dynamics of the SoPrA arm by fitting neural-network-based models to trajectory data. This is done by approximating the state evolution as an autoregressive neural network (b(b-m)m), defined as a map f\mathbf{f},

(sk+1,…,sk+m)=f(sk,sk−1,…,sk−b+1,ak,ak+1,…,ak+m−1),(5)(\mathbf{s}_{k+1},\ldots,\mathbf{s}_{k+m})=\mathbf{f}\left(\mathbf{s}_{k},\mathbf{s}_{k-1},\ldots,\mathbf{s}_{k-b+1},\mathbf{a}_{k},\mathbf{a}_{k+1},\ldots,\mathbf{a}_{k+m-1}\right), \tag*{(5)}

where bb represents the memory window, and mm the forecasting window. The training data used to fit the neural networks is generated by issuing random-walk-like control actions that uniformly explore the state space. Importantly, to ensure a fair comparison, we keep the total amount of time-series training data collected for the neural network consistent with that collected for the aSSM model in § 1.1. The exact training details for all neural network-based models on this dataset are provided in the Supplementary Information. We learn four neural-network-based models: two multilayer perceptrons (MLPs), one with (b(b-m)≡(3m) \equiv(3-$1) and the other with (b(b-m)≡(3m) \equiv(3-$8), a recurrent neural network (RNN), and a Transformer both with (b−m)≡(32−8)(b-m) \equiv(32-8) and trained using state-of-the-art autoregressive training methods from Li et al. [26].

We test the predictive power of all these models, including our aSSM model, on 50 previously unseen continuous random control action sequences, uniformly sampled from the entire control action space. We pass these control action sequences to the SoPrA arm to generate the true responses, and we plot this with the model predictions in the xtip−ytipx_{\mathrm{tip}} - y_{\mathrm{tip}} plane. We present these results in Fig. 3 for four randomly chosen control action sequence runs. We observe that the aSSM model across these sets has the lowest mean error, and its predictions lie close to the true response of the arm.

The RNN and Transformer perform decently, qualitatively tracking the target, but remain inaccurate. In particular, the RNN tracks the true response reasonably well in the xtip>0x_{\mathrm{tip}} > 0 region but loses accuracy in other regions. In contrast, the Transformer uniformly exhibits larger errors throughout the workspace. On the other hand, both vanilla MLPs with simpler autoregressive network structures perform extremely poorly, with the error even diverging on some runs. Increasing the memory window from b=1b = 1 to b=8b = 8 occasionally reduces the error but also suffers from the same instabilities for other random runs. None of the autoregressive models provides a consistent and accurate prediction, which is not surprising, as the SoPrA arm does not exhibit significant memory effects due to material hysteresis or actuation delays in the collected data. We provide a detailed comparison and reasoning for these model behaviors in § 1.4.

Prediction results comparing the aSSM-reduced model against neural-network-based dynamics models

Figure 3. Prediction results comparing the aSSM-reduced model against neural-network-based dynamics models. Plots in the xtip−ytipx_{\mathrm{tip}} - y_{\mathrm{tip}} plane for model predictions on four randomly sampled continuous control action sequences compared against the true response of the SoPrA arm (black). The mean error [mm] of the model prediction is reported in the top-left corner of each plot.

aSSM-based RL policy for the soft arm

We use the learned aSSM model from § 1.1 to learn a reinforcement learning (RL) policy for tracking 2D random target tracks Γ(t)={Γk}k=1T\mathbf{\Gamma}(t) = \{\mathbf{\Gamma}_k\}_{k=1}^{T} in the xtip−ytipx_{\mathrm{tip}} - y_{\mathrm{tip}} plane of the SoPrA arm. The RL policy training proceeds by learning a conditional probability distribution over the space of actions, πθ(a,⋅)\pi_{\theta}(a,\cdot), as a neural network.

A control action ak\mathbf{a}_k at time step kk is sampled from πθ\pi_{\theta} given an observation vector that depends on the current state sk\mathbf{s}_k, the previous state sk−1\mathbf{s}_{k-1}, the previously sampled control action ak−1\mathbf{a}_{k-1}, and the next target state Γk+1\mathbf{\Gamma}_{k+1}. This sampled control action is passed to the aSSM model, which predicts the next state sk+1\mathbf{s}_{k+1}. A reward ρk=ρ(sk+1,ak,Γk+1)\rho_k = \rho(\mathbf{s}_{k+1},\mathbf{a}_k,\mathbf{\Gamma}_{k+1}) is evaluated based on these predictions, the control action that generated it, and the target state. A cumulative reward sum, called the episode return, is computed after rolling out the model and policy across the target track horizon.

Following standard offline model-based RL techniques, we launch and compute episode returns in parallel across a large set of initial conditions sampled from the training data used to fit these models. This procedure is referred to as imagination in the RL community. The weights θ\theta of πθ\pi_{\theta} are updated to maximize the mean episode return over an imagination cycle, and each update is defined as a training iteration of the RL policy. In § 3.3, we provide details on the problem formulation and the RL training algorithm, and in the Supplementary Information, we provide the reward function, the hyperparameters, and the number of initial conditions launched in one imagination cycle.

We perform the same RL training procedure using only the RNN and Transformer models discussed in § 1.2, since these models provided stable predictions. In Extended Data Figure 2, we plot the mean imagination episode return for increasing training iterations to validate if the policy updates indeed converge by maximizing the reward. We plot these for three policies: aSSM-RL, a policy trained on an aSSM model; RNN-RL, a policy trained on an RNN model; and Transformer-RL, a policy trained on a Transformer model. We observe that aSSM-RL smoothly and rapidly reaches a good mean imagination episode return value within 200 iterations, and steadily improves as the training iterations increase. Both the RNN-RL and Transformer-RL only reach a decent mean imagination episode return value after 800 training iterations. Compared to aSSM-RL, training neural network model-based policies often stagnates, and improvements come as short performance gains. They also show erratic behavior as the hyperparameter is swept (shaded region in Extended Data Figure 2), signaling the black-box nature of these models.

Extended Data Figure 2. [Convergence of model-based RL policies trained offline for closed-loop trajectory tracking]. Plots for the evolution of the mean imagination episode return for aSSM-RL (green), RNN-RL (yellow), and Transformer-RL (red) as the policies are trained. The solid lines for each policy depict the average mean imagination episode return across hyperparameter sweeps, and the shaded region shows the deviation.

We perform a total of 5, 000 training iterations, which ensures convergence of all policies. We note that the successful performance of final converged policies on the model does not imply the same on the hardware SoPrA arm. We follow up on this point with a discussion in § 1.4 that systematically compares various features of these models for hardware deployment.

Pareto dominance in model complexity, learning time, and accuracy

We compare here the data-driven models and the RL policies trained on them in terms of model complexity, computation time, and ability to accurately capture hardware dynamics and performance accuracy in closed-loop on hardware.

Our first comparison is a Pareto plot in Figure 4a that focuses only on the performance of the dynamical models. We use the 50 randomized control action experiments from § 1.2 to plot the mean prediction error of the models across these runs along the vertical axis in Figure 4a. The horizontal axis in this figure shows the time taken to learn the various models, with the training dataset size held constant. The number of model parameters is denoted by bubbles, with each bubble’s radius indicating the size of the model. Across all metrics, the aSSM model is Pareto-dominant and defines the Pareto front. As a simple polynomial model (see § 3.1 and eqs. (17)–(18)), the aSSM model requires only 5 minutes to learn, which is orders of magnitude lower than the learning time of the other models. The RNN model lies closest to the Pareto front, where it performs reasonably well in terms of predictive accuracy. However, it is ultimately too complex, requiring significantly longer learning times on the order of hours. The Transformer model is much further from the Pareto front. Despite having fewer parameters than the RNN model, its architecture requires a massive 100 hours of learning time. The instability of MLPs keeps them out of this plot. They are easier to learn because of their moderate sizes, but their erratic accuracy prevents their use for controls. Hence, we omit these models for learning control policies.

Figure 4. [Pareto plot comparisons for learned models and policies]. (a) Mean prediction error of the SoPrA arm’s tip position for the random unseen control action experiments in § 1.2, plotted against model learning time, with bubble size denoting the number of model parameters. The aSSM model is Pareto dominant across these metrics. (b) Mean prediction error of the SoPrA arm’s tip position across the closed-loop runs in § 1.5, plotted against policy training time, with bubble size denoting the number of model parameters. The aSSM-RL policy is Pareto dominant across these metrics.

For the trained model-based RL policies, we offer similar comparisons of the figure, in Figure 4b, but now with the vertical axis denoting mean closed-loop prediction accuracy on hardware across the tasks presented in § 1.5. The horizontal axis of the figure denotes the time taken to train the RL policies. The aSSM-RL is still a Pareto-dominant model. In this case, the RNN-RL is close to the Pareto front but ultimately loses out on closed-loop accuracy, and the RNN model architecture is approximately 500 times more complex than the aSSM model, which adds to the computational burden of training the RL policy only. We note here that Transformer, though considered state-of-the-art, appears to perform the worst across all metrics. Overall, we attribute these trends in neural network models to the limited dataset they can access during training, a practical constraint imposed by hardware.

Closed-loop target tracking results

We finally close the results section, with closed-loop experiments on the hardware SoPrA arm using our trained aSSM-RL policy. We set up two target shapes in the xtip−ytipx_{\mathrm{tip}} - y_{\mathrm{tip}} plane, the lemniscate resembling a Figure-8 pattern, and a square. The lemniscate has a maximum span of 100 [mm] in length and 50 [mm] in width, and the target track moves at a speed of 17.5 [mm/s]. The square’s side length is 100 [mm]. Both of these targets span the maximal regions of the SoPrA arm’s workspace (see Figure 2a). For the square target, we additionally consider three cases varying in speed. The slowest has a speed of 27.5 [mm/s], while the next two are 1.6 and 3.2 times faster.

In Figure 5a, we provide closed-loop results for these target tracks with comparisons against RNN-RL, Transformer-RL, and also aSSM-MPC, a model predictive controller (MPC) similar to the one used in Kaundinya et al. [22], minimizing a quadratic cost to track the target. We quantitatively evaluate the closed-loop prediction error to the target and plot these as bar plots in Figure 5b. We note that our error penalizes spatial track-following. If we also account for strict temporal accuracy, all closed-loop policy prediction errors increase by a factor of 2 [mm] on the lemniscate and the slow square due to hardware latency (see the Supplementary Information). The aSSM-RL method achieves the best performance across various target tracks, with aSSM-MPC following it closely as expected. Both neural-network-based models trace out a track, but as the error increases, they become unstable, causing them to oscillate a bit and leading to larger errors.

We recall that the validity of using the aSSM model for controls relies on the time-scale separation assumption. We can quantitatively verify the assumption on a target track by evaluating a non-dimensional slowness measure rsr_s, defined in eq. (14). The lemniscate and the slow-speed square have rs=0.1r_s = 0.1. The faster squares have an rs=0.3r_s = 0.3 and rs=0.8r_s = 0.8. We expect the theory to hold when rs≪1r_s \ll1, and in practical problems, we have observed that even for rs<1r_s < 1 we obtain good closed-loop performance (see Kaundinya et al. [21] and Kaundinya et al. [22]). This is indeed the case across the target tracks considered here, with performance degrading slightly as the target speed increases. The aSSM-MPC’s limitation is apparent in the fastest target tracks, since it operates at 50 Hz due to the computational burden of solving the problem in real time. The aSSM-RL policy, on the other hand, operates at 100 Hz, which avoids the latency issue of MPC methods.

Figure 5. [Closed-loop experiments using learned model-based RL policies]. (a) Closed-loop results for four target tracks for aSSM-MPC (blue), aSSM-RL (green), RNN-RL (yellow), and Transformer-RL (red). Solid lines show the mean over 10 repeated runs, and shaded regions show the deviation. (b) Average tracking error compared across all target tracks. The deviation bar indicates the performance variation across 10 repeated runs of the closed-loop experiments.

We also provide snapshots of the SoPrA arm during closed-loop operation for all target tracks and across all policies in Figure 6. We note that the aSSM-RL achieves simpler nonlinear shape profiles for the arm, ensuring easy tracking. In contrast, the RNN-RL and transformer-RL assume extreme bending shapes, which prevent continuous, consistent tracking of the target.

Snapshots of the SoPrA arm during closed-loop experiments for all target tracks

Figure 6. [Closed-loop control of SoPrA]. Snapshots of the SoPrA arm during the various closed-loop experiments for all the target tracks. Each snapshot comes from one run, randomly selected from the 10 repeated runs shown in Figure 5a. The traced path by the tip of the arm is plotted as a thick line for each prediction in its respective color.

Closed-loop results with potential hardware upgrades in the pressure-driven actuation system

In § 1.1, in particular in Extended Data Figure 1, we highlighted the hardware limitation that the controller-commanded pressure does not exactly match the measured pressure in the chambers. We expected this mismatch to cause accuracy issues during closed-loop experiments on hardware. Indeed, we observe that no policy prediction is spared, in Figure 5a, they appear as minor wiggles along the entire closed-loop prediction.

Upgrading to more expensive pressure valves is expected to improve accuracy by reducing the mismatch between commanded and measured pressure caused by the dynamic nature of the tasks. It will also allow the practitioner to test their methods on tasks involving faster target tracks. To aid the practitioner in deciding on the upgrade, from the mismatch data in Extended Data Figure 1, we evaluated numerically that a moving-average procedure with a window of 0.3 seconds (the current time lag in the pressure valve between commanded and measured) removes most of the wiggles, thus mimicking closed-loop predictions when the mismatch is absent. We applied this filtering procedure to our policy prediction from § 1.5 and report these results in Extended Data Fig. 3a, and also provide revised mean error bar plots in Extended Data Fig. 3b.

The results show promise that upgrading to better valves guarantees improved performance across all target tracks. Only large oscillations intrinsic to the SoPrA arm are visible, which the aSSM model accurately models and limits compared to the RNN and Transformer models. The error bars also point to an interesting trend. Our heuristic numerical filtering procedure reduces errors in the RNN-RL and Transformer-RL predictions to a greater extent than in the aSSM model, indicating that, despite these pressure mismatch issues, our aSSM-RL method explores the best possible policy to reduce the existing hardware mismatch error.

Discussion

We have demonstrated that hardware soft robots can be accurately controlled for dynamic target tracking via Reinforcement Learning (RL) control policies trained solely on a faithful, reduced dynamics model of the soft robot. This advance is made possible by the fact that the dominant dynamics of soft robots lie on a very low-dimensional adiabatic spectral submanifold (aSSM). Recent methods allow for the rapid extraction of reduced models based on aSSM theory from hardware data alone.

For a pneumatically actuated SoPrA soft robotic arm, we have learned a 6D aSSM model from only partial observational data recorded by cameras (see § 1.1). The learned 6D aSSM model delivers strong predictive capability and generalizes to time-varying control actions not present in the training dataset (see § 1.2). When compared with state-of-the-art autoregressive neural-network-based models, our aSSM model is Pareto dominant across prediction accuracy and model learning time, offering the fastest model to learn and the simplest with parameters in the thousands compared to the others in the millions (see § 1.4). This simplicity also allows the aSSM model to be used in classical model predictive control schemes for real-time control, which are otherwise intractable to use with large models (see aSSM-MPC results in § 1.5).

RL control policies trained on our aSSM model have shown stable performance across a wide range of RL hyperparameters and converge rapidly in a couple of hours, unlike policies trained on autoregressive neural-network-based models such as the RNN and the Transformer (see § 1.3). These RL policies were trained without interacting with the hardware soft robot and only explore the control action space using models learned on two hours of hardware data. By machine-learning standards, this is a small dataset, so one would expect the resulting RL policies to transfer poorly to the real world.

Our closed-loop results in § 1.5 support this expectation: the RNN-RL and Transformer-RL policies produce large errors when tracking targets in real time. In contrast, the aSSM-RL policy provides consistent, accurate closed-loop performance. This is because, the aSSM provides a faithful reduced-order representation of the robot’s dynamics and therefore requires only data capturing the dominant behaviors that are ubiquitous and thus do not demand large volumes of data, unlike black-box approaches. Our discovery of reliable, data-efficient aSSM-RL policies for soft robots opens exciting avenues for applying these techniques to challenging, application-oriented problems in interactive environments, such as surgical assistance or underwater exploration. Learning aSSM-RL policies for soft robotic hands performing dynamic dexterous manipulation is already within reach and currently under exploration. A longer-term goal is to learn aSSM models from video alone, without physical markers. Policies designed using such models will allow us to control complex soft robot morphologies and operate them in unstructured environments.

Methods

Adiabatic Spectral Submanifolds: Theory and Data-Driven Methodology

Theory

The governing equations of soft robots can be written abstractly as a first-order dynamical system

x˙=F(x,a(t)),x∈X⊆Rn,a∈C⊆Rna,(6)\dot{x} = \mathbf{F}(x,a(t)), \qquad x \in\mathcal{X} \subseteq\mathbb{R}^{n}, \qquad a \in\mathcal{C} \subseteq\mathbb{R}^{n_a}, \tag*{(6)}

with nn typically very large due to the continuum nature of the problem. Here xx is the full phase space of the robot, aa is the control action drawn from the admissible set C\mathcal{C}, and F\mathbf{F} is a smooth nonlinear function that captures the geometric and material deformations of the arm.

For a static control input aˉ∈C\bar{a} \in\mathcal{C}, the robot deforms into a static configuration. Assuming that the collection of all these static configurations are attracting across C\mathcal{C}, they form a normally attracting critical manifold (see Fenichel [14]),

L0={(x,aˉ)∈X×C:x=xˉ(aˉ)}.(7)\mathcal{L}_0 = \left\{(x,\bar{a}) \in\mathcal{X} \times\mathcal{C} : x = \bar{x}(\bar{a})\right\}. \tag*{(7)}

If we further assume the linearization about each static configuration has a strict spectral splitting after dd slowest eigenvalues, uniformly over C\mathcal{C},

Re⁡(λn(aˉ))≤⋯≤Re⁡(λd+1(aˉ))<Re⁡(λd(aˉ))≤⋯≤Re⁡(λ1(aˉ))<0,(8)\operatorname{Re}(\lambda_n(\bar{a})) \le\cdots\le\operatorname{Re}(\lambda_{d+1}(\bar{a})) < \operatorname{Re}(\lambda_d(\bar{a})) \le\cdots\le\operatorname{Re}(\lambda_1(\bar{a})) < 0, \tag*{(8)}

we can define the slow spectral subbundle E(aˉ)\mathbf{E}(\bar{a}) associated with the eigenspaces of the dd slowest eigenvalues. Under further non-resonance conditions, this subbundle of the linearized problem survives as a slow spectral submanifold bundle

A0=W(E(aˉ)),(9)\mathcal{A}_0 = \mathcal{W}\bigl(\mathbf{E}(\bar{a})\bigr), \tag*{(9)}

which is also normally attracting and anchored to the critical manifold L0\mathcal{L}_0 (see Haller and Kaundinya [16]). The slow spectral submanifold bundle A0\mathcal{A}_0 captures the dominant dynamics of system eq. (6) with static control actions.

Haller and Kaundinya [16] and Kaundinya et al. [21] show that both L0\mathcal{L}_0 and A0\mathcal{A}_0 survive into nearby diffeomorphic invariant manifolds, Lϵ\mathcal{L}_{\epsilon} the slow manifold and Aϵ\mathcal{A}_{\epsilon} the adiabatic spectral submanifold (aSSM) when the control action is of the form

a(t)=as(ϵt)+ad(t),(10)a(t) = a^{s}(\epsilon t) + a^{d}(t), \tag*{(10)}

with ∥ad(t)∥<Δ\lVert a^{d}(t) \rVert< \Delta, and 0<ϵ≪10 < \epsilon\ll1. General control actions in control operations admit this splitting, in which as(t)a^{s}(t) is relatively slowly varying but can be large compared to a small, uniformly bounded deviation control action ad(t)a^{d}(t). We refer to Haller and Kaundinya [16] for details of the aSSM-reduced model derived from invariance equations and to Kaundinya et al. [21] for details in the control setting.

On hardware, the full phase space x\mathbf{x} is only partially observable: cameras record only a sparse set of positional data s∈S⊆Rns\mathbf{s} \in\mathcal{S} \subseteq\mathbb{R}^{n_s} with ns≪nxn_s \ll n_x. In that case, the critical manifold definition carries over to the observable state space as sˉ(aˉ)\bar{\mathbf{s}}(\bar{\mathbf{a}}). The state s\mathbf{s} has no velocity information and hence does not guarantee embedding the dynamics of the dd-dimensional fibers of the frozen aSSM geometry A0\mathcal{A}_0. To successfully embed the aSSM and its dynamics in an ambient space, by Takens’ theorem (see Takens [39]), we need to delay embed the positional observable s\mathbf{s}, at least p≥⌈2d+1ny⌉p \ge\left\lceil\frac{2d+1}{n_y} \right\rceil times.

In this delay-embedded observable space, assuming the control action satisfies eq. (10), we denote V(a(t))∈R(p+1)ns×d\mathbf{V}(\mathbf{a}(t)) \in\mathbb{R}^{(p+1)n_s \times d} whose columns span the time-varying slow spectral subspace E(a(t))\mathbf{E}(\mathbf{a}(t)) along the trajectory μ(sˉ(a(t)))\boldsymbol{\mu}(\bar{\mathbf{s}}(\mathbf{a}(t))), and r∈Rd\mathbf{r} \in\mathbb{R}^{d} the reduced coordinate along this subbundle, the leading-order aSSM-reduced model takes the form

r=VT(a(t))μ(s−sˉ(a(t))),r˙=R(r,sˉ(a(t))),s=W(r,sˉ(a(t)))+sˉ(a(t)),(11)\begin{aligned} \mathbf{r} &= \mathbf{V}^{\mathsf{T}}(\mathbf{a}(t))\boldsymbol{\mu}\left(\mathbf{s}-\bar{\mathbf{s}}(\mathbf{a}(t))\right),\\ \dot{\mathbf{r}} &= \mathbf{R}\left(\mathbf{r},\bar{\mathbf{s}}(\mathbf{a}(t))\right),\\ \mathbf{s} &= \mathbf{W}\left(\mathbf{r},\bar{\mathbf{s}}(\mathbf{a}(t))\right)+\bar{\mathbf{s}}(\mathbf{a}(t)), \tag*{(11)} \end{aligned}

where W(⋅)\mathbf{W}(\cdot) is the aSSM parametrization and R(⋅)\mathbf{R}(\cdot) is the aSSM reduced dynamics.

Remark 1. [Discrete-time aSSM-reduced model for reinforcement learning (RL)] Most model-based RL algorithms require a discrete dynamics model. We obtain the discrete reduced model by integrating the continuous leading-order dynamics (11) over a time step Δt\Delta t under a zero-order hold, i.e., the action is held fixed at ak\mathbf{a}_k over [tk,tk+Δt)[t_k,t_k+\Delta t). This results in a discrete map of the aSSM-reduced model,

rk=VT(ak)μ(sk−sˉ(ak)),rk+1=Rd(R;rk,sˉ(ak),Δt),sk+1=W(rk+1,sˉ(ak))+sˉ(ak),(12)\begin{aligned} \mathbf{r}_k &= \mathbf{V}^{\mathsf{T}}(\mathbf{a}_k)\boldsymbol{\mu}\left(\mathbf{s}_k-\bar{\mathbf{s}}(\mathbf{a}_k)\right),\\ \mathbf{r}_{k+1} &= \mathbf{R}_d\left(\mathbf{R};\mathbf{r}_k,\bar{\mathbf{s}}(\mathbf{a}_k),\Delta t\right),\\ \mathbf{s}_{k+1} &= \mathbf{W}\left(\mathbf{r}_{k+1},\bar{\mathbf{s}}(\mathbf{a}_k)\right)+\bar{\mathbf{s}}(\mathbf{a}_k), \tag*{(12)} \end{aligned}

where Rd(⋅)\mathbf{R}_d(\cdot) is the discrete reduced dynamics map approximated via numerical methods from the continuous aSSM reduced dynamics R(⋅)\mathbf{R}(\cdot).

Remark 2. [Higher-order correction to the aSSM-reduced model] Kaundinya et al. [22] provides higher-order corrections to the aSSM-reduced dynamics, which depend on the control action rate. Based on the assumptions of the control action splitting in eq. (10), the higher-order corrected aSSM reduced model is of the form

r˙=R(r,sˉ(a(t)))+B(a(t))Dasˉ(a(t))a˙+O((ϵ+Δ)2,ϵ∣r∣,Δ∣r∣),(13)\dot{\mathbf{r}}=\mathbf{R}\left(\mathbf{r},\bar{\mathbf{s}}(\mathbf{a}(t))\right)+\mathbf{B}(\mathbf{a}(t))\mathbf{D}_{\mathbf{a}}\bar{\mathbf{s}}(\mathbf{a}(t))\dot{\mathbf{a}}+O\left((\epsilon+\Delta)^2,\epsilon|\mathbf{r}|,\Delta|\mathbf{r}|\right), \tag*{(13)}

with B(a(t))\mathbf{B}(\mathbf{a}(t)) a calibration matrix learned from the leading-order aSSM-reduced model. We ignore this correction here, since for our pneumatically actuated arm, the inconsistencies between the commanded and measured pressures in the actuation system make it impossible to learn an accurate calibration matrix.

Remark 3. [Time-scale separation for target tracking] Our aSSM-reduced models are valid only under the time-scale separation assumption stated in § 1.1. We quantify this with the non-dimensional slowness measure rsr_s for practical application,

rs=1tf−ti∫titf∥r˙(τ)∥ dτ1td∫0td∥s˙decay(τ)∥ dτ,(14)r_s=\frac{\frac{1}{t_f-t_i}\int_{t_i}^{t_f}\|\dot{\mathbf{r}}(\tau)\|\,\mathrm{d}\tau}{\frac{1}{t_d}\int_{0}^{t_d}\|\dot{\mathbf{s}}_{\mathrm{decay}}(\tau)\|\,\mathrm{d}\tau}, \tag*{(14)}

where sdecay\mathbf{s}_{\mathrm{decay}} is a decaying trajectory of the robot, tdt_d is the duration of that decay, and the overbar denotes an average across the various decays obtained by passing step

control actions spanning CC. The numerator is the mean speed along the target Γ(t)\Gamma(t) over its horizon [ti,tf][t_i,t_f]. The theory is guaranteed for rs≪1r_s \ll1, but our method performs well in practice up to rs≈1r_s \approx1 (see Kaundinya et al. [21] and Kaundinya et al. [22]).

Data-driven methodology

We collect the robot’s response to step control actions in the observable space. The step control actions are generated by performing jumps, each held for tdecayt_{\mathrm{decay}} seconds, between the mm most distant control actions for every element of the static action set, which we sample by Latin hypercube sampling (see McKay et al. [28]). We set the total number of jumps, m=4m=4, for the step data collection for the SoPrA arm. Passing these control actions gives static pairs {sstatici,astatici}i=1N\{\mathbf{s}_{\mathrm{static}}^i,\mathbf{a}_{\mathrm{static}}^i\}_{i=1}^{N} and their accompanying decay trajectories {sdecayi,adecayi}i=1mN\{\mathbf{s}_{\mathrm{decay}}^i,\mathbf{a}_{\mathrm{decay}}^i\}_{i=1}^{mN}.

  • Step 1. [Learning the critical manifold]. We learn sˉ(a)\bar{\mathbf{s}}(\mathbf{a}) as a nonlinear static map by regressing the static configurations onto their control actions,

sˉ⋆=arg⁡min⁡sˉ∑i=1N∥sstatici−sˉ(astatici)∥2.(15)\bar{\mathbf{s}}^\star= \underset{\bar{\mathbf{s}}}{\arg\min}\sum_{i=1}^{N}\left\lVert\mathbf{s}_{\mathrm{static}}^i-\bar{\mathbf{s}}\left(\mathbf{a}_{\mathrm{static}}^i\right)\right\rVert^2 . \tag*{(15)}

We seek the critical manifold as a polynomial expansion of order kak_a,

sˉ(a)=∑0≤∣α∣≤kacαaα,(16)\bar{\mathbf{s}}(\mathbf{a})=\sum_{0\leq\lvert\alpha\rvert\leq k_a}\mathbf{c}_{\alpha}\mathbf{a}^{\alpha}, \tag*{(16)}

where α\alpha is a multi-index, and cα\mathbf{c}_{\alpha} are the polynomial coefficients. The expansion is linear in cα\mathbf{c}_{\alpha}, so the regression reduces to a linear least-squares problem with a closed-form solution. We can also use radial-basis and neural-network approximations for fitting sˉ(a)\bar{\mathbf{s}}(\mathbf{a}) (see Kaundinya et al. [22] for details).

  • Step 2. [Learning the aSSM parametrization and reduced dynamics]. We center each decay about its static configuration to obtain centered, {scenteredi,adecayi}i=1mN\{\mathbf{s}_{\mathrm{centered}}^i,\mathbf{a}_{\mathrm{decay}}^i\}_{i=1}^{mN}. We later delay embed this data pp times through the map μ\mu. The delays are chosen such that the condition (p+1)ns≥2d+1(p+1)n_s\geq2d+1 is satisfied. We collect the mm decays that share a static configuration, and perform an economy singular value decomposition to obtain chart maps {Vi}i=1N\{\mathbf{V}^i\}_{i=1}^{N}, which we interpolate over astatic\mathbf{a}_{\mathrm{static}} or, when the local charts nearly coincide, use a single global chart. The data in the reduced coordinates is then obtained by Ri=(Vi)T[μ(scenteredi1),…,μ(scenteredim)]\mathcal{R}^i=(\mathbf{V}^i)^{\mathsf{T}}[\mu(\mathbf{s}_{\mathrm{centered}}^{i1}),\ldots,\mu(\mathbf{s}_{\mathrm{centered}}^{im})] over the mm decays sharing the iith static configuration, and finite differencing this results in the velocity data R˙i\dot{\mathcal{R}}^i.

We seek the leading-order aSSM parametrization W(⋅)\mathbf{W}(\cdot) and reduced dynamics R(⋅)\mathbf{R}(\cdot) as a polynomial expansion of the form,

W(r,s)=∑1≤∣α∣≤kWr∑0≤∣β∣≤kWaWαβrαsβ,(17)\mathbf{W}(\mathbf{r},\mathbf{s})=\sum_{1\leq\lvert\alpha\rvert\leq k_W^r}\sum_{0\leq\lvert\beta\rvert\leq k_W^a}\mathbf{W}_{\alpha\beta}\mathbf{r}^{\alpha}\mathbf{s}^{\beta}, \tag*{(17)}
R(r,s)=∑1≤∣α∣≤kRr∑0≤∣β∣≤kRaRαβrαsβ,(18)\mathbf{R}(\mathbf{r},\mathbf{s})=\sum_{1\leq\lvert\alpha\rvert\leq k_R^r}\sum_{0\leq\lvert\beta\rvert\leq k_R^a}\mathbf{R}_{\alpha\beta}\mathbf{r}^{\alpha}\mathbf{s}^{\beta}, \tag*{(18)}

where α\alpha and β\beta are multi-indices over r\mathbf{r} and s\mathbf{s} respectively, kWrk_W^r, kWak_W^a and kRrk_R^r, kRak_R^a denote the corresponding polynomial orders, and Wαβ\mathbf{W}_{\alpha\beta}, Rαβ\mathbf{R}_{\alpha\beta} are the unknown coefficients. We fit these by minimizing the reconstruction and dynamics regression losses on the centered decay data,

W⋆=arg⁡min⁡W∑i∥scenteredi−W(Ri,sstatici)∥2,(19)\mathbf{W}^{\star}=\underset{\mathbf{W}}{\arg\min}\sum_i\left\lVert\mathbf{s}_{\mathrm{centered}}^i-\mathbf{W}\left(\mathcal{R}^i,\mathbf{s}_{\mathrm{static}}^i\right)\right\rVert^2 , \tag*{(19)}
R⋆=arg⁡min⁡R ∑i∥R^i−R(Ri,sstatici)∥2,\mathbf{R}^{\star} = \underset{\mathbf{R}}{\arg\min}\ \sum_i \left\lVert\widehat{\mathbf{R}}^{i} - \mathbf{R}(\mathbf{R}^{i},s_{\mathrm{static}}^{i}) \right\rVert^2,

where Scenteredi=[scenteredi1,…,scenteredim]S_{\mathrm{centered}}^{i} = [s_{\mathrm{centered}}^{i1},\ldots,s_{\mathrm{centered}}^{im}]. All parameters concerning the aSSM-reduced model are chosen to minimize the normalized mean trajectory error on unseen test decay data.

Offline model-based reinforcement learning

We learn a policy entirely from the learned dynamics model. The dynamics model is identified offline from the dataset D\mathcal{D} and is held fixed thereafter. No interaction with the hardware takes place during policy optimization. The policy πθ\pi_{\theta} acts on the observation

ok={sk,sk−1,ak−1,Γk+1},(20)o_k = \{s_k, s_{k-1}, a_{k-1}, \Gamma_{k+1}\}, \tag*{(20)}

where sks_k is the state obtained from passing control action ak−1a_{k-1} to the dynamics model and Γk+1\Gamma_{k+1} the target state to be reached. The control action needed to reach the target state Γk+1\Gamma_{k+1} is sampled as,

ak∼πθ(⋅∣ok).(21)a_k \sim\pi_{\theta}(\cdot\mid o_k). \tag*{(21)}

We note, that for obtaining oko_k requires sks_k that needs to evaluated using the aSSM model in eq. (12) or the autoregressive model eq. (5). The aSSM model requires the control action at k−1k-1, whereas the autoregressive model requires past control actions based on the model’s memory window. The aSSM model needs past states only to set an initial condition, but the dynamical rollouts are history-independent. The autoregressive model also needs a history of states to perform the rollout.

Algorithm 1 RL policy optimization by imagination on the learned dynamics model

Require: Dynamics model either an aSSM model eq. (12) or a neural-network-based model eq. (5) identified offline from D\mathcal{D} and held fixed

Require: Dataset D\mathcal{D}, target distribution T\mathcal{T}, horizon TT, number of parallel imagined rollouts NN

  1. Initialize policy πθ\pi_{\theta}

  2. for learning iterations =1,2,…=1,2,\ldots do

  3. Imagination: sample NN initial conditions and target tracks, then roll out in parallel

  4. for i=1,…,Ni=1,\ldots,N in parallel do

  5. Sample warm-start history and initial actions from D\mathcal{D}, giving the initial condition of rollout ii

  6. Sample a target track {Γk(i)}k=1T\{\Gamma_k^{(i)}\}_{k=1}^{T} from T\mathcal{T}

  7. end for

  8. for k=0,1,…,T−1k=0,1,\ldots,T-1 do

  9. for i=1,…,Ni=1,\ldots,N in parallel do

  10. Assemble ok(i)o_k^{(i)} by Eq. (20)

  11. Sample ak(i)∼πθ(⋅∣ok(i))a_k^{(i)} \sim\pi_{\theta}(\cdot\mid o_k^{(i)})

  12. Advance one step to sk+1(i)s_{k+1}^{(i)} using the dynamics model

  13. Evaluate reward ρk(i)\rho_k^{(i)} by Eq. (22)

  14. end for

  15. end for

  16. Update πθ\pi_{\theta} on the batch of NN imagined rollouts using PPO

  17. end for

The target horizon length is TT steps. Thus, the model must perform TT rollouts to produce a prediction based on sampled control-action sequences. At each step of the rollout, we evaluate the reward, defined by

ρk=ρ(sk+1,ak,Γk+1).(22)\rho_k = \rho(\mathbf{s}_{k+1}, \mathbf{a}_k, \boldsymbol{\Gamma}_{k+1}). \tag*{(22)}

We provide Algorithm 1 for training a policy that maximizes the above reward along the target track across different initial conditions.

Acknowledgements

This work has been partially supported by grant 200021_214908 from the Swiss National Science Foundation (SNF).

Competing Interests

All authors declare no financial or non-financial competing interests.

Extended Data

Figure 3 (PDF p. 25). (a) Closed-loop results for four target tracks for aSSM-MPC (blue), aSSM-RL (green), RNN-RL (yellow), and Transformer-RL (red), with numerical filtering applied to the results to demonstrate the improvement expected when we use upgraded pressure valves. (b) Revised average tracking error compared across all the target tracks when commanded-to-measure pressure mismatch is accounted for numerically. The percentage number on top of each bar shows the extent of error removed by the numerical filter.

References

  1. [1]J. I. Alora, M. Cenedese, G. Haller, and M. Pavone. Discovering dominant dynamics for nonlinear continuum robot control. npj Robotics, 3(1):5, 2025. doi: 10.1038/s44182-025-00021-8. URL https://doi.org/10.1038/s44182-025-00021-8.
  2. [2]J. Axås, M. Cenedese, and G. Haller. Fast data-driven model reduction for nonlinear dynamical systems. Nonlinear Dyn., 111(9):7941–7957, 2023.
  3. [3]U. Berdica, M. Jackson, N. E. Veronese, J. Foerster, and P. Majolino. Reinforcement learning controllers for soft robots using learned environments. In Proc. IEEE Int. Conf. Soft Robot. (RoboSoft), pages 933–939, 2024. doi: 10.1109/RoboSoft60065.2024.10522003.DOI
  4. [4]L. Bettini, B. Kaszás, B. Zybach, J. Dual, and G. Haller. Data-driven nonlinear model reduction to spectral submanifolds via oblique projection. Chaos, 35(4):043135, Apr. 2025. doi: 10.1063/5.0243849. URL https://doi.org/10.1063/5.0243849.
  5. [5]L. Bettini, A. Kazemipour, R. K. Katzschmann, and G. Haller. General oblique projections for model reduction via spectral submanifolds. Chaos, 36(5):053122, 2026.DOI
  6. [6]L. Bettini, A. Kazemipour, R. K. Katzschmann, and G. Haller. Nonlinear spectral modeling and control of soft-robotic muscles from data, 2026. URL https://arxiv.org/abs/2601.03247. arXiv:2601.03247.
  7. [7]D. Braganza, D. M. Dawson, I. D. Walker, and N. Nath. A neural network controller for continuum robots. IEEE Trans. Robot., 23(6):1270–1277, 2007. doi: 10.1109/TRO.2007.906248.DOI
  8. [8]G. Buza and G. Haller. Spectral submanifold reduction for PDEs describing nonlinear continuum vibrations, 2026. URL https://arxiv.org/abs/2607.10675. arXiv:2607.10675.
  9. [9]M. Cenedese, J. Axås, and G. Haller. SSMLearn. https://github.com/haller-group/SSMLearn, 2021. GitHub repository.
  10. [10]M. Cenedese, J. Axås, B. Bäuerlein, K. Avila, and G. Haller. Data-driven modeling and prediction of non-linearizable dynamics via spectral submanifolds. Nat. Commun., 13(1):872, 2022.DOI
  11. [11]M. Cenedese, J. Axås, H. Yang, M. Eriten, and G. Haller. Data-driven nonlinear model reduction to spectral submanifolds in mechanical systems. Philos. Trans. R. Soc. A, 380:20210194, 2022.DOI
  12. [12]A. Centurelli, L. Arleo, A. Rizzo, S. Tolu, C. Laschi, and E. Falotico. Closed-loop dynamic control of a soft manipulator using deep reinforcement learning. IEEE Robot. Autom. Lett., 7(2):4741–4748, 2022. doi: 10.1109/LRA.2022.3146903.DOI
  13. [13]F. Faure, C. Duriez, H. Delingette, J. Allard, B. Gilles, S. Marchesseau, H. Talbot, H. Courtecuisse, G. Bousquet, I. Peterlik, et al. Sofa: A multi-model framework for interactive physical simulation. Soft tissue biomechanical modeling for computer assisted surgery, pages 283–321, 2012.DOI
  14. [14]N. Fenichel. Geometric singular perturbation theory for ordinary differential equations. J. Differ. Equ., 31(1):53–98, 1979.DOI
  15. [15]G. Haller. Modeling Nonlinear Dynamics from Equations and Data — with Applications to Solids, Fluids, and Controls. SIAM, Philadelphia, PA, 2025.DOI
  16. [16]G. Haller and R. S. Kaundinya. Nonlinear model reduction to temporally aperiodic spectral submanifolds. Chaos, 34(4):043152, Apr. 2024. ISSN 1054-1500. doi: 10.1063/5.0187080. URL https://doi.org/10.1063/5.0187080.
  17. [17]G. Haller and S. Ponsioen. Nonlinear normal modes and spectral submanifolds: existence, uniqueness and use in model reduction. Nonlinear Dyn., 86(3):1493–1534, 2016.arxiv.org/abs/1602.00560
  18. [18]G. Haller, B. Kaszás, A. Liu, and J. Axås. Nonlinear model reduction to fractional and mixed-mode spectral submanifolds. Chaos, 33(6):063138, 2023.DOI
  19. [19]J. Ibarz, J. Tan, C. Finn, M. Kalakrishnan, P. Pastor, and S. Levine. How to train your robot with deep reinforcement learning: lessons we have learned. Int. J. Robot. Res., 40(4-5):698–721, 2021. doi: 10.1177/0278364920987859. URL https://doi.org/10.1177/0278364920987859.
  20. [20]M. Janner, J. Fu, M. Zhang, and S. Levine. When to trust your model: model-based policy optimization. In Adv. Neural Inf. Process. Syst. (NeurIPS), Red Hook, NY, 2019. Curran Associates, Inc.arxiv.org/abs/1906.08253
  21. [21]R. S. Kaundinya, J. I. Alora, J. G. Matt, L. A. Pabon, M. Pavone, and G. Haller. Data-driven soft robot control via adiabatic spectral submanifolds. Int. J. Robot. Res., 2026. doi: 10.1177/02783649261461630.DOI
  22. [22]R. S. Kaundinya, M. Leone, R. Sabella, H. Buurmeijer, L. Pabon, J. I. Alora, M. Pavone, and G. Haller. Data-Driven Precision Control of Soft Helicoid Robots. Preprint, 2026.arxiv.org/abs/2611.11709
  23. [23]R. S. Kaundinya, I. Thiel, B. Kaszás, S. Jain, and G. Haller. Predicting generalized steady states in aperiodically forced mechanical systems. J. Sound Vib., 643:120026, 2026. ISSN 0022-460X. doi: https://doi.org/10.1016/j.jsv.2026.120026. URL https://www.sciencedirect.com/science/article/pii/S0022460X26003871.
  24. [24]J. Kober, J. A. Bagnell, and J. Peters. Reinforcement learning in robotics: A survey. Int. J. Robot. Res., 32(11):1283–1274, 2013. doi: 10.1177/0278364913495721. URL https://doi.org/10.1177/0278364913495721.
  25. [25]O. Kroemer, S. Niekum, and G. Konidaris. A review of robot learning for manipulation: Challenges, representations, and algorithms. J. Mach. Learn. Res., 22(30):1–82, 2021. URL http://jmlr.org/papers/v22/19-804.html.
  26. [26]C. Li, A. Krause, and M. Hutter. Robotic world model: A neural network simulator for robust policy optimization in robotics. In Adv. Neural Inf. Process. Syst. (NeurIPS), 2025. URL https://arxiv.org/abs/2501.10100.
  27. [27]V. Makoviychuk, L. Wawrzyniak, Y. Guo, M. Lu, K. Storey, M. Macklin, D. Hoeller, N. Rudin, A. Allshire, A. Handa, and G. State. Isaac gym: High performance GPU based physics simulation for robot learning. In Adv. Neural Inf. Process. Syst. (NeurIPS), Datasets and Benchmarks Track, 2021. URL https://openreview.net/forum?id=fgFBtYgJQX-.arxiv.org/abs/2108.10470
  28. [28]M. D. McKay, R. J. Beckman, and W. J. Conover. A comparison of three methods for selecting values of input variables in the analysis of output from a computer code. Technometrics, 21(2):239–245, 1979. ISSN 00401706.
  29. [29]M. Mekkattu, M. Y. Michelis, and R. K. Katzschmann. Sors: A modular, high-fidelity simulator for soft robots. In Proc. IEEE Int. Conf. Soft Robot. (RoboSoft), pages 223–230, 2026. doi: 10.1109/RoboSoft67810.2026.11522869.DOI
  30. [30]T. Miki, J. Lee, J. Hwangbo, L. Wellhausen, V. Koltun, and M. Hutter. Learning robust perceptive locomotion for quadrupedal robots in the wild. Sci. Robot., 7(62):eabk2822, 2022. doi: 10.1126/scirobotics.abk2822. URL https://www.science.org/doi/abs/10.1126/scirobotics.abk2822.DOI
  31. [31]A. A. Morsy, Z. Xu, P. Tiso, and G. Haller. Data-driven reduction of the finite-element model of a Tribomechadynamics benchmark problem. Nonlinear Dyn., 113(18):24539–24555, 2025. doi: 10.1007/s11071-025-11378-8.DOI
  32. [32]F. Nan, H. Ma, Q. Guan, J. Hughes, M. Muehlebach, and M. Hutter. Efficient model-based reinforcement learning for robot control via online optimization. Int. J. Robot. Res., 0(0):02783649261457560, 2026. doi: 10.1177/02783649261457560. URL https://doi.org/10.1177/02783649261457560.
  33. [33]N. Naughton, J. Sun, A. Tekinalp, T. Parthasarathy, G. Chowdhary, and M. Gazzola. Elastica: A compliant mechanics environment for soft robotic control. IEEE Robot. Autom. Lett., 6(2):3389–3396, 2021. doi: 10.1109/LRA.2021.3063698.DOI
  34. [34]M. S. Nazeer, C. Laschi, and E. Falotico. Rl-based adaptive controller for high precision reaching in a soft robot arm. IEEE Trans. Robot., 40:2498–2512, 2024. doi: 10.1109/TRO.2024.3381558.DOI
  35. [35]T. Parthasarathy, S. H. Kim, S. Cui, and M. Gazzola. Elastica++: A high-performance, multiphysics framework for large interacting assemblies of cosserat rods, 2026. URL https://arxiv.org/abs/2605.13766. arXiv:2605.13766.
  36. [36]S. Satheeshbabu, N. K. Uppalapati, T. Fu, and G. Krishnan. Continuous control of a soft continuum arm using deep reinforcement learning. In Proc. IEEE Int. Conf. Soft Robot. (RoboSoft), pages 497–503, 2020. doi: 10.1109/RoboSoft48309.2020.9116003.DOI
  37. [37]J. Schulman, F. Wolksi, P. Dhariwal, A. Radford, and O. Klimov. Proximal policy optimization algorithms, 2017. URL https://arxiv.org/abs/1707.06347. arXiv:1707.06347.
  38. [38]R. S. Sutton and A. G. Barto. Reinforcement learning: An introduction. MIT Press, Cambridge, MA, 2 edition, 2018.
  39. [39]F. Takens. Detecting strange attractors in turbulence. In D. Rand and L. Young, editors, Dynamical Systems and Turbulence, Warwick 1980, pages 366–381. Springer, Berlin, Heidelberg, 1981.DOI
  40. [40]T. G. Thuruthlel, E. Falotico, F. Renda, and C. Laschi. Model-based reinforcement learning for closed-loop dynamic control of soft robotic manipulators. IEEE Trans. Robot., 35:124–134, 2019. URL https://api.semanticscholar.org/Corpus ID:59618928.
  41. [41]Y. Toshimitsu, K. W. Wong, T. Buchner, and R. Katzschmann. Sopra: Fabrication & dynamic modeling of a scalable soft continuum robotic arm with integrated proprioceptive sensing. In Proc. IEEE/RSJ Int. Conf. Intell. Robots Syst. (IROS), pages 653–660, 2021. doi: 10.1109/IROS51168.2021.9636539.DOI
  42. [42]Y. Toshimitsu, B. Forrai, B. G. Cangan, U. Steger, M. Knecht, S. Weirich, and R. K. Katzschmann. Getting the ball rolling: Learning a dexterous policy for a biomimetic tendon-driven hand with rolling contact joints. In Proc. IEEE-RAS Int. Conf. Humanoid Robots (Humanoids), pages 1–7, 2023. doi: 10.1109/Humanoids57100.2023.10375231.DOI
  43. [43]X. Zhang, C. Li, H. Mo, Y. Jiang, W. Cao, X. Xu, W. Jiang, Z. Bing, Y. Yang, X. Li, Y. Yang, H. Lu, L.-l. Zeng, A. Knoll, D. Hu, L. Wen, and W. Pan. Reinforcement learning in linear embedding space unlocks generalizable control across soft robot configurations. Nat. Commun., 17(1):7327, 2026. doi: 10.1038/s41467-026-72491-9. URL https://doi.org/10.1038/s41467-026-72491-9.
  44. [44]H. Zheng, B. Sukhija, C. Li, K. Iten, A. Krause, and R. K. Katzschmann. Learning soft robotic dynamics with active exploration, 2025. URL https://arxiv.org/abs/2510.27428. arXiv:2510.27428.

Paper details

vlnc:W_3zp4k4nh
Version
v1 · latest
Published
Fresh — added to the Hub within the last 30 days
Added
License
CC BY 4.0
Usage

views · 0

downloads · 0

A view counts once per browser per day, a download each time the PDF is requested.

  • Computer Science

Not recorded for this paper

View statement
Looking for related papers…
Contents