WorksComputer Science
Recurrent Temporal Restricted Boltzmann Machines as Unsupervised Encoders for Biomechanical Movement Sequences
Abstract
Biomechanical movement recordings are continuous, high-dimensional time series, and analyzing them without manually labeling them requires representations that capture how movements evolve over time. Most unsupervised representation-learning pipelines, however, treat each observation frame independently and discard this temporal structure, while energy-based models that could capture it have not extensively been adapted to biomechanical signals. So, this paper implements the Recurrent Temporal Restricted Boltzmann Machine (RTRBM), a recurrent energy-based model, into an unsupervised encoder that maps windows of movement data to fixed-length embeddings, and tests what its recurrent structure contributes. We adapt the model to continuous-valued signals, stack it into a deep architecture, and integrate it into the SIT-FUSE segmentation pipeline. We evaluate the embeddings on simulated musculoskeletal reaching and on two real gait datasets, using clustering, reconstruction, and walking-speed classification of unseen subjects, and compare them against non-recurrent and non-probabilistic baselines, including a control that removes the recurrent connection. Clustering the embeddings recovers movement structure tied to progress toward the target without any labels, which also transfers to unseen episodes. On gait data, the embeddings outperform PCA and autoencoder baselines under a nonlinear probe. However, removing the recurrent connection does not yield a measurable loss on the reaching task, and simple per-channel summary statistics classify walking speed at least as well as the embeddings. The model is therefore a working, verified encoder for SIT-FUSE, while the evidence for a benefit of recurrence is mixed, and the learned embeddings did not outperform simple summary statistics on gait.
Introduction
Biomechanical time series are high-dimensional, continuous, and inherently sequential. As features like joint kinematics and contact forces evolve in ways that reflect an underlying motor plan, significant information in a movement is spread across time. Therefore, treating each observation as an independent sample takes away that structure and reduces a movement to simply a set of disconnected snapshots.
The typical fix for this has been supervision, by training a classifier to predict movement phases from annotated recordings ([1]). However, exhaustively hand-labeling recordings is costly, and models trained on labeled recordings often degrade when the sensor placement, device, or subject population changes ([6, 8]). Meanwhile, unsupervised methods that can directly find temporal structure from raw data alleviate the effort required to produce labels, instead shifting efforts towards interpreting the structure a model finds. This overall motivates the use of unsupervised methods here.
Restricted Boltzmann Machines (RBMs) are undirected energy-based models used in unsupervised representation learning [10]; stacking them into Deep Belief Networks (DBNs) trained layer-by-layer helps to pull out abstract features [10]. Feeding these hidden representations into downstream clustering is the approach SIT-FUSE [13] takes for self-supervised segmentation of remote sensing imagery; the code here is a sequence-aware extension of that pipeline, and we use biomechanical reaching trajectories to validate it. Every standard RBM processes each input frame by itself. Nothing conditions the hidden state at time on what the model saw at , so no temporal context carries forward.
The Recurrent Temporal Restricted Boltzmann Machine [18] closes that gap (Figure 1). It augments the RBM with a learned hidden-to-hidden weight matrix that carries mean-field hidden probabilities forward in time, modulating the hidden bias at each step. It defines a generative distribution over whole sequences, and the original RTRBM paper showed that it generates more realistic motion-capture sequences and bouncing-ball videos than the temporal RBM (TRBM) it modifies [18]. Its motion-capture experiments used Gaussian visible units with a fixed unit variance. Biomechanical recordings combine joint angles, velocities, and muscle activations whose variability differs from channel to channel, so we extend the model with a learned per-feature variance, which requires care in how the variance is trained [7] and improves held-out reconstruction (§ 3.1). Temporal RBMs have so far been applied to human movement mainly to generate, predict, or fill in motion-capture sequences [19, 17, 18]. To our knowledge, they have not been used as unsupervised encoders that turn windows of biomechanical signals into fixed-length embeddings for clustering, which is the role we develop here.
Figure 1. Three ways to add memory to an RBM. (a) A standard RBM has no temporal connection; each frame is modeled independently. (b) The RTRBM used in this work reuses its own hidden units as the recurrent signal: sets the hidden bias at through a single learned weight matrix (red). (c) The RNN-RBM keeps a separate recurrent state , updated from the current frame and the previous , which then sets the RBM’s bias one step ahead (teal): more expressive, but with a second set of parameters to train and coordinate.
A related model, known as the RNN-RBM [4], conditions each RBM in the sequence on the hidden state of an RNN instead of on a direct hidden-to-hidden connection. This gives it more expressive power, but also makes it harder to train and requires separate optimization of the RNN and RBM components. The RTRBM’s single recurrent matrix is a simpler inductive bias that suits the short 20-step windows used here.
This paper describes how we developed the RTRBM into a working temporal encoder for SIT-FUSE. We implement the Bernoulli RTRBM and a Gaussian-visible variant with a learned per-feature variance [7], trained with a two-phase variance schedule, gradient clipping, and a sampled negative particle that keep training stable on continuous multichannel signals. We stack RTRBM layers into a Recurrent Temporal Deep Belief Network (RTDBN) that turns each movement window into one fixed-length embedding by temporal mean pooling, with windowing that never spans two recordings. The implementation follows Learnergy’s conventions, and its training gradients agree with exact maximum likelihood on small enumerable models. Our work is also integrated into SIT-FUSE as a selectable encoder. We evaluate it on simulated reaching trajectories from MyoSuite [5], where -means clusters of embeddings are organized around target distance and closing rate, as well as on two real gait cohorts, where the embeddings predict walking speed in unseen subjects better than PCA and autoencoder embeddings under a random-forest probe, but not better than simple per-channel summary statistics. Further, an ablation study finds no benefit from the recurrent connection itself on the reach data (§ 3.5).
Methods
Restricted Boltzmann Machines
An RBM defines a joint distribution over visible units and binary hidden units through the energy:
where , are bias vectors and is the weight matrix. The bipartite structure makes block Gibbs sampling tractable, and parameters are learned by Contrastive Divergence (Hinton, [9]), which runs a -step Gibbs chain from a data point to approximate the model expectation.
The Recurrent Temporal RBM
The RTRBM (Sutskever et al., [18]) adds a recurrent weight matrix and a learned initial hidden state . For a sequence , the hidden bias at step becomes:
where are the mean-field hidden probabilities from the previous step. The per-step conditionals are:
Gaussian Visible Units and Learned Variance
Binary visible outputs do not fit continuous biomechanical signals. Swapping the Bernoulli layer for Gaussian units changes the energy to:
with visible conditional and hidden conditional:
Making a learnable per-feature parameter lets each channel’s output scale adapt to its own variance. This is the parameterization from Cho et al. [7], which we call RTVarianceGaussianRBM.
Training RTVarianceGaussianRBM
We follow the two-phase schedule of Cho et al. [7]: is frozen at 1 for 15 epochs while , , , , and stabilize, then unfrozen for 15 more. We also clip gradients (global max norm 1.0) and clamp after every optimizer step.
is optimized directly as a standard deviation. and are initialized from ; all bias vectors (, , ) are initialized to zero; is initialized to 1. We use plain SGD, with batch size 32, on CPU. We selected each model’s learning rate by a grid search on validation reconstruction MSE, which was 1.0 for the canonical and models, 0.3 for the fixed-variance Gaussian model, and 0.1 for the Bernoulli model. In the canonical reach model, the learning rate of the group containing and is multiplied by 0.01; on the gait data this multiplier is 1, so all weights share one learning rate. Unless stated otherwise, every result below is over ten seeds (1–10); the seed controls the 70/15/15 episode split, weight initialization, and -means initialization, and models are compared paired by seed (same split). We report mean SD, paired differences with 95% bootstrap confidence intervals.
CD-1 is performed independently at each timestep, conditioned on the previous step’s mean-field hidden probability . Within a single timestep’s Gibbs step, the visible unit is sampled from its Gaussian conditional; this sample is the negative particle in the energy term, while the reconstruction-quality metric uses the conditional mean.
Algorithm: CD-1 training step for RTVarianceGaussianRBM, one window
Input: window v_{1..T} in R^{T x n_v}, previous states h_0
h_prev <- h_0 (broadcast to batch)
total_cost <- 0
for t = 1 to T:
// positive phase (teacher-forced on real data)
pos_probs, pos_states <- hidden_sampling(v_t, h_prev)
// negative phase (CD-1, one Gibbs step)
neg_h <- pos_states
v_recon, v_recon_mean <- visible_sampling(neg_h) // v_recon ~ N(mean, sigma^2)
neg_probs, neg_h <- hidden_sampling(v_recon, h_prev)
v_recon <- detach(v_recon) // stop-gradient
cost_t <- energy(v_t, h_prev) - energy(v_recon, h_prev)
mse_t <- sum((v_t - v_recon_mean)^2) / batch_size // metric only (mean)
total_cost <- total_cost + cost_t
// advance recurrent state using REAL data (teacher forcing), not reconstruction
h_prev <- hidden_sampling(v_t, h_prev).probs
total_cost.backward()
clip_grad_norm(params, max_norm=1.0)
optimizer.step()
sigma <- clamp(sigma, 0.1, 10.0)
On the reach data, we train on overlapping windows of length with stride 1 to keep BPTT tractable and yield more training samples; the gait data use non-overlapping windows (§ 2.9).
Recurrent Temporal Deep Belief Network
The RTDBN stacks RTRBM layers using greedy layer-wise pretraining (Hinton et al. [10]). After the final layer, we collapse the temporal hidden states into a single embedding per window by mean pooling (Figure 2):

Figure 2. The full RTDBN pipeline for a single window of length : each frame passes through the RTRBM layer, whose recurrent connection carries information from preceding frames; then, the resulting hidden sequence is mean-pooled into one embedding per window, which -means then assigns to a movement-phase cluster.
All experiments here use a single RTRBM layer; stacking more is implemented but unnecessary at this data scale.
Clustering
We cluster the embeddings with -means (, 20 random restarts), instead of the Invariant Information Clustering (IIC) loss Ji et al. (2019) that SIT-FUSE’s full pipeline normally uses. We use -means as a deliberately decoupled clustering readout, separate from encoder training.
We also report silhouette score Rousseeuw [16], which measures how much closer each point is to points in its own cluster than the points of other clusters, and normalized cluster entropy:
is uniform cluster sizes; is complete collapse. Because silhouette depends on embedding geometry, we also report (i) held-out per-channel of the mean reconstruction (reconstruction from sampled hidden states), summarized by its median over channels and by channel group (positions 0–19, velocities 20–39, tip position 40–42, reach error 43–45, muscle activations 46–79), and (ii) ridge-regression probes (penalty 1.0) fit on the train+validation embeddings and scored by on the test embeddings, for the following targets: the change in reach error over the window (reach trend), mean joint speed, and the relative position of the window within its episode (phase). We also report normalized mutual information (NMI) between the -means clusters and ten equal phase bins.
Data
We generate reaching data from the myoArmReachFixed-v0 environment in MyoSuite Caggiano et al. [5], which uses MuJoCo (Multi-Joint dynamics with Contact) Todorov et al. [21] to simulate a physiological upper-limb model on a fixed-target reaching task. We let 100 episodes run for 151 steps each, with an 80-dimensional observation vector: joint positions (20), joint velocities (20), reach-site position (3), reach error (displacement between target and reach site) (3), and muscle activations (34).
Actions are drawn from a momentum-biased random policy. At each timestep, an action is drawn from a Gaussian centered on a slowly-adapting drift vector; if the previous step reduced the distance to the target, the drift vector is nudged toward the action that produced the improvement, but it decays back toward zero if not.
We partition into 70 training, 15 validation, and 15 test set episodes, ensuring that overlapping windows from the same episode do not cross a split boundary. A StandardScaler (zero-mean, unit-variance per channel) is fit on training episodes and applied to validation and test data at inference time.
We extend SIT-FUSE’s data pipeline to handle multi-trial biomechanical recordings. A causal sliding window of length is applied within each episode before pooling.
Software Implementation, Verification and Integration with SIT-FUSE
Library. The temporal models are implemented as Learnergy classes: RTRBM (Bernoulli), RTGaussianRBM (fixed variance), RTVarianceGaussianRBM (learned per-feature variance), and RTDBN, which stacks RTRBM layers greedily.
Verification. We checked the training gradients against exact maximum likelihood on enumerable models. With the exact negative-phase expectation, the contrastive-divergence surrogate used by the code equals the automatic-differentiation gradient of the exact sequence log-likelihood, taken through the recurrent chain of the learned-variance model. The conditionals implied by the joint energy equal those the classes sample from.
Integration with SIT-FUSE. Setting encoder_type: rtdbn in a SIT-FUSE configuration file selects the temporal encoder. The stages are: encoder pretraining, the clustering stage, inference that writes one cluster label per window for the held-out splits, FLOP counting, and per-cluster TimeSHAP attributions over time steps and channels.
Real Gait Data and Probe Protocol
To test the embeddings outside simulation, we use two independent human gait cohorts where the task is to classify walking speed (three classes, chance = 0.333) from joint-angle windows of subjects that were not used for training. Cohort 1 contains 57 subjects walking at three speeds (0.5, 0.75 and 1.0 m/s), with 66 inertial-sensor-derived joint-angle channels at 60 Hz ([12, 14]). Cohort 2 is the dataset of Bailey et al. [3], released through SimTK (Bailey et al. [2]): 14 healthy adults walking on a treadmill at 70%, 100%, and 130% of their preferred speed, with IMU-driven joint angles from inverse kinematics; we keep the 18 rotational channels that vary by more than one degree (a label-free rule) and drop the first five seconds of each trial.
Windows are non-overlapping with frames, standardized with training-subject statistics. Splits are at the subject level (70/15/15 for cohort 1; 10/2/2 subjects for cohort 2), with ten splits per cohort. Each model produces a 64-dimensional embedding (the RTRBM variants by temporal mean pooling), trained for 30 epochs with the schedule of §2.4 and a learning rate chosen only from validation reconstruction error (0.1 for cohort 1, 0.3 for cohort 2). The comparison encoders are frame-wise PCA, window PCA, an MLP autoencoder, and a GRU autoencoder, all with 64-dimensional embeddings, and a training-free baseline whose embedding is the per-channel mean, standard deviation, minimum and maximum over the window (4× the number of channels: 264 features for cohort 1 and 72 for cohort 2). Two probes are fit on frozen embeddings and scored by test accuracy on unseen subjects: logistic regression and a random forest (40 trees). Contrasts are paired over the ten splits with 95% bootstrap intervals and Holm correction.
Results
Model Comparison
We run three RTRBM variants at , , CD-1, , for 30 epochs and ten seeds, changing the visible unit parameterization: Bernoulli units on min–max scaled data, and fixed-variance and learned-variance Gaussian units on standardized data. Aggregate MSE is not comparable across these (the Bernoulli model’s MSE is computed against binary samples and its data are on a different scale), so we compare held-out per-channel computed from the mean reconstruction. The Bernoulli model reaches a median of , the fixed-variance Gaussian model , and RTVarianceGaussianRBM . The paired difference between the learned- and fixed-variance models is (95% CI ) in median and in held-out MSE ( versus ; CI ). With the learned variance, joint kinematics reconstruct well (median 0.86 for positions and 0.76 for velocities) and muscle activations moderately (0.56) (Figure 3).

Figure 3. (a) Training MSE divided by total window variance (mean SD over ten seeds) for the fixed-variance Gaussian (red) and learned-variance Gaussian (green) RTRBMs; the dashed line marks epoch 15, when is unfrozen. The Bernoulli model is omitted from (a) because its MSE is measured against binary samples. (b) Median per-channel held-out for all three variants (bars: mean SD; dots: individual seeds).
RTDBN Embeddings and Movement Clustering
Applying the single-layer RTDBN (, ) to embeddings from the frozen encoder gives 11,220 train+validation windows split across 10 populated clusters, with normalized entropy and a silhouette score of (ten seeds). The absolute silhouette is low, indicating overlapping rather than well-separated clusters, as expected for windows sampled along continuous trajectories. Cluster agreement with ten equal phase bins is moderate (NMI ), and a linear probe recovers the window’s relative position in its episode from the embedding with .
A PCA projection of the embeddings, colored by cluster, is shown in Figure 4.

Figure 4. PCA of 64-dimensional RTDBN embeddings for 11,220 windows (seed 1), colored by -means cluster (). PC1 and PC2 account for 31.1% and 12.5% of variance.
Looking at representative windows from the three largest clusters (seed 1) shows a kinematic reading organized around distance to the target and closing rate. The largest cluster () is near the target (mean reach-error norm 0.143 m) but still closing it (falling by 0.077 m within the window, compared to a population mean reach-error norm of 0.358 m); the second () is also near the target (0.151 m) and nearly static (falling by only 0.018 m, with the lowest mean muscle-activation norm of the three being 1.449); and the third () is far from the target (0.597 m) and is closing slowly (0.066 m). (Figure 5).

Figure 5. Eight randomly drawn example windows from each of the three largest clusters (rows; seed 1), cluster sizes 2,174, 1,370, and 1,344, across four observation-derived channels (columns): reach-error norm, joint-speed norm, muscle-activation norm, and tip displacement from the start of the window. Thin lines are individual windows, while the bold line is the per-cell mean.
Across all 10 clusters, the mean reach-error norm falls into three bands (about 0.14–0.23 m, 0.43–0.50 m, and 0.58–0.65 m) and the closing rate varies within each band; the two fastest-closing clusters ( and ) fall by 0.167 m and 0.157 m and have the highest joint speed (0.090) and tip displacement (0.328) of all ten. A more in-depth breakdown is provided in the appendix (Figure 7, Table 5). These readings are merely descriptive as they are for one seed, and cluster identities and sizes differ across seeds.

Figure 7. Reach-error magnitude by -means cluster, all 10 clusters (seed 1; 8 example windows per cluster, thin lines; mean, bold line).
| Cluster | Reach err. | Trend | Joint spd. | Muscle eff. | Tip disp. | |
| 0 | 2,174 | 0.143 | −0.077 | 0.069 | 1.535 | 0.164 |
| 1 | 1,194 | 0.142 | −0.032 | 0.071 | 1.502 | 0.172 |
| 2 | 1,207 | 0.577 | −0.122 | 0.082 | 1.555 | 0.224 |
| 3 | 700 | 0.496 | −0.157 | 0.084 | 1.666 | 0.328 |
| 4 | 713 | 0.226 | +0.004 | 0.076 | 1.500 | 0.202 |
| 5 | 935 | 0.430 | −0.022 | 0.070 | 1.570 | 0.205 |
| 6 | 706 | 0.488 | −0.028 | 0.066 | 1.525 | 0.132 |
| 7 | 1,344 | 0.597 | −0.066 | 0.063 | 1.517 | 0.141 |
| 8 | 877 | 0.652 | −0.167 | 0.090 | 1.512 | 0.265 |
| 9 | 1,370 | 0.151 | −0.018 | 0.063 | 1.449 | 0.127 |
Table 5. Per-cluster kinematic summary, all 10 clusters. Trend is the change in reach-error norm from the start to the end of the window (negative = closing distance to target). Joint speed, muscle effort, and tip displacement are window-mean norms.
Test Split Generalization
We apply the trained encoder and -means centroids from the training split to the 15 held-out test episodes, which played no role in training. The split yields 1,980 windows. Held-out reconstruction MSE, computed with the same procedure on both sets, is on the test episodes and on the validation episodes, which is consistent with no overfitting to the training episodes. All 10 clusters stay populated on the test windows, the normalized entropy is , and the silhouette score is on the test windows versus on train+validation. This indicates that the clustering is stable under an episode-disjoint split within our simulation setting.
Ablation Study
We vary (with fixed) and (with fixed), measuring reconstruction MSE, silhouette score, and normalized entropy (Table 1).
| Configuration | Test MSE | MSE/() | Median | Silhouette | |
| , | |||||
| , | |||||
| , | |||||
| , | |||||
| , | |||||
| , | |||||
| , |
Table 1. Ablation results (five seeds, mean; learning rates selected at , and not re-tuned). Test MSE is summed over the window ( SD over seeds); MSE/() divides it by to make window lengths comparable; is the median per-channel held-out value. All configurations produce 10 populated clusters.
Window length has almost no effect once reconstruction error is normalized by the window size: MSE/() is 0.32–0.35 and median is 0.60–0.61 for every , and silhouette varies only between 0.140 and 0.153 (within roughly one seed SD). Hidden size matters for reconstruction: median held-out rises from 0.549 at to 0.610 at and 0.678 at . Silhouette moves in the opposite direction (0.212, 0.153, 0.141).
Determining if Recurrence Matters: A Ablation
The ablations above vary window length and hidden size, but do not test the paper’s central premise directly: does the recurrent connection improve the representation? To explore this, we retrain the canonical , configuration with initialized to zero and frozen throughout training, which removes the hidden state’s dependence on its own history and reduces the model to a per-frame Gaussian RBM with learned variance, mean-pooled over the window as before.
No metric in Table 2 differs between the two models: the paired difference in test MSE ( minus canonical) is (95% bootstrap CI , paired ), in median (CI ), and in test silhouette ; no probe difference has a confidence interval excluding zero (joint speed: , CI ). Because the CIs are narrow, a benefit of larger than about 19 units of test MSE (3.5%) or 0.005 in median can be excluded at this sample size.
| Canonical ( free) | ||
| Test MSE | ||
| Median channel (test) | ||
| Trainval silhouette | ||
| Test silhouette | ||
| Probe : phase | ||
| Probe : reach trend | ||
| Probe : joint speed |
Table 2. ablation (ten seeds, mean SD; models paired by seed). Identical data, split, and training schedule to the canonical model; only is zeroed and frozen. Probe columns are test of a ridge probe on the embedding.
Number of Clusters
We sweep to verify that holds up and to look for degenerate behavior at high (Figure 6). Silhouette declines overall, from 0.247 at down to 0.142 at (seed 1) and is roughly flat at 0.14–0.16 for . The downward trend is expected when clusters are only weakly separated, as for windows sampled along continuous trajectories, because the silhouette score then tends to favor fewer, larger clusters [20]. The within-cluster sum of squares decreases from 33,148 to 17,546 across the full range with no sharp elbow at any single ; (silhouette 0.157) is not distinguished by either curve, and it matches the fixed clustering resolution used throughout the rest of the paper’s comparisons.

Figure 6. Silhouette score (blue, left) and inertia (orange, right) versus . Silhouette declines overall. Inertia decreases smoothly with no sharp elbow; the red line marks , the value used throughout the rest of the paper.
Comparison with Non-RBM Baselines
We compared the RTRBM with plain non-probabilistic models on the same ten splits, scaling, windows, -means, and probes: PCA on the flattened window or frame by frame, a frame-wise autoencoder, and a causal GRU autoencoder in which each frame is reconstructed from the recurrent state after seeing it (the closest non-probabilistic analogue of the RTRBM). All use 64-dimensional embeddings, the same 30 training epochs, and a learning rate from chosen on validation reconstruction error on one seed (Table 3).
| Model | Test MSE | Median | Silhouette | Phase | Speed | Reach trend |
| RTRBM, canonical | ||||||
| RTRBM, | ||||||
| PCA, flattened windows | ||||||
| PCA, frame-wise | ||||||
| Frame-wise autoencoder | ||||||
| Causal GRU autoencoder |
Table 3. RTRBM variants and non-RBM baselines on the held-out reach windows (ten seeds, mean s.d.). MSE is summed over the 20 frames and 80 channels of a window and averaged over windows (the RBM numbers use sampled binary hidden states; the baselines are deterministic). Silhouette is computed on the train and validation episodes (). Probe columns are held-out of a ridge probe on the embedding.
The RTRBM variants are not competitive reconstructors. Frame-wise PCA with 64 components reconstructs the held-out windows essentially exactly (median 1.000), because the 80 channels are highly redundant, and the autoencoders reach median of at least 0.998, against 0.608 for the canonical RTRBM and 0.615 for the variant. The baselines are better on the probes for the reach quantities (reach trend: 0.66 for windowed PCA against 0.37 for the canonical RTRBM; paired differences are positive for every baseline, Holm-adjusted $p < 0.001). The RTRBM embeddings are better on the speed probe (mean absolute joint velocity: 0.79 against at most 0.35 for every baseline in Table 3). The model has the same speed probe (0.82), so this is not due to recurrence.
Walking-Speed Classification on Two Gait Cohorts
Table 4 gives the held-out-subject accuracies of the two probes. Among the learned encoders, the RTRBM embedding gives the highest random-forest accuracy in both cohorts: 0.577 against 0.456 to 0.515 for the four learned baselines in cohort 1, and 0.610 against 0.413 to 0.559 in cohort 2. All four contrasts hold after Holm correction in both cohorts (cohort 1: +0.062 to +0.121, Holm ; cohort 2: +0.051 to +0.197, Holm ). In the linear probes the RTRBM embeddings are better than frame-wise PCA, window PCA and the MLP autoencoder in cohort 2 (Holm to 0.033) and than window PCA in cohort 1 (Holm ), and the remaining contrasts do not separate the encoders, so we do not claim an advantage there. With the 0.01× multiplier on used for the reach data, the random-forest accuracies are 0.542 (cohort 1) and 0.602 (cohort 2), which is also above all four learned baselines.
| Probe | RTRBM | RTRBM | PCA frame | PCA window | MLP-AE | GRU-AE | Stats |
| Cohort 1 (57 subj.) linear | 0.514 | 0.470 | 0.504 | 0.465 | 0.503 | 0.497 | 0.730 |
| Cohort 1 (57 subj.) random forest | 0.577 | 0.515 | 0.456 | 0.515 | 0.463 | 0.466 | 0.649 |
| Cohort 2 (14 subj.) linear | 0.529 | 0.518 | 0.437 | 0.455 | 0.446 | 0.500 | 0.567 |
| Cohort 2 (14 subj.) random forest | 0.610 | 0.591 | 0.413 | 0.559 | 0.498 | 0.479 | 0.649 |
Table 4. Walking-speed accuracy on unseen subjects (mean over ten subject-level splits; chance = 0.333). RTRBM: learned-variance Gaussian RTRBM; removes the recurrence; Stats: per-channel mean, standard deviation, minimum and maximum of the window (no training). Bold: best in row.
Removing the recurrence lowers random-forest accuracy by 0.062 in cohort 1 (95% CI [+0.046, +0.078]) but only by 0.019 in cohort 2 (CI [−0.017, +0.061]), so recurrence appears to help in cohort 1 and is not distinguishable from zero in cohort 2. However, this claim is limited to these cohorts and task.
The training-free summary-statistics baseline is stronger than the RTRBM embedding. In cohort 1, it reaches 0.730 with the linear probe and 0.649 with the random forest, compared to 0.514 and 0.577 for the RTRBM (paired differences +0.216, 95% CI [+0.174, +0.258], and +0.072, CI [+0.048, +0.094]; Holm and over the four contrasts). In cohort 2, it is numerically higher as well (0.567 and 0.649 against 0.529 and 0.610), but the differences are within noise (+0.038, CI [−0.008, +0.078], and +0.039, CI [−0.009, +0.087]; Holm ). So the advantage over frame-wise and window PCA and over the autoencoders does not show that the RTRBM extracts walking-speed information that simple handcrafted features lack. The summary statistics use as many dimensions as the 64-dimensional learned embeddings, and we ran this comparison after seeing the other results.
Discussion
Across ten seeds, multiple results emerged. First, learned-variance Gaussian visible units reconstruct held-out windows substantially better than Bernoulli and fixed-variance units (median 0.61 versus 0.09 and 0.57). Second, the clusters transfer to unseen episodes: the -means centroids sort 15 held-out episodes into the same 10 groups with test silhouette at least as high as on the training episodes (0.170 ± 0.018 versus 0.154 ± 0.013) and held-out reconstruction error as low as on validation episodes, which indicates that the clusters reflect properties of the reaching movement rather than of individual episodes.
One likely reason for the W′ = 0 result is that reconstruction can be solved without memory; the canonical and W′ = 0 models reach the same reconstruction error (§ 3.5), so a reconstruction-based objective gives W′ little to learn. We therefore do not interpret the null as evidence that recurrence cannot help, but instead as a property of reconstruction-based training on this dataset.
The data used in this study is intentionally limited in scope. Each episode uses a random action policy on a single fixed target, which gives us windows that are kinematically diverse but not behaviorally diverse. Meanwhile, variable-target reaching, task-directed policies, or real sensor data would give the encoder more to differentiate and would likely, as a result, produce richer cluster structure.
For future SIT-FUSE applications to sequential data, the results suggest the following practical use of the toolkit. The RTRBM/RTDBN embedding is a compact (64-dimensional) window descriptor that a nonlinear probe or a clustering stage can use directly, and on real gait data it is a better descriptor of walking speed than PCA or autoencoder embeddings under a random-forest probe, but not better than simple per-channel summary statistics (§ 3.8). The per-window outputs and TimeSHAP reports of the SIT-FUSE integration (§ 2.8) then allow the resulting clusters to be inspected by channel and time step. In future work, larger and more varied cohorts and joint training with the information-theoretic clustering head are some natural next evaluations.
Conclusion
RTVarianceGaussianRBM, with its per-feature learned variance, a validation-selected learning rate and a two-phase warmup schedule, reconstructs held-out reaching windows far better than Bernoulli or fixed-variance alternatives (median 0.61 versus 0.09 and 0.57; ten seeds). The RTDBN collapses the resulting temporal hidden states into fixed-length embeddings whose -means clusters carry a reading in terms of distance to the target and closing rate and transfer to held-out episodes. On two real gait cohorts the same embeddings classify walking speed in held-out subjects better than PCA and autoencoder embeddings under a random-forest probe, but not better than per-channel summary statistics, and the verified implementation runs end to end inside SIT-FUSE, from pretraining and clustering to per-window outputs and TimeSHAP attributions.
Broader Impact Statement
This work develops unsupervised tools for analyzing biomechanical movement data. The simulated experiments use synthetic data, while the gait experiments use two publicly released datasets of healthy adults walking on a treadmill, with subjects identified only by code. The first dataset is released under a CC BY 4.0 license with written informed consent and institutional review board approval [12]; the second is distributed through SimTK [2] with its associated publication [3]. These methods make no clinical decisions, and any downstream use would need additional validation; we are also not aware of any significant negative impacts.
Implementation Details
All models are implemented in PyTorch, extending the Learnergy library [15] and following its existing conventions. Source code for the temporal model classes, the dataset class, the experiments and the SIT-FUSE integration is provided as anonymized supplementary material, and will be released publicly upon acceptance.
Hyperparameters were selected by validation reconstruction MSE on one seed (seed 42, a split different from seeds 1–10); the grids were learning rate (screening for 10 epochs at coarse spacing, then 30 epochs at for the variance-Gaussian family), and the learning-rate multiplier .
Supplementary: All-Cluster Kinematic Breakdown
Figure 7 and Table 5 extend Figure 5 to all 10 clusters, not only the three largest.
References
- [1]Ali H. Al-Timemy, Claudio Castellini, Javier Escudero, Rami Khushaba, and Silvia Muceli. Editorial: Current trends in deep learning for movement analysis and prosthesis control. Frontiers in Neuroscience, 16:889202, 2022. doi: 10.3389/fnins.2022.889202.
- [2]Christopher A. Bailey, Thomas K. Uchida, Julie Nantel, and Ryan B. Graham. IMU-driven (OpenSense) and optoelectronic-driven (OpenSim) kinematic models for several 7-minute conditions of continuous treadmill gait (bailey2021_imu). SimTK project, 2021a. URL https://simtk.org/projects/bailey2021_imu.
- [3]Christopher A. Bailey, Thomas K. Uchida, Julie Nantel, and Ryan B. Graham. Validity and sensitivity of an inertial measurement unit-driven biomechanical model of motor variability for gait. Sensors, 21(22): 7690, 2021b. doi: 10.3390/s21227690.DOI
- [4]Nicolas Boulanger-Lewandowski, Yoshua Bengio, and Pascal Vincent. Modeling temporal dependencies in high-dimensional sequences: Application to polyphonic music generation and transcription. In International Conference on Machine Learning, 2012.
- [5]Vittorio Caggiano, Huawei Wang, Guillaume Durandau, Massimo Sartori, and Vikash Kumar. MyoSuite – a contact-rich simulation suite for musculoskeletal motor control. arXiv preprint arXiv:2205.13600, 2022.arxiv.org/abs/2205.13600
- [6]Avijoy Chakma, Abu Zaher Md Faridee, Indrajeet Ghosh, and Nirmalya Roy. Domain adaptation for inertial measurement unit-based human activity recognition: A survey. arXiv preprint arXiv:2304.06489, 2023.arxiv.org/abs/2304.06489
- [7]Kyunghyun Cho, Alexander Ilin, and Tapani Raiko. Improved learning of Gaussian-Bernoulli restricted Boltzmann machines. In International Conference on Artificial Neural Networks, pp. 10–17. Springer, 2011.DOI
- [8]Sourish Gunesh Dhekane and Thomas Ploetz. Transfer learning in human activity recognition: A survey. arXiv preprint arXiv:2401.10185, 2024.arxiv.org/abs/2401.10185
- [9]Geoffrey E Hinton. Training products of experts by minimizing contrastive divergence. Neural Computation, 14(8):1771–1800, 2002.pubmed.ncbi.nlm.nih.gov/12180402
- [10]Geoffrey E Hinton, Simon Osindero, and Yee-Whyteh. A fast learning algorithm for deep belief nets. Neural Computation, 18(7):1527–1554, 2006.pubmed.ncbi.nlm.nih.gov/16764513
- [12]Rateb Katmah, Aamna AlShehhi, Doua Kosaji, Nour Al-Rahmani, Muhammad Abdullah, Abdul Aziz Vaqar Hulleck, and Kinda Khalaf. A multimodal gait dataset of brain activity, muscle activity, kinematics and ground forces in young adults (version 1.0.0), 2026. URL https://doi.org/10.13026/r0ea-7161. RRID:SCR_007345. License: CC BY 4.0.
- [13]Nicholas LaHaye, Michael J. Garay, Brian Bue, Hesham El-Askary, and Erik Linstead. A quantitative validation of multi-modal image fusion and segmentation for object detection and tracking. Remote Sensing, 13(12):2364, 2021. doi: 10.3390/rs13122364.DOI
- [14]Tom Pollard, Benjamin E. Moody, Li-wei H. Lehman, Brian J. Gow, Chrystinne Fernandes, Chen Xie, Alistair Johnson, Roger G. Mark, and Thomas Heldt. PhysioNet as a global platform for biomedical research. Nature Health, 1(8):792–795, 2026. doi: 10.1038/s44360-026-00096-z.DOI
- [15]Mateus Roder and Gustavo de Rosa. Learnergy: Energy-based machine learners, 2020. URL https://github.com/gugarosa/learnergy.arxiv.org/abs/2003.07443
- [16]Peter J. Rousseeuw. Silhouettes: A graphical aid to the interpretation and validation of cluster analysis. Journal of Computational and Applied Mathematics, 20:53–65, 1987. doi: 10.1016/0377-0427(87)90125-7.DOI
- [17]Sainbayar Sukhbaatar, Takaki Makino, Kazuyuki Aihara, and Takashi Chikayama. Robust generation of dynamical patterns in human motion by a deep belief nets. In Proceedings of the Asian Conference on Machine Learning, volume 20 of Proceedings of Machine Learning Research, pp. 231–246. PMLR, 2011.
- [18]Ilya Sutskever, Geoffrey E Hinton, and Graham W Taylor. The recurrent temporal restricted boltzmann machine. In Advances in Neural Information Processing Systems, volume 21, 2008.
- [19]Graham W. Taylor, Geoffrey E. Hinton, and Sam T. Roweis. Two distributed-state models for generating high-dimensional time series. Journal of Machine Learning Research, 12(28):1025–1068, 2011.DOI
- [20]Zongyue Teng, Jun Yan, Dandan Liu, and Panpan Zhang. When does the silhouette score work? a comprehensive study in network clustering. arXiv preprint arXiv:2512.24841, 2025.arxiv.org/abs/2512.24841
- [21]Emanuel Todorov, Tom Erez, and Yuval Tassa. MuJoCo: A physics engine for model-based control. In IEEE/RSJ International Conference on Intelligent Robots and Systems, pp. 5026–5033. IEEE, 2012.DOI
