Comparing Temporal Deep Learning Architectures for Discrete Cross-User Myoelectric Control

Introduction

As human-machine interaction (HMI) continues to expand beyond conventional computing environments, the need is arising for new enabling technologies to replace traditional inputs. Leveraging the physiological electromyogram (EMG) signal, the electrical potentials generated by muscles when they are contracted, myoelectric control is one possibility for these emerging use-cases [1]. However, despite its promise and history of success in controlling powered prostheses, the widespread use of this technology for HMI has largely been hindered by the need for regular and personalized end-user calibration [2].

Compared to the traditional use case of continuous myoelectric control, whereby decisions are made on small windows of data [3], discrete control maps dynamic gestures (e.g., a double tap or finger snap) to event-based commands [2]. Although developing true cross-user models that require no training data from the end user is challenging for continuous control, recent work is beginning to show that the temporal structure of dynamic gestures may help overcome this challenge [1, 4]. Moreover, discrete control might even be more appropriate for HMI as many inputs, such as button clicks and swipes, are inherently event-based [2, 5, 6].

Although previous work has demonstrated the feasibility of cross-user discrete control without recalibration [1, 4], little work has evaluated techniques and approaches to optimize these systems. This work correspondingly benchmarks five temporal deep learning architectures and compares hand-crafted feature extraction to machine-learned CNN-based approaches on the 612-user EMG-EPN dataset [7]. Additionally, to test the real-world performance and generalizability of these models, a never released dataset with 14 participants was used, where users were randomly prompted to elicit discrete gestures during a continuous data collection. This required that these gestures be detected and segmented from the continuous input stream, which is often ignored in offline studies. The results suggest, for the first time, that even under these real-world settings (i.e., data collected years apart, in a different country, and in the context of randomly prompted gestures), classification accuracies can reach up to 96% on data from unseen users. These results provide valuable insights for the future development of discrete cross-user myoelectric control systems and further highlight the potential viability of this promising technology.

Methods

Datasets

This work leveraged two datasets that were recorded nearly five years apart: (1) EMG-EPN-612 [7] and (2) MyoDisCo [8]. Each dataset was recorded with the Myo Armband, an 8-channel surface-EMG device with a sampling rate of 200 Hz [9]. The armband was placed in approximately the same location on each subject, with the LED on the extensor and the USB port facing down toward the hand. The six common gestures used here were: (1) hand open, (2) hand close, (3) wrist flexion, (4) wrist extension, (5) double tap, and (6) rest.

1. EMG-EPN-612: Recorded by Benalcazar et al., this is the largest publicly available EMG dataset, with recordings from 612 users [7]. During the data collection, users were prompted to elicit each discrete gesture at some point during a five-second window (this included the ramp on and off of the gesture). Indices indicating the start and end of each gesture are included in the dataset’s metadata. Labelled data are available for 50 repetitions of each gesture from 306 users and 25 repetitions from the other 306 (a total of 137,700

discrete gesture templates). A more detailed description of this dataset can be found in [4] and [10].

2. MyoDisCo: The MyoDisCo dataset was initially recorded to evaluate the impact of confounding factors for discrete myoelectric control [8]. However, a new piece of this dataset that has never been analyzed or released is the continuous gesture collection (see Figure 1). During this collection, participants were randomly prompted every 3-5 seconds (by a pop-up on a computer screen) to elicit one of the discrete gestures. No feedback was given to participants during this session. This collection aimed to provide a more real-world benchmark to evaluate the performance of discrete models when the active gesture must be segmented from the continuous EMG stream. This data collection was approved by the University of New Brunswick Research Ethics Board (REB 2023-140).

Classification

The deep learning architecture used for classifying the discrete gestures was inspired by the two closest previous works [1], [4]. The data for each discrete gesture in the dataset were first processed and windowed (size=50 ms and increment=25 ms). For the handcrafted feature case, the wavelet energy (WENG) feature was extracted from each window [11]. For the machine-learned feature extraction case, three 1D convolutional layers (kernel size of 3 and stride of 1) were applied to the raw windowed data with output channel sizes of 16, 32, and 64. Each layer maintained constant padding and had batch normalization, max pooling, a ReLU activation function, and dropout of 0.2. Each template (padded with trailing zeros to ensure constant template sizes in the dataset) was then passed through a network of 1 to 7 temporal layers with respective hidden sizes of 128 neurons and dropout of 0.2. Only the final output encodings from the last window of the gesture were forwarded to the classification network and used to update the network. The classification network was composed of three linear layers of size 128, 64, and 32 before being projected to a final layer of 6 neurons representing the 6 possible classes. ReLU activation functions were added between each linear layer. An Adam optimizer [12] and the cross-entropy loss function were used in all cases. Additionally, gradient norm clipping was employed to mitigate the issue of exploding gradients during training. Each model was trained for 50 epochs with an initial learning rate of 0.001 and a batch size of 1000. 80% of the labelled EMG-EPN dataset was used for training (428 users), and the other 20% (128 users) for validation. No additional tuning or training was done when evaluating the models with the MyoDisCo dataset. All code was developed in Python and leveraged PyTorch [13] for deep learning, and LibEMG [14] for EMG-specific data processing.

1. Algorithms: This work evaluated five temporal architectures. The LSTM, bi-directional LSTM (BILSTM), and GRU all had identical structures as highlighted above. For the RNN, however, the nonlinearity added between each layer was modified from Tanh to ReLU to account for vanishing gradients that were observed during tuning. The transformer was designed with a $d_{model}$ of 128 to have a similar hidden representation size as the hidden layers of the other models. The $dim_{feedforward}$ was set to 512, the number of attention heads to eight, and positional embeddings were learned during training

Gesture Segmentation

As participants returned to rest between gestures in the continuous data collection, an energy-based thresholding approach could be used to extract potential gesture templates from the continuous stream of EMG data [15], [16]. The Teager-Kaiser energy was taken across each channel of EMG data and then averaged and smoothed using a window size of 150 ms with an increment of 25 ms [17]. The first step in the gesture extraction process involved waiting until the energy fell below a desired threshold $t_{ready}$. Next, a gesture was assumed to have started when the energy surpassed an onset threshold $t_{onset}$. The gesture extractor then waited until the energy reached its maximum amplitude, with a buffer (i.e., it waited up to 100 ms without an increase before stopping). Finally, the gesture detector waited until the energy reached a local minimum and the extracted gesture was passed through the classifier. Based on pilot studies, $t_{ready}$ and $t_{onset}$ were set to 75 and 150, respectively.

Statistical Analysis

Using a Shapiro-Wilk test, all data were first checked to see if they followed a normal distribution. As all data followed a non-normal distribution, Friedman tests were used to check for significant differences between the factors of interest and, post-hoc, paired Nemenyi tests were used to check for significant differences between specific pairs. A non-parametric Wilcoxon test was applied for any comparisons where the number of groups was less than three.

Results

Offline Results

The results (see Figure 2) focus on the dependent variable classification accuracy and three independent variables: (1) MODEL, (2) FEATURE, and (3) LAYERS.

1) Model: When averaging across LAYERS and FEATURE, a significant effect of MODEL was found ($\chi^{2}(4)=248$, $p=2.2 \times10^{-16}$). Pairwise comparisons highlighted significant differences (of p<0.001 unless specified) between the following: BILSTM (92.1%) and GRU (92.9%), BILSTM and RNN (89.8%), BILSTM and Transformer (91.6%) (p=0.023), GRU and LSTM (91.9%), GRU and RNN, GRU and Transformer, LSTM and RNN, and finally, RNN and Transformer.

When taking the best configuration for each model, a significant effect of MODEL was found ($\chi^{2}(4)=23$, $p=0.0001$). Significant differences were found between RNN (layers=5, cnn=False) (91.9%) and LSTM (layers=3, cnn=False) (92.8%) (p=0.009) and RNN and GRU (layers=7, cnn=True) (93.5%) (p=0.0003). No significant differences were found for the BILSTM (layers=2, cnn=True) (92.8%) or Transformer (layers=7, cnn=True) (92.8%).

2) Parameters: The models in ascending order of average number of parameters were the RNN (140916), GRU (360257), LSTM (469927), Transformer (804033), and BILSTM (1265780).

3) Layers: A significant effect of LAYERS was found when averaging across FEATURE and MODEL ($\chi^{2}(4)=215$, $p=2.2 \times10^{-16}$). Pairwise comparisons highlighted significant differences (p<0.001 unless noted) between the following: 1 (90.6%) and 2 (91.6%), 1 and 3 (92.0%), 1 and 5 (92.1%), 1 and 7 (92.0%), 2 and 3, 2 and 5, and finally, 2 and 7 (p=0.009). No significant effect of LAYERS was found when comparing the best configuration from the different number of layers ($\chi^{2}(4)=4$, $p=0.34$).

4) Feature Extraction: When averaging across MODEL and LAYERS, no significant differences (p=0.052) were found between the handcrafted (WENG) (91.4%) and machine-learned (CNN) (91.9%) features. Additionally, no significant differences (p=0.29) were found when comparing the best configuration with handcrafted (model=GRU, layers=5) (92.8%) and machine learned (model=GRU, layers=7) (93.5%) features.

Continuous Stream Evaluation

The results from the MyoDisCo dataset are shown in Figure 3. Using the energy-based gesture extraction strategy, false positive and negative rates (for detecting gestures) were found to be 1.7% and 0.3% with an average delay from prompt to prediction (including the elicitation of the gesture) of 700 ms. When averaging across LAYERS and FEATURE a significant effect of MODEL was found ($\chi^{2}(4)=32$, $p=1.6 \times10^{-6}$). Pairwise comparisons highlighted significant differences between the following: BILSTM (92.7%) and GRU (94.4%) (p=0.028), GRU and RNN (90.4%) (p<0.001), GRU and Transformer (88.5%) (p<0.001), LSTM (92.8%) and RNN (p=0.028), and finally, LSTM and Transformer (p=0.009). No significant differences were found ($\chi^{2}(4)=3$, $p=0.5$) when comparing the best configuration for each model: BILSTM (layers=3, cnn=True) (94.4%), GRU (layers=2, cnn=True) (96.1%), LSTM (layers=2, cnn=True) (94.2%), RNN (layers=5, cnn=False) (94.3%), and Transformer (layers=3, cnn=True) (91.8%).

Discussion

This paper evaluated five temporal deep learning architectures for discrete myoelectric control. The GRU consistently performed best, followed by the BILSTM, LSTM, Transformer, and RNN. Subtle differences in the generalization of some models (e.g., BILSTM) to the MyoDisCo dataset may be due to differences in how gestures were segmented (it is unknown how segmentation was achieved in the EPN dataset). This suggests that minor variations in the framing of input templates may affect model robustness. While initial differences were significant when averaging across configurations, these results were less pronounced when comparing the optimized configuration from each model. This suggests that all of the proposed architectures are viable if configured correctly.

The BILSTM and Transformer architectures required substantially more parameters than the other models. As these control systems will likely be deployed on embedded systems, this becomes an important design consideration. While increasing the layers improved performance on the EPN dataset, the best models on MyoDisCo were not necessarily the largest, suggesting that the additional parameters did not lead to improved generalizability. While the CNN-based feature extraction did not significantly improve performance, the highest-performing models on the MyoDisCo dataset leveraged the CNN architecture, indicating that it generalizes quite well. Future work could likely optimize these systems further for improved performance. Overall, these results provide valuable insights into the future design of discrete cross-user myoelectric control systems.