betaFeedback

WorksComputer Science

Self-Supervised Pretraining for Label-Efficient Facial Expression Recognition using Smart Glasses

Abstract

Facial Expression Recognition (FER) on optomyographic (OMG) smart glasses such as OCOsense enables privacy-preserving, camera-free affect sensing for health and well-being. Most FER systems are fully supervised, i.e., they rely either on hand-crafted expert features with classical machine learning, or on end-to-end deep models trained from labelled data. Both depend on large quantities of labelled expressions, which are costly to collect and limit deployment in real-world, personalized settings. We investigate Self-Supervised Learning (SSL) as a route to label-efficient FER on these glasses. We pretrain a Vision Transformer with a masked-autoencoder objective on unlabelled OCOsense recordings and finetune it for five-class FER under person-independent cross-validation. Across label budgets from 5% to 100%, pretraining consistently improves performance; with only 5% of the labels, the pretrained model outperforms the from-scratch model trained with all labels. Our results show that SSL pretraining is a practical way to reduce labelling cost for smart eyewear FER, and we discuss the implications for future personalization and continual adaptation.

ACM Reference Format: Zhengtao Gao, Francesco Bombassei De Bona, Leonardo Alchieri, Ivana Kiprijanovska, Simon Stankoski, Bojan Sofronievski, Hristijan Gjoreski, Marc Langheinrich, and Martin Gjoreski. 2026. Self-Supervised Pretraining for Label-Efficient Facial Expression Recognition using Smart Glasses. In Companion of the 2026 ACM International Joint Conference on Pervasive and Ubiquitous Computing (UbiComp Companion ’26), October 11–15, 2026, Shanghai, China. ACM, New York, NY, USA, 7 pages. https://doi.org/10.1145/3798063.3839146

Introduction

Mental-health and well-being disorders are among the leading causes of disability worldwide, affecting roughly 16–20% of the population in many regions, yet most individuals rarely receive timely or personalized support [2]. Wearable devices now make continuous, in-the-wild behavioral and physiological monitoring feasible at scale, opening the door to earlier and more personalized well-being sensing [3]. A persistent obstacle, however, is robust generalization across users, devices, and real-world conditions. Behavioral and affective signals vary substantially from person to person, and models that perform well on a training cohort often degrade on new, unseen users [11]. Methods that adapt to individuals without demanding large per-user labelled datasets are therefore key to enabling practical well-being sensing with smart devices.

Smart eyewear is becoming a practical platform for continuous, context-aware health and well-being sensing. Because the frame sits stably on the face and head, it can host sensors that observe facial-muscle activity and head motion without a camera pointed at the wearer, sidestepping the positioning and privacy problems that limit camera-based facial analysis on wearables. Optomyographic (OMG) glasses such as OCOsense [1] use non-contact optical sensors to measure skin movement over major facial muscles, and have been shown to support facial expression recognition (FER) for affect and well-being monitoring [15, 21].

Despite these advances, FER systems for smart glasses—across camera [26], Electromyography (EMG) [4, 10], Electrooculography (EOG) [25], acoustic [18, 29], capacitive [17], and OMG sensing [15, 21]—are predominantly fully supervised. They typically extract hand-crafted features for classical machine learning classifiers or train deep networks end-to-end, in both cases depending on labelled expression data. Because facial and expression morphology varies across users, models trained on one cohort transfer imperfectly to new, unseen users—the person-independent domain-shift problem in wearable sensing [11]. Collecting large labelled datasets per deployment is impractical, motivating methods that learn useful representations from unlabelled data. This limitation is shared by the OMG-based OCOsense systems on which we perform our experiments [1, 15, 21].

We study Self-Supervised Learning (SSL) as a way to reduce the labelling burden for FER. SSL aims to learn transferable representations from large amounts of unlabelled data through automatically generated supervisory signals. We pretrain a Vision-Transformer (ViT) encoder on unlabelled smart-glass data with a masked-autoencoder (MAE) objective, then finetune it for FER under a person-independent cross-validation. Our primary research question is: can self-supervised pretraining improve OMG-based FER in label-scarce settings?

Our contributions are the following: (i) We present, to our knowledge, the first self-supervised pretraining approach for FER on OMG smart glasses; and (ii) we characterize label-efficiency by varying the labelled-data budget, while comparing our SSL-pretrained approach against supervised training from scratch.

Related Work

FER using smart glasses. Smart-eyewear facial sensing has been explored using cameras, EMG, EOG, active acoustics, and capacitive sensors [4, 10, 17, 18, 25, 26, 29]. These modalities offer different trade-offs in privacy, skin contact, robustness, and sensing range. However, none of these studies investigates SSL pretraining for label-efficient FER.

Researchers have explored the use of OMG-based OCOsense glasses for FER [15]. Their feature-based approach extracts statistical descriptors from OMG sensor data and trains a Random Forest classifier to recognize four posed expressions and a neutral state. Using leave-one-subject-out cross-validation on data from 27 participants, the authors report 93% accuracy and a macro F1 score of 0.90. They further show that head movement does not significantly degrade sensing performance, whereas glasses placement has a noticeable effect. A subsequent study [21] explored end-to-end deep learning for FER using OCOsense glasses. The authors considered six classes: the same five with an additional “other” category including eating, speaking, and bruxism (teeth grinding). They compared seven convolutional architectures together with the ConvBoost augmentation framework [27] on data from 55 participants. Their best performing model, Attention ConvLSTM with ConvBoost, achieved a macro F1 score of 0.78. The authors highlight that the inclusion of the “other” class and the larger cohort substantially increase task difficulty compared with the original five-class posed setting.

SSL for wearable sensing. Researchers have recently explored the use of SSL for wearable sensing [19], e.g. masked-autoencoder pretraining (MAE) [12] on unlabelled wearable data [24], and a growing line of work builds foundation models over large heterogeneous physiological data, from wearables to modality-specific encoders such as EMG [6, 9, 14, 24]. These studies show that SSL representations transfer across downstream tasks and can reduce labelled-data requirements, including for affect and emotion recognition from wearable physiological signals [7, 23]. However, most existing efforts focus on wrist- and body-worn signals, while, to the best of our knowledge, no work has explored SSL pretraining for facial sensing with on-glasses physiological sensors.

Device and Data

OCOsense smart glasses. We use the OCOsense smart glasses [1]. Figure 1 illustrates the OCOsense glasses and the placement of the facial-sensing sensors on the glasses frame. The device carries non-contact optical sensors that measure skin movement over the frontalis, zygomaticus, corrugator, orbicularis, and temple regions, as well as a 9-axis IMU. All sensors have a sampling rate of 50 Hz, with the OMG and proximity sensors operating 4 to 30 mm above the skin. For FER, we exclude the magnetometer and use a total of 30 sensor channels, comprising 6 IMU channels (3-axis accelerometer and 3-axis gyroscope), 6 Euler-angle channels (sine and cosine representations of roll, pitch, and yaw), 12 navigation channels (x/y displacement at the left/right cheek, brow, and temple), and 6 proximity channels corresponding to the six sensing locations. These signals form the multichannel windows used throughout our pretraining and FER experiments.

OCOsense OMG smart glasses and their sensors

Figure 1. OCOsense OMG smart glasses and their sensors.

Signal preprocessing. Each stream is filtered, standardized per channel using session-level statistics, and segmented into fixed windows (50% overlap for MAE pretraining, 90% for FER, with majority-vote labels).1 At 50 Hz the pretraining and FER windows are the (500, 30) and (50, 30) encoder inputs of Section 4.

Pretraining dataset: CLUES. We pretrain on unlabelled OCOsense recordings from the CLUES dataset, collected from 52 participants across two sites [16]. The CLUES dataset contains OCOsense recordings in a lab-controlled environment, where users performed cognitive-load tasks alternating with relaxed breaks, eliciting naturally occurring facial and head movements. After preprocessing, CLUES yielded 29,722 overlapping 10 s windows, corresponding to 82.6 hours of model input.

Downstream FER dataset: OCOGEM. For the supervised FER task we used an extended version of the OCOGEM dataset [21], in which 75 participants performed deliberately posed facial expressions (smile, frown, eyebrow raise, and squeezed eyes) while wearing the OCOsense glasses, with a neutral configuration maintained between poses. Expressions were always performed in a fixed order, cued by an instruction video with visual (on-screen emoticons) and auditory (beep) prompts. This protocol yields a labelled dataset spanning variation in expression intensity, duration (1–3 seconds), and head movement. The pretraining corpus and the FER cohort do not share participants.

Method

Our pipeline consists of MAE pretraining followed by supervised FER finetuning. We also consider a supervised from-scratch baseline that uses the same encoder architecture but is trained from scratch on the FER task without MAE pretraining.

Tokenization and encoder. Each input sample is a multichannel sensor window from the smart glasses. Along the channel dimension, channels are grouped by sensor modality: IMU, Euler-angle, navigation-sensor, and proximity-sensor channels. The input sensor window is then partitioned into non-overlapping two-dimensional patches along the time and sensor dimensions. The patch size is (5, 6) along the temporal and channel dimensions, respectively, so channel boundaries align with modality groups. Consequently, each patch contains channels from only a single sensor modality and can be interpreted as a short temporal segment from that modality. All sensor patches are then projected into token vectors using a shared linear projection layer. The tokenization process is illustrated in Figure 2.

Figure 2. Illustration of the sensor arrangement and tokenization process. Multichannel sensor windows are partitioned into non-overlapping patches along the temporal and channel dimensions and projected into patch tokens for the ViT encoder.

The resulting token sequence is processed by a ViT encoder [8]. To preserve the two-dimensional structure of the original window, we apply 2D RoPE [28] in the transformer blocks. We also prepend four learnable register tokens [5] to the patch-token sequence, allowing the model to store and propagate global information that might otherwise be forced into ordinary patch tokens.

Masked-autoencoder pretraining. We use MAE for pretraining [12]: patch tokens are randomly masked independently for each sample. Only the visible tokens, together with the register tokens, are passed through the encoder. The encoded visible tokens are projected into a decoder space, concatenated with learned mask tokens, restored to the original patch order, and processed by a lightweight decoder that reconstructs the original patch contents.

The reconstruction target is the original sensor window, and the loss is computed only over masked patches as:

LMAE=∑imi⋅MSE⁡(p^i,pi)∑imi,mi∈{0,1},(1)\mathcal{L}_{\mathrm{MAE}} = \frac{\sum_i m_i \cdot\operatorname{MSE}(\hat{p}_i,p_i)}{\sum_i m_i}, \qquad m_i \in\{0,1\}, \tag*{(1)}

where pip_i is the original patch, p^i\hat{p}_i is the reconstructed patch, and mi=1m_i = 1 indicates that patch ii was masked.

Data augmentation. During pretraining we apply four augmentations to the encoder input while keeping the reconstruction target as the clean signal, so the model learns representations robust to deployment nuisances. Each augmentation is applied independently with probability p=0.5p = 0.5. Channel switching exploits the bilateral symmetry of the navigation-sensor layout: we exchange the paired left/right navigation channels and flip the sign of their lateral (xx-axis) coordinates, modelling side-dependent sensor discrepancies. Jitter adds i.i.d. Gaussian noise (σ=0.03\sigma= 0.03) to the IMU channels to mimic measurement noise. Scaling multiplies each channel by a per-window factor drawn from N(1,0.022)\mathcal{N}(1,0.02^2), exposing the model to cross-session amplitude differences that survive session-level normalization. Magnitude warping multiplies each channel by a smooth random curve (interpolated from a few control points, magnitude 0.02) that varies slowly around one, approximating slow gain drift over the window.

Finetuning for FER. After pretraining, we discard the MAE decoder and attach a linear classification head to the pretrained ViT encoder. The window representation is obtained by global average pooling over the encoder’s patch-token outputs. During finetuning, the entire model is optimized end-to-end for the five-class FER.

Evaluation protocol. We evaluate our experiments using a person-independent five-fold cross-validation, i.e., data from each participant is present inside a single fold only. In each cross-validation run, one fold is used as the test set. From the remaining four folds, 20% of the participants are held out for validation, and the rest are used for training. Validation data are used for checkpoint selection.

Model Implementation Details. The ViT encoder has an embedding dimension of 128, six Transformer blocks, four attention heads, four register tokens, and 2D RoPE. The decoder uses an embedding dimension of 64, three Transformer blocks, and four attention heads. MAE pretraining and FER training use the (500, 30) and (50, 30) input windows, respectively, both with a patch size of (5, 6). All models are optimized using AdamW optimizer [20] with a batch size of 128. For MAE pretraining, we use a learning rate of 3×10−43 \times10^{-4}, betas (0.9, 0.95), weight decay 10−410^{-4}, gradient clipping of 1.0, and a masking ratio of 75%. For FER training, we use betas (0.9, 0.99), weight decay 10−410^{-4}, gradient clipping of 1.0, label smoothing of 0.05, and head dropout of 0.5. The learning rates are 4×10−44 \times10^{-4} for training the from-scratch baseline and 2×10−52 \times10^{-5} for fine-tuning the MAE-pretrained model, respectively. All models use a learning-rate schedule consisting of 5% linear warmup followed by cosine decay. MAE pretraining is performed for 100 epochs. To ensure comparable optimization across the varying training-set sizes, models in the label-efficiency experiments are trained for a fixed budget of 22,200 optimization steps and validated every 3,700 steps. Sensor-modality ablations use the complete training set and run for six epochs per fold, with validation after every epoch. In both settings, the checkpoint maximizing the sum of validation macro F1 and AUC is used to evaluate on the test set.

Experiments and Results

Label availability analysis

Overall results. To probe label efficiency, we subsample the downstream training data within each person-independent fold to 5%, 10%, 20%, 40%, 80% and 100%. Sampling is performed within each user-label group so that the reduced training sets remain balanced across participants and expression classes; validation and test sets are left intact.

At each labelled-data budget, we compare the MAE-pretrained model with the supervised from-scratch baseline. We also report the results from a random baseline, which samples uniformly the five expression classes (averaged across five seeds).

Figure 3 shows the FER F1-score across labelled-data budgets, and Table 1 reports the corresponding accuracy, macro F1, AUC, and MCC. The MAE-pretrained model consistently outperforms the supervised from-scratch baseline across all metrics and budgets, with the largest gain at the 5% budget.

Figure 3. FER data efficiency comparison between MAE-pretrained model and from-scratch baseline.

DataAccF1AUCMCC
FSPretrainFSPretrainFSPretrainFSPretrain
5%0.868 ± 0.0040.898 ± 0.0060.802 ± 0.0060.845 ± 0.0080.945 ± 0.0060.977 ± 0.0030.768 ± 0.0090.818 ± 0.010
10%0.883 ± 0.0040.903 ± 0.0040.823 ± 0.0080.851 ± 0.0050.968 ± 0.0020.978 ± 0.0020.792 ± 0.0090.827 ± 0.006
20%0.887 ± 0.0050.908 ± 0.0050.829 ± 0.0060.859 ± 0.0070.971 ± 0.0020.980 ± 0.0020.799 ± 0.0080.836 ± 0.008
40%0.892 ± 0.0040.910 ± 0.0050.837 ± 0.0060.862 ± 0.0080.975 ± 0.0030.980 ± 0.0010.810 ± 0.0080.840 ± 0.009
80%0.893 ± 0.0050.911 ± 0.0060.836 ± 0.0080.863 ± 0.0080.973 ± 0.0030.980 ± 0.0020.808 ± 0.0100.841 ± 0.010
100%0.894 ± 0.0040.912 ± 0.0050.839 ± 0.0060.865 ± 0.0080.974 ± 0.0030.981 ± 0.0020.810 ± 0.0060.843 ± 0.009

Table 1. FER data efficiency comparison between the from-scratch baseline and the MAE-pretrained model.

Label availability analysis on individual facial expressions. We next analyze how label availability affects the different facial expression classes. For each model and label budget, we sum the confusion-matrix counts across the five participant-disjoint test folds. Because these test folds jointly cover the complete dataset, each sample contributes exactly once to the aggregated confusion matrix. After row normalization, Figure 4 reports the element-wise difference

Figure 4. Data-scaling effect on class-level predictions. The matrices show 100%-minus-5% confusion differences for the pretrained model and the from-scratch model. Rows denote ground-truth labels and columns denote predicted labels.

CM100%−CM5%CM_{100\%}-CM_{5\%} for the MAE-pretrained model and the from-scratch baseline. For both models, the larger budget primarily increases the diagonal entries while reducing off-diagonal confusions, indicating improved class separation. The one exception is Smile for the from-scratch model, whose diagonal entry decreases slightly, by 0.7 percentage points, from the 5% to the 100% budget. The largest improvements are observed for the more expressive classes, particularly Eyebrow Raise and Frown. For the MAE-pretrained model, additional labels mainly reduce expressive samples being misclassified as Neutral, whereas the from-scratch model exhibits larger changes in the Neutral class as well as confusions among Frown, Eyebrow Raise, and Squeezed Eyes. The relatively smaller improvement for the Neutral class in the pretrained model suggests that the 5%-label model already learns a reliable representation for neutral expressions, allowing the additional labels to focus primarily on refining the more challenging expressive classes.

Pretraining analysis on individual facial expressions. We further investigate how MAE pretraining affects class-level predictions at two labelled-data budgets, 5% and 100%. Using the same fold aggregation and normalization procedure, Figure 5 reports the element-wise difference CMpretrained−CMscratchCM_{\mathrm{pretrained}} - CM_{\mathrm{scratch}} at each budget. At the 5% label budget, the improvement on the Neutral class provided by pretraining is even larger than the improvement obtained by increasing the from-scratch model from 5% to 100% labelled data. Moreover,

Figure 5. Class-level effect of MAE pretraining at fixed labelled-data budgets. Each matrix shows the confusion difference between the pretrained model and the from-scratch model at the same budget. Rows denote ground-truth labels and columns denote predicted labels.

MAE pretraining yields clear classification improvements for Neutral, Frown, Eyebrow Raise, and Squeezed Eyes, while substantially reducing several off-diagonal confusions. These results suggest that pretraining is most beneficial when labelled data are scarce, as it provides a feature representation that already captures much of the underlying expression structure before supervised FER training. At the 100% label budget, the gains from pretraining mainly arise from reducing samples from other expression classes being misclassified as Neutral.

Sensor-modality Ablation

We perform ablation studies to understand which modality impacts FER performance the most. For each sensor-modality combination in Table 2, we train a separate supervised from-scratch baseline using only the selected input channels.

InputAccF1AUCMCC
∗ Baseline0.200±0.0000.162±0.0000.500±0.0000.000±0.000
IMU0.667±0.0060.373±0.0090.786±0.0050.343±0.011
Euler0.645±0.0060.305±0.0060.730±0.0090.254±0.012
Prox0.750±0.0150.611±0.0170.885±0.0110.544±0.024
Nav0.872±0.0090.805±0.0130.965±0.0040.771±0.015
IMU + Euler0.662±0.0090.388±0.0070.791±0.0040.345±0.011
Prox + Nav0.888±0.0060.833±0.0080.972±0.0030.800±0.011
All0.892±0.0050.836±0.0070.970±0.0030.808±0.009

Table 2. Sensor-modality ablation for FER. Navigation and proximity channels carry most of the discriminative signal.

We report in Table 2 the cross-validation performance, in terms of accuracy, F1-score, AUC, and MCC, across different sensor modalities. A random baseline is also included for reference. We find that the optical facial-sensing channels lead to higher performance: navigation is the single modality with the highest results, and when combining navigation and proximity sensors we find that the performance is very similar to that obtained when using all modalities. IMU and Euler-angle features lead to lower performance when used alone: we speculate that gross head motion and orientation are not sufficient alone for FER. Nonetheless, using all sensors gives slightly higher accuracy, F1, and MCC than proximity and navigation alone (which retains a marginally higher AUC), suggesting that motion and orientation features potentially provide complementary context in a limited amount of cases.

Discussion

Why label efficiency matters here. Personalized affect sensing on smart eyewear must adapt to new wearers whose expression morphology was never seen during training. Gathering many labelled expressions from every new user is impractical, so the operative question is not only “how accurate is the model with all labels” but “how little labelled data does it need to become useful.” Our results indicate that MAE pretraining shifts the accuracy-vs-class curve upward, with the clearest benefit when labels are scarce.

Generalization across users and tasks. Both the MAE-pretrained model and the from-scratch baseline remain substantially above random baseline under participant-disjoint evaluation, even when trained with only 5% of the labelled data. This indicates that OMG signals contain expression-related patterns that generalize to unseen users rather than relying only on participant-specific characteristics. A notable finding is that the benefits of SSL pretraining transfer across both tasks and participant cohorts. The CLUES recordings were not collected for FER, and the pretraining and downstream datasets contain disjoint participants. Nevertheless, MAE pretraining on CLUES consistently improves FER on unseen users. This suggests that pretraining captures reusable facial-sensor dynamics instead of features specific to a particular task or cohort. In practice, useful FER representations may therefore be learned from unlabelled recordings collected during other smart-glass activities, reducing the need to label large amounts of expression data.

Positioning against supervised state-of-the-art. As mentioned in § 1, researchers have created end-to-end fully supervised approaches to perform FER using OCOsense. The difference in reported FER performance between our approach and their state-of-the-art results can be attributed to differences in experimental protocols. For example, we had a different dataset and user pool compared to Kiprijanovska et al. [15], who trained and validated over 27 participants while we have data from 75 users. Moreover, Marinova et al. [21] performed FER over six classes, i.e., they included an “other” class, which we did not use. Indeed, we position our contribution as orthogonal to existing work, i.e., we focus our work on investigating the impact of label availability more than creating an end-to-end state-of-the-art system.

Relationship to personalization. Prior OCOsense work suggested SSL as a promising approach for addressing person-specific variation [15]. Our person-independent results provide initial support for this direction: MAE pretraining improves FER on unseen users relative to an otherwise identical from-scratch model, with the largest benefit when labelled data are scarce. Cross-user generalization and personalization are complementary in this setting. A pretrained encoder can provide a general representation of facial-sensor dynamics, which could subsequently be adapted to a new wearer using a small amount of labelled calibration data. It could also support continual adaptation using unlabelled recordings collected during regular use. Such adaptation may help account for individual facial morphology, differences in glasses fit and placement, device re-mounting, and changes across recording sessions. Although we do not explicitly personalize or adapt models in this work, future studies should quantify per-user improvements, determine how much calibration data are required, and evaluate whether continual adaptation remains stable over extended deployments.

Privacy, ethics, and responsible deployment. Although OCOsense does not use a camera or microphone, facial-expression inferences remain sensitive personal data. Deployment therefore requires informed and revocable consent, clear purpose limitation, and privacy-preserving processing where feasible. Performance should also be evaluated across demographic and cultural groups because expression morphology and display norms vary between users. Finally, predicted expressions should be treated as behavioral signals rather than direct measurements of emotional state, particularly in workplace or health-related applications.

Limitations and Future Work

Acted versus real-life expressions. The labelled expressions in our training data are acted, i.e., posed, voluntary facial movements performed on cue under a controlled protocol. Acted expressions are typically more exaggerated and stereotyped than spontaneous expressions [22] and are often recorded as clearly segmented, isolated expression episodes. This also makes the FER task comparatively easy for these sensors (high accuracy even at 5% labels), which likely compresses the dynamic range of the label-efficiency effect we observe. How well label-efficient FER trained on acted expressions translates to real-life, in-the-wild use therefore remains an open question that must be tested directly—ideally on spontaneous expressions collected during everyday activities [13], across diverse users and sessions, and over multiple days to capture device re-mounting and day-to-day variation.

Other limitations. Our evaluation is limited to a single SSL objective and backbone, namely MAE pretraining with a ViT encoder. This choice allows us to isolate the effect of pretraining on label efficiency, but it leaves open whether similar gains would be obtained with other SSL paradigms, such as contrastive or joint-embedding predictive objectives, or with alternative architectures, including the convolutional and recurrent models used by prior supervised FER systems. In addition, the augmentation magnitudes and per-augmentation application probabilities were not tuned, and we did not ablate the individual contribution of each augmentation to downstream FER performance.

Conclusion

We presented a self-supervised approach to facial expression recognition on OMG smart glasses, using masked-autoencoder pretraining over the glasses’ facial-sensing channels followed by person-independent finetuning. Across labelled-data budgets from 5% to 100%, pretrained models consistently outperform from-scratch models, demonstrating substantially improved label efficiency. Remarkably, the MAE-pretrained model trained with only 5% of the labelled data already achieves better performance than the from-scratch model trained with the entire labelled dataset. Moreover, we also investigate which sensor-modality combination leads to higher performance in FER. We find that the proximity and navigation channels alone perform on par with using all sensor modalities: all sensors give slightly higher accuracy, F1, and MCC, while proximity and navigation give a slightly higher AUC. These results can guide smart eyewear system design. Overall, we see self-supervised, label-efficient FER as a practical building block for next-generation smart eyewear health and well-being systems.

Acknowledgments

This work is supported by the Swiss National Science Foundation (SNSF) through the grant PZ00P2_216405 for the project “XAI-PAC”.

References

  1. [1]James Archer, Ifigeneia Mavridou, Simon Stankoski, M. John Broulidakis, Andrew Cleal, Piotr Walas, Mohsen Fatoorechi, Hijriian Gjoreski, and Charles Nduka. 2023. OCOsense smart glasses for analyzing facial expressions using optomyographic sensors. IEEE Pervasive Computing 22, 3 (2023), 18–26.DOI
  2. [2]Daniel Arias, Shekhar Saxena, and Stéphane Verguet. 2022. Quantifying the global burden of mental disorders and their economic value. EClinicalMedicine 54 (2022).DOI
  3. [3]M John Broulidakis, Ivana Kiprijanovska, Liberty Severs, Simon Stankoski, Martin Gjoreski, Ifigeneia Mavridou, Hristijan Gjoreski, Sophia Cox, Daisy Bradwell, James M Stone, et al. 2023. Optomyography-based sensing of facial expression derived arousal and valence in adults with depression. Frontiers in Psychiatry 14 (2023), 123433.DOI
  4. [4]Ho-Seung Cha, Seong-Jun Choi, and Chang-Hwan Im. 2020. Real-time recognition of facial expressions using facial electromyograms recorded around the eyes for social virtual reality applications. IEEE Access 8 (2020), 62065–62075.DOI
  5. [5]Timothée Darcet, Maxime Oquab, Julien Mairal, and Piotr Bojanowski. 2024. Vision transformers need regex. In International conference on learning representations, Vol. 2024. 2632–2652.arxiv.org/abs/2309.16588
  6. [6]Federico Del Pup and Manfredo Atzori. 2023. Applications of Self-Supervised Learning to Biomedical Signals: A Survey. IEEE Access 11 (2023), 144180–144203. doi:10.1109/ACCESS.2023.3344531
  7. [7]Vijay Dossaryake, Sachith Seneviratne, Rajib Rana, Elliott Wen, Tharindu Kaluarachchi, and Suranga Nanayakkara. 2022. SigRep: Toward Robust Wearable Emotion Recognition With Contrastive Representation Learning. IEEE Access 10 (2022), 18105–18120. doi:10.1109/ACCESS.2022.3149509
  8. [8]Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xi-ahoua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, Jakob Uszkoreit, and Neil Houlsby. 2021. An image is worth 16x16 words: Transformers for image recognition at scale. In International Conference on Learning Representations (ICLR).
  9. [9]Matteo Fasulo, Giusy Scapone, Thorir Mar Ingolfsson, Yawei Li, Luca Benini, and Andrea Cossettini. 2025. TinyMyo: a Tiny Foundation Model for Flexible EMG Signal Processing at the Edge. arXiv:2512.15729 [eess.SP] doi:10.48550/arXiv.2512.15729DOI
  10. [10]Martin Gjoreski, Ivana Kiprijanovska, Simon Stankoski, Ifigeneia Mavridou, M. John Broulidakis, Hristijan Gjoreski, and Charles Nduka. 2022. Facial EMG sensing for monitoring affect using a wearable device. Scientific Reports 12, 1 (2022), 16876.DOI
  11. [11]Yunjo Han, Panyu Zhang, Minseo Park, and Uichin Lee. 2024. Systematic evaluation of personalized deep learning models for affect recognition. Proceedings of the ACM on Interactive, Mobile, Wearable and Ubiquitous Technologies 8, 4 (2024), 1–35.DOI
  12. [12]Kaiming He, Xinlei Chen, Saining Xie, Yanghao Li, Piotr Dollár, and Ross Girshick. 2022. Masked autoencoders are scalable vision learners. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR). 16000–16009.arxiv.org/abs/2111.06377
  13. [13]Matthias Jammot, Björn Braun, Paul Streli, Rafael Wampfler, and Christian Holz. 2025. egoEMOTION: Egocentric Vision and Physiological Signals for Emotion and Personality Recognition in Real-world Tasks. In Advances in Neural Information Processing Systems (NeurIPS) Datasets and Benchmarks Track.
  14. [14]Justin Khasentino, Anastasiya Belyaeva, Xin Liu, Zhun Yang, Nicholas A Furlotte, Chace Lee, Erik Schenck, Yojan Patel, Jian Cui, Logan Douglas Schneider, et al. 2025. A personal health large language model for sleep and fitness coaching. Nature Medicine 31, 10 (2025), 3394–3403.DOI
  15. [15]Ivana Kiprijanovska, Simon Stankoski, M. John Broulidakis, James Archer, Mohsen Fatoorechi, Martin Gjoreski, Charles Nduka, and Hristijan Gjoreski. 2023. Towards smart glasses for facial expression recognition using OMG and machine learning. Scientific Reports 13, 1 (2023), 16043.
  16. [16]Ana Krstevska, Shivalika Goyal, Linda Fiorini, Francesco Bombassei De Bona, Daniil Kirilenko, Mohan Li, Sebastian Trojer, Tomi Božak, Zoja Anžur, Mitja Luštřek, Hristijan Gjoreski, Marc Langheinrich, Gašper Slapničar, and Martin Gjoreski. 2026. CLUES: Cognitive Load Understanding through Experimental Sensing Dataset. Proceedings of the ACM on Interactive, Mobile, Wearable and Ubiquitous Technologies 10, 3, Article 116 (Sept. 2026), 33 pages. doi:10.1145/3831656DOI
  17. [17]Francesco Latino, Matteo Rossi, Andrea Costanzo Palmisciano, Andrea Farabbi, Athul Krishnan, Niccolò Antonello, Luca Mainardi, and Pietro Cerveri. 2026. Turning glasses into sensors: capacitive facial expression recognition under frame geometry constraints. Physiological Measurement 47, 5 (2026), 055011.DOI
  18. [18]Ke Li, Ruidong Zhang, Siyuan Chen, Boao Chen, Mose Sakashita, François Guimbretière, and Cheng Zhang. 2024. Eyecho: Continuous and low-power facial expression tracking on glasses. In Proceedings of the 2024 CHI Conference on Human Factors in Computing Systems. 1–24.DOI
  19. [19]Aleksеj Logacjov. 2024. Self-supervised learning for accelerometer-based human activity recognition: A survey. Proceedings of the ACM on Interactive, Mobile, Wearable and Ubiquitous Technologies 8, 4 (2024), 1–42.DOI
  20. [20]Ilya Loshchilov and Frank Hutter. 2017. Decoupled weight decay regularization. arXiv preprint arXiv:1711.05101 (2017).arxiv.org/abs/1711.05101
  21. [21]Matea Marinova, Emiliya Chona, Andrej Kotеvski, Borjan Sazdov, Ivana Kiprijanovska, Simon Stankoski, Martin Gjoreski, Charles Nduka, and Hristijan Gjoreski. 2025. Deep learning for facial expression and human activity recognition using smart glasses. IEEE Access 13 (2025).DOI
  22. [22]Brais Martinez, Michel F. Valstar, Bihan Jiang, and Maja Pantic. 2019. Automatic Analysis of Facial Actions: A Survey. IEEE Transactions on Affective Computing 10, 3 (2019), 325–347. doi:10.1109/TAFFC.2017.2731763
  23. [23]Kevin G. Montero Quispe, Daniel M. S. Utiyama, Eulanda M. dos Santos, Horácio A. B. F. Oliveira, and Eduardo J. P. Souto. 2022. Applying Self-Supervised Representation Learning for Emotion Recognition Using Physiological Signals. Sensors 22, 23 (2022), 9102. doi:10.3390/s22239102
  24. [24]Girish Narayanaswamy, Xin Liu, Kumar Ayush, Yuzhe Yang, Xuhai Xu, Shun Liao, Jake Garrison, Shyam Tailor, Jacob Sunshine, Yun Liu, et al. 2025. Scaling wearable foundation models. In International Conference on Learning Representations (ICLR), Vol. 2025. 59382–59413.
  25. [25]Soha Rostaminia, Alexander Lamson, Subhransu Maji, Tauhidur Rahman, and Deepak Ganesan. 2019. W! nce: Unobtrusive sensing of upper facial action units with ego-based eyewear. Proceedings of the ACM on Interactive, Mobile, Wearable and Ubiquitous Technologies 3, 1 (2019), 1–26.DOI
  26. [26]Muhammad Sajjad, Fath U Min Ullah, M. H. Ullah, Georgia Christodoulou, Faouzi Alaya Cheikh, Mohammad Hijji, Khan Muhammad, and Joel JPC Rodrigues. 2023. A comprehensive survey on deep facial expression recognition: challenges, applications, and future guidelines. Alexandria Engineering Journal 68 (2023), 817–840.DOI
  27. [27]Shuai Shao, Yu Guan, Bing Zhai, Paolo Missier, and Thomas Plötz. 2023. Conv-Boost: Boosting ConvNets for sensor-based activity recognition. Proceedings of the ACM on Interactive, Mobile, Wearable and Ubiquitous Technologies 7, 2 (2023), 1–21.DOI
  28. [28]Jianlin Su, Murtadha Ahmed, Yu Lu, Shengfeng Pan, Wen Bo, and Yunfeng Liu. 2024. Roformer: Enhanced transformer with rotary position embedding. Neurocomputing 568 (2024), 127063.arxiv.org/abs/2104.09864
  29. [29]Wentao Xie, Qian Zhang, and Jin Zhang. 2021. Acoustic-based upper facial action recognition for smart eyewear. Proceedings of the ACM on Interactive, Mobile, Wearable and Ubiquitous Technologies 5, 2 (2021), 1–28.DOI

Paper details

vlnc:W_rgkpfup9
Version
v1 · latest
Published
Fresh — added to the Hub within the last 30 days
Added
License
CC BY 4.0
Usage

views · 0

downloads · 0

A view counts once per browser per day, a download each time the PDF is requested.

  • Computer Science

Not recorded for this paper

View statement
Looking for related papers…
Contents