betaFeedback

WorksComputer Science

Exploring Speaker-Aware Engagement Modeling for Children with Autism Spectrum Disorder

Abstract

Prior work on engagement modeling in children with autism spectrum disorder (ASD) has demonstrated substantial individual variability in behavioral features, particularly in visual modalities. In this work, we extend the literature to acoustic features in naturalistic interactions, where multiple speakers and atypical interaction patterns complicate speaker-specific analysis. We investigated whether speaker diarization improved acoustic engagement modeling during month-long, in-home socially assistive robot (SAR) interventions with seven children with ASD. We trained classical machine learning models using acoustic features extracted from diarized child and parent/caregiver audio and compared their performance with models trained on undiarized audio. Diarized audio consistently yielded higher area under the receiver operating characteristic curve (AUROC) values, although these differences were not statistically significant. More importantly, the relative contributions of child and caregiver acoustic features varied across participants, indicating that the acoustic information associated with engagement was not consistent across children. These findings extend evidence of participant-level variability in ASD engagement modeling to the acoustic domain and demonstrate the feasibility of isolating speaker-specific vocalizations for studying engagement in complex, naturalistic interactions.

Introduction

Modeling engagement behavior of children with autism spectrum disorder (ASD) in in-home contexts is challenging because of variability in both naturalistic environments and behavioral expressions of engagement. Prior work has explored multimodal approaches to engagement modeling [1, 2]; however, in naturalistic, in-home settings children often interact1

with caregivers and others in the environment [3], and such interactions influence engagement. Therefore, engagement modeling requires consideration of both child behavior and the surrounding social context [4].

Prior work on automated engagement modeling for children with ASD has highlighted substantial variability in behavioral expressions of engagement [1, 2]. Personalized and multimodal approaches have used visual, audio, and motion cues to estimate social engagement during child-robot interactions, with vocalizations providing complementary information [5, 1, 2]. However, naturalistic interactions often involve multiple speakers, complicating the attribution of acoustic information to the target individual [2]. Speaker diarization has been used in ASD settings to distinguish child vocalizations from clinicians, caregivers, and other sources in noisy recordings [6], demonstrating the feasibility of isolating child speech and motivating the use of speaker-specific acoustic features for engagement modeling.

Informed by prior work demonstrating substantial individual variability in ASD engagement modeling, we extend the state-of-the-art by focusing on acoustic engagement modeling in naturalistic interactions. We investigated the effect of speaker diarization on acoustic engagement modeling using data from a month-long, in-home socially assistive robot (SAR) intervention with seven children with ASD [7]. We trained multiple classical machine learning models using acoustic features extracted from diarized child and caregiver audio and compared their performance with models trained on features extracted from undiarized audio. Diarized audio consistently yielded higher area under the receiver operating characteristic curve (AUROC) values, although these differences were not statistically significant. We further examined the relative contributions of child and caregiver acoustic features to engagement classification and found that these contributions varied across participants. Together, these findings extend evidence of individual variability in ASD modeling to the acoustic domain and characterize how speaker-specific acoustic information contributes to engagement modeling in naturalistic interactions.

Data

The data used in this work were collected during a month-long, in-home socially assistive robot (SAR) intervention with seven children with autism spectrum disorder (ASD), ages 3–7 years [7]. Participants interacted with a tabletop SAR and completed space-themed mathematics games on a touchscreen tablet while the robot provided feedback and encouragement verbally and through screen-based facial expressions and body movement. The collected data included webcam video and audio, and game-performance data, comprising approximately 21 hours of interactions across seven participants and multiple sessions. Study participants and their families determined session frequency; as a result, the dataset includes from as few as four sessions to as many as 15 sessions for a participant. We address issues of data and class imbalance during model training and evaluation (§ 3.3).

In all sessions, caregivers were present in the background. The dataset had been human-annotated for binary engagement in prior work by Jain et al. [1]. Each video frame was labeled as engaged (1) if the child participant was listening or responding to the SAR’s prompts, interacting with the tablet, or seeking guidance from others in the room. All other behaviors were labeled as disengaged (0). We used these annotations in our modeling, described next.

Methodology

Diarization Model Evaluation

To evaluate speaker diarization, two authors independently annotated 14 audio files, representing approximately 20% of the dataset, using ELAN1. Inter-annotator reliability was evaluated on 20% of this subset using correctness, defined as the proportion of speech correctly labeled, and diarization error rate (DER), which accounts for missed speech, false alarms, and speaker confusion. Using one annotator as reference, the annotations achieved an average DER of 10.0% and correctness of 96.8%.

ModelDER (%)FA (%)MS (%)SC (%)
NeMo Neural50.98±10.8922.36±6.7912.31±4.4616.30±5.65
Pyannote29.99±9.316.69±2.684.76±3.3318.54±7.11

Table 2. Comparison of diarization models shown with 95% confidence intervals.

We considered four speaker diarization models: NVIDIA NeMo End-to-End Sortformer [8, 9], NVIDIA NeMo Cascaded Clustering Diarizer [8], NVIDIA NeMo Cascaded Neural Diarizer [8, 10], and Pyannote [11]. Because speaker overlap was frequent, we excluded the NeMo Cascaded Clustering Diarizer as it did not explicitly predict overlapping speech. We also excluded the NeMo End-to-End Sortformer because it did not adequately distinguish child vocalizations from the robot’s. We therefore evaluated Pyannote and the NeMo Cascaded Neural Diarizer in detail.

Both models were evaluated using original and denoised audio; denoising was performed using the model described by Défosséz et al. [12]. We evaluated automatic and predetermined speaker-count configurations and compared all configurations using DER. For the NeMo Cascaded Neural Diarizer, we additionally evaluated combinations of voice activity detection and speaker embedding models.

The best-performing Pyannote configuration achieved a mean DER of 29.9±9.3%29.9 \pm9.3\%, compared with 50.9±10.9%50.9 \pm10.9\% for the best-performing NeMo Cascaded Neural Diarizer configuration. Pyannote produced significantly lower DER, t(13)=−5.31t(13) = -5.31, p<.001p < .001, and was selected for subsequent analyses2. The selected configuration used original audio and automatically estimated the number of speakers. DER was calculated using non-overlapping speech to assess speaker identification and separation without the uncertainty associated with overlapping segments. This configuration eliminated the need for manual speaker counting and additional denoising, avoiding potential alterations to acoustic features.

Despite naturalistic background noise and frequent speaker overlap, the selected Pyannote configuration achieved a mean DER of 29.9±9.3%29.9 \pm9.3\%. This result was comparable to the DIHARD III Track 2 baseline DER of 19.37–28.10% for diarization from raw audio [13]. Xu et al. [14] reported a DER of 34.6% for Pyannote on child-adult interactions involving children with ASD, excluding overlapping speech, which decreased to 26.4% after fine-tuning. The DER obtained in this work, therefore, fell between their pretrained and fine-tuned Pyannote results on a related child-adult interaction dataset.

Data Preprocessing

Following speaker diarization, we segmented the continuous recordings into speaker-specific audio clips using the temporal boundaries and speaker labels produced by Pyannote, as shown in Figure 1. Overlapping speech was excluded to retain only non-overlapping audio for each speaker This resulted in a substantial decrease of available audio data. Although this is a disadvantage of isolating speaker-specific acoustic features in naturalistic environments, it was necessary in order to get acoustic features specific to only one speaker. We identified the speaker labels corresponding to the child participant and caregivers involved in the primary activity and retained only their audio. The resulting clips were segmented into one-second intervals following Jain et al. [1], and segments shorter than 60 ms were discarded [15]. Clip statistics are reported in Table 1.

ParticipantUndiarized audioDiarized audio
EngagedDisengagedEngaged (participant)Engaged (caregiver)Disengaged (participant)Disengaged (caregiver)
P5551911031457683235130
P7702863275962056161660
P959826689471856684463
P119273477023878831618234
P125388380313061597899935
P1731813655684928482475
P188028422511911076586711

Table 1. Number of audio clips for the different data configurations. Data between engaged and disengaged clips is slightly imbalanced, depending on the participant, and is more imbalanced for diarized audio, with engaged clips being more frequent than disengaged clips.

Audio pre-processing, handling overlapping speaker segments, multiple speakers within an engagement window, and clipping of audio segments

Figure 1. Audio pre-processing, handling overlapping speaker segments, multiple speakers within an engagement window, and clipping of audio segments.

We extracted acoustic features from the speaker-specific clips and corresponding undiarized audio using the extended Geneva Minimalistic Acoustic Parameter Set (eGeMAPS), which comprises 88 features related to acoustic properties, including pitch, voice quality, and loudness [15]. eGeMAPS also includes cepstral parameters relevant to modeling affective states [15]. An additional speaker id variable was added along with the eGeMAPS features to distinguish between features from child and caregiver audio. The duration of the clip was also added as input to the models.

Engagement Modeling and Evaluation

We trained seven different machine learning models using their default hyperparameters (following Jain et al. [1]): neural network with one hidden layer with 100 nodes (NN), k-nearest neighbors, gradient boosted decision trees (DTs), random decision tree forest, naive Bayes, support vector machine (SVM), and logistic regression. We accounted for class imbalance (Table 1) by weighing engaged/disengaged classes by the inverse of their frequency in order to penalize incorrect model predictions for the minority disengaged class.

We trained the models based on the personalized model evaluation setup proposed by Shi et al. [16]. In that setup, evaluation was performed on a held-out session of a target participant, while the training set consisted of the available sessions from all other participants, together with the earlier available sessions of the target participant. This is repeated for all sessions for each participant. Thus, the model is exposed to both population-level patterns learned from other participants and participant-specific patterns learned from the target participant’s earlier sessions. This adoption of a session-based model evaluation strategy also accounts for the temporal nature of the multi-session dataset and avoids potential temporal leakage, since models are evaluated on chronologically subsequent sessions rather than randomly partitioned instances within a session. Since the number of sessions, and consequently the number of audio clips, varied across participants, we report weighted averages for all metrics to place more importance on model performance for participants with more data. We report F1 score, accuracy, and area under the receiver operating characteristic curve (AUROC) metrics. AUROC is primarily used to evaluate model performance given the class imbalance (Table 1).

Results

Across the seven evaluated models, we compared performance using diarized and undiarized audio data. We report the three models with the highest AUROC under either condition: NN, SVM, and gradient boosted DT. Across these models, AUROC ranged from 0.594 to 0.610, indicating modest discrimination between engaged and disengaged clips and aligning with prior findings on the limited contribution of eGeMAPS acoustic features to engagement classification [1]. All three models achieved higher AUROC with diarized audio than with undiarized audio (Figure 2); however, a one-sided Wilcoxon signed-rank test indicated that the difference was not statistically significant (NN: W=16.00W = 16.00, p=0.41p = 0.41, SVM: W=15.00W = 15.00, p=0.47p = 0.47, gradient boosted DT: W=19.0W = 19.0, p=0.23p = 0.23). Thus, diarization consistently corresponded with higher discrimination between engagement states, although the statistical analysis did not establish a significant improvement.

Neural network comparison of diarized and undiarized features

(a) (a) Neural network

SVM comparison of diarized and undiarized features

(b) (b) SVM

Gradient boosted decision tree comparison of diarized and undiarized features

(c) (c) Gradient Boosted Decision Tree

Figure 2. Comparison of three different models trained on diarized versus undiarized eGeMAPs features: (a) neural network, (b) support vector machine, and (c) gradient boosted decision tree.

AUROC complemented the class-specific F1 scores by measuring discrimination across classification thresholds, which was particularly relevant given the class imbalance in the data (Table 1). Engaged-class F1 scores were significantly higher with diarized audio than with undiarized audio for SVM model (p<.05p < .05), whereas disengaged-class F1 scores did not differ significantly for three models (p>.05p > .05). The predominance of engaged clips likely contributed to the higher engaged-class F1 scores, while the lower disengaged-class performance aligned with prior findings on the greater variability of disengaged behaviors [1]. Overall, diarization significantly improved classification of the majority engaged class and consistently increased AUROC, although the AU-

ROC improvement was not statistically significant.

Lastly, for the highest performing model (neural network), we explored the contribution of caregiver-only and child-only acoustic features towards model performance. As shown in Table 3, we see that for five of the child participants (7, 9, 11, 12 and 18), the acoustic features from the child-only speech segments resulted in a higher AUROC compared to caregiver-only speech. For the other two participants (5 and 17), the opposite was the case. Some caregivers were much more involved and prompted the child participants actively, while others were less involved, which may explain why, for some participants, the caregiver’s acoustic features were more predictive of participant engagement compared to participant acoustic features, while for others, the participant’s acoustic features were more predictive. These results demonstrate that, depending on the participant, the importance of the caregiver’s acoustic features can vary.

ParticipantChild SpeakerCaregiver Speaker
AUROCF1 (Engaged)F1 (Disengaged)AccuracyAUROCF1 (Engaged)F1 (Disengaged)Accuracy
p50.6120.8420.1840.7400.6150.7590.2490.661
p70.6340.6530.3740.5780.6240.7460.3200.645
p90.5790.4250.6290.5820.4940.4100.4140.481
p110.6270.6340.4660.5920.6050.8070.2200.703
p120.5560.2710.4920.4200.5430.3860.5110.463
p170.5720.5720.4940.5790.6090.6310.4270.552
p180.6220.7510.2660.6540.5150.7010.2380.581

Table 3. Comparison of model performance using child-speaker and caregiver-speaker audio for all participants.

Conclusion

This work examined the effect of speaker diarization on acoustic engagement classification during in-home SAR interventions with children with ASD. Across the highest-performing models, diarized audio consistently produced higher AUROC values than undiarized audio, although the differences were not statistically significant. Diarization significantly improved engaged-class F1 scores for SVM, whereas disengaged-class performance remained low and did not differ significantly between conditions. We further examined child and caregiver acoustic features separately and found that their contributions to engagement classification varied across participants, highlighting individual variability in the acoustic correlates of engagement. Overall, these findings demonstrated the potential value of speaker-specific acoustic information while emphasizing the importance of individual differences for engagement modeling in naturalistic interactions. Overall, these findings extend prior evidence of individual variability in ASD engagement modeling to the acoustic domain and demonstrate the feasibility and potential value of isolating speaker-specific vocalizations for engagement modeling in complex, naturalistic interactions.

Acknowledgements

Kaleen Shrestha is supported by the National Science Foundation CISE Graduate Fellowships under Grant No. 2313998.

Compliance with Ethical Standards

The data used in this work was previously collected with approval granted by the University of Southern California Institutional Review Board (UP-16-00755).

References

  1. [1]Shomik Jain, Balasubramanian Thiagarajan, Zhonghao Shi, Caitlyn Clabaugh, and Maja J Matarić, “Modeling engagement in long-term, in-home socially assistive robot interventions for children with autism spectrum disorders,” Science robotics, vol. 5, no. 39, pp. eaa z3791, 2020.
  2. [2]Ognjen Rudovic, Jaeryoung Lee, Miles Dai, Björn Schuller, and Rosalind W Picard, “Personalized machine learning for robot perception of affect and engagement in autism therapy,” Science Robotics, vol. 3, no. 19, pp. eaad6760, 2018.
  3. [3]Arodiami Chorianopoulou, Efthymios Tzinis, Elias Iosif, Asimenia Papoulidi, Christina Papailiou, and Alexandros Potamianos, “Engagement detection for children with autism spectrum disorder,” in 2017 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP). IEEE, 2017, pp. 5055–5059.DOI
  4. [4]SunKyoung Kim, Masakazu Hirokawa, Atsushi Funahashi, and Kenji Suzuki, “Parental influence in disengagement during robot-assisted activities: A case study of a parent and child with autism spectrum disorder,” Multimodal Technologies and Interaction, vol. 6, no. 5, pp. 39, 2022.
  5. [5]Hifza Javed, WonHyong Lee, and Chung Hyuk Park, “Toward an automated measure of social engagement for children with autism spectrum disorder—a personalized computational modeling approach,” Frontiers in Robotics and AI, vol. 7, pp. 43, 2020.
  6. [6]Alex Gorodetski, Ilan Dinstein, and Yaniv Zigel, “Speaker diarization during noisy clinical diagnoses of autism,” in 2019 41st Annual International Conference of the IEEE Engineering in Medicine and Biology Society (EMBC). IEEE, 2019, pp. 2593–2596.DOI
  7. [7]Caitlyn Clabaugh, Kartik Mahajan, Shomik Jain, Roxanna Pakkar, David Becerra, Zhonghao Shi, Eric Deng, Rhianna Lee, Gisele Ragusa, and Maja Matarić, “Long-term personalization of an in-home socially assistive robot for children with autism spectrum disorders,” Frontiers in Robotics and AI, vol. 6, pp. 110, 2019.
  8. [8]Oleksii Kuchaiev, Jason Li, Huyen Nguyen, Oleksii Hrinchuk, Ryan Leary, Boris Ginsburg, Jonathan M. Cohen, et al., “Nemo: a toolkit for building ai applications using neural modules,” arXiv preprint arXiv:1909.09577, 2019.arxiv.org/abs/1909.09577
  9. [9]Taejin Park, Ivan Mendennikov, Kunal Dhawan, Weiqing Wang, He Huang, Nithin Rao Koluguri, Krishna C Puvvada, Jagadeesh Balam, and Boris Ginsburg, “Sortformer: A novel approach for permutation-resolved speaker supervision in speech-to-text systems,” in Proceedings of the 42nd International Conference on Machine Learning, Aarti Singh, Maryam Fazel, Daniel Hsu, Simon Lacoste-Julien, Felix Berkenkamp, Tegan Maharaj, Kiri Wagstaff, and Jerry Zhu, Eds. 13–19 Jul 2025, vol. 267 of Proceedings of Machine Learning Research, pp. 48153–48169, PMLR.
  10. [10]NVIDIA Corporation, “Diarization msdd telephonic,” 2023, Version 1.0.1.
  11. [11]Hervé Bredin, “pyannote.audio 2.1 speaker diarization pipeline: principle, benchmark, and recipe,” in Proc. INTERSPEECH 2023, 2023.DOI
  12. [12]Alexandre Défossez, Gabriel Synnaeve, and Yossi Adi, “Real time speech enhancement in the waveform domain,” in Proceedings of Interspeech 2020, 2020, pp. 3291–3295.arxiv.org/abs/2006.12847
  13. [13]The Third DIHARD Speech Diarization Challenge, “The third dihard speech diarization challenge: Results,” 2021.
  14. [14]A. Xu, K. Huang, T. Feng, L. Shen, H. Tager-Flusberg, and S. Narayanan, “Exploring speech foundation models for speaker diarization in child-adult dyadic interactions,” arXiv preprint arXiv:2406.07890, 2024.arxiv.org/abs/2406.07890
  15. [15]Florian Eyben, Klaus R Scherer, Björn W Schuller, Johan Sundberg, Elisabeth André, Carlos Busso, Laurence Y Devillers, Julien Epps, Petri Laukka, Shrikanth S Narayanan, et al., “The geneva minimalistic acoustic parameter set (gemaps) for voice research and affective computing,” IEEE transactions on affective computing, vol. 7, no. 2, pp. 190–202, 2015.DOI
  16. [16]Zhonghao Shi, Thomas R Groechel, Shomik Jain, Kourtney Chima, Ognjen Rudovic, and Maja J Matarić, “Toward personalized affect-aware socially assistive robot tutors for long-term interventions with children with autism,” ACM Transactions on Human-Robot Interaction (THRI), vol. 11, no. 4, pp. 1–28, 2022.

Paper details

vlnc:W_uhsea8um
Version
v1 · latest
Published
Fresh — added to the Hub within the last 30 days
Added
License
CC BY 4.0
Usage

views · 2

downloads · 0

A view counts once per browser per day, a download each time the PDF is requested.

  • Computer Science

Not recorded for this paper

View statement
Looking for related papers…
Contents