betaFeedback

WorksComputer Science

Personalized Visual Support After Automatic Speech Recognition

Abstract

Many spoken-language interfaces stop at producing a transcript. This project examines how a second layer of optional text and pictorial support can be adapted to individual needs. We specify four presentation levels and a behavior-based selection policy driven by replay requests, response latency and recent comprehension. After expert review of the mappings, we will compare fixed and adaptive presentation using counterbalanced spoken instructions. The study is designed to measure comprehension, usability and which modality listeners rely on, not to claim that transcript embellishment uniformly benefits listeners.

Introduction

A transcript is a useful accessibility feature, but it carries almost the same informational organization as the original utterance. The question considered here is whether a listener should always receive the same presentation or be offered an amount of structure that changes with task performance.

Speech recognition now supports practical transcription in many everyday situations, but a transcript does not by itself guarantee comprehension. Large-scale weak supervision has improved robust speech recognition across acoustic conditions [1]. For listeners dealing with rapid instructions, unfamiliar vocabulary or competing demands, the next problem is deciding which information to present and in what form.

The evidence on added text is mixed, so this decision cannot be settled once for everyone. A review of more than one hundred studies of video captioning reports gains in comprehension, attention and memory, particularly for viewers working in a non-native language, although these studies concern video rather than spoken task instructions [2]. Multimedia-learning research cautions against presenting identical printed and spoken streams, because processing capacity in each channel is limited [3]. The expertise reversal effect makes a related point: guidance that helps less knowledgeable learners can lose its value, or become counterproductive, as knowledge increases [4]. Taken together, these findings argue for support that depends on the listener’s current need rather than staying constant.

Reading support studies suggest that sentence segmentation, pictograms and labels need not help every learner equally [5]. Automated text-to-pictogram mapping has been assessed in several languages and can supply candidate visual supports, though its reliability depends on lexical coverage and context [6]. Pictogram norming points the same way: in a study of 1,525 pictograms from the ARASAAC collection, meanings were on average hard to guess unaided yet were rated as closely related to their intended referents, with nouns scoring highest on both measures [7].

This supports pairing pictograms with text, and suggests that action pictograms need particularly careful review. The proposed project concerns the selection of these resources after speech recognition, rather than the recognition algorithm itself.

Differences in language-related processing should not be collapsed into a single severity scale. A variety of impairments were shown across different conditions [8], [9], [10], [11], [12]. Multiple studies made attempts to decode neural markers during language processing in such conditions [13], [14], [15], [16], [17], [18], as well as develop interfaces [19], [20] for patients to communicate [21] and interact with text [22], illustrating that access modality and understanding are distinct problems. Our task thus concerns comprehension of heard instructions, not decoding intended speech.

We address three questions. First, does behavior-adaptive presentation improve comprehension of action order relative to a fixed format? Second, does it do so without offering support that listeners then reject? Third, which source (audio, text or images) do listeners actually use to resolve each task?

System Concept

A spoken instruction will be transcribed and aligned with sentence or clause boundaries. The listener can receive the original transcript, a segmented transcript, a segmented transcript with selected pictograms, or a display emphasizing ordered action steps (Table 1). The acoustic recording will remain available for replay. Figure 1 shows how these parts connect and how observed behavior feeds back into the choice of level.

LevelDisplayAdded over previous levelReason for a separate level
1Original transcriptText aligned with the recordingBaseline; text may help or duplicate the audio
2Segmented transcriptClause or sentence breaksIsolates segmentation from semantic support
3Segmented transcript with pictogramsReviewed icons for actions, objects and locationsVisual cue kept beside the label, since icons alone are often hard to guess
4Ordered action stepsSegments laid out as a numbered sequence in the original wordingTargets the order errors counted in the primary endpoint

Table 1. The four presentation levels.

System flow diagram showing spoken input, transcription, segmentation, pictogram store, display levels, listener behavior, adaptation policy, and profile

Figure 1. System flow: prepared text and pictograms feed one active display level, and observed listener behavior updates that level through the adaptation policy.

Selection will initially depend on observable behavior: replay requests, answers to a small comprehension probe, response times, and user choices. Each trial yields a difficulty event (a replay request, an incorrect probe answer, or a response slower than a threshold calibrated on the training set), an unaided success, or neither. A simple rule will offer a richer display following repeated difficulty and reduce it when the person succeeds without assistance. The reasoning follows ability-based design, where the system adjusts to demonstrated capabilities rather than requiring a fixed user type [23]; the rule is deliberately symmetric, because support that helps at one level of knowledge can become redundant at another [4], [5].

Because comprehension depends on integration across clauses, visually marking every concept may be counterproductive. The construction-integration account of discourse comprehension provides one reason to examine segmentation and semantic support separately [24]. Segmentation has a practical precedent in subtitling: in an eye-tracking study of live respoken subtitles, chunked text reduced the time participants spent reading [25]. Whether it also improves understanding is what Level 2 is meant to test. Icons will be limited to actions, objects and locations after human review for polysemy, a restriction consistent with pictogram norms in which word concreteness and imageability were the factors most strongly correlated with how readily pictograms were guessed and matched to their meanings [7].

A lightweight personalized profile will keep track of preferred aid levels. To prevent overreaction to a single slow response, the policy will change format only after two comparable events and will preserve a manual override. In the terms of Plass and Pawar’s taxonomy of adaptivity [26], the policy thus combines adjustment during a task with a profile that accumulates over longer use. Such conservative adaptation is attractive for an initial prototype, where reliable estimation of individual preferences is more important than algorithmic complexity.

The transcription stage will preserve word-level timing so that text segments remain associated with the original recording. Whisper-style recognizers [1] do not provide reliable word-level timestamps out of the box, but voice-activity detection followed by forced phoneme alignment can supply them [27]. We will develop a conservative sentence and clause segmenter and evaluate it manually on the instructional materials, reporting how often it is uncertain. When segmentation is uncertain, the display will fall back to punctuation-defined units rather than invent a syntactic analysis.

Pictograms will be generated offline, checked by language specialists and stored with their source phrase and sense. The interface therefore tests support selection rather than conflating that test with stochastic changes in image selection.

Instructions will be sampled from everyday procedural domains such as finding a room, preparing an object or following an administrative sequence. The materials will avoid domain expertise and medical advice, and each item will have an independently reviewed answer key. Two recordings of each instruction will provide acoustic variation without introducing uncontrolled differences in content. Listeners may replay the entire recording or a selected clause; replay controls will be available in every condition so that the adaptive system does not gain an unfair advantage merely by offering basic audio access.

Planned Evaluation

The pilot will begin with approximately 24 adult volunteers, deliberately selected to span a range of self-reported reading and listening preferences. Short spoken task instructions will be recorded by two speakers and distributed in matched sets. A fixed-format condition and a behavior-adaptive condition will be compared within participant, with items and presentation order counterbalanced.

Accuracy on action-order questions will be the primary endpoint. Replay count, completion time, requests to remove scaffolds and perceived effort (for example the effort item of the NASA-TLX [28]) are secondary. The adaptation policy will be calibrated on a short training set and evaluated on unseen instructions; mixed-effects analysis with crossed random effects for participants and items, and with random slopes justified by the design [29], will account for repeated items. No improvement is assumed, and estimates from a feasibility sample will be reported with intervals rather than treated as definitive efficacy evidence. The caution is warranted by the sample size: pilot estimates of variability are known to be imprecise [30], and simulation work on parallel-group pilots recommends well over the 24 participants planned here for stable variance estimates [31]. The within-participant comparison removes between-listener variance from the contrast, which helps, but the study remains a feasibility pilot.

We will quantify how often the system selects a support level that the listener subsequently rejects, alongside the usual comprehension score. This is needed to distinguish a true gain from a policy that provides assistance so frequently that the participant feels obliged to dismiss it.

Before participant testing, an expert panel will rate pictogram appropriateness and whether the segmented layout preserves order and scope; the pictogram ratings resemble the translucency judgments used in pictogram norming [7]. Disagreement will be resolved through review rather than by choosing the majority image automatically, and agreement among panelists will be summarized with a chance-corrected coefficient such as Krippendorff’s alpha [32] so that low-agreement items are flagged for review. For the behavioral experiment, four balanced item lists will rotate across presentation conditions. The same participant will never answer the same question twice. This design sacrifices some within-item repetition in order to avoid an obvious learning effect from repeated exposure to the correct actions.

We will examine task success at the level of both individual steps and complete sequences. A participant who identifies all objects but reverses the action order will not be counted as having understood the instruction. Response time will be measured from the end of the spoken input and interpreted together with replay count. We will also estimate how often the adaptive rule changes the selected format and whether such changes follow a genuine drop in performance. The comparison with fixed support will use the same stimuli, question types and opportunity to request help.

Discussion

The project should clarify whether adaptive text and visual presentation is useful after spoken input, and which additional costs appear when visual materials are introduced. Clinical populations would require separate recruitment and accessibility review. In particular, variation across an adult convenience sample cannot establish benefit for children with developmental conditions or people with acquired aphasia. Findings will likewise be limited to short, everyday procedural instructions from two speakers; longer discourse, specialized vocabulary or noisy listening conditions may behave differently, and the effect of recognition errors, which would appear on screen as if they had been said, is not examined here.

The principal uncertainty is whether visual support benefits auditory comprehension or simply provides a substitute source of information. Both may be legitimate assistive functions, but they answer different scientific questions. We will report whether participants used audio, text or images when resolving each task, and avoid claims about improved speech perception when the data show only better access to a written representation. Which source resolves each item also matters for the cognitive-load argument: in the dual-channel account of multimedia learning, pictures alongside narration engage separate channels, whereas printed text duplicating narration is the case that account warns against.

References

  1. [1]A. Radford, J. W. Kim, T. Xu, G. Brockman, C. Mcleavey, and I. Sutskever, “Robust Speech Recognition via Large-Scale Weak Supervision,” in Proceedings of the 40th International Conference on Machine Learning, PMLR, Jul. 2023, pp. 28492–28518. Accessed: Nov. 03, 2026. [Online]. Available: https://proceedings.mlr.press/v202/radford23a.htmlarxiv.org/abs/2212.04356
  2. [2]M. A. Gernsbacher, “Video Captions Benefit Everyone,” Policy Insights Behav Brain Sci, vol. 2, no. 1, pp. 195–202, Oct. 2015, doi: 10.1177/2372732215602130.
  3. [3]R. E. Mayer and R. Moreno, “Nine Ways to Reduce Cognitive Load in Multimedia Learning,” Educational Psychologist, vol. 38, no. 1, pp. 43–52, Jan. 2003, doi: 10.1207/S15326985EP3801_6.DOI
  4. [4]S. Kalyuga, “Expertise Reversal Effect and Its Implications for Learner-Tailored Instruction,” Educ Psychol Rev, vol. 19, no. 4, pp. 509–539, Dec. 2007, doi: 10.1007/s10648-007-9054-3.DOI
  5. [5]S. Jhilal, E. Pasqua, C. Marchesi, R. Corradi, and M. Galletti, “Tailoring AI-Driven Reading Scaffolds to the Distinct Needs of Neurodiverse Learners,” in Artificial Intelligence in Education, E. G. Blanchard, G. Chen, M. Chi, and S. Isotani, Eds., Cham: Springer Nature Switzerland, 2027, pp. 505–520. doi: 10.1007/978-3-032-29760-0_52.
  6. [6]S. Jhilal and M. Galletti, “Robust Multilingual Text-to-Pictogram Mapping for Scalable Reading Rehabilitation,” in 2026 IEEE International Conference on Advanced Learning Technologies (ICALT), Jul. 2026, pp. 290–294. doi: 10.1109/ICALT68022.2026.00091.
  7. [7]E. Díez, A. M. Díez-Álamo, M. A. Alonso, D. Z. Wojcik, and A. Fernandez, “Transparency and translucency indices for 1,525 pictograms from the Aragonese Portal of Augmentative and Alternative Communication,” Front. Psychol., vol. 15, Oct. 2024, doi: 10.3389/fpsyg.2024.1467796.DOI
  8. [8]L. Fostick and H. Revah, “Dyslexia as a multi-deficit disorder: Working memory and auditory temporal processing,” Acta Psychologica, vol. 183, pp. 19–28, Feb. 2018, doi: 10.1016/j.actpsy.2017.12.010.DOI
  9. [9]S. Jhilal, N. Molinaro, and A. Klimovich-Gray, “Non-verbal skills in auditory word processing: implications for typical and dyslexic readers,” Language, Cognition and Neuroscience, vol. 40, no. 3, pp. 341–359, Mar. 2025, doi: 10.1080/23273798.2024.2438012.
  10. [10]M. Mody and J. W. Belliveau, “Speech and Language Impairments in Autism: Insights from Behavior and Neuroimaging,” N Am J Med Sci (Boston), vol. 5, no. 3, pp. 157–161, 2013, doi: 10.7156/v5i3p157.DOI
  11. [11]S. M. Redmond, “Language Impairment in the Attention-Deficit/Hyperactivity Disorder Context,” Journal of Speech, Language, and Hearing Research, vol. 59, no. 1, pp. 133–142, Feb. 2016, doi: 10.1044/2015_JSLHR-L-15-0038.DOI
  12. [12]S. Pirila et al., “Language and motor speech skills in children with cerebral palsy,” Journal of Communication Disorders, vol. 40, no. 2, pp. 116–128, Mar. 2007, doi: 10.1016/j.jcomdis.2006.06.002.DOI
  13. [13]X. Zheng, J. Araújo, Q. Busson, and U. Goswami, “Towards a Brain–Computer Interface (BCI) for Improving Phonological Processing in Developmental Dyslexia: An Exploratory Study,” European Journal of Neuroscience, vol. 64, no. 7, p. e70709, 2026, doi: 10.1111/ejn.70709.DOI
  14. [14]P. De Clercq et al., “Detecting Post-Stroke Aphasia Via Brain Responses to Speech in a Deep Learning Framework,” in 2024 46th Annual International Conference of the IEEE Engineering in Medicine and Biology Society (EMBC), Jul. 2024, pp. 1–5. doi: 10.1109/EMBC53108.2024.10781830.
  15. [15]S. Jhilal, S. Martin, and A.-L. Giraud, “Transfer Learning from Imagenet for Meg-Based Decoding of Imagined Speech,” in 2026 IEEE 23rd International Symposium on Biomedical Imaging (ISBI), Apr. 2026, pp. 1–5. doi: 10.1109/ISBI61048.2026.11515534.
  16. [16]P. León-Cabrera et al., “Neural signatures of predictive language processing in Parkinson’s disease with and without mild cognitive impairment,” Cortex, vol. 141, pp. 112–127, Aug. 2021, doi: 10.1016/j.cortex.2021.03.032.
  17. [17]Y. Chen et al., “Clinical Diagnostic and Prognostic Value of Residual Language Learning Ability in Patients with Disorders of Consciousness,” J Neurosci, vol. 45, no. 22, p. e1684242025, May 2025, doi: 10.1523/JNEUROSCI.1684-24.2025.DOI
  18. [18]S. Petit, N. A. Badcock, T. Grootswagers, and A. Woolgar, “Unconstrained multivariate EEG decoding can help detect lexical-semantic processing in individual children,” Sci Rep, vol. 10, no. 1, p. 10849, Jul. 2020, doi: 10.1038/s41598-020-67407-6.DOI
  19. [19]A. B. Silva, K. T. Littlejohn, J. R. Liu, D. A. Moses, and E. F. Chang, “The speech neuroprosthesis,” Nat. Rev. Neurosci., vol. 25, no. 7, pp. 473–492, Jul. 2024, doi: 10.1038/s41583-024-00819-9.DOI
  20. [20]S. Lhilal, S. Marchesotti, B. Thirion, B. Soudrie, A.-L. Giraud, and E. Mandonnet, “Implantable Neural Speech Decoders: Recent Advances, Future Challenges,” Neurorehabil Neural Repair, vol. 40, no. 2, pp. 157–172, Feb. 2026, doi: 10.1177/15459683251369468.
  21. [21]H. Peracha et al., “A Personalized and Adaptable User Interface for a Speech and Cursor Brain-Computer Interface,” in Proceedings of the 2026 CHI Conference on Human Factors in Computing Systems, in CHI ’26. New York, NY, USA: Association for Computing Machinery, Apr. 2026, pp. 1–12. doi: 10.1145/372318.3790750.
  22. [22]T. Gupta, L. Aflatоony, and L. Leonard, “Augmentally: A Reading Assistant Application for Children with Dyslexia,” in Proceedings of the 23rd International ACM SIGACCESS Conference on Computers and Accessibility, in ASSETS ’21. New York, NY, USA: Association for Computing Machinery, Oct. 2021, pp. 1–3. doi: 10.1145/3441852.3476530.
  23. [23]J. O. Wobbrock, S. K. Kane, K. Z. Gajos, S. Harada, and J. Froehlich, “Ability-Based Design: Concept, Principles and Examples,” ACM Trans. Access. Comput., vol. 3, no. 3, p. 9:1–9:27, Apr. 2011, doi: 10.1145/1952383.1952384.DOI
  24. [24]W. Kintsch, “The role of knowledge in discourse comprehension: A construction-integration model,” Psychological Review, vol. 95, no. 2, pp. 163–182, 1988, doi: 10.1037/0033-295X.95.2.163.DOI
  25. [25]D. J. Rajendran, A. T. Duchowski, P. Orero, J. Martínez, and P. Romero-Fresco, “Effects of text chunking on subtitling: A quantitative and qualitative examination,” Perspectives, vol. 21, no. 1, pp. 5–21, Mar. 2013, doi: 10.1080/0907676X.2012.722651.DOI
  26. [26]J. L. Plass and S. Pawar, “Toward a taxonomy of adaptivity for learning,” Journal of Research on Technology in Education, vol. 52, no. 3, pp. 275–300, Jul. 2020, doi: 10.1080/15391523.2020.1719943.DOI
  27. [27]M. Bain, J. Huh, T. Han, and A. Zisserman, “WhisperX: Time-Accurate Speech Transcription of Long-Form Audio,” presented at the Proc. Interspeech 2023, 2023, pp. 4489–4493. doi: 10.21437/Interspeech.2023-78.DOI
  28. [28]S. G. Hart and L. E. Staveland, “Development of NASA-TLX (Task Load Index): Results of Empirical and Theoretical Research,” in Human Mental Workload, vol. 52, P. A. Hancock and N. Meshkati, Eds., in Advances in Psychology, vol. 52. , North-Holland, 1988, pp. 139–183. doi: 10.1016/S0166-4115(08)62386-9.DOI
  29. [29]D. J. Barr, R. Levy, C. Scheepers, and H. J. Tily, “Random effects structure for confirmatory hypothesis testing: Keep it maximal,” Journal of Memory and Language, vol. 68, no. 3, pp. 255–278, Apr. 2013, doi: 10.1016/j.jml.2012.11.001.DOI
  30. [30]A. L. Whitehead, S. A. Julius, C. L. Cooper, and M. J. Campbell, “Estimating the sample size for a pilot randomised trial to minimise the overall trial sample size for the external pilot and main trial for a continuous outcome variable,” Stat Methods Med Res, vol. 25, no. 3, pp. 1057–1073, Jun. 2016, doi: 10.1177/0962280215588241.DOI
  31. [31]M. D. Teare, M. Dimairo, N. Shephard, A. Hayman, A. Whitehead, and S. J. Walters, “Sample size requirements to estimate key design parameters from external pilot randomised controlled trials: a simulation study,” Trials, vol. 15, p. 264, Jul. 2014, doi: 10.1186/1745-6215-15-264.DOI
  32. [32]K. Krippendorff, “Computing Krippendorff’s Alpha-Reliability,” 2013, University of Pennsylvania. [Online]. Available: https://repository.upenn.edu/handle/20.500.14332/2089

Paper details

vlnc:W_hb3sm8mj
Version
v1 · latest
Published
Fresh — added to the Hub within the last 30 days
Added
License
CC BY 4.0
Usage

views · 4

downloads · 0

A view counts once per browser per day, a download each time the PDF is requested.

  • Computer Science

Not recorded for this paper

View statement
Looking for related papers…
Contents