betaFeedback

WorksComputer Science

AdaptType: Accessible and Customizable Multimodal Text Entry in Virtual Reality for People with Upper-Body Motor Impairments

Abstract

Text entry in virtual reality (VR) is cumbersome, especially for people with upper-body motor impairments. Existing techniques often demand both fine and gross motor control, yet users' motor abilities vary widely. To address this range of needs, we present AdaptType, a multimodal VR text entry system that enables users to configure input techniques that best suit their abilities, and scaffolds customization through ability-based recommendations. Through a study with 10 participants, we found that customized multimodal techniques improved text entry speed compared to gaze- or hand-only techniques, and complemented voice for error correction, decreasing failure rate from 15.1% to 2.0%. We found that AdaptType yielded high text entry speed (36.26 WPM) and accuracy (0.36% error rate), with low task load and high usability ratings. We also found ability-based recommendations effectively scaffolded customization. Our work contributes more accessible and personalized VR text entry solutions for people with upper-body motor impairments.

AdaptType enables users with upper-body motor impairments to select VR text entry methods and configure input modalities

Figure 1. AdaptType enables users with upper-body motor impairments to select VR text entry methods that best suit their abilities by offering conversational configurations of input modalities, pointer and selection triggers, keyboard settings, and more, scaffolding these end-user customizations through ability-based recommendations. AdaptType makes VR text entry more accessible for users with upper-body motor impairments, exhibiting high text entry speed and accuracy, low task load, and high usability.

Introduction

Despite the rapid development of virtual reality (VR) devices and sensing capabilities, text entry in VR systems remains cumbersome, especially for people with upper-body motor impairments. People with upper-body motor impairments often face significant barriers while interacting with VR systems [27, 97, 110], as most existing input techniques require both fine and gross motor control. Unlike text entry using physical keyboards or touchscreen devices, VR text entry often relies on mid-air pointing or gestures that require higher levels of dexterity [15, 77]. These mid-air actions offer no physical support for the user to rest their hand, wrist, or arm (the “gorilla arm problem” [39, 46, 69]), and thus may cause severe fatigue and discomfort, especially for people with upper-body motor impairments.

Various techniques have been developed to support VR text entry, for example, by leveraging physical keyboards [37], gaze [59, 78], or voice [74, 81]. However, these techniques each have their own issues for people with motor impairments. Hand-based keyboard interactions may exclude users with limited dexterity; gaze input requires no motor function but can be relatively slow [78] and sometimes fatiguing [93], and may interfere with other tasks such as browsing [58]; voice dictation is often fast and easy to use, yet can be unreliable in noisy environments [13, 86] or for users with impaired speech [25, 112]. Prior work also found that single-modality techniques are often insufficient to accommodate the diverse range of needs for people with upper-body motor impairments, and that one modality that works well for one person may be unusable for another [25, 51].

Importantly, motor impairments are heterogeneous: some people retain strong gross motor abilities but have limited finger dexterity, while others may experience the reverse [110]. This diverse range of needs suggests that there is unlikely a one-size-fits-all solution for accessible VR text entry [40, 85, 88].

To address this wide range of user needs, we developed AdaptType, a highly customizable multimodal VR text entry system that enables users to configure input techniques to suit their own abilities. The system implements a comprehensive end-user customization framework that includes five categories of customizable settings: (1) input modalities, (2) auxiliary pointers, (3) selection triggers, (4) keyboard interface settings, and (5) modes of text entry. Beyond selecting from a pre-defined set of pointing and triggering gestures, AdaptType also allows users to specify customized gestures that suit their fine or gross motor abilities. To help users navigate the large number of options and reduce cognitive burden in customization, the system leverages large-language models (LLMs) to provide ability-based recommendations to scaffold the customization process. Based on these recommendations, the system allows users to further adjust customization settings through conversational interactions.

Over multiple preliminary sessions with three people with upper-body motor impairments, we co-designed the AdaptType system, beginning with an initial prototype we developed and using participants’ feedback and insights to guide iterative refinements toward the final working system. We then evaluated our final AdaptType system through a lab study with 10 participants with a range of upper-body motor disabilities. During the study sessions, participants completed text entry tasks using their customized technique, as well as using hand-, gaze-, and voice-only techniques; participants then iterated on their customization choices based on the experience.

Specifically, our evaluation answered two main research questions:

  • RQ1. Do customized multimodal text entry methods make VR text entry more accessible for people with upper-body motor impairments? If so, in which respects?

  • RQ2. What approaches can effectively support customization of VR text entry for people with upper-body motor impairments?

Our study results showed that customized multimodal techniques improved text entry speed compared to gaze- or hand-only techniques, and complemented voice for error correction, decreasing failure rate from 15.1% to 2.0%. We found that AdaptType yielded high text entry speed (36.26 WPM) and accuracy (0.36% error rate), with low task load and high usability ratings. Additionally, we found that AdaptType’s ability-based recommendations effectively scaffolded selection of text entry methods that matched participants’ motor abilities, and that iterative customization showed trends of improvement in text entry performance, task load, and usability.

Note that the goal of our study was not to provide a comprehensive comparison of all VR text entry modalities; nor was it to show that customized techniques outperform every baseline in every respect. Rather, our study demonstrates how AdaptType makes VR text entry accessible by accommodating a range of motor abilities and enabling selection of techniques that best match each user’s abilities. Our findings also shed light on design implications for accessible personalized text entry, suggesting that text entry interfaces should offer ability-based recommendations to scaffold customization, and should guide users to iteratively refine their customization choices over time.

The main contributions of this paper are:

  • A framework for end-user customization that supports accessible and multimodal text entry for people with upper-body motor impairments;

  • The AdaptType system, a mixed-initiative [45] and ability-based [103, 104] VR text entry system that offers a rich set of customization options and provides recommendations to scaffold customizations based on users’ abilities;

  • Empirical findings on customization choices, input behavior, text entry performances, and preferences with different modalities in VR text entry for people with upper-body motor impairments.

Related Work

Ability-Based Design

Our work builds upon the philosophy of Ability-Based Design (ABD) introduced by Wobbrock et al. [103, 104]. ABD is a design approach that centers the abilities of users throughout the design process, with the goal of creating systems that “leverage the full range of human potential” [104]. In an ability-based system, the responsibility for adaptation lies with the system. It is the system that adjusts to the user’s specific abilities, rather than requiring the user to conform to the system’s constraints. Systems might also be highly configurable by end-users so as to better match their abilities.

A number of prior systems have been informed by ABD principles, from automatic user interface generation [30, 31, 32] to enhancing outdoor play experiences for children [2]. Recent systems adopt signals such as gaze, motion, and touch patterns to support adaptation, for example, characterizing specific fine motor challenges through touch behavior [48], improving touch accuracy for people with motor impairments [65, 66], assisting with reading resumption [61], and supporting adaptive app creation [49]. Within the domain of text entry, prior work proposed techniques for dynamically adapting keyboards based on detected walking [35] or hand posture [36] for improved accuracy, and ability-based optimization of keyboard layouts based on user movement patterns [62] and for older adults [82, 83, 84].

While much of the existing work in ABD emphasizes adaptivity—that is, systems that automatically adapt based on sensed or inferred user abilities—there is a limited body of work focused on adaptability, where users themselves actively configure or customize the system to suit their unique abilities. Our work offers a new perspective by supporting adaptability in the context of VR text entry, allowing users to personalize their interactive experiences based on their specific motor abilities.

Virtual Reality Accessibility for Upper-Body Motor Impairments

Current VR technologies present substantial barriers to people with disabilities [34]. Mott et al. [64] identified seven categories of access barriers in VR and highlighted the need for design for customization and diversity, and through an ability-based approach. Gerling et al. [33] also emphasized designing for experience for equitable access to VR. A review by Dudley et al. [19] on VR/AR accessibility further highlighted the need of accommodating a combination of impairments rather than optimizing for single conditions.

Prior work has explored accessible VR interaction through scene-viewing and point-of-interest techniques [27, 50], mappings between bimanual and unimanual interactions [109], and multimodal input and geometric motion mapping [97, 98], and highlighted accessibility as a key design dimension in locomotion techniques [14]. In particular, Di Luca et al. [14] found that locomotion techniques may be more accessible when they are abstract, and less accessible when they resemble real-world movement. Most recently, Pococke et al. [76] showed that default freehand gestures, including pinch and grasp, cause accessibility barriers because they rely heavily on the index finger. Franz et al. [28] also suggested having a range of accessible techniques to choose from based on user preferences. These findings directly motivated our decision to make the selection trigger configurable rather than defaulting to pinch.

User-Defined Gestures

A complementary line of work approached accessible input through user-defined gestures, in both traditional 2D GUIs and 3D interfaces. Following Wobbrock et al.’s work on eliciting user-defined gestures [102, 105] and the $1 unistroke recognizer [106], subsequent recognizers have supported multistroke [3, 4] and point-cloud [94, 95] gestures. Through elicitation, gesture sets have been developed for wheelchair users [10], wrist-worn devices [108], and augmented reality [75]. In particular, Zhao et al. [117] and Tian et al. [92] elicited above-the-neck gestures and gestures for spinal muscular atrophy respectively. Our work draws inspiration from prior research on the preferences of people with upper-body motor impairments in personalizing mid-air gestures [110]—even within the relatively specific population of users with motor disabilities, there exists considerable variation in which body parts individuals use to perform upper-body gestures. This diversity highlights the need for personalized input systems that can adapt to and leverage the unique motor capabilities of each user, rather than relying on a one-size-fits-all approach.

Virtual Reality Text Entry Methods

Text entry in VR has been an active research area. A survey by Dube and Arif [16] classified more than 30 techniques across 11 categories and concluded that different approaches are effective under different situations: a user might, for example, use a pointing-based technique for short inputs and switch to a physical keyboard for extended writing. We therefore organize prior work by input modality.

Physical keyboards when used in VR achieved the highest reported speeds. Grubert et al. [37] reported 41.2–43.7 WPM, but required external hardware. Mid-air hand typing on virtual keyboards was the most widely studied alternative, including two-finger [18] and ten-finger variants [26, 111], and pinch selection [5, 21]. Word-gesture typing methods adapted swipe keyboards to VR using 6-DOF controllers [11] and force-based digital thimbles [17]. Gaze- and head-based typing provided accessible alternatives to hand-based interactions: dwell-based gaze typing [59, 78] were accelerated through multi-threshold dwell schemes [67] and fused with gesture typing via implicit gaze [116]. Head-based tapping, dwelling, and gesturing on virtual keyboards [54, 107, 113] provided additional hands-free options. Speech input were explored both standalone [74, 81] and in combination with a mid-air keyboard to reach 28 WPM [1].

Pointing and other hand-based text entry common in consumer VR has been a barrier for users with upper-limb motor impairments [64, 76]. Alternative modalities each have their own accessibility trade-offs: gaze typing can be fatiguing and sensitive to calibration [59], speech input struggles with ambient noise [13, 86] and impaired speech [25, 112], and word-gesture or mid-air techniques rely on sustained arm movements and fine motor control. The modality that is most usable for one person may therefore be unusable for another [25]. Our AdaptType system instead lets users configure their modality, selection trigger, and entry granularity, making the input choice more accessible and personalized.

Co-Design of the AdaptType System

Prior work suggested that text entry systems should be customizable [51, 64, 72] and multimodal [25, 51, 91], yet how such customization may be achieved to accommodate a range of motor abilities remains unclear. Prior work also elicited gesture preferences of people with upper-body motor impairments [110], but did not establish which gestures might be feasible for extended VR text entry sessions. We therefore developed AdaptType through a co-design process with three people with upper-body motor impairments.

Co-Design Process

We began by building an initial prototype of AdaptType based on prior work insights [25, 51, 64, 110]. The initial prototype integrated gaze, hand, and voice input, and implemented two gestures for users with limited fine/gross motor function: a finger pinch gesture that required no gross motor function, and a hand wave gesture that required no fine motor function, based on personalized gestures elicited in prior work [110]. This prototype allowed us to probe the open design questions through hands-on sessions.

We then worked with three participants with upper-body motor impairments to co-design the AdaptType system. The three participants were all women, on average 70.3 years old (SD=8.0SD = 8.0), and had varying fine and gross motor functions.1 In a hands-on session, each participant configured a text entry method by working through the customization options, then entered at least 10 phrases from the MacKenzie and Soukoreff phrase set [57] using their customized technique. Participants also explored state-of-the-art VR text entry techniques such as using a hand raycast and a pinch trigger, when feasible given their motor abilities. Participants then shared their experience and feedback, and brainstormed what their “ideal keyboard” would be based on the experience.

Design Considerations

Our co-design work helped reveal three primary design considerations, described below.

Custom gestures should be performable with the hand/finger supported. Our initial prototype used a hand wave gesture in which participants move the whole forearm to trigger selection [110]. Participants confirmed the hand wave required no fine motor control, but reported fatigue within a few trials. One participant mentioned “it was much easier to control the cursor with my hand resting on the mouse and sliding on the desktop.” One participant used one hand to hold the other wrist in mid-air while exploring the hand-based technique, and reported it made the gesture easier by “stabilizing [their] wrist.” The findings suggest that customized gestures for VR text entry should not only consider whether gross/fine motor function is required, but also whether the user’s hand/finger can rest in a comfortable position while performing it. Therefore, we redesigned the custom triggers around movements performed against a supporting surface—a desktop, armrest, or the user’s lap (§ 4.3).

Custom gestures should support a flexible range of motion. Although finger pinch was technically feasible for users with mild fine motor impairments, one participant still found this gesture difficult to use, as gestural detection failed when their thumb and index finger were not “wide open.” They also reported fatigue with their fingers after usage. This finding indicated that custom gestures should support a flexible range of motion, and informed our design of calibrating custom pointers and triggers to each user’s range of motion. In the iterated design, a user can lightly lift up their palm or finger, curl up their wrist, or use other gestures that work best with their motor function.

Effective customization requires scaffolding beyond enumeration. During the sessions, all participants spent over an hour exploring all customization options and configuring their personalized input method. Participants found the process “overwhelming,” and repeatedly asked to revisit settings as they could no longer clearly recall which ones had worked well for them. This finding indicated that enumeration would fail when the customizable system offers too many available options, and motivated our design of narrowing the choice and offering ability-based recommendations as a starting point (§ 4.4), rather than presenting users with the full set of options upfront.

AdaptType

In this section, we present the design and implementation of the AdaptType system in two main respects: (1) the end-user customization framework, and (2) the scaffolding customization process based on users’ motor abilities.

The End-User Customization Framework

We include a summary of all the customization options AdaptType supports in Figure 2. There are five categories of customization settings: (1) input modality, (2) auxiliary pointer (when the input modality is voice), (3) selection trigger, (4) keyboard interface settings, and (5) text entry mode. We conceptualized the input modality, auxiliary pointer, and selection trigger following anatomies of 3D pointing in prior work [73, 96].

The AdaptType end-user customization framework

Figure 2. The AdaptType end-user customization framework. There are five categories of customization settings: (1) input modality, (2) auxiliary pointer (when the input modality is voice), (3) selection trigger, (4) keyboard interface settings, and (5) text entry mode. Voice command is also included as a stand-alone feature. The “L” and “R” labels represent the handedness of modalities, auxiliary pointers, and selection triggers. The “C” labels indicate that the modalities, auxiliary pointers, and selection triggers can be calibrated to match the user’s abilities, e.g., posture or range of motion.

Setting 1: Input Modality

The “input modality” refers to the way a user indicates their desired text to input, whether by pointing at a visual target, or by speaking using voice recognition.

Gaze. Users look at the keys to enter text. The gaze input modality is implemented using the Meta OVR Eye library to obtain the user’s gaze direction in 3D world coordinates.

Hand/Arm (Gross Motor Function). Users control a hand-based pointer emitted from the palm using their left/right hand or forearm to point at the keys. This modality is implemented using the Meta OVRHand library to obtain real-time position and direction of the palm, from which a hand-based raycast is computed. Users may manipulate the pointer using wrist or forearm movements, requiring only gross motor control. By default, the hand raycast directly maps the palm’s six degrees-of-freedom (DoF) position and direction to the pointer’s origin and direction. However, certain regions of the keyboard may be difficult or unreachable for some users under this mapping. To address this, AdaptType provides a calibration mechanism that allows users to draw a rectangular area in mid-air that they are comfortable reaching with their hand and forearm. The system then computes a mapping from the detected palm pose to the raycast such that the full keyboard remains reachable while the user’s hand stays within their specified area.

Finger (Fine Motor Function). Users control a finger-based pointer emitted from their left/right index finger to point at the keys. This modality is implemented using the Meta OVRHand and OVRSkeleton libraries to obtain real-time hand joint positions, from which a finger-based raycast is computed. Users manipulate the pointer through index finger movements, relying on fine motor control without requiring gross arm movement. AdaptType provides a similar calibration mechanism as for hand/arm pointing that enables access to the full keyboard.

Head. Users control a head-based pointer emitted between the eyes and in the head’s forward orientation to point at the keys. This modality is implemented using the Meta OVREye library to obtain real-time eye anchor positions, from which a head-directed raycast is computed. The head pointer is calibrated using similar approaches as the hand and finger pointers to address challenges with reachability.

Voice. Users dictate the text they wish to enter. This modality is implemented using the NativeWebSocket open source library2 and OpenAI’s real-time transcription API,3 using the gpt-4o-mini-transcribe model. Transcribed speech is treated as input text once a complete transcription segment is returned by the API.

Auxiliary Pointer

The “auxiliary pointer” refers to the way a user indicates a visual target when their primary input modality is voice, when there is no pointer that allows for target pointing and selections. Thus, we offer an auxiliary pointer option including a similar set of options as in Section 4.1.1: gaze, hand/arm, finger, and head.

Selection Trigger

The “selection trigger” refers to the way a user sends a selection event to the system to actuate whatever they are indicating with their “input modality.”

Pinch. Users close their thumb and index finger to perform a “pinch” gesture, and open their fingers to deactivate the trigger (Figure 3d). The pinch gesture is detected using the Meta OVRHand library. This selection trigger requires high finger dexterity; meanwhile, this trigger can be performed on the desktop, armrest, user’s lap, or any position visible to the headset, thus requiring little to no gross motor control.

Example trigger gestures

Figure 3. A non-exhaustive set of example trigger gestures a user can select. (a) Hand tap on armrest; (b) Finger tap on desktop; (c) Hand tap on lap; (d) Wrist curl; (e) Pinch.

Custom Finger Gesture. Users perform a custom finger gesture using their index finger; for example, by tapping on a surface such as a desktop, armrest, or lap (Figure 3c), while the hand and wrist remain supported. This method derives an implicit index-finger raycast from hand joint positions and triggers selection when the raycast’s vertical angle exceeds a threshold, with a 100 ms smoothing window applied. The default threshold is set to zero, assuming a flat surface. To calibrate this gesture, users perform the gesture (e.g., tapping) 10 times, and the system records the range of vertical motion of the implicit finger raycast, setting the trigger threshold to the median of this range.

Custom Hand/Arm Gesture. Users perform a custom hand/arm gesture by moving their entire hand or forearm; for example, by tapping on a surface such as a desktop, armrest, or lap (Figure 3a). The custom hand/arm gesture is calibrated using a similar approach as the custom finger gesture.

Note that although the implementation of custom finger and hand/arm gestures are based on heuristics, the detection mechanism supports a surprisingly rich set of custom gestures without requiring resource-intensive model training. For example, aside from the “finger tap” (Figure 3c) and “hand tap” (Figure 3a) gestures as default options, users can perform a wrist curl (Figure 3b), a finger curl, a hand flip, and many other gestures that align with their motor abilities, as long as the vertical movement is detectable.

Dwell. Users trigger selection by dwelling on a target. The default dwell time is 600 ms, a duration that balances speed, accuracy, and usability as prior work recommends [71].

Voice Command. We also include a “voice command” feature as a stand-alone control mechanism. Unlike pointer-based inputs, voice commands do not function as selection triggers but instead directly manipulate text and interface elements. When voice command is enabled, the system sends the transcribed text with the current input box content to a GPT-4o-mini endpoint to determine whether the input corresponds to a command (e.g., “enter,” “clear,” or “change the first word to chocolate”) or text entry. The system then executes the inferred command or updates the text accordingly.

Setting 4: Keyboard Size and Position

Users can adjust the keyboard size/depth and position through voice control such as “move keyboard up,” “move keyboard left,” “move keyboard closer,” “move keyboard farther,” etc. The keyboard depth control is restricted within a bound so that the keyboard does not exceed ±35∘\pm35^\circ horizontally and vertically from the center of view, a comfortable range within a user’s view [8], and that each key has minimum width of 5∘5^\circ to ensure gaze pointing usability.

Setting 5: Text Entry Mode

Using the selected input modality and selection trigger, users can enter text character by character by pointing at the keys. Alternatively, a swipe-based method is available in which users maintain the trigger in an active state while moving across keys and release the trigger to input a word.4 When voice is selected as the input modality, text is entered at word/phrase-level, while character-level editing remains available in parallel.

Text Entry Interface Visualization and Feedback Implementation

The text entry interface is positioned at the center of a vertical 2D task plane five meters away from the user, with a keyboard in the middle and left and right panels showing configuration information and usage tips, respectively (Figure 4). The hand- and finger-based pointers are visualized as red beams emitted from the user’s hand or finger toward the keyboard, and keys are highlighted in yellow when hovered over, similar to state-of-the-art VR pointing visualizations. The raycast beam is not visualized for gaze or head input, as visible cursors can degrade gaze performance due to cursor chasing [59]. Selection triggers are visualized using status bars, which appear gray when inactive and green when activated (Figure 4 and Figure 5a), except for dwell, where the selected key turns green from the yellow hover highlight to indicate activation.

AdaptType text entry interface

Figure 4. The AdaptType text entry interface. The keyboard builds on top of the standard QWERTY keyboard layout with function keys at the top of the keyboard: “sel word” to select the word at the current cursor location, “<” and “>” to move the cursor to the left/right, “backspace” to delete a character or the selected text, and “clear” to clear the input text box. Users can also place the cursor using their selected pointer to point at the intended location, then activating the selection trigger (for example, looking at the letter “T” and performing a pinch). The left information panel shows the currently selected customization options, and the right panel displays hints for keyboard voice commands. The user is currently looking at the letter T and performing a pinch gesture with their right hand, as indicated by the right green status bar.

The settings menu of AdaptType

Figure 5. The settings menu of AdaptType, which allows users to customize their text entry method: (a) an example settings menu displaying most of the available customization options for a user with less severe motor impairments; (b) an example settings menu displaying the available customization, for example, when a user does not have fine motor function and has impaired neck movement.

Ability-Based Recommendations of Customization Settings

While AdaptType offers a rich set of customization options that enable users to select input methods aligned with their abilities (Figure 5a), the breadth of these options may be overwhelming. Users may also be unaware of the full range of options or which options best suit their needs [23, 24, 27]. To address this challenge, AdaptType guides customization through an ability-based recommendation of settings leveraging large-language models (LLMs).

Requesting Recommendations from LLMs. We abstract users’ anonymized responses describing specific aspects of their motor abilities from the screening questionnaire, including condition descriptions, 11 ratings using the QuickDASH questionnaire [38], four ratings of lateralized motor abilities (left/right, fine/gross), one rating of neck movement, and one rating of speaking ability. A request is then sent to a gpt-4o-mini endpoint to generate five recommended combinations in the format <InputModality|SelectionTrigger>, with voice excluded from the recommendations. Upon receiving the response, the system parses the recommended combinations and iterates through them to identify the first combination in which the input modality and selection trigger are compatible (e.g., a user cannot use the same hand to manipulate the pointer and perform a tapping selection gesture).

Propagating Recommended Presets. The AdaptType system offers three presets for each user to explore. Note that the presets do not necessarily represent what is likely the best option for the user; instead, they aim to span a variety of customization options that may be feasible for the user, without overwhelming the user with the number of options. The first valid <InputModality|SelectionTrigger> pair parsed from the GPT response is used as Preset 1, emphasizing manual control. The system then iterates through the selection triggers compatible with the user’s abilities to identify an alternative trigger (AltTrigger) that differs from the initial recommendation. This trigger is paired with the gaze input modality as the LLM recommendation prioritizes hand- or finger-based modalities unless motor impairments are severe. Thus, gaze input and AltTrigger form Preset 2, which emphasizes alternative input and triggering strategies. For Preset 3, the system selects voice as the input modality and identifies a third compatible selection trigger,5 along with an AltPointer not used in Presets 1 or 2 as an auxiliary input. Voice input, the AltPointer, and the selected trigger together form Preset 3. Collectively, these three presets span a range of interaction strategies aligned with the user’s motor abilities, from manual pointer-based control to voice-driven input.

Identifying Available Customization Options. In addition to the LLM-based recommendations, AdaptType applies a heuristic-based filter to exclude customization options that are likely inaccessible given the user’s motor abilities. Input modalities and selection triggers involving the hand, finger, or head are removed if the user indicates an “unable” (5/5) rating for the corresponding motor ability. Furthermore, head- and finger-based input modalities and auxiliary pointers are excluded when users report more than moderate impairment (> 3/5), as these interactions require finer motor control or impose greater strain on the neck when using a head-mounted display.

End-User Customization through Voice Commands. Based on the recommendations, users can further configure their text entry method using conversational interactions. Users can say “open settings menu” to access the configuration interface (Figure 5) and “confirm” to apply the current selection and return to the text entry interface. Within the settings menu, users can issue commands such as “select voice input” or “select pinch trigger” to modify settings.

Study Method

We conducted a lab study with 10 participants to evaluate AdaptType. The goal of the study was to validate the effectiveness of AdaptType in enabling text entry configurations that matched user abilities, to understand users’ customization behaviors and preferences, and to investigate whether customized multimodal techniques improved the accessibility of VR text entry.

Participants

We recruited 10 participants with upper-body motor impairments (Table 1) through email solicitations, organizational newsletters, and snowball sampling. On average, participants were 56.4 years old (SD=15.2SD = 15.2), with 5 women and 5 men. Reported health conditions included spinal cord injury (4), amputation (2), facioscapulohumeral muscular dystrophy (2), shoulder pain (1), and autonomic small fiber dysfunction.

IDGenderAgeCondition
P1Man51Spinal cord injury
P2Man69Shoulder pain
P3Man23Spinal cord injury
P4Woman71Autonomic small fiber neuropathy, cervical radiculopathy
P5Woman60Amputation
P6Woman67Facioscapulohumeral muscular dystrophy
P7Man38Facioscapulohumeral muscular dystrophy
P8Man63Amputation
P9Woman59Spinal cord injury
P10Woman63Spinal cord injury

caption: Participant demographics. Condition indicates each participant’s self-reported physical condition influencing their upper-body motor ability.

:::

neuropathy and cervical radiculopathy (1). Participants reported having moderate expertise with text entry using physical keyboards (M=3.0M = 3.0, SD=1.3SD = 1.3, on a novice–expert scale of 1–5), virtual keyboards (M=3.1M = 3.1, SD=1.5SD = 1.5), and voice input (M=3.3M = 3.3, SD=1.5SD = 1.5), and having novice-level experience with VR (M=1.2M = 1.2, SD=0.6SD = 0.6). Participants were compensated $40 per hour, pro-rated to each half-hour, plus transportation costs.

Apparatus

We used a Meta Quest Pro headset (1800 × 1920 px per eye, 90 Hz refresh rate, 106° × 96° FoV) for our study, with 72 Hz eye tracking rate and 60 Hz hand tracking rate. The AdaptType prototype was a Unity app (Figure 4 and Figure 5, version 6000.1.2f1) built on a Windows 11 laptop with a 2.4 GHz Intel i7 CPU and GeForce GTX 4060 GPU, running on the Meta Quest Pro headset via a Meta Link cable (Figure 6). Study sessions were conducted in a lab setting, with participants seated at a desk (Figure 6).

Illustration of the study setup

Figure 6. Illustration of the study setup. A participant seated at a desk wearing the Meta Quest Pro headset, connected to an experiment laptop facing the experimenter.

Study Procedure

Study sessions were conducted in person with approval from our university’s Institutional Review Board (IRB). Participants completed a screening survey including demographic questions and 17 questions on their motor and speaking abilities (11 questions of the QuickDASH questionnaire [38], with an additional four questions on lateral fine/gross motor abilities, one question on neck movement, and one question on speaking ability), whose responses were used to generate ability-based recommendations. After a brief introduction, participants were fitted with the headset and completed an eye-tracking calibration procedure through the Meta Quest Pro calibration dialog. Each session lasted approximately two hours, and short breaks were offered throughout the session.

Part 1: Initial Customization. Participants were guided to use AdaptType to select their customized text entry technique, based on the three ability-based recommended keyboard presets (§ 4.4). Participants completed a calibration process for each new custom trigger/pointer, and transcribed one sentence with each recommendation. Participants were then instructed to open the settings menu (Figure 5) and freely explore other available options.6 Participants could adjust their customization choices until they were satisfied with their configuration after successfully completing one full sentence with the selected technique.

Part 2: Text Entry Tasks. Participants then completed text entry tasks using four techniques (Table 2, left) × two tasks (Table 2, right). After completing the tasks for each technique, participants completed the NASA-TLX questionnaire [41, 42] and a usability questionnaire adapted from the System Usability Scale [6], respectively for the editing and the transcription task.

TechniqueDescription
Custom-InitialThe participant’s personalized AdaptType configuration selected in Part 1.
GazeGaze for pointing with dwell selection.
HandA hand raycast for pointing, with a pinch selection trigger.7
VoiceDictation for text entry and voice commands for editing, with no keyboard or pointer.
TaskDescription
EditingParticipants corrected a partially entered phrase to match a target. Errors were seeded by perturbing the target at the character level: for a phrase of length NN, between 5% and 15% of the characters were selected at random and modified by substitution, deletion, or insertion, each with equal probability. Only lowercase letters were substituted or inserted. This yielded 1–4 edits for phrases of approximately 30 characters. Participants completed two practice trials, then a two-minute test block.
TranscriptionParticipants entered complete target phrases from scratch. Participants completed two practice trials, then a five-minute test block.

Table 2. The four text entry techniques compared in Part 2 of the study (left) and the two tasks participants completed with each technique (right). Technique order was counterbalanced using a balanced Latin square [101]. Target phrases in both tasks were randomly drawn from the MacKenzie and Soukoreff phrase set [57]. Participants were instructed to keep going until signified to stop, when (1) they had reached the test block time bound (2 minutes for editing, 5 minutes for transcription), and (2) they had completed at least five trials. Participants were also advised to proceed to the next trial if one editing or transcription trial was taking longer than 60 or 90 seconds, respectively. We used this method to ensure reasonable study duration in cases of excessively long trials.

Part 3: Iterated Customization. If a participant completed Parts 1 & 2 in less than two hours and reported no significant fatigue, they also then completed a round of iterated customization. Participants were given the option to change their customization settings and adopt features that they found useful based on their experiences in Parts 1 & 2, yielding the Custom-Iterated technique. Participants then used this technique to complete the transcription task and the NASA-TLX and usability questionnaires. Participants also shared additional thoughts on their experience.8

Data

We collected a total of 782 trials from the 10 participants, including 244 trials using the Custom-Initial text entry technique, 119 trials using the Gaze technique, 51 trials using the Hand technique, 257 trials using the Voice technique, and 111 trials using the Custom-Iterated technique. There were 184 total trials collected from the editing task and 598 total trials collected from the transcription task. We excluded P1 and P2’s data for the Voice technique from the analyses, as the voice input implementation was updated after P2, thereby affecting the quality of voice command processing. The Hand technique was omitted for P1, P3, P6, P8, P9 and P10, who did not have the fine and gross motor control required to use it. Among the collected data, 17 trials were failed attempts (2.2%),9 which we excluded from the text entry performance analyses of speed, accuracy, and throughput, leaving 765 trials. The Custom-Iterated technique was collected only for P3–P8, and only for the transcription task.

Measures

We measure the text entry experience using metrics in three respects: (1) performance, (2) perceived task load, and (3) usability ratings. For transcription tasks, we report standard text entry metrics: speed (WPM) [56], corrected error rate (CER), uncorrected error rate (UER), and total error rate (TER) [87, 114]. We additionally report throughput, which combines speed and accuracy into a single measure of efficiency [115]. For editing tasks, we report duration (the elapsed time from the first input event to trial confirmation) and the UER of the final edited sentence relative to the target phrase. Participants filled out the NASA Task Load Index [41, 42] for each technique, which captures mental demand, physical demand, temporal demand, performance, effort, and frustration. We additionally administered a five-item usability questionnaire covering ease of use, ease of learning, comfort, fatigue, and anticipated future use. All items used a 1–10 scale, where 1 indicated the most positive rating and 10 the least.

Analysis

For transcription speed (WPM) and editing duration, we ran linear mixed models (LMM) [53, 99] with Technique as a fixed factor and Subject and Phrase as random factors. For throughput, which had a single value per participant × technique, we ran an LMM with Technique as the fixed factor and Subject as a random factor. For accuracy, we modeled UER, CER, and TER as error counts over a total character denominator [87], fitting beta-binomial generalized linear mixed models (GLMM) [7] with Technique as a fixed factor and Participant and Phrase as random factors. We ran mixed ordinal logistic regression models [43] to compare task load and usability ratings, modeling Technique as a fixed factor and Subject as a random factor. Post hoc pairwise comparisons were corrected with Holm’s sequential Bonferroni procedure [44] following significant omnibus tests.

We ran a paired-samples tt-test [89] to compare text entry throughput before and after iterative customization, as throughput had a single value per participant per technique. We used Wilcoxon signed-rank tests [100] to compare the task load and usability ratings before and after iterative customization. We also ran an LMM [29, 53] to compare the transcription WPM and binomial generalized linear mixed models (GLMM) [7] for UER, CER, TER before and after iterative customization, using Technique (Custom-Initial vs. Custom-Iterated) as a fixed factor and Subject and Phrase as random factors.

Given the small sample size (10 participants, 765 trials), our analyses have limited statistical power for detecting population-level effects. Moreover, participants exhibited heterogeneity in their text entry behavior, performance, and preferences, which may further obscure effects that generalize across participants. Therefore, in addition to inferential statistical analyses, we report descriptive statistics characterizing AdaptType’s performance, task load, usability, and the coverage of ability-based recommendations. We also examine participants’ customization choices and individual differences on a case-by-case basis to capture insights that may not be reflected in aggregate results.

Results

In this section, we report our user study results and how they address our two research questions:

  • RQ1. Do customized multimodal text entry methods using AdaptType make VR text entry more accessible for people with upper-body motor impairments? If so, in which respects?

  • RQ2. What approaches can effectively support customization of VR text entry for people with upper-body motor impairments?

RQ1. Does AdaptType Make VR Text Entry More Accessible?

Finding #1: AdaptType yielded high performance, low task load, and high usability across diverse motor abilities.

Among the six participants who completed the iterated customization phase (Part 3, Section 5.3), they achieved an average text entry speed of 36.26 WPM (SD=11.36SD = 11.36) and uncorrected error rate of 0.36% (SD=2.52%SD = 2.52\%) using the Custom-Iterated technique, and it exhibited the highest throughput (10.12 bits/s, SD=2.06SD = 2.06) among all techniques (compared to 9.19 bits/s for Voice, 7.45 bits/s for Custom-Initial, 2.06 bits/s for Hand, and 2.02 bits/s for Gaze). Participants found their Custom-Iterated technique to exhibit low physical demand (M=2.2M = 2.2, SD=1.3SD = 1.3), low mental demand (M=2.2M = 2.2, SD=1.0SD = 1.0), high performance (M=3.2M = 3.2, SD=1.5SD = 1.5), low effort (M=2.8M = 2.8, SD=1.3SD = 1.3), and low frustration (M=2.5M = 2.5, SD=1.1SD = 1.1). Participants also found their Custom-Iterated technique easy to use (M=2.2M = 2.2, SD=1.2SD = 1.2), comfortable (M=2.0M = 2.0, SD=1.1SD = 1.1), and causing little fatigue (M=2.7M = 2.7, SD=0.8SD = 0.8). Participants indicated very high interest in future use (M=1.7M = 1.7, SD=1.0SD = 1.0) of this technique. Participants showed high satisfaction with their iterated customization, regarding their customized techniques as “this works great” (P3) and “that’s the one” (P7). P8 also commented that “this is really easy, I’m super excited about it.”

Finding #2: Customized, multimodal techniques exhibited text entry speed comparable to Voice and higher than Gaze and Hand.

With regard to text entry speed, both Custom-Initial and Voice had relatively high text entry speed in WPM (M=35.03M = 35.03, SD=16.78SD = 16.78 and M=33.43M = 33.43, SD=9.24SD = 9.24, respectively) in the transcription task, compared to Gaze (M=6.54M = 6.54, SD=1.63SD = 1.63) and Hand (M=6.82M = 6.82, SD=2.09SD = 2.09). Technique had a significant effect on WPM (F3.444,3=120.44F_{3.444,3} = 120.44, p<.001p < .001). The Custom-Initial technique had higher WPM than Gaze (t446=16.12t_{446} = 16.12, p<.001p < .001) and Hand (t441=11.10t_{441} = 11.10, p<.001p < .001); the Voice technique had higher WPM than Gaze (t442=14.83t_{442} = 14.83, p<.001p < .001) and Hand (t442=10.32t_{442} = 10.32, p<.001p < .001). We found no significant differences in WPM between Custom-Initial and Voice (t448=1.83t_{448} = 1.83, p=.135p = .135), or between Gaze and Hand (t441t_{441} = 0.69, pp = .489). In the editing task, Technique had a significant effect on Duration (F3,166.3F_{3,166.3} = 2.85, pp = .039): Custom-Initial took longer than Voice (MM = 33.7s vs. 24.4s; t165t_{165} = 2.74, pp = .041), with no other pairwise difference reaching significance.

Finding #3: Voice excelled in transcription accuracy and task load, yet struggled with error correction, which multimodality remedied.

With regard to text entry accuracy, Technique had a significant effect on UER (χ32\chi^2_3 = 12.66, pp = .005), CER (χ32\chi^2_3 = 24.48, pp < .001) and TER (χ32\chi^2_3 = 29.80, pp < .001) in the transcription task—while all techniques exhibited low UER (< 1.5%), Voice had the lowest UER of all at 0.22% (SD = 0.23). For the editing task, we omit statistical analysis with UER, as only 17 of the 174 trials (9.8%) had a non-zero UER. Descriptively, Voice had the highest editing UER (MM = 0.96%, SD = 2.50%), compared to Custom-Initial (MM = 0.63%, SD = 0.78%), Gaze (MM = 0.32%, SD = 0.64%) and Hand (MM = 0.38%, SD = 0.55%). Voice also had the highest variance in UER across participants, with the highest being 7.1% (P5), while all other techniques’ highest participant-average UER remained below 3% (Figure 8).

Performance measures of the five techniques in the transcription and editing tasks

Figure 8. Performance measures of the five techniques in the transcription and editing tasks. Note that Custom-Iterated is included in the figure to show the overall performance of different techniques, but was excluded from the comparison analysis § 6.1.2 as it was not counterbalanced for Order. Overall, Custom (both Initial and Iterated) and Voice techniques exhibited high speed and throughput, while Gaze and Hand were relatively slow and inefficient. In the transcription tasks when participants entered sentences from scratch, Voice exhibited the lowest uncorrected error rate. However, in the editing tasks when participants corrected existing errors in sentences, Voice exhibited the highest error rate, and became especially inaccurate for some participants (e.g., P5), while other techniques remained low in error rate. Put together, customized multimodal techniques offer both the benefit of fast input comparable to Voice and high accuracy in error correction comparable to Gaze or Hand.

Notably, the accuracy analysis was conducted based on successful trials after removing failed attempts. With regard to failed attempts, however, Voice had the highest failure rate in the editing task (15.1%) among all techniques, compared to 2.0% for Custom-Initial, 1.8% for Gaze, 0.0% for Hand. In the transcription task, Hand had the highest failure rate (12.0%) among all techniques, compared to 1.5% for Custom-Initial, 0.9% for Custom-Iterated, and 0.0% for Voice. Among the four techniques for which both editing and transcription data is available, only Custom-Initial and Gaze exhibited consistently low failure rate within 2% across editing and transcription tasks.

With regard to task load and usability, Technique had a significant effect on mental demand (χ32\chi^2_3 = 10.13, pp = .017), physical demand (χ32\chi^2_3 = 11.06, pp = .011), performance (χ32\chi^2_3 = 9.18, pp = .027), effort (χ32\chi^2_3 = 15.72, pp = .001), frustration (χ32\chi^2_3 = 14.92, pp = .002), ease of use (χ32\chi^2_3 = 14.17, pp = .003), comfort (χ32\chi^2_3 = 13.25, pp = .004), fatigue (χ32\chi^2_3 = 8.63, pp = .035), and future use (χ32\chi^2_3 = 10.25, pp = .017) in the transcription task, with a marginal effect on temporal demand (χ32\chi^2_3 = 7.15, pp = .067). In the editing task, Technique had a significant effect on frustration (χ32\chi^2_3 = 10.10, pp = .018) and a marginal effect on effort (χ32\chi^2_3 = 6.75, pp = .080). All significant or marginal pairwise comparisons are shown in Table 4. All pairwise comparisons favored Voice, yet fewer ratings showed significant differences in editing than transcription in omnibus tests. Meanwhile, individual differences remained. For example, P6 rated Voice to have the highest average task load (MM = 5.83, SD = 0.41) and worst usability (MM = 6.00, SD = 0.00) among all techniques.

Task LoadUsability
Mental DemandPhysical DemandTemporal DemandPerformanceEffortFrustrationEase of UseEase of LearningComfortFatigueFuture Use
Transcription
Custom-Initial–Voice2.70 *2.78 *—2.54 ·3.29 **3.22 **3.23 **—3.12 *2.61 ·2.50 ·
Voice–Gaze-2.58 *-2.50 ·——-3.10 **-2.85 *-2.68 *—-2.67 *-2.45 ·-2.50 ·
Voice–Hand—-2.43 ·——-2.26 ·—-2.30 ·————
Editing
Custom-Initial–Voice————2.40 ·2.74 *—————
Voice–Gaze—————-2.63 *—————

Table 4. ZZ-statistics for significant or marginal pairwise comparisons on the NASA-TLX task load and usability items, from mixed ordinal logistic regressions with Holm correction within each item. A positive ZZ means the first technique was rated less favorably. Technique pairs with no significant or marginal comparison on any item are omitted. Custom-Iterated is excluded, as it always ran last, was not counterbalanced for Order, and had no editing task. Significance indicators: · p<.10p < .10, *p<.05p < .05, **p<.01p < .01, ***p<.001p < .001.

RQ2. What approaches can effectively support customization of VR text entry?

Finding #4: AdaptType’s ability-based recommendations effectively scaffolded selection of text entry methods

Failed Attempts by Technique and Task

Figure 9. The rate of failed attempts for each technique by task. Voice had a zero failure rate with transcription, yet had a high failure rate (15.1%) with editing, when participants were instructed to correct errors in an existing sentence. Hand had a zero failure rate with editing, yet had a high failure rate (12.0%) with transcription, when participants were instructed to enter a full sentence from scratch. Note that only transcription data is available for the Custom-Iterated technique. Among the rest of the four techniques, Custom-Initial and Gaze were the only techniques exhibiting consistently low failure rates (≤2%\le2\%) across editing and transcription tasks.

TranscriptionEditing
UERCERTERUER
MM (SDSD)MM (SDSD)MM (SDSD)MM (SDSD)
Custom-Initial0.92 (1.13)7.44 (7.19)8.37 (7.06)0.63 (0.78)
Custom-Iterated0.38 (0.76)6.12 (6.57)6.50 (6.49)n/a
Voice0.22 (0.23)4.42 (5.47)4.64 (5.39)0.96 (2.50)
Gaze1.03 (1.65)3.83 (3.76)4.85 (4.25)0.32 (0.64)
Hand1.09 (0.90)3.04 (1.28)4.12 (1.99)0.38 (0.55)
Transcription
UERCERTER
Custom-Initial–Voice3.07 *3.49 **4.54 ***
Custom-Initial–Gaze———
Custom-Initial–Hand—-2.18 ·—
Voice–Gaze-2.43 ·-4.04 ***-4.55 ***
Voice–Hand-3.09 *-4.35 ***-4.36 ***
Gaze–Hand———

Table 3. Left: error rates by technique, as participant means in percent. Right: zz-statistics for significant or marginal pairwise comparisons. Measures with no significant or marginal differences are omitted from the right-hand block. Custom-Iterated is reported descriptively but excluded from the comparisons, as it always ran last and was not counterbalanced for Order. CER and TER do not apply to the editing task, where participants corrected pre-seeded errors rather than entering text from scratch. Significance indicators: · p<.10p < .10, *p<.05p < .05, **p<.01p < .01, ***p<.001p < .001.

that matched participants’ upper-body motor abilities. Participants selected a wide range of different input modalities, pointing methods, and selection triggers (Table 5). Nine of the 10 participants selected voice as their primary input modality, and one selected right hand. Seven participants selected gaze as the auxiliary pointer, one selected right hand, and one selected the head pointer. Selection triggers were the most varied: finger tapping on the desktop (3), hand tapping on the desktop (2) or on the armrest (1), gaze dwelling (2), wrist curling up on the armrest (1), and pinch (1). In the iterated customization choices, all six participants selected voice as their primary input modality; four participants selected gaze as the auxiliary pointer, and two selected the right hand. Participants also chose a variety of selection triggers, including hand tapping on the desktop (1) or on the armrest (1), finger tapping on the desktop (1), wrist curling on the armrest (1), gaze dwelling (2), and pinch (1).

IDInitial Input ModalityInitial Auxiliary PointerInitial Selection TriggerIterated Input ModalityIterated Auxiliary PointerIterated Selection TriggerAdditional FeatureSelf-Reported Fine Motor Function LeftSelf-Reported Fine Motor Function RightSelf-Reported Gross Motor Function LeftSelf-Reported Gross Motor Function RightSelf-Rep. Voice
P1VoiceGazeDwell––––55553
P2Right Hand–Left Pinch––––11111
P3VoiceRight HandLeft Hand (curling left wrist on armrest)VoiceRight HandLeft Hand (curling left wrist on armrest)Voice command55221
P4VoiceGazeRight Finger Tap (on desktop)VoiceGazeRight Finger Tap (on desktop)Voice command23322
P5VoiceGazeRight Finger Tap (on desktop)VoiceRight HandRight PinchVoice command51111
P6VoiceGazeLeft Hand Tap (on armrest)VoiceGazeLeft Hand Tap (on armrest)Voice command44453
P7VoiceGazeRight Finger Tap (on desktop)VoiceGazeDwellVoice command11331
P8VoiceHeadLeft Hand Tap (on desktop)VoiceGazeDwellVoice command45111
P9VoiceGazeLeft Hand Tap (on desktop)––––44451
P10VoiceGazeDwell––––42311

Table 5. Customization choices and self-reported motor and voice function for each participant. Overall, participants selected customization options that aligned with their self-reported motor abilities. For example, P3 retained much better gross motor function than fine motor function on both sides, thus selected right hand and left hand wrist curl as the pointing and trigger gestures, which required no fine motor function; P7 retained much better fine motor function than gross motor function, thus selected the right finger desktop tap trigger gesture and gaze dwell, which required no gross motor function; P5 only had impaired fine motor function on the left side, thus selected the right hand as the pointer and right finger tap and right pinch as the triggers.

Participants selected customization options that aligned with their motor abilities (Table 5). All five participants with severe bilateral fine motor impairment (self-reported 4 or 5 on both sides) selected either gaze dwelling or a whole-hand/arm selection trigger, requiring no fine motor function. All three participants who retained unimpaired fine motor function on at least one side (self-reported 1) selected a finger-based trigger on that side. P1 reported no fine or gross motor function on either side (5 across all four ratings), and selected a fully limb-free configuration using voice and gaze. P10 was the only exception to this pattern, selecting the same limb-free configuration despite retaining moderate gross and fine motor function on the right side, suggesting that limb-free configurations could be selected by preference as well as by necessity. Overall, participant selected customization options aligned with their motor abilities. This alignment additionally addresses RQ1, showing that AdaptType enabled users to select input modalities and triggers that suited their abilities and needs across diverse motor abilities.

The ability-based recommendations offered by AdaptType (Section 4.2) also aligned with participants’ customization choices (Figure 10). Among the 10 participants, all selected input modalities offered in the three presets recommended (100%), eight selected auxiliary pointers offered in the recommendations (80%), and eight selected triggers offered in the recommendations (80%). Among the six participants who completed the iterated customization, all six selected input modalities (100%) and auxiliary pointers (100%) offered in the recommendations, and four selected triggers offered in the recommendations (66.7%). The alignment between the recommendations and participants’ customization choices indicates the effectiveness of ability-based recommendations in scaffolding customization.

Coverage of users’ customization choices in recommended presets for (a) initial customization and (b) iterated customization

Figure 10. Coverage of users’ customization choices in recommended presets for (a) initial customization and (b) iterated customization. Green bars represent participant choices offered in a preset and with the same role. Blue bars represent choices offered in a preset with a different role, e.g., when gaze was offered as an input modality, yet a participant selected it as the auxiliary pointer. Yellow bars represent handedness variant of a preset option, e.g., when right pinch was offered in a preset, yet a participant selected left pinch. Red bars represent selections not offered in any preset, and gray bars represent cases when the auxiliary pointer was not selected (not applicable). All participants selected input modalities offered in the recommendations during both initial and iterated customization. For the auxiliary pointer, most participants’ choices were offered in the recommendations, but some with a different role. For the selection trigger, most participants’ choices were offered in the recommendations, while some updated their choices to triggers not offered in any preset in the iterated customization.

Finding #5: Iterative customization showed trends of improvement in text entry performance, task load, and usability.

After iterative customization, the average text entry throughput increased from 7.53 bits/s (SD=2.83SD = 2.83) to 10.12 bits/s (SD=2.06SD = 2.06), with a 34.4% improvement from Custom-Initial to Custom-Iterated. The paired-samples tt-test showed a significant difference (t5=3.51t_5 = 3.51, p=.017p = .017). The average text entry speed increased slightly from 31.54 WPM (SD=6.88SD = 6.88) to 35.88 WPM (SD=4.08SD = 4.08), with a 13.7% improvement from Custom-Initial to Custom-Iterated. We found a marginal effect of Technique on text entry speed (WPM) (F1,203.6=3.57F_{1,203.6} = 3.57, p=.060p = .060), a significant effect on UER (χ12=6.28\chi^2_1 = 6.28, p=.012p = .012), which decreased from 0.27% to 0.05%, a marginal effect on TER (χ12=3.63\chi^2_1 = 3.63, p=.057p = .057), and no effect on CER (χ12=2.07\chi^2_1 = 2.07, p=.150p = .150).

Among task load and usability ratings, marginal significance was detected in Wilcoxon signed-rank tests with mental demand (z=−2.07z = -2.07, p=.053p = .053), physical demand (z=−1.80z = -1.80, p=.090p = .090), and comfort (z=−2.00z = -2.00, p=.072p = .072). No other significant or marginal differences were found. Although only marginal significances were detected, descriptive statistics showed consistent trends of improvement after iterative customization (Figure 11). Participants also reported having better understanding of their ideal customization after extended usage and trial-and-error with different input modalities over tasks. For example, P7 switched from hand-based trigger to dwell trigger, as they “liked the flow once [they were] familiar with it”; and P5 switched from gaze-based pointing to hand-based pointing, as they had enough gross motor function to manipulate the hand pointer, yet gaze tracking “didn’t work well” for them.

Bar charts showing task load and usability ratings, throughput, speed, and error rates for initial and iterated customization

Figure 11. M±SDM \pm SD of performance measures, task load, and usability ratings of the initial vs. iterated customization (n=6n = 6).

Discussion

In this section, we discuss findings from our case studies, implications for future designs, and limitations of this work.

Accommodating a Range of Motor Abilities

A central motivation for AdaptType is that motor impairments are heterogeneous—some users retain strong gross motor ability but limited finger dexterity, while others may experience the reverse [110]. Our 10 participants had a wide range of upper-body motor disabilities that affected different aspects of their gross and fine motor abilities, from spinal cord injury and amputation to neuropathy and muscular dystrophy. Our study results illustrate how the customization framework accommodated this diversity: participants with severe bilateral fine motor impairment (e.g., P1, P3, P6, P8) selected gaze- or hand/arm-based selection triggers, while participants who retained unimpaired fine motor function on at least one side selected a finger-based technique (e.g., P2, P5, P7) (§ 6.3.1). Across participants with different motor abilities, AdaptType yielded high performance (36.26 WPM, 0.36% uncorrected error rate, 10.12 bits/s), low task load (M=2.58M = 2.58, SD=1.08SD = 1.08), and high usability (M=2.13M = 2.13, SD=0.93SD = 0.93), which demonstrates the system’s capability in supporting accessible VR text entry for people with varying upper-body motor impairments (§ 6.1.1).

Beyond findings that characterize holistic behavior patterns and preferences of all participants, individual differences remained, which further highlights the heterogeneity of the population and the need for customization—for example, P10 found Gaze highly usable, while P3 experienced extremely high task load and low usability with it; P9 rated Voice to be the best in task load and usability among all techniques, yet P6 had the worst experience with it (Figure 7). Such variations indicate that there is no single best solution in VR text entry for people with upper-body motor impairments.

Participant-level comparison of five techniques in performance, task load, and usability by participant and technique

Figure 7. Participant-level comparison of five techniques in performance (throughput, WPM, UER), average task load, and average usability measures, among the six participants who completed both the initial and the iterated customization. The scales have been adjusted so that “taller” bars always indicate “better” experiences. A black cell indicates that technique is completely inaccessible to the participant. Participants’ experiences with different modalities were highly varied: (1) the Custom-Iterated technique exhibited consistently high performance, low task load, and high usability; (2) Voice worked well for many participants, yet still failed for some (e.g., P6); (3) Gaze was highly varied across participants; and (4) Hand was entirely inaccessible to about half of the participants.

The central contribution of AdaptType is not that it outperforms every single baseline in every respect, but rather that it enables users with diverse motor abilities to configure input methods in a way that best suits their abilities.

Why Not Just Use Voice Input?

Admittedly, voice is a powerful modality in text entry for people with upper-body motor impairments [12, 51, 68, 79]. Its popularity and advantages were also reflected in our data: voice dictation had been widely adopted in daily usage among the participants (M=3.3M = 3.3, SD=1.5SD = 1.5 in self-reported expertise, on a 1–5 scale), and nine out of 10 participants selected voice as their main input modality during customization, which may have partly contributed to the high text entry speed of the customized techniques. Our analyses also showed that Voice overall exhibited high speed, low transcription accuracy, low task load, and high usability (§ 6.1.2 and 6.1.3).

However, voice-only techniques can be limited in some use cases, and prior work has highlighted the need for multimodal alternatives beyond single-modality input for people with upper-body motor impairments [51]. Here, we discuss limitations of voice input that AdaptType can remedy:

Voice does not work for everyone. Despite the general trends that favored Voice in our statistical analyses, pronounced individual differences remained. For example, P6 had facioscapulohumeral muscular dystrophy and self-reported having impaired voice (3 on a scale of 1–5). Accordingly, P6 rated the voice-only technique to have the highest task load (M=5.83M = 5.83, SD=0.41SD = 0.41) and the lowest usability (M=6.00M = 6.00, SD=0.00SD = 0.00) among all techniques and among all participants. Such experiences could occur for people with similar voice impairments, for example, due to age, dysarthria [60], or other muscular or neurological conditions.

Voice is not as effective in error correction as in transcription. Through our editing task, we aimed to compare customized multimodal techniques against single-modality techniques when errors occurred. We showed that Voice was highly effective in transcription tasks; however, when it came to editing, it had a notably high failure rate (15.1%) and relatively high variation in uncorrected error rate among successful trials (SDSD = 2.5%, max = 7.1%). Meanwhile, although Voice excelled in most transcription task load and usability ratings, this advantage lessened when it came to editing (Table 4).

Multimodal techniques improve user agency compared to using voice only. In our study, participants also indicated interest in using pointer-based techniques aside from voice input due to the capabilities of making fine edits. For example, P3 mentioned that “there are still scenarios where further adjustments would be needed,” and P4 liked the multimodal pointer-based technique because they “feel like [they] have better control.”

Voice may not be ideal for some usage scenarios. Prior research has suggested that voice input may lead to privacy and security issues when used in public [20, 52]. Aside from these concerns, participants also mentioned scenarios in which voice dictation had failed for them, for example, “when I lost my voice” (P3) or when “listening to podcast” (P7). Participants believed that multimodal text entry could be helpful in these scenarios when voice input may not be ideal.

While we highlight these limitations of voice, this paper does not argue that pointer-based techniques should replace voice input for people with upper-body motor impairments. Rather, customized multimodal techniques complement voice dictation in cases where voice may fail or be less preferred—by supporting error correction, improving user agency, and offering flexibility in different usage scenarios.

Addressing Ability-Based Design Principles and “Consequences” in Accessibility

The goal of AdaptType is to accommodate a range of motor abilities, which aligns with the core idea of Ability-Based Design [103, 104]. Thus, we examined our system against the seven principles of Ability-Based Design [103] (Table 6, left). AdaptType addresses most principles through its highly customizable interface, while some principles remain open for future work, such as Performance and Context. Additionally, we examined our system against the nine Ability Heuristics (Table 6, right), which are used to conduct ability-based accessibility inspections [63]. AdaptType satisfies most heuristics, particularly through its highly flexible task flow and multimodal approach, while some heuristics point to future work improvements, such as Ability Data Transparency and Help, Support, and Community.

PrincipleHow AdaptType Addresses ItHeuristicHow AdaptType Addresses It
AbilityThe system is inherently centered around abilities by providing customization options and recommendations based on each user’s gross and fine motor abilities.AdaptabilityThe system provides a highly customizable framework that enables users to configure input methods to their motor abilities.
AccountabilityThe system is adaptable by the user in its provision of a rich set of configuration options.Equitable ExperienceAlthough user choices varied, the customized text entry method yielded consistent high performance and usability across participants who chose different methods.
AvailabilityWhile this system is built on commodity hardware and software, future work should support deployment of the system on lower-cost devices than the Meta Quest Pro.Flexible Task CompletionUsers can freely switch between input modalities, and auxiliary input pointers are provided.
AdaptabilityThe system enables VR text entry methods that can be adapted to suit each user.Efficiency and Effectiveness of User ActionUsers can enter text in VR with ease using the system, with high text entry performance and low task load.
TransparencyThe system provides visual and audio feedback on user actions, and always allows users to override system recommendations.Multiple ModalitiesThe system combines the gaze, hand and finger, head, and voice modalities to enable a multimodal text entry experience.
PerformanceFuture work could track a user’s text entry performance in real time to suggest potential changes and improvements in a user’s customizations.Understandable MessagesThe system provides clear instructions for users and offers hints for conversational commands.
ContextFuture work could detect a user’s task, activity, or situational context and make suggestions for adjustments accordingly.Ease of AdoptionThe system runs on commodity hardware, while future work could expand its applicability to lower-cost devices.
Ability Data TransparencyFuture work could improve the transparency of ability data, and inform users of how their ability data is being used to produce recommendations.
Help, Support, & CommunityFuture work could offer support for troubleshooting and community feedback or support.

Table 6. How AdaptType addresses the seven principles of Ability-Based Design [103] (left) and the nine Ability Heuristics [63] (right).

Prior work has also emphasized that accessibility should be considered not only in terms of “technical feasibility,” but also in terms of the broader “consequences” of usage [55]. This perspective highlights that technologies should not simply make interaction possible for people with disabilities, but should also account for discomfort that may arise from their use. A technology should not be described as “accessible” if it causes significant fatigue, discomfort, or other negative consequences after use. In our study, some participants were technically able to “use” the state-of-the-art method based on hand pointing and pinch gestures. For example, P4 achieved seemingly reasonable text entry accuracy with Hand (2.15% uncorrected error rate), yet reported experiencing severe fatigue with their hand after. In contrast, AdaptType mitigates the possible negative consequences of usage, and no participant reported having severe fatigue with the Custom-Iterated technique despite it was the last technique evaluated.

Design Implications and Generalizability Beyond Virtual Reality

Aside from its artifact contributions, this work also contributes empirical insights and design implications on personalized text entry for accessibility. One major co-design insight that informed our system was that participants were overwhelmed by the large number of customization options. Prior work found that while customization is often desirable, users may struggle to navigate the available options and may not be aware of which options work best for them [9, 22, 25, 80, 90]. Our finding aligns with prior work, and showed the effectiveness of ability-based recommendations in addressing this “awareness” gap by scaffolding customization using AI recommendations (§ 6.3.1). Additionally, we found that iterative customization exhibited trends of improvement in text entry performance, task load, and usability (§ 6.3.2). With highly configurable systems, users may experience steep learning curves with various modalities (“with practice I’ll definitely get better,” P9), and iteration through trial-and-error may improve the outcome of customization. This finding also aligns with prior work showing users’ customization preferences evolve with usage [47, 72]. Together, our findings suggest that personalized text entry interfaces should: (1) offer ability-based recommendations to reduce the selection space and scaffold customization, and (2) guide users to iteratively refine their customization choices over time.

While our system focuses on text entry in VR, its end-user customization framework may generalize beyond virtual environments. We selected the VR scenario because the head-mounted display served as an integrated device that supported a wide range of sensing and interaction capabilities, including gaze tracking, hand and finger tracking, head tracking, voice input, and interface rendering. That said, similar functionality can be achieved outside of VR using installed devices (e.g., the Tobii eye tracker and the OptiTrack motion capture system), enabling deployment without a headset or a fully virtual environment. Participants also expressed interest in using AdaptType beyond VR contexts. For example, P7 noted that “it’ll be very cool to use this keyboard without the headset, so I could type with my eyes at home,” highlighting the potential transferability of AdaptType’s customization framework to broader multimodal text entry scenarios with personalized interaction techniques.

Limitations and Future Work

One limitation of this work is the small sample size (10 participants, 782 trials), which significantly limits the power of inferential analyses and generalizability of our findings. Due to the small sample size, we rely on descriptive statistics alongside inferential statistics to illustrate trends and showcase the capabilities of AdaptType. We also discuss individual differences among participants, which aligns with the heterogeneity of the user population.

With regard to its implementation, AdaptType relies on state-of-the-art gaze- and hand-tracking through the Meta Quest Pro and the OVREye, OVRHand, and OVRSkeleton libraries. Tracking accuracy and latency remain limiting factors, and future systems could benefit from higher-fidelity tracking techniques. A primary contribution of AdaptType was its end-user customization framework rather than a novel gesture recognition technique; thus, our system used threshold-based heuristics with user calibration, which may be less robust than personalized gesture recognition models trained from a set of collected user gestures. Future work could involve more robust recognition techniques, for example, similar to the $1 gesture recognizer [106] or using transfer learning [70]. Furthermore, our custom finger and hand/arm gesture detection relies on visible motion, which is inherently limited for users who retain little to no visible upper-body motor function. Future systems could incorporate additional sensing modalities such as electromyography (EMG) to detect muscle activations that do not produce observable motion.

In our study, participants were allowed to customize numerous aspects of the system, from keyboard size and position to pointing and triggering modalities, utilizing a wide range of interaction techniques. Controlled comparisons accounting for all personalization factors can be hardly captured within a single experiment, and the effect of each specific factor on the performance differences remain unclear. Thus, our study should be regarded as an exploratory study showcasing the customization capability of AdaptType and its benefits, rather than a comprehensive comparison of various input modalities or customization options. The Custom-Iterated condition was also not counterbalanced like the other four conditions, as this condition required completion of the other four. Although we removed this condition from the cross-modality analyses, this limitation may result in biases that favored the iterated customization when compared to the initial customization. Additionally, only six participants completed the the iterated customization, as extended VR headset interactions beyond two hours can cause severe fatigue and discomfort for participants with upper-body motor impairments. During our study, we observed a steep learning curve with the various techniques, and the two-hour study was relatively short for full familiarization. Given the limited session duration, our findings regarding iterative customization were only exploratory, and future work should include large-scale and longitudinal studies on how text entry behavior and customization preferences evolve over time. Furthermore, we included the editing task in our study to showcase the benefits of multimodality in error correction, as natural error correction behavior can be hard to control and analyze in a lab study. That said, our editing task remains rather artificial, and future work should investigate natural error correction behavior with real-world tasks “in the wild.”

Conclusion

We have presented AdaptType, an accessible and customizable multimodal VR text entry system that can be adapted to the diverse abilities and needs of people with upper-body motor impairments. By supporting a flexible combination of input modalities, offering a rich set of configuration settings, and providing ability-based recommendations to scaffold customization, our system allows users to create personalized VR text entry experiences that align with their abilities. Through a lab study with 10 participants with a range of upper-body motor disabilities, we found that customized multimodal techniques improved text entry speed compared to gaze- and hand-only techniques, and complemented voice input for error correction. Participants selected customization options that matched their upper-body motor abilities, achieved high text entry speed and accuracy with low task load. Participants found their customized techniques easy to use and learn, comfortable, causing little fatigue, and welcomed for future use. Our findings demonstrated the capability of AdaptType to accommodate a wide range of motor abilities, and supporting ability-based accessible VR text entry through end-user customization.

Acknowledgments

The authors thank the Here and Now Project, Sue Spaulding, Kathleen Quin Voss, Momona Yamagami, and Qingyang Wan. This work was supported in part by the University of Washington Center for Research and Education on Accessible Technology and Experiences (CREATE). Any opinions, findings, conclusions or recommendations expressed in our work are those of the authors and do not necessarily reflect those of any supporter.

References

References

  1. [1]Jiban Adhikary and Keith Vertanen. 2021. Text Entry in Virtual Environments using Speech and Handwriting on Visualizations. IEEE Transactions on Visualization and Computer Graphics 27, 5 (2021), 2648–2658. doi:10.1109/TVCG.2021.3067776
  2. [2]Ahmad Alsaleem, Ross Imburgia, Andrew Merryweather, Jeffrey Rosenbluth, Stephen Trapp, and Jason Wiese. 2020. Applying Ability-Based Design Principles to Adaptive Outdoor Activities. In Proceedings of the 2020 ACM Designing Interactive Systems Conference (DIS ’20). Association for Computing Machinery, New York, NY, USA, 1–12. doi:10.1145/3357236.3359508DOI
  3. [3]Lisa Anthony and Jacob O Wobbrock. 2010. A lightweight multistroke recognizer for user interface prototypes. In Proceedings of Graphics Interface 2010. 245–252.
  4. [4]Lisa Anthony and Jacob O Wobbrock. 2012. $ N$-protractor: a fast and accurate multistroke recognizer. In Proceedings of Graphics Interface 2012. 117–120.
  5. [5]Doug A Bowman, Vinh Q Ly, and Joshua M Campbell. 2001. Pinch keyboard: Natural text input for immersive virtual environments. (2001). Technical Report. Department of Computer Science, Virginia Polytechnic Institute and State University.
  6. [6]John Brooke. 1996. Sus: a “quick and ‘dirty’ usability. Usability evaluation in industry 189, 3 (1996), 189–194.DOI
  7. [7]Mollie E Brooks, Kasper Kristensen, Koen J Van Benthem, Arni Magnusson, Casper W Berg, Anders Nielsen, Hans J Skaug, Martin Machler, and Benjamin M Bolker. 2017. glmmTMB balances speed and flexibility among packages for zero-inflated generalized linear mixed modeling. (2017).
  8. [8]Charlie S Burlingham, Naveen Sendhilnathan, Oleg Komogortsev, T Scott Murdison, and Michael J Proulx. 2024. Motor “laziness” constrains fixation selection in real-world tasks. Proceedings of the National Academy of Sciences 121, 12 (2024), e2302239121.DOI
  9. [9]Ana Caraban, Evangelos Karapanos, Daniel Gonçalves, and Pedro Campos. 2019. 23 Ways to Nudge: A Review of Technology-Mediated Nudging in Human-Computer Interaction. In Proceedings of the 2019 CHI Conference on Human Factors in Computing Systems (CHI ’19). Association for Computing Machinery, New York, NY, USA, 1–15. doi:10.1145/3290605.3300733DOI
  10. [10]Patrick Carrington, Amy Hurst, and Shaun K Kane. 2014. The gest-rest: a pressure-sensitive chairable input pad for power wheelchair armrests. In Proceedings of the 16th international ACM SIGACCESS conference on Computers & accessibility. 201–208.DOI
  11. [11]Sibo Chen, Junce Wang, Santiago Guerra, Neha Mittal, and Soravis Prakka-makul. 2019. Exploring Word-gesture Text Entry Techniques in Virtual Reality. In Extended Abstracts of the 2019 CHI Conference on Human Factors in Computing Systems (Glasgow, Scotland, UK) (CHI EA ’19). Association for Computing Machinery, New York, NY, USA, 1–6. doi:10.1145/3290607.3312762DOI
  12. [12]Stefano Dalla Gasperina, Marta Gandolla, Alessandra Calcagno, Andrea Costa, Lorenzo Aquilante, Markus Puchinger, Margit Gfoehler, Francesco Braghin, and Alessandra Pedrocchi. 2018. Multi-modal human-machine control interfaces of upper limb motorized exoskeletons for severely impaired patients. In 2018 7th IEEE International Conference on Biomedical Robotics and Biomechatronics (Biorob). IEEE, 491–496.
  13. [13]Jessyca L Derby, Emily A Rickel, Kelly J Harris, Jade A Lovell, and Barbara S Chaparro. 2020. “We didn’t catch that!” using voice text input on a mixed reality headset in noisy environments. In Proceedings of the human factors and ergonomics society annual meeting, Vol. 64. SAGE Publications Sage CA: Los Angeles, CA, 2102–2106.
  14. [14]Massimiliano Di Luca, Hasti Seifi, Simon Egan, and Mar Gonzalez-Franco. 2021. Locomotion vault: the extra mile in analyzing vr locomotion techniques. In Proceedings of the 2021 CHI conference on human factors in computing systems, 1–10.
  15. [15]Tafadzwa Joseph Dube and Ahmed Sabbir Arif. 2019. Text entry in virtual reality: A comprehensive review of the literature. In International Conference on Human-Computer Interaction. Springer, 419–437.DOI
  16. [16]Tafadzwa Joseph Dube and Ahmed Sabbir Arif. 2019. Text Entry in Virtual Reality: A Comprehensive Review of the Literature. In Human-Computer Interaction. Recognition and Interaction Technologies: Thematic Area, HCI 2019, Held as Part of the 21st HCI International Conference, HCI 2019, Orlando, FL, USA, July 26–31, 2019, Proceedings, Part II (Orlando, FL, USA). Springer, Springer-Verlag, Berlin, Heidelberg, 419–437. doi:10.1007/978-3-030-22643-5_33DOI
  17. [17]Tafadzwa Joseph Dube, Kevin Johnson, and Ahmed Sabbir Arif. 2022. Shapeshifter: Gesture Typing in Virtual Reality with a Force-based Digital Thimble. In Extended Abstracts of the 2022 CHI Conference on Human Factors in Computing Systems (New Orleans, LA, USA) (CHI EA ’22). Association for Computing Machinery, New York, NY, USA, Article 230, 9 pages. doi:10.1145/3491101.3519679DOI
  18. [18]John Dudley, Hrooje Benko, Daniel Widgor, and Per Ola Kristensson. 2019. Performance Envelopes of Virtual Keyboard Text Input Strategies in Virtual Reality. In 2019 IEEE International Symposium on Mixed and Augmented Reality (ISMAR). 289–300. doi:10.1109/ISMAR.2019.00027DOI
  19. [19]John Dudley, Lulu Yin, Vanja Garaj, and Per Ola Kristensson. 2023. Inclusive Immersion: a review of efforts to improve accessibility in virtual reality, augmented reality and the metaverse. Virtual Real. 27, 4 (Sept. 2023), 2989–3020. doi:10.1007/s10055-023-00850-8DOI
  20. [20]Aarthi Esawara Moorthy and Kim-Phuong L Vu. 2015. Privacy concerns for use of voice activated personal assistant in the public space. International Journal of Human-Computer Interaction 31, 4 (2015), 307–335.
  21. [21]Jacqui Fishampour, Kenrick Kin, and Matt Wong. 2020. PinchType: Text Entry for Virtual and Augmented Reality Using Comfortable Thumb to Fingertip Pinches. In Extended Abstracts of the 2020 CHI Conference on Human Factors in Computing Systems (Honolulu, HI, USA) (CHI EA ’20). Association for Computing Machinery, New York, NY, USA, 1–7. doi:10.1145/3334480.3382888
  22. [22]Leah Findlater and Joanna McGrenere. 2004. A comparison of static, adaptive, and adaptable menus. In Proceedings of the SIGCHI conference on Human factors in computing systems. 89–96.DOI
  23. [23]Leah Findlater and Joanna McGrenere. 2007. Evaluating Reduced-Functionality Interfaces According to Feature Findability and Awareness. In Human-Computer Interaction – INTERACT 2007, Cécilia Baranauskas, Philippe Palanque, Julio Abascal, and Simone Diniz Junqueira Barbosa (Eds.). Springer, Berlin, Heidelberg, 592–605. doi:10.1007/978-3-540-74796-3_59DOI
  24. [24]Leah Findlater and Joanna McGrenere. 2008. Impact of Screen Size on Performance, Awareness, and User Satisfaction with Adaptive Graphical User Interfaces. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems (CHI ’08). Association for Computing Machinery, New York, NY, USA, 1247–1256. doi:10.1145/1357054.1357249DOI
  25. [25]Leah Findlater and Joanna McGrenere. 2010. Beyond Performance: Feature Awareness in Personalized Interfaces. International Journal of Human-Computer Studies 68, 3 (March 2010), 121–137. doi:10.1016/j.ijhcs.2009.10.002DOI
  26. [26]Conor R. Foy, John J. Dudley, Aakar Gupta, Hrvoje Benko, and Per Ola Kristensson. 2021. Understanding, Detecting and Mitigating the Effects of Coactivations in Ten-Finger Mid-Air Typing in Virtual Reality. In Proceedings of the 2021 CHI Conference on Human Factors in Computing Systems (Yokohama, Japan) (CHI ’21). Association for Computing Machinery, New York, NY, USA, Article 287, 11 pages. doi:10.1145/3411764.3445671DOI
  27. [27]Rachel R Franz, Jacob O Wobbrock, Yi Cheng, and Leah Findlater. 2019. Perception and adoption of mobile accessibility features by older adults experiencing ability changes. In The 21st International ACM SIGACCESS Conference on Computers and Accessibility. 267–278.DOI
  28. [28]Rachel F Franz, Jiaqian Yu, and Jacob O Wobbrock. 2023. Comparing locomotion techniques in virtual reality for people with upper-body motor impairments. In Proceedings of the 25th international ACM SIGACCESS conference on computers and accessibility. 1–15.DOI
  29. [29]Brigitte N Frederick. 1999. Fixed-, Random-, and Mixed-Effects ANOVA Models: A User-Friendly Guide for Increasing the Generalizability of ANOVA Results.
  30. [30]Krzyszof Z Gajos, Daniel S Weld, and Jacob O Wobbrock. 2010. Automatically generating personalized user interfaces with Supple. Artificial intelligence 174, 12-13 (2010), 910–950.DOI
  31. [31]Krzysztof Z Gajos, Jacob O Wobbrock, and Daniel S Weld. 2007. Automatically generating user interfaces adapted to users’ motor and vision capabilities. In Proceedings of the 20th Annual ACM Symposium on User Interface Software and Technology. 231–240.DOI
  32. [32]Krzysztof Z Gajos, Jacob O Wobbrock, and Daniel S Weld. 2008. Improving the performance of motor-impaired users with automatically-generated, ability-based interfaces. In Proceedings of the ACM SIGCHI conference on Human Factors in Computing Systems. 1257–1266.DOI
  33. [33]Kathrin Gerling, Anna-Lena Meiners, Louisa Schumm, Jen Rixen, Marvin Wolf, Zeynep Yildiz, Dmitry Alexandrovsky, and Merlin Opp. 2025. An equitable experience? How HCI research conceptualizes accessibility of virtual reality in the context of disability. ACM Transactions on Accessible Computing 18, 4 (2025), 1–35.
  34. [34]Kathrin Gerling and Katja Spiel. 2021. A Critical Examination of Virtual Reality Technology in the Context of the Minority Body. In Proceedings of the 2021 CHI Conference on Human Factors in Computing Systems (Yokohama, Japan) (CHI ’21). Association for Computing Machinery, New York, NY, USA, Article 599, 14 pages. doi:10.1145/3411764.3445196DOI
  35. [35]Mayank Goel, Alex Jansen, Travis Mandel, Shwetak N. Patel, and Jacob O. Wobbrock. 2012. WalkType: Using Accelerometer Data to Accommodate Situational Impairments in Mobile Touch Screen Text Entry. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems (CHI ’12). Association for Computing Machinery, New York, NY, USA, 2687–2696. doi:10.1145/2207676.2208662DOI
  36. [36]Mayank Goel, Alex Jansen, Travis Mandel, Shwetak N. Patel, and Jacob O. Wobbrock. 2013. ContextType: Using Hand Posture Information to Improve Mobile Touch Screen Text Entry. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems (CHI ’13). Association for Computing Machinery, New York, NY, USA, 2795–2798. doi:10.1145/2470654.2481386DOI
  37. [37]Jens Grubert, Lukas Witzani, Eyal Ofek, Michel Pahud, Matthias Kranz, and Per Ola Kristensson. 2018. Text Entry in Immersive Head-Mounted Display-Based Virtual Reality Using Standard Keyboards. In 2018 IEEE Conference on Virtual Reality and 3D User Interfaces (VR). 159–166. doi:10.1109/VR.2018.8446059DOI
  38. [38]Christina Gummesson, Michael M Ward, and Isam Atroshi. 2006. The shortened disabilities of the arm, shoulder and hand questionnaire (Quick DASH): validity and reliability based on responses within the full-length DASH. BMC musculoskeletal disorders 7, 1 (2006), 44.pubmed.ncbi.nlm.nih.gov/16709254
  39. [39]Jeffrey T Hansberger, Chao Peng, Shannon L Mathis, Vaidyanath Areyur Shanthakumar, Sarah C Meacham, Lizhuo Cao, and Victoria R Blakely. 2017. Dispelling the gorilla arm syndrome: the viability of prolonged gesture interactions. In International conference on virtual, augmented and mixed reality. Springer, 505–520.DOI
  40. [40]Simon Harper. 2007. Is there design-for-all? Universal Access in the Information Society 6, 1 (2007), 111–113.DOI
  41. [41]Sandra G. Hart and Lowell E. Staveland. 1988. Development of NASA-TLX (Task Load Index): Results of Empirical and Theoretical Research. In Advances in Psychology, Peter A. Hancock and Najmedin Meshkati (Eds.). Human Mental Workload, Vol. 52. North-Holland, 139–183. doi:10.1016/S0166-4115(08)62386-9
  42. [42]Sandra G. Hart and Lowell E. Staveland. 1988. Development of NASA-TLX (Task Load Index): Results of Empirical and Theoretical Research. In Advances in Psychology, Peter A. Hancock and Najmedin Meshkati (Eds.). Human Mental Workload, Vol. 52. North-Holland, 139–183. doi:10.1016/S0166-4115(08)62386-9DOI
  43. [43]Donald Hedeker and Robert D. Gibbons. 1994. A Random-Effects Ordinal Regression Model for Multilevel Analysis. Biometrics 50, 4 (1994), 933–944. jstor:2533433 doi:10.2307/2533433DOI
  44. [44]Sture Holm. 1979. A Simple Sequentially Rejective Multiple Test Procedure. Scandinavian Journal of Statistics 6, 2 (1979), 65–70. jstor:4615733 https://www.jstor.org/stable/4615733DOI
  45. [45]Eric Horvitz. 1999. Principles of Mixed-Initiative User Interfaces. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems (CHI ’99). Association for Computing Machinery, New York, NY, USA, 159–166. doi:10.1145/302979.303030DOI
  46. [46]Sujin Jang, Wolfgang Stuerzlinger, Satyajit Ambike, and Karthik Ramani. 2017. Modeling cumulative arm fatigue in mid-air interaction based on perceived exertion and kinetics of arm motion. In Proceedings of the 2017 CHI Conference on Human Factors in Computing Systems. 3328–3339.
  47. [47]Junhan Kong, Jacob O Wobbrock, Tianyuan Cai, and Zoya Bylinskii. 2025. Supporting mobile reading while walking with automatic and customized font size adaptations. In Proceedings of the 2025 CHI Conference on Human Factors in Computing Systems. 1–13.DOI
  48. [48]Junhan Kong, Mingyuan Zhong, James Fogarty, and Jacob O Wobbrock. 2022. Quantifying Touch: New Metrics for Characterizing What Happens During a Touch. In Proceedings of the 24th International ACM SIGACCESS Conference on Computers and Accessibility. 1–13.DOI
  49. [49]Junhan Kong, Mingyuan Zhong, James Fogarty, and Jacob O Wobbrock. 2024. The Ability-Based Design Mobile Toolkit (ABD-MT): Developer Support for Runtime Interface Adaptation Based on Users’ Abilities. Proceedings of the ACM on Human-Computer Interaction 8, MHCI (2024), 1–26.
  50. [50]Rachel L. Franz, Sasa Junuzovic, and Martez Mott. 2021. Nearmi: A framework for designing point of interest techniques for VR users with limited mobility. In Proceedings of the 23rd International ACM SIGACCESS Conference on Computers and Accessibility. 1–14.DOI
  51. [51]Franklin Mingzhe Li, Michael Xieyang Liu, Yang Zhang, and Patrick Carrington. 2022. Freedom to Choose: Understanding Input Modality Preferences of People with Upper-body Motor Impairments for Activities of Daily Living. In Proceedings of the 24th International ACM SIGACCESS Conference on Computers and Accessibility. 1–16.
  52. [52]Jingjin Li, Chao Chen, Mostafa Rahimi Azghadi, Hossein Ghodosi, Lei Pan, and Jun Zhang. 2023. Security and privacy problems in voice assistant applications: A survey. Computers & Security 134 (2023), 103448.
  53. [53]R. C. Littell, P. R. Henry, and C. B. Ammerman. 1998. Statistical Analysis of Repeated Measures Data Using SAS Procedures. Journal of Animal Science 76, 4 (April 1998), 1216–1231. doi:10.2527/1998.7641216xDOI
  54. [54]Xueshi Liu, Dengfy Yu, Hai-Ning Liang, Wenge Xu, Yuzheng Chen, Xiang Li, and Khalad Hasan. 2020. Exploration of Hands-free Text Entry Techniques for Virtual Reality. In 2020 IEEE International Symposium on Mixed and Augmented Reality (ISMAR). 344–349. doi:10.1109/ISMAR50242.2020.00061
  55. [55]Kelly Mack, Emma J McDonnell, Leah Findlater, and Heather D Evans. 2022. Chronically under-addressed: Considerations for HCI accessibility practice with chronically ill people. In Proceedings of the 24th International ACM SIGACCESS conference on computers and accessibility. 1–15.DOI
  56. [56]I. Scott MacKenzie. 2002. A Note on Calculating Text Entry Speed. http://www.yorku.ca/mack/RN-TextEntrySpeed.html.
  57. [57]I Scott MacKenzie and R William Soukoreff. 2003. Phrase sets for evaluating text entry techniques. In CHI’03 extended abstracts on Human factors in computing systems. 754–755.DOI
  58. [58]Paul P Maglio, Rob Barrett, Christopher S Campbell, and Ted Selker. 2000. SUITOR: An attentive information system. In Proceedings of the 5th international conference on Intelligent user interfaces. 169–176.
  59. [59]Päivi Majaranta and Kari-Jouko Räihä. 2002. Twenty years of eye typing: systems and design issues. In Proceedings of the 2002 Symposium on Eye Tracking Research & Applications (ETRA ’02). Association for Computing Machinery, New York, NY, USA, 15–22. doi:10.1145/507072.507076DOI
  60. [60]Meethu Malu, Pramod Chundury, and Leah Findlater. 2018. Exploring accessible speech input interfaces for people with upper body motor impairments. In Proceedings of the 2018 CHI Conference on Human Factors in Computing Systems. 1–12.
  61. [61]Alexander Mariakakis, Mayank Goel, Md Tanvir Islam Aumi, Shwetak N. Patel, and Jacob O. Wobbrock. 2015. SwitchBack: Using Focus and Saccade Tracking to Guide Users’ Attention for Mobile Task Resumption. In Proceedings of the 33rd Annual ACM Conference on Human Factors in Computing Systems (CHI ’15). Association for Computing Machinery, New York, NY, USA, 2953–2962. doi:10.1145/2702123.2702539DOI
  62. [62]Claire L Mitchell, Gabriel J Cler, Susan K Fager, Paola Contessa, Serge H Roy, Gianluca De Luca, Joshua C Kline, and Jennifer M Vojtech. 2022. Ability-based methods for personalized keyboard generation. Multimodal technologies and interaction 6, 8 (2022), 67.DOI
  63. [63]Claire L Mitchell, Junhyeon Kong, Jesse J Martinez, Shaun K Kane, Amy J Ko, Alexis Hiniker, and Jacob O Wobbrock. 2026. Ability Heuristics for Conducting Accessibility Inspections. In Proceedings of the 2026 CHI Conference on Human Factors in Computing Systems. 1–15.DOI
  64. [64]Martez Mott, John Tang, Shaun Kane, Edward Cutrell, and Meredith Ringel Morris. 2020. “I just went into it assuming that I wouldn’t be able to have the full experience”: Understanding the Accessibility of Virtual Reality for People with Limited Mobility. In Proceedings of the 22nd International ACM SIGACCESS Conference on Computers and Accessibility (Virtual Event, Greece (ASSETS ’20). Association for Computing Machinery, New York, NY, USA, Article 43, 13 pages. doi:10.1145/3373625.3416998DOI
  65. [65]Marteze E. Mott, Radu-Daniel Vatavu, Shaun K Kane, and Jacob O Wobbrock. 2016. Smart touch: Improving touch accuracy for people with motor impairments with template matching. In Proceedings of the 2016 CHI conference on human factors in computing systems. 1934–1946.
  66. [66]Martez E. Mott and Jacob O. Wobbrock. 2019. Cluster Touch: Improving Touch Accuracy on Smartphones for People with Motor and Situational Impairments. In Proceedings of the 2019 CHI Conference on Human Factors in Computing Systems (CHI ’19). Association for Computing Machinery, New York, NY, USA, 1–14. doi:10.1145/3290605.3300257DOI
  67. [67]Aunnoy K Mutasim, Mohammad Raihanul Bashar, Christof Leutteroth, Anil Ufuk Batmaz, and Wolfgang Stuerzlinger. 2025. There Is More to Dwell Than Meets the Eye: Toward Better Gaze-Based Text Entry Systems With Multi-Threshold Dwell. In Proceedings of the 2025 CHI Conference on Human Factors in Computing Systems (CHI ’25). Association for Computing Machinery, New York, NY, USA, Article 1088, 18 pages. doi:10.1145/3706598.3713781
  68. [68]Obianuju Okafor. 2023. Voice vs Mouse: The Input Preference of People with Upper-Limb Impairments in a Multi-modal Block-Based Programming Environment. In International Conference on Human-Computer Interaction. Springer, 183–191.
  69. [69]Eduardo GQ Palmeira, Alexandre Campos, Ígor A Moraes, Alexandre G De Sequeira, and Marcelo GG Ferreira. 2023. Quantifying the ’Gorilla Arm’ Effect in a Virtual Reality Text Entry Task via Ray-Casting: A Preliminary Single-Subject Study. In Proceedings of the 25th Symposium on Virtual and Augmented Reality. 274–278.DOI
  70. [70]Sinno Jialin Pan and Qiang Yang. 2009. A survey on transfer learning. IEEE Transactions on knowledge and data engineering 22, 10 (2009), 1345–1359.DOI
  71. [71]Yesaya Tommy Paulus and Gerard Bastiaan Remijn. 2021. Usability of various dwell times for eye-gaze-based object selection with eye tracking. Displays 67 (2021), 101997.DOI
  72. [72]Hamza Peracha, Carrina Iacobacci, Tyler Singer-Clark, Leigh R Hochberg, Sergey D. Stavisky, David M Brandman, and Nicholas S Card. 2026. A personalized and adaptable user interface for a speech and cursor brain-computer interface. In Proceedings of the 2026 CHI Conference on Human Factors in Computing Systems. 1–12.
  73. [73]Ken Pfeuffer, Benedikt Mayer, Diako Mardanbegi, and Hans Gellersen. 2017. Gaze+ pinch interaction in virtual reality. In Proceedings of the 5th symposium on spatial user interaction. 99–108.DOI
  74. [74]Sebastian Pick, Andrew S. Puika, and Torsten W. Kuhlen. 2016. SWIFTER: Design and evaluation of a speech-based text input metaphor for immersive virtual environments. In 2016 IEEE Symposium on 3D User Interfaces (3DUI). 109–112. doi:10.1109/3DUI.2016.7460039
  75. [75]Thadammir Piumsomboon, Adrian Clark, Mark Billinghurst, and Andy Cockburn. 2013. User-defined gestures for augmented reality. In CHI’13 Extended Abstracts on Human Factors in Computing Systems. 955–960.
  76. [76]Lauren Pococke, Crescent Jicol, Christof Lutteroth, and Christopher Clarke. 2025. How Accessible are Virtual Reality Freehand Gestures? Understanding Barriers for Users with Upper Limb Motor Impairments. In Proceedings of the 27th International ACM SIGACCESS Conference on Computers and Accessibility (ASSETS ’25). Association for Computing Machinery, New York, NY, USA, Article 4, 18 pages. doi:10.1145/3663547.3746402DOI
  77. [77]Ivan Poupyrev, Tadao Ichikawa, Suzanne Weghorst, and Mark Billinghurst. 1998. Electric object manipulation in virtual environments: empirical evaluation of interaction techniques. In Computer graphics forum, Vol. 17. Wiley Online Library, 41–52.DOI
  78. [78]Vijay Rajanna and John Paulin Hansen. 2018. Gaze typing in virtual reality: impact of keyboard design, selection method, and motion. In Proceedings of the 2018 ACM Symposium on Eye Tracking Research & Applications (Warsaw, Poland) (ETRA ’18). Association for Computing Machinery, New York, NY, USA, Article 15, 10 pages. doi:10.1145/3204493.3204541DOI
  79. [79]Lucas Rosenblatt, Patrick Carrington, Kotaro Hara, and Jeffrey P Bigham. 2018. Vocal programming for people with upper-body motor impairments. In Proceedings of the 15th International Web for All Conference. 1–10.DOI
  80. [80]Quentin Roy, Futian Zhang, and Daniel Vogel. 2019. Automation Accuracy Is Good, but High Controllability May Be Better. In Proceedings of the 2019 CHI Conference on Human Factors in Computing Systems (CHI ’19). Association for Computing Machinery, New York, NY, USA, 1–8. doi:10.1145/3290605.3300750
  81. [81]Sherry Ruan, Jacob O. Wobbrock, Kenny Liu, Andrew Ng, and James A. Landay. 2018. Comparing Speech and Keyboard Text Entry for Short Messages in Two Languages on Touchscreen Phones. Proc. ACM Interact. Mob. Wearable Ubiquitous Technol. 1, 4, Article 159 (Jan. 2018), 23 pages. doi:10.1145/3161187DOI
  82. [82]Sayan Sarcar, Jussi Jokinen, Antti Oulasvirta, Xiangshi Ren, Chaklam Silpasuwanchai, and Zhenxin Wang. 2017. Ability-based optimization: Designing smartphone text entry interface for older adults. In IFIP Conference on Human-Computer Interaction. Springer, 326–331.DOI
  83. [83]Sayan Sarcar, Jussi PP Jokinen, Antti Oulasvirta, Zhenxin Wang, Chaklam Silpasuwanchai, and Xiangshi Ren. 2018. Ability-based optimization of touchscreen interactions. IEEE Pervasive Computing 17, 1 (2018), 15–26.DOI
  84. [84]Sayan Sarcar, Jussi Jokinen, Antti Oulasvirta, Chaklam Silpasuwanchai, Zhenxin Wang, and Xiangshi Ren. 2016. Towards ability-based optimization for aging users. In Proceedings of the International Symposium on Interactive Technology and Aging Populations. 77–86.DOI
  85. [85]Ben Shneiderman. 2003. Universal usability: Pushing human–computer interaction research to empower every citizen. In Media Access. Routledge, 275–286.
  86. [86]Pitch Sinlapakunatkul, Katlyn S Skilton, Jose N Mathew, and Barbara S Chaparro. 2022. The effects of background noise on user experience and performance of mixed reality voice dictation. In Proceedings of the Human Factors and Ergonomics Society Annual Meeting, Vol. 66. SAGE Publications Sage CA: Los Angeles, CA, 1028–1032.
  87. [87]R. William Soukoreff and I. Scott MacKenzie. 2003. Metrics for text entry research: an evaluation of MSD and KSPC, and a new unified error metric. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems (Ft. Lauderdale, Florida, USA) (CHI ’03). Association for Computing Machinery, New York, NY, USA, 113–120. doi:10.1145/642611.642632DOI
  88. [88]Constantine Stephanidis. 2001. Adaptive Techniques for Universal Access. User Modeling and User-Adapted Interaction 11, 1 (March 2001), 159–179. doi:10.1023/A:1011144232235DOI
  89. [89]Student. 1908. The probable error of a mean. Biometrika (1908), 1–25.
  90. [90]Wolfgang Stuertzlinger, Olivier Chapuis, Dusty Phillips, and Nicolas Russell. 2006. User Interface Façades: Towards Fully Adaptable User Interfaces. In Proceedings of the 19th Annual ACM Symposium on User Interface Software and Technology (UIST ’06). Association for Computing Machinery, New York, NY, USA, 309–318. doi:10.1145/1166253.1166301
  91. [91]Bernhard Suhm, Brad Myers, and Alex Waibel. 2001. Multimodal error correction for speech user interfaces. ACM transactions on computer-human interaction (TOCHI) 8, 1 (2001), 60–98.
  92. [92]Jingze Tian, Yingna Wang, Keye Yu, Liyi Xu, Junan Xie, Franklin Mingzhe Li, Yafeng Niu, and Mingming Fan. 2024. Designing Upper-Body Gesture Interaction with and for People with Spinal Muscular Atrophy in VR. In Proceedings of the 2024 CHI Conference on Human Factors in Computing Systems (Honolulu, HI, USA) (CHI ’24). Association for Computing Machinery, New York, NY, USA, Article 60, 19 pages. doi:10.1145/3613904.3642884DOI
  93. [93]Mindaugas Vasiljevas, Tadas Gedminas, A Ševčenko, M Jančiukas, Thomas Blažauskas, and Rober tas Damaševičius. 2016. Modelling eye fatigue in gaze spelling task. In 2016 IEEE 12th International Conference on Intelligent Computer Communication and Processing (ICCP). IEEE, 95–102.DOI
  94. [94]Radu-Daniel Vatavu, Lisa Anthony, and Jacob O Wobbrock. 2012. Gestures as point clouds: a $ P recognizer for user interface prototypes. In Proceedings of the 14th ACM international conference on Multimodal interaction. 273–280.DOI
  95. [95]Radu-Daniel Vatavu, Lisa Anthony, and Jacob O Wobbrock. 2018. Q $ A super-quick, articulation-invariant stroke-gesture recognizer for low-resource devices. In Proceedings of the 20th International Conference on Human-Computer Interaction with Mobile Devices and Services. 1–12.DOI
  96. [96]Uta Wagner, Mathias N Lystbæk, Pavel Manakhov, Jens Emil Sloth Grønbæk, Ken Pfeuffer, and Hans Gellersen. 2023. A Fitts’ law study of gaze-hand alignment for selection in 3d user interfaces. In Proceedings of the 2023 CHI Conference on Human Factors in Computing Systems. 1–15.DOI
  97. [97]Johann Wentzel, Sasa Junuzovic, James Devine, John Porter, and Martez Mott. 2022. Understanding how people with limited mobility use multi-modal input. In Proceedings of the 2022 CHI Conference on Human Factors in Computing Systems. 1–17.DOI
  98. [98]Johann Wentzel, Alessandra Luz, Martez E Mott, and Daniel Vogel. 2025. MotionBlocks: Modular Geometric Motion Remapping for More Accessible Upper Body Movement in Virtual Reality. In Proceedings of the 2025 CHI Conference on Human Factors in Computing Systems. 1–16.DOI
  99. [99]Brady T. West, Kathleen B. Welch, and Andrzej T. Galecki. 2014. Linear Mixed Models: A Practical Guide Using Statistical Software (2nd ed.). CRC Press, Boca Raton, FL.
  100. [100]Frank Wilcoxon. 1945. Individual comparisons by ranking methods. Biometrical bulletin 1, 6 (1945), 80–83.
  101. [101]Evan James Williams. 1949. Experimental designs balanced for the estimation of residual effects of treatments. Australian Journal of Scientific Research Series A: Physical Sciences 2, 2 (1949), 149–168.DOI
  102. [102]Jacob O Wobbrock, Htet Htet Aung, Brandon Rothrock, and Brad A Myers. 2005. Maximizing the guessability of symbolic input. In CHI’05 extended abstracts on Human Factors in Computing Systems. 1869–1872.DOI
  103. [103]Jacob O. Wobbrock, Krzysztof Z. Gajos, Shaun K. Kane, and Gregg C. Vanderheiden. 2018. Ability-Based Design. Commun. ACM 61, 6 (May 2018), 62–71. doi:10.1145/3148051DOI
  104. [104]Jacob O. Wobbrock, Shaun K. Kane, Krzysztof Z. Gajos, Susumu Harada, and Jon Froehlich. 2011. Ability-Based Design: Concept, Principles and Examples. ACM Trans. Access. Comput. 3, 3 (April 2011), 9:1–9:27. doi:10.1145/1952383.1952384DOI
  105. [105]Jacob O Wobbrock, Meredith Ringel Morris, and Andrew D Wilson. 2009. User-defined gestures for surface computing. In Proceedings of the SIGCHI conference on human factors in computing systems. 1083–1092.DOI
  106. [106]Jacob O Wobbrock, Andrew D Wilson, and Yang Li. 2007. Gestures without libraries, toolkits or training: a $1 recognizer for user interface prototypes. In Proceedings of the 20th annual ACM symposium on User interface software and technology. 159–168.
  107. [107]Wenge Xu, Hai-Ning Liang, Yuxuan Zhao, Tianyu Zhang, Difeng Yu, and Diego Monteiro. 2019. RingText: Well-free and hands-free Text Entry for Mobile Head-Mounted Displays using Head Motions. IEEE Transactions on Visualization and Computer Graphics 25, 5 (2019), 1991–2001. doi:10.1109/TVCG.2019.2898736DOI
  108. [108]Xuhai Xu, Jun Gong, Carolina Brum, Lilian Liang, Bongsoo Suh, Shivam Kumar Gupta, Yash Agarwal, Laurence Lindsey, Runchang Kang, Behrooz Shahsavar, et al. 2022. Enabling hand gesture customization on wrist-worn devices. In Proceedings of the 2022 CHI Conference on Human Factors in Computing Systems. 1–19.DOI
  109. [109]Momona Yamagami, Sasa Junuzovic, Mar Gonzalez-Franco, Eyal Ofek, Edward Cutrell, John R Porter, Andrew D Wilson, and Martez E Mott. 2022. Two-in-one: A design space for mapping unimanual input into bimanual interactions in vrDOI
  110. [110]Momona Yamagami, Alexandra A Portnova-Fahreeva, Junhan Kong, Jacob O Wobbrock, and Jennifer Mankoff. 2023. How do people with limited movement personalize upper-body gestures? considerations for the design of personalized and accessible gesture interfaces. In Proceedings of the 25th International ACM SIGACCESS Conference on Computers and Accessibility. 1–15.DOI
  111. [111]Xin Yi, Chun Yu, Mingrui Zhang, Sida Gao, Ke Sun, and Yuanchun Shi. 2015. ATK: Enabling Ten-Finger Freehand Typing in Air Based on 3D Hand Tracking Data. In Proceedings of the 28th Annual ACM Symposium on User Interface Software & Technology (Charlotte, NC, USA) (UIST ’15). Association for Computing Machinery, New York, NY, USA, 539–548. doi:10.1145/2807442.2807504DOI
  112. [112]Victoria Young and Alex Mihailidis. 2010. Difficulties in automatic speech recognition of dysarthric speakers and implications for speech-based applications used by the elderly: A literature review. Assistive Technology 22, 2 (2010), 99–112.DOI
  113. [113]Chun Yu, Yizheng Gu, Zhican Yang, Xin Yi, Hengliang Luo, and Yuanchun Shi. 2017. Dwell or Gesture? Exploring Head-Based Text Entry Techniques for HMDs. In Proceedings of the 2017 CHI Conference on Human Factors in Computing Systems (Denver, Colorado, USA) (CHI ’17). Association for Computing Machinery, New York, NY, USA, 4479–4488. doi:10.1145/3025453.3025964DOI
  114. [114]Mingrui Ray Zhang and Jacob O Wobbrock. 2019. Beyond the input stream: Making text entry evaluations more flexible with transcription sequences. In Proceedings of the 32nd Annual ACM Symposium on User Interface Software and Technology. 831–842.DOI
  115. [115]Mingrui Ray Zhang, Shumin Zhai, and Jacob O Wobbrock. 2019. Text entry throughput: Towards unifying speed and accuracy in a single performance metric. In Proceedings of the 2019 CHI conference on human factors in computing systems. 1–13.DOI
  116. [116]Maozheng Zhao, Alec M Pierce, Ran Tan, Ting Zhang, Tianyi Wang, Tanya R. Jonker, Hrvoje Benko, and Aakar Gupta. 2023. Gaze Speedup: Eye Gaze Assisted Gesture Typing in Virtual Reality. In Proceedings of the 28th International Conference on Intelligent User Interfaces (Sydney, NSW, Australia) (IUI ’23). Association for Computing Machinery, New York, NY, USA, 595–606. doi:10.1145/3581641.3584072DOI
  117. [117]Xuan Zhao, Mingming Fan, and Teng Han. 2022. “I Don’t Want People to Look At Me Differently”: Designing User-Defined Above-the-Neck Gestures for People with Upper Body Motor Impairments. In Proceedings of the 2022 CHI Conference on Human Factors in Computing Systems (New Orleans, LA, USA) (CHI ’22). Association for Computing Machinery, New York, NY, USA, Article 1, 15 pages. doi:10.1145/3491102.3517552DOI

Paper details

vlnc:W_zp68wt78
Version
v1 · latest
Published
Fresh — added to the Hub within the last 30 days
Added
License
CC BY 4.0
Usage

views · 0

downloads · 0

A view counts once per browser per day, a download each time the PDF is requested.

  • Computer Science

Not recorded for this paper

View statement
Looking for related papers…
Contents