betaFeedback

WorksComputer Science

VoiceGuard: An Audio-Native LLM-VLA Pipeline for Wheelchair-Mounted Assistive Manipulation

Abstract

Wheelchair-mounted robotic arms can help people with limited upper-limb mobility reach and manipulate everyday objects, but operating them typically requires stepwise joystick control and frequent mode switching. Natural-language interfaces could let users express task-level goals, yet they must handle ambiguous requests, distinguish commands from nearby conversation, and support interruption. We present VoiceGuard, a voice interface that separates spoken intent from robot execution through structured request validation and a local stop mechanism. On-device detectors handle the wake phrase and stop commands, an audio-native large language model (LLM) routes each request into a structured function packet, a rule-based ToolDispatcher accepts, rejects, or clarifies the packet, and a SmolVLA vision-language-action (VLA) policy executes accepted tasks on a low-cost SO-101 arm. Preliminary tests showed that the system correctly handled 55 of 60 voice commands, including ambiguous and non-directed utterances, completed 16 of 20 deliveries, and stopped the arm on all 20 stop commands with a median latency of 210 ms. We demonstrate the feasibility of combining flexible spoken input with constrained task execution and local interruption for wheelchair-mounted assistive manipulation.

CCS Concepts: • Human-centered computing → Accessibility systems and tools; Interaction techniques.

Additional Key Words and Phrases: assistive robotics, wheelchair-mounted robotic arm, voice interaction, large language models, vision-language-action models

Authors’ Contact Information: Bang Xiao, NYU Shanghai, Shanghai, China, [email protected]; Rodolfo Cossovich, Carleton University, Ottawa, Canada, [email protected].

Introduction

Wheelchair-mounted robotic arms help people with limited upper-limb mobility reach and grasp nearby objects [21]. While this technology has the potential to reduce the caregiving time needed for daily tasks, their user interfaces are typically driven through a joystick one axis and one mode at a time [18]. Even performing a simple task like picking up a cup requires the users to steer the gripper along each axis, switch modes to rotate the wrist, and switch again to close the gripper.

With the introduction of Large Language Models (LLMs) and Vision-Language-Action (VLA) Models, task-level control makes robotic arms control more feasible [17, 22]. In assistive use, however, the arm moves toward the user’s body while the user often cannot physically intervene [2, 24], and speech from caregivers reaches the microphone as easily as the user’s. A voice interface for this setting therefore needs a checkable step between language and motion, and a way to stop the arm at any moment.

We present VoiceGuard, a voice-controlled system for a wheelchair-mounted robotic arm that validates every request before execution. This paper makes three contributions: (1) an interface architecture design that separates spoken intent from arm motion through an on-device stop path and a validated, logged packet; (2) a working prototype integrating the proposed architecture on a low-cost SO-101 arm mounted on a wheelchair; and (3) preliminary tests of routing, delivery, and stopping performance.

Related Work

Research with people with physical disabilities has shown that retrieving everyday objects ranks among their highest priorities in their daily life activities [4, 7, 11]. Commercial wheelchair-mounted arms address their interface using a joystick [15, 21].

Intelligent Interfaces for assistive arms: Existing interfaces reduce the burden of joystick control through shared autonomy, target selection, and speech. Shared-autonomy systems automate mode switching [10] or infer the user’s goal [13], but keep the user on the joystick. Selection-based systems let users pick a target by eye gaze or pointing [1, 6, 12, 19]. LLM-based speech interfaces for feeding [26] and joint wheelchair and arm control [17] accept free-form requests. Our system differs by placing validation and stopping outside the model.

Language and VLA models for robot action: Technically, our work builds on LLM task planning and VLA policies. SayCan, Code as Policies, and ProgPrompt use LLMs to select or program robot tasks [3, 16, 23], and VLA models such as RT-2, OpenVLA, and SmolVLA map an instruction and camera images to actions [5, 14, 22]. These approaches were mostly evaluated in tabletop settings, where model output is executed directly. We adapt them for assistive use by restricting the LLM to a fixed function schema and giving each VLA policy only the task instruction it was trained on.

Safe motion near the user: Robots that work next to people rely on low-level protections such as collision detection from joint torque or motor current and limits on speed, force, and power [9, 25]. These protections respond to contact or exceeded limits, and do not capture what the user wants the arm to do. Our work adds a software layer above them: the ToolDispatcher bounds each task before the arm moves, and the local stop lets the user halt motion by voice at any point.

System Design

We designed the system around four goals: to let users phrase task-level requests in their own words (DG1); to check, reject, or clarify every request before the arm moves, and log it (DG2); to keep stopping independent of network or model failures (DG3); and to send audio off the device only after the wake phrase, so that nearby conversations never reach the cloud model (DG4). We realize these goals with six components (Figure 1). The local gate listens on the device for the wake phrase and for stop commands. After the wake phrase, the audio-native router turns the spoken request into a structured function packet. The ToolDispatcher checks the packet and decides whether it may run. If it passes, the SmolVLA policy generates arm actions, and the ArmController sends them to the SO-101 arm.

Figure 1. Pipeline overview. After the wake phrase, the Router turns the request into a Packet, the ToolDispatcher checks it, and SmolVLA and the ArmController carry it out. At any moment, “stop” at the Local Gate skips this path and halts the arm.

Audio-Native Routing

The local gate runs two detectors on the device: a wake-phrase detector, which we set to “Hey Rudi,” that opens a streaming session to an audio-native LLM (DG4), and a stop and pause detector that stays always on. The LLM, Gemini 3.5 Flash-Lite in our prototype, returns a function call over a fixed schema of supported functions (DG1); any audio-native model with function calling can replace it. We route raw audio instead of a transcript, which removes a separate speech-to-text stage and its errors on short commands.

The function call becomes a packet with six fields (Figure 1, top right). For “bring the cup up so I can drink,” the router fills target=cup and destination=mouth_pose, names the action module, lists the checks the packet must pass, and records a transcript so that each decision can be audited (DG2). “Bring that” yields a clarification request, and a remark addressed to a caregiver yields no packet.

Validation

The ToolDispatcher applies five checks in order: (1) the function and fields match the schema; (2) the target and destination are present and unambiguous; (3) a trained module exists and the task is authorized; (4) the controller is ready, the target is visible, and no interruption is active; and (5) step size, speed, and reach stay within configured limits. A packet that passes all checks is accepted; a missing or ambiguous argument returns a clarification request, and any other failure is rejected without motion. Every packet and outcome is logged.

Execution and Stop

An accepted packet is converted into the fixed task instruction its module was trained on, so the policy never receives open-ended speech. Each module is a SmolVLA policy [22] fine-tuned for one target and destination pair. The policy takes the instruction together with wrist and agent camera images and outputs action chunks, which the ArmController converts into LeRobot commands (move, grasp, release, home) for the SO-101 arm. Stop and pause go from the local gate straight to the ArmController, so the arm halts without a network round trip or a model call (DG3).

Prototype and Interaction

The prototype mounts an SO-101 arm beside a wheelchair tray, with a wrist camera and a fixed agent camera (Figure 2). The local gate runs openWakeWord on a laptop, and the router uses the Gemini Live API with Gemini 3.5 Flash-Lite. Each SmolVLA module was fine-tuned on 50 teleoperated demonstrations and runs on an RTX 4070 GPU through LeRobot. A delivery takes one utterance (Figure 2). “Stop” or “pause” halts the arm at any point, and “bring that” leads to a spoken prompt to name the object.

Figure 2. A delivery on the physical prototype, from the ready pose (a) to the requested pose (e).

Preliminary Evaluation

We tested the prototype’s three core functions (Table 1). For command routing, the first author said 10 phrasings for each of four supported requests plus 20 ambiguous or non-directed utterances; a command counted as correct when it produced the expected packet, clarification request, or no packet. Deliveries used the setup in Figure 2. For stopping, we used the word “stop” while the arm approached the target, and we measured latency from the end of the utterance to the arm’s last joint motion. Most failed deliveries were missed grasps, and delivery success varied with lighting and background, as reported for other VLA policies [8, 20].

FunctionTrialsResult
Command routing6055/60 correct packet, clarification, or no packet
Physical execution2016/20 deliveries completed
Stop mechanism2020/20 halted; median latency 210 ms

Table 1. Preliminary tests on the physical prototype.

Conclusion and Future Work

We presented VoiceGuard, a voice interface for a wheelchair-mounted robotic arm that turns spoken requests into validated packets and keeps stop on the device; in our prototype, it halted the arm on every stop command. More broadly, by treating language-model output as a proposal to check before execution, such systems can help assistive robotics move from step-by-step teleoperation toward task-level requests the user still controls.

Our tests involved lab members running trained routines on a fixed tray setup. A study with wheelchair users is now under ethics review and will examine how they phrase requests and respond to clarification prompts. We also plan to run routing on the device, removing the need for a network connection, and to improve the robustness of the VLA modules to the lighting and background changes that lowered delivery success in our tests.

References

  1. [1]R. A. M. Abyasiri, A. G. B. P. Jayasekara, R. A. R. C. Gopura, and Kazuo Kiguchi. 2024. Intelligent Object Manipulation for a Wheelchair-Mounted Robotic Arm. Journal of Robotics 2024 (2024), 1516737. doi:10.1155/2024/1516737
  2. [2]Adedire D. Adesiji, Segun E. Ibitoye, Rasheedat M. Mahamood, Olalekan A. Olayemi, Peter O. Omoniyi, Tien-Chien Jen, and Esther T. Akinlabi. 2026. Safety Considerations in Deployment of Robotic Systems – A Systematic Review. Journal of Field Robotics 43, 1 (2026), 5–33. doi:10.1002/rob.70022DOI
  3. [3]Michael Ahn, Anthony Brohan, Noah Brown, Yevgen Chebotar, Omar Cortes, Byron David, Chelsea Finn, Chuyuan Fu, Keerthana Gopalakrishnan, Karol Hausman, Alex Herzog, Daniel Ho, Jasmine Hsu, Julian Ibarz, Brian Ichter, Alex Irpan, Eric Jang, Rosario Jauregui Ruzon, Kyle Jeffrey, Sally Jesmonth, Nikhil J. Joshi, Ryan Julian, Dmitry Kalashnikov, Yuheng Kuang, Kuang-Huei Lee, Sergey Levine, Yao Lu, Linda Luu, Carolina Parada, Peter Pastor, Jornell Quiambao, Kanishka Rao, Jarek Rettinghouse, Diego Reyes, Pierre Sermanet, Nicolas Sievers, Clayton Tan, Alexander Toshev, Vincent Vanhoucke, Fei Xia, Ted Xiao, Peng Xu, Sichun Xu, Mengyuan Yan, and Andy Zeng. 2023. Do As I Can, Not As I Say: Grounding Language in Robotic Affordances. In Proceedings of the Conference on Robot Learning (CoRL).
  4. [4]Kim D. Anderson. 2004. Targeting Recovery: Priorities of the Spinal Cord-Injured Population. Journal of Neurotrauma 21, 10 (2004), 1371–1383. doi:10.1089/neu.2004.21.1371DOI
  5. [5]Anthony Brohan, Noah Brown, Justice Carbajal, Yevgen Chebotar, Xi Chen, Krzysztof Choromanski, Tianli Ding, Danny Driess, Avinava Dubey, Chelsea Finn, Pete Florence, Chuyuan Fu, Montse Gonzalez Arenas, Keerthana Gopalakrishnan, Kehang Han, Karol Hausman, Alexander Herzog, Jasmine Hsu, Brian Ichter, Alex Irpan, Nikhil Joshi, Ryan Julian, Dmitry Kalashnikov, Yuheng Kuang, Isabel Leal, Lisa Lee, Tsang-Wei Edward Lee, Sergey Levine, Yao Lu, Henryk Michalewski, Igor Mordatch, Karl Pertsch, Kanishka Rao, Krista Reymann, Michael Ryoo, Grecia Salazar, Pannag Sanketi, Pierre Sermanet, Jaspiar Singh, Anikait Singh, Radu Soricut, Huong Tran, Vincent Vanhoucke, Quan Vuong, Azyan Wahid, Stefan Welker, Paul Wohlhart, Jialin Wu, Fei Xia, Ted Xiao, Peng Xu, Sichun Xu, Tianhe Yu, and Brianna Zitkovich. 2023. RT-2: Vision-Language-Action Models Transfer Web Knowledge to Robotic Control. In Proceedings of the Conference on Robot Learning (CoRL).
  6. [6]Young Sang Choi et al. 2009. Hand It Over or Set It Down: A User Study of Object Delivery with an Assistive Mobile Manipulator. In IEEE International Conference on Robotics and Automation (ICRA).DOI
  7. [7]Young Sang Choi, Travis Deyle, Tiffany Chen, Jonathan D. Glass, and Charles C. Kemp. 2009. A List of Household Objects for Robotic Retrieval Prioritized by People with PLS. In 2009 IEEE International Conference on Rehabilitation Robotics. 510–517. doi:10.1109/ICORR.2009.5209484DOI
  8. [8]Senyu Fei, Syin Wang, Junhao Shi, Zihao Dai, Jikun Cai, Pengfang Qian, Li Ji, Xinzhe He, Shiduo Zhang, Zhaoye Fei, Jinlan Fu, Jingjing Gong, and Xipeng Qiu. 2025. LIBERO-Plus: In-depth Robustness Analysis of Vision-Language-Action Models. arXiv preprint arXiv:2510.13626 (2025).arxiv.org/abs/2510.13626
  9. [9]Sami Hadjadin, Alessandro De Luca, and Alin Albu-Schäffer. 2017. Robot Collisions: A Survey on Detection, Isolation, and Identification. IEEE Transactions on Robotics 33, 6 (2017), 1292–1312. doi:10.1109/TRO.2017.2723903
  10. [10]Laura V. Herlant, Rachel M. Holladay, and Siddhartha S. Srinivasa. 2016. Assistive Teleoperation of Robot Arms via Automatic Time-Optimal Mode Switching. In ACM/IEEE International Conference on Human-Robot Interaction (HRI). 35–42.DOI
  11. [11]N. Hutmacher, A. Bellwald, R. Rätz, G. Gruener, P. Eichelberger, N. Lutz, V. Steiner, and A. M. Raab. 2025. Identification of Needs for an Assistive Robotic Arm in Individuals with Tetraplegia: A Mixed-Methods Approach. Journal of NeuroEngineering and Rehabilitation 22 (2025), 113. doi:10.1186/s12984-025-01642-8DOI
  12. [12]Advait Jain and Charles C. Kemp. 2010. EL-E: An Assistive Mobile Manipulator that Autonomously Fetches Objects from Flat Surfaces. Autonomous Robots 28, 1 (2010), 45–64.DOI
  13. [13]Siddarth Jain and Brenna Argall. 2019. Probabilistic Human Intent Recognition for Shared Autonomy in Assistive Robotics. ACM Transactions on Human-Robot Interaction 9, 1 (2019).DOI
  14. [14]Moo Jin Kim, Karl Pertsch, Siddharth Karamcheti, Ted Xiao, Ashwin Balakrishna, Suraj Nair, Rafael Rafailov, Ethan Foster, Grace Lam, Pannag Sanketi, Quan Vuong, Thomas Kollar, Benjamin Burchfiel, Russ Tedrake, Dorsa Sadigh, Sergey Levine, Percy Liang, and Chelsea Finn. 2024. OpenVLA: An Open-Source Vision-Language-Action Model. arXiv:2406.09246arxiv.org/abs/2406.09246
  15. [15]Kinova. [n. d.]. Jaco Robotic Arm. https://assistive.kinovarobotics.com/product/jaco-robotic-arm. Accessed: 2026-06-22.
  16. [16]Jacky Liang, Wenlong Huang, Fei Xia, Peng Xu, Karol Hausman, Brian Ichter, Pete Florence, and Andy Zeng. 2023. Code as Policies: Language Model Programs for Embodied Control. In IEEE International Conference on Robotics and Automation (ICRA). 9493–9500.arxiv.org/abs/2209.07753
  17. [17]Guangping Liu, Nicholas Hawkins, Billy Madden, Tipu Sultan, and Madi Babasial. 2026. A Dialogue-Based Human-Robot Interaction Protocol for Wheelchair and Robotic Arm Integrated Control. arXiv:2602.06243arxiv.org/abs/2602.06243
  18. [18]Veronique Maheu, Philippe S. Archambault, Julie Frappier, and François Routhier. 2011. Evaluation of the JACO Robotic Arm: Clinico-Economic Study for Powered Wheelchair Users with Upper-Extremity Disabilities. In 2011 IEEE International Conference on Rehabilitation Robotics. 1–5. doi:10.1109/ICORR.2011.5975397DOI
  19. [19]Anujith Muraleedharan, Anamika J H, Himanshu Vishwakarma, Kudrat Kashyap, and Pradipta Biswas. 2024. Eye-Gaze-Enabled Assistive Robotic Stamp Printing System for Individuals with Severe Speech and Motor Impairment. In Companion Proceedings of the 29th International Conference on Intelligent User Interfaces (IUI Companion ’24). Association for Computing Machinery, New York, NY, USA, 6 pages. doi:10.1145/3640544.3645236DOI
  20. [20]Wilbert Pumacay, Ishika Singh, Jiafei Duan, Ranjay Krishna, Jesse Thomason, and Dieter Fox. 2024. THE COLOSSEUM: A Benchmark for Evaluating Generalization for Robotic Manipulation. arXiv preprint arXiv:2402.08191 (2024).arxiv.org/abs/2402.08191
  21. [21]Md Mahbubur Rahman, Nayan Banik, Md Samiul Haque Sunny, Md Ishrak Islam Zarif, David Bedolla-Martinez, Katie Schultz, Sheikh Iqbal Ahamed, and Mohammad H. Rahman. 2025. Wheelchair-Mounted Robotic Arms: A Systematic Review of Technical Design and Activities of Daily Living Outcomes. Disability and Rehabilitation: Assistive Technology 20, 7 (2025), 2532–2556. doi:10.1080/17483107.2025.2547042DOI
  22. [22]Mustafa Shukor, Dana Aubakirova, Francesco Capuano, Pepijn Kooijmans, Steven Palma, Adil Zouitine, Michel Aractingi, Caroline Pascal, Martino Russi, Andres Marafioti, Simon Alibert, Matthieu Cord, Thomas Wolf, and Remi Cadene. 2025. SmolVLA: A Vision-Language-Action Model for Affordable and Efficient Robotics. arXiv:2506.01844arxiv.org/abs/2506.01844
  23. [23]Ishika Singh, Valts Blukis, Arsalan Mousavian, Ankit Goyal, Danfei Xu, Jonathan Tremblay, Dieter Fox, Jesse Thomason, and Animesh Garg. 2023. ProgPrompt: Generating Situated Robot Task Plans using Large Language Models. In IEEE International Conference on Robotics and Automation (ICRA). 11523–11530.arxiv.org/abs/2209.11302
  24. [24]Milos Vasic and Aude Billard. 2013. Safety Issues in Human-Robot Interactions. In 2013 IEEE International Conference on Robotics and Automation (ICRA). 197–204. doi:10.1109/ICRA.2013.6630576
  25. [25]Valeria Villani, Fabio Pini, Francesco Leali, and Cristian Secchi. 2018. Survey on Human–Robot Collaboration in Industrial Settings: Safety, Intuitive Interfaces and Applications. Mechatronics 55 (2018), 248–266. doi:10.1016/j.mechatronics.2018.02.009DOI
  26. [26]Jessie Yuan, Janavi Gupta, Akhil Padmanabha, Zulehka Karachiwalla, Carmel Majidi, Henny Admoni, and Zackory Erickson. 2024. Towards an LLM-Based Speech Interface for Robot-Assisted Feeding. In Adjunct Proceedings of the 37th Annual ACM Symposium on User Interface Software and Technology (UIST Adjunct). Association for Computing Machinery, New York, NY, USA, 1–4. doi:10.1145/3672539.3686759DOI

Paper details

vlnc:W_xvqthx6z
Version
v1 · latest
Published
Fresh — added to the Hub within the last 30 days
Added
License
CC BY 4.0
Usage

views · 0

downloads · 0

A view counts once per browser per day, a download each time the PDF is requested.

  • Computer Science

Not recorded for this paper

View statement
Looking for related papers…
Contents