betaFeedback

WorksComputer Science

Talk to Act: A Synthetic Multi-Turn Dialogue Benchmark for Human Intent Extraction and Action Planning in Robotics

Abstract

Multi-turn conversational human-robot interaction (HRI) continues to be a challenging problem in robotics, as many existing systems rely on rigid, command-driven interfaces that can be unintuitive for users. Recent advances in Large Language Models (LLMs) provide a promising direction, given their ability to process natural language, generate structured outputs, and support tool-based reasoning, all of which are relevant for interpreting user intent in robotic settings. However, progress in this area is constrained by the limited availability of datasets and benchmarks that capture the nuances of real-world human communication, such as multi-turn dialogue, ambiguity, and context-dependent requests. In this work, we present a synthetically generated multi-turn dialogue dataset for HRI, annotated with action intents, object references, and task plans across a range of scenarios and user personas. Our primary focus is on evaluating the capability of small, open-weight language models for robotic tasks, particularly in settings where local deployment and efficiency are important. To this end, we frame intent extraction as a tool-calling problem where each predicted action corresponds to a structured function call with arguments grounded in the dialogue and benchmark models ranging from 270M to 8B parameters on their ability to perform intent extraction and action planning without task-specific fine-tuning. Our results show that these models can achieve reasonable performance on the task even at relatively small scales and without any task-specific fine-tuning. This suggests that compact, locally deployable models already provide a useful baseline for conversational robotic systems. We expect that task-specific fine-tuning and more realistic training data will enhance performance further and make such models increasingly suitable for real-world robotic applications.

Introduction

Human-Robot Interaction (HRI) has emerged as a critical frontier in robotics, enabling robots to operate alongside humans in shared environments such as assisted living facilities, warehouses, and domestic settings. As robots move from controlled industrial cells into homes, care environments, and shared workspaces, communication becomes a core capability rather than a peripheral interface.

In recent years, substantial progress has been made in robotics, particularly in perception and control. Large embodied datasets and generalist robot policies have advanced robotic performance. Vision-language-action models (VLAs) have demonstrated the effectiveness of web-scale pretraining in enhancing semantic generalization in robotic control. However, interaction between humans and robots is still unnatural. A robot that is capable of perceiving objects and executing movements with precision may still struggle when a user makes an underspecified, multistep, or revised request in everyday language.

Nevertheless, natural language is the most intuitive communication medium for people. Previous studies have contrasted natural language with complex graphical interfaces, structured command languages, and formal abstractions, which require training before a user can effectively control a robot. For robots in homes, assistive scenarios, or shared human environments, it is often unrealistic to expect the human to learn a rigid interaction protocol. A more realistic objective is for robots to be able to interpret natural, unconstrained, and potentially messy human language as it is actually spoken. However, unrestricted language is challenging because human requests are often context-dependent, incomplete, and ambiguous.

Recent advancements in large language models (LLMs) have introduced new opportunities for integrating natural language into robotic tasks. Models such as GPT-5, Claude and Gemini demonstrate strong capabilities in common sense reasoning, following instructions, and tool calling — properties that are directly relevant to interpreting user intent and translating it into structured robot actions. LLMs ability to handle compositional instructions and generate structured outputs makes LLMs a promising foundation for the interaction layer. Furthermore, LLMs offer a promising approach to generating synthetic data, enabling the creation of extensive and diverse dialogue corpora that would otherwise be too costly to collect through human annotation.

Although LLMs have great potential in HRI, existing approaches tend to reduce interaction to single-turn intent extraction within narrowly defined task domains [1]. However, real human communication is multi-turn, context-dependent, and often under-specified. Users revise goals, refer to objects implicitly, and issue instructions that are only interpretable when considering the prior conversational context.

While large frontier models may be attractive for intent understanding due to their strong reasoning capabilities, their deployment in robotic systems is constrained by latency, connectivity requirements, cost, and privacy considerations. In contrast, recent compact open-weight models such as Phi-4, Gemma series, and Qwen series demonstrate that smaller models are becoming increasingly capable at instruction following and structured output generation, making them compelling candidates for on-device or edge-assisted HRI pipelines. Whether such models can reliably handle the full complexity of natural, multi-turn human dialogue in HRI contexts, however, remains an open question. Alongside language, perception capabilities have also advanced significantly. Modern models such as SAM 3 [2] and Grounded DINO [3] now support open-vocabulary segmentation and detection driven by natural language queries. Yet current HRI systems rarely leverage these capabilities, leaving a gap between what perception models can accept and what interaction pipelines actually provide. A well-designed dialogue system that extracts structured referential expressions from human conversation could directly bridge this gap, feeding natural language grounded outputs into modern perception models.

A key bottleneck in advancing natural language-based HRI is the lack of datasets that reflect the complexity of real human communication. Existing benchmarks are limited in scale, domain diversity, and conversational depth. To address this, we introduce a synthetically generated multi-turn dialogue dataset for HRI, designed to capture natural, ambiguous, and context-dependent human instructions annotated with action intent, action plans, and object intent. The dataset spans a variety of real-world scenarios and user personas, reflecting the diversity of people and contexts a robot may encounter in practice. We further present an initial benchmark of small open-weight language models on this dataset, exploring their ability to extract user intent and task plans from unconstrained human dialogue in HRI settings.

Prior work on natural-language HRI spans intent recognition methods, conversational embodied datasets, and LLM/VLM-based robot systems. A recent review of industrial collaborative robotics describes intention recognition as covering rule-based, probabilistic, machine learning, and deep learning approaches utilizing physical, physiological, and contextual cues. The review highlights the importance of understanding human intentions for safe and effective collaboration [4]. Earlier grounded-language work addressed this problem through explicit structured reasoning. Vanzo et al. used a combination of discriminative structured learning, distributional semantics, and contextual evidence from the environment to resolve ambiguities in robotic commands [5]. CoBaIR proposed an explainable Bayesian framework for context-based intention recognition in low-data settings [6]. In physical HRI, Lai et al. inferred user intent from force interactions alone through physical human–robot interaction primitives [7]. These approaches are important because they show that grounding, context, and interpretability are essential, but they are not designed around open-ended, multi-turn conversational language as the primary input for intent extraction and task planning.

Conversation-oriented HRI work pushed the field closer to natural communication, but with different evaluation goals. Papaioannou et al. showed that a hybrid chat-and-task dialogue system can improve perceived interaction quality, yet their main emphasis is adaptive dialogue management for engagement rather than structured extraction of intent and task plans from dialogue [8]. In embodied AI, ALFRED established a benchmark for long-horizon household instruction following from language and egocentric vision, but it is fundamentally organized around directive-to-action mapping rather than conversation [9]. TEACh extends the embodied setting substantially by introducing over 3,000 unconstrained human-human dialogues in simulation, including clarification, correction, and interleaving of language with actions, and defines benchmarks for execution from dialogue history, trajectory from dialogue, and two-agent task completion [10]. These resources are highly relevant to conversational robotics, but they remain oriented toward embodied execution rather than a standalone benchmark with explicit labels for intent extraction and task-planning supervision.

Recent LLM, VLM and VLA work demonstrates the promise of language models in robotics, but mostly from the perspective of control or full-agent behavior. Gao et al. fine-tuned GPT-3.5 to interpret assembly instructions and generate corrective actions in a manufacturing case study [11]. RobotGPT addresses the instability of directly using ChatGPT-generated code for manipulation and instead learns a more reliable manipulation agent from ChatGPT supervision [12]. LLaRA and Hi Robot show that conversation-style supervision and synthetic language data can substantially improve open-ended robot instruction following for vision-language-action policies [13], [14], while Wang et al. develop a tool-augmented industrial HRC agent with memory and reasoning mechanisms for aerospace wire-harness assembly [15]. Collectively, these papers show that language models can support robot reasoning, planning, and interaction, but they do not directly target the question we study here: whether small language models can be used for human robot communication specifically for extracting human intent and producing structured task plans from natural multi-turn HRI dialogue.

A final line of work is especially important for motivating realistic data design. Xu et al. demonstrate that synthetic dialogue generation can scale to 10,000 multi-party conversations for proactive task inference [16]. However, such datasets focus on diverse dialogue between users and do not address intent extraction or task planning in the context of robotics, leaving a gap that our work directly targets. These findings motivate our design choices directly. Our dataset is fully synthetically generated using LLMs and partially verified by human evaluators to ensure quality and naturalness. It consists of multi-turn dialogues spanning diverse scenarios and user personas, with each conversation paired with explicit annotations for action intent, object intent, and task planning, targeting the conversational language layer as a practical problem in HRI.

Method

Synthetic Dataset Generation

We used Qwen 3.6 [17] as the backbone for synthetic data generation; the full pipeline is illustrated in Figure 2.

Overview of the synthetic data generation pipeline

Figure 2. Overview of the synthetic data generation pipeline.

Each conversation consists of at least ten turns between a user and an assistant, spanning everyday chitchat to explicit task-based requests. We formulate intent parsing as a tool-calling problem, where the model must invoke one of four predefined actions, pick, place, pour, and handoff, in response to user requests. Each conversation contains more than one task request, requiring the model to plan multiple actions per conversation while maintaining state across turns. For example, if the robot has already picked an object, it cannot pick again until it has placed it, and the model must track this state throughout the dialogue. The dataset covers a range of scenarios from household tasks to warehouse settings, and includes diverse user personas to reflect realistic interaction conditions. Every generated sample is validated against a predefined JSON schema and then verified and corrected by an LLM to ensure consistent, well-formed annotations. In total, we collected 977 conversations. We release the dataset publicly to facilitate future research in this area.1

Action Schema and Object Intent

We define the following four actions: pick, place, pour, and handoff. Each action is represented by a structured JSON object containing parameters that describe the action type and the objects involved. Object parameters include a name and a full grounding phrase extracted from the conversation. Optional properties include color, shape, material, and texture. The place action also captures the spatial relationship between the target and a reference object. This structured representation is directly motivated by downstream perception requirements. The object name can serve as a query for closed-vocabulary detectors, such as YOLO. Meanwhile, the grounding description, which is a rich, natural language phrase derived from the conversation that can be used as a prompt for open-vocabulary perception models, such as Grounded DINO [3] and SAM 3 [2], which accept free-form text input. Thus, the extracted object intent acts as a bridge between natural human dialogue and modern perception pipelines without requiring manual reformulation of user instructions. An overview of the full pipeline is illustrated in Figure 1.

Overview of the proposed HRI framework

Figure 1. Overview of the proposed HRI framework. A multi-turn conversation between a user and a robot is processed by context-aware intent parsing tools. Ambiguous references (e.g., “that”, “the other one”) are resolved using the dialogue context and action history. The parser outputs action intent (pick, place, pour, handoff) and object intent (target object, source object, properties, and description) as structured JSON.

Data Verification and Correction

The generated conversations were reviewed for logical inconsistencies and minor errors using an LLM-based evaluator based on Qwen 3.6 and corrector pipeline, where a correction prompt was used to identify and fix state tracking violations and incoherent action sequences. After this automated correction pass, the data quality was found to be satisfactory for benchmarking purposes.

Test Case Construction and Evaluation

The dataset is split into training and test sets using a random 80/20 split at the conversation level. The training set is retained for future fine-tuning of small language models. Test cases are generated from the test set conversations, where each test case represents the conversation up to the next tool call. Subsequent test cases from the same conversation include the history of prior turns and ground truth responses, allowing evaluation of model performance under increasing context lengths. We evaluate a range of open-weight small language models spanning from 270 million to 8 billion parameters, assessing their ability to extract user intent and generate correct action plans without any task-specific fine-tuning.

Results

Dataset Statistics

The dataset comprises 977 conversations generated across 40 unique combinations of user persona, tone, and environment. Environments include homes, old age homes, restaurants, offices, supermarkets, educational institutions, airports, and warehouses, covering a broad range of realistic HRI scenarios. Figure 3 shows an example conversation from the dataset, illustrating how ambiguity is resolved across turns before the model issues a structured tool call with full object grounding information.

Example conversation hri_home_office_0058

Figure 3. Example conversation hri_home_office_0058.

Evaluation Metrics

We evaluate model performance using two groups of metrics.

The first is Goal Achievement Accuracy, which measures whether a complete task was accomplished with the correct action plan, in the correct order, and without introducing extra or missing steps. Formally, for a conversation with a ground truth action sequence A∗=(a1∗,a2∗,…,an∗)A^{*} = (a_{1}^{*},a_{2}^{*},\ldots,a_{n}^{*}) and a predicted sequence A^=(a^1,a^2,…,a^m)\hat{A} = (\hat{a}_{1},\hat{a}_{2},\ldots,\hat{a}_{m}), goal achievement accuracy is defined as:

Goal Achievement Accuracy=1N∑i=1N1[A^i=Ai∗](1)\text{Goal Achievement Accuracy} = \frac{1}{N}\sum_{i=1}^{N}\mathbf{1}[\hat{A}_{i}=A_{i}^{*}] \tag*{(1)}

where NN is the total number of instances of task request. The second group consists of Action Extraction Metrics computed at the individual action level across all conversations. For each predicted action, we compute precision, recall, and accuracy where a true positive is a correctly predicted action, a false positive is an action predicted but absent from the ground truth, and a false negative is a ground truth action that was not predicted.

Benchmark Results

We evaluate five small open-weight language models out of the box without any task-specific fine-tuning: FunctionGemma (270M), Gemma4:E2B, Gemma4:E4B, Ministral3:3B, and Ministral3:8B, where E denotes effective parameters. Results are reported in Table I.

All models were evaluated using Ollama as the local inference backend, ensuring consistent and reproducible deployment conditions across models. Gemma4:E4B achieved the best performance across all metrics, with a goal achievement accuracy of 63.6% and an action-level F1 of 74.2%, demonstrating that a 4 billion effective parameter model can handle a meaningful portion of natural multi-turn HRI dialogue without any task-specific fine-tuning. Notably, FunctionGemma at only 270 million parameters produced competitive results relative to its size, suggesting that models explicitly trained for function calling retain useful structure even in complex conversational settings. Overall, the results are encouraging given that no fine-tuning was performed on any of the evaluated models as shown in Figure 4. The scores reflect the inherent difficulty of the task, where conversations are multi-turn, ambiguous, and require state tracking across actions, and suggest that targeted fine-tuning on this dataset has strong potential to further improve performance, which we leave for future work.

Comparison of five models on Goal Accuracy, Action Accuracy, and Action F1

Figure 4. Comparison of five models (FunctionGemma-270M, Gemma4:E2B, Gemma4:E4B, Ministral3:3B, and Ministral3:8B) on Goal Accuracy, Action Accuracy, and Action F1. Gemma4:E4B scores highest on all three metrics (63.6%, 59.0%, and 74.2%). FunctionGemma-270M scores lowest on Goal Accuracy and Action Accuracy, and Ministral3:3B scores lowest on Action F1.

ModelGoal Acc. (%)Action Level Prec. (%)Action Level Rec. (%)Action Level F1 (%)Action Level Acc. (%)
FunctionGemma-270M32.354.751.753.236.2
Gemma4:E2B55.974.158.865.648.8
Gemma4:E4B63.674.574.074.259.0
Ministral3:3B41.153.951.452.635.7
Ministral3:8B49.163.367.865.548.7

Table 1. Benchmark Results of Small Language Models on the HRI Intent Extraction Dataset

Conclusion

We presented a synthetically generated, multi-turn dialogue dataset for human-robot interaction (HRI), designed to reflect the complexity and naturalness of real human communication. Each conversation is annotated with structured action intent, object intent and task plans, providing a benchmark for evaluating language models on intent extraction and task planning in robotic settings. We evaluated five small, open-weight language models without any task-specific fine-tuning, demonstrating that compact, edge-deployable models can achieve promising performance on this challenging task. These results suggest that small language models are a viable option for on-device HRI pipelines and that fine-tuning these models on this dataset could significantly improve performance.

There are several areas for future work. Firstly, the dataset will be expanded to include more conversations, additional scenarios and a broader range of user personas. Secondly, we intend to fine-tune small language models on the training set in order to evaluate the potential benefits of task-specific adaptation. Thirdly, a wider range of models will be evaluated to provide a more comprehensive benchmark. Fourthly, we will incorporate similarity-based metrics for evaluating object description quality to provide a more detailed assessment of object intent extraction. Fifthly, we intend to incorporate robot state feedback into the dialogue pipeline to enable the robot to communicate its current state naturally, allowing for more grounded and coherent multi-turn interactions. Finally, we plan to incorporate vision and other modalities alongside conversational intent extraction to move towards a fully multimodal human-robot interaction (HRI) system.

References

  1. [1]Y. Lai, S. Yuan, Y. Nassar, M. Fan, T. Weber, and M. Rätsch, “Nvp-hri: Zero shot natural voice and posture-based human–robot interaction via large language model,” Expert Systems with Applications, vol. 268, p. 126360, 2025. [Online]. Available: https://www.sciencedirect.com/science/article/pii/S0957417424032275DOI
  2. [2]N. Carion, L. Gustafson, Y.-T. Hu, S. Debnath, R. Hu, D. Suris, C. Ryali, K. V. Alwala, H. Khedr, A. Huang, J. Lei, T. Ma, B. Guo, A. Kalla, M. Marks, J. Greer, M. Wang, P. Sun, R. Rädle, T. Aforaus, E. Mavroudi, K. Xu, T.-H. Wu, Y. Zhou, L. Momeni, R. Hazra, S. Ding, S. Vaze, F. Porcher, F. Li, S. Li, A. Kamath, H. Cheng, P. Dollár, N. Ravi, K. Saenko, P. Zhang, and C. Feichtenhofer, “Sam 3: Segment anything with concepts,” 2025. [Online]. Available: https://arxiv.org/abs/2511.16719
  3. [3]S. Liu, Z. Zeng, T. Ren, F. Li, H. Zhang, J. Yang, C. Li, J. Yang, H. Su, J. Zhu, et al., “Grounding dino: Marrying dino with grounded pre-training for open-set object detection,” arXiv preprint arXiv:2303.05499, 2023.arxiv.org/abs/2303.05499
  4. [4]M. Kekana, S. Du, N. Steyn, A. Benali, and H. Djerroud, “A Review of Human Intention Recognition Frameworks in Industrial Collaborative Robotics,” Robotics, vol. 14, no. 12, Nov. 2025. [Online]. Available: https://www.mdpi.com/2218-6581/14/12/174DOI
  5. [5]A. Vanzo, D. Croce, E. Bastianelli, R. Basili, and D. Nardi, “Grounded language interpretation of robotic commands through structured learning,” Artificial Intelligence, vol. 278, p. 103181, Jan. 2020. [Online]. Available: https://www.sciencedirect.com/science/article/pii/S0004370218302935DOI
  6. [6]A. Lubitz, L. Gutzeit, and F. Kirchner, “CoBaiR: A Python Library for Context-Based Intention Recognition in Human-Robot Interaction,” in 2023 32nd IEEE International Conference on Robot and Human Interactive Communication (RO-MAN). Busan, Korea, Republic of: IEEE, Aug. 2023, pp. 2003–2009. [Online]. Available: https://ieeexplore.ieee.org/document/10395881/arxiv.org/abs/2308.05627
  7. [7]Y. Lai, G. Paul, Y. Cui, and T. Matsubara, “User intent estimation during robot learning using physical human robot interaction primitives,” Autonomous Robots, vol. 46, no. 2, pp. 421–436, Feb. 2022. [Online]. Available: https://doi.org/10.1007/s10514-021-10030-9
  8. [8]I. Papaioannou, C. Dondrup, J. Novikova, and O. Lemon, “Hybrid chat and task dialogue for more engaging HRI using reinforcement learning,” in 2017 26th IEEE International Symposium on Robot and Human Interactive Communication (RO-MAN), Aug. 2017, pp. 593–598, ISSN: 1944-9437. [Online]. Available: https://ieeexplore.ieee.org/document/8172363/DOI
  9. [9]M. Shridhar, J. Thomson, D. Gordon, Y. Bisk, W. Han, R. Mottaghi, L. Zettlemoyer, and D. Fox, “ALFRED: A Benchmark for Interpreting Grounded Instructions for Everyday Tasks,” in 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), June 2020, pp. 10737–10746, iSSN: 2575-7075. [Online]. Available: https://ieeexplore.ieee.org/document/9156851
  10. [10]A. Padmakumar, J. Thomason, A. Shrivastava, P. Lange, A. Narayan-Chen, S. Gella, R. Piramuthu, G. Tur, and D. Hakkani-Tur, “TEACh: Task-driven Embodied Agents that Chat,” Dec. 2021, arXiv:2110.00534 [cs]. [Online]. Available: http://arxiv.org/abs/2110.00534
  11. [11]F. Gao, L. Xia, J. Zhang, S. Liu, L. Wang, and R. X. Gao, “Integrating Large Language Model for Natural Language-Based Instruction toward Robust Human-Robot Collaboration,” Procedia CIRP, vol. 130, pp. 313–318, Jan. 2024. [Online]. Available: https://www.sciencedirect.com/science/article/pii/S2212827124012502DOI
  12. [12]Y. Jin, D. Li, Y. A. J. Shi, P. Hao, F. Sun, J. Zhang, and B. Fang, “RobotGPT: Robot Manipulation Learning From ChatGPT,” IEEE Robotics and Automation Letters, vol. 9, no. 3, pp. 2543–2550, Mar. 2024. [Online]. Available: https://ieeexplore.ieee.org/abstract/document/10412086arxiv.org/abs/2312.01421
  13. [13]X. Li, C. Mata, J. Park, K. Kahatapitiya, Y. S. Jang, J. Shang, K. Ranasinghe, R. Burgert, M. Cai, Y. J. Lee, and M. S. Ryoo, “LLaRA: Supercharging Robot Learning Data for Vision-Language Policy,” Jan. 2025, arXiv:2406.20095 [cs]. [Online]. Available: http://arxiv.org/abs/2406.20095
  14. [14]L. X. Shi, B. Ichter, M. Equi, L. Ke, K. Pertsch, O. Vuong, J. Tanner, A. Walling, H. Wang, N. Fusai, A. Li-Bell, D. Driess, L. Groom, S. Levine, and C. Finn, “Hi Robot: Open-Ended Instruction Following with Hierarchical Vision-Language-Action Models,” July 2025, arXiv:2502.19417 [cs]. [Online]. Available: http://arxiv.org/abs/2502.19417
  15. [15]Y. Wang, Q. Guo, L. Zheng, B. Wang, P. Zheng, and Z. Qi, “LLM based autonomous agent of human-robot collaboration for aerospace wire harnessing assembly,” Robotics and Computer-Integrated Manufacturing, vol. 97, p. 103120, Feb. 2026. [Online]. Available: https://linkinghub.elsevier.com/retrieve/pii/S0736584525001747DOI
  16. [16]H. Xu, C. Li, X. Yuan, T. Zhi, and H. Liu, “A simulated dataset for proactive robot task inference from streaming natural language dialogues,” Scientific Data, vol. 12, no. 1, p. 1405, Aug. 2025. [Online]. Available: https://www.nature.com/articles/s41597-025-05727-wDOI
  17. [17]Qwen Team, “Qwen3.6-35B-A3B: Agentic coding power, now open to all,” April 2026. [Online]. Available: https://qwen.ai/blog?id=qwen3.6-35b-a3b

Paper details

vlnc:W_dtjanpy5
Version
v1 · latest
Published
Fresh — added to the Hub within the last 30 days
Added
License
CC BY 4.0
Usage

views · 8

downloads · 0

A view counts once per browser per day, a download each time the PDF is requested.

  • Computer Science

Not recorded for this paper

View statement
Looking for related papers…
Contents