betaFeedback

WorksComputer Science

Computer-Use Agents Are Embodied Generalists

Abstract

In many areas of AI, general-purpose models have increasingly replaced handcrafted methods and specialized architectures. We argue that embodied AI is poised for a similar shift. Our key claim is that embodied tasks can be formulated as interaction with digital software. Specifically, we build software that renders an agent's view of a physical environment on a screen and provides controls for acting in it. A computer-use agent (CUA) can then perform embodied tasks through the same screenshots and keyboard and mouse actions that it uses to operate ordinary software. Using a single CUA design and changing only the software, our entries win first-, second-, third-, and sixth-place prizes across four ECCV workshop competitions, without any competition-specific agent tuning. These results suggest that the skills CUAs learn on software, such as visual grounding, precise clicking, and exploration, already carry over to embodied tasks, giving CUAs a far broader scope in embodied AI than previously realized. We therefore advocate that computer-use agents can be embodied generalists. Further, we advocate that, by following the training strategy that made computer-use agents general, embodied AI can converge on a single general model as well, ushering in a new era of general embodied AI. Website: https://pranjal2041.github.io/cua-embodied

Introduction

In many areas of AI, general-purpose models have increasingly replaced handcrafted methods and specialized architectures [48, 41] (Sutton 2019). We argue that embodied AI, which studies agents that perceive and act in physical environments, is poised for a similar shift [44, 24]. In particular, we explore connecting embodied AI with computer-use agents: general-purpose digital agents that operate diverse software applications through screenshots, mouse clicks, and keyboard inputs. Although operating a computer and acting in a physical environment may seem quite different, we observe that both involve similar challenges, including visual grounding, exploration, and adapting actions based on feedback from the environment [31] (Yang et al. 2025a; Zhang et al. 2025a).

Our key insight is that embodied tasks can be formulated as interaction with digital software. Specifically, we construct software that renders observations on a screen and provides controls for acting in the physical environment Figure 1. A computer-use agent can then observe the environment, take actions, and inspect their consequences using the same interface that it uses to operate ordinary desktop software. Task-specific functionality resides in the software rather than the agent design, allowing us to use existing computer-use agents across different embodied tasks without developing specialized models.

Figure 1. Computer-use agents can operate any software through keyboard and mouse actions while observing the screen. We argue that physical environments are no different. Once exposed through software, they become just another interface for the same agent.

To investigate our approach, we develop software applications for four embodied AI tasks: vision-language navigation, dialogue-driven navigation, language-guided 3D segmentation, and referring video segmentation. We use a single, minimal computer-use agent harness across all four applications, without competition-specific agent tuning, and enter 4 ECCV 2026 workshop competitions. Our

Preprint. entries place first in OpenSUN3D, second in MeViSv2-Text, third in DialNav, and sixth in the VLNVerse competitions. These results demonstrate that computer-use agents can achieve strong performance across diverse embodied tasks by interacting with task-specific software through a shared agent design.

Beyond these results, we argue that embodied AI can follow the same path that computer use has already taken. Only two years ago, computer-use agents faced the challenges that embodied AI faces today. They struggled to click precisely and relied on overlays or separate grounding models to locate targets [25] (Yang et al. 2023; Zheng et al. 2024), much as most VLAs today add a dedicated action head for continuous control (Physical Intelligence et al., 2025; NVIDIA et al., 2025). They also lagged far behind agents with task-specific APIs (Xie et al., 2024; Aggarwal and Welleck, 2026c; Song et al., 2025). Today, the same model that writes code and solves mathematics problems can also operate computers by writing click coordinates as ordinary text [9] (Quiros 2026), and the strongest agents perform comparably whether they act through the screen or through APIs (NeoCognition, 2026). This progress came from training for precise visual grounding, building environments for reinforcement learning, and making interaction fast enough for real-time use [25, 8, 7] (Wang et al. 2026). Each of these has a direct counterpart in embodied AI. Simulators can generate grounding data at scale (Yuan et al., 2024), embodied environments can be used for reinforcement learning [37], and faster inference can enable real-time control [14]. Since our approach exposes embodied environments through software, advances in computer-use agents can also transfer to embodied AI directly, and embodied tasks can become part of the training of future general-purpose agents. We therefore believe that embodied AI will converge on a single general model, just as computer use did [10] (OpenAI 2026c).

Methodology

Our goal is to provide sparks of evidence for our hypothesis that the capabilities computer-use agents learn on digital software also generalize to embodied environments. We entered four different ECCV workshop competitions, and evaluated a single computer-use agent on their official test sets. We first describe the computer-use agent shared across all four, and then the software we built for each competition.

A shared computer-use agent

We keep the computer-use agent’s design fixed and construct software for each competition. The agent receives the task description, the software’s instructions, and any reference images or videos. Let qq denote these task inputs, SkS_k the software for competition kk, and sts_t its state at step tt. The software presents a visual observation oto_t, consisting of a screenshot and any video returned by an operation. Given the interaction history ht=(o0,a0,…,ot)h_t = (o_0, a_0, \ldots, o_t), the agent π\pi chooses a keyboard or mouse action

at∼π(⋅∣q,ht),(st+1,ot+1)=Sk(st,at).(1)a_t \sim\pi(\cdot\mid q,h_t), \qquad(s_{t+1},o_{t+1}) = S_k(s_t,a_t). \tag*{(1)}

The software executes the action and returns the updated observation. The agent repeats this loop until it presses submit through the interface, at which point the software exports the trajectory or prediction.

We use Gemini 3.5 Flash and Gemini 3.6 Flash with the same agent implementation, system prompt, and keyboard and mouse action space across the four software applications. Each application provides controls for its internal operations, including movement, dialogue, segmentation, and mask propagation. The agent chooses when to use these operations, interprets their outputs, and decides what to do next. It can issue individual inputs or batches of up to 16 inputs; each operation finishes before the next begins, and a screenshot follows the batch. We retain the interaction text and remove the oldest ten screenshots when the context reaches twenty. Complete reference videos and generated overlay videos remain in context. Figure 2 shows the four interfaces.

Four software application interfaces: VLNVerse, DialNav, OpenSUN3D, and MeViSv2

Figure 2. The shared computer-use agent operates four software applications through keyboard and mouse actions. Each application provides task-specific controls and displays the outcomes.

VLNVerse: navigation through a 3D scene

VLNVerse [39] requires navigating to a destination described in language. We construct a 3D scene viewer with a first-person image and controls for translation and rotation. The agent moves forward, backward, sideways, or vertically using keyboard shortcuts or on-screen buttons. It can turn, look up or down, and adjust the movement distance and rotation angle. Each operation updates the camera pose and renders a new image. The agent uses these observations to identify landmarks, explore rooms, and approach the target. The same viewer supports the coarse and fine tracks. Selecting the stop control saves the trajectory and final pose.

DialNav: navigation with a conversational guide

DialNav [27] requires reaching a destination through dialogue with a Guide. Our software displays a 360° panorama, numbered exits to adjacent viewpoints, a target clue, and a message panel. The agent types descriptions and questions, reads the Guide’s replies, and clicks exits to move. A Guide model within the software receives the destination, a scene atlas, and route previews in a separate context. It infers the agent’s location from its descriptions and returns natural-language directions. The agent uses these directions and the panorama to choose each movement. The stop control saves the trajectory and dialogue.

OpenSUN3D: language-guided 3D segmentation

The OpenSUN3D challenge [49] uses SceneFun3D [20] tasks that describe an action and require a 3D mask of the corresponding interaction region, such as a handle for opening a drawer. Our software renders the colored Faro point cloud and provides free-camera movement, rotation, and point selection. Reference videos supply scene context and shortcuts to recorded camera poses. The agent explores the 3D scene, approaches the target, and clicks the relevant component. The renderer maps this click to a visible Faro point. Within the software, PinPoint3D (Zhang et al., 2025b) produces point-prompted masks, while a VOLT model fine-tuned on SceneFun3D supplies affordance-based proposals. The interface displays each candidate from four viewpoints. The agent can refine a selection with positive or negative points, move to another viewpoint, or make a new selection. It then chooses which masks to save, retaining their original Faro point indices.

MeViSv2: referring video segmentation

The MeViSv2 [21] text track requires segmenting the objects described by a motion expression throughout a video. Our software provides a video player, a frame selector, and controls for adding, refining, and removing objects. The agent receives the complete video and expression, chooses a frame, and clicks each target. SAM3 [17] segments these objects and propagates their masks through the video. The agent receives the complete tracked-mask overlay and can inspect individual frames. It can add positive or negative points, remove an object, or select another object before running propagation again. The save control exports the masks of all remaining objects. A no-target control exports an empty mask sequence.

Results

Competition results

The same computer-use agent, with only its software changed, won prizes in all four competitions, spanning navigation, dialogue-driven navigation, 3D segmentation, and video segmentation. It placed first in OpenSUN3D, second in MeViSv2-Text, third in DialNav, and sixth in VLNVerse. We report the official test scores of our competition entries, together with the best other entry in each competition, in Table 1. In OpenSUN3D, the agent finished ahead of every other entry by more than 4 points in AP50 and 10 points in AP25. There, it explored a 3D scan and clicked the exact part to interact with, such as a drawer handle, much as it clicks a small button on a screen. In VLNVerse, the agent placed sixth on success but followed the reference routes more closely than all but two of the 74 entries, with the third-highest NDTW on both tracks. In MeViSv2-Text, it placed second ahead of an entry with a higher J&FJ \& F, because it more often recognized when no object matched the expression (88.89 versus 83.33 no-target accuracy). Each of these results rests on abilities the agent already possesses on digital software, namely locating targets, pointing at them precisely, and exploring its environment.

CompetitionPlaceMetricOursBest other
VLNVerse6thAvg SR@387.3090.70
NDTW (coarse / fine)95.0 / 96.896.3 / 97.6
DialNav3rdEndpoint success rate65.26–
SPL52.87–
OpenSUN3D1stAP5025.8821.45
AP2536.5226.50
MeViSv2-Text2ndFinal86.7290.81
J&F74.9679.22

Table 1. Official test scores of our competition entries and the best other entry for each metric, taken from the official challenge results (EMR, 2026; EAD, 2026; OpenSUN3D, 2026; Liu et al., 2026). All scores are expressed on a 0–100 scale.

VLNVerse’s SR@3 measures the fraction of episodes ending within three metres of the target. Avg SR@3 averages it over the coarse and fine tracks, and NDTW measures how closely a path follows the reference route. DialNav reports the endpoint success rate and success weighted by path length (SPL). OpenSUN3D evaluates 3D instance masks using average precision at 50% and 25% intersection over union. MeViSv2’s Final score averages J&FJ \& F with the accuracy of predicting target presence and absence.

GUI operations or text APIs?

A natural question is whether we need to construct GUIs or could instead provide direct text APIs for navigation and other operations. We view computer use as operating a digital interface. GUIs and APIs are two choices for that interface. We expect sufficiently capable computer-use agents to achieve the same task performance with either interface, although efficiency may differ.

We evaluate this by comparing GUI and API interaction on a subset of tasks from each of the four competitions. Each pair uses the same model, task inputs, and software operations. We compare successive generations of Gemini Flash to examine how improvements in computer use affect the performance gap between interfaces.

Table 2 shows the results on the validation sets. The two interfaces performed within four points of each other, and the GUI scored higher in two of the four competitions.

CompetitionMetricAPIGUIGUI − API
VLNVerseAvg SR@383.985.2+1.3
DialNavEndpoint success rate67.463.4−4.0
OpenSUN3DAP5026.728.1+1.4
MeViSv2-TextFinal95.894.6−1.2

Table 2. GUI and API interaction on the validation set of each competition, with the same model, task inputs, and software operations. All scores are expressed on a 0–100 scale.

This expectation is consistent with prior work. The performance gap between GUI-only and API-assisted interaction has shown to be narrowing across claude generations (Aggarwal and Welleck, 2026c), while more recently, the strongest evaluated agents achieve comparable or higher task success through GUIs than through APIs (NeoCognition, 2026).

Position: Computer-use agents are embodied generalists

Claim 1: Representing physical interfaces in software lets us apply the methods that improved computer-use agents to embodied AI

The challenges facing embodied AI/robotics today are similar to those faced by computer-use agents two years ago. Both require an agent to ground fine-grained actions in visual observations and adapt as its environment changes [18, 35] (Yuan et al. 2024; Yang et al. 2025a) (Figure 3). For computer use, this means locating a target on a screen and clicking at the correct coordinates. For robotics, it means locating targets in three dimensions and coordinating motor commands to achieve the intended positions and orientations. Representing physical interfaces in software or simulations lets us apply the same methods for training agents to perform these interactions. Our results in § 3 offer initial evidence: using the same agent and changing only the software, we performed competitively in four embodied competitions.

Figure 3. Computer use and embodied AI require much of the same from an agent. In both, it must see its surroundings, locate a target, act on it precisely, issue discrete commands, and explore. The main difference is time, as the screen often changes slowly while the physical world keeps moving.

Fine-grained motor control and real-time physical interaction may seem out of reach for computer-use agents. Yet only two years ago, precise clicking seemed equally out of reach: computer-use agents struggled to operate digital interfaces (Xie et al., 2024) and lagged behind agents equipped with task-specific APIs (Aggarwal and Welleck, 2026c; Song et al., 2025). As discussed in the previous section, the strongest agents now achieve comparable task performance through either interface (NeoCognition, 2026). We therefore argue for following the same development strategies in embodied AI, from learning precise, visually grounded actions to improving reliability and speed.

Claim 2: Embodied AI will converge on a single general model, as computer use did

Much of embodied AI today is built on vision-language-action models (VLAs), a special case of vision-language models further fine-tuned for robotic control [15, 33, 13, 32]. Since robot actions are continuous, and discrete tokens seem too coarse to express them precisely, most VLAs also add a dedicated action head, typically trained with diffusion or flow matching [13, 36, 32] (NVIDIA et al. 2025). But computer-use agents faced exactly this problem only two years ago. A screen of a thousand by a thousand pixels is effectively a continuous space, and early agents could not act in it precisely on their own, relying instead on Set-of-Mark overlays (Yang et al., 2023; Xie et al., 2024) or separate grounding models [25]. Yet with enough training, neither turned out to be necessary: today, a single model operates computers by simply writing click coordinates as ordinary text tokens, without any special head or tokenizer [18, 12] (Qin et al. 2025).

We therefore believe that embodied AI will follow the same path, and that specialized action heads will go the way of specialized grounding models [26, 28]. In fact, RT-2 and OpenVLA already show that robot actions can be expressed as tokens in the language model’s own vocabulary, without any action head [15, 33]. To support this claim, we perform a preliminary investigation in Gym-Anything for Robotics [7]. We train π0.5\pi_{0.5} (Physical Intelligence et al., 2025) and a standard vision-language model (Qwen3.5) with 9B parameters on the same 1,000 demonstrations for two tasks in a gift box simulation with two robot arms. We add no action head and do not even change the tokenizer. The only change is the format in which actions are written as text, similar to computer-use agents. Yet this plain vision-language model matches π0.5\pi_{0.5}, succeeding in 56% of attempts, against 59% for π0.5\pi_{0.5} across 1000 unseen instances. The model is also fast. As with π0.5\pi_{0.5}, the model receives a new camera image every second and either writes an action, which can span a longer duration, or chooses not to act. The 9B model takes about 400 milliseconds on average to generate an action on a single L40S GPU. Because each action can span a longer duration, decoding takes up only about 20% of the total time and prefill about 10%, with the remaining 70% idle. However, this setting has only two robot arms. More complex robots, such as humanoids, have many more motors, and writing every state and action for them as plain text would make each action slower to generate. Such robots still need a more efficient format for states and actions, and possibly a better tokenizer [51, 32]. And even if we use additional action heads, or whatever that format turns out to be, the most important thing is, embodied AI will end up exactly where computer use did, with a single model that writes code, solves mathematics problems, operates computers, and controls robots [24] (Yang et al. 2025a; Reed et al. 2022).

The path to embodied generalists follows the same playbook as computer-use agents

What enabled the progress of computer-use agents over the last two years? We view the development of computer-use agents in three phases, each with a direct counterpart in embodied AI.

The first phase was visual grounding. Digital interfaces provide a way to generate supervision automatically: the underlying software can expose the identities and coordinates of interface elements, which can then be paired with screenshots. This approach has been used to generate grounding data at scale [25]. Simulators offer the same opportunity for embodied AI: they can expose the identities and 3D positions of objects, which can be paired with rendered views to generate grounding data at scale (Yuan et al., 2024). Of course, this requires simulations with high enough visual fidelity, an area where coding models are clearly and continuously improving.

The second phase was building environments for reinforcement learning, following concurrent approaches used in other domains such as coding [50] (Yang et al. 2025b). Automated pipelines made it possible to generate tasks, initialize environments, and verify outcomes at scale [8] (Wang et al. 2026). These environments then let agents learn through reinforcement learning at scale. For embodied AI, this means creating such environments in the form of simulations with realistic physics, with tasks and verifiers available through software. We further argue that embodied environments have an even stronger asymmetry between task creation and verification [7], which can be used to provide more accurate rewards or to increase task difficulty over time. We also note that reinforcement learning benefits from a strong initial policy, and training such a policy requires trajectories for pretraining or midtraining. We are therefore not advocating against teleoperation, whether in the real world or in simulation, or against egocentric videos, although this data may matter mainly in pretraining and midtraining, as in the training recipes of existing LLMs. How much of this data is needed also remains unclear, but based on preliminary results presented in this paper, we believe that perhaps much of it will not be needed, and that research on environments for reinforcement learning will remain more important.

The third phase is speed and real-time interaction, which is especially important for robotics. An agent must choose and execute actions quickly enough to keep up with changes in its environment. This requires optimizing completion time alongside task success [7]. Even without such optimization, the 9B model in Claim 2 took about 400 milliseconds per action and was idle for 70% of the time. We can pursue this through rewards for completing tasks successfully with less reasoning or in less time [2, 11], faster inference hardware [38, 1, 29], and harnesses that interleave quick actions with slower planning (Shi et al., 2025; NVIDIA et al., 2025). Such agentic harnesses organize how an agent plans, acts, and checks its progress [46, 34], and embodied tasks can reuse them once they are exposed through software.

The sequence is therefore the same: learn grounding, build environments for reinforcement learning, and optimize for real-time interaction.

So what should be next steps for Embodied AI?

We therefore advocate for 3 research directions we believe would become fundamental for embodied AI research: a.) improving visual grounding through data generated automatically from simulators (Pfaff et al., 2026; Yuan et al., 2024), b.) building robotic environments as simulations for robotics, exposed through software that computer-use agents can operate and be RL trained on [7, 30] (Wang et al. 2024), and c.) optimizing agents for real-time interaction [8] (Tang et al. 2026b) through architecture research [19] (Tang et al. 2026a; Aggarwal and Welleck 2026d) and by rewarding speed alongside task success [11, 2]. These directions call on embodied AI researchers to expose their simulators and robots through software, and on developers of computer-use agents to include such software in training.

Conclusion

In this paper, we argued that computer-use agents can already act as embodied generalists, because embodied tasks can be formulated as interaction with digital software. Using a single computer-use agent and changing only the software, we won prizes in four ECCV workshop competitions, from navigation to video segmentation. Operating that software through a GUI cost little compared to a text API, and a vision-language model with no action head matched π0.5\pi_{0.5} when trained on the same robot demonstrations. We then traced how computer-use agents became general and superhuman over the last two years, through visual grounding, environments for reinforcement learning, and speed, and argued that embodied AI should follow the same playbook. We believe embodied AI is headed the same way, and that its generalist version will be the same model that already writes code, solves mathematics problems, and operates digital computers.

Acknowledgements

We thank Daytona for their generous support in sandbox compute credits, and Google for their generous support through a Gemini API research grant. We thank the organizers of the VLNVerse, DialNav, OpenSUN3D, and MeViSv2 competitions and their ECCV 2026 workshops. All opinions expressed in this paper are the authors’ own and do not necessarily reflect the views of our sponsors or the competition organizers.

References

  1. [1]Dennis Abts, Jonathan Ross, Jonathan Sparling, Mark Wong-VanHaren, Max Baker, Tom Hawkins, Andrew Bell, John Thompson, Temesgen Kahsai, Garrin Kimmell, Jennifer Hwang, Rebekah Leslie-Hurd, Michael Bye, E. R. Creswick, Matthew Boyd, Mahitha Venigalla, Evan Laforge, Jon Purdy, Purushotham Kamath, Dinesh Maheshwari, Michael Beidler, Geert Rosseel, Omar Ahmad, Gleb Gagarin, Richard Czekalski, Ashay Rane, Sahil Parmar, Jeff Werner, Jim Sproch, Adrian Macias, and Brian Kurtz. Think fast: A tensor streaming processor (TSP) for accelerating deep learning workloads. In ACM/IEEE International Symposium on Computer Architecture (ISCA), pp. 145–158, 2020.DOI
  2. [2]Pranjal Aggarwal and Sean Welleck. L1: Controlling how long a reasoning model thinks with reinforcement learning. In Conference on Language Modeling (COLM), 2025.arxiv.org/abs/2503.04697
  3. [7]Pranjal Aggarwal, Lawrence Keunho Jang, Sean Welleck, Daniel Fried, Ruslan Salakhutdinov, and Jing Yu Koh. cua-speedrun: Standardized benchmarking of the speed of computer-use agents, 2026a. URL https://arxiv.org/abs/2609.40284.
  4. [8]Pranjal Aggarwal, Graham Neubig, and Sean Welleck. Gym-Anything: Turn any software into an agent environment, 2026b. URL https://arxiv.org/abs/2604.06126.
  5. [9]Anthropic. Introducing computer use, a new Claude 3.5 Sonnet, and Claude 3.5 Haiku, 2024. URL https://www.anthropic.com/news/3-5-models-and-computer-use.
  6. [10]Anthropic. Introducing Claude Opus 5.5, 2026. URL https://www.anthropic.com/claude-opus-5-5.
  7. [11]Daman Arora and Andrea Zanette. Training language models to reason efficiently. In Advances in Neural Information Processing Systems (NeurIPS), 2025.arxiv.org/abs/2502.04463
  8. [12]Shuai Bai, Keqin Chen, Xuejing Liu, Jialin Wang, Wenbin Ge, Sibo Song, Kai Dang, Peng Wang, Shijie Wang, Jun Tang, Humen Zhong, Yuanzhi Zhu, Mingkun Yang, Zhaohai Li, Jianqiang Wan, Pengfei Wang, Wei Ding, Zheren Fu, Yiheng Xu, Jiabo Ye, Xi Zhang, Tianbao Xie, Zesen Cheng, Hang Zhang, Zhibo Yang, Haiyang Xu, and Junyang Lin. Qwen2.5-VL technical report, 2025. URL https://arxiv.org/abs/2502.13923.
  9. [13]Kevin Black, Noah Brown, Danny Driess, Adnan Esmail, Michael Equi, Chelsea Finn, Niccolo Fusai, Lachy Groom, Karol Hausman, Brian Ichter, Szymon Jakubczak, Tim Jones, Liyiming Ke, Sergey Levine, Adrian Li-Bell, Mohith Mothukuri, Suraj Nair, Karl Pertsch, Lucy Xiaoyang Shi, James Tanner, Quan Vuong, Anna Walling, Haohuan Wang, and Ury Zhilinsky. π₀: A vision-language-action flow model for general robot control. In Robotics: Science and Systems (RSS), 2025.DOI
  10. [14]Kevin Black, Manuel Y. Galliker, and Sergey Levine. Real-time execution of action chunking flow policies. In Advances in Neural Information Processing Systems (NeurIPS), 2025.arxiv.org/abs/2506.07339
  11. [15]Anthony Brohan, Noah Brown, Justice Carbajal, Yevgen Chebotar, Xi Chen, Krzysztof Choromanski, Tianli Ding, Danny Driess, Avinava Dubey, Chelsea Finn, Pete Florence, Chuyuan Fu, Montse Gonzalez Arenas, Keerthana Gopalakrishnan, Kehang Han, Karol Hausman, Alexander Herzog, Jasmine Hsu, Brian Ichter, Alex Irpan, Nikhil Joshi, Ryan Julian, Dmitry Kalashnikov, Yuheng Kuang, Isabel Leal, Lisa Lee, Tsang-Wei Edward Lee, Sergey Levine, Yao Lu, Henry Michalewski, Igor Mordatch, Karl Pertsch, Kanishka Rao, Krista Reymann, Michael Ryoo, Grecia Salazar, Pannag Sanketi, Pierre Sermanet, Jaspiar Singh, Anikait Singh, Radu Soricut, Huong Tran, Vincent Vanhoucke, Quan Vuong, Ayzaan Wahid, Stefan Welker, Paul Wohlhart, Jialin Wu, Fei Xia, Ted Xiao, Peng Xu, Sichun Xu, Tianhe Yu, and Brianna Zitkovich. RT-2: Vision-Language-Action models transfer web knowledge to robotic control. In Conference on Robot Learning (CoRL), 2023.
  12. [17]Nicolas Carion, Laura Gustafson, Yuan-Ting Hu, Shouhbik Debnath, Ronghang Hu, Didac Suris, Chaitanya Ryali, Kalyan Vasudev Alwala, Haitham Khedr, Andrew Huang, Jie Lei, Tengyu Ma, Baishan Guo, Arpit Kalla, Markus Marks, Joseph Greer, Meng Wang, Peize Sun, Roman Rädle, Triantafyllos Afouras, Efrosyni Mavroudi, Katherine Xu, Tsung-Han Wu, Yu Zhou, Liliane Momeni, Rishi Hazra, Shuangrui Ding, Sagar Vaze, Francois Porcher, Feng Li, Siyuan Li, Aishwarya Kamath, Ho Kei Cheng, Piotr Dollár, Nikhila Ravi, Kate Saenko, Pengchuan Zhang, and Christoph Feichtenhofer. SAM 3: Segment anything with concepts, 2025. URL https://arxiv.org/abs/2511.16719.
  13. [18]Kanzhi Cheng, Qiushi Sun, Yougang Chu, Fangzhi Xu, Yantao Li, Jianbing Zhang, and Zhiyong Wu. SeeClick: Harnessing GUI grounding for advanced visual GUI agents. In Annual Meeting of the Association for Computational Linguistics (ACL), pp. 9313–9332, 2024.
  14. [19]Rohan Choudhury, Guanglei Zhu, Sihan Liu, Koichiro Niinuma, Kris M. Kitani, and László Jeni. Don’t look twice: Faster video transformers with run-length tokenization. In Advances in Neural Information Processing Systems (NeurIPS), 2024.arxiv.org/abs/2411.05222
  15. [20]Alexandros Delitzas, Ayca Takmaz, Federico Tombari, Robert Sumner, Marc Pollefeys, and Francis Engelmann. SceneFun3D: Fine-grained functionality and affordance understanding in 3D scenes. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 14531–14542, 2024.DOI
  16. [21]Henghui Ding, Chang Liu, Shuting He, Kaining Ying, Xudong Jiang, Chen Change Loy, and Yu-Gang Jiang. MeViS: A multi-modal dataset for referring motion expression video segmentation. IEEE Transactions on Pattern Analysis and Machine Intelligence, 47(12):11400–11416, 2025.DOI
  17. [24]Gemini Robotics Team, Saminda Abeyruwan, Joshua Ainslie, Jean-Baptiste Alayrac, Montserrat Gonzalez Arenas, Travis Armstrong, Ashwin Balakrishna, Robert Baruch, Maria Bauza, Michiel Blokzijl, Steven Bohez, Konstantinos Bousmalis, Anthony Brohan, Thomas Buschmann, Arunku-mar Byravan, Serkan Cabi, Ken Caluwaerts, Federico Casarini, Oscar Chang, Jose Enrique Chen, Xi Chen, Hao-Tien Lewis Chiang, Krzysztof Choromanski, David D’Ambrosio, Sudeep Dasari, Todor Davchev, Coline Devin, Norman Di Palo, Tianli Ding, Adil Dostmohamed, Danny Driess, Yilun Du, Debidatta Dwibedi, Michael Elad, Claudio Fantacci, Cody Fong, Erik Frey, Chuyuan Fu, Marissa Giustina, Keerthana Gopalakrishnan, Laura Graesser, Leonard Hasenclever, Nicolas Heess, Brandon Hernaez, Alexander Herzog, R. Alex Hofer, Jan Humplik, Atil Iscen, Mithun George Jacob, Deepali Jain, Ryan Julian, Dmitry Kalashnikov, M. Emre Karagozler, Stefani Karp, Chase Kew, Jerad Kirkland, Sean Kirmani, Yuheng Kuang, Thomas Lampe, Antoine Laurens, Isabel Leal, Alex X. Lee, Tsang-Wei Edward Lee, Jacky Liang, Yixin Lin, Sharath Maddineni, Anirudha Majumdar, Assaf Hurwitz Michaelty, Robert Moreno, Michael Neunert, Francesco Nori, Carolina Parada, Emilio Parisotto, Peter Pastor, Acorn Pooley, Kanishka Rao, Krista Reymann, Dorsa Sadigh, Stefano Saliceti, Pannag Sanketi, Pierre Sermanet, Dhruv Shah, Mohit Sharma, Kathryn Shea, Charles Shu, Vikas Sindhwani, Sumeet Singh, Radu Soricut, Jost Tobias Springenberg, Rachel Sterneck, Razvan Surdulescu, Jie Tan, Jonathan Tompson, Vincent Vanhoucke, Jake Varley, Grace Vesom, Giulia Vezzani, Oriol Vinyals, Ayzaan Wahid, Stefan Welker, Paul Wohlhart, Fei Xia, Ted Xiao, Annie Xie, Jinyu Xie, Peng Xu, Sichun Xu, Ying Xu, Zhuo Xu, Yuxiang Yang, Rui Yao, Sergey Yaroshenko, Wenhao Yu, Wentao Yuan, Jingwei Zhang, Tingnan Zhang, Allan Zhou, and Yuxiang Zhou. Gemini Robotics: Bringing AI into the physical world, 2025. URL https://arxiv.org/abs/2503.20020.
  18. [25]Boyu Gou, Ruohan Wang, Boyuan Zheng, Yanan Xie, Cheng Chang, Yiheng Shu, Huan Sun, and Yu Su. Navigating the digital world as humans do: Universal visual grounding for GUI agents. In International Conference on Learning Representations (ICLR), 2025.arxiv.org/abs/2410.05243
  19. [26]Ankit Goyal, Hugo Hadfield, Xuning Yang, Valts Blukis, and Fabio Ramos. VLA-0: Building state-of-the-art VLAs with zero modification, 2025. URL https://arxiv.org/abs/2510.13054.
  20. [27]Leekyeung Han, Hyunji Min, Gyeom Hwangbo, Jonghyun Choi, and Paul Hongsuck Seo. DialNav: Multi-turn dialog navigation with a remote guide. In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), pp. 8514–8523, 2025.arxiv.org/abs/2509.12894
  21. [28]Asher J. Hancock, Xindi Wu, Lihan Zha, Olga Russakovsky, and Anirudha Majumdar. Actions as language: Fine-tuning VLMs into VLAs without catastrophic forgetting. In International Conference on Learning Representations (ICLR), 2026.arxiv.org/abs/2509.22195
  22. [29]Congjie He, Yeqi Huang, Pei Mu, Ziming Miao, Jilong Xue, Lingxiao Ma, Fan Yang, and Luo Mai. WaferLLM: Large language model inference at wafer scale. In USENIX Symposium on Operating Systems Design and Implementation (OSDI), pp. 257–273, 2025.arxiv.org/abs/2502.04563
  23. [30]Pu Hua, Minghuan Liu, Annabella Macaluso, Yunfeng Lin, Weinan Zhang, Huazhe Xu, and Lirui Wang. GenSim2: Scaling robot data generation with multi-modal and reasoning LLMs. In Conference on Robot Learning (CoRL), 2024.arxiv.org/abs/2410.03645
  24. [31]Wenlong Huang, Fei Xia, Ted Xiao, Harris Chan, Jacky Liang, Pete Florence, Andy Zeng, Jonathan Tompson, Igor Mordatch, Yevgen Chebotar, Pierre Sermanet, Tomas Jackson, Noah Brown, Linda Luu, Sergey Levine, Karol Hausman, and Brian Ichter. Inner Monologue: Embodied reasoning through planning with language models. In Conference on Robot Learning (CoRL), 2022.arxiv.org/abs/2207.05608
  25. [32]Kento Kawaharazuka, Jihoon Oh, Jun Yamada, Ingmar Posner, and Yuke Zhu. Vision-language-action models for robotics: A review towards real-world applications. IEEE Access, 13:162467–162504, 2025.DOI
  26. [33]Moo Jin Kim, Karl Pertsch, Siddharth Karamcheti, Ted Xiao, Ashwin Balakrishna, Suraj Nair, Rafael Rafailov, Ethan Foster, Grace Lam, Pannag Sanketi, Quan Vuong, Thomas Kollar, Benjamin Burchfiel, Russ Tedrake, Dorsa Sadigh, Sergey Levine, Percy Liang, and Chelsea Finn. OpenVLA: An open-source vision-language-action model. In Conference on Robot Learning (CoRL), 2024.arxiv.org/abs/2406.09246
  27. [34]Jing Yu Koh, Ruslan Salakhutdinov, and Daniel Fried. Multi-agent computer use, 2026. URL https://arxiv.org/abs/2606.01533.
  28. [35]Jason Lee, Jiafei Duan, Haoquan Fang, Yuquan Deng, Shuo Liu, Boyang Li, Bohan Fang, Jieyu Zhang, Yi Ru Wang, Sangho Lee, Winson Han, Wilbert Pumacay, Angelica Wu, Rose Hendrix, Karen Farley, Eli VanderBilt, Ali Farhadi, Dieter Fox, and Ranjay Krishna. MolmoAct: Action reasoning models that can reason in space. In IEEE International Conference on Robotics and Automation (ICRA), pp. 2585–2592, 2026.arxiv.org/abs/2508.07917
  29. [36]Qixiu Li, Yaobo Liang, Zeyu Wang, Lin Luo, Xi Chen, Mozheng Liao, Fangyun Wei, Yu Deng, Sicheng Xu, Yizhong Zhang, Xiaofan Wang, Bei Liu, Jianlong Fu, Jiammin Bao, Dong Chen, Yuanchun Shi, Jiaolong Yang, and Baining Guo. CogACT: A foundational vision-language-action model for synergizing cognition and action in robotic manipulation, 2024. URL https://arxiv.org/abs/2411.19650.
  30. [37]Haozhan Li, Yuxin Zuo, Jiale Yu, Yuhao Zhang, Zhaohui Yang, Kaiyan Zhang, Xuekai Zhu, Yuchen Zhang, Tianxing Chen, Ganqu Cui, Dehui Wang, Dingxiang Luo, Yuchen Fan, Youbang Sun, Jia Zeng, Jiangmiao Pang, Shanghang Zhang, Yu Wang, Yao Mu, Bowen Zhou, and Ning Ding. SimpleVLA-RL: Scaling VLA training via reinforcement learning. In International Conference on Learning Representations (ICLR), 2026.arxiv.org/abs/2509.09674
  31. [38]Sean Lie. Cerebras architecture deep dive: First look inside the hardware/software co-design for deep learning. IEEE Micro, 43(3):18–30, 2023.DOI
  32. [39]Sihao Lin, Zerui Li, Xunyi Zhao, Gengze Zhou, Liuyi Wang, Rong Wei, Rui Tang, Juncheng Li, Hanqing Wang, Jiangmiao Pang, Anton van den Hengel, Jiajun Liu, and Qi Wu. VLVNVerse: A benchmark for vision-language navigation with versatile, embodied, realistic simulation and evaluation, 2025. URL https://arxiv.org/abs/2512.19021.
  33. [41]Thang Luong and Edward Lockhart. Advanced version of Gemini with Deep Think officially achieves gold-medal standard at the International Mathematical Olympiad, 2025. URL https://deepmind.google/discover/blog/advanced-version-of-gemini-with-deep-think-officially-achieves-gold-medal-standard-at-the-int
  34. [44]Open X-Embodiment Collaboration, Abby O’Neill, Abdul Rehman, Abhiram Maddıkuri, Abhishek Gupta, Abhishek Padalkar, Abraham Lee, Acorn Pooley, Agrim Gupta, Ajay Mandlekar, Ajinkya Jain, Albert Tung, Alex Bewley, Alex Herzog, Alex Irpan, Alexander Khazatsky, Anant Rai, Anchit Gupta, Andrew Wang, Anikait Singh, Animesh Garg, Aniruddha Kembhavi, Annie Xie, Anthony Brohan, Antonin Raffin, Archit Sharma, Arefeh Yavary, Arhan Jain, Ashwin Balakrishna, Ayzaan Wahid, Ben Burgess-Limerick, Beomjoon Kim, Bernhard Schölkopf, Blake Wulfe, Brian Ichter, Cewu Lu, Charles Xu, Charlotte Le, Chelsea Finn, Chen Wang, Chenfeng Xu, Cheng Chi, Chenguang Huang, Christine Chan, Christopher Agia, Chuer Pan, Chuyuan Fu, Coline Devin, Danfei Xu, Daniel Morton, Danny Driess, Daphne Chen, Deepak Pathak, Dhruv Shah, Dieter Büchler, Dinesh Jayaraman, Dmitry Kalashnikov, Dorsa Sadigh, Edward Johns, Ethan Foster, Fangchen Liu, Federico Ceola, Fei Xia, Feiyu Zhao, Freek Stulp, Gaoyue Zhou, Gaurav S. Sukhatme, Gautam Salhotra, Ge Yan, Gilbert Feng, Giulio Schiavi, Glen Berseth, Gregory Kahn, Guanzhi Wang, Hao Su, Hao-Shu Fang, Haochen Shi, Henghui Bao, Heni Ben Amor, Henrik I Christensen, Hiroki Furuta, Homer Walke, Hongjie Fang, Huy Ha, Igor Mordatch, Ilija Radosavovic, Isabel Leal, Jacky Liang, Jad Abou-Chakra, Jaehyung Kim, Jaimyn Drake, Jan Peters, Jan Schneider, Jasmine Hsu, Jeannette Bohg, Jeffrey Bingham, Jeffrey Wu, Jensen Gao, Jiaheng Hu, Jiajun Wu, Jialin Wu, Jiankai Sun, Jianlan Luo, Jiayuan Gu, Jie Tan, Jihoon Oh, Jimmy Wu, Jingpei Lu, Jingyun Yang, Jitendra Malik, João Silvério, Joey Hejna, Jonathan Booher, Jonathan Tompson, Jonathan Yang, Jordi Salvador, Joseph J. Lim, Junhyek Han, Kaiyuan Wang, Kanishka Rao, Karl Pertsch, Karol Hausman, Keegan Go, Keerthana Gopalakrishnan, Ken Goldberg, Kendra Byrne, Kenneth Oslund, Kento Kawaharazuka, Kevin Black, Kevin Lin, Kevin Zhang, Kiana Ehsani, Kiran Lekkala, Kirsty Ellis, Krishan Rana, Krishnan Srinivasan, Kuan Fang, Kunal Pratap Singh, Kuo-Hao Zeng, Kyle Hatch, Kyle Hsu, Laurent Itti, Lawrence Yunliang Chen, Lerrel Pinto, Li Fei-Fei, Liam Tan, Linxi “Jim” Fan, Lionel Ott, Lisa Lee, Luca Weihs, Magnum Chen, Marion Lepert, Marius Memmel, Masayoshi Tomizuka, Masha Itkina, Mateo Guaman Castro, Max Spero, Maximilian Du, Michael Ahn, Michael C. Yip, Mingtong Zhang, Mingyu Ding, Minho Heo, Mohan Kumar Srirama, Mohit Sharma, Moo Jin Kim, Naoaki Kanazawa, Nicklas Hansen, Nicolas Heess, Nikhil J Joshi, Niko Suenderhauf, Ning Liu, Norman Di Palo, Nur Muhammad Mahi Shafiullah, Oier Mees, Oliver Kroemer, Osbert Bastani, Pannag R Sanketi, Patrick “Tree” Miller, Patrick Yin, Paul Wohlhart, Peng Xu, Peter David Fagan, Peter Mitrano, Pierre Sermanet, Pieter Abbeel, Priya Sundaresan, Qiuyu Chen, Quan Vuong, Rafael Rafailov, Ran Tian, Ria Doshi, Roberto Martín-Martín, Rohan Baijal, Rosario Scalise, Rose Hendrix, Roy Lin, Runjia Qian, Ruohan Zhang, Russell Mendonca, Rutav Shah, Ryan Hoque, Ryan Julian, Samuel Bustamante, Sean Kirmani, Sergey Levine, Shan Lin, Sherry Moore, Shikhar Bahl, Shivin Dass, Shubham Sonawani, Shuran Song, Sichun Xu, Siddhant Haldar, Siddharth Karamcheti, Simeon Adebola, Simon Guist, Soroush Nasiriany, Stefan Schaal, Stefan Welker, Stephen Tian, Subramanian Ramamoorthy, Sudeep Dasari, Suneel Belkhale, Sungjae Park, Suraj Nair, Suvir Mirchandani, Takayuki Osa, Tanmay Gupta, Tatsuya Harada, Tatsuya Matsushima, Ted Xiao, Thomas Kollar, Tianhe Yu, Tianli Ding, Todor Davchev, Tony Z. Zhao, Travis Armstrong, Trevor Darrell, Trinity Chung, Vidhi Jain, Vincent Vanhoucke, Wei Zhan, Wenxuan Zhou, Wolfram Burgard, Xi Chen, Xiaolong Wang, Xinghao Zhu, Xinyang Geng, Xiyuan Liu, Xu Liangwei, Xuanlin Li, Yao Lu, Yecheng Jason Ma, Yejin Kim, Yevgen Chebotar, Yifan Zhou, Yifeng Zhu, Yilin Wu, Ying Xu, Yixuan Wang, Yonatan Bisk, Yoonyoung Cho, Youngwoon Lee, Yuchen Cui, Yue Cao, Yueh-Hua Wu, Yujin Tang, Yuke Zhu, Yunchu Zhang, Yunfan Jiang, Yunshuang Li, Yunzhu Li, Yusuke Iwasawa, Yutaka Matsuo, Zehan Ma, Zhuo Xu, Zichen Jeff Cui, Zichen Zhang, and Zipeng Lin. Open X-Embodiment: Robotic learning datasets and RT-X models. In IEEE International Conference on Robotics and Automation (ICRA), pp. 6892–6903, 2024.
  35. [46]OpenAI. Introducing GPT-5.4, 2026. URL https://openai.com/index/introducing-gpt-5-4/.
  36. [48]OpenAI, Ahmed El-Kishky, Alexander Wei, Andre Saraiva, Borys Minaiev, Daniel Selsam, David Dohan, Francis Song, Hunter Lightman, Ignasi Clavera, Jakub Pachocki, Jerry Tworek, Lorenz Kuhn, Lukasz Kaiser, Mark Chen, Max Schwarzer, Mostafa Rohanimanesh, Nat McAleese, o3 contributors, Oleg Mürk, Rhythm Garg, Rui Shu, Szymon Sidor, Vineet Kosaraju, and Wenda Zhou. Competitive programming with large reasoning models, 2025. URL https://arxiv.org/abs/2502.06807.
  37. [49]OpenSUN3D. 7th Workshop on Open-World 3D Scene Understanding and Representations, in conjunction with ECCV 2026, 2026. URL https://open3d.github.io/.
  38. [50]Jiayi Pan, Xingyao Wang, Graham Neubig, Naydeep Jaitly, Heng Ji, Alane Suhr, and Yizhe Zhang. Training software engineering agents and verifiers with SWE-Gym. In International Conference on Machine Learning (ICML), 2025.arxiv.org/abs/2412.21139
  39. [51]Karl Pertsch, Kyle Stachowicz, Brian Ichter, Danny Driess, Suraj Nair, Quan Vuong, Oier Mees, Chelsea Finn, and Sergey Levine. FAST: Efficient action tokenization for vision-language-action models, 2025. URL https://arxiv.org/abs/2501.09747.

Paper details

vlnc:W_y485224d
Version
v1 · latest
Published
Fresh — added to the Hub within the last 30 days
Added
License
CC BY 4.0
Usage

views · 11

downloads · 2

A view counts once per browser per day, a download each time the PDF is requested.

  • Computer Science

Not recorded for this paper

View statement
Looking for related papers…
Contents