betaFeedback

WorksComputer Science

Gym-Anything-RT: Scaling Real-Time Environments for Vision-Language Agents

Abstract

AI agents have made substantial progress in domains where they have a non-trivial amount of time to deliberate between actions, such as mathematics, software engineering, and performing tasks on desktop applications. It is less clear how such models fare in settings that demand real-time interaction, such as playing a video game or operating a physical device. In these settings, the environment state may evolve as the agent is reasoning, or even as the underlying model is performing a forward pass. To gain insight into the real-time interaction capabilities of vision-language agents, we introduce Gym-Anything-RT. Our key idea is to construct visual minigames that demand a portfolio of capabilities required for interactive real-time settings. Gym-Anything-RT uses an agentic pipeline to automatically generate such environments based on a taxonomy of six capabilities required for real-time interaction, including the ability to perceive the environment and act under an evolving environment state. A key feature of Gym-Anything-RT is that it allows for controlled comparison of real-time interaction capabilities: we can run the same minigame in real time, or pause or slow down the environment to relax the requirement for real-time actions. We use Gym-Anything-RT to construct 100 real-time minigames and evaluate AI agents on them. We find large capability gaps in real-time interaction. For example, GPT-6 Astra (xHigh) achieves a pass rate of only 9.0% on the real-time minigames, which rises to 75.0% when the environment is paused. We also identify intriguing behaviors by the most powerful frontier agents: for instance, when given the ability to write code, these agents write code that bypasses the need to perceive and act in real time. We publicly release Gym-Anything-RT and the generated minigames to spur further research on AI agents in real-time interactive settings.

Introduction

AI agents have made substantial progress in domains where they have time to deliberate between actions, such as mathematics [16], software engineering [26], and performing tasks on desktop applications [3, 24] (Yuan et al. 2026). However, these agents have a key limitation: they are slow, often taking more than ten seconds to choose a single action. This is acceptable when the environment waits for the agent, but not in settings that demand real-time interaction, such as playing a video game or operating a physical device [6] (Zhang et al. 2026). There, the environment keeps changing while the agent reasons, or even while the model performs a forward pass. By the time an action executes, it may no longer be valid, and the agent cannot react to what happened in the meantime.

Existing evaluations shed little light on this limitation. Computer-use benchmarks wait for the agent between actions, so they never test acting under an evolving state [3] (Xie et al. 2024; Yuan et al. 2026). Video game benchmarks do run in real time. However, frontier models make little progress on them [35] (Zhang et al. 2026; Ying et al. 2026). Furthermore, each game combines many capabilities, such as 3D spatial understanding, temporal reasoning, and learning unfamiliar controls, in a single fixed task. This makes it difficult to tell whether models fail because they are too slow or because they lack some other capabilities. Separating these two causes requires controlled comparisons that vary how much time an agent has to respond on the same task.

To address these challenges, we introduce GYM-ANYTHING-RT, a framework for creating environments for studying the real-time interaction capabilities of vision-language agents. Our key idea is to construct visual minigames that demand a portfolio of capabilities required for interactive real-time settings, while avoiding the complexity of complete games. For instance, in Rotating On-Screen Keyboard Figure 1, the agent must enter a code by clicking keys that keep moving in three-dimensional space between its observation and its action. Furthermore, we build an evaluation stack that controls the clock of each minigame, so we can run the same task live at its natural speed, slow it down to give the agent more time to respond, or pause it between observation and action to remove the need for real-time actions. Since each minigame is a program, we can also modify it, for instance, to remove its temporal dynamics or to increase its difficulty. By comparing an agent’s performance on these variants of the same task, we can tell whether it fails because it cannot act in time as the environment changes, or because the task is hard for other reasons.

Examples of real-time minigames generated with GYM-ANYTHING-RT

Figure 1. Examples of real-time minigames generated with GYM-ANYTHING-RT. The 100 minigames in our benchmark cover different combinations of capabilities, with controlled variations in difficulty and interaction.

Building such minigames by hand is laborious, since each needs its own visuals, dynamics, controls, and verifier. We therefore build an agentic pipeline that generates the minigames automatically. We first survey prior work on embodied agents [43, 49] (Yang et al. 2025c), video games (Zhang et al., 2026), and computer use [11] (Xie et al. 2024) to identify six capabilities required for real-time interaction, such as 3D visual understanding, temporal reasoning, and acting under an evolving environment state, and select minigames that cover the capabilities. We then build on the creator-auditor procedure of Gym-Anything [3], in which one coding agent builds an environment and an independent agent audits it. In our setting, a creator agent implements each minigame as a program, and additionally writes an oracle solution that uses the program’s internal state to show that the task is solvable. Independent audit agents then test the minigame and report problems to the creator agent, which fixes them until the minigame behaves as intended. Finally, review agents label the capabilities that each minigame requires. Each program also defines four easier difficulty levels and a simplified interaction mode, which let us evaluate weaker models and study what makes a task hard.

Using this pipeline, we construct a benchmark of 100 real-time minigames and their non-real-time counterparts, and use it to evaluate frontier and open-weight models. We find that current agents perform poorly in real time. GPT-6 Astra (xHigh) solves only 9.0% of the real-time tasks, while it solves 66.0% of their non-real-time counterparts, which we obtain by modifying each program so that it no longer requires real-time interaction. Other models perform even worse: Claude Opus 5.5 (xHigh) and GPT-6 Luna (xHigh) solve only 5.0% and 1.0% of the real-time tasks, and the open-weight Qwen3.8-27B and Qwen3.5-9B solve only 1.0% and 0.0%, respectively. In addition, we find that for frontier models, the primary bottleneck is speed. On the same real-time tasks, slowing the environment by 64× raises Astra’s pass rate from 9.0% to 57.0%, and pausing it raises the pass rate to 75.0%. Since we also control difficulty and the interaction interface, we further identify temporal reasoning and exploration of unfamiliar interfaces as additional sources of difficulty.

Interestingly, we also find that frontier agents can partially offset their poor real-time capabilities through their strong coding and reasoning capabilities. Specifically, when we allow Astra to run code, it writes scripts that perceive and act on its behalf between its own responses. For instance, in Rotating On-Screen Keyboard, it locates the moving keys using image template matching and clicks them within 200 ms. As a result, its real-time pass rate rises from 9.0% to 31.0%.

In summary, our contributions are three-fold. First, we introduce GYM-ANYTHING-RT, a framework that uses coding agents to automatically generate real-time visual minigames as programs, with oracle solutions and controlled difficulty and interaction. Its evaluation stack runs each minigame live, slowed down, or paused. Second, using GYM-ANYTHING-RT, we construct a benchmark of 100 real-time minigames and their non-real-time counterparts for studying the real-time interaction capabilities of vision-language agents. Third, through extensive evaluations, we show that current agents perform poorly in real time, that speed is the primary bottleneck for frontier models, and that frontier agents can partially offset their slow responses by writing code.

Methodology

Our goal is to develop a suite of minigames that test the real-time interaction capabilities of vision-language agents in a controllable way. In particular, we construct minigames in which the time available to respond can be controlled while the underlying task stays the same. Formally, each minigame is an executable environment E\mathcal{E} whose program defines its internal state, controls, and clock. At each step, the agent observes a screenshot oto_t and issues mouse and keyboard actions ata_t, while the environment keeps evolving as the agent chooses ata_t. We say that a task requires real-time interaction when the agent must use recent visual evidence to choose an action for the state in which it will execute, so that changes during the delay between observation and action can reduce that action’s value (Appendix § 15).

Each minigame has ten configurations Ed,i\mathcal{E}_{d,i}, where d∈{1,…,5}d \in\{1,\ldots,5\} specifies the difficulty and i∈{simplified,full}i \in\{\text{simplified},\text{full}\} specifies the interaction mode. Each configuration can be run with a slowdown ss: live execution at its natural speed (s=1s=1), slowed execution (s>1s>1), in which the clock advances at 1/s1/s of its normal rate, and paused execution, in which the environment freezes until the agent acts. In addition, each real-time minigame has a non-real-time counterpart E′\mathcal{E}', obtained through a minimal change to its timing (§ 2.3). Since a broken or unsolvable minigame would be counted as an agent failure, every configuration must also be solvable and objectively gradable. We construct these minigames with coding agents, that is, LLM agents that can write and run code and inspect screenshots in a sandbox. All agents are instances of GPT-6 Astra run through the Codex CLI. Each agent takes on one of four roles: survey agents collect candidate minigames, creator agents implement real-time versions of them, audit agents verify them, and review agents calibrate difficulty and annotate capabilities. Figure 2 summarizes this procedure.

Overview of our construction method

Figure 2. Overview of our construction method. A creator agent implements selected minigames with controllable difficulty and interaction, constructs oracle solutions, and records evidence of their behavior. An independent auditor checks the implementation and evidence, returning issues for revision until the minigame passes the audit. Review agents calibrate difficulty and annotate capability requirements for controlled model evaluation.

Task Discovery and Selection

We prompt survey agents to search prior work, public demonstrations, visual puzzles, online videos, and social media for challenges that satisfy four criteria: they can be formulated as self-contained minigames, require visual understanding and grounding, require actions through a visual interface, and have objectively verifiable outcomes. These include tasks that involve moving objects, hidden information, physical manipulation, or changes that occur between actions. For every source, the survey agent records its provenance, objective, visual structure, required interactions, temporal behavior, failure and recovery process, and information needed to verify success.

We manually review the first 15 minigames and use them as quality standards for subsequent agent iterations. We select candidates with a traceable source, distinct visual and interaction requirements, an objectively gradable outcome, and meaningful mouse or keyboard actions. A source need not involve real-time interaction itself, since the creator agent develops its mechanic into a real-time task (§ 2.2). We exclude tasks dominated by OCR, static image classification, arithmetic, or question answering, since these tasks do not require the forms of visual interaction that we seek to measure.

Creator-Auditor Construction

To construct a real-time minigame E\mathcal{E} from a source PP, we build upon the creator-auditor procedure from Gym-Anything ([3]), in which a creator agent builds each environment and independent audit agents verify it. In our setting, the creator agent writes a program that implements the minigame and exposes a common interface for screenshot observations as well as mouse and keyboard actions. Crucially, we ask the creator agent to turn the task posed by the source into one that requires real-time interaction, as defined above, whether or not the source itself does. Adding decorative motion, a countdown, or faster animation is not sufficient. For example, Rotating On-Screen Keyboard requires the agent to click keys that keep moving between observation and action.

The creator agent implements the generator, interface, dynamics, action handling, and grader of E\mathcal{E}. It also constructs an oracle solver that uses the privileged state of the environment to determine a valid solution and executes it through the same action space available to the agent. A successful oracle execution therefore demonstrates that the task is solvable and that its action and grading paths agree. The creator agent records the implementation, oracle trajectory, and visual and temporal evidence for these claims (Appendix § 9.§ 1).

Each implementation is then audited by a fresh audit agent that cannot modify the program. The audit agent inspects the code and evidence, runs the environment when necessary, and checks generation, solvability, rendering, action handling, grading, and recovery from failure. It also checks the real-time requirement itself, searching for shortcuts that would let an agent avoid acting in real time, such as a brake that halts the moving parts or periodic motion that permits a precomputed solution. The audit agent returns issues to the creator agent for revision, and the loop repeats until an audit passes.

Non-Real-Time Counterparts

A low pass rate on a real-time minigame does not by itself show that real-time interaction is the cause, since the underlying task may also be hard. We therefore construct, for each real-time minigame E\mathcal{E}, a non-real-time counterpart Eˉ\bar{\mathcal{E}} that poses the same task without requiring real-time interaction. The creator agent makes a minimal change to the timing of E\mathcal{E}, while preserving its goal, controls, difficulty, and the information available to the agent. An independent audit agent then checks across all ten configurations that the two versions differ only in their timing. Comparing an agent on E\mathcal{E} and Eˉ\bar{\mathcal{E}} thus lets us separate the effect of the real-time requirement from the difficulty of the underlying task.

Controlled Task Variations

To also evaluate weaker models and study what makes a task hard, we construct easier variants of each minigame. We vary the difficulty dd and the interaction mode ii while holding the other and the random seed fixed, for both E\mathcal{E} and its counterpart.

A difficulty-review agent first reads the generator, visible interface, grader, verifier, and oracle solver to determine what the original minigame requires. It compares this configuration with an approved set of reference minigames and assigns it to one of five levels. The creator agent preserves the original configuration at that level and constructs the remaining levels around it by changing the amount of information, number of dependencies, speed, precision, dynamics, or simultaneous control required for success. A difficulty-review agent then checks each adjacent pair and rejects any spurious difficulty increase, such as requesting repeated actions or adding ambiguity. The agent also ensures that the same types of perception, memory, decisions, and control are required across difficulty levels. For example, the levels of Rotating On-Screen Keyboard (Figure 3) vary the code length, the number and size of keys, and the keyboard motion.

Rotating On-Screen Keyboard

Figure 3. Rotating On-Screen Keyboard. The model must enter a code by clicking keys that move between observation and action.

The creator agent also implements simplified and full interaction for every difficulty level. Simplified interaction exposes proxy controls, while full interaction requires the direct mouse or keyboard manipulation appropriate to the minigame. For example, in Rotating On-Screen Keyboard, simplified interaction lets the agent type the code directly with keyboard actions, while full interaction requires clicking the rotating keys. For a fixed seed and difficulty, the audit agent verifies that both modes generate the same minigame and differ only in the controls used to act on it. Appendices § 9.§ 2 and § 9.§ 3 give examples of both variations.

Controlling Time

To control how much time an agent has to respond, we build an evaluation stack on top of Gym-Anything ([3]) that controls the clock of each minigame. During live execution, the task continues to change from observation through action execution at its natural speed. In Rotating On-Screen Keyboard, for example, a key can move between observation and action, so the agent must account for its position when the click executes. Since any delay between an action and its effect counts against the agent, we also build a direct connection protocol. A central server still launches and manages many sandboxes in parallel, but each agent exchanges actions and screenshots over a direct connection to its own sandbox, bypassing the server. This keeps the median time from sending an action to receiving the resulting screenshot below 50 ms in our timing tests (Appendix § 12.§ 1). For slowed execution, the stack scales every clock of the minigame, including its timers, animations, and server-side clocks, so that a slowdown of ss advances the game by one second during ss seconds of real time. Action delivery and screenshot capture still run in real time, so the slowdown changes the time available to respond while the agent’s interaction stays the same. During paused execution, the task advances through an observation window and then freezes on the final frame until the action is executed. All settings use the same minigame, seed, difficulty, interaction mode, and verifier. Appendix § 12 describes the evaluation protocol.

Capability Review

To relate agent failures to the capabilities a minigame requires, a capability-review agent labels each configuration of both the real-time minigames and their counterparts for 3D visual understanding, temporal reasoning, reasoning and planning, and exploration and interface understanding (see Appendix § 11 for definitions). It reviews the game’s program, interface, verifier, oracle solution, and recorded behavior to identify what a solution using screenshots and mouse or keyboard actions requires. Each label is supported by code or behavioral evidence. A real-time review agent checks the real-time requirement separately for each configuration, using the implementation and recorded behavior (Appendix § 15).

Gym-Anything-RT

We use the pipeline from § 2 to construct Gym-Anything-RT, comprising 100 real-time minigames and their 100 non-real-time counterparts. Each also has four easier difficulty levels and a simplified interaction mode, giving 2,000 configurations in total. We evaluate models using the live, slowed, and paused settings defined in § 2.5.

Capability Coverage

Table 1 summarizes Gym-Anything-RT. Using the labels from the capability review (§ 2.6), 31% of the real-time minigames require 3D visual understanding at some difficulty level, all of them require temporal reasoning, 96% require reasoning and planning, and 52% require exploration and interface understanding. Their counterparts require the same capabilities at similar rates, except temporal reasoning, which drops to 82% once the real-time requirement is removed. Table 7 lists all minigames and their capability labels.

StatisticCount
Real-time minigames100
Non-real-time counterparts100
Difficulty levels5
Interaction modes2
Configurations2,000
Real-time1,000
Non-real-time1,000

Table 1. Benchmark statistics.

Minigame3DTREMinigame3DTRE
After Hours at the Reliquary◦••Impossible Ecology•••
Anthill Front•••Impossible Panorama••••
Apothecary Dead Reckoning•••Isometric Voxel Extraction Mine•◦••
Ballast Lantern••Lampwrights Program•◦•
Bandaged Rose Window•••Lantern Loft•••
Blind Corridor Oscilloscope•••Lanternfin Dive•••
Blind Dice Courier••Lanternwing Roundup•••
Bomb Manual From Hell◦•Letter Rapids••
Branch Repair••••LIDAR Blacksite•••
Bureaucratic Signature Trap•••Live Control-Flow Wiring Lab•••
Cell Gatekeeper••Living Scaffold••⋄
Charter of the Nine Cantons••Magnetic-Stripe Purgatory•••
Circle Limit Twist◦••Motion-Only Ghost Jigsaw••
Clockbeat Catacomb••Museum of Lost Gestures••
Clockwork Clutch Safe◦••Occlusion Shell Swindle•••
Clockwork Doppelgänger Customs••••Parallax / Inertial Jigsaw Alignment•••
Cloudpost Circuit•••Parallax Orchard••••
Cloudstep Caddie•••Parallel Grillmaster◦•
Collision Chimes◦••Pearl Lattice•••
Concertina Courier••Photograph Eats the Room◦••◦
Consent Gauntlet◦••Pocket Locksmith••••
Consequences Boss••Polarized Palimpsest••
Coordinates by Another Name•••Polycube Parcel•••◦
Courtesy Junction••Polyrhythm Customs••
Crackglaze Crossing••⋄Popup Exorcist•••
CRAFTCHA: Alchemy Bench••Rayglass Vault•••
Crash-Deadline Hovercar••Recursive Dollhouse Smuggling••
Crater Walker•••Reflow Vitrine•••
Cursor-Controlled Constellation Hunt•••Reload Interruption••
Dead Man’s Switch••Reveal to Identify•••
Downsky Causeway•••Ribbon Switchboard••
Dual-Projection Sculpture Rig•◦••Rising Causeway•••⋄
Elastic Membrane Sorter••Rorschach / Subjective Prompt With A Fixed Rubric•••
Elbow Engine••Rotate The Wrong Thing Upright•••
Ember Anvil◦◦••Rotating On-Screen Keyboard◦◦
Facet Lantern••••Shadow Crime Lab•••
Firewatch Fold•••Slot-Reel Character Capture◦
First Change Memory•••Surveyors Toybox•••
Five-Second Rule••The Flat Prisoner•••
Five-Step Verification Reactor••The Last Carbon Isles•••
Flip-Gate Cascade•••The Unlabeled Drawer◦•
Fluke Census•••The Unwatched Wing••
Forced-Perspective Moving Day•◦•Three-Camera Claw Machine•••
Four-Pane Pilgrimage••Tiny FPS Customs••••
Four-Tab Robot Handshake••⋄Tomographic Baggage Surgery••••
Funeral Ritual•••Top-Face Dice Arithmetic•••
Gyroscopic Tilt Board••Two-Lamp Dyeworks•••
Hearthlift Courier••••Wonky Text Under Hostile Rendering•••
Hologram Silhouette Foundry•••Wrong Number◦•◦
Horizon Relay⋄◦•Zero-G Cable Autopsy••

Table 7. Minigames in GYM-ANYTHING-RT. Capability labels: 3D visual understanding (3D), temporal reasoning (T), reasoning and planning (R), and exploration and interface understanding (E). A capability is marked when it is required at some difficulty level. •: required by the real-time minigame and its non-real-time counterpart. ◦: required only by the real-time minigame. ⋄: required only by the counterpart.

Evaluation Protocol

We evaluate models by pass rate, time to solve, and inference cost, and compare these metrics across real-time minigames and their counterparts, slowdowns, difficulty levels, interaction modes, and capability requirements. Models receive task instructions and interact through screenshots, mouse actions, and keyboard input, without access to program internals or oracle solutions. Task-specific verifiers determine whether each attempt succeeds. Appendix § 13 describes the models, prompts, budgets, and cost computation.

Results and Analysis

Experimental Setup

We evaluate Qwen3.5-9B, Qwen3.8-27B, GPT-6 Luna with xHigh reasoning effort, Claude Opus 5.5 with xHigh effort, and GPT-6 Astra with low and xHigh reasoning effort. We evaluate models in the live setting with full interaction at the highest difficulty, and § 4.9 reports results on the easier difficulty levels. Following prior work on computer-use agents ([2] (Xie et al. 2024)), models observe the environment only through screenshots and act only through mouse and keyboard actions. Models cannot run code or access the page source or program internals. In the coding experiment, we additionally allow models to run code in an isolated sandbox, where they can request screenshots and execute the same mouse and keyboard actions through an API. Interaction with the environment remains the same. Further experiments vary difficulty, interaction mode, the real-time setting, and whether interface rules are provided.

Current Agents Perform Poorly in Real Time

Figure 4 compares the pass rates of each model on the real-time tasks and on their non-real-time counterparts (§ 2.5). Even the strongest model, GPT-6 Astra (xHigh), solves only 9.0% of the real-time tasks at the highest difficulty, compared with 66.0% of their non-real-time counterparts. Interestingly, higher reasoning effort helps only when the task waits for the model: from Low to xHigh effort, Astra’s pass rate rises from 58.0% to 66.0% on the non-real-time counterparts, but only from 8.0% to 9.0% on the real-time tasks. Claude Opus 5.5 (xHigh) solves 5.0% of the real-time tasks and 57.0% of their counterparts. GPT-6 Luna (xHigh) performs worse on both, solving 1.0% of the real-time tasks and 17.0% of their counterparts, and the open-weight Qwen3.8-27B and Qwen3.5-9B solve only 1.0% and 0.0% of the real-time tasks. These two models also solve only 2.0% and 1.0% of the counterparts, so their failures are not specific to real time. Since each real-time task and its counterpart share the same minigame, these results show that acting in real time, rather than the underlying task, accounts for most failures of frontier models. In the following sections, we examine why.

Pass rates for real-time and non-real-time tasks across models

Figure 4. Pass rates (%) on GYM-ANYTHING-RT at difficulty 5 with full interaction in the live setting. Each model is evaluated on the 100 real-time tasks and their 100 non-real-time counterparts. Even the strongest model performs poorly on tasks requiring real-time interaction.

Speed Is the Primary Bottleneck for Frontier Models

To study how response speed limits performance, we use our evaluation stack to slow down the environment while leaving the model’s inference unchanged (§ 2.5). A slowdown of ss advances the environment clock at 1/s1/s of its normal rate, giving the model more time to respond before the task changes, while pausing freezes the environment until the model acts. Figure 5 reports the pass rate of GPT-6 Astra on the same 100 real-time tasks at the highest difficulty as we slow the environment. With xHigh reasoning effort, pass rate rises steadily from 9% at normal speed to 17% at 4×, 48% at 16×, and 57% at 64× slowdown, and reaches 75% when the environment is paused. Pausing similarly raises GPT-6 Luna’s pass rate on real-time tasks from 6.0% to 29.8% across all difficulty levels. Since the tasks are unchanged and only the time available to respond grows, these results show that for frontier models, slow responses are the primary reason for failure in real time.

Effect of environment speed on Astra

Figure 5. Effect of environment speed on Astra. Pass rates increase as the environment slows, with xHigh remaining below its paused performance at 64× slowdown.

Moreover, the benefit of additional time grows with reasoning effort. At normal speed, Low and xHigh effort perform similarly, 8% and 9%, since additional reasoning also takes additional time. As the environment slows, xHigh pulls ahead, reaching 57% at 64× compared with 42% for Low, and

75% compared with 58% when paused. Interestingly, even at 64×64 \times slowdown, Astra (xHigh) remains 18 points below its paused performance, suggesting that some tasks remain difficult as long as the environment keeps changing at all.

Strong Coding Capabilities Partially Offset Weak Real-Time Capabilities

Since frontier models are strong coding agents, we next examine whether they can use code to compensate for their slow responses. We allow models to run code in an isolated sandbox, where they can request screenshots and execute the same mouse and keyboard actions through an API (Appendix § 13.§ 5). The action space is unchanged, but models can now write scripts that observe and act without waiting for the model’s next response. Table 2 reports results at the highest difficulty with full interaction in the live setting. Coding raises the pass rate of Astra (xHigh) on real-time tasks from 9.0% to 31.0%, a 3.4-fold increase, compared with a gain of only 8.0 points on their non-real-time counterparts, from 66.0% to 74.0%. The gain also grows with model capability: Astra (Low) improves from 8.0% to 14.0%, and Luna only from 1.0% to 5.0%.

ModelReal-timeNon-real-time
CodingNon-codingCodingNon-coding
GPT-6 Astra (Low)14.08.063.058.0
GPT-6 Astra (xHigh)31.09.074.066.0
GPT-6 Luna (xHigh)5.01.018.017.0

Table 2. Effect of coding on real-time performance. Astra benefits most from coding on tasks requiring real-time interaction.

To understand this gain, we inspect Astra’s trajectories. Astra writes image-processing scripts that perceive and act between its own responses. For instance, in Rotating On-Screen Keyboard, it locates the moving keys using image template matching and clicks them within 200 ms. In Clockwork Clutch Safe, where moving gears must be aligned, it tracks gear markers and times clutch releases using estimates of their motion (Appendix § 14). In effect, the model moves the real-time part of the task into code, and reasons only about what the code should do. This suggests that similar scripts could help slow models act in other real-time domains, such as video games and physical interfaces. Nonetheless, coding still leaves Astra far below its paused performance of 75.0%, and such scripts are difficult to write when the relevant visual information cannot be extracted reliably or prepared for in advance.

Models Are Slow and Expensive

Real-time applications require agents that are not only accurate, but also fast and affordable. Table 3 reports the pass rate, average time, and average inference cost per attempt on the 100 real-time tasks at the highest difficulty. Astra (xHigh) achieves the highest pass rate, 9.0%, but takes 7.2 minutes and costs $4.50 per attempt on average. Lowering its reasoning effort to Low reduces the time to 4.3 minutes and the cost to $2.76, with a similar pass rate of 8.0%, consistent with additional reasoning helping little when the environment does not wait. Claude Opus 5.5 (xHigh) costs the most, $6.40 per attempt, and takes 15.8 minutes, while solving only 5.0% of the tasks. Luna is far cheaper at $0.07 per attempt, but takes 6.4 minutes and solves only 1.0% of the tasks. Qwen3.8-27B and Qwen3.5-9B cost 0.64and0.64 and0.62 per attempt, but take about ten times longer than Luna, 63.0 and 75.9 minutes, while solving only 1.0% and 0.0% of the tasks. As a case study, we compare these results with those of a human author who, like the evaluated models, sees each minigame for the first time. On 20 randomly sampled tasks, the human solves 9 tasks, or 45.0%, in an average of 2.2 minutes per task (Appendix § 13.7). This is five times the pass rate of Astra (xHigh), in less than a third of its average time. These results show that progress in real-time interaction requires improving not only the pass rate of models, but also their speed and cost ([1, 6]).

AgentPass rate ↑\uparrow (%)Avg. time ↓\downarrow (min)Avg. cost ↓\downarrow ($)
Qwen3.5-9B0.075.90.62
Qwen3.8-27B1.063.00.64
GPT-6 Luna1.06.40.07
Claude Opus 5.55.015.86.40
GPT-6 Astra (Low)8.04.32.76
GPT-6 Astra (xHigh)9.07.24.50
Human45.02.2–

Table 3. Efficiency on the real-time configurations at difficulty 5. All models are slower and less successful than the human author, who is evaluated on 20 randomly sampled tasks.

Pausing Also Helps When Temporal Reasoning Is Required

Besides response speed, our controls let us study which other capabilities limit agents. We first examine whether the benefit of pausing is limited to real-time tasks. Figure 6 compares live and paused execution for GPT-6 Luna with full interaction, averaged across all five difficulty levels. Interestingly, pausing also helps on the non-real-time counterparts that require temporal reasoning, raising Luna’s pass rate from 20.4% to 31.4%, while the gain on counterparts without temporal reasoning, from 45.2% to 52.7%, is not statistically significant (p=0.443p = 0.443). Since these counterparts do not require acting in real time, this suggests that models also struggle to reason about scenes that change over time, even when they need not act quickly, identifying temporal reasoning as a further source of difficulty. Moreover, even when paused, Luna solves only 29.8% of the real-time tasks, showing that for weaker models, response speed is only one of several bottlenecks.

Bar chart comparing Live and Paused pass rates for real-time required, non-real-time temporal reasoning required, and non-real-time temporal reasoning not required tasks. Values are 6.0 and 29.8; 20.4 and 31.4; 45.2 and 52.7.

Figure 6. Effect of pausing on Luna. Pausing improves performance when real-time interaction or temporal reasoning is required. pp-values are Holm-adjusted.

Performance Decreases as More Capabilities Are Required

Since review agents label the capabilities each configuration requires, we can also relate performance to the demands of a task. Figure 7 reports Luna’s pass rate as a function of the number of required capabilities, using full interaction across all five difficulty levels. In the live setting, pass rate decreases from 37.0% when one capability is required to 7.2% when all four are required, and paused execution shows the same trend, from 55.6% to 23.2%. Since the decline persists when the environment is paused, tasks become harder as capabilities are combined, even without time pressure. However, since capability requirements overlap across tasks, this analysis cannot isolate the effect of a single capability. We therefore examine interface exploration directly through a controlled experiment in § 4.8.

Line chart of Live and Paused pass rates as a function of the number of required capabilities, from 0 to 4.

Figure 7. Performance with increasing capability requirements. Luna’s pass rate generally decreases as more of the four task-level capabilities are required.

Providing Interface Rules Improves Performance

Many minigames require the agent to learn how an unfamiliar interface behaves before it can solve the task. To isolate this demand, we append an environment-specific rulebook describing the controls, action effects, and world rules to Luna’s task prompt (Appendix § 13.§ 6), and evaluate with full interaction in the live setting on minigames requiring exploration and interface understanding. Figure 8 shows that providing these rules raises Luna’s pass rate from 11.3% to 16.2% across the five difficulty levels. The model must still solve the same minigame and carry out its solution through the same controls, so this gain comes from knowing the rules in advance. This identifies interface exploration as a further source of difficulty: Luna can use interface knowledge that it struggles to acquire through interaction.

Bar chart comparing pass rates without rules and with rules, with values 11.3 and 16.2.

Figure 8. Effect of providing interface rules. Luna solves more tasks when given the rules, across real-time and non-real-time environments.

Performance Decreases with Difficulty and Improves with Simplified Interaction

Finally, we turn to the easier difficulty levels and simplified interaction. Table 4 reports Luna’s pass rates in the live setting with full interaction across all five difficulty levels. While Luna solves only 1.0% of the real-time tasks at the highest difficulty, it solves 15.0% at difficulty 1, so the easier levels leave room to measure progress for weaker models. On their non-real-time counterparts, its pass rate falls from 35.0% to 17.0%. The gap between the two stays between 14.0 and 27.0 points at every difficulty, so acting in real time is a major source of failure even on the easiest tasks. To study difficulty without this effect, Figure 9 reports Luna’s pass rate on real-time tasks across the five difficulty levels under paused execution. Averaged across both interaction modes, pass rate decreases from 53.0% at difficulty 1 to 15.5% at difficulty 5, showing that the difficulty levels provide a controlled range of challenge. Simplified interaction raises the overall pass rate from 29.8% to 33.2%, with larger gains at higher difficulty. Since both modes realize the same minigame, this gain shows that some failures arise from operating the full controls rather than from solving the task. However, the decline with difficulty persists under simplified interaction, showing that the challenges introduced at higher difficulty levels are not only about the controls.

Line chart comparing Simplified and Full controls across difficulty levels 1 to 5. Simplified values are 53, 41, 30, 23, and 19; Full values are 53, 41, 29, 14, and 12.

Figure 9. Effect of difficulty and interaction. Under paused execution, simplified controls improve Luna’s performance at higher difficulties on real-time tasks.

Difficulty
12345All
Real-time15.06.04.04.01.06.0
Non-real-time35.033.022.018.017.025.0
Gap20.027.018.014.016.019.0

Table 4. Pass rates (%) of GPT-6 Luna (xHigh) across difficulty levels, with full interaction in the live setting. Gap is the pass rate on the non-real-time counterparts minus the pass rate on the real-time tasks.

Related Work

Real-time interaction. Acting in a changing environment requires accounting for the delay between observing it and acting on it. Robotics methods handle this delay through asynchronous execution or by predicting the state at execution time [6, 47, 51], and reinforcement learning studies decision making under action and observation delays [39, 18]. For computer-use agents, OSWorld-Human measures latency and finds that agents take 2.7 to 4.3 times more steps than necessary [1]. Game benchmarks either pause the game during inference [35] or compare paused and real-time play [38] (Zhang et al. 2026), and Real-Time Reasoning Gym adds time pressure to text-based games [58]. However, these comparisons cover only a few visual games or text-based environments. Gym-Anything-RT pairs each of 100 real-time minigames with a non-real-time counterpart and controls the clock of each minigame, so that we can vary how much time an agent has to respond on the same task.

Benchmarks for interactive visual agents. Benchmarks for vision-language agents cover websites [27] (Zhou et al. 2024), desktop and mobile applications [7, 40] (Xie et al. 2024; Yuan et al. 2026), professional software [2] Aggarwal et al., 2026), games [36, 22, 35, 4] (Zhang et al. 2026; Zheng et al. 2025a; Ying et al. 2026), and embodied simulators [43, 46, 49] (Yang et al. 2025c). However, most of these environments wait for the agent between actions, and each task requires many capabilities at once, so low performance does not reveal which capability limits the agent. Prior works address the latter by mapping tasks to capabilities (Wu et al., 2023), evaluating perception, reasoning, and decision making separately [56, 29] (Xu et al. 2026), or varying one condition of a task, such as difficulty, the initial state, or the interface [15, 30] (Zheng et al. 2025b; Zhao et al. 2025). GYM-ANYTHING-RT instead varies the real-time requirement, difficulty, and interaction interface of the same minigame.

Capability-specific evaluation of VLMs. Many benchmarks test individual capabilities of VLMs through questions about images or videos, such as 3D spatial understanding [55, 33, 25] (Yang et al. 2025a), temporal reasoning [31, 44] (Zhang et al. 2024), GUI grounding [11, 28], and abstract reasoning [13, 21, 48]. However, the model answers a question about a fixed input, without acting on what it perceives. iVISPAR and VisGym evaluate visual reasoning through interaction [34, 57]. GYM-ANYTHING-RT further requires agents to perceive and act while the minigame keeps changing.

Automated environment construction. Recent works use coding agents and LLMs to construct computer-use environments [3, 53, 8], text-based environments (Zhang et al., 2025), and environments with tasks of varying difficulty [23, 19], while procedural generators vary the complexity of task instances [14, 42, 12, 50, 9]. However, none of these produce visual minigames whose timing, difficulty, and interaction interface can be varied while preserving the underlying task. We extend the creator and audit agents of Gym-Anything [3] to construct such minigames, each with a non-real-time counterpart and an oracle solution. Appendix I provides a more detailed related work.

Conclusion

In this work, we introduced GYM-ANYTHING-RT, a scalable framework for creating environments for studying the real-time interaction capabilities of vision-language agents. In GYM-ANYTHING-RT, coding agents implement and audit each environment as a visual minigame program with an oracle solution, and our evaluation stack controls the clock of each minigame. As a result, we can vary how much time an agent has to respond, whether the task requires real-time interaction, its difficulty, and its interaction interface on the same minigame. Using GYM-ANYTHING-RT, we constructed a benchmark of 100 real-time minigames and showed that current agents perform poorly in real time, with the strongest agent, GPT-6 Astra (xHigh), solving only 9.0% of the real-time tasks compared with 66.0% of their non-real-time counterparts. We further showed that speed is the primary bottleneck for frontier models, and that temporal reasoning and exploration of unfamiliar interfaces are additional sources of difficulty. Interestingly, we also find that frontier agents can partially offset their poor real-time capabilities by writing code that perceives and acts on their behalf.

More broadly, our results suggest that progress in real-time interaction requires agents that are not only capable but also fast. We hope that GYM-ANYTHING-RT and the released minigames provide a foundation for future work on real-time agents, including faster models, agents that move the time-critical parts of a task into code, and training on real-time environments.

Acknowledgements

We thank Daytona for their generous support in sandbox compute credits. We thank Jing Yu Koh, Lawrence Jang, Seungone Kim, Zeyu Zheng, and other members of the L3 Lab for feedback and helpful discussions. Pranaj Aggarwal is supported by a SoftBank Group-Arm Fellowship. This work is partially supported by the National Science Foundation under Grant No. DMS-2502281.

Limitations

We focus on evaluating existing agents, although our programmatic task generators and automatic verifiers also provide a basis for training. Studying whether training on these minigames improves real-time interaction is an important direction for future work.

Our minigames capture specific demands of real-time environments to enable controlled comparisons across capabilities, difficulty, and interaction. Whether improvements on these tasks transfer to complete games and physical interfaces remains an open question.

Finally, GYM-ANYTHING-RT covers a selected set of capabilities and task mechanics. Our creation–audit framework provides a way to extend this coverage with additional minigames and capability combinations.

Minigame Construction

This appendix details the construction procedure summarized in § 2. All agents in the construction pipeline use GPT-6 Astra.

Creator and Audit Agents

The creator agent receives one selected minigame together with its survey records and source links. It also inspects previously approved minigames as working examples of the environment structure. The creator implements the environment specification, task, generator, browser interface, server grader, exported verifier, oracle solver, difficulty and interaction controls, real-time settings, and tests. It may create or edit only the files of its own minigame and cannot modify the shared runtime, evaluator, or clock. The creator runs all ten difficulty and interaction configurations in the browser in the live and paused settings and checks generation, grading, result export, verification, failure, and retry. It records screenshots, recordings, and command outputs as evidence, together with a solution video in which the oracle solver completes the task in the browser.

A fresh audit agent then inspects the environment and its evidence without modifying any files. It checks the following items and ends its report with a verdict of PASS or REVISION_REQUIRED.

  • (a) Difficulty baseline. The generated task is assigned to the appropriate level by comparison with approved reference minigames, without relying on the environment name or the creator’s description.

  • (b) Difficulty variations. Levels 1 to 5 increase in what the agent must perceive, decide, remember, or control. Adding repetitive steps is not sufficient.

  • (c) Interaction. For the same seed and difficulty, simplified and full interaction use the same minigame, information, goal, timing, physics, tolerances, and success condition. Full interaction requires the intended mouse or keyboard operations on the minigame.

  • (d) Real time. In live execution, the environment continues while the model produces an action. In paused execution, it remains frozen until the action arrives, and the action then executes with the intended behavior. The real-time configurations satisfy the definition in Appendix § 15, and no mechanism lets the agent avoid acting in real time, such as a brake that halts moving parts or repeating motion that permits a precomputed solution.

  • (e) End-to-end behavior. All configurations load, remain playable, produce the correct grading result, export the result, handle failure, and allow another attempt.

  • (f) Evidence. The audit agent ignores comments and written claims that the running environment does not support. Screenshots, recordings, frame sequences, exported results, and browser behavior serve as evidence, and missing visual evidence is a severe issue.

The audit agent also checks that the creator changed only files within its permitted scope. The creator revises the implementation in response to the report, and a new audit agent repeats the audit.

Difficulty Levels

The five difficulty levels, from very easy (level 1) to very hard (level 5), are defined in the same way for every minigame in GYM-ANYTHING-RT. A level describes the exact configuration the agent receives, and the original configuration of a minigame can belong to any level. A difficulty-review agent assigns levels by comparison with approved reference minigames, including Isometric Voxel Extraction Mine (level 1), Cursor-Controlled Constellation Hunt (level 2), Gyroscopic Tilt Board (level 3), Blind Dice Courier (level 4), and Exact-Change Candy Cascade (level 5).

To construct the other levels, the creator changes parameters that alter the problem the agent must solve. Depending on the minigame, these include the information available to the agent, visual ambiguity, the number of variables handled together, dependencies between decisions, planning depth, motion speed and timing windows, action precision, and recovery options. More steps, rounds, or waiting do not by themselves make a configuration harder. For every pair of adjacent levels, the creator must state what changes in the agent’s decision or control problem. Table 5 shows the five levels of Rotating On-Screen Keyboard.

LevelCode lengthKeysKeyboard motion
1312 large keysGentle, begins after the second character
2420 keysBegins after the first character
3526 keysFaster, begins after the first character
4532 keysFaster, changes direction more often
5832 smaller keysFast, sharply irregular, starts immediately

Table 5. Difficulty levels of Rotating On-Screen Keyboard. The keyboard rotates in three dimensions with irregularly timed motion that never repeats within an attempt.

Interaction Modes

Section 2.4 defines the two interaction modes. For a fixed seed and difficulty, both modes generate the same world, information, goal, and action effects. The grader rejects actions submitted through the controls of the other mode. Table 6 shows examples.

MinigameSimplifiedFull
Rotating On-Screen KeyboardType the code on the physical keyboard.Click each on-screen key while it moves.
Clockwork Clutch SafeClick a labeled release or engage button beneath each seal.Drag each seal’s clutch lever between its coupled and free positions.
Parallel GrillmasterSelect a food, then click buttons to start cooking and move it to the tray.Drag each food from the order to the grill and then to the serving tray.
LIDAR BlacksiteHold labeled movement and turning buttons, and use buttons to scan, pick up, and verify.Hold movement keys, drag the view to turn, click the scene to scan or pick up the beacon, and walk through the exit gate.

Table 6. Examples of simplified and full interaction.

Real-Time Revisions

We revised minigames whose earlier versions did not require real-time interaction or allowed an agent to avoid it. For 64 minigames, a creator agent developed a new real-time minigame from the same source, as described in Section 2.2. In 35 of these, the conversion removed the need for exploration, and a further revision restored it while keeping the goal, mechanic, and real-time execution. We also revised 23 minigames that already had real-time configurations, for instance to remove a brake that halts moving parts or to replace repeating motion that permits a precomputed solution. The remaining 13 minigames keep their original implementation, and GYM-ANYTHING-RT uses the revised versions of all others.

Non-Real-Time Counterparts

Section 2.3 describes how we construct a non-real-time counterpart for each real-time minigame. The counterpart keeps the generator, goal, controls, difficulty parameters, available information, and grader of its real-time minigame, and changes only how the environment evolves over time. For example, in Rotating On-Screen Keyboard, the keyboard keeps its initial layout while the agent decides. In Clockwork Clutch Safe, each drive segment holds until the agent operates a clutch, and in Parallel Grillmaster, each food settles at its golden point once cooking starts. An independent audit agent compares the two versions across all ten configurations and returns any difference other than timing to the creator for repair. For the 35 minigames revised to restore exploration, we rebuilt the counterparts from the revised versions.

Minigame List

Table 7 lists the 100 real-time minigames in GYM-ANYTHING-RT and the capabilities that each minigame and its non-real-time counterpart require.

Capability Annotation Criteria

GYM-ANYTHING-RT studies six capabilities. Review agents annotate the four capabilities below. The remaining two, interaction and real-time interaction, are varied through the interaction mode and the real-time setting. Appendix § 15 defines when a configuration requires real-time interaction.

The authors define the following criteria for capability annotation. Review agents apply them to the requirements of a solution using screenshots and visible mouse or keyboard controls.

Visual understanding. We classify visual understanding as 2D or 3D according to the spatial understanding required to solve the task. Decorative perspective alone is insufficient for a 3D label.

Temporal reasoning. A configuration requires this capability when every general screenshot-only solution must interpret change, motion, order, or duration, remember relevant earlier state that is no longer visible, or control when and for how long an action remains active. The temporal relationship must affect action choice or execution. A simple visible sequence, idle animation, or automatic transition to a static result does not establish this requirement.

Reasoning and planning. Solving requires inferring constraints or choosing actions whose consequences affect later decisions. The annotation identifies the relevant inference or action dependency. Basic visual recognition alone does not establish this requirement.

Exploration and interface understanding. The agent must interact to reveal relevant information or learn how the interface behaves before solving the task. Routine feedback, an automatic transition that supplies the next required information, or following explicit controls does not establish this requirement.

Review agents inspect the implementation and recorded behavior for each difficulty and interaction mode. Each annotation is supported by code or behavioral evidence.

Real-Time Evaluation Protocol

The shared runtime controls the execution schedule, and the task clock begins when the environment starts. In live execution, an observation request returns a current frame while the task clock continues. Paused execution advances the task through an observation window and returns a chronological sequence of frames. The clock then stops on the final frame until the action executes and resumes for the next observation window.

The observation window and number of frames are set per minigame and are the same for its non-real-time counterpart. The window ranges from 80 to 800 ms with three to six frames.

Action and Screenshot Latency

Direct connection protocol. Our action and screenshot path builds on a fast input and output path that we previously developed for a single agent controlling a single sandbox. That design assumes a one-to-one pairing between agent and sandbox. Gym-Anything {(citep}(Aggarwal et al., 2026)}, in contrast, runs many sandboxes behind a central server that launches them and forwards requests to them, and forwarding every action and screenshot through this server adds latency. We therefore separate the two roles. The central server still allocates sandboxes and manages their lifecycle, but actions and screenshots bypass it. The agent’s client runs on the same machine as its sandbox and keeps a persistent direct HTTP connection to an input service inside the sandbox, which executes each action and returns its acknowledgement over this connection. Screenshots are captured natively inside the

sandbox as full-resolution JPEG images, encoded once, and returned without resizing or re-encoding. A single request can execute an action and return the subsequent screenshot.

Measurement. We measure latency from immediately before the HTTP request until its complete response, including the image, has arrived. Model inference and subsequent image loading and decoding are outside this measurement.

In isolated timing tests with one client, we measured 150 sequential requests per condition after five warm-up requests. Median latencies were 39.2 ms for Rotating On-Screen Keyboard, 42.5 ms for an idle LIDAR Blacksite scene, and 46.8 ms while issuing LIDAR scan clicks. The corresponding 95th percentiles were 52.1, 55.3, and 70.7 ms.

Evaluation Details

This appendix details the evaluation protocol summarized in § 3.2.

Models and Harnesses

We evaluate GPT-6 Astra with low and xHigh reasoning effort and GPT-6 Luna with xHigh reasoning effort through the Codex CLI. Each GPT model receives a single computer tool. We disable the shell, code execution, web search, browser, and built-in computer-use features. Claude Opus 5.5 with xHigh effort runs through Claude Code with all built-in tools disabled, and receives the same computer tool through an MCP server. Qwen3.5-9B and Qwen3.8-27B are served through the Tinker API with temperature 1.0, top-p 0.95, top-k 20, and at most 2,048 output tokens per response. They use their native computer-use function format with coordinates on a 1000 × 1000 grid, and their context contains the last 100 turns with at most 20 screenshots.

Action Space and Prompt

For the GPT models, each call to the computer tool contains an ordered list of mouse and keyboard actions and returns a screenshot after the actions complete. The actions include moving, clicking, dragging, scrolling, typing, pressing keys, holding and releasing keys and mouse buttons, and waiting. The model instructions and the text after each task description both instruct all models to use only the visible controls of the task webpage, without writing code or inspecting the page. Figure 10 shows the complete prompt for Rotating On-Screen Keyboard in the live setting. In the paused setting, the paragraph following Temporal mode is replaced by the text in Figure 11.

Prompt for Rotating On-Screen Keyboard in the live setting

Figure 10. Prompt for Rotating On-Screen Keyboard in the live setting.

Timing instructions in the paused setting
The task clock is paused while you reason and while inputs are delivered.
Each screenshot request or step response advances one configured observation
window, returning its frames in chronological order. The task then freezes on
the final frame, and the next input acts on that state. A wait sleeps in wall
time but does not advance the paused task clock. To hold a key or mouse button
while the task advances, send down in one request and up in a later request.
Additional screenshot requests advance additional windows while the input
stays held. Down, wait, up in a single batch does NOT advance paused task time
between down and up.

Figure 11. Timing instructions in the paused setting.

Budgets and Outcomes

Each attempt allows at most 100 steps. A step is one request that contains one or more actions in order. Requests that only return a screenshot do not consume steps. Each attempt has a time limit of six hours, except that attempts of Astra (xHigh) on the real-time tasks at 1×, 4×, and 16× slowdown and in the paused setting have a limit of one hour. Attempts that reach the time limit count as failures. We exclude and rerun attempts that end with a provider or infrastructure error. The task verifier determines whether each attempt succeeds.

Time and Cost

Time is the wall-clock duration of an attempt from the moment the minigame’s clock starts to the end of its episode. It excludes the time spent setting up the environment. Cost is the API-equivalent cost of the recorded token usage,

cost⁡=(ninput−ncached)pinput+ncachedpcached+noutputpoutput106,\operatorname{cost} = \frac{(n_{\mathrm{input}} - n_{\mathrm{cached}})p_{\mathrm{input}} + n_{\mathrm{cached}}p_{\mathrm{cached}} + n_{\mathrm{output}}p_{\mathrm{output}}}{10^{6}},

where reasoning tokens are counted once as output tokens. Table 8 lists the prices. For GPT-6 Astra and Luna, we use OpenAI API prices,2 and requests with more than 272K input tokens are charged

ModelInputCached inputOutput
Qwen3.5-9B0.17–0.25
Qwen3.8-27B0.200.092.50
GPT-6 Luna0.100.010.50
GPT-6 Astra10.001.0050.00

Table 8. Prices used to compute cost, in USD per million tokens.

twice the input and cached-input prices and 1.5 times the output price. For the Qwen models, we use OpenRouter prices as of October 2026.3 OpenRouter lists no discounted price for cached input tokens of Qwen3.5-9B. We therefore charge its cached input tokens at the regular input price. For Claude Opus 5.5, we use the cost that Claude Code reports for each session, which applies Anthropic API prices to the recorded token usage.

Coding Protocol

In the coding experiment, the model runs in the Codex CLI with its standard tools inside an isolated sandbox. It receives an HTTP endpoint that accepts the same actions and returns screenshots, together with a short Python client. The model can write programs that process screenshots and issue actions through this endpoint. Access to the task machine, page source, and browser developer tools remains prohibited.

Interface Rulebooks

For the experiment in § 4.8, each rulebook describes a minigame’s controls, action effects, fixed world rules, feedback, submission, and recovery. It excludes generated answers, target coordinates, hidden state, and seed-specific layouts or solutions. The rulebook is appended to the task prompt, and all other settings remain unchanged.

Human Baseline

One of the authors played 20 tasks randomly sampled from the 100 real-time tasks at difficulty 5 with full interaction in the live setting, seeing each minigame for the first time. Each task used the same seed that the models received. The author solved 9 of the 20 tasks (45.0%), with an average time of 132.7 seconds, or 2.2 minutes, per task.

Examples of Model-Written Controllers

We examine saved trajectories from Astra’s coding evaluation at difficulty 5 with full interaction in the live setting. In Rotating On-Screen Keyboard, the model uses perspective transforms and image templates to locate requested keys, converts their positions to screen coordinates, and clicks them. In Wrong Number, it fits curves to waveform pixels from screenshots, estimates the phase error, and adjusts a slider. In Clockwork Clutch Safe, it tracks the marker on each gear, estimates the marker’s

@references.viewer rax? angular speed from recent screenshots, and releases each clutch when the marker is predicted to reach its target angle. All three attempts pass the task verifier.

Image-based control also appears in failed attempts. In Gyroscopic Tilt Board, Astra detects the ball and direction indicator, estimates velocity, and moves the tilt control along a sequence of waypoints, but the attempt fails. These examples show how scripts can process observations and issue actions between model responses.

Real-Time Definition and Review Prompts

Definition

The real-time label is assigned to a fixed difficulty and interaction configuration. Let HtH_t denote the visible interaction history through task time tt, ItI_t the agent’s own prior actions and their task times, and WtwW_t^w the visible window from t−wt-w to tt. Let uu denote the terminal task outcome, with less task-active continuation time after delivery of the next action used only to break ties between equal outcomes. Expectations use visible task information and average over future randomness. Private generator or page state is unavailable to the policy.

A grouped control mode DD contains every occurrence, across seeds, rounds, and attempts, of the same outcome-affecting control mode. Let CΔ(D)C_\Delta(D) contain the histories in DD from which success remains possible after a delay Δ\Delta with no new action. Only histories that become unsalvageable after this delay are excluded. Let AΔ∗(Ht)A^*_\Delta(H_t) be the set of actions maximizing expected uu when delivered at t+Δt+\Delta and followed by optimal continuation. Let HtτH_t^\tau be the history after τ\tau with no new action, and Q(Ht,a)Q(H_t,a) the expected uu when aa is the next action and is followed by optimal continuation.

A configuration satisfies the real-time definition if and only if there exist a grouped control mode DD, a delay Δ>0\Delta> 0, and a bounded visual window ww such that CΔ(D)C_\Delta(D) is nonempty and all three conditions hold:

(i) No policy using only the agent’s own action history and task times selects an optimal delayed action for every viable history:

∄q:It↦asuch that∀Ht∈CΔ(D), q(It)∈AΔ∗(Ht).\not\exists q : I_t \mapsto a \quad\text{such that} \quad\forall H_t \in C_\Delta(D),\ q(I_t) \in A^*_\Delta(H_t).

(ii) A policy using that history and the recent visual window can select an optimal delayed action for every viable history:

∃g:(It,Wtw)↦asuch that∀Ht∈CΔ(D), g(It,Wtw)∈AΔ∗(Ht).\exists g : (I_t, W_t^w) \mapsto a \quad\text{such that} \quad\forall H_t \in C_\Delta(D),\ g(I_t, W_t^w) \in A^*_\Delta(H_t).

(iii) For some Ht∈CΔ(D)H_t \in C_\Delta(D), action aa, and 0<τ<Δ0 < \tau< \Delta, the environment evolves autonomously to HtτH_t^\tau, success remains possible, and the action loses value or advantage over no action:

Q(Htτ,a)<Q(Ht,a)orQ(H_t^\tau,a) < Q(H_t,a) \quad\text{or}
Q(Htτ,a)−Q(Htτ,no-op)<Q(Ht,a)−Q(Ht,no-op).Q(H_t^\tau,a) - Q(H_t^\tau,\mathrm{no\text{-}op}) < Q(H_t,a) - Q(H_t,\mathrm{no\text{-}op}).

An unavailable or ineffective action has the value of no action. The strict loss in (iii) must arise from non-clock task-state evolution. Less remaining deadline, later completion alone, a fixed wait that enables an action, and a fixed animation whose result remains available do not satisfy this condition.

All three conditions must be witnessed by the same visible autonomous process and the same DD, Δ\Delta, and ww. The evidence within WtwW_t^w must support prediction of the process’s effect at t+Δt+\Delta and determine the delayed-optimal action in (i) and (ii). The evidence can occur across frames or in an accumulated trace, chart, or trajectory within one screenshot. Unrelated animation and future randomness independent of the visible window cannot supply this evidence.

The witness delay Δ\Delta is chosen on the time scale of the task behavior. It begins within an active control opportunity and may extend beyond the point where that opportunity degrades or closes. It is independent of the current model provider’s latency. The configured observation window and frame count bound ww. A stationary policy over the visibly evidenced dynamic state is allowed.

Pre-run exception. For the configuration-level label, a configuration is classified as not real-time if at least 50% of its generated instances admit a successful pre-run solution. After preparation, such a solution starts the autonomous outcome phase and sends no further outcome-affecting actions until that phase ends. Final submission or certification is administrative and is excluded from this action count. Reviewers establish this threshold from source evidence or a reproducible seed probe. If the threshold is unresolved and could change a positive mathematical classification, the reviewer returns unresolved. Failure of any mathematical condition already establishes a negative classification.

Review Prompts

The following are the prompt templates used to review real-time labels. The first template assigns one configuration to a reviewer. The second assigns the configurations of one minigame to a reviewer, with a separate judgment for each configuration. Both use the same definition and pre-run exception.

One Configuration per Review

# Independent real-time classification prompt

Prompt version: `selected-100-pending-v1`

Reuses the historical real-time pilot prompt, adding the settled pre-run threshold and
explicit report destination.

Frozen definition source: `docs/real-time.md`

Frozen definition SHA-256: `885cd87d0c59d5017b673a2b8696f251e31eedf0c8e388e32d47101625bcaf09`

Use the configuration and SOURCE_ROOT, CASE_INDEX, and OUTPUT_PATH values supplied by the
launcher. Read source only from that frozen SOURCE_ROOT, not the current checkout. Paths below
are relative to SOURCE_ROOT.

---

You are an independent reviewer classifying exactly one Gym-Anything-RT configuration.

## Configuration

- Public environment name: `{public_environment_name}`
- Environment ID: `{environment_id}`
- Difficulty: `{difficulty}`
- Interaction mode: `{interaction_mode}`

Decide whether this exact configuration is real-time under the frozen mathematical definition
below.

Do not use an existing classification, capability annotation, audit result, result matrix, or
another reviewer's answer. Do not infer the answer from the environment name or task
description.

Read `AGENTS.md` and `docs/real-time.md` completely. Then read the complete relevant
implementation:

1. the exact difficulty and interaction profile in `controls.json`;
2. the generator;
3. the visible browser implementation and relevant CSS;
4. the grader and verifier; and
5. the solver, when present.

Study a normal screenshot-only solution using visible task controls. Private state and
scripted solvers are implementation evidence, not information available to the policy.

## Frozen mathematical definition

```text
H_t                 visible interaction history through task time t
I_t                 the agent's own prior actions and task times in H_t
W_t^τ               the visible window from t−τ through t
u                   terminal task outcome, with less task-active continuation
                    time after delivery of the next action used only to break
                    ties between equal outcomes
D                   one grouped control move
C_Delta(D)          histories in D from which success remains possible after Delta
                    of no new action
A*_Delta(H_t)       actions maximising expected u when delivered at t+Delta,
                    followed by optimal continuation
H_t^tau             history tau later if the agent supplies no new action
Q(H_t, a)           expected u if a is the next action, followed by optimal continuation
Expectations use only visible task information and average over future randomness; private
generator or page state is not available to the policy. For a fixed difficulty and interaction
configuration, `D` contains every occurrence across seeds, rounds, and attempts of the same
outcome-affecting control mode. It is not a hand-picked early interval. `C_Delta(D)` removes
only histories whose outcome is already unsalvageable after the delay.

A fixed difficulty and interaction configuration of `E` is real-time iff there are a grouped
control mode `D`, a delay `Delta > 0`, and a bounded visual window `w` such that `C_Delta(D)`
is nonempty and:

    (i)   there is no q : I_t -> a
          with q(I_t) in A*_Delta(H_t) for every H_t in C_Delta(D)

    (ii)  there is a g : (I_t, W_t^w) -> a
          with g(I_t, W_t^w) in A*_Delta(H_t)
          for every H_t in C_Delta(D)

    (iii) for some H_t in C_Delta(D), action a, and 0 < tau < Delta,
          the environment changes autonomously from H_t to H_t^tau,
          success remains possible from H_t^tau, and

          Q(H_t^tau, a) < Q(H_t, a)

          or

          Q(H_t^tau, a) - Q(H_t^tau, no-op)
            < Q(H_t, a) - Q(H_t, no-op).

An unavailable or ineffective action has the value of no-op. The strict loss in clause (iii) must be caused by non-clock task-state evolution, not merely by less remaining deadline or later completion. An action becoming enabled or more effective after a fixed wait does not satisfy the clause. A fixed animation that later enables controls or permanently archives its result is not an expiring opportunity.

The three clauses must be witnessed by the same phenomenon. The autonomous evolution in clause (iii) must be visibly evidenced within W_t^w, and that evidence must be what makes the delayed-optimal action depend on observation in clauses (i) and (ii). The window must support prediction of the evolution's effect at t+Delta; unrelated idle animation or future randomness independent of the window cannot supply clause (iii).

The evidence may appear across consecutive frames or as an accumulated visible trace, chart, or trajectory in one screenshot. A single screenshot is allowed when it visibly contains the relevant temporal evidence. This does not admit a static puzzle next to an unrelated animation.

Delta is a reported witness horizon on the time scale of the task behavior, not the current provider's response latency. It must begin inside a nontrivial active control opportunity and may extend across the point at which that opportunity gets worse or closes. The configured observation_window_ms and frames_per_observation bound w; they do not force every environment to need every captured frame.

The conjunction has these required consequences:

  • Clause (i) rules out a fixed schedule or a policy driven only by the agent's own action clock.

  • Clause (ii) requires a bounded recent visual window to be sufficient. This excludes unobservable randomness and information that must be recalled only from outside the window.

  • Clause (iii) requires a useful action opportunity to get worse inside the delay because the task state changes. This excludes fixed waits, minimum-duration holds solvable from the agent's own clock, and information that remains actionable indefinitely.

  • A stationary visual policy is allowed. It can map the currently evidenced changing state to an action without using absolute task time.

  • All three clauses must use one common D, Delta, w, and visible autonomous process.

Pre-run threshold

For the configuration-level catalog label, a configuration is Not real-time when at least 50% of its generated instances admit a successful pre-run solution: after preparation, the solution starts the autonomous outcome phase and sends no more outcome-affecting actions until that phase ends. Final submission or certification is administrative and does not count as an action during the run.

Record whether the threshold is met, with source-structural evidence or an actual reproducible seed probe. A failed existing solver does not prove that no pre-run solution exists. Do not invent a percentage. If the threshold cannot be determined and it could change a mathematical Yes into No, return unresolved. A configuration that already fails the mathematical definition is No regardless of an unresolved pre-run rate.

Review procedure

  1. State the controls available in this exact difficulty and interaction mode.

  2. Describe every period in which the task changes without a new agent action.

  3. State whether an outcome-affecting control remains available during each such period.

  4. Enumerate the legitimate grouped control modes D.

  5. Try to construct an observation-free policy q using only I_t.

  6. Try to construct a bounded-window policy g using only I_t and W_t^w.

  7. Check whether the same visible autonomous process causes an available action to become unavailable or strictly less effective within some Delta.

  8. For yes, give one concrete common witness: D, Delta, w, the visible process, why q fails, why g succeeds, and the expiring or degrading action.

  9. For no, identify the failed clause and explain why no legitimate choice of D, Delta, and w repairs it, or establish that the pre-run threshold overrides a mathematical Yes.

  10. Return unresolved instead of guessing when the complete source does not determine the answer.

Return exactly one JSON object:

{
  "case_index": "{case_index}",
  "public_environment_name": "{public_environment_name}",
  "environment_id": "{environment_id}",
  "difficulty": "{difficulty}",
  "interaction_mode": "{interaction_mode}",
  "label": "yes | no | unresolved",
  "controls_available": [],
  "autonomous_processes": [],
  "grouped_control_modes": [],
  "delta_ms": null,
  "window_ms": null,
  "clause_i": {"holds": true, "reason": ""},
  "clause_ii": {"holds": true, "reason": ""},
  "clause_iii": {"holds": true, "reason": ""},
  "common_visible_process": "",
  "pre_run": {
    "threshold": 0.5,
    "at_least_half": null,
    "rate": null,
    "basis": "source_structural | seeded_probe | unresolved | not_applicable",
    "reason": "",
    "seed_probe": null
  },
  "files_read_completely": [],
  "source_evidence": [
    {"path": "", "lines": "", "claim": ""}
  ],
  "uncertainties": []
}

Write the JSON object, with numeric case_index and difficulty, to the assigned OUTPUT_PATH using apply_patch. This one report is the only permitted persistent output. Do not edit puzzle sources, definitions, historical audits, or another report. Do not inspect other first-pass reviews, existing classification/capability files, evaluation results, or previous conversations. The audit manifest is parent-only; the launcher gives you your exact assignment.

You may run read-only source/math probes when needed, with PYTHONDONTWRITEBYTECODE=1 and without modifying source. For a seed probe, include seeds, numerator/denominator, and reproducible probe code in pre_run.seed_probe. No browser or desktop access, external websites/services, or seeded agents, or empirical CUA evaluation. A source-review report may be unresolved; do not guess to finish.

files_read_completely and source_evidence must use repository-relative paths. Read each complete implementation file needed by the assigned mechanic, including shared helpers actually used. If a shared file is long, read it in successive ranges to EOF; keyword searches do not substitute for full reads. Record concrete code evidence for the exact difficulty and interaction. Do not make changes to obtain a preferred label.

Configurations Grouped by Minigame

Per-game real-time classification prompt

Prompt version: selected-100-per-game-v2

Preserves the first-phase mathematical definition, pre-run threshold, and evidence requirements. Changes only the assignment unit: one reviewer reads one game once and judges its requested difficulty/interaction configurations together.

Frozen definition source: docs/real-time.md

Frozen definition SHA-256: 885cd87d0c59d5017b673a2b8696f251e31eedf0c8e388e32d47101625bcaf09

Use the game assignment and SOURCE_ROOT, CASE_INDEX, CONFIGURATIONS, and OUTPUT_PATH values supplied by the launcher. Read source only from that frozen SOURCE_ROOT, not the current checkout. Paths below are relative to SOURCE_ROOT.


You are an independent reviewer classifying one Gym-Anything-RT game across its difficulty and interaction configurations in a single review.

Assignment

The launcher supplies the exact public environment name, environment ID, game CASE_INDEX, and CONFIGURATIONS list. Each configuration has its own numeric case_index, difficulty, and interaction_mode. Normally the list contains all ten configurations: five difficulties crossed with Full and Simplified. A partially completed game may supply only its unfinished configurations. Read all ten active profiles together to understand their differences, but return judgments only for the supplied CONFIGURATIONS. Do not inspect or infer the already completed judgments.

Read each shared implementation file once, then apply the definition separately to every requested configuration. Explicitly check mode-specific inputs and level-dependent behavior; do not copy one answer across the game without checking those differences. These are multiple configuration judgments from one fresh game-level reviewer, not independent reviewer contexts per configuration.

Do not use an existing classification, capability annotation, audit result, result matrix, or another reviewer's answer. Do not infer the answer from the environment name or task description.

Read AGENTS.md and docs/real-time.md completely. Then read the complete relevant implementation:

  1. the exact difficulty and interaction profile in controls.json;

  2. the generator;

  3. the visible browser implementation and relevant CSS;

  4. the grader and verifier; and

  5. the solver, when present.

Study a normal screenshot-only solution using visible task controls. Private state and scripted solvers are implementation evidence, not information available to the policy.

Frozen mathematical definition

H_t             visible interaction history through task time t
I_t             the agent's own prior actions and task times in H_t
W_t^w           the visible window from t−w through t
u               terminal task outcome, with less task-active continuation
                time after delivery of the next action used only to break
                ties between equal outcomes
D               one grouped control mode
C_Delta(D)      histories in D from which success remains possible after Delta
                of no new action
A*_Delta(H_t)   actions maximising expected u when delivered at t+Delta,
                followed by optimal continuation
H_t^tau         history tau later if the agent supplies no new action
Q(H_t, a)       expected u if a is the next action, followed by optimal continuation

Expectations use only visible task information and average over future randomness; private generator or page state is not available to the policy. For a fixed difficulty and interaction configuration, D contains every occurrence across seeds, rounds, and attempts of the same outcome-affecting control mode. It is not a hand-picked early interval. C_Delta(D) removes only histories whose outcome is already unsalvageable after the delay.

A fixed difficulty and interaction configuration of E is real-time iff there are a grouped control mode D, a delay Delta > 0, and a bounded visual window w such that C_Delta(D) is nonempty and:

(i)     there is no q : I_t -> a
        with q(I_t) in A*_Delta(H_t) for every H_t in C_Delta(D)

(ii)    there is a q : (I_t, W_t^w) -> a
        with q(I_t, W_t^w) in A*_Delta(H_t)
        for every H_t in C_Delta(D)

(iii)   for some H_t in C_Delta(D), action a, and 0 < tau < Delta,
        the environment changes autonomously from H_t to H_t^tau,

success remains possible from HtτH_t^\tau, and

Q(Htτ,a)<Q(Ht,a)Q(H_t^\tau, a) < Q(H_t, a)

or

Q(Htτ,a)−Q(Htτ,no-op)<Q(Ht,a)−Q(Ht,no-op).Q(H_t^\tau, a) - Q(H_t^\tau, \text{no-op}) < Q(H_t, a) - Q(H_t, \text{no-op}).

An unavailable or ineffective action has the value of no-op. The strict loss in clause (iii) must be caused by non-clock task-state evolution, not merely by less remaining deadline or later completion. An action becoming enabled or more effective after a fixed wait does not satisfy the clause. A fixed animation that later enables controls or permanently archives its result is not an expiring opportunity.

The three clauses must be witnessed by the same phenomenon. The autonomous evolution in clause (iii) must be visibly evidenced within W_t^w, and that evidence must be what makes the delayed-optimal action depend on observation in clauses (i) and (ii). The window must support prediction of the evolution's effect at t+Δt+\Delta; unrelated idle animation or future randomness independent of the window cannot supply clause (iii).

The evidence may appear across consecutive frames or as an accumulated visible trace, chart, or trajectory in one screenshot. A single screenshot is allowed when it visibly contains the relevant temporal evidence. This does not admit a static puzzle next to an unrelated animation.

Delta is a reported witness horizon on the time scale of the task behavior, not the current provider's response latency. It must begin inside a nontrivial active control opportunity and may extend across the point at which that opportunity gets worse or closes. The configured observation_window_ms and frames_per_observation bound w; they do not force every environment to need every captured frame.

The conjunction has these required consequences:

  • Clause (i) rules out a fixed schedule or a policy driven only by the agent's own action clock.

  • Clause (ii) requires a bounded recent visual window to be sufficient. This excludes unobservable randomness and information that must be recalled only from outside the window.

  • Clause (iii) requires a useful action opportunity to get worse inside the delay because the task state changes. This excludes fixed waits, minimum-duration holds solvable from the agent's own clock, and information that remains actionable indefinitely.

  • A stationary visual policy is allowed. It can map the currently evidenced changing state to an action without using absolute task time.

  • All three clauses must use one common D, Delta, w, and visible autonomous process.

Pre-run threshold

For the configuration-level catalog label, a configuration is Not real-time when at least 50% of its generated instances admit a successful pre-run solution: after preparation, the solution starts the autonomous outcome phase and sends no more outcome-affecting actions until that phase ends. Final submission or certification is administrative and does not count as an action during the run.

Record whether the threshold is met, with source-structural evidence or an actual reproducible seed probe. A failed existing solver does not prove that no pre-run solution exists. Do not invent a percentage. If the threshold cannot be determined and it could change a mathematical Yes into No, return unresolved. A configuration that already fails the mathematical definition is No regardless of an unresolved pre-run rate.

Review procedure

  1. State the controls available in this exact difficulty and interaction mode.

  2. Describe every period in which the task changes without a new agent action.

  3. State whether an outcome-affecting control remains available during each such period.

  4. Enumerate the legitimate grouped control modes D.

  5. Try to construct an observation-free policy q using only I_t.

  6. Try to construct a bounded-window policy g using only I_t and W_t^w.

  7. Check whether the same visible autonomous process causes an available action to become unavailable or strictly less effective within some Delta.

  8. For Yes, give one concrete common witness: D, Delta, w, the visible process, why q fails, why g succeeds, and the expiring or degrading action.

  9. For No, identify the failed clause and explain why no legitimate choice of D, Delta, and w repairs it, or establish that the pre-run threshold overrides a mathematical Yes.

  10. Return unresolved instead of guessing when the complete source does not determine the answer.

Return exactly one game-level JSON object with numeric game_case_index, environment_id, public_environment_name, and configuration_reports. configuration_reports is an array containing exactly one object for each supplied configuration, with no duplicates or omitted or extra configurations. Each array member uses this schema (case_index is the configuration case_index, not the game case_index):

{
  "case_index": "{case_index}",
  "public_environment_name": "{public_environment_name}",
  "environment_id": "{environment_id}",
  "difficulty": "{difficulty}",
  "interaction_mode": "{interaction_mode}",
  "label": "yes | no | unresolved",
  "controls_available": [],
  "autonomous_processes": [],
  "grouped_control_modes": [],
  "delta_ms": null,
  "window_ms": null,
  "clause_i": {"holds": true, "reason": ""},
  "clause_ii": {"holds": true, "reason": ""},
  "clause_iii": {"holds": true, "reason": ""},
  "common_visible_process": "",
  "pre_run": {
    "threshold": 0.5,
    "at_least_half": null,
    "rate": null,
    "basis": "source_structural | seeded_probe | unresolved | not_applicable",
    "reason": "",
    "seed_probe": null
  },
  "files_read_completely": [],
  "source_evidence": [
    {"path": "", "lines": "", "claim": ""}
  ],
  "uncertainties": []
}

Write the single game-level JSON object to the assigned OUTPUT_PATH using apply_patch. Every configuration_reports member must contain numeric case_index and difficulty. The game-level object must identify the assigned game and contain the complete requested configuration set. This one game report is the only permitted persistent output. Do not edit puzzle sources, definitions, historical audits, or another report. Do not inspect other first-pass reviews, existing classification/capability files, evaluation results, or previous conversations. The audit manifest is parent-only; the launcher gives you your exact assignment.

You may run read-only source/math probes when needed, with PYTHONDONTWRITEBYTECODE=1 and without modifying source. For a seed probe, include seeds, numerator/denominator, and reproducible probe code in pre_run.seed_probe. No browser or desktop access, external websites/services, nested agents, or empirical CUA evaluation. A source-review report may be unresolved; do not guess to finish.

files_read_completely and source_evidence must use repository-relative paths. Read each complete implementation file needed by the assigned mechanic, including shared helpers actually used. If a shared file is long, read it in successive ranges to EOF; keyword searches do not substitute for full reads. Record concrete code evidence for the exact difficulty and interaction. Do not make changes to obtain a preferred label.

Before finishing, load your JSON and check the game identity, exact requested configuration coverage, numeric IDs, label/clause consistency, and all cited file paths and line bounds. Shared evidence may be reused where it applies, but every configuration must remain independently understandable. A structural check is not evidence that the substantive classification is correct. Do not read other reviewers' reports or the audit manifest.

Extended Related Work

This appendix expands on the related work discussion in § 5.

Benchmarks for interactive visual agents. Existing benchmarks for vision-language agents span web, desktop, mobile, game, and embodied environments. Web benchmarks such as WebArena and VisualWebArena provide self-hosted websites with realistic tasks [27] (Zhou et al. 2024), while desktop benchmarks such as OSWorld and Windows Agent Arena evaluate open-ended tasks across operating system applications [7] (Xie et al. 2024). OSWorld 2.0 further extends this setting to 108 long-horizon workflows, where the best agent completes only 20.6% of tasks (Yuan et al., 2026). On mobile, AndroidWorld provides tasks across 20 Android apps [40], and Programming with Pixels and Gym-Anything extend computer-use evaluation to software engineering and to 200 professional software applications [2, 3]. Game benchmarks evaluate the same models on complete games, ranging from classic video games (Zhang et al., 2026) and reinforcement learning environments such as NetHack [36] to platformers, puzzle games, and browser games [22, 35] (Zheng et al. 2025a; Ying et al. 2026). ARC-AGI-3 evaluates exploration and goal inference in turn-based abstract games, where humans solve all environments while frontier models solve less than 1% [4]. Embodied benchmarks similarly evaluate agents in simulated homes and physical environments [43, 46, 49] (Yang et al. 2025c). However, since each task in these benchmarks requires many capabilities at once, low performance does not identify which capability limits the agent.

Several works attempt to attribute performance to individual capabilities. SmartPlay maps each of its six games to a subset of nine agent capabilities (Wu et al., 2023), LVLM-Playground evaluates perception, question answering, rule following, and end-to-end play separately on the same board games [56], and Embodied Agent Interface decomposes embodied decision making into separate modules, arguing that final success rates cannot pinpoint which ability is missing [29]. VS-Bench similarly reports perception, strategic reasoning, and decision making separately (Xu et al., 2026). Other benchmarks vary conditions of the same task, such as difficulty tiers [15] (Zheng et al. 2025b), initial states (Zhao et al., 2025), or interface changes between actions [30]. However, these works either operate through text or turn-based interfaces, or vary a single condition without controlling the capabilities a task requires. In contrast, GYM-ANYTHING-RT annotates the capabilities required by each configuration, and varies difficulty, interaction interface, and real-time setting on the same minigame.

Capability-specific evaluation of VLMs. A large body of work evaluates individual capabilities of VLMs through question answering. For 3D spatial understanding, VSI-Bench, MindCube, 3DSR-Bench, and OmniSpatial evaluate spatial relations, viewpoint changes, and mental models of scenes [55, 33, 25] (Yang et al. 2025a), while BLINK shows that perception tasks that humans solve easily remain difficult for multimodal models [20]. For temporal reasoning, TempCompass, TOMATO, and Vinoground evaluate speed, direction, and event order in videos [31] (Shanguang et al. 2024; Zhang et al. 2024), with TOMATO showing that many questions in earlier video benchmarks can be answered from a single frame. For GUI grounding, SeeClick and ScreenSpot-Pro evaluate whether models can locate screen elements from instructions [11, 28]. Finally, PuzzleVQA, AlgoPuzzleVQA, and VisualPuzzles evaluate abstract and algorithmic reasoning over images [13, 21, 48], and MMMU and MMMU-Pro evaluate multidisciplinary reasoning over images and text (Yue et al., 2024; 2025). However, in all of these benchmarks the model answers a question about a static input, without having to act on its understanding. Closer to our work, IVISPAR evaluates the spatial reasoning of VLM agents on a sliding-tile puzzle presented in 3D, 2D, and text [34], and VisGym provides 17 interactive environments with configurable difficulty and input representation [57]. GYM-ANYTHING-RT complements these works by covering temporal reasoning, exploration and interface understanding, and real-time interaction, and by annotating the capabilities required by each configuration.

Real-time interaction. Acting in dynamic environments requires accounting for the delay between observing the environment and executing an action. In robotics, real-time action chunking and asynchronous inference execute previously planned actions while the next chunk is computed [6, 47], and VLASH conditions the policy on the state predicted at execution time [51]. Reinforcement learning has similarly studied decision making under action and observation delays [39, 18]. For computer-use agents, OSWorld-Human measures end-to-end latency and finds that agents take 2.7 to 4.3 times more steps than necessary [1]. Game benchmarks have taken different approaches to latency. GameWorld pauses the game during inference and reports a separate real-time study with two models [35], VideoGameBench runs games in real time and additionally provides a Lite version that pauses the emulator while the agent is thinking (Zhang et al., 2026), and Orak compares paused and real-time play on StarCraft II [38]. Real-Time Reasoning Gym advances text-based games after a fixed number of generated tokens, comparing reactive and planning agents under increasing time pressure [58]. Our work complements these efforts by evaluating the same minigame and seed in the live, slowed, and paused settings, and by pairing each real-time minigame with a non-real-time counterpart.

Automated environment construction. A growing body of work uses coding agents and LLMs to construct environments for agents. Gym-Anything uses a creation agent and an independent audit agent to convert 200 software applications into computer-use environments [3]. CUA-Gym separates an agent that constructs environment states from one that writes reward functions [53], and GUI-GENESIS converts real applications into lightweight web environments with executable rewards [8]. AutoEnv generates 36 text-based environments with 358 levels (Zhang et al., 2025), while AgentGen and OMNI-EPIC generate environments together with tasks of varying difficulty [23, 19]. Similar pipelines construct software engineering tasks from code repositories [37] (Yang et al. 2025b) and 3D scenes for embodied agents (Yang et al., 2024). Procedural generation further provides task instances with adjustable complexity, including Procgen games [14], MiniHack and BabyAI [42, 12], and verifiable reasoning problems in Reasoning Gym and Enigmata [50, 9]. However, these approaches construct software, text, or game environments, and do not produce interactive visual minigames whose difficulty and interaction interface can be varied while preserving the underlying task. We extend the creator-auditor approach to such minigames, where each minigame has a non-real-time counterpart, five difficulty levels, two interaction modes, and an oracle solution.

CAPTCHAs and visual puzzles. Many of the seed minigames in GYM-ANYTHING-RT are derived from CAPTCHAs and puzzle games. CAPTCHA research has studied whether automated solvers can pass challenges designed to distinguish humans from bots. Recent works solve reasoning and visual CAPTCHAs with LLMs and VLMs [17, 52], analyze which types of challenges remain difficult for these models [54] (Wu et al. 2025), and train GUI agents on interactive CAPTCHAs [10]. Open CaptchaWorld evaluates web agents on 20 types of interactive CAPTCHAs, where the best agent achieves 40.0% compared to 93.3% for humans [32]. Beyond CAPTCHAs, KORGym, DiG-bench, and VGRP-Bench evaluate multi-turn reasoning in text, game, and grid puzzles [45, 5, 41]. However, these works primarily target security or reasoning in isolation. In contrast, GYM-ANYTHING-RT uses such challenges as seed minigames, and adds controlled capability requirements, difficulty levels, interaction interfaces, and real-time settings.

References

  1. [1]Reyna Abhyankar, Qi Qi, and Yiying Zhang. OSWorld-Human: Benchmarking the Efficiency of Computer-Use Agents, 2026. URL https://arxiv.org/abs/2506.16042.
  2. [2]Pranjal Aggarwal and Sean Welleck. Programming with Pixels: Can Computer-Use Agents do Software Engineering?, 2025. URL https://arxiv.org/abs/2502.18525.
  3. [3]Pranjal Aggarwal, Graham Neubig, and Sean Welleck. Gym-Anything: Turn any Software into an Agent Environment, 2026. URL https://arxiv.org/abs/2604.06126.
  4. [4]ARC Prize Foundation. ARC-AGI-3: A New Challenge for Frontier Agentic Intelligence, 2026.arxiv.org/abs/2603.24621
  5. [5]Ruairidh M. Battleday, Kai Sandbrink, Jimi Cullen-Drohan, Zihan Yan, Timothy Muller, Clare Maguire, Ales K. Dubcek, Fraser Greenlee-Scott, Sukrit Sumant, Tri Dao, Jürgen Schmidhuber, Michal Valko, Joshua Tenenbaum, Thomas L. Griffiths, Zeb Kurth-Nelson, and James C. R. Whittington. DiG-bench: Discovery in Games, 2026. URL https://arxiv.org/abs/2608.12593.
  6. [6]Kevin Black, Manuel Y. Galliker, and Sergey Levine. Real-Time Execution of Action Chunking Flow Policies, 2025. URL https://arxiv.org/abs/2506.07339.
  7. [7]Rogerio Bonatti, Dan Zhao, Francesco Bonacci, Dillon Dupont, Sara Abdali, Yinheng Li, Yadong Lu, Justin Wagle, Kazuhito Koishida, Arthur Bucker, Lawrence Jang, and Zack Hui. Windows Agent Arena: Evaluating Multi-Modal OS Agents at Scale, 2024.arxiv.org/abs/2409.08264
  8. [8]Yuan Cao, Dezhi Ran, Mengzhou Wu, Yuzhe Guo, Xin Chen, Ang Li, Gang Cao, Gong Zhi, Hao Yu, Linyi Li, Wei Yang, and Tao Xie. GUI-GENESIS: Automated Synthesis of Efficient Environments with Verifiable Rewards for GUI Agent Post-Training, 2026. URL https://arxiv.org/abs/2602.14093.
  9. [9]Jiangjie Chen, Qianyu He, Siyu Yuan, Aili Chen, Zhicheng Cai, Weinan Dai, Hongli Yu, Qiying Yu, Xuefeng Li, Jiaze Chen, Hao Zhou, and Mingxuan Wang. Enigmata: Scaling Logical Reasoning in Large Language Models with Synthetic Verifiable Puzzles, 2025. URL https://arxiv.org/abs/2505.19914.
  10. [10]Yuxi Chen, Haoyu Zhai, Chenkai Wang, Rui Yang, Lingming Zhang, Gang Wang, and Huan Zhang. CAPTCHA Solving for Native GUI Agents: Automated Reasoning-Action Data Generation and Self-Corrective Training. In International Conference on Machine Learning (ICML), 2026. arXiv:2603.23559.arxiv.org/abs/2603.23559
  11. [11]Kanzhi Cheng, Qiushi Sun, Yougang Chu, Fangzhi Xu, Li YanTao, Jianbing Zhang, and Zhiyong Wu. SeeClick: Harnessing GUI Grounding for Advanced Visual GUI Agents. In Lun-Wei Ku, Andre Martins, and Vivek Srikumar (eds.), Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pp. 9313–9332, Bangkok, Thailand, August 2024. Association for Computational Linguistics. doi: 10.18653/v1/2024.acl-long.505. URL https://aclanthology.org/2024.acl-long.505/.DOI
  12. [12]Maxime Chevalier-Boisvert, Dzmitry Bandanau, Salem Lahlou, Lucas Willems, Chitwan Saharia, Thien Huu Nguyen, and Yoshua Bengio. BabyAI: A Platform to Study the Sample Efficiency of Grounded Language Learning. In International Conference on Learning Representations (ICLR), 2019.
  13. [13]Yew Ken Chia, Vernon Toh Yan Han, Deepanway Ghosal, Lidong Bing, and Soujanya Poria. PuzzleVQA: Diagnosing Multimodal Reasoning Challenges of Language Models with Abstract Visual Patterns, 2024. URL https://arxiv.org/abs/2403.13315.
  14. [14]Karl Cobbe, Christopher Hesse, Jacob Hilton, and John Schulman. Leveraging Procedural Generation to Benchmark Reinforcement Learning, 2019. URL https://arxiv.org/abs/1912.01588.
  15. [15]Mariya Davydo­va, Daniel Jeffries, Patrick Barker, Arturo Márquez Flores, and Sinéad Ryan. OSUniverse: Benchmark for Multimodal GUI-navigation AI Agents, 2025.
  16. [16]DeepSeek-AI. DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learning. arXiv preprint arXiv:2501.12948, 2025.DOI
  17. [17]Gelei Deng, Haoran Qu, Yi Liu, Jie Zhang, Tianwei Zhang, and Yang Liu. Oedipus: LLM-enhanced Reasoning CAPTCHA Solver, 2024. URL https://arxiv.org/abs/2405.07496.
  18. [18]Esther Derman, Gal Dalal, and Shie Mannor. Acting in Delayed Environments with Non-Stationary Markov Policies. In International Conference on Learning Representations (ICLR), 2021. arXiv:2101.11992.arxiv.org/abs/2101.11992
  19. [19]Maxence Faldor, Jenny Zhang, Antoine Cully, and Jeff Clune. OMNI-EPIC: Open-endedness via Models of human Notions of Interestingness with Environments Programmed in Code, 2024. URL https://arxiv.org/abs/2405.15568.
  20. [20]Xingyu Fu, Yushi Hu, Bangzheng Li, Yu Feng, Haoyu Wang, Xudong Lin, Dan Roth, Noah A. Smith, Wei-Chiu Ma, and Ranjay Krishna. BLINK: Multimodal Large Language Models Can See but Not Perceive, 2024. URL https://arxiv.org/abs/2404.12390.
  21. [21]Deepanway Ghosal, Vernon Toh Yan Han, Chia Yew Ken, and Soujanya Poria. Are Language Models Puzzle Prodigies? Algorithmic Puzzles Unveil Serious Challenges in Multimodal Reasoning, 2024. URL https://arxiv.org/abs/2403.03864.
  22. [22]Lanxiang Hu, Mingjia Huo, Yuxuan Zhang, Haoyang Yu, Eric P. Xing, Ion Stoica, Tajana Rosing, Haojian Jin, and Hao Zhang. Imgame-Bench: How Good are LLMs at Playing Games?, 2025a.DOI
  23. [23]Mengkang Hu, Pu Zhao, Can Xu, Qingfeng Sun, Jianguang Lou, Qingwei Lin, Ping Luo, and Saravan Rajmohan. AgentGen: Enhancing Planning Abilities for Large Language Model based Agent via Environment and Task Generation. In Proceedings of the ACM SIGKDD Conference on Knowledge Discovery and Data Mining (KDD), 2025b.arxiv.org/abs/2408.00764
  24. [24]Lawrence Keunho Jang, Jing Yu Koh, Daniel Fried, and Ruslan Salakhutdinov. Odysseys: Benchmarking Web Agents on Realistic Long Horizon Tasks, 2026. URL https://arxiv.org/abs/2604.24964.
  25. [25]Mengdi Jia, Zekun Qi, Shaochen Zhang, Wenyao Zhang, Xinqiang Yu, Jiawei He, He Wang, and Li Yi. OmniSpatial: Towards Comprehensive Spatial Reasoning Benchmark for Vision Language Models, 2025. URL https://arxiv.org/abs/2506.03135.
  26. [26]Carlos E. Jimenez, John Yang, Alexander Wettig, Shunyu Yao, Kexin Pei, Ofir Press, and Karthik Narasimhan. SWE-bench: Can Language Models Resolve Real-World GitHub Issues? In The Twelfth International Conference on Learning Representations, 2024.arxiv.org/abs/2310.06770
  27. [27]Jing Yu Koh, Robert Lo, Lawrence Jang, Vikram Duvvur, Ming Chong Lim, Po-Yu Huang, Graham Neubig, Shuyan Zhou, Ruslan Salakhutdinov, and Daniel Fried. VisualWebArena: Evaluating Multimodal Agents on Realistic Visual Web Tasks. In Proceedings of the Annual Meeting of the Association for Computational Linguistics (ACL), 2024. arXiv:2401.13649.arxiv.org/abs/2401.13649
  28. [28]Kaixin Li, Ziyang Meng, Hongzhan Lin, Ziyang Luo, Yuchen Tian, Jing Ma, Zhiyong Huang, and Tat-Seng Chua. ScreenSpot-Pro: GUI Grounding for Professional High-Resolution Computer Use, 2025. URL https://arxiv.org/abs/2504.07981.
  29. [29]Manling Li, Shiyu Zhao, Qineng Wang, Kangrui Wang, Yu Zhou, Sanjana Srivastava, Cem Gokmen, Tony Lee, Li Erran Li, Ruohan Zhang, Weiyu Liu, Percy Liang, Li Fei-Fei, Jiayuan Mao, and Jiajun Wu. Embodied Agent Interface: Benchmarking LLMs for Embodied Decision Making. In Advances in Neural Information Processing Systems (NeurIPS) Datasets and Benchmarks Track, 2024. arXiv:2410.07166.arxiv.org/abs/2410.07166
  30. [30]Enqi Liu, Liyuan Pan, Zhi Gao, Yan Yang, Chenrui Shi, Yang Liu, Jingrong Wu, and Qing Li. Benchmarking and Improving GUI Agents in High-Dynamic Environments, 2026.arxiv.org/abs/2604.25380
  31. [31]Yuanxin Liu, Shicheng Li, Yi Liu, Yuxiang Wang, Shuhuai Ren, Lei Li, Sishuo Chen, Xu Sun, and Lu Hou. TempCompass: Do Video LLMs Really Understand Videos? In Lun-Wei Ku, Andre Martins, and Vivek Srikumar (eds.), Findings of the Association for Computational Linguistics: ACL 2024, pp. 8731–8772, Bangkok, Thailand, August 2024. Association for Computational Linguistics. doi: 10.18653/v1/2024.findings-acl.517. URL https://aclanthology.org/2024.findings-acl.517/.DOI
  32. [32]Yaxin Luo, Zhaoyi Li, Jiacheng Liu, Jiacheng Cui, Xiaohan Zhao, and Zhiqiang Shen. Open CaptchaWorld: A Comprehensive Web-based Platform for Testing and Benchmarking Multimodal LLM Agents, 2025. URL https://arxiv.org/abs/2505.24878.
  33. [33]Wufei Ma, Haoyu Chen, Guofeng Zhang, Yu-Cheng Chou, Jieneng Chen, Celso M de Melo, and Alan Yuile. 3DSRBench: A Comprehensive 3D Spatial Reasoning Benchmark, 2024. URL https://arxiv.org/abs/2412.07825.
  34. [34]Julius Mayer, Mohamad Ballout, Serwan Jassim, Farbod Nosrat Nezami, and Elia Bruni. iVISPAR – An Interactive Visual-Spatial Reasoning Benchmark for VLMs, 2025.arxiv.org/abs/2502.03214
  35. [35]Mingyu Ouyang, Siyuan Hu, Kevin Qinghong Lin, Hwee Tou Ng, and Mike Zheng Shou. GameWorld: Towards Standardized and Verifiable Evaluation of Multimodal Game Agents, 2026. URL https://arxiv.org/abs/2604.07429.
  36. [36]Davide Paglieri, Bartlomiej Cupiał, Samuel Coward, Ulyana Piterbarg, Maciej Wolczyk, Akbir Khan, Eduardo Pignatelli, Łukasz Kuciński, Lerrel Pinto, Rob Fergus, Jakob Nicolaus Foerster, Jack Parker-Holder, and Tim Rocktäschel. BALROG: Benchmarking Agentic LLM and VLM Reasoning On Games. In International Conference on Learning Representations (ICLR), 2025. arXiv:2411.13543.arxiv.org/abs/2411.13543
  37. [37]Jiayi Pan, Xingyao Wang, Graham Neubig, Navdeep Jaitly, Heng Ji, Alane Suhr, and Yizhe Zhang. Training Software Engineering Agents and Verifiers with SWE-Gym. In Proceedings of the International Conference on Machine Learning (ICML), 2025.arxiv.org/abs/2412.21139
  38. [38]Dongmin Park, Minky u Kim, Beongjun Choi, Junhyuck Kim, Keon Lee, Jonghyun Lee, Inkyu Park, Byeong-Uk Lee, Jaeyoung Hwang, Jaewoo Ahn, et al. Orak: A Foundational Benchmark for Training and Evaluating LLM Agents on Diverse Video Games, 2025.arxiv.org/abs/2506.03610
  39. [39]Simon Ramstedt and Christopher Pal. Real-Time Reinforcement Learning. In Advances in Neural Information Processing Systems (NeurIPS), 2019. arXiv:1911.04448.arxiv.org/abs/1911.04448
  40. [40]Christopher Rawles, Sarah Clinckemaille, Yifan Chang, Jonathan Waltz, Gabrielle Lau, Marybeth Fair, Alice Li, William Bishop, Wei Li, Folawiyo Campbell-Aj ala, Daniel Toyama, Robert Berry, Divya Tyamagundlu, Timothy Lillicrap, and Oriana Riva. AndroidWorld: A Dynamic Benchmarking Environment for Autonomous Agents, 2024.arxiv.org/abs/2405.14573
  41. [41]Yufan Ren, Konstantinos Tertikas, Shalini Maiti, Junlin Han, Tong Zhang, Sabine Süsstrunk, and Filippos Kokkinos. VGRP-Bench: Visual Grid Reasoning Puzzle Benchmark for Large Vision-Language Models, 2025. URL https://arxiv.org/abs/2503.23064.
  42. [42]Mikayel Samvelyan, Robert Kirk, Vitaly Kurin, Jack Parker-Holder, Minqi Jiang, Eric Hambro, Fabio Petroni, Heinrich Küttler, Edward Grefenstette, and Tim Rocktäschel. MiniHack the Planet: A Sandbox for Open-Ended Reinforcement Learning Research. In Advances in Neural Information Processing Systems Datasets and Benchmarks Track, 2021.arxiv.org/abs/2109.13202
  43. [43]Manolis Savva, Abhishek Kadian, Oleksandr Maksymets, Yili Zhao, Erik Wijmans, Bhavana Jain, Julian Straub, Jia Liu, Vladlen Koltun, Jitendra Malik, Devi Parikh, and Dhruv Batra. Habitat: A Platform for Embodied AI Research, 2019. URL https://arxiv.org/abs/1904.01201.
  44. [44]Ziyao Shangguan, Chuhan Li, Yuxuan Ding, Yanan Zheng, Yilun Zhao, Tesca Fitzgerald, and Arman Cohan. TOMATO: Assessing Visual Temporal Reasoning Capabilities in Multimodal Foundation Models, 2024. URL https://arxiv.org/abs/2410.23266.
  45. [45]Jiajun Shi, Jian Yang, Jiaheng Liu, Xingyuan Bu, Jiangjie Chen, Junting Zhou, Kaijing Ma, Zhoufutu Wen, Bingli Wang, Yancheng He, Liang Song, Hualei Zhu, Shilong Li, Xingjian Wang, Wei Zhang, Ruibin Yuan, Yifan Yao, Wenjun Yang, Yunli Wang, Siyuan Fang, Siyu Yuan, Qianyu He, Xiangru Tang, Yingshui Tan, Wangchunshu Zhou, Zhaoxiang Zhang, Zhoujun Li, Wenhao Huang, and Ge Zhang. KORGym: A Dynamic Game Platform for LLM Reasoning Evaluation, 2025. URL https://arxiv.org/abs/2505.14552.
  46. [46]Mohit Shridhar, Jesse Thomason, Daniel Gordon, Yonatan Bisk, Winson Han, Roozbeh Mottaghi, Luke Zettlemoyer, and Dieter Fox. ALFRED: A Benchmark for Interpreting Grounded Instructions for Everyday Tasks. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), June 2020.arxiv.org/abs/1912.01734
  47. [47]Mustafa Shukor, Dana Aubakiróva, Francesco Capuano, Pepijn Kooijmans, Steven Palma, Adil Zouitine, Michel Arachtig, Caroline Pascal, Martino Russi, Andres Marafioti, Simon Alibert, Matthieu Cord, Thomas Wolf, and Remi Cadene. SmolVLA: A Vision-Language-Action Model for Affordable and Efficient Robotics, 2025. URL https://arxiv.org/abs/2506.01844.
  48. [48]Yueqi Song, Tianyue Ou, Yibo Kong, Zecheng Li, Graham Neubig, and Xiang Yue. VisualPuzzles: Decoupling Multimodal Reasoning Evaluation from Domain Knowledge, 2025. URL https://arxiv.org/abs/2504.10342.
  49. [49]Sanjana Srivastava, Chengshu Li, Michael Lingelbach, Roberto Martín-Martín, Fei Xia, Kent Vainio, Zheng Lian, Cem Gokmen, Shyamal Buch, C. Karen Liu, Silvio Savarese, Hyowon Gweon, Jiajun Wu, and Li Fei-Fei. BEHAVIOR: Benchmark for Everyday Household Activities in Virtual, Interactive, and Ecological Environments, 2021. URL https://arxiv.org/abs/2108.03332.
  50. [50]Zafir Stojanovski, Oliver Stanley, Joe Sharratt, Richard Jones, Abdulhakeem Adefioye, Jean Kaddour, and Andreas Köpf. REASONING GYM: Reasoning Environments for Reinforcement Learning with Verifiable Rewards. In Advances in Neural Information Processing Systems (NeurIPS), 2025.arxiv.org/abs/2505.24760
  51. [51]Jiaming Tang, Yufei Sun, Yilong Zhao, Shang Yang, Yujun Lin, Zhuoyang Zhang, James Hou, Yao Lu, Zhijian Liu, and Song Han. VLASH: Real-Time VLAs via Future-State-Aware Asynchronous Inference, 2025. URL https://arxiv.org/abs/2512.01031.
  52. [52]Xiwen Teoh, Yun Lin, Siqi Li, Ruofan Liu, Avi Sollomoni, Yaniv Harel, and Jin Song Dong. Are CAPTCHAs Still Bot-hard? Generalized Visual CAPTCHA Solving with Agentic Vision Language Model. In 34th USENIX Security Symposium (USENIX Security 25), 2025.
  53. [53]Bowen Wang, Dunjie Lu, Junli Wang, Tianyi Bai, Shixuan Liu, Zhipeng Zhang, Haiquan Wang, Hao Hu, Tianbao Xie, Shuai Bai, Dayiheng Liu, Que Shen, Junyang Lin, and Tao Yu. CUA-Gym: Scaling Verifiable Training Environments and Tasks for Computer-Use Agents, 2026a. URL https://arxiv.org/abs/2605.25624.
  54. [54]Junyu Wang, Changjia Zhu, Yuanbo Zhou, Lingyao Li, Xu He, Mingkui Wei, and Junjie Xiong. COGNITION: From Evaluation to Defense against Multimodal LLM CAPTCHA Solvers, 2025a. URL https://arxiv.org/abs/2512.02318. USENIX Security 2026.
  55. [55]Qineng Wang, Baiqiao Yin, Pingyue Zhang, Jianshu Zhang, Kangrui Wang, Zihan Wang, Jieyu Zhang, Keshigeyan Chandrasegaran, Han Liu, Ranjay Krishna, Saining Xie, Jiajun Wu, Li Fei-Fei, and Manling Li. MindCube: Spatial Mental Modeling from Limited Views, 2025b. URL https://arxiv.org/abs/2506.21458.
  56. [56]Xinyu Wang, Bohan Zhuang, and Qi Wu. Are Large Vision Language Models Good Game Players? In International Conference on Learning Representations (ICLR), 2025c. arXiv:2503.02358.arxiv.org/abs/2503.02358
  57. [57]Zirui Wang, Junyi Zhang, Jiaxin Ge, Long Lian, Letian Fu, Lisa Dunlap, Ken Goldberg, XuDong Wang, Ion Stoica, David M. Chan, Sewon Min, and Joseph E. Gonzalez. VisGym: Diverse, Customizable, Scalable Environments for Multimodal Agents, 2026b. URL https://arxiv.org/abs/2601.16973.
  58. [58]Yule Wen, Yixin Ye, Yanzhe Zhang, Diyi Yang, and Hao Zhu. Real-Time Reasoning Agents in Evolving Environments. In International Conference on Learning Representations (ICLR), 2026. arXiv:2511.04898.arxiv.org/abs/2511.04898

Paper details

vlnc:W_cucm2qsk
Version
v1 · latest
Published
Fresh — added to the Hub within the last 30 days
Added
License
CC BY 4.0
Usage

views · 12

downloads · 1

A view counts once per browser per day, a download each time the PDF is requested.

  • Computer Science

Not recorded for this paper

View statement
Looking for related papers…
Contents