WorksComputer Science
GenXR-Bench: Benchmarking Coding Agents for Interactive XR Development
Abstract
Coding agents are rapidly advancing from single-file generation to long-horizon repository-scale engineering, yet the evaluation of their capabilities to author interactive extended reality (XR) applications remains an open challenge. Unlike traditional software, XR applications operate within dynamic 3D physical environments where runtime robustness depends jointly on continuous user motion, spatial input, and environmental sensing. We introduce GenXR-Bench, the first benchmark for the end-to-end evaluation of coding agents in XR software development. GenXR-Bench contains 112 expert-authored tasks spanning 24 feature implementations, 21 bug repairs, and 67 application generation tasks. Held-out tests combine scripted and adaptive interactions with programmatic and visual assertions to evaluate core functionality and runtime behavior. GenXR-Bench's testing suite achieves 96.5% accuracy against blinded expert judgments of 124 candidate applications executed on consumer XR hardware. Evaluating 10 models on GenXR-Bench, we find that the strongest configuration achieves only 50.9% overall Pass@1. Failure analysis identifies recurring misunderstandings in asynchronous sensor lifecycles, physical grounding, and embodied interaction semantics. GenXR-Bench establishes a reproducible testbed for evaluating coding agents in the interactive and physically situated conditions of spatial computing.
Introduction
Coding agents have become increasingly capable of building interactive software. Recent benchmarks have expanded beyond algorithmic problems (Jimenez et al., [29]) to end-to-end interactive applications, evaluating how models handle dynamic user input in web and desktop environments ((Lu et al., 2025; Zhang et al., 2025)). Extended reality (XR) introduces a distinct set of challenges for code generation. Unlike 2D software governed by discrete event loops, XR applications operate within continuous 3D physical spaces. Functional correctness depends on the real-time coordination of head and hand kinematics, spatial coordinate transformations, environmental sensing, mixed-reality rendering, and physical ergonomics. Consequently, generating XR software requires reasoning about spatial mathematics, runtime sensor lifecycles, and embodied interaction semantics that are absent in flat-screen software engineering.
Despite these demands, evaluating coding agents in XR remains largely unexplored due to an execution and verification bottleneck. Existing coding benchmarks rely on programmatic unit tests (Jimenez et al., [29]) or browser action runners (Lu et al., 2025), which cannot verify spatial scene graphs or continuous 3D tracking inputs. Conversely, 3D code-generation benchmarks focus primarily on generating static shapes rather than reactive software (Gao et al., [17]), while prior XR benchmarks focus on hardware performance and runtime systems rather than evaluating generative code synthesis [27, 2] (Popescu et al. 2025). Evaluating agent-generated XR software requires an automated and comprehensive testing suite capable of simulating physical environments, exercising embodied user actions, and verifying runtime behavior across diverse implementations.
We present GenXR-Bench, the first benchmark for end-to-end evaluation of coding agents in XR software development. GenXR-Bench comprises 112 expert-curated tasks across two tiers: (1) SDK engineering tasks, consisting of 24 feature implementations and 21 bug repairs in core subsystems such as occlusion shaders, raycast reprojection, and gestural state machines; (2) Application generation tasks, consisting of 67 end-to-end experiences spanning education, healthcare, daily living, design, and entertainment, distilled from a taxonomy of 436 peer-reviewed XR systems papers. To support reproducible, headless execution without the proprietary GUI navigation and multi-gigabyte build toolchains of desktop game engines, all tasks are built on XR Blocks [35], an open-source WebXR framework. To test candidate implementation at scale, we introduce GenXR-Test, the first set of testing tools that pair deterministic kinematic actors with adaptive VLM-controlled agents in simulated XR environments, evaluating runtime results through both scene-graph assertions and scoped visual checks.
To validate our authored evaluation suite, eight blinded XR experts evaluated 124 candidate applications executed on consumer XR hardware. Across 372 evaluation runs, our automated suite achieved 96.5% mean accuracy against human expert ground truth with a 4.0% flakiness rate, demonstrating that simulator-based evaluation reliably reflects physical device execution. We evaluate 10 model configurations on the full benchmark. The strongest configuration achieves an overall Pass@1 of 50.9%, with performance dropping to 29.2% on SDK feature tasks. Diagnostic analysis reveals recurring failure modes across models: 64.4% of failed runs exhibit coordinate-transformation errors, 44.9% encounter asynchronous sensor initialization races, and 34.5% fail to distinguish bimanual interaction semantics.
In summary, our contributions are as follows:
GenXR-Bench: An open-source benchmark of 112 expert-authored tasks evaluating coding agents on XR software engineering, spanning SDK-level feature development, bug repair, and end-to-end application synthesis.
GenXR-Test: XR tooling for automated evaluations using scripted kinematic actors, adaptive VLM agents, runtime state assertions, and scoped visual checks in simulated 3D environments.
Hardware-Grounded Validation: A validation study across 372 physical XR device trials demonstrating that simulator-based evaluation achieves 96.5% agreement with expert human judgment with low flakiness (4.0%).
Empirical Analysis of Coding Agents in XR: An evaluation of 10 model configurations identifying key bottlenecks in frontier models, including coordinate-frame handling, sensor lifecycles, and embodied interaction semantics.
Related Work
Interactive Software Engineering Benchmarks
Interactive software often requires its own set of runtime evaluations. Recent agent benchmarks treat interactivity as the benchmark itself, using operating systems and interactive worlds to evaluate agents in controlled application environments ([31] (Xie et al. 2024; Rawles et al. 2025; Trivedi et al. 2024)), while embodied-agent benchmarks similarly test long action sequences in fixed 3D environments ([32] (Puig et al. 2018; Savva et al. 2019; Shridhar et al. 2020)). Interactive software generation benchmarks often leverage these systems to evaluate generated software: website development benchmarks use agent-controlled browser actions to evaluate functionality ([31] (Lu et al. 2025)); ArtifactsBench ((Zhang et al., 2025)), Interaction2Code ((Xiao et al., 2025)), and InteractScience ([9]) combine runtime interaction and rendering with multimodal evaluations using vision-language models as judges ([6] (Zheng et al. 2023; Wu et al. 2024)); game-development benchmarks ([10, 34, 8] (Luo et al. 2026)) combine functional, visual, and behavioral evaluations with interaction testing. GenXR-Bench extends runtime evaluations to XR software by simulating embodied interactions and environmental sensing to become the first benchmark to formalize evaluations of physically grounded interactive applications beyond relying purely on LLM judges. A systematic comparison of GenXR-Bench against these existing benchmarks is shown in Table 1.
| Domain | Benchmark | Primary tasks | ER | IT | VT | Evaluation |
| SWE | SWE-bench ([29]) | Repository repair | ✗ | ✗ | ✗ | Repository tests |
| SWE | SWE-bench Multimodal ((Yang et al., 2025)) | Repository repair | ✗ | ✓ | ✓ | Repository tests; UI-event checks; pixel-level visual tests |
| Web | Interaction2Code ((Xiao et al., 2025)) | Interactive page reconstruction | ✗ | ✓ | ✓ | Interaction tests; visual metrics; human assessment |
| Web | WebGen-Bench ((Lu et al., 2025)) | Website generation | ✗ | ✓ | ✓ | Browser-agent tests; VLM judge |
| Games | GameDevBench ([10]) | Feature implementation | ✗ | ✓ | ✗ | Deterministic engine tests |
| Games | GameCraft-Bench ((Luo et al., 2026)) | Game generation | ✗ | ✓ | ✓ | Solver-submitted input replay; VLM judge |
| Games | OpenGame-Bench ([28]) | Browser-game generation | ✗ | ✓ | ✓ | Automated play; runtime metrics; VLM judge |
| Games | GameXpert-Bench ([8]) | Game generation; repair; refinement | ✗ | ✓ | ✓ | Behavioral probes; task-specific judgments |
| 3D | WorldCoder-Bench ([37]) | Interactive world generation | ✗ | ✓ | ✗ | Expert-authored runtime contracts |
| Visual | ArtifactsBench ((Zhang et al., 2025)) | Artifact generation | ✗ | ✓ | ✓ | Interaction screenshots; VLM checklists |
| Visual | InteractScience ([9]) | Scientific demo generation | ✗ | ✓ | ✓ | Scripted interaction; assertions; visual assessment |
| XR | VCXR60 ([15]) | App generation | ✓ | ✗ | ✗ | Error-free simulator startup |
| XR | GenXR-Bench (Ours) | SDK features; repair; app generation | ✓ | ✓ | ✓ | Expert-authored interaction tests; runtime assertions; scoped VLM judge |
Table 1. Comparison with representative coding benchmarks. ER: embodied reasoning; IT: interaction-based testing; VT: visual testing. ER refers to evaluated capabilities of the benchmark (e.g., applications related to physical user interactions or spatial computing) while IT and VT refer to evaluation methods.
Generative 3D and XR Authoring
Natural-language 3D authoring predates coding agents, with early systems such as WordsEye ([13]) and the framework proposed by Chang et al. (2014) translating text into virtual scenes. Recent approaches use language models to generate procedural content and interactive environments ((Sun et al., 2025; Yang et al., 2024b,c)). As a result, code-based 3D generation benchmarks increasingly extend evaluation beyond simple primitives to perceptually and physically correct models [17, 37, 23]. In XR, Text2VRScene (Yin et al., 2024) and VRCopilot (Zhang et al., 2024) allow users to use language to author new scenes and layouts. LLMR generates XR application features in real time [14]; Dream-CodeVR generates object behaviors from speech within VR [18]; and LLMER produces structured specifications that control XR objects [7]. GenAssist explores iterative agentic rendering loops to refine XR applications guided by human ratings (Srinidhi et al., 2026). XARP [4] and XR Blocks [35] introduce agent-native XR authoring toolkits for easy XR code generation. Closest to our work, Du et al. [15] demonstrate that current LLMs can author XR applications that initialize without errors. GenXR-Bench continues this work, building upon the XR Blocks framework validated by Du et al. [15] to systematically verify whether candidate implementations satisfy task-specific functional and interactive requirements across diverse XR development tasks.

Figure 2. Authoring workflow of GenXR-Bench. Human icon indicates human-authored and robot icon indicates LLM-authored. Both icons indicate LLM generated with human guidance.
Automated XR Testing
XR presents unique evaluation challenges because runtime behaviors depend continuously on the user’s kinematics, physical environment, and spatial interaction dynamics [3]. Automated XR testing spans from replaying device input with state checkpoints [2], categorizing WebXR [36] and Unity (Rzig et al., 2023) bugs, and formalizing XR interface heuristics [21] (Nielsen & Molich 1990; Utting et al. 2012). The OpenXR Conformance Test Suite checks whether XR runtimes implement OpenXR interfaces consistently (Khronos OpenXR Working Group, 2025). VR-ReST derives functional tests from requirements and scene-graph structure [12]. Other work simulates user actions through camera-based exploration and scripted XR controllers [21] (Wang 2022; Wang et al. 2023; Zhu et al. 2025). XRintTest constructs an interaction-flow graph in existing Unity scenes [22]. Commercial solutions such as Meta XR Operator (Meta, 2026) integrate an LLM-driven tester into XR development to run natural-language tasks in Unity. Closest to our work are XR benchmarks that target end-to-end systems [27], machine-learning workloads [33], and performance criteria (Popescu et al., 2025). We extend prior works by introducing GenXR-Test, which empowers task authors to define implementation-agnostic functional tests executed deterministically within a reproducible simulator environment.
GenXR-Bench
GenXR-Bench consists of 112 expert-curated XR software engineering tasks. Each task contains a natural-language specification, an editable project scaffold, and a held-out evaluation suite. The specification defines the required implementation and observable behavior. The scaffold provides the relevant code, tools, interfaces, documentation, and assets needed to complete the task. The evaluation suite contains human-authored tests written using GenXR-Test described in § 3.4.
All tasks use XR Blocks 0.21.0 [35], a web framework for building cross-platform XR applications. Prior work has demonstrated that LLMs can generate complete XR Blocks applications from a single prompt ([15]). Using a common web framework reduces variation caused by engine-specific editors, deployment workflows, and device toolchains that often require complex GUI navigation. All agents were given access to XR Blocks DevTools, a suite of tools for debugging, running, and developing XR Blocks applications that simplifies multimodal development.
Benchmark Composition
GenXR-Bench covers three types of XR development tasks motivated by real-world use cases: feature implementation, bug repair, and application generation. Examples are shown in Appendix § 12.
SDK feature implementation. Feature tasks require agents to implement a missing XR Blocks capability while retaining the surrounding SDK, interfaces, documentation, and relevant examples. Tasks cover capabilities such as hand interaction, sensing, spatial design, and rendering that form the primitive features for core XR development. These tasks are designed to evaluate an agent’s ability to implement and understand low-level XR functionality and focus on software engineering.
SDK bug repair. Repair tasks begin with a snapshot of the XR Blocks SDK containing a reproduced behavioral issue. The specification describes the observable failure and required behavior without identifying the faulty implementation. Agents must locate and repair the defect while preserving related functionality.
Application generation. Application tasks require agents to build a complete XR experience from a natural-language specification. Each specification outlines the intended experience, interaction flow, spatial behavior, and success conditions. Agents must combine SDK capabilities and design a working application that meets the specification.
Benchmark Curation and Construction
The benchmark tasks were curated and sourced by five XR experts, including professional XR engineers, XR researchers, and contributors to XR Blocks. All tasks were validated on XR hardware by reviewers to ensure that they accurately reflect real-world usage.
SDK tasks. To source the SDK tasks, we interviewed core XR Blocks contributors and inspected all 552 pull requests and issues available as of August 24, 2026. Maintainer reports and pull requests were pooled into a candidate list of potential features and prior bugs, then filtered based on their relevance to XR engineering, evaluation feasibility, and scope. An author converted each selected candidate into a self-contained benchmark task. For feature tasks, the author removed the target implementation while retaining its public interface and context. For repair tasks, the author reproduced the selected defect in a pinned XR Blocks snapshot. SDK tasks contain both fail-to-pass tests for the target behavior and pass-to-pass tests for related behavior that test writers identified as potentially affected by candidate implementations. Authors were instructed to create adversarial solutions and conduct three rounds of test-suite refinement.
Application tasks. We derived an application taxonomy from 436 XR systems papers published at top human-computer interaction and spatial computing venues. Using this taxonomy, gemini-3.5-flash generated 1,000 candidate application ideas. XR experts selected 67 candidates based on XR specificity, realism, implementation feasibility, and evaluation feasibility. Following Luo et al. (2026), we expanded the selected ideas into full specifications using gemini-3.6-flash, manually reviewed and edited by XR experts. Appendix § 10 describes the full procedure.
Authors then created an evaluation suite for each application task. Since there can be multiple valid implementations, we do not require a canonical reference solution. Authors refined each specification and evaluation suite using expert-reviewed valid and adversarial candidates, revising the task specification over up to 10 iterations. Tests were designed to assess the required interactive experience while accommodating differences between implementations.
Benchmark Statistics
GenXR-Bench contains 112 tasks consisting of 24 SDK feature implementations, 21 bug repairs, and 67 app generation tasks. XR experts rated 28 tasks as easy, 38 as medium, and 46 as hard based on subsystem breadth, interaction length, and expected implementation effort. Experts also assigned labels based on each task’s coverage of core XR capabilities, including interaction, user interfaces (UI), hand recognition, virtual object manipulation, depth sensing, object placement, perception, drawing, audio, and head tracking. A task could receive multiple capability labels depending on its scope. Figure 3 summarizes the task types and capability coverage.

Figure 3. Composition of GenXR-Bench. a) Distribution of task type and difficulty. The donut’s inner ring shows task type, while the outer ring shows easy (E), medium (M), and hard (H) task counts within each type. b) Distribution of XR capabilities evaluated across tasks. Each task can have multiple capability labels.
GenXR-Test Evaluation Framework
To build each evaluation suite, we introduce GenXR-Test, a set of testing tools for the creation of adaptive interaction tests. Each held-out evaluation suite combines actors that simulate user actions and environmental inputs with assertions that check the resulting application behavior. This allows tests to assess SDK implementations and generated applications under controlled runtime conditions. A custom simulator provides fully articulated hand inputs as well as various physical scenes and objects to simulate real-world XR environments. Figure 4 illustrates an example evaluation test. More information on the implementation of our evaluation suite can be found in Appendix § 9.

Figure 4. Example of a human-written test for the bumble-calm task, which asks coding agents to implement a simple calming game where users hold a palm up to summon a bee.
Evaluation Actors. Actors are testing functions that simulate head, hand, controller, gesture, object, environment, and time inputs. A scripted actor runs a predefined action sequence, such as performing a hand gesture, pressing a specified control, or following a path. Authors use scripted actors when the required action can be specified before execution, improving reproducibility and reducing variation in test execution. An adaptive actor is a VLM-powered agent that follows a narrowly scoped natural-language instruction when the required action cannot be reliably determined without knowledge of the candidate implementation, following prior work (Lu et al., 2025). For example, it may locate and activate a generated control or find an object needed for a later assertion. Authors use adaptive actors when differences in layout or interaction flow make fixed action sequences unreliable.
Evaluation Assertions. After an actor produces a runtime state, assertions determine whether the required functional behavior has occurred. Programmatic assertions inspect runtime properties such as object transforms, spatial relationships, interaction state, collisions, tracking state, sensor outputs, and emitted events. For example, a test might check that a virtual object follows the hand and disappears upon colliding with a wall. Authors use programmatic assertions whenever the required behavior can be checked deterministically, focusing on observable outcomes to reduce flakiness (Luo et al., 2014). VLM-based assertions assess visual outcomes that are difficult to express directly as programmatic checks. Each assertion receives rendered images from the simulated user’s point of view and a narrowly scoped, human-authored criterion. These criteria describe an expected visual outcome rather than asking the VLM to judge the overall function of the experience, which limits judging failures. This design follows the findings of Cho et al. [11]; Huang et al. [25] and is evaluated in § 4.
Semantic Tags. Valid applications can use different identifiers and scene hierarchies for equivalent objects. Tasks in GenXR-Bench require evaluated coding agents to assign semantic tags to specified objects. Examples include controls, targets, and interactable objects. These tags allow actions and assertions to locate functionally equivalent objects without requiring a fixed implementation. Semantic roles are defined in the task specification and are limited to descriptions of the objects to be tagged, reducing the risk of exposing tests. Task authors are also explicitly instructed to limit the use of semantic tags to avoid forcing a single correct implementation.
Evaluating Application Design
An application can satisfy its functional requirements while remaining difficult or uncomfortable to use. We therefore report a separate design score for application-generation tasks, following prior work in website generation (Lu et al., 2025). During test execution, we record videos of the evaluator running the candidate application and use a VLM as a judge to rate the application against ten XR design heuristics, each on a 1–5 Likert scale. The design score is intended to supplement functional evaluations. Details on the development of the XR heuristics and the alignment of the VLM judge with human evaluators are provided in Appendix § 11 and Appendix § 13, respectively.
Analysis of App Verifier Reliability
We validate our evaluation suite through expert evaluations of 124 candidate applications across 62 app tasks in GenXR-Bench that did not require a specific physical object or setting. The candidates were generated with gemini-3.5-flash and gemini-3.7-flash. We purposefully select gemini-3.5-flash to add a greater diversity of generated app quality.
Eight XR experts participated in the evaluation. Each candidate received independent binary judgments of functional correctness from two evaluators, each of whom assessed the full functionality of the application. In cases of disagreement, a third independent evaluator re-examined the candidate and made the final decision. Evaluators were blind to the candidate source and verifier results and interacted freely with the application. For each candidate app, at least one evaluation used XR hardware (Galaxy XR, Meta Quest 3) to ensure alignment with real use cases.
We ran the automated evaluation suite three times for each candidate, producing 372 evaluation runs. As shown in Table 2, the verifier aligns strongly with human evaluations (accuracy: 96.5%) and is reproducible (flakiness: 4.0%). These results indicate that the evaluation suite reliably reflects expert assessments of XR app functionality. Disagreements occurred primarily in applications where experts performed tasks through interaction behaviors that the automated tests did not accommodate, such as a ball spawning at the wrist rather than the fingertips. These cases present a limitation of the verifier when candidate implementations depart from expected spatial interaction conventions.
| App Source | Accuracy | Precision | Recall | Fleiss’ | Flakiness |
| gemini-3.5-flash | 0.962 0.009 | 0.958 0.072 | 0.778 0.000 | 0.948 | 0.016 |
| gemini-3.7-flash | 0.968 0.032 | 1.000 0.000 | 0.923 0.077 | 0.909 | 0.065 |
| Overall | 0.965 0.020 | 0.989 0.019 | 0.886 0.057 | 0.929 | 0.040 |
Table 2. Auto-verifier alignment with expert human ground truth and execution stability across three repeated runs (124 applications, 372 runs). Classification metrics report the sample mean and standard deviation across runs (Mean STD). App Source: model used to generate the applications; Accuracy: proportion of classifications that match human ground truth; Precision: proportion of predicted passes that are human-labeled passes; Recall: proportion of human-labeled passes identified by the verifier; Fleiss’ : agreement across repeated runs [16]; Flakiness: proportion of applications whose verifier outputs differ across runs. The overall results are shown in bold.
Evaluations
Evaluation Setup
Evaluation environment. Following the procedures in the DeepSWE benchmark (Huang et al., [26]), we evaluated all model configurations using mini-swe-agent (Yang et al., 2024a) harness within the Harbor Framework [24]. Agents could access XR Blocks source code, documentation, examples, and relevant dependency documentation, while other network access was restricted. Adaptive actors and VLM-based assertions used gemini-3.8-flash.
Model choices. We select a variety of different models to evaluate. From the Google Gemini family, we evaluate gemini-3.1-pro-preview, gemini-3.5-flash, gemini-3.6-flash, gemini-3.7-flash, and gemini-3.8-flash, all at the high thinking level. From the OpenAI GPT family, we evaluate gpt-5.6-luna and gpt-5.6-sol, both at the xhigh reasoning level. From the Anthropic Claude family, we evaluate claude-opus-5 at the medium reasoning level2. For open-weight models, we evaluate deepseek-v4-pro-0813 and deepseek-v4.1-flash at the xhigh reasoning level.
Metrics. For all evaluations, we use Pass@1, the percentage of tasks completed successfully in one attempt, to evaluate the functionality of proposed solutions. Tasks are only considered successful when all required tests pass. For application-generation tasks, we report the mean of the 10 heuristic ratings produced by the VLM judge described in § 3.5 as the Design Score. This metric supplements the evaluation of functional correctness through Pass@1.
Discussion of Results
We now discuss our findings from evaluating coding agents as described in Table 3. Detailed results and findings are found in Appendix § 14.
| Model | Pass@1 (%) Overall | Repair | Feature | App | Design Score | T/E | Turns | Tokens (M) | Cost ($) |
| claude-opus-5 | 50.9 | 85.7 | 29.2 | 47.8 | 3.86 | 0.44 | 74.1 | 6.17 | 5.74 |
| deepseek-v4.1-flash | 46.4 | 95.2 | 20.8 | 40.3 | 3.81 | 0.22 | 114.9 | 14.77 | 0.71 |
| gpt-5.6-sol | 44.6 | 95.2 | 29.2 | 34.3 | 4.03 | 0.81 | 26.4 | 1.74 | 2.95 |
| gemini-3.8-flash | 42.9 | 90.5 | 25.0 | 34.3 | 3.71 | 0.89 | 127.0 | 11.91 | 1.73 |
| gemini-3.7-flash | 41.1 | 90.5 | 29.2 | 29.9 | 3.89 | 0.96 | 69.6 | 4.14 | 0.84 |
| deepseek-v4-pro-0813 | 39.3 | 85.7 | 12.5 | 34.3 | 3.53 | 0.38 | 90.5 | 8.77 | 0.63 |
| gemini-3.6-flash | 31.2 | 85.7 | 20.8 | 17.9 | 3.32 | 0.96 | 58.7 | 3.08 | 0.67 |
| gpt-5.6-luna | 30.4 | 85.7 | 16.7 | 17.9 | 3.48 | 0.41 | 33.8 | 2.22 | 0.36 |
| gemini-3.5-flash | 29.5 | 85.7 | 12.5 | 17.9 | 3.58 | 0.93 | 51.2 | 1.81 | 0.95 |
| gemini-3.1-pro-preview | 23.2 | 81.0 | 12.5 | 9.0 | 2.37 | 0.32 | 37.5 | 0.67 | 0.64 |
Table 3. Evaluation results across models. Pass@1 reports task success overall and by task category. All metrics are means across applicable runs. Design Score uses a 1–5 Likert scale. T/E is the test-to-edit ratio, computed as the ratio turns spent authoring and using unit tests and runtime tests over turns spent editing (Unit + Runtime)/Edit. Turns, Cost, and Tokens report the mean interaction turns, dollar cost, and token usage per task, respectively.
XR development remains challenging even for frontier models. Across GenXR-Bench, XR development remains challenging even for frontier models, with the best performing model only reaching a Pass@1 of 50.9 (claude-opus-5). XR functionality and design performance diverged where claude-opus-5 achieves the highest application-generation Pass@1 at 47.8%, while gpt-5.6-sol achieves the highest mean Design Score at 4.03. Notably, while SDK repair scores were high, SDK implementation and application-generation scores remained low, indicating that behavioral repair and bug identification pose a far easier challenge than end-to-end XR app implementation. Models further from the frontier were far less capable, failing on foundational spatial-computing concepts or completely unable to create core application objects. These smaller models suffer heavily from web development coding hallucinations, poor metric understanding, and API hallucinations. Frontier models are more capable of avoiding these errors but still fail at higherorder spatial reasoning, including bimanual coordination (Figure 12), room geometry visualizations (Figure 13), and physical alignment (Figure 14).

Figure 12. Examples of generated apps for the sight-magnifier task. The task requires a daily-living app in which users can form a box with their fingers and see a magnified view of its contents. Models struggled with hand-gesture recognition: they either failed to detect the gesture or aligned the magnified area poorly with the user’s view plane; only claude-opus-5 and deepseek-v4.1-flash effectively identified the hand framing gesture but only claude-opus-5 properly configured the magnification panel.

Figure 13. Examples of generated apps for the walking-clearance-visualization task. The task requires a simple app that visualizes walking clearance to help building owners identify accessibility challenges. Blue and orange boxes are simulated physical objects where we should expect they change the walking clearance visualization. Models struggled with depth tracking and often failed to calculate or render clearance accurately at runtime like the failures seen in gpt-5.6-sol and gemini-3.8-flash while claude-opus-5 and deepseek-v4.1-flash successfully implemented the visualization.

Figure 14. Examples of generated apps for the palm-controller task. The task requires a simple app where when the user flips their hand over, two buttons spawn that, when clicked, create different shapes in the palm. While all models shown were successful, the buttons were often placed deep into the hand due to using the hand root as a coordinate root which affected the overall design of the app.
Model-written tests are not indicative of success. Notably, models were relatively incapable of writing effective tests where the testing-to-editing ratio for each model was not indicative of overall success. Instead, models tend to overwrite unit tests that did not validate actual XR functionality and fail to have the foresight to predict real runtime conditions. Models that were more willing to evaluate code through runtime tests were also more successful. We report a detailed breakdown of agent tool usage in Table 11.
| Model | Calls | Expl. | Edit | E/E | Unit | Runt. | T/E |
| claude-opus-5 [medium] | 58.9 | 55.2 | 23.1 | 2.39 | 7.30 | 2.83 | 0.44 |
| deepseek-v4.1-flash [xhigh] | 109.3 | 101.3 | 47.6 | 2.13 | 9.70 | 0.68 | 0.22 |
| gpt-5.6-sol [xhigh] | 58.9 | 43.6 | 14.2 | 3.07 | 10.75 | 0.75 | 0.81 |
| gemini-3.8-flash [high] | 107.7 | 68.0 | 12.7 | 5.36 | 11.22 | 0.12 | 0.89 |
| gemini-3.7-flash [high] | 68.6 | 45.6 | 8.9 | 5.11 | 8.58 | 0.00 | 0.96 |
| deepseek-v4-pro-0813 [xhigh] | 77.0 | 66.9 | 22.9 | 2.93 | 8.55 | 0.03 | 0.38 |
| gemini-3.6-flash [high] | 64.0 | 44.4 | 10.2 | 4.37 | 9.70 | 0.00 | 0.96 |
| gpt-5.6-luna [xhigh] | 53.3 | 44.0 | 17.3 | 2.54 | 7.08 | 0.00 | 0.41 |
| gemini-3.5-flash [high] | 60.1 | 42.4 | 5.5 | 7.78 | 4.78 | 0.28 | 0.93 |
| gemini-3.1-pro-preview [high] | 39.6 | 29.6 | 7.5 | 3.98 | 2.33 | 0.03 | 0.32 |
Table 11. Development tool usage across models. Counts are means per task. Calls: total tool calls; Expl.: code reads and searches; Edit: code edits; E/E: exploration-to-edit ratio (Expl./Edit); Unit: unit test executions using Vitest or npm test; Runt.: runtime test executions using the DevTools simulator or Chromium; T/E: test-to-edit ratio ((Unit + Runt.)/Edit).
Failure Analysis and Directions for Improvement
XR experts then manually analyzed task failure modes and grouped them into recurring categories. These failures indicate potential openings for improving the ability of agents to create spatial and embodied experiences. We provide a more detailed analysis in Appendix § 15.
Asynchronous initialization and environmental updates. Startup race conditions occurred in 44.9% of failed runs. Implementations began gameplay updates or sensing-dependent calculations before the required assets or tracking state were available, causing runtime exceptions. Other implementations checked for detected surfaces only at startup and did not respond when the surfaces became available later. This behavior also limited the functionality of some applications that passed the tests. For instance, in the lascaux-torch task shown in Figure 11, successful implementations made virtual walls to simulate a cave whereas an XR developer might have developed shaders to place cave paintings on physical walls. These findings suggest a need for better handling of initialization dependencies and continuous sensing input throughout execution.

Figure 11. Examples of generated apps for the lascaux-torch task. The task requires an educational app in which users can explore cave paintings in their room. The gpt-5.6-sol app failed to start due to hallucinating a reticle configuration that prevented it from loading. The other apps were functional but created virtual walls instead of rendering cave paintings on the physical environment.
Coordinate transformations. Coordinate-frame errors occurred in 64.4% of failed runs. Implementations compared positions expressed relative to different parent objects without first transforming them to a common coordinate frame. This was especially apparent for objects parented by user transforms like the hand as is visible in Figure 14 where models anchored buttons directly to hands’ root joint rather than placing them onto the palm surface. These failures highlight recurring difficulties in maintaining a consistent spatial model of the physical world, the user, and virtual content in XR development.
Embodied interaction detection and usability. Current models still lack an embodied understanding of how humans exist and interact in XR and spatial computing settings. Agents often struggled with hand interactions and asymmetric hand gestures (34.5% of failures), accepting interactions from either hand interchangeably and failing to enforce separate tracking semantics. Other failures involved gesture recognition, as shown in Figure 12, where candidate applications often failed to detect the framing pose. Agents often treated physical touch as a discrete click, relying on Boolean bounding-box intersections without tracking continuous penetration along surface normals. This often ten caused unstable contact detection during interactions. Finally, agents routinely failed to account for human ergonomics, placing interactive buttons out of comfortable reach, positioning menus behind the user, or rendering spatial UI at disproportionate scales, degrading ergonomic usability.
Conclusion
We present GenXR-Bench, the first benchmark to evaluate the functional and interactive correctness of agent-generated XR software. To create GenXR-Bench, five XR experts hand-wrote and curated 112 tasks spanning core XR engineering workflows within XR Blocks, using our novel automated XR testing framework, GenXR-Test. We find that frontier coding agents still struggle with XR development tasks, especially when tasks require continuous reasoning over embodied interactions and asynchronous XR runtime state. Our findings highlight the need to improve the embodied understanding of agents. We envision GenXR-Bench as a foundational testbed for advancing spatially situated software engineering across spatial AI, perception, and embodied interactions.
AI Use Disclosure
In this work, we used generative AI tools for dataset generation, benchmark authoring, data analysis, paper writing and editing. We did not use generative AI tools to develop conceptual frameworks. We developed GenXR-Bench alongside human evaluators, ensuring that domain experts guided all parts of this work and reviewed all AI-assisted work. We take responsibility for the final content of this work, including text, claims, and artifacts produced with the aid of generative AI.
Ethics Statement
This work investigates LLM-based code generation for XR applications to improve our understanding of how models reason about spatial computing and implement it in concrete applications. Our data collection was limited to real-world XR frameworks and was guided by contributors to ensure that all data is controlled and accessible. We take care to anonymize all potentially personally identifiable information. XR applications carry risks, including placing users at risk of physical harm by obscuring real-world hazards, so we limit our scope to safe, non-sensitive settings. All pretrained models used are publicly available and are used in accordance with their licenses. We believe our work contributes to the study of frontier coding capabilities.
Reproducibility Statement
We have made considerable efforts to ensure the reproducibility of our results. All prompts used for data generation or evaluation are published in our appendix. We include details of dataset construction, preprocessing, and evaluation metrics. Our benchmark will be released as open source upon publication. Furthermore, we implement the benchmark using the Harbor Framework ([24]) to support reproduction and access.
GenXR-Bench Implementation
GenXR-Bench is built on the XR Blocks framework [35]. We pin all benchmark tasks to a stable version of XR Blocks 0.21.0, remove extraneous add-ons, and precache all runtime network downloads to support network-isolated execution in Harbor. The following XR Blocks features enable XR evaluations in simulated environments, which simplifies much of the complexity in other XR frameworks like Unity, Godot, or Unreal Engine that require in-depth understanding and usage of GUI features to run the engine. Instead, XR Blocks is implemented through common web frameworks like Three.js and WebXR. Thus, XR Blocks remains a strong environment to evaluate XR understanding and can be viewed as a viable native engine equivalent. The following XR Blocks capabilities enable simulated XR evaluations.
Articulated hand simulations. To simulate XR interactions accurately, we use the articulated hand simulations provided by XR Blocks. The pose of each hand is represented by XYZ joint rotations and computed through forward kinematics over the XR hand skeleton. Tests can select common pose presets or specify joint rotations directly. Joint limits prevent actors from producing unrealistic hand poses. All poses are injected directly into the XR Blocks runtime to support standard gesture-recognition implementations and XR interaction logic in candidate applications.
Simulated physical environments. XR Blocks’ simulator also supports multiple human-authored physical environments. These environments simulate physical room information by injecting it into plane, mesh, depth, and object-detection pathways at runtime. Each environment includes navigation meshes that enable realistic movement. GenXR-Bench uses two scenes: Living Room and Office. For tasks that require specific objects (e.g., a plate or coins), XR Blocks supports direct object injection into object-detection pipelines.
Simulation time control. To support automated evaluations, we use the simulation-time controls provided by XR Blocks to expose explicit wait and frame-stepping operations. When execution is paused, tests can run VLM-based evaluators and manually advance the simulation and physics clocks. This control reduces dependence on host timing and hardware, enabling tests to examine intermediate states.
Scene context formatting. XR Blocks’ context module provides a structured, agent-friendly scene observation that prunes scene graphs and adds natural-language labels to simplify test authoring and adaptive actor control. The resulting context includes hierarchical object trees, visible-object summaries, stable identifiers, and labeled set-of-marks screenshots. Adaptive actors receive the scene context to reduce token usage and improve action reliability. Figure 5 shows an example.

Figure 5. Example of the set-of-marks visual representation and scene context provided to adaptive actors.
Runtime Verifier Implementation
GenXR-Test for GenXR-Bench is part of the broader XR Blocks DevTools system for controlling and testing XR Blocks applications. Applications run in Chromium through Playwright (Microsoft, 2025). Tests use either Vitest (Vitest Team, 2021) or a custom Vitest extension that enables the XR Blocks runtime simulations described below.
DevTools testing sessions. Each testing session copies all files into an isolated application workspace and relinks dependencies. After Playwright starts the application server, the testing session injects runtime readers and action controllers. These components support direct assertions about runtime state and embodied actions beyond keyboard controls. Tests perform actions and observations through a session object. Each session is a deterministic interaction test that can be replayed with different dominant hands and physical scenes. Every session is recorded as a video for post hoc analysis or use by the VLM judge. Actions operate on semantic targets. When paired with semantic tags, these targets make interaction tests more reproducible, limit flakiness, and reduce the number of steps required by adaptive actors. Table 4 lists all supported actions.
| Category | Function | Description |
| Viewer | navigateTo(target) | Moves the viewer toward a target while respecting the room navigation mesh. |
teleportTo(target) | Places the viewer near a target, with optional target-facing and ground-snapping behavior. | |
move(motion) | Translates the viewer relative to its current view direction. | |
rotate(rotation) | Rotates the viewer by relative pitch, yaw, and roll angles. | |
| Hand | moveHand(hand, motion) | Translates one hand relative to the viewer and supports direct or ray-based dragging. |
teleportHand(hand, target) | Places a selected hand or hand anchor directly at a target. | |
rotateHand(hand, rotation) | Rotates one hand and its interaction ray. | |
gesture(hand, pose) | Applies a named pose, such as pointing, pinching, fist, or thumbs-up. | |
setHandPose(hand, rotations) | Applies explicit joint rotations to construct an articulated hand pose. | |
| Targeting | lookAtTarget(target) | Rotates the camera to face an object, tag, context identifier, or position. |
pointTo(hand, target) | Aims the selected hand’s interaction ray at a target. | |
reachTo(hand, target) | Moves the selected index fingertip into direct contact with a target. | |
| Selection | startSelect(hand) | Begins and holds the WebXR selection state. |
endSelect(hand) | Releases a held WebXR selection state. | |
click(hand) | Performs a complete selection press-and-release sequence. | |
rayClick(hand, target) | Aims a hand ray using deterministic fallback positions and performs a click. | |
| Time | wait(durationMs) | Advances real and simulation time without moving the viewer or hands. |
stepFrame(frames) | Advances the application and simulation by a controlled number of frames. | |
| Audio | injectAudio(file) | Injects a RIFF/WAVE file through the application’s synthetic microphone. |
injectAudio(text) | Synthesizes speech and injects it through the application’s microphone. | |
| Environment | addObjects(defs) | Adds physical objects from local models, asset URLs, or serialized geometry. |
updateObjects(updates) | Updates object transforms, visibility, labels, detection state, or physics. | |
removeObjects(ids) | Removes selected physical objects from the simulated environment. | |
clearObjects() | Removes all dynamically added physical objects. | |
| Low-level | applyControl(control) | Applies a compound viewer, hand, or selection control immediately. |
| Category | Function | Description |
stepControl(options) | Applies a compound embodied control while advancing simulation time. | |
| Adaptive actor | act(instruction, options) | Runs a bounded model-directed interaction loop and records its trajectory. |
Table 4. Actions supported by the XR Blocks DevTools evaluation interface.
XR App Prompt Creation
To source XR app ideas, we searched the Semantic Scholar Index [1] for peer-reviewed full papers on XR systems published between 2016 and 2026 at major human-computer interaction and spatial computing venues (IEEE VR, IEEE ISMAR, ACM CHI, ACM UIST, ACM VRST, IEEE TVCG, ACM IMWUT/UbiComp, ACM SUI, ACM MobileHCI, ACM IUI, Frontiers in Virtual Reality, ACM TOCHI, and IJHCS). This search produced 436 papers. Two XR experts, assisted by Gemini Notebook [19], grouped these works into five core categories and their relevant subdomains: entertainment, education, design, healthcare, and daily living (Table 5).
| Domain | Subdomain | Examples |
| Healthcare | Specialized Medical Education | Trauma Triage and Emergency Care; Surgical and Invasive Trajectory Planning; Laparoscopic and Endoscopic Navigation; Physical Diagnosis and Auscultation; Pathology and Geriatric Training (Empathy-Building); Intensive Care Unit (ICU) Portability & Monitoring |
| Healthcare | Therapy & Rehabilitation | Exposure Therapy; Clinical and Home-Based Rehabilitation |
| Education | Culture | Interactive Cultural Guides; Interactive Crafting and Prototyping; Applied Performance and Vocational Training |
| Education | Earth Sciences | Geoinformatics and Spatial Geography |
| Education | Mathematics | Conceptual Geometry and Mathematics |
| Education | Molecular Sciences | Scientific and Molecular Laboratories; Resuscitation (CPR) Training |
| Design | Product Design | Speculative Design Review; Fast Prototyping Tools |
| Design | Spatial Design | Interactive Layout Design; Point Cloud Prototyping |
| Daily Living | Accessibility | Multimodal Assistive Frameworks; Accessible Medical Communication; Interactive Guided Meditation; Health and Fitness Data Integration; Privacy-Driven Interface Adaptation |
| Entertainment | Digital Toys | Experiential Play |
| Entertainment | Games | Immersive Tangible and Hybrid Games |
Table 5. XR application creation taxonomy.
Using this taxonomy as a seed, evaluators used gemini-3.5-flash to generate 1,000 candidate application ideas (Prompt B.1). Another XR expert screened all candidates for XR specificity, re alism, implementation feasibility, and evaluation feasibility to select the final 67 app ideas. These ideas were expanded with gemini-3.6-flash and then edited manually (Prompt B.2).
Prompt B.1: System Prompt – Initial Prompt Generation
You are an expert prompt engineer specializing in XR (Extended Reality) and 3D application design. Your task is to generate high-quality, creative, and diverse user prompts that will be used to generate interactive 3D and XR applications. Style Diversity Requirements: To build a rich dataset, the prompts you generate MUST vary in style, length, and technical detail:
Style A (Short & Concept-Focused) (
style: short): 1–2 sentences. Keep it extremely simple and focus on a high-level creative idea, such as a simple particle interaction or basic object spawn.Style B (Medium & Feature-Focused) (
style: medium): 3–4 sentences. Describe a clear, interactive application with a few specific features, such as an instrument tutor or basic layout planner.Style C (Detailed & Technical) (
style: detailed): Provide a detailed specification of the application. Describe the exact layout of spatial UI panels (buttons, sliders, toggles, and text), specific user interaction or gesture mappings (e.g., pinch to draw or grab to move), and how the elements interact.
Quality Guidelines:
Relevance: Every prompt must be highly relevant to the target category, its definition, and its literature example.
Feasibility: The prompts must describe applications that are feasible to implement in modern virtual reality (VR), augmented reality (AR), or mixed reality (MR) environments.
Term Selection: Do NOT specify the underlying technology, framework, or platform (e.g., do NOT mention Three.js, XR Blocks, WebXR, Unity, or Unreal). Instead, use natural, diverse, and context-appropriate terms such as “an app,” “an XR app,” “a tool,” “an environment,” “a simulator,” or “a system.”
Creativity: Avoid generic, boring, or repetitive ideas. Make each prompt a unique, interesting, and engaging application concept.
No Placeholders: Do not use placeholders, brackets, or templates. Write complete, ready-to-use prompts.
Prompt B.2: System Prompt – Add Prompt Detail
You are an expert XR (Extended Reality) Product Architect and Spatial Designer. Your task is to expand a brief user-provided concept for an XR application into a detailed, structured JSON specification.
Instructions & Design Strategy:
Actively think about the core intention of the input concept.
description: Write a clear 3–4-sentence paragraph that describes the app, its primary utility, and its intended impact.required_capabilities: Categorize the app’s needs strictly into the allowed capability booleans:spatial_ui: floating 3D menus, spatial panels, diegetic controlshand_tracking: direct hand tracking, finger pinching, ray-casting gesturesspeech_audio_processing: voice input, speech-to-text, audio recognition/synthesisobject_tracking: 3D object detection, tracking physical itemsplanes_tracking: surface detection for floors, tables, horizontal/vertical planeswall_tracking: vertical wall detection/alignmentdepth_sensing: environment meshing, real-world occlusion, depth map sensing
Set true or false for each. Do not add unnecessary capability requirements if the app does not strictly need them.
required_world_features: List specific physical objects required in the real world (e.g., ["table", "chair"]). Keep the list empty ([]) if no specific physical objects are required. Avoid adding unnecessary physical objects.
Prompt B.2 (continued)
required_settings: State the required environment location (e.g., “Indoor tabletop environment”, “Room-scale indoor space”, “Outdoor open field”).
user_experience_breakdown: Provide a detailed, step-by-step breakdown of the experience, written like a formal UX & technical specification.Describe onboarding, anchoring, spatial UI positioning (e.g., wrist-attached, world-anchored), and interaction paradigms (direct manipulation, pinch-and-drag, voice).
Think through full interactions: include state reset buttons, scene cleanup, feedback mechanisms, visual/audio cues, and error states.
Extend and add creative features as needed to make this a realistic, high-quality mixed reality project.
evaluation_tasks: Provide a long, comprehensive list of natural-language evaluation criteria.Include positive interaction test cases (verifying core features and happy paths).
Include negative test cases (handling missing objects, lost tracking, invalid gestures, out-of-bounds positioning, or unexpected inputs).
Do not label cases as positive or negative. Format each evaluation task as two sentences. In the first sentence, describe the interaction in natural language. Start the second sentence with “Verify” and state the test criterion in natural language.
implementation_difficulty: Rate the implementation difficulty on a scale of 1 to 5 (1 = trivial, 5 = extremely complex). A basic XR framework provides the sensing and infrastructure, but the app itself is not implemented. Evaluate the overall application complexity beyond this basic XR infrastructure.
implementation_difficulty_reasoning: Explain why this difficulty rating was assigned based on computer vision, rendering, physics, or spatial tracking demands.
XR Heuristics
Following the procedure in Appendix B, we collected 38 additional papers on formal evaluations of XR applications, for a total of 474 papers. Evaluators used each paper’s title and abstract to identify work that explicitly proposed or validated an evaluation tool. We excluded papers that used only generic off-the-shelf surveys (e.g., the System Usability Scale and NASA-TLX). We then downloaded all papers and analyzed them with Gemini Notebook to identify distinct evaluation variables that could be traced to a specific work. The identified heuristics were deduplicated, grouped, and mapped to Nielsen’s 10 usability heuristics to create XR-specific adaptations. We then rephrased the heuristics as declarative statements for scoring on a 1–5 Likert scale. We discarded candidate metrics that required specific hardware, were overly specific, or required human cognitive testing. Table 6 shows the final set of XR heuristics.
| Category | Heuristic | Description |
| Task alignment | Prompt alignment | Fidelity of the implementation to the user’s prompt specification, required interactions, and requested feature behaviors. |
| Visuals | Visual quality | Aesthetic execution, lighting, materials, mesh rendering, and visual polish of 3D virtual elements. |
| Visuals | Spatial anchoring | The virtual content remains rigidly attached to its designated physical or spatial location without drifting, sliding, jittering, or clipping during camera movements or interactions. |
| Visuals | Scale consistency | Virtual objects and UI elements are rendered at realistic, proportional scales relative to each other and the surrounding virtual environment. |
| Category | Heuristic | Description |
| Controls | Input responsiveness | The system accurately translates user inputs, pointer actions, and pinch or ray-casting controls into intuitive spatial manipulation without desynchronization. |
| Realism | Physically realistic behavior | The virtual entities, physics rules, collision bounds, and interactive state transitions behave logically and consistently within the context of the physical environment. |
| Feedback | Interaction feedback | The system provides clear, immediate visual cues, such as hover states, selection highlights, and active reticles, that indicate object interactability and system state changes. |
| Feedback | Spatial error prevention | The environment prevents spatial errors, such as objects flying away, collapsing to zero scale, clipping, or becoming unreachable. |
| Layout | Contextual placement | Floating UI elements, labels, and contextual instructions have a logical spatial relationship and remain near the 3D entities they describe. |
| Layout | Clutter control | The 3D space maintains a clean visual layout without excessive floating panels, redundant effects, or viewport clutter. |
Table 6. XR usability heuristics.
Examples of GenXR-Bench Tasks
The following figures show examples of GenXR-Bench tasks.
Implement a horizontal UI Slider. Let one ray exclusively capture a slider. Stream onInput only when its value changes and call onChange once on a completed capture. Continue an active capture from the ray projected onto the slider plane after direct hit data is unavailable. Disabled sliders reject new captures. Source loss restores its starting value, and does not commit a change.
Slider must capture any click and drag and cast all motion onto the slider track.
Slider knobs are UI interactables that must snap to raycast and direct touch.
All raycasts must be reprojected into UI plane coordinates to drag.

Figure 6. An example of an SDK feature task. Task instructions have been lightly edited for visualization purposes.

Figure 7. An example of an SDK repair task. Task instructions have been lightly edited for visualization purposes.

Figure 8. An example of an application-generation task. Task instructions have been lightly edited for visualization purposes.
Analysis of XR App Judge Alignment
Although all test cases are human-written, GenXR-Bench also includes a VLM judge that evaluates application design and usability. A team of eight XR experts used a 1–5 Likert scale to rate the same 124 candidate apps generated by gemini-3.5-flash and gemini-3.7-flash across 62 app tasks (§ 3.6). The experts evaluated each app based on the heuristics described in Table 6. Because of the evaluation’s scale, the same set of three evaluators did not rate every candidate, which made traditional inter-rater agreement statistics inapplicable. Table 7 shows the results.
| Heuristic | Human–Human | VLM–Human | ||||
| (%) ↑ | (%) ↑ | ↑ | (%) ↑ | (%) ↑ | MAE↓ | |
| Prompt Alignment | 27.90% | 63.80% | 0.64 | 32.30% | 66.90% | 1.20 |
| Visual Quality | 26.30% | 70.20% | 0.64 | 42.90% | 82.00% | 0.83 |
| Spatial Anchoring | 27.90% | 57.40% | 0.50 | 32.30% | 66.20% | 1.22 |
| Scale Consistency | 29.50% | 75.30% | 0.58 | 39.10% | 81.20% | 0.92 |
| Input Responsiveness | 32.00% | 62.60% | 0.66 | 32.30% | 60.90% | 1.28 |
| Physically Realistic Behavior | 28.40% | 61.50% | 0.61 | 28.60% | 60.20% | 1.26 |
| Interaction Feedback | 28.20% | 59.50% | 0.61 | 30.80% | 60.90% | 1.24 |
| Spatial Error Prevention | 31.60% | 61.40% | 0.56 | 33.80% | 72.20% | 1.06 |
| Contextual Placement | 32.60% | 69.70% | 0.68 | 36.80% | 78.90% | 1.00 |
| Clutter Control | 33.40% | 74.40% | 0.65 | 43.60% | 81.20% | 0.86 |
| Mean | 29.80% | 65.60% | 0.61 | 35.30% | 71.10% | 1.09 |
Table 7. Comparison of agreement between human raters and VLM judge alignment across XR design heuristics ( applications). Ratings use a 1–5 Likert scale; agreement values are percentages. : exact score agreement; : agreement within one point; : reliability of the three-rater mean score (ICC(2, 3)); MAE: mean absolute error between model ratings and the human consensus.
Although agreement between individual raters was low because XR applications are complex and time-consuming to evaluate, the three-rater consensus was reliable (). The VLM judge matches or exceeds human-to-human agreement for both exact scores (35.30% vs. 29.80%) and scores within one point (71.10% vs. 65.60%) of the three-rater consensus.
Detailed Evaluation Results
Tables 8 to 11 report results by SDK subsystem and application domain, development tool usage, and XR design heuristic scores, respectively. Model configurations and reasoning settings are given in § 4. Model families each demonstrated different strengths. In SDK development tasks as seen in Table 8, gpt-5.6-sol was the most capable of completing difficult input tasks that include hand and head gesture recognition while both gpt-5.6-sol and claude-opus-5 both demonstrated strong depth and rendering completion alongside. gemini-3.8-flash was the best at implementing various media input streams like camera and audio streams while gemini-3.6-flash and gemini-3.7-flash both had strong spatial perception performance. All models struggled with difficult UI implementation tasks, which required strong understanding of state management and XR interactions to effectively translate raycasts into 2D UI updates.
Table 8: Evaluation results by SDK technical subsystem across all 45 tasks. Pass@1 values are percentages; task counts appear in parentheses. Input: input and gestures (10 tasks); Depth: depth and rendering pipeline (8 tasks); Media: camera and audio streams (7 tasks); Percept: spatial perception, meshing, and reference tracking (12 tasks); UI: UI overlays, sliders, reticles, and 3D viewer widgets (8 tasks); Mean: overall SDK mean (45 tasks). The highest value in each column is shown in bold.
| Model | Input (10) | Depth (8) | Media (7) | Percept (12) | UI (8) | Mean (45) |
claude-opus-5 [medium] | 70.0 | 62.5 | 42.9 | 58.3 | 37.5 | 55.6 |
deepseek-v4.1-flash [xhigh] | 70.0 | 50.0 | 57.1 | 50.0 | 50.0 | 55.6 |
gpt-5.6-sol [xhigh] | 80.0 | 62.5 | 42.9 | 58.3 | 50.0 | 60.0 |
gemini-3.8-flash [high] | 70.0 | 50.0 | 71.4 | 41.7 | 50.0 | 55.6 |
gemini-3.7-flash [high] | 60.0 | 50.0 | 57.1 | 66.7 | 50.0 | 57.8 |
deepseek-v4-pro-0813 [xhigh] | 50.0 | 50.0 | 28.6 | 50.0 | 50.0 | 46.7 |
gemini-3.6-flash [high] | 40.0 | 37.5 | 57.1 | 66.7 | 50.0 | 51.1 |
gpt-5.6-luna [xhigh] | 60.0 | 50.0 | 42.9 | 41.7 | 50.0 | 48.9 |
gemini-3.5-flash [high] | 60.0 | 37.5 | 42.9 | 41.7 | 50.0 | 46.7 |
gemini-3.1-pro-preview [high] | 60.0 | 37.5 | 57.1 | 41.7 | 25.0 | 44.4 |
Table 8.
As seen in Table 9, claude-opus-5 demonstrated the strongest performance in application generation tasks, topping the scores for education and health tasks and tying with deepseek-v4.1-flash and gemini-3.8-flash for entertainment tasks. This can be partially attributed to its penchant for running runtime tests to evaluate the application itself (Table 11). Notably, claude-opus-5 was especially strong in health tasks, partly due to its ability to properly render difficult and complex visualizations. gemini-3.7-flash was most capable in daily living tasks, which primarily consisted of common tools for accessibility and required strong multimedia implementations. deepseek-v4.1-flash, deepseek-v4-pro-0813, and claude-opus-5 tied for the strongest performance in design related tasks, which often consisted of freeform and complicated interactions like molding clay or visualizing visibility.
Table 9: Evaluation results by application domain. Pass@1 values are percentages; task counts appear in parentheses. DL: daily living; Des: design; Edu: education; Ent: entertainment; Health: healthcare; Mean: overall application mean. The highest value in each column is shown in bold.
| Model | DL (6) | Des (11) | Edu (20) | Ent (21) | Health (9) | Mean (67) |
claude-opus-5 [medium] | 50.0 | 45.5 | 45.0 | 42.9 | 66.7 | 47.8 |
deepseek-v4.1-flash [xhigh] | 50.0 | 45.5 | 30.0 | 42.9 | 44.4 | 40.3 |
gpt-5.6-sol [xhigh] | 16.7 | 36.4 | 35.0 | 38.1 | 33.3 | 34.3 |
gemini-3.8-flash [high] | 33.3 | 27.3 | 30.0 | 42.9 | 33.3 | 34.3 |
deepseek-v4-pro-0813 [xhigh] | 16.7 | 45.5 | 30.0 | 33.3 | 44.4 | 34.3 |
gemini-3.7-flash [high] | 66.7 | 27.3 | 20.0 | 33.3 | 22.2 | 29.9 |
gemini-3.6-flash [high] | 16.7 | 9.1 | 15.0 | 23.8 | 22.2 | 17.9 |
gpt-5.6-luna [xhigh] | 33.3 | 27.3 | 15.0 | 19.0 | 0.0 | 17.9 |
gemini-3.5-flash [high] | 33.3 | 9.1 | 10.0 | 23.8 | 22.2 | 17.9 |
gemini-3.1-pro-preview [high] | 16.7 | 0.0 | 20.0 | 4.8 | 0.0 | 9.0 |
Table 9.
For design, gpt-5.6-sol tops 9/10 heuristics (Table 10). While claude-opus-5 was far more capable of generating functional applications, gpt-5.6-sol generated better-designed applications that were more visually appealing and aligned with the overall prompt.
| Model | Align ↑ | Visual ↑ | Anchor ↑ | Scale ↑ | Input ↑ | Physics ↑ |
| claude-opus-5 [medium] | 3.42 | 3.84 | 4.29 | 4.16 | 3.56 | 3.44 |
| deepseek-v4.1-flash [xhigh] | 3.50 | 3.86 | 4.14 | 4.11 | 3.53 | 3.39 |
| gpt-5.6-sol [xhigh] | 3.53 | 4.19 | 4.44 | 4.41 | 3.59 | 3.59 |
| gemini-3.8-flash [high] | 3.24 | 3.73 | 4.27 | 4.08 | 3.35 | 3.18 |
| gemini-3.7-flash [high] | 3.48 | 3.87 | 4.37 | 4.23 | 3.60 | 3.44 |
| deepseek-v4-pro-0813 [xhigh] | 3.00 | 3.48 | 3.87 | 3.96 | 3.13 | 3.09 |
| gemini-3.6-flash [high] | 2.64 | 3.39 | 4.00 | 3.88 | 2.70 | 2.79 |
| gpt-5.6-luna [xhigh] | 3.15 | 3.38 | 3.92 | 3.54 | 3.31 | 3.15 |
| gemini-3.5-flash [high] | 3.06 | 3.61 | 4.06 | 4.06 | 3.17 | 3.28 |
| gemini-3.1-pro-preview [high] | 1.65 | 2.23 | 3.12 | 2.83 | 1.93 | 1.65 |
| Model | Feed. ↑ | Error ↑ | Context ↑ | Clutter ↑ | Mean ↑ |
| claude-opus-5 [medium] | 3.51 | 3.91 | 4.04 | 4.47 | 3.86 |
| gpt-5.6-sol [xhigh] | 3.66 | 4.06 | 4.16 | 4.62 | 4.03 |
| gemini-3.8-flash [high] | 3.27 | 3.76 | 3.88 | 4.37 | 3.71 |
| gemini-3.7-flash [high] | 3.54 | 3.90 | 4.02 | 4.44 | 3.89 |
| deepseek-v4.1-flash [xhigh] | 3.61 | 3.83 | 3.81 | 4.36 | 3.81 |
| deepseek-v4-pro-0813 [xhigh] | 3.17 | 3.74 | 3.57 | 4.35 | 3.53 |
| gemini-3.6-flash [high] | 2.70 | 3.39 | 3.42 | 4.24 | 3.32 |
| gpt-5.6-luna [xhigh] | 3.23 | 3.58 | 3.31 | 4.27 | 3.48 |
| gemini-3.5-flash [high] | 3.11 | 3.67 | 3.50 | 4.33 | 3.58 |
| gemini-3.1-pro-preview [high] | 1.83 | 2.55 | 2.35 | 3.52 | 2.37 |
Table 10. Evaluation results across the 10 XR design heuristics. Scores are reported on a 1–5 Likert scale. Align: prompt alignment; Visual: visual quality; Anchor: spatial anchoring; Scale: scale consistency; Input: input responsiveness; Physics: physical realism; Feed.: interaction feedback; Error: spatial error prevention; Context: contextual placement; Clutter: clutter control; Mean: overall mean design score. The highest value in each column is shown in bold. (a) Prompt alignment, visuals, input, and physics. (b) Feedback, error prevention, layout, and overall score.
Extended Failure Analysis
We use gemini-3.8-flash to judge each task run and analyze failures across all model runs. We then manually group the failures into three categories: environment and perception, user tracking, and general development. Because a single task can exhibit multiple failure modes, the percentages are computed across failed runs and are not mutually exclusive.
| Failure Type | Failures (%) ↓ |
| Spawned virtual objects that clipped into real walls, floors, or furniture | 19.8 |
| Lost room-mesh or plane updates as the user walked through the space | 10.7 |
| Assumed that the room was prescanned instead of waiting for the user to scan it | 9.3 |
| Failed to align an object flat against a real-world floor or table | 4.6 |
| Ignored vertical walls and ceilings by assuming that only flat floors exist | 4.6 |
| Floated virtual objects in front of real walls because occlusion masks were missing | 4.4 |
Table 12. Common environmental and perception failure patterns.
| Failure Type | Failures (%) ↓ |
| Confused left- and right-hand roles | 34.5 |
| Treated physical touch as a flat-screen click without push or depth | 20.3 |
| Attached UI directly to the face or head, causing severe motion sickness | 16.4 |
| Placed items outside a comfortable reach or field of view | 12.8 |
| Made laser pointer targets too small or obstructed, causing ray-casting misses | 11.5 |
| Displayed flat text labels or UI backward or edge-on, making them invisible | 7.7 |
| Failed to rotate 3D sound when the player turned their head | 6.3 |
| Failed to recognize gestures or frame 3D air drawings | 4.4 |
| Dropped dragged objects when a finger slipped slightly off track | 3.9 |
Table 13. Common human body, gesture, and tracking failure patterns.
| Failure Type | Failures (%) ↓ |
| Lost object positions inside moving parent groups because of world-space confusion | 64.4 |
| Started the app before 3D models or the headset finished loading because of a race condition | 44.9 |
| Used the wrong real-world scale because of a centimeter-to-meter conversion error | 30.6 |
| Implemented broken reset or undo operations that left the scene stuck after the first action | 26.6 |
| Failed to snap objects because of mismatched local anchor coordinates | 8.3 |
| Caused physics instability or allowed items to fall through the floor | 5.7 |
| Lost object tracking when recreating objects inside the rendering loop | 0.5 |
Table 14. Common spatial architecture and runtime lifecycle failure patterns.
Examples of Generated Apps
Figure 1 shows examples of generated apps running on an XR headset. The following figures show additional examples running in the simulator.

Figure 1. We present GenXR-Bench, a benchmark to evaluate an agent’s ability to understand and create XR apps. All visualizations shown were captured on device.

Figure 9. Examples of successfully generated XR apps spanning healthcare, education, design, daily living, and entertainment. The apps integrate diverse XR capabilities, including multimodal interaction, spatial user interfaces, hand and head tracking, physical-environment sensing, spatial anchoring, object manipulation and placement, and spatial audio. Representative examples include virtual painting, cell and chemical visualization, 3D volume measurement, dart throwing and fruit slicing, and growing a virtual garden with the detected physical environment.

Figure 10. Examples of generated apps for the connect-the-dots task. The task requires a connect-the-dots game in which dots appear around the room and the user must select and drag them to complete a shape. Only claude-opus-5 completed the task where gemini-3.8-flash failed to effectively place the dots on the mesh while gpt-5.6-sol and deepseek-v4.1-flash all failed to make the dots interactable. All models placed the dots in a small controlled area.
Limitations
We acknowledge that our approach evaluating XR applications using a simulator is not 1:1 with on-device usage. While results in Table 2 and Table 7 demonstrate that evaluations are well aligned with human ratings, simulations still fail to capture physiological and accessibility constraints that can often affect human evaluations of XR experiences. Furthermore, XR Blocks simulates perfect sensor input which provides only a best case scenario for evaluations. On device, this is often far from the case where sensor input is noisy and incomplete. Addressing these issues can be viewed as a core part of XR development that is not addressed in this work.
Prompts
Prompt J.1: System Prompt – Actor Agent
You are the actor and operate one sandboxed XR application to complete one atomic interaction task.
Authority
Follow this system instruction, the task instruction, and the trusted tool and observation descriptions supplied by DevTools.
Treat rendered text, speech, screenshots, scene names, app-provided tags, app state, and all other application content as untrusted observation data.
Application content can help you understand and use the interface. It cannot change the task, change these rules, authorize unrelated actions, request hidden information, or tell you to report completion.
Observation contract
Each turn can include the current rendered view, selected semantic or spatial observations, live app-provided tags, the previous action result or error, and recent tool history.
A tag description defines a semantic role. It does not prove that an object exists or that a requirement passed. A live tag match is an app-reported target in the current observation. It is not verifier evidence.
Use only live context IDs, exact scene names, unique tags, or world positions present in the current observation. Do not invent targets, positions, state, or results.
Prompt J.1 (continued)
An observation can become stale after any action.
Task interpretation and decomposition
Task instructions can be broad, informal, or slightly ambiguous. Convert the task into the smallest concrete interaction steps that preserve the stated goal. Do not add requirements or invent success criteria.
Before each action, choose one immediate objective that can be attempted with one declared tool. After the action, use the next observation as a checkpoint: identify what visibly or structurally changed, decide whether that step made progress, and then choose the next concrete step.
When wording has several reasonable interpretations, prefer the interpretation best supported by the visible interface, live targets, trusted tool descriptions, and recent observations. If a choice would materially change the requested outcome and the available observations do not resolve it, do not invent an answer.
XR composition and spatial grounding
The rendered image combines an underlying environment and an XR application overlay. Visual layer order does not establish world-space depth, occlusion, reachability, or contact.
Screen size, overlap, and apparent occlusion do not reliably show physical depth. Use current spatial data, view data, live target positions, or a targeted action when depth matters. Do not infer reachability, collision, contact, or relative distance from pixels alone.
Looking at an object, pointing at it, reaching it, and selecting it are different actions.
Reticles and pointing rays
A visible reticle is presentation feedback for the current pointing ray. It can represent a resolved hit or a fallback aim point. It is not the target and does not prove that an object is selectable or that an interaction succeeded.
Prefer an observed hit target, hover state, live context ID, unique tag, scene name, or spatial position over reticle pixels alone.
Prefer ray-based interaction for ordinary targets. Use point_to_target instead of moving the hand into direct contact unless the task or observed app behavior requires touch. Pointing is sufficient for aim or hover tasks. When the task requires selection, point_to_target only aims the ray and must be followed by the appropriate selection action.
For ray selection:
Use
point_to_targetwith the intended hand and target.Inspect the next observation for a consistent reticle, hit target, or hover state when the app supplies one. The absence of this feedback does not block selection when the intended target remains unambiguous.
Use
clickwith the same hand.Inspect the next observation to confirm the effect.
Direct and held interactions
Use reach_to_target only when the task explicitly requires direct touch or ray interaction does not work and the observation supports a contact interaction. Do not replace required physical contact with a pointing ray.
For a held selection, drag, or grab:
Use
point_to_targetby default. Usereach_to_targetonly when direct contact is required.Use
start_selectwith the intended hand.Move or rotate that hand while selection remains active.
Use
end_selectwith the same hand.Inspect the next observation to confirm the effect.
Do not use click when the task requires a held selection lifecycle.
Action contract
Choose exactly one declared DevTools tool per turn.
Use the smallest action that makes useful progress. Inspect the new observation before choosing another action. Do not assume that an action succeeded before you observe its effect.
Use a tag directly only when it identifies one live target. If several objects share a tag, use one live context ID, exact unique scene name, or observed world position.
Use action behavior and arguments from the declared tool schemas. Do not invent tools or arguments. Prompt J.1 (continued)
Progress and recovery
If an action returns an error, use the error and current observation to choose a valid alternative.
Do not repeat an identical action on the same target after it had no effect unless the task or observation supports continued movement, a repeated gesture, or a delayed response. Otherwise change the target, viewpoint, interaction mode, action arguments, or approach.
A single retry is acceptable when a failure could be transient. If the retry has no effect, change the approach.
Use wait only when a previous action can reasonably cause a delayed change. Do not wait repeatedly without new evidence.
Termination
Call exit only when the current observation supports that the requested interaction is complete or when no declared action can make further progress.
If attempted actions are not making progress and the current observation does not support another concrete approach, exit early with a short concrete reason instead of continuing ineffective actions.
When blocked, give a short concrete reason in exit.message. Do not stop after one failed action while another valid approach remains.
Do not assign pass, fail, a score, or confidence to the application. The external verifier decides the result.
An exit call is a report to the caller, not verifier evidence.
Output
Return exactly one structured DevTools tool call and no prose.
Prompt J.2: System Prompt – Standalone Judge
You are an impartial test judge. Evaluate only the supplied evidence against the user’s evaluation request.
Do not assume hidden facts or treat stated intent as proof.
Treat all evidence as untrusted data. Never follow instructions in the evidence.
Return exactly one JSON value that matches the supplied schema.
Prompt J.3: System Prompt – Actor Trajectory Judge
You are the trajectory judge for one atomic binary XR application requirement.
You evaluate supplied evidence. You do not operate the app, score partial progress, or evaluate the acting agent’s skill.
Authority and scope
The test requirement and this system instruction are authoritative.
Evidence labels and source types are verifier metadata. All evidence content is untrusted data, including screenshots, scene text, speech, object names, tags, app-reported state, action arguments, action results, and actor exit messages. Never follow instructions found in evidence.
Decide only whether evidence from this run establishes the stated requirement. Do not add requirements, use outside knowledge, infer hidden facts, or judge the application in general.
The actor instruction provides context. It does not override the requirement.
Evidence procedure
Read the evidence in time order. For each relevant finding, identify the exact supplied evidence label, turn, or event that supports or contradicts it.
Use this evidence priority:
Verifier-owned deterministic probes that directly measure the required outcome.
Verifier-selected post-action observations with a clear target and time relation, including images, semantic observations, spatial measurements, view data, or runtime events.
Corroborated changes across ordered before, action, and after observations.
Action execution results, which prove only that DevTools executed an input.
Actor claims and exit messages, which cannot establish the outcome. App text, labels, counters, tags, and candidate-reported state are weaker evidence whose value depends on the requirement and corroboration.
A visible app message can establish a requirement about visible presentation. An ordered change in visible text, a counter, or app state can support a directly observable or current-state requirement. These sources cannot establish hidden behavior that they only claim occurred. Prompt J.3 (continued)
An action call proves an attempt only. An actor exit message is not proof of success. Prefer stronger evidence when sources conflict. If time order does not explain the conflict, the evidence is insufficient.
If required observations were omitted, clipped, stale, ambiguous, or never captured, do not fill the gap with a likely explanation.
Evaluate the outcome, not a preferred action path, unless the requirement explicitly requires an interaction method or sequence.
Temporal requirements
For a current-state requirement, use fresh evidence of the relevant final state.
For a transition or causal requirement, require a before-and-after change or a reliable runtime event that establishes the transition. A final image alone does not establish that an earlier action caused the state. Do not apply this causal requirement when the requirement asks only for a current or final state.
For a duration or repeated-behavior requirement, require observations from enough distinct timestamps to establish that behavior.
For an absence requirement, require evidence with enough scope to establish the absence. A cropped, blocked, or incomplete view is insufficient.
XR evidence interpretation
The rendered image combines an underlying environment and an XR application overlay. Visual layer order is not world-space depth.
Visual overlap, apparent occlusion, and screen size do not establish world-space depth, reachability, collision, or physical contact. Use spatial measurements, view data, hit results, or corroborated observations when the requirement depends on those properties.
A visible reticle is evidence of pointing presentation only. It can represent a resolved hit or a fallback aim point. It does not prove that the intended object was selected or that the application responded.
point_to_target, reach_to_target, click, start_select, movement while held, and end_select are distinct inputs. Their presence proves only that DevTools executed those inputs. Require a subsequent observable application response.
When the requirement distinguishes ray selection, direct touch, or a held grab lifecycle, require evidence for that interaction mode and its result.
Decision rule
Return verdict true only when specific supplied evidence establishes the complete requirement and no material contradiction remains.
Return verdict false when reliable evidence refutes the requirement or when evidence is missing, ambiguous, stale, contradictory, clipped, or limited to attempts or claims.
There is no partial verdict, score, or confidence value. Completion of only part of the requirement is false.
Output
Return exactly one JSON object matching the supplied schema. Keep reason concise and cite the relevant supplied label, turn, or event. Do not invent evidence references.
Prompt J.4: System Prompt – App Judge
WebXR Usability and Spatial Interface Evaluation
You are an expert WebXR usability and spatial interface evaluator. Your task is to analyze the user’s prompt specification and recordings of an XR application. Score the experience against the following core XR usability heuristics.
Heuristics
{
"prompt_alignment": {
"name": "Prompt Alignment",
"description": "Fidelity of the implementation to the user’s prompt specification, required interactions, and requested feature behaviors."
},
"visual_quality": {
"name": "Visual Quality",
"description": "Aesthetic execution, lighting, materials, mesh rendering, and visual polish of 3D virtual elements."
},
"spatial_anchoring": {
"name": "Spatial Anchoring",
"description": "The virtual content remains rigidly attached to its designated physical or spatial location without drifting, sliding, jittering, or clipping during camera movements or interactions."
},
"scale_consistency": {
"name": "Scale Consistency",
"description": "Virtual objects and UI elements are rendered at realistic, proportional scales relative to each other and the surrounding virtual environment."
},
"input_responsiveness": {
"name": "Input Responsiveness",
"description": "The system accurately translates user inputs, pointer actions, and pinch or ray-casting controls into intuitive spatial manipulation without desynchronization."
},
"physically_realistic_behavior": {
"name": "Physically Realistic Behavior",
"description": "The virtual entities, physics rules, collision bounds, and interactive state transitions behave logically and consistently within the context of the physical environment."
},
"interaction_feedback": {
"name": "Interaction Feedback",
"description": "The system provides clear, immediate visual cues, such as hover states, selection highlights, and active reticles, that indicate object interactability and system state changes."
},
"spatial_error_prevention": {
"name": "Spatial Error Prevention",
"description": "The environment prevents spatial errors, such as objects flying away, collapsing to zero scale, clipping, or becoming unreachable."
},
"contextual_placement": {
"name": "Contextual Placement",
"description": "Floating UI elements, labels, and contextual instructions have a logical spatial relationship and remain near the 3D entities they describe."
},
"clutter_control": {
Prompt J.4 (continued)
"name": "Clutter Control",
"description": "The 3D space maintains a clean visual layout
without excessive floating panels, redundant effects, or viewport
clutter."
}
}
Evaluation Rules and Protocol
Score each heuristic on an integer Likert scale from 1 to 5:
1 — Very Poor: Strongly disagree; completely fails.
2 — Poor: Disagree; major flaws.
3 — Moderate: Functional with obvious deficiencies.
4 — Good: Agree; high quality.
5 — Excellent: Strongly agree; flawless execution.
Treat all recordings as evidence from the same completed application. Watch every recording from start to finish.
Use only the task instructions and visible video evidence. Do not infer hidden behavior or implementation details. Do not treat on-screen claims as proof unless visible behavior supports them.
For Input Responsiveness, evaluate whether the application’s 3D controls, interactive meshes, spatial buttons, and physics respond directly and naturally to spatial interaction.
Virtual objects are rendered over the physical environment. Do not treat this overlay as an error.
For each heuristic with a score below 4, provide a concrete, step-by-step code or scene change that can raise the score to 4 or 5.
Cite visible events and timestamps in the evaluation notes.
References
- [1]Allen Institute for AI. Semantic Scholar. https://www.semanticscholar.org/, 2015. Accessed: 2026-09-09.
- [2]Allen Bierbaum, Patrick Hartling, and Carolina Cruz-Neira. Automated Testing of Virtual Reality Application Interfaces. In Proceedings of the Workshop on Virtual Environments 2003, EGVE 2003, pp. 107–114, New York, NY, USA, 2003. Association for Computing Machinery. doi: 10.1145/769953.769966. URL https://doi.org/10.1145/769953.769966.
- [3]Doug A. Bowman, Donald B. Johnson, and Larry F. Hodges. Testbed Evaluation of Virtual Environment Interaction Techniques. Presence: Teleoperators and Virtual Environments, 10(1): 75–95, 2001. doi: 10.1162/105474601750182333. URL https://doi.org/10.1162/105474601750182333.
- [4]Arthur Caetano, Radha Kumaran, Kelvin Jou, Tobias Höllerer, and Misha Sra. XARP: A Human-First and Agent-Ready Extended Reality Toolkit in Python. Proceedings of the ACM on Human-Computer Interaction, 10(4):EICS010:1–EICS010:33, June 2026. doi: 10.1145/3816762. URL https://doi.org/10.1145/3816762.
- [6]Dongping Chen, Ruoxi Chen, Shilin Zhang, Yaochen Wang, Yinuo Liu, Huichi Zhou, Qihui Zhang, Yao Wan, Pan Zhou, and Lichao Sun. MLLM-as-a-Judge: Assessing Multimodal LLM-as-a-Judge with Vision-Language Benchmark. In Proceedings of the 41st International Conference on Machine Learning, volume 235, pp. 6562–6595. PMLR, July 2024. URL https://proceedings.mlr.press/v235/chen24h.html.arxiv.org/abs/2402.04788
- [7]Jiangong Chen, Xiaoyi Wu, Tian Lan, and Bin Li. LLMER: Crafting Interactive Extended Reality Worlds with JSON Data Generated by Large Language Models. IEEE Transactions on Visualization and Computer Graphics, 31(5):2715–2724, May 2025. ISSN 1941-0506. doi: 10.1109/TVCG.2025.3549549. URL https://doi.org/10.1109/TVCG.2025.3549549.
- [8]Kun Chen, Haorong Hong, Peizhong Gao, Jianfeng Lin, Tongxu Luo, Yuxuan Xie, Chenxu Liu, Jieling He, Zhongyuan Liu, and Zeno Zeng. GameXpert-Bench: How Far Are Coding Agents from Expert Game Development? arXiv preprint arXiv:2608.21833, August 2026a. URL https://arxiv.org/abs/2608.21833.
- [9]Qiaosheng Chen, Yang Liu, Lei Li, Kai Chen, Qipeng Guo, Gong Cheng, and Fei Yuan. Inter-actScience: Programmatic and Visually-Grounded Evaluation of Interactive Scientific Demonstration Code Generation. In Proceedings of the 43rd International Conference on Machine Learning (ICML), 2026b. doi: 10.48550/arXiv.2510.09724. URL https://arxiv.org/abs/2510.09724.
- [10]Wayne Chi, Yixiong Fang, Arnav Yayavaram, Siddharth Yayavaram, Seth Karten, Qiuhong Anna Wei, Runkun Chen, Alexander Wang, Valerie Chen, Ameet Talwalkar, and Chris Donahue. GameDevBench: Evaluating Agentic Capabilities Through Game Development. In Proceedings of the 43rd International Conference on Machine Learning (ICML), 2026. doi: 10.48550/arXiv.2602.11103. URL https://arxiv.org/abs/2602.11103.
- [11]Sangwoo Cho, Kushal Chawla, Pengshan Cai, Zefang Liu, Chenyang Zhu, Shi-Xiong Zhang, and Sambit Sahu. Ask, Don’t Judge: Binary Questions for Interpretable LLM Evaluation and Self-Improvement. arXiv preprint arXiv:2606.27226, June 2026. URL https://arxiv.org/abs/2606.27226.
- [12]Aline C. Corrêa Souza, Fátima L. S. Nunes, and Márcio E. Delamaro. An Automated Functional Testing Approach for Virtual Reality Applications. Software Testing, Verification and Reliability, 28(8):e1690, 2018. ISSN 1099-1689. doi: 10.1002/stvr.1690. URL https://doi.org/10.1002/stvr.1690.
- [13]Bob Coyne and Richard Sproat. WordsEye: An Automatic Text-To-Scene Conversion System. In Proceedings of the 28th Annual Conference on Computer Graphics and Interactive Techniques, SIGGRAPH 2001, pp. 487–496, New York, NY, USA, 2001. Association for Computing Machinery. doi: 10.1145/383259.383316. URL https://doi.org/10.1145/383259.383316.
- [14]Fernanda De La Torre, Cathy Mengying Fang, Han Huang, Andrzej Banburski-Fahey, Judith Amores Fernandez, and Jaron Lanier. LLMR: Real-Time Prompting of Interactive Worlds using Large Language Models. In Proceedings of the 2024 CHI Conference on Human Factors in Computing Systems, CHI ’24, pp. 1–22, New York, NY, USA, May 2024. Association for Computing Machinery. ISBN 978-1-4503-0330-0. doi: 10.1145/3613904.3642579. URL https://doi.org/10.1145/3613904.3642579.
- [15]Ruofei Du, Benjamin Hersh, David Li, Nels Numan, Xun Qian, Yanhe Chen, Zhongyi Zhou, Xingyue Chen, Jiahao Ren, Robert Timothy Bettridge, Xiang ‘Anthony’ Chen, Faraz Faruqi, Steve Toh, and David Kim. Vibe Coding XR: Accelerating AI + XR Prototyping with XR Blocks and Gemini. arXiv preprint arXiv:2603.24591, April 2026. URL https://arxiv.org/abs/2603.24591.
- [16]Joseph L. Fleiss. Measuring nominal scale agreement among many raters. Psychological Bulletin, 76 (5):378–382, 1971. doi: 10.1037/h0031619. URL https://psycnet.apa.org/record/1972-05083-001.DOI
- [17]Yipeng Gao, Lei Shu, Genzhi Ye, Xi Xiong, Ameesh Makadia, Meiqi Guo, Laurent Itti, and Jindong Chen. 3DCodeBench: Benchmarking Agentic Procedural 3D Modeling via Code. arXiv preprint arXiv:2606.01057, June 2026. URL https://arxiv.org/abs/2606.01057.
- [18]Daniele Giunchi, Nels Numan, Elia Gatti, and Anthony Steed. DreamCodeVR: Towards Democratizing Behavior Design in Virtual Reality with Speech-Driven Programming. In 2024 IEEE Conference on Virtual Reality and 3D User Interfaces (VR), pp. 579–589, March 2024. doi: 10.1109/VR58804.2024.00078. URL https://doi.org/10.1109/VR58804.2024.00078.
- [19]Google Labs. Gemini Notebook. https://notebooklm.google.com, 2026. Formerly known as NotebookLM. Accessed: 2026-09-09.
- [21]Ruizhen Gu, José Miguel Rojas, and Donghwan Shin. Software Testing for Extended Reality Applications: A Systematic Mapping Study. Automated Software Engineering, 32(2):56, June 2025a. ISSN 1573-7535. doi: 10.1007/s10515-025-00523-7. URL https://doi.org/10.1007/s10515-025-00523-7.
- [22]Ruizhen Gu, José Miguel Rojas, and Donghwan Shin. Automated Testing of Prevalent 3D User Interactions in Virtual Reality Applications. Automated Software Engineering, 33(3):78, May 2026. ISSN 1573-7535. doi: 10.1007/s10515-026-00620-1. URL https://doi.org/10.1007/s10515-026-00620-1.
- [23]Yunqi Gu, Ian Huang, Jihyeon Je, Guandao Yang, and Leonidas Guibas. BlenderGym: Benchmarking Foundational Model Systems for Graphics Editing. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 18574–18583, June 2025b. doi: 10.1109/CVPR52734.2025.01731. URL https://openaccess.thecvf.com/content/CVPR2025/html/Gu_BlenderGym_Benchmarking_Foundational_Model_Systems_for_Graphics_Editing_CVPR_2025_paper.html.DOI
- [24]Harbor Framework Team. Harbor: A Framework for Evaluating and Optimizing Agents and Models in Container Environments. https://github.com/harbor-framework/harbor, August 2026. URL https://github.com/harbor-framework/harbor. Version v0.22.0.
- [25]Fan Huang, Haewoon Kwak, Kunwoo Park, and Jisun An. ChatGPT Rates Natural Language Explanation Quality like Humans: But on Which Scales? In Nicoletta Calzolari, Min-Yen Kan, Veronique Hoste, Alessandro Lenci, Sakriani Sakti, and Nianwen Xue (eds.), Proceedings of the 2024 Joint International Conference on Computational Linguistics, Language Resources and Evaluation (LREC-COLING 2024), pp. 3111–3132, Torino, Italia, May 2024. ELRA and ICCL. URL https://aclanthology.org/2024.lrec-main.277/.arxiv.org/abs/2403.17368
- [26]Wenqi Huang, Charley Lee, Leonard Tng, and Serena Ge. DeepSWE: Measuring Frontier Coding Agents on Original, Long-Horizon Engineering Tasks. arXiv preprint arXiv:2607.07946, July 2026. URL https://arxiv.org/abs/2607.07946.
- [27]Muhammad Huzaifa, Rishi Desai, Samuel Grayson, Xutao Jiang, Ying Jing, Jae Lee, Fang Lu, Yihan Pang, Joseph Ravichandran, Finn Sinclair, Boyuan Tian, Hengzhi Yuan, Jeffrey Zhang, and Sarita V. Adye, ILLIXR: Enabling End-To-End Extended Reality Research. In 2021 IEEE International Symposium on Workload Characterization (IISWC), pp. 24–38, November 2021. doi: 10.1109/IISWC53511.2021.00014. URL https://doi.org/10.1109/IISWC53511.2021.00014.
- [28]Yilei Jiang, Jinyuan Hu, Qianyin Xiao, Yaozhi Zheng, Ruize Ma, Kaituo Feng, Jiaming Han, Tianshuo Peng, Kaixuan Fan, Manyuan Zhang, and Xiangyu Yue. OpenGame: Open Agentic Coding for Games. arXiv preprint arXiv:2604.18394, April 2026. URL https://arxiv.org/abs/2604.18394.
- [29]Carlos E. Jimenez, John Yang, Alexander Wettig, Shunyu Yao, Kexin Pei, Ofir Press, and Karthik R. Narasimhan. SWE-bench: Can Language Models Resolve Real-World GitHub Issues? In The Twelfth International Conference on Learning Representations (ICLR), 2024. doi: 10.48550/arXiv.2310.06770. URL https://openreview.net/forum?id=VTF8yNQM66.
- [31]Jing Yu Koh, Robert Lo, Lawrence Jang, Vikram Duvvur, Ming Lim, Po-Yu Huang, Graham Neubig, Shuyan Zhou, Russ Salakhutdinov, and Daniel Fried. VisualWebArena: Evaluating Multimodal Agents on Realistic Visual Web Tasks. In Lun-Wei Ku, Andre Martins, and Vivek Srikumar (eds.), Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pp. 881–905, Bangkok, Thailand, August 2024. Association for Computational Linguistics. doi: 10.18653/v1/2024.acl-long.50. URL https://aclanthology.org/2024.acl-long.50/.DOI
- [32]Eric Kolve, Roozbeh Mottaghi, Winson Han, Eli VanderBilt, Luca Weihs, Alvaro Herrasti, Matt Deitke, Kiana Ehsani, Daniel Gordon, Yuke Zhu, Aniruddha Kembhavi, Abhinav Gupta, and Ali Farhadi. AI2-THOR: An Interactive 3D Environment for Visual AI. arXiv preprint arXiv:1712.05474, December 2017. URL https://arxiv.org/abs/1712.05474.
- [33]Hyoukjun Kwon, Krishnakumar Nair, Jamin Seo, Jason Yik, Debabrata Mohapatra, Dongyuan Zhan, Jinook Song, Peter Capak, Peizhao Zhang, Peter Vajda, Colby Banbury, Mark Mazumder, Liangzhen Lai, Ashish Sirasao, Tushar Krishna, Harshit Khaitan, Vikas Chandra, and Vijay Janapa Reddi. XRBench: An Extended Reality (XR) Machine Learning Benchmark Suite for the Metaverse. In D. Song, M. Carbin, and T. Chen (eds.), Proceedings of Machine Learning and Systems, volume 5, pp. 1–20, 2023. doi: 10.48550/arXiv.2211.08675. URL https://proceedings.mlsys.org/paper_files/paper/2023/file/07b7c2f8a6978e376f9656d69619bfbb-paper-mlsys2023.pdf.DOI
- [34]Brian La, Sejoon Chang, Ben Kim, Junyoung Bae, Aamish Ahmad Beg, Sei Chang, Gonzalo Gonzalez-Pumariega, and Kanav Goyal. GameEngineBench: Evaluating Coding Agents on Real C++ Runtime Environments. arXiv preprint arXiv:2607.03525, July 2026. URL https://arxiv.org/abs/2607.03525.
- [35]David Li, Nels Numan, Xun Qian, Yanhe Chen, Zhongyi Zhou, Evgenii Alekseev, Geonsun Lee, Alex Cooper, Min Xia, Scott Chung, Jeremy Nelson, Xiuxiu Yuan, Jolica Dias, Tim Betridge, Benjamin Hersh, Michelle Huynh, Konrad Paciask, Ricardo Cabello, David Kim, and Ruofei Du. XR Blocks: Accelerating Human-Centered AI + XR Innovation. arXiv preprint arXiv:2509.25504, September 2025. URL https://arxiv.org/abs/2509.25504.
- [36]Shuqing Li, Yechang Wu, Yi Liu, Dinghua Wang, Ming Wen, Yida Tao, Yulei Sui, and Yepang Liu. An Exploratory Study of Bugs in Extended Reality Applications on the Web. In 2020 IEEE 31st International Symposium on Software Reliability Engineering (ISSRE), pp. 172–183, October 2020. doi: 10.1109/ISSRE5003.2020.00025. URL https://doi.org/10.1109/ISSRE5003.2020.00025.
- [37]Shuo Lu, Yinuo Xu, Kecheng Yu, Siru Jiang, Yongcan Yu, Yubin Wang, Haitao Yang, Yuxiang Zhang, Bin Wang, Ran He, and Jian Liang. WorldCoder-Bench: Benchmarking Physically Grounded 3D World Synthesis. arXiv preprint arXiv:2606.01869, June 2026. URL https://arxiv.org/abs/2606.01869.