WorksComputer Science
3D GameDev Arena: Benchmarking Coding Agents That Plan, Build, and Playtest extreme Long-Horizon Browser Games
Abstract
Coding agents now resolve repository issues and generate compact playable games, but whether they can build a large 3D game from scratch remains open. Such games tightly couple mechanics, scenes, assets, and interaction logic, and their quality rests on player enjoyment, which existing benchmarks rarely assess. Thus, we introduce 3D GameDev Arena, a first extreme long-horizon benchmark of 108 from-scratch game-development tasks across 14 genres. Given only a text specification and UI/UX reference images, a fully automated plan–implement–test–repair pipeline builds each task into a self-contained browser game. Runs average 1,000 LLM calls ; the longest uses 14,407 calls, 2.13B tokens, and 608 assets. Evaluation spans (1) objective code-level functional acceptance against the game specification, (2) agent-based QA and playtesting of 3D presentation and playability, and (3) blind pairwise Bradley-Terry human preference in a public arena. The preliminary findings indicate that long-horizon 3D game development remains far from solved, and code-level correctness alone does not dictate player preference. The public arena is hosted online.
Introduction
Coding agents have climbed a steep ladder of software engineering tasks: they resolve real GitHub issues in existing repositories [17, 22], execute complex terminal workflows [31], and sustain autonomous development of full applications over multiple days [42]. Building games is the next frontier. Recent benchmarks show that agents can turn a prompt into a compact playable game [9, 10, 28, 47]. But compiling cleanly and running without crashes is only the baseline: the ultimate measure of a game is whether it is fun. This raises a sharper question:
Can coding agents build a large 3D game from scratch, and will the resulting game actually be fun to play?
Answering this question requires a benchmark that re-
flects real 3D game development, yet existing benchmarks fall short in three ways (Table 1). First, real development builds a 3D game from scratch. An agent should start from an empty folder, design its own codebase, and organize dozens of 3D assets into a working world. Instead, prior benchmarks hand agents ready-made templates [13, 46], pre-loaded 2D asset packs [28], simple 3D shapes, built-in test hooks [10, 13, 47], or even a working reference game [9, 10]. Second, real 3D development takes hours and about a thousand tool calls. Without templates or starter assets, building and debugging a playable 3D game is a long-horizon process. Yet existing benchmarks cap a run at one to two hours or at a few hundred steps (Table 1), leaving this scale untested. Third, games should be fun to play. A game can pass every unit test and still feel boring; whether it is fun is subjective and should be judged by human preference. Yet existing benchmarks rely mainly on code checks, rule checks, and model judges, and rarely measure human preference.
| Benchmark | 3D/Web | Input | Starter code | Run length | Evaluation |
| GameCraft [28] | ×/× | T,A | NR | 128 tool calls; 2 h cap | Replay + VLM rubric |
| GameXpert [9] | ✓/✓ | T,S | No (Gen); Yes (Fix/Opt) | NR | Runtime tests, agent judge, human ratings |
| GameASG [47] | ✓/✓ | T | No | 120-turn cap | Static + executable browser checks |
| GameGen-Verifier [20] | —/✓ | T | NR | NR | State injection, VLM, assertions |
| WebGameBench [46] | —/✓ | T | Template | 20–24 turns; 2 h cap | Browser agent judge |
| SWE-Game [10] | ✓/× | T,A,V,S | Varies by mode | 57–111 tool calls | Evaluator checks, reference comparison, VLM rubric |
| CEGame-Bench [13] | ✓/× | T,A | Starter project | ≤295 steps; 1 h cap | Hidden checks, player journey, VLM judge |
| 3D GameDev Arena | ✓/✓ | T,I | No | 692–1,067 tool calls; 2.6–7.9 h | Code tests; agent playtesting; human preference |
Table 1. Comparison with game-generation benchmarks: task inputs, the length of an agent run, and evaluation.
Building such a benchmark platform poses its own practical challenges. First, tasks must specify rich, diverse 3D worlds clearly while leaving agents free to build from an empty folder. Second, because every agent invents its own codebase, automated evaluation cannot assume interfaces fixed in advance: rule checks have to be adapted to each game’s code after it is built, and 3D visuals and control feel have to be judged from the game as it actually runs in a browser. Third, judging whether a game is genuinely fun requires running live, blind head-to-head playtests and turning noisy human votes into reliable rankings.
To address these gaps, we introduce 3D GameDev Arena (Figure 1), in which a fully automated pipeline carries an agent from planning through implementation, build, and autonomous playtest to a self-contained browser game. Our contributions are:

Figure 1. Overview of 3D GameDev Arena. A: a harness–model pair turns the task specification into a browser game. The resulting candidate is then evaluated by two independent tracks that share three steps: freeze the standard before any candidate is seen, capture candidate-adapted evidence, and score it. B: source-adapted tests check non-visual rules (§ 4.1). C: recorded browser play is judged for 3D presentation and playability (§ 4.2). The task note, harness–model pairs, and subitem rows are illustrative (Appendix § 12).
A from-scratch, long-horizon 3D benchmark. 108 tasks across 14 genres and two difficulty tiers, each a vertical slice of a popular game, given as a text specification with reference images. Agents get no starter code, bundled assets, or test hooks, and must find or generate their own
3D assets; a median build takes 2.6–7.9 hours and 692–1,067 tool calls (§ 3).
Three-level evaluation, from code to player. Functional tests keep the rubric fixed and adapt the test code to each game, so no hooks are needed during development; agent playtesting scores 3D presentation and playability from recorded browser play; blind pairwise human votes give a Bradley–Terry preference score (§ 4).
Findings on what makes a preferred game. The best of four configurations scores 64.7/100 functionally, and two configurations tied on functional score differ more than twofold in human preference. Development logs show that agents rarely lose track of the goal; the preferred ones test the running game, not just its code (§ 5–§ 6).
Related Work
Game Generation Benchmarks. Game benchmarks for coding agents range from single-file 2D Pygame programs [45] and localized edits inside existing Godot and Unreal projects [6, 11, 24] to end-to-end generation of whole games across engines and web runtimes [8, 21, 29, 36, 47]. Compared with these (Table 1; Appendix § 9.§ 1), 3D GameDev Arena asks for full 3D games whose 3D models the agent must find or generate itself, starts from an empty, stack-agnostic workspace with no template or test interface, and adds blind human preference to rule compliance.
Evaluating Generated Games. Prior work checks generated games with white-box state and assertion tests [6, 47] or with vision- and GUI-based playtesting [18, 45]; pairwise human preference with Bradley–Terry rankings [3, 12, 38] is standard for text and web generation but rare for 3D games. We combine all three (Appendix § 9.§ 2).
Game Generation Agents. Systems range from multi-agent frameworks for lightweight 2D games [16, 35] and coding agents with GUI playtesting [21, 34] to Unity and Unreal pipelines for 3D [14, 44]. 3D GameDev Arena is tied to none of them and evaluates any coding agent (Appendix § 9.§ 3).
Benchmark
Tasks are drawn from widely played, top-charting commercial titles on Steam, online PC platforms, and console storefronts. A full commercial production, especially an AAA release, has sprawling progression systems and tens of hours of content, so we cut each large title into one to three self-contained vertical slices. A slice isolates one signature gameplay mode at full mechanical depth: an action role-playing title inspired by Monster Hunter becomes a single boss-arena fight, and an arcade racer inspired by Crazyracing KartRider a single multi-lap circuit.
Benchmark Construction
To turn a slice into an unambiguous engineering prompt that keeps the intended experience, we write its specification on Guardiola’s Gameplay Loop model [15], which describes a game as recurring cycles of player objectives, actions, state transitions, and feedback. Every specification runs 2,500–5,100 words and has the same five sections (§ 3.2).
Coverage by Genre and Difficulty. The benchmark covers 14 canonical game genres at two implementation-difficulty tiers (Figure 2). Medium tasks are mostly 2D, 2.5D, and 3D games with bounded spatial dynamics and simple asset pipelines. Hard tasks are all 3D vertical slices modeled on commercial and AAA productions; they demand complex spatial reasoning, skeletal and structural mesh integration, camera control, and real-time 3D physics. Every genre has at least three distinct tasks in each tier.

Figure 2. Task distribution. The inner ring gives the number of tasks in each of the 14 genres; the outer ring splits each genre into Medium and Hard tasks.
Task Specifications
The five sections are (1) Role and Objective, (2) Game Overview and Player Experience, (3) Project Setup and Deliverables, (4) Complete Game Design, and (5) Acceptance Criteria; the fifth is written by a separate agent from the other four (Appendix § 10.§ 1). Complete Game Design ties player roles, camera perspectives, and input bindings to state variables, resource economies, entity interactions, the core gameplay loop [15], and terminal win, loss, and retry transitions. Table 8 (Appendix § 10) shows the five sections for two representative 3D tasks, a MOBA lane duel inspired by League of Legends and an arcade racing circuit inspired by Crazyracing KartRider.
| Specification Section | Formal Purpose | Instantiation A: MOBA Slice ( Nexus Duel) | Instantiation B: Arcade Racing Slice (Town Highway) |
| Role and Objective | Defines the target software artifact, player/AI configuration, and functional task scope. | Construct a playable 3D browser MOBA duel pitting one human-controlled champion against an autonomous AI opponent, featuring six selectable champions and terminating upon Nexus destruction. | Construct a single-player 3D arcade kart-racing adaptation featuring the Town Highway circuit, three selectable drivers, a Cotton PRO kart, and a two-lap race against three AI competitors. |
| Game Overview and Player Experience | Articulates the intended player experience, core loop pacing, mechanical highlights, and visual direction. | Deliver an 8–15-minute tactical lane duel centered on minion last-hitting, skillshot evasion, damage trading, itemization, and wave-assisted structure pushing. | Deliver a 2–3-minute race set in a vibrant cartoon town, emphasizing corner entry, drift initiation, counter-steering, grip recovery, and strategic nitro-booster overtaking. |
| Project Setup and Deliverables | Establishes engine/rendering constraints, build contracts, and self-contained packaging rules. | Deliver complete source code and a root-level build.sh script compiling an entry index.html, with all runtime assets bundled locally offline. | Deliver complete source code and a root-level build.sh compiling index.html with all runtime assets bundled locally. |
| Complete Game Design | Formalizes controls, state variables, numerical parameters, state transitions, and termination rules. | The Nexus holds 2,500 HP and remains invulnerable while its defensive lane turret stands. Champion eliminations create wave-pushing windows rather than immediate victory. | Sustained moving drifts charge a gauge; reaching 100 grants one booster (up to two stored slots). Pressing Ctrl consumes one booster only when no boost is currently active. |
| Acceptance Criteria | Translates design rules into deterministic, reproducible checks with observable runtime outcomes. | Attacking the Nexus while its turret is intact must leave Nexus HP unchanged; destroying the turret and subsequently reducing Nexus HP to zero must trigger the victory state while logging all HP deltas and state transitions. | Verify that stationary steering yields zero gauge charge. Execute valid drifts to earn and trigger a booster; stored count must decrement by one and kart velocity must increase, while repeated Ctrl inputs during active boost must neither consume extra stock nor extend boost duration. |
Table 8. Canonical five-part structure of benchmark task specifications, illustrated with two representative Hard-tier 3D vertical slices: a MOBA lane duel (1v1 Nexus Duel, modeled after League of Legends) and an arcade kart racer (Town Highway Circuit, modeled after Crazyracing KartRider).
Generation protocol
An agent starts from an empty workspace with a free choice of technology stack and only an index.html entry requirement. There are no test hooks: the specification prescribes no state-reporting interface and no place where tests must go. Every game is built in two stages, planning and execution (Figure 1A), by a coding agent augmented with Agent Skills [1, 25, 40, 41]: packages of workflows, verification scripts, and domain constraints loaded on demand. In the planning stage, a lead agent uses architectural-planning and game-orchestration skills [30, 39] to inspect the workspace contract and write an implementation blueprint covering core mechanics, state organization, controls, and 3D asset sourcing; no assets are supplied, so agents search the web for 3D models and generate missing ones with Meshy [32]. In the execution stage, an execution agent turns this blueprint into the playable deliverable with subagent-driven development and interactive browser-verification skills [33, 39], implementing game systems and playtesting until all acceptance criteria are met. Appendix § 15 gives the skill configurations, the 3D asset pipeline, and the exact prompts.
Evaluated configurations. We evaluate four configurations, each pairing a native coding-agent harness with one model at the highest reasoning setting of that harness: GPT-6.1 Sol and GPT-6 Astra in Codex CLI, GLM-5.3 in Zcode, and Kimi K3 in Kimi CLI (Table 13, Appendix § 15). Configuration IDs follow a harness–model pattern, such as codex-gpt-6.1-sol; because each model runs in exactly one harness, we refer to a configuration by its short name, such as gpt-6.1-sol.
| Configuration ID | Short name | Harness | Model |
| codex-gpt-6.1-sol | gpt-6.1-sol | Codex CLI 0.167.0 | GPT-6.1 Sol |
| codex-gpt-6-astra | gpt-6-astra | Codex CLI 0.167.0 | GPT-6 Astra |
| zcode-glm-5.3 | glm-5.3 | Zcode 0.16.9 | GLM-5.3 |
| kimi-k3 | kimi-k3 | Kimi CLI 2.6.1 | Kimi K3 |
Table 13. Evaluated configurations. Short names are used in the text and in Table 4–Table 6.
T: text; S: source; A: assets; I: images; V: reference video. Inputs span task modes. 3D/Web: ✓ yes, × no, — unreported. Starter code means a supplied game implementation; evaluator-only references do not count. NR: not reported. Template: workspace scaffold, contents unspecified. Starter project: engine project with bundled assets and naming tables but no game logic.
Run length is the agent’s effort on one task as each paper reports it; “cap” is an imposed limit. Ours: range of the per-configuration medians of tool calls and of build time (median: 881–1,980 tool calls; Table 3).
| Configuration | Norm. Time (h) | Tok () | Tok () | Tool Calls | Turns / Sub-tasks | Source Files | Clean LOC | BT Score | Raw Win |
| codex-gpt-6.1-sol | 3.82 [1.98, 8.29] | 162.0 [84.2, 317.5] | 0.624 [0.391, 1.222] | 1,067 [595, 2, 016] | 1,186 / 18.5 | 35.0 [25.5, 53.0] | 2,328 [1, 523, 4, 666] | 69.11 | 62.8% |
| codex-gpt-6-astra | 2.72 [1.86, 5.48] | 121.5 [77.7, 212.9] | 0.532 [0.346, 0.853] | 921 [602, 1, 647] | 1,044 / 17.0 | 39.5 [32.2, 54.5] | 2,885 [2, 151, 4, 046] | 69.24 | 66.2% |
| zcode-glm-5.3 | 2.65 [1.75, 5.01] | 70.1 [40.1, 104.0] | 0.456 [0.259, 0.733] | 692 [426, 1, 151] | 645 / 25.0 | 82.0 [57.0, 122.0] | 13,824 [8, 603, 20, 406] | 35.78 | 30.6% |
| kimi-k3 | 7.11 [4.02, 11.36] | 202.1 [91.7, 305.4] | 0.694 [0.369, 1.572] | 1,045 [529, 2, 115] | 756 / 7.0 | 51.5 [38.5, 81.2] | 7,572 [4, 644, 13, 564] | 25.86 | 27.4% |
Table 3. Long-horizon engineering effort, codebase size, and human Arena preference.
Evaluation
We evaluate each game at three levels. Functional evaluation runs the game’s code against the specification and checks whether it does what was asked (§ 4.1; Figure 1B). Agent playtesting scores 3D presentation and playability, that is, what a player sees and how the game feels to play (§ 4.2; Figure 1C). Finally, human players compare games in blind pairs, which tells us which games people actually prefer (§ 4.3).
Non-Visual Functional Evaluation
Functional evaluation runs a game’s own production logic against the requirements of its task, in three stages (Figure 1B; worked example in Appendix § 12). Its inputs are the candidate’s registered source code and the acceptance criteria, which a separate agent derives from the rest of the task specification before any game is built (Figure B.1). The evaluation is non-visual: tests call the game’s code or drive it with scripted inputs in a headless browser, and use no visible browser window, screenshot, or model judge. It thus verifies game-state logic, not what is visible or usable on screen.
Stage 1: Freeze the rubric. Before any candidate code is seen, a spec reader splits each acceptance criterion into subitems and freezes them, keeping the subitem IDs, the verbatim requirement text, and the behaviors that must hold. Every candidate for a task is judged against this same rubric.
Stage 2: Adapt and run the tests. Tests are written after the build is finished and run on a copy of the delivered game. Test writers may attach debug interfaces to the copy that expose existing state but never change game functions, rules, or state; preconditions are reached through the game’s own operations, unlike in playtesting (§ 4.2). Candidates implement the same requirement behind different interfaces, so tests are written per candidate. Test writers inspect the candidate’s interfaces, state, clocks, and entry points, and several independent writers produce unit tests for each subitem. A 30% attack windup, for example, is checked through a duration field in one game and through start-to-release timestamps in another. A separate reviewer audits the tests and may revise them once; it may change test and driver code only, never the game code or the rubric. An isolated worker then runs every assertion against the real production functions on fresh fixtures. Expected values come from the specification alone, never from the candidate’s own output or formula, so real functional failures stay failures; infrastructure errors are logged separately.
Stage 3: Score. A subitem passes only if all of its assertions execute and pass, and each passed subitem earns an equal share of its criterion, however many assertions it has. Let be the criteria of game that were validly evaluated and the number of passed subitems of criterion . Criteria carry equal weight:
Criteria that could not be evaluated, for instance because of a test or infrastructure error, are left out of and are not counted as failures (Appendix § 12). If is empty the score is undefined, not zero. A configuration’s score is the mean over its games with a defined score, so real zeros stay in the mean.
3D Presentation and Playability
A game can pass every functional test and still be hard to look at or hard to play, so we also score what the player experiences, on two axes. 3D presentation asks whether the rendered world lets the player see what is happening and what matters. It is a holistic quality that is easier to compare than to rate, so we score it pairwise. Playability asks whether actions get clear responses and whether the player knows where the game stands. It is local, so we rate it one moment at a time (Appendix Figure 4).
Situations. A planner model reads only the task specification, the brief the developers received, and lists the key situations a player meets in ordinary play, such as entering the main scene or reaching the results screen, each with what the player should be able to notice. The plan is fixed before any game is played, so all games for a task are judged on the same moments.
Recording. An executor agent plays each game in the browser and records a short burst of frames and state readouts at each situation. It uses ordinary inputs where it can and otherwise a thin evaluation-time adapter that reads or sets the game’s own state but never implements game rules. Each moment is labeled natural or assisted (Appendix § 13.§ 4); an assisted moment shows the experience in that state, not that play reaches it. All judging uses only these frozen recordings.
3D presentation. A small fixed set of reference games, spanning the quality range, sits on a common Bradley–Terry scale with strengths . A judge model compares the recordings of game with those of each reference side by side, with identities hidden, and grades each outcome. We fit the strength with the references held fixed and report the expected win rate against them (details in Appendix § 13.§ 1):
Because the references never change, scores are comparable across tasks and do not depend on which other games are in the pool.
Playability. Playability covers control, whether an action gets a clear and timely response, and flow, whether the player knows the current state and the next step at turning points such as onboarding, failure, or the ending. The judge sees one situation at a time, never another game, and picks a level on a rubric written in terms of the player’s experience (Appendix § 13.§ 2), or abstains when the frames are not enough. With the level of moment and the rated moments of game for aspect ,
where are the aspects with at least one rated moment. Situations that cannot be observed in a game are left out of both scores, and we report the observed fraction as Coverage.
Human Preference
In a public arena, a participant plays two games that different configurations built for the same task and votes for the better one. Presentation order is randomized, and configuration identities stay hidden until the vote is stored. A vote is a preference for either game, a tie, or both bad; ties and both-bad votes count as half a win for each side, and skipped comparisons are discarded.
We fit a Bradley–Terry model with one strength per configuration , shared across tasks, and report the BT Score, the expected win rate of a configuration against an average opponent:
where is the logistic function and is the number of configurations. The BT Score adjusts for the strength of the opponents a configuration actually met. The strengths are fitted by regularized maximum likelihood, with each vote weighted by the inverse probability that its matchup was sampled. We also report Raw Win, the plain share of a configuration’s comparisons that it won, which makes no such adjustment. Appendix § 14 gives the likelihood, the weights, and the Raw Win formula.
Results
Table 2 reports the three evaluation levels of § 4 for each configuration. We draw three conclusions:
| Configuration | Functional Score | Agent playtesting 3D Presentation | Agent playtesting Playability | Agent playtesting Coverage | Human preference BT Score | Human preference Raw Win |
| codex-gpt-6.1-sol | 64.7 | 57.8 | 68.4 | 91.3 | 69.11 | 62.8 |
| codex-gpt-6-astra | 59.4 | 58.5 | 69.5 | 95.8 | 69.24 | 66.2 |
| kimi-k3 | 59.8 | 42.3 | 64.9 | 95.9 | 25.86 | 27.4 |
| zcode-glm-5.3 | 51.2 | 23.5 | 54.9 | 93.2 | 35.78 | 30.6 |
Table 2. Main results.
From-scratch 3D game development is far from solved. Even the best configuration earns under two-thirds of the functional score, and no configuration reaches 70 on either agent-playtesting score.
Correct code does not make a preferred game.
kimi-k3andgpt-6-astraare tied on the functional score, yet the BT score ofgpt-6-astrais more than twice that ofkimi-k3. Functional tests alone would rank the two as equals.The closer a measure is to the player, the wider the gap. The best and worst configurations are less than 15 points apart on the functional score and on playability, 35 points apart on 3D presentation, and more than 40 points apart in human preference. Agent playtesting sees the visual gap that functional tests miss, but it ranks
kimi-k3well above , and players do not. The three levels are therefore complementary, not interchangeable.
Where do these gaps come from? To find out, we go back to the development logs and analyze how the agents work: where their effort goes, how they debug, how they use 3D assets, and whether they stay on task over a long run. § 6 reports this analysis as a series of takeaways.
Analysis
We present the analysis as takeaways: each states one finding in a sentence and supports it with a short paragraph, with the full numbers in the tables. The takeaways answer four questions: where the engineering effort goes and what it buys (§ 6.1), how agents use 3D assets (§ 6.2), what agent playtesting does and does not capture (§ 6.3), and whether agents stay on task over a long run (§ 6.4).
Engineering Effort, Debugging, and Preference
Takeaway 1: Writing more code does not make a better game; top models keep codebases small and modular.
The two configurations that players prefer leave the smallest codebases, with medians under 3,000 clean lines of code
Table 2. Main results. Each column group is one evaluation level of § 4. Functional reports the functional score from non-visual tests (§ 4.1). Agent playtesting reports 3D presentation and playability, and the share of planned situations that were observed (Coverage; § 4.2). Human preference reports the Bradley–Terry expected win rate () against an average opponent (BT Score) and the empirical pairwise win rate (Raw Win; § 4.3). All values are on a 0–100 scale. Best score per column in bold.
Table 3. Long-horizon engineering effort, codebase size, and human Arena preference. Values report Median [Q1, Q3]. Using measured 8K-input end-to-end total output speeds (26.93–33.33 tok/s), Norm. Time normalizes active execution duration to a uniform 30 tok/s output rate. Clean LOC counts handwritten source lines of code after removing blank lines, comments, build bundles, and third-party libraries. BT Score reports the weighted Bradley–Terry expected win rate () against an average opponent, alongside empirical Raw Win rate.
(Table 3). kimi-k3 and glm-5.3 write roughly three to six times as much, in files at least twice as long. Large files make it harder for an agent to keep track of state shared across modules, so later edits tend to break earlier features.
Table 4. Token breakdown and human preference. Per-game means. Uncached In is new text entering the context, Cache Read is its history re-read from the cache, and Cache Replay is their ratio.
| Config. | BT Score | Uncached In | Cache Read | Cache Replay |
| gpt-6.1-sol | 69.11 | 6.82M | 307.41M | 45.1× |
| gpt-6-astra | 69.24 | 5.15M | 200.46M | 38.9× |
| kimi-k3 | 25.86 | 4.91M | 241.04M | 49.1× |
| glm-5.3 | 35.78 | 1.86M | 82.27M | 44.3× |
Table 4. Token breakdown and human preference.
Takeaway 2: Almost all input tokens are cached context re-read at every tool call; a runaway total signals a stalled build, not progress.
New text entering the context (file reads, compiler errors, browser logs) stays at a few million tokens per build. Totals reach tens to hundreds of millions because every tool call re-reads the project history from the prefix cache, 39–49 times over (Table 4), so totals track the number of tool calls and cache discounts keep long builds affordable. More is not better: within every configuration, builds that become playable use 20%–30% fewer tool calls than builds that stall. Splitting work into sub-tasks keeps each history short; kimi-k3 does this least, and even at equal output speed its median build takes roughly two to two and a half times as long as those of the Codex configurations (Table 3).
Takeaway 3: Extra tokens buy higher Arena preference only when spent on live testing.
More tokens alone do not make a preferred game: kimi-k3 spends about three times as many as glm-5.3 and is preferred less (Table 3 and Table 4). The Codex configurations, which players prefer, also spend more than glm-5.3, but they debug mostly by live testing, with full builds and browser checks, where kimi-k3 mostly runs static checks that never render a frame (Table 5). Even there the return flattens: gpt-6.1-sol outspends gpt-6-astra by a third to a half, and players rate the two alike.
| Config. | Debug Calls | tsc+vitest | build+test | Smoke | Func. Score | BT / Raw Win |
| gpt-6.1-sol | 103.0 (184.6) | 8.4% | 83.3% | 8.1% | 64.7% | 69.1 / 62.8% |
| gpt-6-astra | 55.0 (120.9) | 11.2% | 76.8% | 11.4% | 59.4% | 69.2 / 66.2% |
| kimi-k3 | 159.5 (67.9) | 63.5% | 17.7% | 16.9% | 59.8% | 25.9 / 27.4% |
| glm-5.3 | 105.3 (120.1) | 63.4% | 15.2% | 21.1% | 51.2% | 35.8 / 30.6% |
Table 5. Debugging and outcomes.
Table 5. Debugging and outcomes. Debug Calls: median (mean) per game, split into static and unit checks (tsc+vitest), project build+test, and live browser smoke tests.
Takeaway 4: What an agent debugs matters more than how often it debugs: static unit tests raise functional scores, whereas runtime testing wins human preference.
Two comparisons show why functional scores and player preference diverge (Table 5). In the same harness, more debugging raises the functional score but not preference: gpt-6.1-sol runs nearly twice as many debug commands per game as gpt-6-astra and scores higher on the functional tests, yet players rate the two alike. Across harnesses, the kind of debugging decides. kimi-k3 has the highest median debug count and matches gpt-6-astra on the functional score, yet its BT score is well under half as high. It spends 63.5% of its debugging on type checks and isolated unit tests, which pass without rendering a single frame. The two Codex configurations spend about 90% on full builds, integration tests, and live browser checks, which is where broken hitboxes, camera framing, and controls show up.
3D Asset Strategies and Player Enjoyment
Table 6. 3D asset strategies in the playable builds we inspected. Builds rely on built-in primitives only (Prim), retrieved external models only (Ret), AI-generated models only (Gen), hybrid external models (Hyb), or dynamic loaders (Dyn): • at least a quarter of builds, ◦ fewer, — none.
| Config. | Prim | Ret | Gen | Hyb | Dyn | Generated | Retrieved | Primitives |
gpt-6-astra | • | ◦ | ◦ | ◦ | • | < 1 | < 1 | 17 |
gpt-6.1-sol | • | • | — | — | • | 0 | < 1 | 8 |
kimi-k3 | — | • | — | — | — | 0 | 13 | 60 |
glm-5.3 | • | — | — | — | — | 0 | 0 | 11 |
Table 6. 3D asset strategies in the playable builds we inspected.
Takeaway 5: Adding more 3D objects does not make a better game.
kimi-k3 puts by far the most 3D content into its games, on average 13 downloaded models and 60 primitives per game, yet falls in the lower tier of human preference (Table 2 and Table 6). In agent playtesting, its games fall short on 3D presentation rather than on playability (Takeaway 7). A likely reason is that downloaded meshes arrive with their own scale, orientation, and framing, which the agent must reconcile with the rest of the scene.
Takeaway 6: Wiring AI-generated 3D models into live gameplay remains an open bottleneck for coding agents.
Even with 3D generation tools available, only gpt-6-astra integrates AI-generated 3D models into playable final builds, and only in a small share of them; no other configuration has a generated mesh in working runtime code (Table 6). Calling an API to create a static mesh is straightforward, but autonomously checking its proportions, pivot point, and materials and rigging it into interactive game logic remains largely unsolved.
3D Presentation and Playability
Takeaway 7: 3D presentation separates agents far more than moment-level playability, and neither captures a whole play session.
Playability spans only about 15 points across configurations, also when only naturally reached moments are counted, because at the planned key moments most games already give readable feedback and a clear next step (Appendix § 13.§ 4). 3D presentation spans 35 points: kimi-k3 nearly matches the top configurations on playability but trails by 16 points on presentation, consistent with its reliance on downloaded assets (Takeaway 5), and most glm-5.3 games sit at the bottom of the scale (Appendix Table 10). Neither score separates the two Codex configurations (Appendix Table 11), and neither do players. Both scores put kimi-k3 above glm-5.3, which players do not: our recordings are short silent clips of key moments, whereas Arena players play a whole session.
Takeaway 8: Agents build clear menus and stage transitions far more reliably than responsive 3D action controls.
Flow, whether prompts, menus, and stage transitions make the state and next goal clear, beats Control, whether an input gets immediate, readable feedback, in 86% of all games and by a similar margin in every configuration (Appendix Table 10). The two scales are not calibrated against each other, so the consistency across configurations is the more robust signal. About half of the glm-5.3 games score below 3 on Control, where a player can no longer tell whether an input registered. A likely reason is that flow is built from 2D overlays and state machines familiar from web development, whereas clear control needs input handling, camera framing, sightlines, and hit feedback to work together in real time.
Long-Horizon Goal Retention and Plan Adaptation
A build runs for hours and around a thousand model calls, and the agent’s context is compacted several times on the way, so we check whether a long run stays on its goal. We go back through the development logs, cut each run into four quarters (Q1–Q4), sample operations from each, and have an LLM reviewer answer two questions about every sampled operation. Goal retention (GR) asks whether the operation still serves the task, including any change approved along the way [2, 19]. Plan adherence (PA) asks whether it follows the plan the agent made before it started coding [19, 26]. Both measure whether the agent is still working on the right thing, not whether the work succeeds. Table 7 also reports the largest tasks separately, where runs are longest (details in Appendix § 16).
| Config. | Goal retention (%) | Plan adherence (%) | |||||||
| Q1 | Q2 | Q3 | Q4 | Q1 | Q2 | Q3 | Q4 | ||
| gpt-6.1-sol | 100 | 100 | 100 | 100 | 100 | 100 | 96.2 | 91.4 | |
| gpt-6-astra | 100 | 100 | 100 | 100 | 100 | 100 | 100 | 100 | |
| kimi-k3 | 84.5 | 94.0 | 98.8 | 96.2 | 100 | 100 | 100 | 98.6 | |
| Config. | Full run (%) | Calls by task size | |||||||
| GR | GR{sup}†` | Small | Medium | Large | BT | ||||
| gpt-6.1-sol | 99.7 | 99.9 | 849 | 1,186 | 1,645 | 69.1 | |||
| gpt-6-astra | 100 | 100 | 971 | 1,132 | 1,404 | 69.2 | |||
| kimi-k3 | 96.1 | 99.3 | 601 | 730 | 761 | 25.9 |
Table 7. Goal retention (GR) and plan adherence (PA) over a run. Top: large tasks, by quarter of the run (Q1 is the first, Q4 the last). Bottom: all builds, with full-run GR as judged and with process-rule violations left aside (GR†), median model calls per build (including sub-agents) by task size, and BT score.
Takeaway 9: Agents do not forget the game: late in a long run, they are still working on what was asked.
No configuration drifts. Goal retention is between 96% and 100% over full runs and stays at 94% or above on the largest tasks; where it dips on those tasks, as for kimi-k3, the dip comes in the first quarter of a run, not the last (Table 7; Appendix Table 14). Forgetting the goal is therefore not what separates good builds from bad ones: kimi-k3 stays on task almost as well as the Codex configurations, yet players rarely prefer its games. What differs is how the work is checked in the running game.
Takeaway 10: When agents do step off the path, they leave the plan or break a process rule; they do not abandon the game.
Two things do change. First, agents follow their initial plan less closely near the end: on large tasks, the plan adherence of gpt-6.1-sol falls from 100% to 91.4% in the last quarter while its goal retention stays at 100%. Nearly all of this off-plan work still serves the goal, typically scope approved after the plan was written, so it is adaptation, not drift, and a long run is better judged against its current goal than its first plan. Second, the few real violations concern how the agent was told to work, such as starting to code when asked only to plan, not what to build. Setting them aside raises the goal retention of kimi-k3 from 96.1% to 99.3%.
Conclusion
We introduced 3D GameDev Arena, a benchmark of 108 tasks for long-horizon, from-scratch 3D game development, evaluated at three levels: functional tests, agent playtesting, and blind human preference. The task is far from solved: the best configuration earns under two-thirds of the functional score, and correct code does not make a game that players prefer. Our analysis traces this gap to how agents check their work, not to how well they remember it. Agents do not forget what they are building, even late in runs that last hours. But those that debug mainly with type checks and isolated unit tests pass functional tests without producing a game people want to play, while the preferred agents spend their debugging on full builds and live browser checks. The main limitation of 3D GameDev Arena is the time its evaluation takes: every game needs its own adapted tests, a recorded play session, and human votes, which limits how many agents and repeated runs can be evaluated; making it faster is the natural next step.
LLM Usage Statement
We used an LLM-based assistant (Claude, Anthropic) to polish the manuscript. All technical content, experiments, results, and claims are the authors’ own, and all edited text was reviewed by the authors.
Extended Related Work
This appendix gives the full discussion behind the three shortened paragraphs in § 2.
Game Generation Benchmarks
Recent benchmarks have moved beyond generic software engineering testbeds [23, 27] to evaluate coding agents on game creation. Early efforts such as V-GameGym [45] focus on single-file 2D Pygame synthesis, whereas engine-grounded benchmarks—including GameDevBench [11], GameEngineBench [24], and GameLogicBench [6]—assess localized feature implementation or tutorial-derived edits inside pre-structured Godot and Unreal repositories. At the full-project scale, JAMER [36], GameCraft-Bench [29], OpenGame-Bench [21], GameXpert-Bench [8], and GameASG-Bench [47] evaluate end-to-end game generation across game engines and web runtimes. 3D GameDev Arena differs from these benchmarks in three respects: (i) whereas existing suites predominantly target 2D games or provide pre-bundled project assets, we target full-scale 3D games where agents must autonomously retrieve or synthesize dozens of heterogeneous 3D models from scratch; (ii) rather than constraining agents to fixed engine templates, localized edits, or pre-declared test interfaces [47], we enforce a zero-scaffolding, stack-agnostic workspace over multi-hour horizons; and (iii) beyond rule compliance, we establish blind human A/B playtesting to measure player preference. Table 1 compares design choices and is not a performance ranking.
Evaluating Generated Games
Assessing generated games requires verifying dynamic, stateful interactions that static code inspection and compilation checks cannot capture. Prior work has explored three main verification paradigms. First, white-box state and assertion testing checks runtime invariants directly, ranging from tick-level engine assertions [6], runtime state injection [20], and replayable demonstration traces [29] to pre-declared snapshot contracts [47]. Second, vision- and GUI-based playtesting builds on automated game QA [48] by deploying multimodal judges and browser/GUI agents to interact with rendered frames and adjudicate behavioral rubrics, as in V-GameGym [45], OpenGame-Bench [21], PlayEval [34], and PlaytestArena [18]. Third, pairwise human preference modeling via Bradley–Terry rankings [3] has become standard for open-ended text and web generation [12, 38], yet remains largely absent from 3D game benchmarks. Our evaluation pipeline bridges these paradigms by combining non-intrusive evaluation-time state adapters, automated browser playtesting agents, and blind human A/B preference rankings.
Game Generation Agents
Early multi-agent software frameworks such as ChatDev [35], MetaGPT [16], and GameGPT [7] simulate studio roles to translate natural-language prompts into lightweight 2D games, but frequently suffer from cross-file inconsistencies and asset hallucinations at scale. Recent domain-specialized coding agents address long-horizon coherence through structured workflows and feedback loops: OpenGame [21] pairs a specialized code model with reusable template and debugging skills for 2D web games, while PlayCoder [34] and Play2Code [18] couple code-writing agents with GUI playtesters for iterative refinement. In 3D environments, world-model approaches synthesize interactive video frames directly [4, 5] without explicit software engineering, whereas engine-centric multi-agent pipelines—such as DreamGarden [14], UniGen [43], and AutoUE [44]—orchestrate asset retrieval, procedural scene layout, and C++/C# scripting inside Unity or Unreal Engine. Rather than tying evaluation to a single bespoke pipeline, our benchmark provides an open, stack-agnostic testbed for evaluating arbitrary frontier coding agents on autonomous 3D game production.
Task Specification Format
Every task specification follows the five-part schema introduced in § 3.2. Table 8 shows how the five parts are instantiated for two representative Hard-tier 3D vertical slices.
Acceptance Criteria
The Acceptance Criteria section is written after the first four sections of a specification are final. A separate agent takes these four sections as input and writes the fifth: clear, reproducible, objectively verifiable test cases in natural language. No game exists at this point. The prompt below also fixes the rules that test code must follow later, when the criteria are run against a finished game, which is why it refers to game source code. In short, tests are non-visual: they simulate real inputs in a headless browser and read runtime state and logs to check numerical rules, map collisions, input responses, and state transitions. They are derived from the requirements, never from a game’s own code, and they run on a copy of the delivered game, whose functions and rules they must not change.
Non-Visual Basic Functional Acceptance Constraints Prompt
You are responsible for formulating multiple basic functional acceptance criteria based on the given game Prompt. Focus on determining whether core operations, numerical rules, maps and collisions, and state transitions meet the requirements. Output clear, reproducible, and objectively verifiable natural-language test case descriptions, without generating game implementation code.
Game Code and Browser Constraints
Game source code, configurations, maps, assets, and dependency files must all remain read-only. Subsequent test implementations may only write or modify test-side code, configurations, and reports.
Adding game debug interfaces is permitted, but you must strictly refrain from modifying existing game functions, replacing game functions, altering collision or win/loss rules, or directly setting states such as health, gold, position, or cooldowns to establish test scenarios. Preconditions must be reached through existing game operations.
Invoking interactive Chrome Debug / Chrome DevTools tools, visible browser windows, or existing user browser sessions is prohibited. Test scripts are permitted to launch an isolated headless Chromium/Chrome instance on their own to conduct programmatic testing and read-only inspection via Playwright or that instance’s CDP.
Using Playwright’s
-headed,-ui,-debug, and Inspector is prohibited; among these,-debugopens a browser window and Inspector (Playwright Official Documentation).Do not use screenshots, videos, pixels, colors, image recognition, or manual visual inspection as acceptance evidence. Real keyboard and mouse inputs, DOM/accessibility text, existing runtime states, events and console logs, as well as read-only inspection of source code and static data, are permitted.
Basic Functional Testing Focus
Values and Resources: Check the initial values, increments/decrements, consumption, caps, damage, rewards, cooldowns, and settlement rules declared in the source Prompt.
For relevant operations, separately check insufficient resources, exact resources, surplus resources, as well as repeated operations, invalid targets, and operations during cooldowns.
Check whether a single valid event produces the specified number of deductions, damage, or rewards, and whether failed or rejected operations maintain the states that should remain unchanged.
Expected values must be derived independently from the Prompt; do not copy calculation results from the game implementation as the correct answer. For values, units, boundaries, or tolerances not specified in the source text, do not invent them yourself. Numerical test cases shall use equivalence partitioning and boundary value analysis (ISTQB Testing Techniques, Section 4.2).
Maps and Collisions: Check the scenes, spawn points, objectives, obstacles, zones, and map transitions required by the source Prompt.
Verify whether spawning and movement are valid, whether collision and out-of-bounds handling are correct, and whether mandatory objectives can be reached through normal operations.
For readable map data, graph search may be used to check paths; conditions such as passage direction, character colliders, jumping ability, doors, and keys must be considered—do not treat all maps as undirected planar grids. Separately record “a path exists in the map data” and “the character can traverse the path via real operations.” The existence of configurations, navigation returning a path, or the presence of scene objects cannot individually prove that runtime behavior is correct.
Operations and States: Verify through real inputs that primary operations actually change the corresponding states; do not merely check that functions or event listeners exist.
Cover start, pause, resume, win/loss, failure, retry, and scene transitions according to the source Prompt, checking for illegal phase operations, duplicate submissions, and residual old inputs.
Include at least one reachable core gameplay loop. Fixed seeds and deterministic repetition are checked only if already supported by the game or explicitly required by the Prompt; do not override random numbers or the game clock to manufacture consistent results.
Test Case Writing and Verdict Criteria
Each test case verifies only one independent objective and includes: ID, source clause reference, preconditions, actions, evidence, pass criteria, and fail criteria.
Provide positive, rejection, and boundary scenarios for key rules; isolate other mechanisms to prevent them from
masking the result under test.
Do not use undefined vague criteria such as “normal,” “nearby,” “sufficient,” or “obvious.” When rules conflict or lack basis, list only the affected sub-assertions as to-be-confirmed.
Static evidence can only support static conclusions; dynamic features must have evidence of actual operations and state changes. The test model should be a simplified model built from requirements, not a direct copy of the implementation under test. When existing states or logs cannot be obtained, clearly state the evidence gap and mark it as BLOCKED; do not modify the game to adapt to the test, nor directly mark it as PASS or FAIL.
Mark unexecuted test cases as pending execution; do not claim they have already passed. After execution, record actual inputs, pre- and post-states, expected values, actual values, and failure reproduction steps.
Sources and reference images.
Tasks are curated from widely played, top-charting commercial titles across major distribution ecosystems, including Steam, online PC platforms, and console storefronts. Where a task provides reference images, they communicate intended scene composition, visual aesthetics, or spatial layouts. They serve strictly as non-executable design constraints: a reference image neither supplies an interactive scene graph nor establishes that a generated build satisfies functional requirements.
Functional Evaluation: Details
This appendix gives the details of the three stages of § 4.1; Figure 3 shows a worked example.
Figure 3. Non-visual functional acceptance. Gray probe cards show green windup (1.S1) and orange critical-damage (1.S2) excerpts for illustrative Criterion 1; full probes cover its frozen rubric. Schematic APIs run on separate fresh production fixtures (setup, actions, and remaining checks omitted). At 1 Hz, 30% windup is 0.3 s under shared timing tolerance : GPT reads recorded duration and damage, while Kimi uses accepted-start-to-release timestamps (seconds) and HP loss. Damage assumes base 100, ordinary 2× critical hits, sufficient HP, and no mitigation, shields, healing, extra damage, or other modifiers (GPT/Kimi denote illustrative origins). Credit requires all subitem assertions to execute and pass, and repairs modify tests only.
Illustrative content of Figure 1. The Borderlands note, the harness–model pairs, and the MOBA subitem in Figure 1 are illustrative: the two candidate rows of the subitem are historical assertions and the third is synthetic, not a current-protocol result.
Inputs and scope. The protocol takes the acceptance criteria, which a separate agent derives from the finished specification before any candidate is built (Appendix § 10.§ 1), and the registered source code of the candidate. Registration records provenance; it does not assert byte-identity with a published build. Tests run on a copy of the delivered game, whose source stays read-only. The channel uses no visible browser window, pixels, screenshots, visual classifiers, or model judges: it calls production handlers and code-level view models directly or drives them with scripted inputs in a headless browser, and may read DOM text, runtime state, events, and logs. It therefore verifies state logic, not screen visibility or usability.
Writing and auditing tests (Stage 2). Given the frozen subitem IDs, the candidate’s codebase is inspected (interfaces, state, clocks, and entry points), and multiple independent, randomized tasks per subitem write unit tests adapted to that codebase. The reviewer agent that audits the tests revises them at most once and modifies only test or driver code. Programmatic checks on subitem IDs, allowed files, and syntax run alongside the review. The result is multiple assertions per subitem.
Execution (Stage 2). An isolated worker executes the assertions one by one against real production functions, commands, or event paths on fresh fixtures; an engine-native driver may expose state without replacing the tested logic. The worker logs inputs, before and after states, events, resource changes, observed and expected values, tolerances, and clock boundaries. Expected values must come independently from the specification and legal preconditions, never from observed outputs or the candidate’s tested formula. Source-text checks apply only to inherently static rules, and passing requires actual assertion execution, not author claims or forced outcomes. Real functional failures remain failures, and infrastructure errors are logged separately.
Validity and scoring (Stage 3). The set of validly evaluated criteria excludes explicit, spec-grounded OUT_OF_SCOPE items, TEST_ERRORs, unexecuted BLOCKED claims, infrastructure errors, and missing, stale, or unrun results. For , subitem gets if and only if all of its assertions execute and pass, and otherwise, including when worker execution proves it BLOCKED; . Each passed subitem earns of its criterion (for example when ), regardless of how many assertions were generated for it. Criteria carry equal weight, and legacy weight fields are unused. Across a configuration’s declared task set, the overall score is the equal mean of FunctionalScore over games with defined scores (undefined if there are none), which keeps real zero scores in the mean and excludes invalid or missing runs.
3D Presentation and Playability: Details and Validation
Presentation scoring
The reference set has four games from three tasks; the references themselves are not scored. Each game is compared with every reference twice, once in each left–right order, giving eight comparisons. A judged margin of slight, clear, or large counts as an outcome of 0.70, 0.85, or 0.95 for the winner, and a tie as 0.5. We estimate with a weak prior toward the scale center () and give a score only when all comparisons are valid. A game that loses all eight comparisons by a clear margin scores 17.2.
Playability scale
For each moment, the judge receives the definition of one aspect, the five levels in Table 9, one short example per level, and the moment itself. The examples come from genres outside the benchmark, such as card games or golf, so that the judge compares experiences rather than mechanics. The judge first describes what the player sees, then picks a level and explains why it is not the neighboring one.
Table 9. Five-level playability scale (translated from the judge prompt).
| Level | Control: does the action get a readable response? | Flow: does the player know the state and the next step? |
| 1 | The player probably cannot tell whether the action worked, or misreads the result. | The player probably misses the change, or thinks the game is stuck or broken. |
| 2 | Something changes, but the player has to search or guess what it means. | The player notices a change but has to guess the state or the next step. |
| 3 | The result is recognizable, but feedback is weak, far from where the player is looking, or hard to tell apart. | The state is clear, but the player has to explore to find out how to continue. |
| 4 | The result is clear, timely, and where the player is looking; only minor roughness. | State and next step are both clear; only small details are missing. |
| 5 | As 4, and the kind, size, and target of the result are obvious at a glance. | As 4, and the player also knows what to expect, such as a time limit or the cost of failing. |
Table 9. Five-level playability scale (translated from the judge prompt).
Table 10 gives the per-configuration scores behind Takeaways 8 and 9.
| Config. | 3D Pres. | Playability | Control | Flow | BT |
| gpt-6.1-sol | 57.8 (61.4) | 68.4 (68.8) | 3.40 (60.1) | 4.07 (76.8) | 69.11 |
| gpt-6-astra | 58.5 (59.9) | 69.5 (70.0) | 3.49 (62.2) | 4.07 (76.7) | 69.24 |
| kimi-k3 | 42.3 (43.4) | 64.9 (65.0) | 3.35 (58.7) | 3.84 (71.1) | 25.86 |
| glm-5.3 | 23.5 (17.2) | 54.9 (55.8) | 2.88 (47.1) | 3.51 (62.6) | 35.78 |
Table 10. Automated 3D Presentation and checkpoint Playability. 3D Pres. (0–100) measures reference-anchored visual scene quality (mean, with median in parentheses). Playability (0–100) averages Control and Flow (1–5 rubric means, rescaled to 0–100 in parentheses).
Configuration comparisons
Table 11 gives the paired differences behind the results. Each configuration covers a different subset of tasks, so we compare two configurations only on the tasks where both have both scores, and report the mean difference with a 95% percentile bootstrap interval over tasks. Reference games are excluded.
| Pair | Pres. diff | Play. diff |
| gpt-6.1-sol – gpt-6-astra | −2.0 [−5.3, +1.2] | −1.4 [−3.1, +0.5] |
| gpt-6.1-sol – kimi-k3 | +17.4 [+13.1, +21.8] | +4.2 [+1.9, +6.6] |
| gpt-6.1-sol – glm-5.3 | +32.4 [+26.7, +38.2] | +10.6 [+7.0, +14.0] |
| gpt-6-astra – kimi-k3 | +17.8 [+12.4, +23.2] | +5.2 [+3.0, +7.3] |
| gpt-6-astra – glm-5.3 | +34.8 [+29.0, +40.3] | +13.5 [+9.9, +17.0] |
| kimi-k3 – glm-5.3 | +20.2 [+14.0, +26.4] | +8.7 [+5.0, +12.6] |
Table 11. Paired differences (first minus second configuration) on shared tasks. Brackets: 95% bootstrap interval over tasks (2,000 draws).
How moments were reached
Across the four configurations in the results, 37% of the recorded moments were reached naturally and 63% with assistance; about 6% of the planned moments were not recorded. About half of the assisted moments only had their precondition set through the adapter, with no help inside the recorded window, so all feedback in the frames comes from the game itself; in the other half the adapter was also used during the recording. Per-game reports keep all three labels. The narrow range of playability across configurations is not an artifact of assisted arrival: recomputed from naturally reached moments alone, playability is 69.3, 68.3, 63.7, and 54.5 for gpt-6-astra, gpt-6.1-sol, kimi-k3, and glm-5.3, close to the values over all moments (69.5, 68.4, 64.9, 54.9). Each moment is rated on an absolute five-level scale, and 76%–85% of the rated moments of every configuration fall at level 3 or 4.
Worked example
Figure 4 traces one task through situation planning, recording, and both judging formats.

Figure 4. 3D presentation and playability evaluation. Key situations are planned from the task specification alone, then played and recorded once per game. Example: two agents on the same task, SP3D-001 (lane runner): codex-gpt-6.1-sol and zcode-glm-5.3. Top: the same moment in both games; the judge faults zcode-glm-5.3 for a tutorial panel that keeps covering the runner and the near track. On the Bradley–Terry scale the reference strengths are fixed. codex-gpt-6.1-sol () wins all 8 comparisons with the 4 references and scores 75.5 on presentation; zcode-glm-5.3 () loses all 8 clearly and scores 17.2. Bottom: the same 3 of 11 judged moments in each game, with an excerpt of the judge’s reason and how the moment was reached; playability is 66.5 versus 34.4. Quotes are translated from the judge’s output and shortened.
The last two rows explain the split in § 4.2. Rated one moment at a time, presentation misses an obvious repaint: the judge even mentions the flat colors in its description, yet gives the same level. Seen side by side, the change is easy to spot. Playability shows the opposite pattern. Under pairwise comparison, the repaint leaks into control and flow, while per-moment rating ignores it and still picks up the removed feedback.
Human Preference: Details
This appendix completes § 4.3. Responses are a preference for either game, a tie, both games bad, or skip. Ties and both-bad responses remain distinct in the records but each contributes a half-win to both configurations; skips contribute no evidence to the ranking.
Fitting the strengths. The BT Score of Equation (4) is the expected win rate of configuration against a uniform average over the other configurations (). The parameters are fitted by maximizing a weighted, L2-regularized log-likelihood
| Question | Test | Result |
| Is it stable? | Rate the same moments twice | identical in all cases |
| Swap the per-level examples for a different set, or drop them | game scores almost unchanged () | |
| Does each score track its own property? | Remove the game’s hit and action feedback effects | control drops; flow and presentation unchanged |
| Repaint the scene structure in one flat color | presentation drops; control and flow unchanged | |
| Why two formats? | Rate presentation one moment at a time | order agreement drops (23 / 8 / 2); the repaint goes unnoticed |
| Judge playability by pairwise comparison | the repaint also lowers control and flow |
Table 12. Validation of the 3D presentation and playability scores on an early subset, run before the full evaluation. Agreement counts same-task game pairs whose order matches the direct pairwise comparison (agree / reverse / tie). The perturbation rows edit one game and replay the same situations.
over the valid votes,
where indexes the valid votes, and are the two configurations shown in vote , and is the outcome for : 1 if wins, 0 if wins, and for a tie or both-bad. Skipped votes are not valid votes and do not enter the sum.
Sampling weights and regularization. The weight corrects for matchups being drawn with unequal probability. Let be the conditional probability that the arena’s sampler selected the type of matchup that vote belongs to. We set
where runs over the valid votes, so that the weights have mean one. A rarely sampled matchup thus counts more per vote. The weight depends only on the sampling design: it is neither a measure of rater reliability nor a model score. The L2 term keeps the strengths finite when a configuration wins or loses every comparison it appears in. Its strength is fixed at and is not learned from the votes: a larger shrinks the strengths toward zero and gives more conservative scores, whereas a smaller follows the observed votes more closely but becomes more extreme with few votes.
Raw Win. Raw Win is the unweighted share of a configuration’s valid comparisons that it won, with ties and both-bad responses counted as half-wins for both sides:
where , , and are the numbers of clear wins, ties, and both-bad responses of configuration , and is the number of all its non-skipped comparisons; skips are removed from both the numerator and the denominator. Raw Win uses neither the sampling weights nor any per-voter weighting, and it does not use the strength model, so unlike the BT Score it does not account for which opponents a configuration happened to meet.
Reporting. Preference measures relative player judgment and is reported separately from both automatic scores and generation success. A completed study must disclose sampling probabilities, repeat-vote handling, valid vote counts, and uncertainty clustered by participant or anonymous session.
Skill-Augmented Generation Protocol and Prompts
This appendix details the modular skills, 3D asset acquisition rules, and exact orchestration prompts used in the two-stage game generation protocol introduced in § 3.3. Agent Skills [1, 25, 40, 41] are composable packages of procedural workflows, verification scripts, and domain constraints that an agent loads progressively on demand.
Evaluated configurations. Table 13 lists the four evaluated configurations with their harness versions. Each pairs a native coding-agent harness with one model and runs at the highest reasoning setting available in its harness. Each model runs in exactly one harness, so a short name identifies a configuration.
Two-Stage Skill Workflow and Asset Pipeline
Stage 1: Architectural Planning and Asset Orchestration. Given a benchmark task specification, a lead orchestration agent inspects the target repository structure, manifest schema, and build contract without writing production game code. Domain-level coordination is governed by the iterative browser verification guidelines from develop-web-game [33]. For 3D tasks, dedicated subagents are spawned to prepare environmental meshes and textures. Subagents prioritize retrieving open, royalty-free runtime assets formatted in glTF 2.0 [37]—utilizing compact binary (.glb) scene hierarchies, Physically Based Rendering (PBR) materials. Only when suitable public structural models cannot be located do subagents invoke meshy-3d-generation [32] for text- or image-to-3D synthesis (capped at five generated models per project). Once technical risks and workspace identifiers are resolved, the lead agent invokes writing-plans [39] to produce a concrete, file-level implementation blueprint coupling a single unified simulation state to controls, physics, onboarding UI, and acceptance checks.
Stage 2: Subagent-Driven Plan Execution. An execution agent receives the approved Stage 1 blueprint and produces the final deliverable using superpowers:subagent-driven-development and superpowers:executing-plans [39]. To prevent context degradation across multi-file implementations, subagent-driven-development dispatches isolated implementer subagents per planned task, enforces post-task specification-compliance and code-quality review gates, and records state transitions in a persistent progress ledger. Throughout execution, agents perform live headless-browser debugging and interactive playtesting [33]—prioritizing simulation fidelity and gameplay correctness over raw frame rate ( FPS)—until the root build.sh script compiles a self-contained index.html satisfying all acceptance criteria.
Orchestration and Execution Prompts
Stage 1: Lead Orchestration and Planning Prompt
You are the lead orchestration agent responsible for generating this browser game. Use $threejs-game-director as the primary orchestration skill, alongside $develop-web-game for iterative browser verification, $writing-plans for architectural planning, $threejs-3d-generator for structural 3D models, and automatically invoke relevant gameplay, graphics, UI, asset, audio, browser-debugging, performance, and QA skills when appropriate.
Hard Constraints:
Session Isolation: Conduct all generation within this conversation thread; do not invoke external platform-level automated generation workflows.
Contract Discovery: Inspect the repository first to infer the exact artifact format, directory layout, manifest schema, build contract, and database linkage expected by the harness (e.g.,
<WORKSPACE_ROOT>/games/<TASK_ID>/<HARNESS_MODEL>/<RESULT_ID>/).Database Integrity & Zero Cross-Contamination: Do not alter benchmark task prompts or generator scripts, confirm
<TASK_ID>before updating records, and never read or reuse files or assets from other games on the host machine.Subagent Asset Pipeline & glTF Delivery: Spawn subagents to prepare environmental and texture assets, prioritizing open, royalty-free 3D assets in Khronos glTF 2.0 /
.glbformat (https://www.khronos.org/gltf/). Invoke @meshy-3d-generation only when suitable public structural meshes cannot be located (capped at 5 generated models per project).Onboarding & QA Pass: Include an interactive tutorial/onboarding overlay explaining objectives and controls, followed by a comprehensive code review, browser QA pass, and bug-fix iteration prior to packaging.
Current Task (Planning Phase Only): Do not begin implementation yet. First, (1) determine the storage layout, manifest schema, build contract, and status-update mechanism; (2) audit available agent skills and local CLI tools; (3) identify technical bottlenecks, ambiguities, or contradictions in the specification;
Game Specification Input: <INSERT_FIVE_PART_TASK_SPECIFICATION>
Stage 2: Subagent-Driven Execution Handoff Prompt
Implement the approved plan using superpowers:subagent-driven-development and superpowers:executing-plans
Long-Horizon Analysis: Protocol and Full Results
This appendix explains how the goal retention (GR) and plan adherence (PA) numbers of § 6.4 were obtained and gives the full results.
What we sample. We work from the development logs of finished builds. An operation is one step in which the main agent does real work with a tool, such as editing a file or running a command; steps that only wait, coordinate, or plan are skipped. We line up the operations of a run in order, cut them into four quarters by count (not by clock time), and draw up to eight at random from each quarter, which samples about 8% of all operations. We also draw a smaller sample of operations by sub-agents; these are judged against the task the sub-agent was given and are kept out of the rates reported here. Two limits remain: the filter that selects operations misses some terminal commands issued through a wrapper, and the reviewer does not always see the full context around an operation.
How an operation is judged. A reviewer answers the two questions of § 6.4 for every sampled operation. Each answer is one of three labels: aligned (A), the operation serves the goal or follows the plan; deviating (D), there is clear evidence that it does not; or unclear (U), the logs are not enough to tell. If the task is changed along the way, the goal changes with it, but the plan stays the first one the agent wrote. Operations carried out before that plan existed are left out of plan adherence, and an operation whose link to the plan cannot be established is unclear, not aligned.
Who reviews. The reviewers are LLM agents (Codex), not people. A random fifth of the sampled operations was reviewed a second time by a reviewer that did not see the first labels, and 99% of these pairs are usable. On goal labels for main-agent operations the two reviews agree in 92.3% of cases (; is modest because almost all labels are A). On plan labels for sub-agent operations agreement is weak (), so sub-agent labels are not used for the rates in this paper. Every operation first labeled as deviating was also checked by a different reviewer; these extra checks are not part of the agreement figures. The two reviews are kept apart, but they are not fully blind to which configuration produced a run, and two LLM reviewers can make the same mistake. The tables use the labels after reconciliation.
From labels to rates. Because we draw at most eight operations per quarter, a sampled operation from a long quarter stands for more operations than one from a short quarter, and we weight it accordingly. With and the weighted counts of aligned and deviating operations in one build, we report
the share of aligned operations among those that could be judged; this is the GR and PA of the main text. Unclear operations are left out of the rate. A configuration’s value is the plain average over its builds, so every build counts the same however long it ran. A build whose logs cannot be reliably matched to it gives no rate. A rate of 100% means that every sampled operation was aligned, not that every operation was.
Large tasks. Tasks are ranked by the structure of their specification, namely how many groups of requirements it contains and how long its longest chain of dependent steps is, and split into thirds: small, medium, and large. This reflects the size of the task, not how hard people find it or how long an agent takes.
Process rules. Goal retention also covers instructions about how to work, such as staying read-only, only planning during the planning stage, and where to put temporary files. Breaking such a rule counts as a deviation even when the agent is still building the right game. Table 14 therefore repeats the full-run goal retention with these violations counted as unclear instead of deviating (GR†). This does not excuse the violations, and the deviations that remain are not a pure measure of memory; it shows how much of a lower goal retention comes from process rules and not from the game itself.
| Configuration | All builds GR | All builds GR | Large tasks GR | Large tasks GR |
| gpt-6-astra | 100.00 | 100.00 | 100.00 | 100.00 |
| gpt-6.1-sol | 99.66 | 99.94 | 100.00 | 100.00 |
| kimi-k3 | 96.06 | 99.28 | 94.51 | 97.64 |
| glm-5.3 | 100.00 | 100.00 | 100.00 | 100.00 |
Table 14. Full-run goal retention as judged (GR) and with process-rule violations set aside (GR). Values are percentages. For GR, operations that broke a process rule are counted as unclear, not as aligned.
References
- [1]Anthropic. Agent skills: An open standard for modular procedural knowledge (SKILL.md). https://agentskills.io/, 2025. Open specification for on-demand progressive disclosure of instructions, scripts, and resources in coding agents. 2, 20
- [2]Rauno Arike, Elizabeth Donoway, Henning Bartsch, and Marius Hobbhahn. Technical report: Evaluating goal drift in language model agents, 2025. 8arxiv.org/abs/2505.02709
- [3]Ralph Allan Bradley and Milton E. Terry. Rank analysis of incomplete block designs: I. The method of paired comparisons. Biometrika, 39(3/4):324–345, 1952. 2, 13DOI
- [4]Jake Bruce, Michael D. Dennis, Ashley Edwards, Jack Parker-Holder, Yuge Shi, Edward Hughes, Matthew Lai, Aditi Mavalankar, Richie Steigerwald, Chris Apps, Yusuf Aytar, Sarah Maria Elisabeth Bechtle, Feryal Behbahani, Stephanie C. Y. Chan, Nicolas Heess, Lucy Gonzalez, Simon Osindero, Sherjil Ozair, Scott Reed, Jingwei Zhang, Konrad Zolna, Jeff Clune, Nando de Freitas, Satinder Singh, and Tim Rocktaschel. Genie: Generative interactive environments. In Proceedings of the 41st International Conference on Machine Learning (ICML), pages 4603–4623, 2024. 13arxiv.org/abs/2402.15391
- [5]Haoxuan Che, Xuanhua He, Quande Liu, Cheng Jin, and Hao Chen. GameGen-X: Interactive open-world game video generation. In The Thirteenth International Conference on Learning Representations (ICLR), 2025. 13arxiv.org/abs/2411.00769
- [6]Xinyu Che, Yunfei Ge, Shihao Li, Yanchen Liu, Hang Yan, Xinping Lei, Yanghai Wang, Zixuan Dong, Yifan Yao, Qianqian Xie, Letian Zhu, and Jiaheng Liu. GameLogicBench: Evaluating coding agents on runtime game logic with tick-level state assertions. arXiv preprint arXiv:2609.21562, 2026. 2, 13arxiv.org/abs/2609.21562
- [7]Dake Chen, Hanbin Wang, Yunhao Huo, Yuzhao Li, and Haoyang Zhang. GameGPT: Multi-agent collaborative framework for game development. arXiv preprint arXiv:2310.08067, 2023. 13arxiv.org/abs/2310.08067
- [8]Kun Chen, Haorong Hong, Peizhong Gao, Jianfeng Lin, Tongxu Luo, Yuxuan Xie, Chenxu Liu, Jieling He, Zhongyuan Liu, and Zeno Zeng. GameXpert-Bench: How far are coding agents from expert game development? arXiv preprint arXiv:2608.21833, 2026. 2, 13arxiv.org/abs/2608.21833
- [9]Kun Chen, Haorong Hong, Peizhong Gao, Jianfeng Lin, Tongxu Luo, Yuxuan Xie, Chenxu Liu, Jieling He, Zhongyuan Liu, and Zeno Zeng. GameXpert-Bench: How Far Are Coding Agents from Expert Game Development?, 2026. 1, 4arxiv.org/abs/2608.21833
- [10]Xiaoyu Chen, Lai Wei, Jin Wang, Xiangyu Zou, Rouchen Fan, Enze Luo, Mingzhe Yao, Jiahui Zhu, Yuhua Wen, Linghe Kong, and Weiran Huang. SWE-Game: Can Coding Agents Build the Games We Want?, 2026. 1, 4arxiv.org/abs/2609.33678
- [11]Wayne Chi, Yixiong Fang, Arnav Yayavaram, Siddharth Yayavaram, Seth Karten, Qiuhong Anna Wei, Runkun Chen, Alexander Wang, Valerie Chen, Ameet Talwalkar, and Chris Donahue. GameDevBench: Evaluating agentic capabilities through game development. In Proceedings of the 43rd International Conference on Machine Learning (ICML), 2026. 2, 13arxiv.org/abs/2602.11103
- [12]Wei-Lin Chiang, Lianmin Zheng, Ying Sheng, Anastassios Nikolas Angelopoulos, Tianle Li, Dacheng Li, Banghua Zhu, Hao Zhang, Michael J. Jordan, Joseph E. Gonzalez, and Ion Stoica. Chatbot arena: An open platform for evaluating LLMs by human preference. In Proceedings of the 41st International Conference on Machine Learning (ICML), 2024. 2, 13
- [13]Zixuan Dong, Yanghai Wang, Hanyan Bian, Jiaxi Tao, Junze Li, Jiadong Wang, Xinyu Che, Lin Liu, Baoyun Peng, Xiaodong Wang, and Jiaheng Liu. From code to playable systems: Cegame-bench for long-horizon game creation agents, 2026. 1, 4arxiv.org/abs/2610.26098
- [14]Sam Earle, Samyak Parajuli, and Andrzej Banburski-Fahey. DreamGarden: A designer assistant for growing games from a single prompt. In Proceedings of the 2025 CHI Conference on Human Factors in Computing Systems (CHI), 2025. 2, 13
- [15]Emmanuel Guardiola. The gameplay loop: a player activity model for game design and analysis. In Proceedings of the 13th International Conference on Advances in Computer Entertainment Technology, pages 1–7. ACM, 2016. 2
- [16]Sirui Hong, Mingchen Zhuge, Jonathan Chen, Xiawu Zheng, Yuheng Cheng, Jinlin Wang, Ceyao Zhang, Zili Wang, Steven Ka Shing Yau, Zijuan Lin, Liyang Zhou, Chenyu Ran, Lingfeng Xiao, Chenglin Wu, and Jürgen Schmidhuber. MetaGPT: Meta programming for a multi-agent collaborative framework. In The Twelfth International Conference on Learning Representations (ICLR), 2024. 2, 13arxiv.org/abs/2308.00352
- [17]Wenqi Huang, Charley Lee, Leonard Tng, and Serena Ge. Deepswe: Measuring frontier coding agents on original, long-horizon engineering tasks, 2026. 1arxiv.org/abs/2607.07946
- [18]Yixu Huang, Bo Li, Na Li, Zhe Wang, Kaijie Chen, Haonan Ge, Qingyi Si, Yuanzhe Shen, Ruihan Yang, Guangjing Wang, and Hongcheng Guo. GUI agents for continual game generation. In Findings of the Association for Computational Linguistics: EMNLP 2026, 2026. 2, 13arxiv.org/abs/2605.28258
- [19]Allison Sihan Jia, Daniel Huang, Nikhil Vytla, Seung Won Wilson Yoo, Nirvika Choudhury, Shayak Sen, John C. Mitchell, and Anupam Datta. What is your agent’s GPA? a framework for evaluating agent goal-plan-action alignment, 2025. 8arxiv.org/abs/2510.08847
- [20]Chaobo Jia, Ruize Wan, Tianyu Sun, Weihao Tan, Borui Wan, Yuxuan Tong, Guangming Sheng, and Huaiyu Xu. GameGen-Verifier: Parallel keypoint-based verification for LLM-generated games via runtime state injection. arXiv preprint arXiv:2605.07442, 2026. 4, 13arxiv.org/abs/2605.07442
- [21]Yilei Jiang, Jinyuan Hu, Qianyin Xiao, Yaozhi Zheng, Ruize Ma, Kaituo Feng, Jiaming Han, Tianshuo Peng, Kaixuan Fan, Manyuan Zhang, and Xiangyu Yue. OpenGame: Open agentic coding for games. arXiv preprint arXiv:2604.18394, 2026. 2, 13arxiv.org/abs/2604.18394
- [22]Carlos E. Jimenez, John Yang, Alexander Wettig, Shunyu Yao, Kexin Pei, Ofir Press, and Karthik R. Narasimhan. SWE-bench: Can Language Models Resolve Real-World GitHub Issues?, 2024. 1arxiv.org/abs/2310.06770
- [23]Carlos E. Jimenez, John Yang, Alexander Wettig, Shunyu Yao, Kexin Pei, Ofir Press, and Karthik R. Narasimhan. SWE-bench: Can language models resolve real-world GitHub issues? In The Twelfth International Conference on Learning Representations (ICLR), 2024. 13arxiv.org/abs/2310.06770
- [24]Brian La, Sejoon Chang, Ben Kim, Junyoung Bae, Aamish Ahmad Beg, Sei Chang, Gonzalo Gonzalez-Pumariega, and Kanav Goyal. GameEngineBench: Evaluating coding agents on real C++ runtime environments. arXiv preprint arXiv:2607.03525, 2026. 2, 13arxiv.org/abs/2607.03525
- [25]Xiangyi Li, Wenbo Chen, Yimin Liu, Shenghan Zheng, Xiaokun Chen, Yifeng He, Yubo Li, Bingran You, Haotian Shen, Jiankai Sun, et al. SkillsBench: Benchmarking how well agent skills work across diverse tasks. arXiv preprint arXiv:2602.12670, 2026. 2, 20arxiv.org/abs/2602.12670
- [26]Shuyang Liu, Saman Dehghan, Jatin Ganhotra, Martin Hirzel, and Reyhan Jabbarnvand. From plan to action: How well do agents follow the plan?, 2026. 8DOI
- [27]Zimu Lu, Yunqiao Yang, Houxing Ren, Haotian Hou, Han Xiao, Ke Wang, Weikang Shi, AoJun Zhou, Mingjie Zhan, and Hongsheng Li. WebGen-Bench: Evaluating LLMs on generating interactive and functional websites from scratch. In Advances in Neural Information Processing Systems (NeurIPS), Datasets and Benchmarks Track, 2025. 13arxiv.org/abs/2505.03733
- [28]Tongxu Luo, Rongsheng Wang, Jiaxi Bi, Chenming Xu, Zhengyang Tang, Jianlong Chen, Juhao Liang, Ke Ji, Shuqi Guo, Yuhao Du, Fan Bu, Wenyu Du, Xiaotong Zhang, Kyle Li, Shaobo Wang, Linfeng Zhang, Yuxuan Liu, Xin Lai, Chenxin Li, Yiduo Guo, Zhexin Zhang, Xinyuan Wang, Tianyi Bai, Ziniu Li, and Benyou Wang. GameCraft-Bench: Can Agents Build Playable Games End-to-End in a Real Game Engine?, 2026. 1, 4arxiv.org/abs/2606.17861
- [29]Tongxu Luo, Rongsheng Wang, Jiaxi Bi, Chenming Xu, Zhengyang Tang, Jianlong Chen, Juhao Liang, Ke Ji, Shuqi Guo, Yuhao Du, Fan Bu, Wenyu Du, Xiaotong Zhang, Kyle Li, Shaobo Wang, Linfeng Zhang, Yuxuan Liu, Xin Lai, Chenxin Li, Yiduo Guo, Zhexin Zhang, Xinyuan Wang, Tianyi Bai, Ziniu Li, and Benyou Wang. GameCraft-Bench: Can agents build playable games end-to-end in a real game engine? arXiv preprint arXiv:2606.17861, 2026. 2, 13arxiv.org/abs/2606.17861
- [30]Majid Manzarpour. Three.js game skills: Agent skills for building playable, polished Three.js browser games. https://github.com/majidmanzarpour/threejs-game-skills, 2026. Includes threejs-game-director and threejs-3d-generator. 2
- [31]Mike A. Merrill, Alexander G. Shaw, Nicholas Carlini, Boxuan Li, Harsh Raj, Ivan Bercovich, Lin Shi, Jeong Yeon Shin, Thomas Walshe, E. Kelly Buchanan, Junhong Shen, Guanghao Ye, Haowei Lin, Jason Poulos, Maoyu Wang, Marianna Nezhurina, Jenia Jitsev, Di Lu, Orfeas Menis Mastromichalakis, Zhiwei Xu, Zizhao Chen, Yue Liu, Robert Zhang, Leon Liangyu Chen, Anurag Kashyap, Jan-Lucas Uslu, Jeffrey Li, Jianbo Wu, Minghao Yan, Song Bian, Vedang Sharma, Ke Sun, Steven Dillmann, Akshay Anand, Andrew Lampoutakoun, Bardia Koopah, Changran Hu, Etash Guha, Gabriel H. S. Dreiman, Jiacheng Zhu, Karl Krauth, Li Zhong, Niklas Muennighoff, Robert Amanfu, Shangyin Tan, Shreyas Pimpalkoankar, Tushar Aggarwal, Xiangming Lin, Xin Lan, Xuandong Zhao, Yiqing Liang, Yuanli Wang, Zilong Wang, Changzhi Zhou, David Heineman, Hange Liu, Harsh Trivedi, John Yang, Junhong Lin, Manish Shetty, Michael Yang, Nabil Omi, Negin Raoof, Shanda Li, Terry Yue Zhuo, Wuwei Lin, Yiwei Dai, Yuxin Wang, Wenhao Chai, Shang Zhou, Dariush Wahdany, Ziyu She, Jiaming Hu, Zhikang Dong, Yuxuan Zhu, Sasha Cui, Ahson Saiyed, Arinbjörn Kolbeinsson, Jesse Hu, Christopher Michael Rytting, Ryan Marten, Yixin Wang, Alex Dimakis, Andy Konwinski, and Ludwig Schmidt. Terminal-bench: Benchmarking agents on hard, realistic tasks in command line interfaces, 2026. 1
- [32]Meshy. Meshy 3D agent: AI agent skills for the Meshy 3D generation platform. https://github.com/meshy-dev/meshy-3d-agent, 2026. Includes meshy-3d-generation for text/image-to-3D meshes and PBR texturing. 2, 21
- [33]OpenAI. Develop web game skill — OpenAI skills catalog for Codex. https://github.com/openai/skills, 2026. Playwright-driven iterative web game development and deterministic frame-stepping skill (develop-web-game). 2, 21
- [34]Zhiyuan Peng, Wei Tao, Xin Yin, Chenhao Ying, Yuan Luo, and Yiwen Guo. PlayCoder: Making LLM-generated GUI code playable. In Proceedings of the ACM on Software Engineering (FSE), 2026. 2, 13
- [35]Chen Qian, Wei Liu, Hongzhang Liu, Nuo Chen, Yufan Dang, Jiahao Li, Cheng Yang, Weize Chen, Yusheng Su, Xin Cong, Juyuan Xu, Dahai Li, Zhiyuan Liu, and Maosong Sun. ChatDev: Communicative agents for software development. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 15174–15186, 2024. 2, 13arxiv.org/abs/2307.07924
- [36]Jianwen Sun, Chuanhao Li, Zizhen Li, Yukang Feng, Fanrui Zhang, Yifei Huang, Yu Dai, and Kaipeng Zhang. JAMER: Project-level code framework dataset and benchmark on professional game engines. In Proceedings of the 2026 Conference on Empirical Methods in Natural Language Processing (EMNLP), 2026. 2, 13arxiv.org/abs/2606.19830
- [37]The Khronos 3D Formats Working Group. glTF 2.0 specification — runtime 3D asset delivery (ISO/IEC 12113:2022). https://www.khronos.org/gltf/2022. Royalty-free specification for efficient transmission and loading of 3D scenes, PBR materials, and KTX 2.0 textures. 21
- [38]Aryan Vichare, Anastasios N. Angelopoulos, Wei-Lin Chiang, Kelly Tang, and Luca Manolache. WebDev Arena: A live LLM leaderboard for web app development. https://arena.ai/blog/webdev-arena, 2025. 2, 13
- [39]Jesse Vincent. Superpowers: An agentic skills framework and software development methodology. https://github.com/obra/superpowers, 2025. Includes writing-plans, subagent-driven-development, and executing-plans. 2, 21
- [40]Guanzhi Wang, Yuqi Xie, Yunfan Jiang, Ajay Mandlekar, Chaowei Xiao, Yuke Zhu, Linxi Fan, and Anima Anandkumar. Voyager: An open-ended embodied agent with large language models. Transactions on Machine Learning Research (TMLR), 2024. 2, 20arxiv.org/abs/2305.16291
- [41]Renjun Xu and Yang Yan. Agent skills for large language models: Architecture, acquisition, security, and the path forward. arXiv preprint arXiv:2602.12430, 2026. 2, 20arxiv.org/abs/2602.12430
- [42]Haoyang Yan, Min le Su, Hangfan Zhang, Zhanhao Li, Chen Zhang, Shao Zhang, Yang Chen, Lei Bai, and Shuyue Hu. Harness-of-Harness: Multi-Day Autonomous Software Development with Continual Improvement, 2026. 1arxiv.org/abs/2609.01481
- [43]Runxin Yang, Yuxuan Wan, Shuqiang Li, and Michael R. Lyu. 90% faster, 100% code-free: MLLM-driven zero-code 3D game development. In Proceedings of the IEEE/ACM 48th International Conference on Software Engineering (ICSE), 2026. 13arxiv.org/abs/2509.26161
- [44]Lei Yin, Wentao Cheng, Zhida Qin, Tianyu Huang, Yidong Li, and Gangyi Ding. AutoUE: Automated generation of 3D games in Unreal Engine via multi-agent systems. In Findings of the Association for Computational Linguistics: ACL 2026, 2026. 2, 13
- [45]Wei Zhang, Jian Yang, Renshuai Tao, Linzheng Chai, Shuyue Guo, Jiajun Wu, Xiaoming Chen, Ganqu Cui, Ning Ding, Xander Xu, Hu Wei, and Bowen Zhou. V-GameGym: Visual game generation for code large language models. In Findings of the Association for Computational Linguistics: ACL 2026, pages 5613–5641, 2026. 2, 13arxiv.org/abs/2509.20136
- [46]Wenyu Zhang, Guoliang You, Tianlun, Haotian Zhao, Tianshu Zhu, Haoran Wang, Xiaoxuan Tang, Mingyang Dai, Jingnan Gu, Daxiang Dong, and Jianmin Wu. WebGameBench: Requirement-to-Application Evaluation for Coding Agents via Browser-Native Games, 2026. 1, 4arxiv.org/abs/2605.17637
- [47]Xiuhui Zhang, Yi Chen, Shusheng Xu, Fan Li, Huan Wang, Tongkai Yang, and Binhang Yuan. GameASG-Bench: Benchmarking autonomous game generation for game development. arXiv preprint arXiv:2609.21293, 2026. 1, 2, 4, 13arxiv.org/abs/2609.21293
- [48]Yan Zheng, Xiaofei Xie, Ting Su, Lei Ma, Jianye Hao, Zhaopeng Meng, Yang Liu, Ruimin Shen, Yingfeng Chen, and Changjie Fan. Wuji: Automatic online combat game testing using evolutionary deep reinforcement learning. In Proceedings of the 34th IEEE/ACM International Conference on Automated Software Engineering (ASE), pages 772–784, 2019. 13DOI
