WorksComputer Science
Design Anything, Design at Scale
Abstract
As multimodal resources are transformed into media outputs, agentic AI is expected to stay aligned with design intent over long horizons. Designers follow briefs specifying goals, constraints, and deliverables, with later work building on earlier decisions. Recent work reuses design experience across tasks, yet alignment among artifacts within one project remains challenging. Agents therefore need to preserve and apply this intent across domains and as projects grow. We introduce Design Anything, a graph-based harness that preserves design intent outside the model and binds deliverables to the workflow steps that produce them through an intent-execution graph. A task contract retains constraints and decisions, selected designs become explicit inputs to later generations, and a deterministic controller uses bindings and execution records to check output types and counts, identify outstanding deliverables, and guide subsequent turns. We also introduce DesignAnything-Bench, with 228 tasks across 19 design domains, including 38 project families at scales of 6, 14, and 24 artifacts. With the LLM held fixed, Design Anything satisfies 85.0% and 92.5% of the brief's factual constraints with DeepSeek V4.1 Flash High and GPT-5.6 Sol High, respectively, exceeding the strongest same-model baselines by 15.4% and 14.4%. These gains reach 25.1% and 23.0% on 24-artifact tasks. Under each model, Design Anything leads or ties in constraint satisfaction in 18 of 19 domains. Professional designers also prefer its complete deliverable sets over those of same-model baselines in a blind evaluation.
DA Project page https://designanything.gemus.ai
Benchmark and evaluator https://github.com/Gemus-AI/DesignAnything
Dataset https://huggingface.co/datasets/Gemus-AI/DesignAnything-Bench
Gemus platform https://gemus.ai/community/collections/design-anything
Contact: ✉[email protected] ✉[email protected] ✉[email protected]
Introduction
Human design activity often begins with a specific need, such as developing a product, staging an exhibition, or building a brand. Designers work from a design brief, which states the initial design intent: the project’s goals, constraints, and required deliverables. Such projects usually call for a set of coordinated artifacts, including concept images, renderings, specification documents, videos, and slide decks, and later work builds on earlier design decisions, such as the selected concept and references [7, 8, 4].
Generative models and design tools can now produce individual artifacts [25, 29, 38, 6] (ByteDance Seed 2026). Recent work on agentic artifact creation highlights the challenge of preserving requirements and earlier decisions throughout multi-step production [30]. Long-horizon tasks remain difficult [13, 20, 33]. Models do not reliably use the information in long contexts [19, 9], perform worse when requirements arrive over multiple turns [14], and, in long tool-use runs, stop too early, verify too little, and lose constraints [37, 23]. Staying aligned with design intent means delivering every requested artifact, meeting each constraint in every artifact it concerns, and carrying earlier design decisions forward so that the artifacts stay consistent; all three become harder as deliverables multiply. An agent’s performance depends not only on its large language model (LLM) but also on its harness, the software layer that manages tool calls, context, and multi-step execution [36, 27, 15, 32, 35, 1, 21].
Existing studies have explored harness mechanisms for sustained execution, including persistent requirement records and independent review of completion status [22], as well as canvases that organize artifacts and their reference relationships [18]. Other studies reuse design experience accumulated over earlier tasks [21]. These approaches either rely on a model to judge whether the work is complete or check one artifact at a time. Graph-based agent orchestration, recently discussed as graph engineering, makes task dependencies and execution state explicit [12, 31, 5]. For long-horizon design, the challenge for a harness is to ensure that recorded constraints and design decisions continue to guide artifact production and to reliably identify unfinished deliverables. Therefore, this raises a natural question: how can agentic AI consistently align with design intent over long-horizon tasks?
To this end, we propose Design Anything, a graph-based harness for long-horizon agentic design whose key idea is to keep design intent outside the model, in an explicit state the harness can check, and to bind each deliverable to the workflow steps that produce it; we call this state the intent–execution graph (Figure 3). A persistent task contract records the deliverables, constraints, and design decisions made during the run, so later turns still see them. When a later artifact builds on a selected concept or a supplied reference, the agent links it into the graph as an explicit input, so the artifact is generated from it rather than from a fresh description (Figure 14). After each turn, a deterministic controller uses the bindings, the execution records, and checks on output kind and count to find the outstanding deliverables, and starts another turn that names them; a run succeeds only when none remain. The same configuration serves every domain (Figure 2; Appendix § 9).
Figure 2. Design anything across twelve design domains. Each group shows artifacts from a single run of Design Anything with GPT-5.6 Sol High, using the same configuration for every domain, without skills written for a particular design domain or task-specific hints.
Figure 3. Overview of Design Anything. Within a run (❶), the intent–execution graph (❷) records design intent and binds deliverables to workflow nodes. The agent loop (❸) builds and executes the workflow; the delivery loop (❹) uses execution records to identify outstanding deliverables and decide the next action.
Figure 14. Recorded cross-modal dependencies in the LUMA project. This DeepSeek V4.1 Flash High task requests 24 artifacts in nine numbered groups. Selected real outputs show how one run reuses the chosen Continuum direction board across design boards, video, and a presentation; orange arrows mark recorded input connections. The bottom panel redraws an excerpt of the workflow graph that the agent built in this run: the 18 recorded connections that take the selected Continuum board as an input; text prompts, provided references, and the other 61 connections are omitted.
Evaluating such a harness requires a suitable benchmark. Some existing benchmarks include professional tasks with several deliverables ([17] (Sun et al. 2026)), among them design and creative tasks ([17]); others focus on workflow construction, tool planning, UI reproduction, poster generation, or single-image quality ([34, 11, 10, 28, 26, 16]). However, results at one project scale, the number of requested artifacts, do not show whether an agent stays aligned with design intent as the same project grows. We introduce DesignAnything-Bench for this purpose, with 228 tasks across 19 design domains, including 38 project families each posed at 6, 14, and 24 requested artifacts with shared background and inputs. Design Anything satisfies 85.0% of the brief’s factual constraints (such as names and dimensions) with DeepSeek V4.1 Flash High and 92.5% with GPT-5.6 Sol High. Against the strongest baseline on the same LLM, it leads by 15.4% and 14.4% over all tasks and by 25.1% and 23.0% on 24-artifact tasks (Figure 1), and it leads or ties in 18 of 19 domains under each model (Figure 5). In a blind evaluation of complete output sets by professional designers, the two Design Anything configurations receive the two highest estimated preference scores among the five configurations. Our contributions are threefold:
Figure 1. Design Anything outperforms the best same-model baseline at every scale. Colored bars show the strongest baseline and purple segments the gain of Design Anything; labels give its rate and lead in percentage points.
Figure 5. Figure 5 Design Anything leads or ties in 18 of 19 domains per model. Cells show constraint satisfaction (%) on 12 tasks per domain; bold marks the within-model best. Parentheses give applicable-constraint counts. DS and GPT denote DeepSeek V4.1 Flash High and GPT-5.6 Sol High.
Design Anything Harness. We propose Design Anything, a harness that keeps design intent outside the model in an intent–execution graph binding each deliverable to the workflow steps that produce it.
DesignAnything-Bench. We introduce DesignAnything-Bench, a 228-task benchmark spanning 19 design domains, with 38 project families scaled from 6 to 24 artifacts, to test whether agents stay aligned with design intent across domains and as projects grow.
Evaluation. With the LLM fixed, Design Anything satisfies more constraints than the strongest baseline overall and at every project scale, leads or ties in 18 of 19 domains, and receives the highest designer preference.
Method
Design Anything keeps design intent in an explicit state outside the model and operates on it with two loops (Figure 3). After formalizing what alignment with design intent requires of a run (❶), we describe this state, the intent–execution graph (❷), which binds each deliverable to the workflow steps that produce it. The agent loop (❸) builds and executes this graph, and the delivery loop (❹) checks it to find unfinished deliverables and decide the next action.
Problem formulation
Given a design brief and reference materials , we hold the LLM fixed and write the artifacts exported by Design Anything as . The brief states the project’s goals, constraints , and deliverables . Several deliverables may share a selected concept, as with product views, a specification document, and a slide deck. In this setting, staying aligned with design intent requires a run (❶ in Figure 3) to produce an artifact for every , meet each constraint in every deliverable it concerns, and carry earlier design decisions into later artifacts. These properties are measured from exported artifacts (§ 3.2).
For constraint satisfaction, the expected task-level score is
where is the task distribution and is defined in Appendix § 6.§ 3. As deliverables multiply, each turn needs the relevant constraints and earlier decisions, together with the current delivery status.
Intent–execution graph
The intent–execution graph (❷ in Figure 3) stores the task contract , workflow graph with nodes , and execution records :
Here contains recorded design decisions, and binds deliverables to their producing nodes.
The task contract keeps deliverables, constraints, and recorded design decisions available across turns, together with goals, success criteria, and open questions. The same contract format and delivery rules apply across domains; domain-specific constraints come from the brief.
To carry selected designs into later artifacts, the execution layer represents their reuse as explicit data dependencies in , as in graph-based agent orchestration [31, 5]. is a typed directed acyclic graph that only the agent edits during a run, through tool calls (§ 2.3). Edges connect compatible output and input ports. Each node , such as producing in Figure 3, computes its output from its configuration and the outputs of its parent nodes .
A selected concept or supplied reference connected this way enters later generation as an explicit input, rather than being reconstructed from a fresh description. In Figure 3, node is configured to use ’s output as an input; Figure 14 illustrates such connections in a recorded design project. Appendix § 7.5 measures how often generations receive reference inputs.
To determine unfinished work, the binding connects deliverables in to their producing nodes in . Together with , which records node states and controller decisions, it lets the delivery loop derive outstanding deliverables (§ 2.4). Appendix § 5 gives the implementation details.
Agent loop
The agent loop builds and revises this state through checked tool operations (❸ in Figure 3). Each turn lasts until the model stops calling tools. The model observes , comprising the canvas that renders and its artifacts and a state summary regenerated between turns.
The model selects operations from four tool families, in any order: PLAN records or revises intent in ; BUILD creates or edits nodes, edges, and deliverable bindings in ; EXECUTE runs nodes; and AUDIT reads , , and . An operation takes effect only if a deterministic validity check accepts it:
where is the operation’s state transition. checks tool arguments, node configurations, references, port types, and bindings; a rejected operation leaves the state unchanged and returns an actionable error.
Delivery loop
The delivery loop uses the bindings to identify unfinished deliverables when the model ends a turn (❹ in Figure 3). A deterministic controller computes delivery status from and :
Here requires every bound executable node to have finished executing, and requires the outputs counted for to belong to that deliverable and pass the checks on output kind and item count. Thus includes both unbound deliverables and those whose bound work remains unfinished, such as before its node produces in Figure 3.
To direct subsequent work, let contain running jobs and delegated tasks, and contain questions and decisions awaiting input. The controller also runs when a job finishes or an awaited input arrives. A run ends successfully only when
A model’s statement that it has finished is therefore insufficient (❶ in Figure 3). After applying failure checks, the controller waits while or is non-empty, ends the run when Stop holds, and otherwise starts a new turn naming the outstanding deliverables and their blocking reasons. Failures or lack of progress can instead end the run unsuccessfully; Appendix § 5 specifies the control order and stopping rules.
The controller checks delivery using declared node capabilities and output item counts. Constraint satisfaction and consistency are evaluated from the exported artifacts (§ 3). We also compare the full harness with reduced execution configurations to assess delivery and efficiency (§ 3.5).
Experiments
Holding the LLM fixed, we ask: (1) Can one harness fulfill briefs across domains (Anything)? (2) Can it do so as projects require more artifacts (At Scale)? (3) How does the full harness compare with reduced configurations in delivery and efficiency? We also ask which delivered sets professional designers prefer.
Benchmark: DesignAnything-Bench
Task composition
The benchmark spans 19 design domains with 12 tasks each (Figure 4). Tasks supply English briefs, input materials, and delivery requirements. For matched-scale comparisons, 38 project families share background and inputs across 6-, 14-, and 24-artifact versions (114 tasks). The 57 reference-material tasks test matching supplied objects; 57 breadth tasks add subjects and audiences. Each of these independent projects requests 6 or 14 artifacts. In total, 228 tasks represent 152 distinct projects (Appendix § 6.§ 1).
Figure 4. DesignAnything-Bench contains 228 tasks across 19 domains. In (b), the inner ring shows task conditions; the outer ring shows requested artifact counts. Only matched-scale tasks reuse the same 38 families across scales; each other task is an independent project at one scale. Artifact count measures scale.
Evaluation measures
In our work, delivery completion, constraint satisfaction, and cross-artifact consistency measure observable aspects of alignment in exported artifacts. Automatic evaluation follows Agent-as-a-Judge Zhuge et al., [39] (Appendix Table 3).
Requirement fulfillment measures how well outputs meet the brief. Delivery completion is the share of requested files delivered, capped per category. Constraint satisfaction is the share of applicable factual constraints satisfied in every eligible artifact. For illustration, delivering three requested posters gives 100% delivery, but a name required on all three and correct on only two fails that constraint. Table 1 pools satisfied constraints, not fully successful tasks. Reference fidelity measures the share of output–reference pairs depicting the same object (Appendix § 6.§ 3).
Cross-artifact consistency measures agreement among outputs. Identity consistency compares outputs with a designated output from the same run (the anchor), measuring the share of pairs depicting the same object. Reference fidelity instead uses a supplied asset. Set consistency scores shared visual rules, such as palette and typography, on a 100-point rubric.
Design quality (100-point scale) covers technical quality, aesthetics, text rendering, and domain criteria (Li et al., [16]). Aggregate image-checklist means are multiplied by aggregate delivery completion, reflecting both image quality and missing work (Appendix § 6.§ 3). A vision-language judge scores the checklists; Appendix § 6.§ 4 documents calibration of the other automatic judges against human annotations.
Designer preference asks professional designers which delivered set they would hand to the client, comparing outputs for the same brief without seeing system names and allowing ties. Table 1 summarizes these choices using Bradley–Terry estimates (§ 3.6).
Experimental setup
Systems. In the main comparison, a configuration pairs a system with an LLM. We group configurations by LLM. On DeepSeek V4.1 Flash High, we compare Design Anything with the design agents OpenDesign (Open Design contributors, 2026) and JarvisHub (Lin et al., [18]); on GPT-5.6 Sol High, we compare it with Codex CLI. All systems generate images with gpt-image-2.5-sunburst. These comparisons evaluate each complete system with its own tools and execution policies (Appendix Table 4).
Table 1 Results on DesignAnything-Bench (228 tasks). Fulfillment, identity consistency, and designer preference are percentages; other scores use a 100-point scale. Higher is better; bold marks the within-model best. Design quality is weighted by delivery completion; designer preference uses Bradley–Terry estimates (§ 3.6).
| System | Delivery completion | Constraint satisfaction | Reference fidelity | Identity consistency | Set consistency | Technical quality | Aesthetics | Text rendering | Domain criteria | Designer preference |
| DeepSeek V4.1 Flash High | ||||||||||
| Design Anything | 96.6 | 85.0 | 75.6 | 84.5 | 87.9 | 88.4 | 87.1 | 93.0 | 92.0 | 60.7 |
| OpenDesign | 95.2 | 69.6 | 55.0 | 75.2 | 88.3 | 85.1 | 82.3 | 80.6 | 85.5 | 21.8 |
| JarvisHub | 87.6 | 68.0 | 49.6 | 68.5 | 65.4 | 80.9 | 80.2 | 84.7 | 84.2 | 49.1 |
| GPT-5.6 Sol High | ||||||||||
| Design Anything | 97.3 | 92.5 | 79.3 | 90.2 | 87.7 | 88.8 | 88.5 | 93.8 | 92.5 | 74.0 |
| Codex | 98.1 | 78.1 | 71.3 | 75.3 | 89.8 | 88.9 | 87.9 | 94.3 | 90.8 | 44.4 |
Table 1.
Protocol. All systems receive the same briefs and materials except on three tasks of one book project family (Appendix § 6.§ 1), without skills written for a particular design domain or task-specific hints. Each task contributes one final result per configuration, scored by a common evaluator. We compare scores on the same tasks and estimate uncertainty using 95% bootstrap intervals over project families (Appendix § 7.§ 4).
Benchmark results
Overall and cross-domain results
Overall comparison. Across 228 tasks in 19 domains (Table 1), Design Anything leads each LLM group on constraint satisfaction (85.0% on DeepSeek; 92.5% on GPT), reference fidelity, and identity consistency. OpenDesign and Codex score higher on set consistency; Codex also has higher point estimates for delivery, technical quality, and text rendering. JarvisHub leaves 12.4% of artifacts undelivered. Design Anything’s strengths thus lie in satisfying constraints and preserving object identity across outputs, one aspect of carrying design decisions forward. Appendix § 7.5 examines reference reuse and its association with identity consistency within JarvisHub.
Cross-domain results. Figure 5 tests how widely this advantage extends across the 19 domains, with 12 tasks per domain. Design Anything leads or ties in 18 under each LLM. JarvisHub leads in product design on DeepSeek, and Codex in exhibition design on GPT. This broad coverage supports Anything: our method fulfills constraints across diverse design domains.
_Qualitative comparison._ Figure 6 compares five configurations on one 14-artifact animation brief, a railway story in which two otters swap matching cases. Both Design Anything configurations and Codex are credited with all 14 artifacts, JarvisHub with 13, and OpenDesign with 11. Credit, however, does not ensure correct content: JarvisHub turns the otters into humans in its color script, Codex reuses one image for two action phases, and Design Anything (DeepSeek) leaves the cases unreturned in its final keyframe. File counts alone thus cannot show whether a long-horizon project stays faithful to its brief. Enlarged previews appear in Figure 12–Figure 13.
Figure 6. Same-task comparison across five configurations. Red markers highlight selected content errors; slashed circles indicate items not credited by the evaluator.
Figure 12. Enlarged outputs for Figure 6: DeepSeek V4.1 Flash High. Each block shows one configuration’s 14 requested artifacts in deliverable order, with the markers of Figure 6.
Figure 13. Enlarged outputs for Figure 6: GPT-5.6 Sol High. The layout follows Figure 12.
Results at scale
To test At Scale, Figure 7 compares five configurations on 38 matched project families at 6, 14, and 24 artifacts. At 24, Design Anything achieves 85.7% constraint satisfaction on DeepSeek and 91.9% on GPT, leading OpenDesign by 25.1% and Codex by 23.0%. The constraint-satisfaction advantage thus persists as the same projects require more artifacts.
Figure 7. Design Anything retains its constraint-satisfaction advantage from 6 to 24 artifacts. Panels show (a) delivery completion and (b) constraint satisfaction on 38 matched families. Solid lines with filled markers denote DS; dashed lines with hollow markers denote GPT.
Design Anything’s advantage at larger scales lies in satisfying the brief’s constraints throughout the delivered set. On 24-artifact tasks, Codex delivers over 97% of requested artifacts but satisfies 68.9% of constraints, compared with 91.9% for Design Anything using the same LLM. Each constraint counts as satisfied only when it holds in every eligible artifact, so this gap reflects more reliable fulfillment across the set, beyond delivering the requested files. Paired bootstrap comparisons and sensitivity checks for missing deliveries support this conclusion (Appendix § 7.§ 4).
Figure 8 illustrates how the outputs expand across three matched architectural design tasks, from spatial views and drawings to shading details, documents, and video.
Figure 8. Design Anything (GPT-5.6 Sol High) on RIVER COMMONS with 6, 14, and 24 requested artifacts. Stacks denote documents; play icons denote videos. Counts give credited artifacts.
Ablation studies
To test the task-tracking and delivery mechanisms in Figure 3, we compare three configurations on 19 DeepSeek V4.1 Flash High tasks requesting 24 artifacts, one per domain. The reduced harness removes the delivery loop (❹), task contract (❷), and structured planning tools used by the agent loop (❸); Appendix Table 14 gives the full configurations. Stop at model end ends the run when the LLM stops calling tools; Generic prompt instead asks it to keep working. Complete uses the full harness.
Complete raises delivery from 73.9% to 96.1% and constraint satisfaction from 57.5% to 90.4% relative to Stop at model end, eliminating all six runs below half delivery (Table 2). Generic prompt also recovers these early stops, but issues 36.1 generation requests per run versus 24.1 for Complete. With numerically similar delivery (96.1% versus 96.9%), Complete incurs about one-third the mean LLM cost: $0.43 versus $1.30 on 17 paired tasks with complete telemetry.
| Configuration | Delivery (%) | Runs < 50% | Constraint (%) | Gen. requests | Cost ($)† |
| Stop at model end | 73.9 | 6 | 57.5 | 18.6 | 0.48 |
| Generic prompt | 96.9 (+23.0) | 0 | 75.3 (+17.8) | 36.1 | 1.30 |
| Complete (Design Anything) | 96.1 (+22.1) | 0 | 90.4 (+32.9) | 24.1 | 0.43 |
Table 2. Delivery-loop study on 19 paired 24-artifact tasks (DeepSeek V4.1 Flash High). Parentheses give gains over Stop at model end; “Runs < 50%” counts runs below half delivery. Requests and costs are per-run means; †cost uses LLM list prices on 17 paired tasks with complete telemetry.
Together, these mechanisms support high delivery with less repeated generation. Appendix § 8 reports paired tests and further planning-interface and state-summary studies.
Human evaluation
To assess the delivered sets from a professional design perspective, twelve designers in five disciplines compare outputs on 185 tasks, yielding 1,019 blind pairwise judgments. They choose which set they would hand to the client, with ties allowed. Design Anything is preferred over each baseline using the same LLM: designers choose it in 82% of judgments against OpenDesign, 70% against Codex, and 51% against JarvisHub, compared with 37% for JarvisHub and 12% ties (Figure 9b).
Figure 9. The two Design Anything configurations have the highest estimated designer preference. (a) Bradley–Terry estimates with 95% task-and-designer bootstrap intervals (1,019 judgments, 185 tasks, 12 designers). (b) Pairwise choices with ties; counts judgments. DA denotes Design Anything; the bold last row compares its DeepSeek and GPT configurations.
Our GPT and DeepSeek configurations rank first and second in estimated designer preference (74.0% and 60.7%), both with 95% intervals above 50% (Figure 9a). These Bradley–Terry estimates average each configuration’s fitted preference probabilities against the other four; direct comparisons between our two configurations connect the LLM groups. The pairwise choices and joint ranking show that designers more often select Design Anything’s delivered sets for client handoff, complementing the automatic measures with professional judgment of the project as a whole. Appendix § 6.5 gives the evaluation and estimation details.
Conclusion
We presented Design Anything, a harness that keeps design intent outside the model in an intent–execution graph, which binds each deliverable to the workflow steps that produce it, and DesignAnything-Bench, 228 long-horizon design tasks across 19 domains and three project scales. With the LLM fixed, Design Anything satisfies more of the brief’s constraints than the strongest baseline overall and at every project scale, leads or ties in 18 of 19 domains, and receives the highest designer preference. In the study of the delivery loop (§ 3.5), the full harness delivers nearly as many artifacts as a harness that answers every stop with a generic prompt to keep working, at about one-third of its LLM cost. Limitations are discussed in Appendix § 10.
Implementation and Runtime Details
Task contract
The task contract, the intent layer of the intent–execution graph, is stored as a structured project object visible on the canvas. It holds the final deliverable description, stable deliverable identifiers, public success criteria, constraints, decisions marked as coming from the user or the agent, and open questions. It holds no execution phase or completion counter, so revising it updates intent rather than recording progress.
Graph construction and artifact ownership
The binding is implemented through delivery groups, named sets of nodes. Graph-building operations accept typed node descriptions, configurations, connections, and the delivery group that receives them. They can create new nodes or associate eligible existing nodes with that group. Groups explicitly declare which contract deliverables they implement; an empty declaration denotes supporting work. Shared groups require output ownership information so that an artifact for one deliverable is not automatically counted toward another. Dependency declarations and port-level edges serve different purposes: the former express ordering between groups, while the latter carry concrete inputs to operations.
Operational delivery checks
For each deliverable, the controller finds its bound nodes through the delivery groups that claim it and through node ownership, and examines the status of the bound executable nodes; data-only inputs and annotations do not count as artifact-generating execution. In groups shared by several deliverables, an execution failure that cannot be attributed to one item remains a group-level failure signal rather than being assigned to an item.
Output-form checks use node capability declarations. An image-producing operation cannot satisfy a readable-document deliverable merely because its image resembles a document, and a minimum-page declaration requires an implementation capable of paginated output without verifying the realized page count. Quantity checks count supported output items rather than generation nodes. The benchmark’s artifact evaluation is therefore necessary even when every bound node has finished.
Observation and new turns
The state summary is a compact projection of workflow state regenerated between turns; detailed read tools expose individual nodes, connections, configurations, and outputs.
After a turn ends, the controller reads the workflow graph and the execution records and lists obligations for the root goal, deliverables, decisions, required questions, and execution. Each obligation is open, waiting, in progress, satisfied, failed, or cancelled. The controller then selects the next action in this order:
Stop on cancellation, loss of edit access, execution failures, or a repeated-error limit.
Wait if a turn, job, or delegated execution is still active, or if a decision requested from the user is still pending.
Terminate unsuccessfully on failed obligations, or successfully when the root obligation is satisfied and no unresolved obligation remains. The root obligation is derived, not declared: it is satisfied only when every deliverable is delivered and no required question, pending decision, or running execution remains.
For remaining work, check progress and resource limits; either start another model turn or stop with an explicit reason.
The message starting a new turn names the outstanding deliverables and, for each, the blocking reason, such as a missing output form or a quantity shortfall. The model then decides whether to read state, modify a contract, build or edit a graph, execute an operation, delegate work, or request a decision.
Progress monitoring separates obligation changes from observable workflow changes. Repeatedly restating a plan is not sufficient evidence of artifact progress. Progress limits stop repeated new turns that make no progress, while allowing active execution and pending decisions; the thresholds and run limits are listed in Appendix C.1.
Automatic verifier
When every deliverable is delivered, an automatic verifier reviews the workflow graph in the background: deterministic checks first list structural problems, such as unconnected nodes, and the verifier checks them against the success criteria and returns concrete findings to the agent. Visual design critique is available only as a tool the agent may call. Goal completion remains governed by the control layer. The verifier is enabled in the main comparison and the Complete delivery-loop configuration, and disabled in the two reduced delivery-loop configurations and all planning configurations. Recorded run settings are in Appendix C.1; component-study configurations are in Appendix D.
Benchmark Construction and Evaluation Protocol
This appendix describes task construction, output processing, metric definitions, judge calibration, and the designer study. Complete results and sensitivity analyses are in Appendix C; controlled component studies and visual evidence follow in Appendices D and E.
Task construction and benchmark independence
The 19 domains are product design; vehicle styling; furniture and home products; architecture; interior design; exhibition design; landscape design; fashion and textiles; jewelry and accessories; brand identity; graphic design and advertising; books and illustration; packaging; UI/UX; information design and data visualization; animation and storyboarding; game art and worldbuilding; e-commerce product imagery; and advertising and fashion photography. Figure 4 summarizes their equal representation. The task counts below distinguish scale variants from distinct project identities.
| Task condition | Tasks/domain | Tasks by artifact count | Total tasks | Projects | ||
| 6 | 14 | 24 | ||||
| Matched scales | 38 | 38 | 38 | 114 | 38 | |
| Reference materials | 3 | 26 | 31 | 0 | 57 | 57 |
| Breadth | 3 | 19 | 38 | 0 | 57 | 57 |
| Total | 12 | 83 | 107 | 38 | 228 | 152 |
Table 1 (PDF p. 16).
Each domain contributes two base project families: one asks the system to propose alternatives and select one, and the other fixes the direction in the brief. Each family has three scale versions with shared background and inputs, yielding tasks. Only the 14- and 24-artifact versions request process artifacts, such as alternative directions and specifications. The three reference-material and three breadth projects per domain are independent projects, adding 114 tasks. Each requests either 6 or 14 artifacts, with no additional scale variants of the same project. Thus tasks correspond to project identities. Only the base tasks support within-family scale comparisons.
Evaluation specifications. Each specification translates the public brief and supplied materials into scoring criteria. Requested categories specify expected quantities, mark core categories, and distinguish process from final deliverables. Additional rules specify factual constraints, source and target artifacts for comparisons, and when a criterion is inapplicable. Systems receive only the English brief and materials.
Benchmark independence. Tasks were authored with LLM assistance from a per-domain topic plan and reviewed by the authors, with English scenarios as the only system input. No internal label of Design Anything enters scoring. During development, Design Anything was run on a separate 20-task pilot corpus and on small subsets of the final corpus (5-task subsets and 13 base-scale tasks) to fix infrastructure failures. No brief was revised in response to any system’s score except under a recorded defect rule applied identically to all systems.
Each system kept whatever general-purpose skills its product bundles; none used skills written for a particular design domain, which could supply prebuilt design procedures, and none received task-specific hints.
Execution and scoring use frozen task specifications, and each run is scored against the specification of the brief it received. The three tasks of one book project family (6, 14, and 24 artifacts) exist in two versions: OpenDesign and JarvisHub received the earlier archival photography brief, while both Design Anything configurations and Codex received the later historical astronomy plate brief; the other 225 tasks have identical briefs and specifications for all systems.
Output processing and assignment
The evaluator consumes a package of delivered files and evaluation specifications, independent of the system’s internal representation. It can therefore compare a canvas-based workflow with a file-based agent without requiring the latter to reconstruct a canvas. Import records distinguish supplied inputs, primary deliverables, and derived representations. Multiple file formats of the same work are associated to avoid counting a preview and its source as two deliveries. Input materials copied into an output directory remain inputs, identified by their content.
HTML decks are rendered by their own page structure, and PDF and presentation files keep their page and text information; a file whose pages cannot be read makes a page minimum unmeasurable rather than failed. Rendered deck pages can be inspected for consistency but do not count as separately requested artifacts.
To assign images to requested categories, the classifier scores each category and an unassigned option for every image; the assignment fills each category up to its expected quantity and uses each image at most once. Remaining images may be recorded as overflow without raising completion above the expected quantity. This matching uses the artifact and the specification, not Design Anything node labels, group names, or execution order. The classifier and evaluator versions are given in Appendix § 7, and the classifier’s calibration in Appendix § 6.§ 4.
Measures, eligibility, and aggregation
Table 3 summarizes the measures used in the main comparison. The definitions below follow the same order and specify the evidence and counting unit of each measure.
| Measure | Compared against | Definition | Scored by |
| Requirement fulfillment | |||
| Delivery completion | Brief: categories, quantities | Requested files delivered, capped per category | Classifier, count |
| Constraint satisfaction | Brief: factual constraints | Factual constraints satisfied in every targeted artifact (Eq. (7)) | Criterion check |
| Reference fidelity | Client’s reference assets | Output–reference pairs depicting the same object | Calibrated judge |
| Cross-artifact consistency | |||
| Identity consistency | The run’s anchor artifact | Output–anchor pairs depicting the same object | Calibrated judge |
| Set consistency | The run’s whole set | 100-point rubric score for shared visual rules | Calibrated judge |
| Design quality | |||
| Technical, aesthetics, text, domain | Image checklist and delivery requirements | Checklist mean × delivery completion (100-point scale) ([16]) | VLM judge, count |
| Human evaluation | |||
| Preference | Another system, same brief | Blinded pairwise choice; Bradley–Terry strength | Designers |
Table 3. Measures of DesignAnything-Bench. The measures assess fulfillment of the brief, agreement with the client’s materials or the run’s own outputs, design quality, and designer preference between systems on the same brief.
Delivery completion. For requested category with expected and delivered quantities and , delivery completion is ; excess outputs in one category cannot offset a shortfall in another. A file’s category assignment and delivered status are distinct from its compliance with format or content requirements. Page minima belong to the supplementary task-level outcomes in Appendix § 7.§ 3, rather than delivery completion.
Constraint satisfaction. Constraint satisfaction applies each constraint-specific criterion to its eligible targets. Let be the set of applicable factual constraints, the target set for constraint , and indicate whether artifact satisfies it. With the target eligibility and missing-delivery rules below, the task-level score is
Most targets are generated images (85–89% of target files for every system).
Constraint targets and unavailable evidence. Targets are delivered files in the named categories, capped at the requested quantity, or only each category’s final file when the specification says so. The following rules distinguish absent deliveries from unavailable evidence:
If none of the target categories contains a delivered file, the constraint counts as unsatisfied and stays in the denominator. If only some categories are empty, the constraint is checked on the delivered files alone.
Files whose text transcription failed are removed from the target set. The constraint is excluded if no targeted file remains after this removal.
The constraint is also excluded when its target categories are empty but a delivered file of an accepted type received no valid category from the classifier.
Constraint check. For each constraint, the evaluator transcribes artifact text, removes whitespace and dash characters, and normalizes full-width forms before exact, case-sensitive matching against the required wording. This defines .
Reference fidelity and cross-artifact consistency. Consistency comparisons are scoped to properties that the project is supposed to preserve. Alternative concepts need not look identical, and a legitimate refinement need not reproduce a sketch literally. The reference source determines the measure: agreement with a supplied reference asset is reference fidelity, a requirement-fulfillment measure, whereas agreement among outputs of the same run is identity consistency or set consistency. Unavailable pair judgments are distinguished from mismatches: those caused by the system’s own outputs stay in the denominators of reference fidelity and identity consistency, and those caused by an external reference are excluded.
Set consistency. The reported set-consistency score uses only rubric items labeled consistency, excluding separate compliance items. Item scores of 0, 1, and 2 map to 0, 60, and 100; inapplicable items are excluded and unmet items contribute zero. Within a task, the evaluator averages rule-level consistency means equally. Across tasks, it weights these task scores by their numbers of scored consistency items.
Design quality. The image-checklist component of design quality covers technical quality, aesthetics, text rendering, and domain-specific criteria. A criterion may be inapplicable, for example anatomical fidelity in an image without a person. The scores in Appendix § 7.7 use a checklist rubric judged by the same DeepSeek Flash model used for category assignment (Appendix § 7), with reasoning disabled. The judge sees each delivered image of a core image category on its own and marks every checklist item 0, 1, 2, or not applicable. Item marks are averaged within each sub-criterion, the sub-criteria are averaged within each of the four groups, and the result is mapped to a 100-point scale. A task’s score for a group averages the images within each core category and then averages over categories, so a category that asks for many images does not outweigh the others.
Let be the image-checklist score for task and group . A task has no score for a group when none of its judged images has an applicable item in that group; such a task is left out of that group’s average rather than counted as zero. With the set of tasks that have a score for group , design quality is
where is the system’s pooled delivery completion over all requested artifact types, as defined above. Both factors are kept at full precision, and only the product is rounded. Design quality combines the visual quality of delivered images with project delivery coverage; missing work reduces all four dimensions, and excess files do not offset shortfalls in other categories. It aggregates image-level assessments rather than introducing a whole-set visual rubric. Constraint satisfaction and reference fidelity separately assess compliance with the brief and supplied assets; set consistency assesses shared visual rules.
Rendering and missing scores. On pages that an importer rendered from HTML or rasterized from vector files, the three items for noise, edge clarity, and resolution are marked not applicable, because they measure our renderer rather than the design. An image whose judge call fails is left unscored. This judge has not been calibrated against designers on the present artifacts; Appendix § 7.7 instead reports a second judge on a stratified sample.
Aggregation. Delivery completion, constraint satisfaction, reference fidelity, and identity consistency pool their counting units over tasks. In particular, the reported constraint-satisfaction rate pools passed and applicable constraints; Equation (1) instead expresses expected task-level constraint satisfaction. Set consistency averages rule-level scores within a task and weights tasks by their numbers of scored items. For design quality, image-checklist scores average within each task and then over tasks, and the resulting mean is multiplied by pooled delivery completion (Equation (8)). Appendix § 7.§ 2 reports the corresponding observed counts.
Judge calibration
For the identity and set-consistency checks below, two professional designers independently labeled samples drawn from the final benchmark artifacts of the three DeepSeek-backed systems. The first five items of each sample are warm-up items and do not enter the statistics. A judge qualifies when Cohen’s and raw agreement is at least 85%. Human-judge agreement is computed on the items where both designers agree.
Identity judge. A single judge decides whether two images depict the same object, character, logo, or space; it serves both reference fidelity and identity consistency. Of 108 sampled pairs, 103 are analyzed. Human-human agreement is with 96.1% raw agreement. On the 99 consensus pairs, human-judge agreement is with 93.9%. By reference source, is 0.932 for pairs within a run (identity consistency, ) and 0.830 for pairs against supplied assets (reference fidelity, ). By subject, is 1.000 for characters, 1.000 for logos, 0.805 for products, and 0.826 for spaces.
Set-consistency judge. Of 117 sampled rubric items labeled consistency, 112 are analyzed. Human-human agreement is with 92.9%. On the 104 consensus items, human-judge agreement is with 92.3%. Removing the items that the judge answered not applicable leaves 100 adjusted items. On them, human-judge is 0.804 and 0.836 for the two designers and human-human is 0.901, so the qualification does not rest on not-applicable answers.
Direction of judge error. Agreement statistics do not show whether judge errors favor a particular system, so we also compare the judge with the designers per system on the consensus items. The identity judge labels more pairs as the same subject than the designers do: by 2.9 points for Design Anything, 2.9 for OpenDesign, and 14.3 for JarvisHub. On the 100-point scale, the set-consistency judge scores Design Anything 9.2 points below the designers, while it scores JarvisHub 1.6 and OpenDesign 2.0 points above them. Neither judge’s error favors Design Anything.
Category assignment. The classifier assigns delivered artifacts to the requested categories used to compute delivery completion. Calibration covers 120 tasks. Human-human agreement is Cohen’s ; human-classifier agreement is , using the two annotators’ consensus labels as the reference.
Text transcription. For text transcription used in constraint checks, human-human agreement is Cohen’s . Human-machine agreement is on the subset where the two annotators agree, using their consensus annotations as the reference.
Designer evaluation protocol
Participants and assignment. The designer study (§ 3.6) uses the same run of each system that the automatic measures score, so no run is selected by appearance. Twelve designers rate 185 tasks in five discipline groups: visual communication (four designers; brand identity, graphic design and advertising, packaging, books and illustration, information design, photography, and e-commerce imagery), digital interaction (two; UI/UX, game art, animation and storyboarding), product (two; product design, vehicle styling, furniture), spatial (two; architecture, interior, exhibition, and landscape design), and fashion and accessories (two). Within a group, tasks are stratified by domain, task condition, and project scale and dealt to designers in rotation; each designer first rates three warm-up tasks, and some tasks are rated a second time by another designer of the group.
Pairings and displayed evidence. Baseline comparisons pair systems within an LLM: Design Anything, OpenDesign, and JarvisHub pairwise on DeepSeek, and Design Anything against Codex on GPT; Design Anything on DeepSeek is also compared directly with Design Anything on GPT, which connects the two model groups on one Bradley–Terry scale. Each comparison shows every file the evaluator assigned to a requested category, in the brief’s deliverable order.
Whole-set judgments and estimation. Designers see the brief with a reference translation, never system names, reasoning, or completion claims. Sides are randomized. They choose the set they would rather hand to the client, answering left, right, or approximately equal. The study yielded 1,019 whole-set judgments, which support the results in § 3.6. Whole-set preference is estimated with a Bradley–Terry model ([2]), , fitted to the 1,019 judgments. A tie contributes half a win to each side. For each of the five configurations, we average its fitted preference probabilities against the other four, weighting opponents equally regardless of the number of observed comparisons. This is the probability of beating a uniformly sampled other configuration, not a raw head-to-head win fraction. We report 95% intervals from 2,000 bootstrap replicates over tasks and designers.
System Configurations and Complete Results
Controlled comparisons and experimental settings
Table 4 identifies what each reported comparison holds fixed and what it varies. The two main model groups compare complete systems: holding the LLM fixed does not hold media-generation tools, orchestration, or execution policies fixed, so their differences describe complete system configurations. No Codex run reached its 150-minute limit.
Table 4. Controlled comparison matrix. The final column states the comparison of interest; the rows include both interventions and observational analyses. The main model groups share briefs except on three tasks of one book project family (Appendix § 6.§ 1) and use a common evaluator. The delivery-loop study uses the same briefs and evaluator across its configurations; the planning and state-summary studies use separate tasks and harness versions (Appendix § 8).
| Comparison | Tasks | Held fixed | Varied factor |
| DeepSeek group (Table 1a) | 228 | DeepSeek V4.1 Flash High for every agent and sub-agent; gpt-image-2.5-sunburst image generation | Harness and agent: Design Anything, OpenDesign, JarvisHub |
| GPT group (Table 1b) | 228 | GPT-5.6 Sol High; gpt-image-2.5-sunburst image generation | Harness and agent: Design Anything, Codex |
| Project scale (§ 3.4.2) | 38 × 3 | Project family, background, and inputs; system | Requested artifacts: 6, 14, 24; process artifacts |
| Reference coverage (Appendix § 7.5) | 228 | DeepSeek V4.1 Flash High; within JarvisHub, the harness | Reference coverage (observed, not assigned) |
| Delivery loop (§ 3.5) | 19 | Design Anything on DeepSeek; 24-artifact tasks, one per domain | Delivery loop and other channels for tracking the task |
| Planning interface (Appendix § 8.§ 2) | 8 | Design Anything; other channels for tracking the task | Planning tools: structured, text, none |
| State summary (Appendix § 8.§ 3) | 10 pairs | Design Anything; direct read tools | Summary on or off |
| Human evaluation (§ 3.6) | 185 | Scored runs of each system | System within a model; the two Design Anything configurations compared directly |
All five configurations use the same 228 task identifiers, with frozen evaluation specifications matched to the brief version each received (corpus v0.6; Appendix § 6.§ 1). Four configurations are scored with evaluator v1.18. OpenDesign is scored with evaluator v1.19, which reads HTML deliverables as a browser displays them: it runs the local scripts they reference, and a full-page rendering may fill an image slot when the HTML file itself fills none. Only OpenDesign’s HTML deliverables draw their content with scripts, so only its runs were re-imported and rescored. Category assignment uses deepseek-flash with reasoning disabled, the same model alias used for the image-checklist component of design quality.
Each task contributes one final result per configuration. The reported tables aggregate these per-task evaluation records. Independent scorings of the same outputs differ by about three points on judged measures, so we do not interpret differences of that size.
Recorded runtime configuration. Table 5 lists the recorded settings; Appendix § 5 describes the mechanisms.
Table 5. Recorded runtime configuration of Design Anything. Settings apply to both configurations unless noted.
| Setting | Recorded configuration |
| Harness version | DeepSeek V4.1 Flash High runs: one controller version. GPT-5.6 Sol High runs: a later version of the same code that adds the retries listed under Controller limits and updates image analysis and image repair. |
| Agent and role models | Every role (orchestrator, builder, verifier, summarizer, and specialist helpers), text-generation nodes, and image analysis use the configuration’s LLM. High in the configuration names denotes the reasoning setting: DeepSeek V4.1 Flash runs at its default high reasoning, with reasoning disabled for image analysis; GPT-5.6 Sol receives reasoning effort high for the main agent and specialist helpers. |
| Generation backends | Images: gpt-image-2.5-sunburst, medium quality, 2K. Video: a third-party video model. |
| Skills | General-purpose skills bundled with the product; none written for a particular design domain |
| Other tools | Workspace reading and editing, contract and workflow construction, synchronous and batched execution, on-demand image analysis, clarifying questions, web search, memory, and delegation to helper agents |
| Context policy | Tool results are elided beyond 50% of the context window (1M tokens for DeepSeek V4.1 Flash High, 272k for GPT-5.6 Sol High); history is summarized beyond 85%, keeping the last six steps; the state summary is always on |
| Resource limits | Per turn: 120 tool calls, 120 model steps, 90 minutes. Per run: 100 turns. Per task: 150 minutes wall clock (DeepSeek V4.1 Flash High) or 150–240 minutes (GPT-5.6 Sol High). Tokens are recorded but not limited. |
| Checkpoint policy | A scripted user accepts approval and review requests, picks the recommended option (otherwise the first option that does not defer), and answers free-text questions with a fixed message stating that the brief is complete; a repeated question fails the run |
| Automatic verifier | On in the main comparison and the Complete delivery-loop configuration; off in the two reduced delivery-loop configurations and all planning configurations (Appendix § 8) |
| Setting | Recorded configuration (continued) |
| Controller limits | The run stops after two controller decisions without obligation progress, after three without new output, or after consecutive new turns for obligations; at most 100 turns per run. GPT-5.6 Sol High runs also retry on rate limits and transport interruptions. |
Metric counts and aggregate results
Table 6 gives the observed numerators, denominators, and coverage counts underlying the main results. Scoring definitions and eligibility rules are in Appendix § 6; differences in applicable evidence explain why some denominators vary across systems.
| System | Delivery | Constraints | Reference | Identity | Set items |
| Design Anything (DS) | 2809/2908 | 767/902 | 1063/1407 | 554/656 | 606 |
| OpenDesign (DS) | 2768/2908 | 616/885 | 768/1396 | 467/621 | 585 |
| JarvisHub (DS) | 2547/2908 | 608/894 | 658/(1127–1326) | 411/600 | 502 |
| Design Anything (GPT) | 2830/2908 | 834/902 | 1122/1414 | 598/663 | 614 |
| Codex (GPT) | 2853/2908 | 699/895 | 1016/1425 | 498/661 | 550 |
Table 6. Counts underlying Table 1.
Table 6 Counts underlying Table 1. Columns count delivered/requested items (delivery completion), passed/applicable constraints (constraint satisfaction), same/eligible pairs against supplied reference assets (reference fidelity), same/eligible pairs within the run (identity consistency), and scored rubric items (set consistency). JarvisHub’s reference-fidelity denominator is a range: 1,326 counts the judge’s abstentions as mismatches and 1,127 excludes them. The last column is a coverage count, not a simple pooled-mean denominator.
The primary counts follow the unavailable-evidence rules in Appendix § 6.§ 3. Table 10 shows how excluding judge abstentions changes the two pair measures.
| System | Identity consistency | Reference fidelity | ||||
| Abstained | Counted | Excluded | Abstained | Counted | Excluded | |
| Design Anything (DS) | 41/656 | 84.5 | 90.1 | 137/1407 | 75.6 | 83.7 |
| OpenDesign (DS) | 93/621 | 75.2 | 88.4 | 351/1396 | 55.0 | 73.5 |
| JarvisHub (DS) | 38/600 | 68.5 | 73.1 | 199/1326 | 49.6 | 58.4 |
| Design Anything (GPT) | 23/663 | 90.2 | 93.4 | 113/1414 | 79.3 | 86.2 |
| Codex (GPT) | 97/661 | 75.3 | 88.3 | 195/1425 | 71.3 | 82.6 |
Table 10. Judged image-pair measures with judge abstentions counted as mismatches (reported) and excluded. Abstention counts are attributed to the system’s own evidence; for reference fidelity, abstentions attributed to the external reference are excluded in both columns.
Results by task condition
Table 7 checks the five configurations across three project scales, breadth tasks, and reference-material tasks. Design Anything has the highest constraint satisfaction within each model group in every condition, reaching 85.3% and 96.0% on breadth tasks. OpenDesign delivers more on breadth tasks under DeepSeek, and Codex delivers more at every base scale under GPT. These results extend the Anything and At Scale findings across conditions while distinguishing brief alignment from delivery quantity.
Table 7. Design Anything leads same-model baselines in constraint satisfaction in every task condition. Shaded rows are Design Anything; bold marks the best delivery, constraint, and identity score within each LLM group and condition. These scores are percentages; set consistency uses a 100-point scale. Supplementary P/F/U counts denote task passes/failures/unmeasurable outcomes for all or final deliverables (Appendix § 7.§ 3); they are distinct from constraint satisfaction. DS: DeepSeek V4.1 Flash High; GPT: GPT-5.6 Sol High.
| System | Delivery | Constraints | Identity | Set | P/F/U (all) | P/F/U (final) |
| 6 artifacts (38 tasks) | ||||||
| Design Anything (DS) | 96.5 | 86.4 | 87.7 | 93.6 | 27/10/1 | 27/10/1 |
| OpenDesign (DS) | 95.6 | 70.6 | 78.1 | 86.9 | 22/14/2 | 22/14/2 |
| JarvisHub (DS) | 93.4 | 76.3 | 76.9 | 75.0 | 21/15/2 | 21/15/2 |
| Design Anything (GPT) | 97.4 | 97.4 | 97.0 | 94.9 | 31/7/0 | 31/7/0 |
| Codex (GPT) | 99.6 | 85.1 | 93.9 | 92.1 | 32/6/0 | 32/6/0 |
| 14 artifacts (38 tasks) | ||||||
| Design Anything (DS) | 96.6 | 82.1 | 79.0 | 86.7 | 21/15/2 | 24/12/2 |
| OpenDesign (DS) | 90.4 | 58.7 | 61.0 | 82.7 | 12/23/3 | 12/23/3 |
| JarvisHub (DS) | 88.7 | 61.9 | 71.9 | 59.4 | 7/29/2 | 12/24/2 |
| Design Anything (GPT) | 96.8 | 94.2 | 89.9 | 82.7 | 20/17/1 | 25/12/1 |
| Codex (GPT) | 97.9 | 76.3 | 69.6 | 85.2 | 15/23/0 | 19/18/1 |
| 24 artifacts (38 tasks) | ||||||
| Design Anything (DS) | 96.5 | 85.7 | 86.3 | 87.0 | 14/24/0 | 15/23/0 |
| OpenDesign (DS) | 95.0 | 60.6 | 68.7 | 87.7 | 19/18/1 | 19/18/1 |
| JarvisHub (DS) | 81.8 | 54.4 | 60.3 | 61.0 | 4/33/1 | 7/30/1 |
| Design Anything (GPT) | 95.7 | 91.9 | 87.8 | 87.2 | 14/22/2 | 15/21/2 |
| Codex (GPT) | 98.6 | 68.9 | 66.7 | 91.1 | 19/18/1 | 20/17/1 |
| Breadth (57 tasks) | ||||||
| Design Anything (DS) | 96.1 | 85.3 | 85.9 | 85.0 | 30/22/5 | 32/20/5 |
| OpenDesign (DS) | 97.5 | 73.4 | 80.1 | 91.7 | 33/22/2 | 33/22/2 |
| JarvisHub (DS) | 91.6 | 74.3 | 71.8 | 69.6 | 26/28/3 | 30/24/3 |
| Design Anything (GPT) | 98.8 | 96.0 | 88.8 | 88.3 | 41/15/1 | 41/15/1 |
| Codex (GPT) | 97.7 | 80.3 | 77.5 | 90.3 | 30/24/3 | 34/19/4 |
| Ref. materials (57 tasks) | ||||||
| Design Anything (DS) | 97.3 | 85.5 | 81.1 | 89.7 | 31/23/3 | 31/23/3 |
| OpenDesign (DS) | 97.1 | 80.0 | 90.5 | 89.6 | 35/21/1 | 35/21/1 |
| JarvisHub (DS) | 88.8 | 70.2 | 71.4 | 63.9 | 24/32/1 | 24/32/1 |
| Design Anything (GPT) | 98.6 | 84.1 | 94.0 | 87.7 | 34/22/1 | 35/21/1 |
| Codex (GPT) | 97.5 | 79.1 | 84.8 | 89.1 | 29/25/3 | 33/20/4 |
| All (228 tasks) | ||||||
| Design Anything (DS) | 96.6 | 85.0 | 84.5 | 87.9 | 123/94/11 | 129/88/11 |
| OpenDesign (DS) | 95.2 | 69.6 | 75.2 | 88.3 | 121/98/9 | 121/98/9 |
| JarvisHub (DS) | 87.6 | 68.0 | 68.5 | 65.4 | 82/137/9 | 94/125/9 |
| Design Anything (GPT) | 97.3 | 92.5 | 90.2 | 87.7 | 140/83/5 | 147/76/5 |
| Codex (GPT) | 98.1 | 78.1 | 75.3 | 89.8 | 125/96/7 | 138/80/10 |
Each base scale contains the same 38 project families, so the three scale rows of a family are dependent observations. Set consistency per condition uses the same nested aggregation as Table 1, applied to the tasks of that condition.
Supplementary task-level outcomes. Table 7 also reports, for completeness, a supplementary task-level outcome: a task passes when every requested category is filled, every constraint the specification marks as critical holds, and the page minima of presentations and reports are met. The all column applies this test to every deliverable, including requested intermediate outputs, and the final column to final deliverables only. Full-corpus pass fractions use all 228 tasks as the denominator; condition-specific counts use the tasks in that condition. Unmeasurable outcomes have four sources: a report or slide deck delivered in a format without a readable page count, so its page minimum cannot be checked, which is the most common source; an original file the evaluator cannot read; a run the harness recorded as an infrastructure failure; and a critical constraint that could not be judged. Any measured failure of these conditions makes the task fail; an unmeasurable task has no measured failure but lacks evidence for a pass. Unmeasurable tasks remain in the denominator.
Domain coverage. The domain-level comparison is reported in Figure 5 in the main text. Each domain has 12 tasks and 25 to 80 applicable constraints, so these rates describe breadth of performance rather than establishing a significant difference in every domain.
Robustness of the paired comparisons
Paired uncertainty and growth with scale. Accounting for related tasks preserves the same-model advantage. All paired constraint-satisfaction gaps at 6, 14, and 24 artifacts have 95% intervals above zero (Table 8). For the project-family-clustered bootstrap, the 152 distinct projects are resampled with replacement 4,000 times, the three scale variants of a family are drawn together, and intervals are 95% percentile intervals. The point-estimate constraint-satisfaction gaps widen from 6 to 24 artifacts against each baseline, but the interval for this growth excludes zero only against JarvisHub. This distinction separates an advantage at a given scale from evidence that the advantage grows with scale.
| Pair | All tasks | 6 art. | 14 art. | 24 art. | Growth |
| Design Anything (DeepSeek) − JarvisHub | 17.0 [12.4, 22.2] | 10.0 [0.3, 19.6] | 20.1 [7.2, 34.9] | 31.3 [18.6, 45.8] | 21.3 [3.8, 40.0] |
| Design Anything (DeepSeek) − OpenDesign | 15.4 [9.4, 21.5] | 15.7 [2.1, 30.8] | 23.3 [9.9, 35.5] | 25.1 [12.3, 38.4] | 9.4 [−5.3, 24.0] |
| Design Anything (GPT) − Codex | 14.4 [9.3, 19.3] | 12.3 [6.0, 20.0] | 17.9 [10.2, 26.4] | 23.0 [9.0, 36.4] | 10.6 [−3.5, 24.7] |
Table 8. Paired constraint-satisfaction gaps with family-clustered 95% bootstrap intervals (points). Growth is the gap at 24 artifacts minus the gap at 6.
Sensitivity to missing deliveries. The same-model advantage is not explained by counting wholly missing output categories as unsatisfied constraints. Table 9 repeats the comparison after excluding those constraints. A constraint none of whose target categories contains a delivered file counts as unsatisfied (Appendix § 6.§ 3); excluding such constraints, 2 to 39 per configuration, changes the same-model gaps over all tasks by at most 1.9 points, and every paired gap at 14 and 24 artifacts still excludes zero. The growth of the gap from 6 to 24 artifacts becomes 10.0 points [−4.2, 24.3] against Codex and 18.9 [0.8, 37.6] against JarvisHub, so the growth interval still excludes zero against JarvisHub but not against Codex. The growth against JarvisHub shrinks mainly because of its 12 such constraints at 24 artifacts, which narrow the gap there from 31.3 to 26.9 points.
| System | All tasks | 6 art. | 14 art. | 24 art. |
| Design Anything (DS) | 86.2 (12) | 87.5 (2) | 83.7 (3) | 85.7 (0) |
| OpenDesign (DS) | 70.3 (9) | 70.6 (0) | 61.1 (6) | 61.0 (1) |
| JarvisHub (DS) | 71.1 (39) | 79.5 (6) | 64.0 (5) | 58.8 (12) |
| Design Anything (GPT) | 92.7 (2) | 98.0 (1) | 94.2 (0) | 91.9 (0) |
| Codex (GPT) | 78.5 (5) | 85.1 (0) | 76.3 (0) | 68.9 (0) |
Table 9. Constraint satisfaction (%) without constraints on wholly undelivered categories. Constraints none of whose target categories contains a delivered file are removed from numerator and denominator; parentheses give how many were removed. Reported rates, which count them as unsatisfied, are in Tables 1 and Table 7.
Sensitivity to judge abstention. Table 10 tests whether judge abstentions change the reference-fidelity and identity-consistency ranking on the 228 tasks. Excluding them raises Design Anything identity consistency to 90.1% and 93.4%. The baselines also improve, but remain below Design Anything on both measures within each model group. The ranking is therefore the same under either treatment of unavailable pair evidence.
The table reports both denominator choices; most abstentions concern pairs whose views are not comparable, such as a plan drawing against a photograph, or a whole object against a detail.
Diagnostic: partial coverage of factual constraints. The diagnostic tests whether unmet constraints are absent from every target or fulfilled only in part of the delivered set. Of Codex’s 191 unmet constraints with per-file records, 117 hold in at least half of their targets: many failures are omissions within a delivered set. Table 11 decomposes Equation (7) to distinguish partial coverage from satisfaction in every target. Each factual constraint has a criterion and target files , as defined in Appendix § 6.§ 3. *Per-file constraint coverage pools the successful constraint–target checks across the evaluated tasks:
| System | Targets per constraint | Per-file coverage | Constraint satisfaction |
| Design Anything (DS) | 2.21 / 3.38 / 5.76 | 87.9 / 89.2 / 97.0 | 86.4 / 82.1 / 85.7 |
| OpenDesign (DS) | 2.32 / 3.42 / 5.84 | 74.7 / 76.4 / 75.3 | 70.6 / 58.7 / 60.6 |
| JarvisHub (DS) | 2.14 / 3.29 / 5.30 | 90.2 / 86.3 / 85.5 | 76.3 / 61.9 / 54.4 |
| Design Anything (GPT) | 2.21 / 3.47 / 5.68 | 99.1 / 98.3 / 97.9 | 97.4 / 94.2 / 91.9 |
| Codex (GPT) | 2.31 / 3.37 / 5.92 | 87.3 / 88.6 / 86.1 | 85.1 / 76.3 / 68.9 |
Table 11. Constraint satisfaction decomposed by target file. Columns give mean target files per constraint, per-file constraint coverage (%), and constraint satisfaction (%) at 6 / 14 / 24 artifacts.
where contains the evaluated constraint–target pairs with per-file records. Constraint-specific checks and target eligibility follow Appendix § 6.§ 3; text normalization and transcription exclusions apply as above. The denominator counts pairs, not distinct files: a file targeted by two constraints contributes two checks. A factual constraint satisfied in two of three target files gives coverage but an unsatisfied constraint; wholly undelivered categories are handled separately above.
On GPT-5.6 Sol, Design Anything’s per-file constraint coverage is 97.9–99.1% across scales, versus 86.1–88.6% for Codex (Figure 10). Mean targets per constraint increase from about 2.2 to 3.4 to 5.7, averaging the five system means; Table 11 gives each system’s values. No system’s coverage declines by more than 5 points from 6 to 24 artifacts, but each constraint must hold across more files. This diagnostic describes final-output coverage. Constraint satisfaction additionally depends on how failed checks are distributed among constraints; neither statistic identifies when a constraint was lost during execution.
Figure 10. High per-file coverage can coexist with unmet constraints across a set. Curves compare five configurations on 38 matched families, showing successful factual constraint–target checks as a percentage of evaluated pairs at each scale. Filled markers and solid lines denote DeepSeek V4.1 Flash High; hollow markers and dashed lines denote GPT-5.6 Sol High. A constraint passes only if every eligible target satisfies it.
Reference coverage
We compare recorded reference reuse on the 228 DeepSeek tasks and examine its association with identity consistency. On these tasks, Design Anything uses references in 95.3% of image generations, versus 39.4% for JarvisHub. Reference coverage measures this share of generation requests. The recorded LUMA project illustrates how a selected concept feeds later boards, video, and a presentation (Figure 14); here we quantify reference use across runs.
For Design Anything, references are read from the generation requests recorded in each run; for JarvisHub, from the reference edges of its exported canvas graph. Across these tasks, Design Anything issues 2,680 image generations and 4 tasks fall below one-half reference coverage; JarvisHub issues 4,258 and 130 tasks fall below one half (three JarvisHub tasks produced no image). On the 24-artifact tasks, coverage reaches 97.9% for Design Anything and 40.2% for JarvisHub; 74.9% of the referenced generations of Design Anything take an image generated earlier in the same run.
Figure 11b examines reference reuse and identity consistency within JarvisHub’s 103 tasks with identity pairs; Design Anything’s coverage is close to its ceiling in every condition. Higher coverage is associated with higher identity consistency (Spearman , ), including after ranking within conditions (, ). The associations with set consistency, constraint satisfaction, and reference fidelity are weaker. Reference reuse is thus most closely associated with object identity in this diagnostic; observed coverage does not isolate the workflow graph’s causal contribution.
Identity consistency per task is the pair-level rate within a run, the same quantity that Table 1 pools. Tasks have a median of five identity pairs, so per-task values are coarse and 46 of the 103 tasks sit at 0 or 100%. The identity correlation is positive in every condition: 0.73 at 6 artifacts (), 0.53 at 14 (), 0.60 at 24 (), 0.37 on breadth tasks (), and 0.13 on reference-material tasks (). Ranks taken within each condition are pooled for the adjusted correlation above; correlations with set consistency, constraint satisfaction, and reference fidelity over all tasks are 0.26, 0.10, and 0.05.
Resource use in the main benchmark
LLM input volume increases with project scale under both configurations. Table 12 reports median input tokens per task for Design Anything: on DeepSeek, the median rises from 1.4M at 6 artifacts to 6.6M at 14 and 9.1M at 24; on GPT, the orchestrating agent alone reads 5.6M, 16.6M, and 18.4M. The controlled delivery-loop cost comparison is in Appendix § 8.§ 1.
| Condition | DeepSeek V4.1 Flash High | GPT-5.6 Sol High |
| 6 artifacts | 1.36 | 5.59 |
| 14 artifacts | 6.58 | 16.57 |
| 24 artifacts | 9.06 | 18.38 |
| Breadth | 4.22 | 16.42 |
| Reference materials | 2.37 | 12.34 |
| All | 4.04 | 14.63 |
Table 12. Median LLM input tokens per task for Design Anything (millions). The GPT-5.6 Sol High column counts the orchestrating agent only.
The DeepSeek counts come from per-task token records; the GPT counts come from each run’s records and cover the orchestrating agent only, excluding helper sessions, so the two columns are not directly comparable. Because token records also differ in scope across harnesses, we do not compare tokens or cost across systems.
Figure 11. Design Anything uses references more often; reference use is associated with identity consistency within JarvisHub. Panel (a) shows the share of image generations with at least one reference input, pooled over each condition’s tasks, for Design Anything and JarvisHub on DeepSeek V4.1 Flash High. Breadth and Reference denote breadth and reference-material tasks. Panel (b) plots JarvisHub’s reference coverage against identity consistency, with one point per task with identity pairs (Spearman , ).
Design quality
Table 13 compares five configurations on four delivery-weighted design-quality dimensions across the 228 tasks. Design Anything leads on all four under DeepSeek and scores 88.5 on aesthetics and 92.5 on domain criteria under GPT. Codex leads on GPT technical quality and text rendering by 0.1 and 0.5 points, respectively, although differences this small are below the observed judging variability.
| System | Technical | Aesthetics | Text | Domain |
| Design Anything (DS) | 88.4 | 87.1 | 93.0 | 92.0 |
| Design Anything (GPT) | 88.8 | 88.5 | 93.8 | 92.5 |
| OpenDesign (DS) | 85.1 | 82.3 | 80.6 | 85.5 |
| JarvisHub (DS) | 80.9 | 80.2 | 84.7 | 84.2 |
| Codex (GPT) | 88.9 | 87.9 | 94.3 | 90.8 |
Table 13. Design quality (100-point scale). Scores combine four image-checklist dimensions with project delivery coverage (Equation (8)).
Scoring coverage. All 228 tasks of every configuration were judged under the protocol in Appendix § 6.§ 3. The judged images per configuration are 2,089 for Design Anything (DS), 2,102 for Design Anything (GPT), 2,024 for OpenDesign, 1,920 for JarvisHub, and 2,126 for Codex; at most four images per configuration were left unscored after a failed judge call.
Judge sensitivity. Claude Opus 4.8 rescored the same delivered images on a stratified 15-task sample (three per condition) for Design Anything (GPT), OpenDesign, and JarvisHub; Codex was not included. This check tests the image-checklist component of design quality. On the sample’s checklist scores, before weighting by delivery completion, both judges place Design Anything and JarvisHub within 2.4 points of each other on every dimension and above OpenDesign, with per-task Spearman correlations of 0.37–0.60 (the DeepSeek scores for Design Anything (GPT) come from an earlier pass of the same judge). This analysis characterizes judge sensitivity of the checklist component rather than whole-set designer preference.
Ablation Studies
The studies below examine the delivery loop, planning interfaces, and state summaries. In the delivery-loop study, Stop at model end versus Generic prompt tests prompting after a normal stop within the same reduced harness; Generic prompt versus Complete compares a bundle of mechanisms and its execution efficiency. The planning and state-summary studies examine narrower changes separately. The delivery-loop study uses 19 tasks from the main corpus; planning and state-summary studies use separate tasks and harness versions. Each study reports its own configuration and contributes no observations to the 228-task benchmark aggregates. No study uses skills written for a particular design domain.
Delivery loop and execution efficiency
This study asks whether replying after the model stops restores delivery, and how the complete harness changes repeated generation and cost. It uses the 24-artifact version of one base family per domain (19 tasks), the family whose brief asks the system to propose alternative directions and select one, with the LLM and image-generation configuration of the main comparison. Each configuration has one run per task, including a fresh Complete run separate from the main comparison. The verifier is enabled for Complete and disabled for the two reduced configurations. All three are scored with the same evaluator. Approval requests raised through the checkpoint tool receive the same scripted answers in every configuration; a question written into a final message ends the turn, and Stop at model end provides no reply.
Configuration definitions. Stop at model end removes the delivery-loop controller, the contract and planning tools, batched asynchronous execution, and every channel that restates the task or its progress, such as reminders of the original brief and progress lines in the status message. Synchronous execution, read tools, context compaction, scripted approval answers, and retries after transport failures remain, since a generic tool-using harness has them. A run ends when the model ends a turn without calling a tool. Generic prompt keeps every removal and, whenever the model ends a turn, sends the fixed user message “Continue working until everything the original brief asks for is fully delivered. If everything is already delivered, say so briefly and make no canvas changes.” The message names no outstanding item. The loop stops after two consecutive rounds without canvas change, after 100 turns, or at the 150-minute wall-clock limit. Complete is the full harness of § 2. Stop at model end versus Generic prompt tests the addition of the fixed prompt; Generic prompt versus Complete compares the combined mechanisms in Table 14.
| Mechanism | Stop at model end | Generic prompt | Complete |
| After the model stops | End run | Fixed prompt to keep working | Delivery loop (§ 2.4) |
| Contract and planning tools | Removed | Removed | Retained |
| Task and progress reminders | Removed | Removed | Retained |
| Automatic verifier | Disabled | Disabled | Enabled |
| Batched asynchronous execution | Removed | Removed | Retained |
Table 14. Delivery-loop configurations. The first two configurations differ only in a generic reply after stopping within the reduced harness. Complete restores the full set of mechanisms.
Delivery and constraint outcomes. Table 2 compares three configurations on the 19 paired 24-artifact tasks. Complete achieves 96.1% delivery and 90.4% constraint satisfaction. It improves both over Stop at model end ( and ), but its delivery and constraint differences from Generic prompt are not statistically significant ( and ). This supports fulfillment gains over stopping unaided on larger projects; the comparison with Generic prompt does not isolate one component.
Paired tests are two-sided exact Wilcoxon signed-rank tests paired by task; tasks with equal values are excluded and tied absolute differences receive average ranks. Against the complete harness, Stop at model end delivers less on 8 tasks and more on 3 and satisfies fewer constraints on 11 tasks and more on none. Generic prompt delivers more on 7 tasks and less on 4, with 8 ties; constraint satisfaction favors Complete on 6 tasks and Generic prompt on 1.
Early stops and constraint outcomes. The “Runs < 50%” column counts runs delivering fewer than half of their requested artifacts. Six Stop at model end runs fall below half delivery against no Complete run (Fisher exact test, two-sided , one-sided ). All six end within three turns: three ask for confirmation of a direction or a costly next stage, two announce further deliverables, and one exhausts the output-token limit before building anything. Generic prompt recovers each to 23 or 24 deliverables.
Missing deliveries explain only part of the constraint gap. Constraints on wholly undelivered categories count as unsatisfied (Appendix § 6.§ 3); these six runs account for at most 12 of the 24 fewer constraints satisfied relative to Complete. At least 12 fewer satisfied constraints therefore occur in runs that deliver 20 or more of their 24 artifacts.
Repeated generation. On the same 19 tasks (Table 2), the generic configuration issues more generation requests than the complete harness on 15 tasks and fewer on 3 (); counting only completed requests gives 14 and 3 (). The extra requests come after the deliverables are largely in place, when the prompt keeps the agent regenerating delivered work; in one task the generic configuration issues 68 requests against 27. This supports an efficiency advantage of the full harness on these larger projects, alongside the cost comparison below.
LLM cost. On the 17 paired tasks with complete telemetry, Complete costs $0.43 per run versus $1.30 for Generic prompt (Table 15). The generic configuration takes more agent steps on all 17 tasks, and more input tokens and LLM cost on 15 of 17 tasks ( for each). On the same 17 tasks we detect no difference between the Stop at model end configuration and the complete harness in steps (8 higher, 9 lower), input tokens (7 and 10), or cost (9 and 8). Mean LLM costs are $0.48 and $0.43, respectively; this sample does not establish cost equivalence.
| Measurement | Stop at model end | Generic prompt | Complete |
| Agent steps | 31.2 | 79.7 | 36.4 |
| Input tokens (M) | 16.9 | 41.8 | 17.9 |
| LLM cost (USD) | 0.48 | 1.30 | 0.43 |
| Generation requests | 18.3 | 37.1 | 23.8 |
| Delivery completion (%) | 72.1 | 96.8 | 96.1 |
Table 15. Delivery-loop study: mean cost per run on 17 paired tasks. Two Generic prompt runs reached the wall-clock limit without step or token counts and are excluded with their pairs. Input tokens (millions) and LLM cost include sub-agents; cost uses list prices.
Planning-interface study
This study asks whether structured planning interfaces improve delivery when other channels for tracking requirements remain available. It compares three planning interfaces. Structured retains the structured project-plan and workflow-construction tools. Text plan replaces those tools with an instruction to maintain a free-text deliverable checklist. Planning tools off removes them without a replacement instruction. Delivery groups, outstanding-work messages, direct canvas reads, status tools, retained history, and new model turns remain available in every configuration. The automatic verifier is disabled in all three configurations, including Structured, to keep its availability fixed.
On eight tasks, delivery completion is 7.7 points lower with Planning tools off than with Structured, and 2.1 points lower with Text plan. These comparisons measure the incremental contribution of structured planning interfaces alongside the other channels for tracking the task. Those retained channels offer a possible explanation for the modest gaps.
State-summary efficiency
The state-summary study tests whether the summary reduces direct inspection of the canvas. It removes the summary while retaining direct read tools and measures the fraction of tool calls devoted to active canvas reading. Nine of ten paired tasks read more without the summary; the geometric mean ratio of reading fractions is 2.06 (one-sided exact sign test, ).
The paired result shows a larger share of tool calls devoted to direct state inspection without the summary; the experiment does not measure final design quality.
Qualitative Examples
The visual evidence complements the domain and scale results in § 3.4: Figure 12–Figure 13 enlarge the matched comparison in Figure 6; Figure 14 traces reuse of a selected design within one 24-artifact project; and Table 16 indexes the gallery by domain and requested artifact count. These recorded examples illustrate delivered work and its dependencies.
| Figure | Domain | Artifacts |
| 15 | Fashion (first project) | 24 |
| 16 | Game art | 24 |
| 17 | Brand identity | 24 |
| 18 | Jewelry | 24 |
| 19 | Landscape design | 24 |
| 20 | Photography | 24 |
| 21 | Fashion (second project) | 24 |
| 22 | Packaging | 24 |
| 23 | UI/UX | 24 |
| 24 | Exhibition design; landscape design | 14 each |
| 25 | Furniture; product design | 6 each |
Table 16. Gallery index. Domains and requested artifact counts for thirteen recorded projects, selected for display.
Enlarged outputs for Figure 6
Figure 12 and Figure 13 enlarge the 14-artifact animation brief of Figure 6 for all five configurations. Each block keeps the deliverable order, credit counts, and markers of Figure 6. Documents show one page and videos one frame; credit counts follow the evaluator. OpenDesign places two keyframes on each page and its three act storyboards on one page, so these images repeat across positions.
Keyframes S01–S06 are shots numbered in the brief. Red markers identify five content errors checked against the brief. In the S06 keyframe of Design Anything (DeepSeek), each traveler still holds the other’s case. JarvisHub’s color script replaces the otter characters with humans; its complete storyboard departs from the fixed actions of S03, S04, and S06 and omits both dialogue lines; and its Act 3 storyboard replaces the wheeled cases with cardboard boxes. Codex’s Act 2 storyboard repeats one image for both S03 phases. The markers are examples, not an exhaustive audit. A slashed circle means the evaluator did not credit the requested artifact, not that no file was exported; OpenDesign, for example, exported a shot table, a web-page animatic, and a presentation that the evaluator did not credit to these requested artifacts.
A recorded project and its shared dependencies
This case asks how an earlier design decision becomes an input to later image, video, and presentation outputs within one 24-artifact project.
Figure 14 uses the LUMA product-design task from the main-comparison run of Design Anything on DeepSeek V4.1 Flash High. Its public brief requests AI glasses and a matching charging case, with 24 artifacts in nine groups: three direction boards, one selection record, two specifications, six anatomy/detail boards, five wearing/use boards, four case boards, one material board, one video, and one presentation. Shared constraints concern the chosen product identity, dimensions and docking, operating states, and material choices.
The written selection record chooses Continuum from three directions. Its direction-board node has 18 recorded outgoing connections, which make it an input to all 16 anatomy, wearing, case, and material boards, to video generation, and to the presentation. The agent-built workflow graph has 79 connections in total. In the top panel, the orange arrows draw these 18 connections once per group; the bottom panel draws each of them.
This run delivers all 24 artifacts and satisfies 6 of its 7 constraints; the identity judge rates 16 of 18 image pairs as the same product. The unmet constraint shows that reuse alone does not guarantee full fulfillment of the brief.
Agent-built workflow graphs and image boards of thirteen projects
Figure 15 to Figure 25 show thirteen more projects that we selected for display, each from one recorded run of Design Anything with GPT-5.6 Sol High as the LLM. The gallery illustrates the range of domains and project scales handled by the same harness.

Figure 15. A 24-artifact fashion project (I). Each block shows, from top to bottom, the workflow graph the agent built during the run, its delivered image boards in deliverable order, and up to three video frames and three presentation pages. Projects were selected for display.

Figure 25. Two 6-artifact projects (XI). The upper and lower blocks show furniture and product design, respectively. Both follow the layout of Figure 15.

Figure 16. A 24-artifact game-art project (II). The layout follows Figure 15.

Figure 17. A 24-artifact brand-identity project (III). The layout follows Figure 15.

Figure 18. A 24-artifact jewelry project (IV). The layout follows Figure 15.

Figure 19. A 24-artifact landscape-design project (V). The layout follows Figure 15.

Figure 20. A 24-artifact photography project (VI). The layout follows Figure 15.

Figure 21. A second 24-artifact fashion project (VII). The layout follows Figure 15.

Figure 22. A 24-artifact packaging project (VIII). The layout follows Figure 15.

Figure 23. A 24-artifact UI/UX project (IX). The layout follows Figure 15 and includes three direction boards.

Figure 24. Two 14-artifact projects (X). The upper and lower blocks show exhibition design and landscape design, respectively. Both follow the layout of Figure 15.
Limitations
Evaluation and run variability. Each configuration contributes one final result per task; repeated runs would further characterize run-to-run variation. The image-checklist component of design quality relies on a vision-language judge. Whole-set designer preferences complement these measurements without identifying which properties drive preference.
Scope of system comparisons. The main comparisons hold the LLM fixed and evaluate complete systems, including their tools and execution policies. The delivery-loop comparison bundles several mechanisms; planning-interface and state-summary studies examine narrower changes within their own configurations (Appendix § 8).
Runtime checks and design intent. The model chooses the bindings between deliverables and workflow steps. The controller checks their execution, output forms, and quantities, but cannot determine whether a bound artifact is the one the brief requested or whether its content satisfies the constraints. Artifact evaluation remains necessary.
Task and interaction coverage. The benchmark uses English briefs and a scripted client for checkpoint questions. For a fair comparison, it includes only modalities that every compared system can generate: Design Anything also generates 3D meshes and audio (grey in Figure 3), but not all compared systems do. We leave these modalities, other languages, and live client feedback to future evaluations.
References
- [1]Saaket Agashe, Jiuzhou Han, Shuyu Gan, Jiachen Yang, Ang Li, and Xin Wang. Agent S: An open agentic framework that uses computers like a human. In The Thirteenth International Conference on Learning Representations, 2025.
- [2]Ralph Allan Bradley and Milton E. Terry. Rank analysis of incomplete block designs: I. the method of paired comparisons. Biometrika, 39(3/4):324–345, 1952.DOI
- [4]Kees Dorst and Nigel Cross. Creativity in the design process: co-evolution of problem–solution. Design Studies, 22(5):425–437, 2001.DOI
- [5]Yuyuan Feng, Zhishang Xiang, Chaobin Yang, Qichao Ma, Zerui Chen, Yujing Zhang, Ke Huang, Chuanjie Wu, Zhaoxu Liu, Yili Wang, Xin He, Jiapu Wang, Zijin Hong, Hao Chen, Yuanchen Bei, Kun Wang, Shengyuan Chen, Ningyu Zhang, Enyan Dai, Linhao Luo, Qingyi Pan, Qi Wang, Wenqi Fan, Guangjing Wang, Na Zhou, Yangqiu Song, Xin Wang, Zechao Li, Xia Hu, Qing Li, Xiao Huang, Zhihong Zhang, Jinsong Su, Qinggang Zhang, and Yi Chang. Graph engineering in the era of LLM agents: From individual intelligence to system intelligence. arXiv preprint arXiv:2608.21156, 2026.arxiv.org/abs/2608.21156
- [6]Jiaxin Ge, Zora Zhiruo Wang, Xuhui Zhou, Yi-Hao Peng, Sanjay Subramanian, Qinyue Tan, Maarten Sap, Alane Suhr, Daniel Fried, Graham Neubig, and Trevor Darrell. Auto Present: Designing structured visuals from scratch. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 2902–2911, 2025.
- [7]John S. Gero. Design prototypes: A knowledge representation schema for design. AI Magazine, 11(4):26–36, 1990. doi: 10.1609/aimag.v11i4.854.
- [8]John S. Gero and Udo Kannengiesser. The situated function–behaviour–structure framework. Design Studies, 25(4):373–391, 2004. doi: 10.1016/j.destud.2003.10.010.DOI
- [9]Kelly Hong, Anton Troynikov, and Jeff Huber. Context rot: How increasing input tokens impacts LLM performance. Technical report, Chroma, 2025. https://trychroma.com/research/context-rot.
- [10]Daeheon Jeong, Seoyeon Byun, Kihoon Son, Dae Hyun Kim, and Juho Kim. CANVAS: A benchmark for vision-language models on tool-based user interface design. In Proceedings of the AAAI Conference on Artificial Intelligence, 2026.
- [11]Dayeon Ki, Tianyi Zhou, Marine Carpuat, Gang Wu, Puneet Mathur, and Viswanathan Swaminathan. GraphicBench: A planning benchmark for graphic design with language agents. arXiv preprint arXiv:2504.11571, 2025.arxiv.org/abs/2504.11571
- [12]Sehoon Kim, Suhong Moon, Ryan Tabrizi, Nicholas Lee, Michael W. Mahoney, Kurt Keutzer, and Amir Gholami. An LLM compiler for parallel function calling. In Proceedings of the 41st International Conference on Machine Learning, volume 235 of Proceedings of Machine Learning Research, pages 24370–24391. PMLR, 2024. https://proceedings.mlr.press/v235/kim24y.html.arxiv.org/abs/2312.04511
- [13]Thomas Kwa, Ben West, Joel Becker, Amy Deng, Katharyn Garcia, Max Hasin, Sami Jawhar, Megan Kinniment, Nate Rush, Sydney Von Arx, Ryan Bloom, Thomas Broadley, Haoxing Du, Brian Goodrich, Nikola Jurkovic, Luke Harold Miles, Seraphina Nix, Tao Lin, Chris Painter, Neev Parikh, David Rein, Lucas Jun Koba Sato, Hjalmar Wijk, Daniel M. Ziegler, Elizabeth Barnes, and Lawrence Chan. Measuring AI ability to complete long software tasks. In Advances in Neural Information Processing Systems, 2025.arxiv.org/abs/2503.14499
- [14]Philippe Laban, Hiroaki Hayashi, Yingbo Zhou, and Jennifer Neville. LLMs get lost in multi-turn conversation. In International Conference on Learning Representations, 2026.arxiv.org/abs/2505.06120
- [15]Yoonho Lee, Roshen Nair, Qizheng Zhang, Kangwook Lee, Omar Khattab, and Chelsea Finn. Meta-Harness: End-to-end optimization of model harnesses. arXiv preprint arXiv:2603.28052, 2026.arxiv.org/abs/2603.28052
- [16]Niantong Li, Guangzheng Hu, Weixu Qiao, Ying Ba, Qichen Hong, Shijun Shen, Jinlin Wang, Fan Zhou, Jianye Kang, Xin Shang, Ziyi He, Wei Wang, Dalin Li, Jiahao Li, Jie Zhang, Kaiyuan Gao, Kun Yan, Lihan Jiang, Ningyuan Tang, Shengming Yin, Tianhe Wu, Xiao Xu, Xiaoyue Chen, Yuxiang Chen, Yan Shu, Yanran Zhang, Yilei Chen, Yixian Xu, Zekai Zhang, Zhendong Wang, Zihiao Liu, Zikai Zhou, Hongzhu Shi, Yi Wang, Bing Zhao, Hu Wei, Lin Qu, and Chenfei Wu. Qwen-Image-Bench: From generation to creation in text-to-image evaluation. arXiv preprint arXiv:2605.28091, 2026a.arxiv.org/abs/2605.28091
- [17]Wanli Li, Bowen Zhou, Yunyao Yu, Zhou Xu, Yifan Yang, Dongsheng Li, and Caihua Shan. WeaveBench: A long-horizon, real-world benchmark for computer-use agents with hybrid interfaces. arXiv preprint arXiv:2606.09426, 2026b.arxiv.org/abs/2606.09426
- [18]Yunlong Lin, Zixu Lin, Zhaohu Xing, Biqiang Li, Chenxin Li, Haonan Wang, Haitao Wu, Hengyu Liu, Jianghai Chen, Kaituo Feng, Kaixin Li, Shawn Chen, Shijue Huang, Sixiang Chen, Tsung-Yi Ho, Wenxuan Huang, Xiangyan Liu, Xiaomeng Hu, Xuanhua He, Yan Sun, Yunqing Zhao, Zhiqin Yang, Zehan Wang, Zhengyang Tang, Tianyu Pang, and Xiangyu Yue. JarvisHub: An open harness for canvas-native multimodal creative agents. arXiv preprint arXiv:2607.23588, 2026.arxiv.org/abs/2607.23588
- [19]Nelson F. Liu, Kevin Lin, John Hewitt, Ashwin Paranjape, Michele Bevilacqua, Fabio Petroni, and Percy Liang. Lost in the middle: How language models use long contexts. Transactions of the Association for Computational Linguistics, 12:157–173, 2024a.arxiv.org/abs/2307.03172
- [20]Xiao Liu, Hao Yu, Hanchen Zhang, Yifan Xu, Xuanyu Lei, Hanyu Lai, Yu Gu, Hangliang Ding, Kaiwen Men, Kejuan Yang, Shudan Zhang, Xiang Deng, Aohan Zeng, Zhengxiao Du, Chenhui Zhang, Sheng Shen, Tianjun Zhang, Yu Su, Huan Sun, Minlie Huang, Yuxiao Dong, and Jie Tang. AgentBench: Evaluating LLMs as agents. In International Conference on Learning Representations, 2024b.arxiv.org/abs/2308.03688
- [21]Yaxin Luo, Haobin Jiang, Jialv Zou, Xu Huang, Wenhao Yan, Haodong Li, Zhengrong Yue, Jing Li, Xiaofu Chen, Xiaohan Zhao, Jiacheng Liu, Jiacheng Cui, Zhiqiang Shen, and Xiaotong Li. AutoDesign: Meta-harness optimization for long-horizon agentic design. arXiv preprint arXiv:2608.13560, 2026.arxiv.org/abs/2608.13560
- [22]Ziyu Ma, Hailang Huang, Shun Zou, Yong Wang, Shidong Yang, Yiming Hu, Fei Wei, and XiangXiang Chu. LongHorizon-Harness: Advancing long-horizon agents for real-world tasks. arXiv preprint arXiv:2608.01964, 2026.arxiv.org/abs/2608.01964
- [23]Mike A. Merrill, Alexander G. Shaw, Nicholas Carlini, Boxuan Li, Harsh Raj, Ivan Bercovich, Lin Shi, Jeong Yeon Shin, Thomas Walshe, E. Kelly Buchanan, Junhong Shen, Guanghao Ye, Haowei Lin, Jason Poulos, Maoyu Wang, Marianna Nezhurina, Jenia Jitsev, Di Lu, Orfeas Menis Mastromichalakis, Zhiwei Xu, Zizhao Chen, Yue Liu, Robert Zhang, Leon Liangyu Chen, Anurag Kashyap, Jan-Lucas Uslu, Jeffrey Li, Jianbo Wu, Minghao Yan, Song Bian, Vedang Sharma, Ke Sun, Steven Dillmann, Akshay Anand, Andrew Lanpoutakoun, Bardia Koopah, Changran Hu, Etash Guha, Gabriel H. S. Dreiman, Jiacheng Zhu, Karl Krauth, Li Zhong, Niklas Muennighoff, Robert Amanfu, Shangyin Tan, Shreyas Pimpalgaonkar, Tushar Aggarwal, Xiangning Lin, Xin Lan, Xuandong Zhao, Yiqing Liang, Yuanli Wang, Zilong Wang, Changzhi Zhou, David Heineman, Hange Liu, Harsh Trivedi, John Yang, Junhong Lin, Manish Shetty, Michael Yang, Nabil Omi, Negin Raoof, Shanda Li, Terry Yue Zhuo, Wuwei Lin, Yiwei Dai, Yuxin Wang, Wenhao Chai, Shang Zhou, Dariush Wahdany, Ziyu She, Jiaming Hu, Zhikang Dong, Yuxuan Zhu, Sasha Cui, Ahson Saiyed, Arinbjörn Kolbeinsson, Jesse Hu, Christopher Michael Rytting, Ryan Marten, Yixin Wang, Alex Dimakis, Andy Konwinski, and Ludwig Schmidt. Terminal-Bench: Benchmarking agents on hard, realistic tasks in command line interfaces. In International Conference on Learning Representations, 2026.
- [25]OpenAI. Introducing ChatGPT Images 2.0, 2026. https://openai.com/index/introducing-chatgpt-images-2-0/. Published 2026-04-21. Accessed September 21, 2026.
- [26]Wei Pang, Kevin Qinghong Lin, Xiangru Jian, Xi He, and Philip Torr. Paper2Poster: Towards multimodal poster automation from scientific papers. In Advances in Neural Information Processing Systems, 2025.arxiv.org/abs/2505.21497
- [27]Noah Shinn, Federico Cassano, Ashwin Gopinath, Karthik Narasimhan, and Shunyu Yao. Reflexion: Language agents with verbal reinforcement learning. In Advances in Neural Information Processing Systems, 2023.arxiv.org/abs/2303.11366
- [28]Chenglei Si, Yanzhe Zhang, Ryan Li, Zhengyuan Yang, Ruibo Liu, and Diyi Yang. Design2Code: Benchmarking multimodal code generation for automated front-end engineering. In Proceedings of the 2025 Conference of the Nations of the Americas Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume 1: Long Papers), pages 3956–3974. Association for Computational Linguistics, 2025. doi: 10.18653/v1/2025.naacl-long.199. https://aclanthology.org/2025.naacl-long-199/. Yiyou Sun, Xinyang Han, Weichen Zhang, Yuanbo Pang, Tianyu Wang, Yuhan Cao, Yixiao Huang, Chris Duruoiu, Haoyun Zhang, Jeffrey Lin, Weishu Zhang, Tyler Zeng, Ying Yan, Bo Liu, Hanson Wen, Mingyang Xu, Xiaoyuan Liu, Zimeng Chen, Weiyan Shi, Amanda Dsouza, Vincent Sunn Chen, Patrick Bryant, Carl Boettiger, Yamini Rangan, Bradley Rothenberg, Kyle Steinfeld, Arvind Rao, Tapio Schneider, Georgios Yannakakis, Laure Zanna, Kaan Ozbay, Ida Sim, Tarek Zohdi, George Em Karniadakis, Jack Gallant, Teresa Head-Gordon, Yushan Li, Wenxi Deng, Tao Sun, Huiqi Wang, Zhun Wang, Justin Xu, Chris Yuhao Liu, Yafei Cheng, Rongwang Hu, Aras Bacho, Shengcao Cao, Zengyi Qin, Yixiong Chen, Hengduan Fan, Hao Liu, Lin Zeng, Shashank Muralidhar Bharadwaj, Litian Gong, Yingxuan Yang, Maojia Song, Ruheng Wang, Zongzheng Zhang, Honglin Bao, Shuo Lu, Jianhong Tu, Zhonghua Wang, Zheng Zhang, Zijiao Chen, Yanqiong Jiang, Zhendong Li, Bohan Lyu, Chang Ma, Peiran Xu, Benran Zhang, Shangding Gu, Haoyue Hua, Haoyang Li, Wanzhe Liao, Chengzhi Liu, Junbo Peng, Haoran Sun, Zechen Xu, Bo Chen, Jiayi Cheng, Yi Jiang, Keying Kuang, Yuan Li, Youbang Pan, Ziyan Rao, Alexander Schubert, Yifan Shen, Vincent Siu, Xiatao Sun, Kangqi Zhang, Xiaopan Zhang, Yuchen Zhu, Ishaan Singh Chandok, Lei Ding, Jingxuan Fan, Andrew Glover, Jiaming Hu, Yiran Hu, Wenbo Huang, Zixin Jiang, Haoran Jin, Lukas Kim, Ming Liu, Yang Liu, Alireza Rafiei, Xuhuan Shen, Kunyang Sun, Sophia Sun, Ting Sun, Eric Wang, Yixin Wang, Hanwen Xing, Sihan Xu, Yuzheng Xu, Zhongxing Xu, Zhiling Yan, Boqin Yuan, Ruiqi Zhang, Yifan Zhang, Zibo Zhao, Liana, Santanu Bosu Antu, Haoyue Bai, Carlo Bosio, Joseph Cavanagh, Patricia Cavazos-Rehg, Tianxing Chen, Xuewen Chen, Yipu Chen, Chenyu Zhu, Chen Dai, Stefano De Castro, Yunfu Deng, Kaustubh Dhole, Jiayuan Ding, Chenchen Du, Zehang Du, Hao Fan, Run-Ze Fan, Hengyu Fu, Shi Gu, Yifan Gu, Charlie Guo, Baihe Huang, Baixiang Huang, Rimika Jaiswal, Zhihan Jiang, Ran Jin, Erin Kasson, Xin Lan, Joseph Lee, Deren Lei, Chenyu Li, Daofeng Li, Haitao Li, Hongwei Li, Jingyan Li, Xiao Li, Yi Li, Yinsheng Li, Yuangang Li, Zhixu Li, Wenyu Liang, Longtai Liao, Kevin Qinghong Lin, Andy Zeyi Liu, Che Liu, Jiaming Liu, Kaiyuan Liu, Xuan Liu, Pan Lu, Wenbo Lv, Yicheng Lyu, Qiuyang Mang, Kyle Montgomery, Yuzhou Nie, Ruoxi Ning, Jorin Overwiening, Xu Pan, Layna Paraboschi, Core Francisco Park, Justin Purnomo, Swati Rajwal, Scott Rankin, Bixuan Ren, Yiren Rong, HaoYang Shang, Ventus Shaw, Fiona Shen, Jiawei Shen, Minqi Shi, Shi Qiu, Huaxiu Yao, Tianneng Shi, Jonah So, Vladislav Susoy, Hannah Szlyk, Haocheng Wang, Jialu Wang, Wei Wang, Xinyu Wang, Zehao Wang, Dowling Wong, Angela Wu, Dehao Wu, Fangyu Wu, Mengyuan "Millie" Wu, Yu Wu, Yuchen Wu, Yuhao Wu, Qingpo Wuwu, Weihang Xiao, Yongyi Xiong, Fan Xu, Ruiling Xu, Mingxuan Yan, Benjamin Yang, Jirong Yang, Sen Yang, Xiaoli Yang, Yushi Yang, Haoran Ye, Xiaohua Yu, Zhengming Yu, Chenlong Zhang, Chi Zhang, Hanning Zhang, Hanwen Zhang, Junge Zhang, Kunpeng Zhang, Song Zhang, Wenjin Zhang, Wenshuo Zhang, Ying Zhang, Yizhi Zhang, Brian Zhao, Qijian Zhao, Yimin Zhao, Yuhaohua Zheng, Liwei Zhou, Tianyue Zhou, Sichen Zhu, Siqi Zhu, Yan Zhu, Yishu Zhu, Jierui Zuo, Chonghao Cai, Helena Casademunt, Wenjia Chen, Cheng Cheng, Nawen Deng, Rao Fu, Tianfu Fu, Yifan Han, He Ren, Zhenyu He, Qiao Jin, Langlang Li, Luetai Li, Sylvia Liu, Lu Lu, Luqing Zhou, Subhabrata Mukherjee, Yunqi Ouyang, Yin Ren, Dawei Shi, Haoran Wu, Zhiyue Wu, Hannah Yao, Zhuoran Yi, Jenny Yu, Rhea Zhan, Hang Zhou, Blake Zhu, Junfan Zhu, Alan Yuille, Yang Liu, Russell Alan Poldrack, Jiachen Li, Zhenglu Li, Molei Tao, Jing Huang, Wenqi Shi, Costas Spanos, Lichao Sun, Chenguang Wang, Orson Xu, Zhen Dong, Hector Gomez, Aylin Caliskan, Ali Emami, Haimin Hu, Zhi Li, Lihui Liu, Murphy Niu, Yi Shao, Jianxin Sun, Mikko Tolonen, Ting Wang, Sanjiv Das, Yanjun Gao, Wenbo Guo, Erika J Schneider, Zhiyong Lu, Yian Ma, Mark Mueller, Radha Poovendran, Somayeh Sojoudi, Yinglun Zhu, and Dawn Song. Agents’ Last Exam. arXiv preprint arXiv:2606.05405, 2026.
- [29]Tripo. Tripo OpenAPI documentation: Introduction, 2026. https://docs.tripo3d.ai/get-started/introduction.html. Accessed September 21, 2026.
- [30]Tianfu Wang, Zhezheng Hao, Xilin Xia, Lixin Liu, Mengkang Hu, Hongzhang Liu, Xi Chen, Ziyan Liu, Xiankun Lin, Weijia Zhang, Nicholas Jing Yuan, and Hui Xiong. Agentic artifact creation: Systems, evaluation, principles, and opportunities. arXiv preprint arXiv:2608.28122, 2026.arxiv.org/abs/2608.28122
- [31]Hu Wei. From agent loops to structured graphs: A scheduler-theoretic framework for LLM agent execution. arXiv preprint arXiv:2604.11378, 2026.arxiv.org/abs/2604.11378
- [32]Yuhao Wu, Jingyuan Zhang, Jiajun Shi, Xinping Lei, Qingshui Gu, Yuxuan Zhang, Zexuan Wang, Chen He, Chen Huang, Maojia Song, Zhiyuan Zeng, Shaowen Wang, Jinkai Liu, Yunfeng Shi, Jiaheng Liu, Shen Yan, Wenhao Huang, Ge Zhang, and Wenxuan Zhang. HarnessDev: Can LLMs create and evolve their own agent harness? arXiv preprint arXiv:2609.01437, 2026.arxiv.org/abs/2609.01437
- [33]Tianbao Xie, Danyang Zhang, Jixuan Chen, Xiaochuan Li, Siheng Zhao, Ruisheng Cao, Toh Jing Hua, Zhoujun Cheng, Dongchan Shin, Fangyu Lei, Yitao Liu, Yiheng Xu, Shuyan Zhou, Silvio Savarese, Caiming Xiong, Victor Zhong, and Tao Yu. OSWorld: Benchmarking multimodal agents for open-ended tasks in real computer environments. In Advances in Neural Information Processing Systems, Datasets and Benchmarks Track, 2024.arxiv.org/abs/2404.07972
- [34]Xiangyuan Xue, Zeyu Lu, Di Huang, Zidong Wang, Wanli Ouyang, and Lei Bai. ComfyBench: Benchmarking LLM-based agents in ComfyUI for autonomously designing collaborative AI systems. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 24614–24624, 2025.arxiv.org/abs/2409.01392
- [35]John Yang, Carlos E. Jimenez, Alexander Wettig, Kilian Lieret, Shunyu Yao, Karthik Narasimhan, and Ofir Press. SWE-agent: Agent-computer interfaces enable automated software engineering. In Advances in Neural Information Processing Systems, volume 37, 2024.arxiv.org/abs/2405.15793
- [36]Shunyu Yao, Jeffrey Zhao, Dian Yu, Nan Du, Izhak Shafran, Karthik Narasimhan, and Yuan Cao. ReAct: Synergizing reasoning and acting in language models. In International Conference on Learning Representations, 2023.arxiv.org/abs/2210.03629
- [37]Mengqi Yuan, Zilong Zhou, Xinhuang Xiong, Weiming Wu, Jiayang Sun, Jiamin Song, Kaiqian Cui, Bowen Wang, Haoyuan Wu, Yitong Li, Dunjie Liu, Haikong Lu, Qi Zhen, Xinyuan Wang, Jiadi Deng, Yuhao Yang, Cheng Chen, Bouyan Zheng, Alex Su, Xiao Yu, Hao Zou, Saaket Agashe, Xing Han Lu, Manpreet Kaur, Zhengyang Qi, Vincent Sann Chen, Frederic Sala, Dayiheng Liu, Junyang Lin, Zhou Yu, Yu Su, Siva Reddy, Xin Eric Wang, Peng Qi, Tianbao Xie, and Tao Yu. OSWorld 2.0: Benchmarking computer use agents on long-horizon real-world tasks. arXiv preprint arXiv:2606.29537, 2026.arxiv.org/abs/2606.29537
- [38]Hao Zheng, Xinyan Guan, Hao Kong, Wenkai Zhang, Jia Zheng, Weixiang Zhou, Hongyu Lin, Yaojie Lu, Xianpei Han, and Le Sun. PPTAgent: Generating and evaluating presentations beyond text-to-slides. In Proceedings of the 2025 Conference on Empirical Methods in Natural Language Processing, pages 14402–14418. Association for Computational Linguistics, 2025.arxiv.org/abs/2501.03936
- [39]Mingchen Zhuge, Changsheng Zhao, Dylan R. Ashley, Wenyi Wang, Dmitrii Kshbullin, Yunyang Xiong, Zechun Liu, Ernie Chang, Raghuraman Krishnamoorthi, Yuandong Tian, Yangyang Shi, Vikas Chandra, and Jurgen Schmidhuber. Agent-as-a-Judge: Evaluate agents with agents. In Proceedings of the 42nd International Conference on Machine Learning, volume 267 of Proceedings of Machine Learning Research, pages 80569–80611. PMLR, 2025.













