WorksComputer Science
FinCUABuild: Can Agents Build Reliable Benchmarks for Dynamic Financial Computer Use?
Abstract
Financial scenarios are diverse and complex, spanning varying data conditions, tool configurations, and workflows. Yet existing CUA, Computer-Using Agent, evaluation tasks remain largely manually constructed, limiting scalable coverage of real-world financial scenarios. Then, can agents autonomously construct diverse CUA evaluation tasks for financial scenarios? Evaluating this capability poses three key challenges: scenario coverage of construction requests, fair comparison across construction methods, and reliable assessment of generated task quality. To solve these, we introduce FinCUABuildBench, a benchmark for evaluating financial CUA task construction, featuring: (i) 576 construction requests covering 24 financial workflows and three types of runtime variation; (ii) standardized input, budget, and output specifications; and (iii) a task qualification mechanism based on execution tests and quality checks. We further introduce FinCUABuildAgent, a multi-agent system for automatically constructing dynamic financial CUA evaluation tasks. It consists of three modules that jointly construct tasks, environments, and validators. On FinCUABuildBench, under the same model backbone, existing agent-based construction methods achieve strict qualification rates of only 1.3-8.3%, while FinCUABuildAgent reaches 31.3%. Downstream evaluations further show that the constructed tasks can effectively differentiate CUA task-execution capabilities. These results demonstrate that agents can autonomously construct financial CUA tasks with meaningful evaluation value, offering a practical path toward broader evaluation coverage in financial scenarios. Code: https://github.com/FengxianJi/FinCUABuild
Introduction
Evaluation of computer-use agents (CUAs) has expanded to real-world web environments, desktop interactions, and enterprise workflows ([?]; [8]; [?]; [11]). Existing interactive benchmarks further provide controllable application states, dynamically instantiated tasks, and policy-constrained tool interactions, laying the foundation for evaluating complex workflows ([?]; [?]; [?]; [?]; [9]). In finance, workflows involve cross-file retrieval, spreadsheet operations, computation, and reporting, and must cover diverse data conditions, tool combinations, and business processes. SpreadsheetBench and Finch respectively explore real-world spreadsheet manipulation and enterprise-level financial workflow evaluation. Notably, even with LLM-assisted task mining, Finch still required over 700 hours of expert annotation ([19]; [7]). Such construction costs hinder the scalable expansion of evaluation tasks and motivate our central question: Can agents autonomously construct CUA evaluation tasks that cover diverse financial scenarios?
Existing work still has two key limitations in constructing dynamic financial CUA tasks. First, there is no unified benchmark for evaluating task construction capabilities, as existing financial benchmarks mainly assess agents on completing predefined tasks ([?]; [?]; [?]; [?]). Second, there is no construction agent tailored to this setting. Although prior automated construction methods have explored goal-driven generation, multi-agent collaboration, and task synthesis, they provide limited support for jointly constructing financial evidence, application states, dynamic events, and verification rules ([?]; [?]; [?]). Therefore, we study both how to evaluate financial CUA task construction capabilities and how to improve them.
Addressing these two issues requires tackling challenges in both construction-capability evaluation and automated task construction. For evaluation, (i) construction requests should cover diverse financial workflows and runtime variations, rather than increasing diversity through superficial paraphrasing; (ii) differences in input information, tool access, and resource budgets can confound method comparisons, requiring standardized evaluation conditions; and (iii) a plausible task description does not guarantee an executable environment, valid runtime events, or correct verification, so the quality of the complete task package must be assessed. For construction, (i) abstract capability targets must be translated into concrete dependencies among evidence, states, and actions to form executable tasks; (ii) tasks and verifiers may jointly accept incorrect outcomes, requiring independent correctness auditing; and (iii) errors can propagate across dependencies among tasks, environments, and verifiers, requiring failure localization and targeted repair of affected artifacts.
To address these challenges, we introduce FinCUABuildBench, shown as Figure 1(a), a benchmark for evaluating the construction of dynamic financial CUA tasks. For scenario coverage, the benchmark includes 576 construction requests derived from 24 financial workflow categories, 192 workflow templates, and three types of runtime variation: state updates, tool failures, and policy changes. For fair comparison, it standardizes construction inputs, resource budgets, and output specifications so that different methods are evaluated under comparable conditions. For quality assessment, it combines financial evidence checking, reference execution, reset-and-replay testing, dynamic-event validation, and verifier testing to determine whether a generated task is qualified. We further introduce FinCUABuildAgent, shown as Figure 1(b), which uses capability-dependency compilation to translate target capabilities into task requirements, independent quality auditing to inspect tasks and verifiers, and targeted feedback-driven repair to resolve cross-stage errors, jointly constructing tasks, environments, and verifiers. Under the same model backbone, FinCUABuildAgent achieves a strict qualification rate of 31.3%, outperforming general-purpose construction agents at 1.3–8.3%. The 144 qualified tasks reveal differences in downstream CUAs’ dynamic adaptation capabilities, while the qualification rate on 90 held-out construction requests is 13.3%. These results support the evaluation value of automated task construction, while also indicating substantial room for improved generalization.
Figure 1. Overview of FinCUABuildAgent. The upper panel shows the construction of a FinCUABuildBench episode, while the lower panel groups eleven specialist agents and their representative tools by stage. Labels 01–11 denote agents – in the text; shortened agent names are used for compactness.
Our main contributions and findings are: (1) A benchmark for benchmark construction. We introduce FinCUABuildBench, a 576-slot construction benchmark spanning 24 workflows, 192 WorkflowCores, and three runtime-dynamic profiles. Its unified episode contract evaluates groundedness, replayability, verifier reliability, dynamicity, capability necessity, safety, and redundancy; (2) A capability-guided multi-agent framework. We propose FinCUABuildAgent, which models dependencies among states, evidence, operators, and tools. Specialized agents jointly ground financial sources, build environments and runtime events, construct verifiers, and iteratively test and repair auditable CUA episodes; (3) A controlled empirical evaluation. Under the same model backbone, FinCUABuildAgent achieves a 31.3% strict qualification rate, compared with 1.3–8.3% for general-purpose construction agents. The 144 qualified tasks distinguish downstream CUAs in dynamic adaptation, while a 13.3% rate on 90 held-out requests indicates cross-workflow transferability.
Related Work
Financial benchmarks and workflow agents. Financial benchmarks first emphasized report-based question answering and numerical reasoning through FinQA, TAT-QA, ConvFinQA, FinanceBench, and MultiHiertt (Chen et al. 2022a; Zhu et al. 2021; Chen et al. 2022b; Islam et al. 2023; Zhao et al. 2022), then expanded to knowledge-intensive reasoning and financial data analysis in FinanceMATH, FinBen, and FinDABench (Zhao et al. 2024; [17]; Xie et al. 2024a). Spreadsheet-oriented work moves toward executable artifact manipulation: SheetCopilot and SheetAgent study spreadsheet control, while SpreadsheetBench and Finch cover realistic workbook and enterprise finance and accounting tasks ([15]; [4]; [19]; [7]). Recent financial-agent work further targets professional spreadsheets, financial research, banking workflows, MCP-based tool use, and execution-grounded safety ([13]; Wang2026; [14]; Zhu2026; Yang2026). FinCUABuildBench evaluates the construction process itself: whether a method can produce a replayable episode whose evidence, application state, tools, post-start event, reset logic, and verifier remain jointly consistent.
Computer-use agents and stateful tool use. General CUA benchmarks span realistic web and desktop interaction (Zhou2024; Xie2024b), compositional enterprise work and professional data workflows ([8]; [1]; [3]), and controllable application and mobile environments (Trivedi2024; Rawles2025). ToolSandbox and -bench add stateful, multi-turn, or policy-constrained tool interaction ([18]; Yao2025). These studies evaluate agent execution in supplied task environments. FinCUABuildBench instead targets the synthesis of a complete package comprising the environment, task, event, and verifier.
Automated benchmark construction. Automated construction ranges from Dynabench’s human–model loop to graph-generated or combinatorial evaluations in DyVal and SKILL-MIX ([12]; Zhu2024; Yu2024). AutoBencher optimizes declared desiderata, BENCHAGENTS coordinates planning, generation, verification, and evaluation, and the Benchmark Agent preprint adds design, grounding, and allocation stages ([16]; [2]; Xiong2026). APIGen-MT generates verified multi-turn task blueprints, while EnvScaler and AgentSynth synthesize tool environments, terminal validators, or verified long-horizon tasks ([20]; Song2026; Xie2026b; Zhang2026). We instantiate AutoBencher, BENCHAGENTS, and Benchmark Agent as matched construction baselines. FinCUABuildAgent further couples financial grounding, cross-artifact state, deterministic post-start events, reset and replay semantics, and mutation-tested hidden verifiers within a single release contract.
FinCUABuildBench
Problem Formulation
We study the automatic construction of benchmarks for dynamic financial computer use. Let be the number of preregistered target slots, indexed by . The construction request for slot is defined as
where specifies the financial workflow and specifies the target capability. The sets , , and provide the admissible sources, application environments, and builder-side tools, respectively, while contains the safety, privacy, resource, and release constraints.
Let denote the set of benchmark-construction methods. Given , a method generates a candidate episode
Here, is the user-visible instruction; is a versioned source bundle; is the resettable initial state; contains the terminal goal predicates; is the evaluated agent’s ToolGraph; is a deterministic runtime-event program; is a hidden verifier; is a reset and replay manifest; and specifies permissions and forbidden effects.
A candidate is qualified only if it satisfies every mandatory constraint in and passes every preregistered episode-level hard gate: L-Acc, Trace, Exec, verifier pass, Dyn, CapMatch, and nonduplication. The strict conjunction is formalized in (25). H-Acc and CapCov are reported as complementary sample-level and portfolio-level diagnostics and do not enter the episode-level release indicator.
FinCUABuildBench Construction
FinCUABuildBench is designed to evaluate whether a construction method can generate complete dynamic financial computer-use episodes. It covers 24 financial workflow categories spanning reporting and analysis, planning and accounting operations, markets, risk and treasury, and compliance and governance. For each category, we define eight semantically distinct WorkflowCores, each specified by its business objective, source family, artifact contract, and target capability path. Each WorkflowCore is instantiated under the three runtime-dynamic profiles summarized in Figure 2, defining 576 preregistered target slots in total.
Figure 2. The FinCUABuildBench task space. 24 financial workflow categories, each with eight semantically distinct WorkflowCores, instantiated under three runtime-dynamic profiles.
P1: State Update, covering post-start changes to financial data or application state, such as an amended filing, refreshed market value, changed invoice status, collaborator edit, or updated ledger record;
P2: Tool Failure, covering disruptions to tools or interfaces, such as a timeout, session expiration, delayed recalculation, unavailable primary tool, partial write, or changed interface state;
P3: Policy Change, covering changes to permissions, approvals, or compliance requirements, such as an approval rejection, modified threshold, permission restriction, compliance hold, or newly introduced instruction that conflicts with the applicable policy.
Each runtime event is injected after execution begins, is deterministically replayable, changes at least one action required for success, and leaves at least one feasible completion path. To prevent workflow-level leakage, the P1–P3 instances of the same WorkflowCore are assigned to the same split. Each episode follows the contract in (2). Public workflows use versioned, hashed sources, whereas enterprise workflows use privacy-safe synthetic fixtures with cross-file consistency checks. The evaluated agent observes only the task inputs, workspace, and permitted tools; the event controller, reference goal state, oracle, verifier, and scoring logic remain evaluator-private.
Candidate episodes undergo iterative generation, audit, and repair, and only those satisfying the strict release criterion enter the qualified pool. For downstream evaluation, we sample two qualified episodes from each of the category–profile cells, yielding 144 episodes. All selected episodes pass independent execution, deterministic replay, verifier, event-validity, and safety audits, and the frozen reference solver succeeds on all of them. The subset is fixed before any downstream-agent results are observed.
Automation-Point Verification
FinCUABuildBench evaluates executions through automation points, which are programmatically verifiable semantic checkpoints in a financial workflow. For each episode , we organize the task objective, artifact contract, evidence requirements, and runtime event into an automation-point DAG , where contains the automation points and encodes their prerequisite and event-order dependencies. A point may represent grounding a financial fact, updating an artifact, transferring information across tools, or adapting to a runtime event. Low-level interface actions are excluded.
For condition , let be the evaluated episode indices and let be the nonempty set of points applicable to episode . For run of agent , where is the number of repeated runs, let contain the observed application states, produced artifacts, and execution and evidence trace. For , let and be Boolean predicates that check the semantic outcome and its dependency constraints, respectively. The binary node-pass indicator is
where equals one when is true and zero otherwise. Because verification uses semantic outcomes, different valid tools and action orders may satisfy the same automation point.
Let be preregistered point weights satisfying , and let equal one when a forbidden effect occurs. The run-level automation-point completion score and its aggregate are
Table 2 reports APComp static and APComp dynamic. Each automation point is binary, while the run-level score is fractional. Static completion evaluates baseline deliverable points, whereas dynamic completion evaluates event-transition and post-event adaptation points. These condition-specific point sets are complementary rather than nested, so neither aggregate is constrained to be smaller than the other.
where contains only episodes to which dimension applies. Artifact, Evidence, Trace Integrity, Financial Grounding, Sheet Contract, and Cross-Artifact Consistency use their corresponding verifier checks. Replan Recovery and Fallback Recovery are restricted to runs in which replanning or an allowed fallback is required. Safety Violation is the mean forbidden-effect indicator .
FinCUABuildAgent
FinCUABuildAgent operationalizes dynamic financial benchmark construction as a capability-guided and validation-gated multi-agent pipeline. Given a preregistered target slot, it first refines the request into a grounded WorkflowCore and then compiles it into an executable episode. Figure 1 summarizes the construction stages and their realization through specialist agents and stage-specific tools.
FinCUABuildAgent Architecture
The Factory Controller coordinates eleven specialist agents through a generate–audit–repair workflow and routes failed candidates to the responsible stage. The Benchmark Scout identifies financially meaningful workflow candidates, which are then formalized by the Workflow Miner and Process Analyst in terms of objectives, inputs, applications, and deliverables. The Provenance and Grounding Agent verifies the entity, reporting period, unit, and financial sources. The Taxonomy and Allocation Agent assigns each grounded workflow to the benchmark taxonomy, balances portfolio coverage, and removes semantic duplicates.
Given a grounded WorkflowCore and its target capability, the Capability and Dependency Compiler specifies the goal conditions, evidence dependencies, intermediate states, and semantic operations required to instantiate the capability. The Workflow and Episode Designer integrates these elements into a user-facing task with artifact, evidence, safety, and runtime-event specifications. The required operations are then mapped by the Tool Architect and Synthesizer to an episode-level ToolGraph with valid fallback paths. The Environment and Dynamics Composer materializes the financial workspace, initial application state, reset configuration, and replayable runtime event to produce an executable episode.
The Independent Oracle and Verifier Agent derives the hidden goal state and verification conditions from grounded evidence before observing any candidate solution. The Reference Solver executes the task through the same interface available to evaluated CUA agents, confirming that the episode remains solvable after the runtime change. The Red-Team and Calibration Agent tests incorrect financial values, stale evidence, ignored events, unauthorized effects, and shortcut solutions. The Factory Controller releases only episodes that pass these checks and returns failed episodes for targeted revision. Separating ToolGraph specification from environment materialization and committing the oracle before reference execution reduce cross-stage leakage and preserve independence between episode construction and qualification.
Typed Agent–Tool Orchestration
FinCUABuildAgent connects its eleven specialist agents through a typed orchestration graph , where . Each agent receives the validated artifacts produced by its parent agents, invokes a role-specific allowlist of tools , and returns an artifact together with its tool-call trace . The handoff relation checks the required artifact type, source lineage, episode version, and cross-stage dependencies. We activate the downstream stage according to
Here, contains traces in which every tool invocation is authorized and satisfies its declared input–output contract. Thus, only when both the artifact and its tool-use trace pass validation; otherwise, the failed contract is returned to the responsible agent for repair.
Agents – handle workflow discovery and grounding. They identify candidate workflows, retrieve relevant source entities, and apply licensing and deduplication constraints so that only consistent and non-redundant WorkflowCores enter the graph.
Agents – compile and materialize executable episodes. They assemble goal predicates, state representations, and capability dependencies, select compatible schemas and runtime-event families, and synthesize interface adapters when needed. Contract, permission, and runtime-conformance checks precede the instantiation of a reproducible environment with a fixed initial state.
Agents – perform qualification and assurance through separate oracle, reference-execution, and red-team pathways. The oracle inspects hidden goal states and simulates event effects under controlled resets, while the reference solver uses only episode-visible interfaces to produce executable traces. The red-team evaluator probes stale evidence, leakage paths, invalid shortcuts, and unauthorized side effects. Contract or handoff violations trigger rollback and recomputation of the affected downstream artifacts.
Experiments
Experimental Setup
Benchmarks. We evaluate benchmark construction on Fin-CUABuildBench, which contains 576 target slots across 24 financial workflow categories and three dynamic profiles: State Update, Tool Failure, and Policy Change. To assess the downstream diagnostic value of generated tasks, we evaluate CUA agents on a balanced subset of 144 qualified episodes, with three independent runs per agent. We further test construction transfer on 90 held-out requests covering three unseen workflow scenarios: FP&A Variance Investigation, AP Exception Handling, and Portfolio Risk & Compliance.
Baseline methods. We compare FinCUABuildAgent against rule-based templates, LLM prompting (with and without tools), tool-augmented prompting, and adapted benchmark-generation systems including AutoBencher [16], BENCHAGENTS [2], and Benchmark Agent [?], as well as a single-agent tool-using baseline. We evaluate multiple backbones including MiniMax-M3, GPT-5.5, Qwen3.5-397B-A17B, Kimi-K3, and MiMo-V2.5-Pro depending on the setting.
Downstream evaluation on the 144-task subset covers general-purpose agents, financial agents, and controlled baselines. We also report held-out transfer and ablations against a budget-matched collapsed-controller variant and component removals. The collapsed controller in Table 4 is an internal topology ablation and is distinct from the external single-agent baseline in Table 1.
| Family | Generation method | Backbone | H-Acc ↑ | L-Acc ↑ | Trace ↑ | Exec ↑ | V-Acc ↑ | Dyn ↑ | CapMatch ↑ | CapCov ↑ | Dup ↓ | Qual ↑ |
| Rule-based | Deterministic templates | n/a | 0.0% | 11.1% | 74.7% | 68.4% | 69.2% | 29.6% | 0.0% | 0.0% | 33.3% | 0.0% |
| Direct LLM no tools | Direct prompting | MiniMax-M3 | 1.6% | 24.3% | 42.4% | 46.0% | 54.2% | 36.5% | 23.6% | 5.2% | 28.1% | 11.3% |
| GPT-5.5 | 3.2% | 27.3% | 38.4% | 56.6% | 37.6% | 17.4% | 31.9% | 3.1% | 34.6% | 8.7% | ||
| Qwen3.5-397B-A17B | 2.1% | 26.5% | 42.7% | 57.6% | 38.9% | 43.7% | 45.4% | 8.3% | 45.8% | 6.8% | ||
| Kimi K3 | 6.0% | 32.3% | 53.1% | 52.5% | 45.2% | 20.4% | 46.9% | 4.2% | 45.2% | 12.2% | ||
| MiMo-V2.5-Pro | 2.4% | 25.0% | 39.6% | 55.6% | 37.5% | 52.5% | 35.7% | 5.0% | 41.8% | 12.3% | ||
| Direct LLM + finance tools | Tool-augmented prompting | MiniMax-M3 | 12.9% | 21.1% | 67.1% | 57.5% | 54.2% | 38.8% | 56.9% | 31.6% | 37.4% | 17.5% |
| Qwen3.5-397B-A17B | 13.5% | 25.1% | 65.2% | 52.3% | 38.9% | 43.7% | 51.3% | 35.8% | 40.8% | 12.8% | ||
| Kimi K3 | 14.1% | 25.0% | 62.1% | 56.2% | 43.6% | 37.3% | 49.7% | 30.0% | 38.1% | 10.9% | ||
| MiMo-V2.5-Pro | 12.1% | 35.9% | 63.1% | 57.6% | 38.9% | 43.6% | 45.8% | 28.3% | 46.5% | 10.8% | ||
| Agentic baselines | Adapted AutoBencher | MiniMax-M3 | 12.0% | 10.4% | 11.1% | 11.1% | 10.9% | 10.9% | 10.4% | 37.5% | 64.1% | 8.3% |
| Adapted BENCHAGENTS | MiniMax-M3 | 5.2% | 5.0% | 5.0% | 5.0% | 5.0% | 5.0% | 5.0% | 12.5% | 65.5% | 1.3% | |
| Adapted Benchmark Agent | MiniMax-M3 | 7.8% | 7.6% | 8.0% | 8.0% | 7.8% | 8.0% | 7.5% | 25.0% | 63.0% | 4.5% | |
| Single Agent + finance tools | MiniMax-M3 | 6.8% | 7.3% | 7.3% | 7.3% | 7.3% | 7.3% | 7.3% | 25.0% | 64.3% | 6.9% | |
| Ours | FinCUABuildAgent Multi-Agent | MiniMax-M3 | 33.4% | 55.7% | 89.1% | 73.8% | 72.3% | 70.0% | 78.5% | 50.7% | 12.5% | 31.3% |
| Qwen3.5-397B-A17B | 30.4% | 53.1% | 90.3% | 73.1% | 64.7% | 90.3% | 67.9% | 58.3% | 12.2% | 41.9% | ||
| Kimi K3 | 35.2% | 50.8% | 85.4% | 65.9% | 71.7% | 85.5% | 66.1% | 58.3% | 8.9% | 40.3% | ||
| MiMo-V2.5-Pro | 28.6% | 49.7% | 89.3% | 81.8% | 78.1% | 72.6% | 70.5% | 44.8% | 11.9% | 34.9% |
Table 1. Benchmark-generation quality across financial CUA construction methods. Agentic baselines use MiniMax-M3 under matched construction settings. Values are averaged over the preregistered construction runs and then macro-averaged across WorkflowCores.
| Variant | L-Acc ↑ | Trace ↑ | Exec ↑ | V-Acc ↑ | Dyn ↑ | CapCov ↑ | Dup ↓ |
| FinCUABuildAgent Multi-Agent | 55.7% | 89.1% | 73.8% | 72.3% | 70.0% | 50.7% | 12.5% |
| Collapsed controller (budget matched) | 26.0% | 22.9% | 18.8% | 40.1% | 12.5% | 27.3% | 49.7% |
| w/o Grounding & Provenance | 37.5% | 79.0% | 10.4% | 88.5% | 38.4% | 0.0% | 25.0% |
| w/o Dependency Compiler | 0.0% | 40.0% | 86.6% | 0.0% | 0.0% | 0.0% | 100.0% |
| w/o Tool Architect | 0.0% | 3.2% | 0.0% | 1.7% | 0.6% | 0.0% | 100.0% |
| w/o Dynamics Composer | 1.0% | 10.2% | 1.6% | 1.0% | 5.8% | 1.6% | 96.7% |
| w/o Independent Oracle | 8.9% | 61.6% | 58.4% | 52.0% | 63.2% | 35.9% | 43.2% |
| w/o Red Team & Repair | 0.5% | 78.9% | 69.8% | 68.8% | 73.2% | 39.6% | 20.8% |
Table 4. Component and topology ablations of FinCUABuildAgent with MiniMax-M3 on the same 576 target slots as the corresponding full-system row in Table 1. The collapsed-controller variant is budget matched; all rows follow the same construction-run aggregation and scoring protocol.
Evaluation protocol. All methods share the same target specifications, resource budgets, and evaluation interfaces. Tool access follows the declared baseline setting and is held fixed within each matched comparison; no method has access to gold ToolGraphs or reference trajectories. The primary metric is Qualified Task Rate (Qual), the fraction of target slots that pass the strict conjunction of every preregistered episode-level hard gate and mandatory constraint in Γ. H-Acc is a stratified human audit and is not included in this conjunction; V-Acc is reported as a mean verifier score, while strict qualification uses its thresholded episode-level form. We further report L-Acc, Trace, Exec, V-Acc, Dyn, CapMatch, CapCov, and Dup as diagnostic metrics. Construction metrics are macro-averaged over WorkflowCores. Human and LLM-based assessments are method-blind, while execution-derived metrics are computed automatically from frozen artifacts and traces. Full metric definitions are provided in the supplementary material.
Main Result
Table 1 shows that FinCUABuildAgent provides the strongest overall profile on the coupled construction metrics. Finance tools alone consistently improve provenance and capability coverage, but their effects on dynamicity, verifier reliability, and strict qualification remain uneven. Across the four backbones shared with tool-augmented prompting, FinCUABuildAgent raises Dyn by 29.0–48.2 percentage points, reduces Dup by 24.9–34.6 points, and improves strict Qual by 13.8–29.4 points. Qwen3.5 achieves the highest Qual at 41.9%. The adapted agentic baselines reach only 1.3–8.3% Qual under MiniMax-M3, indicating that generic benchmark-generation pipelines do not directly recover the coupled financial grounding, executable state, runtime event, and verification contracts required here.
Diagnostic Effectiveness of Generated Tasks
Table 2 evaluates whether the qualified tasks constructed by FinCUABuildAgent provide meaningful diagnostic signals for downstream CUAs. Since the frozen reference solver succeeds on every selected task while evaluated agents obtain non-saturated and differentiated scores, the generated tasks are both feasible and discriminative. Native general-agent frameworks achieve 0.622–0.660 Static Completion and near-perfect financial grounding but only 0.004–0.043 Dynamic Completion, showing that grounding and general tool use do not ensure adaptation to runtime events. The external structured configuration reaches the highest Dynamic Completion (0.676), confirming that stronger performance is attainable; its Dynamic score exceeds its Static score because the metrics use different, non-nested automation-point sets. Meanwhile, the matched SingleController combines near-perfect artifact, evidence, static-completion, and cross-artifact scores with only 0.258 Dynamic Completion. Together, these results show that the generated tasks expose complementary failure modes in financial grounding, task completion, and runtime adaptation.
| Category | Evaluated Agent / Configuration | Artifact ↑ | Evidence ↑ | Trace Integrity ↑ | Static Completion ↑ | Dynamic Completion ↑ | Financial Grounding ↑ | Sheet Contract ↑ | Cross-Artifact Consistency ↑ | Replan Recovery ↑ | Fallback Recovery ↑ | Safety Violation ↓ |
| Direct MiniMax protocols | Text-only | 0.709 | 0.000 | 0.384 | 0.320 | 0.081 | 0.000 | 0.610 | 0.018 | 0.132 | 0.147 | 0.153 |
| Direct MiniMax protocols | Artifact draft | 0.687 | 0.000 | 0.384 | 0.326 | 0.283 | 0.167 | 0.610 | 0.084 | 0.631 | 0.430 | 0.100 |
| Direct MiniMax protocols | Structured evidence | 0.719 | 0.493 | 0.257 | 0.520 | 0.069 | 0.929 | 0.608 | 0.217 | 0.151 | 0.100 | 0.009 |
| Direct MiniMax protocols | Structured + verifier repair | 0.721 | 0.631 | 0.320 | 0.532 | 0.000 | 0.930 | 0.611 | 0.280 | 0.000 | 0.000 | 0.023 |
| Direct MiniMax protocols | External structured agent | 0.722 | 0.620 | 0.149 | 0.524 | 0.676 | 0.941 | 0.628 | 0.130 | 0.771 | 0.771 | 0.021 |
| Native general-agent frameworks | AutoGen GroupChat | 0.701 | 0.159 | 0.033 | 0.641 | 0.043 | 1.000 | 0.739 | 0.154 | 0.277 | 0.230 | 0.035 |
| Native general-agent frameworks | Magentic-One | 0.702 | 0.219 | 0.045 | 0.646 | 0.004 | 0.998 | 0.749 | 0.211 | 0.027 | 0.022 | 0.035 |
| Native general-agent frameworks | CrewAI | 0.709 | 0.101 | 0.021 | 0.660 | 0.026 | 0.994 | 0.812 | 0.090 | 0.169 | 0.141 | 0.021 |
| Native general-agent frameworks | LangGraph | 0.690 | 0.096 | 0.022 | 0.622 | 0.041 | 0.993 | 0.731 | 0.089 | 0.262 | 0.219 | 0.035 |
| Native general-agent frameworks | CAMEL | 0.684 | 0.116 | 0.025 | 0.623 | 0.033 | 0.997 | 0.711 | 0.110 | 0.212 | 0.174 | 0.028 |
| Financial-agent candidates | FinRobot | 0.684 | 0.341 | 0.005 | 0.667 | 0.000 | 1.000 | 0.695 | 0.005 | 0.000 | 0.000 | 0.319 |
| Financial-agent candidates | TradingAgents | 0.692 | 0.366 | 0.006 | 0.639 | 0.000 | 1.000 | 0.588 | 0.005 | 0.000 | 0.000 | 0.241 |
| Financial-agent candidates | FinTeam | 0.689 | 0.397 | 0.009 | 0.479 | 0.224 | 0.982 | 0.619 | 0.006 | 0.417 | 0.132 | 0.403 |
| Financial-agent candidates | FinGPT | 0.689 | 0.400 | 0.004 | 0.583 | 0.364 | 1.000 | 0.703 | 0.003 | 0.250 | 0.104 | 0.329 |
| Financial-agent candidates | FinRL | 0.690 | 0.387 | 0.008 | 0.643 | 0.015 | 1.000 | 0.674 | 0.007 | 0.025 | 0.015 | 0.227 |
| Matched agent controls | Genuine SingleController | 0.998 | 0.995 | 0.451 | 0.997 | 0.258 | 0.995 | 0.998 | 0.995 | 0.613 | 0.437 | 0.000 |
| Matched agent controls | ReAct | 0.606 | 0.035 | 0.013 | 0.375 | 0.228 | 0.496 | 0.571 | 0.004 | 0.556 | 0.411 | 0.005 |
Table 2. Diagnostic evaluation on 144 qualified tasks sampled from FinCUABuildBench. Each agent or configuration runs all 144 tasks with three independent replicates, yielding 432 trajectories. Static and Dynamic Completion are mean condition-specific automation-point completion scores computed from the same trajectories over complementary, non-nested point sets; the remaining columns average their applicable verifier checks.
Generalization to Held-Out Workflows
Table 3 shows uneven transfer to held-out WorkflowCores within the covered workflow taxonomy. V-Acc remains at least 86.3% in every workflow–profile cell, whereas only 13.3% of episodes satisfy the strict qualification criterion. The gap is most pronounced under P3: 46.7% of the episodes are executable, but only 6.7% strictly qualify. This suggests that the independent oracle transfers reliably from frozen evidence and typed goal predicates, while coupling a policy change to a replayable environment and a capability-essential post-event solution remains difficult. FP&A achieves higher aggregate Exec (56.7%) and Qual (20.0%) than AP exception handling (43.3%, 10.0%) and portfolio risk and compliance (33.3%, 10.0%), although the latter attains higher CapMatch. Overall, the results support partial within-taxonomy WorkflowCore transfer, without implying generalization to entirely unseen workflow categories.
| Scenario | Profile | Exec ↑ | V-Acc ↑ | Dyn ↑ | CapMatch ↑ | Qual ↑ |
| FP&A Variance Investigation | P1: State Update | 60.0% | 95.8% | 40.0% | 40.0% | 30.0% |
| FP&A Variance Investigation | P2: Tool Failure | 50.0% | 90.0% | 30.0% | 20.0% | 20.0% |
| FP&A Variance Investigation | P3: Policy Change | 60.0% | 100.0% | 30.0% | 10.0% | 10.0% |
| AP Exception Handling | P1: State Update | 40.0% | 95.8% | 20.0% | 20.0% | 10.0% |
| AP Exception Handling | P2: Tool Failure | 50.0% | 100.0% | 50.0% | 30.0% | 20.0% |
| AP Exception Handling | P3: Policy Change | 40.0% | 88.7% | 20.0% | 30.0% | 0.0% |
| Portfolio Risk & Compliance | P1: State Update | 30.0% | 95.8% | 20.0% | 50.0% | 20.0% |
| Portfolio Risk & Compliance | P2: Tool Failure | 30.0% | 86.3% | 30.0% | 20.0% | 0.0% |
| Portfolio Risk & Compliance | P3: Policy Change | 40.0% | 96.1% | 30.0% | 40.0% | 10.0% |
Table 3. Held-out workflow generalization under a frozen FinCUABuildAgent configuration. Each of three unseen financial scenarios contains ten held-out WorkflowCores instantiated once under P1–P3, yielding 90 episodes.
Component and Topology Ablations
Table 4 contrasts diagnostic failure patterns. Under the matched budget, staged coordination raises Exec from 18.8% for the collapsed controller to 73.8%, showing that typed handoffs reduce inconsistencies among goal predicates, Tool-Graphs, environment states, and verification logic. Removing Grounding & Provenance yields 88.5% V-Acc but only 10.4% Exec, showing that verifier accuracy alone cannot ensure a grounded, executable task portfolio. Removing the Dependency Compiler, Tool Architect, or Dynamics Composer yields Dup of at least 96.7% and CapCov of at most 1.6%, identifying these stages as the contract spine from semantic intent to executable dynamics. Removing the Independent Oracle reduces V-Acc from 72.3% to 52.0% and L-Acc from 55.7% to 8.9%, while raising Dup from 12.5% to 43.2%. Re-
moving red-team repair preserves several structural metrics but reduces L-Acc to 0.5% and raises Dup to 20.8%. These patterns show complementary stage contributions.
Conclusion
We introduced FinCUABuildBench, a 576-slot benchmark for constructing dynamic financial CUA tasks across 24 workflow categories and three runtime-dynamic profiles, together with FinCUABuildAgent, a capability-guided multi-agent framework for producing grounded, executable, replayable, and verifiable episodes. Against matched tool-augmented prompting across four backbones, FinCUABuildAgent improves Dyn by 29.0–48.2 points and strict Qual by 13.8–29.4 points while reducing Dup by 24.9–34.6 points; Qwen3.5 achieves the highest Qual of 41.9%. The generated qualified tasks yield non-saturated and differentiated automation-point completion scores across existing agents, demonstrating their downstream diagnostic value. Dynamic Completion is lower than Static Completion in 16 of 17 configurations, revealing limited adaptation to post-start changes. Held-out evaluation reaches 13.3% strict Qual and shows partial transfer within the covered taxonomy, while ablations support staged multi-agent coordination. Together, these results frame benchmark construction as a systems problem requiring joint control over grounding, state, dynamics, and verification. Overall, validation-gated construction provides a practical basis for dynamic financial CUA evaluation, with remaining challenges in qualification yield, broader transfer, production coverage, source licensing, and verifier maintenance.
Benchmark-Generation Metric Definitions
Let denote a generation method, let be its preregistered number of target episode slots, and let contain the candidate assigned to each slot in one construction run. An unfilled, malformed, or non-materialized slot is represented by an invalid candidate and receives zero on downstream hard gates. The definitions below specify one run; when the preregistered protocol contains multiple independent construction runs, the reported value is the mean of the run-level metric. Unless stated otherwise, metrics lie in , are macro-averaged across WorkflowCores, and are converted to percentages in Table 1.
Human Acceptance Rate (H-Acc). Financial-domain reviewers assess financial correctness, workflow realism, cross-component consistency, and alignment with the target capability. Let be reviewer ’s decision for episode , and let be a stratified, method-blind sample of target slots, with invalid slots treated as rejected. We define
Because H-Acc is estimated on a stratified sample rather than on every target slot, it is reported as a method-blind external audit and is not included in the strict episode-level qualification indicator.
LLM-as-Judge Acceptance Rate (L-Acc). A method-blind judge receives the workflow seed, target capability, generated instruction, source/state summary, WorkflowGraph, ToolGraph, runtime event, and goal/verifier synopsis. For judge and rubric dimension , let be the score, its preregistered weight, and the mean across judges. Let
We use
The minimum-dimension constraint prevents a strong average from hiding a critical failure.
Source & State Traceability Rate (Trace). Let be the required provenance atoms for episode , including source identifier, version or accession, hash, period, concept, unit, initial-state hash, event-state transition, goal predicate, and verifier reference. With denoting schema, referential, and hash validity, define
Execution & Replay Success Rate (Exec). Let and indicate successful environment boot and ToolGraph execution. Let , , and indicate reset run , reference-solver run , and event replay . Define
Static controls set the event-replay product to one.
Verifier Accuracy (V-Acc). Let be semantically incorrect mutations and be valid alternative solutions. For hidden verifier , define
A missing verifier or missing required test set contributes zero and sets . Thus, the reported V-Acc column is a continuous mean, while strict qualification uses the thresholded indicator .
Valid Dynamic Task Rate (Dyn). Let be the planned dynamic slots. For , indicates a post-start event; indicates deterministic replay; and are the valid plan and goal before and after the event; and and are event-ignoring and event-adaptive outcomes. We define
Target Capability Match Rate (CapMatch). Let indicate that episode targets capability , and let be the method-blind judge’s alignment decision. Define the capability-removal gap and shortcut success rate . For preregistered margins and ,
where each target slot has at least one preregistered capability and an empty or malformed assignment sets . We define
Duplicate Task Rate (Dup). For two episodes, we combine similarities over instruction text, WorkflowGraph, ToolGraph, goal/verifier structure, and source identity. Let and let be nonnegative weights summing to one:
Here, denotes the preregistered comparison set for episode among the valid outputs generated by method , excluding the episode itself. An episode is marked as duplicate if it has at least one comparison whose composite similarity reaches . Comparisons use structure-aware dimensions rather than surface text alone. The comparison order is fixed before evaluation; candidates within a WorkflowCore are compared through structure-aware dimensions rather than text alone.
Strict Qualification Indicator. Let
be the preregistered episode-level hard-gate set. The corresponding binary indicators are , , , , , , and , as defined above. Let indicate satisfaction of every mandatory safety, privacy, permission, resource, materialization, and forbidden-effect constraint in . The strict episode-level qualification indicator is
A missing, malformed, or non-materialized component sets its corresponding gate to zero.
Capability Coverage Rate (CapCov). Let be the preregistered capability ontology, the required number of strictly qualified episodes for capability , its importance weight, and the strict qualification indicator defined in (25). Weighted coverage is
Qualified Task Rate (Qual). Using the strict episode-level qualification indicator defined in (25), we define
Qual is the primary end-to-end construction metric, while the remaining metrics diagnose individual aspects of benchmark quality.
References
- [1]Boisvert, L.; Thakkar, M.; Gasse, M.; Caccia, M.; Le Sellier De Chezelles, T.; Cappart, Q.; Chapados, N.; Lacoste, A.; and Drouin, A. 2024. WorkArena++: Towards Compositional Planning and Reasoning-based Common Knowledge Work Tasks. In Advances in Neural Information Processing Systems, volume 37.
- [2]Butt, N.; Chandrasekaran, V.; Joshi, N.; Nushi, B.; and Balachandran, V. 2024. BENCHAGENTS: Automated Benchmark Creation with Agent Interaction. arXiv preprint arXiv:2410.22584.
- [3]Cao, R.; Lei, F.; Wu, H.; Chen, J.; Fu, Y.; Gao, H.; Xiong, X.; Zhang, H.; Mao, Y.; Hu, W.; Xie, T.; Xu, H.; Zhang, D.; Wang, S.; Sun, R.; Yin, P.; Xiong, C.; Ni, A.; Liu, Q.; Zhong, V.; Chen, L.; Yu, K.; and Yu, T. 2024. Spider2-V: How Far Are Multimodal Agents From Automating Data Science and Engineering Workflows? In Advances in Neural Information Processing Systems, volume 37.arxiv.org/abs/2407.10956
- [4]Chen, Y.; Yuan, Y.; Zhang, Z.; Zheng, Y.; Liu, J.; Ni, F.; Hao, J.; Mao, H.; and Zhang, F. 2025. SheetAgent: Towards a Generalist Agent for Spreadsheet Reasoning and Manipulation via Large Language Models. In Proceedings of the ACM Web Conference 2025, 158–177.
- [7]Dong, H.; Zhang, P.; Gao, Y.; Dong, X.; Cheng, Y.; Lu, M.; Yakefu, A.; and Zheng, S. 2026. Finch: Benchmarking Finance & Accounting across Spreadsheet-Centric Enterprise Workflows. In Findings of the Association for Computational Linguistics: ACL 2026, 10768–10794.
- [8]Drouin, A.; Gasse, M.; Caccia, M.; Laradji, I. H.; Del Verme, M.; Marty, T.; Boisvert, L.; Thakkar, M.; Cappart, Q.; Vazquez, D.; Chapados, N.; and Lacoste, A. 2024. WorkArena: How Capable Are Web Agents at Solving Common Knowledge Work Tasks? arXiv:2403.07718.arxiv.org/abs/2403.07718
- [9]Elbadry, R.; Ahmad, S.; Heakl, A.; Bouch, D.; Ahsan, M.; AlMahri, M.; Khalil, M. E.; Wang, Y.; Lahlou, S.; Ananidou, S.; Stoyanov, V.; Huang, J.; Peng, X.; Nakov, P.; and Xie, Z. 2026. SAHM: A Benchmark for Arabic Financial and Shari’ah-Compliant Reasoning. In Proceedings of the 64th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), 34509–34536.arxiv.org/abs/2604.19098
- [11]Ji, F.; Li, Y.; Yang, J.; Wu, J.; Zhang, F.; Cui, Z.; Xie, Y.; Peng, M.; Xie, Q.; Chen, X.; et al. 2026. Style Wins, Substance Loses: A Diagnosis of LLM-as-Judge in Idea Generation. arXiv preprint arXiv:2608.01666.arxiv.org/abs/2608.01666
- [12]Kiela, D.; Bartolo, M.; Nie, Y.; Kaushik, D.; Geiger, A.; Wu, Z.; Vidgen, B.; Prasad, G.; Singh, A.; Ringshia, P.; Ma, Z.; Thrush, T.; Riedel, S.; Waseem, Z.; Stenetorp, P.; Jia, R.; Bansal, M.; Potts, C.; and Williams, A. 2021. Dynabench: Rethinking Benchmarking in NLP. In Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, 4110–4124.arxiv.org/abs/2104.14337
- [13]Kundurthy, S.; Na, C.; Moraine, C.; Mohta, A.; Winter, C.; Fang, G.; Ling, J.; Strubell, E.; and Kirshner, Z. 2026. BlueFin: Benchmarking LLM Agents on Financial Spreadsheets. arXiv:2605.30907.arxiv.org/abs/2605.30907
- [14]Lau, E.; Dücker, M.; Chaudhary, R.; Goh, H. W.; Wei, R.; Kumar, V.; Qunbar, S.; Gogia, G.; Liu, Y.; Millslagle, S.; Borrazjanizadeh, N.; Tkachenko, U.; Danquah, S. E.; Schweicker, C.; Karumathil, V.; Devalaraju, A.; Sandadi, V.; Nam, H.; Arani, P.; Epps, R.; Arif, A.; Bhaiwala, S.; Northcutt, C.; Wang, S.; Athalye, A.; Mueller, J.; and Guzmán, F. 2026. BankerToolBench: Evaluating AI Agents in End-to-End Investment Banking Workflows. arXiv:2604.11304.arxiv.org/abs/2604.11304
- [15]Li, H.; Su, J.; Chen, Y.; Li, Q.; and Zhang, Z.-X. 2023. Sheet-Copilot: Bringing Software Productivity to the Next Level through Large Language Models. In Advances in Neural Information Processing Systems, volume 36, 4952–4984.
- [16]Li, X. L.; Kaiyom, F.; Liu, E. Z.; Mai, Y.; Liang, P.; and Hashimoto, T. 2025. AutoBencher: Towards Declarative Benchmark Construction. In International Conference on Learning Representations.arxiv.org/abs/2407.08351
- [17]Liu, S.; Zhao, S.; Jia, C.; Zhuang, X.; Long, Z.; Zhou, J.; Zhou, A.; Lan, M.; and Chong, Y. 2025. FinlDABench: Benchmarking Financial Data Analysis Ability of Large Language Models. In Proceedings of the 31st International Conference on Computational Linguistics, 710–725.
- [18]Lu, J.; Holleis, T.; Zhang, Y.; Aumayer, B.; Nan, F.; Bai, H.; Ma, S.; Ma, S.; Li, M.; Yin, G.; Wang, Z.; and Pang, R. 2025. ToolSandbox: A Stateful, Conversational, Interactive Evaluation Benchmark for LLM Tool Use Capabilities. In Findings of the Association for Computational Linguistics: NAACL 2025, 1160–1183.
- [19]Ma, Z.; Zhang, B.; Zhang, J.; Yu, J.; Zhang, X.; Zhang, X.; Luo, S.; Wang, X.; and Tang, J. 2024. SpreadsheetBench: Towards Challenging Real World Spreadsheet Manipulation. arXiv:2406.14991.arxiv.org/abs/2406.14991
- [20]Prabhakar, A.; Liu, Z.; Zhu, M.; Zhang, J.; Awalgoankar, T. M.; Wang, S.; Liu, Z.; Chen, H.; Hoang, T.; Niebles, J. C.; Heinecke, S.; Yao, W.; Wang, H.; Savarese, S.; and Xiong, C. 2025. Agentic-PM: Agentic Pipeline for Multi-Turn Data Generation via Simulated Agent-Human Interplay. In Advances in Neural Information Processing Systems, volume 38.

