betaFeedback

WorksComputer Science

FinCUABuild: Can Agents Build Reliable Benchmarks for Dynamic Financial Computer Use?

Abstract

Financial scenarios are diverse and complex, spanning varying data conditions, tool configurations, and workflows. Yet existing CUA, Computer-Using Agent, evaluation tasks remain largely manually constructed, limiting scalable coverage of real-world financial scenarios. Then, can agents autonomously construct diverse CUA evaluation tasks for financial scenarios? Evaluating this capability poses three key challenges: scenario coverage of construction requests, fair comparison across construction methods, and reliable assessment of generated task quality. To solve these, we introduce FinCUABuildBench, a benchmark for evaluating financial CUA task construction, featuring: (i) 576 construction requests covering 24 financial workflows and three types of runtime variation; (ii) standardized input, budget, and output specifications; and (iii) a task qualification mechanism based on execution tests and quality checks. We further introduce FinCUABuildAgent, a multi-agent system for automatically constructing dynamic financial CUA evaluation tasks. It consists of three modules that jointly construct tasks, environments, and validators. On FinCUABuildBench, under the same model backbone, existing agent-based construction methods achieve strict qualification rates of only 1.3-8.3%, while FinCUABuildAgent reaches 31.3%. Downstream evaluations further show that the constructed tasks can effectively differentiate CUA task-execution capabilities. These results demonstrate that agents can autonomously construct financial CUA tasks with meaningful evaluation value, offering a practical path toward broader evaluation coverage in financial scenarios. Code: https://github.com/FengxianJi/FinCUABuild

Introduction

Evaluation of computer-use agents (CUAs) has expanded to real-world web environments, desktop interactions, and enterprise workflows ([?]; [8]; [?]; [11]). Existing interactive benchmarks further provide controllable application states, dynamically instantiated tasks, and policy-constrained tool interactions, laying the foundation for evaluating complex workflows ([?]; [?]; [?]; [?]; [9]). In finance, workflows involve cross-file retrieval, spreadsheet operations, computation, and reporting, and must cover diverse data conditions, tool combinations, and business processes. SpreadsheetBench and Finch respectively explore real-world spreadsheet manipulation and enterprise-level financial workflow evaluation. Notably, even with LLM-assisted task mining, Finch still required over 700 hours of expert annotation ([19]; [7]). Such construction costs hinder the scalable expansion of evaluation tasks and motivate our central question: Can agents autonomously construct CUA evaluation tasks that cover diverse financial scenarios?

Existing work still has two key limitations in constructing dynamic financial CUA tasks. First, there is no unified benchmark for evaluating task construction capabilities, as existing financial benchmarks mainly assess agents on completing predefined tasks ([?]; [?]; [?]; [?]). Second, there is no construction agent tailored to this setting. Although prior automated construction methods have explored goal-driven generation, multi-agent collaboration, and task synthesis, they provide limited support for jointly constructing financial evidence, application states, dynamic events, and verification rules ([?]; [?]; [?]). Therefore, we study both how to evaluate financial CUA task construction capabilities and how to improve them.

Addressing these two issues requires tackling challenges in both construction-capability evaluation and automated task construction. For evaluation, (i) construction requests should cover diverse financial workflows and runtime variations, rather than increasing diversity through superficial paraphrasing; (ii) differences in input information, tool access, and resource budgets can confound method comparisons, requiring standardized evaluation conditions; and (iii) a plausible task description does not guarantee an executable environment, valid runtime events, or correct verification, so the quality of the complete task package must be assessed. For construction, (i) abstract capability targets must be translated into concrete dependencies among evidence, states, and actions to form executable tasks; (ii) tasks and verifiers may jointly accept incorrect outcomes, requiring independent correctness auditing; and (iii) errors can propagate across dependencies among tasks, environments, and verifiers, requiring failure localization and targeted repair of affected artifacts.

To address these challenges, we introduce FinCUABuildBench, shown as Figure 1(a), a benchmark for evaluating the construction of dynamic financial CUA tasks. For scenario coverage, the benchmark includes 576 construction requests derived from 24 financial workflow categories, 192 workflow templates, and three types of runtime variation: state updates, tool failures, and policy changes. For fair comparison, it standardizes construction inputs, resource budgets, and output specifications so that different methods are evaluated under comparable conditions. For quality assessment, it combines financial evidence checking, reference execution, reset-and-replay testing, dynamic-event validation, and verifier testing to determine whether a generated task is qualified. We further introduce FinCUABuildAgent, shown as Figure 1(b), which uses capability-dependency compilation to translate target capabilities into task requirements, independent quality auditing to inspect tasks and verifiers, and targeted feedback-driven repair to resolve cross-stage errors, jointly constructing tasks, environments, and verifiers. Under the same model backbone, FinCUABuildAgent achieves a strict qualification rate of 31.3%, outperforming general-purpose construction agents at 1.3–8.3%. The 144 qualified tasks reveal differences in downstream CUAs’ dynamic adaptation capabilities, while the qualification rate on 90 held-out construction requests is 13.3%. These results support the evaluation value of automated task construction, while also indicating substantial room for improved generalization.

Figure 1. Overview of FinCUABuildAgent. The upper panel shows the construction of a FinCUABuildBench episode, while the lower panel groups eleven specialist agents and their representative tools by stage. Labels 01–11 denote agents A01A_{01}–A11A_{11} in the text; shortened agent names are used for compactness.

Our main contributions and findings are: (1) A benchmark for benchmark construction. We introduce FinCUABuildBench, a 576-slot construction benchmark spanning 24 workflows, 192 WorkflowCores, and three runtime-dynamic profiles. Its unified episode contract evaluates groundedness, replayability, verifier reliability, dynamicity, capability necessity, safety, and redundancy; (2) A capability-guided multi-agent framework. We propose FinCUABuildAgent, which models dependencies among states, evidence, operators, and tools. Specialized agents jointly ground financial sources, build environments and runtime events, construct verifiers, and iteratively test and repair auditable CUA episodes; (3) A controlled empirical evaluation. Under the same model backbone, FinCUABuildAgent achieves a 31.3% strict qualification rate, compared with 1.3–8.3% for general-purpose construction agents. The 144 qualified tasks distinguish downstream CUAs in dynamic adaptation, while a 13.3% rate on 90 held-out requests indicates cross-workflow transferability.

Financial benchmarks and workflow agents. Financial benchmarks first emphasized report-based question answering and numerical reasoning through FinQA, TAT-QA, ConvFinQA, FinanceBench, and MultiHiertt (Chen et al. 2022a; Zhu et al. 2021; Chen et al. 2022b; Islam et al. 2023; Zhao et al. 2022), then expanded to knowledge-intensive reasoning and financial data analysis in FinanceMATH, FinBen, and FinDABench (Zhao et al. 2024; [17]; Xie et al. 2024a). Spreadsheet-oriented work moves toward executable artifact manipulation: SheetCopilot and SheetAgent study spreadsheet control, while SpreadsheetBench and Finch cover realistic workbook and enterprise finance and accounting tasks ([15]; [4]; [19]; [7]). Recent financial-agent work further targets professional spreadsheets, financial research, banking workflows, MCP-based tool use, and execution-grounded safety ([13]; Wang2026; [14]; Zhu2026; Yang2026). FinCUABuildBench evaluates the construction process itself: whether a method can produce a replayable episode whose evidence, application state, tools, post-start event, reset logic, and verifier remain jointly consistent.

Computer-use agents and stateful tool use. General CUA benchmarks span realistic web and desktop interaction (Zhou2024; Xie2024b), compositional enterprise work and professional data workflows ([8]; [1]; [3]), and controllable application and mobile environments (Trivedi2024; Rawles2025). ToolSandbox and τ\tau-bench add stateful, multi-turn, or policy-constrained tool interaction ([18]; Yao2025). These studies evaluate agent execution in supplied task environments. FinCUABuildBench instead targets the synthesis of a complete package comprising the environment, task, event, and verifier.

Automated benchmark construction. Automated construction ranges from Dynabench’s human–model loop to graph-generated or combinatorial evaluations in DyVal and SKILL-MIX ([12]; Zhu2024; Yu2024). AutoBencher optimizes declared desiderata, BENCHAGENTS coordinates planning, generation, verification, and evaluation, and the Benchmark Agent preprint adds design, grounding, and allocation stages ([16]; [2]; Xiong2026). APIGen-MT generates verified multi-turn task blueprints, while EnvScaler and AgentSynth synthesize tool environments, terminal validators, or verified long-horizon tasks ([20]; Song2026; Xie2026b; Zhang2026). We instantiate AutoBencher, BENCHAGENTS, and Benchmark Agent as matched construction baselines. FinCUABuildAgent further couples financial grounding, cross-artifact state, deterministic post-start events, reset and replay semantics, and mutation-tested hidden verifiers within a single release contract.

FinCUABuildBench

Problem Formulation

We study the automatic construction of benchmarks for dynamic financial computer use. Let N⋆N^{\star} be the number of preregistered target slots, indexed by i∈{1,…,N⋆}i \in\{1,\ldots,N^{\star}\}. The construction request for slot ii is defined as

Ii=(wi,κi,S,A,U,Γ),(1)\mathcal{I}_i = (w_i,\kappa_i,\mathcal{S},\mathcal{A},\mathcal{U},\Gamma), \tag*{(1)}

where wiw_i specifies the financial workflow and κi\kappa_i specifies the target capability. The sets S\mathcal{S}, A\mathcal{A}, and U\mathcal{U} provide the admissible sources, application environments, and builder-side tools, respectively, while Γ\Gamma contains the safety, privacy, resource, and release constraints.

Let M\mathcal{M} denote the set of benchmark-construction methods. Given Ii\mathcal{I}_i, a method m∈Mm \in\mathcal{M} generates a candidate episode

xi=m(Ii)=(ui,Di,s0,i,Gi,Ti,Ei,Vi,Ri,Πi).(2)x_i = m(\mathcal{I}_i) = (u_i,D_i,s_{0,i},G_i,T_i,\mathcal{E}_i,V_i,R_i,\Pi_i). \tag*{(2)}

Here, uiu_i is the user-visible instruction; DiD_i is a versioned source bundle; s0,is_{0,i} is the resettable initial state; GiG_i contains the terminal goal predicates; TiT_i is the evaluated agent’s ToolGraph; Ei\mathcal{E}_i is a deterministic runtime-event program; ViV_i is a hidden verifier; RiR_i is a reset and replay manifest; and Πi\Pi_i specifies permissions and forbidden effects.

A candidate is qualified only if it satisfies every mandatory constraint in Γ\Gamma and passes every preregistered episode-level hard gate: L-Acc, Trace, Exec, verifier pass, Dyn, CapMatch, and nonduplication. The strict conjunction is formalized in (25). H-Acc and CapCov are reported as complementary sample-level and portfolio-level diagnostics and do not enter the episode-level release indicator.

qi=1[Γi∧LAcci∧Tracei∧Execi∧VPassi∧Dyni∧CapMatchi∧¬Dupi].(3)q_i = 1\left[\Gamma_i \land\mathrm{LAcc}_i \land\mathrm{Trace}_i \land\mathrm{Exec}_i \land\mathrm{VPass}_i \land\mathrm{Dyn}_i \land\mathrm{CapMatch}_i \land\neg\mathrm{Dup}_i\right]. \tag*{(3)}

FinCUABuildBench Construction

FinCUABuildBench is designed to evaluate whether a construction method can generate complete dynamic financial computer-use episodes. It covers 24 financial workflow categories spanning reporting and analysis, planning and accounting operations, markets, risk and treasury, and compliance and governance. For each category, we define eight semantically distinct WorkflowCores, each specified by its business objective, source family, artifact contract, and target capability path. Each WorkflowCore is instantiated under the three runtime-dynamic profiles summarized in Figure 2, defining 576 preregistered target slots in total.

Figure 2. The FinCUABuildBench task space. 24 financial workflow categories, each with eight semantically distinct WorkflowCores, instantiated under three runtime-dynamic profiles.

  • P1: State Update, covering post-start changes to financial data or application state, such as an amended filing, refreshed market value, changed invoice status, collaborator edit, or updated ledger record;

  • P2: Tool Failure, covering disruptions to tools or interfaces, such as a timeout, session expiration, delayed recalculation, unavailable primary tool, partial write, or changed interface state;

  • P3: Policy Change, covering changes to permissions, approvals, or compliance requirements, such as an approval rejection, modified threshold, permission restriction, compliance hold, or newly introduced instruction that conflicts with the applicable policy.

Each runtime event is injected after execution begins, is deterministically replayable, changes at least one action required for success, and leaves at least one feasible completion path. To prevent workflow-level leakage, the P1–P3 instances of the same WorkflowCore are assigned to the same split. Each episode follows the contract in (2). Public workflows use versioned, hashed sources, whereas enterprise workflows use privacy-safe synthetic fixtures with cross-file consistency checks. The evaluated agent observes only the task inputs, workspace, and permitted tools; the event controller, reference goal state, oracle, verifier, and scoring logic remain evaluator-private.

Candidate episodes undergo iterative generation, audit, and repair, and only those satisfying the strict release criterion enter the qualified pool. For downstream evaluation, we sample two qualified episodes from each of the 24×324 \times3 category–profile cells, yielding 144 episodes. All selected episodes pass independent execution, deterministic replay, verifier, event-validity, and safety audits, and the frozen reference solver succeeds on all of them. The subset is fixed before any downstream-agent results are observed.

Automation-Point Verification

FinCUABuildBench evaluates executions through automation points, which are programmatically verifiable semantic checkpoints in a financial workflow. For each episode ii, we organize the task objective, artifact contract, evidence requirements, and runtime event into an automation-point DAG GiAP=(Vi,Ei)\mathcal{G}^{\mathrm{AP}}_i = (\mathcal{V}_i, \mathcal{E}_i), where Vi\mathcal{V}_i contains the automation points and Ei\mathcal{E}_i encodes their prerequisite and event-order dependencies. A point may represent grounding a financial fact, updating an artifact, transferring information across tools, or adapting to a runtime event. Low-level interface actions are excluded.

For condition χ∈{static,dynamic}\chi\in\{\mathrm{static}, \mathrm{dynamic}\}, let Xχ\mathcal{X}_\chi be the evaluated episode indices and let Vi,χ⊆Vi\mathcal{V}_{i,\chi} \subseteq\mathcal{V}_i be the nonempty set of points applicable to episode ii. For run r∈{1,…,ni}r \in\{1,\ldots,n_i\} of agent aa, where ni≥1n_i \ge1 is the number of repeated runs, let ωi,χa,r\omega^{a,r}_{i,\chi} contain the observed application states, produced artifacts, and execution and evidence trace. For j∈Vi,χj \in\mathcal{V}_{i,\chi}, let ϕij,χ\phi_{ij,\chi} and dep⁡ij,χ\operatorname{dep}_{ij,\chi} be Boolean predicates that check the semantic outcome and its dependency constraints, respectively. The binary node-pass indicator is

zij,χa,r=1[ϕij,χ(ωi,χa,r)∧dep⁡ij,χ(ωi,χa,r)],(4)z^{a,r}_{ij,\chi} = \mathbf{1}\left[\phi_{ij,\chi}\left(\omega^{a,r}_{i,\chi}\right) \land\operatorname{dep}_{ij,\chi}\left(\omega^{a,r}_{i,\chi}\right)\right], \tag*{(4)}

where 1[P]\mathbf{1}[P] equals one when PP is true and zero otherwise. Because verification uses semantic outcomes, different valid tools and action orders may satisfy the same automation point.

Let βij,χ>0\beta_{ij,\chi} > 0 be preregistered point weights satisfying ∑j∈Vi,χβij,χ=1\sum_{j \in\mathcal{V}_{i,\chi}} \beta_{ij,\chi} = 1, and let νi,χa,r∈{0,1}\nu^{a,r}_{i,\chi} \in\{0,1\} equal one when a forbidden effect occurs. The run-level automation-point completion score and its aggregate are

Ci,χa,r=(1−νi,χa,r)∑j∈Vi,χβij,χzij,χa,r,(5)C^{a,r}_{i,\chi} = \left(1-\nu^{a,r}_{i,\chi}\right)\sum_{j \in\mathcal{V}_{i,\chi}} \beta_{ij,\chi}z^{a,r}_{ij,\chi}, \tag*{(5)}
APComp⁡χ(a)=1∣Xχ∣∑i∈Xχ1ni∑r=1niCi,χa,r.\operatorname{APComp}_{\chi}(a) = \frac{1}{\lvert\mathcal{X}_{\chi}\rvert}\sum_{i \in\mathcal{X}_{\chi}}\frac{1}{n_i}\sum_{r=1}^{n_i} C^{a,r}_{i,\chi}.

Table 2 reports APComp static and APComp dynamic. Each automation point is binary, while the run-level score is fractional. Static completion evaluates baseline deliverable points, whereas dynamic completion evaluates event-transition and post-event adaptation points. These condition-specific point sets are complementary rather than nested, so neither aggregate is constrained to be smaller than the other.

Diag⁡k(a)=1∣Xk∣∑i∈Xk1ni∑r=1ni1∣Cik∣∑c∈Cikpicka,r.(6)\operatorname{Diag}_{k}(a) = \frac{1}{\lvert\mathcal{X}_{k}\rvert}\sum_{i \in\mathcal{X}_{k}}\frac{1}{n_i}\sum_{r=1}^{n_i}\frac{1}{\lvert C_{ik}\rvert}\sum_{c \in C_{ik}}p^{a,r}_{ick}. \tag*{(6)}

where Xk\mathcal{X}_k contains only episodes to which dimension kk applies. Artifact, Evidence, Trace Integrity, Financial Grounding, Sheet Contract, and Cross-Artifact Consistency use their corresponding verifier checks. Replan Recovery and Fallback Recovery are restricted to runs in which replanning or an allowed fallback is required. Safety Violation is the mean forbidden-effect indicator νi,χa,r\nu_i,\chi_{a,r}.

FinCUABuildAgent

FinCUABuildAgent operationalizes dynamic financial benchmark construction as a capability-guided and validation-gated multi-agent pipeline. Given a preregistered target slot, it first refines the request into a grounded WorkflowCore and then compiles it into an executable episode. Figure 1 summarizes the construction stages and their realization through specialist agents and stage-specific tools.

FinCUABuildAgent Architecture

The Factory Controller coordinates eleven specialist agents through a generate–audit–repair workflow and routes failed candidates to the responsible stage. The Benchmark Scout identifies financially meaningful workflow candidates, which are then formalized by the Workflow Miner and Process Analyst in terms of objectives, inputs, applications, and deliverables. The Provenance and Grounding Agent verifies the entity, reporting period, unit, and financial sources. The Taxonomy and Allocation Agent assigns each grounded workflow to the benchmark taxonomy, balances portfolio coverage, and removes semantic duplicates.

Given a grounded WorkflowCore and its target capability, the Capability and Dependency Compiler specifies the goal conditions, evidence dependencies, intermediate states, and semantic operations required to instantiate the capability. The Workflow and Episode Designer integrates these elements into a user-facing task with artifact, evidence, safety, and runtime-event specifications. The required operations are then mapped by the Tool Architect and Synthesizer to an episode-level ToolGraph with valid fallback paths. The Environment and Dynamics Composer materializes the financial workspace, initial application state, reset configuration, and replayable runtime event to produce an executable episode.

The Independent Oracle and Verifier Agent derives the hidden goal state and verification conditions from grounded evidence before observing any candidate solution. The Reference Solver executes the task through the same interface available to evaluated CUA agents, confirming that the episode remains solvable after the runtime change. The Red-Team and Calibration Agent tests incorrect financial values, stale evidence, ignored events, unauthorized effects, and shortcut solutions. The Factory Controller releases only episodes that pass these checks and returns failed episodes for targeted revision. Separating ToolGraph specification from environment materialization and committing the oracle before reference execution reduce cross-stage leakage and preserve independence between episode construction and qualification.

Typed Agent–Tool Orchestration

FinCUABuildAgent connects its eleven specialist agents through a typed orchestration graph GO=(A,EO)\mathcal{G}^{O} = (\mathcal{A},\mathcal{E}^{O}), where A=A01,…,A11\mathcal{A} = A_{01},\ldots,A_{11}. Each agent AkA_k receives the validated artifacts hpa(k)h_{\mathrm{pa}(k)} produced by its parent agents, invokes a role-specific allowlist of tools Uk\mathcal{U}_k, and returns an artifact hkh_k together with its tool-call trace ℓk\ell_k. The handoff relation Rk\mathcal{R}_k checks the required artifact type, source lineage, episode version, and cross-stage dependencies. We activate the downstream stage according to

αk=1[(hpa(k),hk)∈Rk;∧;ℓk∈Trace⁡(Uk)].(7)\alpha_k = \mathbf{1}\left[(h_{\mathrm{pa}(k)},h_k) \in\mathcal{R}_k;\land;\ell_k \in\operatorname{Trace}(\mathcal{U}_k)\right]. \tag*{(7)}

Here, Trace⁡(Uk)\operatorname{Trace}(\mathcal{U}_k) contains traces in which every tool invocation is authorized and satisfies its declared input–output contract. Thus, αk=1\alpha_k = 1 only when both the artifact and its tool-use trace pass validation; otherwise, the failed contract is returned to the responsible agent for repair.

Agents A01A_{01}–A04A_{04} handle workflow discovery and grounding. They identify candidate workflows, retrieve relevant source entities, and apply licensing and deduplication constraints so that only consistent and non-redundant WorkflowCores enter the graph.

Agents A05A_{05}–A08A_{08} compile and materialize executable episodes. They assemble goal predicates, state representations, and capability dependencies, select compatible schemas and runtime-event families, and synthesize interface adapters when needed. Contract, permission, and runtime-conformance checks precede the instantiation of a reproducible environment with a fixed initial state.

Agents A09A_{09}–A11A_{11} perform qualification and assurance through separate oracle, reference-execution, and red-team pathways. The oracle inspects hidden goal states and simulates event effects under controlled resets, while the reference solver uses only episode-visible interfaces to produce executable traces. The red-team evaluator probes stale evidence, leakage paths, invalid shortcuts, and unauthorized side effects. Contract or handoff violations trigger rollback and recomputation of the affected downstream artifacts.

Experiments

Experimental Setup

Benchmarks. We evaluate benchmark construction on Fin-CUABuildBench, which contains 576 target slots across 24 financial workflow categories and three dynamic profiles: State Update, Tool Failure, and Policy Change. To assess the downstream diagnostic value of generated tasks, we evaluate CUA agents on a balanced subset of 144 qualified episodes, with three independent runs per agent. We further test construction transfer on 90 held-out requests covering three unseen workflow scenarios: FP&A Variance Investigation, AP Exception Handling, and Portfolio Risk & Compliance.

Baseline methods. We compare FinCUABuildAgent against rule-based templates, LLM prompting (with and without tools), tool-augmented prompting, and adapted benchmark-generation systems including AutoBencher [16], BENCHAGENTS [2], and Benchmark Agent [?], as well as a single-agent tool-using baseline. We evaluate multiple backbones including MiniMax-M3, GPT-5.5, Qwen3.5-397B-A17B, Kimi-K3, and MiMo-V2.5-Pro depending on the setting.

Downstream evaluation on the 144-task subset covers general-purpose agents, financial agents, and controlled baselines. We also report held-out transfer and ablations against a budget-matched collapsed-controller variant and component removals. The collapsed controller in Table 4 is an internal topology ablation and is distinct from the external single-agent baseline in Table 1.

FamilyGeneration methodBackboneH-Acc ↑L-Acc ↑Trace ↑Exec ↑V-Acc ↑Dyn ↑CapMatch ↑CapCov ↑Dup ↓Qual ↑
Rule-basedDeterministic templatesn/a0.0%11.1%74.7%68.4%69.2%29.6%0.0%0.0%33.3%0.0%
Direct LLM no toolsDirect promptingMiniMax-M31.6%24.3%42.4%46.0%54.2%36.5%23.6%5.2%28.1%11.3%
GPT-5.53.2%27.3%38.4%56.6%37.6%17.4%31.9%3.1%34.6%8.7%
Qwen3.5-397B-A17B2.1%26.5%42.7%57.6%38.9%43.7%45.4%8.3%45.8%6.8%
Kimi K36.0%32.3%53.1%52.5%45.2%20.4%46.9%4.2%45.2%12.2%
MiMo-V2.5-Pro2.4%25.0%39.6%55.6%37.5%52.5%35.7%5.0%41.8%12.3%
Direct LLM + finance toolsTool-augmented promptingMiniMax-M312.9%21.1%67.1%57.5%54.2%38.8%56.9%31.6%37.4%17.5%
Qwen3.5-397B-A17B13.5%25.1%65.2%52.3%38.9%43.7%51.3%35.8%40.8%12.8%
Kimi K314.1%25.0%62.1%56.2%43.6%37.3%49.7%30.0%38.1%10.9%
MiMo-V2.5-Pro12.1%35.9%63.1%57.6%38.9%43.6%45.8%28.3%46.5%10.8%
Agentic baselinesAdapted AutoBencherMiniMax-M312.0%10.4%11.1%11.1%10.9%10.9%10.4%37.5%64.1%8.3%
Adapted BENCHAGENTSMiniMax-M35.2%5.0%5.0%5.0%5.0%5.0%5.0%12.5%65.5%1.3%
Adapted Benchmark AgentMiniMax-M37.8%7.6%8.0%8.0%7.8%8.0%7.5%25.0%63.0%4.5%
Single Agent + finance toolsMiniMax-M36.8%7.3%7.3%7.3%7.3%7.3%7.3%25.0%64.3%6.9%
OursFinCUABuildAgent Multi-AgentMiniMax-M333.4%55.7%89.1%73.8%72.3%70.0%78.5%50.7%12.5%31.3%
Qwen3.5-397B-A17B30.4%53.1%90.3%73.1%64.7%90.3%67.9%58.3%12.2%41.9%
Kimi K335.2%50.8%85.4%65.9%71.7%85.5%66.1%58.3%8.9%40.3%
MiMo-V2.5-Pro28.6%49.7%89.3%81.8%78.1%72.6%70.5%44.8%11.9%34.9%

Table 1. Benchmark-generation quality across financial CUA construction methods. Agentic baselines use MiniMax-M3 under matched construction settings. Values are averaged over the preregistered construction runs and then macro-averaged across WorkflowCores.

VariantL-Acc ↑Trace ↑Exec ↑V-Acc ↑Dyn ↑CapCov ↑Dup ↓
FinCUABuildAgent Multi-Agent55.7%89.1%73.8%72.3%70.0%50.7%12.5%
Collapsed controller (budget matched)26.0%22.9%18.8%40.1%12.5%27.3%49.7%
w/o Grounding & Provenance37.5%79.0%10.4%88.5%38.4%0.0%25.0%
w/o Dependency Compiler0.0%40.0%86.6%0.0%0.0%0.0%100.0%
w/o Tool Architect0.0%3.2%0.0%1.7%0.6%0.0%100.0%
w/o Dynamics Composer1.0%10.2%1.6%1.0%5.8%1.6%96.7%
w/o Independent Oracle8.9%61.6%58.4%52.0%63.2%35.9%43.2%
w/o Red Team & Repair0.5%78.9%69.8%68.8%73.2%39.6%20.8%

Table 4. Component and topology ablations of FinCUABuildAgent with MiniMax-M3 on the same 576 target slots as the corresponding full-system row in Table 1. The collapsed-controller variant is budget matched; all rows follow the same construction-run aggregation and scoring protocol.

Evaluation protocol. All methods share the same target specifications, resource budgets, and evaluation interfaces. Tool access follows the declared baseline setting and is held fixed within each matched comparison; no method has access to gold ToolGraphs or reference trajectories. The primary metric is Qualified Task Rate (Qual), the fraction of target slots that pass the strict conjunction of every preregistered episode-level hard gate and mandatory constraint in Γ. H-Acc is a stratified human audit and is not included in this conjunction; V-Acc is reported as a mean verifier score, while strict qualification uses its thresholded episode-level form. We further report L-Acc, Trace, Exec, V-Acc, Dyn, CapMatch, CapCov, and Dup as diagnostic metrics. Construction metrics are macro-averaged over WorkflowCores. Human and LLM-based assessments are method-blind, while execution-derived metrics are computed automatically from frozen artifacts and traces. Full metric definitions are provided in the supplementary material.

Main Result

Table 1 shows that FinCUABuildAgent provides the strongest overall profile on the coupled construction metrics. Finance tools alone consistently improve provenance and capability coverage, but their effects on dynamicity, verifier reliability, and strict qualification remain uneven. Across the four backbones shared with tool-augmented prompting, FinCUABuildAgent raises Dyn by 29.0–48.2 percentage points, reduces Dup by 24.9–34.6 points, and improves strict Qual by 13.8–29.4 points. Qwen3.5 achieves the highest Qual at 41.9%. The adapted agentic baselines reach only 1.3–8.3% Qual under MiniMax-M3, indicating that generic benchmark-generation pipelines do not directly recover the coupled financial grounding, executable state, runtime event, and verification contracts required here.

Diagnostic Effectiveness of Generated Tasks

Table 2 evaluates whether the qualified tasks constructed by FinCUABuildAgent provide meaningful diagnostic signals for downstream CUAs. Since the frozen reference solver succeeds on every selected task while evaluated agents obtain non-saturated and differentiated scores, the generated tasks are both feasible and discriminative. Native general-agent frameworks achieve 0.622–0.660 Static Completion and near-perfect financial grounding but only 0.004–0.043 Dynamic Completion, showing that grounding and general tool use do not ensure adaptation to runtime events. The external structured configuration reaches the highest Dynamic Completion (0.676), confirming that stronger performance is attainable; its Dynamic score exceeds its Static score because the metrics use different, non-nested automation-point sets. Meanwhile, the matched SingleController combines near-perfect artifact, evidence, static-completion, and cross-artifact scores with only 0.258 Dynamic Completion. Together, these results show that the generated tasks expose complementary failure modes in financial grounding, task completion, and runtime adaptation.

CategoryEvaluated Agent / ConfigurationArtifact ↑Evidence ↑Trace Integrity ↑Static Completion ↑Dynamic Completion ↑Financial Grounding ↑Sheet Contract ↑Cross-Artifact Consistency ↑Replan Recovery ↑Fallback Recovery ↑Safety Violation ↓
Direct MiniMax protocolsText-only0.7090.0000.3840.3200.0810.0000.6100.0180.1320.1470.153
Direct MiniMax protocolsArtifact draft0.6870.0000.3840.3260.2830.1670.6100.0840.6310.4300.100
Direct MiniMax protocolsStructured evidence0.7190.4930.2570.5200.0690.9290.6080.2170.1510.1000.009
Direct MiniMax protocolsStructured + verifier repair0.7210.6310.3200.5320.0000.9300.6110.2800.0000.0000.023
Direct MiniMax protocolsExternal structured agent0.7220.6200.1490.5240.6760.9410.6280.1300.7710.7710.021
Native general-agent frameworksAutoGen GroupChat0.7010.1590.0330.6410.0431.0000.7390.1540.2770.2300.035
Native general-agent frameworksMagentic-One0.7020.2190.0450.6460.0040.9980.7490.2110.0270.0220.035
Native general-agent frameworksCrewAI0.7090.1010.0210.6600.0260.9940.8120.0900.1690.1410.021
Native general-agent frameworksLangGraph0.6900.0960.0220.6220.0410.9930.7310.0890.2620.2190.035
Native general-agent frameworksCAMEL0.6840.1160.0250.6230.0330.9970.7110.1100.2120.1740.028
Financial-agent candidatesFinRobot0.6840.3410.0050.6670.0001.0000.6950.0050.0000.0000.319
Financial-agent candidatesTradingAgents0.6920.3660.0060.6390.0001.0000.5880.0050.0000.0000.241
Financial-agent candidatesFinTeam0.6890.3970.0090.4790.2240.9820.6190.0060.4170.1320.403
Financial-agent candidatesFinGPT0.6890.4000.0040.5830.3641.0000.7030.0030.2500.1040.329
Financial-agent candidatesFinRL0.6900.3870.0080.6430.0151.0000.6740.0070.0250.0150.227
Matched agent controlsGenuine SingleController0.9980.9950.4510.9970.2580.9950.9980.9950.6130.4370.000
Matched agent controlsReAct0.6060.0350.0130.3750.2280.4960.5710.0040.5560.4110.005

Table 2. Diagnostic evaluation on 144 qualified tasks sampled from FinCUABuildBench. Each agent or configuration runs all 144 tasks with three independent replicates, yielding 432 trajectories. Static and Dynamic Completion are mean condition-specific automation-point completion scores computed from the same trajectories over complementary, non-nested point sets; the remaining columns average their applicable verifier checks.

Generalization to Held-Out Workflows

Table 3 shows uneven transfer to held-out WorkflowCores within the covered workflow taxonomy. V-Acc remains at least 86.3% in every workflow–profile cell, whereas only 13.3% of episodes satisfy the strict qualification criterion. The gap is most pronounced under P3: 46.7% of the episodes are executable, but only 6.7% strictly qualify. This suggests that the independent oracle transfers reliably from frozen evidence and typed goal predicates, while coupling a policy change to a replayable environment and a capability-essential post-event solution remains difficult. FP&A achieves higher aggregate Exec (56.7%) and Qual (20.0%) than AP exception handling (43.3%, 10.0%) and portfolio risk and compliance (33.3%, 10.0%), although the latter attains higher CapMatch. Overall, the results support partial within-taxonomy WorkflowCore transfer, without implying generalization to entirely unseen workflow categories.

ScenarioProfileExec ↑V-Acc ↑Dyn ↑CapMatch ↑Qual ↑
FP&A Variance InvestigationP1: State Update60.0%95.8%40.0%40.0%30.0%
FP&A Variance InvestigationP2: Tool Failure50.0%90.0%30.0%20.0%20.0%
FP&A Variance InvestigationP3: Policy Change60.0%100.0%30.0%10.0%10.0%
AP Exception HandlingP1: State Update40.0%95.8%20.0%20.0%10.0%
AP Exception HandlingP2: Tool Failure50.0%100.0%50.0%30.0%20.0%
AP Exception HandlingP3: Policy Change40.0%88.7%20.0%30.0%0.0%
Portfolio Risk & ComplianceP1: State Update30.0%95.8%20.0%50.0%20.0%
Portfolio Risk & ComplianceP2: Tool Failure30.0%86.3%30.0%20.0%0.0%
Portfolio Risk & ComplianceP3: Policy Change40.0%96.1%30.0%40.0%10.0%

Table 3. Held-out workflow generalization under a frozen FinCUABuildAgent configuration. Each of three unseen financial scenarios contains ten held-out WorkflowCores instantiated once under P1–P3, yielding 90 episodes.

Component and Topology Ablations

Table 4 contrasts diagnostic failure patterns. Under the matched budget, staged coordination raises Exec from 18.8% for the collapsed controller to 73.8%, showing that typed handoffs reduce inconsistencies among goal predicates, Tool-Graphs, environment states, and verification logic. Removing Grounding & Provenance yields 88.5% V-Acc but only 10.4% Exec, showing that verifier accuracy alone cannot ensure a grounded, executable task portfolio. Removing the Dependency Compiler, Tool Architect, or Dynamics Composer yields Dup of at least 96.7% and CapCov of at most 1.6%, identifying these stages as the contract spine from semantic intent to executable dynamics. Removing the Independent Oracle reduces V-Acc from 72.3% to 52.0% and L-Acc from 55.7% to 8.9%, while raising Dup from 12.5% to 43.2%. Re-

moving red-team repair preserves several structural metrics but reduces L-Acc to 0.5% and raises Dup to 20.8%. These patterns show complementary stage contributions.

Conclusion

We introduced FinCUABuildBench, a 576-slot benchmark for constructing dynamic financial CUA tasks across 24 workflow categories and three runtime-dynamic profiles, together with FinCUABuildAgent, a capability-guided multi-agent framework for producing grounded, executable, replayable, and verifiable episodes. Against matched tool-augmented prompting across four backbones, FinCUABuildAgent improves Dyn by 29.0–48.2 points and strict Qual by 13.8–29.4 points while reducing Dup by 24.9–34.6 points; Qwen3.5 achieves the highest Qual of 41.9%. The generated qualified tasks yield non-saturated and differentiated automation-point completion scores across existing agents, demonstrating their downstream diagnostic value. Dynamic Completion is lower than Static Completion in 16 of 17 configurations, revealing limited adaptation to post-start changes. Held-out evaluation reaches 13.3% strict Qual and shows partial transfer within the covered taxonomy, while ablations support staged multi-agent coordination. Together, these results frame benchmark construction as a systems problem requiring joint control over grounding, state, dynamics, and verification. Overall, validation-gated construction provides a practical basis for dynamic financial CUA evaluation, with remaining challenges in qualification yield, broader transfer, production coverage, source licensing, and verifier maintenance.

Benchmark-Generation Metric Definitions

Let m∈Mm \in\mathcal{M} denote a generation method, let Nm∗N_m^{*} be its preregistered number of target episode slots, and let Bm={xi}i=1Nm⋆B_m = \{x_i\}_{i=1}^{N_m^\star} contain the candidate assigned to each slot in one construction run. An unfilled, malformed, or non-materialized slot is represented by an invalid candidate and receives zero on downstream hard gates. The definitions below specify one run; when the preregistered protocol contains multiple independent construction runs, the reported value is the mean of the run-level metric. Unless stated otherwise, metrics lie in [0,1][0,1], are macro-averaged across WorkflowCores, and are converted to percentages in Table 1.

Human Acceptance Rate (H-Acc). Financial-domain reviewers assess financial correctness, workflow realism, cross-component consistency, and alignment with the target capability. Let hir∈{0,1}h_{ir} \in\{0,1\} be reviewer rr’s decision for episode ii, and let Hm\mathcal{H}_m be a stratified, method-blind sample of target slots, with invalid slots treated as rejected. We define

HAcc⁡(m)=1∣Hm∣∑i∈Hm1[1∣Ri∣∑r∈Rihir≥τH].(8)\operatorname{HAcc}(m) = \frac{1}{|\mathcal{H}_m|} \sum_{i \in\mathcal{H}_m} \mathbf{1}\left[\frac{1}{|\mathcal{R}_i|}\sum_{r \in\mathcal{R}_i}h_{ir} \ge\tau_{\mathrm{H}}\right]. \tag*{(8)}

Because H-Acc is estimated on a stratified sample rather than on every target slot, it is reported as a method-blind external audit and is not included in the strict episode-level qualification indicator.

LLM-as-Judge Acceptance Rate (L-Acc). A method-blind judge receives the workflow seed, target capability, generated instruction, source/state summary, WorkflowGraph, ToolGraph, runtime event, and goal/verifier synopsis. For judge jj and rubric dimension dd, let sijd∈[1,5]s_{ijd} \in[1,5] be the score, wdw_d its preregistered weight, and sˉid\bar{s}_{id} the mean across judges. Let

ℓi=1[min⁡d∈Dsˉid≥τmin⁡∧∑d∈Dwdsˉid≥τL].\ell_i = \mathbf{1}\left[\min_{d \in\mathcal{D}}\bar{s}_{id} \ge\tau_{\min} \land\sum_{d \in\mathcal{D}}w_d\bar{s}_{id} \ge\tau_{\mathrm{L}}\right].

We use

LAcc⁡(m)=1Nm⋆∑i=1Nm⋆ℓi.(9)\operatorname{LAcc}(m) = \frac{1}{N_m^\star}\sum_{i=1}^{N_m^\star}\ell_i. \tag*{(9)}

The minimum-dimension constraint prevents a strong average from hiding a critical failure.

Source & State Traceability Rate (Trace). Let Pi\mathcal{P}_i be the required provenance atoms for episode ii, including source identifier, version or accession, hash, period, concept, unit, initial-state hash, event-state transition, goal predicate, and verifier reference. With valid⁡i(p)\operatorname{valid}_i(p) denoting schema, referential, and hash validity, define

ti=∏p∈Pi1[valid⁡i(p)=1].t_i = \prod_{p \in\mathcal{P}_i}\mathbf{1}[\operatorname{valid}_i(p)=1].
Trace⁡(m)=1Nm⋆∑i=1Nm⋆ti.(10)\operatorname{Trace}(m) = \frac{1}{N_m^\star}\sum_{i=1}^{N_m^\star}t_i. \tag*{(10)}

Execution & Replay Success Rate (Exec). Let bib_i and gig_i indicate successful environment boot and ToolGraph execution. Let riℓr_{i\ell}, oiqo_{iq}, and ηik\eta_{ik} indicate reset run ℓ\ell, reference-solver run qq, and event replay kk. Define

ρirep=(∏ℓ=1Lriℓ)(∏q=1Qoiq)(∏k=1Kiηik),(11)\rho_i^{\mathrm{rep}} = \left(\prod_{\ell=1}^{L}r_{i\ell}\right)\left(\prod_{q=1}^{Q}o_{iq}\right)\left(\prod_{k=1}^{K_i}\eta_{ik}\right), \tag*{(11)}
ei=bigiρirep,(12)e_i = b_i g_i \rho_i^{\mathrm{rep}}, \tag*{(12)}
Exec⁡(m)=1Nm⋆∑i=1Nm⋆ei.(13)\operatorname{Exec}(m) = \frac{1}{N_m^\star}\sum_{i=1}^{N_m^\star}e_i. \tag*{(13)}

Static controls set the event-replay product to one.

Verifier Accuracy (V-Acc). Let Zi−\mathcal{Z}_i^{-} be semantically incorrect mutations and Zi+\mathcal{Z}_i^{+} be valid alternative solutions. For hidden verifier Vi(z)∈{0,1}V_i(z) \in\{0,1\}, define

VMR⁡i=1∣Zi−∣∑z∈Zi−1[Vi(z)=0],(14)\operatorname{VMR}_i = \frac{1}{|\mathcal{Z}_i^{-}|}\sum_{z \in\mathcal{Z}_i^{-}}\mathbf{1}[V_i(z)=0], \tag*{(14)}
VAA⁡i=1∣Zi+∣∑z∈Zi+1[Vi(z)=1],(15)\operatorname{VAA}_i = \frac{1}{|\mathcal{Z}_i^{+}|}\sum_{z \in\mathcal{Z}_i^{+}}\mathbf{1}[V_i(z)=1], \tag*{(15)}
vi=1[VMR⁡i+VAA⁡i2≥τV],(16)v_i = \mathbf{1}\left[\frac{\operatorname{VMR}_i+\operatorname{VAA}_i}{2}\ge\tau_{\mathrm{V}}\right], \tag*{(16)}
VAcc⁡(m)=1Nm⋆∑i=1Nm⋆VMR⁡i+VAA⁡i2.(17)\operatorname{VAcc}(m) = \frac{1}{N_m^\star}\sum_{i=1}^{N_m^\star}\frac{\operatorname{VMR}_i+\operatorname{VAA}_i}{2}. \tag*{(17)}

A missing verifier or missing required test set contributes zero and sets vi=0v_i=0. Thus, the reported V-Acc column is a continuous mean, while strict qualification uses the thresholded indicator viv_i.

Valid Dynamic Task Rate (Dyn). Let Dm\mathcal{D}_m be the planned dynamic slots. For i∈Dmi \in\mathcal{D}_m, tievt>ti0t_i^{\mathrm{evt}}>t_i^0 indicates a post-start event; rievtr_i^{\mathrm{evt}} indicates deterministic replay; (πi−,Gi−)(\pi_i^{-},G_i^{-}) and (πi+,Gi+)(\pi_i^{+},G_i^{+}) are the valid plan and goal before and after the event; and yiignorey_i^{\mathrm{ignore}} and yiadapty_i^{\mathrm{adapt}} are event-ignoring and event-adaptive outcomes. We define

cievt=1[tievt>ti0∧(πi−,Gi−)≠(πi+,Gi+)],(18)c_i^{\mathrm{evt}} = \mathbf{1}\left[t_i^{\mathrm{evt}}>t_i^0 \land(\pi_i^{-},G_i^{-})\ne(\pi_i^{+},G_i^{+})\right], \tag*{(18)}
didyn=cievtrievt1[Vi(yiignore)=0∧Vi(yiadapt)=1],(19)d_i^{\mathrm{dyn}} = c_i^{\mathrm{evt}}r_i^{\mathrm{evt}}\mathbf{1}\left[V_i(y_i^{\mathrm{ignore}})=0 \land V_i(y_i^{\mathrm{adapt}})=1\right], \tag*{(19)}
Dyn⁡(m)=1∣Dm∣∑i∈Dmdidyn.(20)\operatorname{Dyn}(m) = \frac{1}{|\mathcal{D}_m|}\sum_{i \in\mathcal{D}_m}d_i^{\mathrm{dyn}}. \tag*{(20)}

Target Capability Match Rate (CapMatch). Let aic∈{0,1}a_{ic}\in\{0,1\} indicate that episode ii targets capability cc, and let jicj_{ic} be the method-blind judge’s alignment decision. Define the capability-removal gap Δic=p^ifull−p^i−c\Delta_{ic}=\hat{p}_i^{\mathrm{full}}-\hat{p}_i^{-c} and shortcut success rate si=p^ishortcuts_i=\hat{p}_i^{\mathrm{shortcut}}. For preregistered margins δc\delta_c and ϵ\epsilon,

ci=∏c:aic=1jic1[Δic≥δc∧si≤ϵ],c_i = \prod_{c:a_{ic}=1}j_{ic}\mathbf{1}[\Delta_{ic}\ge\delta_c \land s_i\le\epsilon],

where each target slot has at least one preregistered capability and an empty or malformed assignment sets ci=0c_i=0. We define

CapMatch⁡(m)=1Nm⋆∑i=1Nm⋆ci.(21)\operatorname{CapMatch}(m) = \frac{1}{N_m^\star}\sum_{i=1}^{N_m^\star}c_i. \tag*{(21)}

Duplicate Task Rate (Dup). For two episodes, we combine similarities over instruction text, WorkflowGraph, ToolGraph, goal/verifier structure, and source identity. Let K={text,wf,tool,goal,source}\mathcal{K} = \{\text{text}, \text{wf}, \text{tool}, \text{goal}, \text{source}\} and let λk\lambda_k be nonnegative weights summing to one:

Sij=∑k∈KλkSij(k),∑k∈Kλk=1,(22)S_{ij} = \sum_{k \in\mathcal{K}} \lambda_k S_{ij}^{(k)}, \qquad\sum_{k \in\mathcal{K}} \lambda_k = 1, \tag*{(22)}
zi=1[max⁡j∈Nm(i)∖{i}Sij≥τdup],(23)z_i = \mathbf{1}\left[\max_{j \in\mathcal{N}_m(i)\setminus\{i\}} S_{ij} \geq\tau_{\mathrm{dup}}\right], \tag*{(23)}
Dup⁡(m)=1Nm⋆∑i=1Nm⋆zi.(24)\operatorname{Dup}(m) = \frac{1}{N_m^\star}\sum_{i=1}^{N_m^\star} z_i. \tag*{(24)}

Here, Nm(i)\mathcal{N}_m(i) denotes the preregistered comparison set for episode ii among the Nm⋆N_m^\star valid outputs generated by method mm, excluding the episode itself. An episode is marked as duplicate if it has at least one comparison whose composite similarity reaches τdup\tau_{\mathrm{dup}}. Comparisons use structure-aware dimensions rather than surface text alone. The comparison order is fixed before evaluation; candidates within a WorkflowCore are compared through structure-aware dimensions rather than text alone.

Strict Qualification Indicator. Let

Ghard={LAcc,Trace,Exec,VPass,Dyn,CapMatch,Unique}\mathcal{G}_{\mathrm{hard}} = \{\mathrm{LAcc}, \mathrm{Trace}, \mathrm{Exec}, \mathrm{VPass}, \mathrm{Dyn}, \mathrm{CapMatch}, \mathrm{Unique}\}

be the preregistered episode-level hard-gate set. The corresponding binary indicators are ℓi\ell_i, tit_i, eie_i, viv_i, didynd_i^{\mathrm{dyn}}, cic_i, and 1−zi1-z_i, as defined above. Let γi∈{0,1}\gamma_i \in\{0,1\} indicate satisfaction of every mandatory safety, privacy, permission, resource, materialization, and forbidden-effect constraint in Γ\Gamma. The strict episode-level qualification indicator is

qi=γiℓitieivididynci(1−zi).(25)q_i = \gamma_i \ell_i t_i e_i v_i d_i^{\mathrm{dyn}} c_i (1-z_i). \tag*{(25)}

A missing, malformed, or non-materialized component sets its corresponding gate to zero.

Capability Coverage Rate (CapCov). Let C\mathcal{C} be the preregistered capability ontology, ρc\rho_c the required number of strictly qualified episodes for capability cc, πc>0\pi_c > 0 its importance weight, and qiq_i the strict qualification indicator defined in (25). Weighted coverage is

CapCov⁡(m)=∑c∈Cπc1[∑iqiaic≥ρc]∑c∈Cπc.(26)\operatorname{CapCov}(m) = \frac{\sum_{c \in\mathcal{C}} \pi_c \mathbf{1}\left[\sum_i q_i a_{ic} \geq\rho_c\right]}{\sum_{c \in\mathcal{C}} \pi_c}. \tag*{(26)}

Qualified Task Rate (Qual). Using the strict episode-level qualification indicator defined in (25), we define

Qual⁡(m)=1Nm⋆∑i=1Nm⋆qi.(27)\operatorname{Qual}(m) = \frac{1}{N_m^\star}\sum_{i=1}^{N_m^\star} q_i. \tag*{(27)}

Qual is the primary end-to-end construction metric, while the remaining metrics diagnose individual aspects of benchmark quality.

References

  1. [1]Boisvert, L.; Thakkar, M.; Gasse, M.; Caccia, M.; Le Sellier De Chezelles, T.; Cappart, Q.; Chapados, N.; Lacoste, A.; and Drouin, A. 2024. WorkArena++: Towards Compositional Planning and Reasoning-based Common Knowledge Work Tasks. In Advances in Neural Information Processing Systems, volume 37.
  2. [2]Butt, N.; Chandrasekaran, V.; Joshi, N.; Nushi, B.; and Balachandran, V. 2024. BENCHAGENTS: Automated Benchmark Creation with Agent Interaction. arXiv preprint arXiv:2410.22584.
  3. [3]Cao, R.; Lei, F.; Wu, H.; Chen, J.; Fu, Y.; Gao, H.; Xiong, X.; Zhang, H.; Mao, Y.; Hu, W.; Xie, T.; Xu, H.; Zhang, D.; Wang, S.; Sun, R.; Yin, P.; Xiong, C.; Ni, A.; Liu, Q.; Zhong, V.; Chen, L.; Yu, K.; and Yu, T. 2024. Spider2-V: How Far Are Multimodal Agents From Automating Data Science and Engineering Workflows? In Advances in Neural Information Processing Systems, volume 37.arxiv.org/abs/2407.10956
  4. [4]Chen, Y.; Yuan, Y.; Zhang, Z.; Zheng, Y.; Liu, J.; Ni, F.; Hao, J.; Mao, H.; and Zhang, F. 2025. SheetAgent: Towards a Generalist Agent for Spreadsheet Reasoning and Manipulation via Large Language Models. In Proceedings of the ACM Web Conference 2025, 158–177.
  5. [7]Dong, H.; Zhang, P.; Gao, Y.; Dong, X.; Cheng, Y.; Lu, M.; Yakefu, A.; and Zheng, S. 2026. Finch: Benchmarking Finance & Accounting across Spreadsheet-Centric Enterprise Workflows. In Findings of the Association for Computational Linguistics: ACL 2026, 10768–10794.
  6. [8]Drouin, A.; Gasse, M.; Caccia, M.; Laradji, I. H.; Del Verme, M.; Marty, T.; Boisvert, L.; Thakkar, M.; Cappart, Q.; Vazquez, D.; Chapados, N.; and Lacoste, A. 2024. WorkArena: How Capable Are Web Agents at Solving Common Knowledge Work Tasks? arXiv:2403.07718.arxiv.org/abs/2403.07718
  7. [9]Elbadry, R.; Ahmad, S.; Heakl, A.; Bouch, D.; Ahsan, M.; AlMahri, M.; Khalil, M. E.; Wang, Y.; Lahlou, S.; Ananidou, S.; Stoyanov, V.; Huang, J.; Peng, X.; Nakov, P.; and Xie, Z. 2026. SAHM: A Benchmark for Arabic Financial and Shari’ah-Compliant Reasoning. In Proceedings of the 64th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), 34509–34536.arxiv.org/abs/2604.19098
  8. [11]Ji, F.; Li, Y.; Yang, J.; Wu, J.; Zhang, F.; Cui, Z.; Xie, Y.; Peng, M.; Xie, Q.; Chen, X.; et al. 2026. Style Wins, Substance Loses: A Diagnosis of LLM-as-Judge in Idea Generation. arXiv preprint arXiv:2608.01666.arxiv.org/abs/2608.01666
  9. [12]Kiela, D.; Bartolo, M.; Nie, Y.; Kaushik, D.; Geiger, A.; Wu, Z.; Vidgen, B.; Prasad, G.; Singh, A.; Ringshia, P.; Ma, Z.; Thrush, T.; Riedel, S.; Waseem, Z.; Stenetorp, P.; Jia, R.; Bansal, M.; Potts, C.; and Williams, A. 2021. Dynabench: Rethinking Benchmarking in NLP. In Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, 4110–4124.arxiv.org/abs/2104.14337
  10. [13]Kundurthy, S.; Na, C.; Moraine, C.; Mohta, A.; Winter, C.; Fang, G.; Ling, J.; Strubell, E.; and Kirshner, Z. 2026. BlueFin: Benchmarking LLM Agents on Financial Spreadsheets. arXiv:2605.30907.arxiv.org/abs/2605.30907
  11. [14]Lau, E.; Dücker, M.; Chaudhary, R.; Goh, H. W.; Wei, R.; Kumar, V.; Qunbar, S.; Gogia, G.; Liu, Y.; Millslagle, S.; Borrazjanizadeh, N.; Tkachenko, U.; Danquah, S. E.; Schweicker, C.; Karumathil, V.; Devalaraju, A.; Sandadi, V.; Nam, H.; Arani, P.; Epps, R.; Arif, A.; Bhaiwala, S.; Northcutt, C.; Wang, S.; Athalye, A.; Mueller, J.; and Guzmán, F. 2026. BankerToolBench: Evaluating AI Agents in End-to-End Investment Banking Workflows. arXiv:2604.11304.arxiv.org/abs/2604.11304
  12. [15]Li, H.; Su, J.; Chen, Y.; Li, Q.; and Zhang, Z.-X. 2023. Sheet-Copilot: Bringing Software Productivity to the Next Level through Large Language Models. In Advances in Neural Information Processing Systems, volume 36, 4952–4984.
  13. [16]Li, X. L.; Kaiyom, F.; Liu, E. Z.; Mai, Y.; Liang, P.; and Hashimoto, T. 2025. AutoBencher: Towards Declarative Benchmark Construction. In International Conference on Learning Representations.arxiv.org/abs/2407.08351
  14. [17]Liu, S.; Zhao, S.; Jia, C.; Zhuang, X.; Long, Z.; Zhou, J.; Zhou, A.; Lan, M.; and Chong, Y. 2025. FinlDABench: Benchmarking Financial Data Analysis Ability of Large Language Models. In Proceedings of the 31st International Conference on Computational Linguistics, 710–725.
  15. [18]Lu, J.; Holleis, T.; Zhang, Y.; Aumayer, B.; Nan, F.; Bai, H.; Ma, S.; Ma, S.; Li, M.; Yin, G.; Wang, Z.; and Pang, R. 2025. ToolSandbox: A Stateful, Conversational, Interactive Evaluation Benchmark for LLM Tool Use Capabilities. In Findings of the Association for Computational Linguistics: NAACL 2025, 1160–1183.
  16. [19]Ma, Z.; Zhang, B.; Zhang, J.; Yu, J.; Zhang, X.; Zhang, X.; Luo, S.; Wang, X.; and Tang, J. 2024. SpreadsheetBench: Towards Challenging Real World Spreadsheet Manipulation. arXiv:2406.14991.arxiv.org/abs/2406.14991
  17. [20]Prabhakar, A.; Liu, Z.; Zhu, M.; Zhang, J.; Awalgoankar, T. M.; Wang, S.; Liu, Z.; Chen, H.; Hoang, T.; Niebles, J. C.; Heinecke, S.; Yao, W.; Wang, H.; Savarese, S.; and Xiong, C. 2025. Agentic-PM: Agentic Pipeline for Multi-Turn Data Generation via Simulated Agent-Human Interplay. In Advances in Neural Information Processing Systems, volume 38.

Paper details

vlnc:W_d3ztvcnw
Version
v1 · latest
Published
Added
License
CC BY 4.0
Usage

views · 0

downloads · 0

A view counts once per browser per day, a download each time the PDF is requested.

  • Computer Science

Not recorded for this paper

View statement
Looking for related papers…
Contents