WorksComputer Science
A Taxonomy and Taxonomy-Guided Benchmark Reduction for Spreadsheet Tasks
Abstract
Spreadsheets let millions of end users without programming training do computational work, and teams are increasingly building AI agents to help them. Evaluating each candidate agent configuration across many tasks is costly. Benchmark reduction can lower this cost by selecting a small subset of tasks whose outcomes predict full-benchmark scores, but selecting tasks only for predictive accuracy can leave whole categories of spreadsheet work untested. We contribute a taxonomy of spreadsheet tasks, grounded in a review of benchmarks and refined with the Excel Agent product team at Microsoft, and a benchmark-reduction method that favors taxonomic diversity among similarly predictive choices. On benchmark logs from Microsoft, taxonomy-guided selection broadens coverage at little or no cost in predictive accuracy and often improves prediction on configurations excluded from selection. Broader coverage can therefore support, rather than compromise, predictive accuracy. Generalization remains limited across time: subsets selected using little recent history predict poorly for configurations much later in time. Beyond benchmark reduction, the taxonomy provides a reusable framework for auditing gaps in task coverage, as teams at Microsoft have done.
Introduction
Major technology companies and startups are building AI-powered spreadsheet agents: systems that let users express intent in natural language and autonomously carry out the corresponding spreadsheet operations. Academic research on such agents is growing [1, 2, 3], mirroring the broader trend toward coding agents in software engineering and data work [4, 5], in this case in an end-user programming environment. If effective, these agents can democratize complex data operations in a domain where errors are notoriously prevalent [6, 7].
Ensuring the quality of these agents is challenging. Given a goal such as “clean this sales export and build a month-over-month revenue summary,” an agent plans and carries out a sequence of operations—inserting formulas, creating pivot tables, writing and running scripts—over a live workbook. A task rarely has a single correct answer (“total revenue by region” can be a pivot table, a SUMIF formula, or hand-entered values), and correctness can hinge on workbook state the grid does not expose, such as formulas and cross-sheet references; with natural-language ambiguity and model nondeterminism, ground truth is hard to establish [8, 9]. Evaluation is also expensive. Running the agent on a task—a workbook state paired with a natural-language instruction—consumes tokens and time, and the cost multiplies because the evaluation target is not the LLM alone but a configuration: a combination of LLM, system prompts, tool interfaces, execution harness, and guardrails (validity or safety checks that can block or rewrite an action), any of which can affect outcomes, and all of which teams iterate on routinely. As a concrete illustration of this cost, after integrating a new frontier model into Microsoft’s Excel Agent, we estimated that a single full-benchmark run took about four hours and cost roughly $1,000 in tokens at retail prices. Exhaustive evaluation of every candidate configuration is impractical in a typical CI/CD setting [10, 11], which demands principled test minimization [12]: selecting a reduced benchmark, a small subset of tasks that preserves the predictive accuracy of the full benchmark—how well the subset’s outcomes recover the decision-relevant quantities the full benchmark would report (e.g., pass rates, configuration rankings).
A subset chosen on purely predictive grounds, however, may concentrate in a narrow region of the task space, dominated by a few similar task types while whole categories of behavior go untested. While a few ad-hoc task categorizations exist [1, 2, 13], such coverage has not been defined for spreadsheet AI agents. This gap is our starting point.
In this paper we develop a taxonomy of spreadsheet tasks covering properties of the spreadsheet and natural language instruction. We build the taxonomy following the iterative method of Nickerson et al. [14], grounding it in a literature review of spreadsheet benchmarks and refining it through product team feedback at Microsoft, where Excel Agent is being built and improved. The taxonomy provides an externally observable structure for describing what a benchmark covers, selecting reduced benchmarks, and interpreting failures. We do not claim a single “right” taxonomy—that would overfit current benchmarks and agents—but a reusable framework that can be adapted as the domain evolves [15].
We then use the taxonomy to guide minimization. Treating tasks as features and configurations as data points, we use greedy forward selection with regularized regression to identify a predictive subset; when candidates are near-tied on predictive accuracy, a taxonomy-aware tie-breaker selects the one that maximizes diversity. On real logs from Microsoft, when we evaluate the reduced benchmark on configurations not used to select it (honest validation), selecting for coverage costs little predictive accuracy and often improves it: diversity behaves less as a trade-off against accuracy than as a regularizer over greedy selection—giving up a little fit on the selection data for better prediction on configurations not yet seen—strongest when data is noisy and the task budget is small. The reduced benchmarks do not, however, transfer to configurations far later in time when trained on little recent history, which limits where taxonomy-guided minimization can be used. Matched-budget controls—uniform random, taxonomy-stratified random, diversity-only, and random choice within the same tolerance band—show that the coverage gain comes from the taxonomy rather than from relaxing the greedy objective alone, and that a predictive stage remains necessary. Repeating the analysis on the subset of the logs with no missing outcomes, where every configuration is scored on one common reference set, reproduces the pattern, and taxonomy guidance again beats matched random choice within the band under all three validation strategies, so the result is not an artifact of imputation.
Concretely, our contributions are:
A taxonomy of spreadsheet tasks for evaluating AI agents, synthesized from the literature and industrial practice (§ 3).
A taxonomy-guided benchmark reduction methodology that selects a predictive reduced benchmark and breaks near-ties by taxonomic diversity. On real industrial logs—the recorded outcomes of past evaluations, i.e., which tasks passed or failed under each configuration—this tie-breaker widens taxonomy coverage with little or no loss of predictive accuracy and acts as a regularizer, as matched-budget controls and a complete-case replication confirm (§ 3-C and § 4).
Experience applying the taxonomy at Microsoft to audit coverage gaps between benchmarks and production traffic, and articulating the method’s practical limits (§ 6).
Related Work
We review three lines of work: methods for reducing evaluation cost, from software-testing minimization to LLM benchmark subset selection; agent and spreadsheet benchmarks; and how researchers define coverage for black-box systems through diversity and taxonomies.
Efficient Test and Benchmark Reduction
When rerunning a full test suite on every change is too costly, software teams cut the cost by selecting, minimizing, or prioritizing tests—running a representative subset, dropping redundant tests, or running the most informative tests first [12, 16, 17]—a pressure that long-running CI pipelines intensify [18]. To decide which tests are worth keeping, classical methods measure coverage: how much of the program’s internals—its statements, or artificially injected faults—the tests exercise [19]. Such code-level coverage is unavailable when a “test” is a natural-language task run against a black-box system, so an externally observable artifact—such as the requirement a test relates to—can serve as the selection signal instead [20].
The same problem arises in ML-system testing [21, 22, 23] and LLM evaluation, where the adequacy criterion shifts from code coverage to preserving aggregate scores or rankings. A growing family of methods selects small static subsets (coresets) that approximate full-suite scores or preserve model rankings [24, 25, 26], though the best subset can depend on whether it is model-agnostic or model-aware [27, 28].
Other methods instead exploit learnable structure in modeltask performance: Ye et al. [29] search for a “small-bench” via LLM capability predictability; Zhao et al. [30] frame evaluation as matrix completion; Item Response Theory estimates remaining outcomes from partial observations by modeling item difficulty and discrimination [31, 24, 32]; and Active Evaluation Acquisition [32] and Decentralized Arena [33] choose what to evaluate next from observed results. The matrix-completion and active-evaluation methods spend an observation budget: they choose which missing configuration–task outcomes to acquire or estimate next. We instead fix a reusable task suite under a task budget, so the two settings are similar in spirit but not directly comparable; the adaptive methods also assume a missingness pattern that production logs do not have (§ 4-A). We do not compare against them and claim no superiority over them.
These subsets can be fragile: prompt sensitivity [34, 35] and distribution shift [36, 37] can alter which items are informative, motivating selection criteria that do not rely solely on historical performance.
Our problem is closest to static subset selection under a fixed budget: we choose which spreadsheet tasks to run, to preserve both decision-relevant aggregates (pass rate, configuration rankings) and structural breadth. Our pure predictive greedy selector (, § 4) belongs to this family and represents it in our experiments; we did not run the published instantiations because they fit on dense modeltask matrices and estimate one model’s score rather than rankings across configurations.
AI Agent and Spreadsheet Benchmarks
Agent benchmarks span code (SWE-bench [4], AgentBench [5]), web and desktop (WebArena [38], OSWorld [39]), multi-step reasoning (GAIA [40]), and API tool use (ToolLLM [41]), but none addresses selecting a representative subset when full evaluation is too costly, nor offers a coverage framework for minimization.
For spreadsheet agents specifically, SheetCopilot [1] defines six task categories, SheetAgent [2] introduces five broad types with 36 subtypes paired with reasoning challenges, SpreadsheetBench [3] collects 912 real forum questions, MiMoTable [13] categorizes tasks by meta-operation, and InstructExcel [42] provides crowdsourced natural-language instructions over real Excel files. Each defines its own ad hoc task categories; none amounts to a systematic taxonomy designed for coverage-oriented benchmark reduction. Black-Box Coverage via Diversity and Taxonomies: When source-level coverage is unavailable—as in our black-box setting, and in DNN testing, where neuron-coverage metrics correlate weakly with fault detection [43, 44, 45, 46, 47]—a common alternative is diversity: selecting dissimilar test cases to broaden exploration of the fault space [48, 17, 49, 50], which is competitive with white-box techniques when source-level information is inaccessible [51]. Diversity often correlates positively with fault detection [52, 44, 53], is grounded theoretically by the observation that most faults stem from interactions of few parameters [54], is operationalized by multi-objective and distance-aware methods in black-box DNN testing [55, 56], and has been combined with fault-proneness estimates to prioritize tests [57]; Elgendy et al. [58] map diversity-based testing.
Diversity, however, raises the question: with respect to what? A common answer is a taxonomy—a set of dimensions serving as an operational coverage scheme. Autonomous-vehicle testing derives scenario taxonomies via clustering or driving-dimension characterization (e.g., [59]); LLM testing explores model knowledge boundaries through knowledge graphs [60]; and web testing uses defect taxonomies to build smaller, more effective suites [61, 62].
Knowledge Gap and Problem Framing
Classical minimization defines test adequacy through structural coverage—statement or mutation coverage—which cannot be measured for an LLM-based agent. Coreset methods for LLM benchmarks select on historical performance correlations, with no notion of which behaviors a subset exercises, and turn fragile once those correlations break down under prompt sensitivity or drift over time. Diversity is the standard black-box substitute for coverage, but it presupposes dimensions along which to be diverse—and existing spreadsheet and agent benchmarks supply only ad-hoc task categories, not a systematic taxonomy built for coverage-oriented selection. Pairing a domain-grounded taxonomy with a predictive subset-selection procedure is our response—but it requires a specific framing of the selection problem.
Why a Reduced Benchmark: Production agent teams continuously compare candidate configurations across a large design space: cheaper or faster LLMs that may underperform slightly but win on cost or latency; fine-tuned variants whose quality is uncertain until measured; and harness-only changes that keep the model fixed—alternative spreadsheet representations, tool schemas, retrieval settings, safety policies, or new tool integrations. Each needs a full-benchmark score before it can be compared with the current default, yet a typical CI/CD cycle exercises thousands of tasks under rate limits. A reduced benchmark—a small, fixed subset whose outcomes approximate the full-benchmark score for any new configuration—trades a one-time curation cost for a recurring saving.
Why this works: The framing is well-posed when configurations vary along a few latent capabilities and tasks are redundant within them: a small set of informative tasks with learned weights can then recover the full-benchmark score (App. B gives a stylized example).
Tasks as Features, Configurations as Datapoints: This structure—a few latent capabilities with many redundant tasks—suggests the general framing. A uniform random sample of tasks would suffice only if every task correlated equally well with the full-benchmark score, but tasks are heterogeneous—they probe different operations and difficulty levels—so a small random sample can over-represent one region and predict poorly. We therefore choose tasks for how well they predict that score, casting the choice as supervised learning: tasks are features and configurations are datapoints. Each past configuration has been run against many tasks, giving a vector of pass/fail outcomes whose average—its full-benchmark score—is the label, and selecting a reduced benchmark is feature selection.
This requires a pool of past configurations with known full-benchmark scores, which may seem self-defeating if running the full benchmark is too costly. The resolution is temporal: teams do run the full benchmark periodically—to certify releases, set baselines, or vet major model upgrades—and these runs accumulate over a project’s lifetime. Whether the backlog is large and varied enough to support selection is an empirical question we examine using logs from Microsoft (Section V).
These logs are also the right kind of training data: they do not require new data collection and arise from realistic axes of variation—model swaps, prompt edits, retrieval changes. What we extract from them is a property of the tasks, not of any single configuration: which tasks are informative about aggregate behavior and which are redundant. A configuration that will never be deployed is therefore still useful training data, and the signal depends only on relative differences across configurations, easier to estimate than any absolute score. When extrapolation across model generations is a concern, the training pool can be restricted to a recent slice.
In the dominant coreset framing, one picks a subset to estimate one fixed model’s score; we instead pick a subset to compare and rank many candidate configurations.
From framing to method: This framing imposes two requirements. First, predictive accuracy alone is not enough: a reduced benchmark chosen purely to maximize —the fraction of variance in the full-benchmark score that a reduced benchmark’s predictions explain—can cluster in narrow regions of the task space, so we need a coverage criterion meaningful without source-level signals—a domain-grounded taxonomy answering “with respect to what” (Section IV). Second, we need a predictive selection algorithm that picks a small, weighted benchmark under that taxonomy, whose design decisions we test empirically on real logs (Section V).
The Taxonomy of Spreadsheet Tasks
This section presents a taxonomy of externally observable properties of the workbook and the task instruction—the dimensions over which we operationalize coverage. We describe first how we identified its dimensions (§ 3-A) and then the result (§ 3-B).
A workbook contains one or more spreadsheets (sheets, or “tabs”), each a two-dimensional grid of cells, and a spreadsheet contains one or more tables—contiguous regions holding a coherent unit of data (e.g., a header row with its records). We present the dimensions primarily at the table level, the finest granularity at which most are expressed and the natural scope of many tasks (lookup, aggregation, cleaning).1
Methodology for Building our Taxonomy
Building a taxonomy requires choosing both which dimensions of variation to include and, for each, how to measure it—the concrete measure or measures that assign a task a value along that dimension. These decisions are interdependent: the right dimensions emerge only from examining real tasks, yet examining tasks presupposes at least a tentative scheme. We resolve this circularity by following the iterative taxonomy-development method of Nickerson et al. [14], which alternates deductive (top-down) cycles—deriving candidate dimensions from theory and testing them against data—with inductive (bottom-up) cycles—examining data to find dimensions not yet in the scheme—until a set of ending conditions is met.
Nickerson et al. [14] give two kinds of ending condition. The objective ones hold when every object examined has been classified and a full iteration does not add, merge, or split any dimension or value; the subjective ones hold when the taxonomy is concise, robust, comprehensive, extendible, and explanatory. A single guiding criterion governs what the taxonomy includes, and every dimension must serve it.
Instantiation: We apply this method to AI spreadsheet agents. Our objects of interest are spreadsheet tasks, each defined by a workbook state and a natural-language instruction. Our guiding criterion is that every dimension capture an externally observable property of the workbook or instruction—one relevant for describing, selecting, and interpreting agent evaluations—since agent internals are inaccessible in our black-box setting. Both cycle types require concrete tasks, so we need a task corpus to iterate over.
This corpus is a convenience sample drawn from one popular public benchmark, SpreadsheetBench [3], and an internal benchmark under development at Microsoft. Inspecting every public benchmark (§ 1.1) was impractical; combining a public and an internal source lets the taxonomy reflect both the academic view of spreadsheet tasks and the workloads of a production agent.
Identifying dimensions: In deductive (conceptual-to-empirical) cycles we started from theory: drawing on the benchmarks surveyed in § 1.1, we proposed candidate dimensions, applied them to corpus tasks, and revised them when a dimension did not vary (an early file format candidate:
| Dimension | Description | Example Measures [Sources] |
| Properties of a Table | ||
| Table size | Table grid dimensions | Number of rows & columns [3, 63, 64, 65, 2] |
| Vocabulary size | Size of vocabulary | Number of unique values [66] |
| Header type | The structure of the header row/column | A categorical label, e.g., simple (single-row or single-column header) vs complex (e.g., hierarchical) [13, 3, 67] |
| Structure | The structure of the table | A categorical label, e.g., flat vs hierarchical [68, 67, 69, 3] |
| Data types | The types of data contained in the table | A list, e.g., string, numeric, date [66, 63, 69] |
| Data cleanliness | The cleanliness of the data in the table (e.g., missing entries, inconsistently formatted values) | A categorical label, e.g., clean vs noisy [70, 71] |
| Language | The primary language used in the table | A categorical label, e.g., English [66, 13] |
| Properties of a Spreadsheet or the Workbook | ||
| Size | The size of the parent (spreadsheet/workbook) (as a set of components (tables/sheets) or the file as a whole. | Number of sheets [65, 2, 3, 13, 72]; Number of tables [3, 13, 72]; File size on disk [42, 64] |
| Structural complexity | The overall structural complexity of the parent. | A derived rating that aggregates table-level structure and size signals, e.g., summarized by their principal components [13] |
| Domain / Topic | The domain or topic of the workbook | A categorical label, e.g., architecture, finance, education, accounting [73, 74, 66, 13, 67, 63, 75, 64, 65, 2]; NAICS industry sector [this work]; Workbook primary intent [this work] |
| Properties of the Task Instructions (and Task As a Whole) | ||
| Clarity | The clarity of the natural language instructions | A categorical label, e.g., clear vs ambiguous [65, 2, 69]; Instruction specificity rated on a 1–5 rubric [this work] |
| Operations | Specific operations that are part of the instructions, with or without their objects | List of query verb-noun phrases (actions + objects) [65, 2, 3]; List of met operations (e.g., lookup, edit, calculate, compare, visualize, reason) [42, 13, 65, 2] |
| Task type | The type of the task to be performed | A categorical label, e.g., table Q&A, table manipulation, data analysis [1, 76, 13, 71, 77, 67, 78, 75, 79, 64, 69, 65, 2]; Action type: ask vs. act vs. unknown [this work]; Multiple operationalizations (thematic analysis, capability-, and intent-based) [this work] |
| Task difficulty | Task horizon or complexity | Number of operations/reasoning steps required [65, 2]; A weighted average of the difficulties of the task’s meta operations (e.g., lookup and edit can be considered the easiest, while reasoning can be considered the hardest) [13, 65, 2] |
| LLM capabilities | The capabilities expected of the language model in handling the task | E.g., error detection, symbolic reasoning, domain understanding, numerical abstraction, explanation [80, 75, 81, 69, 65, 2] |
Table I. Our taxonomy dimensions and the literature informing them. Each dimension of variation of the workbook (or its components) or of the task instructions has one or more concrete measures.
nearly every task is an .xlsx workbook), was redundant (a per-table “structural complexity” rating that restated Header type, Structure, and Table size, kept only as an aggregate workbook property), or conflated separable axes (a single “type” dimension, split into Task type and the Operations it comprises) [61]. In inductive (empirical-to-conceptual) cycles we started from data, open-coding corpus tasks to find recurring properties not yet captured and grouping them into new or revised dimensions; data cleanliness emerged this way, after we repeatedly found tables with missing values that our theory-derived dimensions did not capture. Open coding also yielded candidate measures and value sets for each dimension. We iterated through both cycle types, with rounds of discussion between two researchers [61, 82], until the taxonomy stabilized and met our ending conditions.
Choosing concrete measures and stopping: For each dimension we then chose one or more concrete measures, using programmatic extraction for objective workbook structure and LLM-assisted coding for interpretive task or instruction properties. We allowed multiple measures when a dimension had several useful measures: for example, Task type can be grounded in instruction themes, required agent capabilities, or workbook intent, each with different tradeoffs. We stopped when every corpus task could be classified by every dimension and the taxonomy satisfied the subjective ending conditions above (concise, robust, comprehensive for our scope, extendible, and explanatory). App. C gives the full coding and termination details.
The Resulting Taxonomy
Table I summarizes the taxonomy: at the table level it captures size, vocabulary, headers, structure, data types, cleanliness, and language; at the spreadsheet/workbook level scale, structural complexity, and domain; and at the task level clarity, operations, task type, difficulty, and required LLM capabilities. Because the table lists only representative measures and values, App. A enumerates the complete value sets and App. D gives SpreadsheetBench examples for each tier.
Validity of the Taxonomy
Our main evaluation—whether the taxonomy enables effective benchmark reduction—is the subject of § 4. First we establish a precondition, descriptive validity: that the dimensions discriminate among tasks, partitioning a real benchmark into substantively different groups. We illustrate this on SpreadsheetBench [3], which we annotated end-to-end following the labeling procedure of § 3-A. Concretely, we compute objective measures such as workbook size from the workbook metadata, and use an LLM (GPT-4.1) for subjective measures such as task type and instruction specificity, prompted with the measure’s definition and value set as a rubric.
One author spot-checked the LLM labels by hand-coding a sample, concentrating on the most subjective dimensions, task type and workbook intent, and a second author reviewed the disagreements. The checks did not reveal any major concerns beyond the subjectivity one expects from these dimensions. Disagreements arose with sparse workbooks, whose few filled cells do not reveal the domain or purpose, and with instructions that combine operations, where more than one task type fits: for example, highlighting each row’s best value is both Comparison, Ranking & Extremes and Visualization, Formatting & Presentation, and the LLM tends to label anything that fits nowhere else as Formulas, Logic & Automation, which may inflate that category’s count in Figure 1(a). In these cases neither the human nor the LLM was clearly right, and a more detailed coding guide would be needed to resolve them. The distributions below show that the dimensions separate tasks into different groups (descriptive validity); they do not show that every label is right, and labels affect selection only through the tie-break (§ 6).
Figure 1. (a) Task type. (b) Steps implied (task horizon). (c) Instruction specificity. (d) Workbook intent (domain). SpreadsheetBench tasks across four taxonomy dimensions: a few categories dominate while many are sparsely populated, so the dimensions discriminate among tasks (descriptive validity) and benchmark coverage of the taxonomy is highly uneven.
Figure 1 shows four of the most interpretable dimensions. Task type (a) is heavy-tailed: Formulas, Logic & Automation alone accounts for over 400 tasks (a count that the catch-all labeling noted above may inflate); Extraction, Lookup, Join & Reference and Filtering & Selection follow in the low hundreds; and Grouping & Categorization and Structure, Layout & Data Organization appear only a handful of times. Figure 1(b) shows steps implied, our proxy for task horizon (task difficulty in § 3-B): unimodal but right-skewed, peaking at three to four steps, with a thin tail of long-horizon tasks demanding seven or more. Instruction specificity (c; the clarity dimension) is concentrated at the high end: nearly all instructions are Well- or Very Well-Specified, so SpreadsheetBench barely probes the underspecified regime. Workbook intent (d; domain) spans more than two dozen domains but clusters in Data Entry & Validation, Financial Management, and Data Analysis, with a long tail (e.g., Risk Assessment, Legal & Contractual) represented by a few tasks each.
Each dimension, then, is populated across multiple, clearly distinguishable levels, so the dimensions discriminate among tasks; catch-all labeling inflates one task-type category, but the remaining task types and the other three dimensions show the same spread. The skew also affects evaluation, even if the largest category is overstated: because a few task types and domains dominate and many are represented by a handful of tasks, uniform random sampling over-represents the head and under-samples the tail, so benchmark size does not guarantee taxonomy coverage—which motivates the taxonomy-aware reduction we study next.
Taxonomy-Driven Benchmark Reduction
We need two things (§ 2): a coverage criterion grounded in the spreadsheet taxonomy (§ 3) and a predictive selector that uses it. This section makes both concrete and evaluates them on real industrial logs (the recorded pass/fail outcomes of past evaluations). Our method is to represent the logs as a configuration-by-task matrix, run a greedy predictive selector over it, and break ties in predictive accuracy by taxonomic diversity. Exact ties are rare, so relaxing “tie” to within of the best predictor widens the band of contenders and increases diversity, but might cost predictive accuracy if the looser contenders are weaker predictors. We answer four research questions:
RQ1 How much predictive accuracy does selecting for taxonomic coverage cost? (§ 4-F)
RQ2 When does coverage help most? (§ 4-G)
RQ3 Does the reduced benchmark survive temporal shift—does it stay predictive as configurations evolve over time? (§ 4-H)
RQ4 Is the benefit attributable to the taxonomy itself, and does it survive when every configuration is scored on one common, fully observed reference task set? (§ 4-§ 1)
We study one design choice experimentally—how hard to push for diversity ()—along with two axes that determine when it helps (the task budget and a data-quality filter); how to impute missing entries and weight tasks we decide by reasoned argument below.
Data Representation
We arrange the logs into a configuration-by-task matrix ( configurations, tasks), with where observed and missing otherwise. Missingness is common—partial runs across CI cycles, skipped or rerun tasks, inconsistent storage as infrastructure evolves—so is sparse and non-uniform. Each row of is one recorded benchmark execution; several executions can share a treatment definition (e.g., reruns of one prompt variant), so rows are not fully independent, a point we return to in § 4-E and § 6.
We treat one filter as an experimental lever: each configuration’s observed-outcome rate—the fraction of its tasks actually observed—must meet a threshold
min_completion_rate; lowering it allows more but noisier configurations, and sweeping it probes sensitivity to data noise (§ 4-G).
Predictive Minimization via Greedy Forward Selection
Our prediction target is each configuration’s full-benchmark score —its mean outcome over the tasks observed for that configuration, (App. § 5.4)—collected into , which we predict from a small subset of tasks () together with a weighting over them. Because differs across configurations, this target is configuration-specific: a difference in can reflect which tasks were run, and how hard they were, as well as how well the configuration did. We keep it for the principal analysis because it is the target the logs support, and we test in § 4-§ 1 whether the conclusions hold when all configurations share one fully observed task set. We fix the weighting first, since it also scores candidates during selection.
Weighting via Ridge regression: Each selected task has a learned weight, and we predict a configuration’s full-benchmark score as the weighted sum of its outcomes on those tasks. With many candidate tasks, few configurations, and many near-identical tasks, least squares is unstable, so we use Ridge regression, which keeps the weights small and stable through an penalty (; objective and closed form in App. § 15). We interpret the weights as a stable recipe for combining task outcomes; they do not explain why individual tasks pass or fail.
Selection via greedy forward selection: Exact subset selection is combinatorial, so we build greedily: from , each step adds the task that most improves the Ridge fit, scored by inner cross-validated (5-fold, shuffled, fixed seed), until or the inner reaches a convergence threshold; this yields a ranked, inspectable list (App. § 16).
Taxonomy-Guided Diversity
Since greedy selection only optimizes predictive accuracy, we use the taxonomy to inject a coverage objective into selection, breaking near-ties by taxonomic diversity.
Taxonomy diversity score: We score how well a reduced benchmark spans the taxonomy of § 3-B by the unweighted mean of per-dimension coverage scores,
where is every taxonomy dimension of Table I except data cleanliness (not operationalized), and each measures coverage of dimension by a form suited to its measure type—range coverage, normalized entropy, or label coverage (full per-dimension definitions in App. § 17).
Diversity tie-breaker via an -tolerance band: Exact ties in inner cross-validation (CV) are rare, but at each greedy step many candidates are nearly tied. We use these near-ties: among all candidates within of the best , we add the one that most increases the diversity score :
Predictive accuracy stays primary; raising widens the band, trading up to of inner-CV for coverage (§ 4-E, § 4-F).
Measuring Generalization Without Selection Bias
Because greedy selection optimizes an inner cross-validation objective, evaluating the reduced benchmark on the same configurations would be biased by selection. We therefore wrap the selection-and-fit pipeline in an outer validation loop: each outer split holds out configurations for judging, and the rest are used for greedy selection, inner 5-fold CV, and Ridge fitting. This yields an honest free of selection bias in the sense of Cawley and Talbot [83]; for comparison, we also report an optimistic that selects tasks once on all data and therefore leaks selection into the held-out folds. We use three outer strategies—shuffled Nested -Fold, Walk-Forward Expanding, and Walk-Forward Sliding—defined in App. § 1. Their hierarchy separates contemporaneous generalization from temporal generalization: the K-Fold/Expanding gap isolates temporal leakage, while the Expanding/Sliding gap isolates the cost of discarding older training data.
Data and Sweep Grid
We evaluate on benchmark logs with configurations filtered to a single production-candidate configuration family, yielding configurations (126 execution IDs, since some configurations were executed more than once) and candidate tasks after dropping all-missing and zero-variance columns. We sweep the full grid , , and min_completion_rate (the completion-rate filter defined in § 4-A) across the three outer-loop strategies above, with Ridge () and 5-fold inner CV as defined above throughout. RQ1–RQ3 each take up one design question, drawing on the representative grid cell , min_completion_rate together with the full min_completion_rate grids; throughout, honest is the out-of-fold measure defined above and optimistic its selection-biased foil.
Matched-budget controls (RQ4): To isolate what taxonomy guidance contributes, we compare it at the same budget against five controls: (i) pure predictive greedy selection (); (ii) -band random, which forms the same near-tie band but picks a member uniformly at random instead of by diversity—the control that separates the taxonomy’s contribution from relaxing the greedy objective alone; (iii) uniform random selection of tasks; (iv) diversity-only selection, which greedily maximizes with no predictive stage; and (v) taxonomy-stratified random selection, which draws tasks at random across taxonomy strata. Randomized controls are averaged over three seeds; deterministic methods are run once. We run the controls on two datasets: the broader filtered logs with mean imputation (min_completion_rate = 25\%; 126 execution IDs and 852 nonconstant candidate tasks after filtering, missing outcomes filled by training-fold per-task means), and the complete-case matrix described next.
Complete-case common-reference matrix (RQ4): Because is computed over a configuration-specific task set, we also built a complete execution-by-task matrix with no missing cells: 99 execution IDs by 715 common tasks (70,785 observed outcomes), with 595 selection candidates after removing zero-variance tasks. Every target and prediction is thus computed on the same observed reference set, without imputation. The 99 execution IDs correspond to only 46 distinct treatment definitions (internally, flights), the rest being repeated executions, so we present this as a complete-case sensitivity analysis; it is not an independent replication. On this matrix we run the sweep under all three validation strategies at six task budgets, and all five controls at .
RQ1: How much predictive accuracy does selecting for taxonomic coverage cost?
Most directly, we fix a contemporaneous configuration family and ask: if we reduce the full benchmark to tasks, how much predictive accuracy do we lose? Figure 2b shows that the optimistic —where selection and scoring share one configuration family—stays high and falls only slightly as grows, while Figure 2(c) shows the taxonomy diversity score rising steeply and saturating by –.

Figure 2. All at , min_completion_rate = 25%, vs. . (a) Honest weighted-Ridge for the three outer-loop strategies: K-Fold Expanding Sliding throughout, each curve non-monotone, rising at small-to-moderate before falling. (b) Optimistic (independent of the outer strategy), nearly monotone-decreasing. (c) Taxonomy diversity , rising from pure greedy selection () and saturating by –.
Together, the two show that near-complete coverage of the task taxonomy costs almost nothing in predictive accuracy over a known configuration family.
The stricter question—how well the reduced benchmark predicts new configurations—is where the honest of Figure 2 enters, and the absolute numbers drop sharply: honest is well below its optimistic counterpart across the sweep, and under the walk-forward strategies it is sometimes negative (§ 4-H). We therefore interpret the honest curves relatively, as a regularization diagnostic rather than a claim of strong prediction on new configurations. Within that interpretation they are informative: the honest curve is frequently non-monotone in , rising to a peak at intermediate before falling, whereas the optimistic decreases almost monotonically. This divergence is a regularization signature: pure greedy selection () over-optimizes for training correlations that do not transfer, and a moderate partly corrects it, so under out-of-fold evaluation maximization is not the best selection rule. Table II quantifies the effect at vs. in four cells of the grid: in three of four the diversity tie-breaker improves honest , and in the first it turns a negative positive—though several cells remain negative in absolute terms, so the gain is only reduced overfitting.
| Scenario | min compl. | honest | ||
| Nested -Fold | 1% | −0.157 | +0.059 | +0.22 (neg. pos.) |
| Nested -Fold | 5% | +0.050 | +0.159 | +0.11 () |
| Walk-Forward Sliding | 5% | −1.242 | −0.277 | +0.97 (4.5× less negative) |
| Walk-Forward Expanding | 5% | −0.253 | −0.316 | −0.06 (slightly worse) |
Table II. Honest at vs. , , across four representative cells. Moderate can reduce selection-bias overfitting.
Past the early band the two curves diverge: coverage saturates (Figure 2(c))—once the band reaches most taxonomy bins, widening it adds little coverage—while honest continues to decline past –.
A second measure supports this interpretation: the gap between optimistic and honest —a direct measure of selection-bias overfitting—is universally positive, largest for the walk-forward methods and smallest for K-Fold, and moderate tends to shrink it.
RQ2: When does coverage help most?
The regularization benefit is not uniform; it concentrates in the regimes where greedy selection is most prone to overfitting. Figure 4–Figure 9 in the appendix show the honest surface over the full min_completion_rate grid as a function of , for each validation strategy, and two patterns emerge. First, the benefit is concentrated in the low-min_completion_rate rows—the runs in which we admit configurations even though many of their task outcomes are missing. Noisier configurations make the inner cross-validation scores noisier too, so the tiny gaps that the pure greedy approach chases are mostly noise, and breaking those near-ties by taxonomic diversity helps more—the regime where the non-monotone honest- curve is clearest (the low-min_completion_rate rows of Table II). Second, the benefit is concentrated in the small- columns: at the sweep shows visible gains, while at the curves flatten and the choice of has much less effect, because the model then has enough features to fit well regardless of which tasks were selected. Because a modest reduced benchmark already captures most of the recoverable signal, the open problem is how to spend a small budget well, which is where the diversity tie-breaker helps most.

Figure 4. Nested K-Fold: honest weighted-Ridge vs. across grid for .

Figure 9. Walk-Forward Sliding: honest weighted-Ridge vs. across min_completion_rate grid for .
RQ3: Does the reduced benchmark survive temporal shift?
Only partly. Future configurations are a stricter test than interchangeable draws from one population, and the three validation instruments of § 4-D separate the two interpretations. Figure 2 shows the ordering Nested -Fold Walk-Forward Expanding Walk-Forward Sliding holding across the sweep, with Sliding often turning negative. A negative honest means the model predicts worse than the mean ; it arises because is small, the matrix is sparse, Ridge is deliberately linear, and the sliding window forces generalization across substantial temporal shift. The two gaps of § 4-D decompose this difficulty: the K-Fold/Expanding gap measures the benefit of temporal leakage, and the Expanding/Sliding gap the cost of discarding older history.
RQ4: Is the benefit due to the taxonomy, and does it survive a common reference task set?
Yes on both counts on the complete-case matrix, where every configuration is scored on the same 715 tasks; on the broader, imputed logs the coverage gain holds but the predictive advantage over random choice within the band does not hold at every .
Complete-case matrix: The diversity–accuracy pattern of RQ1 reproduces on the zero-missing matrix (Table IV, App. § 20). At , moving from to raised the diversity of the reduced benchmark selected on all data from 0.649 to 0.815 and improved honest weighted-Ridge under all three validation strategies. Across the six task budgets, moving from to increased diversity in all 18 validation-by-budget comparisons and honest in 13 of the 18. Since every cell is observed and every configuration is scored on the same 715 tasks, the pattern cannot come from imputation or from differences in which tasks ran. Among the controls, taxonomy-guided selection substantially increases diversity relative to pure greedy, uniform random, and -band random selection at the same budget. Diversity-only selection maximizes coverage but predicts poorly, so the predictive stage is necessary. And taxonomy-guided selection outperforms its matched -band-random control—same band, random choice within it—under all three validation strategies at both and , so the gain over pure greedy selection comes from choosing within the band by taxonomic diversity.
| Method | Nested | Expanding | Sliding | Diversity | |
| Greedy predictive | – | 0.689 | 0.379 | 0.472 | 0.649 |
| Taxonomy-guided | 0.10 | 0.693 | 0.482 | 0.479 | 0.820 |
| Taxonomy-guided | 0.05 | 0.731 | 0.493 | 0.496 | 0.815 |
| -band random | 0.10 | 0.569 | 0.277 | 0.225 | 0.485 |
| -band random | 0.05 | 0.626 | 0.275 | 0.324 | 0.588 |
| Uniform random | – | 0.567 | 0.301 | 0.338 | 0.490 |
| Diversity-only | – | 0.490 | -0.014 | 0.067 | 0.820 |
| Taxonomy-stratified | – | 0.602 | 0.205 | 0.217 | 0.665 |
Table IV. MATCHED-BUDGET COMPARISON ON THE ZERO-MISSING COMMON-REFERENCE RECTANGLE (99 EXECUTION IDS, 715 COMMON TASKS, 595 NONCONSTANT CANDIDATES, ). VALUES ARE HONEST WEIGHTED OUTER-VALIDATION . RANDOMIZED METHODS REPORT THE MEAN OVER THREE SEEDS. DIVERSITY IS MEASURED ON THE SUITE SELECTED USING THE FULL COMPLETE-CASE DATASET AND IS AVERAGED ACROSS SEEDS FOR RANDOMIZED METHODS.
Broader, imputed logs: On the broader filtered logs, taxonomy guidance again substantially increases diversity and usually improves honest over matched -band random selection: at it is higher under all three validation strategies. The one exception is Nested -Fold at , where -band random is marginally higher (0.401 vs. 0.396). The complete-case result is uniform and the imputed result is not, which suggests that the taxonomy’s advantage over random choice within the band depends on , on missingness, and on how well calibrated the inner-CV predictor is; a systematic analysis is future work. Our claim is therefore narrower: taxonomy guidance gives a controllable increase in coverage at little or no cost in honest predictive accuracy, and, on a fully observed common reference set, predicts better than matched random choice within the same band.
Across the four questions, then, the evidence supports the finding anticipated in § 4: the diversity tie-breaker behaves as a regularizer over greedy selection rather than a trade-off—we discuss what this means for practice in § 6.
Discussion
Interpreting the Results
The diversity tie-breaker as a regularizer: Across RQ1–RQ3, the diversity tie-breaker is better understood not as a trade-off but as a regularizer over greedy selection: pure greedy selection () over-optimizes training correlations that do not transfer to held-out configurations, and breaking near-ties by taxonomic diversity partly corrects this.
RQ4 attributes the correction to the taxonomy: random choice within the same band does not reproduce the coverage gain, and on the complete-case matrix it predicts worse. Ridge regularizes the learned weights, and the tie-breaker regularizes which tasks greedy selection picks; both give up a little fit on the selection data for better held-out prediction.
When a reduced benchmark can be trusted: The honest- results limit what a reduced benchmark should be used for. For contemporaneous configurations—ablations, hyperparameter variants, release candidates of one base model—it is a sound instrument for summarizing or ranking them, and a taxonomy-spanning one has construct and face validity that a coverage-blind top- lacks, resists redundancy, and is harder to game if the full-benchmark aggregate is later reweighted. Across time, we advise caution: the low-rank structure the method assumes holds under the independent and identically distributed (i.i.d.) assumption but not stably as configurations evolve (the Walk-Forward Sliding turns negative).
Task-count reduction: The compression we study is steep in task count: tasks drawn from roughly a thousand (about at ), with the saving recurring on every evaluation while the curation cost is paid once. Task count is not cost, however: tasks differ substantially in runtime and token usage, and the log snapshot we analyze does not include token-usage telemetry or comparable wall-clock measurements. Our claims are therefore about task-count reduction, not measured monetary, token, or runtime savings, which depend on the heterogeneous costs of the selected tasks; a cost-aware selector (a budget in tokens or minutes rather than tasks) and prospective measurement are future work.
Implications
For agent teams: choosing and refreshing: A small tolerance band, of roughly 0.05–0.1, is the operating point we recommend. At a moderate the reduced benchmark covers nearly the whole taxonomy with no loss in honest relative to pure greedy selection, and often a small gain that, as the absolute numbers make clear, reflects reduced overfitting rather than strong prediction on new configurations; beyond –0.3 coverage has saturated while honest keeps falling. No single is uniformly optimal: 0.1 worked best on the principal sweep and 0.05 was more reliable on the complete-case matrix, so the regularizer’s strength should be tuned to the noise level and budget at hand, with the strongest case for moderate in the noisy, low-min_completion_rate, small- regimes of § 4-G. Because the reduced benchmark does not transfer stably across time (§ 4-H), it should also be refreshed—reselected—when the configuration population drifts, rather than kept as a fixed proxy for future configurations.
For agent teams: weighting the taxonomy: Our diversity metric weights every taxonomy dimension equally, but a team shipping charting features cares more about visualization-related task types and LLM capabilities than about the full range of header types. The metric extends to a relevance-weighted mean, with weights stated by the team or derived from production telemetry (the task types users issue, and especially those where the agent fails or users retry); the change reshapes but leaves the predictive band untouched, so the regularizer interpretation still holds.
For agent teams: auditing coverage gaps: Apart from selection, the dimensions of § 3 provide a coordinate system for auditing an evaluation portfolio: labeling both existing benchmark tasks and a sample of production traffic along the same dimensions reveals regions of the task space that users exercise but the benchmarks under-cover. Teams at Microsoft used the taxonomy this way, applying it to telemetry and internal benchmarks to identify task types and workbook conditions their benchmarks did not adequately cover. Gap analysis thus identifies which regions need new tasks, while taxonomy-aware selection compresses what is already covered.
For benchmark builders: Our descriptive analysis (§ 3-C) is itself actionable: SpreadsheetBench, one of the most widely used public spreadsheet benchmarks, is dominated by Formulas, Logic & Automation (a plurality the rubric’s catch-all tendency may inflate; § 3-C), almost all Well-Specified, and barely probes regions that are important in practice—data cleaning over noisy workbooks, underspecified or ambiguous instructions (Figure 1). We encourage benchmark builders to populate these regions deliberately and to report coverage along taxonomy dimensions rather than raw task count, so that size is not confused with breadth.
For other agent domains: Our approach is not specific to spreadsheets: agent benchmarks in other domains (coding, web, desktop) likewise define ad-hoc task categories without a taxonomy designed for coverage-oriented selection (§ 1.1). Practitioners there could follow the same construction process [14], label their tasks, and apply taxonomy-aware minimization; validating this is future work.
Limitations
Several caveats temper these results, most rooted in the small, noisy logs. Few, correlated configurations, one source. All three honest- estimators are high-variance (especially Walk-Forward Sliding), and the rows of are not independent: several executions can share one treatment (99 executions for 46 treatments in the complete-case matrix), which weakens the i.i.d. assumption behind Nested -Fold. All evidence comes from one industrial source; replication on another agent benchmark is needed before generalizing. Missing data and the target. In the principal analysis each averages over its own configuration-specific task set and missing cells are mean-imputed, which attenuates the coefficients of heavily-missing columns and may not match the missingness mechanism. The complete-case analysis (RQ4) removes both confounds within its rectangle, but may not generalize to the excluded executions and tasks, and imputation can still affect the exact estimates in the broader analysis. Label reliability. The labels on which depends are LLM-assigned and were spot-checked by hand on a sample (§ 3-C); we did not compute a formal agreement statistic, and a stratified two-rater study with per-dimension agreement () remains future work. Because they are used only in the tie-break, label noise cannot change the predictive band . The honest- results use task outcomes, not labels, so the benefit we measured was obtained despite imperfect labels. The coverage figures, computed from the same labels, are the one quantity noise could overstate. With uninformative labels the tie-breaker would degrade to -band random choice, so beating that control on the complete-case matrix is indirect evidence that the labels are informative. A perturbation analysis of label errors would quantify how much noise the tie-break tolerates. The metric. is unstable when is small, so mean absolute error or rank correlation may be more informative; we treat the validation hierarchy as a diagnostic. A more reliable reduced benchmark needs more configurations, a stable target (a fully observed reference set, as in RQ4), less selection freedom (smaller , stronger ), tuned to the noise level, and reporting which tasks recur across outer folds.
Conclusion
Our taxonomy of spreadsheet tasks gives benchmark coverage a concrete, externally observable meaning—the “with respect to what” that ad-hoc benchmark categories lack—and our taxonomy-guided selector turns that meaning into reduced benchmarks. On real logs from Microsoft, selecting for taxonomic coverage costs little predictive accuracy and often gains it, behaving as a regularizer over greedy selection, strongest where data is noisy and budgets are small; matched-budget controls rule out a relaxed greedy objective as the source of the gain, and a complete-case replication rules out imputation. Honest evaluation also marks the limits—poor transfer across a long time gap when trained on little recent history, and a reduction measured in task count, not yet in cost—so the practical recipe is a small, taxonomy-spanning reduced benchmark, refreshed as the configurations drift.
Data Availability: The benchmark logs are confidential and cannot be shared. We will release the analysis code, with confidential identifiers removed, an input schema, run commands, seeds, hyperparameters, and instructions for rerunning it on another configuration-by-task matrix.
Acknowledgments: The authors used AI assistants (GitHub Copilot and Claude Code, on Anthropic’s Claude and OpenAI’s GPT models) to draft portions of the manuscript from our notes and analyses, to write code behind many of the numbers and figures (which we checked), and to copy-edit prose under our direction; LLMs are also a component of the taxonomy-construction method (§ 3-A). All research questions, study design, analysis decisions, and interpretations are our own, and we take full responsibility for the paper’s content.
Complete Taxonomy Value Sets
Table I lists only example measures and values for each dimension. For completeness, we enumerate below the full value set of every close-ended (categorical or multi-label) measure that appears in Table I, grouped by the three tiers of the taxonomy. Numeric measures (e.g., table and workbook size, or task difficulty), which take ranges rather than discrete values, and open-ended measures (e.g., data types, language, and free-text operation phrases), which admit no fixed value set, are therefore not listed. Where a per-value definition is given, it is taken (lightly edited) from the labeling prompt or capability schema used to assign the value; bare enumerations are listed without definitions. These value sets are produced programmatically from the labeling configuration so that they stay in sync with the implementation.
Table Properties
Header type (2 values).: The structure of a table’s header (see also Figure 3).

Figure 3. Illustrative spreadsheet structures: (a) flat column-major layout with a simple header; (b) flat row-major layout; (c) non-flat layout with a hierarchical header; (d) two distinct table-like regions on one sheet.
Simple: A single header row only.
Complex: Multiple header rows or one or more header columns present.
Structure (4 values).: The layout of a table, as detected from its orientation and headers.
columnMajorFlat: Flat table with headers in the first row and data in columns below.
rowMajorFlat: Flat table with headers in the first column and data in rows to the right.
hierarchical: Table with nested or multi-level row or column headers.
other: A table structure that does not fit the standard patterns.
Spreadsheet and Workbook Properties
Workbook primary intent (31 values).: The purpose of the workbook (the Domain/Topic dimension), assigned by an LLM from the workbook content.
Financial Management: budget, revenue, expenses, financial reporting, cash flow, profit and loss, balance sheet, financial planning, investment, funding sources
Project Planning: project management, timeline, task tracking, milestone, resource allocation, Gantt chart, progress monitoring, project schedule
Data Analysis: data analysis, statistical analysis, data visualization, summary statistics, pivot table, data aggregation, trend analysis, comparative analysis
Regulatory Compliance: compliance, regulatory reporting, audit, legal compliance, certification, documentation, validation, reporting requirements
Human Resources: payroll, salary, employee information, attendance, leave management, compensation, benefits, staff, personnel costs
Inventory Tracking: inventory, stock management, product catalog, item code, quantity, procurement, supply chain, warehouse
Academic Administration: student data, academic year, enrollment, grade, course, subject, teacher, school, educational assessment
Healthcare Records: patient data, diagnosis, procedure code, healthcare provider, medical billing, insurance, clinical data, treatment, hospital
Event Scheduling: calendar, schedule, event, dates, planning, timetable, session, participant, attendance
Construction Management: construction, project cost, bill of quantities, contractor, materials, site location, infrastructure, building type
Grant and Funding: grant application, funding request, grant allocation, award, program funding, grant reporting, subrecipient
Risk Assessment: risk management, risk assessment, risk register, mitigation, failure analysis, safety, audit, action plan
Sales and Marketing: sales, product, customer, order, pricing, marketing, campaign, revenue, sales tracking
Community Engagement: community development, stakeholder, outreach, engagement, social impact, volunteerism, public health
Sustainability Reporting: sustainability, environmental impact, energy consumption, waste management, greenhouse gas emissions, social responsibility
Transportation Data: transportation, transit operations, vehicle types, route, ridership, infrastructure, maintenance, public transport
Scientific Research: experimental data, gene expression, sample identification, biological data, measurement, data analysis, research study
Legal and Contractual: contract, agreement, legal status, contract type, terms, compliance, documentation, approval
Demographic Analysis: population, age group, gender, region, demographic data, census, household, survey
Insurance Administration: insurance, premium, coverage, claim, policy, benefit, deductible, plan, provider
Real Estate Management: property, address, value, rent, ownership, real estate, property type, appraisal
Resource Allocation: resource planning, resource allocation, bandwidth, man-hours, team, assignment, utilization
Educational Resources: textbook, curriculum, lesson, teaching materials, resource type, subject area, academic module
Data Entry and Validation: data entry, validation, checklist, template, instructions, error checking, data quality
Sports Management: team, match, league, player, score, tournament, competition, performance tracking
Procurement and Purchasing: procurement, purchasing, vendor, supplier, order, invoice, contract, product catalog
Taxation and Accounting: tax, income tax, deduction, ac-
counting, journal entry, financial statement, tax calculation
Healthcare Services: healthcare, service type, facility, provider, patient, treatment, care, reimbursement
Organizational Directory: organization, contact information, address, phone number, email, staff directory, department
Metadata and Structure: worksheet structure, metadata, creator information, modification date, data structure, spreadsheet type
Categorization and Taxonomy: taxonomy, classification, category, code mapping, data categorization, thematic analysis, spreadsheet taxonomy
NAICS industry sector (20 values): A complementary Domain/Topic measure: the 2-digit North American Industry Classification System sector inferred from the workbook content.
11 – Agriculture, Forestry, Fishing and Hunting
21 – Mining, Quarrying, and Oil and Gas Extraction
22 – Utilities
23 – Construction
31 – Manufacturing
42 – Wholesale Trade
44 – Retail Trade
48 – Transportation and Warehousing
51 – Information
52 – Finance and Insurance
53 – Real Estate and Rental and Leasing
54 – Professional, Scientific, and Technical Services
55 – Management of Companies and Enterprises
56 – Administrative and Support and Waste Management and Remediation Services
61 – Educational Services
62 – Health Care and Social Assistance
71 – Arts, Entertainment, and Recreation
72 – Accommodation and Food Services
81 – Other Services (except Public Administration)
92 – Public Administration
Task and Instruction Properties
Instruction specificity (5-point scale): The Clarity dimension, rated against a fixed rubric.
1 (Very Ambiguous): Impossible to act on without significant clarification. Lacks any specific target or action.
2 (Ambiguous): Missing critical information. Has multiple plausible but different interpretations.
3 (Reasonably Specified): The main intent is clear, but minor assumptions are needed.
4 (Well-Specified): The query is clear, direct, and contains enough information to be executed with high confidence.
5 (Very Well-Specified): Perfectly clear and unambiguous, likely referencing specific columns, sheets, or values, leaving no room for interpretation.
Meta operation – operation types (35 values): The action component of the Operations dimension; a task may carry several. add, align, allocate, calculate, compare, copy, count, create, disable, display, fill, filter, fix, format, freeze, hide, identify, link, merge, navigate, paste, protect, refresh, remove, rename, replace, resize, retain, save, sort, split, summarize, unmerge, validate, None.
Meta operation – operation objects (44 values): The object each operation acts upon; a task may carry several. aggregation, alignment, border, cell, cell reference, chart, column, comment, connection, dashboard, data, error, filter, footer, format, formula, header, hyperlink, image, macro, Machine Learning Model, named range, number, object, page layout, pane, picture, PivotTable, protection, row, shape, size, slicer, sparkline, style, table, text box, theme, tool, validation rule, value, workbook, worksheet, None.
Task type – action type (3 values): The coarsest operationalization of Task type: whether the instruction asks for information or directs an action.
ask: Questions about insights, explanations, or recommendations without an explicit instruction to modify the workbook.
act: Directives to create, edit, format, calculate, or otherwise change or produce spreadsheet artefacts.
unknown: The request could be interpreted either way or lacks sufficient detail.
Task type – intent themes (12 values): One operationalization of Task type, derived by thematic analysis of the instruction text.
Filtering & Selection: Filtering, subsetting, excluding, or deduplicating records/entities based on criteria or thresholds.
Aggregation & Summarization: Summing, counting, frequency, totals, breakdowns, subtotals, and high-level numeric summaries.
Grouping & Categorization: Grouping, segmenting, classifying, labeling categories or entities for organization or comparative frames.
Comparison, Ranking & Extremes: Comparing entities/groups/time periods, ranking/order, identifying max/min/top/bottom/extremes and differences.
Statistics, Percentages & Metrics: Averages, medians, ratios, percentages, rates, statistical and correlation measures.
Temporal & Trend Analysis: Time-based or sequential analysis: trends, patterns over periods, period-over-period comparisons.
Extraction, Lookup, Join & Reference: Retrieving specific values, lookups, joins, merges, matching, referencing or attributing data.
Data Cleaning, Quality & Error Handling: Handling missing/invalid data, validation, error/outlier/anomaly detection, deduplication for quality, formula error resolution.
Transformation, Feature & Text Engineering: Transforming structure or formats, feature engineering, mapping,
text parsing/splitting/concatenation, numeric/date conversions, sequence generation.
Visualization, Formatting & Presentation: Charts, graphs, dashboards, conditional formatting, highlighting, styling, and presentational layout.
Formulas, Logic & Automation: Creating/modifying formulas, conditional logic, automation (macros/VBA), dynamic calculation behavior.
Structure, Layout & Data Organization: Managing tables, pivot tables, sheets, columns, layout, data movement/placement, consolidation, access/permissions.
LLM capabilities (8 categories, 20 subcategories).: The agent skills a task requires; also serves as the capability-based operationalization of Task type. Listed as category (with its subcategories).
Analysis — All forms of data analysis and exploration
Analyze Descriptively: Basic Multi-Dimensional Analysis (MDA) operations on tabular entities
Analyze Advanced: Sophisticated analytical techniques determined by schema and relationships
Visualization — Creating visual representations and interactive data exploration tools
Create Charts: Creating, modifying, or analyzing charts and visualizations
Create PivotTables: Creating, modifying, or analyzing PivotTables and PivotCharts
Data Management — Operations for acquiring, organizing, and maintaining data
Import Data: Importing data from files that Excel handles natively
Clean Data: Cleaning, organizing, or restructuring messy data
Shape Data: Transforming data structure and layout
Create Tables: Creating structured tables or converting ranges to tables
Validate Data: Implementing data validation rules or checking data quality
Style Data: Applying visual formatting and styling to data and worksheets
Computation — Mathematical operations and calculations
Use Formulas: Creating, modifying, or using Excel formulas and functions
Run Python: Using Python within Excel for advanced computations
Content Processing — Working with text and non-tabular content
Analyze Text: Analyzing, processing, or manipulating text data
Summarize Text: Summarizing or condensing text content
Translate Text: Translating text between different languages
Process Images: Working with images, PDFs, or visual content
Integration — Connecting with external systems and collaboration
Use M365: Using Microsoft 365 integrations and cloud services
Connect External: Connecting to external systems and data sources
Collaborate: Adding comments, sharing, or collaborative features
Automation — Automating repetitive tasks and workflows
Automate Tasks: Creating macros, VBA scripts, or other automation
Other — Actions that don’t fit into other specific categories
Toy Reduced-Benchmark Example
Suppose a benchmark of 1,000 spreadsheet tasks, with configurations varying primarily along two latent capabilities: formula reasoning and data cleaning. A configuration strong on formula reasoning but weak on data cleaning passes most formula tasks (say, 600/700) but few data-cleaning tasks (50/300), for an overall pass rate near 0.65; a configuration with the opposite profile reaches a similar rate for opposite reasons. Because pass rate is an average over outcomes, it can in principle be reconstructed from any small set of tasks that captures both factors—here, two well-chosen tasks plus a learned weighting from their outcomes to the full-benchmark score. Conversely, if 200 formula tasks measure the same sub-skill, dropping 199 of them costs almost nothing in predictive accuracy; the survivor represents its cluster.
Table 3 makes this concrete on a stylized 4 × 6 example in which F1–F2 and D1–D3 are perfectly correlated: four of the six tasks are redundant, and the two-task reduced benchmark {F1, D1} reconstructs every configuration’s full-benchmark pass rate exactly.
| Formula tasks | Data-clean. tasks | |||||||
| Config | F1 | F2 | F3 | D1 | D2 | D3 | (full) | ({F1, D1}) |
| (F+, D−) | 1 | 1 | 0 | 0 | 0 | 0 | 0.33 | 0.33 |
| (F−, D+) | 0 | 0 | 1 | 1 | 1 | 1 | 0.67 | 0.67 |
| (F+, D+) | 1 | 1 | 1 | 1 | 1 | 1 | 1.00 | 1.00 |
| (F−, D−) | 0 | 0 | 0 | 0 | 0 | 0 | 0.00 | 0.00 |
Table 3. Stylized 4 × 6 instance of the mechanism described in the text: formula tasks F1–F3 and data-cleaning tasks D1–D3, full-benchmark pass rate , and its estimate from the two-task reduced benchmark {F1, D1} with learned ridge weights .
Taxonomy Construction Details
Choosing concrete measures: With a dimension established, we must decide how to measure it: which one or more concrete measures assign a task a value along the dimension, and which values each measure allows. We deliberately allow a single dimension to have more than one measure, because an axis of variation can often be operationalized in different ways with distinct tradeoffs (the Task type dimension below is our clearest example). This extends the single-characteristic-set view of Nickerson et al. [14]—in whose terms a measure’s value set is the dimension’s set of characteristics—and is reflected in the rightmost column of Table I.
Objective (structural) measures can be extracted programmatically from workbook metadata (e.g., cell counts, merged-cell detection, hierarchical structure), without human judgment. Subjective (interpretive) measures require interpreting the task instruction or spreadsheet context, and we operationalized them in one of two ways depending on whether the measure’s value set was emergent or fixed in advance. When the value set had to be discovered from the data, we used LLM-assisted thematic analysis following Naeem et al. [84], who validated their approach against manual thematic analysis. For each task, we gave GPT-4.1 the axis to code along (e.g., task type), the task instruction, and a sample of the workbook data, and the LLM produced initial codes for that task. After coding all corpus tasks this way, we had the LLM group the per-task codes into higher-level themes, which became the measure’s value set. We reviewed these groupings and iterated on the prompts and codebook (e.g., after noticing a few instances of prompt bleeding/context leakage) until the categories stabilized. When the value set was fixed in advance, we instead had the LLM rate each task against a predefined rubric—for instance, scoring instruction specificity on a 1–5 scale—so no theming step was required.
The task type dimension is our clearest case of a dimension admitting several candidate measures. We explored alternatives because it is arguably the most consequential for the downstream evaluation, and because “task type” is not one obvious concept—different operationalizations are study-design alternatives with distinct tradeoffs. We compared four, mirroring the deductive/inductive alternation above. First, we applied bottom-up thematic analysis [84] of the natural-language instructions, producing fine-grained action-oriented categories (e.g., Aggregation & Summarization, Comparison & Ranking, Filtering & Selection, Formulas & Automation); these are well-grounded in the data but may yield more categories than are practical for test selection. Second, we tried LLM-amplified few-shot clustering [85], which is query-efficient—the LLM labels most items and routes uncertain cases near the decision boundary to a human judge—but in our setting it yielded coarser, more function-oriented groups (e.g., Data Analysis, Data Filtering, Formula, Pivot Table) that risk merging meaningfully different tasks. Third, product managers at Microsoft did top-down coding, defining task types by the capabilities the agent requires. This produced an agent-centric hierarchy (e.g., Descriptive Analysis, Computation—Using Formulas, Visualization—Creating PivotTables) that is directly useful for capability-gap analysis but reflects the product team’s framing and may not generalize beyond a specific agent. Finally, we tried coding the workbook’s primary intent rather than the natural-language instruction, which produced purpose-oriented types (e.g., Data Entry & Validation, Financial Management) but can conflate tasks whose instructions differ substantially yet operate on similar workbooks. Thus, even for a single dimension, one can vary both what is coded (the natural language task instruction, the implied workbook intent, or the required agent capabilities) and how (thematic analysis, few-shot clustering, or top-down decomposition), and each choice has different tradeoffs in granularity, generalizability, and practical utility.
We faced a similar choice for Domain/Topic. Coding workbook intent, as above, groups tasks by what the spreadsheet is for, but two tasks with the same purpose (e.g., financial reporting) may involve very different data domains. We therefore added a complementary operationalization that assigns, via LLM labeling of the workbook content, a sector from the North American Industry Classification System (NAICS),4 the standard U.S. federal taxonomy for classifying businesses by industry (e.g., sector 52 Finance and Insurance, sector 44–45 Retail Trade). NAICS sector provides a standardized, reproducible domain label, while workbook intent captures purpose-level variation that sector codes miss; keeping both gives the taxonomy finer-grained coverage of the Domain/Topic space.
Validation and termination: Because the process is iterative, we needed criteria for when to stop. We terminated refinement when the taxonomy satisfied two groups of ending conditions [14]. Objectively: (i) for single-label measures each task receives exactly one value (e.g., header type is either simple or hierarchical, not both); (ii) for multi-label measures a task may receive several values, because a single instruction can involve multiple operations or skills (e.g., a task’s meta operations may include lookup, calculate, and visualize); and (iii) every task in the corpus is classified by every dimension. Subjectively, we required the taxonomy to be concise, robust, comprehensive for our scope, extendible, and explanatory for error analysis [14]. To check these conditions, two researcher co-authors reviewed the full taxonomy, discussed disagreements, merged redundant categories, and clarified definitions, incorporating practitioner feedback from internal presentations and sessions with researchers and product managers at Microsoft to prioritize evaluation-relevant distinctions [82].
Taxonomy Dimension Examples
The examples below use SpreadsheetBench [3], which is public. We refer to tasks by their SpreadsheetBench identifiers (e.g., 13–1). Because the original SpreadsheetBench tasks are phrased as how-to questions to a chatbot, we rephrased them into imperative instructions for an AI agent while keeping all other task details unchanged.
Table properties
The unit of analysis is the workbook, but several dimensions describe the structure of individual tables within it; our SpreadsheetBench annotations record these signals via per-table metadata and sheet-/workbook-level aggregates. The simpler structural measures—table size (rows and columns), vocabulary size (number of unique values), data types (e.g., date, numeric, text), and language (almost always English)—are as defined in Table I; two dimensions warrant illustration. Header type captures whether headers are simple (Figure 3(a)) vs. hierarchical with merged cells (Figure 3(c)). Structure captures table layout: flat column-major means each column is an attribute and each row a record (Figure 3(a)); flat row-major transposes this so that each row is an attribute and each column a record (Figure 3(b)); non-flat covers crosstabs, pivoted layouts, and other arrangements where neither reading direction alone suffices (Figure 3(c)). Finally, data cleanliness captures noise such as duplication or inconsistent formatting.5
Spreadsheet and workbook properties
Size captures the scale of the parent spreadsheet or workbook as a whole—its number of sheets, its number of table-like regions, and its on-disk file size. For example, task 13-1 explicitly spans two sheets (“Combine data from the ‘RANGES’ sheet into the ‘LISTS’ sheet . . . ”) and its workbook contains six table regions (three on each sheet), whereas task 19-7 uses a single sheet with a single table region (Figure 3(d) illustrates a sheet with multiple regions). We include file size to capture scaling effects for the evaluation harness. Structural complexity aggregates these structural signals (e.g., size and layout) into an overall rating. For instance, a workbook with 5208 rows and five columns (task 19-7) likely represents a very different workload than one with 85 rows and nine columns (task 13-1). Domain/Topic captures the subject matter and the primary intent of the workbook (recall § 3-A). For example, the workbook for task 17-35 is a supplier invoice register whose columns record dates, supplier names, tax-identification numbers, invoice numbers, and monetary amounts for pharmaceutical vendors—placing it in NAICS sector 42 Wholesale Trade; its inferred workbook intent is Data Filtering.
Task properties
Clarity captures how well-specified the instruction is, on a scale from 1 to 5: some tasks provide detailed ranges and constraints (e.g., clarity score 5 for task 19-7: “split data from the ‘LASTROW’ sheet into two separate sheets named ‘MINUS’ and ‘PLUS’, based on negative and positive values found in column E”), while others are underspecified. Operations capture the action–object intent signals in the instruction at two levels. At the surface level, query verb–noun phrases record specific actions and their objects, e.g., task 13-1 includes phrases like “delete old ranges” and “add TOTAL row to sum amounts,” while task 19-7 includes “create macro” and “delete existing sheets.” At a more abstract level, meta operations capture composable operation primitives: task 17-35 includes filter (data), validate (cell), and add (column); task 13-1 includes remove (data), merge (data), and add (row); and task 19-7 includes remove (worksheet), split (data), and allocate (cell).
Task type captures the coarse-grained kind of work a task demands. SpreadsheetBench distinguishes explicit questions (ask, e.g., “Explain what is wrong with the VBA code . . .” in task 209-30) from action requests (act, e.g., “Create a macro to split data . . .” in task 19-7), and from cases where the instruction is too vague to determine intent (unknown in task 55224). Task difficulty reflects both the task’s horizon (the number of operations/reasoning steps required) and its operation mix; for example, task 19-7 explicitly bundles deleting sheets, splitting data, placing outputs into specific ranges, and inserting total rows—multi-step macro creation over a 5208 × 5 table—versus smaller single-formula tasks.
LLM capabilities capture the types of agent skills required, such as automation and formula use (e.g., Automation.Automate Tasks and Computation.Use Formulas in task 19-7). Although some overlap exists between Task type and LLM capabilities (e.g., a “data visualization” task type often requires a “charting” capability), they capture different facets: Task type is grounded in properties of the workbook–instruction input—consistent with our criterion of externally observable properties—whereas LLM capabilities reflects the agent-centric view originating from Microsoft’s product managers. Keeping them apart lets the taxonomy distinguish what the user asks for from what the agent must be able to do.
Formalization of the selection task
Let be a set of configurations (e.g., candidate model variants). Let be the set of tasks. For each configuration and task we observe an outcome (fail/pass), with missingness allowed when a task was not executed.
We define the full-benchmark score for configuration as the mean success across all tasks observed for that configuration:
where denotes tasks with non-missing outcomes for . Note that is configuration-specific; the complete-case analysis of § 4-§ 1 instead scores every configuration on one common, fully observed .
Goal. Choose a small subset , , such that a score computed from only tasks in accurately predicts across configurations.
Scope of (which configurations do we care about?). A reasonable concern is that some configurations in historical logs are “old” (e.g., obsolete model generations) and may never be deployed again. We do not assume that all historical configurations are decision-relevant. Instead, should be understood as the set of configurations relevant to a particular evaluation goal (e.g., a current model family, a specific tool stack, or a time window). The predictive objective is then: within that scope, can a small reduced benchmark predict the full-benchmark score well enough to speed up iteration and to compare candidates on the Pareto frontier of quality vs. cost/latency? Operationally, this can be enforced by filtering or weighting configurations by relevance (model family, recency, tool availability), and our scripts already support configuration-family filtering (prefix) and explicit exclusions.
Ridge Regression and Conditioning
With a small number of configurations and many candidate tasks (), ordinary least squares (OLS) is ill-conditioned: many tasks are redundant or strongly correlated, so small changes in the observed outcomes (or the particular train/test split) can produce large swings in fitted coefficients. Ridge regression addresses this by adding an regularization term that shrinks coefficients toward zero. Intuitively, Ridge prefers explanations of that use moderate weights spread across correlated tasks rather than extremely large positive/negative weights that cancel each other. This improves stability (and often generalization) by reducing estimator variance, at the cost of introducing a small bias (a standard bias–variance tradeoff).
In our setting, these properties matter for two practical reasons. First, the design matrix of (largely binary) task outcomes can make nearly singular when or when many tasks are near-duplicates; Ridge yields a unique, well-behaved solution in this regime. Second, we interpret the resulting coefficients as an importance weighting over the selected tasks, producing a weighted proxy score that is less sensitive to noise than an unregularized fit.
Objective.: Given a candidate set of columns , we fit
where denotes the imputed matrix restricted to columns in and is a regularization constant. The penalty yields the closed-form solution , which is well-defined even when is singular. Increasing makes the fit less sensitive to idiosyncratic patterns in small datasets (more shrinkage), while decreasing it approaches OLS (less shrinkage, higher variance); we use throughout.
Greedy Forward Selection Procedure
Selecting the best subset of tasks is combinatorial, so we build with greedy forward selection. Starting from , at each step we evaluate every remaining candidate task by fitting Ridge on under 5-fold inner cross-validation (shuffled -fold, fixed seed) and computing the cross-validated (CV) . The candidate that yields the highest inner CV is added to :
We stop once , or when the inner CV exceeds a convergence threshold. Besides being tractable, the procedure produces a ranked list that practitioners can inspect. The diversity tie-breaker of § 4 modifies only the selection rule , replacing the single best candidate with a diversity-maximizing one drawn from the -band of near-best candidates.
Algorithm 1: Greedy Selection
Input: Task Set Output: Greedy Selected Task Set
for in do
return
Algorithm 2: Modified Greedy Selection
Input: Task Set , Tolerance Band Output: Selected Task Set
for in do
return
Taxonomy Diversity Score
Each taxonomy dimension contributes a coverage score to the diversity score of § 4, with a form that depends on its measure type:
where, over a set of tasks , is the spread of ’s numeric values and the set of labels present; is the selected tasks’ distribution over ’s categories and the number of those categories globally. Numeric measures (e.g., table and workbook size, clarity, task difficulty) use range coverage; single-label categorical measures (e.g., header type, table structure, task type, domain) use entropy; and multi-label measures (e.g., meta operations and required LLM capabilities) use label coverage. Where a dimension is captured by several numeric signals (e.g., workbook size and structural complexity), we summarize them by their top principal components, which we use as numeric measures.
Outer-Split Validation Strategies
The three outer-loop strategies of § 4-D differ in how configurations are partitioned:
Nested -Fold. Configurations are shuffled and partitioned into 5 outer folds. In each outer fold the training configurations are used exclusively for selection and fitting, the held-out configurations only for prediction, and the final is computed from the concatenated out-of-fold predictions.
Walk-Forward Expanding. Configurations are sorted chronologically by build id and partitioned into equal-size sequential windows. The outer loop advances forward in time: fold trains on windows and tests on window .
Walk-Forward Sliding. Identical to expanding, but the training window has a fixed size equal to twice the test window, so that if the test window covers indices , the training window covers . Older data is dropped as evaluation advances, so each fold trains on a limited recent history.
On Mean-Imputation
We impute each missing entry with its per-task mean
(the task’s average outcome over the configurations that ran it), writing for the resulting dense matrix when observed and otherwise), so that configurations can be compared on common ground. Within each outer validation fold we compute over the training configurations only and identify zero-variance tasks on the training fold alone, so a held-out configuration never contributes to the values imputed for it. The final selection on all data uses means over all configurations. The RQ4 broader analysis (§ 4-§ 1) follows the same training-fold convention. Configurations retained by the min_completion_rate filter are still mean-imputed at the task level in exactly this way. The obvious alternatives to mean-imputation are each biased in this log regime: dropping configurations with any missing outcome discards most data when is large; dropping ever-missing tasks favors those that happen to run more often; and averaging over only the observed selected tasks introduces a variable denominator, scoring different configurations on different effective subsets. A learned imputer (e.g., -nearest-neighbors or low-rank matrix completion) would instead add a genuine second model that stacks prediction errors into the regression’s inputs and leaks information across tasks. Mean-imputation is the simplest well-defined baseline. It is, moreover, a placeholder rather than a predictive step: ignores everything about configuration and is identical for every configuration missing task , so an imputed cell cannot help separate configurations—it only dilutes the signal in the observed entries, never manufactures new signal. The sole predictive step is the Ridge fit over the selected columns; the cost of imputation is statistical rather than architectural, attenuating the fitted coefficients of heavily-missing columns (a bias we revisit in § 6).
Matched-Budget Controls on the Complete-Case Matrix
Table IV compares taxonomy-guided selection with the five controls of § 4-E at on the zero-missing common-reference matrix (999 execution IDs by 715 common tasks, 595 nonconstant candidates); § 4-§ 1 summarizes the result. At , taxonomy-guided selection has the highest honest under all three validation strategies, and at both values it exceeds its matched -band-random control under all three. At its diversity matches that of diversity-only selection, which has the lowest honest of any method.
Honest over the full min_completion_rate grid
Figures 4–9 report the honest weighted-Ridge vs. across the full min_completion_rate grid, one figure per validation strategy. Across all three, the regularization benefit of moderate is most pronounced at low min_completion_rate and small .

Figure 5. Nested -Fold: honest weighted-Ridge vs. across min_completion_rate grid for .

Figure 6. Walk-Forward Expanding: honest weighted-Ridge vs. across grid for .

Figure 7. Walk-Forward Expanding: honest weighted-Ridge vs. across grid for .

Figure 8. Walk-Forward Sliding: honest weighted-Ridge vs. across min_completion_rate grid for .
References
- [1]H. Li et al., “SheetCopilot: Bringing software productivity to the next level through large language models,” NeurIPS, vol. 36, pp. 4952–4984, 2023.arxiv.org/abs/2305.19308
- [2]Y. Chen et al., “SheetAgent: Towards a generalist agent for spreadsheet reasoning and manipulation via large language models,” in WWW, 2025, pp. 158–177.arxiv.org/abs/2403.03636
- [3]Z. Ma et al., “SpreadsheetBench: Towards challenging real world spreadsheet manipulation,” NeurIPS, vol. 37, pp. 94 871–94 908, 2024.arxiv.org/abs/2406.14991
- [4]C. E. Jimenez et al., “SWE-bench: Can language models resolve real-world GitHub issues?” in ICLR, 2024.arxiv.org/abs/2310.06770
- [5]X. Liu et al., “AgentBench: Evaluating LLMs as agents,” in ICLR, 2024.arxiv.org/abs/2308.03688
- [6]R. R. Panko, “What we know about spreadsheet errors,” JOUEC, vol. 10, no. 2, pp. 15–21, 1998.DOI
- [7]F. Hermans et al., “Detecting and visualizing inter-worksheet smells in spreadsheets,” in ICSE. IEEE, 2012, pp. 441–451.DOI
- [8]A. Aleti, “Software testing of generative ai systems: Challenges and opportunities,” in ICSE-FoSE. IEEE, 2023, pp. 4–14.arxiv.org/abs/2309.03554
- [9]F. Dobslaw et al., “Challenges in testing large language model based software: A faceted taxonomy,” arXiv:2503.00481, 2025.arxiv.org/abs/2503.00481
- [10]A. Memon et al., “Taming Google-scale continuous testing,” in ICSE-SEIP. IEEE, 2017, pp. 233–242.DOI
- [11]K. Herzig et al., “The art of testing less without sacrificing quality,” in ICSE. IEEE, 2015, pp. 483–493.
- [12]S. Yoo et al., “Regression testing minimization, selection and prioritization: A survey,” STVR, vol. 22, no. 2, pp. 67–120, 2012.
- [13]Z. Li et al., “MiMoTable: A multi-scale spreadsheet benchmark with meta operations for table reasoning,” arXiv:2412.11711, 2024.arxiv.org/abs/2412.11711
- [14]R. C. Nickerson et al., “A method for taxonomy development and its application in information systems,” Eur. J. Inf. Syst., vol. 22, no. 3, pp. 336–359, 2013.DOI
- [15]B. Beizer, Software testing techniques. dreamtech Press, 2003.
- [16]F. Elberzhager et al., “Reducing test effort: A systematic mapping study on existing approaches,” Inf. Softw. Technol., vol. 54, no. 10, pp. 1092–1106, 2012.DOI
- [17]H. Hemmati et al., “Prioritizing manual test cases in rapid release environments,” STVR, vol. 27, no. 6, p. e1609, 2017.DOI
- [18]R. Cheng et al., “Revisiting test-case prioritization on long-running test suites,” in ISSTA, 2024, pp. 615–627.DOI
- [19]H.-Y. Hsu et al., “Mints: A general framework and tool for supporting test suite minimization,” in ICSE, 2009, pp. 419–429.DOI
- [20]R. Pan et al., “Requirements coverage-guided minimization for natural language test cases,” arXiv:2505.20004, 2025.DOI
- [21]Y. Li et al., “Prioritizing test cases for deep learning-based video classifiers,” EMSE, vol. 29, no. 5, p. 111, 2024.DOI
- [22]X. Dang et al., “Test input prioritization for machine learning classifiers,” IEEE TSE, vol. 50, no. 3, pp. 413–442, 2024.DOI
- [23]M. Openja et al., “An empirical study of testing machine learning in the wild,” ACM TOSEM, vol. 34, no. 1, pp. 1–63, 2024.arxiv.org/abs/2312.12604
- [24]F. M. Polo et al., “tinybenchmarks: evaluating llms with fewer examples,” arXiv:2402.14992, 2024.arxiv.org/abs/2402.14992
- [25]R. Vivek et al., “Anchor points: Benchmarking models with much fewer examples,” in EACL, 2024, pp. 1576–1601.arxiv.org/abs/2309.08638
- [26]H. Zhao et al., “Bento: Benchmark task reduction with in-context transferability,” arXiv:2410.13804, 2024.arxiv.org/abs/2410.13804
- [27]P. Yuan et al., “Beyond one-size-fits-all: Tailored benchmarks for efficient evaluation,” in ACL, 2025.arxiv.org/abs/2502.13576
- [28]Y. Perlitz et al., “Efficient benchmarking (of language models),” in NAACL-HLT, 2024, pp. 2519–2536.
- [29]Q. Ye et al., “How predictable are large language model capabilities? a case study on big-bench,” in EMNLP, 2023, pp. 7493–7517.
- [30]Q. Zhao et al., “Can we predict performance of large models across vision-language tasks?” arXiv:2410.10112, 2024.arxiv.org/abs/2410.10112
- [31]L. Cai et al., “Item response theory,” Annu. Rev. Stat. Appl., vol. 3, no. 1, pp. 297–321, 2016.
- [32]Y. Li et al., “Active evaluation acquisition for efficient llm benchmarking,” arXiv:2410.09592, 2024.
- [33]Y. Yin et al., “Decentralized arena: Towards democratic and scalable automatic evaluation of language models,” arXiv:2505.12808, 2025.arxiv.org/abs/2505.12808
- [34]A. Chatterjee et al., “Posix: A prompt sensitivity index for large language models,” in EMNLP, 2024.
- [35]F. Maia Polo et al., “Efficient multi-prompt evaluation of llms,” in NeurIPS, vol. 37, 2024, pp. 22 483–22 512.arxiv.org/abs/2405.17202
- [36]L. Pacchiardi et al., “100 instances is all you need: predicting the success of a new llm on unseen data by testing on a few instances,” arXiv:2409.03563, 2024.arxiv.org/abs/2409.03563
- [37]G. Zhang et al., “How benchmark prediction from fewer data misses the mark,” arXiv:2506.07673, 2025.arxiv.org/abs/2506.07673
- [38]S. Zhou et al., “WebArena: A realistic web environment for building autonomous agents,” in ICLR, 2024.arxiv.org/abs/2307.13854
- [39]T. Xie et al., “OSWorld: Benchmarking multimodal agents for open-ended tasks in real computer environments,” in NeurIPS Datasets & Benchmarks, 2024.arxiv.org/abs/2404.07972
- [40]G. Mialon et al., “GAIA: A benchmark for general AI assistants,” in ICLR, 2024.arxiv.org/abs/2311.12983
- [41]Y. Qin et al., “ToolLLM: Facilitating large language models to master 16000+ real-world APIs,” in ICLR, 2024.arxiv.org/abs/2307.16789
- [42]J. Payan et al., “InstructExcel: A benchmark for natural language instruction in Excel,” arXiv:2310.14495, 2023.
- [43]Z. Li et al., “Structural coverage criteria for neural networks could be misleading,” in ICSE-NIER, 2019, pp. 89–92.DOI
- [44]Z. Aghababaeyan et al., “Black-box testing of deep neural networks through test case diversity,” IEEE TSE, vol. 49, no. 5, pp. 3182–3204, 2023.DOI
- [45]S. Gerasimou et al., “Importance-driven deep learning system testing,” in ICSE, 2020, pp. 702–713.
- [46]K. Pei et al., “DeepXplore: Automated whitebox testing of deep learning systems,” in SOSP, 2017, pp. 1–18.
- [47]J. Kim et al., “Guiding deep learning system testing using surprise adequacy,” in ICSE, 2019, pp. 1039–1049.DOI
- [48]T. Y. Chen et al., “Adaptive random testing: The art of test case diversity,” JSS, vol. 83, no. 1, pp. 60–66, 2010.DOI
- [49]H. Hemmati et al., “Achieving scalable model-based testing through test case diversity,” ACM TOSEM, vol. 22, no. 1, pp. 1–42, 2013.DOI
- [50]R. Feldt et al., “Test set diameter: Quantifying the diversity of sets of test cases,” in ICST, 2016.DOI
- [51]C. Henard et al., “Comparing white-box and black-box test prioritization,” in ICSE, 2016, pp. 523–534.DOI
- [52]T. Zhodinasab et al., “DeepHyperion: exploring the feature space of deep learning-based systems through illumination search,” in ISSTA, 2021, pp. 79–90.
- [53]H. Fahmy et al., “Supporting deep neural network safety analysis and retraining through heatmap-based unsupervised learning,” IEEE Trans. Reliab., vol. 70, no. 4, pp. 1641–1657, 2021.DOI
- [54]D. R. Kuhn et al., “Software fault interactions and implications for software testing,” IEEE TSE, vol. 30, no. 6, pp. 418–421, 2004.DOI
- [55]Z. Aghababaeyan et al., “DeepGD: A multi-objective black-box test selection approach for deep neural networks,” ACM TOSEM, vol. 33, no. 6, pp. 1–29, 2024.DOI
- [56]Z. Li et al., “Distance-aware test input selection for deep neural networks,” in ISSTA, 2024, pp. 248–260.DOI
- [57]M. Mahdieh et al., “Test case prioritization using test case diversification and fault-proneness estimations,” Autom. Softw. Eng., vol. 29, no. 2, p. 50, 2022.DOI
- [58]I. T. Elgendy et al., “A systematic mapping study of the metrics, uses and subjects of diversity-based testing techniques,” STVR, vol. 35, no. 2, p. e1914, 2025.DOI
- [59]N. Neelofar et al., “Towards reliable AI: Adequacy metrics for ensuring the quality of system-level testing of autonomous vehicles,” in ICSE, 2024, pp. 1–12.arxiv.org/abs/2311.08049
- [60]S. S. Rajan et al., “Knowledge-based consistency testing of large language models,” arXiv:2407.12830, 2024.arxiv.org/abs/2407.12830
- [61]A. Marchetto et al., “An empirical validation of a web fault taxonomy and its usage for web testing,” J. Web Eng., pp. 331–345, 2009.DOI
- [62]H. K. Rajaram et al., “Taxonomy-based testing and validation of a new defect classification for health software,” J. Softw. Evol. Process, vol. 31, no. 1, p. e1985, 2019.DOI
- [63]L. Cui et al., “NLCTables: A dataset for marrying natural language conditions with table discovery,” in SIGIR, 2025, pp. 3638–3647.arxiv.org/abs/2504.15849
- [64]X. Wu et al., “TableBench: A comprehensive and complex benchmark for table question answering,” in AAAI, vol. 39, no. 24, 2025, pp. 25497–25506.
- [65]Y. Chen et al., “SheetAgent: A generalist agent for spreadsheet reasoning and manipulation via large language models,” in LLMs and Cognition, 2024.arxiv.org/abs/2403.03636
- [66]G. Eggert et al., “TabLib: A dataset of 627m tables with context,” arXiv:2310.07875, 2023.arxiv.org/abs/2310.07875
- [67]P. Wu et al., “RealHiTBench: A comprehensive realistic hierarchical table benchmark for evaluating LLM-based table analysis,” arXiv:2506.13405, 2025.arxiv.org/abs/2506.13405
- [68]Y. Sui et al., “Table meets LLM: Can large language models understand structured table data? a benchmark and empirical study,” in WSDM, 2024, pp. 645–654.
- [69]W. Lu et al., “Large language model for table processing: A survey,” Front. Comput. Sci., vol. 19, no. 2, 2025.DOI
- [70]A. Singha et al., “Tabular representation, noisy operators, and impacts on table structure understanding tasks in LLMs,” arXiv:2310.10358, 2023.arxiv.org/abs/2310.10358
- [71]P. Li et al., “Table-GPT: Table fine-tuned GPT for diverse table tasks,” PACMMOD, vol. 2, no. 3, pp. 1–28, 2024.DOI
- [72]J. Wu et al., “Mmqa: Evaluating LLMs with multi-table multi-hop complex questions,” in ICLR, 2025, p. 1.
- [73]X. He et al., “Anameta: A table understanding dataset of field metadata knowledge shared by multi-dimensional data analysis tasks,” arXiv:2209.00946, 2022.arxiv.org/abs/2209.00946
- [74]M. Hulsbos et al., “GitTables: A large-scale corpus of relational tables,” PACMMOD, vol. 1, no. 1, pp. 1–17, 2023.
- [75]S. Ashury-Tahan et al., “The mighty ToRR: A benchmark for table reasoning and robustness,” arXiv:2502.19412, 2025.
- [76]T. Zhang et al., “TableLlama: Towards open large generalist models for tables,” arXiv:2311.09206, 2023.arxiv.org/abs/2311.09206
- [77]X. Zhang et al., “TableLLM: Enabling tabular data manipulation by LLMs in real office usage scenarios,” arXiv:2403.19318, 2024.arxiv.org/abs/2403.19318
- [78]J. Xing et al., “MMTU: A massive multi-task table understanding and reasoning benchmark,” arXiv:2506.05587, 2025.arxiv.org/abs/2506.05587
- [79]D. Yi et al., “TablePilot: Recommending human-preferred tabular data analysis with large language models,” arXiv:2503.13262, 2025.arxiv.org/abs/2503.13262
- [80]S. Thorne, “Large language models for spreadsheets: Benchmarking progress and evaluating performance with FLARE,” in EuSpRIG, 2025.arxiv.org/abs/2506.17330
- [81]C. Li et al., “TReB: A comprehensive benchmark for evaluating table reasoning capabilities of large language models,” arXiv:2506.18421, 2025.DOI
- [82]M. L. Lupetti et al., “(Un)making AI magic: A design taxonomy,” in CHI, 2024, pp. 1–21.DOI
- [83]G. C. Cawley et al., “On over-fitting in model selection and subsequent selection bias in performance evaluation,” JMLR, vol. 11, pp. 2079–2107, 2010.DOI
- [84]M. Naeem et al., “Thematic analysis and artificial intelligence: A step-by-step process for using ChatGPT in thematic analysis,” Int. J. Qual. Methods, vol. 24, 2025.DOI
- [85]V. Viswanathan et al., “Large language models enable few-shot clustering,” TACL, vol. 12, pp. 321–333, 2024.
