betaFeedback

WorksComputer Science

A Taxonomy and Taxonomy-Guided Benchmark Reduction for Spreadsheet Tasks

Abstract

Spreadsheets let millions of end users without programming training do computational work, and teams are increasingly building AI agents to help them. Evaluating each candidate agent configuration across many tasks is costly. Benchmark reduction can lower this cost by selecting a small subset of tasks whose outcomes predict full-benchmark scores, but selecting tasks only for predictive accuracy can leave whole categories of spreadsheet work untested. We contribute a taxonomy of spreadsheet tasks, grounded in a review of benchmarks and refined with the Excel Agent product team at Microsoft, and a benchmark-reduction method that favors taxonomic diversity among similarly predictive choices. On benchmark logs from Microsoft, taxonomy-guided selection broadens coverage at little or no cost in predictive accuracy and often improves prediction on configurations excluded from selection. Broader coverage can therefore support, rather than compromise, predictive accuracy. Generalization remains limited across time: subsets selected using little recent history predict poorly for configurations much later in time. Beyond benchmark reduction, the taxonomy provides a reusable framework for auditing gaps in task coverage, as teams at Microsoft have done.

Introduction

Major technology companies and startups are building AI-powered spreadsheet agents: systems that let users express intent in natural language and autonomously carry out the corresponding spreadsheet operations. Academic research on such agents is growing [1, 2, 3], mirroring the broader trend toward coding agents in software engineering and data work [4, 5], in this case in an end-user programming environment. If effective, these agents can democratize complex data operations in a domain where errors are notoriously prevalent [6, 7].

Ensuring the quality of these agents is challenging. Given a goal such as “clean this sales export and build a month-over-month revenue summary,” an agent plans and carries out a sequence of operations—inserting formulas, creating pivot tables, writing and running scripts—over a live workbook. A task rarely has a single correct answer (“total revenue by region” can be a pivot table, a SUMIF formula, or hand-entered values), and correctness can hinge on workbook state the grid does not expose, such as formulas and cross-sheet references; with natural-language ambiguity and model nondeterminism, ground truth is hard to establish [8, 9]. Evaluation is also expensive. Running the agent on a task—a workbook state paired with a natural-language instruction—consumes tokens and time, and the cost multiplies because the evaluation target is not the LLM alone but a configuration: a combination of LLM, system prompts, tool interfaces, execution harness, and guardrails (validity or safety checks that can block or rewrite an action), any of which can affect outcomes, and all of which teams iterate on routinely. As a concrete illustration of this cost, after integrating a new frontier model into Microsoft’s Excel Agent, we estimated that a single full-benchmark run took about four hours and cost roughly $1,000 in tokens at retail prices. Exhaustive evaluation of every candidate configuration is impractical in a typical CI/CD setting [10, 11], which demands principled test minimization [12]: selecting a reduced benchmark, a small subset of tasks that preserves the predictive accuracy of the full benchmark—how well the subset’s outcomes recover the decision-relevant quantities the full benchmark would report (e.g., pass rates, configuration rankings).

A subset chosen on purely predictive grounds, however, may concentrate in a narrow region of the task space, dominated by a few similar task types while whole categories of behavior go untested. While a few ad-hoc task categorizations exist [1, 2, 13], such coverage has not been defined for spreadsheet AI agents. This gap is our starting point.

In this paper we develop a taxonomy of spreadsheet tasks covering properties of the spreadsheet and natural language instruction. We build the taxonomy following the iterative method of Nickerson et al. [14], grounding it in a literature review of spreadsheet benchmarks and refining it through product team feedback at Microsoft, where Excel Agent is being built and improved. The taxonomy provides an externally observable structure for describing what a benchmark covers, selecting reduced benchmarks, and interpreting failures. We do not claim a single “right” taxonomy—that would overfit current benchmarks and agents—but a reusable framework that can be adapted as the domain evolves [15].

We then use the taxonomy to guide minimization. Treating tasks as features and configurations as data points, we use greedy forward selection with regularized regression to identify a predictive subset; when candidates are near-tied on predictive accuracy, a taxonomy-aware tie-breaker selects the one that maximizes diversity. On real logs from Microsoft, when we evaluate the reduced benchmark on configurations not used to select it (honest validation), selecting for coverage costs little predictive accuracy and often improves it: diversity behaves less as a trade-off against accuracy than as a regularizer over greedy selection—giving up a little fit on the selection data for better prediction on configurations not yet seen—strongest when data is noisy and the task budget is small. The reduced benchmarks do not, however, transfer to configurations far later in time when trained on little recent history, which limits where taxonomy-guided minimization can be used. Matched-budget controls—uniform random, taxonomy-stratified random, diversity-only, and random choice within the same tolerance band—show that the coverage gain comes from the taxonomy rather than from relaxing the greedy objective alone, and that a predictive stage remains necessary. Repeating the analysis on the subset of the logs with no missing outcomes, where every configuration is scored on one common reference set, reproduces the pattern, and taxonomy guidance again beats matched random choice within the band under all three validation strategies, so the result is not an artifact of imputation.

Concretely, our contributions are:

  • A taxonomy of spreadsheet tasks for evaluating AI agents, synthesized from the literature and industrial practice (§ 3).

  • A taxonomy-guided benchmark reduction methodology that selects a predictive reduced benchmark and breaks near-ties by taxonomic diversity. On real industrial logs—the recorded outcomes of past evaluations, i.e., which tasks passed or failed under each configuration—this tie-breaker widens taxonomy coverage with little or no loss of predictive accuracy and acts as a regularizer, as matched-budget controls and a complete-case replication confirm (§ 3-C and § 4).

  • Experience applying the taxonomy at Microsoft to audit coverage gaps between benchmarks and production traffic, and articulating the method’s practical limits (§ 6).

Related Work

We review three lines of work: methods for reducing evaluation cost, from software-testing minimization to LLM benchmark subset selection; agent and spreadsheet benchmarks; and how researchers define coverage for black-box systems through diversity and taxonomies.

Efficient Test and Benchmark Reduction

When rerunning a full test suite on every change is too costly, software teams cut the cost by selecting, minimizing, or prioritizing tests—running a representative subset, dropping redundant tests, or running the most informative tests first [12, 16, 17]—a pressure that long-running CI pipelines intensify [18]. To decide which tests are worth keeping, classical methods measure coverage: how much of the program’s internals—its statements, or artificially injected faults—the tests exercise [19]. Such code-level coverage is unavailable when a “test” is a natural-language task run against a black-box system, so an externally observable artifact—such as the requirement a test relates to—can serve as the selection signal instead [20].

The same problem arises in ML-system testing [21, 22, 23] and LLM evaluation, where the adequacy criterion shifts from code coverage to preserving aggregate scores or rankings. A growing family of methods selects small static subsets (coresets) that approximate full-suite scores or preserve model rankings [24, 25, 26], though the best subset can depend on whether it is model-agnostic or model-aware [27, 28].

Other methods instead exploit learnable structure in model×\timestask performance: Ye et al. [29] search for a “small-bench” via LLM capability predictability; Zhao et al. [30] frame evaluation as matrix completion; Item Response Theory estimates remaining outcomes from partial observations by modeling item difficulty and discrimination [31, 24, 32]; and Active Evaluation Acquisition [32] and Decentralized Arena [33] choose what to evaluate next from observed results. The matrix-completion and active-evaluation methods spend an observation budget: they choose which missing configuration–task outcomes to acquire or estimate next. We instead fix a reusable task suite under a task budget, so the two settings are similar in spirit but not directly comparable; the adaptive methods also assume a missingness pattern that production logs do not have (§ 4-A). We do not compare against them and claim no superiority over them.

These subsets can be fragile: prompt sensitivity [34, 35] and distribution shift [36, 37] can alter which items are informative, motivating selection criteria that do not rely solely on historical performance.

Our problem is closest to static subset selection under a fixed budget: we choose which spreadsheet tasks to run, to preserve both decision-relevant aggregates (pass rate, configuration rankings) and structural breadth. Our pure predictive greedy selector (ϵ=0\epsilon=0, § 4) belongs to this family and represents it in our experiments; we did not run the published instantiations because they fit on dense model×\timestask matrices and estimate one model’s score rather than rankings across configurations.

AI Agent and Spreadsheet Benchmarks

Agent benchmarks span code (SWE-bench [4], AgentBench [5]), web and desktop (WebArena [38], OSWorld [39]), multi-step reasoning (GAIA [40]), and API tool use (ToolLLM [41]), but none addresses selecting a representative subset when full evaluation is too costly, nor offers a coverage framework for minimization.

For spreadsheet agents specifically, SheetCopilot [1] defines six task categories, SheetAgent [2] introduces five broad types with 36 subtypes paired with reasoning challenges, SpreadsheetBench [3] collects 912 real forum questions, MiMoTable [13] categorizes tasks by meta-operation, and InstructExcel [42] provides crowdsourced natural-language instructions over real Excel files. Each defines its own ad hoc task categories; none amounts to a systematic taxonomy designed for coverage-oriented benchmark reduction. Black-Box Coverage via Diversity and Taxonomies: When source-level coverage is unavailable—as in our black-box setting, and in DNN testing, where neuron-coverage metrics correlate weakly with fault detection [43, 44, 45, 46, 47]—a common alternative is diversity: selecting dissimilar test cases to broaden exploration of the fault space [48, 17, 49, 50], which is competitive with white-box techniques when source-level information is inaccessible [51]. Diversity often correlates positively with fault detection [52, 44, 53], is grounded theoretically by the observation that most faults stem from interactions of few parameters [54], is operationalized by multi-objective and distance-aware methods in black-box DNN testing [55, 56], and has been combined with fault-proneness estimates to prioritize tests [57]; Elgendy et al. [58] map diversity-based testing.

Diversity, however, raises the question: with respect to what? A common answer is a taxonomy—a set of dimensions serving as an operational coverage scheme. Autonomous-vehicle testing derives scenario taxonomies via clustering or driving-dimension characterization (e.g., [59]); LLM testing explores model knowledge boundaries through knowledge graphs [60]; and web testing uses defect taxonomies to build smaller, more effective suites [61, 62].

Knowledge Gap and Problem Framing

Classical minimization defines test adequacy through structural coverage—statement or mutation coverage—which cannot be measured for an LLM-based agent. Coreset methods for LLM benchmarks select on historical performance correlations, with no notion of which behaviors a subset exercises, and turn fragile once those correlations break down under prompt sensitivity or drift over time. Diversity is the standard black-box substitute for coverage, but it presupposes dimensions along which to be diverse—and existing spreadsheet and agent benchmarks supply only ad-hoc task categories, not a systematic taxonomy built for coverage-oriented selection. Pairing a domain-grounded taxonomy with a predictive subset-selection procedure is our response—but it requires a specific framing of the selection problem.

Why a Reduced Benchmark: Production agent teams continuously compare candidate configurations across a large design space: cheaper or faster LLMs that may underperform slightly but win on cost or latency; fine-tuned variants whose quality is uncertain until measured; and harness-only changes that keep the model fixed—alternative spreadsheet representations, tool schemas, retrieval settings, safety policies, or new tool integrations. Each needs a full-benchmark score before it can be compared with the current default, yet a typical CI/CD cycle exercises thousands of tasks under rate limits. A reduced benchmark—a small, fixed subset whose outcomes approximate the full-benchmark score for any new configuration—trades a one-time curation cost for a recurring saving.

Why this works: The framing is well-posed when configurations vary along a few latent capabilities and tasks are redundant within them: a small set of informative tasks with learned weights can then recover the full-benchmark score (App. B gives a stylized example).

Tasks as Features, Configurations as Datapoints: This structure—a few latent capabilities with many redundant tasks—suggests the general framing. A uniform random sample of tasks would suffice only if every task correlated equally well with the full-benchmark score, but tasks are heterogeneous—they probe different operations and difficulty levels—so a small random sample can over-represent one region and predict poorly. We therefore choose tasks for how well they predict that score, casting the choice as supervised learning: tasks are features and configurations are datapoints. Each past configuration has been run against many tasks, giving a vector of pass/fail outcomes whose average—its full-benchmark score—is the label, and selecting a reduced benchmark is feature selection.

This requires a pool of past configurations with known full-benchmark scores, which may seem self-defeating if running the full benchmark is too costly. The resolution is temporal: teams do run the full benchmark periodically—to certify releases, set baselines, or vet major model upgrades—and these runs accumulate over a project’s lifetime. Whether the backlog is large and varied enough to support selection is an empirical question we examine using logs from Microsoft (Section V).

These logs are also the right kind of training data: they do not require new data collection and arise from realistic axes of variation—model swaps, prompt edits, retrieval changes. What we extract from them is a property of the tasks, not of any single configuration: which tasks are informative about aggregate behavior and which are redundant. A configuration that will never be deployed is therefore still useful training data, and the signal depends only on relative differences across configurations, easier to estimate than any absolute score. When extrapolation across model generations is a concern, the training pool can be restricted to a recent slice.

In the dominant coreset framing, one picks a subset to estimate one fixed model’s score; we instead pick a subset to compare and rank many candidate configurations.

From framing to method: This framing imposes two requirements. First, predictive accuracy alone is not enough: a reduced benchmark chosen purely to maximize R2R^2—the fraction of variance in the full-benchmark score that a reduced benchmark’s predictions explain—can cluster in narrow regions of the task space, so we need a coverage criterion meaningful without source-level signals—a domain-grounded taxonomy answering “with respect to what” (Section IV). Second, we need a predictive selection algorithm that picks a small, weighted benchmark under that taxonomy, whose design decisions we test empirically on real logs (Section V).

The Taxonomy of Spreadsheet Tasks

This section presents a taxonomy of externally observable properties of the workbook and the task instruction—the dimensions over which we operationalize coverage. We describe first how we identified its dimensions (§ 3-A) and then the result (§ 3-B).

A workbook contains one or more spreadsheets (sheets, or “tabs”), each a two-dimensional grid of cells, and a spreadsheet contains one or more tables—contiguous regions holding a coherent unit of data (e.g., a header row with its records). We present the dimensions primarily at the table level, the finest granularity at which most are expressed and the natural scope of many tasks (lookup, aggregation, cleaning).1

Methodology for Building our Taxonomy

Building a taxonomy requires choosing both which dimensions of variation to include and, for each, how to measure it—the concrete measure or measures that assign a task a value along that dimension. These decisions are interdependent: the right dimensions emerge only from examining real tasks, yet examining tasks presupposes at least a tentative scheme. We resolve this circularity by following the iterative taxonomy-development method of Nickerson et al. [14], which alternates deductive (top-down) cycles—deriving candidate dimensions from theory and testing them against data—with inductive (bottom-up) cycles—examining data to find dimensions not yet in the scheme—until a set of ending conditions is met.

Nickerson et al. [14] give two kinds of ending condition. The objective ones hold when every object examined has been classified and a full iteration does not add, merge, or split any dimension or value; the subjective ones hold when the taxonomy is concise, robust, comprehensive, extendible, and explanatory. A single guiding criterion governs what the taxonomy includes, and every dimension must serve it.

Instantiation: We apply this method to AI spreadsheet agents. Our objects of interest are spreadsheet tasks, each defined by a workbook state and a natural-language instruction. Our guiding criterion is that every dimension capture an externally observable property of the workbook or instruction—one relevant for describing, selecting, and interpreting agent evaluations—since agent internals are inaccessible in our black-box setting. Both cycle types require concrete tasks, so we need a task corpus to iterate over.

This corpus is a convenience sample drawn from one popular public benchmark, SpreadsheetBench [3], and an internal benchmark under development at Microsoft. Inspecting every public benchmark (§ 1.1) was impractical; combining a public and an internal source lets the taxonomy reflect both the academic view of spreadsheet tasks and the workloads of a production agent.

Identifying dimensions: In deductive (conceptual-to-empirical) cycles we started from theory: drawing on the benchmarks surveyed in § 1.1, we proposed candidate dimensions, applied them to corpus tasks, and revised them when a dimension did not vary (an early file format candidate:

DimensionDescriptionExample Measures [Sources]
Properties of a Table
Table sizeTable grid dimensionsNumber of rows & columns [3, 63, 64, 65, 2]
Vocabulary sizeSize of vocabularyNumber of unique values [66]
Header typeThe structure of the header row/columnA categorical label, e.g., simple (single-row or single-column header) vs complex (e.g., hierarchical) [13, 3, 67]
StructureThe structure of the tableA categorical label, e.g., flat vs hierarchical [68, 67, 69, 3]
Data typesThe types of data contained in the tableA list, e.g., string, numeric, date [66, 63, 69]
Data cleanlinessThe cleanliness of the data in the table (e.g., missing entries, inconsistently formatted values)A categorical label, e.g., clean vs noisy [70, 71]
LanguageThe primary language used in the tableA categorical label, e.g., English [66, 13]
Properties of a Spreadsheet or the Workbook
SizeThe size of the parent (spreadsheet/workbook) (as a set of components (tables/sheets) or the file as a whole.Number of sheets [65, 2, 3, 13, 72]; Number of tables [3, 13, 72]; File size on disk [42, 64]
Structural complexityThe overall structural complexity of the parent.A derived rating that aggregates table-level structure and size signals, e.g., summarized by their principal components [13]
Domain / TopicThe domain or topic of the workbookA categorical label, e.g., architecture, finance, education, accounting [73, 74, 66, 13, 67, 63, 75, 64, 65, 2]; NAICS industry sector [this work]; Workbook primary intent [this work]
Properties of the Task Instructions (and Task As a Whole)
ClarityThe clarity of the natural language instructionsA categorical label, e.g., clear vs ambiguous [65, 2, 69]; Instruction specificity rated on a 1–5 rubric [this work]
OperationsSpecific operations that are part of the instructions, with or without their objectsList of query verb-noun phrases (actions + objects) [65, 2, 3]; List of met operations (e.g., lookup, edit, calculate, compare, visualize, reason) [42, 13, 65, 2]
Task typeThe type of the task to be performedA categorical label, e.g., table Q&A, table manipulation, data analysis [1, 76, 13, 71, 77, 67, 78, 75, 79, 64, 69, 65, 2]; Action type: ask vs. act vs. unknown [this work]; Multiple operationalizations (thematic analysis, capability-, and intent-based) [this work]
Task difficultyTask horizon or complexityNumber of operations/reasoning steps required [65, 2]; A weighted average of the difficulties of the task’s meta operations (e.g., lookup and edit can be considered the easiest, while reasoning can be considered the hardest) [13, 65, 2]
LLM capabilitiesThe capabilities expected of the language model in handling the taskE.g., error detection, symbolic reasoning, domain understanding, numerical abstraction, explanation [80, 75, 81, 69, 65, 2]

Table I. Our taxonomy dimensions and the literature informing them. Each dimension of variation of the workbook (or its components) or of the task instructions has one or more concrete measures.

nearly every task is an .xlsx workbook), was redundant (a per-table “structural complexity” rating that restated Header type, Structure, and Table size, kept only as an aggregate workbook property), or conflated separable axes (a single “type” dimension, split into Task type and the Operations it comprises) [61]. In inductive (empirical-to-conceptual) cycles we started from data, open-coding corpus tasks to find recurring properties not yet captured and grouping them into new or revised dimensions; data cleanliness emerged this way, after we repeatedly found tables with missing values that our theory-derived dimensions did not capture. Open coding also yielded candidate measures and value sets for each dimension. We iterated through both cycle types, with rounds of discussion between two researchers [61, 82], until the taxonomy stabilized and met our ending conditions.

Choosing concrete measures and stopping: For each dimension we then chose one or more concrete measures, using programmatic extraction for objective workbook structure and LLM-assisted coding for interpretive task or instruction properties. We allowed multiple measures when a dimension had several useful measures: for example, Task type can be grounded in instruction themes, required agent capabilities, or workbook intent, each with different tradeoffs. We stopped when every corpus task could be classified by every dimension and the taxonomy satisfied the subjective ending conditions above (concise, robust, comprehensive for our scope, extendible, and explanatory). App. C gives the full coding and termination details.

The Resulting Taxonomy

Table I summarizes the taxonomy: at the table level it captures size, vocabulary, headers, structure, data types, cleanliness, and language; at the spreadsheet/workbook level scale, structural complexity, and domain; and at the task level clarity, operations, task type, difficulty, and required LLM capabilities. Because the table lists only representative measures and values, App. A enumerates the complete value sets and App. D gives SpreadsheetBench examples for each tier.

Validity of the Taxonomy

Our main evaluation—whether the taxonomy enables effective benchmark reduction—is the subject of § 4. First we establish a precondition, descriptive validity: that the dimensions discriminate among tasks, partitioning a real benchmark into substantively different groups. We illustrate this on SpreadsheetBench [3], which we annotated end-to-end following the labeling procedure of § 3-A. Concretely, we compute objective measures such as workbook size from the workbook metadata, and use an LLM (GPT-4.1) for subjective measures such as task type and instruction specificity, prompted with the measure’s definition and value set as a rubric.

One author spot-checked the LLM labels by hand-coding a sample, concentrating on the most subjective dimensions, task type and workbook intent, and a second author reviewed the disagreements. The checks did not reveal any major concerns beyond the subjectivity one expects from these dimensions. Disagreements arose with sparse workbooks, whose few filled cells do not reveal the domain or purpose, and with instructions that combine operations, where more than one task type fits: for example, highlighting each row’s best value is both Comparison, Ranking & Extremes and Visualization, Formatting & Presentation, and the LLM tends to label anything that fits nowhere else as Formulas, Logic & Automation, which may inflate that category’s count in Figure 1(a). In these cases neither the human nor the LLM was clearly right, and a more detailed coding guide would be needed to resolve them. The distributions below show that the dimensions separate tasks into different groups (descriptive validity); they do not show that every label is right, and labels affect selection only through the tie-break (§ 6).

Figure 1. (a) Task type. (b) Steps implied (task horizon). (c) Instruction specificity. (d) Workbook intent (domain). SpreadsheetBench tasks across four taxonomy dimensions: a few categories dominate while many are sparsely populated, so the dimensions discriminate among tasks (descriptive validity) and benchmark coverage of the taxonomy is highly uneven.

Figure 1 shows four of the most interpretable dimensions. Task type (a) is heavy-tailed: Formulas, Logic & Automation alone accounts for over 400 tasks (a count that the catch-all labeling noted above may inflate); Extraction, Lookup, Join & Reference and Filtering & Selection follow in the low hundreds; and Grouping & Categorization and Structure, Layout & Data Organization appear only a handful of times. Figure 1(b) shows steps implied, our proxy for task horizon (task difficulty in § 3-B): unimodal but right-skewed, peaking at three to four steps, with a thin tail of long-horizon tasks demanding seven or more. Instruction specificity (c; the clarity dimension) is concentrated at the high end: nearly all instructions are Well- or Very Well-Specified, so SpreadsheetBench barely probes the underspecified regime. Workbook intent (d; domain) spans more than two dozen domains but clusters in Data Entry & Validation, Financial Management, and Data Analysis, with a long tail (e.g., Risk Assessment, Legal & Contractual) represented by a few tasks each.

Each dimension, then, is populated across multiple, clearly distinguishable levels, so the dimensions discriminate among tasks; catch-all labeling inflates one task-type category, but the remaining task types and the other three dimensions show the same spread. The skew also affects evaluation, even if the largest category is overstated: because a few task types and domains dominate and many are represented by a handful of tasks, uniform random sampling over-represents the head and under-samples the tail, so benchmark size does not guarantee taxonomy coverage—which motivates the taxonomy-aware reduction we study next.

Taxonomy-Driven Benchmark Reduction

We need two things (§ 2): a coverage criterion grounded in the spreadsheet taxonomy (§ 3) and a predictive selector that uses it. This section makes both concrete and evaluates them on real industrial logs (the recorded pass/fail outcomes of past evaluations). Our method is to represent the logs as a configuration-by-task matrix, run a greedy predictive selector over it, and break ties in predictive accuracy by taxonomic diversity. Exact ties are rare, so relaxing “tie” to within ϵ\epsilon of the best predictor widens the band of contenders and increases diversity, but might cost predictive accuracy if the looser contenders are weaker predictors. We answer four research questions:

RQ1 How much predictive accuracy does selecting for taxonomic coverage cost? (§ 4-F)

RQ2 When does coverage help most? (§ 4-G)

RQ3 Does the reduced benchmark survive temporal shift—does it stay predictive as configurations evolve over time? (§ 4-H)

RQ4 Is the benefit attributable to the taxonomy itself, and does it survive when every configuration is scored on one common, fully observed reference task set? (§ 4-§ 1)

We study one design choice experimentally—how hard to push for diversity (ϵ\epsilon)—along with two axes that determine when it helps (the task budget kk and a data-quality filter); how to impute missing entries and weight tasks we decide by reasoned argument below.

Data Representation

We arrange the logs into a configuration-by-task matrix X∈Rn×pX \in\mathbb{R}^{n \times p} (n=∣C∣n = |C| configurations, p=∣T∣p = |T| tasks), with Xc,t=rc,tX_{c,t} = r_{c,t} where observed and missing otherwise. Missingness is common—partial runs across CI cycles, skipped or rerun tasks, inconsistent storage as infrastructure evolves—so XX is sparse and non-uniform. Each row of XX is one recorded benchmark execution; several executions can share a treatment definition (e.g., reruns of one prompt variant), so rows are not fully independent, a point we return to in § 4-E and § 6.

We treat one filter as an experimental lever: each configuration’s observed-outcome rate—the fraction of its tasks actually observed—must meet a threshold

min_completion_rate; lowering it allows more but noisier configurations, and sweeping it probes sensitivity to data noise (§ 4-G).

Predictive Minimization via Greedy Forward Selection

Our prediction target is each configuration’s full-benchmark score ycy_c—its mean outcome over the tasks observed for that configuration, Tc\mathcal{T}_c (App. § 5.4)—collected into y∈Rny \in\mathbb{R}^{n}, which we predict from a small subset of tasks SS (∣S∣=k|S| = k) together with a weighting over them. Because Tc\mathcal{T}_c differs across configurations, this target is configuration-specific: a difference in ycy_c can reflect which tasks were run, and how hard they were, as well as how well the configuration did. We keep it for the principal analysis because it is the target the logs support, and we test in § 4-§ 1 whether the conclusions hold when all configurations share one fully observed task set. We fix the weighting first, since it also scores candidates during selection.

Weighting via Ridge regression: Each selected task has a learned weight, and we predict a configuration’s full-benchmark score as the weighted sum of its outcomes on those tasks. With many candidate tasks, few configurations, and many near-identical tasks, least squares is unstable, so we use Ridge regression, which keeps the weights small and stable through an ℓ2\ell_2 penalty (α=0.01\alpha= 0.01; objective and closed form in App. § 15). We interpret the weights β^(S)\hat{\beta}(S) as a stable recipe for combining task outcomes; they do not explain why individual tasks pass or fail.

Selection via greedy forward selection: Exact subset selection is combinatorial, so we build SS greedily: from S0=∅S_0 = \varnothing, each step adds the task that most improves the Ridge fit, scored by inner cross-validated R2R^2 (5-fold, shuffled, fixed seed), until ∣S∣=k|S| = k or the inner R2R^2 reaches a convergence threshold; this yields a ranked, inspectable list (App. § 16).

Taxonomy-Guided Diversity

Since greedy selection only optimizes predictive accuracy, we use the taxonomy to inject a coverage objective into selection, breaking near-ties by taxonomic diversity.

Taxonomy diversity score: We score how well a reduced benchmark SS spans the taxonomy of § 3-B by the unweighted mean of per-dimension coverage scores,

Div⁡(S)=1∣D∣∑j∈DDiv⁡j(S),(1)\operatorname{Div}(S) = \frac{1}{|\mathcal{D}|} \sum_{j \in\mathcal{D}} \operatorname{Div}_{j}(S), \tag*{(1)}

where D\mathcal{D} is every taxonomy dimension of Table I except data cleanliness (not operationalized), and each Div⁡j(S)∈[0,1]\operatorname{Div}_{j}(S) \in[0,1] measures coverage of dimension jj by a form suited to its measure type—range coverage, normalized entropy, or label coverage (full per-dimension definitions in App. § 17).

Diversity tie-breaker via an ϵ\epsilon-tolerance band: Exact ties in inner cross-validation (CV) R2R^2 are rare, but at each greedy step many candidates are nearly tied. We use these near-ties: among all candidates within ϵ\epsilon of the best R2R^2, we add the one that most increases the diversity score Div⁡(⋅)\operatorname{Div}(\cdot):

Bϵ(St−1)={q∉St−1:RinnerCV2(St−1∪{q})≥Rbest2−ϵ},(2)B_{\epsilon}(S_{t-1}) = \{q \notin S_{t-1} : R^{2}_{\mathrm{innerCV}}(S_{t-1} \cup\{q\}) \ge R^{2}_{\mathrm{best}} - \epsilon\}, \tag*{(2)}
q∗=arg⁡max⁡q∈Bϵ(St−1)Div⁡(St−1∪{q}).(3)q^{*} = \arg\max_{q \in B_{\epsilon}(S_{t-1})} \operatorname{Div}(S_{t-1} \cup\{q\}). \tag*{(3)}

Predictive accuracy stays primary; raising ϵ\epsilon widens the band, trading up to ϵ\epsilon of inner-CV R2R^2 for coverage (§ 4-E, § 4-F).

Measuring Generalization Without Selection Bias

Because greedy selection optimizes an inner cross-validation objective, evaluating the reduced benchmark on the same configurations would be biased by selection. We therefore wrap the selection-and-fit pipeline in an outer validation loop: each outer split holds out configurations for judging, and the rest are used for greedy selection, inner 5-fold CV, and Ridge fitting. This yields an honest R2R^2 free of selection bias in the sense of Cawley and Talbot [83]; for comparison, we also report an optimistic R2R^2 that selects tasks once on all data and therefore leaks selection into the held-out folds. We use three outer strategies—shuffled Nested KK-Fold, Walk-Forward Expanding, and Walk-Forward Sliding—defined in App. § 1. Their hierarchy separates contemporaneous generalization from temporal generalization: the K-Fold/Expanding gap isolates temporal leakage, while the Expanding/Sliding gap isolates the cost of discarding older training data.

Data and Sweep Grid

We evaluate on benchmark logs with configurations filtered to a single production-candidate configuration family, yielding n=67n = 67 configurations (126 execution IDs, since some configurations were executed more than once) and p=869p = 869 candidate tasks after dropping all-missing and zero-variance columns. We sweep the full grid ϵ∈{0,0.01,0.05,0.1,0.2,0.3,0.4,0.5}\epsilon\in\{0, 0.01, 0.05, 0.1, 0.2, 0.3, 0.4, 0.5\}, k∈{7,10,15,20,25,30}k \in\{7, 10, 15, 20, 25, 30\}, and min_completion_rate ∈{1%,5%,10%,15%,20%,25%,30%}\in\{1\%, 5\%, 10\%, 15\%, 20\%, 25\%, 30\%\} (the completion-rate filter defined in § 4-A) across the three outer-loop strategies above, with Ridge (α=0.01\alpha= 0.01) and 5-fold inner CV as defined above throughout. RQ1–RQ3 each take up one design question, drawing on the representative grid cell k=10k = 10, min_completion_rate =25%= 25\% together with the full k×k \times min_completion_rate grids; throughout, honest R2R^2 is the out-of-fold measure defined above and optimistic R2R^2 its selection-biased foil.

Matched-budget controls (RQ4): To isolate what taxonomy guidance contributes, we compare it at the same budget k=10k=10 against five controls: (i) pure predictive greedy selection (ϵ=0\epsilon= 0); (ii) ϵ\epsilon-band random, which forms the same near-tie band BϵB_{\epsilon} but picks a member uniformly at random instead of by diversity—the control that separates the taxonomy’s contribution from relaxing the greedy objective alone; (iii) uniform random selection of kk tasks; (iv) diversity-only selection, which greedily maximizes Div⁡(S)\operatorname{Div}(S) with no predictive stage; and (v) taxonomy-stratified random selection, which draws tasks at random across taxonomy strata. Randomized controls are averaged over three seeds; deterministic methods are run once. We run the controls on two datasets: the broader filtered logs with mean imputation (min_completion_rate = 25\%; 126 execution IDs and 852 nonconstant candidate tasks after filtering, missing outcomes filled by training-fold per-task means), and the complete-case matrix described next.

Complete-case common-reference matrix (RQ4): Because ycy_c is computed over a configuration-specific task set, we also built a complete execution-by-task matrix with no missing cells: 99 execution IDs by 715 common tasks (70,785 observed outcomes), with 595 selection candidates after removing zero-variance tasks. Every target and prediction is thus computed on the same observed reference set, without imputation. The 99 execution IDs correspond to only 46 distinct treatment definitions (internally, flights), the rest being repeated executions, so we present this as a complete-case sensitivity analysis; it is not an independent replication. On this matrix we run the ϵ\epsilon sweep under all three validation strategies at six task budgets, and all five controls at k=10k=10.

RQ1: How much predictive accuracy does selecting for taxonomic coverage cost?

Most directly, we fix a contemporaneous configuration family and ask: if we reduce the full benchmark to kk tasks, how much predictive accuracy do we lose? Figure 2b shows that the optimistic R2R^2—where selection and scoring share one configuration family—stays high and falls only slightly as ϵ\epsilon grows, while Figure 2(c) shows the taxonomy diversity score Div⁡(S)\operatorname{Div}(S) rising steeply and saturating by ϵ≈0.1\epsilon\approx0.1–0.20.2.

Three plots showing honest weighted-Ridge $R^2$, optimistic $R^2$, and taxonomy diversity $\operatorname{Div}(S)$ as functions of $\epsilon$

Figure 2. All at k=10k=10, min_completion_rate = 25%, vs. ϵ\epsilon. (a) Honest weighted-Ridge R2R^2 for the three outer-loop strategies: K-Fold >> Expanding >> Sliding throughout, each curve non-monotone, rising at small-to-moderate ϵ\epsilon before falling. (b) Optimistic R2R^2 (independent of the outer strategy), nearly monotone-decreasing. (c) Taxonomy diversity Div⁡(S)\operatorname{Div}(S), rising from pure greedy selection (ϵ=0\epsilon=0) and saturating by ϵ≈0.1\epsilon\approx0.1–0.20.2.

Together, the two show that near-complete coverage of the task taxonomy costs almost nothing in predictive accuracy over a known configuration family.

The stricter question—how well the reduced benchmark predicts new configurations—is where the honest R2R^2 of Figure 2 enters, and the absolute numbers drop sharply: honest R2R^2 is well below its optimistic counterpart across the sweep, and under the walk-forward strategies it is sometimes negative (§ 4-H). We therefore interpret the honest curves relatively, as a regularization diagnostic rather than a claim of strong prediction on new configurations. Within that interpretation they are informative: the honest curve is frequently non-monotone in ϵ\epsilon, rising to a peak at intermediate ϵ\epsilon before falling, whereas the optimistic R2R^2 decreases almost monotonically. This divergence is a regularization signature: pure greedy selection (ϵ=0\epsilon=0) over-optimizes for training correlations that do not transfer, and a moderate ϵ\epsilon partly corrects it, so under out-of-fold evaluation R2R^2 maximization is not the best selection rule. Table II quantifies the effect at ϵ=0\epsilon=0 vs. ϵ=0.1\epsilon=0.1 in four cells of the k=10k=10 grid: in three of four the diversity tie-breaker improves honest R2R^2, and in the first it turns a negative R2R^2 positive—though several cells remain negative in absolute terms, so the gain is only reduced overfitting.

Scenariomin compl.ϵ=0\epsilon=0ϵ=0.1\epsilon=0.1Δ\Delta honest R2R^2
Nested KK-Fold1%−0.157+0.059+0.22 (neg. →\to pos.)
Nested KK-Fold5%+0.050+0.159+0.11 (≈3×\approx3\times)
Walk-Forward Sliding5%−1.242−0.277+0.97 (4.5× less negative)
Walk-Forward Expanding5%−0.253−0.316−0.06 (slightly worse)

Table II. Honest R2R^2 at ϵ=0\epsilon=0 vs. ϵ=0.1\epsilon=0.1, k=10k=10, across four representative cells. Moderate ϵ\epsilon can reduce selection-bias overfitting.

Past the early band the two curves diverge: coverage saturates (Figure 2(c))—once the band reaches most taxonomy bins, widening it adds little coverage—while honest R2R^2 continues to decline past ϵ≈0.2\epsilon\approx0.2–0.30.3.

A second measure supports this interpretation: the gap between optimistic and honest R2R^2—a direct measure of selection-bias overfitting—is universally positive, largest for the walk-forward methods and smallest for K-Fold, and moderate ϵ\epsilon tends to shrink it.

RQ2: When does coverage help most?

The regularization benefit is not uniform; it concentrates in the regimes where greedy selection is most prone to overfitting. Figure 4–Figure 9 in the appendix show the honest R2R^2 surface over the full k×k \times min_completion_rate grid as a function of ϵ\epsilon, for each validation strategy, and two patterns emerge. First, the benefit is concentrated in the low-min_completion_rate rows—the runs in which we admit configurations even though many of their task outcomes are missing. Noisier configurations make the inner cross-validation scores noisier too, so the tiny R2R^2 gaps that the pure greedy approach chases are mostly noise, and breaking those near-ties by taxonomic diversity helps more—the regime where the non-monotone honest-R2R^2 curve is clearest (the low-min_completion_rate rows of Table II). Second, the benefit is concentrated in the small-kk columns: at k∈{7,10}k \in\{7, 10\} the ϵ\epsilon sweep shows visible gains, while at k∈{20,30}k \in\{20, 30\} the curves flatten and the choice of ϵ\epsilon has much less effect, because the model then has enough features to fit well regardless of which tasks were selected. Because a modest reduced benchmark already captures most of the recoverable signal, the open problem is how to spend a small budget well, which is where the diversity tie-breaker helps most.

K-Fold weighted $R^2$ versus $\epsilon$ across $k$ and min_completion_rate grid

Figure 4. Nested K-Fold: honest weighted-Ridge R2R^2 vs. ϵ\epsilon across k×min_completion_ratek \times\mathrm{min\_completion\_rate} grid for k∈{7,10,15}k \in\{7, 10, 15\}.

Walk-Forward Sliding weighted $R^2$ vs. $\varepsilon$ across $k \times$ min_completion_rate grid

Figure 9. Walk-Forward Sliding: honest weighted-Ridge R2R^2 vs. ε\varepsilon across k×k \times min_completion_rate grid for k∈{20,25,30}k \in\{20, 25, 30\}.

RQ3: Does the reduced benchmark survive temporal shift?

Only partly. Future configurations are a stricter test than interchangeable draws from one population, and the three validation instruments of § 4-D separate the two interpretations. Figure 2 shows the ordering Nested KK-Fold >> Walk-Forward Expanding >> Walk-Forward Sliding holding across the ϵ\epsilon sweep, with Sliding often turning negative. A negative honest R2R^2 means the model predicts worse than the mean ycy_c; it arises because nn is small, the matrix is sparse, Ridge is deliberately linear, and the sliding window forces generalization across substantial temporal shift. The two gaps of § 4-D decompose this difficulty: the K-Fold/Expanding gap measures the benefit of temporal leakage, and the Expanding/Sliding gap the cost of discarding older history.

RQ4: Is the benefit due to the taxonomy, and does it survive a common reference task set?

Yes on both counts on the complete-case matrix, where every configuration is scored on the same 715 tasks; on the broader, imputed logs the coverage gain holds but the predictive advantage over random choice within the band does not hold at every ϵ\epsilon.

Complete-case matrix: The diversity–accuracy pattern of RQ1 reproduces on the zero-missing matrix (Table IV, App. § 20). At k=10k=10, moving from ϵ=0\epsilon=0 to ϵ=0.05\epsilon=0.05 raised the diversity of the reduced benchmark selected on all data from 0.649 to 0.815 and improved honest weighted-Ridge R2R^2 under all three validation strategies. Across the six task budgets, moving from ϵ=0\epsilon=0 to ϵ=0.1\epsilon=0.1 increased diversity in all 18 validation-by-budget comparisons and honest R2R^2 in 13 of the 18. Since every cell is observed and every configuration is scored on the same 715 tasks, the pattern cannot come from imputation or from differences in which tasks ran. Among the controls, taxonomy-guided selection substantially increases diversity relative to pure greedy, uniform random, and ϵ\epsilon-band random selection at the same budget. Diversity-only selection maximizes coverage but predicts poorly, so the predictive stage is necessary. And taxonomy-guided selection outperforms its matched ϵ\epsilon-band-random control—same band, random choice within it—under all three validation strategies at both ϵ=0.05\epsilon=0.05 and ϵ=0.1\epsilon=0.1, so the gain over pure greedy selection comes from choosing within the band by taxonomic diversity.

Methodϵ\epsilonNested R2R^2Expanding R2R^2Sliding R2R^2Diversity
Greedy predictive–0.6890.3790.4720.649
Taxonomy-guided0.100.6930.4820.4790.820
Taxonomy-guided0.050.7310.4930.4960.815
ϵ\epsilon-band random0.100.5690.2770.2250.485
ϵ\epsilon-band random0.050.6260.2750.3240.588
Uniform random–0.5670.3010.3380.490
Diversity-only–0.490-0.0140.0670.820
Taxonomy-stratified–0.6020.2050.2170.665

Table IV. MATCHED-BUDGET COMPARISON ON THE ZERO-MISSING COMMON-REFERENCE RECTANGLE (99 EXECUTION IDS, 715 COMMON TASKS, 595 NONCONSTANT CANDIDATES, k=10k = 10). VALUES ARE HONEST WEIGHTED OUTER-VALIDATION R2R^2. RANDOMIZED METHODS REPORT THE MEAN OVER THREE SEEDS. DIVERSITY IS MEASURED ON THE SUITE SELECTED USING THE FULL COMPLETE-CASE DATASET AND IS AVERAGED ACROSS SEEDS FOR RANDOMIZED METHODS.

Broader, imputed logs: On the broader filtered logs, taxonomy guidance again substantially increases diversity and usually improves honest R2R^2 over matched ϵ\epsilon-band random selection: at ϵ=0.05\epsilon=0.05 it is higher under all three validation strategies. The one exception is Nested KK-Fold at ϵ=0.1\epsilon=0.1, where ϵ\epsilon-band random is marginally higher (0.401 vs. 0.396). The complete-case result is uniform and the imputed result is not, which suggests that the taxonomy’s advantage over random choice within the band depends on ϵ\epsilon, on missingness, and on how well calibrated the inner-CV predictor is; a systematic analysis is future work. Our claim is therefore narrower: taxonomy guidance gives a controllable increase in coverage at little or no cost in honest predictive accuracy, and, on a fully observed common reference set, predicts better than matched random choice within the same band.

Across the four questions, then, the evidence supports the finding anticipated in § 4: the diversity tie-breaker behaves as a regularizer over greedy selection rather than a trade-off—we discuss what this means for practice in § 6.

Discussion

Interpreting the Results

The diversity tie-breaker as a regularizer: Across RQ1–RQ3, the diversity tie-breaker is better understood not as a trade-off but as a regularizer over greedy selection: pure greedy selection (ϵ=0\epsilon=0) over-optimizes training correlations that do not transfer to held-out configurations, and breaking near-ties by taxonomic diversity partly corrects this.

RQ4 attributes the correction to the taxonomy: random choice within the same band does not reproduce the coverage gain, and on the complete-case matrix it predicts worse. Ridge regularizes the learned weights, and the tie-breaker regularizes which tasks greedy selection picks; both give up a little fit on the selection data for better held-out prediction.

When a reduced benchmark can be trusted: The honest-R2R^2 results limit what a reduced benchmark should be used for. For contemporaneous configurations—ablations, hyperparameter variants, release candidates of one base model—it is a sound instrument for summarizing or ranking them, and a taxonomy-spanning one has construct and face validity that a coverage-blind top-kk lacks, resists redundancy, and is harder to game if the full-benchmark aggregate is later reweighted. Across time, we advise caution: the low-rank structure the method assumes holds under the independent and identically distributed (i.i.d.) assumption but not stably as configurations evolve (the Walk-Forward Sliding R2R^2 turns negative).

Task-count reduction: The compression we study is steep in task count: k∈{7,…,30}k \in\{7,\ldots,30\} tasks drawn from roughly a thousand (about 100×100\times at k=10k=10), with the saving recurring on every evaluation while the curation cost is paid once. Task count is not cost, however: tasks differ substantially in runtime and token usage, and the log snapshot we analyze does not include token-usage telemetry or comparable wall-clock measurements. Our claims are therefore about task-count reduction, not measured monetary, token, or runtime savings, which depend on the heterogeneous costs of the selected tasks; a cost-aware selector (a budget in tokens or minutes rather than tasks) and prospective measurement are future work.

Implications

For agent teams: choosing ϵ\epsilon and refreshing: A small tolerance band, ϵ\epsilon of roughly 0.05–0.1, is the operating point we recommend. At a moderate ϵ\epsilon the reduced benchmark covers nearly the whole taxonomy with no loss in honest R2R^2 relative to pure greedy selection, and often a small gain that, as the absolute numbers make clear, reflects reduced overfitting rather than strong prediction on new configurations; beyond ϵ≈0.2\epsilon\approx0.2–0.3 coverage has saturated while honest R2R^2 keeps falling. No single ϵ\epsilon is uniformly optimal: 0.1 worked best on the principal sweep and 0.05 was more reliable on the complete-case matrix, so the regularizer’s strength should be tuned to the noise level and budget at hand, with the strongest case for moderate ϵ\epsilon in the noisy, low-min_completion_rate, small-kk regimes of § 4-G. Because the reduced benchmark does not transfer stably across time (§ 4-H), it should also be refreshed—reselected—when the configuration population drifts, rather than kept as a fixed proxy for future configurations.

For agent teams: weighting the taxonomy: Our diversity metric weights every taxonomy dimension equally, but a team shipping charting features cares more about visualization-related task types and LLM capabilities than about the full range of header types. The metric extends to a relevance-weighted mean, with weights stated by the team or derived from production telemetry (the task types users issue, and especially those where the agent fails or users retry); the change reshapes Div⁡(⋅)\operatorname{Div}(\cdot) but leaves the predictive band BϵB_{\epsilon} untouched, so the regularizer interpretation still holds.

For agent teams: auditing coverage gaps: Apart from selection, the dimensions of § 3 provide a coordinate system for auditing an evaluation portfolio: labeling both existing benchmark tasks and a sample of production traffic along the same dimensions reveals regions of the task space that users exercise but the benchmarks under-cover. Teams at Microsoft used the taxonomy this way, applying it to telemetry and internal benchmarks to identify task types and workbook conditions their benchmarks did not adequately cover. Gap analysis thus identifies which regions need new tasks, while taxonomy-aware selection compresses what is already covered.

For benchmark builders: Our descriptive analysis (§ 3-C) is itself actionable: SpreadsheetBench, one of the most widely used public spreadsheet benchmarks, is dominated by Formulas, Logic & Automation (a plurality the rubric’s catch-all tendency may inflate; § 3-C), almost all Well-Specified, and barely probes regions that are important in practice—data cleaning over noisy workbooks, underspecified or ambiguous instructions (Figure 1). We encourage benchmark builders to populate these regions deliberately and to report coverage along taxonomy dimensions rather than raw task count, so that size is not confused with breadth.

For other agent domains: Our approach is not specific to spreadsheets: agent benchmarks in other domains (coding, web, desktop) likewise define ad-hoc task categories without a taxonomy designed for coverage-oriented selection (§ 1.1). Practitioners there could follow the same construction process [14], label their tasks, and apply taxonomy-aware minimization; validating this is future work.

Limitations

Several caveats temper these results, most rooted in the small, noisy logs. Few, correlated configurations, one source. All three honest-R2R^2 estimators are high-variance (especially Walk-Forward Sliding), and the rows of XX are not independent: several executions can share one treatment (99 executions for 46 treatments in the complete-case matrix), which weakens the i.i.d. assumption behind Nested KK-Fold. All evidence comes from one industrial source; replication on another agent benchmark is needed before generalizing. Missing data and the target. In the principal analysis each ycy_c averages over its own configuration-specific task set and missing cells are mean-imputed, which attenuates the coefficients of heavily-missing columns and may not match the missingness mechanism. The complete-case analysis (RQ4) removes both confounds within its rectangle, but may not generalize to the excluded executions and tasks, and imputation can still affect the exact estimates in the broader analysis. Label reliability. The labels on which Div⁡(S)\operatorname{Div}(S) depends are LLM-assigned and were spot-checked by hand on a sample (§ 3-C); we did not compute a formal agreement statistic, and a stratified two-rater study with per-dimension agreement (κ\kappa) remains future work. Because they are used only in the tie-break, label noise cannot change the predictive band BϵB_{\epsilon}. The honest-R2R^2 results use task outcomes, not labels, so the benefit we measured was obtained despite imperfect labels. The coverage figures, computed from the same labels, are the one quantity noise could overstate. With uninformative labels the tie-breaker would degrade to ϵ\epsilon-band random choice, so beating that control on the complete-case matrix is indirect evidence that the labels are informative. A perturbation analysis of label errors would quantify how much noise the tie-break tolerates. The metric. R2R^2 is unstable when Var⁡(y)\operatorname{Var}(y) is small, so mean absolute error or rank correlation may be more informative; we treat the validation hierarchy as a diagnostic. A more reliable reduced benchmark needs more configurations, a stable target (a fully observed reference set, as in RQ4), less selection freedom (smaller kk, stronger α\alpha), ϵ\epsilon tuned to the noise level, and reporting which tasks recur across outer folds.

Conclusion

Our taxonomy of spreadsheet tasks gives benchmark coverage a concrete, externally observable meaning—the “with respect to what” that ad-hoc benchmark categories lack—and our taxonomy-guided selector turns that meaning into reduced benchmarks. On real logs from Microsoft, selecting for taxonomic coverage costs little predictive accuracy and often gains it, behaving as a regularizer over greedy selection, strongest where data is noisy and budgets are small; matched-budget controls rule out a relaxed greedy objective as the source of the gain, and a complete-case replication rules out imputation. Honest evaluation also marks the limits—poor transfer across a long time gap when trained on little recent history, and a reduction measured in task count, not yet in cost—so the practical recipe is a small, taxonomy-spanning reduced benchmark, refreshed as the configurations drift.

Data Availability: The benchmark logs are confidential and cannot be shared. We will release the analysis code, with confidential identifiers removed, an input schema, run commands, seeds, hyperparameters, and instructions for rerunning it on another configuration-by-task matrix.

Acknowledgments: The authors used AI assistants (GitHub Copilot and Claude Code, on Anthropic’s Claude and OpenAI’s GPT models) to draft portions of the manuscript from our notes and analyses, to write code behind many of the numbers and figures (which we checked), and to copy-edit prose under our direction; LLMs are also a component of the taxonomy-construction method (§ 3-A). All research questions, study design, analysis decisions, and interpretations are our own, and we take full responsibility for the paper’s content.

Complete Taxonomy Value Sets

Table I lists only example measures and values for each dimension. For completeness, we enumerate below the full value set of every close-ended (categorical or multi-label) measure that appears in Table I, grouped by the three tiers of the taxonomy. Numeric measures (e.g., table and workbook size, or task difficulty), which take ranges rather than discrete values, and open-ended measures (e.g., data types, language, and free-text operation phrases), which admit no fixed value set, are therefore not listed. Where a per-value definition is given, it is taken (lightly edited) from the labeling prompt or capability schema used to assign the value; bare enumerations are listed without definitions. These value sets are produced programmatically from the labeling configuration so that they stay in sync with the implementation.

Table Properties

Header type (2 values).: The structure of a table’s header (see also Figure 3).

Illustrative spreadsheet structures

Figure 3. Illustrative spreadsheet structures: (a) flat column-major layout with a simple header; (b) flat row-major layout; (c) non-flat layout with a hierarchical header; (d) two distinct table-like regions on one sheet.

  • Simple: A single header row only.

  • Complex: Multiple header rows or one or more header columns present.

Structure (4 values).: The layout of a table, as detected from its orientation and headers.

  • columnMajorFlat: Flat table with headers in the first row and data in columns below.

  • rowMajorFlat: Flat table with headers in the first column and data in rows to the right.

  • hierarchical: Table with nested or multi-level row or column headers.

  • other: A table structure that does not fit the standard patterns.

Spreadsheet and Workbook Properties

Workbook primary intent (31 values).: The purpose of the workbook (the Domain/Topic dimension), assigned by an LLM from the workbook content.

  • Financial Management: budget, revenue, expenses, financial reporting, cash flow, profit and loss, balance sheet, financial planning, investment, funding sources

  • Project Planning: project management, timeline, task tracking, milestone, resource allocation, Gantt chart, progress monitoring, project schedule

  • Data Analysis: data analysis, statistical analysis, data visualization, summary statistics, pivot table, data aggregation, trend analysis, comparative analysis

  • Regulatory Compliance: compliance, regulatory reporting, audit, legal compliance, certification, documentation, validation, reporting requirements

  • Human Resources: payroll, salary, employee information, attendance, leave management, compensation, benefits, staff, personnel costs

  • Inventory Tracking: inventory, stock management, product catalog, item code, quantity, procurement, supply chain, warehouse

  • Academic Administration: student data, academic year, enrollment, grade, course, subject, teacher, school, educational assessment

  • Healthcare Records: patient data, diagnosis, procedure code, healthcare provider, medical billing, insurance, clinical data, treatment, hospital

  • Event Scheduling: calendar, schedule, event, dates, planning, timetable, session, participant, attendance

  • Construction Management: construction, project cost, bill of quantities, contractor, materials, site location, infrastructure, building type

  • Grant and Funding: grant application, funding request, grant allocation, award, program funding, grant reporting, subrecipient

  • Risk Assessment: risk management, risk assessment, risk register, mitigation, failure analysis, safety, audit, action plan

  • Sales and Marketing: sales, product, customer, order, pricing, marketing, campaign, revenue, sales tracking

  • Community Engagement: community development, stakeholder, outreach, engagement, social impact, volunteerism, public health

  • Sustainability Reporting: sustainability, environmental impact, energy consumption, waste management, greenhouse gas emissions, social responsibility

  • Transportation Data: transportation, transit operations, vehicle types, route, ridership, infrastructure, maintenance, public transport

  • Scientific Research: experimental data, gene expression, sample identification, biological data, measurement, data analysis, research study

  • Legal and Contractual: contract, agreement, legal status, contract type, terms, compliance, documentation, approval

  • Demographic Analysis: population, age group, gender, region, demographic data, census, household, survey

  • Insurance Administration: insurance, premium, coverage, claim, policy, benefit, deductible, plan, provider

  • Real Estate Management: property, address, value, rent, ownership, real estate, property type, appraisal

  • Resource Allocation: resource planning, resource allocation, bandwidth, man-hours, team, assignment, utilization

  • Educational Resources: textbook, curriculum, lesson, teaching materials, resource type, subject area, academic module

  • Data Entry and Validation: data entry, validation, checklist, template, instructions, error checking, data quality

  • Sports Management: team, match, league, player, score, tournament, competition, performance tracking

  • Procurement and Purchasing: procurement, purchasing, vendor, supplier, order, invoice, contract, product catalog

  • Taxation and Accounting: tax, income tax, deduction, ac-

counting, journal entry, financial statement, tax calculation

  • Healthcare Services: healthcare, service type, facility, provider, patient, treatment, care, reimbursement

  • Organizational Directory: organization, contact information, address, phone number, email, staff directory, department

  • Metadata and Structure: worksheet structure, metadata, creator information, modification date, data structure, spreadsheet type

  • Categorization and Taxonomy: taxonomy, classification, category, code mapping, data categorization, thematic analysis, spreadsheet taxonomy

NAICS industry sector (20 values): A complementary Domain/Topic measure: the 2-digit North American Industry Classification System sector inferred from the workbook content.

  • 11 – Agriculture, Forestry, Fishing and Hunting

  • 21 – Mining, Quarrying, and Oil and Gas Extraction

  • 22 – Utilities

  • 23 – Construction

  • 31 – Manufacturing

  • 42 – Wholesale Trade

  • 44 – Retail Trade

  • 48 – Transportation and Warehousing

  • 51 – Information

  • 52 – Finance and Insurance

  • 53 – Real Estate and Rental and Leasing

  • 54 – Professional, Scientific, and Technical Services

  • 55 – Management of Companies and Enterprises

  • 56 – Administrative and Support and Waste Management and Remediation Services

  • 61 – Educational Services

  • 62 – Health Care and Social Assistance

  • 71 – Arts, Entertainment, and Recreation

  • 72 – Accommodation and Food Services

  • 81 – Other Services (except Public Administration)

  • 92 – Public Administration

Task and Instruction Properties

Instruction specificity (5-point scale): The Clarity dimension, rated against a fixed rubric.

  • 1 (Very Ambiguous): Impossible to act on without significant clarification. Lacks any specific target or action.

  • 2 (Ambiguous): Missing critical information. Has multiple plausible but different interpretations.

  • 3 (Reasonably Specified): The main intent is clear, but minor assumptions are needed.

  • 4 (Well-Specified): The query is clear, direct, and contains enough information to be executed with high confidence.

  • 5 (Very Well-Specified): Perfectly clear and unambiguous, likely referencing specific columns, sheets, or values, leaving no room for interpretation.

Meta operation – operation types (35 values): The action component of the Operations dimension; a task may carry several. add, align, allocate, calculate, compare, copy, count, create, disable, display, fill, filter, fix, format, freeze, hide, identify, link, merge, navigate, paste, protect, refresh, remove, rename, replace, resize, retain, save, sort, split, summarize, unmerge, validate, None.

Meta operation – operation objects (44 values): The object each operation acts upon; a task may carry several. aggregation, alignment, border, cell, cell reference, chart, column, comment, connection, dashboard, data, error, filter, footer, format, formula, header, hyperlink, image, macro, Machine Learning Model, named range, number, object, page layout, pane, picture, PivotTable, protection, row, shape, size, slicer, sparkline, style, table, text box, theme, tool, validation rule, value, workbook, worksheet, None.

Task type – action type (3 values): The coarsest operationalization of Task type: whether the instruction asks for information or directs an action.

  • ask: Questions about insights, explanations, or recommendations without an explicit instruction to modify the workbook.

  • act: Directives to create, edit, format, calculate, or otherwise change or produce spreadsheet artefacts.

  • unknown: The request could be interpreted either way or lacks sufficient detail.

Task type – intent themes (12 values): One operationalization of Task type, derived by thematic analysis of the instruction text.

  • Filtering & Selection: Filtering, subsetting, excluding, or deduplicating records/entities based on criteria or thresholds.

  • Aggregation & Summarization: Summing, counting, frequency, totals, breakdowns, subtotals, and high-level numeric summaries.

  • Grouping & Categorization: Grouping, segmenting, classifying, labeling categories or entities for organization or comparative frames.

  • Comparison, Ranking & Extremes: Comparing entities/groups/time periods, ranking/order, identifying max/min/top/bottom/extremes and differences.

  • Statistics, Percentages & Metrics: Averages, medians, ratios, percentages, rates, statistical and correlation measures.

  • Temporal & Trend Analysis: Time-based or sequential analysis: trends, patterns over periods, period-over-period comparisons.

  • Extraction, Lookup, Join & Reference: Retrieving specific values, lookups, joins, merges, matching, referencing or attributing data.

  • Data Cleaning, Quality & Error Handling: Handling missing/invalid data, validation, error/outlier/anomaly detection, deduplication for quality, formula error resolution.

  • Transformation, Feature & Text Engineering: Transforming structure or formats, feature engineering, mapping,

text parsing/splitting/concatenation, numeric/date conversions, sequence generation.

  • Visualization, Formatting & Presentation: Charts, graphs, dashboards, conditional formatting, highlighting, styling, and presentational layout.

  • Formulas, Logic & Automation: Creating/modifying formulas, conditional logic, automation (macros/VBA), dynamic calculation behavior.

  • Structure, Layout & Data Organization: Managing tables, pivot tables, sheets, columns, layout, data movement/placement, consolidation, access/permissions.

LLM capabilities (8 categories, 20 subcategories).: The agent skills a task requires; also serves as the capability-based operationalization of Task type. Listed as category (with its subcategories).

  • Analysis — All forms of data analysis and exploration

    • Analyze Descriptively: Basic Multi-Dimensional Analysis (MDA) operations on tabular entities

    • Analyze Advanced: Sophisticated analytical techniques determined by schema and relationships

  • Visualization — Creating visual representations and interactive data exploration tools

    • Create Charts: Creating, modifying, or analyzing charts and visualizations

    • Create PivotTables: Creating, modifying, or analyzing PivotTables and PivotCharts

  • Data Management — Operations for acquiring, organizing, and maintaining data

    • Import Data: Importing data from files that Excel handles natively

    • Clean Data: Cleaning, organizing, or restructuring messy data

    • Shape Data: Transforming data structure and layout

    • Create Tables: Creating structured tables or converting ranges to tables

    • Validate Data: Implementing data validation rules or checking data quality

    • Style Data: Applying visual formatting and styling to data and worksheets

  • Computation — Mathematical operations and calculations

    • Use Formulas: Creating, modifying, or using Excel formulas and functions

    • Run Python: Using Python within Excel for advanced computations

  • Content Processing — Working with text and non-tabular content

    • Analyze Text: Analyzing, processing, or manipulating text data

    • Summarize Text: Summarizing or condensing text content

    • Translate Text: Translating text between different languages

    • Process Images: Working with images, PDFs, or visual content

  • Integration — Connecting with external systems and collaboration

    • Use M365: Using Microsoft 365 integrations and cloud services

    • Connect External: Connecting to external systems and data sources

    • Collaborate: Adding comments, sharing, or collaborative features

  • Automation — Automating repetitive tasks and workflows

    • Automate Tasks: Creating macros, VBA scripts, or other automation

  • Other — Actions that don’t fit into other specific categories

Toy Reduced-Benchmark Example

Suppose a benchmark of 1,000 spreadsheet tasks, with configurations varying primarily along two latent capabilities: formula reasoning and data cleaning. A configuration strong on formula reasoning but weak on data cleaning passes most formula tasks (say, 600/700) but few data-cleaning tasks (50/300), for an overall pass rate near 0.65; a configuration with the opposite profile reaches a similar rate for opposite reasons. Because pass rate is an average over outcomes, it can in principle be reconstructed from any small set of tasks that captures both factors—here, two well-chosen tasks plus a learned weighting from their outcomes to the full-benchmark score. Conversely, if 200 formula tasks measure the same sub-skill, dropping 199 of them costs almost nothing in predictive accuracy; the survivor represents its cluster.

Table 3 makes this concrete on a stylized 4 × 6 example in which F1–F2 and D1–D3 are perfectly correlated: four of the six tasks are redundant, and the two-task reduced benchmark {F1, D1} reconstructs every configuration’s full-benchmark pass rate exactly.

Formula tasksData-clean. tasksycy_cy^c\hat{y}_c
ConfigF1F2F3D1D2D3(full)({F1, D1})
c1c_1 (F+, D−)1100000.330.33
c2c_2 (F−, D+)0011110.670.67
c3c_3 (F+, D+)1111111.001.00
c4c_4 (F−, D−)0000000.000.00

Table 3. Stylized 4 × 6 instance of the mechanism described in the text: formula tasks F1–F3 and data-cleaning tasks D1–D3, full-benchmark pass rate ycy_c, and its estimate y^c\hat{y}_c from the two-task reduced benchmark {F1, D1} with learned ridge weights β^=(13,23)\hat{\beta} = \left(\frac{1}{3}, \frac{2}{3}\right).

Taxonomy Construction Details

Choosing concrete measures: With a dimension established, we must decide how to measure it: which one or more concrete measures assign a task a value along the dimension, and which values each measure allows. We deliberately allow a single dimension to have more than one measure, because an axis of variation can often be operationalized in different ways with distinct tradeoffs (the Task type dimension below is our clearest example). This extends the single-characteristic-set view of Nickerson et al. [14]—in whose terms a measure’s value set is the dimension’s set of characteristics—and is reflected in the rightmost column of Table I.

Objective (structural) measures can be extracted programmatically from workbook metadata (e.g., cell counts, merged-cell detection, hierarchical structure), without human judgment. Subjective (interpretive) measures require interpreting the task instruction or spreadsheet context, and we operationalized them in one of two ways depending on whether the measure’s value set was emergent or fixed in advance. When the value set had to be discovered from the data, we used LLM-assisted thematic analysis following Naeem et al. [84], who validated their approach against manual thematic analysis. For each task, we gave GPT-4.1 the axis to code along (e.g., task type), the task instruction, and a sample of the workbook data, and the LLM produced initial codes for that task. After coding all corpus tasks this way, we had the LLM group the per-task codes into higher-level themes, which became the measure’s value set. We reviewed these groupings and iterated on the prompts and codebook (e.g., after noticing a few instances of prompt bleeding/context leakage) until the categories stabilized. When the value set was fixed in advance, we instead had the LLM rate each task against a predefined rubric—for instance, scoring instruction specificity on a 1–5 scale—so no theming step was required.

The task type dimension is our clearest case of a dimension admitting several candidate measures. We explored alternatives because it is arguably the most consequential for the downstream evaluation, and because “task type” is not one obvious concept—different operationalizations are study-design alternatives with distinct tradeoffs. We compared four, mirroring the deductive/inductive alternation above. First, we applied bottom-up thematic analysis [84] of the natural-language instructions, producing fine-grained action-oriented categories (e.g., Aggregation & Summarization, Comparison & Ranking, Filtering & Selection, Formulas & Automation); these are well-grounded in the data but may yield more categories than are practical for test selection. Second, we tried LLM-amplified few-shot clustering [85], which is query-efficient—the LLM labels most items and routes uncertain cases near the decision boundary to a human judge—but in our setting it yielded coarser, more function-oriented groups (e.g., Data Analysis, Data Filtering, Formula, Pivot Table) that risk merging meaningfully different tasks. Third, product managers at Microsoft did top-down coding, defining task types by the capabilities the agent requires. This produced an agent-centric hierarchy (e.g., Descriptive Analysis, Computation—Using Formulas, Visualization—Creating PivotTables) that is directly useful for capability-gap analysis but reflects the product team’s framing and may not generalize beyond a specific agent. Finally, we tried coding the workbook’s primary intent rather than the natural-language instruction, which produced purpose-oriented types (e.g., Data Entry & Validation, Financial Management) but can conflate tasks whose instructions differ substantially yet operate on similar workbooks. Thus, even for a single dimension, one can vary both what is coded (the natural language task instruction, the implied workbook intent, or the required agent capabilities) and how (thematic analysis, few-shot clustering, or top-down decomposition), and each choice has different tradeoffs in granularity, generalizability, and practical utility.

We faced a similar choice for Domain/Topic. Coding workbook intent, as above, groups tasks by what the spreadsheet is for, but two tasks with the same purpose (e.g., financial reporting) may involve very different data domains. We therefore added a complementary operationalization that assigns, via LLM labeling of the workbook content, a sector from the North American Industry Classification System (NAICS),4 the standard U.S. federal taxonomy for classifying businesses by industry (e.g., sector 52 Finance and Insurance, sector 44–45 Retail Trade). NAICS sector provides a standardized, reproducible domain label, while workbook intent captures purpose-level variation that sector codes miss; keeping both gives the taxonomy finer-grained coverage of the Domain/Topic space.

Validation and termination: Because the process is iterative, we needed criteria for when to stop. We terminated refinement when the taxonomy satisfied two groups of ending conditions [14]. Objectively: (i) for single-label measures each task receives exactly one value (e.g., header type is either simple or hierarchical, not both); (ii) for multi-label measures a task may receive several values, because a single instruction can involve multiple operations or skills (e.g., a task’s meta operations may include lookup, calculate, and visualize); and (iii) every task in the corpus is classified by every dimension. Subjectively, we required the taxonomy to be concise, robust, comprehensive for our scope, extendible, and explanatory for error analysis [14]. To check these conditions, two researcher co-authors reviewed the full taxonomy, discussed disagreements, merged redundant categories, and clarified definitions, incorporating practitioner feedback from internal presentations and sessions with researchers and product managers at Microsoft to prioritize evaluation-relevant distinctions [82].

Taxonomy Dimension Examples

The examples below use SpreadsheetBench [3], which is public. We refer to tasks by their SpreadsheetBench identifiers (e.g., 13–1). Because the original SpreadsheetBench tasks are phrased as how-to questions to a chatbot, we rephrased them into imperative instructions for an AI agent while keeping all other task details unchanged.

Table properties

The unit of analysis is the workbook, but several dimensions describe the structure of individual tables within it; our SpreadsheetBench annotations record these signals via per-table metadata and sheet-/workbook-level aggregates. The simpler structural measures—table size (rows and columns), vocabulary size (number of unique values), data types (e.g., date, numeric, text), and language (almost always English)—are as defined in Table I; two dimensions warrant illustration. Header type captures whether headers are simple (Figure 3(a)) vs. hierarchical with merged cells (Figure 3(c)). Structure captures table layout: flat column-major means each column is an attribute and each row a record (Figure 3(a)); flat row-major transposes this so that each row is an attribute and each column a record (Figure 3(b)); non-flat covers crosstabs, pivoted layouts, and other arrangements where neither reading direction alone suffices (Figure 3(c)). Finally, data cleanliness captures noise such as duplication or inconsistent formatting.5

Spreadsheet and workbook properties

Size captures the scale of the parent spreadsheet or workbook as a whole—its number of sheets, its number of table-like regions, and its on-disk file size. For example, task 13-1 explicitly spans two sheets (“Combine data from the ‘RANGES’ sheet into the ‘LISTS’ sheet . . . ”) and its workbook contains six table regions (three on each sheet), whereas task 19-7 uses a single sheet with a single table region (Figure 3(d) illustrates a sheet with multiple regions). We include file size to capture scaling effects for the evaluation harness. Structural complexity aggregates these structural signals (e.g., size and layout) into an overall rating. For instance, a workbook with 5208 rows and five columns (task 19-7) likely represents a very different workload than one with 85 rows and nine columns (task 13-1). Domain/Topic captures the subject matter and the primary intent of the workbook (recall § 3-A). For example, the workbook for task 17-35 is a supplier invoice register whose columns record dates, supplier names, tax-identification numbers, invoice numbers, and monetary amounts for pharmaceutical vendors—placing it in NAICS sector 42 Wholesale Trade; its inferred workbook intent is Data Filtering.

Task properties

Clarity captures how well-specified the instruction is, on a scale from 1 to 5: some tasks provide detailed ranges and constraints (e.g., clarity score 5 for task 19-7: “split data from the ‘LASTROW’ sheet into two separate sheets named ‘MINUS’ and ‘PLUS’, based on negative and positive values found in column E”), while others are underspecified. Operations capture the action–object intent signals in the instruction at two levels. At the surface level, query verb–noun phrases record specific actions and their objects, e.g., task 13-1 includes phrases like “delete old ranges” and “add TOTAL row to sum amounts,” while task 19-7 includes “create macro” and “delete existing sheets.” At a more abstract level, meta operations capture composable operation primitives: task 17-35 includes filter (data), validate (cell), and add (column); task 13-1 includes remove (data), merge (data), and add (row); and task 19-7 includes remove (worksheet), split (data), and allocate (cell).

Task type captures the coarse-grained kind of work a task demands. SpreadsheetBench distinguishes explicit questions (ask, e.g., “Explain what is wrong with the VBA code . . .” in task 209-30) from action requests (act, e.g., “Create a macro to split data . . .” in task 19-7), and from cases where the instruction is too vague to determine intent (unknown in task 55224). Task difficulty reflects both the task’s horizon (the number of operations/reasoning steps required) and its operation mix; for example, task 19-7 explicitly bundles deleting sheets, splitting data, placing outputs into specific ranges, and inserting total rows—multi-step macro creation over a 5208 × 5 table—versus smaller single-formula tasks.

LLM capabilities capture the types of agent skills required, such as automation and formula use (e.g., Automation.Automate Tasks and Computation.Use Formulas in task 19-7). Although some overlap exists between Task type and LLM capabilities (e.g., a “data visualization” task type often requires a “charting” capability), they capture different facets: Task type is grounded in properties of the workbook–instruction input—consistent with our criterion of externally observable properties—whereas LLM capabilities reflects the agent-centric view originating from Microsoft’s product managers. Keeping them apart lets the taxonomy distinguish what the user asks for from what the agent must be able to do.

Formalization of the selection task

Let C\mathcal{C} be a set of configurations (e.g., candidate model variants). Let T\mathcal{T} be the set of tasks. For each configuration c∈Cc \in\mathcal{C} and task t∈Tt \in\mathcal{T} we observe an outcome rc,t∈{0,1}r_{c,t} \in\{0,1\} (fail/pass), with missingness allowed when a task was not executed.

We define the full-benchmark score for configuration cc as the mean success across all tasks observed for that configuration:

yc=1∣Tc∣∑t∈Tcrc,t,(4)y_c = \frac{1}{\lvert\mathcal{T}_c \rvert} \sum_{t \in\mathcal{T}_c} r_{c,t}, \tag*{(4)}

where Tc⊆T\mathcal{T}_c \subseteq\mathcal{T} denotes tasks with non-missing outcomes for cc. Note that Tc\mathcal{T}_c is configuration-specific; the complete-case analysis of § 4-§ 1 instead scores every configuration on one common, fully observed T\mathcal{T}.

Goal. Choose a small subset S⊂TS \subset\mathcal{T}, ∣S∣=k|S| = k, such that a score computed from only tasks in SS accurately predicts ycy_c across configurations.

Scope of C\mathcal{C} (which configurations do we care about?). A reasonable concern is that some configurations in historical logs are “old” (e.g., obsolete model generations) and may never be deployed again. We do not assume that all historical configurations are decision-relevant. Instead, C\mathcal{C} should be understood as the set of configurations relevant to a particular evaluation goal (e.g., a current model family, a specific tool stack, or a time window). The predictive objective is then: within that scope, can a small reduced benchmark predict the full-benchmark score well enough to speed up iteration and to compare candidates on the Pareto frontier of quality vs. cost/latency? Operationally, this can be enforced by filtering or weighting configurations by relevance (model family, recency, tool availability), and our scripts already support configuration-family filtering (prefix) and explicit exclusions.

Ridge Regression and Conditioning

With a small number of configurations and many candidate tasks (p≫np \gg n), ordinary least squares (OLS) is ill-conditioned: many tasks are redundant or strongly correlated, so small changes in the observed outcomes (or the particular train/test split) can produce large swings in fitted coefficients. Ridge regression addresses this by adding an ℓ2\ell_2 regularization term that shrinks coefficients toward zero. Intuitively, Ridge prefers explanations of yy that use moderate weights spread across correlated tasks rather than extremely large positive/negative weights that cancel each other. This improves stability (and often generalization) by reducing estimator variance, at the cost of introducing a small bias (a standard bias–variance tradeoff).

In our setting, these properties matter for two practical reasons. First, the design matrix X~\widetilde{X} of (largely binary) task outcomes can make X~⊤X~\widetilde{X}^{\top}\widetilde{X} nearly singular when p>np > n or when many tasks are near-duplicates; Ridge yields a unique, well-behaved solution in this regime. Second, we interpret the resulting coefficients as an importance weighting over the selected tasks, producing a weighted proxy score that is less sensitive to noise than an unregularized fit.

Objective.: Given a candidate set of columns SS, we fit

β^(S)=arg⁡min⁡β ∥y−X~Sβ∥22+α∥β∥22,(5)\widehat{\beta}(S) = \underset{\beta}{\arg\min}\ \lVert y-\widetilde{X}_S\beta\rVert_2^2+\alpha\lVert\beta\rVert_2^2, \tag*{(5)}

where X~S\widetilde{X}_S denotes the imputed matrix restricted to columns in SS and α>0\alpha> 0 is a regularization constant. The penalty α∥β∥22\alpha\lVert\beta\rVert_2^2 yields the closed-form solution β^=(X~S⊤X~S+αI)−1X~S⊤y\widehat{\beta} = (\widetilde{X}_S^{\top}\widetilde{X}_S+\alpha I)^{-1}\widetilde{X}_S^{\top}y, which is well-defined even when X~S\widetilde{X}_S is singular. Increasing α\alpha makes the fit less sensitive to idiosyncratic patterns in small datasets (more shrinkage), while decreasing it approaches OLS (less shrinkage, higher variance); we use α=0.01\alpha= 0.01 throughout.

Greedy Forward Selection Procedure

Selecting the best subset of kk tasks is combinatorial, so we build SS with greedy forward selection. Starting from S0=∅S_0 = \varnothing, at each step tt we evaluate every remaining candidate task qq by fitting Ridge on St−1∪{q}S_{t-1} \cup\{q\} under 5-fold inner cross-validation (shuffled KK-fold, fixed seed) and computing the cross-validated (CV) R2R^2. The candidate that yields the highest inner CV R2R^2 is added to StS_t:

q∗=arg⁡max⁡q∈Q∖St−1 RinnerCV2(St−1∪{q}).(6)q^* = \underset{q\in\mathcal{Q}\setminus S_{t-1}}{\arg\max}\ R^2_{\mathrm{innerCV}}(S_{t-1}\cup\{q\}). \tag*{(6)}

We stop once ∣S∣=k|S| = k, or when the inner CV R2R^2 exceeds a convergence threshold. Besides being tractable, the procedure produces a ranked list that practitioners can inspect. The diversity tie-breaker of § 4 modifies only the selection rule q∗q^*, replacing the single best candidate with a diversity-maximizing one drawn from the ϵ\epsilon-band of near-best candidates.

Algorithm 1: Greedy Selection

Input: Task Set QQ Output: Greedy Selected Task Set SkS_k

  1. S0=ϕS_0 = \phi

  2. for tt in {1…k}\{1\ldots k\} do

  3. qt∗=arg⁡max⁡q∈Q∖St−1 RinnerCV2(St−1∪{q})q_t^* = \underset{q\in Q\setminus S_{t-1}}{\arg\max}\ R^2_{\mathrm{innerCV}}(S_{t-1}\cup\{q\})

  4. St=St−1∪{qt∗}S_t = S_{t-1}\cup\{q_t^*\}

  5. return SkS_k

Algorithm 2: Modified Greedy Selection

Input: Task Set QQ, Tolerance Band ϵ\epsilon Output: Selected Task Set SkS_k

  1. S0=ϕS_0 = \phi

  2. for tt in {1…k}\{1\ldots k\} do

  3. Rbest2=max⁡q∈Q∖St−1 RinnerCV2(St−1∪{q})R^2_{\mathrm{best}} = \underset{q\in Q\setminus S_{t-1}}{\max}\ R^2_{\mathrm{innerCV}}(S_{t-1}\cup\{q\})

  4. Bϵ(St−1)={q∈Q∖St−1:RinnerCV2(St−1∪{q})≥Rbest2−ϵ}B_\epsilon(S_{t-1}) = \{q\in Q\setminus S_{t-1}: R^2_{\mathrm{innerCV}}(S_{t-1}\cup\{q\}) \ge R^2_{\mathrm{best}}-\epsilon\}

  5. qt∗=arg⁡max⁡q∈Bϵ(St−1) Div⁡(St−1∪{q})q_t^* = \underset{q\in B_\epsilon(S_{t-1})}{\arg\max}\ \operatorname{Div}(S_{t-1}\cup\{q\})

  6. St=St−1∪{qt∗}S_t = S_{t-1}\cup\{q_t^*\}

  7. return SkS_k

Taxonomy Diversity Score Div⁡(S)\operatorname{Div}(S)

Each taxonomy dimension j∈Dj \in\mathcal{D} contributes a coverage score Div⁡j(S)∈[0,1]\operatorname{Div}_j(S)\in[0,1] to the diversity score Div⁡(S)\operatorname{Div}(S) of § 4, with a form that depends on its measure type:

Div⁡j(S)={rj(S)/rj(T)(numeric),H(PS)/log⁡kT(categorical),∣Lj(S)∣/∣Lj(T)∣(multi-label),(7)\operatorname{Div}_j(S)= \begin{cases} r_j(S)/r_j(\mathcal{T}) & \text{(numeric)},\\ H(P_S)/\log k_{\mathcal{T}} & \text{(categorical)},\\ |L_j(S)|/|L_j(\mathcal{T})| & \text{(multi-label)}, \end{cases} \tag*{(7)}

where, over a set of tasks A\mathcal{A}, rj(A)=max⁡t∈Axt,j−min⁡t∈Axt,jr_j(\mathcal{A})=\max_{t\in\mathcal{A}}x_{t,j}-\min_{t\in\mathcal{A}}x_{t,j} is the spread of jj’s numeric values and Lj(A)=L_j(\mathcal{A})= ⋃t∈ALt,j\bigcup_{t\in A} L_{t,j} the set of labels present; PSP_S is the selected tasks’ distribution over jj’s categories and kTk_T the number of those categories globally. Numeric measures (e.g., table and workbook size, clarity, task difficulty) use range coverage; single-label categorical measures (e.g., header type, table structure, task type, domain) use entropy; and multi-label measures (e.g., meta operations and required LLM capabilities) use label coverage. Where a dimension is captured by several numeric signals (e.g., workbook size and structural complexity), we summarize them by their top principal components, which we use as numeric measures.

Outer-Split Validation Strategies

The three outer-loop strategies of § 4-D differ in how configurations are partitioned:

  • Nested KK-Fold. Configurations are shuffled and partitioned into 5 outer folds. In each outer fold the training configurations are used exclusively for selection and fitting, the held-out configurations only for prediction, and the final R2R^2 is computed from the concatenated out-of-fold predictions.

  • Walk-Forward Expanding. Configurations are sorted chronologically by build id and partitioned into equal-size sequential windows. The outer loop advances forward in time: fold ii trains on windows 1,…,i1,\ldots,i and tests on window i+1i+1.

  • Walk-Forward Sliding. Identical to expanding, but the training window has a fixed size equal to twice the test window, so that if the test window covers indices [t,t+w)[t,t+w), the training window covers [max⁡(0,t−2w),t)[\max(0,t-2w),t). Older data is dropped as evaluation advances, so each fold trains on a limited recent history.

On Mean-Imputation

We impute each missing entry with its per-task mean

μt=1∣Ct∣∑c∈CtXc,t\mu_t = \frac{1}{|C_t|}\sum_{c\in C_t} X_{c,t}

(the task’s average outcome over the configurations CtC_t that ran it), writing X~\widetilde{X} for the resulting dense matrix (X~c,t=Xc,t(\widetilde{X}_{c,t}=X_{c,t} when observed and μt\mu_t otherwise), so that configurations can be compared on common ground. Within each outer validation fold we compute μt\mu_t over the training configurations only and identify zero-variance tasks on the training fold alone, so a held-out configuration never contributes to the values imputed for it. The final selection on all data uses means over all configurations. The RQ4 broader analysis (§ 4-§ 1) follows the same training-fold convention. Configurations retained by the min_completion_rate filter are still mean-imputed at the task level in exactly this way. The obvious alternatives to mean-imputation are each biased in this log regime: dropping configurations with any missing outcome discards most data when pp is large; dropping ever-missing tasks favors those that happen to run more often; and averaging over only the observed selected tasks introduces a variable denominator, scoring different configurations on different effective subsets. A learned imputer (e.g., kk-nearest-neighbors or low-rank matrix completion) would instead add a genuine second model that stacks prediction errors into the regression’s inputs and leaks information across tasks. Mean-imputation is the simplest well-defined baseline. It is, moreover, a placeholder rather than a predictive step: μt\mu_t ignores everything about configuration cc and is identical for every configuration missing task tt, so an imputed cell cannot help separate configurations—it only dilutes the signal in the observed entries, never manufactures new signal. The sole predictive step is the Ridge fit over the selected columns; the cost of imputation is statistical rather than architectural, attenuating the fitted coefficients of heavily-missing columns (a bias we revisit in § 6).

Matched-Budget Controls on the Complete-Case Matrix

Table IV compares taxonomy-guided selection with the five controls of § 4-E at k=10k=10 on the zero-missing common-reference matrix (999 execution IDs by 715 common tasks, 595 nonconstant candidates); § 4-§ 1 summarizes the result. At ϵ=0.05\epsilon=0.05, taxonomy-guided selection has the highest honest R2R^2 under all three validation strategies, and at both ϵ\epsilon values it exceeds its matched ϵ\epsilon-band-random control under all three. At ϵ=0.10\epsilon=0.10 its diversity matches that of diversity-only selection, which has the lowest honest R2R^2 of any method.

Honest R2R^2 over the full k×k \times min_completion_rate grid

Figures 4–9 report the honest weighted-Ridge R2R^2 vs. ϵ\epsilon across the full k×k \times min_completion_rate grid, one figure per validation strategy. Across all three, the regularization benefit of moderate ϵ\epsilon is most pronounced at low min_completion_rate and small kk.

K-Fold weighted $R^2$ vs. $\varepsilon$ across $k$ and min_completion_rate grid

Figure 5. Nested KK-Fold: honest weighted-Ridge R2R^2 vs. ε\varepsilon across k×k \times min_completion_rate grid for k∈{20,25,30}k \in\{20, 25, 30\}.

Walk-Forward Expanding weighted $R^2$ vs. $\epsilon$ across $k \times \mathrm{min\_completion\_rate}$ grid

Figure 6. Walk-Forward Expanding: honest weighted-Ridge R2R^2 vs. ϵ\epsilon across k×min_completion_ratek \times\mathrm{min\_completion\_rate} grid for k∈{7,10,15}k \in\{7, 10, 15\}.

Walk-Forward Expanding weighted $R^2$ versus $\epsilon$ across the $k \times \mathrm{min\_completion\_rate}$ grid

Figure 7. Walk-Forward Expanding: honest weighted-Ridge R2R^2 vs. ϵ\epsilon across k×min_completion_ratek \times\mathrm{min\_completion\_rate} grid for k∈{20,25,30}k \in\{20, 25, 30\}.

Walk-Forward Sliding: honest weighted-Ridge $R^2$ vs. $\varepsilon$ across $k \times$ `min_completion_rate` grid

Figure 8. Walk-Forward Sliding: honest weighted-Ridge R2R^2 vs. ε\varepsilon across k×k \times min_completion_rate grid for k∈{7,10,15}k \in\{7, 10, 15\}.

References

  1. [1]H. Li et al., “SheetCopilot: Bringing software productivity to the next level through large language models,” NeurIPS, vol. 36, pp. 4952–4984, 2023.arxiv.org/abs/2305.19308
  2. [2]Y. Chen et al., “SheetAgent: Towards a generalist agent for spreadsheet reasoning and manipulation via large language models,” in WWW, 2025, pp. 158–177.arxiv.org/abs/2403.03636
  3. [3]Z. Ma et al., “SpreadsheetBench: Towards challenging real world spreadsheet manipulation,” NeurIPS, vol. 37, pp. 94 871–94 908, 2024.arxiv.org/abs/2406.14991
  4. [4]C. E. Jimenez et al., “SWE-bench: Can language models resolve real-world GitHub issues?” in ICLR, 2024.arxiv.org/abs/2310.06770
  5. [5]X. Liu et al., “AgentBench: Evaluating LLMs as agents,” in ICLR, 2024.arxiv.org/abs/2308.03688
  6. [6]R. R. Panko, “What we know about spreadsheet errors,” JOUEC, vol. 10, no. 2, pp. 15–21, 1998.DOI
  7. [7]F. Hermans et al., “Detecting and visualizing inter-worksheet smells in spreadsheets,” in ICSE. IEEE, 2012, pp. 441–451.DOI
  8. [8]A. Aleti, “Software testing of generative ai systems: Challenges and opportunities,” in ICSE-FoSE. IEEE, 2023, pp. 4–14.arxiv.org/abs/2309.03554
  9. [9]F. Dobslaw et al., “Challenges in testing large language model based software: A faceted taxonomy,” arXiv:2503.00481, 2025.arxiv.org/abs/2503.00481
  10. [10]A. Memon et al., “Taming Google-scale continuous testing,” in ICSE-SEIP. IEEE, 2017, pp. 233–242.DOI
  11. [11]K. Herzig et al., “The art of testing less without sacrificing quality,” in ICSE. IEEE, 2015, pp. 483–493.
  12. [12]S. Yoo et al., “Regression testing minimization, selection and prioritization: A survey,” STVR, vol. 22, no. 2, pp. 67–120, 2012.
  13. [13]Z. Li et al., “MiMoTable: A multi-scale spreadsheet benchmark with meta operations for table reasoning,” arXiv:2412.11711, 2024.arxiv.org/abs/2412.11711
  14. [14]R. C. Nickerson et al., “A method for taxonomy development and its application in information systems,” Eur. J. Inf. Syst., vol. 22, no. 3, pp. 336–359, 2013.DOI
  15. [15]B. Beizer, Software testing techniques. dreamtech Press, 2003.
  16. [16]F. Elberzhager et al., “Reducing test effort: A systematic mapping study on existing approaches,” Inf. Softw. Technol., vol. 54, no. 10, pp. 1092–1106, 2012.DOI
  17. [17]H. Hemmati et al., “Prioritizing manual test cases in rapid release environments,” STVR, vol. 27, no. 6, p. e1609, 2017.DOI
  18. [18]R. Cheng et al., “Revisiting test-case prioritization on long-running test suites,” in ISSTA, 2024, pp. 615–627.DOI
  19. [19]H.-Y. Hsu et al., “Mints: A general framework and tool for supporting test suite minimization,” in ICSE, 2009, pp. 419–429.DOI
  20. [20]R. Pan et al., “Requirements coverage-guided minimization for natural language test cases,” arXiv:2505.20004, 2025.DOI
  21. [21]Y. Li et al., “Prioritizing test cases for deep learning-based video classifiers,” EMSE, vol. 29, no. 5, p. 111, 2024.DOI
  22. [22]X. Dang et al., “Test input prioritization for machine learning classifiers,” IEEE TSE, vol. 50, no. 3, pp. 413–442, 2024.DOI
  23. [23]M. Openja et al., “An empirical study of testing machine learning in the wild,” ACM TOSEM, vol. 34, no. 1, pp. 1–63, 2024.arxiv.org/abs/2312.12604
  24. [24]F. M. Polo et al., “tinybenchmarks: evaluating llms with fewer examples,” arXiv:2402.14992, 2024.arxiv.org/abs/2402.14992
  25. [25]R. Vivek et al., “Anchor points: Benchmarking models with much fewer examples,” in EACL, 2024, pp. 1576–1601.arxiv.org/abs/2309.08638
  26. [26]H. Zhao et al., “Bento: Benchmark task reduction with in-context transferability,” arXiv:2410.13804, 2024.arxiv.org/abs/2410.13804
  27. [27]P. Yuan et al., “Beyond one-size-fits-all: Tailored benchmarks for efficient evaluation,” in ACL, 2025.arxiv.org/abs/2502.13576
  28. [28]Y. Perlitz et al., “Efficient benchmarking (of language models),” in NAACL-HLT, 2024, pp. 2519–2536.
  29. [29]Q. Ye et al., “How predictable are large language model capabilities? a case study on big-bench,” in EMNLP, 2023, pp. 7493–7517.
  30. [30]Q. Zhao et al., “Can we predict performance of large models across vision-language tasks?” arXiv:2410.10112, 2024.arxiv.org/abs/2410.10112
  31. [31]L. Cai et al., “Item response theory,” Annu. Rev. Stat. Appl., vol. 3, no. 1, pp. 297–321, 2016.
  32. [32]Y. Li et al., “Active evaluation acquisition for efficient llm benchmarking,” arXiv:2410.09592, 2024.
  33. [33]Y. Yin et al., “Decentralized arena: Towards democratic and scalable automatic evaluation of language models,” arXiv:2505.12808, 2025.arxiv.org/abs/2505.12808
  34. [34]A. Chatterjee et al., “Posix: A prompt sensitivity index for large language models,” in EMNLP, 2024.
  35. [35]F. Maia Polo et al., “Efficient multi-prompt evaluation of llms,” in NeurIPS, vol. 37, 2024, pp. 22 483–22 512.arxiv.org/abs/2405.17202
  36. [36]L. Pacchiardi et al., “100 instances is all you need: predicting the success of a new llm on unseen data by testing on a few instances,” arXiv:2409.03563, 2024.arxiv.org/abs/2409.03563
  37. [37]G. Zhang et al., “How benchmark prediction from fewer data misses the mark,” arXiv:2506.07673, 2025.arxiv.org/abs/2506.07673
  38. [38]S. Zhou et al., “WebArena: A realistic web environment for building autonomous agents,” in ICLR, 2024.arxiv.org/abs/2307.13854
  39. [39]T. Xie et al., “OSWorld: Benchmarking multimodal agents for open-ended tasks in real computer environments,” in NeurIPS Datasets & Benchmarks, 2024.arxiv.org/abs/2404.07972
  40. [40]G. Mialon et al., “GAIA: A benchmark for general AI assistants,” in ICLR, 2024.arxiv.org/abs/2311.12983
  41. [41]Y. Qin et al., “ToolLLM: Facilitating large language models to master 16000+ real-world APIs,” in ICLR, 2024.arxiv.org/abs/2307.16789
  42. [42]J. Payan et al., “InstructExcel: A benchmark for natural language instruction in Excel,” arXiv:2310.14495, 2023.
  43. [43]Z. Li et al., “Structural coverage criteria for neural networks could be misleading,” in ICSE-NIER, 2019, pp. 89–92.DOI
  44. [44]Z. Aghababaeyan et al., “Black-box testing of deep neural networks through test case diversity,” IEEE TSE, vol. 49, no. 5, pp. 3182–3204, 2023.DOI
  45. [45]S. Gerasimou et al., “Importance-driven deep learning system testing,” in ICSE, 2020, pp. 702–713.
  46. [46]K. Pei et al., “DeepXplore: Automated whitebox testing of deep learning systems,” in SOSP, 2017, pp. 1–18.
  47. [47]J. Kim et al., “Guiding deep learning system testing using surprise adequacy,” in ICSE, 2019, pp. 1039–1049.DOI
  48. [48]T. Y. Chen et al., “Adaptive random testing: The art of test case diversity,” JSS, vol. 83, no. 1, pp. 60–66, 2010.DOI
  49. [49]H. Hemmati et al., “Achieving scalable model-based testing through test case diversity,” ACM TOSEM, vol. 22, no. 1, pp. 1–42, 2013.DOI
  50. [50]R. Feldt et al., “Test set diameter: Quantifying the diversity of sets of test cases,” in ICST, 2016.DOI
  51. [51]C. Henard et al., “Comparing white-box and black-box test prioritization,” in ICSE, 2016, pp. 523–534.DOI
  52. [52]T. Zhodinasab et al., “DeepHyperion: exploring the feature space of deep learning-based systems through illumination search,” in ISSTA, 2021, pp. 79–90.
  53. [53]H. Fahmy et al., “Supporting deep neural network safety analysis and retraining through heatmap-based unsupervised learning,” IEEE Trans. Reliab., vol. 70, no. 4, pp. 1641–1657, 2021.DOI
  54. [54]D. R. Kuhn et al., “Software fault interactions and implications for software testing,” IEEE TSE, vol. 30, no. 6, pp. 418–421, 2004.DOI
  55. [55]Z. Aghababaeyan et al., “DeepGD: A multi-objective black-box test selection approach for deep neural networks,” ACM TOSEM, vol. 33, no. 6, pp. 1–29, 2024.DOI
  56. [56]Z. Li et al., “Distance-aware test input selection for deep neural networks,” in ISSTA, 2024, pp. 248–260.DOI
  57. [57]M. Mahdieh et al., “Test case prioritization using test case diversification and fault-proneness estimations,” Autom. Softw. Eng., vol. 29, no. 2, p. 50, 2022.DOI
  58. [58]I. T. Elgendy et al., “A systematic mapping study of the metrics, uses and subjects of diversity-based testing techniques,” STVR, vol. 35, no. 2, p. e1914, 2025.DOI
  59. [59]N. Neelofar et al., “Towards reliable AI: Adequacy metrics for ensuring the quality of system-level testing of autonomous vehicles,” in ICSE, 2024, pp. 1–12.arxiv.org/abs/2311.08049
  60. [60]S. S. Rajan et al., “Knowledge-based consistency testing of large language models,” arXiv:2407.12830, 2024.arxiv.org/abs/2407.12830
  61. [61]A. Marchetto et al., “An empirical validation of a web fault taxonomy and its usage for web testing,” J. Web Eng., pp. 331–345, 2009.DOI
  62. [62]H. K. Rajaram et al., “Taxonomy-based testing and validation of a new defect classification for health software,” J. Softw. Evol. Process, vol. 31, no. 1, p. e1985, 2019.DOI
  63. [63]L. Cui et al., “NLCTables: A dataset for marrying natural language conditions with table discovery,” in SIGIR, 2025, pp. 3638–3647.arxiv.org/abs/2504.15849
  64. [64]X. Wu et al., “TableBench: A comprehensive and complex benchmark for table question answering,” in AAAI, vol. 39, no. 24, 2025, pp. 25497–25506.
  65. [65]Y. Chen et al., “SheetAgent: A generalist agent for spreadsheet reasoning and manipulation via large language models,” in LLMs and Cognition, 2024.arxiv.org/abs/2403.03636
  66. [66]G. Eggert et al., “TabLib: A dataset of 627m tables with context,” arXiv:2310.07875, 2023.arxiv.org/abs/2310.07875
  67. [67]P. Wu et al., “RealHiTBench: A comprehensive realistic hierarchical table benchmark for evaluating LLM-based table analysis,” arXiv:2506.13405, 2025.arxiv.org/abs/2506.13405
  68. [68]Y. Sui et al., “Table meets LLM: Can large language models understand structured table data? a benchmark and empirical study,” in WSDM, 2024, pp. 645–654.
  69. [69]W. Lu et al., “Large language model for table processing: A survey,” Front. Comput. Sci., vol. 19, no. 2, 2025.DOI
  70. [70]A. Singha et al., “Tabular representation, noisy operators, and impacts on table structure understanding tasks in LLMs,” arXiv:2310.10358, 2023.arxiv.org/abs/2310.10358
  71. [71]P. Li et al., “Table-GPT: Table fine-tuned GPT for diverse table tasks,” PACMMOD, vol. 2, no. 3, pp. 1–28, 2024.DOI
  72. [72]J. Wu et al., “Mmqa: Evaluating LLMs with multi-table multi-hop complex questions,” in ICLR, 2025, p. 1.
  73. [73]X. He et al., “Anameta: A table understanding dataset of field metadata knowledge shared by multi-dimensional data analysis tasks,” arXiv:2209.00946, 2022.arxiv.org/abs/2209.00946
  74. [74]M. Hulsbos et al., “GitTables: A large-scale corpus of relational tables,” PACMMOD, vol. 1, no. 1, pp. 1–17, 2023.
  75. [75]S. Ashury-Tahan et al., “The mighty ToRR: A benchmark for table reasoning and robustness,” arXiv:2502.19412, 2025.
  76. [76]T. Zhang et al., “TableLlama: Towards open large generalist models for tables,” arXiv:2311.09206, 2023.arxiv.org/abs/2311.09206
  77. [77]X. Zhang et al., “TableLLM: Enabling tabular data manipulation by LLMs in real office usage scenarios,” arXiv:2403.19318, 2024.arxiv.org/abs/2403.19318
  78. [78]J. Xing et al., “MMTU: A massive multi-task table understanding and reasoning benchmark,” arXiv:2506.05587, 2025.arxiv.org/abs/2506.05587
  79. [79]D. Yi et al., “TablePilot: Recommending human-preferred tabular data analysis with large language models,” arXiv:2503.13262, 2025.arxiv.org/abs/2503.13262
  80. [80]S. Thorne, “Large language models for spreadsheets: Benchmarking progress and evaluating performance with FLARE,” in EuSpRIG, 2025.arxiv.org/abs/2506.17330
  81. [81]C. Li et al., “TReB: A comprehensive benchmark for evaluating table reasoning capabilities of large language models,” arXiv:2506.18421, 2025.DOI
  82. [82]M. L. Lupetti et al., “(Un)making AI magic: A design taxonomy,” in CHI, 2024, pp. 1–21.DOI
  83. [83]G. C. Cawley et al., “On over-fitting in model selection and subsequent selection bias in performance evaluation,” JMLR, vol. 11, pp. 2079–2107, 2010.DOI
  84. [84]M. Naeem et al., “Thematic analysis and artificial intelligence: A step-by-step process for using ChatGPT in thematic analysis,” Int. J. Qual. Methods, vol. 24, 2025.DOI
  85. [85]V. Viswanathan et al., “Large language models enable few-shot clustering,” TACL, vol. 12, pp. 321–333, 2024.

Paper details

vlnc:W_kmtmk6mr
Version
v1 · latest
Published
Fresh — added to the Hub within the last 30 days
Added
License
CC BY 4.0
Usage

views · 0

downloads · 0

A view counts once per browser per day, a download each time the PDF is requested.

  • Computer Science

GitHub Copilot, Claude Code, Anthropic’s Claude, GPT (multiple versions)

View statement
Looking for related papers…
Contents