betaFeedback

WorksComputer Science

GRACE: Parallel Best-of-N Multi-Agent Analysis of Closed-Form Questions over Dataframes with Guardrail Selection and Adaptive Retry

Abstract

LLM agents answer closed-form questions over a tabular dataframe by writing and executing Python, almost always as a single rollout: one interpretation, one script, one result, accepted without comparison. The few pipelines that produce several candidates reconcile them by answer-agreement voting, which needs a single comparable scalar and always returns one, even when every candidate is wrong. We present GRACE (Generative, Ranked, Adaptive Code Ensemble), a production-oriented pipeline that instead runs parallel best-of-N selection specialised to dataframe code. N agents run concurrently under a strict, machine-checkable output contract, diversified by persona hints, per-retry rotation, and a two-regime temperature schedule. Rather than vote, GRACE judges the executed artefacts: a pairwise tournament pre-filters the pool and a domain-specialised guardrail either selects the best candidate or, when none is defensible, rejects the whole pool and abstains with a diagnosis (an option voting structurally lacks), which in turn drives an adaptive retry with per-agent feedback. On InfiAgent-DABench (DAEval, M = 246), using Qwen3.6-35B-A3B as the backbone, GRACE raises answer match from 39% for a single-rollout baseline to 71% (+32 pp, 1.8x), recovering 53% of previously wrong answers. On the same DAEval task, the smaller dense Qwen3.5-9B lifts answer match from 38% to 63% (+25 pp, 1.7x). A controlled 3 x 3 study that sweeps the number of parallel agents against the retry budget finds, on the 35B backbone, the two levers largely substitutable: once either one is generous, the other adds little, identifying a cheap operating point for deployment. The same directional pattern replicates on the independent DataBench benchmark, again on the Qwen3.6-35B-A3B backbone, where answer match rises by 15 percentage points (from 70% to 85%) across 400 sampled questions.

Introduction

Natural-language interfaces to tabular data increasingly delegate the analytical step to an LLM code agent: given a dataframe and a question in natural language, the agent writes Python, executes it in a sandbox, and returns what the code produced: a value, a table, or a chart. We target the closed-form case: each question has a single well-defined answer computed by executable Python over one in-memory dataframe, the formulation measured by benchmarks such as InfiAgent-DABench [1] and instantiated by a wave of analysis agents that Section 2 surveys. Two assumptions recur across them. First, each commits to a single solution trajectory: one reading of an often under-specified question, one script, one result, accepted with no comparison against alternatives; even the systems that iterate refine that single trajectory rather than choosing among independent candidates. Second, each commits to that single result unconditionally, returning whatever the code produced, with no mechanism to recognise when no candidate is defensible and decline. The ability to abstain with a diagnosis, rather than emit a confident wrong answer, is a capability these single-trajectory pipelines simply lack.

Inference-time selection over multiple candidates (generate several solutions, keep the best) is among the most reliable accuracy levers known for code and reasoning, yet it has received limited systematic study in agents answering closed-form questions over a dataframe. The reason is that the selection methods proven elsewhere lean on signals this setting does not provide: a single analytical question has no held-out validation metric and no test suite to score candidates against, and its result is a structured artefact (a named dataframe, sometimes with charts) rather than a scalar that answer-agreement voting could agree. Selection here must therefore be label-free, judge the executed artefacts directly, and, unlike majority voting, keep the option to abstain when no candidate is defensible. § 2 develops each of these requirements.

This paper presents GRACE (Generative, Ranked, Adaptive Code Ensemble), a production-oriented multi-agent pipeline that brings parallel best-of-NN selection to this label-free regime. GRACE dispatches NN diversified code-generation agents in parallel under a strict machine-checkable output contract, then selects among their executed results with a two-stage judge: a single-elimination pairwise tournament that pre-filters the pool, then a domain-specialised guardrail that either approves the best candidate or rejects the whole pool. On rejection it triggers an adaptive retry that re-diversifies the pool, routes a targeted critique to each agent, and forces every retry to add a new code invariant rather than a cosmetic edit. Our work is an integrated instantiation of these ingredients for the label-free dataframe-analysis regime: candidate selection is performed at the executed-artefact level and can abstain, rather than relying on scalar answer agreement or validation metrics. We make four contributions:

  1. Parallel best-of-NN code generation for label-free dataframe analysis (§ 3.2): NN concurrent agents under a strict output contract, diversified by persona hints, per-retry persona rotation, and a two-regime temperature schedule, with a neutral anchor agent that always reflects the model’s default behaviour. Parallel sampling itself is not our novelty. Our contribution is to adapt it to label-free dataframe analysis by enforcing an output contract and diversifying rollouts through personas, rotation, and temperature scheduling.

  1. An integrated artefact-level selector with abstention (§ 3.4 and § 3.5): we compose an untrained pairwise tournament with a domain-specialised post-execution guardrail that judges the executed dataframe artefacts and their generating code, either selecting the strongest candidate or rejecting the pool. The contribution is not the pairwise-judge primitive itself, which we adopt from prior work showing that LLMs compare two candidates more reliably than they score or rank many at once [2, 3, 4], but its integration with dataframe artefact contracts, pool-level rejection, abstention, and targeted retry feedback in a label-free setting.

  1. Adaptive retry with invariant-forcing feedback (§ 3.5): rejection re-diversifies the pool and routes per-agent critiques, with every code retry required to add a structural assertion rather than a surface fix.

  1. A controlled 3×33 \times3 study with paired-bootstrap significance testing and equivalence analysis (§ 4 and § 5): a paired-bootstrap analysis under Benjamini–Hochberg control that sweeps the number of parallel agents against the retry budget (jointly, where prior best-of-NN data and code work scales one axis and fixes the other), isolating a substitutability effect between the two levers and a named cost–accuracy operating point for deployment.

Background and Related Work

LLM-based data-analysis agents. A growing body of work casts tabular analysis as code generation over a dataframe: the agent writes Python, executes it, and returns the result. These code-writing agents [5, 6, 7, 8, 9] differ in how they plan but build on standard agent primitives and orchestration frameworks such as ReAct, Reflexion, and AutoGen [10, 11, 12], and all share two limitations: each explores a single solution trajectory per question—iterating or debugging within it, but never generating independent candidates to choose among—and commits to its result unconditionally; the two assumptions GRACE drops. A separate line of data-science automation does select among candidates: SPIO, AIDE, and SELA [13, 14, 15] run explicit generate–evaluate–select search, and DS-Agent [16] revises via case-based performance feedback. Either way they rank or refine multi-step ML pipelines against a held-out validation metric—a near-ground-truth signal that the single closed-form questions we target do not provide, so our selection must instead be label-free.

Best-of-NN selection. Inference-time selection over multiple candidates is among the most reliable accuracy levers for LLMs: Self-Consistency [17] majority-votes over sampled chains of thought, inference-scaling reports near-log-linear gains across hundreds of samples on code and math [18, 19], and More Agents [20] shows accuracy rising with sampled-agent count under voting—which our NN-sweep revisits for dataframe code. Each of these moves a single lever, though: the parallel axis is scaled with no sequential-retry counterpart and no operating-point rule (More Agents leaves cost optimisation to future work), while sequential self-refinement fixes the retry budget by hand [21]. Only Snell et al. [19] sweep both jointly, but on MATH (a competition-mathematics reasoning benchmark) and with a difficulty-dependent, non-substitutable conclusion; our significance-controlled sweep of agents against retries has no counterpart for label-free dataframe code.

The selection step itself comes in several forms. Some train a verifier to score candidates [22, 23, 24]; others need no training, either clustering programs by their execution output (CodeT [25], AlphaCode [26]) or prompting an LLM to judge the candidates directly (Universal Self-Consistency [27], Mixture-of-Agents [28]), with the diversity of the candidate pool itself a further lever (SimpleStrat [29]). Text-to-SQL is where generate-and-select is already standard—CHASE-SQL [30] trains a pairwise selector, PairJudge RM [3] runs a pairwise knockout, and MAC-SQL [31] coordinates several agents—yet even there the only label-free selector is answer-agreement voting [17]. None of these mechanisms has reached closed-form dataframe agents, where the two assumptions of § 1 persist: those agents [6, 8, 9] stay single-rollout, and voting cannot abstain. GRACE closes this gap by composing these ingredients into a label-free selector that judges executed artefacts.

LLM-as-judge, verification, and guardrails. LLM-as-judge approaches directly assign scores to candidate outputs [2, 32, 33]. A consistent finding is that LLMs are more reliable when comparing two candidates than when evaluating a single one in isolation [2] or ranking multiple candidates simultaneously [4]. A related sequential paradigm is iterative self-refinement [21, 11, 34]. In parallel, guardrail frameworks such as NeMo [35] and Constitutional AI [36] primarily focus on safety constraints rather than candidate selection.

GRACE departs from prior work on three fronts: its guardrail judges post-execution artefacts (the resulting named dataframe) rather than textual outputs, extending the execution signal of LEVER [24] to a best-of-NN setting instead of a re-ranking pipeline; it combines the pairwise tournament with a guardrail that validates at most two finalists and can reject them outright, rather than relying on a single selection mechanism; and on rejection it regenerates the entire pool rather than refining a single output. Finally, unlike CHASE-SQL’s trained binary selector [30], our pairwise judge is prompt-based and needs no training, and its critiques for losing candidates double as retry feedback (§ 3.5).

The GRACE Pipeline

GRACE maps an input dataframe and a natural-language task to a single named result dataframe, optionally accompanied by chart artefacts when the task requires them, with lightweight user-facing metadata, with no human in the loop. The architecture is organised into three stages, detailed in Figure 1. First, a dataframe-description agent constructs the shared schema used by NN parallel code-generation agents. Second, the executed candidates pass through a selection stage composed of an optional pairwise tournament and a post-execution guardrail. Third, an accepted result is annotated with metadata, whereas a rejected pool triggers an adaptive retry or, after budget exhaustion, a terminal failure analysis. The central controller coordinates six agent roles while keeping candidate generation, selection, and retry feedback logically separated.

Detailed architecture of the GRACE pipeline

Figure 1. Detailed architecture of the GRACE pipeline. The controller first builds the shared dataframe description and dispatches NN parallel code-generation agents. Executed candidates that satisfy the output contract enter the selection stage: for N≥3N\ge3, a pairwise tournament reduces the pool to two finalists; otherwise, the tournament is bypassed. The guardrail then accepts one candidate or rejects the pool. Rejection triggers agent-specific feedback, persona rotation, and a new generation attempt until the retry budget rr is exhausted. Accepted artefacts receive metadata, whereas terminal failures produce a diagnostic report.

Throughout the paper, NN denotes the number of code-generation agents executed in parallel within each attempt, while rr denotes the maximum number of pool-level retries allowed after guardrail rejection. Thus, r=0r=0 permits only the initial attempt, whereas r=4r=4 permits the initial attempt followed by at most four retries.

Dataframe Description

A dedicated agent previews the first twenty rows and emits a per-column semantic description (meaning, domain, role) merged with data types and sample values into a compact schema, df_info. Computed once and reused by every downstream prompt, it is the sole authoritative description of the dataframe, keeping prompts compact and disambiguating similarly-named columns.

Parallel Code Generation and Execution

A configurable number NN of code agents run concurrently, each interpreting the task independently, generating a self-contained Python script, and executing it in a sandbox over a fixed library set (NumPy, pandas, scikit-learn, PyTorch, statsmodels, Plotly). A per-agent timeout terminates stragglers without affecting the others, so the round is bounded by the slowest surviving agent. On an execution error (uncaught exception, missing result variable, or contract violation) an agent may retry internally to fix its own code before its slot is marked failed, repairing low-cost mistakes locally. Because this repair runs before selection, every candidate reaching the tournament or guardrail has already executed and satisfied the contract: the judges compare only runnable results, never broken code, and an agent that still fails or times out contributes no candidate rather than a faulty one.

A post-execution validator enforces a three-clause contract: (i) a single-entry dictionary whose value is a pandas DataFrame (scalars, multi-table results, lists, and empty frames forbidden); (ii) optional chart artefacts, when needed to answer the task, stored outside the result dataframe and linked back to it through lightweight provenance metadata; (iii) a user-meaningful dataframe name (the dictionary key) for the metadata stage. Structural failures (scalar leakage, empty frames, dangling charts) thus become trivially detectable and, on our benchmarks, fall to near-zero, leaving semantic error as the dominant residual failure mode (§ 5).

Diversification: Personas, Rotation, and Temperature

Agents are perturbed along three axes to reduce the risk that the pool collapses onto one interpretation. The motivating premise is a general one: across the generation-diversity literature, injecting variation into a generator—through sampling temperature, prompt-level stratification, or agent composition—helps it escape a single dominant mode and cover more of the solution space than repeated near-greedy decoding [29, 37, 28]. We take this principle as motivated design and instantiate it as three concrete levers; we do not re-validate the principle here, nor do we claim each lever is independently supported, and ablations isolating their individual contributions are left to future work. (i) Personas. Each agent beyond the first carries one of four fixed preambles that bias its decisional stance without prescribing implementation (full text in Section A, item A). Agent 1 carries no persona and serves as a neutral anchor, identical to the N=1N=1 pipeline, so the model’s most-likely answer is always in the pool. (ii) Rotation. At each retry the persona offset shifts by one, so an agent whose persona led it off the right track receives a different bias next time and the guardrail’s per-agent critique (Section 3.5) does not fight the same biased prior every iteration. (iii) Temperature. A single value, uniform across agents within an attempt, follows a two-regime schedule, where tt denotes the attempt index, with t=1t=1 indicating the initial attempt:

T(t)={0.65t=11.0t≥2(1)T(t) = \begin{cases} 0.65 & t=1 \\ 1.0 & t\ge2 \end{cases} \tag*{(1)}

The first regime favours clean, coherent code; the second escapes interpretation modes that per-agent advice cannot dislodge. Diversity is delegated to personas and rotation rather than temperature, since near-greedy temperatures starve selection of diversity while accuracy plateaus across a moderate-to-high band, and prompt-level stratification yields more useful diversity than temperature alone [29, 37].

Pairwise Tournament

For N≥3N\ge3 valid results a single-elimination bracket runs before the guardrail; for N<3N<3 it is bypassed, recovering the no-tournament behaviour exactly. We adopt rather than propose it: that LLMs compare two candidates more reliably head-to-head than they rank many at once is an established finding we take from the literature and apply, not one we test [2, 3, 4]. Each match compares two candidates (code plus a preview of the executed result dataframe) against the task and must enumerate concrete defects in both before declaring a winner [2]; the JSON verdict carries a winner, a reason, and a one-to-two-sentence loser_critique stored per agent for routing on rejection. The bracket halts at two survivors, not one (N−2N-2 judge calls), so both finalists keep an advice channel into the guardrail.

Guardrail Validation and Adaptive Retry

Because the tournament always precedes it (§ 3.4), the guardrail receives at most two candidates—the two finalists when N≥2N\ge2, or the single candidate when N=1N=1—each as its Python code paired with a preview of the executed result dataframe (so it judges the program and its artefact jointly, not the artefact alone), and emits a structured JSON verdict in one of two shapes.

Case A: at least one candidate is correct. It selects the candidate with the fewest and least severe defects and emits a user-facing narrative in business language, stripped of implementation detail and of any reference to the selection process, so the artefact is directly consumable by a front-end.

Case B: no candidate is correct. It emits a global root-cause Advice and a per-agent critique list (Per_agent_advice), one targeted entry per finalist it evaluated. This triggers a pool-level retry, up to rr attempts, that updates the pool at once: temperature steps (Equation (1)), personas rotate (Section 3.3), global advice enters every prompt, and each per-agent critique reaches only its agent. The feedback channel routes by tournament outcome: the (up to two) finalists receive their Per_agent_advice, while each candidate eliminated earlier in the bracket (only when N≥3N\ge3) receives the loser_critique from the match that knocked it out (global → all, per-agent → finalists, loser-critique → eliminated), so no agent retries on stale feedback. Every code retry must moreover add a new explicit invariant, assertion, or state variable addressing the root cause: patching only the symptom is forbidden, pushing retries toward structural rather than cosmetic fixes, on the hypothesis that the residual failure after one attempt is rarely fixable by a surface change.

Failure Analysis and Metadata

Two terminal roles close the pipeline. If the retry budget is exhausted with no approval, a failure-analysis agent returns a diagnostic narrative (what each agent attempted, the root cause, the evolution across retries) in place of a result, so an unrecoverable case degrades into actionable feedback rather than a silent error. On success, a metadata agent adds concise titles and descriptions to the accepted dataframe and to any optional chart artefacts. These annotations are retained for front-end consumption, but they are not part of the experimental metric.

Experimental Setup

We evaluate on DAEval (InfiAgent-DABench [1]), closed-form data-analysis questions over CSV files with structured expected answers. The metric is answer match: correct if all expected values match, under a 1% relative / 0.01 absolute numeric tolerance and case-insensitive string comparison. Each prediction is a match or a wrong response (structurally valid but not matching gold) with structural failures to near-zero due to the strict output contract. The evaluation therefore targets the analytical answer, not presentation quality.

The pipeline uses the Google Agent Development Kit with LiteLLM for model-agnostic access and Langfuse for observability; code runs in a sandbox over the library set of § 3.2, with artefacts on S3-compatible storage. Our primary backbone, shared by all agents and the judge/guardrail, is Qwen3.6-35B-A3B [38], a general-purpose mixture-of-experts model with strong coding ability (35B total / 3B active per token, 8 routed plus 1 shared of 256 experts, 262,144-token native context), served at full precision (BF16); unless stated otherwise, all reported results use this backbone. To probe how the pipeline degrades under a smaller, dense model, we additionally run the DAEval 3×33 \times3 sweep with Qwen3.5-9B [39] (9B dense parameters) under an otherwise identical configuration. The purpose of the 9B experiment is therefore not to provide a complete cross-benchmark model comparison, but to test whether the interaction between agents and retries persists under a smaller backbone. DAEval was selected for this probe because it is the primary and more analytically demanding benchmark in our evaluation. Extending the full 3×33 \times3 grid to DataBench would require a further 3,600 question-level executions, in addition to retries and judging calls, and was outside the computational scope of this study. Token usage and wall-clock time are measured per question via Langfuse; the full agent and persona prompts are reproduced in Appendix § 10.

The experiments were orchestrated from a workstation with 13 GiB of memory and Python 3.12.3. Both Qwen models were hosted internally by Engineering Ingegneria Informatica S.p.A., served on a single NVIDIA H100 GPU, and accessed through LiteLLM. All sweep cells used the same remote infrastructure and model settings. Agents were submitted concurrently, and token usage and end-to-end latency were recorded through Langfuse.

We sweep a 3×33 \times3 grid over two levers: the number of parallel agents (the pool size N∈{1,2,4}N \in\{1,2,4\}) and the retry budget r∈{0,4,10}r \in\{0,4,10\} (the pool-level guardrail-rejection rounds of § 3.5). The range is deliberately small: each GRACE agent is a full code-generation, execution, and judging rollout, not a token-cheap sample, so the question is how little of each lever serves. We use the M=246M=246 DAEval questions with a well-formed, machine-checkable gold answer (the rest of the 257-question set have malformed or missing values, dropped uniformly across cells), so every configuration sees the identical question set in identical order, paired at the question level. The cheapest cell (N=1,r=0)(N=1,r=0) is the minimal single-rollout baseline. For benchmark context, InfiAgent-DABench reports a GPT-4 baseline of 78.99% on DAEval [1]. This value is provided only as an external reference because the backbone and evaluation harness differ from ours, preventing a controlled system-level comparison.

The comparison between this minimal baseline and the remaining sweep cells is intentionally an inference-scaling comparison rather than a compute-matched architectural ablation: increasing NN or rr necessarily increases the opportunity to generate a correct candidate. Consequently, the baseline-to-best gain measures the practical end-to-end benefit of allocating additional inference compute to the complete pipeline, but does not by itself establish that the same budget would be used less effectively by repeated independent sampling or generic sequential self-refinement. To partially separate candidate generation from selection without additional model runs, Section 5.1 compares the tournament+guardrail against majority answer-agreement voting and random selection on exactly the same candidate pools. These selectors therefore receive identical generated candidates and share the same candidate-generation budget, although the tournament+guardrail incurs additional judging calls and is not compute-matched in total inference cost.

We repeat the identical 3×33 \times3 sweep, backbone, and metric on a second, independent benchmark, DataBench [40], a large-scale QA corpus over real-world tables spanning five domains, with typed closed-form answers (boolean, category, number, lists). The two probe different question styles: DAEval targets data-analysis concepts from expert interviews—summary statistics, distribution and correlation analysis, outlier detection, preprocessing, feature engineering, machine learning [1], often requiring a multi-step statistical or modelling computation—whereas DataBench poses fact-oriented questions answerable by simpler column operations (counting, filtering, aggregation, top-kk ranking, threshold comparisons), with correlation and machine learning largely absent. This difficulty gap partly explains DataBench’s far higher single-rollout baseline (Section 5). Representative questions from both benchmarks are reported in Table 1. To bound cost we draw a fixed subset of M′=400M'=400 questions (80 tables ×\times 5, seed 12345)

BenchmarkRepresentative questions
DAEval• Calculate the mean, median, and population standard deviation of passenger fares separately for each passenger class, and interpret the results.<br>• Encode categorical variables and train and evaluate a regression model to predict passenger survival.
DataBench• How many billionaires are there in the Technology category?<br>• List the three countries with the most billionaires.

Table 1. Representative questions from the two evaluation benchmarks. DAEval includes multi-step statistical and modelling tasks, whereas DataBench mainly targets direct retrieval, aggregation, and ranking over tabular data.

from the full 1,829, stratified by file; the answer types come out near-balanced (boolean 79, category 75, number 92, list[number] 81, list[category] 73) and the subset is held constant and ordered across all nine cells, as for DAEval. For external context, the SemEval-2025 Task 8 leaderboard reports a best score of 95.0%, a best proprietary-system score of 89.9%, and several competitive systems in the mid-to-high 80% range [41]. These scores were obtained on the full test set with different models and harnesses. They are therefore reported as benchmark references rather than as directly comparable baselines.

We estimate match-rate differences with a paired bootstrap over the common question set, using 10,000 resamples and percentile confidence intervals. Multiple comparisons are controlled with the Benjamini–Hochberg procedure at q=0.1q=0.1. Significance symbols report adjusted values (* padj<0.1p_{\mathrm{adj}}<0.1, ** <0.05<0.05, *** <0.01<0.01). Token usage and wall-clock time are normalised to the median of the single-rollout baseline.

For the four plateau comparisons underlying the substitutability claim, we additionally test statistical equivalence using two one-sided tests. Equivalence is established when the 90% paired interval lies within a benchmark-specific margin equal to the median measurement resolution: ±5.7\pm5.7 pp for DAEval and ±3.75\pm3.75 pp for DataBench. This prevents a non-significant difference from being interpreted automatically as evidence of equal performance.

Results

DAEval with the 35B backbone. The complete pipeline improves answer match from 39% for the single-rollout baseline to a maximum of 71% (Table 2). With no retries, increasing the number of agents raises accuracy substantially, reaching approximately 62% with four agents. A retry budget produces a larger improvement for the single-agent configuration and brings all configurations into narrow high-accuracy region. Every non-baseline configuration significantly outperforms the baseline after correction for multiple comparisons. At the question level, the best configuration recovers more than half of the answers missed by the baseline while preserving almost all previously correct answers. The simple-effect analysis in Figure 2a shows that agents and retries are largely substitutable for this backbone: agents are most useful when retries are unavailable, whereas retries are most useful with a single agent. Once either resource is sufficiently available, increasing the other produces only minor differences. The equivalence analysis supports this interpretation for all four plateau comparisons.

Single-factor simple effects on DAEval for the Qwen3.6-35B-A3B and Qwen3.5-9B backbones

Figure 2. Single-factor simple effects on DAEval (M=246M=246) under the paired-bootstrap + BH protocol of § 4: retries at fixed NN (left) and agents at fixed rr (right). (a) With the primary 35B-A3B MoE backbone, retries are decisive only at N=1N=1 and agents only at r=0r=0, with nothing significant beyond one step of either lever. (b) The same view for the smaller Qwen3.5-9B backbone (downscaling probe). Significance stars denote the Benjamini–Hochberg-adjusted pp-value (* padj<0.1p_{\mathrm{adj}}<0.1, ** <0.05<0.05, *** <0.01<0.01; n.s. otherwise).

DAEval with the 9B backbone. The smaller dense backbone exhibits the same overall benefit at a lower final accuracy. The single-rollout baseline is comparable to that of the 35B model, but the gap between the backbones emerges when additional agents and retries are introduced. The dense model reaches a maximum answer match of 63%, compared with 71% for the MoE backbone. Retries again provide most of their benefit by the first tested retry level. Unlike the 35B model, however, the 9B model continues to benefit from additional parallel agents even when retries are available. This suggests that the smaller model gains more from independent interpretations of the question, whereas the stronger backbone can more often correct an initial interpretation through retry feedback. Detailed cell-level results are reported in Table 2 and Figure 2b.

DAEval (M=246M=246) — Qwen3.6-35B-A3B (MoE)
match rate (%)r=0r=0r=4r=4r=10r=10baseline →\to best (N=2N=2, r=4r=4), 246 paired
N=1N=139.066.766.7from correct (96)95 kept, 1 regress
N=2N=256.170.769.5from wrong (150)79 recovered (net +78)
N=4N=461.868.370.3
DAEval (M=246M=246) — Qwen3.5-9B (dense)
match rate (%)r=0r=0r=4r=4r=10r=10baseline →\to best (N=4N=4, r=4r=4), 246 paired
N=1N=137.853.552.0from correct (93)82 kept, 11 regress
N=2N=245.960.259.4from wrong (153)74 recovered (net +63)
N=4N=449.263.460.6
DataBench (M′=400M'=400) — Qwen3.6-35B-A3B (MoE)
match rate (%)r=0r=0r=4r=4r=10r=10baseline →\to best (N=4N=4, r=10r=10), 400 paired
N=1N=169.884.583.5from correct (279)270 kept, 9 regress
N=2N=279.082.583.0from wrong (121)70 recovered (net +61)
N=4N=478.582.285.0

Table 2. Answer-match rate across the 3×33 \times3 sweep, where NN is the number of parallel code-generation agents per attempt and rr is the maximum number of pool-level retries after guardrail rejection. For each backbone and benchmark, the table also reports the paired transition from the single-rollout baseline to the best-performing cell; baseline cell (N=1N=1, r=0r=0) and best cell in bold. Top: DAEval (M=246M=246) under the primary 35B-A3B MoE backbone; middle: DAEval under the smaller dense Qwen3.5-9B backbone (downscaling probe, § 4); bottom: DataBench (M′=400M'=400, 35B-A3B). Single-factor simple effects for DAEval are in Figure 2.

DataBench with the 35B backbone. DataBench independently reproduces the main pattern despite its substantially higher single-rollout baseline. Answer match rises from approximately 70% to 85%, and all non-baseline configurations significantly outperform the baseline. As on DAEval, retries are most useful with a single agent, while additional agents contribute mainly when retries are unavailable. Performance above the baseline is concentrated within a narrow range, making the inexpensive single-agent configuration with a moderate retry budget nearly as accurate as the most expensive configuration. Equivalence is established for two of the four plateau comparisons; the remaining differences are small but cannot be declared equivalent under the stricter DataBench margin. The complete results are reported in Table 2.

The high-accuracy plateau also defines a clear cost–accuracy frontier. On both benchmarks, a single agent with a moderate retry budget approaches the best accuracy at approximately 1.1×1.1 \times the baseline token cost. On DAEval, two agents with the same retry budget attain the highest accuracy at approximately twice the baseline cost. Four-agent configurations consume substantially more tokens without a corresponding improvement, while latency remains close to the baseline because agents execute concurrently. Thus, the practical trade-off is driven primarily by token usage rather than wall-clock time.

Selection Quality

End-to-end accuracy cannot distinguish coverage failures, where no correct candidate is generated, from selection failures. We therefore instrument a separate DAEval run at (N=4N=4, r=4r=4) with the 35B backbone and grade every candidate against gold. Its 246 questions produce 263 selection decisions because retries re-enter selection. Each pool contains on average 3.66 valid candidates, 2.32 of which are correct, but candidates rarely produce equivalent artefacts.

Comparison with voting. To control for the main source of inference scaling, we compare three selectors on exactly the same 263 candidate pools (Table 3). All methods therefore receive identical generated candidates and have the same candidate-generation budget; only the selection mechanism changes. Candidates are clustered by output equivalence (matching type, shape, columns, and preview) so that answer-agreement voting has a well-defined input. The tournament+guardrail is correct on 66.9% of decisions—86% of the way to the 77.6% oracle ceiling. It significantly beats majority answer-agreement voting (63.1%; +3.8 pp, paired-bootstrap 95% CI [+0.3, +7.3], p=0.03p=0.03) and a random pick (**62.9%; +4.0 pp, p=0.02p=0.02). Thus, additional candidate generation alone does not explain the complete benefit: holding the generated pool fixed, the proposed selector extracts a modest but statistically significant improvement over voting. This is a controlled comparison of selection quality, not a claim of equal total inference cost, because the tournament and guardrail require additional judge calls.

correct candidate in pool?sel. correctsel. wrongrejectedtotal
yes176 (86.3%)17 (8.3%)11 (5.4%)204
no–53 (89.8%)6 (10.2%)59
all176 (66.9%)7017263

Table 3. Selection quality on DAEval (N=4N=4, r=4r=4; 263 decisions), each conditioned on whether the pool held a correct candidate (within-row % in parentheses). Realized selection (176/263=66.9% correct) is bounded by the oracle ceiling (204/263=77.6%, a correct candidate present) and beats the label-free baselines majority answer-agreement voting (63.1%) and random pick (62.9%), which are statistically indistinguishable.

Voting, by contrast, is statistically no better than random (+0.2 pp, p=0.66p=0.66): the candidates rarely coincide, so there is little agreement to count—in 63.9% of decisions no two agree at all, and a majority cluster of size ≥2\ge2 exists in only 34.2%. The pools are already majority-correct, so a random pick alone secures 62.9%.

Selection and abstention. Conditioning on whether the pool contains a correct candidate reveals two distinct behaviours (Table 3). Selection is the stronger one: when a correct candidate is available, the selector chooses it in 86.3% of cases. Most of the remaining oracle gap is caused either by choosing an incorrect candidate or by rejecting a pool that already contains a valid answer. Abstention is considerably weaker. When every candidate is incorrect, the guardrail rejects the pool in only 10.2% of cases and usually accepts an incorrect answer. The system is therefore better at ranking available candidates than at recognising that none is defensible.

Retry after rejection. Rejected pools trigger a re-diversified retry. Almost half of the affected questions eventually reach a correct final answer, although recovery is more frequent when the rejected pool already contained a valid candidate. These values are based on a small number of rejected questions and should therefore be interpreted qualitatively. Retry also explains why per-question accuracy exceeds per-decision selection accuracy. A rejected decision counts as unsuccessful in the selection analysis but can still produce a correct answer in a later attempt.

Discussion

The central finding, on the 35B backbone, is that the two ways of spending more compute—running more agents in parallel (NN) or allowing more retries (rr)—are largely interchangeable: once one of them is generous, adding the other barely helps (Figure 2a; the flat r≥4r \ge4 plateau in Table 2). A plausible explanation is that the remaining errors are mostly semantic misreadings after structural failures have been filtered by the output contract. The output contract removes almost all structural failures, so what is left is the model misreading the question—and for that kind of error a failed attempt is still useful, because it shows what went wrong and lets the next try, whether a fresh parallel agent or a retry, start better informed. Parallel agents and retries are thus two ways of taking more shots at the same target, which is why having plenty of one makes the other redundant. This agrees with More Agents [20] and differs from Snell et al. [19], where the best split between parallel and sequential compute depends on problem difficulty. Consistent with this, the loop recovers 53% of initially wrong answers (§ 5): high for a fully automatic system, plausibly because executing the generated Python yields concrete feedback and each retry is forced to add a new assertion rather than a cosmetic patch (§ 3.5). The weaker 9B backbone sharpens the picture: there parallel agents keep helping even under a generous retry budget (§ 5), because a smaller model’s single reading is less often the right one, so the extra interpretations that more agents supply carry a benefit retries cannot—whereas at 35B, whose first reading is usually right, that benefit has largely vanished and the two levers collapse into substitutes.

The pattern is not a single-benchmark artefact: it survives DataBench’s far higher baseline and halved headroom (§ 5). Substitutability still holds on the plateau, even if added retries now buy a small but significant increment at N≥2N \ge2 rather than ceasing to help past N=1N=1. One caveat tempers the cross-benchmark comparison: DataBench’s high baseline is a poor proxy for a lower effect of typed answers—driven by its categorical questions rather than its booleans, which are in fact among the hardest types here—so it overestimates the true difficulty gap.

For deployment, this substitutability together with the cost-efficiency frontier (§ 5) points to (N=2,r=4)(N=2,r=4) as the operating point for standard workloads and (N=1,r=4)(N=1,r=4) as a budget fallback; at these pool sizes (N<3N<3) the tournament is bypassed (§ 3.4), so deployment leans on the guardrail alone. This is an operating point, not a ceiling: our questions are closed-form over clean single CSVs, and we cannot exclude that more complex dataframes or harder, more open-ended questions—where a single reading is less likely to be right—would reward the broader interpretive diversity of N=4N=4, exactly the regime in which the pairwise tournament also becomes active.

The selection analysis (§ 5.1) shows the two links are unequal. Selection is the strong one—a correct candidate, when present, is usually recovered despite the pool rarely agreeing on it, which is why the tournament+guardrail clears a voting baseline that the unscored artefacts leave no better than random. Abstention is the weak link: the guardrail tells good candidates from bad yet seldom recognises when all are wrong, so it mostly false-accepts all-wrong pools. This weak abstention directly limits accuracy: all-wrong pools are usually false-accepted, whereas correctly rejecting them would trigger a re-diversified retry and create another opportunity for recovery. Rejections of pools containing a correct candidate are often recovered on retry, while genuinely all-wrong pools rarely turn around. Abstention is thus the axis to sharpen next—a capability the single-rollout and voting baselines lack entirely—and, the residual failure being semantic, the next gains lie in the guardrail/retry layer rather than in adding generators.

Four limitations bound these results. First, DAEval supplies each question’s source CSV as a fixed, authoritative input. The data itself can be messy—missing values, outliers, and preprocessing are part of the task—but the table is the correct one for the question, so these numbers isolate the analytical stage. In deployment the dataframe is instead a fallible upstream output (a wrong table, or one carrying extraction or ETL errors), which compounds an error rate this benchmark does not exercise. End-to-end evaluation with imperfect upstream data is future work.

Second, the baseline and multi-agent/retry configurations are not compute-matched. Their comparison measures the practical benefit of inference scaling through the complete pipeline, but does not establish superiority over repeated sampling or generic self-refinement under an equal total budget. The same-pool analysis partially controls this confound: given identical candidates, tournament+guardrail selection outperforms majority voting by 3.8 pp (§ 5.1), although it still incurs additional judge calls. A fully compute-matched comparison requires new executions and is left to future work.

Third, the sweep does not isolate the marginal impact of personas, rotation, temperature, invariant forcing, or the pairwise tournament, which we adopt from prior work showing that LLMs judge pairwise comparisons more reliably than absolute scores or full rankings [2, 3, 4], rather than advancing it as an independently measured contribution. A per-component ablation that isolates each lever is therefore also left to future work.

Finally, answer match treats all errors uniformly and cannot distinguish near-misses from semantically incorrect answers or complete failures. A future qualitative error analysis should separate failures in interpretation, computation, formatting, and abstention.

Conclusion

GRACE addresses a gap between single-rollout dataframe agents, validation-metric pipeline selectors, and scalar-voting tabular-QA ensembles by making best-of-NN selection label-free, artefact-level, and able to abstain: diversified parallel agents under a strict contract, a pairwise tournament composed with a post-execution guardrail, and an adaptive retry that forces structural rather than cosmetic fixes. On DAEval, with the 35B-A3B backbone, increasing inference compute through the complete pipeline lifts answer match from 39% to 71% and recovers 53% of previously wrong answers. A controlled same-pool comparison further shows that, for identical generated candidates, tournament+guardrail selection improves over majority answer-agreement voting by 3.8 pp. The sweep also exposes a substitutability between agents and retries that identifies an efficient operating point. DataBench reproduces the same directional plateau pattern (+15 pp, 70 → 85%), with equivalence partially established there (two of four plateau steps clear TOST). A smaller dense backbone, Qwen3.5-9B, serves as a downscaling probe on DAEval, where the same pattern holds at a lower level (+25 pp, 38 → 63%): retries again saturate at r=4r=4, while—unlike at 35B—added parallel agents keep helping even under a generous retry budget.

Acknowledgments

This work is framed within the Project “AVANT” – Project no. IPCEI-CL_00000005 – Application protocol no. 108421 of 14/05/2024 – CUP B89J24002920005 – Grant decree no. 1322 of August 8, 2024 – financed by the European Union – NextGenerationEU (IPCEI Funding).

Declaration on Generative AI

During the preparation of this work, the authors used Microsoft Copilot for grammar checking, spelling correction, and text rephrasing. The authors subsequently reviewed and edited all generated suggestions and take full responsibility for the content of the manuscript.

Agent Personas and Prompt Templates

We reproduce the four personas and three prompt templates (code/ML agent, pairwise judge, post-execution guardrail). {braces} are runtime placeholders; >= replaces the original “≥” glyph.

A. Personas (P1P_1–P4P_4). One preamble is prepended to every non-anchor agent (Agent 1, the anchor, carries none); the offset rotates by one each retry (§ 3.3).

P1P_1, schema-strict / minimal. Treat the provided df_info as the only definitive source of truth about the dataframe. Use only columns and dtypes that are explicitly listed; never infer the existence of fields, categories, or relationships that are not stated. Prefer the smallest, most direct code that answers the question without adding checks beyond what is required.

P2P_2, defensive / robust. Be explicit about edge cases: handle potential NaN values, verify dtypes before operations that assume them, guard against empty groups or single-class targets, and prefer explicit conditions over implicit defaults. Add assertions or state variables that make assumptions about the data visible in the code.

P3P_3, exploratory / chart-generous. When the question implies inspecting a group of similar variables, produce the full set of plots (one per variable), not just an example. When in doubt about the right grain, lean toward keeping the result at the individual-row level with derived columns added, rather than aggregating early.

P4P_4, pragmatic / direct. Go as directly as possible to what the user question literally asks. Avoid pre-processing, feature engineering, or aggregations that the question does not require. Pick the simplest aggregation or operation that matches the wording of the request.

B. Code/ML agent. Prepended for agents 2..NN with a line |PERSONA: <one of P1..P4>|.

You are an execution-only ML agent, given a dataframe at {df_path} with schema {df_info}. Load it with the provided load_df helper, never pd.read_csv; use only schema columns—referencing non-existing columns is a semantic error.

Generate self-contained Python and execute it through execute_generated_code; do not print, output raw code, or add explanations outside the executed analysis. Reason only over the current request, code, error, and provided retry context.

Handle missing values explicitly before modelling, transforming, aggregating, or asserting. Critical operations (loading, fitting, transforming, predicting) must expose success or failure through explicit state, assertions, or control-flow checks. Silent exception handling is forbidden: a caught exception must record its cause, and downstream operations must verify the required state exists.

After execution, follow one of two paths. If the output contains `"COMPLETED": true, return only a business-language summary of the findings. Otherwise treat the run as failed: identify the earliest causal failure and the violated implicit assumption, then revise the code to enforce that assumption explicitly. Each retry must add at least one new invariant, assertion, or state variable for the root cause; cosmetic or trivial equivalence fixes are invalid.

The script must define exactly one result dictionary entry:

result = {
    "<descriptive_name>":
    {"type": "dataframe", "value":
    <pandas.DataFrame>}
}

Only "type": "dataframe" is allowed; scalars, lists, dicts, plots, metrics, and empty frames are forbidden (even scalar answers must be wrapped in a non-empty dataframe). Consolidate multiple tables into one, kept at the finest grain the answer or any chart needs.

When charts are required, save them only inside {agent_output_dir_<i>} via os.path.join, never inside result, each generated from the dataframe in result; produce the full set when the question needs one chart per variable. For each Plotly chart save the triplet .plotly.json, .png, .recipe.json, whose recipe field data_csv must match the single key in result.

For ML operations, calling predict, transform, or accessing learned attributes such as coef_ or classes_ requires explicit proof that fitting completed. If fitting fails or is skipped, downstream ML operations must not run.

Retry feedback, when present, is provided through: {retry_advice} for global guardrail advice, {retry_advice_<i>} for this agent’s critique, {error_<i>} for the previous error, and {python_code_<i>} for the previous code.

User request: "{ML_question}".

C. Pairwise judge.

The pairwise judge receives a natural-language question {ML_question} and two executed candidate analyses, {pairwise_A_ML} and {pairwise_B_ML}, including both their Python code and produced results. Its task is to select the candidate that better answers the question.

Before deciding, the judge internally critiques both candidates. It checks whether each candidate uses only existing dataframe columns, handles missing values before modelling or aggregation, chooses the correct metric or aggregation, returns a result dataframe at the required grain, produces all required plots, and fully answers the question rather than only part of it. Semantic defects such as wrong columns, wrong aggregation, missing NaN handling, or incomplete plots are considered more severe than stylistic issues.

The judge selects the candidate with the fewest and least severe defects, preferring the simpler one when roughly equivalent. It returns only JSON with three fields: winner ("A" or "B"), reason (the decisive defect), and loser_critique (one or two concrete sentences on what the loser should fix next).

D. Guardrail.

The guardrail receives a natural-language question {ML_question}, the executed analyses and their Python code {ml_results}, and any previous errors {error_guardrail}. Its role is to decide whether at least one analysis correctly answers the question and, if so, to select the best one.

Before deciding, the guardrail critiques each analysis against the same checklist as the pairwise judge (valid columns; missing values handled before modelling or aggregation; the metric or aggregation semantically required; result at the correct grain; all required plots; the full question answered). Plot completeness is judged from the code: if the question requires multiple plots, e.g. one per variable, missing plots make the analysis incorrect even if the numeric result is plausible.

The guardrail selects the analysis with the fewest and least severe defects. Semantic errors such as wrong columns, wrong aggregation, missing NaN handling that affects the result, or incomplete plots make an analysis incorrect. If multiple analyses are valid, the guardrail prefers the simpler and more direct one.

On success it returns JSON with Correct_response="true", Correct_analysis_number (the selected index), and Detailed_analysis: a user-facing final report with the relevant numbers and patterns, never mentioning code, rejected analyses, or the selection process. On failure it returns Correct_response="false", Advice (the dominant root cause), and Per_agent_advice: one concise, analysis-specific critique each, citing real column names, avoiding code, and stating the concrete fix for the next attempt.

References

  1. [1]Hu, X., Zhao, Z., Wei, S., et al.: InfiAgent-DABench: Evaluating Agents on Data Analysis Tasks. In: International Conference on Machine Learning (ICML) (2024). arXiv:2401.05507arxiv.org/abs/2401.05507
  2. [2]Zheng, L., Chiang, W.-L., Sheng, Y., et al.: Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena. In: NeurIPS Datasets and Benchmarks Track (2023). arXiv:2306.05685arxiv.org/abs/2306.05685
  3. [3]Liu, Y., Yao, Z., Min, R., Cao, Y., Hou, L., Li, J.: PairJudge RMI: Perform Best-of-N Sampling with Knockout Tournament. arXiv:2501.13007 (2025)
  4. [4]Qin, Z., Jagerman, R., Hui, K., et al.: Large Language Models are Effective Text Rankers with Pairwise Ranking Prompting. In: Findings of the Association for Computational Linguistics: NAACL 2024. arXiv:2306.17563arxiv.org/abs/2306.17563
  5. [5]Zhang, W., Shen, Y., Tan, Z., Hou, G., Lu, W., Zhuang, Y.: Data-Copilot: Bridging Billions of Data and Humans with Autonomous Workflow. arXiv:2306.07209 (2023)
  6. [6]Hong, S., Lin, Y., Liu, B., et al.: Data Interpreter: An LLM Agent for Data Science. In: Findings of ACL (2025). arXiv:2402.18679arxiv.org/abs/2402.18679
  7. [7]Qiao, B., Li, L., Zhang, X., et al.: TaskWeaver: A Code-First Agent Framework. arXiv:2311.17541 (2023)arxiv.org/abs/2311.17541
  8. [8]Sun, M., Han, R., Jiang, B., Qi, H., Sun, D., Yuan, Y., Huang, J.: LAMBDA: A Large Model Based Data Agent. arXiv:2407.17535 (2024)
  9. [9]You, Z., Zhang, Y., Xu, D., Lou, Y., Yan, Y., Wang, W., Zhang, H., Huang, Y.: DatawiseAgent: A Notebook-Centric LLM Agent Framework for Adaptive and Robust Data Science Automation. In: Proc. EMNLP (2025). arXiv:2503.07044
  10. [10]Yao, S., Zhao, J., Yu, D., Du, N., Shafran, I., Narasimhan, K., Cao, Y.: ReAct: Synergizing Reasoning and Acting in Language Models. In: International Conference on Learning Representations (ICLR) (2023). arXiv:2210.03629
  11. [11]Shinn, N., Cassano, F., Berman, E., Gopinath, A., Narasimhan, K., Yao, S.: Reflexion: Language Agents with Verbal Reinforcement Learning. In: Advances in Neural Information Processing Systems (NeurIPS) (2023). arXiv:2303.11366
  12. [12]Wu, Q., Bansal, G., Zhang, J., et al.: AutoGen: Enabling Next-Gen LLM Applications via Multi-Agent Conversation. arXiv:2308.08155 (2023)
  13. [13]Seo, W., Lee, J., Shao, Y., Zhou, Q., Lee, S., Bu, Y.: SPIO: Ensemble and Selective Strategies via LLM-Based Multi-Agent Planning in Automated Data Science. arXiv:2503.23314 (2025)
  14. [14]Jiang, Z., Schmidt, D., Srikanth, D., Xu, D., Kaplan, I., Jacenko, D., Wu, Y.: AIDE: AI-Driven Exploration in the Space of Code. arXiv:2502.13138 (2025)
  15. [15]Chi, Y., Lin, Y., Hong, S., Pan, D., Fei, Y., Mei, G., Liu, B., Pang, T., Kwok, J., Zhang, C., Liu, B., Wu, C.: SELA: Tree-Search Enhanced LLM Agents for Automated Machine Learning. arXiv:2410.17238 (2024)
  16. [16]Guo, S., Deng, C., Wen, Y., Chen, H., Chang, Y., Wang, J.: DS-Agent: Automated Data Science by Empowering Large Language Models with Case-Based Reasoning. In: Proc. ICML (2024). arXiv:2402.17453
  17. [17]Wang, X., Wei, J., Schuurmans, D., Le, Q., Chi, E., Narang, S., Chowdhery, A., Zhou, D.: Self-Consistency Improves Chain of Thought Reasoning in Language Models. In: International Conference on Learning Representations (ICLR) (2023). arXiv:2203.11171
  18. [18]Brown, B., Juravsky, J., Ehrlich, R., Clark, R., Le, Q.V., Ré, C., Mirhoseini, A.: Large Language Monkeys: Scaling Inference Compute with Repeated Sampling. arXiv:2407.21787 (2024)
  19. [19]Snell, C., Lee, J., Xu, K., Kumar, A.: Scaling LLM Test-Time Compute Optimally Can Be More Effective Than Scaling Model Parameters. arXiv:2408.03314 (2024)
  20. [20]Li, J., Zhang, Q., Yu, Y., Fu, Q., Ye, D.: More Agents Is All You Need. Transactions on Machine Learning Research (TMLR) (2024). arXiv:2402.05120arxiv.org/abs/2402.05120
  21. [21]Madaan, A., Tandon, N., Gupta, P., et al.: Self-Refine: Iterative Refinement with Self-Feedback. In: Advances in Neural Information Processing Systems (NeurIPS) (2023). arXiv:2303.17651arxiv.org/abs/2303.17651
  22. [22]Cobbe, K., Kosaraju, V., Bavarian, M., et al.: Training Verifiers to Solve Math Word Problems. arXiv:2110.14168 (2021)
  23. [23]Lightman, H., Kosaraju, V., Burda, Y., et al.: Let’s Verify Step by Step. In: International Conference on Learning Representations (ICLR) (2024). arXiv:2305.20050arxiv.org/abs/2305.20050
  24. [24]Ni, A., Iyer, S., Radev, D., Stoyanov, V., Yih, W.-t., Wang, S.I., Lin, X.V.: LEVER: Learning to Verify Language-to-Code Generation with Execution. In: Proc. ICML (2023). arXiv:2302.08468arxiv.org/abs/2302.08468
  25. [25]Chen, B., Zhang, F., Nguyen, A., Zan, D., Lin, Z., Lou, J.-G., Chen, W.: CodeT: Code Generation with Generated Tests. In: International Conference on Learning Representations (ICLR) (2023). arXiv:2207.10397arxiv.org/abs/2207.10397
  26. [26]Li, Y., Choi, D., Chung, J., et al.: Competition-Level Code Generation with AlphaCode. Science 378(6624), 1092–1097 (2022). arXiv:2203.07814DOI
  27. [27]Chen, X., Aksitov, R., Alon, U., Ren, J., Xiao, K., Yin, P., Prakash, S., Sutton, C., Wang, X., Zhou, D.: Universal Self-Consistency for Large Language Model Generation. arXiv:2311.17311 (2023)arxiv.org/abs/2311.17311
  28. [28]Wang, J., Wang, J., Athiwaratkun, B., Zhang, C., Zou, J.: Mixture-of-Agents Enhances Large Language Model Capabilities. In: International Conference on Learning Representations (ICLR) (2025). arXiv:2406.04692arxiv.org/abs/2406.04692
  29. [29]Wong, J., Orlovskiy, Y., Luo, M., Seshia, S.A., Gonzalez, J.E.: SimpleStrat: Diversifying Language Model Generation with Stratification. arXiv:2410.09038 (2024)arxiv.org/abs/2410.09038
  30. [30]Pourreza, M., Li, H., Sun, R., Chung, Y., Talaei, S., Kakkar, G.T., Gan, Y., Saberi, A., Ozcan, F., Arik, S.O.: CHASE-SQL: Multi-Path Reasoning and Preference Optimized Candidate Selection in Text-to-SQL. In: International Conference on Learning Representations (ICLR) (2025). arXiv:2410.01943arxiv.org/abs/2410.01943
  31. [31]Wang, B., Ren, C., Yang, J., Liang, X., Bai, J., Chai, L., Yan, Z., Zhang, Q.-W., Yin, D., Sun, X., Li, Z.: MAC-SQL: A Multi-Agent Collaborative Framework for Text-to-SQL. In: Proc. COLING (2025). arXiv:2312.11242arxiv.org/abs/2312.11242
  32. [32]Liu, Y., Iter, D., Xu, Y., Wang, S., Xu, R., Zhu, C.: G-Eval: NLG Evaluation using GPT-4 with Better Human Alignment. In: Proc. EMNLP (2023). arXiv:2303.16634arxiv.org/abs/2303.16634
  33. [33]Kim, S., Shin, J., Cho, Y., et al.: Prometheus: Inducing Fine-Grained Evaluation Capability in Language Models. In: International Conference on Learning Representations (ICLR) (2024). arXiv:2310.08491arxiv.org/abs/2310.08491
  34. [34]Gou, Z., Shao, Z., Gong, Y., Shen, Y., Yang, Y., Duan, N., Chen, W.: CRITIC: Large Language Models Can Self-Correct with Tool-Interactive Critiquing. In: International Conference on Learning Representations (ICLR) (2024). arXiv:2305.11738arxiv.org/abs/2305.11738
  35. [35]Rebedea, T., Dinu, R., Sreedhar, M., Parisien, C., Cohen, J.: NeMo Guardrails: A Toolkit for Controllable and Safe LLM Applications with Programmable Rails. In: Proc. EMNLP: System Demonstrations (2023). arXiv:2310.10501arxiv.org/abs/2310.10501
  36. [36]Bai, Y., Kadavath, S., Kundu, S., Askell, A., et al.: Constitutional AI: Harmlessness from AI Feedback. arXiv:2212.08073 (2022)arxiv.org/abs/2212.08073
  37. [37]Sheng, L., Xu, S.-S.: CSC-SQL: Corrective Self-Consistency in Text-to-SQL via Reinforcement Learning. arXiv:2505.13271 (2025)arxiv.org/abs/2505.13271
  38. [38]Qwen Team: Qwen3.6-35B-A3B: Agentic Coding Power, Now Open to All. https://qwen.ai/blog?id=qwen3.6-35b-a3b (2026)
  39. [39]Qwen Team: Qwen3.5-9B. Hugging Face model card, https://huggingface.co/Qwen/Qwen3.5-9B (2026)
  40. [40]Osés Grijalba, J., Ureña-López, L.A., Martínez Cámara, E., Camacho-Collados, J.: Question Answering over Tabular Data with DataBench: A Large-Scale Empirical Evaluation of LLMs. In: Proc. LREC-COLING, pp. 13471–13488 (2024)DOI
  41. [41]Osés Grijalba, J., Ureña-López, L.A., Martínez Cámara, E., Camacho-Collados, J.: SemEval-2025 Task 8: Question Answering over Tabular Data. In: Proc. 19th International Workshop on Semantic Evaluation (SemEval-2025), pp. 2512–2522. Association for Computational Linguistics (2025)

Paper details

vlnc:W_5hj52kpw
Version
v1 · latest
Published
Fresh — added to the Hub within the last 30 days
Added
License
CC BY 4.0
Usage

views · 1

downloads · 0

A view counts once per browser per day, a download each time the PDF is requested.

  • Computer Science

Microsoft Copilot

View statement
Looking for related papers…
Contents