betaFeedback

WorksComputer Science

LuthorSQL: A One-Pass Text-to-SQL Model as System 1 for a Slower LLM Agent

Abstract

Frontier LLM agents that query databases with tools are among the most accurate text-to-SQL systems, but per-query calls to a hosted model are slow, costly, and expose sensitive organizational queries to external APIs. For example, a Claude Opus 5.5 agent serving 10,000 queries a day costs over $70,000 a year at list price. To what extent can local hardware already owned by an organization offload this burden without compromising the agent's accuracy? We present LuthorSQL, a non-autoregressive 7B text-to-SQL model that serves as a rapid System 1 gate for a slower LLM agent (System 2), escalating queries only when uncertain. LuthorSQL makes a single forward pass to predict slot scores over a bounded grammar for query plans, compiles and runs up to 32 candidate plans, and chooses an answer using a trained validator. On its own, LuthorSQL reaches 68.03% execution accuracy on BIRD's hidden test set at 0.84 s per question on a single GPU, maintains its accuracy in 9 GB with 8-bit quantized weights, and transfers zero-shot to Snowflake, BigQuery, MySQL, PostgreSQL, and DuckDB within 3.1 points of its native SQLite accuracy. Combined with Claude Opus 5.5 in a pre-registered confidence cascade over 227 questions on databases published after Opus's training cutoff, LuthorSQL answers nearly half of all questions locally and cuts API cost by 33.3% at the same accuracy. Finally, contamination probing shows that Opus reproduces 48-92% of public benchmark gold queries verbatim without seeing the schema; however, its tool-assisted accuracy holds on renamed schemas and reaches 95.6-97.4% on post-cutoff databases. Code and data: https://github.com/IAMIbrahimmemon/luthorsql

Introduction

Most questions asked of an organization’s data are routine: how many orders shipped last month, which customers lapsed, what a region earned. On BIRD (Li et al., [40]), a benchmark of large real databases with noisy values, many high-scoring systems write SQL token by token and sample several queries per question before voting, reranking or revising ([50, 37, 69]), and LLM agents that explore the database with tools are among the most accurate. Run as such an agent, Claude Opus 5.5 answered 77.26% of 796 questions on held-out BIRD training databases correctly in our runs, at about 2 cents per question at list price. For an organization asking ten thousand questions a day that is about 200aday,over200 a day, over70,000 a year, with every question sent to a hosted model, even though BIRD hands the agent an expert-written hint per question that real users do not. Yet of the 311 questions it answered correctly in a 400-question sample, our one-pass 7B model alone also got 228 right.

Kahneman’s split between a fast, automatic System 1 and a slow, deliberate System 2 (Kahneman, [34]) fits this situation, as do model cascades that send easy inputs to a cheap model (Chen et al., [11]), and our design was inspired by TypeSafe’s Jev, a “System One” model that returns typed decisions instead of free text (Almeida, [4]). Our System 1 is a one-pass, non-autoregressive 7B model that runs on hardware the organization already owns, where each query it answers costs only electricity and never leaves that machine; System 2, the agent, is called only when System 1 is unsure. We ask how much of the agent’s bill one local model can remove without costing accuracy, whether such a model is tied to the SQL engine it was trained on, and how far an agent’s accuracy on public benchmarks can be trusted when the agent may have seen them.

LuthorSQL goes back to slot filling (Xu et al., [68]; Yu et al. [71]). A LoRA adapter (Hu et al., [29]) on OmniSQL-7B (Li et al., [37]) reads the schema and question once and fills every slot of a bounded plan grammar in parallel; a deterministic compiler turns each plan into SQL, taking joins from the foreign-key graph; and a second adapter scores the executed candidates and picks one (Figure 1). The grammar covers 84.6% of BIRD dev gold queries, one reason to pair System 1 with an agent. Both adapters sit on the same frozen backbone weights, which can be loaded once, so LuthorSQL is one 7B model with two small adapters, not an ensemble of LLMs; LuthorSQL means this two-adapter system throughout (single-adapter variant: Appendix § 12; path to this design and failed attempts: Appendices § 10 and § 11).

How LuthorSQL answers a question

Figure 1. How LuthorSQL answers a question. The slot heads, read after the question, point to schema columns and to a stored value; the word links are drawn for illustration, since the model scores slot states, not word alignments. No question word names characters; the plan lists it only in its require slot, and the compiler builds the join along the foreign-key path between movie and actor. The same pass yields up to 32 distinct plans (21 for this question), found by best-first search, which are compiled, executed and scored by the validator. Example: BIRD train question 740 (movie, a held-out database), answered correctly.

Contributions.

  • LuthorSQL, a one-pass slot-filling System 1 on a 7B backbone with a typed plan grammar, lazy kk-best candidates and a validator trained on hard negatives: 68.03% on BIRD’s hidden test set at 0.84 s per question on one GPU, the same accuracy with 8-bit weights in about 9 GB, and, translated without retraining, 57.1–59.3% on Snowflake, BigQuery, MySQL, PostgreSQL and DuckDB against 60.17% in SQLite on BIRD dev.

  • A confidence cascade with an LLM agent, tested in registered experiments on data that played no part in choosing it. Calling Opus 5.5 only below a validator margin of 3.0 answered about half of the questions locally and cut the cost per question by 24.8%, 32.9% and 32.5% in three runs on 348 held-out BIRD questions (30.1% pooled) and by 19.9% on 693 KaggleDBQA and Spider-Realistic questions compared with Opus alone, at accuracy within 1.2 points of it (none of the differences significant; Figure 2, Table 2). Against Opus called directly through the API, the saving is 33.3% on 227 questions over unseen databases at equal accuracy.

RES run 1RES poolExternalFresh
Questions / DBs348 / 83×348 / 8693 / 27227 / 8
Agent calls50.9%50.9%51.9%55.1%
EX Opus (B0)80.1779.5088.3197.36
EX Opus + cands (B1)80.4680.7586.87–
EX cascade (C)81.3280.4688.1794.27
EX System 163.22–70.2763.88
C − B1+0.86−0.29+1.30–
95% interval[−2.6, 3.6][−2.8, 1.5][−2.0, 4.9]
C − B0+1.15+0.96+0.14−3.08
95% interval[0.0, 3.0][−0.6, 2.9][−3.4, 3.2][−6.0, 0.0]

Table 2. The four pre-registered tests of the t=3.0t = 3.0 cascade (EX in %; differences in points with database sign-flip intervals). RES: RESERVED; pool: three complete runs of every arm, EX averaged; External: KaggleDBQA test and Spider-Realistic; Fresh: new questions on post-cutoff databases (no B1 arm). Cost savings are in Figure 2.

Cost saving of the cascade

Figure 2. Cost saving of the t=3.0t = 3.0 cascade in the held-out tests against Opus alone (B0, circles) and Opus with System 1’s candidates on every question (B1, squares), with System 1 on local hardware. RES: RESERVED (348 BIRD questions; pool: three runs); External: KaggleDBQA and Spider-Realistic (693); Fresh: 227 post-cutoff questions. Dashed line: 30%.

  • A memorization check for LLM baselines, which we recommend other work run. Without any schema, Opus reproduces 48–92% of the gold queries of BIRD train, KaggleDBQA and Spider-Realistic verbatim, so thinly prompted LLM scores on these sets partly reflect recall; with every name in the test databases changed, and on 227 new questions over databases published after its training cutoff (97.4% for Opus alone, 94.3% for the cascade), its accuracy with a tool holds up (§ 7.4).

Related Work

Slot filling and parallel generation. SQLNet ([68]) filled a fixed sketch with its own prediction module per slot, after Seq2SQL ([75]) had split the query into classified and decoded parts; both targeted single-table WikiSQL questions. Multi-table work moved to grammar-based decoders and intermediate representations decoded step by step ([71, 26]), building on sketch-then-fill decoding ([15]). Non-autoregressive decoders ([25, 21]) and SmBoP ([55]) cut the number of sequential steps, and diffusion language models ([48, 70, 23]) denoise iteratively, and Ma et al. [45] evaluate them on text-to-SQL. Sketch slot filling has also been applied to multi-table Spider queries ([12]), and SemQL and NatSQL derive joins from the foreign-key graph ([26, 19]). Our closed slot grammar, with expression templates mined from training data, is filled by a 7B backbone in one forward pass that returns kk-best plans.

Selection. Picking among candidates by execution agreement ([66]), a trained verifier or an LLM judge is common ([47, 50]). Closest to our validator, GradeSQL scores candidate queries with an outcome reward model at test time ([61]); XiYan-SQL combines several generators and reorganizes their candidates for a selection model ([20, 43]), and MSc-SQL has a small model critique several sampled queries together ([24]). R3-SQL groups candidates by execution result and ranks the groups ([27]), as MBR-exec and CodeT use execution agreement ([59, 10]); our validator instead scores each candidate on its own, and we group by result only to compute its margin and to show the agent one entry per result (more in Appendix § 21). Our candidates come mainly from lazy kk-best enumeration over factorized decisions ([30]); fixed-seed samples from the same slot scores fill any shortfall.

Cascades and fast/slow systems. FrugalGPT ([11]) sends a query through a chain of models and stops when a scorer trusts the answer, AutoMix routes to a larger model when a smaller model’s self-verified answer looks wrong ([1]), and RouteLLM trains routers on preference data to choose between a strong and a weak model ([49]); confidence-based deferral is well studied ([62, 33, 14]), including on answer consistency ([73]), like our vote-agreement baseline. For text-to-SQL, Malekpour et al. (2024) route among LLMs and argue that cascades cannot be used because a weak model’s SQL cannot be judged before falling back; our validator margin is such a judgment. Ze-roNL2SQL lets small fine-tuned models write an SQL sketch that an LLM completes ([17]), and EllieSQL routes each question to a cheaper or costlier pipeline by its estimated complexity ((Zhu et al., 2025)). TypeSafe’s Jev, a “System One” model that returns typed decisions with probabilities instead of text and that TypeSafe describes as outputting all probabilities in parallel rather than token by token, inspired our System 1 ([4]). SwiftSage pairs a small fast model with a slow LLM planner ([41]). Our setting differs in that the fast model is a one-pass slot parser whose executed, result-grouped kk-best list and validator margin serve both as the routing signal and as the agent’s starting point.

Evaluation. Execution accuracy on a single database can accept a wrong query that happens to match ([75]); our scores share this limitation. BIRD’s labels have known errors ([67]), so we report the original dev gold and label BIRD’s 2025 corrected gold ([6]) separately, and we test the main differences with paired significance tests ([16]), clustered by database. Benchmark contamination can inflate LLM scores ([57]); guided completion of benchmark instances detects it ([22]), and text-to-SQL accuracy has been shown to drop on data an LLM cannot have seen ([54]). We apply such probes to the agent baseline itself and add a renamed-schema test and a post-cutoff evaluation set.

System 1: LuthorSQL

Given a natural-language question qq, an external knowledge hint (“evidence”) ee, and a relational database DD with schema SS and stored contents, the text-to-SQL task is to produce an executable SQL query yy. Following the BIRD benchmark [40], a predicted query yy is evaluated by execution accuracy: yy is correct if executing it on DD yields the exact same set of result rows as the ground-truth query y∗y^*, written

exec⁡(y,D)=exec⁡(y∗,D).\operatorname{exec}(y,D)=\operatorname{exec}(y^*,D).

Within our fast/slow architecture, System 1’s role is to return an accurate candidate query from one generator pass and one validator pass on local hardware while producing a score that § 5 converts into a routing confidence for System 2.

Overview

Instead of writing SQL token by token, LuthorSQL predicts the structured query plan in a single forward pass. From the predicted query plan, LuthorSQL then compiles the executable SQL code. After factorizing the plan score, LuthorSQL enumerates top-kk choices of that plan, and chooses the final answer through a trained validator (Figure 1). Every step is designed to alleviate different failure modes of small language models:

  • Structured plans: Constraining output to discrete slot decisions prevents the model from generating syntax errors or hallucinating string literals.

  • Deterministic compiler: Inferring table joins over the foreign-key graph relieves the neural network from writing complex join clauses.

  • Factorized scoring: Scoring slots independently allows kk-best candidate plans to be enumerated without autoregressive beam search.

  • Full-query validator: Scoring the compiled SQL string allows a second adapter to inspect the complete query structure and select among executed candidates.

Plans and Grammar Coverage

A query plan pp is a bounded record of discrete choices that abstracts away surface SQL syntax while capturing the semantic intent of a query. Formally, each slot σ\sigma represents a single structural decision over a finite option set Oσ(q,D)O_{\sigma}(q,D), where options correspond to schema items in SS, small closed label sets, or grounded literal candidates. A complete plan pp assigns one choice to each active slot, and our plan grammar GG defines the space of all valid combinations of slot choices.

To ensure the model never invents filter values, filter slots can only point to a finite set of literal candidates L(q,D)L(q,D) extracted from three sources: (1) database values matched by a text span in qq or ee; (2) numbers, dates, quoted strings, and capitalized phrases parsed directly from qq and ee; and (3) the complete value lists of low-cardinality columns containing at most 12 distinct values. A plan holds one to four projections—each pairing a schema column or row count with an aggregation operator (none, count, count distinct, sum, avg, min, or max) and an optional expression template—alongside up to four filters, each specifying a column, an operator, and a pointer into L(q,D)L(q,D). Additional slots govern grouping, ordering, limits, DISTINCT, date parts, ratios, OR groups, and a single level of derived tables (full slot specification and mined template counts in Appendix § 19).

We define the coverage of GG as the proportion of benchmark queries with at least one plan p∈Gp \in G that when compiled becomes a query that returns exec⁡(y∗,D)\operatorname{exec}(y^*,D). We observe 1,297/1,534 (84.6%) of the BIRD dev queries and 670/796 (84.2%) of the DECISION queries covered by converting the gold queries to plans and testing them by execution. An oracle slot model would return 1,280 (83.4%) of the dev responses. The remaining queries involve novel expression templates, ungroundable literal values, CTEs, and scalar subqueries. To address those queries we are using a cascade with an unconstrained LLM agent.

Scoring Plans in One Forward Pass

Our generator scores every slot in a single forward pass so that the log-probability of any complete plan decomposes as a sum over its individual decisions. We use OmniSQL-7B ([37, 31]) equipped with a LoRA adapter as the encoder backbone. We serialize the database schema SS (tables, columns, types, descriptions, and sample values) as a text prefix, followed by qq, ee, and one learned embedding token per slot σ\sigma. A single forward pass over this sequence produces a contextualized hidden state hσh_{\sigma} for every slot.

To represent the option set Oσ(q,D)O_{\sigma}(q,D), each schema item is encoded as the mean of the hidden states across its serialized schema line, while each literal candidate in L(q,D)L(q,D) combines the hidden state of its mention span in qq, the schema state of its host column, and an optional “Values:” suffix entry. Specialized lightweight heads then score the decisions: pointer heads compare hσh_{\sigma} against schema options; choice heads apply a softmax over small categorical label sets; pairwise heads pass (hσ,option)(h_\sigma, \text{option}) pairs through a small MLP to score aggregations, operators, templates, and constants conditioned on the chosen column; and value heads score literal candidates. Because these decisions factorize, the total score of a plan pp is the sum of log-probabilities over the active decisions d∈pd \in p:

score⁡(p)=∑d∈plog⁡P(d∣q,e,D)\operatorname{score}(p) = \sum_{d \in p} \log P(d \mid q, e, D)

We train the generator with cross-entropy against execution-verified gold plans. For set-valued slots such as filters and literal candidates, where multiple target choices PP are simultaneously valid against a negative set NN, we optimize a multi-positive cross-entropy objective in which each positive i∈Pi \in P competes against all negatives without suppressing other positives:

Lset=−∑i∈Plog⁡esiesi+∑j∈Nesj\mathcal{L}_{\mathrm{set}} = -\sum_{i \in P} \log\frac{e^{s_i}}{e^{s_i} + \sum_{j \in N} e^{s_j}}

From Plan to SQL

A deterministic compiler translates each plan pp into executable SQL and constructs all table joins automatically, ensuring that a join fails only if a model hint is incorrect or the shortest relational path does not match the question’s intent. Given the set of tables referenced by a plan’s columns, the compiler connects them via a minimum-cost Steiner tree over the database’s foreign-key graph, guided by optional model-predicted join hints that specify required bridge tables or preferred foreign keys (declared keys cost 1, inferred keys more and hinted keys less; dead-end bridge tables are pruned). The compiler emits native SQLite syntax and transpiles queries to other enterprise SQL dialects when targeting external database engines.

Candidate Enumeration

Because score⁡(p)\operatorname{score}(p) decomposes additively across slots, we can extract the top-kk highest-scoring plans directly from the single forward pass without rerunning the backbone. A best-first priority queue pops plans in increasing order of their log-probability gap from the greedy best plan, lazily expanding only the conditional decisions that a plan actively uses. We discard plans that fail to compile or that compile to a duplicate SQL string already in the pool, filling any remaining shortfall via fixed-seed sampling from the slot distributions. This yields up to k=32k = 32 distinct SQL queries per question, each executed read-only against DD with a 3 s timeout to form our set of executed candidates. We partition these candidates into result groups, where each group consists of all candidates that return an identical execution result table.

Picking an Answer with the Validator

Because slot decisions are scored factorially, the generator cannot evaluate global interactions across a complete SQL query; we therefore introduce a validator, VALHARD, to inspect and score each compiled candidate in full. VALHARD is a second LoRA adapter attached to the same frozen OmniSQL-7B backbone with a linear scoring head. For each candidate cc, it reads the serialized schema SS, question qq, evidence ee, and compiled SQL string, outputting a scalar quality score s(q,c)s(q,c) without access to the candidate’s generation rank, plan score, or execution result. We define a candidate as live if it executes within the timeout and returns at least one non-NULL value. At inference time, LuthorSQL outputs the highest-scoring live candidate:

y^=arg⁡max⁡c live s(q,c)\hat{y} = \underset{c \ \mathrm{live}}{\arg\max}\ s(q,c)

falling back to the highest-scoring candidate overall if no candidate is live.

We train VALHARD with the generator’s k=32k = 32 candidate lists of BIRD-train databases. It is trained using hard negatives covering common ranking errors: near-miss wrong result groups and synthetic query perturbations (e.g., COUNT → COUNT DISTINCT switching, filter removal, ORDER BY switching) kept only if they return a new, wrong, non-null result on execution. Although the validator scores each candidate on its own, its training loss treats each result group as one composite option. The log-mean-exp of the result group candidates’ scores is used to compute the probability mass to avoid overestimating the probability mass for replicated SQL strings (Appendix § 13).

Pairing System 1 with an LLM Agent

System 2 is Claude Opus 5.5 run as an agent.1 Its database tool is a read-only SQLite runner, called through the shell, that can also print the schema (CREATE statements, three sample rows per table and BIRD’s column descriptions); as a Claude Code agent it also has file and shell tools, which it used to read its questions and write its answers, and it is instructed to execute at most eight queries before giving one final SQL query. Alone, this agent scores 77.26% on our 796-question selection split, against 59.05% for System 1. We combine the two in two ways.

Candidate handoff. System 1 runs first. Its candidates (up to 32) are grouped by execution result and the top eight groups, ordered by validator score, go into the agent’s prompt as labeled options, each with its SQL, a short preview of its result and how many of its candidates returned it. The agent keeps the same tool and the eight-query limit; its prompt adds the candidates and asks it to pick a clearly right one without further exploration, or to write its own query only when it is confident every candidate is wrong. On 126 of the 796 questions, the ones whose gold query the grammar cannot express, System 1 was not run in our DECISION experiments, a choice that uses gold; there the agent gets no candidates (under the handoff prompt) and System 1 is scored as wrong; 70 of the 400 sampled questions are of this kind. In the cascade these 126 and 26 more questions whose candidates form fewer than two result groups have no margin, so 152 questions always go to the agent. We test whether checking a short list of executed answers takes fewer tool rounds than building a query from the schema up.

Confidence cascade. System 1 answers alone when it is confident and passes the question to the handoff setup otherwise. Confidence is the validator’s score margin between its top two result groups, and on DECISION the 126 out-of-grammar questions always go to the agent; the held-out tests route every question by the margin alone, with no gold. We pick the threshold with a rule written down before the run: on a 0.25 grid from 2.0 to 6.0, take the threshold that maximizes accuracy minus cost, valuing a right answer at $1 and an agent call at its list price.

Neither setup changes System 1.

Experimental Setup

Data and splits. All training data is public: BIRD train ([40]), Spider train ([72]) and a sample of SynSQL-2.5M ([37]), each gold query kept only if its compiled plan returns the gold result (details in Appendix § 14).

BIRD dev (1,534 questions, 11 databases) is scored with the original 2024-06-27 gold, as in most prior work, and we label rescoring against BIRD’s 2025 revision ([6]) wherever it appears. From October 8 on, every design choice was made on DECISION, 796 questions from 8 BIRD-train databases that our adapters never trained on; RESERVED holds 348 more questions from the same databases. OmniSQL’s own fine-tuning data includes all of BIRD train, so these databases are held out from our adapters but not from the backbone; Appendix § 14 lists the databases and other limits on how clean the splits are. We allowed at most three looks at BIRD dev from October 8: LuthorSQL used the first and the System 2 dev run the second.

Metric and tests. For queries that lie outside the plan grammar G\mathcal{G}, System 1 still outputs its highest-scoring valid plan when evaluated on BIRD dev, whereas on our held-out DECISION split out-of-grammar questions are strictly scored as incorrect in all standalone System 1 arms. Latency measurements are reported on a seeded 200-question DECISION sample at batch size 1 (averaging 29.6 distinct candidates per question; full hardware details in Appendix § 14). Paired comparisons use an exact two-sided sign-flip test over per-database summed differences, with 95% intervals by inversion; with 8 databases the smallest attainable pp is 2/28≈0.0082/2^{8} \approx0.008, so we also report question-level McNemar wins and losses.

System 2 runs. Each agent run gives one Claude Code subagent 25 questions at a time, with the read-only SQL tool and the prompts in Appendix § 20. The handoff comparison uses a fixed 400-question DECISION sample with the two arms interleaved; the cascade threshold is chosen on all 796 questions after a second handoff run on the other 396. Costs are Opus 5.5 list prices applied to the subagents’ usage logs, not invoices (Appendix § 14).

Held-out tests. The cascade was tested on three held-out sets, each registered before any agent ran on it: RESERVED; KaggleDBQA’s test split ([35]) and Spider-Realistic ([13]), on 27 databases, two of which overlap System 1’s training data (§ 7.3); and a new set of 227 questions over 8 databases built from public data dated after the agent’s training cutoff (Appendix § 16.§ 4). Memorization probes, a renamed-schema test and attempts to raise accuracy were registered the same way (Appendices § 16.§ 3 and § 16.§ 5); one later exploratory search was not. Except for the API runs (Appendix § 16.§ 6), the agent ran as Claude Code subagents in a sandbox holding only the databases and the SQL tool, and every transcript was scanned for access outside it.

Results

System 1 Standalone Performance and Latency

LuthorSQL achieves competitive accuracy among fast local models while operating in under a second on a single GPU. On BIRD dev, the two-adapter System 1 reaches 59.39% execution accuracy (911 of 1,534 questions), placing it roughly on par with multi-sample diffusion language models with a reported latency of over five seconds (a preprint result under different conditions and hardware), though behind slower autoregressive models that generate SQL token by token such as its OmniSQL-7B backbone (62.6% in our harness; Appendix § 15). As shown in Table 6 (Appendix § 15), generating only the single greedy plan takes just 0.13 s end-to-end on one H100 PCIe and achieves 49.12% on DECISION. Expanding to k=32k = 32 candidates and reranking with VALHARD adds nearly ten percentage points of accuracy (+9.93 points) for roughly 0.7 s of additional compute, yielding a median end-to-end latency of 0.84 s per question ($0.64 per 1,000 queries).

Mode (R1 generator)DEC.DevH100A100
First plan only49.12–0.13 s0.18 s
k=32k=32 + VALHARD59.0559.390.84 s1.23 s
k=64k=64 + repair + val.59.55–1.60 s2.37 s
Batched, k=32k=32 + VALHARD
questions / s1.090.77
$ per 1,000 questions | | | 0.64 | 0.72

Table 6. Speed tiers of the two-adapter System 1 pipeline (bold: the frozen LuthorSQL configuration). EX (%) on DECISION (796 held-out training questions) and BIRD dev (original gold, all 1,534); times are p50 per question at batch size 1, end to end, measured on 200 DECISION questions with model load excluded (H100 PCIe and A100 80GB PCIe); cost at $2.50/h (H100) and $2.00/h (A100).

Zero-shot dialect transfer. Because LuthorSQL predicts an engine-agnostic query plan before compiling to SQL, it transfers across enterprise database engines without retraining. Transpiling the SQLite candidates and executing them natively on all 1,534 BIRD dev questions yields 57.1–59.3% on Snowflake, MySQL, DuckDB, PostgreSQL and BigQuery, compared to 60.17% in SQLite on the same database files (Table 9 in Appendix § 15). A single local checkpoint thus serves six distinct SQL engines within a 3.1-point accuracy band.

EXSimpleMod.Chall.
BIRD dev (1,534)
SQLite60.1770.1646.7739.31
Snowflake59.3269.3046.1237.93
MySQL57.6367.1444.8337.93
DuckDB57.4367.6843.9735.17
PostgreSQL57.3766.5945.4736.55
BigQuery57.1167.1443.1037.93

Table 9. System 1 on other engines without retraining. Share of candidates that run: MySQL 96%, Snowflake 83%, PostgreSQL 83%, DuckDB 80%, BigQuery 76%; stricter GROUP BY rules and typing account for most failures.

Generalization to the BIRD Hidden Test Set

On BIRD’s official hidden test set of 1,789 questions, LuthorSQL reaches 68.03% execution accuracy (69.26% Soft F1, 65.17% R-VES; Table 1), and our single-adapter variant achieves 68.19% (Appendix § 12), matching the published single-model leaderboard score of the 32B Arctic-ExCoT model [74] at a fraction of the parameter count. Our test score is 8.64 points above our dev score, a gap we cannot explain without test labels, so we rely on BIRD dev and our held-out splits for all controlled ablations.

SimpleModerateChallengingTotal
Test EX77.2464.8643.5168.03
Test Soft F177.7966.7445.7769.26
Test R-VES75.0460.8540.6965.17
Dev EX69.4145.6939.3159.39
Test − dev EX+7.83+19.17+4.20+8.64

Table 1. LuthorSQL (two adapters) on BIRD’s hidden test set (1,789 questions: 949 simple, 555 moderate, 285 challenging), run by BIRD, against our single BIRD dev scoring (original gold, 1,534 questions).

Pre-Registered Cascade Evaluations

Routing queries between System 1 and the Claude Opus 5.5 agent via our pre-registered validator margin threshold (t=3.0t = 3.0) answers roughly half of all questions locally and reduces API costs by 16–34% with no significant loss of accuracy (Figure 2, Table 2). Across three complete runs on RESERVED (348 held-out BIRD questions), the cascade handled 49.1% of queries locally on System 1 and cut total list-price cost by 34.3% pooled against Opus with candidates (and 20.9% against Opus alone in Run 1, including rented H100 GPU cost), while matching the agent’s accuracy within statistical noise (−0.29 points pooled against Opus with candidates, +0.96 points against Opus alone). On our second pre-registered benchmark combining KaggleDBQA test and Spider-Realistic (693 questions across 27 databases), the cascade routed 51.9% of questions to the agent, reducing cost by

31.7% against Opus with candidates (+1.30 points accuracy) and 16.2% against Opus alone (−0.14 points accuracy). While the first RESERVED run failed our pre-registered rule, missing its cost, time-saving and lower-bound bars (29.9% vs. 30%, 12.3% vs. 25%, and −2.60 vs. −2.50 points), the pooled runs and external benchmarks confirm that validator-margin routing consistently outperforms vote-agreement, question-length, and random routing baselines at the same call budget (Appendix § 16).

Local hardware and direct API deployment. When System 1 runs on hardware an organization already owns, its marginal cost is purely electricity; counting local compute as zero increases the cascade’s cost savings against standalone Opus to 30.1% pooled on RESERVED and 19.9% on the external benchmark (Figure 2). Furthermore, quantizing the shared backbone to 8-bit weights allows both adapters to run in about 9 GB of unified memory on a 24 GB Apple M3 laptop, matching GPU accuracy on 239 RESERVED questions (159 vs. 158 correct). Finally, when calling Opus directly through the plain Messages API, the t=3.0t = 3.0 cascade reduces API cost by 33.3% on our post-cutoff fresh benchmark at statistically indistinguishable accuracy (95.15% vs. 95.59%, p=1.0p = 1.0); on RESERVED, measured afterwards, it saved 35.3% but at 2.0 points lower accuracy (p=0.03p = 0.03; Appendix § 16.§ 6).

Benchmark Memorization vs. True Agent Generalization

Because frontier LLMs may have memorized public text-to-SQL benchmarks during pretraining, we ran pre-registered contamination probes before trusting our System 2 baseline. Given only a question and database name—with no schema and no database tool—Opus 5.5 generated executable queries that returned the gold result on 79–95% of instances across four public splits and reproduced 48–92% of gold SQL queries verbatim, down to dataset-specific alias conventions and backtick formatting (Table 8 in Appendix § 15). Standard schema-prompted evaluations on these public sets therefore mix reasoning with verbatim recall.

Spider-R.KaggleBIRD tr.Mini-Dev
Questions508185348500
EX, no schema94.6979.4679.3186.20
Gold SQL verbatim92.3%53.0%48.3%58.4%
EX with database (B0)92.7276.2280.17–
Exact completions22 / 1666 / 477 / 891 / 25

Table 8. Memorization probes on Opus 5.5 (no database, no tool). EX, no schema: given only the question and the database name; verbatim: the gold query reproduced after whitespace and quote normalization. Exact completions: of 100 half-questions, second halves reproduced word for word when told / not told the dataset name. BIRD tr.: RESERVED; Mini-Dev: the SQLite version of BIRD Mini-Dev, drawn from BIRD dev, scored on BIRD dev’s databases rather than Mini-Dev’s own copy (2 of its 500 gold queries failed to run there and count as wrong).

However, two controlled robustness tests suggest that Opus’s high accuracy as a tool-using agent comes mostly from database exploration rather than memorization. First, when we systematically renamed every table and column in the RESERVED databases using a non-Claude model and paraphrased the questions, the routed agent’s accuracy remained unchanged (72.88% vs. 72.32% on original schemas, p=1.0p = 1.0). Second, on our newly constructed benchmark of 227 verified questions over 8 databases published after the agent’s training cutoff (July–October 2026), System 1 answered 96.1% of its kept questions correctly, the routed agent scored 92.80% against 95.2% for Opus alone on the same questions (the registered endpoint), and the full cascade achieved 94.27% accuracy against 97.36% for Opus alone (a 3.08-point gap, not significant) while cutting cost by 20.1% (Table 2).

Conclusion

Organizations deploying natural-language database interfaces do not have to choose between the accuracy of a frontier LLM agent and the speed, privacy, and zero marginal cost of a small local model. LuthorSQL, a one-pass slot-filling model, reaches 68.03% execution accuracy on BIRD’s hidden test set in 0.84 s on a single GPU, and transfers zero-shot across six SQL engines without retraining. Placed in front of Claude Opus 5.5 as a fast System 1 gate, its validator margin routes roughly half of incoming queries locally and cuts end-to-end API costs by 16–34% across pre-registered held-out benchmarks with no significant loss of the agent’s accuracy. Finally, our contamination probes show that while frontier LLMs have memorized large portions of public text-to-SQL benchmarks verbatim, tool-assisted agents maintain high accuracy on obfuscated schemas and post-cutoff databases.

Limitations and Future Work

Limitations. The grammar expresses 84.6% of BIRD dev gold queries; questions needing CTEs, correlated subqueries, window functions or unseen templates usually go wrong, which is one reason to route them to an agent. The cascade has not yet been run on BIRD’s hidden test set, and its held-out tests are small (8, 27 and 8 databases), so intervals are wide and results vary between runs; RESERVED’s first registered run failed its registered bars (§ 7.3). Time savings hold against Opus with candidates but not against Opus alone, since the routed questions are the slow ones, and agent time depends on load. Costs are list prices applied to logged usage, not invoices, and the agents ran inside Claude Code, whose fixed per-session cost makes part of the saving come from running fewer sessions; through the plain API the saving is registered on the fresh set only. Opus has probably seen every public set we used (§ 7.4); the renamed-schema test blocks verbatim recall but not recall of a query’s structure, and the fresh set is easier than BIRD and was checked by LLM agents and spot-checked by hand rather than reviewed in full. On BIRD dev the agent is much stronger on confident questions, so the t=3.0t = 3.0 threshold lost 2.93 points there; on DECISION, out-of-grammar questions were routed by gold coverage, which the held-out tests avoid. These questions were identified by converting their gold SQL, information a deployed system would not have, so the DECISION cascade results assume a perfect out-of-grammar detector, and in the handoff the agent saw an empty list on these questions (70 of the 400) instead of System 1’s usually wrong candidates, which can mislead it; this may flatter the handoff’s accuracy and change its time savings. DECISION and RESERVED’s databases are in OmniSQL’s fine-tuning data. In designing VALHARD, the hard-negative perturbation categories were chosen after inspecting an earlier validator’s errors on DECISION, the same split used to evaluate VALHARD, so its +1.76-point gain on DECISION may be optimistic (the BIRD dev score of 59.39% played no part in its design). Furthermore, VALHARD narrowly missed our pre-written database-level significance bar on DECISION (database sign-flip p=0.19p = 0.19, question-level McNemar p=0.04p = 0.04), though we retained it as no subsequent validator modification outperformed it (Appendix § 11). BIRD’s test score is 8.64 points above our dev score and we cannot say why without test labels; it comes from BIRD’s emailed result and is not yet on the public leaderboard. Everything here is English with foreign-key joins; System 1 runs on MySQL, PostgreSQL, DuckDB, Snowflake and BigQuery through translation of its SQLite output rather than natively, the cascade’s agent was tested on SQLite only, and the system needs read access to the database.

Future work. Next steps are the t=3.0t = 3.0 cascade on BIRD’s hidden test set, a plain-API cost measurement, native generation in other SQL dialects, caching the validator’s schema pass per database (about half of System 1’s time), and a detector that routes out-of-grammar questions before System 1 runs.

Ethics Statement

All training and evaluation data are public; the evaluation sets are KaggleDBQA (CC BY-SA 4.0), Spider-Realistic (built from Spider dev, CC BY-SA 4.0) and the fresh set, whose sources’ licenses are in Appendix § 16.§ 4. The reported system uses BIRD (CC BY-SA 4.0), Spider (CC BY-SA 4.0) and SynSQL-2.5M (Apache-2.0); earlier rounds also used WikiSQL (Zhong et al., [75]) (BSD-3-Clause), Spider-Syn (Gan et al., [18]) (MIT), SParC and CoSQL (both CC BY-SA 4.0). OmniSQL-7B and Qwen2.5-14B-Instruct (Qwen et al., [52]) (the model that wrote R1’s rare-skeleton questions) are released under Apache-2.0. The BIRD-Platinum labels of Zhu et al. (2026), for which we found no license, were used for research only: in VALHARD’s training lists, which makes them part of the reported system, and in other validator and judge training and one generator experiment. No private or proprietary data was used. LLM outputs (answers from Opus 5.5, Sonnet, Haiku, GLM 5.3, GLM 5.3 Flash and DeepSeek V4.1 Flash, and judge verdicts) were used for evaluation, and some training questions were written by LLMs: DeepSeek V4.1 Flash and Claude sub-agents in rounds 2–3, DeepSeek V4.1 Flash for one negative-result experiment, and Qwen2.5-14B for R1’s rare skeletons. A text-to-SQL system can return a confident wrong answer to a user who cannot read SQL; the validator margin used by the cascade is one signal for flagging such cases, not a guarantee. Ibrahim Memon designed and ran the experiments and wrote the paper; Armin Karamzade gave feedback on the framing and presentation of the paper; Hanif Jhaveri advised on the work; Roy Fox supervised the work. An AI assistant (Claude, through Claude Code) wrote much of the experiment code under the first author’s direction and helped prepare the manuscript from the first author’s notes and experiment logs; the first author revised the text and checked every number against those logs, and all authors take responsibility for the content.

Acknowledgments

We thank the BIRD team for evaluating our submission on the hidden test set.

How We Got Here

All stages used public data only. The first version, a bi-encoder on the openjev Qwen3.5-4B model ([3]), encoded question and columns separately; the wrong output column caused 48–61% of its errors, so later versions put the schema in front of the question. An autoregressive detour (Qwen2.5-Coder-7B, [31]; OmniSQL-32B) scored higher but cost seconds per question, so we returned to one pass on OmniSQL-7B with grammar v3, which covers 1,297 of 1,534 dev gold queries (84.6%). Pooling two runs’ candidates gave 59.06%; the gain came from the larger candidate list, so we kept one run (R1) with up to 32 distinct plans and retrained its validator on hard negatives (VALHARD). Because no further selection change passed its bar (Appendix § 11) and 15% of dev questions lie outside the grammar, where a model that writes SQL freely does better (35.9% against 14.8% for our system on those 237 questions), we paired System 1 with an agent instead of growing the grammar.

What Did Not Work

Table 4 lists the attempts that did not survive; their scripts, costs and raw outputs are in the code release. No selection change after VALHARD passed: off-the-shelf and fine-tuned judges did not beat the trained 7B validator by the per-database test, and a judge combination that gained 2.01 points on DECISION fell to +0.29 on RESERVED after we had tried about eleven configurations on the same questions, so the first number mostly measured our search. No generation change passed either: doubling the training steps, execution-reward RL, extra LLM-written rows, corrected labels and a larger grammar all landed within noise or below. Two seeds of one recipe differ by 0.50 points on DECISION, which is why single-run generator claims needed +2.0.

AttemptΔ\Delta EXVerdict
Selection
LLM judge, top 4 (think off/on)−0.25/+0.63-0.25/+0.63no bar
8 decision-model APIs−2.39-2.39 to +1.88+1.88no pass
Two-judge agreement+2.01+2.01RES. +0.29+0.29
Fine-tuned 27B judge+0.75+0.75no pass
Gated pairwise duel+0.50+0.50kill
Third validator in the pool−0.38-0.38kill
Retrained validator (VAL2)−0.38-0.38kill
Results-aware validator−0.75-0.75kill
VAL3 recipe / joint scoring−1.01/−4.15-1.01/-4.15kill
VAL4, unseen-database lists−0.13-0.13kill
Free pick rules, same pool−0.57-0.57†no pass
Generation
RL, GRPO/pass@8+0.13/−0.25+0.13/-0.25kill
LoRA soup of two runs−1.26-1.26kill
Decode-time constraints−0.50-0.50 to +0.13+0.13drop
+33k LLM-written rows+0.50+0.50kill
Corrected training labels−0.75-0.75kill
Grammar v4 + descriptions−1.26-1.26kill
Doubling training steps−0.07-0.07‡flat
Question-variant ensemble+0.63+0.63no pass
One adapter for both jobs
Soup of runs 1 and 2−1.76-1.76miss
2-layer rerank head−1.01-1.01miss

Table 4. Attempts measured on DECISION (796 questions, original gold) against the system current at the time, with bars written before scoring except for the LLM judge (no bar: exploratory, no gain) and seven of the eight decision-model APIs (exploratory). RES. = the same change scored once on RESERVED. †On RESERVED. ‡On BIRD dev (original gold). The single-adapter rows are against the two-adapter LuthorSQL (Table 5).

SystemEXΔ\DeltaDB ppMcNemarOr.@32Sel.
Two adapters59.05–––67.8459.05
Run 158.29−0.75-0.750.594+22/−2867.5958.42
Run 258.54−0.50-0.500.750+24/−2868.5958.42
Soup57.29−1.76-1.760.156+15/−2967.5957.91
2-layer head58.04−1.01-1.010.445+20/−2867.5958.67

Table 5. Single-adapter arms on DECISION (796 questions, original gold; out-of-grammar questions count as wrong): runs 1 and 2, their weight average (soup) and run 1’s recipe with a 2-layer rerank head. Δ\Delta is against the two-adapter LuthorSQL, with the database sign-flip pp and McNemar wins/losses. Or.@32: oracle over the arm’s own list of up to 32 candidates. Sel.: the arm’s scorer picking from R1’s own candidates.

One adapter for generation and selection

BIRD judged that two separately trained adapters belong on its Overall leaderboard and invited a single-adapter resubmission for the Single Trained Model track, so we built one. It keeps R1’s rank-32 adapter and slot heads and adds a zero-initialized linear rerank head on the same adapter’s last hidden state, so no second adapter exists. Each training step combines one VALHARD validator batch with one generator chunk replayed from R1’s training mix; no DECISION, RESERVED or dev question is in the training data, and inference is unchanged with one backbone in memory.

Run 1 (2 epochs, generator weight 1.0) and run 2 (3 epochs, weight 0.5, chosen by a rule written before run 1 was scored) are 0.75 and 0.50 points below the two-adapter system, not significantly (Table 5), though the intervals are wide (run 2: [−4.06, +2.74]), so this is not evidence of equal accuracy. The gap is mostly in selection: on R1’s candidates both pick at 58.42% against VALHARD’s 59.05%. A weight average (soup) of the two runs and a 2-layer rerank head did worse. We resubmitted run 2 to BIRD for its Single Trained Model track. On the hidden test set it scores 68.19% EX (simple 77.03, moderate 65.77, challenging 43.51), 69.35% Soft F1 and 64.58% R-VES, against 68.03%, 69.26% and 65.17% for the two adapters, so one adapter loses no test accuracy. On BIRD dev (original gold, scored only after the test result came back) it scores 59.84% (918/1,534) against 59.39%.

VALHARD training data

We built VALHARD after reading the earlier validator’s misses on held-out training databases. Of the 84 questions where it missed although a right candidate was in its list, the right result was its second choice for 58; the clear error kinds were counting at the wrong grain, a dropped or wrong filter, join, literal or sort, and the wrong output columns (34 of the 84 by an automatic bucketing; 11 more were LIMIT or tie cases and 39 were unclassified). Its training lists come from the generator’s own k=32k=32 candidates on 42 BIRD-train databases (3,830 lists, with the expert-corrected BIRD-train labels of Zhu et al. (2026) (BIRD-Platinum) where available), with one live representative per result signature, and two sources of hard negatives: the closest right/wrong pair for each wrong result class

(28,450 mined, 16,804 of them in the training lists), and synthetic sqlglot edits of right queries of the same kinds (count versus count distinct, an added join, a dropped filter, a flipped order, a dropped or added projection), each kept only if it runs and returns a new, wrong, non-null result. The loss treats each result group as one option, scoring it by the log-mean-exp of its members.

Training and measurement details

Training. All training data is public: BIRD train ([40]), Spider train ([72]) and a sample of SynSQL-2.5M ([37]); each gold query is kept only if its compiled plan returns the gold result. We add “twin” questions that differ from an original in one word and have a different answer, and 5,483 questions written by Qwen2.5-14B-Instruct ([52]) for rare plan skeletons; BIRD train is repeated three times. R1 uses AdamW (learning rate 10−410^{-4} for the LoRA weights, rank 32, α=64\alpha=64, all linear layers; 10−310^{-3} for the heads), cosine decay and bf16, with 8 questions from each of 4 databases per step. It ran 6,714 steps in 6.7 h on one A100; we keep the step-6,000 checkpoint, the best by greedy accuracy on 1,455 held-out BIRD-train questions. VALHARD trained in about 2.1 h on one H100.

Latency. 200 seeded DECISION questions at batch size 1 after three warm-up questions, counting every stage and excluding model load and one-off per-database setup, on one H100 PCIe and on an A100 80 GB with a CUDA 12.2 driver (BIRD’s environment). Dollar figures use $2.50/h (H100) and $2.00/h (A100). On a 24 GB Apple M3 laptop (PyTorch on the Apple GPU), both adapters share one backbone whose linear layers hold 8-bit weights with one scale per output channel; LoRA weights, embeddings and heads stay bf16. The run covered 239 RESERVED questions on 6 of the 8 databases, with other applications open, before memory pressure from those applications stopped it; it gave the GPU run’s answer on 207 of them (87%), 159 the right against 158. In bf16 the same set-up peaks at 17.2 GB.

System 2 protocol. For the Claude agents the eight-query limit was an instruction: they sometimes batched several queries into one tool call, so compliance could not be checked exactly. The open-weight API agents in Appendix § 15 had the limit enforced in code. Each chunk of up to 25 questions is one Claude Code subagent with the SQL tool and the prompts of Appendix § 20; tokens and time come from the subagents’ usage logs. Costs are Opus 5.5 list prices (input $4/M tokens, five-minute cache writes $5/M, cache reads $0.20/M, output $20/M, output taken as the larger of logged tokens and written characters / 4), not invoices. The DECISION handoff comparison used a fixed 400-question sample with the two arms interleaved; the threshold was then chosen on all 796 after a second handoff run on the other 396. The Opus agent-alone DECISION score (77.26%) comes from an earlier October 10 run; in the Opus DECISION runs gold SQL sat in the agent’s repository behind an instruction only, while all later runs (dev, other agents, held-out tests) used a sandbox holding only the databases and the tool, and their transcripts were scanned for access outside

it. GLM 5.3, GLM 5.3 Flash and DeepSeek V4.1 Flash ran through the Ollama Cloud API in a plain tool loop at temperature 0, one question per conversation, with the eight-execution limit enforced in code.

Held-out tests. RESERVED’s registration fixed the endpoints, the bars (cost saving ≥30%\ge30\% against Opus with candidates with a bootstrap lower bound ≥20%\ge20\%; accuracy lower bound ≥−2.5\ge-2.5 points; time saving ≥25%\ge25\%) and predictions before any agent ran. System 1 ran gold-free on every question on a rented H100 ($0.00085 per question including model load), and the routing share was recorded before the cascade arm ran. Each arm ran in chunks from a single database, in interleaved waves; intervals resample databases 10,000 times.

Splits and dev looks. DECISION and RESERVED come from 8 held-out BIRD-train databases (hockey, student_loan, synthea, university, car_retails, law_episode, cs_semester, movie), split by database, grammar coverage and gold problems, 30% of each group to RESERVED. RESERVED was scored by System 1 during development (60.34% with out-of-grammar questions counted as wrong; 63.22% in Table 2, where the gold-free package answers them too), so it is not a clean final test for System 1, though never with an agent. R1’s checkpoint was selected on these databases, and many of their questions were in an earlier held-out set used before October 8 (Appendix § 17). From October 8 we allowed at most three BIRD dev looks: LuthorSQL used the first, the System 2 dev run the second.

Scoring. System 1 scores (including the dev look) use a 3 s limit per query and at most 100,000 rows; agent experiments use 30 s (BIRD dev: 60 s) and no row cap. Bars were written before scoring for the registered tests only; the DECISION handoff and cascade runs were exploratory, and the threshold was chosen by a rule written before the full run.

Additional results

System 1 alone. LuthorSQL scores 59.39% on BIRD dev (911 of 1,534; three gold queries fail to run and count as wrong), with 69.41% on simple, 45.69% on moderate and 39.31% on challenging questions, and 55.61% on the 2025 corrected gold, though it saw the original question text. One generator and one validator score about the same as the earlier system that pooled two runs (59.06%), and against OmniSQL-7B (62.58%) that pooled system was level on the 85% of dev questions the grammar covers and 21 points behind on the rest (14.77% against 35.86%). On an A100 the package picks the same SQL as the frozen run on 170 of 183 questions, with accuracy equal to within one question.

Table 7 places LuthorSQL next to the diffusion and autoregressive models evaluated by Ma et al. [45]: it is roughly level with their best diffusion result (58.7%, an 8-sample vote; they report a latency of 5.66 s without saying whether per question, against our 0.84 s per question), under different conditions and hardware, while stronger SQL-writing models remain ahead: Qwen2.5-Coder-14B at 66.1 with an 8-sample vote (61.5 as a single answer) in their runs, and OmniSQL-7B at 62.6 in ours. On BIRD dev the package’s pick differed from the scored run on 59 of 1,534 questions (3.8%), which bounds the implementation drift there.

ModelGreedyVote/pickLatency (s)
LLaDA2.1-mini (DLM)48.858.75.66
WeDLM-8B (DLM)51.453.00.18
Qwen2.5-Coder-7B (AR)50.958.20.31
Qwen2.5-Coder-14B (AR)61.566.1–
OmniSQL-7B (AR)*62.6––
Our AR writer (7B)*57.863.2§–
LuthorSQL*–59.39‖0.84

Table 7. BIRD dev EX (%, original gold). The first block is copied from Ma et al. [45], an unreviewed preprint that does not say whether BIRD’s evidence was given (v1, Tables 1 and 7, unchanged in v2’s Tables 2 and 10; latency on A100s as they report it), prompted with retrieval-augmented schema context in a setting they call zero-shot. *Our runs, given BIRD’s evidence; the writer and LuthorSQL are fine-tuned on public training data, and OmniSQL-7B is the released checkpoint decoded greedily. Vote/pick: majority vote over 8 samples (Maj@8) unless marked. §Vote over 17 candidates. ‖Validator pick from up to 32 distinct candidates, not a vote; latency is the p50 on one H100 PCIe over 200 DECISION questions, not timed on BIRD dev.

Other SQL dialects

System 1 compiles plans to SQLite. To run it on other engines we translate every candidate with sqlglot (SQLite to the target dialect), match table and column names to the target catalog without regard to case, qualify tables with their schema or dataset on Snowflake and BigQuery, and, for PostgreSQL, add type rules for what SQLite coerces implicitly (string literals against text columns, dates read as text for LIKE, SUBSTRING and comparisons, LIKE as ILIKE, JULIANDAY through the epoch); these rules were written after reading PostgreSQL errors on Mini-Dev. Translated candidates are executed on the target database and picked by the package’s rule (the validator reads SQL text, so its scores carry over). No model is retrained or rerun: we reuse the frozen package run on dev. BIRD’s MySQL and PostgreSQL dumps hold all 11 dev databases, built from BIRD’s updated dev files (5 of 11 differ from the 2024-06-27 release), so every engine is scored on those files: we copy the same SQLite files into native DuckDB files, into one Snowflake schema and one BigQuery dataset per database (columns SQLite stores with mixed types become text; 11 BigQuery column names with characters it rejects are renamed and mapped back); on all 1,534 questions a prediction is right when its rows equal the SQLite gold rows after normalizing values (numbers rounded to 6 places, dates as ISO strings). On Mini-Dev, where BIRD publishes gold for each dialect, the official rule gives 50.4% (SQLite), 49.2% (MySQL) and 48.8% (PostgreSQL; 48.4% without the type rules).

Handoff (DECISION, exploratory). Given System 1’s candidates, the agent shows no measurable change in accuracy (78.00% against 77.75%; p=1.0p = 1.0) and does less work: 28% less agent time and 18% fewer logged tokens. Of the 330 questions with candidates it took one on 279 (87% right) and wrote its own query on the other 121 (57% right); in an earlier pilot, on questions with no right candidate it scored 43.2% with candidates against 47.2% alone, so wrong candidates can mislead it. The longer prompt cancels the saving from shorter answers, so the cost per call stays at about 2 cents; the money is saved by not calling the agent.

Cascade threshold (DECISION). The rule picks t=3.0t = 3.0: System 1 answers 45.4% of the questions alone and the combined system scores 78.39%, 0.50 points below always calling the agent.

Opus alone+ System 1Change
EX (400 questions)77.7578.00+0.25
Agent time, total1,587 s1,146 s−28%
Time, median chunk91 s56 s−38%
Tool calls (all)207169−18%
Output text (chars/4)54.7k33.3k−39%
Cache writes842k1,025k+22%
Cache reads11.8M10.8M−9%
$/question (list) | ≈0.019 | ≈0.020 | ≈+4%

Table 10. Candidate handoff on a 400-question DECISION sample. Times and tokens include the same agent-harness overhead in both arms. Output text is an estimate (output characters / 4); logged output-token counts were not lower.

agent (p=0.50p = 0.50, [−1.44, +0.50]), with 54.6% of the agent calls (Table 11). Here the 126 out-of-grammar questions were routed by gold coverage; routing by the margin alone, as in the held-out tests (402 of 796 routed), gives 77.64%, 1.25 points below always calling the agent (both reuse the agent’s earlier answers, whose candidate lists were empty for those questions). Because the threshold was picked on these questions, 78.39% is an in-sample estimate; between 2.25 and 4.5 the curve stays within four questions, and leaving out one database at a time the rule still picks 3.0 on six of eight folds. The threshold of 2.5, picked earlier on the 400-question sample, failed on BIRD dev (below), so we re-chose it on all 796; RESERVED and the external sets played no part.

Margin ≥t\ge tEXAgent callsChange
always agent78.89100%–
2.077.7645.6%−1.13
2.578.1449.5%−0.75
3.078.3954.6%−0.50
3.578.2759.3%−0.63
4.078.5266.6%−0.38
5.078.7785.8%−0.13

Table 11. Confidence cascade on all 796 DECISION questions. System 1 answers alone when its validator margin is at least tt; otherwise the agent gets the handoff prompt. System 1 alone scores 58.92 here: the cascade picks among result groups, which breaks one tie differently from the official pick (59.05). Opus alone scores 77.26. Bold: the threshold picked by the rule fixed before the run, which charged every agent call the same average list-price cost, estimated at the time; measured held-out results are in Table 2 and Figure 2.

Other agents (DECISION). To check that the threshold is not tied to Opus we repeated both arms with five agents weaker than Opus on this split: Claude Sonnet 5.5 and Haiku 5.5 (claude-sonnet-5-5, claude-haiku-5-5), GLM 5.3, GLM 5.3 Flash and DeepSeek V4.1 Flash (Table 12). Each gains +2.51 to +4.27 points from the candidates (significant at the database level only for Haiku), and at t=3.0t = 3.0 the largest observed loss against always calling the agent is 0.13 points (only DeepSeek’s +1.01 is significant). Above a margin of 3.5 the open models are up to 3.7 points below System 1, so sending them confident questions hurts. Our predictions, written before these runs and committed with the results, failed for the chosen threshold of Haiku (2.25) and the open models (2.0, on curves flat within 0.3 points from 2.0 to 4.0) and for GLM 5.3 Flash’s predicted gain of at least 10 points (+9.55); GLM 5.3 was added after the registration. What held is that t=3.0t = 3.0 costs little with every agent we tried.

AgentAlone+ S1t=3.0t=3.0Change
Opus 5.577.2678.8978.39−0.50
Sonnet71.1174.5074.37−0.13
Haiku68.3472.6172.610.00
GLM 5.3*66.2170.4870.60+0.13
GLM 5.3 Flash*65.9568.4768.84+0.38
DeepSeek V4.1 Fl.*65.9569.2270.23+1.01

Table 12. The same cascade with six agents (all 796 DECISION questions; System 1 alone 58.92). + S1: the agent with System 1’s candidates on every question; t=3.0t=3.0: the cascade; Change: cascade minus + S1. The agent sees the same 54.6% of questions in every row, since routing does not depend on the agent; the t=3.0t=3.0 column combines System 1’s answers on the kept questions with the + S1 arm’s answers on the routed ones, rather than being a separate run. *Open-weight models run through Ollama Cloud with a plain API loop (one question per conversation, schema in the first message, the eight-query limit enforced in code) instead of Claude Code subagents, so their absolute scores are not directly comparable with the Claude rows. Fl.: Flash.

BIRD dev. On all 1,534 dev questions, Opus with System 1’s candidates scores 84.49%, 25.10 points above System 1 alone and not significantly different from Opus alone (84.03%; p=0.70p = 0.70). The cascade saves less there: at t=3.0t = 3.0 it scores 81.55%, and the registered t=2.5t = 2.5 scored 80.70%, 3.78 points below always calling the agent, failing its non-inferiority bar. System 1’s confidence transferred (its accuracy by margin band is close to DECISION’s above 2.5), but the agent’s gain did not: above 2.5 Opus adds 1.2–2.6 points per band on DECISION and 12–15 on dev (Figure 3). Part of that may be recall: 20.9% of Opus’s answers alone match the gold SQL text, and on 83 dev questions whose gold result changed in BIRD’s 2025 revision, Opus with candidates leans to the old gold (39 against 26) while System 1 leans to the new (24 against 15), though with 83 questions none of these splits is significant. The lesson for a cascade is to measure the agent’s gain over the small model in each confidence band on questions the agent has not seen.

Accuracy by System 1’s validator margin

Figure 3. Accuracy by System 1’s validator margin (gap between its top two result groups; “none” = fewer than two groups, including every out-of-grammar question). System 1’s accuracy rises with its margin on both splits, so the margin transfers. The agent’s gain does not: above a margin of 2.5, Opus with candidates adds 1.2–2.6 points per band on DECISION (796 questions, 77–152 per band) but 12.5–15.4 in the two bands from 2.5 to 4.1 on BIRD dev and 2.2–2.8 above them (1,534 questions, 84–237 per band), which is why a threshold chosen on DECISION loses more on dev. Original gold.

Held-out tests, probes and tuning

Every experiment here except the exploratory search at the end was registered, with endpoints and decision rules, before any agent ran on its data (Appendix § 16.§ 7), and every agent transcript was scanned for access outside its sandbox.

External test

Against Opus with candidates, total time per question (summed agent time plus System 1’s GPU time) fell by 12.3%, 23.6% and 21.7% in the three RESERVED runs (19.3% pooled; 2.90 against 3.59 s per question on average) and by 17.1% [12.5, 21.4] on the external test. Against a prompt-only control with empty candidate lists, Opus with candidates scored −0.9, +1.7 and +1.4 points in the three RESERVED runs (all p>0.5p > 0.5), so the candidates add no clear accuracy. KaggleDBQA’s test split (185 questions, 8 databases) and Spider-Realistic (508 questions, 19 databases) were run blind: System 1 saw questions and schemas without SQL, and the agents only the databases, column descriptions and the tool. The registered claim pooled both sets (27 database clusters). On the reg-

ArmAgent callsEXCorrected
System 1 alone0%59.3955.67*
Opus alone100%84.0377.25
Opus + System 1100%84.4976.27
Cascade t=2.5t=2.549.5%80.7072.88
Cascade t=3.0t=3.055.0%81.5573.66

Table 13. All 1,534 BIRD dev questions (our second dev look). EX uses the original gold; Corrected rescores the same answers against BIRD’s 2025 revision of the same question ids and is not a leaderboard number. *55.61 under the 3 s System 1 scorer (§ 7); this table uses the 60 s scorer with no row cap; the two differ by one question, which we have not traced. t=2.5t=2.5 was the threshold pre-registered for this run from the 400-question sample; t=3.0t=3.0 is shown for reference and was chosen after.

istered clean subset of 130 KaggleDBQA questions, without two databases that overlap System 1’s training data, the saving against Opus with candidates was 18.0% [3.6, 32.9]; per set it was 25.3% on KaggleDBQA and 34.6% on Spider-Realistic. Simulated with the candidate arm’s answers at the same call rate, routing by the margin scored 85.57 against 81.82 for vote agreement, 80.81 for question length and 78.89 at random.

Memorization probes and renamed schemas

Probe agents had no database and no tool. In the schema-free probe each got the question and database name and wrote one SQLite query, scored by execution and by verbatim match after normalizing whitespace, quotes, case and backticks. In the completion probe each question was cut after half its words and completed with or without the dataset’s name, in separate agents; guided minus general ROUGE-L was +0.11 to +0.53 on the four sets (p<0.001p < 0.001 each). For the renamed-schema test, a non-Claude model (DeepSeek V4.1 Flash) renamed every table and column of the 8 RESERVED databases with no shared word of three or more letters; gold and candidates were rewritten with a SQL parser and kept only if they returned the original result (all 177 gold queries passed), and questions and evidence were reworded with every literal kept. The routed agent ran unchanged.

Fresh unseen set

Eight SQLite databases were built, by Claude sub-agents doing data plumbing only, from public data dated 1 July 2026 or later (small reference tables excepted), each with at least three related tables and declared foreign keys: FDA enforcement reports (openFDA, CCO), USGS earthquakes and NWS alerts (US public domain), HM Land Registry price-paid data (OGL v3.0: “Contains HM Land Registry data © Crown copyright and database right 2021. This data is licensed under the Open Government License v3.0.”), the 2026 World Cup knockout matches (openfootball, CCO), Chicago crimes and traffic crashes (City of Chicago Data Terms of Use) and Formula 1 races (Jolpica-F1, CC BY-NC-SA 4.0). An MLB database was excluded because its terms forbid redistribution. DeepSeek V4.1 Flash wrote 40 questions per database with evidence and gold SQL that had to execute, be deterministic and return 1 to 1,000 rows; GLM 5.3 judged each item and the first 30 per database were kept. By an amendment recorded before any model ran, independent LLM agents (model not recorded; possibly Claude) wrote their own query before seeing the gold and judged all 240 items; 13 items with a majority of “wrong” verdicts were removed, leaving 227, and an author checked 20 by hand and found all correct. Counting the 13 removed items as misses gives 92.1% for Opus alone and 89.2% for the cascade. The questions often name columns or codes, so they are easier than BIRD’s.

Raising accuracy at the same cost

We registered variants of the routed agent and tested them on RESERVED’s renamed copies, split by database into a development and a confirmation half (Table 14); none gained the required 2 points, and the best combination, run end to end on all of RESERVED, scored 80.17% against the cascade’s 80.46% at a saving of only 19.1%, so it was not adopted. While scoring them we found and fixed a bug that had costed the baseline over all routed questions but each variant over its half, which had made the top-3 variant look 20% cheaper instead of 11%. An exploratory search with 55 agents on DECISION only ($2.98 of GPU) found no routing change worth more than 0.63 points.

Variant (routed questions)EX− V0Cost vs V0
Development half, renamed (92 questions; V0 76.09)
Answer first76.090.001.09
Answer first, top 3 cands75.00−1.090.92
Second attempt, top 376.090.000.89
+ independent 2nd agent, vote72.83−3.261.71
Confirmation half, renamed (85 questions; V0 69.41)
Second attempt, top 363.53−5.881.14

Table 14. Attempts to raise the routed agent’s accuracy at the same cost. V0: the cascade’s routed arm. Cost: agent dollars per question relative to V0 on the same questions.

Opus through the API

The API agent (experiments/plain_api/) calls Opus 5.5 through the Messages API with one conversation per question: the API-agent prompt of Appendix K, the schema output of the SQL tool in the user message, the read-only SQL tool with the eight-execution limit enforced in code, a submit tool, at most 14 turns, default sampling, no extended thinking, and prompt caching on the system prompt and the schema, so a database’s schema is written to the cache once. Costs are its logged usage at the list prices of Section 5. The first registered API test on RESERVED kept the Claude Code design, in which each routed question also shows System 1’s top eight candidate groups. Through the API those candidates add about 1,250 uncached input tokens and a third more output per routed question ($0.0223 against $0.0122 without them) and no accuracy (86.21% against 86.49% for Opus alone), so that cascade cost 18.3% [2.3, 31.3] more than Opus alone, at 84.20% accuracy; inside Claude Code the same difference was hidden by the fixed session cost. Forwarding only the question was then registered for the fresh set before any API call there (Section 6.3). The fresh run cost $2.47 and the RESERVED run $9.62.

Registration record

The registrations were private files in our development repository, each hashed (SHA-256) when written, with run records and amendments appended below the registered text; every stored hash equals that of the file’s opening section except the external test’s, whose placeholder for run details was filled in place, so its hash matches its first committed version. Table 15 gives the commit times, which are our own records rather than an external timestamp; RESERVED run 1’s registration and the cross-agent predictions were first committed together with their results. Future registrations will use an external registry.

ExperimentRegisteredResult
RESERVED run 1both 20th21:45
RESERVED runs 2–3, external20th 22:11 22:37; 21st 00:04
Memorization probe21st 00:1200:19
Mini-Dev probe21st 08:5409:00
Renamed schemas21st 11:0211:21
Fresh set21st 14:2917:55
Tuning rounds 1–221st 18:42, 18:5418:47, 18:57
Final configuration C*21st 19:1819:27
API, RESERVED26th 12:2212:44
API, fresh set (question only)26th 13:1513:20

Table 15. Commit times of each registration and of its first result in the development repository.

Development-set use and possible leakage

BIRD dev was a development set in earlier rounds; after October 8, 2026 nothing was fitted or tuned on it, and LuthorSQL’s configuration was hashed before its one scoring run. Some evidence-repair rules of earlier systems were written after reading dev misses; we report those systems with a version rebuilt without them, and LuthorSQL does not use that repair. Two Spider-train databases overlap dev (formula_1; soccer_1 with european_football_2); retraining without them changed DECISION accuracy by −1.01-1.01 points (p=0.42p = 0.42), and on the other nine dev databases LuthorSQL scores 60.52%. OmniSQL’s fine-tuning data contains all of BIRD train, so DECISION and RESERVED are held out from our adapters only. Dev gold has been public since 2023, and Opus’s answers alone match its gold text on 20.9% of questions, so every agent setting was chosen on DECISION and dev agent scores are not treated as estimates for unseen questions.

Compute and reproducibility

Provider billing records show 227.10 rented GPU-hours and $464.46 from September 30 to October 13, 2026 (121.29 A100-hours, 96.38 H100-hours, 9.42 on other cards), covering every stage in Table 3, failed runs included; the later single-adapter work cost about $26.6 by our run logs. R1 trained on one A100 80 GB and VALHARD on one H100. The package (Python 3.12, PyTorch 2.11.0, Transformers 5.19.0, PEFT 0.21.2) loads the 7B backbone once per adapter (peak about 42 GB) and answers 1.09 questions per second batched on one H100 PCIe. It was sent to BIRD on October 13, 2026, and BIRD’s test result reached us on October 15. The DECISION and RESERVED ids are in the ancillary files with their SHA-256 hashes (of the ids sorted as strings and joined by newlines). The code, registrations, scorers, result files and prompts are at https://github.com/IAMIbrahimemon/luthorsql; the adapters at https://huggingface.co/IAMIbrahim/luthorsql-bird (single adapter: luthorsql-bird-single) and the fresh set at https://huggingface.co/datasets/IAMIbrahim/luthorsql-fresh-unseen, for research use.

Date (2026)StageModelBIRD devWhat it taught us
Sep 30Round 14B bi-encoder fills a plan19.3Wrong output column caused 48–61% of errors; question and schema never met inside the model.
Oct 1–3Rounds 2–34B, schema in context, reranker–Grammar v2 covers 70.9% of dev; best held-out system 73.3% against a top-8 ceiling of 77.3%.
Oct 3–4Rounds 4–57B autoregressive SQL writer63.2*Writing SQL beat filling slots on the same held-out list (first answer 83.4 against 78.4 for the slot model); a trained selector lost 2.7 points to plain voting.
Oct 4Round 6OmniSQL-32B, three prompts68.3†A 32B track would at best edge past open 32B models; we went back to one pass.
Oct 4Slot pilotOmniSQL-7B slot model37.5Grammar v3 lifts coverage to 84.6%; 42 ms per question.
Oct 5–7R1, R2+ validator, evidence repair56.78/56.71Validator over top 8 and a deterministic repair; doubling the training steps changed nothing.
Oct 8ENS1two runs pooled59.06The gain came from a larger candidate pool, not from voting validators.
Oct 9–11System 1R1, k=32k=32, VALHARD59.39One generator and one validator score about the same as the pooled pair; frozen and scored once.
Oct 12System 1+2+ Opus 5.5 agent–Handoff and cascade measured on held-out training databases (Appendix § 15).
Oct 18System 1+2all of dev, threshold on 79684.49‡Agent with candidates; the cascade threshold from 400 questions failed on dev and was re-chosen on all 796 (t=3.0t = 3.0).

Table 3. Milestones, with BIRD dev execution accuracy (%, original gold, all 1,534 questions unless marked). *Execution vote over greedy plus 16 samples; greedy 57.8. †Corrected 2025 gold, vote over three prompt styles; not comparable to the other rows. Rounds 2–4 were not scored on BIRD dev; the Rounds 4–5 row gives round 5’s score. ‡With Opus 5.5 as the agent on every question; possibly partly inflated by memorization (§ 7.4).

Plan slots

Table 16 lists the slots of the plan grammar (v3).

SlotValues
projections (1–4)schema column or row count
aggregationnone, count, count distinct, sum, avg, min, max
expr. template533 templates (422 over columns, 111 over projections)
numeric constantclosed list (71)
conditional aggregatecondition on projection 1, 2 or both
filters (0–4)column, operator, literal pointer
subquery filterone inner plan (one projection)
group by (0–2), order bycolumns, direction
HAVING (0–1)aggregate, column, operator, literal
limit, distinctsmall choices
date part, ratio, ORchoices
time grainday, week, month, quarter, year
set operationunion, intersect, except, with a second plan
derived table (wrap)aggregate over inner group-by
required tables (0–2)tables joined without a used column
join hintspreferred bridge tables / keys

Table 16. Slots of the plan grammar (v3). Ranges give how many of each slot a plan can hold.

System 2 prompt and tool

The routed agent received the prompt below, one per chunk of up to 25 questions, with CHUNK and OUT replaced by file paths. The agent-alone prompt, an example candidate list and the plain-API loop for the open-weight agents are in the code release. The tool is a read-only SQLite runner: --schema prints the CREATE statements, three sample rows per table and BIRD’s column descriptions, and a query prints the column names and at most 20 rows.

You are answering BIRD text-to-SQL questions (SQLite) together
with a fast text-to-SQL system. For each question that
system already proposed candidate queries; they are
grouped by identical execution result and listed best
first by its own confidence (n_of_32 = how many of its
32 samples gave that result; "result" = a short preview
of the rows). Some questions have no candidates.

Input: CHUNK (JSONL: question_id, db_id, question, evidence,
candidates[{label, sql, result, n_of_32}]).

Output: append one JSON line per question to OUT as soon as it
is done:
{"question_id": <exactly as given>, "pick": "<label>"}
if one candidate answers the question correctly,
{"question_id": <exactly as given>, "pick": "NONE", "sql":
"<your own SQLite query>"} if none is correct or there
are none.
Exactly one line per question.

Tool (the ONLY thing you may use besides reading CHUNK and
writing OUT):
python3 TOOLS/sql.py <db_id> --schema        # tables,
    sample rows, column descriptions
python3 TOOLS/sql.py <db_id> "<SQL>"          # runs read-
    only SQL, prints up to 20 rows
Get each DB's schema only if you need it, and reuse it for
later questions on the same DB.

Rules (strict -- this is a benchmark):
- Do NOT read any other file or directory (no repo files, no
  gold SQL, no other chunks/answers, no web).
- At most 8 SQL executions per question. Use the candidates to
  save work: when one is clearly right, pick it without
  extra exploration; write your own SQL only when you are
  confident every candidate is wrong.
- Judge correctness as the benchmark does: the result set must
  equal the intended answer (row order ignored). Use the
  evidence: it defines terms and formulas as the benchmark
  intends. Exactly the asked columns, nothing extra.
- If you write any helper script, put it ONLY in OUT's dir +
  "/../work/<chunk name>/"; never create or edit shared
  files (other agents run in parallel).
Final message: "done N/N, picked K, own M".

Further related work

Autoregressive text-to-SQL. Early systems used grammar-constrained and execution-guided decoding [58, 65], alongside encoders that link question words to schema items [63, 38]; LLM systems later added decomposition, self-correction, multi-agent pipelines and retrieval-based schema linking [51, 60, 64], and benchmarks such as Spider 2.0 now pose enterprise-scale workflows for such agents [36]. Several high-scoring open models build on large SQL corpora: CodeS [39] pretrains open models for SQL, OmniSQL [37] fine-tunes 7B to 32B models on millions of synthetic examples, and Arctic-Text2SQL-R1 [69] trains with execution rewards starting from OmniSQL checkpoints. We use OmniSQL-7B as our backbone but never decode text from it.

Slot filling and joins. Before multi-table benchmarks, SQLova, X-SQL and HydraNet filled WikiSQL’s sketch slots from pretrained encoders [32, 28, 44]. RYANSQL is the closest multi-table precedent: it splits nested SQL into non-nested SELECT statements and fills each with sketch-based slot filling on Spider [12]; our grammar is flatter (no nesting) but fills its slots from one pass of an LLM backbone and returns kk-best plans. Our compiler’s join inference over the foreign-key graph follows natural-language interfaces that connect the tables a query mentions with a Steiner tree over the schema or ontology graph, as in ATHENA and Templar [56, 5]. Pointing at stored values rather than generating them, as our filter slots do, appears in BRIDGE and ValueNet [42, 8].

Selection and verification. Contextual AI’s BIRD system trains a reward model to pick among a generator’s own candidates [2], as VALHARD trains on hard negatives drawn from our generator’s candidate lists; it is described in a blog post and code release, not a paper.

Robustness and contamination. Rajkumar et al. [53] gave Codex questions with no database information and argued qualitatively against memorization of Spider; our schema-free probe makes the same kind of test systematic and adds verbatim matching. Dr.Spider and Spider-Syn perturb schemas and questions [9, 18], as our renamed-schema test does for the agent, and LiveSQLBench releases questions over newly built databases [7], the idea behind our post-cutoff set.

References

  1. [1]Pranjal Aggarwal, Aman Madaan, Ankit Anand, Srividya Pranavi Potharaju, Swaroop Mishra, Pei Zhou, Aditya Gupta, Dheeraj Rajagopal, Karthik Kappaganthu, Yiming Yang, Shyam Upadhyay, Manaal Faruqui, and Mausam. 2024. AutoMix: Automatically mixing language models. In Advances in Neural Information Processing Systems, volume 37, pages 131000–131034.arxiv.org/abs/2310.12963
  2. [2]Sheshansh Agrawal and Thien Nguyen. 2025. Open-sourcing the best local text-to-SQL system: Lessons from BIRD and the path to enterprise-scale. Contextual AI blog.
  3. [3]AlexWortega. 2026. openjev: Qwen3.5 trained as a jev model. Hugging Face model card. Checkpoint qwen3.5-4b-nli-v5. MIT license. Accessed 2026-10-01.
  4. [4]Diogo Almeida. 2026. Introducing system one models & Jev. TypeSafe AI blog. Accessed 2026-10-21.
  5. [5]Christopher Baik, H. V. Jagadish, and Yunyao Li. 2019. Bridging the semantic gap with SQL query logs in natural language interfaces to databases. In 2019 IEEE 35th International Conference on Data Engineering (ICDE), pages 374–385. IEEE.DOI
  6. [6]BIRD Team. 2025a. BIRD-SQL dev set, 2025-11-06 revision. Accessed October 26, 2026 (commit 3c11fb1).
  7. [7]BIRD Team. 2025b. LiveSQLBench: A dynamic and contamination-free benchmark for evaluating LLMs on real-world text-to-SQL tasks. https://github.com/bird-bench/livesqlbench. Project page https://livesqlbench.ai. Not peer-reviewed.
  8. [8]Ursin Brunner and Kurt Stockinger. 2021. ValueNet: A natural language-to-SQL system that learns from database information. In 2021 IEEE 37th International Conference on Data Engineering (ICDE), pages 2177–2182. IEEE.arxiv.org/abs/2006.00888
  9. [9]Shuaichen Chang, Jun Wang, Mingwen Dong, Lin Pan, Henghui Zhu, Alexander Hanbo Li, Wuwei Lan, Sheng Zhang, Jiarong Jiang, Joseph Lilien, Steve Ash, William Yang Wang, Zhiqiu Wang, Vittorio Castelli, Patrick Ng, and Bing Xiang. 2023. Dr.Spider: A diagnostic evaluation benchmark towards text-to-SQL robustness. In International Conference on Learning Representations (ICLR).arxiv.org/abs/2301.08881
  10. [10]Bei Chen, Fengji Zhang, Anh Nguyen, Daoguang Zan, Zeqi Lin, Jian-Guang Lou, and Weizhu Chen. 2023. CodeT: Code generation with generated tests. In International Conference on Learning Representations (ICLR).arxiv.org/abs/2207.10397
  11. [11]Lingjiao Chen, Matei Zaharia, and James Zou. 2024. FrugalGPT: How to use large language models while reducing cost and improving performance. Transactions on Machine Learning Research. Featured Certification.arxiv.org/abs/2305.05176
  12. [12]DongHyun Choi, Myeong Cheol Shin, EungGyun Kim, and Dong Ryeol Shin. 2021. RYANSQL: Recursively applying sketch-based slot fillings for complex text-to-SQL in cross-domain databases. Computational Linguistics, 47(2):309–332.arxiv.org/abs/2004.03125
  13. [13]Xiang Deng, Ahmed Hassan Awadallah, Christopher Meek, Oleksandr Polozov, Huan Sun, and Matthew Richardson. 2021. Structure-grounded pretraining for text-to-SQL. In Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies.DOI
  14. [14]Dujian Ding, Ankur Mallick, Chi Wang, Robert Sim, Subhobrata Mukherjee, Victor Rühle, Laks V. S. Lakshmanan, and Ahmed Hassan Awadallah. 2024. Hybrid LLM: Cost-efficient and quality-aware query routing. In International Conference on Learning Representations (ICLR).arxiv.org/abs/2404.14618
  15. [15]Li Dong and Mirella Lapata. 2018. Coarse-to-fine decoding for neural semantic parsing. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics, pages 731–742.arxiv.org/abs/1805.04793
  16. [16]Rotem Dror, Gili Baumer, Segev Shlomov, and Roi Reichart. 2018. The hitchhiker’s guide to testing statistical significance in natural language processing. In Proceedings of the 56th Annual Meeting of the Association of the Association for Computational Linguistics (Volume 1: Long Papers), pages 1383–1392.DOI
  17. [17]Ju Fan, Zihui Gu, Songyue Zhang, Yuxin Zhang, Zui Chen, Lei Cao, Guoliang Li, Samuel Madden, Xiaoyong Du, and Nan Tang. 2024. Combining small language models and large language models for zero-shot NL2SQL. Proceedings of the VLDB Endowment, 17(11):2750–2763.DOI
  18. [18]Yujian Gan, Xinyun Chen, Qiuping Huang, Matthew Purver, John R. Woodward, Jinxia Xie, and Pengsheng Huang. 2021a. Towards robustness of text-to-SQL models against synonym substitution. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers), pages 2505–2515, Online. Association for Computational Linguistics.
  19. [19]Yujian Gan, Xinyun Chen, Jinxia Xie, Matthew Purver, John R. Woodward, John Drake, and Qiaofu Zhang. 2021b. Natural SQL: Making SQL easier to infer from natural language specifications. In Findings of the Association for Computational Linguistics: EMNLP 2021, pages 2030–2042, Punta Cana, Dominican Republic. Association for Computational Linguistics.arxiv.org/abs/2109.05153
  20. [20]Yingqi Gao, Yifu Liu, Xiaoxia Li, Xiaorong Shi, Yin Zhu, Yiming Wang, Shiqi Li, Wei Li, Yuntao Hong, Zhiling Luo, Jinyang Gao, Liyu Mou, and Yu Li. 2024. A preview of XiYan-SQL: A multi-generator ensemble framework for text-to-SQL. Preprint, arXiv:2411.08599.arxiv.org/abs/2411.08599
  21. [21]Marjan Ghazvininejad, Omer Levy, Yinhan Liu, and Luke Zettlemoyer. 2019. Mask-predict: Parallel decoding of conditional masked language models. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), pages 6112–6121.arxiv.org/abs/1904.09324
  22. [22]Shahriar Golchin and Mihai Surdeanu. 2024. Time travel in LLMs: Tracing data contamination in large language models. In The Twelfth International Conference on Learning Representations.arxiv.org/abs/2308.08493
  23. [23]Shansan Gong, Ruixiang Zhang, Huangjie Zheng, Jiatao Gu, Navdeep Jaitly, Lingpeng Kong, and Yizhe Zhang. 2026. DiffuCoder: Understanding and improving masked diffusion models for code generation. In The Fourteenth International Conference on Learning Representations (ICLR).arxiv.org/abs/2506.20639
  24. [24]Satya Krishna Gorti, Ilan Gofman, Zhaoyan Liu, Jiapeng Wu, Noël Voutisis, Guangwei Yu, Jesse C. Cresswell, and Rasa Hosseinzadeh. 2025. MSc-SQL: Multi-sample critiquing small language models for text-to-SQL translation. In Proceedings of the 2025 Conference of the Nations of the Americas Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume 1: Long Papers), pages 2145–2160, Albuquerque, New Mexico. Association for Computational Linguistics.
  25. [25]Jiatao Gu, James Bradbury, Caiming Xiong, Victor O.K. Li, and Richard Socher. 2018. Non-autoregressive neural machine translation. In International Conference on Learning Representations.arxiv.org/abs/1711.02281
  26. [26]Jiaqi Guo, Zecheng Zhan, Yan Gao, Yan Xiao, Jian-Guang Lou, Ting Liu, and Dongmei Zhang. 2019. Towards complex text-to-SQL in cross-domain database with intermediate representation. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 4524–4535.arxiv.org/abs/1905.08205
  27. [27]Hojae Han, Yeonseok Jeong, Seung-won Hwang, Zhewei Yao, and Yuxiong He. 2026. R^3-SQL: Ranking reward and resampling for text-to-SQL. In Findings of the Association for Computational Linguistics:
  28. [28]Pengcheng He, Yi Mao, Kaushik Chakrabarti, and Weizhu Chen. 2019. X-SQL: Reinforce schema representation with context. Preprint, arXiv:1908.08113.arxiv.org/abs/1908.08113
  29. [29]Edward J. Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzi Li, Shean Wang, Lu Wang, and Weizhu Chen. 2022. LoRA: Low-rank adaptation of large language models. In International Conference on Learning Representations.arxiv.org/abs/2106.09685
  30. [30]Liang Huang and David Chiang. 2005. Better k-best parsing. In Proceedings of the Ninth International Workshop on Parsing Technology, pages 53–64.
  31. [31]Binyuan Hui, Jian Yang, Zeyu Cui, et al. 2024. Qwen2.5-Coder technical report. Preprint, arXiv:2409.12186.arxiv.org/abs/2409.12186
  32. [32]Wonseok Hwang, Jiyeong Yim, Seunghyun Park, and Minjoon Seo. 2019. A comprehensive exploration on WikiSQL with table-aware word contextualization. Preprint, arXiv:1902.01069. KR2ML Workshop at NeurIPS 2019.arxiv.org/abs/1902.01069
  33. [33]Wittawat Jitkrittum, Neha Gupta, Aditya Krishna Menon, Harikrishna Narasimhan, Ankit Singh Rawat, and Sanjiv Kumar. 2023. When does confidence-based cascade deferral suffice? In Advances in Neural Information Processing Systems, volume 36, pages 9891–9906. Curran Associates, Inc.arxiv.org/abs/2307.02764
  34. [34]Daniel Kahneman. 2011. Thinking, Fast and Slow. Farrar, Straus and Giroux, New York.
  35. [35]Chia-Hsuan Lee, Oleksandr Polozov, and Matthew Richardson. 2021. KaggIeDBQA: Realistic evaluation of text-to-SQL parsers. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers).
  36. [36]Fangyu Lei, Jixuan Chen, Yuxiao Ye, Ruisheng Cao, Dongchan Shin, Hongjin Su, Zhaoqing Suo, Hongcheng Gao, Wenjing Hu, Pengcheng Yin, Victor Zhong, Caiming Xiong, Ruoxi Sun, Qian Liu, Sida Wang, and Tao Yu. 2025. Spider 2.0: Evaluating language models on real-world enterprise text-to-SQL workflows. In The Thirteenth International Conference on Learning Representations (ICLR).arxiv.org/abs/2411.07763
  37. [37]Haoyang Li, Shang Wu, Xiaokang Zhang, Xinmei Huang, Jing Zhang, Fuxin Jiang, Shuhai Wang, Tieying Zhang, Jianjun Chen, Rui Shi, Hong Chen, and Cuiping Li. 2025. OmniSQL: Synthesizing high-quality text-to-SQL data at scale. Proceedings of the VLDB Endowment, 18(11):4695–4709.arxiv.org/abs/2503.02240
  38. [38]Haoyang Li, Jing Zhang, Cuiping Li, and Hong Chen. 2023a. REDSQL: Decoupling schema linking and skeleton parsing for text-to-SQL. In Proceedings of the AAAI Conference on Artificial Intelligence, pages 13067–13075.
  39. [39]Haoyang Li, Jing Zhang, Hanbing Liu, Ju Fan, Xiao-aokang Zhang, Jun Zhu, Renjie Wei, Hongyan Pan, Cuiping Li, and Hong Chen. 2024. CodeS: Towards building open-source language models for text-to-SQL. Proceedings of the ACM on Management of Data, 2(3):1–28.arxiv.org/abs/2402.16347
  40. [40]Jinyang Li, Binyuan Hui, Ge Qu, Jiaxi Yang, Binhua Li, Bowen Li, Bailin Wang, Bowen Qin, Ruiying Geng, Nan Huo, Xuanhe Zhou, Chenhao Ma, Guoliang Li, Kevin C. C. Chang, Fei Huang, Reynold Cheng, and Yongbin Li. 2023b. Can LLM already serve as a database interface? A Big bench for large-scale database grounded text-to-SQLs. In Advances in Neural Information Processing Systems (Datasets and Benchmarks Track), volume 36.
  41. [41]Bill Yuchen Lin, Yicheng Fu, Karina Yang, Faezeh Brahman, Shiyu Huang, Chandra Bhagavatula, Prithviraj Ammanabrolu, Yejin Choi, and Xiang Ren. 2023. SwiftSage: A generative agent with fast and slow thinking for complex interactive tasks. In Advances in Neural Information Processing Systems, volume 36, pages 23813–23825.arxiv.org/abs/2305.17390
  42. [42]Xi Victoria Lin, Richard Socher, and Caiming Xiong. 2020. Bridging textual and tabular data for cross-domain text-to-SQL semantic parsing. In Findings of the Association for Computational Linguistics: EMNLP 2020, pages 4870–4888, Online. Association for Computational Linguistics.arxiv.org/abs/2012.12627
  43. [43]Yifu Liu, Yin Zhu, Yingqi Gao, Zhiling Luo, Xiaoxia Li, Xiaorong Shi, Yuntao Hong, Jinyang Gao, Yu Li, Bolin Ding, and Jingren Zhou. 2026. XiYan-SQL: A novel multi-generator framework for text-to-SQL. IEEE Transactions on Knowledge and Data Engineering, 38(4):2474–2487. https://doi.org/10.1109/TKDE.2026.3657851.
  44. [44]Qin Lyu, Kaushik Chakrabarti, Shobhit Hathi, Souvik Kundu, Jianwen Zhang, and Zheng Chen. 2020. Hybrid ranking network for text-to-SQL. Preprint, arXiv:2008.04759.
  45. [45]Peixian Ma, Xialie Zhuang, Jiantao Tan, Changlun Li, Ruirui Chen, and Chengwei Qin. 2026. Are diffusion language models good database analysts? Preprint, arXiv:2605.27791. Preprint; numbers taken from v1.arxiv.org/abs/2605.27791
  46. [47]Ansong Ni, Srini Iyer, Dragomir Radev, Veys Stoyanov, Wen-tau Yih, S ida I. Wang, and Xi Victoria Lin. 2023. LEVER: Learning to verify language-to-code generation with execution. In Proceedings of the 40th International Conference on Machine Learning, volume 202 of Proceedings of Machine Learning Research, pages 26106–26128.arxiv.org/abs/2302.08468
  47. [48]Shen Nie, Fengqi Zhu, Zebin You, Xiaolu Zhang, Jingyang Ou, Jun Hu, Jun Zhou, Yankai Lin, Ji-Rong Wen, and Chongxuan Li. 2025. Large language diffusion models. In The Thirty-ninth Annual Conference on Neural Information Processing Systems (NeurIPS).arxiv.org/abs/2502.09992
  48. [49]Isaac Ong, Amjad Almahairi, Vincent Wu, Wei-Lin Chiang, Tianhao Wu, Joseph E. Gonzalez, M Waleed Kadous, and Ion Stoica. 2025. RouteLLM: Learning to route LLMs from preference data. In The Thirteenth International Conference on Learning Representations (ICLR).arxiv.org/abs/2406.18665
  49. [50]Mohammadreza Pourreza, Hailong Li, Ruoxi Sun, Yeounoh Chung, Shayan Talaei, Gaurav Tarlok Kakkar, Yu Gan, Amin Saberi, Fatma Ozcan, and Sercan O. Arik. 2025. CHASE-SQL: Multi-path reasoning and preference optimized candidate selection in text-to-SQL. In The Thirteenth International Conference on Learning Representations (ICLR).arxiv.org/abs/2410.01943
  50. [51]Mohammadreza Pourreza and Davood Rafiei. 2023. DIN-SQL: Decomposed in-context learning of text-to-SQL with self-correction. In Advances in Neural Information Processing Systems, volume 36.arxiv.org/abs/2304.11015
  51. [52]Qwen, An Yang, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chengyuan Li, Daiyiheng Liu, Fei Huang, Haoran Wei, Huan Lin, Jian Yang, Jianhong Tu, Jianwei Zhang, Jianxin Yang, Jiaxi Yang, Jingren Zhou, Junyang Lin, and 24 others. 2024. Qwen2.5 technical report. Preprint, arXiv:2412.15115.
  52. [53]Nitarshan Rajkumar, Raymond Li, and Dzmitry Bahdanau. 2022. Evaluating the text-to-SQL capabilities of large language models. Preprint, arXiv:2204.00498.arxiv.org/abs/2204.00498
  53. [54]Federico Ranaldi, Elena Sofia Ruzzetti, Dario Onorati, Leonardo Ranaldi, Cristina Giannone, Andrea Favalli, Raniero Romagnoli, and Fabio Massimo Zanzotto. 2024. Investigating the impact of data contamination of large language models in text-to-SQL translation. In Findings of the Association for Computational Linguistics: ACL 2024.DOI
  54. [55]Ohad Rubin and Jonathan Berant. 2021. SmBoP: Semi-autoregressive bottom-up semantic parsing. In Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 311–324.
  55. [56]Diptikalyan Saha, Avrila Floratou, Karthik Sankaranarayanan, Umar Farooq Minhas, Ashish R. Mittal, and Fatma Özcan. 2016. ATHENA: An ontology-driven system for natural language querying over relational data stores. Proceedings of the VLDB Endowment, 9(12):1209–1220.DOI
  56. [57]Oscar Sainz, Jon Ander Campos, Iker García-Ferrero, Julen Etxaniz, Oier Lopez de Lacalle, and Eneko Agirre. 2023. NLP evaluation in trouble: On the need to measure LLM data contamination for each benchmark. In Findings of the Association for Computational Linguistics: EMNLP 2023.
  57. [58]Torsten Scholak, Nathan Schucher, and Dzmitry Bahdanau. 2021. PICARD: Parsing incrementally for constrained auto-regressive decoding from language models. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pages 9895–9901.
  58. [59]Freda Shi, Daniel Fried, Marjan Ghazvininejad, Luke Zettlemoyer, and Sida I. Wang. 2022. Natural language to code translation with execution. In Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, pages 3533–3546, Abu Dhabi, United Arab Emirates. Association for Computational Linguistics.arxiv.org/abs/2204.11454
  59. [60]Shayan Talaei, Mohammadreza Pourreza, Yu-Chen Chang, Azalia Mirhoseini, and Amin Saberi. 2024. CHESS: Contextual harnessing for efficient SQL synthesis. Preprint, arXiv:2405.16755.arxiv.org/abs/2405.16755
  60. [61]Mattia Tritto, Giuseppe Farano, Dario Di Palma, Gaetano Rossiello, Dharmashankar Subramanian, Fedelucio Narducci, and Tommaso Di Noia. 2026. GradeSQL: Outcome reward models for intelligent text-to-SQL generation from LLMs. Journal of Intelligent Information Systems, 64(5):2545–2568. https://doi.org/10.1007/s10844-026-01071-6.
  61. [62]Neeraj Varshney and Chitta Baral. 2022. Model cascading: Towards jointly improving efficiency and accuracy of NLP systems. In Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, pages 11007–11021, Abu Dhabi, United Arab Emirates. Association for Computational Linguistics.
  62. [63]Bailin Wang, Richard Shin, Xiaodong Liu, Oleksandr Polozov, and Matthew Richardson. 2020. RAT-SQL: Relation-aware schema encoding and linking for text-to-SQL parsers. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 7567–7578.arxiv.org/abs/1911.04942
  63. [64]Bing Wang, Changyu Ren, Jian Yang, Xinnian Liang, Jiaqi Bai, Lin Zheng Chai, Zhao Yan, Qian-Wen Zhang, Di Yin, Xing Sun, and Zhoujun Li. 2025. MAC-SQL: A multi-agent collaborative framework for text-to-SQL. In Proceedings of the 31st International Conference on Computational Linguistics, pages 540–557, Abu Dhabi, UAE. Association for Computational Linguistics.
  64. [65]Chenglong Wang, Kedar Tatwawadi, Marc Brockschmidt, Po-Sen Huang, Yi Mao, Oleksandr Polozov, and Rishabh Singh. 2018. Robust text-to-SQL generation with execution-guided decoding. Preprint, arXiv:1807.03100.arxiv.org/abs/1807.03100
  65. [66]Xuezhi Wang, Jason Wei, Dale Schuurmans, Quoc Le, Ed Chi, Sharan Narang, Aakanksha Chowdhery, and Denny Zhou. 2023. Self-consistency improves chain of thought reasoning in language models. In The Eleventh International Conference on Learning Representations (ICLR).arxiv.org/abs/2203.11171
  66. [67]Niklas Wretblad, Fredrik Riseby, Rahul Biswas, Amin Ahmadi, and Oskar Holmström. 2024. Understanding the effects of noise in text-to-SQL: An examination of the BIRD-bench benchmark. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers), pages 356–369, Bangkok, Thailand. Association for Computational Linguistics.
  67. [68]Xiaojun Xu, Chang Liu, and Dawn Song. 2017. SQL-Net: Generating structured queries from natural language without reinforcement learning. Preprint, arXiv:1711.04436.arxiv.org/abs/1711.04436
  68. [69]Zhewei Yao, Guoheng Sun, Łukasz Borchmann, Zheyu Shen, Minghang Deng, Bohan Zhai, Hao Zhang, Ang Li, and Yuxiong He. 2026. Arctic-Text2SQL-R1: Simple rewards, strong reasoning in text-to-SQL. In Findings of the Association for Computational Linguistics: ACL 2026, pages 26966–26995.arxiv.org/abs/2505.20315
  69. [70]Jiacheng Ye, Zhihui Xie, Lin Zheng, Jiahui Gao, Zirui Wu, Xin Jiang, Zhenguo Li, and Linpeng Kong. 2025. Dream 7B: Diffusion large language models. Preprint, arXiv:2508.15487.arxiv.org/abs/2508.15487
  70. [71]Tao Yu, Michihiro Yasunaga, Kai Yang, Rui Zhang, Dongxu Wang, Zifan Li, and Dragomir Radev. 2018a. SyntaxSQLNet: Syntax tree networks for complex and cross-domain text-to-SQL task. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pages 1653–1663.DOI
  71. [72]Tao Yu, Rui Zhang, Kai Yang, Michihiro Yasunaga, Dongxu Wang, Zifan Li, James Ma, Irene Li, Qingning Yao, Shanelle Roman, Zilin Zhang, and Dragomir Radev. 2018b. Spider: A large-scale human-labeled dataset for complex and cross-domain semantic parsing and text-to-SQL task. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pages 3911–3921.arxiv.org/abs/1809.08887
  72. [73]Murong Yue, Jie Zhao, Min Zhang, Liang Du, and Ziyu Yao. 2024. Large language model cascades with mixture of thought representations for cost-efficient reasoning. In International Conference on Learning Representations (ICLR).arxiv.org/abs/2310.03094
  73. [74]Bohan Zhai, Canwen Xu, Yuxiong He, and Zhewei Yao. 2025. ExCoT: Optimizing reasoning for text-to-SQL with execution feedback. arXiv preprint arXiv:2503.19988.arxiv.org/abs/2503.19988
  74. [75]Ruiqi Zhong, Tao Yu, and Dan Klein. 2020. Semantic evaluation for text-to-SQL with distilled test suites. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing, pages 396–411.arxiv.org/abs/1709.00103

Paper details

vlnc:W_72npy9ps
Version
v1 · latest
Published
Fresh — added to the Hub within the last 30 days
Added
License
CC BY 4.0
Usage

views · 2

downloads · 0

A view counts once per browser per day, a download each time the PDF is requested.

  • Computer Science

Not recorded for this paper

View statement
Looking for related papers…
Contents