WorksComputer Science
FlinkEval: A Synthetic Corpus for Evaluating Natural Language Interfaces to Streaming SQL
Abstract
Existing natural-language-to-SQL benchmarks primarily evaluate queries over bounded relational data, leaving streaming semantics underexplored. We introduce FlinkEval, an execution-grounded synthetic benchmark comprising 3,903 question–SQL pairs across 45 industry domains and 45 semantic labels. We create diverse schemas and domain-plausible, temporally consistent data through iterative generation and validation. SQL patterns collected from the human-written NL2SQL pairs and vendor documentation drive quota-balanced generation, and execution with a repair loop on Confluent Cloud Apache Flink produces reference queries covering windowing, temporal joins, pattern matching, nested types, and data manipulation. The corpus includes paired schema-explicit and business-language requests, with annotations for analysis by query pattern, complexity, and execution mode. A seven-level Question Specification Ladder examines semantics-preserving reformulation and underspecified requests augmented with oracle-derived clues. Baseline evaluations combine execution outcomes with adjudicated LLM judgments, revealing challenges in pattern composition and request specification.
Introduction
Natural-language interfaces to databases (NL2SQL) have advanced substantially through benchmarks such as Spider (Yu et al., 2018), Spider 2.0 [6], and BIRD [7]. These benchmarks primarily address queries over bounded relational data. In contrast, streaming engines such as Apache Flink operate over evolving inputs whose interpretation depends on event time, watermarks, temporal relationships, and changelog behaviour. Consequently, successful translation requires more than selecting tables and columns: the generated query must respect the target engine’s temporal and operational semantics.
We study this problem in Confluent Cloud (CC), a managed platform built on Apache Kafka and Apache Flink. Its SQL interface supports continuously updated streaming queries, bounded snapshots of stream state, and background data manipulation (DML) that writes results to Kafka topics. For example, a syntactically valid pattern-matching query may fail when its input produces updates rather than append-only events. Such constraints motivate evaluation against an operational engine (Appendix § 11.§ 1).
A second challenge concerns the specification of user intent. A business request such as “Which of my recent ad impressions were actually viewable?” may leave the data source or the meaning of “recent” implicit. Some omitted requirements may be recoverable from the available schema or domain definitions; others may require clarification. This distinction matters for evaluation: disagreement with a single reference query need not establish that a prediction is incompatible with the request [1, 8] (Sarwar et al. 2026). Conversely, a plausible prediction does not demonstrate that the user’s intended semantics have been resolved.
To investigate both challenges, we introduce FlinkEval, a synthetic benchmark grounded in a live CC Flink deployment. We generate candidate SQL from domain schemas, retain queries that pass execution validation, and construct corresponding natural-language requests. The accompanying Question Specification Ladder distinguishes identifier abstraction and paraphrasing from potentially underspecified requests augmented with oracle-derived clues. It supports comparisons across information conditions without presuming that every omitted detail constitutes genuine ambiguity.
Our evaluation addresses two questions: how reliably do current LLMs generate Flink SQL across execution modes and query patterns, and how does performance depend on the specification provided by the request? Five-model comparisons reveal substantial differences in execution reliability and performance on queries combining multiple patterns. A separate specification study finds similar accuracy under identifier abstraction and paraphrasing, but markedly lower agreement with the gold SQL for unclarified requests. The sensitivity of this result to the judging criterion motivates distinguishing reference agreement from validated resolution of user intent.
Our contributions are threefold:
A streaming NL2SQL corpus. We synthesise 50 domain schemas and temporally consistent data through structural validation, live integrity checks, and plausibility assessment (Appendix § 7), and generate 3,903 NL–SQL pairs across 45 domains and 45 semantic labels from 46 SQL patterns collected from human-written pairs and vendor documentation (Appendix § 8.1.1). Every reference SQL executes on a live CC Flink cluster and returns a non-empty result (Appendix § 8.2.1).
A five-model evaluation. We score execution outcomes with an adjudicated three-judge panel and characterise performance by query complexity, execution mode, and semantic pattern (§ 5.1).
The Question Specification Ladder. We analyse seven specification conditions under ground-truth-anchored and face-value criteria (§ 5.2).
Related Work
NL2SQL benchmarks. Spider (Yu et al., 2018) established cross-domain NL2SQL evaluation with 10,181 questions over 200 databases, and BIRD [7] added 12,751 queries over large real-world databases with external knowledge. FlinkEval complements these resources by evaluating temporal and operational constraints specific to streaming SQL.
Ambiguity, underspecification, and clarification. AmbiQT [1] constructs questions with two plausible SQL interpretations arising from lexical or structural ambiguity. CLAR-ITY (Sarwar et al., 2026) distinguishes lexical and semantic ambiguity from unanswerability and studies partially helpful clarification. PLANPOOL [8] defines underspecification through multiple plausible SQL semantics and separates execution correctness from grounded resolution. FlinkEval draws on these distinctions to compare linguistic reformulation with omitted semantic information (§ 3.1).
Synthetic data for NL2SQL. Wu et al. (2021) generate synthetic question–SQL pairs through SQL-query generation and hierarchical SQL-to-question generation, improving cross-domain text-to-SQL parsing. Complementary approaches use intermediate representations and schema linking [5], or decompose LLM-based SQL generation into reasoning and correction stages (Pourreza and Rafiei, 2023). Our approach uses execution on a live cluster to validate synthetic reference queries against the target engine’s constraints.
LLM-as-judge. LLM-based evaluation offers a scalable alternative to manual assessment, although judgments can depend on evaluator biases and calibration [9] (Zheng et al. 2023). We apply this approach to streaming SQL through panel adjudication, in which a third judge resolves disagreements between two primary judges. Comparing ground-truth-anchored and face-value criteria further exposes sensitivity to how the evaluation treats implicit requirements.
Streaming SQL. Apache Flink unifies stream and batch processing [2]. Its SQL interface provides window TVFs, watermarks, and temporal joins [3], and Confluent Cloud adds dialect extensions that FlinkEval targets.
The NL2Flink-SQL Task
Given a natural-language question and a set of Flink table DDLs , a system under evaluation produces a Flink SQL statement . FlinkEval provides a gold SQL and, for bounded queries, a reference result set obtained by executing on a live CC Flink cluster.
Question specification and underspecification
Requests vary in both linguistic form and how fully they specify intended SQL semantics. We distinguish Contextually recoverable underspecification, replacing technical identifiers with everyday expressions without intentionally altering query constraints, from Contextually unresolved underspecification, omitting or leaving implicit SQL-relevant information such as projections, sources, joins, predicates, time windows, ranking, aggregation, or business metrics.
tbl:1 defines our seven-level Question Specification Ladder, which operationalises variation in request specification rather than a universal ordering of difficulty. The table lists the levels from the most to the least specified request (L0 to L2.1); within L2 the rows run from three appended clues (L2.4) down to none (L2.1), the base request that the others extend. L1.1 and L1.2 are generated to preserve semantics and are verified for equivalence. L2.1 denotes a potentially underspecified generation condition relative to a single gold SQL; it does not establish item-level ambiguity. In the corpus, nl_input_generated stores the schema-explicit source form (L0) and nl_question_simple the unclarified business-language base form (L2.1).
Levels L2.1–L2.4 share the same base question and differ only in how many clues are appended. The LLM ranks clues by their estimated usefulness in resolving omitted SQL-relevant details (typically sources, filters, aggregation keys, or time intervals).
Evaluation metrics
We report execution outcomes and LLM judgments, which capture complementary aspects of correctness.
Execution-error rate. Execution-error rate is the proportion of evaluated predictions with a recorded error during SQL submission or result retrieval under the benchmark’s execution protocol (Appendix § 9). It captures operational failures, including parsing, validation, and observed runtime errors, rather than agreement with reference results. For streaming queries, the observation period is bounded; absence of an error does not establish indefinite execution success or semantic correctness. We do not report execution accuracy (ExAcc): comparing evolving streaming results requires aligned replay, defined observation horizons, and handling of refractions (Appendix § 9).
LLM-as-judge (judge-pass rate). Our primary metric is judge-pass rate, following the LLM-as-judge paradigm of Zheng et al. (2023) and G-Eval Liu et al. [9]. A judge LLM receives the question, the DDLs, the gold SQL and its execution output, and the predicted SQL and its execution output, returning a ternary verdict: pass (1.0), maybe (0.5), or fail (0.0). Judge-pass rate is the fraction of pass verdicts.
We use a three-judge panel. Two primary judges score each prediction independently, and a third model serves as adjudicator in case of disagreement. Judge-pass rate counts only final scores equal to 1.0. We use non-pass for all scores below 1.0.
FlinkEval Construction
FlinkEval was built iteratively, with execution on a live CC Flink cluster as the common acceptance criterion. Human expertise enters at three points: subject-matter experts (SMEs) wrote and validated the initial NL2FlinkSQL pairs from which most SQL patterns are derived, audited a sample of synthetic pairs to identify systematic defects, and formulated the engine rules that the generator enforces as guardrails and lint checks. Generation, execution and labelling are otherwise automatic.
Synthetic Generation of the Benchmark
We start from 30 NL–SQL pairs written by SMEs; their statement mix follows observed usage of the CC Flink SQL workspace and addresses everyday analytical questions rather than dialect edge cases. From these pairs and vendor documentation we derive a catalogue of 46 SQL patterns, extend every schema with an append-only, watermarked
| Level | Operational label | Specification condition |
| L0 | Schema-Explicit Specification | Fully specified NL with explicit technical schema identifiers, operations, predicates, values, and relevant parameters. |
| L1.1 | Schema-Abstracted Specification | Replace table and column identifiers with natural-language equivalents, aiming to preserve complete intended SQL semantics. |
| L1.2 | Semantics-Preserving Paraphrase | Reformulate in ordinary business language without deliberately removing necessary information. |
| L2.4 | Clue-Augmented Request (+3) | Append the top three ranked SQL-relevant clues to the L2.1 base request. |
| L2.3 | Clue-Augmented Request (+2) | Append the top two ranked clues to the same base request. |
| L2.2 | Clue-Augmented Request (+1) | Append the top ranked clue to the same base request. |
| L2.1 | Unclarified Underspecified Request | Business-language request before clues; omitted constraints may be recoverable from context or require clarification. |
caption: Question Specification Ladder in FlinkEval, ranging from schema-explicit requests to unclarified, potentially underspecified business-language requests; rows run from most to least specified. L2.2–L2.4 augment L2.1 with ranked oracle-derived clues. Results appear in § 5.2, Table 4.
| Level | Specification condition | GT% | 95% CI | Face% | 95% CI | Gap | Err.% |
| L0 | Schema-Explicit | 79.5 | 76.1–82.5 | 79.7 | 76.3–82.7 | 0.2 | 21.3 |
| L1.1 | Schema-Abstracted | 78.0 | 74.5–81.1 | 79.2 | 75.7–82.2 | 1.2 | 22.2 |
| L1.2 | Paraphrase | 78.2 | 74.7–81.3 | 78.2 | 74.7–81.3 | 0.0 | 22.2 |
| L2.4 | Clue-Augmented (+3) | 51.3 | 47.3–55.3 | 73.7 | 70.0–77.0 | 22.3 | 22.2 |
| L2.3 | Clue-Augmented (+2) | 46.3 | 42.4–50.3 | 74.2 | 70.5–77.5 | 27.8 | 21.7 |
| L2.2 | Clue-Augmented (+1) | 40.2 | 36.3–44.1 | 73.2 | 69.5–76.6 | 33.0 | 21.3 |
| L2.1 | Unclarified | 27.0 | 23.6–30.7 | 74.8 | 71.2–78.1 | 47.8 | 21.2 |
Table 4. GPT-6-Astra on the Question Specification Ladder ( test-split questions per level, release v3.1; adjudicated judge panel). Face: judge (face-value prompt v3) treating the gold SQL as one acceptable reading; gap: face minus GT (pp); err.: statements that fail to execute, which includes the 75 background-mode statements (12.5% of questions): all of them fail at L0–L1.2 and 53–60 at L2.1–L2.4. Clues are ranked oracle additions, not clarification turns. Rows follow the order of Table 1.
:::
stream table and a table with nested columns (Appendix § 7), and generate queries under a quota schedule, SME-derived dialect guardrails, and a repair loop (Appendix § 8.2.1). The guardrails encode the failure modes the SME audit exposed: time-relative predicates on historic data, streaming constructs applied to upsert tables, Top-N rankings without a unique tie-breaker, and pattern matching or OVER windows over non-append input.
Benchmark statistics
Table 2 summarises the benchmark. It contains 3,903 NL–SQL pairs over 50 schemas and 404 table DDLs in 45 industry domains, between 57 and 89 pairs per schema; four domains (hospital, insurance, retail, sports) are covered by two or three schemas. Each pair has two generated requests, the L0 schema-explicit form (49.7 words on average) and the L2.1 business-language form (24.1 words), and five further ladder variants (L1.1–L2.4), for 27,321 natural-language questions in total. All five variants pass LLM verification for 3,492 pairs (89.5%); the remaining 411 pairs keep at least one variant that failed verification, mostly L1.1 identifier abstraction (374 pairs).
| Scale | ||
| NL–SQL pairs | 3,903 | |
| Domains / schemas / tables | 45 / 50 / 404 | |
| Observed patterns / groups | 45 / 14 | |
| Generated questions (7 levels) | 27,321 | |
| Complexity tier | ||
| Tier 1 (single pattern) | 1,659 | 42.5% |
| Tier 2 (two patterns) | 1,342 | 34.4% |
| Tier 3 (three or more) | 902 | 23.1% |
| Execution mode | ||
| Snapshot | 1,677 | 43.0% |
| Streaming | 1,828 | 46.8% |
| Background (DML) | 398 | 10.2% |
Table 2. The FlinkEval corpus. Generated questions: two stored requests (L0, L2.1) and five ladder variants per pair. Per-group pair counts appear with the pattern catalogue (Table 6).
Complexity is assigned by pattern count; 57.5% of the pairs combine two or more patterns. Snapshot and streaming distinguish bounded from ongoing result retrieval; background identifies DML statements that write to sinks. These are operational categories rather than disjoint SQL pattern classes. Every group is covered, and 13 of the 14 groups contain all of their patterns (Table 6).
| Group | Name | Patterns |
| G1 | Windowed aggregation | tumble_tvf, hop_tvf, session_window_tvf, windowed_count_having, cumulate_tvf* |
| G2 | Window join | window_join |
| G3 | OVER windows | over_rows_preceding, over_range_interval_preceding, over_window_unbounded, lag_over_window, lead_over_window, avg_over_partition |
| G4 | Dedup / Top-N | dedup_rownumber_eq1, streaming_topn_rownumber |
| G5 | Stream–stream joins | stream_stream_interval_join, unbounded_inner_join, unbounded_left_join |
| G6 | Temporal / enrichment | temporal_join, multi_join_chain, two_table_join_agg, self_join_aggregation |
| G7 | Filter / conditional | filter_with_interval_bound, null_empty_filter, case_when_transform, in_exists_subquery, nullif_coalesce* |
| G8 | Snapshot / aggregation | group_by_order_limit, set_snapshot_mode, conditional_aggregation* |
| G9 | CEP | match_recognize, union_all_last_value |
| G10 | JSON / serialisation | json_value_extraction, json_value_cast, metadata_virtual_column |
| G12 | Sink / background | execute_statement_set, ctas_passthrough, regexp_replace_mask |
| G13 | Schema / configuration | set_state_ttl, faker_connector_ddl |
| G14 | Scalar expressions | convert_tz, timestampdiff, time_series_math° |
| G15 | Complex / nested types* | cross_join_unnest, nested_row_access, array_functions, json_parsing_functions |
Table 6. SQL pattern groups in FlinkEval. G11 (UDFs, extension and ML functions) is intentionally absent: the taxonomy defines it, but its constructs are disabled because they need registered JARs or models that the environment does not provide; group identifiers are kept stable rather than renumbered. *Added in the expanded catalogue. °In the catalogue but not generated, because the environment lacks a stable reference result. The patterns span all three execution modes (streaming, snapshot, background/DML).
Each pair contains the L0 and L2.1 requests, the reference SQL, semantic labels, the planned and observed execution mode, the number of repairs, and the target pattern; the ladder variants are stored in sidecar files (Appendix § 8.§ 5).
Experiments
We investigate two research questions: RQ1, how generation quality varies across models and streaming SQL constructs; and RQ2, how request specification affects reference agreement and judged acceptability. The studies share an execution and adjudication protocol but use different question samples. The model comparison (§ 5.1) was run on the held-out test split of release v3.1, and so was the Ladder study (§ 5.2).
Shared evaluation protocol. We evaluate simple baselines rather than optimising them. A single LLM receives a two-message prompt (system and user) with the natural-language question and the domain table DDLs; no few-shot examples, retrieval or repair are used. Every prediction is executed on the live Confluent Cloud cluster and scored by the adjudicated three-judge panel of § 3.2 (Claude Opus 5.5, GPT-6-Astra, Gemini 3.6 Flash, seed 0).
Both studies provide DDLs for all tables in the question’s domain as schema context (about 1.6k prompt tokens on average). The model comparison uses L0 requests; the specification study varies the level while holding the domain context fixed, enabling within-question comparisons of the information supplied by the request.
Model comparison on schema-explicit requests
Experimental setup. We evaluate every model on the same 666 questions of the held-out test split of release v3.1 (781 questions), which cover all 45 domains (11–42 per domain). We exclude the 115 test questions whose reference SQL uses a TUMBLE or HOP group window: submitted as batch queries they trigger a known Confluent Cloud Flink defect (SQL-5248) that leaves the statement pending and pages the service’s on-call engineers, so they could not be run safely. The exclusion depends only on the reference query and is therefore identical for every model. Each of five models receives the L0 question and the domain-specific table DDLs, yielding 3,330 statements that are submitted for cluster execution. The models are GPT-6-Astra, GPT-5.6-Sol, Claude Opus 5, and Gemini 3.6 Flash as frontier models, and GPT-OSS-120B as an open-weight model (served on watsonx.ai). The judging procedure completed for all 3,330 statements.
Overall comparison. The largest performance difference in Table 3 separates the four frontier models (65–80% judge-pass rate) from GPT-OSS-120B (35.0%; against every frontier model, exact McNemar). Claude Opus 5 and GPT-6-Astra are the two strongest models and cannot be distinguished (); each is significantly above GPT-5.6-Sol () and Gemini 3.6 Flash (), and GPT-5.6-Sol is above Gemini 3.6 Flash (). Tests are unadjusted. Discordant-pair counts are in Appendix § 10.
| Slice | GPT-6 | Claude | GPT-5.6 | Gemini | OSS-120B |
| All questions (666) | 79.3 | 79.6 | 70.0 | 65.0 | 35.0 |
| 95% CI (Wilson) | 76.0–82.2 | 76.4–82.5 | 66.4–73.3 | 61.3–68.5 | 31.5–38.7 |
| Execution errors | 21.5 | 18.8 | 25.5 | 30.3 | 49.4 |
| Pass & ran | 68.3 | 68.3 | 59.9 | 56.0 | 34.8 |
| Complexity tier | |||||
| Tier 1 (single pattern, 336) | 83.3 | 84.8 | 75.0 | 69.6 | 39.6 |
| Tier 2 (two patterns, 230) | 79.1 | 79.6 | 70.0 | 65.2 | 32.6 |
| Tier 3 (three or more, 100) | 66.0 | 62.0 | 53.0 | 49.0 | 25.0 |
| Execution mode | |||||
| Snapshot (216) | 83.3 | 84.3 | 75.0 | 69.4 | 44.4 |
| Streaming (369) | 74.8 | 74.0 | 64.2 | 60.4 | 37.1 |
| Background (81) | 88.9 | 92.6 | 82.7 | 74.1 | 0.0 |
Table 3. Model comparison on the 666 test-split questions (L0 question, domain DDL context, adjudicated ground-truth-anchored judge panel). Rows 1–2: judge-pass rate (%) and Wilson 95% CI. Execution errors: share of statements that fail to execute (all 81 background statements fail for every model, see text). Pass & ran: share of questions with score 1.0 and no execution error. Slice sizes in parentheses; slices are not weighted to reproduce the corpus composition and snapshot queries exclude the 115 TUMBLE/HOP questions (§ 5.1). Best value per row in bold; pattern groups are in Appendix § 10 (Table 11).
Table 3 shows lower accuracy on tier 3 than on tier 1 for every model, suggesting that pattern composition remains challenging. GPT-6-Astra scores 83.3 / 79.1 / 66.0% on tiers 1 / 2 / 3; the decline is comparable or larger for Claude Opus 5 (84.8 / 79.6 / 62.0%) and Gemini 3.6 Flash (69.6 / 65.2 / 49.0%), and the tier-3 estimates remain uncertain ().
Every model is less accurate on streaming queries than on snapshot ones, by 7–11 points. All 81 background (DML) statements in the sample fail to execute for every model: they write to sink tables that do not exist in the test environment, or return no result set. The background-mode pass rates (74–93% for the frontier models) therefore reflect judged SQL quality rather than successful execution; the same qualification applies to part of the “statements and set operations” group. GPT-OSS-120B receives no credit on any background statement: none of its 81 answers uses an EXECUTE STATEMENT SET block, and the judges treat several standalone INSERT statements as invalid for a single job.
Question Specification Ladder
To address RQ2, we compare performance across the ladder under two criteria: agreement with a single gold SQL, and acceptability under plausible readings of the presented request.
Each question is presented at the seven specification levels defined in § 3.1 (Table 1).
Experimental setup. We use the held-out test split of release v3.1 and keep the 600 questions, from all 45 domains, for which all five generated variants passed verification and whose reference SQL does not use a TUMBLE or HOP window (§ 5.1); of the 781 test questions, 115 are excluded for the window defect and a further 66 lack a complete set of verified variants. GPT-6-Astra generates SQL for each question at all seven levels, yielding 4,200 statements. The L0 statements are those of the model comparison. The same adjudicated three-judge panel evaluates every statement twice. Twelve generated statements (1 / 3 / 1 / 7 at L2.4 / L2.3 / L2.2 / L2.1) use a TUMBLE or HOP window although the reference does not; they were not submitted to the cluster and are judged without an execution result. The ground-truth-anchored criterion uses the gold SQL to interpret the request; the “face-value” criterion treats it as one acceptable reading. We report pass rates with Wilson 95% confidence intervals and compare levels on the same questions with a paired exact McNemar test.
Effect of request specification. Table 4 shows similar accuracy for the three conditions intended to preserve complete specification. Identifier abstraction and paraphrasing (L1.1, L1.2) do not significantly change the pass rate (L0 79.5%, L1.1 78.0%, L1.2 78.2%; paired exact McNemar ). The unclarified base condition (L2.1) instead yields 27.0%, 52.5 points below L0 (337 questions change from pass to non-pass and 22 in the opposite direction, ). Oracle clues partially recover reference agreement: one clue raises the pass rate by 13.2 points (27.0 → 40.2%, ), a second by a further 6.1 points (46.3%, ) and the third by 5.0 points (51.3%, ). The first clue closes about 26% of the L1.2–L2.1 gap; with three clues the rate is still 26.9 points below L1.2 ().
Sensitivity to the evaluation criterion. Face-value judgments vary less across conditions: 78.2–79.7% for L0–L1.2, 73.2–74.8% for L2.4–L2.1, a drop of 4.5 points between L1.2 and L2.4 () and no significant change among the four L2 levels (for example L2.1 vs. L2.2, ). The two criteria give similar aggregate rates for L0–L1.2 (gap 0.0–1.2 points, although at L0 59 questions pass only under face-value and 58 only under ground truth) and diverge to 22.3, 27.8, 33.0 and 47.8 points for L2.4 to L2.1; at L2.1, 303 questions pass under face-value only against 16 under the ground-truth criterion only. Reference agreement and judged plausibility thus diverge as explicit constraints are removed.
This is consistent with an information-specification bottleneck, although L2 changes wording and omits several details at once, which prevents attribution to individual omissions.
Conclusion
We introduced FlinkEval, an execution-grounded benchmark for natural-language interfaces to streaming Flink SQL. Its cross-domain corpus enables evaluation of temporal and operational requirements that are underrepresented in conventional NL2SQL benchmarks. The five-model comparison identifies two persistent challenges: generating executable statements for the target engine and preserving intended semantics when multiple query patterns are combined.
On the Ladder, unclarified requests lose most reference agreement, oracle clues recover only part of it, and the two judging criteria diverge.
Limitations
Corpus and deployment scope. FlinkEval is grounded in a single Confluent Cloud cluster and reflects the schema and data patterns of that deployment. Generalisability to other Flink environments (e.g., Apache Flink on-premise, other managed Flink services) requires additional validation.
Specification validity. The unclarified base requests (L2.1) and oracle clues (L2.2–L2.4) are LLM-generated and not human-authored, which may introduce systematic biases toward phrasings the generator itself favours. Broad paraphrasing confounds linguistic changes with semantic omissions, and the ladder lacks per-omission recoverability annotations. The single-gold reference need not exhaust plausible intents, and L1 preservation is checked by an LLM rather than an independent human study. Oracle-ranked clues do not measure deployable retrieval or interactive clarification.
Evaluation reliability. The result-stability issues in § 3.2 complicate execution-based comparison, while LLM judgments introduce evaluator uncertainty. The panel invokes adjudication on a non-negligible share of predictions (§ 5.1), and the evaluator pool overlaps with the generator families; face-value judgments are not calibrated against human annotations. The five-model comparison uses 666 questions of the v3.1 test split and a single run per model; the 115 test questions that use TUMBLE or HOP windows are excluded because of a Confluent Cloud defect (SQL-5248), so batch tumbling and hopping window aggregates are not measured in either study. The Ladder study uses a single model (GPT-6-Astra) and a single run; full-corpus evaluation is available only for GPT-6-Astra on v2. These choices limit conclusions about small model differences and run-to-run variability.
Data Generation: Schemas, Population, and Integrity
This appendix describes the construction of the database contents used by the cluster-grounded pipeline in Section B.2.1.
Schema design and coverage
The source portfolio comprises 50 domain models across 45 business domains. Multiple models can represent different aspects of the same domain. The portfolio comprises 404 table DDLs: each model contributes an event stream and a nested-type table in addition to its base tables. Its ten structural families are normalised entity networks, header–detail, many-to-many associations, hierarchies and graphs, supertype–subtype, star, snowflake, fact constellations, temporal/event history, and flexible or semistructured data. The sixteen targeted secondary features vary cardinality, optionality, key design, relationship roles, join paths, attributed associations, join depth, parallel child collections, grain and aggregation, temporal relationships, tenant-scoped identity, connectivity, denormalisation, recursive topology, subtype constraints, and flexible value shape. These describe the source schemas; the SQL-pattern taxonomy in Table 6 describes the generated queries.
For each target schema, an LLM receives the schema-design specification, the domain-specification format, summaries of the other schemas, and Flink capability constraints. It specifies each table’s row grain, columns, primary and unique keys, relationships, value rules, and append/upsert semantics, together with a real-time component, concrete data scenarios, and query intents. A deterministic validator checks structural consistency, including key references, compatible column types, tenant-scoped joins, and watermark columns. An LLM evaluator assesses whether the generated schema instantiates the intended structural family and features; both assessments inform revision. Accepted specifications are compiled into Flink DDL, including primary keys declared NOT ENFORCED and event-time watermarks.
Population workflows
The workflow-generation model receives the domain specification, generated DDL, an authoring guide with known Flink constraints, and a reference workflow. It produces dependency-ordered steps using three population strategies:
Seed: small reference and dimension tables receive fixed, domain-coherent rows with stable identifiers.
Faker: bounded, append-only Flink Faker sources generate raw identifiers, categorical choices, numbers, and time offsets. An
INSERT...SELECTcomputes the target columns, drawing foreign keys from existing parents and deriving dependent values together.Derived: child records, events, versions, and aggregate snapshots are computed from populated tables in snapshot mode. Parent-key-based identifiers and sequence numbers preserve record linkage and ordering.
Explicitly constructed records instantiate boundary conditions, such as an open-ended version, an empty nested collection, or events on opposite sides of a window boundary. Restricting these conditions to designated records preserves variation in the remaining population. Workflow execution follows the dependency order, parallelises independent steps, and records statement and assertion outcomes. Bounded generation ensures termination while retaining event-time and changelog structure in the resulting Kafka-backed tables.
Integrity validation and plausibility assessment
A static linter checks that workflow DDL matches the specification, sources are bounded, dependencies are valid, and every model table is populated and checked. Live SQL assertions then test row counts, key uniqueness, foreign-key resolution (including composite keys), allowed values, ranges, and domain-specific constraints such as non-overlapping intervals and subtype consistency. Each assertion returns a scalar compared with an expected value or bounds; statistical properties can be reported as warnings. Failed statements and assertions provide feedback for workflow revision.
An additional LLM judge reviews the seed values, Faker settings, and derivation SQL for value plausibility, within-row and cross-table coherence, codes and formats, locale coherence, and distributional and temporal plausibility. Each finding identifies the affected step and cites supporting SQL, which is verified against the workflow before revision. A repair is accepted only after linting, a fresh live population run, and re-review yield fewer errors, or the same errors with fewer warnings. The repair mechanism preserves DDL and rejects deletion of existing checks, error-to-warning downgrades, and removal of expectations.
For example, the charging workflow initially assigned 1.25 kWh to a two-second interval on a 150 kW connector, implying 2,250 kW. The repair retained the temporal boundary case but reduced its energy to approximately 0.0028 kWh and added an assertion comparing interval power with connector capacity. A documented 50-schema generation run passed all 4,207 live integrity checks; iterative plausibility assessment reduced error findings from 52 to four, leaving 46 schemas without error-level findings. These are data-generation checks, complementary to the benchmark-query execution filter in Section B.2.1. Realism judgments remain model-based: the same model reviews its own repairs, and residual temporal and distributional regularities can remain.
Metadata and regeneration
The population procedure attaches descriptive model.* metadata to Schema Registry subjects, recording table grains, keys, relationships, measures, and constraints alongside the physical schemas. These properties document the domain model; integrity is checked by the population workflow’s assertions. Versioned specifications, workflows, and generation logs support rebuilding and inspecting the environment. Fresh Faker draws and run-relative timestamps can change individual rows across population runs, so exact data reuse requires preserving the populated topics or copying a captured dataset using schema-aware export and import, which preserve the record contents while remapping schema identifiers to the destination environment.
Corpus Construction: Patterns, Generation, and Labels
Pattern acquisition
SQL pattern taxonomy
We derive SQL query patterns from the human-validated seed corpus and supplement them with patterns from the official Confluent Cloud Flink SQL documentation and how-to guides. The resulting taxonomy comprises 46 semantic patterns organised into 14 query groups (Table 6). These groups organise candidate generation. The retained corpus contains 45 observed labels, which the experiments aggregate into nine reporting groups (Appendix § 8.§ 4); generation groups and evaluation groups therefore serve different purposes.
| Source | Patterns | Assign. |
| SME-written seed pairs | 26 | 4,617 |
| Long-running statement examples | 5 | 782 |
| Flink how-to guides | 6 | 236 |
| Coverage expansion | 9 | 1,773 |
| Total | 46 | 7,408 |
Table 5. Sources of the 46 SQL patterns; Assign. counts label assignments (a pair can carry several labels).
Seed-derived patterns. We first abstracted the reference SQL of the SME-written seed pairs into reusable constructs. Each pair contributes the construct named in its rationale (temporal_join, session_window_tvf, streaming_topn_rownumber, ...), and multi-construct pairs contribute several; for instance, the consecutive-orders fraud check contributes lag_over_window and timestampdiff, and its alternative formulation match_recognize. Each pattern records the identifiers of the seed pairs it was derived from, which preserves the link between generated queries and the human-written examples: the entry in Table 7 carries suite_ids 4, 18 and 25, the seed pairs that define temporal_join. This pass yielded 26 patterns, still the bulk of the catalogue (62% of label assignments). Five seed pairs, all plain CREATE TABLE or pass-through INSERT statements, define no query pattern.
| Entry 24706241 (water domain) | |
| Pattern / provenance | temporal_join (G6); suite_ids 4, 18, 25; tier 1; streaming; one repair round |
| Tables | water_33__dim_utility and water_33__dim_region, both upsert tables with a primary key |
| L0 (schema-explicit) | Enrich each utility record from water_33__dim_utility — utility_code, utility_name and region_code — with the region_name from water_33__dim_region as it stood at the moment that utility record arrived, matching on region_code. |
| L2.1 (business) | Can you show my utilities with the name of the region each one belongs to? |
| Reference SQL | SELECT u.utility_code, u.utility_name, u.region_code, r.region_name FROM ‘water_33__dim_utility‘ u JOIN ‘water_33__dim_region‘ FOR SYSTEM_TIME AS OF u.‘$rowtime‘ AS r ON u.region_code = r.region_code |
| Stored result | e.g. (U005, Cascadia Public Water, R05, Pacific Ownership Reporting) |
Table 7. An example entry generated from the temporal_join pattern, which was derived from several seed pairs (#4, #18 and #25). The entry stores the L0 and L2.1 requests, the reference SQL executed on the cluster, and a captured result; L1.1–L2.4 variants are stored in a sidecar file (Section 5.2).
Documentation-derived patterns. We then added patterns from Confluent’s material: five from a collection of long-running statement examples (two- and three-table join aggregations, self-join statistics, and unbounded inner and left joins), and six from the CC Flink how-to guides (online), including multi-stream UNION ALL with LAST_VALUE, field masking with REGEXP_REPLACE, state-TTL and snapshot-mode settings, and Faker-backed table definitions. SQL examples from both sources were also executed on the cluster; the 16 how-to examples that run were retained as reference material. The dialect rules in the generator prompt draw on the CC Flink SQL reference ([3]) and on the dialect gaps observed in the seed pairs.
Coverage expansion. An SME audit of a pilot sample showed that common analytical constructs were missing entirely. The catalogue adds nine patterns: CUMULATE windows, CASE WHEN derivations, IN/EXISTS subqueries, COALESCE/NULLIF handling, conditional aggregation with FILTER and HAVING, CROSS JOIN UNNEST, ROW field access, array and map functions, and JSON structure functions (JSON_QUERY, JSON_EXISTS). These are grouped under a new group G15 (complex and nested types) or existing groups. The nested and JSON patterns are reachable only through the extension tables described in Appendix § 7.
Resulting taxonomy. The catalogue contains 46 patterns in 14 groups (Table 6). One pattern (time_series_math) is not generated, because it requires synthetic signal generation, which yields no stable reference result. The detectors were revised after generation (e.g., interval joins with an offset lower bound, and unbounded OVER windows in their BETWEEN form), and all labels were recomputed from the stored SQL.
Benchmark generation
Cluster-grounded generation
We extract Avro/JSON schemas, including nested types, from the environment’s Schema Registry and construct Flink DDLs annotated with the descriptive model.* metadata (grain, keys, changelog mode, allowed values). Generation then proceeds per schema in five steps.
Quota planning. A planned share is assigned to ten pattern buckets, for example 18% window TVFs, 13% nested types, 12% deduplication and Top-N, 10% interval joins, 10% background statement sets, and 6% pattern matching. Each bucket is split evenly across the patterns the schema can support; the share of unsupported patterns is redistributed. Each task is also assigned an intended execution mode (target 45% snapshot, 45% streaming, 10% background) and zero, one or two secondary constructs (target 40 / 40 / 20%), such as a CASE derivation, a TIMESTAMPDIFF duration or a join with an FK-linked table, to produce multi-pattern queries.
Capability-based routing. Each pattern declares the table capabilities it needs (an append stream with a watermark, two such streams, an upsert dimension with a primary key, an ARRAY, ROW or JSON column), and tasks are routed only to tables that provide them. Routing prevents a frequent failure mode: streaming constructs applied to updating tables.
Guided generation. An LLM (Claude Opus 5) writes one statement per task, given the routed DDLs, up to twelve previously accepted corpus queries from the same pattern group as examples, and a set of dialect guardrails derived from the SME audit and the dialect gaps observed in the seed pairs. Global guardrails forbid time-relative predicates on historic data, restrict streaming constructs to watermarked append tables, require a unique tie-breaker in Top-N rankings, and require matching timestamp types in interval arithmetic; pattern-specific guardrails state the required form (e.g., SESSION windows are streaming-only on CC).
Acceptance and repair. Each candidate must (i) pass a static lint that encodes the known defect classes, (ii) contain the target construct and match the planned execution mode, (iii) be accepted by the CDC Flink planner (EXPLAIN) and execute on the cluster, and (iv) return at least one row on the live data. A failing candidate is returned to the LLM with the lint violations, the Flink error, or the empty result as feedback, up to two times. Background statements are validated by EXPLAIN on each inserted query, and CREATE TABLE ... AS statements by executing their inner query. Across the run, 375 tasks were rejected after two repairs (137 still returned no rows, 118 still failed on Flink, 120 still failed the lint, of which 103 lacked the target construct); of the accepted pairs, 852 (21.8%) required at least one repair.
Question generation and deduplication. For each accepted query, the LLM writes the L0 schema-explicit request and the L2.1 business-language request; rows with an empty or truncated request are discarded. Near-duplicate pairs are removed by NL and SQL similarity (threshold 0.95), which dropped 309 of 4,212 candidates. Semantic labels are recomputed from the SQL with the pattern detectors, and the complexity tier is the number of distinct labels. A final pass generates the intermediate specification levels (L1.1–L2.4) with an LLM verifier (§ 5.2).
Execution acceptance establishes compatibility with the target environment and a non-empty reference result; alignment of each request with its query is produced by the LLM and is not independently validated.
Semantic labels
Table 8 presents representative labels, grouped by SQL pattern area. Each pair carries one or more labels, so the 3,903 pairs yield 7,408 assignments over 45 observed labels (Table 6). The five most frequent labels account for 2,226 assignments (30.0%): two_table_join_agg (597), timestampdiff (474), tumble_tvf (433), convert_tz (396) and array_functions (326). Filters with literal rather than clock-relative bounds are not generation targets but are still detected in 264 and 222 queries (null_empty_filter, filter_with_interval_bound).
| Area | Labels |
| Window TVFs | tumble_tvf, hop_tvf, session_window_tvf, windowed_count_having |
| OVER windows | over_rows_preceding, over_range_interval_preceding, over_window_unbounded |
| Joins | temporal_join, unbounded_left_join, stream_stream_interval_join, multi_join_chain, window_join |
| Aggregation | two_table_join_agg, group_by_order_limit |
| Dedup/Top-N | dedup_rownumber_eq1, streaming_topn_rownumber |
| CEP | match_recognize |
| JSON/CDC | json_value_cast, json_value_extraction |
| Filtering | null_empty_filter, filter_with_interval_bound, timestampdiff |
Table 8. Semantic labels grouped by Flink SQL pattern area (partial listing of the 45 observed labels).
Evaluation groups
Table 11 and Table 10 aggregate the 45 semantic labels into nine groups, following the pattern areas of Table 8, extended so that every label belongs to exactly one group. Corpus sizes (questions touching the group, over all 3,903 pairs) are given in parentheses.
| Pass% | 95% CI | Err. | ||
| All | 666 | 79.3 | 76.0–82.2 | 143 |
| Window TVFs | 57 | 91.2 | 81.1–96.2 | 3 |
| OVER windows | 81 | 74.1 | 63.6–82.4 | 13 |
| Joins | 121 | 71.9 | 63.3–79.1 | 17 |
| Aggregation | 167 | 80.2 | 73.5–85.6 | 9 |
| Dedup / Top-N | 28 | 96.4 | 82.3–99.4 | 2 |
| CEP | 41 | 53.7 | 38.7–67.9 | 2 |
| JSON, nested | 128 | 67.2 | 58.7–74.7 | 28 |
| Filters, scalar | 251 | 77.7 | 72.1–82.4 | 24 |
| Statements | 132 | 78.8 | 71.1–84.9 | 95 |
Table 10. GPT-6-Astra performance by pattern group on the comparison sample (judge-pass rate with Wilson 95% CI). Err.: statements that failed to execute. A question with several labels contributes to every group.
| Pattern group | GPT-6 | Claude | GPT-5.6 | Gemini | OSS-120B | |
| Window TVFs | 57 | 91.2 | 75.4 | 42.1 | 36.8 | 8.8 |
| OVER windows | 81 | 74.1 | 84.0 | 79.0 | 70.4 | 44.4 |
| Joins | 121 | 71.9 | 69.4 | 67.8 | 63.6 | 45.5 |
| Aggregation | 167 | 80.2 | 74.9 | 74.3 | 72.5 | 52.7 |
| Dedup / Top-N | 28 | 96.4 | 96.4 | 96.4 | 96.4 | 64.3 |
| Pattern matching (CEP) | 41 | 53.7 | 61.0 | 29.3 | 29.3 | 0.0 |
| JSON and nested types | 128 | 67.2 | 63.3 | 56.2 | 51.6 | 24.2 |
| Filters, scalar functions | 251 | 77.7 | 80.9 | 64.5 | 62.2 | 30.3 |
| Statements, set operations | 132 | 78.8 | 84.1 | 78.0 | 61.4 | 10.6 |
Table 11. Judge-pass rate (%) by pattern group for the model comparison ( = 666; best value per row in bold). A question with several labels contributes to every group, so rows do not sum to 666. The Window TVF group contains only session_window_tvf and cumulate_tvf here, because the TUMBLE and HOP questions are excluded.
Window TVFs (852) tumble_tvf, hop_tvf, session_window_tvf, cumulate_tvf, windowed_count_having.
OVER windows (405) over_rows_preceding, over_range_interval_preceding, over_window_unbounded, avg_over_partition, lag_over_window, lead_over_window.
Joins (678) temporal_join, stream_stream_interval_join, multi_join_chain, window_join, unbounded_left_join, unbounded_inner_join, self_join_aggregation.
Aggregation (899) two_table_join_agg, group_by_order_limit, conditional_aggregation.
Dedup / Top-N (220) dedup_rownumber_eq1, streaming_topn_rownumber.
Pattern matching, CEP (183) match_recognize.
JSON and nested types (657) json_value_cast, json_value_extraction, json_parsing_functions, array_functions, cross_join_unnest, nested_row_access.
Filters and scalar functions (1,903) null_empty_filter, filter_with_interval_bound, timestampdiff, convert_tz, regexp_replace_mask, case_when_transform, nullif_coalesce, in_exists_subquery.
Statements and set operations (605) execute_statement_set, union_all_last_value, metadata_virtual_column, ctas_passthrough, set_state_ttl, set_snapshot_mode, faker_connector_ddl.
Corpus entry schema
Each JSONL entry contains the following fields: stmt_num (unique integer ID), source_file (schema identifier) and domain, sql_cc (validated Flink SQL), nl_input_generated (schema-explicit source form), nl_question_simple (unclarified business-language base form), what_it_exercises_generated (pattern description), labels (list of semantic tags) and group_ids, execution_mode (SNAPSHOT / STREAMING / BACKGROUND, the last denoting DML), complexity_tier (1, 2, or 3), and ddl_refs (list of referenced table names). Entries additionally record the generation plan and outcome: target_label, quota_bucket, planned_mode, planned_tier, planned_secondaries, and generation_repairs (number of repair rounds, 0–2). Paired ground-truth files (.gt.jsonl) store ground_truth_df (up to 200 result rows as JSON), ground_truth_df_truncated, execution_time_ms, and sql_execution_error. Difficulty sidecars (.difficulty.jsonl) store the L1.1–L2.4 requests, the ranked clues, and the verifier outcome per level.
Evaluation Protocol and Configuration
Model identifiers and prompts. The model-comparison commands use azure/gpt-6-astra, azure/gpt-5.6-sol, aws/claude-opus-5, gcp/gemini-3.6-flash, and watsonx/openai/gpt-oss-120b. Both experiments use generation prompt version v1. The runner defaults are temperature 0.2 and a maximum of 4,096 generated tokens. The judge configurations identify aws/claude-opus-5-5, azure/gpt-5-astra, and gcp/gemini-3.6-flash; the scoring adapter prefixes these identifiers with litellm:. Ground-truth-anchored and face-value evaluation use separate prompt configurations with the same model pool and role-assignment seed. The face-value results use prompt version v3, which tells the judge that the question abstracts the SQL (extra specificity is not a defect), shows each statement’s execution status (ran, failed, or not executed) as fact, and narrows maybe to a single unverifiable requirement. An earlier prompt (v1) without these elements gave 86.8–88.0% at L0–L1.2 and 75.7–77.7% at L2.4–L2.1 on the same predictions; the two versions are not comparable. The variant-generation records identify azure/gpt-5.6-sol as both generator and verifier, with prompt version difficulty_variants/v1; the generation script sets temperatures of 0.3 and 0.0 for these respective roles.
Prediction execution. The execution runner infers snapshot or streaming retrieval from the predicted SQL. Its defaults are a 300-second submission/completion timeout, a 1,000-row prediction-output cap, and a 60-second streaming read budget after submission. A streaming read ends when the row cap or read deadline is reached, or the cursor reports no further results. An empty fetch is followed by a 0.2-second pause. The read deadline is checked between fetches, so a blocking client call can extend the observed duration. The runner then requests that the statement stop and waits for confirmation. Snapshot retrieval fetches up to one row beyond the cap to detect truncation.
Execution duration covers submission and result collection and, for streaming retrieval, the stop request. Submission or retrieval exceptions are recorded as execution errors; an exception while stopping is logged separately. Success therefore means no recorded error during the observation procedure, rather than indefinite runtime correctness. Prediction-output limits are distinct from the corpus reference-capture limits.
Streaming execution accuracy
We do not report execution accuracy (ExAcc), conventionally defined by agreement between predicted and reference SQL results. Applying this metric to streaming data requires an explicit comparison protocol for evolving inputs, event-time and watermark alignment, and updates or retractions in output streams. Clock-relative predicates and truncated captures further complicate comparison. We leave the extension and validation of ExAcc to future work, including aligned input snapshots or replay, defined observation horizons, and comparison of evolving results.
Confidence intervals
Pass rates are proportions; Table 9 shows the 95% confidence interval half-width (Wilson method, ) for selected sample sizes at . For passes among questions, , the interval is
| Sample size | 95% CI half-width (pp) |
| 10 | ±26.3 |
| 30 | ±16.8 |
| 100 | ±9.6 |
| 394 | ±4.9 |
| 400 | ±4.9 |
| 666 | ±3.8 |
Table 9. 95% confidence interval half-widths (Wilson) for pass-rate estimates at various sample sizes, assuming (worst case).
At it is symmetric about the observed proportion; otherwise we report its endpoints. For example, 362 passes among 394 questions (91.9%) yield an interval of 88.8–94.2%. The v3 model comparison () has a worst-case half-width of 3.8 points.
Statistical scope. The five-model comparison uses 666 questions, limiting the resolution of small performance differences (Table 9). Subgroup estimates have smaller denominators than the overall comparison, for example 28 Dedup / Top-N questions. The judge pool overlaps with three generator families (Claude, GPT-6-Astra, Gemini 3.6 Flash), although the Claude judge and generator use different versions (Opus 5.5 and Opus 5). Shared model biases may favour related outputs; the adjudication protocol does not establish the absence of this bias. Each model and condition is evaluated in a single run.
Additional Experimental Results
Query-pattern analysis. Table 11 and Table 10 identify substantial variation across query patterns. Dedup / Top-N (96.4%), window TVFs (91.2%), aggregation (80.2%), filters and scalar functions (77.7%) and statements (78.8%) yield relatively high pass rates for GPT-6-Astra; the window-TVF group is restricted to session and cumulate windows in this release (Table 11). Pattern matching with MATCH_RECOGNIZE is the weakest group for GPT-6-Astra (53.7%, 22 of 41), GPT-5.6-Sol and Gemini 3.6 Flash (both 29.3%), and Claude Opus 5 reaches 61.0%; GPT-OSS-120B passes none. Window TVFs separate the models most: GPT-6-Astra reaches 91.2%, Claude Opus 5 75.4%, and GPT-5.6-Sol and Gemini 3.6 Flash only 42.1% and 36.8%. Joins (71.9% for GPT-6-Astra) and JSON and nested types (67.2%) are the next weakest groups. Combining joins with other patterns lowers GPT-6-Astra’s pass rate (tier 1: 30 of 43; tier 2: 32 of 38; tier 3: 25 of 40). The temporal join label is a clear failure point: 12 of GPT-6-Astra’s 16 temporal-join statements are rejected because they use PROCTIME(), which Confluent’s Flink dialect does not support, and only 2 of 16 pass for GPT-6-Astra, Claude Opus 5 and GPT-5.6-Sol alike (Gemini 3.6 Flash: 0). This illustrates dialect-specific constraints that the table DDL alone does not reveal. Pooling the three strongest models, the lowest-performing individual labels with at least eight questions are temporal_join (12%), lead_over_window (28%), filter_with_interval_bound (47%) and match_recognize (48%).
Of GPT-6-Astra’s 143 execution errors, all 81 are background-mode statements. Of the remaining 62, 12 are the PROCTIME() errors above, 7 are TIMESTAMP_LTZ arithmetic errors, and 43 are miscellaneous parse, planner or validation errors (for example, unsupported MATCH_RECOGNIZE clauses or unresolved functions). Of the 62 non-background errors, 1 is nevertheless judged pass; of the 81 background errors, 72 are.
Per-domain results
Per-domain variation. The v3.1 comparison sample provides only 11–42 questions per domain, limiting the precision of domain-level estimates. The domain analysis below is therefore from an earlier run on the full corpus of release v2 (same L0 question, domain DDL and judge) and was not repeated on v3.1. Its pass rate is 81.7% (95% CI 80.5–82.9), compared with 79.3% for GPT-6-Astra on the v3.1 test sample of Table 3; the two runs use different corpus releases and question sets. (Table 12, Appendix). Domain rates range from 68.9% (banking) to 90.0% (mining), with a median of 82.2% and a standard deviation of 4.3 pp. No domain retains the 100% accuracy observed for some small domain subsets in the sample. The lowest pass rates occur in banking (68.9%), aid (71.6%), retail (73.9%), weather (74.3%) and restaurant (75.0%). The observed heterogeneity is not statistically significant under a common-rate model (, 44 d.f., ), and only mining, banking and aid have a 95% interval that excludes the overall rate. These results provide insufficient evidence for a systematic domain effect. Schema size also exhibits no observed association with accuracy: domains contain 4–14 tables, and table count is uncorrelated with pass rate (Spearman ).
| Domain | Tab. | Pass% | Domain | Tab. | Pass% | Domain | Tab. | Pass% |
| advertising | 7 | 78.6 | farming | 8 | 83.1 | pipeline | 7 | 81.1 |
| aid | 4 | 71.6 | forestry | 5 | 85.4 | property | 5 | 82.2 |
| aquaculture | 5 | 80.4 | homecare | 7 | 87.6 | recruitment | 7 | 83.7 |
| banking | 6 | 68.9 | hospital | 13 | 82.7 | recycling | 9 | 80.6 |
| build | 6 | 84.4 | hotel | 6 | 80.8 | renewable | 5 | 80.8 |
| charging | 7 | 85.7 | insurance | 10 | 84.7 | research | 7 | 80.4 |
| cloud | 6 | 76.7 | laboratory | 7 | 76.8 | restaurant | 7 | 75.0 |
| construction | 6 | 82.9 | livestock | 5 | 86.7 | retail | 14 | 73.9 |
| courts | 6 | 85.0 | maritime | 7 | 83.8 | rideshare | 6 | 82.1 |
| cybersecurity | 8 | 82.6 | mining | 5 | 90.0 | sports | 11 | 79.6 |
| education | 4 | 86.6 | newsroom | 6 | 83.8 | telecom | 5 | 85.3 |
| elections | 5 | 77.9 | parcel | 5 | 84.2 | transit | 7 | 87.1 |
| electricity | 4 | 77.9 | parking | 6 | 83.8 | water | 7 | 85.0 |
| entertainment | 5 | 80.5 | permits | 8 | 80.9 | weather | 4 | 74.3 |
| exchange | 6 | 80.8 | pharmacy | 6 | 81.3 | wildlife | 6 | 82.3 |
Table 12. GPT-6-Astra on the full corpus of release v2 (L0, domain DDL context, ground-truth-anchored judge), by domain. Tab.: number of tables in the domain schema. This run has not been repeated on v3.1.
Paired comparisons. Discordant-pair counts (first model only / second model only) for the v3.1 comparison are: GPT-6-Astra vs. Claude Opus 5 57 / 59 (); GPT-6-Astra vs. GPT-5.6-Sol 102 / 40 (); GPT-6-Astra vs. Gemini 3.6 Flash 127 / 32; Claude Opus 5 vs. GPT-5.6-Sol 101 / 37; Claude Opus 5 vs. Gemini 3.6 Flash 122 / 25; GPT-5.6-Sol vs. Gemini 3.6 Flash 74 / 41 (); every frontier model vs. GPT-OSS-120B at least 215 / 15. Tests are not corrected for multiple comparisons. Adjudication was required for 483 of the 3,330 judgments (14.5%), from 12.0% (GPT-OSS-120B) to 17.9% (GPT-6-Astra) across generators.
Illustrative Examples
Illustrative Flink SQL constructs
The following examples illustrate the temporal operations and data manipulation constructs motivating FlinkEval. They are schematic examples rather than additional benchmark measurements.
Windowed aggregation. A streaming query can count orders per customer in five-minute event-time windows, using the window boundary columns supplied by the TVF [4]:
SELECT customer_id,
window_start AS ws,
COUNT(*) AS order_count
FROM TUMBLE(TABLE orders,
DESCRIPTOR($rowtime$),
INTERVAL '5' MINUTE)
GROUP BY customer_id, window_start,
window_end
Temporal enrichment. An event-time temporal join associates an order with the exchange rate valid at the order’s event time, assuming a versioned rate table keyed by currency and suitable event-time watermarks:
SELECT o.order_id, o.amount,
r.rate, o.amount * r.rate AS usd
FROM orders AS o
JOIN exchange_rates FOR SYSTEM_TIME
AS OF o.$rowtime$ AS r
ON o.currency = r.currency
Background data manipulation. Background statements materialise query results in sink tables backed by Kafka topics:
INSERT INTO `customer_summary`
SELECT customer_id,
COUNT(*) AS total_orders,
SUM(amount) AS total_spend
FROM `orders`
GROUP BY customer_id
Confluent’s ON CONFLICT extension additionally supports deduplication:
UPSERT INTO `loyalty_scores`
SELECT customer_id, score
FROM enriched_events
ON CONFLICT DO DEDUPLICATE
Potential clarification interaction. The following dialogue illustrates a possible extension to interactive clarification; the current experiments use appended oracle clues.
User: Which of my recent ad impressions were actually viewable?
System: I can help with that. Which table contains your impression data, and over what time window should “recent” be defined?
User: The fact_impression table, last 2 hours.
System: SELECT * FROM ‘fact_impression‘ WHERE served_at > CURRENT_TIMESTAMP - INTERVAL ’2’ HOUR AND viewable = TRUE
Illustrative analysis of omitted constraints
Table 13 presents a schematic insurance example of how question specification affects SQL evaluation. The requests and SQL are illustrative, rather than a traced corpus entry or recorded model prediction; no empirical judge verdicts are associated with this example. The two requests illustrate the L0 and L2.1 specification conditions.
| L0: Explicit Specification | Show me all active policies where the policy_end_date is within the next 30 days, including policy_id, holder_name, premium_amount, and policy_end_date from insurance_42__dim_policy. |
| L2.1: Unclarified Underspecified Request | Which of my policies are about to expire? |
| Illustrative reference SQL | SELECT policy_id, holder_name, premium_amount, policy_end_date FROM insurance_42__dim_policy WHERE policy_end_date BETWEEN CURRENT_DATE AND CURRENT_DATE + INTERVAL ’30’ DAY AND status = ’ACTIVE’ |
| Candidate SQL | SELECT * FROM insurance_42__dim_policy WHERE policy_end_date <= CURRENT_TIMESTAMP + INTERVAL ’30’ DAY |
Table 13. Schematic insurance example of omitted query constraints. The candidate changes the projection, omits the active-policy and lower-date filters, and uses a timestamp-relative upper bound. L0 and L2.1 illustrate paired specification conditions; the example reports no model judgments or live execution results.
The candidate differs from the illustrative reference SQL in four respects: (1) SELECT * instead of the four explicitly requested columns; (2) the status = ’ACTIVE’ predicate is absent, so cancelled and lapsed policies can also be returned; (3) the lower bound policy_end_date >= CURRENT_DATE is absent, allowing already-expired policies; and (4) the upper bound is expressed relative to CURRENT_TIMESTAMP rather than CURRENT_DATE, changing the temporal expression. The effect of the last difference depends on the column type and the engine’s date/timestamp coercion rules; it does not by itself establish a type error.
Under reference-anchored evaluation, the reference supplies the projection, active-policy filter, and bounded 30-day horizon. Under face-value evaluation, L0 explicitly supplies these requirements, whereas L2.1 leaves the projection, status restriction, and precise horizon unstated. Returning already-expired policies remains problematic even for “about to expire”: the candidate has no lower date bound. For example, an active policy that ended yesterday satisfies the candidate’s upper-bound condition but is excluded by the reference. This logical counterexample illustrates a semantic discrepancy without relying on a particular result capture or judge score.
References
- [1]Adithya Bhaskar, Tushar Tomar, Ashutosh Sathe, and Sunita Sarawagi. 2023. Benchmarking and improving text-to-SQL generation under ambiguity. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, pages 7053–7074. Association for Computational Linguistics.arxiv.org/abs/2310.13659
- [2]Paris Carbone, Asterios Katsifodimos, Stephan Ewen, Volker Markl, Seif Haridi, and Kostas Tzoumas. 2015. Apache Flink: Stream and batch processing in a single engine. IEEE Data Engineering Bulletin, 38(4):28–38.
- [3]Confluent. 2026a. Flink SQL and Table API reference for Confluent Cloud for Apache Flink. Accessed October 28, 2026.
- [4]Confluent. 2026b. Windowing table-valued functions in Confluent Cloud for Apache Flink. Accessed October 28, 2026.
- [5]Jiaqi Guo, Zecheng Zhan, Yan Gao, Yan Xiao, Jian-Guang Lou, Ting Liu, and Dongmei Zhang. 2019. Towards complex text-to-SQL in cross-domain database with intermediate representation. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 4524–4535, Florence, Italy. Association for Computational Linguistics.arxiv.org/abs/1905.08205
- [6]Fangyu Lei, Jixuan Chen, Yuxiao Ye, Ruisheng Cao, Dongchan Shin, Hongjin Su, Zhaoqing Suo, Hongcheng Gao, Wenjing Hu, Pengcheng Yin, Victor Zhong, Caiming Xiong, Ruoxi Sun, Qian Liu, Sida Wang, and Tao Yu. 2025. Spider 2.0: Evaluating language models on real-world enterprise text-to-SQL workflows. In Proceedings of the International Conference on Learning Representations (ICLR).arxiv.org/abs/2411.07763
- [7]Jinyang Li, Binyuan Hui, Ge Qu, Jiaxi Yang, Binhua Li, Bowen Li, Bailin Wang, Bowen Qin, Ruiying Geng, Nuo Hao, Xuanhe Zhou, Chenhao Ma, Guoliang Li, Kevin Chang, Fei Huang, Reynold Cheng, and Yongbin Li. 2023. Can LLM already serve as a database interface? a Big bench for large-scale database grounded text-to-SQLs. In Advances in Neural Information Processing Systems, volume 36.arxiv.org/abs/2305.03111
- [8]Wen-Zhi Li, Yue Gong, Konstantinos Kanellis, and Balakrishnan Murali Narayanaswamy. 2026. Beyond correctness: Resolving underspecification in agentic text-to-SQL. arXiv preprint arXiv:2610.02739v1.arxiv.org/abs/2610.02739
- [9]Yang Liu, Dan Iter, Yichong Xu, Shuohang Wang, Ruochen Xu, and Chenguang Zhu. 2023. G-Eval: NLG evaluation using GPT-4 with better human alignment. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, pages 2511–2522, Singapore. Association for Computational Linguistics.arxiv.org/abs/2303.16634