betaFeedback

WorksComputer Science

Fidelity Collapses with Table Size, Not Turn Count: Delivery Architecture in a Deployed Tool-Using Agent

Abstract

When a multi-turn agent delivers a tool's structured records into the model's context and re-emits them each turn, every turn can corrupt data the tool computed correctly. We measure this in a production award-travel agent: two architectures over identical frozen payloads, lineage-matched prompts and sampling, differing only in delivery architecture. SYNTHESIS feeds full record sets into the planner's context and answers follow-ups in-head; BYPASS, the deployed system, keeps records in session state, translates follow-ups into arguments for a deterministic filter, and delivers rows over a typed side channel. Across 24 dialogues (339 scored turns per condition), BYPASS scores 92.3% of turns fully correct, SYNTHESIS 45.7% (paired gap +46.6pp [+35.7, +57.1]). Varying delivered-table size by capture date, SYNTHESIS collapses from near-perfect at 10 rows or fewer to 8% or less beyond about 74 rows. On a content-controlled prefix ladder the collapse threshold orders by model capability, from under 10 rows (GPT-4o-mini) to about 198 or more (Gemini 2.5 Pro), while BYPASS stays size-flat for all four models. The collapse is expensive: beyond 100 rows, where almost no SYNTHESIS answer is correct, each correct one costs roughly 200x BYPASS's flat $0.012. Practitioners converged on typed side-channel delivery for rendering, not correctness; we show it was, unknowingly, also a correctness decision.

Introduction

Structured hallucination, in which an agent retrieves correct structured data through a tool call and then corrupts field values while formatting them, is invisible to ordinary hallucination detection [18]: validators see valid JSON, schema-parse metrics log success [17]. Prior work established the phenomenon single-turn; a deployed award-travel session runs five to twelve turns, each another synthesis pass over

the same record set. Whether the corruption compounds, and what it scales with, is this paper’s question.

The stake is architectural: agent frameworks default to returning a tool’s result into the model’s context for the model to re-emit: every framework we surveyed does so, or requires wiring an opt-in bypass [22, 35, 8, 23, 24]. The alternative (records as structured state, follow-ups answered by a deterministic tool, rows streamed on a typed side channel) exists [6], adopted for rendering, with correctness only an unmeasured aside [3] (CopilotKit contributors 2026); the corruption itself is documented both single-turn (Yang et al., 2026) and, over long agentic workflows, at roughly a quarter of document content [21]. To our knowledge, no prior work joins the two: a controlled A/B on the same record sets, with row- and field-level fidelity on both arms. We measure the corruption in a production multi-agent award-and-cash flight system built on Google’s Agent Development Kit (ADK) as a fork of the adk-samples travel-concierge [13]: the fork-point architecture (full row lists into the planner’s context, follow-ups in-head) is a real system contract, not a strawman, and the deployed system is the same tree with only delivery replaced; both run over the same frozen vendor payloads.

Contributions.

  • A controlled size manipulation, measured on both arms of a production system, testing an architecture the field adopted on faith. Typed side-channel delivery shipped for UX, never correctness (LangGraph in 2024, AG-UI in 2025, Vercel Data Parts in 2025, Google A2UI in 2026). Twenty dialogues over four routes vary delivered size (4→194 rows); a six-route ladder holds content constant via K-row prefixes (Figure 2); a 3×3 Latin square separates size from position (p=3×10−10p = 3 \times10^{-10} vs 0.36). SYNTHESIS collapses with size on every design; BYPASS is flat at 99.2%; a byte-identical control table holds its level even where treatment scored zero. Strict per-turn field fidelity gives BYPASS 92.3% vs SYNTHESIS 45.7%, paired gap +46.6pp, replicating across three model families at +46.6 to +55.8pp: capability buys threshold, architecture buys immunity (Figure 1).

Figure 1. Four models, one content-controlled ladder: SYNTHESIS fidelity vs K-row prefixes of one frozen payload. Capability buys threshold, architecture buys immunity: collapse orders by capability (under 10 → ≳198 rows); the BYPASS band is size-flat for all four (140/144 runs). nn=6 runs per (model, K) cell.

Figure 2. Turn fidelity vs K: the delivered table is the first K rows of each route’s fixed payload, so size varies only by appending rows (six routes pooled). Light dots are per-route points at each K (n=3n=3 runs each, hence quantised to 0/33/67/100%); the band is a Wilson 95% interval on the six routes pooled. Flash SYNTHESIS decays monotonically to 6% at K=100; BYPASS is 100% in all 93 runs.

  • A cost and latency account, not only an accuracy one. BYPASS’s per-turn cost and latency are flat in table size; SYNTHESIS’s scale with it, reaching $2.53 per correct answer beyond 100 rows against BYPASS’s flat $0.012 (Figure 3). Swapping the synthesis model from the production Flash to Gemini 2.5 Pro over the same 24 dialogues lifts synthesis fidelity from Flash’s 46% to 72%, at 5× Flash’s per-turn cost. Renting fidelity from a bigger model is the expensive substitute for not re-emitting the data.

Figure 3. Cost per correct answer by delivered-size band (size-ladder sweeps; mean turn cost ÷ accuracy, log scale). BYPASS is flat at ≈$0.012; SYNTHESIS climbs to $2.53 beyond 100 rows, ≈200×. Point labels give SYNTHESIS’s per-band accuracy (one correct SYNTHESIS turn in the >100 band; the multiple is an order-of-magnitude estimate). Unequal nn: the starved ORD-NRT run’s two BYPASS turns (0 of 72 rows delivered) count in ≤16, its only misses; two >100-row SYNTHESIS turns emitted nothing scoreable (Appendix § 10).

  • A published negative result, with diagnosis, and evidence that failures are not independent. Our pre-registered test for super-linear error compounding against the accumulation-only null 1−(1−p)n1 - (1-p)^n finds none, because the null saturates when per-turn failure is large. Its replacement, lag-1 conditional hazard (does this turn’s failure predict the next turn’s? §5.6), agrees with value-level tracing: synthesis failures propagate, each corrupted emission becoming the next turn’s input. BYPASS recomputes every turn from stored state; propagation is precluded by the delivery path, and the 0/25 reflects that no counterexample arose.

  • A measurement methodology with computed ground truth, and its integrity report. Expert-demonstrated dialogues replay against frozen vendor payloads with computed ground truth (production filter evaluator; 408 frozen captures; detail in § 4). Practising it surfaced eight instrument defects (three in our own ground truth, § 7.1), plus two ground-truth semantics corrections and three production bugs, all fixed; the headline holds under both instrument versions.

Related Work

Structured output: validity is not correctness. Constrained decoding guarantees shape, not content: validity hits 100% while accuracy falls and wrong-but-valid outputs dominate [27]. Hoang [17] measured the resulting structured hallucination in a production travel agent single-turn, introducing the field-fidelity metric and zero-out policy we adopt. This paper compares two delivery architectures on a new dataset in a multi-turn design (results are not compared to that study’s).

Multi-turn agent evaluation. τ\tau-bench established multi-turn tool-agent evaluation (Yao et al., 2024); a 39% average single- to multi-turn drop with poor recovery is documented [20]; self-conditioning, in which past errors raise later error rates, is the closest mechanism match [32]. Closest to us, “Beyond Task Completion” reports 27–78% of τ\tau-bench successes as corrupt [5]: procedural integrity on τ\tau-bench’s mock domains (airline and retail), where ours is field- and row-level data fidelity on real records, indexed by delivered record-set size, with the corruption self-inflicted.

Architectures that keep data out of the model. CodeAct routes actions through executable code [37]; practitioner channels exist (LangGraph’s custom stream mode, the Vercel AI SDK’s data parts, the AG-UI protocol), but no framework we surveyed ships typed structured delivery as the default end-to-end rendering path, and the public motivation is rendering predictability, never correctness [6].

Evaluation methodology. Turns within a dialogue are not independent and turn-level significance claims frequently fail cluster-robust correction [30]; every gap interval here re-samples dialogues, not turns. The full survey, including the structured-output infrastructure literature and the closest dialogue benchmark (BOULDER (Kartáč et al., [19]): it computes over given records; we ask whether the set survives delivery), is in Appendix G.

System and Conditions

The deployed system

The system under study, AwardWise, is a multi-agent travel concierge on Google’s ADK: a planning agent routes requests; sub-agents wrap the award (Seats.aero) and cash (SerpAPI) search pipelines, which write the canonicalised record set to session state. Two properties matter here.

  1. Search results do not enter the model’s context. The sub-agent’s function response carries a count and the search parameters; records go to session state, and the client receives rows as typed events alongside, not through, the model’s token stream.

  1. The model translates each follow-up’s intent into arguments; a deterministic tool evaluates them. The context tool evaluates filter arguments (vocabulary in C.10) over the stored record set. The remaining translation job (the full language-understanding problem of tool use, scope anaphora pinned by convention, § 4.2) is where BYPASS’s residual failures live (§ 5.4). The architecture changes the failure surface, not the difficulty: a mistranslated argument yields one wrong-but-real subset at any size, where re-emission puts every field of every row at stake on every turn.

We name the conditions for what happens to the records: in SYNTHESIS the record set passes through the model’s generation and is re-emitted; in BYPASS records reach the interface as typed data and are never generated. The distinction is the delivery; model, sampling and instruction lineage are held constant. BYPASS is this system unmodified, driven over frozen payloads: its single model-side error source is translation, and records are never re-typed, so field values cannot mutate in transit.

SYNTHESIS: the counterfactual fork-point architecture

The system is a fork of the adk-samples travel-concierge ([13]), whose contract is the canonical ADK pattern: a flight search sub-agent

returns a full structured row list into the planner’s context, there is no context tool, and follow-ups are processed in-head. SYNTHESIS rebuilds that architecture inside the mature product, so delivery is the only difference: the production tree deep-copied with exactly four deltas: a lineage-matched instruction variant, context tools removed, records injected mechanically into each sub-agent’s summary so SYNTHESIS is not penalised for sub-agent transcription noise, and output schemas adjusted so the injected records survive (all four in Appendix C). The planner emits each turn’s table through an output schema (flights, summary), parsed structurally rather than scraped from markdown; everything else (model, sampling, other sub-agents, tools, state writes) is identical to production.

Three consequences (in full in Appendix C): translation error is common-mode: the same planner with the same instruction lineage decides what the user asked for in both conditions, so the residual gap is attributable to delivery; mechanical delivery makes SYNTHESIS stronger than its fork-true form, which passes records through an LLM emission before the planner; and BYPASS’s advantage is joint: synthesis bypass plus structured-state preservation, not separated here (Table 1; Limitations). Both conditions run Gemini 2.5 Flash at temperature 0.1 / top_p 0.5, inherited by lineage rather than chosen, so temperature cannot explain the gap.

ComponentSYNTHESISBYPASS
Translate (NL → filter args)model (identical)model (identical)
Record storagemodel context, re-emitted every turnsession state, written once
Follow-up filterin-context, by the modeldeterministic evaluator
Delivery to UIparsed from model outputtyped side channel

Table 1. What differs between the conditions. The headline gap measures the joint effect of the last three rows; per-factor attribution requires unbundled conditions we leave to follow-up work.

Method

A note on instrument versions: partway through the study our measurement surfaced defects in production and in the instrument itself (§ 7); after fixing them we declared an instrument freeze and re-ran every experiment. We label runs before those fixes pre-fix and runs on the frozen instrument post-fix; every headline number is post-fix. Pre-fix numbers appear only in the labelled instrument-effect accounting (§ 7.3), the attribution argument (§ 6.2), and the appendices.

Freeze real, then replay

The study is a freeze-real-then-replay design: real searches ran through the production search tools, not raw vendor APIs (hub routing, deduplication and canonicalisation are part of the system under study), and everything they returned was frozen. The frozen corpus is 408 captures over 36 routes holding 9,809 records (largest single payload 510 rows), spanning four date bands and two cabins per route (composition and provenance in Appendix § 12). Experiments serve the vendor boundary from these captures, never a live vendor call.

Dialogue authoring

Dialogue templates derive from demonstration sessions conducted by an author, an experienced award-travel practitioner, over several rounds exercising representative award-hunting workflows. Observed phrasings form the allowed vocabulary, and the 24 dialogues vary along four knobs; the knobs, six dialogue arcs and turn families are inventoried in Appendix § 12.

Two authoring rules keep turns scoreable rather than arguable: no heuristic constraints (“red-eye” is banned; the mapping is pinned or phrased explicitly, “departing after 9pm”) and no undefined-aggregate superlatives (a superlative must name a total order; Appendix § 12 works the example); the second was adopted mid-study, retiring such a turn shipped in five dialogues (§ 7). Thresholds and boundary probes are pinned so competing readings cannot both be defensible (Appendix § 12).

A third convention pins down scope semantics, since every turn’s ground truth depends on what a follow-up operates over: a new filter composes with the active scope (“only morning departures” narrows the current table); a refer-back restores the view where its referent was introduced (“show all the morning ones again” returns to the table that named “the morning ones”, dropping filters and limits applied after it); and a constraint revision re-executes the turn that set the constraint against that turn’s own base (“actually, up to 87,499 points is fine” re-derives the cap from the full table, not the narrowed one; the deliberately odd cap is a pinned boundary probe, one point below the payload’s real 87,500 value). A turn meaning to reset must name its base explicitly (“across ALL results. . . ”).

87,500 value). A turn meaning to reset must name its base explicitly (“across ALL results. . . ”).

The oracle

Per-turn ground truth is computed, not annotated: each dialogue’s frozen captures are replayed through the production search tools, leaving exactly the session state a real search leaves, and each filter turn’s pinned arguments are passed to the production context tool’s evaluator over that state; the returned envelope (the tool’s function response: count and arguments, rows by reference) is the ground truth, normalised by the production canonicalisation layer at a fixed SHA, applied identically to both conditions. A liveness verifier rejects empty or trivially-repeated ground truth on every turn before any model is called (§ 7; the verifier and the oracle’s by-reference read path in Appendix § 12).

Metric

The primary metric is per-turn field fidelity with strict count-mismatch zero-out, the policy adopted from the single-turn study ([17]); field-level scoring against source records descends from data-to-text evaluation ([39]).1 Emitted records are aligned with ground-truth records on the identity key (flight_number, date, source); scored fields are those present and non-empty in the ground-truth record, drawn from a fixed fifteen-field set; comparison is exact after production canonicalisation (field list and key rationale in Appendix § 12). When the emitted count equals the ground-truth count, turn fidelity is the fraction of scored fields, over all aligned record pairs, whose emitted value matches the ground truth. If the counts differ, the turn’s fidelity is instead zero, regardless of how many individual fields match. This is strict on purpose since the harm is downstream, i.e., the user acts on the table as shown. A silently missing award flight is a missed booking the user never learns existed.

A turn counts as correct iff it produced an artifact at all and its zero-out fidelity is 1.0. Turns where the deployed condition answered from the count-only envelope without materialising rows currently count as failures, so **BYPASS’s published rate is a lower bound. Partial-credit fidelity (the field-match fraction without the zero-out: 98.2% for BYPASS vs 82.7% for SYNTHESIS) and the ordered-turn axis (order-exact: 94.2% vs 51.8%) are computed alongside every turn and reported in full in Appendix § 12. Search turns establish context and are not scored; award-versus-cash comparison turns exercise a different production pipeline and are out of scope.

Runs, repeats, and the analysis layer

Each dialogue is run k=3k=3 times per condition at production sampling; repeats are independent, not seeded replays, so results reproduce distributionally, not turn-for-turn (Limitations). All published aggregates load through a single corpus module that decides what counts: network errors (429/503) are holes a recovery re-run may fill (a 429 is not data), and a run with no scoreable output scores zeros on its expected filter turns (inclusion rules, per-run trace contents and the analysis-layer inventory in Appendix § 12).

Hypotheses. H1a: the fidelity gap appears at the first filter turn. H1b: synthesis failures are not independent across turns; history predicts failure. H1c: synthesis fidelity degrades with the size of the delivered record set. A turn-count super-linearity conjecture (failures compound faster with depth than an independence null) was pre-registered but unsupported, published as a diagnosed negative result (§5.6). The size instruments (§ 5.1, § 5.3, § 7.4) were designed after the main corpus’s first-turn rates were observed, each pre-specified (cells, byte-identical scored turns, expected ground-truth counts) before its runs. A second family, H2 (demonstration-versus-instruction), was stated as motivation and not measured (Limitations; Appendix B). Verdicts: H1c supported as a controlled manipulation (§ 5.1); H1a supported (§ 5.2); H1b supported (§5.6).

Scale. 408 captures / 9,809 records over 36 routes; 24 dialogues × k=3k=3 runs × 2 conditions; 339 scored turns per condition carry the headline, on one model (Gemini 2.5 Flash; scale table: Table 7, Appendix § 12), with further families in § 5.3 (Table 10, Appendix § 15).

Frozen captures / records408 / 9,809
Routes36
Dialogues24
Runs per condition72 (24 dialogues × k=3k=3)
Scored turns per condition339
of which turn-matched264
Conditions2
Models1 (Gemini 2.5 Flash)

Table 7. Main-study scale. Capture, record, and route totals are the full frozen corpus, which also serves the size and prefix instruments (Appendix § 10); further families replicate at the same 339-turn scale (§ 5.3, F.1). The headline is the full 339 scored turns per condition; the 264 turn-matched turns (identical turn sets in both instrument versions’ corpora) isolate the instrument effect (§ 7.3).

modelBYPASSSYNTHESISgap
Gemini 2.5 Flash92.3% [89.0, 94.7]45.7% [40.5, 51.0]+46.6pp [+35.7, +57.1]
Gemini 2.5 Pro92.6% [89.3, 95.0]72.0% [67.0, 76.5]+20.6pp [+11.7, +31.4]
GPT-4o-mini67.0% [61.8, 71.8]17.4% [13.7, 21.8]+49.6pp [+39.2, +60.2]
Claude Haiku 4.582.9% [78.5, 86.5]27.1% [22.7, 32.1]+55.8pp [+41.1, +69.9]

Table 10. Cross-family replication at full-matrix scale (n=339n=339 scored turns per cell, kk1–3; Wilson 95% CIs on cells, dialogue-level bootstrap CIs on gaps).

Results

Delivered table size drives the synthesis collapse

The central result is a controlled manipulation: we vary only the size of the table delivered into the agent’s context, holding route, cabin, filter vocabulary, turn wording and turn position fixed, and score the same filter turn.

Content-controlled (top-K), replicated over six routes. From each of six routes’ fixed payloads (112–198 rows) we deliver the first K rows: a true prefix, so the small table is literally the opening of the large one and only the count differs. The same query (“only afternoon departures”) is asked at every K; each cell pools 6 routes × 3 repeats (n=18n=18; per-cell table: Table 6, Appendix § 10).

With only K varying, pooled Flash SYNTHESIS decays monotonically from 83% at K=5 to 6% at K=100 while BYPASS is perfect in all 93 runs (Figure 2); FRA-BKK’s full 198-row payload, the one route run at K=198, also scores 0/3. The decay is graded, not a cliff: single routes at n=3n=3 per cell quantise to 0/33/67/100%, but pooled over six routes the curve is a slope, and a Cochran–Armitage trend test with log-K scores rejects a flat profile (z=−5.07z=-5.07, two-sided p=3.9×10−7p=3.9\times10^{-7}; cluster-bootstrap robustness in Appendix § 15; FRA-BKK failure anatomy in Appendix § 10). No change of route, date, price distribution, or query is needed. Table size alone produces it.

Cross-route replication (date-varied). Independently, on four routes we vary delivered size by capture date; each route is a single-cabin ladder (Figure 4 and Table 5, Appendix § 10).

Figure 4. Turn fidelity vs table size. SYNTHESIS falls from 75–100% on the smallest tables to ≤8% beyond 74 rows in every route that reaches that size (one 0%→8% wobble at FRA-BKK’s largest cell); BYPASS is 100% everywhere except a single dip (70 rows, 83%), one starved run (§7); 99.2% overall.

routerows → SYNTHESIS acc.BYPASS
FRA-BKK9 → 92% · 74 → 0% · 194 → 8%100%
ORD-NRT11 → 75% · 70 → 17% · 174 → 0%100%*
ATL-AMS4 → 100% · 27 → 42% · 44 → 25%100%
BOS-DUB53 → 42%100%

Table 5. Date-varied ladders. *ORD-NRT’s BYPASS column excludes one starved run (10/12 at the 70-row cell, the single dip visible in Figure 4); with it, 99.2% overall (238/240 scored turns). Figure 3’s n row shares this accounting: the starved run’s two turns count in the ≤16 band at their delivered size of 0 rows, and two SYNTHESIS turns (d27 sweep k2 and d33 sweep k3, the 194 and 174-row cells; delivered payloads 198 and 190 rows) emitted nothing scoreable and drop from the >100 count.

SYNTHESIS falls from 75–100% on the smallest tables to near-zero beyond ∼74\sim74 rows in every route that reaches that size (the sole exception: FRA-BKK’s 8% at 194 against 0% at 74, one turn in twelve); BYPASS is flat at 99.2% (238/240), its two misses a single starved run (the search sub-agent never fired; § 7.2). A control leg, a byte-identical 16-row table in all twenty dialogues, scores SYNTHESIS 77.5% and BYPASS 100%, and holds even inside dialogues whose treatment turn scored zero (Appendix A): the failure tracks the table delivered at that turn, not the depth of the dialogue.

The date-varied ladder confounds size with content; the top-K ladder does not, and they agree: the effect survives under a pure prefix manipulation. A size-by-position Latin square replicates the collapse at every position with no detectable position effect (§ 7.4).

The gap at corpus scale

Over 24 authored dialogues, three repeats: BYPASS 92.3% (313/339), SYNTHESIS 45.7% (155/339), gap +46.6pp (dialogue-level bootstrap CI [+35.7, +57.1]). The gap is present at the first filter turn (not an accumulation effect) and never smaller than the per-route size ladders imply. The direction is near-unanimous at the dialogue level: BYPASS beats SYNTHESIS in 22 of 24 paired dialogues (one tie, one loss; exact sign test p=5.7×10−6p=5.7 \times10^{-6}), in 24 of 24 on GPT-4o-mini and 21 of 24 on Claude Haiku (§ 5.3): 67 wins against 2 losses across the three families.

The gap is not specific to one model family, and a tier upgrade only rents it back

With both conditions re-run over the full 24-dialogue corpus on two further families with the same agent stack (sampling caveat: Limitations; n=339n=339 scored turns per cell, the headline’s corpus), the paired gap holds across all three families: +46.6pp [+35.7, +57.1] (Gemini 2.5 Flash), +49.6pp [+39.2, +60.2] (GPT-4o-mini), +55.8pp [+41.1, +69.9] (Claude Haiku 4.5); per-condition cells with Wilson 95% CIs are in Appendix F (Table 10). The gap converges: +46.6 to +55.8pp across three families, every family’s condition intervals disjoint. Family determines who fails how, not how much the architecture costs; BYPASS’s floor (67.0–92.3% across the three families) tracks translation quality (Table 11), yet no family’s BYPASS falls below any of the three families’ SYNTHESIS; the exception is the Pro tier (72.0%, at 12×12 \times BYPASS-on-Flash’s price).

The failure shapes are family-specific. Gemini corrupts: real rows, wrong set (§ 5.4). GPT garbles: unparseable JSON mid-table, type-broken fields, dropped required fields. Claude omits, concentrated on larger tables: well-formed JSON that drops the required flights array while the prose summary counts the delivered table correctly (an accurate count attached to a missing table). One mechanism, three masks (specifics, per-family latencies, and a small-table pilot artifact in Appendix F).

The taxonomy has a load axis, and it is the paper’s thesis in one picture. Re-run on all four models with identical K-row prefixes of the same frozen 198-row payload, the content-controlled ladder (§ 5.1) shows the collapse threshold ordering by model capability: under 10 rows for GPT-4o-mini, 10–25 for Claude Haiku, 50–100 for Gemini Flash, while Gemini Pro alone degrades gracefully (perfect through 25 rows, 83% at 50, 67% at 100, and 50% at the full 198), and BYPASS is size-flat for all four models (140/144 across the ladder; Figure 1). Capability buys threshold; architecture buys size-immunity. Cells pool six repeats.

A same-family tier upgrade only rents the gap back: over the full corpus (n=339n=339 per cell) Gemini 2.5 Pro halves the gap rather than closing it (BYPASS 92.6% vs SYNTHESIS 72.0%, +20.6pp [+11.7, +31.4] against Flash’s +46.6pp) at 5× Flash’s per-turn cost and 1.6× its latency. BYPASS is tier-insensitive (92.3% Flash, 92.6% Pro), and BYPASS on the cheap production Flash beats SYNTHESIS on Pro at one twelfth the price (detail in Appendix F). Tier is the top rung of Figure 1’s ordering: it raises the collapse threshold; it does not remove the mechanism.

What failure looks like

Post-fix, SYNTHESIS no longer fabricates on delivered data (§ 6): of 99 failed SYNTHESIS turns in the size ladder, 96 have partial-credit fidelity ≈1.0\approx1.0: the rows emitted are real, but the set boundary is wrong. The dominant mode is truncation with a cascade: a turn truncates a 74-row result to 55 emitted, and the next turn filters that emission rather than the true set, the pre-fix mechanism (§ 6) on real rows. BYPASS cannot exhibit either mode; its failures are wrong-but-real filter arguments: of its 26 failed turns, 22 applied wrong arguments, 3 applied the right arguments over an under-delivered

@end_mid=1 result, and one produced no artifact (anatomy in Appendix § 15).

Cost and latency, not only accuracy

On the full 24-dialogue grid, mean per-turn latency is BYPASS 4.9s vs SYNTHESIS 31.7s, a 6.5× penalty. BYPASS’s latency and cost are flat in table size because it re-emits nothing, while SYNTHESIS’s scale with the set it must reproduce (Figure 5, Appendix § 15). At the largest tables a synthesis turn averages 146 seconds (194 at worst) and costs 8.7× BYPASS’s flat per-turn cost, for an 8%-fidelity answer. Dividing cost by accuracy (Figure 3): BYPASS produces a correct answer for ≈$0.012 at every table size, while SYNTHESIS’s cost per correct answer climbs to $2.53 beyond 100 rows, roughly 200×; tables that size are precisely the regime a search product exists for. Model tier does not escape this arithmetic (§ 5.3; Appendix § 15): a model upgrade rents part of what the architecture gives outright, and the rent is high.

Figure 5. Per-turn latency (top) and cost (bottom) against the actual delivered table size, over the size ladder’s scored filter turns. SYNTHESIS scales from ~5s to a 146s mean turn and, at the largest size, 8.7× BYPASS’s per-turn cost; BYPASS is flat at ~3.4s and ~$0.012 at every size. At the smallest tables SYNTHESIS is marginally cheaper, crossing above BYPASS by 25 rows. (§ 5.5’s 4.9s/31.7s are all-turns means over the full grid; this plot’s means cover the ladder’s scored filter turns only. § 5.1’s design labels are post-seat-filter counts; this figure uses the true delivered sizes.)

Errors cluster, only synthesis propagates values, and the depth test finds no super-linearity

Two independent instruments show synthesis failures are not independent draws, and bypass failures are. Lag-1 conditioning: after a failed turn, SYNTHESIS fails again 71.9% of the time against 31.4% after a clean turn (a recovery rate of 28.1%), while BYPASS recovers 95.7%. Value-tracing (a deliberately liberal upper bound; C.4): 53 of SYNTHESIS’s 177 traceable failures (30%) are propagated; 0 of BYPASS’s 25 are: for BYPASS no model-emitted set exists to trace, so propagation is precluded and the zero reflects that no counterexample arose. The mechanism is architectural: BYPASS recomputes each turn from pre-served state; SYNTHESIS filters its own poisoned output, and its residual failures are isolated translation slips.

Our pre-registered turn-count super-linearity test finds none. The reason is structural: with SYNTHESIS’s 56.9% first-turn failure rate the accumulation-only null saturates, leaving almost no measurable headroom above it; the test is degenerate in its own high-error regime (construction, excess-by-position and sample sizes in Appendix § 12). This redirected the analysis to table size (§ 5.1), which is monotone, controlled, and does not saturate.

Qualitative failure analysis

A grounding census classifies every emitting SYNTHESIS turn under both instrument versions (production model, 1,306 turns) by grounding: is each emitted flight number present in anything ever delivered into the run? The result is categorical: fabrication tracks grounding absence; corruption tracks re-emission. Under neither version did the model invent a table when the queried table had been delivered; every from-nothing fabrication sits over a missing one, and on fully empty context (31 turns, each downstream of a starved search) fabricated tables average 1.4–2.1 rows. A tiny emission is thus a signature of absence (full decomposition in Appendix § 13).

After the prompt fix: the fabrication is gone, the corruption is not

On the frozen instrument we audited every failed SYNTHESIS turn in the size ladder, 99 of them. 96 of the 99 carry partial-credit fidelity ≈1.0: every row the model emits is real. Fabricated tables were never a data-present phenomenon; post-fix, fabrication persists exactly where residual starvation persists (19 empty-context turns, mean 1.4 rows) and nowhere else. What did not vanish is the set error. The dominant modes:

  • Truncation. Asked to show a 74-row result, the model emits 55 of the 74 (real rows, correctly formatted, silently incomplete; d26_fra_bkk_size_m_early t2: GT 74, emitted 55, partial 1.0).

  • Cascade on its own emission. The next turn then filters the emitted 55 rather than the true 74: asked for the afternoon subset (GT 11), it returns 45: real rows over a corrupted base and a corrupted scope (same dialogue, t3). This is Exhibit 1’s propagation mechanism (Appendix § 13), on real rows instead of invented ones.

  • Identifier splicing under load. In 25 of 943 post-fix populated emitting turns an identifier is corrupted inside an otherwise-real row set, concentrated in large-table runs (specifics in D.1): field-level corruption that count-based checks cannot see.

The pre-fix runs pin the attribution (Exhibits 1–2, Appendix D)

The obvious deflationary reading, that synthesis merely had a bad prompt, is answered by comparing runs before and after the prompt repair. Pre-fix, the production prompt itself taught fabrication (§ 7): in one starved run SYNTHESIS answered a points filter over an undelivered payload with a plausible schema-valid table, then filtered its own fiction for five more turns; constrained decoding guaranteed the shape of the lie. BYPASS’s failure on the same starvation was visible emptiness: its table comes from state, and empty state renders empty (turn by turn in Appendix § 13, Table 8–Table 9). Repairing the prompt removed the fabrication but not the failure: fabrication tracked grounding absence under both prompts, while set corruption survived every prompt repair, scales with the delivered table (§ 5.1), and BYPASS cannot exhibit it. Attributing the gap to bad prompting is backwards: the better the prompt got, the more purely the residual gap measured the architecture.

tuser turnactive scopeGTemitted
1“Find award flights from LHR to FCO…”search: should deliver 48 rows; never firesnot scored—
2“Show me ones less than 10,800 points”<10,800<10{,}800 pts272
3“Only morning departures please”+ morning (composes)192
4“Of those, at least 2 seats”+ ≥2\ge2 seats (composes)172
5“Give me the cheapest one”cheapest of the 171 (of 17)1
6“Show all the morning ones again”back to t3’s full morning table192
7“And the 3 earliest of those”3 earliest of those 193 (of 19)2

Table 8. Exhibit 1 (d02_lhr_fco_close, sweep k1): a starved SYNTHESIS run fabricates a two-row table and then filters the fiction. GT is the oracle’s expected emission per turn; “emitted” is what the model displayed.

tuser turnactive scopeGTemitted
1“Find award flights from JFK to LHR…”search: should deliver 74 rows; never firesnot scored—
2“less than 30k points”<30k<30\mathrm{k} pts (over the full 74)232 (fabricated)
3“morning departures”+ morning (composes)62 (the same fiction re-emitted)
4“Actually, up to 87,499 points is fine!”cap relaxed to ≤87,499\le87{,}499 over the full table (morning no longer active)445 (more fiction)
5“cheapest one”cheapest of the 441 (of 44)1

Table 9. Exhibit 2 (d23_jfk_lhr_relax_close, sweep k2): starved SYNTHESIS fabricates placeholder rows, then fabricates more to fill a relaxed cap. The t4 view grows past t2’s 23 because the user is backing off the budget set at t2: “up to 87,499 points is fine” asks about everything under the new cap, not just the morning subset they had drilled into (the scope semantics of §4.2). Contrast t3, which narrows, and Exhibit 1’s t6, which returns to an earlier view.

Measurement integrity

We found three defects in our own ground truth, and report them

The range/flex arc initially scored BYPASS 14/27 (52%) vs SYNTHESIS 16/27 (59%), the only arc synthesis won. Challenged rather than published, it dissolved into three instrument defects (full narratives in Appendix § 14); after correction: BYPASS 25/25 (100%), SYNTHESIS 12/25 (48%), a 59-point swing (from −7pp to +52pp; turn accounting in E.1). The § 4.3 liveness verifier exists because of this, the corrected arc replaces the defective one in both conditions, and the same arc captured a user-facing production bug, fixed (Appendix § 14).

The starvation event: diagnosed and largely removed

In pre-fix runs, the search sub-agent sometimes never fired despite a ready verdict, failing the dialogue wholesale (21.2% of SYNTHESIS search turns, 5.2% of BYPASS’s). The mechanisms differ by condition (Appendix § 14); repairs took the post-fix rates on the main 24-dialogue corpus to 0.0% (BYPASS) and 9.7% (SYNTHESIS) (starved runs persist on other legs; § 5.1), and affected turns are reported as failures, not excluded. Excluding starved runs widens the pre-fix gap (re-scored: +35.3 → +37.5pp): the confound suppressed the effect rather than manufacturing it. Post-fix, excluding the 11 of 72 starved SYNTHESIS runs (BYPASS loses none) lifts SYNTHESIS to 47.4% (gap +44.9pp): the headline does not rest on residual starvation.

Instrument fixes: one move unexplained

Fixing the instrument moved the headline on the comparable subset (the 19 dialogues whose turn sets are identical under both instrument versions; 264 turns per condition) from +33.7pp to +49.6pp. The BYPASS side is accounted for (removing starvation and the fabrication-teaching prompt lifted it from 81.4% to 91.3%); the SYNTHESIS side is not: two fixes expected to raise it by roughly 4pp preceded a fall from 47.7% to 41.7%, a −6.0pp residual exceeding the pre-fix corpus’s own sweep-to-sweep (repeat-to-repeat) spread, which we flag as unexplained rather than absorb (full accounting in Appendix § 14). We declared the freeze before the final runs and report both readings; under either instrument version the gap exceeds +33pp.

A factorial separates size from position

In the 24-dialogue corpus, table size and turn position are correlated, so observational stratification alone cannot separate a size account from a first-turn account. The size ladders (§ 5.1) counterbalance position, and a 3×3 size-by-position Latin square settles it: size at fixed position (slot) is decisive (92→33→0% at slot 1; χ2=39.4\chi^2=39.4, p=3×10−10p=3 \times10^{-10}, surviving cluster bootstrap in 2000 of 2000 resamples) while no position main effect is detectable (χ2=0.8\chi^2=0.8, p=0.36p=0.36); BYPASS crosses the same square at 106/108. Regression, design and the secondary texture are in Appendix § 14.

Conclusion

In a deployed multi-turn agent, re-emitting a tool’s records through the model’s context each turn costs 46.6pp of per-turn field fidelity against carrying them as state on a typed side channel. The load variable is the size of the re-emitted record set: synthesis fidelity falls from 75–100% to ≤8%\le8\% beyond ∼74\sim74 rows, while BYPASS holds 99.2% at flat latency and cost. The gap appears at the first filter turn, the hazard is flat with depth, and our pre-registered super-linearity test finds none: its null saturates in the high-error regime it targeted. Synthesis failures are not independent (§5.6); BYPASS structurally cannot inherit one: it never re-emits records. Practitioners adopted typed side-channel delivery for rendering; this paper supplies the joined measurement from one deployed system, suggesting the field’s UX-motivated convergence was also, unknowingly, a correctness decision.

Limitations

Hypothesis H2 is motivated, not measured. Earlier engineering observations suggested that an answer-shaped demonstration in context overrides an on-point prompt prohibition (Appendix B reproduces the pilot that motivates this); the pre-registered 2×22 \times2 design of seed shape ×\times prompt strength remains future work, and this paper asserts nothing about H2.

The size manipulation has bounded coverage. The content-controlled ladder covers six routes but one cabin, with per-route ceilings of 112–198 rows against the 510-row payloads we hold; we declined a true ∼200+\sim200+ band through range captures because that would exchange the position confound for a query-shape confound at exactly the cell carrying the effect. Whether the decay moves with route complexity, cabin or record width is open, and the cell resolution localises the shape only coarsely: the pooled decay is monotone from 83% at K=5K=5 to 6% at K=100K=100, with no single knee we can name.

Headline numbers from one family; no seeds, no bit-reproducibility. The headline number is Gemini 2.5 Flash’s; § 5.3’s family and tier legs replicate the gap at the same scale, so the residual limitation is reproducibility, not coverage. Runs are independent repeats at production sampling (temperature 0.1 / top_p 0.5) with no random seed set, on a non-bit-deterministic serving stack: reproducible distributionally, not turn-for-turn; payload identity is derivable from dialogue specifications and capture manifests, but no content hash is in the traces.

Cross-family sampling parity is imperfect: Anthropic’s API rejects temperature and top_p together, so Claude runs clear top_p; Appendix § 14.§ 5 reports the measured direction of that deviation (a Gemini-only ablation, the one family where the combination is settable): if the direction transfers across families, the parity break biases against BYPASS and the Claude gap is conservative.

The comparison is joint by construction. As stated in § 3.2, BYPASS combines synthesis bypass with structured-state preservation and we do not separate them. A natural further condition would keep the same cross-turn state mechanism but deliver through the LLM; we did not run it, because it corresponds to no deployed pattern we are aware of. The cost is that our effect size is attributable to the pair, not to either member. One partial unbundling comes free: every scored turn of the content-controlled ladder (§ 5.1) is its run’s first filter turn, so no prior emission exists for state preservation to protect, and the collapse appears anyway; the synthesis half of the pair suffices to produce the size effect.

Statistical resolution. CIs are wide (dialogue-level bootstrap: BYPASS 88.3–95.9, SYNTHESIS 34.8–57.2; gap [+35.7, +57.1]) because n=24n=24 dialogues is the resampling unit, and strata are small: the largest-size ×\times deepest-position stratum holds 7 turns, and the two deepest turn positions are reached by only 8 and 3 dialogues respectively.

External validity and scope. Captures are restricted to non-stop and one-stop itineraries; turn scripts derive from expert demonstration sessions, not production traffic; every payload is a flight itinerary from one deployment; hotels and other verticals are out of scope. Scope conventions are common-mode: the § 4.2 rulebook was fixed before runs and applied identically to both arms, so a defective convention shifts both conditions’ levels rather than the between-arm contrast (the § 7.1 corrections were applied to both arms). Ground truth and the deployed arm share the production evaluator, so a shared evaluator defect would depress SYNTHESIS alone; the checks that do not share that code path are the authored expected-row assertions (Appendix § 10), the pre-run liveness verifier (§ 4.3), and the hand audits of all 99 ladder failures against the frozen payloads (§ 6.1). An independent re-implementation of the filter semantics, checking the oracle from outside the production codebase, is the verification this design still lacks (independence detail: C.2). The corpus-level marginal (+46.6pp) depends on the authored mix of delivered-table sizes; the size-conditional curves (Figure 1 and Figure 2, Table 5 and 6) are the transportable result, and readers can reweight them under any deployment’s size distribution. Vendor terms preclude releasing the payloads; the method (§4) replicates on any deployment’s own data.

Other open items. An outcome category for count-only-envelope turns would raise BYPASS’s measured rate honestly; a greedy-decoding ablation and a temperature sweep were designed and not run.

LLM and Agent Usage Disclosure

Research system: the system under study is itself a multi-agent system (Google ADK, Gemini 2.5 Flash). Both conditions were driven against production code; ground truth was computed from frozen real vendor payloads (Seats.aero, SerpAPI), not model-generated.

Affiliation: the authors are affiliated with AwardWise, the company that operates the production system under study.

Experimental design and analysis: the study design (the two conditions, the size and prefix manipulations, the dialogue scripts, the fidelity metric and its zero-out policy, and every ground-truth semantics decision for scope composition, refer-back restoration, and constraint revision) was specified and adjudicated by the authors, who personally audited the exhibits, per-turn tables, and failure cases reported here.

Writing and code: an AI assistant (Claude, Anthropic) was used substantially throughout: under the authors’ direction it drafted and revised the paper’s prose, implemented the replay harness and analysis code, executed experiment runs, and performed reference verification. Every quantitative claim traces to a committed artifact produced by deterministic scripts. The authors reviewed the analyses, directed each correction, and bear sole responsibility for all claims, interpretations, and final wording.

Artifact availability: the camera-ready artifact will release the 24 dialogue templates with knob specifications and the §4.2 scope-semantics rulebook; the replay harness, including the four-delta SYNTHESIS construction (C.9) and the model-override layer implementing sampling parity and the top_p ablation (E.5); the analysis layer, which reproduces every published number from committed result files; capture manifests (producing git SHA, request parameters, and payload content hashes); per-run traces with vendor payload contents stripped or hashed (tool calls, arguments, token usage, per-turn scores); and the internal pre-registration text with its commit hashes. Vendor terms preclude releasing the frozen payloads themselves.

The Size-Manipulation Dialogues and Per-Route Ladders

The manipulation should be checkable rather than taken on trust, so the dialogue set is reproduced here. The tables below describe the original d25–d30 six-dialogue design, which sits within the larger 20-dialogue ladder of § 5.1. Every dialogue has the same six-turn shape (search, two scored filter turns, second search, two more scored filter turns) with turns 2, 3, 5 and 6 scored and byte-identical across all six dialogues (Table 3); the only textual difference is the date in the two search turns, the manipulation instrument, because the capture date determines how large the delivered table is (Table 2). Each dialogue contains one treatment leg (size varies) and one control leg (always the same close_in capture: 25 delivered rows, 16 after the seat filter); the _early / _late suffix counterbalances which leg comes first in the dialogue.

tspeaker turnroleground truth
1“Find award flights from FRA to BKK on 2027-06-24, economy class”search (establishes context)—
2“At least 2 seats please”scored · fresh filter9 rows
3“Only afternoon departures”scored · composes with previous3 rows
4“Find award flights from FRA to BKK on 2026-08-14, economy class”search (establishes context)—
5“Same here — at least 2 seats”scored · fresh filter16 rows
6“Afternoon departures only, please”scored · composes with previous5 rows

Table 3. The dialogue script, shown for d25 (treatment first); the other five differ only in the two dates; their ground-truth counts differ accordingly (Table 4).

Turn 2/5 applies min_seats=2; turn 3/6 adds the published afternoon bucket (12:00–17:59) and composes with the previous turn. Ground truth is computed by the production filter evaluator over the frozen payload, never annotated (Table 4).

dialoguetreatment legcontrol leg
d25 (S, early)9 → 316 → 5
d26 (M, early)74 → 1116 → 5
d27 (L, early)194 → 8716 → 5
d28 (S, late)9 → 316 → 5
d29 (M, late)74 → 1116 → 5
d30 (L, late)194 → 8716 → 5

Table 4. Ground-truth row counts per cell.

The control leg is identical in all six dialogues (16 → 5 rows): synthesis runs whose 194-row treatment turn earned zero credit (one via an un-scoreable emission) still average 90% (9/10 control turns) on the byte-identical 16-row control leg of the same dialogues, which is what separates delivered-table load from context accumulation. Every scored turn passed an oracle liveness gate before any model was run (empty or trivially-repeated ground truth is rejected at authoring time), and the authored expected_gt_rows is asserted against the evaluator’s total_matched at run time, so payload drift cannot silently rescale the treatment variable.

The per-route date-varied ladders (§5.1)

Each route is a single-route, single-cabin ladder varying delivered size by capture date; Figure 4 plots these cells (Table 5).

The top-K ladder cells

The five pooled cells of the content-controlled (top-K) ladder (§ 5.1); these are exactly Figure 2’s plotted points (Table 6).

Failure anatomy on the content-controlled (top-K) ladder, FRA-BKK. Three failures were residual starvation (the search sub-agent reported success but its inner search never delivered rows, § 7.2; one in the full-payload K=198 cell, outside this table’s K range), and the mid-K failures are genuine truncations whose self-reported counts exceed their emitted rows by one (claimed 12, emitted 11; claimed 19, emitted 18). The K=100 collapse is starvation-free: all three FRA-BKK runs received their 100 rows and truncated their answers (20/30/34 rows emitted against a ground truth of 42).

dialoguetreatment sizeleg orderfirst search datesecond search date
d25_fra_bkk_size_s_early9 rowstreatment first2027-06-24 (release_edge)2026-08-14 (close_in)
d26_fra_bkk_size_m_early74 rowstreatment first2026-10-14 (near)2026-08-14 (close_in)
d27_fra_bkk_size_l_early194 rowstreatment first2027-04-14 (far)2026-08-14 (close_in)
d28_fra_bkk_size_s_late9 rowscontrol first2026-08-14 (close_in)2027-06-24 (release_edge)
d29_fra_bkk_size_m_late74 rowscontrol first2026-08-14 (close_in)2026-10-14 (near)
d30_fra_bkk_size_l_late194 rowscontrol first2026-08-14 (close_in)2027-04-14 (far)

caption: tbl:2: The six size-manipulation cells. Sizes are § 5.1’s design labels (post-seat-filter counts); the delivered payloads are 17, 103 and 198 rows (control 25).

:::

The Demonstration-versus-Instruction Pilot

During earlier engineering work, seeding a session with an answer-shaped model turn (“Found 40 award options . . . ” followed by a rendered table) made the agent pattern-complete the next filter request without calling the filter tool, fabricating a count (40→10, 7→2, 2→1 across three probes). A local A/B on one route, cabin and date (5 runs per variant): an answer-shaped seed with an explicitly strengthened system-prompt prohibition produced 0/5 grounded turns; a tool-trajectory seed produced 5/5 under either prompt. nn is small, uncounterbalanced, one route and one model: not a controlled experiment, motivating only the hypothesis that in-context demonstration governs grounding more strongly than an on-point prompt prohibition. The pre-registered design, a 2×22 \times2 of seed shape × prompt strength over ∼30 stratified seeded contexts × 5 runs per cell, is future work.

Method and Analysis Detail

The undefined-aggregate rule, worked

“Which day is cheapest?”, asked after a date-range search whose result spans several days, names no total order: ranking days requires a day-level aggregate over per-flight prices (the day’s minimum fare? its mean? award points or cash?) that the utterance never picks, so several answers are defensible and the turn is unscoreable by construction. The § 4.2 rule retired exactly such a turn, which had shipped in five dialogues; its replacement is an unambiguous window-spanning filter turn (“Morning departures across all the days”; Appendix § 14).

Thresholds are authored as payload-pinned percentiles with a lint asserting 20–80% survival; boundary probes are pinned strictly between two

K delivered5102550100
SYNTHESIS83% (15/18)72% (13/18)44% (8/18)39% (7/18)6% (1/18)
BYPASS100% (18/18)100% (18/18)100% (18/18)100% (18/18)100% (18/18)

caption: Content-controlled prefix ladder pooled over six routes. FRA-BKK’s single-route full-payload cell (K=198, n=3n=3: SYNTHESIS 0/3, BYPASS 3/3) is not pooled here; it completes § 5.1’s 93-run BYPASS total.

adjacent real values, so strict-versus-inclusive readings cannot both be defensible.

The oracle’s read path

Because the deployed system serves large result sets by reference (rows in state, envelope carrying a pointer), the oracle reads the full served row set through the same state path the client’s stream uses, rather than trusting the envelope’s inline rows. The liveness verifier of § 4.3 asserts, before any model is called, that no turn’s ground truth is empty or a trivial repeat of the previous turn’s, on every turn of every dialogue. Three properties keep the oracle independent of the arm that shares its code. First, it executes the authored arguments, pinned in the dialogue spec, never the model’s: BYPASS’s own translations are scored against it and fail when the divergence changes the selected rows (22 wrong-argument failures; F.5), so the deployed arm is not correct by construction. Second, the authored expected_gt_rows, derived by hand from each frozen payload, is asserted against the evaluator’s total_matched on every scored turn of every dialogue, across all turn families (C.10). Third, the evaluator is challenged rather than trusted: row-level hand audits cover all 99 ladder failures and both exhibits (§ 6.1, Appendix § 13), and the § 7.1 challenge found three ground-truth defects, fixed with both arms re-scored. We use the production evaluator because it implements exactly what the oracle needs, a deterministic filter over frozen state under the § 4.2 semantics; a second implementation built only for scoring would add disagreements needing their own adjudication without removing the shared-specification risk. An independent re-implementation of the § 4.2 semantics is the check this design still lacks (Limitations).

Partial credit and order, in full

Partial-credit fidelity over aligned pairs, post-fix: 98.2% for BYPASS and 82.7% for SYNTHESIS (mean over scored turns with an emitted artifact, n=338n=338 and n=333n=333; pre-fix: 93.3% and 72.8%), a 15.5-point gap against the zero-out headline’s 46.6.

Turns flagged ordered (limit-NN superlatives and explicit re-sort requests) are scored on emitted sequence as order-exact (the emitted sequence equals the ground-truth sequence) and order agreement (the fraction of positions holding the correct record). On the post-fix corpus’s 86 (BYPASS) and 85 (SYNTHESIS) ordered turns: order-exact 94.2% vs 51.8%, mean order agreement 95.9% vs 58.2% (pre-fix: 93.7% vs 61.2% and 94.2% vs 64.0%).

The analysis layer’s instruments

The corpus module’s inclusion rules: three independent sweeps per condition (the k=3k=3 repeats); the range/flex arc taken only from its corrected rerun (§ 7.1), the defective original excluded in both conditions; recovery re-runs may fill a dialogue lost to a network error; a run producing no scoreable output is scored as zeros on its expected filter turns.

Beyond the aggregates named in § 4.5, the analysis layer computes per-position marginal hazard; cumulative failure versus the accumulation-only null 1−(1−p)n1-(1-p)^n; and a per-dialogue counterfactual ratio: observed failures per dialogue divided by what an independent-failures null with the same per-turn rate predicts: 1.0 means failures land as if independent, above 1 means they cluster (0.95×\times for SYNTHESIS; C.11).

Value-tracing labels. A failed turn is traceable when a comparable row set exists for it (SYNTHESIS: the model-emitted table; BYPASS: the typed side-channel artifact; turns with no scoreable artifact are grounding failures, excluded). A traceable failure is propagated when its emitted rows are consistent with filtering a prior turn’s emission rather than the ground truth (set equality or subset over the identity keys of prior emissions), and independent when the error is fresh. When both explanations fit, the labeler resolves to propagated, so the propagated share is an upper bound; for BYPASS no model-emitted set exists, current or prior, so its zero is structural rather than a survived test.

The depth test’s degenerate null

With SYNTHESIS’s 56.9% first-turn failure rate, the accumulation-only null 1−(1−p)n1-(1-p)^n saturates at 81% by position 2 and 92% by position 3, leaving at most 8–19pp of measurable headroom above the null: super-linearity of any plausible magnitude is undetectable in this regime. SYNTHESIS’s marginal hazard is flat with depth (57/58/60/39/55% at positions 1–5); BYPASS’s position-1 hazard is 0.0% on the main 24-dialogue corpus: all 26 of its failed turns (§ 5.4) sit at later positions.

The SYNTHESIS construction’s consequences, in full

  • Translation error is common-mode. The same planner, with the same instruction lineage, decides what the user asked for in both conditions, so the residual gap is attributable to delivery.

  • Mechanical delivery makes SYNTHESIS stronger than its fork-true form. In the real fork-point system the sub-agent’s records pass through an LLM emission before reaching the planner; we remove that step, so our SYNTHESIS is a charitable version of the architecture it stands for.

  • BYPASS’s advantage is joint, and we name it. BYPASS benefits jointly from synthesis bypass (records never re-typed) and structured-state preservation (each follow-up recomputes from the original set). We do not separate them: the deployed system does both, and no deployed pattern we know of does one without the other, so the comparison is the pair, not a single-factor ablation.

Corpus composition and capture provenance

The 408 captures decompose as 270 fixed-date award, 18 award date-range, and 120 cash. The four date bands are close-in (≈\approx3 weeks out), near (≈\approx3 months), far (≈\approx9 months), and release-edge (≈\approx11 months, the edge of the booking window where award seats are first released). Captures are restricted at capture time to non-stop and one-stop itineraries with the pre-filter count recorded in each manifest, so the restriction is never a silent truncation; each capture records the git SHA that produced it, the request, and per-call vendor traffic.

Scale

Table 7 reproduces the study’s scale.

The four deltas, in full

The production agent tree is deep-copied in memory with exactly four deltas: (1) the planner’s instruction replaced by a variant walked block-by-block against production, removing only blocks that presuppose the context tool and client-side rendering; this variant, with the sampling inherited unchanged (§ 3.2), is what the abstract’s “lineage-matched” names: the same production lineage in both arms, differing only in delivery-entailed blocks; (2) the deterministic context tools removed from the planner’s toolset; (3) each flight-search sub-agent given a callback replacing its LLM-generated summary with {total_found, search_params, records} read verbatim from state, so the row list is delivered mechanically; (4) those clones dropping their output schema, which would otherwise strip the injected records. Neither condition sets an output-token cap: the only generation parameters configured in either arm are temperature 0.1 and top_p 0.5 (§ 3.2; Claude inherits temperature only, E.5); provider defaults apply identically across arms within each family. Each delta is entailed by the swap: in-context delivery cannot coexist with a deterministic context tool, with instruction blocks that presuppose it, or with output schemas that strip the injected rows; the abstract’s “differing only in delivery architecture” quantifies over this one variable, implemented as these four deltas.

Dialogue inventory

Six arcs cover narrow-and-pick, award/cash comparison, release-edge with truthful empty results, range/flex, multi-table refer-back, and compound power-user filters; turn families span the filter vocabulary: time buckets, day-rollover arrival filters, seat and program constraints, superlatives with sort and limit, negated predicates, refer-back, and cumulative narrowing. The four variation knobs are payload swap, order permutation, follow-up addition or omission, and phrasing alternates. Dialogue identifiers encode route and design cell (d02_lhr_fco_close, d26_fra_bkk_size_m_early; Appendix § 10).

Operational detail moved from the body

The fifteen scored fields are flight number, airline, booking source, mileage cost, taxes, cabin, date, origin, destination, departure and arrival timestamps, duration, stops, remaining seats, and cash price. source is included in the identity key because the same flight legitimately appears once per booking program. The much smaller partial-credit gap is the expected signature: SYNTHESIS frequently emits a nearly-right set, which is still a wrong answer to a filter question. Turns flagged ordered (limit-N superlatives and explicit re-sorts) are scored on emitted sequence, reported as their own axis and never folded into fidelity; mean order agreement is 95.9% vs 58.2% (C.3).

Every run persists a readable trace and a complete raw record (tool calls, model texts, token usage, final state, per-turn scores). The analysis layer computes dialogue-level bootstrap CIs (2000 resamples, dialogue as the unit, following Schessl, 2026) plus per-position and lag-1 hazards, cumulative failure versus the accumulation null, accuracy by table size, and runs a value-tracing labeler that marks each traceable failure independent or propagated (definitions in C.4).

§5.6’s cut detail, with C.5’s test construction: SYNTHESIS’s excess over the accumulation-only null is negative at every position (−14.8 to −22.3pp at positions 2–5) and its failures per dialogue land at 0.95× the per-dialogue independence prediction (C.4); lag-1 samples are n=146/121n=146/121 (SYNTHESIS, 71.9% vs 31.4%) and n=23/244n=23/244 (BYPASS, 4.3% vs 10.2%; the nominal inversion, a lower hazard after a failure, is a single-failure cell, 1/23, consistent with independence rather than any protective effect).

The Exhibits and the Fabrication Census in Full

The census decomposition

Of 1,275 context-populated emitting turns (both corpora, production model), 1,245 emit only delivered rows; the 30 exceptions decompose into identifier-level corruption of real rows (25 turns: connecting segments of multi-leg itineraries dropped or recombined, or a single garbled code, inside an otherwise-real table; e.g., a real “..., BR62” tail swapped to “BR68”, one run splicing 24 of its 103 emitted rows) and five turns in two pre-fix multi-search dialogues where the queried award table itself had never been delivered (the context held only cash rows) and the model produced two-row placeholder tables (UA123, JL789, KL456) for the missing set. Every from-nothing fabrication in either corpus sits over a missing queried table. All exhibits are schema-valid SYNTHESIS turns, and the census finds no third variant in which a delivered table is present and a fabricated one replaces it.

Exhibit 1: fabrication under grounding absence (the plausible face)

Dialogue d02_lhr_fco_close, sweep k1. The frozen payload is LHR→FCO on 2026-08-14, economy: 48 real records whose only mileage values are {7500, 14000, 15000, 22500, 25500, 28200, 47500} and whose booking sources are {aeroplan, alaska, delta, flyingblue, qantas, virginatlantic}. The readiness gate is a deterministic pre-search check that all required slots are filled; its “Ready” verdict licenses the search dispatch. In this run it returned “Ready”, but the search sub-agent was never invoked (the pre-fix starvation event, §7.2), so the context held no award records at any turn. Asked at turn 2 for “ones less than 10,800 points” (ground truth 27 rows, all at exactly 7,500), SYNTHESIS emitted (identity fields shown; the records’ other fields elided):

{"flights": [
  {"flight_number": "BA234",
   "mileage_cost": "10000",
   "source": "avios"},
  {"flight_number": "VS123",
   "mileage_cost": "9500",
   "source": "virginatlantic"}],
 "summary": "I found 2 award flights for less than
  10,800 points."}

None of it exists in the payload: not the flight numbers, not the mileage values, not the source avios; schema_parsed: true. The fabrications are plausible: under the user’s cap, on carriers that credibly fly London→Rome, with textbook flight numbers: nothing distinguishes the reply from a genuine two-row result. And the fiction propagates: the remaining turns filter, sort and re-emit the invented rows. In Table 8 (and Table 9), each row’s GT is the correct table after executing that turn’s request under the scope shown, computed by the oracle from the frozen payload and independent of what the run did.

Fidelity is zero for the rest of the dialogue because the model is filtering fiction; on the same starved dialogue, BYPASS emits zero rows, a visible failure.

Exhibit 2: fabrication under starvation

Dialogue d23_jfk_lhr_relax_close, sweep k2. At turn 1 the readiness gate returned “Ready” but the award search sub-agent was never invoked, so no records ever entered the context: the frozen payload this dialogue should have delivered holds 74 rows (the same no-search event appears in BYPASS runs; see § 7). Asked at turn 2 for flights under 30k points, SYNTHESIS made zero tool calls and emitted two placeholder-shaped records (UA123 / 25000 / united_miles, DL789 / 28000 / delta_skymiles): round point values, flight numbers absent from the frozen payload, non-production source strings. Asked at turn 4 to widen the cap (“Actually, up to 87,499 points is fine!”), the model does not return to any source: it generates three additional fictional rows (BA456 / 35000 / avios, VS101 / 40000, AA200 / 80000), one flight per major carrier, to fill the widened window (Table 9).

Under the same starvation, BYPASS emits zero rows (visibly empty, so the user retries) where SYNTHESIS emits a complete, plausible, schema-valid table that is entirely false.

Integrity Detail

The three ground-truth defects, in full

(1) An unscoreable turn: “Which day is cheapest?” has no defined aggregate (Appendix § 12) and was retired under the § 4.2 rule, replaced with an unambiguous window-spanning filter turn (“Morning departures across all the days”). (2) Stale downstream ground truth: after the replacement, later turns’ ground truth was not re-composed, penalising the deployed condition for being right: one turn demanded 64 rows (date only) while BYPASS’s 9 (morning ∩ date) was correct under our own scope convention. The arc’s scored-turn count drops from 27 to 25 in the correction: for the two payloads with zero morning departures (d16, d17) the replacement turn is itself the seats filter, so each dialogue’s later standalone “Of those, at least 2 seats” turn became a no-op duplicate and was removed in the re-composition; both conditions had answered both turns correctly, so the drop favours neither side. (3) A hardcoded day pick (range-start + 2) on one payload selected a day with zero morning departures, unscoring every downstream turn.

The production bug the same arc captured: a date-window argument the context tool did not accept was silently dropped by the framework, so a user asking for a week received one day (2 rows instead of 6); post-fix the same turns return 467 / 6 / 6 rows, exact. A third production fix from the same study: duplicate cash itineraries had been reaching users (61 rows across 21 captures; now zero). The two ground-truth semantics corrections of § 1 are § 4.2’s refer-back and constraint-revision scope rulings.

The starvation mechanisms

For BYPASS, the production prompt’s worked examples taught the model to answer instead of searching; the fabrication fix (§ 6.1) repaired it, taking the rate on affected dialogues from 14.2% to 1.4% at the time of the production fix (on the post-fix main corpus the BYPASS rate is 0.0%; § 7.2). For SYNTHESIS, output_schema biases the planner toward emitting a schema-valid “searching…” object in place of the tool call, an ADK-level behaviour that an ordering instruction mostly but not fully counteracts; the residual post-fix 9.7% is schema pressure winning over the instruction. The schema-versus-tool-call tension is documented in the framework’s own tracker: declared output schemas are unreliably honoured when tools are present ([1]), and an input-schema variant suppresses an inner agent’s tool calls entirely ([2]).

The position question: the three instruments’ design

The size ladders (§ 5.1) counterbalance position: each size band appears at both an early and a late position. An observational regression over all 527 scored SYNTHESIS turns of the main 24-dialogue corpus and the size ladders (logistic, correct ∼log⁡2(size)+position\sim\log_{2}(\mathrm{size}) + \mathrm{position}) puts the size term at

χ2=83.3\chi^2=83.3 (p=7×10−20p=7\times10^{-20}) with position controlled, leaving a small residual position term (χ2=6.1\chi^2=6.1, p=0.013p=0.013) that observational data cannot adjudicate (accumulated context grows with position by construction), and that the Latin square was designed to settle. The square: two routes, three delivered sizes (9/11, 70/74, 174/194 rows) each appearing at each dialogue slot exactly once per route, byte-identical scored turns in every slot, oracle-verified identical ground truth for a size at every slot.

The square’s cell pattern hints at one real secondary texture: small-table accuracy drifts down when large tables precede it in context (92/75/58% across slots) while large-table accuracy does not worsen, a mild accumulated-context tax, far from accounting for the size effect.

The headline movement, accounted in full

The BYPASS side: removing starvation (0.0% post-fix) and the fabrication-teaching prompt lifted BYPASS from 81.4% to 91.3% (+9.9pp). The SYNTHESIS side: two of the eight fixes (timestamp canonicalisation and tie-break re-authoring) removed penalties only SYNTHESIS could incur and were expected to raise it by roughly 4pp, yet SYNTHESIS fell from 47.7% to 41.7% (−6.0pp).

Per-dialogue movement is large in both directions (+41.7 to −46.7pp on individual dialogues), consistent with SYNTHESIS’s high run-to-run variance, but the aggregate move exceeds the pre-fix corpus’s own sweep-to-sweep spread and we do not have a mechanism for it.

Sampling-parity ablation (Limitations)

Claude runs with top_p unset because Anthropic’s API rejects temperature and top_p together. A Gemini ablation (the only family where the combination is settable) measured clearing top_p at −15.7pp for BYPASS and −5.9pp for SYNTHESIS; if the direction transfers across families, the parity break biases against BYPASS and the Claude gap is conservative.

Results Detail

Cross-family full matrix

Table 10 gives the per-condition cells behind §5.3’s paired gaps (n=339n=339 scored turns per cell: all 24 dialogues; Wilson 95% CIs on cells, dialogue-level bootstrap CIs on gaps; the cell intervals pool turns and are descriptive; inferential gap claims rest on the dialogue-level CIs). Extending the repeats from three to five sweeps for the two non-Gemini families (GPT-4o-mini and Claude Haiku 4.5; $n=565 per cell) moves no cell by more than 1.1pp and no gap by more than 1.4pp.

Trend-test robustness

Because repeats within a route are correlated, the decreasing top-KK trend was also verified under route-level cluster bootstrap: it survives, appearing in 2000 of 2000 resamples, i.e., pp below the bootstrap’s resolution floor (p<5×10−4p < 5 \times10^{-4}).

Family failure shapes, specifics

GPT’s garbling includes type-broken fields (“21h 55m” where the schema wants an integer) and responses that drop required fields (a flights array with no summary, the mirror image of Claude’s omission), including 13 runs with no scoreable output at all, four of them because the re-emitted tables inflated the transcript past the model’s 128k context window; BYPASS, whose context never carries the table, has no such run. Claude’s omission mode appeared on thirty-five runs: twenty-nine with an intact prose summary counting the delivered table correctly (“Found 48 award flights. . . ”) and six emitting an empty object. Omission is even cheap (4.0 s per Claude synthesis turn; not emitting the table is fast), where garbling is not (16.0 s per GPT synthesis turn). A three-dialogue pilot had suggested Claude’s synthesis was markedly stronger (64.7% there); that was an artifact of the pilot’s small tables, which rarely trigger the omission mode. GPT’s ladder zero at K=5K=5 (0/6) against 33% at K=10K=10 is a real non-monotonicity of its garble-dominated mode. A power note on size-flat BYPASS: a zero-failure cell excludes, with 80% probability, only a true per-turn decay of at least 8.5pp at n=18n=18 (prefix-ladder cells) and 23.5pp at n=6n=6 (family-ladder cells); the exclusion of a large-size decay rests on pooling across the four instruments (93/93 prefix ladder, 140/144 family ladder, 238/240 date-varied ladder, 106/108 Latin square), where every BYPASS miss is starvation-shaped (empty delivery, no rows reaching state) and none is size-graded.

The tier’s residual failures and per-answer costs

Gemini 2.5 Pro’s full-corpus cells are in the cross-family matrix (F.1); a Pro synthesis turn averages 35.5 s against Flash’s 22.8 s (scored filter turns, the same turn class Figure 5 plots; § 5.5’s 31.7 s is the all-turns mean; 1.6×) and $0.141 against

Flash’s 0.028(5×),twelvetimesBYPASS−on−Flash’s0.028 (5\times), twelve times BYPASS-on-Flash’s0.012. Residual synthesis failures carry the size signature of § 5.1 (in the tier pilot runs: near-miss truncation on 40–77-row tables, 77 delivered → 74/76 emitted; collapse at 138 → 0 and 76 emitted).

On the content-controlled ladder Pro’s correct full-payload answers cost $0.67 each (3 of 6 correct at a mean $0.336 per turn) against BYPASS’s flat $0.012. A pre-fix probe on five failure-selected dialogues agreed with the tier comparison’s direction: Flash-BYPASS 60.0% vs Pro-SYNTHESIS 52.0%.

Failure anatomy

Beyond truncation, SYNTHESIS over-emits by carrying a filter it should have dropped. On the deployed Gemini 2.5 Flash, BYPASS’s 22 wrong-argument failures are translation and scope errors; corpus-wide, its applied arguments differ from the authored spec on 27.2% of argument-recoverable scored turns (92/338 of 339), but 70 of those 92 argument sets (76%) still select exactly the right rows: argument-level noise absorbed by the deterministic evaluator, where synthesis would have re-typed every field.

Wrong-argument failures order exactly inversely with the four BYPASS accuracies (Table 11), including the Pro/Flash flip at the top, and the harmless share rises with BYPASS accuracy (41% to 82%): BYPASS’s floor is translation quality (how well a model maps each follow-up’s intent into filter arguments, § 3.1), measured.

modelBYPASS turns correctargument divergenceof which harmlessfailed turns from wrong args
GPT-4o-mini67.0%45.6% (150/329)41% (61/150)79% (89/112)
Claude Haiku 4.582.9%32.0% (103/322)63% (65/103)66% (38/58)
Gemini 2.5 Flash92.3%27.2% (92/338)76% (70/92)85% (22/26)
Gemini 2.5 Pro92.6%31.2% (105/336)82% (86/105)76% (19/25)

Table 11. Cross-family translation error over scored BYPASS turns (n=339n=339 per model; rows ordered by ascending BYPASS accuracy). Divergence = applied filter arguments differ from the authored spec; denominators are turns with trace-recoverable applied arguments (the shortfall of 10/17/1/3 turns is all failures). Harmless = divergent arguments that still select exactly the right rows, so divergent minus harmless = the wrong-argument failures. Last column: wrong-argument failures over all failed turns; the remainder are 13/3/3/3 failures with right arguments plus the 10/17/1/3 unrecoverable, closing each total (112/58/26/25).

Extended related work and system detail

The body’s § 2 keeps the closest works; the full positioning:

Structured output: validity is not correctness. Constrained decoding guarantees shape, not content: format restriction degrades reasoning ([34]), generation shifts onto lower-confidence paths ([29]), validity hits 100% while accuracy falls and wrong-but-valid outputs dominate ([27]), task accuracy coexists with zero format accuracy ([11]); grammar-constrained infrastructure ([38, 9]) and schema-compliance benchmarks ([12]) show the shape guarantee is real. Hoang [17] measured the resulting structured hallucination in a production travel agent single-turn, introducing the field-fidelity metric and zero-out policy we adopt. This paper compares two delivery architectures on a new dataset in a multi-turn design (results are not compared to that study’s), showing the failure survives into a multi-turn agentic pipeline over real vendor records, where corrupted output re-enters the context and is filtered again.

Multi-turn agent evaluation and error compounding. Tool-use work established reliable invocation ([31, 25] (Yao et al. 2023)); our subject is what happens to the returned data afterwards. τ\tau-bench established multi-turn tool-agent evaluation ((Yao et al., 2024)), MINT showed multi-turn ability is not predicted by single-turn ([36]), a 39% average single- to multi-turn drop with poor recovery is documented ([20]), and self-conditioning, in which past errors raise later error rates, is the closest mechanism match ([32]); failure taxonomies ((Zhu et al., 2025)), stability analyses ([40]), reliability framing ((Rabanser et al., 2026)), a Tool-Skip category matching our secondary observation ([33]), and injected corruption producing over-trust and reduced recovery ((Zhu et al., 2026)) situate the problem. Closest are three unreviewed preprints: AgentProp-Bench (synthetic tasks, injected errors) defers multi-turn injection (our question) to future work ([16]); “Beyond Task Completion” reports 27–78% of τ\tau-bench successes as corrupt ([5]): procedural integrity on τ\tau-bench’s mock domains (airline and retail), where ours is field- and row-level data fidelity; “Reason Less, Verify More” finds, on a budget agent, 78% of failures are silent wrong-state and gates the write side ([28]), where ours bypasses the read side. All index degradation by turn count or injected fault on simulated or injected data; ours is indexed by delivered record-set size, on real data, and the corruption is self-inflicted.

Architectures that keep data out of the model. CodeAct routes actions through executable code ([37]); “heavy brain, precise hands” designs pair LLM planning with a deterministic backend ((Zeng et al., 2026)); trace-level compliance verification adds a deterministic gate ([10]); the prompt-injection literature converges on context-minimization ([4]). Our mechanism is in this family; its practitioner form exists in LangGraph’s custom stream mode, the Vercel AI SDK’s data parts, and the AG-UI protocol, but no framework we surveyed ships typed structured delivery as the default end-to-end rendering path, and the public motivation is rendering predictability, never correctness [6]. Survey method: for each framework (LangGraph, Vercel AI SDK, CrewAI, Microsoft Agent Framework, OpenAI Responses API, CopilotKit/AG-UI; documentation accessed 28 July 2026) we read the current public streaming and rendering documentation and asked one question: does structured tool output reach the client renderer without re-entering the model’s context on the documented default path? Default means the quickstart path, not an available opt-in.

perform hub routing, deduplication and canonicalisation before writing the record set to session state.

Evaluation methodology. Turns within a dialogue are not independent, and turn-level significance claims frequently fail cluster-robust correction [30]; every gap interval here therefore resamples dialogues, not turns, a choice survey work also supports [15].

Reasoning inside dialogue. Closest in setting is BOULDER [19]: reasoning over MultiWOZ-derived travel domains, isolated and embedded in dialogue, with a substantial isolated-to-dialogue drop attributed largely to multi-turn framing. BOULDER measures whether the model computes correctly over records it was given; we measure whether the record set survives delivery; it has no synthesis-bypassing arm and no row- or field-level fidelity metric, and its ablations cannot separate the assistant’s dual generation role from longer context; the generation burden can only be removed architecturally.

Deployed-system detail

Award inventory is seats bookable with loyalty points, priced per booking program, so the same flight legitimately appears once per program; cash is ordinary priced tickets. The search sub-agents

References

  1. [1]ADK contributors. 2025. ADK issue #3969: Output schema consistently ignored by LLM agent with tools. GitHub issue, opened 18 December 2025.
  2. [2]ADK contributors. 2026. ADK issue #5926: AgentTool with input_schema and LiteLLM/Claude skips inner agent tool calls. GitHub issue, opened 1 June 2026.
  3. [3]Anthropic. 2025. Code execution with MCP. Engineering blog post, 4 November 2025.
  4. [4]Luca Beurer-Kellner et al. 2025. Design patterns for securing LLM agents against prompt injections. Preprint, arXiv:2506.08837.
  5. [5]Hongliu Cao, Ilias Driouch, and Eoin Thomas. 2026. Beyond task completion: Revealing corrupt success in LLM agents through procedure-aware evaluation. Preprint, arXiv:2603.03116. Unreviewed preprint; τ-bench mocks with a simulated user.arxiv.org/abs/2603.03116
  6. [6]CopilotKit. 2026. Controlled generative UI; AG-UI protocol specification. Online documentation, accessed 28 July 2026. AG-UI specification: https://docs.ag-ui.com.
  7. [8]CrewAI. 2026. CrewAI documentation. Online documentation, accessed 28 July 2026.
  8. [9]Yixin Dong et al. 2024. XGrammar: Flexible and efficient structured generation engine for large language models. Preprint, arXiv:2411.15100.arxiv.org/abs/2411.15100
  9. [10]Laïla Elkoussy and Julien Perez. 2026. AgentLTL: A trace-verification framework for measuring, enforcing, and training procedural compliance in tool-using LLM agents. Preprint, arXiv:2607.02599.arxiv.org/abs/2607.02599
  10. [11]Cosimo Galeone, Minsu Park, Giuseppe Ettorre, and Daniele Ligorio. 2026. When correct isn’t usable: Improving structured output reliability in small language models. Preprint, arXiv:2605.02363.arxiv.org/abs/2605.02363
  11. [12]Saibo Geng et al. 2025. JSONSchemaBench: A rigorous benchmark of structured outputs for language models. Preprint, arXiv:2501.10868. Cited for schema-compliance measurement only.arxiv.org/abs/2501.10868
  12. [13]Google. 2025. ADK samples: travel-concierge. GitHub repository google/adk-samples.
  13. [15]Shengyue Guan, Jindong Wang, Jiang Bian, Bin Zhu, Jiang-yang Lou, and Haoyi Xiong. 2025. Evaluating LLM-based agents for multi-turn conversations: A survey. Preprint, arXiv:2503.22458.DOI
  14. [16]Bhaskar Gurram. 2026. Evaluating tool-using language agents: Judge reliability, propagation cascades, and runtime mitigation in AgentProp-Bench. Preprint, arXiv:2604.16706. Unreviewed preprint; synthetic tasks, injected faults.
  15. [17]Tung Thanh Hoang. 2026. Structured hallucination in tool-using agents: Measuring and mitigating LLM synthesis corruption in production. In Second Workshop on Agents in the Wild: Safety, Security, and Beyond. AIWILD @ ICML 2026.
  16. [18]Ziwei Ji et al. 2023. Survey of hallucination in natural language generation. ACM Computing Surveys. arXiv:2202.03629.DOI
  17. [19]Ivan Kartáč, Mateusz Lango, and Ondřej Dušek. 2026. Reasoning gets harder for LLMs inside a dialogue. In Proceedings of ACL 2026, pages 12263–12303.
  18. [20]Philippe Laban, Hiroaki Hayashi, Yingbo Zhou, and Jennifer Neville. 2025. LLMs get lost in multi-turn conversation. Preprint, arXiv:2505.06120.arxiv.org/abs/2505.06120
  19. [21]Philippe Laban, Tobias Schnabel, and Jennifer Neville. 2026. LLMs corrupt your documents when you delegate. Preprint, arXiv:2604.15597.arxiv.org/abs/2604.15597
  20. [22]LangChain. 2026. LangGraph documentation: custom stream mode. Online documentation, accessed 28 July 2026.
  21. [23]Microsoft. 2026. Microsoft agent framework documentation: workflow events. Online documentation, accessed 28 July 2026.
  22. [24]OpenAI. 2026. OpenAI responses API documentation. Online documentation, accessed 28 July 2026.
  23. [25]Shishir G. Patil et al. 2024. Gorilla: Large language model connected with massive APIs. In NeurIPS 2024. arXiv:2305.15334.arxiv.org/abs/2305.15334
  24. [27]Jaideep Ray. 2026. The constraint tax: Measuring validity-correctness tradeoffs in structured outputs for small language models. Preprint, arXiv:2605.26128.arxiv.org/abs/2605.26128
  25. [28]Vikas Reddy, Sumanth Reddy Challaram, and Abhishek Basu. 2026. Reason less, verify more: Deterministic gates recover a silent policy-violation failure mode in tool-using LLM agents. Preprint, arXiv:2607.07405. Venue self-declared and unverified.arxiv.org/abs/2607.07405
  26. [29]Maximilian Schall and Gerard de Melo. 2025. The hidden cost of structure: How constrained decoding affects language model performance. In Proceedings of RANLP 2025.DOI
  27. [30]Ferdinand M. Schessl. 2026. The autocorrelation blind spot: Why 42% of turn-level findings in LLM conversation analysis may be spurious. Preprint, arXiv:2604.14414.arxiv.org/abs/2604.14414
  28. [31]Timo Schick et al. 2023. Toolformer: Language models can teach themselves to use tools. Preprint, arXiv:2302.04761.arxiv.org/abs/2302.04761
  29. [32]Akshit Sinha, Arvindh Arun, Shashwat Goel, Steffen Staab, and Jonas Geiping. 2025. The illusion of diminishing returns: Measuring long-horizon execution in LLMs. Preprint, arXiv:2509.09677.arxiv.org/abs/2509.09677
  30. [33]Harsh Soni. 2026. ToolFailBench: Diagnosing tool-use failures in LLM agents. In Workshop on Agents in the Wild (AIWILD) and Workshop on Failure Modes of Agentic AI (FAGEN) @ ICML 2026. ArXiv:2607.04686.arxiv.org/abs/2607.04686
  31. [34]Zhi Rui Tam et al. 2024. Let me speak freely? A study on the impact of format restrictions on performance of large language models. In EMNLP 2024 Industry Track. ArXiv:2408.02442.arxiv.org/abs/2408.02442
  32. [35]Vercel. 2026. Vercel AI SDK documentation: Data parts. Online documentation, accessed 28 July 2026.
  33. [36]Xingyao Wang, Zihan Wang, et al. 2024a. MINT: Evaluating LLMs in multi-turn interaction with tools and language feedback. In ICLR 2024. ArXiv:2309.10691.arxiv.org/abs/2309.10691
  34. [37]Xingyao Wang et al. 2024b. Executable code actions elicit better LLM agents. In ICML 2024. ArXiv:2402.01030.arxiv.org/abs/2402.01030
  35. [38]Brandon T. Willard and Rémi Louf. 2023. Efficient guided generation for large language models. Preprint, arXiv:2307.09702.arxiv.org/abs/2307.09702
  36. [39]Sam Wiseman, Stuart Schieber, and Alexander Rush. 2017. Challenges in data-to-document generation. In Proceedings of EMNLP 2017.arxiv.org/abs/1707.08052
  37. [40]Weimin Xiong, Ke Wang, Yifan Song, et al. 2025. More vulnerable than you think: On the stability of tool-integrated LLM agents. Preprint, arXiv:2506.21967.arxiv.org/abs/2506.21967

Paper details

vlnc:W_8784s849
Version
v1 · latest
Published
Fresh — added to the Hub within the last 30 days
Added
License
CC BY 4.0
Usage

views · 11

downloads · 0

A view counts once per browser per day, a download each time the PDF is requested.

  • Computer Science

Claude

View statement
Looking for related papers…
Contents