betaFeedback

WorksComputer Science

Right Numbers, Wrong Questions: Evaluating LLMs on Underspecified Financial Queries

Abstract

Financial assistants answer questions that sound factual but depend on what the user leaves unsaid: what a data figure measures, which product variant is meant, and the investor's situation and the rules that apply. An assistant that ignores these can still answer with a real number from its data, one that answers a different question. We call this frame collapse. We study it in Indian mutual funds, where product variants and tax rules are formally defined, so the correct answer can be computed for every reading. One study tests each challenge. Given an ambiguously labelled figure, models used it, often under the user's label. Given every alternative labelled, they listed them all, but the layout decided which came first. Missing a fact about the investor, weaker agents assumed a default or an outdated tax rule; requiring the fact did not guarantee its use. Because such answers match their data, checks against the data pass them. An AI judge caught mislabelled figures only when told what each field means, and still passed answers that silently chose another reading. We show how these failures arise, not how often, and propose controls for financial assistants.

Introduction

Financial assistants answer questions that sound factual but rarely are. The answer to “What is this fund’s return?” or “How much will I get if I sell?” depends on things the user does not say. The assistant must work these out before answering, and three things make that hard:

  1. Its data can be ambiguous. A field named growth may hold a fund category’s net inflow or its change in assets; a benchmark figure may come from the price index or the total-return index.

  1. Products come in variants. Each mutual fund is sold in two plans, Direct and Regular, with the same portfolio but different fees (Appendix A). Investors also put money in either as a lump sum or monthly. Each variant has its own return.

  1. The answer depends on the investor and the rules. What a sale pays out depends on who sells, for example whether they live in India, and on the tax rule in force.

Take “What are the returns of HDFC Flexi Cap Fund?” Over the same three years, its Direct plan returned 14.72% a year and its Regular plan 13.97%. Both figures are correct. Which one answers the user depends on the plan they hold, and the question does not say. On Rs. 10 lakh (Rs. 1 million), the gap is about Rs. 29,300.

An assistant that leaves these questions unsettled still answers, with a real number that answers a different question. We call this frame collapse: the model commits to one interpretation of an underspecified question without establishing or stating it. Unlike a numerical error or a hallucination, the figure is real and traceable to the evidence, so a check that verifies an answer against its evidence passes it.

We study frame collapse in Indian mutual funds. Plans, options and investing patterns are formally defined, NAV records are public, and tax rules depend on the investor and changed recently, so we can compute the correct answer for every reading. The setting matters in practice too: the regulator holds firms responsible for what their AI tools say [25]. Each study tests one challenge, and Figure 1 shows a real case from each:

Figure 1. One real case for each challenge. Model answers are quoted from the study logs, shortened with ellipses (currency shown as Rs.). (1) Study 1, a lookup tool returning one ambiguously named record. (2) One Study 2 question asked twice, with the table’s plan columns in each order; each figure is that plan’s three-year annualised return. (3) A Study 3 run of a model chosen because it failed in exploratory runs: a profile tool holding the investor’s NRI status was available but never called. The panels illustrate each failure; § 5 reports how often it occurred, and Appendix Figure 4 summarises every measurement.

  • Ambiguous data (Study 1). Given one ambiguously named figure that was not the quantity asked for, seven of eight models used it, often under the user’s label. Given every candidate, labelled, they rarely erred.

  • Product variants (Study 2). Given both plans, labelled, most current models listed both, but the table’s column order decided which came first.

  • The investor and the rules (Study 3). Weaker agents assumed a default investor or an outdated tax rule. Requiring the fact got it fetched, but not always used. The newest models gathered it unprompted.

  • Checking. An AI judge caught mislabelled figures only when told what each data field means, and still passed answers that silently chose another reading. A prompt that checks the data against the question cut wrong answers, mostly by asking the user back.

One description fits these results: what reaches the model’s context sets the interpretation, and defaults fill the gaps. We formed it after the studies and did not test it, and the most careful current models were the clearest exceptions. The studies show how these failures arise, not how often, and AI models coded all answers, with a human check on a sample (§ 7).

Contributions. (1) A taxonomy of how a financial question can be underspecified, with a materiality rule (§ 2). (2) Three studies, one per challenge, with a computed reference for each reading (§ 4–§ 5). (3) Evidence on what a judge needs to catch these answers, and what a prevention check costs. (4) Controls for financial assistants, separating tested from proposed (§ 6). Appendix A is a glossary of Indian terms; Appendix § 10 gives full counts, denominators and statistics.

The Problem: What Must Be Settled Before Answering

Table 1 extends the opening example: one question, five defensible readings, five correct figures. Each answers a different question.

Plan, option, modeMeasure, periodReturn
Regular, Growth3-year annualised13.97%
Direct, Growth3-year annualised14.72%
Regular, IDCW payout3-year XIRR, before tax15.12%
Direct, Growth; Rs. 5,000 a month (SIP)5-year XIRR13.66%
Direct, Growth; one lump sum5-year annualised15.67%

Table 1. Five defensible readings of “What are the returns of HDFC Flexi Cap Fund?”, computed from AMFI NAV records through 30 September 2026, net of fund expenses, before exit load and tax. XIRR is the annualised return over dated cash flows (glossary, Appendix A). The IDCW (payout) figure is the investor’s own cash-flow return, not a fund return; it exceeds Growth even though that investor ends with less (Appendix § 10.§ 1).

Dimensions. A question has one appropriate answer only once its material dimensions are settled. Table 2 groups fourteen dimensions: what is measured, how, against what, and in which context. We compiled the taxonomy from regulatory definitions and practice; it is not measured from real queries.

GroupDimensionsExample of an unsupported interpretation
A. ReferentA1 entity (fund, fund house, company, person, index; renamed schemes); A2 plan and option (Direct/Regular ×\times Growth/IDCW); A3 unit and scale“HDFC” read as HDFC Bank when HDFC’s fund house was meant; a Regular-plan NAV given for a Direct holding
B. MeasurementB1 metric (absolute, annualised, XIRR; asset change vs net flow); B2 horizon and as-of date; B3 mode (lump sum vs SIP); B4 gross or net basis (before or after exit load, transaction taxes and tax)A 3-year total return reported as a 3-year annualised return; a redemption credit quoted before tax
C. EvaluativeC1 comparator (total-return vs price index, category); C2 objective; C3 risk measure“Beat the Nifty” judged against the price index
D. ContextualD1 jurisdiction; D2 tax status and regime (including treaty relief); D3 market regime; D4 data and rule vintageA capital-gains rule superseded in July 2024 applied to a 2026 sale

Table 2. Fourteen dimensions along which a financial question can be underspecified.

Materiality and appropriate behaviour. A dimension is material when changing it moves the answer beyond the reporting tolerance; we score only material dimensions. The appropriate behaviour, in order: answer under the investor’s known values; otherwise state the basis used and name the material alternatives (for an unknown plan, both plans, labelled); ask only when no basis exists; and decline when the data cannot answer. Investor-specific dimensions take three steps: gather the investor’s facts, apply the rule to them, and use the rule version in force.

Forms of failure. Table 3 separates the failures we observe. Three are frame collapse: the answer commits to one value of a material open dimension, or attaches a measure to a figure, although nothing in the model’s context settled it, and does not say so. The fourth, headline prioritisation, is weaker: both figures are shown and labelled, and only their order comes from the layout. The studies score these forms differently, so they do not yield one failure rate.

FormWhat the answer doesExample from our studiesTested inScored by
(i) MislabelAttaches the user’s label to a figure that measures something elseA month’s change in assets reported as the net inflowStudy 1; judgesCodebook
(ii) Silent readingAnswers under another admissible reading without saying soA lump-sum return given to an investor who invests monthlyStudies 1 and 2Codebook; extraction
(iii) Headline prioritisationShows and labels both readings, but the layout decides which leadsThe first column’s plan headlinedStudy 2Code (order)
(iv) Omitted conditionLeaves out a decisive investor fact, or uses a superseded ruleAn NRI’s bank credit quoted before tax is withheldStudy 3Code, tool logs

Table 3. The forms of failure, where each is tested and how it is scored. Forms (i), (ii) and (iv) are frame collapse; (iii) is a related, weaker presentation effect.

Related Work

Ambiguity and clarification. Ambiguous questions admit several correct answers [17], and models rarely resolve them unprompted [12, 35]. Benchmarks test whether models ask for missing information [13, 20], and asking can be learned [1, 11]. Conditional QA asks which conditions apply [29], and time-sensitive QA whether knowledge is current [30]. Most financial benchmarks fully specify the question [3, 9]; RealFin removes premises from exam questions [5]. Closest to us, FinInteract [31] pairs default and intended readings of ambiguous financial questions and finds interaction, not financial knowledge, to be the bottleneck. Ambiguous database questions have reference answers for each interpretation [2, 22]. We use underspecification for questions, not for model pipelines [6]. We add computed references for each reading in a regulated domain, score the basis an answer states, and manipulate where the interpretation comes from, rather than only asking whether a model asks.

Context, order and tools. Irrelevant or counterfactual context misleads models [18, 27]. They favour options by position [19, 36] and are sensitive to table and prompt format [23, 28]. Agents over-rely on tools [32, 33] and choose them by their descriptions [8]. Models accept questionable presuppositions [10, 34] and defer to a user’s stated view [26]; a mislabel accepts the label the user’s wording presupposes. Models also hold knowledge they do not use [4, 7]. Our context differs: it is not misleading, but true under another interpretation.

Grounding and judges. Answer checks usually test whether an answer is supported by its evidence, through attribution metrics [16, 21] or LLM judges [37]. Evaluator verdicts on underspecified queries change once user context is supplied [15]. A frame-collapsed answer is supported by its evidence, so we test what else a judge needs, borrowing field descriptions from text-to-SQL [14].

Study Design

Each study tests one challenge by changing one thing the model sees (Table 4). Studies 1 and 2 leave out the investor; Study 3 adds the investor and the tax rule. Two further tests reuse Study 1’s answers: can a judge catch the failure, and can a prompt prevent it?

Study 1: ambiguous data. We wrote 22 questions in three wordings: precise, vague, and a vague paraphrase. Each came with no data, a thin record (one record whose field name does not say exactly what its figure measures) or labelled records (every candidate, each labelled). The data appeared as a lookup tool’s output under the instruction “If data is provided, use it”. Scheme figures are illustrative. We also ran each item with the question rewritten to name the field it needs. Because the three models are older, we replicated the thin-data test on GPT-6.1 Sol, Gemini 3.8 Flash and three small open-weight models (Llama 3.1 8B, Mistral Nemo 12B, Qwen3 8B).

Study 2: product variants. Eleven current models (Claude Opus 5.5, GPT-6.1 Sol, GPT-6 Astra, Gemini 3.8 Flash, Grok 4.7, Kimi K3, GLM-5.3, DeepSeek V4.1 Flash, Qwen3.5-122B, Llama 4 Maverick, gpt-oss-120b) answered questions on real AMFI NAV records for ten funds. They gave either a fund’s three-year return, with the plan open or stated, or the return of an investor “investing since 2021”, with the mode open or stated. We counterbalanced the order of the Direct and Regular columns, in a table or in a tool’s search ranking.

Study 3: the investor and the rules. An agent with tools for the investor’s profile, holdings, NAVs and scheme facts answered two redemption questions, each turning on one profile fact (Table 5). Each arm changes one element from its comparator: a mention (a tool description only), a demand (a tool argument or an online checklist requiring the fact), or rule supply (the current rule as a tool or prompt text). Each arm ran at least five times per case.

Study 1: ambiguous dataStudy 2: product variantsStudy 3: the investor and the rules
What changesNo data, a thin record or labelled records, returned by a lookup tool; precise or vague wordingPlan or mode open or stated; Direct or Regular column first (counterbalanced); table in the prompt or toolsWhether the agent’s interface mentions the decisive fact, demands it, or supplies the current rule
ModelsGPT-4.1, Gemini 2.5 Flash, Qwen3.8 27B; replicated on five moreEleven current modelsGemini 3.1 Pro and GPT-4.1, selected for failing; five further models as checks
Scale22 questions; 1,188 answers148 items; 3,256 answersTwo redemption cases; 350 planned runs, 80 check and final runs
OutcomeWrong answer; mislabelSilent on the open dimension; which plan leadsFetched the fact; figure within tolerance
Scored byAn AI coder with a fixed codebookCode, from two AI extractors’ readingsCode and tool logs
Read with careCodebook stricter than the definition; illustrative figuresRates over answers both extractors agree on; extractors are tested modelsTwo cases; models chosen for failing; mechanisms, not rates

Table 4. Design of the three studies. Each changes one thing that reaches the model’s context. Cohorts and denominators: Appendix B.3.

Checks and prevention. AI judges scored Study 1’s 162 thin-data answers, with and without the data, a neutral data dictionary and the user’s intended reading. A data-match check, a prompt that states what the question asks for and what each field measures before answering, ran on Study 1’s thin items with one open-weight model.

Measurement. Code or a fixed codebook decides correctness; models only extract or code what an answer did. References for Studies 2 and 3 are computed in code from AMFI NAV records and the Income-tax Act, 2025. Three limits apply throughout. One AI coder applied Study 1’s codebook, checked by blind recodes, a coder from another model family, a model-free text rule and, on 50 answers, a finance-trained human coder. Study 2’s two extractors (Claude Opus 5.5 and GPT-6.1 Sol) are also tested models, and its rates count only answers both agree on. Study 3 scores the figure, not its disclosure, and uses models chosen because they failed in exploratory runs. Appendices B.1–B.4 give tolerances, coding checks and statistics.

Results

The results follow the three challenges of Figure 1, with Study 3 split into a missing fact and a demanded fact. Sections 5.5 and 5.6 ask whether such answers can be caught or prevented. Appendix Figure 4 summarises every measurement.

Ambiguous data: one figure in context is adopted (Study 1)

Given one figure, models committed to it. On the same 14 questions and three wordings, the three Study 1 models gave 7 wrong answers of 126 with labelled data. When a lookup tool returned a thin record whose figure was not the quantity asked for, they gave 96 of 126. Only 1 of the 162 thin-data answers asked the user anything.

The commitment took two forms. A mislabel attaches the user’s label to the figure. Asked for the net inflow and given {"category": "Equity", "growth": 84955}, a month’s change in assets, GPT-4.1 replied that “the net inflow ... was Rs. 84,955 crore”: 2.9 times the actual net inflow of Rs. 29,329 crore. On precise wording, 26 of 42 thin answers carried the user’s label; none of the 42 labelled answers did. A silent reading is true under another interpretation but does not say so. AI coders disagree on how the 96 split between the two forms, but agree on whether an answer is wrong (kappa 0.70–0.95 across three coders). A finance-trained human coder who blind-recoded a sample of 50 answers agreed with the codebook on both (kappa 0.90 on wrong versus not; 33 of 34 on the form).

A mislabel can reverse a decision. All nine thin answers to one question said the large-cap category beat the Nifty 50, citing 7.6%; three called this the total-return index (TRI), SEBI’s required benchmark since 2018. But 7.6% was the price index’s return. The category’s 8.2% beat the price index, not the TRI’s 8.9%.

The pattern held across models, with one exception. Of five further models, blind-coded on the same questions, four were wrong on 99 to 113 of 126 answers. GPT-6.1 Sol was wrong on 38 and mostly flagged the gap or asked. Overall, seven of eight models were wrong in 69–90% of these answers (Table 9). Attaching the label partly reflects a reasonable presupposition that the tool returned what was asked [34].

Model (samples per item)RecordPrecise mislabels (of 42)Wrong, figure not the asked quantity (of 126)Flagged or asked, same answers (of 126)Asked (of 162)
Study 1: GPT-4.1, Gemini 2.5 Flash, Qwen3.8 27B (1)As given2696181
GPT-6.1 Sol (3)As given10388843
Gemini 3.8 Flash (3)As given2799121
Llama 3.1 8B (3)As given / accurate34 / 21109 / 9112 / 250 / 0
Mistral Nemo 12B (3)As given / accurate35 / 25113 / 1004 / 201 / 0
Qwen3 8B (3)As given / accurate32 / 21106 / 9216 / 213 / 4

Table 9. Thin-data behaviour across models, and with accurate records (fields renamed, entity and plan named). The 42 are 14 questions × 3 models (Study 1) or × 3 samples of one model. The three small models’ rows come from one arm-hidden recode of both arms (their first coding differs by at most 3); replications used larger output limits.

Accurate field names helped only where the record named a different thing. Thin and labelled records differ in naming and in whether the asked figure is present. A planned arm on the three small models separated the two: it kept the single record but renamed its fields to say what each measures (aum_change_cr instead of growth). Mislabels fell from 42 to 10 of 54 where the record now named a different company, scheme, plan or method. Where it named a close relative of the asked measure, such as asset change for net inflow, they barely moved (59 to 57 of 72): the user’s wording set the label. The overall effect is suggestive, not confirmatory (Appendix B.5).

Product variants: set out, but in layout order (Study 2)

Capability was not the problem: with plan and mode specified, Claude Opus 5.5 and GPT-6.1 Sol were right on every Study 2 item, and most models on 75–99%.

Given every candidate, labelled, most models set out the alternatives. Study 1’s vague questions with labelled data were flagged or asked in 82 of 108 answers. In Study 2, Claude Opus 5.5 and GPT-6.1 Sol named the open plan or mode in every agreed answer. Five models (Gemini 3.8 Flash, Grok 4.7, Qwen3.5-122B, gpt-oss-120b, Llama 4 Maverick) gave 140 of the 149 silent agreed answers, out of 713 agreed open-item answers that stated a figure (Figure 2a–Figure 2b; coding exceptions in Appendix B.6).

Figure 2. Study 2, per model, over agreed answers that stated a figure. (a, b) Silent selection (form ii, silent reading): answered without naming the open plan or mode, or asking. (c) Share of plan answers headlining the first column’s plan (order counterbalanced; 50% = no order effect); most of these answers show both plans. Open: NAV table in the prompt; filled: data fetched through tools; bars: Wilson 95% intervals; grey: fewer than 10 answers. *Gemini 3.8 Flash’s plan cases are side-by-side layouts that an audit coded as flagged.

The layout, however, decided which came first. Nothing in the question or records says which plan to list first. In 130 of 145 answers that showed both plans, the first-listed plan was the first column’s. Of the ten models with such answers, nine followed the first column more often and one tied (model-level sign test p = 0.004). Swapping the columns flipped the leading plan in 61 of 84 matched pairs (Figure 2c). Nothing was hidden: both figures were present and labelled. The layout set the priority; we did not test whether readers act on the first figure.

The investor and the rules: a missing fact is filled by a default (Study 3)

When a needed fact was missing from context, a default filled the gap. With residency removed from its profile, Gemini 3.1 Pro gave the resident (no-withholding) figure in 20 of 20 runs and never flagged the gap. In Study 2, all 81 agreed silent mode answers assumed a lump sum, although the investor wrote “I have been investing... since 2021” and the table showed no cash flows. GPT-4.1 and GPT-4.1-mini, with a knowledge cutoff before India’s July 2024 Budget, applied the capital-gains rule it superseded. These defaults are consistent with model priors.

The omission did not transfer to frontier agents. In final planned runs, GPT-6.1 Sol and Gemini 3.8 Flash fetched the fact unprompted and answered both cases correctly in all 20 runs. Gemini 3.5 Flash also gathered it unaided (10 of 10), and GPT-5.5 showed no gap. Study 3 thus shows a mechanism in some models, not a general rate.

A demanded fact is fetched, but not always used (Study 3)

A demand, unlike a mention, made a failing agent fetch the fact. With a demand, as a tool argument or a one-line checklist, Gemini 3.1 Pro fetched the decisive fact in 10 of 10 runs. With only a mention in a tool description, it did so in 1 of 10 (one-sided Fisher p=6.0×10−5p = 6.0 \times10^{-5}; Figure 3a). When the demand named the decisive fact, it answered both cases correctly in every run. A demand for unrelated facts got the profile fetched, but the non-resident case right in at most 1 of 5.

Figure 3. Study 3 arms. (a) Gemini 3.1 Pro: share of runs that fetched the decisive fact. (b) GPT-4.1: share of runs with the correct figure. Filled circles: a demand or fix; open circles: a mention, no demand or a stale rule; grey squares: no fix; grey circle: field-named tools without a demand. Lines: Wilson 95% intervals; five runs per case, two cases per arm unless a case is named.

Fetching did not ensure use. Gemini 3.1 Pro fetched the investor’s NRI status in 15 of 15 runs, yet repeated a compute tool’s resident-based figure in 14. GPT-4.1 fetched the fact under a checklist in 10 of 10 runs and was correct in none (Figure 3b), for a different reason in each case. In the non-resident case, knowledge was not the gap: GPT-4.1 stated the 30% withholding rate when asked, yet quoted the bank credit before tax in 10 of 10 agent runs. It applied the withholding only when the question explicitly asked for the amount after tax (10 of 10). In the booked-gains case, it consistently produced the superseded rule. Supplying the current rule as prompt text fixed this (5 of 5 against 0 of 5); a tool computing tax under the superseded rule was relayed in 5 of 5.

Can a check see it?

A frame-collapsed answer agrees with its own data, so a judge that checks only support can accept it, as the answering model did. We gave two judges Study 1’s 162 thin answers and varied what they saw (Table 6).

Told readingSeesMislabels named (of 44)Wrong rated correct (of 96)Acceptable rejected (of 61)
YesQuestion141815
Yes+ data174315
Yes+ data + dictionary40023
No+ data235315
No+ data + dictionary404217

Table 6. GPT-6.1 Sol as a judge of Study 1’s 162 thin answers: 44 mislabels, 96 wrong answers (26 precise, 70 vague) and 61 acceptable (right, flagged or asked) answers. Both judges, all arms and tests: Appendix § 10.§ 8.

The data alone did not help; the data’s meaning did. Told the intended reading and shown the data, GPT-6.1 Sol named 17 of 44 mislabels. With a neutral data-dictionary entry for each field, written from the source record without reference to the question, it named 40 of 44 (9 of 13 questions improved, none worsened; sign p=0.004p = 0.004). False alarms on acceptable answers rose from 15 to 23 of 61.

The gain held without the intended reading. A deployed judge does not know what the user meant, so planned arms removed the reading and reworded the rubric to ‘answers the question as asked’. With the dictionary, Sol still named 40 of 44 mislabels, against 23 with the data alone, with 17 false alarms of 61.

Silent readings, however, passed. In that condition Sol passed 42 of the 52 silent readings of vague questions, while rating every mislabel wrong. A silent reading does answer a vague question as asked; it lacks only a stated interpretation. The dictionary catches mislabels (i). Catching silent readings (ii) would need a rubric that requires a stated interpretation, or a reference for each reading; we tested neither. As a release gate, the dictionary judge would also reject about a third of acceptable answers. The small open-weight judge behaved alike (Appendix § 10.§ 8).

Can it be prevented, and at what cost?

§ 5.4 showed that, for the agents selected for failing, a demand repaired fetching and a supplied current rule repaired the rule version. On the answer side, we tested a data-match check on Study 1’s thin items with one open-weight model (gpt-oss-20b).

The check cut wrong answers, mostly by asking. Wrong answers fell from 38 to 9 of 53 (14 of 18 questions improved, none worsened), and appropriate behaviour rose from 12 to 29. But questions back to the user rose from 1 to 33 of 53, and plainly correct answers fell from 11 to 0. On labelled data, where no question was needed, it still asked in 21 of 53. If an unnecessary question costs a fraction of a wrong answer, the check beats both doing nothing and always asking only when that fraction is between a half and about one (Appendix § 10.9). It needs a rule that lets the model answer when the data match.

Cheaper interventions showed no effect. Mentioning the facts in a tool description did not make the failing agent fetch them (1 of 10 against 2 of 10 at baseline), though ten runs per arm cannot exclude a moderate effect. Rewriting Study 1’s questions to name the needed field left wrong answers unchanged (131 to 130 of 558 paired answers). Removing the instruction ‘If data is provided, use it’ did not reduce mislabels.

Discussion

What the studies show together

One description fits the three studies: the interpretation a model adopts tracks what reaches its context, and defaults fill what the context leaves open. We formed it after the studies. A single candidate is committed to; labelled candidates are set out in layout order; a missing fact is filled by a default; a demanded fact is fetched, but not always used.

Three manipulations could have contradicted it and did not. Labelled data removed mislabels (64 of 66 precise answers right). Swapping the columns flipped the leading plan (61 of 84 pairs). A demand, unlike a mention, made a failing agent fetch the fact (10 of 10 against 1 of 10).

Three limits bound it. First, context is not sufficient: a retrieved fact was not always used, and on close measures the user’s wording outweighed an accurate record. Second, it depends on the model. GPT-6.1 Sol still mislabelled 10 of 42 precise answers; Claude Opus 5.5 was tested only in Study 2; and the agent omissions did not transfer to GPT-6.1 Sol or Gemini 3.8 Flash. In the most careful current models, only layout order and, for judges, silent readings persisted. Third, when a model chooses its own context, as the frontier agents did, the description predicts nothing in advance.

Two models were tested in all three settings. GPT-6.1 Sol mostly flagged or asked in Studies 1 and 2 and answered every Study 3 run correctly. Gemini 3.8 Flash mislabelled (27 of 42 precise answers) and gave silent readings on most tabled mode questions (8 of 13), yet fetched the missing fact in every Study 3 run. This fits the description, but because the studies also differ in task, format and scoring, it does not show that context is the cause.

Controls for a financial assistant

The studies point to one principle: settle the material dimensions against the question, the data and the investor’s records before answering, and label the answer with the interpretation used. Table 7 turns this into controls, separating tested from proposed. The tested parts were not tested together.

#ControlFormEvidence here
1Take each decisive investor fact as a required, typed input from the records(iv)Made both failing agents fetch it (GPT-4.1 via a checklist); fixed one; frontier agents did not need it
2Supply the rule in force from a dated source; compute tax in code(iv)Current rule fixed GPT-4.1 (5 of 5 vs 0 of 5); a stale rule tool was relayed
3State what each data field measures; check it against the question(i)Helped where the record named a different entity (exploratory); prompt check over-asked
4Evaluate with the data’s meaning; add a reference for each readingJudgesDictionary tested; per-reading references untested
5Label every figure with the interpretation usedAllProposed
6Order alternatives by the user’s own holding(iii), (ii)Proposed; order effect observed
7Keep servicing separate from adviceAdviceProposed
8Log interpretation, facts and rule versionAuditProposed

Table 7. Controls, the failure each addresses, and the evidence here. Rows 1–4 have evidence on few models and cases; rows 5–8 are proposed and untested. Full table with Indian implementations: Appendix § 10.11.

By money at stake, mislabels (a verdict reversed, a flow overstated 2.9-fold) and omitted investor conditions (a credit overstated by Rs. 16,484 on one redemption) matter most in our cases; we did not test headline prioritisation on readers. SEBI makes regulated firms responsible for their AI tools’ outputs [25], and execution-only platforms may not give advice [24]. The controls apply in any jurisdiction but were tested only in India; other markets have analogues, such as the UK’s shared capital-gains exemption. Appendix § 10.11 gives Indian implementations and the full evidence.

Limitations

Coding. AI models did all coding (Study 1) and extraction (Study 2); a finance-trained colleague who is not an author checked a sample. On 50 answers enriched for wrong ones, the human, a model-free rule, blind coders and a coder from another model family agree on whether an answer is wrong (kappa 0.70–0.95; the human 0.90). The other-family coder recoded 10 of 15 silent readings as mislabels, but the human kept all 15, which supports the split between forms (i) and (ii), though on one coder and 50 answers. Study 2’s extractors are also tested models.

Samples. Study 1’s figures are illustrative and its main models older; the replications bound this. The accurate-record arm ran on three small models. Study 3’s models were selected for failing, and its arms ran five times on two cases; its tolerance admits an omitted cess. The judge arms rest on 162 answers.

Scope. One jurisdiction and product family. We did not measure how often real users leave each dimension open, so nothing here estimates prevalence. Appendix § 10.12 discusses what extending the studies beyond India would show, and lists further limits.

Conclusion

A number can be present in an assistant’s evidence, correct under some reading, and still answer a question the user did not ask. In our studies, models often neither established nor stated the interpretation they adopted; it came from what was nearest: a field name, a column order, a default about mode, residency or tax. Because such answers trace to their data, a check of support passes them, and a judge recognised most mislabels only when told what each field measures. Benchmarks should therefore ship a computed reference for every admissible reading and score whether a model established the reading, not just the number. Judges should be tested without the intended reading, which a deployed judge never has. The next test is the proposed combination of required investor facts, dated rules and labelled interpretations, evaluated together on real user queries with human coding; it would also estimate how often frame collapse occurs. Tax, insurance, banking and health questions share this structure of regulated variants and personal conditions, and are natural next domains.

A Glossary of Indian Terms

Table 8 defines the Indian mutual-fund and tax terms used in the paper. Three words name three different things: a plan is Direct or Regular, an option is Growth or IDCW, and a mode is a lump sum or an SIP.

TermMeaning
Plan: Direct, RegularEvery scheme is offered in two plans that hold the same portfolio and differ only in where the investor bought it. A Direct plan is bought without a distributor (from the fund house, an execution-only platform or an adviser). A Regular plan is bought through a distributor, and its expense ratio includes the distributor’s commission. The two plans therefore have different NAVs and returns.
Option: Growth, IDCWWithin each plan, the Growth option pays nothing out, so gains stay in the NAV. The income distribution cum capital withdrawal (IDCW) option pays out distributions, which can include the investor’s own capital; its NAV falls with each payout.
Mode: lump sum, SIPHow the money went in: one single investment, or a systematic investment plan (SIP) that invests a fixed amount every month.
NAVNet asset value per unit: the daily price at which units are bought and redeemed.
Expense ratio; exit loadThe annual fee taken from the fund’s assets (already reflected in the NAV); a fee charged on redemption within a set period.
Annualised return; XIRRAnnualised return: the return per year between two dates for a single investment. XIRR: the annualised return computed over dated cash flows, as in an SIP or IDCW payouts (an internal rate of return).
AUM; net inflowAssets under management. Net inflow is money invested minus money redeemed; a change in AUM also includes market movements, so the two differ.
Price index; TRIA price index tracks prices only; a total-return index (TRI) also reinvests dividends. SEBI has required funds to be benchmarked against a TRI since 2018.
Nifty 50An index of fifty large Indian companies.
Equity-oriented fund; debt fundTax classes. Long-term gains (held over 12 months) on listed equity and equity-oriented fund units are taxed at 12.5% above an annual exemption. Gains on debt funds (since April 2025, funds with more than 65% in debt and money-market instruments) bought on or after 1 April 2023 are taxed at the investor’s slab rate, whatever the holding period.
Slab rateThe investor’s income-tax bracket rate.
Long-term gains exemptionRs. 1.25 lakh per tax year (Rs. 1 lakh before 23 July 2024), shared across all long-term gains on listed equity and equity-oriented fund units in that year.
NRINon-resident Indian: an investor not resident in India for tax purposes.
TDS; bank creditTax deducted at source: tax withheld before redemption proceeds reach the investor’s bank. For an NRI’s debt-fund gain, 30% plus cess. The bank credit is the amount actually paid into the bank, after any TDS.
CessA 4% levy charged on top of the tax.
PAN; folioPermanent Account Number, India’s tax identifier; an investor’s account with a fund house.
Tax yearThe April–March year; tax year 2026–27 runs from April 2026 to March 2027.
July 2024 Budget; Income-tax Act, 2025The 2024 Budget raised the long-term rate on equity gains from 10% to 12.5% and the exemption from Rs. 1 lakh to Rs. 1.25 lakh, from 23 July 2024. The Income-tax Act, 2025 is the law in force from 1 April 2026.
Lakh; croreOne lakh is 100,000 and one crore is 10 million. Indian digit grouping writes 536,350 as 5,36,350.
SEBI; AMFIThe Securities and Exchange Board of India, the market regulator; the Association of Mutual Funds in India, the industry body whose records give each fund’s NAVs.
Execution-only platformA platform that executes transactions in Direct plans and may not give advice.

Table 8. Glossary of Indian mutual-fund and tax terms.

Detailed Reporting

This appendix reports the counts, denominators, coding checks and statistics behind the main text, which states only what the argument needs. Nothing here changes a main-text claim.

Terms, tolerances and conventions

Terms. A reference answer (gold) is computed in code. Precise wording fully specifies the question; there the open element is what the data field measures. A mislabel attaches the user’s specific label to a figure that measures something else. A silent reading answers under another interpretation without saying so and without a false label (form ii; the codebook’s ‘wrong reading’); wrong means a mislabel or a silent reading. A flag names the unresolved dimension or the assumption used. Agreed answers are those on which both Study 2 extractors agree; answers they disagree on are left out of agreed rates.

Departures from the definition. Study 1’s codebook is stricter than the definition: it counts an answer that names the figure’s true measure without naming the alternative as wrong, so relative to the definition its counts lean high (a blind recode confirmed every agreed wrong code but cannot show how many wrong answers were missed). On wrong versus not, a coder from another model family, gpt-oss-20b, agreed at kappa 0.82, two blind Claude coders at 0.70 and 0.95, and a human coder at 0.90. Study 2 counts labelled side-by-side answers as flagged; five unworded Gemini 3.8 Flash layouts, which an audit codes as flagged, are kept as silent and marked in Figure 2. Study 3 scores the figure, not the disclosure.

Tolerances. The reporting tolerance is 5% of the reference (at least 0.5 points) in Study 2 and 0.5% (credits) or 8% (tax) in Study 3, with floors of Rs. 100 and Rs. 200. The Study 3 tolerances admit an omitted 4% cess but not a missed withholding (3.1% of the credit), an unshared exemption (30% below the tax reference) or the superseded rule (10% below). In Study 2 the other plan’s figure lies outside the tolerance for all ten funds, but only narrowly for HDFC Flexi Cap Direct (gap 0.75, tolerance 0.74 points), so plan is scored from the answer’s stated label rather than its value; a 0.25-point tolerance would change 28 of 1,591 specified-item scores.

Table 1. Each return period starts at the first NAV published on or after its start date; fund factsheets instead use the last NAV before a non-business start date. The IDCW figure is the payout investor’s money-weighted outcome: it exceeds Growth, yet that investor receives less in payouts plus final holding value (Rs. 144.1 against Rs. 147.9 per Rs. 100 invested, before the slab-rate tax on payouts). Not a valid alternative: reading the Regular IDCW NAV as though it were the Growth NAV gives 3.62%, a misreading. Direct plans are bought without a distributor (from the fund house, an execution-only platform or an adviser); the NAV records are published through the Association of Mutual Funds in India (AMFI), the industry body. Across ten funds, the Direct-Regular three-year difference on Rs. 10 lakh is Rs. 21,800–48,300.

Currency and Table 5. One lakh is 100,000 and one crore 10 million; Indian digit grouping writes 536,350 as 5,36,350. The NRI overstatement of Rs. 16,484 is about USD 190 at Rs. 87 per dollar. Figures are rounded to the rupee separately (the overstatement is computed before rounding); stamp duty on purchase (about Rs. 25) is ignored.

Study 3 caseInvestor and holdingRule that decides itReference answerIf the fact is missed
Non-resident withholdingNRI; Rs. 5 lakh in a debt fund (HDFC Corporate Bond) bought January 2025; gain Rs. 52,835Debt funds (since April 2025, funds with more than 65% in debt and money-market instruments) bought on or after 1 April 2023 are taxed at slab rate, whatever the holding period; for an NRI, 30% plus 4% cess is withheld at sourceBank credit Rs. 5,36,350Rs. 5,52,835 (overstated by Rs. 16,484)
Booked gainsResident; Rs. 5 lakh in an equity fund (Parag Parikh Flexi Cap) bought June 2023; Rs. 60,000 of long-term gains on listed shares already booked this tax yearLong-term gains on listed equity and equity-oriented fund units (held over 12 months) taxed at 12.5% plus cess above a Rs. 1.25 lakh annual exemption shared with other such gains (before 23 July 2024: 10% above Rs. 1 lakh)Gain Rs. 2,66,118; tax Rs. 26,145 (tax year 2026-27)Rs. 18,345 if the exemption is not shared; Rs. 23,516 under the superseded rule

Table 5. The two Study 3 cases. Neither question mentions tax status or other gains. Both redeem at the 30 September 2026 NAV and assume no surcharge, a PAN on file, no treaty relief and no exit load. The NRI reference is the bank credit after TDS, not final tax; the resident reference is final tax, assuming other income above the basic exemption and no carried-forward losses. Rounding conventions: Appendix B.1.

Evidence summary

Figure 4 collects the main measurements behind § 5, grouped by what reached the model’s context.

Figure 4. Evidence summary: what reached the model’s context, and what it did. Each block is one situation; each line is one measurement with its count and 95% interval. Red: failures (lower is better); green: appropriate behaviour (higher is better); open circles: share following the layout (dashed line at 50%: no order effect). [M] a manipulation; [C] a comparison across models or answers; [S] runs of a model chosen because it failed in exploratory runs. Lines are comparable within a block, not across blocks: the studies use different models and scoring rules (§ 4). Populations and interval methods: Appendix § 10.§ 3.

Cohorts and denominators

Study 1. Each run is 2222 questions ×3\times3 wordings ×3\times3 data conditions ×3\times3 models = 594 answers; with the explicit-field restatement run, 1,188 in all. The denominator ladder for the original arm: 594 generated → 558 included, after excluding the 36 thin-data answers to four questions whose thin label was itself wrong → 162 with thin data (18 questions) → 126 whose figure was not the asked quantity (14 questions) → 42 of those on precise wording. All 96 wrong thin answers fall among the 126. Figure 4’s Study 1 vague lines use the 18 questions with a usable thin record. Original and restatement arms are reported separately.

Study 2. 148 items ×11\times11 models ×\times table and tools = 3,256 answers. Silence rates condition on extractor agreement and on the answer stating a figure (713 agreed open-item answers stated one). Specified-item accuracy is over specified items, not over the whole 148-item inventory.

Judges. The 162 thin answers partition into 96 codebook-wrong (26 precise, 70 vague; 44 of them mislabeled, 26 precise and 18 vague), 61 right, flagged or asked, and 5 coded as date confusion.

Data-match. Headline counts (38 to 9 of 53 wrong) use the first coder on the 53 thin items. The cost analysis uses a blind recode pooling thin and labelled items, 107 answers per policy.

Study 3. 350 planned runs on Gemini 3.1 Pro and GPT-4.1, with GPT-4.1-mini and GPT-5.5 as checks; separately, 40 check runs on Gemini 3.5 Flash and 40 final planned runs on GPT-6.1 Sol and Gemini 3.8 Flash. Study 3’s exploratory runs are about 1,100 earlier agent answers from four models.

Figure 4. Study 1 rows are GPT-4.1, Gemini 2.5 Flash and Qwen3.8 27B; Study 2 rows the eleven current models. Study 2 tools lines count agreed answers that stated a figure; the three models that never fetched both plans give 38 of the 44 silent answers on the one-plan line, which is why its interval is wide. The [S] line pools Gemini 3.1 Pro’s baseline (fetched the fact 2 of 10) and mention (1 of 10) arms as 3 of 20. Intervals are clustered by question (Study 1) or model (Study 2; the lump-sum line uses Wilson, its cluster interval being degenerate), else Wilson.

Coding and statistical methods

In Study 1 one AI coder (Claude) coded every answer, aided by a model-free text rule that flags a restated thin value with no correcting label nearby. A human recoded a 50-answer sample blind; the full codes matched in 43 of 50 (kappa 0.81). Study 2’s two extractors (Claude Opus 5.5, GPT-6.1 Sol) read each answer and record its headline figure, plan, method, mode and any flag or question. A model-based consistency check of 100 agreed answers found no misread headline figure, series, method or mode, and no sign of an extractor favouring its own model. Because both are also tested models, a model outside the panel (gpt-oss-20b) re-abstracted all 160 of their open answers: the 23 it marked as not flagged all showed both plans, labelled, and none committed silently. gpt-oss-20b is also the data-match model and the small judge.

Intervals are 95% bootstrap with questions as clusters for Study 1, clustered by model where Study 2 models differ (with 7–11 model clusters these are approximate and likely too narrow), and Wilson for counts. Tests are exact (one-sided Fisher for planned directional contrasts, McNemar or sign), Holm-corrected within Study 3’s planned contrasts and Study 2’s per-model order tests; unplanned contrasts are marked exploratory. McNemar tests treat answers as independent; question-level sign tests are the conservative reading and are given alongside. Study 3’s Fisher tests treat repeated runs of one prompt as independent draws and pool its two cases; they show that an arm changed a model’s behaviour on these cases, not that the effect generalises to other cases.

Study 1: replications, accurate records and further results

Table 9 gives thin-data behaviour across models. On the 126 answers whose figure was not the asked quantity, the seven models other than GPT-6.1 Sol were wrong in 69–90% (Gemini 3.8 Flash 99, the small models 106–113 as given); GPT-6.1 Sol was wrong in 30%.

Accurate-record arm. The arm was planned; departing from the plan, the records also named the entity and plan explicitly. The planned test, a question-level sign test, was not significant for precise mislabels (101 to 67 of 126; 8 questions fewer, 2 more; p=0.11p = 0.11) and was for wrong answers (328 to 283 of 378; p=0.039p = 0.039); a Holm correction added after the plan leaves neither significant. The unadjusted question-clustered intervals (-48 to -8 and -23 to -2 points; a percentile bootstrap over 14 questions, which tends to be too narrow with so few clusters) exclude zero, so we read the arm as suggestive, not confirmatory. The model-free rule, which cannot see the arm, moved the same way (mislabel candidates 105 to 43 of 378; sign test p=0.002p = 0.002). The split by record type is exploratory; on the asset-change example mislabels went from 6 to 9 of 9. On vague questions the removed mislabels became answers that named the figure’s true measure without the alternative, which the codebook counts as wrong; coders saw each record, so this shift is partly by definition. Removing the instruction ‘If data is provided, use it’ did not reduce mislabels.

Explicit-field restatement. Wrong answers 131 to 130 of 558 paired answers (90% interval -1.7 to +1.4 points); on thin data 96 to 94 of 162, where an effect of about 5 points cannot be excluded.

Self-chosen context (Study 2, tools). When models chose what entered their context, the evidence is weaker: answers that fetched one plan series were silent on the plan in 44 of 65 against 4 of 80 when both were fetched, but this pooled contrast mostly reflects which models never fetched both (38 of their 42 answers). Within the four models that fetched one plan in some answers and both in others, silence was 6 of 23 against 1 of 43, five of the six from GLM-5.3. In exploratory agent runs a tool that reported only the lump-sum annualised return misled 35 of 72 SIP answers.

Study 2: further results and coding exceptions

Accuracy on specified items. gpt-oss-120b was right on 58% with tools; Llama 4 Maverick on 69% with the table and 13% with tools.

Silence. The extractors disagreed on some of Claude Opus 5.5’s and GPT-6.1 Sol’s tool answers; counting every disagreement as silent gives upper bounds of 10% and 30%. If Gemini 3.8 Flash’s five unworded side-by-side layouts are coded as flagged, the five models give 135 of 144 silent agreed answers. Nine of the 149 fall on Mirae’s mode items, where mode is immaterial, so they are not frame collapse.

Order. Kimi K3 followed the first column only partly, as it also favoured Direct. Among the seven models that followed the first column, 49 of 50 pairs flipped (a subset selected on the outcome); a pair flips when the headline follows the first column in both orders. In one model the layout also set a committed answer: Grok 4.7 called Regular “the standard plan” in 8 of 10 answers that listed it first and none that listed it second (exploratory).

Lump-sum default. The 81 agreed silent mode answers that assumed a lump sum came from seven models, most from Llama 4 Maverick and Qwen3.5-122B; 72 of 72 on material items; over all answers, including four tool calls printed as text, 83 of 85. Three models favoured the Direct plan whatever the column order (right only when the investor’s records say Direct).

Study 3: further results

Defaults. With residency removed, Gemini 3.1 Pro filled a residency field with ‘resident’ in all 15 runs that read the profile. The compute tool that produced the resident-based figure did not, by design, take residency.

Gathering arms. Field-named profile tools without a demand fetched the fact in 9 of 10 runs (correct in 8). With a demand for the decisive fact, Gemini 3.1 Pro answered correctly whether its arguments were named (15 of 15 per case) or uninformatively named (5 of 5 per case). With the fact missing, it filled a required residency field with ‘resident’.

GPT-4.1. It ignored tool arguments (1 of 10). With the facts preloaded it stayed wrong on the original question (0 of 5). It applied the 30% withholding only when asked for the amount after tax, in the question itself or in a follow-up turn (10 of 10, five each; one-sided Fisher p=3.3×10−4p = 3.3 \times10^{-4}). Four of those ten were within Rs. 120 of the reference and six omitted the 4% cess (Rs. 580–725 high, about 0.1% of the credit, against Rs. 16,484 when the withholding is missed), so under strict scoring, which counts a missing cess as wrong, the contrast is 4 of 10 against 0 of 5 (p=0.15p = 0.15). Because this rewords the user’s question rather than adding a platform control, we read it as evidence that GPT-4.1 had the rate but did not apply it unprompted, not as a deployable fix. When asked directly, GPT-4.1 justified the 30% rate with a superseded 36-month holding test.

Rule version. The current rule as prompt text gave GPT-4.1 5 of 5 against 0 of 5 with the facts alone (p=0.004p = 0.004), and a tool computing the tax under it 5 of 5. The facts-alone arm was appended to the plan after 140 earlier runs but before its own, and neither plan specified this Fisher contrast. The plans set threshold readings; a Holm correction over nine Fisher contrasts, added later, keeps six, including both demand contrasts and the rule as text for GPT-4.1, and drops three. In exploratory runs, required plan, method and horizon fields gave no violation in 144 runs.

Frontier checks. GPT-6.1 Sol and Gemini 3.8 Flash were also correct under strict scoring, and in all 20 runs with a demand. Zero failures in 20 baseline runs, 10 per model, gives 95% Wilson upper limits on the per-run failure probability of 16% (pooled, treating runs as independent and the two models as sharing one rate) and 28% per model; it says nothing about other cases.

Judges: all arms and tests

Table 10 gives both judges in every arm. The question-only arm was planned together with an arm whose notes sometimes stated the verdict (40 of 44 named for Sol, 28 for gpt-oss-20b); the neutral data and dictionary arms replaced it after a first review, pre-specified before their own runs; the arms without the reading were planned and use the reworded rubric.

JudgeTold the intended readingSeesMislabels named (of 44)Wrong rated correct: precise (of 26)Wrong rated correct: vague (of 70)Acceptable rejected (of 61)
GPT-6.1 SolYesQuestion1414415
GPT-6.1 SolYes+ data17152815
GPT-6.1 SolYes+ data + dictionary400023
GPT-6.1 SolNo+ data2354815
GPT-6.1 SolNo+ data + dictionary4004217
gpt-oss-20bYesQuestion517937
gpt-oss-20bYes+ data3162123
gpt-oss-20bYes+ data + dictionary2441221
gpt-oss-20bNo+ data785311
gpt-oss-20bNo+ data + dictionary2434310

Table 10. Judges on the 162 Study 1 thin answers. Mislabels named: codebook mislabels the judge labelled as such (the planned outcome); 44 mislabels cover all wordings (26 precise, 18 vague). Wrong rated correct: codebook-wrong answers the judge passed (96 = 26 precise + 70 vague). Acceptable rejected: right, flagged or asked answers called wrong.

Tests. With the reading, dictionary against data alone for Sol: answer-level McNemar p=1.5×10−6p = 1.5 \times10^{-6}; 9 of 13 questions improved, none worsened, sign p=0.004p = 0.004; Sol rated all 44 mislabels wrong. Data alone against question alone: 17 against 14 of 44 (p=0.55p = 0.55). Without the reading, dictionary against data alone: p=7.6×10−5p = 7.6 \times10^{-5}; 6 questions improved, none worsened, sign p=0.03p = 0.03; it passed no wrong precise answer. The small judge passed 43 of 70 wrong vague answers without the reading.

Data alone (post hoc). The data alone can hurt. Shown the data, Sol passed 28 of 70 wrong answers to vague questions against 4 without it (precise: 15 against 14). Over all 96 wrong answers, 7 questions moved towards passing and 3 away (sign p=0.34p = 0.34); on vague answers alone, 6 and none (sign p=0.03p = 0.03; post hoc). A simpler check, that every quoted percentage appears in the tool output, passed 31 of 35 misleading SIP answers in exploratory agent runs.

Data-match check

On labelled data the check left wrong answers at a similar level (8 to 9) while asking in 21 of 53. On the cost score (blind recode; thin and labelled items pooled, 107 answers per policy) it beats both doing nothing and always asking only when an unnecessary question costs between a half and about one wrong answer; at a half it ties always asking (33 against 33) and at one it beats doing nothing by a single point (46 against 47). Always asking would have scored 41 of 53 on appropriate behaviour.

Claims, evidence and status

Table 11 lists each main claim with its evidence, scoring, status and scope.

Controls in full

Table 12 expands Table 7. Where the platform owns the data, the first control is a data contract; a model-side check is the backstop for third-party feeds. The proposed layered design loads the facts the rule takes as inputs (row 1), supplies the rule in force with its legal basis (row 2) and checks the data against the question (row 3).

Table 12: Controls for a financial assistant: jurisdiction-agnostic control, failure addressed, an Indian example, and the evidence our studies give for it. Rows 1 and 2 have planned evidence for the failing models only; row 4 has planned evidence for the data dictionary, not for reference answers; row 3 has planned evidence only for a prompt-level check that over-asked, and exploratory evidence for metadata; rows 5–8 are untested.

#Control (any jurisdiction)Failure addressedExample implementation in IndiaEvidence here
1Make each decisive fact a required, typed input taken from the investor’s records (or asked of the investor where only they hold it, such as gains booked elsewhere), enforced in code; never accept a model-supplied default(iv)Consolidated account statements; tax status on each folioFor Gemini 3.1 Pro a demand for the decisive fact fetched it and gave correct answers (two cases); GPT-4.1 fetched it under a checklist but stayed wrong unless the current rule was supplied or the user explicitly asked for the bank credit after TDS (a diagnostic, not a control); a model filled a required field with ‘resident’ when the fact was missing; frontier models gathered it unasked (20 of 20), so for them the control is a safeguard against regression (untested), not a fix shown to be needed
2Supply rules from a dated source and compute the tax in code(iv)A rulebook re-tested when the Budget, tax notifications or SEBI circulars changeRule as text 5 of 5 vs 0 of 5 on the version case; a tool with the superseded rule was relayed 5 of 5 (one model); legal-basis text not significant after Holm
3Give data fields metadata saying what they measure; check them against the question, with a rule that answers when they match (the check tested here lacked one and over-asked)(i)Metadata separating total-return from price indices, net flows from asset changesAccurate records cut mislabels where they named a different entity (exploratory, 3 small models), not for close measures; the data-match check (a prompt, not metadata) cut wrong answers 38 to 9 but asked back 33 times and lost all 11 plainly correct answers
4Evaluate with the data’s meaning (tested); add code-checked reference answers for each reading (untested as a judge input)Judges passing wrong answersPre-release tests of extracted figures against computed valuesAn accurate, author-written dictionary let a strong judge name 40 of 44 mislabels (i) with or without the intended reading (17 to 23 of 61 false alarms, against 15 with the data alone); without the reading, silent readings (ii) passed (42 of 52); a check that quoted percentages appear in the tool output passed 31 of 35 misleading answers (exploratory)
5Label every figure with the interpretation used: variant, measure, period, as-of date, fees, tax basisAll fourState plan, option, return method and pre- or post-taxNot tested (the central principle)
6Order alternatives by the user’s own holding, not by layout or search rank(iii), (ii)The investor’s plan; else a fixed, disclosed order, never the plan paying the seller moreNot tested; effect observed (Study 2)
7Keep servicing separate from adviceAdvice in servicingExecution-only platforms route suitability questions to a registered adviserNot tested; outside the SIP question, 11 of 144 suitability answers recommended outright, 33 conditionally
8Log the interpretation, facts and rule version of each answerAudit trailA per-answer record for the responsibility SEBI places on firmsNot tested

Table 12.

Further limitations

One market. All three studies use Indian mutual funds, so they cannot show how the failures vary across markets. Models may know some markets’ products and rules better than others, for example markets that are better represented in their training data, and less familiar markets may produce more defaults and outdated rules. Repeating the studies in other markets, from widely documented ones such as the US and UK to smaller markets with less public documentation of their products and tax rules, would show how much performance depends on the market. The taxonomy would need further values there, such as real versus nominal returns and tax wrappers.

Other limits. 53 of Qwen3.8 27B’s Study 1 answers may have been truncated (none a thin-data wrong answer). Study 3’s rule tool cited the repealed Act’s section (figures unaffected). Two judges were tested without the intended reading, with a reworded rubric. A planned resolution-layer test on Study 2 was stopped after 6 of 148 items because its pre-step assumed one default plan, whereas the right plan is the one in the investor’s records.

ClaimEvidence and designScoringStatusScope
F1. When a lookup tool returns one ambiguously named figure in reply to the question, models attach the user’s label to it or silently answer another reading: (i), (ii)Study 1: thin vs labelled data; accurate-record arm; five replicationsCodebook; blind, model-free and human checksDesign fixed in advance; codebook after collection; accurate-record arm planned (entity naming a deviation)7 of 8 models (GPT-6.1 Sol mostly flagged or asked; accurate-record arm: 3 small open-weight); 14 questions whose thin figure was not the asked quantity; illustrative figures
F2. With plan or mode open, several models answer without naming it (ii)Study 2: open vs specified items (comparison across 11 models)Code, two extractorsPlannedTwo dimensions, 10 funds
F3. Layout sets which plan leads (iii)Study 2: counterbalanced column order (manipulation)CodePlanned; committed-answer effect exploratory9 of 10 models; committed answers in one
F4. Two agents selected for failing omitted conditions they could fetch (iv); a demand for the decisive fact made Gemini 3.1 Pro answer correctly; GPT-4.1 needed the current rule, or an explicit request for the bank credit after TDSStudy 3: one-element arms (manipulation)Code and tool logsPre-specified threshold readings; six of nine Fisher contrasts survive a Holm correction added laterTwo cases; models selected for failing; did not transfer to two frontier models (40 runs)
F5. A judge named most mislabels when given an accurate data dictionary, with or without the intended reading; without the reading it passed silent readings of vague questionsJudge arms on Study 1 answers (manipulation)Codebook as referencePlanned outcome; the data arms with the reading added after a first reviewTwo judges, each with and without the reading (rubric reworded without it); 162 answers
F6. Checking data against the question cuts errors, mostly by askingData-match check vs baseline (paired)Codebook; blind recodePlanned; scoring deviatedOne open-weight model; always asking scores higher

Table 11. Claims, evidence and status. A manipulation varies one element with everything else fixed; a comparison contrasts models or items.

References

  1. [1]C. Andukuri, J.-P. Fränken, T. Gerstenberg, and N. D. Goodman. 2024. STaR-GATE: Teaching Language Models to Ask Clarifying Questions. In COLM. arXiv:2403.19154arXiv
  2. [2]A. Bhaskar, T. Tomar, A. Sathe, and S. Sarawagi. 2023. Benchmarking and Improving Text-to-SQL Generation under Ambiguity. In EMNLP. arXiv:2310.13659arXiv
  3. [3]Z. Chen et al. 2021. FinQA: A Dataset of Numerical Reasoning over Financial Data. In EMNLP. arXiv:2109.00122arXiv
  4. [4]Y. Cheng, C. Fan, M. JafariRaviz, K. Rezaei, and S. Feizi. 2026. Model-Adaptive Tool Necessity Reveals the Knowing-Doing Gap in LLM Tool Use. arXiv preprint. arXiv:2605.14038arXiv
  5. [5]Y. Dai, Y. Lin, Z. Xie, and Y. Wang. 2026. RealFin: How Well Do LLMs Reason About Finance When Users Leave Things Unsaid? arXiv preprint. arXiv:2602.07096arXiv
  6. [6]A. D’Amour et al. 2022. Underspecification Presents Challenges for Credibility in Modern Machine Learning. Journal of Machine Learning Research 23(226) (2022). arXiv:2011.03395arXiv
  7. [7]P. Hager et al. 2024. Evaluation and Mitigation of the Limitations of Large Language Models in Clinical Decision-Making. Nature Medicine 30(9):2613-2622 (2024). https://doi.org/10.1038/s41591-024-03097-1
  8. [8]M. M. Hasan, H. Li, G. K. Rajbahadur, B. Adams, and A. E. Hassan. 2026. Model Context Protocol (MCP) Tool Descriptions Are Smelly! Towards Improving AI Agent Efficiency with Augmented MCP Tool Descriptions. arXiv preprint. arXiv:2602.14878arXiv
  9. [9]P. Islam et al. 2023. FinanceBench: A New Benchmark for Financial Question Answering. arXiv preprint. arXiv:2311.11944arXiv
  10. [10]N. Kim, P. M. Htut, S. R. Bowman, and J. Petty. 2023. (QA)²: Question Answering with Questionable Assumptions. In ACL. arXiv:2212.10003arXiv
  11. [11]L. Kuhn, Y. Gal, and S. Farquhar. 2022. CLAM: Selective Clarification for Ambiguous Questions with Generative Language Models. arXiv preprint. arXiv:2212.07769arXiv
  12. [12]P. Laban, H. Hayashi, Y. Zhou, and J. Neville. 2026. LLMs Get Lost In Multi-Turn Conversation. In ICLR. arXiv:2505.06120arXiv
  13. [13]B. Z. Li, B. K. Kim, and Z. Wang. 2025. QuestBench: Can LLMs Ask the Right Question to Acquire Information in Reasoning Tasks?. In NeurIPS Datasets and Benchmarks Track. arXiv:2503.22674arXiv
  14. [14]J. Li et al. 2023. Can LLM Already Serve as a Database Interface? A Big Bench for Large-Scale Database Grounded Text-to-SQLs. In NeurIPS Datasets and Benchmarks. arXiv:2305.03111arXiv
  15. [15]Malaviy, C., Chang, J. C., Roth, M., Iyver, M., Yatskar, K., & Lo, K. (2025). Contextualized Evaluations: Judging Language Model Responses to Underspecified Queries. Transactions of the Association for Computational Linguistics, 13. arXiv:2411.07237
  16. [16]S. Min et al. 2023. FActScore: Fine-grained Atomic Evaluation of Factual Precision in Long Form Text Generation. In EMNLP. arXiv:2305.14251arXiv
  17. [17]S. Min, J. Michael, H. Hajishirzi, and L. Zettlemoyer. 2020. AmbigQA: Answering Ambiguous Open-domain Questions. In EMNLP. arXiv:2004.10645arXiv
  18. [18]Ming, Y., & Lyu, Z. (2025). FaithEval: Can Your Language Model Stay Faithful to Context, Even If “The Moon is Made of Marshmallows”.. In ICLR. arXiv:2410.03727
  19. [19]P. Pezeshkpour and E. Hruschka. 2024. Large Language Models Sensitivity to The Order of Options in Multiple-Choice Questions. In Findings of NAACL. arXiv:2308.11483arXiv
  20. [20]C. Qian et al. 2024. Tell Me More! Towards Implicit User Intention Understanding of Language Model Driven Agents. In ACL. arXiv:2402.09205arXiv
  21. [21]H. Rashkin et al. 2023. Measuring Attribution in Natural Language Generation Models. Computational Linguistics (2023). arXiv:2112.12870arXiv
  22. [22]I. Saparina and M. Lapata. 2024. AMBROSIA: A Benchmark for Parsing Ambiguous Questions into Database Queries. In NeurIPS Datasets and Benchmarks. arXiv:2406.19073arXiv
  23. [23]M. Sclar, Y. Choi, Y. Tsvetkov, and A. Suhr. 2024. Quantifying Language Models’ Sensitivity to Spurious Features in Prompt Design or: How I Learned to Start Worrying about Prompt Formatting. In ICLR. arXiv:2310.11324arXiv
  24. [24]India, S. and E. B. of . (2023). Regulatory framework for Execution Only Platforms for facilitating transactions in direct plans of schemes of Mutual Funds (SEBI/HO/IMD/IMD-PoD-1/P/CIR/2023/86). <a href="https://doi.org/https://www.sebi.gov.in/legal/circulars/jun-2023/regulatory-framework-for-execution-only-platforms-for-facilitating-transactions-in-direct-plans-of-schemes-of-mutual-funds_72479.html" rel="noopener noreferrer" target="_blank">https://doi.org/https://www.sebi.gov.in/legal/circulars/jun-2023/regulatory-framework-for-execution-only-platforms-for-facilitating-transactions-in-direct-plans-of-schemes-of-mutual-funds_72479.html</a>
  25. [25]India, S. and E. B. of . (2025). SEBI (Intermediaries) (Amendment) Regulations, 2025 (No. SEBI/LAD-NRO/GN/2025/226). <a href="https://doi.org/https://www.sebi.gov.in/legal/regulations/feb-2025/securities-and-exchange-board-of-india-intermediaries-amendment-regulations-2025_91809.html" rel="noopener noreferrer" target="_blank">https://doi.org/https://www.sebi.gov.in/legal/regulations/feb-2025/securities-and-exchange-board-of-india-intermediaries-amendment-regulations-2025_91809.html</a>
  26. [26]M. Sharma, M. Tong, T. Korbak, et al. 2024. Towards Understanding Sycophancy in Language Models. In ICLR. arXiv:2310.13548arXiv
  27. [27]Shi, F., Chen, X., Misra, R., Sales, D., Shi, C. H., Schärli, N., & Zhou, D. (2023). Large Language Models Can Be Easily Distracted by Irrelevant Context. In ICML. arXiv:2302.00093
  28. [28]Y. Sui, M. Zhou, M. Zhou, M. Han, S. Han, and D. Zhang. 2024. Table Meets LLM: Can Large Language Models Understand Structured Table Data? A Benchmark and Empirical Study. In WSDM. arXiv:2305.13062arXiv
  29. [29]H. Sun, W. W. Cohen, and R. Salakhutdinov. 2022. Conditional-QA: A Complex Reading Comprehension Dataset with Conditional Answers. In ACL. arXiv:2110.06884arXiv
  30. [30]Vu, T., Myer, N., Wang, X., Constant, J., Wei, J., Wei, C., Tar, Y.-H., Sung, D., Zhou, Q., Le, N., & Luong, T. (2024). FreshLLMs: Refreshing Large Language Models with Search Engine Augmentation. In Findings of ACL. arXiv:2310.03214
  31. [31]X. Wang, T. S. T. Kwok, Z. Tai, and G. Cheng. 2026. FinInteract: Benchmarking Clarification and Intent Integration in Ambiguous Financial Question Answering. arXiv preprint. arXiv:2609.24002arXiv
  32. [32]K. Wu, E. Wu, and J. Zou. 2024. ClashEval: Quantifying the Tug-of-War between an LLM’s Internal Prior and External Evidence. In NeurIPS Datasets and Benchmarks. arXiv:2404.10198arXiv
  33. [33]H. Yang, W. Song, T. Kim, J. Song, S. Park, and Y. Jo. 2026. Agents’ Overreliance on Unreliable Tools. arXiv preprint. arXiv:2609.05587arXiv
  34. [34]Yu, S., Lin, S., Zettlemoyer, L., & Hajishirzi, H. (2023). CREPE: Open-Domain Question Answering with False Presuppositions. In ACL. arXiv:2211.17257
  35. [35]T. Zhang, P. Qin, Y. Deng, et al. 2024. CLAMBER: A Benchmark of Identifying and Clarifying Ambiguous Information Needs in Large Language Models. In ACL. arXiv:2405.12063arXiv
  36. [36]Zheng, C., Zhou, H., Feng, J., Zhou, M., & Huang, M. (2024). Large Language Models Are Not Robust Multiple Choice Selectors. In ICLR. arXiv:2309.03882
  37. [37]L. Zheng et al. 2023. Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena. In NeurIPS Datasets and Benchmarks. arXiv:2306.05685arXiv

Paper details

vlnc:W_9zx3e5es
Version
v1 · latest
Published
Fresh — added to the Hub within the last 30 days
Added
License
CC BY 4.0
Usage

views · 0

downloads · 0

A view counts once per browser per day, a download each time the PDF is requested.

  • Computer Science

Not recorded for this paper

View statement
Looking for related papers…
Contents