betaFeedback

WorksComputer Science

When to Stop Asking: Factor Coverage Bias and LLM Determinability Judgments with Underspecified Information in Medicine

Abstract

Clinical decision-making often begins with underspecified information and requires not only gathering missing information but also knowing when to stop. This calls for judging determinability (whether the available information is sufficient for a decision) by considering whether missing information could change the conclusion, not simply the amount of available decision-relevant information. Reliance on this amount as a cue for determinability, which we term factor coverage bias, could undermine the trustworthiness of large language models (LLMs) through premature conclusions or excessive information gathering, such as unnecessary invasive tests. Yet whether LLMs exhibit this bias remains insufficiently studied. To address this gap, we construct a controlled pairwise benchmark in which LLMs identify the determinable case within each determinable–undeterminable pair. We add a decision-relevant factor to either side while keeping each case's determinability unchanged, and similarly add irrelevant text to distinguish this bias from context-length effects. Adding a relevant factor made LLMs more likely to select the augmented case as determinable even when it remained undeterminable, an effect that persisted with increased reasoning effort, while adding irrelevant text made them less likely to select the augmented case, an effect that reasoning substantially reduced. These findings identify factor coverage bias as a failure mode distinct from context-length effects and a potential risk to LLM trustworthiness. Our work provides an evaluation framework for LLM trustworthiness under underspecification.

Introduction

Recent LLMs have reached near-ceiling performance on benchmarks based on medical licensing examinations (Tordjman et al., 2025; Liu et al., 2026). The challenge is shifting from knowledge to reasoning as experts do in practice (Li et al., 2024; Watanabe et al., 2026). Examinations, however, typically provide the information needed for a decision, whereas clinical practice often involves underspecified information (Hersh et al., 2013; Tobin-Schnittger et al., 2018), limiting what these examinations can reveal about reasoning under such conditions (Hager et al., 2024). In clinical practice, experts tend to focus information gathering on what is needed for a decision, while trainees tend to collect more unnecessary information (Bowen, 2006; McQuade et al., 2024). When inputs are underspecified, as in patient-facing history taking and decision support for physicians outside their specialty, LLMs need not only to gather missing information but also to know when to stop, avoiding premature conclusions and excessive information gathering. In clinical practice, unnecessary questioning in emergencies or unnecessary invasive testing can harm patients (Grady and Redberg, 2010), so gathering every potentially relevant detail is not necessarily appropriate even if the final conclusion is correct. Prior work examines abstention under insufficient information and gathering additional information (Machcha et al., 2026; Li et al., 2024), but stopping once information is sufficient has received less attention. Watanabe et al. (2026) defined determinability by whether missing information could change the conclusion and demonstrated failures in LLM determinability judgments, without fully examining which input cues drive them. Failures in determinability judgment may undermine LLM trustworthiness beyond medicine, including coding assistance, legal consultation, and customer support, where robust responses to underspecified inputs require clarifying user intent and gathering necessary information, yet asking about every potentially relevant detail burdens users and reduces efficiency (Fang et al., 2026; Vincent et al., 2026; Mass et al., 2022).

LLM biases, such as sycophancy (Sharma et al., 2024), can undermine trustworthiness by reducing robustness. LLMs are also sensitive to context length and distraction by irrelevant content [3] (Shi et al. 2023; Du et al. 2025). Inputs grow through both relevant information and irrelevant text, especially during retrieval or multi-turn interactions. However, more relevant information does not necessarily imply determinability. Few factors can suffice, while many can still be insufficient. LLMs may still use the amount of available decision-relevant information as a cue for determinability, a tendency we term factor coverage bias (Figure 1). This bias could lead to premature conclusions or excessive information gathering and may require different mitigation from context-length effects, because filtering irrelevant text would not prevent reliance on the amount of relevant information. Adding relevant information also lengthens the context, but whether factor coverage biases determinability judgments independently of context length remains insufficiently studied.

Concept of factor coverage bias

Figure 1. Concept of factor coverage bias. Determinability depends on whether missing information could change the conclusion. An LLM exhibiting factor coverage bias tends to use the amount of available decision-relevant information as a cue for determinability. This can lead to errors in deciding when to stop gathering information: premature conclusions when many factors are available but still insufficient, and excessive information gathering when few factors already suffice.

To address these gaps, we construct a controlled pairwise benchmark of 1,080 pairs based on clinical scoring systems to examine the bias clearly. Each pair contains a determinable and an undeterminable case, both with missing information, and LLMs identify which is determinable. We test for factor coverage bias by adding a decision-relevant factor to either side while keeping each case’s determinability unchanged and measuring how the judgment shifts. We similarly add irrelevant text of comparable length to either side to distinguish this bias from context-length effects. Across recent open-weight and proprietary LLMs, adding a relevant factor shifts judgments toward determinable even when the augmented case remains undeterminable, an effect that persists with increased reasoning effort, while adding irrelevant text shifts judgments toward undeterminable, an effect that reasoning substantially reduces. These differences in direction and response to reasoning identify factor coverage bias as a failure mode distinct from context-length effects. Our main contributions are as follows.

  • We define and demonstrate factor coverage bias, the tendency to use the amount of available decision-relevant information as a cue for determinability, as a potential risk to LLM trustworthiness.

  • We distinguish factor coverage bias from context-length effects through their different directions and responses to reasoning, informing future efforts to develop trustworthy LLMs.

  • We release a controlled pairwise benchmark for evaluating factor coverage bias in LLMs.

Related Work

LLM behavior under underspecified input. Underspecified inputs are a practical challenge in LLM deployment (Yang et al., 2026; Laban et al., 2025). Missing information in the input is distinct from gaps in the model’s knowledge (Ren et al., 2026). Prior work has evaluated LLM responses to underspecified inputs by examining whether they ask clarifying questions (Zhang et al., 2024; Dong et al., 2025; Li et al., 2025; Luo et al., 2025; Fang et al., 2026), abstain from answering (Kirichenko et al., 2025), or flag what is missing (Vincent et al., 2026). Few evaluate whether the conclusion could still change given what is missing (Watanabe et al., 2026), and systematic biases in this judgment remain insufficiently understood. Our work characterizes these biases.

Heuristics in LLM judgment. Input-driven biases undermine the robustness and trustworthiness of LLMs. Known examples include sycophancy ((Sharma et al., 2024)) and evaluators’ preference for longer responses ((Zheng et al., 2023; Dubois et al., 2024)). Adding irrelevant text or increasing context length can reduce accuracy ([3] (Shi et al. 2023; Du et al. 2025)), but the direction of the resulting answer shifts remains underexplored. Increasing supporting evidence shifts answers toward the supported side ((Xie et al., 2024; Naphade, 2026)), but these studies do not examine information sufficiency judgments. In these studies, the amount of relevant information and context length also change together, leaving their effects difficult to distinguish, although they may require different mitigation. We address these gaps by comparing additions of relevant information and irrelevant text while holding determinability fixed and examining how judgments shift.

LLM benchmarks in medicine. Medical benchmarks for LLMs have largely assumed complete information, as in those derived from licensing examinations ((Jin et al., 2021; Pal et al., 2022)). Work on incomplete information has evaluated abstention ((Machcha et al., 2026; Cocchieri et al., 2026)), requests for additional information ([5]), and interactive information gathering ((Li et al., 2024; Schmidgall et al., 2026)). Clin-DetBench ((Watanabe et al., 2026)) evaluates determinability judgment directly, but because its items are constructed by progressively removing information from complete cases, factor coverage and determinability are correlated by construction, so the benchmark cannot detect whether a model responds to factor coverage rather than to determinability. We manipulate the two independently and evaluate this failure mode.

Methodology

Motivation

We adopt a pairwise format because it controls for each model’s propensity to abstain, which confounds single-case judgments, and because pairwise comparison amplifies superficial cues ((Jeong et al., 2025)), making bias easier to detect. Case presentations are constructed from clinical scoring systems. The aim is not to evaluate the ability to apply scoring rules; rather, because the judgment is determined by item values and a threshold, the correct label can be constructed uniquely, which yields a robust experimental setting.

Preliminary: Clinical Scoring Systems

Clinical scoring systems are widely used rule-based tools that assign points to clinical findings and guide decisions such as treatment initiation or risk stratification according to whether the total score reaches a threshold. A representative example, CHADS2_2, assesses the risk of stroke in patients with atrial fibrillation. It assigns one point each to congestive heart failure (C), hypertension (H), age ≥75\ge75 (A), and diabetes (D), and two points to a history of stroke or transient ischemic attack (S2_2), and classifies a total score ≥2\ge2 (i.e., threshold θ=2\theta= 2) as high risk (Table 1). We use 15 scoring systems that are recommended in clinical guidelines and have explicit thresholds, each with 4–9 scored items (Table A.2).

ItemPoints if positive
Congestive heart failure1
Hypertension1
Age ≥75\ge751
Diabetes mellitus1
Prior stroke or TIA2

Table 1. CHADS2_2 scoring system. Total scores range from 0 to 6; we use a threshold of θ=2\theta= 2.

Definition of Determinability

When the values of some scored items are missing, the exact total score cannot be computed, but the judgment can still be determined if all possible total scores fall on the same side of the threshold. Let Smin⁡S_{\min} and Smax⁡S_{\max} denote the minimum and maximum possible total scores given the available information, obtained by assigning each item with a missing value its minimum and maximum possible points, respectively. With threshold θ\theta, a case is determinable if Smin⁡≥θS_{\min} \ge\theta or Smax⁡<θS_{\max} < \theta, and undeterminable if Smin⁡<θ≤Smax⁡S_{\min} < \theta\le S_{\max}.

For example, in CHADS2_2 (θ=2\theta= 2), a patient aged 80 with a history of heart failure has a total score of at least 2 regardless of the remaining missing items and is therefore determinable. In contrast, a patient with hypertension but without heart failure or diabetes, whose age and stroke history are missing, has a possible total score ranging from 1 to 4 and is therefore undeterminable. Determinability thus depends on whether missing information could change the conclusion, rather than the amount of available relevant information.

Task Formulation

Figure 2 provides an overview of our benchmark. Each pair contains a determinable and an undeterminable case constructed from the same clinical scoring system. The model identifies, in a forced binary choice, which case contains sufficient information to settle the judgment under that system. Both cases have missing scored items. One is determinable, meaning that the judgment is the same for all possible values of the missing items, while the other is undeterminable, meaning that the judgment depends on those values.

Overview of the pairwise benchmark

Figure 2. Overview of our pairwise benchmark. LLMs identify which case is determinable. We add a decision-relevant factor to either the determinable or the undeterminable case. We similarly add irrelevant text to either case, yielding four conditions in total. Each addition preserves both cases’ determinability. We compare the resulting judgment shifts to distinguish factor coverage bias from context-length effects.

Dataset Construction

Materials. Case presentations are constructed from pre-written text chunks that explicitly indicate whether each documented scored item is positive (scoring) or negative (non-scoring). Table A.3 illustrates these chunks using CHADS2_2. Irrelevant sentences describe medically plausible details, such as drug allergies, current medications, or family history, that do not affect the score (Table A.4). A board-certified internal medicine physician checked the chunks against item definitions and verified that irrelevant sentences neither directly nor implicitly indicated any scored item. For example, irrelevant sentences were checked for mentions of antidiabetic drugs in scoring systems that include diabetes. Determinability labels are assigned automatically by computing the possible total score range (Smin⁡S_{\min}, Smax⁡S_{\max}) and comparing it with the threshold. The prompt template and examples of the actual prompts are provided in Appendix § 10.

Baseline pairs. A baseline pair consists of a determinable case and an undeterminable case with the same number of missing items and comparable length to minimize differences in information amount and context length at baseline. Eight pairs were sampled uniformly at random within each scoring system, yielding 120 baseline pairs, with correct answers balanced between A and B. Each baseline pair is used in nine conditions, including baseline, yielding 1,080 pairs in total (120 × 9). Baseline cases average 41 tokens, with a mean prompt length of 140 tokens (Table A.5).

Adding a relevant factor. To test for factor coverage bias, we examine whether adding relevant information shifts judgments toward the augmented case as determinable. A relevant factor is realized as a positive or negative chunk for a previously missing scored item (Table A.3), which we add to either case of a baseline pair. Baseline pairs are constructed so that the undeterminable case remains undeterminable after the addition, while the determinable case also retains its label. This increases both the amount of available relevant information and the context length without changing either case’s determinability. The addition averages 8 tokens.

Adding irrelevant text. To distinguish factor coverage bias from context-length effects, we similarly add irrelevant sentences (Table A.4) to either case without changing its determinability. The length-matched condition adds one sentence with the same token count as the added relevant factor for the same scoring system (mean 8 tokens). To examine larger length increases, the 2× and 4× conditions extend the case to approximately twice and four times its original length, adding 43 and 127 tokens on average, respectively. Token counts by condition are reported in Table A.6.

Evaluation

For each model and each condition, we use the error rate against the correct label (1 − accuracy) as the metric. The effect of an addition is quantified by attraction. Because the correct answer is always the determinable side, a tendency to select the side to which text was added increases errors when the addition is made to the undeterminable side and decreases errors when it is made to the determinable side. We therefore define attraction = ErrUnd−ErrDet\mathrm{Err}_{\mathrm{Und}}-\mathrm{Err}_{\mathrm{Det}}, where ErrUnd\mathrm{Err}_{\mathrm{Und}} and ErrDet\mathrm{Err}_{\mathrm{Det}} are the error rates when the text is added to the undeterminable and determinable side, respectively. Attraction is positive when the judgment is drawn toward the added side, negative when the added side is avoided, and near 0 when an addition merely degrades both sides equally (nondirectional noise). We obtain 95% confidence intervals (CIs) by a pair-level paired bootstrap (10,000 resamples, percentile method).

Experiments

Experimental Setup

We evaluated GPT-5.5, Claude Opus 4.5, and three open-weight models, Qwen3.5-27B, Qwen3.5-9B, and Qwen3.6-35B-A3B, through their providers’ APIs, with the Qwen models accessed via OpenRouter. Reasoning effort was varied through the thinking level for Opus 4.5 (none/low/medium/high), the reasoning-effort parameter for GPT-5.5 (none/low/high/max), and thinking off/on for the Qwen models (reported as none/on). Temperature was left at the provider default, and each instance was queried once. The prompt was a single zero-shot template containing the scoring system, the decision target, and the two cases (Figure A.2); no judgment procedure was supplied, because our question is whether models make the judgment autonomously.

Overall Error Rates

Table 2 reports error rates and changes from baseline. Enabling reasoning reduced error rates across all five models and all conditions shown, with baseline error rates falling from 0.342–0.425 to 0.017–0.200 (see also Figure A.1 in the appendix). Adding a relevant factor to the determinable case reduced or left unchanged the error rate, whereas adding it to the undeterminable case increased the error rate in every model and reasoning setting. With reasoning enabled, adding a relevant factor to the undeterminable case produced the highest error rate for every model among the conditions shown. Without reasoning, irrelevant text showed the opposite pattern, increasing errors when added to the determinable case and decreasing errors when added to the undeterminable case; these changes were generally smaller with reasoning. The next section quantifies these directional shifts as attraction. Results including intermediate reasoning levels and the 2× and 4× conditions are provided in Table A.1.

ModelReasoningBaselineAdd relevant factor DetAdd relevant factor UndAdd irrelevant text (matched) DetAdd irrelevant text (matched) Und
GPT-5.5none0.3420.233 (−0.108)0.525 (+0.183)0.475 (+0.133)0.267 (−0.075)
max0.0170.017 (±0.000)0.050 (+0.033)0.025 (+0.008)0.025 (+0.008)
Opus 4.5none0.4250.358 (−0.067)0.525 (+0.100)0.558 (+0.133)0.392 (−0.033)
high0.1250.042 (−0.083)0.233 (+0.108)0.142 (+0.017)0.133 (+0.008)
Qwen3.5-27Bnone0.3420.225 (−0.117)0.492 (+0.150)0.542 (+0.200)0.183 (−0.158)
on0.1000.050 (−0.050)0.242 (+0.142)0.208 (+0.108)0.108 (+0.008)
Qwen3.5-9Bnone0.3750.350 (−0.025)0.433 (+0.058)0.617 (+0.242)0.275 (−0.100)
on0.1670.100 (−0.067)0.233 (+0.067)0.192 (+0.025)0.217 (+0.050)
Qwen3.6-35B-A3Bnone0.3750.367 (−0.008)0.508 (+0.133)0.500 (+0.125)0.308 (−0.067)
on0.2000.108 (−0.092)0.292 (+0.092)0.233 (+0.033)0.192 (−0.008)

Table 2. Error rates. Det/Und denote the determinable/undeterminable case to which text is added; values in parentheses are changes from baseline. Irrelevant text is length-matched to the added relevant factor. Reasoning indicates the setting used, with max for GPT-5.5 and high for Opus 4.5 representing the highest reasoning levels evaluated; for the Qwen models, none/on correspond to thinking off/on.

Factor Coverage Bias Is Distinct from Context-Length Effects

Figure 3 shows the attraction (§ 3.6) and its 95% CI for each model with reasoning off and on, for the addition of one relevant factor (left) and the addition of length-matched irrelevant text (right); on denotes the highest level available (max for GPT-5.5 and high for Opus 4.5).

Attraction with 95% CIs for relevant factor and length-matched irrelevant text

Figure 3. Attraction with 95% CIs for the addition of one relevant factor and of length-matched irrelevant text. Positive values indicate that the judgment was drawn toward the added side.

For the addition of a relevant factor, attraction was positive across all models and reasoning levels, indicating a tendency to select the case with added relevant information as determinable even with reasoning enabled. For GPT-5.5 with reasoning, the 95% CI excluded zero, but the point estimate remained positive (0.033). In contrast, adding irrelevant text produced negative attraction in all five models without reasoning, indicating a tendency to regard the augmented case as undeterminable. Reasoning substantially reduced this effect. Because the length-matched addition averaged only about 8 tokens, we also added irrelevant text to extend cases to approximately twice and four times their original length (Figure 4). Without reasoning, these larger additions amplified the shift toward undeterminable. Reasoning substantially reduced this effect, although statistically significant negative attraction remained in some settings, including Opus 4.5 at both lengths.

Attraction with 95% CIs for irrelevant text at 2× and 4× case length

Figure 4. Attraction with 95% CIs for irrelevant text at 2×2 \times and 4×4 \times case length. Positive values indicate that the judgment was drawn toward the added side.

LLM judgments shifted in response to input changes despite an unchanged correct answer, indicating a lack of robustness. The shift toward cases with added relevant information suggests that models use the amount of available relevant information as a cue for determinability. These findings demonstrate factor coverage bias. Its direction and response to reasoning distinguish factor coverage bias from context-length effects.

Discussion

We identify factor coverage bias, the tendency to use the amount of available relevant information as a cue for determinability, as a bias in LLM judgment distinct from context-length effects. By distorting determinability judgments, this bias may undermine LLM trustworthiness and usability through unnecessary clarification or premature conclusions. These risks are particularly relevant to tasks that require robust responses to underspecified inputs, including medicine, coding assistance, legal consultation, and customer support (Li et al., 2024; Fang et al., 2026; Vincent et al., 2026; Mass et al., 2022). In medicine, the consequences may be especially serious, ranging from incorrect decisions through premature conclusions (Graber et al., 2005) to patient harm from unnecessary invasive testing (Ganguli et al., 2019) and burdensome questioning. Trustworthy LLMs therefore require both

knowledge and the reasoning abilities that experts use in practice. Judging determinability is one such ability.

Given the substantial improvement in determinability judgments with reasoning, allocating inference resources to these judgments is an important design consideration. However, factor coverage bias persisted with reasoning, motivating further training and evaluation using accurately labeled determinable and undeterminable inputs. Such labels are difficult to obtain because experts do not necessarily document their judgments of information sufficiency during information gathering, even when the final conclusion is recorded. Our work addresses this challenge by synthesizing cases with determinability labels, providing an evaluation framework that could be extended to other domains for developing trustworthy LLMs.

Conclusion

We defined and demonstrated factor coverage bias, the tendency to use the amount of available decision-relevant information as a cue for determinability. This bias may undermine LLM trustworthiness under underspecification through premature conclusions and excessive information gathering. Its direction and response to reasoning distinguish it from context-length effects, informing future efforts to develop trustworthy LLMs. We release a controlled pairwise benchmark for evaluating this bias, providing an evaluation framework for developing trustworthy LLMs.

Limitations

This study has six limitations. First, the evaluation is limited to five models and to a single synthetic medical benchmark generated by concatenating text templates from clinical scoring systems with explicit decision criteria; it does not include judgments on which clinicians themselves might disagree. In addition, the text chunks were verified by a single physician. Because this verification consists of checking each chunk against explicit item definitions rather than exercising clinical judgment, we expect little inter-rater variability, but we did not measure it.

Second, our evaluation covers only relatively short contexts. Although reasoning substantially reduced negative attraction from irrelevant text, we did not test whether this reduction holds at substantially greater context lengths. The irrelevant-text conditions were designed to distinguish factor coverage bias from context-length effects within our benchmark, rather than to characterize long-context behavior comprehensively.

Third, the pairwise design was chosen to observe the bias while suppressing the influence of each model’s abstention tendency; whether the same bias appears in pointwise judgments of a single case has not been tested. Nor have we tested settings closer to actual use, in which information accumulates sequentially through history taking or retrieval-augmented generation; whether the bias manifests there as premature conclusion of information gathering is a question that directly corresponds to the motivation of this study but remains open.

Fourth, this study targets whether models can make the judgment autonomously, using a single zero-shot template that does not supply the reasoning procedure an expert would follow. Prior work has shown that providing the judgment procedure through few-shot prompting improves the accuracy of determinability judgment (Watanabe et al. (2026)), and the present bias would presumably be mitigated as well; however, because our question is whether LLMs apply such an expert procedure on their own, mitigation by externally supplying the procedure is outside the scope of this study.

Fifth, the amount of relevant information could not be varied: adding further relevant factors to the undeterminable case eventually makes both cases of a pair determinable, so the relevant condition was restricted to a single factor. Even a single factor, however, produced a shift comparable to that of irrelevant text (Table 2).

Finally, for GPT-5.5 with reasoning enabled, the error rates were near the floor (0.017–0.050); because attraction is defined as a difference in error rates, the range of values it can take is compressed in this setting. The point estimate was positive, as for the other models, but the confidence interval included 0, and we cannot distinguish whether this reflects the absence of this bias or low statistical power due to the small number of errors.

Ethical Considerations

Our benchmark consists exclusively of synthetic cases constructed from publicly available scoring criteria and does not include any patient information or private clinical data. This benchmark is intended solely for research and evaluation purposes and is not designed for direct use in clinical practice or patient-facing applications. The proposed framework does not replace clinical judgment or human supervision. To ensure transparency and reproducibility, the dataset is publicly released under the MIT License.

Additional Results

Table A.1 reports the error rate for every model and reasoning level under all nine conditions, including the intermediate levels of GPT-5.5 and Opus 4.5 and the 2×2 \times and 4×4 \times conditions, and Figure A.1 plots each run against its median number of reasoning tokens. Within each model the error rate fell roughly monotonically as the reasoning level, and with it the number of reasoning tokens, increased. The only cells in which the highest level did not lower the error rate are the conditions in which 2×2 \times or 4×4 \times irrelevant text is added to the undeterminable case, for the Qwen models, where the avoidance of the side carrying irrelevant text happens to favor the correct answer without reasoning and reasoning removes it. Across models, however, more reasoning tokens did not mean fewer errors: at their highest levels the three Qwen models consumed more reasoning tokens than GPT-5.5 and Opus 4.5, yet reduced their baseline error rate less (from 0.342–0.375 to 0.100–0.200, against 0.342 to 0.017 for GPT-5.5 and 0.425 to 0.125 for Opus 4.5). The gain from reasoning is therefore bounded by the model rather than by the amount of reasoning tokens.

Clinical Scoring Systems

Table A.2 lists the 15 scoring systems with the rule used to generate the ground-truth labels and the decision target embedded in the prompt. For the two systems whose rule is an upper bound (Bishop and Centor), the definition of § 3 applies with the inequalities reversed.

Dataset Details

Table A.3 lists the scored-item chunks for the CHADS2_2 Score, from which the case presentations are built (§ 3), and Table A.4 the irrelevant text for the same system; one fixed irrelevant text was written per scoring system and amount (15×3=4515 \times3 = 45 texts), the three amounts are nested, and the same text is added whichever side receives it. Table A.5 gives the composition of the 120 baseline pairs by scoring system, and Table A.6 reports the measured amount of text added in each condition and the resulting prompt length. Figures A.3 and A.4 (Appendix § 10) show one pair built from these chunks under the baseline condition and the four conditions in which text is added to the undeterminable side.

Prompt Template and Examples

Figure A.2 shows the template, identical for every instance and every model; it names the scoring system but does not list its items or threshold, and no judgment procedure is supplied (§ 4). Figures A.3 and A.4 show the prompts actually sent for one pair of the CHADS2_2 Score under the baseline condition and the four conditions in which text is added to the undeterminable side; the conditions that add text to the determinable case add the same text to Case A instead, and text added relative to the baseline is underlined.

ModelReasoningBaselineAdd relevant factor DetAdd relevant factor Undmatched Detmatched Und2× Det2× Und4× Det4× Und
GPT-5.5none0.3420.2330.5250.4750.2670.5420.2580.5000.267
low0.0670.0170.1000.0670.0420.0580.0580.0580.083
high0.0330.0330.0580.0330.0250.0250.0500.0330.050
max0.0170.0170.0500.0250.0250.0250.0250.0250.025
Opus 4.5none0.4250.3580.5250.5580.3920.5750.3830.6580.317
low0.2250.0830.4170.2830.2170.3250.2170.3580.183
medium0.1080.0920.2670.2000.1330.2000.1000.2080.108
high0.1250.0420.2330.1420.1330.1670.1000.1750.092
Qwen3.5-27Bnone0.3420.2250.4920.5420.1830.7500.1330.7830.100
on0.1000.0500.2420.2080.1080.2170.1330.2080.200
Qwen3.5-9Bnone0.3750.3500.4330.6170.2750.7330.1830.8000.133
on0.1670.1000.2330.1920.2170.2420.2170.2330.233
Qwen3.6-35B-A3Bnone0.3750.3670.5080.5000.3080.5580.2250.7670.200
on0.2000.1080.2920.2330.1920.3000.2170.2920.217

Table A.1. Error rate for every model, reasoning level, and condition. Det/Und denote the determinable/undeterminable case of each pair to which the text is added. Matched, 2×, and 4× are the amounts of irrelevant text (Table A.6). The rows reported as reasoning on in the main text are max for GPT-5.5, high for Opus 4.5, and on for Qwen; none denotes no reasoning (thinking off for Qwen).

Error rate versus the number of reasoning tokens per run, by model and condition

Figure A.1. Error rate versus the number of reasoning tokens per run, by model and condition. GPT-5.5 and Opus 4.5 have four reasoning levels and the Qwen models two (thinking off/on). For Opus 4.5, reasoning tokens are estimated as output tokens minus the visible answer because the API does not report them separately, so its none level is not at zero.

Table A.2: The 15 clinical scoring systems used. Items is the number of scored items and Range the range of possible totals; Rule is the threshold and direction used to generate the ground-truth labels; the decision target is the wording embedded in the prompt. Ref. gives the original publication of each system together with a current clinical guideline in which it is used.

Scoring systemItemsRangeRuleDecision target (as used in the prompt)Ref.
Alvarado Score80–10≥7\ge7acute appendicitis is suggested[1] (Di Saverio et al. 2020)
Apgar Score50–10≥7\ge7the newborn’s condition is normal[4] (Committee on Fetus and Newborn 2015)
BISAP Score50–5≥3\ge3the patient is at higher risk of mortalityWu et al., 2008; Tenner et al., 2013
Bishop Score50–13≤7\le7the cervix is unfavorable for induction[6, 2]
Blatchford Score80–23≥1\ge1the patient is at high risk and needs intervention[8] (National Institute for Health and Care Excellence 2012)
CHA2_2DS2_2-VASc Score70–9≥2\ge2the patient is at high stroke risk and needs anticoagulationLip et al., 2010; Hindricks et al., 2021
CHADS2_2 Score50–6≥2\ge2the patient is at high stroke risk and needs anticoagulationGage et al., 2001; Takase et al., 2024
CURB-65 Score50–5≥3\ge3the pneumonia is severe and warrants hospitalizationLim et al., 2003; Metlay et al., 2019
Centor Score50–5≤2\le2no further testing or antibiotics are neededCentor et al., 1981; Shulman et al., 2012
Child-Pugh Classification50–15≥10\ge10the cirrhosis is class CPugh et al., 1973; European Association for the Study of the Liver, 2018
Glasgow-Imrie Score80–8≥3\ge3this is severe acute pancreatitis[7] (Imrie et al. 1978; Tenner et al. 2013)
HAS-BLED Score90–9≥3\ge3the patient is at high bleeding riskPisters et al., 2010; Hindricks et al., 2021
Khorana Score50–6≥3\ge3the patient is at high risk of venous thromboembolismKhorana et al., 2008; Key et al., 2023
ORBIT Bleeding Score50–7≥4\ge4the patient is at high risk of major bleedingO’Brien et al., 2015; Hindricks et al., 2021
SIRS Criteria40–4≥2\ge2the patient meets the criteria for SIRSBone et al., 1992; Tenner et al., 2013

Table A.2.

Table A.3: Examples of pre-written scored-item chunks for CHADS2_2. Positive chunks indicate that the scoring criterion is met, while negative chunks indicate that it is not. Missing item values are represented by omitting the corresponding chunks.

ItemPositive chunkNegative chunk
Congestive heart failurea history of congestive heart failureno history of congestive heart failure
Hypertensiona history of hypertensionno history of hypertension
Age ≥75\ge75An 80-year-oldA 68-year-old
Diabetes mellitusa history of diabetes mellitusno history of diabetes mellitus
Prior stroke or TIAa prior strokeno prior stroke or TIA

Table A.3.

Table A.4: Examples of pre-written irrelevant text for CHADS2_2. These sentences do not correspond to any scoring item. Each condition includes the text from the preceding condition plus the sentences shown in its row. Sentence and token counts are cumulative.

AmountSentences added
matched 1 sentence 7 tokensFamily history included maternal osteoarthritis.
2× 2 sentences 33 tokens+ The patient reported a severe allergy to macrolide antibiotics, which historically caused significant gastrointestinal upset and a diffuse skin rash during childhood.
4× 5 sentences 98 tokens+ According to the medical record, the patient took oral levothyroxine each morning to manage a long-standing history of uncomplicated primary hypothyroidism. Current regular medications included a daily oral zinc tablet and a nonprescription antihistamine taken during the spring months for seasonal allergic rhinitis. The patient had a penicillin allergy.

Table A.4.

Scoring systemMissingDet marginUnd marginDet tokensUnd tokens
Alvarado Score4 [2–5]0 [0–1]1 [1–3]48.2 (8.6)51.6 (10.4)
Apgar Score3 [2–3]0 [0–0]2 [1–3]23.5 (5.0)27.8 (5.3)
BISAP Score2 [2–2]0 [0–0]1 [1–1]32.8 (3.4)32.9 (2.4)
Bishop Score2 [2–2]0 [0–1]1 [1–3]40.2 (2.9)39.9 (2.3)
Blatchford Score4 [2–5]3 [0–9]1 [1–1]44.6 (8.0)40.6 (8.3)
CHA₂DS₂-VASc Score2 [2–5]1 [0–4]1 [1–2]41.6 (8.2)45.4 (9.9)
CHADS₂ Score3 [2–4]0 [0–1]1 [1–2]32.2 (4.2)32.9 (3.8)
CURB-65 Score2 [2–2]0 [0–0]1 [1–1]52.5 (3.4)50.1 (1.9)
Centor Score2 [2–3]0 [0–1]1 [1–2]26.9 (3.8)26.8 (5.1)
Child-Pugh Classification2 [2–2]0 [0–1]2 [1–2]44.2 (6.4)42.5 (5.3)
Glasgow-Imrie Score3 [2–4]0 [0–2]1 [1–2]66.9 (9.7)64.6 (8.0)
HAS-BLED Score4 [3–6]0 [0–1]1 [1–2]40.6 (8.5)40.8 (7.6)
Khorana Score2 [2–3]0 [0–1]1 [1–2]44.8 (5.4)47.2 (6.8)
ORBIT Bleeding Score2 [2–3]0 [0–1]2 [1–3]31.9 (4.8)30.4 (3.7)
SIRS Criteria2 [2–2]0 [0–0]1 [1–1]39.4 (3.1)37.9 (1.6)
All (120 pairs)2 [2–6]0 [0–9]1 [1–3]40.7 (11.9)40.8 (11.4)

Table A.5. Composition of the 120 baseline pairs by scoring system: eight pairs per system, four with the correct answer A and four with B. Missing = number of missing scored items per case, median [range]. Det margin = distance of the determinable side from the threshold (0 = exactly at it); Und margin = points the undeterminable side would need to move to become determinable. Tokens are mean (SD) per case.

ConditionAdded tokensPrompt tokens
Baseline—140
Rel to Det8147
Rel to Und8148
Irr (matched)8148
Irr (2×)43182
Irr (4×)127267

Table A.6. Amount of text added per case in each condition. Rel/Irr: addition of a relevant factor/irrelevant text; Det/Und: the determinable/undeterminable case receiving it (Irr rows apply to either side). Added tokens count only the inserted text; prompt tokens are the mean length of the complete prompt actually sent to the model, i.e., the instruction template plus the case description and any added text. Both columns report means. Token counts use tiktoken (o200k_base); baseline case descriptions average 41 tokens.

Prompt template

Figure A.2. The prompt template, identical for every instance and every model, sent as a single user message with no system prompt.

Prompts sent for one pair under Baseline, Rel to Und, and Irr to Und (matched)

Figure A.3. The prompts actually sent for one pair (CHADS₂ Score) under Baseline, Rel to Und, and Irr to Und (matched); Figure A.4 continues with 2× and 4×. Case A, the determinable side and correct answer, is identical throughout; the underlined text is added to Case B: a missing scored item (age) in Rel to Und, and one, two, or five irrelevant sentences (Table A.4) in Irr to Und. Rel to Det and Irr to Det add the text to Case A instead. Title bars give the token counts (tiktoken o200k_base).

The prompts for the same pair under Irr to Und with the 2× and 4× amounts of irrelevant text

Figure A.4. The prompts for the same pair under Irr to Und with the 2×2 \times and 4×4 \times amounts of irrelevant text (continued from Figure A.3). The irrelevant text is nested: each amount extends the preceding one (Table A.4). Text added relative to the baseline is underlined.

References

  1. [1]A. Alvarado. 1986. A practical score for the early diagnosis of acute appendicitis. Annals of Emergency Medicine, 15(5):557–564.pubmed.ncbi.nlm.nih.gov/3963537
  2. [2]American College of Obstetricians and Gynecologists. 2009. ACOG Practice Bulletin No. 107: Induction of Labor. Obstetrics & Gynecology, 114(2 Part 1):386.DOI
  3. [3]Chen Amiraz, Florin Cuconasu, Simone Filice, and Zohar Karmin. 2025. The Distracting Effect: Understanding Irrelevant Passages in RAG. In Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 18228–18258, Vienna, Austria. Association for Computational Linguistics.
  4. [4]V. Apgar. 1953. A proposal for a new method of evaluation of the newborn infant. Current Researches in Anesthesia & Analgesia, 32(4):260–267.
  5. [5]Rahul K. Arora, Jason Wei, Rebecca Soskin Hicks, Preston Bowman, Joaquin Quiñonero-Candela, Foivos Tsimpourlas, Michael Sharman, Meghan Shah, Andrea Vallone, Alex Beutel, Johannes Heidecke, and Karan Singhal. 2025. HealthBench: Evaluating Large Language Models Towards Improved Human Health. Preprint, arXiv:2505.08775.arxiv.org/abs/2505.08775
  6. [6]E. H. Bishop. 1964. PELVIC SCORING FOR ELECTIVE INDUCTION. Obstetrics and Gynecology, 24:266–268.pubmed.ncbi.nlm.nih.gov/14199536
  7. [7]S. L. Blamey, C. W. Imrie, J. O’Neill, W. H. Gilmour, and D. C. Carter. 1984. Prognostic factors in acute pancreatitis. Gut, 25(12):1340–1346.pubmed.ncbi.nlm.nih.gov/6510766
  8. [8]O. Blatchford, W. R. Murray, and M. Blatchford. 2000. A risk score to predict need for treatment for upper-gastrointestinal haemorrhage. Lancet (London, England), 356(9238):1318–1321.pubmed.ncbi.nlm.nih.gov/11073021

Paper details

vlnc:W_sgkhfxf8
Version
v1 · latest
Published
Fresh — added to the Hub within the last 30 days
Added
License
CC BY 4.0
Usage

views · 0

downloads · 0

A view counts once per browser per day, a download each time the PDF is requested.

  • Computer Science

Not recorded for this paper

View statement
Looking for related papers…
Contents