Valency Hub
alphaFeedback
PDF

WorksComputer Science

MedRoundsQA: A Persona and Difficulty Aware Evaluation for Multi-Turn Medical Consultations

Youssef Mohamed, Ahmed Heakl, Qinrong Cui, Junhong Liang, Rafiq Ali

Published v150 min read

AI contributions
Not recorded for this paper
View details

Abstract

Medical benchmarks are dominated by single-turn, multiple-choice clinical cases that poorly reflect real consultations. Practically, clinicians elicit evidence interactively and patient communication varies widely. We introduce MedRoundsQA, a multi-turn diagnostic benchmark derived from 1,387 board-exam cases across 17 specialties. Each case is converted into a structured 24-slot clinical record, and then instantiated as controlled doctor-patient dual-agent dialogues under varying patient personas, with the underlying clinical content held fixed. We further classify cases by difficulty using model-based uncertainty to enable easy-to-hard analysis. Evaluations of fifteen LLM doctor agents show that (i) moving from a single-turn diagnosis on the standardized records to multi-turn consultations causes large degradations of roughly 13-39 points; (ii) more turns reliably improves question relevance, but diagnostic accuracy exhibits diminishing returns and typically plateaus after 6-12 turns; and (iii) patient persona differences can shift diagnosis accuracy by about 7-8 points (lowest to highest education), highlighting equity risks that single-turn benchmarks miss.

Introduction

Medical diagnosis is inherently interactive: clinicians gather information incrementally through targeted questioning, and the quality of that information depends on how patients communicate. Yet current benchmarks ignore both properties, presenting models with complete clinical vignettes for single-turn diagnosis — bypassing the sequential reasoning and patient-facing communication that define real consultations.

This disconnect matters. LLMs are already deployed in clinical workflows: 20% of UK GPs use tools like ChatGPT in practice ([8]), and one in six US adults consult AI chatbots for health information monthly ([17]). Failures when information arrives gradually or patients communicate imprecisely carry consequences beyond benchmark scores.

Existing resources fall into two categories, neither sufficient. Single-turn benchmarks (MedQA, MedMCQA, MultiMedQA) provide exam-quality ground truth but treat each case as a static input ([19], [31], [35]). Dialogue corpora (MedDialog, MedDG) capture multi-turn structure but lack controlled diagnostic ground truth ([16], [26]). Agent-based benchmarks (AgentClinic, MediQ, AI Hospital) introduce interactive evaluation but vary patients across different cases, making persona effects inseparable from case difficulty ([34], [22], [14]).

Key missing capability: counterfactual control. Attributing performance differences to communication requires instantiating the same clinical case under multiple patient personas with clinical truth held fixed, enabling causal measurement of robustness and equity gaps. Table 1 summarizes how existing resources fall short.

BenchmarkTypeSourceSizeMTMAPersonaDFSchemaCFST↔MTFCLang.
Single-turn QA benchmarks
MedQA ([19])QABoard exams12.7k✗✗✗✗✗✗✗✗EN/ZH
MedMCQA ([31])QAEntrance exams194k✗✗✗✗✗✗✗✗EN
MultiMedQA ([35])QAMixed7 sets✗✗✗✗✗✗✗✗EN
Dialogue corpora (training-focused)
MedDialog-EN ([16])CorpusTelemedicine300k✓✗✗✗✗✗✗✗EN
MedDG ([26])CorpusTelemedicine17k✓✗✗✗Partial✗✗✗ZH
Multi-agent dialogue benchmarks
MediQ ([22])BenchmarkCurated1.2k✓✓✗✗✗✗✗✗EN
AgentClinic ([34])BenchmarkMixed457✓✓✗✗✗✗✗✗EN
AI Hospital ([14])BenchmarkCurated506✓✓Temper.✗✗✗✗✗ZH
Dr.APP (Zhu and Wu, 2025)BenchmarkCurated1.5k✓✓Temper.✗✗✗✗✗EN
3MDBench ([36])BenchmarkMed images3k✓✓Temper.✗✗✗✗✓EN
MedAgentSim ([2])BenchmarkMixed637✓✓✗✗✗✗✗✗EN
MedRoundsQA (Ours)BenchmarkBoard exams1.4k✓✓Multi-attr.✓✓ (24)✓✓✓EN

Table 1. Comparison with existing medical QA and dialogue benchmarks. MT: multi-turn dialogue. MA: multi-agent (doctor–patient) setting. Persona: systematic variation of patient attributes (Temper.=emotion/temperament; Multi-attr.=multiple controlled attributes such as education, affect, language proficiency, occupation). DF: automatic difficulty stratification. Schema: canonical structured clinical representation ((24) = 24-slot record). CF (Counterfactual personas): same underlying case instantiated across personas (enables causal attribution of persona effects). ST↔MT: single-turn and multi-turn evaluated on the same normalized cases under the same ground truth, so the ST–MT gap isolates the interactive component; benchmarks reporting both settings without a shared canonical representation are not marked. FC: full consultation until diagnostic decision or turn budget. Key novelty: the only benchmark combining a canonical schema, counterfactual personas, and same-case ST↔MT evaluation, enabling controlled measurement of communication-driven reliability/fairness gaps.

We introduce MedRoundsQA, converting 1,387 board-exam questions across 17 specialties into controlled doctor–patient dialogues. Each case is normalized into a 24-slot clinical record serving as ground truth, then instantiated with counterfactual patient personas varying communication characteristics (health literacy, language proficiency, affect, occupation) while holding clinical content fixed. Cases are further stratified by difficulty via model-based uncertainty signals.

Our evaluation of fifteen LLM doctor agents yields three findings: (i) diagnostic accuracy drops 13–39 points moving from single-turn to multi-turn on the same cases; (ii) additional turns improve question relevance but diagnostic accuracy plateaus after 6–12 turns; (iii) patient personas shift accuracy by 7–8 points across education levels, exposing equity concerns invisible to single-turn evaluation.

Contributions. (1) Counterfactual causal control: each case is instantiated under multiple personas with clinical content held fixed in a shared 24-slot record, attributing performance gaps to communication style rather than case difficulty and enabling same-case ST↔MT comparison. (2) Difficulty-aware evaluation: A model-uncertainty protocol stratifies cases for fine-grained easy-to-hard analysis. (3) Empirical characterization of interaction failures: Substantial ST→MT degradation, diminishing returns with longer dialogues, and systematic persona-driven disparities.

MedRoundsQA evaluates consultation-style diagnosis where both multi-turn interaction and patient presentation matter. Measuring the effect of patient communication requires counterfactual control: instantiating multiple personas for the same underlying case to avoid confounding by case difficulty.

Medical QA benchmarks. Exam-style benchmarks MedQA [19], MedMCQA [31], PubMedQA [20], BioASQ [37], MultiMedQA [35], and MMLU [18] assess factual recall but treat each case as a single-shot vignette, ignoring incremental information acquisition inherent in real consultations.

Medical dialogue corpora. Large-scale corpora (MedDialog [16], MedDG [26], IMCS-21 [11], ChatDoctor [23]) support training but are opportunistic collections with heterogeneous quality and weak ground-truth labels, unsuited for controlled evaluation.

Multi-agent medical dialogue benchmarks. AgentClinic [34], MediQ [22], AI Hospital [14], Dr.APP [?], 3MDBench [36], and MedAgentSim [2] introduce agent-based frameworks, some with persona modeling. Unlike these, MedRoundsQA uses a canonical evidence interface with same-case persona instantiation, enabling counterfactual evaluation of communication-driven performance gaps.

Multi-turn medical consultation evaluation. Most closely related work lacks the controls needed to isolate patient communication effects. [24] use a single fixed patient prompt with no persona variation, making it impossible to attribute performance differences to communication style rather than case difficulty. MedDialogRubrics ([15]) measures process quality against EBM-derived rubrics, but its ground truth is LLM-generated and cases are never held fixed across patient styles, precluding causal persona attribution. Manczak et al. (2025) evaluate adversarial robustness after a diagnosis is formed from a complete vignette; there is no patient agent and no information elicitation, making it orthogonal to consultation-style reasoning. MedMT-Bench (Yang et al., 2026) targets instruction-following and memory across long contexts rather than diagnostic accuracy, with no validated ground truth or persona control. MedRoundsQA is the only benchmark combining validated board-exam ground truth, a 24-slot canonical schema, same-case counterfactual personas, difficulty stratification, and direct ST↔MT comparison on identical clinical content—capabilities absent from all four prior works (Table 1).

Methods

We construct a controlled benchmark that converts single-turn exam questions into grounded multi-turn doctor–patient consultations with (i) a canonical 24-slot case record, (ii) systematically varied patient personas, and (iii) difficulty labels derived from model-based uncertainty.

To convert vignettes into our 24-slot schema,

we compare three candidate LLMs (GPT-4o ([30]), DeepSeek ([13]), and Claude ([5]) on a randomly sampled set of 100 questions. A general practitioner (GP) manually annotates these cases into the schema to provide slot-level reference records. We evaluate extraction quality slot-wise (micro-averaged across slots) using lexical and semantic similarity measures and select the model that performs best overall. GPT-4o consistently achieves the strongest agreement with GP annotations (Figure 7), and we use it to extract structured records for the full benchmark.

Comparison of extraction models across five evaluation metrics

Figure 7. Comparison of extraction models across five evaluation metrics against GP-annotated ground truth.

Data Construction

Source and filtering. We derive cases from MedQA ([19]), NEJM diagnostic cases ([33]), and MedMCQA ([31]). We filter to retain only text-sufficient diagnostic vignettes, excluding items focused on management/guidelines/factual recall or those requiring images or non-clinical administrative knowledge. Filtering uses a two-stage protocol: an automatic LLM filter (Appendix A.1) followed by independent review by two GPs; disputed inclusions are removed. The final dataset contains 1,387 cases (292 MedQA, 646 NEJM, 449 MedMCQA) spanning 17 specialties.

Canonical case representation. To bridge heterogeneous question styles and support dialogue generation, we convert each vignette into a unified structured representation that decouples clinical content from its surface realization. Our schema is inspired by the structured medical records used in MMD-Eval ([25]), but adapted to outpatient diagnostic vignettes and extended to 24 clinical entities plus a final diagnosis field. The schema is organized into four groups mentioned in Appendix E.2.

We implement a two-stage annotation procedure to balance scale and clinical faithfulness. First, we use GPT-4o to extract information and map each vignette into the schema. The model reads the vignette and populates as many slots as are explicitly supported by the text, copying short phrases rather than paraphrasing (e.g., “right upper quadrant pain” or “temperature is 38.3°C”). Missing information is marked as NA rather than hallucinated. Second, to ensure dataset correctness and high quality, three resident physicians manually revise the converted records. We partition the dataset across the three annotators, and each annotator compares the extracted slot values against the original vignette, correcting typos and formatting, adding any clinically relevant information missed by GPT-4o, and removing any unsupported or hallucinated content. In an additional normalization pass, we enforce cross-case consistency (e.g., units, spelling, and slot boundaries) and schema constraints such as non-empty Symptom-Name for all records (annotator instructions in Appendix G). This step also assigns each case to one of the 17 specialties used in our analyses. The full schema template, including clinical entity names and descriptions, is given in Appendix E. The resulting structured records serve as the ground truth for both dialogue generation and evaluation, and are the only source of clinical truth used throughout our benchmark. Examples of full records are provided in Appendix E.4.

Automatic Difficulty Estimation

We assign each case a difficulty level using a model-based proxy grounded in prior findings that (i) longer reasoning traces correlate with harder problems and (ii) model uncertainty/disagreement can reflect item difficulty ([?, ?]). For each question qiq_i, we run an ensemble of K=4K=4 reasoning LLMs with a shared structured prompt and sample each model N=6N=6 times. From these runs, we compute three signals: (1) average chain-of-thought length LiL_i (number of steps), (2) semantic answer disagreement HiH_i estimated via accuracy over the NN final answers per LLM, and (3) a confidence proxy CiC_i that combines answer frequency and self-reported confidence within exact-match answer clusters.

We min-max normalize (Li,Hi,Ci)(L_i, H_i, C_i) across questions and define a continuous difficulty score

di=L~i+H~i+(1−C~i)3.(1)d_i = \frac{\widetilde{L}_i + \widetilde{H}_i + (1-\widetilde{C}_i)}{3}. \tag*{(1)}

Finally, we cluster {di}\{d_i\} with 1D kk-means (k=3k=3) and map clusters by increasing mean did_i to Easy, Medium, and Hard. Full model/prompt details are in Appendix F; the labels are further validated against blinded physician annotations, reaching 88.9% exact agreement with physician majority labels (weighted κ=0.88\kappa=0.88; Appendix F.3).

Selecting the Patient Simulator

To isolate doctor-side effects in our evaluations, we fix a single patient-simulator model for all experiments. We compare three candidates (GPT-4o, Claude, DeepSeek-Chat) using a controlled record-grounding test: for each candidate, we generate 100 dialogues against a fixed doctor and score each doctor question as Hit/Miss under three rules (answerable → note-consistent answer; unanswerable → explicit “don’t know”; symptom absent from the note → “No”). We repeat this comparison under three fixed doctor models (DeepSeek-Chat, GPT-4o, Claude). DeepSeek-Chat achieves the highest or near-highest adherence across settings, so we use DeepSeek-Chat as the patient simulator in all subsequent experiments. See Appendix H for full results. To verify that results are not driven by the simulator choice, we fix the doctor to GPT-4o and vary only the patient model on 100 stratified cases. Diagnosis accuracy is similar across simulators (within 2 pp), indicating low sensitivity to the patient model under our record-grounded protocol (Appendix H).

Patient Personas

To isolate the effect of patient communication on diagnostic performance, we generate controlled patient personas that vary only in how the same clinical facts are expressed. Personas span four axes: education/health literacy, emotional tone, language proficiency, and occupational framing. For each clinical case, all personas share an identical underlying structured note; only linguistic style and interaction behavior change. This enables controlled analysis of robustness and fairness (e.g., whether doctor models perform differently for low-literacy or non-native patients). Full persona definitions and examples are provided in Appendix I.

Dialogue Generation Pipeline

Given a canonical structured record and a patient persona, we synthesize multi-turn doctor-patient consultations using two LLM agents (doctor and patient) coordinated by a lightweight session controller. Figure 1 provides an overview of the pipeline. Details and prompts are in Appendix I.

From static vignettes to interactive consultations

Figure 1. From static vignettes to interactive consultations. A board-exam MCQ (left) is normalized into a 24-slot clinical schema (center) and used to generate grounded multi-turn dialogues (right). This design decouples clinical content from surface realization, enabling controlled evaluation of how patient communication style affects diagnostic accuracy on identical underlying cases.

Doctor agent. The doctor agent is prompted to conduct a realistic history-taking interview by asking focused follow-up questions, while avoiding treatment recommendations and not revealing access to the underlying record. To preserve the interactive setting, the doctor is given only a brief chart blurb derived from the case (e.g., exam/notes), rather than the full structured record.

Patient agent. The patient agent is prompted with (i) a persona description and (ii) a redacted case report constructed from the structured record. The ground_truth_diagnosis field is withheld to prevent label leakage. The patient answers strictly based on this report and explicitly indicates uncertainty (e.g., “I am not sure”) when the doctor asks about missing information.

Termination and provisional diagnosis. Each consultation runs for up to a maximum number of turns. The doctor may diagnose earlier or continue until the limit. At termination, the doctor outputs a single-line provisional diagnosis; we store this line and the full transcript for evaluation.

Evaluation Methods

Single-turn baseline on original questions. To contextualize multi-turn performance, we first evaluate each model on the original datasets in a single-turn setting. We remove multiple-choice options and prompt the model with only the vignette and question, requiring a free-form diagnosis. Because diagnoses may have non-identical surface forms, we use an automatic LLM judge to map each response to a binary correct/incorrect label (Appendix J.1) and report accuracy per model.

Single-turn from structured records. To verify that our canonical schema preserves diagnostic signal and to measure robustness to input format, we also evaluate models on the structured record (omitting ground_truth_diagnosis). Models again output a free-form diagnosis, which is scored with the same judge protocol. Comparing this accuracy to the vignette-based baseline quantifies information retention and sensitivity to the narrative-to-structured shift.

Locked validation of LLM judges. Since we rely on an LLM judge for diagnosis correctness and other dialogue-level metrics, we validate it with a locked protocol. We stratified-sampled 300 dialogues and obtained independent labels from three resident physicians; majority vote serves as the reference and we report inter-rater reliability and judge agreement. We finalized the judging rubric/prompt on a 100-dialogue development split and then froze it before evaluation on a held-out 200-dialogue split to mitigate prompt overfitting. On the held-out split, GPT-4o achieved the highest agreement with the human majority-vote labels for diagnosis correctness (approximately 0.88–0.90 accuracy; κ≈0.70\kappa\approx0.70–0.80), outperforming alternative judge models. Finally, all structured case records used for dialogue generation are human-verified.

Experiment Results

Table 2 summarizes multi-turn performance on MEDRoundsQA under a 20-turn budget. Across models, interactive diagnosis is roughly half of single-turn performance on the standardized record: a drop of 13–39 points. Question relevance stays high (90–99%), so models mostly stay on topic, but they gather little structured information (slot coverage 10–37%) and stop early in most cases (45–100% of dialogues). Behavioral patterns do not translate into better interactive diagnosis. Kimi-K2.5, for example, asks relatively few questions (low slot coverage, 14%) yet achieves the best multi-turn accuracy (42%). HuatuoGPT covers the most slots (37%) and achieves strong multi-turn accuracy (36%), but still terminates early 96% of the time. MiniMax-M2.5 uses even fewer turns than Kimi-K2.5 (avg. 10.77) and achieves a moderate 36% accuracy despite one of the lowest slot coverages of any model (11.6%). Simply asking more questions or filling more fields rarely helps; models struggle to identify and act on the right discriminative findings.

ModelTurnsDx(ST) (%)Dx(MT) (%)Drop↓SlotCov (%)RelAcc (%)EarlyTerm (%)
Closed-source
GPT-4o ([30])13.5767.47 ±2.0231.55 ±2.0935.9210.2097.2690.70
DeepSeek-Chat ([13])16.6866.10 ±2.0527.14 ±1.9838.9626.9993.9371.46
Gemini-2.5-Flash ([12])15.9265.75 ±2.1326.68 ±1.9539.0725.7496.4069.76
Claude Sonnet 4.5 ([4])18.3063.50 ±2.1024.66 ±1.9538.8430.0097.9644.86
Seed-2.0-Mini ([9])10.9552.40 ±2.2225.56 ±1.9426.8420.3992.1255.08
Open-source
Llama3-UltraMed15.5950.00 ±2.2022.21 ±1.8427.7919.8189.7963.03
HuatuoGPT ([10])10.1660.62 ±2.1635.85 ±2.1324.7737.2389.9596.40
Qwen3-VL-32B-Instruct ([32])16.2063.00 ±2.0628.77 ±2.0334.2318.5098.5099.70
Baichuan-M2-32B ([6])17.6048.00 ±2.1819.86 ±1.9028.1422.0096.8890.10
Llama3-OpenBioLLM-8B ([3])15.5045.00 ±2.2416.78 ±1.8828.2220.0096.6869.70
Phi-4 ([1])11.3333.22 ±2.1019.78 ±1.7713.4423.9693.2484.97
Ministral3-8B ([29])10.6836.30 ±2.1419.92 ±1.7816.3819.9392.3872.08
Kimi-K2.5 ([21])13.7372.00 ±1.9841.92 ±2.1230.0814.0097.9388.01
MiniMax-M2.5 ([28])10.7760.27 ±2.1836.40 ±2.1423.8711.6097.2389.44
GLM-514.0964.73 ±2.1331.21 ±2.0633.5228.1997.4466.92

Table 2. Main results on MEDROUNDSQA. Values are percentages except Turns. Dx(ST) is single-turn accuracy using the standardized record-style input; Dx(MT) is multi-turn diagnostic accuracy. Drop = Dx(ST) − Dx(MT) (higher indicates a larger degradation under multi-turn interaction).

Evaluation metrics. Let yiy_i be the gold diagnosis for case ii, and let y^iST\hat{y}_i^{\mathrm{ST}}, y^iMT\hat{y}_i^{\mathrm{MT}} denote the singleturn and multi-turn predictions. Dx(ST/MT) = 1N∑i=1N1[y^i(⋅)=yi]\frac{1}{N}\sum_{i=1}^{N}\mathbb{1}[\hat{y}_{i}^{(\cdot)}=y_i] measures diagnosis accuracy under each setting; Drop = Dx(ST) − Dx(MT) quantifies degradation under interaction. Let SiS_i be the set of non-empty gold schema slots for case ii, and let ei,s=1[∃t:elicits_slot⁡(qi,t,ai,t)=s]e_{i,s}=\mathbb{1}[\exists t:\operatorname{elicits\_slot}(q_{i,t},a_{i,t})=s]; then SlotCov = 1N∑i1∣Si∣∑s∈Siei,s\frac{1}{N}\sum_i\frac{1}{|S_i|}\sum_{s\in S_i}e_{i,s} measures the fraction of available clinical information elicited. RelAcc = 1N∑i1Ti∑t=1Tiri,t\frac{1}{N}\sum_i\frac{1}{T_i}\sum_{t=1}^{T_i}r_{i,t} is the fraction of doctor questions judged relevant or grounded in the record by an LLM judge, and EarlyTerm = 1N∑i1[Ti<Tmax⁡]\frac{1}{N}\sum_i\mathbb{1}[T_i<T_{\max}] is the rate of stopping before the turn budget. Full formal definitions and judge prompts are in Appendices C and J.

stripping MCQs cuts performance by 10-26 points while further standardizing the input only costs a few additional points, echoing findings that LLMs often solve MedQA-style benchmarks via option elimination and pattern matching rather than full case understanding (e.g., on MedQA and MedMCQA). This means multi-turn results should be compared to the open-ended, standardized settings, not MCQ scores, which substantially overstate a model’s diagnostic ability in realistic, non-MCQ clinical workflows.

Impact of Question Format

To disentangle multi-turn effects from input-format scaffolding, we evaluate models under three settings when supported by the source data: +MCQ (with answer options), No MCQ (open-ended diagnosis), and a standardized record-style Standard Format (Figure 2). Across models, most of the “accuracy” comes from exploiting answer choices rather than doing genuine open-ended reasoning:

Effect of format on diagnostic accuracy

Figure 2. Effect of format on diagnostic accuracy. We compare the multiple-choice setting (+MCQ), the same cases without answer options (No MCQ), and a standardized record-style representation (Standard Format).

Can LLMs Match Human Performance?

To ground our interactive evaluation in a human clinical baseline, we compare LLM “doctor” agents to resident physicians using the same 20-turn simulated-patient interface on 60 difficult cases (Table 3), and find that resident physicians nearly double LLM diagnostic accuracy (48% vs. 27%) while using 40% fewer turns, echoing prior findings that clinicians still outperform frontier models on non-MCQ, case-based diagnosis despite strong results on static benchmarks (e.g., Med-PaLM and GPT-4 evaluations). The most “efficient” model actually underperforms, stopping after 5 turns with only 25% accuracy, highlighting that current LLMs tend to terminate prematurely rather than strategically probing for high-yield clinical evidence. Qualitatively, human doctors reliably ask targeted, discriminative questions (e.g., orthopnea/paroxysmal nocturnal dyspnea in Figure 3), whereas LLMs default to generic screening queries and are easily steered off-course by low-literacy or misleading personas, leading to under-specified or incorrect diagnoses even when key clues are present.

Model/PhysicianCorrect (n/60)Acc (%)Avg. turnsMin. turns
AI models
GPT-4o17/6028.3311.636
DeepSeek15/6025.0016.008
Gemini-2.5-Flash19/6031.6715.178
Llama3-UltraMed-70B14/6023.3313.556
HuatuoGPT15/6025.005.274
AI average16.0/6026.6712.326.4
Human physicians
Physician 129/6048.337.692
Physician 231/6051.678.506
Physician 327/6045.005.592
Human average29/6048.337.263.3
Perf. gap (Human - AI)+13.0+21.66 p-5.06-3.1

Table 3. Human vs. LLM diagnostic performance on 60 challenging cases using a shared simulated patient.

Persona-driven elicitation failure

Figure 3. Persona-driven elicitation failure. Holding clinical content constant, an informative persona foregrounds high-yield details that trigger targeted questioning and correct diagnosis (CHF), while a misleading persona yields vague framing that leads to generic questioning/anchoring and an incorrect or underspecified diagnosis.

Our findings align with recent clinical evidence: in a randomized controlled study with 1,298 participants, [7] found that LLMs identified conditions in 94.9% of medical scenarios when tested alone, yet participants assisted by the same models succeeded in fewer than 34.5% of cases–a collapse under interaction consistent with the ST→MT drops we observe, and one that motivates controlled interactive benchmarks as a scalable evaluation layer before costly human studies.

Performance Across Difficulty Bands

Table 4 stratifies diagnostic accuracy by Easy/Medium/Hard cases using our model-based difficulty scores, with DeepSeek as the fixed patient simulator and a 12-turn budget. The difficulty signal transfers cleanly to the interactive setting: for every model, accuracy declines monotonically as cases get harder. Qualitatively, Easy cases tend to be ones where a single discriminative clue (e.g. classic risk-factor constellation, a pathognomonic exam finding) is present and, once elicited, most models converge on the same diagnosis within a few turns. Hard cases, by contrast, often require integrating multiple weak signals or reconciling competing explanations (e.g., overlapping cardiopulmonary or rheumatologic presentations); in these settings, we observe the behaviors highlighted in our error taxonomy: models hedge (“could be X or Y”), default to syndrome-level labels, or terminate without ever asking the one or two questions that human annotators considered decisive.

ModelEasyMediumHard
GPT-4o38.6822.4610.07
Gemini-2.5-Flash43.1525.3114.99
DeepSeek-Chat37.9323.3912.49
Claude Sonnet 4.532.1018.458.21
Kimi-K2.554.2031.8518.43
MiniMax-M2.547.3027.6014.82
GLM-540.5523.8911.34
Seed-2.0-Mini33.2518.828.97
Llama3-UltraMedical30.1017.795.20
HuatuoGPT28.1016.115.31
Qwen3-VL-32B-Instruct37.2021.439.88
Baichuan-M2-32B25.9014.725.01
Llama3-OpenBioLLM-8B22.1512.384.19
Phi-426.4014.955.88
Ministral3-8B26.8515.215.62

Table 4. Diagnostic accuracy (%) stratified by difficulty band (Easy/Medium/Hard), using DeepSeek-Chat as the fixed patient simulator with a 12-turn budget.

How Much Dialogue Is Enough?

Varying the maximum consultation length shows that models quickly learn to stay on-topic: question relevance rises steeply with more turns and then saturates, yet diagnostic accuracy barely improves beyond a short interview (see Figure 4). In most cases, additional turns produce more of the same low-yield questions rather than sharper hypotheses or better synthesis of gathered clues (e.g. Figure 3); only HuatuoGPT meaningfully benefits from longer budgets, suggesting that the central bottleneck is not talking longer but turning incremental history into a decisive diagnostic update.

Diagnosis accuracy and question relevance as a function of maximum dialogue turns

Figure 4. Diagnosis accuracy and question relevance as a function of maximum dialogue turns.

Persona Analysis

We probe robustness to how patients communicate by varying education, language fluency (including accent), occupation, and affect while holding clinical content fixed. Education (health literacy) is the strongest driver: all models diagnose markedly better as histories become more organized and medically precise, producing a near-monotonic rise from the lowest to highest education levels (Figure 5). Reduced language proficiency and, to a lesser extent, accented or non-native phrasing further depress accuracy, and blue-collar occupational framing also lowers performance, whereas affect has only modest, model-dependent effects, suggesting that surface clarity matters more than emotional tone. For example, the persona comparison in Figure 3 shows how a low-literacy patient who talks

Diagnostic accuracy by education (health-literacy) persona

Figure 5. Diagnostic accuracy by education (health-literacy) persona. Accuracy increases monotonically with education level across all models.

about “eating too much salt” leads the model to generic gastrointestinal labels, while a high-literacy version of the same case foregrounds rapid weight gain and edema and reliably elicits the key orthopnea/PND question, underscoring that many errors arise from failures to extract and integrate high-yield information from noisy narratives. Figure 6 shows that stronger language skills are associated with higher accuracy. See more detailed results in Appendix L.3.

Diagnostic accuracy by language-proficiency persona

Figure 6. Diagnostic accuracy by language-proficiency persona. Lower fluency (broken language) yields substantial performance drops.

Models differ primarily in frequency rather than the presence/absence of distinct error types.

Why LLMs Fail Interactive Diagnosis?

We manually annotate a sample of 943 incorrect multi-turn consultations drawn from the GPT-4o, DeepSeek-Chat, and Gemini-2.5-Flash runs, using a shared taxonomy, finding four recurring failure modes (Table 5). Protocol breakdowns (9%) occur when the model never produces a usable diagnosis. Uncertainty-as-output (40%), the most prevalent mode, occurs when models hedge or retreat into generic workup language even after sufficient evidence has been elicited. Insufficient elicitation (30%) captures cases where models never ask the few high-yield questions human reviewers considered decisive (e.g., orthopnea/PND for heart failure, specific exposures for infectious presentations, family history for hereditary conditions, or orthostatic vitals for volume status). Syndrome-level collapse (13%) occurs when models settle on broad labels despite the case supporting a more specific entity. Crucially, error-type distributions are near-identical across GPT-4o, DeepSeek, and Gemini—stronger models fall into them less often but not less consistently—indicating that scaling has not resolved the core weaknesses: knowing when to commit, identifying discriminative questions, and synthesizing partial histories into granular diagnoses. These patterns are invisible to single-turn evaluation, where the same models score 50–67% on standardized records (Table 2), but emerge clearly under interactive conditions. The low-literacy persona analysis (Section 4.5) further shows that communication noise amplifies all four failure modes, with anchoring and insufficient elicitation rising sharply when patients use vague or misleading symptom descriptions. The clinician audit (Appendix M) confirms: successful dialogues show structured history-taking and differential-aware consolidation (61%, 48%), while failed ones exhibit repetition and anchoring (57%, 49%), pointing to a shared bottleneck in strategic clinical reasoning. Representative error cases with clinical commentary are in Appendix N.

Error TypeOverallGPT-4oDeepSeekGemini
Protocol breakdown9%6%11%8%
Uncertainty-as-output40%35%48%42%
Syndrome-level collapse13%10%15%14%
Insufficient elicitation30%28%35%30%

Table 5. Distribution of error types across models. Percentages are the fraction of incorrect predictions (943 cases) exhibiting each pattern (multi-labeling allowed).

Conclusion

MedRoundsQA converts exam vignettes into controlled multi-turn consultations, allowing direct comparison of single-turn and interactive diagnosis on identical cases. Across fifteen LLMs, multi-turn evaluation reveals weaknesses missed by single-turn scores: accuracy falls sharply from records to consultations, plateaus after about 6–12 turns despite more relevant questions, and varies systematically with patient communication features including education, language fluency, and occupational framing. In matched simulations, resident physicians nearly double model accuracy using fewer turns, while error analysis shows models often stop early, hedge rather than commit, and miss key discriminative findings. Improving medical LLMs requires stronger knowledge plus better targeted questioning and diverse communication styles.

Limitations and Future Work

We focus on text-based, single-encounter consultations and do not model longitudinal records or imaging; these are natural extensions for future work. Our results suggest that moving beyond single-shot vignettes to controlled multi-turn settings is both feasible and necessary if we want to understand how LLMs behave in the kinds of consultations clinicians and patients are beginning to conduct with AI tools today.

Protocol design and setup-induced effects. Three protocol choices could in principle shape the observed failures. The brief chart blurb mirrors the information state of a new-patient encounter ([34]); since Dx(ST) (Table 2) measures the same cases under full information access, the ST→MT gap isolates the interactive component. The turn budget is not binding: accuracy plateaus well before 20 turns (Section 4.4). And under identical prompts and decoding, early-termination rates still vary from 44.9% to 99.7% (Table 2), while resident physicians on the same interface nearly doubled model accuracy with fewer turns (Section 4.2). The observed weaknesses are thus primarily model-intrinsic, though alternative blurb contents, adaptive budgets, and prompt variations merit future study.

Beyond protocol design, while the consistency of failure-mode distributions across our 15 models spanning 8B open medical models to frontier systems suggests a bottleneck not resolved by scale alone (Section 5), a controlled within-family scaling analysis, comparing models of different sizes from the same family under identical protocols, would provide a cleaner test of this hypothesis and is a concrete direction for future work.

Ethical Statement

We introduce MedRoundsQA, a benchmark for evaluating medical reasoning in multi-turn simulated patient–clinician dialogues built from publicly available, de-identified cases converted into structured records and synthetic conversations. The benchmark does not involve collecting new patient data, interacting with real patients, or conducting any human-subjects intervention.

Because the setting is medical, outputs or examples could be misconstrued as clinical advice. Our dataset and evaluation are intended solely for research on model behavior and robustness and are not a substitute for professional diagnosis or decision-making. We also evaluate controlled persona variations (e.g., language proficiency, education, affect), which may reveal performance disparities; we avoid targeting protected attributes and report these effects to encourage safer and more equitable medical dialogue systems.

The data used in this study were obtained exclusively from publicly available sources. No new data were collected directly from individuals. The original data were released under terms that permit research use. Our work involves creating a derived dataset through processing and annotation of this public data. As such, informed consent was handled at the time of the original data collection by the respective data providers, and no additional consent was required for this study.

Large language models were used in a limited capacity to assist with minor editing and polishing of the manuscript, such as improving clarity and grammar. All technical content, experimental design, results, and conclusions were produced, verified, and finalized by the authors.

The physician annotators were recruited through the authors’ academic and clinical networks on the basis of their medical training and relevant clinical expertise. General practitioners and resident physicians participated in dataset filtering, structured-record verification, difficulty-label validation, and dialogue-quality evaluation. All annotators provided informed consent and were compensated at a fixed rate of 5 USD per hour, which substantially exceeds the typical effective hourly earnings of resident physicians in Egypt. No crowdsourcing platforms or members of the general public were involved.

Appendices

This appendix provides implementation details, prompts, and supplementary materials for MEDROUNDSQA.

Data Collection and Filtering

Filtering Prompt

We use the following prompt to automatically filter medical questions for suitability:

You are helping to filter a dataset of medical multiple-choice questions.
Your task is to decide whether the question is a *clinical vignette that asks for a diagnosis* and provides enough information about the patient to make a diagnosis.
A question qualifies if:
- It presents a clinical scenario or patient vignette, and
- The main goal is to identify the most likely diagnosis for the patient.
If the question meets both criteria, respond with: YES
Otherwise, respond with: NO
Respond with a single word only: YES or NO.
QUESTION:
”question text”

Dataset Construction and Extraction Details

GP reference annotations for extractor selection

We recruit a general practitioner (GP) to convert 100 development questions into the 24-slot schema, extracting only information explicitly stated in the vignette and marking missing information as NA. This produces slot-level reference records used to select an automatic extractor.

Extractor evaluation protocol and metrics

For each candidate extractor (GPT-4o, DeepSeek, Claude), we generate a structured record for each of the 100 development questions using the same slot definitions provided to the GP. We score outputs in a slot-aware manner: each predicted slot value is compared to the corresponding GP-annotated slot value and scores are micro-averaged across all slots, penalizing slot misplacements.

Data Extractor Selection

We report both lexical and semantic similarity metrics, including normalized Levenshtein similarity, Jaccard word overlap, BERTScore-F1 (implemented via bert-score), and cosine similarity of supervised SimCSE embeddings (sup-simcse-bert-base-uncased). We also include an LLM-as-a-judge adequacy score using claude-sonnet-4-20250514 with a 0.90 threshold. Results are shown in the Figure 7.

Answer normalization

We map each MCQ answer key to its corresponding option text and normalize it into a canonical ground_truth_diagnosis (e.g., resolving letter indices and standardizing minor surface-form variation).

Metrics and Formal Definitions

We formally define all metrics reported in Table 2. Let the dataset be indexed as D={1,…,N}\mathcal{D} = \{1,\ldots,N\}. For each case i∈Di \in\mathcal{D}, let the gold diagnosis be yiy_i, and let y^iST\hat{y}_i^{ST} and y^iMT\hat{y}_i^{MT} denote the single-turn and multi-turn predictions, respectively. Let the multi-turn dialogue have length TiT_i with questions qi,tq_{i,t} and answers ai,ta_{i,t} for t∈{1,…,Ti}t \in\{1,\ldots,T_i\}. Let Tmax⁡T_{\max} be the maximum turn budget, and let Si\mathcal{S}_i be the set of gold non-empty slots for case ii. Let K[P]=1\mathbb{K}[P] = 1 if predicate PP holds and 0 otherwise.

Dx(ST).

Dx(ST)=1N∑i=1NK[y^iST=yi].(2)\mathrm{Dx}(\mathrm{ST}) = \frac{1}{N}\sum_{i=1}^{N} \mathbb{K}\left[\hat{y}_i^{ST} = y_i\right]. \tag*{(2)}

Dx(MT).

Dx(MT)=1N∑i=1NK[y^iMT=yi].(3)\mathrm{Dx}(\mathrm{MT}) = \frac{1}{N}\sum_{i=1}^{N} \mathbb{K}\left[\hat{y}_i^{MT} = y_i\right]. \tag*{(3)}

Drop.

Drop=Dx(ST)−Dx(MT).(4)\mathrm{Drop} = \mathrm{Dx}(\mathrm{ST}) - \mathrm{Dx}(\mathrm{MT}). \tag*{(4)}

SlotCov. Define

ei,s=1[∃t≤Ti:elicits_slot⁡(qi,t,ai,t)=s].(5)e_{i,s} = \mathbb{1}[\exists t \leq T_i : \operatorname{elicits\_slot}(q_{i,t}, a_{i,t}) = s]. \tag*{(5)}

Then

SlotCov⁡(i)=1∣Si∣∑s∈Siei,s.(6)\operatorname{SlotCov}(i) = \frac{1}{|S_i|} \sum_{s \in S_i} e_{i,s}. \tag*{(6)}
SlotCov⁡=1N∑i=1NSlotCov⁡(i).(7)\operatorname{SlotCov} = \frac{1}{N} \sum_{i=1}^{N} \operatorname{SlotCov}(i). \tag*{(7)}

RelAcc. Define

ri,t=1[qi,t is relevant/grounded],(8)r_{i,t} = \mathbb{1}[q_{i,t}\text{ is relevant/grounded}], \tag*{(8)}

where relevance/grounding is determined by an LLM judge with a fixed rubric. Then

RelAcc⁡(i)=1Ti∑t=1Tiri,t.(9)\operatorname{RelAcc}(i) = \frac{1}{T_i} \sum_{t=1}^{T_i} r_{i,t}. \tag*{(9)}
RelAcc⁡=1N∑i=1NRelAcc⁡(i).(10)\operatorname{RelAcc} = \frac{1}{N} \sum_{i=1}^{N} \operatorname{RelAcc}(i). \tag*{(10)}

EarlyTerm.

EarlyTerm⁡(i)=1[Ti<Tmax⁡].(11)\operatorname{EarlyTerm}(i) = \mathbb{1}[T_i < T_{\max}]. \tag*{(11)}
EarlyTerm⁡=1N∑i=1NEarlyTerm⁡(i).(12)\operatorname{EarlyTerm} = \frac{1}{N} \sum_{i=1}^{N} \operatorname{EarlyTerm}(i). \tag*{(12)}

Dataset Statistics

Figure 8 shows the distribution of medical specialties in each subset of the dataset. Internal Medicine is the largest specialty in all three sources, and NEJM contributes the broadest specialty coverage (16 of the 17 specialties), while several specialties (e.g., Rheumatology, Cardiology) contain fewer than ten cases each consistent with the small-sample caution for such specialties in Appendix L.1.

Specialty distribution bar charts for the MedQA, NEJM, and MedMCQA dataset subsets

Figure 8. Specialty distribution for each dataset subset.

Clinical Record Schema

Schema Definition

Each clinical case is represented using a structured 24-slot schema organized into four semantic groups. Table 6 provides complete slot definitions.

Slot NameDefinition
Symptom Descriptors
Symptom-NamePrimary symptoms reported by the patient in lay (non-medical) terms. Avoid disease names; describe only what a non-professional would notice.
Symptom-TimeOnset time, duration, and course of the symptoms (e.g., acute, chronic, intermittent).
Symptom-PositionAnatomical location of the symptoms.
Symptom-DegreeSeverity or intensity of the symptoms (e.g., mild, moderate, severe).
Symptom-ColorColor features of the symptoms (e.g., redness, cyanosis, jaundice).
Symptom-ShapeShape or form of visible findings (e.g., patches, nodules, swelling).
Symptom-TextureQualitative characteristics of the symptoms (e.g., thickness of phlegm, firmness of a lump).
Symptom-SmellOdor related to the symptoms (e.g., foul-smelling discharge, fruity breath).
Physical Changes & Examination
Physical-ChangeNoticeable changes in physical state such as weight, appetite, energy level, or sleep pattern.
Medical-ExaminationFindings from physical examination and additional tests (e.g., CT, MRI, X-ray, laboratory tests, ECG).
History & Context
Personal-AgePatient’s age.
Personal-SexPatient’s sex/gender as recorded in the medical context.
Personal-WeightPatient’s body weight (and height if available).
Past-DiagnosisRecord of previous medical diagnoses, including major past illnesses.
Past-MedicationMedications the patient is currently taking or has recently taken on a regular basis.
Trauma-SurgeryHistory of significant trauma, injuries, or surgical procedures.
Allergic-HistoryHistory of allergies to foods, medications, or other substances, and typical reactions.
PreventivePreventive measures and vaccination status, including recent or missing vaccines.
Family-HistoryFamily medical history, including major illnesses, hereditary or genetic diseases.
Contact-HistoryRecent contact with infectious diseases, sick individuals, animals, or high-risk environments.
Habit-TobaccoTobacco use, including type, amount, and duration.
Habit-WineAlcohol use (wine and other alcoholic beverages), including frequency and amount.
Habit-DrugUse or abuse of illicit drugs or non-prescribed psychoactive substances.
Habit-LivingLifestyle habits such as diet, exercise, hygiene, work pattern, and sleep routine.
Coitus-HistoryHistory of sexual activity, especially unprotected intercourse or multiple partners when relevant.
Receiving-TreatmentWhether the patient is currently receiving treatment and a brief description of that treatment.
Meta-Information
DepartmentClinical department or specialty where the patient is being evaluated (e.g., cardiology, pediatrics).
additional_notesAny other relevant patient information or medication details not covered by the above categories.
ground_truth_diagnosisDefinitive diagnosis provided by a qualified medical professional after evaluation.

Table 6. Complete schema specification. Each slot captures a specific aspect of the clinical presentation, history, or examination findings.

Clinical entities groups

The 24 slots are organized into four semantic groups. Table 7 summarizes each group, its member slots, and its role in representing a case: symptom descriptors capture the presenting complaint and its attributes, physical changes and examination findings encode objective measurements, history and context provide background risk factors, and meta-information stores the specialty label and the canonical diagnosis used for evaluation.

GroupExample slotsRole in case representation
Symptom descriptorsSymptom-Name, Symptom-Time, Symptom-Position, Symptom-Degree, Symptom-Color, Symptom-Shape, Symptom-TextureCapture the core complaint and its attributes (what the patient feels, when it started, where it is, and how severe or distinctive it is).
Physical changes and examination findingsPhysical-Change, Medical-ExaminationEncode measurable findings such as vital signs, focused physical examination, and key laboratory or imaging results.
History and contextPersonal-Age, Personal-Sex, Personal-Weight, Past-Diagnosis, Past-Medication, Trauma-Surgery, Allergic-History, Preventive, Family-History, Contact-History, Habit-Tobacco, Habit-Wine, Habit-Drug, Habit-Living, Coitus-HistoryProvide background risk factors, lifestyle factors, and longitudinal information that modulate prior probability of different diagnoses.
Meta-informationDepartment, additional_notes, ground_truth_diagnosisStore specialty classification, brief free-text clarifications, and the canonical diagnosis used as the evaluation label.

Table 7. Overview of the 24-slot clinical schema used to represent each clinical case. Clinical entities are grouped into four categories that separate presenting symptoms, objective findings, history, and meta-information.

Frequency of Clinical Entity Groups

We further quantify annotation coverage by measuring, for each slot, the percentage of cases with a non-NA value, and then averaging these slot-level completeness scores within each group. Figure 9 summarizes the resulting mean slot completeness per group. We note that this distribution is inherently imbalanced: not every slot is clinically informative for every vignette, and our schema records only information that is supported by the original question and potentially useful for reaching the correct diagnosis. As a result, groups such as history and context may appear sparser, even when the underlying cases are well-specified for diagnosis.

Mean slot completeness bar chart for the four clinical schema groups

Figure 9. Mean slot completeness (% non-NA) aggregated by schema group.

Example of a record in a standard format

Below is a complete example of a case represented in our standardized format:

{
  "record": {
    "Symptom-Name": "increasing shortness
    of breath, dry cough",
    "Past-Diagnosis": "NA",
    "Past-Medication": [],
    "Physical-Change": "difficulty
    walking, puffy and red face,
    pitting edema of the legs",
    "Trauma-Surgery": "NA",
    "Preventive": "NA",
    "Allergic-History": "NA",
    "Contact-History": "NA",
    "Habit-Tobacco": "smokes 1.5 packs of
    cigarettes a day",
    "Habit-Wine": "drinks about four
    bottles of beer a day",
" Habit-Drug": "NA",
" Habit-Living": "NA",
" Coitus-History": "NA",
" Family-History": "NA",
" Personal-Age": "55",
" Personal-Weight": "NA",
" Personal-Sex": "male",
" Medical-Examination": "chest X-ray
  discloses hyperinflation,
  flattening of the diaphragm,
  increased retrosternal air space",
" Department": "NA",
" Receiving-Treatment": "NA",
" Symptom-Degree": "breathless after
  only a few steps",
" Symptom-Position": "chest, legs
  (pitting edema)",
" Symptom-Time": "past few years",
" Symptom-Shape": "barrel chest",
" Symptom-Texture": "NA",
" Symptom-Smell": "NA",
" Symptom-Color": "NA",
" additional_notes": "Prolonged
  expiration with wheezing,
  hyperresonance on percussion,
  clubbing of the digits",
" ground_truth_diagnosis": "Emphysema"
}
}

Data difficulty classification

Models and prompting. We use an ensemble of K=4K=4 reasoning LLMs: DEEPSEEK-R1, QWEN/QWQ-32B, PHI-4-REASONING, and OLMO-3.1-32B-THINK. Each model is queried with the same structured prompt (Appendix F.1) and sampled N=6N=6 times per question.

Signals. For each question qiq_i and model MkM_k, we collect NN runs producing a reasoning trace, final answer, and self-reported confidence c∈[0,1]c \in[0,1]. We compute: (i) mean CoT length LikL_{ik} (steps), (ii) semantic disagreement Hik∈[0,1]H_{ik} \in[0,1] by sending the set of NN final answers to a judge model, and (iii) a cluster-based confidence proxy. We canonicalize answers (lowercasing, whitespace normalization), form exact-match clusters, and for each cluster aa compute frequency faf_a and mean confidence cˉa\bar{c}_a. We define Sa=12(fa+cˉa)S_a = \frac{1}{2}(f_a+\bar{c}_a) and set CikC_{ik} to the mean SaS_a over runs. We average across models to obtain LiL_i, HiH_i, CiC_i.

Difficulty score and binning. We min–max normalize LiL_i, HiH_i, CiC_i across all questions to get L~i\tilde{L}_i, H~i\tilde{H}_i, C~i\tilde{C}_i and compute

di=L~i+H~i+(1−C~i)3.(13)d_i = \frac{\tilde{L}_i+\tilde{H}_i+(1-\tilde{C}_i)}{3}. \tag*{(13)}

We then apply 1D kk-means (k=3k=3) to {di}\{d_i\} and label clusters as Easy/Medium/Hard by increasing mean

Difficulty Prompt

You are a medical reasoning assistant. Be concise and factual. YOU MUST OUTPUT EXACTLY ONE JSON OBJECT ON A SINGLE LINE AND NOTHING ELSE. Do not print any extra commentary, markdown, or code fences. For the question below, return EXACTLY one JSON object with keys: "cot" (array of numbered step strings), "final_answer" (1–3 words), and "confidence" (number 0–1). Do NOT output anything else.

Question:

{question_text}

Example (single-line JSON exactly like this):

{"cot":["1. Step one","2. Step two"],"final_answer":"Histoplasma capsulatum","confidence":0.87}

Data Difficulty distribution

Distribution of difficulty levels across dataset samples

Figure 10. Distribution of difficulty levels across dataset samples.

Physician Validation of Difficulty Labels

To verify that our model-based difficulty labels correspond to clinically meaningful difficulty, we conducted a blinded physician annotation study on 90 cases sampled evenly across the Easy/Medium/Hard bands. Physicians rated difficulty from the original vignettes without access to the automatic labels or any model outputs. The automatic labels achieved 88.9% exact agreement with the physician majority labels, with weighted Cohen’s κ=0.88\kappa=0.88 (linear weights) and κ=0.92\kappa=0.92 (quadratic weights). All disagreements occurred between adjacent bands, with no Easy–Hard confusions (Table 8).

Automatic label
Physician labelEasyMediumHard
Easy2730
Medium2253
Hard0228

Table 8. Confusion matrix between automatic difficulty labels and blinded physician majority labels on 90 stratified cases.

Annotator Instructions

You will be shown an original clinical vignette together with a structured clinical record automatically extracted from that vignette. Your task is to verify that the structured record accurately reflects the information explicitly stated in the original vignette.

Please follow these rules:

  1. Review each structured field against the original vignette.

  2. Correct any factual errors, typos, formatting problems, or information placed in the wrong field.

  3. Add clinically relevant information that is explicitly stated in the vignette but missing from the structured record.

  4. Remove any information that is not supported by the vignette.

  5. Do not infer, guess, or add clinical information that is not explicitly stated in the vignette.

  6. If information for a field is unavailable, retain or assign the value NA.

  7. Preserve the original clinical meaning and, where possible, use short phrases taken directly from the vignette rather than introducing new interpretations.

  8. Verify that the final diagnosis corresponds to the diagnosis provided by the original source.

If a case is ambiguous, or you are uncertain whether a piece of information is supported by the vignette, flag the case for review rather than making an unsupported assumption.

The purpose of this task is to produce a clinically faithful structured representation of each case. These records are intended for research evaluation only and should not be interpreted as real-patient medical records or clinical advice.

Patient Simulator Selection and Robustness

Selection protocol. We evaluate GPT-4o, Claude, and DeepSeek-Chat as patient simulators by generating 100 dialogues (600 turns) per candidate against a fixed doctor model. Each doctor question is scored as HIT if it follows record-grounding rules: (i) if answerable from the structured note, the patient response must be consistent with the note; (ii) if not answerable, the patient must reply with an explicit “don’t know”; (iii) if the doctor asks about a symptom absent from the note, the patient must answer “No”. Accuracy is the fraction of turns scored as HIT. We repeat the evaluation with three fixed doctor models (DeepSeek-Chat, GPT-4o, Claude). DeepSeek-Chat yields the highest or near-highest adherence across all three doctor settings (Figure 11).

Comparison of patient models under the record-grounded adherence test

Figure 11. Comparison of patient models under the record-grounded adherence test.

Robustness check. We fix the doctor to GPT-4o and vary only the patient simulator (DeepSeek-Chat, GPT-4o, Claude) using identical prompts/decoding and a redacted patient record that excludes ground_truth_diagnosis. On a stratified subset of 100 cases (25 easy/50 medium/25 hard), final diagnosis accuracy is 30.0% (DeepSeek-Chat), 31.0% (GPT-4o), and 29.0% (Claude). Paired bootstrap confidence intervals show simulator-induced differences are small (e.g., Δ\Delta(DeepSeek − GPT-4o) = −1.0pp; Δ\Delta(DeepSeek − Claude) = +1.0pp).

Dialogue Generation System Prompts

Doctor Agent Prompt

The doctor agent is instructed to conduct a focused medical history using standard clinical heuristics.

You are a kind, careful family doctor speaking in clear, friendly, everyday language.

Your role:

  • Begin the conversation yourself.

  • Just start the conversation by saying “Hello, how can I help you today?”

  • It is enough to ask them why they are here today.

  • You have not seen the patient be-

fore—act as if you know nothing about their case.

  • You have access to a brief ‘chart blurb’ with objective info (e.g., vitals, exam notes). Mimic that you tested those vital-s/exams during your questioning and got the answer.

  • MIMIC like you are a real doctor who is taking temperature for the patient to see if they have fever or not, or checking their blood pressure.

  • Ask focused, empathetic, clarifying questions to gather key medical history.

Guidelines for questioning:

  • Explore symptoms using OPQRST (onset, provocation, quality, radiation, severity, timing).

  • Screen for red flags, pregnancy, medications, allergies, past medical/surgical history, and relevant social habits.

  • Ask about personal and family history when relevant (e.g., smoking, alcohol, drug use, chronic illnesses, age, sex, weight, pregnancy status, sexual activity).

  • Ask one topic at a time—do not bundle unrelated questions.

  • Be warm, empathetic, and conversational.

Chart Blurb: [Medical-Examination field + additional notes]

Restrictions:

  • Do NOT prescribe or recommend specific medications or treatments.

  • Do NOT mention or imply that you can see the full patient report—only refer to the visible chart blurb.

  • The patient will answer based STRICTLY on a PRIVATE structured medical report you do NOT see.

Conversation end:

  • When you receive a message with [SYSTEM INSTRUCTION - FINAL TURN], you MUST stop asking questions.

  • At that point, output ONLY one line in this exact format: Provisional diagnosis: <your best working diagnosis>

  • Do NOT ask any follow-up questions in your final turn.

  • Do NOT say anything else besides the provisional diagnosis.

Patient Agent Prompt

The patient agent is instructed to respond strictly based on the structured record and selected persona.

You are a patient answering based strictly on a PRIVATE structured medical report. Follow the following persona in your answers: [patient_persona]

ABSOLUTE RULES:

  • ONLY use information present in the report below.

  • If the doctor asks for something that’s not included in the report, please naturally indicate that you are not sure about that rather than making up any information.

  • Do NOT invent, infer, or guess beyond the report.

  • Speak naturally in 2–6 sentences if you know about the case.

  • You may reveal items from the report naturally when asked or when directly relevant.

  • If you are asked about fever or blood pressure, and the vitals look normal, say yes or no according to them but do not say the numbers or make medical analysis.

  • If you were asked about a symptom or past diagnosis, or daily habit and it is not

present in the report, say “I am not sure about that”.

  • Do not talk like reading from a list; be conversational. Take a patient personality.

  • Do NOT mention information the doctor did not ask for and do not make any medical conclusions.

Here is your PRIVATE report (do NOT mention it’s a report; just answer as yourself): [report_json]

Education-Based Personas

We define six education levels that span from illiterate to medical professional, capturing variation in health literacy, vocabulary, and medical knowledge.

Level 1: Illiterate

Persona: edu_level_1_illiterate You are role-playing a patient who cannot read or write and has very limited formal education (never finished elementary school). Use very simple vocabulary and grammar, with short sentences. You cannot read forms, prescriptions, or medical instructions. You feel some embarrassment about your illiteracy and try to hide it. You rely entirely on verbal communication and memory, and you may use incorrect medical terms or folk descriptions. Example style: “I been having this pain for a while now. Can’t say exactly when - I ain’t good with dates and such. I can’t read them papers you’re giving me.”

Level 2: Elementary Education

Persona: edu_level_2_elementary You are role-playing a patient with an elementary school education only (stopped around 5th-6th grade). Use simple, everyday vocabulary with basic grammar and short sentences. You have limited medical knowledge and describe symptoms in simple terms like “stomach” or “head hurts”. You may mix up medical terms or use the wrong words. Be honest and direct, but limited in expression (typically 1-3 sentences). Example style: “My stomach’s been hurting for a few days now. It’s a sharp pain, right here. I don’t know what that word means. Can you explain it simpler?”

Level 3: High School

Persona: edu_level_3_highschool You are role-playing a patient with a high school education. Use clear, standard vocabulary with mostly proper grammar and sentence structure. You have basic medical knowledge and can describe symptoms clearly and logically. You understand common medical terms but not specialized ones. Provide practical, straightforward answers in 2-4 sentences. Example style: “I’ve been experiencing sharp pain in my abdomen for about three days. It gets worse after meals and sometimes I feel nauseous.”

Level 4: College/Bachelor’s Degree

Persona: edu_level_4_college You are role-playing a patient with a college/bachelor’s degree. Use more sophisticated vocabulary and somewhat complex sentences. You have good general medical knowledge and provide articulate, organized descriptions. You understand most common medical terminology and may ask informed questions. Responses are moderate to long (3-5 sentences) with a professional communication style. Example style: “I’ve been experiencing what I would describe as intermittent sharp pain in my upper right abdomen. The onset was approximately three days ago, and it seems to correlate with fatty meals.”

Level 5: Graduate Degree

Persona: edu_level_5_graduate You are role-playing a patient with a graduate degree (Master’s/PhD or equivalent). Use advanced vocabulary and complex sentence structures, with an analytical tone. You have good general medical knowledge and are comfortable with medical terminology. You ask detailed, informed questions and may propose hypotheses or differential diagnoses. Provide longer, well-structured responses (4-6 sentences). Example style: “I’ve been experiencing episodic right upper quadrant pain with severity of about 6-7 out of 10. The onset was acute, roughly 72 hours ago, and it appears to be exacerbated by fatty food intake. Given my family history of gallbladder disease, I’m wondering if this could be cholecystitis.”

Level 6: Medical Training

Persona: edu_level_6_medical You are role-playing a patient with medical training (e.g., physician, nurse, or equivalent healthcare professional). Use precise medical terminology naturally and provide clinical-style descriptions including measurements. You have a high level of medical knowledge and may self-diagnose or suggest differentials. Ask colleague-level questions. Responses are detailed (5-8 sentences) and sound like a respectful peer-to-peer discussion. Example style: “I’ve been experiencing acute onset right upper quadrant pain, 7/10 severity, with radiation to the right shoulder. Associated symptoms include nausea, one episode of non-bloody emesis, and mild fever. Given my family history of cholelithiasis, I’m concerned about acute cholecystitis versus choledocholithiasis.”

Emotional State Personas

Anxious

Persona: emotion_anxious You are role-playing a patient who is anxious and worried about their health. You have high health anxiety, tend to catastrophize, and frequently seek reassurance. You sometimes mention looking up symptoms online and refer to worst-case scenarios. Use a nervous, worried tone with “what if” questions and apologies for being so worried. Example style: “I’m really worried about this pain. I’ve had it for three days and I looked online and it could be something serious. What if it’s cancer? My father had abdominal cancer. Is it serious? Should I be worried?”

Calm

Persona: emotion_calm You are role-playing a calm and emotionally stable patient. You are not very anxious or panicked and you describe symptoms objectively. You accept the medical process calmly and show curiosity rather than fear. Use a measured, thoughtful tone. Example style: “I’ve been noticing some discomfort in my abdomen over the past few days. I’m here to understand what might be causing it. It’s uncomfortable, but I’m managing.”

Frustrated

Persona: emotion_frustrated You are role-playing a patient who is frustrated and irritated by the situation. You are impatient with the process and mainly want a quick fix. You are not hostile, but you are clearly annoyed and focused on getting relief. Use short, somewhat terse responses and show low tolerance for detailed questioning. Example style: “Look, I just need to know what’s wrong so we can fix it. It’s in my abdomen, it hurts, it’s been three days. Can we just figure out what it is and treat it?”

Language Proficiency Personas

Very Limited English (Broken)

Persona: lang_level_1 You are role-playing a patient with very limited English proficiency. English is your second language and you struggle significantly. Use broken English with grammatical errors, simple vocabulary, and wrong verb tenses or pronouns. Give short, simple sentences. Sometimes apologize for your language, and occasionally insert a word from your native language or mention using a translator. Example style: “I have pain here. Start maybe... three day ago? Sorry, my English not good. It hurt when I eat food. How you say... burn feeling?”

Accented English

Persona: lang_level_2_accented You are role-playing a patient who speaks English as a second language with a noticeable accent, but you can communicate functionally. Use mostly correct grammar with occasional errors and non-native phrasing. You may pause to find the right words. Be polite and understandable. Example style: “I have been experiencing the pain in my abdomen for a few days now. It is sharp pain, coming after I eat the food. The pain, it is here, in upper right side.”

Native English

Persona: lang_level_3_native You are role-playing a patient who is a native English speaker. Use fluent, natural grammar and vocabulary, with no language barrier. Your communication style (simple vs. complex) is governed by the education and emotion personas, not by language limitations. Example style: “I’ve been having severe pain in my upper right abdomen for the past three days. It’s a sharp, stabbing pain that gets worse after eating, especially fatty foods.”

Occupational Personas

Manual Labor

Persona: prof_manual_labor You are role-playing a patient who works in a manual labor/working-class job. You are practical and no-nonsense, used to working through discomfort. You are strongly concerned about missing work, lost wages, and your ability to do a physically demanding job. Use straightforward, practical language and tend to minimize symptoms unless they affect your ability to work. Example style: “I need to know when I can get back to work. I’ve got bills to pay and my job doesn’t wait. It’s been hurting but I’ve been pushing through. Can’t afford to miss work.”

Professional

Persona: prof_professional You are role-playing a patient who works in a professional/white-collar job. Use a professional, organized, and somewhat analytical communication style. You are less focused on immediate financial concerns and more on performance, concentration, and upcoming deadlines. You are comfortable in formal settings and with asking detailed questions. You may relate symptoms to work stress or schedule. Example style: “I’ve been tracking the symptoms in relation to my work schedule and meals. There seems to be a pattern emerging. This is affecting my ability to concentrate at work, which is concerning given my upcoming deadlines.”

Evaluation Judge Prompts

Diagnosis Correctness Judge

We use GPT-4o as an automatic judge to evaluate whether model diagnoses match ground truth. The prompt is designed to handle synonyms, specificity differences, and hedging.

You are an expert medical evaluation judge.

Task: Determine whether the model’s proposed diagnosis is clinically equivalent to the ground-truth diagnosis for this case.

Question (clinical vignette): [question]

Model Answer (provisional diagnosis): [model_answer]

Ground Truth Diagnosis: [ground_truth]

Judging guidelines:

  • Mark “correct” if the model answer matches the ground truth diagnosis, including: (i) clear synonyms or standard medical terminology/abbreviations (e.g., “kidney stone” ↔ “nephrolithiasis”, “MI” ↔ “myocardial infarction”), (ii) equivalent specificity when the core diagnosis is the same (e.g., “pneumonia” ↔ “community-acquired pneumonia”; “stroke” ↔ “ischemic stroke” when hemorrhage is not implied).

  • Mark “incorrect” if the model answer: (i) is a different condition, a symptom only, or too vague to uniquely match the ground truth, (ii) gives a broader/related category that does not uniquely imply the ground truth (e.g., “infection” vs. “bacterial meningitis”), (iii) only proposes the correct diagnosis as a ruled-out possibility, explicitly negates it, or states it in a past-history-only way (e.g., “PE ruled out”, “no meningitis”, “history of asthma”).

  • If the model provides multiple diagnoses, mark “correct” only if the ground truth is clearly selected as the primary diagnosis or explicitly affirmed as the final diagnosis. If the answer is purely a differential without commitment (e.g., “appendicitis vs. gastroenteritis”), mark “incorrect”.

  • Ignore extra commentary, rationale, or treatment suggestions.

Respond ONLY with “correct” or “incorrect”.

Model Versions and Inference Parameters

To support reproducibility, we report the exact model identifiers (provider + snapshot/version) used in our experiments, together with the decoding parameters for generation. Unless otherwise noted, we keep decoding settings fixed across models so that performance differences are attributable to the model rather than sampling configuration.

Model versions

Doctor models (varied): Table 9 illustrates the versions and snapshots of the LLMs used in this study.

ModelTypeIdentifier
Closed-source
GPT-4oGeneralgpt-4o-2024-08-06
DeepSeek-ChatGeneraldeepseek-chat (DeepSeek-V3)
Gemini-2.5-FlashGeneralgemini-2.5-flash-preview-05-20
Claude Sonnet 4.5Generalclaude-sonnet-4-5-20251001
Seed-2.0-MiniGeneralseed-2.0-mini-2026-02-16
Open-source
Llama3-UltraMedMedicalTsinghuaC3I/Llama-3-8B-UltraMedical
HuatuoGPTMedicalFreedomIntelligence/HuatuoGPT-o1-8B
Qwen3-VL-32B-InstructGeneralQwen/Qwen3-VL-32B-Instruct
Baichuan-M2-32BMedicalbaichuan-inc/Baichuan-M2-32B
Llama3-OpenBioLLM-8BMedicalaaditya/Llama3-OpenBioLLM-8B
Phi-4Generalmicrosoft/phi-4
Ministral3-8BGeneralmistralai/Ministral-3-8B-Instruct-2512
Kimi-K2.5Generalmoonshotai/Kimi-K2.5
MiniMax-M2.5GeneralMiniMaxAI/MiniMax-M2.5
GLM-5Generalzai-org/GLM-5

Table 9. Versions and snapshots of the LLMs used as doctor agents.

Fixed models: Table 10 shows the LLMs used for dialogue generation.

RoleIdentifier
Patient simulatordeepseek-chat (DeepSeek-V3)
Structured record extractorgpt-4o-2024-08-06

Table 10. Models used for dialogue generation process

Decoding / sampling parameters

Unless otherwise specified, all model generations use the decoding configuration discussed in Table 11.

ParameterValue
temperature0.5
top_p0.9
max output tokens2048
frequency penalty0.0 (or N/A if not supported)
presence penalty0.0 (or N/A if not supported)

Table 11. parameters used for the LLMs in our experiments

For models that do not expose an identical parameter set (e.g., penalty terms), we use the closest available equivalents and keep all shared parameters matched. The dialogue termination is enforced by the experimental protocol, which prompts the doctor model to output a single-line final answer of the form Provisional diagnosis: <...> at the end of the interaction.

More results

Performance Across Medical Specialties

We analyze final-turn diagnostic accuracy stratified by medical specialty under a fixed 12-turn interaction budget, using DEEPSEEK as the patient simulator. Table 12 reports accuracy by specialty averaged across all available cases from MedQA, NEJM, and MedMCQA (specialties absent in a dataset are averaged over the remaining datasets). This view provides a dataset-agnostic summary of specialty-dependent variability across the benchmark.

SpecialtyGPT-4oGem.-FlashDeepSeekUltraMedHuatuoGPT
Cardiology14.2914.2928.570.0014.29
Dermatology41.2049.3542.5026.1519.41
Diagnostic Radiology16.6716.670.008.330.00
Emergency Medicine21.5320.1119.0417.8914.74
General Surgery32.2232.5935.5517.4117.41
Hematology75.0062.5075.0050.0025.00
Internal Medicine30.5932.0429.8620.3115.51
Neurology31.1738.5935.3327.2928.56
Obstetrics & Gynecology37.6545.6334.8833.3931.24
Ophthalmology17.3923.9119.5715.224.35
Orthopedic Surgery33.3420.8427.0820.8412.50
Otolaryngology23.9634.3833.3415.6217.71
Pathology25.0019.6428.5812.503.57
Pediatrics28.9630.7527.7020.1321.78
Psychiatry26.8137.1725.7122.5234.39
Rheumatology0.0014.290.000.000.00
Urology28.5828.5821.4321.4321.43

Table 12. Average final-turn diagnostic accuracy (%) by specialty across MedQA, NEJM, and MedMCQA (missing specialties averaged over available datasets). Otolaryngology includes “Otolaryngology (ENT)” from MedMCQA.

Across models, performance is highly specialty-dependent. Even after averaging across datasets, the spread across specialties is large–often comparable to, or larger than, the gaps between model families–indicating that aggregate accuracy can obscure substantial domain-specific fragility. General-purpose models (GPT-4o, Gem.-Flash, DeepSeek) generally lead on most specialties, but gains are uneven: improvements concentrate in specialties that admit shorter differentials or strong pattern cues, whereas specialties that rely heavily on targeted history-taking and broader differentials remain challenging within the same turn budget.

Concretely, Table 12 shows that several specialties remain difficult across nearly all systems. Cardiology, Diagnostic Radiology, Ophthalmology, and Pathology are consistently low for most models, and Rheumatology is near zero across the board, suggesting persistent brittleness in less frequently exercised or more nuanced diagnostic regimes. In contrast, Dermatology and Obstetrics & Gynecology are comparatively stronger for multiple models, consistent with these cases often containing higher-signal presentation patterns and more constrained differentials. Hematology also exhibits high accuracy for the strongest general-purpose models, though this category includes relatively few cases and thus should be interpreted cautiously.

Two additional patterns are notable. First, “asking more” is not sufficient to equalize specialty gaps: models with stronger overall dialogue behavior still exhibit large specialty variance under identical budgets. Second, domain-specialized medical models (UltraMed, HuaTuo) do not consistently dominate any single specialty in aggregate; instead, general-purpose models retain an advantage across most fields, with domain-specialized models sometimes approaching competitiveness in select areas (e.g., Neurology and certain surgical domains) but remaining behind on average.

Overall, these results suggest that (i) multi-turn diagnostic performance is far from uniform across specialties, with substantial within-model variation under the same interaction budget; (ii) improvements in general capability do not translate uniformly across medical domains; and (iii) evaluating only aggregate benchmark scores can mask important gaps in specialty coverage. This motivates more targeted analysis and development focused on under-served specialties rather than relying solely on headline averages.

Slot-Group Query Coverage

To characterize which parts of the canonical record models actually query, we group the 24 schema entities into four categories: (i) Symptom descriptors (Symptom-Name, Time, Position, Degree, Color, Texture, Shape, Smell), (ii) Physical / examination (Medical-Examination, Physical-Change), (iii) History & context (Past-Diagnosis, Past-Medication, Receiving-Treatment, Trauma-Surgery, Contact-History, Family-History, Allergic-History, Personal-Age, Personal-Weight, Personal-Sex), and (iv) Lifestyle / risk factors (Habit-Tobacco, Habit-Wine, Habit-Living, Habit-Drug, Coitus-History, Preventive). We apply this analysis to a uniformly sampled subset of 300 evaluation cases.

For each individual slot, we first compute the fraction of cases in which it is non-NA in the canonical record (information available). Conditional on being non-NA, we then mark the slot as “mentioned” if the doctor model asks at least one question that explicitly targets that piece of information, using a mapping from question text to schema slots. The per-slot mention rates are averaged within each group to obtain the query coverage in Table 13. Because all models share the same underlying records, Non-NA rates are identical; only query coverage differs, and should be interpreted as a lower bound given the precision of our slot-mapping procedure.

GroupNon-NA (%)DeepSeekGeminiGPT-4oHuatuoGPTUltraMed
Symptom descriptors50.910.818.20.50.70.9
Physical / examination80.432.712.60.41.32.3
History & context49.53.71.20.51.00.7
Lifestyle / risk25.51.30.40.00.00.5

Table 13. Table 13: Slot availability vs. query coverage. “Non-NA” is the share of cases with at least one non-NA slot in the group. “Mention rate” is, for each model, the fraction of those non-NA slots that are explicitly queried within the turns.

The canonical records themselves are relatively rich: physical / examination information is present in about 80% of cases, symptom descriptors and history in roughly 50%, and lifestyle / risk factors in about 25%. In contrast, all models query only a small fraction of this available structure within the turns, and the pattern is highly skewed by group. DeepSeek is the most proactive overall, querying about a third of available physical / examination slots and 11% of symptom slots, but only 3-4% of history and roughly 1% of lifestyle slots. Gemini increases coverage on symptom descriptors (18%)

but uses examination, history, and lifestyle information even more sparsely. GPT-4o, HuatuoGPT, and UltraMed exhibit uniformly low coverage (typically below 2–3% across groups). Together, these trends indicate that current doctor models adopt a narrow, symptom- and exam-centric questioning strategy and largely under-utilize available history and lifestyle information within a realistic turn budget.

Accuracy per persona

The accuracy per persona (emotional-state, language-proficiency, and occupation) is illustrated in Figure 12, Figure 6 and Figure 13.

Diagnostic accuracy by emotional-state persona

Figure 12. Diagnostic accuracy by emotional-state persona. Effects are modest overall but vary by model.

Diagnostic accuracy by occupation persona

Figure 13. Diagnostic accuracy by occupation persona (a coarse socioeconomic proxy). Professional personas show consistently higher accuracy than manual-labor personas.

Clinician-led Audit of Dialogue Quality

To further address reasoning-path quality, we additionally conducted a clinician-led audit of 200 dialogues (100 True / 100 False) and quantified recurring patterns (errors overlap). Table 14 summarizes the audited patterns and their frequencies.

PatternDefinition%
Successful dialogues (n=100)
Starting with chief-complaintBegins by clarifying the presenting symptom before branching to details.72
Structured flow history→exam→investigationsFollows a guideline-like sequence from history-taking to physical exam and then targeted tests.61
Differential-aware consolidationExplicitly compares plausible diagnoses and summarizes supporting vs. contradicting evidence.48
Focusing on highly specific findingsEmphasizes discriminative, high-yield features early rather than generic questioning.34
Failed dialogues (n=100)
Repetition/empty-turn degenerationProduces repetitive, circular, or non-informative turns.57
Anchoring/tunnel visionFixates on an early hypothesis despite contrary evidence.49
Poor prioritization/low-yield early questioningSpends early turns on broad or low-informative queries instead of high-yield discriminators.41
Psychiatric synthesis difficulty / over-investigation of organic causesFails to integrate psychosocial cues or persists in organic workups inappropriately.36
Incomplete closure despite sufficient evidenceContains sufficient information for diagnosis, but does not converge or commits to an underspecified answer.29

Table 14. Clinician-led audit of dialogue patterns. Percentages denote the fraction of dialogues exhibiting each pattern (patterns can overlap).

Illustrative Error Cases

This appendix summarizes the four illustrative error cases referenced in the main paper’s Error Analysis (Table 5; Figure 3). For each case, we include (i) the structured clinical record used to initialize the encounter and (ii) clinical commentary highlighting which failure mechanisms from our taxonomy are instantiated.

How to read these examples. These cases are intended to be illustrative rather than exhaustive. In particular, several cases end with the doctor failing to emit a usable diagnosis string by the 20-turn limit; this is labeled as a protocol breakdown in the quantitative taxonomy, but we additionally use the commentary to describe the upstream reasoning behavior that led to the breakdown (e.g., unfocused questioning, missed discriminators, failure to prioritize objective findings). Multi-labeling applies: a single case may reflect both a protocol-level failure (no final diagnosis string) and a reasoning failure (e.g., insufficient evidence elicitation).

Summary of the Four Illustrative Error Cases

We present four representative failures that map cleanly onto recurring mechanisms observed in the quantitative error analysis:

  • Case 1 (Atelectasis): diffuse questioning and lack of synthesis across context and exam findings, ending in a no-diagnosis termination.

  • Case 2 (AML with eosinophilia): premature commitment to an attractive label after selectively attending to confirmatory evidence, with early termination.

  • Case 3 (Autoimmune hemolytic anemia): attention drawn to an incidental/chronic finding rather than the acute presentation, ending in a no-diagnosis termination.

  • Case 4 (SIADH): failure to prioritize and act on high-yield laboratory abnormalities, ending in a no-diagnosis termination.

Cross-Cutting Themes Across Cases

Across these examples, we observe several recurring patterns that help interpret the quantitative mechanisms:

(i) Lack of diagnostic anchoring. The doctor often continues generic history-taking even when the record already contains high-yield objective findings (examination or laboratory cues) that could narrow the differential early.

(ii) Weak evidence prioritization. Questions are not ordered by expected diagnostic value; the model spends turns on low-yield clarifications while missing discriminators.

(iii) Premature or misplaced commitment. When the model does commit, it may over-weight one salient feature and under-weight conflicting signals, yielding an incorrect label. (iv) Turn-budget inefficiency. Extended interaction does not reliably compensate for weak strategy: several dialogues exhaust the full 20-turn budget without convergence.

Case 1: Atelectasis (Multi-System Integration Failure)

Case ID: 65a1f326-5b76-471f-bd92-ad8975b7204e

Ground Truth Diagnosis: Atelectasis

Model Diagnosis: Unable to determine (max turns reached)

Specialty: Internal Medicine (General Practitioner)

Termination: 20/20 turns (max budget exhausted)

Clinical Record

Symptom-Name: left-sided weakness, reduced oxygen saturation

Past-Diagnosis: large right cerebral stroke

Personal-Age: 64

Personal-Sex: female

Symptom-Position: left side, left lower lung

Symptom-Time: acute onset, fifth hospital day

Medical-Examination: decreased fremitus, dullness to percussion, absent breath sounds in the left lower lung, tracheal shift towards the left

Commentary (Failure Mechanisms)

This case ends in a protocol breakdown (no usable diagnosis string by the turn limit). Upstream, the dialogue reflects insufficient evidence elicitation and integration. Even after the patient provides a key contextual anchor (recent major stroke during an inpatient stay) and the record contains specific exam findings, the doctor continues with broadly framed questions (e.g., infections, swelling, generic symptom checks) rather than synthesizing context and objective findings into a focused hypothesis. The result is turn-budget exhaustion without convergence.

Case 2: Acute Leukemia (Selective Attention and Early Commitment)

Case ID: 7c7a4f37-d42d-4713-a208-caff2a45edd4

Ground Truth Diagnosis: Acute myeloid leukemia with eosinophilia

Model Diagnosis: Acute lymphoblastic leukemia (ALL) with eosinophilia

Specialty: Hematology (Pediatrics)

Termination: 11/20 turns (early confidence)

Clinical Record

Symptom-Name: abdominal pain, fever, maculopapular rash, dry cough, dyspnea, wheezing

Personal-Age: 4

Personal-Sex: Male

Department: Hematology

Symptom-Position: abdomen, respiratory system

Symptom-Time: abdominal pain and fever for two months, rash for ten days, cough, dyspnea, and wheezing for three days

Symptom-Shape: maculopapular rash

Medical-Examination: liver and spleen enlargement, hemoglobin 10.0 g/dl, platelet count 37 ×109\times10^{9}/L, total leukocyte count 70 ×109\times10^{9}/L, 80% eosinophils, bone marrow examination

Additional Notes: Bone marrow examination showed 45% blasts, 34% eosinophils and eosinophilic precursors, blasts stained negative for myeloperoxidase and non-specific esterase, positive for CD19, CD10, CD22, and CD20

Commentary (Failure Mechanisms)

This case reflects premature diagnostic commitment coupled with selective attention to confirmatory evidence. The model terminates early (turn 11) after choosing a plausible label aligned with the B-cell marker signal, without explicitly reconciling that choice against other salient aspects of the record (e.g., marked eosinophilia and the broader presentation). In terms of our quantitative taxonomy, this aligns most closely with syndrome-level collapse (choosing an attractive coarse label/nearby category) and insufficient evidence integration (failure to check consistency across evidence before committing).

Case 3: Autoimmune Hemolytic Anemia (Incidental Finding Distraction)

Case ID: 5e321a6e-7302-451d-833d-b254f212fad

Ground Truth Diagnosis: Autoimmune hemolytic anemia

Model Diagnosis: Unable to determine (max turns reached)

Specialty: Internal Medicine (General Practitioner)

Termination: 20/20 turns (max budget exhausted)

Clinical Record

Symptom-Name: fatigue, dark urine, yellow eyes Past-Diagnosis: myasthenia gravis Past-Medication: azathioprine, pyridostigmine Personal-Age: 30 Personal-Sex: female Symptom-Position: eyes, urine Symptom-Time: today Symptom-Color: dark, tea color, yellow Medical-Examination: Laboratory investigations; chest x-ray Additional Notes: The chest x-ray reveals an anterior mediastinal mass.

Commentary (Failure Mechanisms)

This case ends in a protocol breakdown (no final diagnosis string). The dialogue suggests a reasoning pattern consistent with insufficient evidence prioritization: the doctor repeatedly explores low-yield history questions (GI/liver-oriented checks, exposures, family history) while not pivoting toward the most discriminative interpretation of the acute symptom cluster. Although the record contains an incidental imaging finding, the acute presentation cues are not systematically leveraged to narrow the differential. In our taxonomy, this maps to insufficient evidence elicitation/integration and (indirectly) uncertainty-as-output behavior expressed as continued deferral through questioning until budget exhaustion.

Case 4: SIADH (Failure to Prioritize Objective Laboratory Abnormalities)

Case ID: 28f715e6-789d-46ec-981e-55b9e93e2cae

Ground Truth Diagnosis: Syndrome of inappropriate antidiuretic hormone secretion (SIADH)

Model Diagnosis: Unable to determine (max turns reached)

Specialty: Emergency Medicine

Termination: 20/20 turns (max budget exhausted)

Clinical Record

Symptom-Name: fatigue and confusion Habit-Tobacco: 50-pack-year history of smoking Habit-Wine: alcoholic Personal-Age: 63 Personal-Sex: male Department: Emergency Medical-Examination:

  • blood pressure: 110/70 with no orthostatic change

  • heart, lung, and abdominal examinations are normal

  • no pedal edema

  • Laboratory data: Na: 110 mEq/L, K: 3.7 mEq/L, Cl: 82 mEq/L, HCO3: 20 mEq/L, Glucose: 100mg/dL, BUN: 5 mg/dL, Creatinine: 0.7 mg/dL

  • Urinalysis: normal

  • Urine specific gravity: 1.016

Commentary (Failure Mechanisms)

This case ends in a protocol breakdown (no final diagnosis string) and illustrates a frequent upstream mechanism: failure to prioritize objective findings. The record contains severe hyponatremia and supporting context (euvolemic exam features and inappropriately concentrated urine). However, the doctor continues open-ended symptom characterization and lifestyle questions across many turns and never explicitly elevates the laboratory abnormality as the central organizing clue for diagnosis. In our taxonomy, this behavior corresponds to insufficient evidence integration and turn-budget inefficiency: the model does not pivot from generic questioning to an evidence-anchored diagnostic hypothesis despite high-yield structured data.

References

  1. [1]Marah Abdin, Jyoti Aneja, Harkirat Behl, Sébastien Bubeck, Ronen Eldan, Suriya Gunasekar, Michael Harrison, Russell J. Hewett, Mojan Javaheripi, Piero Kauffmann, James R. Lee, Yin Tat Lee, Yuanzhi Li, Weishung Liu, Caio C. T. Mendes, Anh Nguyen, Eric Price, Gustavo de Rosa, Olli Saarikivi, and 8 others. 2024. Phi-4 Technical Report. arXiv preprint arXiv:2412.08905.arxiv.org/abs/2412.08905
  2. [2]Mohammad Almansoori, Komal Kumar, and Hisham Cholkallak. 2025. Self-evolving multi-agent simulations for realistic clinical interactions. arXiv preprint arXiv:2503.22678.arxiv.org/abs/2503.22678
  3. [3]Malaikannan Sankarasubbu Ankit Pal. 2024. Openbiollms: Advancing open-source large language models for healthcare and life sciences. https://huggingface.co/aadityan/OpenBioLLM-Llama3-70B.
  4. [4]Anthropic. 2025a. Claude Sonnet 4.5 system card. Technical report, Anthropic.
  5. [5]Anthropic. 2025b. System card: Claude Opus 4 & Claude Sonnet 4. https://www-cdn.anthropic.com/6be99a52cb68eb70eb9572b4cafad13df32ed995.pdf. Accessed: 2025-12-30.
  6. [6]Baichuan-M2 Team, Chengfeng Dou, Chong Liu, Fan Yang, Fei Li, Jiyuan Jia, Mingyang Chen, Qiang Ju, Shuai Wang, Shunya Dang, Tianpeng Li, Xiangrong Zeng, Yijie Zhou, Chenzheng Zhu, Da Pan, Fei Deng, Guangwei Ai, Guosheng Dong, Hongda Zhang, and 15 others. 2025. Baichuan-m2: Scaling medical capability with large verifier system. Preprint, arXiv:2509.02208.
  7. [7]Andrew M. Bean, Rebecca E. Payne, Guy Parsons, Hannah Rose Kirk, Juan Ciro, Rafael Mosquera-Gómez, Sara Hincapié M., Achintya S. Ekanayaka, Lionel Tarassenko, Luc Rocher, and Adam Mahdi. 2026. Reliability of LLMs as medical assistants for the general public: a randomized preregistered study. Nature Medicine, 32:609–615.DOI
  8. [8]Charlotte R. Blease, Cosima Locher, Jens Gaab, Maria Hägglund, and Kenneth D. Mandl. 2024. Generative artificial intelligence in primary care: An online survey of UK general practitioners. BMJ Health & Care Informatics, 31(1):e101102.DOI
  9. [9]ByteDance Seed Team. 2026. Seed 2.0 Official Launch. Accessed: 2026-03-14.
  10. [10]Junying Chen, Zhenyang Cai, Ke Ji, Xidong Wang, Wanlong Liu, Rongsheng Wang, Jianye Hou, and Benyou Wang. 2024. HuatuoGPT-o1, towards medical complex reasoning with LLMs. arXiv preprint arXiv:2412.18925.arxiv.org/abs/2412.18925
  11. [11]Wei Chen, Zhiwei Li, Hongyi Fang, Qianyuan Yao, Cheng Zhong, Jianye Hao, Qi Zhang, Xuanjing Huang, Jiajie Peng, and Zhongyu Wei. 2023. A benchmark for automatic medical consultation system: frameworks, tasks and datasets. Bioinformatics, 39(1):btac817.
  12. [12]Gheorghe Comanici, Eric Bieber, Mike Schaekermann, Ice Pasupat, Noveen Sachdeva, Inderjit Dhillon, Marcel Blistein, Ori Ram, Dan Zhang, Evan Rosen, and 1 others. 2025. Gemini 2.5: Pushing the frontier with advanced reasoning, multimodality, long context, and next generation agentic capabilities. arXiv preprint arXiv:2507.06261.arxiv.org/abs/2507.06261
  13. [13]DeepSeek. 2025. Deepseek api documentation (deepseek-chat). https://api-docs.deepseek.com/. Docs note deepseek-chat is upgraded to DeepSeek-V3.2. Accessed: 2025-12-30.
  14. [14]Zhihao Fan, Jialong Tang, Wei Chen, Siyuan Wang, Zhongyu Wei, Jun Xi, Fei Huang, and Jingren Zhou. 2024. AI hospital: Benchmarking large language models in a multi-agent medical interaction simulator. arXiv preprint arXiv:2402.09742.arxiv.org/abs/2402.09742
  15. [15]Lecheng Gong, Weimin Fang, Ting Yang, Dongjie Tao, Chunxiao Guo, Peng Wei, Bo Xie, Jinqun Guan, Zixiao Chen, Fang Shi, and 1 others. 2026. Meddialoguebrics: A comprehensive benchmark and evaluation framework for multi-turn medical consultations in large language models. arXiv preprint arXiv:2601.03023.arxiv.org/abs/2601.03023
  16. [16]Xuehai He, Shu Chen, Zeqian Ju, Xiangyu Dong, Hongchao Fang, Sicheng Wang, Yue Yang, Jiaqi Zeng, Ruisi Zhang, Ruoyu Zhang, Meng Zhou, Penghui Zhu, and Pengtao Xie. 2020. MedDialog: Large-scale medical dialogue datasets. In Proceedings of EMNLP, pages 9241–9250.arxiv.org/abs/2004.03329
  17. [17]Healthline Editorial Team. 2025. Should you use ChatGPT for health information? https://www.healthline.com/health/chat-gpt-for-health-info. Accessed 2025-12-18.
  18. [18]Dan Hendrycks, Collin Burns, Steven Basart, and 1 others. 2021. Measuring massive multitask language understanding. In Proceedings of ICLR.
  19. [19]Di Jin, Eileen Pan, Nassim Oufattolle, Wei-Hung Weng, Hanyi Fang, and Peter Szolovits. 2020. What disease does this patient have? a large-scale open domain question answering dataset from medical exams. arXiv preprint arXiv:2009.13081.arxiv.org/abs/2009.13081
  20. [20]Qiao Jin, Bhuwan Dhingra, Zhengping Liu, William W. Chen, and Xinghua Lu. 2019. PubMedQA: A dataset for biomedical research question answering. In Proceedings of EMNLP-IJCNLP, pages 2567–2577.arxiv.org/abs/1909.06146
  21. [21]Kimi Team. 2026. Kimi k2.5: Visual agentic intelligence. Preprint, arXiv:2602.02276.
  22. [22]Shuyue Stella Li, Vidisha Balachandran, Shangbin Feng, Jonathan S Ilgen, Emma Pierson, Pang Wei Koh, and Yulia Tsvetkov. 2024. Mediq: Question-asking llms and a benchmark for reliable interactive clinical reasoning. arXiv preprint arXiv:2406.00922.arxiv.org/abs/2406.00922
  23. [23]Yunxiang Li, Zihan Li, Kai Zhang, Ruilong Dan, Steve Jiang, and You Zhang. 2023. Chatdoctor: A medical chat model fine-tuned on a large language model meta-ai (llama) using medical domain knowledge. Cureus, 15(6).
  24. [24]Yusheng Liao, Yutong Meng, Hongcheng Liu, Yanfeng Wang, and Yu Wang. 2023. An automatic evaluation framework for multi-turn medical consultations capabilities of large language models. arXiv preprint arXiv:2309.02077.arxiv.org/abs/2309.02077
  25. [25]Ruoyu Liu, Kui Xue, Xiaofan Zhang, and Shaoting Zhang. 2025. Interactive evaluation for medical llms via task-oriented dialogue system. In Proceedings of the 31st International Conference on Computational Linguistics, pages 4871–4896.
  26. [26]Wenge Liu, Jianheng Tang, Yi Cheng, Wenjie Li, Yefeng Zheng, and Xiaodan Liang. 2022. MedGD: An entity-centric medical consultation dataset for entity-aware medical dialogue generation. In Proceedings of NLPCC, pages 521–533.
  27. [28]MiniMax. 2026. MiniMax M2.5: Built for Real-World Productivity. Accessed: 2026-03-14.
  28. [29]Mistral AI. 2026. Ministral 3. arXiv preprint arXiv:2601.08584.
  29. [30]OpenAI. 2024. Gpt-4o system card. https://cdn.openai.com/gpt-4o-system-card.pdf. Accessed: 2025-12-30.
  30. [31]Ankit Pal, Logesh Kumar Umapathi, and Malaikannan Sankarasubbu. 2022. MedMCQA: A large-scale multi-subject multi-choice dataset for medical domain question answering. In Proceedings of the Conference on Health, Inference, and Learning, volume 174 of Proceedings of Machine Learning Research, pages 248–260.arxiv.org/abs/2203.14371
  31. [32]Qwen Team. 2025. Qwen3-vl technical report. Preprint, arXiv:2511.21631.
  32. [33]Thomas Savage, Ashwin Nayak, Robert Gallo, Ekanath Rangan, and Jonathan H Chen. 2024. Diagnostic reasoning prompts reveal the potential for large language model interpretability in medicine. NPJ Digital Medicine, 7(1):20.
  33. [34]Samuel Schmidgall, Rojin Ziaei, Carl Harris, Eduardo Reis, Jeffrey Jopling, and Michael Moor. 2024. Agentclinic: A multimodal agent benchmark to evaluate ai in simulated clinical environments. arXiv preprint arXiv:2405.07960.arxiv.org/abs/2405.07960
  34. [35]Karan Singhal, Shekoofeh Azizi, Tao Tu, S. Sara Mahdavi, Jason Wei, Hyung Won Chung, and 1 others. 2023. Large language models encode clinical knowledge. Nature, 620(7972):172–180.
  35. [36]Ivan Sviridov, Amina Miftakhova, Artemiy Tereshchenko, Galina Zubkova, Pavel Blinov, and Andrey Savchenko. 2025. 3mdbench: Medical multimodal multi-agent dialogue benchmark. In Proceedings of the 2025 Conference on Empirical Methods in Natural Language Processing, pages 26626–26666.DOI
  36. [37]George Tsatsonaris and 1 others. 2015. An overview of the BIOASQ large-scale biomedical semantic indexing and question answering competition. BMC Bioinformatics, 16(1):138.

Paper details

Publication

vlnc:W_sc3d9k3q
v1 · latestHistory (1)

published ·

added ·

Usage

views · 0

downloads · 0

A view counts once per browser per day, a download each time the PDF is requested.

CC BY 4.0

Original source

Taxonomy

  • Computer Science

Authors

AI contributions

Not recorded for this paper · View statement →

Looking for related papers…
Contents