WorksComputer Science
DiagTrace: An Interactive Benchmark for Clinical Decision Making
Abstract
Large language models (LLMs) have shown strong performance on medical question answering and diagnostic reasoning benchmarks. However, existing evaluations typically provide all patient information upfront and assess only the final diagnosis or rationale. This setup ignores the need to acquire diagnostic evidence incrementally from incomplete information in real-world diagnosis. To bridge this gap, we introduce DiagTrace, a source-grounded benchmark for evaluating whether LLMs can reconstruct the documented diagnostic trajectory through interactive information seeking. Each benchmark instance is built from a PubMed Central case report and includes an initial presentation, an event bank, a clinical course, and diagnostic references. DiagTrace contains 510 cases across eight specialties, with 225 used to evaluate four LLMs on clinical-course alignment, interaction quality, and efficiency. Results reveal trade-offs in coverage, source-consistent ordering, and turn use.
Introduction
Recent medical large language models (LLMs) have primarily been evaluated in static settings ((Singhal et al., 2023; Qiu et al., 2025; Wang et al., 2024)), where a fixed set of patient findings is provided and LLMs predict a diagnosis or rationale. While useful for measuring medical knowledge and reasoning, these benchmarks do not test how a model actively acquires clinically relevant information.
This gap is important because clinical diagnosis is not only an inference problem, but also an information-acquisition process, including deciding what to ask, what to examine, what to test, and when the accumulated findings are sufficient. Recent interactive medical benchmarks move closer to this setting by requiring models to ask follow-up questions or request tests over multiple turns ([9, 11, 4, 2] (Tu et al. 2025; Sviridov et al. 2025; Werthaim et al. 2026)). However, they typically prioritize final diagnostic accuracy or dialogue quality, leaving it under-evaluated whether a model can actively seek the right information at the right time.
In this work, we ask: given the correct diagnosis, can a model actively reconstruct the diagnostic trajectory through interactive information seeking? This formulation matters for two reasons. First, by providing the final diagnosis, it separates information acquisition from diagnostic inference. Second, it supports evaluation under a fixed turn budget, requiring the model to prioritize informative requests and balance information coverage against interaction cost ((Yue et al., 2026)).
We introduce DIAGTRACE, a benchmark built from PubMed Central (PMC) case reports for evaluating interactive diagnostic reconstruction ((Zhao et al., 2023; Zhang et al., 2025; National Library of Medicine, National Center for Biotechnology Information, 2026)). Each case includes an initial patient presentation, an event bank, a clinical course, and final diagnostic references. During evaluation, a clinician (evaluated) agent receives the initial presentation and final diagnosis, and must interact with a source-grounded report agent under a fixed turn budget to recover the supporting diagnostic trajectory. We evaluate this process at the trajectory level by comparing the model’s request trajectory with a source-derived reference using metrics like clinical course alignment, interactive quality, and turn efficiency and availability.
DIAGTRACE contains 510 cases spanning eight clinical specialties, with 225 cases used for comprehensive evaluation. Our results reveal substantial variation in models’ ability to acquire diagnostic evidence, highlighting trade-offs between information coverage, source-consistent ordering, and interaction efficiency.
Our contributions are threefold: (i) a source– grounded benchmark derived from case reports; (ii) a clinical interactive task that isolates information acquisition from diagnosis inference; and (iii) a trajectory-level evaluation protocol for measuring what information models request, in what order, and at what interaction cost.
Related Work
Medical LLM evaluation has traditionally focused on static question answering and structured diagnostic reasoning benchmarks ([3] (Singhal et al. 2023; Wang et al. 2024; Qiu et al. 2025; Zhou et al. 2026; Yang et al. 2026)). While these benchmarks assess clinical knowledge and reasoning quality, the relevant findings are provided rather than actively acquired. In contrast, DIAGTRACE evaluates how clinician agents actively reconstruct a diagnostic trajectory through sequential information seeking.
Interactive clinical benchmarks move closer to real diagnostic settings by evaluating models under partial information through multi-turn dialogue or agent-based environments ([9, 11, 4, 2, 5, 10] (Tu et al. 2025; Saab et al. 2026; Sviridov et al. 2025; Werthaim et al. 2026)). More recent systems further study autonomous diagnostic agents that perform active information acquisition ([6, 8] (Sanghvi et al. 2026)). DIAGTRACE is complementary to this line of work by evaluating not only whether an agent reaches the diagnosis, but whether its information-request trajectory matches a source-derived reference (Yue et al., 2026). Our prior work explored source-grounded diagnostic dialogues (Wang et al., 2026), while DiagTrace focuses on evaluating interactive evidence acquisition.
Methods
Our pipeline has three stages: (1) reference resource construction, (2) an interactive protocol, and (3) an evaluation protocol (Figure 1).
Figure 1. Overview of DIAGTRACE. (A) PMC case reports are converted into validated reference resources. (B) A diagnosis-aware clinician agent interacts with a source-grounded report agent under a fixed turn budget to reconstruct the documented diagnostic trajectory. (C) Interactions are evaluated using clinical course alignment, interactive quality, turn efficiency and availability.
Reference resource construction
Construction. We selected PMC case reports from eight clinical specialties (Table 1). Each report is converted into a structured reference resource with four components (Prompt 1): (1) Initial presentation, describing the patient’s presenting context; (2) Event bank, containing documented findings across 10 types (Table 2); (3) Clinical course, capturing the patient’s progression, including diagnostic workup, treatment, follow-up, and outcomes when available; and (4) Diagnostic references, including the final diagnosis and explicitly reported diagnostic reasoning.
| Specialty | Cases | Events | Figure-linked rate |
| Dermatology | 49 | 541 | 0.325 |
| Emergency Medicine | 25 | 307 | 0.218 |
| Internal Medicine | 24 | 293 | 0.270 |
| Nephrology | 29 | 366 | 0.235 |
| Neurology | 95 | 1,128 | 0.239 |
| Oncology | 112 | 1,284 | 0.321 |
| Ophthalmology | 43 | 518 | 0.311 |
| Radiology | 133 | 1,474 | 0.325 |
| All | 510 | 5,911 | 0.293 |
Table 1. Characteristics of DIAGTRACE.
| Category | Weight | Distribution |
| Pathology | 2.00 | 7.2% |
| Imaging | 1.50 | 22.7% |
| Laboratory tests | 1.00 | 14.9% |
| Microbiology | 1.00 | 2.7% |
| Procedure/intervention | 1.00 | 5.8% |
| Treatment/medication | 0.75 | 10.8% |
| Admission/course | 0.75 | 6.7% |
| Clinical history | 0.50 | 13.3% |
| Physical examination | 0.50 | 8.2% |
| Other/off-protocol | 0.25 | 7.7% |
Table 2. Request/response taxonomy
Data completeness validation. The reference resource is validated before use to ensure schema completeness, source grounding, ordering consistency, and leakage control. Source events must preserve provenance, populate required fields, and correctly mark diagnosis-revealing content for protection. The interactive experiments use separately extracted, source-quoted event banks available for 502 cases.
Quality audit. We audit 100 sampled cases using GPT-4.1 mini and independently repeat the evaluation with Gemini 2.5 Flash using identical frozen inputs, prompts, and scoring procedures. Both audits use three metrics (Appendix § 7.§ 1 and Prompt 2): (i) Finding recall, (ii) Event faithfulness, and (iii) Reasoning support. For comparison, we use strict scoring for finding recall and reasoning support, and lenient scoring for event faithfulness.
Human source-event audit. Two independent annotators reviewed 52 unique extracted events against the original PMC articles. We sampled eight articles from the 225-case evaluation subset, one per specialty, and selected five events per article. We additionally examined the complete 11-event trajectories of two articles, yielding 52 unique events after accounting for overlap. Annotators assessed source support and fidelity without access to automated judgments, event categories, or diagnostic-role labels, and also examined the complete trajectories for source-order consistency.
Interactive reconstruction protocol
A representative episode is shown in Figure 3. Each benchmark instance initializes two agents. The clinician agent is the evaluated agent and receives the initial presentation and the final diagnosis. The report agent is initialized separately and grounded in the reference resource. All interactions are text-only; image-related evidence is conveyed through source-derived text and figure captions rather than raw image pixels.
Figure 3. Example diagnosis-aware interactive reconstruction episode. The clinician agent is initialized with the initial presentation and known diagnosis, actively requests source-backed information, and reconstructs the diagnostic trajectory under a fixed turn budget.
At each interactive turn, the clinician agent issues a focused request, such as asking for additional history, examination findings, test results, or clinical-course information. Request types follow prior medical-agent and clinical-interaction benchmarks ([5, 9]) and are summarized in Table 2. The report agent returns at most one matching source-backed event from the reference resource, or an unavailable status. When multiple events match, the earliest unreleased eligible event in the clinical course is returned. The report agent cannot interpret findings, infer undocumented content, or reveal the final diagnosis.
The interaction terminates when the clinician agent submits a reconstructed diagnostic trajectory or exhausts the turn budget. The resulting trace records the clinician’s requests, report-agent responses, and the final reconstructed course.
The clinician-agent and report-agent prompts are provided in Prompt 3.
Evaluation protocol
We evaluate the interaction at the trajectory level. Both clinician requests and response events are mapped to a shared clinical information-category taxonomy (Table 2). This allows comparison between the predicted and reference clinical course even when wording differs.
Since not all categories contribute equally to diagnostic reconstruction, we heuristically assign each category a weight reflecting its relative importance. These weights are used consistently across all weighted evaluation metrics. Formal definitions are provided in Appendix § 7.§ 5.
Clinical Course Alignment. We report three metrics. (1) Tversky alignment measuring category-level coverage with penalties for missed and unnecessary requests. (2) Longest-common-sub-sequence recall measuring recovery of the source clinical course in relative order. (3) Pairwise order accuracy measuring whether two request pairs preserve source order.
Coverage and request behavior. To quantify whether the clinician agent requests the right clinical information categories compared with the source-derived evidence categories, we report precision, recall, and F1 using the request categories that are matched, missed, and additionally requested, respectively. We also report the Off-protocol rate and Redundant rate to characterize invalid or repeated information acquisition.
Event-level recall. To complement category-level metrics, we measure the fraction of eligible source events released during each interaction, excluding events already provided in the initial presentation. Event-level recall is computed separately for each case and then macro-averaged across the evaluation cases. Unlike category-level recall, this metric captures the retrieval of multiple distinct events within the same information category.
LLM-as-judge. We use LLM-as-judge to evaluate four 5-point dimensions (Prompt 4): (1) Factuality, (2) Validity, (3) Coherence, and (4) Utility.
Turn efficiency and availability. We report the mean turns used under the fixed 15-turn budget. We also track answer yield, the fraction of intermediate turns in which the report agent returns a source-backed event, and unavailable-turn rate, the fraction of intermediate turns in which no source-backed event is returned.
Fixed-checklist baseline. To examine whether category-level metrics reward non-adaptive information seeking, we evaluate an untuned fixed-checklist policy on 16 specialty-stratified cases from the matched subset. The policy issues the same sequence of clinical requests for every case without using the diagnosis, case content, or previous responses. It interacts with the unchanged report agent under the same 15-turn budget. For comparison, we truncate the checklist trace to each model’s case-specific number of acquisition turns and compute paired differences in macro-averaged event-level recall (Appendix § 7.§ 3).
Full-scale sensitivity. To assess the robustness of the matched-subset results, we additionally compute trajectory-alignment, weighted F1, and event-level recall metrics on each model’s full set of completed cases (Appendix § 7.§ 4).
Experimental setup
We evaluate four clinician agents: GPT-4.1 mini (OpenAI, 2025), Claude Haiku 4.5 [1], Gemini 2.5 Flash [7], and Kimi K2.5 (Moonshot AI, 2026) (Appendix § 7.6). Each interaction is run once per case with a maximum budget of 15 clinician turns. GPT-4.1 mini, Claude Haiku 4.5, Gemini 2.5 Flash, and Kimi K2.5 completed 502, 498, 502, and 225 cases, respectively. We report matched comparisons on the 225-case intersection, with Kimi’s lower completion count due to API failures and throughput limits rather than performance-based selection.
GPT-4.1 mini is used for benchmark construction, benchmark-quality audits, and FVCU quality evaluation. It also serves as the report agent during interactive reconstruction.
Results
Statistics of DIAGTRACE
DIAGTRACE contains 510 selected PMC case reports across eight specialties (Table 1). Source-grounded construction succeeded for all selected cases, producing 5,911 source events. Each case contains an average of 11.59 source events. The event bank is clinically diverse: the most frequent evidence types are imaging, history, routine laboratory tests, treatment response, physical examination, pathology, procedures, and outcomes (Table 2). A substantial fraction of events are linked to article figures or captions, reflecting the image-rich nature of PMC case reports.
Reference resource quality
LLM-based quality audit. Because DIAGTRACE relies on LLM-assisted construction, we audited 100 sampled cases using GPT-4.1 mini and independently repeated the evaluation with Gemini 2.5 Flash. GPT-4.1 mini achieved 0.982 finding recall, 0.999 event faithfulness, and 0.988 reasoning support, while Gemini obtained 0.955, 0.997, and 0.992, respectively. These scores use strict criteria for finding recall and reasoning support and lenient criteria for event faithfulness. The two audits agreed on 97.61% of event labels (1,143/1,171) and 98.30% of reasoning labels (520/529). The similar aggregate scores suggest that the quality estimates are not specific to GPT-4.1 mini, although both audits remain LLM-based.
Human source-event audit. Two independent annotators reviewed 52 unique extracted events against the original PMC articles and agreed on 48 labels (92.3%; Cohen’s ; weighted ). The annotators judged 50/52 and 52/52 events, respectively, to be supported by the source articles. For 20 events overlapping with the original GPT-4.1 mini audit, GPT’s faithfulness judgments agreed with the two annotators on 18/20 and 20/20 events, respectively. Both annotators identified discrepancies in article narrative order in the two fully audited trajectories (Appendix § 7.§ 2). These findings support the source grounding of the sampled events but do not establish complete finding coverage or source-order accuracy.
Interactive reconstruction results
Table 3 reports interactive reconstruction results on the 225-case subset. The models show differ ent trade-offs between category coverage, source-consistent ordering, and interaction efficiency. Kimi K2.5 achieves the highest Tversky alignment (0.399) and LCS recall (0.561), while Claude Haiku 4.5 achieves the highest pairwise order accuracy (0.690). GPT-4.1 mini and Gemini 2.5 Flash show comparable Tversky alignment (0.360 and 0.359, respectively). GPT-4.1 mini uses the fewest turns on average, whereas Gemini 2.5 Flash uses the most turns and has a higher unavailable-turn rate.
| Metric | GPT | Claude | Gemini | Kimi |
| Request-category alignment | ||||
| Tversky alignment | 0.360 | 0.334 | 0.359 | 0.399 |
| LCS recall | 0.465 | 0.510 | 0.500 | 0.561 |
| Pairwise order accuracy | 0.646 | 0.690 | 0.654 | 0.601 |
| Coverage decomposition and request behavior | ||||
| Precision | 0.626 | 0.560 | 0.607 | 0.623 |
| Recall | 0.525 | 0.582 | 0.573 | 0.672 |
| F1 | 0.592 | 0.589 | 0.603 | 0.638 |
| Event-level recall | 0.577 | 0.665 | 0.655 | 0.661 |
| Off-protocol rate | 0.000 | 0.003 | 0.005 | 0.002 |
| Redundant rate | 0.099 | 0.170 | 0.150 | 0.202 |
| Interactive FVCU quality | ||||
| Factuality | 4.81 | 4.82 | 4.75 | 4.85 |
| Validity | 4.84 | 4.84 | 4.78 | 4.87 |
| Coherence | 4.84 | 4.87 | 4.79 | 4.92 |
| Utility | 4.22 | 4.20 | 4.11 | 4.29 |
| Turn efficiency and availability | ||||
| Mean turns used | 9.93 | 11.31 | 11.88 | 10.28 |
| Answer yield | 0.525 | 0.611 | 0.544 | 0.642 |
| Unavailable-turn rate | 0.351 | 0.272 | 0.348 | 0.250 |
Table 3. Interactive reconstruction results.
Coverage decomposition and request behavior. Kimi K2.5 achieves the highest weighted recall (0.672) and F1 (0.638), while GPT-4.1 mini obtains the highest precision (0.626). GPT-4.1 mini also has the lowest redundant category rate, reflecting a more concise information-seeking strategy. Off-protocol request rates remain low across all models, suggesting that differences mainly arise from category selection, omission, ordering, and repeated requests rather than invalid interaction behavior.
Event-level recall. Claude Haiku 4.5 achieves the highest macro-averaged event-level recall (0.665), followed by Kimi K2.5 (0.661), Gemini 2.5 Flash (0.655), and GPT-4.1 mini (0.577). Although Kimi achieves the highest category-level alignment, Claude retrieves the largest proportion of eligible source events. This difference suggests that category-level alignment and event-level evidence acquisition capture complementary aspects of information-seeking behavior.
Fixed-checklist baseline. On 16 specialty-stratified cases, all four clinician models achieved higher event-level recall than the fixed checklist at matched turn budgets, with mean paired improvements of 0.022 (GPT), 0.097 (Claude), 0.159 (Gemini), and 0.207 (Kimi) (Table 4). The 95% paired-bootstrap confidence intervals excluded zero for Gemini and Kimi, but not for GPT or Claude. These exploratory results suggest that adaptive information seeking can improve evidence recovery beyond a fixed request sequence.
| Model | Model | Checklist | 95% CI | |
| GPT | 0.546 | 0.524 | +0.022 | [−0.055, 0.104] |
| Claude | 0.638 | 0.540 | +0.097 | [−0.019, 0.216] |
| Gemini | 0.689 | 0.531 | +0.159 | [0.047, 0.277] |
| Kimi | 0.656 | 0.449 | +0.207 | [0.108, 0.327] |
Table 4. Event-level recall on the 16-case fixed-checklist evaluation. Checklist scores are matched to each model’s case-specific acquisition-turn budget. Differences and confidence intervals are based on paired case-level comparisons.
Category-wise results. Laboratory, microbiology, imaging, and pathology evidence is requested relatively consistently across models, whereas treatment-related and physical-examination evidence is requested less frequently. Procedure/intervention and admission/course categories show greater variation across models (Figure 2).

Figure 2. Category-wise recall. Each cell is the fraction of cases in which a source-derived category was requested by the clinician agent.
Full-scale sensitivity. Results on the full completed-case sets (502 GPT, 498 Claude, and 502 Gemini cases) were broadly consistent with the matched-225 results, with absolute changes of at most 0.037 across five principal metrics (Table 5). Full-scale estimates are descriptive because the case sets differ across models.
| Model () | Tversky | LCS | Pair. | F1 | Event R |
| GPT (225) | .360 | .465 | .646 | .592 | .577 |
| GPT (502) | .393 | .502 | .669 | .618 | .575 |
| Claude (225) | .334 | .510 | .690 | .589 | .665 |
| Claude (498) | .362 | .537 | .679 | .610 | .666 |
| Gemini (225) | .359 | .500 | .654 | .603 | .655 |
| Gemini (502) | .380 | .535 | .670 | .622 | .654 |
| Kimi (225) | .399 | .561 | .601 | .638 | .661 |
Table 5. Matched-subset and full completed-case sensitivity results. Pair. denotes pairwise order accuracy; Event R denotes event-level recall. Full-case estimates are not strictly paired across models.
Overall, the results suggest that stronger category coverage does not always coincide with the best pairwise ordering or the shortest interaction.
Conclusion
We introduced DIAGTRACE, a source-grounded benchmark for evaluating interactive reconstruction of diagnostic trajectory from clinical case reports. Unlike static diagnosis benchmarks, it measures how models acquire supporting information. Experiments on 225 cases reveal trade-offs in coverage, ordering, and interaction efficiency.
Limitations
The benchmark also has several limitations. First, source order reflects the published article narrative rather than an adjudicated clinical timeline. Human auditing also identified article-order discrepancies that may affect order-sensitive metrics, including LCS recall and pairwise order accuracy. Second, clinical case reports are selective summaries and may not reflect routine workflows or the full space of diagnostic interactions. Third, benchmark construction and quality auditing rely in part on LLM assistance, making continued validation important.
Ethical Considerations
DIAGTRACE is intended solely for research evaluation and not for clinical decision support or patient care. It is constructed from publicly available PMC case reports and may inherit their selection and reporting biases. Because the clinician agent is given the final diagnosis, benchmark scores measure diagnostic-trajectory reconstruction rather than autonomous diagnostic ability and should not be interpreted as evidence of clinical safety. LLM-assisted benchmark construction and evaluation may also introduce residual errors despite the validation procedures.
Acknowledgments
We thank Ranxuan Li and Bicheng Chen from Weill Cornell Medicine for their contributions to the human validation of the benchmark, including the independent audit of source–event mappings. We also thank the anonymous reviewers and the area chair for their constructive feedback. This research was supported in part by the National Institutes of Health (NIH) under grants R01LM014306, R01LM014573, and UG3TR005954.
Appendix
Reference Resource Audit Metrics
We audit a sampled subset of benchmark references to assess source grounding, leakage control, and clinical usability.
Finding recall measures whether the event bank captures the major diagnosis-relevant findings. Under strict scoring, it is the fraction of diagnosis-relevant findings in the source case report that are judged fully represented in the extracted event bank: .
Event faithfulness assesses whether extracted events are supported by source quotes. Under lenient scoring, it is the fraction of extracted events that are judged faithful or partially faithful to their supporting source quotations: .
Reasoning support evaluates whether the event bank contains information supporting the article-derived diagnostic rationale. Under strict scoring, it is the fraction of article-derived diagnostic reasoning items that are judged fully supported by the extracted event bank: .
Supplementary Audit Findings
Article-order discrepancies. Both annotators identified article-order discrepancies in the two fully reviewed trajectories (PMC10906755 and PMC9721097). Although the extracted events were source-supported, their ordering differed from the published narratives. These discrepancies concern article presentation order rather than verified clinical chronology and may affect order-sensitive evaluation metrics.
Fixed-Checklist Baseline
Sampling and protocol. We sampled 16 cases from the shared 225-case subset, with two cases per specialty, using a fixed random seed (20260916). The checklist followed 14 acquisition requests in a fixed order: history, physical examination, laboratory tests, microbiology, imaging, pathology, procedures, treatment response, clinical course, and then repeated history, examination, laboratory tests, microbiology, and imaging. The final turn was reserved for trajectory submission. The policy did not access case-specific information or previous responses and used the unchanged GPT-4.1 mini report agent. All 16 checklist episodes completed successfully.
Turn matching and evaluation. To account for differences in interaction length, each checklist trajectory was truncated to the corresponding clinician model’s number of non-final turns for that case. Event-level recall was calculated using unique event IDs returned with a found status, divided by the number of eligible reference events not included in the initial presentation. We calculated recall separately for each case and averaged across the 16 cases. Model-minus-checklist differences were computed for each paired case, with 95% confidence intervals estimated using 10,000 paired bootstrap resamples.
Full-scale Sensitivity Analysis
We recomputed the evaluation metrics from saved completed episodes using the corrected request-category classifier and canonical specialty–PMCID case identities. Table 5 compares results on the shared 225-case subset with each model’s full set of completed cases.
Across GPT-4.1 mini, Claude Haiku 4.5, and Gemini 2.5 Flash, absolute changes in the five principal metrics were at most 0.037. Kimi K2.5 had no additional completed cases beyond the matched subset. The matched subset was more balanced across specialties, whereas the remaining completed cases were concentrated in radiology and oncology. Thus, the stability of aggregate results does not establish that the matched subset is representative of the full dataset.
Evaluation metrics
Clinical Course Alignment Metrics
The interaction trace is converted into a clinical-request sequence and a reference sequence . Both requests and responses are mapped to the shared taxonomy (Table 2). Overly broad requests or requests for retrospective diagnostic justification are mapped to Other/off-protocol.
To emphasize category coverage and clinical course order, adjacent duplicate categories are first collapsed, and only the first occurrence of each category is retained. The resulting compact sequences therefore contain at most one occurrence of each request category.
The initial presentation is removed because it is provided before the interaction begins. Final-turn request/response and system-control turns (e.g., generic wait or reassessment without a concrete information request/response) are also excluded.
Reassessment turns that explicitly request documented clinical-course updates, follow-up status, interval change, or treatment response are retained. Requests that receive unavailable responses are also retained, because they reflect the clinician agent’s information-seeking strategy under a fixed turn budget.
Evaluation compares the generated sequence with the reference sequence using the following metrics.
Tversky alignment measures category coverage while penalizing both missed and unnecessary requests. Let , , and denote the total weights of categories that appear in both and , only in , and only in , respectively. The score is defined as:
Following [5], we set and , placing a stronger penalty on unnecessary requests because they consume the limited turn budget.
Longest-common-subsequence (LCS) recall measures the extent to which is recovered in the correct relative order:
Because this metric is recall-oriented, additional requests do not directly reduce the score, but missing requests and order changes can reduce the score.
Pairwise order accuracy evaluates ordering consistency among shared categories. Let be the set of category pairs such that precedes in and both categories also occur in . Each pair receives weight:
Let indicator denote if precedes in . The score is defined as:
Cases with fewer than two shared categories are excluded.
Coverage and request behavior metrics
For each case, let , , and denote the total weights of request categories that are matched, missed, and additionally requested, respectively, as defined in the weighted Tversky alignment. The metrics are defined as:
Off-protocol rate measures the proportion of intermediate interaction turns that produce an off-protocol report-agent status:
where is the number of intermediate turns containing an overly broad, invalid, or overly diagnostic request, and is the total number of intermediate interaction turns.
Redundant rate measures whether the clinician agent re-issues a request. Let denote the request sequence after collapsing adjacent duplicate requests. A request is considered redundant when it has appeared previously in . The redundant rate is defined as:
where is the number of redundant category requests.
Implementation details
Each interactive reconstruction used a fixed budget of 15 clinician-report turns. Each clinical agent was evaluated once per case without sampling multiple stochastic trajectories.
All API requests used a 600-second timeout. For Claude Haiku 4.5, Gemini 2.5 Flash, and Kimi K2.5, we set temperature to 0 and the maximum output length to 4,096 tokens. Top-, seed, and reasoning-effort parameters were left at their provider defaults.
GPT-4.1 mini was accessed through Azure OpenAI using JSON-mode chat-completion calls with the deployment-level decoding configuration. Temperature, top-, maximum output length, seed, and reasoning effort were not specified.
Transient connection and timeout failures were retried using the same prompts and inference configuration.
Data and Code Availability
The DiagTrace benchmark data and interaction traces are publicly available at https://github.com/Fanncg1229/Diagtrace. The corrected camera-ready release includes the canonical event banks, matched and full-scale interaction episodes, evaluation manifests, and fixed-checklist artifacts under camera_ready_release/. The evaluation code, corrected aggregate results, and reproducibility instructions are available at https://github.com/Fanncg1229/DiagTrace-code. The repositories provide instructions for reproducing the deterministic evaluation metrics and figures.
Prompts
Prompt 1: Reference resource construction
Given the complete PMC case report, construct a source-grounded reference resource for an interactive diagnostic benchmark.
Extract: (1) an initial presentation that does not reveal the final diagnosis; (2) a source-ordered event bank of documented clinical findings; (3) clinical-course information derived from the event bank; and (4) article-derived diagnostic references, including the final diagnosis and diagnostic reasoning when explicitly available.
Each source event must be grounded in the report text and include provenance, source order, evidence type, diagnostic role, and figure or caption linkage when applicable. Do not infer unsupported findings, introduce external medical knowledge, or treat article order as an adjudicated clinical timeline. Mark diagnosis-revealing content for protected handling. Return valid JSON only.
Prompt 2: Reference resource quality audit
You are auditing a source-grounded reference resource for an interactive medical diagnosis benchmark. Evaluate benchmark data quality, not downstream model performance. Use only the supplied source report text and extracted reference resource. Do not add external medical facts.
First, identify a compact inventory of 8–15 major diagnosis-relevant findings from the source report. The inventory should span the diagnostic trajectory when present, including initial presentation, history, examination, laboratory or microbiology results, imaging, pathology or molecular confirmation, procedures, treatment response, follow-up, and clinical course.
For each source finding, decide whether it is represented in the source-ordered event bank using one of four labels: full, partial, none, or contradicted.
For each extracted event, decide whether it is faithful to its supporting source quotation using one of four labels: faithful, partial, unsupported, or contradicted.
For each article-derived diagnostic reasoning item, decide whether it is supported by the source-ordered event bank and diagnostic references using full, partial, none, or contradicted.
Return valid JSON only.
Prompt 3: Interactive reconstruction protocol
Diagnosis-aware clinician agent. You are given an initial presentation and the article-reported final diagnosis. Your task is not to re-diagnose the patient, but to reconstruct the source-derived diagnostic trajectory that supports the known diagnosis. At each non-final turn, issue one focused clinical request, such as asking about history, examination findings, tests, documented results, procedures, treatment response, follow-up, or clinical-course updates.
Use the known diagnosis to guide information seeking, but treat only report-agent replies as observed case information. Do not treat textbook expectations or general medical knowledge as facts about the case. Maintain a working trajectory reconstruction, confirmed findings, remaining information gaps, and confidence in the sufficiency of the acquired information. When sufficient information has been acquired, submit a final reconstructed trajectory grounded only in released event identifiers. Return valid JSON only.
Source-grounded report agent. You are given the source-ordered event bank extracted from the case report. For each clinician request, return at most one responsive, previously unreleased, source-backed event, or return an unavailable status if no eligible event is documented. When multiple events match, prefer the earliest eligible event in preserved source order.
Do not diagnose, interpret clinical significance, recommend next steps, infer undocumented facts, or expose retrospective diagnostic synthesis. Do not convert absence from the source into a negative clinical finding. Return valid JSON only.
Prompt 4: Interactive FVCU quality audit
You are evaluating an interactive diagnostic-trajectory reconstruction episode. Evaluate the diagnosis-aware clinician agent, not the report agent. Use only the provided reference resource and interaction trace.
Score four FVCU dimensions on a 1–5 scale.
Factuality: whether the clinician agent’s case-specific claims, confirmed findings, and final reconstruction are grounded in report-agent responses available by that turn.
Validity: whether the clinician agent’s requests and interpretations are clinically reasonable given the known diagnosis and acquired information.
Coherence: whether the working reconstruction, confirmed findings, missing information, and confidence evolve consistently across turns.
Utility: whether the clinician agent uses the fixed turn budget to acquire high-yield documented information needed to reconstruct the source-derived diagnostic trajectory.
Return valid JSON only, including one score and brief rationale for each dimension.
Initialization
INITIAL PRESENTATION
69-year-old woman with abdominal pain, back pain, and rectal tenesmus.
KNOWN DIAGNOSIS
Stanford Type A aortic dissection.
CLINICIAN RECEIVES
Initial presentation + known final diagnosis.
Turn 1 | Request physical-examination evidence
CLINICIAN AGENT · REQUEST EXAM
Abdominal exam findings, pulse discrepancy, palpable mass?
REPORT AGENT · SOURCE-BACKED RESPONSE
Diffuse abdominal tenderness; no pulse discrepancies.
ACQUIRED EVIDENCE + REMAINING GAPS
Initial symptoms plus exam findings suggest an acute abdominal/vascular process; key imaging still missing.
Turn 2 | Request imaging evidence
CLINICIAN AGENT · REQUEST RESULT
Results of chest/abdominal CT with contrast?
REPORT AGENT · SOURCE-BACKED RESPONSE
CT showed Stanford Type A aortic dissection from aortic root to aortic bifurcation.
WORKING RECONSTRUCTION
Known diagnosis is now supported by source-backed imaging evidence.
Turn 3 | Request clinical-course evidence
CLINICIAN AGENT · REQUEST COURSE
Subsequent treatment, clinical course, and outcome?
REPORT AGENT · SOURCE-BACKED RESPONSE
Blood pressure and heart rate control; emergent cardiothoracic surgery; full recovery.
WORKING RECONSTRUCTION
Trajectory now includes presentation, confirmatory imaging, management, and outcome.
Final reconstructed trajectory
EVIDENCE PATH
Initial abdominal/back pain and rectal tenesmus -> exam with diffuse abdominal tenderness -> CT-confirmed Stanford Type A aortic dissection -> blood pressure control and emergent surgery -> full recovery.
References
- [1]Anthropic. 2025. Claude Haiku 4.5. https://platform.claude.com/docs/en/models/haiku-4-5/overview. Claude model documentation. Accessed: 2026-08-02.
- [2]Xi Chen, Hanyu Zhou, Huahui Yi, Mingke You, WeiZhi Liu, Li Wang, Ziyuan Qin, Hairui Li, Xue Zhang, Yingman Guo, Songlin Li, Yunan Hu, Qianjiang Xiong, Rongxiao Li, Lei Fan, Qicheng Lao, Weili Fu, Jian Li, and Kang Li. 2026. Grounding large language models in clinical diagnostics. Nature Communications, 17:4401.DOI
- [3]Yera Choi, Yeong Hwa Kim, JaeDeok Lee, Taekang Kim, Sangdoo Yun, and Seong-Eun Moon. 2026. ASCENT: A benchmark for evaluating and advancing stepwise diagnostic reasoning in large language models on common clinical scenarios. In Proceedings of the 7th Conference on Health, Inference, and Learning, volume 333 of Proceedings of Machine Learning Research, pages 404–426. PMLR.
- [4]Zhihao Fan, Lai Wei, Jialong Tang, Wei Chen, Siyuan Wang, Zhongyu Wei, Jun Xie, Fei Huang, and Jingren Zhou. 2025. AI hospital: Benchmarking large language models in a multi-agent medical interaction simulator. In Proceedings of the 31st International Conference on Computational Linguistics, pages 10183–10213, Abu Dhabi, UAE. Association for Computational Linguistics.
- [5]Dyke Ferber, Lars Hilgers, Christiane Höper, Benedict Kinny-Köster, Jan-Niklas Eckardt, Katharina Egger-Heidrich, Marius Bill, Martin M. K. Schneider, Jan Clusmann, Lejla Kadric, Marcel Oehme, Maximilian Mayrhofer-Schmid, Alexander Oeser, Georg Wölflein, Isabella C. Wiest, Jan Moritz Miedeke, A. John Iafrate, Daniel Truhn, Dirk Jäger, and Jakob Nikolas Kather. 2026. Towards autonomous medical artificial intelligence agents. Nature, 655:1282–1291.DOI
- [6]Yicheng Gao, Xiaolin Zhou, Yahan Li, Yue Zhao, and Ruishan Liu. 2026. MedExAgent: Training LLM agents to ask, examine, and diagnose in noisy clinical environments. Preprint, arXiv:2605.07058.arxiv.org/abs/2605.07058
- [7]Google. 2026. Gemini 2.5 Flash. https://ai.google.dev/gemini-api/docs/models/gemini-2.5-flash. Gemini API model documentation. Accessed: 2026-08-02.
- [8]Hsin-Ling Hsu, Zizheng Wang, Donghua Zhang, Nai-Chia Chen, Jerry Wang, Jun-En Ding, Chia-Hsuan Hsu, Guoan Wang, Feng Liu, Fang-Ming Hung, Chenwei Wu, and Liyue Shen. 2026. MedAction: Towards active multi-turn clinical diagnostic LLMs. Preprint, arXiv:2605.07305.arxiv.org/abs/2605.07305
- [9]Shuyue Stella Li, Vidhisha Balachandran, Shangbin Feng, Jonathan S. Ilgen, Emma Pierson, Pang Wei Koh, and Yulia Tsvetkov. 2024. MediQ: Question-asking LLMs and a benchmark for reliable interactive clinical reasoning. In Advances in Neural Information Processing Systems, volume 37, pages 28858–28888.arxiv.org/abs/2406.00922
- [10]Cheng Liang, Pengcheng Qiu, Ya Zhang, Yanfeng Wang, Chaoyi Wu, and Weidi Xie. 2026. Evaluating large language models in dynamic clinical decision-making with standardized patient cases. arXiv preprint arXiv:2606.05112.arxiv.org/abs/2606.05112
- [11]Ruoyu Liu, Kui Xue, Xiaofan Zhang, and Shaoting Zhang. 2025. Interactive evaluation for medical LLMs via task-oriented dialogue system. In Proceedings of the 31st International Conference on Computational Linguistics, pages 4871–4896, Abu Dhabi, UAE. Association for Computational Linguistics.

