betaFeedback

WorksComputer Science

ClinVerse: A Realistic Multi-Agent Diagnostic Environment for Training and Evaluating Medical Agents

Abstract

Disease diagnosis is a key application of large language model (LLM) agents in healthcare. However, existing medical agents are primarily trained and evaluated with static question-answering data, providing limited opportunities to learn active information acquisition, examination decision-making, and dynamic clinical interaction. To address this limitation, we present ClinVerse, an interactive, real-data-grounded multi-agent environment designed for training and evaluating medical diagnostic agents. Built on ClinPatient, a scalable dataset comprising 150k diagnostic cases derived from real-world electronic health records, ClinVerse supports coordinated interactions among six specialized clinical roles. Specifically, a doctor agent starts from an initial patient complaint, progressively gathers clinical evidence through patient interviews and examination requests, and ultimately produces a diagnosis. For each interaction trajectory, a rule-based scoring framework quantifies information acquisition, examination ordering, and diagnostic accuracy. These trajectory-level scores serve as reward signals for agent training while also enabling scalable evaluation. In experiments, we evaluate various general-purpose and medical-domain models and show that nine self-evolution methods across three categories consistently improve agents' diagnostic capabilities within ClinVerse. We further analyze the interplay among model capability, environment feedback, and self-evolution methods.

Introduction

To alleviate the shortage of healthcare resources, medical agents driven by large language models (LLMs) are increasingly being integrated into modern clinical practice, with assisting diagnosis emerging as one of their most important applications [6, 28]. To ensure their reliability in real-world clinical settings, extensive research has been devoted to evaluating and advancing their diagnostic capabilities [8, 32, 35].

As shown in Table 1, early studies predominantly assess and train medical agents within static single-turn question-answering (QA) formats derived from medical licensing exams [9, 20, 40, 48]. This paradigm provides medical agents with pre-defined clinical information and requires them to directly formulate a diagnosis, offering standardization and efficiency in model training and evaluation. However, in real-world clinical practice, physicians must actively gather information through multi-turn patient consultations, coordinate with laboratory and radiology departments to request diagnostic tests, and progressively refine their diagnostic hypotheses based on newly acquired evidence [2, 12]. This fundamental discrepancy highlights that static QA tasks primarily assess medical agents’ diagnostic reasoning capabilities while overlooking their information acquisition capabilities, thereby failing to fully capture the dynamic and interactive nature of clinical diagnosis.

WorkVenueCasesData SourceRolesScalableReal DataSME
Static Question-answering
MedQA [9]arXiv 202061kMedical licensing exams—❌✅❌
CMExam [20]NeurIPS 202368kChinese medical licensing exam—❌✅❌
MedXpertQA [48]ICML 20254,460Medical examinations—❌✅❌
EHRBench [40]KDD 2026960kMIMIC-III, MIMIC-IV, etc.—✅✅❌
Interactive Clinical Environments
Agent Hospital [14]arXiv 202410kLLM-generated cases🧑‍⚕️ 👩‍🔬 👩‍⚕️ 🧑‍⚖️ ⚙️✅❌✅
AIHospital [4]COLING 2025506Web-based clinical cases🧑‍⚕️ 👩‍🔬 🧑‍⚕️ 🧑‍⚖️❌✅❌
SDBench [27]Microsoft AI 2025304NEJM-CPC🧑‍⚕️ 👩‍⚕️ 🧑‍⚖️❌✅✅
MedSP [16]arXiv 20261,638MedEdPORTAL🧑‍⚕️ 👩‍⚕️ 🧑‍⚖️❌✅❌
MIRA [5]Nature 2026574Expert-processed MIMIC-IV🧑‍⚕️ 👩‍⚕️ 🧑‍⚖️❌✅❌
ClinVerseOurs150kMIMIC-IV, SICdb, etc.🧑‍⚕️ 👩‍🔬 👩‍⚕️ 🧑‍ patient 🧑‍⚖️ ⚙️✅✅✅

Table 1. Comparison of static medical QA and interactive clinical environments. ClinVerse is grounded in 150k scalable real-world patient data, and supports predicting missing examination results to enable flexible interactions of 6 roles. Role icons denote medical agent, laboratory physician, nurse, patient, judge, and environment controller. SME means supporting missing examinations.

These limitations motivate a shift from static QA toward interactive simulations of real-world clinical environments that better reflect real-world multi-role interactions. Within such environments, medical agents can continuously improve their diagnostic reasoning capability and information acquisition capability, through iterative interaction, feedback, and self-evolution. Compared with static QA, this shifts the training paradigm from taking medical exams to undergoing clinical residency training.

Despite this promising vision, constructing interactive clinical environments remains challenging. The first challenge is how to prevent information leakage while supporting open-ended clinical interactions. During patient simulation, granting simulators access to complete clinical records may inadvertently expose ground-truth diagnoses or unrevealed examination results, undermining the authenticity of clinical interactions. Moreover, when agents request examinations absent from electronic health records (EHRs), existing environments [4, 16, 27] often return "unavailable" or "normal", potentially exposing missingness patterns that agents can exploit as diagnostic shortcuts. These issues highlight the difficulty of preserving controlled information access while enabling flexible clinical interactions.

The second challenge is how to provide reliable trajectory-level feedback for evaluating and improving medical agents.

Unlike static QA tasks that primarily assess final-answer correctness, interactive clinical diagnosis involves multi-turn trajectories of information acquisition, examination decisions, and diagnostic reasoning, requiring evaluation of both the clinical decision-making process and its outcomes [23, 35]. However, expert-based evaluation, as adopted in MIRA [5], is costly and difficult to scale, while LLM-as-a-judge approaches [4, 14] may introduce model-dependent biases and sensitivity to prompting strategies [36, 47].

To address these challenges, we propose ClinVerse, an interactive, EHR-grounded multi-agent environment designed for training and evaluating medical diagnostic agents. To achieve scalability and open-endedness while preserving clinical grounding, we develop an EHR-to-environment transformation framework that converts real-world clinical records into interactive diagnostic environments. Specifically, an automated data construction pipeline enables the scalable construction of ClinPatient, a dataset comprising 150k EHR-derived patient cases, reducing dependence on extensive manual curation. We further introduce a dual-layer patient simulator with a self-check mechanism that separates patient-observable information from latent clinical states, mitigating unintended disclosure of privileged evidence. Meanwhile, missing examination results in EHRs are addressed by ClinLab, a trained physiological-state predictor that infers clinically plausible results.

To provide scalable and reproducible trajectory-level feedback, we develop a rule-based evaluation framework grounded in structured clinical evidence and ICD-10 diagnosis labels. Rather than relying on subjective judgments of entire conversations, this framework evaluates interaction trajectories using deterministic rules across three complementary dimensions: information acquisition, examination coverage, and diagnostic accuracy. These metrics characterize both the evidence acquisition process and the resulting diagnostic outcomes, providing consistent feedback for agent evaluation and subsequent self-evolution.

Built upon these components, ClinVerse coordinates six specialized roles to simulate the core workflow of clinical diagnosis, where a doctor agent, coordinated by the environment controller, progressively acquires patient information through multi-turn consultations with the observable patient simulator, requests examination results from the laboratory physician supported by the latent patient simulator, and ultimately formulates a diagnosis. The resulting interaction trajectory is then evaluated by a judge.

Consequently, ClinVerse can serve as both an evaluation platform and a training environment for medical agents. We evaluate nine general-purpose and medical-domain agents, identifying their strengths and limitations in interactive diagnosis. We further investigate multiple self-evolution methods across three levels, demonstrating that agents can improve their capabilities by learning from interaction trajectories generated within ClinVerse. Beyond performance improvements, our analyses reveal that evidence acquisition remains a major bottleneck, while collecting more information does not necessarily lead to better diagnoses, highlighting the importance of evidence prioritization and integration. We also examine how trajectory effectiveness depends on the capabilities of source and target models, and whether learned skills transfer to other clinical environments.

Our main contributions are summarized as follows:

  • We introduce ClinVerse, a realistic diagnostic environment that supports open-ended multi-turn interactions among six specialized roles and enables both the self-evolving and evaluation of medical agents.

  • We develop a scalable pipeline for transforming EHR data into diagnostic cases and use it to construct ClinPatient, a dataset containing 150k patient cases. We design a dual-layer patient simulator with a self-check mechanism and train a physiological-state predictor to infer missing examination results.

  • We demonstrate the effectiveness of ClinVerse as a training environment and validate its reliability as an evaluation platform, and further analyze the interplay among the environment, model, and self-evolution methods.

Setup of ClinVerse

As illustrated in Figure 1, ClinVerse is a realistic clinical diagnostic environment grounded in real-world EHR data. It comprises six specialized roles that interact freely throughout the consultation, enabling the doctor agent to progressively gather evidence and produce a final diagnosis. The resulting interaction trajectories are evaluated using three categories of rule-based criteria and can subsequently be utilized for both model assessment and training.

Overview of ClinVerse

Figure 1. An overview of ClinVerse. Built upon 150k EHR-derived dataset ClinPatient, a dual-layer patient simulator ensures controlled information disclosure, while the laboratory physician retrieves recorded examination results or predicts missing ones using ClinLab. ClinVerse integrates six specialized roles to support multi-turn open-ended clinical interaction. The resulting interaction trajectories are evaluated using three-dimensional rule-based metrics and further utilized for three levels of agent self-evolution strategies.

Task Definition

For each patient, we construct an EHR-derived clinical case as

C=(U,E,y),(1)C = (\mathcal{U}, \mathcal{E}, y), \tag*{(1)}

where U\mathcal{U} and E\mathcal{E} denote the patient’s clinical information units and examination items, respectively, and y∈YICDy \in\mathcal{Y}_{\mathrm{ICD}} is the ground-truth diagnosis represented by an ICD-10 code.2

The clinical information unit set is defined as

U={u1,u2,…,uM},(2)\mathcal{U} = \{u_1, u_2, \ldots, u_M\}, \tag*{(2)}

where each umu_m represents patient-specific clinical information, such as a symptom, medical history, medication, or physical sign. At the beginning of a consultation, the doctor is provided with an initial subset U0⊂U\mathcal{U}_0 \subset\mathcal{U}, which contains the patient’s chief complaint and other initially disclosed information, while the remaining information units are hidden and can be obtained through interaction with patients.

The examination item set is defined as

E={e1,e2,…,eN},(3)\mathcal{E} = \{e_1, e_2, \ldots, e_N\}, \tag*{(3)}

where each ene_n denotes an examination item, such as a laboratory test, vital sign measurement, imaging examination, or other auxiliary examination. The doctor can retrieve an item ene_n by requesting the corresponding examination.

At turn tt, the doctor receives the currently collected evidence

xt=(Ut,Et),(4)x_t = (\mathcal{U}_t, \mathcal{E}_t), \tag*{(4)}

where U0⊆Ut⊆U\mathcal{U}_0 \subseteq\mathcal{U}_t \subseteq\mathcal{U} and Et⊆E\mathcal{E}_t \subseteq\mathcal{E} are the revealed patient information units and retrieved examination items, respectively. The doctor can query the patient for additional information, request examinations, or terminate the interaction by issuing a diagnosis.

The task output is a predicted ICD-10 code:

y^∈YICD.(5)\hat{y} \in\mathcal{Y}_{\mathrm{ICD}}. \tag*{(5)}

where the ICD-10 code inherently specifies both the disease category and its fine-grained subtype.

Overall, the task requires the doctor agent to interact iteratively with the patient to collect information units and request relevant examinations to learn about the patient’s physiological state. Once the agent determines that sufficient information has been collected, it provides a final diagnosis of the patient’s underlying disease.

Six Roles in ClinVerse

To simulate realistic clinical interactions while maintaining controlled information access, ClinVerse comprises six specialized roles: an observable patient simulator, a latent patient simulator, a laboratory physician, a doctor agent, an environment controller, and a diagnostic judge. As illustrated in Figure 1, the doctor agent interacts with the patient simulators and laboratory physician through the environment controller, progressively acquiring clinical evidence before submitting a final diagnosis. The diagnostic judge evaluates the resulting interaction trajectory, providing feedback for agent evaluation and subsequent self-evolution.

Observable Patient Simulator. Existing patient simulators [13, 27] often grant a single model access to the complete patient record, increasing the risk of information leakage under leading or adversarial queries. To address this issue, we adopt a dual-layer patient simulation architecture that separates patient-observable information from latent clinical states. The observable patient simulator is restricted to patient-reportable information, including demographics, chief complaints, subjective symptoms, medical history, and personality traits, while excluding ground-truth diagnoses, clinician-only records, and unrevealed examination results. A self-check mechanism further verifies each generated response against the patient’s accessible information and underlying case evidence, revising or rejecting privileged, unsupported, or premature disclosures. Validly revealed information units are recorded for subsequent trajectory-level evaluation.

Latent Patient Simulator. Complementing the observable patient simulator, the latent patient simulator maintains the complete EHR-derived case state, including the ground-truth diagnosis, longitudinal medical history, physiological measurements, and examination results. It does not interact directly with the doctor agent but provides case-specific clinical information to authorized roles, particularly the laboratory physician. By separating patient-reportable evidence from privileged clinical records, the dual-layer architecture preserves the partial observability of clinical diagnosis and mitigates information leakage. The effectiveness of this design, together with the self-check mechanism, is evaluated in § 6.1.

Laboratory Physician. The laboratory physician handles physical examination requests, laboratory tests, and imaging examinations issued by the doctor agent. As illustrated in Figure 1, each request follows a two-stage process: if the corresponding result is recorded in the original EHR, the laboratory physician retrieves it from the latent patient simulator; otherwise, it invokes ClinLab, a trained physiological-state predictor, to infer a clinically plausible result from the case context and available observations. This design avoids directly returning missingness responses for unrecorded examinations, as in prior environments [4, 5], thereby supporting flexible examination requests while reducing the risk of exploiting systematic missingness as a diagnostic shortcut. Both recorded and predicted results are standardized into structured reports, including measurements, units, reference ranges, and abnormality indicators when applicable. The training and evaluation of ClinLab are detailed in Appendix I and Section 6.2, respectively.

Doctor Agent. The doctor agent is the primary role evaluated and trained within ClinVerse. Starting from a limited initial patient presentation, it progressively acquires clinical evidence and formulates diagnostic hypotheses through multi-turn interactions. Its action space comprises three types: patient inquiry, examination requests, and final diagnosis submission. Examination requests cover physical examinations, laboratory tests, microbiological tests, and imaging studies. At each turn, the doctor selects an action based on the accumulated interaction history, reflecting the sequential process of clinical evidence acquisition and diagnostic reasoning.

reasoning. The consultation terminates when the doctor submits a final diagnosis or reaches the interaction limit.

Environment Controller. The environment controller serves as the central coordinator of clinical interactions, connecting the doctor agent with the supporting roles while enforcing information access boundaries. It validates the doctor’s actions, routes patient inquiries to the observable patient simulator and examination requests to the laboratory physician, and returns the corresponding observations to the doctor. Throughout the consultation, it maintains the interaction history and records information disclosure and examination events as hidden metadata for subsequent evaluation. This centralized coordination ensures that the doctor obtains clinical evidence only through authorized interactions, preserving the partial observability and consistency of the diagnostic process.

Diagnostic Judge. The diagnostic judge evaluates the complete interaction trajectory without participating in the consultation. Using the ground-truth case information and interaction metadata maintained by the environment controller, it applies deterministic scoring rules across three dimensions: patient information collection, examination coverage, and diagnostic accuracy at different levels of disease granularity. These scores provide reproducible feedback on both evidence acquisition and diagnostic outcomes, supporting systematic agent evaluation and subsequent self-evolution. The scoring criteria are detailed in Section 2.4.

Interaction Flow

As illustrated in Figure 1, each diagnostic episode begins with the environment controller initializing an EHR-derived case from ClinPatient and providing the doctor agent with a limited initial presentation. At each turn, the doctor selects a clinical action based on the accumulated interaction history, which is validated and routed by the environment controller. Patient inquiries are directed to the observable patient simulator, which generates self-checked responses based on patient-accessible information. Physical examination and test requests are handled by the laboratory physician, which retrieves recorded findings from the latent patient simulator or invokes ClinLab to predict unavailable results. The returned observations are incorporated into the interaction history, allowing the doctor to progressively update its diagnostic reasoning and determine subsequent actions.

Rule-based Scoring

Given a complete interaction trajectory of TT turns, we evaluate agent behavior along three dimensions: patient information collection, examination coverage, and diagnostic agreement with the hidden case label. We design deterministic scoring rules to enable reproducible and scalable evaluation while avoiding the inconsistency of LLM-based judges and the cost of expert annotation. These metrics assess how effectively agents acquire and utilize case-specific evidence, rather than judging the clinical optimality of every possible decision path.

Information Collection Rate. Let U\mathcal{U}, whose extraction details are provided in Section 3.3, denote the normalized set of patient-reportable information units extracted during case construction, and let U0⊆U\mathcal{U}_0 \subseteq\mathcal{U} denote the subset already disclosed in the initial presentation. The remaining units,

Uask=U∖U0,(6)\mathcal{U}^{\mathrm{ask}} = \mathcal{U} \setminus\mathcal{U}_0, \tag*{(6)}

represent the information that can be elicited through subsequent interaction with the observable patient simulator.

Each unit is assigned an importance weight w(u)∈{3,2,1}w(u) \in\{3,2,1\}, corresponding to key, supporting, and background evidence, respectively. Let UTrev⊆Uask\mathcal{U}^{\mathrm{rev}}_T \subseteq\mathcal{U}^{\mathrm{ask}} denote the set of distinct units revealed to the doctor over the current trajectory. We define the information collection rate as

Rinfo=∑u∈Uaskw(u)1[u∈UTrev]∑u∈Uaskw(u).(7)R_{\mathrm{info}} = \frac{\sum_{u \in\mathcal{U}^{\mathrm{ask}}} w(u)\mathbf{1}\left[u \in\mathcal{U}^{\mathrm{rev}}_T\right]}{\sum_{u \in\mathcal{U}^{\mathrm{ask}}} w(u)}. \tag*{(7)}

This metric measures the proportion of undisclosed patient-reportable evidence successfully elicited by the doctor, while assigning greater credit to diagnostically salient information. The weighting scheme emphasizes key symptoms, relevant medical history, medications, and allergies, and prevents the score from being dominated by numerous low-value details. For cases in which Uask\mathcal{U}^{\mathrm{ask}} is empty, we set Rinfo=1R_{\mathrm{info}} = 1 by convention.

Test Coverage Rate. Let Etest⊆E\mathcal{E}^{\mathrm{test}} \subseteq\mathcal{E} denote the set of initially hidden and retrievable laboratory, microbiology, and radiology findings associated with the case. Let ETret\mathcal{E}^{\mathrm{ret}}_T denote the distinct examination items returned by the laboratory physician in response to the doctor’s requests. We measure overall and abnormal test coverage as

Rtest=∣ETret∩Etest∣∣Etest∣,Rabnormal=∣ETret∩Eabnormal∣∣Eabnormal∣.(8)R_{\mathrm{test}} = \frac{\left|\mathcal{E}^{\mathrm{ret}}_T \cap\mathcal{E}^{\mathrm{test}}\right|}{\left|\mathcal{E}^{\mathrm{test}}\right|}, \qquad R_{\mathrm{abnormal}} = \frac{\left|\mathcal{E}^{\mathrm{ret}}_T \cap\mathcal{E}^{\mathrm{abnormal}}\right|}{\left|\mathcal{E}^{\mathrm{abnormal}}\right|}. \tag*{(8)}

The metrics are computed over distinct case-supported findings rather than the raw number of orders. Therefore, repeated requests do not increase the score, and unsupported requests do not artificially inflate coverage. Here, Eabnormal\mathcal{E}^{\mathrm{abnormal}} denotes the initially hidden laboratory and microbiology findings marked as abnormal. We set the corresponding score to 1 when its denominator is empty. Thus, RabnormalR_{\mathrm{abnormal}} measures whether the doctor surfaces clinically salient abnormal test results.

Diagnosis Accuracy. To provide partial credit for clinically related predictions while maintaining a standardized and language-independent evaluation, we adopt the ICD-10 coding system and compare the predicted code y^\hat{y} with the ground-truth code yy at two levels of granularity. Let b(⋅)b(\cdot) denote the corresponding ICD-10 block and c3(⋅)c_3(\cdot) denote the three-character category. We define

Acat=1[b(y^)=b(y)],Asub=1[c3(y^)=c3(y)].(9)A_{\mathrm{cat}} = \mathbf{1}\left[b(\hat{y}) = b(y)\right], \qquad A_{\mathrm{sub}} = \mathbf{1}\left[c_3(\hat{y}) = c_3(y)\right]. \tag*{(9)}

Here, AcatA_{\mathrm{cat}} measures whether the predicted diagnosis falls within the correct ICD-10 block, while AsubA_{\mathrm{sub}} evaluates the finer-grained match at the three-character category level. All three types of metrics can be computed efficiently, making them suitable both for scalable model evaluation and as reward signals for model training.

Construction of ClinPatient

To support scalable patient simulation in ClinVerse, we model each patient from three complementary dimensions and construct a patient pool ClinPatient through a unified data-processing pipeline that converts raw EHR records into diagnosis-oriented cases. Designed to preserve the fidelity of real-world clinical data while

remaining extensible, this pipeline can be applied to additional EHR sources, allowing new patient cases to be continuously incorporated into the environment.

From EHRs to a Diagnostic Dataset

We construct ClinPatient from structured records in SICdb [30] and MIMIC-IV v3.1 [10], together with clinical narratives from MIMIC-IV-Note [11]. These sources are integrated into admission-level datasets, SICdb-Ext and MIMIC-IV-Ext, by aggregating demographics, ICD diagnoses, medications, laboratory and microbiology results, radiology records, and clinical narratives extracted from discharge summaries. We retain only admissions with sufficient evidence, remove diagnosis-revealing text, and preserve only the Findings section of radiology reports. The primary diagnosis is normalized to ICD-10 and stored as the hidden ground-truth label yy, while the processed records provide the acute clinical context and retrievable examination evidence E\mathcal{E} without exposing clinician-only information to the observable patient simulator. The statistics of ClinPatient are shown in Figure 2.

Figure 2. Dataset Statistics of ClinPatient.

Data Augmentation

To ensure the realism of the environment, we model patients along three complementary dimensions, including patient profile, clinical status, and social attributes. Because real-world admission records often lack social attributes of patients, we augment each case with a conditionally generated Synthea3 profile containing patient-reportable information, including prior conditions, long-term medications, allergies, immunization history, and social history. Generation is conditioned on age, sex, and secondary ICD diagnoses, while excluding the primary diagnosis to prevent label leakage. Inconsistent profiles and events occurring after admission or revealing the diagnosis are removed. Each case is also assigned a Big Five personality profile along the OCEAN dimensions to control conversational behaviors such as information disclosure, response conciseness, and anxiety expression [22]. Personality affects only dialogue generation, while all clinical observations and diagnostic targets remain grounded in the original EHR, enabling richer patient simulation without introducing diagnostic shortcuts.

Information Unit Extraction

To support controlled information disclosure and trajectory-level evaluation, we transform long content into a structured set of atomic information units, following prior work [29]. Specifically, we extract facts from demographics, the history of present illness, and other patient-related contents, and normalize it into a set U\mathcal{U}. Each unit is assigned a semantic category, an importance label, and a stable identifier. The subset revealed in the initial presentation is denoted by U0⊆U\mathcal{U}_0 \subseteq\mathcal{U}, while the remaining units must be acquired through interaction with the observable patient simulator. Examination-derived evidence, including physical examination, laboratory, microbiology, and radiology findings, is stored separately in E\mathcal{E} and accessed through the laboratory physician. This separation preserves role-specific information boundaries and provides structured supervision for the scoring functions in Section 2.4.

Experiments

We envision CLINVERSE as an interactive clinical consultation environment that supports both the evaluation and training of medical agents. In this section, we empirically assess its effectiveness for training and evaluation through a series of experiments.

Experimental Setup

Metrics. We evaluate doctor agents using five rule-based metrics introduced in Section 2.4. All metrics are computed on 100 cases randomly sampled from a held-out test set with 5 independent runs.

Models. We evaluate doctor agents from diverse model families and parameter scales, including general-purpose and medical-domain models such as Lingshu-32B [42], Baichuan-M2-32B [34], and the HealthGPT-Pro series [19]. We also include a full-information setting in which complete patient information and examination results are provided directly for agents, whose results are presented in Appendix E. This setting removes information-access constraints and serves as a reference for interpreting agents’ evidence-acquisition and diagnostic capabilities.

Training Methods. We categorize self-evolution methods into three levels. L1 is offline adaptation, including MemRL-offline [46], SkillRL [39], Trace2Skill [26], and MedAgent-Zero [14]. L2 represents online self-evolution, including MemRL-online [46], Dynamic Cheatsheet [33], and ACE [45]. L3 focuses on recursive self-improvement, including Self-Taught Optimizer [44], Meta Context Engineering [43], and MedRSI [37]. Detailed descriptions are provided in Appendix B.

ClinVerse as an Evaluation Platform

As presented in Table 2, we evaluate agents’ performance across the full consultation process, covering patient information elicitation, examination ordering, and evidence integration for final diagnosis. GPT-5.6-Luna achieves the strongest overall performance among the evaluated agents, yet a clear gap remains relative to the full-information reference, indicating that effective evidence acquisition is still a major bottleneck in interactive diagnosis. The five metrics further reveal distinct behavioral profiles that cannot be captured by final diagnostic accuracy alone. For example, Baichuan-M2-32B performs well in identifying relevant and abnormal examinations but does not consistently convert this advantage into more accurate diagnoses, suggesting limitations in evidence integration or clinical reasoning. In contrast, Lingshu-32B achieves comparatively strong diagnostic performance despite rarely requesting examinations, implying a greater reliance on prior knowledge or initially available information. By separately evaluating information seeking, examination decisions, and diagnostic inference under controlled information boundaries, ClinVerse exposes where each doctor agent succeeds or fails throughout the consultation process.

Doctor AgentInfo Collection Rate (%)Test Order Coverage (%)Abnormal Test Coverage (%)Diagnosis Category Accuracy (%)Diagnosis Subtype Accuracy (%)
L0: Static Agents
Lingshu-32B28.82 ± 2.690.58 ± 0.181.32 ± 0.2928.20 ± 2.1711.60 ± 1.95
Baichuan-M2-32B26.90 ± 0.7256.30 ± 1.9265.60 ± 2.1329.41 ± 3.4019.22 ± 4.02
HealthGPT-Pro-4B12.79 ± 1.8722.04 ± 1.1824.24 ± 1.6025.67 ± 3.069.00 ± 1.73
HealthGPT-Pro-8B10.62 ± 1.4133.90 ± 2.7336.56 ± 3.4324.00 ± 8.3411.60 ± 5.46
GPT-4o-mini27.49 ± 0.4147.45 ± 2.4260.98 ± 2.6432.20 ± 1.1019.20 ± 2.39
GPT-5.6-Luna35.55 ± 1.2956.63 ± 2.0668.01 ± 2.1940.37 ± 1.9232.89 ± 1.68
Claude-Sonnet-4.633.45 ± 1.0655.93 ± 1.9067.42 ± 2.1338.07 ± 1.8724.19 ± 2.29
Qwen3-8B14.33 ± 1.0740.70 ± 1.8348.44 ± 2.4323.20 ± 2.397.80 ± 2.77
Qwen3-32B13.96 ± 1.3440.84 ± 1.8548.06 ± 1.8026.04 ± 2.6912.43 ± 1.12
L1: Offline Adaptation
+ Trace2Skill14.48 ± 0.9338.26 ± 3.6449.09 ± 4.4829.20 ± 2.4916.60 ± 4.45
+ SkillRL15.46 ± 0.7137.08 ± 1.4147.10 ± 2.4927.60 ± 2.5116.40 ± 3.65
+ MemRL-offline14.36 ± 1.0043.02 ± 1.7954.16 ± 1.2631.40 ± 4.1616.80 ± 3.03
+ MedAgent-Zero12.48 ± 0.6525.00 ± 2.0632.92 ± 2.7229.25 ± 3.7713.00 ± 2.58
L2: Online Self-Evolution
+ MemRL-online15.50 ± 8.9640.34 ± 1.9349.65 ± 2.7130.87 ± 4.6219.33 ± 3.95
+ Dynamic Cheatsheet11.48 ± 0.7649.33 ± 0.6657.45 ± 1.5533.40 ± 5.8620.60 ± 4.83
+ ACE13.48 ± 1.3839.83 ± 2.7751.63 ± 3.1730.40 ± 2.3020.00 ± 3.16
L3: Recursive Self-Improvement
+ Self-Taught Optimizer12.89 ± 1.1640.32 ± 0.9750.85 ± 4.0729.40 ± 6.1117.60 ± 4.98
+ Meta Context Engineering12.84 ± 1.7442.11 ± 2.1951.43 ± 3.1228.40 ± 5.3216.40 ± 4.22
+ MedRSI13.26 ± 0.3639.88 ± 3.1449.38 ± 4.9831.40 ± 4.5117.80 ± 4.21
ReferenceGPT-5.6-Sol + Full Information54.6045.80

Table 2. Performance comparison of different doctor models and training methods. Results are grouped by four levels and reported as mean ± standard deviation (%). Methods prefixed with “+” are applied to Qwen3-32B; bold denotes the best result among these adaptation methods. GPT-5.6-Sol with full information is included as a performance reference.

ClinVerse as an Evolution Environment

Beyond evaluating static agents, we further investigate how three categories of adaptation and evolution strategies improve Qwen3-32B within ClinVerse. As shown in Table 2, these methods produce distinct capability profiles. MemRL-online mainly improves Information Collection Rate and Diagnosis Subtype Accuracy, although its information-collection performance exhibits substantial variance, whereas MemRL-offline yields more balanced gains in examination coverage and diagnosis. Dynamic Cheatsheet achieves the strongest overall adaptation results, substantially improving Test Order Coverage, Abnormal Test Coverage, and both diagnostic metrics despite collecting less patient-reported information. This pattern suggests that adaptation can improve diagnosis by shifting evidence acquisition toward more informative examinations rather than uniformly increasing the amount of collected evidence.

All three categories improve diagnostic performance, but their efficiency differs. Online Self-Evolution achieves larger diagnostic gains using 1,000 optimization trajectories than Offline Adaptation using 5,000, indicating substantially greater experience efficiency.

Recursive Self-Improvement provides more limited gains under the same setting, suggesting that optimizing the update mechanism introduces additional search and credit-assignment demands that are not necessarily compensated for by the available feedback.

Further Analysis

Info Acquisition & Diagnostic Convergence

Figure 3(a–Figure 3b) shows that information acquisition is concentrated in the early consultation turns and then exhibits clear diminishing returns, whereas diagnostic agreement generally continues to increase as the interaction proceeds. Stronger models achieve both higher information coverage and more consistent diagnoses, suggesting that effective evidence acquisition facilitates diagnostic convergence. Notably, agreement can continue improving after information collection has largely plateaued, indicating that later turns may help models confirm and integrate existing evidence rather than acquire new facts. The occasional fluctuations in agreement further reflect evidence-driven revision of intermediate hypotheses during the consultation.

Three-panel plots showing information collection rate, diagnostic agreement, and action-type distributions across dialogue turns

Figure 3. (a) Average information collection rate across dialogue turns. Information is acquired rapidly early in the consultation and gradually plateaus; (b) Average agreement between turn-level and final diagnoses, which increases as more evidence is collected; (c) Distribution of the four action types across turns, showing a general progression from patient inquiry and examination to testing and final diagnosis.

Temporal Analysis of Clinical Actions

To analyze interaction dynamics, we visualize the turn-level action distributions of three doctor agents. As shown in Figure 3(c), all models exhibit a common coarse-grained workflow: early turns focus on patient inquiry, middle turns increasingly involve physical examinations and laboratory tests, and final diagnoses dominate toward the end. However, the transitions vary across models. GPT-5.6-Luna and Qwen3-32B shift relatively smoothly between action types, whereas Baichuan-M2-32B shows more abrupt changes, suggesting less stable long-horizon planning. These results show that ClinVerse captures not only final performance but also the temporal organization of clinical decision-making.

Cross-Model Learning

Building on Table 3, we further examine how trajectory source and quality affect different target models. Trajectories generated by Qwen3-32B improve Qwen3-series models but provide little benefit to stronger GPT-series models and can even degrade performance, especially for GPT-4o-mini. Reducing the number of trajectories also weakens the gains, indicating that quantity still matters when the teacher remains fixed. However, trajectory quality is more influential: a much smaller set generated by GPT-5.6-Luna yields substantially larger improvements on Qwen3 models than a larger set from Qwen3-32B. These results suggest that trajectory-based training depends strongly on the capability gap between teacher and target models: stronger teachers provide effective supervision for weaker targets, whereas trajectories from weaker or comparable models offer limited value and may introduce harmful guidance.

Traj ModelTraj NumDoctor ModelICTOCATCDCADSA
Qwen3-32B5000GPT-4o-mini↓ 0.30↓ 6.16↓ 11.41↓ 8.20↓ 4.15
Qwen3-32B5000GPT-5.6-Luna↓ 1.86↓ 1.94↓ 0.65↓ 1.31↑ 1.49
Qwen3-32B5000Qwen3-8B↑ 0.86↓ 2.55↑ 2.65↑ 0.97↑ 3.77
Qwen3-32B5000Qwen3-32B↑ 0.40↑ 2.18↑ 6.10↑ 5.36↑ 4.37
Qwen3-32B500Qwen3-32B↓ 0.55↑ 1.60↑ 3.79↑ 0.52↑ 4.44
GPT-5.6-Luna500GPT-4o-mini↑ 4.24↑ 0.86↑ 1.32↑ 1.80↑ 11.80
GPT-5.6-Luna500GPT-5.6-Luna↓ 0.56↓ 2.94↓ 1.41↑ 4.73↑ 4.36
GPT-5.6-Luna500Qwen3-8B↑ 8.19↑ 8.71↑ 14.07↑ 0.80↑ 8.31
GPT-5.6-Luna500Qwen3-32B↑ 6.72↑ 0.87↑ 5.27↑ 5.79↑ 6.37

Table 3. Performance variation of models training with MemRL using trajectories from different models. All results are reported in percentage (%) and the five metrics are abbreviated forms of these reported in Table 2. Cross-model adaptation depends strongly on the source–target capability relationship, with stronger-model trajectories often providing greater benefits despite smaller data volumes.

Cross-Environment Generalization

To evaluate whether ClinVerse-derived experiences transfer beyond the environment, we test the adapted Qwen3-32B agents on AIHospital [4] without further adaptation. As shown in Table 4, all evolution methods consistently improve most metrics, demonstrating that acquired skills from ClinVerse provide transferable supervision across environments. Moreover, we find that most offline adaptation methods maintain relatively stable MVME scores, an LLM-as-a-judge metric for consultation quality, while improving diagnostic accuracy. In contrast, Online Self-Evolution and RSI substantially reduce MVME while achieving larger diagnostic gains. This is consistent with results in Table 2, where diagnostic accuracy improves despite limited gains in information acquisition.

Doctor AgentMVMERecallPrecisionF1
Qwen3-32B45.6718.8130.4223.25
+ MemRL-offline47.40↑1.7320.41↑1.6032.78↑2.3625.16↑1.91
+ SkillRL44.20↓1.4719.69↑0.8831.28↑0.8624.17↑0.92
+ MedAgent-Zero45.73↑0.0618.93↑0.1231.47↑1.0523.55↑0.30
+ Trace2Skill45.13↓0.5422.94↑4.1335.89↑5.4727.99↑4.74
+ Dynamic Cheatsheet39.45↓6.2226.12↑7.3136.85↑6.4330.57↑7.32
+ STOP42.32↓3.3526.46↑7.6539.44↑9.0231.67↑8.42

Table 4. Generalization performance of Qwen3-32B adapted with different training methods using interaction trajectories from ClinVerse and evaluated on AIHospital.

One possible explanation is that offline methods rely more on accumulated experience and therefore tend to collect additional information for case matching, whereas Online Self-Evolution and RSI continuously refine their strategies toward more efficient decision paths. These results suggest that different evolution paradigms improve agents along distinct capability dimensions.

Cross-Methods Improvement

We further investigate how self-evolution methods affect models of different scales. As shown in Figure 4, all methods yield positive gains, with performance progressively approaching the full-information reference. Improvements in information collection capability, however, are not always accompanied by gains in diagnostic accuracy. For Qwen3-32B, stronger information collection generally correlates with higher diagnostic accuracy, whereas GPT-5.6-Luna can achieve better diagnostic performance with less collected information. This indicates that a substantial portion of patient information may be redundant or weakly relevant to diagnosis, and that self-evolution can help models focus on more informative evidence while reducing unnecessary information acquisition. We also observe a scale-dependent pattern for the L3 RSI method. It provides larger gains for Mid-tier models, while weaker and stronger models benefit less, consistent with the trend reported by Lin et al. [18].

Figure 4. Information collection and diagnostic reasoning capabilities across 3 levels of evolution methods and backbone models. Information collection capability is calculated as the average of ICR, TOC, and ATC, while diagnostic accuracy is calculated as the average of DCA and DSA. Greater information acquisition does not necessarily yield better diagnostic accuracy, highlighting the importance of how medical agents select and utilize clinical evidence.

Ablation Study

Robustness of Patient Info Boundaries

Examination results of the dual-layer patient simulator and self-check mechanism described in § 2.2 are presented in Table 5, with details in the Appendix § 17. The dual-layer design restricts the observable patient simulator from accessing privileged clinical evidence, whereas self-checking filters responses that may disclose information beyond the permitted scope. Their combination achieves the lowest information leakage rate and the highest refusal rate for boundary-violating queries. The results indicate that access control and response verification address distinct sources of leakage and are both necessary for maintaining partial observability.

SettingArbitrary Disclosure RateTest-related Disclosure RateRefusal Rate
Single&Neutral47.2%87.0%10.0%
Dual-layer&Neutral36.8%66.0%19.7%
Single&Self-check25.5%67.0%62.8%
Dual-layer&Self-check5.7%13.5%84.7%

Table 5. Comparison of information disclosure and refusal behaviors under different patient simulation settings.

Fidelity of Physiological-State Prediction

Table 6 presents the evaluation results of the physiological-state predictor ClinLab-2B for examinations whose results are absent from the source EHR. Metrics are detailed in Appendix I. ClinLab-2B consistently outperforms the general-purpose baselines in both numerical accuracy and abnormality recall, suggesting that task-specific training is more effective than simply increasing model scale. By generating plausible examination results while retaining clinically relevant abnormalities, the predictor supports flexible test ordering without allowing missingness in the original EHR to become an unintended diagnostic cue.

ModelMAPEMedian MAPEMAENormRefErrAnomaly Recall
Qwen3.5-2B57.3627.1777.21777.2330.68
Qwen3.5-27B35.3311.5422.452.0159.69
DeepSeek-V4-Flash35.4712.5061.631.9263.95
GPT-4o-mini44.5218.4357.284.7369.04
ClinLab-2B19.463.7014.620.3676.92

Table 6. Performance comparison across different models as physiological-state predictor. Our trained model ClinLab accurately predicts examination outcomes while achieving high recall for abnormal findings.

Conclusion

We propose ClinVerse, a scalable real-data-grounded multi-agent environment for medical diagnostic agents. Built on ClinPatient, it supports realistic interactions, flexible examinations, and rule-based evaluation of evidence acquisition and diagnosis. Experiments validate the effectiveness of ClinVerse as an evolution environment and its reliability as an evaluation platform.

Limitations and Ethical Considerations

This study uses only de-identified clinical data from SICdb [30], MIMIC-IV v3.1 [10], and MIMIC-IV-Note [11], accessed through PhysioNet under the required credentialing and Data Use Agreement procedures. We used these data solely for research and made no attempt to identify patients or institutions. Raw EHRs were transformed into structured admission-level cases, and doctor agents accessed only de-identified, role-specific information. Any released artifacts will comply with original licenses and will not include raw records or re-identifiable information.

Generative AI Usage

Generative AI tools were used only in an assistive capacity during manuscript preparation and implementation. Specifically, LLM were used for language polishing, grammar and typographical-error checking, and assistance with drafting, debugging, and documenting research code. All AI-assisted text and code were reviewed, revised, and verified by the authors. Singh, Maryam Fazel, Daniel Hsu, Simon Lacoste-Julien, Felix Berkenkamp, Tegan Maharaj, Kiri Wagstaff, and Jerry Zhu (Eds.). PMLR / OpenReview.net. https://proceedings.mlr.press/v267/zuo25a.html

Related Work

Diagnostic Decision-Making Evaluation

Existing evaluations of medical agents can be broadly divided into static and interactive paradigms. Static benchmarks typically formulate diagnosis as a question-answering task using cases derived from medical licensing examinations [9, 20], textbooks [7], or EHRs [40], with answer accuracy serving as the primary metric. These benchmarks are effective for measuring medical knowledge and diagnostic reasoning under fixed evidence, but they provide limited insight into how agents acquire information or make sequential clinical decisions. More recent benchmarks [14, 17, 41] therefore introduce multi-turn consultation settings in which agents interact with simulated patients and other clinical roles, enabling evaluation of both diagnostic outcomes and the decision processes that produce them.

Evaluating such interaction trajectories, however, remains challenging. MIRA [5] relies on case-specific rules and expert review to assess clinically meaningful behaviors, achieving high evaluation fidelity at substantial annotation cost. Other systems [4, 41] improve scalability by using LLM-based judges to score interactions across multiple dimensions, but their results may vary with the evaluator model, prompt design, and scoring criteria. In contrast, ClinVerse adopts a rule-based evaluation framework that derives reproducible scores directly from hidden case states and recorded interaction events, covering information collection, examination ordering, and diagnostic accuracy without requiring per-case expert annotation or model-based judging.

Interactive Clinical Environments

Recent studies [1, 3, 15, 21, 24, 25, 38] have explored interactive clinical environments for evaluating medical agents beyond static question answering. Agent Hospital [14] constructs a flexible hospital sandbox by synthesizing patients and clinical roles from external knowledge sources, but its fully synthetic design may compromise clinical fidelity. In contrast, AIHospital [4] and SDBench [27] ground patient simulation in cases collected from online resources or medical literature, providing more realistic clinical scenarios while remaining limited by the scale, completeness, and examination coverage of the source cases. AgentClinic [31] introduces a multimodal clinical environment with four interacting roles, enabling patient consultations, examination requests, and diagnostic decision-making across diverse clinical scenarios. However, its text-parsed interaction protocol and reliance on predefined case findings, with potentially normal responses for unavailable examinations, limit structured clinical interactions and examination flexibility. MIRA [5] further improves evaluation fidelity through expert-curated cases and fine-grained, case-specific assessment criteria, but the associated annotation burden limits scalability.

Overall, existing environments face a trade-off among interaction flexibility, clinical grounding, and scalable evaluation. ClinVerse differs from these systems by constructing interactive cases from

real-world EHRs at scale, providing structured tool-call-based clinical actions, and supporting examination requests beyond the observations explicitly recorded in each source case through ClinLab-based prediction. These designs enable broader and more consistent diagnostic interactions while facilitating subsequent agent self-improvement.

Training Methods Used in Experiments

We adapt doctor agents within ClinVerse using nine methods. Most of the methods keep the backbone LLM frozen and improve agent behavior through external skills or memories derived from interaction trajectories.

Trace2Skill [26] collects successful and failed clinical trajectories from a frozen doctor agent and analyzes them separately to extract reusable behaviors and causally grounded corrections. These trajectory-level lessons are converted into local skill patches, then hierarchically deduplicated and consolidated into a coherent clinical skill directory. The resulting skill is directly supplied at inference time without model updates or per-case memory retrieval.

SkillRL [39] distills successful trajectories into effective decision patterns and failed trajectories into counterfactual lessons describing errors and preferable actions. It organizes this knowledge into a hierarchical SkillBank containing general clinical principles and task-specific guidance for stages such as history collection, investigation, and diagnosis. Our parameter-free adaptation retrieves relevant skills as contextual guidance while omitting the SFT and reinforcement-learning updates of the original framework. We additionally apply the full SkillRL training pipeline to Qwen3-8B, with details and results reported in Appendix C.

MemRL [46] represents each clinical episode as an intent – experience – utility triplet in an external memory bank initialized from offline rollouts. At inference time, it first recalls semantically similar experiences and then re-ranks them using learned utility, reducing the influence of relevant-looking but unhelpful memories. Environmental feedback subsequently updates the utilities of retrieved experiences and may add new memories, enabling online adaptation without changing model weights.

MedAgent-Zero [14] maintains separate Medical Record Libraries and Experience Bases for the examination, diagnosis, and treatment stages. Successful cases are stored as validated examples, whereas failed cases are reflected upon and summarized into transferable corrective principles. For each new clinical decision, similar records and failure-derived guidance are jointly retrieved to support reasoning, allowing both repositories—and thus the doctor agent—to improve continually without parameter updates.

Dynamic Cheatsheet [33] gives a black-box LLM a persistent, self-curated memory that accumulates concise strategies, code snippets, and corrections across test-time interactions. After each case, a curator examines the current memory, query, and attempted solution to add useful generalizations and revise or prune unreliable entries; the revised cheatsheet then conditions subsequent cases. This enables online adaptation without ground-truth labels, human feedback, or changes to model parameters, while limiting the context growth caused by appending the full interaction history.

ACE [45] treats the agent context as an evolving playbook and separates adaptation into generator, reflector, and curator roles. The generator acts using the current playbook, the reflector extracts generalizable lessons from the resulting trajectory and natural execution feedback, and the curator integrates compact delta updates before semantically deduplicating overlapping entries. We use its online setting so that later consultations receive the refined playbook; these localized updates preserve useful clinical detail and mitigate the brevity bias and context collapse associated with repeatedly rewriting the entire context.

Self-Taught Optimizer [44] begins with an LLM-powered improver program that proposes multiple revisions to an input program, evaluates them with a task-specific utility function, and returns the best candidate. The seed improver is then applied to its own source code, allowing utility feedback to produce a stronger update procedure whose emergent search strategies can include beam search, genetic algorithms, and simulated annealing. The improved improver is subsequently used to optimize agent solutions, yielding a form of recursive self-improvement in which the scaffolding evolves while the backbone LLM remains frozen.

Meta Context Engineering [43] formulates adaptation as bilevel optimization that co-evolves context-engineering skills and the context artifacts produced by those skills. At the meta level, agentic crossover synthesizes a revised skill from the task specification and the history of prior skills, executions, and evaluations; at the base level, an agent executes that skill to construct context as flexible files and code, including both static resources and query-dependent operations such as retrieval and filtering. Iterative evaluation selects improved skill–context pairs, allowing both what the doctor agent learns and how it organizes and updates that knowledge to evolve without modifying the backbone model.

MedRSI [37] enables recursive self-improvement by transforming diagnostic failures into new clinical capabilities while keeping the backbone LLM frozen. It prioritizes failures according to their potential clinical consequences and identifies missing capabilities through trajectory analysis and reflection. These capability gaps are addressed through automated code generation, tool composition, or task-specific model training, producing new clinical tools for measurement, prediction, and evidence integration. To ensure reliable adaptation, MedRSI adopts a fast-discovery, slow-registration mechanism, where candidate tools are evaluated across subsequent patient cohorts and incorporated into a persistent tool registry only after demonstrating sustained diagnostic improvements without increasing clinical risk. This iterative process allows the doctor agent to autonomously expand its capabilities without updating the backbone LLM.

Effectiveness of ClinVerse for GRPO Training

To further examine whether ClinVerse can support parameter-updating reinforcement learning, we adapt the GRPO stage of SkillRL to the ClinVerse interaction and reward interface and optimize an SFT-initialized Qwen3-8B doctor agent for 30 steps. During training, the agent conducts multi-turn clinical encounters and receives the rule-based ClinVerse score as its reward, allowing the policy to learn directly from its information-gathering and diagnostic decisions.

As shown in Table 7, SkillRL-GRPO consistently improves examination decisions and diagnostic performance, indicating that the reward signals provided by ClinVerse can effectively guide policy optimization. Notably, the trained model achieves subtype-level diagnostic performance close to the Full Info reference, despite operating under the original partially observable setting. Meanwhile, the decrease in information collection suggests that the optimized policy relies more on examination-based evidence and diagnostic inference rather than extensive patient inquiry. These results demonstrate that ClinVerse not only provides effective training signals for GRPO but also enables fine-grained analysis of how optimization reshapes clinical decision-making behaviors.

ModelInfo. CollectionTest OrderAbnormal TestDiag. CategoryDiag. Subtype
Qwen3-8B14.3 ± 1.140.7 ± 1.848.4 ± 2.423.2 ± 2.47.8 ± 2.8
+ SkillRL-GRPO6.7 ± 0.245.2 ± 1.557.6 ± 1.728.6 ± 2.116.6 ± 1.1
+ Full Info–––32.617.5

Table 7. Performance of Qwen3-8B before and after SkillRL-GRPO training. Interactive results are reported as mean ± standard deviation over five runs of 100 cases. The Full Info reference is evaluated on 200 cases and does not involve information collection or test ordering.

Impact of Different Training Methods on Model Behavior

Impact on Information Collection

Figure 5 shows that all training methods enable agents to acquire most patient-reportable information during the early stages of consultation, after which the collection rate gradually saturates. However, different methods exhibit distinct acquisition patterns. MemRL-online accelerates early evidence acquisition and reaches a stable level faster, while SkillRL and MemRL-online achieve the strongest overall information collection capability. In contrast, only limited improvements are provided by Trace2Skill and MemRL-offline over the original model, and MedAgent-Zero even reduces information coverage. These results indicate that trajectory-based adaptation influences not only the amount of information collected but also the timing of evidence acquisition. In particular, utility-aware memory retrieval encourages more efficient early-stage questioning, whereas improvements in diagnostic performance do not necessarily imply more comprehensive information gathering.

Figure 5. Comparison of Information Collection Rate of Different Training Methods with Qwen3-32B

Impact on Action Taken

The turn-level action distributions in Figure 6 reveal that different training methods induce distinct interaction strategies. Trace2Skill and SkillRL maintain a relatively smooth clinical workflow, gradually transitioning from patient inquiry to examination requests and final diagnosis. MemRL-online produces a more structured and stage-wise interaction pattern, with each action type concentrated within specific phases of the consultation, enabling rapid progression toward diagnosis. In contrast, MedAgent-Zero maintains prolonged patient inquiry throughout the interaction and performs fewer diagnostic actions, suggesting that its retrieved experiences encourage extended but less targeted information gathering. These results show that trajectory-based adaptation not only improves final performance but also reshapes the temporal organization of clinical decision-making, with different methods achieving distinct trade-offs between exploration and efficiency.

Figure 6. Comparison of Action Taken Ratio of Different Training Methods with Qwen3-32B

Performances of Different Models Under Full Information Setting

Table 8 reports diagnostic performance when complete patient information and examination results are directly provided, isolating evidence integration and diagnostic reasoning from information acquisition. Compared with the interactive setting, the ranking of models changes substantially, suggesting that strong diagnostic reasoning with complete evidence does not necessarily translate into effective evidence acquisition and interaction planning. GPT-series models generally benefit more from full information, while Qwen3, HealthGPT-Pro, and Lingshu models show relatively smaller gaps between constrained and unconstrained settings. Model scaling also exhibits different effects across model families, indicating that larger parameter sizes do not uniformly improve fine-grained diagnosis. Moreover, all models show a consistent performance drop

Doctor ModelDiagnosis Category Accuracy (%)Diagnosis Subtype Accuracy (%)
GPT-5.6-Luna48.0038.00
GPT-5.6-Sol54.0045.00
Qwen3-8B32.6017.50
Qwen3-32B35.0021.60
HealthGPT-Pro-4B32.509.50
HealthGPT-Pro-8B32.4017.80
Lingshu-32B35.2022.60
GPT-4o-mini38.0027.50

Table 8. Diagnostic performance of different models under the full-information setting.

from category-level to subtype-level diagnosis, demonstrating that fine-grained clinical reasoning remains challenging even when information access is no longer a limitation.

Details of Cross-Methods Improvement

Table 9 provides the complete numerical results underlying Figure 4 and the cross-method analysis in § 5.5. While the figure highlights relative changes in information collection and diagnostic accuracy, the table reports the absolute values of all five evaluation metrics for every model–method combination.

Doctor ModelMethodICTOCATCDCADSA
Qwen3-32BStatic13.9640.8448.0626.0412.43
Qwen3-32BMemRL14.3643.0254.1631.4016.80
Qwen3-32BDynamic Cheatsheet11.4849.3357.4533.4020.60
Qwen3-32BSelf-Taught Optimizer12.8940.3250.8529.4017.60
GPT-4o-miniStatic27.4947.4560.9832.2019.20
GPT-4o-miniMemRL28.0650.2164.3538.0026.00
GPT-4o-miniDynamic Cheatsheet29.3650.4464.1034.0026.00
GPT-4o-miniSelf-Taught Optimizer27.1249.6568.2237.0026.00
GPT-5.6-LunaStatic35.5556.6368.0140.0032.00
GPT-5.6-LunaMemRL34.9953.6966.6045.1037.25
GPT-5.6-LunaDynamic Cheatsheet29.8055.6771.3141.0035.00
GPT-5.6-LunaSelf-Taught Optimizer32.0754.7672.6241.0032.00

Table 9. Performance of three doctor models under static inference and three adaptation methods. IC, TOC, ATC, DCA, and DSA denote Info Collection, Test Order Coverage, Abnormal Test Coverage, Diagnosis Category Accuracy, and Diagnosis Subtype Accuracy, respectively.

Transfer from AIHospital to ClinVerse

In § 5.4, we demonstrate that the capabilities acquired through training in ClinVerse can be transferred to AIHospital. We further examine the reverse transfer direction by adapting Qwen3-32B with MemRL, SkillRL, and MedAgent-Zero using trajectories collected from 406 AIHospital [4] cases, then evaluating the agents in ClinVerse. Table 10 reports changes relative to the unadapted agents. Most method show no improvement: MemRL and MedAgent-Zero decline across the reported metrics, while SkillRL improves information collection and examination coverage but yields only small diagnostic gains. These results suggest that adaptation in AIHospital does not consistently translate into improved performance in ClinVerse, and further demonstrate the effectiveness and validity of ClinVerse’s design.

Table 10: Changes relative to Baseline for Qwen3-32B agents adapted in AIHospital and evaluated in ClinVerse. Metric abbreviations follow Table 2. Upward and downward arrows indicate increases and decreases, respectively.

MethodICTOCATCDCADSA
MemRL↓ 2.24↓ 4.80↓ 4.71↓ 1.60↓ 1.00
SkillRL↑ 3.23↑ 2.12↑ 3.49↑ 0.40↑ 1.20
MedAgent-Zero↓ 5.71↓ 15.89↓ 18.29↓ 7.00↓ 4.60

Table 10.

Diagnosis with an Oracle ICD-10 Block

We further investigate fine-grained diagnostic reasoning under an oracle block-conditioned setting, where models are provided with complete patient information, examination results, and the ground-truth ICD-10 block as a hard constraint. The task is to predict the corresponding three-character diagnostic category within the given block, and predictions violating the constraint are considered incorrect. As shown in Table 11, oracle block information substantially improves subtype-level diagnosis across models, indicating that coarse disease localization accounts for a major portion of diagnostic errors. Nevertheless, considerable performance gaps remain between models, demonstrating that fine-grained category selection requires additional clinical reasoning and constraint-following ability beyond access to the correct diagnostic region. The consistent ranking with the unconditioned full-information setting further suggests that stronger models maintain advantages even when the search space is narrowed by oracle information.

Table 11: Diagnosis subtype accuracy under full information with the ground-truth ICD-10 block provided as a hard constraint. The model must predict the correct three-character category within the given block.

Doctor ModelDiagnosis Subtype Accuracy (%)
GPT-4o-mini51↑ 31.80
GPT-5.6-Luna73↑ 40.11
Qwen3-32B45↑ 32.57
Lingshu-32B49↑ 37.40
Baichuan-M2-32B14↓ 5.22

Table 11.

Training Detail of Physiological-state Predictor

Training. We construct the training set from laboratory measurements in MIMIC-IV-Ext and split samples by hospital admission to prevent admission-level context leakage between training and evaluation. For each eligible measurement, we apply a leave-one-out masking strategy, where the target laboratory result is removed while all remaining clinical information, including demographics, history, physical examination, diagnoses, medications, microbiology results, and hospital course, is preserved. The model receives the masked test name, category, unit, and reference range, and is trained to predict only the missing result value. This formulation simulates real scenarios where requested examinations are absent from the source EHR while preventing direct target copying. To emphasize clinically meaningful cases, all abnormal measurements are retained and normal measurements are subsampled. We fine-tune a Qwen3.5-2B backbone with LoRA under the causal language modeling objective. During training, system, user, and padding tokens are assigned the ignore index −100-100, ensuring that only the generated laboratory result contributes to the loss. For samples exceeding the context limit, input pies are truncated while preserving the target response.

Evaluation Metrics. Given the ground-truth value yiy_i and prediction y^i\hat{y}_i, we evaluate numerical accuracy using four metrics. MAPE computes the mean absolute percentage error, 1N∑i∣y^i−yi∣/∣yi∣\frac{1}{N}\sum_i |\hat{y}_i-y_i|/|y_i|, while Median MAPE reports the median sample-level percentage error to reduce sensitivity to outliers. MAE measures the absolute deviation between predictions and ground truth, and Norm-RefErr normalizes the absolute error by the width of the corresponding reference interval, ∣y^i−yi∣/(ui−li)|\hat{y}_i-y_i|/(u_i-l_i), enabling comparison across heterogeneous laboratory tests. We further measure Anomaly Recall, which evaluates whether predicted values correctly identify ground-truth abnormal results based on test-specific reference ranges. Lower values indicate better performance for MAPE, Median MAPE, MAE, and NormRefErr, while higher values indicate better performance for Anomaly Recall. Following the evaluation protocol, zero-valued targets are excluded from numerical error metrics, unstable tests with per-test MAPE above 200% are removed, and individual percentage errors are clipped at 200% when computing aggregate MAPE.

Information-Boundary Stress Test and Interaction Impact

Stress-test protocol. The ablation in Table 5 evaluates information-boundary robustness under adversarial queries rather than routine consultation. For each of 100 cases, we issue six boundary-seeking queries covering diagnosis requests, examination-result requests, and attempts to obtain complete medical records. We compare different combinations of information architecture (single simulator vs. dual layer) and response policy (neutral vs. self-check) under identical cases and prompts, resulting in 600 responses per setting.

Metric definitions. Arbitrary Disclosure Rate measures the proportion of responses containing any suspected privileged disclosure, including explicit diagnosis leakage or unsupported objective findings. Test-related Disclosure Rate focuses on leakage of examination results among test-related queries. Refusal Rate measures responses containing explicit refusal or unavailability statements. These metrics are computed independently, and disclosure labels are obtained through deterministic lexical screening rather than manual clinical judgment.

Impact on ordinary interaction. The high refusal rate observed under the adversarial protocol reflects the intentionally boundary-violating nature of the queries and does not represent routine consultation behavior. To examine practical impact, we further analyze 98 completed Qwen3-32B trajectories from the main evaluation, containing 393 patient-query turns. Only a small fraction of responses match the refusal lexicon, and most correspond to normal uncertainty about symptoms, history, or medications rather than information-boundary violations. Only two cases involve explicit requests for unavailable diagnostic or examination results, resulting in a negligible boundary-triggered refusal frequency. These cases appropriately redirect the doctor to obtain objective evidence through the laboratory physician.

These results indicate that the proposed boundary mechanism can resist explicit information leakage attempts while minimally affecting normal patient interaction.

ClinLab Validity Analysis

To investigate whether ClinLab-generated examination results affect diagnostic reliability, we conduct two additional experiments examining the robustness of diagnostic performance to predicted values and missing examination results.

Robustness to ClinLab predictions. We randomly replace 10% and 20% of available EHR examination values with ClinLab predictions and evaluate the resulting diagnostic performance. As shown in Table 12, both category and subtype accuracy remain stable across different replacement ratios, suggesting that the introduction of ClinLab-generated values does not substantially compromise overall diagnostic performance.

Replacement RatioDCADSA
0%26.04 ± 2.6912.43 ± 1.12
10%26.41 ± 2.3013.22 ± 1.48
20%28.27 ± 1.9214.61 ± 1.52

Table 12. Diagnostic performance under different proportions of ClinLab-predicted examination values.

Impact of missing examination results. We further compare ClinLab-based prediction with explicitly returning null for unavailable examinations. As shown in Table 13, removing ClinLab leads to higher diagnostic accuracy, particularly at the category level. One possible explanation is that explicit missingness provides additional cues about the underlying diagnosis, which agents may exploit rather than relying solely on acquired clinical evidence.

SettingDCADSA
w/ ClinLab35.5831.73
w/o ClinLab39.1731.91

Table 13. Diagnostic performance with and without ClinLab.

These results suggest that diagnostic performance is relatively robust to ClinLab predictions, while the treatment of missing examinations can influence evaluation outcomes. Nevertheless, stable aggregate accuracy does not rule out individual cases where incorrect predicted values affect diagnostic reasoning. Further provenance-aware analysis is needed to distinguish decisions supported by observed EHR evidence from those influenced by ClinLab-generated results.

Subgroup Robustness Analysis

We evaluate Qwen3-32B across patient-disjoint cohorts sampled from MIMIC-IV-Ext, covering different genders, age groups, and disease categories. Table 14 presents the subgroup results across five evaluation metrics. As shown in Table 14, diagnostic accuracy is relatively stable across age groups despite variations in information collection and examination coverage. Male and female cohorts also exhibit similar evidence-acquisition behavior but differ in diagnostic accuracy, indicating a disparity that warrants further investigation; unequal disease composition prevents a causal interpretation. More pronounced differences arise across diseases: heart failure is diagnosed substantially more accurately than sepsis, while pneumonia achieves the highest examination coverage without comparable diagnostic performance. These results suggest that evidence acquisition alone does not determine diagnostic success; identifying and integrating clinically discriminative findings remain critical.

SubgroupICRTOCATCCategory Acc.Subtype Acc.
Male13.235.646.942.024.0
Female13.234.245.824.015.0
Age 18–4415.239.248.533.018.0
Age 45–6412.037.944.931.018.0
Age 65–7412.938.645.730.018.0
Age 75+11.141.051.233.016.0
Sepsis (A41)10.836.745.72.52.5
Heart failure (I50)14.440.345.271.071.0
Pneumonia (J18)12.951.562.928.027.0
Procedure complications (T81)12.231.841.023.015.0
Urinary-system diseases13.139.944.923.018.0

Table 14. Subgroup performance of the fixed Qwen3-32B doctor agent. All values are percentages over completed evaluations. ICR, TOC, and ATC denote information collection rate, test-order coverage, and abnormal-test coverage, respectively. Category and subtype accuracy measure agreement at the ICD-10 block and three-character category levels. The urinary-system cohort includes N10–N16 and N30–N39.

We further examine three illustrative trajectories (Table 15). In the heart-failure case, targeted cardiac examinations reveal elevated NT-proBNP and pulmonary edema, supporting a correct diagnosis at the evaluated ICD-10 levels despite limited overall information collection. In contrast, the sepsis case focuses on biliary pathology, and the pneumonia case favors heart failure based on pulmonary edema and cardiomegaly; both disagree with the recorded primary diagnosis despite high examination coverage. Together, these examples illustrate that diagnostic performance depends not only on obtaining evidence but also on interpreting competing findings and aligning the final diagnosis with the underlying condition. However, the available observations do not establish that the reference diagnosis was unequivocally inferable in either mismatched case; examination results may also include laboratory-provider predictions.

CohortCase IDReferencePredictionICRTOCATCCategory Acc.Subtype Acc.
Heart failure10199560_22434262I50.23I50.16.443.1100.0100.0100.0
Sepsis10651989_24805766A41.9K80.319.567.4100.00.00.0
Pneumonia10333909_27924784J18.9I5032.791.7100.00.00.0

Table 15. Illustrative diagnostic trajectories across disease cohorts. All scores are percentages. Case identifiers refer to the frozen evaluation records. Reference and predicted diagnoses retain their original ICD-10 codes, while diagnostic accuracy is evaluated at the coarser levels defined in Section 2.4.

Case Study

Case Overview. We qualitatively analyze two Qwen3-32B trajectories from the same MIMIC-IV-Ext evaluation split used in the main experiments. To provide representative examples, we select cases with above-median information collection and test coverage among the completed trajectories. We include one successful case with a coherent evidence-acquisition process and one failed case with substantial evidence collection but an incorrect final diagnosis, allowing us to examine different sources of diagnostic errors. Table 16 summarizes the selected cases. These examples are intended to illustrate the behaviors captured by the decomposed metrics rather than characterize their overall frequency.

Success: iterative evidence acquisition supports the correct ICD-10 category
PresentationA 57-year-old patient with hypertension, hyperlipidemia, diabetes, peripheral neuropathy, and current smoking presented with worsening positional dizziness, episodic severe hypertension, and bilateral hand paresthesia. Examination showed a left cervical bruit and subtle right-sided neurological abnormalities.
DiagnosisAgent output: cerebral infarction, coded as I63. Reference: occlusion/stenosis of a carotid artery with cerebral infarction (I63.239).
Metric scoresRinfo=0.159R_{\mathrm{info}} = 0.159 Rtest=0.409R_{\mathrm{test}} = 0.409 Rabnormal=0.111R_{\mathrm{abnormal}} = 0.111 Acat=1A_{\mathrm{cat}} = 1 Asub=1A_{\mathrm{sub}} = 1
Failure: high evidence coverage still leads to a symptom-level answer
PresentationA 73-year-old patient with diabetes, diastolic heart failure, hypertension, possible coronary artery disease, and chronic kidney disease presented with progressive dyspnea and dry cough. Symptoms worsened with exertion and recumbency, but there was no chest pain.
DiagnosisAgent output: dyspnea and dry cough, mapped to R06.0. Reference: subendocardial infarction/NSTEMI (I21.4).
Metric scoresRinfo=0.419R_{\mathrm{info}} = 0.419 Rtest=0.729R_{\mathrm{test}} = 0.729 Rabnormal=0.667R_{\mathrm{abnormal}} = 0.667 Acat=0A_{\mathrm{cat}} = 0 Asub=0A_{\mathrm{sub}} = 0

Table 16. Representative successful and failed Qwen3-32B trajectories.

We further presents the successful trajectory in a turn-by-turn format as shown in Table 17. Patient responses are translated into English, while lengthy laboratory and radiology reports are abbreviated with diagnostically relevant findings preserved. During interaction, the observable patient simulator provides patient-reportable information, whereas the latent patient simulator supplies privileged clinical states only through the laboratory physician. The doctor can therefore flexibly alternate between patient inquiry and objective evidence acquisition based on the evolving diagnostic hypothesis.

TurnInformation routeActual interaction
Init.EC → DEC: Revealed a 57-year-old patient with hypertension, hyperlipidemia, diabetes with peripheral neuropathy, and current smoking. Admission examination included a left cervical bruit, subtle right-sided weakness and clumsiness, and an unsteady gait.
1D ↔ OPSD: “Describe your main symptoms, their onset and duration, and what worsens or relieves them.” OPS: “I have recurrent dizziness, especially when standing. My blood pressure sometimes reaches 190–210, and both hands feel numb and tingly.”
2D ↔ OPSD: “When did the dizziness begin, and was the onset sudden or gradual?” OPS: “It began about one month ago, initially only occasionally, but has worsened over the past few days and is most noticeable whenever I stand.”
3D ↔ OPSD: “Does the dizziness occur with nausea, blurred vision, palpitations, or sweating?” OPS: “I sometimes have transient blurred vision, but no nausea, palpitations, or sweating.”
4D → LP<br>LP ↔ LPS<br>LP → DD: Requested vital-sign and neurological examinations to evaluate dizziness, blood-pressure fluctuation, and paresthesia. LP (from LPS): BP 115/74, left cervical bruit, distal sensory loss, slow/clumsy right-hand movements, and an unsteady gait.
5D → LP<br>LP ↔ LPS/ClinLab<br>LP → DD: Ordered a metabolic panel, cardiac markers, and inflammatory markers. LP (from LPS/ClinLab): Glucose 214 mg/dL; creatinine 0.8 mg/dL; sodium 138 mEq/L; potassium 4.3 mEq/L; troponin T < 0.01 ng/mL; CK-MB 3.0 ng/mL. Additional requested items absent from the source record were generated by ClinLab.
6D → LP<br>LP ↔ LPS<br>LP → DD: Ordered non-contrast head CT to assess a possible cerebrovascular event. LP (from LPS): No hemorrhage, edema, mass, or large-territory infarction; a small left frontal hypodensity could represent an old lacunar infarct or prominent perivascular space; intracranial vascular calcifications were present.
7D ↔ OPSD: “Have you noticed headache, speech difficulty, limb weakness, or other visual abnormalities?” OPS: “I have a headache around my left eye, but no speech difficulty or limb weakness.”
8D → LP<br>LP ↔ LPS<br>LP → DD: Ordered brain MRI to clarify the indeterminate left frontal CT finding. LP (from LPS): Multiple diffusion-restricted foci consistent with infarction, including new left frontal and bilateral paramedian parietal lesions; a small amount of blood products was present in the left frontal focus.
FinalD → EC<br>EC → DJD: Submitted cerebral infarction (I63). DJ: The code matched the ICD-10 block and three-character category of the reference carotid occlusion/stenosis with cerebral infarction (I63.239), although the agent did not specify the carotid mechanism.

Table 17. Turn-by-turn successful diagnostic trajectory for a de-identified case (10006457_27894366). D denotes Doctor, OPS Observable Patient Simulator, LPS Latent Patient Simulator, LP Laboratory Physician, EC Environment Controller, and DJ Diagnostic Judge. Patient responses are filtered by the self-check mechanism before being returned to D, while clinical evidence is provided through LP with support from LPS and ClinLab when needed. Blue, orange, and green rows represent patient interaction, evidence acquisition, and diagnosis submission, respectively.

Analysis. The successful case illustrates an evidence-driven but non-linear diagnostic process. After initial patient inquiry, the agent progressively requested neurological examination, laboratory tests, and imaging. When CT failed to provide definitive evidence, the agent conducted further targeted inquiry and ordered MRI, whose findings supported the final diagnosis at both category and subtype levels. However, it failed to identify the underlying vascular mechanism captured by the reference diagnosis, leading to incomplete abnormal-test coverage. This case demonstrates that ClinVerse can distinguish between achieving the required diagnostic granularity and fully characterizing the underlying pathology.

The failed case highlights a different limitation, where evidence acquisition was adequate but clinical interpretation was insufficient. Although the patient presented with risk factors and abnormal findings suggestive of ischemia, the agent failed to integrate these signals and terminated with only a symptom-level diagnosis. This example shows that collecting relevant evidence alone does not guarantee accurate diagnosis. By separately evaluating information acquisition and diagnostic reasoning, ClinVerse can identify whether failures arise from insufficient evidence gathering or ineffective evidence integration.

References

  1. [1]Mohammed Khaleed Almansoori, Komal Kumar, and Hisham Chlolkkal. 2025. MedAgentSim: Self-evolving Multi-agent Simulations for Realistic Clinical Interactions. In Medical Image Computing and Computer Assisted Intervention - MICCAI 2025 - 28th International Conference, Daejeon, South Korea, September 23-27, 2025, Proceedings, Part IX (Lecture Notes in Computer Science, Vol. 15968), James C. Gee, Daniel C. Alexander, Jaesung Hong, Juan Eugenio Iglesias, Carole H. Sudre, Archana Venkataraman, Polina Golland, Jong Hyo Kim, and Jinah Park (Eds.). Springer, 362–372. doi:10.1007/978-3-032-05114-1_35
  2. [2]Erin P Balogh, Bryan T Miller, and John R Ball. 2015. Improving diagnosis in health care. (2015).DOI
  3. [3]Sirry Chen, Jieyi Wang, Wei Chen, and Zhongyu Wei. 2026. SpeechMedAssist: Efficiently and Effectively Adapting Speech Language Models for Medical Consultation. In Proceedings of the 64th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), ACL 2026, San Diego, California, United States, July 2-7, 2026, Maria Liakata, Viviane P. Moreira, Jiajun Zhang, and David Jurgens (Eds.). Association for Computational Linguistics, 30914–30935. https://aclanthology.org/2026.acl-long.1428/
  4. [4]Zhihao Fan, Lai Wei, Jialong Tang, Wei Chen, Siyuan Wang, Zhongyu Wei, and Fei Huang. 2025. AI Hospital: Benchmarking Large Language Models in a Multi-agent Medical Interaction Simulator. In Proceedings of the 31st International Conference on Computational Linguistics, COLING 2025, Abu Dhabi, UAE, January 19-24, 2025, Owen Rambow, Leo Wanner, Marianna Apidianaki, Hend Al-Khalifa, Barbara Di Eugenio, and Steven Schockaert (Eds.). Association for Computational Linguistics, 10183–10213. https://aclanthology.org/2025.coling-main.680/arxiv.org/abs/2402.09742
  5. [5]Dyke Ferber, Lars Hilgers, Christiane Höper, Benedict Kinny-Köster, Jan-Niklas Eckardt, Katharina Egger-Heidrich, Marius Bill, Martin KK Schneider, Jan Clusmann, Lejla Kadric, et al. 2026. Towards autonomous medical artificial intelligence agents. Nature (2026), 1–10.DOI
  6. [6]Kenneth A Fleming, Susan Horton, Michael L Wilson, Rifat Atun, Kristen DeStigter, John Flanigan, Shahin Sayed, Pierrick Adam, Bertha Aguilar, Savvas Andronikou, et al. 2021. The Lancet Commission on diagnostics: transforming access to diagnostics. The Lancet 398, 10315 (2021), 1997–2050.DOI
  7. [7]Aya E Fouda, Abdelrahman A Hassan, Radwa J Hanafy, and Mohammed E Fouda. 2026. PsychiatryBench: a multi-task benchmark for LLMs in psychiatry. npj Digital Medicine (2026).DOI
  8. [8]Paul Hager, Friederike Jungmann, Robbie Holland, Kunal Bhagat, Inga Hubrecht, Manuel Knauer, Jakob Vielhauer, Marcus Makowski, Rickmer Braren, Georgios Kaissis, et al. 2024. Evaluation and mitigation of the limitations of large language models in clinical decision-making. Nature medicine 30, 9 (2024), 2613–2622.DOI
  9. [9]Di Jin, Eileen Pan, Nassim Oufattolle, Wei-Hung Weng, Hanyi Fang, and Peter Szolovits. 2020. What Disease does this Patient Have? A Large-scale Open Domain Question Answering Dataset from Medical Exams. CoRR abs/2009.13081 (2020). arXiv:2009.13081 https://arxiv.org/abs/2009.13081
  10. [10]Alistair Johnson, Tom Pollard, Steven Horng, Leo Anthony Celi, and Roger Mark. 2024. MIMIC-IV. PhysioNet (Oct. 2024). Version 3.1. doi:10.13026/kpb9-mt58
  11. [11]Alistair Johnson, Tom Pollard, Steven Horng, Leo Anthony Celi, and Roger Mark. 2023. MIMIC-IV-Note: Deidentified free-text clinical notes. PhysioNet (Jan. 2023). Version 2.2. doi:10.13026/1n74-ne17DOI
  12. [12]Jerome P Kassirer. 1983. Teaching clinical medicine by iterative hypothesis testing: let’s preach what we practice. 921–923 pages.pubmed.ncbi.nlm.nih.gov/6888486
  13. [13]Daeun Kyung, Hyunseung Chung, Seongsu Bae, Jiho Kim, Jae Ho Sohn, Taerim Kim, Soo Kyung Kim, and Edward Choi. 2025. PatientSim: A Persona-Driven Simulator for Realistic Doctor-Patient Interactions. CoRR abs/2505.17818 (2025). arXiv:2505.17818 doi:10.48550/ARXIV.2505.17818DOI
  14. [14]Junkai Li, Siyu Wang, Meng Zhou, Yitao Li, Yunghwei Lai, Xinhui Kang, Weizhi Ma, and Yang Liu. 2024. Agent Hospital: A Simulacrum of Hospital with Evolvable Medical Agents. CoRR abs/2405.02957 (2024). arXiv:2405.02957 doi:10.48550/ARXIV.2405.02957
  15. [15]Shuyue Stella Li, Vidishia Balachandran, Shangbin Feng, Jonathan S Ilgen, Emma Pierson, Pang Wei Koh, and Yulia Tsvetkov. 2024. Mediq: Question-asking llms and a benchmark for reliable interactive clinical reasoning. In The Thirty-eighth Annual Conference on Neural Information Processing Systems.arxiv.org/abs/2406.00922
  16. [16]Cheng Liang, Pengcheng Qiu, Ya Zhang, Yanfeng Wang, Chaoyi Wu, and Weidi Xie. 2026. Evaluating Large Language Models in Dynamic Clinical Decision-Making with Standardized Patient Cases. CoRR abs/2606.05112 (2026). arXiv:2606.05112 doi:10.48550/ARXIV.2606.05112DOI
  17. [17]Ernest Lim, Yajie Vera He, Jared Josefowitz, Kate Preston, Mohita Chowdhury, Louis Williams, Aisling Higham, Katrina Mason, Mariane Melo, Tom Lawton, et al. 2025. MATRIX: Multi-Agent simulation Framework for safe Interactions and contextual clinical conversational evaluation. arXiv preprint arXiv:2508.19163 (2025).arxiv.org/abs/2508.19163
  18. [18]Minhua Lin, Juncheng Wu, Zijun Wang, Zhan Shi, Yisi Sang, Bing He, Zewen Liu, Tianxin Wei, Zongyu Wu, Zhiwei Zhang, Dakuo Wang, Xiang Zhang, Benoît Dumoulin, Chiang Xie, Yuyin Zhou, Shuhang Wang, and Hanqing Li. 2026. Harness Benefit: Disentangling Evolution Capabilities in Self-Evolving LLM Agents. CoRR abs/2605.30621 (2026). arXiv:2605.30621 doi:10.48550/ARXIV.2605.30621
  19. [19]Tianwei Lin, Wenqiao Zhang, Sijing Li, Yuqian Yuan, Binhe Yu, Haoyuan Li, Wanggui He, Hao Jiang, Mengze Li, Xiaohui Song, Siliang Tang, Jun Xiao, Hui Lin, Yueting Zhuang, and Beng Chin Ooi. 2025. HealthGPT: A Medical Large Vision-Language Model for Unifying Comprehension and Generation via Heterogeneous Knowledge Adaptation. arXiv:2502.09838 [cs.CV] https://arxiv.org/abs/2502.09838
  20. [20]Junling Liu, Peilin Zhou, Yining Hua, Dading Chong, Zhongyu Tian, Andrew Liu, Helin Wang, Chenyu You, Zhenhua Guo, Lei Zhu, and Michael Lingzhi Li. 2023. Benchmarking Large Language Models on CMExam - A comprehensive Chinese Medical Exam Dataset. In Advances in Neural Information Processing Systems 36: Annual Conference on Neural Information Processing Systems 2023, NeurIPS 2023, New Orleans, LA, USA, December 10 - 16, 2023, Alice Oh, Tristan Naumann, Amir Globerson, Kate Saenko, Moritz Hardt, and Sergey Levine (Eds.). http://papers.nips.cc/paper_files/paper/2023/hash/a48ad12d588c597f4725a8b84af647b5-Abstract-Datasets_and_Benchmarks.htmlarxiv.org/abs/2306.03030
  21. [21]Ruoqi Liu, Imran O. Mohiuddin, Austin J. Schoeffler, Kavita Renduchintala, Ashwin Nayak, Prasantha L. Vemu, Shivam Vedak, Kameron C. Black, John L. Havlik, Isaac Ogunmola, Stephen P. Ma, Roopa Dhatt, and Jonathan H. Chen. 2026. PhysicianBench: Evaluating LLM Agents in Real-World EHR Environments. CoRR abs/2605.02240 (2026). arXiv:2605.02240 doi:10.48550/ARXIV.2605.02240
  22. [22]Robert R McCrae and Oliver P John. 1992. An introduction to the five-factor model and its applications. Journal of personality 60, 2 (1992), 175–215.pubmed.ncbi.nlm.nih.gov/1635039
  23. [23]Nikita Mehandru, Brenda Y Miao, Eduardo Rodriguez Almaraz, Madhumita Sushil, Atul J Butte, and Ahmed Alaa. 2024. Evaluating large language models as agents in the clinic. NPJ digital medicine 7, 1 (2024), 84.DOI
  24. [24]Xinyi Mou, Xuanwen Ding, Qi He, Liang Wang, Jingcong Liang, Xinnong Zhang, Jibo Sun, Jiayu Lin, Jie Zhou, Xuanjing Huang, and Zhongyu Wei. 2026. From Individual to Society: A Survey on Social Simulation Driven by Large Language Model-based Agents. ACM Comput. Surv. 58, 11 (2026), 280:1–280:41. doi:10.1145/3800683DOI
  25. [25]Ning Mu, Zhongzhen Huang, Yannian Gu, Shengqian Qin, Shaoting Zhang, and Xiaofan Zhang. 2026. EHRWorld: A Patient-Centric Medical World Model for Long-Horizon Clinical Trajectories. arXiv preprint arXiv:2602.03569 (2026).arxiv.org/abs/2602.03569
  26. [26]Jingwei Ni, Yihao Liu, Xinpeng Liu, Yutao Sun, Mengyu Zhou, Pengyu Cheng, Dexin Wang, Erchao Zhao, Xiaoxi Jiang, and Guanyun Jiang. 2026. Trace2Skill: Distill Trajectory-Local Lessons into Transferable Agent Skills. CoRR abs/2603.25158 (2026). arXiv:2603.25158 doi:10.48550/ARXIV.2603.25158DOI
  27. [27]Harsha Nori, Mayank Daswani, Christopher Kelly, Scott M. Lundberg, Marco Tulio Ribeiro, Marc Wilson, Xiaoxuan Liu, Nikesh Sounderajah, Jonathan Carlson, Matthew P. Lungren, Bay Gross, Peter Hames, Mustafa Suleyman, Dominic King, and Eric Horvitz. 2025. Sequential Diagnosis with Language Models. CoRR abs/2506.22405 (2025). arXiv:2506.22405 doi:10.48550/ARXIV.2506.22405DOI
  28. [28]Stephen G Pauker and Jerome P Kassirer. 1980. The threshold approach to clinical decision making. New England Journal of Medicine 302, 20 (1980), 1109–1117.
  29. [29]Chuhan Qiao, Jianghua Huang, Daxing Zhao, Ziding Liu, Yanjun Shen, Bing Cheng, Wei Lin, and Kai Wu. 2026. MedConsultBench: A Full-Cycle, Fine-Grained, Process-Aware Benchmark for Medical Consultation Agents. arXiv preprint arXiv:2601.12661 (2026).arxiv.org/abs/2601.12661
  30. [30]Niklas Rodemund, Andreas Kokoefer, Bernhard Wernly, and Crispiana Cozowicz. 2023. Salzburg Intensive Care database (SICdb), a freely accessible intensive care database. PhysioNet (Feb. 2023). Version 1.0.5. doi:10.13026/67ww-6m34
  31. [31]Samuel Schmidgall, Rojin Ziaei, Carl Harris, Ji Woong Kim, Edwardo Pontes Reis, Jeffrey Jopling, and Michael Moor. 2026. AgentClinic: a multimodal benchmark for tool-using clinical AI agents. npj Digit. Medicine 9, 1 (2026). doi:10.1038/S41746-026-02674-7DOI
  32. [32]Karan Singhal, Shekoofeh Azizi, Tao Tu, S Sara Mahdavi, Jason Wei, Hyung Won Chung, Nathan Scales, Ajay Tanwani, Heather Cole-Lewis, Stephen Pfohl, et al. 2023. Large language models encode clinical knowledge. Nature 620, 7972 (2023), 172–180.DOI
  33. [33]Mirac Suzgun, Mert Yüksekögünül, Federico Bianchi, Dan Jurafsky, and James Zou. 2026. Dynamic Cheatsheet: Test-Time Learning with Adaptive Memory. In Proceedings of the 19th Conference of the European Chapter of the Association for Computational Linguistics, EACL 2026 - Volume 1: Long Papers, Rabat, Morocco, March 24-29, 2026, Vera Demberg, Kentaro Inui, and Lluís Márquez (Eds.). Association for Computational Linguistics, 7080–7106. doi:10.18653/V1/2026.EACL-LONG.333DOI
  34. [34]M2 Team, Chengfeng Dou, Chong Liu, Fan Yang, Fei Li, Jiyuan Jia, Mingyang Chen, Qiang Ju, Shuai Wang, Shunya Dang, Tianpeng Li, Xiangrong Zeng, and et al. Yijie Zhou. 2025. Baichuan-M2: Scaling Medical Capability with Large Verifier System. arXiv:2509.02208 [cs.LG] https://arxiv.org/abs/2509.02208
  35. [35]Tao Tu, Mike Schaekermann, Anil Palepu, Khaled Saab, Jan Freyberg, Ryutaro Tanaka, Amy Wang, Brenna Li, Mohamed Amin, Yong Cheng, et al. 2025. Towards conversational diagnostic artificial intelligence. Nature 642, 8067 (2025), 442–450.DOI
  36. [36]Peiyi Wang, Lei Li, Liang Chen, Zefan Cai, Dawei Zhu, Binghuai Lin, Yunbo Cao, Lingpeng Kong, Qi Liu, Tianyu Liu, and Zhifang Sui. 2024. Large Language Models are not Fair Evaluators. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), ACL 2024, Bangkok, Thailand, August 11-16, 2024, Lun-Wei Ku, Andre Martins, and Vivek Srikumar (Eds.). Association for Computational Linguistics, 9440–9450. doi:10.18653/V1/2024.ACL-LONG.511DOI
  37. [37]Junde Wu, Jiayuan Zhu, Minghao Hu, Fenglin Liu, and Jiazhen Pan. 2026. MedRSI: Recursive Self-Improvement for Medical Agents via Clinically Aligned Self-Evolution. CoRR abs/2609.24838 (2026). arXiv:2609.24838 doi:10.48550/ARXIV.2609.24838
  38. [38]Minghao Wu, Yuting Yan, Zhenyang Cai, Ke Ji, Chuangsen Fang, Ziying Sheng, Xidong Wang, Rongsheng Wang, Hejia Zhang, Shuang Li, Benyou Wang, and Hongyuan Zha. 2026. Agentifying Patient Dynamics within LLMs through Interacting with Clinical World Model. CoRR abs/2605.14723 (2026). arXiv:2605.14723 doi:10.48550/ARXIV.2605.14723DOI
  39. [39]Peng Xia, Jianwen Chen, Hanyang Wang, Jiaqi Liu, Kaide Zeng, Yu Wang, Siwei Han, Yiyang Zhou, Xujian Zhao, Haifeng Chen, Zeyu Zheng, Cihang Xie, and Huaxiu Yao. 2026. SkillRL: Evolving Agents via Recursive Skill-Augmented Reinforcement Learning. CoRR abs/2602.08234 (2026). arXiv:2602.08234 doi:10.48550/ARXIV.2602.08234DOI
  40. [40]Yuzhang Xie, Keqi Han, Yunpeng Xiao, Hejie Cui, Guanchen Wu, Ziyang Zhang, Kai Shu, Jiaying Lu, Xiao Hu, and Carl Yang. 2026. EHRBench: An Automated Benchmark for EHR-based Benchmark for Clinical Decision Making in LLMs. CoRR abs/2605.30637 (2026). arXiv:2605.30637 doi:10.48550/ARXIV.2605.30637DOI
  41. [41]Ran Xu, Yuchen Zhuang, Yishan Zhong, Yue Yu, Zifeng Wang, Xiangru Tang, Hang Wu, May D Wang, Peifeng Ruan, Donghan Yang, et al. 2025. MedAgent-Gym: A Scalable Agentic Training Environment for Code-Centric Reasoning in Biomedical Data Science. arXiv preprint arXiv:2506.04405 (2025).arxiv.org/abs/2506.04405
  42. [42]Weiwen Xu, Hou Pong Chan, Long Li, Mahani Aljunied, Ruifeng Yuan, Jianyu Wang, Chengxiao Xiao, Guizhen Chen, Chaoqun Liu, Zhaodonghu Li, et al. 2025. Lingshu: A Generalist Foundation Model for Unified Multimodal Medical Understanding and Reasoning. arXiv preprint arXiv:2506.07044 (2025).arxiv.org/abs/2506.07044
  43. [43]Haoran Ye, Xuning He, Vincent Arak, Haonan Dong, and Guojie Song. 2026. Meta Context Engineering via Agentic Skill Evolution. CoRR abs/2601.21557 (2026). arXiv:2601.21557 doi:10.48550/ARXIV.2601.21557
  44. [44]Eric Zelikman, Elian Lorcher, Lester Mackey, and Adam Tauman Kalai. 2023. Self-Taught Optimizer (STOP): Recursively Self-Improving Code Generation. CoRR abs/2310.02304 (2023). arXiv:2310.02304 doi:10.48550/ARXIV.2310.02304
  45. [45]Qizheng Zhang, Changran Hu, Shubhangi Upasani, Boyuan Ma, Fenglu Hong, Yamsidhar Kamanuru, Jay Rainton, Chen Wu, Mengmeng Ji, Hancheng Li, Urmish Thakker, James Zou, and Kunle Olukotun. 2025. Agentic Context Engineering: Evolving Contexts for Self-Improving Language Models. CoRR abs/2510.04618 (2025). arXiv:2510.04618 doi:10.48550/ARXIV.2510.04618DOI
  46. [46]Shengtao Zhang, Jiaqiang Wang, Ruiwen Zhou, Junwei Liao, Yuchen Feng, Weinan Zhang, Ying Wen, Zhiyu Li, Feiyu Xiong, Yutao Qi, Bo Tang, and Muning Wen. 2026. MemRL: Self-Evolving Agents via Runtime Reinforcement Learning on Episodic Memory. CoRR abs/2601.03192 (2026). arXiv:2601.03192 doi:10.48550/ARXIV.2601.03192DOI
  47. [47]Lianmin Zheng, Wei-Lin Chiang, Ying Sheng, Siyuan Zhuang, Zhanghao Wu, Yonghao Zhuang, Zi Lin, Zhuohan Li, Dacheng Li, Eric P. Xing, Hao Zhang, Joseph E. Gonzalez, and Ion Stoica. 2023. Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena. In Advances in Neural Information Processing Systems 36: Annual Conference on Neural Information Processing Systems 2023, NeurIPS 2023, New Orleans, LA, USA, December 10 - 16, 2023, Alice Oh, Tristan Naumann, Amir Globerson, Kate Saenko, Moritz Hardt, and Sergey Levine (Eds.). http://papers.nips.cc/paper_files/paper/2023/hash/91f18a1287b398d378eff22505bf41832-Abstract-Datasets_and_Benchmarks.htmlarxiv.org/abs/2306.05685
  48. [48]Yuxin Zuo, Shang Qu, Yifei Li, Zhang-Ren Chen, Xuekai Zhu, Ermo Hua, Kaiyan Zhang, Ning Ding, and Bowen Zhou. 2025. MedXpertQA: Benchmarking Expert-Level Medical Reasoning and Understanding. In Forty-second International Conference on Machine Learning, ICML 2025, Vancouver, BC, Canada, July 13-19, 2025 (Proceedings of Machine Learning Research, Vol. 267), Aarti Singh, Maryam Fazel, Daniel Hsu, Simon Lacoste-Julien, Felix Berkenkamp, Tegan Maharaj, Kiri Wagstaff, and Jerry Zhu (Eds.). PMLR / OpenReview.net. https://proceedings.mlr.press/v267/zuo25a.htmlarxiv.org/abs/2501.18362

Paper details

vlnc:W_be5dt967
Version
v1 · latest
Published
Fresh — added to the Hub within the last 30 days
Added
License
CC BY 4.0
Usage

views · 0

downloads · 0

A view counts once per browser per day, a download each time the PDF is requested.

  • Computer Science

LLM

View statement
Looking for related papers…
Contents