betaFeedback

WorksComputer Science

CODA: Cohort- and Drift-aware Foundation Model for Multimodal Clinical Reasoning

Abstract

Recent advances in multimodal foundation models have demonstrated strong performance across diverse clinical tasks. However, existing approaches predominantly operate at the individual sample level and fail to model the interplay between population-level cohort patterns and patient-specific variation that underlie real-world clinical reasoning. In practice, clinicians routinely reason through patient cohorts sharing similar demographic, physiological, or pathological characteristics, while accounting for patient-specific deviations from the group prototype. Motivated by this gap, we propose CODA, a cohort- and drift-aware foundation model for multimodal clinical reasoning. CODA explicitly integrates cohort structure into the modeling pipeline through two complementary mechanisms: cohort-aware population stratification and drift-aware individual adaptation. In particular, we leverage reinforcement learning to discover clinically meaningful patient cohorts, and further model each patient's deviation from cohort prototypes as a structured drift signal encoding fine-grained individual heterogeneity. These cohort- and drift-aware representations are incorporated into the foundation model to enable reasoning that is simultaneously informed by population-level patterns and sensitive to individual-level variation. Extensive experiments on real-world electronic health record (EHR) benchmarks demonstrate that CODA achieves consistent effectiveness and strong generalizability across diverse clinical tasks, including closed-QA, open-QA, and report generation.

Introduction

Generative AI has rapidly evolved from text-only large language models (LLMs) [1, 2] to large vision–language models (LVLMs) [3], enabling unified reasoning over both textual and visual modalities. This paradigm shift is particularly consequential in the medical domain, where clinical decision-making inherently relies on heterogeneous data sources, including medical images, EHR data, and free-text reports [4, 5, 6, 7]. By integrating these modalities within a single framework, recent approaches have demonstrated strong capabilities in medical question answering [8], multimodal

Preprint. understanding [9], and medical decision support [10]. These advances have driven notable progress in applications such as medical knowledge acquisition and education, treatment planning, and patient outcome prediction [11, 12].

Despite these advances, existing medical foundation models [13, 14, 15] are developed in a task-centric manner, where reasoning is performed on individual queries with limited consideration of broader clinical context. However, clinicians rarely reason about a patient in isolation. Instead, they implicitly compare the current case with previously encountered patients, drawing on experience with recurring patterns across similar populations [16]. Without this kind of contextual grounding, medical foundation models struggle to capture meaningful clinical variation, often leading to fragmented or overly simplified reasoning.

A natural way to incorporate such context is through patient cohorts [17]. By grouping patients with similar clinical profiles, such as demographics or specific health conditions, cohorts help guide how individual cases are interpreted and managed. Figure 1 illustrates the transition from conventional instance-level reasoning to cohort-aware clinical reasoning. Take the age cohorts as an example, a sudden episode of chest pain may be treated as a potential medical emergency for elderly patients with a history of heart disease, often requiring immediate hospital evaluation, whereas the same symptoms for young and healthy individuals may be attributed to benign causes such as muscle strain or stress. This type of cohort-based reasoning allows clinical decisions to be grounded in population-level patterns.

Rethinking Clinical Reasoning: From instance-level predictions to cohort-aware modeling

Figure 1. Rethinking Clinical Reasoning: From instance-level predictions to cohort-aware modeling.

Furthermore, relying on cohort structure alone is insufficient to support accurate clinical decisions. Even within the same cohort, patients may differ in ways that lead to different clinical judgments. For example, two elderly patients with similar cardiovascular risk factors may both have chest pain, yet one may show stable vital signs and mild symptoms that allow for conservative monitoring, while the other may exhibit severe pain that demands immediate intervention. Such differences in symptoms and clinical signals often determine the final intervention, yet are not well captured by coarse cohort patterns. As a result, effective clinical decision-making requires jointly modeling cohort structure and individual heterogeneity, a capability that remains underexplored in existing approaches.

To this end, we propose CODA2, a COhort- and Drift-Aware foundation model that explicitly characterizes both cohort structure and individual heterogeneity. Our framework consists of two key components. First, we formulate cohort construction as a reinforcement learning process that automatically partitions patients into clinically meaningful groups. This process builds a cohort tree, where each internal node corresponds to a feature-based status and action, and each leaf represents a cohort. A cohort encoder is then employed to encode these cohorts, and the resulting embeddings are injected into the model input to provide cohort-level context. Second, we move beyond static cohorts and introduce a continuously evolving cohort prototype space, where cohort representations are dynamically updated. For each patient, we further compute a cohort-drift signal that captures how the individual deviates from the corresponding cohorts. This signal reflects patient-specific characteristics that are not explained by shared cohort patterns, and is incorporated into the model to refine its reasoning at the individual level. These cohort-aware and drift-aware representations enable more precise and context-aware clinical reasoning. We summarize our main contributions as follows:

  • We propose a cohort- and drift-aware foundation model for multimodal clinical reasoning that jointly incorporates population-level cohort structure and patient-specific heterogeneity.

  • We present a reinforcement learning-based cohort discovery and dynamic prototype updates strategy, enabling adaptive and data-driven cohort modeling.

  • We explicitly model the individual deviations from cohort prototypes, and incorporate such signals into the backbone through drift-conditioned adapters.

  • Extensive experiments demonstrate that CODA consistently outperforms both general-purpose and medical-specific baselines across closed-QA, open-QA, and report generation tasks on diverse real-world EHR datasets, highlighting its effectiveness in enhancing clinical reasoning.

Preliminary

Multimodal Clinical Data Representation in Foundation Models

Modern foundation models encode heterogeneous clinical data by projecting multimodal inputs into a unified token embedding space. Given the input x={I,X}x = \{\mathcal{I}, \mathcal{X}\} and ground truth output sequence yy, where I∈RH×W×3\mathcal{I} \in\mathbb{R}^{H \times W \times3} represents a medical image and X={x1,x2,…,xn,q}\mathcal{X} = \{x_1, x_2, \ldots, x_n, \mathbf{q}\} denotes nn serialized clinical records and the question q\mathbf{q}, existing LVLMs first encode the visual and textual inputs separately, and then fuse them into a shared embedding space as

Z(0)=[fI→t(EI(I)) ; Et(X)]∈R(LI+Lt)×dt(1)\mathbf{Z}^{(0)} = [f_{\mathcal{I}\to t}(\mathcal{E}_{\mathcal{I}}(\mathcal{I}))\,;\,\mathcal{E}_{t}(\mathcal{X})] \in\mathbb{R}^{(L_{\mathcal{I}}+L_t)\times d_t} \tag*{(1)}

where EI(⋅)\mathcal{E}_{\mathcal{I}}(\cdot) is a vision encoder and Et(⋅)\mathcal{E}_{t}(\cdot) is the textual encoder. LIL_{\mathcal{I}} and LtL_t denote the numbers of visual and textual tokens, respectively. fI→tf_{\mathcal{I}\to t} maps the visual embeddings into the shared embedding space, and dtd_t denotes the dimensions of the shared embedding space.

The fused embedding is then processed by LbL_b Transformer blocks F\mathcal{F}, Z(ℓ)=F(ℓ)(Z(ℓ−1))\mathbf{Z}^{(\ell)} = \mathcal{F}^{(\ell)}(\mathbf{Z}^{(\ell-1)}), ℓ=1,…,Lb\ell= 1, \ldots, L_b yielding the final contextualized representation Z(Lb)\mathbf{Z}^{(L_b)}. Conditioned on Z(Lb)\mathbf{Z}^{(L_b)}, the model generates the output sequence autoregressively with learnable parameters θ\theta as

pθ(y∣x)=∏i=1∣y∣pθ(yi∣y<i,Z(Lb))(2)p_\theta(y \mid x) = \prod_{i=1}^{|y|} p_\theta(y_i \mid y_{<i}, \mathbf{Z}^{(L_b)}) \tag*{(2)}

Reinforcement Learning

Reinforcement learning (RL) is a general framework designed for sequential decision-making, typically formulated as a Markov Decision Process (MDP) defined as

M=(S,A,P,r,γ),s∈S, a∈A(3)\mathcal{M} = (\mathcal{S}, \mathcal{A}, P, r, \gamma), \quad s \in\mathcal{S},\ a \in\mathcal{A} \tag*{(3)}

where S\mathcal{S} denotes the state space, A\mathcal{A} denotes the action space, PP denotes the transition dynamics. rr is the reward function, and γ∈[0,1]\gamma\in[0,1] is the discount factor. Given a state ss, a policy πϕ(a∣s)\pi_\phi(a \mid s) specifies a distribution over the action space. The policy induces a trajectory τ\tau with probability p(τ)p(\tau):

τ=(s1,a1,s2,a2,…,sT),p(τ)=p(s1)∏k=1Tπϕ(ak∣sk)P(sk+1∣sk,ak)(4)\tau= (s_1, a_1, s_2, a_2, \ldots, s_T), \quad p(\tau) = p(s_1)\prod_{k=1}^{T}\pi_\phi(a_k \mid s_k)P(s_{k+1} \mid s_k,a_k) \tag*{(4)}

The objective of RL is to learn a policy that maximizes the expected cumulative reward as

max⁡πϕEτ∼pτ(⋅∣πϕ)[∑k=1Tγk−1r(sk,ak)],(5)\max_{\pi_\phi}\mathbb{E}_{\tau\sim p_\tau(\cdot\mid\pi_\phi)}\left[\sum_{k=1}^{T}\gamma^{k-1}r(s_k,a_k)\right], \tag*{(5)}

where pτ(⋅∣πϕ)p_\tau(\cdot\mid\pi_\phi) denotes the trajectory distribution induced by policy πϕ\pi_\phi.

Methodology

CODA Overview

As illustrated in Figure 2, CODA adopts a two-stage formulation to enable cohort-aware clinical reasoning. In the first stage, we learn a hierarchical cohort tree T\mathcal{T} that explicitly partitions the patient population P\mathcal{P} into a clinically meaningful cohort set C\mathcal{C}. In the cohort tree, each internal node parameterizes a feature-based state ss and partition action aa, while each leaf node represents a cohort c∈Cc \in\mathcal{C}. These partition actions are optimized with reinforcement learning, allowing the cohort structure to be shaped not only by clinical heterogeneity but also by its utility for downstream generation. With the learned cohorts in C\mathcal{C}, we derive a cohort-aware representation for each sample and inject it into the backbone, thereby providing explicit population-level context rather than relying on the model to infer such structure implicitly from its parameters alone. In the second stage, building upon this cohort-level grounding, we maintain dynamically updated cohort prototypes hch_c during training and characterize each sample by its cohort-relative deviation Δ(x,c)\Delta(x,c). This cohort-relative deviation captures fine-grained patient-specific variability beyond shared cohort characteristics, enabling the model to reason jointly over stable population patterns and individualized clinical departures. The cohort- and drift-aware framework is encapsulated in the following integrated objective:

Overview of CODA for cohort- and drift-aware multimodal clinical reasoning

Figure 2. Overview of CODA for cohort- and drift-aware multimodal clinical reasoning.

J(θ,πϕ)=∑x,y[Eτ∼pπϕ(⋅∣x)[R(x,τ)]+H(pπϕ(⋅∣x))⏟Cohort-aware Population Stratification+log⁡pθ(y∣x,Δ(x,c))⏟Drift-aware Individual Adaptation](6)\mathcal{J}(\theta,\pi_\phi)=\sum_{x,y}\left[\underbrace{\mathbb{E}_{\tau\sim p_{\pi_\phi}(\cdot\mid x)}[\mathcal{R}(x,\tau)]+\mathcal{H}(p_{\pi_\phi}(\cdot\mid x))}_{\text{Cohort-aware Population Stratification}}+\underbrace{\log p_\theta(y\mid x,\Delta(x,c))}_{\text{Drift-aware Individual Adaptation}}\right] \tag*{(6)}

where θ\theta denotes the learnable model parameters and πϕ\pi_\phi is the cohort-partitioning policy, R(x,τ)\mathcal{R}(x,\tau) is the trajectory-level cohort discovery reward, H(pπϕ(⋅∣x))\mathcal{H}(p_{\pi_\phi}(\cdot\mid x)) is the policy entropy for cohort partition exploration.

Cohort-aware Population Stratification

To formalize the context-dependency inherent in clinical heuristics, CODA eschews the conventional assumption of a homogeneous patient distribution. Instead, CODA introduces a hierarchical stratification process that partitions the patient population space P\mathcal{P} into a set of clinically distinct cohorts C\mathcal{C}. This process is governed by two synergistic components, i.e., an RL-based cohort discovery mechanism and a cohort context injection schema that injects cohort-level context into the model.

RL-based Cohort Discovery. We formulate cohort discovery as a hierarchical policy learning problem, where a binary-tree-structured policy πϕ\pi_\phi recursively partitions the patient population P\mathcal{P} into a set of cohorts C\mathcal{C}. Formally, we define a cohort tree as a hierarchical binary tree T=(V,E)\mathcal{T}=(V,E), where V={v1,…,vN}V=\{v_1,\ldots,v_N\} is the set of nodes and E⊆V×VE\subseteq V\times V is the set of directed edges representing parent–child relationships. Each node v∈Vv\in V corresponds to a subset of patients Pv⊆P\mathcal{P}_v\subseteq\mathcal{P}, with the root node representing the entire population. Internal nodes correspond to decision points that further partition the patient subset, while leaf nodes represent terminal cohorts corresponding to clinically homogeneous patient groups. Each node vv is further associated with (i) a state representation svs_v that summarizes the statistical and structural properties of the corresponding patient subset, and (ii) a partition action ava_v induced by the policy, which determines how the node is expanded or terminated.

The cohort tree thereby induces a recursive partition of the population space, where each root-to-leaf path encodes a sequence of feature-based decisions that uniquely determines a cohort.

Specifically, for each sample x\mathbf{x}, we extract a clinical feature u=g(x)\mathbf{u}=g(\mathbf{x}), where g(⋅)g(\cdot) maps serialized EHR records into a vector of statistical features capturing patient demographics, clinical conditions, and laboratory signals. Each node vv is represented by a compact state vector:

sv=[μ(UPv);σ(UPv);∣Pv∣;dv],(7)s_v=\left[\mu(\mathcal{U}_{\mathcal{P}_v});\sigma(\mathcal{U}_{\mathcal{P}_v});|\mathcal{P}_v|;d_v\right], \tag*{(7)}

where UPv={ui}i∈Pv\mathcal{U}_{\mathcal{P}_v}=\{\mathbf{u}_i\}_{i\in\mathcal{P}_v} denotes the collection of feature vectors of patients in node vv. μ(⋅)\mu(\cdot) and σ(⋅)\sigma(\cdot) denote the mean and standard deviation, respectively. ∣Pv∣|\mathcal{P}_v| is the number of patients and dvd_v is the depth of node vv.

At each node vv, the cohort policy πϕ\pi_{\phi} induces a stochastic action ava_v over the patient subset Pv\mathcal{P}_v to specify whether to stop, the selected medical feature, and the corresponding split threshold, where action ava_v is defined as:

av=(zv,jv,tv)∼πϕ(⋅∣sv).(8)a_v=(z_v,j_v,t_v)\sim\pi_{\phi}(\cdot\mid s_v). \tag*{(8)}

where zv∈{0,1}z_v\in\{0,1\} is a binary stopping variable, with zv=1z_v=1 indicating that vv becomes a terminal cohort. jvj_v is the selected feature, and tvt_v is the split threshold. When zv=0z_v=0, the node is expanded by partitioning Pv\mathcal{P}_v into two disjoint subsets:

PvL={i∈Pv∣ui,jv≤tv},PvR=Pv∖PvL,(9)\mathcal{P}_{v_L}=\{i\in\mathcal{P}_v\mid u_{i,j_v}\leq t_v\},\qquad\mathcal{P}_{v_R}=\mathcal{P}_v\setminus\mathcal{P}_{v_L}, \tag*{(9)}

where vLv_L and vRv_R denote the left and right child nodes, respectively.

The reward function is designed to encourage each split to produce more compact and balanced child cohorts while penalizing unnecessary tree depth:

rv=(σ(UPv)−∑k∈{L,R}∣Pvk∣∣Pv∣σ(UPvk))+(1−∣∣PvL∣−∣PvR∣∣∣Pv∣)−dvdmax⁡,(10)r_v=\left(\sigma(\mathcal{U}_{\mathcal{P}_v})-\sum_{k\in\{L,R\}}\frac{|\mathcal{P}_{v_k}|}{|\mathcal{P}_v|}\sigma(\mathcal{U}_{\mathcal{P}_{v_k}})\right)+\left(1-\frac{\left||\mathcal{P}_{v_L}|-|\mathcal{P}_{v_R}|\right|}{|\mathcal{P}_v|}\right)-\frac{d_v}{d_{\max}}, \tag*{(10)}

where dmax⁡d_{\max} is a normalization constant denoting the maximum depth. Here, the first term measures the reduction in intra-cohort variance after splitting, the second term discourages highly imbalanced partitions, and the last term penalizes overly deep trees.

Cohort Context Encoding. For each cohort cc, we derive a representative embedding via a similarity-aware aggregation over its members, which is then mapped to a cohort embedding ec\mathbf{e}_c and injected into the multimodal input sequence for the patients Pc\mathcal{P}_c within cohort cc to condition generation:

Z(0)=[fI→t(EI(I));ec;Et(X)],ec=fc(∑i∈Pcexp⁡(cos⁡(hi,hˉc))∑j∈Pcexp⁡(cos⁡(hj,hˉc))hi),(11)\mathbf{Z}^{(0)}=\left[f_{I\rightarrow t}(\mathcal{E}_I(\mathcal{I}));\mathbf{e}_c;\mathcal{E}_t(\mathcal{X})\right],\qquad\mathbf{e}_c=f_c\left(\sum_{i\in\mathcal{P}_c}\frac{\exp(\cos(\mathbf{h}_i,\bar{\mathbf{h}}_c))}{\sum_{j\in\mathcal{P}_c}\exp(\cos(\mathbf{h}_j,\bar{\mathbf{h}}_c))}\mathbf{h}_i\right), \tag*{(11)}

where fcf_c is a learnable mapping function, hi=fu→h(ui)\mathbf{h}_i=f_{u\rightarrow h}(\mathbf{u}_i) is the learned patient representation based on statistical feature ui\mathbf{u}_i via function fu→hf_{u\rightarrow h}, and hˉc\bar{\mathbf{h}}_c is the centroid of cohort cc in the feature space.

To jointly optimize cohort discovery and cohort-aware generation, we define a unified objective that integrates policy learning and downstream generation:

Lcohort=ET∼pπ(⋅∣πϕ)[∑v∈V(−log⁡πϕ(av∣sv)Rv−H(πϕ(⋅∣sv)))]+λgLgen,(12)\mathcal{L}_{\mathrm{cohort}}=\mathbb{E}_{\mathcal{T}\sim p_{\pi}(\cdot\mid\pi_{\phi})}\left[\sum_{v\in\mathcal{V}}\left(-\log\pi_{\phi}(a_v\mid s_v)R_v-\mathcal{H}(\pi_{\phi}(\cdot\mid s_v))\right)\right]+\lambda_g\mathcal{L}_{\mathrm{gen}}, \tag*{(12)}

where Rv=rv−LgenR_v=r_v-\mathcal{L}_{\mathrm{gen}} denotes the generation-aware reward and H(πϕ(⋅))\mathcal{H}(\pi_{\phi}(\cdot)) is the entropy of the policy that encourages exploration. Lgen\mathcal{L}_{\mathrm{gen}} is the downstream generation loss with λg\lambda_g balancing policy learning and generation performance.

In this way, a cohort tree T\mathcal{T} and cohort embeddings ec\mathbf{e}_c are jointly learned. The resulting structure captures population-level heterogeneity via hierarchical partitioning, while the learned embeddings encode cohort-specific context for clinical reasoning. Through the proposed generation-aware objective, cohort discovery is directly aligned with multimodal reasoning, enabling the model to move beyond purely structure-driven partitioning toward reasoning-aware population stratification.

Drift-aware Individual Adaptation

While the cohort tree T\mathcal{T} provides population-level structure, clinical reasoning further requires capturing patient-specific deviations from cohort prototypes. To this end, we introduce a drift-aware individual adaptation mechanism that explicitly models the cohort-relative deviation of each sample and integrates it into the generation process.

Cohort Prototype Representation. For each cohort cc, we maintain a learnable prototype representation hc\mathbf{h}_c, which summarizes the central clinical characteristics of the cohort and is initialized using the previously learned cohort embedding ec\mathbf{e}_c. During training, these prototypes are dynamically updated in an online manner based on the incoming samples. Specifically, for a sample x\mathbf{x} of patient pp assigned to cohort cc with feature representation hp\mathbf{h}_p, we update the corresponding prototype using an exponential moving average mechanism:

hc←(1−βw(p,c))hc+βw(p,c)hp∥hp∥2,(13)\mathbf{h}_c \leftarrow(1-\beta w(p,c))\mathbf{h}_c+\beta w(p,c)\frac{\mathbf{h}_p}{\lVert\mathbf{h}_p\rVert_2}, \tag*{(13)}

where w(p,c)∈[0,1]w(p,c) \in[0,1] denotes the assignment confidence of patient pp to cohort cc, and the feature is ℓ2\ell_2-normalized to stabilize prototype updates. hc\mathbf{h}_c replaces ec\mathbf{e}_c in Z(0)\mathbf{Z}^{(0)} and serves as a conditioning signal in the multimodal input sequence, analogous to ec\mathbf{e}_c.

Drift-aware Deviation Modeling. While the cohort tree T\mathcal{T} captures population-level structure, clinical reasoning further requires modeling patient-specific deviations from cohort prototypes. To this end, we introduce a unified drift-aware deviation mechanism that explicitly characterizes individual departures from cohort-level patterns and injects them into the generation process.

Given a sample x\mathbf{x} with feature representation hp\mathbf{h}_p and assigned cohort cc, we compute its cohort-relative deviation as Δ(x,c)=hp−hc\Delta(\mathbf{x},c)=\mathbf{h}_p-\mathbf{h}_c, which captures patient-specific variation beyond the cohort prototype. We first transform this deviation into a compact drift-aware fusion code:

dx=LN⁡(Wd[Δ(x,c);hp;hc;Δ(x,c)⊙hc]+bd),(14)\mathbf{d}_x=\operatorname{LN}\left(\mathbf{W}_d[\Delta(\mathbf{x},c);\mathbf{h}_p;\mathbf{h}_c;\Delta(\mathbf{x},c)\odot\mathbf{h}_c]+\mathbf{b}_d\right), \tag*{(14)}

where [⋅;⋅][\cdot;\cdot] denotes concatenation, ⊙\odot denotes element-wise interaction, LN⁡(⋅)\operatorname{LN}(\cdot) denotes layer normalization, Wd\mathbf{W}_d is a learnable weight matrix and bd\mathbf{b}_d is a learnable bias.

The drift code is then used to modulate the hidden representation at transformer block ℓ\ell through a residual adapter. Compared to a simple additive bias, which applies a uniform shift, our formulation introduces a multiplicative interaction between the deviation and the hidden representation. This enables context-dependent, feature-wise modulation, allowing the model to adapt differently across features and tokens.

Z~(ℓ)=Z(ℓ)+Wℓα((1+tanh⁡(Wℓγdx))⊙GELU⁡(WℓαZ(ℓ))+Wℓβdx),(15)\widetilde{\mathbf{Z}}^{(\ell)}=\mathbf{Z}^{(\ell)}+\mathbf{W}_{\ell}^{\alpha}\left((1+\tanh(\mathbf{W}_{\ell}^{\gamma}\mathbf{d}_x))\odot\operatorname{GELU}(\mathbf{W}_{\ell}^{\alpha}\mathbf{Z}^{(\ell)})+\mathbf{W}_{\ell}^{\beta}\mathbf{d}_x\right), \tag*{(15)}

where Wℓo\mathbf{W}_{\ell}^{o} is a learnable output projection that maps the adapted signal back to the hidden space. The term GELU⁡(WℓαZ(ℓ))\operatorname{GELU}(\mathbf{W}_{\ell}^{\alpha}\mathbf{Z}^{(\ell)}) corresponds to a nonlinear transformation of the transformer output, capturing content-dependent features. The modulation term 1+tanh⁡(Wℓγdx)1+\tanh(\mathbf{W}_{\ell}^{\gamma}\mathbf{d}_x) acts as a feature-wise scaling factor conditioned on the cohort-relative deviation, where tanh⁡(⋅)\tanh(\cdot) ensures bounded and stable modulation. Finally, Wℓβdx\mathbf{W}_{\ell}^{\beta}\mathbf{d}_x introduces an additive bias term that shifts the representation based on the deviation signal.

Cohort-aware Generation. The adapted hidden representation Z~(Lb)\widetilde{\mathbf{Z}}^{(L_b)} at the final transformer block LbL_b is then used for autoregressive generation:

pθ(y∣x)=∏i=1∣y∣pθ(yi∣y<i,Z~(Lb)).(16)p_{\theta}(\mathbf{y}\mid\mathbf{x})=\prod_{i=1}^{|\mathbf{y}|}p_{\theta}\left(y_i\mid y_{<i},\widetilde{\mathbf{Z}}^{(L_b)}\right). \tag*{(16)}

To facilitate individual deviation modeling and enforce inter-cohort separability, we introduce a drift loss that explicitly encodes both intra-cohort consistency and inter-cohort discrimination by minimizing the deviation from the assigned cohort prototype while imposing a margin-based constraint against alternative cohorts.

Ldrift=Lgen+λd(∥Δ(x,c)∥22+max⁡(0,m−hpThc∥hp∥∥hc∥+hpThc′∥hp∥∥hc′∥)).(17)\mathcal{L}_{\mathrm{drift}}=\mathcal{L}_{\mathrm{gen}}+\lambda_d\left(\lVert\Delta(\mathbf{x},c)\rVert_2^2+\max\left(0,m-\frac{\mathbf{h}_p^{\mathsf{T}}\mathbf{h}_c}{\lVert\mathbf{h}_p\rVert\lVert\mathbf{h}_c\rVert}+\frac{\mathbf{h}_p^{\mathsf{T}}\mathbf{h}_{c'}}{\lVert\mathbf{h}_p\rVert\lVert\mathbf{h}_{c'}\rVert}\right)\right). \tag*{(17)}

where m>0m > 0 is the cohort margin, c′c' denotes the cohort with the second highest confidence to pp, and λd\lambda_d controls the trade-off between generation performance and cohort-relative deviation modeling.

Unlike standard generation, where the model relies solely on input features, our formulation conditions the generation process on both the cohort prototype hch_c and the cohort-relative deviation Δ(x,c)\Delta(x, c). This allows the model to first anchor predictions to population-level patterns, and then refine them based on patient-specific variations. Overall, this design enables a unified integration of cohort-level regularities and individual-level deviations, leading to context-aware and fine-grained generation.

Experiments

Experimental Setup

Datasets. We conduct extensive experiments on four real-world EHR benchmarks, i.e., EHR-Bench [18], DIYHealth-Bench [19], MIMIC-CXR [20], and PadChest [21]. Specifically, the admissions, services, ingredient events (IE), and provider order entry (POE) tasks from EHR-Bench, together with MIMIC-CXR, are treated as in-distribution (ID) datasets, while the Transfers task from EHR-Bench, Obesity from DIYHealth-Bench, and PadChest are used as out-of-distribution (OOD) datasets. The model is trained on the ID training datasets and evaluated on both ID and OOD settings. Through these datasets, we aim to provide a comprehensive assessment of the effectiveness and generalizability of CODA in answering practical medical questions and generating clinical reports.

Baselines. We evaluate CODA against a broad set of baselines, including state-of-the-art generalist models, (e.g., LLaVA 1.5 [22], Llama 3 [23], InternVL 3 [24], Qwen3 VL [25], Gemma 3 [26], Grok [27], Claude 4.5 Haiku [28] and GPT 5.4 Mini [29]), as well as medical-specific models (e.g., LLaVA-Med v1.5 [8], Med-Flamingo [13], HuatuoGPT-Vision [14], HealthGPT [9], Lingshu [30], and Hulu-Med [31]). All models are evaluated under closed-form multiple-choice question answering (Closed-QA), open-ended question reasoning (Open-QA), and radiology report generation.

Training Strategy. Our training pipeline consists of three stages. (i) Cross-Modal Alignment. To adapt the visual and textual alignment to the medical domain, we train the vision projector of the backbone using a mixture of medical VQA and QA datasets, including MedQuAD [32], PubMedVision [33], SLAKE [34], and VQA-RAD [35]. (ii) Cohort-aware Population Stratification. With the backbone frozen, we train only the RL-related modules on a subset (15%) of the in-distribution data to learn a hierarchical cohort structure. (iii) Drift-aware Individual Adaptation. We fine-tune only the LoRA modules together with deviation-aware components to model patient-specific variations conditioned on cohort context.

Evaluation Metrics. For Closed-QA, we adopt Accuracy (ACC) and Matthews Correlation Coefficient (MCC) [36]. ACC quantifies the proportion of correctly predicted answers, reflecting exact match performance. MCC provides a balanced evaluation by capturing the agreement between predicted and true labels across all classes. For Open-QA, we use BLEU-1 [37] and F1-BioBERT (F1-Bio) [38]. BLEU-1 evaluates lexical overlap based on unigram precision, serving as a signal of surface-level correctness. F1-Bio measures semantic consistency by computing similarity in a contextual embedding space derived from BioBERT [39], enabling robust evaluation beyond surface-level matching. For report generation, we employ F1-RadGraph (F1-Rad) [40] and F1-Bio. F1-Rad assesses the correctness of clinical entities and their relations by comparing structured representations extracted from RadGraph [40], capturing clinical fidelity at a fine-grained level. Please refer to Appendix § 8 for more details about the datasets, model implementation, and evaluation metrics.

Main Results

The experimental results for Closed-QA, Open-QA, and report generation are presented in Table 1 and Table 2. We consider three variants: (i) CODA with a Qwen3 backbone, (ii) CODA with an LLaMA3 backbone, and (iii) CODA with a Qwen3 backbone but without LoRA. We summarize the key findings as follows. (i) State-of-the-art performance. Three variants of CODA consistently outperform all baselines across all tasks, yielding average gains of 42.18% in ACC for Closed-QA, 6.01% in F1-Bio for Open-QA, and 63.82% in F1-Rad for report generation. These results demonstrate the effectiveness of CODA and suggest that explicitly modeling cohort structure and cohort drift provides a principled mechanism for capturing both population-level patterns and individual-level variations, resulting in consistently superior performance. (ii) Out-of-Distribution Generalization. Despite being trained solely on ID data, CODA generalizes well to OOD settings, as evidenced by its strong performance on the Obesity dataset, which represents a home-care scenario with limited EHR availability, as well as on PadChest, where the scale of EHR data is substantially smaller. These results indicate that CODA is robust to both data scarcity and domain shift, maintaining reliable performance across diverse clinical environments. (iii) Robustness across backbones. CODA consistently improves performance across diverse backbone architectures (e.g., Qwen3-VL and LLaMA-based models), demonstrating that the proposed cohort-aware mechanism is model-agnostic and broadly applicable. (iv) Limitations of existing Med-LVLMs. Existing Med-LVLMs are typically trained on large-scale corpora and a diverse set of tasks, largely mirroring general LVLM training paradigms with domain-specific data substitution. However, such models often overlook clinically meaningful reasoning processes and fail to incorporate domain-specific inductive biases or mechanisms. As a result, when data scale is no longer the primary bottleneck, their performance remains constrained by a lack of clinically grounded modeling.

DomainModelAdmissions ACCAdmissions MCCIE ACCIE MCCPOE ACCPOE MCCServices ACCServices MCCTransfers ACCTransfers MCCObesity ACCObesity MCCAvg. ACCAvg. MCC
General DomainLLaVA 1.5-7B40.0020.0619.44-7.4857.0042.8263.5951.4560.6047.6161.9047.9750.4233.74
General DomainLLaMA 3-11B32.8010.7426.524.8535.4014.5543.6425.3237.0016.2954.1239.5838.2518.56
General DomainInternVL 3-8B28.205.0618.94-6.5246.8029.1342.3923.3527.002.5870.9259.6139.0418.87
General DomainQwen 3-VL-8B36.4015.3538.3817.6440.4020.4960.6048.0753.8038.6369.6758.1049.8833.05
General DomainGemma 3-12B18.80-8.0619.95-6.8335.2013.5558.8545.2050.0033.4271.0760.2442.3122.92
General DomainGrok 4.223.60-1.5831.578.5635.2013.4325.190.1740.6020.4556.9239.6935.5113.45
General DomainClaude 4.5 Haiku35.6014.0929.296.5140.4020.7842.6423.5442.6023.5147.9032.6139.7420.17
General DomainGPT 5.4 Mini34.6012.6042.3923.0844.4026.0243.1424.1748.4031.1459.7245.9245.4427.15
Medical DomainLLaVA-Med v1.5-7B21.003.8325.002.2158.0045.0144.8927.4434.4017.2958.4845.8540.2923.60
Medical DomainMed-Flamingo-7B17.40-3.868.08-3.1121.803.1825.694.2827.801.8417.423.6019.700.99
Medical DomainHuatuoGPT-Vision-7B44.0026.0040.6620.7848.4032.1160.1046.9457.8044.6067.9656.0453.1537.74
Medical DomainHealthGPT-14B39.6019.5325.511.5338.2018.0433.4212.1042.0023.5555.6842.7139.0719.58
Medical DomainLingshu-7B44.4026.2645.4528.0650.6034.1078.5571.5657.0044.2670.6360.5257.7744.13
Medical DomainHulu-Med-7B43.6025.1226.012.3953.2037.7560.6047.8436.0014.5665.4753.5647.4830.20
Medical DomainCODA-Qwen3-8B76.6068.7792.4289.9382.8077.0593.5291.3975.2066.8972.3261.2882.1475.88
Medical DomainCODA-Llama3-8B76.6068.7792.9390.5982.0076.0092.2789.7174.8066.3172.9061.7181.9275.52
Medical DomainCODA-Qwen3-8B w/o LoRA72.8063.7592.1789.5982.4076.5385.2980.5570.2060.6172.1660.9879.1772.00

Table 1. Comparison of CODA with baselines under Closed-QA settings.

DomainModelIDOODAvg.
AdmissionsIEPOEServicesMIMIC-CXRTransfersObesityPadChestOpen-QAReport
F1-BioBLEU-1F1-BioBLEU-1F1-BioBLEU-1F1-BioBLEU-1F1-RadF1-BioF1-BioBLEU-1F1-BioBLEU-1F1-RadF1-BioF1-BioBLEU-1F1-RadF1-Bio
General DomainLLaVA 1.5-7B75.9536.9175.1629.2175.6633.5175.0834.121.7968.1275.7941.2772.239.9911.7568.7874.9830.836.7768.45
LLaMA 3-11B74.6835.6572.7324.3874.8336.1874.9734.344.3865.0174.8838.0170.518.687.5462.2873.7729.545.9663.64
InternVL 3-8B72.2024.4373.9125.0172.6123.9072.3923.836.3861.6273.9626.1667.905.349.3261.4572.1621.457.8561.54
Qwen 3-VL-8B68.8218.3168.7717.1468.4716.8268.0717.098.2363.6372.2326.9668.956.408.1566.4269.2217.128.1965.03
Gemma 3-12B72.3326.5671.8625.3073.5730.0774.5733.9810.2768.7574.7225.5866.934.0712.3165.3572.3324.2611.2967.05
Grok 4.269.6119.7168.9719.7168.6117.8269.6519.8613.5272.6271.2125.3267.635.2912.7470.3169.2817.9513.1371.46
Claude 4.5 Haiku65.1215.3666.5017.2465.3115.3865.5715.1810.5467.7567.7820.8464.807.678.5461.9765.8515.289.5464.86
GPT 5.4 Mini68.8623.1565.5526.8069.5923.7768.8422.8415.8872.3071.2430.6565.787.0710.3568.3968.3122.3813.1170.34
Medical DomainLLaVA-Med v1.5-7B73.858.7175.276.3075.5910.0974.447.720.8067.1176.8511.7674.9415.143.8566.7975.169.952.3366.95
Med-Flamingo-7B58.6411.8655.5111.3658.6813.0458.6111.560.1440.5462.249.5157.954.912.6352.0858.6110.371.3846.31
HuatuoGPT-Vision-7B74.3333.9374.5133.4274.0132.4574.0932.507.7770.4574.6933.6672.5610.467.6064.6074.0329.407.6867.53
HealthGPT-14B69.9129.2770.6727.1671.0326.8772.1528.229.0573.0466.5827.6966.497.492.0368.4869.4724.455.5470.76
Lingshu-7B76.1738.1774.6835.0175.8833.4776.2633.2316.1878.1976.1135.1972.9511.161.2368.8575.3431.048.7173.52
Hulu-Med-7B76.2034.7974.4314.5575.4730.7475.6227.6416.3175.8476.0626.0673.7912.455.5269.1575.2624.3710.9172.49
CODA-Qwen3-8B80.8252.4582.2151.6981.0251.9780.2149.1917.4979.0379.5746.4275.3715.6525.5273.9379.8744.5621.5176.48
CODA-Llama3-8B78.4946.3878.7446.1177.3046.0576.3544.5117.3178.5677.4946.1974.9514.5224.0971.7777.2240.6320.7075.17
CODA-Qwen3-8B w/o LoRA80.2950.5081

Table 2. Comparison of CODA with baselines under Open-QA and report generation settings.

Ablation Study and In-Depth Analysis

Component-wise Ablation Study. To assess the contributions of key components in CODA, we conduct an ablation study, with results reported in Table 3 and full results in Appendix § 10.§ 1. Removing the Cohort Context Encoding leads to consistent performance degradation in both in-distribution and out-of-distribution settings, particularly for Closed-QA and report generation tasks. This suggests that explicitly modeling cohort structure is essential for capturing population-level regularities. Removing deviation modeling results in additional performance drops, especially on Open-QA tasks, indicating that cohort-level representations alone are insufficient to capture fine-grained, patient-specific variations required for accurate clinical reasoning. Overall, the full model (CODA) achieves the best performance across all settings, demonstrating that the joint modeling of cohort structure and individual-level deviations is critical for robust generalization.

ModelIDOOD
Closed-QAOpen-QAReportClosed-QAOpen-QAReport
ACCMCCF1-BioBLEU-1F1-RadF1-BioACCMCCF1-BioBLEU-1F1-RadF1-Bio
Vanilla Backbone43.9525.3968.5317.348.2363.6361.7448.3670.5916.688.1566.42
w/o Cohort85.5880.6980.1149.1815.7378.5872.5662.1976.8228.8324.6072.51
w/o Deviation85.5680.7380.1048.8616.3378.9473.0562.9576.6528.8225.1972.41
CODA86.3481.7881.0651.3217.4979.0373.7664.0877.4731.0325.5273.93

Table 3. Ablation study results of CODA. "w/o Cohort" removes the Cohort Context Encoding, while "w/o Deviation" removes Drift-aware Deviation Modeling.

Comparison with Fine-tuned Baselines. To fully validate the effectiveness of CODA, we compare CODA-Qwen3-8B with two representative baselines, Lingshu-7B and InternVL 3-8B, with results presented in Figure 3 and full results in Appendix E.2. CODA consistently achieves the best performance across all tasks and metrics. These results demonstrate that CODA not only improves predictive accuracy but also enhances semantic consistency and clinical fidelity, highlighting the effectiveness of its cohort-aware and deviation-aware modeling in complex medical reasoning tasks. Interpretation Study. We visualize the learned patient representations of CODA on OOD tasks using t-SNE [41]. For clarity, we present results from the top-10 largest cohorts. As shown in Figure 4, samples from the same cohort form compact clusters, while different cohorts are largely well separated, indicating intra-cohort consistency and inter-cohort separability. We also observe mild overlap among cohorts 27, 28, and 29, which can be attributed to their shared ancestor nodes in the cohort hierarchy, leading to partially aligned representations. Overall, the learned embeddings exhibit coherent and discriminative geometric structures, where samples are grouped according to shared population-level patterns, suggesting that cohort semantics are effectively captured in the embedding space.

Comparisons with fine-tuned baselines

Figure 3. Comparisons with fine-tuned baselines.

Visualization of embeddings within different cohorts

Figure 4. Visualization of embeddings within different cohorts.

Efficiency Study. To evaluate the computational efficiency of CODA, we compare it with representative general and medical baselines. The results are summarized in Table 4. Despite introducing cohort-aware population stratification and drift-aware individual adaptation, CODA maintains competitive inference efficiency while achieving substantially stronger clinical reasoning performance. Specifically, CODA achieves 7.95 s/sample and 20.19 tokens/s throughput, remaining comparable to widely used LLaVA-based baselines while exhibiting substantially lower sample latency than models with higher output complexity, such as InternVL3-8B and Qwen3-VL-8B. In addition, CODA maintains a practical memory footprint for standard single-GPU deployment, while the additional FLOPs/token introduced by cohort-aware conditioning remain modest. Overall, the results demonstrate that CODA achieves a favorable effectiveness–efficiency trade-off, enabling enhanced population-grounded clinical reasoning without sacrificing practical inference efficiency.

ModelTokens (tokens/sample)Latency (s/sample)Token Throughput (tokens/s)Peak GPU Mem. (GB)FLOPs (G/token)
LLaVA 1.5-7B187.318.6221.7313.2713.60
LLaVA-Med v1.5-7B69.563.2121.6714.2114.60
Hulu-Med-7B72.755.7928.9515.6514.40
Lingshu-7B167.625.7925.1815.6514.40
InternVL 3-8B417.5519.5321.3815.0014.80
Qwen 3-VL-8B545.6523.1523.5716.4715.03
CODA-8B160.517.9520.1917.2515.69

Table 4. Efficiency comparison of CODA against different baselines.

Expert Review

To assess the practical significance of CODA, we conduct an expert review involving both GPT-5.4 and human experts. Specifically, we randomly sample 250 Open-QA instances from the OOD Transfers task and 250 report generation instances from the OOD PadChest task. We compare against five strong baselines, including Grok4.2, Hulu-Med, Qwen3 VL, Lingshu, and LLaVA 1.5.

For each sample, experts are asked to select their first preference, i.e., the response that is most accurate and clinically reliable, from six candidate answers. The results are presented in Figure 5. CODA is consistently chosen as the first preference by both human experts and GPT-5.4, demonstrating its superior clinical utility and robustness. Among the baselines, Grok-4.2 and Lingshu are ranked as the second-best models by human experts and GPT-5.4, respectively. Despite these variations, all baselines remain substantially behind CODA, indicating that the proposed cohort- and drift-aware mechanisms provide consistent and reliable improvements.

Expert review results on the first preference

Figure 5. Expert review results on the first preference.

We provide additional in-depth analyses on few-shot settings, hyperparameter sensitivity, and case studies in Appendices E.3, E.4, and F, further supporting the generality and reliability of CODA.

Conclusions

Generative AI has significantly advanced AI for healthcare. However, current medical foundation models largely overlook the interplay between population-level cohort patterns and patient-specific variation that underlies real-world clinical reasoning. In this paper, we propose CODA, a cohort- and drift-aware medical foundation model that explicitly discovers patient cohorts and integrates them into the reasoning pipeline. CODA introduces cohort-aware population stratification and drift-aware individual adaptation, providing a principled and effective framework for enhancing medical reasoning. Extensive experiments demonstrate that CODA consistently outperforms both general-purpose and medical-specific baselines across diverse clinical tasks, including closed-QA, open-QA, and report generation. Despite its effectiveness, CODA currently leverages structured feature representations and summary statistics for cohort construction, which may introduce additional preprocessing overhead in certain scenarios. Developing more automated and end-to-end approaches for cohort discovery and integration remains an important direction for future work.

Notations

To provide a comprehensive overview of the notations used throughout the paper, we present a summary of notations in Table 5 as a quick reference to facilitate the understanding and recall of each symbol.

NotationDescription
xxInput instance, consisting of clinical records, a question and an image.
yyThe ground truth output sequence.
xix_iThe i-th clinical records in xx.
qqThe question in xx.
I\mathcal{I}Input image of height HH, width WW, and three RGB channels.
X\mathcal{X}The textual input, including clinical records and a question.
Z(0)\mathcal{Z}^{(0)}Initial multimodal embedding, formed by visual and text embeddings.
Z(ℓ)\mathcal{Z}^{(\ell)}Output of ℓ\ell-th transformer block.
LbL_bThe number of transformer blocks in the backbone.
fI→tf_{I\to t}The function that maps the visual embeddings into the shared textual embedding space.
EI(⋅)\mathcal{E}_I(\cdot)The visual encoder that maps images to embeddings.
Et(⋅)\mathcal{E}_t(\cdot)The textual encoder that maps words to embeddings.
LIL_IThe number of visual tokens in image I\mathcal{I}, similar to LtL_t.
dtd_tThe dimensions of shared textual embedding space.
pθp_\thetaThe conditional distribution parameterized by a large language model with parameters θ\theta.
M\mathcal{M}Markov decision process (MDP).
SS, AA, PP, rr and γ\gammaState space, action space, transition dynamics, reward function and discount factor, respectively.
πϕ\pi_\phiPolicy, πϕ(a∣s)\pi_\phi(a\mid s) parameterized by ϕ\phi, which in this work corresponds to the cohort-partitioning policy.
τ\tauTrajectory generated by the policy, i.e., a sequence of states and actions.
T(ℓ)\mathcal{T}^{(\ell)}The ℓ\ell-th Transformer block.
T\mathcal{T}The learned hierarchical cohort tree.
P\mathcal{P}The set of patients.
C\mathcal{C}The set of cohorts.
hCh_{\mathcal{C}}The cohort prototypes of cohorts.
Δ(x,c)\Delta(x,c)The deviation between input xx and cohort cc.
θ\thetaThe learnable model parameters.
pτ(⋅∣πϕ)p_\tau(\cdot\mid\pi_\phi)The trajectory distribution induced by policy πϕ\pi_\phi.
R(x,τ)\mathcal{R}(x,\tau)The trajectory-level cohort discovery reward.
H(p)\mathcal{H}(p)Entropy of a distribution pp.
pπϕ(⋅∣x)p_{\pi_\phi}(\cdot\mid x)Policy distribution conditioned on input xx.
g(⋅)g(\cdot)The function that maps serialized EHR records into a vector of statistical features.
VVThe set of nodes in T\mathcal{T}, similar to EE, which is the set of edges in T\mathcal{T}.
μ(⋅)\mu(\cdot)The mean function.
σ(⋅)\sigma(\cdot)The standard deviation function.
dvd_vDepth of node vv.
UPv\mathrm{UP}_vSet of statistical features of patients Pv\mathcal{P}_v.
zvz_vBinary stopping variable for node vv.
jvj_vThe selected feature for node vv.
tvt_vThe split threshold for node vv.
dmax⁡d_{\max}The maximum depth.
ece_cCohort embedding that will be injected into the multimodal input.
Pc\mathcal{P}_cThe patients within cohort cc.
Lgen\mathcal{L}_{\mathrm{gen}}Downstream generation loss.
RvR_vThe generation-aware reward.
hph_pLearned representation of patient pp.
WℓαW^\alpha_\ellLearnable weight matrix, similar to WℓβW^\beta_\ell, WℓγW^\gamma_\ell, WℓφW^\varphi_\ell and WdW_d.

Table 5. Notations.

Related Work

Foundation Models in General Domain. The rapid development of LLMs has significantly reshaped modern AI, enabling strong general-purpose reasoning and coherent, human-like interactions. Beginning with generative pre-trained Transformers (GPT) [1, 2], it has been widely proven that scaling model size and training data leads to substantial improvements in general-purpose question answering across a broad range of tasks [42, 43, 44]. Subsequent advances further enhanced vision-language alignment and multimodal understanding through instruction tuning [45, 22, 46] and reinforcement learning from human feedback (RLHF) [47]. Recent studies explore multimodal alignment via Wasserstein barycenters [48] and fine-grained image-text alignment for zero-shot anomaly detection [49]. Beyond generative modeling, representation learning approaches, including embedding models such as E5 [50] and dense retrieval frameworks like DPR [51], have supported retrieval augmented generation (RAG) [52] by providing high-quality semantic representations. These approaches allow LLMs to incorporate external knowledge, thereby improving factual consistency and mitigating hallucinations. More recently, advanced LLMs, such as Claude [28], GPT-5 [53] and Gemini [54], have demonstrated strong performance in complicated multimodal understanding and generation tasks. Yet, the general-purpose LLMs are not readily applicable to specialized domains such as healthcare. In such high-stakes settings, effective reasoning requires not only general linguistic competence but also a deep understanding of domain-specific knowledge and complex clinical data.

Foundation Models in Medical Domain. Following the success of general-purpose LLMs, extensive efforts have been made to adapt these models to the medical domain. Early efforts focus on domain-specific pretraining, where models such as BioBERT [39], PubMedBERT [55] and ClinicalBERT [56] are trained on large-scale biomedical corpora to learn medical domain knowledge and preserve medical semantics. More recently, Med-PaLM [57] and HuatuoGPT [58] are proposed to extend LLMs to clinical applications, including medical question answering, report generation, and medical decision support. At the same time, multimodal medical LVLMs are developed to integrate heterogeneous clinical data. For example, LLaVA-Med [8] and Med-Flamingo [13] enables joint reasoning over radiology images and medical reports. Building upon these advances, subsequent models, exemplified by HealthGPT [9] and HuLu-Med [31], further scale medical corpora and task diversity to enhance the medical understanding of complex clinical tasks involving both structured and unstructured data. Despite these improvements, existing models mainly operate at the individual instance level, treating each patient independently and relying on implicit medical knowledge learned by model parameters. However, real-world clinical reasoning is inherently cohort-based, where clinicians often compare patients within similar populations and reason about deviations from the typical patterns. The lack of explicit modeling of cohort structure and patient-level derivation limits the ability of current foundation models to capture clinically meaningful heterogeneity.

Implementation Details

Datasets

The experiments are conducted on four real-world EHR benchmarks: EHR-Bench [18], DIYHealth-Bench [19], MIMIC-CXR [20], and PadChest [21]. The task description for each dataset is introduced in Table 6. EHR-Bench and DIYHealth-Bench provide raw QA pairs. For the MIMIC-CXR and PadChest datasets, we further preprocess the EHR records and radiology reports into unified QA-pair formats to ensure consistency with EHR-Bench and DIYHealth-Bench. To balance evaluation reliability and computational efficiency, datasets containing more than 2,500 samples are split by reserving 500 samples for testing and using the remaining samples for training. For smaller datasets, we adopt a 4:1 training-to-test split ratio.

DatasetTask Description
AdmissionsPredict the next admission type from EHR records (Closed-QA) and provide clinical reasoning (Open-QA).
ServicesPredict the next clinical department from EHR records (Closed-QA) and provide clinical reasoning (Open-QA).
Ingredient Events (IE)Predict the next administered medication or substance from EHR records (Closed-QA) and provide clinical reasoning (Open-QA).
Provider Order Entry (POE)Predict the next clinical order type from EHR records (Closed-QA) and provide clinical reasoning (Open-QA).
TransfersPredict the next care location from EHR records (Closed-QA) and provide clinical reasoning (Open-QA).
ObesityAssess obesity risk from self-reported EHR records (Closed-QA) and provide supporting analysis (Open-QA).
MIMIC-CXRGenerate radiology reports from EHR records and chest X-ray images.
PadChestGenerate radiology reports from a few EHR records and chest X-ray images.

Table 6. Task descriptions across datasets.

Model Implementation Details

We adopt Qwen3-8B and LLaMA3-8B as backbone models. The projector fI→tf_{I \to t} is implemented as a two-layer MLP. Both g(⋅)g(\cdot) and fcf_c are instantiated as MLPs, and fu→hf_{u \to h} is an MLP-based encoder. The maximum depth dmax⁡d_{\max} is set to 100. We set λg=0.3\lambda_g = 0.3 and λd=10−4\lambda_d = 10^{-4} across all experiments. The coefficient β\beta is set to 0.01 following prior work [59, 60], and the margin mm is set to 0.2 following prior study [61]. The assignment confidence of patient pp to cohort cc is computed based on the similarity between hph_p and hch_c.

We present the detailed configurations for the training process. The specific settings are listed in Table 7. These hyperparameters are set up following prior studies [9, 22, 8, 19]. The architectural parameters are listed in Table 8.

ConfigurationStage (i)Stage (ii)Stage (iii)
OptimizerAdamWAdamWAdamW
Adapter LR/5e-51e-4
Learning Rate2e-51e-42e-5
Weight Decay/0.010.01
Dropout Rate0.050.10.1
LR SchedulerWarmupConstantCosine w/ Warmup
Max Sequence Length32k32k32k

Table 7. Summary of configurations for the training process.

ModuleParameterValue / Shape
TokenizerVocabulary size151,670
Backbone (Qwen3-VL)Hidden size4096
Backbone (Qwen3-VL)# Transformer layers36
Backbone (Qwen3-VL)Embedding matrix(151670, 4096)
Backbone (Qwen3-VL)Self-attn projectionsq: (4096, 4096), k/v: (4096, 1024), o: (4096, 4096)
Backbone (Qwen3-VL)MLPgate/up: (4096, 12288), down: (12288, 4096)
LoRARank rr16
LoRAMatrix Aq/k/v/o/gate/up: (16, 4096), down: (16, 12288)
LoRAMatrix Bq/o: (4096,16), k/v: (1024,16), gate/up: (12288,16), down: (4096,16)
Stage iiCohort embedding dim1024
Stage iiPrefix length8
Stage iiPrefix projector(4096, 4096) →\to (4096, 8 ×\times 4096)
Stage iiCohort encoder hidden dim4096
Stage iiiDrift dim1024
Stage iiiAdapter bottleneck dim256
Stage iiiAdapter (down / up)(4096, 256), (256, 4096)
Stage iiiDrift generator(1024, 512)
Stage iiiGating module(1024, 4096)
Cohort Memory# Cohorts34
Cohort MemoryPrototype matrix(34, 1024)

Table 8. Architectural parameters of CODA.

Experimental Environment

All experiments are conducted on a server equipped with an AMD EPYC 9334 CPU (32 cores, 2.7 GHz), 128 GB RAM, and an NVIDIA H100 NVL GPU with CUDA 12.9. The system runs Ubuntu 24.04 with Linux kernel 6.8.0-79-generic.

Evaluation Metrics

Evaluation Metrics for Closed-QA

  • Accuracy. Accuracy (ACC) evaluates the overall prediction performance by computing the fraction of correctly classified instances among all samples. It provides a coarse-grained measure of classification effectiveness, especially under balanced label distributions.

ACC=TP+TNTP+TN+FP+FN(18)ACC = \frac{TP + TN}{TP + TN + FP + FN} \tag*{(18)}

where TP, TN, FP, and FN are the numbers of true positives, true negatives, false positives, and false negatives, respectively.

  • Matthews Correlation Coefficient. The Matthews Correlation Coefficient (MCC) provides a holistic evaluation of classification performance by capturing the correlation between predicted and true labels through the full confusion matrix. In contrast to accuracy, MCC remains robust under class imbalance, as it simultaneously considers all types of prediction outcomes and their interactions across classes. As a normalized correlation measure, MCC ranges from −1-1 to 11, where 11 indicates perfect alignment, 00 corresponds to chance-level prediction, and −1-1 reflects complete disagreement.

MCC=N∑kCkk−∑kTkPk(N2−∑kPk2)(N2−∑kTk2)(19)\mathrm{MCC} = \frac{N\sum_k C_{kk} - \sum_k T_kP_k}{\sqrt{(N^2-\sum_k P_k^2)(N^2-\sum_k T_k^2)}} \tag*{(19)}

where C∈RK×KC \in\mathbb{R}^{K \times K} denotes the confusion matrix, and CklC_{kl} represents the number of samples whose ground-truth class is kk and predicted class is ll. Tk=∑lCklT_k = \sum_l C_{kl} and Pk=∑lClkP_k = \sum_l C_{lk} denote the total number of true and predicted instances for class kk, respectively. N=∑k∑lCklN = \sum_k \sum_l C_{kl} is the total number of samples, and KK is the number of classes.

Evaluation Metrics for Open-QA

  • F1-BioBERT. F1-BioBERT evaluates semantic consistency between generated answers and ground-truth responses by leveraging contextual embeddings from BioBERT [39]. Instead of relying on exact lexical matching, it computes similarity at the representation level, enabling robust assessment of paraphrased or semantically equivalent expressions in medical text. Specifically, precision and recall are defined based on semantic matching between predicted and reference tokens (or spans) in the embedding space, and the final score is computed as their harmonic mean:

F1-BioBERT=2⋅Precision⋅RecallPrecision+Recall.(20)\mathrm{F1\text{-}BioBERT} = \frac{2 \cdot\mathrm{Precision} \cdot\mathrm{Recall}}{\mathrm{Precision} + \mathrm{Recall}}. \tag*{(20)}

Here, matches are determined according to similarity scores derived from BioBERT representations, with higher scores indicating stronger semantic alignment.

  • BLEU-1. BLEU-1 evaluates lexical fidelity by measuring the modified unigram precision between the generated text and the reference. Compared with higher-order BLEU variants, BLEU-1 focuses on word-level overlap and is therefore suitable for assessing whether key lexical content is preserved.

BLEU-1=BP⋅exp⁡(log⁡p1),(21)\mathrm{BLEU\text{-}1} = \mathrm{BP} \cdot\exp(\log p_1), \tag*{(21)}

where p1p_1 is the modified unigram precision:

p1=∑w∈Vmin⁡(Count⁡pred(w),Count⁡ref(w))∑w∈VCount⁡pred(w).(22)p_1 = \frac{\sum_{w \in\mathcal{V}} \min\left(\operatorname{Count}_{\mathrm{pred}}(w), \operatorname{Count}_{\mathrm{ref}}(w)\right)}{\sum_{w \in\mathcal{V}} \operatorname{Count}_{\mathrm{pred}}(w)}. \tag*{(22)}

Here, V\mathcal{V} denotes the unigram vocabulary, while Count⁡pred(w)\operatorname{Count}_{\mathrm{pred}}(w) and Count⁡ref(w)\operatorname{Count}_{\mathrm{ref}}(w) denote the frequency of unigram ww in the generated and reference texts, respectively. BP is the brevity penalty that discourages overly short generations.

Evaluation Metrics for Report Generation

  • F1-RadGraph. F1-RadGraph evaluates the quality of generated radiology reports by measuring the overlap between predicted and reference clinical graphs constructed using RadGraph [40]. Specifically, it assesses the correctness of extracted medical entities and their associated relations, thereby capturing both factual accuracy and clinical consistency at the structured level. The metric is computed as the harmonic mean of precision and recall over matched graph elements:

F1-RadGraph=2⋅Precision⋅RecallPrecision+Recall(23)\mathrm{F1\text{-}RadGraph} = \frac{2 \cdot\mathrm{Precision} \cdot\mathrm{Recall}}{\mathrm{Precision} + \mathrm{Recall}} \tag*{(23)}

where precision and recall are defined based on the matching between predicted and ground-truth RadGraph annotations, with true positives corresponding to correctly identified entities and relations, and false positives/negatives representing mismatched or missing clinical concepts.

Cohort Structure

The learned cohort tree provides an interpretable hierarchical decomposition of the patient space, where each internal node corresponds to a clinically meaningful decision criterion, and each leaf node defines a distinct cohort, as shown in Figure 6. At the top levels, the model prioritizes global clinical signals such as cardiac-related markers, care context (e.g., discharge disposition and observation status), and multimorbidity, indicating that population stratification is first driven by overall severity and healthcare utilization. As the hierarchy progresses, the tree further branches along complementary axes, including chronic disease patterns (e.g., cardiometabolic conditions) and imaging-derived phenotypes (e.g., pleural effusion and pneumothorax), demonstrating that the model disentangles heterogeneous patient populations using semantically coherent, multimodal criteria. At deeper levels, the structure shifts toward fine-grained refinement, where demographic attributes (e.g., age and gender) and laboratory-derived signals (e.g., hematologic and metabolic abnormalities) are used to differentiate patients with subtle physiological variations. Notably, certain features recur across different branches, reflecting context-dependent decision boundaries rather than fixed global rules. In addition, the tree exhibits adaptive depth, with simpler cohorts identified early and more complex populations refined through deeper branches.

Figure 6. Illustration of Cohort Structure.

Overall, this hierarchy reveals that the proposed method learns clinically grounded and interpretable cohort structures, organizing patients into meaningful subgroups that support coherent and reliable downstream reasoning.

Supplementary Experimental Results

Supplementary Ablation Study

Tables 9 and 10 provide extended ablation results across closed-QA, open-QA, and report generation settings. Consistent with the main paper, removing either cohort-aware population stratification or drift-aware individual adaptation leads to performance degradation across both in-distribution and out-of-distribution tasks, demonstrating that the two components are complementary and jointly contribute to robust multimodal clinical reasoning.

Under closed-QA settings, CODA achieves the best overall performance across all tasks and evaluation metrics. Similar trends are observed in open-QA and report generation settings. CODA consistently achieves the strongest performance across both language-level and clinical evaluation metrics. Overall, the results demonstrate that population-level cohort structure and patient-specific deviation modeling are both critical for reliable multimodal clinical reasoning.

ModelAdmissions ACCAdmissions MCCIE ACCIE MCCPOE ACCPOE MCCServices ACCServices MCCTransfers ACCTransfers MCCObesity ACCObesity MCCAvg. ACCAvg. MCC
LoRA Only74.6065.7891.9289.0481.2074.8392.0289.4273.6064.6570.3058.4880.6173.70
w/o Cohort75.2066.5792.1789.5782.2076.2592.7790.3874.2065.1670.9259.2181.2474.52
w/o Deviation75.8067.3792.1789.5182.0076.0092.2790.0674.4065.7071.7060.2081.3974.81
CODA76.6068.7792.4289.9382.8077.0593.5291.3975.2066.8972.3261.2882.1475.88

Table 9. Ablation Study of CODA under Closed-QA settings.

ModelAdmissions F1-BioAdmissions BLEU-1IE F1-BioIE BLEU-1POE F1-BioPOE BLEU-1Services F1-BioServices BLEU-1MIMIC-CXR F1-RadMIMIC-CXR F1-BioTransfers F1-BioTransfers BLEU-1Obesity F1-BioObesity BLEU-1PadChest F1-RadPadChest F1-BioOpen-QA F1-BioOpen-QA BLEU-1Report F1-RadReport F1-Bio
LoRA Only79.0646.0980.9447.2080.1650.2778.8746.2415.4678.7478.6144.1874.6113.4424.6872.2478.7141.2420.0775.49
w/o Cohort79.7148.0281.1049.0880.4452.0379.2047.6015.7378.5878.5843.6675.0714.0124.6072.5179.0242.4020.1775.55
w/o Deviation79.8048.6481.1848.9980.0650.5079.3447.3216.3378.9478.5543.6874.7613.9625.1972.4178.9542.1820.7675.67
CODA80.8252.4582.2151.6981.0251.9780.2149.1917.4979.0379.5746.4275.3715.6525.5273.9379.8744.5621.5176.48

Table 10. Ablation Study of CODA under Open-QA and report generation settings.

Supplementary Comparisons with Fine-tuned Baselines

Tables 11 and 12 provide the full comparisons between CODA and representative LoRA fine-tuned multimodal medical foundation models under closed-QA, open-QA, and report generation settings. CODA consistently achieves the best overall performance across both in-distribution and out-of-distribution tasks.

ModelAdmissions ACCAdmissions MCCIE ACCIE MCCPOE ACCPOE MCCServices ACCServices MCCTransfers ACCTransfers MCCObesity ACCObesity MCCAvg. ACCAvg. MCC
InternVL-3-8B76.0067.9891.1888.5181.2074.9393.2791.0274.6065.7571.2359.5281.2574.62
Lingshu-7B76.2068.2491.6788.8581.2074.9693.2791.0774.4065.5271.3860.2481.3574.81
CODA-Qwen3-8B76.6068.7792.4289.9382.8077.0593.5291.3975.2066.8972.3261.2882.1475.88

Table 11. Comparison of CODA with LoRA fine-tuned baselines under Closed-QA settings.

ModelAdmissions F1-BioAdmissions BLEU-1IE F1-BioIE BLEU-1POE F1-BioPOE BLEU-1Services F1-BioServices BLEU-1MIMIC-CXR F1-RadMIMIC-CXR F1-BioTransfers F1-BioTransfers BLEU-1Obesity F1-BioObesity BLEU-1PadChest F1-RadPadChest F1-BioOpen-QA F1-BioOpen-QA BLEU-1Report F1-RadReport F1-Bio
InternVL-3-8B80.7352.1081.5450.2280.9151.5180.1849.3916.5978.6979.1446.0672.1711.2123.8572.0579.1143.7420.2275.37
Lingshu-7B80.6050.0881.5650.2680.9951.6580.0149.0316.8978.8879.8246.4272.7211.0223.5871.9079.4444.3419.7475.79
CODA80.8252.4582.2151.6981.0251.9780.2149.1917.4979.0379.5746.4275.3715.6525.5273.9379.8744.5621.5176.48

Table 12. Comparison of CODA with LoRA fine-tuned baselines under Open-QA and report generation settings.

Comparison with Few-shot Baselines

To further evaluate CODA against strong baselines, we augment several representative models with few-shot demonstrations. The results are reported in Table 13 and Table 14. Due to the rich and heterogeneous nature of EHR data, each sample is relatively long, posing challenges for models with limited context length constraints. While InternVL 3 and Hulu-Med benefit from few-shot demonstrations, LLaVA 1.5 and Lingshu exhibit performance degradation, suggesting insufficient capability to effectively process long-context inputs. In particular, the extreme degradation observed in LLaVA 1.5 under the 2-shot setting can also be attributed to its 4K context window, which is insufficient to accommodate both demonstrations and the query. Overall, these results further highlight the effectiveness of CODA.

ModelIDOODAvg.
AdmissionsIEPOEServicesTransfersObesity
ACCMCCACCMCCACCMCCACCMCCACCMCCACCMCCACCMCC
LLaVA 1.5-7B 1-shot35.6017.7320.200.9441.8026.7055.3641.8354.6039.3358.7944.2744.3928.46
LLaVA 1.5-7B 2-shot10.605.688.595.7214.2011.5918.2016.1846.2034.5749.6135.9924.5718.29
InternVL 3-8B 1-shot38.0017.6429.045.5545.4027.3655.3640.5338.2017.9653.6537.4043.2824.41
InternVL 3-8B 2-shot42.6023.7430.306.8243.4024.7256.8642.4642.6023.9253.5036.9444.8826.43
lingshu-7B 1-shot38.8019.0134.8514.6249.8033.2569.0858.8349.2033.0865.0151.8251.1235.10
lingshu-7B 2-shot45.6027.9938.1317.7148.4031.2670.3260.5751.0034.9768.5856.0553.6738.09
Hulu-Med-7B 1-shot43.4024.7830.817.8657.6043.6665.8454.5349.2032.2860.1947.3451.1735.07
Hulu-Med-7B 2-shot49.6033.0234.0912.3359.6046.3168.5858.5949.4032.6063.6150.7854.1538.94
CODA-Qwen3-8B76.6068.7792.4289.9382.8077.0593.5291.3975.2066.8972.3261.2882.1475.88

Table 13. Comparison of CODA with few-shot baselines under Closed-QA settings.

ModelAdmissions F1-BioAdmissions BLEU-1IE F1-BioIE BLEU-1POE F1-BioPOE BLEU-1Services F1-BioServices BLEU-1MIMIC-CXR F1-RadMIMIC-CXR F1-BioTransfers F1-BioTransfers BLEU-1Obesity F1-BioObesity BLEU-1PadChest F1-RadPadChest F1-BioOpen-QA F1-BioOpen-QA BLEU-1Report F1-RadReport F1-Bio
LLaVA 1.5-7B 1-shot72.4223.9168.4517.8771.8422.5272.9322.994.1768.1976.0728.9673.1111.489.2567.8372.4721.296.7168.01
LLaVA 1.5-7B 2-shot51.986.1450.4913.7871.8422.5272.9322.994.1768.1976.0728.9673.1111.489.2567.8372.4721.296.7168.01
InternVL 3-8B 1-shot74.4027.9475.7028.5074.8928.0674.9228.329.1867.7775.4430.3269.467.048.6363.2174.1325.038.9165.49
InternVL 3-8B 2-shot75.4830.0076.2430.1476.6930.3475.6130.279.3268.2576.4131.9669.456.988.3163.7674.7626.628.8266.01
lingshu-7B 1-shot77.0938.6175.6533.6074.5633.6476.7634.1111.9875.1977.9732.1773.0410.677.8567.3676.0429.919.9271.27
lingshu-7B 2-shot77.5539.2776.1830.5776.7434.7977.2134.8911.5374.5877.2533.6372.8810.547.6766.7575.9030.619.6070.55
Hulu-Med-7B 1-shot76.9331.1675.4224.0876.1030.4076.5629.8813.8675.7776.9126.7872.8710.588.8664.9075.8025.3211.3670.34
Hulu-Med-7B 2-shot77.0432.1975.8323.0776.2530.0676.1629.0414.5073.0277.9129.9572.8515.368.2965.5275.9625.7811.4070.92
CODA-Qwen3-8B80.0252.4582.2351.9781.0250.9780.2149.1917.4979.0379.9142.4578.5716.5525.5275.9980.7744.7821.5174.92

Table 14. Comparison of CODA with few-shot baselines under Open-QA and report generation settings.

Hyperparameter Study

We conduct a sensitivity study on two key hyperparameters, λg\lambda_g and λd\lambda_d, with results shown in Figure 7. As observed in Figure 7a–Figure 7c), performance consistently peaks at moderate values of λg\lambda_g (around 0.3–0.5), highlighting the importance of balancing the generation objective and cohort-aware signals. Both smaller and larger values lead to performance degradation, corresponding to the under- and over-dominance of the generation objective, respectively.

Hyperparameter study of $\lambda_g$ under Closed-QA setting

(a) Hyperparameter study of λg\lambda_g under Closed-QA setting.

Hyperparameter study of $\lambda_g$ under Open-QA setting

(b) Hyperparameter study of λg\lambda_g under Open-QA setting.

Hyperparameter study of $\lambda_g$ under Report Generation setting

(c) Hyperparameter study of λg\lambda_g under Report Generation setting.

Hyperparameter study of $\lambda_d$ under Closed-QA setting

(d) Hyperparameter study of λd\lambda_d under Closed-QA setting.

Hyperparameter study of $\lambda_d$ under Open-QA setting

(e) Hyperparameter study of λd\lambda_d under Open-QA setting.

Hyperparameter study of $\lambda_d$ under Report Generation setting

(f) Hyperparameter study of λd\lambda_d under Report Generation setting.

Figure 7. Hyperparameter study on λg\lambda_g and λd\lambda_d.

In contrast, Figure 7d–Figure 7f) shows that the model is relatively insensitive to λd\lambda_d, with stable performance across several orders of magnitude. The best results are typically achieved around 10−410^{-4} or 10−110^{-1}, suggesting that the drift loss primarily acts as a regularizer.

Overall, these results demonstrate that our method achieves robust performance without requiring precise hyperparameter tuning.

Case Study

Supplementary Experimental Results

Case Study

Figure 8. A case study of Transfers task under Closed-QA setting.

Figure 9. A case study of PadChest dataset under Report Generation setting.

Broader Impacts

Current medical foundation models overlook patient-specific characteristics, which are critical for accurate clinical decision-making. By explicitly modeling cohort structure and individual-level deviations, CODA has the potential to support clinicians in delivering more personalized care, particularly in settings with heterogeneous patient populations, and to enhance transparency by exposing cohort-level patterns and patient-specific factors that influence model predictions.

At the same time, several risks should be considered. First, cohort construction may inadvertently encode biases present in the underlying data, potentially leading to disparate performance across demographic groups. Second, reliance on structured features and summary statistics may limit applicability in scenarios where such information is incomplete or noisy. Third, as with other generative models, there remains a risk of producing plausible but incorrect outputs, which could be misinterpreted in high-stakes clinical contexts.

To mitigate these risks, future work should incorporate fairness-aware cohort construction, uncertainty estimation, and human-in-the-loop validation to ensure safe deployment. In addition, rigorous evaluation across diverse populations and clinical settings is necessary to assess generalizability and prevent unintended harms. Overall, while CODA offers a step toward more interpretable and patient-aware medical AI, careful consideration of these societal implications is essential for responsible use.

References

  1. [1]Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. Language models are few-shot learners. Advances in Neural Information Processing Systems, 33:1877–1901, 2020.arxiv.org/abs/2005.14165
  2. [2]OpenAI. Gpt-4 technical report, 2023.arxiv.org/abs/2303.08774
  3. [3]Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. In International conference on machine learning, pages 8748–8763. PmLR, 2021.arxiv.org/abs/2103.00020
  4. [4]Gang Chen, Changshuo Liu, Gene Anne Ooi, Marcus Tan, Zhongle Xie, Jianwei Yin, James Wei Luen Yip, Wenqiao Zhang, Jiaqi Zhu, and Beng Chin Ooi. Generative ai for healthcare: Fundamentals, challenges, and perspectives. arXiv preprint arXiv:2510.24551, 2025.arxiv.org/abs/2510.24551
  5. [5]Jiarui Ye and Hao Tang. Multimodal large language models for medicine: A comprehensive survey. arXiv preprint arXiv:2504.21051, 2025.arxiv.org/abs/2504.21051
  6. [6]Béria Chingnabé Kalpélbé, Angel Gabriel Adaambiik, and Wei Peng. Vision language models in medicine. arXiv preprint arXiv:2503.01863, 2025.
  7. [7]Jiaqi Zhu, Naili Xing, Hexiang Pan, Haotian Gao, Jianwei Yin, Xiaokui Xiao, and Beng Chin Ooi. Logictree-rag: Logic tree-guided retrieval-augmented generation for long-form patent drafting. arXiv preprint arXiv:2609.30943, 2026.arxiv.org/abs/2609.30943
  8. [8]Chunyuan Li, Cliff Wong, Sheng Zhang, Naoto Usuyama, Haotian Liu, Jianwei Yang, Tristan Naumann, Hoifung Poon, and Jianfeng Gao. Llava-med: Training a large language-and-vision assistant for biomedicine in one day. Advances in Neural Information Processing Systems, 36:28541–28564, 2023.
  9. [9]Tianwei Lin, Wenqiao Zhang, Sijing Li, Yuqian Yuan, Binhe Yu, Haoyuan Li, Wanggui He, Hao Jiang, Mengze Li, Xiaohui Song, et al. HealthGPT: A medical large vision-language model for unifying comprehension and generation via heterogeneous knowledge adaptation. In Proceedings of Forty-second International Conference on Machine Learning (ICML), 2025.arxiv.org/abs/2502.09838
  10. [10]Jianing Qiu, Kyle Lam, Guohao Li, Amish Acharya, Tien Yin Wong, Ara Darzi, Wu Yuan, and Eric J Topol. Llm-based agentic systems in medicine and healthcare. Nature Machine Intelligence, 6(12):1418–1420, 2024.DOI
  11. [11]Karan Singhal, Tao Tu, Juraj Gottweis, Rory Sayres, Ellery Wulczyn, Mohamed Amin, Le Hou, Kevin Clark, Stephen R Pfohl, Heather Cole-Lewis, et al. Toward expert-level medical question answering with large language models. Nature medicine, 31(3):943–950, 2025.DOI
  12. [12]Kai Zhang, Rong Zhou, Eashan Adhikarla, Zhiling Yan, Yixin Liu, Jun Yu, Zhengliang Liu, Xun Chen, Brian D Davidson, Hui Ren, et al. A generalist vision–language foundation model for diverse biomedical tasks. Nature medicine, 30(11):3129–3141, 2024.
  13. [13]Michael Moor, Qian Huang, Shirley Wu, Michihiro Yasunaga, Yash Dalmia, Jure Leskovec, Cyril Zakka, Eduardo Pontes Reis, and Pranav Rajpurkar. Med-flamingo: a multimodal medical few-shot learner. In Machine Learning for Health (ML4H), pages 353–367. PMLR, 2023.arxiv.org/abs/2307.15189
  14. [14]Junying Chen, Chi Gui, Ruyi Ouyang, Anningzhe Gao, Shunian Chen, Guiming Hardy Chen, Xidong Wang, Ruifei Zhang, Zhenyang Cai, Ke Ji, et al. Huatuogpt-vision, towards injecting medical visual knowledge into multimodal llms at scale. arXiv preprint arXiv:2406.19280, 2024.arxiv.org/abs/2406.19280
  15. [15]Andrew Sellergren, Sahar Kazemzadeh, Tiam Jaroensri, Atilla Kiraly, Madeleine Traverse, Timo Kohlberger, Shawn Xu, Fayaz Jamil, Cían Hughes, Charles Lau, et al. Medgemma technical report. arXiv preprint arXiv:2507.05201, 2025.arxiv.org/abs/2507.05201
  16. [16]Eta S Berner. Clinical decision support systems, volume 233. Springer, 2007.DOI
  17. [17]Xiaofeng Wang and Michael W Kattan. Cohort studies: design, analysis, and reporting. Chest, 158(1):S72–S78, 2020.DOI
  18. [18]Yusheng Liao, Chaoyi Wu, Junwei Liu, Shuyang Jiang, Pengcheng Qiu, Haowen Wang, Yun Yue, Shuai Zhen, Jian Wang, Qianrui Fan, et al. Ehr-1: A reasoning-enhanced foundational language model for electronic health record analysis. arXiv preprint arXiv:2510.25628, 2025.arxiv.org/abs/2510.25628
  19. [19]Changshuo Liu, Wu Junran, Zhongle Xie, Wenqiao Zhang, Kaiping Zheng, Jiaqi Zhu, Qingpeng Cai, Ooi Gene Anne, Marcus Chun Jin Tan, Jianwei Yin, et al. Diyhealth suite: Dataset, model, and benchmark for health management at home.
  20. [20]Alistair EW Johnson, Tom J Pollard, Seth J Berkowitz, Nathaniel R Greenbaum, Matthew P Lungren, Chih-ying Deng, Roger G Mark, and Steven Horng. Mimic-cxr, a de-identified publicly available database of chest radiographs with free-text reports. Scientific data, 6(1):317, 2019.DOI
  21. [21]Aurelia Bustos, Antonio Pertusa, Jose-Maria Salinas, and Maria De La Iglesia-Vaya. Padchest: A large chest x-ray image dataset with multi-label annotated reports. Medical image analysis, 66:101797, 2020.DOI
  22. [22]Haotian Liu, Chunyuan Li, Qingyang Wu, and Yong Jae Lee. Visual instruction tuning. Advances in Neural Information Processing Systems, 36:34892–34916, 2023.
  23. [23]Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Amy Yang, Angela Fan, et al. The llama 3 herd of models. arXiv e-prints, pages arXiv–2407, 2024.arxiv.org/abs/2407.21783
  24. [24]Jinguo Zhu, Weiyun Wang, Zhe Chen, Zhaoyang Liu, Shenglong Ye, Lixin Gu, Hao Tian, Yuchen Duan, Weijie Su, Jie Shao, et al. Internvl3: Exploring advanced training and test-time recipes for open-source multimodal models. arXiv preprint arXiv:2504.10479, 2025.arxiv.org/abs/2504.10479
  25. [25]An Yang, Anfeng Li, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chang Gao, Chenggen Huang, Chenxu Lv, et al. Qwen3 technical report. arXiv preprint arXiv:2505.09388, 2025.arxiv.org/abs/2505.09388
  26. [26]Gemma Team, Aishwarya Kamath, Johan Ferret, Shreya Pathak, Nino Vieillard, Ramona Merhej, Sarah Perrin, Tatiana Matejovicova, Alexandre Ramé, Morgane Rivière, et al. Gemma 3 technical report. arXiv preprint arXiv:2503.19786, 2025.
  27. [27]xAI. Grok 4 technical overview. https://x.ai/blog, 2025. Technical report.
  28. [28]AI Anthropic. The claude 3 model family: Opus, sonnet, haiku. Claude-3 Model Card, 1(1):4, 2024.
  29. [29]Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, et al. Gpt-4 technical report. arXiv preprint arXiv:2303.08774, 2023.arxiv.org/abs/2303.08774
  30. [30]Weiwen Xu, Hou Pong Chan, Long Li, Mahani Aljunied, Ruifeng Yuan, Jianyu Wang, Chenghao Xiao, Guizhen Chen, Chaoqun Liu, Zhaodonghui Li, et al. Lingshu: A generalist foundation model for unified multimodal medical understanding and reasoning. arXiv preprint arXiv:2506.07044, 2025.arxiv.org/abs/2506.07044
  31. [31]Songtao Jiang, Yuan Wang, Sibo Song, Tianxiang Hu, Chenyi Zhou, Bin Pu, Yan Zhang, Zhibo Yang, Yang Feng, Joey Tianyi Zhou, et al. Hulu-med: A transparent generalist model towards holistic medical vision-language understanding. arXiv preprint arXiv:2510.08668, 2025.arxiv.org/abs/2510.08668
  32. [32]Asma Ben Abacha and Dina Demner-Fushman. A question-entailment approach to question answering. BMC bioinformatics, 20(1):511, 2019.
  33. [33]Junying Chen, Chi Gui, Ruyi Ouyang, Anningzhe Gao, Shunian Chen, Guiming Chen, Xidong Wang, Zhenyang Cai, Ke Ji, Xiang Wan, et al. Towards injecting medical visual knowledge into multimodal llms at scale. In Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, pages 7346–7370, 2024.DOI
  34. [34]Bo Liu, Li-Ming Zhan, Li Xu, Lin Ma, Yan Yang, and Xiao-Ming Wu. Slake: A semantically-labeled knowledge-enhanced dataset for medical visual question answering. In 2021 IEEE 18th international symposium on biomedical imaging (ISBI), pages 1650–1654. IEEE, 2021.arxiv.org/abs/2102.09542
  35. [35]Jason J Lau, Soumya Gayen, Asma Ben Abacha, and Dina Demner-Fushman. A dataset of clinically generated visual questions and answers about radiology images. Scientific data, 5(1):180251, 2018.DOI
  36. [36]Brian W Matthews. Comparison of the predicted and observed secondary structure of t4 phage lysozyme. Biochimica et Biophysica Acta (BBA)-Protein Structure, 405(2):442–451, 1975.pubmed.ncbi.nlm.nih.gov/1180967
  37. [37]Kishore Papineni, Salim Roukos, Todd Ward, and Wei-Jing Zhu. Bleu: a method for automatic evaluation of machine translation. In Proceedings of the 40th annual meeting of the Association for Computational Linguistics, pages 311–318, 2002.
  38. [38]Tianyi Zhang, Varsha Kishore, Felix Wu, Kilian Q Weinberger, and Yoav Artzi. Bertscore: Evaluating text generation with bert. In International Conference on Learning Representations.
  39. [39]Jinhyuk Lee, Wonjin Yoon, Sungdong Kim, Donghyeon Kim, Sunkyу Kim, Chan Ho So, and Jaewoo Kang. Biobert: a pre-trained biomedical language representation model for biomedical text mining. Bioinformatics, 36(4):1234–1240, 2020.DOI
  40. [40]Saahil Jain, Ashwin Agrawal, Adriel Saporta, Steven QH Truong, Du Nguyen Duong, Tan Bui, Pierre Chambon, Yuhao Zhang, Matthew P Lungren, Andrew Y Ng, et al. Radgraph: Extracting clinical entities and relations from radiology reports. arXiv preprint arXiv:2106.14463, 2021.arxiv.org/abs/2106.14463
  41. [41]Laurens van der Maaten and Geoffrey Hinton. Visualizing data using t-sne. Journal of machine learning research, 9(Nov):2579–2605, 2008.
  42. [42]Renqian Luo, Liai Sun, Yingce Xia, Tao Qin, Sheng Zhang, Hoifung Poon, and Tie-Yan Liu. Biogpt: generative pre-trained transformer for biomedical text generation and mining. Briefings in bioinformatics, 23(6):bbac409, 2022.DOI
  43. [43]Yizhen Luo, Jiahпuan Zhang, Siqi Fan, Kai Yang, Massimo Hong, Yushuai Wu, Mu Qiao, and Zaiqing Nie. Biomedgpt: An open multimodal large language model for biomedicine. IEEE Journal of Biomedical and Health Informatics, 2024.DOI
  44. [44]Hongyang Yang, Xiao-Yang Liu, and Christina Dan Wang. Fingpt: Open-source financial large language models. arXiv preprint arXiv:2306.06031, 2023.arxiv.org/abs/2306.06031
  45. [45]Jason Wei, Maarten Bosma, Vincent Y Zhao, Kelvin Guo, Adams Wei Yu, Brian Lester, Nan Du, Andrew M Dai, and Quoc V Le. Finetuned language models are zero-shot learners. arXiv preprint arXiv:2109.01652, 2021.arxiv.org/abs/2109.01652
  46. [46]Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Fei Xia, Ed Chi, Quoc V Le, Denny Zhou, et al. Chain-of-thought prompting elicits reasoning in large language models. Advances in neural information processing systems, 35:24824–24837, 2022.arxiv.org/abs/2201.11903
  47. [47]Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, et al. Training language models to follow instructions with human feedback. Advances in neural information processing systems, 35:27730–27744, 2022.arxiv.org/abs/2203.02155
  48. [48]Xiaole Tang, Jiayi Xu, Xiang Gu, Yan Yang, and Jian Sun. Binding multiple modalities via multimodal wasserstein barycenter. arXiv preprint arXiv:2609.33800, 2026.arxiv.org/abs/2609.33800
  49. [49]Jiaqi Zhu, Shaofeng Cai, Fang Deng, Beng Chin Ooi, and Junran Wu. Do LLMs understand visual anomalies? uncovering LLM’s capabilities in zero-shot anomaly detection. In Proceedings of the 32nd ACM International Conference on Multimedia, pages 48–57, 2024.arxiv.org/abs/2404.09654
  50. [50]Liang Wang, Nan Yang, Xiaolong Huang, Binxing Jiao, Linjun Yang, Daxin Jiang, Rangan Majumder, and Furu Wei. Text embeddings by weakly-supervised contrastive pre-training. arXiv preprint arXiv:2212.03533, 2022.arxiv.org/abs/2212.03533
  51. [51]Vladimir Karpukhin, Barlas Oguz, Sewon Min, Patrick Lewis, Ledell Wu, Sergey Edunov, Danqi Chen, and Wen-tau Yih. Dense passage retrieval for open-domain question answering. In Proceedings of the 2020 conference on empirical methods in natural language processing (EMNLP), pages 6769–6781, 2020.arxiv.org/abs/2004.04906
  52. [52]Patrick Lewis, Ethan Perez, Aleksandra Piktus, Fabio Petroni, Vladimir Karpukhin, Naman Goyal, Heinrich Küttler, Mike Lewis, Wen-tau Yih, Tim Rocktäschel, et al. Retrieval-augmented generation for knowledge-intensive nlp tasks. Advances in neural information processing systems, 33:9459–9474, 2020.arxiv.org/abs/2005.11401
  53. [53]Aaditya Singh, Adam Fry, Adam Perelman, Adam Tart, Adi Ganesh, Ahmed El-Kishky, Aidan McLaughlin, Aiden Low, AJ Ostrow, Akhila Ananthram, et al. Openai gpt-5 system card. arXiv preprint arXiv:2601.03267, 2025.arxiv.org/abs/2601.03267
  54. [54]Gemini Team, Rohan Anil, Sebastian Borgeaud, Jean-Baptiste Alayrac, Jiahui Yu, Radu Soricut, Johan Schalkwyk, Andrew M Dai, Anja Hauth, Katie Millican, et al. Gemini: a family of highly capable multimodal models. arXiv preprint arXiv:2312.11805, 2023.
  55. [55]Yu Gu, Robert Tinn, Hao Cheng, Michael Lucas, Naoto Usuyama, Xiaodong Liu, Tristan Naumann, Jianfeng Gao, and Hoifung Poon. Domain-specific language model pretraining for biomedical natural language processing. ACM Transactions on Computing for Healthcare (HEALTH), 3(1):1–23, 2021.DOI
  56. [56]Kexin Huang, Jaan Altosaar, and Rajesh Ranganath. Clinicalbert: Modeling clinical notes and predicting hospital readmission. arXiv preprint arXiv:1904.05342, 2019.arxiv.org/abs/1904.05342
  57. [57]Karan Singhal, Shekoofeh Azizi, Tao Tu, S Sara Mahdavi, Jason Wei, Hyung Won Chung, Nathan Scales, Ajay Tanwani, Heather Cole-Lewis, Stephen Pfohl, et al. Large language models encode clinical knowledge. Nature, 620(7972):172–180, 2023.DOI
  58. [58]Hongbo Zhang, Junying Chen, Feng Jiang, Fei Yu, Zhihong Chen, Guiming Chen, Jianquan Li, Xiangbo Wu, Zhang Zhiyi, Qingying Xiao, et al. Huatuogpt, towards taming language model to be a doctor. In Findings of the association for computational linguistics: EMNLP 2023, pages 10859–10855, 2023.arxiv.org/abs/2305.15075
  59. [59]Jean-Bastien Grill, Florian Strub, Florent Altché, Corentin Tallec, Pierre Richemond, Elena Buchatskaya, Carl Doersch, Bernardo Avila Pires, Zhaohan Guo, Mohammad Gheshlaghi Azar, et al. Bootstrap your own latent—a new approach to self-supervised learning. Advances in neural information processing systems, 33:21271–21284, 2020.arxiv.org/abs/2006.07733
  60. [60]Antti Tarvainen and Harri Valpola. Mean teachers are better role models: Weight-averaged consistency targets improve semi-supervised deep learning results. Advances in neural information processing systems, 30, 2017.arxiv.org/abs/1703.01780
  61. [61]Florian Schroff, Dmitry Kalenichenko, and James Philbin. Facenet: A unified embedding for face recognition and clustering. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 815–823, 2015.DOI

Paper details

vlnc:W_5bakh5h7
Version
v1 · latest
Published
Fresh — added to the Hub within the last 30 days
Added
License
CC BY 4.0
Usage

views · 0

downloads · 0

A view counts once per browser per day, a download each time the PDF is requested.

  • Computer Science

Not recorded for this paper

View statement
Looking for related papers…
Contents