WorksComputer Science
SMILE: A Structured Multi-Agent Clinical Reasoning for Liver Ultrasound Understanding
Abstract
Automated liver ultrasound report generation improves the diagnosis of liver diseases by producing clinically structured descriptions of anatomical and disease-related findings. However, current generative vision-language models (VLMs) often rely on a single reasoning pathway that maps ultrasound appearance directly to textual descriptions. In addition to the inherent vision-language modality gap, this design is limited for liver ultrasound because disease interpretation is constrained by subtle visual evidence, heterogeneous disease patterns, and limited paired supervision. To address these limitations, we propose SMILE, a Structured Multi-agent clInicaL rEasoning framework that uses multiple complementary specialist agents to interpret ultrasound images and consolidates their outputs through a judge-guided consensus mechanism. Our approach employs three specialized agents: an image interpretation agent for morphological descriptions, a clinical-stage reasoning agent, and a visual question answering (VQA) agent for targeted queries. These agent responses are then aggregated and scrutinized by a judge VLM using a consensus filtering prompt to produce a unified, clinically coherent output. Across a public liver ultrasound dataset and an internal clinical dataset, SMILE consistently outperforms state-of-the-art VLMs, achieving up to 19.0% improvement in ROUGE-1 and 18.8% improvement in clinical F1-score, while reaching VQA F1-scores of 0.82 and 0.65 on public and internal datasets, respectively. These results demonstrate the potential of SMILE for data-scarce and limited-supervision clinical settings. Our models, datasets, and code are available \href{https://github.com/CMME-Lab/SMILE}{here}.
Introduction
Ultrasound imaging plays a central role in the evaluation of liver diseases due to its accessibility, real-time feedback, cost-effectiveness, and non-invasive assessment [13]. However, reliable diagnosis of liver diseases remains challenging because liver pathology presents heterogeneous and often ill-defined sonographic patterns, and ultrasound images are affected by data-acquisition artifacts such as speckle noise and acoustic shadowing [11, 32]. As a representative example, staging liver fibrosis from conventional ultrasound is particularly difficult because early fibrotic changes may not produce overt structural abnormalities [4, 40].
Recent advances in artificial intelligence (AI) have enabled automated analysis of ultrasound images [16, 20]. In particular, large language models (LLMs) have recently been explored for medical image understanding due to their strong capabilities in semantic abstraction and structured reasoning [14, 27, 41]. Unlike purely discriminative vision models, the integration of LLMs with vision systems can externalize intermediate reasoning steps through natural language, providing traceable and clinically interpretable explanations [6]. However, in liver ultrasound analysis, the primary challenge extends beyond text generation itself. The visual evidence provided by ultrasound is often under-constrained, allowing multiple clinically plausible interpretations to coexist for a single image. Conventional fine-tuning of single-agent LLMs implicitly assumes that a solitary reasoning trajectory is sufficient to resolve this ambiguity [18]. Under limited supervision and small-data regimes, this assumption leads to unstable reasoning behavior, loss of context awareness, and hallucinated findings that are weakly grounded in the image [1, 36, 44].
To improve the reasoning capabilities of LLMs, several studies have explored collaboration among multiple large language models for complex medical reasoning tasks [7, 17, 31, 33, 43]. These methods improve the reasoning of a LLM through role-playing, where each agent assumes a distinct expert perspective and proposes diverse plans that capture different aspects of the clinical problem. However, the benefit of such reasoning models depends strongly on the design of the collaboration mechanism among LLMs [33]. Moreover, without explicit constraints that enforce complementary reasoning, multi-agent systems may generate homogeneous, low-quality responses and induce misconceptions [5, 12]. These challenges are further highlighted in clinical imaging tasks, where limited access to independent and modality-specific evidence can reduce agent diversity and amplify unsupported clinical interpretations.
To address these limitations, we propose SMILE, a structured multi-agent clinical reasoning framework for liver ultrasound image analysis. Rather than depending on a single reasoning pathway, our approach decomposes the diagnostic process into complementary agents: (1) an image interpretation agent that converts visual features into structured morphological descriptions, (2) a clinical-stage reasoning agent guided by pathology-aligned supervision, and (3) a visual question answering (VQA) agent that improves diagnostic performance through targeted queries. These agentic responses are then evaluated by a dedicated LLM, referred to as the judge, that improves the consistency and clinical plausibility of the agents’ responses and synthesizes them into a unified, diagnostically coherent output (Figure 1). Unlike existing multi-LLM medical reasoning frameworks, which operate on a specific clinical modality and require resource-intensive post-training, we only employ lightweight agents that can be adapted through simple fine-tuning with limited labeled datasets, making the framework suitable for data-scarce scenarios.

Figure 1. Comparison with existing VLMs. Top (Conventional VLM): Utilizing a single VLM, pre-trained on a single source of clinical information, which generates responses with hallucinated findings. Bottom (Our Method): SMILE uses complementary specialist agents adapted with radiology-aligned and pathology-aligned supervision to extract complementary clinical evidence from liver ultrasound images. Red text indicates hallucinated findings, and green text indicates findings corrected by SMILE.
Our contributions can be summarized as follows: 1) We formulate liver ultrasound understanding as an under-constrained clinical reasoning problem; 2) We propose a structured multi-agent framework that stabilizes reasoning through cross-agent consistency; 3) We demonstrate improved robustness and interpretability by assessing medical image understanding under limited data conditions.
Related Work
Vision Language Models in Medical Report Generation
Medical report generation has evolved from task-specific image-to-text generation toward generative vision-language models (VLMs) with broader clinical reasoning ability. Early methods such as R2Gen used memory-driven Transformer architectures to generate long clinical reports directly from medical images [8]. Subsequent report generation methods addressed the cross-modal semantic gap between visual findings and clinical language, aiming to reduce inconsistent or visually unsupported descriptions [23, 37, 39]. With the rise of large language models in natural text generation, recent VLMs have moved beyond task-specific decoders by connecting visual encoders with pretrained language models for open-ended multi-modal generation. Flamingo and Med-Flamingo introduced few-shot multi-modal learning through visual resampling and gated cross-attention [2, 29]. LLaVA-Med further introduced biomedical visual instruction tuning by adapting a general vision-language assistant with biomedical captions and instruction-following data [21]. Recent state-of-the-art VLMs such as Qwen2-VL improve image understanding through dynamic resolution processing and multi-modal positional encoding, while medical VLMs such as MedGemma introduce medical-specific visual encoders to further improve the domain adaptation [30, 35]. Despite the remarkable performance on task-specific benchmarks, these VLMs focus on a single reasoning pathway, which limits the clinical correctness.
Multi-Agent Vision Language Models in Medical Report Generation
Researchers have explored several strategies to combine the strengths of multiple LLMs, including ensembling (e.g., Mixture-of-Experts), knowledge distillation, and multi-agent debate (MAD) [19, 25, 31]. While Mixture-ofExperts and knowledge distillation improve capacity, efficiency, or output quality, they rely on post-training or aggregation mechanisms, and thus do not explicitly support interactive, evidence-grounded reasoning at inference time. Inspired by Minsky’s Society of Mind [28], MAD instead allows multiple agents to assume different roles, exchange intermediate analyses, and reach a final decision through discussion or consensus [7, 10, 24].
Compared with general-purpose multi-agent frameworks, MedAgents [31] extended this idea to zero-shot medical reasoning by assigning domain expert roles and using multi-round discussion before final decision making. MDAgents [17] further adapted the collaboration structure according to medical task complexity, showing that the organization of agents can affect both efficiency and accuracy. Despite improvements over single LLMs, these models mainly rely on role-based discussion or response aggregation over already language-based evidence, which can lead to reasoning drift, low-quality feedback, and homogeneous outputs [33]. We argue that medical image understanding requires a different form of collaboration, where agents learn from specialized clinical evidence sources and their outputs are consolidated through complementary reasoning under a training-free judge mechanism.
Method
Overview of the LLM Council
We define SMILE as a council of LLMs specifically designed to analyze complementary attributes of liver ultrasound images. Each agent is fine-tuned on unique target datasets under a supervised fine-tuning (SFT) approach, which is illustrated in Figure 2(a). We adopt parameter-efficient low-rank adaptation (LoRA) [15], where agent-specific learnable parameters are introduced while keeping the shared multi-modal backbone frozen. Once all agents have been specialized, we ask them to collaborate under the guidance of an anchor LLM, referred to as the judge, which refines the outputs of each agent and provides both criticism and a verdict for their responses. As shown in Figure 2(b), a judge can provide the final caption under a defined set of rules.

Figure 2. An overview of our proposed method. (a) Training procedure of a single agent as explained in § 3.3. (b) Inference from multi-agent under consensus filtering by a judge to obtain a high-fidelity liver ultrasound report.
Problem Setup
Let denote the training dataset, where is a liver ultrasound case and is the corresponding ground-truth caption. Our goal is to approximate the conditional distribution of a clinically coherent report given an ultrasound image, denoted as , where represents the final consensus caption generated through multi-agent reasoning and judge aggregation.
Agent-Level Conditional Modeling
Let denote the agent-specific training dataset, where is the ground-truth caption for agent , represented as a token sequence . Each agent employs a frozen multi-modal backbone parameterized by and is adapted via the LoRA module with learnable parameters . By conditioning on an agent-specific prompt , the conditional distribution modeled by agent is defined as:
At inference time, the generated caption of agent is sampled as:
Each agent is trained independently using the SFT approach by minimizing the token-level negative log-likelihood. Specifically, the objective for agent can be written as:
Inference and Consolidation with a Judge
Once all agents are trained, their generated captions are aggregated and criticized by a judge model. The judge is a frozen LLM parameterized by and conditioned on a dedicated prompt . We refer to as the consensus filtering prompt, as it instructs the judge to extract a consensus from the complementary agentic responses by filtering the non-visual grounding attributes. The complete consensus-filtering prompt is provided in Supp. Note D.
Finally, the judge models the conditional distribution as:
and produces the final consensus caption
Experiments
Datasets
Dataset-1. We utilize a publicly available Liver Ultrasound dataset consisting of 1,183 liver ultrasound images and their corresponding radiology reports [22]. For each patient, the dataset provides two distinct liver views along with a textual description of the overall liver appearance. We divide the dataset at the patient level into training, validation, and test sets with a 0.70/0.10/0.20 ratio.
Dataset-2. We retrospectively collected an independent multi-modal liver fibrosis ultrasound dataset under a protocol approved by the Institutional Review Board of the College of Medicine, The Catholic University of Korea (KC25ZISI0058). Dataset includes 1,417 liver ultrasound images obtained from 286 patients (mean age: 55.18 16.04 years; 105 males (36.7%) and 181 females (63.3%)), along with 286 corresponding clinical descriptions. The ultrasound examinations consist of standardized liver views, including transverse and longitudinal scans of the left hepatic lobe, subcostal and intercostal scans of the right hepatic lobe, and hepatorenal comparative views. The clinical descriptions comprise imaging findings, histopathological examination results, and fibrosis stages based on the METAVIR grading system. The distribution of fibrosis stages () is as follows: F0, 34 (11.9%); F1, 107 (37.4%); F2, 38 (13.3%); F3, 67 (23.4%); F4, 40 (14.0%). We divide the patients into training and testing subsets with a ratio of 0.80/0.20. Additionally, we further split the training dataset, holding out 10% of it for validation.
Benchmarks
Baselines
We compare SMILE with three open source vision language models, including Qwen-2B [35], MedMO-4B [9], and MedGemma-4B [30]. All baselines are fine-tuned on the training subset of each benchmark dataset, following the same preprocessing and evaluation protocol to ensure a fair comparison with SMILE.
We further compare with MedAgents [31], a training-free multi-agent LLM for collaborative medical reasoning. To ensure a fair comparison, we follow the role-playing architecture and multi-stage collaboration protocol from the official implementation, including expert gathering, individual expert analysis, synthesized report generation, collaborative consultation, and final decision-making. Since the original MedAgents framework is designed for text-based medical reasoning, we adapt its prompts and agent roles to the liver ultrasound setting while preserving its original collaboration structure. Specifically, the individual experts, radiology- and pathology-specialized agents, have been fine-tuned on their respective domain-specific datasets. The task-adapted MedAgents configuration and collaboration protocol are detailed in Supp. Note A.1.
Evaluation Metrics
For natural language generation (NLG), we compare the generated text with ground truth reports and calculate ROUGE-1 (R-1), ROUGE-L (R-L) [26], METEOR (M) [3], and BERTScore (BS) [42] for assessing the linguistic quality of generated responses. For clinical efficacy (CE), we extract the appearance-based common findings from generated reports and report accuracy, precision, recall, and F1-score.
We evaluate clinical correctness using closed-set visual question answering (VQA). We derive a set of questions from appearance-based ultrasound findings and clinical attributes described in the ground truth reports of both datasets. We report accuracy and F1-score for VQA evaluation.
We utilize MedCLIP [38] to encode the ultrasound images and reports into visual and textual embeddings, respectively. We measure the image text alignment by computing the cosine similarity between the visual embedding of each liver ultrasound image and the text embedding of its generated report. A detailed discussion on evaluation metrics is provided in Supp. Note B.
Implementation Details
Role of Agents. We define three agents, represented by , , and , to decompose liver ultrasound understanding into complementary reasoning tasks. Agent A generates structured descriptions of appearance-based features, including echogenicity patterns, parenchymal texture, organ boundaries, and anatomical visibility. Its role is restricted to visually verifiable findings, without inferring pathological conclusions beyond the image evidence. Agent B performs pathology-oriented diagnostic reasoning and provides higher-level interpretations related to diffuse liver disease and fibrosis-relevant findings. Agent C follows a VQA setting, where it answers a predefined attribute question bank with a closed answer set.
Training Strategy. We select our agents and the judge model from a pool of open-source VLMs, including Qwen-2B, MedMO-4B, and MedGemma-4B. Specifically, we design the training procedure to utilize target captions from Dataset-1 () and Dataset-2 () to train Agent A and Agent B, respectively. To enforce visual grounding and reduce hallucinations and response repetition, we train Agent C on our VQA-driven questions derived from both datasets. All agents are trained independently on their respective datasets using SFT with parameter-efficient fine-tuning method (PeFT) [34].
Post-training and Hyperparameter Selection. During training, we select hyperparameters on the validation subsets by performing a bounded grid search over LoRA configurations. We search over rank , scale , learning rate in , and batch size in . The selected configurations are MedGemma-4B for Agent A () and Qwen-2B for both Agents B&C (), all trained with learning rate . LoRA is applied to all linear projections for A/B and only query/key projections for C. A zero-shot MedGemma-4B acts as a judge during inference, and aggregates all responses as detailed in 3.4. All models are implemented in PyTorch with CUDA using publicly available model weights and official implementations. All experiments are conducted on two A6000 GPU nodes. Additional implementation details are provided in Supp. Notes A and E.
Results
Medical Image Understanding
Table 1 reports the performance of SMILE against fine-tuned VLMs and a multi-agent VLM baseline. SMILE achieves the best overall performance among fine-tuned open-source VLMs on both datasets. On Dataset-1, our method outperforms the strongest fine-tuned baseline, MedMO-4B, improving ROUGE-1 by 4.3% from 0.69 to 0.72, ROUGE-L by 3.4% from 0.59 to 0.61, and METEOR by 19.1% from 0.47 to 0.56. Under CE evaluation, SMILE improves the F1-score by 18.8% from 0.69 to 0.82 compared with MedMO-4B. Similarly, on Dataset-2, SMILE outperforms the strongest fine-tuned baseline, MedGemma-4B, improving both ROUGE-1 and ROUGE-L by 19.0% and 28.2%, respectively.
| Dataset | Model | R-1 | R-L | M | BS | Acc | Prec | Rec | F1 | P | L | FLOPs |
| D-1 | Qwen-2B [35] | 0.10 | 0.09 | 0.02 | 0.81 | 0.88 | 0.88 | 0.05 | 0.01 | 4.36 | 13.2 | 3.9 |
| MedMO-4B [9] | 0.69 | 0.59 | 0.47 | 0.93 | 0.94 | 0.84 | 0.59 | 0.69 | 80.54 | 23.8 | 7.0 | |
| MedGem-4B [30] | 0.59 | 0.48 | 0.49 | 0.90 | 0.94 | 0.78 | 0.66 | 0.72 | 77.00 | 25.6 | 7.6 | |
| MedAgents [31] | 0.60 | 0.45 | 0.50 | 0.90 | 0.94 | 0.77 | 0.70 | 0.73 | 55.63 | 228.9 | 81.9 | |
| SMILE (Ours) | 0.72 | 0.61 | 0.56 | 0.93 | 0.96 | 0.85 | 0.80 | 0.82 | 55.63 | 74.3 | 34.0 | |
| D-2 | Qwen-2B [35] | 0.19 | 0.19 | 0.08 | 0.74 | – | – | – | – | 4.36 | 15.5 | 6.1 |
| MedMO-4B [9] | 0.40 | 0.38 | 0.29 | 0.80 | – | – | – | – | 80.54 | 28.1 | 11.0 | |
| MedGem-4B [30] | 0.42 | 0.39 | 0.35 | 0.78 | – | – | – | – | 77.00 | 30.2 | 11.9 | |
| MedAgents [31] | 0.37 | 0.33 | 0.31 | 0.77 | – | – | – | – | 55.63 | 266.1 | 116.7 | |
| SMILE (Ours) | 0.50 | 0.50 | 0.32 | 0.81 | – | – | – | – | 55.63 | 81.8 | 35.3 |
Table 1. Quantitative comparison of the proposed SMILE with finetuned state-of-the-art models. We report the NLG, CE and computational cost on the test subsets of Dataset-1 and Dataset-2. The best results are highlighted in bold. P, L, and FLOPs denote trainable parameters, inference latency, and floating-point operations, reported in millions (), seconds, and trillion () operations, respectively.
We further compare SMILE with MedAgents to evaluate the effectiveness of our designed framework for liver ultrasound understanding. On Dataset-1, SMILE improves over MedAgents by 0.12 in ROUGE-1, 0.16 in ROUGE-L, 0.06 in METEOR, and 0.09 in CE-F1. On Dataset-2, SMILE also improves over MedAgents by 0.13 in ROUGE-1, 0.17 in ROUGE-L, and 0.01 in METEOR.
SMILE achieves stronger medical image understanding across NLG and CE metrics despite requiring fewer trainable parameters than the single VLM baselines, such as MedMO-4B and MedGemma-4B. Compared with MedAgents under the same trainable parameter count of 55.63M, SMILE requires much less inference compute, with – fewer FLOPs and 3.1× lower latency on Dataset-1 (3.31× and 3.25× on Dataset-2). These results show that the role specialization and consensus filtering improve the use of both model capacity and efficiency of inference cost (see more analysis in Supp. Note E).
Evaluation on Visual Question Answering
Table 2 summarizes closed-ended VQA performance across 837 and 108 evaluated questions for Dataset-1 and Dataset-2, respectively. On Dataset-1, SMILE achieves the highest accuracy and F1-score, with scores of 0.77 and 0.82, respectively. Compared with the strongest baseline by F1-score, Qwen-2B, SMILE improves F1-score from 0.77 to 0.82, corresponding to a relative improvement of 6.5%. It also improves over MedMO-4B from 0.76 to 0.82 and over MedGemma-4B from 0.62 to 0.82. Similarly, on Dataset-2, SMILE achieves the highest F1-score of 0.65, improving over MedMO-4B from 0.57 to 0.65, MedGemma-4B from 0.61 to 0.65, and Qwen-2B from 0.44 to 0.65.
Table 2. VQA-based clinical verification on Dataset-1 (D-1) and Dataset-2 (D-2)
Image Text Alignment
Table 3 reports the cosine similarity between visual embeddings from liver ultrasound images and text embeddings from generated reports. SMILE achieves the highest image text alignment on both datasets, with a mean similarity of 0.51 on Dataset-1 and 0.36 on Dataset-2. On Dataset-1, SMILE improves the MedGemma-4B, from 0.49 to 0.51. It also improves over MedAgents from 0.45 to 0.51. These results suggest that the reports generated by SMILE are more closely aligned with the corresponding ultrasound images in the shared embedding space.
Table 3. Image text alignment between liver ultrasound images and generated reports. We report the mean and standard deviation of cosine similarity between visual and textual embeddings.
Ablation Studies
We perform ablation studies on agent specialization, agent collaboration, judge inputs, fusion strategy, and council composition, as shown in Table 4 (see implementation details in Supp. Note A). We note that each agent specializes in specific aspects of liver ultrasound knowledge; however, on its own, it struggles to achieve optimal understanding performance. For example, Agent A outperforms other single-agent VLMs by achieving ROUGE-L of 0.37, METEOR of 0.24, and CE-F1 of 0.53. While Agent B struggles to perform on-par with Agent A because Agent A has been finetuned on the specialized knowledge for appearance-based interpretation as compared to Agent B, which focuses on pathology-motivated cues. Agent C also performs poorly as a report generator, with ROUGE-L of 0.14 and METEOR of 0.06, but its CE-F1 of 0.26 shows that the VQA agent still contributes some finding-level evidence despite weak free text generation.
| Setting | R-1 | R-L | M | CE-F1 | P (M) | Calls | L (s) | FLOPs |
| Single Agents | ||||||||
| Agent A (Radiology Specialist) | 0.44{sup}† | 0.37{sup}† | 0.24{sup}† | 0.53 | 38.50 | 1 | 25.6 | 7.6 |
| Agent B (Pathology Specialist) | 0.25{sup}† | 0.24{sup}† | 0.05{sup}† | 0.03 | 14.61 | 1 | 13.2 | 3.9 |
| Agent C (VQA Specialist) | 0.18{sup}† | 0.14{sup}† | 0.06{sup}† | 0.26 | 2.52 | 1 | 49.4 | 14.6 |
| Pairwise Agents | ||||||||
| A + B | 0.70 | 0.62 | 0.50{sup}† | 0.70 | 53.11 | 3 | 70.2 | 24.4 |
| A + C | 0.42{sup}† | 0.33{sup}† | 0.16{sup}† | 0.73 | 41.02 | 3 | 58.7 | 27.9 |
| B + C | 0.29{sup}† | 0.21{sup}† | 0.13{sup}† | 0.62 | 17.13 | 3 | 46.3 | 24.2 |
| Judge Input | ||||||||
| SMILE (Judge with image only) | 0.17{sup}† | 0.14{sup}† | 0.06{sup}† | 0.20 | 0.00 | 1 | 26.7 | 8.6 |
| SMILE (Judge with agent outputs only) | 0.48{sup}† | 0.38{sup}† | 0.32{sup}† | 0.74 | 55.63 | 4 | 69.6 | 29.6 |
| Fusion Controls | ||||||||
| SMILE (Judge with an ensembling prompt) | 0.41{sup}† | 0.32{sup}† | 0.20{sup}† | 0.53 | 55.63 | 4 | 74.3 | 34.1 |
| SMILE (Judge with a Cross Attention Module) | 0.23{sup}† | 0.17{sup}† | 0.09{sup}† | 0.42 | 120.81 | 4 | 68.5 | 28.6 |
| Council and Judge Controls | ||||||||
| SMILE (Homogeneous Council) | 0.68{sup}† | 0.55{sup}† | 0.57 | 0.77 | 60.97 | 4 | 84.8 | 33.1 |
| SMILE (General Agents) | 0.30{sup}† | 0.21{sup}† | 0.13{sup}† | 0.72 | 0.00 | 4 | 74.3 | 34.0 |
| SMILE (Qwen-2B as a Judge) | 0.57{sup}† | 0.45{sup}† | 0.36 | 0.77 | 55.63 | 4 | 59.0 | 27.7 |
| SMILE (Ours) | 0.72 | 0.61 | 0.56 | 0.82 | 55.63 | 4 | 74.3 | 34.1 |
Table 4. Ablation analysis of SMILE on Dataset-1. We report single agents, pairwise agents, judge-input controls, fusion controls, and judge-backbone ablations. denotes the number of trainable parameters in millions, Calls denotes the number of model invocations per patient, denotes average inference latency per patient in seconds, and FLOPs denote floating-point operations, reported in trillions (). CE-F1 represents the clinical efficacy F1-score. For the NLG metrics, {sup}† indicates that the complete SMILE model significantly outperforms the corresponding ablation according to a two-sided Wilcoxon signed-rank test with Holm correction (). Unmarked NLG values indicate that the difference is not statistically significant. The statistical testing protocol is detailed in Supp. Note F.
Pairwise Agents. We perform pairwise ablations of our council, which shows that agent collaboration improves image understanding of liver diagnosis. Specifically, when combined Agents A and B, it gives the strongest pairwise report generation scores, increasing ROUGE-L from 0.37 to 0.62 and METEOR from 0.24 to 0.50 compared with Agent A alone. However, its CE-F1 remains 0.70, which is further improved by adding the VQA agent, and finally enforces the constraints upon council toward clinical entity preservation. We also observe that the A+C setting increases CE-F1 from 0.53 to 0.73 compared with Agent A alone, even though its ROUGE-L decreases from 0.37 to 0.33.
Judge Ablations. We isolate the contribution of the judge through input, fusion, and council controls: 1) using the image alone yields only 0.17 ROUGE-1 and 0.20 CE-
F1, whereas providing agent outputs without visual input improves these scores to 0.48 and 0.74, showing that agent-derived responses provide stronger clinical evidence to the judge; 2) simple aggregation remains insufficient, as an ensembling prompt achieves CE-F1 of 0.53, identical to Agent A alone, while Cross Attention fusion reaches only 0.23 ROUGE-1 and 0.42 CE-F1 despite requiring 120.81M trainable parameters; 3) a homogeneous council reaches 0.68 ROUGE-1 and 0.77 CE-F1, remaining below SMILE at 0.72 and 0.82 while requiring more trainable parameters (60.97M vs. 55.63M) and higher latency (84.8s vs. 74.3s); 4) general zero-shot agents achieve only 0.30 ROUGE-1 despite using the same four model calls, showing that additional inference alone does not guarantee better performance; and 5) replacing the MedGem-4B judge with Qwen-2B shows that specialized agents provide sufficiently informative clinical evidence for consolidation across different judge LLM. Overall, these results, including statistically significant improvements in NLG, suggest that SMILE benefits from complementary agent specialization and criticism-based consensus filtering rather than increased model capacity or inference cost.
Qualitative Analysis:
To further highlight the importance of a multi-agent LLM council in providing the complementary reasoning paths for liver ultrasound, we present a case study from VQA task and a case from Report Generation Task in (a) and (b) Figure S1. In example (a), the image shows diffused echogenicity in the parenchymal region. Initially, Agent C fails to answer correctly, as do other single LLM baselines. However, Agent A, which reasons over grounded captions learned during fine-tuning, and Agent B, which provides fibrosis hints based on contrast patterns, together supply the contextual information needed for the council to self-correct without any external context. Example (b) highlights an important failure mode of the council. The reference report describes a plump liver with increased parenchymal echogenicity, whereas Agent A incorrectly characterizes the liver as normal in shape and echogenicity. In this case, Agent C correctly identifies increased echogenicity, yet the judge fails to incorporate this evidence and does not recover the reference liver morphology and echogenicity. This case shows that consensus aggregation can fail when correct evidence from one specialist is ignored relative to conflicting agent responses (see more case-studies in Supp. Note C.)

Figure S1. Qualitative case studies for closed-set VQA. For each image and question we compare the responses generated by our multi-agent council against the reference answer. Red indicates incorrect predictions, green indicates correct predictions, and orange highlights complementary information contributed by individual agents during the reasoning process. Examples (a) and (b) are taken from Dataset-1 and Dataset-2, respectively.

Figure 3. Qualitative analysis of SMILE. (a) Successful correction using complementary agent evidence. (b) Failure case where the judge overlooks a correct agent finding. Red, green, and orange denote incorrect, correct, and complementary findings, respectively.
Discussion
Complementary roles of radiology- and pathology-aligned supervision. SMILE leverages two complementary sources of clinical supervision instead of relying on a single source. Radiology-aligned supervision captures visually observable findings, including echogenicity, parenchymal texture, anatomical visibility, and lesion appearance. Pathology-aligned supervision provides disease-level interpretive context through textual staging labels (e.g., fibrosis grade, steatosis score). Instead of forcing a single VLM to map images directly to both visual descriptions and disease stages, we assign each supervision target to a specialized agent. Finally, a training-free judge VLM, aligns the outputs of these agents by retaining findings supported by complementary evidence.
Benefits of consensus filtering over role-based aggregation. Role-based aggregation, as used in MedAgents, improves general medical QA through iterative expert discussion. However, it is less effective for ultrasound-specific visual reasoning because it lacks an explicit mechanism to verify whether a stated finding is grounded in the image. In SMILE, each agent is prompted to provide a complementary source of evidence: radiology-oriented morphology, pathology-aligned disease interpretation, and structured fact verification. Our prompting strategy ensures that the judge does not simply merge these outputs or select the majority response. Instead, it cross-references each candidate’s findings against the complementary evidence from the other agents, as shown in Figure S1. We observe that findings unsupported by cross-agent evidence or contradicted by other agents are filtered out, reducing hallucination-related errors in the final output.
Clinical adaptability under limited-data conditions. SMILE improves clinical adaptability by reducing the training and inference requirements that often limit medical multi-agent systems in data-scarce radiology settings. First, SMILE does not require end-to-end fine-tuning of a large VLM. Each agent is adapted through lightweight LoRA for a specific supervision source. Second, SMILE does not rely on multiple rounds of agent debate. Each agent produces its report once, while a zero-shot VLM acting as the judge performs consensus filtering, reducing inference complexity and limiting repeated reasoning errors compared with multi-agent systems that require iterative discussion. Third, SMILE enables learning from unpaired radiology and pathology supervision without requiring each ultrasound case to include both a radiology report and a pathology-based description. Instead, agents are specialized using their respective supervision sources, and their outputs are integrated at inference through judge-guided consensus filtering. This design makes SMILE better suited for heterogeneous clinical datasets, where data are often uni-modal, institution-dependent, and collected under different reporting protocols.
Limitations. Our results also highlight the limitations of direct fine-tuning in low-data medical VLM settings. Although fine-tuned single models improve task adaptation, they remain sensitive to dataset bias and incomplete clinical supervision. If all agents miss the same finding or produce the same incorrect observation, the judge may not recover the correct interpretation. Furthermore, while we avoid iterative debate, the parallel forward pass of three specialized agents increases inference latency compared to a single model. Future work should evaluate SMILE across larger multi-center cohorts and diverse clinical tasks to assess its broader generalization.
Conclusion
We propose SMILE, a structured multi-agent framework that enhances interpretability in liver ultrasound understanding. By leveraging agents trained on diverse information sources, our approach provides complementary context to generate visually-grounded and accurate medical reports. Experimental results demonstrate that SMILE significantly outperforms conventional methods in both image understanding and clinical diagnostic performance. Future work will focus on incorporating reinforcement learning-based policy optimization for judge decisions and developing more sophisticated collaborative rules, which could drive further breakthroughs in both natural imaging and clinical applications.
Acknowledgement
This work was supported by the National Research Foundation of Korea (NRF) grant funded by the Korea government (MSIT) (No. RS-2024-00335185). This work was also supported by the Korea Medical Device Development Fund grant funded by the Korea government (the Ministry of Science and ICT, the Ministry of Trade, Industry and Energy, the Ministry of Health & Welfare, the Ministry of Food and Drug Safety) (Project Number: RS-2026-25543484).
Baseline and Control Implementations
Task Adapted MedAgents
MedAgents [31] is originally formulated as a zero-shot collaborative medical reasoning framework. For a fair task-specific comparison, we instantiate the three expert roles corresponding to the SMILE radiology, pathology, and VQA specialists using the same adapted models employed in our council. Any additional experts recruited by the MedAgents protocol retain their original zero-shot behavior. The coordination and discussion procedure is also preserved from the original implementation. This design provides both methods with comparable task-specific specialist knowledge while retaining their respective collaboration mechanisms.
Task adaptation. The three adapted experts use the same backbones and LoRA configurations as their corresponding SMILE agents. Their combined trainable parameter budget is 55.63M, which matches the SMILE council. The remaining MedAgents experts and coordination components receive no task-specific adaptation.
Collaboration protocol. We retain the complete MedAgents inference procedure, which consists of expert recruitment, independent analysis, collaborative revision, and final synthesis. The three task-adapted specialists participate within the recruited expert panel, while any remaining recruited experts continue to operate in the original zero-shot setting. During collaborative revision, each participating expert receives the current responses of the other experts before producing its updated assessment. The final moderator then synthesizes the accumulated discussion into the output. All model invocations required by this protocol are included in the computational accounting reported in Table 1 of the main paper.
At an identical trainable parameter budget, the resulting MedAgents implementation requires the FLOPs and the latency of SMILE on Dataset-1. The comparison therefore evaluates the two collaboration strategies under matched task-specific adaptation capacity while accounting for their complete inference procedures.
Reproducibility. We will release the task-adapted MedAgents implementation together with the SMILE training and inference code through the project repository with the camera-ready version.
Cross-Attention Fusion
This control replaces prompt-level consolidation with representation-level fusion, following the gated cross-attention mechanism introduced in Flamingo [2]. The judge backbone remains the MedGemma-4B model used by SMILE, but it is trained here rather than frozen.
| Setting | Value |
| Adapter | LoRA |
| Radiology expert | MedGemma-4B, |
| Pathology expert | Qwen2-VL-2B, |
| VQA expert | Qwen2-VL-2B, restricted |
| Additional MedAgents experts | Zero shot |
| Coordinator / moderator | Frozen |
| Optimiser | AdamW, weight decay 0.01 |
| Learning rate | |
| Epochs / batch | 3 / 1 with gradient accumulation |
| Trainable budget | 55.63M |
| Maximum new tokens | 512 |
Table S1. Hyperparameters of Task-Adapted MedAgents
Architecture. The three specialist outputs are encoded once by a frozen Bio_ClinicalBERT text encoder, and a learned agent-type embedding is added to each stream so that the judge can distinguish its source. A Q-Former-style resampler then compresses the encoded evidence into 8 latent queries at an internal width of 512. Three gated cross-attention blocks are inserted into the MedGemma-4B decoder at layers 8, 17, and 25 of 34, each computing . Every gate is initialised to zero, so that at the first optimisation step and the model is numerically identical to a plain LoRA baseline. Any subsequent divergence therefore measures how much evidence the gates admit, and it does so independently of any caption metric.
Training and budget. The backbone is held frozen in 4-bit precision and adapted with LoRA , trained for one epoch on Dataset-1 with the fusion module in fp32 and the language model in bf16. The trainable budget comprises 26.68M for the fusion module, covering the resampler, the three cross-attention blocks and their gates, together with 38.50M for the LoRA adapter. The trained judge therefore accounts for 65.18M parameters, and the control totals 120.81M once the 55.63M specialist adapters are included. Constraining the cross-attention to an internal width of 512 is what keeps the module comparable in size to the council, since an unconstrained variant operating at the full decoder width of 2560 requires 211.82M parameters for the fusion module alone. This control reaches 0.23 ROUGE-1 and 0.42 CE-F1, against 0.72 and 0.82 for SMILE, and it tests whether representation-level fusion on its own can reproduce rule-constrained consensus consolidation.
Council Controls
We construct two council variants to examine specialist supervision and backbone heterogeneity while preserving the overall SMILE inference structure.
General-agent council. The general-agent council preserves the same three agent roles, four model invocations, role prompts, and MedGemma-4B judge used by SMILE. The specialist agents receive no task-specific LoRA adaptation and instead operate directly from their pretrained checkpoints under the corresponding role instructions. This control retains the council structure and inference budget while removing task-specific specialist supervision. It therefore evaluates the contribution provided by pretrained zero-shot agents and role prompting alone.
Homogeneous council. The homogeneous council retains task-specific specialist supervision and the same judge-mediated consensus procedure as SMILE, but all specialist roles are instantiated from the MedGemma-4B backbone family. Each role is adapted using its corresponding supervision source. The radiology, pathology, and VQA roles therefore remain functionally specialized, while backbone diversity is removed. This control enables the contribution of role-specific adaptation to be examined under a common backbone family.
The general and homogeneous councils use the same final MedGemma-4B judge and the same consensus-filtering procedure as SMILE. Their performance, trainable parameters, model calls, latency, and FLOPs are reported in Table 4 of the main paper.
Supplementary Note B: Clinical Evaluation Protocol
We use complementary automatic metrics for report similarity, clinical-entity preservation, finding-level verification, and image–text alignment. These metrics are not interpreted as direct measures of clinical factuality, which would require finding-level clinical annotation.
Natural language generation
ROUGE-1, ROUGE-L [26], METEOR [3] and BERTScore [42] are used as report-similarity measures: they quantify lexical and embedding-level agreement between a generated report and its reference. We interpret them as report-similarity measures rather than direct evidence of clinical correctness; Table S5 illustrates this limitation.
| Patient 216747 | |
| Reference report | The liver was plump in shape, with a smooth capsule, dense and enhanced echoes in the parenchyma […] After cholecystectomy, the upper segment of the extrahepatic bile duct is about _SCM_ wide, with no abnormal echo in the segment. […] |
| Agent A: Radiology Specialist | The liver has a normal size and shape, with a smooth capsule. The liver parenchyma demonstrates normal echogenicity and echotexture. The gallbladder is normal in size and echogenicity, with no obvious abnormal echo in the cavity. […] |
| Agent B: Pathology specialist | Lobular activity, none; portal–periportal activity, none; steatosis, mild; fibrosis, none. |
| Agent C: Structured VQA | Echogenicity increased? yes · Echotexture coarse? no · Attenuation increased? no |
| SMILE Judge : Final Response | The liver parenchyma shows no definite focal abnormality. The gallbladder size and echogenicity are normal, and no obvious abnormal echo is identified in the cavity. The visualized biliary structures show no definite dilation. […] |
| Patient 235520 | |
| Reference report | The liver is plump in shape and has a smooth capsule. A cystic structure can be seen in the right lobe of the liver, about 2DS in size, with a clear border and enhanced rear echo. The echo of the remaining parenchyma is finely enhanced. […] |
| Agent A: Radiology Specialist | The liver has a normal size and shape, with a smooth capsule. The liver parenchyma demonstrates relatively homogeneous echotexture. There is no evidence of a focal lesion in the visualized liver parenchyma. […] |
| Agent B: Pathology specialist | Lobular activity, none; portal–periportal activity, mild; steatosis, none; fibrosis, none. |
| Agent C: Structured VQA | Echogenicity increased? yes · Echotexture coarse? no · Attenuation increased? no |
| SMILE Judge : Final Response | The liver parenchyma appears relatively homogeneous, with no focal lesion identified in the available views. No additional focal abnormality is evident in the visualized hepatic parenchyma. […] |
| Failure locus | the right-lobe cystic structure is missed by the specialist and remains absent from the judge output. The structured VQA questions assess diffuse parenchymal attributes and do not interrogate focal lesions. |
Table S5. Qualitative case studies (failure): correct findings absent from the available evidence propagate through consensus (patients 216747 and 235520, Dataset-1). A prior cholecystectomy (top) and a right-lobe cystic structure (bottom) are recorded in the reference (underlined) but the correct findings are unavailable in the specialist evidence, so the consolidated report contradicts or omits them (red). This delimits the mechanism: consensus filtering is effective when corroborating evidence is available but remains limited when the correct attribute is absent from all specialist outputs. Because the surrounding template text is reproduced correctly, -gram overlap with the reference is largely unaffected, illustrating the limitation of report-similarity metrics (Sec. B). _SCM_ is the corpus’ masked-measurement token.
Clinical Efficacy
Clinical efficacy (CE) measures whether a generated report preserves clinically relevant entities from the reference report.
Entity vocabulary. We start from the 16 appearance-based seed terms provided with Dataset-1. Several seed terms refer to the same underlying clinical concept and are therefore consolidated into canonical entities. To account for lexical variation in radiology reporting, we use an LLM to propose morphological variants, plural forms, adjectival forms, and common clinical abbreviations for each seed term. Every proposed variant is subsequently verified against the Dataset-1 report corpus before inclusion. This process yields 13 canonical entities and 100 accepted surface variants:
| Canonical entity | Surface variants |
| portal vein | portal vein, main portal vein, portal venous |
| bile duct | bile duct(s), intrahepatic bile ducts, biliary |
| gallbladder | gallbladder, GB |
| pancreas | pancreas, pancreatic |
| spleen | spleen, splenic |
| parenchyma | parenchyma, parenchymal, parenchymal echo, spleen/pancreas parenchymal |
| echogenicity | echogenicity, echogenic, echogenicities |
| echotexture | echotexture, echotextural, coarse, heterogeneous, heterogeneity |
| attenuation | attenuation, acoustic attenuation, posterior attenuation |
| capsule | capsule, capsular |
| lesion | lesion(s), space-occupying |
| cyst | cyst, cysts, cystic |
| nodule | nodule, nodules |
Table S2. Clinical entity vocabulary used for CE evaluation. Representative surface variants are shown for the 13 canonical entities derived from the Dataset-1 seed terminology. The complete vocabulary contains 100 accepted surface variants.
Severity handling. CE is evaluated at the canonical-entity level. Severity modifiers remain in the report text but are not scored as independent entities. CE therefore measures entity preservation rather than agreement in severity.
Matching and scoring. Each report is converted into a binary presence vector over the canonical entities. An entity is marked as present when any accepted surface variant occurs in the lower-cased, whitespace-normalised text. Accuracy, precision, recall, and F1-score are computed between the reference and generated vectors using micro aggregation over entities and reports. The current CE protocol is presence-based and polarity-agnostic. For example, both “no dilation of the bile ducts” and “the bile ducts were dilated” register the bile duct entity. CE should therefore be interpreted as preservation of clinically relevant report content and considered together with the closed-set VQA evaluation, which tests finding-level polarity.
Closed-Set Visual Question Answering
VQA provides finding-level verification of clinically defined ultrasound attributes. For each dataset, three predefined questions are derived from the corresponding reference findings and answered independently for each patient:
| Dataset | Question |
| D-1 | Is the liver parenchymal echogenicity diffusely increased in the imaged portion? Is the hepatic echotexture coarse or heterogeneous in the imaged portion? Is posterior acoustic attenuation increased in the imaged portion? |
| D-2 | Is steatosis or fatty change reported for the liver, and is it visually assessable? Is cirrhosis reported, and are cirrhotic morphological features visually assessable? Is a focal hepatic lesion reported, and is it visually assessable? |
Table S3. List of closed-ended Questions for VQA
Answers are restricted to {yes, no, not assessable}. Accuracy is computed over the evaluable patient-question pairs, while F1-score is obtained by macro aggregation across questions and patients. We report both metrics because the answer distributions are imbalanced across datasets and question categories. Qualitative examples are provided in Figure S1.
Image–Text Alignment
Cross-modal similarity provides an auxiliary measure of whether the semantic content of a generated report remains aligned with the source image. Such alignment is relevant to multimodal downstream applications, including image-report retrieval and representation learning. We therefore compute cosine similarity between the ultrasound image embedding and generated-report embedding using the fixed MedCLIP [38] image-text encoder.
MedCLIP was pretrained using MIMIC-CXR and CheXpert chest radiograph data rather than liver ultrasound, and we therefore expect limited sensitivity to fine-grained sonographic differences. This domain mismatch may partly explain the relatively small absolute differences observed between systems. We retain MedCLIP because it provides a publicly available medical contrastive image-text representation and has been evaluated for both zero-shot prediction and image-text retrieval [38]. All methods are evaluated using the same frozen encoder, so we interpret the resulting values only as a relative auxiliary alignment measure rather than as evidence of clinical correctness. ... Future work can evaluate ultrasound-specific or SigLIP-style medical contrastive encoders once validated representations for this modality become available.
Supplementary Note C: Success and Failure Analysis
Successful Consensus Correction
Table S4 shows representative cases in which conflicting specialist evidence is reconciled and the judge recovers clinically relevant reference findings.
| Patient 225145 | Reference report (ground truth): The liver has a plump shape, a smooth capsule, dense and enhanced parenchymal echoes, slightly attenuated backfield echoes, unclear display of the portal vein system, and enhanced liver and kidney echo contrast. No dilation of intrahepatic bile ducts was found. The inner diameter of the main portal vein is within the normal range. [. . .] |
| Patient 225145 | Agent A : Radiology specialist response: The liver has a normal size and shape. The capsule is not clearly visualized. The liver parenchyma demonstrates normal echogenicity with a relatively homogeneous echotexture. Posterior acoustic attenuation is not evident. [. . .] |
| Patient 225145 | Agent B : pathology specialist: Lobular activity, mild; portal–periportal activity, none; steatosis, none; fibrosis, none. |
| Patient 225145 | Agent C : structured VQA: Echogenicity increased? yes · Echotexture coarse? yes · Attenuation increased? no (reference: yes) |
| Patient 225145 | SMILE judge: The hepatic parenchyma demonstrates increased echogenicity with a coarse and heterogeneous appearance. Liver size and capsular contour cannot be confidently characterized from the available views. Posterior acoustic attenuation is not clearly demonstrated. [. . .] |
| Patient 219190 | Reference report (ground truth): The liver has a plump shape, a smooth capsule, dense and uneven parenchymal echoes, attenuated rear echoes, enhanced liver and kidney echoes, and unclear display of the portal venous system. [. . .] |
| Patient 219190 | Agent A : Radiology specialist: The liver has a normal size and shape, with the capsule not clearly visualized. The parenchymal echoes are relatively homogeneous, and posterior attenuation is unremarkable. The portal venous system shows no obvious abnormality. [. . .] |
| Patient 219190 | Agent B : Pathology specialist: Steatohepatitis, mild; lobular activity, none; portal–periportal activity, none; fibrosis, none; steatosis, none. |
| Patient 219190 | Agent C : structured VQA: Echogenicity increased? yes · Echotexture coarse? yes · Attenuation increased? no (reference: yes) |
| Patient 219190 | SMILE Judge : Final Response: A plump shape of the liver is present, with a smooth capsule. The liver parenchyma demonstrates dense and uneven echoes, with attenuated rear echoes. The liver and kidney echoes are enhanced, and the portal venous system is not clearly displayed. [. . .] |
Table S4. Qualitative case studies (success): the judge overrides erroneous specialist readings (patients 225145 and 219190, Dataset-1). In both cases the radiology specialist reads the liver as normal and the VQA specialist answers the posterior-attenuation question incorrectly (red), yet the consolidated report re-establishes the reference findings marked in green, namely increased parenchymal echogenicity, a coarse or heterogeneous echo-texture, and the plump hepatic contour, because the remaining evidence outputs contradict the erroneous reading. [. . .] marks elided template organ sentences (bile ducts, portal vein, gallbladder, pancreas, spleen).
Failure Modes
Table S5 presents the complementary failure mode: the correct finding or clinical status is absent from the available specialist evidence and therefore remains incorrect or missing after consolidation.
Qualitative VQA Examples
Figure S1 provides representative closed-set VQA predictions from both datasets.
Supplementary Note D: Consensus-Filtering Judge Prompt
Figure S2 reproduces the complete criticism-based consensus-filtering instruction given to the SMILE judge, exactly as issued at inference time: the judge receives both views, the candidate reports of Agents A and B, and Agent C’s structured answers; it must audit itself against three rule families (R1–R3), criticise each source separately, and return the final caption as strict JSON. The prompt makes the consolidation rules explicit: the mandatory incorporation rule projects Agent C’s three parenchymal attributes into the final caption, constraint 4 strips Candidate B’s non-imaging pathology content, and the per-source criticisms force the judge to commit to a verdict before writing the consolidated report.

Figure S2. The complete consensus-filtering judge prompt. System and user messages issued to the SMILE judge for every patient, verbatim from the inference code; only the per-case content is replaced by /slots. The three rule families audited in rules_check (R1: non-imaging content, R2: diagnosis claims, R3: agent/VQA mentions) mirror hard constraints 1–3, and the mandatory incorporation rule couples the caption to Agent C’s three parenchymal attributes.
Supplementary Note E: Model Capacity and Computational Cost
Trainable Adaptation Capacity
To examine whether greater trainable capacity explains the observed gains, we first strengthen the single-VLM baselines used throughout the revised evaluation. MedGemma-4B is increased to LoRA rank , corresponding to 77.00M trainable parameters, while MedM0-4B uses 80.54M trainable parameters. Both exceed the 55.63M trainable budget of SMILE, yet neither recovers its overall Dataset-1 performance in Table 1–Table 3 of the main paper.
We further analyze this trend by scaling MedGemma-4B over LoRA ranks , corresponding to 38.50M, 77.00M, and 115.50M trainable parameters. Figure S3 shows that increasing adaptation capacity improves the single-model baseline consistently. From to , ROUGE-1 increases from 0.44 to 0.70, ROUGE-L from 0.37 to 0.59, and CE-F1 from 0.53 to 0.78. However, the model uses approximately the trainable parameters of SMILE while remaining below it on ROUGE-1 (0.70 vs. 0.72), ROUGE-L (0.59 vs. 0.61), and CE-F1 (0.78 vs. 0.82).
Figure S3. Effect of increasing single-model adaptation capacity. Performance of MedGemma-4B with increasing LoRA rank is compared with SMILE on Dataset-1. Increasing the trainable capacity improves the single-model baseline, while SMILE retains stronger ROUGE-1, ROUGE-L, and CE-F1 performance at a lower trainable parameter budget.
Computational Cost Accounting
We use the generally used evaluation protocol for the computational-cost columns in the main tables. Every required model invocation is charged to the corresponding system. Single-model baselines and MedAgents are allocated output budgets of 256 and 512 tokens per generative call, respectively. Dataset-1 contains 2 views per patient (512 image tokens), whereas Dataset-2 contains an average of 3.96 views per patient (1015 image tokens).
| Component | Backbone | Rank | Trainable |
| Agent A (radiology) | MedGemma-4B | 38.50M | |
| Agent B (pathology) | Qwen2-VL-2B | 14.61M | |
| Agent C (VQA){sup}* | Qwen2-VL-2B | 2.52M | |
| Judge | MedGemma-4B | frozen | 0.00M |
| SMILE total | 55.63M |
Table S6. Where SMILE’s trainable parameters sit. The aggregation step is frozen, so extending the council with a new specialist requires only the specialist adapter and no judge retraining. The parameters are partitioned across independently trained, individually reusable specialists: the largest single training run (38.50M) is smaller than the MedGemma-4B (77.00M) and MedMO-4B (80.54M) single-model baselines, each of which trains one adapter in a single run. {sup}*Agent C adapts a restricted module set (query and key projections only), which is why its budget is well below Agent B’s at the same rank.
Supplementary Note F: Statistical Testing
For each NLG metric we compute per-patient scores for both systems on the same test cohort and apply a two-sided Wilcoxon signed-rank test to the paired differences. We select a rank-based test because per-patient NLG scores are bounded and do not follow a normal distribution. We then correct for multiple comparisons across the family of tests within each table using the Holm–Bonferroni procedure. A dagger (†) denotes ; unmarked NLG values are not statistically significant at that level.
Scope of inference. The Wilcoxon analysis quantifies paired output differences on the same patients and is not used as an estimate of training-seed variability. CE-F1 is reported separately because the main-table significance markers are restricted to patient-level NLG metrics. This scope is stated explicitly in the revised Table 4 caption.
References
- [1]Aisha Alansari and Hamzah Luqman. Large language models hallucination: A comprehensive survey. arXiv preprint arXiv:2510.06265, 2025. 2arxiv.org/abs/2510.06265
- [2]Jean-Baptiste Alayrac, Jeff Donahue, Pauline Luc, Antoine Miech, Iain Barr, Yana Hasson, Karel Lenc, Arthur Mensch, Katherine Millican, Malcolm Reynolds, et al. Flamingo: a visual language model for few-shot learning. Advances in neural information processing systems, 35:23716–23736, 2022. 2, 12arxiv.org/abs/2204.14198
- [3]Satanjeev Banerjee and Alon Lavie. Meteor: An automatic metric for evaluation with improved correlation with human judgments. In Proceedings of the acl workshop on intrinsic and extrinsic evaluation measures for machine translation and/or summarization, pages 65–72, 2005. 4, 14
- [4]Ramón Bataller, David A Brenner, et al. Liver fibrosis. J. Clin. Invest., 115(2):209–218, 2005. 2
- [5]Jonas Becker, Lars Benedikt Kaesberg, Andreas Stephan, Jan Philip Wahle, Terry Ruas, and Bela Gipp. Stay focused: Problem drift in multi-agent debate. In Findings of the Association for Computational Linguistics: EACL 2026, pages 5068–5102, 2026. 2
- [6]Junying Chen, Chi Gui, Ruyi Ouyang, Anningzhe Gao, Shunian Chen, Guiming Hardy Chen, Xindong Wang, Zhenyang Cai, Ke Ji, Xiang Wan, and Benyou Wang. Towards injecting medical visual knowledge into multimodal LLMs at scale. In Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, pages 7346–7370, Miami, Florida, USA, 2024. Association for Computational Linguistics. 2
- [7]Justin Chen, Swarnadeep Saha, and Mohit Bansal. Reconcile: Round-table conference improves reasoning via consensus among diverse llms. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 7066–7085, 2024. 2, 3
- [8]Zhihong Chen, Yan Song, Tsung-Hui Chang, and Xiang Wan. Generating radiology reports via memory-driven transformer. In Proceedings of the 2020 conference on empirical methods in natural language processing (EMNLP), pages 1439–1449, 2020. 2arxiv.org/abs/2010.16056
- [9]Ankan Deria, Komal Kumar, Adinath Madhavrao Dukre, Eran Segal, Salman Khan, and Imran Razzak. Medmo: Grounding and understanding multimodal large language model for medical images, 2026. 4, 5, 6arxiv.org/abs/2602.06965
- [10]Yilun Du, Shuang Li, Antonio Torralba, Joshua B Tenenbaum, and Igor Mordatch. Improving factuality and reasoning in language models through multiagent debate. arXiv preprint arXiv:2305.14325, 2023. 3arxiv.org/abs/2305.14325
- [11]Julia C D’Souza, Laith R Sultan, Stephen J Hunt, Susan M Schultz, Angela K Brice, Andrew KW Wood, and Chandra M Sehgal. B-mode ultrasound for the assessment of hepatic fibrosis: A quantitative multiparametric analysis for a radiomics approach. Scientific Reports, 9(1):8708, 2019. 2DOI
- [12]Andrew Estornell and Yang Liu. Multi-llm debate: Framework, principals, and interventions. Advances in Neural Information Processing Systems, 37:28938–28964, 2024. 2DOI
- [13]Mireen Friedrich-Rust, Katrin Wunder, Susanne Kriener, Fariba Sotoudeh, Swantje Richter, Joerg Bojunga, Eva Hermann, Thierry Poynard, Christoph F Dietrich, Johannes Vermehren, et al. Liver fibrosis in viral hepatitis: noninvasive assessment with acoustic radiation force impulse imaging versus transient elastography. Radiology, 252(2):595–604, 2009. 1
- [14]Takanobu Hirosawa, Ren Kawamura, Yukinori Harada, Kazuya Mizuta, Kazuki Tokumasu, Yuki Kaji, Tomoharu Suzuki, and Taro Shimizu. Chatgpt-generated differential diagnosis lists for complex case-derived clinical vignettes: diagnostic accuracy evaluation. JMIR Med. Inform., 11:e48808, 2023. 2DOI
- [15]Edward J. Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen. Lora: Low-rank adaptation of large language models, 2021. 3arxiv.org/abs/2106.09685
- [16]Yunsang Joo, Hyun-Cheol Park, O-Joun Lee, Changhan Yoon, Moon Hyung Choi, and Chang Choi. Classification of liver fibrosis from heterogeneous ultrasound image. Ieee Access, 11:9920–9930, 2023. 2DOI
- [17]Yubin Kim, Chanwoo Park, Hyewon Jeong, Yik S Chan, Xuhai Xu, Daniel McDuff, Hyeonhoon Lee, Marzyeh Ghassemi, Cynthia Breazeal, and Hae W Park. Mdagents: An adaptive collaboration of llms for medical decision-making. Advances in Neural Information Processing Systems, 37:79410–79452, 2024. 2, 3arxiv.org/abs/2404.15155
- [18]Yubin Kim, Hyewon Jeong, Shan Chen, Shuyue Stella Li, Chanwoo Park, Mingyu Lu, Kumail Alhamoud, Jimin Mun, Cristina Grau, Minseok Jung, et al. Medical hallucinations in foundation models and their impact on healthcare. arXiv preprint arXiv:2503.05777, 2025. 2
- [19]Byung-Kwan Lee, Ryo Hachiuma, Yong Man Ro, Frank Wang, and Yueh-Hua Wu. Unified reinforcement and imitation learning for vision-language models. Advances in Neural Information Processing Systems, 38:156508–156534, 2026. 2arxiv.org/abs/2510.19307
- [20]Jeong Hyun Lee, Ijin Joo, Tae Wook Kang, Yong Han Paik, Dong Hyun Sim, Sang Yun Ha, Kyunga Kim, Choonghwan Choi, Gunwoo Lee, Jonghyon Yi, et al. Deep learning with ultrasonography: automated classification of liver fibrosis using a deep convolutional neural network. Eur. Radiol., 30(2):1264–1273, 2020. 2DOI
- [21]Chunyuan Li, Cliff Wong, Sheng Zhang, Naoto Usuyama, Haotian Liu, Jianwei Yang, Tristan Naumann, Hoifung Poon, and Jianfeng Gao. Llava-med: Training a large language-and-vision assistant for biomedicine in one day. Advances in Neural Information Processing Systems, 36:28541–28564, 2023. 2arxiv.org/abs/2306.00890
- [22]Jun Li, Tongkun Su, Baoliang Zhao, Faqin Lv, Qiong Wang, Nassir Navab, Ying Hu, and Zhongliang Jiang. Ultrasound report generation with cross-modality feature alignment via unsupervised guidance. IEEE Trans. Med. Imaging, 44(1):19–30, 2025. 4arxiv.org/abs/2406.00644
- [23]Mingjie Li, Wenjia Cai, Karin Verspoor, Shirui Pan, Xiaodan Liang, and Xiaojun Chang. Cross-modal clinical graph transformer for ophthalmic report generation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 20656–20665, 2022. 2arxiv.org/abs/2206.01988
- [24]Tian Liang, Zhiwei He, Wenxiang Jiao, Xing Wang, Yan Wang, Rui Wang, Yujiu Yang, Shuming Shi, and Zhaopeng Tu. Encouraging Divergent Thinking in Large Language Models through Multi-Agent Debate. In Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, pages 17889–17904. Association for Computational Linguistics, 2024. 3arxiv.org/abs/2305.19118
- [25]Bin Lin, Zhenyu Tang, Yang Ye, Jinfa Huang, Junwu Zhang, Yatian Pang, Peng Jin, Muran Ning, Jiebo Luo, and Li Yuan. Moe-llava: Mixture of experts for large vision-language models. IEEE Transactions on Multimedia, 2026. 2arxiv.org/abs/2401.15947
- [26]Chin-Yew Lin. Rouge: A package for automatic evaluation of summaries. In Text summarization branches out, pages 74–81, 2004. 4, 14
- [27]Xu Liu, Chaoli Duan, Min-kyu Kim, Lu Zhang, EunJin Jee, Beenu Maharjan, Yuwei Huang, Dan Du, and Xian Jiang. Claude 3 opus and chatgpt with gpt-4 in dermoscopic image analysis for melanoma diagnosis: comparative performance analysis. JMIR Med. Inform., 12:e59273, 2024. 2DOI
- [28]Marvin Minsky. Society of mind. Simon and Schuster, 1986. 3
- [29]Michael Moor, Qian Huang, Shirley Wu, Michihiro Yasunaga, Yash Dalmia, Jure Leskovec, Cyril Zakka, Eduardo Pontes Reis, and Pranav Rajpurkar. Med-flamingo: a multimodal medical few-shot learner. In Machine learning for health (ML4H), pages 353–367. PMLR, 2023. 2arxiv.org/abs/2307.15189
- [30]Andrew Sellgeren, Sahar Kazemzadeh, Tiam Jaroensri, Atilla Kiraly, Madeleine Traverse, Timo Kohlberger, Shawn Xu, Fayaz Jamil, Cían Hughes, Charles Lau, et al. Medgemma technical report. arXiv preprint arXiv:2507.05201, 2025. 2, 4, 5, 6
- [31]Xiangru Tang, Anni Zou, Zhuosheng Zhang, Ziming Li, Yilun Zhao, Xingyao Zhang, Arman Cohan, and Mark Gerstein. Medagents: Large language models as collaborators for zero-shot medical reasoning. In Findings of the Association for Computational Linguistics: ACL 2024, pages 599–621, 2024. 2, 3, 4, 5, 6, 12arxiv.org/abs/2311.10537
- [32]Hisham Tchelebi, Philip W. Ralls, Randall Radin, and Edward Grant. Sonography of diffuse liver disease. J. Ultrasound Med., 21(9):1023–1032, 2002. 2
- [33]Khanh-Tung Tran, Dung Dao, Minh-Duong Nguyen, Quoc-Viet Pham, Barry O’Sullivan, and Hoang D Nguyen. Multi-agent collaboration mechanisms: A survey of llms. arXiv preprint arXiv:2501.06322, 2025. 2, 3arxiv.org/abs/2501.06322
- [34]Luping Wang, Sheng Chen, Linnan Jiang, Shu Pan, Runze Cai, Sen Yang, and Fei Yang. Parameter-efficient fine-tuning in large language models: a survey of methodologies. Artif. Intell. Rev., 58(8):227, 2025. 5DOI
- [35]Peng Wang, Shuai Bai, Sinan Tan, Shijie Wang, Zhihao Fan, Jinze Bai, Kevin Chen, Xuejing Liu, Jialin Wang, Wenbin Ge, Yang Fan, Kai Dang, Mengfei Du, Xuancheng Ren, Rui Men, Dayiheng Liu, Chang Zhou, Jingren Zhou, and Jun-yang Lin. Qwen2-vl: Enhancing vision-language model’s perception of the world at any resolution, 2024. 2, 4, 5, 6arxiv.org/abs/2409.12191
- [36]Yihan Wang, Andrew Bai, Nanyun Peng, and Cho-Jui Hsieh. On the loss of context-awareness in general instruction fine-tuning. arXiv preprint arXiv:2411.02688, 2024. 2arxiv.org/abs/2411.02688
- [37]Zhanyu Wang, Mingkang Tang, Lei Wang, Xiu Li, and Luping Zhou. A medical semantic-assisted transformer for radiographic report generation. In International Conference on Medical Image Computing and Computer-Assisted Intervention, pages 655–664. Springer, 2022. 2arxiv.org/abs/2208.10358
- [38]Zifeng Wang, Zhenbang Wu, Dinesh Agarwal, and Jimeng Sun. Medclip: Contrastive learning from unpaired medical images and text. In Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, pages 3876–3887, 2022. 4, 15arxiv.org/abs/2210.10163
- [39]Zhanyu Wang, Lingqiao Liu, Lei Wang, and Luping Zhou. R2gengpt: Radiology report generation with frozen llms. Meta-Radiology, 1(3):100033, 2023. 2arxiv.org/abs/2309.09812
- [40]Thomas A Wynn and Thirumai R Ramalingam. Mechanisms of fibrosis: therapeutic translation for fibrotic disease. Nat. Med., 18(7):1028–1040, 2012. 2DOI
- [41]Xintian Yang, Tongxin Li, Qin Su, Yaling Liu, Chenxi Kang, Yong Lyu, Lina Zhao, Yongzhan Nie, and Yanglin Pan. Application of large language models in disease diagnosis and treatment. Chin. Med. J., 138(02):130–142, 2025. 2DOI
- [42]Tianyi Zhang, Varsha Kishore, Felix Wu, Kilian Q Weinberger, and Yoav Artzi. Bertscore: Evaluating text generation with bert. arXiv preprint arXiv:1904.09675, 2019. 4, 14arxiv.org/abs/1904.09675
- [43]Weitong Zhang, Mengyun Qiao, Chengqi Zhang, Steven Niederer, Paul M Matthews, Wenjia Bai, and Bernhard Kainz. Multi-agent reasoning for cardiovascular imaging phenotype analysis. In International Conference on Medi cal Image Computing and Computer-Assisted Intervention, pages 429–439. Springer, 2025. 2
- [44]Jingming Zhu, Songyang Zhang, Xinyu Fang, Haodong Duan, Dahua Lin, and Kai Chen. Prosa: Assessing and understanding the prompt sensitivity of llms. In Findings of the Association for Computational Linguistics: EMNLP 2024, pages 1950–1976, 2024. 2
