betaFeedback

WorksComputer Science

How Well Do Generalist CT Foundation Models Differentiate Liver Tumors?

Abstract

Generalist CT foundation models detect many findings with high performance, but their ability to differentiate detected lesions remains largely unevaluated. This retrospective study evaluated two generalist models, RADAR and Merlin, for differentiating liver tumors using a public single-institution multiphase CT dataset (517 cases: hepatocellular carcinoma, intrahepatic cholangiocarcinoma, breast and colorectal cancer liver metastases, and hemangioma). The models were evaluated in a zero-shot setting using their existing outputs and with linear probing, in which only a linear classifier was trained on frozen image features (stratified fivefold cross-validation repeated five times). In the zero-shot setting, RADAR achieved an accuracy of 54.4% (95% CI: 49.9, 58.8) and a κκ of 0.37 for four-class differentiation at the portal venous phase and predicted 31% (129 of 421) of malignant tumors as hemangioma. With linear probing of four-phase features, its accuracy improved to 79.9% (95% CI: 76.6, 83.4) with a κκ of 0.72, and 1% (five of 421) of malignant tumors were predicted as hemangioma; Merlin achieved an accuracy of 60.9% (95% CI: 56.9, 65.0). Generalist CT foundation models showed limited zero-shot ability to differentiate liver tumors, which improved substantially with linear probing alone.

Introduction

Generalist CT foundation models trained on large collections of CT images and radiology reports can address many findings with a single model [1, 2, 3, 4, 5]. RADAR showed high performance in detecting 146 findings at contrast-enhanced abdominal CT and outperformed most of 26 radiologists in a reader study, and it has been reported as an “expert-level” generalist artificial intelligence (AI) [4].

However, a key role of radiologists is to provide a differential diagnosis for the lesions they detect. Liver tumors are a typical example: hepatocellular carcinoma, intrahepatic cholangiocarcinoma, liver metastasis, and hemangioma require different management, and misclassifying a malignant tumor as hemangioma may delay treatment. Multiphase contrast-enhanced CT is the standard examination, with enhancement across phases providing key clues [6, 7, 8], and previous AI approaches have mainly used models trained specifically for this task [9, 10]. In contrast, generalist models have mainly been evaluated for detecting individual findings; for example, the evaluation of RADAR consisted of independent binary decisions on each finding, not multiclass differentiation among candidate diagnoses [4, 11]. The ability to identify what a detected lesion is has rarely been evaluated, although AI should be evaluated in its intended clinical task [12].

The purpose of this study was to assess how well two generalist CT foundation models differentiate liver tumors at multiphase CT, both with their existing outputs (zero-shot) and with linear probing [13, 14], in which only a linear classifier is trained on frozen image features. Contrasting the two settings clarifies how much differential diagnostic ability the models have and to what extent their existing outputs draw it out.

Materials and Methods

Dataset

This retrospective study used only public, de-identified data, and institutional review board approval was not required. We used MCT-LTDiag (Harvard Dataverse, doi:10.7910/DVN/S3RW15) [15], a single-institution dataset of 517 patients with four-phase contrast-enhanced CT (noncontrast [NC], arterial [AP], portal venous [PVP], and delayed [DP]) collected between 2014 and 2024: hepatocellular carcinoma (HCC; n=103n = 103), intrahepatic cholangiocarcinoma (ICC; n=100n = 100), breast cancer liver metastasis (BCLM; n=115n = 115), colorectal cancer liver metastasis (CRLM; n=103n = 103), and hepatic hemangioma (HH; n=96n = 96). All cases were included. Tumor masks provided with the dataset were not used.

Models

RADAR [4] is a vision-language model trained on approximately 400 000 contrast-enhanced abdominal CT examinations; it builds organ-level features and outputs scores for 146 findings, including hepatocellular carcinoma, cholangiocarcinoma, metastasis, and hemangioma. Its training data came from a different institution (the First Affiliated Hospital, Zhejiang University School of Medicine) [4], so this was an external evaluation. Merlin [3] is a vision-language model trained on 15 331 abdominal CT scans from a single academic medical center (97% portal venous phase) with multitask supervision from diagnosis codes and organ-wise radiology report sections; its inflated three-dimensional ResNet152 summarizes the abdomen into a 2048-dimensional feature and outputs probabilities for 1692 phenotype codes (phecodes). We used the publicly released weights without further training. Because both models accept a single phase, each phase was processed independently using the official preprocessing (Appendix E1).

Zero-Shot Evaluation

To evaluate native performance without domain adaptation, we used the model outputs as they are. For RADAR, the finding with the highest score among the four liver tumor findings was taken as the prediction. For Merlin, the highest probability among phecodes 155.1 (HCC), 155 (ICC), and 198.4 (metastasis) was taken as the prediction (155 overlaps with 155.1; both are proxy labels); because no liver-specific phecode for hemangioma was available, the 421 cases without HH were evaluated.

Linear Probing

Linear probing evaluates the ability obtainable by adapting only the classification stage. We used the 2048-dimensional Merlin feature and the 256-dimensional RADAR liver organ feature, for each phase and for the concatenation of the four phases. An L2-regularized logistic regression for the five tumor types was evaluated with stratified fivefold cross-validation repeated five times, with standardization and regularization strength determined within training folds only (Appendix E1).

Statistical Analysis

Following the benchmark of the original dataset publication [15], we evaluated four-class differentiation with BCLM and CRLM combined as liver metastasis (LM), reporting per-class specificity, sensitivity, F1 score, and area under the receiver operating characteristic curve (AUC) and overall accuracy and Cohen κ\kappa. For linear probing, the BCLM and CRLM probabilities were summed, and out-of-fold probabilities were averaged over the five repeats. For Merlin zero-shot, the same metrics were computed for three-class differentiation excluding HH. The 95% CIs were obtained by case-level bootstrapping (2000 resamples). Changes from zero-shot evaluation to linear probing were estimated by paired bootstrapping (Appendix E1). Analyses used scikit-learn 1.7.2; code is available at https://github.com/yamagishi0824/liver-tumor-ddx-ct-fm.

Results

Patients

The mean age was 55.8 years ±\pm 12.1 (SD), and 308 of 517 patients (60%) were female; demographic characteristics by tumor type are shown in Table S1.

Tumor typeNo. of casesAge (mean ±\pm SD), yFemaleCirrhosis
HCC10359.2 ±\pm 10.721 (20)68 (66)
ICC10060.3 ±\pm 10.453 (53)14 (14)
BCLM11552.9 ±\pm 12.2115 (100)2 (2)
CRLM10360.6 ±\pm 10.443 (42)0 (0)
HH9645.7 ±\pm 9.676 (79)1 (1)
All51755.8 ±\pm 12.1308 (60)85 (16)

Table S1. Patient Characteristics

Zero-Shot Performance

In zero-shot evaluation (Table 1), RADAR achieved an accuracy of 54.4% (95% CI: 49.9, 58.8) and a κ\kappa of 0.37 at PVP, the best-performing phase, and 30.8%–41.6% at the other phases (Table S4). Its sensitivity was 84.4% for HH but 22.0% for ICC, and 31% (129 of 421) of malignant tumors were predicted as hemangioma. Merlin achieved an accuracy of 45.6% (95% CI: 40.9, 50.4) and a κ\kappa of 0.14 for three-class differentiation excluding HH at PVP (52% when always predicting metastasis) (Table S5). Normalized zero-shot scores were spread across the four findings for every tumor type, whereas linear-probe probabilities concentrated on the correct type (Figure 3).

RADAR Zero-shotRADAR Linear probeMerlin Zero-shotMerlin Linear probe
HCCSpecificity94.20 (91.83–96.53)95.65 (93.53–97.52)78.62 (73.98–82.90)89.61 (86.67–92.55)
HCCSensitivity49.51 (39.99–59.09)76.70 (68.18–84.62)43.69 (34.04–53.40)47.57 (37.86–57.01)
HCCF157.30 (47.83–65.66)79.00 (72.34–84.76)41.67 (32.71–49.58)50.26 (40.91–58.29)
HCCAUC67.13 (60.51–73.73)95.38 (93.47–97.14)72.82 (66.90–78.04)85.02 (80.76–88.64)
ICCSpecificity97.36 (95.72–98.82)90.65 (87.77–93.26)72.27 (67.37–77.00)92.57 (89.92–94.87)
ICCSensitivity22.00 (13.89–30.63)60.00 (50.00–69.73)22.00 (14.02–30.61)26.00 (17.07–34.74)
ICCF133.08 (22.22–43.75)60.30 (51.89–68.06)20.85 (13.66–28.29)33.12 (22.70–42.43)
ICCAUC64.82 (57.91–70.93)87.18 (83.24–90.81)62.05 (55.55–68.02)67.59 (61.28–73.53)
HHSpecificity69.36 (64.90–73.62)98.81 (97.66–99.76)—90.74 (87.96–93.51)
HHSensitivity84.38 (76.92–91.14)92.71 (87.23–97.75)—63.54 (53.92–73.96)
HHF152.94 (45.58–59.33)93.68 (89.84–96.94)—62.24 (53.97–70.09)
HHAUC60.87 (55.50–66.23)99.51 (98.87–99.90)—89.34 (85.92–92.47)
LMSpecificity75.92 (71.16–80.72)85.95 (82.09–89.78)64.53 (57.84–70.92)70.23 (64.75–75.15)
LMSensitivity58.26 (51.35–64.87)84.86 (80.18–89.30)57.34 (50.72–63.51)82.11 (76.92–86.85)
LMF160.91 (55.19–66.52)83.15 (79.35–86.67)60.24 (54.50–65.32)73.66 (69.03–77.94)
LMAUC71.33 (66.63–75.89)93.22 (90.96–95.32)36.74 (31.18–42.03)84.99 (81.56–88.04)
OverallAccuracy54.35 (49.90–58.80)79.88 (76.60–83.37)45.61 (40.86–50.36)60.93 (56.86–65.00)
OverallCohen’s κ0.37 (0.31–0.42)0.72 (0.67–0.76)0.14 (0.07–0.21)0.43 (0.37–0.49)

Note.—Values are percentages with 95% CIs in parentheses, except Cohen κ (0–1). Zero-shot: portal venous phase; RADAR, scores of four liver tumor findings; Merlin, phecode probabilities (HCC, 155.1; ICC, 155; LM, 198.4). Because no liver-specific phecode for hemangioma was available, Merlin zero-shot results are for three-class differentiation in the 421 cases without HH; all other columns are four-class results in 517 cases. Linear probe: four-phase concatenation; RADAR, liver organ features (256 dimensions); Merlin, 2048-dimensional image features; the BCLM and CRLM probabilities were summed. LM = liver metastasis (BCLM + CRLM).

Table 1. Four-Class Differentiation in Zero-Shot Evaluation and with Linear Probing

Distribution of RADAR outputs by true tumor type

Figure 3. Distribution of RADAR outputs by true tumor type. Top: zero-shot scores of the four liver tumor findings at the portal venous phase, normalized to sum to 1 in each case for display; the original scores are computed independently for each finding and are not probabilities. Bottom: linear-probe probabilities (four-phase concatenation; BCLM and CRLM probabilities summed). Boxes indicate the interquartile range and median, whiskers 1.5 times the interquartile range, and dots individual cases. LM = liver metastasis.

NCAPPVPDP
HCCSpecificity92.51 (89.93–95.16)95.41 (93.32–97.35)94.20 (91.83–96.53)89.61 (86.57–92.42)
HCCSensitivity43.69 (33.70–53.19)34.95 (25.55–43.96)49.51 (39.99–59.09)35.92 (26.36–45.14)
HCCF150.28 (40.96–59.30)45.57 (35.15–54.44)57.30 (47.83–65.66)40.44 (31.05–48.81)
HCCAUC60.34 (53.63–66.82)66.39 (60.31–72.62)67.13 (60.51–73.73)66.45 (60.50–72.40)
ICCSpecificity96.64 (94.78–98.32)92.09 (89.49–94.51)97.36 (95.72–98.82)91.37 (88.63–93.93)
ICCSensitivity21.00 (13.04–29.21)30.00 (21.21–38.94)22.00 (13.89–30.63)26.00 (17.78–34.26)
ICCF131.11 (20.44–40.91)36.81 (27.58–45.56)33.08 (22.22–43.75)32.10 (22.53–40.54)
ICCAUC64.99 (58.94–70.67)64.44 (57.67–70.50)64.82 (57.91–70.93)61.00 (54.32–67.48)
HHSpecificity61.28 (56.51–65.78)44.89 (40.05–49.30)69.36 (64.90–73.62)44.42 (39.62–48.96)
HHSensitivity41.67 (32.00–51.43)82.29 (74.26–89.69)84.38 (76.92–91.14)57.29 (47.87–67.03)
HHF126.76 (20.28–33.34)38.82 (32.57–44.65)52.94 (45.58–59.33)28.57 (22.53–34.50)
HHAUC50.72 (44.44–56.30)79.57 (47.83–83.83)60.87 (55.50–66.23)69.71 (64.12–75.02)
LMSpecificity68.56 (63.23–73.46)84.62 (80.72–88.50)75.92 (71.16–80.72)84.95 (80.87–89.00)
LMSensitivity50.00 (43.64–56.54)19.27 (14.08–24.34)58.26 (51.35–64.87)18.81 (13.73–23.98)
LMF151.78 (46.00–57.34)27.45 (20.76–33.75)60.91 (55.19–66.52)26.97 (20.29–33.54)
LMAUC59.70 (54.56–64.85)32.79 (26.39–37.92)71.33 (66.63–75.89)35.11 (30.20–40.20)
OverallAccuracy41.59 (37.33–45.84)36.17 (31.72–40.04)54.35 (49.90–58.80)30.75 (26.89–34.82)
OverallCohen’s κ0.19 (0.13–0.24)0.17 (0.12–0.22)0.37 (0.31–0.42)0.10 (0.05–0.15)

Table S4. RADAR Zero-Shot Four-Class Differentiation by Phase

NCAPPVPDP
HCCSpecificity63.52 (57.96–68.81)66.35 (61.09–71.43)78.62 (73.98–82.90)77.36 (72.76–81.73)
HCCSensitivity66.02 (57.00–75.00)51.46 (41.38–61.47)43.69 (34.04–53.40)39.81 (30.39–50.00)
HCCF147.39 (40.00–54.23)40.30 (32.63–47.83)41.67 (32.71–49.58)37.96 (29.09–46.01)
HCCAUC73.73 (68.05–79.06)61.54 (55.25–67.83)72.82 (66.90–78.04)65.27 (59.36–71.12)
ICCSpecificity76.95 (72.12–81.60)71.65 (66.67–76.53)72.27 (67.37–77.00)65.42 (59.82–70.37)
ICCSensitivity27.00 (18.09–36.08)25.00 (16.48–34.07)22.00 (14.02–30.61)30.00 (20.79–39.05)
ICCF126.87 (18.65–35.19)23.15 (15.69–30.96)20.85 (13.66–28.29)24.90 (17.35–31.94)
ICCAUC57.77 (51.28–63.54)58.73 (52.44–64.80)62.05 (55.55–68.02)61.65 (54.99–67.61)
LMSpecificity76.35 (70.37–82.07)69.95 (63.40–76.38)64.53 (57.84–70.92)66.01 (59.11–72.82)
LMSensitivity40.37 (33.48–47.03)38.53 (32.14–44.68)57.34 (50.72–63.51)44.95 (38.23–51.31)
LMF149.72 (42.61–55.87)46.28 (39.77–52.12)60.24 (54.50–65.32)50.91 (44.33–56.71)
LMAUC40.42 (34.98–46.20)45.75 (40.25–51.37)36.74 (31.18–42.03)38.62 (33.20–44.29)
OverallAccuracy43.47 (38.48–48.46)38.48 (33.97–43.00)45.61 (40.86–50.36)40.14 (35.15–44.89)
OverallCohen’s κ0.15 (0.08–0.22)0.07 (0.01–0.14)0.14 (0.07–0.21)0.08 (0.01–0.15)

Table S5. Merlin Zero-Shot Three-Class Differentiation (HH Excluded) by Phase

Linear-Probe Performance

With linear probing of four-phase features (Table 1), RADAR achieved an accuracy of 79.9% (95% CI: 76.6, 83.4) and a κ\kappa of 0.72 (95% CI: 0.67, 0.76), with AUCs of 0.87 or higher for all classes; Merlin achieved 60.9% (95% CI: 56.9, 65.0) and 0.43 (95% CI: 0.37, 0.49). Four-phase concatenation improved accuracy more for RADAR than for Merlin (Table S2).

FeaturesPhaseAccuracyBalanced accuracyMacro AUCmAPBenign vs malignant AUCHCC vs HH AUC
RADAR (liver, pre-projection, 256-d)NC0.608 ±\pm 0.0290.611 ±\pm 0.0300.857 ±\pm 0.0150.653 ±\pm 0.0130.937 ±\pm 0.0060.945 ±\pm 0.004
RADAR (liver, pre-projection, 256-d)AP0.655 ±\pm 0.0370.660 ±\pm 0.0370.890 ±\pm 0.0180.690 ±\pm 0.0110.978 ±\pm 0.0060.977 ±\pm 0.006
RADAR (liver, pre-projection, 256-d)PVP0.650 ±\pm 0.0410.655 ±\pm 0.0400.892 ±\pm 0.0190.699 ±\pm 0.0140.994 ±\pm 0.0010.994 ±\pm 0.001
RADAR (liver, pre-projection, 256-d)DP0.603 ±\pm 0.0460.608 ±\pm 0.0460.860 ±\pm 0.0190.637 ±\pm 0.0070.963 ±\pm 0.0060.971 ±\pm 0.005
RADAR (liver, pre-projection, 256-d)4-phase concat0.706 ±\pm 0.0290.711 ±\pm 0.0290.915 ±\pm 0.0130.746 ±\pm 0.0160.992 ±\pm 0.0020.986 ±\pm 0.002
Merlin (2048-d)NC0.523 ±\pm 0.0520.521 ±\pm 0.0520.816 ±\pm 0.0240.538 ±\pm 0.0120.869 ±\pm 0.0050.917 ±\pm 0.007
Merlin (2048-d)AP0.515 ±\pm 0.0370.515 ±\pm 0.0370.804 ±\pm 0.0240.534 ±\pm 0.0050.857 ±\pm 0.0060.898 ±\pm 0.005
Merlin (2048-d)PVP0.537 ±\pm 0.0420.535 ±\pm 0.0430.830 ±\pm 0.0240.575 ±\pm 0.0090.881 ±\pm 0.0060.927 ±\pm 0.007
Merlin (2048-d)DP0.540 ±\pm 0.0390.537 ±\pm 0.0390.819 ±\pm 0.0220.557 ±\pm 0.0190.871 ±\pm 0.0160.891 ±\pm 0.010
Merlin (2048-d)4-phase concat0.562 ±\pm 0.0480.559 ±\pm 0.0480.838 ±\pm 0.0230.592 ±\pm 0.0110.887 ±\pm 0.0050.928 ±\pm 0.003

Table S2. Linear-Probe Results by Phase (Primary Features)

Linear probing increased accuracy by 25.5 percentage points (95% CI: 20.9, 30.2) for RADAR and by 19.2 (95% CI: 13.5, 24.9) for Merlin (matched three-class setting), mainly through LM sensitivity (Figure 1). Similar improvements were observed with PVP features only (Table S6). In five-class classification (Figure 2), RADAR accuracy was 81% for HCC, 65% for ICC, 60% for BCLM, 67% for CRLM, and 93% for HH, with remaining errors mainly among BCLM, CRLM, and ICC. Malignant tumors were predicted as HH in five of 421 cases (1%) by RADAR and 44 of 421 (10%) by Merlin, and, in the HCC-versus-HH comparison (Figure 2, bottom), HCC was predicted as HH in five of 103 (5%) and 18 of 103 (17%), respectively.

Change from zero-shot evaluation to linear probing in sensitivity, specificity, F1 score, and one-vs-rest AUC

Figure 1. Change from zero-shot evaluation to linear probing in (A) sensitivity, (B) specificity, and (C) F1 score, and (D) one-vs-rest AUC for each tumor type, for RADAR (blue) and Merlin (orange). Points indicate the difference (linear probe minus zero-shot, in percentage points; AUC multiplied by 100), and lines indicate 95% CIs from paired case-level bootstrapping (2000 resamples). RADAR: four classes in all 517 cases. Merlin: because no liver-specific phecode for hemangioma was available, both conditions were evaluated as three-class differentiation (HCC, ICC, and LM) in the 421 cases without HH; the linear-probe prediction was the largest of the HCC, ICC, and LM probabilities of the five-class model, which had been trained including HH. The two conditions still differ in input phases (zero-shot, portal venous phase only; linear probe, four-phase concatenation) and in how predictions are obtained (finding scores or phecode probabilities vs a classifier trained with cross-validation), and the change reflects both; the results were similar with portal venous phase features only (Table S6). LM = liver metastasis.

Confusion matrices for linear probing with four-phase concatenation

Figure 2. Confusion matrices for linear probing with four-phase concatenation. Rows indicate the reference standard and columns the predictions; shading indicates row-normalized percentages. Top: five-class classification (left, RADAR; right, Merlin). Bottom: benign (HH) versus malignant, and HCC versus HH (HCC and HH cases only). Each cell shows the number of cases and the row percentage. In the HCC versus HH matrices, the prediction is the larger of the HCC and HH probabilities from the same five-class model.

RADAR PVP onlyRADAR Four-phaseMerlin PVP onlyMerlin Four-phase
HCCSensitivity+21.4 (11.8 to 31.9)+27.2 (17.3 to 38.0)+7.8 (−3.9 to 18.9)+8.7 (−3.2 to 20.2)
HCCSpecificity−1.0 (−3.4 to 1.7)+1.4 (−1.0 to 3.9)+6.6 (1.7 to 11.7)+9.1 (4.5 to 13.9)
HCCF1+14.3 (6.5 to 23.0)+21.7 (13.5 to 31.0)+10.6 (1.1 to 19.9)+13.4 (3.6 to 22.9)
HCCAUC+26.8 (20.2 to 33.1)+28.2 (21.8 to 34.8)+9.4 (4.3 to 14.1)+10.8 (5.6 to 16.0)
ICCSensitivity+27.0 (17.5 to 36.5)+38.0 (27.8 to 48.5)−1.0 (−11.6 to 10.0)+5.0 (−6.3 to 15.6)
ICCSpecificity−7.2 (−10.0 to −4.6)−6.7 (−9.5 to −4.2)+17.8 (11.9 to 23.6)+16.8 (10.6 to 22.9)
ICCF1+18.5 (8.7 to 29.0)+27.2 (16.6 to 38.3)+6.6 (−4.4 to 17.6)+12.5 (0.8 to 22.9)
ICCAUC+20.7 (14.1 to 27.0)+22.4 (15.0 to 29.3)+3.6 (−3.2 to 10.5)+3.8 (−2.5 to 10.0)
HHSensitivity+6.2 (−2.1 to 15.1)+8.3 (−1.0 to 17.6)——
HHSpecificity+30.2 (25.8 to 34.8)+29.5 (25.1 to 34.1)——
HHF1+41.1 (34.1 to 48.4)+40.7 (34.2 to 48.2)——
HHAUC+38.6 (33.5 to 43.9)+38.6 (33.5 to 43.9)——
LMSensitivity+26.1 (17.9 to 33.9)+26.6 (18.5 to 34.4)+29.4 (22.1 to 37.0)+30.7 (23.2 to 38.2)
LMSpecificity+6.4 (0.7 to 12.1)+10.0 (4.9 to 15.5)−3.4 (−12.6 to 5.6)−1.0 (−10.1 to 7.7)
LMF1+20.0 (13.8 to 26.0)+22.2 (16.4 to 28.3)+17.5 (11.8 to 23.7)+19.1 (13.5 to 24.9)
LMAUC+21.5 (17.2 to 26.0)+21.9 (17.5 to 26.5)+46.3 (39.0 to 53.5)+47.2 (39.5 to 54.4)
OverallAccuracy+21.7 (17.2 to 26.3)+25.5 (20.9 to 30.2)+16.9 (11.4 to 22.3)+19.2 (13.5 to 24.9)

Table S6. Change from Zero-Shot Evaluation to Linear Probing with Portal Venous Phase Features Only

Discussion

Generalist CT foundation models showed limited zero-shot ability to differentiate liver tumors, but linear probing of their frozen features substantially improved differentiation and reduced malignant tumors predicted as hemangioma from 31% to 1% with RADAR.

The limited zero-shot performance likely reflects how these models were trained. RADAR determines independently, for each finding, whether it is present, rather than choosing among candidate diagnoses; its diagnostic findings, such as hepatocellular carcinoma and hemangioma, are also handled as independent decisions. Each score is computed independently from presence and absence prompts and is not a relative probability among candidate diagnoses [11], and its absolute value requires calibration before clinical interpretation [16]; simply choosing the highest score biased predictions toward hemangioma (Figure 3). This quantitatively supports, in 517 cases, the observation that finding-level scores do not form a coherent differential diagnosis [11]. In contrast, linear probing substantially improved differentiation, suggesting that finding-focused training had latently encoded the information needed to differentiate liver tumors but that the existing outputs did not draw it out. The best result in the benchmark of the original dataset publication, which included deep learning models such as UniFormer [17], was an accuracy of 70.6% with a κ\kappa of 0.56 (a radiomics-based support vector machine) [15]; the linear-probe result of RADAR was numerically higher, but direct comparison is limited because the benchmark used selected tumors (10–150 mm), tumor-mask crops, and region-level evaluation. The smaller gain with Merlin may reflect differences in pretraining data, supervision, and feature extraction (whole-abdomen vs organ-level features), although their contributions were not separated.

Both models accept a single phase, so phase features were concatenated after independent inference. Temporal enhancement patterns are key to differentiating liver tumors, but the models are not designed to capture them, and even RADAR still confused ICC with metastases. Models that handle multiple phases simultaneously will be increasingly important [18].

Our study had limitations. First, data were from a single institution. Second, tumor detection was not evaluated. Third, the results may not apply to other organs or diseases. Fourth, because we focused on the native performance of frozen features, the basis of model decisions was not examined, and shortcuts based on features other than the tumor cannot be ruled out [19, 20].

In conclusion, generalist CT foundation models showed limited zero-shot ability to differentiate liver tumors. However, adapting only a linear classifier on top of the frozen image encoder substantially improved this ability, to values numerically higher than those reported for task-specific models under different conditions. To use these models to support differential diagnosis, evaluation of differential diagnostic ability is needed in addition to finding detection, and this ability can be drawn out with lightweight adaptation.

References

Supplemental Material

Appendix E1: Supplemental Methods

Dataset. According to the original publication [15], cases were collected at Peking Union Medical College Hospital between August 2014 and January 2024 with institutional ethics approval (approval no. K23N0520; informed consent waived). Inclusion required a pathologic or clinical diagnosis and complete contrast-enhanced CT; cases with poor image quality, unclear tumor margins, or a maximum tumor diameter smaller than 1 cm were excluded. The four phases were registered by the dataset providers.

Models and preprocessing. RADAR: publicly released pretrained weights (checkpoint_radar_pretrain.pth) and the official inference code. Volumes were reoriented from LPS to LAS to match the official demonstration case, resampled to 1×1×51 \times1 \times5 mm, and clipped to −300 to 400 HU with normalization to 0–1. The 18 living findings include hepatocellular carcinoma, intrahepatic cholangiocarcinoma, metastasis, and hemangioma. Merlin: publicly released weights (stanfordmimi/Merlin, merlin-vlm 0.0.7) with the package preprocessing (reorientation to RAS, resampling to 1.5×1.5×31.5 \times1.5 \times3 mm, clipping to −1000 to 1000 HU with normalization to 0–1, and padding and center cropping to 224×224×160224 \times224 \times160). Inference was performed on a single NVIDIA GH200 GPU (PyTorch 2.14.1, MONAI 1.6.1; transformers 4.25.1 for RADAR).

Feature extraction. Merlin: the globally pooled 2048-dimensional feature (primary), the 512-dimensional contrastive embedding, and the 1692 phecode probabilities. RADAR: the inference code was minimally modified to save liver organ features before projection into the text space (256 dimensions; primary) and after projection and normalization (256 dimensions).

Change from zero-shot evaluation. Changes from zero-shot evaluation to linear probing were estimated by paired bootstrapping, using the same resampled cases for both conditions; for Merlin, both conditions were evaluated in the 421 cases without HH, with the linear-probe prediction restricted to the HCC, ICC, and LM probabilities.

Linear probing. Five-class L2-regularized logistic regression with standardization. Outer evaluation: stratified fivefold cross-validation repeated five times (25 folds). In each outer training fold, the scaler was fitted to the entire outer training fold before selecting C from 11 candidate values (10−410^{-4} to 10110^{1}) by inner fivefold cross-validation using balanced accuracy. The four phases of a case were treated as one sample.

Supplementary metrics. Five-class accuracy, balanced accuracy, macro AUC, and mean average precision (mAP) are reported as mean ±\pm SD across 25 test folds. Benign versus malignant: HH regarded as benign, with 1−P(HH)1 - P(\mathrm{HH}) as the score. HCC versus HH: HCC and HH cases only, with P(HCC)/(P(HCC)+P(HH))P(\mathrm{HCC}) / (P(\mathrm{HCC}) + P(\mathrm{HH})) as the score. In confusion matrices, the class with the highest probability (or the larger of P(HCC)P(\mathrm{HCC}) and P(HH)P(\mathrm{HH})) was taken as the prediction.

Note.—Data are numbers of patients with percentages in parentheses unless otherwise indicated, summarized from the patient information provided with the dataset. Cirrhosis includes entries recorded as “Y” with an etiology; one BCLM case with uncertain status was counted as no cirrhosis.

Note.—Five-class classification. Values are mean ±\pm SD across 25 test folds (stratified fivefold cross-validation repeated five times). Benign vs malignant AUC: HH regarded as benign, with 1−P(HH)1 - P(\mathrm{HH}) as the score. HCC vs HH AUC: HCC and HH cases only, with P(HCC)/(P(HCC)+P(HH))P(\mathrm{HCC}) / (P(\mathrm{HCC}) + P(\mathrm{HH})) as the score. With random selection among five classes, accuracy is 0.20 and mAP is approximately 0.20.

FeaturesPhaseAccuracyBalanced accuracyMacro AUCmAPBenign vs malignant AUCHCC vs HH AUC
RADAR (liver, post-projection, 256-d)NC0.610 ± 0.0280.612 ± 0.0290.855 ± 0.0180.648 ± 0.0170.935 ± 0.0060.952 ± 0.007
RADAR (liver, post-projection, 256-d)AP0.647 ± 0.0470.653 ± 0.0460.879 ± 0.0220.672 ± 0.0130.971 ± 0.0050.974 ± 0.003
RADAR (liver, post-projection, 256-d)PVP0.650 ± 0.0480.654 ± 0.0480.890 ± 0.0230.688 ± 0.0220.989 ± 0.0030.991 ± 0.002
RADAR (liver, post-projection, 256-d)DP0.585 ± 0.0380.590 ± 0.0380.843 ± 0.0220.601 ± 0.0100.942 ± 0.0070.960 ± 0.008
RADAR (liver, post-projection, 256-d)4-phase concat0.693 ± 0.0310.698 ± 0.0310.912 ± 0.0150.747 ± 0.0240.989 ± 0.0030.987 ± 0.003
Merlin (512-d)NC0.503 ± 0.0520.502 ± 0.0520.800 ± 0.0270.508 ± 0.0190.859 ± 0.0050.907 ± 0.006
Merlin (512-d)AP0.495 ± 0.0360.495 ± 0.0360.791 ± 0.0260.511 ± 0.0080.844 ± 0.0090.889 ± 0.005
Merlin (512-d)PVP0.515 ± 0.0440.513 ± 0.0440.818 ± 0.0220.552 ± 0.0070.875 ± 0.0030.919 ± 0.006
Merlin (512-d)DP0.528 ± 0.0450.526 ± 0.0460.813 ± 0.0240.546 ± 0.0190.863 ± 0.0130.884 ± 0.009
Merlin (512-d)4-phase concat0.542 ± 0.0440.540 ± 0.0450.824 ± 0.0260.561 ± 0.0250.879 ± 0.0070.919 ± 0.006
Merlin (phecode probabilities, 1692)NC0.491 ± 0.0370.488 ± 0.0370.781 ± 0.0200.485 ± 0.0060.854 ± 0.0040.908 ± 0.006
Merlin (phecode probabilities, 1692)AP0.461 ± 0.0390.460 ± 0.0400.748 ± 0.0290.455 ± 0.0070.816 ± 0.0050.865 ± 0.006
Merlin (phecode probabilities, 1692)PVP0.502 ± 0.0420.500 ± 0.0410.795 ± 0.0260.504 ± 0.0110.852 ± 0.0040.902 ± 0.006
Merlin (phecode probabilities, 1692)DP0.475 ± 0.0420.472 ± 0.0430.770 ± 0.0210.474 ± 0.0070.832 ± 0.0060.869 ± 0.009
Merlin (phecode probabilities, 1692)4-phase concat0.508 ± 0.0430.505 ± 0.0420.797 ± 0.0260.505 ± 0.0050.848 ± 0.0070.878 ± 0.008

Table S3. Linear-Probe Results by Phase (Secondary Features)

Note.—Same format as tbl:2.

Note.—Values are percentages with 95% CIs in parentheses, except Cohen κ (0–1). The finding with the highest score among four findings (hepatocellular carcinoma, intrahepatic cholangiocarcinoma, metastasis, and hemangioma) was taken as the prediction. AUCs are one-vs-rest AUCs of the corresponding finding score for each class.

Note.—Values are percentages with 95% CIs in parentheses, except Cohen κ (0–1). Because no liver-specific phecode for hemangioma was available, the 421 cases without HH were evaluated, with the highest probability among phecodes 155.1 (HCC), 155 (ICC), and 198.4 (LM) taken as the prediction.

Note.—Values are changes from zero-shot evaluation (PVP) to linear probing (percentage points; AUC multiplied by 100), with 95% CIs from paired bootstrapping in parentheses. PVP only, linear probing with portal venous phase features only, so that the input phase matches the zero-shot evaluation; four-phase, the values shown in Figure 1. RADAR, four classes in 517 cases; Merlin, three classes in the 421 cases without HH. LM = liver metastasis.

References

  1. [1]Moor M, Banerjee O, Abad ZSH, et al. Foundation models for generalist medical artificial intelligence. Nature 2023;616(7956):259–265. doi:10.1038/s41586-023-05881-4DOI
  2. [2]Hamamci IE, Er S, Wang C, et al. Generalist foundation models from a multimodal dataset for 3D computed tomography. Nat Biomed Eng 2026;10(8):1610–1628. doi:10.1038/s41551-025-01599-yDOI
  3. [3]Blankemeier L, Kumar A, Cohen JP, et al. Merlin: a computed tomography vision–language foundation model and dataset. Nature 2026;652:1318–1328. doi:10.1038/s41586-026-10181-8DOI
  4. [4]Zhang Q, Zhang J, Cao W, et al. An expert-level generalist AI for abdominal CT diagnosis. Science 2026;393:eaec6129. doi:10.1126/science.aec6129DOI
  5. [5]Yamamoto K, Kikuchi T. TotalFM: an organ-separated 3D-CT foundation model leveraging large-scale routine clinical radiology data. arXiv 2601.00260 [preprint]. Posted January 1, 2026. doi:10.48550/arXiv.2601.00260arxiv.org/abs/2601.00260
  6. [6]Singal AG, Llovet JM, Yarchoan M, et al. AASLD Practice Guidance on prevention, diagnosis, and treatment of hepatocellular carcinoma. Hepatology 2023;78(6):1922–1965. doi:10.1097/HEP.0000000000000466DOI
  7. [7]Chernyak V, Fowler KJ, Kamaya A, et al. Liver Imaging Reporting and Data System (LI-RADS) version 2018: imaging of hepatocellular carcinoma in at-risk patients. Radiology 2018;289(3):816–830. doi:10.1148/radiol.2018181494DOI
  8. [8]Nino-Murcia M, Olcott EW, Jeffrey RB Jr, et al. Focal liver lesions: pattern-based classification scheme for enhancement at arterial phase CT. Radiology 2000;215(3):746–751. doi:10.1148/radiology.215.3.r00jn03746
  9. [9]Yasaka K, Akai H, Abe O, Kiryu S. Deep learning with convolutional neural network for differentiation of liver masses at dynamic contrast-enhanced CT: a preliminary study. Radiology 2018;286(3):887–896. doi:10.1148/radiol.2017170706DOI
  10. [10]Ying H, Liu X, Zhang M, et al. A multicenter clinical AI system study for detection and diagnosis of focal liver lesions. Nat Commun 2024;15:1131. doi:10.1038/s41467-024-45325-9DOI
  11. [11]Choi JI, Kim H, Park SH. What “expert-level” generalist abdominal CT artificial intelligence reveals about expert radiologists. Korean J Radiol 2026;27:e102. doi:10.3348/kjr.2026.1767DOI
  12. [12]Park SH, Han K, Jang HY, et al. Methods for clinical evaluation of artificial intelligence algorithms for medical diagnosis. Radiology 2023;306(1):20–31. doi:10.1148/radiol.220182DOI
  13. [13]Alain G, Bengio Y. Understanding intermediate layers using linear classifier probes. arXiv 1610.01644 [preprint]. Posted October 5, 2016. doi:10.48550/arXiv.1610.01644DOI
  14. [14]Radford A, Kim JW, Hallacy C, et al. Learning transferable visual models from natural language supervision. In: Proceedings of the 38th International Conference on Machine Learning. PMLR 2021;139:8748–8763arxiv.org/abs/2103.00020
  15. [15]Wu X, Su H, Hua Y, et al. A multi-phase CT dataset for automated differential diagnosis of liver tumors. Sci Data 2026;13:31. doi:10.1038/s41597-025-06343-4DOI
  16. [16]Myong Y, Jeon S, Sim Y. Uncover this tech term: model calibration in the artificial intelligence era. Korean J Radiol 2026;27:e101. doi:10.3348/kjr.2026.1648DOI
  17. [17]Li K, Wang Y, Gao P, et al. UniFormer: unified transformer for efficient spatiotemporal representation learning. In: International Conference on Learning Representations (ICLR) 2022. arXiv 2201.04676. doi:10.48550/arXiv.2201.04676DOI
  18. [18]Zhu S, Zou M, Wu Q, et al. STD-Net: a spatio-temporal decoupling network for multiphasic liver lesion segmentation and characterization. NPJ Digit Med 2026;9:13. doi:10.1038/s41746-025-02181-1DOI
  19. [19]Geirhos R, Jacobsen JH, Michaelis C, et al. Shortcut learning in deep neural networks. Nat Mach Intell 2020;2(11):665–673. doi:10.1038/s42256-020-00257-zDOI
  20. [20]DeGrave AJ, Janizek JD, Lee SI. AI for radiographic COVID-19 detection selects shortcuts over signal. Nat Mach Intell 2021;3(7):610–619. doi:10.1038/s42256-021-00338-7DOI

Paper details

vlnc:W_vdyu5ew2
Version
v1 · latest
Published
Fresh — added to the Hub within the last 30 days
Added
License
CC BY 4.0
Usage

views · 7

downloads · 0

A view counts once per browser per day, a download each time the PDF is requested.

  • Computer Science

Not recorded for this paper

View statement
Looking for related papers…
Contents