In this paper
AbstractIntroductionResultsMisalignment of Existing Metrics with CLARITYApplication to Conformal PredictionDiscussionMethodsStudy design and data sourceReport generationClinical admissibility criteriaRadiologist evaluationAutomatic metric evaluationConformal predictionStatistical analysisEthicsData availabilityCode availabilityAuthor contributionsAcknowledgementsCompeting interestsReferencesWorksComputer Science
A Clinical Evaluation of Conformal Prediction for Chest Radiograph Report Generation
Abstract
While multimodal large language models (MLLMs) have improved automated radiology report generation, they remain prone to hallucinations, necessitating clinically meaningful risk-control methods. Conformal prediction (CP) provides statistical guarantees, yet existing CP approaches rely on uncalibrated automated extractors and fail to account for the clinical relevance of errors. We introduce clinical admissibility criteria (CLARITY), a framework for evaluating radiology reports that explicitly distinguishes clinically consequential errors from minor deviations. To eliminate reliance on automatic judges, we collected radiologist annotations of MLLM-generated reports using CLARITY. By benchmarking CP strategies and automatic evaluation metrics against these expert annotations, we demonstrate that commonly used admissibility proxies systematically fail to detect clinically relevant errors. Our findings reveal that automated admissibility judgments are largely decoupled from true clinical admissibility.
Introduction
Machine learning has achieved striking performance in medical applications ranging from breast cancer screening [1] to cardiovascular risk prediction [2]. Automated radiology report generation has progressed rapidly [3], yet safe clinical deployment requires trustworthy uncertainty estimates. Miscalibration and hallucination remain key failure modes [4, 5]: overconfident or fabricated claims can directly mislead clinical decisions [6]. Conformal prediction (CP) addresses this problem by producing sets of candidate reports with finite-sample guarantees that at least one admissible report is included with a certain target probability [7, 8].
For both calibration and evaluation, CP methods require an admissibility function that determines whether a generated report is acceptable. Existing approaches use proxies such as linguistic metrics, including BLEU [9] and ROUGE [10], or automated clinical extractors such as CheXbert [11] and RadGraph [12]. Linguistic metrics are sensitive to wording and do not directly encode clinical correctness, whereas learned extractors inherit model and reference-report errors and do not represent the asymmetric consequences of different mistakes. Encoding generated and reference reports as label sets also fails to capture the clinical relevance of individual findings or errors in the reference report itself.
Evaluation research has consequently moved toward embedding similarity, structured clinical extraction, and domain-specific factuality measures. Among these approaches, BERTScore compares contextual representations [13]; RadCliQ combines report metrics to better reflect radiologist error assessments [14]; GREEN identifies clinically significant errors in candidate reports [15]; and RaTE-Score and FORTE emphasize clinical entities and structured radiology concepts [16, 17]. Safety-oriented generation methods have similarly included grounding statements to image evidence [18], black-box uncertainty quantification [19], and conformal methods for language and vision-language models [7, 8, 20, 21]`. Any conformal guarantee, however, remains defined relative to its chosen admissibility event.
We address this gap by introducing CLARITY, a radiologist-grounded framework that distinguishes clinically consequential errors from minor omissions and stylistic differences. We use CLARITY to benchmark CP methods and quantify the extent to which automatic metrics align with the resulting operational definition of clinical admissibility.
Our main contributions are:
Clinically grounded admissibility criteria. We introduce CLARITY, a radiologist-grounded framework that assesses report admissibility by distinguishing clinically relevant errors from minor omissions and stylistic differences.
Expert-annotated dataset. We obtain board-certified radiologist annotations for reports generated by a MLLM, releasing the dataset for reproducible benchmarking.
Misalignment of automatic metrics with radiologist evaluation. We systematically compare 14 automatic metric variants with CLARITY and show that their alignment with radiologist-assessed clinical admissibility is metric-dependent and incomplete. In particular, CheXbert positive-label F1, a commonly used admissibility proxy, shows no positive alignment with the radiologist endpoint, while even the strongest metric is not an interchangeable substitute for expert evaluation.
CP benchmarking. We systematically compare CP methods under clinically grounded evaluation, including unsuccessful calibration runs, and characterize the coverage–set-size trade-offs observed under different admissibility definitions.
Results
Our primary experiment tests agreement between existing automatic metrics and the radiologist assessments defined by CLARITY. We then examine the downstream consequences of disagreement by calibrating and evaluating conformal prediction with either CheXbert or CLARITY admissibility.
Misalignment of Existing Metrics with CLARITY
The automatic metrics did not uniformly recover radiologist-assessed admissibility. An empirical exploratory analysis shows that report-level CheXbert positive-label F1 was weakly negatively associated with CLARITY (, unadjusted two-sided ). The primary study-level association was also negative and was not significant after multiplicity correction (, 95% CI -0.339–0.093; Holm-adjusted ). CheXbert therefore provided no evidence of positive alignment with the radiologist endpoint despite its use as an admissibility proxy in prior CP pipelines (Figure 2).
Figure 2. Comparison of CheXbert and CLARITY. Distribution of CheXbert positive-label F1 scores for 970 generated reports from 97 studies, grouped by radiologist-assessed CLARITY admissibility. Points show individual reports.
As shown in Table 1, the degree of alignment varied substantially across metrics. GREEN had the strongest study-level association (, 95% CI 0.562–0.798; Holm-adjusted ), plausibly because it penalizes clinically significant errors while discounting insignificant ones, thereby most closely reflecting the CLARITY relevance dimension. RadCliQ, BERTScore, custom pooled FORTE, RaTE-Score, semb-score, FORTE degree F1, RadGraph F1, and BLEU also had positive associations with Holm-adjusted . By contrast, the FORTE impression, feature, and landmark components were not significant after correction, and CheXbert positive F1 was negatively correlated. Among the FORTE variants, the custom pooled score had the strongest association (), followed by degree (), impression (), feature (), and landmark (). At report level, 39.4% of generated reports were CLARITY-admissible. Automatic metrics therefore captured useful aggregate signal to markedly different degrees, but even the strongest association remained imperfect; none was an interchangeable substitute for the radiologist-grounded endpoint.
| Metric | 95% CI | |||
| GREEN [15] | 97/970 | 0.700 | [0.562, 0.798] | |
| −RadCliQ-v1 [14] | 97/970 | 0.479 | [0.295, 0.630] | |
| −RadCliQ-v0 [14] | 97/970 | 0.476 | [0.288, 0.630] | |
| BERTScore [13] | 97/970 | 0.467 | [0.282, 0.624] | |
| FORTE pooled F1 (custom) [17] | 97/970 | 0.432 | [0.247, 0.586] | |
| RaTE-Score [16] | 97/970 | 0.402 | [0.212, 0.572] | |
| semb-score [14] | 97/970 | 0.386 | [0.181, 0.565] | |
| FORTE degree F1 [17] | 97/970 | 0.383 | [0.208, 0.536] | |
| RadGraph-F1 [12] | 97/970 | 0.372 | [0.180, 0.540] | 0.00103 |
| BLEU [9] | 97/970 | 0.273 | [0.060, 0.461] | 0.0339 |
| FORTE impression F1 [17] | 96/957 | 0.235 | [0.016, 0.445] | 0.0854 |
| FORTE feature F1 [17] | 97/956 | 0.229 | [0.026, 0.399] | 0.0854 |
| FORTE landmark F1 [17] | 97/925 | 0.135 | [-0.068, 0.310] | 0.373 |
| CheXbert positive F1 [11] | 97/970 | -0.126 | [-0.339, 0.093] | 0.373 |
Table 1. Association of automatic metrics with CLARITY. Spearman correlations are computed after study-level averaging; 95% confidence intervals use 2,000 study-cluster bootstrap samples and two-sided values use Holm correction across 14 metric variants. RadCliQ scores are sign-inverted so higher values consistently indicate better reports. The sample-size column gives studies/reports after metric-specific exclusion of undefined values.
Application to Conformal Prediction
At the primary CheXbert-F1 threshold of 0.4, all fixed disjoint configurations produce finite sets. At , CLM exceeds the 0.70 coverage target but requires 9.59–9.91 of ten reports, whereas SCOPE-Gen produces smaller sets of 4.70–5.00 reports but reaches only 0.67–0.68 coverage. At , CLM again exceeds the 0.55 target with sets of 8.05–10.00 reports; SCOPE-Gen returns sets of 2.00–2.23 reports but reaches only 0.53–0.54 coverage. Thus, transfer from CheXbert calibration to CLARITY evaluation exposes a selectivity–coverage trade-off rather than uniformly preserving the nominal target.
The sensitivity analysis in Table 4 shows that this conclusion depends materially on the CheXbert-F1 threshold. Raising the threshold from 0.4 to 0.5 generally enlarges the SCOPE-Gen sets; at , both CLM configurations become invalid. At 0.6, no method produces a finite set for , while the finite sets contain 5.42–9.91 reports. Threshold selection therefore changes both feasibility and practical utility and should be treated as part of the admissibility definition rather than as a benign tuning choice. A similar low-selectivity pattern appears under CLARITY calibration: at , CLM returns 7.80–7.84 reports on average, compared with 3.05–3.07 for SCOPE-Gen.
| Method | Score | ||||
| 0.30 | SCOPE-Gen | count | 0.67 / 5.00 | 0.71 / 7.00 | – |
| SCOPE-Gen | max | 0.68 / 4.70 | 0.71 / 6.41 | – | |
| CLM | count | 0.75 / 9.59 | – | – | |
| CLM | max | 0.76 / 9.91 | – | – | |
| 0.45 | SCOPE-Gen | count | 0.53 / 2.00 | 0.60 / 3.00 | 0.70 / 6.00 |
| SCOPE-Gen | max | 0.54 / 2.23 | 0.54 / 2.44 | 0.71 / 5.42 | |
| CLM | count | 0.77 / 10.00 | 0.77 / 10.00 | 0.74 / 9.16 | |
| CLM | max | 0.67 / 8.05 | 0.69 / 6.88 | 0.76 / 9.91 |
Table 4. Sensitivity of the fixed disjoint CheXbert-to-CLARITY evaluation to the CheXbert positive-label F1 admissibility threshold. Each cell reports coverage / prediction-set size for the single fixed split; “–” indicates that no finite prediction set could be calibrated.
Across the random-split experiments, SCOPE-Gen generally yields smaller prediction sets than CLM. This advantage must be interpreted together with calibration stability. At , some CLM summaries are based on only two to six valid runs, and the SCOPE-Gen max configuration under CLARITY is based on 31 valid runs. Means computed from these successful subsets can be optimistic and do not describe all 100 attempted splits. At , SCOPE-Gen is stable across all random splits, whereas CLM remains invalid in 45–52% of labeled-set runs. Collectively, the results show that coverage, prediction-set size, and calibration failure must be considered jointly; an admissibility proxy that yields nominal coverage can still be impractical or fail to transfer to clinically grounded evaluation.
Discussion
The central finding is that existing automatic metrics recover radiologist-assessed clinical admissibility to markedly different and incomplete degrees. GREEN shows by far the strongest association with CLARITY (), plausibly because its error counts target clinically significant errors while discounting insignificant ones—the design that most closely mirrors the CLARITY relevance dimension. RadCliQ, BERTScore, custom pooled FORTE, RaTE-Score, semb-score, FORTE degree F1, RadGraph-F1, and BLEU also capture meaningful study-level variation. However, the component-level FORTE results are heterogeneous, and CheXbert positive-label F1 shows no positive association with CLARITY. This result is especially consequential because CheXbert has been used to define admissibility in prior conformal report-generation pipelines.
The mismatch reflects two structural limitations rather than merely a difference in metric sensitivity. First, as depicted in Figure 1, reference-based automatic labelers are bound by the reference text: if a generated report correctly identifies a finding omitted by the original radiologist, CheXbert can penalize it as a hallucination. This limitation is consequential because expert radiologists found 30/97 (30.9%) ground-truth reports in our dataset to be clinically inadmissible. Second, label-matching metrics ignore asymmetric clinical consequences. For example, failing to mention a pneumothorax and hallucinating a minor bone change each count as a single error under CheXbert, whereas CLARITY uses its relevance dimension to distinguish errors that could affect patient management. GREEN’s comparatively strong association supports the importance of explicitly modeling clinical significance, but even GREEN leaves a gap to the radiologist endpoint. Automatic metrics may therefore remain useful for scalable model development, ranking, or cohort-level monitoring, while the present evidence does not support treating any single metric as an interchangeable replacement for image-grounded radiologist evaluation.
Figure 1. Qualitative comparison of admissibility assessments. Reference-based automatic assessment maps a candidate and reference report to clinical labels and then compares those labels. This procedure can penalize a correct finding that was omitted from the reference and treats discrepancies equally without accounting for their clinical consequence. CLARITY instead uses image-grounded radiologist judgments of both relevance and accuracy. The labels and arrows, rather than color alone, distinguish the assessment pathways.
The conformal prediction experiments demonstrate that admissibility is not a neutral implementational detail: changing the criterion used to identify an acceptable report changes both the frequency of admissible calibration examples and the resulting coverage–set-size trade-off. Under the fixed disjoint configuration at the CheXbert-F1 threshold of 0.4, finite sets are possible at both target coverages, but SCOPE-Gen falls slightly below the nominal targets while CLM attains them using much larger sets. Increasing the threshold enlarges sets or makes calibration infeasible, culminating in no finite configuration at F1 . Thus, the metric mismatch propagates into the feasibility and practical utility of proxy-calibrated prediction sets.
Within this application, SCOPE-Gen generally produces smaller sets than CLM, particularly at , but method comparisons cannot be reduced to average set size. Several CLM configurations, and some SCOPE-Gen configurations at , fail to produce finite sets for substantial fractions of the random splits. Moreover, the reported means are conditional on successful runs. This is particularly important where only two to six of 100 CLM runs are valid: those entries provide little evidence about expected behavior under a new split. Reporting alongside coverage and set size exposes this failure mode, while the large finite sets distinguish nominal coverage from practical utility.
CLARITY addresses two limitations of reference-based admissibility. First, readers assess reports against the images rather than treating the original report as an error-free gold standard; this matters because 30 of 97 original reports are themselves rated inadmissible under the rubric. Second, the relevance label makes the admissibility event asymmetric: only findings judged relevant are required to be correct. This better reflects the fact that reporting errors differ in clinical importance, while remaining a transparent operational rule that can be audited at the category level. The rule should be viewed as an evaluation framework rather than a complete model of downstream harm. Future work could adapt category definitions or relevance thresholds to particular clinical workflows and should test whether those adaptations preserve calibration across institutions and reader populations.
The primary metric finding suggests a division of labor between automated and expert evaluation. Automatic metrics can support large-scale screening and identify broad changes in report quality, while smaller image-grounded expert datasets can determine whether each proxy remains aligned with the clinical endpoint of interest. Such validation is necessary before metric scores are interpreted as evidence of clinical admissibility, particularly when the metric reduces a report to lexical or extracted-label agreement.
The CP application demonstrates the downstream importance of this alignment check because conformal guarantees apply to the defined loss, not automatically to clinical safety. A deployment study should therefore report evidence that its admissibility function remains aligned with the target clinical endpoint, alongside nominal coverage, calibration failures, prediction-set size, and the prevalence of admissible candidates.
This study has several limitations. Only 97 of the 293 eligible studies were annotated, and the labeled cohort comprises the sequence of cases completed by the radiologists within the allotted annotation time rather than a randomized sample. Selection effects may therefore influence both admissibility prevalence and metric associations. The data come from a single public dataset, and all candidates were generated by one model variant with ten pre-generated reports per study. The findings may not generalize to other institutions, modalities, generators, prompts, or decoding procedures.
CLARITY is also limited to its 12 categories and operational relevance judgments; clinically important phenomena outside the rubric may be missed. The metric analysis establishes association rather than interchangeability with the radiologist endpoint. Missing component-level FORTE values arise when no terms from a category are matched, producing unequal denominators and particularly sparse within-study information for the feature and landmark components, whereas GREEN covers all 970 reports in the final labeled cohort. The automatic metrics were not validated as replacement admissibility functions in otherwise identical CP pipelines.
Finally, the secondary CP application uses small calibration and test samples. Invalid conformal runs and the instability of conditional summaries at limit the precision of comparisons between methods, while the disjoint results represent one fixed split. Although the disjoint comparison uses the same primary CheXbert threshold as the random-split experiments, the sensitivity analysis confirms that feasibility and set size depend strongly on this threshold. Larger, prospectively sampled, multi-institutional studies with repeated independent annotations are needed before either the metric relationships or the conformal results can be interpreted as deployment-ready evidence.
We introduced CLARITY, a radiologist-grounded operational definition of admissibility for automated chest-radiograph reports, and used it to test whether 14 automatic metric variants recover expert assessments of clinical admissibility. Our central finding is that alignment with radiologist evaluation is highly metric-dependent and incomplete. CheXbert positive-label F1, despite its use as an admissibility proxy in prior conformal pipelines, shows no positive alignment with the radiologist endpoint. GREEN aligns most strongly at study level, but even this association is imperfect and does not make the metric interchangeable with image-grounded expert review. The observed mismatch reflects limitations that label and text comparisons cannot resolve alone: reference reports may themselves contain errors, and superficially similar discrepancies can have substantially different clinical relevance.
We then examined how this mismatch propagates to conformal prediction. Changing the admissibility definition alters calibration feasibility, empirical coverage, and prediction-set size; SCOPE-Gen generally produces smaller sets than CLM, but can miss nominal coverage, whereas CLM often requires nearly all candidates or fails to calibrate on some splits. These results illustrate a practical consequence of the primary metric finding: statistical guarantees defined through an automatic proxy should not be interpreted as guarantees of radiologist-assessed clinical admissibility without direct validation. The expert-annotated benchmark provides a basis for more reproducible evaluation of metric alignment and clinically grounded risk control in radiology report generation.
Methods
Study design and data source
We conducted a retrospective methodological study using MIMIC-CXR [22], a de-identified database containing 377,110 chest radiographs from 227,827 imaging studies with free-text reports. Because MedGemma was trained on the official MIMIC-CXR training and evaluation partitions, we restricted the analysis to the test partition. This partition contains 3,269 imaging studies from 293 patients. Multiple studies from the same patient would violate the patient-level exchangeability assumption underlying the conformal analyses [23]; we therefore retained the chronologically first study for each patient, yielding 293 independent study units. The first study was selected by a prespecified deterministic rule rather than according to image or report content.
Report generation
We used the 27-billion-parameter instruction-tuned MedGemma medical vision-language model [24]. The associated reports were divided into sections by rule-based parsing. Text from the Indication or History section was supplied as the clinical Reason, frontal and available lateral chest radiographs were supplied as image input, and the Findings section served as the reference report. A standardized prompt was used to generate ten candidate reports for each of the 293 studies. Preliminary inspection found the instruction-tuned checkpoint more coherent and precise than the pretrained checkpoint, and all subsequent analyses used the instruction-tuned variant.
Clinical admissibility criteria
CLARITY assesses a report for relevance as Relevant or Irrelevant and for accuracy as Correct, Incorrect, or Missing in each of 12 clinical finding categories:
Diaphragm
Pleural effusion
Pneumothorax
Mediastinum
Heart silhouette
Cardiopulmonary congestion
Pneumonia
Lung opacity
Osseous structures
Soft tissue
Abdomen
Foreign material
The categories are largely inspired by the popular CheXpert labels [25] and modified for higher relevance and better coverage of the possible medical findings. The relevance label is central to CLARITY rather than an auxiliary annotation: it determines whether an incorrect or missing finding constitutes a clinically consequential error. By separating relevance from accuracy, CLARITY distinguishes errors that could affect patient management from minor omissions or deviations that should not render an otherwise useful report inadmissible. This distinction is essential because counting all discrepancies equally would obscure the practical clinical utility of generated reports.
Radiologist evaluation
Two board-certified radiologists reviewed the DICOM images, the clinical REASON, the original FINDINGS, and the generated reports corresponding to 97 studies using the CLARITY rubric. Readers were instructed to evaluate each report against the images; the original FINDINGS was treated as a reference, was evaluated under the same rubric, and was not assumed to be wholly correct. Each report received 24 annotations comprising one relevance and one accuracy label for each category.
Automatic metric evaluation
We evaluated 14 automatic metric variants. These comprised RadCliQ-v0 and RadCliQ-v1, BERTScore, RaTE-Score, semb-score, RadGraph-F1, GREEN, BLEU, CheXbert positive-label F1, four official FORTE components, and a custom pooled FORTE score. CheXbert uncertain labels were treated as not positive, and the 12 report pairs with two empty positive-label sets were assigned an F1 score of one. RaTE-Score was computed using RaTE-NER-DeBERTa, BioLORD-2023-C, and the long affinity matrix. GREEN was computed with the GREEN-RadLlama2-7b checkpoint [15] using deterministic decoding. FORTE degree, landmark, feature, and impression F1 scores were computed separately. The custom pooled FORTE score is a category-tagged union of the four components and is not an official FORTE statistic. Component F1 was undefined when no terms in that category were matched; these values were treated as missing rather than assigned a numerical score.
Conformal prediction
We evaluated Conformal Language Modeling (CLM) [7] and the generation stage of Conformal Generative Modeling with Improved Sample Efficiency through Sequential Greedy Filtering (SCOPE-Gen) [8]. Both methods use an admissibility function that labels a candidate report as admissible or inadmissible. The strict CheXbert definition is
where is the reference report and is the -th CheXbert label [11, 26]. Exact matching produced too few admissible candidates for useful calibration. We therefore used a soft-label definition based on positive-label F1: the binary loss was , and the primary loss threshold was 0.6, so reports with were admissible. The fixed disjoint analysis was repeated at thresholds of 0.5 and 0.6. The alternative CLARITY function labeled a report admissible according to the category-level rule described above.
The labeled random-split analyses used 970 candidates from 97 studies, with 77 studies for calibration and 20 for testing in each 80:20 split. The full-set CheXbert analysis used all 293 studies, with 234 for calibration and 59 for testing. The fixed disjoint analysis calibrated on the 196 studies without radiologist labels and evaluated on all 97 labeled studies; no patient or study appeared in both sets. We evaluated conformal miscoverage levels , corresponding to target coverages of 0.70 and 0.55. For SCOPE-Gen, the count nonconformity score was prediction-set cardinality, whereas max was the maximum included candidate score plus 0.1 for every included report. CLM used both lambda components and a maximum of 20.
Each same-label configuration was evaluated across 100 study-level random calibration/test splits. A run was valid only if it produced a finite prediction set. Coverage and prediction-set size are reported as the mean and standard deviation across valid runs, together with , the number of valid runs among the 100 attempts. The fixed disjoint analysis used one split and is reported without a variability estimate. Results conditional on a small are not estimates over all attempted splits.
Statistical analysis
The study, rather than the individual generated report, was the primary independent sampling unit. The primary metric analysis averaged each automatic metric and the binary CLARITY endpoint over the ten reports within each study and measured their association using Spearman’s rank correlation. Spearman correlation was selected because the hypothesis concerned monotonic association and did not require normally distributed metric values. The primary analysis included 97 studies and 970 reports for complete metrics. Metric-specific complete-case sample sizes were 96 studies and 957 reports for FORTE impression F1, 97 and 956 for FORTE feature F1, and 97 and 925 for FORTE landmark F1; all other variants included 97 studies and 970 reports. No values were imputed.
Two-sided values were computed for each Spearman correlation, and a family-wise hypothesis-testing alpha of 0.05 was used. Holm’s procedure adjusted the 14 primary values for multiple comparisons. The 95% confidence intervals are percentile intervals from 2,000 cluster-bootstrap replicates generated by sampling the 97 study identifiers with replacement and retaining all reports from each sampled study. The report-level CheXbert analysis was exploratory, unadjusted, and did not account for within-study dependence, so the study-level analysis was used for inference. Exact values are reported in the text and Table 1. Normality testing was not applicable to the nonparametric rank analysis.
The conformal analyses were descriptive and did not use null-hypothesis tests. Their denotes conformal miscoverage and is distinct from the 0.05 hypothesis-testing alpha. Values following in Table 2 and Table 3 are standard deviations across valid random splits.
| Method | Metric | CheXbert (full set) | CheXbert (disjoint CLARITY eval.) | CheXbert (labeled set) | CLARITY |
| 234 | 196 | 77 | 77 | ||
| 59 | 97 | 20 | 20 | ||
| SCOPE-Gen (count) | Coverage | 0.67 | |||
| Pred. set size | 5.00 | ||||
| 100 | 1 | 100 | 100 | ||
| SCOPE-Gen (max) | Coverage | 0.68 | |||
| Pred. set size | 4.70 | ||||
| 100 | 1 | 90 | 31 | ||
| CLM (count) | Coverage | 0.75 | |||
| Pred. set size | 9.59 | ||||
| 44 | 1 | 2 | 6 | ||
| CLM (max) | Coverage | 0.76 | |||
| Pred. set size | 9.91 | ||||
| 43 | 1 | 4 | 3 |
Table 2. Empirical coverage and prediction-set size for conformal miscoverage . Random-split columns report the mean standard deviation (s.d.) over the finite runs among 100 attempted study-level splits and the number of finite runs. The fixed disjoint CheXbert-to-CLARITY evaluation uses the primary F1 threshold of 0.4 and reports one run.
| Method | Metric | CheXbert (full set) | CheXbert (disjoint CLARITY eval.) | CheXbert (labeled set) | CLARITY |
| 234 | 196 | 77 | 77 | ||
| 59 | 97 | 20 | 20 | ||
| SCOPE-Gen (count) | Coverage | 0.53 | |||
| Pred. set size | 2.00 | ||||
| 100 | 1 | 100 | 100 | ||
| SCOPE-Gen (max) | Coverage | 0.54 | |||
| Pred. set size | 2.23 | ||||
| 100 | 1 | 100 | 100 | ||
| CLM (count) | Coverage | 0.77 | |||
| Pred. set size | 10.00 | ||||
| 100 | 1 | 48 | 48 | ||
| CLM (max) | Coverage | 0.67 | |||
| Pred. set size | 8.05 | ||||
| 100 | 1 | 50 | 55 |
Table 3. Empirical coverage and prediction-set size for conformal miscoverage . Random-split columns report the mean standard deviation (s.d.) over the finite runs among 100 attempted study-level splits and the number of finite runs. The fixed disjoint CheXbert-to-CLARITY column uses the primary F1 threshold of 0.4 and reports one run.
Ethics
MIMIC-CXR contains de-identified human clinical data distributed through PhysioNet under credentialed access and a data-use agreement [22].
Data availability
MIMIC-CXR is available through PhysioNet under credentialed access and its data-use agreement [22]. Derived data, including the generated candidate reports and CLARITY annotations, will be made available after successful publication at PhysioNet, according to the MIMIC-CXR data-use conditions.
Code availability
The supporting codebase for this study is made publicly available and can be accessed via this link: github.com/nikoskont/clarity.
Author contributions
N.K. and K.-R.K. conducted the experimental analysis and authored the manuscript. P.K. contributed to the CLARITY method design and labelled reports. A.E. labelled reports. C.B., L.K and M.M. provided feedback on the study design and supported manuscript authorship. S.A. and M.M. provided supervision. All authors reviewed and approved the final manuscript.
Acknowledgements
This research received no specific grant from any funding agency in the public, commercial, or not-for-profit sectors.
Competing interests
The authors declare no competing interests.
References
- [1]Kristina Lång et al. Artificial intelligence-supported screen reading versus standard double reading in the Mammography Screening with Artificial Intelligence trial (MASAI): a clinical safety analysis of a randomised, controlled, non-inferiority, single-blinded, screening accuracy study. The Lancet Oncology, 24(8):936–944, 2023.
- [2]Frederike Lübeck et al. Adaptable cardiovascular disease risk prediction from heterogeneous data using large language models. arXiv:2505.24655, pages 1–29, 2025.
- [3]Yilin Li, Chao Kong, Guosheng Zhao, and Zijian Zhao. Automatic radiology report generation with deep learning: a comprehensive review of methods and advances. Artificial Intelligence Review, 58(11):1–42, 2025.
- [4]Ben Van Calster, David J. McLernon, Maarten van Smeden, Laure Wynants, and Ewout W. Steyerberg. Calibration: the Achilles heel of predictive analytics. BMC Medicine, 17(230):1–7, 2019.
- [5]Yubin Kim et al. Medical hallucination in foundation models and their impact on healthcare. arXiv:2503.05777, pages 1–61, 2025.
- [6]Elham Asgari et al. A framework to assess clinical safety and hallucination rates of LLMs for medical text summarisation. npj Digital Medicine, 8(274):1–15, 2025.
- [7]Victor Quach et al. Conformal language modeling. International Conference on Learning Representations, pages 1–28, 2024.arxiv.org/abs/2306.10193
- [8]Klaus-Rudolf Kladny, Bernhard Schölkopf, and Michael Muehlebach. Conformal generative modeling with improved sample efficiency through sequential greedy filtering. International Conference on Learning Representations, pages 1–30, 2025.
- [9]Kishore Papineni, Salim Roukos, Todd Ward, and Wei-Jing Zhu. BLEU: A Method for automatic evaluation of machine translation. Annual Meeting of the Association for Computational Linguistics, pages 311–318, 2002.
- [10]Chin-Yew Lin. ROUGE: A package for automatic evaluation of summaries. Text summarization branches out, pages 74–81, 2004.
- [11]Akshay Smit et al. Combining automatic labelers and expert annotations for accurate radiology report labeling using BERT. Empirical Methods in Natural Language Processing, pages 1500–1519, 2020.
- [12]Saahil Jain et al. RadGraph: Extracting Clinical Entities and Relations from Radiology Reports. PhysioNet, 2021. Version 1.0.0.
- [13]Tianyi Zhang, Varsha Kishore, Felix Wu, Kilian Q. Weinberger, and Yoav Artzi. BERTScore: Evaluating text generation with BERT. In International Conference on Learning Representations, 2020.arxiv.org/abs/1904.09675
- [14]Feiyang Yu et al. Evaluating progress in automatic chest x-ray radiology report generation. Patterns, 4(9):100802, 2023.DOI
- [15]Sophie Ostmeier et al. GREEN: Generative radiology report evaluation and error notation. In Findings of the Association for Computational Linguistics: EMNLP 2024, pages 374–390, 2024.DOI
- [16]Weike Zhao et al. RaTEScore: A metric for radiology report generation. In Conference on Empirical Methods in Natural Language Processing, pages 15004–15019, 2024.arxiv.org/abs/2406.16845
- [17]Cheng-Yi Li et al. Towards a holistic framework for multimodal LLM in 3D brain CT radiology report generation. Nature Communications, 16:2258, 2025.DOI
- [18]Shruthi Bannur et al. MAIRA-2: Grounded radiology report generation. arXiv:2406.04449, pages 1–72, 2024.arxiv.org/abs/2406.04449
- [19]Chenyu Wang, Weichao Zhou, Shantanu Ghosh, Kayhan Batmanghelich, and Wenchao Li. Semantic consistency-based uncertainty quantification for factuality in radiology report generation. Findings of the Association for Computational Linguistics, pages 1739–1754, 2025.DOI
- [20]Hooman Shahrokhi, Devjeet Raj Roy, Yan Yan, Venera Arnaoudova, and Janaradhan Rao Doppa. Conformal prediction sets for deep generative models via reduction to conformal regression. Uncertainty in Artificial Intelligence, pages 1–31, 2025.arxiv.org/abs/2503.10512
- [21]Zhuohang Li et al. Towards statistical factuality guarantee for large vision-language models. Conference on Empirical Methods in Natural Language Processing, pages 11435–11456, 2025.arxiv.org/abs/2502.20560
- [22]Alistair E. W. Johnson et al. MIMIC-CXR, a de-identified publicly available database of chest radiographs with free-text reports. Scientific Data, 6(317):1–8, 2019.DOI
- [23]Ryan Tibshirani. Conformal Prediction. UC Berkeley, Lecture Notes, pages 1–15, 2023.
- [24]Andrew Sellergren et al. MedGemma Technical Report. arXiv:2507.05201, pages 1–62, 2025.DOI
- [25]Jeremy Irvin et al. CheXpert: A large chest radiograph dataset with uncertainty labels and expert comparison. AAAI Conference on Artificial Intelligence, pages 590–597, 2019.arxiv.org/abs/1901.07031
- [26]Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. BERT: Pre-training of deep bidirectional Transformers for language understanding. Conference of the North American Chapter of the Association for Computational Linguistics, pages 4171–4186, 2019.arxiv.org/abs/1810.04805

