In this paper
AbstractIntroductionResultsSubgroup discovery exposes higher performance gaps than traditional subgroup analysisDiscovered subgroups are diverse and often align with clinical or acquisition-related conceptsUse case: subgroup annotations for improving model robustnessSubgroup discovery passes sanity checks on synthetic dataFeature extractors trained on natural images are sufficient for exposing meaningful performance gapsDiscussionMethodsDataset descriptionPreliminaries: subgroup discovery algorithmStatistical analysis of subgroup performanceStatistical analysis of metadata profilesSynthetic scenario with generated artifactsExperimental setupPerspective: regulatory implicationsData availabilityCode availabilityAcknowledgmentsAuthor ContributionsCompeting InterestsReferencesWorksComputer Science
Detection and Validation of Hidden Subgroup Disparities in Clinical AI
Abstract
Machine learning (ML) models used in clinical workflows often perform systematically differently across patient groups, raising concerns about equity and patient safety. Traditional subgroup analyses can expose such disparities only when rich metadata (e.g., patient demographics) is available and captures the sources of performance variation, conditions rarely met in practice. Algorithmic subgroup discovery methods offer a solution by identifying coherent groups from learned feature representations, but because true underlying stratifications are unknown, these methods are hard to evaluate. As a result, subgroup discovery has not been systematically applied or assessed for performance monitoring. Our main objective is to evaluate algorithmic subgroup discovery for performance monitoring, assessing the discovered subgroups' performance gaps and semantic coherence. Investigating cardiomegaly classification in a dataset of 223,228 chest X-rays, we showed that the discovered subgroups exhibited larger performance gaps (accuracies of 0.61-0.90) than traditional metadata-based analysis (0.80-0.94) and were frequently interpretable using patient, acquisition, and model metadata. As a proof of concept, we also showed that annotations from learned subgroups can be used to mitigate the biases they expose. We conclude that subgroup discovery is a practical and effective tool to enhance the performance validation and monitoring of ML systems in healthcare.
Introduction
Machine learning (ML) algorithms are implemented and commercialised as medical artificial intelligence (AI) devices at an unprecedented pace. In the US, for example, the FDA had approved 1,247 AI-assisted medical devices by May 2025, with 523 of these approvals occurring within the last 24 months [1]. This growth reflects the vast potential of AI for patient care. However, research shows that even the most capable AI models can reinforce societal biases and fail unpredictably in clinical settings [2]. Consequently, regulatory agencies worldwide are tightening policies governing the development and deployment of medical AI to enhance its fairness and robustness.
A critical concern is that ML models may perform systematically differently across patient subgroups [3, 4, 5, 6]. This has hampered past attempts at safely deploying medical AI, particularly in underserved populations [3, 4]. Model performance can depend on many factors [7], including patient attributes (e.g., sex, age, ethnicity) and image characteristics (e.g., image quality, artifacts, device manufacturer). Chest X-rays model’s performances, for example, were shown to depend on biological sex [8, 9], support devices such as tracheotomy tubes and chest drains [10, 11], artifacts (e.g., laterality markers, highlight arrows) [12], and projection views [12] (anterior-posterior (AP) and posterior-anterior (PA)). Not understanding when and why an ML model systematically fails poses an avoidable danger to patients and delays the development of more robust models. Furthermore, systematic hidden model failures may violate regulators’ demands for safety and effectiveness for all intended use populations of a medical device.
These challenges are central to ensuring the safe and equitable application of AI in healthcare. Recent guidelines for trustworthy medical AI development, reporting, and deployment increasingly stress the importance of detailed subgroup analyses (e.g., TRIPOD+AI [13]) and systematic bias assessments (e.g., FUTURE-AI [14], PROBAST+AI [15]). In response, current frameworks call on AI developers and dataset curators to explicitly identify and annotate potential sources of bias throughout the model development pipeline. Together with clinical metadata, such as patient attributes and data-specific characteristics, the annotation can then be used to define patient subgroups to assess group fairness and evaluate model robustness. In general, the richer the available annotations, the more precisely relevant subgroups can be defined, increasing the chance of uncovering meaningful performance gaps.
However, acquiring such rich metadata is inherently challenging: dataset collection is both time- and personnel-intensive, and privacy concerns can limit the scope of metadata collection. In practice, this can lead to severe reporting gaps, as exposed by a recent scoping review of FDA-approved medical AI devices [16]. According to their analysis, less than 4% of approved devices provide information on race or ethnicity, less than 1% report on socioeconomic factors, and less than 19% report on age subgroups [16].
Even under ideal conditions, where multiple metadata attributes are available for a large, heterogeneous population, metadata may only partially align with the major hidden factors of variation that affect ML model performance. As a result, traditional metadata-defined subgroups may fail to reveal relevant performance gaps. Metadata annotations often reflect concepts that are meaningful for human interpretation and categorization, yet may be irrelevant to how machine learning models stratify data to make predictions. For example, stratifying a chest X-ray dataset by patient sex may be of limited value for auditing performance, since observed differences are often driven by correlated factors such as the presence of support devices (e.g., tubes and lines) rather than sex itself [9, 11]. In contrast, stratifying by projection (AP vs PA) is crucial. Standard radiology interpretation emphasizes that the PA view is preferred for assessing cardiomegaly because the AP view magnifies the cardiac silhouette [17]. Accordingly, deep learning models perform better on PA studies than on AP studies for cardiomegaly-related labels (see Table 2). Unrecognized subsets of data are referred to as hidden stratifications. As we will demonstrate in this study, even in one of the most comprehensively annotated large-scale datasets available, CheXpertPlus [18], which contains over 80 annotated factors including patient socio-demographics, comorbidities, and image characteristics for more than 200,000 samples, hidden stratifications remain prevalent. While manual inspection of model errors can sometimes reveal these issues, it is generally infeasible at scale and particularly challenging in medical imaging, where relevant differences across samples are often subtle.
| SG | Category | Description | Accuracy | Sensibility | Specificity | AUROC |
| A5 | ⚙ | Positive preds | ||||
| A0 | 👤 | High BMI, males | ||||
| A12 | 👤 | Males, unknown | ||||
| A9 | 👤 📷 ⚙ | Females, positive preds, support devices | ||||
| A8 | 📷 | Support devices, interpreter needed | ||||
| A6 | – | Unknown | ||||
| A11 | ⚙ | Negative preds, unknown | ||||
| A4 | 📷 | Support devices, unknown | ||||
| A14 | 📷 | Support devices, unknown | ||||
| A2 | 📷 | Lateral samples, unknown | ||||
| A1 | 👤 📷 | Lateral images, healthy, males | ||||
| A13 | 📷 | Lateral images | ||||
| A10 | 👤 📷 ⚙ | Positive preds, PA, pacemaker, Pacific Islander | ||||
| A3 | 👤 📷 | Healthy males, PA | ||||
| A7 | 📷 | PA, unknown |
Table 2. Performance by discovered subgroups. Subgroups are categorized into patient (👤), acquisition (📷), and model (⚙) factors. Values are mean 95% confidence-interval half-width. PA = posterior-to-anterior, AP = anterior-to-posterior.
Algorithmic subgroup discovery [19, 20] appears as a potential solution for this problem, as it can learn data stratifications that are relevant to a specific ML prediction model and dataset. The subgroup divisions are found by training unsupervised clustering methods (e.g., mixture models, k-means) on learned feature representations of the data. Aside from the predictions from the investigated model, subgroup discovery methods also leverage complementary knowledge from models trained with different objectives (e.g., multi-modal, generative) or with more data. These models have been used so far only on toy datasets with predefined biases for easier evaluation. We believe they might also be useful for performance evaluation and model surveillance, but they have yet to be proven effective in real-world medical contexts. The main challenge of using subgroup discovery techniques is how to validate the discovered subgroups, as hidden stratifications by definition lack ground-truth labels for evaluation.
In this article, we thoroughly investigate algorithmic subgroup discovery [19, 21] for detailed model performance evaluation (Figure 1). For subgroup discovery techniques to be effective, they should satisfy two key criteria: (1) they must reveal substantial performance disparities across groups, and (2) the identified subgroups should exhibit semantic coherence. That is, subgroup members should exhibit interpretable characteristics, whether related to patient (demographics, comorbidities), acquisition, or model factors. We define our research questions as:
Figure 1. From traditional subgroup analysis to algorithmic subgroup discovery for performance monitoring. (a) Traditional subgroup analysis detects disparate patient outcomes, but it is limited to annotated metadata. (b) Subgroup discovery reveals hidden stratifications but lacks performance validation. (c) We bridge this gap by applying subgroup discovery for performance analysis in both (d) controlled synthetic settings and (e) real-world scenarios with unknown subgroups.
Do algorithmic subgroup discovery systematically expose performance gaps across different metrics?
Are the discovered subgroups semantically coherent, and can we associate them with existing metadata?
Can the discovered subgroups be leveraged to mitigate the performance disparities they expose?
We employ a richly annotated real-world radiology dataset of chest x-ray images for a cardiomegaly-versus-all task to support a comprehensive evaluation. Using this dataset, we first contrast performance gaps identified through metadata-defined subgroups with those revealed by subgroup discovery. We then characterize the discovered subgroups with our proposed method for creating metadata profiles, i.e. sets of metadata attributes that explain the similarities among samples within the same subgroup.
We argue that subgroup discovery can be an effective and easily implemented tool to enhance the performance validation and monitoring of ML systems in medicine. In this article, we expand on our prior work [21], where we demonstrated preliminary evidence that algorithmic subgroup discovery can expose systematic performance gaps of medical ML models. Specifically, we propose a way to characterise found subgroups using existing metadata, which we call subgroup profiles. Moreover, we report additional metrics beyond accuracy and perform a statistical analysis of our results. Overall, our main contributions are:
We provide the first comprehensive evidence that subgroup discovery can systematically expose performance gaps in medical imaging, identifying meaningful subgroups in both synthetic and real-world settings.
We demonstrate that discovered subgroups exhibit significantly larger performance disparities than conventional demographic metadata, revealing critical gaps missed by traditional fairness auditing.
We show that discovered subgroups are cohesive and that many exhibit statistically significant associations with annotated metadata.
We demonstrate that discovered subgroups can be effective for mitigating biases.
Results
We trained a deep learning model for cardiomegaly classification on CheXpertPlus [18], a real-world chest X-ray dataset richly annotated with metadata, including patient demographics, acquisition details, and other factors (see first two columns of Table 1 for summary statistics). Our training, validation, and test sets follow an 80/10/10 split, with 178,684 / 22,263 / 22,281 images, respectively. The model achieved a balanced accuracy of 0.70 in cardiomegaly classification and an area under the receiver-operating characteristic curve (AUROC) of 0.85 on the test set. Traditional subgroup analysis (metadata-based) revealed slight performance disparities between subgroups (see Table 1).
| Metadata Subgroups | Sample Size | Accuracy | Sensitivity | Specificity | AUROC |
| Full dataset | 22,281 | ||||
| Sex: Male | 13,134 | ||||
| Sex: Female | 9,127 | ||||
| Age: 0–60 | 10,032 | ||||
| Age: 60+ | 12,229 | ||||
| Race: White | 12,635 | ||||
| Race: Black | 1,164 | ||||
| Race: Asian | 2,418 | ||||
| Recent BMI: Normal (18.5 - 25) | 6,492 | ||||
| Recent BMI: Morbidly Obese (>40) | 711 | ||||
| Projection: PA | 2,960 | ||||
| Projection: AP | 16,032 | ||||
| Support Devices: Positive | 11,501 | ||||
| View: Frontal | 18,894 | ||||
| View: Lateral | 3,287 | ||||
| Pneumothorax: Positive | 1,696 | ||||
| Edema: Positive | 5,454 | ||||
| Lung Lesion: Positive | 908 |
Table 1. Table 1: Traditional subgroup analysis. Subgroup performance metrics reported as value half 95% CI. PA = posterior-to-anterior, AP = anterior-to-posterior.
Then, we applied a simplified version [21] of a state-of-the-art subgroup discovery method [19] to discover subgroups algorithmically from learned feature representations (see details in the Methods Section). In short, the method used the cardiomegaly classification model’s predictions and feature representations obtained from an external non-medical visual-language model [22], and applied clustering to identify relevant subgroups. This resulted in 15 discovered subgroups (denoted A0–A14), each containing between 223 and 4,354 images (blue bars in Figure 2 A). The prevalence of cardiomegaly varied across the subgroups between 8% and 61% (light blue bars in Figure 2 B). In contrast, traditional metadata contained between 711 and 18,894 samples (red bars in Figure 2 A), and cardiomegaly prevalence varied across metadata subgroups between 10% and 24% (light red bars in Figure 2 B). We report cardiomegaly prevalence because it highlights heterogeneity across subgroups with respect to the target pathology and because variation in prevalence directly affects measured performance, particularly given that specificity is typically higher than sensitivity.
Figure 2. Performances of discovered subgroups (blue) and traditional metadata subgroups (red). Filled points indicate statistically significant performance disparities from its complement. For each group, we show their size (A), cardiomegaly class ratio (B), and their performance in prevalence-dependent (C) and independent (D-F) metrics.
Subgroup discovery exposes higher performance gaps than traditional subgroup analysis
Discovered subgroups showed larger performance disparities than traditional metadata-defined subgroups (Figure 2, panels C–F). Accuracy varied substantially across discovered subgroups, ranging from 0.61 to 0.92, which may partly reflect differences in class prevalence (0.08–0.61). However, we also observed statistically significant differences in prevalence-independent metrics: sensitivity ranged from 0.10 to 0.98, and AUROC from 0.67 to 0.89. This shows that prevalence alone is insufficient to justify the disparities observed among the discovered subgroups. Overall, sensitivity differed significantly in 7 of 15 subgroups, specificity in 10 of 15, and accuracy in 11 of 15 (see details about the statistical study in the Methods Section). In contrast, metadata-defined subgroups showed smaller but still significant gaps, with accuracy ranging from 0.80 to 0.94, sensitivity from 0.39 to 0.60, and AUROC from 0.82 to 0.88. Across these, sensitivity differed significantly in 6 of 17 subgroups and accuracy in 8 of 17.
The largest disparities were concentrated in a few specific subgroups. Among discovered subgroups, groups A5 and A10 showed the highest sensitivity (0.984 and 0.772, respectively) and the lowest specificity (0.017 and 0.756). The pronounced sensitivity–specificity imbalance in subgroup A5 reflects a strong bias toward positive cardiomegaly predictions, resulting in the only significant AUC disparity (0.188) across all experiments and the largest accuracy gap (0.260). Subgroup A9, despite a relatively low prevalence of positive samples, also showed a marked reduction in accuracy of 0.101. Among metadata-defined subgroups, Black patients exhibited the largest disparity, with sensitivity increased by 0.13, specificity decreased by 0.05, and accuracy decreased by 0.03 relative to the average. Regarding acquisition factors, posteroanterior (PA) samples showed the largest disparities, with sensitivity increased by 0.12 and accuracy by 0.03.
Discovered subgroups are diverse and often align with clinical or acquisition-related concepts
We next verified whether the discovered subgroups could be characterised with semantic concepts. For this, we used the rich metadata available in the CheXpertPlus [18] dataset, and created metadata profiles for each discovered subgroup by measuring the association between the metadata attributes and each subgroup (see the Methods section for details). The resulting metadata profiles can be analysed as heatmaps for each discovered subgroup (columns in Figure 3A), with bold colours indicating a strong association between a subgroup and a metadata attribute. We divide the subgroups into three major categories: patient, acquisition, and model factors.
Figure 3. Characteristics of discovered subgroups. (A) Discovered subgroup’s metadata profile. Each subgroup (column) is characterized by a set of metadata (row), and its effect (cells) is measured by Cohen’s h for categorical values and Cohen’s d for continuous ones. (B) This characterization enables us to understand the subgroups, identify key characteristics important to the classification models, and create performance gaps. For a few selected subgroups, we show four randomly selected images from this group, and their most distinctive characteristic.
Patient factors include metadata commonly used in traditional subgroup analyses, such as patient demographics and comorbidities. These appear relevant to some subgroups but only to a limited extent. The only ethnicity that showed a significant association was “Pacific Islander” on subgroup 10. Males and females showed a mild influence on five (one-third) groups. Medical findings, such as co-occurring lung lesions, showed mild to moderate impact. Of the 12 co-morbidities (excluding No Findings) annotated in CheXpert, only 3 showed significant associations, with Edema being the most common, appearing in 7 groups.
Acquisition factors were the most strongly associated with our discovered subgroups. Image characteristics such as Frontal/Lateral and AP/PA strongly influenced subgroups, as different views and projections consistently clustered together and were rarely assigned to mixed subgroups. Since these were often binary complementary values (e.g., frontal vs. lateral, AP vs. PA), a high positive association on one led to a high negative association on the other. External objects, such as support devices, were significantly associated with some of the identified subgroups. Since these objects appeared highly relevant to our investigated models, we included annotations of pacemakers in our analysis. For this, we used publicly available annotations [23], which covered 200 samples of our test set. Even if annotated for only a small subset of the dataset, pacemakers became a defining attribute in one of our worst-performing subgroups (SG A10).
Model factors, such as the classification scores, were a defining attribute, particularly for subgroups 5 and 10, the worst-performing subgroups. That shows that these subgroups contained samples with systematically higher predicted model scores (positive class predictions) than the other subgroups.
The metadata profiles described above and shown in Figure 3 allow an augmented and enriched version of traditional subgroup analysis. Based on the obtained metadata profiles for the discovered subgroups, we created an enriched subgroup analysis table (Table 2) to showcase how subgroup discovery could be implemented to complement traditional subgroup analysis for model validation. As an important limitation, the discovered subgroups could only be explained with metadata that were available during analysis. While we could show that discovered subgroups were strongly linked with some semantic features, some factors remained unknown. For example, subgroups A1 and A13 had similar profiles but differed by 6 percentage points in accuracy, and subgroups A4 and A14 differed by 24 percentage points in sensitivity. Subgroup discovery as a tool can therefore expose differences beyond available metadata, and could be used to trigger further investigations by domain experts to identify the missing defining attributes of these subgroups.
Use case: subgroup annotations for improving model robustness
Here, we present model debiasing as a downstream use case of subgroup discovery. Typically, model debiasing and domain generalisation rely on subgroup annotations (e.g., image domain, patient sex, presence of artifacts). The subgroup labels are then used during model training to improve worst-group performance and thus improve model robustness and fairness. Subgroup discovery can automatically generate such annotations and identify underperforming subgroups and potential biases, supporting subsequent model improvement through debiasing.
Manual inspection revealed a consistent presence of pacemakers in SG A10, which was also the second-worst-performing group. This suggests that the model relied on pacemaker features, leading to reduced performance. While pacemakers were not the only defining attribute of this low-performing subgroup, nor did the subgroup include all pacemaker samples, we hypothesized that improving robustness with respect to Subgroup A10 could mitigate performance drops in the pacemaker cohort. If effective, subgroup discovery could replace much of the manual annotation needed to identify artifacts such as pacemakers.
We used groupDRO [24], a standard domain generalization approach that minimizes the worst subgroup error. We created an annotation indicating whether a sample belonged to Subgroup A10. For evaluation, we used a publicly available partial annotation of pacemakers [23], which covered 200 samples in our test set. Although small, this subset reflects real-world conditions, as such annotations are costly to obtain. After hyperparameter tuning and training, the groupDRO model improved balanced accuracy on pacemaker cases by 3.2 percentage points (from 70.0 to 73.2). All training and validation, including hyperparameter tuning, used only the subgroup-based annotations, while the pacemaker reference labels were reserved for final evaluation on the test set.
Subgroup discovery passes sanity checks on synthetic data
In real-world medical imaging, detailed metadata are often scarce, and when available, they may not reflect the features that most influence the model’s predictions. Subgroup discovery is designed for exactly these situations, where few or no annotations exist. However, this makes its evaluation challenging, since we cannot rely on supervised ground truth to assess the discovered subgroups.
To address this, we created a synthetic version of CheXpert that simulates performance disparities while providing access to known, ground-truth subgroups. Following previous work [25], we added two clinically inspired artifacts correlated with the positive disease label: vertical lines of hyperintense signal and a hospital tag in the bottom-left corner. Each artifact combination defines a ground-truth subgroup (no artifacts, only vertical line, only tag, both). We trained the classification model and applied subgroup discovery on this biased dataset, without using any artifact information. Evaluation was then performed on an unbiased test set, in which artifacts were equally likely to appear in both positive and negative samples. This experiment tests whether subgroup discovery methods can (1) identify and isolate these subgroups and (2) capture the performance disparities introduced by the artifacts.
In Figure 4A, we showed that subgroup discovery successfully isolated all combinations of artifacts. The artifact concepts were only mildly mixed across subgroups (Figure 4B), because these are highly impactful features by design (high-bias scenario). As this synthetic data is an altered version of CheXpert, it also retains the dataset’s original characteristics, such as frontal/lateral and AP/PA, which might still affect subgroups’ composition. Here we had access to the metadata inherited from CheXpert, as shown in previous results, but we omitted them for clarity.
Figure 4. Sanity checks of subgroup discovery. (A) Effects of known simulated artifacts. Subgroup discovery successfully dedicates subgroups for each possible combination of the presence of the two artifacts. (B) Random samples from the found subgroups. (C) Detailed subgroup accuracies, reproduced from [21]. Darker dots represent found subgroups with higher effects. These capture the true performance gap characterized by hidden subgroups, which are overlooked by traditional subgroup analysis based on a single artifact (known subgroups), and by overall performance.
In Figure 4C we showed that subgroup discovery captured the ground-truth (based on the introduced artifacts) hidden subgroup performances. We contrasted the discovered subgroups’ performances with those calculated having access to none (“overall acc.” column), one (“known sgs.”), or both artifacts (“hidden sgs.”). Having none or one artifact annotated represents the more realistic scenario, as biases are often not annotated. Having all artifacts annotated serves as ground truth for the discovered subgroups. When only one artifact is known, subgroup analysis based on the known artifact unsurprisingly revealed increasing performance gaps, but missed the much larger, hidden performance gaps caused by the second artifact.
Feature extractors trained on natural images are sufficient for exposing meaningful performance gaps
In the main results presented above, we used a pretrained natural image feature extractor (CLIP [22]) in the subgroup discovery step. Next, we investigated whether representations learned from biomedical data led to better stratification of disease-related features in medical images. For this, we used Biomed-CLIP [26], a foundation model trained on a wide range of biomedical multimodal datasets, as a feature extractor for subgroup discovery.
Overall, BioMedCLIP and generalist CLIP led to similar performance disparities, but different metadata profiles (see Table 3 vs. Table 2). The subgroups obtained with BioMedCLIP were more strongly associated with comorbidities such as lung diseases. For generalist CLIP, no subgroup was linked to lung diseases, but BioMedCLIP found positive associations in 4 out of 15 subgroups. Further, it identified a subgroup of older patients but missed links with BMI, potentially because age was more often mentioned in radiology reports used for training the BioMedCLIP foundation model. Despite these differences, our results show that even feature extractors trained on natural images can expose meaningful performance gaps in real-world medical data distributions. The different subgroup characteristics call for combining external models in future work to yield even more diverse subgroups.
| SG | Category | Description | Accuracy | Sensibility | Specificity | AUROC |
| B4 | acquisition | Lateral | ||||
| B0 | patient, acquisition | Young, healthy, lateral | ||||
| B1 | acquisition | Lateral, no support devices | ||||
| B6 | patient, acquisition, model | PA, healthy, negative preds | ||||
| B2 | model | Positive preds | ||||
| B9 | acquisition, model | Pacemaker, PA, positive preds | ||||
| B8 | patient, model | Pleural effusion, consolidation, negative preds | ||||
| B13 | patient, model | Pleural effusion, negative preds | ||||
| B5 | acquisition | PA samples, unknown | ||||
| B14 | patient | Older patients | ||||
| B11 | patient | Healthy, unknown | ||||
| B3 | patient, acquisition | Lung opacity, effusion, support devices | ||||
| B10 | patient, acquisition, model | Lung opacity, consolidation, support devices, negative preds | ||||
| B7 | patient, model | Support devices, negative preds | ||||
| B12 | acquisition | Support devices, unknown |
Table 3. Discovered subgroups using BioMedCLIP at threshold 0.5. Subgroups are categorized into patient, acquisition, and model factors. Metrics are reported as mean half 95% CI. PA = posterior-to-anterior.
Discussion
Deep learning models frequently show systematic performance differences across patient populations, making careful subgroup analyses essential for uncovering these disparities. Traditional subgroup analysis relies on metadata such as sex or age to stratify performance, but these attributes often fail to align with the model’s underlying decision concepts. As a result, conventional approaches may miss important systematic differences. Subgroup discovery offers a way to identify more relevant groups, yet its evaluation remains difficult because ground-truth subgroup annotations are unavailable for real-world data.
We explored this challenging problem using a real-world chest X-ray dataset 1 chosen for its extensive metadata. Even under these unusually favourable conditions, we show that metadata alone does not adequately reveal meaningful subgroup disparities. In contrast, subgroup discovery uncovered groups exhibiting larger and statistically significant disparities across both prevalence-dependent metrics (e.g., accuracy) and prevalence-independent metrics (e.g., sensitivity, specificity, AUC). These findings extend prior work on subgroup discovery [^19, 21] by proposing a way to characterise found subgroups using existing metadata, which we call subgroup profiles, and by thoroughly evaluating the performance gaps beyond accuracy and performing a rigorous statistical analysis.
Revealing performance disparities in high-performing models addresses only part of the challenge. It is equally important to show that the identified subgroups are internally coherent, that is, that their samples share a meaningful, potentially interpretable concept. To support interpretability, we examined whether specific metadata values were overrepresented within each subgroup relative to all remaining samples. Interpretability is a multifaceted concept 2 involving content properties (fidelity to the model’s internal behaviour), presentation properties (how explanations are communicated), and userfacing properties (whether explanations are practically relevant to the end user). Our pipeline contributes to each axis: the statistical analysis and sanity checks establish content fidelity; the metadata profiling translates subgroup structure into clinically legible terms; and the actionability analysis below assesses user-facing relevance to determine whether and how to act.
The EU AI Act [28] operationalizes this final step for high-risk medical AI, requiring that systems enable deployers to "interpret outputs correctly and use the system appropriately" (Article 13), and establishing further obligations for data governance (Article 10), robustness (Article 15), human oversight (Article 14), and post-market surveillance (Article 72).
Regardless of whether a subgroup can be fully characterised, the discovery of a performance disparity enables an immediate response: flagging the affected group for deferral to specialist review and for stratified performance monitoring, addressing human oversight and post-market surveillance obligations. For example, SG A0, associated with high-BMI males, shows a sensitivity of 0.10 and a specificity of 0.95, indicating that positive cases in this group are rarely detected. A similar pattern appears in SG B14, associated with older patients (sensitivity 0.17, specificity 0.97). In both cases, patients with these characteristics can be flagged for manual assessment rather than relying on AI-based predictions, preventing potentially life-threatening missed diagnoses.
Beyond deferral, the recommended intervention depends on how a subgroup’s characteristics interact with the classification target. This requires domain expertise and may involve both clinicians and computer scientists. We distinguish three scenarios. Firstly, the factors associated with a subgroup can be clinically relevant to the disease (e.g., patient BMI and age). In this case, subgroup-level model recalibration may be appropriate to improve the model and mitigate the performance gap. Secondly, the associated factor introduces systematic imaging distortions that compromise diagnostic accuracy, for instance, differences between AP and PA chest projections [17]. Here, an appropriate short-term response may be to flag the patient for additional imaging to reduce ambiguity. In the longer term, such findings motivate targeted robustness testing and corrections. Thirdly, the identified characteristic may confound the classifier, as with the pacemaker-enriched SG A10. Here, targeted domain adaptation using subgroup labels can mitigate the disparity [29], as we demonstrated in our proof-of-concept experiment. In each case, the subgroup discovery can identify potential problem sources, and specialist assessment determines the appropriate response.
Some subgroups in our analyses could not be attributed to any annotated clinical, demographic, or acquisition factor. While metadata-based characterisation reduces the burden of manual interpretation and scales to large datasets, it is inherently bounded by what is annotated. For underperforming subgroups with no apparent clinical explanation, the appropriate response starts with a domain expert examining the flagged samples to determine whether they share a concept not captured by available metadata annotations. Once such a concept is identified, it may be linked to one of the scenarios and appropriate interventions described above. If no unifying concept emerges, as with SG A5, whose systematically inflated classifier confidence persists after controlling for all annotated factors, or SG A6, which shows no strong association with any specific characteristic, the discovered subgroups remain actionable: they provide a basis for deferral given their known elevated error rate, and they serve as challenging holdout subgroups for ongoing model validation.
Several limitations should be considered. The metadata-based characterisation, while extensive by the standards of publicly available datasets, is constrained by what was annotated. Factors not captured in the metadata (e.g., disease stratifications, patient positioning details) may drive subgroup structure that our profiling cannot expose. Finally, our analysis identified associations between subgroup membership and metadata factors, but it does not establish causal mechanisms for the observed performance gaps.
We posit that the current study represents a significant advance in the ability of model developers and evaluators to better assess the safety and effectiveness of a model across a range of patient populations. Subgroup discovery can unveil ongoing problems and provide informative labels for model debiasing. We conclude that subgroup discovery should be highly relevant as a performance monitoring and reporting tool, and argue that it should accompany traditional subgroup analysis as an additional safeguard during real-world ML validation and deployment.
Methods
We seek subgroup divisions that expose large systematic performance gaps in a target classification model while preserving subgroup cohesion, so that the model’s performance within a subgroup can be attributed to a shared characteristic. Metadata-based subgroups are inherently cohesive because a semantic concept, such as patient age or sex, defines the division. However, we hypothesise that these attributes do not adequately reflect the main factors that affect model performance, which often results in relatively small performance gaps (Figure 1a). Instead, we propose to use subgroup discovery techniques (Figure 1b) for subgroup performance analysis in hidden stratifications (Figure 1c).
First, we train the classification model (see “Experimental setup”) for detecting cardiomegaly following the train/val/test splits. Then, we fit a simplified version [21] of DOMINO [19] for subgroup discovery on the validation set (see “Preliminaries: subgroup discovery algorithm”) to identify latent subgroups. Finally, we predict the subgroup assignments of test samples for subsequent analysis.
We propose a strategy to evaluate subgroup discovery for performance analysis for the first time in a real-world medical data distribution (Figure 1e). Here, we first calculate performance gaps using different metrics and performing statistical tests (see “Statistical analysis of subgroup performance”). Then, we use existing metadata to characterize the discovered subgroups (see “Statistical analysis of metadata profiles”). As a sanity check, we inject synthetic artifacts to create clinically-inspired subgroups (see “Synthetic scenario with generated artifacts”) for which ground-truth model performance is available (Figure 1d).
Dataset description
We selected a dataset that provides comprehensive metadata coverage. CheXpert-Plus [18] is an extension of CheXpert [30] and provides metadata that allows for a challenging comparison to our discovered subgroups. The metadata includes patient demographics (e.g., sex, age), comorbidities (e.g., edema, fracture), and artifacts, totalling 20 attributes. Our training, validation, and test sets follow an 80/10/10 division, with a total of 178,684 / 22,263 / 22,281 images respectively.
Preliminaries: subgroup discovery algorithm
We use DOMINO [19], a simple yet effective approach for subgroup discovery. First, a feature representation is extracted from each image using an external pretrained model such as CLIP [22], followed by dimensionality reduction using principal component analysis. In addition, softmax predictions are obtained from the target classification model. While the model predictions encapsulate characteristics important for the classification task, the external model helps identify task-agnostic features such as artifacts. Next, the samples are clustered into subgroups using a generalised Gaussian Mixture Model (GMM) by minimising the following objective (similar to [19]):
where balances the influence of predicted labels and embeddings in the slicing decision. In contrast to the original DOMINO [19], we remove classification labels in the GMM, enabling subgroup discovery in post-deployment scenarios with unlabeled test sets. We use explicit validation and test set splits, fitting DOMINO on the validation set and inferring subgroups on the test set.
Statistical analysis of subgroup performance
We computed subgroup performance metrics by estimating prevalence-dependent (accuracy), and independent metrics (sensitivity, specificity, and AUROC) within each predefined or discovered subgroup. We evaluated accuracy, sensitivity, and specificity at a fixed decision threshold of 0.5, and obtained their confidence intervals using Wilson’s score intervals at the 95% level. For AUROC, we report a 95% bootstrap interval (2.5th and 97.5th percentiles) using 1,000 resamples drawn with replacement within each subgroup. Resamples that contained only one class were skipped.
To assess disparities, each subgroup was compared to its complement (all remaining examples not belonging to the subgroup). For sensitivity, specificity, and accuracy, we used a two-sample test for proportions by taking the difference in the rates, dividing that difference by its estimated standard error, and obtaining a value that reflects how far apart the two rates are under the assumption that they are truly equal. Analogously, for AUROC, we use DeLong’s method for two independent samples. Since we conducted many subgroup comparisons, we applied a Bonferroni correction to the -values to keep the false-positive rate below 0.05. We considered a difference significant when the corrected -value was below this threshold.
Statistical analysis of metadata profiles
To characterize the discovered subgroups with semantic information, we used the available metadata to create metadata profiles describing how each attribute relates to each subgroup. For each subgroup, we compared the distribution of each metadata attribute within the subgroup with that of the rest of the dataset. For categorical attributes (e.g., sex, race, view position), we tested each category separately (e.g., sex = male) against the combined set of all other categories. Missing categorical values were recoded as an explicit “Unknown” category and included in the analysis. For CheXpert comorbidities, missing and negative labels were treated as absent, while uncertain labels were kept as a separate level. Differences in proportions were measured using Cohen’s h, where positive values indicate that a category is more common within the subgroup and negative values indicate it is less common. For continuous attributes (e.g., age, BMI), we computed Cohen’s d, the standardized mean difference between the samples within and outside the subgroup, with positive values indicating higher means within the subgroup.
Statistical significance was assessed using a chi-square test for categorical and an independent-samples t-test for continuous variables. We applied the Benjamini–Hochberg false discovery rate (FDR) correction across all subgroup–attribute pairs and considered results significant at . Only attributes with both significant results and an absolute effect size of at least 0.3 were retained for visualization. All significant standardized effect sizes were compiled into a heatmap (Figure 3), where each column corresponds to a discovered subgroup and each row to a metadata attribute. Colors represent the direction and strength of association after standardization within metric type (Cohen’s h or d). Positive values indicate attributes that are more common or higher within a subgroup, and negative values indicate attributes that are less common or lower.
Synthetic scenario with generated artifacts
As a sanity check, we evaluate subgroup discovery in a simulated scenario where ground truth subgroups and subgroup performances are known. To simulate performance disparities, we add artifacts spuriously correlated with the positive disease label, similar to standard practice in shortcut learning research [31, 32, 33, 25]. We adapt CheXpertPlus with two clinically-inspired artifacts following previous work [25] (Figure 1b): a hospital tag on the bottom left, and vertical lines of hyperintense signal.
The two artifacts are independently correlated with the label: each is inserted into positive samples with probability and into negative samples with probability , yielding four ground-truth subgroups. Training and validation sets are generated from this biased version of the data and used to train the target classification model and select its hyperparameters. The validation set is also used to fit DOMINO. For testing, we use an unbiased test set (), facilitating fair comparisons across training bias levels.
Experimental setup
For all experiments, we trained ResNet-50 [34] classification models using SGD and searched over learning rates of with the weight decay of . Models were selected based on validation balanced accuracy. We chose the task of “cardiomegaly vs. all”. For subgroup discovery, we always used 15 subgroups. The external model is the pretrained CLIP [22] for all scenarios, and we included BiomedCLIP [26] for our real-world experiments. We used the publicly available versions of these pretrained models from the OpenAI repository and HuggingFace, respectively. In the synthetic scenario, we considered hypersensitivities as a known attribute and hospital tags as a hidden stratification and set the bias level as .
Perspective: regulatory implications
Subgroup discovery has direct implications for regulatory oversight of clinical AI systems. To illustrate this connection, we consider how a chest X-ray classification tool developed as Software as a Medical Device would operate under the Medical Device Regulation (MDR). The MDR requires manufacturers to maintain a post-market surveillance system that continually assesses model performance and safety throughout the device lifetime (Article 83(1) MDR). This includes procedures for identifying situations in which the device may place patients at increased risk and for taking appropriate action when such concerns arise (Article 83(4) MDR). Subgroup discovery supports these obligations by revealing groups of patients for whom performance differs substantially from the overall model behaviour. It complements rather than replaces evaluations based on predefined socio-demographic groups, as it can uncover additional patient clusters with meaningful performance gaps, thereby strengthening trustworthiness.
As part of this surveillance process, the manufacturer will collect real-world data from routine clinical use. Subgroup discovery can be applied to model outputs on the validation dataset to partition samples into groups that show distinct performance profiles. This method requires only model outputs and a feature representation extracted by a separate model, so it can be deployed without retraining or access to model internals. Once subgroups are defined, the manufacturer can infer subgroup membership for new patients and determine whether an individual belongs to a group with lower validated accuracy. If this occurs, it can serve as an early warning signal that the model may not be reliable for that specific case. The associated metadata profiles provide additional context, enabling clinicians to assess whether the patient exhibits characteristics associated with poor performance and to decide whether to use or override the model output.
If an alert is raised because a new patient belongs to a subgroup with lower validated performance, it would trigger the procedures described in the manufacturer’s post-market surveillance plan (Article 84 of the MDR). The manufacturer would assess whether these alerts reflect isolated cases or indicate a broader limitation of the model. If corrective measures are required, such as model updates, our bias mitigation experiment shows that subgroup discovery can supply the labels needed for targeted retraining that improves performance for a clinically important group.
Data availability
The CheXpertPlus dataset used in the current study is publicly available at https://aimi.stanford.edu/datasets/chexpert-plus. A list of CheXpert images labeled for the presence of pacemakers, which we used for validation of our method, is publicly available at https://github.com/nina-weng/FastDiME_Med/blob/master/shortcut_annotations/che_pm_shortcut_labels.csv.
Code availability
The code for training and evaluating the models used in this work is available at https://github.com/alceubissoto/hidden-sg-disparities.
Acknowledgments
Calculations were performed on UBELIX (https://www.id.unibe.ch/hpc), the HPC cluster at the University of Bern. This work was supported by the Diabetes Center Berne (AB, TH, TF, LK); the Carl Zeiss Foundation in the project “Certification and Foundations of Safe Machine Learning Systems in Healthcare” (SS); the Deutsche Forschungsgemeinschaft (DFG), EXC number 2064/1, Project number 39072764 (SS, CB); strategic funding of the medical faculty of the University of Bern (TH); DTU Compute, the Technical University of Denmark (NW); Otto Mønsteds Fond (NW); IDA and Berg-Nielsen Study and Support Fund (NW); and DTU Travel Stipend (NW).
Author Contributions
A.B. and L.K. designed the study. A.B. performed the experiments and analyses and wrote the first draft of the manuscript. L.K. supervised the project and acquired funding. T.H., T.F., and N.W. contributed to methodological discussions. All authors (A.B., T.H., T.F., N.W., S.S., C.B., L.K.) contributed to the interpretation of the results and reviewed, edited, and approved the final manuscript.
Competing Interests
The authors declare no competing interests.
References
- [1]U.S. Food and Drug Administration, Ai-enabled medical devices, https://www.fda.gov/medical-devices/software-medical-device-samd/artificial-intelligence-enabled-medical-devices, accessed: 28 August 2025 (2025). 1
- [2]T. Zack, E. Lehman, M. Suzgun, J. A. Rodriguez, L. A. Celi, J. Gichoya, D. Jurafsky, P. Szolovits, D. W. Bates, R.-E. E. Abdulnour, et al., Assessing the potential of gpt-4 to perpetuate racial and gender biases in health care: a model evaluation study, The Lancet Digital Health 6 (1) (2024) e12–e22.
- [3]L. Seyyed-Kalantari, H. Zhang, M. B. McDermott, I. Y. Chen, M. Ghassemi, Underdiagnosis bias of artificial intelligence algorithms applied to chest radiographs in under-served patient populations, Nature medicine 27 (12) (2021) 2176–2182.
- [4]E. Christodoulou, A. Reinke, R. Houhou, P. Kalinowski, S. Erkan, C. H. Sudre, N. Burgos, S. Boutaj, S. Loizillon, M. Solal, et al., Confidence intervals uncovered: Are we ready for real-world medical imaging ai?, in: International Conference on Medical Image Computing and Computer-Assisted Intervention (MICCAI), Springer, 2024, pp. 124–132.arxiv.org/abs/2409.17763
- [5]L. M. Koch, C. F. Baumgartner, P. Berens, Distribution shift detection for the postmarket surveillance of medical ai algorithms: A retrospective simulation study, npj Digital Medicine (2024).DOI
- [6]L. Oakden-Rayner, J. Dunnmon, G. Carneiro, C. Re, Hidden stratification causes clinically meaningful failures in machine learning for medical imaging, in: ACM Conference on Health, Inference, and Learning, 2020.
- [7]S. G. Finlayson, A. Subbaswamy, K. Singh, J. Bowers, A. Kupke, J. Zittrain, I. S. Kohane, S. Saria, The Clinician and Dataset Shift in Artificial Intelligence, New England Journal of Medicine 385 (3) (2021) 283–286.DOI
- [8]A. J. Larrazabal, N. Nieto, V. Peterson, D. H. Milone, E. Ferrante, Gender imbalance in medical imaging datasets produces biased classifiers for computer-aided diagnosis, Proceedings of the National Academy of Sciences 117 (23) (2020) 12592–12594.DOI
- [9]N. Weng, S. Bigdeli, E. Petersen, A. Feragen, Are sex-based physiological differences the cause of gender bias for chest x-ray diagnosis?, in: Workshop on Clinical Image-Based Procedures, Springer, 2023, pp. 142–152.arxiv.org/abs/2308.05129
- [10]A. Jiménez-Sánchez, D. Juodelyte, B. Chamberlain, V. Cheplygina, Detecting shortcuts in medical images-a case study in chest x-rays, in: 2023 IEEE 20th International Symposium on Biomedical Imaging (ISBI), IEEE, 2023, pp. 1–5.
- [11]V. Olesen, N. Weng, A. Feragen, E. Petersen, Slicing through bias: explaining performance gaps in medical image analysis using slice discovery methods, in: MICCAI Workshop on Fairness of AI in Medical Imaging, Springer, 2024, pp. 3–13.
- [12]A. J. DeGrave, J. D. Janizek, S.-I. Lee, Ai for radiographic covid-19 detection selects shortcuts over signal, Nature Machine Intelligence 3 (7) (2021) 610–619.DOI
- [13]G. S. Collins, K. G. Moons, P. Dhiman, R. D. Riley, A. L. Beam, B. Van Calster, M. Ghassemi, X. Liu, J. B. Reitsma, M. Van Smeden, et al., Tripod+ ai statement: updated guidance for reporting clinical prediction models that use regression or machine learning methods, bmj 385 (2024).DOI
- [14]K. Lekadir, A. F. Frangi, A. R. Porras, B. Glocker, C. Cintas, C. P. Langlotz, E. Weicken, F. W. Asselbergs, F. Prior, G. S. Collins, et al., Future-ai: international consensus guideline for trustworthy and deployable artificial intelligence in healthcare, bmj 388 (2025).DOI
- [15]K. G. Moons, J. A. Damen, T. Kaul, L. Hooft, C. A. Navarro, P. Dhiman, A. L. Beam, B. Van Calster, L. A. Celi, S. Denaxas, et al., Probast+ ai: an updated quality, risk of bias, and applicability assessment tool for prediction models using regression or artificial intelligence methods, bmj 388 (2025).DOI
- [16]V. Muralidharan, B. A. Adewale, C. J. Huang, M. T. Nta, P. O. Ademiju, P. Pathmarajah, M. K. Hang, O. Adesanya, R. O. Abdullateef, A. O. Babatunde, et al., A scoping review of reporting gaps in fda-approved ai medical devices, NPJ Digital Medicine 7 (1) (2024) 273.DOI
- [17]S. Raoof, D. Feigin, A. Sung, S. Raoof, L. Irugulapati, E. C. Rosenow III, Interpretation of plain chest roentgenogram, Chest 141 (2) (2012) 545–558.DOI
- [18]P. Chambon, J.-B. Delbrouck, T. Sounack, S.-C. Huang, Z. Chen, M. Varma, S. Q. Truong, C. T. Chuong, C. P. Langlotz, Chexpert plus: Augmenting a large chest x-ray dataset with text radiology reports, patient demographics and additional image formats, arXiv preprint arXiv:2405.19538 (2024). 2, 3, 5, 9, 11arxiv.org/abs/2405.19538
- [19]S. Eyuboglu, M. Varma, K. K. Saab, J.-B. Delbrouck, C. Lee-Messer, J. Dunnmon, J. Zou, C. Re, Domino: Discovering systematic errors with cross-modal embeddings, in: International Conference on Learning Representations, 2022. 2, 3, 9, 11arxiv.org/abs/2203.14960
- [20]A. Subbaswamy, B. Sahiner, N. Petrick, V. Pai, R. Adams, M. C. Diamond, S. Saria, A data-driven framework for identifying patient subgroups on which an ai/machine learning model may underperform, npj Digital Medicine 7 (1) (2024) 334. 2DOI
- [21]A. Bissoto, T.-D. Hoang, T. Flühmann, S. Sun, C. F. Baumgartner, L. M. Koch, Subgroup performance analysis in hidden stratifications, in: International Conference on Medical Image Computing and Computer-Assisted Intervention (MICCAI), Springer, 2025, pp. 594–603. 2, 3, 8, 9, 11, 16
- [22]A. Radford, J. W. Kim, C. Hallacy, A. Ramesh, G. Goh, S. Agarwal, G. Sastry, A. Askell, P. Mishkin, J. Clark, et al., Learning transferable visual models from natural language supervision, in: International Conference on Machine Learning (ICML), PMLR, 2021, pp. 8748–8763. 4, 8, 11, 12arxiv.org/abs/2103.00020
- [23]N. Weng, P. Pegios, E. Petersen, A. Feragen, S. Bigdeli, Fast diffusion-based counterfactuals for shortcut removal and generation, in: European Conference on Computer Vision (ECCV), Springer, 2024, pp. 338–357. 6, 7arxiv.org/abs/2312.14223
- [24]S. Sagawa, P. W. Koh, T. B. Hashimoto, P. Liang, Distributionally robust neural networks for group shifts: On the importance of regularization for worst-case generalization, in: International Conference on Learning Representations (ICLR), 2020. 7arxiv.org/abs/1911.08731
- [25]S. Sun, L. M. Koch, C. F. Baumgartner, Right for the wrong reason: Can interpretable ml techniques detect spurious correlations?, in: Proc. Medical Image Computing and Computer Assisted Interventions MICCAI, 2023. 7, 12arxiv.org/abs/2307.12344
- [26]S. Zhang, Y. Xu, N. Usuyama, H. Xu, J. Bagga, R. Tinn, S. Preston, R. Rao, M. Wei, N. Valluri, C. Wong, A. Tupini, Y. Wang, M. Mazzola, S. Shukla, L. Liden, J. Gao, A. Crabtree, B. Piening, C. Bifulco, M. P. Lungren, T. Naumann, S. Wang, H. Poon, A multimodal biomed-ical foundation model trained from fifteen million image–text pairs, NEJM AI 2 (1) (2024). doi:10.1056/AIoa2400640. URL https://ai.nejm.org/doi/full/10.1056/AIoa2400640 8, 12DOI
- [28]Regulation (eu) 2024/1689 of the european parliament and of the council of 13 june 2024 laying down harmonised rules on artificial intelligence and amending regulations (ec) no 300/2008, (eu) no 167/2013, (eu) no 168/2013, (eu) 2018/858, (eu) 2018/1139 and (eu) 2019/2144 and directives 2014/90/eu, (eu) 2016/797 and (eu) 2020/1828 (artificial intelligence act) (text with eea relevance) (Jul 2024). 10
- [29]A. Alloula, C. Jones, B. Glocker, B. Papiez, Subgroups matter for robust bias mitigation, in: International Conference on Machine Learning (ICML), 2025. URL https://openreview.net/forum?id=P0RkH1RT5z 10arxiv.org/abs/2505.21363
- [30]J. Irvin, P. Rajpurkar, M. Ko, Y. Yu, S. Ciurea-Ilcus, C. Chute, H. Marklund, B. Haghgoo, R. Ball, K. Shpanskaya, et al., Chexpert: A large chest radiograph dataset with uncertainty labels and expert comparison, in: AAAI Conference on Artificial Intelligence, 2019. 11arxiv.org/abs/1901.07031
- [31]M. Roschewitz, R. Mehta, C. Jones, B. Glocker, Automatic dataset shift identification to support root cause analysis of ai performance drift, arXiv preprint arXiv:2411.07940 (2024). 12
- [32]N. Bayasi, J. Fayyad, A. Bissoto, G. Hamarneh, R. Garbi, Biaspruner: Debiased continual learning for medical image classification, in: International Conference on Medical Image Computing and Computer-Assisted Intervention (MICCAI), Springer, 2024, pp. 90–101. 12arxiv.org/abs/2407.08609
- [33]A. Bissoto, C. Barata, E. Valle, S. Avila, Even small correlation and diversity shifts pose dataset-bias issues, Pattern Recognition Letters 45 (2024). 12
- [34]K. He, X. Zhang, S. Ren, J. Sun, Deep residual learning for image recognition, in: Proc. Conference on Computer Vision and Pattern Recognition (CVPR), 2016. 12arxiv.org/abs/1512.03385