Counterfactual Marginalisation: Framework for Evaluating Robustness to Nuisance Variables

Introduction

Deep learning models have achieved strong performance in medical image analysis, but high accuracy on held-out data does not guarantee clinically appropriate decision rules. In practice, models may rely on demographic, acquisition-related, or site-specific shortcuts rather than pathology-relevant evidence [12]. Causal modelling provides a principled way to study these issues by asking, for example, how a prediction would change if a patient’s demographic attributes were intervened upon while other patient-specific factors were held fixed [21]. Recent work has used Structural Causal Model (SCM)-based generative models to produce medical image CFs for explanation, auditing and augmentation [20, 22, 4, 9]. Here, instead, we use CFs at test time to define evaluation metrics for medical image classifiers. Given an SCM-based CF generator, we intervene on nuisance parent variables such as age or sex, and generate CF versions of each test image [9]. We then develop a range of metrics that use these CFs to measure the sensitivity, stability and performance of a predictor. This setting is closely related to work on fairness and robustness in medical imaging [12, 16], where prior studies have shown that models can exhibit demographic disparities and can recover sensitive attributes directly from images [28, 2]. Our contributions are as follows:

• We formulate counterfactual marginalisation over variables as a test-time evaluation procedure for medical image classifiers.

• We introduce intervention-aware evaluation metrics, including counterfactual marginal risk, interventional expected risk, counterfactual stability, and worst-case counterfactual risk. This extends counterfactuals from explanation and augmentation to principled robustness evaluation.

• We demonstrate that our approach exposes biases in predictors better than traditional metrics and also enables measuring sensitivity to nuisance variables without requiring disease labels.

Background

Point-estimate evaluation can miss brittle model behaviour: small, plausible changes to inputs or logits may alter predictions. This concern echoes work on flat minima, where models in wider low-loss regions tend to generalise better than sharp solutions [8, 5]. Ensembles and Bayesian model averaging address a related issue by marginalising over parameters to improve robustness and uncertainty estimates [13, ?, 26], while test-time augmentation similarly averages predictions over transformed images [1, 14]. Our counterfactual marginalisation can be viewed as a causal analogue: instead of averaging over ad-hoc image transformations, we average over SCM-defined interventions on parent variables such as age or sex. This in turn motivates the design of metrics that assess not only average performance, which often is insufficient for describing clinical utility [19], but also metrics that measure sensitivity, calibration, and worst-case behaviour across clinically meaningful perturbations.

Counterfactual Marginalisation for Evaluation

Let $g_{\theta}(x) \in[0,1]$ denote the predicted probability of disease for image $x$ and $y \in\{0,1\}$ the corresponding ground-truth label. We partition the parent variables as $\mathrm{pa}=(\mathbf{a},\mathbf{c})$, where $\mathbf{a}$ is the subset of parent variables to be marginalised, e.g. age and sex, and $\mathbf{c}$ denotes the remaining parent variables that are kept fixed. Given an observed sample $(x_i,\mathbf{a}_i,\mathbf{c}_i,y_i)$, we generate counterfactual images by intervening on $\mathbf{a}$ while preserving the inferred exogenous noise for that individual through the standard abduction–action–prediction procedure. This yields a set of counterfactual images $\tilde{x}_i(\tilde{\mathbf{a}}) \sim P_{x \mid\operatorname{do}(\mathbf{a}=\tilde{\mathbf{a}}),\mathbf{c}_i,u_i}$, where $u_i$ denotes the abducted exogenous variables for sample $i$. To remove the influence of the nuisance demographic variables, we define a target distribution $q(\tilde{\mathbf{a}})$ over interventions. This may be chosen as the empirical test-set distribution, a uniform distribution over demographic strata, or an external population distribution. We then define the counterfactually marginalised prediction for sample $i$ as

In practice, this expectation is approximated by Monte Carlo sampling:

Counterfactual Risk

Using (1), we define the counterfactual marginal risk and interventional expected risk as

where $L_i(\tilde{a}) = \ell\left(g_\theta\left(\tilde{x}_i(\tilde{a})\right), y_i\right)$ and $\ell$ is binary cross-entropy or another appropriate loss. The former evaluates predictive performance after marginalising out the nuisance demographic variables, while the latter averages the loss over counterfactual interventions before aggregation. To measure the extent to which the predictor is vulnerable to the most adverse demographic interventions, we consider a tail-risk measure based on the conditional value at risk (CVaR). For a chosen tail level $\alpha\in(0, 1]$, the counterfactual tail risk is

where $\mathrm{CVaR}_{\alpha}$ is the expected loss within the worst $\alpha$-fraction of interventions under $q(\tilde{a})$. Let $\mathcal{R}_{\mathrm{WC}}=\mathcal{R}_{\mathrm{CVaR},0}$ denote the risk of the worst-case intervention.

Counterfactual Calibration

While $\mathcal{R}_{\mathrm{CM}}$ and $\mathcal{R}_{\mathrm{IE}}$ measure predictive accuracy under counterfactual marginalisation, they do not directly assess whether the resulting probabilities are well calibrated. To handle this, we report the counterfactual Brier score and counterfactual expected calibration error (ECE):

where $I_{1},\ldots,I_{B}$ denote bins of samples partitioned according to their marginalised predicted probabilities $\bar{p}_{i}$. These measure whether the probabilities obtained after marginalisation remain clinically interpretable as calibrated risk estimates.

Counterfactual Stability

We define a counterfactual stability score based on the variation of the model output across interventions:

A low value of $S_{\mathrm{var}}$ indicates that the predicted probability is stable under changes in the nuisance demographic variables. For threshold-based clinical decision making, we additionally define a counterfactual flip rate

where $\tau$ is a fixed decision threshold. This metric captures how often a prediction changes solely because of an intervention on the nuisance parent variables. We also measure how often the binary decision made on the observed image differs from the decision made using the counterfactually marginalised prediction. This observed–marginalised disagreement rate is defined as

This metric quantifies the extent to which conventional observational evaluation would lead to a different clinical decision than intervention-aware evaluation. For tasks in which the predicted score is used for prioritisation or triage, it is important that the relative ranking of patients remains stable after marginalisation [3, 17]. Let $r_{i}$ denote the rank of the observed prediction $g_{\theta}(x_{i})$ across the test set, and let $\bar{r}_{i}$ denote the rank of the marginalised prediction $\bar{p}_{i}$. We quantify counterfactual rank stability using the Spearman rank correlation measure:

Theoretical Advantages of Counterfactual Evaluation

As well as the clear advantage of being able to generate samples from underrepresented classes, we also have that for quantities that are identified observationally, such as the mean prediction within a demographic subgroup, the CF estimator achieves lower variance by exploiting the full test set rather than the subgroup alone. Let $z$ denote a particular attribute value (e.g. $A = \mathrm{Female}$), $n_z = |\{i : a_i = z\}|$ the number of test samples in that subgroup, and $n$ the total test size. The standard observational estimator of $\mu_z = \mathbb{E}[g(X) \mid A = z]$ is

The CF estimator generates a counterfactual image at $A = z$ for every test patient and averages: $\hat{\mu}_z^{\mathrm{CF}} = \frac{1}{n}\sum_{i=1}^{n}g(\tilde{x}_i(z))$. Under correct SCM specification, $\hat{\mu}_z^{\mathrm{CF}}$ is unbiased for $\mu_z$. Its variance decomposes via the law of total variance, conditioning on the abducted exogenous noise $U_i$:

The first term is the variance of the posterior mean (signal); the second is the expected posterior variance (SCM generation noise $\sigma_{\mathrm{gen}}^2$). The exogeneous noise $U$ accounts for all unobserved background variables and individual traits, serving as a sufficient statistic for individual-level causal queries. Then, by the Rao-Blackwell theorem, $\operatorname{Var}[\mathbb{E}[g(\tilde{X}(z)) \mid U]] \leq\operatorname{Var}[g(\tilde{X}(z))] = \operatorname{Var}[g(X) \mid A=z]$, so

Proposition 1 (Variance reduction). Let $\sigma^2 = \operatorname{Var}[g(X) \mid A = z]$. The CF estimator satisfies

whenever $\sigma_{\mathrm{gen}}^2 \leq\sigma^2(1 - n_z/n)$. The advantage grows as the subgroup becomes rarer: as $n_z/n \to0$, the bound reduces to $\operatorname{Var}[\hat{\mu}_z^{\mathrm{CF}}] \leq\sigma^2/n + \sigma_{\mathrm{gen}}^2/n \ll\sigma^2/n_z$.

This is directly analogous to the efficiency gain of a within-subjects design over a between-subjects design in experimental settings [23].

Experiments

Datasets

To demonstrate the utility of our metrics, we use two large scale chest x-ray datasets, CheXpert [10] and MIMIC-CXR [11]. We perform binary classification of Pleural Effusion (PE), existence or not. To ensure a controlled evaluation, we balance the test split so that male and female patients appear in equal numbers, and disease prevalence is equalised across sex and 3 age groups ($<45$, $45$-$65$, $>65$). The final balanced test sets for CheXpert (C) and MIMIC (M) contain 5,180 and 28,189 studies respectively. We use these two datasets to run three experimental settings: (1) Train and test the PE classifier on C; (2) Train and test on M; (3) Train on C and test on $M_{\mathrm{nat}}$ with natural demographic distributions. For all 3 settings we train the CF generator on C.

Predictors and Training Settings

The causal graph encodes the following parents of the image: sex, age, race, lung size, and pleural effusion. All experiments vary the sex attribute ($\tilde{a} \in\{\mathrm{Male}, \mathrm{Female}\}$) and age (8 uniformly-spaced values from 20 to 90 years), yielding $K = 2 \times8 = 16$ counterfactual images per patient. The marginalisation distribution $q(\tilde{A})$ is uniform over the $K$ intervention values in all experiments. The baseline predictor (basic) is trained on the full training set without any demographic manipulation. To study how well our method assesses bias, we create a first set of biased predictors by adding a fixed scalar offset $\beta\in\{0.5, 1.0, 2.0\}$ to the baseline logit for female patients at test time: $g_{\beta}(x,s) = \sigma(\operatorname{logit}(g_{0}(x)) + \beta\cdot s)$, where $s \in\{0,1\}$ is the sex label. Because the offset is a monotone transformation applied uniformly within each sex group, within-group rankings are preserved and hence stratified AUC is invariant to $\beta$ by construction. We then train three sex-biased predictors on a corrupted dataset in which Male+disease and Female+healthy samples are dropped with probabilities 0.8, 0.9, and 0.95 respectively, inducing a spurious correlation between sex and PE. We also train predictors with a continuous age bias that induces a correlation between older age and positive PE. For each training sample with normalised age $a \in[0,1]$ and binary PE label $y$, the concordance score is defined as $c_i = a_i \cdot1[y_i = 1] + (1-a_i) \cdot1[y_i = 0]$, and a sample is removed from training with probability $\beta_{\mathrm{age}} \cdot(1-c_i)$, where $\beta_{\mathrm{age}} \in\{0.8,0.9,0.95\}$ controls the strength of the induced correlation.

Results

tbl:3a shows that in the CheXpert experiment (Setting 1), our CF metrics discern the biases in the various predictors better than the standard risk $R_{\mathrm{orig}}$ (cross entropy) calculated only on original data, which sometimes decreases when bias increases (e.g. Logit+0.5 $\to$ Logit+1 $\to$). To summarise comparisons, we define the Correct Transition Count (CTC) as the number of transitions with increasing predictor bias (out of 9 total: 3 for logit bias, Basic $\to$ Logit+0.5 $\to$ +1 $\to$ +2, and similarly 3 for sex and age bias) where the risk also increases. Higher is better and our counterfactual metrics outperform the standard risk estimation $R_{\mathrm{orig}}$. All results shown are averages of 3 repetitions per experiment with varying seeds. Similar results were found for setting 2 and 3 (See Appendix A for details).

Moreover, biased predictors should be worse calibrated [18]. Figure 2 compares the original ECE [7] and marginalised ECE ([]{#eq:7}) for each predictor, where we find that scores of the latter increase more consistently with increasing predictor bias. This suggests that CF marginalised ECE can better estimate model calibration.

Predictor	$R_{\mathrm{orig}}$	$R_{\mathrm{CM}}$	$R_{\mathrm{IE}}$	$R_{\mathrm{CVaR}(0.50)}$	$R_{\mathrm{CVaR}(0.25)}$	$R_{\mathrm{CVaR}(0.10)}$	$R_{\mathrm{WC}}$
Basic	0.444	0.501	0.594	1.002	1.326	1.549	1.852
Logit +0.5	0.445	0.612	0.724	1.027	1.346	1.597	1.845
Logit +1	0.443	0.629	0.771	1.068	1.424	1.699	1.962
Logit +2	0.502	0.677	0.919	1.072	1.431	1.716	1.997
Sex ($\alpha= 0.80$)	0.453	0.610	0.687	1.192	1.618	1.991	1.390
Sex ($\alpha= 0.90$)	0.452	0.614	0.699	1.195	1.612	1.968	2.338
Sex ($\alpha= 0.95$)	0.443	0.614	0.699	1.203	1.622	1.982	2.361
Age ($\alpha= 0.80$)	0.494	0.613	0.750	1.126	1.511	1.817	2.111
Age ($\alpha= 0.90$)	0.509	0.612	0.754	1.225	1.654	2.007	2.340
Age ($\alpha= 0.95$)	0.512	0.613	0.758	1.518	2.046	2.495	2.893
CTC ($\uparrow$)	6.33	8.33	8.66	8.33	8.00	8.00	7.33

Finally, we assess our stability metrics. When ground truth disease labels are available, bias is often estimated by calculating the performance gap between the best and worst performing data stratification for a variable of interest. Therefore, separately for each attribute (sex, race, age), we calculate the worst-case gap in AUC ($\Delta AUC$) between relevant data stratifications (e.g. male/female for sex) for the 1 non-biased (basic) predictor and each of the 3 sex-biased and 3 age-biased predictors. We repeat for 3 random seeds and thus obtain 21 values of $\Delta AUC$ per attribute (sex, race, age). We then study how well our stability metrics correlate with $\Delta AUC$ and Figure 3 demonstrates strong correlation, implying our metrics are useful bias estimators. Table 5c further shows that values of our stability metrics tend to follow changes in model bias. These results suggests that our metrics can be used as bias estimators without requiring ground truth classification labels, replacing the use of stratified performance gap analysis, which requires ground truth labels, when these are not available.

Table 5: Prediction stability under marginalisation at threshold $\tau= 0.5$ (uniform prior). $S_{\mathrm{var}}$ = variance of marginalised predictions; $S_{\mathrm{flip}}$ = label-flip rate at $\tau$; $D_{\mathrm{obs}}$ = mean absolute difference between original and marginalised scores; $\rho$ = Spearman rank correlation; $\tau_b$ = Kendall $\tau_b$; $|\Delta\bar{r}|$ = mean absolute rank change.

Predictor	$S_{\mathrm{var}}$	$S_{\mathrm{flip}}$	$D_{\mathrm{obs}}$	$\rho$	$\tau_b$	$|\Delta\bar{r}|$
Basic	0.0396	0.247	0.199	0.691	0.476	913.5
Logit +0.5	0.0530	0.271	0.216	0.653	0.471	918.4
Logit +1	0.0574	0.289	0.235	0.653	0.469	928.5
Logit +2	0.0685	0.335	0.291	0.641	0.466	934.2
Sex ($\alpha= 0.80$)	0.0426	0.265	0.218	0.653	0.476	920.6
Sex ($\alpha= 0.90$)	0.0501	0.256	0.208	0.645	0.479	922.4
Sex ($\alpha= 0.95$)	0.0489	0.263	0.222	0.644	0.464	932.5
Age ($\alpha= 0.80$)	0.0566	0.267	0.203	0.652	0.477	913.3
Age ($\alpha= 0.90$)	0.0567	0.270	0.205	0.642	0.466	937.8
Age ($\alpha= 0.95$)	0.0567	0.279	0.208	0.589	0.416	1050.2

Predictor	$S_{\mathrm{var}}$	$S_{\mathrm{flip}}$	$D_{\mathrm{obs}}$	$S_{\mathrm{rank}}$
Basic	0.0198	0.143	0.069	0.920
Logit +0.5	0.0212	0.144	0.064	0.918
Logit +1	0.0359	0.188	0.115	0.907
Logit +2	0.0502	0.243	0.173	0.847
Sex ($\alpha= 0.80$)	0.0270	0.146	0.068	0.916
Sex ($\alpha= 0.90$)	0.0305	0.156	0.079	0.906
Sex ($\alpha= 0.95$)	0.0284	0.158	0.082	0.903
Age ($\alpha= 0.80$)	0.0263	0.167	0.077	0.917
Age ($\alpha= 0.90$)	0.0308	0.170	0.079	0.916
Age ($\alpha= 0.95$)	0.0314	0.174	0.096	0.901

Conclusion

This work introduced counterfactual marginalisation as a test-time framework for evaluating medical image classifiers under controlled interventions on nuisance parent variables. By averaging predictions over SCM-generated counterfactual variants, the framework provides intervention-aware measures of risk, calibration, stability, tail sensitivity, and ranking robustness. Results on CheXpert and MIMIC demonstrate that counterfactual evaluation can expose sensitivity to demographic interventions even when standard point-estimate metrics appear stable. A limitation is that the metrics depend on the realism and causal validity of the counterfactual generator, so they should be used alongside generator validation and conventional external evaluation, rather than as a replacement. Future work should study sensitivity to causal graph specification, intervention distributions, generator uncertainty, and extensions to multi-label settings.

Full Results

Risk

	Predictor	$R_{\mathrm{orig}}$	$R_{\mathrm{CM}}$	$R_{\mathrm{IE}}$	$R_{\mathrm{CVaR}}(0.50)$	$R_{\mathrm{CVaR}}(0.25)$	$R_{\mathrm{CVaR}}(0.10)$	$R_{\mathrm{WC}}$
(a) Trained and tested on CheXpert ($N = 5180$).
	Basic	0.444	0.501	0.594	1.002	1.326	1.549	1.852
	Logit +0.5	0.445	0.612	0.724	1.027	1.346	1.597	1.845
	Logit +1	0.443	0.629	0.771	1.068	1.424	1.699	1.962
	Logit +2	0.502	0.677	0.919	1.072	1.431	1.716	1.997
	Sex ($\alpha= 0.80$)	0.453	0.610	0.687	1.192	1.618	1.991	1.390
	Sex ($\alpha= 0.90$)	0.452	0.614	0.699	1.195	1.612	1.968	2.338
	Sex ($\alpha= 0.95$)	0.443	0.614	0.699	1.203	1.622	1.982	2.361
	Age ($\alpha= 0.80$)	0.494	0.613	0.750	1.126	1.511	1.817	2.111
	Age ($\alpha= 0.90$)	0.509	0.612	0.754	1.225	1.654	2.007	2.340
	Age ($\alpha= 0.95$)	0.512	0.613	0.758	1.518	2.046	2.495	2.893
	CTC ($\uparrow$)	6.33	8.33	8.66	8.33	8.00	8.00	7.33
(b) Trained and tested on MIMIC ($N = 28189$).
	Basic	0.335	0.425	0.536	0.790	1.031	1.190	1.283
	Logit +0.5	0.342	0.454	0.556	0.827	1.084	1.252	1.351
	Logit +1	0.359	0.469	0.590	0.897	1.161	1.370	1.494
	Logit +2	0.424	0.511	0.694	0.880	1.181	1.377	1.505
	Sex ($\alpha= 0.80$)	0.375	0.455	0.570	1.055	1.290	1.459	1.563
	Sex ($\alpha= 0.90$)	0.371	0.458	0.583	1.215	1.427	1.580	1.681
	Sex ($\alpha= 0.95$)	0.384	0.459	0.581	1.232	1.499	1.724	1.864
	Age ($\alpha= 0.80$)	0.335	0.425	0.536	0.857	1.160	1.360	1.549
	Age ($\alpha= 0.90$)	0.351	0.565	0.734	0.887	1.142	1.370	1.501
	Age ($\alpha= 0.95$)	0.366	0.610	0.771	0.880	1.191	1.411	1.505
	CTC ($\uparrow$)	7.00	8.33	8.00	8.00	7.66	7.00	7.33
(c) Trained on CheXpert, tested on MIMIC ($N = 28189$).
	Basic	0.397	0.466	0.549	0.769	0.977	1.121	1.210
	Logit +0.5	0.402	0.486	0.566	0.822	1.055	1.223	1.328
	Logit +1	0.422	0.495	0.602	0.913	1.176	1.379	1.506
	Logit +2	0.508	0.536	0.722	1.177	1.506	1.778	1.945
	Sex ($\alpha= 0.80$)	0.418	0.497	0.558	0.761	0.941	1.064	1.140
	Sex ($\alpha= 0.90$)	0.395	0.499	0.565	0.793	1.023	1.183	1.281
	Sex ($\alpha= 0.95$)	0.437	0.530	0.597	0.829	1.042	1.193	1.287
	Age ($\alpha= 0.80$)	0.416	0.490	0.568	0.897	1.056	1.143	1.464
	Age ($\alpha= 0.90$)	0.422	0.511	0.573	0.809	1.015	1.166	1.563
	Age ($\alpha= 0.95$)	0.402	0.516	0.614	0.827	1.062	1.233	1.642
	CTC ($\uparrow$)	6.66	8.00	8.33	8.33	7.66	7.66	7.33

Calibration

Predictor	Marginalised Brier	Marginalised ECE	Marginalised MCE	Original Brier	Original ECE	Original MCE
(a) Trained and tested on CheXpert ($N = 5180$).
Predictor	Brier	ECE	MCE	Brier	ECE	MCE
Basic	0.202	0.083	0.171	0.143	0.030	0.080
Logit +0.5	0.208	0.125	0.219	0.143	0.019	0.077
Logit +1	0.216	0.153	0.240	0.149	0.043	0.121
Logit +2	0.237	0.211	0.306	0.180	0.116	0.228
Sex ($\alpha= 0.80$)	0.218	0.100	0.184	0.146	0.017	0.046
Sex ($\alpha= 0.90$)	0.223	0.104	0.232	0.145	0.042	0.119
Sex ($\alpha= 0.95$)	0.224	0.111	0.233	0.142	0.017	0.078
Age ($\alpha= 0.80$)	0.216	0.091	0.164	0.158	0.058	0.104
Age ($\alpha= 0.90$)	0.216	0.099	0.159	0.162	0.057	0.089
Age ($\alpha= 0.95$)	0.228	0.103	0.171	0.165	0.056	0.079
(b) Trained and tested on MIMIC ($N = 28189$).
Predictor	Brier	ECE	MCE	Brier	ECE	MCE
Basic	0.139	0.088	0.244	0.102	0.012	0.067
Logit +0.5	0.143	0.111	0.284	0.104	0.029	0.093
Logit +1	0.150	0.136	0.323	0.109	0.050	0.146
Logit +2	0.167	0.179	0.378	0.128	0.091	0.212
Sex ($\alpha= 0.80$)	0.152	0.126	0.299	0.193	0.133	0.262
Sex ($\alpha= 0.90$)	0.186	0.128	0.303	0.212	0.154	0.271
Sex ($\alpha= 0.95$)	0.207	0.135	0.296	0.241	0.181	0.328
Age ($\alpha= 0.80$)	0.141	0.094	0.251	0.113	0.039	0.149
Age ($\alpha= 0.90$)	0.150	0.105	0.252	0.114	0.049	0.188
Age ($\alpha= 0.95$)	0.154	0.138	0.259	0.118	0.042	0.073
(c) Trained on CheXpert and tested on MIMIC ($N = 28189$).
Predictor	Brier	ECE	MCE	Brier	ECE	MCE
Basic	0.155	0.059	0.103	0.125	0.056	0.147
Logit +0.5	0.157	0.074	0.161	0.126	0.030	0.077
Logit +1	0.161	0.096	0.206	0.134	0.019	0.034
Logit +2	0.179	0.150	0.278	0.164	0.075	0.146
Sex ($\alpha= 0.80$)	0.156	0.061	0.116	0.132	0.075	0.163
Sex ($\alpha= 0.90$)	0.169	0.061	0.139	0.125	0.065	0.182
Sex ($\alpha= 0.95$)	0.172	0.076	0.132	0.140	0.072	0.187
Age ($\alpha= 0.80$)	0.159	0.063	0.141	0.131	0.040	0.075
Age ($\alpha= 0.90$)	0.167	0.064	0.137	0.135	0.028	0.083
Age ($\alpha= 0.95$)	0.170	0.096	0.163	0.127	0.015	0.048

Stability

Predictor	$S_{\mathrm{var}}$	$S_{\mathrm{flip}}$	$D_{\mathrm{obs}}$	$\rho$	$\tau_b$	$|\Delta\bar{r}|$
Basic	0.0358	0.148	0.069	0.932	0.783	2143
Logit +0.5	0.0367	0.148	0.071	0.930	0.782	2059
Logit +1	0.0385	0.153	0.080	0.909	0.760	2242
Logit +2	0.0433	0.170	0.108	0.877	0.745	2731
Sex ($\alpha= 0.80$)	0.0406	0.158	0.102	0.919	0.758	2041
Sex ($\alpha= 0.90$)	0.0497	0.196	0.137	0.897	0.713	3013
Sex ($\alpha= 0.95$)	0.0514	0.191	0.139	0.865	0.671	3386
Age ($\alpha= 0.80$)	0.0412	0.155	0.087	0.928	0.773	2244
Age ($\alpha= 0.90$)	0.0477	0.151	0.089	0.917	0.770	2285
Age ($\alpha= 0.95$)	0.0487	0.168	0.097	0.888	0.746	2330

Predictor	$S_{\mathrm{var}}$	$S_{\mathrm{flip}}$	$D_{\mathrm{obs}}$	$\rho$	$\tau_b$	$|\Delta\bar{r}|$
Basic	0.0358	0.148	0.069	0.920	0.765	2339
Logit +0.5	0.0212	0.144	0.064	0.918	0.751	2489
Logit +1	0.0359	0.188	0.115	0.907	0.726	2790
Logit +2	0.0502	0.243	0.173	0.847	0.646	3801
Sex ($\alpha= 0.80$)	0.0270	0.146	0.068	0.916	0.756	2430
Sex ($\alpha= 0.90$)	0.0305	0.156	0.079	0.906	0.743	2552
Sex ($\alpha= 0.95$)	0.0284	0.158	0.082	0.903	0.738	2605
Age ($\alpha= 0.80$)	0.0263	0.167	0.077	0.917	0.750	2380
Age ($\alpha= 0.90$)	0.0308	0.170	0.079	0.916	0.748	2403
Age ($\alpha= 0.95$)	0.0314	0.174	0.096	0.901	0.735	2431

Details of the Generative Model

For completeness, we summarise the semi-supervised deep structural causal model of [9], which we use throughout this work to generate counterfactual images. We refer the reader to the original paper for full derivations; here we restate only what is needed to make our experiments reproducible.

Setup

The endogenous variables comprise an image $x$ and a set of causal variables $y$, denoted $y^{*}$ when observed and $y^{\prime}$ when predicted. Latent variables $z = z_{1:L}$ form part of the exogenous noise for $x$ and are modelled with a hierarchical latent structure [4, 24]. In addition to this generative backbone, the model contains a predictive component that infers $y$ from $x$, which is what permits counterfactual generation when labels are missing. As in [9], we present the case of a single cause variable $y_C$ and a single effect variable $y_E$; the construction extends to any finite number of variables under an arbitrary causal structure.

Training Objective

Training proceeds by partitioning the data according to which labels are available. For fully labelled samples drawn from $\mathcal{D}_L$, the ELBO gives

where $\mathcal{L}(x,y)$ is the ELBO of a conditional VAE and $p_\theta(\cdot)$ are Gaussian priors. For unlabelled samples from $\mathcal{D}_U$, the labels are predicted via $y^{\prime} \sim q_\phi(y\mid x)$ and regularised against their priors:

When only the cause $y_C$ is observed, for $(x,y_C)\in\mathcal{D}_C$,

and when only the effect $y_E$ is observed, for $(x,y_E)\in\mathcal{D}_E$,

Since the parent predictors $q_\phi(y_E\mid x)$ and $q_\phi(y_C\mid x,y_E)$ would otherwise only receive gradient when their targets are unobserved, an explicit classification term is added on the labelled subset [15], yielding the total objective

In the final term the labelled variables $y_i$ are placed in a topological ordering [6] beginning at the root nodes, so that for every $i$ the ancestors of $y_i$ lie in $y_{<i}$ and its descendants in $y_{>i}$.

Two key implementation details of [9] are retained. First, for discrete variables whose true label $y^*$ is unavailable, the corresponding term is inversely weighted by the entropy $H_\phi(y^* \mid x)$ of the predicted label distribution — e.g. the expectation in (16) is scaled by $1-H_\phi(y_E^* \mid x)$ — using predictive uncertainty as a proxy for how far the imputed label should be trusted. Second, the expectations over labels in (15) are computed by summing over all values of a discrete $y$, $\mathbb{E}_{q_\phi(y\mid x)}[f(y,\cdot)]=\sum_y q_\phi(y\mid x)f(y,\cdot)$ [15], which becomes expensive as the number of variables or classes grows. Rather than resorting to Monte-Carlo estimation, training begins on the labelled subset alone, so that the predictors $q_\phi(\cdot\mid x)$ are already accurate before being used to impute labels for the remainder of the data.

Counterfactual Regularisation

The model additionally employs a causal form of consistency regularisation [27], in which perturbations of the input are restricted to interventions on the DAG over the causal variables. Given an image $x$ with variables $(y_C,y_E)$, intervening on the effect via $\operatorname{do}(y_E=\tilde{e})$ produces $\tilde{x}$ with variables $(\tilde{y}_C,\tilde{y}_E)$. If the DAG holds, the cause is invariant to this intervention, so predicting $\tilde{y}_C\sim q_\phi(\cdot\mid\tilde{x})$ and minimising $D(y_C,\tilde{y}_C)$ — for a suitable distance $D(\cdot,\cdot)$ — penalises violations of that invariance; if $y_C$ is unobserved it is first imputed as $y_C \sim q_\phi(\cdot\mid x, y_E)$. Intervening on the cause instead should change the effect: $\tilde{y}_E \sim q_\phi(\cdot\mid\tilde{x})$ is compared against the counterfactual value of $y_E$ under the intervention on $y_C$, with $y_E \sim q_\phi(\cdot\mid x)$ imputed when unlabelled. This regularises both the generative component and the causal mechanisms $p_\theta(y_i \mid y_{<i})$.

Counterfactual Generation

At inference time, counterfactuals $\tilde{x}$ are produced using the causal mechanisms $p_\theta(y_i \mid y_{<i})$. Abduction requires the structural assignments to be invertible in the exogenous noise, so each mechanism is encoded as an invertible map $g_{y_{<i}}(u)$, $u \sim p(u)$, parameterised by the causal parents $y_{<i}$ [20]. Each counterfactual value is then obtained in topological order as $\tilde{y}_i = g_{\tilde{y}_{<i}}\left(g_{y_{<i}}^{-1}(y_i)\right)$, and any missing $y_i$ is imputed beforehand with the trained predictor $q_\phi(y_i \mid x, y_{>i})$, as illustrated in Figure 4 (right).

Example Counterfactuals

Below we provide some example counterfactuals produced by the model and used for testing with our proposed metrics.