betaFeedback

WorksMathematics

Joint Learning of Factual and Counterfactual Worlds for Causal Prior-data Fitted Networks

Abstract

Estimating treatment effects is central to science across many disciplines and requires counterfactual reasoning about alternative worlds. In the real world, however, each unit reveals its outcome only under the factual world. Thus, the factual and counterfactual outcomes can never be jointly learned by any model trained on real data. The problem is therefore tractable only in synthetic settings, where both worlds can be generated. Recent progress on prior-data fitted networks (PFNs) has shown that models pre-trained purely on synthetic data can transfer to real-world prediction. This paradigm has since carried over to causal effect estimation. Despite this fully synthetic setup, existing causal PFNs carry the real-world constraint into their training objective, confining learning to a single world's outcome per unit. The dependence between the two worlds is thereby lost, and the density of the treatment effect relies by default on independence, which is implausible in practice. We therefore propose joint learning, a training setup in which a causal PFN learns the joint distribution of the factual and counterfactual outcomes of each unit. From the joint distribution, we obtain both the response and the individual treatment effect densities, and at inference, we apply optimal transport across units to obtain an approximation of the average treatment effect density. Ours is therefore the first causal PFN approach to provide all three densities together while accounting for the cross-world dependence. We apply joint learning and optimal transport to existing causal PFNs and evaluate it across semi-synthetic, synthetic, and real benchmarks spanning diverse causal scenarios. The jointly trained models significantly improve upon their baselines in point estimation and uncertainty quantification in most settings, indicating that joint learning is a preferable training setup for causal PFNs.

Introduction

Treatment effect estimation is central to medicine, economics, finance, and public policy [18, 4]. The effect can be considered at the unit or at the population level [15]. At the unit level, it is the difference between a unit’s factual and counterfactual outcomes; its estimation is therefore inherently a joint learning problem over them. In practice, however, each unit is observed only under the factual world, so the pair can never be jointly learned from real data [13, 32, 29, 15]. The only viable option is therefore learning each outcome separately [33, 2]. Yet at best, this recovers the two marginal distributions; the dependence between the outcomes remains unidentified even under randomization [13, 11]. Assuming the outcomes are independent makes learning practical and solves the joint problem but is implausible [3] (Vansteelandt and Lancker 2024), and such misspecification carries a real inferential cost [10]. Joint learning at unit level therefore remains tractable only in synthetic settings, in which both outcomes can be generated.

At the population level, in contrast to unit level, the average treatment effect depends only on the two marginal distributions, which randomization identifies. Randomized controlled trials are therefore the ideal setup for it, but are often unethical, expensive, slow, or infeasible [14]. In these cases, one has to rely on observational data. Recovering the marginals from observational data, however, requires strong assumptions, such as unconfoundedness, positivity, or a valid instrument [14, 28, 8] (Wager and Athey 2018). These assumptions are difficult to justify in practice: confounders are frequently unmeasured, valid instruments seldom exist, and the causal graph is rarely known fully.

To address the assumption problem, a family of causal PFNs has recently been proposed [31, 30, 5, 22, 36]. A PFN is a transformer that is trained on synthetic data and returns a posterior-predictive distribution [24]. PFNs achieve state-of-the-art performance on tabular prediction on observational data [16, 17], showing that a model trained purely on synthetic data can transfer to real-world prediction. Causal PFNs adopt the same architecture but predict interventional rather than observational outcomes [35]. They resolve the assumption problem by encoding the identification and structural assumptions in the training prior, and have outperformed traditional causal machine-learning methods [8, 26, 19, 20] (Wager and Athey 2018) on standard tabular benchmarks [31, 5, 30].

Causal PFNs thus provide a promising direction by addressing the assumption problem at the population level. However, critical gaps remain. First, existing models report only a point estimate of the average treatment effect, and they fail to provide a mechanism to estimate its density. Second, they do not address the joint learning problem at the unit level. Despite the fully synthetic setup, most existing models carry the real-world constraint into their training objective, confining learning to a single world’s outcome per unit [31, 30, 5]. These models therefore cannot learn the cross-world dependence and default to the implausible independence assumption. Training a model on the treatment effect directly avoids this assumption [22] but discards the response information. Hence, a decision maker cannot obtain the response, individual treatment effect, and average treatment effect densities together from any existing causal PFN. To address this, we propose joint learning, a training setup in which a causal PFN learns the joint distribution of each unit’s factual and counterfactual outcomes. From this joint distribution, the response and individual treatment effect densities follow, and optimal transport across units at inference provides a calibrated average treatment effect density. Ours is thus the first causal PFN approach to provide all three densities while accounting for the cross-world dependence. We apply joint learning with optimal transport to existing causal PFNs and evaluate it across semi-synthetic, synthetic, and real benchmarks spanning diverse causal scenarios. The jointly trained models significantly improve upon their baselines in point estimation and uncertainty quantification in most settings, indicating that joint learning is a preferable training setup for causal PFNs.

Background and related work

Structural causal model and interventions

We work with an observational dataset Dobs={(Xi,Ti,Yi)}i=1n\mathcal{D}_{\mathrm{obs}} = \{(X_i, T_i, Y_i)\}_{i=1}^{n}, where each row records a unit’s covariates Xi∈RdX_i \in\mathbb{R}^{d}, the treatment it received Ti∈{0,1}T_i \in\{0,1\}, and the outcome observed under that treatment, Yi∈RY_i \in\mathbb{R}.

We assume Dobs\mathcal{D}_{\mathrm{obs}} is generated by a structural causal model [28, 29] (SCM) ψ=(G,f,pε)\psi= (G, f, p_{\varepsilon}) over {X,T,Y}\{X, T, Y\}: GG a directed graph, f={fV}V∈{X,T,Y}f = \{f_V\}_{V \in\{X,T,Y\}} structural equations shared across all units, pεp_{\varepsilon} the exogenous noise distribution. Each row of Dobs\mathcal{D}_{\mathrm{obs}} is one sample εi∼pε\varepsilon_i \sim p_{\varepsilon} propagated through ff. An intervention do⁡(T=t)\operatorname{do}(T = t) under SCMs sets the treatment to value tt and severs its dependence on its causal parents, while leaving every other structural equation untouched. This defines the potential outcome:

Ydo⁡(t)(x,ε)=fY(x,t,pa⁡−T(ψ;x,ε),εY),t∈{0,1},(1)Y_{\operatorname{do}(t)}(x,\varepsilon) = f_Y\left(x,t,\operatorname{pa}_{-T}(\psi;x,\varepsilon),\varepsilon_Y\right), \quad t \in\{0,1\}, \tag*{(1)}

where pa⁡−T(ψ;x,ε)\operatorname{pa}_{-T}(\psi;x,\varepsilon) denotes the values of YY’s graph-parents other than TT, propagated through ψ\psi. The individual treatment effect (ITE) of a unit is the difference of its two potential outcomes,

δ:=Ydo⁡(1)−Ydo⁡(0).(2)\delta:= Y_{\operatorname{do}(1)} - Y_{\operatorname{do}(0)}. \tag*{(2)}

The conditional average treatment effect (CATE) is its mean at covariate value xx, τ(x):=E[δ∣X=x]\tau(x) := \mathbb{E}[\delta\mid X = x], and the average treatment effect (ATE) is the average of the CATE over the covariate population, EX[τ(X)]\mathbb{E}_{X}[\tau(X)].

Joint learning of outcomes

The dependence between a unit’s two potential outcomes in Equation (2) is captured by the cross-world correlation (Bodik et al., 2026):

ρ(x)=corr⁡(Ydo⁡(1),Ydo⁡(0)∣X=x).(3)\rho(x) = \operatorname{corr}(Y_{\operatorname{do}(1)}, Y_{\operatorname{do}(0)} \mid X = x). \tag*{(3)}

Since in real data each unit is observed under a single treatment, ρ(x)\rho(x) is not identifiable [32, 13]. Hence, models trained on real data [2, 21] (Vlontzos et al. 2023; Yoon et al. 2018) cannot learn this correlation directly. This limitation does not apply to models trained on synthetic data, where both potential outcomes of every unit can be computed and ρ(x)\rho(x) is therefore known. Since the two outcomes are defined by the same exogenous variables [28], ρ(x)\rho(x) is determined by the mechanisms of the SCM. A model can therefore be trained on data simulated from SCMs, with the pair (ydo⁡(0),ydo⁡(1))(y_{\operatorname{do}(0)}, y_{\operatorname{do}(1)}) of every unit as its target, which makes the joint law P(Ydo⁡(0),Ydo⁡(1)∣X=x)P(Y_{\operatorname{do}(0)}, Y_{\operatorname{do}(1)} \mid X = x) a direct learning target. We refer to training on such a pair as joint learning.

Prior-data fitted networks for Bayesian inference

Prior-Data Fitted Networks (PFNs) [24] are neural networks trained to perform Bayesian prediction. Under a prior p(ψ)p(\psi) over SCMs, the target is the posterior predictive distribution

p(y∣x,Dobs)=∫p(y∣x,ψ) p(ψ∣Dobs) dψ.(4)p(y \mid x, \mathcal{D}_{\mathrm{obs}}) = \int p(y \mid x,\psi)\,p(\psi\mid\mathcal{D}_{\mathrm{obs}})\,d\psi. \tag*{(4)}

A PFN is trained once, offline, on synthetic datasets generated from the prior to approximate this distribution. At inference time, it returns the distribution for a new DobsD_{\mathrm{obs}} in a single forward pass, without gradient updates.

Causal prior-data fitted networks

Standard PFNs such as TabPFN (Hollmann et al., [17]) approximate the distribution in Equation (4) for an observational outcome yy, whereas causal PFNs approximate it for an interventional outcome Ydo(t)Y_{\mathrm{do}(t)}. Since this distribution depends on the prior p(ψ)p(\psi), existing causal PFNs differ mainly in what causal structures their priors commit to. Do-PFN (Robertson et al., [31]) samples SCMs from a broad prior with no identifiability restrictions and the model is trained to predict interventional outcomes from DobsD_{\mathrm{obs}} alone, without knowledge of the graph. Use What You Know (UWYK) (Reuter et al., [30]) uses an unrestricted prior similar to Do-PFN, but supplies causal graph knowledge at inference time as a partially known ancestral matrix T~∈{−1,0,1}(d+2)×(d+2)\widetilde{T} \in\{-1, 0, 1\}^{(d+2)\times(d+2)}, injected into the feature-wise attention of the transformer, so that a single model spans the range from no graph information to full ancestral knowledge. CausalPFN (Balazadeh et al., [5]) builds its prior from real and synthetic tables, taking random columns as the expected potential outcomes mtm_t and the noise terms ηt\eta_t, with Ydo(t)=mt+ηtY_{\mathrm{do}(t)} = m_t + \eta_t. The model is trained on mtm_t instead of Ydo(t)Y_{\mathrm{do}(t)}. This removes the random noise ηt\eta_t from its target, but variation of mtm_t remains, which comes from characteristics that xx does not capture. The treatment is drawn as T∼Bernoulli⁡(σ(f(X)))T \sim\operatorname{Bernoulli}(\sigma(f(X))) for a random propensity function ff that depends on XX alone, which guarantees ignorability by design. Its estimates are therefore valid only on data that satisfy this assumption. CausalFM (Ma et al., [22]) uses separate priors encoding different identifiability constraints. Each model thus satisfies its constraint by design but covers only that setting. It predicts the distribution of the ITE δ\delta directly, without modeling either Ydo(0)Y_{\mathrm{do}(0)} or Ydo(1)Y_{\mathrm{do}(1)}; because the map (Ydo(0),Ydo(1))↦δ(Y_{\mathrm{do}(0)}, Y_{\mathrm{do}(1)}) \mapsto\delta is not injective, neither potential outcome can be recovered from δ\delta alone.

Despite these different approaches, no causal PFN performs the joint learning of § 2.2. Since Do-PFN, UWYK and CausalPFN are trained on one outcome per unit, the dependence between a unit’s two potential outcomes is never learned, and they are treated as independent. CausalFM is trained on their difference δ\delta, which does not determine their joint law. Two limitations follow.

Statistical efficiency. Single-outcome models never see both outcomes of the same unit, which limits how efficiently they learn the treatment effect. They see each unit under one treatment only, and learn the other outcome from different units. Their estimate of the treatment effect therefore carries the noise of both outcomes. A model trained on both sees the two outcomes of each unit. When p(x)>0p(x) > 0, the noise they share cancels in their difference, so this model can learn the treatment effect from fewer units. § 3.1 discusses this gap.

Distributional information. The second limitation concerns the densities that existing causal PFNs report. A decision maker needs the response densities (e.g., whether a treated unit’s outcome exceeds a clinical or economic threshold), and the ITE and ATE densities (e.g., whether to intervene at all). To our knowledge, no causal PFN reports an ATE density, as for the response and ITE densities, each model falls short in a different way. CausalFM provides an ITE density, but no response densities. Do-PFN, UWYK and CausalPFN can provide both,1 but their ITE density treats the two outcomes as independent. This assumption is likely to be violated in practice [3] (Vansteelandt and Lancker 2024; Bodik et al. 2026) and may introduce spurious treatment effects. Our proposed joint learning, combined with an optimal-transport aggregation at inference, resolves these gaps. It provides statistical efficiency, gives both the response and ITE densities without an independence assumption, and adds a calibrated approximation of the ATE density.

Methodology

We approximate the joint posterior predictive distribution of a unit’s factual and counterfactual outcomes. For a query with covariates xx, observational context Dobs\mathcal{D}_{\mathrm{obs}}, and treatments t≠t′t \ne t', it is defined as

p(ydo(t),ydo(t′)∣x,Dobs)=∫p(ydo(t),ydo(t′)∣x,ψ)p(ψ∣Dobs) dψ.(5)p(y_{\mathrm{do}(t)}, y_{\mathrm{do}(t')} \mid x, \mathcal{D}_{\mathrm{obs}}) = \int p(y_{\mathrm{do}(t)}, y_{\mathrm{do}(t')} \mid x, \psi) p(\psi\mid\mathcal{D}_{\mathrm{obs}})\,\mathrm{d}\psi. \tag*{(5)}

The uncertainty decomposition and consistency result of Robertson et al. [31] extend directly to this distribution (Appendix § 12). At inference, the ITE density of each unit is computed from the predicted joint, and optimal transport aggregates these into the ATE density. Figure 1 gives an overview of the method. § 3.1 establishes the statistical efficiency of the joint objective, and § 3.2 details how the ITE and ATE densities are obtained.

Figure 1. Overview of joint learning. Pre-training: for each unit of a sampled SCM, the two potential outcomes are generated in a factual world, under do⁡(t)\operatorname{do}(t), and a counterfactual world, under do⁡(t′)\operatorname{do}(t'), which share the same exogenous noise ε\varepsilon. The model qθq_\theta receives an observational set and the covariates of query units, whose outcomes are unknown, predicts the joint distribution of their two potential outcomes, and is trained on the negative log-likelihood of the true pair. Inference: the difference of the two outcomes under the predicted joint gives the ITE distribution of each unit, and optimal transport aggregates these into the ATE distribution.2

Statistical efficiency of the joint objective

We estimate the CATE τ(x)\tau(x) by predicting the distribution of the ITE δ\delta at xx. Its mean is the point estimate τ^(x)\hat{\tau}(x), and its quantiles give a posterior predictive interval for δ\delta. As defined in § 2.1, each potential outcome is a function of the covariates and the exogenous noise, Ydo(t)(x,ε)Y_{\mathrm{do}(t)}(x,\varepsilon), and both potential outcomes of a unit share the same ε\varepsilon. Let μt(x)\mu_t(x) and σt2(x)\sigma_t^2(x) denote the mean and variance of Ydo(t)(x,ε)Y_{\mathrm{do}(t)}(x,\varepsilon) over ε\varepsilon given X=xX=x, and ρ(x)\rho(x) the cross-world correlation of Equation (3). Under an SCM ψ\psi, the variance of the ITE at xx is

vψ(x)=σ12(x)+σ02(x)−2ρ(x) σ1(x) σ0(x),(6)v_\psi(x) = \sigma_1^2(x) + \sigma_0^2(x) - 2\rho(x)\,\sigma_1(x)\,\sigma_0(x), \tag*{(6)}

and by the law of total variance, the posterior predictive variance of δ\delta decomposes as

Var⁡(δ∣x,Dobs)=Var⁡(τψ(x)∣Dobs)+E[vψ(x)∣Dobs],(7)\operatorname{Var}(\delta\mid x,\mathcal{D}_{\mathrm{obs}}) = \operatorname{Var}(\tau_\psi(x) \mid\mathcal{D}_{\mathrm{obs}}) + \mathbb{E}[v_\psi(x) \mid\mathcal{D}_{\mathrm{obs}}], \tag*{(7)}

where τψ(x)\tau_\psi(x) is the CATE under ψ\psi. The first term, the uncertainty about the CATE, shrinks as more data are observed; the second, the spread of units with the same xx, does not, and it depends on ρ(x)\rho(x). A CATE posterior contains only the first term; the two are equal only when δ\delta is deterministic given xx.

The second term is where the learning objectives mainly differ, since it includes ρ(x)\rho(x). The single-outcome models see each unit under a single treatment, thus discarding the dependence and treating the two outcomes as independent, which implies ρ(x)=0\rho(x)=0. This creates a mismatch with what we expect in real datasets. Setting ρ(x)\rho(x) to a different value at inference can address the mismatch, but choosing it correctly is not a trivial task. From Dobs\mathcal{D}_{\mathrm{obs}} alone, one can only get a conservative bound on ρ(x)\rho(x). Any value is therefore a random guess within this bound and does not improve with more data.

Choosing ρ(x)\rho(x) is, however, not the only problem. Even if the true ρ(x)\rho(x) were known, it would not suffice in general, since the two predicted distributions also contain the model’s uncertainty about μ0\mu_0 and μ1\mu_1, whose dependence ρ(x)\rho(x) does not describe (Appendix § 13). Joint learning instead approximates the second term with the ρ(x)\rho(x) of each SCM, weighted by how well that SCM fits DobsD_{\mathrm{obs}}; as each SCM contributes μ0\mu_0 and μ1\mu_1 together, the uncertainty about them keeps its dependence as well. As more data is observed, this weight concentrates on the SCMs compatible with DobsD_{\mathrm{obs}}, and the estimate improves. The remaining uncertainty about ρ(x)\rho(x) then reduces to the uncertainty over these SCMs.

The objectives also differ in how efficiently they learn τ(x)\tau(x) from finite training data. A single-outcome model learns each outcome from different units, so its estimate carries the full noise of both. A jointly trained model learns from both outcomes of the same unit. Their difference δ\delta carries less noise whenever ρ(x)>0\rho(x)>0, so it can learn τ(x)\tau(x) more efficiently. The joint objective also weighs errors differently. Under the joint loss, a predicted τ(x)\tau(x) that is off by ee increases the loss by at least e22Var⁡(δ)\frac{e^2}{2\operatorname{Var}(\delta)}, against e24(σ02+σ12)\frac{e^2}{4(\sigma_0^2+\sigma_1^2)} under the single-outcome loss (Proposition 2). The joint loss thus penalizes these errors more when ρ(x)>0\rho(x)>0, which can direct more of the learning toward τ(x)\tau(x). Both arguments assume that the objectives learn the same outcomes. A model trained to predict mtm_t instead learns τ(x)\tau(x) without the idiosyncratic noise ηt\eta_t, and can therefore be more efficient than the joint. However, this comes at a cost, since without that noise its response densities would be overconfident. The efficiency gaps between these objectives vanish with unlimited training data and model capacity, since under the same prior every objective then recovers the same posterior mean of τ(x)\tau(x). Yet, the estimates of Equation (7) still differ, and only joint learning estimates it consistently. Hence, the joint objective provides the largest information set to learn the ITE and response densities.

Data efficiency. As the single-outcome observation is a randomised function of the joint one, any parameter that can be estimated from it can be estimated from the joint at least as well (Lemma 1). Nothing is therefore lost by joint training. In addition to this preservation, joint training also gains information from two sources. Each unit contributes an additional outcome, and observing the two outcomes together makes their dependence learnable. To separate the two sources, we conduct an ablation in which the baseline is trained on the same number of outcomes as the joint model.

Computational cost and resolution. For PFN models with a one-dimensional output, the output is a distribution over a grid of KK points, and the output layer costs O(K)O(K) per query. For the joint model, the output is two-dimensional, and the output layer costs O(K2)O(K^2) per query. To compare against the published baselines at a similar cost, each joint model uses the square root of its baseline’s grid size. The coarser grid reduces the model’s representational capacity and places it at a disadvantage relative to its baseline. Improvements under this setup therefore reflect a conservative gain. The resolution loss can be mitigated at inference by a density estimator such as kernel density estimation ([27]), a Gaussian mixture ([23]), or MALC ((Danimisan et al., 2026)), at the cost of additional inference time; our results use the grid without it. We further conduct an ablation in which each baseline is re-trained on the grid of its joint model, for a matched comparison. The comparisons thus show the trade-off between cost and performance, that is, whether, under a fixed budget, one should prioritize resolution or joint learning.

Distributional inference for ITE and ATE

Let x1,…,xnx_1,\ldots,x_n be a sample of query covariates. For a query at xix_i, the ITE density is the distribution of δ\delta under the joint of Equation (5), with t=1t=1 and t′=0t'=0,

fδ(i)(d)=∫p(ydo(t′)=y,ydo(t)=y+d∣xi,Dobs) dy,(8)f_{\delta}^{(i)}(d)=\int p\left(y_{\mathrm{do}(t')}=y,y_{\mathrm{do}(t)}=y+d\mid x_i,\mathcal{D}_{\mathrm{obs}}\right)\,\mathrm{d}y, \tag*{(8)}

which is computed from the output of the joint (Appendix § 13). Since the ATE is the average of the ITEs across units, these densities need to be aggregated into one. A naive pointwise average would not describe the average effect, since differences between the units would inflate its spread. We therefore average their quantile functions instead and scale the result by conformal calibration ((Vovk et al., 2005)). This gives fATEf_{\mathrm{ATE}}, a calibrated approximation of the ATE density, as the scaled 2-Wasserstein barycenter of the ITE densities,

QATE(u)=τˉ+s(1n∑i=1nQδ(i)(u)−τˉ),u∈(0,1),(9)Q_{\mathrm{ATE}}(u)=\bar{\tau}+s\left(\frac{1}{n}\sum_{i=1}^{n}Q_{\delta}^{(i)}(u)-\bar{\tau}\right),\qquad u\in(0,1), \tag*{(9)}

where Qδ(i)Q_{\delta}^{(i)} is the quantile function of fδ(i)f_{\delta}^{(i)} and ss the scaling parameter, calibrated on the model’s prior. Its mean τˉ\bar{\tau} is the average of the per-query ITE means, leaving the ATE point estimate unchanged. Appendix § 14 gives the derivation, and Figure 4 illustrates the procedure.

Figure 4. ATE density via optimal transport on one IHDP realization. Faint curves are per-query ITE densities, the bold curve the ATE density, and the dashed line the true ATE. Top row shows the single-outcome models, bottom row their jointly trained (JT) variants.

Experimental Setup

We evaluate our proposed joint learning method across three causal PFN models and four benchmarks, measuring point estimation and uncertainty calibration. This section describes the compared baselines, ablations, benchmarks, and evaluation metrics.

Compared baselines. We evaluate the effect of joint learning on three major causal PFN models: Do-PFN [31], UWYK [30], and CausalPFN [5]. For each model, we hold the prior over SCMs and the transformer architecture fixed, generate both potential outcomes of each unit, and change only the learning objective to the joint learning of § 2.2. The jointly trained variant is denoted as JT throughout the results. We exclude another influential causal PFN model, CausalFM [22], for two reasons. It trains a separate model per identification setting, which is computationally heavy to replicate across our ablation setups. It also already supervises the ITE δ\delta, so joint learning would be expected to add its response densities rather than change its ITE results.

We compare each JT model against its published checkpoint, except for CausalPFN. It has two training stages, and the first cannot be trained on paired outcomes. Hence, we re-train its baseline from the second (causal) stage and refer to it as CausalPFN-C. We provide the details and the results against the published two-stage checkpoint on Appendix H.7.Additionally, UWYK’s authors provide two training options, with a binary or a continuous treatment. The published checkpoint covers only the continuous one, while our benchmarks have a binary treatment, and we therefore re-train the binary version under the same protocol and report it in Appendix H.8. Both comparisons lead to conclusions similar to those of our main results.

Cross-world correlation of the baselines. As discussed in § 3.1, the single-outcome baselines treat the two outcomes as independent, which implies ρ(x)=0\rho(x) = 0, although another value of ρ(x)\rho(x) can be set at inference. To test whether this closes the gap to joint learning, we report each baseline under ρ∈{−0.5,0,0.5,1}\rho\in\{-0.5, 0, 0.5, 1\} or a finer grid for an averaged comparison, applied through a Gaussian copula, while the JT models use their learned ρ(x)\rho(x).

Ablations. We conduct two ablations, following the arguments in § 3.1. In the first, each baseline is re-trained at the reduced resolution KK of its JT counterpart. This isolates the effect of joint learning from the effect of resolution. In the second, each baseline is given the same amount of data information. This isolates the effect of the dependence from the effect of the additional data. Each baseline thus enters with three variants: its JT version, a single-outcome version at the matched resolution KK, and a single-outcome version with the matched label budget. Together with the CausalPFN second-stage and UWYK binary re-trainings, this amounts to eleven trained models in total. The ablation results are reported in Appendix H.§ 5 and computation details in Appendix § 18.

Benchmarks. We evaluate on four benchmarks with semi-synthetic, synthetic, and real datasets. The semi-synthetic benchmark, RealCause [25], generates the two potential outcomes independently, reflecting the ρ=0\rho= 0 setting. The two synthetic benchmarks have dependent outcomes. ComplexMech, adopted from UWYK, draws random structural causal models with nonlinear mechanisms. The case study, adopted from Do-PFN, covers six causal scenarios: observed confounding, observed mediation, their combination, unobserved confounding, and frontdoor and backdoor structures. The real benchmark, Twins [21], consists of twin pairs whose two outcomes are treated as the factual and counterfactual outcomes according to the twins’ weight at birth. Due to the nature of twins, dependence between the outcomes therefore also exists in this case. For Twins, we follow Bodik (2026) and predict the outcome of one twin from the outcome of the other. Appendix § 16 describes how each benchmark is constructed, and Table 6 reports how the cross-world correlation varies across realizations.

BenchmarkDataset(A) no mediator(B) no YY-descendantComputableρ\rho (mean ±\pm sd)
RealCauseIHDP✓✓all0.002 ±\pm 0.037
ACIC✓✓all0.003 ±\pm 0.013
CPS✓✓all−0.001 ±\pm 0.009
PSID✓✓all0.003 ±\pm 0.020
Case studyObserved Confounder✓✓1800 / 18001.000 ±\pm 0.000
Backdoor✓✓1800 / 18001.000 ±\pm 0.001
Frontdoor✓✓1800 / 18001.000 ±\pm 0.001
Unobserved Confounder✓✓1800 / 18000.996 ±\pm 0.013
Observed Mediator×✓0 / 1800–
Observed Mediator + Confounder×✓0 / 1800–
ComplexMechn=5n = 553 / 10069 / 10037 / 1000.821 ±\pm 0.408
n=10n = 1032 / 10045 / 10015 / 1000.867 ±\pm 0.281
n=20n = 2012 / 10022 / 1001 / 1000.215
n=30n = 309 / 10014 / 1003 / 1000.958 ±\pm 0.072
n=40n = 4012 / 10012 / 1002 / 1001.000 ±\pm 0.000
n=50n = 5011 / 10014 / 1001 / 1000.946
Twinsd=1,d=10d = 1, d = 10n/an/aempirical0.59 ±\pm 0.15

Table 6. Cross-world correlation per benchmark. ρ(x)\rho(x) (Eq. (3)) is computed exactly from the data-generating process when no covariate is a consequence of the treatment or of the outcome, the assumption ε⊥X\varepsilon\perp X of Bodik et al. (2026): xx is fixed and the remaining noise, shared by both outcomes, is redrawn. Check (A): no covariate is a mediator (T→Xk→YT \to X_k \to Y). Check (B): no covariate is a descendant of YY. For ComplexMech, whose graph is drawn per dataset, the checks give the number of datasets that pass. ρ\rho is computed on the datasets passing both: per dataset the median of ρ(x)\rho(x) over its queries, then mean ±\pm sd over datasets. Datasets in which no noise reaches YY have a deterministic ITE given xx and are counted as ρ=1\rho= 1. For Twins, the two outcomes belong to different individuals, there is no data-generating process to compute ρ(x)\rho(x) from, and checks (A) and (B) do not apply. Following the statistical computation of Bodik et al. (2026), we report the correlation between the twins’ outcomes within the three most frequent value combinations of each pair of the first three covariates, as mean ±\pm sd over the groups in which it is defined. It is the same on the binary and on the dequantized outcomes (Table 7).

Evaluation metrics. From each predicted ITE distribution, we evaluate its mean, the CATE estimate, by PEHE\sqrt{\mathrm{PEHE}}, and its 95% posterior predictive interval by the standardised interval score IS0.05/σY\mathrm{IS}_{0.05}/\sigma_Y (Gneiting and Raftery, [12]), which combines coverage and length into a single metric by penalising misses in proportion to their distance from the interval. The ATE estimate is evaluated by its relative error ϵATE\epsilon_{\mathrm{ATE}}. Across datasets, all metrics are summarised by the geometric mean, Averageg_g, and differences between models are tested for significance with Hochberg correction. Definitions and rationales are given in Appendix § 15.

Results

We report the results on the semi-synthetic (§ 5.1), synthetic (§ 5.2), and real (§ 5.3) benchmarks. Due to space constraints, for uncertainty quantification, we discuss only the ITE in detail in the main text. Nevertheless, similar results also hold for the ATE, and our JT models attain significantly better interval scores than their baselines in 42 of the 48 comparisons across model families and benchmarks. We provide the full results and discussion for it in Appendix H.§ 4.

Semi-synthetic

RealCause is a commonly used benchmark in causal inference, but it generates the factual and counterfactual outcomes independently (Table 6). In reality, the two outcomes of a unit are expected to depend on each other (Bodik et al., 2026; Andrews and Didelez, [3]; Vansteelandt and Lancker, 2024). We therefore treat RealCause as a robustness test in which the dependence assumption is violated. As causal PFN evaluations on this benchmark commonly focus on point estimation, and due to space constraints, we report point estimates in Table 1 and defer interval calibration to Figure 6 (Appendix H.§ 1).

ModelIHDP PEHE↓\sqrt{\mathrm{PEHE}} \downarrowIHDP ϵATE↓\epsilon_{\mathrm{ATE}} \downarrowACIC PEHE↓\sqrt{\mathrm{PEHE}} \downarrowACIC ϵATE↓\epsilon_{\mathrm{ATE}} \downarrowCPS PEHE↓\sqrt{\mathrm{PEHE}} \downarrowCPS ϵATE↓\epsilon_{\mathrm{ATE}} \downarrowPSID PEHE↓\sqrt{\mathrm{PEHE}} \downarrowPSID ϵATE↓\epsilon_{\mathrm{ATE}} \downarrowPSID-B PEHE↓\sqrt{\mathrm{PEHE}} \downarrowPSID-B ϵATE↓\epsilon_{\mathrm{ATE}} \downarrowAverageg_g PEHE↓\sqrt{\mathrm{PEHE}} \downarrowAverageg_g ϵATE↓\epsilon_{\mathrm{ATE}} \downarrow
Do-PFN6.07±0.96.07 \pm0.90.93±0.40.93 \pm0.44.11±0.54.11 \pm0.50.66±0.00.66 \pm0.012014±3212014 \pm320.88±0.00.88 \pm0.020907±13820907 \pm1380.93±0.00.93 \pm0.022892±16122892 \pm1611.09±0.01.09 \pm0.01.22±0.11.22 \pm0.10.89±0.10.89 \pm0.1
Do-PFN JT5.72±0.85.72 \pm0.80.60±0.20.60 \pm0.23.65±0.63.65 \pm0.60.53±0.10.53 \pm0.111121±2311121 \pm230.53±0.00.53 \pm0.015844±15715844 \pm1570.10±0.00.10 \pm0.016356±15716356 \pm1570.18±0.00.18 \pm0.01.03∗∗±0.11.03^{**} \pm0.10.31∗∗±0.00.31^{**} \pm0.0
UWYK No-Anc6.29±0.86.29 \pm0.80.67±0.10.67 \pm0.13.35±0.53.35 \pm0.50.45±0.10.45 \pm0.112562±3712562 \pm370.96±0.00.96 \pm0.022443±13222443 \pm1320.98±0.00.98 \pm0.021886±13621886 \pm1360.94±0.00.94 \pm0.01.20±0.11.20 \pm0.10.76±0.00.76 \pm0.0
UWYK No-Anc JT4.51±0.64.51 \pm0.60.38±0.10.38 \pm0.12.74±0.52.74 \pm0.50.13±0.00.13 \pm0.012299±3112299 \pm310.88±0.00.88 \pm0.021426±14521426 \pm1450.85±0.00.85 \pm0.020039±14120039 \pm1410.68±0.00.68 \pm0.01.05∗∗±0.11.05^{**} \pm0.10.48∗∗±0.00.48^{**} \pm0.0
UWYK Anc5.49±0.85.49 \pm0.80.49±0.10.49 \pm0.12.77±0.52.77 \pm0.50.23±0.10.23 \pm0.110773±5810773 \pm580.61±0.00.61 \pm0.023304±15623304 \pm1561.06±0.01.06 \pm0.019039±21219039 \pm2120.62±0.00.62 \pm0.01.07±0.11.07 \pm0.10.54±0.00.54 \pm0.0
UWYK Anc JT4.30±0.64.30 \pm0.60.33±0.10.33 \pm0.12.72±0.52.72 \pm0.50.12±0.00.12 \pm0.012475±2812475 \pm280.92±0.00.92 \pm0.021850±13621850 \pm1360.90±0.00.90 \pm0.021467±14121467 \pm1410.86±0.00.86 \pm0.01.06±0.11.06 \pm0.10.49±0.00.49 \pm0.0
CausalPFN-C0.82±0.10.82 \pm0.10.05±0.00.05 \pm0.01.41±0.21.41 \pm0.20.09±0.00.09 \pm0.010043±18110043 \pm1810.35±0.00.35 \pm0.018281±29718281 \pm2970.68±0.00.68 \pm0.018645±31418645 \pm3140.71±0.00.71 \pm0.00.60±0.00.60 \pm0.00.24±0.00.24 \pm0.0
CausalPFN-C JT1.20±0.11.20 \pm0.10.07±0.00.07 \pm0.01.26±0.21.26 \pm0.20.09±0.00.09 \pm0.010902±17610902 \pm1760.39±0.00.39 \pm0.015557±21215557 \pm2120.25±0.00.25 \pm0.016769±31016769 \pm3100.33±0.00.33 \pm0.00.61±0.00.61 \pm0.00.18±0.00.18 \pm0.0

Table 1. Unit- and population-level point estimation errors on RealCause across five datasets (mean ±\pm standard error of the mean over realizations). Lower is better. Bold marks the better model within each family. Significance on Averageg_g at the 0.1 and 0.05 level is indicated by * and **, respectively. Averageg_g is the geometric mean over datasets, of PEHE\sqrt{\mathrm{PEHE}} standardised by each dataset’s outcome scale σY\sigma_Y and of ϵATE\epsilon_{\mathrm{ATE}}.

Figure 6. Interval score IS0.05/σY\mathrm{IS}_{0.05}/\sigma_Y (lower is better) of the 95% posterior predictive interval for the ITE on RealCause across five datasets (PSID-B: PSID balanced). Each row is one model family: its single-outcome model under each assumed ρ\rho (blue) and its JT model with the learned dependence (orange). Bars show the mean over realizations ±\pm the standard error. The last column, Averageg_g, is the geometric mean over datasets ±\pm its standard error, with the lowest value in bold. Significance against the JT model at the 0.1 and 0.05 level is indicated by * and **, respectively. Each panel has its own scale.

Finding 1: JT models improve point estimation even under independent generation. Do-PFN JT and UWYK No-Anc JT significantly improve upon their baselines on average in both unit and population level metrics, with a lower error on all ten cells. UWYK Anc JT and CausalPFN-C JT have a lower error on six of the ten cells, but on average neither differs significantly from its baseline.

Synthetic

Both synthetic benchmarks generate the two outcomes under dependence, and their cross-world correlation can be computed for most datasets (Table 6). On the case studies the correlation ρ≈1\rho\approx1 wherever it can be computed, hence ρ=1\rho= 1 matches the cross-world correlation of the data-generating process. On ComplexMech, ρ\rho is computable for fewer datasets and varies across them, with means between 0.82 and 1 for most graph sizes, making ρ=0.5\rho= 0.5 and ρ=1\rho= 1 the closest approximations among the values we assume. We focus on the interval score, which reflects the dependence that joint learning targets, and report it in Figure 2 and Figure 3. Due to space constraints, point estimation is reported in Appendix H.§ 2. In brief, joint learning also significantly improves it for Do-PFN and UWYK, reducing PEHE\sqrt{\mathrm{PEHE}} on average by 40–62% and 11–51%, respectively. For CausalPFN-C, whose baseline learns the expected outcomes without the idiosyncratic noise (§ 3.1), the baseline holds a slight advantage, and the JT model is at most 9% worse.

Figure 2. Interval score IS0.05/σY\mathrm{IS}_{0.05}/\sigma_Y (lower is better) of the 95% posterior predictive interval for the ITE on the six causal scenarios of the Do-PFN case studies. Each row is one model family: its single-outcome model under each assumed ρ\rho (blue) and its JT model with the learned dependence (orange). Bars show the mean over realizations ±\pm the standard error. The last column, Averageg_g, is the geometric mean over datasets ±\pm its standard error, with the lowest value in bold. Significance against the JT model at the 0.1 and 0.05 level is indicated by * and **, respectively. Each panel has its own scale.

Figure 3. Interval score IS0.05/σY\mathrm{IS}_{0.05}/\sigma_Y on ComplexMech, per graph size nn, as in Figure 2.

Finding 2: When the dependence is present, joint learning significantly improves the posterior predictive intervals at every tested ρ\rho; on the case studies, the improvement for CausalPFN-C occurs only at matched resolution. On the case studies, the Do-PFN and UWYK JT models significantly improve the interval score of their baselines under every tested ρ\rho, by 55–77%. This holds even when the baselines are given the true cross-world correlation, ρ=1\rho= 1. For CausalPFN-C, the baseline attains the better interval score on the case studies, since the reduced resolution costs the JT model the most, but once the resolution is matched, the JT model is significantly better under every tested ρ\rho, by 6–44% (Appendix H.§ 5). On ComplexMech, this large resolution cost of CausalPFN-C is outweighed by the gains of joint learning. Hence, every JT model significantly improves the interval score of its baseline by 18–78% under every tested ρ\rho.

Real

On Twins, we follow Bodik (2026): for each pair, one twin is observed and the other, its counterfactual, is predicted from it. The observed twin is informative about the other because the two share unobserved characteristics, which induce dependence between them. A single-outcome model can use the observed twin only through an assumed cross-world correlation, which the observed data do not identify. We therefore evaluate it over the range of ρ\rho compatible with the observed data. We evaluate the models on the dequantized outcomes with d=1d = 1 and d=10d = 10 covariates and report the mean squared error only, since the binary nature of the outcome leaves the interval score uninformative even after dequantization.

Finding 3: On real paired outcomes, joint learning on average significantly outperforms a baseline that selects its cross-world correlation uniformly from the range compatible with the data. A JT model predicts the counterfactual twin by conditioning its joint on the observed twin, with the dependence taken from the SCMs that fit DobsD_{\mathrm{obs}} (§ 3.1). The baseline must instead assume ρ\rho, and since DobsD_{\mathrm{obs}} carries no information about it, any value within the compatible range is a random guess. Averaged over these guesses, every JT model significantly reduces the mean squared error of its baseline at both covariate sets, by 22–48% (Table 2). The baseline reaches the JT model only when its guess falls mostly near the reported range of 0.5–0.7 ((Bodik et al., 2026)), which requires knowing the outcomes that are being predicted.

Do-PFNUWYK No-AncUWYK AncCausalPFN-C
BaselineJTBaselineJTBaselineJTBaselineJT
MSEd=1_{d=1} ↓0.206 ±0.000.141{sup}∗∗` ±0.000.202 ±0.000.117{sup}∗∗` ±0.000.212 ±0.000.118{sup}∗∗` ±0.000.210 ±0.000.164{sup}∗∗` ±0.00
MSEd=10_{d=10} ↓0.203 ±0.000.131{sup}∗∗` ±0.000.202 ±0.000.116{sup}∗∗` ±0.000.222 ±0.000.115{sup}∗∗` ±0.000.213 ±0.000.160{sup}∗∗` ±0.00

Table 2. Counterfactual prediction on dequantized Twins: mean squared error of the counterfactual twin given the factual twin and dd covariates, averaged over three jitter widths. The baseline is averaged over ρ\rho drawn uniformly from the range compatible with the data. Mean ± standard error, lower is better. Bold marks the better model of each family. Significance at the 0.1 and 0.05 level is indicated by ∗ and ∗∗, respectively.

Discussion and Conclusion

Although causal prior-data fitted networks are a step forward, at the current stage they do not solve causal inference. We discuss what this means for their evaluation and training.

Causal inference carries strict identifiability limits. The Markov equivalence class and the function class of the underlying SCM cannot be recovered from observational data ([34, 29]). Two PFN models of equal capacity can favour different members of the same equivalence class, and point estimation alone cannot separate them. Evaluation should therefore prioritise uncertainty estimation, with coverage and density estimation given a higher standing as standard metrics in the field.

Beyond how we evaluate these models, what we evaluate them on should also be scrutinized. Real datasets rarely offer an objectively accepted ground truth for causal inference, so evaluation mostly falls to semi-synthetic and synthetic benchmarks. There, what one believes about the connection of the factual and counterfactual worlds shapes the design of the data generation process. This belief can instead be informed by the evidence in the literature, which indicates that the two worlds are rarely independent and are typically positively correlated ([3] (Bodik et al. 2026; Vansteelandt and Lancker 2024)). The Twins ([21]) dataset is one of the motivating applications for this. Benchmarks may therefore benefit from spanning levels of this dependence rather than assuming independence, bringing the setup closer to reality.

This connection between the two worlds should also be carried to the training side. Jointly learning both factual and counterfactual worlds for every unit captures their dependence. In our results, this significantly improves point estimation and posterior predictive intervals in most settings, for some models even with bins up to 32 times wider. Hence, under a limited budget, joint learning should be favoured, provided that the learning regime and the standardization of the outcomes keep the resolution cost moderate enough for the model to capture the dependence. If the budget is not limited, raising the resolution removes this cost and only strengthens the case for joint learning (Appendix H.§ 5). As our results suggest, providing the same amount of data is not equivalent: seeing the same number of worlds does not substitute for learning them together.

Taken together, these considerations provide three main recommendations for where the field should move. Evaluation should measure the uncertainty that identifiability leaves behind, benchmarks should carry the connection between the worlds that reality carries, and training should learn the two worlds jointly. As evaluation and benchmarks move in this direction, the gains of joint learning would only become clearer. Although the field has much progress ahead, we conclude that, with its learning approach closer to reality, joint learning overall is a preferable training setup for causal prior-data fitted networks.

Acknowledgments

This material is based in part upon work supported by the German Federal Ministry of Education and Research (BMBF): Tübingen AI Center, FKZ: 01IS18039B; by the Machine Learning Cluster of Excellence, EXC number 2064/1 – Project number 390727645; by NSERC Discovery Grant RGPIN-2025-06491; by an NSERC Canada Graduate Research Scholarship – Master’s (CGRS M); by the University of Toronto’s Acceleration Consortium, which receives funding from the Canada First Research Excellence Fund (CFREF); Resources used in preparing this research project were provided, in part, by the Digital Research Alliance of Canada; the Province of Ontario; the Swiss AI Compute grants; the Government of Canada through CIFAR; and companies sponsoring the Vector Institute. We thank Rahul Shrestha, Yongjin Yang, Erivan Inan, and Angelo Huang for their helpful comments. We further acknowledge the work of AdminBot for administrative support that streamlined our research workflow.

AI use statement

In this work, we used generative AI tools (Claude Opus 5.5) to assist in writing and formatting, editing software code, summarizing existing literature, and targeted information retrieval in papers verified by the authors.

We did not use generative AI tools to generate datasets, develop theoretical models or conceptual frameworks, formulate mathematical claims or critical ingredients for proofs, assist in writing proofs, propose or refine research hypotheses, or design or provide feedback on the research methodology or experiments. The rest of the required disclosure tasks are not applicable to this work. Synthetic data used in training and evaluation were generated using the benchmark mechanisms described in § 4, § 5 and Appendix § 16 rather than by generative AI tools.

We have reviewed all AI-assisted work. AI-generated code was verified and tested for correctness before any interpretation of results. We take responsibility for the final content of this work, including text, claims or artifacts produced with the aid of generative AI.

Reproducibility statement

We provide the methodological and experimental details needed to reproduce the work throughout the paper and appendices. § 3 and Appendix B specify the joint learning architecture and training objective, while Appendix § 12 provides the proofs of the theoretical results. Appendices § 13 and § 14 describe the ITE- and ATE-density inference procedures, including optimal-transport aggregation. § 4 and Appendix § 16 describe the experimental setup and benchmark construction, including modifications to existing benchmarks, and Appendix § 15 defines the evaluation protocol and metrics. Appendix H provide the additional results and ablations, including the matched-resolution, matched-label, correlation, and checkpoint comparisons. Finally, Appendix § 18 reports the computational environment, software versions, hardware, model sizes, training steps, throughput, and compute cost. The code is available on GitHub.

Overview

The appendix is organised as follows. Appendix B details the joint output distribution and its training loss. Appendix C contains the theoretical results, namely the proof of Lemma 1, the extension of the results of Robertson et al. [31] to the joint distribution, and the comparison of the joint and single-outcome losses. Appendix D describes how the ITE density is obtained for each model type, and Appendix E how the ATE density is obtained by optimal transport and calibrated on the model’s prior. Appendix F defines the evaluation protocol and metrics, and Appendix G the construction of the four benchmarks. Appendix H collects the additional results, including the interval score on RealCause, the point estimation on the case studies and ComplexMech, the counterfactual prediction on Twins, the ATE density on the benchmarks, the matched-resolution and matched-label ablations, the robustness to the cross-world correlation, and the comparisons against the published CausalPFN and re-trained UWYK checkpoints. Appendix I lists the computational details.

Joint Output Distribution

The joint distribution predicted by a jointly trained model (Equation (5)) must return a finite density at every point of R2\mathbb{R}^{2} so that the log-loss is well-posed on unbounded outcomes, while remaining expressive enough on the bounded region I=[a,b]\mathcal{I} = [a,b].3

Grid and indexing. Partition I\mathcal{I} into a uniform K×KK \times K grid of square bins of width h:=(b−a)/Kh := (b-a)/K. Bins are indexed by (j0,j1)∈{0,…,K−1}2(j_0,j_1) \in\{0,\ldots,K-1\}^{2}; bin Bj0,j1B_{j_0,j_1} is the h×hh \times h square with lower-left corner (a+j0h,a+j1h)(a+j_0h,a+j_1h). For any y∈[a,b]y \in[a,b] and coordinate k∈{0,1}k \in\{0,1\}, the containing bin index along that coordinate is

jk(y):=⌊(y−a)/h⌋.(10)j_k(y) := \left\lfloor(y-a)/h \right\rfloor. \tag*{(10)}

Per-query outputs. For each query the transformer emits K2+9+4K^{2}+9+4 real numbers, grouped into three tensors mapped to normalised parameters by fixed link functions:

  • Interior logits ℓ∈RK2\ell\in\mathbb{R}^{K^{2}} softmaxed into a K×KK \times K probability table over the bins,

pj0,j1(X,Dobs)=exp⁡(ℓj0,j1)∑i0,i1exp⁡(ℓi0,i1);(11)p_{j_0,j_1}(X,\mathcal{D}_{\mathrm{obs}}) = \frac{\exp(\ell_{j_0,j_1})}{\sum_{i_0,i_1}\exp(\ell_{i_0,i_1})}; \tag*{(11)}
  • Region logits u∈R9u \in\mathbb{R}^{9} softmaxed into non-negative mixture weights across the nine regions defined below,

wr(X,Dobs)=exp⁡(ur)∑s=19exp⁡(us),r∈{1,…,9};(12)w_r(X,\mathcal{D}_{\mathrm{obs}}) = \frac{\exp(u_r)}{\sum_{s=1}^{9}\exp(u_s)}, \qquad r \in\{1,\ldots,9\}; \tag*{(12)}
  • Raw tail scales (σL(0),raw,σR(0),raw,σL(1),raw,σR(1),raw)(\sigma_L^{(0),\mathrm{raw}},\sigma_R^{(0),\mathrm{raw}},\sigma_L^{(1),\mathrm{raw}},\sigma_R^{(1),\mathrm{raw}}), one for each coordinate–direction pair, passed through softplus with a small floor ε>0\varepsilon> 0 and scaled by hh,

σd(k)(X,Dobs)=h⋅(softplus⁡(σd(k),raw)+ε),k∈{0,1}, d∈{L,R}.(13)\sigma_d^{(k)}(X,\mathcal{D}_{\mathrm{obs}}) = h \cdot\left(\operatorname{softplus}(\sigma_d^{(k),\mathrm{raw}})+\varepsilon\right), \qquad k \in\{0,1\},\ d \in\{L,R\}. \tag*{(13)}

The hh prefactor in equation (13) makes the tail scales compatible with the interior resolution: doubling KK halves both the bin width and the initial tail width, so the density is invariant under resolution changes at initialisation.

Nine regions. The plane R2\mathbb{R}^{2} is partitioned into nine regions in three groups: (1) the interior square I\mathcal{I}, where both y0,y1∈[a,b]y_{0}, y_{1} \in[a,b]; (2) four edge strips, where exactly one of y0,y1y_{0}, y_{1} lies outside [a,b][a,b]; and (3) four corner quadrants, where both lie outside. Every query point by default falls in one region, so at inference the mixture weight wrw_{r} selects a single component; only its density contributes to ff and to log⁡f\log f.

Interior region (y0,y1∈[a,b]y_{0}, y_{1} \in[a,b]). The histogram is converted to a piecewise-constant density by dividing by the bin area,

finner(y0,y1∣X,Dobs)=pj0(y0),j1(y1)h2.(14)f_{\mathrm{inner}}(y_{0}, y_{1} \mid X, \mathcal{D}_{\mathrm{obs}}) = \frac{p_{j_{0}(y_{0}),j_{1}(y_{1})}}{h^{2}}. \tag*{(14)}

The h2h^{2} factor is the Jacobian from bin probability to density on [a,b]2[a,b]^{2}.

Edge regions (yi∈[a,b],y1−i∉[a,b]y_{i} \in[a,b], y_{1-i} \notin[a,b]). Consider the representative subcase y0∈[a,b]y_{0} \in[a,b] and y1<ay_{1} < a; the other three edges are symmetric. We factor the edge density as p(y1)p(y0∣y1)p(y_{1})p(y_{0} \mid y_{1}), model the marginal p(y1)p(y_{1}) by a half-Gaussian anchored at the boundary, and approximate the conditional p(y0∣y1)p(y_{0} \mid y_{1}) by the inner region at the boundary:

fedge(y0,y1∣X,Dobs)=pL(1)(y1) g(y0∣y1=a),(15)f_{\mathrm{edge}}(y_{0}, y_{1} \mid X, \mathcal{D}_{\mathrm{obs}}) = p_{L}^{(1)}(y_{1})\,g(y_{0} \mid y_{1}=a), \tag*{(15)}

with

pL(1)(y1)=2σL(1)ϕ(y1−aσL(1)),g(y0∣y1=a)=pj0(y0),0(∑ipi0,0)h,(16)p_{L}^{(1)}(y_{1}) = \frac{2}{\sigma_{L}^{(1)}}\phi\left(\frac{y_{1}-a}{\sigma_{L}^{(1)}}\right), \qquad g(y_{0} \mid y_{1}=a) = \frac{p_{j_{0}(y_{0}),0}}{\left(\sum_i p_{i0,0}\right)h}, \tag*{(16)}

where ϕ\phi is the standard normal density. Holding p(y0∣y1)p(y_{0} \mid y_{1}) fixed at y1=ay_{1}=a is exact when Y0⊥Y1Y_{0} \perp Y_{1} and otherwise introduces a bias that grows with the strength of dependence and the distance past the boundary; for a bivariate normal with correlation ρ\rho and tail scale σ\sigma the excess NLL per edge observation is approximately ρ2σ2/π(1−ρ2)\rho^{2}\sigma^{2}/\pi(1-\rho^{2}). The expression grows without bound as ∣ρ∣→1|\rho| \to1. The approximation, however, enters only the edge regions, which hold the observations with exactly one outcome outside the grid.

Corner regions (y0,y1∉[a,b]y_{0}, y_{1} \notin[a,b]). Let (s0,s1)∈{L,R}2(s_{0},s_{1}) \in\{L,R\}^{2} select the quadrant with corner (c0,c1)(c_{0},c_{1}), with ck=ac_{k}=a if sk=Ls_{k}=L and ck=bc_{k}=b if sk=Rs_{k}=R. The corner density is a bivariate normal centred at the corner, with correlation ρ\rho (defined below) and standardised coordinates y~k=(yk−ck)/σsk(k)\tilde{y}_{k}=(y_{k}-c_{k})/\sigma_{s_{k}}^{(k)}:

fcorner(y0,y1∣X,Dobs)=1Z(s0,s1,ρ)ϕ2(y~0,y~1;ρ).(17)f_{\mathrm{corner}}(y_{0},y_{1} \mid X,\mathcal{D}_{\mathrm{obs}}) = \frac{1}{Z(s_{0},s_{1},\rho)}\phi_{2}(\tilde{y}_{0},\tilde{y}_{1};\rho). \tag*{(17)}

The truncation constant ZZ is the probability mass the bivariate normal places in the quadrant selected by (s0,s1)(s_{0},s_{1}); by Sheppard’s identity,

Z(s0,s1,ρ)={14+arcsin⁡ρ2πif s0=s1,14−arcsin⁡ρ2πif s0≠s1.(18)Z(s_{0},s_{1},\rho)= \begin{cases} \frac{1}{4}+\frac{\arcsin\rho}{2\pi} & \text{if } s_{0}=s_{1},\\ \frac{1}{4}-\frac{\arcsin\rho}{2\pi} & \text{if } s_{0}\ne s_{1}. \end{cases} \tag*{(18)}

Correlation inherited from the interior. The correlation ρ\rho in equation (17)–equation (18) is not a separate trained parameter by design choice. It is defined as the Pearson correlation of the discrete distribution {pj0,j1}\{p_{j_0,j_1}\} evaluated at bin midpoints mk(jk):=a+h(jk+12)m_k(j_k) := a + h\left(j_k + \frac{1}{2}\right):

yˉk=∑j0,j1pj0,j1mk(jk),\bar{y}_k = \sum_{j_0,j_1} p_{j_0,j_1}m_k(j_k),
vk=∑j0,j1pj0,j1(mk(jk)−yˉk)2,v_k = \sum_{j_0,j_1} p_{j_0,j_1}\left(m_k(j_k)-\bar{y}_k\right)^2,
c=∑j0,j1pj0,j1(m0(j0)−yˉ0)(m1(j1)−yˉ1),c = \sum_{j_0,j_1} p_{j_0,j_1}\left(m_0(j_0)-\bar{y}_0\right)\left(m_1(j_1)-\bar{y}_1\right),
ρ=cv0v1+ϵ.(19)\rho= \frac{c}{\sqrt{v_0v_1+\epsilon}}. \tag*{(19)}

Tying ρ\rho to the interior grid removes one degree of freedom, keeps the corner coupling consistent with what the interior region already represents, and prevents the tail and interior components from disagreeing about the dependence between Ydo(0)Y_{\mathrm{do}(0)} and Ydo(1)Y_{\mathrm{do}(1)}.

Full mixture and training loss. The joint density is

f(y0,y1∣X,Dobs)=w1finner+∑r=25wrfedge(r)+∑r=69wrfcorner(r),(20)f(y_0,y_1 \mid X,\mathcal{D}_{\mathrm{obs}}) = w_1 f_{\mathrm{inner}} + \sum_{r=2}^{5} w_r f_{\mathrm{edge}}^{(r)} + \sum_{r=6}^{9} w_r f_{\mathrm{corner}}^{(r)}, \tag*{(20)}

and the negative log-density loss is evaluated at the region of the observed label (y0⋆,y1⋆)(y_0^\star,y_1^\star); the region indicator selects a single component, so only its weight and parameters receive gradient at that point. Because the region assignment is deterministic in the label, the mixture weights ww are trained purely from which region observations fall into, while the interior grid, tail scales, and derived correlation are trained within each region.

Theoretical Results

Proof of Lemma 1

Lemma 1 Let ZiR:=(Xi,Ydo(0)(xi,ϵi),Ydo(1)(xi,ϵi))Z_i^{\mathrm{R}} := (X_i,Y_{\mathrm{do}(0)}(x_i,\epsilon_i),Y_{\mathrm{do}(1)}(x_i,\epsilon_i)) denote the per-unit observation under the joint objective and ZiU:=(Xi,Ti,Ydo(Ti)(xi,ϵi))Z_i^{\mathrm{U}} := (X_i,T_i,Y_{\mathrm{do}(T_i)}(x_i,\epsilon_i)) the per-unit observation under the single-outcome objective, with TiT_i drawn from any distribution on {0,1}\{0,1\} independent of ϵi\epsilon_i. Then ZUZ^{\mathrm{U}} is a randomised function of ZRZ^{\mathrm{R}}: sampling TT from the same distribution and setting ZU=(X,T,Ydo(T))Z^{\mathrm{U}}=(X,T,Y_{\mathrm{do}(T)}) recovers the law of the single-outcome observation from that of the paired one. Consequently, any parameter of the joint law P⋆(Ydo(0),Ydo(1)∣X)P^\star(Y_{\mathrm{do}(0)},Y_{\mathrm{do}(1)} \mid X) estimable from ZUZ^{\mathrm{U}} is estimable from ZRZ^{\mathrm{R}} at least as well.

Proof. Let λ\lambda denote the law of TT on {0,1}\{0,1\}, chosen independently of ϵi\epsilon_i. Define the measurable map Φ(x,y0,y1;t):=(x,t,yt)\Phi(x,y_0,y_1;t):=(x,t,y_t); then for any ZRZ^{\mathrm{R}} with joint law P⋆P^\star,

ZU=dΦ(ZR;T),T∼λ⊥ZR,(21)Z^{\mathrm{U}} \mathrel{\overset{d}{=}} \Phi(Z^{\mathrm{R}};T), \qquad T \sim\lambda\perp Z^{\mathrm{R}}, \tag*{(21)}

recovers the law of the marginal observation from the joint one. The map is measurable and marginalises over TT without reference to P⋆P^\star, so the experiment observing ZUZ^{\mathrm{U}} is a garbling of the one observing ZRZ^{R}. By Blackwell’s comparison of experiments, any procedure based on ZUZ^{U} can be matched by one based on ZRZ^{R}, and the Fisher information about any parameter of P⋆P^{\star} is no larger under ZUZ^{U}; hence any parameter estimable from ZUZ^{U} is estimable from ZRZ^{R} at least as well.

Extension of the results of Robertson et al. (2025)

Robertson et al. [31] factorise the posterior predictive distribution of an interventional outcome as p(y∣x,do⁡(t),Dobs)=∬p(y∣x,do⁡(t),ψ)p(ψ∣r)p(r∣Dobs) dr dψp(y \mid x, \operatorname{do}(t), \mathcal{D}_{\mathrm{obs}}) = \iint p(y \mid x, \operatorname{do}(t), \psi) p(\psi\mid r) p(r \mid\mathcal{D}_{\mathrm{obs}})\,\mathrm{d}r\,\mathrm{d}\psi, where rr is the representative of the observational equivalence class of ψ\psi (their Equation 4), and show in their Proposition 3 that, for observational data drawn i.i.d. from an SCM ψ0\psi_0, this distribution converges weakly to p(y∣x,do⁡(t),[ψ0])p(y \mid x, \operatorname{do}(t), [\psi_0]) as the number of observations grows. Neither result uses the form of the target beyond two properties: it takes values in a complete separable metric space, and, given ψ\psi, its distribution does not depend on Dobs\mathcal{D}_{\mathrm{obs}}. Both hold for the pair (Ydo⁡(0),Ydo⁡(1))(Y_{\operatorname{do}(0)}, Y_{\operatorname{do}(1)}), which takes values in R2\mathbb{R}^{2} and, given ψ\psi, is generated by the two interventions from the same exogenous noise, independently of Dobs\mathcal{D}_{\mathrm{obs}}. Replacing yy by (y0,y1)(y_0,y_1) in their factorisation gives the same three sources of uncertainty for Equation (5): the noise within an SCM, the unidentifiability within an equivalence class, and the epistemic uncertainty about the class. Their proof of Proposition 3 then applies with Borel sets B⊆R2B \subseteq\mathbb{R}^{2}, where the two tower steps and Doob’s martingale convergence theorem give the weak convergence of p(y0,y1∣x,Dobs)p(y_0,y_1 \mid x,\mathcal{D}_{\mathrm{obs}}) to p(y0,y1∣x,[ψ0])p(y_0,y_1 \mid x,[\psi_0]).

Comparison of the joint and single-outcome losses

Proposition 2. Let pp be the distribution of (Ydo⁡(0),Ydo⁡(1))(Y_{\operatorname{do}(0)},Y_{\operatorname{do}(1)}) given xx, with finite second moments, marginals p0p_0 and p1p_1, variances σ02\sigma_0^2 and σ12\sigma_1^2, correlation ρ\rho, and δ=Ydo⁡(1)−Ydo⁡(0)\delta= Y_{\operatorname{do}(1)} - Y_{\operatorname{do}(0)}. For a prediction whose mean of δ\delta is off by ee, let CJT(e)C_{\mathrm{JT}}(e), CML(e)C_{\mathrm{ML}}(e) and CBase(e)C_{\mathrm{Base}}(e) be the smallest expected loss increase per unit under the joint loss, KL⁡(p∥q)\operatorname{KL}(p \mathbin{\|} q), under the matched-label loss, KL⁡(p0∥q0)+KL⁡(p1∥q1)\operatorname{KL}(p_0 \mathbin{\|} q_0) + \operatorname{KL}(p_1 \mathbin{\|} q_1), and under the single-outcome loss, which scores one outcome per unit, each with probability 1/21/2, 12KL⁡(p0∥q0)+12KL⁡(p1∥q1)\frac{1}{2}\operatorname{KL}(p_0 \mathbin{\|} q_0) + \frac{1}{2}\operatorname{KL}(p_1 \mathbin{\|} q_1). To second order in ee,

CJT(e)=e22Var⁡p(δ),CML(e)=e22(σ02+σ12),CBase(e)=e24(σ02+σ12).C_{\mathrm{JT}}(e) = \frac{e^2}{2\operatorname{Var}_p(\delta)}, \qquad C_{\mathrm{ML}}(e) = \frac{e^2}{2(\sigma_0^2+\sigma_1^2)}, \qquad C_{\mathrm{Base}}(e) = \frac{e^2}{4(\sigma_0^2+\sigma_1^2)}.

Since Var⁡p(δ)=σ02+σ12−2ρσ0σ1\operatorname{Var}_p(\delta) = \sigma_0^2+\sigma_1^2-2\rho\sigma_0\sigma_1, CJT(e)>CML(e)C_{\mathrm{JT}}(e) > C_{\mathrm{ML}}(e) whenever ρ>0\rho> 0, and since Var⁡p(δ)≤(σ0+σ1)2≤2(σ02+σ12)\operatorname{Var}_p(\delta) \leq(\sigma_0+\sigma_1)^2 \leq2(\sigma_0^2+\sigma_1^2), CJT(e)≥CBase(e)C_{\mathrm{JT}}(e) \geq C_{\mathrm{Base}}(e) for every ρ\rho.

Proof. To second order, KL⁡(p∥q)=12∫(q−p)2/p\operatorname{KL}(p \mathbin{\|} q) = \frac{1}{2}\int(q-p)^2/p. Minimising this subject to ∫(q−p)=0\int(q-p)=0 and ∫δ(q−p)=e\int\delta(q-p)=e gives q−p=cp(δ−Epδ)q-p = c p(\delta-\mathbb{E}_p\delta) with c=e/Var⁡p(δ)c=e/\operatorname{Var}_p(\delta), and the value e2/(2Var⁡p(δ))e^2/(2\operatorname{Var}_p(\delta)). For a single outcome, the same argument gives the cost et2/(2σt2)e_t^2/(2\sigma_t^2) of a shift ete_t in its mean. Minimising e02/(2σ02)+e12/(2σ12)e_0^2/(2\sigma_0^2)+e_1^2/(2\sigma_1^2) subject to e1−e0=ee_1-e_0=e gives e2/(2(σ02+σ12))e^2/(2(\sigma_0^2+\sigma_1^2)) for the matched-label loss, and the single-outcome loss is half of it.

ITE Density Estimation

This section details how the ITE density fδf_\delta is obtained for the single-outcome and the jointly trained models.

Single-outcome models. A single-outcome model outputs the two marginal densities p^(y∣x,do(0))\hat{p}(y \mid x, do(0)) and p^(y∣x,do(1))\hat{p}(y \mid x, do(1)) separately, with CDFs F^0\hat{F}_0 and F^1\hat{F}_1. Since it does not learn their dependence, a value of ρ\rho has to be assumed. Under a Gaussian copula with parameter ρ\rho, the ITE density is

fδ(d)=∫p^(y+d∣x,do(1))p^(y∣x,do(0))cρ(F^1(y+d),F^0(y)) dy,(22)f_{\delta}(d) = \int\hat{p}(y+d \mid x, do(1))\hat{p}(y \mid x, do(0))c_{\rho}\left(\hat{F}_1(y+d),\hat{F}_0(y)\right)\,dy, \tag*{(22)}

where, with a=Φ−1(u)a=\Phi^{-1}(u) and b=Φ−1(v)b=\Phi^{-1}(v),

cρ(u,v)=11−ρ2exp⁡(−ρ2(a2+b2)−2ρab2(1−ρ2)).(23)c_{\rho}(u,v) = \frac{1}{\sqrt{1-\rho^2}}\exp\left(-\frac{\rho^2(a^2+b^2)-2\rho ab}{2(1-\rho^2)}\right). \tag*{(23)}

At ρ=0\rho=0, cρ=1c_{\rho}=1 and the ITE density reduces to the convolution of the marginals under independence, the default of these models, computed on the grid as the discrete cross-correlation of the two bin vectors. At ρ=1\rho=1, it is the comonotone limit.

Why the true ρ\rho is not sufficient. The copula is applied to the predicted distributions of the two outcomes, whose spread contains both the noise and the model’s uncertainty about μ0\mu_0 and μ1\mu_1. The true correlation ρ(x)\rho(x) of Equation (3) describes only the noise, so even when it is known, it does not in general describe how the two predicted outcomes depend on each other. A simple example helps us illustrate this. Let Ydo(t)=μt+εY_{do(t)}=\mu_t+\varepsilon with shared noise ε∼N(0,1)\varepsilon\sim N(0,1), so that the true correlation is ρ(x)=1\rho(x)=1, and let the model’s uncertainty about μ1\mu_1 and μ0\mu_0 be independent, with variance 1 each. Since the shared noise cancels in the ITE, the correct variance of the ITE is Var⁡(μ1−μ0)=2\operatorname{Var}(\mu_1-\mu_0)=2. Coupling the two predicted distributions at the true value ρ=1\rho=1, however, gives a variance of 0, and assuming independence gives 4. Giving a single-outcome model the true ρ(x)\rho(x) therefore does not recover the correct ITE distribution. The dependence it would need varies with the model’s uncertainty about each Dobs\mathcal{D}_{\mathrm{obs}}, and a single-outcome PFN outputs only the two separate predicted distributions, so this dependence can neither be specified in advance nor recovered from its output. The joint model does not face this problem, since it learns the two outcomes together and therefore learns their dependence as well.

Jointly trained models. A jointly trained model outputs p^(ydo(0),ydo(1)∣x,Dobs)\hat{p}(y_{do(0)},y_{do(1)} \mid x,\mathcal{D}_{\mathrm{obs}}) directly, so the ITE density requires no assumption on the dependence. It is the pushforward of the joint under the difference map, as in Equation (8), and it is computed from the two parts of its output. On the interior K×KK\times K grid, of bin width hh with centres (y0i,y1j)(y_0^i,y_1^j), the model places probability pijp_{ij}, summing to one over the interior, on cell (i,j)(i,j) and spreads it uniformly over the cell. Under the difference, each cell becomes a triangle of half-width hh centred at y1j−y0iy_1^j-y_0^i. The remaining mass lies in the eight tail regions of Equation (20), and each tail component f(r)f^{(r)} becomes the density fδ(r)(d)=∫f(r)(y,y+d) dyf_{\delta}^{(r)}(d)=\int f^{(r)}(y,y+d)\,dy. The ITE density is therefore the mixture

fδ(d)=w1∑ijpij1hΛ(d−(y1j−y0i)h)+∑r=29wrfδ(r)(d),Λ(u)=max⁡(0,1−∣u∣),(24)f_{\delta}(d)=w_1\sum_{ij}p_{ij}\frac{1}{h}\Lambda\left(\frac{d-(y_1^j-y_0^i)}{h}\right)+\sum_{r=2}^{9}w_r f_{\delta}^{(r)}(d),\qquad\Lambda(u)=\max(0,1-|u|), \tag*{(24)}

with the region weights wrw_r of Equation (20). The learned dependence enters through the joint probabilities pijp_{ij}.

ATE Density via Optimal Transport

Section 3.2 defines the ATE density fATEf_{\mathrm{ATE}} as the scaled, uniformly-weighted 2-Wasserstein barycenter of the per-query ITE densities {fδ(i)}i=1n\{f_{\delta}^{(i)}\}_{i=1}^{n} through its quantile function in equation (9). Since each fδ(i)f_{\delta}^{(i)} is continuous (Equation (24)), its CDF inverts cleanly and the barycenter follows directly. In this segment we report the derivation, the grid-level algorithm, the calibration of the scale, and finally an illustration of the process (Figure 4).

Notation. Let FiF_i and Qi:=Fi−1Q_i := F_i^{-1} be the CDF and quantile function of the per-query ITE density fδ(i)f_{\delta}^{(i)}, and let α∈Δn−1\alpha\in\Delta^{n-1} be nonnegative weights summing to 1. On R\mathbb{R} the weighted 2-Wasserstein barycenter of {fδ(i)}i=1n\{f_{\delta}^{(i)}\}_{i=1}^{n} admits the closed-form quantile representation [1]

Q⋆(u)=∑i=1nαiQi(u),u∈(0,1),(25)Q^{\star}(u) = \sum_{i=1}^{n} \alpha_i Q_i(u), \qquad u \in(0,1), \tag*{(25)}

and the barycenter in equation (9) is its uniform-weight case αi=1/n\alpha_i = 1/n, before scaling.

Grid-level implementation. Because Q⋆Q^{\star} is defined explicitly by equation (25), no iterative optimisation is needed. Given the per-query densities sampled on a common grid x1<⋯<xMx_1 < \cdots< x_M with spacing Δx\Delta x, the barycenter density is produced in three quantile passes and one density-recovery pass:

  1. Discretise the probability axis by τj:=(j−12)/Mτ\tau_j := (j - \frac{1}{2})/M_{\tau} for j=1,…,Mτj = 1,\ldots,M_{\tau} (we use Mτ=4001M_{\tau} = 4001).

  1. For each query i∈{1,…,n}i \in\{1,\ldots,n\}, compute the CDF by trapezoidal integration, renormalise, and invert to obtain the quantile function at every τj\tau_j:

Fi(xk)←∑ℓ≤k12(p^i(xℓ−1)+p^i(xℓ))Δx,Fi←Fi/Fi(xM),Qi(τj)←Fi−1(τj).F_i(x_k) \leftarrow\sum_{\ell\le k} \frac{1}{2}\bigl(\hat{p}_i(x_{\ell-1}) + \hat{p}_i(x_\ell)\bigr)\Delta x,\qquad F_i \leftarrow F_i/F_i(x_M),\qquad Q_i(\tau_j) \leftarrow F_i^{-1}(\tau_j).
  1. Average the quantile functions pointwise per equation (25): Q⋆(τj)=∑iαiQi(τj)Q^{\star}(\tau_j) = \sum_i \alpha_i Q_i(\tau_j).

  1. Recover the barycenter density on the original grid: invert Q⋆Q^{\star} back to a CDF F⋆F^{\star}, take a central-difference derivative, and renormalise to unit integral,

f^ATE(xk)←F⋆(xk+1)−F⋆(xk−1)2Δx,f^ATE←f^ATE/(Δx∑kf^ATE(xk)).\hat{f}_{\mathrm{ATE}}(x_k) \leftarrow\frac{F^{\star}(x_{k+1}) - F^{\star}(x_{k-1})}{2\Delta x},\qquad\hat{f}_{\mathrm{ATE}} \leftarrow\hat{f}_{\mathrm{ATE}}/\left(\Delta x\sum_k \hat{f}_{\mathrm{ATE}}(x_k)\right).

Every step is O(n⋅M)O(n \cdot M) vector arithmetic; producing f^ATE\hat{f}_{\mathrm{ATE}} therefore adds little cost to computing the per-query ITE densities it aggregates.

Calibration of the scale. The barycenter fixes the location and shape of fATEf_{\mathrm{ATE}}, and the factor ss in equation (9) sets its scale. We set ss once per model by split conformal calibration (Vovk et al., 2005). For each model, we draw N=500N = 500 datasets from its training prior, where the true ATE is known, and compute the barycenter of each. For dataset jj, let rjr_j be the smallest factor for which the central 95% interval of the rescaled barycenter contains the true ATE. The scale ss is the ⌈0.95(N+1)⌉\lceil0.95(N + 1)\rceil-th smallest of r1,…,rNr_1,\ldots,r_N, which guarantees a coverage of at least 95% on new datasets from the same prior. Rescaling around τˉ\bar{\tau} leaves the ATE point estimate unchanged. For the single-outcome models, whose ITE densities depend on ρ\rho, ss is set separately for each ρ\rho, and for UWYK separately for the No-Anc and Anc variants. Appendix § 15 reports the in-prior coverage of the resulting intervals.

Scope and future directions. The scaled barycenter is a first step toward an ATE density for causal PFNs, and it approximates, rather than equals, the posterior of the ATE. Averaging quantile functions couples the units comonotonically, whereas the exact posterior depends on the joint posterior of the ITEs across units, which share the uncertainty about the SCM. The calibration of ss matches the spread of the density to the ATE errors on the model’s prior, as the held-out coverage in Table 4 confirms, and the resulting densities let the JT models attain better ATE interval scores than their baselines in 42 of the 48 comparisons (Appendix H.§ 4). The calibration, however, does not fix the shape of the density. We therefore see this construction as a practical starting point on which future work can build, for example by modeling the dependence between units or by learning the ATE density directly.

ρ=−0.5\rho=-0.5ρ=0\rho=0ρ=0.5\rho=0.5ρ=1\rho=1JT
Do-PFN95.0±1.095.0 \pm1.094.8±1.094.8 \pm1.094.8±1.094.8 \pm1.095.2±1.095.2 \pm1.095.2±1.095.2 \pm1.0
UWYK No-Anc95.4±0.995.4 \pm0.995.4±0.995.4 \pm0.994.8±1.094.8 \pm1.095.0±1.095.0 \pm1.094.8±1.094.8 \pm1.0
UWYK Anc95.4±0.995.4 \pm0.995.0±1.095.0 \pm1.095.2±1.095.2 \pm1.094.8±1.094.8 \pm1.094.8±1.094.8 \pm1.0
CausalPFN-C95.2±1.095.2 \pm1.095.2±1.095.2 \pm1.095.0±1.095.0 \pm1.094.6±1.094.6 \pm1.094.8±1.094.8 \pm1.0

Table 4. In-prior calibration of the ATE density. Cells give the coverage (%) of its 95% interval on held-out datasets ±\pm SE over the 500 datasets of each model’s prior, with the scale ss set on the other four of five folds. Single-outcome models under each ρ\rho, and JT models with the learned dependence.

Illustration. Figure 4 shows the aggregation on one IHDP realization. The faint curves are the per-query ITE densities, and the bold curve is the ATE density. The barycenter averages the locations of the per-query densities while preserving their shape. The Do-PFN and UWYK JT models concentrate the per-query densities, and their ATE density tightens around the true value. CausalPFN-C JT looks wider than its baseline, mainly because its bin width is 32 times larger.

Evaluation Metrics

Evaluation protocol. For each benchmark setting we draw BB SCMs ψ1,…,ψB\psi_1,\ldots,\psi_B independently. From each ψb\psi_b we generate a single observational set Dobs(b)\mathcal{D}_{\mathrm{obs}}^{(b)}, independently across SCMs, and QQ query covariates xb1,…,xbQx_{b1},\ldots,x_{bQ}, each with a target effect τb(xbq)\tau_b(x_{bq}) and a population target ATEb\mathrm{ATE}_b, defined below, and the realized ITE δbq\delta_{bq} of each query unit. Each model is conditioned on Dobs(b)\mathcal{D}_{\mathrm{obs}}^{(b)} and returns, for every query, a CATE estimate τ^b(xbq)\hat{\tau}_b(x_{bq}) and an ITE density, and, per SCM, an ATE estimate ATE^b\widehat{\mathrm{ATE}}_b.

Point estimation. Per SCM we compute

PEHEb=(1Q∑q=1Q(τ^b(xbq)−τb(xbq))2)1/2,ϵATE,b=∣ATE^b−ATEb∣max⁡(∣ATEb∣,0.1).(26)\sqrt{\mathrm{PEHE}_b} = \left(\frac{1}{Q}\sum_{q=1}^{Q}\left(\hat{\tau}_b(x_{bq})-\tau_b(x_{bq})\right)^2\right)^{1/2}, \qquad\epsilon_{\mathrm{ATE},b} = \frac{\left|\widehat{\mathrm{ATE}}_b-\mathrm{ATE}_b\right|}{\max\left(\left|\mathrm{ATE}_b\right|,0.1\right)}. \tag*{(26)}

and report their means over the BB SCMs; the floor of 0.1 keeps ϵATE\epsilon_{\mathrm{ATE}} finite when the true ATE is close to zero, as in the case studies with a zero mean shift. The target τb(xbq)\tau_b(x_{bq}) is the analytical CATE μ1(xbq)−μ0(xbq)\mu_1(x_{bq})-\mu_0(x_{bq}) where the data-generating process provides one (IHDP and ACIC); elsewhere it is the effect of the query unit itself. It is the realized ITE δbq\delta_{bq} on CPS, PSID and ComplexMech, and on the case studies. ATEb\mathrm{ATE}_b is the corresponding mean. Evaluating against a unit’s own effect τ~bq\tilde{\tau}_{bq} rather than the CATE τb(xbq)\tau_b(x_{bq}) does not affect the comparison. Write ηbq=τ~bq−τb(xbq)\eta_{bq}=\tilde{\tau}_{bq}-\tau_b(x_{bq}), so that E[ηbq∣xbq]=0\mathbb{E}[\eta_{bq}\mid x_{bq}]=0. Since the query unit’s outcomes are not seen by the model, ηbq\eta_{bq} is independent of τ^b(xbq)\hat{\tau}_b(x_{bq}) given xbqx_{bq}, and the cross term vanishes:

E[(τ^b(xbq)−τ~bq)2]=E[(τ^b(xbq)−τb(xbq))2]+E[ηbq2].(27)\mathbb{E}\left[(\hat{\tau}_b(x_{bq})-\tilde{\tau}_{bq})^2\right]=\mathbb{E}\left[(\hat{\tau}_b(x_{bq})-\tau_b(x_{bq}))^2\right]+\mathbb{E}\left[\eta_{bq}^2\right]. \tag*{(27)}

The second term does not depend on the model, so the ranking is unchanged in expectation, and the error against the CATE is smaller than the reported one by E[ηbq2]\mathbb{E}[\eta_{bq}^2].

Posterior predictive intervals. For each query, the 95% posterior predictive interval PIb(xbq)=[lbq,ubq]\mathrm{PI}_b(x_{bq})=[l_{bq},u_{bq}] is the central interval between the 2.5% and 97.5% quantiles of the predicted ITE density, and its length is ubq−lbqu_{bq}-l_{bq}.

Interval score. We evaluate these intervals by the interval score ([12]) at α=0.05\alpha=0.05, which combines coverage and length into a single metric,

ISbq=(ubq−lbq)+2α(lbq−δbq)1{δbq<lbq}+2α(δbq−ubq)1{δbq>ubq}.(28)\mathrm{IS}_{bq}=(u_{bq}-l_{bq})+\frac{2}{\alpha}(l_{bq}-\delta_{bq})\mathbf{1}\{\delta_{bq}<l_{bq}\}+\frac{2}{\alpha}(\delta_{bq}-u_{bq})\mathbf{1}\{\delta_{bq}>u_{bq}\}. \tag*{(28)}

It equals the interval length when the realized ITE is covered, and adds a penalty proportional to the distance of a miss otherwise. Within each SCM we average it over queries and normalise it by the outcome scale, yielding one value per observational set,

Sb=1QσY∑q=1QISbq.(29)S_b=\frac{1}{Q\sigma_Y}\sum_{q=1}^{Q}\mathrm{IS}_{bq}. \tag*{(29)}

The reported score is the mean across SCMs, Sˉ=1B∑b=1BSb\bar{S}=\frac{1}{B}\sum_{b=1}^{B}S_b. Since the SCMs and their observational sets are drawn independently, S1,…,SBS_1,\ldots,S_B are i.i.d., so Sˉ\bar{S} is an unbiased estimate of the expected score over the benchmark’s SCM distribution, and its standard error is sS/Bs_S/\sqrt{B}, with sSs_S the sample standard deviation of the SbS_b. Queries within an SCM share one posterior and are therefore dependent; aggregating them into SbS_b before computing uncertainty accounts for this. The benchmark SCMs are not drawn from the models’ training priors, so these intervals carry no coverage guarantee, which holds only on average over SCMs drawn from the prior. The benchmark results therefore show how the models transfer to test data; the in-prior check, where the guarantee holds, is also reported.

Evaluation of CausalPFN. The CausalPFN baseline is trained on mtm_t rather than on the outcomes. This does not change what it predicts in our setting: in our design both outcomes of a unit receive the same idiosyncratic noise draw, which cancels in their difference, so the ITE equals m1−m0m_1-m_0 and the target of the model is the ITE itself. We therefore treat its prediction as an ITE distribution like every other model. The alternative is not viable either. The true CATE is available only on IHDP and ACIC, so evaluating CausalPFN against it would exclude the model from the uncertainty metrics on most datasets, and would leave the models without a common target.

Averages across datasets. For a metric mm where lower is better, with values m1,…,mKm_1,\ldots,m_K on the KK datasets or settings of a comparison, we report the geometric mean

Averageg=(∏k=1Kmk)1/K,(30)\mathrm{Average}_g=\left(\prod_{k=1}^{K}m_k\right)^{1/K}, \tag*{(30)}

with its standard error obtained by the delta method. We report PEHE\sqrt{\mathrm{PEHE}}, the standard metric in the causal inference literature, in its usual units, which differ across datasets. An arithmetic mean of these values would be dominated by the datasets with the largest scale, whereas the ratio of two geometric means does not depend on the scale of any dataset. The percentage improvements in the text are therefore computed from Averageg\mathrm{Average}_g, and do not depend on how the metrics are normalised. The printed Averageg\mathrm{Average}_g of PEHE\sqrt{\mathrm{PEHE}} is standardised by σY\sigma_Y, which leaves these percentages unchanged. The same mean is used for ϵATE\epsilon_{\mathrm{ATE}} and for IS0.05/σY\mathrm{IS}_{0.05}/\sigma_Y, so that all comparisons are made on the same basis.

Significance. For two compared models, let A1A_1 and A2A_2 be their Averageg\mathrm{Average}_g over the KK datasets of a benchmark, computed from the per-dataset means mjkm_{jk} with standard errors sjks_{jk}. By the delta method, the standard error of each Averageg\mathrm{Average}_g is

SE(Aj)=AjK(∑k=1K(sjk/mjk)2)1/2.(31)\mathrm{SE}(A_j)=\frac{A_j}{K}\left(\sum_{k=1}^{K}\left(s_{jk}/m_{jk}\right)^2\right)^{1/2}. \tag*{(31)}

We compare the two by the zz-statistic and its two-sided pp-value,

z=A1−A2SE(A1)2+SE(A2)2,p=2(1−Φ(∣z∣)),(32)z=\frac{A_1-A_2}{\sqrt{\mathrm{SE}(A_1)^2+\mathrm{SE}(A_2)^2}},\qquad p=2(1-\Phi(|z|)), \tag*{(32)}

with Φ\Phi the standard normal distribution function. On Twins, the same test is applied to the mean squared error at each covariate set. The baseline’s error is first averaged over the values of ρ\rho in the range compatible with the data, then over the three jitter widths, as is the JT model’s error over the widths. The standard error of such an average is bounded by the average of the standard errors, which holds whatever the dependence between the averaged terms, and makes the test conservative. The comparisons of one model family within a figure or table, across the assumed values of ρ\rho and, for the ablations, across both ablations, form one family of MM tests. Their pp-values are corrected with the Hochberg procedure. The pp-values are ordered, p(1)≤⋯≤p(M)p_{(1)}\le\cdots\le p_{(M)}, and at level α\alpha every comparison with p(i)≤p(k)p_{(i)}\le p_{(k)} is significant, where kk is the largest index with p(k)≤α/(M−k+1)p_{(k)}\le\alpha/(M-k+1). We apply this at α=0.1\alpha=0.1 and α=0.05\alpha=0.05, and mark the significant differences with * and **, respectively. The procedure assumes positive dependence among the tests, which holds here, since all comparisons of a family share the same JT model and are evaluated on the same SCMs. The test also treats the two models as independent, although both are evaluated on the same SCMs. As their errors are expected to be positively correlated, this overstates the standard error of the difference, so the test is conservative.

In-prior calibration. We report the calibration of our trained models on data drawn from each model’s own training prior, where the ground truth is known, with a new SCM drawn from the prior for every dataset. A posterior predictive interval averages over SCMs according to their posterior, so when the posterior is exact, its coverage equals the nominal level on average over SCMs drawn from the prior. This applies to the predicted outcomes of every model and to the ITE of the JT models; the ITE interval of a single-outcome model also depends on how its two predicted outcomes are combined.

Each dataset has 1000 observational units and 500 query units. Whether a query’s realized ITE falls in its interval is averaged within a dataset, as queries share a context, and the standard error is taken over datasets. Table 3 reports the coverage of each outcome and of the ITE. We report the ITE with shared noise in the two potential outcomes for every model, as the single-outcome models never see the two outcomes of a unit together.

Table 3: In-prior calibration, new SCM each time. Cells give the coverage of the 95% interval ±\pm SE over SCMs, where each SCM gives one dataset. Ydo(1)Y_{\mathrm{do}(1)}, Ydo(0)Y_{\mathrm{do}(0)}: the interval of each predicted outcome, from the marginals of the joint for the JT models. UWYK-B: UWYK re-trained with binary treatments (Appendix H.8). ∗{}^{*}UWYK’s published checkpoint, trained with continuous treatments and evaluated on its own continuous-treatment prior; its columns are the two outcomes Ydo(t1)Y_{\mathrm{do}(t_1)}, Ydo(t0)Y_{\mathrm{do}(t_0)} at two treatment levels t0<t1t_0 < t_1 drawn from that prior, and their contrast Ydo(t1)−Ydo(t0)Y_{\mathrm{do}(t_1)} - Y_{\mathrm{do}(t_0)}.

ModelYdo(1)Y_{\mathrm{do}(1)}Ydo(0)Y_{\mathrm{do}(0)}ITE
Do-PFN0.954±0.000.954 \pm0.000.929±0.000.929 \pm0.000.953±0.010.953 \pm0.01
Do-PFN JT0.964±0.000.964 \pm0.000.949±0.010.949 \pm0.010.944±0.010.944 \pm0.01
UWYK-B0.842±0.010.842 \pm0.010.856±0.010.856 \pm0.010.837±0.010.837 \pm0.01
UWYK∗{}^{*}0.872±0.010.872 \pm0.010.879±0.010.879 \pm0.010.916±0.010.916 \pm0.01
UWYK JT0.972±0.000.972 \pm0.000.971±0.000.971 \pm0.000.963±0.000.963 \pm0.00
CausalPFN-C0.943±0.000.943 \pm0.000.943±0.000.943 \pm0.000.964±0.000.964 \pm0.00
CausalPFN-C JT0.937±0.000.937 \pm0.000.946±0.000.946 \pm0.000.946±0.000.946 \pm0.00
SCMs (RR)100010001000
Queries per SCM (QQ)500500500
Observational units per SCM100010001000

Table 3.

Most models are close to nominal on both the outcomes and the ITE: Do-PFN, Do-PFN JT, UWYK JT, and both CausalPFN-C models lie between 0.93 and 0.97. UWYK-B under-covers even in its own prior, with 0.84–0.86 on the outcomes and 0.84 on the ITE, and so does UWYK∗{}^{*}, with 0.87–0.88 on the outcomes and 0.92 on the ITE. Figure 5 shows the correlation of the predicted pair (Ydo(0)Y_{\mathrm{do}(0)}, Ydo(1)Y_{\mathrm{do}(1)}) that the JT models output on datasets drawn from their training priors: between the 5% and 95% quantiles over datasets, it ranges from about 0 to above 0.9 for Do-PFN JT and UWYK JT, and from 0.01 to 0.30 for CausalPFN-C JT. The correlation is lower for CausalPFN-C JT for two reasons. First, CausalPFN’s prior gives the two outcomes a weaker dependence to learn. In this prior, an outcome is the sum of an expected outcome mtm_t and a noise term ηt\eta_t. In our JT prior, the noise ηt\eta_t is the same for both outcomes of a unit, so it makes them move together. The expected outcomes m0m_0 and m1m_1, however, also vary between units with the same covariates (§ 2.4), and this variation is not the same for the two outcomes. In about four out of five datasets drawn from the prior, it accounts for more than half of an outcome’s variance given xx, so only the remaining part comes from the shared noise. The two outcomes of a unit are therefore only moderately correlated in this prior, and the dependence the model can learn is correspondingly weaker. Second, the grid is coarse for this model. On standardized outcomes, each bin is 0.625 wide, so about half of the training data falls within two bins (Appendix H.§ 5). When both outcomes fall in the same bin, the model cannot tell how they move together, which pulls its predicted correlation down further. Both causes stem from choices made for single-outcome training, which we keep unchanged so that each JT model differs from its baseline only in its learning objective. A prior in which the two worlds share more of their variation, or a grid adapted to joint learning, would let the model learn the cross-world dependence more fully. Designing priors and output distributions for joint learning is therefore a natural direction for future work.

Figure 5. Correlation of the predicted pair (Ydo(0),Ydo(1))(Y_{\mathrm{do}(0)}, Y_{\mathrm{do}(1)}) output by the JT models in their own training prior. Each model is evaluated on 1000 datasets drawn from its training prior; for every query, ρ\rho is the correlation of the predicted joint distribution of (Ydo(0),Ydo(1))(Y_{\mathrm{do}(0)}, Y_{\mathrm{do}(1)}), and each dataset contributes the median over its queries. Bars give the share of datasets; titles give the mean ±\pm sd over datasets.

For the ATE density, Table 4 reports the coverage of its 95% interval on held-out datasets. The 500 datasets of each model’s prior are split into five folds, and the scale ss (Appendix § 14) is set on four of them and evaluated on the fifth. Every model attains close to the nominal level under every ρ\rho, between 94.6% and 95.4%.

Benchmark Construction

RealCause. We use the RealCause datasets [25] directly without any modification. Its generative models fit the outcome distribution under each treatment separately and sample the factual and counterfactual outcomes as independent draws. Table 6 confirms this. PSID (bal.) uses the PSID samples with an observational context that keeps every treated unit and 500 randomly drawn controls, so its query units and realized ITEs are those of PSID. For UWYK Anc, we supply the ancestral matrix in which the treatment is an ancestor of the outcome and every covariate is an ancestor of both, with all other entries left unknown. This matches the structure of RealCause’s generator, which models the treatment given the covariates and the outcome given the covariates and the treatment.

Case study. We take the six causal scenarios of Do-PFN [31], keeping both the causal graphs and the structural equations. Two types of variations are added. First, a constant is added to the treated outcome equation, shifting the mean ITE to −2-2, 00, and +2+2; a zero-mean benchmark rewards a model that predicts no effect. Hence, this variation removes that favouritism. Reported numbers pool the three shifts as one dataset. Second, the covariate dimension is varied over d∈{5,10,20,30,40,50}d \in\{5, 10, 20, 30, 40, 50\} by adding covariates; Table 5 reports the resulting composition per scenario. Both potential outcomes of a unit are generated by propagating the same exogenous draw under do⁡(T=0)\operatorname{do}(T=0) and do⁡(T=1)\operatorname{do}(T=1), so the noise entering YY is shared by both outcomes and cancels in the ITE; the ITE then varies at fixed xx only through hidden variables, noticeably under Unobserved Confounder and negligibly under Backdoor and Frontdoor (Table 6).

ddCobsC_{\mathrm{obs}}CobsC_{\mathrm{obs}}MunobsM_{\mathrm{unobs}}MobsM_{\mathrm{obs}}CobsC_{\mathrm{obs}}MobsM_{\mathrm{obs}}CobsC_{\mathrm{obs}}CunobsC_{\mathrm{unobs}}CobsC_{\mathrm{obs}}MunobsM_{\mathrm{unobs}}
55515325151
10101011055101101
2020202201010202202
3030303301515303303
4040404402020404404
5050505502525505505

Table 5. Composition of the six case studies as we vary the observed covariate count dd. The number of hidden nodes is max⁡(1,⌊d/10⌋)\max(1,\lfloor d/10\rfloor). They are not considered covariates, so ∣X∣=d|X| = d in every cell. A node counts as a confounder (CC) if it is a parent of both TT and YY, and as a mediator (MM) if it lies on a T→⋅→YT \to\cdot\to Y path; subscripts denote observed, unobserved, and hidden nodes.

ComplexMech. ComplexMech is adopted from UWYK [30] and samples a fresh SCM with nonlinear mechanisms for every realization. Its graph size nn is the number of nodes of the SCM. We modify it by binarizing the treatment. A uniformly selected treatment node is binarized by thresholding at the median of its own sample, which keeps the two treatment groups balanced; the threshold is frozen and reused for the interventional passes, and the observational context is re-drawn after binarization. Realizations without a directed path T→YT \to Y are discarded, and, as in the case studies, the two interventional passes share the exogenous draw. The original benchmark evaluates interventional distributions and draws no matched pair of potential outcomes, so the ITE is undefined there; sharing the draw defines it as the SCM counterfactual (Pearl, [28]).

The generator mixes two noise types. In the additive case, Y=σ(f(pa⁡))+εY = \sigma(f(\operatorname{pa})) + \varepsilon, the shared noise of YY cancels when the two outcomes are subtracted; in the non-additive case it enters before the activation and does not cancel. Table 6 gives ρ(x)\rho(x) for the realizations in which it is computable exactly. Because the noise type and the mechanisms are drawn per realization, the correlation differs across realizations.

Twins. Twins (Louizos et al., [21]) contains U.S. twin births from 1989 to 1991, restricted to same-sex pairs with both birth weights below 2 kg: 11,984 pairs. The treatment considered here is being the heavier twin (T=1T = 1) and the outcome is one-year mortality, so under this design both potential outcomes of a pair are observed. We use the dataset as prepared by Bodik (2026), in which one twin of each pair is the factual one, and the other is the counterfactual. We also follow their evaluation: given a pair’s covariates and the outcome of its factual twin, the outcome of the counterfactual twin is predicted, and the prediction is scored by the mean squared error of the predicted mean against the counterfactual twin’s outcome. For UWYK Anc, we supply the ancestral matrix in which the treatment is an ancestor of the outcome and every covariate is an ancestor of both, with all other entries left unknown, as on RealCause. This treats every covariate as a confounder, which might not necessarily reflect the causal structure of Twins. Hence, any performance degradation can be explained with that. It is important to note that we do not know the true causal structure in the dataset, and no ancestral information variant does not suffer from this potential cost.

Dequantization. The models predict continuous outcomes, whereas one-year mortality is binary. We therefore add a uniform jitter on [−w,w][-w, w] to the outcome of each twin, for w∈{0.1,0.25,0.5}w \in\{0.1, 0.25, 0.5\}, and report results averaged over the three widths. Since w≤0.5w \le0.5, thresholding at 1/21/2 recovers the binary outcome. Both the factual outcomes given to the models and the counterfactual outcomes they predict are dequantized. The jitters of the two twins of a pair are coupled through a Gaussian copula, whose correlation is set such that the correlation between the twins’ outcomes, given the covariates, equals that of the binary data. Table 7 shows that the correlation of Table 6 is unchanged by the dequantization.

GroupBinaryd=10d = 10, w=0.1w = 0.1d=10d = 10, 0.25d=10d = 10, 0.5d=1d = 1, w=0.1w = 0.1d=1d = 1, 0.25d=1d = 1, 0.5
X1=0,X2=0X_1 = 0, X_2 = 00.6530.6520.6540.6440.6530.6480.651
X1=0,X2=1X_1 = 0, X_2 = 10.6460.6460.6360.6470.6440.6540.648
X1=1,X2=0X_1 = 1, X_2 = 00.2410.2770.3680.4810.2910.2770.385
X1=0,X3=0X_1 = 0, X_3 = 00.5870.5880.5940.6110.5860.5920.613
X1=0,X3=1X_1 = 0, X_3 = 10.6900.6890.6830.6710.6910.6910.677
X1=1,X3=0X_1 = 1, X_3 = 0–0.0150.4620.3640.1350.2890.350
X2=0,X3=0X_2 = 0, X_3 = 00.5730.5750.5860.6000.5750.5730.598
X2=0,X3=1X_2 = 0, X_3 = 10.7030.7010.7000.6780.7040.7010.691
X2=1,X3=0X_2 = 1, X_3 = 00.6070.6070.6050.6250.6020.6220.638
Mean ± sd0.59±0.150.59 \pm0.150.59±0.140.59 \pm0.140.60±0.100.60 \pm0.100.62±0.060.62 \pm0.060.59±0.130.59 \pm0.130.59±0.140.59 \pm0.140.61±0.100.61 \pm0.10

Table 7. Correlation between the twins’ outcomes, computed following Bodik et al. (2026), within the three most frequent value combinations of each pair of the first three covariates X1X_1, X2X_2, X3X_3, which are binary. Binary: on the original 0/1 mortality outcomes. ww: half-width of the uniform jitter added to each outcome, with the jitter coupled given the first d=10d = 10 or d=1d = 1 covariates. A dash marks a group without deaths, in which the correlation is undefined. The last row is the mean ±\pm sd over the groups in which it is defined.

Range of ρ\rho. A single-outcome model sees only one twin of each pair. The data therefore cannot tell it how the outcomes of the two twins are correlated. However, we can still limit how large or how small this correlation can be. From the observed twins, we estimate the outcome distribution of each twin given the covariates. The two outcomes are most positively correlated when the twins are ordered in the same way under both distributions, and most negatively correlated when they are ordered in opposite ways. These two cases bound the range of ρ\rho compatible with the data. We compute this range for each pair and take the median over the pairs, which gives [−0.4,0.8][-0.4, 0.8], [−0.7,0.9][-0.7, 0.9] and [−0.9,1][-0.9, 1] for w=0.1w = 0.1, 0.250.25 and 0.50.5, at both covariate sets. The baseline is evaluated over this range in steps of 0.10.1.

Interval score. On Twins, we report the mean squared error only. For a binary outcome, a 95% interval can contain only survival, death, or both, and it contains both whenever each has a probability above 2.5%. A model that predicts a 3% chance of death and one that predicts 70% therefore give the same interval and the same interval score, although their predictions differ greatly. The interval score is thus insensitive to how well the probability of the counterfactual twin’s death, given the factual twin, is predicted. Dequantization by large scales might change this, however under the current range it only spreads each outcome over a narrow band around 0 or 1. The interval must therefore still cover both bands whichever probability is predicted, and the interval score cannot show whether a model has used the factual twin. The mean squared error, in contrast, scores the predicted probability itself and therefore rewards using it. Hence, on Twins, the interval score remains uninformative even after dequantization, while the mean squared error is more informative about the models’ performance, and we report it alone.

Additional Results

Throughout this appendix, a model is described as better or worse than another only when the difference is significant (Appendix § 15). Otherwise, the two are said not to differ significantly.

Interval score on RealCause

Finding 4: When the two outcomes are independent, whether joint learning helps depends on the correlation the baseline assumes. RealCause generates the two outcomes independently. A baseline that assumes independence is therefore correct, while the JT models, having learned from their prior that the outcomes are dependent, are misspecified. Against this correct baseline, two of the four JT models have a 14–27% higher interval score, and the other two do not differ significantly. The further the baseline’s assumed correlation moves away from independence, the more misspecified the baseline becomes, and the balance shifts towards joint learning. Under a moderate assumed correlation, ρ=−0.5\rho= -0.5 or 0.50.5, the results are mixed, and when the baseline assumes perfect dependence, every JT model improves the interval score by 62–80%.

Point estimation on the case studies and ComplexMech

Finding 5: When the dependence is present, joint learning significantly improves point estimation for Do-PFN and UWYK, while for CausalPFN the baseline keeps an advantage on the case studies. On average, joint learning significantly reduces the error of the estimated treatment effects by 40% on the case studies and 62% on ComplexMech for Do-PFN, and by 11–51% for the UWYK variants. For CausalPFN, whose baseline already removes the idiosyncratic noise ηt\eta_t, the JT model does not differ significantly from its baseline on ComplexMech, and its error is significantly higher on the case studies, by 8%. The ATE error follows the same pattern for Do-PFN and UWYK, with significant reductions of 15–29% on the case studies and 21–70% on ComplexMech. For CausalPFN, the ATE error of the JT model is significantly higher on the case studies and does not differ significantly on ComplexMech, so its gain from joint learning is limited to the posterior predictive intervals.

Counterfactual prediction on Twins

Figure 9 and Figure 10 show, for each jitter width, the mean squared error of each baseline under every ρ\rho in the range compatible with the data, together with its average over this range and the error of the JT model. The error of the baseline falls as its assumed ρ\rho gets closer to the reported range of 0.5–0.7 (Bodik et al., 2026).

Figure 9. Mean squared error on Twins at d=10d = 10, per jitter width ww (columns) and model family (rows): the baseline under each ρ\rho in the range compatible with the data (blue, ±\pm standard error), its average over the range (dashed), and the JT model (orange, ±\pm standard error).

Figure 10. Mean squared error on Twins at d=1d = 1, as in Figure 9.

The jitter width enters the comparison in two ways. It adds variance to the outcomes, which both models predict and which therefore raises the error of both. It also widens the range of ρ\rho compatible with the data, since the added spread is not tied together by the observed twins, and a wider range adds strongly negative values to the baseline’s average. To ensure that this widening does not drive the result, we repeat the comparison with a single range for all widths, the narrowest one, [−0.4,0.8][-0.4, 0.8], which every width admits. Every JT model remains significantly better than its baseline at both covariate sets, by 7–37%.

ATE density on the benchmarks

Figure 11–Figure 13 report the interval score of the 95% interval of the ATE density (§ 3.2) on RealCause, the case studies and ComplexMech, and Appendix H.§ 5 against the matched-resolution and matched-label baselines. Across the four model families, the four values of ρ\rho and the three benchmarks, the JT models attain a significantly lower interval score in 42 of the 48 comparisons. The exceptions are UWYK Anc on RealCause, where the outcomes are generated independently, so the baseline is correctly specified under ρ≤0.5\rho\le0.5, and CausalPFN-C under ρ=1\rho= 1 on the case studies and ComplexMech, where CausalPFN-C JT still attains a significantly lower interval score than its matched-resolution baseline.

Figure 11. Interval score IS0.05/σY\mathrm{IS}_{0.05}/\sigma_Y (lower is better) of the 95% interval of the ATE density on RealCause across five datasets (PSID-B: PSID balanced). Each row is one model family: its single-outcome model under each ρ\rho (blue) and its JT model with the learned dependence (orange). Bars show the mean over realizations ±\pm the standard error. The last column, Averageg_g, is the geometric mean over datasets ±\pm its standard error, with the lowest value in bold. Significance against the JT model at the 0.1 and 0.05 level is indicated by * and **, respectively. Each panel has its own scale.

Figure 13. Interval score IS0.05/σY\mathrm{IS}_{0.05}/\sigma_Y (lower is better) of the 95% interval of the ATE density on ComplexMech, per graph size nn. Each row is one model family: its single-outcome model under each ρ\rho (blue) and its JT model with the learned dependence (orange). Bars show the mean over realizations ±\pm the standard error. The last column, Averageg\mathrm{Average}_g, is the geometric mean over datasets ±\pm its standard error, with the lowest value in bold. Significance against the JT model at the 0.1 and 0.05 level is indicated by * and **, respectively. Each panel has its own scale.

Figure 7. PEHE\sqrt{\mathrm{PEHE}} (top) and relative ATE error ϵATE\epsilon_{\mathrm{ATE}} (bottom) on the six case studies; mean ±\pm standard error, lower is better. Each row is one model family: its baseline (Base, blue) and its JT model (orange); each panel has its own scale. The last column, Averageg\mathrm{Average}_{g}, is the geometric mean over datasets ±\pm its standard error, with PEHE\sqrt{\mathrm{PEHE}} standardised by each dataset’s outcome scale σY\sigma_Y, and the lowest value in bold. Significance against the JT model at the 0.1 and 0.05 level is indicated by ∗\ast and ∗∗\ast\ast, respectively.

Figure 8. PEHE\sqrt{\mathrm{PEHE}} (top) and relative ATE error ϵATE\epsilon_{\mathrm{ATE}} (bottom) on ComplexMech per graph size; mean ±\pm standard error, lower is better. Each row is one model family: its baseline (Base, blue) and its JT model (orange); each panel has its own scale. The last column, Averageg\mathrm{Average}_{g}, is the geometric mean over datasets ±\pm its standard error, with PEHE\sqrt{\mathrm{PEHE}} standardised by each dataset’s outcome scale σY\sigma_Y, and the lowest value in bold. Significance against the JT model at the 0.1 and 0.05 level is indicated by ∗* and ∗∗**, respectively.

Matched-resolution and matched-label ablations

In Section 3.1 we discuss that joint learning is more efficient due to two reasons: learning the dependence of outcomes and simply learning more by additional labels. The two ablations separate them.

In the matched-resolution version, each baseline is re-trained at its JT counterpart’s grid resolution (K=10K=10 for Do-PFN, K=32K=32 for UWYK and CausalPFN), removing the resolution advantage of the baseline. In the matched-label version, each baseline is trained on both generated outcomes of every unit, receiving the same data information as the JT model without learning the pair together.

Figure 14, Figure 15, and Figure 16 report the calibration with posterior predictive intervals for the ITE and Figure 17–Figure 19 for the ATE, and Figure 20–Figure 22 the point estimation.

Figure 14. Coupling sensitivity of the ablation baselines on RealCause: interval score IS0.05/σY\mathrm{IS}_{0.05}/\sigma_Y (lower is better) of the 95% posterior predictive interval for the ITE. The rows above the dashed line are the matched-resolution ablations of Appendix H.§ 5 (green) and those below are the matched-label ablations (purple), each under every assumed ρ\rho, with its family’s JT model and the learned dependence (orange). Bars show the mean ±\pm standard error, and each panel has its own scale. The last column, Averageg_g, is the geometric mean over datasets ±\pm its standard error, with the lowest value in bold. Significance against the JT model at the 0.1 and 0.05 level is indicated by * and **, respectively.

Figure 15. Coupling sensitivity of the ablation baselines on the six case studies: interval score IS0.05/σYIS_{0.05}/\sigma_Y (lower is better) of the 95% posterior predictive interval for the ITE. The rows above the dashed line are the matched-resolution ablations of Appendix H.§ 5 (green) and those below are the matched-label ablations (purple), each under every assumed ρ\rho, with its family’s JT model and the learned dependence (orange). Bars show the mean ±\pm standard error, and each panel has its own scale. The last column, AveragegAverage_g, is the geometric mean over datasets ±\pm its standard error, with the lowest value in bold. Significance against the JT model at the 0.1 and 0.05 level is indicated by * and **, respectively.

Figure 16. Coupling sensitivity of the ablation baselines on ComplexMech per graph size nn: interval score IS0.05/σY\mathrm{IS}_{0.05}/\sigma_Y (lower is better) of the 95% posterior predictive interval for the ITE. The rows above the dashed line are the matched-resolution ablations of Appendix H.§ 5 (green) and those below are the matched-label ablations (purple), each under every assumed ρ\rho, with its family’s JT model and the learned dependence (orange). Bars show the mean ±\pm standard error, and each panel has its own scale. The last column, Averageg_g, is the geometric mean over datasets ±\pm its standard error, with the lowest value in bold. Significance against the JT model at the 0.1 and 0.05 level is indicated by * and **, respectively.

Figure 17. Interval score IS0.05/σY\mathrm{IS}_{0.05}/\sigma_Y (lower is better) of the 95% interval of the ATE density on RealCause, against the ablation baselines. The rows above the dashed line are the matched-resolution ablations (green) and those below are the matched-label ablations (purple), each under every ρ\rho, with its family’s JT model and the learned dependence (orange). Bars show the mean over realizations ±\pm the standard error. The last column, Averageg_g, is the geometric mean over datasets ±\pm its standard error, with the lowest value in bold. Significance against the JT model at the 0.1 and 0.05 level is indicated by * and **, respectively. Each panel has its own scale.

Figure 19. Interval score IS0.05/σY\mathrm{IS}_{0.05}/\sigma_Y (lower is better) of the 95% interval of the ATE density on ComplexMech per graph size nn, against the ablation baselines. The rows above the dashed line are the matched-resolution ablations (green) and those below are the matched-label ablations (purple), each under every ρ\rho, with its family’s JT model and the learned dependence (orange). Bars show the mean over realizations ±\pm the standard error. The last column, Averageg_g, is the geometric mean over datasets ±\pm its standard error, with the lowest value in bold. Significance against the JT model at the 0.1 and 0.05 level is indicated by * and **, respectively. Each panel has its own scale.

Figure 20. PEHE\sqrt{\mathrm{PEHE}} (top) and relative ATE error ϵATE\epsilon_{\mathrm{ATE}} (bottom) of the matched-resolution (green) and matched-label (purple) ablations and the JT models (orange) on RealCause; mean ±\pm standard error, lower is better; each panel has its own scale. The last column, Averageg_g, is the geometric mean over datasets ±\pm its standard error, with PEHE\sqrt{\mathrm{PEHE}} standardised by each dataset’s outcome scale σY\sigma_Y, and the lowest value in bold. Significance against the JT model at the 0.1 and 0.05 level is indicated by * and **, respectively.

Figure 22. PEHE\sqrt{\mathrm{PEHE}} (top) and relative ATE error εATE\varepsilon_{\mathrm{ATE}} (bottom) of the matched-resolution (green) and matched-label (purple) ablations and the JT models (orange) on ComplexMech per graph size nn; mean ±\pm standard error, lower is better; each panel has its own scale. The last column, Averageg_g, is the geometric mean over datasets ±\pm its standard error, with PEHE\sqrt{\mathrm{PEHE}} standardised by each dataset’s outcome scale σY\sigma_Y, and the lowest value in bold. Significance against the JT model at the 0.1 and 0.05 level is indicated by * and **, respectively.

Finding 6: Resolution limits CausalPFN-C, and learning the two worlds separately does not substitute for learning them together.

On RealCause, where the outcomes are independent, the comparison again depends on the correlation the baselines assume, as in Finding 4. Against their matched-resolution baselines, Do-PFN JT and CausalPFN-C JT are better under three of the four assumed values and do not differ significantly under the fourth. Against the other ablations the results are mixed, and CausalPFN-C JT is worse than its matched-label baseline unless that baseline assumes perfect dependence.

On the synthetic benchmarks, where the outcomes are dependent, every JT model attains a lower interval score than its matched-resolution baseline under every assumed ρ\rho: 3.2–8.3 times lower for Do-PFN, 1.3–2.7 for UWYK and 1.1–1.8 for CausalPFN-C on the case studies, and 3.2–6.4 and 1.2–3.6 times lower for Do-PFN and UWYK on ComplexMech, where CausalPFN-C JT is comparable to its matched-resolution baseline or ahead of it. Giving the baselines the same data as the JT models does not close the gap either. Against their matched-label baselines, the JT models of Do-PFN and UWYK attain a lower interval score under every assumed ρ\rho, 2.2–3.3 and 2.9–5.8 times lower on the case studies, and on ComplexMech, UWYK JT is 1.5–5.0 times lower, while

Figure 12. Interval score IS0.05/σY\mathrm{IS}_{0.05}/\sigma_Y (lower is better) of the 95% interval of the ATE density on the six causal scenarios of the Do-PFN case studies. Each row is one model family: its single-outcome model under each ρ\rho (blue) and its JT model with the learned dependence (orange). Bars show the mean over realizations ±\pm the standard error. The last column, Averageg_g, is the geometric mean over datasets ±\pm its standard error, with the lowest value in bold. Significance against the JT model at the 0.1 and 0.05 level is indicated by * and **, respectively. Each panel has its own scale.

Do-PFN JT is lower only under ρ=1\rho= 1, by 2.0 times. CausalPFN-C JT is 1.3–1.9 times lower on ComplexMech, while on the case studies its matched-label baseline, which retains the fine grid, scores lower. Hence, for both the ITE and the ATE, although the additional data helps, seeing the same number of worlds does not substitute for learning them together.

On Twins, averaged over the range of ρ\rho compatible with the data, every JT model attains a significantly lower mean squared error than both its matched-resolution and its matched-label baseline, at both covariate sets, by 20–61% (Figure 23 and Figure 24).

Figure 23. Mean squared error on Twins at d=10d = 10 against the ablation baselines, as in Figure 9: the matched-resolution (green) and the matched-label (purple) baseline under each ρ\rho in the range compatible with the data, their averages over the range (dashed), and the JT model (orange).

Figure 24. Mean squared error on Twins at d=1d = 1 against the ablation baselines, as in Figure 23.

It is important to highlight the results for CausalPFN-C particularly. The matched-resolution comparison is the most informative for this model, since its baseline benefits most from the fine grid. During training, CausalPFN outputs the response on a fixed grid over [−10,10][-10, 10] with a bin width of ≈0.02\approx0.02 on standardized outcomes. For our jointly trained variant the width grows 32-fold to 0.625, so about half of the training data falls within two bins; for Do-PFN and UWYK, the share of data within two bin widths is dramatically smaller. Once this advantage is removed at matched resolution, CausalPFN-C JT attains a lower interval score under every assumed ρ\rho on the case studies (1.1–1.8 times) and under three of the four on RealCause, and a comparable or lower on ComplexMech. Joint learning therefore improves CausalPFN-C as well; the gain was concealed only by comparing models at different resolutions. Resolution, however, is not a fundamental limitation of joint learning: it can be raised at a higher computational cost or, where that is not feasible, the grid can be adapted to joint learning, for example by narrowing the interior region or changing the standardization of the outcomes. We do not apply such changes here, as our aim is to measure what switching an existing model to joint learning changes, with everything else held fixed.

Figure 20–Figure 22 report the point estimation of the ablations. Against both ablations, the JT models of Do-PFN and UWYK keep lower unit- and population-level errors on the synthetic benchmarks. On the case studies, these errors are lower by 36–72% and 15–72%, respectively. On ComplexMech, they are lower by 20–74% and 17–84%, respectively, with one exception at the unit level. On RealCause, the population-level error is lower, by 16–48%. For CausalPFN-C the picture is mixed. On the case studies, its population-level error is 63% and 26% higher than that of the matched-label and matched-resolution baselines, respectively. Its unit-level error is 5% higher than that of the matched-label baseline and 13% lower than that of the matched-resolution baseline. On RealCause and ComplexMech, it does not differ significantly from either ablation.

Figure 18. Interval score IS0.05/σYIS_{0.05}/\sigma_Y (lower is better) of the 95% interval of the ATE density on the six case studies, against the ablation baselines. The rows above the dashed line are the matched-resolution ablations (green) and those below are the matched-label ablations (purple), each under every ρ\rho, with its family’s JT model and the learned dependence (orange). Bars show the mean over realizations ±\pm the standard error. The last column, AverageqAverage_q, is the geometric mean over datasets ±\pm its standard error, with the lowest value in bold. Significance against the JT model at the 0.1 and 0.05 level is indicated by * and **, respectively. Each panel has its own scale.

Figure 21. PEHE\sqrt{\mathrm{PEHE}} (top) and relative ATE error ϵATE\epsilon_{\mathrm{ATE}} (bottom) of the matched-resolution (green) and matched-label (purple) ablations and the JT models (orange) on the six case studies; mean ±\pm standard error, lower is better; each panel has its own scale. The last column, Averageg\mathrm{Average}_{g}, is the geometric mean over datasets ±\pm its standard error, with PEHE\sqrt{\mathrm{PEHE}} standardised by each dataset’s outcome scale σY\sigma_Y, and the lowest value in bold. Significance against the JT model at the 0.1 and 0.05 level is indicated by * and **, respectively.

Robustness to the cross-world correlation ρ(x)\rho(x)

In our results, we evaluate the models under various correlation structures: a strongly correlated one on the case studies, one that varies widely across datasets on ComplexMech, and a moderate one on Twins. In this section, we test them further in a controlled setting, where the correlation can be computed directly. The four case studies with pre-treatment covariates, IHDP, ACIC and ComplexMech with five nodes allow this, so we generate them with the outcome noise coupled across the two outcomes at a chosen level and group the datasets by their ρ\rho (Table 8). The single-outcome baselines are reported under each assumed ρ∈{−0.5,0,0.5,1}\rho\in\{-0.5, 0, 0.5, 1\}, so in every bin of the true ρ\rho, at least one assumed value lies within the bin.

ModelAt inferenceTrue ρ\rhoAverage
ModelAt inference[−0.5,0)[-0.5, 0)[0,0.5)[0, 0.5)[0.5,1][0.5, 1]Average
Do-PFNρ=−0.5\rho= -0.571%100%100%90%
Do-PFNρ=0\rho= 057%71%71%67%
Do-PFNρ=0.5\rho= 0.571%71%71%71%
Do-PFNρ=1\rho= 1100%71%71%81%
Do-PFNAverage75%79%79%77%
UWYK No-Ancρ=−0.5\rho= -0.586%100%100%95%
UWYK No-Ancρ=0\rho= 071%86%86%81%
UWYK No-Ancρ=0.5\rho= 0.586%71%71%76%
UWYK No-Ancρ=1\rho= 1100%86%71%86%
UWYK No-AncAverage86%86%82%85%
UWYK Ancρ=−0.5\rho= -0.586%86%100%90%
UWYK Ancρ=0\rho= 071%71%86%76%
UWYK Ancρ=0.5\rho= 0.543%43%29%38%
UWYK Ancρ=1\rho= 1100%86%71%86%
UWYK AncAverage75%71%71%73%
CausalPFN-Cρ=−0.5\rho= -0.514%14%14%14%
CausalPFN-Cρ=0\rho= 014%14%14%14%
CausalPFN-Cρ=0.5\rho= 0.514%14%14%14%
CausalPFN-Cρ=1\rho= 186%57%43%62%
CausalPFN-CAverage32%25%21%26%
CausalPFN-C (resolution)ρ=−0.5\rho= -0.586%100%100%95%
CausalPFN-C (resolution)ρ=0\rho= 071%86%100%86%
CausalPFN-C (resolution)ρ=0.5\rho= 0.529%0%43%24%
CausalPFN-C (resolution)ρ=1\rho= 114%14%14%14%
CausalPFN-C (resolution)Average50%50%64%55%
Benchmarks7777

Table 8. Percentage of benchmarks on which the JT model attains a lower interval score IS0.05\mathrm{IS}_{0.05} of the 95% posterior predictive interval for the ITE than its single-outcome baseline, on seven benchmarks with a controlled cross-world correlation: the four case studies with pre-treatment covariates, IHDP, ACIC and ComplexMech (n=5n = 5). Each column groups the datasets by their true ρ\rho, and each row gives the ρ\rho used by the baseline at inference. The Average row and column average these percentages, and the last row gives the number of benchmarks. CausalPFN-C (resolution) compares CausalPFN-C JT with its matched-resolution baseline.

Finding 7: The advantage of joint learning on Do-PFN and UWYK holds across the tested range of dependence, while CausalPFN-C is about even with its baseline at matched resolution. Averaged over the true ρ\rho and the ρ\rho used at inference, the JT models attain a lower interval score on 77% of the benchmarks for Do-PFN, and on 85% and 73% for UWYK No-Anc and UWYK Anc (Table 8). CausalPFN-C JT does so on 26%, and on 55% against its matched-resolution baseline.

CausalPFN two-stage comparison

The published CausalPFN is trained in two stages: the first predicts observational outcomes on real datasets, and the second carries those weights over and predicts interventional outcomes on the synthetic prior. The first stage does not transfer to the joint setting: its weights do not match the bin structure of the joint output, and its real datasets contain no counterfactual outcomes to pair. We therefore apply joint learning to the second, causal stage and re-train the baseline from the same stage under the same protocol; the main results compare against this re-trained baseline, CausalPFN-C. Figure 25 compares CausalPFN-C JT against the published two-stage checkpoint instead, in point estimation and in calibration, with both evaluated against the ITE (Appendix § 15).

Figure 25. Averageg_g of PEHE\sqrt{\mathrm{PEHE}}, ϵATE\epsilon_{\mathrm{ATE}} and the interval score IS0.05/σY\mathrm{IS}_{0.05}/\sigma_Y (lower is better) of CausalPFN-C JT (orange) against the published two-stage CausalPFN checkpoint (blue), instead of the re-trained CausalPFN-C of the main results, on RealCause, the case studies and ComplexMech (rows). The interval score shows the baseline under each assumed ρ\rho and the JT model with its learned dependence. Geometric mean over datasets ±\pm its standard error, with PEHE\sqrt{\mathrm{PEHE}} standardised by each dataset’s outcome scale σY\sigma_Y. The lowest value of each panel is in bold. Significance against the JT model at the 0.1 and 0.05 level is indicated by * and **, respectively.

Finding 8: Against the published CausalPFN, joint learning trades point estimation for better intervals. The published two-stage checkpoint has a clear advantage in point estimation. It is better in both metrics on the case studies and in the error of the estimated treatment effects on RealCause, while the two are comparable on ComplexMech. The posterior predictive intervals, however, favour joint learning. On the case studies, CausalPFN-C JT attains a lower interval score than the published checkpoint under every assumed ρ\rho (1.96 against 2.14–3.44). On RealCause it attains a lower interval score under three of the four assumed values of ρ\rho, and on ComplexMech the two are comparable (6.22 against 6.10–15.26).

UWYK binary re-training

UWYK provides two training options, binary and continuous treatment prediction. The published checkpoint covers only the continuous version, so for fairness we also re-train the binary version. Figure 26 compares the JT model against the re-trained binary version, UWYK-B, for both the ancestral and no-ancestral variants.

Figure 26. Averageg_g of PEHE\sqrt{\mathrm{PEHE}}, ϵATE\epsilon_{\mathrm{ATE}} and the interval score IS0.05/σY\mathrm{IS}_{0.05}/\sigma_Y (lower is better) of UWYK JT (orange) against the re-trained binary-treatment baseline UWYK-B (blue), on RealCause, the case studies and ComplexMech, for the no-ancestral variant (No-Anc, above the dashed line) and the ancestral variant (Anc, below it). The interval score shows the baseline under each assumed ρ\rho and the JT model with its learned dependence. Geometric mean over datasets ±\pm its standard error, with PEHE\sqrt{\mathrm{PEHE}} standardised by each dataset’s outcome scale σY\sigma_Y. The lowest value of each panel is in bold. Significance against the JT model at the 0.1 and 0.05 level is indicated by * and **, respectively.

Finding 9: Re-training UWYK on binary treatments does not change the main results. In point estimation, UWYK JT improves upon UWYK-B on the synthetic benchmarks, for both variants: by 25–52% and 29–45% on the case studies, and 21–30% and 19–32% on ComplexMech, for the no-ancestral and ancestral variants respectively. On RealCause, only the no-ancestral variant improves, by 14–39%. The posterior predictive intervals follow the same pattern as against the published checkpoint. On the synthetic benchmarks, where the outcomes are dependent, UWYK JT

attains a lower interval score than UWYK-B under every assumed ρ\rho, for both variants: 2.8–4.4 and 2.4–2.9 times lower on the case studies, and 1.9–4.6 and 1.5–3.5 times lower on ComplexMech. On RealCause, where the outcomes are independent, UWYK-B attains a lower interval score under most assumed values of ρ\rho for the ancestral variant (three of four), consistent with Finding 4.

Computational Details

Table 9 reports the inference time per dataset. Each JT model needs roughly half the forward time of its single-outcome baseline, as one pass yields the joint while the baselines require one pass per treatment arm. The ITE and ATE densities are post-processing on the CPU; the ATE barycenter adds about 0.01 s.

ModelVariantGridOutput dimTotal paramsThroughput (s/step)StepsGPU-hoursForward (s)ATE density (s)
Do-PFNPublished1001007,336,721—262,144—0.2750.010
Do-PFNJT10210^2102+13=11310^2+13=1137,346,7180.223 (L40S)262,14416.20.1360.010
Do-PFNMatched resolution10107,267,2410.216 (L40S)262,14415.70.2740.010
Do-PFNMatched labels1001007,336,7210.284 (L40S)262,14420.60.2730.009
UWYKContinuous (published)10001000+4=10041000+4=100411,186,463—50,000—0.1260.010
UWYKBinary (retrained)10001000+4=10041000+4=100411,186,4631.63 (H100)50,00022.70.1270.010
UWYKJT32232^2322+13=103732^2+13=103714,010,9451.90 (H100)50,00026.40.0730.010
UWYKMatched resolution3232+4=3632+4=3610,937,6871.71 (Blackwell)50,00023.80.1260.010
UWYKMatched labels10001000+4=10041000+4=100411,186,4631.69 (Blackwell)50,00023.50.1270.010
CausalPFNTwo-stage (published)10241024+10=10341024+10=103418,844,034———0.0320.009
CausalPFNStage-2 (reproduction)10241024+10=10341024+10=103418,844,0342.57 (H100)50,00035.70.0320.009
CausalPFNJT32232^2322+13+10=104732^2+13+10=104718,852,0021.96 (H100)50,00027.20.0160.009
CausalPFNMatched resolution3232+10=4232+10=4218,080,1941.90 (H100)50,00026.40.0310.009
CausalPFNMatched labels10241024+10=10341024+10=103418,844,0342.60 (H100)50,00036.10.0320.009

Table 9. Training cost per baseline, ablation, and jointly trained versions. Forward and ATE-density inference time per dataset on RealCause IHDP (realization 0; 672 context units, 75 queries), median over 20 timed forward runs after 3 warm-up runs on a single NVIDIA L40S, and over 5 runs of the ATE-density step. Output dimension counts the outputs per query: the grid logits plus the models’ auxiliary outputs, such as tail and region parameters (+13 for the JT models, Appendix B; +0, +4, and +10 for the Do-PFN, UWYK, and CausalPFN single-outcome models). Parameter counts are deduplicated tensor counts of the checkpoints. Published models are not trained by us, so their training columns are not applicable.

Each model was trained on a single GPU of one of three types: NVIDIA L40S (48 GB) for the Do-PFN models, NVIDIA H100 (80 GB, SXM5) for the CausalPFN models and the UWYK JT and binary models, and NVIDIA RTX PRO 6000 Blackwell Server Edition (96 GB) for the UWYK matched-resolution and matched-label models. Table 9 reports the hardware, throughput, and cost of each run.

For evaluation, inference was done using a mix of single NVIDIA L40S GPUs (48 GB) per task, or Intel Granite Rapids Xeon CPUs (8 cores, 64GB at a time).

The implementation uses Python 3.11, PyTorch 2.11.0, NumPy 1.26.4, SciPy 1.15.2, scikit-learn 1.2.2, and XGBoost 3.2.0, the last of which enters results through the synthetic-prior mechanism sampler.

References

  1. [1]Martial Agueh and Guillaume Carlier. Barycenters in the wasserstein space. SIAM Journal on Mathematical Analysis, 43(2):904–924, 2011. doi: 10.1137/100805741.DOI
  2. [2]Ahmed M. Alaa, Michael Weisz, and Mihaela van der Schaar. Deep counterfactual networks with propensity-dropout. arXiv preprint arXiv:1706.05966, 2017.arxiv.org/abs/1706.05966
  3. [3]Ryan M. Andrews and Vanessa Didelez. Insights into the cross-world independence assumption of causal mediation analysis. Epidemiology, 32(2):209–219, 2021.DOI
  4. [4]Susan Athey and Guido W. Imbens. The state of applied econometrics: Causality and policy evaluation. Journal of Economic Perspectives, 31(2):3–32, May 2017. doi: 10.1257/jep.31.2.3.DOI
  5. [5]Vahid Balazadeh, Hamidreza Kamkari, Valentin Thomas, Junwei Ma, Bingru Li, Jesse C. Cresswell, and Rahul Krishnan. CausalPFN: Amortized causal effect estimation via in-context learning. In The Thirty-ninth Annual Conference on Neural Information Processing Systems, 2025.arxiv.org/abs/2506.07918
  6. [8]Victor Chernozhukov, Denis Chetverikov, Mert Demirer, Esther Duflo, Christian Hansen, Whitney Newey, and James Robins. Double/debiased machine learning for treatment and structural parameters. The Econometrics Journal, 21(1):C1–C68, 02 2018. ISSN 1368-4221.
  7. [10]Peng Ding and Tirthankar Dasgupta. A potential tale of two-by-two tables from completely randomized experiments. Journal of the American Statistical Association, 111(513):157–168, 2016.arxiv.org/abs/1501.02389
  8. [11]Yanqin Fan and Sang Soo Park. Sharp bounds on the distribution of treatment effects and their statistical inference. Econometric Theory, 26(3):931–951, 2010. ISSN 02664666, 14694360.DOI
  9. [12]Tilmann Gneiting and Adrian E Raftery. Strictly proper scoring rules, prediction, and estimation. Journal of the American Statistical Association, 102(477):359–378, 2007. doi: 10.1198/016214506000001437.DOI
  10. [13]James J. Heckman, Jeffrey Smith, and Nancy Clements. Making the most out of programme evaluations and social experiments: Accounting for heterogeneity in programme impacts. The Review of Economic Studies, 64(4):487–535, 1997. ISSN 00346527, 1467937X.DOI
  11. [14]Miguel A. Hernán and James M. Robins. Causal Inference: What If. Chapman & Hall/CRC, Boca Raton, 2020.
  12. [15]Paul W. Holland. Statistics and causal inference. Journal of the American Statistical Association, 81 (396):945–960, 1986. ISSN 01621459, 1537274X.
  13. [16]Noah Hollmann, Samuel Müller, Katharina Eggensperger, and Frank Hutter. TabPFN: A transformer that solves small tabular classification problems in a second. In The Eleventh International Conference on Learning Representations, 2023.arxiv.org/abs/2207.01848
  14. [17]Noah Hollmann, Samuel Müller, Lennart Prücker, Arjun Krishnakumar, Max Körfer, Shi Bin Hoo, Robin Tibor Schirrmeister, and Frank Hutter. Accurate predictions on small data with a tabular foundation model. Nature, 637(8045):319–326, 2025.DOI
  15. [18]Guido W. Imbens and Donald B. Rubin. Causal Inference for Statistics, Social, and Biomedical Sciences: An Introduction. Cambridge University Press, 2015.
  16. [19]Edward H. Kennedy. Towards optimal doubly robust estimation of heterogeneous causal effects. Electronic Journal of Statistics, 17(2):3008 – 3049, 2023. doi: 10.1214/23-EJS2157.DOI
  17. [20]Sören R. Künzel, Jasjeet S. Sekhon, Peter J. Bickel, and Bin Yu. Metalearners for estimating heterogeneous treatment effects using machine learning. Proceedings of the National Academy of Sciences, 116(10):4156–4165, 2019.DOI
  18. [21]Christos Louizos, Uri Shalit, Joris Mooij, David Sontag, Richard Zemel, and Max Welling. Causal effect inference with deep latent-variable models. arXiv preprint arXiv:1705.08821, 2017.arxiv.org/abs/1705.08821
  19. [22]Yuchen Ma, Dennis Frauen, Emil Javurek, and Stefan Feuerriegel. Foundation models for causal inference via prior-data fitted networks. In The Fourteenth International Conference on Learning Representations, 2026.arxiv.org/abs/2506.10914
  20. [23]Geoffrey J. McLachlan, Sharon X. Lee, and Suren I. Rathnayake. Finite mixture models. Annual Review of Statistics and Its Application, 6:355–378, 2019. doi: 10.1146/annurev-statistics-031017-100325.
  21. [24]Samuel Müller, Noah Hollmann, Sebastian Pineda Arango, Josif Grabocka, and Frank Hutter. Transformers can do bayesian inference. In International Conference on Learning Representations, 2022.arxiv.org/abs/2112.10510
  22. [25]Brady Neal, Chin-Wei Huang, and Sunand Raghupathi. Realcause: Realistic causal inference benchmarking, 2021.arxiv.org/abs/2011.15007
  23. [26]Xinkun Nie and Stefan Wager. Quasi-oracle estimation of heterogeneous treatment effects. arXiv preprint arXiv:1712.04912, 2020.arxiv.org/abs/1712.04912
  24. [27]Emanuel Parzen. On estimation of a probability density function and mode. The Annals of Mathematical Statistics, 33(3):1065–1076, September 1962. doi: 10.1214/aoms/1177704472.DOI
  25. [28]Judea Pearl. Causality: Models, Reasoning and Inference. Cambridge University Press, Cambridge, 2 edition, 2009.
  26. [29]Jonas Peters, Dominik Janzing, and Bernhard Schölkopf. Elements of Causal Inference: Foundations and Learning Algorithms. Adaptive Computation and Machine Learning. MIT Press, Cambridge, MA, 2017. ISBN 978-0-262-03731-0.
  27. [30]Arik Reuter, Anish Dhir, Cristiana Diaconu, Jake Robertson, Ole Ossen, Frank Hutter, Adrian Weller, Mark van der Wilk, and Bernhard Schölkopf. Use what you know: Causal foundation models with partial graphs. In Forty-third International Conference on Machine Learning, 2026.arxiv.org/abs/2602.14972
  28. [31]Jake Robertson, Arik Reuter, Siyuan Guo, Noah Hollmann, Frank Hutter, and Bernhard Schölkopf. Do-PFN: In-context learning for causal effect estimation. In The Thirty-ninth Annual Conference on Neural Information Processing Systems, 2025.arxiv.org/abs/2506.06039
  29. [32]Donald B. Rubin. Estimating causal effects of treatments in randomized and nonrandomized studies. Journal of Educational Psychology, 66(5):688–701, 1974. doi: 10.1037/h0037350.DOI
  30. [33]Uri Shalit, Fredrik D. Johansson, and David Sontag. Estimating individual treatment effect: generalization bounds and algorithms. arXiv preprint arXiv:1606.03976, 2017.arxiv.org/abs/1606.03976
  31. [34]Peter Spirtes, Clark Glymour, and Richard Scheines. Causation, Prediction, and Search. The MIT Press, 01 2001. ISBN 9780262284158.
  32. [35]Christopher Stith, Hossein Rahmani, and Jesse C. Cresswell. Causal foundation models. arXiv preprint arXiv:2609.03003, 2026.arxiv.org/abs/2609.03003
  33. [36]Dennis Thumm, Arik Reuter, Jake Robertson, Shi Bin Hoo, Adrian Weller, Frank Hutter, Ying Chen, and Bernhard Schölkopf. Causal foundation models for time series based on prior-data fitted networks. In 2nd ICML Workshop on Foundation Models for Structured Data, 2026.

Paper details

vlnc:W_yax4uqfa
Version
v1 · latest
Published
Fresh — added to the Hub within the last 30 days
Added
License
CC BY 4.0
Usage

views · 0

downloads · 0

A view counts once per browser per day, a download each time the PDF is requested.

  • Mathematics

Claude Opus 5.5

View statement
Looking for related papers…
Contents