betaFeedback

WorksComputer Science

Inversion-Free High-Dimensional Counterfactual Inference

Abstract

Counterfactual inference is the task of answering hypothetical "what-if" queries about a unit, given an observational dataset, modeling assumptions, and a causal graph. This enables a modeler to reason about hypothetical scenarios that cannot otherwise be answered without assumptions or a causal graph. The simpler case involves a causal graph with scalar 1-D variables that denote different attributes in the data-generation process. While the literature is clear in the 1-D case, extending this to settings with high-dimensional nodes (e.g., images) remains challenging. Conventionally, in the literature, methods recover the latent state from the factual image (inversion), generate counterfactual attributes from the graph, and render the counterfactual image from the latent state using the updated attributes. In our work, we investigate the necessity of the inversion step in computing high-dimensional counterfactuals. We first discuss moving from the scalar 1-D case to the higher-dimensional case. Secondly, we discuss the relevance of the inversion step and evaluate an inversion-free counterpart repurposed from the image-editing literature. Finally, on image datasets such as CelebA and ADNI, we benchmark our approach using two measures: namely, effectiveness and identity preservation. While effectiveness is a measure of how faithfully the counterfactual attribute is rendered, identity preservation measures whether the same unit (image) is being rendered, both of which are crucial for high-dimensional counterfactual inference. We present the tradeoffs of using inversion-based and inversion-free approaches in our work.

Introduction

Answering hypothetical questions about individual cases are common in data analysis and hard to answer without assumptions. Would this patient’s chances of survival looked different had the disease progressed further? This is a counterfactual query, and Pearl’s abduction–action–prediction procedure [1] answers the query when the variables concerned are scalars, under some assumptions that are by now well understood. However, most of the variables of interest are not single-dimensional scalars. A brain scan or a photograph carries several attributes at once, many of them causally related, and answering a counterfactual query about such a record means producing the record itself rather than a number. This has become feasible, with generative models capable of rendering an image conditioned on a requested set of attributes.

The pipelines that answer such queries share a common structure. A causal model over the attributes, a generative backbone that renders the image, and between them sits a step that recovers a latent state from the factual image, intended to make the result depict the same unit rather than a fresh image carrying the requested attributes. This state is taken from a learned encoder [2, 3] or from running the backbone backward to a noise map [4, 5, 6], and the step is also referred to as “abduction”, taking its name after the procedure of inverting a structural equation the modeler has fitted in the standard 1-D case. The name is warranted when the image mechanism is trained as part of the causal model, and identification in high dimensions has been established on that basis [7], under conditions that are demanding in practice. We investigate the role of this inversion step in computing counterfactuals in the higher dimensional case.

Our contributions are as follows:

  • Analysing the issues in moving from 1-D counterfactual inference to higher dimensional counterfactual inference, discussing approaches in the literature and identification arguments involved to tackle the higher dimensional case.

  • Presenting the role of inversion step, conventionally used by most works in computing higher-dimensional counterfactuals, and proposing an inversion-free pipeline that does not generate the latent during the edit.

  • Testing the inversion-based approaches against our proposed inversion-free approach on image datasets like CelebA, a corpus of celebrity images and ADNI, consisting of brain MRI scans. We provide tradeoffs on two measures namely effectiveness and identity preservation.

We classify works on counterfactual inference into three categories: starting with works relevant to traditional counterfactual inference, literature dealing with using deep networks for the task and the body of work relevant to counterfactual queries over images.

a) Counterfactual Inference: Counterfactual queries are posed within the Structural Causal Model framework [1], to answer a query about a specific unit through the abduction–action–prediction procedure and, with the axiomatic characterization of Galles and Pearl [8]. The difficulty lies in abduction, and it is resolved by restricting the functional form of the mechanism, a restriction that the literature has progressively weakened. Additive noise models fix how the exogenous term enters, so that it is recovered as a residual while the dependence on the parents remains nonlinear [9]. Post-nonlinear models relax additivity itself by composing the additive form with an invertible output nonlinearity, which is inverted before the residual is taken [10]. Nasr-Esfahany et al. [11] unify these under bijective generation mechanisms and establish counterfactual identifiability for three causal structures, in the Markovian case relevant here, strict monotonicity in the exogenous argument. The other two cases admit unobserved confounding, identifying the mechanism through an instrumental variable or a backdoor set in place of the Markovian assumption. Recently another approach by Von Kuglegen et al. [12] popularised backtracking counterfactuals. This paradigm allows for the same causal graph while computing the counterfactual, breaking away from the traditional interventionist account.

The two transports, inversion and inversion-free

Figure 1. The two transports, both given the same counterfactual attributes obtained by propagating the antecedent on age through the structural equations. Left: inversion carries the factual to noise under the source condition and regenerates under the target. Right: the inversion-free map moves the factual directly, by accumulated velocity differences between the two conditions, always staying on the image manifold.

b) Deep Networks for Counterfactual Inference: Scaling causal inference to high-dimensional data has led to the development of Deep Structural Causal Models (DSCMs) [2], which leverage normalizing flows and VAEs to parameterize exogenous noise. Neural Causal Models [13] further extend this by providing frameworks for identification and estimation in complex, non-linear environments. Recent research has specifically addressed the problem of Deep Backtracking [14], proposing methods that ensure counterfactual explanations remain ”causally compliant” by allowing background variables to be updated. Other developments include handling categorical background variables [15] and ensuring end-to-end consistency for counterfactual fairness [16], ensuring that deep estimators do not introduce spurious correlations during the abduction phase.

c) Image Editing: In the visual domain, counterfactual editing seeks to modify specific image attributes while preserving the underlying causal structure. Early generative approaches often conflated causal dynamics with image rendering, but recent work on Augmented SCMs (ASCM) [17] has formalized the separation of mixing functions from causal mechanisms. High-fidelity image counterfactuals have been achieved using probabilistic causal models [3], which utilize latent space abduction to ensure realistic reconstructions. The emergence of diffusion-based models has further pushed the state-of-the-art. Diffusion Causal Models [4] allow for fine-grained counterfactual estimation by treating the reverse diffusion process as a causal mechanism. More recently, semantic abduction techniques [5] have improved the alignment between textual prompts and causal interventions, enabling diffusion models to generate images that are not only visually convincing but also causally consistent with complex counterfactual queries. A parallel line in the vision literature avoids the round trip altogether, editing along the generative path [18, 19]. These methods carry no causal model over the attributes, but supply the transports compared here against inversion.

Preliminaries

Lowercase letters denote scalars, lowercase boldface vectors, uppercase boldface matrices, and calligraphic letters sets and tuples that are not random variables. Plain uppercase letters denote random variables and their lowercase counterparts realized values. We write P(⋅)P(\cdot) for probability distributions and do⁡(⋅)\operatorname{do}(\cdot) for Pearl’s intervention operator. The remainder of this section focuses on structural causal models, counterfactuals, the notion of counterfactual identifiability and some basics of flow-based generative models that we later use.

a) Structural Causal Models: A Structural Causal Model (SCM) is a tuple M=⟨U,V,F,P(U)⟩\mathcal{M}=\langle U,V,\mathcal{F},P(U)\rangle, where UU is a set of exogenous noise variables with prior P(U)P(U), VV is a set of endogenous variables, and F={fi}\mathcal{F}=\{f_i\} is a set of structural equations vi=fi(pa⁡i,ui)v_i=f_i(\operatorname{pa}_i,u_i) mapping parents and noise to each variable [1]. Each SCM induces a causal diagram G\mathcal{G} in which directed edges encode direct causal influence and bidirected edges encode shared exogenous noise. An intervention do⁡(vi=v∗)\operatorname{do}(v_i=v^*) replaces fif_i with the constant v∗v^*, yielding a submodel Mvi=v∗\mathcal{M}_{v_i=v^*} used for interventional and counterfactual queries.

b) Counterfactuals: Counterfactuals reason about a specific unit while conditioning on what was actually observed. The hypothesised condition Vi=v∗V_i=v^* is the antecedent. Counterfactuals are obtained as follows:

  1. Abduction. Update the noise to the posterior consistent with the evidence, P(U∣V=v)P(U\mid V=v). When M\mathcal{M} is bijective in its noise, this posterior collapses to the point u=F−1(v)u=F^{-1}(v).

  2. Action. Apply do⁡(Vi=v∗)\operatorname{do}(V_i=v^*) to obtain the submodel Mvi=v∗\mathcal{M}_{v_i=v^*}.

  3. Prediction. Evaluate the queried variable in Mvi=v∗\mathcal{M}_{v_i=v^*} using the abducted noise uu.

c) Counterfactual Identifiability: Two SCMs can agree on the observational distribution and still disagree on a counterfactual, since matching the conditional law of a variable given its parents leaves free how exogenous values are paired with outcomes (Figure 3). A counterfactual query is identifiable when no such disagreement is possible, that is, when every SCM consistent with the observational distribution assigns it the same value [1]. Restricting the class restores this: with each fif_i bijective in its exogenous argument abduction returns a point, and under a Markovian diagram with fif_i strictly monotone the counterfactual is identified [11].

Abduction at fixed parents

Figure 3. Abduction at fixed parents. Solid curves are fv(a,b,⋅)f_v(a,b,\cdot) at the observed brain volume bb, dashed curves the same mechanism under do⁡(B=b∗)\operatorname{do}(B=b^*). In (a) the observed vv has a single preimage u^v\hat{u}_v, and the counterfactual is read off the dashed curve at that point. In (b) the mechanism attains vv at two exogenous values, and the two preimages give different counterfactuals.

d) Flow-based generative models: Flow-based generative models learn a distribution mapping samples from a noise distribution to a target image distribution [20]. Flow matching learns to transport noise to data along a prescribed interpolation. Rectified flow [21] prescribes the straight-line interpolation

zt=(1−t)x+tϵ,ϵ∼N(0,I),(1)z_t=(1-t)x+t\epsilon,\qquad\epsilon\sim\mathcal{N}(0,I), \tag*{(1)}

so that t=0t=0 is the image and t=1t=1 is noise, a network V(z,t,c)V(z,t,c) is trained to predict the velocity ϵ−x\epsilon-x of this path from ztz_t alone, given a condition cc. Generation integrates z˙t=V(zt,t,c)\dot{z}_t=V(z_t,t,c) from t=1t=1 to t=0t=0 starting at a noise draw, running the same field from t=0t=0 upward instead carries an image to the noise state that would have produced it. This

second direction is the inversion step, and the round trip invert under csrcc_{\mathrm{src}}, then regenerate under ctarc_{\mathrm{tar}} is the mechanism by which flow and diffusion backbones implement abduction and prediction steps at the image level.

Understanding Counterfactual Inference in High-Dimensional Settings

We first consider counterfactual identification in the scalar case, where the assumptions are well understood and abduction reduces to inverting the structural equation. Introducing an image node, we then examine what assumptions are required for identification in higher dimensions, and how it is approached in practice. We close by reviewing how abduction is implemented in the current literature, and by separating two notions the scalar case allows to coincide: abduction and inversion.

Counterfactual Identification in the Scalar 1-D Case

We introduce a running example based on structural brain imaging, to which we return to in Section VI. Genotype, sex and age are roots; brain volume is their common child; ventricular volume responds negatively to brain volume, since the two partition a fixed intracranial space, and to age directly (Figure 2). The exogenous terms absorb the subject-specific variation the observed attributes leave unexplained, and are what abduction must recover.

Figure 2. Attribute-level diagram and structural equations used as a running example, following Melistas et al. [22]: ApoE-ϵ4\epsilon4 status apap, sex ss, age aa, brain volume bb and ventricular volume vv. The image node is introduced in Section IV-B.

Consider a subject with observed attributes and the antecedent do(B=b∗B=b^*). Genotype, sex and age are non-descendants of BB and retain their observed values, so the query reduces to re-evaluating fvf_v at (a,b∗,uv)(a,b^*,u_v). Everything therefore depends on recovering uvu_v from the factual, and the data alone does not determine this. The observational distribution constrains fvf_v only through the conditional law of VV given its parents. Figure 3 shows two mechanisms that agree on that law and disagree on the counterfactual, because they pair exogenous values with outcomes differently.

Imposing restrictions on the causal models by assuming their functional forms, we can remove the indeterminacy that leads to counterfactual non-identification. If the mechanism is strictly monotone in its exogenous argument at every configuration of the parents, the map uv↦vu_v \mapsto v is invertible, the posterior collapses to the point uv=fv−1(v;a,b)u_v=f_v^{-1}(v;a,b), and the counterfactual is point-identified [11]. Along with monotonicity, two more conditions are assumed in this setting. The diagram is known, and the model is Markovian, so the exogenous variables are mutually independent.

Counterfactual Inference over a Higher Dimensional Node

We now consider a brain MRI scan that is rendered as a function of brain and ventricular volume at once. The causal mechanism is therefore separated from the rendering mechanism, also advocated by Pan and Bareinboim [17], shown in Figure 4. We take gg to be invertible with respect to the attributes, so a labelling map ℓ\ell recovers them from any image in the support, which is what makes the two volumes readable off a scan at all.

The two-stage model

Figure 4. The two-stage model. The attribute mechanisms and their exogenous variables are those of Fig. 2. All five attributes pa=(ap,s,a,b,v)\mathbf{pa}=(ap,s,a,b,v) are rendered into the image by gg (grey edges), together with a high-dimensional exogenous state uxu_x carrying everything the diagram does not name. The labelling map ℓ\ell recovers the attributes from the image.

Abduction step can now be split in two phases. Recovering the attribute-level noise uap,…,uvu_{ap},\ldots,u_v is the scalar problem of Section IV-A, unchanged. Though recovering uxu_x is different in nature. It carries everything not specified in the attribute graph, the slice depth, the cortical folding pattern, the scanner and its intensity profile, and it is high-dimensional. No mechanism has been specified for it, so there is no fitted function to invert in the sense of Section IV-A. Also monotonicity does not transfer easily: it is a condition on a scalar argument, whereas gg takes a latent state of many thousands of dimensions. In practice uxu_x is whatever a generative backbone returns when run in reverse.

Abduction v.s. Inversion

A counterpart to monotonicity does exist in the higher dimensional case making identification possible in higher dimensions. But the condition must be imposed while the model is trained. De Sousa Ribeiro et al. [7] require the image mechanism to be the map induced by a dynamic optimal transport flow, which generalises monotonicity to d>1d>1 and identifies the counterfactual. Nothing applied at inference time can supply this to a backbone pretrained for generation. Table I places the literature against that standard. Each method factorises the image noise differently, and abduction returns whatever that factorisation names: an encoder latent and a reparameterisation residual for the variational mechanisms [22], a single latent for the adversarial one, and for Rasal et al. [5] a spatial noise map, a semantic code and a sequence of guidance tokens from the same image. Abduction then returns whatever these models declare, so the states they recover are properties of the model rather than of the images.

Definition 4.1 (Abduction): Given a structural causal model M\mathcal{M} and evidence ee, abduction returns the posterior over the exogenous variables of M\mathcal{M} consistent with ee.

Definition 4.2 (Inversion): Given a conditional generator G(z,c)G(z,c) and an image xx, inversion returns a state zz satisfying G(z,c)≈xG(z,c)\approx x.

Definition 4.1 is relative to a declared model and inherits whatever that model’s assumptions secure, while Definition 4.2 is a numerical procedure that secures nothing on its own. The two coincide when the network is the model’s rendering mechanism, which the trained entries of Table I arrange for and a frozen backbone does not. What remains in the latter case is a recovery step carrying no guarantee, which makes its contribution an empirical question. Section V takes it up.

Flexibility of Transport and an Inversion-Free Pipeline

We first separate the attribute mechanism from the transport map that generates the output. We then identify composition, the requirement that a null antecedent return the factual, as the one further property the semantics places on the map itself. Using the freedom these results leave, we construct a pipeline that dispenses with inversion, and close with the properties to be tested empirically.

Separating the Attribute Mechanism from the Transport

Write a pipeline as a pair (A,T)(\mathcal{A}, T). The attribute mechanism A\mathcal{A} takes the factual image xx and an antecedent, reads off pa=ℓ(x)\mathrm{pa} = \ell(x), abducts the attribute-level noise and propagates the antecedent through the structural equations to obtain the counterfactual attributes pa∗\mathrm{pa}^{*}. The transport TT takes the factual image together with a condition cpa∗c_{\mathrm{pa}^{*}} built from those attributes, and returns an image realising them. In the running example A\mathcal{A} is the computation of Section IV-A and TT is whatever produces the edited image. The two are specified independently, and every method in Table 1 pairs some A\mathcal{A} with some TT.

MethodImage mechanismExogenous state
Pawlowski et al. [2]VAE, trainedlatent + residual
De Sousa Ribeiro et al. [3]HVAE, trainedlatents + residual
Melistas et al. [22]GAN, trainedsingle latent
Sanchéz and Tsafriris [4]diffusion, traineduncond. noise map
Rasal et al. [5]diff. AE, trainednoise map, semantic code, guidance tokens
De Sousa Ribeiro et al. [7]OT-coupled flowflow noise map
Tong et al. [6]SD1.5 frozen, adapter trainedcond. noise map

Table 1. Image mechanisms and the exogenous states they recover.

Definition 5.1 (Faithful transport): A transport TT is faithful if, for every factual image xx in the support and every attribute vector pa∗\mathrm{pa}^{*} in the range of A\mathcal{A}, the image it returns satisfies ℓ(T(x,cpa∗))=pa∗\ell(T(x,c_{\mathrm{pa}^{*}})) = \mathrm{pa}^{*}.

Proposition 5.2 (Invariance of attribute-level metrics under faithful transports): Assume the two-stage model of Section IV-B. Let T1T_1 and T2T_2 be faithful in the sense of Definition 5.1. Then (A,T1)(\mathcal{A},T_1) and (A,T2)(\mathcal{A},T_2) induce the same joint distribution of factual and counterfactual attributes, and any functional of that joint distribution is therefore invariant to which of the two is used.

Proof. Fix a factual image xx and an antecedent, and let pa=ℓ(x)\mathrm{pa} = \ell(x) and pa∗\mathrm{pa}^{*} be the attributes A\mathcal{A} returns. For j∈{1,2}j \in\{1,2\} write xj∗=Tj(x,cpa∗)x_j^{*} = T_j(x,c_{\mathrm{pa}^{*}}). The labelling map ℓ\ell is a function of the image alone, so ℓ(xj∗)\ell(x_j^{*}) is determined by xj∗x_j^{*} and does not depend on the trajectory through image space by which it was reached. By Definition 5.1, ℓ(xj∗)=pa∗\ell(x_j^{*}) = \mathrm{pa}^{*} almost surely for j=1,2j = 1,2. The pair (ℓ(x),ℓ(xj∗))(\ell(x),\ell(x_j^{*})) therefore equals (pa,pa∗)(\mathrm{pa},\mathrm{pa}^{*}) almost surely for both transports, and its distribution over factual images is fixed by the distribution of (pa,pa∗)(\mathrm{pa},\mathrm{pa}^{*}), which A\mathcal{A} determines from the factual image and the antecedent alone. The second claim follows, a functional of a distribution being determined by it. ■

The proposition says that a faithful transport is invisible to anything read off the attributes of the counterfactual image generated. Effectiveness is measured this way, by predicting the intervened attribute from the counterfactual image, so it cannot tell two such transports apart. Section VI nonetheless reports differences in it. The reason is that no transport on a pretrained backbone is faithful in the exact sense of Definition 5.1: a requested attribute is delivered only in part. A gap in effectiveness therefore records how much of the request each transport failed to deliver, and not which of them took the better route to the counterfactual.

Exact Composition Under a Null Antecedent

One property escapes Proposition 5.2. A counterfactual under a null antecedent must return the factual, since setting a variable to the value it already holds changes nothing. This is the composition axiom of Galles and Pearl [8], used as an evaluation metric for image counterfactuals by Monteiro et al. [23], and unlike identity preservation it can be checked without knowing the true rendering mechanism.

Proposition 5.3 (Exactness under a null antecedent): Let the antecedent set the intervened attribute to its observed value, so that pa∗=pa\mathrm{pa}^{*} = \mathrm{pa}. Then the transport of (2) returns the factual latent unchanged, for any step count NN, any number of noise draws MM, and any solver whose update is a linear combination of evaluations of the integrand.

Proof. Since pa∗=pa\mathrm{pa}^{*} = \mathrm{pa}, the two conditions coincide, ctar=csrcc_{\mathrm{tar}} = c_{\mathrm{src}}, so V(⋅,t,ctar)\mathcal{V}(\cdot,t,c_{\mathrm{tar}}) and V(⋅,t,csrc)\mathcal{V}(\cdot,t,c_{\mathrm{src}}) are the same function of state and time. The coupling z^ttar=z^t+(z^tsrc−x)\hat{z}_t^{\mathrm{tar}} = \hat{z}_t + (\hat{z}_t^{\mathrm{src}} - x) gives z^ttar=z^tsrc\hat{z}_t^{\mathrm{tar}} = \hat{z}_t^{\mathrm{src}} whenever z^t=x\hat{z}_t = x, in which case the two fields are evaluated at the same state, time and condition and the integrand of (2) vanishes for every draw ϵ\epsilon, hence in expectation. The trajectory starts at ztmax⁡=xz_{t_{\max}} = x, if zt=xz_t = x at the current step the update is zero and zt=xz_t = x at the next, so zt=xz_t = x throughout. ■

The round trip has no such guarantee. Inversion and regeneration are exact inverses only in the continuum limit, so at finite NN the two integrations accumulate discretisation error that does not cancel. The error shrinks with NN but is never zero, and it is paid on every query, including those where no edit is requested. Table IV measures the gap this predicts: under an antecedent restoring the observed value the displacement map returns to 0.0470 in LPIPS, against 0.4345 for the round trip.

InversionInversion-free
Effectiveness
Delivered (mm3\mathrm{mm}^{3}) ↑\uparrow+41,459 ±\pm 4,671+23,634 ±\pm 2,732
Realised ↑\uparrow82.1%46.8%
Identity (LPIPS)
Reconstruction ↓\downarrow0.05080.0223
Edit magnitude, median (max)0.149 (0.618)0.062 (0.108)
Do-then-undo ↓\downarrow0.43450.0470
Unstable outputs ↓\downarrow15/2730/273

Table IV. Counterfactual editing on ADNI, do(vent_vol→p95)do(\mathrm{vent\_vol}\to p_{95}). On the same backbone. n=91n = 91 subjects ×\times 3 axial levels; ±\pm is a 95% CI for the mean. Instrument floor 2,066 mm3\mathrm{mm}^{3}. Unstable is edit LPIPS > 0.30, unchanged at 3×\times median (9/273 against 0/273). Edit magnitude is not ranked.

An Inversion-Free Counterfactual Pipeline

We now give a pipeline that constructs no exogenous latent state at any point. It has three parts: the attribute mechanism producing pa∗\mathrm{pa}^{*}, a conditioning interface turning those attributes into an offset the frozen backbone can read, and the transport realising it (Fig. 1).

Conditioning interface

  1. Conditioning interface: The backbone reads an image as a grid of nn patches. A single vector added identically to every patch can request more of a quantity but cannot say where the change belongs, which is the wrong parameterisation for an attribute confined to one anatomical region. We instead give each attribute ii its own spatial pattern Mi∈Rn×dM_i \in\mathbb{R}^{n \times d}, one vector per patch, and add ∑iaiMi\sum_i a_i M_i to the patch embeddings before the first transformer block, where aia_i is the value of attribute ii. Each attribute contributes a fixed pattern whose amplitude is its value, so the request carries a location as well as a magnitude. The patterns are zero-initialised and attached at the input, so the pretrained weights are untouched and the frozen model is exactly recovered at initialisation. We write cpa∗c_{\mathrm{pa}^{*}} for the offset built from the counterfactual attributes.

Transport

  1. Transport: The backbone is a flow model: a network V(zt,t,c)\mathcal{V}(z_t,t,c) predicting, along the straight interpolation between an image xx and a noise draw ϵ\epsilon, the velocity carrying one to the other under a condition cc. Rather than integrate to noise and back, we adopt the displacement map of Kulikov et al. [18] unmodified and transport the image directly. The source state is the factual image noised along that path, z^tsrc=(1−t)x+tϵ\hat{z}_t^{\mathrm{src}} = (1-t)x+t\epsilon, and the target state is the same point carried forward by however far the edit has travelled, z^ttar=zt+(z^tsrc−x)\hat{z}_t^{\mathrm{tar}} = z_t + (\hat{z}_t^{\mathrm{src}}-x), so the two stay a fixed displacement apart and differ only in

the condition at which the field is evaluated. That difference drives the edit,

dzt=Eϵ[V(z^ttar,t,cpa∗)−V(z^tsrc,t,cpa)]dt,(2)d z_t = \mathbb{E}_{\epsilon}\left[\mathcal{V}(\hat{z}_t^{\mathrm{tar}},t,c_{\mathrm{pa}^{*}})-\mathcal{V}(\hat{z}_t^{\mathrm{src}},t,c_{\mathrm{pa}})\right]dt, \tag*{(2)}

integrated from tmax⁡t_{\max} to 0 with ztmax⁡=xz_{t_{\max}}=x and the expectation estimated over MM noise draws. The trajectory starts at a real image and moves only by accumulated velocity differences, requiring no inversion, no optimisation and no access to model internals.

Properties Left to Empirical Comparison

Few questions remain open, and they decide the choice between the two families in practice. Whether the counterfactual depicts the same subject lies outside Proposition 5.2, the exogenous state being no attribute of the diagram. How much of a requested change is actually delivered lies outside it too, faithfulness in the sense of Definition 5.1 being an idealisation a real backbone meets only approximately.

A transport that leaves the image untouched attains a perfect preservation score, so identity is informative only where the edit demonstrably occurred. Section VI adopts this convention and reports the comparison on CelebA, on a from-scratch checkpoint isolating the map, and on brain MRI with a continuous attribute.

Empirical Analysis

Below we compare inversion and inversion free methods using existing pretrained flow based generative backbones as well as scratch trained models for reference. Later we choose a case brain MRI images where the interventions are over continous attributes to contrast both family of methods (See Figure 1).

Benchmarking Pretrained Flow Based Models on CelebA

Setup

CelebA dataset [24] is used in this experiment, a split of 162,770 training/19,962 test samples, with the attributes Young (aa), Male (gg), No Beard (brbr) and Bald (blbl) over the causal graph as shown in the Figure 5. Each test image, is queried under the four single-node antecedents do(vS=1−vS)do(v_S=1-v_S). The counterfactual attributes are obtained by propagating each through the SCM. Interventions on the roots displace both leaves, interventions on the leaves must leave their non-descendants at the observed values. All methods see an identical query set.

Figure 5. The attribute-level diagram assumed on CelebA.

Baselines and protocol

Methods are grouped by whether a latent state is recovered at all rather than by architecture. The first group recovers one explicitly: VAE, HVAE and GAN are deep structural causal models using a learned encoder [2, 3], and Causal-Adapter recovers a latent state from a frozen SD1.5 by DDIM inversion [6]. FlowEdit integrates the displacement dynamics of Eq. (2) on a frozen SD3-medium [25] with the counterfactual attributes supplied through the target condition. Effectiveness is the F1 of an anti-causal classifier for the intervened attribute, pooled over the entire test split. Realism is FID [26], which compares generated and real images in the feature space of a pretrained network and is computed once per intervention across all samples. Identity is the LPIPS [27] between each counterfactual and its corresponding factual, a perceptual distance that better tracks human similarity judgments than pixel norms.

Analysis

Explicit inversion attains higher effectiveness, inversion-free higher identity preservation. SD3-inv and FlowEdit share the same frozen backbone. They differ only in the transport. The comparison between them therefore isolates the map. Inversion is more effective on three of four attributes. The gap is 0.307 on No_Beard and 0.366 on Bald. The inversion-free map stays closer to the factual on all four. On Young it records 0.078 against 0.369. On the same attribute the two are within 0.010 on effectiveness. Causal-Adapter and SD3-inv are the only methods to exceed 0.4 on every attribute. FlowEdit (inversion-free) attains LPIPS between 0.055 and 0.124 on the attributes where the edit is realised, against 0.119–0.246 for Causal-Adapter, and achieves the lowest identity cost on do(Young)do(Young) and do(No_Beard)do(No\_Beard) among methods whose edit took effect.

Bald is the attribute on which FlowEdit fails (0.039), despite attaining the highest F1(aa) (0.866). The backbone is frozen and receives the antecedent only through the target condition. So an attribute that it does not already represent densely cannot be requested effectively. Baldness is rare in web-scale pretraining relative to age.

The from-scratch baselines fail for reasons specific to each. The VAE produces blurred reconstructions, with FID exceeding 57 on every antecedent and effectiveness of 0.004 on Bald. The HVAE alters global contrast in place of the attribute, which yields the lowest FID on aa and gg and the lowest identity cost on gg (0.076) while failing the effectiveness criterion on two attributes. The GAN departs from the image manifold under some antecedents and incurs the highest identity cost of any successful edit (0.285 on Bald). The identity values of 0.26–0.36 reported for the VAE and GAN are therefore attributable largely to reconstruction error rather than to the intervention.

Scratch Trained Models

a) Setup: The comparison in Section VI-A varies the backbone, the training corpus and the transport simultaneously, and consequently cannot attribute an observed difference to any one of them. This section holds every factor fixed except the map. All rows of Table 3 are generated by a single

Inference mapFwd./queryEffectiveness F1 ↑ aaIdentity LPIPS ↓ aa
copy input00.1420.3490.0500.0070.0000.0000.0000.000
sample at pa∗pa^{*}320.5690.9920.3120.7000.4950.5170.5080.517
inversion-free, M=8M=85120.6740.9940.3610.724<u>0.070</u><u>0.325</u><u>0.128</u>0.430
+ FlowAlign, γ=0.01\gamma=0.015120.4200.9680.2470.5500.0330.2390.1040.402
inversion, N=32N=3264<u>0.610</u>0.992<u>0.314</u>0.7170.1480.3430.1880.432
inversion, N=256N=2565120.5390.9870.2970.6980.1160.3350.147<u>0.426</u>

Table 3. One model, several inference maps. All rows share the same scratch-trained DiT-B/2 (131 M, SD3 latents, independent coupling), so no difference between them is attributable to training. The first two rows are bounds and are excluded from the ranking; best per column in bold, runner-up underlined. Identity is LPIPS(x′x^{\prime}, xx) per intervention. The sample at pa∗pa^{*} bound never reads the factual image.

MethodBackbone\multicolumn{4}{c}{Effectiveness F1 ↑\uparrow}\multicolumn{4}{c}{Realism FID ↓\downarrow}\multicolumn{4}{c}{Identity LPIPS ↓\downarrow}
aaggbrbrblblaaggbrbrblblaaggbrbrblbl
Explicit inversion — encoder; DDIM for Causal-Adapter, flow-ODE for SD3-inv
VAEdeep-SCM0.3900.9900.2840.00460.063.459.957.40.2830.2950.2830.284†
HVAEdeep-SCM0.2460.742__0.476__0.1222.25.925.631.70.026†0.0760.1380.229†
GANdeep-SCM0.4710.9890.2730.34022.126.623.024.30.2560.3630.262__0.285__
Causal-AdapterSD1.5 + adapter0.530__0.997__0.4240.4978.29.08.017.70.1660.246__0.119__0.240
SD3-invSD3 (2B, frozen)__0.856__0.9370.522__0.405__33.147.033.137.40.3690.4640.3510.397
Inversion Free
FlowEditSD3 (2B, frozen)0.8660.9270.2150.03910.89.63.93.50.078__0.124__0.0550.049†

Table 2. CelebA counterfactuals, grouped by whether the exogenous state is recovered at all. Best per column in bold, runner-up underlined. Cells marked † are excluded from the ranking.

CelebA counterfactuals with four antecedents, one method per column

Figure 6. CelebA counterfactuals with four antecedents, one method per column.

intervention plan is done as described in Section VI-A. The rows differ only in the procedure by which the counterfactual image is obtained from the factual one.

b) Inference maps: Four maps are compared on a single checkpoint, all at guidance weight ω=2\omega=2 [28].

Inversion-free. The displacement map integrates the dynamics of Eq. (2) directly from the factual image, estimating the expectation over M=8M=8 noise draws at N=32N=32 steps, at 2NM2NM forward passes per query. Its regularised variant adds the source-consistency term of FlowAlign [19] at γ=0.01\gamma=0.01, which at each step contracts the accumulated displacement toward the factual rather than allowing it to accumulate unconstrained.

Inversion. The round trip recovers a noise map from the factual under the source condition and regenerates under the target condition, following a single trajectory rather than an expectation, at 2N2N forward passes. We report it at N=32N=32 and at N=256N=256.

Two bounds frame the comparison rather than competing in it: copying the input, which measures what the classifier assigns to an unaided image, and sampling at the counterfactual parent configuration, which measures the effectiveness attainable once the factual is disregarded entirely. Both are excluded from the ranking. Metrics follow Section VI-A.

c) Analysis: At matched budget the inversion-free map is ahead on all four effectiveness columns and on three of four identity columns, and the round trip only closes the gap by running at N=32N=32, at one eighth of the forward passes. The FlowAlign regulariser moves along the same trade-off rather than improving on it, gaining 0.028 in identity under do(Bald)do(Bald) at a cost of 0.174 in effectiveness.

Identity does separate the two maps, but only on some attributes. Under do(Young)do(Young) the inversion-free map (no path-reg) records an LPIPS of 0.070 against 0.148 for the round trip, and under do(NoBeard)do(No_Beard) 0.128 against 0.188, in both cases staying closer to the factual. Under do(Bald)do(Bald) the two are indistinguishable, at 0.430 and 0.432. What predicts this is how close the maps sit to the from-noise bound of 0.517: on Bald both are near it, and the prototype collapse described below leaves no identity for any map to preserve, whereas on Young there is room between them and the bound.

CelebA counterfactuals from the scratch-trained model

Figure 7. CelebA counterfactuals from the scratch-trained model, one inference map per column.

The from-scratch checkpoint also answers many queries with a similar output regardless of the factual, a failure neither effectiveness nor identity registers: the outputs carry the intervened attribute, and their perceptual distance to the factual is comparable to that of a subject-specific counterfactual.

Case study: Counterfactuals on brain MRIs

Setup

Our final setting is over structural brain MRIs from the ADNI dataset [29], a public study of aging and Alzheimer’s disease. We remove non-brain tissue from each scan and align it to a standard anatomical template, as also done in previous work [22], so that a given structure appears at approximately the same location in every image. We retain the ten central axial slices per subject used in prior work on this benchmark and report results at three of these levels. The attribute we intervene on is the volume of the ventricles, the fluid-filled cavities at the center of the brain, which enlarge as surrounding tissue is lost. Two features distinguish this setting from Section VI-A. The attribute is continuous, so an antecedent specifies how much change is requested and not merely its direction. And its effect is confined to a small, anatomically fixed region, so a successful edit must move one part of the image and leave the rest alone.

Modeling

The conditioning interface is that of § 5-C, with one attribute pattern per node of the causal graph. Because every scan is aligned to a common template, a given structure occupies approximately the same patches across subjects, so a fixed spatial pattern per attribute is well-defined across the dataset. Without alignment, the same pattern would map to different anatomy in each scan. The model is trained under the flow-matching objective alone, with the attribute dropped for a fixed fraction of samples so that classifier-free guidance is available without further training. One frozen backbone serves both transports, so every difference in Table IV is attributable to the map.

Analysis

Effectiveness is reported in the units of the attribute: delivered is the measured change in ventricular volume and realised that change as a fraction of the change requested. Identity is split across three axes. Reconstruction is the LPIPS between the factual and what the map returns before any intervention, isolating the cost the map imposes on the factual itself. Edit magnitude is the LPIPS under do(vent_vol →p95\to p_{95}). Do-then-undo applies the antecedent and then restores the observed value, the continuous-attribute analog of the null antecedent of Proposition 5.3.

Neither map wins outright. Inversion delivers about twice the requested change, realising 82.1% of it against 46.8%. But it displaces the factual twice as much before any intervention is applied, a cost paid on every subject, and it fails outright on 15 of 273 subjects where the inversion-free map fails on none. No subject fails under both maps, so these are failures of the transport on particular images rather than hard cases in the data.

Counterfactual editing examples for inversion and inversion-free methods

Figure 8. do(vent_vol→p95)do(\mathrm{vent\_vol}\to p_{95}) on four held-out subjects, taken at even quantiles of the requested change among stable outputs.

Composition shows the largest gap. When the antecedent restores the observed value, the two conditions coincide, the integrand of Eq. (2) vanishes, and the inversion-free map returns to 0.0470. The round trip has no such guarantee and records 0.4345. Inversion is useful when the requested edit is to be effective, while the inversion-free counterpart proves to be useful when some parts of the anatomy must remain untouched.

Conclusion

This work examined the role of the inversion step in high-dimensional counterfactual inference. Moving from the 1-D setting to the high-dimensional one splits the exogenous state into an attribute-level component and an image-level one. The attribute-level noise is recovered as in the scalar case, under the same identifying assumptions. For the image-level node there is no fitted mechanism to invert, and monotonicity is not available as an identifying assumption. Inversion applied over a backbone recovers a latent state from the factual image, but bears no identification guarantees, making its contribution an empirical question. This leaves room for an inversion-free alternative. On CelebA and ADNI we observe tradeoffs rather than dominance: inversion realises a larger fraction of the requested change, while the inversion-free construction departs less from the factual and returns it exactly under a null antecedent. The choice between them can be governed by whether an application prioritises the magnitude of the edit or the preservation of what was not intervened upon.

Our analysis covers a couple of inversion-free transports, and both work the same way, so a transport built on another principle may land elsewhere on the effectiveness–identity trade-off. We have also set the identification route of [7] aside, since it requires the image mechanism to be trained under an optimal-transport condition that a pretrained backbone does not meet.

As future work, we intend to explore applications where identity preservation is paramount, such as longitudinal medical imaging. We also intend to extend the comparison to complex causal graphs and to antecedents on several nodes at once, where the requested attributes compete for the same region of the image.

References

  1. [1]J. Pearl, Causality. Cambridge university press, 2009.
  2. [2]N. Pawlowski, D. Coelho de Castro, and B. Glocker, “Deep structural causal models for tractable counterfactual inference,” in Advances in Neural Information Processing Systems, vol. 33, 2020, pp. 857–869.arxiv.org/abs/2006.06485
  3. [3]F. De Sousa Ribeiro, T. Xia, M. Monteiro, N. Pawlowski, and B. Glocker, “High fidelity image counterfactuals with probabilistic causal models,” in Proceedings of the 40th International Conference on Machine Learning, ser. ICML’23. JMLR.org, 2023.arxiv.org/abs/2306.15764
  4. [4]P. Sanchez and S. A. Tsaftaris, “Diffusion causal models for counterfactual estimation,” in Conference on Causal Learning and Reasoning. PMLR, 2022, pp. 647–668.arxiv.org/abs/2202.10166
  5. [5]R. R. Rasal, A. Kori, F. D. S. Ribeiro, T. Xia, and B. Glocker, “Diffusion counterfactual generation with semantic abduction,” in International Conference on Machine Learning. PMLR, 2025, pp. 51 201–51 228.arxiv.org/abs/2506.07883
  6. [6]L. Tong, Z. Liu, C. Lu, D. Oglic, T. Diethe, P. Teare, S. A. Tsaftaris, and C. Jin, “Causal-adapter: Taming text-to-image diffusion for faithful counterfactual generation,” arXiv preprint arXiv:2509.24798, 2025.arxiv.org/abs/2509.24798
  7. [7]F. De Sousa Ribeiro, A. Santhirasekaram, and B. Glocker, “Counterfactual identifiability via dynamic optimal transport,” Advances in Neural Information Processing Systems, vol. 38, pp. 147 855–147 903, 2026.arxiv.org/abs/2510.08294
  8. [8]D. Galles and J. Pearl, “An axiomatic characterization of causal counterfactuals,” Foundations of Science, vol. 3, no. 1, pp. 151–182, 1998.DOI
  9. [9]P. Hoyer, D. Janzing, J. M. Mooij, J. Peters, and B. Schölkopf, “Nonlinear causal discovery with additive noise models,” Advances in neural information processing systems, vol. 21, 2008.
  10. [10]K. Zhang and A. Hyvärinen, “On the identifiability of the post-nonlinear causal model,” arXiv preprint arXiv:1205.2599, 2012.arxiv.org/abs/1205.2599
  11. [11]A. Nasr-Esfahany, M. Alizadeh, and D. Shah, “Counterfactual identifiability of bijective causal models,” in International conference on machine learning. PMLR, 2023, pp. 25 733–25 754.
  12. [12]J. Von Kügelgen, A. Mohamed, and S. Beckers, “Backtracking counterfactuals,” in Conference on Causal Learning and Reasoning. PMLR, 2023, pp. 177–196.arxiv.org/abs/2211.00472
  13. [13]K. Xia, Y. Pan, and E. Bareinboim, “Neural causal models for counterfactual identification and estimation,” in The Eleventh International Conference on Learning Representations, 2023.arxiv.org/abs/2210.00035
  14. [14]K.-R. Klady, J. von Kügelgen, B. Schölkopf, and M. Muehlebach, “Deep backtracking counterfactuals for causally compliant explanations,” Transactions on Machine Learning Research, 2024.
  15. [15]E. De Brouwer, “Deep counterfactual estimation with categorical background variables,” Advances in Neural Information Processing Systems, vol. 35, pp. 35 213–35 225, 2022.arxiv.org/abs/2210.05811
  16. [16]Y. Ma, V. Melnychuk, D. Frauen, and S. Feuerriegel, “Consistent end-to-end estimation for counterfactual fairness,” in Fifth Conference on Causal Learning and Reasoning, 2026.
  17. [17]Y. Pan and E. Bareinboim, “Counterfactual image editing,” in International Conference on Machine Learning. PMLR, 2024, pp. 39 087–39 101.arxiv.org/abs/2403.09683
  18. [18]V. Kulikov, M. Kleiner, I. Huberman-Spiegelglas, and T. Michaeli, “Flowedit: Inversion-free text-based editing using pre-trained flow models,” in Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), October 2025, pp. 19 721–19 730.arxiv.org/abs/2412.08629
  19. [19]J. Kim, Y. Hong, J. Park, and J. C. Ye, “Flowalign: Trajectory-regularized, inversion-free flow-based image editing,” in International Conference on Learning Representations, vol. 2026, 2026, pp. 121 814–121 846.arxiv.org/abs/2505.23145
  20. [20]Y. Lipman, R. T. Chen, H. Ben-Hamu, M. Nickel, and M. Le, “Flow matching for generative modeling,” arXiv preprint arXiv:2210.02747, 2022.arxiv.org/abs/2210.02747
  21. [21]X. Liu, C. Gong, and Q. Liu, “Flow straight and fast: Learning to generate and transfer data with rectified flow,” arXiv preprint arXiv:2209.03003, 2022.arxiv.org/abs/2209.03003
  22. [22]T. Melistas, N. Spyrou, N. Gkouti, P. Sanchez, A. Vlontzos, Y. Panagakis, G. Papanastasiou, and S. A. Tsaftaris, “Benchmarking counterfactual image generation,” Advances in Neural Information Processing Systems, vol. 37, pp. 133 207–133 230, 2024.arxiv.org/abs/2403.20287
  23. [23]M. Monteiro, F. D. S. Ribeiro, N. Pawlowski, D. C. Castro, and B. Glocker, “Measuring axiomatic soundness of counterfactual image models,” arXiv preprint arXiv:2303.01274, 2023.arxiv.org/abs/2303.01274
  24. [24]Z. Liu, P. Luo, X. Wang, and X. Tang, “Deep learning face attributes in the wild,” in Proceedings of the IEEE international conference on computer vision, 2015, pp. 3730–3738.arxiv.org/abs/1411.7766
  25. [25]P. Esser, S. K ulal, A. Blattmann, R. Entenzari, J. Müller, H. Saini, Y. Levi, D. Lorenz, A. Sauer, F. Boesel et al., “Scaling rectified flow transformers for high-resolution image synthesis,” in Forty-first international conference on machine learning, 2024.arxiv.org/abs/2403.03206
  26. [26]M. Heusel, H. Ramsauer, T. Unterthiner, B. Nessler, and S. Hochreiter, “Gans trained by a two time-scale update rule converge to a local nash equilibrium,” Advances in neural information processing systems, vol. 30, 2017.
  27. [27]R. Zhang, P. Isola, A. A. Efros, E. Shechtman, and O. Wang, “The unreasonable effectiveness of deep features as a perceptual metric,” in 2018 IEEE/CVF conference on computer vision and pattern recognition. IEEE, 2018, pp. 586–595.arxiv.org/abs/1801.03924
  28. [28]J. Ho and T. Salimans, “Classifier-free diffusion guidance,” arXiv preprint arXiv:2207.12598, 2022.arxiv.org/abs/2207.12598
  29. [29]C. R. Jack Jr et al., “The alzheimer’s disease neuroimaging initiative (adni): Mri methods,” Journal of Magnetic Resonance Imaging: An Official Journal of the International Society for Magnetic Resonance in Medicine, vol. 27, no. 4, pp. 685–691, 2008.DOI

Paper details

vlnc:W_sk4nf8zg
Version
v1 · latest
Published
Fresh — added to the Hub within the last 30 days
Added
License
CC BY 4.0
Usage

views · 6

downloads · 0

A view counts once per browser per day, a download each time the PDF is requested.

  • Computer Science

Not recorded for this paper

View statement
Looking for related papers…
Contents