Introduction
Identifying the causal predictors of a target response is an important problem in causal inference. However, data from a single observational distribution generally cannot distinguish causal predictors from noncausal variables that are associated with the response. Data collected across varying settings and conditions, broadly called environments , offer an additional source of information for causal discovery. Although interventions or distributional changes may alter the distributions of covariates and their associations with the response, the causal mechanism governing the response remains invariant across environments, provided it is not directly intervened upon.
This idea underlies the invariant causal prediction framework of Peters et al. [18 ] . Let S ⋆ S^{\star} S ⋆ denote the causal parent set of a target response Y Y Y . If the causal mechanism for Y Y Y is unchanged, then the conditional distribution of Y Y Y given X S ⋆ X_{S^{\star}} X S ⋆ remains invariant across environments. This establishes a connection between invariance and causality: the true causal parent set must yield an invariant conditional distribution, and any predictor set exhibiting such invariance is a natural candidate for the causal parent set. Motivated by this connection, Peters et al. [18 ] identify invariant predictor sets across environments and take their intersection to recover a subset of S ⋆ S^{\star} S ⋆ .
Since the work by Peters et al. [18 ] , exploiting invariance across environments has become an important strategy for causal discovery. One line of research estimates invariant predictive models through optimization, typically by incorporating a tractable property of invariance into the estimation objective. Under suitable identifiability conditions, these methods can recover the true causal model. Arjovsky et al. [1 ] provide an early example with invariant risk minimization (IRM), which seeks a data representation such that the same prediction rule minimizes the risk in every environment. However, IRM does not generally guarantee recovery of the causal parent set, even in linear settings, and its practical implementation relies on highly heuristic nonconvex optimization that can be sensitive to initialization and tuning parameters [11 , 20 ] .
Other approaches focus on estimating causal regression coefficients. Rothenhäusler et al. [21 ] propose Causal Dantzig, which estimates causal coefficients in linear structural models using invariant moment conditions under additive interventions. More recently, Wang et al. [24 ] developed NegDRO, a nonconvex minimax approach that exploits invariance of prediction risks to identify causal coefficients. Although both Causal Dantzig and NegDRO can accommodate hidden confounding, they rely on specific additive intervention mechanisms and assume a linear structural causal model for the entire system, which may limit their applicability.
In contrast, EILLS, introduced by Fan et al. [5 ] , and CoCo, proposed by Yin et al. [27 ] , do not require a specific intervention mechanism. Both approaches exploit the first order optimality conditions of least squares regression across environments to identify causal coefficients. EILLS incorporates an invariance penalty into the regression objective, together with indicators for selected features, whereas CoCo relies on continuous nonconvex optimization. Another method is Bayesian invariant prediction (BIP), proposed by Wu et al. (2025), which uses latent binary indicators to select variables whose conditional relationship with the response remains invariant across environments.
Among these methods, only EILLS and BIP explicitly select subsets of predictors when identifying an invariant model, while sparse formulations of Causal Dantzig perform variable selection through ℓ 1 \ell_{1} ℓ 1 regularization. However, the exact implementations of EILLS and BIP require exhaustive enumeration over candidate predictor subsets, making them computationally prohibitive even in moderate dimensions. Although their approximate implementations alleviate this computational burden, they lack any optimality guarantees. In contrast, ℓ 1 \ell_{1} ℓ 1 -regularized Causal Dantzig admits a computationally tractable convex formulation, but its reliance on a convex relaxation of variable selection may lead to shrinkage biases and inaccurate support recovery.
The methods discussed above, including IRM, Causal Dantzig, NegDRO, EILLS, CoCo, and BIP, generally require sufficient intervention heterogeneity to guarantee recovery of the causal parent set. Without such heterogeneity, the variables discovered by these methods may include noncausal predictors, and recovery of even a subset of the true causal parent set is not guaranteed.
An alternative line of work seeks to identify a subset of the causal parent set while controlling type-I error, without requiring full identifiability. As discussed earlier, Peters et al. [18 ] introduce invariant causal prediction (ICP), which tests invariance over all candidate predictor subsets and takes the intersection of the accepted sets. However, exhaustive enumeration of predictor subsets becomes computationally prohibitive as the dimension increases. To address this limitation, Martinet et al. [15 ] formulate causal predictor discovery as a multiple testing problem instead of exhaustive subset enumeration. These tests use Wasserstein variance minimization (WVM) to assess whether the conditional distribution of the response can remain invariant across environments when a given variable is excluded. However, WVM remains computationally expensive. Moreover, both ICP and WVM can have limited statistical power in identifying causal predictors, particularly at moderate sample sizes.
Gaps and our contributions
Gaps and our contributions. The preceding discussion highlights two related but distinct tasks in invariant causal prediction:
(T1) When S ⋆ S^{\star} S ⋆ is identifiable, how can we accurately recover the entire causal parent set?
(T2) When S ⋆ S^{\star} S ⋆ is not fully identifiable, how can we identify a subset of S ⋆ S^{\star} S ⋆ while controlling false causal discoveries?
As shown in Table 1 , existing approaches either do not explicitly perform variable selection or generally focus on only one of these two goals. To address both tasks, we introduce an invariance score that quantifies the degree of invariance of a predictor set S S S , with smaller scores indicating greater invariance. For Task (T1), we incorporate this score as a regularization term in the risk minimization objective to encourage the selection of an invariant predictor set. We establish identifiability conditions for S ⋆ S^{\star} S ⋆ and show that the resulting estimator recovers S ∗ S^{*} S ∗ under suitable regularity conditions. For Task (T 2 T2 T 2 ), we adopt the multiple testing framework of Martinet et al. [15 ] . Specifically, for each variable k k k , we search for the minimum invariance score among all predictor sets that exclude k k k . If even this minimum score is sufficiently large, it provides evidence that no invariant predictor set can exclude k k k . Consequently, k k k must belong to the intersection of all invariant predictor sets and hence to the true causal parent set S ∗ S^{*} S ∗ . We further develop a multiplier bootstrap procedure to approximate the distribution of the minimum score and establish asymptotic type-I error control for false causal discoveries.
Table 1. Comparison of related work and our contributions. Note that ICP and WVM can recover S ∗ S^{*} S ∗ only under identifiability.
Notably, both tasks involve optimization over discrete predictor subsets. To avoid exhaustive subset enumeration, we develop convex mixed-integer programming (MIP) formulations for the invariance score and its subsequent optimization problems. Subset selection is naturally represented through binary variables, and the resulting formulations have convex continuous relaxations, enabling efficient optimization and certificates of global optimality. Unlike existing approaches ICP, EILLS and BIP, our MIP framework allows the use of branch-and-bound algorithms that (i) efficiently prune candidate subsets to reduce the computational burden and (ii) return solutions with guaranteed optimality gaps below a prespecified tolerance. We further develop a scalable coordinate descent heuristic that efficiently finds solutions of high quality. When incorporated into the MIP framework, these solutions can be equipped with certificates of optimality gaps.
Our invariance score is constructed based on the least-squares coefficient and risk (LSCR) invariance principle for linear models. This principle requires both the population ordinary least squares (OLS) coefficients and the corresponding OLS risks to remain invariant across environments. Unlike Causal Dantzig and NegDRO, neither the linear structural causal model we consider nor the LSCR principle imposes specific restrictions on the intervention mechanisms, provided that the causal mechanism for the response remains unchanged. Moreover, tests based on the LSCR principle demonstrate greater statistical power than ICP and WVM in our experiments. This improvement can be attributed, in part, to the stronger invariance restrictions imposed by LSCR compared with tests on the regression residuals used in ICP, which can fail to detect changes in regression coefficients. In contrast, WVM involves a highly nonconvex optimization problem, making it susceptible to suboptimal solutions that can compromise its empirical performance. Our MIP framework can also be extended to settings with latent confounding by adopting a different invariance score.
Finally, we demonstrate the performance of our methods using both synthetic data and real data collected from a tightly controlled physical system. First, our estimator for Task (T1) achieves substantially higher accuracy in recovering causal parents than competing methods as the dimension increases. Second, our procedure for Task (T2) consistently achieves higher statistical power than competing methods while maintaining type-I error control. For both tasks, our methods remain computationally competitive.
The remainder of the paper is organized as follows. § 2 introduces the problem setup and the LSCR invariance principle. § 3 presents the MIP formulations for the invariance score and the two complementary tasks. Sections 4 and 5 investigate Task (T1) and Task (T2), respectively. Sections 6 and 7 evaluate the proposed approach through synthetic and real data experiments. Section 8 extends our framework to settings with nonlinear causal effects or hidden confounding. Finally, Section 9 discusses future research directions. The code for implementing our methods and reproducing the numerical experiments is available at https://github.com/Xiaozhu-Zhang1998/mip_inv_pred.
Other related literature
This section reviews three additional lines of related research not discussed earlier and highlights their connections to and distinctions from our proposed framework.
Distributional robustness and domain generalization. Beyond causal predictor selection, invariance has also been widely used to improve prediction under distributional shifts. One line of work focuses on identifying invariant predictor subsets that generalize to unseen environments. Rojas-Carulla et al. [20 ] study this problem in causal transfer learning, while Pfister et al. [19 ] combine regression models based on invariant predictor subsets to improve predictive performance across environments. Another line of work focuses on distributional robustness. Meinshausen and Bühlmann [16 ] introduce maximin effects for robust regression with heterogeneous data, while anchor regression (Rothenhäusler et al., [22 ] ) and DRIG (Shen et al., [23 ] ) aim to achieve robust prediction under distributional shifts caused by interventions. For probabilistic prediction, Henzi et al. [10 ] propose invariant probabilistic prediction (IPP), which uses proper scoring rules to encourage invariance of prediction risks across environments. While these approaches primarily aim to improve prediction under distributional shifts, our work focuses on identifying causal predictors through invariance across environments.
Beyond linear invariant prediction. Another line of work extends invariant causal prediction beyond linear models. Heinze-Deml et al. [9 ] develop nonlinear and nonparametric versions of ICP using statistical tests for invariance across environments. Kook et al. [12 ] extend ICP to general response types, including categorical, count, and censored outcomes, through transformation models. More recently, Gu et al. (2025) propose focused adversarial invariant regularization, which uses neural networks and adversarial optimization to identify invariant predictors in nonparametric regression models. These approaches extend invariant causal learning to more flexible model classes, whereas our work focuses on linear models to develop mixed-integer convex formulations with statistical and optimization guarantees. Nonetheless, we describe how our MIP formulations can be extended to handle nonlinear additive models in Section 8.
Mixed-integer optimization in statistical learning. Recent advances in mixed-integer programming have made combinatorial optimization increasingly practical for statistical learning. Bertsimas et al. [3 ] develop a mixed-integer optimization approach to best subset selection, using initial high-quality solutions to accelerate branch-and-bound algorithms. Bertsimas and Van Parys [4 ] further introduce a binary convex reformulation and cutting-plane algorithms for scalable ℓ 0 \ell_0 ℓ 0 -regularized regression. Hazimeh and Mazumder [8 ] develop efficient coordinate descent and local combinatorial search algorithms for best subset selection. Similar ideas have also been applied to other statistical problems, including multi-task learning (Behдин et al., 2026) and Bayesian network structure learning (Manzour et al., [14 ] ). These studies demonstrate that exploiting problem structures can substantially improve the computational efficiency of discrete optimization. Convex mixed-integer programming has also been used to learn entire causal graphs (Küçükyavuz et al., [13 ] ; Manzour et al. [14 ] ; Xu et al. (2025); Zhang et al. [28 ] ), although this is a substantially more challenging problem that typically requires modeling assumptions on the entire system of variables.
Notation
For a random variable Z Z Z with observations { Z i } i = 1 m \{Z_i\}_{i=1}^{m} { Z i } i = 1 m , we define the empirical squared norm by ∥ Z ∥ m 2 : = m − 1 ∑ i = 1 m Z i 2 \lVert Z\rVert_m^2 := m^{-1}\sum_{i=1}^{m} Z_i^2 ∥ Z ∥ m 2 := m − 1 ∑ i = 1 m Z i 2 . Similarly, for two random variables Z Z Z and W W W with paired observations { ( Z i , W i ) } i = 1 m \{(Z_i,W_i)\}_{i=1}^{m} {( Z i , W i ) } i = 1 m , we define the empirical inner product by ⟨ Z , W ⟩ m : = m − 1 ∑ i = 1 m Z i W i \langle Z,W\rangle_m := m^{-1}\sum_{i=1}^{m} Z_iW_i ⟨ Z , W ⟩ m := m − 1 ∑ i = 1 m Z i W i . For any index set S ⊆ [ p ] S \subseteq[p] S ⊆ [ p ] and random vector U ∈ R p U \in\mathbb{R}^{p} U ∈ R p , we use U S ∈ R ∣ S ∣ U_S \in\mathbb{R}^{|S|} U S ∈ R ∣ S ∣ to denote the subvector of U U U indexed by S S S . For a vector w ∈ R p w \in\mathbb{R}^{p} w ∈ R p , we use supp ( w ) = { j : w j ≠ 0 } \operatorname{supp}(w)=\{j:w_j\ne0\} supp ( w ) = { j : w j = 0 } to denote its support. Lastly, for two sequences a n a_n a n and b n b_n b n , we write a n ≲ b n a_n\lesssim b_n a n ≲ b n if there exists a constant C > 0 C>0 C > 0 and n 0 n_0 n 0 such that a n ≤ C b n a_n\le Cb_n a n ≤ C b n for all n ≥ n 0 n\ge n_0 n ≥ n 0 . We write a n ≳ b n a_n\gtrsim b_n a n ≳ b n if both a n ≲ b n a_n\lesssim b_n a n ≲ b n and a n ≳ b n a_n\gtrsim b_n a n ≳ b n hold. We write a n = o ( b n ) a_n=o(b_n) a n = o ( b n ) if a n / b n → 0 a_n/b_n\to0 a n / b n → 0 as n → ∞ n\to\infty n → ∞ .
Setup: causality and invariant predictions from heterogeneous data
Linear structural causal model
We suppose we have access to observations of variables under different environments, such as experimental conditions in which some of the variables may have been manipulated, that is, received interventions. To represent this setting, we consider covariates X ∈ R p X \in\mathbb{R}^{p} X ∈ R p , and response variable Y ∈ R Y \in\mathbb{R} Y ∈ R . The interventions on these variables take values in the set E \mathcal{E} E ; each e ∈ E e \in\mathcal{E} e ∈ E represents a different environment that generates the random vector ( X e , Y e ) (X^{e}, Y^{e}) ( X e , Y e ) . We posit that for every e ∈ E e \in\mathcal{E} e ∈ E , the covariates X e X^{e} X e and the response Y e Y^{e} Y e are related via the following linear structural causal model (SCM) for some distribution F \mathcal{F} F :
Y e = β ⋆ T X e + ε e , supp ( β ⋆ ) = S ⋆ , ε e ⊥ X S ⋆ e , ε e ∼ F , E [ ε e ] = 0. (1) Y^{e} = \beta^{\star\mathsf{T}} X^{e} + \varepsilon^{e}, \quad\operatorname{supp}(\beta^{\star}) = S^{\star}, \quad\varepsilon^{e} \perp X_{S^{\star}}^{e}, \quad\varepsilon^{e} \sim\mathcal{F}, \quad\mathbb{E}[\varepsilon^{e}] = 0. \tag*{(1)} Y e = β ⋆ T X e + ε e , supp ( β ⋆ ) = S ⋆ , ε e ⊥ X S ⋆ e , ε e ∼ F , E [ ε e ] = 0. (1)
Here, β ⋆ ∈ R p \beta^{\star} \in\mathbb{R}^{p} β ⋆ ∈ R p is a vector encoding causal relationships between the covariates and the response variable, namely β j ⋆ ≠ 0 \beta_{j}^{\star} \ne0 β j ⋆ = 0 if covariate X j e X_{j}^{e} X j e is a causal parent of Y e Y^{e} Y e in the graph among ( X e , Y e ) (X^{e}, Y^{e}) ( X e , Y e ) . The subset S ⋆ ⊆ { 1 , 2 , … , p } S^{\star} \subseteq\{1,2,\ldots,p\} S ⋆ ⊆ { 1 , 2 , … , p } represents the covariates that are causal parents of Y e Y^{e} Y e , and the coefficients β S ⋆ ⋆ \beta_{S^{\star}}^{\star} β S ⋆ ⋆ represent the strength of these causal effects. The SCM (1) thus posits that the structure and strength of the causal parents of Y e Y^{e} Y e do not change across e ∈ E e \in\mathcal{E} e ∈ E . The noise variable ε e ∈ R \varepsilon^{e} \in\mathbb{R} ε e ∈ R is assumed to be independent of the causal parents X S ⋆ e X_{S^{\star}}^{e} X S ⋆ e . This independence condition corresponds to a causal sufficiency assumption, namely that there is no unobserved confounding simultaneously affecting the covariates and the response variable. In Appendix D, we discuss extensions to the setting with latent confounding. Importantly, ε e \varepsilon^{e} ε e can have dependencies with other coordinates of X e X^{e} X e outside the index set S ⋆ S^{\star} S ⋆ . Such dependencies are expected when some of the covariates are descendants of Y Y Y in the causal graph.
The condition that the distribution of ε e \varepsilon^{e} ε e remains the same across all e ∈ E e \in\mathcal{E} e ∈ E means that the response variable does not receive a direct intervention, although its marginal distribution could still be changed due to interventions on its causal ancestors. Finally, the zero-mean condition on ε e \varepsilon^{e} ε e can be assumed without loss of generality, as a nonzero mean can be absorbed into the term β ⋆ T X e \beta^{\star\mathsf{T}} X^{e} β ⋆ T X e by augmenting the covariates with a constant intercept variable. The parameter β ⋆ \beta^{\star} β ⋆ and the distribution F \mathcal{F} F are not assumed to be known.
For every e ∈ E e \in\mathcal{E} e ∈ E , we assume that we have n e n^{e} n e independent and identically distributed observations of ( X e , Y e ) (X^{e}, Y^{e}) ( X e , Y e ) from the SCM (1) . We further assume that the samples are independent across environments.
LSCR-invariant prediction models
The SCM (1) exhibits important invariances that we highlight in this section. For any S ⊆ [ p ] S \subseteq[p] S ⊆ [ p ] and e ∈ E e \in\mathcal{E} e ∈ E , define
β e , S : = arg min β ~ ∈ R p , supp ( β ~ ) ⊆ S E [ ( Y e − β ~ T X e ) 2 ] , ν e , S : = E [ ( Y e − β e , S T X e ) 2 ] , (2) \beta^{e,S} := \underset{\widetilde{\beta} \in\mathbb{R}^{p},\ \operatorname{supp}(\widetilde{\beta}) \subseteq S}{\arg\min}\ \mathbb{E}\left[\left(Y^{e} - \widetilde{\beta}^{\mathsf{T}}X^{e}\right)^{2}\right], \qquad\nu^{e,S} := \mathbb{E}\left[\left(Y^{e} - \beta^{e,S\mathsf{T}}X^{e}\right)^{2}\right], \tag*{(2)} β e , S := β ∈ R p , supp ( β ) ⊆ S arg min E [ ( Y e − β T X e ) 2 ] , ν e , S := E [ ( Y e − β e , S T X e ) 2 ] , (2)
as the population OLS solution when linearly regressing Y e Y^{e} Y e on X S e X_{S}^{e} X S e , and the corresponding risk. We propose the following invariance principle based on the OLS coefficients and risk.
Definition 1 (LSCR-invariant prediction model). Let supp ( β ) ⊆ S \operatorname{supp}(\beta) \subseteq S supp ( β ) ⊆ S for β ∈ R p \beta\in\mathbb{R}^{p} β ∈ R p . We say that β \beta β is a least-squares coefficient and risk (LSCR) invariant prediction model if:
β = β e , S and ν e , S = ν f , S for every e , f ∈ E . \beta= \beta^{e,S} \quad\text{and} \quad\nu^{e,S} = \nu^{f,S} \quad\text{for every } e,f \in\mathcal{E}. β = β e , S and ν e , S = ν f , S for every e , f ∈ E . The corresponding LSCR-invariant collection is defined as:
S L S C R ( E ) : = { S ⊆ { 1 , 2 , … , p } : β e , S = β f , S and ν e , S = ν f , S , ∀ e , f ∈ E } , \mathcal{S}_{\mathrm{LSCR}}(\mathcal{E}) := \left\{S \subseteq\{1,2,\ldots,p\} : \beta^{e,S} = \beta^{f,S}\ \text{and}\ \nu^{e,S} = \nu^{f,S},\ \forall e,f \in\mathcal{E}\right\}, S LSCR ( E ) := { S ⊆ { 1 , 2 , … , p } : β e , S = β f , S and ν e , S = ν f , S , ∀ e , f ∈ E } , which contains all subsets of covariates that result in LSCR-invariant prediction models.
An LSCR prediction model β \beta β is an OLS solution that remains invariant across environments and whose corresponding residual risk is also constant across environments. Under this definition, the invariant coefficient vector and its support determine each other. Specifically, if β \beta β is LSCR-invariant, then its support S = supp ( β ) S = \operatorname{supp}(\beta) S = supp ( β ) necessarily belongs to S L S C R ( E ) \mathcal{S}_{\mathrm{LSCR}}(\mathcal{E}) S LSCR ( E ) . Conversely, if S ∈ S L S C R ( E ) S \in\mathcal{S}_{\mathrm{LSCR}}(\mathcal{E}) S ∈ S LSCR ( E ) , then the associated invariant coefficient is uniquely determined as the population OLS coefficient restricted to S S S . In particular,
β = β e , S = arg min β ~ ∈ R p , supp ( β ~ ) ⊆ S ∑ e ∈ E E [ ( Y e − β ~ T X e ) 2 ] . (3) \beta= \beta^{e,S} = \underset{\widetilde{\beta} \in\mathbb{R}^{p},\ \operatorname{supp}(\widetilde{\beta}) \subseteq S}{\arg\min}\ \sum_{e \in\mathcal{E}} \mathbb{E}\left[\left(Y^{e} - \widetilde{\beta}^{\mathsf{T}}X^{e}\right)^{2}\right]. \tag*{(3)} β = β e , S = β ∈ R p , supp ( β ) ⊆ S arg min e ∈ E ∑ E [ ( Y e − β T X e ) 2 ] . (3)
Thus, the same coefficient is obtained from each individual environment as well as from the pooled population distribution. Throughout the paper, we use the term model to refer interchangeably to the coefficient vector β \beta β and its support S S S whenever the correspondence is clear from context.
Under the SCM in (1), both the causal coefficient vector β ⋆ \beta^\star β ⋆ , which links X S ⋆ e X_{S^\star}^{e} X S ⋆ e to Y e Y^{e} Y e , and the distribution of the noise term ε e \varepsilon^{e} ε e are invariant over e ∈ E e \in\mathcal{E} e ∈ E . It therefore follows that β ⋆ \beta^\star β ⋆ is an LSCR prediction model. However, as illustrated in the following example, unless the data across the environments are sufficiently heterogeneous, the collection S L S C R ( E ) \mathcal{S}_{\mathrm{LSCR}}(\mathcal{E}) S LSCR ( E ) may also contain models that include spurious covariates. At one extreme, if there is only a single environment, then every subset S ⊆ [ p ] S \subseteq[p] S ⊆ [ p ] belongs to S L S C R ( E ) \mathcal{S}_{\mathrm{LSCR}}(\mathcal{E}) S LSCR ( E ) .
Example 1. Consider the linear SCM in Figure 1 with E = { 1 , 2 , 3 } \mathcal{E} = \{1,2,3\} E = { 1 , 2 , 3 } . The set S ⋆ = { 1 , 3 } S^\star= \{1,3\} S ⋆ = { 1 , 3 } denotes the true causal parent set of the response Y Y Y . We consider several candidate subsets and their corresponding population OLS solutions β e , S \beta^{e,S} β e , S and risks ν e , S \nu^{e,S} ν e , S in Table 2 . First, the set S 1 = { 1 , 3 } S_1 = \{1,3\} S 1 = { 1 , 3 } is the true causal parent set of Y Y Y , and is therefore LSCR-invariant. In particular, β e , S 1 = β ⋆ \beta^{e,S_1} = \beta^\star β e , S 1 = β ⋆ and ν e , S 1 = σ 2 \nu^{e,S_1} = \sigma^2 ν e , S 1 = σ 2 for all e ∈ E e \in\mathcal{E} e ∈ E . Interestingly, the set S 2 = { 1 , 2 } S_2 = \{1,2\} S 2 = { 1 , 2 } is also LSCR-invariant, even though X 2 X_2 X 2 is not a direct parent of Y Y Y . This is because the invariant relationship X 3 e = γ 3 X 2 e + ε 3 e X_3^e = \gamma_3 X_2^e + \varepsilon_3^e X 3 e = γ 3 X 2 e + ε 3 e allows X 2 X_2 X 2 to serve as an alternative predictor for X 3 X_3 X 3 . However, if interventions were also imposed on γ 3 \gamma_3 γ 3 or η 3 \eta_3 η 3 , then S 2 S_2 S 2 would no longer be LSCR-invariant. Thus, distinguishing alternative invariant predictors from the true causal parents requires sufficiently rich intervention mechanisms. In contrast, for S 3 = { 4 } S_3 = \{4\} S 3 = { 4 } , both β e , S 3 \beta^{e,S_3} β e , S 3 and ν e , S 3 \nu^{e,S_3} ν e , S 3 vary across environments because V 1 ≠ V 2 ≠ V 3 V_1 \ne V_2 \ne V_3 V 1 = V 2 = V 3 where V e = Var ( Y e ) V_e = \operatorname{Var}(Y^e) V e = Var ( Y e ) . Therefore, S 3 S_3 S 3 is not LSCR-invariant. By enumerating all subsets in this example, we obtain
S L S C R ( E ) = { { 1 , 2 } , { 1 , 3 } , { 1 , 2 , 3 } , { 1 , 2 , 4 } , { 1 , 3 , 4 } , { 1 , 2 , 3 , 4 } } , \mathcal{S}_{\mathrm{LSCR}}(\mathcal{E}) = \{\{1,2\},\ \{1,3\},\ \{1,2,3\},\ \{1,2,4\},\ \{1,3,4\},\ \{1,2,3,4\}\}, S LSCR ( E ) = {{ 1 , 2 } , { 1 , 3 } , { 1 , 2 , 3 } , { 1 , 2 , 4 } , { 1 , 3 , 4 } , { 1 , 2 , 3 , 4 }} , where every invariant set contains either { 1 , 2 } \{1,2\} { 1 , 2 } or { 1 , 3 } \{1,3\} { 1 , 3 } as a subset.
Figure 1. SCM over four variables across three environments. The hammer indicates the intervened variable. Environment 1 represents an observational setting, and in environments 2 and 3, variables X 1 X_1 X 1 and X 2 X_2 X 2 have received stochastic interventions.
Table 2. Population OLS coefficients and risks for different candidate predictor sets. Here V e : = Var ( Y e ) V_e := \operatorname{Var}(Y^e) V e := Var ( Y e ) .
Although the LSCR principle requires invariance only of the first two conditional moments, the SCM (1) may exhibit even stronger invariance. In particular, the conditional distribution Y e ∣ X S ⋆ e Y^e \mid X_{S^\star}^e Y e ∣ X S ⋆ e is the same for all environments e ∈ E e \in\mathcal{E} e ∈ E . Note that when Y e ∣ X S e ∼ N ( X e T β e , S , ν e , S ) Y^e \mid X_S^e \sim\mathcal{N}(X^{e\mathsf{T}}\beta^{e,S},\nu^{e,S}) Y e ∣ X S e ∼ N ( X e T β e , S , ν e , S ) , LSCR invariance is equivalent to invariance of the full conditional distribution. More generally, this equivalence continues to hold whenever the conditional distribution is fully characterized by E [ Y e ∣ X S e ] = X e T β e , S \mathbb{E}[Y^e \mid X_S^e] = X^{e\mathsf{T}}\beta^{e,S} E [ Y e ∣ X S e ] = X e T β e , S and Var ( Y e ∣ X S e ) = ν e , S \operatorname{Var}(Y^e \mid X_S^e) = \nu^{e,S} Var ( Y e ∣ X S e ) = ν e , S . In addition to being equivalent under these settings, LSCR invariance is considerably more amenable to convex mixed-integer optimization formulations, which enable optimality guarantees. We therefore focus on LSCR invariance throughout the paper.
Comparison to other invariance notions
We compare the LSCR invariance principle with other notions of invariance used in related work. First, the LSCR principle is essentially the same as that used in ICP [18 ] , but the two methods employ different testing procedures. For each candidate predictor set S S S , ICP fits a pooled linear regression and tests whether the residual means and variances are invariant across environments using t t t -tests and F F F -tests with Bonferroni correction. However, differences in environment specific regression coefficients may not manifest as differences in the means or variances of the pooled residuals. Consequently, for a subset S S S , ICP can fail to detect violations of invariance in the conditional distribution Y e ∣ X S e Y^e \mid X_S^e Y e ∣ X S e even at the population level, as illustrated in the following example.
Example 2. Consider two environments with the invariant causal mechanism Y e = X 1 e + ε e Y^e = X_1^e + \varepsilon^e Y e = X 1 e + ε e , e ∈ { 1 , 2 } e \in\{1,2\} e ∈ { 1 , 2 } . Suppose additionally, X 1 e ∼ N ( 0 , v e ) X_1^e \sim N(0,v_e) X 1 e ∼ N ( 0 , v e ) and X 2 e = X 1 e + U e X_2^e = X_1^e + U^e X 2 e = X 1 e + U e , U e ∼ N ( 0 , u e ) U^e \sim N(0,u_e) U e ∼ N ( 0 , u e ) with X 1 e X_1^e X 1 e , ε e \varepsilon^e ε e and U e U^e U e being mutually independent. Suppose v 1 = 1 v_1 = 1 v 1 = 1 , v 2 = 4 v_2 = 4 v 2 = 4 , u 1 = 4 u_1 = 4 u 1 = 4 and u 2 = 1 u_2 = 1 u 2 = 1 . Consider set S = { 2 } S = \{2\} S = { 2 } . It is straightforward to see that the distribution Y e ∣ X S e Y^e \mid X_S^e Y e ∣ X S e is not invariant across e e e , since β 1 , S ≠ β 2 , S \beta^{1,S} \ne\beta^{2,S} β 1 , S = β 2 , S . However, a straightforward calculation shows that the pooled residuals have identical distributions across the two environments. Consequently, the criterion used by ICP fails to reject S S S . Since ICP takes the intersection of all non-rejected sets, retaining S = { 2 } S = \{2\} S = { 2 } alongside { 1 } \{1\} { 1 } results in an empty intersection, thereby failing to identify the parent set { 1 } \{1\} { 1 } .
This example illustrates the limits of the tests ICP uses for quantifying invariance and helps explain the greater statistical power of our MIP testing procedure for moderate sample sizes, as shown in Figure 5 .
Figure 5. Performance of Example 5 as sample size n e n^{e} n e varies. Left panel: proportion of 200 trials in which the true plausible causal predictor set S ( E ) S(E) S ( E ) is recovered. Middle panel: coverage probability, defined as the proportion of 200 trials in which S ^ ( E ) ⊆ S ∗ \widehat{S}(E) \subseteq S^{*} S ( E ) ⊆ S ∗ , with the dashed line indicating the target level 0.95. Right panel: average runtime over 200 trials, shown on a logarithmic scale. We compare our proposed methods, MIP-test-2 and MIP-test-4 , with ICP [18 ] and WVM [15 ] . Solid lines show averages and shaded bands indicate ± 1 \pm1 ± 1 standard error. Additional experimental details are provided in Appendix G.5.
Second, the condition β = β e , S \beta= \beta^{e,S} β = β e , S for all e ∈ E e \in\mathcal{E} e ∈ E in Definition 1 is essentially equivalent to the invariance principle E [ ( Y e − β ⊤ X S e ) X S e ] = 0 \mathbb{E}[(Y^e-\beta^\top X_S^e)X_S^e]=0 E [( Y e − β ⊤ X S e ) X S e ] = 0 for all e ∈ E e \in\mathcal{E} e ∈ E , used in EILLS [5 ] . However, by additionally requiring invariance of ν e , S \nu^{e,S} ν e , S across environments, our proposed LSCR principle can reduce the collection of invariant predictor sets. As illustrated in Example 3, this additional requirement also weakens the identification condition. Intuitively, exploiting more information from the conditional distribution helps distinguish incorrect models through comparisons across environments.
Relatedly, IRM [1 ] seeks a representation under which the same prediction rule minimizes risk across environments. When the representation selects a predictor subset and the loss is squared error, this condition reduces to invariance of the corresponding population OLS coefficients. CoCo [27 ] similarly exploits coefficient invariance through first order optimality conditions. Like EILLS, both methods rely solely on coefficient invariance, potentially admitting more invariant predictor sets and requiring stronger conditions for causal parent identification.
Third, Causal Dantzig [21 ] exploits invariance of the moment E [ X e ( Y e − β ⊤ X e ) ] \mathbb{E}[X^e(Y^e-\beta^\top X^e)] E [ X e ( Y e − β ⊤ X e )] across environments, without requiring it to vanish. NegDRO [24 ] instead uses risk invariance, requiring E [ ( Y e − β ⊤ X e ) 2 ] = E [ ( Y f − β ⊤ X f ) 2 ] \mathbb{E}[(Y^e-\beta^\top X^e)^2] = \mathbb{E}[(Y^f-\beta^\top X^f)^2] E [( Y e − β ⊤ X e ) 2 ] = E [( Y f − β ⊤ X f ) 2 ] for all e , f ∈ E e,f \in\mathcal{E} e , f ∈ E . Both methods can accommodate hidden confounding, but require specific additive intervention mechanisms and a linear structural model for the entire system. Moreover, NegDRO does not explicitly perform variable selection.
Finally, Martinet et al. [15 ] consider a distributional invariance principle that requires the full distributions of prediction residuals to remain identical across environments, namely Y e − f ( X S e ) = d Y f − f ( X S f ) Y^{e}-f(X_{S}^{e}) \overset{d}{=} Y^{f}-f(X_{S}^{f}) Y e − f ( X S e ) = d Y f − f ( X S f ) for all e , f ∈ E e,f\in\mathcal{E} e , f ∈ E . Compared with LSCR, this principle exploits information beyond the first two moments and may therefore provide greater statistical power in certain settings. However, testing this invariance requires computationally demanding, generally nonconvex Wasserstein variance minimization, making variable selection challenging in practice. A related approach is BIP (Wu et al. (2025)), which models invariance of the full conditional distribution through a Bayesian framework, requiring p e ( y ∣ x S ) = p f ( y ∣ x S ) p_{e}(y\mid x_{S})=p_{f}(y\mid x_{S}) p e ( y ∣ x S ) = p f ( y ∣ x S ) for all e , f ∈ E e,f\in\mathcal{E} e , f ∈ E . BIP introduces latent binary indicators for variable selection and performs posterior inference over candidate predictor sets. However, its exact implementation requires exhaustive subset enumeration.
Invariant learning via convex mixed-integer programming
An invariance score and two invariant learning tasks
A key component of invariant learning is to quantify the extent to which a candidate model remains stable across environments. Motivated by the LSCR principle, we define the following invariance scores, in population and finite sample, for a given subset S ⊆ [ p ] S\subseteq[p] S ⊆ [ p ] :
I ( S ) : = ∑ e , f ∈ E e < f ω e , f [ ∥ β e , S − β f , S ∥ 2 2 + ρ ( ν e , S − ν f , S ) 2 ] , I ^ ( S ) : = ∑ e , f ∈ E e < f ω e , f [ ∥ β ^ e , S − β ^ f , S ∥ 2 2 + ρ ( ν ^ e , S − ν ^ f , S ) 2 ] . (4) \begin{aligned}
I(S)&:=\sum_{\substack{e,f\in\mathcal{E}\\e<f}}\omega^{e,f}\left[\left\|\beta^{e,S}-\beta^{f,S}\right\|_{2}^{2}+\rho\left(\nu^{e,S}-\nu^{f,S}\right)^{2}\right],\\
\widehat{I}(S)&:=\sum_{\substack{e,f\in\mathcal{E}\\e<f}}\omega^{e,f}\left[\left\|\widehat{\beta}^{e,S}-\widehat{\beta}^{f,S}\right\|_{2}^{2}+\rho\left(\widehat{\nu}^{e,S}-\widehat{\nu}^{f,S}\right)^{2}\right]. \tag*{(4)}
\end{aligned} I ( S ) I ( S ) := e , f ∈ E e < f ∑ ω e , f [ β e , S − β f , S 2 2 + ρ ( ν e , S − ν f , S ) 2 ] , := e , f ∈ E e < f ∑ ω e , f [ β e , S − β f , S 2 2 + ρ ( ν e , S − ν f , S ) 2 ] . (4)
The score I ( S ) I(S) I ( S ) measures violations of the two invariance conditions in Definition 1 through the weighted pairwise discrepancies between β e , S \beta^{e,S} β e , S and β f , S \beta^{f,S} β f , S , and between ν e , S \nu^{e,S} ν e , S and ν f , S \nu^{f,S} ν f , S , across environments. Here, the weights ω e , f > 0 \omega^{e,f}>0 ω e , f > 0 satisfy ∑ e , f ∈ E , e < f ω e , f = 1 \sum_{e,f\in\mathcal{E},e<f}\omega^{e,f}=1 ∑ e , f ∈ E , e < f ω e , f = 1 , and we typically choose ω e , f = n e n f / m \omega^{e,f}=n^{e}n^{f}/m ω e , f = n e n f / m where m = ∑ e ′ , f ′ ∈ E , e ′ < f ′ n e ′ n f ′ m=\sum_{e',f'\in\mathcal{E},e'<f'}n^{e'}n^{f'} m = ∑ e ′ , f ′ ∈ E , e ′ < f ′ n e ′ n f ′ . Moreover, ρ > 0 \rho>0 ρ > 0 is a regularization parameter that weighs different invariance conditions; we discuss its choices below. By construction, I ( ⋅ ) I(\cdot) I ( ⋅ ) is non-negative and attains the value of zero if and only if S ∈ S L S C R ( E ) S\in\mathcal{S}_{\mathrm{LSCR}}(\mathcal{E}) S ∈ S LSCR ( E ) . We therefore use I ( S ) I(S) I ( S ) to quantify the degree of invariance of S S S , with smaller values indicating greater invariance across environments.
The corresponding finite-sample invariance score, I ^ ( S ) \widehat{I}(S) I ( S ) , replaces β e , S \beta^{e,S} β e , S and ν e , S \nu^{e,S} ν e , S by their empirical counterparts β ^ e , S \widehat{\beta}^{e,S} β e , S and ν ^ e , S \widehat{\nu}^{e,S} ν e , S . This empirical invariance score can serve as a building block for a broad range of invariant learning tasks. We describe these tasks below.
(Task 1) Invariance-regularized prediction. Let R ^ ( β ) \widehat{R}(\beta) R ( β ) denote a weighted mean squared error computed over the pooled data, of a linear model with coefficient vector β ∈ R p \beta\in\mathbb{R}^{p} β ∈ R p . (We will describe its definition in Section 4). We consider the following optimization problem, which trades off predictive accuracy and invariance across environments while encouraging sparsity through regularization parameters λ n , γ ≥ 0 \lambda_{n},\gamma\geq0 λ n , γ ≥ 0 :
min S ⊆ [ p ] , β ∈ R p R ^ ( β ) + γ I ^ ( S ) + λ n ∣ S ∣ subject to: supp ( β ) ⊆ S . (5) \min_{\substack{S\subseteq[p],\,\beta\in\mathbb{R}^{p}}}\ \widehat{R}(\beta)+\gamma\widehat{I}(S)+\lambda_{n}|S|\quad\text{subject to: }\operatorname{supp}(\beta)\subseteq S. \tag*{(5)} S ⊆ [ p ] , β ∈ R p min R ( β ) + γ I ( S ) + λ n ∣ S ∣ subject to: supp ( β ) ⊆ S . (5)
Here, the prediction term R ^ ( β ) \widehat{R}(\beta) R ( β ) favors models with good predictive performance across environments, the invariance term I ^ ( S ) \widehat{I}(S) I ( S ) favors sets of predictors with strong invariance across environments, and the sparsity term ∣ S ∣ |S| ∣ S ∣ penalizes the inclusion of additional predictors. The coefficient vector β \beta β is linked to the invariance and sparsity terms only through the predictor set S S S . In particular, both β \beta β and the environment specific regression coefficients { β ^ e , S } e ∈ E \{\widehat{\beta}^{e,S}\}_{e\in\mathcal{E}} { β e , S } e ∈ E (that define I ^ ( S ) \widehat{I}(S) I ( S ) ) are restricted to the predictors in S S S . For any fixed S S S , the optimal β \beta β is simply the weighted least-squares estimate obtained by regressing the pooled response { Y e } e ∈ E \{Y^{e}\}_{e\in\mathcal{E}} { Y e } e ∈ E on the pooled predictors { X S e } e ∈ E \{X_{S}^{e}\}_{e\in\mathcal{E}} { X S e } e ∈ E .
Note that the prediction term R ^ ( β ) \widehat{R}(\beta) R ( β ) can help distinguish among multiple invariant sets. In particular, minimizing γ I ^ ( S ) + λ n ∣ S ∣ \gamma\widehat{I}(S)+\lambda_{n}|S| γ I ( S ) + λ n ∣ S ∣ alone can be less powerful in recovering the true set of causal parents S ∗ S^{*} S ∗ , since other subsets may also exhibit invariance across environments. Example 1 illustrates this point: when interventions occur only in environments e = 2 e=2 e = 2 and e = 3 e=3 e = 3 , both S 1 S_{1} S 1 and S 2 S_{2} S 2 are LSCR-invariant, but the causal parent set S 1 S_{1} S 1 achieves a smaller prediction error than S 2 S_2 S 2 . In Section 4, we establish conditions under which the estimator (5) recovers the causal parent set S ⋆ S^\star S ⋆ in finite samples.
Reformulation of optimization problems as convex mixed-integer programs
The formulations (5) and (6) do not lead to convenient optimization procedures. This difficulty arises for two reasons. First, for a given support S S S , the quantity β ^ e , S \widehat{\beta}^{e,S} β e , S in I ^ ( S ) \widehat{I}(S) I ( S ) is determined by S S S through the implicit OLS normal equations. Second, the squared discrepancy ( ν ^ e , S − ν ^ f , S ) 2 (\widehat{\nu}^{e,S}-\widehat{\nu}^{f,S})^2 ( ν e , S − ν f , S ) 2 in I ^ ( S ) \widehat{I}(S) I ( S ) generally induces a quartic and non-convex dependence on the regression coefficients, making it unsuitable for exact optimization procedures.
Fortunately, the invariance score can be cast within a mixed-integer programming framework, leading to convex MIP formulations for (5) and (6). We introduce two sets of new decision variables: (i) A set of binary variables { g j ∈ { 0 , 1 } } j = 1 p \{g_j \in\{0,1\}\}_{j=1}^p { g j ∈ { 0 , 1 } } j = 1 p , where g j = 1 g_j=1 g j = 1 whenever j ∈ S j \in S j ∈ S and g j = 0 g_j=0 g j = 0 otherwise; (ii) A set of continuous variables { β e ∈ R p } e ∈ E \{\beta^e \in\mathbb{R}^p\}_{e\in\mathcal{E}} { β e ∈ R p } e ∈ E that parameterize the coefficient for each environment. Then, I ^ ( S ) \widehat{I}(S) I ( S ) can be written as the solution to the following convex MIP:
I ^ ( S ) = I ^ ( g ) : = min { β e ∈ R p } e ∈ E ∑ e , f ∈ E ω e , f [ ∥ β e − β f ∥ 2 2 + ρ ( d e , f ( β e , β f ) ) 2 ] \widehat{I}(S)=\widehat{I}(g):=\min_{\{\beta^e\in\mathbb{R}^p\}_{e\in\mathcal{E}}}\sum_{e,f\in\mathcal{E}}\omega^{e,f}\left[\left\|\beta^e-\beta^f\right\|_2^2+\rho\left(d^{e,f}(\beta^e,\beta^f)\right)^2\right] I ( S ) = I ( g ) := { β e ∈ R p } e ∈ E min e , f ∈ E ∑ ω e , f [ β e − β f 2 2 + ρ ( d e , f ( β e , β f ) ) 2 ]
subject to:
− M n g j ≤ β j e ≤ M n g j , e ∈ E , j ∈ [ p ] , (7) -M_n g_j\leq\beta_j^e\leq M_n g_j,\qquad e\in\mathcal{E},\ j\in[p], \tag*{(7)} − M n g j ≤ β j e ≤ M n g j , e ∈ E , j ∈ [ p ] , (7)
− M ~ n ( 1 − g j ) ≤ ⟨ X j e , Y e − ∑ k ∈ [ p ] β k e X k e ⟩ n e ≤ M ~ n ( 1 − g j ) , e ∈ E , j ∈ [ p ] . (8) -\widetilde{M}_n(1-g_j)\leq\left\langle X_j^e,Y^e-\sum_{k\in[p]}\beta_k^eX_k^e\right\rangle_{n_e}\leq\widetilde{M}_n(1-g_j),\qquad e\in\mathcal{E},\ j\in[p]. \tag*{(8)} − M n ( 1 − g j ) ≤ ⟨ X j e , Y e − k ∈ [ p ] ∑ β k e X k e ⟩ n e ≤ M n ( 1 − g j ) , e ∈ E , j ∈ [ p ] . (8)
Here, the heterogeneity in the OLS risks is written as:
d e , f ( β e , β f ) : = ∥ Y e ∥ n e 2 − ∥ Y f ∥ n f 2 − ∑ j ∈ [ p ] β j e ⟨ X j e , Y e ⟩ n e + ∑ j ∈ [ p ] β j f ⟨ X j f , Y f ⟩ n f . d^{e,f}(\beta^e,\beta^f):=\left\|Y^e\right\|_{n_e}^2-\left\|Y^f\right\|_{n_f}^2-\sum_{j\in[p]}\beta_j^e\left\langle X_j^e,Y^e\right\rangle_{n_e}+\sum_{j\in[p]}\beta_j^f\left\langle X_j^f,Y^f\right\rangle_{n_f}. d e , f ( β e , β f ) := ∥ Y e ∥ n e 2 − Y f n f 2 − j ∈ [ p ] ∑ β j e ⟨ X j e , Y e ⟩ n e + j ∈ [ p ] ∑ β j f ⟨ X j f , Y f ⟩ n f .
See § 1.3 for the notations ∥ ⋅ ∥ n e \|\cdot\|_{n_e} ∥ ⋅ ∥ n e and ⟨ ⋅ , ⋅ ⟩ n e \langle\cdot,\cdot\rangle_{n_e} ⟨ ⋅ , ⋅ ⟩ n e . The constraints (7a) are so-called big-M M M constraints that link the binary variables g g g to the environment specific regression coefficients { β e } e ∈ E \{\beta^e\}_{e\in\mathcal{E}} { β e } e ∈ E . For sufficiently large M n M_n M n , constraints (7a) enforce β j e = 0 \beta_j^e=0 β j e = 0 whenever g j = 0 g_j=0 g j = 0 , and are otherwise nonbinding. Thus, supp ( β e ) ⊆ S \operatorname{supp}(\beta^e)\subseteq S supp ( β e ) ⊆ S . Similarly, for sufficiently large M ~ n \widetilde{M}_n M n , constraints (7b) enforce ⟨ X j e , Y e − ∑ k ∈ [ p ] β k e X k e ⟩ n e = 0 \left\langle X_j^e,Y^e-\sum_{k\in[p]}\beta_k^eX_k^e\right\rangle_{n_e}=0 ⟨ X j e , Y e − ∑ k ∈ [ p ] β k e X k e ⟩ n e = 0 whenever g j = 1 g_j=1 g j = 1 , and are nonbinding when g j = 0 g_j=0 g j = 0 . Together with (7a), these constraints impose the normal equations for the regression of Y e Y^e Y e on X S e X_S^e X S e , so that β S e \beta_S^e β S e is the corresponding OLS coefficient. We discuss choices of M n M_n M n and M ~ n \widetilde{M}_n M n below.
Finally, the objective function reformulates the LSCR invariance score as a convex function of { β e } e ∈ E \{\beta^e\}_{e\in\mathcal{E}} { β e } e ∈ E . Indeed, for an OLS coefficient β e \beta^e β e , we have
ν ^ e , S = ∥ Y e ∥ n e 2 − 2 ∑ j ∈ [ p ] β j e ⟨ X j e , Y e ⟩ n e + ∑ j , k ∈ [ p ] β j e β k e ⟨ X j e , X k e ⟩ n e = ∥ Y e ∥ n e 2 − ∑ j ∈ [ p ] β j e ⟨ X j e , Y e ⟩ n e . \widehat{\nu}^{e,S}=\left\|Y^e\right\|_{n_e}^2-2\sum_{j\in[p]}\beta_j^e\left\langle X_j^e,Y^e\right\rangle_{n_e}+\sum_{j,k\in[p]}\beta_j^e\beta_k^e\left\langle X_j^e,X_k^e\right\rangle_{n_e}=\left\|Y^e\right\|_{n_e}^2-\sum_{j\in[p]}\beta_j^e\left\langle X_j^e,Y^e\right\rangle_{n_e}. ν e , S = ∥ Y e ∥ n e 2 − 2 j ∈ [ p ] ∑ β j e ⟨ X j e , Y e ⟩ n e + j , k ∈ [ p ] ∑ β j e β k e ⟨ X j e , X k e ⟩ n e = ∥ Y e ∥ n e 2 − j ∈ [ p ] ∑ β j e ⟨ X j e , Y e ⟩ n e .
The last equality follows from the OLS normal equations.
Interestingly, for a fixed subset S S S (or its associated binary variables { g j } j = 1 p \{g_j\}_{j=1}^p { g j } j = 1 p ), the feasible region of (7) is a singleton, provided that the OLS solution is unique in each environment. Indeed, as discussed in § 2.2 , the support S S S uniquely determines the corresponding environment specific OLS coefficients { β e , S } e ∈ E \{\beta^{e,S}\}_{e\in\mathcal{E}} { β e , S } e ∈ E . Consequently, the minimization over { β e } e ∈ E \{\beta^e\}_{e\in\mathcal{E}} { β e } e ∈ E may be omitted when evaluating the objective for a fixed g g g . The key benefit of the reformulation (7) is that our two invariant learning tasks can be cast as the following convex quadratic MIPs:
Task 1: min g ∈ { 0 , 1 } p , β ∈ R p R ^ ( β ) + γ I ^ ( g ) + λ n ∑ j ∈ [ p ] g j subject to: − M n g j ≤ β j ≤ M n g j ∀ j ∈ [ p ] , (9) \text{Task 1:}\qquad\min_{g\in\{0,1\}^p,\ \beta\in\mathbb{R}^p}\ \widehat{R}(\beta)+\gamma\widehat{I}(g)+\lambda_n\sum_{j\in[p]}g_j\qquad\text{subject to: }-M_ng_j\leq\beta_j\leq M_ng_j\quad\forall j\in[p], \tag*{(9)} Task 1: g ∈ { 0 , 1 } p , β ∈ R p min R ( β ) + γ I ( g ) + λ n j ∈ [ p ] ∑ g j subject to: − M n g j ≤ β j ≤ M n g j ∀ j ∈ [ p ] , (9)
Task 2: min g ∈ { 0 , 1 } p I ^ ( g ) subject to: g j 0 = 0. (10) \text{Task 2:}\qquad\min_{g\in\{0,1\}^p}\ \widehat{I}(g)\qquad\text{subject to: }g_{j_0}=0. \tag*{(10)} Task 2: g ∈ { 0 , 1 } p min I ( g ) subject to: g j 0 = 0. (10)
Note that the big-M M M constraints in (8) ensure that the support of β \beta β is encoded by g g g : whenever g j = 0 g_j=0 g j = 0 , β j = 0 \beta_j=0 β j = 0 . As discussed next, the formulations (8) and (9) allow us to certify global optimality, stop early with explicit guarantees, and develop scalable heuristics that produce solutions of high quality.
Remark 4. Optimization problems (8) and (9) depend on the parameters M n M_n M n and M ~ n \widetilde{M}_n M n . One can use heuristic approaches Park and Kлабjan [17 ] ; ), Xu et al. (2025) to select them. Specifically, one can first solve (8) or (9)
after relaxing the binary variables to [ 0 , 1 ] [0,1] [ 0 , 1 ] , yielding solutions β ^ e \widehat{\beta}^{e} β e and β ^ \widehat{\beta} β , and then set M n = 2 max e ∈ E ∥ β ^ e ∥ ∞ M_n = 2\max_{e\in\mathcal{E}}\|\widehat{\beta}^{e}\|_{\infty} M n = 2 max e ∈ E ∥ β e ∥ ∞ and M ~ n = 2 ∥ β ^ ∥ ∞ \widetilde{M}_n = 2\|\widehat{\beta}\|_{\infty} M n = 2∥ β ∥ ∞ . In practice, however, we avoid specifying M n M_n M n and M ~ n \widetilde{M}_n M n by using Gurobi’s indicator constraints. These allow us to directly enforce β j e = 0 \beta_j^e = 0 β j e = 0 for all e ∈ E e\in\mathcal{E} e ∈ E and β j = 0 \beta_j = 0 β j = 0 whenever g j = 0 g_j = 0 g j = 0 , leaving Gurobi to handle the corresponding mixed-integer formulation internally.
Exact and scalable heuristic solvers
We discuss two approaches for solving the convex MIPs (8) and (9).
An exact solver. One option is to use the mixed-integer optimization solver Gurobi, which can certify global optimality up to prescribed numerical tolerances. The solver uses a branch-and-bound framework. For both formulations, relaxing the binary constraints g j ∈ { 0 , 1 } g_j\in\{0,1\} g j ∈ { 0 , 1 } to g j ∈ [ 0 , 1 ] g_j\in[0,1] g j ∈ [ 0 , 1 ] yields a convex quadratic program whose optimal value provides a lower bound on the corresponding mixed-integer problem. These lower bounds are used to prune the search tree: if the bound at a node is no smaller than the objective value of the best feasible solution found so far, then that node and all of its descendants can be discarded. Consequently, the solver can certify optimality without enumerating all 2 p 2^p 2 p possible support configurations.
Lower and upper bounds also provide a certificate of solution quality. The best feasible solution, referred to as the incumbent , provides an upper bound on the optimal objective value, whereas the convex relaxations provide a global lower bound. Their difference, referred to as the optimality gap , quantifies the suboptimality of the incumbent solution. In particular, the incumbent is globally optimal when this gap is zero, up to numerical tolerances.
Several practical strategies can improve the performance of this exact approach. A high-quality warm start can provide a strong initial upper bound and thereby accelerate pruning; such a warm start can be obtained using the coordinate descent procedure described next. One may also configure the solver to prioritize improvements in the global lower bound, for example, by setting MIPFocus=2 in Gurobi. Finally, to reduce computation time, an early-stopping tolerance τ e a r l y ≥ 0 \tau^{\mathrm{early}}\geq0 τ early ≥ 0 may be imposed on the optimality gap. If the algorithm terminates with a gap no greater than τ e a r l y \tau^{\mathrm{early}} τ early , the returned solution is guaranteed to be within τ e a r l y \tau^{\mathrm{early}} τ early of the global optimum.
An approximate solver using coordinate descent. Coordinate descent provides a substantially more scalable approach for obtaining high-quality feasible solutions. We observe that, for any fixed g g g , the objective values in (8) and (9) can be evaluated in closed form. This motivates a coordinate descent procedure that initializes from a support g g g and sequentially updates one coordinate at a time. For each j ∈ [ p ] j \in[p] j ∈ [ p ] , if g j = 1 g_j = 1 g j = 1 , the algorithm removes j j j whenever the updated support that sets g j = 0 g_j = 0 g j = 0 yields a smaller objective value; otherwise, the support is left unchanged. Similarly, if g j = 0 g_j = 0 g j = 0 , the algorithm adds j j j whenever the updated support that sets g j = 1 g_j = 1 g j = 1 improves the objective. The procedure cycles through the coordinates until no update to a single coordinate decreases the objective. We also employ several practical strategies to accelerate the algorithm, including rank one updates for objective evaluation and local combinatorial optimization. Further details are provided in Appendix A.2.
Finally, although the coordinate descent procedure is not an exact solver, its solution can be used to initialize the vanilla MIP solver. This provides an initial upper bound of high quality, allowing the MIP solver to further improve the solution and, through its lower bound, either certify global optimality or quantify the remaining optimality gap.
Figure 2 compares the coordinate descent heuristic (CD) with the exact MIP solver, both with and without a warm start. The MIP solver with a warm start is initialized by the solution obtained from coordinate descent, whereas the vanilla MIP is solved without an initial incumbent. All three methods yield essentially identical estimation performance across the considered dimensions. However, coordinate descent is substantially faster and scales much more favorably with p p p . The computational cost of both MIP variants increases rapidly with p p p , although using the CD solution as a warm start yields a modest runtime improvement over the vanilla MIP.
Figure 2. Comparison of coefficient estimation error and runtime for Task 1 among heuristic coordinate descent (CD), MIP, and MIP with a CD warm start. Left panel: coefficient estimation error ∥ β ^ − β ∗ ∥ 2 \|\widehat{\beta}-\beta^*\|_2 ∥ β − β ∗ ∥ 2 across dimensions p p p . Right panel: Runtime across dimensions p p p , shown on a logarithmic scale. Each box summarizes results from 100 independent simulation trials. Implementation details are provided in Appendix G.1.
Task 1: invariance-regularized predictions via convex mixed-integer programming
This section studies the estimator (8) that obtains prediction with invariance regularization. Section 4.1 analyzes the estimator in population and presents conditions for which this estimator identifies the causal parents S ∗ S^* S ∗ and the coefficient vector β ∗ \beta^* β ∗ . Section 4.2 provides high-dimensional guarantees that allow the number of features to grow with the sample size. Throughout, we set the predictor term R ^ ( β ) \widehat{R}(\beta) R ( β ) and its population analogue R ( β ) R(\beta) R ( β ) as:
R ^ ( β ) : = ∑ e ∈ E ω e ∥ Y e − X e β ∥ n e 2 and R ( β ) : = ∑ e ∈ E ω e E [ ( Y e − X e β ) 2 ] . \widehat{R}(\beta) := \sum_{e \in\mathcal{E}} \omega^e \lVert Y^e - X^e\beta\rVert_{n_e}^2 \quad\text{and} \quad R(\beta) := \sum_{e \in\mathcal{E}} \omega^e \mathbb{E}\left[(Y^e - X^e\beta)^2\right]. R ( β ) := e ∈ E ∑ ω e ∥ Y e − X e β ∥ n e 2 and R ( β ) := e ∈ E ∑ ω e E [ ( Y e − X e β ) 2 ] .
Thus R ^ ( β ) \widehat{R}(\beta) R ( β ) is the weighted mean squared error, and R ( β ) R(\beta) R ( β ) is its corresponding population counterpart. The weights ω e > 0 \omega^e > 0 ω e > 0 satisfy ∑ e ∈ E ω e = 1 \sum_{e \in\mathcal{E}} \omega^e = 1 ∑ e ∈ E ω e = 1 . In the finite-sample version, we choose ω e = n e / ∑ e ∈ E n e \omega^e = n^e / \sum_{e \in\mathcal{E}} n^e ω e = n e / ∑ e ∈ E n e for all e ∈ E e \in\mathcal{E} e ∈ E .
Identifiability results in population
The population analog of estimator (8) is given by:
( β ~ , S ~ ) ∈ arg min β ∈ R p , S ⊆ [ p ] ∑ e ∈ E ω e E [ ∣ Y e − β ⊤ X e ∣ 2 ] + γ I ( S ) subject to: supp ( β ) ⊆ S . (11) (\widetilde{\beta}, \widetilde{S}) \in\underset{\beta\in\mathbb{R}^p,S \subseteq[p]}{\arg\min}\ \sum_{e \in\mathcal{E}} \omega^e \mathbb{E}\left[\left|Y^e - \beta^\top X^e\right|^2\right] + \gamma I(S) \quad\text{subject to: } \operatorname{supp}(\beta) \subseteq S. \tag*{(11)} ( β , S ) ∈ β ∈ R p , S ⊆ [ p ] arg min e ∈ E ∑ ω e E [ Y e − β ⊤ X e 2 ] + γ I ( S ) subject to: supp ( β ) ⊆ S . (11)
Here, (10) searches over a subset S S S of covariates and a regression coefficient β \beta β with support S S S . Its objective is a weighted MSE, computed after pooling data from all environments, plus a regularization term I ( S ) I(S) I ( S ) . Conceptually, the minimization can be decomposed into two stages: first optimizing over the support S S S , and then over the coefficient vector β \beta β . The regularization term encourages the selection of a support with a small invariance score, ideally one that is exactly LSCR-invariant. Once the support solution S ~ \widetilde{S} S is determined, the subsequent minimisation over β \beta β involves only the MSE term and therefore yields a weighted OLS estimate restricted to S ~ \widetilde{S} S . If S ~ ∈ S L S C R ( E ) \widetilde{S} \in\mathcal{S}_{\mathrm{LSCR}}(\mathcal{E}) S ∈ S LSCR ( E ) , then by (3), the solution β ~ \widetilde{\beta} β coincides with the environment specific OLS coefficients β e , S ~ \beta^{e,\widetilde{S}} β e , S for all e ∈ E e \in\mathcal{E} e ∈ E .
When the regularization parameters γ \gamma γ and ρ \rho ρ are sufficiently large (which we formalize in Lemma 1), it is apparent that S ~ \widetilde{S} S must satisfy S ~ ∈ S L S C R ( E ) \widetilde{S} \in\mathcal{S}_{\mathrm{LSCR}}(\mathcal{E}) S ∈ S LSCR ( E ) . In this regime, the solution to (10) is equivalent to
arg min β ∈ R p , S ⊆ [ p ] ∑ e ∈ E ω e E [ ∣ Y e − β ⊤ X e ∣ 2 ] subject to: supp ( β ) ⊆ S , I ( S ) = 0. \underset{\beta\in\mathbb{R}^p,S \subseteq[p]}{\arg\min}\ \sum_{e \in\mathcal{E}} \omega^e \mathbb{E}\left[\left|Y^e - \beta^\top X^e\right|^2\right] \quad\text{subject to: } \operatorname{supp}(\beta) \subseteq S,\ I(S)=0. β ∈ R p , S ⊆ [ p ] arg min e ∈ E ∑ ω e E [ Y e − β ⊤ X e 2 ] subject to: supp ( β ) ⊆ S , I ( S ) = 0.
Hence, the problem reduces to selecting the most predictive model among all LSCR-invariant models. As alluded to in Section 3.1, the MSE objective can further distinguish the causal support from other invariant models, particularly when the interventions are not sufficiently heterogeneous. We next present an identification condition under which this distinction is possible, together with the corresponding identification result.
Assumption 1 (Identification). Let G : = { j ∈ [ p ] : ∑ e ∈ E ω e E [ X j e ε e ] ≠ 0 } G := \{j \in[p] : \sum_{e \in\mathcal{E}} \omega^{e} \mathbb{E}[X_{j}^{e} \varepsilon^{e}] \ne0\} G := { j ∈ [ p ] : ∑ e ∈ E ω e E [ X j e ε e ] = 0 } . For any S ⊆ [ p ] S \subseteq[p] S ⊆ [ p ] satisfying S ∩ G ≠ ∅ S \cap G \ne\varnothing S ∩ G = ∅ , there exists some e , f ∈ E e, f \in\mathcal{E} e , f ∈ E such that β e , S ≠ β f , S \beta^{e,S} \ne\beta^{f,S} β e , S = β f , S or ν e , S ≠ ν f , S \nu^{e,S} \ne\nu^{f,S} ν e , S = ν f , S .
In essence, this identification condition requires sufficient environmental heterogeneity to distinguish any model containing a variable in G G G . Variables in G G G are linearly spurious in the sense that they can improve prediction of the noise and hence the prediction risk, despite not belonging to the true causal model. Under the interventions specified in Assumption 1, any model S S S with S ∩ G ≠ ∅ S \cap G \ne\varnothing S ∩ G = ∅ would yield I ( S ) ≠ 0 I(S) \ne0 I ( S ) = 0 , and hence is very likely to be detected.
Assumption 1 is closely related to the identification condition (Condition 4.5) in Fan et al. [5 ] . Indeed, Fan et al. [5 ] require that for any S ⊆ [ p ] S \subseteq[p] S ⊆ [ p ] satisfying S ∩ G ≠ ∅ S \cap G \ne\varnothing S ∩ G = ∅ , there exists some e , f ∈ E e, f \in\mathcal{E} e , f ∈ E such that β e , S ≠ β f , S \beta^{e,S} \ne\beta^{f,S} β e , S = β f , S . This similarity arises because both methods adopt an objective of the form MSE plus an invariance penalty, and their penalty essentially sums ∥ β e , S − β f , S ∥ 2 2 \lVert\beta^{e,S} - \beta^{f,S} \rVert_{2}^{2} ∥ β e , S − β f , S ∥ 2 2 across e , f ∈ E e, f \in\mathcal{E} e , f ∈ E . However, the penalty in Fan et al. [5 ] appears more challenging to formulate as a scalable convex mixed-integer program. In addition, our stronger invariance principle, which also exploits heterogeneity in the OLS risks, yields a weaker identification condition and hence greater power. Example 3 illustrates this advantage.
Assumption 1 can also be compared with the sufficient identification conditions in Peters et al. [18 ] . Their Theorem 2 establishes identifiability under several specific intervention schemes, such as single-variable do or noise interventions on every variable other than the target. In contrast, Assumption 1 does not impose any restrictions on the intervention mechanisms. Moreover, our assumption does not require interventions on variables that are independent of ε e \varepsilon^{e} ε e , such as ancestors of Y e Y^{e} Y e in a standard SEM. In this sense, Assumption 1 is less restrictive.
Example 3. Consider two environments e ∈ { 1 , 2 } e \in\{1, 2\} e ∈ { 1 , 2 } and the linear SCM
X 0 e ∼ N ( 0 , 1 ) , Y e = X 0 e + ε e , X 1 e = a e Y e + η e . X_{0}^{e} \sim\mathcal{N}(0,1), \qquad Y^{e} = X_{0}^{e} + \varepsilon^{e}, \qquad X_{1}^{e} = a_{e}Y^{e} + \eta^{e}. X 0 e ∼ N ( 0 , 1 ) , Y e = X 0 e + ε e , X 1 e = a e Y e + η e . where ε e ∼ N ( 0 , 1 ) \varepsilon^{e} \sim\mathcal{N}(0,1) ε e ∼ N ( 0 , 1 ) and η e ∼ N ( 0 , ν e ) \eta^{e} \sim\mathcal{N}(0,\nu_{e}) η e ∼ N ( 0 , ν e ) . We additionally generate eight independent noise covariates Z 1 e , … , Z 8 e Z_{1}^{e}, \ldots, Z_{8}^{e} Z 1 e , … , Z 8 e . All exogenous variables ε e \varepsilon^{e} ε e , η e \eta^{e} η e and Z e Z^{e} Z e are mutually independent. We set ( a 1 , a 2 ) = ( 1 , 1 / 2 ) (a_{1},a_{2}) = (1,1/2) ( a 1 , a 2 ) = ( 1 , 1/2 ) and ( ν 1 , ν 2 ) = ( 2 , 3 / 2 ) (\nu_{1},\nu_{2}) = (2,3/2) ( ν 1 , ν 2 ) = ( 2 , 3/2 ) . The causal support is S ⋆ = { 0 } S^{\star} = \{0\} S ⋆ = { 0 } while G = { 1 } G = \{1\} G = { 1 } . For the spurious support S = { 1 } S = \{1\} S = { 1 } , the population OLS coefficient and residual risk are β e , S = 2 a e / ( 2 a e 2 + ν e ) \beta^{e,S} = 2a_{e}/(2a_{e}^{2}+\nu^{e}) β e , S = 2 a e / ( 2 a e 2 + ν e ) and ν e , S = 2 − 4 a e 2 / ( 2 a e 2 + ν e ) \nu^{e,S} = 2 - 4a_{e}^{2}/(2a_{e}^{2}+\nu^{e}) ν e , S = 2 − 4 a e 2 / ( 2 a e 2 + ν e ) . Under the above parameter choices, we obtain β 1 , S = β 2 , S = 1 / 2 \beta^{1,S} = \beta^{2,S} = 1/2 β 1 , S = β 2 , S = 1/2 , whereas ν 1 , S ≠ ν 2 , S \nu^{1,S} \ne\nu^{2,S} ν 1 , S = ν 2 , S . As a result, our identification condition holds but that of Fan et al. [5 ] is violated.
For each trial, we generate 100 independent datasets with n e = 500 n^{e} = 500 n e = 500 for both environments. We compute, for each method, the proportion of exact support recovery and the proportion of datasets in which the spurious feature X 1 X_{1} X 1 is selected. Figure 3 shows these proportions averaged over 50 trials. The results demonstrate the limited identification power of EILLS (Fan et al., [5 ] ), which frequently includes the spurious feature X 1 X_{1} X 1 . We also report the performance of ICP (Peters et al., [18 ] ), Causal Dantzig (Rothenhäusler et al., [21 ] ), and BIP (Wu et al., 2025), which also exhibit limited power in this example.
Figure 3. Performance in the perturbed version of Example 3. To avoid relying on a parameter configuration that occurs with probability zero, we set ( a 1 , a 2 ) = ( 1 , 1 / 2 ) (a_1,a_2)=(1,1/2) ( a 1 , a 2 ) = ( 1 , 1/2 ) and ( ν 1 , ν 2 ) = ( 2 , 3 / 2 ) (\nu_1,\nu_2)=(2,3/2) ( ν 1 , ν 2 ) = ( 2 , 3/2 ) , each with an additional small Gaussian perturbation. Left panel: proportion of exact support recovery. Right panel: proportion of including the spurious feature { 1 } \{1\} { 1 } . Each box summarizes results from 50 independent simulation trials. Additional competing methods and full experimental details are provided in Appendix G.2.
Lemma 1. Suppose Assumption 1 holds. Also assume that there exists κ L > 0 \kappa_{L} > 0 κ L > 0 such that E [ X e X e T ] ⪰ κ L I p \mathbb{E}[X^{e}X^{e\mathsf{T}}] \succeq\kappa_{L}I_{p} E [ X e X e T ] ⪰ κ L I p , ∀ e ∈ E \forall e \in\mathcal{E} ∀ e ∈ E . Denote
γ ⋆ : = sup S : S ∩ G ≠ ∅ κ L − 1 ⋅ ∥ ∑ e ∈ E ω e E [ ε e X S e ] ∥ 2 2 I 1 ( S ) + I 2 ( S ) . \gamma^{\star} := \sup_{S:S\cap G\ne\varnothing}\kappa_{L}^{-1}\cdot\frac{\left\lVert\sum_{e\in\mathcal{E}}\omega^{e}\mathbb{E}[\varepsilon^{e}X_{S}^{e}]\right\rVert_{2}^{2}}{I_{1}(S)+I_{2}(S)}. γ ⋆ := S : S ∩ G = ∅ sup κ L − 1 ⋅ I 1 ( S ) + I 2 ( S ) ∑ e ∈ E ω e E [ ε e X S e ] 2 2 . where I 1 ( S ) : = ∑ e < f ω e , f ∥ β e , S − β f , S ∥ 2 2 I_{1}(S) := \sum_{e<f}\omega^{e,f}\lVert\beta^{e,S}-\beta^{f,S}\rVert_{2}^{2} I 1 ( S ) := ∑ e < f ω e , f ∥ β e , S − β f , S ∥ 2 2 and I 2 ( S ) : = ∑ e < f ω e , f ( ν e , S − ν f , S ) 2 I_{2}(S) := \sum_{e<f}\omega^{e,f}(\nu^{e,S}-\nu^{f,S})^{2} I 2 ( S ) := ∑ e < f ω e , f ( ν e , S − ν f , S ) 2 . When γ ∧ γ P ≥ 2 γ ⋆ + ϵ \gamma\wedge\gamma_{P} \ge2\gamma^{\star} + \epsilon γ ∧ γ P ≥ 2 γ ⋆ + ϵ for some ϵ > 0 \epsilon> 0 ϵ > 0 , every minimizer of (11) has coefficient vector β ⋆ \beta^{\star} β ⋆ . Moreover, S ⋆ S^{\star} S ⋆ is the unique smallest selected set among the minimizers.
Lemma 1 presents the identification result of procedure (10). The proof is provided in Appendix § 16 .§ 1 . Under Assumption 1, the score I 1 ( S ) + I 2 ( S ) I_{1}(S)+I_{2}(S) I 1 ( S ) + I 2 ( S ) must be positive. Intuitively, for models S S S satisfying S ∩ G ≠ ∅ S \cap G \ne\varnothing S ∩ G = ∅ , weaker correlations between X S e X_{S}^{e} X S e and ε e \varepsilon^{e} ε e , together with larger invariance scores, reduce the penalty strength required to separate these models from the causal model.
Finite-sample guarantees
We establish statistical guarantees for estimates obtained from (8):
( β ^ , S ^ ) ∈ arg min β ∈ R p , S ⊆ [ p ] ∑ e ∈ E ω e ∥ Y e − X e β ∥ n e 2 + γ ∑ e < f ω e , f [ ∥ β ^ e , S − β ^ f , S ∥ 2 2 + ρ ( ν ^ e , S − ν ^ f , S ) 2 ] + λ n ∣ S ∣ , (\widehat{\beta},\widehat{S})\in\mathop{\arg\min}_{\beta\in\mathbb{R}^{p},\,S\subseteq[p]}\sum_{e\in\mathcal{E}}\omega^{e}\left\|Y^{e}-X^{e}\beta\right\|_{n_{e}}^{2}+\gamma\sum_{e<f}\omega^{e,f}\left[\left\|\widehat{\beta}^{e,S}-\widehat{\beta}^{f,S}\right\|_{2}^{2}+\rho\left(\widehat{\nu}^{e,S}-\widehat{\nu}^{f,S}\right)^{2}\right]+\lambda_{n}|S|, ( β , S ) ∈ arg min β ∈ R p , S ⊆ [ p ] e ∈ E ∑ ω e ∥ Y e − X e β ∥ n e 2 + γ e < f ∑ ω e , f [ β e , S − β f , S 2 2 + ρ ( ν e , S − ν f , S ) 2 ] + λ n ∣ S ∣ ,
subject to: supp ( β ) ⊆ S , \operatorname{supp}(\beta)\subseteq S, supp ( β ) ⊆ S ,
∣ S ∣ ≤ s n . |S|\leq s_{n}. ∣ S ∣ ≤ s n .
Here we further impose the sparsity constraint ∣ S ∣ ≤ s n |S|\leq s_{n} ∣ S ∣ ≤ s n , which restricts the number of selected features to at most s n s_{n} s n . The required conditions on s n s_{n} s n are specified later. The corresponding detailed MIP formulation is provided in Appendix A.1. To reduce computation time, the procedure may be terminated early once the optimality gap falls below a user specified tolerance τ e a r l y > 0 \tau_{\mathrm{early}}>0 τ early > 0 . We denote the corresponding feasible solution by β ^ e a r l y \widehat{\beta}^{\mathrm{early}} β early and S ^ e a r l y \widehat{S}^{\mathrm{early}} S early .
Assumption 2 (Covariate and noise regularity). There exist constants 0 < κ L ≤ κ U < ∞ 0<\kappa_{L}\leq\kappa_{U}<\infty 0 < κ L ≤ κ U < ∞ , σ X ≥ 1 \sigma_{X}\geq1 σ X ≥ 1 and σ ε > 0 \sigma_{\varepsilon}>0 σ ε > 0 independent of n n n , p p p , and e e e , such that, for every e ∈ E e\in\mathcal{E} e ∈ E , the following conditions hold:
(1) The population covariance matrix Σ e : = E [ X e X e ⊤ ] \Sigma^{e}:=\mathbb{E}[X^{e}X^{e\top}] Σ e := E [ X e X e ⊤ ] satisfies κ L I p ⪯ Σ e ⪯ κ U I p \kappa_{L}I_{p}\preceq\Sigma^{e}\preceq\kappa_{U}I_{p} κ L I p ⪯ Σ e ⪯ κ U I p .
(2) The standardized covariate vector is sub-Gaussian: for every v ∈ R p v\in\mathbb{R}^{p} v ∈ R p ,
E [ exp { v ⊤ ( Σ e ) − 1 / 2 X e } ] ≤ exp ( σ X 2 ∥ v ∥ 2 / 2 ) . \mathbb{E}\left[\exp\left\{v^{\top}(\Sigma^{e})^{-1/2}X^{e}\right\}\right]\leq\exp\left(\sigma_{X}^{2}\|v\|^{2}/2\right). E [ exp { v ⊤ ( Σ e ) − 1/2 X e } ] ≤ exp ( σ X 2 ∥ v ∥ 2 /2 ) .
(3) The noise variable ε e \varepsilon^{e} ε e is sub-Gaussian: for every λ ∈ R \lambda\in\mathbb{R} λ ∈ R ,
E [ exp { λ ε e } ] ≤ exp ( λ 2 σ ε 2 / 2 ) . \mathbb{E}\left[\exp\{\lambda\varepsilon^{e}\}\right]\leq\exp\left(\lambda^{2}\sigma_{\varepsilon}^{2}/2\right). E [ exp { λ ε e } ] ≤ exp ( λ 2 σ ε 2 /2 ) .
Assumption 3 (Model sparsity and moderate magnitude of coefficient). The quantity s n s_{n} s n that constrains the number of selected features satisfies s ⋆ ≤ s n s^{\star}\leq s_{n} s ⋆ ≤ s n . Additionally, the true coefficients satisfy
∥ β ⋆ ∥ 2 2 ≲ ( min e ∈ E n e ) 1 − δ / 2 ( s n log p + log ∣ E ∣ ) 1 − δ / 2 \|\beta^{\star}\|_{2}^{2}\lesssim\frac{\left(\min_{e\in\mathcal{E}}n^{e}\right)^{1-\delta/2}}{\left(s_{n}\log p+\log|\mathcal{E}|\right)^{1-\delta/2}} ∥ β ⋆ ∥ 2 2 ≲ ( s n log p + log ∣ E ∣ ) 1 − δ /2 ( min e ∈ E n e ) 1 − δ /2
for some δ ∈ ( 0 , 2 ) \delta\in(0,2) δ ∈ ( 0 , 2 ) .
The conditions in Assumption 2 are adapted from Conditions 4.2–4.4 of Fan et al. [5 ] , which impose uniform covariance and tail bounds across environments. They are very standard in high-dimensional linear regression analysis. From Assumption 3, the true model size, upper bounded by s n s_n s n , may grow with the sample size. The added constraint ∑ j g j ≤ s n \sum_j g_j \le s_n ∑ j g j ≤ s n ensures that the estimated model adheres to the same sparsity level. Similarly, the magnitude of coefficients ∥ β ∗ ∥ 2 2 \lVert\beta^*\rVert_2^2 ∥ β ∗ ∥ 2 2 is allowed to grow with the sample size, but at a rate not too fast. This condition on ∥ β ∗ ∥ 2 2 \lVert\beta^*\rVert_2^2 ∥ β ∗ ∥ 2 2 is required only when ρ ≠ 0 \rho\ne0 ρ = 0 , to control the worst case behavior of the term ( ν ^ e , S − ν ^ f , S ) 2 (\widehat{\nu}^{e,S}-\widehat{\nu}^{f,S})^2 ( ν e , S − ν f , S ) 2 . We may assume that all environments have the same scale of samples: n e ≍ min f ∈ E n f n^e \asymp\min_{f\in\mathcal{E}} n^f n e ≍ min f ∈ E n f , ∀ e ∈ E \forall e\in\mathcal{E} ∀ e ∈ E . Then some example choices for these quantities are: s n ≍ n 0 1 / 5 s_n \asymp n_0^{1/5} s n ≍ n 0 1/5 , ∣ E ∣ ≲ p |\mathcal{E}|\lesssim p ∣ E ∣ ≲ p , δ = 1 / 2 \delta=1/2 δ = 1/2 , and ∥ β ∗ ∥ 2 2 ≲ n 0 3 / 5 / ( log p ) 3 / 4 \lVert\beta^*\rVert_2^2\lesssim n_0^{3/5}/(\log p)^{3/4} ∥ β ∗ ∥ 2 2 ≲ n 0 3/5 / ( log p ) 3/4 .
Theorem 2. Suppose conditions in Assumptions 1–3 and Lemma 1 hold. Suppose the sample size in each environment is sufficiently large: min e ∈ E n e ≳ s n log p + log ∣ E ∣ \min_{e\in\mathcal{E}}n^e \gtrsim s_n\log p+\log|\mathcal{E}| min e ∈ E n e ≳ s n log p + log ∣ E ∣ . Then, the following are true with probability at least 1 − 7 / p 1-7/p 1 − 7/ p :
For the optimal solution β ^ \widehat{\beta} β :
∥ β ^ − β ∗ ∥ 2 2 ≲ λ n s n + s n log p + log ∣ E ∣ min e ∈ E n e . \lVert\widehat{\beta}-\beta^*\rVert_2^2 \lesssim\lambda_n s_n+\frac{s_n\log p+\log|\mathcal{E}|}{\min_{e\in\mathcal{E}}n^e}. ∥ β − β ∗ ∥ 2 2 ≲ λ n s n + min e ∈ E n e s n log p + log ∣ E ∣ . For the early-stopping solution with threshold τ e a r l y \tau^{\mathrm{early}} τ early :
∥ β ^ e a r l y − β ∗ ∥ 2 2 ≲ λ n s n + s n log p + log ∣ E ∣ min e ∈ E n e + τ e a r l y . \lVert\widehat{\beta}^{\mathrm{early}}-\beta^*\rVert_2^2 \lesssim\lambda_n s_n+\frac{s_n\log p+\log|\mathcal{E}|}{\min_{e\in\mathcal{E}}n^e}+\tau^{\mathrm{early}}. ∥ β early − β ∗ ∥ 2 2 ≲ λ n s n + min e ∈ E n e s n log p + log ∣ E ∣ + τ early .
The proof of Theorem 2 is provided in Appendix E.3. When λ n s n ≲ ( s n log p + log ∣ E ∣ ) / min e ∈ E n e \lambda_n s_n\lesssim(s_n\log p+\log|\mathcal{E}|)/\min_{e\in\mathcal{E}}n^e λ n s n ≲ ( s n log p + log ∣ E ∣ ) / min e ∈ E n e , the convergence rate of β ^ \widehat{\beta} β is dominated by the term s n log p / min e ∈ E n e s_n\log p/\min_{e\in\mathcal{E}}n^e s n log p / min e ∈ E n e , thereby accommodating high dimensional settings. Moreover, for the early-stopping estimator β ^ e a r l y \widehat{\beta}^{\mathrm{early}} β early , if τ e a r l y ≲ ( s n log p + log ∣ E ∣ ) / min e ∈ E n e \tau^{\mathrm{early}}\lesssim(s_n\log p+\log|\mathcal{E}|)/\min_{e\in\mathcal{E}}n^e τ early ≲ ( s n log p + log ∣ E ∣ ) / min e ∈ E n e , then early stopping does not affect its convergence rate.
Theorem 3. Suppose conditions in Assumptions 1–3 and of Lemma 1 hold. We also assume that
min j ∈ S ∗ ∣ β j ∗ ∣ 2 ≳ λ n + s n log p + log ∣ E ∣ min e ∈ E n e . (12) \min_{j\in S^*}|\beta_j^*|^2 \gtrsim\lambda_n+\frac{s_n\log p+\log|\mathcal{E}|}{\min_{e\in\mathcal{E}}n^e}. \tag*{(12)} j ∈ S ∗ min ∣ β j ∗ ∣ 2 ≳ λ n + min e ∈ E n e s n log p + log ∣ E ∣ . (12) Take λ n ≳ ( s n log p + log ∣ E ∣ ) / min e ∈ E n e \lambda_n\gtrsim(s_n\log p+\log|\mathcal{E}|)/\min_{e\in\mathcal{E}}n^e λ n ≳ ( s n log p + log ∣ E ∣ ) / min e ∈ E n e . When min e ∈ E n e ≳ s n log p + log ∣ E ∣ \min_{e\in\mathcal{E}}n^e\gtrsim s_n\log p+\log|\mathcal{E}| min e ∈ E n e ≳ s n log p + log ∣ E ∣ , the following are true with probability at least 1 − 7 / p 1-7/p 1 − 7/ p :
The optimal solution model recovers the causal parents: S ^ = S ∗ \widehat{S}=S^* S = S ∗ .
Whenever the early-stopping threshold satisfies τ e a r l y ≲ ( s n log p + log ∣ E ∣ ) / min e ∈ E n e \tau^{\mathrm{early}}\lesssim(s_n\log p+\log|\mathcal{E}|)/\min_{e\in\mathcal{E}}n^e τ early ≲ ( s n log p + log ∣ E ∣ ) / min e ∈ E n e , the resulting model recovers the causal parents S ^ e a r l y = S ∗ \widehat{S}^{\mathrm{early}}=S^* S early = S ∗ .
The proof of Theorem 3 is provided in Appendix E.4. The argument proceeds by showing that the optimal objective value under the true model S ∗ S^* S ∗ is strictly smaller than that under any incorrect model. This separation is ensured, in part, by the beta-min condition in (11), which imposes a lower bound on the signal strength of the true causal predictors so that they can be recovered by the estimated model. It also relies on an appropriate choice of λ n \lambda_n λ n , which must be sufficiently large to discourage the inclusion of redundant features. Furthermore, the sub-optimal solution from early stopping can also achieve perfect recovery if the corresponding threshold τ e a r l y \tau^{\mathrm{early}} τ early is not too large.
Task 2: Testing for causal parents via convex mixed-integer programming
Although procedure (8) can estimate an invariant set, recovering the true causal parent set S ∗ S^* S ∗ generally requires additional, potentially restrictive identifiability assumptions (e.g., Assumption 1). Under substantially weaker requirements, this section studies the task of identifying plausible causal predictors using a procedure of the form (9). In particular, the plausible causal predictor set S ( E ) = ⋂ S ∈ S L C S R ( E ) S \mathcal{S}(\mathcal{E})=\bigcap_{S\in\mathcal{S}_{\mathrm{LCSR}}(\mathcal{E})}S S ( E ) = ⋂ S ∈ S LCSR ( E ) S provides useful information about S ∗ S^* S ∗ . First, since S ( E ) ⊆ S ∗ \mathcal{S}(\mathcal{E})\subseteq S^* S ( E ) ⊆ S ∗ , it provides a lower bound on the causal parent set. At the empirical level, we further show that the estimated plausible causal predictor set, denoted by S ^ ( E ) \widehat{\mathcal{S}}(\mathcal{E}) S ( E ) , satisfies S ^ ( E ) ⊆ S ∗ \widehat{S}(\mathcal{E}) \subseteq S^* S ( E ) ⊆ S ∗ at a designated significance level. This result provides a form of uncertainty quantification by controlling the inclusion of noncausal variables. Second, every variable in S ( E ) S(\mathcal{E}) S ( E ) is a causal parent, and recovering even a subset of causal parents can already be informative and useful for practitioners.
Our uncertainty quantification results are asymptotic, with the sample size growing in each environment. Throughout this section, we assume either that the number of predictors p p p remains fixed as the sample size increases or that the analysis is restricted to a subset of predictors. For example, procedure (8) can be applied to one portion of the data to identify an invariant set. The remaining data can then be used to identify which features in this set are likely to be causal parents.
A preliminary multiple hypothesis framework
To obtain an estimate S ^ ( E ) \widehat{S}(\mathcal{E}) S ( E ) , a naive approach, following Peters et al. [18 ] , is to test each candidate set S S S for LSCR invariance and take the intersection of all sets for which the null hypothesis is not rejected. This, however, scales exponentially with the number of predictors. Instead, we adopt the approach in Martinet et al. [15 ] , which allows us to estimate S ^ ( E ) \widehat{S}(\mathcal{E}) S ( E ) from p p p hypothesis tests. Specifically, for every k ∈ [ p ] k \in[p] k ∈ [ p ] , we consider the null hypothesis:
H 0 , k : k ∉ S ( E ) ⟺ ∃ S ⊆ [ p ] ∖ { k } such that S is LSCR invariant . H_{0,k}: k \notin S(\mathcal{E}) \quad\Longleftrightarrow\quad\exists S \subseteq[p] \setminus\{k\}\ \text{such that } S\ \text{is LSCR invariant}. H 0 , k : k ∈ / S ( E ) ⟺ ∃ S ⊆ [ p ] ∖ { k } such that S is LSCR invariant .
Then, we obtain an estimate S ^ ( E ) \widehat{S}(\mathcal{E}) S ( E ) via
S ^ ( E ) : = { k ∈ [ p ] : H 0 , k is rejected } = { k ∈ [ p ] : T ^ ( k ) > τ ^ k ( α ) } . (13) \widehat{S}(\mathcal{E}) := \{k \in[p] : H_{0,k}\ \text{is rejected}\} = \{k \in[p] : \widehat{T}(k) > \widehat{\tau}_k(\alpha)\}. \tag*{(13)} S ( E ) := { k ∈ [ p ] : H 0 , k is rejected } = { k ∈ [ p ] : T ( k ) > τ k ( α )} . (13)
Here, T ^ ( k ) \widehat{T}(k) T ( k ) represents a test statistic for the hypothesis H 0 , k H_{0,k} H 0 , k and τ ^ k ( α ) \widehat{\tau}_k(\alpha) τ k ( α ) represents the corresponding threshold. For the test to be calibrated, for every k ∈ [ p ] k \in[p] k ∈ [ p ] , we want:
P H 0 , k ( T ^ ( k ) > τ ^ k ( α ) ) ≤ α / p + o ( 1 ) ∀ k ∈ [ p ] , (14) \mathbb{P}_{H_{0,k}}\bigl(\widehat{T}(k) > \widehat{\tau}_k(\alpha)\bigr) \leq\alpha/p + o(1) \quad\forall k \in[p], \tag*{(14)} P H 0 , k ( T ( k ) > τ k ( α ) ) ≤ α / p + o ( 1 ) ∀ k ∈ [ p ] , (14)
where the factor of p p p is due to a Bonferroni correction. Then, the estimated plausible causal predictor set S ^ ( E ) \widehat{S}(\mathcal{E}) S ( E ) is unlikely to contain any noncausal variables:
P ( S ^ ( E ) ⊈ S ∗ ) ≤ P ( H 0 , k is rejected for some noncausal k ) ≤ ∑ k = 1 p P H 0 , k ( T ^ ( k ) > τ ^ k ( α ) ) ≤ α + o ( 1 ) . \mathbb{P}\bigl(\widehat{S}(\mathcal{E}) \nsubseteq S^*\bigr) \leq\mathbb{P}\bigl(H_{0,k}\ \text{is rejected for some noncausal } k\bigr) \leq\sum_{k=1}^{p}\mathbb{P}_{H_{0,k}}\bigl(\widehat{T}(k) > \widehat{\tau}_k(\alpha)\bigr) \leq\alpha+ o(1). P ( S ( E ) ⊈ S ∗ ) ≤ P ( H 0 , k is rejected for some noncausal k ) ≤ k = 1 ∑ p P H 0 , k ( T ( k ) > τ k ( α ) ) ≤ α + o ( 1 ) .
In § 5.2 , we describe how the test statistics T ^ ( k ) \widehat{T}(k) T ( k ) can be obtained by mixed-integer programming. In § 5.3 , we describe a multiplier bootstrapping procedure to obtain the thresholds τ ^ k ( α ) \widehat{\tau}_k(\alpha) τ k ( α ) .
Obtaining test statistics via convex mixed-integer programming
By leveraging our invariance score I ^ ( ⋅ ) \widehat{I}(\cdot) I ( ⋅ ) in (4) and I ^ + ( ⋅ ) \widehat{I}_+(\cdot) I + ( ⋅ ) in Remark 3, we construct two types of test statistics T ^ ( k ) \widehat{T}(k) T ( k ) for the hypothesis H 0 , k H_{0,k} H 0 , k . Let n ∗ : = ∑ e n e n^* := \sum_{e} n_e n ∗ := ∑ e n e . Specifically, we obtain the test statistics via the following optimization problems:
T ^ M I P - t e s t - 2 ( k ) : = min S ⊆ [ p ] n ∗ I ^ ( S ) subject to: k ∉ S , T ^ M I P - t e s t - 4 ( k ) : = min S ⊆ [ p ] n ∗ I ^ + ( S ) subject to: k ∉ S . (15) \begin{aligned}
\widehat{T}_{\mathrm{MIP\text{-}test\text{-}2}}(k) &:= \min_{S \subseteq[p]} n^*\widehat{I}(S) && \text{subject to: } k \notin S, \\
\widehat{T}_{\mathrm{MIP\text{-}test\text{-}4}}(k) &:= \min_{S \subseteq[p]} n^*\widehat{I}_+(S) && \text{subject to: } k \notin S. \tag*{(15)}
\end{aligned} T MIP - test - 2 ( k ) T MIP - test - 4 ( k ) := S ⊆ [ p ] min n ∗ I ( S ) := S ⊆ [ p ] min n ∗ I + ( S ) subject to: k ∈ / S , subject to: k ∈ / S . (15)
With the invariance scores (up to a factor of n ∗ n^* n ∗ ) as its objective, these optimization problems search for the most invariant model that excludes feature X k X_k X k . Specifically, when k ∈ S ( E ) k \in S(\mathcal{E}) k ∈ S ( E ) , excluding X k X_k X k from the feasible models precludes the existence of an invariant set. Consequently, the optimal objective values T ^ M I P - t e s t - 2 ( k ) \widehat{T}_{\mathrm{MIP\text{-}test\text{-}2}}(k) T MIP - test - 2 ( k ) and T ^ M I P - t e s t - 4 ( k ) \widehat{T}_{\mathrm{MIP\text{-}test\text{-}4}}(k) T MIP - test - 4 ( k ) , which quantify the strongest attainable degrees of invariance under this restriction, are expected to be large. In contrast, when k ∉ S ( E ) k \notin S(\mathcal{E}) k ∈ / S ( E ) , excluding X k X_k X k does not preclude the existence of an invariant set S S S , and hence T ^ M I P - t e s t - 2 ( k ) \widehat{T}_{\mathrm{MIP\text{-}test\text{-}2}}(k) T MIP - test - 2 ( k ) and T ^ M I P - t e s t - 4 ( k ) \widehat{T}_{\mathrm{MIP\text{-}test\text{-}4}}(k) T MIP - test - 4 ( k ) will be small. Therefore, the null hypothesis H 0 , k H_{0,k} H 0 , k can be tested by choosing an appropriate threshold τ k \tau_k τ k that separates these two cases. The factor n ∗ n^* n ∗ rescales the invariance score so that the test statistic has a nondegenerate limiting distribution.
As described in § 3 , both of the optimization problems in (15) can be reformulated as a convex MIP. See also Section A.1 for additional details.
Example 4. We continue with the setup in Example 1, which contains four covariates ( X 1 , X 2 , X 3 , X 4 ) (X_1, X_2, X_3, X_4) ( X 1 , X 2 , X 3 , X 4 ) and a response Y Y Y . Among these covariates, only X 1 X_1 X 1 and X 3 X_3 X 3 are causal parents of Y Y Y . We further augment the data with nine independent noncausal variables. Across the three environments, every LSCR-invariant set contains either { 1 , 2 } \{1, 2\} { 1 , 2 } or { 1 , 3 } \{1, 3\} { 1 , 3 } as a subset. Since both { 1 , 2 } \{1, 2\} { 1 , 2 } and { 1 , 3 } \{1, 3\} { 1 , 3 } are themselves LSCR-invariant, it follows that S ( E ) = { 1 } S(\mathcal{E}) = \{1\} S ( E ) = { 1 } . Figure 4 presents the results obtained using MIP-test-2. Only feature X 1 X_1 X 1 has a test statistic exceeding the threshold. This is because, after excluding X 1 X_1 X 1 , no invariant set exists and therefore even the smallest attainable invariance score is large.
Figure 4. Test statistic T ^ M I P − t e s t − 2 ( k ) \widehat{T}_{\mathrm{MIP-test-2}}(k) T MIP − test − 2 ( k ) for every k k k . Orange bars denote the true causal parents, while blue bars denote noncausal variables; the hatched bar indicates the feature in S ( E ) S(\mathcal{E}) S ( E ) . The red dashed segments show the corresponding bootstrap thresholds that are described in § 5.3 . Only X 1 X_1 X 1 exceeds its threshold, correctly identifying S ( E ) = { 1 } S(\mathcal{E}) = \{1\} S ( E ) = { 1 } . Full experimental details are provided in Appendix G.§ 5 .
A bootstrap procedure for testing
Using a multiplier bootstrap procedure, we construct thresholds τ k ( α ) \tau_k(\alpha) τ k ( α ) satisfying condition (13). Since our approach applies to both test statistics T ^ M I P − t e s t − 2 ( k ) \widehat{T}_{\mathrm{MIP-test-2}}(k) T MIP − test − 2 ( k ) and T ^ M I P − t e s t − 4 ( k ) \widehat{T}_{\mathrm{MIP-test-4}}(k) T MIP − test − 4 ( k ) , we present it in a general form.
For any k ∉ S ( E ) k \notin S(\mathcal{E}) k ∈ / S ( E ) , let S k \mathcal{S}_k S k denote the collection of all LSCR invariant sets excluding feature X k X_k X k , and let S ^ \widehat{S} S denote the set encoded by an optimal solution to (14). Suppose that S ^ ∈ S k \widehat{S} \in\mathcal{S}_k S ∈ S k . If we set τ k ( α ) \tau_k(\alpha) τ k ( α ) to be the ( 1 − α / p ) (1-\alpha/p) ( 1 − α / p ) -quantile of the distribution of n ⋆ max S ∈ S k I ^ ( S ) n^\star\max_{S \in\mathcal{S}_k} \widehat{I}(S) n ⋆ max S ∈ S k I ( S ) , then we have P ( T ^ ( k ) > τ k ( α ) ) ≤ α / p \mathbb{P}(\widehat{T}(k) > \tau_k(\alpha)) \leq\alpha/p P ( T ( k ) > τ k ( α )) ≤ α / p . Implementing this strategy requires two key steps. First, because the collection S k \mathcal{S}_k S k may be difficult to estimate exactly, we construct an enlarged collection S k ∘ ⊇ S k \mathcal{S}_k^\circ\supseteq\mathcal{S}_k S k ∘ ⊇ S k such that S ^ ∈ S k ∘ \widehat{S} \in\mathcal{S}_k^\circ S ∈ S k ∘ with high probability. Second, we use a multiplier bootstrap procedure to approximate the distribution of n ⋆ max S ∈ S k ∘ I ^ ( S ) n^\star\max_{S \in\mathcal{S}_k^\circ} \widehat{I}(S) n ⋆ max S ∈ S k ∘ I ( S ) .
More specifically, we split the data into two independent subsamples, D 1 \mathcal{D}_1 D 1 and D 2 \mathcal{D}_2 D 2 . Using D 1 \mathcal{D}_1 D 1 , we construct a candidate collection S k ∘ \mathcal{S}_k^\circ S k ∘ containing up to M ‾ \overline{M} M distinct predictor sets. In practice, M ‾ \overline{M} M can be chosen to provide sufficient diversity among the candidate invariant sets, without requiring the collection to be exhaustive. One strategy is to generate bootstrap datasets from D 1 D_1 D 1 and, for each bootstrap dataset, solve the optimization problem in (14) to obtain an estimated invariant set. This procedure is repeated until M ˉ \bar{M} M ˉ distinct sets have been collected. We use this strategy in practice for computational efficiency. Another strategy is to sequentially retrieve sets S S S in ascending order of the objective value in (14), starting with the minimizer, followed by the second best solution, and so forth, until M ˉ \bar{M} M ˉ sets have been retrieved. This sequential procedure is particularly amenable to mixed-integer programming if we use the procedure described in Remark 1. Details are provided in Appendix B. We also establish a lemma for the second strategy showing that, under a Gaussian model, the resulting collection S k ∘ S_k^\circ S k ∘ contains S k S_k S k with probability tending to one. Moreover, the condition S ^ ∈ S k ∘ \widehat{S} \in S_k^\circ S ∈ S k ∘ is satisfied as well.
We next obtain a threshold for T ^ ( k ) \widehat{T}(k) T ( k ) using the multiplier bootstrap procedure. Specifically, for each bootstrap replication, the algorithm generates independent Gaussian multipliers and uses them to construct bootstrap analogues of the coefficient and risk components of the invariance score for every S ∈ S k ∘ S \in S_k^\circ S ∈ S k ∘ . It then takes the maximum of the resulting bootstrap invariance scores I ^ ( S ) \widehat{I}(S) I ( S ) over S ∈ S k ∘ S \in S_k^\circ S ∈ S k ∘ , producing one bootstrap realization of the maximal fluctuation. Repeating this procedure yields an empirical approximation to the distribution of n ⋆ max S ∈ S k ∘ I ^ ( S ) n^\star\max_{S\in S_k^\circ}\widehat{I}(S) n ⋆ max S ∈ S k ∘ I ( S ) , and the algorithm returns its estimated ( 1 − α / p ) (1-\alpha/p) ( 1 − α / p ) -quantile as the threshold. The multiplier bootstrap procedure, specifically for the test statistic T ^ M I P - t e s t - 2 ( k ) \widehat{T}_{\mathrm{MIP\text{-}test\text{-}2}}(k) T MIP - test - 2 ( k ) is described in Algorithm 1. The extension to T ^ M I P - t e s t - 4 ( k ) \widehat{T}_{\mathrm{MIP\text{-}test\text{-}4}}(k) T MIP - test - 4 ( k ) is provided in Appendix B.2.
Remark 6. Although our method involves discrete variable selection, the multiplier bootstrap in Algorithm 1 is not used to approximate the distribution of the selected predictor set or the MIP optimizer, for which standard bootstrap methods may be invalid due to nonregularity. Instead, conditional on the candidate collection S k ∘ S_k^\circ S k ∘ , we use the multiplier bootstrap to approximate the distribution of the largest sampling fluctuation in the invariance statistics across S ∈ S k ∘ S \in S_k^\circ S ∈ S k ∘ . Since S k ∘ S_k^\circ S k ∘ is constructed using an independent subsample D 1 D_1 D 1 , it can be treated as fixed when performing inference on D 2 D_2 D 2 .
The following theorem establishes asymptotic type-I error control, ensuring that the estimated set S ^ ( E ) \widehat{S}(\mathcal{E}) S ( E ) contains no noncausal variables with probability at least 1 − α 1-\alpha 1 − α . The proof is provided in Appendix F.3.
Theorem 4. Suppose that S ^ ∈ S k \widehat{S} \in S_k S ∈ S k , S k ∘ ⊇ S k S_k^\circ\supseteq S_k S k ∘ ⊇ S k , and n e / n ⋆ → ρ ∈ ( 0 , 1 ) n^e/n^\star\to\rho\in(0,1) n e / n ⋆ → ρ ∈ ( 0 , 1 ) for all e ∈ E e \in\mathcal{E} e ∈ E . Assume that E [ ∥ X e ∥ 2 4 + ∣ Y e ∣ 4 ] < ∞ \mathbb{E}[\lVert X^e\rVert_2^4+\lvert Y^e\rvert^4]<\infty E [∥ X e ∥ 2 4 + ∣ Y e ∣ 4 ] < ∞ and λ min ( Σ e ) > 0 \lambda_{\min}(\Sigma^e)>0 λ m i n ( Σ e ) > 0 . Assume that the bootstrap sample size B n → ∞ B_n \to\infty B n → ∞ as n → ∞ n\to\infty n → ∞ . Then the test statistic T ^ ( k ) \widehat{T}(k) T ( k ) defined in (14), together with the bootstrap threshold τ ^ k ( α ) \widehat{\tau}_k(\alpha) τ k ( α ) obtained from Algorithm 1, provides asymptotic type-I error control for testing H 0 , k H_{0,k} H 0 , k :
P H 0 , k ( T ^ ( k ) > τ ^ k ( α ) ) ≤ α p + o ( 1 ) \mathbb{P}_{H_{0,k}}\left(\widehat{T}(k)>\widehat{\tau}_k(\alpha)\right)\leq\frac{\alpha}{p}+o(1) P H 0 , k ( T ( k ) > τ k ( α ) ) ≤ p α + o ( 1 )
Consequently, for the estimated plausible causal predictor set defined in (12),
lim inf n → ∞ P ( S ^ ( E ) ⊆ S ⋆ ) ≥ 1 − α . \liminf_{n\to\infty}\mathbb{P}\left(\widehat{S}(\mathcal{E})\subseteq S^\star\right)\geq1-\alpha. n → ∞ lim inf P ( S ( E ) ⊆ S ⋆ ) ≥ 1 − α .
Example 5. We use the same setup as in Example 4. Figure 5 compares the performance of four methods for recovering S ( E ) = { 1 } S(\mathcal{E})=\{1\} S ( E ) = { 1 } as the minimum sample size min e ∈ E n e \min_{e\in\mathcal{E}} n^e min e ∈ E n e increases. We consider two approaches using MIP: MIP-test-2, which uses the test statistic in (14) based on two components of the invariance score, and MIP-test-4, given in (18), which incorporates two additional components into the test statistic. The MIP methods, particularly MIP-test-4, achieve nearly perfect recovery at relatively small sample sizes while remaining computationally efficient in this toy example. The performance gap between MIP-test-2 and MIP-test-4 demonstrates the benefit of incorporating the two additional components into the test statistic. ICP also achieves nearly perfect recovery when the sample size is sufficiently large because it uses the same invariance principle as MIP-test-2, although its testing procedure appears less powerful. Both MIP approaches and ICP achieve the desired type-I error control, whereas the WVM fails to maintain the target coverage.
Algorithm 1 Bootstrap estimation of the threshold τ ^ k ( α ) \widehat{\tau}_k(\alpha) τ k ( α )
Input: Multi-environment data { ( X ⋅ , i e , Y i e ) } i = 1 n e \{(X_{\cdot,i}^{e},Y_i^{e})\}_{i=1}^{n^e} {( X ⋅ , i e , Y i e ) } i = 1 n e for e ∈ E e \in\mathcal{E} e ∈ E , collection S k ∘ \mathcal{S}_k^{\circ} S k ∘ , number of bootstrap samples B B B , and significance level α \alpha α .
Output: Bootstrap threshold τ ^ k ( α ) \widehat{\tau}_k(\alpha) τ k ( α ) .
for b = 1 , … , B b=1,\ldots,B b = 1 , … , B do
For each e ∈ E e \in\mathcal{E} e ∈ E , independently sample ξ 1 e , … , ξ n e e ∼ i i d N ( 0 , 1 ) \xi_1^e,\ldots,\xi_{n^e}^e \overset{\mathrm{iid}}{\sim} \mathcal{N}(0,1) ξ 1 e , … , ξ n e e ∼ iid N ( 0 , 1 ) .
for S ∈ S k ∘ S \in\mathcal{S}_k^{\circ} S ∈ S k ∘ do
For each e ∈ E e \in\mathcal{E} e ∈ E , compute the OLS coefficients β ^ e , S \widehat{\beta}^{e,S} β e , S and
r ^ i e , S = Y i e − ( X ⋅ , i e ) ⊤ β ^ e , S , Σ ^ S e = 1 n e ∑ i = 1 n e X S , i e X S , i e ⊤ , \widehat{r}_i^{e,S}=Y_i^e-(X_{\cdot,i}^{e})^{\top}\widehat{\beta}^{e,S},\qquad\widehat{\Sigma}_S^e=\frac{1}{n^e}\sum_{i=1}^{n^e}X_{S,i}^eX_{S,i}^{e\top}, r i e , S = Y i e − ( X ⋅ , i e ) ⊤ β e , S , Σ S e = n e 1 i = 1 ∑ n e X S , i e X S , i e ⊤ ,
G ⋆ , e , S = 1 n e ∑ i = 1 n e ξ i e X S , i e r ^ i e , S , H ⋆ , e , S = 1 n e ∑ i = 1 n e ξ i e { ( r ^ i e , S ) 2 − 1 n e ∑ j = 1 n e ( r ^ j e , S ) 2 } . G^{\star,e,S}=\frac{1}{\sqrt{n^e}}\sum_{i=1}^{n^e}\xi_i^eX_{S,i}^e\widehat{r}_i^{e,S},\qquad H^{\star,e,S}=\frac{1}{\sqrt{n^e}}\sum_{i=1}^{n^e}\xi_i^e\left\{(\widehat{r}_i^{e,S})^2-\frac{1}{n^e}\sum_{j=1}^{n^e}(\widehat{r}_j^{e,S})^2\right\}. G ⋆ , e , S = n e 1 i = 1 ∑ n e ξ i e X S , i e r i e , S , H ⋆ , e , S = n e 1 i = 1 ∑ n e ξ i e { ( r i e , S ) 2 − n e 1 j = 1 ∑ n e ( r j e , S ) 2 } .
For each pair e < f e<f e < f in E \mathcal{E} E , compute
L n ⋆ , e , f , S = n ⋆ n e ( Σ ^ S e ) − 1 G ⋆ , e , S − n ⋆ n f ( Σ ^ S f ) − 1 G ⋆ , f , S , L_n^{\star,e,f,S}=\sqrt{\frac{n^\star}{n^e}}(\widehat{\Sigma}_S^e)^{-1}G^{\star,e,S}-\sqrt{\frac{n^\star}{n^f}}(\widehat{\Sigma}_S^f)^{-1}G^{\star,f,S}, L n ⋆ , e , f , S = n e n ⋆ ( Σ S e ) − 1 G ⋆ , e , S − n f n ⋆ ( Σ S f ) − 1 G ⋆ , f , S ,
V n ⋆ , e , f , S = n ⋆ n e H ⋆ , e , S − n ⋆ n f H ⋆ , f , S . V_n^{\star,e,f,S}=\sqrt{\frac{n^\star}{n^e}}H^{\star,e,S}-\sqrt{\frac{n^\star}{n^f}}H^{\star,f,S}. V n ⋆ , e , f , S = n e n ⋆ H ⋆ , e , S − n f n ⋆ H ⋆ , f , S .
end for
Compute
t k b = max S ∈ S k ∘ { ∑ e < f ω e , f ∥ L n ⋆ , e , f , S ∥ 2 2 + ρ ∑ e < f ω e , f ∣ V n ⋆ , e , f , S ∣ 2 } . t_k^b=\max_{S\in\mathcal{S}_k^{\circ}}\left\{\sum_{e<f}\omega^{e,f}\left\|L_n^{\star,e,f,S}\right\|_2^2+\rho\sum_{e<f}\omega^{e,f}\left|V_n^{\star,e,f,S}\right|^2\right\}. t k b = S ∈ S k ∘ max ⎩ ⎨ ⎧ e < f ∑ ω e , f L n ⋆ , e , f , S 2 2 + ρ e < f ∑ ω e , f V n ⋆ , e , f , S 2 ⎭ ⎬ ⎫ .
end for
Set
τ ^ k ( α ) = quantile ( { t k b } b = 1 B ; 1 − α / p ) . \widehat{\tau}_k(\alpha)=\operatorname{quantile}\left(\{t_k^b\}_{b=1}^{B};1-\alpha/p\right). τ k ( α ) = quantile ( { t k b } b = 1 B ; 1 − α / p ) .
return τ ^ k ( α ) \widehat{\tau}_k(\alpha) τ k ( α ) .
Experiments
We conduct four complementary simulation studies to investigate different aspects of our proposed framework. Across all experiments, data are generated from linear SCMs following (1) . The true causal parent set S ⋆ S^\star S ⋆ and its coefficient vector β ⋆ \beta^\star β ⋆ remain fixed across environments, so that the causal conditional distribution Y e ∣ X S ⋆ e Y^e\mid X_{S^\star}^e Y e ∣ X S ⋆ e is invariant. Meanwhile, the covariate distributions are perturbed through environment specific interventions.
We focus on the estimator for Task 1 in § 6.1 –§ 6.2 and the testing procedure for Task 2 in § 6.3 –§ 6.4 . In § 6.1 , we examine the statistical behavior of our invariant estimator (5) , including its dependence on the number of environments and sample size, as well as its robustness to the choice of tuning parameters. In § 6.2 , we compare its statistical and computational performance with existing invariant prediction methods. In § 6.3 , we compare the procedures developed for Tasks 1 and 2 and investigate how their different goals affect empirical performance. Finally, § 6.4 compares the statistical power, coverage, and computational efficiency of different testing procedures as the number of environments increases. Details of data generation mechanisms and implementation are provided in Appendix G.
Behavior of estimator (5)
We generate data from a linear structural equation model with p = 20 p=20 p = 20 variables and introduce environmental heterogeneity through soft interventions. Across environments, only the linear coefficients used to generate the covariates are perturbed, while the causal mechanism for the response, including its parent set S ∗ S^{*} S ∗ and structural coefficients β ∗ \beta^{*} β ∗ , remains unchanged. In these experiments, we use the exact MIP solver and solve all trials to global optimality.
Figure 6 shows that both sample size and environmental heterogeneity play important roles in recovery. For a fixed sample size, increasing the number of environments substantially reduces the coefficient estimation error and improves support recovery. With more environments, interventions produce stronger variation and hence more information for distinguishing invariant causal predictors from spurious ones, consistent with our identification theory. In particular, perfect recovery is achieved at considerably smaller sample sizes when more environments are available. For a fixed number of environments, performance also improves steadily as the sample size in each environment increases. The coefficient estimation error decreases and the probability of perfect recovery approaches one, in line with our theory.
Figure 6. Statistical behavior and tuning robustness of estimator (5). Left panel: average coefficient estimation error ∥ β ^ − β ∗ ∥ 2 \lVert\hat{\beta}-\beta^*\rVert_2 ∥ β ^ − β ∗ ∥ 2 over 100 trials, as the sample size per environment increases, for ∣ E ∣ ∈ { 2 , 3 , 5 , 10 } |\mathcal{E}| \in\{2,3,5,10\} ∣ E ∣ ∈ { 2 , 3 , 5 , 10 } environments. Middle panel: proportion of 100 trials achieving perfect recovery of S ∗ S^* S ∗ under the same settings. Solid lines show averages and shaded bands indicate ± 1 \pm1 ± 1 standard error. Right panel: average leave-one-environment-out (LOEO) criterion over 50 trials, for each combination of tuning parameters γ \gamma γ and λ n \lambda_n λ n ; red outlines indicate tuning combinations with recovery proportion above 0.95. Additional experimental details are provided in Appendix G.3.
The right panel examines robustness to the regularization parameters γ \gamma γ and λ n \lambda_n λ n , with ρ \rho ρ fixed at 1. Each cell reports the average leave-one-environment-out (LOEO) validation error (see Remark 2), while the red outlines indicate parameter combinations for which the true causal support S ∗ S^{*} S ∗ is recovered. Correct recovery occurs over a broad range of ( γ , λ n ) (\gamma,\lambda_n) ( γ , λ n ) values rather than at a narrowly tuned choice, demonstrating that the proposed estimator is relatively stable with respect to the regularization parameters. Moreover, several of the smallest LOEO validation errors fall within the region of correct recovery, further supporting the effectiveness and robustness of the tuning procedure based on LOEO.
Comparison of estimator (5) with existing methods
To avoid favoring any particular intervention model, we consider a strong intervention regime. Across environments, the mechanisms generating covariates may differ in their graph structures, structural coefficients, and noise distributions, while the causal conditional distribution Y e ∣ X S ∗ e Y^{e} \mid X_{S^{*}}^{e} Y e ∣ X S ∗ e remains unchanged. We let the number of causal predictors increase with the ambient dimension p p p : we set ∣ S ∗ ∣ = ⌊ 0.4 p ⌋ |S^{*}| = \left\lfloor0.4p \right\rfloor ∣ S ∗ ∣ = ⌊ 0.4 p ⌋ when ⌊ 0.4 p ⌋ < 16 \left\lfloor0.4p \right\rfloor< 16 ⌊ 0.4 p ⌋ < 16 , and ∣ S ∗ ∣ = 16 + ⌊ 0.1 p ⌋ |S^{*}| = 16 + \left\lfloor0.1p \right\rfloor ∣ S ∗ ∣ = 16 + ⌊ 0.1 p ⌋ otherwise. We generate 10 environments with n e = 500 n^{e} = 500 n e = 500 observations per environment. Full details of the data generation process are provided in Appendix G.4.
We compare our coordinate descent implementation of (5), MIP-CD , with screened EILLS-s [5 ] , screened ICP-s [18 ] , Causal Dantzig [21 ] , screened BIP-s (Wu et al., 2025), NegDRO [24 ] , CoCo [27 ] , and IRM [1 ] . We use MIP-CD rather than the exact MIP solver because these experiments involve relatively large dimensions and many simulation trials, for which coordinate descent is computationally more efficient.
Since the exact implementations of ICP , BIP , and EILLS require enumeration over candidate predictor subsets, they do not scale to the larger dimensions considered here. Following the screening strategy used by Wu et al. (2025), we first screen the covariates based on the magnitudes of ridge regression coefficients. We keep the top 10 variables for ICP-s and BIP-s and the top 15 variables for EILLS-s . These screening sizes are chosen based on computational feasibility: ICP-s with 10 variables and EILLS-s with 15 variables have comparable runtimes, both exceeding 10 seconds when screening becomes necessary, while the available implementation of BIP is more restrictive. Although approximate implementations are also available, we found that the Gumbel based approximation of EILLS rarely selects any variables in our experiments, making the comparison uninformative. Similarly, the exact implementation of BIP already exhibits low recovery rates at smaller dimensions. We therefore focus on the screened exact implementations of these methods to provide more informative comparisons. Consequently, once ∣ S ∗ ∣ |S^*| ∣ S ∗ ∣ exceeds the screening size, perfect support recovery becomes impossible for these methods, even under ideal screening. In our experiments, this occurs from p = 30 p = 30 p = 30 onward for ICP-s and BIP-s , and from p = 40 p = 40 p = 40 onward for EILLS-s .
The remaining baselines involve different compromises. The methods CoCo , NegDRO , and IRM primarily estimate a coefficient vector rather than performing explicit variable selection. Moreover, Causal Dantzig and NegDRO are developed under additive intervention mechanisms, where interventions are restricted to the noise terms. The comparison therefore evaluates how these methods perform under a more general intervention setting that is not tailored to their respective assumptions.
The right panel compares computational cost. Despite solving a discrete combinatorial optimization problem, our MIP-CD remains highly competitive computationally and is the fastest method for p < 50 p < 50 p < 50 in this experiment. Its runtime increases gradually with the dimension and remains comparable to most competing methods at larger p p p . These results demonstrate the scalability of the proposed MIP approach and its computational advantage over explicit subset enumeration.
Recovering S(E) versus an invariant predictor set
We compare methods designed for the two distinct tasks considered in this paper. The first group, including our proposed MIP-test in § 5 , ICP , and WVM , aims to recover the plausible causal predictor set S ( E ) S(\mathcal{E}) S ( E ) with controlled false discoveries. The second group, including MIP , EILLS and BIP-s , instead aims to identify an invariant set. These two objectives need not coincide when the causal parent set is not identifiable.
Figure 8 shows the trade-off between the two groups of methods using the setup of Example 4. Methods targeting S ( E ) S(\mathcal{E}) S ( E ) , including MIP-test-4 , ICP , and WVM , generally achieve high coverage and few false discoveries, but the selected sets are rarely themselves invariant. In contrast, methods targeting an invariant set, including MIP , EILLS , and BIP-s , recover invariant models much more frequently, but tend to have lower coverage and more false discoveries.
Figure 8. Trade-offs between coverage, invariance, and false discoveries across methods as the sample size per environment varies. Each point corresponds to one sample size n e n^{e} n e ranging from 400 to 10000. Left panel: proportion of 200 trials in which the estimated set is LSCR invariant, versus coverage probability, defined as the proportion of 200 trials in which S ^ ⊆ S ∗ \widehat{S} \subseteq S^* S ⊆ S ∗ . Middle panel: average number of false discoveries ∣ S ^ ∖ S ∗ ∣ |\widehat{S} \setminus S^*| ∣ S ∖ S ∗ ∣ over 200 trials, versus coverage probability. Right panel: average number of false discoveries, versus the proportion of trials in which the estimated set is invariant. Additional experimental details are provided in Appendix G.§ 5 .
This behavior follows from the structure of the example. Here, S ( E ) = { 1 } S(\mathcal{E}) = \{1\} S ( E ) = { 1 } , and multiple invariant models exist: every invariant set contains either { 1 , 2 } \{1, 2\} { 1 , 2 } or { 1 , 3 } \{1, 3\} { 1 , 3 } , whereas their intersection is only { 1 } \{1\} { 1 } . Importantly, S ( E ) S(\mathcal{E}) S ( E ) itself is not invariant. Therefore, methods targeting S ( E ) S(\mathcal{E}) S ( E ) are naturally conservative: they identify predictors shared across all invariant models, leading to few false discoveries, but the resulting set need not itself define an invariant predictive model. By contrast, methods targeting an invariant set seek a richer model satisfying the invariance criterion; because such a set is not unique, however, the selected model need not coincide with the true causal parent set S ⋆ S^\star S ⋆ .
Comparison of testing procedures
We further compare the statistical and computational performance of testing procedures for Task 2, under the intervention regime used in § 6.2 . We generate data from a linear structural equation model with p = 10 p = 10 p = 10 covariates and n e = 500 n^e = 500 n e = 500 observations per environment. In each environment, we generate a directed acyclic graph (DAG) by randomly selecting edges between pairs of nodes according to a common topological ordering, while keeping the edges pointing to the response fixed. The coefficients and noise variances of the covariates also vary across environments, whereas the causal conditional distribution Y e ∣ X S ⋆ e Y^e \mid X^e_{S^\star} Y e ∣ X S ⋆ e remains invariant. Compared with the setting in § 6.3 , the DAG structure and causal parent set are independently generated in each trial. Increasing the number of environments also provides greater heterogeneity across environments and improves causal identification. We compare our proposed methods, MIP-test-2 and MIP-test-4 , with ICP and WVM .
Figure 9 presents the results as the number of environments increases. All four methods maintain coverage above the target level of 0.95, indicating effective control of false causal discoveries in this experiment. Meanwhile, the number of true discoveries increases with the number of environments, as additional environments provide more information for distinguishing causal predictors from spurious variables. Our proposed MIP-test-2 and MIP-test-4 consistently identify more true causal predictors than ICP and WVM , demonstrating greater statistical power. Moreover, both MIP procedures are substantially faster than WVM and remain competitive with ICP .
Figure 9. Performance comparison of testing procedures for Task 2 as the number of environments increases. Left panel: coverage probability, defined as the proportion of 200 trials in which S ^ ( E ) ⊆ S ⋆ \widehat{S}(\mathcal{E}) \subseteq S^\star S ( E ) ⊆ S ⋆ , with the dashed line indicating the target level 0.95. Middle panel: average number of true discoveries ∣ S ^ ( E ) ∩ S ⋆ ∣ |\widehat{S}(\mathcal{E}) \cap S^\star| ∣ S ( E ) ∩ S ⋆ ∣ over 200 trials; the number of true causal parents S ⋆ S^\star S ⋆ is randomly selected from { 4 , 5 } \{4, 5\} { 4 , 5 } . Right panel: average runtime in seconds over 200 trials, shown on a logarithmic scale. Solid lines show averages, and shaded bands indicate ± 1 \pm1 ± 1 standard error. Additional experimental details are provided in Appendix G.§ 6 .
Real data: Causal chambers
Causal chambers [6 ] are devices controlled by computers that allow for manipulating and measuring variables in physical systems, providing a rich testbed for causal discovery algorithms. Following the experimental setup of Gu et al. (2025), we consider the light tunnel chamber, which consists of a controllable light source, two rotating polarizers, and sensors measuring light intensity. We aim to recover the causal parents of the infrared intensity measurement Y = I ~ 3 Y = \tilde{I}_3 Y = I ~ 3 among 11 observed covariates, with the known causal parent set S ⋆ = { R , G , B , θ 1 , θ 2 } S^\star= \{R, G, B, \theta_1, \theta_2\} S ⋆ = { R , G , B , θ 1 , θ 2 } . In particular, the causal effects of the polarizer angles ( θ 1 (\theta_1 ( θ 1 and θ 2 ) \theta_2) θ 2 ) on light intensity I ~ 3 \tilde{I}_3 I ~ 3 are nonlinear.
We use the reference environment and 19 intervention environments corresponding to different manipulations of the physical system. To investigate how the number of environments affects causal variable selection, we vary ∣ E ∣ ∈ { 3 , 4 , … , 10 } |\mathcal{E}| \in\{3, 4, \ldots, 10\} ∣ E ∣ ∈ { 3 , 4 , … , 10 } , with 1000 observations per environment. For each trial, we randomly permute the intervention environments and construct nested environment sets by sequentially adding interventions to the reference environment. We repeat this procedure for 100 randomized environment configurations and compare our proposed methods, MIP and MIP-test-4 , with EILLS , ICP , Causal Dantzig , and BIP . Additionally, we compare with FAIR (Gu et al., 2025), a method based on neural networks for invariant causal learning. This method is relevant here because of the nonlinear causal effects of ( θ 1 , θ 2 ) (\theta_1,\theta_2) ( θ 1 , θ 2 ) on I ~ 3 \tilde{I}_3 I ~ 3 . Additional experimental details are provided in Appendix G.§ 7 .
Figure 10 presents the F1 score, number of false discoveries, and runtime as the number of environments increases. MIP-CD achieves the highest F1 score as more environments become available. In particular, compared with EILLS , MIP-CD achieves a comparable or higher F1 score while producing substantially fewer false discoveries. Meanwhile, MIP-test exhibits more conservative variable selection, achieving lower F1 scores than MIP-CD and EILLS . Even so, it consistently outperforms all other competing methods in terms of F1 score, and produces fewer false discoveries than MIP-CD and EILLS . Notably, FAIR fails to recover any causal predictors in this experiment, possibly due to optimization difficulties associated with its Gumbel procedure. Even with 10,000 optimization iterations and runtimes exceeding 1,000 seconds, the computational effort remains insufficient for FAIR . In contrast, both MIP-CD and MIP-test are substantially more computationally efficient. Overall, these results demonstrate the effectiveness and computational efficiency of our proposed methods in identifying causal predictors from heterogeneous environments in a real physical system.
Figure 10. Performance comparison of invariant causal variable selection methods on the Causal Chamber dataset as the number of environments increases. Left panel: average F1 score over 100 trials, for recovering the true causal parent set S ⋆ S^\star S ⋆ . Middle panel: average number of false discoveries ∣ S ^ ∖ S ⋆ ∣ |\hat{S} \setminus S^\star| ∣ S ^ ∖ S ⋆ ∣ over 100 trials. Right panel: average runtime in seconds over 100 trials, shown on a logarithmic scale. Solid lines show averages, and shaded bands indicate ± 1 \pm1 ± 1 standard error. Additional experimental details are provided in Appendix G.§ 7 .
Extensions to nonlinearity and hidden confounding
Extension to nonlinear causal effects
Our framework can also be extended to settings with nonlinear causal effects. Specifically, consider an additive structural equation model
Y e = ∑ j ∈ S ⋆ f j ⋆ ( X j e ) + ε e , ε e ⊥ X S ⋆ e , ε e ∼ F , E [ ε e ] = 0 , Y^{e} = \sum_{j \in S^{\star}} f_{j}^{\star}(X_{j}^{e}) + \varepsilon^{e}, \qquad\varepsilon^{e} \perp X_{S^{\star}}^{e}, \qquad\varepsilon^{e} \sim\mathcal{F}, \quad\mathbb{E}[\varepsilon^{e}] = 0, Y e = j ∈ S ⋆ ∑ f j ⋆ ( X j e ) + ε e , ε e ⊥ X S ⋆ e , ε e ∼ F , E [ ε e ] = 0 ,
where f j ⋆ f_{j}^{\star} f j ⋆ denotes the nonlinear causal effect of X j X_{j} X j on Y Y Y . To accommodate nonlinear effects, we approximate each f j ⋆ f_{j}^{\star} f j ⋆ using R n R_{n} R n basis functions:
f j ⋆ ( X j e ) ≈ ∑ r = 1 R n θ j r b r ( X j e ) , f_{j}^{\star}(X_{j}^{e}) \approx\sum_{r=1}^{R_{n}} \theta_{jr}b_{r}(X_{j}^{e}), f j ⋆ ( X j e ) ≈ r = 1 ∑ R n θ j r b r ( X j e ) ,
where the coefficients θ j = ( θ j 1 , … , θ j R n ) ⊤ \theta_{j} = (\theta_{j1}, \ldots, \theta_{jR_{n}})^{\top} θ j = ( θ j 1 , … , θ j R n ) ⊤ are treated as a group associated with each original predictor X j X_{j} X j . We can then apply the LSCR invariance principle to the expanded covariates, together with a group ℓ 0 \ell_{0} ℓ 0 -type penalty. The resulting optimization problem can still be formulated as a convex MIP. One limitation is that the LSCR invariance principle may not hold exactly under finite basis expansions. In particular, the population OLS coefficients and residual risks may vary across environments due to approximation errors, even when the underlying nonlinear causal mechanism remains invariant. We therefore consider this extension as an empirical investigation and leave theoretical guarantees to future work. Further details on the formulation and numerical results are provided in Appendix C.
Extension to hidden confounding
We next consider settings with hidden confounding, which require a different invariance principle. Specifically, we consider a linear structural equation model in which exogenous latent variables may affect both the covariates and the response, while interventions change the noise variances of the observed covariates without altering the structural coefficients or the response mechanism. In this setting, the LSCR principle may no longer hold, since the population OLS coefficients can vary across environments even when the underlying causal coefficients remain invariant. Nevertheless, under the noise intervention model, we have
E [ X j e ( Y e − X e ⊤ β ⋆ ) ] = E [ X j f ( Y f − X f ⊤ β ⋆ ) ] , ∀ j ∈ [ p ] , e , f ∈ E . \mathbb{E}\left[X_{j}^{e}\left(Y^{e}-X^{e\top}\beta^{\star}\right)\right] = \mathbb{E}\left[X_{j}^{f}\left(Y^{f}-X^{f\top}\beta^{\star}\right)\right], \quad\forall j \in[p],\ e,f \in\mathcal{E}. E [ X j e ( Y e − X e ⊤ β ⋆ ) ] = E [ X j f ( Y f − X f ⊤ β ⋆ ) ] , ∀ j ∈ [ p ] , e , f ∈ E .
This motivates an alternative invariance principle based on residual moments, which can accommodate hidden confounding and is also considered in Rothenhäusler et al. [21 ] . Based on this principle, we define an invariance score that measures discrepancies in these residual moments across environments:
I h ( β ) = ∑ e < f ω e , f ∑ j = 1 p ( E [ X j e ( Y e − X e ⊤ β ) ] − E [ X j f ( Y f − X f ⊤ β ) ] ) 2 . I_{h}(\beta) = \sum_{e<f} \omega^{e,f}\sum_{j=1}^{p}\left(\mathbb{E}\left[X_{j}^{e}\left(Y^{e}-X^{e\top}\beta\right)\right]-\mathbb{E}\left[X_{j}^{f}\left(Y^{f}-X^{f\top}\beta\right)\right]\right)^{2}. I h ( β ) = e < f ∑ ω e , f j = 1 ∑ p ( E [ X j e ( Y e − X e ⊤ β ) ] − E [ X j f ( Y f − X f ⊤ β ) ] ) 2 .
By construction, I h ( β ) = 0 I_{h}(\beta) = 0 I h ( β ) = 0 if and only if the cross moments between the residuals and covariates remain invariant across environments. We then combine its empirical counterpart with the empirical MSE loss and a sparsity penalty, leading to a convex MIP formulation. Further details on the model, optimization formulation, and numerical experiments are provided in Appendix § 15 .
Discussion
We propose a unified MIP framework for invariant causal variable selection and inference across heterogeneous environments. Based on the LSCR invariance principle, our framework addresses both causal parent recovery under identifiability and causal discovery with type-I error control when full identification is not possible. Our methods enjoy both statistical and optimization guarantees, and experiments on synthetic and real data demonstrate their effectiveness.
Our work opens several directions for future research. First, although our framework can be extended to nonlinear additive models through basis expansions, approximation errors may prevent the LSCR invariance principle from holding exactly. Developing more general methods for nonlinear causal effects while preserving computational tractability and optimization guarantees would be valuable. Second, our identification results. depend on the heterogeneity of the available environments. Investigating how to design informative interventions or select environments to improve causal identification is an important direction. Finally, although our MIP formulations avoid explicit subset enumeration, solving larger problems to optimality can still be computationally challenging. Developing specialized branch-and-bound algorithms that further exploit the invariance score structure may substantially improve computational efficiency.
Acknowledgments
We acknowledge funding from NSF grant DMS-2413074 (Xiaozhu Zhang and Armeen Taeb). Rahul Mazumder acknowledges funding from Office of Naval Research (N000142512504).
References
Details on MIP algorithms
MIP formulations for both tasks
The procedure (8) can be formulated as a convex MIP similarly as in Section 3:
min { β e ∈ R p } e ∈ E g ∈ { 0 , 1 } p ∑ e ∈ E ω e ∥ Y e − ∑ k = 1 p X k e β k e ∥ n e 2 + γ ∑ e , f ∈ E e < f ω e , f [ ∥ β e − β f ∥ 2 2 + ρ ( d e , f ( β e , β f ) ) 2 ] + λ n ∑ j = 1 p g j \min_{\{\beta^e\in\mathbb{R}^p\}_{e\in\mathcal{E}} \atop g\in\{0,1\}^p} \sum_{e\in\mathcal{E}}\omega^e\left\|Y^e-\sum_{k=1}^{p}X_k^e\beta_k^e\right\|_{n_e}^2+\gamma\sum_{e,f\in\mathcal{E}\atop e<f}\omega^{e,f}\left[\|\beta^e-\beta^f\|_2^2+\rho\left(d^{e,f}(\beta^e,\beta^f)\right)^2\right]+\lambda_n\sum_{j=1}^{p}g_j g ∈ { 0 , 1 } p { β e ∈ R p } e ∈ E min e ∈ E ∑ ω e Y e − k = 1 ∑ p X k e β k e n e 2 + γ e < f e , f ∈ E ∑ ω e , f [ ∥ β e − β f ∥ 2 2 + ρ ( d e , f ( β e , β f ) ) 2 ] + λ n j = 1 ∑ p g j
− M n g j ≤ β j e ≤ M n g j , ∀ e ∈ E , j ∈ [ p ] , (16) -M_ng_j\le\beta_j^e\le M_ng_j,\qquad\forall e\in\mathcal{E},\ j\in[p], \tag*{(16)} − M n g j ≤ β j e ≤ M n g j , ∀ e ∈ E , j ∈ [ p ] , (16)
− M n g j ≤ β j ≤ M n g j , ∀ j ∈ [ p ] , (17) -M_ng_j\le\beta_j\le M_ng_j,\qquad\forall j\in[p], \tag*{(17)} − M n g j ≤ β j ≤ M n g j , ∀ j ∈ [ p ] , (17)
− M ~ n ( 1 − g j ) ≤ ⟨ X j e , Y e − ∑ k = 1 p β k e X k e ⟩ n e ≤ M ~ n ( 1 − g j ) , ∀ e ∈ E , j ∈ [ p ] . (18) -\widetilde{M}_n(1-g_j)\le\left\langle X_j^e,Y^e-\sum_{k=1}^{p}\beta_k^eX_k^e\right\rangle_{n_e}\le\widetilde{M}_n(1-g_j),\qquad\forall e\in\mathcal{E},\ j\in[p]. \tag*{(18)} − M n ( 1 − g j ) ≤ ⟨ X j e , Y e − k = 1 ∑ p β k e X k e ⟩ n e ≤ M n ( 1 − g j ) , ∀ e ∈ E , j ∈ [ p ] . (18)
Here, the heterogeneity in the OLS risks is written as:
d e , f ( β e , β f ) : = ∥ Y e ∥ n e 2 − ∥ Y f ∥ n f 2 − ∑ j ∈ [ p ] β j e ⟨ X j e , Y e ⟩ n e + ∑ j ∈ [ p ] β j f ⟨ X j f , Y f ⟩ n f . d^{e,f}(\beta^e,\beta^f):=\|Y^e\|_{n_e}^2-\|Y^f\|_{n_f}^2-\sum_{j\in[p]}\beta_j^e\langle X_j^e,Y^e\rangle_{n_e}+\sum_{j\in[p]}\beta_j^f\langle X_j^f,Y^f\rangle_{n_f}. d e , f ( β e , β f ) := ∥ Y e ∥ n e 2 − ∥ Y f ∥ n f 2 − j ∈ [ p ] ∑ β j e ⟨ X j e , Y e ⟩ n e + j ∈ [ p ] ∑ β j f ⟨ X j f , Y f ⟩ n f .
Compared with (7), the additional selection constraints in (15b) ensure that the coefficient vector β \beta β appearing in the MSE term is restricted to the same set as the environment specific coefficients { β e } e ∈ E \{\beta^e\}_{e\in\mathcal{E}} { β e } e ∈ E .
Similarly, the procedure (9), and equivalently (14), can be formulated as the following convex MIP:
T ^ ( k ) = min { β e ∈ R p } e ∈ E g ∈ { 0 , 1 } p n ∗ ∑ e , f ∈ E e < f ω e , f [ ∥ β e − β f ∥ 2 2 + ρ ( d e , f ( β e , β f ) ) 2 ] , (19) \widehat{T}(k)=\min_{\{\beta^e\in\mathbb{R}^p\}_{e\in\mathcal{E}}\atop g\in\{0,1\}^p}n^*\sum_{e,f\in\mathcal{E}\atop e<f}\omega^{e,f}\left[\|\beta^e-\beta^f\|_2^2+\rho\left(d^{e,f}(\beta^e,\beta^f)\right)^2\right], \tag*{(19)} T ( k ) = g ∈ { 0 , 1 } p { β e ∈ R p } e ∈ E min n ∗ e < f e , f ∈ E ∑ ω e , f [ ∥ β e − β f ∥ 2 2 + ρ ( d e , f ( β e , β f ) ) 2 ] , (19)
β j e = 0 if g j = 0 , ∀ e ∈ E , j ∈ [ p ] , (20) \beta_j^e=0\text{ if }g_j=0,\qquad\forall e\in\mathcal{E},\ j\in[p], \tag*{(20)} β j e = 0 if g j = 0 , ∀ e ∈ E , j ∈ [ p ] , (20)
⟨ X j e , Y e − ∑ k = 1 p β k e X k e ⟩ n e = 0 if g j = 1 , ∀ e ∈ E , j ∈ [ p ] , (21) \left\langle X_j^e,Y^e-\sum_{k=1}^{p}\beta_k^eX_k^e\right\rangle_{n_e}=0\text{ if }g_j=1,\qquad\forall e\in\mathcal{E},\ j\in[p], \tag*{(21)} ⟨ X j e , Y e − k = 1 ∑ p β k e X k e ⟩ n e = 0 if g j = 1 , ∀ e ∈ E , j ∈ [ p ] , (21)
g k = 0. (22) g_k=0. \tag*{(22)} g k = 0. (22)
The constraints (16b) and (16c) have the same interpretations as (7a) and (7b), respectively. The constraint (16d) prevents feature X k X_k X k from being selected.
A heuristic coordinate descent
We focus on the optimization problem (15); the coordinate descent algorithm for (16) can be developed similarly and is simpler.
First, we note that, for any fixed support S S S , the objective function in (15) can be evaluated in closed form, and we denote its value by f ( S ) f(S) f ( S ) . Specifically, let β ^ e , S \widehat{\beta}^{e,S} β e , S denote the environment specific OLS solution and define the pooled OLS solution as
β ^ S : = arg min β ~ ∈ R p , supp ( β ~ ) ⊆ S ∑ e ∈ E ω e E ^ [ ( Y e − β ~ ⊤ X e ) 2 ] . \widehat{\beta}^{S}:=\underset{\widetilde{\beta}\in\mathbb{R}^p,\ \operatorname{supp}(\widetilde{\beta})\subseteq S}{\arg\min}\sum_{e\in\mathcal{E}}\omega^e\widehat{\mathbb{E}}\left[\left(Y^e-\widetilde{\beta}^{\top}X^e\right)^2\right]. β S := β ∈ R p , supp ( β ) ⊆ S arg min e ∈ E ∑ ω e E [ ( Y e − β ⊤ X e ) 2 ] .
Then the objective value depends only on the environment specific and pooled OLS solutions and can be written as
f ( S ) = ∑ e ∈ E ω e ∥ Y e − ∑ k = 1 p X k e β ^ k S ∥ n e 2 + γ ∑ e , f ∈ E e < f ω e , f ∥ β ^ e , S − β ^ f , S ∥ 2 2 + γ ρ ∑ e , f ∈ E e < f ω e , f ( ν ^ e , S − ν ^ f , S ) 2 + λ n ∣ S ∣ . f(S)=\sum_{e\in\mathcal{E}}\omega^e\left\|Y^e-\sum_{k=1}^{p}X_k^e\widehat{\beta}_k^S\right\|_{n_e}^2+\gamma\sum_{e,f\in\mathcal{E}\atop e<f}\omega^{e,f}\|\widehat{\beta}^{e,S}-\widehat{\beta}^{f,S}\|_2^2+\gamma\rho\sum_{e,f\in\mathcal{E}\atop e<f}\omega^{e,f}\left(\widehat{\nu}^{e,S}-\widehat{\nu}^{f,S}\right)^2+\lambda_n|S|. f ( S ) = e ∈ E ∑ ω e Y e − k = 1 ∑ p X k e β k S n e 2 + γ e < f e , f ∈ E ∑ ω e , f ∥ β e , S − β f , S ∥ 2 2 + γ ρ e < f e , f ∈ E ∑ ω e , f ( ν e , S − ν f , S ) 2 + λ n ∣ S ∣.
Consequently, f ( S ) f(S) f ( S ) can be evaluated efficiently as long as the environment specific and pooled OLS solutions can be updated efficiently. In a coordinate descent procedure, each candidate move modifies the current support by only one feature, either adding a feature k ∉ S k\notin S k ∈ / S or removing a feature j ∈ S j\in S j ∈ S . The corresponding inverse Gram matrices and OLS coefficients can therefore be updated using rank one formulas, as described in Algorithms 2 and 3. As a result, matrix inversion is required only once for the initial support, while subsequent candidate supports can be evaluated through inexpensive rank one updates.
Our coordinate descent algorithm updates the support one feature at a time. For each j ∈ [ p ] j \in[p] j ∈ [ p ] , if j ∈ S j \in S j ∈ S , the algorithm removes j j j whenever the updated support S ∖ { j } S \setminus\{j\} S ∖ { j } yields a smaller objective value; otherwise, the support remains unchanged. Similarly, if j ∉ S j \notin S j ∈ / S , the algorithm adds j j j whenever S ∪ { j } S \cup\{j\} S ∪ { j } decreases the objective value. The procedure cycles through the coordinates until no addition or removal of any feature further improves the objective.
When the number of variables p p p is large, repeatedly cycling through all features can be computationally expensive. To accelerate the procedure, we adopt an active set strategy [2 , 8 ] . We begin with an initial active set A a c t i v e ⊆ [ p ] \mathcal{A}_{\mathrm{active}} \subseteq[p] A active ⊆ [ p ] and run coordinate descent restricted to this set, imposing
β j e = 0 and β j = 0 , ∀ j ∉ A a c t i v e , e ∈ E . \beta_j^e = 0 \quad\text{and} \quad\beta_j = 0,\qquad\forall j \notin\mathcal{A}_{\mathrm{active}},\quad e \in\mathcal{E}. β j e = 0 and β j = 0 , ∀ j ∈ / A active , e ∈ E .
Once coordinate descent converges on the restricted problem, we perform one full pass over all p p p features. If no coordinate update decreases the objective, then the current solution is also a coordinatewise local optimum of the original problem. Otherwise, we enlarge the active set to include the newly selected features,
A a c t i v e ← A a c t i v e ∪ { j : β j e ≠ 0 for some e ∈ E or β j ≠ 0 } , \mathcal{A}_{\mathrm{active}} \leftarrow\mathcal{A}_{\mathrm{active}} \cup\{j : \beta_j^e \ne0 \text{ for some } e \in\mathcal{E} \text{ or } \beta_j \ne0\}, A active ← A active ∪ { j : β j e = 0 for some e ∈ E or β j = 0 } ,
and repeat the procedure.
Finally, to further improve the quality of the solution obtained by coordinate descent, we optionally apply a local search procedure [2 , 8 ] . Starting from the converged support S S S , we consider all swaps between two features, in the form S ∖ { j } ∪ { k } S \setminus\{j\} \cup\{k\} S ∖ { j } ∪ { k } , where j ∈ S j \in S j ∈ S and k ∉ S k \notin S k ∈ / S . If any such swap yields a smaller objective value, we update the support accordingly and restart the coordinate descent procedure. If no improving swap is found, the algorithm terminates at the current solution.
The full algorithm is provided in Algorithms 4, 5 and 6.
Enumerating top-k k k invariant sets
As discussed in Remark 1, the top-k k k most invariant predictor sets can be obtained by solving a sequence of MIP problems. Suppose that we have obtained L < k L < k L < k distinct predictor sets S ^ 1 , … , S ^ L \widehat{S}^{1}, \ldots, \widehat{S}^{L} S 1 , … , S L , ordered by their empirical invariance scores. To obtain the next set, we solve
S ^ L + 1 ∈ arg min S ⊆ [ p ] I ^ ( S ) s.t. S ≠ S ^ ℓ , ∀ ℓ ∈ [ L ] . \widehat{S}^{L+1} \in\arg\min_{S \subseteq[p]} \widehat{I}(S) \quad\text{s.t.} \quad S \ne\widehat{S}^{\ell},\quad\forall\ell\in[L]. S L + 1 ∈ arg S ⊆ [ p ] min I ( S ) s.t. S = S ℓ , ∀ ℓ ∈ [ L ] .
Recall that the binary variables g j ∈ { 0 , 1 } g_j \in\{0,1\} g j ∈ { 0 , 1 } indicate whether predictor j j j is selected. The constraint S ≠ S ^ ℓ S \ne\widehat{S}^{\ell} S = S ℓ can be expressed as
∑ j ∈ S ^ ℓ ( 1 − g j ) + ∑ j ∉ S ^ ℓ g j ≥ 1 , ∀ ℓ ∈ [ L ] . (23) \sum_{j \in\widehat{S}^{\ell}} (1-g_j) + \sum_{j \notin\widehat{S}^{\ell}} g_j \ge1,\qquad\forall\ell\in[L]. \tag*{(23)} j ∈ S ℓ ∑ ( 1 − g j ) + j ∈ / S ℓ ∑ g j ≥ 1 , ∀ ℓ ∈ [ L ] . (23)
This constraint requires the new predictor set to differ from S ^ ℓ \widehat{S}^{\ell} S ℓ in at least one variable. Since these constraints are linear in the binary variables, they can be directly added to the MIP formulation of the invariance score without affecting its convexity. By successively adding the exclusion constraints (23) and resolving the MIP, we obtain the top-k k k distinct predictor sets in increasing order of their empirical invariance scores.
Algorithm 2 rank one update when adding a feature
Input: Current support S S S , feature j ∉ S j \notin S j ∈ / S , OLS coefficients { β ^ e , S } e ∈ E \{\widehat{\beta}^{e,S}\}_{e\in\mathcal{E}} { β e , S } e ∈ E and β ^ S \widehat{\beta}^{S} β S , and inverse Gram matrices { ( G S e ) − 1 } e ∈ E \{(G_{S}^{e})^{-1}\}_{e\in\mathcal{E}} {( G S e ) − 1 } e ∈ E and ( G S p o o l ) − 1 (G_{S}^{\mathrm{pool}})^{-1} ( G S pool ) − 1 , where
G S e : = ( X S e ) T X S e , G S p o o l : = ∑ e ∈ E ω e n e ( X S e ) T X S e . G_{S}^{e} := (X_{S}^{e})^{\mathsf{T}}X_{S}^{e}, \qquad G_{S}^{\mathrm{pool}} := \sum_{e\in\mathcal{E}}\frac{\omega^{e}}{n^{e}}(X_{S}^{e})^{\mathsf{T}}X_{S}^{e}. G S e := ( X S e ) T X S e , G S pool := e ∈ E ∑ n e ω e ( X S e ) T X S e . Output: Updated inverse Gram matrices and OLS coefficients on R = S ∪ { j } R=S\cup\{j\} R = S ∪ { j } .
for e ∈ E e\in\mathcal{E} e ∈ E do
Compute
c e = ( X S e ) T X j e , d e = X j e T X j e , z e = X j e T Y e . c^{e}=(X_{S}^{e})^{\mathsf{T}}X_{j}^{e}, \qquad d^{e}=X_{j}^{e\mathsf{T}}X_{j}^{e}, \qquad z^{e}=X_{j}^{e\mathsf{T}}Y^{e}. c e = ( X S e ) T X j e , d e = X j e T X j e , z e = X j e T Y e . Compute
u e = ( G S e ) − 1 c e , δ e = d e − ( c e ) T u e . u^{e}=(G_{S}^{e})^{-1}c^{e}, \qquad\delta^{e}=d^{e}-(c^{e})^{\mathsf{T}}u^{e}. u e = ( G S e ) − 1 c e , δ e = d e − ( c e ) T u e . Update the inverse Gram matrix:
( G R e ) − 1 = [ ( G S e ) − 1 + u e ( u e ) T δ e − u e δ e − ( u e ) T δ e 1 δ e ] . (G_{R}^{e})^{-1}=
\begin{bmatrix}
(G_{S}^{e})^{-1}+\dfrac{u^{e}(u^{e})^{\mathsf{T}}}{\delta^{e}} & -\dfrac{u^{e}}{\delta^{e}}\\
-\dfrac{(u^{e})^{\mathsf{T}}}{\delta^{e}} & \dfrac{1}{\delta^{e}}
\end{bmatrix}. ( G R e ) − 1 = ( G S e ) − 1 + δ e u e ( u e ) T − δ e ( u e ) T − δ e u e δ e 1 . Compute the coefficient of the newly added feature, and update:
β ^ j e , R = z e − ( c e ) T β ^ e , S δ e , β ^ e , R = [ β ^ e , S − u e β ^ j e , R β ^ j e , R ] . \widehat{\beta}_{j}^{e,R}=\frac{z^{e}-(c^{e})^{\mathsf{T}}\widehat{\beta}^{e,S}}{\delta^{e}}, \qquad
\widehat{\beta}^{e,R}=
\begin{bmatrix}
\widehat{\beta}^{e,S}-u^{e}\widehat{\beta}_{j}^{e,R}\\
\widehat{\beta}_{j}^{e,R}
\end{bmatrix}. β j e , R = δ e z e − ( c e ) T β e , S , β e , R = [ β e , S − u e β j e , R β j e , R ] . end for
Compute the pooled quantities
c p o o l = ∑ e ∈ E ω e X S e T X j e n e , d p o o l = ∑ e ∈ E ω e X j e T X j e n e , z p o o l = ∑ e ∈ E ω e X j e T Y e n e . c^{\mathrm{pool}}=\sum_{e\in\mathcal{E}}\omega^{e}\frac{X_{S}^{e\mathsf{T}}X_{j}^{e}}{n^{e}}, \qquad
d^{\mathrm{pool}}=\sum_{e\in\mathcal{E}}\omega^{e}\frac{X_{j}^{e\mathsf{T}}X_{j}^{e}}{n^{e}}, \qquad
z^{\mathrm{pool}}=\sum_{e\in\mathcal{E}}\omega^{e}\frac{X_{j}^{e\mathsf{T}}Y^{e}}{n^{e}}. c pool = e ∈ E ∑ ω e n e X S e T X j e , d pool = e ∈ E ∑ ω e n e X j e T X j e , z pool = e ∈ E ∑ ω e n e X j e T Y e . Compute
u p o o l = ( G S p o o l ) − 1 c p o o l , δ p o o l = d p o o l − ( c p o o l ) T u p o o l . u^{\mathrm{pool}}=(G_{S}^{\mathrm{pool}})^{-1}c^{\mathrm{pool}}, \qquad
\delta^{\mathrm{pool}}=d^{\mathrm{pool}}-(c^{\mathrm{pool}})^{\mathsf{T}}u^{\mathrm{pool}}. u pool = ( G S pool ) − 1 c pool , δ pool = d pool − ( c pool ) T u pool . Update the inverse Gram matrix:
( G R p o o l ) − 1 = [ ( G S p o o l ) − 1 + u p o o l ( u p o o l ) T δ p o o l − u p o o l δ p o o l − ( u p o o l ) T δ p o o l 1 δ p o o l ] . (G_{R}^{\mathrm{pool}})^{-1}=
\begin{bmatrix}
(G_{S}^{\mathrm{pool}})^{-1}+\dfrac{u^{\mathrm{pool}}(u^{\mathrm{pool}})^{\mathsf{T}}}{\delta^{\mathrm{pool}}} & -\dfrac{u^{\mathrm{pool}}}{\delta^{\mathrm{pool}}}\\
-\dfrac{(u^{\mathrm{pool}})^{\mathsf{T}}}{\delta^{\mathrm{pool}}} & \dfrac{1}{\delta^{\mathrm{pool}}}
\end{bmatrix}. ( G R pool ) − 1 = ( G S pool ) − 1 + δ pool u pool ( u pool ) T − δ pool ( u pool ) T − δ pool u pool δ pool 1 . Compute the coefficient of the newly added feature, and update:
β ^ j R = z p o o l − ( c p o o l ) T β ^ S δ p o o l , β ^ R = [ β ^ S − u p o o l β ^ j R β ^ j R ] . \widehat{\beta}_{j}^{R}=\frac{z^{\mathrm{pool}}-(c^{\mathrm{pool}})^{\mathsf{T}}\widehat{\beta}^{S}}{\delta^{\mathrm{pool}}}, \qquad
\widehat{\beta}^{R}=
\begin{bmatrix}
\widehat{\beta}^{S}-u^{\mathrm{pool}}\widehat{\beta}_{j}^{R}\\
\widehat{\beta}_{j}^{R}
\end{bmatrix}. β j R = δ pool z pool − ( c pool ) T β S , β R = [ β S − u pool β j R β j R ] . return { ( G R e ) − 1 , β ^ e , R } e ∈ E \{(G_{R}^{e})^{-1},\widehat{\beta}^{e,R}\}_{e\in\mathcal{E}} {( G R e ) − 1 , β e , R } e ∈ E and { ( G R p o o l ) − 1 , β ^ R } \{(G_{R}^{\mathrm{pool}})^{-1},\widehat{\beta}^{R}\} {( G R pool ) − 1 , β R } .
Algorithm 3 rank one update when removing a feature
Input: Current support S S S , feature j ∈ S j \in S j ∈ S , OLS coefficients { β ^ e , S } e ∈ E \{\widehat{\beta}^{e,S}\}_{e\in\mathcal{E}} { β e , S } e ∈ E and β ^ S \widehat{\beta}^{S} β S , and inverse Gram matrices { ( G S e ) − 1 } e ∈ E \{(G^{e}_{S})^{-1}\}_{e\in\mathcal{E}} {( G S e ) − 1 } e ∈ E and ( G S p o o l ) − 1 (G^{\mathrm{pool}}_{S})^{-1} ( G S pool ) − 1 .
Output: Updated inverse Gram matrices and OLS coefficients on R = S ∖ { j } R=S\setminus\{j\} R = S ∖ { j } .
1: Reorder the coordinates, if necessary, so that feature j j j is the last coordinate of S S S . 2: for e ∈ E e \in\mathcal{E} e ∈ E do 3: Partition
( G S e ) − 1 = [ A e b e ( b e ) T d e ] , β ^ e , S = [ β ^ R e , S β ^ j e , S ] . (G^{e}_{S})^{-1} =
\begin{bmatrix}
A^{e} & b^{e} \\
(b^{e})^{\mathsf{T}} & d^{e}
\end{bmatrix},
\qquad
\widehat{\beta}^{e,S} =
\begin{bmatrix}
\widehat{\beta}^{e,S}_{R} \\
\widehat{\beta}^{e,S}_{j}
\end{bmatrix}. ( G S e ) − 1 = [ A e ( b e ) T b e d e ] , β e , S = [ β R e , S β j e , S ] .
4: Update the inverse Gram matrix and the OLS coefficient:
( G R e ) − 1 = A e − b e ( b e ) T d e , β ^ e , R = β ^ R e , S − b e d e β ^ j e , S . (G^{e}_{R})^{-1} = A^{e} - \frac{b^{e}(b^{e})^{\mathsf{T}}}{d^{e}},
\qquad
\widehat{\beta}^{e,R} = \widehat{\beta}^{e,S}_{R} - \frac{b^{e}}{d^{e}}\widehat{\beta}^{e,S}_{j}. ( G R e ) − 1 = A e − d e b e ( b e ) T , β e , R = β R e , S − d e b e β j e , S .
5: end for 6: Partition the pooled quantities as
( G S p o o l ) − 1 = [ A p o o l b p o o l ( b p o o l ) T d p o o l ] , β ^ S = [ β ^ R S β ^ j S ] . (G^{\mathrm{pool}}_{S})^{-1} =
\begin{bmatrix}
A^{\mathrm{pool}} & b^{\mathrm{pool}} \\
(b^{\mathrm{pool}})^{\mathsf{T}} & d^{\mathrm{pool}}
\end{bmatrix},
\qquad
\widehat{\beta}^{S} =
\begin{bmatrix}
\widehat{\beta}^{S}_{R} \\
\widehat{\beta}^{S}_{j}
\end{bmatrix}. ( G S pool ) − 1 = [ A pool ( b pool ) T b pool d pool ] , β S = [ β R S β j S ] .
7: Update
( G R p o o l ) − 1 = A p o o l − b p o o l ( b p o o l ) T d p o o l , β ^ R = β ^ R S − b p o o l d p o o l β ^ j S . (G^{\mathrm{pool}}_{R})^{-1} = A^{\mathrm{pool}} - \frac{b^{\mathrm{pool}}(b^{\mathrm{pool}})^{\mathsf{T}}}{d^{\mathrm{pool}}},
\qquad
\widehat{\beta}^{R} = \widehat{\beta}^{S}_{R} - \frac{b^{\mathrm{pool}}}{d^{\mathrm{pool}}}\widehat{\beta}^{S}_{j}. ( G R pool ) − 1 = A pool − d pool b pool ( b pool ) T , β R = β R S − d pool b pool β j S .
8: return { ( ( G R e ) − 1 , β ^ e , R ) } e ∈ E \{((G^{e}_{R})^{-1},\widehat{\beta}^{e,R})\}_{e\in\mathcal{E}} {(( G R e ) − 1 , β e , R ) } e ∈ E and { ( ( G R p o o l ) − 1 , β ^ R ) } \{((G^{\mathrm{pool}}_{R})^{-1},\widehat{\beta}^{R})\} {(( G R pool ) − 1 , β R )} .
Algorithm 4 CoordinateUpdate: Coordinate update on a prescribed set
Input: Current support S S S , index set A ⊆ [ p ] \mathcal{A}\subseteq[p] A ⊆ [ p ] , objective function f f f , and indicator converge.
Output: Updated support S S S and indicator changed.
1: changed ← \leftarrow ← False. 2: repeat 3: updated ← \leftarrow ← False. 4: for j ∈ A j\in\mathcal{A} j ∈ A do 5: if j ∈ S j\in S j ∈ S then 6: S ′ ← S ∖ { j } S'\leftarrow S\setminus\{j\} S ′ ← S ∖ { j } . 7: else 8: S ′ ← S ∪ { j } S'\leftarrow S\cup\{j\} S ′ ← S ∪ { j } . 9: end if 10: Evaluate f ( S ′ ) f(S') f ( S ′ ) using the corresponding rank one update. 11: if f ( S ′ ) < f ( S ) f(S')<f(S) f ( S ′ ) < f ( S ) then 12: S ← S ′ S\leftarrow S' S ← S ′ and f ( S ) ← f ( S ′ ) f(S)\leftarrow f(S') f ( S ) ← f ( S ′ ) . 13: Update the inverse Gram matrices and OLS coefficients. 14: updated ← \leftarrow ← True. 15: changed ← \leftarrow ← True. 16: end if 17: end for 18: until updated = False or converge = False 19: return S S S and changed.
Algorithm 5 LocalSearch: Two-feature local search
Input: Current support S S S and objective function f f f .
Output: Updated support S S S and indicator improved .
improved ← \leftarrow ← False .
for j ∈ S j \in S j ∈ S do
for k ∉ S k \notin S k ∈ / S do
S ′ ← ( S ∖ { j } ) ∪ { k } S' \leftarrow(S \setminus\{j\}) \cup\{k\} S ′ ← ( S ∖ { j }) ∪ { k } .
Evaluate f ( S ′ ) f(S') f ( S ′ ) using a rank one removal followed by a rank one addition.
if f ( S ′ ) < f ( S ) f(S') < f(S) f ( S ′ ) < f ( S ) then
S ← S ′ S \leftarrow S' S ← S ′ and f ( S ) ← f ( S ′ ) f(S) \leftarrow f(S') f ( S ) ← f ( S ′ ) .
Update the inverse Gram matrices and OLS coefficients.
improved ← \leftarrow ← True .
return S S S and improved .
end if
end for
end for
return S S S and improved .
Algorithm 6 CoordinateDescent: Coordinate descent
Input: Initial support S S S , initial active set A a c t i v e ⊇ S \mathcal{A}_{\mathrm{active}} \supseteq S A active ⊇ S , and indicator local_search .
Output: Estimated support S ^ \hat{S} S ^ .
Compute the inverse Gram matrices and OLS coefficients on the initial support S S S .
repeat
( S , ⋅ ) ← CoordinateUpdate ( S , A a c t i v e , f , True ) (S, \cdot) \leftarrow\operatorname{CoordinateUpdate}(S, \mathcal{A}_{\mathrm{active}}, f, \textbf{True}) ( S , ⋅ ) ← CoordinateUpdate ( S , A active , f , True ) .
( S , changed ) ← CoordinateUpdate ( S , [ p ] , f , False ) (S, \textbf{changed}) \leftarrow\operatorname{CoordinateUpdate}(S, [p], f, \textbf{False}) ( S , changed ) ← CoordinateUpdate ( S , [ p ] , f , False ) .
if changed = = = True then
A a c t i v e ← A a c t i v e ∪ S \mathcal{A}_{\mathrm{active}} \leftarrow\mathcal{A}_{\mathrm{active}} \cup S A active ← A active ∪ S .
end if
until changed = = = False
if local_search = = = True then
repeat
( S , improved ) ← LocalSearch ( S , f ) (S, \textbf{improved}) \leftarrow\operatorname{LocalSearch}(S, f) ( S , improved ) ← LocalSearch ( S , f ) .
if improved = = = True then
A a c t i v e ← A a c t i v e ∪ S \mathcal{A}_{\mathrm{active}} \leftarrow\mathcal{A}_{\mathrm{active}} \cup S A active ← A active ∪ S .
repeat
( S , ⋅ ) ← CoordinateUpdate ( S , A a c t i v e , f , True ) (S, \cdot) \leftarrow\operatorname{CoordinateUpdate}(S, \mathcal{A}_{\mathrm{active}}, f, \textbf{True}) ( S , ⋅ ) ← CoordinateUpdate ( S , A active , f , True ) .
( S , changed ) ← CoordinateUpdate ( S , [ p ] , f , False ) (S, \textbf{changed}) \leftarrow\operatorname{CoordinateUpdate}(S, [p], f, \textbf{False}) ( S , changed ) ← CoordinateUpdate ( S , [ p ] , f , False ) .
A a c t i v e ← A a c t i v e ∪ S \mathcal{A}_{\mathrm{active}} \leftarrow\mathcal{A}_{\mathrm{active}} \cup S A active ← A active ∪ S .
until changed = = = False
end if
until improved = = = False
end if
S ^ ← S \hat{S} \leftarrow S S ^ ← S .
return S ^ \hat{S} S ^ .
More on the Bootstrap procedure
Procedures that estimate S k ∘ S_k^\circ S k ∘
In § 5.3 , we briefly introduced two procedures for constructing a super-collection of S k S_k S k . The first procedure, which can be described in Algorithm 7, collects M ˉ \bar{M} M ˉ candidate sets by repeatedly resampling the observations in D 1 D_1 D 1 with replacement. The intuition is that these bootstrap perturbations induce sufficient variation in the selected models to recover the invariant sets in S k S_k S k . This procedure, however, is largely heuristic and does not come with strong theoretical guarantees.
Algorithm 7 Construction of S k ∘ S_k^\circ S k ∘ via perturbation
Input: Dataset D 1 D_1 D 1 , feature index k k k , and maximum number of sets M ˉ \bar{M} M ˉ .
Output: A collection of sets S k ∘ S_k^\circ S k ∘ .
Initialize S k ∘ ← ∅ S_k^\circ\leftarrow\emptyset S k ∘ ← ∅ .
for m = 1 , … , M ˉ m = 1,\ldots,\bar{M} m = 1 , … , M ˉ do
Construct a bootstrap sample D 1 ( m ) D_1^{(m)} D 1 ( m ) by drawing observations from D 1 D_1 D 1 with replacement within each environment.
Solve the optimization problem (16) using D 1 ( m ) D_1^{(m)} D 1 ( m ) .
Let S ^ ( m ) \widehat{S}^{(m)} S ( m ) denote the set encoded by the optimal solution.
Update S k ∘ ← S k ∘ ∪ { S ^ ( m ) } S_k^\circ\leftarrow S_k^\circ\cup\{\widehat{S}^{(m)}\} S k ∘ ← S k ∘ ∪ { S ( m ) } .
end for
return S k ∘ S_k^\circ S k ∘ .
The second procedure sequentially retrieves sets S S S in ascending order of the objective value in (16), motivated by the intuition that smaller objective values correspond to more invariant models. Consequently, the top-M ˉ \bar{M} M ˉ sets are expected to contain the invariant sets in S k S_k S k . The details of this procedure are presented in Algorithm 8. Notably, such a sequential retrieval based on the ordering of objective values would not be readily available without the mixed-integer programming formulation, particularly the exclusion constraints imposed in line 3. Furthermore, under the Gaussian model Y e ∣ X S e ∼ N ( X S e T β S e , ν S e ) Y^e \mid X_S^e \sim\mathcal{N}(X_S^{e\mathsf{T}}\beta_S^e,\nu_S^e) Y e ∣ X S e ∼ N ( X S e T β S e , ν S e ) , both requirements for the testing procedure, S k ⊆ S k ∘ S_k \subseteq S_k^\circ S k ⊆ S k ∘ and S ^ ∈ S k \widehat{S} \in S_k S ∈ S k , can be satisfied with probability tending to one. The proof of Lemma 5 is provided in Appendix § 17 .§ 1 .
Lemma 5. Suppose that D 1 D_1 D 1 and D 2 D_2 D 2 each contain half of the original dataset, and ∣ S k ∣ ≤ M ˉ |S_k| \le\bar{M} ∣ S k ∣ ≤ M ˉ . Under the Gaussian model Y e ∣ X S e ∼ N ( X S e T β S e , ν S e ) Y^e \mid X_S^e \sim\mathcal{N}(X_S^{e\mathsf{T}}\beta_S^e,\nu_S^e) Y e ∣ X S e ∼ N ( X S e T β S e , ν S e ) for all e ∈ E e \in\mathcal{E} e ∈ E , for the set S ^ \widehat{S} S encoded by the optimal solution to (16) and the collection S k ∘ S_k^\circ S k ∘ obtained from Algorithm 8, we have
P ( { S k ⊆ S k ∘ } ∩ { S ^ ∈ S k } ) → 1 , \mathbb{P}\left(\{S_k \subseteq S_k^\circ\} \cap\{\widehat{S} \in S_k\}\right) \to1, P ( { S k ⊆ S k ∘ } ∩ { S ∈ S k } ) → 1 , as the sample sizes min e ∈ E n e → ∞ \min_{e\in\mathcal{E}} n^e \to\infty min e ∈ E n e → ∞ .
Gaining more power
Although the two components of the invariance score I ( S ) I(S) I ( S ) , namely the discrepancies in the OLS coefficients and residual variances, are sufficient to characterize LSCR invariance, incorporating additional measures of discrepancy across environments can further improve the power of the resulting test. We additionally consider the following two conditions for a model β \beta β : (i) E [ Y e − X e T β ] = E [ Y f − X f T β ] \mathbb{E}[Y^e-X^{e\mathsf{T}}\beta]=\mathbb{E}[Y^f-X^{f\mathsf{T}}\beta] E [ Y e − X e T β ] = E [ Y f − X f T β ] for all e , f ∈ E e,f\in\mathcal{E} e , f ∈ E ; and (ii) E [ X j e ( Y e − X e T β ) ] = E [ X j f ( Y f − X f T β ) ] \mathbb{E}[X_j^e(Y^e-X^{e\mathsf{T}}\beta)]=\mathbb{E}[X_j^f(Y^f-X^{f\mathsf{T}}\beta)] E [ X j e ( Y e − X e T β )] = E [ X j f ( Y f − X f T β )] for all j ∈ supp ( β ) j\in\operatorname{supp}(\beta) j ∈ supp ( β ) and e , f ∈ E e,f\in\mathcal{E} e , f ∈ E . Both conditions are automatically satisfied by any LSCR-invariant model and therefore do not alter the collection S L S C R ( E ) S_{\mathrm{LSCR}}(\mathcal{E}) S LSCR ( E ) characterized by the LSCR principle. Nevertheless, incorporating their empirical discrepancies into the invariance score can provide additional information for distinguishing non-invariant models, thereby improving the power to detect plausible causal predictors.
Algorithm 8 Sequential retrieval of the top-M ˉ \bar{M} M ˉ sets
Input: Dataset D 1 \mathcal{D}_1 D 1 , feature index k k k , and maximum number of sets M ˉ \bar{M} M ˉ .
Output: A collection of sets S k ∘ \mathcal{S}_k^\circ S k ∘ .
Initialize S k ∘ ← ∅ \mathcal{S}_k^\circ\gets\varnothing S k ∘ ← ∅ .
for m = 1 , … , M ˉ m = 1, \ldots, \bar{M} m = 1 , … , M ˉ do
Solve the mixed-integer program (16) on D 1 \mathcal{D}_1 D 1 , together with exclusion constraints
∑ j ∈ S ( 1 − g j ) + ∑ j ∉ S g j ≥ 1 , ∀ S ∈ S k ∘ . \sum_{j\in S}(1-g_j)+\sum_{j\notin S}g_j \ge1,\quad\forall S\in\mathcal{S}_k^\circ. j ∈ S ∑ ( 1 − g j ) + j ∈ / S ∑ g j ≥ 1 , ∀ S ∈ S k ∘ .
Let S ^ ( m ) \widehat{S}^{(m)} S ( m ) denote the set encoded by the optimal solution.
Update S k ∘ ← S k ∘ ∪ { S ^ ( m ) } \mathcal{S}_k^\circ\gets\mathcal{S}_k^\circ\cup\{\widehat{S}^{(m)}\} S k ∘ ← S k ∘ ∪ { S ( m ) } .
end for
return S k ∘ \mathcal{S}_k^\circ S k ∘ .
The corresponding MIP formulation for testing H 0 , k 0 H_{0,k_0} H 0 , k 0 can be modified as:
min β ∈ R p , { β e ∈ R p } e ∈ E g ∈ { 0 , 1 } p , { t l ≥ 0 } l = 1 4 n ⋆ ( γ 1 t 1 + γ 2 t 2 + γ 3 t 3 + γ 4 t 4 ) \min_{\substack{\beta\in\mathbb{R}^{p},\ \{\beta^{e}\in\mathbb{R}^{p}\}_{e\in\mathcal{E}}\\ g\in\{0,1\}^{p},\ \{t_l\geq0\}_{l=1}^{4}}} n^\star(\gamma_1t_1+\gamma_2t_2+\gamma_3t_3+\gamma_4t_4) β ∈ R p , { β e ∈ R p } e ∈ E g ∈ { 0 , 1 } p , { t l ≥ 0 } l = 1 4 min n ⋆ ( γ 1 t 1 + γ 2 t 2 + γ 3 t 3 + γ 4 t 4 )
∑ e < f ω e , f ∥ β e − β f ∥ 2 2 ≤ t 1 , (24) \sum_{e<f}\omega^{e,f}\lVert\beta^{e}-\beta^{f}\rVert_2^2\leq t_1, \tag*{(24)} e < f ∑ ω e , f ∥ β e − β f ∥ 2 2 ≤ t 1 , (24)
∑ e < f ω e , f ( ∥ Y e ∥ n e 2 − ∥ Y f ∥ n f 2 − ∑ j = 1 p β j e ⟨ X j e , Y e ⟩ n e + ∑ j = 1 p β j f ⟨ X j f , Y f ⟩ n f ) 2 ≤ t 2 , (25) \sum_{e<f}\omega^{e,f}\left(\lVert Y^{e}\rVert_{n_e}^{2}-\lVert Y^{f}\rVert_{n_f}^{2}-\sum_{j=1}^{p}\beta_j^{e}\langle X_j^{e},Y^{e}\rangle_{n_e}+\sum_{j=1}^{p}\beta_j^{f}\langle X_j^{f},Y^{f}\rangle_{n_f}\right)^2\leq t_2, \tag*{(25)} e < f ∑ ω e , f ( ∥ Y e ∥ n e 2 − ∥ Y f ∥ n f 2 − j = 1 ∑ p β j e ⟨ X j e , Y e ⟩ n e + j = 1 ∑ p β j f ⟨ X j f , Y f ⟩ n f ) 2 ≤ t 2 , (25)
∑ e < f ω e , f ( ⟨ Y e − ∑ j = 1 p X j e β j , 1 ⟩ n e − ⟨ Y f − ∑ j = 1 p X j f β j , 1 ⟩ n f ) 2 ≤ t 3 , (26) \sum_{e<f}\omega^{e,f}\left(\left\langle Y^{e}-\sum_{j=1}^{p}X_j^{e}\beta_j,1\right\rangle_{n_e}-\left\langle Y^{f}-\sum_{j=1}^{p}X_j^{f}\beta_j,1\right\rangle_{n_f}\right)^2\leq t_3, \tag*{(26)} e < f ∑ ω e , f ⟨ Y e − j = 1 ∑ p X j e β j , 1 ⟩ n e − ⟨ Y f − j = 1 ∑ p X j f β j , 1 ⟩ n f 2 ≤ t 3 , (26)
∑ e < f ω e , f ( ⟨ Y e − ∑ j = 1 p X j e β j , X k e ⟩ n e − ⟨ Y f − ∑ j = 1 p X j f β j , X k f ⟩ n f ) 2 ≤ t 4 , if g k = 1 , (27) \sum_{e<f}\omega^{e,f}\left(\left\langle Y^{e}-\sum_{j=1}^{p}X_j^{e}\beta_j,X_k^{e}\right\rangle_{n_e}-\left\langle Y^{f}-\sum_{j=1}^{p}X_j^{f}\beta_j,X_k^{f}\right\rangle_{n_f}\right)^2\leq t_4,\quad\text{if }g_k=1, \tag*{(27)} e < f ∑ ω e , f ⟨ Y e − j = 1 ∑ p X j e β j , X k e ⟩ n e − ⟨ Y f − j = 1 ∑ p X j f β j , X k f ⟩ n f 2 ≤ t 4 , if g k = 1 , (27)
β j e = 0 if g j = 0 , ∀ e ∈ E , j ∈ [ p ] , (28) \beta_j^{e}=0\text{ if }g_j=0,\quad\forall e\in\mathcal{E},\ j\in[p], \tag*{(28)} β j e = 0 if g j = 0 , ∀ e ∈ E , j ∈ [ p ] , (28)
β j = 0 if g j = 0 , ∀ j ∈ [ p ] , (29) \beta_j=0\text{ if }g_j=0,\quad\forall j\in[p], \tag*{(29)} β j = 0 if g j = 0 , ∀ j ∈ [ p ] , (29)
⟨ X j e , Y e − ∑ k = 1 p β k e X k e ⟩ n e = 0 if g j = 1 , ∀ e ∈ E , j ∈ [ p ] , (30) \left\langle X_j^{e},Y^{e}-\sum_{k=1}^{p}\beta_k^{e}X_k^{e}\right\rangle_{n_e}=0\text{ if }g_j=1,\quad\forall e\in\mathcal{E},\ j\in[p], \tag*{(30)} ⟨ X j e , Y e − k = 1 ∑ p β k e X k e ⟩ n e = 0 if g j = 1 , ∀ e ∈ E , j ∈ [ p ] , (30)
∑ e ∈ E n e n ⋆ ⟨ X j e , Y e − ∑ k = 1 p β k X k e ⟩ n e = 0 if g j = 1 , j ∈ [ p ] , (31) \sum_{e\in\mathcal{E}}\frac{n_e}{n^\star}\left\langle X_j^{e},Y^{e}-\sum_{k=1}^{p}\beta_kX_k^{e}\right\rangle_{n_e}=0\text{ if }g_j=1,\quad j\in[p], \tag*{(31)} e ∈ E ∑ n ⋆ n e ⟨ X j e , Y e − k = 1 ∑ p β k X k e ⟩ n e = 0 if g j = 1 , j ∈ [ p ] , (31)
g k 0 = 0. (32) g_{k_0}=0. \tag*{(32)} g k 0 = 0. (32)
For the objective function, the first two components coincide with those in formulation (16), through constraints (18a) and (18b). The last two components, introduced through (18c) and (18d), capture the additional discrepancies across environments in the residual mean, corresponding to condition (i), and in the cross moment between residual and active variables, corresponding to condition (ii). Constraints (18e) and (18g) ensure that each β e \beta^{e} β e is the environment specific OLS solution, whereas constraints (18f) and (18h) ensure that β \beta β is the pooled OLS solution based on the full dataset { ( X e , Y e ) } e ∈ E \{(X^{e},Y^{e})\}_{e\in\mathcal{E}} {( X e , Y e ) } e ∈ E of all environments. Importantly, the pooled OLS coefficient is used in the additional discrepancy terms. If the environment specific OLS coefficients were used instead, the corresponding empirical moment conditions would hold automatically by the OLS normal equations: for every active predictor j j j with g j = 1 g_j=1 g j = 1 , ⟨ X j e , Y e − ∑ k = 1 p β k e X k e ⟩ n e = 0 \langle X_j^{e},Y^{e}-\sum_{k=1}^{p}\beta_k^{e}X_k^{e}\rangle_{n_e}=0 ⟨ X j e , Y e − ∑ k = 1 p β k e X k e ⟩ n e = 0 , and ⟨ Y e − ∑ j = 1 p X j e β j e , 1 ⟩ n e = 0 \langle Y^e-\sum_{j=1}^{p}X_j^e\beta_j^e,\mathbf{1}\rangle_{n^e}=0 ⟨ Y e − ∑ j = 1 p X j e β j e , 1 ⟩ n e = 0 . In that case, these quantities would provide no additional information for detecting violations of invariance. Using the pooled OLS solution instead allows the last two components to capture meaningful discrepancies across environments and thereby potentially improve the power of the test.
Using this modified objective value as the test statistic, we accordingly adapt the multiplier bootstrap procedure in Algorithm 1. In line 4, we additionally compute the pooled OLS coefficient β ^ S \hat{\beta}^{S} β ^ S and
r ^ i e , S , p o o l = Y i e − ( X i e ) T β ^ S , Σ ^ S = ∑ e ∈ E n e n ∗ Σ ^ S e , B ∗ , S = ( Σ ^ S ) − 1 [ 1 n ∗ ∑ e ∈ E ∑ i = 1 n e ξ i e X i e r ^ i e , S , p o o l ] . \hat{r}_{i}^{e,S,\mathrm{pool}}=Y_i^e-(X_i^e)^{\mathsf{T}}\hat{\beta}^{S},\qquad\hat{\Sigma}_{S}=\sum_{e\in\mathcal{E}}\frac{n^e}{n^\ast}\hat{\Sigma}^{e}_{S},\qquad B^{\ast,S}=(\hat{\Sigma}_{S})^{-1}\left[\frac{1}{\sqrt{n^\ast}}\sum_{e\in\mathcal{E}}\sum_{i=1}^{n^e}\xi_i^eX_i^e\hat{r}_{i}^{e,S,\mathrm{pool}}\right]. r ^ i e , S , pool = Y i e − ( X i e ) T β ^ S , Σ ^ S = e ∈ E ∑ n ∗ n e Σ ^ S e , B ∗ , S = ( Σ ^ S ) − 1 [ n ∗ 1 e ∈ E ∑ i = 1 ∑ n e ξ i e X i e r ^ i e , S , pool ] .
M ∗ , e , S = 1 n e ∑ i = 1 n e ξ i e ( r ^ i e , S , p o o l − 1 n e ∑ i = 1 n e r ^ i e , S , p o o l ) , M^{\ast,e,S}=\frac{1}{\sqrt{n^e}}\sum_{i=1}^{n^e}\xi_i^e\left(\hat{r}_{i}^{e,S,\mathrm{pool}}-\frac{1}{n^e}\sum_{i=1}^{n^e}\hat{r}_{i}^{e,S,\mathrm{pool}}\right), M ∗ , e , S = n e 1 i = 1 ∑ n e ξ i e ( r ^ i e , S , pool − n e 1 i = 1 ∑ n e r ^ i e , S , pool ) ,
and
Q j ∗ , e , S = 1 n e ∑ i = 1 n e ξ i e ( X i j e r ^ i e , S , p o o l − 1 n e ∑ i = 1 n e X i j e r ^ i e , S , p o o l ) . Q_{j}^{\ast,e,S}=\frac{1}{\sqrt{n^e}}\sum_{i=1}^{n^e}\xi_i^e\left(X_{ij}^e\hat{r}_{i}^{e,S,\mathrm{pool}}-\frac{1}{n^e}\sum_{i=1}^{n^e}X_{ij}^e\hat{r}_{i}^{e,S,\mathrm{pool}}\right). Q j ∗ , e , S = n e 1 i = 1 ∑ n e ξ i e ( X ij e r ^ i e , S , pool − n e 1 i = 1 ∑ n e X ij e r ^ i e , S , pool ) .
In line 5, for each pair e < f e<f e < f in E \mathcal{E} E , we additionally compute
O n ∗ , e , f , S = n ∗ n e M ∗ , e , S − n ∗ n f M ∗ , f , S − ( 1 n e ∑ i = 1 n e X S , i e − 1 n f ∑ i = 1 n f X S , i f ) T B ∗ , S O_{n}^{\ast,e,f,S}=\sqrt{\frac{n^\ast}{n^e}}M^{\ast,e,S}-\sqrt{\frac{n^\ast}{n^f}}M^{\ast,f,S}-\left(\frac{1}{n^e}\sum_{i=1}^{n^e}X_{S,i}^e-\frac{1}{n^f}\sum_{i=1}^{n^f}X_{S,i}^f\right)^{\mathsf{T}}B^{\ast,S} O n ∗ , e , f , S = n e n ∗ M ∗ , e , S − n f n ∗ M ∗ , f , S − n e 1 i = 1 ∑ n e X S , i e − n f 1 i = 1 ∑ n f X S , i f T B ∗ , S
and
K n , j ∗ , e , f , S = n ∗ n e Q j ∗ , e , S − n ∗ n f Q j ∗ , f , S − ( 1 n e ∑ i = 1 n e X i j e X S , i e − 1 n f ∑ i = 1 n f X i j f X S , i f ) T B ∗ , S . K_{n,j}^{\ast,e,f,S}=\sqrt{\frac{n^\ast}{n^e}}Q_{j}^{\ast,e,S}-\sqrt{\frac{n^\ast}{n^f}}Q_{j}^{\ast,f,S}-\left(\frac{1}{n^e}\sum_{i=1}^{n^e}X_{ij}^eX_{S,i}^e-\frac{1}{n^f}\sum_{i=1}^{n^f}X_{ij}^fX_{S,i}^f\right)^{\mathsf{T}}B^{\ast,S}. K n , j ∗ , e , f , S = n e n ∗ Q j ∗ , e , S − n f n ∗ Q j ∗ , f , S − n e 1 i = 1 ∑ n e X ij e X S , i e − n f 1 i = 1 ∑ n f X ij f X S , i f T B ∗ , S .
Finally, the line 7 is modified as:
t k b ← max S ∈ S ∘ { ∑ e < f ω e , f ( γ 1 ∥ L n ∗ , e , f , S ∥ 2 2 + γ 2 ∣ V n ∗ , e , f , S ∣ 2 + γ 3 ∣ O n ∗ , e , f , S ∣ 2 ) + γ 4 max j ∈ S ∑ e < f ω e , f ∣ K n , j ∗ , e , f , S ∣ 2 } . t_k^b\leftarrow\max_{S\in\mathcal{S}^{\circ}}\left\{\sum_{e<f}\omega^{e,f}\left(\gamma_1\lVert L_n^{\ast,e,f,S}\rVert_2^2+\gamma_2\lvert V_n^{\ast,e,f,S}\rvert^2+\gamma_3\lvert O_n^{\ast,e,f,S}\rvert^2\right)+\gamma_4\max_{j\in S}\sum_{e<f}\omega^{e,f}\lvert K_{n,j}^{\ast,e,f,S}\rvert^2\right\}. t k b ← S ∈ S ∘ max ⎩ ⎨ ⎧ e < f ∑ ω e , f ( γ 1 ∥ L n ∗ , e , f , S ∥ 2 2 + γ 2 ∣ V n ∗ , e , f , S ∣ 2 + γ 3 ∣ O n ∗ , e , f , S ∣ 2 ) + γ 4 j ∈ S max e < f ∑ ω e , f ∣ K n , j ∗ , e , f , S ∣ 2 ⎭ ⎬ ⎫ .
The validity of this modified bootstrap procedure can be established using the same argument as in Appendix F.3.
Appendix C. Extension to nonlinear causal effects
We now investigate an extension of our framework to nonlinear structural equation models. Although the LSCR invariance principle is developed for linear regression, our variable selection procedure can be extended to nonlinear additive models using basis expansions. Specifically, consider the additive model
Y e = ∑ j ∈ S ⋆ f j ⋆ ( X j e ) + ε e , ε e ⊥ X S ⋆ e , ε e ∼ F , E [ ε e ] = 0. Y^e=\sum_{j\in S^\star}f_j^\star(X_j^e)+\varepsilon^e,\qquad\varepsilon^e\perp X_{S^\star}^e,\qquad\varepsilon^e\sim\mathcal{F},\qquad\mathbb{E}[\varepsilon^e]=0. Y e = j ∈ S ⋆ ∑ f j ⋆ ( X j e ) + ε e , ε e ⊥ X S ⋆ e , ε e ∼ F , E [ ε e ] = 0.
where the nonlinear functions f j ⋆ f_j^\star f j ⋆ and the noise distribution remain invariant across environments. To accommodate nonlinear effects, we expand each feature X j e X_j^e X j e into R n R_n R n basis functions b r ( X j e ) b_r(X_j^e) b r ( X j e ) , r ∈ [ R n ] r\in[R_n] r ∈ [ R n ] . The nonlinear function f j ⋆ ( X j e ) f_j^\star(X_j^e) f j ⋆ ( X j e ) can then be approximated by a linear combination of these basis functions. Denote the expanded covariate vector by Z j e = ( b 1 ( X j e ) , … , b R n ( X j e ) ) T ∈ R R n Z_j^e=(b_1(X_j^e),\ldots,b_{R_n}(X_j^e))^{\mathsf{T}}\in\mathbb{R}^{R_n} Z j e = ( b 1 ( X j e ) , … , b R n ( X j e ) ) T ∈ R R n , and concatenate those vectors into Z e ∈ R p R n Z^e\in\mathbb{R}^{pR_n} Z e ∈ R p R n . For a candidate predictor set S S S , we consider the additive regression model
f S ( X S e ) = ∑ j ∈ S ∑ r = 1 R n θ j r b r ( X j e ) , f_S(X_S^e)=\sum_{j\in S}\sum_{r=1}^{R_n}\theta_{jr}b_r(X_j^e), f S ( X S e ) = j ∈ S ∑ r = 1 ∑ R n θ j r b r ( X j e ) ,
where θ j = ( θ j 1 , … , θ j R n ) T \theta_j=(\theta_{j1},\ldots,\theta_{jR_n})^{\mathsf{T}} θ j = ( θ j 1 , … , θ j R n ) T denotes the coefficient vector associated with the original predictor X j X_j X j . We now apply the LSCR invariance principle to the expanded covariates. For each environment e e e and candidate set S S S , define the population OLS coefficients and corresponding risks as
θ e , S : = arg min θ : θ j = 0 , j ∉ S E [ ( Y e − θ ⊤ Z e ) 2 ] , ν e , S : = E [ ( Y e − ( θ e , S ) ⊤ Z e ) 2 ] . \theta^{e,S} := \mathop{\arg\min}_{\theta:\theta_j=0,\ j\notin S} \mathbb{E}\left[(Y^e-\theta^{\top}Z^e)^2\right], \qquad\nu^{e,S}:=\mathbb{E}\left[(Y^e-(\theta^{e,S})^{\top}Z^e)^2\right]. θ e , S := arg min θ : θ j = 0 , j ∈ / S E [ ( Y e − θ ⊤ Z e ) 2 ] , ν e , S := E [ ( Y e − ( θ e , S ) ⊤ Z e ) 2 ] .
Let θ ^ e , S \widehat{\theta}^{e,S} θ e , S and ν ^ e , S \widehat{\nu}^{e,S} ν e , S denote their empirical counterparts. The corresponding empirical invariance score is
I ^ n o n l i n e a r ( S ) = ∑ e < f ω ^ e , f [ ∥ θ ^ e , S − θ ^ f , S ∥ 2 2 + ρ ( ν ^ e , S − ν ^ f , S ) 2 ] . \widehat{I}_{\mathrm{nonlinear}}(S)=\sum_{e<f}\widehat{\omega}^{e,f}\left[\left\|\widehat{\theta}^{e,S}-\widehat{\theta}^{f,S}\right\|_2^2+\rho\left(\widehat{\nu}^{e,S}-\widehat{\nu}^{f,S}\right)^2\right]. I nonlinear ( S ) = e < f ∑ ω e , f [ θ e , S − θ f , S 2 2 + ρ ( ν e , S − ν f , S ) 2 ] .
As in our original framework, we combine this score with the pooled prediction loss and a group ℓ 0 \ell_0 ℓ 0 -penalty. Let g j ∈ { 0 , 1 } g_j\in\{0,1\} g j ∈ { 0 , 1 } indicate whether predictor j j j is selected, and introduce coefficients θ e ∈ R p R n \theta^e\in\mathbb{R}^{pR_n} θ e ∈ R p R n for each environment. The resulting MIP formulation is
min θ e ∈ R p R n { θ e } e ∈ E g ∈ { 0 , 1 } p ∑ e ∈ E ω e ∥ Y e − ∑ j = 1 p ∑ r = 1 R n θ j r b r ( X j e ) ∥ n e 2 + λ n ∑ j = 1 p g j + γ ∑ e < f ω e , f [ ∥ θ e − θ f ∥ 2 2 + ρ d e , f ( θ e , θ f ) 2 ] , \min_{\substack{\theta^e\in\mathbb{R}^{pR_n}\\ \{\theta^e\}_{e\in\mathcal{E}}\\ g\in\{0,1\}^p}}
\sum_{e\in\mathcal{E}}\omega^e\left\|Y^e-\sum_{j=1}^{p}\sum_{r=1}^{R_n}\theta_{jr}b_r(X_j^e)\right\|_{n_e}^2
+\lambda_n\sum_{j=1}^{p}g_j
+\gamma\sum_{e<f}\omega^{e,f}\left[\|\theta^e-\theta^f\|_2^2+\rho\,d^{e,f}(\theta^e,\theta^f)^2\right], θ e ∈ R p R n { θ e } e ∈ E g ∈ { 0 , 1 } p min e ∈ E ∑ ω e Y e − j = 1 ∑ p r = 1 ∑ R n θ j r b r ( X j e ) n e 2 + λ n j = 1 ∑ p g j + γ e < f ∑ ω e , f [ ∥ θ e − θ f ∥ 2 2 + ρ d e , f ( θ e , θ f ) 2 ] ,
subject to: θ j r = 0 \theta_{jr}=0 θ j r = 0 if g j = 0 g_j=0 g j = 0 , ∀ j ∈ [ p ] , r ∈ [ R n ] , \quad\forall j\in[p],\ r\in[R_n], ∀ j ∈ [ p ] , r ∈ [ R n ] ,
θ j r e = 0 if g j = 0 , ∀ e ∈ E , j ∈ [ p ] , r ∈ [ R n ] , \theta_{jr}^e=0\ \text{if }g_j=0,\qquad\forall e\in\mathcal{E},\ j\in[p],\ r\in[R_n], θ j r e = 0 if g j = 0 , ∀ e ∈ E , j ∈ [ p ] , r ∈ [ R n ] ,
⟨ b r ( X j e ) , Y e − ∑ k = 1 p ∑ s = 1 R n θ k s e b s ( X k e ) ⟩ n e = 0 if g j = 1 , ∀ e ∈ E , j ∈ [ p ] , r ∈ [ R n ] , \left\langle b_r(X_j^e),Y^e-\sum_{k=1}^{p}\sum_{s=1}^{R_n}\theta_{ks}^e b_s(X_k^e)\right\rangle_{n_e}=0\ \text{if }g_j=1,\qquad\forall e\in\mathcal{E},\ j\in[p],\ r\in[R_n], ⟨ b r ( X j e ) , Y e − k = 1 ∑ p s = 1 ∑ R n θ k s e b s ( X k e ) ⟩ n e = 0 if g j = 1 , ∀ e ∈ E , j ∈ [ p ] , r ∈ [ R n ] ,
where
d e , f ( θ e , θ f ) : = ν ^ e ( θ e ) − ν ^ f ( θ f ) , ν ^ e ( θ e ) : = ∥ Y e ∥ n e 2 − ∑ j = 1 p ∑ r = 1 R n θ j r e ⟨ b r ( X j e ) , Y e ⟩ n e . d^{e,f}(\theta^e,\theta^f):=\widehat{\nu}^e(\theta^e)-\widehat{\nu}^f(\theta^f),\qquad
\widehat{\nu}^e(\theta^e):=\left\|Y^e\right\|_{n_e}^2-\sum_{j=1}^{p}\sum_{r=1}^{R_n}\theta_{jr}^e\left\langle b_r(X_j^e),Y^e\right\rangle_{n_e}. d e , f ( θ e , θ f ) := ν e ( θ e ) − ν f ( θ f ) , ν e ( θ e ) := ∥ Y e ∥ n e 2 − j = 1 ∑ p r = 1 ∑ R n θ j r e ⟨ b r ( X j e ) , Y e ⟩ n e .
For computational efficiency, we may use coordinate descent to obtain approximate solutions, with each coordinate update corresponding to the addition or removal of an entire group of basis functions.
Example 6. We generate data from a nonlinear structural equation model with p = 6 p=6 p = 6 covariates and ∣ S ⋆ ∣ = 2 |S^\star|=2 ∣ S ⋆ ∣ = 2 causal parents. Specifically, the response is generated according to
Y e = ∑ j ∈ S ⋆ β j ⋆ tanh ( 2 sin ( X j e ) ) + ϵ e , ϵ e ∼ N ( 0 , 0.25 ) , Y^e=\sum_{j\in S^\star}\beta_j^\star\tanh\left(2\sin(X_j^e)\right)+\epsilon^e,\qquad\epsilon^e\sim\mathcal{N}(0,0.25), Y e = j ∈ S ⋆ ∑ β j ⋆ tanh ( 2 sin ( X j e ) ) + ϵ e , ϵ e ∼ N ( 0 , 0.25 ) ,
where the causal coefficients remain invariant across environments. We consider five environments, allowing the structural coefficients and noise variances governing the covariates to vary across environments. We approximate the nonlinear effects using a Fourier basis with four basis functions for each predictor, and apply our group-ℓ 0 \ell_0 ℓ 0 formulation using coordinate descent. We compare the resulting procedure, denoted by nonlinear MIP, with nonlinear ICP (Heinze-Deml et al., [9 ] ), WVM (Martinet et al., [15 ] ), and FAIR (Gu et al., 2025).
Figure 11 compares the performance of these methods as the sample size per environment increases. Our proposed nonlinear MIP achieves substantially higher rates of exact support recovery and identifies more true causal predictors than the competing methods. In particular, its recovery probability approaches one as the sample size increases, while the other methods rarely achieve exact recovery in this experiment. Moreover, nonlinear MIP is considerably faster than the competing methods.
Figure 11. Performance comparison under nonlinear causal effects as the sample size n n n increases. Left panel: proportion of 100 trials achieving exact recovery of S ⋆ S^\star S ⋆ . Middle panel: average number of true positives ∣ S ^ ∩ S ⋆ ∣ |\widehat{S} \cap S^\star| ∣ S ∩ S ⋆ ∣ over 100 trials. Right panel: average runtime in seconds over 100 trials, shown on a logarithmic scale. Solid lines show averages, and shaded bands indicate ± 1 \pm1 ± 1 standard error. Additional experimental details are provided in Appendix G.8.
One limitation of this extension is that the LSCR invariance principle may not hold exactly under finite basis expansions. To see this, denote the true nonlinear regression function by f ⋆ ( X S ⋆ e ) = ∑ j ∈ S ⋆ f j ⋆ ( X j e ) f^\star(X_{S^\star}^e)=\sum_{j\in S^\star}f_j^\star(X_j^e) f ⋆ ( X S ⋆ e ) = ∑ j ∈ S ⋆ f j ⋆ ( X j e ) . Suppose that it admits the representation
f ⋆ ( X S ⋆ e ) = ∑ j ∈ S ⋆ ∑ r = 1 R n θ j r ⋆ b r ( X j e ) + r ( X S ⋆ e ) , f^\star(X_{S^\star}^e)=\sum_{j\in S^\star}\sum_{r=1}^{R_n}\theta_{jr}^\star b_r(X_j^e)+r(X_{S^\star}^e), f ⋆ ( X S ⋆ e ) = j ∈ S ⋆ ∑ r = 1 ∑ R n θ j r ⋆ b r ( X j e ) + r ( X S ⋆ e ) ,
where θ ⋆ \theta^\star θ ⋆ denotes the L 2 L_2 L 2 projection coefficients of f ⋆ f^\star f ⋆ onto the span of selected basis functions under the observational distribution, and r r r denotes the corresponding approximation error. Let B S ⋆ e : = { b r ( X j ) } j ∈ S ⋆ , r ∈ [ R n ] B_{S^\star}^{e} := \{b_r(X_j)\}_{j \in S^\star,r \in[R_n]} B S ⋆ e := { b r ( X j ) } j ∈ S ⋆ , r ∈ [ R n ] denote the vector of basis functions, and define Σ S ⋆ e : = E [ B S ⋆ e ( B S ⋆ e ) ⊤ ] \Sigma_{S^\star}^{e} := \mathbb{E}[B_{S^\star}^{e}(B_{S^\star}^{e})^\top] Σ S ⋆ e := E [ B S ⋆ e ( B S ⋆ e ) ⊤ ] . By construction, E P o b s [ B S ⋆ e r ( X S ⋆ ) ] = 0 \mathbb{E}_{P_{\mathrm{obs}}}[B_{S^\star}^{e}r(X_{S^\star})] = 0 E P obs [ B S ⋆ e r ( X S ⋆ )] = 0 , although this orthogonality need not hold in other environments. Assuming Σ S ⋆ e \Sigma_{S^\star}^{e} Σ S ⋆ e is invertible, the population OLS coefficients satisfy
θ S ⋆ e , S ⋆ = θ ⋆ + ( Σ S ⋆ e ) − 1 E [ B S ⋆ e r ( X S ⋆ ) ] . \theta_{S^\star}^{e,S^\star} = \theta^\star+ (\Sigma_{S^\star}^{e})^{-1}\mathbb{E}[B_{S^\star}^{e}r(X_{S^\star})]. θ S ⋆ e , S ⋆ = θ ⋆ + ( Σ S ⋆ e ) − 1 E [ B S ⋆ e r ( X S ⋆ )] .
Similarly, the corresponding residual risk is
ν S ⋆ e , S ⋆ = σ ε 2 + E [ r ( X S ⋆ e ) 2 ] − E [ B S ⋆ e r ( X S ⋆ e ) ] ⊤ ( Σ S ⋆ e ) − 1 E [ B S ⋆ e r ( X S ⋆ e ) ] , \nu_{S^\star}^{e,S^\star} = \sigma_\varepsilon^2 + \mathbb{E}[r(X_{S^\star}^{e})^2] - \mathbb{E}[B_{S^\star}^{e}r(X_{S^\star}^{e})]^\top(\Sigma_{S^\star}^{e})^{-1}\mathbb{E}[B_{S^\star}^{e}r(X_{S^\star}^{e})], ν S ⋆ e , S ⋆ = σ ε 2 + E [ r ( X S ⋆ e ) 2 ] − E [ B S ⋆ e r ( X S ⋆ e ) ] ⊤ ( Σ S ⋆ e ) − 1 E [ B S ⋆ e r ( X S ⋆ e )] ,
where σ ε 2 = E [ ( ε e ) 2 ] \sigma_\varepsilon^2 = \mathbb{E}[(\varepsilon^e)^2] σ ε 2 = E [( ε e ) 2 ] is invariant across environments. When r = 0 r = 0 r = 0 , meaning that the true nonlinear functions lie in the span of the selected basis functions, we have θ S ⋆ e , S ⋆ = θ ⋆ \theta_{S^\star}^{e,S^\star} = \theta^\star θ S ⋆ e , S ⋆ = θ ⋆ and ν S ⋆ e , S ⋆ = σ ε 2 \nu_{S^\star}^{e,S^\star} = \sigma_\varepsilon^2 ν S ⋆ e , S ⋆ = σ ε 2 for all e ∈ E e \in\mathcal{E} e ∈ E . Thus, both conditions of the LSCR invariance principle hold at the true causal parent set S ⋆ S^\star S ⋆ . However, when r ≠ 0 r \ne0 r = 0 , the additional terms in the expressions above may depend on the covariate distributions, which can vary across environments. Consequently, both the population OLS coefficients and residual risks may differ across environments, even though the underlying nonlinear causal mechanism remains invariant.
Increasing the number of basis functions R n R_n R n may reduce the approximation error and bring the invariance score closer to zero for S ⋆ S^\star S ⋆ . However, increasing R n R_n R n introduces more coefficients to estimate, potentially leading to larger statistical errors and higher computational costs. Therefore, the choice of basis dimension involves a trade-off between approximation accuracy and estimation accuracy. We consider this extension as an empirical investigation and leave theoretical guarantees for nonlinear models to future work.
Extensions to when some variables are unobserved
We now discuss an extension of the mixed-integer programming framework for invariant prediction in the presence of hidden variables. We consider a structural equation model with noise interventions and latent confounding. Let H ∈ R q H \in\mathbb{R}^q H ∈ R q denote the hidden variables, X A ∈ R ∣ A ∣ X_A \in\mathbb{R}^{|A|} X A ∈ R ∣ A ∣ the observed ancestors of the causal parents, X S ⋆ ∈ R ∣ S ⋆ ∣ X_{S^\star} \in\mathbb{R}^{|S^\star|} X S ⋆ ∈ R ∣ S ⋆ ∣ the causal parents of Y Y Y , Y ∈ R Y \in\mathbb{R} Y ∈ R the response, and X D ∈ R ∣ D ∣ X_D \in\mathbb{R}^{|D|} X D ∈ R ∣ D ∣ the descendants of Y Y Y . The latent variables H H H may directly affect all of these observed variables, while the intervention mechanism, indexed by e ∈ E e \in\mathcal{E} e ∈ E , is assumed to be unaffected by H H H . Specifically, for each e ∈ E e \in\mathcal{E} e ∈ E ,
H e ∼ N ( 0 , I q ) , X a e = ϕ a ⊤ H e + ε a e , ∀ a ∈ A , X j e = θ j ⊤ X A e + ϕ j ⊤ H e + ε j e , ∀ j ∈ S ⋆ . H^{e} \sim\mathcal{N}(0,I_q), \qquad X_a^{e} = \phi_a^{\top} H^{e} + \varepsilon_a^{e}, \ \forall a \in A, \qquad X_j^{e} = \theta_j^{\top} X_A^{e} + \phi_j^{\top} H^{e} + \varepsilon_j^{e}, \ \forall j \in S^{\star}. H e ∼ N ( 0 , I q ) , X a e = ϕ a ⊤ H e + ε a e , ∀ a ∈ A , X j e = θ j ⊤ X A e + ϕ j ⊤ H e + ε j e , ∀ j ∈ S ⋆ .
The response and its descendants are generated according to
Y e = β S ⋆ ⋆ ⊤ X S ⋆ e + ϕ Y ⊤ H e + ε Y e , X d e = θ d Y e + ϕ d ⊤ H e + ε d e , ∀ d ∈ D . Y^{e} = \beta_{S^{\star}}^{\star\top} X_{S^{\star}}^{e} + \phi_Y^{\top} H^{e} + \varepsilon_Y^{e}, \qquad X_d^{e} = \theta_d Y^{e} + \phi_d^{\top} H^{e} + \varepsilon_d^{e}, \ \forall d \in D. Y e = β S ⋆ ⋆ ⊤ X S ⋆ e + ϕ Y ⊤ H e + ε Y e , X d e = θ d Y e + ϕ d ⊤ H e + ε d e , ∀ d ∈ D .
The noise variables satisfy
ε a e ∼ N ( 0 , ν a e ) , ∀ a ∈ A , ε j e ∼ N ( 0 , ν j e ) , ∀ j ∈ S ⋆ , ε Y e ∼ N ( 0 , ν Y ) , ε d e ∼ N ( 0 , ν d e ) , ∀ d ∈ D \varepsilon_a^{e} \sim\mathcal{N}(0,\nu_a^{e}), \ \forall a \in A, \qquad\varepsilon_j^{e} \sim\mathcal{N}(0,\nu_j^{e}), \ \forall j \in S^{\star}, \qquad\varepsilon_Y^{e} \sim\mathcal{N}(0,\nu_Y), \qquad\varepsilon_d^{e} \sim\mathcal{N}(0,\nu_d^{e}), \ \forall d \in D ε a e ∼ N ( 0 , ν a e ) , ∀ a ∈ A , ε j e ∼ N ( 0 , ν j e ) , ∀ j ∈ S ⋆ , ε Y e ∼ N ( 0 , ν Y ) , ε d e ∼ N ( 0 , ν d e ) , ∀ d ∈ D
where the noise variables are mutually independent and independent of H e H^{e} H e . Across environments, interventions are allowed to change only the noise levels ν j e \nu_j^{e} ν j e of the observed covariates j ∈ A ∪ S ⋆ ∪ D j \in A \cup S^{\star} \cup D j ∈ A ∪ S ⋆ ∪ D , while all structural coefficients, the distribution of H e H^{e} H e , and the noise distribution of the response remain invariant. Thus, the environments induce heterogeneity in the covariate distributions without directly intervening on the response mechanism. Such models with noise interventions provide a standard framework for studying causal identification in the presence of latent confounding and have been widely considered in the literature ([21 , 23 ] ).
Since the latent variables H e H^{e} H e are unobserved, the LSCR invariance principle can no longer be expected to identify the structural coefficient β ⋆ \beta^{\star} β ⋆ . In particular, when S = S ⋆ S = S^{\star} S = S ⋆ , the population OLS coefficient is given by
β S ⋆ e , S ⋆ = β S ⋆ ⋆ + ( Σ S ⋆ e ) − 1 E [ X S ⋆ e H e ⊤ ] ϕ Y . \beta_{S^{\star}}^{e,S^{\star}} = \beta_{S^{\star}}^{\star} + \left(\Sigma_{S^{\star}}^{e}\right)^{-1}\mathbb{E}\left[X_{S^{\star}}^{e}H^{e\top}\right]\phi_Y. β S ⋆ e , S ⋆ = β S ⋆ ⋆ + ( Σ S ⋆ e ) − 1 E [ X S ⋆ e H e ⊤ ] ϕ Y .
The second term represents the bias induced by latent confounding and, in general, depends on the interventions across environments. Consequently, the OLS coefficient on the true causal parent set need not remain invariant across environments, even though the structural coefficient β ⋆ \beta^{\star} β ⋆ itself is invariant. This motivates the need for an alternative invariance principle that can accommodate hidden confounding.
Definition 2 (Hidden-invariant prediction model). We say that β ∈ R p \beta\in\mathbb{R}^{p} β ∈ R p is a hidden invariant prediction model if
E [ X j e ( Y e − X e ⊤ β ) ] = E [ X j f ( Y f − X f ⊤ β ) ] , ∀ j ∈ [ p ] , ∀ e , f ∈ E . \mathbb{E}\left[X_j^{e}\left(Y^{e}-X^{e\top}\beta\right)\right] = \mathbb{E}\left[X_j^{f}\left(Y^{f}-X^{f\top}\beta\right)\right], \qquad\forall j \in[p], \ \forall e,f \in\mathcal{E}. E [ X j e ( Y e − X e ⊤ β ) ] = E [ X j f ( Y f − X f ⊤ β ) ] , ∀ j ∈ [ p ] , ∀ e , f ∈ E .
A prediction model is invariant under hidden confounding, based on Definition 2, if the cross moments between its residual and each covariate remain constant across environments. In particular, for the true structural coefficient β ⋆ \beta^{\star} β ⋆ , the cross moments E [ X j e ( Y e − X e ⊤ β ⋆ ) ] \mathbb{E}[X_j^{e}(Y^{e}-X^{e\top}\beta^{\star})] E [ X j e ( Y e − X e ⊤ β ⋆ )] remain invariant across environments for all j ∈ [ p ] j \in[p] j ∈ [ p ] . This follows because the structural coefficients, the distribution of the latent variables, and the response noise distribution remain unchanged across environments, while interventions only affect the noise variances of the covariates. The same invariance principle is also considered in Rothenhäusler et al. [21 ] . Motivated by this observation, we define the invariance score under hidden confounding
I h ( β ) = ∑ e < f ω e , f ∑ j = 1 p ( E [ X j e ( Y e − X e ⊤ β ) ] − E [ X j f ( Y f − X f ⊤ β ) ] ) 2 . I_h(\beta) = \sum_{e<f}\omega^{e,f}\sum_{j=1}^{p}\left(\mathbb{E}\left[X_j^{e}\left(Y^{e}-X^{e\top}\beta\right)\right]-\mathbb{E}\left[X_j^{f}\left(Y^{f}-X^{f\top}\beta\right)\right]\right)^2. I h ( β ) = e < f ∑ ω e , f j = 1 ∑ p ( E [ X j e ( Y e − X e ⊤ β ) ] − E [ X j f ( Y f − X f ⊤ β ) ] ) 2 .
The score quantifies the degree to which the moments between the residuals and covariates vary across environments. In particular, I h ( β ) = 0 I_h(\beta)=0 I h ( β ) = 0 if and only if these moments are invariant across environments, so that β \beta β satisfies the proposed invariance principle under hidden confounding.
With this newly defined invariance score, we propose a procedure with invariance regularization that estimates β ⋆ \beta^{\star} β ⋆ in the same spirit as (11) :
min β ∈ R p ∑ e ∈ E ω e E [ Y e − β ⊤ X e ] 2 + γ I h ( β ) . \min_{\beta\in\mathbb{R}^{p}} \sum_{e \in\mathcal{E}}\omega^{e}\mathbb{E}\left[Y^{e}-\beta^{\top}X^{e}\right]^2+\gamma I_h(\beta). β ∈ R p min e ∈ E ∑ ω e E [ Y e − β ⊤ X e ] 2 + γ I h ( β ) .
Furthermore, the finite-sample convex MIP formulation is given by:
min β ∈ R p g ∈ { 0 , 1 } p ∑ e ∈ E ω e ∥ Y e − ∑ k = 1 p X k e β k ∥ n e 2 + λ ∑ j = 1 p g j + γ ∑ e < f ω e , f ∑ j = 1 p ( ⟨ X j e , Y e − ∑ k = 1 p X k e β k ⟩ − ⟨ X j f , Y f − ∑ k = 1 p X k f β k ⟩ ) 2 \begin{aligned}
\min_{\substack{\beta\in\mathbb{R}^{p}\\g\in\{0,1\}^{p}}}\;&\sum_{e\in\mathcal{E}}\omega^{e}\left\|Y^{e}-\sum_{k=1}^{p}X_{k}^{e}\beta_{k}\right\|_{n_{e}}^{2}+\lambda\sum_{j=1}^{p}g_{j}\\
&+\gamma\sum_{e<f}\omega^{e,f}\sum_{j=1}^{p}\left(\left\langle X_{j}^{e},Y^{e}-\sum_{k=1}^{p}X_{k}^{e}\beta_{k}\right\rangle-\left\langle X_{j}^{f},Y^{f}-\sum_{k=1}^{p}X_{k}^{f}\beta_{k}\right\rangle\right)^{2}
\end{aligned} β ∈ R p g ∈ { 0 , 1 } p min e ∈ E ∑ ω e Y e − k = 1 ∑ p X k e β k n e 2 + λ j = 1 ∑ p g j + γ e < f ∑ ω e , f j = 1 ∑ p ( ⟨ X j e , Y e − k = 1 ∑ p X k e β k ⟩ − ⟨ X j f , Y f − k = 1 ∑ p X k f β k ⟩ ) 2
subject to: β j = 0 \beta_j=0 β j = 0 if g j = 0 g_j=0 g j = 0 , ∀ j ∈ [ p ] \forall j\in[p] ∀ j ∈ [ p ] .
where a penalty ∑ j = 1 p g j \sum_{j=1}^{p}g_j ∑ j = 1 p g j is introduced to encourage sparsity.
Example 7. We generate data according to the noise-intervention model introduced in this section, with q = 2 q=2 q = 2 , ∣ A ∣ ∈ { 2 , 3 } |A|\in\{2,3\} ∣ A ∣ ∈ { 2 , 3 } , ∣ S ∗ ∣ = 4 |S^*|=4 ∣ S ∗ ∣ = 4 , ∣ D ∣ ∈ { 2 , 3 } |D|\in\{2,3\} ∣ D ∣ ∈ { 2 , 3 } , and eight additional independent covariates. Figure 12 shows that our proposed method, MIP-hidden, achieves the lowest coefficient estimation error and the highest rate of perfect recovery. We focus on comparisons with Causal Dantzig [21 ] and negDRO [24 ] , both of which are also designed to accommodate hidden confounding. One possible explanation for the improved performance of MIP-hidden is that its objective includes both the MSE loss and an ℓ 0 \ell_0 ℓ 0 -type sparsity penalty, in addition to the invariance score. In contrast, Causal Dantzig does not directly account for prediction loss, while negDRO does not explicitly encourage sparsity. The mixed-integer formulation therefore allows MIP-hidden to perform discrete variable selection while also accounting for prediction accuracy, particularly when the underlying causal model is sparse.
Figure 12. Performance comparison under hidden confounding and noise interventions (Example 7) as the sample size n n n increases. Left panel: average coefficient estimation error ∥ β ^ − β ∗ ∥ 2 \|\hat{\beta}-\beta^*\|_2 ∥ β ^ − β ∗ ∥ 2 over 200 trials. Right panel: proportion of 200 trials achieving perfect recovery of S ∗ S^* S ∗ , reported for methods that perform variable selection. Solid lines show averages, and shaded bands indicate ± 1 \pm1 ± 1 standard error. Additional competing methods and full experimental details are provided in Appendix G.§ 9 .
Proofs of results in Section 4
In this section, we provide proofs of results regarding the procedure (10). To facilitate the proof, we define the objective function as F ( β ) F(\beta) F ( β ) , and
R ( β ) : = ∑ e ∈ E ω e E [ Y e − β ⊤ X e ] 2 , I 1 ( S ) : = ∑ e < f ω e , f ∥ β e , S − β f , S ∥ 2 2 , I 2 ( S ) : = ∑ e < f ω e , f ∥ ν e , S − ν f , S ∥ 2 2 . R(\beta):=\sum_{e\in\mathcal{E}}\omega^e\mathbb{E}\left[Y^e-\beta^\top X^e\right]^2,\quad I_1(S):=\sum_{e<f}\omega^{e,f}\left\|\beta^{e,S}-\beta^{f,S}\right\|_2^2,\quad I_2(S):=\sum_{e<f}\omega^{e,f}\left\|\nu^{e,S}-\nu^{f,S}\right\|_2^2. R ( β ) := e ∈ E ∑ ω e E [ Y e − β ⊤ X e ] 2 , I 1 ( S ) := e < f ∑ ω e , f β e , S − β f , S 2 2 , I 2 ( S ) := e < f ∑ ω e , f ν e , S − ν f , S 2 2 .
As a result, I ( S ) = I 1 ( S ) + ρ I 2 ( S ) I(S)=I_1(S)+\rho I_2(S) I ( S ) = I 1 ( S ) + ρ I 2 ( S ) and F ( β ) = R ( β ) + γ I 1 ( S ) + γ ρ I 2 ( S ) F(\beta)=R(\beta)+\gamma I_1(S)+\gamma\rho I_2(S) F ( β ) = R ( β ) + γ I 1 ( S ) + γ ρ I 2 ( S ) . We also use F ^ ( β ) \widehat{F}(\beta) F ( β ) , R ^ ( β ) \widehat{R}(\beta) R ( β ) , I ^ 1 ( S ) \widehat{I}_1(S) I 1 ( S ) and I ^ 2 ( S ) \widehat{I}_2(S) I 2 ( S ) to denote the corresponding finite-sample quantities. We use X S e ∈ R ∣ S ∣ X^e_S\in\mathbb{R}^{|S|} X S e ∈ R ∣ S ∣ to denote the subvector of X e X^e X e indexed by S S S , and X i e ∈ R p X_i^e\in\mathbb{R}^p X i e ∈ R p to denote the i i i -th observation of X e X^e X e . Similarly, we use Y i e ∈ R Y_i^e\in\mathbb{R} Y i e ∈ R to denote the i i i -th observation of Y e Y^e Y e . When needed, we write X ~ e ∈ R n × p \widetilde{X}^e\in\mathbb{R}^{n\times p} X e ∈ R n × p for the corresponding sample matrix, and Y ~ e ∈ R n \widetilde{Y}^e\in\mathbb{R}^n Y e ∈ R n for the corresponding sample vector. We further define the covariance matrix restricted to S S S as Σ S e : = E [ X S e X S e ⊤ ] \Sigma^e_S:=\mathbb{E}[X^e_S X^{e\top}_S] Σ S e := E [ X S e X S e ⊤ ] , and denote its empirical counterpart by Σ ^ S e \widehat{\Sigma}^e_S Σ S e . Finally, for the population OLS coefficient β e , S \beta^{e,S} β e , S defined in (2), we denote its restriction to the coordinates in S S S by β S e , S ∈ R ∣ S ∣ \beta^{e,S}_S\in\mathbb{R}^{|S|} β S e , S ∈ R ∣ S ∣ ; its empirical counterpart is denoted by β ^ S e , S ∈ R ∣ S ∣ \widehat{\beta}^{e,S}_S\in\mathbb{R}^{|S|} β S e , S ∈ R ∣ S ∣ .
Proof of Lemma 1
Before the proof, we restate Lemma 1 in a more detailed form.
Lemma 6. Suppose Assumption 1 and 2 hold. Denote
γ ⋆ : = sup S : S ∩ G ≠ ∅ κ L − 1 ⋅ ∥ ∑ e ∈ E ω e E [ ε e X S e ] ∥ 2 2 I 1 ( S ) + I 2 ( S ) . \gamma^\star:=\sup_{S:S\cap G\ne\varnothing}\kappa_L^{-1}\cdot\frac{\left\|\sum_{e\in\mathcal{E}}\omega^e\mathbb{E}[\varepsilon^eX^e_S]\right\|_2^2}{I_1(S)+I_2(S)}. γ ⋆ := S : S ∩ G = ∅ sup κ L − 1 ⋅ I 1 ( S ) + I 2 ( S ) ∑ e ∈ E ω e E [ ε e X S e ] 2 2 . When γ ∧ γ ρ ≥ 2 γ ⋆ + ϵ \gamma\wedge\gamma\rho\ge2\gamma^\star+\epsilon γ ∧ γ ρ ≥ 2 γ ⋆ + ϵ for some ϵ > 0 \epsilon>0 ϵ > 0 ,
F ( β ) − F ( β ⋆ ) ≥ κ L 2 ∥ β − β ⋆ ∥ 2 2 + ϵ ⋅ ( I 1 ( S ) + I 2 ( S ) ) . F(\beta)-F(\beta^\star)\ge\frac{\kappa_L}{2}\|\beta-\beta^\star\|_2^2+\epsilon\cdot\left(I_1(S)+I_2(S)\right). F ( β ) − F ( β ⋆ ) ≥ 2 κ L ∥ β − β ⋆ ∥ 2 2 + ϵ ⋅ ( I 1 ( S ) + I 2 ( S ) ) .
Proof. We largely follow the proof idea of Theorem A.1 in Fan et al. [5 ] . First, denote Δ β = β − β ⋆ \Delta^\beta=\beta-\beta^\star Δ β = β − β ⋆ , S = supp ( β ) S=\operatorname{supp}(\beta) S = supp ( β ) and S ‾ = S ∪ S ⋆ \overline{S}=S\cup S^\star S = S ∪ S ⋆ . Then for any δ ∈ ( 0 , 1 ) \delta\in(0,1) δ ∈ ( 0 , 1 ) ,
R ( β ) − R ( β ⋆ ) = ∑ e ∈ E ω e ( E [ ( Y e − β ⊤ X e ) 2 ] − E [ ( Y e − β ⋆ ⊤ X e ) 2 ] ) = ∑ e ∈ E ω e ( E [ ( ( β ⋆ − β ) ⊤ X e + ε e ) 2 ] − E [ ( ε e ) 2 ] ) = ∑ e ∈ E ω e ( ( Δ β ) ⊤ Σ e Δ β − 2 ( Δ β ) ⊤ E [ ε e X e ] ) = ∑ e ∈ E ω e ( Δ S ‾ β ) ⊤ Σ S ‾ e Δ S ‾ β − 2 ( Δ β ) ⊤ ⋅ ∑ e ∈ E ω e E [ ε e X S ‾ e ] ≥ ( 1 − δ ) κ L ∥ β − β ⋆ ∥ 2 2 − δ − 1 κ L − 1 ∥ ∑ e ∈ E ω e E [ ε e X S e ] ∥ 2 2 . \begin{aligned}
R(\beta)-R(\beta^\star)
&=\sum_{e\in\mathcal{E}}\omega^e\left(\mathbb{E}\left[(Y^e-\beta^\top X^e)^2\right]-\mathbb{E}\left[(Y^e-\beta^{\star\top}X^e)^2\right]\right)\\
&=\sum_{e\in\mathcal{E}}\omega^e\left(\mathbb{E}\left[((\beta^\star-\beta)^\top X^e+\varepsilon^e)^2\right]-\mathbb{E}\left[(\varepsilon^e)^2\right]\right)\\
&=\sum_{e\in\mathcal{E}}\omega^e\left((\Delta^\beta)^\top\Sigma^e\Delta^\beta-2(\Delta^\beta)^\top\mathbb{E}[\varepsilon^eX^e]\right)\\
&=\sum_{e\in\mathcal{E}}\omega^e(\Delta^\beta_{\overline{S}})^\top\Sigma^e_{\overline{S}}\Delta^\beta_{\overline{S}}-2(\Delta^\beta)^\top\cdot\sum_{e\in\mathcal{E}}\omega^e\mathbb{E}[\varepsilon^eX^e_{\overline{S}}]\\
&\ge(1-\delta)\kappa_L\|\beta-\beta^\star\|_2^2-\delta^{-1}\kappa_L^{-1}\left\|\sum_{e\in\mathcal{E}}\omega^e\mathbb{E}[\varepsilon^eX^e_S]\right\|_2^2.
\end{aligned} R ( β ) − R ( β ⋆ ) = e ∈ E ∑ ω e ( E [ ( Y e − β ⊤ X e ) 2 ] − E [ ( Y e − β ⋆ ⊤ X e ) 2 ] ) = e ∈ E ∑ ω e ( E [ (( β ⋆ − β ) ⊤ X e + ε e ) 2 ] − E [ ( ε e ) 2 ] ) = e ∈ E ∑ ω e ( ( Δ β ) ⊤ Σ e Δ β − 2 ( Δ β ) ⊤ E [ ε e X e ] ) = e ∈ E ∑ ω e ( Δ S β ) ⊤ Σ S e Δ S β − 2 ( Δ β ) ⊤ ⋅ e ∈ E ∑ ω e E [ ε e X S e ] ≥ ( 1 − δ ) κ L ∥ β − β ⋆ ∥ 2 2 − δ − 1 κ L − 1 e ∈ E ∑ ω e E [ ε e X S e ] 2 2 .
Consequently, by taking δ = 1 / 2 \delta=1/2 δ = 1/2 ,
F ( β ) − F ( β ⋆ ) = R ( β ) − R ( β ⋆ ) + γ I 1 ( S ) + γ ρ I 2 ( S ) ≥ κ L 2 ∥ β − β ⋆ ∥ 2 2 − 2 κ L − 1 ∥ ∑ e ∈ E ω e E [ ε e X S e ] ∥ 2 2 + 2 γ ⋆ ( I 1 ( S ) + I 2 ( S ) ) + ( γ − 2 γ ⋆ ) I 1 ( S ) + ( γ ρ − 2 γ ⋆ ) I 2 ( S ) ≥ κ L 2 ∥ β − β ⋆ ∥ 2 2 + ϵ ⋅ ( I 1 ( S ) + I 2 ( S ) ) , \begin{aligned}
F(\beta)-F(\beta^\star)
&=R(\beta)-R(\beta^\star)+\gamma I_1(S)+\gamma\rho I_2(S)\\
&\ge\frac{\kappa_L}{2}\|\beta-\beta^\star\|_2^2-2\kappa_L^{-1}\left\|\sum_{e\in\mathcal{E}}\omega^e\mathbb{E}[\varepsilon^eX^e_S]\right\|_2^2+2\gamma^\star\left(I_1(S)+I_2(S)\right)+(\gamma-2\gamma^\star)I_1(S)+(\gamma\rho-2\gamma^\star)I_2(S)\\
&\ge\frac{\kappa_L}{2}\|\beta-\beta^\star\|_2^2+\epsilon\cdot\left(I_1(S)+I_2(S)\right),
\end{aligned} F ( β ) − F ( β ⋆ ) = R ( β ) − R ( β ⋆ ) + γ I 1 ( S ) + γ ρ I 2 ( S ) ≥ 2 κ L ∥ β − β ⋆ ∥ 2 2 − 2 κ L − 1 e ∈ E ∑ ω e E [ ε e X S e ] 2 2 + 2 γ ⋆ ( I 1 ( S ) + I 2 ( S ) ) + ( γ − 2 γ ⋆ ) I 1 ( S ) + ( γ ρ − 2 γ ⋆ ) I 2 ( S ) ≥ 2 κ L ∥ β − β ⋆ ∥ 2 2 + ϵ ⋅ ( I 1 ( S ) + I 2 ( S ) ) ,
where the last inequality follows from the definition of γ ⋆ \gamma^\star γ ⋆ , as well as the conditions γ ∧ γ ρ ≥ 2 γ ⋆ + ϵ \gamma\wedge\gamma\rho\ge2\gamma^\star+\epsilon γ ∧ γ ρ ≥ 2 γ ⋆ + ϵ .
Preliminary results
In this section, we present several events with high probabilities that relate the population quantities to their empirical counterparts.
Lemma 7. Define the event
T 1 = { max β ∈ B ( s n ) ∣ R ( β ) − R ( β ⋆ ) − R ^ ( β ) + R ^ ( β ⋆ ) ∣ ≤ ( C 0 δ n + κ L 4 ) ∥ β − β ⋆ ∥ 2 2 + ( C 0 ′ δ n ) 2 κ L } . \mathcal{T}_1=\left\{\max_{\beta\in\mathcal{B}(s_n)}\left|R(\beta)-R(\beta^\star)-\widehat{R}(\beta)+\widehat{R}(\beta^\star)\right|\le\left(C_0\delta_n+\frac{\kappa_L}{4}\right)\|\beta-\beta^\star\|_2^2+\frac{(C_0'\delta_n)^2}{\kappa_L}\right\}. T 1 = { β ∈ B ( s n ) max R ( β ) − R ( β ⋆ ) − R ( β ) + R ( β ⋆ ) ≤ ( C 0 δ n + 4 κ L ) ∥ β − β ⋆ ∥ 2 2 + κ L ( C 0 ′ δ n ) 2 } .
where C 0 , C 0 ′ > 0 C_0,C'_0>0 C 0 , C 0 ′ > 0 are constants, B ( s n ) = { β ∈ R p : ∣ supp ( β ) ∣ ≤ s n } \mathcal{B}(s_n)=\{\beta\in\mathbb{R}^p:|\operatorname{supp}(\beta)|\le s_n\} B ( s n ) = { β ∈ R p : ∣ supp ( β ) ∣ ≤ s n } , and
δ n = 2 s n + s n log p + log p ∑ e ∈ E n e + 2 s n + s n log p + log p ∑ e ∈ E n e . \delta_n=\sqrt{\frac{2s_n+s_n\log p+\log p}{\sum_{e\in\mathcal{E}}n^e}+\frac{2s_n+s_n\log p+\log p}{\sum_{e\in\mathcal{E}}n^e}}. δ n = ∑ e ∈ E n e 2 s n + s n log p + log p + ∑ e ∈ E n e 2 s n + s n log p + log p .
Then P ( T 1 ) ≥ 1 − 1 / p \mathbb{P}(\mathcal{T}_1)\ge1-1/p P ( T 1 ) ≥ 1 − 1/ p .
Proof. Appealing to Lemma C.7 of Fan et al. [5 ] , where we let
V ( s n , t ) = s n log ( e p / s n ) + s ⋆ + t ≤ s n ( 1 + log p ) + s n + t , V(s_n,t)=s_n\log(ep/s_n)+s^\star+t\le s_n(1+\log p)+s_n+t, V ( s n , t ) = s n log ( e p / s n ) + s ⋆ + t ≤ s n ( 1 + log p ) + s n + t ,
and let t = log p t=\log p t = log p , we obtain that
max β ∈ B ( s n ) ∣ R ( β ) − R ( β ⋆ ) − R ^ ( β ) + R ^ ( β ⋆ ) ∣ ≤ ( C 0 δ n ∥ β − β ⋆ ∥ 2 2 + C 0 ′ δ n ∥ β − β ⋆ ∥ 2 ) . \max_{\beta\in\mathcal{B}(s_n)}\left|\mathsf{R}(\beta)-\mathsf{R}(\beta^\star)-\widehat{\mathsf{R}}(\beta)+\widehat{\mathsf{R}}(\beta^\star)\right|\le\left(C_0\delta_n\|\beta-\beta^\star\|_2^2+C'_0\delta_n\|\beta-\beta^\star\|_2\right). β ∈ B ( s n ) max R ( β ) − R ( β ⋆ ) − R ( β ) + R ( β ⋆ ) ≤ ( C 0 δ n ∥ β − β ⋆ ∥ 2 2 + C 0 ′ δ n ∥ β − β ⋆ ∥ 2 ) .
with probability at least 1 − 1 / p 1-1/p 1 − 1/ p . The constants C 0 , C 0 ′ > 0 C_0,C'_0>0 C 0 , C 0 ′ > 0 are only associated with κ U \kappa_U κ U , σ x \sigma_x σ x and σ ε \sigma_\varepsilon σ ε . Now applying Young’s inequality yields
C 0 δ n ∥ β − β ⋆ ∥ 2 2 + C 0 ′ δ n ∥ β − β ⋆ ∥ 2 ≤ C 0 δ n ∥ β − β ⋆ ∥ 2 2 + ( C 0 ′ δ n ) 2 4 4 κ L + κ L 4 ∥ β − β ⋆ ∥ 2 2 = ( C 0 δ n + κ L 4 ) ∥ β − β ⋆ ∥ 2 2 + ( C 0 ′ δ n ) 2 κ L . \begin{aligned}
C_0\delta_n\|\beta-\beta^\star\|_2^2+C'_0\delta_n\|\beta-\beta^\star\|_2\le C_0\delta_n\|\beta-\beta^\star\|_2^2+\frac{(C'_0\delta_n)^2}{4}\frac{4}{\kappa_L}+\frac{\kappa_L}{4}\|\beta-\beta^\star\|_2^2 \\
&=\left(C_0\delta_n+\frac{\kappa_L}{4}\right)\|\beta-\beta^\star\|_2^2+\frac{(C'_0\delta_n)^2}{\kappa_L}.
\end{aligned} C 0 δ n ∥ β − β ⋆ ∥ 2 2 + C 0 ′ δ n ∥ β − β ⋆ ∥ 2 ≤ C 0 δ n ∥ β − β ⋆ ∥ 2 2 + 4 ( C 0 ′ δ n ) 2 κ L 4 + 4 κ L ∥ β − β ⋆ ∥ 2 2 = ( C 0 δ n + 4 κ L ) ∥ β − β ⋆ ∥ 2 2 + κ L ( C 0 ′ δ n ) 2 .
The result hence follows.
Lemma 8. Define the event
T 2 = { max ∣ S ∣ ≤ s n e ∈ E ∥ Σ ^ S e − Σ S e ∥ o p ≲ s n log p + log ∣ E ∣ min e ∈ E n e } , \mathcal{T}_2=\left\{\max_{\substack{|S|\le s_n\\e\in\mathcal{E}}}\left\|\widehat{\Sigma}_S^e-\Sigma_S^e\right\|_{\mathrm{op}}\lesssim\sqrt{\frac{s_n\log p+\log|\mathcal{E}|}{\min_{e\in\mathcal{E}}n^e}}\right\}, T 2 = ⎩ ⎨ ⎧ ∣ S ∣ ≤ s n e ∈ E max Σ S e − Σ S e op ≲ min e ∈ E n e s n log p + log ∣ E ∣ ⎭ ⎬ ⎫ , where Σ S e = E [ X S e X S e T ] \Sigma_S^e=\mathbb{E}[X_S^eX_S^{e\mathsf{T}}] Σ S e = E [ X S e X S e T ] . Then P ( T 2 ) ≥ 1 − 2 / p \mathbb{P}(\mathcal{T}_2)\ge1-2/p P ( T 2 ) ≥ 1 − 2/ p .
Proof. Recall that we assume κ L I p ⪯ Σ e ⪯ κ U I p \kappa_LI_p\preceq\Sigma^e\preceq\kappa_UI_p κ L I p ⪯ Σ e ⪯ κ U I p in Assumption 2 part (1). Then for any S ⊆ [ p ] S\subseteq[p] S ⊆ [ p ] , we also have κ L I ∣ S ∣ ⪯ Σ S e ⪯ κ U I ∣ S ∣ \kappa_LI_{|S|}\preceq\Sigma_S^e\preceq\kappa_UI_{|S|} κ L I ∣ S ∣ ⪯ Σ S e ⪯ κ U I ∣ S ∣ . Since
Σ ^ S e = 1 n e ∑ i = 1 n e X S , i e X S , i e T with E [ Σ ^ S e ] = Σ S e , \widehat{\Sigma}_S^e=\frac{1}{n^e}\sum_{i=1}^{n^e}X_{S,i}^eX_{S,i}^{e\mathsf{T}}\quad\text{with}\quad\mathbb{E}\left[\widehat{\Sigma}_S^e\right]=\Sigma_S^e, Σ S e = n e 1 i = 1 ∑ n e X S , i e X S , i e T with E [ Σ S e ] = Σ S e , using the Bernstein inequality for sub-exponential distributions, we have
∥ Σ ^ S e − Σ S e ∥ o p ≤ C 1 σ x 2 κ U ( ∣ S ∣ + t n e + ∣ S ∣ + t n e ) \left\|\widehat{\Sigma}_S^e-\Sigma_S^e\right\|_{\mathrm{op}}\le C_1\sigma_x^2\kappa_U\left(\sqrt{\frac{|S|+t}{n^e}}+\frac{|S|+t}{n^e}\right) Σ S e − Σ S e op ≤ C 1 σ x 2 κ U ( n e ∣ S ∣ + t + n e ∣ S ∣ + t ) with probability at least 1 − 2 e − c 1 t 1-2e^{-c_1t} 1 − 2 e − c 1 t , where C 1 , c 1 > 0 C_1,c_1>0 C 1 , c 1 > 0 are absolute constants. In other words, with probability at least 1 − 2 e − t / ∑ j = 0 ∣ S ∣ ( p j ) ∣ E ∣ 1-2e^{-t}/\sum_{j=0}^{|S|}\binom{p}{j}|\mathcal{E}| 1 − 2 e − t / ∑ j = 0 ∣ S ∣ ( j p ) ∣ E ∣ ,
∥ Σ ^ S e − Σ S e ∥ o p ≲ ∣ S ∣ + log ∑ j = 0 ∣ S ∣ ( p j ) + log ∣ E ∣ + t n e ≲ ∣ S ∣ log ( e p / ∣ S ∣ ) + log ∣ E ∣ + t n e . \left\|\widehat{\Sigma}_S^e-\Sigma_S^e\right\|_{\mathrm{op}}\lesssim\sqrt{\frac{|S|+\log\sum_{j=0}^{|S|}\binom{p}{j}+\log|\mathcal{E}|+t}{n^e}}\lesssim\sqrt{\frac{|S|\log(ep/|S|)+\log|\mathcal{E}|+t}{n^e}}. Σ S e − Σ S e op ≲ n e ∣ S ∣ + log ∑ j = 0 ∣ S ∣ ( j p ) + log ∣ E ∣ + t ≲ n e ∣ S ∣ log ( e p /∣ S ∣ ) + log ∣ E ∣ + t . Now applying the union bound over all supports for S S S with ∣ S ∣ ≤ s n |S|\le s_n ∣ S ∣ ≤ s n and all ∣ E ∣ |\mathcal{E}| ∣ E ∣ environments, and letting t = log p t=\log p t = log p , we have
P ( T 2 ) ≥ 1 − 2 / p . \mathbb{P}(\mathcal{T}_2)\ge1-2/p. P ( T 2 ) ≥ 1 − 2/ p .
Lemma 9. Define the event
T 3 = { max ∣ S ∣ ≤ s n e ∈ E ∥ 1 n e ∑ i = 1 n e X S , i e r i e , S ∥ 2 ≲ ( 1 + ∥ β − β ⋆ ∥ 2 ) s n log p + log ∣ E ∣ min e ∈ E n e , with supp ( β ) ⊆ S } , \mathcal{T}_3 = \left\{ \max_{\substack{|S|\leq s_n\\ e\in\mathcal{E}}} \left\| \frac{1}{n^e}\sum_{i=1}^{n^e} X_{S,i}^e r_{i}^{e,S} \right\|_2 \lesssim(1+\|\beta-\beta^\star\|_2)\sqrt{\frac{s_n\log p+\log|\mathcal{E}|}{\min_{e\in\mathcal{E}} n^e}},\quad\text{with supp}(\beta)\subseteq S \right\}, T 3 = ⎩ ⎨ ⎧ ∣ S ∣ ≤ s n e ∈ E max n e 1 i = 1 ∑ n e X S , i e r i e , S 2 ≲ ( 1 + ∥ β − β ⋆ ∥ 2 ) min e ∈ E n e s n log p + log ∣ E ∣ , with supp ( β ) ⊆ S ⎭ ⎬ ⎫ , where r e , S = Y e − X e T β e , S r^{e,S}=Y^e-X^{e\mathsf{T}}\beta^{e,S} r e , S = Y e − X e T β e , S . Then P ( T 3 ) ≥ 1 − 2 / p \mathbb{P}(\mathcal{T}_3)\geq1-2/p P ( T 3 ) ≥ 1 − 2/ p .
Proof. Denote Z i e , S = X S , i e r i e , S Z_i^{e,S}=X_{S,i}^e r_i^{e,S} Z i e , S = X S , i e r i e , S . Due to the property of the population OLS coefficient β e , S \beta^{e,S} β e , S , we have E [ Z i e , S ] = 0 \mathbb{E}[Z_i^{e,S}]=0 E [ Z i e , S ] = 0 . Now using the Bernstein inequality for sub-exponential distributions, we have
∥ 1 n e ∑ i = 1 n e Z i e , S ∥ 2 ≤ C 2 K e , S ( ∣ S ∣ + t n e + ∣ S ∣ + t n e ) \left\|\frac{1}{n^e}\sum_{i=1}^{n^e} Z_i^{e,S}\right\|_2 \leq C_2K^{e,S}\left(\sqrt{\frac{|S|+t}{n^e}}+\frac{|S|+t}{n^e}\right) n e 1 i = 1 ∑ n e Z i e , S 2 ≤ C 2 K e , S ( n e ∣ S ∣ + t + n e ∣ S ∣ + t ) with probability at least 1 − 2 e − c 2 t 1-2e^{-c_2t} 1 − 2 e − c 2 t , where C 2 , c 2 > 0 C_2,c_2>0 C 2 , c 2 > 0 are absolute constants. However, the number K e , S K^{e,S} K e , S depends on β ⋆ \beta^\star β ⋆ . To see this, note that
K e , S = sup ∥ u ∥ 2 = 1 ∥ u T Z e , S ∥ ψ 1 ≤ sup ∥ u ∥ 2 = 1 ∥ u T X S e ∥ ψ 2 ∥ r e , S ∥ ψ 2 ≲ σ x κ U ∥ r e , S ∥ ψ 2 , K^{e,S}=\sup_{\|u\|_2=1}\|u^\mathsf{T}Z^{e,S}\|_{\psi_1}\leq\sup_{\|u\|_2=1}\|u^\mathsf{T}X_S^e\|_{\psi_2}\|r^{e,S}\|_{\psi_2}\lesssim\sigma_x\sqrt{\kappa_U}\|r^{e,S}\|_{\psi_2}, K e , S = ∥ u ∥ 2 = 1 sup ∥ u T Z e , S ∥ ψ 1 ≤ ∥ u ∥ 2 = 1 sup ∥ u T X S e ∥ ψ 2 ∥ r e , S ∥ ψ 2 ≲ σ x κ U ∥ r e , S ∥ ψ 2 , where the last inequality follows from Assumption 2 part (2). Moreover, for any β \beta β with supp ( β ) ⊆ S \operatorname{supp}(\beta)\subseteq S supp ( β ) ⊆ S ,
r e , S = Y e − X S e T β S e , S = X e T ( β ⋆ − β ) + ε e + X S e T ( β S − β S e , S ) ; r^{e,S}=Y^e-X_S^{e\mathsf{T}}\beta_S^{e,S}=X^{e\mathsf{T}}(\beta^\star-\beta)+\varepsilon^e+X_S^{e\mathsf{T}}(\beta_S-\beta_S^{e,S}); r e , S = Y e − X S e T β S e , S = X e T ( β ⋆ − β ) + ε e + X S e T ( β S − β S e , S ) ; Now let q e , β = X e T ( β ⋆ − β ) + ε e q^{e,\beta}=X^{e\mathsf{T}}(\beta^\star-\beta)+\varepsilon^e q e , β = X e T ( β ⋆ − β ) + ε e , with the population normal equation, we have
0 = E [ X S e r e , S ] = E [ X S e q e , β ] + Σ S e ( β S − β S e , S ) . 0=\mathbb{E}\left[X_S^e r^{e,S}\right]=\mathbb{E}\left[X_S^e q^{e,\beta}\right]+\Sigma_S^e(\beta_S-\beta_S^{e,S}). 0 = E [ X S e r e , S ] = E [ X S e q e , β ] + Σ S e ( β S − β S e , S ) . As a result,
∥ β S − β S e , S ∥ 2 = ∥ ( Σ S e ) − 1 E [ X S e q e , β ] ∥ 2 ≤ 1 κ L sup ∥ u ∥ 2 = 1 ∣ u T E [ X S e q e , β ] ∣ ≤ 1 κ L sup ∥ u ∥ 2 = 1 E [ ( u T X S e ) 2 ] E ( q e , β ) 2 ≲ κ U κ L ∥ q e , β ∥ ψ 2 ≤ κ U κ L ( ∥ X e T ( β ⋆ − β ) ∥ ψ 2 + ∥ ε e ∥ ψ 2 ) ≲ κ U κ L ( σ x κ U ∥ β − β ⋆ ∥ 2 + σ ε ) , \begin{aligned}
\|\beta_S-\beta_S^{e,S}\|_2
&=\left\|(\Sigma_S^e)^{-1}\mathbb{E}\left[X_S^e q^{e,\beta}\right]\right\|_2
\leq\frac{1}{\kappa_L}\sup_{\|u\|_2=1}\left|u^\mathsf{T}\mathbb{E}\left[X_S^e q^{e,\beta}\right]\right|\\
&\leq\frac{1}{\kappa_L}\sup_{\|u\|_2=1}\sqrt{\mathbb{E}[(u^\mathsf{T}X_S^e)^2]}\sqrt{\mathbb{E}(q^{e,\beta})^2}
\lesssim\frac{\sqrt{\kappa_U}}{\kappa_L}\|q^{e,\beta}\|_{\psi_2}\\
&\leq\frac{\sqrt{\kappa_U}}{\kappa_L}\left(\|X^{e\mathsf{T}}(\beta^\star-\beta)\|_{\psi_2}+\|\varepsilon^e\|_{\psi_2}\right)
\lesssim\frac{\sqrt{\kappa_U}}{\kappa_L}\left(\sigma_x\sqrt{\kappa_U}\|\beta-\beta^\star\|_2+\sigma_\varepsilon\right),
\end{aligned} ∥ β S − β S e , S ∥ 2 = ( Σ S e ) − 1 E [ X S e q e , β ] 2 ≤ κ L 1 ∥ u ∥ 2 = 1 sup u T E [ X S e q e , β ] ≤ κ L 1 ∥ u ∥ 2 = 1 sup E [( u T X S e ) 2 ] E ( q e , β ) 2 ≲ κ L κ U ∥ q e , β ∥ ψ 2 ≤ κ L κ U ( ∥ X e T ( β ⋆ − β ) ∥ ψ 2 + ∥ ε e ∥ ψ 2 ) ≲ κ L κ U ( σ x κ U ∥ β − β ⋆ ∥ 2 + σ ε ) , where the inequalities follow from Assumption 2. Consequently, we have
K e , S ≲ ∥ r e , S ∥ ψ 2 ≤ ∥ q e , β ∥ ψ 2 + ∥ X S e T ( β S − β S e , S ) ∥ ψ 2 ≤ ( σ x κ U ∥ β − β ⋆ ∥ 2 + σ ε ) + σ x κ U ∥ β S − β S e , S ∥ 2 ≲ 1 + ∥ β − β ⋆ ∥ 2 . \begin{aligned}
K^{e,S}
&\lesssim\|r^{e,S}\|_{\psi_2}\leq\|q^{e,\beta}\|_{\psi_2}+\|X_S^{e\mathsf{T}}(\beta_S-\beta_S^{e,S})\|_{\psi_2}\\
&\leq(\sigma_x\sqrt{\kappa_U}\|\beta-\beta^\star\|_2+\sigma_\varepsilon)+\sigma_x\sqrt{\kappa_U}\|\beta_S-\beta_S^{e,S}\|_2
\lesssim1+\|\beta-\beta^\star\|_2.
\end{aligned} K e , S ≲ ∥ r e , S ∥ ψ 2 ≤ ∥ q e , β ∥ ψ 2 + ∥ X S e T ( β S − β S e , S ) ∥ ψ 2 ≤ ( σ x κ U ∥ β − β ⋆ ∥ 2 + σ ε ) + σ x κ U ∥ β S − β S e , S ∥ 2 ≲ 1 + ∥ β − β ⋆ ∥ 2 . In other words, with probability at least 1 − 2 e − t ( p ∣ S ∣ ) ∣ E ∣ 1-2e^{-t}\binom{p}{|S|}|\mathcal{E}| 1 − 2 e − t ( ∣ S ∣ p ) ∣ E ∣ ,
∥ 1 n e ∑ i = 1 n e Z i e , S ∥ 2 ≲ ( 1 + ∥ β − β ⋆ ∥ 2 ) ∣ S ∣ log ( e p / ∣ S ∣ ) + log ∣ E ∣ + t n e . \left\|\frac{1}{n^e}\sum_{i=1}^{n^e} Z_i^{e,S}\right\|_2\lesssim(1+\|\beta-\beta^\star\|_2)\sqrt{\frac{|S|\log(ep/|S|)+\log|\mathcal{E}|+t}{n^e}}. n e 1 i = 1 ∑ n e Z i e , S 2 ≲ ( 1 + ∥ β − β ⋆ ∥ 2 ) n e ∣ S ∣ log ( e p /∣ S ∣ ) + log ∣ E ∣ + t . Now applying the union bound over all supports for S S S with ∣ S ∣ ≤ s n |S|\leq s_n ∣ S ∣ ≤ s n and all ∣ E ∣ |\mathcal{E}| ∣ E ∣ environments, and letting t = log p t=\log p t = log p , we have
P ( T 3 ) ≥ 1 − 2 / p . \mathbb{P}(\mathcal{T}_3)\geq1-2/p. P ( T 3 ) ≥ 1 − 2/ p .
Lemma 10. Define the event
T 4 = { max S ⊆ s n ϵ ∈ E ∣ 1 n ϵ ∑ i = 1 n ϵ ( r i ϵ , S ) 2 − E [ ( r ϵ , S ) 2 ] ∣ ≲ ( 1 + ∥ β − β ∗ ∥ 2 ) 2 s n log p + log ∣ E ∣ min ϵ ∈ E n ϵ , with supp ( β ) ⊆ S } , \mathcal{T}_4 = \left\{ \max_{\substack{S \subseteq s_n \\ \epsilon\in\mathcal{E}}} \left| \frac{1}{n^{\epsilon}} \sum_{i=1}^{n^{\epsilon}} (r_i^{\epsilon,S})^2 - \mathbb{E}\left[(r^{\epsilon,S})^2\right] \right| \lesssim(1+\lVert\beta-\beta^* \rVert_2)^2 \sqrt{\frac{s_n \log p+\log|\mathcal{E}|}{\min_{\epsilon\in\mathcal{E}}n^{\epsilon}}},\quad\text{with supp}(\beta)\subseteq S \right\}, T 4 = ⎩ ⎨ ⎧ S ⊆ s n ϵ ∈ E max n ϵ 1 i = 1 ∑ n ϵ ( r i ϵ , S ) 2 − E [ ( r ϵ , S ) 2 ] ≲ ( 1 + ∥ β − β ∗ ∥ 2 ) 2 min ϵ ∈ E n ϵ s n log p + log ∣ E ∣ , with supp ( β ) ⊆ S ⎭ ⎬ ⎫ , where r ϵ , S = Y ϵ − X ϵ T β ϵ , S r^{\epsilon,S}=Y^{\epsilon}-X^{\epsilon\mathsf{T}}\beta^{\epsilon,S} r ϵ , S = Y ϵ − X ϵ T β ϵ , S . Then P ( T 4 ) ≥ 1 − 2 / p \mathbb{P}(\mathcal{T}_4)\geq1-2/p P ( T 4 ) ≥ 1 − 2/ p .
Proof. Recall that in the proof of Lemma 9, we showed that r ϵ , S r^{\epsilon,S} r ϵ , S is sub-Gaussian with ∥ r ϵ , S ∥ ψ 2 ≲ 1 + ∥ β − β ∗ ∥ 2 \lVert r^{\epsilon,S}\rVert_{\psi_2}\lesssim1+\lVert\beta-\beta^*\rVert_2 ∥ r ϵ , S ∥ ψ 2 ≲ 1 + ∥ β − β ∗ ∥ 2 . Then ( r ϵ , S ) 2 − E [ ( r ϵ , S ) 2 ] (r^{\epsilon,S})^2-\mathbb{E}[(r^{\epsilon,S})^2] ( r ϵ , S ) 2 − E [( r ϵ , S ) 2 ] is sub-exponential with ∥ ( r ϵ , S ) 2 − E [ ( r ϵ , S ) 2 ] ∥ ψ 1 ≲ ( 1 + ∥ β − β ∗ ∥ 2 ) 2 \lVert(r^{\epsilon,S})^2-\mathbb{E}[(r^{\epsilon,S})^2]\rVert_{\psi_1}\lesssim(1+\lVert\beta-\beta^*\rVert_2)^2 ∥( r ϵ , S ) 2 − E [( r ϵ , S ) 2 ] ∥ ψ 1 ≲ ( 1 + ∥ β − β ∗ ∥ 2 ) 2 . By Bernstein’s inequality, we have
∣ 1 n ϵ ∑ i = 1 n ϵ ( r i ϵ , S ) 2 − E [ r ϵ , S ] 2 ∣ ≤ C 3 ( K ϵ , S ) 2 ( t n ϵ + t n ϵ ) \left|\frac{1}{n^{\epsilon}}\sum_{i=1}^{n^{\epsilon}}(r_i^{\epsilon,S})^2-\mathbb{E}[r^{\epsilon,S}]^2\right|\leq C_3(K^{\epsilon,S})^2\left(\sqrt{\frac{t}{n^{\epsilon}}}+\frac{t}{n^{\epsilon}}\right) n ϵ 1 i = 1 ∑ n ϵ ( r i ϵ , S ) 2 − E [ r ϵ , S ] 2 ≤ C 3 ( K ϵ , S ) 2 ( n ϵ t + n ϵ t ) with probability at least 1 − e − c 3 t 1-e^{-c_3t} 1 − e − c 3 t , where C 3 , c 3 > 0 C_3,c_3>0 C 3 , c 3 > 0 are absolute constants, and K ϵ , S ≲ 1 + ∥ β − β ∗ ∥ 2 K^{\epsilon,S}\lesssim1+\lVert\beta-\beta^*\rVert_2 K ϵ , S ≲ 1 + ∥ β − β ∗ ∥ 2 . In other words, with probability at least 1 − 2 e − t / ∑ j = 0 s n ( p j ) ∣ E ∣ 1-2e^{-t}/\sum_{j=0}^{s_n}\binom{p}{j}|\mathcal{E}| 1 − 2 e − t / ∑ j = 0 s n ( j p ) ∣ E ∣ ,
∣ 1 n ϵ ∑ i = 1 n ϵ ( r i ϵ , S ) 2 − E [ r ϵ , S ] 2 ∣ ≲ ( 1 + ∥ β − β ∗ ∥ 2 ) 2 ∣ S ∣ log ( e p / ∣ S ∣ ) + log ∣ E ∣ + t n ϵ . \left|\frac{1}{n^{\epsilon}}\sum_{i=1}^{n^{\epsilon}}(r_i^{\epsilon,S})^2-\mathbb{E}[r^{\epsilon,S}]^2\right|\lesssim(1+\lVert\beta-\beta^*\rVert_2)^2\sqrt{\frac{|S|\log(ep/|S|)+\log|\mathcal{E}|+t}{n^{\epsilon}}}. n ϵ 1 i = 1 ∑ n ϵ ( r i ϵ , S ) 2 − E [ r ϵ , S ] 2 ≲ ( 1 + ∥ β − β ∗ ∥ 2 ) 2 n ϵ ∣ S ∣ log ( e p /∣ S ∣ ) + log ∣ E ∣ + t . Now applying the union bound over all supports for S S S with ∣ S ∣ ≤ s n |S|\leq s_n ∣ S ∣ ≤ s n and all ∣ E ∣ |\mathcal{E}| ∣ E ∣ environments, and letting t = log p t=\log p t = log p , we have
P ( T 4 ) ≥ 1 − 2 / p . \mathbb{P}(\mathcal{T}_4)\geq1-2/p. P ( T 4 ) ≥ 1 − 2/ p .
Lemma 11. Under the event T 2 ∩ T 3 ∩ T 4 \mathcal{T}_2\cap\mathcal{T}_3\cap\mathcal{T}_4 T 2 ∩ T 3 ∩ T 4 , for any S S S with ∣ S ∣ ≤ s n |S|\leq s_n ∣ S ∣ ≤ s n and supp ( β ) ⊆ S \operatorname{supp}(\beta)\subseteq S supp ( β ) ⊆ S ,
∣ ℓ 1 ( S ) − ℓ 1 ( S ∗ ) − ℓ ^ 1 ( S ) + ℓ ^ 1 ( S ∗ ) ∣ ≤ ϵ / ( 2 γ ) ⋅ ( ℓ 1 ( S ) + ℓ 2 ( S ) ) + ( ( 16 + 64 ρ 2 ) γ / ϵ + 16 + 64 ρ ) r n 2 + ( 16 γ / ϵ + 8 ) r n 2 ⋅ ∥ β − β ∗ ∥ 2 2 + ( 64 ρ 2 γ / ϵ + 32 ρ ) r n 2 ⋅ ∥ β − β ∗ ∥ 2 2 , \left|\ell_1(S)-\ell_1(S^*)-\widehat{\ell}_1(S)+\widehat{\ell}_1(S^*)\right|
\leq\epsilon/(2\gamma)\cdot(\ell_1(S)+\ell_2(S))+\left((16+64\rho^2)\gamma/\epsilon+16+64\rho\right)r_n^2
+\left(16\gamma/\epsilon+8\right)r_n^2\cdot\lVert\beta-\beta^*\rVert_2^2+(64\rho^2\gamma/\epsilon+32\rho)r_n^2\cdot\lVert\beta-\beta^*\rVert_2^2, ℓ 1 ( S ) − ℓ 1 ( S ∗ ) − ℓ 1 ( S ) + ℓ 1 ( S ∗ ) ≤ ϵ / ( 2 γ ) ⋅ ( ℓ 1 ( S ) + ℓ 2 ( S )) + ( ( 16 + 64 ρ 2 ) γ / ϵ + 16 + 64 ρ ) r n 2 + ( 16 γ / ϵ + 8 ) r n 2 ⋅ ∥ β − β ∗ ∥ 2 2 + ( 64 ρ 2 γ / ϵ + 32 ρ ) r n 2 ⋅ ∥ β − β ∗ ∥ 2 2 , where r n ≍ ( s n log p + log ∣ E ∣ ) / min ϵ ∈ E n ϵ r_n\asymp\sqrt{(s_n\log p+\log|\mathcal{E}|)/\min_{\epsilon\in\mathcal{E}}n^\epsilon} r n ≍ ( s n log p + log ∣ E ∣ ) / min ϵ ∈ E n ϵ , and ϵ > 0 \epsilon>0 ϵ > 0 is defined in Lemma 1.
Proof. Recall ℓ ( S ) = ℓ 1 ( S ) + ρ ℓ 2 ( S ) \ell(S)=\ell_1(S)+\rho\ell_2(S) ℓ ( S ) = ℓ 1 ( S ) + ρ ℓ 2 ( S ) . Therefore, it suffices to bound
∣ ℓ 1 ( S ) − ℓ 1 ( S ∗ ) − ℓ ^ 1 ( S ) + ℓ ^ 1 ( S ∗ ) ∣ and ∣ ℓ 2 ( S ) − ℓ 2 ( S ∗ ) − ℓ ^ 2 ( S ) + ℓ ^ 2 ( S ∗ ) ∣ , \left|\ell_1(S)-\ell_1(S^*)-\widehat{\ell}_1(S)+\widehat{\ell}_1(S^*)\right|
\quad\text{and}\quad
\left|\ell_2(S)-\ell_2(S^*)-\widehat{\ell}_2(S)+\widehat{\ell}_2(S^*)\right|, ℓ 1 ( S ) − ℓ 1 ( S ∗ ) − ℓ 1 ( S ) + ℓ 1 ( S ∗ ) and ℓ 2 ( S ) − ℓ 2 ( S ∗ ) − ℓ 2 ( S ) + ℓ 2 ( S ∗ ) , which are established in Steps I and II of this proof. The final bound is obtained in Step III.
Step I. Bound ∣ ℓ 1 ( S ) − ℓ 1 ( S ∗ ) − ℓ ^ 1 ( S ) + ℓ ^ 1 ( S ∗ ) ∣ \left|\ell_1(S)-\ell_1(S^*)-\widehat{\ell}_1(S)+\widehat{\ell}_1(S^*)\right| ℓ 1 ( S ) − ℓ 1 ( S ∗ ) − ℓ 1 ( S ) + ℓ 1 ( S ∗ ) . For any ϵ ∈ E \epsilon\in\mathcal{E} ϵ ∈ E , the sample error of OLS coefficients satisfies
β ^ ϵ , S − β ϵ , S = ( Σ ^ S ϵ ) − 1 ( 1 n ϵ ∑ i = 1 n ϵ X S , i ϵ r i ϵ , S ) , \widehat{\beta}^{\epsilon,S}-\beta^{\epsilon,S}=(\widehat{\Sigma}^{\epsilon}_S)^{-1}\left(\frac{1}{n^\epsilon}\sum_{i=1}^{n^\epsilon}X_{S,i}^{\epsilon}r_i^{\epsilon,S}\right), β ϵ , S − β ϵ , S = ( Σ S ϵ ) − 1 ( n ϵ 1 i = 1 ∑ n ϵ X S , i ϵ r i ϵ , S ) , and under event T 2 ∩ T 3 \mathcal{T}_2\cap\mathcal{T}_3 T 2 ∩ T 3 ,
∥ β ^ ϵ , S − β ϵ , S ∥ 2 ≤ ( 1 + ∥ β − β ∗ ∥ 2 ) ⋅ r n . (33) \left\|\widehat{\beta}^{\epsilon,S}-\beta^{\epsilon,S}\right\|_2\leq(1+\lVert\beta-\beta^*\rVert_2)\cdot r_n. \tag*{(33)} β ϵ , S − β ϵ , S 2 ≤ ( 1 + ∥ β − β ∗ ∥ 2 ) ⋅ r n . (33)
for any β \beta β with supp ( β ) ⊆ S \operatorname{supp}(\beta) \subseteq S supp ( β ) ⊆ S . Now denote Δ e , f , S : = β e , S − β f , S \Delta^{e,f,S} := \beta^{e,S} - \beta^{f,S} Δ e , f , S := β e , S − β f , S , ζ e , S : = β ^ e , S − β e , S \zeta^{e,S} := \widehat{\beta}^{e,S} - \beta^{e,S} ζ e , S := β e , S − β e , S , and D 1 ( S ) : = ∑ e < f ω e , f ∥ ζ e , S − ζ f , S ∥ 2 2 D_1(S) := \sum_{e<f}\omega^{e,f}\lVert\zeta^{e,S} - \zeta^{f,S}\rVert_2^2 D 1 ( S ) := ∑ e < f ω e , f ∥ ζ e , S − ζ f , S ∥ 2 2 . Then
∣ l ^ 1 ( S ) − l 1 ( S ) ∣ = ∣ ∑ e < f ω e , f ∥ Δ e , f , S + ( ζ e , S − ζ f , S ) ∥ 2 2 − ∑ e < f ω e , f ∥ Δ e , f , S ∥ 2 2 ∣ = ∣ 2 ∑ e < f ω e , f ⟨ Δ e , f , S , ζ e , S − ζ f , S ⟩ + ∑ e < f ω e , f ∥ ζ e , S − ζ f , S ∥ 2 2 ∣ ≤ 2 ( ∑ e < f ω e , f ∥ Δ e , f , S ∥ 2 2 ) 1 / 2 ( ∑ e < f ω e , f ∥ ζ e , S − ζ f , S ∥ 2 2 ) 1 / 2 + ∑ e < f ω e , f ∥ ζ e , S − ζ f , S ∥ 2 2 ≤ 2 l 1 ( S ) D 1 ( S ) + D 1 ( S ) ≤ ϵ 2 γ ⋅ l 1 ( S ) + ( 2 γ ϵ + 1 ) ⋅ D 1 ( S ) . \begin{aligned}
\left|\widehat{l}_1(S)-l_1(S)\right|
&=\left|\sum_{e<f}\omega^{e,f}\left\lVert\Delta^{e,f,S}+(\zeta^{e,S}-\zeta^{f,S})\right\rVert_2^2-\sum_{e<f}\omega^{e,f}\left\lVert\Delta^{e,f,S}\right\rVert_2^2\right|\\
&=\left|2\sum_{e<f}\omega^{e,f}\langle\Delta^{e,f,S},\zeta^{e,S}-\zeta^{f,S}\rangle+\sum_{e<f}\omega^{e,f}\left\lVert\zeta^{e,S}-\zeta^{f,S}\right\rVert_2^2\right|\\
&\le2\left(\sum_{e<f}\omega^{e,f}\left\lVert\Delta^{e,f,S}\right\rVert_2^2\right)^{1/2}\left(\sum_{e<f}\omega^{e,f}\left\lVert\zeta^{e,S}-\zeta^{f,S}\right\rVert_2^2\right)^{1/2}+\sum_{e<f}\omega^{e,f}\left\lVert\zeta^{e,S}-\zeta^{f,S}\right\rVert_2^2\\
&\le2\sqrt{l_1(S)}\sqrt{D_1(S)}+D_1(S)\le\frac{\epsilon}{2\gamma}\cdot l_1(S)+\left(\frac{2\gamma}{\epsilon}+1\right)\cdot D_1(S).
\end{aligned} l 1 ( S ) − l 1 ( S ) = e < f ∑ ω e , f Δ e , f , S + ( ζ e , S − ζ f , S ) 2 2 − e < f ∑ ω e , f Δ e , f , S 2 2 = 2 e < f ∑ ω e , f ⟨ Δ e , f , S , ζ e , S − ζ f , S ⟩ + e < f ∑ ω e , f ζ e , S − ζ f , S 2 2 ≤ 2 e < f ∑ ω e , f Δ e , f , S 2 2 1/2 e < f ∑ ω e , f ζ e , S − ζ f , S 2 2 1/2 + e < f ∑ ω e , f ζ e , S − ζ f , S 2 2 ≤ 2 l 1 ( S ) D 1 ( S ) + D 1 ( S ) ≤ 2 γ ϵ ⋅ l 1 ( S ) + ( ϵ 2 γ + 1 ) ⋅ D 1 ( S ) .
Note that
D 1 ( S ) = ∑ e < f ω e , f ∥ ( β ^ e , S − β e , S ) − ( β ^ f , S − β f , S ) ∥ 2 2 ≤ 4 max e ∥ β ^ e , S − β e , S ∥ 2 2 ≤ 8 r n 2 + 8 ∥ β − β ∗ ∥ 2 2 r n 2 . D_1(S)=\sum_{e<f}\omega^{e,f}\left\lVert(\widehat{\beta}^{e,S}-\beta^{e,S})-(\widehat{\beta}^{f,S}-\beta^{f,S})\right\rVert_2^2\le4\max_e\left\lVert\widehat{\beta}^{e,S}-\beta^{e,S}\right\rVert_2^2\le8r_n^2+8\lVert\beta-\beta^*\rVert_2^2r_n^2. D 1 ( S ) = e < f ∑ ω e , f ( β e , S − β e , S ) − ( β f , S − β f , S ) 2 2 ≤ 4 e max β e , S − β e , S 2 2 ≤ 8 r n 2 + 8 ∥ β − β ∗ ∥ 2 2 r n 2 .
where the last inequality follows from (33) . Moreover,
l ^ 1 ( S ∗ ) − l 1 ( S ∗ ) = l ^ 1 ( S ∗ ) = D 1 ( S ∗ ) ≤ 8 r n 2 , \widehat{l}_1(S^*)-l_1(S^*)=\widehat{l}_1(S^*)=D_1(S^*)\le8r_n^2, l 1 ( S ∗ ) − l 1 ( S ∗ ) = l 1 ( S ∗ ) = D 1 ( S ∗ ) ≤ 8 r n 2 ,
by taking β = β ∗ \beta=\beta^* β = β ∗ under the true support S ∗ S^* S ∗ . Putting them together, we have
∣ l 1 ( S ) − l 1 ( S ∗ ) − l ^ 1 ( S ) + l ^ 1 ( S ∗ ) ∣ ≤ ϵ 2 γ ⋅ l 1 ( S ) + ( 2 γ ϵ + 1 ) D 1 ( S ) + D 1 ( S ∗ ) ≤ ϵ 2 γ ⋅ l 1 ( S ) + ( 16 γ ϵ + 8 ) ( r n 2 + ∥ β − β ∗ ∥ 2 2 r n 2 ) + 8 r n 2 , \begin{aligned}
\left|l_1(S)-l_1(S^*)-\widehat{l}_1(S)+\widehat{l}_1(S^*)\right| \\
&\le\frac{\epsilon}{2\gamma}\cdot l_1(S)+\left(\frac{2\gamma}{\epsilon}+1\right)D_1(S)+D_1(S^*)\le\frac{\epsilon}{2\gamma}\cdot l_1(S)+\left(\frac{16\gamma}{\epsilon}+8\right)(r_n^2+\lVert\beta-\beta^*\rVert_2^2r_n^2)+8r_n^2,
\end{aligned} l 1 ( S ) − l 1 ( S ∗ ) − l 1 ( S ) + l 1 ( S ∗ ) ≤ 2 γ ϵ ⋅ l 1 ( S ) + ( ϵ 2 γ + 1 ) D 1 ( S ) + D 1 ( S ∗ ) ≤ 2 γ ϵ ⋅ l 1 ( S ) + ( ϵ 16 γ + 8 ) ( r n 2 + ∥ β − β ∗ ∥ 2 2 r n 2 ) + 8 r n 2 ,
for any β ∈ R p \beta\in\mathbb{R}^p β ∈ R p with supp ( β ) = S \operatorname{supp}(\beta)=S supp ( β ) = S .
Step II. Bound ∣ l 2 ( S ) − l 2 ( S ∗ ) − l ^ 2 ( S ) + l ^ 2 ( S ∗ ) ∣ \left|l_2(S)-l_2(S^*)-\widehat{l}_2(S)+\widehat{l}_2(S^*)\right| l 2 ( S ) − l 2 ( S ∗ ) − l 2 ( S ) + l 2 ( S ∗ ) . For any ϵ ∈ E \epsilon\in\mathcal{E} ϵ ∈ E , the sample variances of the OLS residuals satisfy
ν ^ e , S − ν e , S = [ 1 n e ∑ i = 1 n e ( r i e , S ) 2 − E [ ( r e , S ) 2 ] ] − ( 1 n e ∑ i = 1 n e X S , i e r i e , S ) ( Σ ^ e ) − 1 ( 1 n e ∑ i = 1 n e X S , i e r i e , S ) , \widehat{\nu}^{e,S}-\nu^{e,S}=\left[\frac{1}{n^e}\sum_{i=1}^{n^e}(r_i^{e,S})^2-\mathbb{E}\left[(r^{e,S})^2\right]\right]-\left(\frac{1}{n^e}\sum_{i=1}^{n^e}X_{S,i}^{e}r_i^{e,S}\right)\left(\widehat{\Sigma}^{e}\right)^{-1}\left(\frac{1}{n^e}\sum_{i=1}^{n^e}X_{S,i}^{e}r_i^{e,S}\right), ν e , S − ν e , S = [ n e 1 i = 1 ∑ n e ( r i e , S ) 2 − E [ ( r e , S ) 2 ] ] − ( n e 1 i = 1 ∑ n e X S , i e r i e , S ) ( Σ e ) − 1 ( n e 1 i = 1 ∑ n e X S , i e r i e , S ) ,
and under T 2 ∩ T 3 ∩ T 4 \mathcal{T}_2\cap\mathcal{T}_3\cap\mathcal{T}_4 T 2 ∩ T 3 ∩ T 4 ,
∣ ν ^ e , S − ν e , S ∣ ≤ ( 1 + ∥ β − β ∗ ∥ 2 ) 2 ⋅ r n (34) \left|\widehat{\nu}^{e,S}-\nu^{e,S}\right|\le(1+\lVert\beta-\beta^*\rVert_2)^2\cdot r_n \tag*{(34)} ν e , S − ν e , S ≤ ( 1 + ∥ β − β ∗ ∥ 2 ) 2 ⋅ r n (34)
for any β \beta β with supp ( β ) ⊆ S \operatorname{supp}(\beta)\subseteq S supp ( β ) ⊆ S . Now denote Δ ~ e , f , S : = ν e , S − ν f , S \widetilde{\Delta}^{e,f,S}:=\nu^{e,S}-\nu^{f,S} Δ e , f , S := ν e , S − ν f , S , ζ ~ e , S : = ν ^ e , S − ν e , S \widetilde{\zeta}^{e,S}:=\widehat{\nu}^{e,S}-\nu^{e,S} ζ e , S := ν e , S − ν e , S , and D 2 ( S ) : = ∑ e < f ω e , f ∣ ζ ~ e , S − ζ ~ f , S ∣ 2 D_2(S):=\sum_{e<f}\omega^{e,f}\left|\widetilde{\zeta}^{e,S}-\widetilde{\zeta}^{f,S}\right|^2 D 2 ( S ) := ∑ e < f ω e , f ζ e , S − ζ f , S 2 . Then as in Step I,
∣ l ^ 2 ( S ) − l 2 ( S ) ∣ ≤ 2 l 2 ( S ) D 2 ( S ) + D 2 ( S ) ≤ ϵ 2 ρ γ ⋅ l 2 ( S ) + ( 2 ρ γ ϵ + 1 ) ⋅ D 2 ( S ) . \left|\widehat{l}_2(S)-l_2(S)\right|\le2\sqrt{l_2(S)}\sqrt{D_2(S)}+D_2(S)\le\frac{\epsilon}{2\rho\gamma}\cdot l_2(S)+\left(\frac{2\rho\gamma}{\epsilon}+1\right)\cdot D_2(S). l 2 ( S ) − l 2 ( S ) ≤ 2 l 2 ( S ) D 2 ( S ) + D 2 ( S ) ≤ 2 ρ γ ϵ ⋅ l 2 ( S ) + ( ϵ 2 ρ γ + 1 ) ⋅ D 2 ( S ) .
Note that
D 2 ( S ) = ∑ e < f ω e , f ∣ ( ν ^ e , S − ν e , S ) − ( ν ^ f , S − ν f , S ) ∣ 2 ≤ 4 max e ∣ ν ^ e , S − ν e , S ∣ 2 ≤ 32 r n 2 + 32 ∥ β − β ∗ ∥ 2 4 r n 2 D_2(S)=\sum_{e<f}\omega^{e,f}\left|(\widehat{\nu}^{e,S}-\nu^{e,S})-(\widehat{\nu}^{f,S}-\nu^{f,S})\right|^2\le4\max_e\left|\widehat{\nu}^{e,S}-\nu^{e,S}\right|^2\le32r_n^2+32\lVert\beta-\beta^*\rVert_2^4r_n^2 D 2 ( S ) = e < f ∑ ω e , f ( ν e , S − ν e , S ) − ( ν f , S − ν f , S ) 2 ≤ 4 e max ν e , S − ν e , S 2 ≤ 32 r n 2 + 32 ∥ β − β ∗ ∥ 2 4 r n 2
where the last inequality follows from (34) . Moreover,
l ^ 2 ( S ∗ ) − l 2 ( S ∗ ) = l ^ 2 ( S ∗ ) = D 2 ( S ∗ ) ≤ 32 r n 2 , \widehat{l}_2(S^*)-l_2(S^*)=\widehat{l}_2(S^*)=D_2(S^*)\le32r_n^2, l 2 ( S ∗ ) − l 2 ( S ∗ ) = l 2 ( S ∗ ) = D 2 ( S ∗ ) ≤ 32 r n 2 ,
by taking β = β ∗ \beta=\beta^* β = β ∗ under the true support S ∗ S^* S ∗ . Putting them together, we have
∣ l 2 ( S ) − l 2 ( S ∗ ) − l ^ 2 ( S ) + l ^ 2 ( S ∗ ) ∣ ≤ ϵ 2 ρ γ ⋅ l 2 ( S ) + ( 2 ρ γ ϵ + 1 ) D 2 ( S ) + D 2 ( S ∗ ) ≤ ϵ 2 ρ γ ⋅ l 2 ( S ) + ( 64 ρ γ ϵ + 32 ) ( r n 2 + ∥ β − β ∗ ∥ 2 4 r n 2 ) + 32 r n 2 . \begin{aligned}
\left|l_2(S)-l_2(S^*)-\widehat{l}_2(S)+\widehat{l}_2(S^*)\right| \\
&\le\frac{\epsilon}{2\rho\gamma}\cdot l_2(S)+\left(\frac{2\rho\gamma}{\epsilon}+1\right)D_2(S)+D_2(S^*)\le\frac{\epsilon}{2\rho\gamma}\cdot l_2(S)+\left(\frac{64\rho\gamma}{\epsilon}+32\right)(r_n^2+\lVert\beta-\beta^*\rVert_2^4r_n^2)+32r_n^2.
\end{aligned} l 2 ( S ) − l 2 ( S ∗ ) − l 2 ( S ) + l 2 ( S ∗ ) ≤ 2 ρ γ ϵ ⋅ l 2 ( S ) + ( ϵ 2 ρ γ + 1 ) D 2 ( S ) + D 2 ( S ∗ ) ≤ 2 ρ γ ϵ ⋅ l 2 ( S ) + ( ϵ 64 ρ γ + 32 ) ( r n 2 + ∥ β − β ∗ ∥ 2 4 r n 2 ) + 32 r n 2 .
Step III. Combine the results. We combine the bounds obtained in the previous steps and conclude that, for any S S S with ∣ S ∣ ≤ s n |S| \le s_n ∣ S ∣ ≤ s n ,
∣ I ( S ) − I ( S ⋆ ) − I ^ ( S ) + I ^ ( S ⋆ ) ∣ ≤ ∣ I 1 ( S ) − I 1 ( S ⋆ ) − I ^ 1 ( S ) + I ^ 1 ( S ⋆ ) ∣ + ρ ∣ I 2 ( S ) − I 2 ( S ⋆ ) − I ^ 2 ( S ) + I ^ 2 ( S ⋆ ) ∣ ≤ ϵ / ( 2 γ ) ⋅ ( I 1 ( S ) + I 2 ( S ) ) + ( ( 16 + 64 ρ 2 ) γ / ϵ + 16 + 64 ρ ) r n 2 + ( 16 γ / ϵ + 8 ) r n 2 ⋅ ∥ β − β ⋆ ∥ 2 2 + ( 64 ρ 2 γ / ϵ + 32 ρ ) r n 2 ⋅ ∥ β − β ⋆ ∥ 2 4 . \begin{aligned}
\left|I(S)-I(S^\star)-\widehat{I}(S)+\widehat{I}(S^\star)\right|
&\le\left|I_1(S)-I_1(S^\star)-\widehat{I}_1(S)+\widehat{I}_1(S^\star)\right| \\
&\quad+ \rho\left|I_2(S)-I_2(S^\star)-\widehat{I}_2(S)+\widehat{I}_2(S^\star)\right| \\
&\le\epsilon/(2\gamma)\cdot\left(I_1(S)+I_2(S)\right)
+\left((16+64\rho^2)\gamma/\epsilon+16+64\rho\right)r_n^2 \\
&\quad+(16\gamma/\epsilon+8)r_n^2\cdot\|\beta-\beta^\star\|_2^2
+(64\rho^2\gamma/\epsilon+32\rho)r_n^2\cdot\|\beta-\beta^\star\|_2^4.
\end{aligned} I ( S ) − I ( S ⋆ ) − I ( S ) + I ( S ⋆ ) ≤ I 1 ( S ) − I 1 ( S ⋆ ) − I 1 ( S ) + I 1 ( S ⋆ ) + ρ I 2 ( S ) − I 2 ( S ⋆ ) − I 2 ( S ) + I 2 ( S ⋆ ) ≤ ϵ / ( 2 γ ) ⋅ ( I 1 ( S ) + I 2 ( S ) ) + ( ( 16 + 64 ρ 2 ) γ / ϵ + 16 + 64 ρ ) r n 2 + ( 16 γ / ϵ + 8 ) r n 2 ⋅ ∥ β − β ⋆ ∥ 2 2 + ( 64 ρ 2 γ / ϵ + 32 ρ ) r n 2 ⋅ ∥ β − β ⋆ ∥ 2 4 .
Lemma 12. Under the event T 2 ∩ T 4 \mathcal{T}_2 \cap\mathcal{T}_4 T 2 ∩ T 4 , for any S S S with ∣ S ∣ ≤ s n |S| \le s_n ∣ S ∣ ≤ s n , the corresponding OLS coefficients
β O L S , S : = arg min supp ( β ) ⊆ S ∑ e ∈ E ω e ∥ Y e − X e β ∥ n e 2 \beta^{\mathrm{OLS},S}:=\underset{\operatorname{supp}(\beta)\subseteq S}{\arg\min}\sum_{e\in\mathcal{E}}\omega^e\left\|Y^e-X^e\beta\right\|_{n_e}^2 β OLS , S := supp ( β ) ⊆ S arg min e ∈ E ∑ ω e ∥ Y e − X e β ∥ n e 2 satisfy
∥ β O L S , S − β ⋆ ∥ 2 2 ≲ 1 + ∥ β ⋆ ∥ 2 2 . \left\|\beta^{\mathrm{OLS},S}-\beta^\star\right\|_2^2\lesssim1+\|\beta^\star\|_2^2. β OLS , S − β ⋆ 2 2 ≲ 1 + ∥ β ⋆ ∥ 2 2 . Proof. Note that
∑ e ω e λ min ( Σ ^ S e ) ⋅ ∥ β O L S , S ∥ 2 2 ≤ ∑ e ω e ( ∥ X S e T β O L S , S ∥ n e 2 + ∥ Y e − X S e β O L S , S ∥ n e 2 ) = ∑ e ω e ∥ Y e ∥ n e 2 , \sum_e\omega^e\lambda_{\min}\left(\widehat{\Sigma}_S^e\right)\cdot\left\|\beta^{\mathrm{OLS},S}\right\|_2^2
\le\sum_e\omega^e\left(\left\|X_S^{e\mathsf{T}}\beta^{\mathrm{OLS},S}\right\|_{n_e}^2+\left\|Y^e-X_S^e\beta^{\mathrm{OLS},S}\right\|_{n_e}^2\right)
=\sum_e\omega^e\left\|Y^e\right\|_{n_e}^2, e ∑ ω e λ m i n ( Σ S e ) ⋅ β OLS , S 2 2 ≤ e ∑ ω e ( X S e T β OLS , S n e 2 + Y e − X S e β OLS , S n e 2 ) = e ∑ ω e ∥ Y e ∥ n e 2 , where the last equality follows from the normal equations. On the one hand, under the event T 2 \mathcal{T}_2 T 2 , we have λ min ( Σ ^ e ) ≳ 1 \lambda_{\min}(\widehat{\Sigma}^e)\gtrsim1 λ m i n ( Σ e ) ≳ 1 . On the other hand, under the event T 4 \mathcal{T}_4 T 4 and by taking S = S ⋆ S=S^\star S = S ⋆ and β = β ⋆ \beta=\beta^\star β = β ⋆ , we have ∥ ε e ∥ n e 2 ≲ r n + σ ε 2 ≳ 1 \|\varepsilon^e\|_{n_e}^2\lesssim r_n+\sigma_\varepsilon^2\gtrsim1 ∥ ε e ∥ n e 2 ≲ r n + σ ε 2 ≳ 1 . As a result,
∥ Y e ∥ n e ≤ ∥ X S ⋆ e β S ⋆ ⋆ ∥ n e + ∥ ε e ∥ n e ≲ ∥ β ⋆ ∥ 2 + 1. \left\|Y^e\right\|_{n_e}\le\left\|X_{S^\star}^e\beta_{S^\star}^\star\right\|_{n_e}+\left\|\varepsilon^e\right\|_{n_e}\lesssim\|\beta^\star\|_2+1. ∥ Y e ∥ n e ≤ ∥ X S ⋆ e β S ⋆ ⋆ ∥ n e + ∥ ε e ∥ n e ≲ ∥ β ⋆ ∥ 2 + 1. Consequently,
∥ β O L S , S − β ⋆ ∥ 2 2 ≤ 2 ∥ β O L S , S ∥ 2 2 + 2 ∥ β ⋆ ∥ 2 2 ≲ 1 + ∥ β ⋆ ∥ 2 2 . \left\|\beta^{\mathrm{OLS},S}-\beta^\star\right\|_2^2
\le2\left\|\beta^{\mathrm{OLS},S}\right\|_2^2+2\left\|\beta^\star\right\|_2^2
\lesssim1+\left\|\beta^\star\right\|_2^2. β OLS , S − β ⋆ 2 2 ≤ 2 β OLS , S 2 2 + 2 ∥ β ⋆ ∥ 2 2 ≲ 1 + ∥ β ⋆ ∥ 2 2 .
Lemma 13. Suppose all conditions of Lemma 6 hold. Under the event T 1 ∩ T 2 ∩ T 3 ∩ T 4 \mathcal{T}_1\cap\mathcal{T}_2\cap\mathcal{T}_3\cap\mathcal{T}_4 T 1 ∩ T 2 ∩ T 3 ∩ T 4 ,
F ^ ( β ) − F ^ ( β ⋆ ) ≳ ∥ β − β ⋆ ∥ 2 2 − s n log p + log ∣ E ∣ min e ∈ E n e − 2 λ n s n , \widehat{F}(\beta)-\widehat{F}(\beta^\star)\gtrsim\|\beta-\beta^\star\|_2^2-\frac{s_n\log p+\log|\mathcal{E}|}{\min_{e\in\mathcal{E}}n_e}-2\lambda_n s_n, F ( β ) − F ( β ⋆ ) ≳ ∥ β − β ⋆ ∥ 2 2 − min e ∈ E n e s n log p + log ∣ E ∣ − 2 λ n s n , where β \beta β is the OLS coefficients β O L S , S \beta^{\mathrm{OLS},S} β OLS , S , and ∣ S ∣ ≤ s n |S|\le s_n ∣ S ∣ ≤ s n .
Proof. Consider the decomposition:
F ^ ( β ) − F ^ ( β ⋆ ) = − γ D I ( S , S ⋆ ) − D R ( β , β ⋆ ) + F ( β ) − F ( β ⋆ ) + λ n ( ∣ S ∣ − s ⋆ ) , \widehat{F}(\beta)-\widehat{F}(\beta^\star)
=-\gamma D_I(S,S^\star)-D_R(\beta,\beta^\star)+F(\beta)-F(\beta^\star)+\lambda_n(|S|-s^\star), F ( β ) − F ( β ⋆ ) = − γ D I ( S , S ⋆ ) − D R ( β , β ⋆ ) + F ( β ) − F ( β ⋆ ) + λ n ( ∣ S ∣ − s ⋆ ) , where
D I ( S , S ⋆ ) : = I ( S ) − I ( S ⋆ ) − I ^ ( S ) + I ^ ( S ⋆ ) , D_I(S,S^\star):=I(S)-I(S^\star)-\widehat{I}(S)+\widehat{I}(S^\star), D I ( S , S ⋆ ) := I ( S ) − I ( S ⋆ ) − I ( S ) + I ( S ⋆ ) , which can be bounded using Lemma 11; and
D R ( β , β ⋆ ) : = R ( β ) − R ( β ⋆ ) − R ^ ( β ) + R ^ ( β ⋆ ) , D_R(\beta,\beta^\star):=R(\beta)-R(\beta^\star)-\widehat{R}(\beta)+\widehat{R}(\beta^\star), D R ( β , β ⋆ ) := R ( β ) − R ( β ⋆ ) − R ( β ) + R ( β ⋆ ) ,
which can be bounded using Lemma 7. Moreover, Lemma 6 provides a bound for F ( β ) − F ( β ∗ ) \mathrm{F}(\beta)-\mathrm{F}(\beta^{*}) F ( β ) − F ( β ∗ ) . Putting them together, we obtain:
F ^ ( β ) − F ^ ( β ∗ ) ≥ − ϵ 2 ⋅ ( I 1 ( S ) + I 2 ( S ) ) − γ ( ( 16 + 64 ρ 2 ) γ / ϵ + 16 + 64 ρ 2 ) r n 2 − γ ( 16 γ / ϵ + 8 ) r n 2 ⋅ ∥ β − β ∗ ∥ 2 2 − γ ( 64 ρ 2 γ / ϵ + 32 ρ ) r n 2 ⋅ ∥ β − β ∗ ∥ 2 4 − ( C 0 δ n + κ L 4 ) ∥ β − β ∗ ∥ 2 2 − ( C 0 ′ δ n ) 2 κ L + 1 2 κ L ∥ β − β ∗ ∥ 2 2 + ϵ ⋅ ( I 1 ( S ) + I 2 ( S ) ) + λ n ( ∣ S ∣ − s ∗ ) ≳ ∥ β − β ∗ ∥ 2 2 − r n 2 − δ n 2 − r n 2 ∥ β − β ∗ ∥ 2 4 − 2 λ n s n . \begin{aligned}
\widehat{\mathrm{F}}(\beta)-\widehat{\mathrm{F}}(\beta^{*}) \ge{}&
-\frac{\epsilon}{2}\cdot(I_{1}(S)+I_{2}(S))-\gamma\left((16+64\rho^{2})\gamma/\epsilon+16+64\rho^{2}\right)r_{n}^{2}\\
&-\gamma(16\gamma/\epsilon+8)r_{n}^{2}\cdot\|\beta-\beta^{*}\|_{2}^{2}
-\gamma(64\rho^{2}\gamma/\epsilon+32\rho)r_{n}^{2}\cdot\|\beta-\beta^{*}\|_{2}^{4}\\
&-\left(C_{0}\delta_{n}+\frac{\kappa L}{4}\right)\|\beta-\beta^{*}\|_{2}^{2}
-\frac{(C_{0}^{\prime}\delta_{n})^{2}}{\kappa_{L}}
+\frac{1}{2}\kappa_{L}\|\beta-\beta^{*}\|_{2}^{2}
+\epsilon\cdot(I_{1}(S)+I_{2}(S))\\
&+\lambda_{n}(|S|-s^{*})\\
\gtrsim{}&\|\beta-\beta^{*}\|_{2}^{2}-r_{n}^{2}-\delta_{n}^{2}-r_{n}^{2}\|\beta-\beta^{*}\|_{2}^{4}-2\lambda_{n}s_{n}.
\end{aligned} F ( β ) − F ( β ∗ ) ≥ ≳ − 2 ϵ ⋅ ( I 1 ( S ) + I 2 ( S )) − γ ( ( 16 + 64 ρ 2 ) γ / ϵ + 16 + 64 ρ 2 ) r n 2 − γ ( 16 γ / ϵ + 8 ) r n 2 ⋅ ∥ β − β ∗ ∥ 2 2 − γ ( 64 ρ 2 γ / ϵ + 32 ρ ) r n 2 ⋅ ∥ β − β ∗ ∥ 2 4 − ( C 0 δ n + 4 κ L ) ∥ β − β ∗ ∥ 2 2 − κ L ( C 0 ′ δ n ) 2 + 2 1 κ L ∥ β − β ∗ ∥ 2 2 + ϵ ⋅ ( I 1 ( S ) + I 2 ( S )) + λ n ( ∣ S ∣ − s ∗ ) ∥ β − β ∗ ∥ 2 2 − r n 2 − δ n 2 − r n 2 ∥ β − β ∗ ∥ 2 4 − 2 λ n s n .
Further note that δ n ≲ r n \delta_{n}\lesssim r_{n} δ n ≲ r n , and by Lemma 12,
r n 2 ∥ β − β ∗ ∥ 2 4 ≲ r n 2 ( 1 + ∥ β ∗ ∥ 2 2 ) ⋅ ∥ β − β ∗ ∥ 2 2 ≲ ( r n 2 + r n δ ) ⋅ ∥ β − β ∗ ∥ 2 2 ≲ r n δ ∥ β − β ∗ ∥ 2 2 , r_{n}^{2}\|\beta-\beta^{*}\|_{2}^{4}\lesssim r_{n}^{2}(1+\|\beta^{*}\|_{2}^{2})\cdot\|\beta-\beta^{*}\|_{2}^{2}\lesssim(r_{n}^{2}+r_{n}^{\delta})\cdot\|\beta-\beta^{*}\|_{2}^{2}\lesssim r_{n}^{\delta}\|\beta-\beta^{*}\|_{2}^{2}, r n 2 ∥ β − β ∗ ∥ 2 4 ≲ r n 2 ( 1 + ∥ β ∗ ∥ 2 2 ) ⋅ ∥ β − β ∗ ∥ 2 2 ≲ ( r n 2 + r n δ ) ⋅ ∥ β − β ∗ ∥ 2 2 ≲ r n δ ∥ β − β ∗ ∥ 2 2 ,
where the second to the last inequality follows from Assumption 3. As a result,
F ^ ( β ) − F ^ ( β ∗ ) ≳ ∥ β − β ∗ ∥ 2 2 − r n 2 − 2 λ n s n . \widehat{\mathrm{F}}(\beta)-\widehat{\mathrm{F}}(\beta^{*})\gtrsim\|\beta-\beta^{*}\|_{2}^{2}-r_{n}^{2}-2\lambda_{n}s_{n}. F ( β ) − F ( β ∗ ) ≳ ∥ β − β ∗ ∥ 2 2 − r n 2 − 2 λ n s n .
□
Proof of Theorem 2
Proof. We first provide the proof for the optimal solution β ^ \widehat{\beta} β . Appealing to the basic inequality and Lemma 13:
∥ β ^ − β ∗ ∥ 2 2 − s n log p + log ∣ E ∣ min e ∈ E n e − 2 λ n s n ≲ F ^ ( β ^ ) − F ^ ( β ∗ ) ≤ 0. \|\widehat{\beta}-\beta^{*}\|_{2}^{2}-\frac{s_{n}\log p+\log|\mathcal{E}|}{\min_{e\in\mathcal{E}}n^{e}}-2\lambda_{n}s_{n}\lesssim\widehat{\mathrm{F}}(\widehat{\beta})-\widehat{\mathrm{F}}(\beta^{*})\le0. ∥ β − β ∗ ∥ 2 2 − min e ∈ E n e s n log p + log ∣ E ∣ − 2 λ n s n ≲ F ( β ) − F ( β ∗ ) ≤ 0.
The result hence follows.
For the early-stopping solution β ^ e a r l y \widehat{\beta}^{\mathrm{early}} β early , we have
F ^ ( β ^ e a r l y ) − F ^ ( β ∗ ) = ( F ^ ( β ^ e a r l y ) − F ^ ( β ^ ) ) + ( F ^ ( β ^ ) − F ^ ( β ∗ ) ) ≤ τ e a r l y + 0 = τ e a r l y . \widehat{\mathrm{F}}(\widehat{\beta}^{\mathrm{early}})-\widehat{\mathrm{F}}(\beta^{*})=\left(\widehat{\mathrm{F}}(\widehat{\beta}^{\mathrm{early}})-\widehat{\mathrm{F}}(\widehat{\beta})\right)+\left(\widehat{\mathrm{F}}(\widehat{\beta})-\widehat{\mathrm{F}}(\beta^{*})\right)\le\tau^{\mathrm{early}}+0=\tau^{\mathrm{early}}. F ( β early ) − F ( β ∗ ) = ( F ( β early ) − F ( β ) ) + ( F ( β ) − F ( β ∗ ) ) ≤ τ early + 0 = τ early .
Then combining with the results in Lemma 13, we have
∥ β ^ e a r l y − β ∗ ∥ 2 2 − s n log p + log ∣ E ∣ min e ∈ E n e − 2 λ n s n ≲ F ^ ( β ^ e a r l y ) − F ^ ( β ∗ ) ≤ τ e a r l y . \|\widehat{\beta}^{\mathrm{early}}-\beta^{*}\|_{2}^{2}-\frac{s_{n}\log p+\log|\mathcal{E}|}{\min_{e\in\mathcal{E}}n^{e}}-2\lambda_{n}s_{n}\lesssim\widehat{\mathrm{F}}(\widehat{\beta}^{\mathrm{early}})-\widehat{\mathrm{F}}(\beta^{*})\le\tau^{\mathrm{early}}. ∥ β early − β ∗ ∥ 2 2 − min e ∈ E n e s n log p + log ∣ E ∣ − 2 λ n s n ≲ F ( β early ) − F ( β ∗ ) ≤ τ early .
The result hence follows. □
Proof of Theorem 3
Proof of part (1)
Proof. This proof proceeds in three steps. In each step, we solve an optimization problem that is more constrained than the original one.
R 1 = { β ∈ R p : supp ( β ) = supp ( β ∗ ) } , \mathcal{R}_{1}=\{\beta\in\mathbb{R}^{p}:\operatorname{supp}(\beta)=\operatorname{supp}(\beta^{*})\}, R 1 = { β ∈ R p : supp ( β ) = supp ( β ∗ )} ,
and derive an upper bound for min β ∈ R 1 F ^ ( β ) − F ^ ( β ∗ ) \min_{\beta\in\mathcal{R}_{1}}\widehat{\mathrm{F}}(\beta)-\widehat{\mathrm{F}}(\beta^{*}) min β ∈ R 1 F ( β ) − F ( β ∗ ) . We denote this upper bound by u u u .
R 2 = { β ∈ R p : ∣ supp ( β ) ∣ ≤ s ∗ , supp ( β ) ≠ supp ( β ∗ ) } , \mathcal{R}_{2}=\{\beta\in\mathbb{R}^{p}:|\operatorname{supp}(\beta)|\le s^{*},\ \operatorname{supp}(\beta)\ne\operatorname{supp}(\beta^{*})\}, R 2 = { β ∈ R p : ∣ supp ( β ) ∣ ≤ s ∗ , supp ( β ) = supp ( β ∗ )} ,
and derive a lower bound for min β ∈ R 2 F ^ ( β ) − F ^ ( β ∗ ) \min_{\beta\in\mathcal{R}_{2}}\widehat{\mathrm{F}}(\beta)-\widehat{\mathrm{F}}(\beta^{*}) min β ∈ R 2 F ( β ) − F ( β ∗ ) . We denote this lower bound by l 1 l_{1} l 1 , and verify that u < l 1 u<l_{1} u < l 1 when min e ∈ E n e \min_{e\in\mathcal{E}}n^{e} min e ∈ E n e is sufficiently large.
R 3 = { β ∈ R p : ∣ supp ( β ) ∣ > s ⋆ } , \mathcal{R}_3 = \{\beta\in\mathbb{R}^p : |\operatorname{supp}(\beta)| > s^\star\}, R 3 = { β ∈ R p : ∣ supp ( β ) ∣ > s ⋆ } ,
and derive a lower bound for min β ∈ R 3 F ^ ( β ) − F ^ ( β ⋆ ) \min_{\beta\in\mathcal{R}_3} \widehat{F}(\beta) - \widehat{F}(\beta^\star) min β ∈ R 3 F ( β ) − F ( β ⋆ ) . We denote this lower bound by l 2 l_2 l 2 , and verify that u < l 2 u < l_2 u < l 2 when min e ∈ E n e \min_{e \in\mathcal{E}} n^e min e ∈ E n e is sufficiently large.
The three steps above guarantee that the solution to the original problem, arg min β F ^ ( β ) \arg\min_\beta\widehat{F}(\beta) arg min β F ( β ) , falls inside R 1 \mathcal{R}_1 R 1 when min e ∈ E n e \min_{e \in\mathcal{E}} n^e min e ∈ E n e is sufficiently large. This proof relies on the event T 1 ∩ T 2 ∩ T 3 ∩ T 4 \mathcal{T}_1 \cap\mathcal{T}_2 \cap\mathcal{T}_3 \cap\mathcal{T}_4 T 1 ∩ T 2 ∩ T 3 ∩ T 4 , which holds with probability at least 1 − 7 / p 1 - 7/p 1 − 7/ p .
Step I. In this step, we use β ˉ \bar{\beta} β ˉ to denote the solution to min β ∈ R 1 F ^ ( β ) − F ^ ( β ⋆ ) \min_{\beta\in\mathcal{R}_1} \widehat{F}(\beta) - \widehat{F}(\beta^\star) min β ∈ R 1 F ( β ) − F ( β ⋆ ) . Note that for any β ∈ R 1 \beta\in\mathcal{R}_1 β ∈ R 1 , we have F ^ ( β ) − F ^ ( β ⋆ ) = R ^ ( β ) − R ^ ( β ⋆ ) \widehat{F}(\beta) - \widehat{F}(\beta^\star) = \widehat{R}(\beta) - \widehat{R}(\beta^\star) F ( β ) − F ( β ⋆ ) = R ( β ) − R ( β ⋆ ) . Therefore, β ˉ \bar{\beta} β ˉ is the pooled OLS coefficients, and
β ˉ S ⋆ : = ( ∑ e ω e X S ⋆ e T X S ⋆ e n e ) − 1 ( ∑ e ω e X S ⋆ e T Y e n e ) = β S ⋆ ⋆ + ( ∑ e ω e X S ⋆ e T X S ⋆ e n e ) − 1 ( ∑ e ω e X S ⋆ e T ϵ e n e ) . \bar{\beta}_{S^\star} := \left(\sum_e \omega^e \frac{X_{S^\star}^{e\mathsf{T}} X_{S^\star}^e}{n^e}\right)^{-1}\left(\sum_e \omega^e \frac{X_{S^\star}^{e\mathsf{T}} Y^e}{n^e}\right) = \beta_{S^\star}^\star+ \left(\sum_e \omega^e \frac{X_{S^\star}^{e\mathsf{T}} X_{S^\star}^e}{n^e}\right)^{-1}\left(\sum_e \omega^e \frac{X_{S^\star}^{e\mathsf{T}} \epsilon^e}{n^e}\right). β ˉ S ⋆ := ( e ∑ ω e n e X S ⋆ e T X S ⋆ e ) − 1 ( e ∑ ω e n e X S ⋆ e T Y e ) = β S ⋆ ⋆ + ( e ∑ ω e n e X S ⋆ e T X S ⋆ e ) − 1 ( e ∑ ω e n e X S ⋆ e T ϵ e ) .
Let A : = ∑ e ω e X S ⋆ e T X S ⋆ e / n e A := \sum_e \omega^e X_{S^\star}^{e\mathsf{T}} X_{S^\star}^e/n^e A := ∑ e ω e X S ⋆ e T X S ⋆ e / n e and b : = ∑ e ω e X S ⋆ e T ϵ e / n e b := \sum_e \omega^e X_{S^\star}^{e\mathsf{T}} \epsilon^e/n^e b := ∑ e ω e X S ⋆ e T ϵ e / n e . Then we can rewrite β ˉ S ⋆ − β S ⋆ ⋆ = A − 1 b \bar{\beta}_{S^\star} - \beta_{S^\star}^\star= A^{-1}b β ˉ S ⋆ − β S ⋆ ⋆ = A − 1 b , and hence
R ^ ( β ˉ ) − R ^ ( β ⋆ ) = ∑ e ω e ( ∥ Y e − X S ⋆ e β ˉ S ⋆ ∥ n e 2 − ∥ Y e − X S ⋆ e β S ⋆ ⋆ ∥ n e 2 ) = ∑ e ω e ( ∥ ϵ e − X S ⋆ e ( β ˉ S ⋆ − β S ⋆ ⋆ ) ∥ n e 2 − ∥ ϵ e ∥ n e 2 ) = ( β ˉ S ⋆ − β S ⋆ ⋆ ) T A ( β ˉ S ⋆ − β S ⋆ ⋆ ) − 2 ( β ˉ S ⋆ − β S ⋆ ⋆ ) T b = ( A − 1 b ) T A ( A − 1 b ) − 2 ( A − 1 b ) T b = − b T A − 1 b = − ( ∑ e ω e X S ⋆ e T ϵ e n e ) T ( ∑ e ω e X S ⋆ e T X S ⋆ e n e ) − 1 ( ∑ e ω e X S ⋆ e T ϵ e n e ) . \begin{aligned}
\widehat{R}(\bar{\beta}) - \widehat{R}(\beta^\star) = \sum_e \omega^e \left(\left\|Y^e - X_{S^\star}^e \bar{\beta}_{S^\star}\right\|_{n^e}^2 - \left\|Y^e - X_{S^\star}^e \beta_{S^\star}^\star\right\|_{n^e}^2\right) \\
&= \sum_e \omega^e \left(\left\|\epsilon^e - X_{S^\star}^e(\bar{\beta}_{S^\star} - \beta_{S^\star}^\star)\right\|_{n^e}^2 - \left\|\epsilon^e\right\|_{n^e}^2\right) \\
&= (\bar{\beta}_{S^\star} - \beta_{S^\star}^\star)^\mathsf{T} A(\bar{\beta}_{S^\star} - \beta_{S^\star}^\star) - 2(\bar{\beta}_{S^\star} - \beta_{S^\star}^\star)^\mathsf{T}b \\
&= (A^{-1}b)^\mathsf{T}A(A^{-1}b) - 2(A^{-1}b)^\mathsf{T}b = -b^\mathsf{T}A^{-1}b \\
&= -\left(\sum_e \omega^e \frac{X_{S^\star}^{e\mathsf{T}}\epsilon^e}{n^e}\right)^\mathsf{T}
\left(\sum_e \omega^e \frac{X_{S^\star}^{e\mathsf{T}}X_{S^\star}^e}{n^e}\right)^{-1}
\left(\sum_e \omega^e \frac{X_{S^\star}^{e\mathsf{T}}\epsilon^e}{n^e}\right).
\end{aligned} R ( β ˉ ) − R ( β ⋆ ) = e ∑ ω e ( Y e − X S ⋆ e β ˉ S ⋆ n e 2 − ∥ Y e − X S ⋆ e β S ⋆ ⋆ ∥ n e 2 ) = e ∑ ω e ( ϵ e − X S ⋆ e ( β ˉ S ⋆ − β S ⋆ ⋆ ) n e 2 − ∥ ϵ e ∥ n e 2 ) = ( β ˉ S ⋆ − β S ⋆ ⋆ ) T A ( β ˉ S ⋆ − β S ⋆ ⋆ ) − 2 ( β ˉ S ⋆ − β S ⋆ ⋆ ) T b = ( A − 1 b ) T A ( A − 1 b ) − 2 ( A − 1 b ) T b = − b T A − 1 b = − ( e ∑ ω e n e X S ⋆ e T ϵ e ) T ( e ∑ ω e n e X S ⋆ e T X S ⋆ e ) − 1 ( e ∑ ω e n e X S ⋆ e T ϵ e ) .
Now denote Σ S : = ∑ e ω e Σ S e \Sigma_S := \sum_e \omega^e \Sigma_S^e Σ S := ∑ e ω e Σ S e and Σ ^ S : = ∑ e ω e Σ ^ S e \widehat{\Sigma}_S := \sum_e \omega^e \widehat{\Sigma}_S^e Σ S := ∑ e ω e Σ S e . Appealing to Weyl’s inequality,
λ min ( Σ ^ S ) ≥ λ min ( Σ S ) − ∥ Σ S − Σ ^ S ∥ o p ≥ λ min ( Σ S ) − ∑ e ω e ∥ Σ S e − Σ ^ S e ∥ o p ≥ λ min ( Σ S ) − s n log p + log ∣ E ∣ min e ∈ E n e ≥ κ L − s n log p + log ∣ E ∣ min e ∈ E n e . \begin{aligned}
\lambda_{\min}(\widehat{\Sigma}_S) \ge\lambda_{\min}(\Sigma_S) - \left\|\Sigma_S - \widehat{\Sigma}_S\right\|_{\mathrm{op}} \ge\lambda_{\min}(\Sigma_S) - \sum_e \omega^e \left\|\Sigma_S^e - \widehat{\Sigma}_S^e\right\|_{\mathrm{op}} \\
&\ge\lambda_{\min}(\Sigma_S) - \sqrt{\frac{s_n \log p + \log|\mathcal{E}|}{\min_{e \in\mathcal{E}} n^e}} \ge\kappa_L - \sqrt{\frac{s_n \log p + \log|\mathcal{E}|}{\min_{e \in\mathcal{E}} n^e}}.
\end{aligned} λ m i n ( Σ S ) ≥ λ m i n ( Σ S ) − Σ S − Σ S op ≥ λ m i n ( Σ S ) − e ∑ ω e Σ S e − Σ S e op ≥ λ m i n ( Σ S ) − min e ∈ E n e s n log p + log ∣ E ∣ ≥ κ L − min e ∈ E n e s n log p + log ∣ E ∣ .
When min e ∈ E n e ≳ s n log p + log ∣ E ∣ \min_{e \in\mathcal{E}} n^e \gtrsim s_n \log p + \log|\mathcal{E}| min e ∈ E n e ≳ s n log p + log ∣ E ∣ , we obtain λ min ( Σ ^ S ) > 0 \lambda_{\min}(\widehat{\Sigma}_S) > 0 λ m i n ( Σ S ) > 0 for all S S S . As a result,
( ∑ e ω e X S ⋆ e T X S ⋆ e n e ) ≻ 0 , \left(\sum_e \omega^e \frac{X_{S^\star}^{e\mathsf{T}}X_{S^\star}^e}{n^e}\right) \succ0, ( e ∑ ω e n e X S ⋆ e T X S ⋆ e ) ≻ 0 ,
and R ^ ( β ˉ ) − R ^ ( β ⋆ ) < 0 \widehat{R}(\bar{\beta}) - \widehat{R}(\beta^\star) < 0 R ( β ˉ ) − R ( β ⋆ ) < 0 when min e ∈ E n e \min_{e \in\mathcal{E}} n^e min e ∈ E n e is sufficiently large. We set the strict upper bound of min β ∈ R 1 F ^ ( β ) − F ^ ( β ⋆ ) \min_{\beta\in\mathcal{R}_1} \widehat{F}(\beta) - \widehat{F}(\beta^\star) min β ∈ R 1 F ( β ) − F ( β ⋆ ) as u = − b T A − 1 b / 2 < 0 u = -b^\mathsf{T}A^{-1}b/2 < 0 u = − b T A − 1 b /2 < 0 .
Step II. In this step, we overload the notation β ˉ \bar{\beta} β ˉ to denote the solution to min β ∈ R 2 { F ^ ( β ) − F ^ ( β ⋆ ) } \min_{\beta\in\mathcal{R}_2} \{\widehat{F}(\beta) - \widehat{F}(\beta^\star)\} min β ∈ R 2 { F ( β ) − F ( β ⋆ )} . We also denote S ˉ : = supp ( β ˉ ) \bar{S} := \operatorname{supp}(\bar{\beta}) S ˉ := supp ( β ˉ ) . Recall from Lemma 13 that for any feasible β \beta β ,
F ^ ( β ) − F ^ ( β ⋆ ) ≳ ∥ β − β ⋆ ∥ 2 2 − s n log p + log ∣ E ∣ min e ∈ E n e + λ n ( ∣ S ˉ ∣ − s ⋆ ) . \widehat{F}(\beta) - \widehat{F}(\beta^\star) \gtrsim\|\beta- \beta^\star\|_2^2 - \frac{s_n \log p + \log|\mathcal{E}|}{\min_{e \in\mathcal{E}} n^e} + \lambda_n(|\bar{S}| - s^\star). F ( β ) − F ( β ⋆ ) ≳ ∥ β − β ⋆ ∥ 2 2 − min e ∈ E n e s n log p + log ∣ E ∣ + λ n ( ∣ S ˉ ∣ − s ⋆ ) .
Now we attempt to connect this lower bound with the number of missed causal parents η 1 : = ∣ S ⋆ ∖ S ˉ ∣ \eta_1 := |S^\star\setminus\bar{S}| η 1 := ∣ S ⋆ ∖ S ˉ ∣ . We also define η 2 : = ∣ S ˉ ∖ S ⋆ ∣ \eta_2 := |\bar{S} \setminus S^\star| η 2 := ∣ S ˉ ∖ S ⋆ ∣ . For any β ˉ ∈ R 2 \bar{\beta} \in\mathcal{R}_2 β ˉ ∈ R 2 , we consider two cases.
_ Case 1:_ If ∣ S ˉ ∣ = s ⋆ |\bar{S}| = s^\star ∣ S ˉ ∣ = s ⋆ , then η 1 = η 2 ≥ 1 \eta_1 = \eta_2 \ge1 η 1 = η 2 ≥ 1 . As a result,
∥ β ˉ − β ⋆ ∥ 2 2 ≥ η 1 min j ∈ S ⋆ ∣ β j ⋆ ∣ 2 ≥ min j ∈ S ⋆ ∣ β j ⋆ ∣ 2 , \|\bar{\beta} - \beta^\star\|_2^2 \ge\eta_1 \min_{j \in S^\star} |\beta_j^\star|^2 \ge\min_{j \in S^\star} |\beta_j^\star|^2, ∥ β ˉ − β ⋆ ∥ 2 2 ≥ η 1 j ∈ S ⋆ min ∣ β j ⋆ ∣ 2 ≥ j ∈ S ⋆ min ∣ β j ⋆ ∣ 2 ,
and consequently,
F ^ ( β ˉ ) − F ^ ( β ⋆ ) ≳ min j ∈ S ⋆ ∣ β j ⋆ ∣ 2 − s n log p + log ∣ E ∣ min e ∈ E n e \widehat{F}(\bar{\beta})-\widehat{F}(\beta^\star)\gtrsim\min_{j\in S^\star}|\beta_j^\star|^2-\frac{s_n\log p+\log|\mathcal{E}|}{\min_{e\in\mathcal{E}}n^e} F ( β ˉ ) − F ( β ⋆ ) ≳ j ∈ S ⋆ min ∣ β j ⋆ ∣ 2 − min e ∈ E n e s n log p + log ∣ E ∣
We call this lower bound l 1 ( 1 ) l_1^{(1)} l 1 ( 1 ) . Appealing to the beta-min condition (11), we have l 1 ( 1 ) ≥ 0 > u l_1^{(1)}\ge0>u l 1 ( 1 ) ≥ 0 > u when min e ∈ E n e \min_{e\in\mathcal{E}}n^e min e ∈ E n e is sufficiently large.
Case 2: If ∣ S ˉ ∣ < s ⋆ |\bar{S}|<s^\star ∣ S ˉ ∣ < s ⋆ , then η 2 ≥ 0 \eta_2\ge0 η 2 ≥ 0 and η 1 = s ⋆ − ∣ S ˉ ∣ + η 2 \eta_1=s^\star-|\bar{S}|+\eta_2 η 1 = s ⋆ − ∣ S ˉ ∣ + η 2 . As a result,
∥ β ˉ − β ⋆ ∥ 2 2 ≥ η 1 min j ∈ S ⋆ ∣ β j ⋆ ∣ 2 ≥ ( s ⋆ − ∣ S ˉ ∣ ) min j ∈ S ⋆ ∣ β j ⋆ ∣ 2 , \|\bar{\beta}-\beta^\star\|_2^2\ge\eta_1\min_{j\in S^\star}|\beta_j^\star|^2\ge(s^\star-|\bar{S}|)\min_{j\in S^\star}|\beta_j^\star|^2, ∥ β ˉ − β ⋆ ∥ 2 2 ≥ η 1 j ∈ S ⋆ min ∣ β j ⋆ ∣ 2 ≥ ( s ⋆ − ∣ S ˉ ∣ ) j ∈ S ⋆ min ∣ β j ⋆ ∣ 2 ,
and consequently,
F ^ ( β ˉ ) − F ^ ( β ⋆ ) ≳ ( s ⋆ − ∣ S ˉ ∣ ) min j ∈ S ⋆ ∣ β j ⋆ ∣ 2 − s n log p + log ∣ E ∣ min e ∈ E n e + λ n ( ∣ S ˉ ∣ − s ⋆ ) ≥ ( s ⋆ − ∣ S ˉ ∣ ) ( min j ∈ S ⋆ ∣ β j ⋆ ∣ 2 − λ n − s n log p + log ∣ E ∣ min e ∈ E n e ) ≥ min j ∈ S ⋆ ∣ β j ⋆ ∣ 2 − λ n − s n log p + log ∣ E ∣ min e ∈ E n e . \begin{aligned}
\widehat{F}(\bar{\beta})-\widehat{F}(\beta^\star)&\gtrsim(s^\star-|\bar{S}|)\min_{j\in S^\star}|\beta_j^\star|^2-\frac{s_n\log p+\log|\mathcal{E}|}{\min_{e\in\mathcal{E}}n^e}+\lambda_n(|\bar{S}|-s^\star)\\
&\ge(s^\star-|\bar{S}|)\left(\min_{j\in S^\star}|\beta_j^\star|^2-\lambda_n-\frac{s_n\log p+\log|\mathcal{E}|}{\min_{e\in\mathcal{E}}n^e}\right)\\
&\ge\min_{j\in S^\star}|\beta_j^\star|^2-\lambda_n-\frac{s_n\log p+\log|\mathcal{E}|}{\min_{e\in\mathcal{E}}n^e}.
\end{aligned} F ( β ˉ ) − F ( β ⋆ ) ≳ ( s ⋆ − ∣ S ˉ ∣ ) j ∈ S ⋆ min ∣ β j ⋆ ∣ 2 − min e ∈ E n e s n log p + log ∣ E ∣ + λ n ( ∣ S ˉ ∣ − s ⋆ ) ≥ ( s ⋆ − ∣ S ˉ ∣ ) ( j ∈ S ⋆ min ∣ β j ⋆ ∣ 2 − λ n − min e ∈ E n e s n log p + log ∣ E ∣ ) ≥ j ∈ S ⋆ min ∣ β j ⋆ ∣ 2 − λ n − min e ∈ E n e s n log p + log ∣ E ∣ .
We call this lower bound l 1 ( 2 ) l_1^{(2)} l 1 ( 2 ) . Again, appealing to the beta-min condition (11), we have l 1 ( 2 ) ≥ 0 > u l_1^{(2)}\ge0>u l 1 ( 2 ) ≥ 0 > u when min e ∈ E n e \min_{e\in\mathcal{E}}n^e min e ∈ E n e is sufficiently large.
Step III. In this step, we overload the notation β ˉ \bar{\beta} β ˉ again to denote the solution to the optimization arg min β ∈ R 3 { F ^ ( β ) − F ^ ( β ⋆ ) } \mathop{\arg\min}_{\beta\in\mathbb{R}^3}\{\widehat{F}(\beta)-\widehat{F}(\beta^\star)\} arg min β ∈ R 3 { F ( β ) − F ( β ⋆ )} . We also overload S ˉ : = supp ( β ˉ ) \bar{S}:=\operatorname{supp}(\bar{\beta}) S ˉ := supp ( β ˉ ) . Similarly as in Step II, we have
F ^ ( β ˉ ) − F ^ ( β ⋆ ) ≳ ∥ β ˉ − β ⋆ ∥ 2 2 − s n log p + log ∣ E ∣ min e ∈ E n e + λ n ( ∣ S ˉ ∣ − s ⋆ ) ≥ λ n − s n log p + log ∣ E ∣ min e ∈ E n e . \widehat{F}(\bar{\beta})-\widehat{F}(\beta^\star)\gtrsim\|\bar{\beta}-\beta^\star\|_2^2-\frac{s_n\log p+\log|\mathcal{E}|}{\min_{e\in\mathcal{E}}n^e}+\lambda_n(|\bar{S}|-s^\star)\ge\lambda_n-\frac{s_n\log p+\log|\mathcal{E}|}{\min_{e\in\mathcal{E}}n^e}. F ( β ˉ ) − F ( β ⋆ ) ≳ ∥ β ˉ − β ⋆ ∥ 2 2 − min e ∈ E n e s n log p + log ∣ E ∣ + λ n ( ∣ S ˉ ∣ − s ⋆ ) ≥ λ n − min e ∈ E n e s n log p + log ∣ E ∣ .
We call this lower bound l 2 l_2 l 2 . With the choice λ n ≳ ( s n log p + log ∣ E ∣ ) / min e ∈ E n e \lambda_n\gtrsim(s_n\log p+\log|\mathcal{E}|)/\min_{e\in\mathcal{E}}n^e λ n ≳ ( s n log p + log ∣ E ∣ ) / min e ∈ E n e and when min e ∈ E n e \min_{e\in\mathcal{E}}n^e min e ∈ E n e is sufficiently large, we have l 2 ≥ 0 > u l_2\ge0>u l 2 ≥ 0 > u .
Proof of part (2)
We use the same proof technique of part (1). From the basic inequality,
F ^ ( β ^ e a r l y ) − F ^ ( β ⋆ ) ≤ F ^ ( β ^ ) − F ^ ( β ⋆ ) + τ e a r l y . \widehat{F}(\widehat{\beta}^{\mathrm{early}})-\widehat{F}(\beta^\star)\le\widehat{F}(\widehat{\beta})-\widehat{F}(\beta^\star)+\tau^{\mathrm{early}}. F ( β early ) − F ( β ⋆ ) ≤ F ( β ) − F ( β ⋆ ) + τ early .
Consequently, the upper bound of Step I becomes u = − b ⊤ A − 1 b / 2 + τ e a r l y u=-b^\top A^{-1}b/2+\tau^{\mathrm{early}} u = − b ⊤ A − 1 b /2 + τ early . When τ e a r l y ≲ ( s n log p + log ∣ E ∣ ) / min e ∈ E n e \tau^{\mathrm{early}}\lesssim(s_n\log p+\log|\mathcal{E}|)/\min_{e\in\mathcal{E}}n^e τ early ≲ ( s n log p + log ∣ E ∣ ) / min e ∈ E n e , we still have l 1 ( 1 ) > u l_1^{(1)}>u l 1 ( 1 ) > u , l 1 ( 2 ) > u l_1^{(2)}>u l 1 ( 2 ) > u , and l 2 > u l_2>u l 2 > u when min e ∈ E n e \min_{e\in\mathcal{E}}n^e min e ∈ E n e is sufficiently large. The result hence follows.
Proofs of results in Section 5
In this section, we present the proofs of results in Section 5 and Appendix B. Without loss of generality, we focus on the case of two environments, E = { e , f } \mathcal{E}=\{e,f\} E = { e , f } . We rewrite γ 1 = γ \gamma_1=\gamma γ 1 = γ and γ 2 = γ ρ \gamma_2=\gamma\rho γ 2 = γ ρ in this section. We denote
U n ( k ) : = min S ∉ S k n ⋆ { γ 1 ∥ β ^ e , S − β ^ f , S ∥ 2 2 + γ 2 ∥ ν ^ e , S − ν ^ f , S ∥ 2 2 } , U_n(k):=\min_{S\notin\mathcal{S}_k}n^\star\left\{\gamma_1\left\|\widehat{\beta}^{e,S}-\widehat{\beta}^{f,S}\right\|_2^2+\gamma_2\left\|\widehat{\nu}^{e,S}-\widehat{\nu}^{f,S}\right\|_2^2\right\}, U n ( k ) := S ∈ / S k min n ⋆ { γ 1 β e , S − β f , S 2 2 + γ 2 ν e , S − ν f , S 2 2 } ,
T n ( k ) : = max S ∈ S k n ⋆ { γ 1 ∥ β ^ e , S − β ^ f , S ∥ 2 2 + γ 2 ∥ ν ^ e , S − ν ^ f , S ∥ 2 2 } , T_n(k):=\max_{S\in\mathcal{S}_k}n^\star\left\{\gamma_1\left\|\widehat{\beta}^{e,S}-\widehat{\beta}^{f,S}\right\|_2^2+\gamma_2\left\|\widehat{\nu}^{e,S}-\widehat{\nu}^{f,S}\right\|_2^2\right\}, T n ( k ) := S ∈ S k max n ⋆ { γ 1 β e , S − β f , S 2 2 + γ 2 ν e , S − ν f , S 2 2 } ,
and
T n ∘ ( k ) : = max S ∈ S k ∘ n ⋆ { γ 1 ∥ β ^ e , S − β ^ f , S ∥ 2 2 + γ 2 ∥ ν ^ e , S − ν ^ f , S ∥ 2 2 } . T_n^\circ(k):=\max_{S\in\mathcal{S}_k^\circ}n^\star\left\{\gamma_1\left\|\widehat{\beta}^{e,S}-\widehat{\beta}^{f,S}\right\|_2^2+\gamma_2\left\|\widehat{\nu}^{e,S}-\widehat{\nu}^{f,S}\right\|_2^2\right\}. T n ∘ ( k ) := S ∈ S k ∘ max n ⋆ { γ 1 β e , S − β f , S 2 2 + γ 2 ν e , S − ν f , S 2 2 } .
We also adopt the notation introduced in Appendix E.
Proof of Lemma 5
Proof. We suppress the parentheses involving k k k in the notation whenever there is no ambiguity. It suffices to show that P H 0 , k ( T n < U n ) → 1 \mathbb{P}_{H_{0,k}}(T_n < U_n) \to1 P H 0 , k ( T n < U n ) → 1 , since it implies that
1 − o ( 1 ) ≤ P H 0 , k ( T n < U n ) ≤ P H 0 , k ( { S k ⊆ S k ∘ } ∩ { S ^ ∈ S k } ) ≤ 1. 1-o(1) \le\mathbb{P}_{H_{0,k}}(T_n < U_n) \le\mathbb{P}_{H_{0,k}}\left(\{S_k \subseteq S_k^\circ\} \cap\{\widehat{S} \in S_k\}\right) \le1. 1 − o ( 1 ) ≤ P H 0 , k ( T n < U n ) ≤ P H 0 , k ( { S k ⊆ S k ∘ } ∩ { S ∈ S k } ) ≤ 1.
Now consider the working model Y e ∣ X S e ∼ N ( ( X S e ) ⊤ β S e , S , ν e , S ) Y^e \mid X_S^e \sim\mathcal{N}((X_S^e)^\top\beta_S^{e,S},\nu^{e,S}) Y e ∣ X S e ∼ N (( X S e ) ⊤ β S e , S , ν e , S ) , where β S e , S \beta_S^{e,S} β S e , S is the population OLS solution of ( X S e , Y e ) (X_S^e,Y^e) ( X S e , Y e ) . Let r e , S = Y e − ( X S e ) ⊤ β S e , S r^{e,S}=Y^e-(X_S^e)^\top\beta_S^{e,S} r e , S = Y e − ( X S e ) ⊤ β S e , S , we have
E [ ( X S e ) ⊤ r e , S ] = 0 , E [ ( r e , S ) 2 ] = ν e , S , ∀ e ∈ E . \mathbb{E}\left[(X_S^e)^\top r^{e,S}\right]=0,\qquad\mathbb{E}\left[(r^{e,S})^2\right]=\nu^{e,S},\qquad\forall e\in\mathcal{E}. E [ ( X S e ) ⊤ r e , S ] = 0 , E [ ( r e , S ) 2 ] = ν e , S , ∀ e ∈ E .
As a result, for the sample OLS solutions,
β ^ S e , S − β ^ S f , S = β S e , S − β S f , S + ( ( X S e ) ⊤ X S e n e ) − 1 ( ( X S e ) ⊤ r e , S n e ) − ( ( X S f ) ⊤ X S f n f ) − 1 ( ( X S f ) ⊤ r f , S n f ) = β S e , S − β S f , S + o p ( 1 ) , \widehat{\beta}_S^{e,S}-\widehat{\beta}_S^{f,S}
=
\beta_S^{e,S}-\beta_S^{f,S}
+
\left(\frac{(X_S^e)^\top X_S^e}{n^e}\right)^{-1}
\left(\frac{(X_S^e)^\top r^{e,S}}{n^e}\right)
-
\left(\frac{(X_S^f)^\top X_S^f}{n^f}\right)^{-1}
\left(\frac{(X_S^f)^\top r^{f,S}}{n^f}\right)
=
\beta_S^{e,S}-\beta_S^{f,S}+o_p(1), β S e , S − β S f , S = β S e , S − β S f , S + ( n e ( X S e ) ⊤ X S e ) − 1 ( n e ( X S e ) ⊤ r e , S ) − ( n f ( X S f ) ⊤ X S f ) − 1 ( n f ( X S f ) ⊤ r f , S ) = β S e , S − β S f , S + o p ( 1 ) ,
and
ν ^ e , S − ν ^ f , S = 1 n e ∑ i = 1 n e ( Y i e − ( β ^ S e , S ) ⊤ X S , i e ) 2 − 1 n f ∑ i = 1 n f ( Y i f − ( β ^ S f , S ) ⊤ X S , i f ) 2 = ν e , S − ν f , S + o p ( 1 ) . \widehat{\nu}^{e,S}-\widehat{\nu}^{f,S}
=
\frac{1}{n^e}\sum_{i=1}^{n^e}\left(Y_i^e-(\widehat{\beta}_S^{e,S})^\top X_{S,i}^e\right)^2
-
\frac{1}{n^f}\sum_{i=1}^{n^f}\left(Y_i^f-(\widehat{\beta}_S^{f,S})^\top X_{S,i}^f\right)^2
=
\nu^{e,S}-\nu^{f,S}+o_p(1). ν e , S − ν f , S = n e 1 i = 1 ∑ n e ( Y i e − ( β S e , S ) ⊤ X S , i e ) 2 − n f 1 i = 1 ∑ n f ( Y i f − ( β S f , S ) ⊤ X S , i f ) 2 = ν e , S − ν f , S + o p ( 1 ) .
Therefore, by continuous mapping theorem, we have
T n / n ⋆ = max S ∈ S k [ γ 1 ∥ β S e , S − β S f , S ∥ 2 2 + γ 2 ∣ ν e , S − ν f , S ∣ 2 ] + o p ( 1 ) T_n/n^\star
=
\max_{S\in S_k}
\left[
\gamma_1\left\|\beta_S^{e,S}-\beta_S^{f,S}\right\|_2^2
+
\gamma_2\left|\nu^{e,S}-\nu^{f,S}\right|^2
\right]
+o_p(1) T n / n ⋆ = S ∈ S k max [ γ 1 β S e , S − β S f , S 2 2 + γ 2 ν e , S − ν f , S 2 ] + o p ( 1 )
and
U n / n ⋆ = min S ∉ S k [ γ 1 ∥ β S e , S − β S f , S ∥ 2 2 + γ 2 ∣ ν e , S − ν f , S ∣ 2 ] + o p ( 1 ) . U_n/n^\star
=
\min_{S\notin S_k}
\left[
\gamma_1\left\|\beta_S^{e,S}-\beta_S^{f,S}\right\|_2^2
+
\gamma_2\left|\nu^{e,S}-\nu^{f,S}\right|^2
\right]
+o_p(1). U n / n ⋆ = S ∈ / S k min [ γ 1 β S e , S − β S f , S 2 2 + γ 2 ν e , S − ν f , S 2 ] + o p ( 1 ) .
For any S ∈ S k S\in S_k S ∈ S k , we have β S e , S = β S f , S \beta_S^{e,S}=\beta_S^{f,S} β S e , S = β S f , S and ν e , S = ν f , S \nu^{e,S}=\nu^{f,S} ν e , S = ν f , S . However, for any S ∉ S k S\notin S_k S ∈ / S k , either β S e , S ≠ β S f , S \beta_S^{e,S}\ne\beta_S^{f,S} β S e , S = β S f , S or ν e , S ≠ ν f , S \nu^{e,S}\ne\nu^{f,S} ν e , S = ν f , S . These imply that
( U n − T n ) / n ⋆ = C + o p ( 1 ) (U_n-T_n)/n^\star=C+o_p(1) ( U n − T n ) / n ⋆ = C + o p ( 1 )
for some constant C > 0 C>0 C > 0 , and the result follows.
In this section, we present several intermediate results that will be used later to prove Theorem 4. Recall that we focus on the case of two environments, E = { e , f } \mathcal{E}=\{e,f\} E = { e , f } . We further denote the random variables constructed from bootstrap (line 8 in Algorithm 1) by
T n ⋆ ( k ) : = max S ∈ S k { γ 1 ∥ L n ⋆ , e , f , S ∥ 2 2 + γ 2 ∣ V n ⋆ , e , f , S ∣ 2 } , T_n^\star(k):=\max_{S\in S_k}\left\{\gamma_1\left\|L_n^{\star,e,f,S}\right\|_2^2+\gamma_2\left|V_n^{\star,e,f,S}\right|^2\right\}, T n ⋆ ( k ) := S ∈ S k max { γ 1 L n ⋆ , e , f , S 2 2 + γ 2 V n ⋆ , e , f , S 2 } ,
and
T n ∘ , ⋆ ( k ) : = max S ∈ S k ∘ { γ 1 ∥ L n ⋆ , e , f , S ∥ 2 2 + γ 2 ∣ V n ⋆ , e , f , S ∣ 2 } . T_n^{\circ,\star}(k):=\max_{S\in S_k^\circ}\left\{\gamma_1\left\|L_n^{\star,e,f,S}\right\|_2^2+\gamma_2\left|V_n^{\star,e,f,S}\right|^2\right\}. T n ∘ , ⋆ ( k ) := S ∈ S k ∘ max { γ 1 L n ⋆ , e , f , S 2 2 + γ 2 V n ⋆ , e , f , S 2 } .
For simplicity, we suppress the parentheses in the notation whenever there is no ambiguity. Note that for every bootstrap realization, we have T n ⋆ ≤ T n ∘ , ⋆ T_n^\star\le T_n^{\circ,\star} T n ⋆ ≤ T n ∘ , ⋆ . We also define
F n , k ( t ) : = P H 0 , k ( T n ≤ t ) , F n , k ⋆ ( t ) : = P H 0 , k ( T n ⋆ ≤ t ∣ D 2 ) , G n , k ⋆ ( t ) : = P H 0 , k ( T n ∘ , ⋆ ≤ t ∣ D 2 ) . F_{n,k}(t):=\mathbb{P}_{H_{0,k}}(T_n\le t),\qquad
F_{n,k}^\star(t):=\mathbb{P}_{H_{0,k}}(T_n^\star\le t\mid D_2),\qquad
G_{n,k}^\star(t):=\mathbb{P}_{H_{0,k}}(T_n^{\circ,\star}\le t\mid D_2). F n , k ( t ) := P H 0 , k ( T n ≤ t ) , F n , k ⋆ ( t ) := P H 0 , k ( T n ⋆ ≤ t ∣ D 2 ) , G n , k ⋆ ( t ) := P H 0 , k ( T n ∘ , ⋆ ≤ t ∣ D 2 ) .
With B n B_n B n i.i.d. bootstrap draws { T n , b ∘ , ⋆ } b = 1 B n \{T_{n,b}^{\circ,\star}\}_{b=1}^{B_n} { T n , b ∘ , ⋆ } b = 1 B n from the bootstrap distribution T n ∘ , ⋆ T_n^{\circ,\star} T n ∘ , ⋆ , we define the empirical bootstrap CDF
G ^ n , k ⋆ ( t ) : = 1 B n ∑ b = 1 B n 1 ( T n , b ∘ , ⋆ ≤ t ) . \widehat{G}_{n,k}^\star(t):=\frac{1}{B_n}\sum_{b=1}^{B_n}\mathbf{1}\left(T_{n,b}^{\circ,\star}\le t\right). G n , k ⋆ ( t ) := B n 1 b = 1 ∑ B n 1 ( T n , b ∘ , ⋆ ≤ t ) .
Lemma 14. Suppose that n e / n ⋆ → ρ e > 0 n^{e}/n^\star\to\rho_e > 0 n e / n ⋆ → ρ e > 0 and n f / n ⋆ → ρ f > 0 n^{f}/n^\star\to\rho_f > 0 n f / n ⋆ → ρ f > 0 . Then the bootstrap distribution F n , k ⋆ F^\star_{n,k} F n , k ⋆ consistently approximates F n , k F_{n,k} F n , k :
sup t ∣ F n , k ( t ) − F n , k ⋆ ( t ) ∣ = o p ( 1 ) . \sup_t \left|F_{n,k}(t)-F^\star_{n,k}(t)\right|=o_p(1). t sup F n , k ( t ) − F n , k ⋆ ( t ) = o p ( 1 ) .
Proof. For notational simplicity, we present the proof for S k = { S , S ˉ } S_k=\{S,\bar{S}\} S k = { S , S ˉ } ; the same argument applies to any fixed finite S k S_k S k by stacking the corresponding components. For each A ∈ S k A\in S_k A ∈ S k , define
L n A : = n ⋆ ( β ^ e , A − β ^ f , A ) , V n A : = n ⋆ ( ν ^ e , A − ν ^ f , A ) . L_n^A:=\sqrt{n^\star}\left(\widehat{\beta}^{e,A}-\widehat{\beta}^{f,A}\right),\qquad V_n^A:=\sqrt{n^\star}\left(\widehat{\nu}^{e,A}-\widehat{\nu}^{f,A}\right). L n A := n ⋆ ( β e , A − β f , A ) , V n A := n ⋆ ( ν e , A − ν f , A ) .
Under H 0 , k H_{0,k} H 0 , k , both S S S and S ˉ \bar{S} S ˉ are invariant. By the asymptotic linearity of the OLS coefficient and residual variance, we have for any A ∈ S k A\in S_k A ∈ S k ,
L n A = n ⋆ n e ( X A e ⊤ X A e n e ) − 1 X A e ⊤ r e , A n e − n ⋆ n f ( X A f ⊤ X A f n f ) − 1 X A f ⊤ r f , A n f , L_n^A=\sqrt{\frac{n^\star}{n^e}}\left(\frac{X_A^{e\top}X_A^e}{n^e}\right)^{-1}\frac{X_A^{e\top}r^{e,A}}{\sqrt{n^e}}-\sqrt{\frac{n^\star}{n^f}}\left(\frac{X_A^{f\top}X_A^f}{n^f}\right)^{-1}\frac{X_A^{f\top}r^{f,A}}{\sqrt{n^f}}, L n A = n e n ⋆ ( n e X A e ⊤ X A e ) − 1 n e X A e ⊤ r e , A − n f n ⋆ ( n f X A f ⊤ X A f ) − 1 n f X A f ⊤ r f , A ,
and
V n A = n ⋆ n e 1 n e ∑ i = 1 n e ( r i e , A ) 2 − n ⋆ n f 1 n f ∑ i = 1 n f ( r i f , A ) 2 + o p ( 1 ) , V_n^A=\sqrt{\frac{n^\star}{n^e}}\frac{1}{\sqrt{n^e}}\sum_{i=1}^{n^e}(r_i^{e,A})^2-\sqrt{\frac{n^\star}{n^f}}\frac{1}{\sqrt{n^f}}\sum_{i=1}^{n^f}(r_i^{f,A})^2+o_p(1), V n A = n e n ⋆ n e 1 i = 1 ∑ n e ( r i e , A ) 2 − n f n ⋆ n f 1 i = 1 ∑ n f ( r i f , A ) 2 + o p ( 1 ) ,
where r e , A = Y e − X A e β ^ e , A r^{e,A}=Y^e-X_A^e\widehat{\beta}^{e,A} r e , A = Y e − X A e β e , A . Now stack these quantities as W n = [ L n S , L n S ˉ , V n S , V n S ˉ ] ⊤ W_n=[L_n^S,L_n^{\bar{S}},V_n^S,V_n^{\bar{S}}]^\top W n = [ L n S , L n S ˉ , V n S , V n S ˉ ] ⊤ ; then we obtain
W n = n ⋆ n e 1 n e ∑ i = 1 n e ψ i e − n ⋆ n f 1 n f ∑ i = 1 n f ψ i f + o p ( 1 ) , W_n=\sqrt{\frac{n^\star}{n^e}}\frac{1}{\sqrt{n^e}}\sum_{i=1}^{n^e}\psi_i^e-\sqrt{\frac{n^\star}{n^f}}\frac{1}{\sqrt{n^f}}\sum_{i=1}^{n^f}\psi_i^f+o_p(1), W n = n e n ⋆ n e 1 i = 1 ∑ n e ψ i e − n f n ⋆ n f 1 i = 1 ∑ n f ψ i f + o p ( 1 ) ,
where g ∈ { e , f } g\in\{e,f\} g ∈ { e , f } ,
ψ i g : = [ ( Σ S g ) − 1 X S , i g ⋅ r i g , S ( Σ S ˉ g ) − 1 X S ˉ , i g ⋅ r i g , S ˉ ( r i g , S ) 2 − ν g , S ( r i g , S ˉ ) 2 − ν g , S ˉ ] , and E [ ψ i g ] = 0. \psi_i^g:=\begin{bmatrix}
(\Sigma_S^g)^{-1}X_{S,i}^g\cdot r_i^{g,S}\\
(\Sigma_{\bar{S}}^g)^{-1}X_{\bar{S},i}^g\cdot r_i^{g,\bar{S}}\\
(r_i^{g,S})^2-\nu^{g,S}\\
(r_i^{g,\bar{S}})^2-\nu^{g,\bar{S}}
\end{bmatrix},
\qquad\text{and}\qquad\mathbb{E}[\psi_i^g]=0. ψ i g := ( Σ S g ) − 1 X S , i g ⋅ r i g , S ( Σ S ˉ g ) − 1 X S ˉ , i g ⋅ r i g , S ˉ ( r i g , S ) 2 − ν g , S ( r i g , S ˉ ) 2 − ν g , S ˉ , and E [ ψ i g ] = 0.
Since n e / n ⋆ → ρ e > 0 n^e/n^\star\to\rho_e>0 n e / n ⋆ → ρ e > 0 and n f / n ⋆ → ρ f > 0 n^f/n^\star\to\rho_f>0 n f / n ⋆ → ρ f > 0 , the multivariate central limit theorem and Slutsky’s theorem imply
W n ⇝ W ∼ N ( 0 , Γ ) , where Γ = 1 ρ e Var ( ψ i e ) + 1 ρ f Var ( ψ i f ) . W_n\rightsquigarrow W\sim N(0,\Gamma),\qquad\text{where}\qquad\Gamma=\frac{1}{\rho_e}\operatorname{Var}(\psi_i^e)+\frac{1}{\rho_f}\operatorname{Var}(\psi_i^f). W n ⇝ W ∼ N ( 0 , Γ ) , where Γ = ρ e 1 Var ( ψ i e ) + ρ f 1 Var ( ψ i f ) .
Now denote the corresponding bootstrap vector by W n ⋆ = [ L n ⋆ , S , L n ⋆ , S ˉ , V n ⋆ , S , V n ⋆ , S ˉ ] ⊤ W_n^\star=[L_n^{\star,S},L_n^{\star,\bar{S}},V_n^{\star,S},V_n^{\star,\bar{S}}]^\top W n ⋆ = [ L n ⋆ , S , L n ⋆ , S ˉ , V n ⋆ , S , V n ⋆ , S ˉ ] ⊤ . Note that
W n ⋆ = n ⋆ n e 1 n e ∑ i = 1 n e ξ i e ψ ^ i e − n ⋆ n f 1 n f ∑ i = 1 n f ξ i f ψ ^ i f , where ψ ^ i g : = [ ( Σ ^ S g ) − 1 X S , i g ⋅ r ^ i g , S ( Σ ^ S ˉ g ) − 1 X S ˉ , i g ⋅ r ^ i g , S ˉ ( r ^ i g , S ) 2 − ν ^ g , S ( r ^ i g , S ˉ ) 2 − ν ^ g , S ˉ ] . W_n^\star=\sqrt{\frac{n^\star}{n^e}}\frac{1}{\sqrt{n^e}}\sum_{i=1}^{n^e}\xi_i^e\widehat{\psi}_i^e-\sqrt{\frac{n^\star}{n^f}}\frac{1}{\sqrt{n^f}}\sum_{i=1}^{n^f}\xi_i^f\widehat{\psi}_i^f,
\qquad\text{where}\qquad
\widehat{\psi}_i^g:=\begin{bmatrix}
(\widehat{\Sigma}_S^g)^{-1}X_{S,i}^g\cdot\widehat{r}_i^{g,S}\\
(\widehat{\Sigma}_{\bar{S}}^g)^{-1}X_{\bar{S},i}^g\cdot\widehat{r}_i^{g,\bar{S}}\\
(\widehat{r}_i^{g,S})^2-\widehat{\nu}^{g,S}\\
(\widehat{r}_i^{g,\bar{S}})^2-\widehat{\nu}^{g,\bar{S}}
\end{bmatrix}. W n ⋆ = n e n ⋆ n e 1 i = 1 ∑ n e ξ i e ψ i e − n f n ⋆ n f 1 i = 1 ∑ n f ξ i f ψ i f , where ψ i g := ( Σ S g ) − 1 X S , i g ⋅ r i g , S ( Σ S ˉ g ) − 1 X S ˉ , i g ⋅ r i g , S ˉ ( r i g , S ) 2 − ν g , S ( r i g , S ˉ ) 2 − ν g , S ˉ .
Conditional on D 2 \mathcal{D}_2 D 2 , we have W n ⋆ ∣ D 2 ∼ N ( 0 , Γ ^ n ) W_n^\star\mid\mathcal{D}_2\sim N(0,\widehat{\Gamma}_n) W n ⋆ ∣ D 2 ∼ N ( 0 , Γ n ) , where
Γ ^ n = n ⋆ ( n e ) 2 ∑ i = 1 n e ψ ^ i e ψ ^ i e ⊤ + n ⋆ ( n f ) 2 ∑ i = 1 n f ψ ^ i f ψ ^ i f ⊤ , \widehat{\Gamma}_n=\frac{n^\star}{(n^e)^2}\sum_{i=1}^{n^e}\widehat{\psi}_i^e\widehat{\psi}_i^{e\top}+\frac{n^\star}{(n^f)^2}\sum_{i=1}^{n^f}\widehat{\psi}_i^f\widehat{\psi}_i^{f\top}, Γ n = ( n e ) 2 n ⋆ i = 1 ∑ n e ψ i e ψ i e ⊤ + ( n f ) 2 n ⋆ i = 1 ∑ n f ψ i f ψ i f ⊤ ,
since E [ ξ i g ξ j g ] = 1 ( i = j ) \mathbb{E}[\xi_i^g\xi_j^g]=\mathbb{1}(i=j) E [ ξ i g ξ j g ] = 1 ( i = j ) for all g ∈ { e , f } g\in\{e,f\} g ∈ { e , f } . Furthermore, by OLS consistency, the law of large numbers, and Slutsky’s theorem, we have
Γ ^ n = 1 ρ e 1 n e ∑ i = 1 n e ψ ^ i e ψ ^ i e ⊤ + 1 ρ f 1 n f ∑ i = 1 n f ψ ^ i f ψ ^ i f ⊤ + o p ( 1 ) = 1 ρ e E [ ψ i e ψ i e ⊤ ] + 1 ρ f E [ ψ i f ψ i f ⊤ ] + o p ( 1 ) = Γ + o p ( 1 ) , \begin{aligned}
\widehat{\Gamma}_n=\frac{1}{\rho_e}\frac{1}{n^e}\sum_{i=1}^{n^e}\widehat{\psi}_i^e\widehat{\psi}_i^{e\top}+\frac{1}{\rho_f}\frac{1}{n^f}\sum_{i=1}^{n^f}\widehat{\psi}_i^f\widehat{\psi}_i^{f\top}+o_p(1) \\
&=\frac{1}{\rho_e}\mathbb{E}\left[\psi_i^e\psi_i^{e\top}\right]+\frac{1}{\rho_f}\mathbb{E}\left[\psi_i^f\psi_i^{f\top}\right]+o_p(1)=\Gamma+o_p(1),
\end{aligned} Γ n = ρ e 1 n e 1 i = 1 ∑ n e ψ i e ψ i e ⊤ + ρ f 1 n f 1 i = 1 ∑ n f ψ i f ψ i f ⊤ + o p ( 1 ) = ρ e 1 E [ ψ i e ψ i e ⊤ ] + ρ f 1 E [ ψ i f ψ i f ⊤ ] + o p ( 1 ) = Γ + o p ( 1 ) ,
Finally, define the continuous function
g ( w ) : = max A ∈ S k { γ 1 ∥ w L A ∥ 2 2 + γ 2 ∣ w V A ∣ 2 } . g(w) := \max_{A \in\mathcal{S}_k} \left\{ \gamma_1 \lVert w_L^A \rVert_2^2 + \gamma_2 \lvert w_V^A \rvert^2 \right\}. g ( w ) := A ∈ S k max { γ 1 ∥ w L A ∥ 2 2 + γ 2 ∣ w V A ∣ 2 } .
Then T n = g ( W n ) T_n = g(W_n) T n = g ( W n ) and T n ⋆ = g ( W n ⋆ ) T_n^\star= g(W_n^{\star}) T n ⋆ = g ( W n ⋆ ) . By the continuous mapping theorem, we have T n ⇝ g ( W ) T_n \rightsquigarrow g(W) T n ⇝ g ( W ) ; conditional on D 2 \mathcal{D}_2 D 2 , we have T n ⋆ ⇝ g ( W ) T_n^\star\rightsquigarrow g(W) T n ⋆ ⇝ g ( W ) in probability. Provided that the distribution function F ( t ) F(t) F ( t ) of g ( W ) g(W) g ( W ) is continuous, Pólya’s theorem yields
sup t ∣ F n , k ( t ) − F ( t ) ∣ → 0 , sup t ∣ F n , k ⋆ ( t ) − F ( t ) ∣ = o p ( 1 ) . \sup_t \lvert F_{n,k}(t) - F(t) \rvert\to0,\qquad\sup_t \lvert F_{n,k}^\star(t) - F(t) \rvert= o_p(1). t sup ∣ F n , k ( t ) − F ( t )∣ → 0 , t sup ∣ F n , k ⋆ ( t ) − F ( t )∣ = o p ( 1 ) .
As a result,
sup t ∣ F n , k ( t ) − F n , k ⋆ ( t ) ∣ ≤ sup t ∣ F n , k ( t ) − F ( t ) ∣ + sup t ∣ F n , k ⋆ ( t ) − F ( t ) ∣ = o p ( 1 ) . \sup_t \lvert F_{n,k}(t) - F_{n,k}^\star(t) \rvert\le\sup_t \lvert F_{n,k}(t) - F(t) \rvert+ \sup_t \lvert F_{n,k}^\star(t) - F(t) \rvert= o_p(1). t sup ∣ F n , k ( t ) − F n , k ⋆ ( t )∣ ≤ t sup ∣ F n , k ( t ) − F ( t )∣ + t sup ∣ F n , k ⋆ ( t ) − F ( t )∣ = o p ( 1 ) .
Lemma 15. The empirical bootstrap distribution G ^ n , k ⋆ \widehat{G}_{n,k}^\star G n , k ⋆ uniformly approximates G n , k ⋆ G_{n,k}^\star G n , k ⋆ as the bootstrap sample size B n → ∞ B_n \to\infty B n → ∞ :
sup t ∣ G ^ n , k ⋆ ( t ) − G n , k ⋆ ( t ) ∣ = o p ( 1 ) . \sup_t \lvert\widehat{G}_{n,k}^\star(t) - G_{n,k}^\star(t) \rvert= o_p(1). t sup ∣ G n , k ⋆ ( t ) − G n , k ⋆ ( t )∣ = o p ( 1 ) .
Proof. By the Dvoretzky–Kiefer–Wolfowitz inequality, for every ϵ > 0 \epsilon> 0 ϵ > 0 ,
P ( sup t ∣ G ^ n , k ⋆ ( t ) − G n , k ⋆ ( t ) ∣ > ϵ ∣ D ) ≤ 2 exp ( − 2 B n ϵ 2 ) . \mathbb{P}\left(\sup_t \lvert\widehat{G}_{n,k}^\star(t) - G_{n,k}^\star(t) \rvert> \epsilon\mid\mathcal{D}\right) \le2\exp(-2B_n\epsilon^2). P ( t sup ∣ G n , k ⋆ ( t ) − G n , k ⋆ ( t )∣ > ϵ ∣ D ) ≤ 2 exp ( − 2 B n ϵ 2 ) .
The result hence follows.
Proof of Theorem 4
Proof. The proof proceeds in two steps. In the first step, we show the type-I error control for each hypothesis H 0 , k H_{0,k} H 0 , k . In the second step, we show the asymptotic probabilistic guarantee for { S ^ ( E ) ⊆ S ⋆ } \{\widehat{S}(\mathcal{E}) \subseteq S^\star\} { S ( E ) ⊆ S ⋆ } .
Step I. Define the sample quantile
t ^ n , k : = inf { t : G ^ n , k ⋆ ( t ) ≥ 1 − α / p } , \widehat{t}_{n,k} := \inf\left\{t : \widehat{G}_{n,k}^\star(t) \ge1 - \alpha/p\right\}, t n , k := inf { t : G n , k ⋆ ( t ) ≥ 1 − α / p } ,
and the distribution discrepancy
Δ 1 , n , k : = sup t ∣ F n , k ( t ) − F n , k ⋆ ( t ) ∣ , Δ 2 , n , k : = sup t ∣ G ^ n , k ⋆ ( t ) − G n , k ⋆ ( t ) ∣ . \Delta_{1,n,k} := \sup_t \lvert F_{n,k}(t) - F_{n,k}^\star(t) \rvert,\qquad\Delta_{2,n,k} := \sup_t \lvert\widehat{G}_{n,k}^\star(t) - G_{n,k}^\star(t) \rvert. Δ 1 , n , k := t sup ∣ F n , k ( t ) − F n , k ⋆ ( t )∣ , Δ 2 , n , k := t sup ∣ G n , k ⋆ ( t ) − G n , k ⋆ ( t )∣ .
By Lemma 14 and 15, we know Δ 1 , n , k + Δ 2 , n , k = o p ( 1 ) \Delta_{1,n,k} + \Delta_{2,n,k} = o_p(1) Δ 1 , n , k + Δ 2 , n , k = o p ( 1 ) , and hence we can choose a deterministic sequence ϵ n ↓ 0 \epsilon_n \downarrow0 ϵ n ↓ 0 such that P H 0 , k ( Δ 1 , n , k + Δ 2 , n , k > ϵ n ) = o ( 1 ) \mathbb{P}_{H_{0,k}}(\Delta_{1,n,k} + \Delta_{2,n,k} > \epsilon_n) = o(1) P H 0 , k ( Δ 1 , n , k + Δ 2 , n , k > ϵ n ) = o ( 1 ) . Now consider the population quantile
q n , k : = F n , k − 1 ( 1 − α / p − ϵ n ) . q_{n,k} := F_{n,k}^{-1}(1 - \alpha/p - \epsilon_n). q n , k := F n , k − 1 ( 1 − α / p − ϵ n ) .
On the event A n , k : = { Δ 1 , n , k + Δ 2 , n , k ≤ ϵ n } A_{n,k} := \{\Delta_{1,n,k} + \Delta_{2,n,k} \le\epsilon_n\} A n , k := { Δ 1 , n , k + Δ 2 , n , k ≤ ϵ n } , for all t < q n , k t < q_{n,k} t < q n , k , we have F n , k ( t ) < 1 − α / p − ϵ n F_{n,k}(t) < 1 - \alpha/p - \epsilon_n F n , k ( t ) < 1 − α / p − ϵ n , and hence
G ^ n , k ⋆ ( t ) ≤ G n , k ⋆ ( t ) + Δ 2 , n , k ≤ F n , k ⋆ ( t ) + Δ 2 , n , k ≤ F n , k ( t ) + Δ 1 , n , k + Δ 2 , n , k < 1 − α / p . \widehat{G}_{n,k}^\star(t) \le G_{n,k}^\star(t) + \Delta_{2,n,k} \le F_{n,k}^\star(t) + \Delta_{2,n,k} \le F_{n,k}(t) + \Delta_{1,n,k} + \Delta_{2,n,k} < 1 - \alpha/p. G n , k ⋆ ( t ) ≤ G n , k ⋆ ( t ) + Δ 2 , n , k ≤ F n , k ⋆ ( t ) + Δ 2 , n , k ≤ F n , k ( t ) + Δ 1 , n , k + Δ 2 , n , k < 1 − α / p .
The second inequality follows from the fact that T n ⋆ ≤ T n ∘ , ⋆ T_n^\star\le T_n^{\circ,\star} T n ⋆ ≤ T n ∘ , ⋆ . This implies that the CDF G ^ n , k ⋆ \widehat{G}_{n,k}^\star G n , k ⋆ cannot reach 1 − α / p 1 - \alpha/p 1 − α / p before q n , k q_{n,k} q n , k , and consequently q n , k ≤ t ^ n , k q_{n,k} \le\widehat{t}_{n,k} q n , k ≤ t n , k on the event A n , k A_{n,k} A n , k . As a result, we have
P H 0 , k ( T n > t ^ n , k ) ≤ P H 0 , k ( { T n > t ^ n , k } ∩ { A n , k } ) + P H 0 , k ( { A n , k } c ) ≤ P H 0 , k ( T n > q n , k ) + P H 0 , k ( Δ 1 , n , k + Δ 2 , n , k > ϵ n ) ≤ α / p + o ( 1 ) . \begin{aligned}
\mathbb{P}_{H_{0,k}}(T_n > \widehat{t}_{n,k})
&\le\mathbb{P}_{H_{0,k}}\left(\{T_n > \widehat{t}_{n,k}\} \cap\{A_{n,k}\}\right) + \mathbb{P}_{H_{0,k}}(\{A_{n,k}\}^{c}) \\
&\le\mathbb{P}_{H_{0,k}}(T_n > q_{n,k}) + \mathbb{P}_{H_{0,k}}(\Delta_{1,n,k} + \Delta_{2,n,k} > \epsilon_n) \\
&\le\alpha/p + o(1).
\end{aligned} P H 0 , k ( T n > t n , k ) ≤ P H 0 , k ( { T n > t n , k } ∩ { A n , k } ) + P H 0 , k ({ A n , k } c ) ≤ P H 0 , k ( T n > q n , k ) + P H 0 , k ( Δ 1 , n , k + Δ 2 , n , k > ϵ n ) ≤ α / p + o ( 1 ) .
Finally, by the definition of T n ( k ) T_n(k) T n ( k ) and the assumption that S ^ ∈ S k \widehat{S} \in\mathcal{S}_k S ∈ S k , we have T ^ ( k ) ≤ T n ( k ) \widehat{T}(k) \le T_n(k) T ( k ) ≤ T n ( k ) . Furthermore, t ^ n , k ∘ \widehat{t}_{n,k}^{\circ} t n , k ∘ coincides with the bootstrap threshold τ ^ k ( α ) \widehat{\tau}_k(\alpha) τ k ( α ) defined in line 10 of Algorithm 1. As a result,
P H 0 , k ( T ^ ( k ) > τ ^ k ( α ) ) ≤ P H 0 , k ( T n ( k ) > t ^ n , k ∘ ) ≤ α / p + o ( 1 ) . \mathbb{P}_{H_{0,k}}\left(\widehat{T}(k) > \widehat{\tau}_k(\alpha)\right) \le\mathbb{P}_{H_{0,k}}\left(T_n(k) > \widehat{t}_{n,k}^{\circ}\right) \le\alpha/p + o(1). P H 0 , k ( T ( k ) > τ k ( α ) ) ≤ P H 0 , k ( T n ( k ) > t n , k ∘ ) ≤ α / p + o ( 1 ) .
Step II. The result follows immediately that
P ( S ^ ( E ) ⊈ S ⋆ ) ≤ ∑ k = 1 p P H 0 , k ( T ^ ( k ) > τ ^ k ( α ) ) ≤ p ⋅ α p + o ( 1 ) = α + o ( 1 ) . \mathbb{P}\left(\widehat{S}(\mathcal{E}) \not\subseteq S^\star\right) \le\sum_{k=1}^{p}\mathbb{P}_{H_{0,k}}\left(\widehat{T}(k) > \widehat{\tau}_k(\alpha)\right) \le p \cdot\frac{\alpha}{p} + o(1) = \alpha+ o(1). P ( S ( E ) ⊆ S ⋆ ) ≤ k = 1 ∑ p P H 0 , k ( T ( k ) > τ k ( α ) ) ≤ p ⋅ p α + o ( 1 ) = α + o ( 1 ) .
Appendix G. Details on experiments
We first describe the synthetic data generation and intervention mechanisms we used. We generate data from linear Gaussian structural equation models over p p p covariates X = ( X 1 , … , X p ) X=(X_1,\ldots,X_p) X = ( X 1 , … , X p ) and a response Y Y Y . We first sample a common topological ordering of the p + 1 p+1 p + 1 variables and place Y Y Y at a prespecified position in this ordering. The causal parent set S ⋆ S^\star S ⋆ of Y Y Y is sampled from the variables preceding Y Y Y , with ∣ S ⋆ ∣ \lvert S^\star\rvert ∣ S ⋆ ∣ drawn between s min s_{\min} s m i n and s max s_{\max} s m a x . For j ∈ S ⋆ j \in S^\star j ∈ S ⋆ , the invariant coefficients are sampled as
β j ⋆ = ξ j U j , ξ j ∈ { − 1 , 1 } , U j ∼ Unif ( l ⋆ , u ⋆ ) , \beta_j^\star= \xi_j U_j,\qquad\xi_j \in\{-1,1\},\qquad U_j \sim\operatorname{Unif}(l_\star,u_\star), β j ⋆ = ξ j U j , ξ j ∈ { − 1 , 1 } , U j ∼ Unif ( l ⋆ , u ⋆ ) ,
independently. Thus, in every environment,
Y = ∑ j ∈ S ⋆ β j ⋆ X j + ϵ Y , ϵ Y ∼ N ( 0 , σ Y 2 ) , Y = \sum_{j\in S^\star} \beta_j^\star X_j + \epsilon_Y,\qquad\epsilon_Y \sim N(0,\sigma_Y^2), Y = j ∈ S ⋆ ∑ β j ⋆ X j + ϵ Y , ϵ Y ∼ N ( 0 , σ Y 2 ) ,
For the remaining structural equations, coefficients are sampled independently as a random sign multiplied by a uniform magnitude on [ l , u ] [l,u] [ l , u ] . The parameter p a c t p_{\mathrm{act}} p act controls the sparsity of the covariate DAG: for each pair of variables consistent with the topological ordering, an edge is included with probability p a c t p_{\mathrm{act}} p act . Conditional on the resulting structural coefficients, observations are generated recursively according to the common topological ordering.
Under the soft-intervention mechanism , the DAG structure is sampled once and shared across all environments. We first generate a baseline coefficient matrix A ( 0 ) A^{(0)} A ( 0 ) . The observational environment uses this baseline mechanism. For each subsequent environment e > 0 e>0 e > 0 , every active edge into a covariate X j X_j X j is independently selected with probability p r e s a m p l e p_{\mathrm{resample}} p resample ; selected coefficients are then redrawn from the same signed uniform distribution, while the edge support remains unchanged. The noise standard deviations of the covariates are sampled once from Unif ( σ X , min , σ X , max ) \operatorname{Unif}(\sigma_{X,\min},\sigma_{X,\max}) Unif ( σ X , m i n , σ X , m a x ) and are also shared across environments. Hence, environments differ through perturbations of existing structural coefficients while preserving both the DAG support and the noise distributions.
Under the all-intervention mechanism , the topological ordering and the response mechanism are still shared across environments, but the mechanisms generating covariates are resampled independently in every environment. Specifically, for each environment and for every admissible pair of covariates, an edge is included independently with probability p a c t p_{\mathrm{act}} p act , and its coefficient is sampled afresh from the signed uniform distribution. In addition, under the all-intervention mechanism, the covariate noise standard deviations are independently resampled in each environment from σ X j ( e ) ∼ Unif ( σ X , min , σ X , max ) \sigma_{X_j}^{(e)} \sim\operatorname{Unif}(\sigma_{X,\min},\sigma_{X,\max}) σ X j ( e ) ∼ Unif ( σ X , m i n , σ X , m a x ) , whereas σ Y \sigma_Y σ Y remains fixed.
We generate data using the soft-intervention mechanism described above. We vary the ambient dimension over p ∈ { 10 , 20 , 30 , 40 , 50 } p \in\{10, 20, 30, 40, 50\} p ∈ { 10 , 20 , 30 , 40 , 50 } and generate 3 environments with n e = 200 n^{e} = 200 n e = 200 observations per environment. For each value of p p p , we conduct 100 independent replications. The SEM is generated with p a c t ∈ { 0.1 , 0.2 } p_{\mathrm{act}} \in\{0.1, 0.2\} p act ∈ { 0.1 , 0.2 } , ℓ = u = 0.1 \ell= u = 0.1 ℓ = u = 0.1 , ℓ ⋆ = 0.5 \ell_{\star} = 0.5 ℓ ⋆ = 0.5 , u ⋆ = 1 u_{\star} = 1 u ⋆ = 1 , σ X , min = 0.5 \sigma_{X,\min} = 0.5 σ X , m i n = 0.5 , σ X , max = 1 \sigma_{X,\max} = 1 σ X , m a x = 1 , σ Y = 0.5 \sigma_Y = 0.5 σ Y = 0.5 , s min = 4 s_{\min} = 4 s m i n = 4 , s max = 5 s_{\max} = 5 s m a x = 5 , and p r e s a m p l e = 0.05 p_{\mathrm{resample}} = 0.05 p resample = 0.05 .
We compare the heuristic coordinate descent (CD) with the vanilla MIP formulation, both with and without a warm start. All three methods use the same objective parameters λ = 0.01 \lambda= 0.01 λ = 0.01 , γ 1 = 5 \gamma_1 = 5 γ 1 = 5 , and ρ = 1 \rho= 1 ρ = 1 . For coordinate descent, we initialize with the empty set. The MIP with a warm start is initialized using the support returned by coordinate descent, whereas the vanilla MIP is initialized from the empty set. For both MIP variants, we set MIPFocus=2 and Cuts=2, and impose a time limit of 180 seconds.
We use the data generation mechanism described in Example 3. To avoid relying on the exact parameter configuration ( a 1 , a 2 ) = ( 1 , 1 / 2 ) (a_1, a_2) = (1, 1/2) ( a 1 , a 2 ) = ( 1 , 1/2 ) and ( v 1 , v 2 ) = ( 2 , 3 / 2 ) (v_1, v_2) = (2, 3/2) ( v 1 , v 2 ) = ( 2 , 3/2 ) , which occurs with probability zero under a continuous parameter distribution, we independently perturb these parameters for each generated dataset according to
a e = ( a e ) 0 + δ a , e , v e = ( v e ) 0 + δ v , e , a_e = (a_e)_0 + \delta_{a,e}, \qquad v_e = (v_e)_0 + \delta_{v,e}, a e = ( a e ) 0 + δ a , e , v e = ( v e ) 0 + δ v , e ,
where δ a , e , δ v , e ∼ i . i . d . N ( 0 , 0.05 2 ) \delta_{a,e}, \delta_{v,e} \overset{\mathrm{i.i.d.}}{\sim} \mathcal{N}(0, 0.05^2) δ a , e , δ v , e ∼ i.i.d. N ( 0 , 0.0 5 2 ) , with ( a 1 ) 0 = 1 , ( a 2 ) 0 = 1 / 2 (a_1)_0 = 1, (a_2)_0 = 1/2 ( a 1 ) 0 = 1 , ( a 2 ) 0 = 1/2 and ( v 1 ) 0 = 2 , ( v 2 ) 0 = 3 / 2 (v_1)_0 = 2, (v_2)_0 = 3/2 ( v 1 ) 0 = 2 , ( v 2 ) 0 = 3/2 .
We compare MIP with EILLS, ICP, Causal Dantzig, and BIP. We describe the implementation details:
MIP: we set λ n = 0.5 \lambda_n = 0.5 λ n = 0.5 , γ 1 = 50 \gamma_1 = 50 γ 1 = 50 , ρ = 1 \rho= 1 ρ = 1 , and s max = 10 s_{\max} = 10 s m a x = 10 ; we use MIPFocus=3 and impose a time limit of 60 seconds.
EILLS: we set its penalty parameter to γ = 50 \gamma= 50 γ = 50 .
ICP: we use significance level α = 0.05 \alpha= 0.05 α = 0.05 .
Causal Dantzig: we use the hidden variable implementation with α = 0.1 \alpha= 0.1 α = 0.1 .
BIP: we restrict the candidate models to size at most 10, use ridge stabilization ϵ = 10 − 4 \epsilon= 10^{-4} ϵ = 1 0 − 4 , and report the maximum a posteriori model. No intercept is included for any of these methods.
Left and middle panels. We generate data using the soft-intervention mechanism described above with ambient dimension p = 20 p = 20 p = 20 . We vary the number of environments over ∣ E ∣ ∈ { 2 , 3 , 5 , 10 } |\mathcal{E}| \in\{2, 3, 5, 10\} ∣ E ∣ ∈ { 2 , 3 , 5 , 10 } and the sample size per environment over n e ∈ { 50 , 75 , … , 275 } n^e \in\{50, 75, \ldots, 275\} n e ∈ { 50 , 75 , … , 275 } . For each combination of ( ∣ E ∣ , n e ) (|\mathcal{E}|, n^e) ( ∣ E ∣ , n e ) , we conduct 100 independent replications. The SEM is generated with p a c t ∈ { 0.1 , 0.2 } p_{\mathrm{act}} \in\{0.1, 0.2\} p act ∈ { 0.1 , 0.2 } , ℓ = u = 0.1 \ell= u = 0.1 ℓ = u = 0.1 , ℓ ⋆ = 0.5 \ell_{\star} = 0.5 ℓ ⋆ = 0.5 , u ⋆ = 1 u_{\star} = 1 u ⋆ = 1 , σ X , min = 0.5 \sigma_{X,\min} = 0.5 σ X , m i n = 0.5 , σ X , max = 1 \sigma_{X,\max} = 1 σ X , m a x = 1 , σ Y = 0.5 \sigma_Y = 0.5 σ Y = 0.5 , s min = 4 s_{\min} = 4 s m i n = 4 , s max = 5 s_{\max} = 5 s m a x = 5 , and p r e s a m p l e = 0.05 p_{\mathrm{resample}} = 0.05 p resample = 0.05 . We apply the coordinate descent implementation of the invariant estimator with λ n = 0.01 \lambda_n = 0.01 λ n = 0.01 , γ 1 = 5 \gamma_1 = 5 γ 1 = 5 , and ρ = 1 \rho= 1 ρ = 1 . Coordinate descent is initialized from the empty set.
Right panel. We use the same soft-intervention mechanism and DGP parameters as above, fixing p = 20 p = 20 p = 20 , ∣ E ∣ = 10 |\mathcal{E}| = 10 ∣ E ∣ = 10 , and n e = 150 n^e = 150 n e = 150 . We vary the tuning parameters over γ ∈ { 1 , 5 , 10 , 15 , 20 } \gamma\in\{1, 5, 10, 15, 20\} γ ∈ { 1 , 5 , 10 , 15 , 20 } and λ n ∈ { 10 − 4 , 10 − 3 , 10 − 2 , 10 − 1 , 1 } \lambda_n \in\{10^{-4}, 10^{-3}, 10^{-2}, 10^{-1}, 1\} λ n ∈ { 1 0 − 4 , 1 0 − 3 , 1 0 − 2 , 1 0 − 1 , 1 } , while fixing ρ = 1 \rho= 1 ρ = 1 . For each choice of ( γ , λ n ) (\gamma, \lambda_n) ( γ , λ n ) , we conduct 50 independent replications. We compute the leave-one-environment-out (LOEO) validation criterion and apply coordinate descent with the corresponding tuning parameters, initialized from the empty set. Each cell in the heatmap reports the average LOEO criterion across replications, while red outlines indicate parameter combinations for which the proportion of perfect recovery of S ⋆ S^{\star} S ⋆ exceeds 0.95.
Figure 7. Performance comparison between our method MIP-CD and the baseline methods as the dimension p p p increases. Left panel: average coefficient estimation error ∥ β ^ − β ∗ ∥ 2 \lVert\hat{\beta}-\beta^*\rVert_2 ∥ β ^ − β ∗ ∥ 2 over 200 trials. Middle panel: proportion of 200 trials achieving perfect recovery of S ∗ S^* S ∗ . Right panel: average runtime in seconds over 200 trials, shown on a logarithmic scale. Solid lines show averages and shaded bands indicate ± 1 \pm1 ± 1 standard error. Additional experimental details are provided in Appendix G.4.
We generate data using the all-intervention mechanism described above. We vary the ambient dimension over p ∈ { 5 , 10 , … , 90 , 100 } p \in\{5,10,\ldots,90,100\} p ∈ { 5 , 10 , … , 90 , 100 } and generate 10 environments with n e = 500 n^{e}=500 n e = 500 observations per environment. The number of causal parents is set to ∣ S ⋆ ∣ = ⌊ 0.4 p ⌋ |S^{\star}|=\lfloor0.4p\rfloor ∣ S ⋆ ∣ = ⌊ 0.4 p ⌋ when ⌊ 0.4 p ⌋ < 16 \lfloor0.4p\rfloor<16 ⌊ 0.4 p ⌋ < 16 , and ∣ S ⋆ ∣ = 16 + ⌊ 0.1 p ⌋ |S^{\star}|=16+\lfloor0.1p\rfloor ∣ S ⋆ ∣ = 16 + ⌊ 0.1 p ⌋ otherwise. For each value of p p p , we conduct 200 independent replications. The SEM is generated with p a c t ∈ { 0.1 , 0.15 } p_{\mathrm{act}}\in\{0.1,0.15\} p act ∈ { 0.1 , 0.15 } , ℓ = 0.2 \ell=0.2 ℓ = 0.2 , u = 0.5 u=0.5 u = 0.5 , ℓ ⋆ = u ⋆ = 0.5 \ell_{\star}=u_{\star}=0.5 ℓ ⋆ = u ⋆ = 0.5 , σ X , min = 0.5 \sigma_{X,\min}=0.5 σ X , m i n = 0.5 , σ X , max = 1 \sigma_{X,\max}=1 σ X , m a x = 1 , and σ Y = 0.5 \sigma_Y=0.5 σ Y = 0.5 .
We compare our proposed method, MIP-CD , with EILLS-s , ICP-s , Causal Dantzig , BIP-s , NegDRO , CoCo , and IRM . For EILLS-s , ICP-s , and BIP-s , we first screen the covariates using the magnitudes of ridge regression coefficients with regularization parameter λ = 1 \lambda=1 λ = 1 , keeping the top 15 variables for EILLS-s and the top 10 for ICP-s and BIP-s . We describe the implementation details below:
MIP-CD : we set ρ = 1 \rho=1 ρ = 1 , while λ n ∈ { 10 − 4 , 10 − 3 , 10 − 2 , 10 − 1 , 1 } \lambda_n\in\{10^{-4},10^{-3},10^{-2},10^{-1},1\} λ n ∈ { 1 0 − 4 , 1 0 − 3 , 1 0 − 2 , 1 0 − 1 , 1 } and γ ∈ { 1 , 5 , 10 , 15 , 20 } \gamma\in\{1,5,10,15,20\} γ ∈ { 1 , 5 , 10 , 15 , 20 } are chosen based on LOEO criterion. Coordinate descent is initialized from the empty set.
EILLS-s : we set its penalty parameter to γ = 20 \gamma=20 γ = 20 .
ICP-s : we use significance level α = 0.05 \alpha=0.05 α = 0.05 .
Causal Dantzig : we use the hidden variable implementation with α = 0.1 \alpha=0.1 α = 0.1 and exclude the intercept.
BIP-s : we enumerate candidate models over the 10 screened variables, use ridge stabilization ϵ = 10 − 4 \epsilon=10^{-4} ϵ = 1 0 − 4 , and report the maximum a posteriori model without an intercept.
NegDRO : we set its penalty parameter to γ = 20 \gamma=20 γ = 20 .
CoCo : we set the regularization parameter to λ = 10 \lambda=10 λ = 10 , learning rate to 0.01, and maximum number of iterations to 10,000, using equal environment weights and excluding the intercept.
IRM : we set the penalty weight to 10, penalty annealing iterations to 100, ℓ 2 \ell_2 ℓ 2 regularization parameter to 10 − 3 10^{-3} 1 0 − 3 , learning rate to 10 − 3 10^{-3} 1 0 − 3 , and number of optimization steps to 500.
We use the structural equation model described in Example 1, with parameters γ 3 = γ 4 = β 1 ⋆ = β 3 ⋆ = 1 \gamma_3=\gamma_4=\beta_1^{\star}=\beta_3^{\star}=1 γ 3 = γ 4 = β 1 ⋆ = β 3 ⋆ = 1 , η 3 = η 4 = σ = 1 \eta_3=\eta_4=\sigma=1 η 3 = η 4 = σ = 1 , and intervention strength a = 3 a=3 a = 3 . We additionally generate nine independent standard Gaussian covariates, yielding an ambient dimension of p = 13 p=13 p = 13 . For each environment, we generate two independent datasets with sample sizes n 1 e = 300 n_1^{e}=300 n 1 e = 300 and n 2 e ∈ { 400 , 800 , … , 10000 } n_2^{e}\in\{400,800,\ldots,10000\} n 2 e ∈ { 400 , 800 , … , 10000 } . For each value of n 2 e n_2^{e} n 2 e , we conduct 200 independent replications. The true causal parent set is S ⋆ = { 1 , 3 } S^{\star}=\{1,3\} S ⋆ = { 1 , 3 } , while the plausible causal predictor set is S ( E ) = { 1 } S(\mathcal{E})=\{1\} S ( E ) = { 1 } .
We compare MIP-test-2 and MIP-test-4 with ICP and WVM , as well as MIP , EILLS , Causal Dantzig , and BIP-s . For MIP-test-2 and MIP-test-4 , we use the first dataset to construct candidate supports and the second dataset for hypothesis testing. For the remaining methods, we combine the two datasets within each environment. We describe the implementation details below:
MIP-test-2 and MIP-test-4 : we set α = 0.05 \alpha=0.05 α = 0.05 , M ‾ = 5 \overline{M}=5 M = 5 , and use B = 300 B=300 B = 300 multiplier bootstrap replications. We set ρ = 1 \rho=1 ρ = 1 for MIP-test-2 and ( γ 1 , γ 2 , γ 3 , γ 4 ) = ( 1 , 1 , 1 , 1 ) (\gamma_1,\gamma_2,\gamma_3,\gamma_4)=(1,1,1,1) ( γ 1 , γ 2 , γ 3 , γ 4 ) = ( 1 , 1 , 1 , 1 ) for MIP-test-4 . Both methods use coordinate descent to solve the underlying optimization problems, with candidate supports obtained through ridge screening.
ICP : we use significance level α = 0.05 \alpha=0.05 α = 0.05 .
WVM : we use significance level α = 0.05 \alpha=0.05 α = 0.05 and B = 300 B=300 B = 300 bootstrap replications.
MIP : we use coordinate descent with λ n = 0.01 \lambda_n=0.01 λ n = 0.01 , γ 1 = 5 \gamma_1=5 γ 1 = 5 , and ρ = 1 \rho=1 ρ = 1 , initialized from the empty set.
EILLS : we set its penalty parameter to γ = 20 \gamma=20 γ = 20 and exclude the intercept.
Causal Dantzig : we use the hidden variable implementation with α = 0.05 \alpha=0.05 α = 0.05 and exclude the intercept.
BIP-s : we first keep the top 10 variables using ridge screening, enumerate candidate subsets of these variables, and report the maximum a posteriori model. We use ridge stabilization ϵ = 10 − 4 \epsilon=10^{-4} ϵ = 1 0 − 4 and exclude the intercept.
We generate data using the all-intervention mechanism described above. We fix the ambient dimension at p = 10 p = 10 p = 10 and the sample size at n e = 500 n^{e} = 500 n e = 500 per environment, while varying the number of environments. For each setting, we conduct 200 independent replications. The SEM is generated with P a c t ∈ { 0.1 , 0.15 } \mathcal{P}_{\mathrm{act}} \in\{0.1, 0.15\} P act ∈ { 0.1 , 0.15 } , ℓ = 0.2 \ell= 0.2 ℓ = 0.2 , u = 0.5 u = 0.5 u = 0.5 , ℓ ⋆ = u ⋆ = 0.5 \ell^{\star} = u^{\star} = 0.5 ℓ ⋆ = u ⋆ = 0.5 , σ X , min = 0.5 \sigma_{X,\min} = 0.5 σ X , m i n = 0.5 , σ X , max = 1 \sigma_{X,\max} = 1 σ X , m a x = 1 , σ Y = 0.5 \sigma_Y = 0.5 σ Y = 0.5 , s min = 4 s_{\min} = 4 s m i n = 4 , and s max = 5 s_{\max} = 5 s m a x = 5 .
We compare MIP-test-2 and MIP-test-4 with ICP and WVM . For both MIP procedures, we randomly split the observations in each environment, using 20% to construct candidate predictor sets and the remaining 80% for hypothesis testing. For ICP and WVM , we use the full dataset. We describe the implementation details below:
MIP-test-2 and MIP-test-4 : we set α = 0.05 \alpha= 0.05 α = 0.05 , M ‾ = 3 \overline{M} = 3 M = 3 , and use B = 500 B = 500 B = 500 multiplier bootstrap replications. We set ρ = 1 \rho= 1 ρ = 1 for MIP-test-2 and ( γ 1 , γ 2 , γ 3 , γ 4 ) = ( 1 , 1 , 1 , 1 ) (\gamma_1, \gamma_2, \gamma_3, \gamma_4) = (1,1,1,1) ( γ 1 , γ 2 , γ 3 , γ 4 ) = ( 1 , 1 , 1 , 1 ) for MIP-test-4 . Both methods use coordinate descent to solve the underlying optimization problems.
ICP : We use significance level α = 0.05 \alpha= 0.05 α = 0.05 .
WVM : We use significance level α = 0.05 \alpha= 0.05 α = 0.05 and B = 300 B = 300 B = 300 bootstrap replications.
Experiment of real data: the causal chambers in Section 7
We consider the light tunnel dataset from Causal Chambers ([6 ] ), following the causal discovery setup of Gu et al. (2025). The response Y = I ~ 3 Y = \widetilde{I}_3 Y = I 3 represents the infrared intensity measured after two polarizers. We consider 11 observed covariates: the RGB intensities ( R , G , B ) (R, G, B) ( R , G , B ) of the light source, the two polarizer angles ( θ 1 , θ 2 ) (\theta_1, \theta_2) ( θ 1 , θ 2 ) , three measurements ( V ~ 1 , V ~ 2 , V ~ 3 ) (\widetilde{V}_1, \widetilde{V}_2, \widetilde{V}_3) ( V 1 , V 2 , V 3 ) of visible light intensity, two additional measurements ( I ~ 1 , I ~ 2 ) (\widetilde{I}_1, \widetilde{I}_2) ( I 1 , I 2 ) of infrared intensity, and the measured electrical current C ~ \widetilde{C} C . The known causal parent set is S ⋆ = { R , G , B , θ 1 , θ 2 } S^{\star} = \{R, G, B, \theta_1, \theta_2\} S ⋆ = { R , G , B , θ 1 , θ 2 } . See Figure 13 for the full ground truth.
Figure 13. Ground truth of the causal structure in the light tunnel chamber, adapted from Gu et al. (2025). The response is Y = I ~ 3 Y = \widetilde{I}_3 Y = I 3 , with causal parent set S ⋆ = { R , G , B , θ 1 , θ 2 } S^{\star} = \{R, G, B, \theta_1, \theta_2\} S ⋆ = { R , G , B , θ 1 , θ 2 } .
We consider one reference environment uniform_reference, and 19 intervention environments corresponding to different manipulations of the physical system. We vary the number of environments over ∣ E ∣ ∈ { 3 , 4 , … , 10 } |E| \in\{3, 4, \ldots, 10\} ∣ E ∣ ∈ { 3 , 4 , … , 10 } , with n e = 1000 n^{e} = 1000 n e = 1000 observations per environment. Specifically, we conduct 100 independent trials. In each trial, we randomly sample 1000 observations from each environment and generate a random permutation of the 19 intervention environments. We then construct nested environment sets by always including the reference environment and sequentially adding intervention environments according to the random ordering. For example, when ∣ E ∣ = 3 |E| = 3 ∣ E ∣ = 3 , we include the reference environment and the first two randomly selected intervention environments. When ∣ E ∣ = 4 |E| = 4 ∣ E ∣ = 4 , we keep these three environments and add the next intervention environment. This procedure continues until ∣ E ∣ = 10 |E| = 10 ∣ E ∣ = 10 , and the selected environments and their corresponding observations remain unchanged within each trial as additional environments are introduced. Finally, all covariates and the response are standardized using the empirical means and standard deviations computed from the reference environment, with the same standardization applied to all intervention environments.
We compare our proposed methods, MIP-CD and MIP-test , with EILLS , ICP , Causal Dantzig , BIP , and FAIR . We describe the implementation details below:
MIP-CD : we set λ n = 0.01 \lambda_n = 0.01 λ n = 0.01 , γ 1 = 5 \gamma_1 = 5 γ 1 = 5 , and ρ = 1 \rho= 1 ρ = 1 . Coordinate descent is initialized from the empty set.
MIP-test : we first use 20% of the observations in each environment to obtain a candidate support using MIP-CD , with the same tuning parameters as above. We then restrict the subsequent variable selection to this candidate support, allowing MIP-test to serve as a diagnostic procedure for MIP-CD . The remaining observations are further divided into two subsets, containing 20% and 80% of the remaining data, respectively, for constructing S k S^k S k and conducting hypothesis tests. We use MIP-test-4 with ( γ 1 , γ 2 , γ 3 , γ 4 ) = ( 1 , 1 , 1 , 1 ) (\gamma_1, \gamma_2, \gamma_3, \gamma_4) = (1, 1, 1, 1) ( γ 1 , γ 2 , γ 3 , γ 4 ) = ( 1 , 1 , 1 , 1 ) , significance level α = 0.05 \alpha= 0.05 α = 0.05 , M ‾ = 5 \overline{M} = 5 M = 5 , and B = 500 B = 500 B = 500 multiplier bootstrap replications. The underlying optimization problems are solved using coordinate descent.
EILLS : we set its penalty parameter to γ = 20 \gamma= 20 γ = 20 and exclude the intercept.
ICP : we use significance level α = 0.05 \alpha= 0.05 α = 0.05 and report the intersection of all accepted predictor sets.
Causal Dantzig : we use the hidden variable implementation with α = 0.05 \alpha= 0.05 α = 0.05 and exclude the intercept.
BIP : we enumerate candidate models of size at most 10 among the 11 observed covariates, use ridge stabilization ϵ = 10 − 4 \epsilon= 10^{-4} ϵ = 1 0 − 4 , and report the maximum a posteriori model without an intercept.
FAIR : we use the implementation based on neural networks with Gumbel variable selection. We set the invariance penalty to γ = 36 \gamma= 36 γ = 36 , the number of optimization iterations to 10,000, and the variable selection threshold to 0.9. We use an accelerated Gumbel temperature annealing schedule to accommodate the reduced number of optimization iterations.
For each randomized environment configuration, we evaluate the estimated causal parent set S ^ \widehat{S} S using the F1 score,
F 1 = 2 ∣ S ^ ∩ S ⋆ ∣ 2 ∣ S ^ ∩ S ⋆ ∣ + ∣ S ^ ∖ S ⋆ ∣ + ∣ S ⋆ ∖ S ^ ∣ \mathrm{F1} = \frac{2|\widehat{S} \cap S^\star|}{2|\widehat{S} \cap S^\star| + |\widehat{S} \setminus S^\star| + |S^\star\setminus\widehat{S}|} F1 = 2∣ S ∩ S ⋆ ∣ + ∣ S ∖ S ⋆ ∣ + ∣ S ⋆ ∖ S ∣ 2∣ S ∩ S ⋆ ∣
the number of false discoveries ∣ S ^ ∖ S ⋆ ∣ |\widehat{S} \setminus S^\star| ∣ S ∖ S ⋆ ∣ , and runtime.
We generate data from the nonlinear structural equation model described in Example 6. We fix the ambient dimension at p = 6 p = 6 p = 6 and the number of environments at ∣ E ∣ = 5 |\mathcal{E}| = 5 ∣ E ∣ = 5 , while varying the sample size per environment over n e ∈ { 100 , 150 , … , 500 } n^e \in\{100, 150, \ldots, 500\} n e ∈ { 100 , 150 , … , 500 } . The true causal parent set has size ∣ S ⋆ ∣ = 2 |S^\star| = 2 ∣ S ⋆ ∣ = 2 . For each sample size, we conduct 100 independent replications. We use the all-intervention mechanism described above, replacing each linear structural relationship with the nonlinear function tanh ( 2 sin ( x ) ) \tanh(2\sin(x)) tanh ( 2 sin ( x )) . The SEM is generated with p a c t = 0.2 p_{\mathrm{act}} = 0.2 p act = 0.2 , ℓ = 0.1 \ell= 0.1 ℓ = 0.1 , u = 0.5 u = 0.5 u = 0.5 , ℓ ⋆ = 0.5 \ell^\star= 0.5 ℓ ⋆ = 0.5 , u ⋆ = 1 u^\star= 1 u ⋆ = 1 , σ X , min = 0.5 \sigma_{X,\min} = 0.5 σ X , m i n = 0.5 , σ X , max = 1 \sigma_{X,\max} = 1 σ X , m a x = 1 , and σ Y = 0.5 \sigma_Y = 0.5 σ Y = 0.5 . Across environments, the structural coefficients and noise variances generating the covariates are independently resampled, while the nonlinear response mechanism remains invariant.
We compare our proposed nonlinear MIP with nonlinear ICP , WVM , and FAIR . We describe the implementation details below:
Nonlinear MIP: We expand each covariate using four Fourier basis functions: sin ( x ) \sin(x) sin ( x ) , cos ( x ) \cos(x) cos ( x ) , sin ( 2 x ) \sin(2x) sin ( 2 x ) and cos ( 2 x ) \cos(2x) cos ( 2 x ) . We set λ n = 0.01 \lambda_n = 0.01 λ n = 0.01 , γ = 0.5 \gamma= 0.5 γ = 0.5 , and ρ = 1 \rho= 1 ρ = 1 . The resulting group ℓ 0 \ell_0 ℓ 0 optimization problem is solved using coordinate descent, initialized from the empty set.
Nonlinear ICP : We use the nonlinearICP package with the invariant residual distribution test and GAM fitting. We set the significance level to α = 0.05 \alpha= 0.05 α = 0.05 and restrict the maximum candidate set size to two.
WVM : We use significance level α = 0.05 \alpha= 0.05 α = 0.05 and B = 300 B = 300 B = 300 bootstrap replications.
FAIR : We use the implementation based on neural networks with Gumbel variable selection. We set the invariance penalty to γ = 36 \gamma= 36 γ = 36 , the number of optimization iterations to 10,000, and the variable selection threshold to 0.9.
We generate data using the noise-intervention model with hidden confounding described in Example 7. We consider ∣ E ∣ = 10 |\mathcal{E}| = 10 ∣ E ∣ = 10 environments and vary the sample size per environment over n e ∈ { 1000 , 1500 , … , 5000 } n^{e} \in\{1000, 1500, \ldots, 5000\} n e ∈ { 1000 , 1500 , … , 5000 } . For each sample size, we conduct 200 independent replications. We set q = 2 q = 2 q = 2 , ∣ S ⋆ ∣ = 4 |S^{\star}| = 4 ∣ S ⋆ ∣ = 4 , and independently sample ∣ A ∣ , ∣ D ∣ ∈ { 2 , 3 } |A|, |D| \in\{2, 3\} ∣ A ∣ , ∣ D ∣ ∈ { 2 , 3 } , with eight additional independent standard Gaussian covariates. The structural coefficients are generated with β S ⋆ = ( 1 , 1 , 1 , 1 ) T \boldsymbol{\beta}_{S^{\star}} = (1, 1, 1, 1)^{\mathsf{T}} β S ⋆ = ( 1 , 1 , 1 , 1 ) T , θ d = 0.1 \theta_{d} = 0.1 θ d = 0.1 for all d ∈ D d \in D d ∈ D . The remaining coefficients for the relationships among H H H , X A X_{A} X A , X S ⋆ X_{S^{\star}} X S ⋆ , and X D X_{D} X D are independently drawn from N ( 0 , 0.3 2 ) \mathcal{N}(0, 0.3^{2}) N ( 0 , 0. 3 2 ) . The coefficients for the direct effect of H H H on Y Y Y are independently drawn from N ( 0 , 1 ) \mathcal{N}(0, 1) N ( 0 , 1 ) .
We set the variance of the response noise to ν Y = 0.2 2 \nu_{Y} = 0.2^{2} ν Y = 0. 2 2 and the variances of the covariate noise terms to ν j e = ( 0.2 + 0.05 ( e − 1 ) ) 2 \nu_{j}^{e} = (0.2 + 0.05(e - 1))^{2} ν j e = ( 0.2 + 0.05 ( e − 1 ) ) 2 for all j ∈ A ∪ S ⋆ ∪ D j \in A \cup S^{\star} \cup D j ∈ A ∪ S ⋆ ∪ D and e ∈ { 1 , … , 10 } e \in\{1, \ldots, 10\} e ∈ { 1 , … , 10 } . All structural coefficients remain fixed across environments.
We compare our proposed method, MIP-hidden , with EILLS-s , Causal Dantzig , BIP-s , negDRO , and CoCo . For EILLS-s and BIP-s , we first screen the covariates using ridge regression, keeping the top 15 and 10 variables, respectively. We describe the implementation details below:
MIP-hidden : we set λ = 0.5 \lambda= 0.5 λ = 0.5 , γ = 300 \gamma= 300 γ = 300 , and s max = 10 s_{\max} = 10 s m a x = 10 . We use MIPFocus=3 and Cuts=3, and impose a time limit of 60 seconds.
EILLS-s : we set its penalty parameter to γ = 20 \gamma= 20 γ = 20 and exclude the intercept.
Causal Dantzig : we use the hidden variable implementation with α = 0.1 \alpha= 0.1 α = 0.1 and exclude the intercept.
BIP-s : we enumerate candidate models over the 10 screened variables, use ridge stabilization ϵ = 10 − 4 \epsilon= 10^{-4} ϵ = 1 0 − 4 , and report the maximum a posteriori model without an intercept.
negDRO : we set its penalty parameter to γ = 20 \gamma= 20 γ = 20 .
CoCo : we set the regularization parameter to λ = 10 \lambda= 10 λ = 10 , learning rate to 0.01, and maximum number of iterations to 10,000, using equal environment weights and excluding the intercept.