betaFeedback

WorksMathematics

Decorrelation Before Variable Selection: Left and Right Transforms and Their Trade-Off

Abstract

Decorrelation has two forms. A left transform keeps coefficients and changes noise; a right transform keeps noise and changes coefficients. Cholesky whitening alone keeps noise variables inactive when true ones come first; decorrelation preserving coefficients and noise loses information.

Introduction

Consider the linear model without intercept

y=Xβ+ε,Cov⁡(ε)=σ2In,(1)y = X\beta+ \varepsilon,\qquad\operatorname{Cov}(\varepsilon) = \sigma^2 I_n, \tag*{(1)}

with X∈Rn×pX \in\mathbb{R}^{n \times p} column-centred, so that rank⁡(X)≤n−1\operatorname{rank}(X) \le n - 1, and β\beta sparse. We call S={j:βj≠0}S = \{j : \beta_j \ne0\} the set of true variables and the others noise variables, and we say that a variable is active in a model if its coefficient in that model is non-zero, and inactive otherwise.

Correlation among the columns of XX can violate the irrepresentable condition [21] for the Lasso [16], weaken marginal screening [2] and make selection unstable [see the discussion in Meinshausen and Bühlmann, 2010]. A common solution in variable selection and screening is to transform the data so that the columns become uncorrelated, or nearly so, before selecting. Linear transforms come in two kinds, left (row) and right (column) transforms, a distinction drawn by Kelner et al. [ [7] for preconditioning in the space of samples and of parameters:

Fy=(FX)β+Fε,Cov⁡(Fε)=σ2FFT;y=(XG)γ+ε,γ=G−1β,Fy = (FX)\beta+ F\varepsilon,\qquad\operatorname{Cov}(F\varepsilon) = \sigma^2 FF^{\mathsf{T}};\qquad y = (XG)\gamma+ \varepsilon,\qquad\gamma= G^{-1}\beta,

with F∈Rm×nF \in\mathbb{R}^{m \times n} and an invertible G∈Rp×pG \in\mathbb{R}^{p \times p}. A left transform keeps the coefficients β\beta but transforms the noise (for whitening, the two views coincide up to which parametrisation is penalised; end of Section 3). A right transform keeps the noise and, for fixed GG, each row depends only on its own observation, but the coefficients, and hence the meaning of the columns, change. Wauthier et al. [2013] compare left preconditioners for the Lasso in a common framework that allows different matrices for XX and yy; we apply the same FF to both, so that model (1) is preserved.

We show that existing methods are, at their core, left transforms (preconditioning, DECO, HOLP screening, Cholesky-based penalisation), right transforms (whitening, Gram–Schmidt orthogonalisation, weighted Lassos) or a combination of both, and we give two results that explain their trade-offs. Proposition 1 shows that, among right whitenings, Cholesky alone keeps noise variables inactive whenever the true variables come first, whatever their number and coefficients. Propositions 2 and 3 show that no fixed linear transform decorrelates the columns while keeping both the coefficients and white noise, unless it discards information; the whitened estimator that underlies fixed-X knockoffs is such a reduced model.

Left transforms

Let X=UDVTX = UDV^{\mathsf{T}} be the compact singular value decomposition, with DD the diagonal matrix of the rank⁡(X)\operatorname{rank}(X) non-zero singular values. Since XX is centred, XXTXX^{\mathsf{T}} is singular. Jia and Rohe [ [5], Wang et al. [ [19] and Wang and Leng [2016] write their transforms or estimators with (XXT)−1(XX^{\mathsf{T}})^{-1} or its square root, which requires XXTXX^{\mathsf{T}} to be invertible. Wang et al. [ [19] note that this fails after standardisation and use a ridge version, and they also use the pseudo-inverse when p≤np \le n; Wang and Leng [2016] also propose a ridge version, recommended when XXTXX^{\mathsf{T}} is close to degeneracy. For centred XX we read (XXT)−1(XX^{\mathsf{T}})^{-1} as the pseudo-inverse (XXT)+=UD−2UT(XX^{\mathsf{T}})^{+} = UD^{-2}U^{\mathsf{T}}. Ridge parameters rr below are positive.

Puffer transform Jia and Rohe [ [5]. Take F=UD−1UTF = UD^{-1}U^{\mathsf{T}}. Then

FX=UVT,(FX)TFX=VVT,Cov⁡(Fε)=σ2UD−2UT.FX = UV^{\mathsf{T}}, \qquad(FX)^{\mathsf{T}}FX = VV^{\mathsf{T}}, \qquad\operatorname{Cov}(F\varepsilon) = \sigma^{2}UD^{-2}U^{\mathsf{T}}.

If rank⁡(X)=p\operatorname{rank}(X) = p, then VVT=IpVV^{\mathsf{T}} = I_p and the columns of FXFX are exactly orthonormal; otherwise, in particular if p≥np \ge n, VVTVV^{\mathsf{T}} is only a projection and the columns are in general not orthogonal. Jia and Rohe [ [5] show that FXFX can satisfy the irrepresentable condition when XX does not, so the preconditioned Lasso is sign consistent in a wider range of settings. The price is the noise: FF inflates the smallest singular values of XX to one, which also inflates and correlates the errors.

DECO Wang et al. [ [19]. Aimed mainly at p≫np \gg n, DECO uses F=(XXT/p)−1/2=p UD−1UTF = (XX^{\mathsf{T}}/p)^{-1/2} = \sqrt{p}\,UD^{-1}U^{\mathsf{T}}, the Puffer transform up to the factor p\sqrt{p}; because XXTXX^{\mathsf{T}} has rank at most n−1n-1 after standardisation, its algorithm uses p(XXT+rIn)−1/2\sqrt{p}(XX^{\mathsf{T}} + rI_n)^{-1/2}. After decorrelation, the variables are split across machines and a sparse regression is fitted to each block. For both transforms, each transformed row typically mixes all nn observations, so a subsample of transformed rows has correlated noise.

HOLP Wang and Leng, [17]; Wang et al. [19]. HOLP ranks the variables by the absolute values of the entries of β^HOLP=XT(XXT)−1y\hat{\beta}^{\mathrm{HOLP}} = X^{\mathsf{T}}(XX^{\mathsf{T}})^{-1}y. With the Puffer FF, (FX)T(Fy)=XTUD−2UTy=XT(XXT)+y(FX)^{\mathsf{T}}(Fy) = X^{\mathsf{T}}UD^{-2}U^{\mathsf{T}}y = X^{\mathsf{T}}(XX^{\mathsf{T}})^{+}y, so HOLP is marginal screening, as in SIS Fan and Lv, [2], applied after left decorrelation. Ridge-HOLP uses XT(XXT+rIn)−1y=(FrX)TFryX^{\mathsf{T}}(XX^{\mathsf{T}} + rI_n)^{-1}y = (F_rX)^{\mathsf{T}}F_ry with Fr=(XXT+rIn)−1/2F_r = (XX^{\mathsf{T}} + rI_n)^{-1/2}, which is also the ridge regression estimate Hoerl and Kennard, [4]; Air-HOLP Joudah et al., [6] chooses rr adaptively.

Cholesky as a left transform. If rank⁡(X)=p\operatorname{rank}(X)=p, write the thin QR factorisation X=QRX=QR, with RR upper triangular with positive diagonal, so that XTX=RTRX^{\mathsf{T}}X=R^{\mathsf{T}}R and RR is the Cholesky factor. Taking F=QTF=Q^{\mathsf{T}} gives

QTy=Rβ+QTε,Cov⁡(QTε)=σ2Ip,Q^{\mathsf{T}}y=R\beta+Q^{\mathsf{T}}\varepsilon,\qquad\operatorname{Cov}(Q^{\mathsf{T}}\varepsilon)=\sigma^{2}I_{p},

and ∥y−Xβ∥2=∥QTy−Rβ∥2+∥(In−QQT)y∥2\lVert y-X\beta\rVert^{2}=\lVert Q^{\mathsf{T}}y-R\beta\rVert^{2}+\lVert(I_{n}-QQ^{\mathsf{T}})y\rVert^{2}, so the least-squares estimate and the Lasso path are unchanged (with the loss scaled by the same constant). The Cholesky matrix penalisation of Nghiem et al. [ [11] has this form: for sliced inverse regression, it solves Σβ=η\Sigma\beta=\eta through Σ=LLT\Sigma=LL^{\mathsf{T}} by a Lasso on LTβ≈L−1ηL^{\mathsf{T}}\beta\approx L^{-1}\eta; with Σ=XTX\Sigma=X^{\mathsf{T}}X and η=XTy\eta=X^{\mathsf{T}}y, this is the Lasso on (QTy,R)(Q^{\mathsf{T}}y,R). The transform keeps β\beta and white noise but does not decorrelate, since RTR=XTXR^{\mathsf{T}}R=X^{\mathsf{T}}X.

Right transforms

A right transform GG whitens XX if GTXTXG=IpG^{\mathsf{T}}X^{\mathsf{T}}XG=I_{p}; this requires rank⁡(X)=p\operatorname{rank}(X)=p, hence p≤n−1p\le n-1 because XX is centred. Writing M=G−1M=G^{-1}, the condition is MTM=XTXM^{\mathsf{T}}M=X^{\mathsf{T}}X, and the transformed coefficient is γ=Mβ\gamma=M\beta. Kessy et al. [ [8] compare the standard choices:

  • ZCA (zero-phase component analysis, or Mahalanobis whitening): G=(XTX)−1/2G=(X^{\mathsf{T}}X)^{-1/2}. The whitened data stay as close as possible to the original data. Zhu et al. [ [22] apply ZCA-type whitening, with an estimated correlation matrix of the predictors, before a Lasso-type selection, noting that, unlike left preconditioning, the error term is unchanged.

  • PCA (principal component analysis): G=VD−1G=VD^{-1}, so XG=UXG=U. Each new column is in general a mix of all original variables.

  • Cholesky: G=R−1G=R^{-1}, so XG=QXG=Q. Computationally this is Gram–Schmidt orthogonalisation: column jj of QQ is the part of XjX_j not explained by X1,…,Xj−1X_1,\ldots,X_{j-1}, normalised to unit length, so the result depends on the ordering of the columns.

A desirable property for selection is that a right transform does not move signal into columns that come later in the ordering than the true variables. The next result shows that this property singles out Cholesky.

Proposition 1. Assume XTXX^{\mathsf{T}}X is positive definite, let M∈Rp×pM\in\mathbb{R}^{p\times p} satisfy MTM=XTXM^{\mathsf{T}}M=X^{\mathsf{T}}X, and fix the ordering of the columns. The following are equivalent:

(a) for every kk and every β\beta with supp⁡(β)⊆{1,…,k}\operatorname{supp}(\beta)\subseteq\{1,\ldots,k\}, we have supp⁡(Mβ)⊆{1,…,k}\operatorname{supp}(M\beta)\subseteq\{1,\ldots,k\}, where supp⁡(v)={j:vj≠0}\operatorname{supp}(v)=\{j:v_j\ne0\};

(b) MM is upper triangular;

(c) M=ΔRM=\Delta R, where Δ\Delta is diagonal with entries ±1\pm1 and RR is the Cholesky factor of XTXX^{\mathsf{T}}X.

Proof. (a)⇒\Rightarrow(b): take β=ej\beta=e_j, whose support lies in {1,…,j}\{1,\ldots,j\}. Then MejMe_j, the jjth column of MM, is supported on {1,…,j}\{1,\ldots,j\}, so Mij=0M_{ij}=0 for i>ji>j. (b)⇒\Rightarrow(a): if MM is upper triangular, then (Mβ)i=∑l≥iMilβl(M\beta)_i=\sum_{l\ge i}M_{il}\beta_l, which vanishes for i>ki>k whenever βl=0\beta_l=0 for l>kl>k. (b)⇒\Rightarrow(c): MM is invertible because XTXX^{\mathsf{T}}X is, so its diagonal entries are non-zero. Let Δ=diag⁡(sign⁡(M11),…,sign⁡(Mpp))\Delta=\operatorname{diag}(\operatorname{sign}(M_{11}),\ldots,\operatorname{sign}(M_{pp})). Then ΔM\Delta M is upper triangular with positive diagonal and (ΔM)T(ΔM)=XTX(\Delta M)^{\mathsf{T}}(\Delta M)=X^{\mathsf{T}}X, so ΔM=R\Delta M=R by uniqueness of the Cholesky factor, and M=ΔRM=\Delta R. (c)⇒\Rightarrow(b) is immediate. ∎

The equivalence of (b) and (c) is the uniqueness of the Cholesky factor, and Kessy et al. [ [8] characterise Cholesky whitening by the analogous triangularity of the cross-product (XG)TX=M(XG)^{\mathsf{T}}X=M. Their whitening matrix acts on a column vector, so it corresponds to GTG^{\mathsf{T}} here; they use the Cholesky factor of the precision matrix, which makes MM lower triangular and amounts to the reversed ordering. Kelner et al. [ [7] precondition the Lasso on the right with a sparse block Cholesky factorisation, so that, for low-treewidth designs, sparse coefficient vectors stay sparse; condition (a) asks for more, that the support stay within the leading coordinates. Triangularity itself is classical, since Gram–Schmidt orthogonalisation is sequential regression [3 Section 3.2.3]; what we add, to our knowledge, is condition (a), which reads it as a selection property and shows that Cholesky is the only whitening with it. The quantifier “for every kk” matters: for a single known kk, any whitening with MM block upper triangular at kk also satisfies (a) for that kk.

Here, active and inactive refer to the coefficients γ=Mβ\gamma=M\beta of the transformed model, not to an estimate; a noise variable with γj≠0\gamma_j \ne0 is a correct selection in the transformed model but a false one in terms of the original variables. Thus, if the true variables come first, Cholesky is, up to signs, the only whitening that keeps the noise variables inactive, γj=0\gamma_j=0 for j>∣S∣j>|S|, for every number of true variables and every coefficient vector on them. Two limitations remain.

  • True variables may become weak. Since γj=∑ℓ≥jRjℓβℓ\gamma_j=\sum_{\ell\ge j}R_{j\ell}\beta_\ell, a true variable that is well explained by earlier ones has a small RjjR_{jj}, and γj\gamma_j can even vanish through cancellation. No information is lost, since the transform is invertible; the shared signal is carried by earlier columns.

  • The ordering must be right. A noise variable placed before a true variable with which it is correlated typically absorbs some of its signal. The requirement that each true variable precede the noise variables correlated with it, which Nouraie et al. [ [12] use for consistent selection, does not keep them inactive. Take true variables T1,T2T_1,T_2 and a noise variable NN in the ordering (T1,N,T2)(T_1,N,T_2), with correlations between the columns ρ(T1,N)=0.6\rho(T_1,N)=0.6, ρ(T1,T2)=0.5\rho(T_1,T_2)=0.5, ρ(N,T2)=0\rho(N,T_2)=0 and β=(1,0,1)\beta=(1,0,1). For unit-norm columns, the Cholesky factor of the correlation matrix gives γ=(1.5,−0.375,0.781)\gamma=(1.5,-0.375,0.781), so NN picks up signal from T2T_2 through T1T_1 and is active. This requirement does ensure γN=0\gamma_N=0 when groups of correlated variables are mutually uncorrelated and, within each group, every noise variable is correlated with every true variable.

When XGXG has orthonormal columns, the Lasso on XGXG is soft-thresholding: minimising 12∥y−XGγ∥2+λ∥γ∥1\frac{1}{2}\lVert y-XG\gamma\rVert^2+\lambda\lVert\gamma\rVert_1 gives γ^j=sign⁡(zj)(∣zj∣−λ)+\hat{\gamma}_j=\operatorname{sign}(z_j)(|z_j|-\lambda)_+, with z=(XG)Ty=Mβ^OLSz=(XG)^{\mathsf{T}}y=M\hat{\beta}^{\mathrm{OLS}} and Cov⁡(z)=σ2Ip\operatorname{Cov}(z)=\sigma^2I_p. So every right whitening thresholds a different linear image of the least-squares estimate, ZCA, PCA and Cholesky differing only in MM, whereas Puffer, for n>pn>p and full-rank XX, thresholds β^OLS\hat{\beta}^{\mathrm{OLS}} itself [14], with correlated noise [5]. Equivalently, a whitened right-transform Lasso is a left transform F=(XG)TF=(XG)^{\mathsf{T}}, which gives Fy=Mβ+FεFy=M\beta+F\varepsilon with FX=MFX=M and Cov⁡(Fε)=σ2Ip\operatorname{Cov}(F\varepsilon)=\sigma^2I_p, followed by a Lasso that penalises MβM\beta instead of β\beta. For Cholesky, F=QTF=Q^{\mathsf{T}} and M=RM=R, so the left and right versions use the same statistic QTyQ^{\mathsf{T}}y and differ only in which parametrisation is penalised.

Combining left and right transforms

A method can use one kind of transform for ranking and the other for selection. DVS [12] ranks the variables by Air-HOLP, applies Gram–Schmidt to the columns of XX in this order, and runs stability selection [10] with the Lasso on QQ; since Ridge-HOLP is screening after a left transform, it uses a left transform for ranking and a right transform for selection. Two general points apply to any such combination. First, the ordering uses yy, so the guarantees above hold for a given ordering. Second, the decorrelation is exact only on the full sample: a subsample QIQ_I of ⌊n/2⌋\lfloor n/2\rfloor rows (index set II), as in stability selection, has E[QITQI]=(∣I∣/n)Id\mathbb{E}[Q_I^{\mathsf{T}}Q_I] = (|I|/n)I_d under uniform subsampling, with d≤rank⁡(X)d \le\operatorname{rank}(X) orthogonalised columns, but each realisation deviates from this, and QITQIQ_I^{\mathsf{T}}Q_I is singular once d>⌊n/2⌋d > \lfloor n/2\rfloor.

Remark. When p>rank⁡(X)p > \operatorname{rank}(X), at most rank⁡(X)\operatorname{rank}(X) columns can be orthogonalised. Once the columns already processed span the column space of XX, every further Gram–Schmidt residual is zero in exact arithmetic; numerically, the normalised residuals are rounding error, giving meaningless unit-norm columns that the Lasso can still select. Orthogonalising only the top d≤rank⁡(X)d \le\operatorname{rank}(X) ranked columns, as screening would, avoids this and, for d≤⌊n/2⌋d \le\lfloor n/2\rfloor, typically keeps QITQIQ_I^{\mathsf{T}}Q_I non-singular, provided these columns are linearly independent; the price is that the true variables must rank in the top dd.

The trade-off

Assume here that XTXX^{\mathsf{T}}X is invertible, so that exact decorrelation is possible; this requires p≤n−1p \le n-1 (XX is centred), so the results below do not cover the p≥np \ge n regime that motivates screening. A decorrelating transform would ideally achieve all three of the following:

  1. uncorrelated columns: the transformed Gram matrix is diagonal;

  2. unchanged meaning: the coefficients are β\beta, up to a fixed diagonal rescaling and relabelling;

  3. unchanged noise: the transformed noise has covariance σ2I\sigma^2 I.

Proposition 2. Let XTXX^{\mathsf{T}}X be invertible and not diagonal, and consider a fixed linear transform, that is, one that does not depend on yy, (y,X)↦(Fy,FXG)(y,X) \mapsto(Fy,FXG) with F∈Rm×nF \in\mathbb{R}^{m\times n} and an invertible G∈Rp×pG \in\mathbb{R}^{p\times p}. If it satisfies (i), (ii) for every β\beta, and (iii), then m<nm < n, that is, the data are reduced to fewer than nn linear combinations of the observations.

Proof. Writing the reduced model as Fy=(FXG)(G−1β)+FεFy=(FXG)(G^{-1}\beta)+F\varepsilon and applying (ii) to β=ej\beta=e_j gives G−1ej=cjeπ(j)G^{-1}e_j=c_je_{\pi(j)} with cj≠0c_j\ne0, for j=1,…,pj=1,\ldots,p; since G−1G^{-1} is invertible, π\pi is a permutation, so GG is a scaled permutation. Condition (iii) means FFT=ImFF^{\mathsf{T}}=I_m, hence m≤nm\le n. If m=nm=n, then FF is orthogonal, so the transformed Gram matrix is GTXTXGG^{\mathsf{T}}X^{\mathsf{T}}XG, which, GG being a scaled permutation, is diagonal only if XTXX^{\mathsf{T}}X is. Hence (i) fails unless m<nm<n.

To our knowledge, Proposition 2 and Proposition 3 below have not been stated in this form. Any transform in this class that achieves (i) must give up (ii), give up (iii), or reduce the data: Cholesky as a right transform gives (i) and (iii); Puffer gives (i) and (ii), and when rank⁡(X)=p\operatorname{rank}(X)=p, FX=UVTFX=U V^{\mathsf{T}} is exactly the ZCA-whitened design, so the two differ only in whether β\beta or the noise is kept; Cholesky as a left transform gives (ii) and (iii) with m=p<nm=p<n, but not (i). Achieving all three requires GG to be a scaled permutation and F=WTF=W^{\mathsf{T}} with W∈Rn×mW\in\mathbb{R}^{n\times m} having orthonormal columns and m<nm<n.

Proposition 3. Let W∈Rn×mW\in\mathbb{R}^{n\times m} have orthonormal columns and suppose the columns of WTXW^{\mathsf{T}}X are orthogonal, with XTWWTX=Λ=diag⁡(μ1,…,μp)X^{\mathsf{T}}WW^{\mathsf{T}}X=\Lambda=\operatorname{diag}(\mu_1,\ldots,\mu_p) and all μj>0\mu_j>0. Then the reduced model WTy=(WTX)β+WTεW^{\mathsf{T}}y=(W^{\mathsf{T}}X)\beta+W^{\mathsf{T}}\varepsilon satisfies (i), (ii) and (iii), and

Λ⪯XTX,henceσ2μj≥σ2[(XTX)−1]jj.\Lambda\preceq X^{\mathsf{T}}X,\qquad\text{hence}\qquad\frac{\sigma^2}{\mu_j}\ge\sigma^2\left[(X^{\mathsf{T}}X)^{-1}\right]_{jj}.

Proof. Cov⁡(W⊤ε)=σ2W⊤W=σ2Im\operatorname{Cov}(W^{\top}\varepsilon)=\sigma^2W^{\top}W=\sigma^2I_m, the coefficients are β\beta, and the Gram matrix is Λ\Lambda. Since In−WW⊤⪰0I_n-WW^{\top}\succeq0, we get X⊤X−Λ=X⊤(In−WW⊤)X⪰0X^{\top}X-\Lambda=X^{\top}(I_n-WW^{\top})X\succeq0. Both Λ\Lambda and X⊤XX^{\top}X are positive definite, so inverting reverses the order, Λ−1⪰(X⊤X)−1\Lambda^{-1}\succeq(X^{\top}X)^{-1}, and comparing diagonal entries gives the bound. □\square

The left-hand side of the bound is the variance of the least-squares estimate of βj\beta_j in the reduced model; the right-hand side is that from the full data, the Gauss–Markov bound for linear unbiased estimators. If X⊤XX^{\top}X is not diagonal then Λ≠X⊤X\Lambda\ne X^{\top}X, so Λ−1−(X⊤X)−1\Lambda^{-1}-(X^{\top}X)^{-1} is non-zero and positive semidefinite, hence has positive trace, and for at least one jj the variance σ2/μj\sigma^2/\mu_j strictly exceeds the full-data variance: the reduction loses information. Under strong correlation [(X⊤X)−1]jj[(X^{\top}X)^{-1}]_{jj} is large for the variables involved, so μj≤1/[(X⊤X)−1]jj\mu_j\le1/[(X^{\top}X)^{-1}]_{jj} is small for them. By contrast, the left Cholesky transform reduces the data to pp dimensions without changing the least-squares estimate or its variance, because it does not decorrelate: the loss comes from (i), not from the reduction itself.

Such a WW need not exist. With m=pm=p, one construction is W=X(X⊤X)−1Λ1/2+ZW=X(X^{\top}X)^{-1}\Lambda^{1/2}+Z, where the columns of ZZ are orthogonal to those of XX and Z⊤Z=Ip−Λ1/2(X⊤X)−1Λ1/2Z^{\top}Z=I_p-\Lambda^{1/2}(X^{\top}X)^{-1}\Lambda^{1/2}. This needs Λ⪯X⊤X\Lambda\preceq X^{\top}X and rank⁡(X⊤X−Λ)≤n−p\operatorname{rank}(X^{\top}X-\Lambda)\le n-p; for example, Λ=λmin⁡(X⊤X)Ip\Lambda=\lambda_{\min}(X^{\top}X)I_p works whenever n≥2p−1n\ge2p-1. For this WW, Λ−1/2W⊤y=β^OLS+Λ−1/2Z⊤ε\Lambda^{-1/2}W^{\top}y=\hat{\beta}^{\mathrm{OLS}}+\Lambda^{-1/2}Z^{\top}\varepsilon, the full-data least-squares estimate plus extra noise. Under Gaussian errors, this has the same distribution as the whitened estimator of Li and Fithian [ [9], with covariance σ2Λ−1\sigma^2\Lambda^{-1} (they add user-generated noise where we use the residual space): their requirement that a diagonal Ω\Omega satisfy Ω⪰(X⊤X)−1\Omega\succeq(X^{\top}X)^{-1} is our Λ⪯X⊤X\Lambda\preceq X^{\top}X with Ω=Λ−1\Omega=\Lambda^{-1}, and they show that the price in variance can be very high when the eigenstructure of X⊤XX^{\top}X is unfavourable.

The fixed-XX knockoff construction [1] supplies such a WW; Li and Fithian [ [9] show that the fixed-XX knockoff filter is equivalent to inference on such a whitened estimator, built from the contrasts Xj−X~jX_j-\widetilde{X}_j [9 eq. (5]; the filter can also use (X+X~)⊤y(X+\widetilde{X})^{\top}y, which, under Gaussian errors, is independent of this estimator, to rank the variables, together with the magnitude of the whitened estimator, so only the estimator it tests with is reduced. With columns normalised to unit length, Σ=X⊤X\Sigma=X^{\top}X and n≥2pn\ge2p, the knockoffs exist when 0⪯diag⁡(s)⪯2Σ0\preceq\operatorname{diag}(s)\preceq2\Sigma; when all sj>0s_j>0, the contrasts Xj−X~jX_j-\widetilde{X}_j are mutually orthogonal with squared norms 2sj2s_j, and taking W=(Xj−X~j)/2sjW=(X_j-\widetilde{X}_j)/\sqrt{2s_j} gives Λ=diag⁡(s)/2\Lambda=\operatorname{diag}(s)/2. The constraint Λ⪯X⊤X\Lambda\preceq X^{\top}X is then exactly diag⁡(s)⪯2Σ\operatorname{diag}(s)\preceq2\Sigma, so the sjs_j must be small under strong correlation, one way to see why knockoffs lose power for highly correlated variables. Barber and Candès [ [1] recommend choosing the sjs_j as large as possible, whereas Spector and Janson [ [15] show for model-XX knockoffs that pushing ss to the boundary can itself reduce power, and propose alternatives within the same constraint that are also more powerful in fixed-XX simulations.

A meaning-preserving alternative. Proposition 2 also shows what remains when (ii) and (iii) are both required without reducing the data: GG must be a scaled permutation, so the columns can only be rescaled. A weighted Lasso, such as the adaptive Lasso [23], is exactly of this form. With positive penalty factors wjw_j, minimising ∥y−Xβ∥2+∑jwj∣βj∣\lVert y-X\beta\rVert^2+\sum_j w_j|\beta_j| is the ordinary Lasso applied to XGXG with G=diag⁡(1/w1,…,1/wp)G=\operatorname{diag}(1/w_1,\ldots,1/w_p), whose coefficients γj=wjβj\gamma_j=w_j\beta_j keep their meaning; a zero weight leaves that variable unpenalised (the objective then has no such GG). The Stable Lasso [13] sets wjw_j to an increasing function of the Air-HOLP rank of variable jj (rank 1 being the most relevant), so that, as in DVS, a left transform supplies the ranking. When the weights depend on yy, the argument applies for given weights. Such methods give up (i), but by the proof of Proposition 2, rescaling, up to a square orthogonal FF and a reordering of the columns, which leave the Lasso unchanged, is the only option that keeps both the meaning of the coefficients and white noise without reducing the data. Like Puffer, their gains come from changing how the Lasso behaves, not from new information.

Discussion

No transformation of the data, linear or not, can create information about β\beta; under Gaussian errors and with X⊤XX^\top X invertible, any unbiased estimate of βj\beta_j has variance at least σ2[(X⊤X)−1]jj\sigma^2[(X^\top X)^{-1}]_{jj}, however the data are transformed. By Propositions 2 and 3, achieving (i)–(iii) together loses information unless the columns of XX are already orthogonal. This is a statement about information, not about procedures: because the Lasso is nonlinear and ignores the noise covariance, a transform can still change how well it selects, as the Puffer results of Jia and Rohe [ [5] show; conversely, Kelner et al. [ [7] construct Gaussian designs of high treewidth for which no right-preconditioned ℓ1\ell_1 program recovers sparse vectors from fewer than polynomially many samples in the treewidth. Table 1 summarises which property each method gives up.

MethodSideUncorrelated columnsKeeps β\betaNoise stays whiteNeeds ordering
Puffer, DECOleftexactly if rank⁡(X)=p\operatorname{rank}(X)=p (Puffer); otherwise not in generalyesnono
Cholesky (Q⊤Q^\top)†leftnoyesyesno
Projection (W⊤W^\top)*leftexactlyyesyesno
ZCA, PCArightexactly; needs rank⁡(X)=p\operatorname{rank}(X)=pnoyesno
Cholesky (Gram–Schmidt)rightexactly on the full sample, for at most rank⁡(X)\operatorname{rank}(X) columnsno; noise variables stay inactive if true ones come firstyes, for a fixed orderingyes
Weighted Lassoright (diagonal)noyes, up to scalingyes, for fixed weightsno

Table 1. Properties of decorrelating transforms. †Reduces the data to pp dimensions without changing the least-squares estimate or the Lasso path; needs rank⁡(X)=p\operatorname{rank}(X)=p; used by Nghiem et al. [ [11]. *Reduces the data and loses information; exists, for example, when rank⁡(X)=p\operatorname{rank}(X)=p and n≥2p−1n \ge2p-1; never, with all μj>0\mu_j>0, when rank⁡(X)<p\operatorname{rank}(X)<p.

In practice, the choice depends on what the analysis needs. If the goal is to select among the original variables and interpret their coefficients, a method that keeps β\beta, such as a weighted Lasso, is the natural choice, and the correlation has to be handled by the selection method itself. If decorrelated columns are worth losing that interpretation, Cholesky is the right transform to use, but its results are only as good as the ordering, and its decorrelation is exact only on the full sample. Methods that keep both β\beta and white noise while decorrelating exist only by losing information, as Proposition 3 shows; the whitened estimator behind fixed-X knockoffs is an example.

Funding

Mahdi Nouraie began this work during his PhD, supported by the Macquarie University Research Excellence Scholarship (20213605), and continued it with support from the Australian Research Council Discovery Project Grant (DP260100348), which also supported Samuel Muller.

Data availability

No data were used for the research described in this article.

Declaration of generative AI and AI-assisted technologies in the manuscript preparation process

During the preparation of this work the authors used Claude (Anthropic) to help edit the text and to check the derivations. After using this tool, the authors reviewed and edited the content as needed and take full responsibility for the content of the published article.

References

  1. [1]Rina Foygel Barber and Emmanuel J. Candès. Controlling the false discovery rate via knockoffs. The Annals of Statistics, 43(5):2055–2085, 2015. doi: 10.1214/15-AOS1337.DOI
  2. [2]Jianqing Fan and Jinchi Lv. Sure independence screening for ultrahigh dimensional feature space. Journal of the Royal Statistical Society: Series B, 70(5):849–911, 2008. doi: 10.1111/j.1467-9868.2008.00674.x.
  3. [3]Trevor Hastie, Robert Tibshirani, and Jerome Friedman. The Elements of Statistical Learning: Data Mining, Inference, and Prediction. Springer, New York, 2 edition, 2009.
  4. [4]Arthur E. Hoerl and Robert W. Kennard. Ridge regression: biased estimation for nonorthogonal problems. Technometrics, 12(1):55–67, 1970. doi: 10.1080/00401706.1970.10488634.DOI
  5. [5]Jinzhu Jia and Karl Rohe. Preconditioning the Lasso for sign consistency. Electronic Journal of Statistics, 9(1):1150–1172, 2015. doi: 10.1214/15-EJS1029.DOI
  6. [6]Ibrahim Joudah, Samuel Muller, and Houying Zhu. Air-HOLP: adaptive regularized feature screening for high dimensional correlated data. Statistics and Computing, 35(3):63, 2025. doi: 10.1007/s11222-025-10599-6.DOI
  7. [7]Jonathan A. Kelner, Frederic Koehler, Raghu Meka, and Dhruv Rohatgi. On the power of preconditioning in sparse linear regression. In 2021 IEEE 62nd Annual Symposium on Foundations of Computer Science (FOCS), pages 550–561, 2021. doi: 10.1109/FOCS52979.2021.00061.DOI
  8. [8]Agnan Kessy, Alex Lewin, and Korbinian Strimmer. Optimal whitening and decorrelation. The American Statistician, 72(4):309–314, 2018. doi: 10.1080/00031305.2016.1277159.DOI
  9. [9]Xiao Li and William Fithian. Whiteout: when do fixed-X knockoffs fail?, 2021. arXiv:2107.06388.arxiv.org/abs/2107.06388
  10. [10]Nicolai Meinshausen and Peter Bühlmann. Stability selection. Journal of the Royal Statistical Society: Series B, 72(4):417–473, 2010. doi: 10.1111/j.1467-9868.2010.00740.x.DOI
  11. [11]Linh H. Nghiem, Francis K. C. Hui, Samuel Müller, and A. H. Welsh. Sparse sliced inverse regression via Cholesky matrix penalization. Statistica Sinica, 32:2431–2453, 2022. doi: 10.5705/ss.202020.0406.DOI
  12. [12]Mahdi Nouraie, Connor Smith, and Samuel Muller. Stability selection via variable decorrelation. Statistics and Computing, 36:160, 2026a. doi: 10.1007/s11222-026-10916-7.DOI
  13. [13]Mahdi Nouraie, Houying Zhu, and Samuel Muller. A stable Lasso, 2026b. arXiv:2511.02306v2.arxiv.org/abs/2511.02306
  14. [14]Karl Rohe. Preconditioning for classical relationships: a note relating ridge regression and OLS p-values to preconditioned sparse penalized regression. Stat, 4(1):157–166, 2015. doi: 10.1002/sta4.86.DOI
  15. [15]Asher Spector and Lucas Janson. Powerful knockoffs via minimizing reconstructability. The Annals of Statistics, 50(1):252–276, 2022. doi: 10.1214/21-AOS2104.DOI
  16. [16]Robert Tibshirani. Regression shrinkage and selection via the lasso. Journal of the Royal Statistical Society: Series B, 58(1):267–288, 1996. doi: 10.1111/j.2517-6161.1996.tb02080.x.DOI
  17. [17]Xiangyu Wang and Chenlei Leng. High dimensional ordinary least squares projection for screening variables. Journal of the Royal Statistical Society: Series B, 78(3):589–611, 2016. doi: 10.1111/rssb.12127.DOI
  18. [19]Xiangyu Wang, Chenlei Leng, and Tom Boot. Wang and Leng (2016), High-dimensional ordinary least-squares projection for screening variables, Journal of the Royal Statistical Society Series B, 78, 589–611. Journal of the Royal Statistical Society: Series B, 83(4):880–881, 2021. doi: 10.1111/rssb.12427.
  19. [21]Peng Zhao and Bin Yu. On model selection consistency of Lasso. Journal of Machine Learning Research, 7:2541–2563, 2006.DOI
  20. [22]Wencan Zhu, Céline Lévy-Leduc, and Nils Ternès. A variable selection approach for highly correlated predictors in high-dimensional genomic data. Bioinformatics, 37(16):2238–2244, 2021. doi: 10.1093/bioinformatics/btab114.DOI
  21. [23]Hui Zou. The adaptive lasso and its oracle properties. Journal of the American Statistical Association, 101(476):1418–1429, 2006. doi: 10.1198/016214506000000735.DOI

Paper details

vlnc:W_p9vuqmmp
Version
v1 · latest
Published
Fresh — added to the Hub within the last 30 days
Added
License
CC BY 4.0
Usage

views · 0

downloads · 0

A view counts once per browser per day, a download each time the PDF is requested.

  • Mathematics

Claude (Anthropic)

View statement
Looking for related papers…
Contents