WorksMathematics
Decorrelation Before Variable Selection: Left and Right Transforms and Their Trade-Off
Abstract
Decorrelation has two forms. A left transform keeps coefficients and changes noise; a right transform keeps noise and changes coefficients. Cholesky whitening alone keeps noise variables inactive when true ones come first; decorrelation preserving coefficients and noise loses information.
Introduction
Consider the linear model without intercept
with column-centred, so that , and sparse. We call the set of true variables and the others noise variables, and we say that a variable is active in a model if its coefficient in that model is non-zero, and inactive otherwise.
Correlation among the columns of can violate the irrepresentable condition [21] for the Lasso [16], weaken marginal screening [2] and make selection unstable [see the discussion in Meinshausen and Bühlmann, 2010]. A common solution in variable selection and screening is to transform the data so that the columns become uncorrelated, or nearly so, before selecting. Linear transforms come in two kinds, left (row) and right (column) transforms, a distinction drawn by Kelner et al. [ [7] for preconditioning in the space of samples and of parameters:
with and an invertible . A left transform keeps the coefficients but transforms the noise (for whitening, the two views coincide up to which parametrisation is penalised; end of Section 3). A right transform keeps the noise and, for fixed , each row depends only on its own observation, but the coefficients, and hence the meaning of the columns, change. Wauthier et al. [2013] compare left preconditioners for the Lasso in a common framework that allows different matrices for and ; we apply the same to both, so that model (1) is preserved.
We show that existing methods are, at their core, left transforms (preconditioning, DECO, HOLP screening, Cholesky-based penalisation), right transforms (whitening, Gram–Schmidt orthogonalisation, weighted Lassos) or a combination of both, and we give two results that explain their trade-offs. Proposition 1 shows that, among right whitenings, Cholesky alone keeps noise variables inactive whenever the true variables come first, whatever their number and coefficients. Propositions 2 and 3 show that no fixed linear transform decorrelates the columns while keeping both the coefficients and white noise, unless it discards information; the whitened estimator that underlies fixed-X knockoffs is such a reduced model.
Left transforms
Let be the compact singular value decomposition, with the diagonal matrix of the non-zero singular values. Since is centred, is singular. Jia and Rohe [ [5], Wang et al. [ [19] and Wang and Leng [2016] write their transforms or estimators with or its square root, which requires to be invertible. Wang et al. [ [19] note that this fails after standardisation and use a ridge version, and they also use the pseudo-inverse when ; Wang and Leng [2016] also propose a ridge version, recommended when is close to degeneracy. For centred we read as the pseudo-inverse . Ridge parameters below are positive.
Puffer transform Jia and Rohe [ [5]. Take . Then
If , then and the columns of are exactly orthonormal; otherwise, in particular if , is only a projection and the columns are in general not orthogonal. Jia and Rohe [ [5] show that can satisfy the irrepresentable condition when does not, so the preconditioned Lasso is sign consistent in a wider range of settings. The price is the noise: inflates the smallest singular values of to one, which also inflates and correlates the errors.
DECO Wang et al. [ [19]. Aimed mainly at , DECO uses , the Puffer transform up to the factor ; because has rank at most after standardisation, its algorithm uses . After decorrelation, the variables are split across machines and a sparse regression is fitted to each block. For both transforms, each transformed row typically mixes all observations, so a subsample of transformed rows has correlated noise.
HOLP Wang and Leng, [17]; Wang et al. [19]. HOLP ranks the variables by the absolute values of the entries of . With the Puffer , , so HOLP is marginal screening, as in SIS Fan and Lv, [2], applied after left decorrelation. Ridge-HOLP uses with , which is also the ridge regression estimate Hoerl and Kennard, [4]; Air-HOLP Joudah et al., [6] chooses adaptively.
Cholesky as a left transform. If , write the thin QR factorisation , with upper triangular with positive diagonal, so that and is the Cholesky factor. Taking gives
and , so the least-squares estimate and the Lasso path are unchanged (with the loss scaled by the same constant). The Cholesky matrix penalisation of Nghiem et al. [ [11] has this form: for sliced inverse regression, it solves through by a Lasso on ; with and , this is the Lasso on . The transform keeps and white noise but does not decorrelate, since .
Right transforms
A right transform whitens if ; this requires , hence because is centred. Writing , the condition is , and the transformed coefficient is . Kessy et al. [ [8] compare the standard choices:
ZCA (zero-phase component analysis, or Mahalanobis whitening): . The whitened data stay as close as possible to the original data. Zhu et al. [ [22] apply ZCA-type whitening, with an estimated correlation matrix of the predictors, before a Lasso-type selection, noting that, unlike left preconditioning, the error term is unchanged.
PCA (principal component analysis): , so . Each new column is in general a mix of all original variables.
Cholesky: , so . Computationally this is Gram–Schmidt orthogonalisation: column of is the part of not explained by , normalised to unit length, so the result depends on the ordering of the columns.
A desirable property for selection is that a right transform does not move signal into columns that come later in the ordering than the true variables. The next result shows that this property singles out Cholesky.
Proposition 1. Assume is positive definite, let satisfy , and fix the ordering of the columns. The following are equivalent:
(a) for every and every with , we have , where ;
(b) is upper triangular;
(c) , where is diagonal with entries and is the Cholesky factor of .
Proof. (a)(b): take , whose support lies in . Then , the th column of , is supported on , so for . (b)(a): if is upper triangular, then , which vanishes for whenever for . (b)(c): is invertible because is, so its diagonal entries are non-zero. Let . Then is upper triangular with positive diagonal and , so by uniqueness of the Cholesky factor, and . (c)(b) is immediate. ∎
The equivalence of (b) and (c) is the uniqueness of the Cholesky factor, and Kessy et al. [ [8] characterise Cholesky whitening by the analogous triangularity of the cross-product . Their whitening matrix acts on a column vector, so it corresponds to here; they use the Cholesky factor of the precision matrix, which makes lower triangular and amounts to the reversed ordering. Kelner et al. [ [7] precondition the Lasso on the right with a sparse block Cholesky factorisation, so that, for low-treewidth designs, sparse coefficient vectors stay sparse; condition (a) asks for more, that the support stay within the leading coordinates. Triangularity itself is classical, since Gram–Schmidt orthogonalisation is sequential regression [3 Section 3.2.3]; what we add, to our knowledge, is condition (a), which reads it as a selection property and shows that Cholesky is the only whitening with it. The quantifier “for every ” matters: for a single known , any whitening with block upper triangular at also satisfies (a) for that .
Here, active and inactive refer to the coefficients of the transformed model, not to an estimate; a noise variable with is a correct selection in the transformed model but a false one in terms of the original variables. Thus, if the true variables come first, Cholesky is, up to signs, the only whitening that keeps the noise variables inactive, for , for every number of true variables and every coefficient vector on them. Two limitations remain.
True variables may become weak. Since , a true variable that is well explained by earlier ones has a small , and can even vanish through cancellation. No information is lost, since the transform is invertible; the shared signal is carried by earlier columns.
The ordering must be right. A noise variable placed before a true variable with which it is correlated typically absorbs some of its signal. The requirement that each true variable precede the noise variables correlated with it, which Nouraie et al. [ [12] use for consistent selection, does not keep them inactive. Take true variables and a noise variable in the ordering , with correlations between the columns , , and . For unit-norm columns, the Cholesky factor of the correlation matrix gives , so picks up signal from through and is active. This requirement does ensure when groups of correlated variables are mutually uncorrelated and, within each group, every noise variable is correlated with every true variable.
When has orthonormal columns, the Lasso on is soft-thresholding: minimising gives , with and . So every right whitening thresholds a different linear image of the least-squares estimate, ZCA, PCA and Cholesky differing only in , whereas Puffer, for and full-rank , thresholds itself [14], with correlated noise [5]. Equivalently, a whitened right-transform Lasso is a left transform , which gives with and , followed by a Lasso that penalises instead of . For Cholesky, and , so the left and right versions use the same statistic and differ only in which parametrisation is penalised.
Combining left and right transforms
A method can use one kind of transform for ranking and the other for selection. DVS [12] ranks the variables by Air-HOLP, applies Gram–Schmidt to the columns of in this order, and runs stability selection [10] with the Lasso on ; since Ridge-HOLP is screening after a left transform, it uses a left transform for ranking and a right transform for selection. Two general points apply to any such combination. First, the ordering uses , so the guarantees above hold for a given ordering. Second, the decorrelation is exact only on the full sample: a subsample of rows (index set ), as in stability selection, has under uniform subsampling, with orthogonalised columns, but each realisation deviates from this, and is singular once .
Remark. When , at most columns can be orthogonalised. Once the columns already processed span the column space of , every further Gram–Schmidt residual is zero in exact arithmetic; numerically, the normalised residuals are rounding error, giving meaningless unit-norm columns that the Lasso can still select. Orthogonalising only the top ranked columns, as screening would, avoids this and, for , typically keeps non-singular, provided these columns are linearly independent; the price is that the true variables must rank in the top .
The trade-off
Assume here that is invertible, so that exact decorrelation is possible; this requires ( is centred), so the results below do not cover the regime that motivates screening. A decorrelating transform would ideally achieve all three of the following:
uncorrelated columns: the transformed Gram matrix is diagonal;
unchanged meaning: the coefficients are , up to a fixed diagonal rescaling and relabelling;
unchanged noise: the transformed noise has covariance .
Proposition 2. Let be invertible and not diagonal, and consider a fixed linear transform, that is, one that does not depend on , with and an invertible . If it satisfies (i), (ii) for every , and (iii), then , that is, the data are reduced to fewer than linear combinations of the observations.
Proof. Writing the reduced model as and applying (ii) to gives with , for ; since is invertible, is a permutation, so is a scaled permutation. Condition (iii) means , hence . If , then is orthogonal, so the transformed Gram matrix is , which, being a scaled permutation, is diagonal only if is. Hence (i) fails unless .
To our knowledge, Proposition 2 and Proposition 3 below have not been stated in this form. Any transform in this class that achieves (i) must give up (ii), give up (iii), or reduce the data: Cholesky as a right transform gives (i) and (iii); Puffer gives (i) and (ii), and when , is exactly the ZCA-whitened design, so the two differ only in whether or the noise is kept; Cholesky as a left transform gives (ii) and (iii) with , but not (i). Achieving all three requires to be a scaled permutation and with having orthonormal columns and .
Proposition 3. Let have orthonormal columns and suppose the columns of are orthogonal, with and all . Then the reduced model satisfies (i), (ii) and (iii), and
Proof. , the coefficients are , and the Gram matrix is . Since , we get . Both and are positive definite, so inverting reverses the order, , and comparing diagonal entries gives the bound.
The left-hand side of the bound is the variance of the least-squares estimate of in the reduced model; the right-hand side is that from the full data, the Gauss–Markov bound for linear unbiased estimators. If is not diagonal then , so is non-zero and positive semidefinite, hence has positive trace, and for at least one the variance strictly exceeds the full-data variance: the reduction loses information. Under strong correlation is large for the variables involved, so is small for them. By contrast, the left Cholesky transform reduces the data to dimensions without changing the least-squares estimate or its variance, because it does not decorrelate: the loss comes from (i), not from the reduction itself.
Such a need not exist. With , one construction is , where the columns of are orthogonal to those of and . This needs and ; for example, works whenever . For this , , the full-data least-squares estimate plus extra noise. Under Gaussian errors, this has the same distribution as the whitened estimator of Li and Fithian [ [9], with covariance (they add user-generated noise where we use the residual space): their requirement that a diagonal satisfy is our with , and they show that the price in variance can be very high when the eigenstructure of is unfavourable.
The fixed- knockoff construction [1] supplies such a ; Li and Fithian [ [9] show that the fixed- knockoff filter is equivalent to inference on such a whitened estimator, built from the contrasts [9 eq. (5]; the filter can also use , which, under Gaussian errors, is independent of this estimator, to rank the variables, together with the magnitude of the whitened estimator, so only the estimator it tests with is reduced. With columns normalised to unit length, and , the knockoffs exist when ; when all , the contrasts are mutually orthogonal with squared norms , and taking gives . The constraint is then exactly , so the must be small under strong correlation, one way to see why knockoffs lose power for highly correlated variables. Barber and Candès [ [1] recommend choosing the as large as possible, whereas Spector and Janson [ [15] show for model- knockoffs that pushing to the boundary can itself reduce power, and propose alternatives within the same constraint that are also more powerful in fixed- simulations.
A meaning-preserving alternative. Proposition 2 also shows what remains when (ii) and (iii) are both required without reducing the data: must be a scaled permutation, so the columns can only be rescaled. A weighted Lasso, such as the adaptive Lasso [23], is exactly of this form. With positive penalty factors , minimising is the ordinary Lasso applied to with , whose coefficients keep their meaning; a zero weight leaves that variable unpenalised (the objective then has no such ). The Stable Lasso [13] sets to an increasing function of the Air-HOLP rank of variable (rank 1 being the most relevant), so that, as in DVS, a left transform supplies the ranking. When the weights depend on , the argument applies for given weights. Such methods give up (i), but by the proof of Proposition 2, rescaling, up to a square orthogonal and a reordering of the columns, which leave the Lasso unchanged, is the only option that keeps both the meaning of the coefficients and white noise without reducing the data. Like Puffer, their gains come from changing how the Lasso behaves, not from new information.
Discussion
No transformation of the data, linear or not, can create information about ; under Gaussian errors and with invertible, any unbiased estimate of has variance at least , however the data are transformed. By Propositions 2 and 3, achieving (i)–(iii) together loses information unless the columns of are already orthogonal. This is a statement about information, not about procedures: because the Lasso is nonlinear and ignores the noise covariance, a transform can still change how well it selects, as the Puffer results of Jia and Rohe [ [5] show; conversely, Kelner et al. [ [7] construct Gaussian designs of high treewidth for which no right-preconditioned program recovers sparse vectors from fewer than polynomially many samples in the treewidth. Table 1 summarises which property each method gives up.
| Method | Side | Uncorrelated columns | Keeps | Noise stays white | Needs ordering |
| Puffer, DECO | left | exactly if (Puffer); otherwise not in general | yes | no | no |
| Cholesky ()† | left | no | yes | yes | no |
| Projection ()* | left | exactly | yes | yes | no |
| ZCA, PCA | right | exactly; needs | no | yes | no |
| Cholesky (Gram–Schmidt) | right | exactly on the full sample, for at most columns | no; noise variables stay inactive if true ones come first | yes, for a fixed ordering | yes |
| Weighted Lasso | right (diagonal) | no | yes, up to scaling | yes, for fixed weights | no |
Table 1. Properties of decorrelating transforms. †Reduces the data to dimensions without changing the least-squares estimate or the Lasso path; needs ; used by Nghiem et al. [ [11]. *Reduces the data and loses information; exists, for example, when and ; never, with all , when .
In practice, the choice depends on what the analysis needs. If the goal is to select among the original variables and interpret their coefficients, a method that keeps , such as a weighted Lasso, is the natural choice, and the correlation has to be handled by the selection method itself. If decorrelated columns are worth losing that interpretation, Cholesky is the right transform to use, but its results are only as good as the ordering, and its decorrelation is exact only on the full sample. Methods that keep both and white noise while decorrelating exist only by losing information, as Proposition 3 shows; the whitened estimator behind fixed-X knockoffs is an example.
Funding
Mahdi Nouraie began this work during his PhD, supported by the Macquarie University Research Excellence Scholarship (20213605), and continued it with support from the Australian Research Council Discovery Project Grant (DP260100348), which also supported Samuel Muller.
Data availability
No data were used for the research described in this article.
Declaration of generative AI and AI-assisted technologies in the manuscript preparation process
During the preparation of this work the authors used Claude (Anthropic) to help edit the text and to check the derivations. After using this tool, the authors reviewed and edited the content as needed and take full responsibility for the content of the published article.
References
- [1]Rina Foygel Barber and Emmanuel J. Candès. Controlling the false discovery rate via knockoffs. The Annals of Statistics, 43(5):2055–2085, 2015. doi: 10.1214/15-AOS1337.DOI
- [2]Jianqing Fan and Jinchi Lv. Sure independence screening for ultrahigh dimensional feature space. Journal of the Royal Statistical Society: Series B, 70(5):849–911, 2008. doi: 10.1111/j.1467-9868.2008.00674.x.
- [3]Trevor Hastie, Robert Tibshirani, and Jerome Friedman. The Elements of Statistical Learning: Data Mining, Inference, and Prediction. Springer, New York, 2 edition, 2009.
- [4]Arthur E. Hoerl and Robert W. Kennard. Ridge regression: biased estimation for nonorthogonal problems. Technometrics, 12(1):55–67, 1970. doi: 10.1080/00401706.1970.10488634.DOI
- [5]Jinzhu Jia and Karl Rohe. Preconditioning the Lasso for sign consistency. Electronic Journal of Statistics, 9(1):1150–1172, 2015. doi: 10.1214/15-EJS1029.DOI
- [6]Ibrahim Joudah, Samuel Muller, and Houying Zhu. Air-HOLP: adaptive regularized feature screening for high dimensional correlated data. Statistics and Computing, 35(3):63, 2025. doi: 10.1007/s11222-025-10599-6.DOI
- [7]Jonathan A. Kelner, Frederic Koehler, Raghu Meka, and Dhruv Rohatgi. On the power of preconditioning in sparse linear regression. In 2021 IEEE 62nd Annual Symposium on Foundations of Computer Science (FOCS), pages 550–561, 2021. doi: 10.1109/FOCS52979.2021.00061.DOI
- [8]Agnan Kessy, Alex Lewin, and Korbinian Strimmer. Optimal whitening and decorrelation. The American Statistician, 72(4):309–314, 2018. doi: 10.1080/00031305.2016.1277159.DOI
- [9]Xiao Li and William Fithian. Whiteout: when do fixed-X knockoffs fail?, 2021. arXiv:2107.06388.arxiv.org/abs/2107.06388
- [10]Nicolai Meinshausen and Peter Bühlmann. Stability selection. Journal of the Royal Statistical Society: Series B, 72(4):417–473, 2010. doi: 10.1111/j.1467-9868.2010.00740.x.DOI
- [11]Linh H. Nghiem, Francis K. C. Hui, Samuel Müller, and A. H. Welsh. Sparse sliced inverse regression via Cholesky matrix penalization. Statistica Sinica, 32:2431–2453, 2022. doi: 10.5705/ss.202020.0406.DOI
- [12]Mahdi Nouraie, Connor Smith, and Samuel Muller. Stability selection via variable decorrelation. Statistics and Computing, 36:160, 2026a. doi: 10.1007/s11222-026-10916-7.DOI
- [13]Mahdi Nouraie, Houying Zhu, and Samuel Muller. A stable Lasso, 2026b. arXiv:2511.02306v2.arxiv.org/abs/2511.02306
- [14]Karl Rohe. Preconditioning for classical relationships: a note relating ridge regression and OLS p-values to preconditioned sparse penalized regression. Stat, 4(1):157–166, 2015. doi: 10.1002/sta4.86.DOI
- [15]Asher Spector and Lucas Janson. Powerful knockoffs via minimizing reconstructability. The Annals of Statistics, 50(1):252–276, 2022. doi: 10.1214/21-AOS2104.DOI
- [16]Robert Tibshirani. Regression shrinkage and selection via the lasso. Journal of the Royal Statistical Society: Series B, 58(1):267–288, 1996. doi: 10.1111/j.2517-6161.1996.tb02080.x.DOI
- [17]Xiangyu Wang and Chenlei Leng. High dimensional ordinary least squares projection for screening variables. Journal of the Royal Statistical Society: Series B, 78(3):589–611, 2016. doi: 10.1111/rssb.12127.DOI
- [19]Xiangyu Wang, Chenlei Leng, and Tom Boot. Wang and Leng (2016), High-dimensional ordinary least-squares projection for screening variables, Journal of the Royal Statistical Society Series B, 78, 589–611. Journal of the Royal Statistical Society: Series B, 83(4):880–881, 2021. doi: 10.1111/rssb.12427.
- [21]Peng Zhao and Bin Yu. On model selection consistency of Lasso. Journal of Machine Learning Research, 7:2541–2563, 2006.DOI
- [22]Wencan Zhu, Céline Lévy-Leduc, and Nils Ternès. A variable selection approach for highly correlated predictors in high-dimensional genomic data. Bioinformatics, 37(16):2238–2244, 2021. doi: 10.1093/bioinformatics/btab114.DOI
- [23]Hui Zou. The adaptive lasso and its oracle properties. Journal of the American Statistical Association, 101(476):1418–1429, 2006. doi: 10.1198/016214506000000735.DOI