In this paper
AbstractIntroductionLow-Complexity Precision Matrix ModelProblem statementModel of the nuisance componentMaximum likelihood nuisance statisticsBuilding the sparse structure and learning its coefficientsDetection of Faint ObjectsMaximum likelihood brightness and uncertaintyDetection criterionUnsupervised Model Complexity SelectionResults on Astronomical DataConsidered modality: direct imaging at high-contrastUnsupervised setting and reliability of the nuisance modelPerformance assessmentConclusions and PerspectivesReferencesWorksPhysics and Astronomy
Learning a model of non-stationary nuisances with non-local correlations for object detection
Abstract
We tackle the problem of detecting faint objects in multivariate data corrupted by a strong and spatially correlated nuisance component in the low-sample regime. Standard covariance-based models often yield unstable or ill-conditioned estimators of the nuisance statistics, severely degrading detection performance. We propose a data-driven approach based on sparse precision matrix modeling, where the precision matrix is approximated through a pivoted sparse Cholesky factorization whose non-zeros coefficients are learned by maximum likelihood. This yields a low-complexity yet expressive statistical model of the nuisance. Building on this model, we derive a generalized likelihood ratio test for faint source detection enabling direct control of the false alarm rate. The approach is evaluated on an astrophysical application: the detection of exoplanets by high contrast imaging, where it achieves state-of-the-art performance.
Introduction
Detecting faint objects in multivariate data corrupted by a structured nuisance component is a central problem in various applications, including biomedical imaging, remote sensing, and astronomy. Many detection methods rely on second-order statistics to model the nuisance, but in practice the number of available samples is often limited. This low-sample, high-dimensional regime leads to unstable or singular covariance estimators, severely degrading maximum likelihood-based inference and detection performance. Achieving a bias–variance trade-off is thus a key challenge in such inverse problems exploiting measurement statistics.
A possible solution is to calibrate the nuisance statistics using additional data independent of the observations of interest. However, this is often impractical, as the statistics can vary significantly across datasets (e.g., due to changing conditions of observation over time in radar imaging or astronomy). Self-calibrating approaches address this challenge through covariance regularization. These include diagonal loading and shrinkage methods [1], [2], which stabilize the estimator by combining the sample covariance with another estimator of lower variance, as well as approaches that impose predefined covariance structures, such as block-diagonal or banded models, when long-range correlations are assumed negligible [3]. Hybrid strategies combining structural assumptions and nonlinear shrinkage have also been proposed [4]. While these methods improve numerical stability, they rely by construction on strong prior assumptions about the covariance structure. A more flexible alternative consists in directly modeling the inverse covariance matrix, or precision matrix, which often admits sparse representations [5]. In particular, sparse Cholesky factorizations provide a structured and computationally efficient way to approximate precision matrices while guaranteeing positive semidefiniteness [6]. However, defining the sparse structure, learning the corresponding non-zero coefficients in a fully data-driven manner, and integrating such models into an optimal detection framework remain challenging.
Our contributions. We propose a low-complexity, data-driven detection framework based on sparse precision matrix modeling of the nuisance component. The precision matrix is approximated by a pivoted sparse Cholesky factorization learned from the data by maximum likelihood, enabling stable estimation in the low-sample regime while capturing long-range (spatial) correlations (Sect. II). Building on this model, we derive a detection test with a constant false alarm rate (Sect. III). The model complexity is selected automatically via cross-validation (Sect. IV). We demonstrate the effectiveness of the proposed approach for exoplanet detection (Sect. V).
Low-Complexity Precision Matrix Model
Problem statement
We consider the general case of observations, where the -th data sample follows the forward model:
where denotes a known (possibly nonlinear) mapping of the unknown parameters , and is a nuisance component with . The mapping depends on the considered task (object detection, image reconstruction, etc.). We also assume , thus defining a low-sample regime.
Model of the nuisance component
We assume that the nuisance terms are random variables that are mutually independent (for ) but follow identical Gaussian distribution:
with the expectation of the nuisance terms and their precision matrix. We denote by the nuisance statistics. Under forward model (1), the probability density function of the -th nuisance term is given by:
where is the determinant of , and is the squared Mahalanobis norm. Under these assumptions, the co-log-likelihood of the data sequence given and writes:
up to an irrelevant additive constant. We address in Sects. II-C-III how to accurately estimate all the unknown parameters, and , from the finite set of data samples.
Maximum likelihood nuisance statistics
Under the models of Sects. II-A-II-B, the maximum likelihood estimators of the statistics knowing reduce to the sample estimators:
with the residuals . Since the rank2 of is at most , a consistent covariance estimate requires . In the considered low-sample regime (), is therefore singular, preventing direct estimation of the precision matrix by inversion of . To cope with this issue, we propose to use a matrix with a lower complexity such as the matrices of A family of Sparse Approximations for whitening and Preconditioning (ASAP, [7]) as an estimator of the precision matrix. We thus approximate by:
where is the composition of a sparse lower triangular matrix and of a permutation matrix , with . The symmetric and non-negative semi-definite precision matrix defined in Eq. (6) can be seen as a pivoted sparse Cholesky factorization [5].
Building the sparse structure and learning its coefficients
We decompose the construction of the precision matrix estimator in (6) into two steps: (i) definition of the sparse structure, and (ii) estimation of the associated coefficients.
Construction of the sparse structure: The structure of is defined by (i) the ordering of the nodes (e.g., the pixels) and (ii), for each node, the set of connected preceding nodes. The ordering is specified by the permutation matrix , while the connectivity is defined by:
where denotes the set of nodes connected to node , and is the sparsity pattern of , that is the set of index pairs of the structural non-zeros of . The model complexity is defined as the mean number of structural non-zeros per row of and thus set the sparsity level of the model statistics. There is a variety of choices to define an ordered sparse structure that differ by the sampling considered. A first solution is to assume (i.e., no permutation) and to select, in lexicographic order, for each node , the previous nodes that are closest to the node :
where is a distance (e.g., Euclidean) between nodes and , and controls the sparsity that is the number of nodes connected to the -th one. A finer strategy uses an ordered random sparse structure based on a random permutation and in which a node is simply connected to the nearest preceding nodes. A related multi-scale structure provides a more homogeneous spatial coverage by constructing the permutation so that the density of the first nodes is approximately uniform whatever .
Learning the sparse coefficients: For a fixed sparsity pattern , the matrix can be learned from the sample covariance (5) in the maximum likelihood sense as:
where denotes the set of pivoted sparse Cholesky factors with given permutation and sparsity pattern . Problem (10) can also be interpreted as the minimization of the Kullback–Leibler divergence between two centered Gaussian distributions, one with covariance and the other with covariance . Given the sparse structure , Problem (10) has a closed-form solution for the structural non-zero of [7]:
where denotes the sample covariance of the permuted residuals, and its restriction to the set of column indices of the structural non-zeros of the -th row of the sparse factor (also corresponding to the nodes directly connected to node in the sparse structure). Equation (11) assumes that index is the last entry of , hence, at position in this set of node indices. The sign ambiguity in (11) does not affect the resulting approximation of in (6).
Detection of Faint Objects
The statistical model of the nuisance introduced in Sect. II-C is general and independent of the specific form of the forward model in Eq. (1). We now specialize this model to the detection of objects parametrized by , with the object brightness and its 2-dimensional position. In this setting, the forward model writes with the (known) footprint in the image of an object at position .
Maximum likelihood brightness and uncertainty
The maximum likelihood estimator of the object brightness knowing its position and the statistics is:
with:
The statistic corresponds to a cross-correlation term, while is a normalization factor. They provide sufficient statistics for the estimation problem. Indeed, under the Gaussian assumption with precision matrix , . It follows that the estimator of the flux standard-deviation is:
Detection criterion
Assuming that the nuisance statistics are known, detection is performed by the generalized likelihood ratio test (GLRT):
Under the null hypothesis and the assumed multivariate Gaussian nuisance model, follows a standard normal distribution , independently of . A single threshold thus ensures a constant false alarm rate (CFAR) detector and yields a natural interpretation of the detection criterion as a signal-to-noise ratio (SNR). Detection is performed by exhaustively evaluating (16) over a discretized (possibly subpixel) grid of candidate positions. Since the sought sources are usually faint, nuisance statistics are estimated under by assuming . After detection, the source flux and position are locally refined by jointly re-estimating the nuisance statistics and source parameters through a fixed-point iterative scheme, alternating closed-form flux updates (12) and continuous position optimization maximizing (16).
Unsupervised Model Complexity Selection
The performance of the proposed detector critically depends on the nuisance model complexity , which governs a bias–variance trade-off. If is too small, the model fails to capture correlations in the data. If is too large, the model may overfit the finite sample set and partially absorb the signal of interest into the estimated nuisance statistics . To set in an unsupervised manner, we assess how well the nuisance model predicts unseen data rather than how well it fits the training samples. We thus select by maximizing a leave-one-out cross-validated predictive log-likelihood (LOOCV), defined as the sum of the co-log-likelihoods of each sample evaluated under a model learned from all remaining samples.
where are the nuisance statistics learned with model complexity from the training set , i.e., excluding sample . The LOOCV criterion therefore quantifies how well a model of complexity generalizes to unseen data. To avoid contamination by real (bright) sources, must be computed in an unbiased manner: pixels potentially affected by sources are masked when learning and/or evaluating the nuisance statistics . In practice, a preliminary detection map is computed using a low-complexity model ( or ) and thresholded at a high level to conservatively identify and mask brighter sources, which are then treated as invalid data in the LOOCV procedure. The optimal model complexity and the corresponding best sparse structure (lexicographic, random, or multiscale) are finally selected as the minimizers of the LOOCV criterion.
Results on Astronomical Data
Considered modality: direct imaging at high-contrast
We consider high-contrast imaging [8], which aims at detecting exoplanets appearing as unresolved point sources around an observed star. The data were acquired with the SPHERE instrument at the Very Large Telescope, combining adaptive optics and a coronagraph mask to suppress part of the starlight. This yields spatio-temporal data cubes with typically frames of pixels. Detection is challenging due to a strong, spatially correlated nuisance caused by residual stellar leakage and optical aberrations, with exoplanet-to-star flux ratio (i.e., contrast) of order to before optical correction and still below in the recorded images. This nuisance is both non-stationary and correlated at multiple spatial scales (local speckles + extended structures). During the acquisition, astrophysical sources follow a known rotation induced by Earth’s rotation, while the nuisance remains quasi-static. In the forward model, thus accounts for this rotation and represents the spatial signature in the -th frame of a source located at the 2D sky position .
State-of-the-art detection methods exploit this signal diversity by learning a nuisance model directly from the data cube . Baseline approaches [9] rely on empirical modeling, such as image combination (cADI) or low-rank approximations via truncated PCA (KLIP), assuming a quasi-static nuisance across exposures. However, as shown in Figure 1(top), the nuisance is not strictly stationary across exposures: subtracting the empirical mean leaves strong correlated residuals. More advanced methods, such as PACO [10], use matched filtering and explicitly model these correlations at a local (patch) scale under a Gaussian assumption, approximating the nuisance covariance by block-diagonal structures regularized by shrinkage. The hybrid deep PACO [11] approach further combines this statistical model with a deep-learning detector trained in a supervised way.

Figure 1. Top row: examples of real residual samples. Other rows: examples of synthetic samples generated from the learned nuisance model. Optimal model complexity has been used for rows (c–e).
Unsupervised setting and reliability of the nuisance model
We assess the relevance of the proposed unsupervised and unbiased LOOCV criterion for selecting the nuisance model complexity by comparing the resulting estimate with a supervised reference . The latter is obtained by maximizing the area under the ROC curve (AUC), which summarizes the precision–recall trade-off via the true positive rate (TPR) and false discovery rate (FDR). The AUC is computed from extensive simulations involving the injection of synthetic sources into real data using model (1). These simulations
are realistic, as the nuisance component comes from real observations and the PSF model is accurately calibrated from instrumental data. Figure 2 shows on 13 datasets that agrees well with when LOOCV is implemented in an unbiased manner (see Sect. IV). The LOOCV criterion also enables comparison of sparse structures, with the multi-scale structure generally performing best. In the following, both the sparse structure and the model complexity are selected fully unsupervised using LOOCV.

Figure 2. Unsupervised model complexity selection using LOOCV. (a) LOOCV criterion without (left) and with (right) unbiasing, for different structures of the sparse precision matrix factor . (b) Comparison between and ; error bars indicate model complexities yielding detection performance (AUC) within 10% of the optimum.
Beyond model selection, we qualitatively assess the expressivity of the proposed Gaussian nuisance model. Using nuisance statistics learned from real data, Figure 1 compares synthetic residual samples generated by correlating i.i.d. Gaussian vectors with the learned sparse precision matrix factor . Ignoring correlations, or modeling them only locally (local covariance or with the lexicographic structure of the sparse precision matrix factor), fails to reproduce the large-scale structures observed in real data. In contrast, the random and multi-scale structures of the sparse precision matrix factor generate very realistic samples, demonstrating the ability of the proposed model to capture both local and long-range correlations.
Performance assessment
We assess the detection performance of the proposed approach by comparing it with the state-of-the-art methods described in Sect. V-A on 13 datasets acquired under various instrumental configurations and observing conditions. Figure 3 shows representative detection maps, while Table I reports quantitative results in terms of average AUC and TPR at a low FDR (0.005), a regime of particular interest in practice. Metrics are computed on datasets processed by all algorithms for both real sources (72 in total) and synthetic sources generated as described in Sect. V-B, allowing evaluation over a wide range of fluxes and positions. The proposed method is highly competitive with respect to state-of-the-art approaches, and often achieves the best performance in the low-FDR regime which is of particular interest for astronomers.

Figure 3. Detection maps on real data. Top: 200 200 pixel region centered on the star. Bottom: full field of view for dataset #2, shown for the three best-performing algorithms. The detection threshold is set to for cADI, PCA, PACO, and the proposed method (SNR-based maps), and to for deep PACO (pseudo-probability map), corresponding to comparable targeted FDRs. deep PACO could not be applied to dataset #12 acquired in a spectral mode inducing highly structured nuisance patterns making the training procedure unstable.
| Metric / Algo. | cADI | PCA | PACO | deep PACO | proposed |
| Real sources | |||||
| AUC | 0.64 | 0.60 | 0.81 | 0.86 | 0.83 |
| TPR (FDR=0.005) | 0.53 | 0.50 | 0.75 | 0.75 | 0.78 |
| Simulated synthetic sources in real nuisance data | |||||
| AUC | 0.59 | 0.55 | 0.74 | 0.90 | 0.89 |
| TPR (FDR=0.005) | 0.52 | 0.49 | 0.72 | 0.77 | 0.86 |
Table 1. Detection metrics on a 200 200 pixel ROI centered on the star. Best scores are shown in bold, second best underlined.
Using the same set of simulated sources, Figure 4 evaluates flux estimation accuracy by comparing to the ground-truth as a function of separation. The estimates are unbiased and in excellent agreement for sources detected at SNR . At lower detection confidence, the estimation degrades, particularly close to the star, where stronger stellar residuals

Figure 4. Estimation metric: color-coded ratio as a function of the groundtruth source flux and of the distance to the star. Overlaid curves indicate sensitivity limits at , 4, and 3 (solid, dashed, and dotted). Sources above these contours are detected with equal or higher confidence.
and reduced signal diversity induced by the limited apparent rotation of the sources make the unmixing task more difficult.
Conclusions and Perspectives
We proposed a data-driven approach to model correlated multivariate data in the low-sample regime. A low-complexity model is learned through a sparse decomposition of the precision matrix, with the model complexity selected via an unsupervised procedure. Integrating this statistical model into a detection framework yields a CFAR detector achieving state-of-the-art performance, particularly at low FDR. Future work will extend this approach to other imaging modalities and tasks, such as multi-spectral high-contrast imaging.
References
- [1]Y. Chen, A. Wiesel, Y.C. Eldar, and A.O. Hero, “Shrinkage algorithms for MMSE covariance estimation,” IEEE TSP, 2010.DOI
- [2]E. Ollila and E. Raninen, “Optimal shrinkage covariance matrix estimation under random sampling from elliptical distributions,” IEEE TSP, 2019.DOI
- [3]P. J. Bickel and E. Levina, “Regularized estimation of large covariance matrices,” 2008.
- [4]O. Ledoit and M. Wolf, “Analytical nonlinear shrinkage of large-dimensional covariance matrices,” The Annals of Statistics, vol. 48, no. 5, pp. 3043–3065, 2020.
- [5]N. J. Higham, “Cholesky factorization,” Wiley interdisciplinary reviews: computational statistics, vol. 1, no. 2, pp. 251–254, 2009.DOI
- [6]F. Schafer, M. Katzfuss, and H. Owhadi, “Sparse Cholesky factorization by Kullback–Leibler minimization,” SIAM ISC, 2021.arxiv.org/abs/2004.14455
- [7]E. Thiébaut, M. Tallon, L. Denis et al., “Beyond FRiM, ASAP: a family of sparse approximation for covariance matrices and preconditioners,” in Adaptive Optics Systems VIII, SPIE, 2022.DOI
- [8]L. Pueyo, “Direct imaging as a detection technique for exoplanets,” Handbook of Exoplanets, 2018.
- [9]C.A. Gomez Gonzalez, O. Wertz, et al., “VIP: Vortex image processing package for high-contrast direct imaging,” AJ, 2017.
- [10]O. Flasseur, L. Denis, É. Thiébaut, and M. Langlois, “Exoplanet detection in angular differential imaging by statistical learning of the nonstationarity patch covariances - The PACO algorithm,” A&A, 2018.DOI
- [11]O. Flasseur, T. Bodrito, J. Mairal et al., “deep PACO: Combining statistical models with deep learning for exoplanet detection and characterization in direct imaging at high contrast,” MNRAS, 2024.arxiv.org/abs/2303.02461