Introduction
Multivariate count time series arise naturally in many biological applications, such as species abundances measured over time and longitudinal gene expression counts from measurements over time. A central goal is to recover an underlying network structure that captures interactions among system components (species in an environment, genes in a cell). However, unlike continuous-valued time series data, which have a well-developed modeling literature [21 ] , multivariate count time series have a less unified modeling framework. Their analysis is often complicated by over-dispersion in addition to the dependence among components and the serial dependence over time inherent to multivariate time series. Related work includes network inference for categorical time series [33 ] and for Poisson and Bernoulli autoregressive models [16 ] . Existing work provides several approaches to the problem, reviewed in Davis et al. [11 ] . Early work developed integer-valued autoregressive models for univariate counts [23 , 2 ] , later extended to the multivariate setting by Latour [19 ] . However, these models do not admit an intuitive interpretation for network inference and are limited in their ability to capture general autocorrelation structures. Recent work addressed some of these limitations. For example, Yu and Zheng (2024) allows for flexible correlation in the noise component, but still does not allow one component to depend on past values of another. Fokianos et al. [14 ] developed multivariate count autoregressive models in which past observed counts directly influence future conditional means, with contemporaneous dependence captured through a copula. We instead treat counts as noisy observations of an underlying process whose dynamics define the target network. This separates temporal interactions in the latent log-rates from Poisson sampling variation.
To address the limitations of existing approaches, we adopt a multivariate Poisson log-normal (PLN) model [1 ] , in which observed counts follow a Poisson distribution with rates driven by a latent Gaussian vector autoregressive (VAR) process. This model naturally accommodates count-valued data, over-dispersion, and dependence both within and across time points. Moreover, the transition matrix of the latent VAR process encodes directed temporal dependencies among components corresponding to Granger causal interactions [29 ] , so its support can be interpreted as the underlying network structure.
Our proposed model is most closely related to Zhang et al. [40 ] , who modeled multivariate Poisson counts using latent Gaussian VAR dynamics and estimated the parameters by Monte Carlo EM, with numerical studies primarily in low-dimensional settings. We focus on sparse recovery of the directed network encoded by the latent transition matrix, developing computationally scalable estimators with support-recovery guarantees. Our framework also accommodates covariates, multiple independent trajectories of unequal length, and ℓ 1 \ell_1 ℓ 1 -penalized estimation for sparse recovery. For network inference, our approach is closer to Chiquet et al. [8 ] , who used a PLN model to infer an undirected network via sparsity in the latent precision matrix. Recent extensions of the PLN framework include classification, clustering, and dimension reduction (see Chiquet et al. [7 ] for an overview), as well as tree-structured PLN models for hierarchical count data [6 ] . Saha et al. [27 ] introduced the Poisson log-normal process, which extends the PLN framework by placing a Gaussian process prior on the Poisson log-rates for count prediction.
Many existing PLN methods rely on variational inference (VI), estimating model parameters by maximizing the evidence lower bound (ELBO) [5 ] . More recent work has proposed likelihood-based alternatives, including maximum marginal likelihood estimation [18 ] , motivated by composite likelihood ideas [10 , 35 ] . Relatedly, Stoehr and Robin [31 ] combine composite likelihood with importance sampling. Full likelihood-based estimation requires integration over the latent trajectories, making computation challenging as the number of components and time points increases. Marginal likelihood approaches, such as Huang et al. [18 ] , reduce this burden by using low-dimensional marginal distributions, but still require numerical integration and optimization.
We propose a method-of-moments (MoM) estimator as well as its l 1 l_{1} l 1 -penalized version to encourage sparsity in high dimensions. Our penalized MoM estimator is related to Düker et al. [13 ] , who studied sparse latent VAR estimation for count time series. Their model generates counts by thresholding latent Gaussian variables, whereas ours uses a Poisson observation model conditional on the latent Gaussian states. Our proposed estimator scales well in high-dimensional settings and has theoretical guarantees on recovering the support of the network of interest under usual conditions for sparse estimation. We also develop a VI estimator and penalized VI estimator to compare our method to the prevailing estimation strategy used by PLN models in the literature.
The main methodological contributions of this work are twofold. First, we formulate network recovery for multivariate count time series under a dynamic PLN model in which the support of the latent VAR transition matrix defines the target directed network. Second, we develop an estimation procedure for sparse network recovery along with theoretical guarantees. We illustrate the practical utility of our method using macroinvertebrate species abundance data from Danish freshwater environments in the TREAM (Time seRies of European freshwAter Macroinvertebrates) dataset [37 ] .
Methodology
Model and network target
We consider n n n independent realizations of a multivariate count time series, where i i i indexes the realization and the timepoints for each realization are indexed along a grid of equally spaced timepoints 1 , 2 , … , T i 1, 2, \ldots, T_i 1 , 2 , … , T i , where T i T_i T i is the latest timepoint at which the realization is observed. At time t t t , let Y t ( i ) ∈ Z ≥ 0 d Y_t^{(i)} \in\mathbb{Z}_{\geq0}^d Y t ( i ) ∈ Z ≥ 0 d , denote the vector of counts for d d d components, X t ( i ) ∈ R p X_t^{(i)} \in\mathbb{R}^p X t ( i ) ∈ R p , the vector of observed covariates (excluding an intercept term), and O t ( i ) ∈ R O_t^{(i)} \in\mathbb{R} O t ( i ) ∈ R an offset. Let Z t ( i ) ∈ R d Z_t^{(i)} \in\mathbb{R}^d Z t ( i ) ∈ R d denote the latent state for sample i i i at time t t t and let u t ( i ) u_t^{(i)} u t ( i ) denote the corresponding noise term. Further, let Σ ∈ R d × d \boldsymbol{\Sigma} \in\mathbb{R}^{d \times d} Σ ∈ R d × d be a positive definite covariance matrix, A ∈ R d × d A \in\mathbb{R}^{d \times d} A ∈ R d × d the latent transition matrix, β 0 ∈ R d \beta_0 \in\mathbb{R}^d β 0 ∈ R d the intercept vector, and β ∈ R p × d \beta\in\mathbb{R}^{p \times d} β ∈ R p × d the matrix of regression coefficients. Since higher-order VAR processes can be written in “companion form” as VAR(1) processes, for simplicity, we focus on a latent VAR(1) model [21 ] . Specifically, we assume
Z t ( i ) = A Z t − 1 ( i ) + u t ( i ) , u t ( i ) ∼ i . i . d . N ( 0 , Σ ) (1) Z_t^{(i)} = A Z_{t-1}^{(i)} + u_t^{(i)}, \qquad u_t^{(i)} \mathrel{\mathop{\sim}^{\mathrm{i.i.d.}}} N(0,\boldsymbol{\Sigma}) \tag*{(1)} Z t ( i ) = A Z t − 1 ( i ) + u t ( i ) , u t ( i ) ∼ i.i.d. N ( 0 , Σ ) (1)
Y t j ( i ) ∣ Z t j ( i ) ∼ Pois. { exp ( O t ( i ) + β 0 j + X t ( i ) ⊤ β j + Z t j ( i ) ) } (2) Y_{tj}^{(i)} \mid Z_{tj}^{(i)} \sim\operatorname{Pois.}\left\{\exp\left(O_t^{(i)} + \beta_{0j} + X_t^{(i)\top}\beta_j + Z_{tj}^{(i)}\right)\right\} \tag*{(2)} Y t j ( i ) ∣ Z t j ( i ) ∼ Pois. { exp ( O t ( i ) + β 0 j + X t ( i ) ⊤ β j + Z t j ( i ) ) } (2)
We additionally assume that, conditional on the latent trajectory, covariates, and offsets, the counts are independent across time points and components. Suppressing the conditioning on covariates and offsets, the joint distribution of the latent states and observed counts factorizes as
p ( y 1 : T , z 1 : T ) = p ( z 1 ) ∏ t = 2 T p ( z t ∣ z t − 1 ) ∏ t = 1 T ∏ j = 1 d p ( y t j ∣ z t j ) . p(y_{1:T}, z_{1:T}) = p(z_1) \prod_{t=2}^{T} p(z_t \mid z_{t-1}) \prod_{t=1}^{T} \prod_{j=1}^{d} p(y_{tj} \mid z_{tj}). p ( y 1 : T , z 1 : T ) = p ( z 1 ) t = 2 ∏ T p ( z t ∣ z t − 1 ) t = 1 ∏ T j = 1 ∏ d p ( y t j ∣ z t j ) .
We assume that the n n n trajectories are independent and follow this model. Further, we assume all eigenvalues of A A A have modulus less than 1 and that the latent process is stationary. For each trajectory, the latent process is independent of the covariates and offsets. Letting Σ Z ∈ R d × d \boldsymbol{\Sigma}_Z \in\mathbb{R}^{d \times d} Σ Z ∈ R d × d denote the stationary covariance matrix of Z t Z_t Z t , we have Σ Z = A Σ Z A ⊤ + Σ \boldsymbol{\Sigma}_Z = A\boldsymbol{\Sigma}_Z A^\top+ \boldsymbol{\Sigma} Σ Z = A Σ Z A ⊤ + Σ [21 ] .
Our target of inference is the network of Granger-causal interactions among the components of the latent process Z t Z_t Z t , as defined in Shojaie and Fox [29 ] . In the proposed model, A j k A_{jk} A j k represents the lagged effect of latent component k k k on latent component j j j , and A j k ≠ 0 A_{jk} \ne0 A j k = 0 corresponds to a directed edge from component k k k to component j j j .
Identifiability
Under conditions on the design matrix of observed covariates and the covariance of the latent VAR process, the proposed model is identifiable.
Lemma 1. Let N t o t : = ∑ i = 1 n T i N_{\mathrm{tot}} := \sum_{i=1}^{n} T_i N tot := ∑ i = 1 n T i , and let X ∈ R N t o t × p \mathbf{X} \in\mathbb{R}^{N_{\mathrm{tot}} \times p} X ∈ R N tot × p denote the matrix of observed covariates obtained by stacking X t ( i ) \mathbf{X}_{t}^{(i)} X t ( i ) over all samples and time points. If the augmented design X ‾ = [ 1 , X ] \overline{\mathbf{X}} = [\mathbf{1}, \mathbf{X}] X = [ 1 , X ] has full column rank, and Σ Z \boldsymbol{\Sigma}_{Z} Σ Z is invertible, then the model is identifiable.
Proof. The proof is given in Appendix A.
We next describe the two proposed estimation procedures for the model parameters. We start with the MoM estimator and then discuss the alternative VI estimator.
Method of Moments Estimator
We develop a method-of-moments estimator for the model parameters θ : = { A , β 0 , β , Σ } \theta:= \{A, \beta_{0}, \beta, \Sigma\} θ := { A , β 0 , β , Σ } , with primary interest in A A A , whose support determines the directed network.
We first define some notation. Let ∥ ⋅ ∥ 1 \lVert\cdot\rVert_{1} ∥ ⋅ ∥ 1 , ∥ ⋅ ∥ 2 \lVert\cdot\rVert_{2} ∥ ⋅ ∥ 2 , and ∥ ⋅ ∥ ∞ \lVert\cdot\rVert_{\infty} ∥ ⋅ ∥ ∞ denote the usual ℓ 1 \ell_{1} ℓ 1 , Euclidean, and maximum norms of a vector. For a matrix M M M ,
∥ M ∥ max = max j , k ∣ M j k ∣ , ∥ M ∥ 1 = max k ∑ j ∣ M j k ∣ , ∥ M ∥ ∞ = max j ∑ k ∣ M j k ∣ . \lVert M \rVert_{\max} = \max_{j,k} |M_{jk}|,\qquad\lVert M \rVert_{1} = \max_{k} \sum_{j} |M_{jk}|,\qquad\lVert M \rVert_{\infty} = \max_{j} \sum_{k} |M_{jk}|. ∥ M ∥ m a x = j , k max ∣ M j k ∣ , ∥ M ∥ 1 = k max j ∑ ∣ M j k ∣ , ∥ M ∥ ∞ = j max k ∑ ∣ M j k ∣.
We write ∥ M ∥ 2 \lVert M \rVert_{2} ∥ M ∥ 2 for its spectral norm, and ∥ M ∥ F \lVert M \rVert_{F} ∥ M ∥ F for its Frobenius norm.
We begin by estimating β \beta β . Let γ ∈ R ( p + 1 ) × d \gamma\in\mathbb{R}^{(p+1)\times d} γ ∈ R ( p + 1 ) × d be defined columnwise by γ j : = [ β 0 j + 1 2 Σ Z , j j , β 1 j , … , β p j ] ⊤ ∈ R p + 1 \gamma_{j} := [\beta_{0j} + \frac{1}{2}\Sigma_{Z,jj}, \beta_{1j}, \ldots, \beta_{pj}]^{\top} \in\mathbb{R}^{p+1} γ j := [ β 0 j + 2 1 Σ Z , j j , β 1 j , … , β p j ] ⊤ ∈ R p + 1 , and index the rows of γ \gamma γ from 0 to p p p . Since γ 1 : p = β \gamma_{1:p} = \beta γ 1 : p = β , estimating γ \gamma γ yields an estimator of β \beta β . Moreover, E [ Y t j ( i ) ∣ X t ( i ) , O t ( i ) ] = exp { O t ( i ) + γ 0 j + X t ( i ) ⊤ γ 1 : p , j } \mathbb{E}[Y_{tj}^{(i)} \mid X_{t}^{(i)}, O_{t}^{(i)}] = \exp\{O_{t}^{(i)} + \gamma_{0j} + X_{t}^{(i)\top}\gamma_{1:p,j}\} E [ Y t j ( i ) ∣ X t ( i ) , O t ( i ) ] = exp { O t ( i ) + γ 0 j + X t ( i ) ⊤ γ 1 : p , j } , so γ \gamma γ can be estimated from the first moment by fitting a Poisson regression. Concretely, we define the objective L j ( g ) L_j(g) L j ( g ) for g = [ g 0 g 1 : p ] ⊤ ∈ R p + 1 g = [g_0\ g_{1:p}]^{\top} \in\mathbb{R}^{p+1} g = [ g 0 g 1 : p ] ⊤ ∈ R p + 1 as
L j ( g ) = − 1 N t o t ∑ i = 1 n ∑ t = 1 T i { Y t j ( i ) ( O t ( i ) + g 0 + X t ( i ) ⊤ g 1 : p ) − exp ( O t ( i ) + g 0 + X t ( i ) ⊤ g 1 : p ) } , L_j(g) = -\frac{1}{N_{\mathrm{tot}}}\sum_{i=1}^{n}\sum_{t=1}^{T_i}\left\{Y_{tj}^{(i)}\left(O_t^{(i)} + g_0 + X_t^{(i)\top}g_{1:p}\right) - \exp\left(O_t^{(i)} + g_0 + X_t^{(i)\top}g_{1:p}\right)\right\}, L j ( g ) = − N tot 1 i = 1 ∑ n t = 1 ∑ T i { Y t j ( i ) ( O t ( i ) + g 0 + X t ( i ) ⊤ g 1 : p ) − exp ( O t ( i ) + g 0 + X t ( i ) ⊤ g 1 : p ) } ,
and obtain estimates for γ j \gamma_j γ j (j = 1 , … , d j = 1, \ldots, d j = 1 , … , d ) as γ ^ j = arg min g L j ( g ) \hat{\gamma}_j = \arg\min_g L_j(g) γ ^ j = arg min g L j ( g ) .
It remains to estimate β 0 \beta_0 β 0 . By the definition of γ \gamma γ , we have β ^ 0 = γ ^ 0 − 1 2 diag ( Σ ^ Z ) \hat{\beta}_0 = \hat{\gamma}_0 - \frac{1}{2}\operatorname{diag}(\hat{\Sigma}_Z) β ^ 0 = γ ^ 0 − 2 1 diag ( Σ ^ Z ) , where diag ( ⋅ ) \operatorname{diag}(\cdot) diag ( ⋅ ) returns the diagonal of a matrix as a vector. We therefore estimate Σ Z \Sigma_Z Σ Z from the second moments E [ Y t j ( i ) Y t k ( i ) ∣ X t ( i ) , O t ( i ) ] \mathbb{E}\left[Y_{tj}^{(i)}Y_{tk}^{(i)} \mid X_t^{(i)}, O_t^{(i)}\right] E [ Y t j ( i ) Y t k ( i ) ∣ X t ( i ) , O t ( i ) ] , for all j , k = 1 , … , d j,k = 1,\ldots,d j , k = 1 , … , d . The explicit expressions for estimating Σ Z \Sigma_Z Σ Z are
exp { Σ ^ Z , j j } = 1 ∑ i = 1 n T i ∑ i = 1 n ∑ t = 1 T i Y t j ( i ) 2 − Y t j ( i ) exp { 2 ( O t ( i ) + γ ^ 0 j + X t ( i ) T γ ^ 1 : p , j ) } , \exp\left\{\hat{\Sigma}_{Z,jj}\right\} = \frac{1}{\sum_{i=1}^{n} T_i}\sum_{i=1}^{n}\sum_{t=1}^{T_i}\frac{Y_{tj}^{(i)2}-Y_{tj}^{(i)}}{\exp\left\{2\left(O_t^{(i)}+\hat{\gamma}_{0j}+X_t^{(i)\mathsf{T}}\hat{\gamma}_{1:p,j}\right)\right\}}, exp { Σ ^ Z , j j } = ∑ i = 1 n T i 1 i = 1 ∑ n t = 1 ∑ T i exp { 2 ( O t ( i ) + γ ^ 0 j + X t ( i ) T γ ^ 1 : p , j ) } Y t j ( i ) 2 − Y t j ( i ) ,
exp { Σ ^ Z , j k } = 1 ∑ i = 1 n T i ∑ i = 1 n ∑ t = 1 T i Y t j ( i ) Y t k ( i ) exp { 2 O t ( i ) + γ ^ 0 j + γ ^ 0 k + X t ( i ) T ( γ ^ 1 : p , j + γ ^ 1 : p , k ) } . \exp\left\{\hat{\Sigma}_{Z,jk}\right\} = \frac{1}{\sum_{i=1}^{n} T_i}\sum_{i=1}^{n}\sum_{t=1}^{T_i}\frac{Y_{tj}^{(i)}Y_{tk}^{(i)}}{\exp\left\{2O_t^{(i)}+\hat{\gamma}_{0j}+\hat{\gamma}_{0k}+X_t^{(i)\mathsf{T}}\left(\hat{\gamma}_{1:p,j}+\hat{\gamma}_{1:p,k}\right)\right\}}. exp { Σ ^ Z , j k } = ∑ i = 1 n T i 1 i = 1 ∑ n t = 1 ∑ T i exp { 2 O t ( i ) + γ ^ 0 j + γ ^ 0 k + X t ( i ) T ( γ ^ 1 : p , j + γ ^ 1 : p , k ) } Y t j ( i ) Y t k ( i ) .
Having estimated β \beta β , we proceed to estimating A A A . Let P ∈ R d × d P \in\mathbb{R}^{d\times d} P ∈ R d × d be a matrix that captures the lag-one cross-moments,
P j k : = log ( E [ Y t j ( i ) Y t − 1 , k ( i ) ∣ X t ( i ) , X t − 1 ( i ) , O t ( i ) , O t − 1 ( i ) ] E [ Y t j ( i ) ∣ X t ( i ) , O t ( i ) ] E [ Y t − 1 , k ( i ) ∣ X t − 1 ( i ) , O t − 1 ( i ) ] ) . P_{jk} := \log\left(\frac{\mathbb{E}\left[Y_{tj}^{(i)}Y_{t-1,k}^{(i)} \mid X_t^{(i)},X_{t-1}^{(i)},O_t^{(i)},O_{t-1}^{(i)}\right]}{\mathbb{E}\left[Y_{tj}^{(i)} \mid X_t^{(i)},O_t^{(i)}\right]\mathbb{E}\left[Y_{t-1,k}^{(i)} \mid X_{t-1}^{(i)},O_{t-1}^{(i)}\right]}\right). P j k := log E [ Y t j ( i ) ∣ X t ( i ) , O t ( i ) ] E [ Y t − 1 , k ( i ) ∣ X t − 1 ( i ) , O t − 1 ( i ) ] E [ Y t j ( i ) Y t − 1 , k ( i ) ∣ X t ( i ) , X t − 1 ( i ) , O t ( i ) , O t − 1 ( i ) ] .
In our model, the lag-one moments imply P = A Σ Z P = A\Sigma_Z P = A Σ Z . Hence, we propose first estimating P P P and then setting A ^ = P ^ Σ ^ Z − 1 \hat{A} = \hat{P}\hat{\Sigma}_Z^{-1} A ^ = P ^ Σ ^ Z − 1 . The explicit estimating equation for each element of P P P is
exp { P ^ j k } = 1 ∑ i = 1 n ( T i − 1 ) ∑ i = 1 n ∑ t = 2 T i Y t j ( i ) Y t − 1 , k ( i ) exp { O t ( i ) + O t − 1 ( i ) + γ ^ 0 j + γ ^ 0 k + X t ( i ) T γ ^ 1 : p , j + X t − 1 ( i ) T γ ^ 1 : p , k } . \exp\left\{\hat{P}_{jk}\right\} = \frac{1}{\sum_{i=1}^{n}(T_i-1)}\sum_{i=1}^{n}\sum_{t=2}^{T_i}\frac{Y_{tj}^{(i)}Y_{t-1,k}^{(i)}}{\exp\left\{O_t^{(i)}+O_{t-1}^{(i)}+\hat{\gamma}_{0j}+\hat{\gamma}_{0k}+X_t^{(i)\mathsf{T}}\hat{\gamma}_{1:p,j}+X_{t-1}^{(i)\mathsf{T}}\hat{\gamma}_{1:p,k}\right\}}. exp { P ^ j k } = ∑ i = 1 n ( T i − 1 ) 1 i = 1 ∑ n t = 2 ∑ T i exp { O t ( i ) + O t − 1 ( i ) + γ ^ 0 j + γ ^ 0 k + X t ( i ) T γ ^ 1 : p , j + X t − 1 ( i ) T γ ^ 1 : p , k } Y t j ( i ) Y t − 1 , k ( i ) .
Finally, leveraging stationarity, we estimate Σ \Sigma Σ by the plug-in estimator Σ ^ = Σ ^ Z − A ^ Σ ^ Z A ^ T \hat{\Sigma} = \hat{\Sigma}_Z - \hat{A}\hat{\Sigma}_Z\hat{A}^{\mathsf{T}} Σ ^ = Σ ^ Z − A ^ Σ ^ Z A ^ T .
We show that under the conditions stated in Appendix § 8.1 , the components of γ ^ \hat{\gamma} γ ^ , Σ ^ Z \hat{\Sigma}_Z Σ ^ Z and P ^ \hat{P} P ^ converge to their target values.
Lemma 2: Under Assumptions (A)–(E), as n → ∞ n \to\infty n → ∞ ,
max 1 ≤ j ≤ d ∥ γ ^ j − γ j ∥ 2 = O p ( log d n ) \max_{1\leq j\leq d}\left\|\hat{\gamma}_j-\gamma_j\right\|_2 = O_p\left(\sqrt{\frac{\log d}{n}}\right) 1 ≤ j ≤ d max ∥ γ ^ j − γ j ∥ 2 = O p ( n log d ) and,
∥ Σ ^ Z − Σ Z ∥ max = O p ( log d n ) , ∥ P ^ − P ∥ max = O p ( log d n ) . \left\|\hat{\Sigma}_Z-\Sigma_Z\right\|_{\max} = O_p\left(\sqrt{\frac{\log d}{n}}\right), \qquad\left\|\hat{P}-P\right\|_{\max} = O_p\left(\sqrt{\frac{\log d}{n}}\right). Σ ^ Z − Σ Z m a x = O p ( n log d ) , P ^ − P m a x = O p ( n log d ) .
Assumptions (A)–(B) state that the number of time-points for each realization of the time series is fixed and does not grow as n n n grows, and the assumed model defined in § 2.1 . The bounded-length condition reflects the longitudinal-data setting we consider, where each subject has a limited number of serial observations. Thus, our asymptotic results concern increasing n n n , allowing d d d to grow while the trajectory lengths remain uniformly bounded. Assumptions (C)–(D) impose moment and compactness conditions on the covariates and parameters, bounded offsets, and uniform positive curvature of the population Poisson criterion. These conditions allow us to establish consistency of γ ^ j \hat{\gamma}_j γ ^ j uniformly over j j j . Assumption (E) bounds the entries of Σ Z \Sigma_Z Σ Z and P P P and restricts the growth of d d d relative to n n n .
Sparse MoM Estimator
To facilitate estimation in high dimensions, we next develop a procedure to estimate sparse A A A matrices.
First, note that our MoM estimator for A A A in § 2.3 is equivalent to solving the optimization problem
A ^ = arg min A ∥ A Σ ^ Z − P ^ ∥ F 2 . \hat{A} = \mathop{\arg\min}_{A} \left\|A\hat{\Sigma}_Z-\hat{P}\right\|_F^2. A ^ = arg min A A Σ ^ Z − P ^ F 2 .
Noticing that the above problem separates over rows, and using the symmetry of Σ ^ Z \hat{\Sigma}_Z Σ ^ Z , each row a ( j ) a^{(j)} a ( j ) for all j ∈ { 1 , 2 , … , d } j \in\{1,2,\ldots,d\} j ∈ { 1 , 2 , … , d } can be estimated as
a ^ ( j ) = arg min a ( j ) ∈ R d ∥ Σ ^ Z a ( j ) − p ^ ( j ) ∥ 2 2 , \hat{a}^{(j)} = \mathop{\arg\min}_{a^{(j)}\in\mathbb{R}^d} \left\|\hat{\Sigma}_Z a^{(j)}-\hat{p}^{(j)}\right\|_2^2, a ^ ( j ) = arg min a ( j ) ∈ R d Σ ^ Z a ( j ) − p ^ ( j ) 2 2 ,
where p ^ ( j ) \hat{p}^{(j)} p ^ ( j ) denotes the j j j -th row of P ^ \hat{P} P ^ .
In light of framing the estimation of A A A as a regression problem, we propose a two-step estimation procedure for estimation of a sparse A A A similar to Belloni and Chernozhukov [4 ] . More specifically, we first use a variable selection procedure to recover the support for each row of A A A . Then, we refit each row on its selected support to obtain the final estimate of A A A . When A A A is not too large, the first step can be accomplished via an exhaustive search over all possible supports and selecting the one that minimizes an information criterion, such as BIC, for the above loss. When it is computationally infeasible to search over all potential supports for A A A , we perform the variable selection step through fitting an initial sparse MoM estimator with a weighted-ℓ 1 \ell_1 ℓ 1 penalty on A A A . Specifically, after estimating β \beta β , Σ Z \mathbf{\Sigma}_Z Σ Z , and P P P as above, we set the weights as w ^ j k = Σ ^ Z , k k 1 / 2 / Σ ^ Z , j j 1 / 2 \widehat{w}_{jk} = \widehat{\Sigma}_{Z,kk}^{1/2}/\widehat{\Sigma}_{Z,jj}^{1/2} w j k = Σ Z , k k 1/2 / Σ Z , j j 1/2 . These weights make penalization comparable across the marginal scales of the latent variables. We find the penalized estimate for each row of A A A with
a ^ λ j ( j ) = arg min a ( j ) ∈ R d ∥ Σ ^ Z a ( j ) − p ^ ( j ) ∥ 2 2 + λ j ∑ k = 1 d w ^ j k ∣ a k ( j ) ∣ . (3) \widehat{a}_{\lambda_j}^{(j)} = \underset{a^{(j)} \in\mathbb{R}^d}{\operatorname{arg\,min}} \left\lVert\widehat{\mathbf{\Sigma}}_Z a^{(j)} - \widehat{p}^{(j)} \right\rVert_2^2 + \lambda_j \sum_{k=1}^{d} \widehat{w}_{jk} \left|a_k^{(j)}\right|. \tag*{(3)} a λ j ( j ) = a ( j ) ∈ R d arg min Σ Z a ( j ) − p ( j ) 2 2 + λ j k = 1 ∑ d w j k a k ( j ) . (3)
The full penalized estimate for A A A is then written as A ^ ℓ 1 = [ a ^ λ 1 ( 1 ) ⋯ a ^ λ d ( d ) ] T ∈ R d × d \widehat{A}_{\ell_1} = \left[\begin{array}{ccc}\widehat{a}_{\lambda_1}^{(1)} & \cdots& \widehat{a}_{\lambda_d}^{(d)}\end{array}\right]^{\mathsf{T}} \in\mathbb{R}^{d \times d} A ℓ 1 = [ a λ 1 ( 1 ) ⋯ a λ d ( d ) ] T ∈ R d × d .
Theorem 1. Under Assumptions (A)–(K) in Appendix § 8.1 , the optimization problem (3) has a unique minimizer for every j = 1 , … , d j = 1,\ldots,d j = 1 , … , d with probability tending to one. The resulting estimators satisfy
Pr { supp ( A ^ ℓ 1 ) = supp ( A ) , sign ( A ^ ℓ 1 ) = sign ( A ) } ⟶ 1. \operatorname{Pr}\left\{\operatorname{supp}(\widehat{A}_{\ell_1}) = \operatorname{supp}(A),\ \operatorname{sign}(\widehat{A}_{\ell_1}) = \operatorname{sign}(A)\right\} \longrightarrow1. Pr { supp ( A ℓ 1 ) = supp ( A ) , sign ( A ℓ 1 ) = sign ( A ) } ⟶ 1. Assumptions (A)–(E) establish the preliminary rates in Lemma 2 . Assumption (F) bounds the sum of absolute covariances in each column of Σ Z \mathbf{\Sigma}_Z Σ Z uniformly as d d d grows. Assumptions (G)–(H) require well-conditioned active Gram matrices and a weighted irrepresentability condition, as is standard in signed support recovery for ℓ 1 \ell_1 ℓ 1 -penalized regression [36 ] . Assumption (I) keeps the marginal latent variances uniformly bounded away from zero, allowing consistent estimation of the ratio weights in (3) . Assumption (J) bounds the entries of A A A , while Assumption (K) specifies penalty sequences that dominate the moment-estimation error while remaining small relative to the weakest nonzero signal.
The optimization problem for the initial penalized estimator in (3) is solved using the fast iterative shrinkage-thresholding algorithm (FISTA) [3 ] . The selection of an appropriate tuning parameter for estimating equations in high-dimensional settings is generally a difficult problem, as discussed in [38 ] and [26 ] . We propose to select the tuning parameters by using a BIC-like criterion. We first fit the penalized estimator over a grid of candidate λ j \lambda_j λ j values, and for each selected λ j \lambda_j λ j , refit on the selected support with
a ^ r e f i t , λ j ( j ) : = arg min a ( j ) ∥ Σ ^ Z a ( j ) − p ^ ( j ) ∥ 2 2 , s.t. supp ( a ( j ) ) = supp ( a ^ λ j ( j ) ) \widehat{a}^{(j)}_{\mathrm{refit},\lambda_j}:=\underset{a^{(j)}}{\arg\min}\left\|\widehat{\Sigma}_Z a^{(j)}-\widehat{p}^{(j)}\right\|_2^2,\qquad\text{s.t. }\operatorname{supp}(a^{(j)})=\operatorname{supp}\left(\widehat{a}^{(j)}_{\lambda_j}\right) a refit , λ j ( j ) := a ( j ) arg min Σ Z a ( j ) − p ( j ) 2 2 , s.t. supp ( a ( j ) ) = supp ( a λ j ( j ) )
where supp ( a ( j ) ) \operatorname{supp}(a^{(j)}) supp ( a ( j ) ) is the support of a ( j ) a^{(j)} a ( j ) . Then, we select λ j \lambda_j λ j by minimizing the following BIC-like criterion,
BIC M o M , j ( λ j ) : = n ∥ Σ ^ Z a ^ r e f i t , λ j ( j ) − p ^ ( j ) ∥ 2 2 + log ( n ) ∣ supp ( a ^ r e f i t , λ j ( j ) ) ∣ . \operatorname{BIC}_{\mathrm{MoM},j}(\lambda_j):=n\left\|\widehat{\Sigma}_Z\widehat{a}^{(j)}_{\mathrm{refit},\lambda_j}-\widehat{p}^{(j)}\right\|_2^2+\log(n)\left|\operatorname{supp}\left(\widehat{a}^{(j)}_{\mathrm{refit},\lambda_j}\right)\right|. BIC MoM , j ( λ j ) := n Σ Z a refit , λ j ( j ) − p ( j ) 2 2 + log ( n ) supp ( a refit , λ j ( j ) ) .
The final estimator for A A A is then obtained as A ^ s p a r s e = [ a ^ r e f i t , λ 1 ∗ ( 1 ) ⋯ a ^ r e f i t , λ d ∗ ( d ) ] T ∈ R d × d \widehat{A}_{\mathrm{sparse}}=\left[\widehat{a}^{(1)}_{\mathrm{refit},\lambda_1^*}\ \cdots\ \widehat{a}^{(d)}_{\mathrm{refit},\lambda_d^*}\right]^\mathsf{T}\in\mathbb{R}^{d\times d} A sparse = [ a refit , λ 1 ∗ ( 1 ) ⋯ a refit , λ d ∗ ( d ) ] T ∈ R d × d , where λ j ∗ \lambda_j^* λ j ∗ denotes the selected λ j \lambda_j λ j for the j j j -th row. For fixed d d d and fixed model parameters, the BIC-type criterion consistently selects the true support, provided Σ Z \Sigma_Z Σ Z is invertible and the candidate supports include the true support with probability tending to one. Indeed, Lemma 2 gives n − 1 / 2 n^{-1/2} n − 1/2 convergence rates for Σ ^ Z \widehat{\Sigma}_Z Σ Z and P ^ \widehat{P} P . The refitted loss on the true support is therefore O p ( n − 1 ) O_p(n^{-1}) O p ( n − 1 ) . Adding unnecessary coefficients can reduce the loss by at most this amount, giving an O p ( 1 ) O_p(1) O p ( 1 ) improvement in the n n n -scaled loss, whereas each additional coefficient incurs a log ( n ) \log(n) log ( n ) penalty. Conversely, invertibility of Σ Z \Sigma_Z Σ Z implies that every support omitting a true nonzero coefficient has strictly positive population loss. Its n n n -scaled loss therefore grows at order n n n , dominating any reduction in the penalty, which is of order log ( n ) \log(n) log ( n ) . Since there are finitely many supports, the true supports are selected simultaneously for all rows with probability tending to one. When d d d grows with n n n , controlling the increasing number of candidate supports may require an additional dimension-dependent complexity penalty, as in the extended BIC (EBIC).
Our final sparse estimate A ^ s p a r s e \widehat{A}_{\mathrm{sparse}} A sparse is a consistent estimator of A A A as a consequence of Theorem 1.
Corollary 1: Let s n = max 1 ≤ j ≤ d ∣ supp ( a ( j ) ) ∣ s_n=\max_{1\leq j\leq d}\left|\operatorname{supp}\left(a^{(j)}\right)\right| s n = max 1 ≤ j ≤ d supp ( a ( j ) ) and ρ n = r n + d r n 2 \rho_n=r_n+dr_n^2 ρ n = r n + d r n 2 , where r n = log d n r_n=\sqrt{\frac{\log d}{n}} r n = n l o g d . Under the assumptions of Theorem 1,
∥ A ^ ℓ 1 − A ∥ max = O p ( s n 3 / 2 ρ n + s n max j λ j ) = o p ( 1 ) . \left\|\widehat{A}_{\ell_1}-A\right\|_{\max}=O_p\left(s_n^{3/2}\rho_n+\sqrt{s_n}\max_j\lambda_j\right)=o_p(1). A ℓ 1 − A m a x = O p ( s n 3/2 ρ n + s n j max λ j ) = o p ( 1 ) .
Let A ^ s p a r s e \widehat{A}_{\mathrm{sparse}} A sparse be obtained by minimizing the unpenalized rowwise loss on the estimated supports S ^ ( j ) = supp ( a ^ j ( j ) ) \widehat{S}^{(j)} = \operatorname{supp}(\widehat{a}^{(j)}_j) S ( j ) = supp ( a j ( j ) ) , with coefficients outside these supports set to zero. If the selected supports equal the true supports with probability tending to one, then
∥ A ^ s p a r s e − A ∥ max = O p ( s n 3 / 2 ρ n ) = o p ( 1 ) . \left\| \widehat{A}_{\mathrm{sparse}} - A \right\|_{\max} = O_p(s_n^{3/2}\rho_n) = o_p(1). A sparse − A m a x = O p ( s n 3/2 ρ n ) = o p ( 1 ) .
MoM Estimator for partially observed time series
Our MoM estimator can also be applied in settings where there are gaps in the observed time series, i.e. some observations are missing. Let R t j ( i ) ∈ { 0 , 1 } R^{(i)}_{tj} \in\{0,1\} R t j ( i ) ∈ { 0 , 1 } be the missingness indicator for observation Y t j ( i ) Y^{(i)}_{tj} Y t j ( i ) so that R t j ( i ) = 1 R^{(i)}_{tj}=1 R t j ( i ) = 1 when Y t j ( i ) Y^{(i)}_{tj} Y t j ( i ) is observed and R t j ( i ) = 0 R^{(i)}_{tj}=0 R t j ( i ) = 0 otherwise, and collect the missingness indicators and observations for all time series realizations, timepoints, and categories as R , Y \mathbf{R},\mathbf{Y} R , Y , respectively. Then, assuming that all covariates are observed and that missingness is random conditional on the covariates and offset—i.e., R ⊥ Y ∣ X , O \mathbf{R} \perp\mathbf{Y}\mid\mathbf{X},\mathbf{O} R ⊥ Y ∣ X , O —we can still obtain unbiased estimating equations for θ \theta θ . First, the objective for estimating each γ j \gamma_j γ j is now written as
L j ( g ) = − 1 ∑ i = 1 n ∑ t = 1 T i R t j ( i ) { ∑ i = 1 n ∑ t = 1 T i R t j ( i ) [ Y t j ( i ) ( O t ( i ) + g 0 + X t ( i ) ⊤ g 1 : p ) − exp ( O t ( i ) + g 0 + X t ( i ) ⊤ g 1 : p ) ] } . L_j(g) = -\frac{1}{\sum_{i=1}^{n}\sum_{t=1}^{T_i}R^{(i)}_{tj}}\left\{\sum_{i=1}^{n}\sum_{t=1}^{T_i}R^{(i)}_{tj}\left[Y^{(i)}_{tj}\left(O^{(i)}_t+g_0+X^{(i)\top}_t g_{1:p}\right)-\exp\left(O^{(i)}_t+g_0+X^{(i)\top}_t g_{1:p}\right)\right]\right\}. L j ( g ) = − ∑ i = 1 n ∑ t = 1 T i R t j ( i ) 1 { i = 1 ∑ n t = 1 ∑ T i R t j ( i ) [ Y t j ( i ) ( O t ( i ) + g 0 + X t ( i ) ⊤ g 1 : p ) − exp ( O t ( i ) + g 0 + X t ( i ) ⊤ g 1 : p ) ] } .
Then, the solutions of the estimating equation for Σ Z \boldsymbol{\Sigma}_Z Σ Z are
exp { Σ ^ Z , j j } = 1 ∑ i = 1 n ∑ t = 1 T i R t j ( i ) ∑ i = 1 n ∑ t = 1 T i R t j ( i ) ( Y t j ( i ) 2 − Y t j ( i ) ) exp { 2 ( O t ( i ) + γ ^ 0 j + X t ( i ) ⊤ γ ^ 1 : p , j ) } . \exp\left\{\widehat{\Sigma}_{Z,jj}\right\}=\frac{1}{\sum_{i=1}^{n}\sum_{t=1}^{T_i}R^{(i)}_{tj}}\sum_{i=1}^{n}\sum_{t=1}^{T_i}\frac{R^{(i)}_{tj}\left(Y^{(i)2}_{tj}-Y^{(i)}_{tj}\right)}{\exp\left\{2\left(O^{(i)}_t+\widehat{\gamma}_{0j}+X^{(i)\top}_t\widehat{\gamma}_{1:p,j}\right)\right\}}. exp { Σ Z , j j } = ∑ i = 1 n ∑ t = 1 T i R t j ( i ) 1 i = 1 ∑ n t = 1 ∑ T i exp { 2 ( O t ( i ) + γ 0 j + X t ( i ) ⊤ γ 1 : p , j ) } R t j ( i ) ( Y t j ( i ) 2 − Y t j ( i ) ) .
exp { Σ ^ Z , j k } = 1 ∑ i = 1 n ∑ t = 1 T i R t j ( i ) R t k ( i ) ∑ i = 1 n ∑ t = 1 T i R t j ( i ) R t k ( i ) Y t j ( i ) Y t k ( i ) exp { 2 O t ( i ) + γ ^ 0 j + γ ^ 0 k + X t ( i ) ⊤ ( γ ^ 1 : p , j + γ ^ 1 : p , k ) } . \exp\left\{\widehat{\Sigma}_{Z,jk}\right\}=\frac{1}{\sum_{i=1}^{n}\sum_{t=1}^{T_i}R^{(i)}_{tj}R^{(i)}_{tk}}\sum_{i=1}^{n}\sum_{t=1}^{T_i}\frac{R^{(i)}_{tj}R^{(i)}_{tk}Y^{(i)}_{tj}Y^{(i)}_{tk}}{\exp\left\{2O^{(i)}_t+\widehat{\gamma}_{0j}+\widehat{\gamma}_{0k}+X^{(i)\top}_t\left(\widehat{\gamma}_{1:p,j}+\widehat{\gamma}_{1:p,k}\right)\right\}}. exp { Σ Z , j k } = ∑ i = 1 n ∑ t = 1 T i R t j ( i ) R t k ( i ) 1 i = 1 ∑ n t = 1 ∑ T i exp { 2 O t ( i ) + γ 0 j + γ 0 k + X t ( i ) ⊤ ( γ 1 : p , j + γ 1 : p , k ) } R t j ( i ) R t k ( i ) Y t j ( i ) Y t k ( i ) .
Next, the solution of the estimating equation for each element of P P P is
exp { P ^ j k } = 1 ∑ i = 1 n ∑ t = 2 T i R t j ( i ) R t − 1 , k ( i ) ∑ i = 1 n ∑ t = 2 T i R t j ( i ) R t − 1 , k ( i ) Y t j ( i ) Y t − 1 , k ( i ) exp { O t ( i ) + O t − 1 ( i ) + γ ^ 0 j + γ ^ 0 k + X t ( i ) ⊤ γ ^ 1 : p , j + X t − 1 ( i ) ⊤ γ ^ 1 : p , k } . \exp\left\{\widehat{P}_{jk}\right\}=\frac{1}{\sum_{i=1}^{n}\sum_{t=2}^{T_i}R^{(i)}_{tj}R^{(i)}_{t-1,k}}\sum_{i=1}^{n}\sum_{t=2}^{T_i}\frac{R^{(i)}_{tj}R^{(i)}_{t-1,k}Y^{(i)}_{tj}Y^{(i)}_{t-1,k}}{\exp\left\{O^{(i)}_t+O^{(i)}_{t-1}+\widehat{\gamma}_{0j}+\widehat{\gamma}_{0k}+X^{(i)\top}_t\widehat{\gamma}_{1:p,j}+X^{(i)\top}_{t-1}\widehat{\gamma}_{1:p,k}\right\}}. exp { P j k } = ∑ i = 1 n ∑ t = 2 T i R t j ( i ) R t − 1 , k ( i ) 1 i = 1 ∑ n t = 2 ∑ T i exp { O t ( i ) + O t − 1 ( i ) + γ 0 j + γ 0 k + X t ( i ) ⊤ γ 1 : p , j + X t − 1 ( i ) ⊤ γ 1 : p , k } R t j ( i ) R t − 1 , k ( i ) Y t j ( i ) Y t − 1 , k ( i ) .
The estimator for A A A is obtained as outlined earlier in § 2.3 and § 2.4 for the standard and sparse MoM estimators, respectively, and the estimator for Σ \boldsymbol{\Sigma} Σ is obtained as specified in § 2.3 .
Variational Inference Estimator
An alternative to the proposed method-of-moments estimator is a likelihood-based estimation strategy. However, exact likelihood-based estimation for our PLN model is challenging because the observed-data likelihood, p θ ( y ) p_\theta(y) p θ ( y ) , requires integrating over the latent states, which is not tractable. A standard approach for latent-variable models is the EM algorithm [12 ] but here the required conditional moments are not available in closed form, and Markov-Chain Monte Carlo-based approximations would be computationally prohibitive in high dimensions. As a result, variational inference (VI) approaches, wherein parameters are estimated by maximizing the evidence lower bound (ELBO), have become prevalent in the PLN literature [8 , 7 ] . In this section, we develop a penalized VI estimator for our PLN model.
Under stationarity, the initial latent density p θ ( z 1 ) p_\theta(z_1) p θ ( z 1 ) is Gaussian with covariance Σ Z \Sigma_Z Σ Z . If one were to derive the ELBO from the full likelihood, the resulting objective would involve log det ( Σ Z ) \log\det(\Sigma_Z) log det ( Σ Z ) and Σ Z − 1 \Sigma_Z^{-1} Σ Z − 1 . Because Σ Z \Sigma_Z Σ Z is defined implicitly by A A A and Σ \Sigma Σ , this would complicate optimization. We therefore base the ELBO on the likelihood conditional on Z 1 ∈ R n × d Z_1 \in\mathbb{R}^{n \times d} Z 1 ∈ R n × d , the collection of the latent states at the first time point across samples, and treat Z 1 Z_1 Z 1 as a set of fixed nuisance parameters to be estimated. Let q Ψ ( Z ) q_\Psi(Z) q Ψ ( Z ) denote a variational approximation to the conditional posterior p θ ( z ∣ y , x , o , z 1 ) p_\theta(z \mid y,x,o,z_1) p θ ( z ∣ y , x , o , z 1 ) of the latent states at times t ≥ 2 t \ge2 t ≥ 2 , where Ψ \Psi Ψ collects the variational means and variances together with the unknown initial-state values. We use a mean-field family for the variational approximation, defined as
Q : = { q : q Ψ ( z ) = ∏ i = 1 n ∏ t = 2 T i ∏ j = 1 d ϕ ( z t j ( i ) ; m t j ( i ) , s t j ( i ) ) } , Q := \left\{q : q_\Psi(z) = \prod_{i=1}^{n}\prod_{t=2}^{T_i}\prod_{j=1}^{d} \phi\left(z_{tj}^{(i)};m_{tj}^{(i)},s_{tj}^{(i)}\right)\right\}, Q := { q : q Ψ ( z ) = i = 1 ∏ n t = 2 ∏ T i j = 1 ∏ d ϕ ( z t j ( i ) ; m t j ( i ) , s t j ( i ) ) } ,
where ϕ ( ⋅ ; μ , σ 2 ) \phi(\cdot;\mu,\sigma^2) ϕ ( ⋅ ; μ , σ 2 ) denotes the univariate Gaussian density with mean μ \mu μ and variance σ 2 \sigma^2 σ 2 . For t ≥ 2 t \ge2 t ≥ 2 , m t j ( i ) m_{tj}^{(i)} m t j ( i ) and s t j ( i ) s_{tj}^{(i)} s t j ( i ) are the variational mean and variance of Z t j ( i ) Z_{tj}^{(i)} Z t j ( i ) , respectively. Because we condition on Z 1 Z_1 Z 1 , the initial states are treated as fixed, with their unknown values estimated through m 1 j ( i ) m_{1j}^{(i)} m 1 j ( i ) . We set s 1 j ( i ) = 0 s_{1j}^{(i)}=0 s 1 j ( i ) = 0 as a notational convention in the ELBO below.
With this variational approximation, the form of the conditional ELBO, up to parameter-independent additive terms, is
ELBO ( y ; Ψ , θ ) = ∑ i = 1 n ∑ t = 1 T i [ y t ( i ) T ( m t ( i ) + μ t ( i ) ) − 1 d T exp { μ t ( i ) + m t ( i ) + 1 2 s t ( i ) } ] − 1 2 ∑ i = 1 n ( T i − 1 ) log det ( Σ ) + 1 2 ∑ i = 1 n ∑ t = 2 T i ∑ j = 1 d log ( s t j ( i ) ) − 1 2 ∑ i = 1 n ∑ t = 1 T i − 1 tr { [ ( m t + 1 ( i ) − A m t ( i ) ) ( m t + 1 ( i ) − A m t ( i ) ) T + S t + 1 ( i ) + A S t ( i ) A T ] Σ − 1 } . \operatorname{ELBO}(y;\Psi,\theta)=\sum_{i=1}^{n}\sum_{t=1}^{T_i}\left[y_t^{(i)\mathsf{T}}\left(m_t^{(i)}+\mu_t^{(i)}\right)-\mathbf{1}_d^{\mathsf{T}}\exp\left\{\mu_t^{(i)}+m_t^{(i)}+\frac{1}{2}s_t^{(i)}\right\}\right]-\frac{1}{2}\sum_{i=1}^{n}(T_i-1)\log\det(\Sigma)+\frac{1}{2}\sum_{i=1}^{n}\sum_{t=2}^{T_i}\sum_{j=1}^{d}\log\left(s_{tj}^{(i)}\right)-\frac{1}{2}\sum_{i=1}^{n}\sum_{t=1}^{T_i-1}\operatorname{tr}\left\{\left[\left(m_{t+1}^{(i)}-Am_t^{(i)}\right)\left(m_{t+1}^{(i)}-Am_t^{(i)}\right)^{\mathsf{T}}+S_{t+1}^{(i)}+AS_t^{(i)}A^{\mathsf{T}}\right]\Sigma^{-1}\right\}. ELBO ( y ; Ψ , θ ) = i = 1 ∑ n t = 1 ∑ T i [ y t ( i ) T ( m t ( i ) + μ t ( i ) ) − 1 d T exp { μ t ( i ) + m t ( i ) + 2 1 s t ( i ) } ] − 2 1 i = 1 ∑ n ( T i − 1 ) log det ( Σ ) + 2 1 i = 1 ∑ n t = 2 ∑ T i j = 1 ∑ d log ( s t j ( i ) ) − 2 1 i = 1 ∑ n t = 1 ∑ T i − 1 tr { [ ( m t + 1 ( i ) − A m t ( i ) ) ( m t + 1 ( i ) − A m t ( i ) ) T + S t + 1 ( i ) + A S t ( i ) A T ] Σ − 1 } .
where μ t ( i ) : = O t ( i ) 1 d + β 0 + β T X t ( i ) \mu_t^{(i)}:=O_t^{(i)}\mathbf{1}_d+\beta_0+\beta^{\mathsf{T}}X_t^{(i)} μ t ( i ) := O t ( i ) 1 d + β 0 + β T X t ( i ) (1 d \mathbf{1}_d 1 d denotes a d d d -dimensional vector of ones), s t ( i ) : = ( s t 1 ( i ) , … , s t d ( i ) ) T s_t^{(i)}:=(s_{t1}^{(i)},\ldots,s_{td}^{(i)})^{\mathsf{T}} s t ( i ) := ( s t 1 ( i ) , … , s t d ( i ) ) T , and S t ( i ) ∈ R d × d S_t^{(i)}\in\mathbb{R}^{d\times d} S t ( i ) ∈ R d × d is the diagonal matrix whose j j j ’th diagonal entry is s t j ( i ) s_{tj}^{(i)} s t j ( i ) .
We estimate θ \theta θ and Ψ \Psi Ψ by maximizing the ELBO via block coordinate ascent. Let M \mathbf{M} M collect all mean parameters, including the initial-state values, and let S \mathbf{S} S collect the variational variance parameters for t ≥ 2 t\ge2 t ≥ 2 . The algorithm cycles through the blocks M \mathbf{M} M , S \mathbf{S} S , Σ − 1 \Sigma^{-1} Σ − 1 , A A A , and [ β 0 , β T ] [\beta_0,\beta^{\mathsf{T}}] [ β 0 , β T ] , updating each one in turn until the increase in ELBO falls below a user-specified tolerance. Although the ELBO is concave in each block separately, it is not jointly concave in all parameters, so convergence to a global maximizer is not guaranteed.
To accommodate estimation in high dimensions, we also develop a penalized VI estimator to encourage estimation of a sparse A A A . Specifically, we first fit the unpenalized VI estimator to obtain pilot estimates A ~ , Σ ~ \widetilde{A},\widetilde{\Sigma} A , Σ , compute Σ ~ Z \widetilde{\Sigma}_Z Σ Z , and set w ^ j k = Σ ~ Z , k k 1 / 2 / Σ ~ Z , j j 1 / 2 \widehat{w}_{jk}=\widetilde{\Sigma}_{Z,kk}^{1/2}/\widetilde{\Sigma}_{Z,jj}^{1/2} w j k = Σ Z , k k 1/2 / Σ Z , j j 1/2 . We then solve
Ψ ^ V I , λ , θ ^ V I , λ = arg min Ψ , θ − ELBO ( y ; Ψ , θ ) + λ ∑ j = 1 d ∑ k = 1 d w ^ j k ∣ A j k ∣ . \widehat{\Psi}_{\mathrm{VI},\lambda},\widehat{\theta}_{\mathrm{VI},\lambda}=\underset{\Psi,\theta}{\arg\min}\quad-\operatorname{ELBO}(y;\Psi,\theta)+\lambda\sum_{j=1}^{d}\sum_{k=1}^{d}\widehat{w}_{jk}|A_{jk}|. Ψ VI , λ , θ VI , λ = Ψ , θ arg min − ELBO ( y ; Ψ , θ ) + λ j = 1 ∑ d k = 1 ∑ d w j k ∣ A j k ∣.
To select λ \lambda λ , the penalized model is fit over a grid of candidate values, and refit on the support selected by θ ^ V I , λ \widehat{\theta}_{\mathrm{VI},\lambda} θ VI , λ , yielding ( θ ^ V I , r e f i t , λ , Ψ ^ V I , r e f i t , λ ) (\widehat{\theta}_{\mathrm{VI,refit},\lambda},\widehat{\Psi}_{\mathrm{VI,refit},\lambda}) ( θ VI , refit , λ , Ψ VI , refit , λ ) . We then choose λ \lambda λ by minimizing the BIC-like criterion:
BIC V I ( λ ) : = − 2 ELBO ( y ; Ψ ^ V I , r e f i t , λ , θ ^ V I , r e f i t , λ ) + log ( n ) ∣ supp ( A ^ V I , r e f i t , λ ) ∣ . \operatorname{BIC}_{\mathrm{VI}}(\lambda):=-2\operatorname{ELBO}(y;\widehat{\Psi}_{\mathrm{VI,refit},\lambda},\widehat{\theta}_{\mathrm{VI,refit},\lambda})+\log(n)\left|\operatorname{supp}(\widehat{A}_{\mathrm{VI,refit},\lambda})\right|. BIC VI ( λ ) := − 2 ELBO ( y ; Ψ VI , refit , λ , θ VI , refit , λ ) + log ( n ) supp ( A VI , refit , λ ) .
Simulation Studies
We conducted simulations to evaluate recovery of the underlying directed networks. Each simulated dataset contained n ∈ { 100 , 250 , 500 } n \in\{100,250,500\} n ∈ { 100 , 250 , 500 } independent trajectories, each observed at T T T time points. The transition matrix A A A was sparse with an expected proportion of nonzero entries per row of either 5% or 10%. Non-zero entries of A A A were set to take values uniformly in either [ 0.25 , 0.5 ] [0.25,0.5] [ 0.25 , 0.5 ] or [ 0.5 , 0.75 ] [0.5,0.75] [ 0.5 , 0.75 ] . We set Σ = diag ( σ 2 ) \Sigma= \operatorname{diag}(\sigma^2) Σ = diag ( σ 2 ) , where σ 2 ∈ R d \sigma^2 \in\mathbb{R}^d σ 2 ∈ R d and the range of values that entries of σ 2 \sigma^2 σ 2 could take varied between [ 0.1 , 0.3 ] [0.1,0.3] [ 0.1 , 0.3 ] or [ 0.5 , 0.7 ] [0.5,0.7] [ 0.5 , 0.7 ] , where each entry sampled uniformly in the given range. Intercepts were generated as β 0 j ∼ N ( 0.2 , 0.1 2 ) \beta_{0j} \sim N(0.2,0.1^2) β 0 j ∼ N ( 0.2 , 0. 1 2 ) and covariate coefficients as β k j ∼ N ( 0 , 0.2 2 ) \beta_{kj} \sim N(0,0.2^2) β k j ∼ N ( 0 , 0. 2 2 ) .
In each simulation, Z 1 Z_1 Z 1 was drawn from the stationary distribution of the latent VAR(1) process, and Z t Z_t Z t for t = 2 , … , T t = 2,\ldots,T t = 2 , … , T was generated recursively according to Eq. (1) . We generated p = 4 p = 4 p = 4 covariates independently as X t k ( i ) ∼ N ( 0 , 2 2 ) X_{tk}^{(i)} \sim N(0,2^2) X t k ( i ) ∼ N ( 0 , 2 2 ) for all t t t , k k k and i i i . The observed counts were then generated as specified in Eq. (2) .
We first compared how the penalized MoM and VI estimators performed in a low-dimensional setting, fixing T = 5 T = 5 T = 5 and d = 10 d = 10 d = 10 . For each setting, we ran 200 simulations and summarized performance by the distributions of the Matthews correlation coefficient (MCC) [22 ] . In each run, the penalized MoM estimator was fit over 100 λ j \lambda_j λ j for each row with values ranging from λ max , j \lambda_{\max,j} λ m a x , j , the smallest value selecting no edges for row j j j , to 0.01 λ max , j 0.01\lambda_{\max,j} 0.01 λ m a x , j . The penalized VI estimator used a common λ \lambda λ across all rows, with 100 candidate values ranging from λ max \lambda_{\max} λ m a x , the smallest value selecting no edges in the full network, to 0.01 λ max 0.01\lambda_{\max} 0.01 λ m a x .
The BIC-selected MoM estimator generally achieved higher MCC than its VI counterpart across the settings considered (Figure 1 ). Recovery generally improved with increasing n n n , with larger improvements for MoM. Based on these results, we focused the subsequent simulations on the MoM estimator in higher dimensions.
Figure 1. Distributions of MCC for the MoM and VI estimators for d = 10 d = 10 d = 10 across values of A A A , Σ \Sigma Σ , and n n n , where the final model was selected according to the BIC criterion. Within each setting, panels are faceted by the density of A A A . The x-axis gives n n n and color distinguishes the MoM estimator (purple) from the VI estimator (dark teal).
For the higher-dimensional settings, we fixed T = 30 T = 30 T = 30 to reflect the TREAM dataset discussed in § 4 . The sparsity pattern and value ranges of A A A , as well as the value ranges of Σ \boldsymbol{\Sigma} Σ were kept as before. We considered a medium-dimensional regime, d = 25 d = 25 d = 25 , resembling the TREAM data, and a higher-dimensional regime, d = 50 d = 50 d = 50 , with n ∈ { 100 , 250 , 500 } n \in\{100, 250, 500\} n ∈ { 100 , 250 , 500 } . For the high-dimensional setting with d = 50 d = 50 d = 50 and values of A A A in the range [ 0.5 − 0.75 ] [0.5 - 0.75] [ 0.5 − 0.75 ] , we allowed 5% non-zero values in each row of A A A . This is because we could not reliably construct A A A matrices with d = 50 d = 50 d = 50 and entries in the [ 0.5 , 0.75 ] [0.5, 0.75] [ 0.5 , 0.75 ] range with all eigenvalues having modulus less than 1 when 10% of entries in each row were nonzero. We aimed to assess how model performance changed as dimensionality increases, and also to benchmark our model selection procedure by comparing how the models selected by the BIC-like criteria from § 2.4 fared against an oracle estimator, defined as the sparsest model attaining the maximum true positive rate (TPR) along the λ \lambda λ -grid.
Here, we present results for d = 25 d = 25 d = 25 with an average of 10% nonzero entries per row and d = 50 d = 50 d = 50 with an average of 5% nonzero entries per row in Figure 2 –Figure 3 . Network recovery generally improved as n n n increased in both settings. The BIC-like criterion often achieved TPR close to the oracle benchmark, except when both the transition coefficients A A A , and the latent noise variances Σ \boldsymbol{\Sigma} Σ were relatively small. However, it selected more false edges than the oracle when both were relatively large. For smaller transition coefficients, the oracle-selected networks showed TPR approaching one and false positive rate (FPR) approaching zero as n n n increased. With larger transition coefficients, the oracle-selected networks continued to exhibit missed and false edges over the sample sizes considered. Additional results for d = 25 d = 25 d = 25 with 5% density and d = 50 d = 50 d = 50 with 10% density are shown in Figure A1 –Figure A2 in Appendix C. False positive rates were generally lower in the sparser settings. However, the effect of coefficient magnitude depended on the setting: for d = 25 d = 25 d = 25 with 5% density, larger transition coefficients improved TPR while FPR remained low, with nearly perfect recovery when the latent noise variances were also relatively large.
Figure 2. Distributions of TPR and FPR for the MoM estimator across values of A A A , Σ \Sigma Σ , and n n n when d = 25 d = 25 d = 25 and on average 10% of each row of A A A is non-zero. Within each setting, panels are faceted by the metric (TPR, FPR). The x-axis gives n n n and color distinguishes the network selection criteria (“BIC” for BIC and “Oracle” for the sparsest network attaining the highest TPR), where “BIC” is in blue and “Oracle” is in orange.
Figure 3. Distributions of TPR and FPR for the MoM estimator across values of A A A , Σ \Sigma Σ , and n n n when d = 50 d = 50 d = 50 and on average 5% of each row of A A A is non-zero. Within each setting, panels are faceted by the metric (TPR, FPR). The x-axis gives n n n and color distinguishes the network selection criteria (“BIC” for BIC and “Oracle” for the sparsest network attaining the highest TPR), where “BIC” is in blue and “Oracle” in orange.
Figure A1. Distributions of TPR and FPR for the MoM estimator across values of A A A , Σ \Sigma Σ , and n n n when d = 25 d = 25 d = 25 and on average 5% of each row of A A A is non-zero. Within each setting, panels are faceted by the metric (TPR, FPR). The x-axis gives n n n and color distinguishes the network selection criteria (“BIC” for BIC and “Oracle” for the sparsest network attaining the highest TPR), where “BIC” is in blue and “Oracle” in orange.
Figure A2. Distributions of TPR and FPR for the MoM estimator across values of A A A , Σ \Sigma Σ , and n n n when d = 50 d = 50 d = 50 and on average 10% of each row of A A A is non-zero. Within each setting, panels are faceted by the metric (TPR, FPR). The x-axis gives n n n and color distinguishes the network selection criteria (“BIC” for BIC and “Oracle” for the sparsest network attaining the highest TPR), where “BIC” is in blue and “Oracle” in orange.
Larger nonzero transition coefficients might be expected to improve network recovery by increasing signal strength. However, the poorer recovery observed with larger coefficients in some settings may reflect accompanying changes in the stationary latent variances. The covariance of the observed values depends exponentially on the entries of Σ Z \boldsymbol{\Sigma}_Z Σ Z , which under our model satisfies Σ Z = A Σ Z A T + Σ \boldsymbol{\Sigma}_Z = A\boldsymbol{\Sigma}_Z A^{\mathsf{T}} + \boldsymbol{\Sigma} Σ Z = A Σ Z A T + Σ , or equivalently, vec ( Σ Z ) = ( I − A ⊗ A ) − 1 vec ( Σ ) \operatorname{vec}(\boldsymbol{\Sigma}_Z) = (I - A \otimes A)^{-1}\operatorname{vec}(\boldsymbol{\Sigma}) vec ( Σ Z ) = ( I − A ⊗ A ) − 1 vec ( Σ ) . Hence, larger coefficients in A A A , can lead to larger stationary latent variances, increasing the variability of the observations and empirical moments and making network recovery more difficult in those settings.
Analysis of Ecological Data
We apply our MoM estimator to infer relationships among freshwater macroinvertebrates, which are often used as bioindicators of aquatic environmental change [17 ] . Understanding these relationships can help disentangle changes in community composition due to species interactions from those driven by environmental factors. The TREAM dataset compiles macroinvertebrate sampling efforts across 22 European countries spanning 1968–2020 [37 ] . Our aim is to estimate how the abundance of each macroinvertebrate group depends on the lagged abundances of other groups in the community. Whereas previous analyses of TREAM have focused on temporal trends in community and functional diversity [15 , 30 , 28 , 9 ] , our analysis instead targets the network of temporal relationships among groups.
We focus on a subset of TREAM, comprising 248 freshwater sites in Denmark sampled between 1992 and 2020. Each site was sampled at most once per year, so each year was treated as a time point. Not all sites were sampled every year. We assumed that missingness was independent of the counts conditional on the covariates and offsets, and used only observed consecutive-year pairs to estimate lagged moments. Organisms were recorded at varying taxonomic resolution, and counts were aggregated to the order-or-higher level, referred to in the dataset as “groups”. This yielded 27 groups in total. If a group was never observed at a site, its absence at that site was treated as a structural zero (the group does not inhabit the site). For groups recorded at least once at a site, absence from a sampled site-year was treated as a sampling zero. Years without sampling at the site remained missing. This assumes that the set of groups present at a site is stable over time. We retained only groups that were present in at least 5% of the available site-year combinations in the dataset and had at least one consecutive-year pair of observations with every other group. This resulted in a final set of 15 groups in our analysis.
For each site-year combination, we included four time-varying covariates: cumulative precipitation in the preceding year (millimeters), the average of monthly maximum temperatures over the preceding 12 months (C°), and the upstream percentages of cropland and urban land cover. Four site-level covariates that were constant over time were also included: Strahler stream order (quantifies river branching complexity), flow accumulation (km², measuring upstream drainage area), elevation (meters), and average stream slope. These covariates were included to control for environmental drivers.
The data were analyzed using the sparse MoM estimator, where due to the low dimensionality of the data we employed exhaustive search to select the final network. As the observed counts for the categories varied greatly in terms of the scale and amount of extreme counts present, we also utilized a stability selection procedure motivated by [24 ] in our analysis to limit the inclusion of spurious edges in the final network. We implemented this by drawing 200 subsamples consisting of 198 sites (80% of the total sites) each, fitting the model for each subsample, and designating the stable selected support as edges that were selected in at least 90% of the subsamples. This procedure yielded a final network that selected all self-edges other than that of Heteroptera and 6 cross-edges. In contrast, fitting the model to the full data yielded a much denser network with all of the self-edges and 125 cross-edges selected. Most edges selected from the full data were not consistently selected across subsamples, indicating substantial sensitivity of the estimated support to the sites included in the analysis. After finding the stable support, we performed the refitting procedure to obtain our final estimate of the network, shown in Figure 4 .
Figure 4. Recovered network for the TREAM data, shown as a bipartite graph. Nodes in the left column represent groups at time t t t , and nodes in the right column represent groups at time t + 1 t + 1 t + 1 . An edge indicates a lagged effect of a group at time t t t on a group at time t + 1 t + 1 t + 1 . Blue edges indicate negative effects, red edges indicate positive effects, and edge widths are proportional to effect size. Self-edges, corresponding to diagonal entries of A A A , are omitted to highlight interactions among distinct groups; all self-edges except that of Heteroptera were selected and positive.
The recovered network identifies edges among the taxonomic orders of Diptera, Bivalvia, Tricladida, Prosobranchia, Hirudinoidea, Heteroptera, and Odonata. The positive edge from Diptera to Hirudinoidea is compatible with reported feeding relationship between Hirudinoidea and Diptera [34 ] . The positive edge from Bivalvia to Megaloptera suggests that findings of aquatic mollusks having a positive impact on abundances of other macrofauna from the meta-analysis by Zieritz et al. [41 ] could also be present in the Danish aquatic sites. We also recovered negative edges from Tricladida to Megaloptera, Heteroptera, and Odonata and a negative edge from Prosobranchia to Odonata. These negative edges may suggest competitive relationships among these taxonomic orders, but require further ecological investigation. A caveat to these results is that they are based on annual sampling of freshwater sites, so our model may not capture relationships as faithfully if the true lagged effect occurs on a different time scale [32 ] .
Discussion
We proposed a framework for inferring directed networks from multivariate count time series using a latent Gaussian VAR process. To estimate the underlying network, we developed MoM and VI estimators, together with their ℓ 1 \ell_1 ℓ 1 -penalized versions for sparse network recovery. We established support recovery and coefficient-consistency results for the penalized MoM estimator under suitable conditions. Our simulation study indicates that the BIC-selected MoM estimator generally outperforms its VI counterpart, and we illustrated the method’s utility on freshwater macroinvertebrate abundance data.
Our theoretical network-recovery guarantees require appropriate row-specific penalty se quences. Although the proposed BIC-like criterion performed well in several simulation settings, it omitted many true edges when both the transition coefficients and the latent noise variances were small. The higher sensitivity of the oracle-selected networks in these settings indicates room for improving the tuning procedure. Future work could investigate alternative model-selection criteria or cross-validation procedures based on projected estimating equations, as considered by Xia and Shojaie [38 ] . Another direction is to extend the proposed estimator to a generalized method-of-moments procedure, weighting the moment conditions according to their variability and dependence to potentially improve efficiency. This would require reliable estimation of the covariance of the moment conditions. It would also be useful to study an asymptotic regime with a fixed number of trajectories and increasing trajectory lengths, under suitable conditions on temporal dependence, complementing our current theory for many short trajectories. Finally, developing statistical inference procedures for the estimated network coefficients remains an important direction for future work.
[ Figure 1 about here. ]
[ Figure 2 about here. ]
[ Figure 3 about here. ]
[ Figure 4 about here. ]
References
Appendix
Appendix A: Proof of Lemma 1: Model Identifiablity
Proof. We establish identifiability of the model by showing that each parameter can be expressed as a function of the moments of Y t ( i ) Y_{t}^{(i)} Y t ( i ) . Let X ( i ) ∈ R T i × p X^{(i)} \in\mathbb{R}^{T_i \times p} X ( i ) ∈ R T i × p be the covariates across all time-points for the i i i -th sample and O ( i ) ∈ R T i O^{(i)} \in\mathbb{R}^{T_i} O ( i ) ∈ R T i be the offsets across all time-points for the i i i -th sample. Then, we have the following expressions for the first and second moments of Y t j ( i ) Y_{tj}^{(i)} Y t j ( i ) , the first moment of Y t j ( i ) Y t k ( i ) Y_{tj}^{(i)}Y_{tk}^{(i)} Y t j ( i ) Y t k ( i ) , and the first moment of Y t − 1 , j ( i ) Y t k ( i ) Y_{t-1,j}^{(i)}Y_{tk}^{(i)} Y t − 1 , j ( i ) Y t k ( i ) , conditional on the covariates, X ( i ) X^{(i)} X ( i ) and offsets O ( i ) O^{(i)} O ( i ) .
E [ Y t j ( i ) ∣ O ( i ) , X ( i ) ] = exp { O t ( i ) + β 0 j + X t ( i ) T β j + 1 2 Σ Z , j j } \mathbb{E}\left[Y_{tj}^{(i)} \mid O^{(i)}, X^{(i)}\right] = \exp\left\{O_t^{(i)}+\beta_{0j}+X_t^{(i)\mathsf{T}}\beta_j+\frac{1}{2}\Sigma_{Z,jj}\right\} E [ Y t j ( i ) ∣ O ( i ) , X ( i ) ] = exp { O t ( i ) + β 0 j + X t ( i ) T β j + 2 1 Σ Z , j j }
E [ Y t j ( i ) 2 ∣ O ( i ) , X ( i ) ] = E [ Y t j ( i ) ∣ O ( i ) , X ( i ) ] ( 1 + E [ Y t j ( i ) ∣ O ( i ) , X ( i ) ] exp { Σ Z , j j } ) \mathbb{E}\left[Y_{tj}^{(i)2} \mid O^{(i)}, X^{(i)}\right] = \mathbb{E}\left[Y_{tj}^{(i)} \mid O^{(i)}, X^{(i)}\right]\left(1+\mathbb{E}\left[Y_{tj}^{(i)} \mid O^{(i)}, X^{(i)}\right]\exp\left\{\Sigma_{Z,jj}\right\}\right) E [ Y t j ( i ) 2 ∣ O ( i ) , X ( i ) ] = E [ Y t j ( i ) ∣ O ( i ) , X ( i ) ] ( 1 + E [ Y t j ( i ) ∣ O ( i ) , X ( i ) ] exp { Σ Z , j j } )
E [ Y t j ( i ) Y t k ( i ) ∣ O ( i ) , X ( i ) ] = E [ Y t j ( i ) ∣ O ( i ) , X ( i ) ] E [ Y t k ( i ) ∣ O ( i ) , X ( i ) ] exp { Σ Z , j k } \mathbb{E}\left[Y_{tj}^{(i)}Y_{tk}^{(i)} \mid O^{(i)}, X^{(i)}\right] = \mathbb{E}\left[Y_{tj}^{(i)} \mid O^{(i)}, X^{(i)}\right]\mathbb{E}\left[Y_{tk}^{(i)} \mid O^{(i)}, X^{(i)}\right]\exp\left\{\Sigma_{Z,jk}\right\} E [ Y t j ( i ) Y t k ( i ) ∣ O ( i ) , X ( i ) ] = E [ Y t j ( i ) ∣ O ( i ) , X ( i ) ] E [ Y t k ( i ) ∣ O ( i ) , X ( i ) ] exp { Σ Z , j k }
E [ Y t j ( i ) Y t − 1 , k ( i ) ∣ O ( i ) , X ( i ) ] = E [ Y t j ( i ) ∣ O ( i ) , X ( i ) ] E [ Y t − 1 , k ( i ) ∣ O ( i ) , X ( i ) ] exp { A j Σ Z , k } \mathbb{E}\left[Y_{tj}^{(i)}Y_{t-1,k}^{(i)} \mid O^{(i)}, X^{(i)}\right] = \mathbb{E}\left[Y_{tj}^{(i)} \mid O^{(i)}, X^{(i)}\right]\mathbb{E}\left[Y_{t-1,k}^{(i)} \mid O^{(i)}, X^{(i)}\right]\exp\left\{A_j\Sigma_{Z,k}\right\} E [ Y t j ( i ) Y t − 1 , k ( i ) ∣ O ( i ) , X ( i ) ] = E [ Y t j ( i ) ∣ O ( i ) , X ( i ) ] E [ Y t − 1 , k ( i ) ∣ O ( i ) , X ( i ) ] exp { A j Σ Z , k }
From above, we can identify Σ Z \Sigma_Z Σ Z by identifying each component of Σ Z \Sigma_Z Σ Z .
Σ Z , j j = log ( E [ Y t j ( i ) 2 ∣ O ( i ) , X ( i ) ] − E [ Y t j ( i ) ∣ O ( i ) , X ( i ) ] ) − 2 log ( E [ Y t j ( i ) ∣ O ( i ) , X ( i ) ] ) \Sigma_{Z,jj} = \log\left(\mathbb{E}\left[Y_{t_j}^{(i)2}\mid O^{(i)},X^{(i)}\right]-\mathbb{E}\left[Y_{t_j}^{(i)}\mid O^{(i)},X^{(i)}\right]\right)-2\log\left(\mathbb{E}\left[Y_{t_j}^{(i)}\mid O^{(i)},X^{(i)}\right]\right) Σ Z , j j = log ( E [ Y t j ( i ) 2 ∣ O ( i ) , X ( i ) ] − E [ Y t j ( i ) ∣ O ( i ) , X ( i ) ] ) − 2 log ( E [ Y t j ( i ) ∣ O ( i ) , X ( i ) ] )
Σ Z , j k = log ( E [ Y t j ( i ) Y t k ( i ) ∣ O ( i ) , X ( i ) ] ) − log ( E [ Y t j ( i ) ∣ O ( i ) , X ( i ) ] ) − log ( E [ Y t k ( i ) ∣ O ( i ) , X ( i ) ] ) \Sigma_{Z,jk} = \log\left(\mathbb{E}\left[Y_{t_j}^{(i)}Y_{t_k}^{(i)}\mid O^{(i)},X^{(i)}\right]\right)-\log\left(\mathbb{E}\left[Y_{t_j}^{(i)}\mid O^{(i)},X^{(i)}\right]\right)-\log\left(\mathbb{E}\left[Y_{t_k}^{(i)}\mid O^{(i)},X^{(i)}\right]\right) Σ Z , j k = log ( E [ Y t j ( i ) Y t k ( i ) ∣ O ( i ) , X ( i ) ] ) − log ( E [ Y t j ( i ) ∣ O ( i ) , X ( i ) ] ) − log ( E [ Y t k ( i ) ∣ O ( i ) , X ( i ) ] )
Let N t o t : = ∑ i = 1 n T i N_{\mathrm{tot}} := \sum_{i=1}^{n} T_i N tot := ∑ i = 1 n T i , X ∈ R N t o t × p \mathbf{X} \in\mathbb{R}^{N_{\mathrm{tot}}\times p} X ∈ R N tot × p be the collection of covariates for each sample and time-point combination, Y ∈ R N t o t × d \mathbf{Y} \in\mathbb{R}^{N_{\mathrm{tot}}\times d} Y ∈ R N tot × d be the collection of the observed counts for each category across each time-point and sample combination, and O ∈ R N t o t \mathbf{O} \in\mathbb{R}^{N_{\mathrm{tot}}} O ∈ R N tot be the collection of the offset term for each sample and time-point combination. By Lemma 1, X ˉ = [ 1 , X ] \bar{\mathbf{X}}=[\mathbf{1},\mathbf{X}] X ˉ = [ 1 , X ] has full column rank. Since Σ Z \Sigma_Z Σ Z has already been identified, the intercept and slopes are identified jointly by
X ˉ ( β 0 j β j ) = log ( E [ Y j ∣ O , X ] ) − O − 1 2 Σ Z , j j 1 , \bar{\mathbf{X}}\begin{pmatrix}\beta_{0j}\\\beta_j\end{pmatrix}=\log\left(\mathbb{E}[\mathbf{Y}_j\mid\mathbf{O},\mathbf{X}]\right)-\mathbf{O}-\frac{1}{2}\Sigma_{Z,jj}\mathbf{1}, X ˉ ( β 0 j β j ) = log ( E [ Y j ∣ O , X ] ) − O − 2 1 Σ Z , j j 1 ,
( β 0 j β j ) = ( X ˉ T X ˉ ) − 1 X ˉ T { log ( E [ Y j ∣ O , X ] ) − O − 1 2 Σ Z , j j 1 } . \begin{pmatrix}\beta_{0j}\\\beta_j\end{pmatrix}=(\bar{\mathbf{X}}^{\mathsf{T}}\bar{\mathbf{X}})^{-1}\bar{\mathbf{X}}^{\mathsf{T}}\left\{\log\left(\mathbb{E}[\mathbf{Y}_j\mid\mathbf{O},\mathbf{X}]\right)-\mathbf{O}-\frac{1}{2}\Sigma_{Z,jj}\mathbf{1}\right\}. ( β 0 j β j ) = ( X ˉ T X ˉ ) − 1 X ˉ T { log ( E [ Y j ∣ O , X ] ) − O − 2 1 Σ Z , j j 1 } .
The logarithm is taken entrywise. Then, we can identify A A A by noting that
A j Σ Z , k = log ( E [ Y t j ( i ) Y t − 1 , k ( i ) ∣ O , X ( i ) ] ) − log ( E [ Y t j ( i ) ∣ O , X ( i ) ] ) − log ( E [ Y t − 1 , k ( i ) ∣ O , X ( i ) ] ) . A_j\Sigma_{Z,k}=\log\left(\mathbb{E}\left[Y_{t_j}^{(i)}Y_{t-1,k}^{(i)}\mid\mathbf{O},\mathbf{X}^{(i)}\right]\right)-\log\left(\mathbb{E}\left[Y_{t_j}^{(i)}\mid\mathbf{O},\mathbf{X}^{(i)}\right]\right)-\log\left(\mathbb{E}\left[Y_{t-1,k}^{(i)}\mid\mathbf{O},\mathbf{X}^{(i)}\right]\right). A j Σ Z , k = log ( E [ Y t j ( i ) Y t − 1 , k ( i ) ∣ O , X ( i ) ] ) − log ( E [ Y t j ( i ) ∣ O , X ( i ) ] ) − log ( E [ Y t − 1 , k ( i ) ∣ O , X ( i ) ] ) .
As we assume that Σ Z \Sigma_Z Σ Z is invertible, this implies that
A Σ Z = P ⟺ A = P Σ Z − 1 , A\Sigma_Z=P\iff A=P\Sigma_Z^{-1}, A Σ Z = P ⟺ A = P Σ Z − 1 ,
where
P j k = log ( E [ Y t j ( i ) Y t − 1 , k ( i ) ∣ O , X ( i ) ] ) − log ( E [ Y t j ( i ) ∣ O , X ( i ) ] ) − log ( E [ Y t − 1 , k ( i ) ∣ O , X ( i ) ] ) P_{jk}=\log\left(\mathbb{E}\left[Y_{t_j}^{(i)}Y_{t-1,k}^{(i)}\mid\mathbf{O},\mathbf{X}^{(i)}\right]\right)-\log\left(\mathbb{E}\left[Y_{t_j}^{(i)}\mid\mathbf{O},\mathbf{X}^{(i)}\right]\right)-\log\left(\mathbb{E}\left[Y_{t-1,k}^{(i)}\mid\mathbf{O},\mathbf{X}^{(i)}\right]\right) P j k = log ( E [ Y t j ( i ) Y t − 1 , k ( i ) ∣ O , X ( i ) ] ) − log ( E [ Y t j ( i ) ∣ O , X ( i ) ] ) − log ( E [ Y t − 1 , k ( i ) ∣ O , X ( i ) ] )
Hence, A A A is identified.
Then, from the stationarity of the latent VAR(1) process, we have that
Σ = Σ Z − A Σ Z A T = Σ Z − P Σ Z − 1 P T , \Sigma=\Sigma_Z-A\Sigma_ZA^{\mathsf{T}}=\Sigma_Z-P\Sigma_Z^{-1}P^{\mathsf{T}}, Σ = Σ Z − A Σ Z A T = Σ Z − P Σ Z − 1 P T ,
so Σ \Sigma Σ is identified. Hence, the model is identifiable.
Appendix
Appendix B: Proof of Support Recovery
In this appendix, L j L_j L j is the empirical Poisson criterion in § 2.3 , and S ( j ) = { k : A j k ≠ 0 } S^{(j)} = \{k : A_{jk} \ne0\} S ( j ) = { k : A j k = 0 } , s j = ∣ S ( j ) ∣ s_j = \lvert S^{(j)}\rvert s j = ∣ S ( j ) ∣ . Individual rows may have empty support. To define the minimum nonzero signal, assume that A A A has at least one nonzero entry. For a matrix M M M , M S M_S M S denotes the columns indexed by S S S and S ( j ) , c S^{(j),c} S ( j ) , c denotes the complementary set.
We use the augmented design X ˉ \bar{X} X ˉ defined in Lemma 1, with X ˉ t ( i ) \bar{X}^{(i)}_t X ˉ t ( i ) denoting the transpose of its row corresponding to trajectory i i i and time t t t . Let
r n = log d n . r_n = \sqrt{\frac{\log d}{n}}. r n = n log d .
Denote s n = max 1 ≤ j ≤ d ∣ S ( j ) ∣ s_n = \max_{1 \le j \le d} \lvert S^{(j)}\rvert s n = max 1 ≤ j ≤ d ∣ S ( j ) ∣ , ρ n = r n + d r n 2 \rho_n = r_n + d r_n^2 ρ n = r n + d r n 2 , and define the minimum nonzero signal by
a min , n = min 1 ≤ j , k ≤ d A j k ≠ 0 ∣ A j k ∣ . a_{\min,n} = \min_{\substack{1 \le j,k \le d \\ A_{jk} \ne0}} \lvert A_{jk}\rvert. a m i n , n = 1 ≤ j , k ≤ d A j k = 0 min ∣ A j k ∣ .
For conciseness, define
H j = Σ Z , S ( j ) T Σ Z , S ( j ) , B j = Σ Z , S ( j ) , c T Σ Z , S ( j ) . H_j = \Sigma_{Z,S^{(j)}}^{\mathsf{T}}\Sigma_{Z,S^{(j)}}, \qquad B_j = \Sigma_{Z,S^{(j),c}}^{\mathsf{T}}\Sigma_{Z,S^{(j)}}. H j = Σ Z , S ( j ) T Σ Z , S ( j ) , B j = Σ Z , S ( j ) , c T Σ Z , S ( j ) .
Also define
w k = Σ Z , k k 1 / 2 , w ^ k = Σ ^ Z , k k 1 / 2 , D = diag ( w 1 , … , w d ) , D ^ = diag ( w ^ 1 , … , w ^ d ) . w_k = \Sigma_{Z,kk}^{1/2}, \qquad\hat{w}_k = \hat{\Sigma}_{Z,kk}^{1/2}, \qquad D = \operatorname{diag}(w_1,\ldots,w_d), \qquad\hat{D} = \operatorname{diag}(\hat{w}_1,\ldots,\hat{w}_d). w k = Σ Z , k k 1/2 , w ^ k = Σ ^ Z , k k 1/2 , D = diag ( w 1 , … , w d ) , D ^ = diag ( w ^ 1 , … , w ^ d ) .
D S D_S D S and D ^ S \hat{D}_S D ^ S denote the diagonal submatrices indexed by S S S . The population and empirical penalty weights are w j k = w k / w j w_{jk} = w_k/w_j w j k = w k / w j and w ^ j k = w ^ k / w ^ j \hat{w}_{jk} = \hat{w}_k/\hat{w}_j w ^ j k = w ^ k / w ^ j , respectively. Empirical standard deviations and ratio weights are evaluated on the event min k Σ ^ Z , k k > 0 \min_k \hat{\Sigma}_{Z,kk} > 0 min k Σ ^ Z , k k > 0 . Lemma 3 shows that this event has probability tending to one.
Assume:
(A) The n n n trajectories are mutually independent. The trajectory lengths are fixed, and there exists a constant T max < ∞ T_{\max} < \infty T m a x < ∞ , not depending on n n n , such that
2 ≤ T i ≤ T max , i = 1 , … , n . 2 \le T_i \le T_{\max}, \qquad i = 1,\ldots,n. 2 ≤ T i ≤ T m a x , i = 1 , … , n .
(B) The latent process is stationary and satisfies the model in § 2.1 . For each trajectory, the latent process is independent of the covariates and offsets.
(C) The covariate dimension p p p is fixed. For every a > 0 a > 0 a > 0 , there exists a constant C a < ∞ C_a < \infty C a < ∞ , independent of n n n , i i i , and t t t , such that
E { exp ( a ∥ X t ( i ) ∥ 2 ) } ≤ C a . \mathbb{E}\left\{\exp\left(a\left\|\mathbf{X}^{(i)}_t\right\|_2\right)\right\} \le C_a. E { exp ( a X t ( i ) 2 ) } ≤ C a .
Moreover, there exists a constant K O < ∞ K_O < \infty K O < ∞ such that
∣ O t ( i ) ∣ ≤ K O almost surely \left|O^{(i)}_t\right| \le K_O \quad\text{almost surely} O t ( i ) ≤ K O almost surely
for all i i i and t t t . The true vectors γ j \gamma_j γ j belong to the interior of a fixed compact convex set Γ ⊂ R p + 1 \Gamma\subset\mathbb{R}^{p+1} Γ ⊂ R p + 1 and remain uniformly separated from its boundary.
(D) For some constant c γ > 0 c_\gamma> 0 c γ > 0 ,
inf n ≥ 1 min 1 ≤ j ≤ d inf g ∈ Γ λ min { E ∇ 2 L j ( g ) } ≥ c γ . \inf_{n \ge1}\min_{1 \le j \le d}\inf_{g \in\Gamma} \lambda_{\min}\left\{\mathbb{E}\nabla^2 L_j(g)\right\} \ge c_\gamma. n ≥ 1 inf 1 ≤ j ≤ d min g ∈ Γ inf λ m i n { E ∇ 2 L j ( g ) } ≥ c γ .
(E) There are constants K M K_M K M , C d < ∞ C_d < \infty C d < ∞ and κ > 0 \kappa> 0 κ > 0 such that
∥ Σ Z ∥ max ∨ ∥ P ∥ max ≤ K M , 2 ≤ d ≤ C d n κ . \left\|\Sigma_Z\right\|_{\max} \vee\left\|P\right\|_{\max} \le K_M,\qquad2 \le d \le C_d n^\kappa. ∥ Σ Z ∥ m a x ∨ ∥ P ∥ m a x ≤ K M , 2 ≤ d ≤ C d n κ .
(F) For some constant C 1 > 0 C_1 > 0 C 1 > 0 , ∥ Σ Z ∥ 1 ≤ C 1 \left\|\Sigma_Z\right\|_1 \le C_1 ∥ Σ Z ∥ 1 ≤ C 1 .
(G) For some constant C 2 > 0 C_2 > 0 C 2 > 0 ,
min 1 ≤ j ≤ d s j > 0 λ min ( H j ) ≥ C 2 . \min_{\substack{1 \le j \le d\\ s_j > 0}} \lambda_{\min}(H_j) \ge C_2. 1 ≤ j ≤ d s j > 0 min λ m i n ( H j ) ≥ C 2 .
(H) For some constant η ∈ ( 0 , 1 ) \eta\in(0,1) η ∈ ( 0 , 1 ) ,
max 1 ≤ j ≤ d s j > 0 ∥ D S ( j ) , c − 1 B j H j − 1 D S ( j ) ∥ ∞ ≤ 1 − η . \max_{\substack{1 \le j \le d\\ s_j > 0}} \left\|D_{S(j),c}^{-1} B_j H_j^{-1} D_{S(j)}\right\|_\infty\le1-\eta. 1 ≤ j ≤ d s j > 0 max D S ( j ) , c − 1 B j H j − 1 D S ( j ) ∞ ≤ 1 − η .
(I) For some constant c w > 0 c_w > 0 c w > 0 ,
min 1 ≤ k ≤ d Σ Z , k k ≥ c w . \min_{1 \le k \le d} \Sigma_{Z,kk} \ge c_w. 1 ≤ k ≤ d min Σ Z , k k ≥ c w .
(J) For some constant C A > 0 C_A > 0 C A > 0 , ∥ A ∥ max ≤ C A \left\|A\right\|_{\max} \le C_A ∥ A ∥ m a x ≤ C A .
(K) The prescribed positive penalty sequences λ j \lambda_j λ j satisfy
s n max 1 ≤ j ≤ d λ j = o ( a min , n ) , s n ρ n = o ( min 1 ≤ j ≤ d λ j ) . \sqrt{s_n}\max_{1 \le j \le d}\lambda_j = o(a_{\min,n}),\qquad s_n\rho_n = o\left(\min_{1 \le j \le d}\lambda_j\right). s n 1 ≤ j ≤ d max λ j = o ( a m i n , n ) , s n ρ n = o ( 1 ≤ j ≤ d min λ j ) .
Proof. [Proof of Lemma 2] We use the following Nagaev inequality [25 Corollary 1.8] , see also (Li and Liu, 2024, Remark 31).
If V 1 , … , V n V_1,\ldots,V_n V 1 , … , V n are independent, centered random variables satisfying E ∣ V i ∣ q < ∞ \mathbb{E}|V_i|^q<\infty E ∣ V i ∣ q < ∞ for every i i i and some q > 2 q>2 q > 2 , then for every x > 0 x>0 x > 0 ,
Pr ( ∣ ∑ i = 1 n V i ∣ > x ) ≤ C q x − q ∑ i = 1 n E ∣ V i ∣ q + 2 exp { − c q x 2 ∑ i = 1 n E ( V i 2 ) } , (4) \operatorname{Pr}\left(\left|\sum_{i=1}^{n}V_i\right|>x\right)\le C_qx^{-q}\sum_{i=1}^{n}\mathbb{E}|V_i|^q+2\exp\left\{-c_q\frac{x^2}{\sum_{i=1}^{n}\mathbb{E}(V_i^2)}\right\}, \tag*{(4)} Pr ( i = 1 ∑ n V i > x ) ≤ C q x − q i = 1 ∑ n E ∣ V i ∣ q + 2 exp { − c q ∑ i = 1 n E ( V i 2 ) x 2 } , (4)
where C q , c q > 0 C_q,c_q>0 C q , c q > 0 depend only on q q q .
Let I n \mathcal{I}_n I n index all scalar trajectory-level quantities to which (A.B1) is applied in this proof. The particular quantity corresponding to each index is specified when it is introduced below.
Since p p p is fixed and each quantity involves at most two response coordinates, ∣ I n ∣ ≤ C d 2 |\mathcal{I}_n|\le Cd^2 ∣ I n ∣ ≤ C d 2 for some constant C < ∞ C<\infty C < ∞ .
Suppose that W i ℓ W_{i\ell} W i ℓ is the trajectory-level contribution indexed by ℓ ∈ I n \ell\in\mathcal{I}_n ℓ ∈ I n and, for some fixed q > 2 q>2 q > 2 ,
sup i , ℓ E ∣ W i ℓ ∣ q ≤ C q . \sup_{i,\ell}\mathbb{E}|W_{i\ell}|^q\le C_q. i , ℓ sup E ∣ W i ℓ ∣ q ≤ C q .
Applying (A.B1) to V i ℓ = W i ℓ − E W i ℓ V_{i\ell}=W_{i\ell}-\mathbb{E}W_{i\ell} V i ℓ = W i ℓ − E W i ℓ and using
E ∣ V i ℓ ∣ q ≤ 2 q E ∣ W i ℓ ∣ q , ∑ i = 1 n E V i ℓ 2 ≤ C n , \mathbb{E}|V_{i\ell}|^q\le2^q\mathbb{E}|W_{i\ell}|^q,\qquad\sum_{i=1}^{n}\mathbb{E}V_{i\ell}^2\le Cn, E ∣ V i ℓ ∣ q ≤ 2 q E ∣ W i ℓ ∣ q , i = 1 ∑ n E V i ℓ 2 ≤ C n ,
gives
Pr ( ∣ 1 n ∑ i = 1 n V i ℓ ∣ > x ) ≤ C q n 1 − q x − q + 2 exp ( − c q n x 2 ) . (5) \operatorname{Pr}\left(\left|\frac{1}{n}\sum_{i=1}^{n}V_{i\ell}\right|>x\right)\le C_qn^{1-q}x^{-q}+2\exp(-c_qnx^2). \tag*{(5)} Pr ( n 1 i = 1 ∑ n V i ℓ > x ) ≤ C q n 1 − q x − q + 2 exp ( − c q n x 2 ) . (5)
Consequently, a union bound and choosing x = M r n x=Mr_n x = M r n yield
Pr ( max ℓ ∈ I n ∣ 1 n ∑ i = 1 n V i ℓ ∣ > M r n ) ≤ C q d 2 n 1 − q ( M r n ) − q + 2 d 2 exp ( − c q n M 2 r n 2 ) ≤ C q M − q d 2 n 1 − q / 2 ( log d ) − q / 2 + 2 d 2 − c q M 2 . (6) \begin{aligned}
\operatorname{Pr}\left(\max_{\ell\in\mathcal{I}_n}\left|\frac{1}{n}\sum_{i=1}^{n}V_{i\ell}\right|>Mr_n\right)\le C_qd^2n^{1-q}(Mr_n)^{-q}+2d^2\exp(-c_qnM^2r_n^2) \\
&\le C_qM^{-q}d^2n^{1-q/2}(\log d)^{-q/2}+2d^{2-c_qM^2}. \tag*{(6)}
\end{aligned} Pr ( ℓ ∈ I n max n 1 i = 1 ∑ n V i ℓ > M r n ) ≤ C q d 2 n 1 − q ( M r n ) − q + 2 d 2 exp ( − c q n M 2 r n 2 ) ≤ C q M − q d 2 n 1 − q /2 ( log d ) − q /2 + 2 d 2 − c q M 2 . (6)
Choose and fix an integer q > 4 κ + 2 q>4\kappa+2 q > 4 κ + 2 . The first term in (A.B3) tends to zero because d ≤ C d n κ d\le C_dn^\kappa d ≤ C d n κ . The second term can be made arbitrarily small by choosing M M M sufficiently large and, for such a fixed M M M , tends to zero when d → ∞ d\to\infty d → ∞ . Thus, any collection of at most order d 2 d^2 d 2 such empirical means is uniformly O p ( r n ) O_p(r_n) O p ( r n ) .
To apply (A.B3), it remains to verify the required q q q ’th-moment bound for trajectory-level contributions appearing in the score and in the moment estimators.
Denote the conditional intensity of the Poisson distribution by
Λ t j ( i ) = exp { O t ( i ) + β 0 j + X t ( i ) T β j + Z t j ( i ) } . \Lambda_{tj}^{(i)}=\exp\left\{O_t^{(i)}+\beta_{0j}+X_t^{(i)\mathsf{T}}\beta_j+Z_{tj}^{(i)}\right\}. Λ t j ( i ) = exp { O t ( i ) + β 0 j + X t ( i ) T β j + Z t j ( i ) } .
For every fixed integer m m m , the m m m ’th moment of a Poisson variable is a polynomial of degree m m m in its intensity, consequently,
E [ ( Y t j ( i ) ) m ∣ X t ( i ) , O t ( i ) , Z t ( i ) ] ≤ C m { 1 + ( Λ t j ( i ) ) m } . \mathbb{E}\left[\left(Y_{tj}^{(i)}\right)^m\mid X_t^{(i)},O_t^{(i)},Z_t^{(i)}\right]\le C_m\left\{1+\left(\Lambda_{tj}^{(i)}\right)^m\right\}. E [ ( Y t j ( i ) ) m ∣ X t ( i ) , O t ( i ) , Z t ( i ) ] ≤ C m { 1 + ( Λ t j ( i ) ) m } .
Conditional independence of the counts given the latent states gives, for j ≠ k j\ne k j = k ,
E [ ( Y t j ( i ) Y t k ( i ) ) q ∣ X t ( i ) , O t ( i ) , Z t ( i ) ] ≤ C q { 1 + ( Λ t j ( i ) ) q } { 1 + ( Λ t k ( i ) ) q } , \mathbb{E}\left[\left(Y_{tj}^{(i)}Y_{tk}^{(i)}\right)^q\mid X_t^{(i)},O_t^{(i)},Z_t^{(i)}\right]\le C_q\left\{1+\left(\Lambda_{tj}^{(i)}\right)^q\right\}\left\{1+\left(\Lambda_{tk}^{(i)}\right)^q\right\}, E [ ( Y t j ( i ) Y t k ( i ) ) q ∣ X t ( i ) , O t ( i ) , Z t ( i ) ] ≤ C q { 1 + ( Λ t j ( i ) ) q } { 1 + ( Λ t k ( i ) ) q } ,
and the same holds at two successive time points. Additionally,
∣ ( Y t j ( i ) ) 2 − Y t j ( i ) ∣ q ≤ C q { ( Y t j ( i ) ) 2 q + ( Y t j ( i ) ) q } , \left|\left(Y_{tj}^{(i)}\right)^2-Y_{tj}^{(i)}\right|^q\le C_q\left\{\left(Y_{tj}^{(i)}\right)^{2q}+\left(Y_{tj}^{(i)}\right)^q\right\}, ( Y t j ( i ) ) 2 − Y t j ( i ) q ≤ C q { ( Y t j ( i ) ) 2 q + ( Y t j ( i ) ) q } ,
hence
E [ ∣ ( Y t j ( i ) ) 2 − Y t j ( i ) ∣ q ∣ X t ( i ) , O t ( i ) , Z t ( i ) ] ≤ C q { 1 + ( Λ t j ( i ) ) 2 q } . \mathbb{E}\left[\left|\left(Y_{tj}^{(i)}\right)^2-Y_{tj}^{(i)}\right|^q\mid X_t^{(i)},O_t^{(i)},Z_t^{(i)}\right]\le C_q\left\{1+\left(\Lambda_{tj}^{(i)}\right)^{2q}\right\}. E [ ( Y t j ( i ) ) 2 − Y t j ( i ) q ∣ X t ( i ) , O t ( i ) , Z t ( i ) ] ≤ C q { 1 + ( Λ t j ( i ) ) 2 q } .
For the latent states, Z t j ( i ) Z_{tj}^{(i)} Z t j ( i ) is centered Gaussian with variance at most K M K_M K M , based on Assumption (E). Since e a ∣ z ∣ ≤ e a z + e − a z e^{a|z|}\le e^{az}+e^{-az} e a ∣ z ∣ ≤ e a z + e − a z , we have
E exp { a ∣ Z t j ( i ) ∣ } ≤ 2 exp { a 2 K M 2 } . \mathbb{E}\exp\left\{a\left|Z_{tj}^{(i)}\right|\right\}\le2\exp\left\{\frac{a^2K_M}{2}\right\}. E exp { a Z t j ( i ) } ≤ 2 exp { 2 a 2 K M } .
Hence, by the Cauchy–Schwarz inequality, for s ∈ { t , t − 1 } s\in\{t,t-1\} s ∈ { t , t − 1 } ,
E exp { a ( ∣ Z t j ( i ) ∣ + ∣ Z s k ( i ) ∣ ) } ≤ [ E e 2 a ∣ Z t j ( i ) ∣ E e 2 a ∣ Z s k ( i ) ∣ ] 1 / 2 ≤ 2 e 2 a 2 K M . \mathbb{E}\exp\left\{a\left(\left|Z_{tj}^{(i)}\right|+\left|Z_{sk}^{(i)}\right|\right)\right\}\le\left[\mathbb{E}e^{2a|Z_{tj}^{(i)}|}\mathbb{E}e^{2a|Z_{sk}^{(i)}|}\right]^{1/2}\le2e^{2a^2K_M}. E exp { a ( Z t j ( i ) + Z s k ( i ) ) } ≤ [ E e 2 a ∣ Z t j ( i ) ∣ E e 2 a ∣ Z s k ( i ) ∣ ] 1/2 ≤ 2 e 2 a 2 K M .
We first apply the concentration bound to the score defining γ ^ \hat{\gamma} γ ^ . For r = 0 , … , p r=0,\ldots,p r = 0 , … , p , define the contribution of trajectory i i i to coordinate r r r of the score for γ j \gamma_j γ j by
W i , j r γ = n N t o t ∑ t = 1 T i X ˉ t r ( i ) [ exp { O t ( i ) + X ˉ t ( i ) T γ j } − Y t j ( i ) ] , W_{i,jr}^{\gamma}=\frac{n}{N_{\mathrm{tot}}}\sum_{t=1}^{T_i}\bar{X}_{tr}^{(i)}\left[\exp\left\{O_t^{(i)}+\bar{X}_t^{(i)\mathsf{T}}\gamma_j\right\}-Y_{tj}^{(i)}\right], W i , j r γ = N tot n t = 1 ∑ T i X ˉ t r ( i ) [ exp { O t ( i ) + X ˉ t ( i ) T γ j } − Y t j ( i ) ] ,
and let W i , j γ = ( W i , j 0 γ , … , W i , j p γ ) T W_{i,j}^{\gamma}=(W_{i,j0}^{\gamma},\ldots,W_{i,jp}^{\gamma})^{\mathsf{T}} W i , j γ = ( W i , j 0 γ , … , W i , j p γ ) T . Then,
∇ L j ( γ j ) = 1 n ∑ i = 1 n W i , j γ . \nabla L_j(\gamma_j)=\frac{1}{n}\sum_{i=1}^{n}W_{i,j}^{\gamma}. ∇ L j ( γ j ) = n 1 i = 1 ∑ n W i , j γ .
The definition of γ j \gamma_j γ j and Assumption (B) imply that E W i , j r γ = 0 \mathbb{E}W_{i,jr}^{\gamma}=0 E W i , j r γ = 0 . Compactness of Γ \Gamma Γ bounds the coefficients in γ j \gamma_j γ j uniformly in j j j . Since β 0 j = γ 0 j − Σ Z , j j / 2 \beta_{0j}=\gamma_{0j}-\Sigma_{Z,jj}/2 β 0 j = γ 0 j − Σ Z , j j /2 , the coefficients in the intensity are also uniformly bounded. The conditional Poisson moment bound, compactness of Γ \Gamma Γ , and boundedness of the offsets give the following bound for the conditional q q q 'th moment of each score summand:
C q ∥ X ˉ t ( i ) ∥ 2 q exp { C q ∥ X ˉ t ( i ) ∥ 2 + q ∣ Z t j ( i ) ∣ } . C_q\left\|\bar{X}_t^{(i)}\right\|_2^q\exp\left\{C_q\left\|\bar{X}_t^{(i)}\right\|_2+q\left|Z_{tj}^{(i)}\right|\right\}. C q X ˉ t ( i ) 2 q exp { C q X ˉ t ( i ) 2 + q Z t j ( i ) } .
Its expectation is uniformly bounded by Cauchy–Schwarz, Assumption (C), and the Gaussian exponential-moment bound above. Since T i ≤ T max T_i\leq T_{\max} T i ≤ T m a x and N t o t ≥ 2 n N_{\mathrm{tot}}\geq2n N tot ≥ 2 n , the moment bounds above and Jensen’s inequality give
sup i , j , r E ∣ W i , j r γ ∣ q ≤ C q . \sup_{i,j,r}\mathbb{E}\left|W_{i,jr}^{\gamma}\right|^q\leq C_q. i , j , r sup E W i , j r γ q ≤ C q .
There are d ( p + 1 ) d(p+1) d ( p + 1 ) score coordinates and p p p is fixed. Therefore, (A.B3) gives
max 1 ≤ j ≤ d ∥ ∇ L j ( γ j ) ∥ 2 = O p ( r n ) . (7) \max_{1\leq j\leq d}\left\|\nabla L_j(\gamma_j)\right\|_2=O_p(r_n). \tag*{(7)} 1 ≤ j ≤ d max ∥ ∇ L j ( γ j ) ∥ 2 = O p ( r n ) . (7)
The Hessian is
∇ 2 L j ( g ) = 1 N t o t ∑ i = 1 n ∑ t = 1 T i exp { O t ( i ) + X ˉ t ( i ) ⊤ g } X ˉ t ( i ) X ˉ t ( i ) ⊤ . \nabla^2L_j(g)=\frac{1}{N_{\mathrm{tot}}}\sum_{i=1}^{n}\sum_{t=1}^{T_i}\exp\left\{O_t^{(i)}+\bar{X}_t^{(i)\top}g\right\}\bar{X}_t^{(i)}\bar{X}_t^{(i)\top}. ∇ 2 L j ( g ) = N tot 1 i = 1 ∑ n t = 1 ∑ T i exp { O t ( i ) + X ˉ t ( i ) ⊤ g } X ˉ t ( i ) X ˉ t ( i ) ⊤ .
Differentiating the Hessian with respect to coordinate g r g_r g r gives
∂ ∂ g r ∇ 2 L j ( g ) = 1 N t o t ∑ i = 1 n ∑ t = 1 T i exp { O t ( i ) + X ˉ t ( i ) ⊤ g } X ˉ t r ( i ) X ˉ t ( i ) X ˉ t ( i ) ⊤ . \frac{\partial}{\partial g_r}\nabla^2L_j(g)=\frac{1}{N_{\mathrm{tot}}}\sum_{i=1}^{n}\sum_{t=1}^{T_i}\exp\left\{O_t^{(i)}+\bar{X}_t^{(i)\top}g\right\}\bar{X}_{tr}^{(i)}\bar{X}_t^{(i)}\bar{X}_t^{(i)\top}. ∂ g r ∂ ∇ 2 L j ( g ) = N tot 1 i = 1 ∑ n t = 1 ∑ T i exp { O t ( i ) + X ˉ t ( i ) ⊤ g } X ˉ t r ( i ) X ˉ t ( i ) X ˉ t ( i ) ⊤ .
Since Γ \Gamma Γ is compact and the offsets are bounded, the absolute values of the scalar summands in the Hessian and its derivatives are bounded, uniformly over g ∈ Γ g\in\Gamma g ∈ Γ , by
C ( 1 + ∥ X ˉ t ( i ) ∥ 2 3 ) exp { C ∥ X ˉ t ( i ) ∥ 2 } . C\left(1+\left\|\bar{X}_t^{(i)}\right\|_2^3\right)\exp\left\{C\left\|\bar{X}_t^{(i)}\right\|_2\right\}. C ( 1 + X ˉ t ( i ) 2 3 ) exp { C X ˉ t ( i ) 2 } .
Assumption (C) gives uniformly bounded q q q 'th moments for these bounds. Their empirical averages are O p ( 1 ) O_p(1) O p ( 1 ) by Markov’s inequality, and their expectations are uniformly bounded. Compactness gives a finite net of Γ \Gamma Γ at any fixed approximation accuracy, with cardinality independent of n , d n,d n , d . The number of scalar Hessian entries evaluated at its points is therefore at most C d 2 Cd^2 C d 2 , with C C C allowed to depend on the net. Applying (A.B3) at these points gives an o p ( 1 ) o_p(1) o p ( 1 ) bound there. The derivative bounds control the difference between any point and a net point. Taking increasingly fine nets after n → ∞ n \to\infty n → ∞ therefore gives
max 1 ≤ j ≤ d sup g ∈ Γ ∥ ∇ 2 L j ( g ) − E ∇ 2 L j ( g ) ∥ 2 = o p ( 1 ) . \max_{1 \le j \le d} \sup_{g \in\Gamma} \left\lVert\nabla^2 L_j(g) - \mathbb{E}\nabla^2 L_j(g) \right\rVert_2 = o_p(1). 1 ≤ j ≤ d max g ∈ Γ sup ∇ 2 L j ( g ) − E ∇ 2 L j ( g ) 2 = o p ( 1 ) .
For any j j j and g ∈ Γ g \in\Gamma g ∈ Γ ,
λ min { ∇ 2 L j ( g ) } ≥ λ min { E ∇ 2 L j ( g ) } − ∥ ∇ 2 L j ( g ) − E ∇ 2 L j ( g ) ∥ 2 . \lambda_{\min}\left\{\nabla^2 L_j(g)\right\} \ge\lambda_{\min}\left\{\mathbb{E}\nabla^2 L_j(g)\right\} - \left\lVert\nabla^2 L_j(g) - \mathbb{E}\nabla^2 L_j(g) \right\rVert_2. λ m i n { ∇ 2 L j ( g ) } ≥ λ m i n { E ∇ 2 L j ( g ) } − ∇ 2 L j ( g ) − E ∇ 2 L j ( g ) 2 .
Therefore, Assumption (D) together with the preceding uniform o p ( 1 ) o_p(1) o p ( 1 ) bound imply that, with probability tending to one,
min 1 ≤ j ≤ d inf g ∈ Γ λ min { ∇ 2 L j ( g ) } ≥ c γ 2 . (8) \min_{1 \le j \le d} \inf_{g \in\Gamma} \lambda_{\min}\left\{\nabla^2 L_j(g)\right\} \ge\frac{c_\gamma}{2}. \tag*{(8)} 1 ≤ j ≤ d min g ∈ Γ inf λ m i n { ∇ 2 L j ( g ) } ≥ 2 c γ . (8)
Uniform separation from the boundary of Γ \Gamma Γ gives a fixed ϵ > 0 \epsilon> 0 ϵ > 0 such that the ball of radius ϵ \epsilon ϵ around each γ j \gamma_j γ j lies in Γ \Gamma Γ . By (A.B5), on its boundary,
L j ( g ) − L j ( γ j ) ≥ − ϵ ∥ ∇ L j ( γ j ) ∥ 2 + c γ 4 ϵ 2 . L_j(g) - L_j(\gamma_j) \ge-\epsilon\left\lVert\nabla L_j(\gamma_j) \right\rVert_2 + \frac{c_\gamma}{4}\epsilon^2. L j ( g ) − L j ( γ j ) ≥ − ϵ ∥ ∇ L j ( γ j ) ∥ 2 + 4 c γ ϵ 2 .
The score bound (A.B4) makes this positive simultaneously for all j j j with probability tending to one. Thus, each ball has an interior minimizer. Convexity of L j L_j L j and strict convexity within Γ \Gamma Γ make it the unique global minimizer γ ^ j \widehat{\gamma}_j γ j . Consequently,
Pr ( γ ^ j ∈ Γ for every 1 ≤ j ≤ d ) ⟶ 1. \Pr\left(\widehat{\gamma}_j \in\Gamma\text{ for every } 1 \le j \le d\right) \longrightarrow1. Pr ( γ j ∈ Γ for every 1 ≤ j ≤ d ) ⟶ 1.
On this event, strong convexity and the minimizing property of γ ^ j \widehat{\gamma}_j γ j imply
0 ≥ L j ( γ ^ j ) − L j ( γ j ) ≥ ∇ L j ( γ j ) ⊤ ( γ ^ j − γ j ) + c γ 4 ∥ γ ^ j − γ j ∥ 2 2 . 0 \ge L_j(\widehat{\gamma}_j) - L_j(\gamma_j) \ge\nabla L_j(\gamma_j)^\top(\widehat{\gamma}_j - \gamma_j) + \frac{c_\gamma}{4}\left\lVert\widehat{\gamma}_j - \gamma_j \right\rVert_2^2. 0 ≥ L j ( γ j ) − L j ( γ j ) ≥ ∇ L j ( γ j ) ⊤ ( γ j − γ j ) + 4 c γ ∥ γ j − γ j ∥ 2 2 .
Consequently,
∥ γ ^ j − γ j ∥ 2 ≤ 4 c γ ∥ ∇ L j ( γ j ) ∥ 2 , \left\lVert\widehat{\gamma}_j - \gamma_j \right\rVert_2 \le\frac{4}{c_\gamma}\left\lVert\nabla L_j(\gamma_j) \right\rVert_2, ∥ γ j − γ j ∥ 2 ≤ c γ 4 ∥ ∇ L j ( γ j ) ∥ 2 ,
and (A.B4) yields
max 1 ≤ j ≤ d ∥ γ ^ j − γ j ∥ 2 = O p ( r n ) . (9) \max_{1 \le j \le d} \left\lVert\widehat{\gamma}_j - \gamma_j \right\rVert_2 = O_p(r_n). \tag*{(9)} 1 ≤ j ≤ d max ∥ γ j − γ j ∥ 2 = O p ( r n ) . (9)
We now apply the same concentration result to the moment estimators. Define the trajectory-level contributions evaluated at the true γ \gamma γ by the following expressions. The first is for j ≠ k j \ne k j = k , whereas the lagged expression is for all j , k j,k j , k ,
W i , j k Σ = n N t o t ∑ t = 1 T i Y t j ( i ) Y t k ( i ) exp { − 2 O t ( i ) − X ˉ t ( i ) ⊤ γ j − X ˉ t ( i ) ⊤ γ k } . W^\Sigma_{i,jk}=\frac{n}{N_{\mathrm{tot}}}\sum_{t=1}^{T_i}Y_{tj}^{(i)}Y_{tk}^{(i)}\exp\left\{-2O_t^{(i)}-\bar{X}_t^{(i)\top}\gamma_j-\bar{X}_t^{(i)\top}\gamma_k\right\}. W i , j k Σ = N tot n t = 1 ∑ T i Y t j ( i ) Y t k ( i ) exp { − 2 O t ( i ) − X ˉ t ( i ) ⊤ γ j − X ˉ t ( i ) ⊤ γ k } .
W i , j j Σ = n N t o t ∑ t = 1 T i { ( Y t j ( i ) ) 2 − Y t j ( i ) } exp { − 2 O t ( i ) − 2 X ˉ t ( i ) ⊤ γ j } . W^\Sigma_{i,jj}=\frac{n}{N_{\mathrm{tot}}}\sum_{t=1}^{T_i}\left\{\left(Y_{tj}^{(i)}\right)^2-Y_{tj}^{(i)}\right\}\exp\left\{-2O_t^{(i)}-2\bar{X}_t^{(i)\top}\gamma_j\right\}. W i , j j Σ = N tot n t = 1 ∑ T i { ( Y t j ( i ) ) 2 − Y t j ( i ) } exp { − 2 O t ( i ) − 2 X ˉ t ( i ) ⊤ γ j } .
W i , j k P = n N t o t − n ∑ t = 2 T i Y t j ( i ) Y t − 1 , k ( i ) exp { − O t ( i ) − O t − 1 ( i ) − X ˉ t ( i ) ⊤ γ j − X ˉ t − 1 ( i ) ⊤ γ k } . W^P_{i,jk}=\frac{n}{N_{\mathrm{tot}}-n}\sum_{t=2}^{T_i}Y_{tj}^{(i)}Y_{t-1,k}^{(i)}\exp\left\{-O_t^{(i)}-O_{t-1}^{(i)}-\bar{X}_t^{(i)\top}\gamma_j-\bar{X}_{t-1}^{(i)\top}\gamma_k\right\}. W i , j k P = N tot − n n t = 2 ∑ T i Y t j ( i ) Y t − 1 , k ( i ) exp { − O t ( i ) − O t − 1 ( i ) − X ˉ t ( i ) ⊤ γ j − X ˉ t − 1 ( i ) ⊤ γ k } .
We now verify the moment bound for these specific summands. For example, for j ≠ k j\ne k j = k , the conditional q q q ’th moment of the summand in W i , j k Σ W^\Sigma_{i,jk} W i , j k Σ satisfies
E [ ∣ Y t j ( i ) Y t k ( i ) e − 2 O t ( i ) − X ˉ t ( i ) ⊤ γ j − X ˉ t ( i ) ⊤ γ k ∣ q ∣ X t ( i ) , O t ( i ) , Z t ( i ) ] ≤ C q e − q { 2 O t ( i ) + X ˉ t ( i ) ⊤ ( γ j + γ k ) } { 1 + ( Λ t j ( i ) ) q } { 1 + ( Λ t k ( i ) ) q } . \mathbb{E}\left[\left.\left|Y_{tj}^{(i)}Y_{tk}^{(i)}e^{-2O_t^{(i)}-\bar{X}_t^{(i)\top}\gamma_j-\bar{X}_t^{(i)\top}\gamma_k}\right|^q\right|X_t^{(i)},O_t^{(i)},Z_t^{(i)}\right]
\le C_qe^{-q\{2O_t^{(i)}+\bar{X}_t^{(i)\top}(\gamma_j+\gamma_k)\}}\left\{1+\left(\Lambda_{tj}^{(i)}\right)^q\right\}\left\{1+\left(\Lambda_{tk}^{(i)}\right)^q\right\}. E [ Y t j ( i ) Y t k ( i ) e − 2 O t ( i ) − X ˉ t ( i ) ⊤ γ j − X ˉ t ( i ) ⊤ γ k q X t ( i ) , O t ( i ) , Z t ( i ) ] ≤ C q e − q { 2 O t ( i ) + X ˉ t ( i ) ⊤ ( γ j + γ k )} { 1 + ( Λ t j ( i ) ) q } { 1 + ( Λ t k ( i ) ) q } .
Expanding the right-hand side gives finitely many exponential terms with bounded covariate coefficients and fixed multiples of the latent coordinates. Their expectations are uniformly finite by Assumption (C), the Gaussian exponential-moment bounds, and Cauchy–Schwarz inequality. The same calculation, using the factorial-moment bound above when j = k j=k j = k and the two-time-point bound for the lagged products, gives a uniformly bounded q q q ’th moment for every time-point summand in W i , j k Σ W^\Sigma_{i,jk} W i , j k Σ and W i , j k P W^P_{i,jk} W i , j k P .
Since T i ≤ T max T_i\le T_{\max} T i ≤ T m a x , Jensen’s inequality gives
∣ ∑ t = 1 T i a t ∣ q ≤ T i q − 1 ∑ t = 1 T i ∣ a t ∣ q ≤ T max q − 1 ∑ t = 1 T i ∣ a t ∣ q . \left|\sum_{t=1}^{T_i}a_t\right|^q\le T_i^{q-1}\sum_{t=1}^{T_i}|a_t|^q\le T_{\max}^{q-1}\sum_{t=1}^{T_i}|a_t|^q. t = 1 ∑ T i a t q ≤ T i q − 1 t = 1 ∑ T i ∣ a t ∣ q ≤ T m a x q − 1 t = 1 ∑ T i ∣ a t ∣ q .
Moreover, N t o t ≥ 2 n N_{\mathrm{tot}}\ge2n N tot ≥ 2 n , so the factors n / N t o t n/N_{\mathrm{tot}} n / N tot and n / ( N t o t − n ) n/(N_{\mathrm{tot}}-n) n / ( N tot − n ) in the definitions of W i , j k Σ W^\Sigma_{i,jk} W i , j k Σ and W i , j k P W^P_{i,jk} W i , j k P are uniformly bounded. Therefore,
sup i , j , k E ∣ W i , j k Σ ∣ q ≤ C q , sup i , j , k E ∣ W i , j k P ∣ q ≤ C q . \sup_{i,j,k}\mathbb{E}|W^\Sigma_{i,jk}|^q\le C_q,\qquad\sup_{i,j,k}\mathbb{E}|W^P_{i,jk}|^q\le C_q. i , j , k sup E ∣ W i , j k Σ ∣ q ≤ C q , i , j , k sup E ∣ W i , j k P ∣ q ≤ C q .
The oracle empirical moments, obtained by using the true γ \gamma γ in place of γ ^ \hat{\gamma} γ ^ , can therefore be written as
M ~ j k Σ = 1 n ∑ i = 1 n W i , j k Σ , M ~ j k P = 1 n ∑ i = 1 n W i , j k P . \widetilde{M}^\Sigma_{jk}=\frac{1}{n}\sum_{i=1}^nW^\Sigma_{i,jk},\qquad\widetilde{M}^P_{jk}=\frac{1}{n}\sum_{i=1}^nW^P_{i,jk}. M j k Σ = n 1 i = 1 ∑ n W i , j k Σ , M j k P = n 1 i = 1 ∑ n W i , j k P .
The moments identities established in Appendix A imply that, with M j k Σ = e Σ Z , j k M^\Sigma_{jk}=e^{\Sigma_{Z,jk}} M j k Σ = e Σ Z , j k and M j k P = M^P_{jk}= M j k P = e j k P , e^{P}_{jk}, e j k P ,
1 n ∑ i = 1 n E W i , j k Σ = M j k Σ , 1 n ∑ i = 1 n E W i , j k P = M j k P . \frac{1}{n}\sum_{i=1}^{n}\mathbb{E}W^{\Sigma}_{i,jk}=M^{\Sigma}_{jk}, \qquad\frac{1}{n}\sum_{i=1}^{n}\mathbb{E}W^{P}_{i,jk}=M^{P}_{jk}. n 1 i = 1 ∑ n E W i , j k Σ = M j k Σ , n 1 i = 1 ∑ n E W i , j k P = M j k P .
It follows directly that
M ~ j k Σ − M j k Σ = 1 n ∑ i = 1 n ( W i , j k Σ − E W i , j k Σ ) , M ~ j k P − M j k P = 1 n ∑ i = 1 n ( W i , j k P − E W i , j k P ) . \widetilde{M}^{\Sigma}_{jk}-M^{\Sigma}_{jk}=\frac{1}{n}\sum_{i=1}^{n}\left(W^{\Sigma}_{i,jk}-\mathbb{E}W^{\Sigma}_{i,jk}\right), \qquad\widetilde{M}^{P}_{jk}-M^{P}_{jk}=\frac{1}{n}\sum_{i=1}^{n}\left(W^{P}_{i,jk}-\mathbb{E}W^{P}_{i,jk}\right). M j k Σ − M j k Σ = n 1 i = 1 ∑ n ( W i , j k Σ − E W i , j k Σ ) , M j k P − M j k P = n 1 i = 1 ∑ n ( W i , j k P − E W i , j k P ) .
There are at most order d 2 d^2 d 2 such moment entries. Applying (A.B3) to the preceding centered representations therefore gives
∥ M ~ Σ − M Σ ∥ max = O p ( r n ) , ∥ M ~ P − M P ∥ max = O p ( r n ) . (10) \left\|\widetilde{M}^{\Sigma}-M^{\Sigma}\right\|_{\max}=O_p(r_n), \qquad\left\|\widetilde{M}^{P}-M^{P}\right\|_{\max}=O_p(r_n). \tag*{(10)} M Σ − M Σ m a x = O p ( r n ) , M P − M P m a x = O p ( r n ) . (10)
It remains to account for replacing γ \gamma γ by γ ^ \widehat{\gamma} γ in the empirical moments. For j ≠ k j\ne k j = k , consider the empirical moment as a function of two arguments:
M ^ j k Σ ( g , h ) : = 1 N t o t ∑ i = 1 n ∑ t = 1 T i Y t j ( i ) Y t k ( i ) exp { − 2 O t ( i ) − X ‾ t ( i ) ⊤ g − X ‾ t ( i ) ⊤ h } . \widehat{M}^{\Sigma}_{jk}(g,h):=\frac{1}{N_{\mathrm{tot}}}\sum_{i=1}^{n}\sum_{t=1}^{T_i}Y_{tj}^{(i)}Y_{tk}^{(i)}\exp\left\{-2O_t^{(i)}-\overline{X}_t^{(i)\top}g-\overline{X}_t^{(i)\top}h\right\}. M j k Σ ( g , h ) := N tot 1 i = 1 ∑ n t = 1 ∑ T i Y t j ( i ) Y t k ( i ) exp { − 2 O t ( i ) − X t ( i ) ⊤ g − X t ( i ) ⊤ h } .
Then M ~ j k Σ \widetilde{M}^{\Sigma}_{jk} M j k Σ is obtained by evaluating this function at ( γ j , γ k ) (\gamma_j,\gamma_k) ( γ j , γ k ) , whereas M ^ j k Σ \widehat{M}^{\Sigma}_{jk} M j k Σ is obtained by evaluating it at ( γ ^ j , γ ^ k ) (\widehat{\gamma}_j,\widehat{\gamma}_k) ( γ j , γ k ) .
Differentiation with respect to either g g g or h h h multiplies each summand by − X ‾ t ( i ) -\overline{X}_t^{(i)} − X t ( i ) . Compactness of Γ \Gamma Γ and boundedness of the offsets therefore give
sup g , h ∈ Γ { ∥ ∇ g M ^ j k Σ ( g , h ) ∥ 2 + ∥ ∇ h M ^ j k Σ ( g , h ) ∥ 2 } ≤ 2 N t o t ∑ i = 1 n ∑ t = 1 T i ∥ X ‾ t ( i ) ∥ 2 Y t j ( i ) Y t k ( i ) exp { 2 K O + C ∥ X ‾ t ( i ) ∥ 2 } . \sup_{g,h\in\Gamma}\left\{\left\|\nabla_g\widehat{M}^{\Sigma}_{jk}(g,h)\right\|_2+\left\|\nabla_h\widehat{M}^{\Sigma}_{jk}(g,h)\right\|_2\right\}
\le\frac{2}{N_{\mathrm{tot}}}\sum_{i=1}^{n}\sum_{t=1}^{T_i}\left\|\overline{X}_t^{(i)}\right\|_2Y_{tj}^{(i)}Y_{tk}^{(i)}\exp\left\{2K_O+C\left\|\overline{X}_t^{(i)}\right\|_2\right\}. g , h ∈ Γ sup { ∇ g M j k Σ ( g , h ) 2 + ∇ h M j k Σ ( g , h ) 2 } ≤ N tot 2 i = 1 ∑ n t = 1 ∑ T i X t ( i ) 2 Y t j ( i ) Y t k ( i ) exp { 2 K O + C X t ( i ) 2 } .
The extra covariate factor and the exponential factor are also controlled by Assumption (C). The conditional Poisson bounds and Cauchy Schwartz inequality thus give bounded q q q ’th moments for the trajectory contributions on the right, by the same calculation used above. Their expectations are uniformly bounded, and (A.B3) bounds their centered averages uniformly over j ≠ k j\ne k j = k . Consequently,
max j ≠ k sup g , h ∈ Γ { ∥ ∇ g M ^ j k Σ ( g , h ) ∥ 2 + ∥ ∇ h M ^ j k Σ ( g , h ) ∥ 2 } = O p ( 1 ) . \max_{j\ne k}\sup_{g,h\in\Gamma}\left\{\left\|\nabla_g\widehat{M}^{\Sigma}_{jk}(g,h)\right\|_2+\left\|\nabla_h\widehat{M}^{\Sigma}_{jk}(g,h)\right\|_2\right\}=O_p(1). j = k max g , h ∈ Γ sup { ∇ g M j k Σ ( g , h ) 2 + ∇ h M j k Σ ( g , h ) 2 } = O p ( 1 ) .
On the high probability event established above, both γ j \gamma_j γ j and γ ^ j \widehat{\gamma}_j γ j belong to Γ \Gamma Γ for every j j j .
Since Γ \Gamma Γ is convex, the line segment joining them is also contained in Γ \Gamma Γ . The mean-value theorem then gives, uniformly over j ≠ k j \ne k j = k ,
∣ M ^ j k Σ − M ~ j k Σ ∣ ≤ O p ( 1 ) ( ∥ γ ^ j − γ j ∥ 2 + ∥ γ ^ k − γ k ∥ 2 ) . \left|\widehat{M}^{\Sigma}_{jk}-\widetilde{M}^{\Sigma}_{jk}\right| \le O_p(1)\left(\|\widehat{\gamma}_j-\gamma_j\|_2+\|\widehat{\gamma}_k-\gamma_k\|_2\right). M j k Σ − M j k Σ ≤ O p ( 1 ) ( ∥ γ j − γ j ∥ 2 + ∥ γ k − γ k ∥ 2 ) .
When j = k j=k j = k , the corresponding empirical moment is
M ^ j j Σ ( g ) : = 1 N t o t ∑ i = 1 n ∑ t = 1 T i { ( Y t j ( i ) ) 2 − Y t j ( i ) } exp { − 2 O t ( i ) − 2 X ‾ t ( i ) ⊤ g } . \widehat{M}^{\Sigma}_{jj}(g):=\frac{1}{N_{\mathrm{tot}}}\sum_{i=1}^{n}\sum_{t=1}^{T_i}\left\{\left(Y_{t j}^{(i)}\right)^2-Y_{t j}^{(i)}\right\}\exp\left\{-2O_t^{(i)}-2\overline{X}_t^{(i)\top}g\right\}. M j j Σ ( g ) := N tot 1 i = 1 ∑ n t = 1 ∑ T i { ( Y t j ( i ) ) 2 − Y t j ( i ) } exp { − 2 O t ( i ) − 2 X t ( i ) ⊤ g } .
Its derivative multiplies each summand by − 2 X ‾ t ( i ) -2\overline{X}_t^{(i)} − 2 X t ( i ) . The same argument therefore gives
max 1 ≤ j ≤ d ∣ M ^ j j Σ − M ~ j j Σ ∣ ≤ O p ( 1 ) max 1 ≤ j ≤ d ∥ γ ^ j − γ j ∥ 2 . \max_{1\le j\le d}\left|\widehat{M}^{\Sigma}_{jj}-\widetilde{M}^{\Sigma}_{jj}\right|\le O_p(1)\max_{1\le j\le d}\|\widehat{\gamma}_j-\gamma_j\|_2. 1 ≤ j ≤ d max M j j Σ − M j j Σ ≤ O p ( 1 ) 1 ≤ j ≤ d max ∥ γ j − γ j ∥ 2 .
For the lagged moment, the corresponding function is
M ^ j k P ( g , h ) : = 1 N t o t − n ∑ i = 1 n ∑ t = 2 T i Y t j ( i ) Y t − 1 , k ( i ) exp { − O t ( i ) − O t − 1 ( i ) − X ‾ t ( i ) ⊤ g − X ‾ t − 1 ( i ) ⊤ h } . \widehat{M}^{P}_{jk}(g,h):=\frac{1}{N_{\mathrm{tot}}-n}\sum_{i=1}^{n}\sum_{t=2}^{T_i}Y_{t j}^{(i)}Y_{t-1,k}^{(i)}\exp\left\{-O_t^{(i)}-O_{t-1}^{(i)}-\overline{X}_t^{(i)\top}g-\overline{X}_{t-1}^{(i)\top}h\right\}. M j k P ( g , h ) := N tot − n 1 i = 1 ∑ n t = 2 ∑ T i Y t j ( i ) Y t − 1 , k ( i ) exp { − O t ( i ) − O t − 1 ( i ) − X t ( i ) ⊤ g − X t − 1 ( i ) ⊤ h } .
Its derivatives multiply the summands by − X ‾ t ( i ) -\overline{X}_t^{(i)} − X t ( i ) and − X ‾ t − 1 ( i ) -\overline{X}_{t-1}^{(i)} − X t − 1 ( i ) , respectively. The same moment and concentration bounds give
max 1 ≤ j , k ≤ d ∣ M ^ j k P − M ~ j k P ∣ ≤ O p ( 1 ) max 1 ≤ j ≤ d ∥ γ ^ j − γ j ∥ 2 . \max_{1\le j,k\le d}\left|\widehat{M}^{P}_{jk}-\widetilde{M}^{P}_{jk}\right|\le O_p(1)\max_{1\le j\le d}\|\widehat{\gamma}_j-\gamma_j\|_2. 1 ≤ j , k ≤ d max M j k P − M j k P ≤ O p ( 1 ) 1 ≤ j ≤ d max ∥ γ j − γ j ∥ 2 .
Using Eq. (A.B6), we conclude that
∥ M ^ Σ − M ~ Σ ∥ max = O p ( r n ) , ∥ M ^ P − M ~ P ∥ max = O p ( r n ) . \left\|\widehat{M}^{\Sigma}-\widetilde{M}^{\Sigma}\right\|_{\max}=O_p(r_n),\qquad\left\|\widehat{M}^{P}-\widetilde{M}^{P}\right\|_{\max}=O_p(r_n). M Σ − M Σ m a x = O p ( r n ) , M P − M P m a x = O p ( r n ) .
Combining these bounds with (A.B7) gives
∥ M ^ Σ − M Σ ∥ max = O p ( r n ) , ∥ M ^ P − M P ∥ max = O p ( r n ) . \left\|\widehat{M}^{\Sigma}-M^{\Sigma}\right\|_{\max}=O_p(r_n),\qquad\left\|\widehat{M}^{P}-M^{P}\right\|_{\max}=O_p(r_n). M Σ − M Σ m a x = O p ( r n ) , M P − M P m a x = O p ( r n ) .
Assumption (E) gives M j k Σ ≥ e − K M M^{\Sigma}_{jk}\ge e^{-K_M} M j k Σ ≥ e − K M and M j k P ≥ e − K M M^{P}_{jk}\ge e^{-K_M} M j k P ≥ e − K M . Since r n → 0 r_n\to0 r n → 0 , all empirical moments are at least e − K M / 2 e^{-K_M}/2 e − K M /2 with probability tending to one. On this event the logarithm is uniformly Lipschitz. Applying it entrywise proves the two remaining rates in Lemma 2 . The estimators are thus well defined on an event whose probability tends to one.
Lemma 3: Let H ^ j = Σ ^ Z , S ( j ) ⊤ Σ ^ Z , S ( j ) \widehat{H}_j=\widehat{\Sigma}^{\top}_{Z,S(j)}\widehat{\Sigma}_{Z,S(j)} H j = Σ Z , S ( j ) ⊤ Σ Z , S ( j ) . Under Assumptions (A)–(G) and (I)–(K), uniformly
over rows with s j > 0 s_j > 0 s j > 0 ,
∥ H ^ j − H j ∥ max = O p ( ρ n ) , ∥ H ^ j − 1 ∥ ∞ = O p ( s j ) . \left\|\widehat{H}_j-H_j\right\|_{\max}=O_p(\rho_n),\qquad\left\|\widehat{H}_j^{-1}\right\|_\infty=O_p(\sqrt{s_j}). H j − H j m a x = O p ( ρ n ) , H j − 1 ∞ = O p ( s j ) .
Moreover, all estimated diagonal entries are positive with probability tending to one, and the penalty weights satisfy
max j , k ∣ w ^ j k − w j k ∣ = O p ( r n ) . \max_{j,k}\left|\widehat{w}_{jk}-w_{jk}\right|=O_p(r_n). j , k max ∣ w j k − w j k ∣ = O p ( r n ) .
Proof. [Proof of Lemma 3 ]
Writing Δ ^ Σ Z = Σ ^ Z − Σ Z \widehat{\Delta}_{\Sigma_Z}=\widehat{\Sigma}_Z-\Sigma_Z Δ Σ Z = Σ Z − Σ Z , we have
∥ H ^ j − H j ∥ max = ∥ Δ ^ Σ Z , S ( j ) T Δ ^ Σ Z , S ( j ) + Σ Z , S ( j ) T Δ ^ Σ Z , S ( j ) + Δ ^ Σ Z , S ( j ) T Σ Z , S ( j ) ∥ max ≤ ∥ Δ ^ Σ Z , S ( j ) ∥ 1 ∥ Δ ^ Σ Z , S ( j ) ∥ max + 2 ∥ Σ Z , S ( j ) ∥ ∞ ∥ Δ ^ Σ Z , S ( j ) ∥ max ≤ d ∥ Δ ^ Σ Z ∥ max 2 + 2 ∥ Σ Z ∥ 1 ∥ Δ ^ Σ Z ∥ max ≤ d ∥ Δ ^ Σ Z ∥ max 2 + 2 C 1 ∥ Δ ^ Σ Z ∥ max . \begin{aligned}
\left\|\widehat{H}_j-H_j\right\|_{\max}
=
\left\|\widehat{\Delta}_{\Sigma_Z,S(j)}^{\mathsf T}\widehat{\Delta}_{\Sigma_Z,S(j)}
+\Sigma_{Z,S(j)}^{\mathsf T}\widehat{\Delta}_{\Sigma_Z,S(j)}
+\widehat{\Delta}_{\Sigma_Z,S(j)}^{\mathsf T}\Sigma_{Z,S(j)}\right\|_{\max} \\
&\leq
\left\|\widehat{\Delta}_{\Sigma_Z,S(j)}\right\|_1
\left\|\widehat{\Delta}_{\Sigma_Z,S(j)}\right\|_{\max}
+2\left\|\Sigma_{Z,S(j)}\right\|_\infty
\left\|\widehat{\Delta}_{\Sigma_Z,S(j)}\right\|_{\max} \\
&\leq d\left\|\widehat{\Delta}_{\Sigma_Z}\right\|_{\max}^2
+2\left\|\Sigma_Z\right\|_1\left\|\widehat{\Delta}_{\Sigma_Z}\right\|_{\max} \\
&\leq d\left\|\widehat{\Delta}_{\Sigma_Z}\right\|_{\max}^2
+2C_1\left\|\widehat{\Delta}_{\Sigma_Z}\right\|_{\max}.
\end{aligned} H j − H j m a x = Δ Σ Z , S ( j ) T Δ Σ Z , S ( j ) + Σ Z , S ( j ) T Δ Σ Z , S ( j ) + Δ Σ Z , S ( j ) T Σ Z , S ( j ) m a x ≤ Δ Σ Z , S ( j ) 1 Δ Σ Z , S ( j ) m a x + 2 Σ Z , S ( j ) ∞ Δ Σ Z , S ( j ) m a x ≤ d Δ Σ Z m a x 2 + 2 ∥ Σ Z ∥ 1 Δ Σ Z m a x ≤ d Δ Σ Z m a x 2 + 2 C 1 Δ Σ Z m a x .
Combining the preceding bound with Lemma 2 , uniformly over j j j ,
∥ H ^ j − H j ∥ max = O p ( d r n 2 + r n ) = O p ( ρ n ) . \left\|\widehat{H}_j-H_j\right\|_{\max}
=O_p(dr_n^2+r_n)=O_p(\rho_n). H j − H j m a x = O p ( d r n 2 + r n ) = O p ( ρ n ) .
Now, we provide a lower bound for λ min ( H ^ j ) \lambda_{\min}(\widehat{H}_j) λ m i n ( H j ) .
λ min ( H ^ j ) = λ min ( H ^ j − H j + H j ) ≥ λ min ( H j ) + λ min ( H ^ j − H j ) ≥ C 2 − ∥ H ^ j − H j ∥ 2 ≥ C 2 − s j ∥ H ^ j − H j ∥ max . \begin{aligned}
\lambda_{\min}(\widehat{H}_j)
=\lambda_{\min}\left(\widehat{H}_j-H_j+H_j\right) \\
&\geq\lambda_{\min}(H_j)+\lambda_{\min}(\widehat{H}_j-H_j) \\
&\geq C_2-\left\|\widehat{H}_j-H_j\right\|_2 \\
&\geq C_2-s_j\left\|\widehat{H}_j-H_j\right\|_{\max}.
\end{aligned} λ m i n ( H j ) = λ m i n ( H j − H j + H j ) ≥ λ m i n ( H j ) + λ m i n ( H j − H j ) ≥ C 2 − H j − H j 2 ≥ C 2 − s j H j − H j m a x .
Since s j ≤ s n s_j\leq s_n s j ≤ s n , Assumption (K) gives s j ρ n ≤ s n ρ n = o ( min j λ j ) s_j\rho_n\leq s_n\rho_n=o(\min_j\lambda_j) s j ρ n ≤ s n ρ n = o ( min j λ j ) . Also, max j λ j = o ( 1 ) \max_j\lambda_j=o(1) max j λ j = o ( 1 ) , because s n ≥ 1 s_n\geq1 s n ≥ 1 and a min , n ≤ C A a_{\min,n}\leq C_A a m i n , n ≤ C A . Hence s n ρ n = o ( 1 ) s_n\rho_n=o(1) s n ρ n = o ( 1 ) , so with probability tending to one λ min ( H ^ j ) > C 2 / 2 \lambda_{\min}(\widehat{H}_j)>C_2/2 λ m i n ( H j ) > C 2 /2 simultaneously for all nonempty supports. Consequently,
∥ H ^ j − 1 ∥ ∞ ≤ s j ∥ H ^ j − 1 ∥ 2 = s j λ min ( H ^ j ) = O p ( s j ) . (11) \left\|\widehat{H}_j^{-1}\right\|_\infty
\leq\sqrt{s_j}\left\|\widehat{H}_j^{-1}\right\|_2
=\frac{\sqrt{s_j}}{\lambda_{\min}(\widehat{H}_j)}
=O_p(\sqrt{s_j}). \tag*{(11)} H j − 1 ∞ ≤ s j H j − 1 2 = λ m i n ( H j ) s j = O p ( s j ) . (11)
By Assumption (I) and Lemma 2 , all estimated diagonal entries are at least c w / 2 c_w/2 c w /2 with probability tending to one. The square-root and inverse-square-root functions are uniformly Lipschitz on this interval. Consequently,
max k ∣ w ^ k − w k ∣ = O p ( r n ) , max k ∣ w ^ k − 1 − w k − 1 ∣ = O p ( r n ) . \max_k |\widehat{w}_k-w_k|=O_p(r_n),\qquad\max_k |\widehat{w}_k^{-1}-w_k^{-1}|=O_p(r_n). k max ∣ w k − w k ∣ = O p ( r n ) , k max ∣ w k − 1 − w k − 1 ∣ = O p ( r n ) .
Since w ^ j k = w ^ k / w ^ j \widehat{w}_{jk}=\widehat{w}_k/\widehat{w}_j w j k = w k / w j ,
∣ w ^ j k − w j k ∣ ≤ w ^ j − 1 ∣ w ^ k − w k ∣ + w k ∣ w ^ j − 1 − w j − 1 ∣ . |\widehat{w}_{jk}-w_{jk}|\leq\widehat{w}_j^{-1}|\widehat{w}_k-w_k|+w_k|\widehat{w}_j^{-1}-w_j^{-1}|. ∣ w j k − w j k ∣ ≤ w j − 1 ∣ w k − w k ∣ + w k ∣ w j − 1 − w j − 1 ∣.
Assumption (E) gives w k ≤ K M w_k\leq\sqrt{K_M} w k ≤ K M . On the event that all estimated diagonal entries are at least c w / 2 c_w/2 c w /2 , we also have w ^ j − 1 ≤ 2 / c w \widehat{w}_j^{-1}\leq\sqrt{2/c_w} w j − 1 ≤ 2/ c w . Therefore,
max j , k ∣ w ^ j k − w j k ∣ = O p ( r n ) . \max_{j,k}|\widehat{w}_{jk}-w_{jk}|=O_p(r_n). j , k max ∣ w j k − w j k ∣ = O p ( r n ) .
We are now ready to prove Theorem 1.
Proof. [Proof of Theorem 1] The problem we wish to solve is
arg min A ∈ R d × d ∥ A Σ ^ Z − P ^ ∥ F 2 + ∑ j = 1 d λ j ∑ k = 1 d w ^ j k ∣ A j k ∣ . \underset{A\in\mathbb{R}^{d\times d}}{\arg\min}\ \left\|A\widehat{\Sigma}_Z-\widehat{P}\right\|_F^2+\sum_{j=1}^{d}\lambda_j\sum_{k=1}^{d}\widehat{w}_{jk}|A_{jk}|. A ∈ R d × d arg min A Σ Z − P F 2 + j = 1 ∑ d λ j k = 1 ∑ d w j k ∣ A j k ∣.
Let a ( j ) a^{(j)} a ( j ) , p ( j ) p^{(j)} p ( j ) , and p ^ ( j ) \widehat{p}^{(j)} p ( j ) denote the corresponding rows as column vectors. Since Σ ^ Z \widehat{\Sigma}_Z Σ Z is symmetric, the problem is equivalent to minimizing
∑ j = 1 d { ∥ Σ ^ Z a ( j ) − p ^ ( j ) ∥ 2 2 + λ j w ^ j ∥ D a ( j ) ∥ 1 } . \sum_{j=1}^{d}\left\{\left\|\widehat{\Sigma}_Z a^{(j)}-\widehat{p}^{(j)}\right\|_2^2+\frac{\lambda_j}{\widehat{w}_j}\left\|D a^{(j)}\right\|_1\right\}. j = 1 ∑ d { Σ Z a ( j ) − p ( j ) 2 2 + w j λ j D a ( j ) 1 } .
We work on the high-probability event in Lemma 3 . Let S ( j ) = { k : A j k ≠ 0 } S^{(j)}=\{k:A_{jk}\neq0\} S ( j ) = { k : A j k = 0 } denote the true support of a ( j ) a^{(j)} a ( j ) . For each j j j , construct a candidate solution by solving the problem restricted to S ( j ) S^{(j)} S ( j ) :
a ~ S ( j ) ( j ) ∈ arg min a S ( j ) ∈ R ∣ S ( j ) ∣ ∥ Σ ^ Z , S ( j ) a S ( j ) − p ^ ( j ) ∥ 2 2 + λ j w ^ j ∥ D ^ S ( j ) a S ( j ) ∥ 1 , \widetilde{a}^{(j)}_{S^{(j)}}\in\underset{a_{S^{(j)}}\in\mathbb{R}^{|S^{(j)}|}}{\arg\min}\ \left\|\widehat{\Sigma}_{Z,S^{(j)}}a_{S^{(j)}}-\widehat{p}^{(j)}\right\|_2^2+\frac{\lambda_j}{\widehat{w}_j}\left\|\widehat{D}_{S^{(j)}}a_{S^{(j)}}\right\|_1, a S ( j ) ( j ) ∈ a S ( j ) ∈ R ∣ S ( j ) ∣ arg min Σ Z , S ( j ) a S ( j ) − p ( j ) 2 2 + w j λ j D S ( j ) a S ( j ) 1 ,
and set a ~ S ( j ) , c = 0 ( j ) \widetilde{a}^{(j)}_{S^{(j),c}=0} a S ( j ) , c = 0 ( j ) .
Then, for any j j j , the KKT condition for the restricted problem implies that there exists
z ~ S ( j ) ( j ) ∈ ∂ ∥ a ~ S ( j ) ( j ) ∥ 1 , ∥ z ~ S ( j ) ( j ) ∥ ∞ ≤ 1 , \widetilde{z}^{(j)}_{S^{(j)}}\in\partial\left\|\widetilde{a}^{(j)}_{S^{(j)}}\right\|_1,\qquad\left\|\widetilde{z}^{(j)}_{S^{(j)}}\right\|_\infty\leq1, z S ( j ) ( j ) ∈ ∂ a S ( j ) ( j ) 1 , z S ( j ) ( j ) ∞ ≤ 1 ,
such that
2 Σ ^ Z , S ( j ) ⊤ [ Σ ^ Z , S ( j ) a ~ S ( j ) ( j ) − p ^ ( j ) ] + λ j w ^ j D ^ S ( j ) z ~ S ( j ) ( j ) = 0. (12) 2\widehat{\Sigma}_{Z,S^{(j)}}^{\top}\left[\widehat{\Sigma}_{Z,S^{(j)}}\widetilde{a}_{S^{(j)}}^{(j)}-\widehat{p}^{(j)}\right]+\frac{\lambda_j}{\widehat{w}_j}\widehat{D}_{S^{(j)}}\widetilde{z}_{S^{(j)}}^{(j)}=0. \tag*{(12)} 2 Σ Z , S ( j ) ⊤ [ Σ Z , S ( j ) a S ( j ) ( j ) − p ( j ) ] + w j λ j D S ( j ) z S ( j ) ( j ) = 0. (12)
Solving for a ~ S ( j ) ( j ) \widetilde{a}_{S^{(j)}}^{(j)} a S ( j ) ( j ) , yields
a ~ S ( j ) ( j ) = H ^ j − 1 [ Σ ^ Z , S ( j ) ⊤ p ^ ( j ) − λ j 2 w ^ j D ^ S ( j ) z ~ S ( j ) ( j ) ] . \widetilde{a}_{S^{(j)}}^{(j)}=\widehat{H}_j^{-1}\left[\widehat{\Sigma}_{Z,S^{(j)}}^{\top}\widehat{p}^{(j)}-\frac{\lambda_j}{2\widehat{w}_j}\widehat{D}_{S^{(j)}}\widetilde{z}_{S^{(j)}}^{(j)}\right]. a S ( j ) ( j ) = H j − 1 [ Σ Z , S ( j ) ⊤ p ( j ) − 2 w j λ j D S ( j ) z S ( j ) ( j ) ] .
Since p ( j ) = Σ Z , S ( j ) a S ( j ) ( j ) p^{(j)}=\Sigma_{Z,S^{(j)}}a_{S^{(j)}}^{(j)} p ( j ) = Σ Z , S ( j ) a S ( j ) ( j ) , and letting Δ ^ p ( j ) = p ^ ( j ) − p ( j ) \widehat{\Delta}_{p^{(j)}}=\widehat{p}^{(j)}-p^{(j)} Δ p ( j ) = p ( j ) − p ( j ) , we obtain
a ~ S ( j ) ( j ) − a S ( j ) ( j ) = H ^ j − 1 [ Σ ^ Z , S ( j ) ⊤ ( Δ ^ p ( j ) − Δ ^ Σ Z , S ( j ) a S ( j ) ( j ) ) − λ j 2 w ^ j D ^ S ( j ) z ~ S ( j ) ( j ) ] . (13) \widetilde{a}_{S^{(j)}}^{(j)}-a_{S^{(j)}}^{(j)}=\widehat{H}_j^{-1}\left[\widehat{\Sigma}_{Z,S^{(j)}}^{\top}\left(\widehat{\Delta}_{p^{(j)}}-\widehat{\Delta}_{\Sigma_Z,S^{(j)}}a_{S^{(j)}}^{(j)}\right)-\frac{\lambda_j}{2\widehat{w}_j}\widehat{D}_{S^{(j)}}\widetilde{z}_{S^{(j)}}^{(j)}\right]. \tag*{(13)} a S ( j ) ( j ) − a S ( j ) ( j ) = H j − 1 [ Σ Z , S ( j ) ⊤ ( Δ p ( j ) − Δ Σ Z , S ( j ) a S ( j ) ( j ) ) − 2 w j λ j D S ( j ) z S ( j ) ( j ) ] . (13)
Therefore,
∥ a ~ S ( j ) ( j ) − a S ( j ) ( j ) ∥ ∞ ≤ ∥ H ^ j − 1 ∥ ∞ [ ∥ Σ ^ Z , S ( j ) ∥ 1 ( ∥ Δ ^ p ( j ) ∥ ∞ + ∥ Δ ^ Σ Z , S ( j ) ∥ max ∥ a S ( j ) ( j ) ∥ 1 ) + λ j 2 w ^ j ∥ D ^ S ( j ) ∥ ∞ ] . \left\|\widetilde{a}_{S^{(j)}}^{(j)}-a_{S^{(j)}}^{(j)}\right\|_{\infty}\leq\left\|\widehat{H}_j^{-1}\right\|_{\infty}\left[\left\|\widehat{\Sigma}_{Z,S^{(j)}}\right\|_1\left(\left\|\widehat{\Delta}_{p^{(j)}}\right\|_{\infty}+\left\|\widehat{\Delta}_{\Sigma_Z,S^{(j)}}\right\|_{\max}\left\|a_{S^{(j)}}^{(j)}\right\|_1\right)+\frac{\lambda_j}{2\widehat{w}_j}\left\|\widehat{D}_{S^{(j)}}\right\|_{\infty}\right]. a S ( j ) ( j ) − a S ( j ) ( j ) ∞ ≤ H j − 1 ∞ [ Σ Z , S ( j ) 1 ( Δ p ( j ) ∞ + Δ Σ Z , S ( j ) m a x a S ( j ) ( j ) 1 ) + 2 w j λ j D S ( j ) ∞ ] .
From Lemma (2) and Assumption (F), uniformly over j j j ,
∥ Σ ^ Z , S ( j ) ∥ 1 ≤ ∥ Σ Z ∥ 1 + d ∥ Σ ^ Z − Σ Z ∥ max = O p ( 1 + d r n ) . (14) \left\|\widehat{\Sigma}_{Z,S^{(j)}}\right\|_1\leq\left\|\Sigma_Z\right\|_1+d\left\|\widehat{\Sigma}_Z-\Sigma_Z\right\|_{\max}=O_p(1+dr_n). \tag*{(14)} Σ Z , S ( j ) 1 ≤ ∥ Σ Z ∥ 1 + d Σ Z − Σ Z m a x = O p ( 1 + d r n ) . (14)
and from Assumption (J) we have ∥ a S ( j ) ( j ) ∥ 1 ≤ C A s j \left\|a_{S^{(j)}}^{(j)}\right\|_1\leq C_As_j a S ( j ) ( j ) 1 ≤ C A s j . Together with Lemmas (2)–(3), and Eq. (A.B8), these bounds yield
∥ a ~ S ( j ) ( j ) − a S ( j ) ( j ) ∥ ∞ = O p ( s j 3 / 2 ρ n + s j λ j ) . \left\|\widetilde{a}_{S^{(j)}}^{(j)}-a_{S^{(j)}}^{(j)}\right\|_{\infty}=O_p\left(s_j^{3/2}\rho_n+\sqrt{s_j}\lambda_j\right). a S ( j ) ( j ) − a S ( j ) ( j ) ∞ = O p ( s j 3/2 ρ n + s j λ j ) .
Consequently,
max 1 ≤ j ≤ d ∥ a ~ S ( j ) ( j ) − a S ( j ) ( j ) ∥ ∞ = O p ( s n 3 / 2 ρ n + s n max j λ j ) . \max_{1\leq j\leq d}\left\|\widetilde{a}_{S^{(j)}}^{(j)}-a_{S^{(j)}}^{(j)}\right\|_{\infty}=O_p\left(s_n^{3/2}\rho_n+\sqrt{s_n}\max_j\lambda_j\right). 1 ≤ j ≤ d max a S ( j ) ( j ) − a S ( j ) ( j ) ∞ = O p ( s n 3/2 ρ n + s n j max λ j ) .
By Assumption (K),
s n 3 / 2 ρ n a min , n ≤ s n ρ n min j λ j s n max j λ j a min , n ⟶ 0. \frac{s_n^{3/2}\rho_n}{a_{\min,n}}\leq\frac{s_n\rho_n}{\min_j\lambda_j}\frac{\sqrt{s_n}\max_j\lambda_j}{a_{\min,n}}\longrightarrow0. a m i n , n s n 3/2 ρ n ≤ min j λ j s n ρ n a m i n , n s n max j λ j ⟶ 0.
Thus, s n 3 / 2 ρ n + s n max j λ j = o ( a min , n ) s_n^{3/2}\rho_n+\sqrt{s_n}\max_j\lambda_j=o(a_{\min,n}) s n 3/2 ρ n + s n max j λ j = o ( a m i n , n ) , so it follows that
max 1 ≤ j ≤ d ∥ a ~ S ( j ) ( j ) − a S ( j ) ( j ) ∥ ∞ = o p ( a min , n ) . \max_{1\leq j\leq d}\left\|\widetilde{a}_{S^{(j)}}^{(j)}-a_{S^{(j)}}^{(j)}\right\|_{\infty}=o_p(a_{\min,n}). 1 ≤ j ≤ d max a S ( j ) ( j ) − a S ( j ) ( j ) ∞ = o p ( a m i n , n ) .
Therefore,
Pr ( max 1 ≤ j ≤ d ∥ a ~ S ( j ) ( j ) − a S ( j ) ( j ) ∥ ∞ < a min , n ) ⟶ 1. \Pr\left(\max_{1\leq j\leq d}\left\|\widetilde{a}_{S^{(j)}}^{(j)}-a_{S^{(j)}}^{(j)}\right\|_{\infty}<a_{\min,n}\right)\longrightarrow1. Pr ( 1 ≤ j ≤ d max a S ( j ) ( j ) − a S ( j ) ( j ) ∞ < a m i n , n ) ⟶ 1.
On this event, for every j j j and every k ∈ S ( j ) k\in S^{(j)} k ∈ S ( j ) ,
∣ a ~ k ( j ) − a k ( j ) ∣ < a min , n ≤ ∣ a k ( j ) ∣ . \left|\widetilde{a}_k^{(j)}-a_k^{(j)}\right|<a_{\min,n}\leq\left|a_k^{(j)}\right|. a k ( j ) − a k ( j ) < a m i n , n ≤ a k ( j ) .
Hence, a ~ k ( j ) ≠ 0 \widetilde{a}^{(j)}_k \ne0 a k ( j ) = 0 , and sign ( a ~ k ( j ) ) = sign ( a k ( j ) ) \operatorname{sign}(\widetilde{a}^{(j)}_k) = \operatorname{sign}(a^{(j)}_k) sign ( a k ( j ) ) = sign ( a k ( j ) ) . Thus, with probability tending to one, the restricted candidate has no false negatives and recovers the signs of all coefficients on the true support. Since a ~ S ( j ) , c ( j ) = 0 \widetilde{a}^{(j)}_{S^{(j),c}} = 0 a S ( j ) , c ( j ) = 0 , by construction, it follows that supp ( a ~ ( j ) ) = S ( j ) \operatorname{supp}(\widetilde{a}^{(j)}) = S^{(j)} supp ( a ( j ) ) = S ( j ) , uniformly over j j j , with probability tending to one. This establishes support and sign recovery for the restricted candidate. To show that this candidate is a solution to the full optimization problem, it remains to construct the inactive subgradient and verify that
∥ z ~ S ( j ) , c ( j ) ∥ ∞ < 1. \left\|\widetilde{z}^{(j)}_{S^{(j),c}}\right\|_{\infty} < 1. z S ( j ) , c ( j ) ∞ < 1.
For convenience, let us define
B ^ j = Σ ^ Z , S ( j ) , c T Σ ^ Z , S ( j ) , R j = p ^ ( j ) − Σ ^ Z , S ( j ) a S ( j ) ( j ) . \widehat{B}_j = \widehat{\Sigma}_{Z,S^{(j),c}}^{\mathsf{T}}\widehat{\Sigma}_{Z,S^{(j)}}, \qquad R_j = \widehat{p}^{(j)} - \widehat{\Sigma}_{Z,S^{(j)}}a^{(j)}_{S^{(j)}}. B j = Σ Z , S ( j ) , c T Σ Z , S ( j ) , R j = p ( j ) − Σ Z , S ( j ) a S ( j ) ( j ) .
Define the inactive subgradient by the full KKT equation:
z ~ S ( j ) , c ( j ) = − 2 w ^ j λ j D ^ S ( j ) , c − 1 Σ ^ Z , S ( j ) , c T ( Σ ^ Z , S ( j ) a ~ S ( j ) ( j ) − p ^ ( j ) ) . \widetilde{z}^{(j)}_{S^{(j),c}} = -\frac{2\widehat{w}_j}{\lambda_j}\widehat{D}^{-1}_{S^{(j),c}}\widehat{\Sigma}_{Z,S^{(j),c}}^{\mathsf{T}}\left(\widehat{\Sigma}_{Z,S^{(j)}}\widetilde{a}^{(j)}_{S^{(j)}}-\widehat{p}^{(j)}\right). z S ( j ) , c ( j ) = − λ j 2 w j D S ( j ) , c − 1 Σ Z , S ( j ) , c T ( Σ Z , S ( j ) a S ( j ) ( j ) − p ( j ) ) .
Combining this with (A.B10) gives
z ~ S ( j ) , c ( j ) = D ^ S ( j ) , c − 1 B ^ j H ^ j − 1 D ^ S ( j ) z ~ S ( j ) ( j ) + 2 w ^ j λ j D ^ S ( j ) , c − 1 ( Σ ^ Z , S ( j ) , c T − B ^ j H ^ j − 1 Σ ^ Z , S ( j ) T ) R j . (15) \begin{aligned}
\widetilde{z}^{(j)}_{S^{(j),c}}
={}&\widehat{D}^{-1}_{S^{(j),c}}\widehat{B}_j\widehat{H}_j^{-1}\widehat{D}_{S^{(j)}}\widetilde{z}^{(j)}_{S^{(j)}} \\
&+\frac{2\widehat{w}_j}{\lambda_j}\widehat{D}^{-1}_{S^{(j),c}}\left(\widehat{\Sigma}_{Z,S^{(j),c}}^{\mathsf{T}}-\widehat{B}_j\widehat{H}_j^{-1}\widehat{\Sigma}_{Z,S^{(j)}}^{\mathsf{T}}\right)R_j. \tag*{(15)}
\end{aligned} z S ( j ) , c ( j ) = D S ( j ) , c − 1 B j H j − 1 D S ( j ) z S ( j ) ( j ) + λ j 2 w j D S ( j ) , c − 1 ( Σ Z , S ( j ) , c T − B j H j − 1 Σ Z , S ( j ) T ) R j . (15)
We would like now to transfer Assumption (H) to its sample counterpart. For any k ∈ S ( j ) , c k \in S^{(j),c} k ∈ S ( j ) , c and ℓ ∈ S ( j ) \ell\in S^{(j)} ℓ ∈ S ( j ) , and from Eq. (A.B11),
∣ Σ ^ Z , k T Σ ^ Z , ℓ − Σ Z , k T Σ Z , ℓ ∣ ≤ ∥ Σ ^ Z , k − Σ Z , k ∥ ∞ ∥ Σ ^ Z , ℓ ∥ 1 + ∥ Σ Z , k ∥ 1 ∥ Σ ^ Z , ℓ − Σ Z , ℓ ∥ ∞ = O p { r n ( 1 + d r n ) + r n } = O p ( ρ n ) , \begin{aligned}
\left|\widehat{\Sigma}_{Z,k}^{\mathsf{T}}\widehat{\Sigma}_{Z,\ell}-\Sigma_{Z,k}^{\mathsf{T}}\Sigma_{Z,\ell}\right|
\le\left\|\widehat{\Sigma}_{Z,k}-\Sigma_{Z,k}\right\|_{\infty}\left\|\widehat{\Sigma}_{Z,\ell}\right\|_1+\left\|\Sigma_{Z,k}\right\|_1\left\|\widehat{\Sigma}_{Z,\ell}-\Sigma_{Z,\ell}\right\|_{\infty} \\
&= O_p\left\{r_n(1+d r_n)+r_n\right\}=O_p(\rho_n),
\end{aligned} Σ Z , k T Σ Z , ℓ − Σ Z , k T Σ Z , ℓ ≤ Σ Z , k − Σ Z , k ∞ Σ Z , ℓ 1 + ∥ Σ Z , k ∥ 1 Σ Z , ℓ − Σ Z , ℓ ∞ = O p { r n ( 1 + d r n ) + r n } = O p ( ρ n ) ,
uniformly in j j j , k k k and ℓ \ell ℓ . Since B ^ j − B j \widehat{B}_j-B_j B j − B j has s j s_j s j columns, it follows that
∥ B ^ j − B j ∥ ∞ = O p ( s j ρ n ) , \left\|\widehat{B}_j-B_j\right\|_{\infty}=O_p(s_j\rho_n), B j − B j ∞ = O p ( s j ρ n ) ,
uniformly in j j j . Using the identity
B ^ j H ^ j − 1 − B j H j − 1 = ( B ^ j − B j ) H ^ j − 1 + B j H j − 1 ( H j − H ^ j ) H ^ j − 1 . \widehat{B}_j\widehat{H}_j^{-1}-B_jH_j^{-1}=(\widehat{B}_j-B_j)\widehat{H}_j^{-1}+B_jH_j^{-1}(H_j-\widehat{H}_j)\widehat{H}_j^{-1}. B j H j − 1 − B j H j − 1 = ( B j − B j ) H j − 1 + B j H j − 1 ( H j − H j ) H j − 1 .
The population weights and their inverses are uniformly bounded. Thus Assumption (H) implies ∥ B j H j − 1 ∥ ∞ = O ( 1 ) \lVert B_jH_j^{-1}\rVert_\infty=O(1) ∥ B j H j − 1 ∥ ∞ = O ( 1 ) uniformly in j j j . Together with the preceding bounds, we obtain
∥ B ^ j H ^ j − 1 − B j H j − 1 ∥ ∞ ≤ ∥ B ^ j − B j ∥ ∞ ∥ H ^ j − 1 ∥ ∞ + ∥ B j H j − 1 ∥ ∞ ∥ H j − H ^ j ∥ ∞ ∥ H ^ j − 1 ∥ ∞ = O p ( s j 3 / 2 ρ n ) . \left\lVert\widehat{B}_j\widehat{H}_j^{-1}-B_jH_j^{-1}\right\rVert_\infty
\leq\left\lVert\widehat{B}_j-B_j\right\rVert_\infty\left\lVert\widehat{H}_j^{-1}\right\rVert_\infty+\left\lVert B_jH_j^{-1}\right\rVert_\infty\left\lVert H_j-\widehat{H}_j\right\rVert_\infty\left\lVert\widehat{H}_j^{-1}\right\rVert_\infty
=O_p\left(s_j^{3/2}\rho_n\right). B j H j − 1 − B j H j − 1 ∞ ≤ B j − B j ∞ H j − 1 ∞ + B j H j − 1 ∞ H j − H j ∞ H j − 1 ∞ = O p ( s j 3/2 ρ n ) .
Consequently,
max 1 ≤ j ≤ d ∥ B ^ j H ^ j − 1 − B j H j − 1 ∥ ∞ = O p ( s n 3 / 2 ρ n ) = o p ( 1 ) . (16) \max_{1\leq j\leq d}\left\lVert\widehat{B}_j\widehat{H}_j^{-1}-B_jH_j^{-1}\right\rVert_\infty
=O_p\left(s_n^{3/2}\rho_n\right)=o_p(1). \tag*{(16)} 1 ≤ j ≤ d max B j H j − 1 − B j H j − 1 ∞ = O p ( s n 3/2 ρ n ) = o p ( 1 ) . (16)
We now compare the sample weighted matrix with its population counterpart. Combining (A.B13) with the bounds for D ^ \widehat{D} D and D ^ − 1 \widehat{D}^{-1} D − 1 established in the proof of Lemma 3 gives
max 1 ≤ j ≤ d ∥ D ^ S ( j ) , C − 1 B ^ j H ^ j − 1 D ^ S ( j ) − D S ( j ) , C − 1 B j H j − 1 D S ( j ) ∥ ∞ = O p ( s n 3 / 2 ρ n + r n ) = o p ( 1 ) . \max_{1\leq j\leq d}\left\lVert\widehat{D}_{S(j),C}^{-1}\widehat{B}_j\widehat{H}_j^{-1}\widehat{D}_{S(j)}-D_{S(j),C}^{-1}B_jH_j^{-1}D_{S(j)}\right\rVert_\infty
=O_p\left(s_n^{3/2}\rho_n+r_n\right)=o_p(1). 1 ≤ j ≤ d max D S ( j ) , C − 1 B j H j − 1 D S ( j ) − D S ( j ) , C − 1 B j H j − 1 D S ( j ) ∞ = O p ( s n 3/2 ρ n + r n ) = o p ( 1 ) .
Consequently, with probability tending to one,
max 1 ≤ j ≤ d ∥ D ^ S ( j ) , C − 1 B ^ j H ^ j − 1 D ^ S ( j ) ∥ ∞ ≤ 1 − η 2 . \max_{1\leq j\leq d}\left\lVert\widehat{D}_{S(j),C}^{-1}\widehat{B}_j\widehat{H}_j^{-1}\widehat{D}_{S(j)}\right\rVert_\infty\leq1-\frac{\eta}{2}. 1 ≤ j ≤ d max D S ( j ) , C − 1 B j H j − 1 D S ( j ) ∞ ≤ 1 − 2 η .
Since ∥ R j ∥ ∞ = O p ( s j r n ) \lVert R_j\rVert_\infty=O_p(s_jr_n) ∥ R j ∥ ∞ = O p ( s j r n ) , uniformly in j j j , and
∥ Σ ^ Z , S ( j ) , C ∥ 1 + ∥ Σ ^ Z , S ( j ) ∥ 1 = O p ( 1 + d r n ) , \left\lVert\widehat{\Sigma}_{Z,S(j),C}\right\rVert_1+\left\lVert\widehat{\Sigma}_{Z,S(j)}\right\rVert_1=O_p(1+dr_n), Σ Z , S ( j ) , C 1 + Σ Z , S ( j ) 1 = O p ( 1 + d r n ) ,
we have that, uniformly in j j j ,
∥ ( Σ ^ Z , S ( j ) , C − B ^ j H ^ j − 1 Σ ^ Z , S ( j ) ⊤ ) R j ∥ ∞ ≤ { ∥ Σ ^ Z , S ( j ) , C ∥ 1 + ∥ B ^ j H ^ j − 1 ∥ ∞ ∥ Σ ^ Z , S ( j ) ∥ 1 } ∥ R j ∥ ∞ = O p { s j r n ( 1 + d r n ) } = O p ( s j ρ n ) . \left\lVert\left(\widehat{\Sigma}_{Z,S(j),C}-\widehat{B}_j\widehat{H}_j^{-1}\widehat{\Sigma}_{Z,S(j)}^\top\right)R_j\right\rVert_\infty
\leq\left\{\left\lVert\widehat{\Sigma}_{Z,S(j),C}\right\rVert_1+\left\lVert\widehat{B}_j\widehat{H}_j^{-1}\right\rVert_\infty\left\lVert\widehat{\Sigma}_{Z,S(j)}\right\rVert_1\right\}\lVert R_j\rVert_\infty
=O_p\{s_jr_n(1+dr_n)\}=O_p(s_j\rho_n). ( Σ Z , S ( j ) , C − B j H j − 1 Σ Z , S ( j ) ⊤ ) R j ∞ ≤ { Σ Z , S ( j ) , C 1 + B j H j − 1 ∞ Σ Z , S ( j ) 1 } ∥ R j ∥ ∞ = O p { s j r n ( 1 + d r n )} = O p ( s j ρ n ) .
Because ∥ z ~ S ( j ) , C ( j ) ∥ ∞ ≤ 1 \lVert\widetilde{z}_{S(j),C}^{(j)}\rVert_\infty\leq1 ∥ z S ( j ) , C ( j ) ∥ ∞ ≤ 1 , and the standard deviations and their inverses are uniformly bounded, together with (A.B12) and Assumption (K) give
max 1 ≤ j ≤ d ∥ z ~ S ( j ) , C ( j ) ∥ ∞ ≤ 1 − η 2 + O p ( s n ρ n min j λ j ) = 1 − η 2 + o p ( 1 ) . \max_{1\leq j\leq d}\left\lVert\widetilde{z}_{S(j),C}^{(j)}\right\rVert_\infty
\leq1-\frac{\eta}{2}+O_p\left(\frac{s_n\rho_n}{\min_j\lambda_j}\right)
=1-\frac{\eta}{2}+o_p(1). 1 ≤ j ≤ d max z S ( j ) , C ( j ) ∞ ≤ 1 − 2 η + O p ( min j λ j s n ρ n ) = 1 − 2 η + o p ( 1 ) .
Thus strict dual feasibility holds simultaneously for all rows, and each restricted candidate is a minimizer of the full rowwise problem. It remains to prove uniqueness. Let a ˉ ( j ) \bar{a}^{(j)} a ˉ ( j ) be any other minimizer. Expanding the difference of the objective values and using 2 Σ ^ Z ( Σ ^ Z a ˉ ( j ) − p ^ ( j ) ) + λ j w ^ j D ^ z ~ ( j ) = 0 2\widehat{\Sigma}_Z(\widehat{\Sigma}_Z\bar{a}^{(j)}-\widehat{p}^{(j)})+\frac{\lambda_j}{\widehat{w}_j}\widehat{D}\widetilde{z}^{(j)}=0 2 Σ Z ( Σ Z a ˉ ( j ) − p ( j ) ) + w j λ j D z ( j ) = 0 gives
0 = ∥ Σ ^ Z ( a ˉ ( j ) − a ~ ( j ) ) ∥ 2 2 + λ j ∑ k = 1 d w ^ j k { ∣ a ˉ k ( j ) ∣ − ∣ a ~ k ( j ) ∣ − z ~ k ( j ) ( a ˉ k ( j ) − a ~ k ( j ) ) } . 0=\left\lVert\widehat{\Sigma}_Z(\bar{a}^{(j)}-\widetilde{a}^{(j)})\right\rVert_2^2+\lambda_j\sum_{k=1}^{d}\widehat{w}_{jk}\left\{\left|\bar{a}_k^{(j)}\right|-\left|\widetilde{a}_k^{(j)}\right|-\widetilde{z}_k^{(j)}\left(\bar{a}_k^{(j)}-\widetilde{a}_k^{(j)}\right)\right\}. 0 = Σ Z ( a ˉ ( j ) − a ( j ) ) 2 2 + λ j k = 1 ∑ d w j k { a ˉ k ( j ) − a k ( j ) − z k ( j ) ( a ˉ k ( j ) − a k ( j ) ) } .
Both terms, and each summand in the second term, are nonnegative. The contribution of the inactive coordinates to the second term is at least
λ j ∑ k ∈ S ( j ) , c w ^ j k ( 1 − ∣ z ~ k ( j ) ∣ ) ∣ a ‾ k ( j ) ∣ . \lambda_j \sum_{k\in S(j),c} \widehat{w}_{jk}\left(1-\left|\widetilde{z}^{(j)}_k\right|\right)\left|\overline{a}^{(j)}_k\right|. λ j k ∈ S ( j ) , c ∑ w j k ( 1 − z k ( j ) ) a k ( j ) .
The weights are positive and strict dual feasibility holds, so a ‾ S ( j ) , c ( j ) = 0 \overline{a}^{(j)}_{S(j),c}=0 a S ( j ) , c ( j ) = 0 . The first term also vanishes, giving
Σ ^ Z , S ( j ) ( a ‾ S ( j ) ( j ) − a ~ S ( j ) ( j ) ) = 0. \widehat{\Sigma}_{Z,S(j)}\left(\overline{a}^{(j)}_{S(j)}-\widetilde{a}^{(j)}_{S(j)}\right)=0. Σ Z , S ( j ) ( a S ( j ) ( j ) − a S ( j ) ( j ) ) = 0.
Positive definiteness of H ^ j \widehat{H}_j H j implies equality of the active coefficients. Thus the minimizer is unique. Combining this with the established support and sign recovery proves
Pr { supp ( A ^ ) = supp ( A ) , sign ( A ^ ) = sign ( A ) } ⟶ 1. \Pr\left\{\operatorname{supp}(\widehat{A})=\operatorname{supp}(A),\ \operatorname{sign}(\widehat{A})=\operatorname{sign}(A)\right\}\longrightarrow1. Pr { supp ( A ) = supp ( A ) , sign ( A ) = sign ( A ) } ⟶ 1.
Proof. [Proof of Corollary 1] On the support-recovery event, A ^ \widehat{A} A equals the restricted candidate constructed in the proof of Theorem 1. The coefficient-error bound established there gives the first rate.
On the same event, the refit uses the true support in every row. For rows with nonempty support, its normal equations give
a ^ S ( j ) ( j ) , r e f i t − a S ( j ) ( j ) = H ^ j − 1 Σ ^ Z , S ( j ) T R j . \widehat{a}^{(j),\mathrm{refit}}_{S(j)}-a^{(j)}_{S(j)}=\widehat{H}_j^{-1}\widehat{\Sigma}^{\mathsf{T}}_{Z,S(j)}R_j. a S ( j ) ( j ) , refit − a S ( j ) ( j ) = H j − 1 Σ Z , S ( j ) T R j .
Using (A.B8), (A.B11), and the previously established bound ∥ R j ∥ ∞ = O p ( s j r n ) \lVert R_j\rVert_{\infty}=O_p(s_jr_n) ∥ R j ∥ ∞ = O p ( s j r n ) yields, uniformly over these rows,
∥ a ^ S ( j ) ( j ) , r e f i t − a S ( j ) ( j ) ∥ ∞ = O p ( s j 3 / 2 r n ( 1 + d r n ) ) = O p ( s j 3 / 2 ρ n ) . \left\lVert\widehat{a}^{(j),\mathrm{refit}}_{S(j)}-a^{(j)}_{S(j)}\right\rVert_{\infty}=O_p\left(s_j^{3/2}r_n(1+dr_n)\right)=O_p\left(s_j^{3/2}\rho_n\right). a S ( j ) ( j ) , refit − a S ( j ) ( j ) ∞ = O p ( s j 3/2 r n ( 1 + d r n ) ) = O p ( s j 3/2 ρ n ) .
Rows with empty support are refitted as zero. Since the support-recovery event has probability tending to one, taking the maximum over rows gives the asserted refit rate.
Finally, Assumption (K) implies
s n 3 / 2 ρ n = o ( a min , n ) , s n max j λ j = o ( a min , n ) . s_n^{3/2}\rho_n=o(a_{\min,n}),\qquad\sqrt{s_n}\max_j\lambda_j=o(a_{\min,n}). s n 3/2 ρ n = o ( a m i n , n ) , s n j max λ j = o ( a m i n , n ) .
Since a min , n ≤ C A a_{\min,n}\le C_A a m i n , n ≤ C A by Assumption (J), both estimators are consistent in the maximum-entry norm.
Appendix C: Additional simulation results
Here, we provide additional simulation results for the settings where d = 25 d = 25 d = 25 and on average 5% of each row of A A A is non-zero and for the settings where d = 50 d = 50 d = 50 and on average 10% of each row of A A A is non-zero.
[fig:5 about here.]
[fig:6 about here.]