Introduction
Advances in machine learning have greatly expanded our ability to learn complex distributions [19 , 31 , 45 ] (Ho et al. 2020). These advances have, in turn, stimulated interest in flexible latent distributions within latent variable models, where the distribution of the observed variables is determined jointly by the conditional response model and the latent distribution. Moving beyond conventional Gaussian latent distributions allows such models to capture richer forms of population heterogeneity, and thereby to support inference on complex latent structures across the social, economic, and health sciences [4 , 5 , 39 ] (Xu et al. 2023). At the same time, these developments raise a fundamental question of whether model parameters remain identifiable when the latent distribution is left unrestricted, a property on which estimation, inference, and interpretation all depend.
This identifiability question is especially relevant under nonlinear models, where the model parameters and the latent distribution jointly affect the observed responses through a nonlinear relationship. As a representative case, we consider a binary response vector X ∈ { 0 , 1 } p X \in\{0,1\}^{p} X ∈ { 0 , 1 } p , whose distribution depends on a d d d -dimensional latent random vector Z ∈ R d Z \in\mathbb{R}^{d} Z ∈ R d . Conditional on Z Z Z , the components { X j } j ≤ p \{X_j\}_{j\le p} { X j } j ≤ p are independent, with
P ( X j = 1 ∣ Z = z ) = f ( β j + α j T z ) , for j = 1 , … , p , (1) \mathbb{P}(X_j = 1 \mid Z = z) = f(\beta_j + \alpha_j^{\mathsf{T}}z), \quad\text{for } j = 1,\ldots,p, \tag*{(1)} P ( X j = 1 ∣ Z = z ) = f ( β j + α j T z ) , for j = 1 , … , p , (1)
where f : R → ( 0 , 1 ) f:\mathbb{R}\to(0,1) f : R → ( 0 , 1 ) is a specified response function. The model parameters are denoted by θ = ( A , β ) \theta=(A,\beta) θ = ( A , β ) , where A = ( α 1 , … , α p ) T ∈ R p × d A=(\alpha_1,\ldots,\alpha_p)^{\mathsf{T}}\in\mathbb{R}^{p\times d} A = ( α 1 , … , α p ) T ∈ R p × d and β = ( β 1 , … , β p ) T ∈ R p \beta=(\beta_1,\ldots,\beta_p)^{\mathsf{T}}\in\mathbb{R}^{p} β = ( β 1 , … , β p ) T ∈ R p . Since Z Z Z is unobserved, the distribution of X X X is obtained by integrating out Z Z Z with respect to its latent distribution G G G . Specifically, for x ∈ { 0 , 1 } p x\in\{0,1\}^{p} x ∈ { 0 , 1 } p ,
p θ , G ( X = x ) = ∫ R d ∏ j = 1 p { f ( β j + α j T z ) } x j { 1 − f ( β j + α j T z ) } 1 − x j d G ( z ) . (2) p_{\theta,G}(X=x)=\int_{\mathbb{R}^{d}}\prod_{j=1}^{p}\{f(\beta_j+\alpha_j^{\mathsf{T}}z)\}^{x_j}\{1-f(\beta_j+\alpha_j^{\mathsf{T}}z)\}^{1-x_j}\,dG(z). \tag*{(2)} p θ , G ( X = x ) = ∫ R d j = 1 ∏ p { f ( β j + α j T z ) } x j { 1 − f ( β j + α j T z ) } 1 − x j d G ( z ) . (2)
We denote the corresponding marginal distribution of the observed variable X X X by P θ , G P_{\theta,G} P θ , G , which depends on both the model parameter and latent distribution. This model covers several important classes of latent models:
Item response theory. In item response theory and related measurement models for educational and psychological assessment, Z Z Z represents unobserved abilities, attitudes, or psychological traits [47 ] , and its distribution G G G can be asymmetric, multimodal, and dependent to capture rich population heterogeneity [56 ] . Parameter α j \alpha_j α j describes how strongly and in which directions item j j j responds to variation in these latent traits.
Latent-class models. When G G G has finite support, (1) becomes a latent class model given by a finite mixture of product Bernoulli distributions [1 , 20 , 36 ] . The support points of G G G represent latent classes with distinct response profiles, while their probability masses give the corresponding population proportions [9 ] . Here, α j \alpha_j α j describes how the response probability for item j j j varies across these latent classes.
Generative models. Under the logistic link f ( ⋅ ) f(\cdot) f ( ⋅ ) , for instance, model (1) describes the Bernoulli observation layer used in sigmoid belief networks and related generative models [24 , 31 , 59 ] . In this setting, Z Z Z is the hidden representation, G G G is its distribution, and ( A , β ) (A,\beta) ( A , β ) parameterize the observation layer.
A fundamental issue shared by these applications is model identifiability. Formally, identifiability requires that, for any ( θ , G ) (\theta,G) ( θ , G ) and ( θ ~ , G ~ ) (\tilde{\theta},\tilde{G}) ( θ ~ , G ~ ) in the admissible parameter space,
P θ , G = P θ ~ , G ~ ⟹ θ = θ ~ . P_{\theta,G}=P_{\tilde{\theta},\tilde{G}} \quad\Longrightarrow\quad\theta=\tilde{\theta}. P θ , G = P θ ~ , G ~ ⟹ θ = θ ~ .
Identifiability plays a fundamental role in both the theory and practice of latent variable modeling. Theoretically, it is a prerequisite for consistent parameter estimation and valid statistical inference. Practically, without identifiability, the data cannot distinguish between parameters that induce the same observed distribution, leaving the interpretation of the estimated model ambiguous. We therefore ask
(Q1) Are the model parameter θ \theta θ and latent distribution G G G identifiable in (1) ?
(Q2) If not, is reliable estimation still possible?
The primary difficulty in answering this is to uniquely disentangle these two components from the marginal distribution P θ , G P_{\theta,G} P θ , G . Since G G G enters the marginal distribution via integration, the resulting ambiguity is of a different and less tractable nature than the familiar indeterminacies in θ \theta θ . To make the problem more tractable, most existing studies examine identifiability under additional restrictions, typically on G G G . For instance, conventional analyses often take G G G to be a d d d -dimensional Gaussian distribution, in which case the identifiability issue reduces to affine indeterminacies that can be resolved through appropriate constraints [3 , 20 ] . However, a Gaussian specification may fail to capture asymmetry, multimodality, and other features of heterogeneous latent populations [39 , 65 ] .
Beyond the Gaussian case, several studies allow non-Gaussian latent distributions under other restrictions. For example, Allman et al. [1 ] consider latent distributions with a known finite number of support points and establish generic identifiability. Restrictions on the response structure are also considered, for instance by Xu [63 ] and Gu and Xu [21 ] , to establish identifiability of restricted latent class models. In deep generative models, identifiability is often achieved using auxiliary variables, such as time segments or class labels [26 , 30 ] , or mixture priors on the latent variables [32 ] . Across these studies, identifiability is established under restrictions that can be difficult to verify in practice. Moreover, these restrictions narrow the model class, leaving open whether identifiability holds when the latent distribution G G G is left essentially unrestricted.
In this work, we show that exact identifiability fails for almost every parameter in model (1) , answering (Q1) in the negative. This is perhaps unsurprising in light of existing non-identifiability results for the one-dimensional item response theory model [22 , 53 ] , yet our result is considerably stronger. For almost every parameter θ \theta θ and latent distribution G G G , we construct alternatives ( θ ~ , G ~ ) (\tilde{\theta},\tilde{G}) ( θ ~ , G ~ ) satisfying P θ , G = P θ ~ , G ~ P_{\theta,G}=P_{\tilde{\theta},\tilde{G}} P θ , G = P θ ~ , G ~ , with θ ~ ≠ θ \tilde{\theta}\ne\theta θ ~ = θ and G ~ \tilde{G} G ~ matching the moments of G G G up to any prespecified order. This reveals a fundamental obstacle to identifiability under flexible latent distributions that finitely many moment conditions on G G G cannot remove.
The more surprising finding is the superpolynomial rate at which this ambiguity vanishes as p p p grows. Specifically, under suitable regularity conditions, the discrepancy D p ( θ , θ ~ ) D_p(\theta,\tilde{\theta}) D p ( θ , θ ~ ) between parameters θ \theta θ , θ ~ \tilde{\theta} θ ~ (defined in (4) below) satisfies
D p ( θ , θ ~ ) ≤ C exp { − c ( log p ) 2 } whenever P θ , G = P θ ~ , G ~ , D_p(\theta,\tilde{\theta}) \le C\exp\left\{-c(\log p)^2\right\} \qquad\text{whenever}\qquad P_{\theta,G}=P_{\tilde{\theta},\tilde{G}}, D p ( θ , θ ~ ) ≤ C exp { − c ( log p ) 2 } whenever P θ , G = P θ ~ , G ~ ,
uniformly over admissible ( θ , G ) (\theta,G) ( θ , G ) , ( θ ~ , G ~ ) (\tilde{\theta},\tilde{G}) ( θ ~ , G ~ ) , for constants C , c > 0 C,c>0 C , c > 0 . Thus, the ambiguity decays faster than any polynomial rate of p p p and becomes negligible on every polynomial scale. The key technical challenge is to bound D p ( θ , θ ~ ) D_p(\theta,\tilde{\theta}) D p ( θ , θ ~ ) uniformly over all parameter pairs that yield the same marginal distribution. We address this challenge by constructing high-order approximations based on conditional moments of some carefully selected observed responses, given the rest of the observed responses. The nonlinear link leads to complicated interactions among conditional latent moments of different orders, requiring careful control of the expansion coefficients and remainder. Choosing an expansion order proportional to log p \log p log p balances the decay of the remainder against the growth of the coefficients in the expansion, yielding the exp { − c ( log p ) 2 } \exp\left\{-c(\log p)^2\right\} exp { − c ( log p ) 2 } bound.
This superpolynomial rate provides a basis for answering the second question (Q2). In particular, we show that, even without exact identifiability, estimation errors for both the model parameters and the latent distribution can converge to zero as the sample size and p p p diverge. We also provide conditions of valid inference for functionals of the model parameters and population-level quantities defined by the latent distribution. These results provide theoretical justification for the estimability of the model parameters and latent distribution when they are learned jointly (e.g. [4 , 44 , 61 ] (Cui, Chen, Wang and Xu 2026; Li and Lee 2026)), in the sense that non-identifiability need not pose an irreducible identifiability barrier to accurate estimation when p p p is large.
Related Work
The vanishing identification ambiguity described above can be termed asymptotic identifiability. For unidimensional nonparametric item response models, Douglas [16 ] established this property for the response functions, and He [23 ] extended the analysis by relaxing derivative bounds. Both analyses rely on transforming the latent variable to a uniform distribution, which is not applicable in multidimensional settings, and neither quantifies the rate of the vanishing ambiguity. Related questions are studied in panel data models, where p p p is the panel length. For instance, Chernozhukov et al. [8 ] studied the identified sets of average and quantile effects in panel models and showed that they shrink to a point as the panel length increases. In fixed-effects logit models, Davezies et al. [14 ] derived sharp bounds on average causal effects.
A related literature examines estimation error arising from misspecification of the latent distribution. For logistic mixed-effects models, Neuhaus et al. [41 ] showed that misspecifying the mixing distribution can lead to inconsistent parameter estimates. For longitudinal models, Rizopoulos et al. [49 ] showed that the effects of such misspecification vanish for certain parameters as the number of measurements increases. For nonlinear panel models, Arellano and Bonhomme [5 ] analysed estimation bias under misspecification of the individual-effects. For generalized linear latent variable models, Ma and Genton (2010) developed semiparametric estimating equations under misspecified latent distributions. Our work studies a different identification problem when the latent distribution is unknown and must be estimated. The resulting identification ambiguity is intrinsic to the model, independent of any specific estimation procedure.
Another related literature develops estimation methods under flexible latent distributions, including spline-based densities ([61 ] ), Dirichlet-process mixtures ([44 ] ), kernel density methods ((Li and Lee, 2026)), and normalizing flows ((Cui, Chen, Wang and Xu, 2026)), among others. These studies report accurate estimation of the model parameters when the latent distribution is fitted jointly, but do not provide statistical guarantees for the resulting estimators. This gap cannot be addressed by classical theory for latent variable models, which generally presupposes that the model parameters are identifiable ([36 ] (Ma and Genton 2010)), a condition that fails for model (1) when G G G is unrestricted.
Identifiability is also a central concern in generative modelling and representation learning, where the goal is to uncover the underlying generative mechanisms or latent representations, under a flexible latent distribution. Such questions arise in unsupervised disentanglement ([38 ] ), identifiable generative models ([30 ] (Xi and Bloem-Reddy 2023)), and causal representation learning ([54 ] ). A prevailing view in this literature is that complex latent structures cannot be identified without additional information or structural constraints, such as auxiliary variables or inductive biases ([30 , 38 ] ). Notably, Xi and Bloem-Reddy (2023) characterized the indeterminacies of generative models with flexible nonlinear generators. Their framework requires a continuous data-generating process and thus excludes discrete observations. Our work offers another route that accepts the failure of exact identifiability and instead quantifies the ambiguity that remains.
Notation
For a positive integer n n n , let [ n ] = { 1 , … , n } [n]=\{1,\ldots,n\} [ n ] = { 1 , … , n } . For a finite set S S S , ∣ S ∣ |S| ∣ S ∣ denotes its cardinality, and x S x_S x S is the subvector of x x x indexed by S S S . For a vector x x x , ∥ x ∥ 2 \lVert x\rVert_2 ∥ x ∥ 2 and ∥ x ∥ ∞ \lVert x\rVert_\infty ∥ x ∥ ∞ denote its ℓ 2 \ell_2 ℓ 2 and ℓ ∞ \ell_\infty ℓ ∞ norms, respectively. For a matrix M = ( M i j ) n × m M=(M_{ij})_{n\times m} M = ( M ij ) n × m , write ∥ M ∥ ∞ = max i ∈ [ n ] , j ∈ [ m ] ∣ M i j ∣ \lVert M\rVert_\infty=\max_{i\in[n],j\in[m]}|M_{ij}| ∥ M ∥ ∞ = max i ∈ [ n ] , j ∈ [ m ] ∣ M ij ∣ and ∥ M ∥ = max ∥ x ∥ = 1 ∥ M x ∥ \lVert M\rVert=\max_{\lVert x\rVert=1}\lVert Mx\rVert ∥ M ∥ = max ∥ x ∥ = 1 ∥ M x ∥ , and write col ( M ) \operatorname{col}(M) col ( M ) for its column space. Let I d I_d I d be the d × d d\times d d × d identity matrix. The class C ∞ C^\infty C ∞ consists of functions with continuous derivatives of all orders. For a multi-index α = ( α 1 , … , α d ) ∈ { 0 , 1 , … } d \alpha=(\alpha_1,\ldots,\alpha_d)\in\{0,1,\ldots\}^d α = ( α 1 , … , α d ) ∈ { 0 , 1 , … } d , write ∣ a ∣ = ∑ k = 1 d a k |a|=\sum_{k=1}^{d}a_k ∣ a ∣ = ∑ k = 1 d a k and z a = ∏ k = 1 d z k a k z^a=\prod_{k=1}^{d}z_k^{a_k} z a = ∏ k = 1 d z k a k . Unless otherwise stated, c , C > 0 c,C>0 c , C > 0 denote constants that may change from line to line.
Problem Setup
In this section, we formalize the model identifiability for (1) . The model suffers from the well-known affine indeterminacy as in latent factor models (Anderson, [3 ] ). Specifically, for any μ ∈ R d \mu\in\mathbb{R}^d μ ∈ R d and invertible H ∈ R d × d H\in\mathbb{R}^{d\times d} H ∈ R d × d , let G ~ \widetilde{G} G denote the distribution of Z ~ = H − 1 ( Z − μ ) \widetilde{Z}=H^{-1}(Z-\mu) Z = H − 1 ( Z − μ ) when Z ∼ G Z\sim G Z ∼ G , and set θ ~ = ( A H , β + A μ ) \widetilde{\theta}=(AH,\beta+A\mu) θ = ( A H , β + A μ ) . A change of variables then gives P θ , G = P θ ~ , G ~ P_{\theta,G}=P_{\widetilde{\theta},\widetilde{G}} P θ , G = P θ , G . We first address this affine indeterminacy, and then formulate identifiability in terms of the remaining ambiguity arising from the interaction between the unknown latent distribution and the model parameters, which is the main focus of our analysis.
First, we adopt a common normalization on the latent variables:
E G ( Z ) = 0 , Cov G ( Z ) = I d . (3) \mathbb{E}_{G}(Z)=0,\qquad\operatorname{Cov}_{G}(Z)=I_d. \tag*{(3)} E G ( Z ) = 0 , Cov G ( Z ) = I d . (3)
When E G Z = m \mathbb{E}_{G}Z=m E G Z = m and Cov G ( Z ) = Σ ≻ 0 \operatorname{Cov}_{G}(Z)=\Sigma\succ0 Cov G ( Z ) = Σ ≻ 0 , we can simply take μ = m \mu=m μ = m and H = Σ 1 / 2 H=\Sigma^{1/2} H = Σ 1/2 to yield a standardized latent distribution without changing the marginal distribution. This normalization can also be replaced by more interpretable procedures; see Remark 1 below.
Normalization (3) still leaves a rotational indeterminacy, as for any orthogonal matrix H ∈ R d × d H\in\mathbb{R}^{d\times d} H ∈ R d × d , the preceding reparameterization preserves both normalization (3) and the marginal distribution. To account for this, we measure the distance metric between any θ = ( A , β ) \theta=(A,\beta) θ = ( A , β ) , θ ~ = ( A ~ , β ~ ) \widetilde{\theta}=(\widetilde{A},\widetilde{\beta}) θ = ( A , β ) as
D p ( θ , θ ~ ) = ∥ β − β ~ ∥ ∞ + ∥ A A T − A ~ A ~ T ∥ ∞ = max j ∈ [ p ] ∣ β j − β ~ j ∣ + max i , j ∈ [ p ] ∣ α i T α j − α ~ i T α ~ j ∣ . (4) D_p(\theta,\widetilde{\theta})=\|\beta-\widetilde{\beta}\|_\infty+\|AA^\mathsf{T}-\widetilde{A}\widetilde{A}^\mathsf{T}\|_\infty=\max_{j\in[p]}|\beta_j-\widetilde{\beta}_j|+\max_{i,j\in[p]}|\alpha_i^\mathsf{T}\alpha_j-\widetilde{\alpha}_i^\mathsf{T}\widetilde{\alpha}_j|. \tag*{(4)} D p ( θ , θ ) = ∥ β − β ∥ ∞ + ∥ A A T − A A T ∥ ∞ = j ∈ [ p ] max ∣ β j − β j ∣ + i , j ∈ [ p ] max ∣ α i T α j − α i T α j ∣. (4)
This is a rotation-invariant distance. In particular, D p ( θ , θ ~ ) = 0 D_p(\theta,\widetilde{\theta})=0 D p ( θ , θ ) = 0 if and only if ( A , β ) = ( A ~ H , β ~ ) (A,\beta)=(\widetilde{A}H,\widetilde{\beta}) ( A , β ) = ( A H , β ) for some orthogonal matrix H H H , and D p ( θ , θ ~ ) D_p(\theta,\widetilde{\theta}) D p ( θ , θ ) controls the maximum rowwise discrepancy between A A A and A ~ \widetilde{A} A after some orthogonal Procrustes alignment (Ten Berge, [58 ] ); see Lemma B.6 .
Our notion of model identifiability is then formulated as follows.
Definition 1. Consider model (1) . For each p p p , let M p \mathcal{M}_p M p be a nonempty set of pairs ( θ , G ) (\theta,G) ( θ , G ) satisfying (3) . The model is said to be exactly identifiable over M p \mathcal{M}_p M p if, for any ( θ , G ) , ( θ ~ , G ~ ) ∈ M p (\theta,G),(\widetilde{\theta},\widetilde{G})\in\mathcal{M}_p ( θ , G ) , ( θ , G ) ∈ M p ,
P θ , G = P θ ~ , G ~ ⟹ D p ( θ , θ ~ ) = 0 P_{\theta,G}=P_{\widetilde{\theta},\widetilde{G}}\quad\Longrightarrow\quad D_p(\theta,\widetilde{\theta})=0 P θ , G = P θ , G ⟹ D p ( θ , θ ) = 0 The model is said to be asymptotically identifiable over ( M p ) p ≥ 1 (\mathcal{M}_p)_{p\geq1} ( M p ) p ≥ 1 if, for any two sequences satisfying ( θ p , G p ) , ( θ ~ p , G ~ p ) ∈ M p (\theta_p,G_p),(\widetilde{\theta}_p,\widetilde{G}_p)\in\mathcal{M}_p ( θ p , G p ) , ( θ p , G p ) ∈ M p ,
P θ p , G p = P θ ~ p , G ~ p , ∀ p ⟹ D p ( θ p , θ ~ p ) → 0 as p → ∞ P_{\theta_p,G_p}=P_{\widetilde{\theta}_p,\widetilde{G}_p},\ \forall p\quad\Longrightarrow\quad D_p(\theta_p,\widetilde{\theta}_p)\to0\ \text{as }p\to\infty P θ p , G p = P θ p , G p , ∀ p ⟹ D p ( θ p , θ p ) → 0 as p → ∞
Exact identifiability is a standard assumption underlying consistency and inference results, whereas asymptotic identifiability is a weaker requirement that applies as the number of observed variables p p p grows. When P θ , G = P θ ~ , G ~ P_{\theta,G}=P_{\tilde{\theta},\tilde{G}} P θ , G = P θ ~ , G ~ , the two parameterizations induce the same distribution of the observed data and hence cannot be distinguished by any data-driven procedure, with the discrepancy D p ( θ , θ ~ ) D_p(\theta,\tilde{\theta}) D p ( θ , θ ~ ) quantifying the resulting population-level ambiguity. Classical theory for consistent point estimation and inference at fixed p p p therefore typically requires this discrepancy to be zero. Nevertheless, under suitable scaling of p p p and the sample size, asymptotic identifiability may still permit consistent estimation and valid inference, particularly when the ambiguity vanishes rapidly.
Remark 1. In Definition 1, the affine indeterminacy is resolved by the normalization (3) together with the rotationally invariant discrepancy D p ( ⋅ , ⋅ ) D_p(\cdot,\cdot) D p ( ⋅ , ⋅ ) . Besides these mathematically convenient conventions, rotation methods are often used in practice to select a simpler and more interpretable representation among equivalent ones [7 ] . Two common settings are as follows:
Orthogonal rotation. Suppose we retain the normalization (3) , and the remaining indeterminacy regards orthogonal transformations, as recovering A A A from A A T AA^{\mathsf{T}} A A T requires choosing an orientation within { A H : H T H = I d } \{AH:H^{\mathsf{T}}H=I_d\} { A H : H T H = I d } . Rotation methods such as varimax [29 , 50 ] can then be used to select some H ∗ H^{*} H ∗ to obtain A H ∗ AH^{*} A H ∗ with a simpler structure.
Oblique rotation. To allow correlated latent variables, condition Cov G ( Z ) = I d \operatorname{Cov}_G(Z)=I_d Cov G ( Z ) = I d can be relaxed to diag { Cov G ( Z ) } = I d \operatorname{diag}\{\operatorname{Cov}_G(Z)\}=I_d diag { Cov G ( Z )} = I d . Starting from a representation satisfying (3) , any invertible H H H satisfying diag ( H H T ) = I d \operatorname{diag}(HH^{\mathsf{T}})=I_d diag ( H H T ) = I d yields an equivalent representation
Z ~ = H Z , A ~ = A H − 1 , Cov ( Z ~ ) = H H T , \tilde{Z}=HZ,\qquad\tilde{A}=AH^{-1},\qquad\operatorname{Cov}(\tilde{Z})=HH^{\mathsf{T}}, Z ~ = H Z , A ~ = A H − 1 , Cov ( Z ~ ) = H H T ,
with diag { Cov G ( Z ~ ) } = I d \operatorname{diag}\{\operatorname{Cov}_G(\tilde{Z})\}=I_d diag { Cov G ( Z ~ )} = I d . The transformed latent coordinates have unit variances and may be correlated. Oblique rotation methods [37 ] (Cui, Chen, Ouyang and Xu 2026) can select some H ∗ H^{*} H ∗ to yield a simpler structure in A ( H ∗ ) − 1 A(H^{*})^{-1} A ( H ∗ ) − 1 .
Main Results
Impossibility of Exact Identifiability
When G G G is unspecified, it ranges over an infinite-dimensional space, and it is commonly believed that exact identifiability cannot hold in this case [22 , 36 , 53 ] . The following result confirms this belief and sharpens it by showing that exact identifiability fails almost everywhere in the parameter space, even when constraining finitely many moments of the latent distribution.
Theorem 1 (Non-identifiability). Assume that f ( x ) f(x) f ( x ) is strictly increasing and analytic with limits f ( + ∞ ) = 1 f(+\infty)=1 f ( + ∞ ) = 1 and f ( − ∞ ) = 0 f(-\infty)=0 f ( − ∞ ) = 0 , and that G G G has a positive C ∞ C^\infty C ∞ density on R d \mathbb{R}^d R d . Then for every finite p ≥ 1 p \ge1 p ≥ 1 , for every θ \theta θ outside a Lebesgue zero set, there exists a neighbourhood U U U of θ \theta θ such that each θ ~ ∈ U \tilde{\theta} \in U θ ~ ∈ U admits some G ~ \tilde{G} G ~ satisfying (3) and
P θ , G = P θ ~ , G ~ P_{\theta,G}=P_{\tilde{\theta},\tilde{G}} P θ , G = P θ ~ , G ~ Consequently, the model is not exactly identifiable in the sense of Definition 1. Moreover, for any fixed integer s ≥ 1 s \ge1 s ≥ 1 with E G ( ∥ Z ∥ 2 s ) < ∞ \mathbb{E}_G(\lVert Z\rVert^{2s})<\infty E G (∥ Z ∥ 2 s ) < ∞ , G ~ \tilde{G} G ~ can be chosen so that
E G ~ [ Z a ] = E G [ Z a ] , for all ∣ a ∣ ≤ 2 s , \mathbb{E}_{\tilde{G}}[Z^a]=\mathbb{E}_G[Z^a],\quad\text{for all } |a|\le2s, E G ~ [ Z a ] = E G [ Z a ] , for all ∣ a ∣ ≤ 2 s , where a a a is a multi-index and Z a = ∏ k = 1 d Z k a k Z^a=\prod_{k=1}^d Z_k^{a_k} Z a = ∏ k = 1 d Z k a k .
This result establishes that, when G G G is unspecified, exact identifiability fails for almost every parameter θ \theta θ , in the sense that the exception is a Lebesgue zero set. This exceptional set is used to exclude degenerate parameter configurations, such as α j = 0 \alpha_j=0 α j = 0 or identical ( α j , β j ) (\alpha_j,\beta_j) ( α j , β j ) across j ∈ [ p ] j\in[p] j ∈ [ p ] . Moreover, this identifiability issue cannot be addressed by restricting to some local neighbourhood ([52 ] ), as every sufficiently small parameter perturbation can be offset by a corresponding change in the latent distribution while preserving the marginal distribution. Finally, in the proof, G G G and G ~ \tilde{G} G ~ are constructed to share the same moments up to any prespecified order, beyond the normalization in (3) , provided these moments exist. Consequently, no fixed finite collection of moment restrictions on the latent distribution can restore exact identifiability.
Example 1. We illustrate the construction in Theorem 1 using a one-dimensional logistic model with f ( t ) = ( 1 + e − t ) − 1 f(t)=(1+e^{-t})^{-1} f ( t ) = ( 1 + e − t ) − 1 and latent distribution G = N ( 0 , 1 ) G=N(0,1) G = N ( 0 , 1 ) , whose density is denoted by ϕ ( ⋅ ) \phi(\cdot) ϕ ( ⋅ ) . Fix an admissible parameter θ = ( A , β ) \theta=(A,\beta) θ = ( A , β ) as in that theorem. For a fixed j ∈ [ p ] j\in[p] j ∈ [ p ] , consider the perturbed alternative θ ε = ( A , β + ε e j ) \theta_\varepsilon=(A,\beta+\varepsilon e_j) θ ε = ( A , β + ε e j ) , where e j ∈ R p e_j\in\mathbb{R}^p e j ∈ R p is the j j j th standard basis vector. We construct a perturbed distribution G ε G_\varepsilon G ε that offsets this parameter change, so that P θ , G = P θ ε , G ε P_{\theta,G}=P_{\theta_\varepsilon,G_\varepsilon} P θ , G = P θ ε , G ε , while matching the moments of G G G through order 2 s 2s 2 s and differing from G G G only within [ − 1 , 1 ] [-1,1] [ − 1 , 1 ] .
Write q S ( θ , z ) = ∏ k ∈ S f ( β k + α k z ) q_S(\theta,z)=\prod_{k\in S}f(\beta_k+\alpha_kz) q S ( θ , z ) = ∏ k ∈ S f ( β k + α k z ) for S ⊆ [ p ] S\subseteq[p] S ⊆ [ p ] and define
v ε ( z ) = [ 1 , z 2 , … , z 2 s , { q S ( θ ε , z ) + q S ( θ ε , − z ) } ∅ ≠ S ⊆ [ p ] ] T ∈ R s + 2 p . v_\varepsilon(z)=\left[1,z^2,\ldots,z^{2s},\left\{q_S(\theta_\varepsilon,z)+q_S(\theta_\varepsilon,-z)\right\}_{\varnothing\ne S\subseteq[p]}\right]^\mathsf{T}\in\mathbb{R}^{s+2^p}. v ε ( z ) = [ 1 , z 2 , … , z 2 s , { q S ( θ ε , z ) + q S ( θ ε , − z ) } ∅ = S ⊆ [ p ] ] T ∈ R s + 2 p . This feature vector encodes the normalization, the latent moments through order 2 s 2s 2 s , and the response probabilities of all subsets of [ p ] [p] [ p ] . Assume it is linearly independent on ( − 1 , 1 ) (-1,1) ( − 1 , 1 ) . To localize
the perturbation to the distribution, we use the smooth bump
w ( z ) = { exp { − 1 / ( 1 − z 2 ) } , ∣ z ∣ < 1 , 0 , ∣ z ∣ ≥ 1. w(z) =
\begin{cases}
\exp\{-1/(1-z^2)\}, & |z| < 1, \\
0, & |z| \ge1.
\end{cases} w ( z ) = { exp { − 1/ ( 1 − z 2 )} , 0 , ∣ z ∣ < 1 , ∣ z ∣ ≥ 1.
Set a weighted Gram matrix M ε = ∫ R w ( z ) v ε ( z ) v ε ( z ) T ϕ ( z ) d z M_\varepsilon= \int_{\mathbb{R}} w(z)v_\varepsilon(z)v_\varepsilon(z)^\mathsf{T}\phi(z)\,dz M ε = ∫ R w ( z ) v ε ( z ) v ε ( z ) T ϕ ( z ) d z and the vector of feature discrepancies b ε = ∫ R { v 0 ( z ) − v ε ( z ) } ϕ ( z ) d z b_\varepsilon= \int_{\mathbb{R}}\{v_0(z)-v_\varepsilon(z)\}\phi(z)\,dz b ε = ∫ R { v 0 ( z ) − v ε ( z )} ϕ ( z ) d z . For sufficiently small ∣ ε ∣ |\varepsilon| ∣ ε ∣ , M ε M_\varepsilon M ε is invertible. Then define the perturbed distribution by G ε G_\varepsilon G ε with density g ε ( z ) = ϕ ( z ) { 1 + w ( z ) v ε ( z ) T M ε − 1 b ε } g_\varepsilon(z) = \phi(z)\{1+w(z)v_\varepsilon(z)^\mathsf{T}M_\varepsilon^{-1}b_\varepsilon\} g ε ( z ) = ϕ ( z ) { 1 + w ( z ) v ε ( z ) T M ε − 1 b ε } . It is easy to check that g ε g_\varepsilon g ε is a positive and symmetric density for sufficiently small ∣ ε ∣ |\varepsilon| ∣ ε ∣ with g ε ( z ) = ϕ ( z ) g_\varepsilon(z)=\phi(z) g ε ( z ) = ϕ ( z ) for ∣ z ∣ ≥ 1 |z|\ge1 ∣ z ∣ ≥ 1 . By construction, one has
∫ R v ε ( z ) g ε ( z ) d z = ∫ R v ε ( z ) ϕ ( z ) d z + { ∫ R w ( z ) v ε ( z ) v ε ( z ) T ϕ ( z ) d z } M ε − 1 b ε = ∫ R v ε ( z ) ϕ ( z ) d z + b ε = ∫ R v 0 ( z ) ϕ ( z ) d z . (5) \begin{aligned}
\int_{\mathbb{R}}v_\varepsilon(z)g_\varepsilon(z)\,dz
&= \int_{\mathbb{R}}v_\varepsilon(z)\phi(z)\,dz
+ \left\{\int_{\mathbb{R}}w(z)v_\varepsilon(z)v_\varepsilon(z)^\mathsf{T}\phi(z)\,dz\right\}M_\varepsilon^{-1}b_\varepsilon\\
&= \int_{\mathbb{R}}v_\varepsilon(z)\phi(z)\,dz+b_\varepsilon
= \int_{\mathbb{R}}v_0(z)\phi(z)\,dz. \tag*{(5)}
\end{aligned} ∫ R v ε ( z ) g ε ( z ) d z = ∫ R v ε ( z ) ϕ ( z ) d z + { ∫ R w ( z ) v ε ( z ) v ε ( z ) T ϕ ( z ) d z } M ε − 1 b ε = ∫ R v ε ( z ) ϕ ( z ) d z + b ε = ∫ R v 0 ( z ) ϕ ( z ) d z . (5)
By symmetry of g ε g_\varepsilon g ε and ϕ \phi ϕ , this identity implies that E G [ Z a ] = E G ε [ Z a ] \mathbb{E}_G[Z^a]=\mathbb{E}_{G_\varepsilon}[Z^a] E G [ Z a ] = E G ε [ Z a ] for all a ≤ 2 s a\le2s a ≤ 2 s and E G [ q S ( θ , Z ) ] = E G ε [ q S ( θ ε , Z ) ] \mathbb{E}_G[q_S(\theta,Z)]=\mathbb{E}_{G_\varepsilon}[q_S(\theta_\varepsilon,Z)] E G [ q S ( θ , Z )] = E G ε [ q S ( θ ε , Z )] for all S S S . Therefore, P θ , G = P θ ε , G ε P_{\theta,G}=P_{\theta_\varepsilon,G_\varepsilon} P θ , G = P θ ε , G ε with D p ( θ ε , θ ) = ∣ ε ∣ D_p(\theta_\varepsilon,\theta)=|\varepsilon| D p ( θ ε , θ ) = ∣ ε ∣ . Analogous constructions can offset any other sufficiently small perturbation through changes in the latent distribution while preserving its moments to any finite order.
Theorem 1 establishes non-identifiability when the latent dimension d d d is fixed. We next show that the latent dimension d d d itself can also be ambiguous. Borrowing terminology from factor analysis ([42 ] ), we call a latent variable a weak factor if the corresponding column of A A A is small. The proposition shows that appending a weak factor to A A A can be offset by a perturbation of the latent distribution, leaving the marginal distribution unchanged. This ambiguity further motivates our treatment of the latent dimension d d d as specified throughout.
Proposition 1 (Ambiguity from a weak factor). *Under the setup of Theorem 1, suppose p > d p>d p > d , rank ( A ) = d \operatorname{rank}(A)=d rank ( A ) = d , and that θ = ( A , β ) \theta=(A,\beta) θ = ( A , β ) is outside the Lebesgue zero set specified by Theorem 1. Let H H H be any distribution on R \mathbb{R} R with mean zero and unit variance, and let G ⊗ H G\otimes H G ⊗ H denote the joint distribution of independent Z ∼ G Z\sim G Z ∼ G and U ∼ H U\sim H U ∼ H . Fix any v ∈ R p ∖ col ( A ) v\in\mathbb{R}^p\setminus\operatorname{col}(A) v ∈ R p ∖ col ( A ) and an integer s ≥ 1 s\ge1 s ≥ 1 with E G ∥ Z ∥ 2 s < ∞ \mathbb{E}_G\lVert Z\rVert^{2s}<\infty E G ∥ Z ∥ 2 s < ∞ . Then, for sufficiently small ∣ ε ∣ |\varepsilon| ∣ ε ∣ , there exists a d d d -dimensional latent distribution G ~ ε \widetilde{G}_\varepsilon G ε satisfying (3) such that
P ( [ A , ε v ] , β ) , G ⊗ H = P ( A , β ) , G ~ ε and E G [ Z a ] = E G ~ ε [ Z a ] for all ∣ a ∣ ≤ 2 s . P_{([A,\varepsilon v],\beta),G\otimes H}=P_{(A,\beta),\widetilde{G}_\varepsilon}
\quad\text{and}\quad
\mathbb{E}_G[Z^a]=\mathbb{E}_{\widetilde{G}_\varepsilon}[Z^a]\quad\text{for all }|a|\le2s. P ([ A , ε v ] , β ) , G ⊗ H = P ( A , β ) , G ε and E G [ Z a ] = E G ε [ Z a ] for all ∣ a ∣ ≤ 2 s . The additional variable U U U is the weak factor, with the corresponding column ε v \varepsilon v ε v of order ε \varepsilon ε .
This result shows that a ( d + 1 ) (d+1) ( d + 1 ) -dimensional model with one weak factor can yield the same marginal distribution for the observed variables as a d d d -dimensional model with a suitably perturbed latent distribution. This reflects the difficulty of detecting weak factors in factor analysis Onatski, [42 ] from the perspective of model identifiability. The equivalence also has a more positive interpretation that a flexible latent distribution can absorb the effect of omitted weak signal factors without changing the model parameters associated with strong factors.
Possibility of Asymptotic Identifiability
While Theorem 1 settles (Q1) in the negative, reliable estimation may still be possible if the model is asymptotically identifiable, which would answer (Q2) in the positive. In this subsection, we establish asymptotic identifiability of the model by deriving a quantitative bound on the discrepancy D p ( θ , θ ~ ) D_p(\theta,\tilde{\theta}) D p ( θ , θ ~ ) that holds uniformly over all { ( θ , G ) , ( θ ~ , G ~ ) } \{(\theta,G),(\tilde{\theta},\tilde{G})\} {( θ , G ) , ( θ ~ , G ~ )} that yield the same marginal distribution. The subsequent Section 3.3 draws two implications of this bound to answer (Q2).
We first introduce two technical assumptions.
Assumption 1. The link function f : R → ( 0 , 1 ) f:\mathbb{R}\to(0,1) f : R → ( 0 , 1 ) satisfies the following conditions:
(i) f f f is analytic with f ′ ( x ) > 0 f'(x)>0 f ′ ( x ) > 0 for every x ∈ R x\in\mathbb{R} x ∈ R , lim x → + ∞ f ( x ) = 1 \lim_{x\to+\infty}f(x)=1 lim x → + ∞ f ( x ) = 1 , and lim x → − ∞ f ( x ) = 0 \lim_{x\to-\infty}f(x)=0 lim x → − ∞ f ( x ) = 0 . Moreover, f ′ ( 0 ) ≥ m f'(0)\ge m f ′ ( 0 ) ≥ m for some constant m > 0 m>0 m > 0 .
(ii) There exists a constant K ≥ 1 K\ge1 K ≥ 1 such that for any integer k ≥ 2 k\ge2 k ≥ 2 , sup x ∈ R ∣ f ( k ) ( x ) k ! f ′ ( x ) ∣ ≤ K k − 1 \sup_{x\in\mathbb{R}}\left|\frac{f^{(k)}(x)}{k!f'(x)}\right|\le K^{k-1} sup x ∈ R k ! f ′ ( x ) f ( k ) ( x ) ≤ K k − 1 .
Assumption 1 imposes regularity conditions on the link function and is satisfied by, for example, the logistic link f ( x ) = ( 1 + e − x ) − 1 f(x)=(1+e^{-x})^{-1} f ( x ) = ( 1 + e − x ) − 1 and the Cauchy link f ( x ) = 1 2 + π − 1 arctan ( x ) f(x)=\frac{1}{2}+\pi^{-1}\arctan(x) f ( x ) = 2 1 + π − 1 arctan ( x ) . The first condition ensures a strictly increasing analytic response function with the usual boundary limits, with the lower bound on f ′ ( 0 ) f'(0) f ′ ( 0 ) preventing it from becoming arbitrarily flat at 0. Condition (ii) is the key technical requirement for the superpolynomial rate. It implies that f f f admits a Taylor expansion to arbitrary order around any point with coefficients bounded relative to f ′ f' f ′ . This uniform control is crucial for our technique to apply; see Remark 2 for more discussion.
Assumption 2. For ( θ , G ) ∈ M p (\theta,G)\in\mathcal{M}_p ( θ , G ) ∈ M p with M p \mathcal{M}_p M p given in Definition 1, we assume
(i) for some constant M > 0 M>0 M > 0 , max j ∈ [ p ] ∥ α j ∥ ≤ M \max_{j\in[p]}\|\alpha_j\|\le M max j ∈ [ p ] ∥ α j ∥ ≤ M ; max j ∈ [ p ] ∣ β j ∣ ≤ M \max_{j\in[p]}|\beta_j|\le M max j ∈ [ p ] ∣ β j ∣ ≤ M ;
(ii) λ min ( p − 1 A T A ) ≥ κ > 0 \lambda_{\min}(p^{-1}A^{\mathsf{T}}A)\ge\kappa>0 λ m i n ( p − 1 A T A ) ≥ κ > 0 for some constant κ \kappa κ ;
(iii) there exist some constants ν > 0 \nu>0 ν > 0 and L L L such that E G [ exp ( ν ∥ Z ∥ 2 ) ] ≤ L \mathbb{E}_G[\exp(\nu\|Z\|^2)]\le L E G [ exp ( ν ∥ Z ∥ 2 )] ≤ L .
Assumption 2 imposes boundedness and nondegeneracy conditions standard in the latent-variable literature. Assumption 2(iii) imposes a uniform sub-Gaussian tail bound on the latent distribution. It accommodates Gaussian latent distributions, widely used in classical item response models [6 , 47 ] , finitely supported discrete latent distributions, as arise in latent-class models [1 , 63 ] , and bounded-support latent distributions (Cui and Xu, 2026a; He, 2024). The condition places no restriction on the shape of the latent distribution, allowing asymmetry and multimodality, and does not require a density or any parametric form.
To quantify the identification ambiguity over the admissible set M p \mathcal{M}_p M p , we define
ω p ( M p ) = sup { D p ( θ , θ ~ ) : ( θ , G ) , ( θ ~ , G ~ ) ∈ M p , P θ , G = P θ ~ , G ~ } . \omega_p(\mathcal{M}_p)=\sup\left\{D_p(\theta,\tilde{\theta}):(\theta,G),(\tilde{\theta},\tilde{G})\in\mathcal{M}_p,\ P_{\theta,G}=P_{\tilde{\theta},\tilde{G}}\right\}. ω p ( M p ) = sup { D p ( θ , θ ~ ) : ( θ , G ) , ( θ ~ , G ~ ) ∈ M p , P θ , G = P θ ~ , G ~ } .
This gives the supremum of the D p D_p D p -diameters of sets of all admissible parameters that yield some common marginal distribution. In Example 1, for instance, it upper bounds ∣ ϵ ∣ |\epsilon| ∣ ϵ ∣ for every perturbation that preserves the marginal distribution. We have the following superpolynomial bound for ω p ( M p ) \omega_p(\mathcal{M}_p) ω p ( M p ) .
Theorem 2 (Asymptotic identifiability). Under Assumptions 1 and 2, the model (1) is asymptotically identifiable. In particular, there exist c c c , C C C , p 0 p_0 p 0 such that for every p ≥ p 0 p\ge p_0 p ≥ p 0 , it holds that
ω p ( M p ) ≤ C exp { − c ( log p ) 2 } . \omega_p(\mathcal{M}_p)\le C\exp\left\{-c(\log p)^2\right\}. ω p ( M p ) ≤ C exp { − c ( log p ) 2 } .
Theorem 2 shows that the identification ambiguity decays as fast as exp { − c ( log p ) 2 } = p − c log p \exp\{-c(\log p)^2\}=p^{-c\log p} exp { − c ( log p ) 2 } = p − c l o g p , and hence faster than any polynomial rate in p p p . The bound holds uniformly over all pairs in M p \mathcal{M}_p M p , with D p ( θ , θ ~ ) D_p(\theta,\tilde{\theta}) D p ( θ , θ ~ ) controlling the entrywise discrepancy between θ \theta θ and θ ~ \tilde{\theta} θ ~ . This essentially quantifies the size of the neighbourhood U \mathcal{U} U in Theorem 1, bounding the discrepancy of admissible alternatives by C exp { − c ( log p ) 2 } C\exp\{-c(\log p)^2\} C exp { − c ( log p ) 2 } in metric D p ( ⋅ , ⋅ ) D_p(\cdot,\cdot) D p ( ⋅ , ⋅ ) . It is therefore suggested that non-identifiability may pose less of an impediment to inference in latent variable models than is commonly expected. The result is stronger than existing asymptotic identifiability results ([16 , 23 ] ), which only establish a qualitative result under unidimensional models. Establishing the superpolynomial rate in Theorem 2 requires a more delicate analysis, which we illustrate as follows.
Remark 2. The primary challenge in establishing Theorem 2 is to approximate the parameters via observation Y Y Y to a sufficiently high order. We start with a conditioning argument. Select a small set S ⊆ [ p ] S\subseteq[p] S ⊆ [ p ] containing the indices of the parameters to approximate together with a few reference indices, and we condition on the remaining responses Y = X S c Y=X_{S^c} Y = X S c . Then Z Z Z can be approximated by its conditional mean E ( Z ∣ Y ) \mathbb{E}(Z\mid Y) E ( Z ∣ Y ) , with residual V = Z − E ( Z ∣ Y ) V=Z-\mathbb{E}(Z\mid Y) V = Z − E ( Z ∣ Y ) . Then for i , j ∈ S i,j\in S i , j ∈ S , we can write
β i = E ( T i ) + E ( δ i ) and α i T α j = Cov ( T i + δ i , T j + δ j ) + E { ( α i T V ) ( α j T V ) } . \beta_i=\mathbb{E}(T_i)+\mathbb{E}(\delta_i)
\quad\text{and}\quad
\alpha_i^{\mathsf{T}}\alpha_j=\operatorname{Cov}(T_i+\delta_i,T_j+\delta_j)+\mathbb{E}\{(\alpha_i^{\mathsf{T}}V)(\alpha_j^{\mathsf{T}}V)\}. β i = E ( T i ) + E ( δ i ) and α i T α j = Cov ( T i + δ i , T j + δ j ) + E {( α i T V ) ( α j T V )} .
where T i = f − 1 { E ( X i ∣ Y ) } T_i=f^{-1}\{\mathbb{E}(X_i\mid Y)\} T i = f − 1 { E ( X i ∣ Y )} is determined by the marginal distribution, whereas δ i = E ( α i T Z + β i T Y ∣ Y ) − T i \delta_i=\mathbb{E}(\alpha_i^{\mathsf{T}}Z+\beta_i^{\mathsf{T}}Y\mid Y)-T_i δ i = E ( α i T Z + β i T Y ∣ Y ) − T i depends on unknown model parameters. Here, δ i \delta_i δ i is a correction term that we separate out to simplify the derivation.
The key difficulty is to approximate δ i \delta_i δ i with sufficiently small error. We work with the conditional latent moments M k , i = E { ( α i T V ) k ∣ Y } M_{k,i}=\mathbb{E}\{(\alpha_i^{\mathsf{T}}V)^k\mid Y\} M k , i = E {( α i T V ) k ∣ Y } , which satisfy ∣ M k , i ∣ ≤ e C k 2 p − k / 2 \lvert M_{k,i}\rvert\le e^{Ck^2}p^{-k/2} ∣ M k , i ∣ ≤ e C k 2 p − k /2 for k ≥ 2 k\ge2 k ≥ 2 (Lemma B.2 ), so that expansions in these moments become more accurate as the order increases. For δ i \delta_i δ i , for instance, to fourth order, it can be approximated by
δ i ≈ − f ′ ′ ( T i ) 2 f ′ ( T i ) M 2 , i − f ′ ′ ′ ( T i ) 6 f ′ ( T i ) M 3 , i − f ( 4 ) ( T i ) 24 f ′ ( T i ) M 4 , i + { f ′ ′ ( T i ) f ′ ′ ′ ( T i ) 4 f ′ ( T i ) 2 − f ′ ′ ( T i ) 3 8 f ′ ( T i ) 3 } M 2 , i 2 + ⋯ \delta_i \approx-\frac{f^{\prime\prime}(T_i)}{2f^\prime(T_i)}M_{2,i}-\frac{f^{\prime\prime\prime}(T_i)}{6f^\prime(T_i)}M_{3,i}-\frac{f^{(4)}(T_i)}{24f^\prime(T_i)}M_{4,i}+\left\{\frac{f^{\prime\prime}(T_i)f^{\prime\prime\prime}(T_i)}{4f^\prime(T_i)^2}-\frac{f^{\prime\prime}(T_i)^3}{8f^\prime(T_i)^3}\right\}M_{2,i}^2+\cdots δ i ≈ − 2 f ′ ( T i ) f ′′ ( T i ) M 2 , i − 6 f ′ ( T i ) f ′′′ ( T i ) M 3 , i − 24 f ′ ( T i ) f ( 4 ) ( T i ) M 4 , i + { 4 f ′ ( T i ) 2 f ′′ ( T i ) f ′′′ ( T i ) − 8 f ′ ( T i ) 3 f ′′ ( T i ) 3 } M 2 , i 2 + ⋯
Here, M 4 , i M_{4,i} M 4 , i and M 2 , i 2 M_{2,i}^2 M 2 , i 2 are both of order p − 2 p^{-2} p − 2 but must be tracked separately. Higher orders generate more moment products of this type, with growing complexity in their coefficients, and it is thus difficult to ensure the precision of the approximation. Moreover, the latent moments are not observed and thus must themselves be approximated from observed quantities, which introduces further error terms that have to cancel successively to reach the desired order.
To handle these calculations, we introduce a symbolic calculus that organizes moment products as the expansion order N N N increases. This separates algebraic bookkeeping from the probabilistic estimates needed for the generally dependent conditional moments. Then for pairs ( θ , G ) (\theta,G) ( θ , G ) and ( θ ~ , G ~ ) (\tilde{\theta},\tilde{G}) ( θ ~ , G ~ ) that yield the same marginal distribution, we can construct the expansions with θ \theta θ first, and then apply the same expansions to θ ~ \tilde{\theta} θ ~ . After careful control of the higher-order terms, we obtain
D p ( θ , θ ~ ) ≤ C D p ( θ , θ ~ ) p + e c N 2 p − ( N + 1 ) / 2 . D_p(\theta,\tilde{\theta}) \leq\frac{CD_p(\theta,\tilde{\theta})}{p}+e^{cN^2}p^{-(N+1)/2}. D p ( θ , θ ~ ) ≤ p C D p ( θ , θ ~ ) + e c N 2 p − ( N + 1 ) /2 .
The first term controls the parameter-dependent differences between the expansions, with the factor p − 1 p^{-1} p − 1 reflecting the order at which the correction terms begin. The second term bounds the order-N N N approximation error uniformly over θ \theta θ and θ ~ \tilde{\theta} θ ~ , which aligns with the bound for latent moments M k , i M_{k,i} M k , i , with e c N 2 e^{cN^2} e c N 2 accounting for the growth of the expansion coefficients. Absorbing the first term and choosing N = ⌊ τ log p ⌋ N=\lfloor\tau\log p\rfloor N = ⌊ τ log p ⌋ for sufficiently small τ > 0 \tau>0 τ > 0 gives the superpolynomial bound.
We further specialize our results to three concrete examples of latent variable models. These specializations follow directly from our main results, but the results may be of independent interest in each setting.
Example 2 (Bifactor Models). The bifactor model is an important latent trait model in psychological and educational assessment ([48 ] ). It specifies one general variable Z 0 Z_0 Z 0 and d d d group variables Z 1 , … , Z d Z_1,\ldots,Z_d Z 1 , … , Z d , giving d + 1 d+1 d + 1 latent variables in total. Each observed response X j X_j X j is allowed to depend only on the general variable Z 0 Z_0 Z 0 and at most one group variable Z g ( j ) Z_{g(j)} Z g ( j ) , where g ( j ) ∈ { 1 , … , d } g(j)\in\{1,\ldots,d\} g ( j ) ∈ { 1 , … , d } . Its
conditional response probability therefore takes the form
pr ( X j = 1 ∣ Z ) = f { β j + λ j 0 Z 0 + λ j 1 Z g ( j ) } . \operatorname{pr}(X_j = 1 \mid Z) = f\left\{\beta_j + \lambda_{j0}Z_0 + \lambda_{j1}Z_{g(j)}\right\}. pr ( X j = 1 ∣ Z ) = f { β j + λ j 0 Z 0 + λ j 1 Z g ( j ) } .
The vector Z = ( Z 0 , Z 1 , … , Z d ) T Z = (Z_0, Z_1, \ldots, Z_d)^\mathsf{T} Z = ( Z 0 , Z 1 , … , Z d ) T has mean zero and covariance Σ = diag ( 1 , R ) \Sigma= \operatorname{diag}(1, R) Σ = diag ( 1 , R ) , where R R R has unit diagonal. This covariance structure requires Z 0 Z_0 Z 0 to be uncorrelated with the group variables, a common restriction used to facilitate interpretation and identification. In particular, Fang et al. [18 ] established identifiability under this restriction with Gaussian latent variables.
Our theory establishes asymptotic identifiability for a broader class of latent distributions beyond the Gaussian family, while retaining the covariance structure. Let λ j ∈ R d + 1 \lambda_j \in\mathbb{R}^{d+1} λ j ∈ R d + 1 have first entry λ j 0 \lambda_{j0} λ j 0 , entry g ( j ) + 1 g(j) + 1 g ( j ) + 1 equal to λ j 1 \lambda_{j1} λ j 1 , and all remaining entries zero. Write Λ = ( λ 1 , … , λ p ) T \Lambda= (\lambda_1, \ldots, \lambda_p)^\mathsf{T} Λ = ( λ 1 , … , λ p ) T . Under suitable regularity conditions (see details in Section B.6 in the Supplementary Material), Theorem 2 implies that, for ( Λ , β , G ) (\Lambda, \beta, G) ( Λ , β , G ) and ( Λ ~ , β ~ , G ~ ) (\widetilde{\Lambda}, \widetilde{\beta}, \widetilde{G}) ( Λ , β , G ) that yield the same marginal distribution,
∥ β − β ~ ∥ ∞ + ∥ Λ ~ − Λ ∥ ∞ + ∥ Σ ~ − Σ ∥ ∞ ≤ C exp { − c ( log p ) 2 } . \lVert\beta- \widetilde{\beta} \rVert_\infty+ \lVert\widetilde{\Lambda} - \Lambda\rVert_\infty+ \lVert\widetilde{\Sigma} - \Sigma\rVert_\infty\leq C\exp\left\{-c(\log p)^2\right\}. ∥ β − β ∥ ∞ + ∥ Λ − Λ ∥ ∞ + ∥ Σ − Σ ∥ ∞ ≤ C exp { − c ( log p ) 2 } .
for some constants C , c > 0 C, c > 0 C , c > 0 , after proper sign change of the latent variables, where Σ \Sigma Σ and Σ ~ \widetilde{\Sigma} Σ are the covariance matrices of G G G and G ~ \widetilde{G} G , respectively.
Example 3 (Independent Component Analysis). Independent component analysis (ICA) aims to recover independent latent variables, also referred to as latent sources, from observed mixtures (Hyvärinen and Pajunen [28 ] ; Khemakhem et al. [30 ] ). Model (1) can be viewed as a binary ICA model when G G G is restricted to have independent components Z 1 , … , Z d Z_1, \ldots, Z_d Z 1 , … , Z d , with A A A being the mixing matrix specifying how the latent sources are combined to determine the response probabilities. Theorem 2 implies that min Q T Q = I d max j ∈ [ p ] ∥ α ~ j − Q α j ∥ ≤ C exp { − c ( log p ) 2 } \min_{Q^\mathsf{T}Q=I_d}\max_{j\in[p]}\lVert\widetilde{\alpha}_j - Q\alpha_j\rVert\leq C\exp\left\{-c(\log p)^2\right\} min Q T Q = I d max j ∈ [ p ] ∥ α j − Q α j ∥ ≤ C exp { − c ( log p ) 2 } . Thus, as the number p p p of observed mixtures grows, the mixing matrix is asymptotically identifiable up to an orthogonal transformation, an indeterminacy that is commonly resolved by independence and non-Gaussianity conditions on the sources (Hyvärinen and Oja [27 ] ). This result provides a quantitative perspective on the identifiability theory of ICA.
Example 4 (Generative Models with Binary Decoders). Model (1) can be viewed as the generative model underlying a variational autoencoder for binary data (Kingma and Welling [31 ] ), with prior G G G and decoder m θ ( z ) = { f ( β j + α j T z ) } j ∈ [ p ] m_\theta(z) = \{f(\beta_j + \alpha_j^\mathsf{T}z)\}_{j\in[p]} m θ ( z ) = { f ( β j + α j T z ) } j ∈ [ p ] . Learning the prior, for example through normalizing flows (Papamakarios et al. [45 ] ), can introduce additional indeterminacies. For instance, Xi and Bloem-Reddy (2023) characterized such indeterminacies, but their analysis is limited to continuous decoders. Our result shows that, under suitable regularity conditions and with G G G left unspecified, any two observationally equivalent representations satisfy ∥ β − β ~ ∥ ∞ + max j ∈ [ p ] ∥ α ~ j − Q α j ∥ ≤ C exp { − c ( log p ) 2 } \lVert\beta- \widetilde{\beta}\rVert_\infty+ \max_{j\in[p]}\lVert\widetilde{\alpha}_j - Q\alpha_j\rVert\leq C\exp\left\{-c(\log p)^2\right\} ∥ β − β ∥ ∞ + max j ∈ [ p ] ∥ α j − Q α j ∥ ≤ C exp { − c ( log p ) 2 } for some orthogonal matrix Q Q Q . Consequently, for each fixed R > 0 R > 0 R > 0 , the decoder
discrepancy satisfies
sup ∥ z ∥ ≤ R ∥ m θ ~ ( Q z ) − m θ ( z ) ∥ ∞ ≤ C R exp { − c ( log p ) 2 } , \sup_{\lVert z\rVert\le R} \lVert m_{\widetilde{\theta}}(Qz)-m_{\theta}(z)\rVert_{\infty} \le C_R\exp\{-c(\log p)^2\}, ∥ z ∥ ≤ R sup ∥ m θ ( Q z ) − m θ ( z ) ∥ ∞ ≤ C R exp { − c ( log p ) 2 } ,
where C R C_R C R is some constant depending on the radius R R R . Thus the decoder is asymptotically identifiable up to a rotation of the latent space, even when the prior is learned flexibly.
Implications
This subsection presents two consequences of Theorem 2, concerning estimability of parameter θ \theta θ and latent distribution G G G to address (Q2). The first concerns parameter estimation.
Corollary 1 (Estimation of model parameter). Under Assumptions 1 and 2 , suppose ( θ 0 , G 0 ) ∈ M p (\theta_0,G_0) \in\mathcal{M}_p ( θ 0 , G 0 ) ∈ M p and ( θ † , G † ) ∈ M p (\theta^\dagger,G^\dagger) \in\mathcal{M}_p ( θ † , G † ) ∈ M p yield the same marginal distribution P θ 0 , G 0 P_{\theta_0,G_0} P θ 0 , G 0 under model (1) . For any estimator θ ^ n \hat{\theta}_n θ ^ n based on n n n independent responses, we have
D p ( θ ^ n , θ 0 ) = D p ( θ ^ n , θ † ) + O P ( exp { − c ( log p ) 2 } ) . (6) D_p(\hat{\theta}_n,\theta_0)=D_p(\hat{\theta}_n,\theta^\dagger)+O_{\mathbb{P}}\left(\exp\{-c(\log p)^2\}\right). \tag*{(6)} D p ( θ ^ n , θ 0 ) = D p ( θ ^ n , θ † ) + O P ( exp { − c ( log p ) 2 } ) . (6)
Thus when n , p → ∞ n,p \to\infty n , p → ∞ and D p ( θ ^ n , θ † ) → 0 D_p(\hat{\theta}_n,\theta^\dagger) \to0 D p ( θ ^ n , θ † ) → 0 , the estimator is consistent up to some orthogonal transformation. Moreover, suppose ψ ( ⋅ ) \psi(\cdot) ψ ( ⋅ ) is a scalar functional satisfying ∣ ψ ( θ ) − ψ ( θ ~ ) ∣ ≤ L ψ D p ( θ , θ ~ ) |\psi(\theta)-\psi(\widetilde{\theta})| \le L_\psi D_p(\theta,\widetilde{\theta}) ∣ ψ ( θ ) − ψ ( θ ) ∣ ≤ L ψ D p ( θ , θ ) for a constant L ψ L_\psi L ψ independent of p p p . If s ^ n , p − 1 { ψ ( θ ^ n ) − ψ ( θ † ) } \hat{s}_{n,p}^{-1}\{\psi(\hat{\theta}_n)-\psi(\theta^\dagger)\} s ^ n , p − 1 { ψ ( θ ^ n ) − ψ ( θ † )} has a limiting distribution as n , p → ∞ n,p \to\infty n , p → ∞ , for some sequence s ^ n , p \hat{s}_{n,p} s ^ n , p satisfying s ^ n , p − 1 exp { − c ( log p ) 2 } → 0 \hat{s}_{n,p}^{-1}\exp\{-c(\log p)^2\} \to0 s ^ n , p − 1 exp { − c ( log p ) 2 } → 0 , then s ^ n , p − 1 { ψ ( θ ^ n ) − ψ ( θ 0 ) } \hat{s}_{n,p}^{-1}\{\psi(\hat{\theta}_n)-\psi(\theta_0)\} s ^ n , p − 1 { ψ ( θ ^ n ) − ψ ( θ 0 )} has the same limiting distribution.
Corollary 1 separates the estimation error into two parts. The identification ambiguity, captured by the second term in (6) , is controlled by Theorem 2 regardless of the estimator. The remaining D p ( θ ^ n , θ † ) D_p(\hat{\theta}_n,\theta^\dagger) D p ( θ ^ n , θ † ) measures the sampling error relative to a target θ † \theta^\dagger θ † satisfying P θ † , G † = P θ 0 , G 0 P_{\theta^\dagger,G^\dagger}=P_{\theta_0,G_0} P θ † , G † = P θ 0 , G 0 for some G † G^\dagger G † , which depends on the specific estimation procedure. The corollary also allows valid inference for ψ ( θ 0 ) \psi(\theta_0) ψ ( θ 0 ) , provided that s ^ n , p − 1 { ψ ( θ ^ n ) − ψ ( θ † ) } \hat{s}_{n,p}^{-1}\{\psi(\hat{\theta}_n)-\psi(\theta^\dagger)\} s ^ n , p − 1 { ψ ( θ ^ n ) − ψ ( θ † )} converges to a normal distribution and the identification ambiguity is negligible relative to the standard error. For root-n n n inference, for instance, the additional requirement is n 1 / 2 exp { − c ( log p ) 2 } → 0 n^{1/2}\exp\{-c(\log p)^2\} \to0 n 1/2 exp { − c ( log p ) 2 } → 0 , which holds, for example, whenever log n = o { ( log p ) 2 } \log n=o\{(\log p)^2\} log n = o {( log p ) 2 } . The result thus provides a basis for consistent estimation and valid inference for procedures converging to population solutions that reproduce the true marginal distribution. Such population solutions arise naturally in marginal likelihood estimation. Identification ambiguity contributes only a superpolynomially small error and therefore need not obstruct consistent estimation.
There have been studies developing asymptotic theory for estimation of model (1) . In the semiparametric literature, for instance, Lindsay et al. [36 ] linked conditional likelihood and mixture likelihood estimation in the Rasch model and related exponential response models. For generalized linear latent variable models, Ma and Genton (2010) developed semiparametric estimating equations and established root-n n n consistency and asymptotic normality. Both studies take model identifiability as given. In a growing-p p p regime, Cui and Xu [13 ] recently established consistency and asymptotic normality of marginal maximum likelihood estimators under a working Gaussian latent distribution, requiring only that the true latent distribution be sub-Gaussian. In their analysis, the working density contributes a leading first-order term to the likelihood expansion, and the resulting asymptotic theory targets a canonical representation that need not coincide with the true parameter.
We next consider the estimation and inference of the latent distribution G G G . We measure the discrepancy between two latent distributions by the 1-Wasserstein distance
W 1 ( G , G ~ ) = inf π ∈ Π ( G , G ~ ) ∫ R d × R d ∥ z − z ~ ∥ d π ( z , z ~ ) , W_1(G,\widetilde{G}) = \inf_{\pi\in\Pi(G,\widetilde{G})} \int_{\mathbb{R}^d\times\mathbb{R}^d} \lVert z-\widetilde{z}\rVert\,d\pi(z,\widetilde{z}), W 1 ( G , G ) = π ∈ Π ( G , G ) inf ∫ R d × R d ∥ z − z ∥ d π ( z , z ) ,
where Π ( G , G ~ ) \Pi(G,\widetilde{G}) Π ( G , G ) is the set of couplings of G G G and G ~ \widetilde{G} G . For an orthogonal matrix Q Q Q , write Q # G Q_{\#}G Q # G for the distribution of Q Z QZ QZ when Z ∼ G Z\sim G Z ∼ G . We have the following result.
Corollary 2 (Estimation of the latent distribution). Under Assumptions 1 and 2, suppose ( θ 0 , G 0 ) , ( θ † , G † ) ∈ M p (\theta_0,G_0),(\theta^\dagger,G^\dagger)\in\mathcal{M}_p ( θ 0 , G 0 ) , ( θ † , G † ) ∈ M p and P θ 0 , G 0 = P θ † , G † P_{\theta_0,G_0}=P_{\theta^\dagger,G^\dagger} P θ 0 , G 0 = P θ † , G † . There exists an orthogonal matrix Q Q Q such that, for any estimator G ^ n \widehat{G}_n G n based on n n n independent responses and having a finite first moment,
W 1 ( Q # G ^ n , G 0 ) = W 1 ( G ^ n , G † ) + O P ( p − 1 / 2 ) . (7) W_1(Q_{\#}\widehat{G}_n,G_0)=W_1(\widehat{G}_n,G^\dagger)+O_{\mathbb{P}}(p^{-1/2}). \tag*{(7)} W 1 ( Q # G n , G 0 ) = W 1 ( G n , G † ) + O P ( p − 1/2 ) . (7) Consequently, when n , p → ∞ n,p\to\infty n , p → ∞ and W 1 ( G ^ n , G † ) → 0 W_1(\widehat{G}_n,G^\dagger)\to0 W 1 ( G n , G † ) → 0 , Q # G ^ n Q_{\#}\widehat{G}_n Q # G n is consistent for G 0 G_0 G 0 in W 1 W_1 W 1 . Moreover, for every L h L_h L h -Lipschitz function h : R d → R h:\mathbb{R}^d\to\mathbb{R} h : R d → R , satisfying ∣ h ( z ) − h ( z ′ ) ∣ ≤ L h ∥ z − z ′ ∥ |h(z)-h(z')|\le L_h\lVert z-z'\rVert ∣ h ( z ) − h ( z ′ ) ∣ ≤ L h ∥ z − z ′ ∥ , we have, almost surely for all sufficiently large p p p ,
∣ E G ^ n [ h ( Q Z ) ] − E G 0 [ h ( Z ) ] ∣ ≤ L h { W 1 ( G ^ n , G † ) + C p − 1 / 2 } . (8) \left|\mathbb{E}_{\widehat{G}_n}[h(QZ)]-\mathbb{E}_{G_0}[h(Z)]\right|\le L_h\left\{W_1(\widehat{G}_n,G^\dagger)+Cp^{-1/2}\right\}. \tag*{(8)} E G n [ h ( QZ )] − E G 0 [ h ( Z )] ≤ L h { W 1 ( G n , G † ) + C p − 1/2 } . (8)
Corollary 2 shows in (7) that, as p → ∞ p\to\infty p → ∞ , Q # G ^ n Q_{\#}\widehat{G}_n Q # G n can be consistent for G 0 G_0 G 0 provided that G ^ n \widehat{G}_n G n converges in W 1 W_1 W 1 to some G † G^\dagger G † that, together with some θ † \theta^\dagger θ † , induces the same marginal distribution as ( θ 0 , G 0 ) (\theta_0,G_0) ( θ 0 , G 0 ) . The identification ambiguity for the latent distribution is of order p − 1 / 2 p^{-1/2} p − 1/2 . For comparison, the minimax W 1 W_1 W 1 rate for estimating a distribution on R d \mathbb{R}^d R d from n n n observations is n − 1 / d n^{-1/d} n − 1/ d for d ≥ 3 d\ge3 d ≥ 3 without involving latent variables [55 ] , so the ambiguity is negligible when p ≫ n 2 / d p\gg n^{2/d} p ≫ n 2/ d .
Equation (8) is useful from a practical view. It shows that the failure of exact identifiability need not prevent accurate estimation of population features of the latent distribution, which are often the quantities of scientific interest. For instance, in a latent-trait measurement model, h ( z ) h(z) h ( z ) may be the success probability on a new testing item for an individual with latent trait z z z , so that E G 0 [ h ( Z ) ] \mathbb{E}_{G_0}[h(Z)] E G 0 [ h ( Z )] is the population success rate on that item. In a Bernoulli generative model for binary images, h ( z ) h(z) h ( z ) may be the mean activation probability over a specified image region under a given decoder, so that E G 0 [ h ( Z ) ] \mathbb{E}_{G_0}[h(Z)] E G 0 [ h ( Z )] is the expected fraction of active pixels in that region. These guarantees provide a theoretical basis for downstream population-level inference.
Simulation Study
We illustrate Theorems 1 and 2 via a numerical study. We use a logistic function for f ( ⋅ ) f(\cdot) f ( ⋅ ) . Consider d = 1 d=1 d = 1 and d = 3 d=3 d = 3 , and for each d d d , we draw ten sets of parameters θ 0 = ( A , β ) \theta_0=(A,\beta) θ 0 = ( A , β ) with independent rows. For d = 1 d=1 d = 1 , α j ∼ U ( 0.5 , 1.5 ) \alpha_j\sim U(0.5,1.5) α j ∼ U ( 0.5 , 1.5 ) . For d = 3 d=3 d = 3 , α j = r j u j \alpha_j=r_j u_j α j = r j u j , where r j ∼ U ( 0.5 , 1.5 ) r_j\sim U(0.5,1.5) r j ∼ U ( 0.5 , 1.5 ) and u j u_j u j is uniform on the unit sphere of R 3 \mathbb{R}^3 R 3 . In both cases, β j ∼ U ( − 1 , 1 ) \beta_j\sim U(-1,1) β j ∼ U ( − 1 , 1 ) . A model with p p p responses uses the first p p p rows. We consider two choices of latent distributions G 0 G_0 G 0 : N ( 0 , I d ) N(0,I_d) N ( 0 , I d ) , and the mixture 0.7 N ( 0 , 0.25 I d ) + 0.3 N ( 2 d − 1 / 2 1 d , 0.25 I d ) 0.7N(0,0.25I_d)+0.3N(2d^{-1/2}\mathbf{1}_d,0.25I_d) 0.7 N ( 0 , 0.25 I d ) + 0.3 N ( 2 d − 1/2 1 d , 0.25 I d ) , with 1 d \mathbf{1}_d 1 d denoting the d d d -dimensional all-1 vector.
For a candidate parameter θ ~ \widetilde{\theta} θ , we measure how well P θ 0 , G 0 P_{\theta_0,G_0} P θ 0 , G 0 can be reproduced with θ ~ \widetilde{\theta} θ by
Δ p ( θ ~ ) = min G ~ ∈ G g r i d , d ∥ p θ ~ , G ~ − p θ 0 , G 0 ∥ ∥ p θ 0 , G 0 ∥ (9) \Delta_p(\widetilde{\theta})=\min_{\widetilde{G}\in\mathcal{G}_{\mathrm{grid},d}}\frac{\left\|p_{\widetilde{\theta},\widetilde{G}}-p_{\theta_0,G_0}\right\|}{\left\|p_{\theta_0,G_0}\right\|} \tag*{(9)} Δ p ( θ ) = G ∈ G grid , d min ∥ p θ 0 , G 0 ∥ p θ , G − p θ 0 , G 0 (9)
where p θ , G = { p θ , G ( x ) } x ∈ { 0 , 1 } p p_{\theta,G}=\{p_{\theta,G}(x)\}_{x\in\{0,1\}^p} p θ , G = { p θ , G ( x ) } x ∈ { 0 , 1 } p is the vector of response probabilities defined in (2) . The class G g r i d , d \mathcal{G}_{\mathrm{grid},d} G grid , d consists of distributions supported on a fixed grid that satisfy (3) and the tail bound E G ~ exp ( ∥ Z ∥ 2 / 4 ) ≤ 2 ( d + 1 ) / 2 \mathbb{E}_{\widetilde{G}}\exp(\|Z\|^2/4)\leq2^{(d+1)/2} E G exp ( ∥ Z ∥ 2 /4 ) ≤ 2 ( d + 1 ) /2 as required by Assumption 2(iii). This grid contains 121 equally spaced points on [ − 12 , 12 ] [-12,12] [ − 12 , 12 ] for d = 1 d=1 d = 1 and 17 3 17^3 1 7 3 points on [ − 6 , 6 ] 3 [-6,6]^3 [ − 6 , 6 ] 3 for d = 3 d=3 d = 3 . We evaluate P θ 0 , G 0 P_{\theta_0,G_0} P θ 0 , G 0 using a finer discretization of G 0 G_0 G 0 , with 241 and 41 3 41^3 4 1 3 points on the respective domains, followed by exact standardization. For d = 1 d=1 d = 1 and p = 7 , 8 , 9 p=7,8,9 p = 7 , 8 , 9 , we discretize G 0 G_0 G 0 and G ~ \widetilde{G} G on shared standardized grids of 257, 513 and 1025 points, respectively, and increase the arithmetic precision to 2500, 4000 and 6500 bits to resolve the smaller perturbations. Thus, Δ p ( θ ~ ) = 0 \Delta_p(\widetilde{\theta})=0 Δ p ( θ ) = 0 if and only if there exists an admissible G ~ ∈ G g r i d , d \widetilde{G}\in\mathcal{G}_{\mathrm{grid},d} G ∈ G grid , d such that P θ , G ~ = P θ 0 , G 0 P_{\theta,\widetilde{G}}=P_{\theta_0,G_0} P θ , G = P θ 0 , G 0 . Numerical equivalence is assessed via this discretized form, with details described in Appendix § 10 .
We examine three perturbation paths: the scale path θ ~ t = ( ( 1 + t ) A , β ) \widetilde{\theta}_t=((1+t)A,\beta) θ t = (( 1 + t ) A , β ) , the intercept path θ ~ t = ( A , β + t e 1 ) \widetilde{\theta}_t=(A,\beta+t e_1) θ t = ( A , β + t e 1 ) , and the random path θ ~ t = θ 0 + t U \widetilde{\theta}_t=\theta_0+tU θ t = θ 0 + t U , where U U U is uniform on the unit sphere of R p ( d + 1 ) \mathbb{R}^{p(d+1)} R p ( d + 1 ) . Along each path, we estimate t p t_p t p as the transition at numerically zero value of Δ p ( θ ~ t ) \Delta_p(\widetilde{\theta}_t) Δ p ( θ t ) . The estimated transitions illustrate the ambiguity along corresponding directions, which serve as a proxy for ω p \omega_p ω p . We adopt this proxy because computing the full diameter ω p \omega_p ω p would require optimizing over all admissible pairs and can be computationally infeasible. Figure 1 and Figure 2 summarize the results for d = 1 d=1 d = 1 and d = 3 d=3 d = 3 , respectively.
Figure 1. Exact equivalence at the population level for d = 1 d = 1 d = 1 .
Figure 2. Exact equivalence at the population level for d = 3 d = 3 d = 3 .
Figure 1 shows that the identification ambiguity decays rapidly as p p p increases when d = 1 d=1 d = 1 . Panel (a) illustrates the discrepancy Δ p ( θ ~ t ) \Delta_p(\widetilde{\theta}_t) Δ p ( θ t ) along the scale path over a high-resolution grid; see Appendix § 10 for detail. The flat segments in the bottom band mark intervals of numerically zero discrepancy, illustrating the non-identifiability established in Theorem 1. These intervals contract sharply, with the endpoint approximately t 6 ≈ 10 − 34 t_6 \approx10^{-34} t 6 ≈ 1 0 − 34 at p = 6 p = 6 p = 6 , indicating that the ambiguity is extremely small when d = 1 d = 1 d = 1 . Panel (b) shows a similar decline in D p ( θ 0 , θ ~ t p ) D_p(\theta_0,\tilde{\theta}_{t_p}) D p ( θ 0 , θ ~ t p ) along all three paths, with comparable patterns under the Gaussian and bimodal latent distributions. Figure 2 shows a similar pattern for d = 3 d = 3 d = 3 , with the ambiguity larger over the displayed range. This can be explained by the greater flexibility resulting from a higher-dimensional latent distribution, which may require more observed responses to achieve a comparable scale of identification ambiguity. Overall, numerical results illustrate how the ambiguity depends on the latent dimension, validating the vanishing ambiguity as p p p grows, established by Theorem 2.
Conclusion
This paper studies identifiability in latent variable models with flexible latent distributions. We show that, when the latent distribution is unknown and learned jointly with the model parameters, exact identifiability fails for almost every parameter, even under moment constraints of any prespecified finite order. Despite this non-identifiability, the model parameters remain estimable. Specifically, we establish that the resulting identification ambiguity vanishes uniformly over admissible parameters at the superpolynomial rate exp { − c ( log p ) 2 } \exp\{-c(\log p)^2\} exp { − c ( log p ) 2 } , where p p p is the number of observed variables. Consequently, any parameter that induces the same marginal distribution as the true parameter can serve as a population target for consistent estimation and valid inference. These results provide a theoretical basis for jointly estimating model parameters and latent distributions as both the sample size and p p p grow.
Our work opens up several possible future research directions. First, an important next step is to develop joint estimators of the model parameters and latent distribution that converge to population solutions inducing the true marginal distribution, as required by Corollaries 1 and 2. Flexible learned priors, including mixture priors [17 , 59 ] and normalizing flows [45 ] , offer promising constructions. Second, our proof techniques may provide a starting point for studying asymptotic identifiability under more general nonlinear decoders and more complex latent structures. Third, from a practical perspective, it would be valuable to develop computable diagnostics that quantify the identification ambiguity at a given finite p p p , which could guide applied researchers in deciding how many observed variables are needed for the estimated parameters to be reliably interpreted.
Preliminaries
We introduce the following additional notation. The unit sphere in R d \mathbb{R}^{d} R d is S d − 1 = { u ∈ R d : ∥ u ∥ = 1 } \mathbb{S}^{d-1} = \{u \in\mathbb{R}^{d} : \lVert u\rVert= 1\} S d − 1 = { u ∈ R d : ∥ u ∥ = 1 } , and e i ( n ) e_{i}^{(n)} e i ( n ) denotes the i i i th standard basis vector in n n n -dimensional Euclidean space, with the superscript ( n ) (n) ( n ) omitted when the dimension is clear from the context. We write N 0 = { 0 , 1 , 2 , … } \mathbb{N}_{0} = \{0,1,2,\ldots\} N 0 = { 0 , 1 , 2 , … } . For a matrix M ∈ R n × m M \in\mathbb{R}^{n \times m} M ∈ R n × m , we write ∥ M ∥ F = ( ∑ i , j M i j 2 ) 1 / 2 \lVert M\rVert_{\mathrm{F}} = \left(\sum_{i,j} M_{ij}^{2}\right)^{1/2} ∥ M ∥ F = ( ∑ i , j M ij 2 ) 1/2 for the Frobenius norm. For index sets I ⊆ [ n ] \mathcal{I} \subseteq[n] I ⊆ [ n ] , J ⊆ [ m ] \mathcal{J} \subseteq[m] J ⊆ [ m ] , write ( M ) I , J = ( M i j ) i ∈ I , j ∈ J (M)_{\mathcal{I},\mathcal{J}} = (M_{ij})_{i\in\mathcal{I},j\in\mathcal{J}} ( M ) I , J = ( M ij ) i ∈ I , j ∈ J . When I = [ n ] \mathcal{I} = [n] I = [ n ] , we write ( M ) , J (M)_{,\mathcal{J}} ( M ) , J for short and ( M ) I , (M)_{\mathcal{I},} ( M ) I , when J = [ m ] \mathcal{J} = [m] J = [ m ] .
For a function h h h , ∥ h ∥ ∞ \lVert h\rVert_{\infty} ∥ h ∥ ∞ denotes its supremum norm. Recall that C ∞ C^{\infty} C ∞ consists of functions with continuous derivatives of all orders. We further let C c ∞ C_{c}^{\infty} C c ∞ denote those with compact support. A function on an open set is analytic if, near every point, it agrees with a convergent power series about that point. A Lebesgue zero set, or null set, is a set of Lebesgue measure zero. An integral with respect to a latent variable in R d \mathbb{R}^{d} R d is written as ∫ ⋅ d z \int\cdot dz ∫ ⋅ d z unless a domain is specified. For a random vector or scalar W W W and 1 ≤ q < ∞ 1 \leq q < \infty 1 ≤ q < ∞ , write
∥ W ∥ q = ( E ∥ W ∥ q ) 1 / q , \lVert W\rVert_{q} = \left(\mathbb{E}\lVert W\rVert^{q}\right)^{1/q}, ∥ W ∥ q = ( E ∥ W ∥ q ) 1/ q ,
where the norm inside the expectation is the Euclidean norm for vectors and absolute value for scalars. We distinguish this L q L_{q} L q norm from the unsubscripted notation ∥ ⋅ ∥ \lVert\cdot\rVert ∥ ⋅ ∥ , which denotes the Euclidean norm for vectors and the operator norm for matrices.
We also introduce basic polynomial notation. For a commutative coefficient ring A \mathcal{A} A , the polynomial ring A [ x 1 , … , x r ] \mathcal{A}[x_{1},\ldots,x_{r}] A [ x 1 , … , x r ] consists of finite sums P ( x ) = ∑ a c a x a P(x) = \sum_{a} c_{a}x^{a} P ( x ) = ∑ a c a x a , where c a ∈ A c_{a} \in\mathcal{A} c a ∈ A , a ∈ N 0 r a \in\mathbb{N}_{0}^{r} a ∈ N 0 r , and x a = ∏ j = 1 r x j a j x^{a} = \prod_{j=1}^{r}x_{j}^{a_{j}} x a = ∏ j = 1 r x j a j is a monomial of total degree ∣ a ∣ \lvert a\rvert ∣ a ∣ . Equality of polynomials means equality of the coefficients of every monomial. An outer polynomial R ( x 1 , … , x r ) R(x_{1},\ldots,x_{r}) R ( x 1 , … , x r ) is a polynomial whose arguments may themselves be polynomials. For instance, if F 1 ( u ) , … , F r ( u ) F_{1}(u),\ldots,F_{r}(u) F 1 ( u ) , … , F r ( u ) are polynomials, then R ( F 1 ( u ) , … , F r ( u ) ) R(F_{1}(u),\ldots,F_{r}(u)) R ( F 1 ( u ) , … , F r ( u )) denotes an outer polynomial of u u u obtained by replacing x i x_{i} x i by F i ( u ) F_{i}(u) F i ( u ) . For example, R ( x 1 , x 2 ) = x 1 2 + x 2 R(x_{1},x_{2}) = x_{1}^{2} + x_{2} R ( x 1 , x 2 ) = x 1 2 + x 2 gives R ( F 1 ( u ) , F 2 ( u ) ) = F 1 ( u ) 2 + F 2 ( u ) R(F_{1}(u),F_{2}(u)) = F_{1}(u)^{2} + F_{2}(u) R ( F 1 ( u ) , F 2 ( u )) = F 1 ( u ) 2 + F 2 ( u ) .
Proofs of Main Results
Proof of Theorem 1
Given ( θ , G ) (\theta,G) ( θ , G ) , we construct perturbation of G G G to offset any sufficiently small perturbation of θ \theta θ and preserves the moments such that E G ~ [ Z a ] = E G [ Z a ] \mathbb{E}_{\widetilde{G}}[Z^{a}] = \mathbb{E}_{G}[Z^{a}] E G [ Z a ] = E G [ Z a ] for all ∣ a ∣ ≤ 2 s \lvert a\rvert\leq2s ∣ a ∣ ≤ 2 s . The construction builds on the idea of moment-matching structure of calibration weighting and empirical likelihood [15 , 46 ] .
Let q S ( θ , z ) = ∏ j ∈ S f ( β j + α j T z ) q_{S}(\theta,z) = \prod_{j\in S} f(\beta_{j} + \alpha_{j}^{\mathsf{T}}z) q S ( θ , z ) = ∏ j ∈ S f ( β j + α j T z ) for any S ⊆ [ p ] S \subseteq[p] S ⊆ [ p ] . When S = ∅ S = \varnothing S = ∅ , let q ∅ ( θ , z ) = 1 q_{\varnothing}(\theta,z) = 1 q ∅ ( θ , z ) = 1 . By conditional independence, E θ , G ∏ j ∈ S X j = ∫ q S ( θ , z ) d G ( z ) \mathbb{E}_{\theta,G}\prod_{j\in S} X_{j} = \int q_{S}(\theta,z)\,dG(z) E θ , G ∏ j ∈ S X j = ∫ q S ( θ , z ) d G ( z ) . For x ∈ { 0 , 1 } p x \in\{0,1\}^{p} x ∈ { 0 , 1 } p and S x = { j : x j = 1 } S_{x} = \{j : x_{j} = 1\} S x = { j : x j = 1 } , inclusion— exclusion gives
P θ , G ( X = x ) = ∑ T ⊆ [ p ] ∖ S x ( − 1 ) ∣ T ∣ ∫ q S x ∪ T ( θ , z ) d G ( z ) . (10) P_{\theta,G}(X=x)=\sum_{T\subseteq[p]\setminus S_x}(-1)^{|T|}\int q_{S_x\cup T}(\theta,z)\,dG(z). \tag*{(10)} P θ , G ( X = x ) = T ⊆ [ p ] ∖ S x ∑ ( − 1 ) ∣ T ∣ ∫ q S x ∪ T ( θ , z ) d G ( z ) . (10)
Thus, it suffices to construct ( θ , G ) (\theta,G) ( θ , G ) and ( θ ~ , G ~ ) (\tilde{\theta},\tilde{G}) ( θ ~ , G ~ ) such that ∫ q S ( θ , z ) d G ( z ) = ∫ q S ( θ ~ , z ) d G ~ ( z ) \int q_S(\theta,z)\,dG(z)=\int q_S(\tilde{\theta},z)\,d\tilde{G}(z) ∫ q S ( θ , z ) d G ( z ) = ∫ q S ( θ ~ , z ) d G ~ ( z ) for every S ≠ ∅ S\ne\varnothing S = ∅ . The proof proceeds in two steps. First, we define a Lebesgue-null exceptional set N p \mathcal{N}_p N p with the required property. Second, for any θ ∉ N p \theta\notin\mathcal{N}_p θ ∈ / N p , any sufficiently close θ ~ \tilde{\theta} θ ~ , and any admissible distribution G G G with density g g g , we construct a perturbed density of the form g ( z ) ( 1 + correction ) g(z)(1+\text{correction}) g ( z ) ( 1 + correction ) so that the distribution G ~ \tilde{G} G ~ satisfies ∫ q S ( θ , z ) d G ( z ) = ∫ q S ( θ ~ , z ) d G ~ ( z ) \int q_S(\theta,z)\,dG(z)=\int q_S(\tilde{\theta},z)\,d\tilde{G}(z) ∫ q S ( θ , z ) d G ( z ) = ∫ q S ( θ ~ , z ) d G ~ ( z ) , as well as other requirements in Theorem 1.
(1) Defining the exceptional set. We first construct a Lebesgue null set N p \mathcal{N}_p N p such that, for any θ ∉ N p \theta\notin\mathcal{N}_p θ ∈ / N p , q S ( θ , z ) q_S(\theta,z) q S ( θ , z ) are linearly independent as functions of z z z , namely, ∑ S ⊆ [ p ] c S q S ( θ , z ) = 0 \sum_{S\subseteq[p]}c_Sq_S(\theta,z)=0 ∑ S ⊆ [ p ] c S q S ( θ , z ) = 0 for every z ∈ R d z\in\mathbb{R}^d z ∈ R d implies c S = 0 c_S=0 c S = 0 for every S ⊆ [ p ] S\subseteq[p] S ⊆ [ p ] . It then suffices to find one θ ∗ \theta^* θ ∗ for which Q S ( θ ∗ , r ) Q_S(\theta^*,r) Q S ( θ ∗ , r ) , defined as
Q S ( θ , r ) = ∫ S d − 1 q S ( θ , r u ) d σ d − 1 ( u ) , S ⊆ [ p ] Q_S(\theta,r)=\int_{\mathbb{S}^{d-1}}q_S(\theta,ru)\,d\sigma_{d-1}(u),\qquad S\subseteq[p] Q S ( θ , r ) = ∫ S d − 1 q S ( θ , r u ) d σ d − 1 ( u ) , S ⊆ [ p ]
are linear independent as functions of r > 0 r>0 r > 0 . Here, σ d − 1 \sigma_{d-1} σ d − 1 is uniform probability measure on S d − 1 \mathbb{S}^{d-1} S d − 1 (σ 0 = ( δ − 1 + δ 1 ) / 2 \sigma_0=(\delta_{-1}+\delta_1)/2 σ 0 = ( δ − 1 + δ 1 ) /2 ). To see this, suppose such θ ∗ \theta^* θ ∗ exists. Then the q S ( θ ∗ , ⋅ ) q_S(\theta^*,\cdot) q S ( θ ∗ , ⋅ ) are linearly independent, since any relation among them remains a relation after spherical averaging to Q S ( θ , r ) Q_S(\theta,r) Q S ( θ , r ) , so we can find z 1 , … , z 2 p ∈ R d z_1,\ldots,z_{2^p}\in\mathbb{R}^d z 1 , … , z 2 p ∈ R d such that det ( q S ( θ ∗ , z i ) ) 1 ≤ i ≤ 2 p , S ⊆ [ p ] ≠ 0 \det\left(q_S(\theta^*,z_i)\right)_{1\le i\le2^p,S\subseteq[p]}\ne0 det ( q S ( θ ∗ , z i ) ) 1 ≤ i ≤ 2 p , S ⊆ [ p ] = 0 . Next, define
N p = { θ : det ( q S ( θ , z i ) ) 1 ≤ i ≤ 2 p , S ⊆ [ p ] = 0 } . (11) \mathcal{N}_p=\left\{\theta:\det\left(q_S(\theta,z_i)\right)_{1\le i\le2^p,S\subseteq[p]}=0\right\}. \tag*{(11)} N p = { θ : det ( q S ( θ , z i ) ) 1 ≤ i ≤ 2 p , S ⊆ [ p ] = 0 } . (11)
Since f f f is analytic, det ( q S ( θ , z i ) ) 1 ≤ i ≤ 2 p , S ⊆ [ p ] \det\left(q_S(\theta,z_i)\right)_{1\le i\le2^p,S\subseteq[p]} det ( q S ( θ , z i ) ) 1 ≤ i ≤ 2 p , S ⊆ [ p ] is also analytic in θ \theta θ . By the definition of θ ∗ \theta^* θ ∗ and { z i } i ∈ [ 2 p ] \{z_i\}_{i\in[2^p]} { z i } i ∈ [ 2 p ] , the determinant is nonzero at θ ∗ \theta^* θ ∗ , and therefore its zero set N p \mathcal{N}_p N p has Lebesgue measure zero. For θ ∉ N p \theta\notin\mathcal{N}_p θ ∈ / N p , matrix ( q S ( θ , z i ) ) i ∈ [ 2 p ] , S ⊆ [ p ] \left(q_S(\theta,z_i)\right)_{i\in[2^p],S\subseteq[p]} ( q S ( θ , z i ) ) i ∈ [ 2 p ] , S ⊆ [ p ] is nonsingular, so the q S ( θ , ⋅ ) q_S(\theta,\cdot) q S ( θ , ⋅ ) are linearly independent.
To show that Q S ( θ ∗ , r ) Q_S(\theta^*,r) Q S ( θ ∗ , r ) are linear independent, we design θ ∗ \theta^* θ ∗ inductively. Suppose A ∗ A^* A ∗ in θ ∗ \theta^* θ ∗ has only the first column nonzero, and let
v S ( u ) = ∏ j ∈ S f ( β j ∗ + α j 1 ∗ u ) and v ˉ S ( r ) = ∫ S d − 1 v S ( r u 1 ) d σ d − 1 ( u ) . v_S(u)=\prod_{j\in S}f(\beta_j^*+\alpha_{j1}^*u)\quad\text{and}\quad\bar{v}_S(r)=\int_{\mathbb{S}^{d-1}}v_S(ru_1)\,d\sigma_{d-1}(u). v S ( u ) = j ∈ S ∏ f ( β j ∗ + α j 1 ∗ u ) and v ˉ S ( r ) = ∫ S d − 1 v S ( r u 1 ) d σ d − 1 ( u ) .
Here, u 1 u_1 u 1 is the first component in u u u . We then choose β j ∗ ∈ R \beta_j^*\in\mathbb{R} β j ∗ ∈ R and α j 1 ∗ ∈ R \alpha_{j1}^*\in\mathbb{R} α j 1 ∗ ∈ R inductively so that ( v ˉ S ) S ⊆ [ k ] (\bar{v}_S)_{S\subseteq[k]} ( v ˉ S ) S ⊆ [ k ] is linearly independent.
When k = 0 k=0 k = 0 , v ˉ ∅ = 1 \bar{v}_{\varnothing}=1 v ˉ ∅ = 1 , and there is nothing to prove. Suppose the assertion holds for k ≥ 0 k\ge0 k ≥ 0 . Let n = 2 k n=2^k n = 2 k . Then ( v S ) S ⊆ [ k ] (v_S)_{S\subseteq[k]} ( v S ) S ⊆ [ k ] is also linearly independent due to the spherical averaging. By the induction hypothesis there are radii 0 < r 1 < ⋯ < r n 0<r_1<\cdots<r_n 0 < r 1 < ⋯ < r n such that ( v ˉ S ( r i ) ) i ≤ n , S ⊆ [ k ] (\bar{v}_S(r_i))_{i\le n,S\subseteq[k]} ( v ˉ S ( r i ) ) i ≤ n , S ⊆ [ k ] is nonsingular, and we fix t 0 > r n t_0>r_n t 0 > r n as a threshold for new functions when moving from k k k to k + 1 k+1 k + 1 :
v ˉ S ∪ { k + 1 } ( r ) = ∫ S d − 1 v S ( r u 1 ) f ℓ ( r u 1 ) d σ d − 1 ( u ) , S ⊆ [ k ] , \bar{v}_{S\cup\{k+1\}}(r)=\int_{\mathbb{S}^{d-1}}v_S(ru_1)f_\ell(ru_1)\,d\sigma_{d-1}(u),\qquad S\subseteq[k], v ˉ S ∪ { k + 1 } ( r ) = ∫ S d − 1 v S ( r u 1 ) f ℓ ( r u 1 ) d σ d − 1 ( u ) , S ⊆ [ k ] ,
For r ≠ t 0 r\ne t_0 r = t 0 , let T d v S ( r ) : = ∫ r u 1 > t 0 v S ( r u 1 ) d σ d − 1 ( u ) T_dv_S(r):=\int_{ru_1>t_0}v_S(ru_1)\,d\sigma_{d-1}(u) T d v S ( r ) := ∫ r u 1 > t 0 v S ( r u 1 ) d σ d − 1 ( u ) , which equals 0 0 0 when 0 < r < t 0 0<r<t_0 0 < r < t 0 . For j = k + 1 j=k+1 j = k + 1 and any ℓ > 0 \ell>0 ℓ > 0 , define α j 1 ∗ = ℓ \alpha^*_{j1}=\ell α j 1 ∗ = ℓ , β j ∗ = − ℓ t 0 \beta^*_j=-\ell t_0 β j ∗ = − ℓ t 0 , and f ℓ ( t ) = f { ℓ ( t − t 0 ) } f_\ell(t)=f\{\ell(t-t_0)\} f ℓ ( t ) = f { ℓ ( t − t 0 )} . Then as ℓ → ∞ \ell\to\infty ℓ → ∞ , f ℓ ( t ) → 1 { t > t 0 } f_\ell(t)\to\mathbf{1}_{\{t>t_0\}} f ℓ ( t ) → 1 { t > t 0 } for t ≠ t 0 t\ne t_0 t = t 0 , and thus since { u : r u 1 = t 0 } \{u:ru_1=t_0\} { u : r u 1 = t 0 } is null under measure σ d − 1 \sigma_{d-1} σ d − 1 for r ≠ t 0 r\ne t_0 r = t 0 , we have convergence
∫ S d − 1 v S ( r u 1 ) f ℓ ( r u 1 ) d σ d − 1 ( u ) → T d v S ( r ) . \int_{\mathbb{S}^{d-1}}v_S(ru_1)f_\ell(ru_1)\,d\sigma_{d-1}(u)\to T_dv_S(r). ∫ S d − 1 v S ( r u 1 ) f ℓ ( r u 1 ) d σ d − 1 ( u ) → T d v S ( r ) .
We then show that ( T d v S ) S ⊆ [ k ] (T_dv_S)_{S\subseteq[k]} ( T d v S ) S ⊆ [ k ] are linearly independent on ( t 0 , ∞ ) (t_0,\infty) ( t 0 , ∞ ) . Let v ( u ) = ∑ S ⊆ [ k ] c S v S ( u ) v(u)=\sum_{S\subseteq[k]}c_Sv_S(u) v ( u ) = ∑ S ⊆ [ k ] c S v S ( u ) with c ≠ 0 c\ne0 c = 0 . Since the v S ( u ) v_S(u) v S ( u ) are linearly independent and analytic, v ( u ) ≢ 0 v(u)\not\equiv0 v ( u ) ≡ 0 has isolated zeros. Thus for some δ > 0 \delta>0 δ > 0 , v ( u ) v(u) v ( u ) has no zero within ( t 0 , t 0 + δ ) (t_0,t_0+\delta) ( t 0 , t 0 + δ ) , namely, it has a constant sign on that interval. For r ∈ ( t 0 , t 0 + δ ) r\in(t_0,t_0+\delta) r ∈ ( t 0 , t 0 + δ ) , the integrand of T d v ( r ) = ∫ u 1 > t 0 / r v ( r u 1 ) d σ d − 1 ( u ) T_dv(r)=\int_{u_1>t_0/r}v(ru_1)\,d\sigma_{d-1}(u) T d v ( r ) = ∫ u 1 > t 0 / r v ( r u 1 ) d σ d − 1 ( u ) has fixed nonzero sign throughout the integration domain u > t 0 / r u>t_0/r u > t 0 / r , within which we have t 0 < r u 1 ≤ r < t 0 + δ t_0<ru_1\le r<t_0+\delta t 0 < r u 1 ≤ r < t 0 + δ as u 1 ≤ 1 u_1\le1 u 1 ≤ 1 when u ∈ S d − 1 u\in\mathbb{S}^{d-1} u ∈ S d − 1 . This domain has positive measure under σ d − 1 \sigma_{d-1} σ d − 1 -measure, so T d v ( r ) ≠ 0 T_dv(r)\ne0 T d v ( r ) = 0 .
With r 1 < ⋯ < r n < t 0 r_1<\cdots<r_n<t_0 r 1 < ⋯ < r n < t 0 as above, we choose t 0 < r n + 1 < ⋯ < r 2 n t_0<r_{n+1}<\cdots<r_{2n} t 0 < r n + 1 < ⋯ < r 2 n such that ( T d v S ( r n + i ) ) i ≤ n , S ⊆ [ k ] (T_dv_S(r_{n+i}))_{i\le n,S\subseteq[k]} ( T d v S ( r n + i ) ) i ≤ n , S ⊆ [ k ] is nonsingular. Note that ( v ˉ S ( r i ) ) 1 ≤ i ≤ 2 k + 1 , S ⊆ [ k + 1 ] (\bar{v}_S(r_i))_{1\le i\le2^{k+1},S\subseteq[k+1]} ( v ˉ S ( r i ) ) 1 ≤ i ≤ 2 k + 1 , S ⊆ [ k + 1 ] converges to
( ( v ˉ S ( r i ) ) i ≤ n , S ⊆ [ k ] 0 ( v ˉ S ( r n + i ) ) i ≤ n , S ⊆ [ k ] ( T d v S ( r n + i ) ) i ≤ n , S ⊆ [ k ] ) , \begin{pmatrix}
(\bar{v}_S(r_i))_{i\le n,S\subseteq[k]} & 0\\
(\bar{v}_S(r_{n+i}))_{i\le n,S\subseteq[k]} & (T_dv_S(r_{n+i}))_{i\le n,S\subseteq[k]}
\end{pmatrix}, ( ( v ˉ S ( r i ) ) i ≤ n , S ⊆ [ k ] ( v ˉ S ( r n + i ) ) i ≤ n , S ⊆ [ k ] 0 ( T d v S ( r n + i ) ) i ≤ n , S ⊆ [ k ] ) ,
whose determinant is nonzero by the linear independence. Consequently, for sufficiently large ℓ \ell ℓ , ( v ˉ S ) S ⊆ [ k + 1 ] (\bar{v}_S)_{S\subseteq[k+1]} ( v ˉ S ) S ⊆ [ k + 1 ] is linearly independent on ( 0 , ∞ ) (0,\infty) ( 0 , ∞ ) , which completes the induction.
Therefore, there is some θ ∗ \theta^* θ ∗ such that Q S ( θ ∗ , r ) Q_S(\theta^*,r) Q S ( θ ∗ , r ) are linearly independent, and by the preceding discussion, N p \mathcal{N}_p N p constructed in (B.2) is the required Lebesgue null set.
(2) Constructing the perturbed density. Now fix θ ∉ N p \theta\notin\mathcal{N}_p θ ∈ / N p and any G G G satisfying (3) . Let g ∈ C ∞ g\in C^\infty g ∈ C ∞ be the density of G G G . Fix s ≥ 1 s\ge1 s ≥ 1 under which E G ∥ Z ∥ 2 s < ∞ \mathbb{E}_G\lVert Z\rVert^{2s}<\infty E G ∥ Z ∥ 2 s < ∞ . The case s = 1 s=1 s = 1 proves the first assertion of the theorem. By definition, we know
{ z a : ∣ a ∣ ≤ 2 s } ∪ { q S ( θ , z ) : ∅ ≠ S ⊆ [ p ] } (12) \{z^a:|a|\le2s\}\cup\{q_S(\theta,z):\varnothing\ne S\subseteq[p]\} \tag*{(12)} { z a : ∣ a ∣ ≤ 2 s } ∪ { q S ( θ , z ) : ∅ = S ⊆ [ p ]} (12)
is linear independent, as the first part concerns polynomials of z z z and the second part concerns bounded functions of linear combination in q S ( θ , z ) q_S(\theta,z) q S ( θ , z ) . For simplicity, we write 2 [ p ] = { S 0 = ∅ , S 1 , … , S n − 1 } 2^{[p]}=\{S_0=\varnothing,S_1,\ldots,S_{n-1}\} 2 [ p ] = { S 0 = ∅ , S 1 , … , S n − 1 } where we slightly abuse the notation to denote n = 2 p n=2^p n = 2 p . For any w ∈ C ∞ ( R d ) w\in C^\infty(\mathbb{R}^d) w ∈ C ∞ ( R d ) , define the linear map
w ( ⋅ ) ↦ [ { ∫ z a w ( z ) g ( z ) d z } ∣ a ∣ ≤ 2 s , { ∫ q S i ( θ , z ) w ( z ) g ( z ) d z } i = 1 n − 1 ] . w(\cdot)\mapsto\left[\left\{\int z^aw(z)g(z)\,dz\right\}_{|a|\le2s},\left\{\int q_{S_i}(\theta,z)w(z)g(z)\,dz\right\}_{i=1}^{n-1}\right]. w ( ⋅ ) ↦ [ { ∫ z a w ( z ) g ( z ) d z } ∣ a ∣ ≤ 2 s , { ∫ q S i ( θ , z ) w ( z ) g ( z ) d z } i = 1 n − 1 ] .
This map is surjective. Otherwise, since the map is linear, the range of the map has dimension strictly less than ∣ { a : ∣ a ∣ ≤ 2 s } ∣ + n − 1 |\{a:|a|\le2s\}|+n-1 ∣ { a : ∣ a ∣ ≤ 2 s } ∣ + n − 1 . Then we can find some nonzero linear combination of the functions in (B.3) that integrates to zero for every w ( ⋅ ) g ( ⋅ ) w(\cdot)g(\cdot) w ( ⋅ ) g ( ⋅ ) . Since the density g ( ⋅ ) g(\cdot) g ( ⋅ ) is positive, the combination would be zero, leading to the contradiction. The surjectivity then implies that there exists some w l ∈ C c ∞ ( R d ) w_l\in C^\infty_c(\mathbb{R}^d) w l ∈ C c ∞ ( R d ) mapped to ( 0 , e l ) (0,e_l) ( 0 , e l ) for any l ∈ [ n − 1 ] l\in[n-1] l ∈ [ n − 1 ] , which thus satisfies
∫ z a w l ( z ) g ( z ) d z = 0 and ∫ q S i ( θ , z ) w l ( z ) g ( z ) d z = δ i l , for each i ∈ [ n − 1 ] (13) \int z^a w_l(z)g(z)\,dz=0 \quad\text{and}\quad\int q_{S_i}(\theta,z)w_l(z)g(z)\,dz=\delta_{il},\ \text{for each }i\in[n-1] \tag*{(13)} ∫ z a w l ( z ) g ( z ) d z = 0 and ∫ q S i ( θ , z ) w l ( z ) g ( z ) d z = δ i l , for each i ∈ [ n − 1 ] (13)
The collection { w l } l ∈ [ n − 1 ] \{w_l\}_{l\in[n-1]} { w l } l ∈ [ n − 1 ] serves the role of bump function w ( z ) w(z) w ( z ) in Example 1. For θ ~ \tilde{\theta} θ ~ , let
M ( θ ~ ) = { M i l ( θ ~ ) } [ n − 1 ] , [ n − 1 ] and b ( θ ~ ) = { b i ( θ ~ ) } [ n − 1 ] M(\tilde{\theta})=\{M_{il}(\tilde{\theta})\}_{[n-1],[n-1]}\quad\text{and}\quad b(\tilde{\theta})=\{b_i(\tilde{\theta})\}_{[n-1]} M ( θ ~ ) = { M i l ( θ ~ ) } [ n − 1 ] , [ n − 1 ] and b ( θ ~ ) = { b i ( θ ~ ) } [ n − 1 ]
where for each i , l ∈ [ n − 1 ] i,l\in[n-1] i , l ∈ [ n − 1 ] , M i l ( θ ~ ) = ∫ q S i ( θ ~ , z ) w l ( z ) g ( z ) d z M_{il}(\tilde{\theta})=\int q_{S_i}(\tilde{\theta},z)w_l(z)g(z)\,dz M i l ( θ ~ ) = ∫ q S i ( θ ~ , z ) w l ( z ) g ( z ) d z and b i ( θ ~ ) = ∫ { q S i ( θ , z ) − q S i ( θ ~ , z ) } g ( z ) d z b_i(\tilde{\theta})=\int\{q_{S_i}(\theta,z)-q_{S_i}(\tilde{\theta},z)\}g(z)\,dz b i ( θ ~ ) = ∫ { q S i ( θ , z ) − q S i ( θ ~ , z )} g ( z ) d z . By definition, as θ ~ → θ \tilde{\theta}\to\theta θ ~ → θ , M ( θ ~ ) → M ( θ ) = I n − 1 M(\tilde{\theta})\to M(\theta)=I_{n-1} M ( θ ~ ) → M ( θ ) = I n − 1 and b i ( θ ~ ) → b i ( θ ) = 0 b_i(\tilde{\theta})\to b_i(\theta)=0 b i ( θ ~ ) → b i ( θ ) = 0 . Then on a neighbourhood U \mathcal{U} U of θ \theta θ , we let
g ~ ( z ) = g ( z ) [ 1 + ∑ l = 1 n − 1 { [ M ( θ ~ ) ] − 1 b ( θ ~ ) } l w l ( z ) ] . (14) \tilde{g}(z)=g(z)\left[1+\sum_{l=1}^{n-1}\left\{\left[M(\tilde{\theta})\right]^{-1}b(\tilde{\theta})\right\}_l w_l(z)\right]. \tag*{(14)} g ~ ( z ) = g ( z ) [ 1 + l = 1 ∑ n − 1 { [ M ( θ ~ ) ] − 1 b ( θ ~ ) } l w l ( z ) ] . (14)
Because { M ( θ ~ ) } − 1 b ( θ ~ ) → 0 \{M(\tilde{\theta})\}^{-1}b(\tilde{\theta})\to0 { M ( θ ~ ) } − 1 b ( θ ~ ) → 0 as θ ~ → θ \tilde{\theta}\to\theta θ ~ → θ , we can shrink U \mathcal{U} U so that ∥ ∑ l = 1 n − 1 [ { M ( θ ~ ) } − 1 b ( θ ~ ) ] l w l ∥ ∞ < 1 / 2 \left\|\sum_{l=1}^{n-1}[\{M(\tilde{\theta})\}^{-1}b(\tilde{\theta})]_l w_l\right\|_\infty<1/2 ∑ l = 1 n − 1 [{ M ( θ ~ ) } − 1 b ( θ ~ ) ] l w l ∞ < 1/2 . This neighbourhood U \mathcal{U} U depends on the chosen s s s , but we suppress this dependence in the notation for simplicity. Then g ~ > 0 \tilde{g}>0 g ~ > 0 and g ~ ∈ C ∞ ( R d ) \tilde{g}\in C^\infty(\mathbb{R}^d) g ~ ∈ C ∞ ( R d ) . By (B.4), we know that ∫ z a g ~ ( z ) d z = ∫ z a g ( z ) d z \int z^a\tilde{g}(z)\,dz=\int z^ag(z)\,dz ∫ z a g ~ ( z ) d z = ∫ z a g ( z ) d z and similar to (5) , that
∫ q S i ( θ ~ , z ) g ~ ( z ) d z = ∫ q S i ( θ , z ) g ( z ) d z . \int q_{S_i}(\tilde{\theta},z)\tilde{g}(z)\,dz=\int q_{S_i}(\theta,z)g(z)\,dz. ∫ q S i ( θ ~ , z ) g ~ ( z ) d z = ∫ q S i ( θ , z ) g ( z ) d z .
Thus the pairs ( θ , G ) (\theta,G) ( θ , G ) and ( θ ~ , G ~ ) (\tilde{\theta},\tilde{G}) ( θ ~ , G ~ ) satisfies the requirement in Theorem 1. Moreover, taking θ ~ = ( A , β + u e 1 ) ∈ U \tilde{\theta}=(A,\beta+ue_1)\in\mathcal{U} θ ~ = ( A , β + u e 1 ) ∈ U with any u ≠ 0 u\ne0 u = 0 gives D p ( θ , θ ~ ) = ∣ u ∣ > 0 D_p(\theta,\tilde{\theta})=|u|>0 D p ( θ , θ ~ ) = ∣ u ∣ > 0 , proving non-identifiability in the sense of Definition 1.
Proof of Proposition 1
Following the definitions in Appendix B.1, let n = 2 p n=2^p n = 2 p and use the dual functions w l w_l w l from (B.4) at the fixed parameter θ \theta θ . Define θ ε = ( [ A , ε v ] , β ) \theta_\varepsilon=([A,\varepsilon v],\beta) θ ε = ([ A , ε v ] , β ) and for i ∈ [ n − 1 ] i\in[n-1] i ∈ [ n − 1 ]
b i ′ ( θ ε ) = E [ ∏ j ∈ S i f ( β j + α j Z + ε v j U ) ] − E [ q S i ( θ , Z ) ] , b_i^{\prime}(\theta_\varepsilon)=\mathbb{E}\left[\prod_{j\in S_i}f(\beta_j+\alpha_jZ+\varepsilon v_jU)\right]-\mathbb{E}[q_{S_i}(\theta,Z)], b i ′ ( θ ε ) = E j ∈ S i ∏ f ( β j + α j Z + ε v j U ) − E [ q S i ( θ , Z )] ,
for Z ∼ G Z\sim G Z ∼ G and U ∼ H U\sim H U ∼ H . Similar to (B.5), let
g ~ ε ( z ) = g ( z ) [ 1 + ∑ l = 1 n − 1 b l ′ ( θ ε ) w l ( z ) ] . \tilde{g}_\varepsilon(z)=g(z)\left[1+\sum_{l=1}^{n-1}b_l^{\prime}(\theta_\varepsilon)w_l(z)\right]. g ~ ε ( z ) = g ( z ) [ 1 + l = 1 ∑ n − 1 b l ′ ( θ ε ) w l ( z ) ] .
Similar to the proof of Theorem 1, one can check that, for sufficiently small ∣ ε ∣ |\varepsilon| ∣ ε ∣ , g ~ ε ( z ) \widetilde{g}_{\varepsilon}(z) g ε ( z ) is a positive smooth density preserving all moments through order 2 s 2s 2 s . For every i i i , it holds that
∫ q S i ( θ , z ) g ~ ε ( z ) d z = E [ q S i ( θ , Z ) ] + b i † ( θ ε ) = E [ ∏ j ∈ S i f ( β j + α j T Z + ε v j U ) ] . \int q_{S_i}(\theta,z)\widetilde{g}_{\varepsilon}(z)\,dz=\mathbb{E}[q_{S_i}(\theta,Z)]+b_i^{\dagger}(\theta_{\varepsilon})=\mathbb{E}\left[\prod_{j\in S_i}f(\beta_j+\alpha_j^{\mathsf{T}}Z+\varepsilon v_jU)\right]. ∫ q S i ( θ , z ) g ε ( z ) d z = E [ q S i ( θ , Z )] + b i † ( θ ε ) = E j ∈ S i ∏ f ( β j + α j T Z + ε v j U ) .
We therefore know P ( [ A , ε v ] , β ) , G ⊗ H = P ( A , β ) , G ~ ε P_{([A,\varepsilon v],\beta),G\otimes H}=P_{(A,\beta),\widetilde{G}_{\varepsilon}} P ([ A , ε v ] , β ) , G ⊗ H = P ( A , β ) , G ε following a similar argument in the inclusion–exclusion argument for (B.1). By the definition of G G G and H H H , we know G ⊗ H G\otimes H G ⊗ H has mean zero and covariance I d + 1 I_{d+1} I d + 1 . The other requirements in Proposition 1 can be similarly verified.
Proof of Theorem 2
Step 0. Preliminaries. To prove Theorem 2, we use a conditioning argument to compare θ \theta θ and θ ~ \widetilde{\theta} θ through high-order expansions of conditional moments. Establishing the superpolynomial rate requires careful control of the expansion terms and remainders as the expansion order increases. To organize these calculations, we introduce a symbolic calculus. We first collect the technical ingredients needed for the proof.
Notation. The entire proof concerns two representations ( θ , G ) (\theta,G) ( θ , G ) and ( θ ~ , G ~ ) (\widetilde{\theta},\widetilde{G}) ( θ , G ) in M p \mathcal{M}_p M p with P θ , G = P θ ~ , G ~ P_{\theta,G}=P_{\widetilde{\theta},\widetilde{G}} P θ , G = P θ , G . For notational simplicity, we index them by r = 0 , 1 r=0,1 r = 0 , 1 ; refer them as model zero, model one; and write α 0 j = α j \alpha_{0j}=\alpha_j α 0 j = α j , α 1 j = α ~ j \alpha_{1j}=\widetilde{\alpha}_j α 1 j = α j , β 0 j = β j \beta_{0j}=\beta_j β 0 j = β j , and β 1 j = β ~ j \beta_{1j}=\widetilde{\beta}_j β 1 j = β j . The rows in A r A_r A r are α r j T \alpha_{rj}^{\mathsf{T}} α r j T and Z r ∼ G r Z_r\sim G_r Z r ∼ G r . All expectations of functions of the responses alone are under the common marginal distribution P θ , G P_{\theta,G} P θ , G , and we use E r \mathbb{E}_r E r to distinguish the two latent representations. With H = 2 K H=2K H = 2 K , we will frequently use the following relationships obtained from Assumption 1(ii):
∣ f ′ ′ ( t ) f ′ ( t ) ∣ ≤ H , f ′ ( t ) ≥ c e − H ∣ t ∣ , and e − H ∣ u ∣ ≤ f ′ ( t + u ) f ′ ( t ) ≤ e H ∣ u ∣ . (15) \left|\frac{f^{\prime\prime}(t)}{f^{\prime}(t)}\right|\le H,\qquad f^{\prime}(t)\ge ce^{-H|t|},\qquad\text{and}\qquad e^{-H|u|}\le\frac{f^{\prime}(t+u)}{f^{\prime}(t)}\le e^{H|u|}. \tag*{(15)} f ′ ( t ) f ′′ ( t ) ≤ H , f ′ ( t ) ≥ c e − H ∣ t ∣ , and e − H ∣ u ∣ ≤ f ′ ( t ) f ′ ( t + u ) ≤ e H ∣ u ∣ . (15)
Write D p = D p ( θ , θ ~ ) D_p=D_p(\theta,\widetilde{\theta}) D p = D p ( θ , θ ) for short, when the argument is clear from the context.
Moment approximation. We now introduce the conditioning trick. Throughout, N ≥ 4 N\ge4 N ≥ 4 denotes an integer approximation order. We select a subset S \mathcal{S} S of size k = O ( N ) k=O(N) k = O ( N ) and condition on the retained responses Y = X S c Y=X_{\mathcal{S}^c} Y = X S c . The set S \mathcal{S} S contains a target set T \mathcal{T} T and a reference set I \mathcal{I} I . The target set contains two elements { i , j } \{i,j\} { i , j } as each time we compare β 0 i , β 1 i \beta_{0i},\beta_{1i} β 0 i , β 1 i and products α 0 i T α 0 j , α 1 i T α 1 j \alpha_{0i}^{\mathsf{T}}\alpha_{0j},\alpha_{1i}^{\mathsf{T}}\alpha_{1j} α 0 i T α 0 j , α 1 i T α 1 j . The reference set contains observed variables whose corresponding parameters are uniformly well-behaved in one representation, and are thereby used for robustly recovering conditional moments. At the same time, S \mathcal{S} S must occupy only a small proportion of all observed variables, so that the rest responses remain informative about Z Z Z and the conditional uncertainty is small. The following lemma guides the selection of these reference groups.
Lemma B.1. Under Assumption 2, there are constants c 0 , c 1 > 0 c_0,c_1>0 c 0 , c 1 > 0 and integer p 0 p_0 p 0 such that for any p ≥ p 0 p\ge p_0 p ≥ p 0 and any J ⊆ [ p ] \mathcal{J}\subseteq[p] J ⊆ [ p ] , with ∣ J ∣ ≤ c 0 p |\mathcal{J}|\le c_0p ∣ J ∣ ≤ c 0 p , we have
∑ j ∈ [ p ] ∖ J α r j α r j T ⪰ ( κ p / 2 ) I d , \sum_{j\in[p]\setminus\mathcal{J}}\alpha_{rj}\alpha_{rj}^{\mathsf{T}}\succeq(\kappa p/2)I_d, j ∈ [ p ] ∖ J ∑ α r j α r j T ⪰ ( κ p /2 ) I d , for r ∈ { 0 , 1 } r\in\{0,1\} r ∈ { 0 , 1 } . Moreover, given any integer N N N and target set T ⊆ [ p ] \mathcal{T}\subseteq[p] T ⊆ [ p ] with ∣ T ∣ ≤ 2 |\mathcal{T}|\le2 ∣ T ∣ ≤ 2 and N d + 2 ≤ c 0 p Nd+2\le c_0p N d + 2 ≤ c 0 p , one can choose disjoint reference sets I 1 , … , I N \mathcal{I}_1,\ldots,\mathcal{I}_N I 1 , … , I N , each of size d d d and excluding T \mathcal{T} T , such that their row matrices B s : = ( A 0 ) I s B_s:=(A_0)_{\mathcal{I}_s} B s := ( A 0 ) I s , under model zero satisfy σ min ( B s ) ≥ c 1 \sigma_{\min}(B_s)\ge c_1 σ m i n ( B s ) ≥ c 1 . The constants are uniform over the two representations and the admissible class.
Proof. See Appendix C.1.
For a fixed target set T = { i , j } \mathcal{T}=\{i,j\} T = { i , j } , choose the reference groups from Lemma B.1 and set I = ⋃ s = 1 N I s \mathcal{I}=\bigcup_{s=1}^{N}\mathcal{I}_s I = ⋃ s = 1 N I s , S = T ∪ I \mathcal{S}=\mathcal{T}\cup\mathcal{I} S = T ∪ I , and Y = X S c Y=X_{\mathcal{S}^{c}} Y = X S c . The same conditioning set is used in both representations. In the scalar illustration below, write S = S \mathcal{S}=S S = S .
Example 5. To better motivate the conditioning trick with moment approximation, we use the case d = 1 d=1 d = 1 as a toy example. For arbitrary i , j ∈ [ p ] i,j\in[p] i , j ∈ [ p ] , we seek to bound β j − β ~ j \beta_j-\widetilde{\beta}_j β j − β j and α i α j − α ~ i α ~ j \alpha_i\alpha_j-\widetilde{\alpha}_i\widetilde{\alpha}_j α i α j − α i α j , and we have selected i , j i,j i , j in S S S . Conditioning on Y = X S c Y=X_{S^c} Y = X S c , we decompose the latent predictor, for i ∈ S i\in S i ∈ S , as
η i = α i Z + β i = T i + δ i + α i V , \eta_i=\alpha_i Z+\beta_i=T_i+\delta_i+\alpha_i V, η i = α i Z + β i = T i + δ i + α i V ,
where
T i = f − 1 E ( X i ∣ Y ) , δ i = α i E ( Z ∣ Y ) + β i − T i , V = Z − E ( Z ∣ Y ) . (16) T_i=f^{-1}\mathbb{E}(X_i\mid Y),\qquad\delta_i=\alpha_i\mathbb{E}(Z\mid Y)+\beta_i-T_i,\qquad V=Z-\mathbb{E}(Z\mid Y). \tag*{(16)} T i = f − 1 E ( X i ∣ Y ) , δ i = α i E ( Z ∣ Y ) + β i − T i , V = Z − E ( Z ∣ Y ) . (16)
This decomposition splits η i \eta_i η i into two parts: α i V \alpha_iV α i V contains the variation in Z Z Z that cannot be predicted by Y Y Y ; and the predictable part α i E ( Z ∣ Y ) + β i \alpha_i\mathbb{E}(Z\mid Y)+\beta_i α i E ( Z ∣ Y ) + β i . When f ( ⋅ ) f(\cdot) f ( ⋅ ) is linear, α i E ( Z ∣ Y ) + β i = T i \alpha_i\mathbb{E}(Z\mid Y)+\beta_i=T_i α i E ( Z ∣ Y ) + β i = T i , and δ i \delta_i δ i serves as correction arising from the nonlinearity of f ( ⋅ ) f(\cdot) f ( ⋅ ) .
As discussed in Remark 2, our primary goal is to approximate δ i \delta_i δ i and the conditional moments M k = E ( V k ∣ Y ) M_k=\mathbb{E}(V^k\mid Y) M k = E ( V k ∣ Y ) for k ≥ 2 k\ge2 k ≥ 2 . Lemmas B.2 and B.3 below shows that V = O L q ( p − 1 / 2 ) V=O_{L^q}(p^{-1/2}) V = O L q ( p − 1/2 ) and δ i = O L q ( p − 1 ) \delta_i=O_{L^q}(p^{-1}) δ i = O L q ( p − 1 ) and M k = O L q ( p − k / 2 ) M_k=O_{L^q}(p^{-k/2}) M k = O L q ( p − k /2 ) for q ≥ 1 q\ge1 q ≥ 1 and k ≥ 2 k\ge2 k ≥ 2 , where O L q O_{L^q} O L q denotes an order bound in the L q L^q L q norm. These rates therefore give only O ( p − 1 ) O(p^{-1}) O ( p − 1 ) bounds on the target parameter differences. Achieving the superpolynomial bound, however, requires much more accurate common approximations to the correction terms. We first construct these under θ \theta θ .
To approximate δ i \delta_i δ i , rewrite the identity E { f ( T i + α i V + δ i ) ∣ Y } = f ( T i ) \mathbb{E}\{f(T_i+\alpha_iV+\delta_i)\mid Y\}=f(T_i) E { f ( T i + α i V + δ i ) ∣ Y } = f ( T i ) as
E [ g T i ( α i V + δ i ) ∣ Y ] = 0 , (17) \mathbb{E}[g_{T_i}(\alpha_iV+\delta_i)\mid Y]=0, \tag*{(17)} E [ g T i ( α i V + δ i ) ∣ Y ] = 0 , (17)
where g t ( u ) = { f ( t + u ) − f ( t ) } / f ′ ( t ) g_t(u)=\{f(t+u)-f(t)\}/f'(t) g t ( u ) = { f ( t + u ) − f ( t )} / f ′ ( t ) . The denominator normalizes the coefficient of the linear term to one. Writing c k i = f ( k ) ( T i ) / k ! f ′ ( T i ) c_{ki}=f^{(k)}(T_i)/k!f'(T_i) c k i = f ( k ) ( T i ) / k ! f ′ ( T i ) , from Taylor expansion of (B.8), we can successively compute approximation to δ i \delta_i δ i with M k M_k M k . For instance, one can expand through fourth order to get
δ i = − c 2 i α i 2 M 2 − c 3 i α i 3 M 3 − c 4 i α i 4 M 4 + ( 3 c 2 i c 3 i − c 2 i 3 ) α i 4 M 2 2 + O L q ( p − 5 / 2 ) . (18) \delta_i=-c_{2i}\alpha_i^2M_2-c_{3i}\alpha_i^3M_3-c_{4i}\alpha_i^4M_4+(3c_{2i}c_{3i}-c_{2i}^3)\alpha_i^4M_2^2+O_{L_q}(p^{-5/2}). \tag*{(18)} δ i = − c 2 i α i 2 M 2 − c 3 i α i 3 M 3 − c 4 i α i 4 M 4 + ( 3 c 2 i c 3 i − c 2 i 3 ) α i 4 M 2 2 + O L q ( p − 5/2 ) . (18)
This shall be implemented in detail in Step 2 below. Next, we use conditional moments of distinct excluded responses following the moment method [2 ] to approximate M k M_k M k . Choose four reference coordinates in S ∖ { i , j } S\setminus\{i,j\} S ∖ { i , j } and relabel them 1 , 2 , 3 , 4 1,2,3,4 1 , 2 , 3 , 4 without loss of generality. Define, for r ∈ [ 4 ] , r\in[4], r ∈ [ 4 ] ,
ζ r = X r − f ( T r ) f ′ ( T r ) , such that E ( ζ r ∣ Z , Y ) = g T r ( α r V + δ r ) . \zeta_r=\frac{X_r-f(T_r)}{f'(T_r)},\qquad\text{such that}\qquad\mathbb{E}(\zeta_r\mid Z,Y)=g_{T_r}(\alpha_rV+\delta_r). ζ r = f ′ ( T r ) X r − f ( T r ) , such that E ( ζ r ∣ Z , Y ) = g T r ( α r V + δ r ) .
Then each O k = ( ∏ r = 1 k α r ) − 1 E ( ∏ r = 1 k ζ r ∣ Y ) O_k=(\prod_{r=1}^k\alpha_r)^{-1}\mathbb{E}(\prod_{r=1}^k\zeta_r\mid Y) O k = ( ∏ r = 1 k α r ) − 1 E ( ∏ r = 1 k ζ r ∣ Y ) has leading term is M k M_k M k for k = 2 , 3 , 4 k=2,3,4 k = 2 , 3 , 4 . Specifically, expanding through fourth order and letting a r = c 2 r α r a_r=c_{2r}\alpha_r a r = c 2 r α r and b r = c 3 r α r 2 b_r=c_{3r}\alpha_r^2 b r = c 3 r α r 2 , we obtain
O 2 = M 2 + ( a 1 + a 2 ) M 3 + ( b 1 + b 2 + a 1 a 2 ) M 4 − ( 2 a 1 2 + 2 a 2 2 + a 1 a 2 ) M 2 2 + O L q ( p − 5 / 2 ) , O 3 = M 3 + ( a 1 + a 2 + a 3 ) ( M 4 − M 2 2 ) + O L q ( p − 5 / 2 ) , O 4 = M 4 + O L q ( p − 5 / 2 ) , and O 2 2 = M 2 2 + O L q ( p − 5 / 2 ) . \begin{aligned}
O_2&=M_2+(a_1+a_2)M_3+(b_1+b_2+a_1a_2)M_4-(2a_1^2+2a_2^2+a_1a_2)M_2^2+O_{L_q}(p^{-5/2}),\\
O_3&=M_3+(a_1+a_2+a_3)(M_4-M_2^2)+O_{L_q}(p^{-5/2}),\\
O_4&=M_4+O_{L_q}(p^{-5/2}),\qquad\text{and}\qquad O_2^2=M_2^2+O_{L_q}(p^{-5/2}).
\end{aligned} O 2 O 3 O 4 = M 2 + ( a 1 + a 2 ) M 3 + ( b 1 + b 2 + a 1 a 2 ) M 4 − ( 2 a 1 2 + 2 a 2 2 + a 1 a 2 ) M 2 2 + O L q ( p − 5/2 ) , = M 3 + ( a 1 + a 2 + a 3 ) ( M 4 − M 2 2 ) + O L q ( p − 5/2 ) , = M 4 + O L q ( p − 5/2 ) , and O 2 2 = M 2 2 + O L q ( p − 5/2 ) .
To eliminate the higher order terms contained in O 2 , O 3 O_2,O_3 O 2 , O 3 , we successively define
M ^ 4 = O 4 M ^ 3 = O 3 − ( a 1 + a 2 + a 3 ) ( O 4 − O 2 2 ) , M ^ 2 = O 2 − ( a 1 + a 2 ) M ^ 3 − ( b 1 + b 2 + a 1 a 2 ) O 4 + ( 2 a 1 2 + 2 a 2 2 + a 1 a 2 ) O 2 2 . (19) \begin{aligned}
\widehat{M}_4&=O_4\qquad\widehat{M}_3=O_3-(a_1+a_2+a_3)(O_4-O_2^2),\\
\widehat{M}_2&=O_2-(a_1+a_2)\widehat{M}_3-(b_1+b_2+a_1a_2)O_4+(2a_1^2+2a_2^2+a_1a_2)O_2^2. \tag*{(19)}
\end{aligned} M 4 M 2 = O 4 M 3 = O 3 − ( a 1 + a 2 + a 3 ) ( O 4 − O 2 2 ) , = O 2 − ( a 1 + a 2 ) M 3 − ( b 1 + b 2 + a 1 a 2 ) O 4 + ( 2 a 1 2 + 2 a 2 2 + a 1 a 2 ) O 2 2 . (19)
Then M ^ k = M k + O L q ( p − 5 / 2 ) \widehat{M}_k=M_k+O_{L_q}(p^{-5/2}) M k = M k + O L q ( p − 5/2 ) for k = 2 , 3 k=2,3 k = 2 , 3 . Substituting M ^ 2 , M ^ 3 , M ^ 4 \widehat{M}_2,\widehat{M}_3,\widehat{M}_4 M 2 , M 3 , M 4 , and O 2 2 O_2^2 O 2 2 for M 2 M_2 M 2 , M 3 M_3 M 3 , M 4 M_4 M 4 , and M 2 2 M_2^2 M 2 2 in (B.9) gives an approximation to δ i \delta_i δ i with error O L q ( p − 5 / 2 ) O_{L_q}(p^{-5/2}) O L q ( p − 5/2 ) .
A general version of approximating the conditional latent moments to order N N N shall be implemented in detail in Step 3 below. This calculation illustrates how successive moment recovery sharpens the initial approximations at a fixed order. At higher orders, recursive substitution generates products such as M 2 M 3 M_2M_3 M 2 M 3 and M 2 3 M_2^3 M 2 3 , making coefficient growth and accumulated errors increasingly difficult to track directly. To obtain bounds valid under growing N N N , p p p and multidimensional d d d , we introduce the symbolic calculus.
Symbolic calculus. We introduce a symbolic calculus to represent combinations of moment products as formal expressions and provide algebraic rules to carry out high-order expansions concisely. Similar to (B.7), let V = Z − E ( Z ∣ Y ) V=Z-\mathbb{E}(Z\mid Y) V = Z − E ( Z ∣ Y ) and δ i = E ( α i T Z + β i ∣ Y ) − T i \delta_i=\mathbb{E}(\alpha_i^{\mathsf{T}}Z+\beta_i\mid Y)-T_i δ i = E ( α i T Z + β i ∣ Y ) − T i from (B.7). We suppress r r r when working within one representation. For a multi-index γ = ( γ 1 , … , γ d ) ∈ N 0 d \gamma=(\gamma_1,\ldots,\gamma_d)\in\mathbb{N}_0^d γ = ( γ 1 , … , γ d ) ∈ N 0 d , write ∣ γ ∣ = ∑ ℓ = 1 d γ ℓ |\gamma|=\sum_{\ell=1}^d\gamma_\ell ∣ γ ∣ = ∑ ℓ = 1 d γ ℓ and V γ = ∏ ℓ = 1 d V ℓ γ ℓ V^\gamma=\prod_{\ell=1}^dV_\ell^{\gamma_\ell} V γ = ∏ ℓ = 1 d V ℓ γ ℓ . Since E ( V ∣ Y ) = 0 \mathbb{E}(V\mid Y)=0 E ( V ∣ Y ) = 0 , the conditional moments of interest begin at ∣ γ ∣ = 2 |\gamma|=2 ∣ γ ∣ = 2 . We symbolize δ i \delta_i δ i , V V V , and E [ V γ ∣ Y ] \mathbb{E}[V^\gamma\mid Y] E [ V γ ∣ Y ] , and δ i \delta_i δ i , which are treated as indeterminates to form a commutative polynomial ring, while T i T_i T i are treated as coefficient. The following two lemmas bound these quantities, which hold under either representation.
Lemma B.2. Under Assumptions 1 and 2, there is a constant C > 0 C > 0 C > 0 such that, for p ≥ p 0 p \ge p_0 p ≥ p 0 , k ≥ 2 k \ge2 k ≥ 2 and q ≥ 1 q \ge1 q ≥ 1 ,
E ∥ V ∥ k ≤ e C k q 2 p − k / 2 and ∥ E ( ∥ V ∥ k ∣ Y ) ∥ q ≤ e C q k q 2 p − k / 2 . (20) \mathbb{E}\lVert V\rVert^k \le e^{Ckq^2}p^{-k/2} \quad\text{and} \quad\left\lVert\mathbb{E}(\lVert V\rVert^k \mid Y)\right\rVert_q \le e^{Cqkq^2}p^{-k/2}. \tag*{(20)} E ∥ V ∥ k ≤ e C k q 2 p − k /2 and E (∥ V ∥ k ∣ Y ) q ≤ e C q k q 2 p − k /2 . (20) The bounds hold uniformly over S S S with ∣ S ∣ ≤ c 0 p |S| \le c_0p ∣ S ∣ ≤ c 0 p and both representations, where c 0 , p 0 c_0,p_0 c 0 , p 0 are as in Lemma Lemma B.1 .
Proof. See Appendix § 9 .§ 2 .
Lemma B.3. Under the setup of Lemma Lemma B.2 , for every q ≥ 1 q \ge1 q ≥ 1 and t ≥ 0 t \ge0 t ≥ 0 , uniformly over i ∈ S i \in S i ∈ S ,
∥ δ i ∥ q ≤ e C q p − 1 , \lVert\delta_i\rVert_q \le e^{Cq}p^{-1}, ∥ δ i ∥ q ≤ e C q p − 1 , E e t ∥ V ∥ + E e t ∣ T i ∣ + E e t ∣ δ i ∣ ≤ e C ( 1 + t 2 ) . \mathbb{E}e^{t\lVert V\rVert}+\mathbb{E}e^{t|T_i|}+\mathbb{E}e^{t|\delta_i|}\le e^{C(1+t^2)}. E e t ∥ V ∥ + E e t ∣ T i ∣ + E e t ∣ δ i ∣ ≤ e C ( 1 + t 2 ) .
Proof. See Appendix § 9 .§ 3 .
We now define the symbolic calculus. First introduce a formal vector v = ( v 1 , … , v d ) v=(v_1,\ldots,v_d) v = ( v 1 , … , v d ) to represent V V V , a formal scalar s s s to represent δ i \delta_i δ i , and moment symbols m γ m_\gamma m γ to represent E ( V γ ∣ Y ) \mathbb{E}(V^\gamma\mid Y) E ( V γ ∣ Y ) for ∣ γ ∣ ≥ 2 |\gamma|\ge2 ∣ γ ∣ ≥ 2 , respectively. The commutative polynomial ring considered is then generated by the indeterminates v 1 , … , v d v_1,\ldots,v_d v 1 , … , v d , s s s , and m γ m_\gamma m γ over coefficients consisting of real-valued functions of the predictors T : = ( T i ) i ∈ S \mathbf{T}:=(T_i)_{i\in S} T := ( T i ) i ∈ S , often denoted by c ( T i ) c(T_i) c ( T i ) . The coefficients are also allowed to depend on model parameters.
For any u u u representing such as symbol, we denote ev ( u ) \operatorname{ev}(u) ev ( u ) the evaluated value (at the observables) of this quantity when substituting v = V v=V v = V , s = δ i s=\delta_i s = δ i , and m γ = E ( V γ ∣ Y ) m_\gamma=\mathbb{E}(V^\gamma\mid Y) m γ = E ( V γ ∣ Y ) into a formal expression. ev ( u ) \operatorname{ev}(u) ev ( u ) is thus a random quantity and u u u is a deterministic symbol. The preceding lemmas give, for q ≥ 1 q\ge1 q ≥ 1 , ℓ ∈ [ d ] \ell\in[d] ℓ ∈ [ d ] , i ∈ S i\in S i ∈ S , and ∣ γ ∣ ≥ 2 |\gamma|\ge2 ∣ γ ∣ ≥ 2 ,
∥ ev ( v ℓ ) ∥ q ≤ C q p − 1 / 2 , ∥ ev ( s ) ∥ q ≤ C q p − 1 , and ∥ ev ( m γ ) ∥ q ≤ e C q ∣ γ ∣ 2 p − ∣ γ ∣ / 2 . (21) \lVert\operatorname{ev}(v_\ell)\rVert_q \le C_qp^{-1/2}, \qquad\lVert\operatorname{ev}(s)\rVert_q \le C_qp^{-1}, \qquad\text{and} \qquad\lVert\operatorname{ev}(m_\gamma)\rVert_q \le e^{Cq|\gamma|^2}p^{-|\gamma|/2}. \tag*{(21)} ∥ ev ( v ℓ ) ∥ q ≤ C q p − 1/2 , ∥ ev ( s ) ∥ q ≤ C q p − 1 , and ∥ ev ( m γ ) ∥ q ≤ e C q ∣ γ ∣ 2 p − ∣ γ ∣/2 . (21)
We use symbolic calculus to organize the expansions and track their coefficients and weights before replacing the corresponding random quantities with the formal symbols. Treating these symbols as algebraically independent provides the convenience of doing high-order expansion on the conditional moments without requiring the random quantities behind the symbols to be probabilistically independent.
According to their scaling with p − 1 / 2 p^{-1/2} p − 1/2 , we assign weight via wt ( ⋅ ) \operatorname{wt}(\cdot) wt ( ⋅ ) as
wt ( v ℓ ) = 1 , wt ( s ) = 2 , and wt ( m γ ) = ∣ γ ∣ . \operatorname{wt}(v_\ell)=1, \qquad\operatorname{wt}(s)=2, \qquad\text{and} \qquad\operatorname{wt}(m_\gamma)=|\gamma|. wt ( v ℓ ) = 1 , wt ( s ) = 2 , and wt ( m γ ) = ∣ γ ∣.
These weights tracks powers of p − 1 / 2 p^{-1/2} p − 1/2 , which are added under multiplication of these quantities. They are different from the degree of the polynomial. For example, m 2 e 1 2 m_{2e_1}^{2} m 2 e 1 2 ( E ( V 1 2 ∣ Y ) ) 2 (\mathbb{E}(V_1^2 \mid Y))^2 ( E ( V 1 2 ∣ Y ) ) 2 and m 4 e 1 m_{4e_1} m 4 e 1 ( E ( V 1 4 ∣ Y ) ) (\mathbb{E}(V_1^4 \mid Y)) ( E ( V 1 4 ∣ Y )) both have weight 4.
We regard v v v , s s s , m γ m_\gamma m γ as algebraically independent variables and work in the polynomial ring they generate, with coefficients depending on the common T i T_i T i and fixed α 0 i \alpha_{0i} α 0 i or α 1 i \alpha_{1i} α 1 i . Addition and subtraction combine coefficients of identical monomials, while multiplication expands products and adds their weights. Under d = 1 d=1 d = 1 for example, the moment symbols can be written as m 2 , m 3 , m 4 , … m_2,m_3,m_4,\ldots m 2 , m 3 , m 4 , … For a fixed coefficient ξ \xi ξ , one can perform the following calculations
( m 2 + ξ m 3 ) + ξ m 3 = m 2 + 2 ξ m 3 ; (m_2+\xi m_3)+\xi m_3=m_2+2\xi m_3; ( m 2 + ξ m 3 ) + ξ m 3 = m 2 + 2 ξ m 3 ;
( m 2 + ξ m 3 ) − ξ m 3 = m 2 ; (m_2+\xi m_3)-\xi m_3=m_2; ( m 2 + ξ m 3 ) − ξ m 3 = m 2 ;
( m 2 + ξ m 3 ) m 2 + m 4 = m 2 2 + ξ m 2 m 3 + m 4 . (m_2+\xi m_3)m_2+m_4=m_2^2+\xi m_2m_3+m_4. ( m 2 + ξ m 3 ) m 2 + m 4 = m 2 2 + ξ m 2 m 3 + m 4 .
The distinct monomials m 2 2 m_2^2 m 2 2 and m 4 m_4 m 4 remain separate.
Next, we define formal expectation E f \mathbb{E}_{\mathrm{f}} E f as the conditional expectation given Y Y Y under the symbolic form. It treats s s s and the moment symbols as coefficients in polynomials, satisfying
E f ( 1 ) = 1 , E f ( v ℓ ) = 0 , E f ( v γ ) = m γ , and E f ( s a m η v γ ) = s a m η m γ . \mathbb{E}_{\mathrm{f}}(1)=1,\qquad\mathbb{E}_{\mathrm{f}}(v^\ell)=0,\qquad\mathbb{E}_{\mathrm{f}}(v^\gamma)=m_\gamma,\qquad\text{and}\qquad\mathbb{E}_{\mathrm{f}}(s^a m^\eta v^\gamma)=s^a m^\eta m_\gamma. E f ( 1 ) = 1 , E f ( v ℓ ) = 0 , E f ( v γ ) = m γ , and E f ( s a m η v γ ) = s a m η m γ .
This defines a linear projection on the polynomial ring under symbolic calculation, as E f [ E f ( ⋅ ) ] = E f ( ⋅ ) \mathbb{E}_{\mathrm{f}}[\mathbb{E}_{\mathrm{f}}(\cdot)]=\mathbb{E}_{\mathrm{f}}(\cdot) E f [ E f ( ⋅ )] = E f ( ⋅ ) . Here, m η = ∏ ∣ ρ ∣ ≥ 2 m ρ m^\eta=\prod_{|\rho|\geq2}m^\rho m η = ∏ ∣ ρ ∣ ≥ 2 m ρ is some monomial with the same multi-index convention as V γ V^\gamma V γ . For any polynomial P P P , formal expectation and evaluation satisfy
ev { E f ( P ) } = E { ev ( P ) ∣ Y } . \operatorname{ev}\{\mathbb{E}_{\mathrm{f}}(P)\}=\mathbb{E}\{\operatorname{ev}(P)\mid Y\}. ev { E f ( P )} = E { ev ( P ) ∣ Y } .
Finally, for a polynomial P P P , denote by [ P ] N [P]_N [ P ] N the truncation that preserves only monomials of weight at most N N N . For example, [ ( v 1 + s ) 2 ] 3 = v 1 2 + 2 v 1 s [(v_1+s)^2]_3=v_1^2+2v_1s [( v 1 + s ) 2 ] 3 = v 1 2 + 2 v 1 s as s 2 s^2 s 2 has weight four. Till now, we have defined a polynomial ring in conditional-moment symbols equipped with multiplication, truncation, and formal expectation.
Remark 4. The use of formal symbols and an evaluation map is related to the algebraic viewpoint of classical umbral calculus [51 ] . Organizing and truncating terms by their asymptotic order also parallels cumulant-based higher-order expansions [60 ] . Here the weights are used to reflect scaling with p − 1 / 2 p^{-1/2} p − 1/2 , allowing conditional latent moments, correction terms, and their products to be handled within one polynomial calculus.
Outline. To bound D p ( θ 0 , θ 1 ) D_p(\theta_0,\theta_1) D p ( θ 0 , θ 1 ) , we refine
β i = E [ T i + δ i ] ≈ E [ T i ] , α i ⊤ α j = Cov ( T i + δ i , T j + δ j ) + E { ( α i ⊤ V ) ( α j ⊤ V ) } ≈ Cov ( T i , T j ) . (22) \begin{aligned}
\beta_i &= \mathbb{E}[T_i+\delta_i]\approx\mathbb{E}[T_i],\\
\alpha_i^\top\alpha_j &= \operatorname{Cov}(T_i+\delta_i,T_j+\delta_j)+\mathbb{E}\{(\alpha_i^\top V)(\alpha_j^\top V)\}\approx\operatorname{Cov}(T_i,T_j). \tag*{(22)}
\end{aligned} β i α i ⊤ α j = E [ T i + δ i ] ≈ E [ T i ] , = Cov ( T i + δ i , T j + δ j ) + E {( α i ⊤ V ) ( α j ⊤ V )} ≈ Cov ( T i , T j ) . (22)
using high-order polynomial approximations to the bias and covariance corrections. The argument has three components: controlling approximation errors under finite products; constructing symbolic expansions of δ i \delta_i δ i and its conditional-mean equation; and expressing the corrections through conditional response moments. We proceed one by one.
Weighted polynomial calculus
For T = ( T i ) i ∈ S \mathbf{T}=(T_i)_{i\in\mathcal{S}} T = ( T i ) i ∈ S and a polynomial P ( m ) = ∑ η a η ( T ) m η P(m)=\sum_\eta a_\eta(\mathbf{T})m^\eta P ( m ) = ∑ η a η ( T ) m η , define
∥ P ( m ) ∥ c o e f f = ∑ η sup T ∣ a η ( T ) ∣ . \lVert P(m)\rVert_{\mathrm{coeff}}=\sum_\eta\sup_{\mathbf{T}}|a_\eta(\mathbf{T})|. ∥ P ( m ) ∥ coeff = η ∑ T sup ∣ a η ( T ) ∣.
Here, a η ( T ) a_\eta(\mathbf{T}) a η ( T ) denotes the coefficient in P ( m ) P(m) P ( m ) associated with monomial m η m^\eta m η , which is itself a function of T i T_i T i s, such as f ′ ′ ( T i ) / f ′ ( T i ) f^{\prime\prime}(T_i)/f^\prime(T_i) f ′′ ( T i ) / f ′ ( T i ) . We use the same definition for polynomials involving v , s v,s v , s . This norm controls the accumulation of coefficients when the approximation order grows. Under this norm, we have for any two polynomials P P P and Q Q Q that ∥ P Q ∥ c o e f f ≤ ∥ P ∥ c o e f f ∥ Q ∥ c o e f f \lVert PQ\rVert_{\mathrm{coeff}}\leq\lVert P\rVert_{\mathrm{coeff}}\lVert Q\rVert_{\mathrm{coeff}} ∥ P Q ∥ coeff ≤ ∥ P ∥ coeff ∥ Q ∥ coeff , ∥ E f ( P ) ∥ c o e f f ≤ ∥ P ∥ c o e f f \lVert\mathbb{E}_{\mathrm{f}}(P)\rVert_{\mathrm{coeff}}\leq\lVert P\rVert_{\mathrm{coeff}} ∥ E f ( P ) ∥ coeff ≤ ∥ P ∥ coeff , and
∥ ev { P ( m ) } ∥ q ≤ ∑ η sup T ∣ a η ( T ) ∣ ∥ ev { m η } ∥ q ≤ ∥ P ∥ c o e f f max η : a η ≠ 0 ∥ ev { m η } ∥ q . \lVert\operatorname{ev}\{P(m)\}\rVert_q\leq\sum_\eta\sup_{\mathbf{T}}|a_\eta(\mathbf{T})|\lVert\operatorname{ev}\{m^\eta\}\rVert_q\leq\lVert P\rVert_{\mathrm{coeff}}\max_{\eta:a_\eta\ne0}\lVert\operatorname{ev}\{m^\eta\}\rVert_q. ∥ ev { P ( m )} ∥ q ≤ η ∑ T sup ∣ a η ( T ) ∣ ∥ ev { m η } ∥ q ≤ ∥ P ∥ coeff η : a η = 0 max ∥ ev { m η } ∥ q .
For the polynomials constructed below, we will show that they satisfy coefficient bounds of the form ∥ P ∥ c o e f f ≤ e C N 2 \lVert P\rVert_{\mathrm{coeff}}\leq e^{CN^2} ∥ P ∥ coeff ≤ e C N 2 , with C C C independent of N N N .
As a concrete example of using the coefficient bound, consider a monomial P = ∏ j = 1 r u j P=\prod_{j=1}^r u_j P = ∏ j = 1 r u j where u j u_j u j represents one of the symbols v ℓ , s , m γ v_\ell,s,m_\gamma v ℓ , s , m γ and has weights η j \eta_j η j , and let η = ∑ j = 1 r η j ≥ 1 \eta=\sum_{j=1}^r\eta_j\geq1 η = ∑ j = 1 r η j ≥ 1 . By (B.12), the L a L^a L a norm of each evaluated u j u_j u j is bounded by e C ( 1 + a ) η j 2 p − η j / 2 e^{C(1+a)\eta_j^2}p^{-\eta_j/2} e C ( 1 + a ) η j 2 p − η j /2 . By Hölder inequality with norm orders q η / η j q\eta/\eta_j q η / η j on each evaluated component of weight η j \eta_j η j , the L q L^q L q norm of this evaluated monomial P P P can be bounded by
∥ ev ( P ) ∥ q ≤ exp ( C ∑ j = 1 r η j 2 + C q η ∑ j = 1 r η j ) p − η / 2 ≤ e C n 2 + C q η 2 p − η / 2 = e C ( 1 + q ) η 2 p − η / 2 . (23) \lVert\operatorname{ev}(P)\rVert_q\leq\exp\left(C\sum_{j=1}^r\eta_j^2+Cq\eta\sum_{j=1}^r\eta_j\right)p^{-\eta/2}\leq e^{Cn^2+Cq\eta^2}p^{-\eta/2}=e^{C(1+q)\eta^2}p^{-\eta/2}. \tag*{(23)} ∥ ev ( P ) ∥ q ≤ exp ( C j = 1 ∑ r η j 2 + C q η j = 1 ∑ r η j ) p − η /2 ≤ e C n 2 + C q η 2 p − η /2 = e C ( 1 + q ) η 2 p − η /2 . (23)
Moreover, by Jensen’s inequality, we know ∥ E [ ev ( P ) ∣ Y ] ∥ q ≤ e C ( 1 + q ) η 2 p − η / 2 \lVert\mathbb{E}[\operatorname{ev}(P)\mid Y]\rVert_q\leq e^{C(1+q)\eta^2}p^{-\eta/2} ∥ E [ ev ( P ) ∣ Y ] ∥ q ≤ e C ( 1 + q ) η 2 p − η /2 . When ∥ P ∥ c o e f f ≤ e C N 2 \lVert P\rVert_{\mathrm{coeff}}\leq e^{CN^2} ∥ P ∥ coeff ≤ e C N 2 for some C C C , and all weights of P P P lie between N + 1 N+1 N + 1 and c N cN c N with fixed c c c , we have
∥ ev ( P ) ∥ q ≤ ∥ P ∥ c o e f f e C ′ ( 1 + q ) N 2 p − ( N + 1 ) / 2 ≤ e C ′ ′ ( 1 + q ) N 2 p − ( N + 1 ) / 2 . (24) \lVert\operatorname{ev}(P)\rVert_q\leq\lVert P\rVert_{\mathrm{coeff}}e^{C'(1+q)N^2}p^{-(N+1)/2}\leq e^{C''(1+q)N^2}p^{-(N+1)/2}. \tag*{(24)} ∥ ev ( P ) ∥ q ≤ ∥ P ∥ coeff e C ′ ( 1 + q ) N 2 p − ( N + 1 ) /2 ≤ e C ′′ ( 1 + q ) N 2 p − ( N + 1 ) /2 . (24)
We next establish that for a sequence of random quantities, if they admits proper polynomial approximation whose coefficient can be controlled in terms of ∥ P ∥ c o e f f \lVert P\rVert_{\mathrm{coeff}} ∥ P ∥ coeff , then the product of these random quantities can be well-approximated by some polynomial as well. In particular, we have
Lemma B.4. Let x 1 , … , x r x_1,\ldots,x_r x 1 , … , x r be random quantities with positive integer weights w 1 , … , w r w_1,\ldots,w_r w 1 , … , w r satisfying ∑ j = 1 r w j ≤ N \sum_{j=1}^{r} w_j \le N ∑ j = 1 r w j ≤ N . Suppose that, for every q ≥ 1 q \ge1 q ≥ 1 , the L q L^q L q norm satisfies ∥ x j ∥ q ≤ e C ( 1 + q ) w j 2 p − w j / 2 \lVert x_j\rVert_q \le e^{C(1+q)w_j^2}p^{-w_j/2} ∥ x j ∥ q ≤ e C ( 1 + q ) w j 2 p − w j /2 , and that each x j x_j x j admits a polynomial approximation F j ( v , s , m ) F_j(v,s,m) F j ( v , s , m ) with weights between w j w_j w j and N N N , such that
∥ x j − ev ( F j ( v , s , m ) ) ∥ q ≤ e C ( 1 + q ) N 2 p − ( N + 1 ) / 2 (25) \lVert x_j-\operatorname{ev}(F_j(v,s,m))\rVert_q \le e^{C(1+q)N^2}p^{-(N+1)/2} \tag*{(25)} ∥ x j − ev ( F j ( v , s , m )) ∥ q ≤ e C ( 1 + q ) N 2 p − ( N + 1 ) /2 (25) for any q ≥ 1 q \ge1 q ≥ 1 and the coefficient of each weight w ∈ { w j , w j + 1 , … , N } w \in\{w_j,w_j+1,\ldots,N\} w ∈ { w j , w j + 1 , … , N } can be controlled by
∥ [ F j ] w − [ F j ] w − 1 ∥ c o e f f ≤ C w , (26) \lVert[F_j]_w-[F_j]_{w-1}\rVert_{\mathrm{coeff}} \le C^w, \tag*{(26)} ∥[ F j ] w − [ F j ] w − 1 ∥ coeff ≤ C w , (26) Then there is a polynomial P r ( v , s , m ) P_r(v,s,m) P r ( v , s , m ) such that, for every q ≥ 1 q \ge1 q ≥ 1 ,
∥ ∏ j = 1 r x j − ev ( P r ( v , s , m ) ) ∥ q ≤ e C ′ ( 1 + q ) N 2 p − ( N + 1 ) / 2 , (27) \left\lVert\prod_{j=1}^{r}x_j-\operatorname{ev}(P_r(v,s,m))\right\rVert_q \le e^{C'(1+q)N^2}p^{-(N+1)/2}, \tag*{(27)} j = 1 ∏ r x j − ev ( P r ( v , s , m )) q ≤ e C ′ ( 1 + q ) N 2 p − ( N + 1 ) /2 , (27) for each q ≥ 1 q \ge1 q ≥ 1 . In particular, P r ( v , s , m ) P_r(v,s,m) P r ( v , s , m ) can be defined inductively by setting P 0 ( v , s , m ) = 1 P_0(v,s,m)=1 P 0 ( v , s , m ) = 1 and P j ( v , s , m ) = [ P j − 1 ( v , s , m ) F j ( v , s , m ) ] N P_j(v,s,m)=[P_{j-1}(v,s,m)F_j(v,s,m)]_N P j ( v , s , m ) = [ P j − 1 ( v , s , m ) F j ( v , s , m ) ] N for j = 1 , … , r j=1,\ldots,r j = 1 , … , r . Moreover, for an outer polynomial R ( x 1 , … , x r ) R(x_1,\ldots,x_r) R ( x 1 , … , x r ) of weight at most N N N with ∥ R ∥ c o e f f ≤ e C N 2 \lVert R\rVert_{\mathrm{coeff}}\le e^{CN^2} ∥ R ∥ coeff ≤ e C N 2 , we have
∥ R ( x 1 , … , x r ) − ev { [ R ( F 1 , … , F r ) ] N } ∥ q ≤ e C ′ ′ ( 1 + q ) N 2 p − ( N + 1 ) / 2 . (28) \lVert R(x_1,\ldots,x_r)-\operatorname{ev}\{[R(F_1,\ldots,F_r)]_N\}\rVert_q \le e^{C''(1+q)N^2}p^{-(N+1)/2}. \tag*{(28)} ∥ R ( x 1 , … , x r ) − ev {[ R ( F 1 , … , F r ) ] N } ∥ q ≤ e C ′′ ( 1 + q ) N 2 p − ( N + 1 ) /2 . (28) Proof. See Appendix C.4.
Step 2. Symbolic expansion of δ i \delta_i δ i We now construct polynomial approximations to δ i \delta_i δ i . Note that the definition of δ i = E ( α i V Z + β i ∣ Y ) − T i \delta_i=\mathbb{E}(\alpha_i^V Z+\beta_i\mid Y)-T_i δ i = E ( α i V Z + β i ∣ Y ) − T i implies
E { f ( T i + α i V V + δ i ) ∣ Y } = E ( X i ∣ Y ) = f ( T i ) . \mathbb{E}\{f(T_i+\alpha_i^V V+\delta_i)\mid Y\}=\mathbb{E}(X_i\mid Y)=f(T_i). E { f ( T i + α i V V + δ i ) ∣ Y } = E ( X i ∣ Y ) = f ( T i ) .
Define g t ( u ) = { f ( t + u ) − f ( t ) } / f ′ ( t ) g_t(u)=\{f(t+u)-f(t)\}/f'(t) g t ( u ) = { f ( t + u ) − f ( t )} / f ′ ( t ) and we have the conditional mean equation
E { g T i ( α i V V + δ i ) ∣ Y } = 0. (29) \mathbb{E}\{g_{T_i}(\alpha_i^V V+\delta_i)\mid Y\}=0. \tag*{(29)} E { g T i ( α i V V + δ i ) ∣ Y } = 0. (29)
The denominator f ′ ( T i ) f'(T_i) f ′ ( T i ) makes the first-order coefficient equal to one. Under nonlinear f f f , directly expanding (B.20) to solve δ i \delta_i δ i requires repeatedly substituting the expansion of δ i \delta_i δ i into itself, generating increasingly complicated products of conditional moments.
We instead replace V V V , δ i \delta_i δ i and conditional moments of V V V by their formal symbols v v v , s s s and m m m , respectively. Specifically, we can symbolize (B.20) into
F i ( s , m ) : = E f g T i ( α i V v + s ) = 0 , wt ( s ) = 2. (30) F_i(s,m):=\mathbb{E}_f g_{T_i}(\alpha_i^V v+s)=0,\qquad\operatorname{wt}(s)=2. \tag*{(30)} F i ( s , m ) := E f g T i ( α i V v + s ) = 0 , wt ( s ) = 2. (30)
The weight constraint wt ( s ) = 2 \operatorname{wt}(s)=2 wt ( s ) = 2 reflects the ∥ δ i ∥ q = O ( p − 1 ) \lVert\delta_i\rVert_q=O(p^{-1}) ∥ δ i ∥ q = O ( p − 1 ) by Lemma B.3 , since each unit of weight corresponds to a factor p − 1 / 2 p^{-1/2} p − 1/2 . Lemma B.5 below gives a unique formal solution s = H i ( m ) s=H_i(m) s = H i ( m ) starting at weight two, whose evaluated truncation approximates δ i \delta_i δ i with L q L^q L q error at most e C q N 2 p − ( N + 1 ) / 2 e^{CqN^2}p^{-(N+1)/2} e C q N 2 p − ( N + 1 ) /2 . This expresses δ i \delta_i δ i in terms of conditional latent moments.
We note that the expansion of F i ( s , m ) \mathcal{F}_i(s,m) F i ( s , m ) consists of the term s s s , which has weight two, and remaining terms of weight at least two. We can therefore determine H i ( m ) H_i(m) H i ( m ) implicitly by matching terms of equal weight. For example, let d = 1 d=1 d = 1 and write c k = c k ( T i ) c_k=c_k(T_i) c k = c k ( T i ) for brevity. Expanding (B.21) through weight four gives
F i ( s , m ) = s + c 2 α i 2 m 2 + c 3 α i 3 m 3 + c 4 α i 4 m 4 + c 2 s 2 + 3 c 3 α i 2 s m 2 + { terms of weight at least five } . (31) \mathcal{F}_i(s,m)=s+c_2\alpha_i^2m_2+c_3\alpha_i^3m_3+c_4\alpha_i^4m_4+c_2s^2+3c_3\alpha_i^2sm_2+\{\text{terms of weight at least five}\}. \tag*{(31)} F i ( s , m ) = s + c 2 α i 2 m 2 + c 3 α i 3 m 3 + c 4 α i 4 m 4 + c 2 s 2 + 3 c 3 α i 2 s m 2 + { terms of weight at least five } . (31)
Writing H i ( m ) = H i , 2 ( m ) + H i , 3 ( m ) + ⋯ H_i(m)=H_{i,2}(m)+H_{i,3}(m)+\cdots H i ( m ) = H i , 2 ( m ) + H i , 3 ( m ) + ⋯ , where H i , r ( m ) H_{i,r}(m) H i , r ( m ) has weight r r r , and matching terms of equal weight in F i ( H i ( m ) , m ) = 0 \mathcal{F}_i(H_i(m),m)=0 F i ( H i ( m ) , m ) = 0 , we obtain H i , 2 ( m ) = − c 2 α i 2 m 2 H_{i,2}(m)=-c_2\alpha_i^2m_2 H i , 2 ( m ) = − c 2 α i 2 m 2 , H i , 3 ( m ) = − c 3 α i 3 m 3 H_{i,3}(m)=-c_3\alpha_i^3m_3 H i , 3 ( m ) = − c 3 α i 3 m 3 , and H i , 4 ( m ) = − c 4 α i 4 m 4 + ( 3 c 2 c 3 − c 2 3 ) α i 4 m 2 2 H_{i,4}(m)=-c_4\alpha_i^4m_4+(3c_2c_3-c_2^3)\alpha_i^4m_2^2 H i , 4 ( m ) = − c 4 α i 4 m 4 + ( 3 c 2 c 3 − c 2 3 ) α i 4 m 2 2 . The same procedure determines all higher-order components, although their expressions become increasingly complicated. Our argument does not give a closed-form expression for the full series. The coefficient and remainder bounds needed for these constructions are collected in the following lemma.
Lemma B.5. Under the current setup, for either representation there is a unique formal series H i ( m ) H_i(m) H i ( m ) starting at weight two such that F i ( H i ( m ) , m ) = 0 \mathcal{F}_i(H_i(m),m)=0 F i ( H i ( m ) , m ) = 0 . Define
G i , N ( v , m ) = [ g T i ( α i ⊤ v + H i ( m ) ) ] N . G_{i,N}(v,m)=[g_{T_i}(\alpha_i^\top v+H_i(m))]_N. G i , N ( v , m ) = [ g T i ( α i ⊤ v + H i ( m )) ] N . The weight-w w w coefficients of H i ( m ) H_i(m) H i ( m ) , for w ≥ 2 w\ge2 w ≥ 2 , and of G i , N ( v , m ) G_{i,N}(v,m) G i , N ( v , m ) , for 1 ≤ w ≤ N 1\le w\le N 1 ≤ w ≤ N , have coefficient norm at most C w C^w C w . The same bounds hold for their first derivatives with respect to each loading coordinate α i ℓ \alpha_{i\ell} α i ℓ . The weight-one term of G i , N ( v , m ) G_{i,N}(v,m) G i , N ( v , m ) is α i ⊤ v \alpha_i^\top v α i ⊤ v . For every q ≥ 1 q\ge1 q ≥ 1 ,
∥ δ i − ev { [ H i ( m ) ] N } ∥ q ≤ e C ( 1 + q ) N 2 p − ( N + 1 ) / 2 , \lVert\delta_i-\operatorname{ev}\{[H_i(m)]_N\}\rVert_q\le e^{C(1+q)N^2}p^{-(N+1)/2}, ∥ δ i − ev {[ H i ( m ) ] N } ∥ q ≤ e C ( 1 + q ) N 2 p − ( N + 1 ) /2 , ∥ g T i ( α i ⊤ V + δ i ) − ev { G i , N ( v , m ) } ∥ q ≤ e C ( 1 + q ) N 2 p − ( N + 1 ) / 2 . \lVert g_{T_i}(\alpha_i^\top V+\delta_i)-\operatorname{ev}\{G_{i,N}(v,m)\}\rVert_q\le e^{C(1+q)N^2}p^{-(N+1)/2}. ∥ g T i ( α i ⊤ V + δ i ) − ev { G i , N ( v , m )} ∥ q ≤ e C ( 1 + q ) N 2 p − ( N + 1 ) /2 . Moreover, ∥ δ i ∥ q ≤ e C ( 1 + q ) p − 1 \lVert\delta_i\rVert_q\le e^{C(1+q)}p^{-1} ∥ δ i ∥ q ≤ e C ( 1 + q ) p − 1 and ∥ g T i ( α i ⊤ V + δ i ) ∥ q ≤ e C ( 1 + q ) p − 1 / 2 \lVert g_{T_i}(\alpha_i^\top V+\delta_i)\rVert_q\le e^{C(1+q)}p^{-1/2} ∥ g T i ( α i ⊤ V + δ i ) ∥ q ≤ e C ( 1 + q ) p − 1/2 .
Proof. See Appendix § 9 .§ 5 .
Lemma B.5 provides the approximations and bounds needed to apply Lemma B.4 to δ i \delta_i δ i and g T i ( α i ⊤ V + δ i ) g_{T_i}(\alpha_i^\top V+\delta_i) g T i ( α i ⊤ V + δ i ) . In the next step, we use these results to construct response-based approximations to ev ( m γ ) \operatorname{ev}(m_\gamma) ev ( m γ ) . Substituting these approximations into [ H i ( m ) ] N [H_i(m)]_N [ H i ( m ) ] N and removing the resulting higher-weight terms yields an accurate approximation to δ i \delta_i δ i .
Step 3. Corrections via conditional response moments.
Recall that our goal is to obtain sharper corrections than (B.13). Both approximation δ i ≈ ev { [ H i ( m ) ] N } \delta_i \approx\operatorname{ev}\{[H_i(m)]_N\} δ i ≈ ev {[ H i ( m ) ] N } established in Lemma B.5 and the conditional covariance E { ( α i T V ) ( α i T V ) } \mathbb{E}\{(\alpha_i^{\mathsf{T}}V)(\alpha_i^{\mathsf{T}}V)\} E {( α i T V ) ( α i T V )} involve conditional latent moments E [ V γ ∣ Y ] = ev ( m γ ) \mathbb{E}[V^\gamma\mid Y]=\operatorname{ev}(m_\gamma) E [ V γ ∣ Y ] = ev ( m γ ) . In this step, we recover these corrections from conditional moments of distinct reference responses from I \mathcal{I} I as given in Lemma B.1. To better motivate the derivation, we first give a cubic calculation as an illustration.
Inspired by the approximations M ^ 2 \widehat{M}_2 M 2 and M ^ 3 \widehat{M}_3 M 3 constructed in (B.10), we aim to construct observable quantities to recover the conditional latent moments through combinations of
ζ h ( s ) = X j h ( s ) − E ( X j h ( s ) ∣ Y ) f ′ ( T j h ( s ) ) , for h ∈ [ d ] , j h ∈ I s , and s ∈ [ N ] . \zeta_h^{(s)}=\frac{X_{j_h^{(s)}}-\mathbb{E}(X_{j_h^{(s)}}\mid Y)}{f'(T_{j_h^{(s)}})},\qquad\text{for }h\in[d],\ j_h\in\mathcal{I}_s,\ \text{and }s\in[N]. ζ h ( s ) = f ′ ( T j h ( s ) ) X j h ( s ) − E ( X j h ( s ) ∣ Y ) , for h ∈ [ d ] , j h ∈ I s , and s ∈ [ N ] .
Here, we denote the selected reference group as I s = { j h ( s ) : h ∈ [ d ] } ⊆ [ p ] \mathcal{I}_s=\{j_h^{(s)}:h\in[d]\}\subseteq[p] I s = { j h ( s ) : h ∈ [ d ]} ⊆ [ p ] . Without loss of generality, assume j 1 ( s ) < j 2 ( s ) < ⋯ < j d ( s ) j_1^{(s)}<j_2^{(s)}<\cdots<j_d^{(s)} j 1 ( s ) < j 2 ( s ) < ⋯ < j d ( s ) for every s ∈ [ N ] s\in[N] s ∈ [ N ] . We aim to construct a linear system from these ζ h ( s ) \zeta_h^{(s)} ζ h ( s ) to form O γ O_\gamma O γ , such that it matches E [ V γ ∣ Y ] \mathbb{E}[V^\gamma\mid Y] E [ V γ ∣ Y ] to the first order.
To achieve this, we work on the symbolic form. First note that under model zero,
E [ ζ h ( s ) ∣ Z 0 , Y ] = g j h ( s ) ( α 0 , j h ( s ) T V 0 + δ 0 , j h ( s ) ) . (32) \mathbb{E}[\zeta_h^{(s)}\mid Z_0,Y]=g_{j_h^{(s)}}\left(\alpha_{0,j_h^{(s)}}^{\mathsf{T}}V_0+\delta_{0,j_h^{(s)}}\right). \tag*{(32)} E [ ζ h ( s ) ∣ Z 0 , Y ] = g j h ( s ) ( α 0 , j h ( s ) T V 0 + δ 0 , j h ( s ) ) . (32)
as f ( T i ) = E [ X i ∣ Y ] f(T_i)=\mathbb{E}[X_i\mid Y] f ( T i ) = E [ X i ∣ Y ] and α i T Z + β i = T i + α i T V + δ i \alpha_i^{\mathsf{T}}Z+\beta_i=T_i+\alpha_i^{\mathsf{T}}V+\delta_i α i T Z + β i = T i + α i T V + δ i for i = j h ( s ) i=j_h^{(s)} i = j h ( s ) . By Lemma B.5, this conditional mean is approximated by ev 0 { G j h ( s ) , N ( v , m ) } \operatorname{ev}_0\{G_{j_h^{(s)},N}(v,m)\} ev 0 { G j h ( s ) , N ( v , m )} , whose leading term is α 0 , j h ( s ) T V 0 \alpha_{0,j_h^{(s)}}^{\mathsf{T}}V_0 α 0 , j h ( s ) T V 0 . Recall that Lemma B.1 ensures that B s : = ( A 0 ) I s B_s:=(A_0)_{\mathcal{I}_s} B s := ( A 0 ) I s , satisfies σ min ( B s ) ≥ c 1 \sigma_{\min}(B_s)\ge c_1 σ m i n ( B s ) ≥ c 1 . The rows of B s B_s B s , ordered as j 1 ( s ) , … , j d ( s ) j_1^{(s)},\ldots,j_d^{(s)} j 1 ( s ) , … , j d ( s ) , give the coefficients of the linear forms α 0 , j h ( s ) T v \alpha_{0,j_h^{(s)}}^{\mathsf{T}}v α 0 , j h ( s ) T v .
But we want to construct v γ v^\gamma v γ from α 0 , j h ( s ) T v \alpha_{0,j_h^{(s)}}^{\mathsf{T}}v α 0 , j h ( s ) T v . We use a tensor-product change of basis Kolda and Bader [33 ] . For any multi-index γ ∈ N 0 d \gamma\in\mathbb{N}_0^d γ ∈ N 0 d with 2 ≤ k = ∣ γ ∣ ≤ N 2\le k=|\gamma|\le N 2 ≤ k = ∣ γ ∣ ≤ N , choose and fix an ordered list ℓ 1 , … , ℓ k ∈ [ d ] \ell_1,\ldots,\ell_k\in[d] ℓ 1 , … , ℓ k ∈ [ d ] such that v γ = ∏ s = 1 k v ℓ s v^\gamma=\prod_{s=1}^k v_{\ell_s} v γ = ∏ s = 1 k v ℓ s . In what follows, we suppress the dependence of these indices on γ \gamma γ when it is clear from the context. With this notation, define b h ( s ) = ( B s − 1 ) ℓ s h b_h^{(s)}=(B_s^{-1})_{\ell_s h} b h ( s ) = ( B s − 1 ) ℓ s h . Then
∑ h = 1 d b h ( s ) α 0 , j h ( s ) T v = e ℓ s T v = v ℓ s \sum_{h=1}^d b_h^{(s)}\alpha_{0,j_h^{(s)}}^{\mathsf{T}}v=e_{\ell_s}^{\mathsf{T}}v=v_{\ell_s} h = 1 ∑ d b h ( s ) α 0 , j h ( s ) T v = e ℓ s T v = v ℓ s
and therefore
∏ s = 1 k { ∑ h = 1 d b h ( s ) α 0 , j h ( s ) T v } = v γ . (33) \prod_{s=1}^k\left\{\sum_{h=1}^d b_h^{(s)}\alpha_{0,j_h^{(s)}}^{\mathsf{T}}v\right\}=v^\gamma. \tag*{(33)} s = 1 ∏ k { h = 1 ∑ d b h ( s ) α 0 , j h ( s ) T v } = v γ . (33)
Thus, we construct
O γ = ∑ h 1 , … , h k = 1 d { ∏ s = 1 k ( B s − 1 ) ℓ s h s } E { ∏ s = 1 k ζ h s ( s ) ∣ Y } , for 2 ≤ ∣ γ ∣ ≤ N (34) O_\gamma=\sum_{h_1,\ldots,h_k=1}^d\left\{\prod_{s=1}^k(B_s^{-1})_{\ell_s h_s}\right\}\mathbb{E}\left\{\prod_{s=1}^k\zeta_{h_s}^{(s)}\mid Y\right\},\qquad\text{for }2\le|\gamma|\le N \tag*{(34)} O γ = h 1 , … , h k = 1 ∑ d { s = 1 ∏ k ( B s − 1 ) ℓ s h s } E { s = 1 ∏ k ζ h s ( s ) ∣ Y } , for 2 ≤ ∣ γ ∣ ≤ N (34)
Here, ℓ s \ell_s ℓ s are implicitly determined by γ \gamma γ . Letting U s = ∑ h = 1 d b h ( s ) ζ h ( s ) U_s=\sum_{h=1}^{d}b_h^{(s)}\zeta_h^{(s)} U s = ∑ h = 1 d b h ( s ) ζ h ( s ) , O γ = E 0 ( ∏ s = 1 k U s ∣ Y ) = E 0 [ ∏ s = 1 k E 0 ( U s ∣ Z 0 , Y ) ∣ Y ] = E 0 [ ∏ s = 1 k { ∑ h = 1 d b h ( s ) E 0 [ ζ h ( s ) ∣ Z 0 , Y ] } ∣ Y ] O_\gamma=\mathbb{E}_0(\prod_{s=1}^{k}U_s\mid Y)=\mathbb{E}_0[\prod_{s=1}^{k}\mathbb{E}_0(U_s\mid Z_0,Y)\mid Y]=\mathbb{E}_0[\prod_{s=1}^{k}\{\sum_{h=1}^{d}b_h^{(s)}\mathbb{E}_0[\zeta_h^{(s)}\mid Z_0,Y]\}\mid Y] O γ = E 0 ( ∏ s = 1 k U s ∣ Y ) = E 0 [ ∏ s = 1 k E 0 ( U s ∣ Z 0 , Y ) ∣ Y ] = E 0 [ ∏ s = 1 k { ∑ h = 1 d b h ( s ) E 0 [ ζ h ( s ) ∣ Z 0 , Y ]} ∣ Y ] . As shown above, E 0 [ ζ h ( s ) ∣ Z 0 , Y ] \mathbb{E}_0[\zeta_h^{(s)}\mid Z_0,Y] E 0 [ ζ h ( s ) ∣ Z 0 , Y ] has conditional mean approximated by some polynomial with leading term α 0 , j h ( s ) ( s ) ⊤ V 0 \alpha_{0,j_h^{(s)}}^{(s)\top}V_0 α 0 , j h ( s ) ( s ) ⊤ V 0 . Then (B.24) and linearity in O γ O_\gamma O γ shows that O γ O_\gamma O γ has leading term E 0 ( V 0 γ ∣ Y ) \mathbb{E}_0(V_0^\gamma\mid Y) E 0 ( V 0 γ ∣ Y ) .
By (B.23) and the independence between responses conditional on ( Z r , Y ) (Z_r,Y) ( Z r , Y ) , we know that
E { ∏ i ∈ S [ X i − E ( X i ∣ Y ) ] ∣ Y } ∏ i ∈ S f ′ ( T i ) = E r [ ∏ i ∈ S g T i ( α r i ⊤ V r + δ r i ) ∣ Y ] . (35) \frac{\mathbb{E}\{\prod_{i\in S}[X_i-\mathbb{E}(X_i\mid Y)]\mid Y\}}{\prod_{i\in S}f'(T_i)}
=
\mathbb{E}_r\left[\prod_{i\in S}g_{T_i}(\alpha_{ri}^{\top}V_r+\delta_{ri})\mid Y\right]. \tag*{(35)} ∏ i ∈ S f ′ ( T i ) E { ∏ i ∈ S [ X i − E ( X i ∣ Y )] ∣ Y } = E r [ i ∈ S ∏ g T i ( α r i ⊤ V r + δ r i ) ∣ Y ] . (35)
for S ⊆ S S\subseteq\mathcal{S} S ⊆ S consisting of excluded indices for S \mathcal{S} S given in Lemma B.1. We first show that this product can be approximated by some polynomial, and then we can obtain the approximation for O γ O_\gamma O γ by taking S = { j h ( s ) : s = 1 , … , k } S=\{j_h^{(s)}:s=1,\ldots,k\} S = { j h ( s ) : s = 1 , … , k } .
We first consider model zero and let ξ i ( r ) = g T i ( α r i ⊤ V r + δ r i ) \xi_i^{(r)}=g_{T_i}(\alpha_{ri}^{\top}V_r+\delta_{ri}) ξ i ( r ) = g T i ( α r i ⊤ V r + δ r i ) for r ∈ { 0 , 1 } r\in\{0,1\} r ∈ { 0 , 1 } . When ∣ S ∣ ≤ N |S|\leq N ∣ S ∣ ≤ N , by Lemma B.5 and the fact that G i , N ( v , m ) G_{i,N}(v,m) G i , N ( v , m ) has weights between one and N N N , with linear part α 0 i ⊤ v \alpha_{0i}^{\top}v α 0 i ⊤ v , we can apply Lemma B.4 treating ξ i ( 0 ) \xi_i^{(0)} ξ i ( 0 ) as the random quantities with weights 1 and G i , N ( v , m ) G_{i,N}(v,m) G i , N ( v , m ) as the polynomial approximation to get
∥ ∏ i ∈ S ξ i ( 0 ) − ev 0 { P S ( v , m ) } ∥ q ≤ e C ( 1 + q ) N 2 p − ( N + 1 ) / 2 , (36) \left\|\prod_{i\in S}\xi_i^{(0)}-\operatorname{ev}_0\{P_S(v,m)\}\right\|_q
\leq e^{C(1+q)N^2}p^{-(N+1)/2}, \tag*{(36)} i ∈ S ∏ ξ i ( 0 ) − ev 0 { P S ( v , m )} q ≤ e C ( 1 + q ) N 2 p − ( N + 1 ) /2 , (36)
where S = { i 1 , … , i ∣ S ∣ } S=\{i_1,\ldots,i_{|S|}\} S = { i 1 , … , i ∣ S ∣ } and P S ( v , m ) : = P S , ∣ S ∣ ( v , m ) P_S(v,m):=P_{S,|S|}(v,m) P S ( v , m ) := P S , ∣ S ∣ ( v , m ) is inductively defined by P S , 0 ( v , m ) = 1 P_{S,0}(v,m)=1 P S , 0 ( v , m ) = 1 and P S , t ( v , m ) = [ P S , t − 1 ( v , m ) G i t , N ( v , m ) ] N P_{S,t}(v,m)=[P_{S,t-1}(v,m)G_{i_t,N}(v,m)]_N P S , t ( v , m ) = [ P S , t − 1 ( v , m ) G i t , N ( v , m ) ] N for t = 1 , 2 , … , ∣ S ∣ t=1,2,\ldots,|S| t = 1 , 2 , … , ∣ S ∣ . Thus P S ( v , m ) = [ ∏ i ∈ S G i , N ( v , m ) ] N P_S(v,m)=[\prod_{i\in S}G_{i,N}(v,m)]_N P S ( v , m ) = [ ∏ i ∈ S G i , N ( v , m ) ] N .
It remains to pass the bound from ∏ i ∈ S ξ i ( 0 ) \prod_{i\in S}\xi_i^{(0)} ∏ i ∈ S ξ i ( 0 ) to the conditional expectation as in (B.26). Let Q S ( m ) = [ E f { P S ( v , m ) } ] N = E f { P S ( v , m ) } Q_S(m)=[\mathbb{E}_f\{P_S(v,m)\}]_N=\mathbb{E}_f\{P_S(v,m)\} Q S ( m ) = [ E f { P S ( v , m )} ] N = E f { P S ( v , m )} . Because E 0 [ ev 0 { P S ( v , m ) } ∣ Y ] = ev 0 ( [ E f { ∏ i ∈ S G i , N ( v , m ) } ] N ) \mathbb{E}_0[\operatorname{ev}_0\{P_S(v,m)\}\mid Y]=\operatorname{ev}_0([\mathbb{E}_f\{\prod_{i\in S}G_{i,N}(v,m)\}]_N) E 0 [ ev 0 { P S ( v , m )} ∣ Y ] = ev 0 ([ E f { ∏ i ∈ S G i , N ( v , m )} ] N ) as formal expectation preserves the weights in each monomial, we have by Jensen inequality that
∥ E 0 ( ∏ i ∈ S ξ i ( 0 ) ∣ Y ) − ev 0 ( Q S ( m ) ) ∥ q = ∥ E 0 [ ∏ i ∈ S ξ i ( 0 ) − ev 0 ( P S ( v , m ) ) ∣ Y ] ∥ q ≤ ∥ ∏ i ∈ S ξ i ( 0 ) − ev 0 ( P S ( v , m ) ) ∥ q ≤ e C ( 1 + q ) N 2 p − ( N + 1 ) / 2 . \begin{aligned}
\left\|\mathbb{E}_0\left(\prod_{i\in S}\xi_i^{(0)}\mid Y\right)-\operatorname{ev}_0(Q_S(m))\right\|_q
&=\left\|\mathbb{E}_0\left[\prod_{i\in S}\xi_i^{(0)}-\operatorname{ev}_0(P_S(v,m))\mid Y\right]\right\|_q\\
&\leq\left\|\prod_{i\in S}\xi_i^{(0)}-\operatorname{ev}_0(P_S(v,m))\right\|_q
\leq e^{C(1+q)N^2}p^{-(N+1)/2}.
\end{aligned} E 0 ( i ∈ S ∏ ξ i ( 0 ) ∣ Y ) − ev 0 ( Q S ( m )) q = E 0 [ i ∈ S ∏ ξ i ( 0 ) − ev 0 ( P S ( v , m )) ∣ Y ] q ≤ i ∈ S ∏ ξ i ( 0 ) − ev 0 ( P S ( v , m )) q ≤ e C ( 1 + q ) N 2 p − ( N + 1 ) /2 .
Therefore, for O γ O_\gamma O γ given in (B.25), we can find a polynomial approximation given by
F γ ( m ) = [ ∑ h 1 , … , h k = 1 d { ∏ s = 1 k ( B s − 1 ) ℓ s h s } E f { ∏ s = 1 k G j h s ( s ) , N ( v , m ) } ] N . (37) F_\gamma(m)=\left[\sum_{h_1,\ldots,h_k=1}^{d}\left\{\prod_{s=1}^{k}(B_s^{-1})_{\ell_s h_s}\right\}\mathbb{E}_f\left\{\prod_{s=1}^{k}G_{j_{h_s}^{(s)},N}(v,m)\right\}\right]_N. \tag*{(37)} F γ ( m ) = h 1 , … , h k = 1 ∑ d { s = 1 ∏ k ( B s − 1 ) ℓ s h s } E f { s = 1 ∏ k G j h s ( s ) , N ( v , m ) } N . (37)
such that ϵ γ = O γ − ev 0 { F γ ( m ) } \epsilon_\gamma=O_\gamma-\operatorname{ev}_0\{F_\gamma(m)\} ϵ γ = O γ − ev 0 { F γ ( m )} satisfies
∥ ϵ γ ∥ q ≤ ∑ h 1 , … , h k = 1 d ∥ ∏ s = 1 k ( B s − 1 ) ℓ s h s ∥ ∥ E 0 ( ∏ s = 1 k ξ j s h s ( 0 ) ∣ Y ) − ev 0 ( [ E F { ∏ s = 1 k G j s h s , N ( s ) ( v , m ) } ] N ) ∥ q ≤ C B k e C ( 1 + q ) N 2 p − ( N + 1 ) / 2 ≤ e C ′ ( 1 + q ) N 2 p − ( N + 1 ) / 2 . (38) \|\epsilon_\gamma\|_q \le\sum_{h_1,\ldots,h_k=1}^{d}\left\|\prod_{s=1}^{k}(B_s^{-1})_{\ell_s h_s}\right\|\left\|\mathbb{E}_0\left(\prod_{s=1}^{k}\xi_{j_s h_s}^{(0)}\mid Y\right)-\operatorname{ev}_0\left(\left[\mathbb{E}_F\left\{\prod_{s=1}^{k}G_{j_s h_s,N}^{(s)}(v,m)\right\}\right]_N\right)\right\|_q
\le C_B^k e^{C(1+q)N^2}p^{-(N+1)/2}\le e^{C'(1+q)N^2}p^{-(N+1)/2}. \tag*{(38)} ∥ ϵ γ ∥ q ≤ h 1 , … , h k = 1 ∑ d s = 1 ∏ k ( B s − 1 ) ℓ s h s E 0 ( s = 1 ∏ k ξ j s h s ( 0 ) ∣ Y ) − ev 0 ( [ E F { s = 1 ∏ k G j s h s , N ( s ) ( v , m ) } ] N ) q ≤ C B k e C ( 1 + q ) N 2 p − ( N + 1 ) /2 ≤ e C ′ ( 1 + q ) N 2 p − ( N + 1 ) /2 . (38)
Here, we use ∏ s = 1 k ∑ h = 1 d ∣ ( B s − 1 ) ℓ s h ∣ ≤ ( d 1 / 2 / c 1 ) k ≤ C B k \prod_{s=1}^{k}\sum_{h=1}^{d}|(B_s^{-1})_{\ell_s h}|\le(d^{1/2}/c_1)^k\le C_B^k ∏ s = 1 k ∑ h = 1 d ∣ ( B s − 1 ) ℓ s h ∣ ≤ ( d 1/2 / c 1 ) k ≤ C B k with C B : = max { 1 , d 1 / 2 / c 1 } C_B:=\max\{1,d^{1/2}/c_1\} C B := max { 1 , d 1/2 / c 1 } , by ∥ B s − 1 ∥ ≤ c 1 − 1 \|B_s^{-1}\|\le c_1^{-1} ∥ B s − 1 ∥ ≤ c 1 − 1 from Lemma B.1 . Moreover, by the previous argument to construct O γ O_\gamma O γ , it is not hard to verify that [ F γ ( m ) ] k = m γ [F_\gamma(m)]_k=m_\gamma [ F γ ( m ) ] k = m γ . Apply a similar argument in showing (B.17), we can obtain
∥ [ F γ ] w − [ F γ ] w − 1 ∥ c o e f f ≤ C w , (39) \|[F_\gamma]_w-[F_\gamma]_{w-1}\|_{\mathrm{coeff}}\le C^w, \tag*{(39)} ∥ [ F γ ] w − [ F γ ] w − 1 ∥ coeff ≤ C w , (39)
for ∣ γ ∣ ≤ w ≤ N |\gamma|\le w\le N ∣ γ ∣ ≤ w ≤ N , and ∥ ∂ α j l [ F γ ] w − ∂ α j l [ F γ ] w − 1 ∥ c o e f f ≤ C w \|\partial_{\alpha_{jl}}[F_\gamma]_w-\partial_{\alpha_{jl}}[F_\gamma]_{w-1}\|_{\mathrm{coeff}}\le C^w ∥ ∂ α j l [ F γ ] w − ∂ α j l [ F γ ] w − 1 ∥ coeff ≤ C w . Also, for ∥ O γ ∥ q \|O_\gamma\|_q ∥ O γ ∥ q , apply
∏ s = 1 k ∑ h = 1 d ∣ ( B s − 1 ) ℓ s h ∣ ≤ C B k , ∥ ξ j ( r ) ∥ a ≤ e C ( 1 + a ) p − 1 / 2 , \prod_{s=1}^{k}\sum_{h=1}^{d}|(B_s^{-1})_{\ell_s h}|\le C_B^k,\qquad\|\xi_{j}^{(r)}\|_a\le e^{C(1+a)}p^{-1/2}, s = 1 ∏ k h = 1 ∑ d ∣ ( B s − 1 ) ℓ s h ∣ ≤ C B k , ∥ ξ j ( r ) ∥ a ≤ e C ( 1 + a ) p − 1/2 ,
and Jensen inequality to get
∥ O γ ∥ q ≤ C B k ∑ h 1 , … , h k = 1 d ∥ E r ( ∏ s = 1 k ξ j s h s ( r ) ∣ Y ) ∥ q ≤ C B k ∑ h 1 , … , h k = 1 d ∏ s = 1 k ∥ ξ j s h s ( r ) ∥ q k ≤ e C ′ ( 1 + q ) k 2 p − k / 2 . (40) \begin{aligned}
\|O_\gamma\|_q\le C_B^k\sum_{h_1,\ldots,h_k=1}^{d}\left\|\mathbb{E}_r\left(\prod_{s=1}^{k}\xi_{j_s h_s}^{(r)}\mid Y\right)\right\|_q \\
&\le C_B^k\sum_{h_1,\ldots,h_k=1}^{d}\prod_{s=1}^{k}\|\xi_{j_s h_s}^{(r)}\|_{qk} \\
&\le e^{C'(1+q)k^2}p^{-k/2}. \tag*{(40)}
\end{aligned} ∥ O γ ∥ q ≤ C B k h 1 , … , h k = 1 ∑ d E r ( s = 1 ∏ k ξ j s h s ( r ) ∣ Y ) q ≤ C B k h 1 , … , h k = 1 ∑ d s = 1 ∏ k ∥ ξ j s h s ( r ) ∥ q k ≤ e C ′ ( 1 + q ) k 2 p − k /2 . (40)
Till now, we establish the polynomial approximations F γ ( m ) F_\gamma(m) F γ ( m ) of order N N N to all constructed observed quantities O γ O_\gamma O γ for 2 ≤ ∣ γ ∣ ≤ N 2\le|\gamma|\le N 2 ≤ ∣ γ ∣ ≤ N , which have their leading terms as ev ( m γ ) \operatorname{ev}(m_\gamma) ev ( m γ ) . Next, we want to use these observed quantities to approximate δ 0 i \delta_{0i} δ 0 i and E 0 { ( α 0 i T V ) ( α 0 j T V ) ∣ Y } \mathbb{E}_0\{(\alpha_{0i}^{\mathsf T}V)(\alpha_{0j}^{\mathsf T}V)\mid Y\} E 0 {( α 0 i T V ) ( α 0 j T V ) ∣ Y } . Recall that we have shown δ i ≈ ev { [ H i ( m ) ] N } \delta_i\approx\operatorname{ev}\{[H_i(m)]_N\} δ i ≈ ev {[ H i ( m ) ] N } in Lemma B.5 . We aim to find some polynomial R i ( O ) ≈ δ 0 i R_i(O)\approx\delta_{0i} R i ( O ) ≈ δ 0 i . One may want to directly use ev 0 { [ H i ( F ( m ) ) ] N } \operatorname{ev}_0\{[H_i(F(m))]_N\} ev 0 {[ H i ( F ( m )) ] N } . While this can keep the leading terms aligned, interactions among the higher-order terms in F ( m ) F(m) F ( m ) produce a more complicated expansion of H i ( F ( m ) ) H_i(F(m)) H i ( F ( m )) . As in the successive recovery of M 4 M_4 M 4 , M 3 M_3 M 3 , M 2 M_2 M 2 in B.10, these terms must be removed before the desired accuracy is attained.
To cancel the higher order terms introduced in F γ F_\gamma F γ , we define an operator U \mathcal{U} U . Let P N \mathcal{P}_N P N be the space of polynomials in these moment symbols whose monomials have weight at most N N N , with coefficient functions of T = ( T i ) i ∈ S \mathbf{T}=(T_i)_{i\in S} T = ( T i ) i ∈ S as in the coefficient norm defined above. For any P ( m ) = ∑ η a η ( T ) m η ∈ P N P(m)=\sum_{\eta}a_{\eta}(\mathcal{T})m^{\eta}\in\mathcal{P}_{N} P ( m ) = ∑ η a η ( T ) m η ∈ P N , define
U P ( m ) : = [ ∑ η a η ( T ) ∏ γ : 2 ≤ ∣ γ ∣ ≤ N F γ ( m ) η γ ] N . (41) \mathcal{U}P(m):=\left[\sum_{\eta}a_{\eta}(\mathcal{T})\prod_{\gamma:2\leq|\gamma|\leq N}F_{\gamma}(m)^{\eta_{\gamma}}\right]_{N}. \tag*{(41)} U P ( m ) := η ∑ a η ( T ) γ : 2 ≤ ∣ γ ∣ ≤ N ∏ F γ ( m ) η γ N . (41)
By definition, we have ( U − I ) N + 1 = 0 (\mathcal{U}-\mathcal{I})^{N+1}=0 ( U − I ) N + 1 = 0 where I \mathcal{I} I is identity mapping here. Therefore we know the inverse of U \mathcal{U} U can be expressed as U − 1 = ∑ l = 0 N { − ( U − I ) } l \mathcal{U}^{-1}=\sum_{l=0}^{N}\{-(\mathcal{U}-\mathcal{I})\}^{l} U − 1 = ∑ l = 0 N { − ( U − I ) } l , which will be used to cancel the high order terms introduced by polynomial approximation F γ F_{\gamma} F γ . We then introduce two technical lemmas.
Lemma B.6. Under Assumption 2, there is a constant C > 0 C>0 C > 0 such that, for p ≥ p 0 p\geq p_{0} p ≥ p 0 ,
min O T O = I d max j ∈ [ p ] ∥ α 1 j − O α 0 j ∥ ≤ C max i , j ∈ [ p ] ∣ α 1 i T α 1 j − α 0 i T α 0 j ∣ . (42) \min_{O^{\mathsf{T}}O=I_{d}}\max_{j\in[p]}\lVert\alpha_{1j}-O\alpha_{0j}\rVert\leq C\max_{i,j\in[p]}\left|\alpha_{1i}^{\mathsf{T}}\alpha_{1j}-\alpha_{0i}^{\mathsf{T}}\alpha_{0j}\right|. \tag*{(42)} O T O = I d min j ∈ [ p ] max ∥ α 1 j − O α 0 j ∥ ≤ C i , j ∈ [ p ] max α 1 i T α 1 j − α 0 i T α 0 j . (42)
Proof. See Appendix § 9 .6.
The next lemma shows that replacing m γ m_{\gamma} m γ with the corresponding O γ O_{\gamma} O γ in U − 1 [ H i ( m ) ] N \mathcal{U}^{-1}[H_i(m)]_N U − 1 [ H i ( m ) ] N can well approximate δ 0 i \delta_{0i} δ 0 i . Write D p = D p ( θ 0 , θ 1 ) D_p=D_p(\theta_0,\theta_1) D p = D p ( θ 0 , θ 1 ) for short.
Lemma B.7. Suppose Assumptions 1 and 2 hold. Under model zero, define polynomials
R i ( m ) = U − 1 [ H i ( m ) ] N , R i j ( m ) = U − 1 E f { ( α 0 i T v ) ( α 0 j T v ) } . (43) R_i(m)=\mathcal{U}^{-1}[H_i(m)]_N,\qquad R_{ij}(m)=\mathcal{U}^{-1}\mathbb{E}_{f}\{(\alpha_{0i}^{\mathsf{T}}v)(\alpha_{0j}^{\mathsf{T}}v)\}. \tag*{(43)} R i ( m ) = U − 1 [ H i ( m ) ] N , R ij ( m ) = U − 1 E f {( α 0 i T v ) ( α 0 j T v )} . (43) They contain only terms of weights between two and N N N , and satisfy
∥ R i ( m ) ∥ c o e f f + ∥ R i j ( m ) ∥ c o e f f ≤ e C N 2 . \lVert R_i(m)\rVert_{\mathrm{coeff}}+\lVert R_{ij}(m)\rVert_{\mathrm{coeff}}\leq e^{CN^2}. ∥ R i ( m ) ∥ coeff + ∥ R ij ( m ) ∥ coeff ≤ e C N 2 . Let R i ( O ) R_i(O) R i ( O ) and R i j ( O ) R_{ij}(O) R ij ( O ) be evaluated value obtained by replacing each moment symbol m γ m_{\gamma} m γ with O γ O_{\gamma} O γ . By Lemma B.6 , we may assume without loss of generality that the two representations have been orthogonally aligned so that max a ∈ [ p ] ∥ α 1 a − α 0 a ∥ ≤ C D p \max_{a\in[p]}\lVert\alpha_{1a}-\alpha_{0a}\rVert\leq CD_p max a ∈ [ p ] ∥ α 1 a − α 0 a ∥ ≤ C D p . For q ∈ { 1 , 2 } q\in\{1,2\} q ∈ { 1 , 2 } and r ∈ { 0 , 1 } r\in\{0,1\} r ∈ { 0 , 1 } , there is a common τ > 0 \tau>0 τ > 0 such that, whenever 4 ≤ N ≤ τ log p 4\leq N\leq\tau\log p 4 ≤ N ≤ τ log p and N d + 2 ≤ c 0 p Nd+2\leq c_0p N d + 2 ≤ c 0 p , with c 0 c_0 c 0 as in Lemma B.1 , the following bound holds:
∥ R i ( O ) − δ r i ∥ q + ∥ R i j ( O ) − E r { ( α r i T V r ) ( α r j T V r ) ∣ Y } ∥ q ≤ r C q D p / p + e C ( 1 + q ) N 2 p − ( N + 1 ) / 2 . (44) \lVert R_i(O)-\delta_{ri}\rVert_q+\left\lVert R_{ij}(O)-\mathbb{E}_{r}\{(\alpha_{ri}^{\mathsf{T}}V_r)(\alpha_{rj}^{\mathsf{T}}V_r)\mid Y\}\right\rVert_q\leq rC_qD_p/p+e^{C(1+q)N^2}p^{-(N+1)/2}. \tag*{(44)} ∥ R i ( O ) − δ r i ∥ q + R ij ( O ) − E r {( α r i T V r ) ( α r j T V r ) ∣ Y } q ≤ r C q D p / p + e C ( 1 + q ) N 2 p − ( N + 1 ) /2 . (44) Here, the first term on the right sides says that, for r = 0 r=0 r = 0 , the D p / p D_p/p D p / p term vanishes. Under this case, the inequality holds for any q ≥ 1 q\geq1 q ≥ 1 .
Proof. See Appendix § 9 .7.
Step 4. Combining the bounds to control D p D_p D p .
Finally, we can compare
β r i = E { E r ( α r i T Z r + β r i ∣ Y ) } = E [ T i + δ r i ] , \beta_{ri} = \mathbb{E}\{\mathbb{E}_r(\alpha_{ri}^{\mathsf{T}}Z_r+\beta_{ri}\mid Y)\}=\mathbb{E}[T_i+\delta_{ri}], β r i = E { E r ( α r i T Z r + β r i ∣ Y )} = E [ T i + δ r i ] ,
α r i T α r j = Cov { E r ( α r i T Z r + β r i ∣ Y ) , E r ( α r j T Z r + β r j ∣ Y ) } + E r { ( α r i T V r ) ( α r j T V r ) } . \alpha_{ri}^{\mathsf{T}}\alpha_{rj}=\operatorname{Cov}\{\mathbb{E}_r(\alpha_{ri}^{\mathsf{T}}Z_r+\beta_{ri}\mid Y),\mathbb{E}_r(\alpha_{rj}^{\mathsf{T}}Z_r+\beta_{rj}\mid Y)\}+\mathbb{E}_r\{(\alpha_{ri}^{\mathsf{T}}V_r)(\alpha_{rj}^{\mathsf{T}}V_r)\}. α r i T α r j = Cov { E r ( α r i T Z r + β r i ∣ Y ) , E r ( α r j T Z r + β r j ∣ Y )} + E r {( α r i T V r ) ( α r j T V r )} .
By Lemma B.7 , we have
∣ β 0 i − β 1 i ∣ ≤ ∥ T i + δ 0 i − ( T i + δ 1 i ) ∥ 1 ≤ ∥ δ 0 i − R i ( O ) ∥ 1 + ∥ δ 1 i − R i ( O ) ∥ 1 ≤ C D p / p + e C N 2 p − ( N + 1 ) / 2 . |\beta_{0i}-\beta_{1i}|\leq\|T_i+\delta_{0i}-(T_i+\delta_{1i})\|_1
\leq\|\delta_{0i}-R_i(O)\|_1+\|\delta_{1i}-R_i(O)\|_1
\leq CD_p/p+e^{CN^2}p^{-(N+1)/2}. ∣ β 0 i − β 1 i ∣ ≤ ∥ T i + δ 0 i − ( T i + δ 1 i ) ∥ 1 ≤ ∥ δ 0 i − R i ( O ) ∥ 1 + ∥ δ 1 i − R i ( O ) ∥ 1 ≤ C D p / p + e C N 2 p − ( N + 1 ) /2 .
For ∣ α 0 i T α 0 j − α 1 i T α 1 j ∣ |\alpha_{0i}^{\mathsf{T}}\alpha_{0j}-\alpha_{1i}^{\mathsf{T}}\alpha_{1j}| ∣ α 0 i T α 0 j − α 1 i T α 1 j ∣ , we first note that by Jensen’s inequality, ∥ E r ( α r i T Z r + β r i ∣ Y ) ∥ 2 ≤ ∥ α r i T Z r + β r i ∥ 2 ≤ M ( 1 + d 1 / 2 ) \|\mathbb{E}_r(\alpha_{ri}^{\mathsf{T}}Z_r+\beta_{ri}\mid Y)\|_2\leq\|\alpha_{ri}^{\mathsf{T}}Z_r+\beta_{ri}\|_2\leq M(1+d^{1/2}) ∥ E r ( α r i T Z r + β r i ∣ Y ) ∥ 2 ≤ ∥ α r i T Z r + β r i ∥ 2 ≤ M ( 1 + d 1/2 ) for r ∈ { 0 , 1 } r\in\{0,1\} r ∈ { 0 , 1 } . Here, ∥ ⋅ ∥ 2 \|\cdot\|_2 ∥ ⋅ ∥ 2 denotes the L 2 L^2 L 2 norm for random variables. Then writing μ r i : = E r ( α r i T Z r + β r i ∣ Y ) = T i + δ r i \mu_{ri}:=\mathbb{E}_r(\alpha_{ri}^{\mathsf{T}}Z_r+\beta_{ri}\mid Y)=T_i+\delta_{ri} μ r i := E r ( α r i T Z r + β r i ∣ Y ) = T i + δ r i for r ∈ { 0 , 1 } r\in\{0,1\} r ∈ { 0 , 1 } , we have ∥ μ r i ∥ 2 ≤ M ( 1 + d 1 / 2 ) \|\mu_{ri}\|_2\leq M(1+d^{1/2}) ∥ μ r i ∥ 2 ≤ M ( 1 + d 1/2 ) . With μ 0 i − μ 1 i = δ 0 i − δ 1 i \mu_{0i}-\mu_{1i}=\delta_{0i}-\delta_{1i} μ 0 i − μ 1 i = δ 0 i − δ 1 i and Lemma B.7 , we know
∥ μ 0 i − μ 1 i ∥ 2 ≤ ∥ δ 0 i − R i ( O ) ∥ 2 + ∥ δ 1 i − R i ( O ) ∥ 2 ≤ C D p / p + e C N 2 p − ( N + 1 ) / 2 . \|\mu_{0i}-\mu_{1i}\|_2\leq\|\delta_{0i}-R_i(O)\|_2+\|\delta_{1i}-R_i(O)\|_2\leq CD_p/p+e^{CN^2}p^{-(N+1)/2}. ∥ μ 0 i − μ 1 i ∥ 2 ≤ ∥ δ 0 i − R i ( O ) ∥ 2 + ∥ δ 1 i − R i ( O ) ∥ 2 ≤ C D p / p + e C N 2 p − ( N + 1 ) /2 .
Finally, using ∣ Cov ( U , V ) ∣ ≤ ∥ U ∥ 2 ∥ V ∥ 2 |\operatorname{Cov}(U,V)|\leq\|U\|_2\|V\|_2 ∣ Cov ( U , V ) ∣ ≤ ∥ U ∥ 2 ∥ V ∥ 2 , one can compute
∣ α 0 i T α 0 j − α 1 i T α 1 j ∣ ≤ ∣ Cov ( μ 0 i − μ 1 i , μ 0 j ) ∣ + ∣ Cov ( μ 1 i , μ 0 j − μ 1 j ) ∣ + ∑ r = 0 1 ∥ E r { ( α r i T V r ) ( α r j T V r ) ∣ Y } − R i j ( O ) ∥ 1 ≤ M ( 1 + d 1 / 2 ) { ∥ μ 0 i − μ 1 i ∥ 2 + ∥ μ 0 j − μ 1 j ∥ 2 } + ∑ r = 0 1 ∥ E r { ( α r i T V r ) ( α r j T V r ) ∣ Y } − R i j ( O ) ∥ 1 ≤ C D p / p + e C N 2 p − ( N + 1 ) / 2 , \begin{aligned}
|\alpha_{0i}^{\mathsf{T}}\alpha_{0j}-\alpha_{1i}^{\mathsf{T}}\alpha_{1j}|
&\leq|\operatorname{Cov}(\mu_{0i}-\mu_{1i},\mu_{0j})|+|\operatorname{Cov}(\mu_{1i},\mu_{0j}-\mu_{1j})| \\
&\quad+\sum_{r=0}^{1}\|\mathbb{E}_r\{(\alpha_{ri}^{\mathsf{T}}V_r)(\alpha_{rj}^{\mathsf{T}}V_r)\mid Y\}-R_{ij}(O)\|_1 \\
&\leq M(1+d^{1/2})\{\|\mu_{0i}-\mu_{1i}\|_2+\|\mu_{0j}-\mu_{1j}\|_2\} \\
&\quad+\sum_{r=0}^{1}\|\mathbb{E}_r\{(\alpha_{ri}^{\mathsf{T}}V_r)(\alpha_{rj}^{\mathsf{T}}V_r)\mid Y\}-R_{ij}(O)\|_1 \\
&\leq CD_p/p+e^{CN^2}p^{-(N+1)/2},
\end{aligned} ∣ α 0 i T α 0 j − α 1 i T α 1 j ∣ ≤ ∣ Cov ( μ 0 i − μ 1 i , μ 0 j ) ∣ + ∣ Cov ( μ 1 i , μ 0 j − μ 1 j ) ∣ + r = 0 ∑ 1 ∥ E r {( α r i T V r ) ( α r j T V r ) ∣ Y } − R ij ( O ) ∥ 1 ≤ M ( 1 + d 1/2 ) { ∥ μ 0 i − μ 1 i ∥ 2 + ∥ μ 0 j − μ 1 j ∥ 2 } + r = 0 ∑ 1 ∥ E r {( α r i T V r ) ( α r j T V r ) ∣ Y } − R ij ( O ) ∥ 1 ≤ C D p / p + e C N 2 p − ( N + 1 ) /2 ,
where the last inequality uses (B.35) at q = 1 , 2 q=1,2 q = 1 , 2 .
Taking the two maxima defining D p D_p D p and increasing C C C gives
D p ≤ C D p / p + e C N 2 p − ( N + 1 ) / 2 , (45) D_p\leq CD_p/p+e^{CN^2}p^{-(N+1)/2}, \tag*{(45)} D p ≤ C D p / p + e C N 2 p − ( N + 1 ) /2 , (45)
with 4 ≤ N ≤ τ log p 4\leq N\leq\tau\log p 4 ≤ N ≤ τ log p and N d + 2 ≤ c 0 p Nd+2\leq c_0p N d + 2 ≤ c 0 p required by Lemma B.1 . When p ≥ 2 C p\geq2C p ≥ 2 C , choose N = ⌊ τ log p ⌋ N=\lfloor\tau\log p\rfloor N = ⌊ τ log p ⌋ and sufficiently small τ \tau τ , we know D p ≤ 2 e − ( τ / 4 ) ( log p ) 2 D_p\leq2e^{-(\tau/4)(\log p)^2} D p ≤ 2 e − ( τ /4 ) ( l o g p ) 2 . The proof is then completed by Lemma B.6 .
Proof of Corollary 1
Proof. Both ( θ 0 , G 0 ) (\theta_0, G_0) ( θ 0 , G 0 ) and ( θ † , G † ) (\theta^\dagger, G^\dagger) ( θ † , G † ) belong to M p \mathcal{M}_p M p and yield the same marginal distributions for the observations. Hence Theorem 2 gives, for all sufficiently large p p p , that D p ( θ † , θ 0 ) ≤ C exp { − c ( log p ) 2 } D_p(\theta^\dagger,\theta_0) \leq C\exp\{-c(\log p)^2\} D p ( θ † , θ 0 ) ≤ C exp { − c ( log p ) 2 } . The triangle inequality holds for D p ( ⋅ , ⋅ ) D_p(\cdot,\cdot) D p ( ⋅ , ⋅ ) by definition, and therefore
∣ D p ( θ ^ n , θ 0 ) − D p ( θ ^ n , θ † ) ∣ ≤ D p ( θ † , θ 0 ) , \left|D_p(\hat{\theta}_n,\theta_0)-D_p(\hat{\theta}_n,\theta^\dagger)\right| \leq D_p(\theta^\dagger,\theta_0), D p ( θ ^ n , θ 0 ) − D p ( θ ^ n , θ † ) ≤ D p ( θ † , θ 0 ) ,
which proves (6) .
For the inference part, the Lipschitz condition and Theorem 2 give ∣ ψ ( θ † ) − ψ ( θ 0 ) ∣ ≤ C L ψ exp { − c ( log p ) 2 } |\psi(\theta^\dagger)-\psi(\theta_0)| \leq CL_\psi\exp\{-c(\log p)^2\} ∣ ψ ( θ † ) − ψ ( θ 0 ) ∣ ≤ C L ψ exp { − c ( log p ) 2 } . After division by s ^ n , p \hat{s}_{n,p} s ^ n , p , this difference is o P ( 1 ) o_P(1) o P ( 1 ) , so the s ^ n , p − 1 { ψ ( θ ^ n ) − ψ ( θ † ) } \hat{s}_{n,p}^{-1}\{\psi(\hat{\theta}_n)-\psi(\theta^\dagger)\} s ^ n , p − 1 { ψ ( θ ^ n ) − ψ ( θ † )} and s ^ n , p − 1 { ψ ( θ ^ n ) − ψ ( θ 0 ) } \hat{s}_{n,p}^{-1}\{\psi(\hat{\theta}_n)-\psi(\theta_0)\} s ^ n , p − 1 { ψ ( θ ^ n ) − ψ ( θ 0 )} have the same limiting distribution by Slutsky’s theorem.
Proof of Corollary 2
For notational convenience we write ( θ 1 , G 1 ) (\theta_1,G_1) ( θ 1 , G 1 ) as ( θ † , G † ) (\theta^\dagger,G^\dagger) ( θ † , G † ) . By Lemma B.6 , we can choose an orthogonal matrix Q Q Q such that max j ∈ [ p ] ∥ Q α 1 j − α 0 j ∥ ≤ C D p ( θ 0 , θ 1 ) \max_{j\in[p]}\|Q\alpha_{1j}-\alpha_{0j}\| \leq CD_p(\theta_0,\theta_1) max j ∈ [ p ] ∥ Q α 1 j − α 0 j ∥ ≤ C D p ( θ 0 , θ 1 ) . Applying Z 1 ↦ Q Z 1 Z_1 \mapsto QZ_1 Z 1 ↦ Q Z 1 replaces the loading vectors by Q α 1 j Q\alpha_{1j} Q α 1 j and the latent distribution by Q # G † Q_{\#}G^\dagger Q # G † . For simplicity, we set the aligned loading vectors and latent distribution again by α 1 j \alpha_{1j} α 1 j and G 1 G_1 G 1 , respectively, and thus max j ∈ [ p ] ∥ α 1 j − α 0 j ∥ ≤ C D p ( θ 0 , θ 1 ) \max_{j\in[p]}\|\alpha_{1j}-\alpha_{0j}\| \leq CD_p(\theta_0,\theta_1) max j ∈ [ p ] ∥ α 1 j − α 0 j ∥ ≤ C D p ( θ 0 , θ 1 ) . In the proof of Lemma B.2 , we construct Z ^ \hat{Z} Z ^ to approximate Z Z Z . Specifically, suppose Z ^ r ( x ) \hat{Z}_r(x) Z ^ r ( x ) minimizes
Φ r ( z ; x ) = 1 p ∑ j = 1 p { ∫ 0 α r j T z + β r j f ( s ) d s − x j ( α r j T z + β r j ) } + ∥ z ∥ 2 2 p . \Phi_r(z;x)=\frac{1}{p}\sum_{j=1}^{p}\left\{\int_{0}^{\alpha_{rj}^{\mathsf T}z+\beta_{rj}}f(s)\,ds-x_j(\alpha_{rj}^{\mathsf T}z+\beta_{rj})\right\}+\frac{\|z\|^2}{2p}. Φ r ( z ; x ) = p 1 j = 1 ∑ p { ∫ 0 α r j T z + β r j f ( s ) d s − x j ( α r j T z + β r j ) } + 2 p ∥ z ∥ 2 .
Each p Φ ( z ; x ) p\Phi(z;x) p Φ ( z ; x ) is strongly convex, so its minimizer exists and is unique. In the proof of Lemma B.2 , (C.6) implies E θ r , G r ∥ Z − Z ^ r ( X ) ∥ 2 ≤ C p − 1 \mathbb{E}_{\theta_r,G_r}\|Z-\hat{Z}_r(X)\|^2 \leq Cp^{-1} E θ r , G r ∥ Z − Z ^ r ( X ) ∥ 2 ≤ C p − 1 for r ∈ { 0 , 1 } r\in\{0,1\} r ∈ { 0 , 1 } here X X X is the observed vector. To bound ∥ Z ^ 0 ( X ) − Z ^ 1 ( X ) ∥ \|\hat{Z}_0(X)-\hat{Z}_1(X)\| ∥ Z ^ 0 ( X ) − Z ^ 1 ( X ) ∥ , we note that because Φ 0 ( ⋅ ; x ) \Phi_0(\cdot;x) Φ 0 ( ⋅ ; x ) is strongly convex and ∂ Z Φ 1 ( Z ^ 1 ( x ) ; x ) = 0 \partial_Z\Phi_1(\hat{Z}_1(x);x)=0 ∂ Z Φ 1 ( Z ^ 1 ( x ) ; x ) = 0 for any x ∈ { 0 , 1 } p x\in\{0,1\}^p x ∈ { 0 , 1 } p , it holds that
∥ Z ^ 0 ( x ) − Z ^ 1 ( x ) ∥ ≤ p ∥ ∂ Z Φ 0 ( Z ^ 1 ( x ) ; x ) − ∂ Z Φ 1 ( Z ^ 1 ( x ) ; x ) ∥ . \|\hat{Z}_0(x)-\hat{Z}_1(x)\| \leq p\|\partial_Z\Phi_0(\hat{Z}_1(x);x)-\partial_Z\Phi_1(\hat{Z}_1(x);x)\|. ∥ Z ^ 0 ( x ) − Z ^ 1 ( x ) ∥ ≤ p ∥ ∂ Z Φ 0 ( Z ^ 1 ( x ) ; x ) − ∂ Z Φ 1 ( Z ^ 1 ( x ) ; x ) ∥.
Since ∫ − ∞ + ∞ f ′ ( s ) d s = 1 \int_{-\infty}^{+\infty}f'(s)\,ds=1 ∫ − ∞ + ∞ f ′ ( s ) d s = 1 , by (B.6), we know 1 ≥ f ′ ( t ) ∫ − ∞ + ∞ e − 2 K ∣ s − t ∣ d s = f ′ ( t ) / K 1\geq f'(t)\int_{-\infty}^{+\infty}e^{-2K|s-t|}\,ds=f'(t)/K 1 ≥ f ′ ( t ) ∫ − ∞ + ∞ e − 2 K ∣ s − t ∣ d s = f ′ ( t ) / K , and thus f f f is K K K -Lipschitz. Then by Assumption 2(i), one can obtain
∥ ∇ Φ 0 ( z ; x ) − ∇ Φ 1 ( z ; x ) ∥ ≤ p − 1 ∥ ∑ j = 1 p { α 0 j f ( α 0 j T z + β 0 j ) − α 1 j f ( α 1 j T z + β 1 j ) − ( α 0 j − α 1 j ) x j } ∥ ≤ C D p ( θ 0 , θ 1 ) ( 1 + ∥ z ∥ ) . \begin{aligned}
\|\nabla\Phi_0(z;x)-\nabla\Phi_1(z;x)\| \leq p^{-1}\left\|\sum_{j=1}^{p}\left\{\alpha_{0j}f(\alpha_{0j}^{\mathsf T}z+\beta_{0j})-\alpha_{1j}f(\alpha_{1j}^{\mathsf T}z+\beta_{1j})-(\alpha_{0j}-\alpha_{1j})x_j\right\}\right\| \\
&\leq CD_p(\theta_0,\theta_1)(1+\|z\|).
\end{aligned} ∥∇ Φ 0 ( z ; x ) − ∇ Φ 1 ( z ; x ) ∥ ≤ p − 1 j = 1 ∑ p { α 0 j f ( α 0 j T z + β 0 j ) − α 1 j f ( α 1 j T z + β 1 j ) − ( α 0 j − α 1 j ) x j } ≤ C D p ( θ 0 , θ 1 ) ( 1 + ∥ z ∥ ) .
For Z ^ 0 ( X ) \widehat{Z}_0(X) Z 0 ( X ) and Z ^ 1 ( X ) \widehat{Z}_1(X) Z 1 ( X ) , we know by (C.4) that ∥ Z ^ 0 ( X ) ∥ ≤ p M \|\widehat{Z}_0(X)\| \le pM ∥ Z 0 ( X ) ∥ ≤ pM and ∥ Z ^ 1 ( X ) ∥ ≤ p M \|\widehat{Z}_1(X)\| \le pM ∥ Z 1 ( X ) ∥ ≤ pM . Combine these estimates and we know ∥ Z ^ 0 ( X ) − Z ^ 1 ( X ) ∥ ≤ C p 2 D p ( θ 0 , θ 1 ) \|\widehat{Z}_0(X)-\widehat{Z}_1(X)\| \le Cp^2D_p(\theta_0,\theta_1) ∥ Z 0 ( X ) − Z 1 ( X ) ∥ ≤ C p 2 D p ( θ 0 , θ 1 ) .
Now, we are ready to bound the Wasserstein distance between G 0 G_0 G 0 and G 1 G_1 G 1 . Let H r H_r H r be the distribution of Z ^ r ( X ) \widehat{Z}_r(X) Z r ( X ) . The joint distribution of ( Z , z ^ r ( X ) ) (Z,\widehat{z}_r(X)) ( Z , z r ( X )) couple G r G_r G r and H r H_r H r , while ( Z ^ 0 ( X ) , Z ^ 1 ( X ) ) (\widehat{Z}_0(X),\widehat{Z}_1(X)) ( Z 0 ( X ) , Z 1 ( X )) couples H 0 H_0 H 0 and H 1 H_1 H 1 . Then we have
W 1 ( G 0 , G 1 ) ≤ W 1 ( G 0 , H 0 ) + W 1 ( H 0 , H 1 ) + W 1 ( H 1 , G 1 ) ≤ ∑ r = 0 1 E θ r , G r ∥ Z − Z ^ r ( X ) ∥ + E P ∥ Z ^ 0 ( X ) − Z ^ 1 ( X ) ∥ ≤ C p − 1 / 2 + C p 2 D p ( θ 0 , θ 1 ) ≤ C ′ p − 1 / 2 \begin{aligned}
W_1(G_0,G_1) \le W_1(G_0,H_0)+W_1(H_0,H_1)+W_1(H_1,G_1) \\
&\le\sum_{r=0}^{1}\mathbb{E}_{\theta_r,G_r}\|Z-\widehat{Z}_r(X)\|+\mathbb{E}_{P}\|\widehat{Z}_0(X)-\widehat{Z}_1(X)\| \\
&\le Cp^{-1/2}+Cp^2D_p(\theta_0,\theta_1)\le C'p^{-1/2}
\end{aligned} W 1 ( G 0 , G 1 ) ≤ W 1 ( G 0 , H 0 ) + W 1 ( H 0 , H 1 ) + W 1 ( H 1 , G 1 ) ≤ r = 0 ∑ 1 E θ r , G r ∥ Z − Z r ( X ) ∥ + E P ∥ Z 0 ( X ) − Z 1 ( X ) ∥ ≤ C p − 1/2 + C p 2 D p ( θ 0 , θ 1 ) ≤ C ′ p − 1/2
for all sufficiently large p p p , by Cauchy–Schwarz and Theorem 2. Then because orthogonal transformations preserve W 1 ( ⋅ , ⋅ ) W_1(\cdot,\cdot) W 1 ( ⋅ , ⋅ ) , we conclude that ∣ W 1 ( Q # G ^ n , G 0 ) − W 1 ( G ^ n , G † ) ∣ ≤ W 1 ( G 0 , Q # G † ) ≤ C p − 1 / 2 |W_1(Q_{\#}\widehat{G}_n,G_0)-W_1(\widehat{G}_n,G^\dagger)|\le W_1(G_0,Q_{\#}G^\dagger)\le Cp^{-1/2} ∣ W 1 ( Q # G n , G 0 ) − W 1 ( G n , G † ) ∣ ≤ W 1 ( G 0 , Q # G † ) ≤ C p − 1/2 .
Finally, for any L h L_h L h -Lipschitz function h h h and distributions μ , ν \mu,\nu μ , ν , integrating ∣ h ( z ) − h ( z ′ ) ∣ ≤ L h ∥ z − z ′ ∥ |h(z)-h(z')|\le L_h\|z-z'\| ∣ h ( z ) − h ( z ′ ) ∣ ≤ L h ∥ z − z ′ ∥ under any coupling and taking the infimum gives ∣ E μ h − E ν h ∣ ≤ L h W 1 ( μ , ν ) |\mathbb{E}_{\mu}h-\mathbb{E}_{\nu}h|\le L_hW_1(\mu,\nu) ∣ E μ h − E ν h ∣ ≤ L h W 1 ( μ , ν ) . With W 1 ( Q # G ^ n , G 0 ) ≤ W 1 ( G ^ n , G † ) + C p − 1 / 2 W_1(Q_{\#}\widehat{G}_n,G_0)\le W_1(\widehat{G}_n,G^\dagger)+Cp^{-1/2} W 1 ( Q # G n , G 0 ) ≤ W 1 ( G n , G † ) + C p − 1/2 , apply this inequality with μ = Q # G ^ n \mu=Q_{\#}\widehat{G}_n μ = Q # G n and ν = G 0 \nu=G_0 ν = G 0 proves (8) .
Details for Example 2
We give sufficient regularity conditions for all p p p :
(i) d ≥ 2 d\ge2 d ≥ 2 is fixed, and both θ , θ ~ \theta,\widetilde{\theta} θ , θ have the same known group membership g ( j ) ∈ { 1 , … , d } g(j)\in\{1,\ldots,d\} g ( j ) ∈ { 1 , … , d } ;
(ii) Σ − 1 / 2 Z \Sigma^{-1/2}Z Σ − 1/2 Z and Σ ~ − 1 / 2 Z ~ \widetilde{\Sigma}^{-1/2}\widetilde{Z} Σ − 1/2 Z satisfy Assumptions 1 and 2 in dimension d + 1 d+1 d + 1 ;
(iii) λ min ( Σ ) \lambda_{\min}(\Sigma) λ m i n ( Σ ) and λ min ( Σ ~ ) \lambda_{\min}(\widetilde{\Sigma}) λ m i n ( Σ ) are bounded away from zero;
(iv) for each g ∈ { 1 , … , d } g\in\{1,\ldots,d\} g ∈ { 1 , … , d } , there are distinct a g , b g a_g,b_g a g , b g with g ( a g ) = g ( b g ) = g g(a_g)=g(b_g)=g g ( a g ) = g ( b g ) = g such that the 2 × 2 2\times2 2 × 2 matrices
L g = ( λ j k ) j ∈ { a g , b g } , k ∈ { 0 , 1 } , L ~ g = ( λ ~ j k ) j ∈ { a g , b g } , k ∈ { 0 , 1 } L_g=(\lambda_{jk})_{j\in\{a_g,b_g\},\,k\in\{0,1\}}, \qquad\widetilde{L}_g=(\widetilde{\lambda}_{jk})_{j\in\{a_g,b_g\},\,k\in\{0,1\}} L g = ( λ j k ) j ∈ { a g , b g } , k ∈ { 0 , 1 } , L g = ( λ j k ) j ∈ { a g , b g } , k ∈ { 0 , 1 }
satisfy σ min ( L g ) ≥ c \sigma_{\min}(L_g)\ge c σ m i n ( L g ) ≥ c and σ min ( L ~ g ) ≥ c \sigma_{\min}(\widetilde{L}_g)\ge c σ m i n ( L g ) ≥ c for some constant c > 0 c>0 c > 0 .
Let a j = Σ 1 / 2 λ j a_j=\Sigma^{1/2}\lambda_j a j = Σ 1/2 λ j and u g = Σ 1 / 2 e g + 1 ( d + 1 ) u_g=\Sigma^{1/2}e_{g+1}^{(d+1)} u g = Σ 1/2 e g + 1 ( d + 1 ) , with analogous tilded notation, where e i ( d + 1 ) e_i^{(d+1)} e i ( d + 1 ) denotes the i i i th standard basis vector in R d + 1 \mathbb{R}^{d+1} R d + 1 . By the covariance structure, we have a j = λ j 0 e 1 ( d + 1 ) + λ j 1 u g ( j ) a_j=\lambda_{j0}e_1^{(d+1)}+\lambda_{j1}u_{g(j)} a j = λ j 0 e 1 ( d + 1 ) + λ j 1 u g ( j ) , where ∥ u g ∥ = 1 \|u_g\|=1 ∥ u g ∥ = 1 and u g T e 1 ( d + 1 ) = 0 u_g^{\mathsf{T}}e_1^{(d+1)}=0 u g T e 1 ( d + 1 ) = 0 . Theorem 2 and Lemma B.6 yield an orthogonal Q Q Q such that
∥ β − β ~ ∥ ∞ + max j ∈ [ p ] ∥ a j − Q a ~ j ∥ ≤ C exp { − c ( log p ) 2 } . (46) \|\beta-\widetilde{\beta}\|_{\infty}+\max_{j\in[p]}\|a_j-Q\widetilde{a}_j\|\le C\exp\{-c(\log p)^2\}. \tag*{(46)} ∥ β − β ∥ ∞ + j ∈ [ p ] max ∥ a j − Q a j ∥ ≤ C exp { − c ( log p ) 2 } . (46)
Fix a group g g g and let ( t 1 , t 2 ) T = ( L ~ g ) − 1 ( 1 , 0 ) T (t_1,t_2)^\mathsf{T}=(\widetilde{L}_g)^{-1}(1,0)^\mathsf{T} ( t 1 , t 2 ) T = ( L g ) − 1 ( 1 , 0 ) T . Condition ( i v ) (iv) ( i v ) gives ∣ t 1 ∣ + ∣ t 2 ∣ ≤ C |t_1|+|t_2|\le C ∣ t 1 ∣ + ∣ t 2 ∣ ≤ C . Since a ~ j = λ ~ j 0 e 1 ( d + 1 ) + λ ~ j 1 u ~ g \widetilde{a}_j=\widetilde{\lambda}_{j0}e_1^{(d+1)}+\widetilde{\lambda}_{j1}\widetilde{u}_g a j = λ j 0 e 1 ( d + 1 ) + λ j 1 u g for items in group g g g , we have e 1 ( d + 1 ) = t 1 a ~ a g + t 2 a ~ b g e_1^{(d+1)}=t_1\widetilde{a}_{a_g}+t_2\widetilde{a}_{b_g} e 1 ( d + 1 ) = t 1 a a g + t 2 a b g . Therefore, we have
∥ Q e 1 ( d + 1 ) − ( t 1 a a g + t 2 a b g ) ∥ ≤ C exp { − c ( log p ) 2 } . \left\|Qe_1^{(d+1)}-(t_1a_{a_g}+t_2a_{b_g})\right\|\le C\exp\{-c(\log p)^2\}. Q e 1 ( d + 1 ) − ( t 1 a a g + t 2 a b g ) ≤ C exp { − c ( log p ) 2 } .
So Q e 1 ( d + 1 ) Qe_1^{(d+1)} Q e 1 ( d + 1 ) is within C exp { − c ( log p ) 2 } C\exp\{-c(\log p)^2\} C exp { − c ( log p ) 2 } of span ( e 1 ( d + 1 ) , u g ) \operatorname{span}(e_1^{(d+1)},u_g) span ( e 1 ( d + 1 ) , u g ) as a a g a_{a_g} a a g and a b g a_{b_g} a b g belongs to this space. Condition ( i i i ) (iii) ( iii ) gives 1 − ∣ u 1 T u 2 ∣ ≥ c > 0 1-|u_1^\mathsf{T}u_2|\ge c>0 1 − ∣ u 1 T u 2 ∣ ≥ c > 0 , where u 1 = Σ 1 / 2 e 2 ( d + 1 ) u_1=\Sigma^{1/2}e_2^{(d+1)} u 1 = Σ 1/2 e 2 ( d + 1 ) and u 2 = Σ 1 / 2 e 3 ( d + 1 ) u_2=\Sigma^{1/2}e_3^{(d+1)} u 2 = Σ 1/2 e 3 ( d + 1 ) . Since u 1 , u 2 u_1,u_2 u 1 , u 2 are unit vectors orthogonal to e 1 ( d + 1 ) e_1^{(d+1)} e 1 ( d + 1 ) according to the structure of Σ \Sigma Σ , the largest eigenvalue of u 1 u 1 T + u 2 u 2 T u_1u_1^\mathsf{T}+u_2u_2^\mathsf{T} u 1 u 1 T + u 2 u 2 T on this orthogonal complement is 1 + ∣ u 1 T u 2 ∣ 1+|u_1^\mathsf{T}u_2| 1 + ∣ u 1 T u 2 ∣ . Using ∥ Q e 1 ( d + 1 ) ∥ = 1 \|Qe_1^{(d+1)}\|=1 ∥ Q e 1 ( d + 1 ) ∥ = 1 and the preceding bounds for the two planes, we obtain
min s 0 ∈ { − 1 , 1 } ∥ Q e 1 ( d + 1 ) − s 0 e 1 ( d + 1 ) ∥ 2 = 2 [ 1 − { ( e 1 ( d + 1 ) ) T Q e 1 ( d + 1 ) } ] ≤ 2 [ 1 − { ( e 1 ( d + 1 ) ) T Q e 1 ( d + 1 ) } 2 ] ≤ 2 1 − ∣ u 1 T u 2 ∣ ∑ g = 1 2 [ 1 − { ( e 1 ( d + 1 ) ) T Q e 1 ( d + 1 ) } 2 − { u g T Q e 1 ( d + 1 ) } 2 ] ≤ C exp { − 2 c ( log p ) 2 } . \begin{aligned}
\min_{s_0\in\{-1,1\}}\left\|Qe_1^{(d+1)}-s_0e_1^{(d+1)}\right\|^2
&=2\left[1-\left\{(e_1^{(d+1)})^\mathsf{T}Qe_1^{(d+1)}\right\}\right]\\
&\le2\left[1-\left\{(e_1^{(d+1)})^\mathsf{T}Qe_1^{(d+1)}\right\}^2\right]\\
&\le\frac{2}{1-|u_1^\mathsf{T}u_2|}\sum_{g=1}^{2}\left[1-\left\{(e_1^{(d+1)})^\mathsf{T}Qe_1^{(d+1)}\right\}^2-\left\{u_g^\mathsf{T}Qe_1^{(d+1)}\right\}^2\right]\\
&\le C\exp\{-2c(\log p)^2\}.
\end{aligned} s 0 ∈ { − 1 , 1 } min Q e 1 ( d + 1 ) − s 0 e 1 ( d + 1 ) 2 = 2 [ 1 − { ( e 1 ( d + 1 ) ) T Q e 1 ( d + 1 ) } ] ≤ 2 [ 1 − { ( e 1 ( d + 1 ) ) T Q e 1 ( d + 1 ) } 2 ] ≤ 1 − ∣ u 1 T u 2 ∣ 2 g = 1 ∑ 2 [ 1 − { ( e 1 ( d + 1 ) ) T Q e 1 ( d + 1 ) } 2 − { u g T Q e 1 ( d + 1 ) } 2 ] ≤ C exp { − 2 c ( log p ) 2 } .
Here each summand is the squared distance from Q e 1 ( d + 1 ) Qe_1^{(d+1)} Q e 1 ( d + 1 ) to span ( e 1 ( d + 1 ) , u g ) \operatorname{span}(e_1^{(d+1)},u_g) span ( e 1 ( d + 1 ) , u g ) . Taking square roots gives ∥ Q e 1 ( d + 1 ) − s 0 e 1 ( d + 1 ) ∥ ≤ C exp { − c ( log p ) 2 } \|Qe_1^{(d+1)}-s_0e_1^{(d+1)}\|\le C\exp\{-c(\log p)^2\} ∥ Q e 1 ( d + 1 ) − s 0 e 1 ( d + 1 ) ∥ ≤ C exp { − c ( log p ) 2 } for some s 0 ∈ { − 1 , 1 } s_0\in\{-1,1\} s 0 ∈ { − 1 , 1 } . With ∥ a j − Q a ~ j ∥ ≤ C exp { − c ( log p ) 2 } \|a_j-Q\widetilde{a}_j\|\le C\exp\{-c(\log p)^2\} ∥ a j − Q a j ∥ ≤ C exp { − c ( log p ) 2 } , projecting the aligned loadings onto the general direction and its orthogonal complement therefore gives, uniformly in j j j , we have
∣ λ j 0 − s 0 λ ~ j 0 ∣ + ∥ λ j 1 u g ( j ) − λ ~ j 1 Q u ~ g ( j ) ∥ ≤ C exp { − c ( log p ) 2 } . |\lambda_{j0}-s_0\widetilde{\lambda}_{j0}|+\|\lambda_{j1}u_{g(j)}-\widetilde{\lambda}_{j1}Q\widetilde{u}_{g(j)}\|\le C\exp\{-c(\log p)^2\}. ∣ λ j 0 − s 0 λ j 0 ∣ + ∥ λ j 1 u g ( j ) − λ j 1 Q u g ( j ) ∥ ≤ C exp { − c ( log p ) 2 } .
By condition ( i v ) (iv) ( i v ) , each group has an anchor whose group loading is bounded away from zero. Choosing a sign s g s_g s g using this anchor gives ∥ u g − s g Q u ~ g ∥ ≤ C exp { − c ( log p ) 2 } \|u_g-s_gQ\widetilde{u}_g\|\le C\exp\{-c(\log p)^2\} ∥ u g − s g Q u g ∥ ≤ C exp { − c ( log p ) 2 } and ∣ λ j 1 − s g λ ~ j 1 ∣ ≤ C exp { − c ( log p ) 2 } |\lambda_{j1}-s_g\widetilde{\lambda}_{j1}|\le C\exp\{-c(\log p)^2\} ∣ λ j 1 − s g λ j 1 ∣ ≤ C exp { − c ( log p ) 2 } for every item in that group. With D = diag ( s 0 , s 1 , … , s d ) D=\operatorname{diag}(s_0,s_1,\ldots,s_d) D = diag ( s 0 , s 1 , … , s d ) , these direction bounds give ∥ Q Σ ~ 1 / 2 − Σ 1 / 2 D ∥ ≤ C exp { − c ( log p ) 2 } \|Q\widetilde{\Sigma}^{1/2}-\Sigma^{1/2}D\|\le C\exp\{-c(\log p)^2\} ∥ Q Σ 1/2 − Σ 1/2 D ∥ ≤ C exp { − c ( log p ) 2 } , which yields ∥ Σ ~ − D Σ D ∥ ≤ C exp { − c ( log p ) 2 } \|\widetilde{\Sigma}-D\Sigma D\|\le C\exp\{-c(\log p)^2\} ∥ Σ − D Σ D ∥ ≤ C exp { − c ( log p ) 2 } . By condition ( i i i ) (iii) ( iii ) , taking positive definite square roots preserves this order, so ∥ Σ ~ 1 / 2 − D Σ 1 / 2 D ∥ ≤ C exp { − c ( log p ) 2 } \|\widetilde{\Sigma}^{1/2}-D\Sigma^{1/2}D\|\le C\exp\{-c(\log p)^2\} ∥ Σ 1/2 − D Σ 1/2 D ∥ ≤ C exp { − c ( log p ) 2 } . Hence ∥ ( Q − D ) Σ ~ 1 / 2 ∥ ≤ C exp { − c ( log p ) 2 } \|(Q-D)\widetilde{\Sigma}^{1/2}\|\le C\exp\{-c(\log p)^2\} ∥ ( Q − D ) Σ 1/2 ∥ ≤ C exp { − c ( log p ) 2 } , and the uniform bound on ∥ Σ ~ − 1 / 2 ∥ \|\widetilde{\Sigma}^{-1/2}\| ∥ Σ − 1/2 ∥ gives
∥ Q − D ∥ ≤ C exp { − c ( log p ) 2 } . \|Q-D\|\le C\exp\{-c(\log p)^2\}. ∥ Q − D ∥ ≤ C exp { − c ( log p ) 2 } .
Combining these bounds with (B.37) proves the displayed bound in Example 2.
Proofs of Technical Lemmas
Throughout this appendix, we use the notation and conventions of Appendix § 7 .§ 3 .
Proof of Lemma B.1
By Assumption 2 ( i i ) 2(ii) 2 ( ii ) , there is some p 0 p_0 p 0 such that for any p ≥ p 0 p \ge p_0 p ≥ p 0 , ∑ j = 1 p α r j α r j T ⪰ κ p I d \sum_{j=1}^{p}\alpha_{rj}\alpha_{rj}^{\mathsf{T}} \succeq\kappa p I_d ∑ j = 1 p α r j α r j T ⪰ κ p I d for r ∈ { 0 , 1 } r \in\{0,1\} r ∈ { 0 , 1 } . Choose c 0 > 0 c_0 > 0 c 0 > 0 sufficiently small that c 0 M 2 ≤ κ / 2 c_0M^2 \le\kappa/2 c 0 M 2 ≤ κ /2 . So for any J ⊆ [ p ] \mathcal{J} \subseteq[p] J ⊆ [ p ] with ∣ J ∣ ≤ c 0 p |\mathcal{J}| \le c_0p ∣ J ∣ ≤ c 0 p , Assumption 2 ( i ) 2(i) 2 ( i ) gives
0 ⪯ ∑ j ∈ J α r j α r j T ⪯ ∣ J ∣ M 2 I d ⪯ ( κ p / 2 ) I d . 0 \preceq\sum_{j \in\mathcal{J}}\alpha_{rj}\alpha_{rj}^{\mathsf{T}} \preceq|\mathcal{J}|M^2I_d \preceq(\kappa p/2)I_d. 0 ⪯ j ∈ J ∑ α r j α r j T ⪯ ∣ J ∣ M 2 I d ⪯ ( κ p /2 ) I d .
This proves the first part.
For the reference sets, we show that they can be selected successively. Suppose that I 1 , … , I s − 1 \mathcal{I}_1,\ldots,\mathcal{I}_{s-1} I 1 , … , I s − 1 have already been chosen and let R s = [ p ] ∖ ( ⋃ t = 1 s − 1 I t ∪ T ) \mathcal{R}_s = [p] \setminus\left(\bigcup_{t=1}^{s-1}\mathcal{I}_t \cup\mathcal{T}\right) R s = [ p ] ∖ ( ⋃ t = 1 s − 1 I t ∪ T ) . Since ∣ T ∣ + ( s − 1 ) d ≤ N d + 2 ≤ c 0 p |\mathcal{T}|+(s-1)d \le Nd+2 \le c_0p ∣ T ∣ + ( s − 1 ) d ≤ N d + 2 ≤ c 0 p , Cauchy–Binet formula imply
∑ I ⊆ R s ∣ I ∣ = d det { ( A 0 ) I } 2 = det ( ∑ j ∈ R s α 0 j α 0 j T ) ≥ ( κ p / 2 ) d . \sum_{\substack{I\subseteq\mathcal{R}_s\\ |I|=d}}\det\{(A_0)_I\}^2
=\det\left(\sum_{j\in\mathcal{R}_s}\alpha_{0j}\alpha_{0j}^{\mathsf{T}}\right)
\ge(\kappa p/2)^d. I ⊆ R s ∣ I ∣ = d ∑ det {( A 0 ) I } 2 = det j ∈ R s ∑ α 0 j α 0 j T ≥ ( κ p /2 ) d .
There are at most ( p d ) ≤ p d / d ! \binom{p}{d} \le p^d/d! ( d p ) ≤ p d / d ! terms in this sum. Thus there exists some I s ⊆ R s \mathcal{I}_s \subseteq\mathcal{R}_s I s ⊆ R s with ∣ I s ∣ = d |\mathcal{I}_s|=d ∣ I s ∣ = d satisfies ∣ det { ( A 0 ) I s } ∣ ≥ ( d ! ) 1 / 2 ( κ / 2 ) d / 2 |\det\{(A_0)_{\mathcal{I}_s}\}| \ge(d!)^{1/2}(\kappa/2)^{d/2} ∣ det {( A 0 ) I s } ∣ ≥ ( d ! ) 1/2 ( κ /2 ) d /2 . Since ∥ B s ∥ ≤ ∥ B s ∥ F ≤ d 1 / 2 M \|B_s\| \le\|B_s\|_{\mathrm{F}} \le d^{1/2}M ∥ B s ∥ ≤ ∥ B s ∥ F ≤ d 1/2 M for B s = ( A 0 ) I s B_s=(A_0)_{\mathcal{I}_s} B s = ( A 0 ) I s , this gives σ min ( B s ) ≥ ∣ det ( B s ) ∣ / ∥ B s ∥ d − 1 ≥ ( d ! ) 1 / 2 ( κ / 2 ) d / 2 ( d 1 / 2 M ) 1 − d : = c 1 \sigma_{\min}(B_s) \ge|\det(B_s)|/\|B_s\|^{d-1} \ge(d!)^{1/2}(\kappa/2)^{d/2}(d^{1/2}M)^{1-d} := c_1 σ m i n ( B s ) ≥ ∣ det ( B s ) ∣/∥ B s ∥ d − 1 ≥ ( d ! ) 1/2 ( κ /2 ) d /2 ( d 1/2 M ) 1 − d := c 1 . In this way, we successively select the reference set with the required properties.
Proof of Lemma B.2
Fix either representation and suppress the index. We follow Proposition B.4(a) in Ostrovskii and Bach [43 ] to construct a regularized convex M-estimator to approximate Z Z Z from Y = X S c Y=X_{\mathcal{S}^{c}} Y = X S c . Specifically, let
Z ^ = arg min z ∈ R d 1 p ∑ j ∈ [ p ] ∖ S { ∫ 0 α j T z + β j f ( s ) d s − X j ( α j T z + β j ) } + ∥ z ∥ 2 2 p . \widehat{Z}=\underset{z\in\mathbb{R}^d}{\arg\min}\ \frac{1}{p}\sum_{j\in[p]\setminus\mathcal{S}}\left\{\int_{0}^{\alpha_j^{\mathsf{T}}z+\beta_j}f(s)\,ds-X_j(\alpha_j^{\mathsf{T}}z+\beta_j)\right\}+\frac{\|z\|^2}{2p}. Z = z ∈ R d arg min p 1 j ∈ [ p ] ∖ S ∑ { ∫ 0 α j T z + β j f ( s ) d s − X j ( α j T z + β j ) } + 2 p ∥ z ∥ 2 .
Then we establish the required bound by controlling Z − Z ^ Z-\widehat{Z} Z − Z and E ( Z − Z ^ ∣ Y ) \mathbb{E}(Z-\widehat{Z}\mid Y) E ( Z − Z ∣ Y ) . First, it is not hard to check that this yields a unique minimizer Z ^ \widehat{Z} Z . The scores of the estimating equation at Z Z Z and Z ^ \widehat{Z} Z are given by
S = 1 p ∑ j ∈ [ p ] ∖ S α j { X j − f ( α j T Z + β j ) } − Z p (47) S=\frac{1}{p}\sum_{j\in[p]\setminus\mathcal{S}}\alpha_j\{X_j-f(\alpha_j^{\mathsf{T}}Z+\beta_j)\}-\frac{Z}{p} \tag*{(47)} S = p 1 j ∈ [ p ] ∖ S ∑ α j { X j − f ( α j T Z + β j )} − p Z (47)
with S ^ = p − 1 ∑ j ∈ [ p ] ∖ S α j { X j − f ( α j T Z ^ + β j ) } − p − 1 Z ^ = 0 \widehat{S}=p^{-1}\sum_{j\in[p]\setminus\mathcal{S}}\alpha_j\{X_j-f(\alpha_j^{\mathsf{T}}\widehat{Z}+\beta_j)\}-p^{-1}\widehat{Z}=0 S = p − 1 ∑ j ∈ [ p ] ∖ S α j { X j − f ( α j T Z + β j )} − p − 1 Z = 0 . Let h = Z ^ − Z h=\widehat{Z}-Z h = Z − Z , and we have
h T S = h T ( S − S ^ ) = h T 1 p ∑ j ∈ [ p ] ∖ S α j { f ( α j T ( Z + h ) + β j ) − f ( α j T Z + β j ) } + ∥ h ∥ 2 p (48) h^{\mathsf{T}}S=h^{\mathsf{T}}(S-\widehat{S})=h^{\mathsf{T}}\frac{1}{p}\sum_{j\in[p]\setminus\mathcal{S}}\alpha_j\{f(\alpha_j^{\mathsf{T}}(Z+h)+\beta_j)-f(\alpha_j^{\mathsf{T}}Z+\beta_j)\}+\frac{\|h\|^2}{p} \tag*{(48)} h T S = h T ( S − S ) = h T p 1 j ∈ [ p ] ∖ S ∑ α j { f ( α j T ( Z + h ) + β j ) − f ( α j T Z + β j )} + p ∥ h ∥ 2 (48)
When h ≠ 0 h \ne0 h = 0 , by integral mean value theorem, the first term on the right side can be written as ( p ∥ h ∥ ) − 1 ∑ j ∈ [ p ] ∖ S ( α j T h ) 2 ∫ 0 ∥ h ∥ f ′ { α j T ( Z + s h / ∥ h ∥ ) + β j } d s (p\lVert h\rVert)^{-1}\sum_{j\in[p]\setminus S}(\alpha_j^{\mathsf T}h)^2\int_{0}^{\lVert h\rVert}f'\{\alpha_j^{\mathsf T}(Z+sh/\lVert h\rVert)+\beta_j\}\,ds ( p ∥ h ∥ ) − 1 ∑ j ∈ [ p ] ∖ S ( α j T h ) 2 ∫ 0 ∥ h ∥ f ′ { α j T ( Z + s h / ∥ h ∥) + β j } d s . On 0 ≤ s ≤ 1 ∧ ∥ h ∥ 0\le s\le1\land\lVert h\rVert 0 ≤ s ≤ 1 ∧ ∥ h ∥ , by Assumption 2(i) and (B.6), we know f ′ ( ⋅ ) ≥ c e − C ∥ Z ∥ f'(\cdot)\ge ce^{-C\lVert Z\rVert} f ′ ( ⋅ ) ≥ c e − C ∥ Z ∥ . With ∑ j ∈ [ p ] ∖ S α j α j T ⪰ ( κ / 2 ) p I d \sum_{j\in[p]\setminus S}\alpha_j\alpha_j^{\mathsf T}\succeq(\kappa/2)pI_d ∑ j ∈ [ p ] ∖ S α j α j T ⪰ ( κ /2 ) p I d by Lemma B.1, we know this part of integral is lower bounded by
c e − C ∥ Z ∥ ( κ / 2 ) ∥ h ∥ ( 1 ∧ ∥ h ∥ ) . ce^{-C\lVert Z\rVert}(\kappa/2)\lVert h\rVert(1\land\lVert h\rVert). c e − C ∥ Z ∥ ( κ /2 ) ∥ h ∥ ( 1 ∧ ∥ h ∥) .
Thus on the event E : = { ∥ S ∥ ≤ c e − C ∥ Z ∥ / 2 } \mathcal{E}:=\{\lVert S\rVert\le ce^{-C\lVert Z\rVert}/2\} E := {∥ S ∥ ≤ c e − C ∥ Z ∥ /2 } , we know from (C.2) that ∥ S ∥ ≥ c e − C ∥ Z ∥ ( 1 ∧ ∥ h ∥ ) \lVert S\rVert\ge ce^{-C\lVert Z\rVert}(1\land\lVert h\rVert) ∥ S ∥ ≥ c e − C ∥ Z ∥ ( 1 ∧ ∥ h ∥) , and therefore ∥ h ∥ ≤ 1 / 2 \lVert h\rVert\le1/2 ∥ h ∥ ≤ 1/2 and
∥ h ∥ ≤ C e C ∥ Z ∥ ∥ S ∥ . (49) \lVert h\rVert\le Ce^{C\lVert Z\rVert}\lVert S\rVert. \tag*{(49)} ∥ h ∥ ≤ C e C ∥ Z ∥ ∥ S ∥ . (49)
Apparently, (C.3) holds when h = 0 h=0 h = 0 . On the other side, on the complement event E c = { ∥ S ∥ > c e − C ∥ Z ∥ / 2 } \mathcal{E}^{c}=\{\lVert S\rVert>ce^{-C\lVert Z\rVert}/2\} E c = {∥ S ∥ > c e − C ∥ Z ∥ /2 } , we know by S ^ = 0 \widehat{S}=0 S = 0 that
∥ Z ^ ∥ = ∥ ∑ j ∈ [ p ] ∖ S α j { X j − f ( α j T Z ^ + β j ) } ∥ ≤ p M (50) \lVert\widehat{Z}\rVert=\left\lVert\sum_{j\in[p]\setminus S}\alpha_j\{X_j-f(\alpha_j^{\mathsf T}\widehat{Z}+\beta_j)\}\right\rVert\le pM \tag*{(50)} ∥ Z ∥ = j ∈ [ p ] ∖ S ∑ α j { X j − f ( α j T Z + β j )} ≤ pM (50)
by Assumption 2(i), which yields
∥ h ∥ ≤ p M + ∥ Z ∥ . (51) \lVert h\rVert\le pM+\lVert Z\rVert. \tag*{(51)} ∥ h ∥ ≤ pM + ∥ Z ∥ . (51)
Next, because each summand in ∑ j ∈ [ p ] ∖ S α j { X j − f ( α j T Z + β j ) } \sum_{j\in[p]\setminus S}\alpha_j\{X_j-f(\alpha_j^{\mathsf T}Z+\beta_j)\} ∑ j ∈ [ p ] ∖ S α j { X j − f ( α j T Z + β j )} is conditionally independent of each other with mean zero, for each component of S ‾ = S + Z / p \overline{S}=S+Z/p S = S + Z / p , denoted by S ‾ k \overline{S}_k S k for k ∈ [ d ] k\in[d] k ∈ [ d ] , one has by Hoeffding’s inequality that
E { e t S ‾ k ∣ Z = z } ≤ exp { t 2 8 p 2 ∑ j ∈ [ p ] ∖ S α j k 2 } ≤ e M 2 t 2 / ( 8 p ) . \mathbb{E}\{e^{t\overline{S}_k}\mid Z=z\}\le\exp\left\{\frac{t^2}{8p^2}\sum_{j\in[p]\setminus S}\alpha_{jk}^2\right\}\le e^{M^2t^2/(8p)}. E { e t S k ∣ Z = z } ≤ exp ⎩ ⎨ ⎧ 8 p 2 t 2 j ∈ [ p ] ∖ S ∑ α j k 2 ⎭ ⎬ ⎫ ≤ e M 2 t 2 / ( 8 p ) .
Thus for (C.1), Markov inequality yields E ( ∥ S ∥ k ∣ Z = z ) ≤ C k k k / 2 { p − k / 2 + p − k ∥ z ∥ k } \mathbb{E}(\lVert S\rVert^k\mid Z=z)\le C^k k^{k/2}\{p^{-k/2}+p^{-k}\lVert z\rVert^k\} E (∥ S ∥ k ∣ Z = z ) ≤ C k k k /2 { p − k /2 + p − k ∥ z ∥ k } for any k ≥ 2 k\ge2 k ≥ 2 . Then (C.3) gives
E ( ∥ h ∥ k 1 E ∣ Z = z ) ≤ e C k ∥ z ∥ C k k k / 2 { p − k / 2 + p − k ∥ z ∥ k } . \mathbb{E}(\lVert h\rVert^k\mathbf{1}_{\mathcal{E}}\mid Z=z)\le e^{Ck\lVert z\rVert}C^k k^{k/2}\{p^{-k/2}+p^{-k}\lVert z\rVert^k\}. E (∥ h ∥ k 1 E ∣ Z = z ) ≤ e C k ∥ z ∥ C k k k /2 { p − k /2 + p − k ∥ z ∥ k } .
For the complementary event, Markov’s inequality at order 3 k 3k 3 k gives P ( E c ∣ Z = z ) ≤ C 3 k ( 3 k ) 3 k / 2 e C k ∥ z ∥ { p − 3 k / 2 + p − 3 k ∥ z ∥ 3 k } \mathbb{P}(\mathcal{E}^c\mid Z=z)\le C^{3k}(3k)^{3k/2}e^{Ck\lVert z\rVert}\{p^{-3k/2}+p^{-3k}\lVert z\rVert^{3k}\} P ( E c ∣ Z = z ) ≤ C 3 k ( 3 k ) 3 k /2 e C k ∥ z ∥ { p − 3 k /2 + p − 3 k ∥ z ∥ 3 k } . Multiplying this bound by ( p M + ∥ z ∥ ) k (pM+\lVert z\rVert)^k ( pM + ∥ z ∥ ) k from (C.5) bounds its contribution to the k k k th moment. In particular, since p ≥ 1 p\ge1 p ≥ 1 and p M + ∥ z ∥ ≤ C p ( 1 + ∥ z ∥ ) pM+\lVert z\rVert\le Cp(1+\lVert z\rVert) pM + ∥ z ∥ ≤ C p ( 1 + ∥ z ∥) , the two event bounds together imply
E ( ∥ h ∥ k ∣ Z = z ) ≤ { C ( 3 k ) 1 / 2 } 3 k p − k / 2 e C k ∥ z ∥ ( 1 + ∥ z ∥ ) 4 k . \mathbb{E}(\lVert h\rVert^k\mid Z=z)\le\{C(3k)^{1/2}\}^{3k}p^{-k/2}e^{Ck\lVert z\rVert}(1+\lVert z\rVert)^{4k}. E (∥ h ∥ k ∣ Z = z ) ≤ { C ( 3 k ) 1/2 } 3 k p − k /2 e C k ∥ z ∥ ( 1 + ∥ z ∥ ) 4 k .
Here, the powers of p p p from the complementary event are p − k / 2 p^{-k/2} p − k /2 and p − 2 k p^{-2k} p − 2 k , both bounded by p − k / 2 p^{-k/2} p − k /2 .
Note that E ∥ Z ∥ k \mathbb{E}\lVert Z\rVert^{k} E ∥ Z ∥ k and E e C k 1 / 2 ∥ Z ∥ \mathbb{E}e^{Ck^{1/2}\lVert Z\rVert} E e C k 1/2 ∥ Z ∥ can be bounded by e C ′ k 2 e^{C'k^{2}} e C ′ k 2 by the sub-Gaussianity of Z Z Z . Therefore, with { C ( 3 k ) 1 / 2 } 3 k ≤ e C ′ k 2 \{C(3k)^{1/2}\}^{3k}\le e^{C'k^{2}} { C ( 3 k ) 1/2 } 3 k ≤ e C ′ k 2 , and ( 1 + x ) 4 k ≤ e 4 k x (1+x)^{4k}\le e^{4kx} ( 1 + x ) 4 k ≤ e 4 k x , e C k ∥ z ∥ ≤ e ν ∥ z ∥ 2 e C ′ k 2 e^{Ck\lVert z\rVert}\le e^{\nu\lVert z\rVert^{2}}e^{C'k^{2}} e C k ∥ z ∥ ≤ e ν ∥ z ∥ 2 e C ′ k 2 , we arrive at
E ∥ Z − Z ^ ∥ k ≤ e C k 2 p − k / 2 . (52) \mathbb{E}\lVert Z-\widehat Z\rVert^{k}\le e^{Ck^{2}}p^{-k/2}. \tag*{(52)} E ∥ Z − Z ∥ k ≤ e C k 2 p − k /2 . (52)
Finally, we note that V = ( Z − Z ^ ) − E ( Z − Z ^ ∣ Y ) V=(Z-\widehat Z)-\mathbb{E}(Z-\widehat Z\mid Y) V = ( Z − Z ) − E ( Z − Z ∣ Y ) as Z ^ \widehat Z Z is measurable with respect to Y Y Y . The triangle inequality and conditional Jensen’s inequality give ∥ V ∥ k ≤ 2 ∥ Z − Z ^ ∥ k \lVert V\rVert_{k}\le2\lVert Z-\widehat Z\rVert_{k} ∥ V ∥ k ≤ 2 ∥ Z − Z ∥ k , proving the first part in (B.11). Applying conditional Jensen’s inequality once more, with moment order k q ≥ 2 kq\ge2 k q ≥ 2 , yields
∥ E ( ∥ V ∥ k ∣ Y ) ∥ q ≤ ( E ∥ V ∥ k q ) 1 / q ≤ e C q k 2 p − k / 2 . \left\lVert\mathbb{E}(\lVert V\rVert^{k}\mid Y)\right\rVert_{q}\le\left(\mathbb{E}\lVert V\rVert^{kq}\right)^{1/q}\le e^{Cqk^{2}}p^{-k/2}. E (∥ V ∥ k ∣ Y ) q ≤ ( E ∥ V ∥ k q ) 1/ q ≤ e C q k 2 p − k /2 .
which completes the proof.
Proof of Lemma B.3
Fix either representation and suppress the index. For any i ∈ S i\in\mathcal{S} i ∈ S , by conditional independence, f ( T i ) = E { f ( α i T Z + β i ) ∣ Y } f(T_i)=\mathbb{E}\{f(\alpha_i^{\mathsf T}Z+\beta_i)\mid Y\} f ( T i ) = E { f ( α i T Z + β i ) ∣ Y } . We first bound T i T_i T i . By (B.6), the function φ ( x ) = f ( H − 1 log x ) \varphi(x)=f(H^{-1}\log x) φ ( x ) = f ( H − 1 log x ) is concave in x > 0 x>0 x > 0 . By Jensen’s inequality, we know f ( T i ) = E { φ ( e H ( α i T Z + β i ) ) ∣ Y } ≤ f [ H − 1 log E { e H ( α i T Z + β i ) ∣ Y } ] f(T_i)=\mathbb{E}\{\varphi(e^{H(\alpha_i^{\mathsf T}Z+\beta_i)})\mid Y\}\le f[H^{-1}\log\mathbb{E}\{e^{H(\alpha_i^{\mathsf T}Z+\beta_i)}\mid Y\}] f ( T i ) = E { φ ( e H ( α i T Z + β i ) ) ∣ Y } ≤ f [ H − 1 log E { e H ( α i T Z + β i ) ∣ Y }] . Because f f f is strictly increasing, we know
T i ≤ H − 1 log E ( e H ( α i T Z + β i ) ∣ Y ) . T_i\le H^{-1}\log\mathbb{E}(e^{H(\alpha_i^{\mathsf T}Z+\beta_i)}\mid Y). T i ≤ H − 1 log E ( e H ( α i T Z + β i ) ∣ Y ) .
A lower bound can be similarly obtained by considering function 1 − f ( − t ) 1-f(-t) 1 − f ( − t ) :
T i ≥ − H − 1 log E ( e − H ( α i T Z + β i ) ∣ Y ) . T_i\ge-H^{-1}\log\mathbb{E}(e^{-H(\alpha_i^{\mathsf T}Z+\beta_i)}\mid Y). T i ≥ − H − 1 log E ( e − H ( α i T Z + β i ) ∣ Y ) .
With δ i = E [ α i T Z + β i ∣ Y ] − T i = H − 1 log { e H E [ α i T Z + β i ∣ Y ] } − T i \delta_i=\mathbb{E}[\alpha_i^{\mathsf T}Z+\beta_i\mid Y]-T_i=H^{-1}\log\{e^{H\mathbb{E}[\alpha_i^{\mathsf T}Z+\beta_i\mid Y]}\}-T_i δ i = E [ α i T Z + β i ∣ Y ] − T i = H − 1 log { e H E [ α i T Z + β i ∣ Y ] } − T i and V = Z − E ( Z ∣ Y ) V=Z-\mathbb{E}(Z\mid Y) V = Z − E ( Z ∣ Y ) , we know
− H − 1 log E ( e H ( α i T V ) ∣ Y ) ≤ δ i ≤ H − 1 log E ( e − H ( α i T V ) ∣ Y ) . -H^{-1}\log\mathbb{E}(e^{H(\alpha_i^{\mathsf T}V)}\mid Y)\le\delta_i\le H^{-1}\log\mathbb{E}(e^{-H(\alpha_i^{\mathsf T}V)}\mid Y). − H − 1 log E ( e H ( α i T V ) ∣ Y ) ≤ δ i ≤ H − 1 log E ( e − H ( α i T V ) ∣ Y ) .
By e x ≤ 1 + x + x 2 e ∣ x ∣ / 2 e^{x}\le1+x+x^{2}e^{|x|}/2 e x ≤ 1 + x + x 2 e ∣ x ∣ /2 , log x ≤ x − 1 \log x\le x-1 log x ≤ x − 1 and E [ V ∣ Y ] = 0 \mathbb{E}[V\mid Y]=0 E [ V ∣ Y ] = 0 , we have
∣ δ i ∣ ≤ H 2 E { ( α i T V ) 2 e H ∣ α i T V ∣ ∣ Y } ≤ H M 2 2 E { ∥ V ∥ 2 e H M ∥ V ∥ ∣ Y } . (53) |\delta_i|\le\frac{H}{2}\mathbb{E}\{(\alpha_i^{\mathsf T}V)^2e^{H|\alpha_i^{\mathsf T}V|}\mid Y\}\le\frac{HM^{2}}{2}\mathbb{E}\{\lVert V\rVert^{2}e^{HM\lVert V\rVert}\mid Y\}. \tag*{(53)} ∣ δ i ∣ ≤ 2 H E {( α i T V ) 2 e H ∣ α i T V ∣ ∣ Y } ≤ 2 H M 2 E {∥ V ∥ 2 e H M ∥ V ∥ ∣ Y } . (53)
For e H M ∥ V ∥ e^{HM\lVert V\rVert} e H M ∥ V ∥ , by Cauchy’s inequality and Jensen’s inequality, one has
E e t ∥ V ∥ ≤ E [ e t ∥ Z ∥ e t ∥ E ( Z ∣ Y ) ∥ ] ≤ ( E e 2 t ∥ Z ∥ ) 1 / 2 ( E e 2 t ∥ E ( Z ∣ Y ) ∥ ) 1 / 2 ≤ E e 2 t ∥ Z ∥ ≤ L e t 2 / ν , \mathbb{E}e^{t\lVert V\rVert}\le\mathbb{E}\left[e^{t\lVert Z\rVert}e^{t\lVert\mathbb{E}(Z\mid Y)\rVert}\right]\le(\mathbb{E}e^{2t\lVert Z\rVert})^{1/2}(\mathbb{E}e^{2t\lVert\mathbb{E}(Z\mid Y)\rVert})^{1/2}\le\mathbb{E}e^{2t\lVert Z\rVert}\le Le^{t^{2}/\nu}, E e t ∥ V ∥ ≤ E [ e t ∥ Z ∥ e t ∥ E ( Z ∣ Y )∥ ] ≤ ( E e 2 t ∥ Z ∥ ) 1/2 ( E e 2 t ∥ E ( Z ∣ Y )∥ ) 1/2 ≤ E e 2 t ∥ Z ∥ ≤ L e t 2 / ν ,
where the last inequality follows from 2 t ∥ Z ∥ ≤ ν ∥ Z ∥ 2 + t 2 / ν 2t\lVert Z\rVert\le\nu\lVert Z\rVert^{2}+t^{2}/\nu 2 t ∥ Z ∥ ≤ ν ∥ Z ∥ 2 + t 2 / ν and the sub-Gaussianity of Z Z Z . Then with ( E ∥ V ∥ 4 q ) 1 / ( 2 q ) ≤ e C q p − 1 (\mathbb{E}\lVert V\rVert^{4q})^{1/(2q)}\le e^{Cq}p^{-1} ( E ∥ V ∥ 4 q ) 1/ ( 2 q ) ≤ e C q p − 1 by Lemma B.2 , we conclude with ∥ E ( ⋅ ∣ Y ) ∥ q ≤ ∥ ⋅ ∥ q \lVert\mathbb{E}(\cdot\mid Y)\rVert_q\le\lVert\cdot\rVert_q ∥ E ( ⋅ ∣ Y ) ∥ q ≤ ∥ ⋅ ∥ q and Hölder’s inequality that
∥ δ i ∥ q ≤ H M 2 2 ( E ∥ V ∥ 4 q ) 1 / ( 2 q ) ( E e 2 q H M ∥ V ∥ ) 1 / ( 2 q ) ≤ e C q p − 1 . \lVert\delta_i\rVert_q \le\frac{HM^2}{2}\left(\mathbb{E}\lVert V\rVert^{4q}\right)^{1/(2q)}\left(\mathbb{E}e^{2qHM\lVert V\rVert}\right)^{1/(2q)} \le e^{Cq}p^{-1}. ∥ δ i ∥ q ≤ 2 H M 2 ( E ∥ V ∥ 4 q ) 1/ ( 2 q ) ( E e 2 q H M ∥ V ∥ ) 1/ ( 2 q ) ≤ e C q p − 1 .
This proves the first part of Lemma B.3 .
For the second part, it suffices to show that both E ( e t ∣ α i T Z + β i ∣ ) \mathbb{E}(e^{t|\alpha_i^{\mathsf{T}}Z+\beta_i|}) E ( e t ∣ α i T Z + β i ∣ ) and E [ e t ∣ T i ∣ ] \mathbb{E}[e^{t|T_i|}] E [ e t ∣ T i ∣ ] can be bounded by e C ( 1 + t 2 ) e^{C(1+t^2)} e C ( 1 + t 2 ) , as Cauchy’s inequality and Jensen’s inequality yield
E e t ∣ δ i ∣ ≤ ( E e 2 t ∣ E [ α i T Z + β i ∣ Y ] ∣ ) 1 / 2 ( E e 2 t ∣ T i ∣ ) 1 / 2 ≤ ( E e 2 t ∣ α i T Z + β i ∣ ) 1 / 2 ( E e 2 t ∣ T i ∣ ) 1 / 2 . \mathbb{E}e^{t|\delta_i|} \le\left(\mathbb{E}e^{2t|\mathbb{E}[\alpha_i^{\mathsf{T}}Z+\beta_i\mid Y]|}\right)^{1/2}\left(\mathbb{E}e^{2t|T_i|}\right)^{1/2} \le\left(\mathbb{E}e^{2t|\alpha_i^{\mathsf{T}}Z+\beta_i|}\right)^{1/2}\left(\mathbb{E}e^{2t|T_i|}\right)^{1/2}. E e t ∣ δ i ∣ ≤ ( E e 2 t ∣ E [ α i T Z + β i ∣ Y ] ∣ ) 1/2 ( E e 2 t ∣ T i ∣ ) 1/2 ≤ ( E e 2 t ∣ α i T Z + β i ∣ ) 1/2 ( E e 2 t ∣ T i ∣ ) 1/2 .
For E ( e t ∣ α i T Z + β i ∣ ) \mathbb{E}(e^{t|\alpha_i^{\mathsf{T}}Z+\beta_i|}) E ( e t ∣ α i T Z + β i ∣ ) , by Assumptions 2, it is easy to check that it can be bounded by e C ( 1 + t 2 ) e^{C(1+t^2)} e C ( 1 + t 2 ) . For E [ e t ∣ T i ∣ ] \mathbb{E}[e^{t|T_i|}] E [ e t ∣ T i ∣ ] , when t ≥ H t \ge H t ≥ H , we note by Jensen’s inequality that, for the convex map x ↦ x H − 1 t x \mapsto x^{H^{-1}t} x ↦ x H − 1 t , there is
E e t ∣ T i ∣ ≤ E e t T i + E e − t T i ≤ E { E ( e H ( α i T Z + β i ) ∣ Y ) } t / H + E { E ( e − H ( α i T Z + β i ) ∣ Y ) } t / H ≤ E e t ( α i T Z + β i ) + E e − t ( α i T Z + β i ) ≤ e C ( 1 + t 2 ) . \begin{aligned}
\mathbb{E}e^{t|T_i|} \le\mathbb{E}e^{tT_i}+\mathbb{E}e^{-tT_i} \le\mathbb{E}\{\mathbb{E}(e^{H(\alpha_i^{\mathsf{T}}Z+\beta_i)}\mid Y)\}^{t/H}+\mathbb{E}\{\mathbb{E}(e^{-H(\alpha_i^{\mathsf{T}}Z+\beta_i)}\mid Y)\}^{t/H} \\
&\le\mathbb{E}e^{t(\alpha_i^{\mathsf{T}}Z+\beta_i)}+\mathbb{E}e^{-t(\alpha_i^{\mathsf{T}}Z+\beta_i)} \le e^{C(1+t^2)}.
\end{aligned} E e t ∣ T i ∣ ≤ E e t T i + E e − t T i ≤ E { E ( e H ( α i T Z + β i ) ∣ Y ) } t / H + E { E ( e − H ( α i T Z + β i ) ∣ Y ) } t / H ≤ E e t ( α i T Z + β i ) + E e − t ( α i T Z + β i ) ≤ e C ( 1 + t 2 ) .
When t < H t < H t < H , E [ e t ∣ T i ∣ ] ≤ E [ e H ∣ T i ∣ ] ≤ e C ( 1 + H 2 ) \mathbb{E}[e^{t|T_i|}] \le\mathbb{E}[e^{H|T_i|}] \le e^{C(1+H^2)} E [ e t ∣ T i ∣ ] ≤ E [ e H ∣ T i ∣ ] ≤ e C ( 1 + H 2 ) . Therefore we complete the second part of Lemma B.3 .
Proof of Lemma B.4
Proof. Since the polynomials F j F_j F j and P j P_j P j share the formal arguments ( v , s , m ) (v,s,m) ( v , s , m ) , we suppress in this proof for simplicity. It suffices to prove (B.18) for P r P_r P r defined by setting P 0 = 1 P_0=1 P 0 = 1 and P j = [ P j − 1 F j ] N P_j=[P_{j-1}F_j]_N P j = [ P j − 1 F j ] N for j = 1 , … , r j=1,\ldots,r j = 1 , … , r . By this definition, we have
∏ j = 1 r x j − ev ( P r ) = ∑ j = 1 r { ev ( P j − 1 ) x j − ev ( P j ) } ∏ l > j x l = ∑ j = 1 r [ ev ( P j − 1 ) { x j − ev ( F j ) } ∏ l > j x l + ev { P j − 1 F j − [ P j − 1 F j ] N } ∏ l > j x l ] . (54) \begin{aligned}
\prod_{j=1}^{r}x_j-\operatorname{ev}(P_r)=\sum_{j=1}^{r}\{\operatorname{ev}(P_{j-1})x_j-\operatorname{ev}(P_j)\}\prod_{l>j}x_l \\
&=\sum_{j=1}^{r}\left[\operatorname{ev}(P_{j-1})\{x_j-\operatorname{ev}(F_j)\}\prod_{l>j}x_l+\operatorname{ev}\{P_{j-1}F_j-[P_{j-1}F_j]_N\}\prod_{l>j}x_l\right]. \tag*{(54)}
\end{aligned} j = 1 ∏ r x j − ev ( P r ) = j = 1 ∑ r { ev ( P j − 1 ) x j − ev ( P j )} l > j ∏ x l = j = 1 ∑ r ev ( P j − 1 ) { x j − ev ( F j )} l > j ∏ x l + ev { P j − 1 F j − [ P j − 1 F j ] N } l > j ∏ x l . (54)
To bound γ j 1 \gamma_{j1} γ j 1 , we first control ev ( P j − 1 ) \operatorname{ev}(P_{j-1}) ev ( P j − 1 ) by showing ∥ P j − 1 ∥ c o e f f ≤ e C N \lVert P_{j-1}\rVert_{\mathrm{coeff}}\le e^{CN} ∥ P j − 1 ∥ coeff ≤ e C N . Note that P j P_j P j is defined inductively by multiplication and truncation at N N N , and thus for any j ≤ w ≤ N j\le w\le N j ≤ w ≤ N , we have
∥ [ P j ] w − [ P j ] w − 1 ∥ c o e f f ≤ ∑ a 1 + ⋯ + a j = w a 1 , … , a j ≥ 1 ∏ i = 1 j ∥ [ F i ] a i − [ F i ] a i − 1 ∥ c o e f f ≤ C w ( w − 1 j − 1 ) ≤ ( 2 C ) w , \lVert[P_j]_w-[P_j]_{w-1}\rVert_{\mathrm{coeff}}\le\sum_{\substack{a_1+\cdots+a_j=w\\a_1,\ldots,a_j\ge1}}\prod_{i=1}^{j}\lVert[F_i]_{a_i}-[F_i]_{a_i-1}\rVert_{\mathrm{coeff}}\le C^w\binom{w-1}{j-1}\le(2C)^w, ∥[ P j ] w − [ P j ] w − 1 ∥ coeff ≤ a 1 + ⋯ + a j = w a 1 , … , a j ≥ 1 ∑ i = 1 ∏ j ∥[ F i ] a i − [ F i ] a i − 1 ∥ coeff ≤ C w ( j − 1 w − 1 ) ≤ ( 2 C ) w ,
by (B.17). Then ∥ P j − 1 ∥ c o e f f ≤ ∑ w = j − 1 N ( 2 C ) w ≤ e C N \lVert P_{j-1}\rVert_{\mathrm{coeff}}\le\sum_{w=j-1}^{N}(2C)^w\le e^{CN} ∥ P j − 1 ∥ coeff ≤ ∑ w = j − 1 N ( 2 C ) w ≤ e C N and ∥ F j ∥ c o e f f ≤ ∑ w = w j N C w ≤ e C N \lVert F_j\rVert_{\mathrm{coeff}}\le\sum_{w=w_j}^{N}C^w\le e^{CN} ∥ F j ∥ coeff ≤ ∑ w = w j N C w ≤ e C N . With (B.14), we know ∥ ev ( P j − 1 ) ∥ a ≤ ∥ P j − 1 ∥ c o e f f e C ( 1 + a ) N 2 ≤ e C ( 1 + a ) N 2 \lVert\operatorname{ev}(P_{j-1})\rVert_a\le\lVert P_{j-1}\rVert_{\mathrm{coeff}}e^{C(1+a)N^2}\le e^{C(1+a)N^2} ∥ ev ( P j − 1 ) ∥ a ≤ ∥ P j − 1 ∥ coeff e C ( 1 + a ) N 2 ≤ e C ( 1 + a ) N 2 . For ∏ l > j x l \prod_{l>j}x_l ∏ l > j x l , we know by $\lVert x_j\rVert_q\le e C ( 1 + q ) w j 2 p − w j / 2 e^{C(1+q)w_j^2}p^{-w_j/2} e C ( 1 + q ) w j 2 p − w j /2 and Hölder inequality that for any q ≥ 1 q \ge1 q ≥ 1 ,
∥ ∏ l > j x l ∥ q ≤ ∏ l > j ∥ x l ∥ q ( ∑ k > j w k ) / w l ≤ e C ( 1 + q ) ( ∑ l > j w l ) 2 p − 1 2 ∑ l > j w l ≤ e C ( 1 + q ) N 2 . \left\|\prod_{l>j}x_l\right\|_q \le\prod_{l>j}\|x_l\|_{q\left(\sum_{k>j}w_k\right)/w_l} \le e^{C(1+q)\left(\sum_{l>j}w_l\right)^2}p^{-\frac{1}{2}\sum_{l>j}w_l} \le e^{C(1+q)N^2}. l > j ∏ x l q ≤ l > j ∏ ∥ x l ∥ q ( ∑ k > j w k ) / w l ≤ e C ( 1 + q ) ( ∑ l > j w l ) 2 p − 2 1 ∑ l > j w l ≤ e C ( 1 + q ) N 2 .
Therefore, for γ j 1 \gamma_{j1} γ j 1 , we have by (B.16) that
∥ γ j 1 ∥ q ≤ ∥ ev ( P j − 1 ) ∥ 3 q ∥ x j − ev ( F j ) ∥ 3 q ∥ ∏ l > j x l ∥ 3 q ≤ e C ( 1 + 3 q ) N 2 { e C ( 1 + 3 q ) N 2 p − ( N + 1 ) / 2 } e C ( 1 + 3 q ) N 2 ≤ e C ′ ( 1 + q ) N 2 p − ( N + 1 ) / 2 . \begin{aligned}
\|\gamma_{j1}\|_q \le\|\operatorname{ev}(P_{j-1})\|_{3q}\|x_j-\operatorname{ev}(F_j)\|_{3q}\left\|\prod_{l>j}x_l\right\|_{3q} \\
&\le e^{C(1+3q)N^2}\left\{e^{C(1+3q)N^2}p^{-(N+1)/2}\right\}e^{C(1+3q)N^2} \\
&\le e^{C'(1+q)N^2}p^{-(N+1)/2}.
\end{aligned} ∥ γ j 1 ∥ q ≤ ∥ ev ( P j − 1 ) ∥ 3 q ∥ x j − ev ( F j ) ∥ 3 q l > j ∏ x l 3 q ≤ e C ( 1 + 3 q ) N 2 { e C ( 1 + 3 q ) N 2 p − ( N + 1 ) /2 } e C ( 1 + 3 q ) N 2 ≤ e C ′ ( 1 + q ) N 2 p − ( N + 1 ) /2 .
Next, we bound γ j 2 \gamma_{j2} γ j 2 . Because ∥ P j − 1 F j − [ P j − 1 F j ] N ∥ c o e f f ≤ ∥ P j − 1 F j ∥ c o e f f ≤ ∥ P j − 1 ∥ c o e f f ∥ F j ∥ c o e f f ≤ e C N \|P_{j-1}F_j-[P_{j-1}F_j]_N\|_{\mathrm{coeff}} \le\|P_{j-1}F_j\|_{\mathrm{coeff}} \le\|P_{j-1}\|_{\mathrm{coeff}}\|F_j\|_{\mathrm{coeff}} \le e^{CN} ∥ P j − 1 F j − [ P j − 1 F j ] N ∥ coeff ≤ ∥ P j − 1 F j ∥ coeff ≤ ∥ P j − 1 ∥ coeff ∥ F j ∥ coeff ≤ e C N , we can apply (B.15) to get ∥ ev { P j − 1 F j − [ P j − 1 F j ] N } ∥ 2 q ≤ e C ( 1 + 2 q ) N 2 p − ( N + 1 ) / 2 \|\operatorname{ev}\{P_{j-1}F_j-[P_{j-1}F_j]_N\}\|_{2q} \le e^{C(1+2q)N^2}p^{-(N+1)/2} ∥ ev { P j − 1 F j − [ P j − 1 F j ] N } ∥ 2 q ≤ e C ( 1 + 2 q ) N 2 p − ( N + 1 ) /2 . Together with ∥ ∏ l > j x l ∥ 2 q ≤ e C ( 1 + 2 q ) N 2 \left\|\prod_{l>j}x_l\right\|_{2q} \le e^{C(1+2q)N^2} ∏ l > j x l 2 q ≤ e C ( 1 + 2 q ) N 2 and the Cauchy inequality, we arrive at
∥ γ j 2 ∥ q ≤ ∥ ev { P j − 1 F j − [ P j − 1 F j ] N } ∥ 2 q ∥ ∏ l > j x l ∥ 2 q ≤ e C ( 1 + q ) N 2 p − ( N + 1 ) / 2 . \|\gamma_{j2}\|_q \le\|\operatorname{ev}\{P_{j-1}F_j-[P_{j-1}F_j]_N\}\|_{2q}\left\|\prod_{l>j}x_l\right\|_{2q} \le e^{C(1+q)N^2}p^{-(N+1)/2}. ∥ γ j 2 ∥ q ≤ ∥ ev { P j − 1 F j − [ P j − 1 F j ] N } ∥ 2 q l > j ∏ x l 2 q ≤ e C ( 1 + q ) N 2 p − ( N + 1 ) /2 .
Combine the upper bounds for γ j 1 \gamma_{j1} γ j 1 and γ j 2 \gamma_{j2} γ j 2 in (C.8), and using r ≤ N r \le N r ≤ N , yields (B.18). Similar calculation follows for outer polynomial R ( x 1 , … , x r ) R(x_1,\ldots,x_r) R ( x 1 , … , x r ) to show (B.19).
Proof of Lemma B.5
Proof. Fix either representation and suppress its index. Table 1 summarizes the formal expansions used throughout this proof. Write c k ( t ) = f ( k ) ( t ) / { k ! f ′ ( t ) } c_k(t)=f^{(k)}(t)/\{k!f'(t)\} c k ( t ) = f ( k ) ( t ) / { k ! f ′ ( t )} . The expressions F i ( s , m ) \mathcal{F}_i(s,m) F i ( s , m ) and H i ( m ) H_i(m) H i ( m ) are formal power series; their finite truncations are polynomials.
Table 1. Formal expansions used in the proof of Lemma B.5.
The series F i ( s , m ) \mathcal{F}_i(s,m) F i ( s , m ) provides approximation to the conditional-mean equation used to determine H i ( m ) H_i(m) H i ( m ) so that G i , N ( v , m ) G_{i,N}(v,m) G i , N ( v , m ) replaces s s s with H ( m ) H(m) H ( m ) in P i , N ( v , s ) P_{i,N}(v,s) P i , N ( v , s ) . Both P i , N ( v , s ) P_{i,N}(v,s) P i , N ( v , s ) and G i , N ( v , m ) G_{i,N}(v,m) G i , N ( v , m ) approximate g T i ( α i T V + δ i ) g_{T_i}(\alpha_i^{\mathsf T}V+\delta_i) g T i ( α i T V + δ i ) in the evaluated value. We first bound the coefficients obtained by recursively solving H i ( m ) H_i(m) H i ( m ) from (B.21). Write b w = ∥ [ H i ( m ) ] w − [ H i ( m ) ] w − 1 ∥ c o e f f b_w=\lVert[H_i(m)]_w-[H_i(m)]_{w-1}\rVert_{\mathrm{coeff}} b w = ∥[ H i ( m ) ] w − [ H i ( m ) ] w − 1 ∥ coeff . For g t ( u ) g_t(u) g t ( u ) , by Assumption 1(ii), we know ∣ c k ( t ) ∣ ≤ K k − 1 |c_k(t)|\le K^{k-1} ∣ c k ( t ) ∣ ≤ K k − 1 and we have by the integral form of remainder that
∣ g t ( u ) − ∑ k = 1 N c k ( t ) u k ∣ ≤ ∣ u ∣ N + 1 N ! f ′ ( t ) ∫ 0 1 ( 1 − s ) N ∣ f ( N + 1 ) ( t + s u ) ∣ d s ≤ ( 1 ) ∣ u ∣ N + 1 N ! ∫ 0 1 ∣ 1 − s ∣ N ( N + 1 ) ! K N f ′ ( t + s u ) f ′ ( t ) d s ≤ ( 2 ) ∣ u ∣ N + 1 K N e H ∣ u ∣ (55) \begin{aligned}
\left|g_t(u)-\sum_{k=1}^{N}c_k(t)u^k\right|&\le\frac{|u|^{N+1}}{N!f'(t)}\int_0^1(1-s)^N|f^{(N+1)}(t+su)|\,\mathrm{d}s\\
&\underset{(1)}{\le}\frac{|u|^{N+1}}{N!}\int_0^1|1-s|^N\frac{(N+1)!K^Nf'(t+su)}{f'(t)}\,\mathrm{d}s\\
&\underset{(2)}{\le}|u|^{N+1}K^Ne^{H|u|} \tag*{(55)}
\end{aligned} g t ( u ) − k = 1 ∑ N c k ( t ) u k ≤ N ! f ′ ( t ) ∣ u ∣ N + 1 ∫ 0 1 ( 1 − s ) N ∣ f ( N + 1 ) ( t + s u ) ∣ d s ( 1 ) ≤ N ! ∣ u ∣ N + 1 ∫ 0 1 ∣1 − s ∣ N f ′ ( t ) ( N + 1 )! K N f ′ ( t + s u ) d s ( 2 ) ≤ ∣ u ∣ N + 1 K N e H ∣ u ∣ (55)
Here, (1) follows from Assumption 1(ii); (2) follows from (B.6). Since c 1 = 1 c_1=1 c 1 = 1 , we can write (B.21) with s = H i ( m ) s=H_i(m) s = H i ( m ) as
H i ( m ) = − ∑ k ≥ 2 c k ( T i ) E f [ { α i T v + H i ( m ) } k ] . (56) H_i(m)=-\sum_{k\ge2}c_k(T_i)\mathbb{E}_{\mathrm{f}}\left[\left\{\alpha_i^{\mathsf T}v+H_i(m)\right\}^k\right]. \tag*{(56)} H i ( m ) = − k ≥ 2 ∑ c k ( T i ) E f [ { α i T v + H i ( m ) } k ] . (56)
To bound the coefficients of H i ( m ) H_i(m) H i ( m ) , we use a majorant argument that records coefficient norms by weight. Introduce a scalar variable t t t , and let [ t w ] [t^w] [ t w ] extract the coefficient of t w t^w t w in a scalar series. A moment symbol m γ m_\gamma m γ is recorded by t ∣ γ ∣ t^{|\gamma|} t ∣ γ ∣ , and the coefficient at t w t^w t w is denoted by [ t w ] t ∣ γ ∣ [t^w]t^{|\gamma|} [ t w ] t ∣ γ ∣ , which equals one when w = ∣ γ ∣ w=|\gamma| w = ∣ γ ∣ and zero otherwise. For H i ( m ) H_i(m) H i ( m ) , the corresponding scalar series is ∑ ℓ ≥ 2 ∥ [ H i ] ℓ − [ H i ] ℓ − 1 ∥ c o e f f t ℓ = ∑ ℓ ≥ 2 b ℓ t ℓ \sum_{\ell\ge2}\lVert[H_i]_\ell-[H_i]_{\ell-1}\rVert_{\mathrm{coeff}}t^\ell=\sum_{\ell\ge2}b_\ell t^\ell ∑ ℓ ≥ 2 ∥[ H i ] ℓ − [ H i ] ℓ − 1 ∥ coeff t ℓ = ∑ ℓ ≥ 2 b ℓ t ℓ , so b w = [ t w ] ∑ ℓ ≥ 2 b ℓ t ℓ b_w=[t^w]\sum_{\ell\ge2}b_\ell t^\ell b w = [ t w ] ∑ ℓ ≥ 2 b ℓ t ℓ . Applying the coefficient norm inequalities to (C.10) yields recursive bounds for b w b_w b w . The majorant argument then constructs a scalar series h ( t ) = ∑ w ≥ 2 h w t w h(t)=\sum_{w\ge2}h_wt^w h ( t ) = ∑ w ≥ 2 h w t w with nonnegative coefficients satisfying b w ≤ h w b_w\le h_w b w ≤ h w . Showing that h h h is analytic near the origin allows us to apply Cauchy’s coefficient estimate [57 Chapter 2, Corollary 4.3] , giving b w ≤ h w ≤ C w b_w\le h_w\le C^w b w ≤ h w ≤ C w .
In particular, equating the weight-w w w components on both sides of (C.10), we obtain
b w ≤ [ t w ] ∑ k = 2 w sup T i ∣ c k ( T i ) ∣ { ∥ α i T v ∥ c o e f f + ∑ ℓ = 2 w − 1 b ℓ t ℓ } k . b_w\le[t^w]\sum_{k=2}^{w}\sup_{T_i}|c_k(T_i)|\left\{\lVert\alpha_i^{\mathsf T}v\rVert_{\mathrm{coeff}}+\sum_{\ell=2}^{w-1}b_\ell t^\ell\right\}^k. b w ≤ [ t w ] k = 2 ∑ w T i sup ∣ c k ( T i ) ∣ { ∥ α i T v ∥ coeff + ℓ = 2 ∑ w − 1 b ℓ t ℓ } k .
By Assumption 2, we have ∥ α i T v ∥ c o e f f = ∑ ℓ = 1 d ∣ α i ℓ ∣ ≤ C 0 \lVert\alpha_i^{\mathsf T}v\rVert_{\mathrm{coeff}}=\sum_{\ell=1}^{d}|\alpha_{i\ell}|\le C_0 ∥ α i T v ∥ coeff = ∑ ℓ = 1 d ∣ α i ℓ ∣ ≤ C 0 . Together with sup T i ∣ c k ( T i ) ∣ ≤ K k − 1 \sup_{T_i}|c_k(T_i)|\le K^{k-1} sup T i ∣ c k ( T i ) ∣ ≤ K k − 1 , we have for a suitable constant C ≥ K ( C 0 ∨ 1 ) C\ge K(C_0\vee1) C ≥ K ( C 0 ∨ 1 ) that
b w ≤ [ t w ] ∑ k = 2 w C k ( t + ∑ ℓ = 2 w − 1 b ℓ t ℓ ) k . b_w\le[t^w]\sum_{k=2}^{w}C^k\left(t+\sum_{\ell=2}^{w-1}b_\ell t^\ell\right)^k. b w ≤ [ t w ] k = 2 ∑ w C k ( t + ℓ = 2 ∑ w − 1 b ℓ t ℓ ) k .
To derive an explicit bound for b w b_w b w , we introduce a series h ( t ) = ∑ w ≥ 2 h w t w h(t)=\sum_{w\ge2}h_wt^w h ( t ) = ∑ w ≥ 2 h w t w satisfying
h ( t ) = ∑ k ≥ 2 C k { t + h ( t ) } k = C 2 { t + h ( t ) } 2 1 − C { t + h ( t ) } = : H ( t , h ) . h(t)=\sum_{k\ge2}C^k\{t+h(t)\}^k=\frac{C^2\{t+h(t)\}^2}{1-C\{t+h(t)\}}=:H(t,h). h ( t ) = k ≥ 2 ∑ C k { t + h ( t ) } k = 1 − C { t + h ( t )} C 2 { t + h ( t ) } 2 =: H ( t , h ) .
Since H ( 0 , 0 ) = 0 H(0,0)=0 H ( 0 , 0 ) = 0 and ∂ h H ( t , h ) ∣ ( t , h ) = ( 0 , 0 ) = 0 \left.\partial_h H(t,h)\right|_{(t,h)=(0,0)}=0 ∂ h H ( t , h ) ∣ ( t , h ) = ( 0 , 0 ) = 0 , implicit function theorem implies that this equation admits some analytic solution with h ( 0 ) = h ′ ( 0 ) = 0 h(0)=h'(0)=0 h ( 0 ) = h ′ ( 0 ) = 0 . As in the recursive construction, we can recursively determine h w > 0 h_w>0 h w > 0 from the above equation. Moreover, if b l ≤ h l b_l\le h_l b l ≤ h l for 2 ≤ l < w 2\le l<w 2 ≤ l < w , we know
b w ≤ [ t w ] ∑ k = 2 w C k ( t + ∑ l = 2 w − 1 h l t l ) k = h w . b_w\le[t^w]\sum_{k=2}^{w}C^k\left(t+\sum_{l=2}^{w-1}h_l t^l\right)^k=h_w. b w ≤ [ t w ] k = 2 ∑ w C k ( t + l = 2 ∑ w − 1 h l t l ) k = h w .
It is easy to check that b 2 ≤ h 2 b_2\le h_2 b 2 ≤ h 2 . Thus inductively, we know b w ≤ h w b_w\le h_w b w ≤ h w for every w ≥ 2 w\ge2 w ≥ 2 . By Cauchy’s estimate for analytic functions, we have
b w = ∥ [ H i ] w − [ H i ] w − 1 ∥ c o e f f ≤ h w ≤ r − w max ∣ t ∣ = r ∣ h ( t ) ∣ ≤ C 1 w . b_w=\left\|[H_i]_w-[H_i]_{w-1}\right\|_{\mathrm{coeff}}\le h_w\le r^{-w}\max_{|t|=r}|h(t)|\le C_1^w. b w = ∥ [ H i ] w − [ H i ] w − 1 ∥ coeff ≤ h w ≤ r − w ∣ t ∣ = r max ∣ h ( t ) ∣ ≤ C 1 w .
Next, we bound the coefficient ∥ ∂ α i ℓ H i , w ( m ) ∥ c o e f f \left\|\partial_{\alpha_{i\ell}}H_{i,w}(m)\right\|_{\mathrm{coeff}} ∥ ∂ α i ℓ H i , w ( m ) ∥ coeff , where H i , w H_{i,w} H i , w is treated as a polynomial in parameter α i ℓ \alpha_{i\ell} α i ℓ . To bound the first-order derivatives, we allow these coordinates to be complex with max ℓ ∣ α i ℓ ∣ ≤ M + 1 \max_\ell|\alpha_{i\ell}|\le M+1 max ℓ ∣ α i ℓ ∣ ≤ M + 1 for M M M given in Assumption 2 ( i ) 2(i) 2 ( i ) . The preceding calculation still applies after increasing C 0 C_0 C 0 as ∑ ℓ ∣ α i ℓ ∣ ≤ d ( M + 1 ) \sum_\ell|\alpha_{i\ell}|\le d(M+1) ∑ ℓ ∣ α i ℓ ∣ ≤ d ( M + 1 ) . Then Cauchy’s formula implies that, at any real α i \alpha_i α i ,
∂ α i ℓ H i , w ( m ; α i ) = ( 2 π i ) − 1 ∮ ∣ z ∣ = 1 z − 2 H i , w ( m ; α i + z e ℓ ) d z , \partial_{\alpha_{i\ell}}H_{i,w}(m;\alpha_i)=(2\pi i)^{-1}\oint_{|z|=1}z^{-2}H_{i,w}(m;\alpha_i+z e_\ell)\,dz, ∂ α i ℓ H i , w ( m ; α i ) = ( 2 π i ) − 1 ∮ ∣ z ∣ = 1 z − 2 H i , w ( m ; α i + z e ℓ ) d z ,
and therefore
∥ ∂ α i ℓ H i , w ∥ c o e f f ≤ sup ∣ z ∣ = 1 ∥ H i , w ( m ; α i + z e ℓ ) ∥ c o e f f ≤ C 2 w . (57) \left\|\partial_{\alpha_{i\ell}}H_{i,w}\right\|_{\mathrm{coeff}}\le\sup_{|z|=1}\left\|H_{i,w}(m;\alpha_i+z e_\ell)\right\|_{\mathrm{coeff}}\le C_2^w. \tag*{(57)} ∥ ∂ α i ℓ H i , w ∥ coeff ≤ ∣ z ∣ = 1 sup ∥ H i , w ( m ; α i + z e ℓ ) ∥ coeff ≤ C 2 w . (57)
Now we move forward to bound δ i − ev { [ H i ( m ) ] N } \delta_i-\operatorname{ev}\{[H_i(m)]_N\} δ i − ev {[ H i ( m ) ] N } . Factorize F i ( s , m ) F_i(s,m) F i ( s , m ) to get F i ( s , m ) = ( s − H i ( m ) ) Q i ( s , m ) F_i(s,m)=(s-H_i(m))Q_i(s,m) F i ( s , m ) = ( s − H i ( m )) Q i ( s , m ) for a formal series Q i ( s , m ) Q_i(s,m) Q i ( s , m ) given by
Q i ( s , m ) = 1 + ∑ k ≥ 2 c k ( T i ) E f { ∑ ℓ = 0 k − 1 ( α i T v + s ) k − 1 − ℓ ( α i T v + H i ( m ) ) ℓ } . (58) Q_i(s,m)=1+\sum_{k\ge2}c_k(T_i)\mathbb{E}_f\left\{\sum_{\ell=0}^{k-1}(\alpha_i^T v+s)^{k-1-\ell}(\alpha_i^T v+H_i(m))^\ell\right\}. \tag*{(58)} Q i ( s , m ) = 1 + k ≥ 2 ∑ c k ( T i ) E f { ℓ = 0 ∑ k − 1 ( α i T v + s ) k − 1 − ℓ ( α i T v + H i ( m ) ) ℓ } . (58)
Writing Q i , N − 1 ( s , m ) = [ Q i ( s , m ) − 1 ] N Q_{i,N}^{-1}(s,m)=[Q_i(s,m)^{-1}]_N Q i , N − 1 ( s , m ) = [ Q i ( s , m ) − 1 ] N , we have
[ s − H i ( m ) ] N = [ [ F i ( s , m ) ] N Q i , N − 1 ( s , m ) ] N . [s-H_i(m)]_N=\left[[F_i(s,m)]_NQ_{i,N}^{-1}(s,m)\right]_N. [ s − H i ( m ) ] N = [ [ F i ( s , m ) ] N Q i , N − 1 ( s , m ) ] N .
In what follows, we omit the common arguments of polynomials when they are clear from the context. It suffices to control the evaluated right side. To control this term, following a similar strategy in the proof of Lemma B.4 , we first control ∥ [ Q i − 1 ] w − [ Q i − 1 ] w − 1 ∥ c o e f f \left\|[Q_i^{-1}]_w-[Q_i^{-1}]_{w-1}\right\|_{\mathrm{coeff}} [ Q i − 1 ] w − [ Q i − 1 ] w − 1 coeff . Apply [ t w ] [t^w] [ t w ] to both side of the equation, and note that the contribution of c k ( T i ) c_k(T_i) c k ( T i ) , α i T v \alpha_i^T v α i T v , s s s , and H i ( m ) H_i(m) H i ( m ) are bounded by K k − 1 K^{k-1} K k − 1 , C 0 t C_0t C 0 t , t 2 t^2 t 2 , and h ( t ) h(t) h ( t ) , respectively, which gives
∥ [ Q i − 1 ] w − [ Q i − 1 ] w − 1 ∥ c o e f f ≤ [ t w ] ∑ k ≥ 2 k C k { t + t 2 + h ( t ) } k − 1 , \left\|[Q_i-1]_w-[Q_i-1]_{w-1}\right\|_{\mathrm{coeff}}\le[t^w]\sum_{k\ge2}kC^k\{t+t^2+h(t)\}^{k-1}, ∥ [ Q i − 1 ] w − [ Q i − 1 ] w − 1 ∥ coeff ≤ [ t w ] k ≥ 2 ∑ k C k { t + t 2 + h ( t ) } k − 1 ,
Let q ( t ) = ∑ k ≥ 2 k C k { t + t 2 + h ( t ) } k − 1 q(t)=\sum_{k\geq2}kC^k\{t+t^2+h(t)\}^{k-1} q ( t ) = ∑ k ≥ 2 k C k { t + t 2 + h ( t ) } k − 1 , which has nonnegative coefficients. Then ∥ [ ( 1 − Q i ) j ] w − [ ( 1 − Q i ) j ] w − 1 ∥ c o e f f ≤ [ t w ] q ( t ) j \|[(1-Q_i)^j]_w-[(1-Q_i)^j]_{w-1}\|_{\mathrm{coeff}}\leq[t^w]q(t)^j ∥ [( 1 − Q i ) j ] w − [( 1 − Q i ) j ] w − 1 ∥ coeff ≤ [ t w ] q ( t ) j . With q ( 0 ) = 0 q(0)=0 q ( 0 ) = 0 , we can select some radii ρ \rho ρ such that q ( ρ ) < 1 / 2 q(\rho)<1/2 q ( ρ ) < 1/2 and the inverse Q i − 1 = ∑ j ≥ 0 ( 1 − Q i ) j Q_i^{-1}=\sum_{j\geq0}(1-Q_i)^j Q i − 1 = ∑ j ≥ 0 ( 1 − Q i ) j satisfies
∥ [ Q i − 1 ] w − [ Q i − 1 ] w − 1 ∥ c o e f f ≤ [ t w ] ∑ j = 0 ∞ q ( t ) j = [ t w ] 1 1 − q ( t ) \|[Q_i^{-1}]_w-[Q_i^{-1}]_{w-1}\|_{\mathrm{coeff}}\leq[t^w]\sum_{j=0}^{\infty}q(t)^j=[t^w]\frac{1}{1-q(t)} ∥ [ Q i − 1 ] w − [ Q i − 1 ] w − 1 ∥ coeff ≤ [ t w ] j = 0 ∑ ∞ q ( t ) j = [ t w ] 1 − q ( t ) 1
Suppose ( 1 − q ( t ) ) − 1 = ∑ n = 0 ∞ d n t n (1-q(t))^{-1}=\sum_{n=0}^{\infty}d_nt^n ( 1 − q ( t ) ) − 1 = ∑ n = 0 ∞ d n t n . Since d n ≥ 0 d_n\geq0 d n ≥ 0 as q ( t ) q(t) q ( t ) has nonnegative coefficients, we know at ρ \rho ρ , there is d w ρ w ≤ ∑ n = 0 ∞ d n ρ n = ( 1 − q ( ρ ) ) − 1 d_w\rho^w\leq\sum_{n=0}^{\infty}d_n\rho^n=(1-q(\rho))^{-1} d w ρ w ≤ ∑ n = 0 ∞ d n ρ n = ( 1 − q ( ρ ) ) − 1 . Thus [ t w ] ( 1 − q ( t ) ) − 1 = d w [t^w](1-q(t))^{-1}=d_w [ t w ] ( 1 − q ( t ) ) − 1 = d w and
∥ [ Q i − 1 ] w − [ Q i − 1 ] w − 1 ∥ c o e f f ≤ ρ − w 1 − q ( ρ ) ≤ 2 ρ − w ≤ C w + 1 . \|[Q_i^{-1}]_w-[Q_i^{-1}]_{w-1}\|_{\mathrm{coeff}}\leq\frac{\rho^{-w}}{1-q(\rho)}\leq2\rho^{-w}\leq C^{w+1}. ∥ [ Q i − 1 ] w − [ Q i − 1 ] w − 1 ∥ coeff ≤ 1 − q ( ρ ) ρ − w ≤ 2 ρ − w ≤ C w + 1 .
We now bound δ i − ev { [ H i ( m ) ] N } \delta_i-\operatorname{ev}\{[H_i(m)]_N\} δ i − ev {[ H i ( m ) ] N } using the preceding truncated identity. Note that
∥ δ i − ev { [ H i ] N } ∥ q ≤ ∥ ev { [ F i ] N } ∥ 2 q ∥ ev ( Q i , N − 1 ) ∥ 2 q + ∥ ev { [ F i ] N Q i , N − 1 − [ F i ] N Q i , N − 1 } ∥ q . (59) \|\delta_i-\operatorname{ev}\{[H_i]_N\}\|_q\leq\|\operatorname{ev}\{[F_i]_N\}\|_{2q}\|\operatorname{ev}(Q_{i,N}^{-1})\|_{2q}+\|\operatorname{ev}\{[F_i]_NQ_{i,N}^{-1}-[F_i]_NQ_{i,N}^{-1}\}\|_q. \tag*{(59)} ∥ δ i − ev {[ H i ] N } ∥ q ≤ ∥ ev {[ F i ] N } ∥ 2 q ∥ ev ( Q i , N − 1 ) ∥ 2 q + ∥ ev {[ F i ] N Q i , N − 1 − [ F i ] N Q i , N − 1 } ∥ q . (59)
For ev { [ F i ] N } \operatorname{ev}\{[F_i]_N\} ev {[ F i ] N } , let P i , N ( v , s ) = ∑ k = 1 N c k ( T i ) ( α i T v + s ) k P_{i,N}(v,s)=\sum_{k=1}^{N}c_k(T_i)(\alpha_i^{\mathsf T}v+s)^k P i , N ( v , s ) = ∑ k = 1 N c k ( T i ) ( α i T v + s ) k . Then [ F i ] N = E f { [ P i , N ] N } [F_i]_N=\mathbb{E}_{f}\{[P_{i,N}]_N\} [ F i ] N = E f {[ P i , N ] N } . It is easy to check by previous bounds that ∥ P i , N ∥ c o e f f ≤ ∑ k = 1 N K k − 1 ( C 0 + 1 ) k ≤ e C N \|P_{i,N}\|_{\mathrm{coeff}}\leq\sum_{k=1}^{N}K^{k-1}(C_0+1)^k\leq e^{CN} ∥ P i , N ∥ coeff ≤ ∑ k = 1 N K k − 1 ( C 0 + 1 ) k ≤ e C N and ∥ [ F i ] N ∥ c o e f f ≤ ∥ P i , N ∥ c o e f f ≤ e C N \|[F_i]_N\|_{\mathrm{coeff}}\leq\|P_{i,N}\|_{\mathrm{coeff}}\leq e^{CN} ∥ [ F i ] N ∥ coeff ≤ ∥ P i , N ∥ coeff ≤ e C N . Let u = α i T V + δ i u=\alpha_i^{\mathsf T}V+\delta_i u = α i T V + δ i . With ∥ u ∥ r ≤ e C ( 1 + r ) p − 1 / 2 \|u\|_r\leq e^{C(1+r)}p^{-1/2} ∥ u ∥ r ≤ e C ( 1 + r ) p − 1/2 and E e a ∣ u ∣ ≤ e C ( 1 + a 2 ) \mathbb{E}e^{a|u|}\leq e^{C(1+a^2)} E e a ∣ u ∣ ≤ e C ( 1 + a 2 ) , we have by (C.9) and Hölder’s inequality, that
∥ ev ( P i , N ) − g T i ( u ) ∥ q ≤ K N ( E e 2 q H ∣ u ∣ ) 1 / ( 2 q ) ∥ u ∥ 2 q ( N + 1 ) N + 1 ≤ K N e C ( 1 + q ) e C ( N + 1 ) { 1 + 2 q ( N + 1 ) } p − ( N + 1 ) / 2 ≤ e C ′ ( 1 + q ) N 2 p − ( N + 1 ) / 2 . (60) \begin{aligned}
\|\operatorname{ev}(P_{i,N})-g_{T_i}(u)\|_q\leq K^N(\mathbb{E}e^{2qH|u|})^{1/(2q)}\|u\|_{2q(N+1)}^{N+1} \\
&\leq K^Ne^{C(1+q)}e^{C(N+1)\{1+2q(N+1)\}}p^{-(N+1)/2} \\
&\leq e^{C'(1+q)N^2}p^{-(N+1)/2}. \tag*{(60)}
\end{aligned} ∥ ev ( P i , N ) − g T i ( u ) ∥ q ≤ K N ( E e 2 q H ∣ u ∣ ) 1/ ( 2 q ) ∥ u ∥ 2 q ( N + 1 ) N + 1 ≤ K N e C ( 1 + q ) e C ( N + 1 ) { 1 + 2 q ( N + 1 )} p − ( N + 1 ) /2 ≤ e C ′ ( 1 + q ) N 2 p − ( N + 1 ) /2 . (60)
We then invoke E [ g T i ( u ) ∣ Y ] = 0 \mathbb{E}[g_{T_i}(u)\mid Y]=0 E [ g T i ( u ) ∣ Y ] = 0 as in (B.20) to get
∥ ev { [ F i ] N } ∥ q = ∥ E [ ev { [ P i , N ] N − P i , N } + ev ( P i , N ) − g T i ( u ) ∣ Y ] ∥ q ≤ ∥ ev { P i , N − [ P i , N ] N } ∥ q + ∥ ev ( P i , N ) − g T i ( u ) ∥ q ≤ e C ( 1 + q ) N 2 p − ( N + 1 ) / 2 , (61) \begin{aligned}
\|\operatorname{ev}\{[F_i]_N\}\|_q=\left\|\mathbb{E}\left[\operatorname{ev}\{[P_{i,N}]_N-P_{i,N}\}+\operatorname{ev}(P_{i,N})-g_{T_i}(u)\mid Y\right]\right\|_q \\
&\leq\|\operatorname{ev}\{P_{i,N}-[P_{i,N}]_N\}\|_q+\|\operatorname{ev}(P_{i,N})-g_{T_i}(u)\|_q \\
&\leq e^{C(1+q)N^2}p^{-(N+1)/2}, \tag*{(61)}
\end{aligned} ∥ ev {[ F i ] N } ∥ q = ∥ E [ ev {[ P i , N ] N − P i , N } + ev ( P i , N ) − g T i ( u ) ∣ Y ] ∥ q ≤ ∥ ev { P i , N − [ P i , N ] N } ∥ q + ∥ ev ( P i , N ) − g T i ( u ) ∥ q ≤ e C ( 1 + q ) N 2 p − ( N + 1 ) /2 , (61)
where the last inequality is due to (B.15) and (C.14).
For ev ( Q i , N − 1 ) \operatorname{ev}(Q_{i,N}^{-1}) ev ( Q i , N − 1 ) , we note that ∥ Q i , N − 1 ∥ c o e f f ≤ 1 + ∑ w = 1 N C w + 1 ≤ e C N \|Q_{i,N}^{-1}\|_{\mathrm{coeff}}\leq1+\sum_{w=1}^{N}C^{w+1}\leq e^{CN} ∥ Q i , N − 1 ∥ coeff ≤ 1 + ∑ w = 1 N C w + 1 ≤ e C N . By (B.14), we have that
∥ ev ( Q i , N − 1 ) ∥ 2 q ≤ ∥ Q i , N − 1 ∥ c o e f f e C ( 1 + 2 q ) N 2 ≤ e C ( 1 + q ) N 2 . \|\operatorname{ev}(Q_{i,N}^{-1})\|_{2q}\leq\|Q_{i,N}^{-1}\|_{\mathrm{coeff}}e^{C(1+2q)N^2}\leq e^{C(1+q)N^2}. ∥ ev ( Q i , N − 1 ) ∥ 2 q ≤ ∥ Q i , N − 1 ∥ coeff e C ( 1 + 2 q ) N 2 ≤ e C ( 1 + q ) N 2 .
Finally, for the last term in (C.13), since the terms discarded from [ F i ] N Q i , N − 1 [F_i]_NQ_{i,N}^{-1} [ F i ] N Q i , N − 1 have weights between N + 1 N+1 N + 1 and 2 N 2N 2 N , we arrive at
∥ [ F i ] N Q i , N − 1 − [ F i ] N Q i , N − 1 ∥ c o e f f ≤ ∥ [ F i ] N ∥ c o e f f ∥ Q i , N − 1 ∥ c o e f f ≤ e C N . \left\|[\mathcal{F}_i]_N\mathcal{Q}_{i,N}^{-1}-[\mathcal{F}_i]_N\mathcal{Q}_{i,N}^{-1}\right\|_{\mathrm{coeff}}\leq\|[\mathcal{F}_i]_N\|_{\mathrm{coeff}}\|\mathcal{Q}_{i,N}^{-1}\|_{\mathrm{coeff}}\leq e^{CN}. [ F i ] N Q i , N − 1 − [ F i ] N Q i , N − 1 coeff ≤ ∥ [ F i ] N ∥ coeff ∥ Q i , N − 1 ∥ coeff ≤ e C N .
Combining these estimates into (C.13) to yield
∥ δ i − ev { [ H i ] N } ∥ q ≤ e C ( 1 + q ) N 2 p − ( N + 1 ) / 2 . (62) \|\delta_i-\operatorname{ev}\{[H_i]_N\}\|_q\leq e^{C(1+q)N^2}p^{-(N+1)/2}. \tag*{(62)} ∥ δ i − ev {[ H i ] N } ∥ q ≤ e C ( 1 + q ) N 2 p − ( N + 1 ) /2 . (62)
We next construct a polynomial approximation and establish an L q L^q L q bound for g T i ( α i ⊤ V + δ i ) g_{T_i}(\alpha_i^\top V+\delta_i) g T i ( α i ⊤ V + δ i ) similarly as for δ i \delta_i δ i . Specifically, let G i , N ( v , m ) = [ g T i ( α i ⊤ v + H i ( m ) ) ] N G_{i,N}(v,m)=[g_{T_i}(\alpha_i^\top v+H_i(m))]_N G i , N ( v , m ) = [ g T i ( α i ⊤ v + H i ( m )) ] N . Its weight-one term is α i ⊤ v \alpha_i^\top v α i ⊤ v . The other coefficient can be bounded in the Taylor series (C.9) by
∥ [ G i , N ( v , m ) ] w − [ G i , N ( v , m ) ] w − 1 ∥ c o e f f ≤ [ t w ] ∑ k = 1 ∞ K k − 1 { C 0 t + h ( t ) } k = [ t w ] C 0 t + h ( t ) 1 − K { C 0 t + h ( t ) } ≤ C w . (63) \|[G_{i,N}(v,m)]_w-[G_{i,N}(v,m)]_{w-1}\|_{\mathrm{coeff}}\leq[t^w]\sum_{k=1}^{\infty}K^{k-1}\{C_0t+h(t)\}^k=[t^w]\frac{C_0t+h(t)}{1-K\{C_0t+h(t)\}}\leq C^w. \tag*{(63)} ∥ [ G i , N ( v , m ) ] w − [ G i , N ( v , m ) ] w − 1 ∥ coeff ≤ [ t w ] k = 1 ∑ ∞ K k − 1 { C 0 t + h ( t ) } k = [ t w ] 1 − K { C 0 t + h ( t )} C 0 t + h ( t ) ≤ C w . (63)
The same Cauchy derivative estimate as in (C.11) gives
max ℓ ∈ [ d ] ∥ ∂ α i ℓ ( [ G i , N ( v , m ) ] w − [ G i , N ( v , m ) ] w − 1 ) ∥ c o e f f ≤ C w . \max_{\ell\in[d]}\|\partial_{\alpha_{i\ell}}([G_{i,N}(v,m)]_w-[G_{i,N}(v,m)]_{w-1})\|_{\mathrm{coeff}}\leq C^w. ℓ ∈ [ d ] max ∥ ∂ α i ℓ ([ G i , N ( v , m ) ] w − [ G i , N ( v , m ) ] w − 1 ) ∥ coeff ≤ C w .
To transfer the approximation of δ i \delta_i δ i to G i , N ( v , m ) G_{i,N}(v,m) G i , N ( v , m ) , let
K N ( v , s , m ) = [ g T i ( α i ⊤ v + s ) − g T i ( α i ⊤ v + H i ( m ) ) s − H i ( m ) ] N . K_N(v,s,m)=\left[\frac{g_{T_i}(\alpha_i^\top v+s)-g_{T_i}(\alpha_i^\top v+H_i(m))}{s-H_i(m)}\right]_N. K N ( v , s , m ) = [ s − H i ( m ) g T i ( α i ⊤ v + s ) − g T i ( α i ⊤ v + H i ( m )) ] N .
Invoke Q i ( s , m ) Q_i(s,m) Q i ( s , m ) in (C.12) to get
K N ( v , s , m ) = [ ∑ k ≥ 1 c k ( T i ) ∑ ℓ = 0 k − 1 ( α i ⊤ v + s ) k − 1 − ℓ ( α i ⊤ v + H i ( m ) ) ℓ ] N K_N(v,s,m)=\left[\sum_{k\geq1}c_k(T_i)\sum_{\ell=0}^{k-1}(\alpha_i^\top v+s)^{k-1-\ell}(\alpha_i^\top v+H_i(m))^\ell\right]_N K N ( v , s , m ) = [ k ≥ 1 ∑ c k ( T i ) ℓ = 0 ∑ k − 1 ( α i ⊤ v + s ) k − 1 − ℓ ( α i ⊤ v + H i ( m ) ) ℓ ] N
Thus we have ∥ K N ∥ c o e f f ≤ ∑ w = 0 N [ t w ] { 1 + q ( t ) } ≤ e C N \|K_N\|_{\mathrm{coeff}}\leq\sum_{w=0}^{N}[t^w]\{1+q(t)\}\leq e^{CN} ∥ K N ∥ coeff ≤ ∑ w = 0 N [ t w ] { 1 + q ( t )} ≤ e C N and ∥ ev { K N } ∥ 2 q ≤ e C ( 1 + q ) N 2 \|\operatorname{ev}\{K_N\}\|_{2q}\leq e^{C(1+q)N^2} ∥ ev { K N } ∥ 2 q ≤ e C ( 1 + q ) N 2 . Suppose K ( v , s , m ) K(v,s,m) K ( v , s , m ) is the untruncated version of K N K_N K N . Then g T i ( α i ⊤ v + s ) − g T i ( α i ⊤ v + H i ( m ) ) = ( s − H i ( m ) ) K ( v , s , m ) g_{T_i}(\alpha_i^\top v+s)-g_{T_i}(\alpha_i^\top v+H_i(m))=(s-H_i(m))K(v,s,m) g T i ( α i ⊤ v + s ) − g T i ( α i ⊤ v + H i ( m )) = ( s − H i ( m )) K ( v , s , m ) . Recall that P i , N ( v , s ) = ∑ k = 1 N c k ( T i ) ( α i ⊤ v + s ) k P_{i,N}(v,s)=\sum_{k=1}^{N}c_k(T_i)(\alpha_i^\top v+s)^k P i , N ( v , s ) = ∑ k = 1 N c k ( T i ) ( α i ⊤ v + s ) k and G i , N ( v , m ) = [ g T i ( α i ⊤ v + H i ( m ) ) ] N G_{i,N}(v,m)=[g_{T_i}(\alpha_i^\top v+H_i(m))]_N G i , N ( v , m ) = [ g T i ( α i ⊤ v + H i ( m )) ] N . Truncation to N N N yields
[ P i , N ( v , s ) ] N − G i , N ( v , m ) = [ ( s − [ H i ( m ) ] N ) K N ] N [P_{i,N}(v,s)]_N-G_{i,N}(v,m)=\left[(s-[H_i(m)]_N)K_N\right]_N [ P i , N ( v , s ) ] N − G i , N ( v , m ) = [ ( s − [ H i ( m ) ] N ) K N ] N
Hölder’s inequality gives
∥ ev { [ P i , N ( v , s ) ] N − G i , N ( v , m ) } ∥ q ≤ ∥ δ i − ev { [ H i ( m ) ] N } ∥ 2 q ∥ ev ( K N ) ∥ 2 q + ∥ ev { ( s − [ H i ( m ) ] N ) K N − [ ( s − [ H i ( m ) ] N ) K N ] N } ∥ q ≤ e C ( 1 + q ) N 2 p − ( N + 1 ) / 2 + { 1 + ∥ [ H i ( m ) ] N ∥ c o e f f } ∥ K N ∥ c o e f f e C ′ ( 1 + q ) N 2 p − ( N + 1 ) / 2 (1) ≤ e C ′ ′ ( 1 + q ) N 2 p − ( N + 1 ) / 2 (2) \begin{aligned}
\| \operatorname{ev}\{[P_{i,N}(v,s)]_N-G_{i,N}(v,m)\}\|_q
&\le\|\delta_i-\operatorname{ev}\{[H_i(m)]_N\}\|_{2q}\|\operatorname{ev}(K_N)\|_{2q}\\
&\quad+\|\operatorname{ev}\{(s-[H_i(m)]_N)K_N-[(s-[H_i(m)]_N)K_N]_N\}\|_q\\
&\le e^{C(1+q)N^2}p^{-(N+1)/2}+\{1+\|[H_i(m)]_N\|_{\mathrm{coeff}}\}\|K_N\|_{\mathrm{coeff}}e^{C'(1+q)N^2}p^{-(N+1)/2}\quad\text{(1)}\\
&\le e^{C''(1+q)N^2}p^{-(N+1)/2}\quad\text{(2)}
\end{aligned} ∥ ev {[ P i , N ( v , s ) ] N − G i , N ( v , m )} ∥ q ≤ ∥ δ i − ev {[ H i ( m ) ] N } ∥ 2 q ∥ ev ( K N ) ∥ 2 q + ∥ ev {( s − [ H i ( m ) ] N ) K N − [( s − [ H i ( m ) ] N ) K N ] N } ∥ q ≤ e C ( 1 + q ) N 2 p − ( N + 1 ) /2 + { 1 + ∥ [ H i ( m ) ] N ∥ coeff } ∥ K N ∥ coeff e C ′ ( 1 + q ) N 2 p − ( N + 1 ) /2 (1) ≤ e C ′′ ( 1 + q ) N 2 p − ( N + 1 ) /2 (2)
Here, in (1), the first term is bounded by (C.16) and (B.15), and the second part is due to truncated terms have weights between N + 1 N+1 N + 1 and 2 N 2N 2 N and coefficient norm at most e C N e^{CN} e C N ; (2) is due to ∥ [ H i ( m ) ] N ∥ c o e f f ≤ e C N \|[H_i(m)]_N\|_{\mathrm{coeff}}\le e^{CN} ∥ [ H i ( m ) ] N ∥ coeff ≤ e C N and ∥ K N ∥ c o e f f ≤ e C N \|K_N\|_{\mathrm{coeff}}\le e^{CN} ∥ K N ∥ coeff ≤ e C N . Therefore, we arrive at
∥ g T i ( α i T v + δ i ) − ev { G i , N ( v , m ) } ∥ q ≤ ∥ g T i ( u ) − ev { P i , N ( v , s ) } ∥ q + ∥ ev { P i , N ( v , s ) − [ P i , N ( v , s ) ] N } ∥ q + ∥ ev { [ P i , N ( v , s ) ] N − G i , N ( v , m ) } ∥ q ≤ e C ( 1 + q ) N 2 p − ( N + 1 ) / 2 . (64) \begin{aligned}
\|g_{T_i}(\alpha_i^{\mathsf T}v+\delta_i)-\operatorname{ev}\{G_{i,N}(v,m)\}\|_q
&\le\|g_{T_i}(u)-\operatorname{ev}\{P_{i,N}(v,s)\}\|_q\\
&\quad+\|\operatorname{ev}\{P_{i,N}(v,s)-[P_{i,N}(v,s)]_N\}\|_q\\
&\quad+\|\operatorname{ev}\{[P_{i,N}(v,s)]_N-G_{i,N}(v,m)\}\|_q\\
&\le e^{C(1+q)N^2}p^{-(N+1)/2}. \tag*{(64)}
\end{aligned} ∥ g T i ( α i T v + δ i ) − ev { G i , N ( v , m )} ∥ q ≤ ∥ g T i ( u ) − ev { P i , N ( v , s )} ∥ q + ∥ ev { P i , N ( v , s ) − [ P i , N ( v , s ) ] N } ∥ q + ∥ ev {[ P i , N ( v , s ) ] N − G i , N ( v , m )} ∥ q ≤ e C ( 1 + q ) N 2 p − ( N + 1 ) /2 . (64)
where the last terms on the right side are bounded by (C.15).
Finally, similar to (C.14), we have by ∣ g t ( u ) ∣ ≤ e H ∣ u ∣ ∣ u ∣ |g_t(u)|\le e^{H|u|}|u| ∣ g t ( u ) ∣ ≤ e H ∣ u ∣ ∣ u ∣ that
∥ g T i ( α i T v + δ i ) ∥ q ≤ ∥ u ∥ 2 q ( E e 2 q H ∣ u ∣ ) 1 / ( 2 q ) ≤ e C ( 1 + 2 q ) p − 1 / 2 e C { 1 + ( 2 q H ) 2 } / ( 2 q ) ≤ e C ( 1 + q ) p − 1 / 2 . \|g_{T_i}(\alpha_i^{\mathsf T}v+\delta_i)\|_q
\le\|u\|_{2q}(\mathbb{E}e^{2qH|u|})^{1/(2q)}
\le e^{C(1+2q)}p^{-1/2}e^{C\{1+(2qH)^2\}/(2q)}
\le e^{C(1+q)}p^{-1/2}. ∥ g T i ( α i T v + δ i ) ∥ q ≤ ∥ u ∥ 2 q ( E e 2 q H ∣ u ∣ ) 1/ ( 2 q ) ≤ e C ( 1 + 2 q ) p − 1/2 e C { 1 + ( 2 q H ) 2 } / ( 2 q ) ≤ e C ( 1 + q ) p − 1/2 .
Proof of Lemma B.6
Write Δ = max i , j ∈ [ p ] ∣ α i 1 α j 1 − α i 0 α j 0 ∣ \Delta=\max_{i,j\in[p]}|\alpha^1_i\alpha^1_j-\alpha^0_i\alpha^0_j| Δ = max i , j ∈ [ p ] ∣ α i 1 α j 1 − α i 0 α j 0 ∣ . Recall that Lemma B.1 finds a reference set I \mathcal{I} I of size d d d for which σ min ( B 0 ) ≥ c 1 \sigma_{\min}(B_0)\ge c_1 σ m i n ( B 0 ) ≥ c 1 , where B r = ( A r ) I , : B_r=(A_r)_{\mathcal{I},:} B r = ( A r ) I , : . Set Q = B 1 T ( B 0 − 1 ) T Q=B_1^{\mathsf T}(B_0^{-1})^{\mathsf T} Q = B 1 T ( B 0 − 1 ) T , and we have
∥ Q T Q − I d ∥ = ∥ B 0 − 1 ( B 1 B 1 T − B 0 B 0 T ) ( B 0 − 1 ) T ∥ ≤ d c 1 − 2 Δ . \|Q^{\mathsf T}Q-I_d\|=\|B_0^{-1}(B_1B_1^{\mathsf T}-B_0B_0^{\mathsf T})(B_0^{-1})^{\mathsf T}\|\le dc_1^{-2}\Delta. ∥ Q T Q − I d ∥ = ∥ B 0 − 1 ( B 1 B 1 T − B 0 B 0 T ) ( B 0 − 1 ) T ∥ ≤ d c 1 − 2 Δ.
When d c 1 − 2 Δ ≤ 1 / 2 dc_1^{-2}\Delta\le1/2 d c 1 − 2 Δ ≤ 1/2 , we know O = Q ( Q T Q ) − 1 / 2 O=Q(Q^{\mathsf T}Q)^{-1/2} O = Q ( Q T Q ) − 1/2 is orthogonal and satisfies ∥ O − Q ∥ ≤ 3 d c 1 − 2 Δ \|O-Q\|\le3dc_1^{-2}\Delta ∥ O − Q ∥ ≤ 3 d c 1 − 2 Δ by the spectral theorem. Since max j ∈ [ p ] ∥ B 1 α j 1 − B 0 α j 0 ∥ ≤ d 1 / 2 Δ \max_{j\in[p]}\|B_1\alpha^1_j-B_0\alpha^0_j\|\le d^{1/2}\Delta max j ∈ [ p ] ∥ B 1 α j 1 − B 0 α j 0 ∥ ≤ d 1/2 Δ , we know that for this fixed O O O , the following holds
∥ α j 1 − O α j 0 ∥ = ∥ O T α j 1 − α j 0 ∥ ≤ ∥ O − Q ∥ ∥ α j 1 ∥ + ∥ B 0 − 1 ∥ ∥ B 1 α j 1 − B 0 α j 0 ∥ ≤ C Δ , \|\alpha^1_j-O\alpha^0_j\|=\|O^{\mathsf T}\alpha^1_j-\alpha^0_j\|
\le\|O-Q\|\|\alpha^1_j\|+\|B_0^{-1}\|\|B_1\alpha^1_j-B_0\alpha^0_j\|
\le C\Delta, ∥ α j 1 − O α j 0 ∥ = ∥ O T α j 1 − α j 0 ∥ ≤ ∥ O − Q ∥∥ α j 1 ∥ + ∥ B 0 − 1 ∥∥ B 1 α j 1 − B 0 α j 0 ∥ ≤ C Δ ,
uniformly in j ∈ [ p ] j \in[p] j ∈ [ p ] . When d c 1 − 2 Δ > 1 / 2 d c_1^{-2}\Delta> 1/2 d c 1 − 2 Δ > 1/2 , take O = I d O=I_d O = I d and use max j ∈ [ p ] ∣ α 1 j − α 0 j ∣ ≤ 2 M ≤ 4 d M c 1 − 2 Δ \max_{j\in[p]}|\alpha_{1j}-\alpha_{0j}| \le2M \le4dMc_1^{-2}\Delta max j ∈ [ p ] ∣ α 1 j − α 0 j ∣ ≤ 2 M ≤ 4 d M c 1 − 2 Δ . This proves (B.33).
Proof of Lemma B.7
Throughout this proof, the series H i ( m ) H_i(m) H i ( m ) and the forward polynomials F γ ( m ) F_\gamma(m) F γ ( m ) are those of model zero. We omit the arguments in the polynomials when they are clear from the context. Write Γ N = { γ ∈ N d : 2 ≤ ∣ γ ∣ ≤ N } \Gamma_N=\{\gamma\in\mathbb{N}^d:2\le|\gamma|\le N\} Γ N = { γ ∈ N d : 2 ≤ ∣ γ ∣ ≤ N } , and let P [ w ] = [ P ] w − [ P ] w − 1 P^{[w]}=[P]_w-[P]_{w-1} P [ w ] = [ P ] w − [ P ] w − 1 denote the component of weight w w w in a polynomial P P P .
By the construction of F γ F_\gamma F γ , we know F γ = m γ + F_\gamma=m_\gamma+ F γ = m γ + terms of weight greater than ∣ γ ∣ |\gamma| ∣ γ ∣ . Consequently, U − I \mathcal{U}-\mathcal{I} U − I strictly increases the minimum weight of every nonconstant polynomial and maps any constant to zero. Thus
U − 1 = ∑ l = 0 N ( I − U ) l and U ∑ l = 0 N ( I − U ) l = I . (65) \mathcal{U}^{-1}=\sum_{l=0}^{N}(\mathcal{I}-\mathcal{U})^l
\quad\text{and}\quad
\mathcal{U}\sum_{l=0}^{N}(\mathcal{I}-\mathcal{U})^l=\mathcal{I}. \tag*{(65)} U − 1 = l = 0 ∑ N ( I − U ) l and U l = 0 ∑ N ( I − U ) l = I . (65)
Next, we calculate the coefficient bound regarding operator U \mathcal{U} U , which is defined by
∥ U ∥ c o e f f = sup P ≠ 0 ∥ U P ∥ c o e f f ∥ P ∥ c o e f f . \|\mathcal{U}\|_{\mathrm{coeff}}=\sup_{P\ne0}\frac{\|\mathcal{U}P\|_{\mathrm{coeff}}}{\|P\|_{\mathrm{coeff}}}. ∥ U ∥ coeff = P = 0 sup ∥ P ∥ coeff ∥ U P ∥ coeff .
where the supremum is taken with respect to all symbolic polynomial P ( m ) = ∑ η a η ( T ) m η P(m)=\sum_{\eta}a_\eta(\mathbf{T})m^\eta P ( m ) = ∑ η a η ( T ) m η where T − ( T i ) i ∈ S \mathbf{T}-(T_i)_{i\in S} T − ( T i ) i ∈ S . For any nonconstant monomial m η = ∏ a = 1 b m γ a m^\eta=\prod_{a=1}^{b}m_{\gamma_a} m η = ∏ a = 1 b m γ a of weights u ≤ N u\le N u ≤ N . Recall that U ( m ) \mathcal{U}(m) U ( m ) is defined by substituting F γ F_\gamma F γ for each moment symbol M γ M_\gamma M γ and truncating to N N N order:
U ( m η ) = [ ∏ a = 1 b F γ a ( m ) ] N . \mathcal{U}(m^\eta)=\left[\prod_{a=1}^{b}F_{\gamma_a}(m)\right]_N. U ( m η ) = [ a = 1 ∏ b F γ a ( m ) ] N .
Since 2 ≤ ∣ γ a ∣ ≤ N 2\le|\gamma_a|\le N 2 ≤ ∣ γ a ∣ ≤ N , we know m m m is of weight u ≥ 2 b u\ge2b u ≥ 2 b . For any u ≤ w ≤ N u\le w\le N u ≤ w ≤ N , the weight-w w w component in U ( m η ) \mathcal{U}(m^\eta) U ( m η ) is given by choosing some weight t a t_a t a component from F γ a F_{\gamma_a} F γ a such that ∑ a = 1 b t a = w \sum_{a=1}^{b}t_a=w ∑ a = 1 b t a = w . Since ∥ F γ a [ t ] ∥ c o e f f ≤ C t \|F_{\gamma_a}^{[t]}\|_{\mathrm{coeff}}\le C^t ∥ F γ a [ t ] ∥ coeff ≤ C t by (B.30), the coefficient norm at weight w w w is bounded by
∥ [ ∏ a = 1 b F γ a ] w − [ ∏ a = 1 b F γ a ] w − 1 ∥ c o e f f ≤ ∑ t 1 + ⋯ + t b = w t a ≥ ∣ γ a ∣ ∏ a = 1 b C t a ≤ C w ( w − u + b − 1 b − 1 ) ≤ C 1 w . \left\|\left[\prod_{a=1}^{b}F_{\gamma_a}\right]_w-\left[\prod_{a=1}^{b}F_{\gamma_a}\right]_{w-1}\right\|_{\mathrm{coeff}}
\le\sum_{\substack{t_1+\cdots+t_b=w\\t_a\ge|\gamma_a|}}\prod_{a=1}^{b}C^{t_a}
\le C^w\binom{w-u+b-1}{b-1}\le C^w_1. [ a = 1 ∏ b F γ a ] w − [ a = 1 ∏ b F γ a ] w − 1 coeff ≤ t 1 + ⋯ + t b = w t a ≥ ∣ γ a ∣ ∑ a = 1 ∏ b C t a ≤ C w ( b − 1 w − u + b − 1 ) ≤ C 1 w .
For constant monomial, this bound holds trivially. Therefore, it is easy to see by summing over all w ≤ N w\le N w ≤ N that ∥ U ∥ c o e f f ≤ e C N \|\mathcal{U}\|_{\mathrm{coeff}}\le e^{CN} ∥ U ∥ coeff ≤ e C N and ∥ U − I ∥ c o e f f ≤ e C N \|\mathcal{U}-\mathcal{I}\|_{\mathrm{coeff}}\le e^{CN} ∥ U − I ∥ coeff ≤ e C N , which gives
∥ U − 1 ∥ c o e f f ≤ ∑ l = 0 N ∥ U − I ∥ c o e f f l ≤ e C N 2 . (66) \|\mathcal{U}^{-1}\|_{\mathrm{coeff}}\le\sum_{l=0}^{N}\|\mathcal{U}-\mathcal{I}\|_{\mathrm{coeff}}^l\le e^{CN^2}. \tag*{(66)} ∥ U − 1 ∥ coeff ≤ l = 0 ∑ N ∥ U − I ∥ coeff l ≤ e C N 2 . (66)
By the definition of R i ( m ) R_i(m) R i ( m ) and R i j ( m ) R_{ij}(m) R ij ( m ) in (B.34) and U \mathcal{U} U , we know [ R i { F ( m ) } ] N = [ H i ( m ) ] N [R_i\{F(m)\}]_N=[H_i(m)]_N [ R i { F ( m )} ] N = [ H i ( m ) ] N and [ R i j { F ( m ) } ] N = E f { ( α 0 i r v ) ( α 0 j r v ) } [R_{ij}\{F(m)\}]_N = \mathbb{E}_{f}\{(\alpha^r_{0i}v)(\alpha^r_{0j}v)\} [ R ij { F ( m )} ] N = E f {( α 0 i r v ) ( α 0 j r v )} . Both of these polynomials start at weight 2, and thus by Lemma B.5 and (C.20), we know ∥ R i ∥ c o e f f ≤ e C N 2 \|R_i\|_{\mathrm{coeff}} \le e^{CN^2} ∥ R i ∥ coeff ≤ e C N 2 and ∥ R i j ∥ c o e f f ≤ e C N 2 \|R_{ij}\|_{\mathrm{coeff}} \le e^{CN^2} ∥ R ij ∥ coeff ≤ e C N 2 . Follow the same calculation above, one can also show that for any 2 ≤ w ≤ N 2 \le w \le N 2 ≤ w ≤ N ,
∥ { R i ( m ) } [ w ] − { R i ( m ) } [ w − 1 ] ∥ c o e f f ≤ e C w 2 and ∥ { R i j ( m ) } [ w ] − { R i j ( m ) } [ w − 1 ] ∥ c o e f f ≤ e C w 2 . \left\|\{R_i(m)\}_{[w]}-\{R_i(m)\}_{[w-1]}\right\|_{\mathrm{coeff}} \le e^{Cw^2}
\quad\text{and}\quad
\left\|\{R_{ij}(m)\}_{[w]}-\{R_{ij}(m)\}_{[w-1]}\right\|_{\mathrm{coeff}} \le e^{Cw^2}. { R i ( m ) } [ w ] − { R i ( m ) } [ w − 1 ] coeff ≤ e C w 2 and { R ij ( m ) } [ w ] − { R ij ( m ) } [ w − 1 ] coeff ≤ e C w 2 .
We first prove (B.35) under model zero, i.e., r = 0 r=0 r = 0 . Recall that we have shown in (B.29) and (B.31) that, for any q ≥ 1 q \ge1 q ≥ 1 , ∥ O γ − ev 0 ( F γ ( m ) ) ∥ q ≤ e C ( 1 + q ) N 2 p − ( N + 1 ) / 2 \|O_\gamma-\operatorname{ev}_0(F_\gamma(m))\|_q \le e^{C(1+q)N^2}p^{-(N+1)/2} ∥ O γ − ev 0 ( F γ ( m )) ∥ q ≤ e C ( 1 + q ) N 2 p − ( N + 1 ) /2 and ∥ O γ ∥ q ≤ e C ( 1 + q ) ∣ γ ∣ 2 p − ∣ γ ∣ / 2 \|O_\gamma\|_q \le e^{C(1+q)|\gamma|^2}p^{-|\gamma|/2} ∥ O γ ∥ q ≤ e C ( 1 + q ) ∣ γ ∣ 2 p − ∣ γ ∣/2 . Therefore by Lemma B.4, we know
∥ ∏ γ ∈ Γ N O γ n γ − ev 0 { [ ∏ γ ∈ Γ N F γ ( m ) n γ ] N } ∥ q ≤ e C ( 1 + q ) N 2 p − ( N + 1 ) / 2 . \left\|\prod_{\gamma\in\Gamma_N}O_\gamma^{n_\gamma}-\operatorname{ev}_0\left\{\left[\prod_{\gamma\in\Gamma_N}F_\gamma(m)^{n_\gamma}\right]_N\right\}\right\|_q
\le e^{C(1+q)N^2}p^{-(N+1)/2}. γ ∈ Γ N ∏ O γ n γ − ev 0 ⎩ ⎨ ⎧ γ ∈ Γ N ∏ F γ ( m ) n γ N ⎭ ⎬ ⎫ q ≤ e C ( 1 + q ) N 2 p − ( N + 1 ) /2 .
The coefficient bound for R i ( m ) R_i(m) R i ( m ) and R i j ( m ) R_{ij}(m) R ij ( m ) therefore gives
∥ R i ( O ) − ev 0 { [ R i ( F ( m ) ) ] N } ∥ q ≤ e C ′ ( 1 + q ) N 2 p − ( N + 1 ) / 2 . (67) \|R_i(O)-\operatorname{ev}_0\{[R_i(F(m))]_N\}\|_q
\le e^{C'(1+q)N^2}p^{-(N+1)/2}. \tag*{(67)} ∥ R i ( O ) − ev 0 {[ R i ( F ( m )) ] N } ∥ q ≤ e C ′ ( 1 + q ) N 2 p − ( N + 1 ) /2 . (67)
∥ R i j ( O ) − ev 0 { [ R i j ( F ( m ) ) ] N } ∥ q ≤ e C ′ ( 1 + q ) N 2 p − ( N + 1 ) / 2 . (68) \|R_{ij}(O)-\operatorname{ev}_0\{[R_{ij}(F(m))]_N\}\|_q
\le e^{C'(1+q)N^2}p^{-(N+1)/2}. \tag*{(68)} ∥ R ij ( O ) − ev 0 {[ R ij ( F ( m )) ] N } ∥ q ≤ e C ′ ( 1 + q ) N 2 p − ( N + 1 ) /2 . (68)
Then for R i ( O ) − δ r i R_i(O)-\delta_{ri} R i ( O ) − δ r i , combining (C.21), Lemma B.5, and [ R i { F ( m ) } ] N = [ H i ( m ) ] N [R_i\{F(m)\}]_N=[H_i(m)]_N [ R i { F ( m )} ] N = [ H i ( m ) ] N , we obtain the bound as required in (B.35) under r = 0 r=0 r = 0 . For R i j ( O ) − E r { ( α r i r V r ) ( α r j r V r ) ∣ Y } R_{ij}(O)-\mathbb{E}_r\{(\alpha^r_{ri}V_r)(\alpha^r_{rj}V_r)\mid Y\} R ij ( O ) − E r {( α r i r V r ) ( α r j r V r ) ∣ Y } , combining (C.22) and [ R i j { F ( m ) } ] N = E f { ( α 0 i r v ) ( α 0 j r v ) } [R_{ij}\{F(m)\}]_N=\mathbb{E}_f\{(\alpha^r_{0i}v)(\alpha^r_{0j}v)\} [ R ij { F ( m )} ] N = E f {( α 0 i r v ) ( α 0 j r v )} proves the required bound in (B.35) under r = 0 r=0 r = 0 .
Finally, we show (B.35) under r = 1 r=1 r = 1 . Following the definition of F γ ( m ) F_\gamma(m) F γ ( m ) in (B.28), let
F γ ( r ) ( m ) = [ ∑ h 1 , … , h k = 1 d { ∏ s = 1 k ( B s − 1 ) ℓ s h s E f { ∏ s = 1 k G j h s ( r ) , N ( r ) ( v , m ) } } ] N . (69) F_\gamma^{(r)}(m)=\left[\sum_{h_1,\ldots,h_k=1}^{d}\left\{\prod_{s=1}^{k}(B_s^{-1})_{\ell_s h_s}\mathbb{E}_f\left\{\prod_{s=1}^{k}G_{j_{h_s}^{(r)},N}^{(r)}(v,m)\right\}\right\}\right]_N. \tag*{(69)} F γ ( r ) ( m ) = h 1 , … , h k = 1 ∑ d { s = 1 ∏ k ( B s − 1 ) ℓ s h s E f { s = 1 ∏ k G j h s ( r ) , N ( r ) ( v , m ) } } N . (69)
Here, k = ∣ γ ∣ k=|\gamma| k = ∣ γ ∣ ; B s − 1 B_s^{-1} B s − 1 are fixed at model zero; G i , N ( r ) ( v , m ) G_{i,N}^{(r)}(v,m) G i , N ( r ) ( v , m ) are the polynomials from Lemma B.5 using α r i \alpha_{ri} α r i . Then F γ ( 0 ) ( m ) = F γ ( m ) F_\gamma^{(0)}(m)=F_\gamma(m) F γ ( 0 ) ( m ) = F γ ( m ) . Under model one, F γ ( r ) ( m ) F_\gamma^{(r)}(m) F γ ( r ) ( m ) has leading weight-k k k component E f [ ∏ s = 1 k { ∑ j h = 1 d ( B s − 1 ) ℓ s h α 1 , j h r v } ] \mathbb{E}_f\left[\prod_{s=1}^{k}\left\{\sum_{j_h=1}^{d}(B_s^{-1})_{\ell_s h}\alpha^r_{1,j_h}v\right\}\right] E f [ ∏ s = 1 k { ∑ j h = 1 d ( B s − 1 ) ℓ s h α 1 , j h r v } ] . Similar to (B.30), we know ∥ [ F γ ( 1 ) ] w − [ F γ ( 1 ) ] w − 1 ∥ c o e f f ≤ C w \|[F_\gamma^{(1)}]_w-[F_\gamma^{(1)}]_{w-1}\|_{\mathrm{coeff}}\le C^w ∥ [ F γ ( 1 ) ] w − [ F γ ( 1 ) ] w − 1 ∥ coeff ≤ C w for k ≤ w ≤ N k\le w\le N k ≤ w ≤ N . In addition, to bound the coefficient in F γ ( 1 ) ( m ) − F γ ( 0 ) ( m ) F_\gamma^{(1)}(m)-F_\gamma^{(0)}(m) F γ ( 1 ) ( m ) − F γ ( 0 ) ( m ) , interpolate α i \alpha_i α i by α a ( λ ) = ( 1 − λ ) α 0 a + λ α 1 a \alpha_a(\lambda)=(1-\lambda)\alpha_{0a}+\lambda\alpha_{1a} α a ( λ ) = ( 1 − λ ) α 0 a + λ α 1 a and let G a , N ( λ ) G_{a,N}(\lambda) G a , N ( λ ) be the polynomial from Lemma B.5 with α a ( λ ) \alpha_a(\lambda) α a ( λ ) . Define F γ ( λ ) F_\gamma(\lambda) F γ ( λ ) by (C.23) with G a , N ( r ) G_{a,N}^{(r)} G a , N ( r ) replaced by G a , N ( λ ) G_{a,N}(\lambda) G a , N ( λ ) . Then
∥ { F γ ( 1 ) ( m ) − F γ ( 0 ) ( m ) } [ w ] − { F γ ( 1 ) ( m ) − F γ ( 0 ) ( m ) } [ w − 1 ] ∥ c o e f f ≤ ∫ 0 1 ∥ { d d λ F γ ( λ ) } [ w ] − { d d λ F γ ( λ ) } [ w − 1 ] ∥ c o e f f d λ ≤ C B k max h 1 , … , h k ∈ [ d ] ∑ s = 1 t 1 + ⋯ + t k = w t 1 , … , t k ≥ 1 k ∥ d d λ [ { G a s , N ( λ ) } [ t s ] − { G a s , N ( λ ) } [ t s − 1 ] ] ∥ c o e f f × ∏ b ≠ s ∥ { G a b , N ( λ ) } [ t b ] − { G a b , N ( λ ) } [ t b − 1 ] ∥ c o e f f . \begin{aligned}
\left\|\{F_\gamma^{(1)}(m)-F_\gamma^{(0)}(m)\}_{[w]}-\{F_\gamma^{(1)}(m)-F_\gamma^{(0)}(m)\}_{[w-1]}\right\|_{\mathrm{coeff}}
&\le\int_0^1\left\|\left\{\frac{d}{d\lambda}F_\gamma(\lambda)\right\}_{[w]}-\left\{\frac{d}{d\lambda}F_\gamma(\lambda)\right\}_{[w-1]}\right\|_{\mathrm{coeff}}\,d\lambda\\
&\le C_B^k\max_{h_1,\ldots,h_k\in[d]}\sum_{\substack{s=1\\t_1+\cdots+t_k=w\\t_1,\ldots,t_k\ge1}}^k\left\|\frac{d}{d\lambda}\left[\{G_{a_s,N}(\lambda)\}_{[t_s]}-\{G_{a_s,N}(\lambda)\}_{[t_s-1]}\right]\right\|_{\mathrm{coeff}}\\
&\qquad\times\prod_{b\ne s}\left\|\{G_{a_b,N}(\lambda)\}_{[t_b]}-\{G_{a_b,N}(\lambda)\}_{[t_b-1]}\right\|_{\mathrm{coeff}}.
\end{aligned} { F γ ( 1 ) ( m ) − F γ ( 0 ) ( m ) } [ w ] − { F γ ( 1 ) ( m ) − F γ ( 0 ) ( m ) } [ w − 1 ] coeff ≤ ∫ 0 1 { d λ d F γ ( λ ) } [ w ] − { d λ d F γ ( λ ) } [ w − 1 ] coeff d λ ≤ C B k h 1 , … , h k ∈ [ d ] max s = 1 t 1 + ⋯ + t k = w t 1 , … , t k ≥ 1 ∑ k d λ d [ { G a s , N ( λ ) } [ t s ] − { G a s , N ( λ ) } [ t s − 1 ] ] coeff × b = s ∏ { G a b , N ( λ ) } [ t b ] − { G a b , N ( λ ) } [ t b − 1 ] coeff .
Here, a s = j h s ( s ) a_s=j_{h_s}^{(s)} a s = j h s ( s ) and the second inequality uses ∏ s = 1 k ∑ h = 1 d ∣ ( B s − 1 ) ℓ s h ∣ ≤ C B k \prod_{s=1}^k\sum_{h=1}^d\left|(B_s^{-1})_{\ell_sh}\right|\le C_B^k ∏ s = 1 k ∑ h = 1 d ( B s − 1 ) ℓ s h ≤ C B k . Since ∥ α a ( λ ) ∥ ≤ M \|\alpha_a(\lambda)\|\le M ∥ α a ( λ ) ∥ ≤ M , with the coefficient and derivative bound in Lemma B.5, one can show ∥ G a , N ( λ ) [ t ] ∥ c o e f f ≤ C t \|G_{a,N}(\lambda)_{[t]}\|_{\mathrm{coeff}}\le C^t ∥ G a , N ( λ ) [ t ] ∥ coeff ≤ C t . The chain rules gives
∥ d d λ [ { G a s , N ( λ ) } [ t s ] − { G a s , N ( λ ) } [ t s − 1 ] ] ∥ c o e f f ≤ ∑ ℓ = 1 d ∣ α 1 a s , ℓ − α 0 a s , ℓ ∣ ∥ ∂ α a s ℓ [ { G a s , N ( λ ) } [ t s ] − { G a s , N ( λ ) } [ t s − 1 ] ] ∥ c o e f f ≤ D p C t s , \begin{aligned}
\left\|\frac{d}{d\lambda}\left[\{G_{a_s,N}(\lambda)\}_{[t_s]}-\{G_{a_s,N}(\lambda)\}_{[t_s-1]}\right]\right\|_{\mathrm{coeff}}
&\le\sum_{\ell=1}^d|\alpha_{1a_s,\ell}-\alpha_{0a_s,\ell}|\left\|\partial_{\alpha_{a_s\ell}}\left[\{G_{a_s,N}(\lambda)\}_{[t_s]}-\{G_{a_s,N}(\lambda)\}_{[t_s-1]}\right]\right\|_{\mathrm{coeff}}\\
&\le D_pC^{t_s},
\end{aligned} d λ d [ { G a s , N ( λ ) } [ t s ] − { G a s , N ( λ ) } [ t s − 1 ] ] coeff ≤ ℓ = 1 ∑ d ∣ α 1 a s , ℓ − α 0 a s , ℓ ∣ ∂ α a s ℓ [ { G a s , N ( λ ) } [ t s ] − { G a s , N ( λ ) } [ t s − 1 ] ] coeff ≤ D p C t s ,
where we denote D p = D p ( θ 0 , θ 1 ) D_p=D_p(\theta_0,\theta_1) D p = D p ( θ 0 , θ 1 ) . Therefore,
∥ { F γ ( 1 ) ( m ) − F γ ( 0 ) ( m ) } [ w ] − { F γ ( 1 ) ( m ) − F γ ( 0 ) ( m ) } [ w − 1 ] ∥ c o e f f ≤ D p C w . (70) \left\|\{F_\gamma^{(1)}(m)-F_\gamma^{(0)}(m)\}_{[w]}-\{F_\gamma^{(1)}(m)-F_\gamma^{(0)}(m)\}_{[w-1]}\right\|_{\mathrm{coeff}}\le D_pC^w. \tag*{(70)} { F γ ( 1 ) ( m ) − F γ ( 0 ) ( m ) } [ w ] − { F γ ( 1 ) ( m ) − F γ ( 0 ) ( m ) } [ w − 1 ] coeff ≤ D p C w . (70)
Write P i j ( r ) ( m ) = E f { ( α r i v ) ( α r j v ) } P_{ij}^{(r)}(m)=\mathbb{E}_f\{(\alpha_{ri}^v)(\alpha_{rj}^v)\} P ij ( r ) ( m ) = E f {( α r i v ) ( α r j v )} for short. Then by definition,
[ { R i ( F ( 1 ) ( m ) ) } ] N − [ H i ( 1 ) ( m ) ] N = [ { R i ( F ( 1 ) ( m ) ) } − { R i ( F ( 0 ) ( m ) ) } ] N − [ H i ( 1 ) ( m ) − H i ( 0 ) ( m ) ] N , [\{R_i(F^{(1)}(m))\}]_N-[H_i^{(1)}(m)]_N=[\{R_i(F^{(1)}(m))\}-\{R_i(F^{(0)}(m))\}]_N-[H_i^{(1)}(m)-H_i^{(0)}(m)]_N, [{ R i ( F ( 1 ) ( m ))} ] N − [ H i ( 1 ) ( m ) ] N = [{ R i ( F ( 1 ) ( m ))} − { R i ( F ( 0 ) ( m ))} ] N − [ H i ( 1 ) ( m ) − H i ( 0 ) ( m ) ] N ,
[ R i j { F ( 1 ) ( m ) } ] N − P i j ( 1 ) ( m ) = [ R i j { F ( 1 ) ( m ) } − R i j { F ( 0 ) ( m ) } ] N − { P i j ( 1 ) ( m ) − P i j ( 0 ) ( m ) } . [R_{ij}\{F^{(1)}(m)\}]_N-P_{ij}^{(1)}(m)=[R_{ij}\{F^{(1)}(m)\}-R_{ij}\{F^{(0)}(m)\}]_N-\{P_{ij}^{(1)}(m)-P_{ij}^{(0)}(m)\}. [ R ij { F ( 1 ) ( m )} ] N − P ij ( 1 ) ( m ) = [ R ij { F ( 1 ) ( m )} − R ij { F ( 0 ) ( m )} ] N − { P ij ( 1 ) ( m ) − P ij ( 0 ) ( m )} .
We first show that the evaluated value of the two polynomials are bounded by C q D p / p C_qD_p/p C q D p / p , and then establish that replacing m γ m_\gamma m γ by O γ O_\gamma O γ introduces errors bounded by e C ( 1 + q ) N 2 p − ( N + 1 ) / 2 e^{C(1+q)N^2}p^{-(N+1)/2} e C ( 1 + q ) N 2 p − ( N + 1 ) /2 .
First, note that ∏ a = 1 b F γ a ( 1 ) − ∏ a = 1 b F γ a ( 0 ) = ∑ a = 1 b ( F γ a ( 1 ) − F γ a ( 0 ) ) ∏ c < a F γ c ( 1 ) ∏ c > a F γ c ( 0 ) \prod_{a=1}^bF_{\gamma_a}^{(1)}-\prod_{a=1}^bF_{\gamma_a}^{(0)}=\sum_{a=1}^b(F_{\gamma_a}^{(1)}-F_{\gamma_a}^{(0)})\prod_{c<a}F_{\gamma_c}^{(1)}\prod_{c>a}F_{\gamma_c}^{(0)} ∏ a = 1 b F γ a ( 1 ) − ∏ a = 1 b F γ a ( 0 ) = ∑ a = 1 b ( F γ a ( 1 ) − F γ a ( 0 ) ) ∏ c < a F γ c ( 1 ) ∏ c > a F γ c ( 0 ) . Combining ∥ { R i ( m ) } [ w ] − { R i ( m ) } [ w − 1 ] ∥ c o e f f ≤ e C w 2 \|\{R_i(m)\}_{[w]}-\{R_i(m)\}_{[w-1]}\|_{\mathrm{coeff}}\le e^{Cw^2} ∥ { R i ( m ) } [ w ] − { R i ( m ) } [ w − 1 ] ∥ coeff ≤ e C w 2 , ∥ { R i j ( m ) } [ w ] − { R i j ( m ) } [ w − 1 ] ∥ c o e f f ≤ e C w 2 \|\{R_{ij}(m)\}_{[w]}-\{R_{ij}(m)\}_{[w-1]}\|_{\mathrm{coeff}}\le e^{Cw^2} ∥ { R ij ( m ) } [ w ] − { R ij ( m ) } [ w − 1 ] ∥ coeff ≤ e C w 2 , and the coefficient bounds for F γ ( r ) F_\gamma^{(r)} F γ ( r ) , F γ ( 1 ) − F γ ( 0 ) F_\gamma^{(1)}-F_\gamma^{(0)} F γ ( 1 ) − F γ ( 0 ) , we know the coefficients of weight w w w in ∏ a = 1 b F γ a ( 1 ) − ∏ a = 1 b F γ a ( 0 ) \prod_{a=1}^bF_{\gamma_a}^{(1)}-\prod_{a=1}^bF_{\gamma_a}^{(0)} ∏ a = 1 b F γ a ( 1 ) − ∏ a = 1 b F γ a ( 0 ) can be bounded by D p e C w 2 D_pe^{Cw^2} D p e C w 2 , as only components of weight at most w w w are involved. With Lemma B.5 and P i j ( r ) P_{ij}^{(r)} P ij ( r ) having leading term at least weight 2, we can also show that the weight-w w w components’ coefficients in { R i ( F ( 1 ) ( m ) ) } N − [ H i ( 1 ) ( m ) ] N \{R_i(F^{(1)}(m))\}_N-[H_i^{(1)}(m)]_N { R i ( F ( 1 ) ( m )) } N − [ H i ( 1 ) ( m ) ] N and [ R i j { F ( 1 ) ( m ) } ] N − P i j ( 1 ) ( m ) [R_{ij}\{F^{(1)}(m)\}]_N-P_{ij}^{(1)}(m) [ R ij { F ( 1 ) ( m )} ] N − P ij ( 1 ) ( m ) can be bounded by D p e C w 2 D_p e^{Cw^2} D p e C w 2 uniformly for 2 ≤ w ≤ N 2 \le w \le N 2 ≤ w ≤ N . With (B.14), we conclude that
∥ ev 1 ( [ R i { F ( 1 ) ( m ) } ] N − [ H i ( 1 ) ] N ) ∥ q + ∥ ev 1 ( [ R i j { F ( 1 ) ( m ) } ] N − P i j ( 1 ) ( m ) ) ∥ q ≤ ∑ w = 2 N D p e C w 2 e C q w 2 p − w / 2 ≤ C ′ D p p . (71) \left\|\operatorname{ev}_1\left(\left[R_i\{F^{(1)}(m)\}\right]_N-\left[H_i^{(1)}\right]_N\right)\right\|_q+\left\|\operatorname{ev}_1\left(\left[R_{ij}\{F^{(1)}(m)\}\right]_N-P_{ij}^{(1)}(m)\right)\right\|_q
\le\sum_{w=2}^{N}D_p e^{Cw^2}e^{Cqw^2}p^{-w/2}\le C'\frac{D_p}{p}. \tag*{(71)} ev 1 ( [ R i { F ( 1 ) ( m )} ] N − [ H i ( 1 ) ] N ) q + ev 1 ( [ R ij { F ( 1 ) ( m )} ] N − P ij ( 1 ) ( m ) ) q ≤ w = 2 ∑ N D p e C w 2 e C q w 2 p − w /2 ≤ C ′ p D p . (71)
Finally, to replace m γ m_\gamma m γ with O γ O_\gamma O γ , we note that since the conditional independent holds also under model one, (B.26) still hold with r = 1 r=1 r = 1 , and therefore with Lemmas B.5 and B.4 , we know
∥ O γ − ev 1 ( F γ ( 1 ) ) ∥ q ≤ e C ( 1 + q ) N 2 p − ( N + 1 ) / 2 . \left\|O_\gamma-\operatorname{ev}_1(F_\gamma^{(1)})\right\|_q\le e^{C(1+q)N^2}p^{-(N+1)/2}. O γ − ev 1 ( F γ ( 1 ) ) q ≤ e C ( 1 + q ) N 2 p − ( N + 1 ) /2 .
Each F γ ( 1 ) F_\gamma^{(1)} F γ ( 1 ) starts at weight ∣ γ ∣ |\gamma| ∣ γ ∣ and has the coefficient bounds established above, and (B.29), (B.31) analogously hold for F γ ( 1 ) F_\gamma^{(1)} F γ ( 1 ) . Repeating the monomial substitution argument leading to (C.21), (C.22) yields
∥ R i ( O ) − ev 1 ( [ R i { F ( 1 ) ( m ) } ] N ) ∥ q ≤ e C ( 1 + q ) N 2 p − ( N + 1 ) / 2 ; \left\|R_i(O)-\operatorname{ev}_1\left(\left[R_i\{F^{(1)}(m)\}\right]_N\right)\right\|_q\le e^{C(1+q)N^2}p^{-(N+1)/2}; R i ( O ) − ev 1 ( [ R i { F ( 1 ) ( m )} ] N ) q ≤ e C ( 1 + q ) N 2 p − ( N + 1 ) /2 ;
∥ R i j ( O ) − ev 1 ( [ R i j { F ( 1 ) ( m ) } ] N ) ∥ q ≤ e C ( 1 + q ) N 2 p − ( N + 1 ) / 2 ; \left\|R_{ij}(O)-\operatorname{ev}_1\left(\left[R_{ij}\{F^{(1)}(m)\}\right]_N\right)\right\|_q\le e^{C(1+q)N^2}p^{-(N+1)/2}; R ij ( O ) − ev 1 ( [ R ij { F ( 1 ) ( m )} ] N ) q ≤ e C ( 1 + q ) N 2 p − ( N + 1 ) /2 ;
For R i R_i R i , combine this with (C.25) and the approximation of δ 1 i \delta_{1i} δ 1 i by ev 1 { [ H i ( 1 ) ] N } \operatorname{ev}_1\{[H_i^{(1)}]_N\} ev 1 {[ H i ( 1 ) ] N } from Lemma B.5 yields the required bound under r = 1 r=1 r = 1 . For R i j R_{ij} R ij , recall that ev 1 ( P i j ( 1 ) ) = E 1 { ( α 1 i V 1 ) ( α 1 j V 1 ) ∣ Y } \operatorname{ev}_1(P_{ij}^{(1)})=\mathbb{E}_1\{(\alpha_{1i}V_1)(\alpha_{1j}V_1)\mid Y\} ev 1 ( P ij ( 1 ) ) = E 1 {( α 1 i V 1 ) ( α 1 j V 1 ) ∣ Y } . Use (C.25) and we prove (B.35) under r = 1 r=1 r = 1 .
Additional Simulation Details
This section describes how the quantities reported in § 4 are computed. The computation has two steps. First, for each candidate θ ~ \widetilde{\theta} θ , we approximate the discrepancy Δ p ( θ ~ ) \Delta_p(\widetilde{\theta}) Δ p ( θ ) in (9) by a penalized nonnegative least-squares fit over the masses that G ~ \widetilde{G} G places on the grid, with the constraints of G g r i d , d \mathcal{G}_{\mathrm{grid},d} G grid , d imposed through a penalty. To distinguish exact equivalence from very small positive discrepancies under the cases with d = 1 d=1 d = 1 , p = 7 , 8 , 9 p=7,8,9 p = 7 , 8 , 9 and d = 3 d=3 d = 3 , p = 9 p=9 p = 9 , we check directly by verifying whether admissible grid masses can reproduce P θ 0 , G 0 P_{\theta_0,G_0} P θ 0 , G 0 , while accounting for rounding error. Second, along each perturbation path θ ~ t \widetilde{\theta}_t θ t , we classify each evaluated value of t t t as numerically equivalent or nonequivalent using these computations, and locate the transition point t p t_p t p by a coarse search followed by bisection in log 10 t \log_{10}t log 10 t . We describe the two steps in turn.
Let z 1 , … , z J z_1,\ldots,z_J z 1 , … , z J be the points of the grid supporting G g r i d , d \mathcal{G}_{\mathrm{grid},d} G grid , d , let p 0 = p θ 0 , G 0 p_0=p_{\theta_0,G_0} p 0 = p θ 0 , G 0 , and for a candidate θ ~ \widetilde{\theta} θ define the 2 p × J 2^p\times J 2 p × J matrix B θ ~ = { Pr θ ~ ( X = x ∣ Z = z ℓ ) } x ∈ { 0 , 1 } p , ℓ ∈ [ J ] B_{\widetilde{\theta}}=\{\Pr_{\widetilde{\theta}}(X=x\mid Z=z_\ell)\}_{x\in\{0,1\}^p,\ell\in[J]} B θ = { Pr θ ( X = x ∣ Z = z ℓ ) } x ∈ { 0 , 1 } p , ℓ ∈ [ J ] . A latent distribution with nonnegative masses w = ( w 1 , … , w J ) T w=(w_1,\ldots,w_J)^\mathsf{T} w = ( w 1 , … , w J ) T on the grid then has response probability vector B θ ~ w B_{\widetilde{\theta}}w B θ w . Membership in G g r i d , d \mathcal{G}_{\mathrm{grid},d} G grid , d imposes the linear constraints
∑ ℓ = 1 J w ℓ = 1 , ∑ ℓ = 1 J w ℓ z ℓ = 0 , ∑ ℓ = 1 J w ℓ z ℓ z ℓ T = I d , ∑ ℓ = 1 J w ℓ exp ( ∥ z ℓ ∥ 2 / 4 ) + s = 2 ( d + 1 ) / 2 . \sum_{\ell=1}^{J} w_{\ell}=1,\qquad\sum_{\ell=1}^{J}w_{\ell}z_{\ell}=0,\qquad\sum_{\ell=1}^{J}w_{\ell}z_{\ell}z_{\ell}^{\mathsf{T}}=I_d,\qquad\sum_{\ell=1}^{J}w_{\ell}\exp\left(\lVert z_{\ell}\rVert^2/4\right)+s=2^{(d+1)/2}. ℓ = 1 ∑ J w ℓ = 1 , ℓ = 1 ∑ J w ℓ z ℓ = 0 , ℓ = 1 ∑ J w ℓ z ℓ z ℓ T = I d , ℓ = 1 ∑ J w ℓ exp ( ∥ z ℓ ∥ 2 /4 ) + s = 2 ( d + 1 ) /2 .
where the first three constraints encode (3) , and the slack variable s ≥ 0 s \ge0 s ≥ 0 turns the tail bound into an equality. We write these R = 2 + d + d ( d + 1 ) / 2 R=2+d+d(d+1)/2 R = 2 + d + d ( d + 1 ) /2 constraints compactly as C w + s e R = b Cw+se_R=b C w + s e R = b , where e R = ( 0 , … , 0 , 1 ) T ∈ R R e_R=(0,\ldots,0,1)^{\mathsf{T}}\in\mathbb{R}^{R} e R = ( 0 , … , 0 , 1 ) T ∈ R R is the last standard basis vector. To put the constraints on a common scale, let D = diag ( a 1 − 1 , … , a R − 1 ) D=\operatorname{diag}(a_1^{-1},\ldots,a_R^{-1}) D = diag ( a 1 − 1 , … , a R − 1 ) , where a r a_r a r is the largest absolute coefficient in row r r r of [ C , e R ] [C,e_R] [ C , e R ] . We then replace the constrained minimization in (9) by the penalized problem
min w ∈ [ 0 , ∞ ) J , s ≥ 0 { ∥ B θ ~ w − p 0 ∥ 2 ∥ p 0 ∥ 2 + λ 2 ∥ D ( C w + s e R − b ) ∥ 2 } . (72) \min_{w\in[0,\infty)^J,\ s\ge0}\left\{\frac{\lVert B_{\widetilde{\theta}}w-p_0\rVert^2}{\lVert p_0\rVert^2}+\lambda^2\lVert D(Cw+se_R-b)\rVert^2\right\}. \tag*{(72)} w ∈ [ 0 , ∞ ) J , s ≥ 0 min { ∥ p 0 ∥ 2 ∥ B θ w − p 0 ∥ 2 + λ 2 ∥ D ( C w + s e R − b ) ∥ 2 } . (72)
Because the objective is a sum of squares of affine functions of ( w , s ) (w,s) ( w , s ) and the only remaining constraints are nonnegativity, problem (D.26) is a nonnegative least-squares problem. We solve it by the active-set method of Lawson and Hanson [34 ] , with λ = 10 15 \lambda=10^{15} λ = 1 0 15 for d = 1 d=1 d = 1 and λ = 10 10 \lambda=10^{10} λ = 1 0 10 for d = 3 d=3 d = 3 . All fits use the fixed grids described in § 4 and are computed in 1500-bit arithmetic for d = 1 d=1 d = 1 and 600-bit arithmetic for d = 3 d=3 d = 3 . Let ( w ^ , s ^ ) (\widehat{w},\widehat{s}) ( w , s ) denote the masses and slack returned by the solver, and define
Δ ^ p ( θ ~ ) = ∥ B θ ~ w ^ − p 0 ∥ ∥ p 0 ∥ . (73) \widehat{\Delta}_p(\widetilde{\theta})=\frac{\lVert B_{\widetilde{\theta}}\widehat{w}-p_0\rVert}{\lVert p_0\rVert}. \tag*{(73)} Δ p ( θ ) = ∥ p 0 ∥ ∥ B θ w − p 0 ∥ . (73)
Since a finite penalty does not enforce the constraints exactly, Δ ^ p \widehat{\Delta}_p Δ p is a numerical approximation to Δ p \Delta_p Δ p .
For the penalized least-squares calculations, we classify an evaluated value t t t as numerically equivalent if Δ ^ p ( θ ~ t ) 2 ≤ τ \widehat{\Delta}_p(\widetilde{\theta}_t)^2\le\tau Δ p ( θ t ) 2 ≤ τ and as nonequivalent otherwise, with τ ≈ 10 − 301 \tau\approx10^{-301} τ ≈ 1 0 − 301 for d = 1 d=1 d = 1 and τ ≈ 10 − 120 \tau\approx10^{-120} τ ≈ 1 0 − 120 for d = 3 d=3 d = 3 . For d = 1 d=1 d = 1 , p = 7 , 8 , 9 p=7,8,9 p = 7 , 8 , 9 and d = 3 d=3 d = 3 , p = 9 p=9 p = 9 , we instead classify t t t as numerically equivalent when we verify that nonnegative grid masses satisfy all the constraints, and as nonequivalent when we verify that no such masses exist. These checks account for rounding error.
To estimate t p t_p t p , we search over a = log 10 t a=\log_{10}t a = log 10 t in two stages. In the coarse stage, the search starts at a = 0 a=0 a = 0 and decreases a a a until it finds a numerically equivalent point or reaches a prescribed lower search limit. Once two consecutive points receive different classifications, write their log values as a L < a U a_L<a_U a L < a U , with the lower point numerically equivalent and the upper point nonequivalent. In the refinement stage, each bisection step evaluates the midpoint a M = ( a L + a U ) / 2 a_M=(a_L+a_U)/2 a M = ( a L + a U ) /2 , corresponding to t = 10 a M t=10^{a_M} t = 1 0 a M , and replaces a L a_L a L by a M a_M a M if this point is numerically equivalent and a U a_U a U by a M a_M a M otherwise. Bisection continues until a U − a L ≤ 0.1 a_U-a_L\le0.1 a U − a L ≤ 0.1 .