Introduction
Many statistical and machine learning parameters have geometric constraints: directions lie on spheres, covariance matrices and diffusion tensors in the symmetric positive definite (SPD) matrix cone, and principal subspaces on a Grassmann manifold. These lead to population problems
x ⋆ ∈ arg min x ∈ M F ( x ) , F ( x ) = E { ℓ ( x ; Z ) } , (1) x_{\star} \in\arg\min_{x \in\mathcal{M}} F(x), \qquad F(x) = \mathbb{E}\{\ell(x; Z)\}, \tag*{(1)} x ⋆ ∈ arg x ∈ M min F ( x ) , F ( x ) = E { ℓ ( x ; Z )} , (1)
where ( M , g ) (\mathcal{M}, g) ( M , g ) is a finite-dimensional Riemannian manifold. With sequential observations or a single sample pass, Riemannian stochastic gradients naturally estimate (1) [5 , 11 , 31 ] . Their optimization and averaging theory is well developed [1 , 6 ] .
Inference is less direct. For an unbiased stochastic Riemannian gradient G ( x , Z ) G(x, Z) G ( x , Z ) , write in the tangent space at a regular minimizer, using ⊗ \otimes ⊗ for tangent outer products,
H = Hess F ( x ⋆ ) , Ω = H − 1 Σ H − 1 , H = \operatorname{Hess} F(x_{\star}), \qquad\Omega= H^{-1}\Sigma H^{-1}, H = Hess F ( x ⋆ ) , Ω = H − 1 Σ H − 1 ,
Σ = E { G ( x ⋆ , Z ) ⊗ G ( x ⋆ , Z ) } . (2) \Sigma= \mathbb{E}\{G(x_{\star}, Z) \otimes G(x_{\star}, Z)\}. \tag*{(2)} Σ = E { G ( x ⋆ , Z ) ⊗ G ( x ⋆ , Z )} . (2)
Averaged stochastic approximation has covariance (2) [27 , 31 ] . Plug-in Wald inference estimates H H H and Σ \Sigma Σ in a common tangent space. This adds a second-order calculation to a first-order optimizer. For affine-invariant matrix means, it involves differentiating a matrix logarithm; for conditional-median sensing, the population Hessian contains an unknown residual density. Even an analytic observation Hessian can be difficult to average accurately: in multi-frequency circular estimation its magnitude grows with frequency while the stochastic gradient remains bounded. A path-based procedure avoids these additional Hessian calculations and their estimation variability.
Self-normalization uses the same partial sum process in numerator and denominator, cancelling the unknown linear covariance factor in Euclidean time series and stochastic approximation [21 , 28 , 29 ] . On a manifold, constructing such sums requires common coordinates and control of nonlinear coordinate errors accumulated along the gradient path.
Classical stochastic approximation and Polyak–Ruppert averaging provide the underlying estimation theory [19 , 27 ] . Riemannian results establish convergence, averaging, streaming k k k -PCA, and fixed-step approximation [5 , 11 , 31 ] . Euclidean online inference uses covariance estimates and batch means [7 ] , perturbed trajectories [14 ] , or HiGrad branching [30 ] . Random scaling instead uses a single path [21 , 22 ] , including stochastic subgradient quantile regression where it avoids residual-density estimation [20 ] . Our construction reduces exactly to Euclidean random scaling; the additional difficulty is preserving its common covariance factor through nonlinear coordinates and a random tangent anchor.
Inference from manifold observations includes intrinsic and extrinsic mean limits [4 ] , distributions meeting the cut locus [17 ] , and empirical likelihood for Fréchet means [18 ] . Here the inferential input is the dependent trajectory of an online optimizer, whose covariance is determined by stochastic gradients.
To construct inference from a single path, we express retained iterates in the tangent space at a burn-in iterate and accumulate their average and centered partial sums in one pass. The central difficulty is to show that nonlinear coordinates preserve the common covariance factor needed for self-normalization. We address this through a joint Polyak–Ruppert expansion of the averaged estimator and integrated path, controlling the nonlinear remainders with conditional second moments under the localization, stability, and martingale conditions in § 3.1 . This expansion yields a Brownian bridge pivot depending only on dimension, and hence confidence regions without Hessian or gradient-noise covariance estimation. Because the same cancellation survives smooth projection and changes of local coordinates, the construction also provides inference for interpretable contrasts such as squared principal-component loadings.
The rest of the paper is organized as follows. Section 2 gives the methodology, Section 3 presents the theoretical results, Section 4 reports simulations and a data illustration, and Section 5 discusses implications. The appendices contain additional experiments, geometric specializations, technical lemmas, and proofs.
Methodology
Let ( M , g ) (\mathcal{M},g) ( M , g ) be a connected d d d -dimensional Riemannian manifold, with tangent inner product ⟨ ⋅ , ⋅ ⟩ x \langle\cdot,\cdot\rangle_x ⟨ ⋅ , ⋅ ⟩ x and norm ∥ ⋅ ∥ x \|\cdot\|_x ∥ ⋅ ∥ x . We write I \mathrm{I} I for the identity operator, ∥ ⋅ ∥ o p \|\cdot\|_{\mathrm{op}} ∥ ⋅ ∥ op for the operator norm, and vec \operatorname{vec} vec for column-stacking of a matrix. On a strongly convex normal neighborhood, Exp x \operatorname{Exp}_x Exp x is the exponential map, Log x \operatorname{Log}_x Log x its local inverse, and P x → y \mathcal{P}_{x\to y} P x → y parallel transport along the minimizing geodesic.
Recursion and averaging in a fixed chart
Let F k = σ ( Z 1 , … , Z k , x 0 ) \mathcal{F}_k=\sigma(Z_1,\ldots,Z_k,x_0) F k = σ ( Z 1 , … , Z k , x 0 ) and let G k + 1 G_{k+1} G k + 1 be an adapted unbiased gradient field: E [ G k + 1 ( x ) ∣ F k ] = grad F ( x ) \mathbb{E}[G_{k+1}(x)\mid\mathcal{F}_k]=\operatorname{grad}F(x) E [ G k + 1 ( x ) ∣ F k ] = grad F ( x ) . For i.i.d. observations, G k + 1 ( x ) = G ( x , Z k + 1 ) G_{k+1}(x)=G(x,Z_{k+1}) G k + 1 ( x ) = G ( x , Z k + 1 ) . We use
x k + 1 = R x k { − η k + 1 G k + 1 ( x k ) } , η k = η 0 k − β , (3) x_{k+1}=\mathcal{R}_{x_k}\{-\eta_{k+1}G_{k+1}(x_k)\},\qquad\eta_k=\eta_0 k^{-\beta}, \tag*{(3)} x k + 1 = R x k { − η k + 1 G k + 1 ( x k )} , η k = η 0 k − β , (3)
where η 0 > 0 \eta_0>0 η 0 > 0 and R x \mathcal{R}_x R x is a C 2 C^2 C 2 retraction, with R x ( 0 ) = x \mathcal{R}_x(0)=x R x ( 0 ) = x and D R x ( 0 ) = I D\mathcal{R}_x(0)=\mathrm{I} D R x ( 0 ) = I . The exponential map R x = Exp x \mathcal{R}_x=\operatorname{Exp}_x R x = Exp x is one such choice. Discard the first b n b_n b n iterates, where the deterministic burn-in satisfies
b n → ∞ , b n / n → 0 , N n = n − b n , (4) b_n\to\infty,\qquad b_n/n\to0,\qquad N_n=n-b_n, \tag*{(4)} b n → ∞ , b n / n → 0 , N n = n − b n , (4)
Thus the number of discarded iterates grows with n n n , while their fraction vanishes and N n / n → 1 N_n/n\to1 N n / n → 1 . This is the diverging burn-in used below. Set a n = x b n a_n=x_{b_n} a n = x b n to fix the tangent space for averaging; if the recursion converges to x ⋆ x_\star x ⋆ , this anchor is consistent. Assumptions 3.1 and 3.4 specify the localization and stability needed for the retained path. On the common normal neighborhood define
v k , n = Log a n ( x k ) ∈ T a n M , b n < k ≤ n , (5) v_{k,n}=\operatorname{Log}_{a_n}(x_k)\in T_{a_n}\mathcal{M},\qquad b_n<k\le n, \tag*{(5)} v k , n = Log a n ( x k ) ∈ T a n M , b n < k ≤ n , (5)
and the fixed-chart Polyak–Ruppert estimate
v ˉ n = N n − 1 ∑ k = b n + 1 n v k , n , x ^ n = Exp a n ( v ˉ n ) . \bar{v}_n=N_n^{-1}\sum_{k=b_n+1}^{n}v_{k,n},\qquad\hat{x}_n=\operatorname{Exp}_{a_n}(\bar{v}_n). v ˉ n = N n − 1 k = b n + 1 ∑ n v k , n , x ^ n = Exp a n ( v ˉ n ) .
Smooth inverse retractions provide alternative coordinates; Proposition B.4 establishes their first order equivalence.
Centered path and confidence region
For 0 ≤ m ≤ N n 0\le m\le N_n 0 ≤ m ≤ N n , with S 0 , n = C 0 , n = 0 S_{0,n}=C_{0,n}=0 S 0 , n = C 0 , n = 0 , define
S m , n = ∑ j = 1 m v b n + j , n , S_{m,n}=\sum_{j=1}^{m}v_{b_n+j,n}, S m , n = j = 1 ∑ m v b n + j , n ,
C m , n = S m , n − m N n S N n , n . C_{m,n}=S_{m,n}-\frac{m}{N_n}S_{N_n,n}. C m , n = S m , n − N n m S N n , n .
The centered sums give
V ^ n = N n − 2 ∑ m = 1 N n C m , n ⊗ C m , n , (6) \hat{V}_n=N_n^{-2}\sum_{m=1}^{N_n}C_{m,n}\otimes C_{m,n}, \tag*{(6)} V ^ n = N n − 2 m = 1 ∑ N n C m , n ⊗ C m , n , (6)
where ( u ⊗ v ) w = u ⟨ v , w ⟩ a n (u\otimes v)w=u\langle v,w\rangle_{a_n} ( u ⊗ v ) w = u ⟨ v , w ⟩ a n . Since the centered sums fluctuate on the N n \sqrt{N_n} N n scale, this normalization gives a nondegenerate limit. For a candidate x x x , define e n ( x ) = v ˉ n − Log a n ( x ) e_n(x)=\bar{v}_n-\operatorname{Log}_{a_n}(x) e n ( x ) = v ˉ n − Log a n ( x ) in orthonormal tangent coordinates and put
T n ( x ) = N n e n ( x ) T V ^ n − 1 e n ( x ) . (7) T_n(x)=N_n e_n(x)^{\mathsf{T}}\hat{V}_n^{-1}e_n(x). \tag*{(7)} T n ( x ) = N n e n ( x ) T V ^ n − 1 e n ( x ) . (7)
Let W d W_d W d be a standard d d d -dimensional Brownian motion on [ 0 , 1 ] [0,1] [ 0 , 1 ] and define
B d ( r ) = W d ( r ) − r W d ( 1 ) , B_d(r)=W_d(r)-rW_d(1), B d ( r ) = W d ( r ) − r W d ( 1 ) ,
Q d = ∫ 0 1 B d ( r ) B d ( r ) T d r , (8) Q_d=\int_0^1 B_d(r)B_d(r)^{\mathsf{T}}\,\mathrm{d}r, \tag*{(8)} Q d = ∫ 0 1 B d ( r ) B d ( r ) T d r , (8)
T d = W d ( 1 ) T Q d − 1 W d ( 1 ) . T_d=W_d(1)^{\mathsf{T}}Q_d^{-1}W_d(1). T d = W d ( 1 ) T Q d − 1 W d ( 1 ) .
For 0 < α < 1 0<\alpha<1 0 < α < 1 , let c d , 1 − α c_{d,1-\alpha} c d , 1 − α be the ( 1 − α ) (1-\alpha) ( 1 − α ) quantile of T d T_d T d and report
C n , 1 − α = { x : T n ( x ) ≤ c d , 1 − α } (9) C_{n,1-\alpha}=\{x:T_n(x)\le c_{d,1-\alpha}\} \tag*{(9)} C n , 1 − α = { x : T n ( x ) ≤ c d , 1 − α } (9)
within the chart domain. In tangent coordinates, (9) is an ellipsoid centered at v ˉ n \bar{v}_n v ˉ n . An eigenvalue λ ^ j \hat{\lambda}_j λ ^ j of V ^ n \hat{V}_n V ^ n gives semiaxis c d , 1 − α λ ^ j / N n \sqrt{c_{d,1-\alpha}\hat{\lambda}_j/N_n} c d , 1 − α λ ^ j / N n ; the exponential map sends this ellipsoid to the manifold.
Algorithm 1 Estimation and inference from a single path
Require: Initial point $x_0$, gradient stream, gains $\eta_k$, sample size $n$, burn-in $b_n$, and critical value $c_{d,1-\alpha}$.
Ensure: Estimate $\hat{x}_n$ and region $C_{n,1-\alpha}$.
1: Run (3) for $b_n$ steps.
2: Set $a_n \leftarrow x_{b_n}$ and $N \leftarrow n-b_n$.
3: Initialize $S=B=0\in T_{a_n}\mathcal{M}$ and $A=0$.
4: for $m=1,\ldots,N$ do
5: Continue (3) to obtain $x_{b_n+m}$.
6: $v_m\leftarrow\operatorname{Log}_{a_n}(x_{b_n+m})$.
7: $S\leftarrow S+v_m$.
8: $A\leftarrow A+S\otimes S$.
9: $B\leftarrow B+mS$.
10: end for
11: $\bar{v}_n\leftarrow S/N$ and $\hat{x}_n\leftarrow\operatorname{Exp}_{a_n}(\bar{v}_n)$.
12: Compute $\widehat{V}_n$ by (10).
13: return $\hat{x}_n$ and $C_{n,1-\alpha}$ from (9).
Computation in one pass
Write N = N n N=N_n N = N n , S m = S m , n S_m=S_{m,n} S m = S m , n , and maintain A N = ∑ m = 1 N S m ⊗ S m A_N=\sum_{m=1}^{N}S_m\otimes S_m A N = ∑ m = 1 N S m ⊗ S m and B N = ∑ m = 1 N m S m B_N=\sum_{m=1}^{N}mS_m B N = ∑ m = 1 N m S m . Expanding the centered products gives
V ^ n = 1 N 2 [ A N − B N ⊗ S N + S N ⊗ B N N + q N N 2 S N ⊗ S N ] , q N = N ( N + 1 ) ( 2 N + 1 ) 6 . (10) \widehat{V}_n=\frac{1}{N^2}\left[A_N-\frac{B_N\otimes S_N+S_N\otimes B_N}{N}+\frac{q_N}{N^2}S_N\otimes S_N\right],\qquad q_N=\frac{N(N+1)(2N+1)}{6}. \tag*{(10)} V n = N 2 1 [ A N − N B N ⊗ S N + S N ⊗ B N + N 2 q N S N ⊗ S N ] , q N = 6 N ( N + 1 ) ( 2 N + 1 ) . (10)
Each update uses O ( d 2 ) O(d^2) O ( d 2 ) arithmetic apart from evaluating the chart, and the accumulators use O ( d 2 ) O(d^2) O ( d 2 ) memory. Full-region inference adds a final matrix solve. Equation (10) avoids storing or replaying the trajectory. A vanishing nonnegative ridge preserves the pivot under nondegeneracy (Proposition B.2 ).
The Brownian critical values are shared across data sets of the same tangent dimension. The bridge has the Karhunen–Loève representation
Q d = ∑ j = 1 ∞ z j z j T ( π j ) 2 , T d = e T Q d − 1 e , Q_d=\sum_{j=1}^{\infty}\frac{z_jz_j^{\mathsf{T}}}{(\pi j)^2},\qquad T_d=e^{\mathsf{T}}Q_d^{-1}e, Q d = j = 1 ∑ ∞ ( π j ) 2 z j z j T , T d = e T Q d − 1 e ,
where e , z 1 , z 2 , … e,z_1,z_2,\ldots e , z 1 , z 2 , … are independent standard d d d -dimensional Gaussian vectors. Nested truncations with at least d d d modes are almost surely positive definite; their quadratic forms decrease to T d T_d T d .
For a truncation after J J J modes, the expected trace of the omitted bridge Gram terms is at most d / ( π 2 J ) d/(\pi^2J) d / ( π 2 J ) . This bounds the matrix tail. Nested truncation gives one-sided approximation of the pivot. The scalar experiments use the critical value c 1 , 0.95 c_{1,0.95} c 1 , 0.95 , displayed as 46.14. Appendix A.1 gives the simulation counts, truncation levels, and full-region critical values.
The tangent basis is arbitrary. For an orthogonal matrix O O O , the estimation error and normalizer become O e n ( x ) Oe_n(x) O e n ( x ) and O V ^ n O T O\widehat{V}_nO^{\mathsf{T}} O V n O T , leaving (7) unchanged. More generally, a jointly smooth family of uniformly C 2 C^2 C 2 local charts with uniformly nonsingular derivatives preserves the first order limit (Proposition B.4 ). This permits inverse-retraction or graph coordinates when they are convenient. In Euclidean space, v k , n = x k − a n v_{k,n}=x_k-a_n v k , n = x k − a n : centering removes the common anchor, x ^ n \hat{x}_n x ^ n is the arithmetic average, and Algorithm 1 reduces exactly to the random scaling procedure of Lee et al. [21 ] .
Wald inference estimates Ω = H − 1 Σ H − 1 \Omega= H^{-1}\Sigma H^{-1} Ω = H − 1 Σ H − 1 and uses chi-square critical values. Self-normalization retains the random path shape L Q d L T LQ_dL^{\mathsf{T}} L Q d L T , where L = H − 1 Σ 1 / 2 L = H^{-1}\Sigma^{1/2} L = H − 1 Σ 1/2 , and cancels L L L by combining numerator and denominator. Both procedures have root-N n N_n N n radii under their respective assumptions; Proposition B.5 gives their volume limits. The computational distinction is estimated Hessian and noise covariance versus centered trajectory accumulation.
Theoretical Results
Assumptions
The theory studies inference at an isolated local minimizer x ⋆ x_\star x ⋆ . Fix orthonormal coordinates on T x ⋆ M T_{x_\star}\mathcal{M} T x ⋆ M and write Δ k = Log x ⋆ ( x k ) \Delta_k = \operatorname{Log}_{x_\star}(x_k) Δ k = Log x ⋆ ( x k ) on the local chart, with an arbitrary measurable extension outside it. The following conditions identify the geometric and stochastic inputs; sufficient conditions for verifying them are given in Section B.1.
Assumption 3.1 (Local geometry and localization). There are strongly convex normal neighborhoods U 0 U_0 U 0 , U 1 U_1 U 1 with x ⋆ ∈ U 0 x_\star\in U_0 x ⋆ ∈ U 0 , U ‾ 0 ⊂ U 1 \overline{U}_0 \subset U_1 U 0 ⊂ U 1 , and U ‾ 1 \overline{U}_1 U 1 compact. Logarithms and parallel transport are smooth on the relevant pairs. For some r R > 0 r_R > 0 r R > 0 , the retraction is C 2 C^2 C 2 on { ( x , v ) : x ∈ U ‾ 0 , ∥ v ∥ x ≤ r R } \{(x,v) : x \in\overline{U}_0, \lVert v\rVert_x \le r_R\} {( x , v ) : x ∈ U 0 , ∥ v ∥ x ≤ r R } and maps this tube into U 1 U_1 U 1 . For b n b_n b n in (4),
P { x k ∈ U 0 for all b n ≤ k ≤ n } → 1. \mathbb{P}\{x_k \in U_0\text{ for all }b_n \le k \le n\} \to1. P { x k ∈ U 0 for all b n ≤ k ≤ n } → 1.
Assumption 3.2 (Regular local mean field). grad F ( x ⋆ ) = 0 \operatorname{grad} F(x_\star) = 0 grad F ( x ⋆ ) = 0 , and H = Hess F ( x ⋆ ) H = \operatorname{Hess} F(x_\star) H = Hess F ( x ⋆ ) is self-adjoint and positive definite. For some ρ ∈ ( 0 , 1 ] \rho\in(0,1] ρ ∈ ( 0 , 1 ] and L F < ∞ L_F < \infty L F < ∞ ,
∥ P x → x ⋆ grad F ( x ) − H Log x ⋆ ( x ) ∥ ≤ L F dist ( x , x ⋆ ) 1 + ρ , x ∈ U 1 . (11) \left\lVert P_{x\to x_\star}\operatorname{grad} F(x) - H\operatorname{Log}_{x_\star}(x)\right\rVert\le L_F\operatorname{dist}(x,x_\star)^{1+\rho}, \qquad x \in U_1. \tag*{(11)} P x → x ⋆ grad F ( x ) − H Log x ⋆ ( x ) ≤ L F dist ( x , x ⋆ ) 1 + ρ , x ∈ U 1 . (11)
Assumption 3.3 (Stochastic gradient). The gradient field is jointly F k + 1 ⊗ B ( U 1 ) \mathcal{F}_{k+1} \otimes\mathcal{B}(U_1) F k + 1 ⊗ B ( U 1 ) -measurable in local trivializations and conditionally unbiased. Put ϵ k + 1 ( x ) = G k + 1 ( x ) − grad F ( x ) \epsilon_{k+1}(x) = G_{k+1}(x) - \operatorname{grad} F(x) ϵ k + 1 ( x ) = G k + 1 ( x ) − grad F ( x ) and ξ k + 1 = ϵ k + 1 ( x ⋆ ) \xi_{k+1} = \epsilon_{k+1}(x_\star) ξ k + 1 = ϵ k + 1 ( x ⋆ ) . For C G < ∞ C_G < \infty C G < ∞ and ν > 0 \nu> 0 ν > 0 ,
E { ∥ G k + 1 ( x ) ∥ x 2 ∣ F k } ≤ C G , (12) \mathbb{E}\{\lVert G_{k+1}(x)\rVert_x^2 \mid\mathcal{F}_k\} \le C_G, \tag*{(12)} E {∥ G k + 1 ( x ) ∥ x 2 ∣ F k } ≤ C G , (12) E { ∥ P x → x ⋆ ϵ k + 1 ( x ) − ξ k + 1 ∥ 2 ∣ F k } ≤ C G dist ( x , x ⋆ ) 2 ν . (13) \mathbb{E}\{\lVert P_{x\to x_\star}\epsilon_{k+1}(x) - \xi_{k+1}\rVert^2 \mid\mathcal{F}_k\} \le C_G\operatorname{dist}(x,x_\star)^{2\nu}. \tag*{(13)} E {∥ P x → x ⋆ ϵ k + 1 ( x ) − ξ k + 1 ∥ 2 ∣ F k } ≤ C G dist ( x , x ⋆ ) 2 ν . (13) These assertions also hold at F k \mathcal{F}_k F k -measurable U 1 U_1 U 1 -valued random arguments.
Assumption 3.4 (Step size and stability in mean square). The gain exponent satisfies
max { 1 / 2 , 1 / ( 1 + ρ ) } < β < 1. (14) \max\{1/2,1/(1+\rho)\} < \beta< 1. \tag*{(14)} max { 1/2 , 1/ ( 1 + ρ )} < β < 1. (14) For a fixed smooth increment cutoff as in Lemma C.3 , independent of n n n , let Δ k , n l o c \Delta^{\mathrm{loc}}_{k,n} Δ k , n loc be the burn-in-reset auxiliary process defined in Section C.1.1, (C.9). It agrees with Δ k \Delta_k Δ k on the retained localization and small-increment event. Some C Δ < ∞ C_\Delta< \infty C Δ < ∞ and n 0 ∈ N n_0 \in\mathbb{N} n 0 ∈ N satisfy
sup n ≥ n 0 sup b n ≤ k ≤ n E ∥ Δ k , n l o c ∥ 2 η k ≤ C Δ . (15) \sup_{n\ge n_0}\sup_{b_n\le k\le n}\frac{\mathbb{E}\left\lVert\Delta^{\mathrm{loc}}_{k,n}\right\rVert^2}{\eta_k} \le C_\Delta. \tag*{(15)} n ≥ n 0 sup b n ≤ k ≤ n sup η k E Δ k , n loc 2 ≤ C Δ . (15)
Assumption 3.5 (Martingale invariance principle). ( ξ k + 1 , F k + 1 ) (\xi_{k+1},\mathcal{F}_{k+1}) ( ξ k + 1 , F k + 1 ) is a martingale difference sequence . Write Σ k + 1 = E ( ξ k + 1 ⊗ ξ k + 1 ∣ F k ) \Sigma_{k+1}=\mathbb{E}(\xi_{k+1}\otimes\xi_{k+1}\mid\mathcal{F}_k) Σ k + 1 = E ( ξ k + 1 ⊗ ξ k + 1 ∣ F k ) for its conditional covariance . For a nonnegative definite Σ \Sigma Σ ,
sup r ∈ [ 0 , 1 ] ∥ N n − 1 ∑ k = b n b n + ⌊ N n r ⌋ − 1 Σ k + 1 − r Σ ∥ o p → p 0. (16) \sup_{r\in[0,1]}\left\|N_n^{-1}\sum_{k=b_n}^{b_n+\lfloor N_n r\rfloor-1}\Sigma_{k+1}-r\Sigma\right\|_{\mathrm{op}}\xrightarrow{p}0. \tag*{(16)} r ∈ [ 0 , 1 ] sup N n − 1 k = b n ∑ b n + ⌊ N n r ⌋ − 1 Σ k + 1 − r Σ op p 0. (16) For each ε > 0 \varepsilon>0 ε > 0 , the conditional Lindeberg condition holds:
N n − 1 ∑ k = b n n − 1 E [ ∥ ξ k + 1 ∥ 2 1 { ∥ ξ k + 1 ∥ > ε N n } ∣ F k ] → p 0. (17) N_n^{-1}\sum_{k=b_n}^{n-1}\mathbb{E}\left[\|\xi_{k+1}\|^2\mathbf{1}\{\|\xi_{k+1}\|>\varepsilon\sqrt{N_n}\}\mid\mathcal{F}_k\right]\xrightarrow{p}0. \tag*{(17)} N n − 1 k = b n ∑ n − 1 E [ ∥ ξ k + 1 ∥ 2 1 { ∥ ξ k + 1 ∥ > ε N n } ∣ F k ] p 0. (17) The requirement β ( 1 + ρ ) > 1 \beta(1+\rho)>1 β ( 1 + ρ ) > 1 controls accumulated mean field errors; β > 1 / 2 \beta>1/2 β > 1/2 controls curvature and retraction errors; β < 1 \beta<1 β < 1 controls the averaging boundary. For a locally Lipschitz Hessian, ρ = 1 \rho=1 ρ = 1 , so the full range 1 / 2 < β < 1 1/2<\beta<1 1/2 < β < 1 is available. Lower Hölder smoothness requires a more rapidly decreasing gain; for example, ρ = 1 / 2 \rho=1/2 ρ = 1/2 gives 2 / 3 < β < 1 2/3<\beta<1 2/3 < β < 1 .
These assumptions can be checked separately for the model and the recursion. Almost sure convergence to x ⋆ x_\star x ⋆ gives retained localization for any diverging burn-in. For a smooth i.i.d. loss, the transported Hessian condition
∥ P x → x ⋆ Hess F ( x ) P x ⋆ → x − H ∥ o p ≤ L dist ( x , x ⋆ ) ρ . (18) \left\|P_{x\to x_\star}\operatorname{Hess}F(x)P_{x_\star\to x}-H\right\|_{\mathrm{op}}\le L\operatorname{dist}(x,x_\star)^\rho. \tag*{(18)} ∥ P x → x ⋆ Hess F ( x ) P x ⋆ → x − H ∥ op ≤ L dist ( x , x ⋆ ) ρ . (18) implies the mean-field expansion. A sufficient condition for noise continuity with ν = 1 \nu=1 ν = 1 is the square-integrable local Hessian envelope
E [ sup x ∈ U 1 ∥ Hess x ℓ ( x ; Z ) ∥ o p 2 ] < ∞ . (19) \mathbb{E}\left[\sup_{x\in U_1}\|\operatorname{Hess}_x\ell(x;Z)\|_{\mathrm{op}}^2\right]<\infty. \tag*{(19)} E [ x ∈ U 1 sup ∥ Hess x ℓ ( x ; Z ) ∥ op 2 ] < ∞. (19) Finite second moments of the true-point gradient then give the martingale covariance and Lindeberg conditions. These checks identify regularity of the statistical model; convergence to the selected minimizer and uniform mean square stability are properties of the stochastic recursion.
For stability, positive definiteness of H H H and the mean-field expansion give, on a sufficiently small neighborhood,
⟨ Δ , P x → x ⋆ grad F ( x ) ⟩ x ⋆ ≥ μ 0 ∥ Δ ∥ 2 , Δ = Log x ⋆ ( x ) , μ 0 > 0. (20) \langle\Delta,P_{x\to x_\star}\operatorname{grad}F(x)\rangle_{x_\star}\ge\mu_0\|\Delta\|^2,\qquad\Delta=\operatorname{Log}_{x_\star}(x),\quad\mu_0>0. \tag*{(20)} ⟨ Δ , P x → x ⋆ grad F ( x ) ⟩ x ⋆ ≥ μ 0 ∥Δ ∥ 2 , Δ = Log x ⋆ ( x ) , μ 0 > 0. (20) Together with conditional second moments, this yields the one-step drift bound (21) for the auxiliary update. Proposition 3.6 converts that drift and a uniform O ( η b n ) O(\eta_{b_n}) O ( η b n ) initial second moment into (15). Consistency of the anchor alone does not imply this initial moment bound. The distinction matters because the retained path starts at a random iterate: localization keeps it in a valid chart, whereas the moment rate controls the size of the errors accumulated in that chart. The uniformity in n n n ensures that the same bound remains valid as the burn-in point changes. Once initialized at the required rate, the drift contracts the existing error and the O ( η k + 1 2 ) O(\eta_{k+1}^2) O ( η k + 1 2 ) term accounts for fresh gradient noise. Appendix B.1 gives the verification details and comparison with existing averaging assumptions.
Proposition 3.6 (A drift bound for one update implies the stability rate). Suppose there is C b < ∞ C_b<\infty C b < ∞ such that, uniformly for all sufficiently large n n n ,
E [ ∥ Δ b n , n l o c ∥ 2 ] ≤ C b η b n , \mathbb{E}\left[\left\|\Delta^{\mathrm{loc}}_{b_n,n}\right\|^2\right]\le C_b\eta_{b_n}, E [ Δ b n , n loc 2 ] ≤ C b η b n ,
and, for b n ≤ k < n b_n \le k < n b n ≤ k < n ,
E ( ∥ Δ k + 1 , n l o c ∥ 2 ∣ F k ) ≤ ( 1 − c 0 η k + 1 ) ∥ Δ k , n l o c ∥ 2 + C 0 η k + 1 2 (21) \mathbb{E}\left(\left\|\Delta^{\mathrm{loc}}_{k+1,n}\right\|^2 \mid\mathcal{F}_k\right) \le(1-c_0\eta_{k+1})\left\|\Delta^{\mathrm{loc}}_{k,n}\right\|^2+C_0\eta_{k+1}^2 \tag*{(21)} E ( Δ k + 1 , n loc 2 ∣ F k ) ≤ ( 1 − c 0 η k + 1 ) Δ k , n loc 2 + C 0 η k + 1 2 (21)
for constants c 0 , C 0 > 0 c_0,C_0>0 c 0 , C 0 > 0 independent of n n n and k k k . If 0 < β < 1 0<\beta<1 0 < β < 1 , then (15) holds. A sufficient condition for (21) , for all sufficiently large n n n , is Assumptions 3.1–3.3 together with (20) on U ‾ 0 \overline{U}_0 U 0 .
Partial sum limit
By Lemma C.4, the measurable extension of Δ k \Delta_k Δ k does not affect the limits below. Define the retained partial sum process
D n ( r ) = 1 N n ∑ k = b n + 1 b n + ⌊ N n r ⌋ Δ k , 0 ≤ r ≤ 1. (22) D_n(r)=\frac{1}{\sqrt{N_n}}\sum_{k=b_n+1}^{b_n+\lfloor N_n r\rfloor}\Delta_k,\qquad0\le r\le1. \tag*{(22)} D n ( r ) = N n 1 k = b n + 1 ∑ b n + ⌊ N n r ⌋ Δ k , 0 ≤ r ≤ 1. (22)
The process in (22) is viewed as a step function in L 2 ( [ 0 , 1 ] ; R d ) L^2([0,1];\mathbb{R}^d) L 2 ([ 0 , 1 ] ; R d ) . The endpoint is included separately because evaluation at r = 1 r=1 r = 1 is not continuous in the L 2 L^2 L 2 topology. This choice matches the statistic: the numerator uses the endpoint, whereas the denominator integrates outer products of the centered path. Joint convergence of these two objects is therefore sufficient for inference. In particular, L 2 L^2 L 2 convergence controls the integrated quadratic form by the Cauchy–Schwarz inequality without requiring uniform convergence of the entire trajectory.
Theorem 3.7. Under Assumptions 3.1–3.5,
{ D n ( ⋅ ) , D n ( 1 ) } ⇝ { − H − 1 Σ 1 / 2 W d ( ⋅ ) , − H − 1 Σ 1 / 2 W d ( 1 ) } (23) \{D_n(\cdot),D_n(1)\}\rightsquigarrow\{-H^{-1}\Sigma^{1/2}W_d(\cdot),-H^{-1}\Sigma^{1/2}W_d(1)\} \tag*{(23)} { D n ( ⋅ ) , D n ( 1 )} ⇝ { − H − 1 Σ 1/2 W d ( ⋅ ) , − H − 1 Σ 1/2 W d ( 1 )} (23) in L 2 ( [ 0 , 1 ] ; R d ) × R d L^2([0,1];\mathbb{R}^d)\times\mathbb{R}^d L 2 ([ 0 , 1 ] ; R d ) × R d , where W d W_d W d is a standard d d d -dimensional Brownian motion.
The endpoint gives the usual averaged-estimator limit, while the L 2 L^2 L 2 component determines the limit of the integrated path products in the normalizer. The proof separates boundary, changing-gain, mean-field, noise, and retraction errors. Their control explains the gain restrictions: squared boundary errors average to order n β − 1 n^{\beta-1} n β − 1 , the accumulated mean-field error is of order n 1 / 2 − β ( 1 + ρ ) / 2 n^{1/2-\beta(1+\rho)/2} n 1/2 − β ( 1 + ρ ) /2 , and quadratic geometric errors are of order n 1 / 2 − β n^{1/2-\beta} n 1/2 − β . Technical lemmas appear in Appendix § 9 ; the proof is in Appendix § 10 .§ 2 .
Self-normalized inference
The endpoint and centered path share L = H − 1 Σ 1 / 2 L=H^{-1}\Sigma^{1/2} L = H − 1 Σ 1/2 . Their limits are − L W d ( 1 ) -LW_d(1) − L W d ( 1 ) and L Q d L T LQ_dL^{\mathsf{T}} L Q d L T , so an invertible L L L cancels in the quadratic statistic. The bridge Gram matrix Q d Q_d Q d is almost surely positive definite and independent of W d ( 1 ) W_d(1) W d ( 1 ) , giving the continuous pivotal law in (8) . The normalizer thus remains random in the limit. Its role is to carry the same unknown linear transformation as the estimation error, so that inversion removes that transformation. This explains why the critical value comes from a Brownian bridge rather than a chi-square law, even though the averaged estimator itself is asymptotically Gaussian.
Theorem 3.8 (Self-normalized Riemannian inference). Suppose Assumptions 3.1–3.5 hold. Let a n = x b n a_n=x_{b_n} a n = x b n and construct x ^ n \widehat{x}_n x n and V ^ n \widehat{V}_n V n from (5) –(6) . The local operations are well-defined with probability tending to one (Lemma C.9). Then:
N n Log x ∗ ( x ^ n ) ⇝ N ( 0 , H − 1 Σ H − 1 ) . (24) \sqrt{N_n}\operatorname{Log}_{x_*}(\widehat{x}_n)\rightsquigarrow\mathcal{N}\left(0,H^{-1}\Sigma H^{-1}\right). \tag*{(24)} N n Log x ∗ ( x n ) ⇝ N ( 0 , H − 1 Σ H − 1 ) . (24)
(b) With
V ~ n = P a n → x ⋆ V ^ n P x ⋆ → a n , \widetilde{V}_n = P_{a_n \to x_\star}\widehat{V}_nP_{x_\star\to a_n}, V n = P a n → x ⋆ V n P x ⋆ → a n ,
we have
V ~ n ⇝ H − 1 Σ 1 / 2 Q d Σ 1 / 2 H − 1 . (25) \widetilde{V}_n \leadsto H^{-1}\Sigma^{1/2}Q_d\Sigma^{1/2}H^{-1}. \tag*{(25)} V n ⇝ H − 1 Σ 1/2 Q d Σ 1/2 H − 1 . (25)
(c) If Σ \Sigma Σ is positive definite, V ^ n \widehat{V}_n V n is invertible with probability tending to one and, at the true parameter,
T n ( x ⋆ ) ⇝ T d . (26) T_n(x_\star) \leadsto T_d. \tag*{(26)} T n ( x ⋆ ) ⇝ T d . (26)
Consequently, for c d , 1 − α c_{d,1-\alpha} c d , 1 − α the ( 1 − α ) (1-\alpha) ( 1 − α ) quantile of T d T_d T d ,
P { x ⋆ ∈ C n , 1 − α } ⟶ 1 − α . (27) \mathbb{P}\{x_\star\in C_{n,1-\alpha}\} \longrightarrow1-\alpha. \tag*{(27)} P { x ⋆ ∈ C n , 1 − α } ⟶ 1 − α . (27)
The same cancellation holds in the practical chart at a n a_n a n . The anchor is computed from the observed path, so it need not be independent of the retained iterates. Uniform local chart expansions handle this dependence by applying the same remainder bound throughout the localization neighborhood, including at the random anchor. Its common offset disappears from every centered sum, while the remaining coordinate error is quadratic in the local iterate error. Mean square stability and β > 1 / 2 \beta> 1/2 β > 1/2 make its accumulated contribution negligible at the N n \sqrt{N_n} N n scale. Appendix § 9 .§ 1 gives the chart expansion, and Appendix § 10 .§ 3 proves the joint bridge limit and Theorem 3.8 .
Inference for smooth functionals
For a reported contrast, only its projected covariance needs to be nondegenerate.
Corollary 3.9 (Smooth functionals). Under Assumptions 3.1–3.5 , let h : M → R q h : \mathcal{M} \to\mathbb{R}^{q} h : M → R q be continuously differentiable near x ⋆ x_\star x ⋆ and set
A h = D h x ⋆ H − 1 Σ 1 / 2 . A_h = Dh_{x_\star}H^{-1}\Sigma^{1/2}. A h = D h x ⋆ H − 1 Σ 1/2 . Suppose A h A_h A h has row rank q q q ; Σ \Sigma Σ itself need not be positive definite. Let
D n = D ( h ∘ Exp a n ) ( v ˉ n ) , V ^ h , n = D n V ^ n D n T . D_n = D(h \circ\operatorname{Exp}_{a_n})(\bar{v}_n), \qquad\widehat{V}_{h,n} = D_n\widehat{V}_nD_n^{\mathsf{T}}. D n = D ( h ∘ Exp a n ) ( v ˉ n ) , V h , n = D n V n D n T . Then
N n { h ( x ^ n ) − h ( x ⋆ ) } ⇝ N ( 0 , A h A h T ) , (28) \sqrt{N_n}\{h(\widehat{x}_n)-h(x_\star)\} \leadsto\mathcal{N}\left(0,A_hA_h^{\mathsf{T}}\right), \tag*{(28)} N n { h ( x n ) − h ( x ⋆ )} ⇝ N ( 0 , A h A h T ) , (28) V ^ h , n \widehat{V}_{h,n} V h , n is invertible with probability tending to one, and
N n { h ( x ^ n ) − h ( x ⋆ ) } T V ^ h , n − 1 { h ( x ^ n ) − h ( x ⋆ ) } ⇝ T q . (29) N_n\{h(\widehat{x}_n)-h(x_\star)\}^{\mathsf{T}}\widehat{V}_{h,n}^{-1}\{h(\widehat{x}_n)-h(x_\star)\} \leadsto T_q. \tag*{(29)} N n { h ( x n ) − h ( x ⋆ ) } T V h , n − 1 { h ( x n ) − h ( x ⋆ )} ⇝ T q . (29) For a scalar functional, report
h ( x ^ n ) ± c 1 , 1 − α V ^ h , n / N n . h(\widehat{x}_n) \pm\sqrt{c_{1,1-\alpha}\widehat{V}_{h,n}/N_n}. h ( x n ) ± c 1 , 1 − α V h , n / N n . Thus a loading uses a scalar critical value even for a subspace with a large tangent dimension. The rank condition requires nondegenerate variation in the reported functional: directions annihilated by D h x ⋆ Dh_{x_\star} D h x ⋆ need not have positive noise variance. This permits inference for an informative contrast even when the full tangent covariance is singular. Conversely, a zero derivative removes the first order fluctuation and calls for a different limit. For example, a squared loading h j ( [ u ] ) = u j 2 h_j([u])=u_j^2 h j ([ u ]) = u j 2 has derivative 2 u j δ j 2u_j\delta_j 2 u j δ j ; a loading at zero is therefore outside the scope of this nondegenerate first order result. Consistent numerical derivatives and a vanishing nonnegative ridge preserve the pivot (Corollary B.1 and Proposition B.2 ); Remark B.3 gives direct normalization of a scalar path.
Geometric models
The construction applies to smooth Riemannian M M M -estimation with i.i.d. observations and stochastic gradient G ( x , Z ) = grad x ℓ ( x ; Z ) G(x,Z)=\operatorname{grad}_{x}\ell(x;Z) G ( x , Z ) = grad x ℓ ( x ; Z ) . When differentiation and expectation can be interchanged,
H = E { Hess x ℓ ( x ⋆ ; Z ) } , Σ = E { grad ℓ ( x ⋆ ; Z ) ⊗ grad ℓ ( x ⋆ ; Z ) } . (30) \begin{aligned}
H &= \mathbb{E}\{\operatorname{Hess}_{x}\ell(x_{\star};Z)\},\\
\Sigma&= \mathbb{E}\{\operatorname{grad}\ell(x_{\star};Z)\otimes\operatorname{grad}\ell(x_{\star};Z)\}. \tag*{(30)}
\end{aligned} H Σ = E { Hess x ℓ ( x ⋆ ; Z )} , = E { grad ℓ ( x ⋆ ; Z ) ⊗ grad ℓ ( x ⋆ ; Z )} . (30)
We verify Assumptions 3.1–3.5 for the following models under the localization and mean square stability conditions in their respective propositions. Smooth local geometry permits the charts in Assumption 3.1. The Hessian and moment calculations below establish Assumptions 3.2 and 3.3, while i.i.d. true-point gradients with finite second moments give Assumption 3.5. For Assumption 3.4, choose the gain exponent in (14) and use the stability rate required by each model proposition, or verify it through Proposition 3.6. Thus localization and stability remain conditions on the recursion. The precise model statements are in Section B.4; initialization is deterministic or independent of the observation stream.
Intrinsic Fréchet means
For a manifold-valued observation Z Z Z , the intrinsic mean minimizes
F ( x ) = 1 2 E { dist ( x , Z ) 2 } . F(x)=\frac{1}{2}\mathbb{E}\{\operatorname{dist}(x,Z)^2\}. F ( x ) = 2 1 E { dist ( x , Z ) 2 } .
Away from the cut locus, the stochastic gradient is
G ( x , Z ) = − Log x ( Z ) . (31) G(x,Z)=-\operatorname{Log}_{x}(Z). \tag*{(31)} G ( x , Z ) = − Log x ( Z ) . (31)
Thus the update moves toward the next observation along its logarithmic direction. At the unique mean μ \mu μ , the operators in (30) become
H μ = E { Hess x 1 2 dist ( x , Z ) 2 ∣ x = μ } , Σ μ = E { Log μ ( Z ) ⊗ Log μ ( Z ) } . (32) \begin{aligned}
H_{\mu} &= \mathbb{E}\left\{\left.\operatorname{Hess}_{x}\frac{1}{2}\operatorname{dist}(x,Z)^2\right|_{x=\mu}\right\},\\
\Sigma_{\mu} &= \mathbb{E}\{\operatorname{Log}_{\mu}(Z)\otimes\operatorname{Log}_{\mu}(Z)\}. \tag*{(32)}
\end{aligned} H μ Σ μ = E { Hess x 2 1 dist ( x , Z ) 2 x = μ } , = E { Log μ ( Z ) ⊗ Log μ ( Z )} . (32)
Positive definiteness of H μ H_{\mu} H μ gives regular local identification. A compact strongly convex support separated from the relevant cut loci, with a square integrable local Hessian/Hölder envelope, is sufficient for smoothness and noise continuity. On a Hadamard manifold, the logarithm is globally defined and finite squared distance gives uniqueness; the local moment and smoothness conditions then verify the remaining model assumptions. Proposition B.7 states the precise alternatives. These means summarize directional, shape, and matrix-valued observations [4 , 23 , 25 ] .
Spherical means. On the unit sphere S q \mathbb{S}^{q} S q , put r = dist ( x , z ) r=\operatorname{dist}(x,z) r = dist ( x , z ) and u = Log x ( z ) / r u=\operatorname{Log}_{x}(z)/r u = Log x ( z ) / r for 0 < r < π 0<r<\pi 0 < r < π . The Hessian of squared distance is
Hess x 1 2 dist ( x , z ) 2 = u ⊗ u + r cot ( r ) ( I − u ⊗ u ) . (33) \operatorname{Hess}_{x}\frac{1}{2}\operatorname{dist}(x,z)^2=u\otimes u+r\cot(r)(I-u\otimes u). \tag*{(33)} Hess x 2 1 dist ( x , z ) 2 = u ⊗ u + r cot ( r ) ( I − u ⊗ u ) . (33)
At r = 0 r=0 r = 0 , its continuous extension is I I I . Formula (33) separates the radial eigenvalue, equal to one, from the transverse eigenvalues r cot ( r ) r\cot(r) r cot ( r ) . Support in a geodesic ball of radius less than π / 4 \pi/4 π /4 provides a simple sufficient condition for uniqueness and local strong convexity [2 ] . Averaging (33) gives H μ H_{\mu} H μ , while logarithmic outer products give Σ μ \Sigma_{\mu} Σ μ . Algorithm 1 obtains the confidence region directly from the iterate path.
SPD matrix means. For S + + p S_{++}^{p} S ++ p with affine invariant metric ⟨ U , V ⟩ X = tr ( X − 1 U X − 1 V ) \langle U,V\rangle_X=\operatorname{tr}(X^{-1}UX^{-1}V) ⟨ U , V ⟩ X = tr ( X − 1 U X − 1 V ) , the logarithm is
Log X ( Z ) = X 1 / 2 log ( X − 1 / 2 Z X − 1 / 2 ) X 1 / 2 . \operatorname{Log}_X(Z)=X^{1/2}\log\left(X^{-1/2}ZX^{-1/2}\right)X^{1/2}. Log X ( Z ) = X 1/2 log ( X − 1/2 Z X − 1/2 ) X 1/2 .
This is a Hadamard manifold with tangent dimension p ( p + 1 ) / 2 p(p+1)/2 p ( p + 1 ) /2 . Finite squared affine invariant distance gives a unique mean M M M . Bounded geodesic support implies the local smoothness and noise-continuity bounds. For unbounded support, the local mean-field and noise-continuity conditions in Proposition B.7 (b) are imposed separately. The covariance factor is H M − 1 Σ M H M − 1 H_M^{-1}\Sigma_MH_M^{-1} H M − 1 Σ M H M − 1 , with both operators defined by (32) ; (B.6) gives the corresponding limit. Self-normalization avoids differentiating the matrix logarithm for Hessian estimation and transporting gradient outer products to an estimated mean. Such means are used in diffusion tensor and covariance valued analysis [10 , 15 , 26 ] .
Streaming principal subspaces
Let C = E ( X X T ) C=\mathbb{E}(XX^{\mathsf T}) C = E ( X X T ) for X ∈ R p X\in\mathbb{R}^p X ∈ R p . Its leading r r r -dimensional eigenspace is a point on Gr ( r , p ) \operatorname{Gr}(r,p) Gr ( r , p ) , with tangent dimension r ( p − r ) r(p-r) r ( p − r ) [12 ] . For an orthonormal representative U U U , use
F ( [ U ] ) = − 1 2 E ∥ U T X ∥ 2 = − 1 2 tr ( U T C U ) , (34) F([U])=-\frac{1}{2}\mathbb{E}\left\|U^{\mathsf T}X\right\|^2=-\frac{1}{2}\operatorname{tr}(U^{\mathsf T}CU), \tag*{(34)} F ([ U ]) = − 2 1 E U T X 2 = − 2 1 tr ( U T C U ) , (34)
and stochastic gradient
G ( [ U ] , X ) = − ( I − U U T ) X X T U . (35) G([U],X)=-(I-UU^{\mathsf T})XX^{\mathsf T}U. \tag*{(35)} G ([ U ] , X ) = − ( I − U U T ) X X T U . (35)
The operator C C C is the second moment; it is the covariance when X X X is centered. Incremental gradient methods include Oja’s update and Grassmann subspace tracking [3 , 24 ] ; averaged Riemannian gradients give an efficient streaming k k k -PCA estimate [31 ] .
Write the spectral decomposition as
C = U ⋆ Λ 1 U ⋆ T + U ⊥ Λ 2 U ⊥ T , λ r > λ r + 1 . C=U_{\star}\Lambda_1U_{\star}^{\mathsf T}+U_{\perp}\Lambda_2U_{\perp}^{\mathsf T},\qquad\lambda_r>\lambda_{r+1}. C = U ⋆ Λ 1 U ⋆ T + U ⊥ Λ 2 U ⊥ T , λ r > λ r + 1 .
Here λ 1 ≥ ⋯ ≥ λ p \lambda_1\ge\cdots\ge\lambda_p λ 1 ≥ ⋯ ≥ λ p are the eigenvalues of C C C , and Λ 1 , Λ 2 \Lambda_1,\Lambda_2 Λ 1 , Λ 2 contain the first r r r and remaining eigenvalues. Represent a tangent perturbation at [ U ⋆ ] [U_{\star}] [ U ⋆ ] by U ⊥ K U_{\perp}K U ⊥ K , with K ∈ R ( p − r ) × r K\in\mathbb{R}^{(p-r)\times r} K ∈ R ( p − r ) × r and the Frobenius inner product. The Hessian acts entrywise:
( H P C A K ) i j = ( λ j − λ r + i ) K i j . (36) (H_{\mathrm{PCA}}K)_{ij}=(\lambda_j-\lambda_{r+i})K_{ij}. \tag*{(36)} ( H PCA K ) ij = ( λ j − λ r + i ) K ij . (36)
Hence the eigengap is the smallest Hessian eigenvalue and identifies the leading subspace. With X ∥ = U ⋆ T X X_{\parallel}=U_{\star}^{\mathsf T}X X ∥ = U ⋆ T X and X ⊥ = U ⊥ T X X_{\perp}=U_{\perp}^{\mathsf T}X X ⊥ = U ⊥ T X , the tangent noise covariance is
Σ P C A = E [ vec ( X ⊥ X ∥ T ) vec ( X ⊥ X ∥ T ) T ] . (37) \Sigma_{\mathrm{PCA}}=\mathbb{E}\left[\operatorname{vec}(X_{\perp}X_{\parallel}^{\mathsf T})\operatorname{vec}(X_{\perp}X_{\parallel}^{\mathsf T})^{\mathsf T}\right]. \tag*{(37)} Σ PCA = E [ vec ( X ⊥ X ∥ T ) vec ( X ⊥ X ∥ T ) T ] . (37)
Since ∥ G ( [ U ] , X ) ∥ ≤ ∥ X ∥ 2 \|G([U],X)\|\le\|X\|^2 ∥ G ([ U ] , X ) ∥ ≤ ∥ X ∥ 2 , the fourth data moment E ∥ X ∥ 4 < ∞ \mathbb{E}\|X\|^4<\infty E ∥ X ∥ 4 < ∞ ensures a finite second moment of the stochastic gradient. Under localization and mean square stability, Proposition B.8 gives the root-N n N_n N n Gaussian limit for 1 / 2 < β < 1 1/2<\beta<1 1/2 < β < 1 . A positive definite tangent covariance yields inference for the full subspace; a nondegenerate derivative contrast yields smooth functional inference. For r = 1 r=1 r = 1 , the parameter is an unoriented direction, making squared loadings natural targets for § 4.2 .
Simulations
Simulation setup
We compare normalization of one path with plug-in Wald inference in four settings: two multi-frequency circular models, conditional-median phase retrieval on R P 2 \mathbb{R}P^2 R P 2 , and conditional-median range localization on S 2 S^2 S 2 .
The first pair varies the maximum frequency while keeping the population Hessian and score variance fixed; the sensing models require residual-density estimation for the Hessian. Each setting uses n = 4000 n = 4000 n = 4000 , 12000 12000 12000 and 500 independent replications, with methods paired on the same averaged estimate. The gain exponent is 0.6 and b n = ⌈ n 1 / 3 ⌉ b_n = \lceil n^{1/3} \rceil b n = ⌈ n 1/3 ⌉ . We report nominal 95% coverage and mean absolute error for the scalar functional. Failures count as noncoverage. Plug-in uses analytic sample Hessians for the circular models and bandwidth multiplier one for sensing; Signed-H accepts any nonzero scalar Hessian. Oracle uses population asymptotic covariance. Appendix § 7 .§ 2 gives the data-generating models and inference rules.
In the circular models, θ ⋆ = 0.35 \theta_\star= 0.35 θ ⋆ = 0.35 and Φ = M θ ⋆ + ϵ \Phi= M\theta_\star+\epsilon Φ = M θ ⋆ + ϵ modulo 2 π 2\pi 2 π . Frequency M M M equals one with probability 0.7 and otherwise is min { m max , ⌊ 20 U − 2 / 3 ⌋ } \min\{m_{\max}, \lfloor20U^{-2/3} \rfloor\} min { m m a x , ⌊ 20 U − 2/3 ⌋} for uniform U U U , with caps 100 and 200. The conditional phase density is { 1 + ( 1.2 / M ) cos ϵ + 0.4 cos ( 2 ϵ ) } / ( 2 π ) \{1 + (1.2/M)\cos\epsilon+ 0.4\cos(2\epsilon)\}/(2\pi) { 1 + ( 1.2/ M ) cos ϵ + 0.4 cos ( 2 ϵ )} / ( 2 π ) . The loss { 1 − cos ( M θ − Φ ) } / M \{1 - \cos(M\theta- \Phi)\}/M { 1 − cos ( M θ − Φ )} / M has bounded score and population operators H = 0.6 H = 0.6 H = 0.6 , Σ = 0.4 \Sigma= 0.4 Σ = 0.4 ; its observation Hessian grows with frequency. We infer sin θ ⋆ \sin\theta_\star sin θ ⋆ , use gain constant 0.8, and project updates onto [ − 0.4 , 1 ] [-0.4, 1] [ − 0.4 , 1 ] .
The sensing observations are Y = f ( x ⋆ , A ) + σ ( A ) ( Z − m Z ) Y = f(x_\star, A)+\sigma(A)(Z-m_Z) Y = f ( x ⋆ , A ) + σ ( A ) ( Z − m Z ) , where Z ∼ 0.8 N ( 0 , 1 ) + 0.2 N ( 0.5 , 0.15 2 ) Z \sim0.8\mathcal{N}(0, 1)+0.2\mathcal{N}(0.5, 0.15^2) Z ∼ 0.8 N ( 0 , 1 ) + 0.2 N ( 0.5 , 0.1 5 2 ) and m Z m_Z m Z is its median. Half absolute residual loss targets the conditional median. Phase retrieval uses f ( x , A ) = ( A T x ) 2 f(x, A) = (A^\mathsf{T}x)^2 f ( x , A ) = ( A T x ) 2 , a uniform spherical design, and target x 1 2 x_1^2 x 1 2 on R P 2 \mathbb{R}P^2 R P 2 ; range localization uses f ( x , A ) = ∥ x − A ∥ f(x, A) = \lVert x - A \rVert f ( x , A ) = ∥ x − A ∥ , six axis-aligned sensors, and target x 1 x_1 x 1 on S 2 S^2 S 2 . These exercise nonlinear coordinates, whereas the circular chart is locally Euclidean. Appendix § 7 .§ 2 specifies scales, initializations, gain constants, and the density-based plug-in rule.
We also study full-region inference for intrinsic means with readily estimated Hessians. Each setting uses 500 independent paths. The sphere model is rotationally symmetric with concentration eight, truncated to a geodesic ball of radius 0.6 about its mean. The positive definite model is Z = M 1 / 2 exp ( A ) M 1 / 2 Z = M^{1/2}\exp(A)M^{1/2} Z = M 1/2 exp ( A ) M 1/2 , where M M M has condition number five and the orthonormal symmetric coordinates of A A A are independent N ( 0 , 0.3 2 ) \mathcal{N}(0, 0.3^2) N ( 0 , 0. 3 2 ) . Both models use exact exponentials and logarithm maps. Gains are 2 k − 0.65 2k^{-0.65} 2 k − 0.65 on the sphere and 1.5 k − 0.6 1.5k^{-0.6} 1.5 k − 0.6 on the SPD matrix cone; burn-in is ⌈ n 1 / 3 ⌉ \lceil n^{1/3} \rceil ⌈ n 1/3 ⌉ .
The settings were selected after exploratory comparisons. The cap-100 design was then evaluated with a separately fixed independent seed set; the other primary settings reuse their fixed simulation runs. Additional comparisons include single-frequency controls and settings with lower self-normalized coverage.
Results
Table 1 shows the largest coverage gain when observation Hessians fluctuate strongly. Accepting negative nonzero Hessian estimates improves short-stream plug-in coverage, but the longer-stream advantage remains. Population-covariance Wald intervals also undercover, so covariance estimation alone does not explain the difference. Sensing gives comparable coverage without residual-density estimation; estimation errors decrease with stream length (Figure 1 ).
Table 1. Nominal 95% scalar coverage. Signed-H permits negative nonzero estimated Hessians; Oracle uses population asymptotic covariance.
Figure 1. Mean absolute error of the common scalar estimator from 500 replications per setting. Each panel has its own vertical scale.
In the intrinsic-mean comparisons, estimation error decreases while self-normalized coverage remains below nominal (Table 2 ). Plug-in performs better at n = 5000 n = 5000 n = 5000 and in most matched scalar controls (Table 5 ). Short-batch uses ⌊ N n ⌋ \lfloor\sqrt{N_n}\rfloor ⌊ N n ⌋ consecutive batches and serves as a failure diagnostic, not an adapted batch-means procedure. Population Wald uses independently approximated population operators; plug-in uses observation-based operators at the fitted estimate (Appendix § 7 .§ 2 ). These comparisons distinguish the gain under strongly varying observation Hessians from settings where curvature is readily estimated. Scalar projection reduces dimension but does not by itself ensure accurate coverage.
Table 2. Nominal 95% full-region coverage and mean squared geodesic error for intrinsic means. Methods share the estimator.
Table 5. Additional scalar coverage comparisons, each with 500 replications. All plug-in bandwidth multipliers are one.
The data illustration analyzes 4898 white wine samples with 11 physicochemical measurements from the UCI Wine Quality data [8 , 9 ] . A preprocessing block of 898 rows fixes centers and scales; the remaining 4000 rows form a randomly ordered PCA stream targeting the leading standardized second moment eigenspace conditional on preprocessing. We use η k = 0.15 k − 0.6 \eta_k = 0.15k^{-0.6} η k = 0.15 k − 0.6 , burn-in 16, and a fixed initial direction. The estimate lies 0.0127 radians from the batch eigenspace and explains 28.3% of the standardized second moment; the sample eigengap is 1.441.
We infer squared loadings h j ( [ u ] ) = u j 2 h_j([u]) = u_j^2 h j ([ u ]) = u j 2 using Corollary 3.9 , projecting the ten-dimensional normalizer onto each squared loading. A common anchor graph chart defines the path average, centered sums, and functional derivatives (Appendix § 7 .§ 4 ). Density, alcohol, total sulfur dioxide, and residual sugar have the largest squared loadings (Table 3 ); intervals are intersected with [ 0 , 1 ] [0, 1] [ 0 , 1 ] . Under an i.i.d. sampling model these intervals are asymptotically marginal for fixed nonderegenerate loadings, conditional on preprocessing. They describe physicochemical variation, excluding sensory quality; ranking the loadings does not confer simultaneous or selective coverage. Appendix § 7 .§ 4 reports derivative and order sensitivity.
Table 3. Largest squared white wine loadings and nominal marginal 95% intervals.
Discussion
One Riemannian gradient path yields confidence regions in one pass without sandwich covariance estimation. The joint endpoint and L 2 L^2 L 2 expansion justifies using Brownian critical values under stability and martingale conditions, avoiding highly variable Hessian estimates and residual-density estimation. Finite-sample undercoverage remains in some settings. Quantitative martingale approximations, geometric remainder bounds, and control of small normalizer eigenvalues could guide gain selection and coverage corrections. The present theory also assumes fixed dimension, local stability, and nondegenerate parameters or functionals at deterministic sample sizes. For increasing dimension or weak identification, projection and regularization are promising directions.
AI use statement
Generative AI tools were used for language editing, literature searches, code development, and checking the authors’ mathematical arguments. The authors verified all AI-assisted material incorporated into the manuscript and code, and take full responsibility for the work.
Additional Experiments
Brownian critical values
Section 2.3 describes simulation of dimension-dependent Brownian critical values. Table 4 lists the critical values for the main and supplementary experiments. Every dimension uses 20000 independent internal draws. Dimensions 1, 2, 3, 5, 6 use 512 Karhunen–Loève modes; dimensions 10, 15, 19 use 8192 modes. The scalar critical value rounds to 46.14; calculations use the unrounded critical value. These internal draws are separate from the 500 outer replications per setting.
Table 4. Brownian 0.95 critical values.
Scalar models and inference rules
The four primary settings separate two obstacles to covariance estimation: a highly variable observation Hessian and a population Hessian that depends on an unknown density. Each uses 500 independent paths at each of n = 4000 , 12000 n = 4000, 12000 n = 4000 , 12000 . Gains are η k = η 0 k − 0.6 \eta_k = \eta_0 k^{-0.6} η k = η 0 k − 0.6 , with b n = ⌈ n 1 / 3 ⌉ b_n = \lceil n^{1/3} \rceil b n = ⌈ n 1/3 ⌉ and N n = n − b n N_n = n - b_n N n = n − b n . The retained sample sizes are 3984 and 11977. All methods use the same fixed-chart averaged estimate, the same N n N_n N n scaling, and no ridge. Self-normalization uses Corollary 3.9 ; the derivative is analytic in the circular models and approximated by centered coordinate differences with step 10 − 3 n − 1 / 4 10^{-3}n^{-1/4} 1 0 − 3 n − 1/4 in the sensing models. Estimation error is ∣ h ( x ^ n ) − h ( x ⋆ ) ∣ |h(\widehat{x}_n) - h(x_\star)| ∣ h ( x n ) − h ( x ⋆ ) ∣ .
Multi-frequency circular estimation. Observe an integer frequency M M M and an angle Φ = M θ ⋆ + ϵ \Phi= M\theta_\star+ \epsilon Φ = M θ ⋆ + ϵ modulo 2 π 2\pi 2 π , with θ ⋆ = 0.35 \theta_\star= 0.35 θ ⋆ = 0.35 . Set M = 1 M = 1 M = 1 with probability 0.7; otherwise take M = min { m max , ⌊ 20 U − 2 / 3 ⌋ } M = \min\{m_{\max}, \lfloor20U^{-2/3} \rfloor\} M = min { m m a x , ⌊ 20 U − 2/3 ⌋} for an independent uniform U U U . The two frequency caps are m max = 100 , 200 m_{\max} = 100, 200 m m a x = 100 , 200 . Conditional on M M M , the phase error has density
p ( ϵ ∣ M ) = 1 + ( 1.2 / M ) cos ϵ + 0.4 cos ( 2 ϵ ) 2 π , − π < ϵ ≤ π . p(\epsilon\mid M) = \frac{1 + (1.2/M)\cos\epsilon+ 0.4\cos(2\epsilon)}{2\pi}, \qquad-\pi< \epsilon\le\pi. p ( ϵ ∣ M ) = 2 π 1 + ( 1.2/ M ) cos ϵ + 0.4 cos ( 2 ϵ ) , − π < ϵ ≤ π .
For loss ℓ ( θ ; M , Φ ) = { 1 − cos ( M θ − Φ ) } / M \ell(\theta; M, \Phi) = \{1-\cos(M\theta-\Phi)\}/M ℓ ( θ ; M , Φ ) = { 1 − cos ( M θ − Φ )} / M , the score is sin ( M θ − Φ ) \sin(M\theta-\Phi) sin ( M θ − Φ ) and its derivative is M cos ( M θ − Φ ) M\cos(M\theta-\Phi) M cos ( M θ − Φ ) . Thus the score is bounded by one, whereas the observation Hessian scales with frequency. Direct integration gives H = 0.6 H = 0.6 H = 0.6 and Σ = 0.4 \Sigma= 0.4 Σ = 0.4 for either cap. Increasing the cap changes observation-Hessian variability without changing these population operators. In particular, at the true parameter the second moment of the observation Hessian is 0.6 E ( M 2 ) 0.6\mathbb{E}(M^2) 0.6 E ( M 2 ) , while its mean remains 0.6.
The reported functional is h ( θ ) = sin θ h(\theta) = \sin\theta h ( θ ) = sin θ . Starting at zero with η 0 = 0.8 \eta_0 = 0.8 η 0 = 0.8 , we project each angular update onto [ − 0.4 , 1 ] [-0.4,1] [ − 0.4 , 1 ] and average in the anchor’s angular chart, denoting the estimate by θ ^ n \widehat{\theta}_n θ n . The positive-Hessian plug-in uses all n n n observations to estimate
H ^ = n − 1 ∑ i M i cos ( M i θ ^ n − Φ i ) , Σ ^ = n − 1 ∑ i ( g i − g ˉ ) 2 , g i = sin ( M i θ ^ n − Φ i ) , \widehat{H} = n^{-1}\sum_i M_i\cos(M_i\widehat{\theta}_n-\Phi_i), \qquad\widehat{\Sigma} = n^{-1}\sum_i(g_i-\bar{g})^2, \qquad g_i = \sin(M_i\widehat{\theta}_n-\Phi_i), H = n − 1 i ∑ M i cos ( M i θ n − Φ i ) , Σ = n − 1 i ∑ ( g i − g ˉ ) 2 , g i = sin ( M i θ n − Φ i ) ,
where g ˉ = n − 1 ∑ i g i \bar{g} = n^{-1}\sum_i g_i g ˉ = n − 1 ∑ i g i . The scalar sandwich is cos 2 ( θ ^ n ) Σ ^ / H ^ 2 \cos^2(\widehat{\theta}_n)\widehat{\Sigma}/\widehat{H}^2 cos 2 ( θ n ) Σ / H 2 , provided H ^ > 10 − 9 \widehat{H} > 10^{-9} H > 1 0 − 9 . It requires no bandwidth. Signed-H uses the same sandwich whenever ∣ H ^ ∣ > 10 − 9 |\widehat{H}| > 10^{-9} ∣ H ∣ > 1 0 − 9 , since scalar variance does not require a positive estimated Hessian; both rules appear in Table 1 . The oracle uses the population asymptotic variance cos 2 ( 0.35 ) 0.4 / 0.6 2 \cos^2(0.35)\,0.4/0.6^2 cos 2 ( 0.35 ) 0.4/0. 6 2 , not the finite-sample distribution of the estimator.
The caps keep H H H and Σ \Sigma Σ fixed, but also change the mean field away from the true parameter and the score-continuity constants. Thus they vary observation Hessians within regular models without holding the entire finite-stream distribution fixed. Angular coordinates are locally Euclidean here; the sensing examples exercise nonlinear manifold coordinates.
Identification and stability can be checked directly. For δ = θ − θ ⋆ \delta=\theta-\theta_\star δ = θ − θ ⋆ ,
F ( θ ) − F ( θ ⋆ ) = E [ 0.6 M 2 { 1 − cos ( M δ ) } ] . F(\theta)-F(\theta_\star)=\mathbb{E}\left[\frac{0.6}{M^2}\{1-\cos(M\delta)\}\right]. F ( θ ) − F ( θ ⋆ ) = E [ M 2 0.6 { 1 − cos ( M δ )} ] .
The positive mass at M = 1 M=1 M = 1 identifies θ ⋆ \theta_\star θ ⋆ uniquely on the circle. On the projection interval, ∣ δ ∣ ≤ 0.75 |\delta|\le0.75 ∣ δ ∣ ≤ 0.75 and F ′ ′ ( θ ) = 0.6 E cos ( M δ ) ≥ 0.6 { 0.7 cos ( 0.75 ) − 0.3 } > 0 F^{\prime\prime}(\theta)=0.6\mathbb{E}\cos(M\delta)\ge0.6\{0.7\cos(0.75)-0.3\}>0 F ′′ ( θ ) = 0.6 E cos ( M δ ) ≥ 0.6 { 0.7 cos ( 0.75 ) − 0.3 } > 0 . Nonexpansiveness of projection and the bounded score give the quadratic drift
E { ( θ k + 1 − θ ⋆ ) 2 ∣ F k } ≤ ( 1 − 2 μ 0 η k + 1 ) ( θ k − θ ⋆ ) 2 + η k + 1 2 , μ 0 = 0.6 { 0.7 cos ( 0.75 ) − 0.3 } . \mathbb{E}\{(\theta_{k+1}-\theta_\star)^2\mid\mathcal{F}_k\}\le(1-2\mu_0\eta_{k+1})(\theta_k-\theta_\star)^2+\eta_{k+1}^2,\qquad\mu_0=0.6\{0.7\cos(0.75)-0.3\}. E {( θ k + 1 − θ ⋆ ) 2 ∣ F k } ≤ ( 1 − 2 μ 0 η k + 1 ) ( θ k − θ ⋆ ) 2 + η k + 1 2 , μ 0 = 0.6 { 0.7 cos ( 0.75 ) − 0.3 } .
This yields E ( θ k − θ ⋆ ) 2 = O ( η k ) \mathbb{E}(\theta_k-\theta_\star)^2=O(\eta_k) E ( θ k − θ ⋆ ) 2 = O ( η k ) and almost-sure convergence by the decreasing-step supermartingale argument. Since the true parameter is interior and the score is bounded, projection is eventually inactive. In a smaller angular neighborhood, the reset auxiliary recursion has strongly monotone drift inside the neighborhood and linear drift outside, with bounded noise. The same drift bound and the burn-in moment bound verify Proposition 3.6. Finite caps give bounded derivatives and ρ = ν = 1 \rho=\nu=1 ρ = ν = 1 ; the i.i.d. bounded true-point scores verify the martingale conditions. Hence this experiment has a regular, identifiable model with strongly varying observation Hessians rather than a singular population Hessian.
Conditional-median sensing. Both sensing models observe Y = f ( x ⋆ , A ) + σ ( A ) ( Z − m Z ) Y=f(x_\star,A)+\sigma(A)(Z-m_Z) Y = f ( x ⋆ , A ) + σ ( A ) ( Z − m Z ) and minimize half absolute residual loss. Let Φ N \Phi_N Φ N and ϕ N \phi_N ϕ N denote the standard normal distribution function and density. The noise variable and its median m Z m_Z m Z satisfy
Z ∼ 0.8 N ( 0 , 1 ) + 0.2 N ( 0.5 , 0.15 2 ) , 0.8 Φ N ( m Z ) + 0.2 Φ N { ( m Z − 0.5 ) / 0.15 } = 0.5 , Z\sim0.8\mathcal{N}(0,1)+0.2\mathcal{N}(0.5,0.15^2),\qquad0.8\Phi_N(m_Z)+0.2\Phi_N\{(m_Z-0.5)/0.15\}=0.5, Z ∼ 0.8 N ( 0 , 1 ) + 0.2 N ( 0.5 , 0.1 5 2 ) , 0.8 Φ N ( m Z ) + 0.2 Φ N {( m Z − 0.5 ) /0.15 } = 0.5 ,
so m Z ≃ 0.2749 m_Z\simeq0.2749 m Z ≃ 0.2749 and the conditional median is f ( x ⋆ , A ) f(x_\star,A) f ( x ⋆ , A ) . The asymmetric mixture introduces density variation near the median without shifting the conditional median.
For known-energy phase retrieval, a unit representative x ∈ S 2 x\in\mathbb{S}^2 x ∈ S 2 denotes a point in R P 2 \mathbb{RP}^2 RP 2 . We take A A A uniform on the sphere of radius 3 \sqrt{3} 3 ,
f ( x , A ) = ( A T x ) 2 , x ⋆ = ( 0.4 , − 0.3 , 1 ) T / 1.25 , σ ( A ) = 0.2 ( 1 + 0.3 A 1 / 3 ) . f(x,A)=(A^\mathsf{T}x)^2,\qquad x_\star=(0.4,-0.3,1)^\mathsf{T}/\sqrt{1.25},\qquad\sigma(A)=0.2(1+0.3A_1/\sqrt{3}). f ( x , A ) = ( A T x ) 2 , x ⋆ = ( 0.4 , − 0.3 , 1 ) T / 1.25 , σ ( A ) = 0.2 ( 1 + 0.3 A 1 / 3 ) .
The sign-invariant functional is h ( x ) = x 1 2 h(x)=x_1^2 h ( x ) = x 1 2 , with true value 0.128. Initialization is ( 0.2 , − 0.1 , 1 ) T / 1.05 (0.2,-0.1,1)^\mathsf{T}/\sqrt{1.05} ( 0.2 , − 0.1 , 1 ) T / 1.05 , η 0 = 0.2 \eta_0=0.2 η 0 = 0.2 , and representatives are aligned before taking projective logarithms. Measurements follow the additive model above, as for background-corrected intensities. Population operators for the oracle are integrated over the spherical design; doubling Gauss–Legendre order from 32 to 64 changes the scalar variance by less than 10 − 9 10^{-9} 1 0 − 9 relatively.
For range localization, x ∈ S 2 x\in\mathbb{S}^2 x ∈ S 2 is a position on a known spherical surface. A sensor A A A is chosen uniformly from ( 2 e 1 , 2 e 2 , 2 e 3 , − 2 e 1 , − 2 e 2 , − 2 e 3 ) (2e_1,2e_2,2e_3,-2e_1,-2e_2,-2e_3) ( 2 e 1 , 2 e 2 , 2 e 3 , − 2 e 1 , − 2 e 2 , − 2 e 3 ) , where e j e_j e j are the standard coordinate vectors, with prediction f ( x , A ) = ∥ x − A ∥ f(x,A)=\lVert x-A\rVert f ( x , A ) = ∥ x − A ∥ . We set x ⋆ = Exp e 3 ( 0.2 , − 0.15 , 0 ) x_\star=\operatorname{Exp}_{e_3}(0.2,-0.15,0) x ⋆ = Exp e 3 ( 0.2 , − 0.15 , 0 ) and take the six noise scales to be 0.08 times six equally spaced values from 0.7 to 1.3, in that sensor order. The reported coordinate is h ( x ) = x 1 h(x)=x_1 h ( x ) = x 1 , approximately 0.1979. Initialization is e 3 e_3 e 3 and η 0 = 0.8 \eta_0=0.8 η 0 = 0.8 . Opposite-sensor squared ranges identify x j x_j x j through { ∥ x + 2 e j ∥ 2 − ∥ x − 2 e j ∥ 2 } / 8 \{\lVert x+2e_j\rVert^2-\lVert x-2e_j\rVert^2\}/8 {∥ x + 2 e j ∥ 2 − ∥ x − 2 e j ∥ 2 } /8 ; all sensor distances are at least one. The oracle is computed by an exact six-sensor sum.
Both models use exponential-map updates. Write r i = Y i − f ( x ^ n , A i ) r_i = Y_i - f(\hat{x}_n, A_i) r i = Y i − f ( x ^ n , A i ) , s i = 1 { r i < 0 } − 0.5 s_i = 1\{r_i < 0\} - 0.5 s i = 1 { r i < 0 } − 0.5 , and let D i D_i D i be the tangent gradient of f f f at x ^ n \hat{x}_n x ^ n . The plug-in estimates
H ^ = n − 1 ∑ i { ϕ N ( r i / t n ) t n D i D i T + s i Hess x f ( x ^ n , A i ) } , \hat{H} = n^{-1}\sum_i\left\{\frac{\phi_N(r_i/t_n)}{t_n}D_iD_i^{\mathsf{T}} + s_i\operatorname{Hess}_x f(\hat{x}_n,A_i)\right\}, H ^ = n − 1 i ∑ { t n ϕ N ( r i / t n ) D i D i T + s i Hess x f ( x ^ n , A i ) } ,
Σ ^ = n − 1 ∑ i ( s i D i − s D ‾ ) ( s i D i − s D ‾ ) T , s D ‾ = n − 1 ∑ i s i D i , \hat{\Sigma} = n^{-1}\sum_i(s_iD_i-\overline{sD})(s_iD_i-\overline{sD})^{\mathsf{T}}, \qquad\overline{sD}=n^{-1}\sum_i s_iD_i, Σ ^ = n − 1 i ∑ ( s i D i − sD ) ( s i D i − sD ) T , sD = n − 1 i ∑ s i D i ,
t n = 1.4826 median i ∣ r i − median j r j ∣ n − 1 / 5 . t_n = 1.4826\operatorname{median}_i\lvert r_i-\operatorname{median}_j r_j\rvert\,n^{-1/5}. t n = 1.4826 median i ∣ r i − median j r j ∣ n − 1/5 .
The bandwidth multiplier is one for every sensing setting. The geometric correction in H ^ \hat{H} H ^ is retained, and inverse action on the functional derivative gives the scalar sandwich variance. Nonpositive Hessians or variances count as failed intervals, without substitution of a population value. These are all-observation comparisons at the common fitted estimate; neither plug-in nor oracle changes that estimate.
To connect these losses to the theory, let D ( A ) = grad x f ( x ⋆ , A ) D(A)=\operatorname{grad}_x f(x_\star,A) D ( A ) = grad x f ( x ⋆ , A ) and p ϵ ( 0 ∣ A ) p_\epsilon(0\mid A) p ϵ ( 0 ∣ A ) be the conditional density at zero of ϵ = Y − f ( x ⋆ , A ) \epsilon=Y-f(x_\star,A) ϵ = Y − f ( x ⋆ , A ) . Differentiating the conditional mean score gives
H = E { p ϵ ( 0 ∣ A ) D ( A ) ⊗ D ( A ) } , Σ = 1 4 E { D ( A ) ⊗ D ( A ) } . H=\mathbb{E}\{p_\epsilon(0\mid A)D(A)\otimes D(A)\}, \qquad\Sigma=\frac{1}{4}\mathbb{E}\{D(A)\otimes D(A)\}. H = E { p ϵ ( 0 ∣ A ) D ( A ) ⊗ D ( A )} , Σ = 4 1 E { D ( A ) ⊗ D ( A )} .
Thus the population Hessian contains a density term even though a single observation’s indicator score is locally constant off its threshold. Smooth predictors, positive smooth error densities and the spanning designs make H H H positive definite and give a quadratic mean-field remainder. An indicator changes only on a residual interval of length proportional to the parameter displacement, giving squared L 2 L^2 L 2 score increments of that order. These verify ρ = 1 \rho=1 ρ = 1 and ν = 1 / 2 \nu=1/2 ν = 1/2 , while bounded scores verify the true-point martingale conditions. Corollary 3.9 then applies under the recursion localization and uniform mean square stability conditions in Assumptions 3.1 and 3.4, with β = 0.6 \beta=0.6 β = 0.6 .
Full-region comparison rules. The intrinsic-mean comparisons in Table 2 use the same fitted estimates and retained sample size N = N n N=N_n N = N n . The short-batch diagnostic takes B = max { 4 , ⌊ N ⌋ } B=\max\{4,\lfloor\sqrt{N}\rfloor\} B = max { 4 , ⌊ N ⌋} consecutive batches of length ℓ = ⌊ N / B ⌋ \ell=\lfloor N/B\rfloor ℓ = ⌊ N / B ⌋ from the retained coordinate path. Writing v ˉ j B \bar{v}_j^B v ˉ j B for batch means and v ˉ B \bar{v}^B v ˉ B for their average,
Ω ^ B = ℓ B − 1 ∑ j = 1 B ( v ˉ j B − v ˉ B ) ( v ˉ j B − v ˉ B ) T . \hat{\Omega}_B=\frac{\ell}{B-1}\sum_{j=1}^{B}(\bar{v}_j^B-\bar{v}^B)(\bar{v}_j^B-\bar{v}^B)^{\mathsf{T}}. Ω ^ B = B − 1 ℓ j = 1 ∑ B ( v ˉ j B − v ˉ B ) ( v ˉ j B − v ˉ B ) T .
The Wald statistic uses the full retained average, scale N N N , and χ d 2 \chi_d^2 χ d 2 critical value. This short-batch schedule is a failure diagnostic, not the adapted batch-means procedure of Chen et al. [7 ] .
Population Wald regions use independently approximated population Hessians and covariances at the true parameter, mapped into the anchor chart. Plug-in regions estimate these operators at the fitted point using observations: all observations for the sphere, and an 80-observation Hessian approximation together with gradient covariance from all observations for positive definite matrices. These regions also use χ d 2 \chi_d^2 χ d 2 critical values.
Estimation error and covariance comparisons
The multi-frequency mean errors decrease from 0.0182 to 0.00961 at cap 100 and from 0.0173 to 0.00966 at cap 200. Phase-retrieval errors decrease from 0.00104 to 0.00060, and range-localization errors from 0.00167 to 0.00098. These are shared point-estimation results, not differences between the inference methods.
The circular plug-in fails its positive-Hessian check in 12 and 0 replications at cap 100 and in 43 and 2 replications at cap 200, in increasing order of n n n . All 500 replications remain in every coverage denominator.
No self-normalized, signed-Hessian or oracle calculation fails, and neither method fails in the selected sensing experiments. Allowing negative nonzero Hessians improves short-stream plug-in coverage, but the longer-stream coverage gap remains (Table 1 ).
Single-frequency and other sensing comparisons. Setting M = 1 M = 1 M = 1 throughout gives a single-frequency control with the same conditional phase density, reported functional, gain, initialization and projection as the multi-frequency models. The two sensing models are also evaluated with Z ∼ N ( 0 , 1 ) Z \sim\mathcal{N}(0,1) Z ∼ N ( 0 , 1 ) and m Z = 0 m_Z = 0 m Z = 0 , leaving their other settings unchanged. Table 5 reports these comparisons together with circular intrinsic means. The single-frequency control at n = 12000 n = 12000 n = 12000 favors plug-in, while phase retrieval remains close under both error distributions. Consequently the gain in the main frequency comparison occurs with strongly varying observation Hessians, not across all circular or density-dependent models.
For the circular intrinsic means, Y = 0.35 + T Y = 0.35 + T Y = 0.35 + T modulo 2 π 2\pi 2 π with
p m ( t ) = { 1 + 0.6 cos t + 0.1 cos ( m t ) + 0.05 ( sin t + 2 sin 2 t ) } / ( 2 π ) , m = 2 , 8. p_m(t) = \{1 + 0.6\cos t + 0.1\cos(mt) + 0.05(\sin t + 2\sin2t)\}/(2\pi), \qquad m = 2,8. p m ( t ) = { 1 + 0.6 cos t + 0.1 cos ( m t ) + 0.05 ( sin t + 2 sin 2 t )} / ( 2 π ) , m = 2 , 8.
Half squared geodesic loss has target mean 0.35 0.35 0.35 , Hessian H = 0.5 H = 0.5 H = 0.5 , and score variance π 2 / 3 − 1.2 + 0.2 / m 2 \pi^2/3 - 1.2 + 0.2/m^2 π 2 /3 − 1.2 + 0.2/ m 2 . The reported mean functional is sin ( 0.35 ) \sin(0.35) sin ( 0.35 ) ; gain, initialization, projection and burn-in equal those in the multi-frequency models. The density at the fitted antipode is estimated by a wrapped Gaussian kernel with bandwidth n − 1 / 5 n^{-1/5} n − 1/5 , and 1 − 2 π p ^ ( θ ^ n + π ) 1 - 2\pi\widehat{p}(\widehat{\theta}_n + \pi) 1 − 2 π p ( θ n + π ) estimates the Hessian. This is a different density-dependent mechanism from the analytic observation Hessian in the multi-frequency experiment.
Details of the white wine analysis
Each row is a sampling unit. The 11 physicochemical variables have no reported missing values; repeated measurement rows are retained. The sensory quality score is excluded. Standardization estimated from the preprocessing block remains fixed for the streaming analysis. Figure 2 displays the squared-loading intervals. The anchor graph chart uses z k = B T u k / ( a T u k ) z_k = B^T u_k/(a^T u_k) z k = B T u k / ( a T u k ) and inverse u a ( z ) = ( a + B z ) / ( 1 + ∥ z ∥ 2 ) 1 / 2 u_a(z) = (a + Bz)/(1 + \lVert z \rVert^2)^{1/2} u a ( z ) = ( a + B z ) / ( 1 + ∥ z ∥ 2 ) 1/2 , where B B B is an orthonormal tangent basis. These coordinates are invariant to changing the sign of u k u_k u k ; choose representatives with a T u k > 0 a^T u_k > 0 a T u k > 0 . Their average and centered sums define the estimate and normalizer, and the derivative is evaluated in the same coordinates. For the graph coordinates used in Section 4.2, with c = a + B z c = a + Bz c = a + B z , the derivative is
Figure 2. Squared white wine loadings with nominal marginal 95% intervals.
∇ z h j { u a ( z ) } = 2 c j 1 + ∥ z ∥ 2 B T e j − 2 c j 2 ( 1 + ∥ z ∥ 2 ) 2 z . \nabla_z h_j\{u_a(z)\} = \frac{2c_j}{1+\lVert z\rVert^2}B^\mathsf{T}e_j - \frac{2c_j^2}{(1+\lVert z\rVert^2)^2}z. ∇ z h j { u a ( z )} = 1 + ∥ z ∥ 2 2 c j B T e j − ( 1 + ∥ z ∥ 2 ) 2 2 c j 2 z .
The reported calculation uses centered differences with fixed step 10 − 5 10^{-5} 1 0 − 5 ; steps 10 − 4 10^{-4} 1 0 − 4 , 10 − 5 10^{-5} 1 0 − 5 , 10 − 6 10^{-6} 1 0 − 6 give stable interval widths. This is a numerical precision check, distinct from the shrinking-step asymptotics of Corollary B.1 . Across ten additional random orders, the angle to the batch eigenspace has median 0.0157 and maximum 0.0425 radians; source-file order gives 0.0596 radians.
Additional Theory
This section collects sufficient conditions, functional inference, structural comparisons, and geometric specializations of Theorems 3.7 and 3.8 . Their proofs are collected in Section D.
Scope and verification of the assumptions
A common normal neighborhood admits a local logarithm chart. Such localizations are standard in stochastic approximation [19 ] and Riemannian gradient theory [5 , 31 ] , and follow from almost sure convergence or projection onto a strongly convex set. Hadamard manifolds have globally single-valued logarithms; local boundedness controls Taylor remainders. For intrinsic means, support separated from the cut locus is a convenient sufficient condition, while specialized theory permits mass near the cut locus [17 ] .
Positive definiteness of H H H is the usual regular identification condition for a root-n n n limit of the parameter vector. If H H H is singular, the convergence rate and limit law may change; smeariness of intrinsic means is one example [13 ] . Assumption 3.2 states only the Taylor remainder used by the proof. The transported Hölder condition (18) implies this remainder bound. The condition β ( 1 + ρ ) > 1 \beta(1+\rho)>1 β ( 1 + ρ ) > 1 controls the accumulated mean field remainder. The inequality β > 1 / 2 \beta>1/2 β > 1/2 controls the accumulated retraction and curvature terms, while β < 1 \beta<1 β < 1 controls the Abel boundary for a generic scalar gain. The boundary choice β = 1 \beta=1 β = 1 generally has a different partial-sum limit and is not covered by the Brownian critical values used here.
Under localization and uniform stability in mean square, the nonlinear remainder bounds use only the conditional second moment of the stochastic gradient. The square integrable Hessian envelope in (19) gives (13) with ν = 1 \nu=1 ν = 1 in an i.i.d. smooth loss model. Conditions (16)–(17) are standard martingale functional central limit conditions [16 ] . For i.i.d. gradients at x ⋆ x_\star x ⋆ , a finite second moment implies them.
The uniform triangular-array rate in (15) is the usual local mean square rate for a stable decreasing-step stochastic approximation, imposed on the auxiliary process used by the proof. Proposition 3.6 derives it from a one-step Lyapunov drift and a uniform burn-in initial bound. In a concrete model, the drift is typically verified from local strong monotonicity of the mean gradient, a second moment bound, and either a projected recursion or an established convergence theorem. The sufficient condition uses the increment-truncated map itself to bound the squared update. It therefore requires neither a fourth gradient moment nor a squared bound on the Taylor remainder. The initial moment bound is not implied by consistency of the anchor.
These conditions follow the established averaging framework. In Euclidean random scaling, Lee et al. [21, Assumption 1] impose a positive definite Hessian, a local Hölder expansion of the mean gradient, polynomial gains with exponent in ( 1 / 2 , 1 ) (1/2,1) ( 1/2 , 1 ) , and martingale noise with covariance and tail conditions. These correspond to Assumptions 3.2–3.5. Their Lyapunov condition implies stability; here retained localization and a uniform mean square rate state that input directly. Their uniform path argument additionally assumes a higher noise moment depending on the gain exponent. Our joint L 2 L^2 L 2 and endpoint analysis controls nonlinear remainders through conditional second moments, with the martingale covariance and Lindeberg requirements in Assumption 3.5.
For Riemannian averaging, Tripuraneni et al. [31, Assumptions 1–6] require iterates in a compact common retraction neighborhood, a positive definite Hessian with local Lipschitz regularity, conditionally unbiased gradients, continuity in mean square, and a bounded conditional fourth gradient moment. Their Assumption 1 gives E ∥ Δ k ∥ 2 = O ( η k ) \mathbb{E}\lVert\Delta_k\rVert^2=O(\eta_k) E ∥ Δ k ∥ 2 = O ( η k ) , the same rate used in (15). The choices ρ = 1 \rho=1 ρ = 1 and ν = 1 \nu=1 ν = 1 recover the usual Lipschitz conditions and gain range. Our formulation assumes retained localization and uniform stability and uses the integrated path expansion to connect the random anchor chart to the Brownian normalizer. Thus the inferential assumptions retain the standard local identification, stability, and martingale structure of averaging theory, with conditions tailored to the joint endpoint and path limit.
A uniform functional limit is still available under an additional maximal condition. For example, if
1 N n max b n ≤ k ≤ n η k − 1 ∥ Δ k ∥ → p 0 , \frac{1}{\sqrt{N_n}}\max_{b_n\le k\le n}\eta_k^{-1}\lVert\Delta_k\rVert\xrightarrow{p} 0, N n 1 b n ≤ k ≤ n max η k − 1 ∥ Δ k ∥ p 0 ,
then the proof of Theorem 3.7 yields convergence in the Skorokhod topology and, because the limit is continuous, in the uniform topology.
Derivative approximation and regularization
The rank condition in Corollary 3.9 is the relevant nondegeneracy requirement for a reported contrast. It can hold even when the full tangent covariance is singular.
Corollary B.1 (Consistent derivative approximation). Under the conditions of Corollary 3.9 , let D ^ n : T a n M → R q \widehat{D}_n:T_{a_n}\mathcal{M}\to\mathbb{R}^{q} D n : T a n M → R q be a random linear map such that, after the natural tangent identifications,
∥ D ^ n − D n ∥ o p → p 0. \lVert\widehat{D}_n-D_n\rVert_{\mathrm{op}}\xrightarrow{p}0. ∥ D n − D n ∥ op p 0. Then (29) remains valid with V ^ h , n \widehat{V}_{h,n} V h , n replaced by
V ^ h , n a p p = D ^ n V ^ n D ^ n T . \widehat{V}^{\mathrm{app}}_{h,n}=\widehat{D}_n\widehat{V}_n\widehat{D}_n^{\mathsf{T}}. V h , n app = D n V n D n T .
In fixed dimension, centered coordinate differences with deterministic steps s n → 0 s_n\to0 s n → 0 satisfy the operator-norm consistency condition with error O p ( s n 2 ) O_p(s_n^2) O p ( s n 2 ) when the relevant pullbacks are uniformly C 3 C^3 C 3 on a common neighborhood containing every perturbed evaluation point. No rate linking s n s_n s n to N n N_n N n is required.
Proposition B.2 (Stability under a vanishing ridge). Suppose the joint endpoint and normalizer convergence in Theorem 3.8 holds, and let all ridges below be nonnegative and possibly data-dependent.
(a) If Σ \Sigma Σ is positive definite and λ n → p 0 \lambda_n\xrightarrow{p}0 λ n p 0 , replacing V ^ n \widehat{V}_n V n by V ^ n + λ n I \widehat{V}_n+\lambda_n I V n + λ n I leaves the full statistic’s limit T d T_d T d unchanged.
(b) Under Corollary 3.9 , if λ h , n → p 0 \lambda_{h,n}\xrightarrow{p}0 λ h , n p 0 , replacing the projected normalizer by V ^ h , n + λ h , n I q \widehat{V}_{h,n}+\lambda_{h,n}I_q V h , n + λ h , n I q , or replacing V ^ h , n a p p \widehat{V}^{\mathrm{app}}_{h,n} V h , n app analogously under Corollary B.1 , leaves the limit T q T_q T q unchanged.
Remark B.3 (Direct normalization of a scalar path). For a C 2 C^2 C 2 scalar h h h , the arithmetic mean and bridge normalizer of h ( x b n + 1 ) , … , h ( x n ) h(x_{b_n+1}),\ldots,h(x_n) h ( x b n + 1 ) , … , h ( x n ) give a direct scalar statistic. On the agreement event in Lemma C.4 , the Taylor remainder is O ( ∥ Δ k , n l o c ∥ 2 ) O\left(\lVert\Delta^{\mathrm{loc}}_{k,n}\rVert^2\right) O ( ∥ Δ k , n loc ∥ 2 ) . Event splitting, (15), and β > 1 / 2 \beta>1/2 β > 1/2 make its accumulated contribution o p ( 1 ) o_p(1) o p ( 1 ) after N n − 1 / 2 N_n^{-1/2} N n − 1/2 scaling. A nondegenerate derivative contrast therefore gives the limit T 1 T_1 T 1 , equivalent to first order to the construction using derivative projection in Corollary 3.9 .
Structural properties and comparison with Wald inference
Proposition B.4 (Coordinate equivariance and Euclidean reduction). Under Assumptions 3.1–3.5, the following statements hold.
(a) The finite sample statistic T n ( x ) T_n(x) T n ( x ) is unchanged by an orthonormal change of basis in T a n M T_{a_n}\mathcal{M} T a n M .
(b) Let A ⊂ U 0 A \subset U_0 A ⊂ U 0 be a neighborhood of x ⋆ x_\star x ⋆ . For a ∈ A a \in A a ∈ A , let χ a \chi_a χ a be a C 2 C^2 C 2 coordinate chart on a common open set containing U 0 U_0 U 0 , with values in T a M T_a\mathcal{M} T a M and χ a ( a ) = 0 \chi_a(a)=0 χ a ( a ) = 0 . Suppose ( a , x ) ↦ χ a ( x ) (a,x)\mapsto\chi_a(x) ( a , x ) ↦ χ a ( x ) is jointly C 2 C^2 C 2 , with derivatives through second order uniformly bounded for a ∈ A a\in A a ∈ A and x ∈ U 0 x\in U_0 x ∈ U 0 , and that D χ a ∣ x ⋆ D\chi_a|_{x_\star} D χ a ∣ x ⋆ and its inverse are uniformly bounded. If Log a ( x ) \operatorname{Log}_a(x) Log a ( x ) is replaced throughout Algorithm 1 by χ a ( x ) \chi_a(x) χ a ( x ) and the average is mapped back by χ a − 1 \chi_a^{-1} χ a − 1 , the alternative-chart average still satisfies (24) . If Σ \Sigma Σ is positive definite, the full statistic has limit T d T_d T d . More generally, for h h h under Corollary 3.9 , derivative projection using D ( h ∘ χ a n − 1 ) D(h\circ\chi_{a_n}^{-1}) D ( h ∘ χ a n − 1 ) at the alternative-coordinate average has limit T q T_q T q whenever A h A_h A h has row rank q q q , even if Σ \Sigma Σ is singular. This functional conclusion also holds for an operator-norm-consistent approximation to that derivative.
(c) If M = R d \mathcal{M}=\mathbb{R}^d M = R d , R x ( v ) = x + v R_x(v)=x+v R x ( v ) = x + v , and χ a ( x ) = x − a \chi_a(x)=x-a χ a ( x ) = x − a , after relabeling the retained segment, Algorithm 1 is exactly the random scaling construction for Polyak–Ruppert averaged stochastic gradient descent in [21 ] .
Part (b) covers inverse retractions that form a uniformly C 2 C^2 C 2 local chart on the retained localization neighborhood. Smooth coordinate choices give the same pivotal first order limit.
For a formal comparison with plug-in covariance estimation, define Ω = H − 1 Σ H − 1 \Omega=H^{-1}\Sigma H^{-1} Ω = H − 1 Σ H − 1 and let Ω ^ n \widehat{\Omega}_n Ω n be an operator on T a n M T_{a_n}\mathcal{M} T a n M such that
P a n → x ⋆ Ω ^ n P x ⋆ → a n → p Ω . (38) P_{a_n\to x_\star}\widehat{\Omega}_nP_{x_\star\to a_n}\xrightarrow{p}\Omega. \tag*{(38)} P a n → x ⋆ Ω n P x ⋆ → a n p Ω. (38)
Let χ d , 1 − α 2 \chi^2_{d,1-\alpha} χ d , 1 − α 2 be the ( 1 − α ) (1-\alpha) ( 1 − α ) quantile of χ d 2 \chi^2_d χ d 2 . The corresponding local Wald region is
C n , 1 − α W = { x : N n ⟨ v ‾ n − Log a n ( x ) , Ω ^ n − 1 { v ‾ n − Log a n ( x ) } ⟩ a n ≤ χ d , 1 − α 2 } . (39) C^{\mathrm{W}}_{n,1-\alpha}=\left\{x:N_n\left\langle\overline{v}_n-\operatorname{Log}_{a_n}(x),\widehat{\Omega}_n^{-1}\{\overline{v}_n-\operatorname{Log}_{a_n}(x)\}\right\rangle_{a_n}\leq\chi^2_{d,1-\alpha}\right\}. \tag*{(39)} C n , 1 − α W = { x : N n ⟨ v n − Log a n ( x ) , Ω n − 1 { v n − Log a n ( x )} ⟩ a n ≤ χ d , 1 − α 2 } . (39)
The Wald region (B.2) and the self-normalized region are understood to be restricted to their common chart domain. Let ω d = π d / 2 / Γ ( 1 + d / 2 ) \omega_d=\pi^{d/2}/\Gamma(1+d/2) ω d = π d /2 /Γ ( 1 + d /2 ) and let vol a n \operatorname{vol}_{a_n} vol a n denote Lebesgue volume in an orthonormal coordinate system of T a n M T_{a_n}\mathcal{M} T a n M . Because the region radii are O p ( N n − 1 / 2 ) O_p(N_n^{-1/2}) O p ( N n − 1/2 ) and the true parameter is an interior point of that domain with probability tending to one, this restriction is asymptotically inactive.
Proposition B.5 (Relation to Wald inference at first order). Suppose the assumptions of Theorem 3.8 and (B.1) hold, Σ \Sigma Σ is positive definite, and Ω ^ n \widehat{\Omega}_n Ω n is self-adjoint and positive definite with probability tending to one. Then the Wald statistic converges to χ d 2 \chi^2_d χ d 2 , while the self-normalized statistic converges to T d T_d T d . Moreover, with L = H − 1 Σ 1 / 2 L=H^{-1}\Sigma^{1/2} L = H − 1 Σ 1/2 ,
N n d / 2 vol a n { Log a n ( C n , 1 − α W ) } → p ω d ( χ d , 1 − α 2 ) d / 2 ∣ det L ∣ , (40) N_n^{d/2}\operatorname{vol}_{a_n}\{ \operatorname{Log}_{a_n}(C^{\mathrm{W}}_{n,1-\alpha})\}\xrightarrow{p}\omega_d(\chi^2_{d,1-\alpha})^{d/2}|\det L|, \tag*{(40)} N n d /2 vol a n { Log a n ( C n , 1 − α W )} p ω d ( χ d , 1 − α 2 ) d /2 ∣ det L ∣ , (40)
N n d / 2 vol a n { Log a n ( C n , 1 − α ) } ⇝ ω d c d , 1 − α d / 2 ∣ det L ∣ det ( Q d ) 1 / 2 . (41) N_n^{d/2}\operatorname{vol}_{a_n}\{\operatorname{Log}_{a_n}(C_{n,1-\alpha})\}\rightsquigarrow\omega_dc_{d,1-\alpha}^{d/2}|\det L|\det(Q_d)^{1/2}. \tag*{(41)} N n d /2 vol a n { Log a n ( C n , 1 − α )} ⇝ ω d c d , 1 − α d /2 ∣ det L ∣ det ( Q d ) 1/2 . (41)
Both procedures share the first order covariance factor ∣ det L ∣ |\det L| ∣ det L ∣ ; the self-normalized volume also contains the random bridge factor det ( Q d ) 1 / 2 \det(Q_d)^{1/2} det ( Q d ) 1/2 .
:::
The volume limits separate the shared covariance factor from the random Brownian bridge shape. Comparing region efficiency would involve choosing a loss for random confidence sets and common higher-order asymptotic conditions.
Wald inference estimates Ω \Omega Ω and uses chi-square critical values. Self-normalization uses the path’s Brownian bridge shape and its pivotal critical values. Other Euclidean studentization procedures use batch means, perturbed trajectories, or HiGrad branching [7 , 14 , 30 ] .
Theoretical results for the geometric models
Section 3.5 gives the objectives, gradients, Hessians, and noise covariances for the geometric examples. The results below collect their sufficient conditions. The i.i.d. stream is independent of initialization.
Smooth Riemannian M M M -estimation
Proposition B.6 (Riemannian M M M -estimation). Suppose x ⋆ x_\star x ⋆ is an isolated minimizer, H H H in (30) is positive definite, and the following conditions hold on a normal neighborhood of x ⋆ x_\star x ⋆ : the mean gradient has the expansion (11) ; E ∥ grad ℓ ( x ⋆ ; Z ) ∥ 2 < ∞ \mathbb{E}\,\lVert\operatorname{grad}\ell(x_\star;Z)\rVert^2 < \infty E ∥ grad ℓ ( x ⋆ ; Z ) ∥ 2 < ∞ ; the stochastic gradient satisfies (13) ; and the stochastic gradient path satisfies localization and (15) . Then, for β \beta β satisfying (14) ,
N n Log x ⋆ ( x ^ n ) ⇝ N ( 0 , H − 1 Σ H − 1 ) . (42) \sqrt{N_n}\operatorname{Log}_{x_\star}(\widehat{x}_n)\rightsquigarrow\mathcal{N}(0,H^{-1}\Sigma H^{-1}). \tag*{(42)} N n Log x ⋆ ( x n ) ⇝ N ( 0 , H − 1 Σ H − 1 ) . (42) If Σ \Sigma Σ is positive definite, (9) has asymptotic coverage 1 − α 1-\alpha 1 − α . More generally, every smooth q q q -dimensional functional satisfying the rank condition in Corollary 3.9 admits the pivot in (29) .
Intrinsic Fréchet means
Write μ ^ n \widehat{\mu}_n μ n for the fixed-chart average x ^ n \widehat{x}_n x n when estimating the intrinsic mean μ \mu μ .
Proposition B.7 (Intrinsic Fréchet mean). Suppose either of the following sets of sufficient conditions holds.
(a) The support of Z Z Z and the retained path lie in a compact strongly convex set separated by a positive distance from the relevant cut loci, and the Hessian of squared distance has a square integrable local Hölder envelope.
(b) The manifold is Hadamard, E dist ( x 0 , Z ) 2 < ∞ \mathbb{E}\,\operatorname{dist}(x_0,Z)^2 < \infty E dist ( x 0 , Z ) 2 < ∞ for some x 0 x_0 x 0 , and the local mean field and noise-continuity conditions hold near μ \mu μ .
Assume H μ H_\mu H μ is positive definite, the path satisfies localization and (15) , and the step-size exponent satisfies (14) for the local mean field exponent ρ \rho ρ . Then
N n Log μ ( μ ^ n ) ⇝ N ( 0 , H μ − 1 Σ μ H μ − 1 ) . (43) \sqrt{N_n}\operatorname{Log}_{\mu}(\widehat{\mu}_n)\rightsquigarrow\mathcal{N}(0,H_\mu^{-1}\Sigma_\mu H_\mu^{-1}). \tag*{(43)} N n Log μ ( μ n ) ⇝ N ( 0 , H μ − 1 Σ μ H μ − 1 ) . (43) If Σ μ \Sigma_\mu Σ μ is positive definite, Algorithm 1 gives an asymptotic ( 1 − α ) (1-\alpha) ( 1 − α ) confidence region for μ \mu μ without estimating either operator in (32) .
The support condition in part (a) gives a regular model with support separated from the cut locus. Refined central limit theorems permit support meeting the cut locus under geometry and local-density conditions [17 ] . Smeary means have different scaling [13 ] ; empirical likelihood is a batch-sample alternative [18 ] . For affine-invariant matrix means, (B.6) applies with μ = M \mu=M μ = M , H μ = H M H_\mu=H_M H μ = H M , and Σ μ = Σ M \Sigma_\mu=\Sigma_M Σ μ = Σ M .
Streaming principal subspaces
Proposition B.8 (Grassmann principal subspace). Assume λ r > λ r + 1 \lambda_r>\lambda_{r+1} λ r > λ r + 1 and E ∥ X ∥ 4 < ∞ \mathbb{E}\,\lVert X\rVert^4 < \infty E ∥ X ∥ 4 < ∞ . Suppose the stochastic Grassmann recursion is localized in the basin of [ U ⋆ ] [U_\star] [ U ⋆ ] and satisfies (15) . Let [ U ^ n ] [\widehat{U}_n] [ U n ] be its fixed-chart average, and represent H P C A H_{\mathrm{PCA}} H PCA in the column-stacked K K K coordinates of § 3.5.2 . For 1 / 2 < β < 1 1/2<\beta<1 1/2 < β < 1 ,
N n vec { U ⊥ T Log [ U ⋆ ] ( [ U ^ n ] ) } ⇝ N ( 0 , H P C A − 1 Σ P C A H P C A − 1 ) . (44) \sqrt{N_n}\operatorname{vec}\!\left\{U_\perp^\mathsf{T}\operatorname{Log}_{[U_\star]}([\widehat{U}_n])\right\}\rightsquigarrow\mathcal{N}\left(0,H_{\mathrm{PCA}}^{-1}\Sigma_{\mathrm{PCA}}H_{\mathrm{PCA}}^{-1}\right). \tag*{(44)} N n vec { U ⊥ T Log [ U ⋆ ] ([ U n ]) } ⇝ N ( 0 , H PCA − 1 Σ PCA H PCA − 1 ) . (44) If Σ P C A \Sigma_{\mathrm{PCA}} Σ PCA is positive definite on the tangent space, Algorithm 1 gives a full confidence region for the principal subspace. Otherwise, Corollary 3.9 applies to every nondegenerate smooth contrast.
Extensions
The first order argument allows any uniformly C 2 C^2 C 2 local inverse retraction with nonsingular derivative on a common retained localization neighborhood, which can reduce computation required for repeated logarithm evaluations. A quotient-manifold extension additionally needs a locally smooth horizontal representation; singular orbits require critical values adapted to changing dimension or isotropy. Dependence is also possible in principle when the leading noise admits a martingale or Poisson-equation decomposition with a Brownian partial sum limit. Long-range dependence would change that limit and hence the critical values.
Further directions include rank adaptation and nonregular inference. Unknown covariance rank, increasing tangent dimension, singular Hessians, smeary intrinsic means, nonsmooth population objectives, and optional stopping all interact with either the convergence rate or the normalizer spectrum. These developments would extend the present fixed-dimensional regular theory.
Technical Lemmas
Local coordinate expansions
This appendix develops the geometric and stochastic approximation expansions used in the proofs. Write b = b n b=b_n b = b n and N = N n N=N_n N = N n throughout. Constants may change from line to line. We first construct local coefficients on the compact normal-coordinate set corresponding to U ‾ 0 \overline{U}_0 U 0 , extend those coefficients to a globally defined burn-in-reset recursion, and only then transfer its limits to the original retained path on a high-probability agreement event.
Lemma C.1 (Differential of the logarithm map). There are constants r 0 , C > 0 r_0,C>0 r 0 , C > 0 such that, whenever x = Exp x ⋆ ( Δ ) x=\operatorname{Exp}_{x_\star}(\Delta) x = Exp x ⋆ ( Δ ) and ∥ Δ ∥ ≤ r 0 \lVert\Delta\rVert\le r_0 ∥ Δ ∥ ≤ r 0 ,
∥ D Log x ⋆ ∣ x − P x → x ⋆ ∥ o p ≤ C ∥ Δ ∥ 2 . (45) \left\lVert D\operatorname{Log}_{x_\star}\big|_x-P_{x\to x_\star}\right\rVert_{\mathrm{op}}\le C\lVert\Delta\rVert^2. \tag*{(45)} D Log x ⋆ x − P x → x ⋆ op ≤ C ∥ Δ ∥ 2 . (45)
Proof. Let γ ( t ) = Exp x ⋆ ( t Δ ) \gamma(t)=\operatorname{Exp}_{x_\star}(t\Delta) γ ( t ) = Exp x ⋆ ( t Δ ) and use a parallel orthonormal frame along γ \gamma γ . For w ∈ T x ⋆ M w\in T_{x_\star}\mathcal{M} w ∈ T x ⋆ M , the differential D Exp x ⋆ ∣ Δ [ w ] D\operatorname{Exp}_{x_\star}\big|_\Delta[w] D Exp x ⋆ Δ [ w ] is the value at t = 1 t=1 t = 1 of the Jacobi field J J J satisfying J ( 0 ) = 0 J(0)=0 J ( 0 ) = 0 and ∇ t J ( 0 ) = w \nabla_tJ(0)=w ∇ t J ( 0 ) = w . After the path is rescaled to unit time, its coordinate vector j ( t ) j(t) j ( t ) satisfies
j ′ ′ ( t ) + R Δ ( t ) j ( t ) = 0 , j ( 0 ) = 0 , j ′ ( 0 ) = w , j''(t)+R_\Delta(t)j(t)=0,\qquad j(0)=0,\qquad j'(0)=w, j ′′ ( t ) + R Δ ( t ) j ( t ) = 0 , j ( 0 ) = 0 , j ′ ( 0 ) = w ,
where sup t ∥ R Δ ( t ) ∥ o p ≤ C ∥ Δ ∥ 2 \sup_t\lVert R_\Delta(t)\rVert_{\mathrm{op}}\le C\lVert\Delta\rVert^2 sup t ∥ R Δ ( t ) ∥ op ≤ C ∥ Δ ∥ 2 by bounded sectional curvature on the local compact set. Twice integrating the equation gives
j ( 1 ) = w − ∫ 0 1 ( 1 − s ) R Δ ( s ) j ( s ) d s . j(1)=w-\int_0^1(1-s)R_\Delta(s)j(s)\,\mathrm{d}s. j ( 1 ) = w − ∫ 0 1 ( 1 − s ) R Δ ( s ) j ( s ) d s .
A Gronwall bound yields sup s ≤ 1 ∥ j ( s ) ∥ ≤ C ∥ w ∥ \sup_{s\le1}\lVert j(s)\rVert\le C\lVert w\rVert sup s ≤ 1 ∥ j ( s )∥ ≤ C ∥ w ∥ and hence ∥ j ( 1 ) − w ∥ ≤ C ∥ Δ ∥ 2 ∥ w ∥ \lVert j(1)-w\rVert\le C\lVert\Delta\rVert^2\lVert w\rVert ∥ j ( 1 ) − w ∥ ≤ C ∥ Δ ∥ 2 ∥ w ∥ . Therefore
D Exp x ⋆ ∣ Δ = P x ⋆ → x { I + E Δ } , ∥ E Δ ∥ o p ≤ C ∥ Δ ∥ 2 . D\operatorname{Exp}_{x_\star}\big|_\Delta=P_{x_\star\to x}\{I+E_\Delta\},\qquad\lVert E_\Delta\rVert_{\mathrm{op}}\le C\lVert\Delta\rVert^2. D Exp x ⋆ Δ = P x ⋆ → x { I + E Δ } , ∥ E Δ ∥ op ≤ C ∥ Δ ∥ 2 .
For r 0 r_0 r 0 small, I + E Δ I+E_\Delta I + E Δ is invertible. Inverting this display proves (C.1).
Lemma C.2 (Small retained increments). Under Assumptions 3.1 and 3.3, and with β > 1 / 2 \beta>1/2 β > 1/2 ,
P ( max b n ≤ k ≤ n − 1 η k + 1 ∥ G k + 1 ( x k ) ∥ x k > r R / 2 , x k ∈ U 0 for all b n ≤ k ≤ n ) ⟶ 0. (46) \mathbb{P}\left(\max_{b_n\le k\le n-1}\eta_{k+1}\lVert G_{k+1}(x_k)\rVert_{x_k}>r_{\mathcal{R}}/2,\ x_k\in U_0\text{ for all }b_n\le k\le n\right)\longrightarrow0. \tag*{(46)} P ( b n ≤ k ≤ n − 1 max η k + 1 ∥ G k + 1 ( x k ) ∥ x k > r R /2 , x k ∈ U 0 for all b n ≤ k ≤ n ) ⟶ 0. (46)
Proof. On the localization event, conditional Markov’s inequality and (12) give
P ( max b n ≤ k ≤ n − 1 η k + 1 ∥ G k + 1 ( x k ) ∥ > r R / 2 , x k ∈ U 0 for all b n ≤ k ≤ n ) \mathbb{P}\left(\max_{b_n\le k\le n-1}\eta_{k+1}\left\|G_{k+1}(x_k)\right\|>r_R/2,\ x_k\in U_0\text{ for all }b_n\le k\le n\right) P ( b n ≤ k ≤ n − 1 max η k + 1 ∥ G k + 1 ( x k ) ∥ > r R /2 , x k ∈ U 0 for all b n ≤ k ≤ n )
≤ 4 r R − 2 ∑ k = b n n − 1 η k + 1 2 E [ ∥ G k + 1 ( x k ) ∥ 2 1 { x k ∈ U 0 } ] ≤ C ∑ k = b n ∞ k − 2 β ⟶ 0 , \begin{aligned}
\le4r_R^{-2}\sum_{k=b_n}^{n-1}\eta_{k+1}^2\mathbb{E}\left[\left\|G_{k+1}(x_k)\right\|^2\mathbf{1}\{x_k\in U_0\}\right] \\
&\le C\sum_{k=b_n}^{\infty}k^{-2\beta}\longrightarrow0,
\end{aligned} ≤ 4 r R − 2 k = b n ∑ n − 1 η k + 1 2 E [ ∥ G k + 1 ( x k ) ∥ 2 1 { x k ∈ U 0 } ] ≤ C k = b n ∑ ∞ k − 2 β ⟶ 0 ,
where the last step uses b n → ∞ b_n\to\infty b n → ∞ and β > 1 / 2 \beta>1/2 β > 1/2 . □
Lemma C.3 (Local coefficient construction). Under Assumptions 3.1–3.3, whenever x k ∈ U ‾ 0 x_k\in\overline{U}_0 x k ∈ U 0 , the increment-truncated update in the proof has normal-coordinate error Δ k = Log x ⋆ ( x k ) \Delta_k=\operatorname{Log}_{x_\star}(x_k) Δ k = Log x ⋆ ( x k ) satisfying
Δ k + 1 = Δ k − η k + 1 ( H Δ k + ξ k + 1 ) + η k + 1 a k + 1 + η k + 1 m k + 1 + η k + 1 2 q k + 1 , (47) \Delta_{k+1}=\Delta_k-\eta_{k+1}(H\Delta_k+\xi_{k+1})+\eta_{k+1}a_{k+1}+\eta_{k+1}m_{k+1}+\eta_{k+1}^2q_{k+1}, \tag*{(47)} Δ k + 1 = Δ k − η k + 1 ( H Δ k + ξ k + 1 ) + η k + 1 a k + 1 + η k + 1 m k + 1 + η k + 1 2 q k + 1 , (47) where a k + 1 a_{k+1} a k + 1 is F k \mathcal{F}_k F k -measurable, m k + 1 m_{k+1} m k + 1 is a martingale difference, and
∥ a k + 1 ∥ ≤ C { ∥ Δ k ∥ 1 + ρ + ∥ Δ k ∥ 2 } , (48) \left\|a_{k+1}\right\|\le C\{\left\|\Delta_k\right\|^{1+\rho}+\left\|\Delta_k\right\|^2\}, \tag*{(48)} ∥ a k + 1 ∥ ≤ C { ∥ Δ k ∥ 1 + ρ + ∥ Δ k ∥ 2 } , (48) E ( ∥ m k + 1 ∥ 2 ∣ F k ) ≤ C { ∥ Δ k ∥ 2 ν ˉ + ∥ Δ k ∥ 2 } , ν ˉ = min ( ν , 1 ) , (49) \mathbb{E}(\left\|m_{k+1}\right\|^2\mid\mathcal{F}_k)\le C\{\left\|\Delta_k\right\|^{2\bar{\nu}}+\left\|\Delta_k\right\|^2\},\qquad\bar{\nu}=\min(\nu,1), \tag*{(49)} E ( ∥ m k + 1 ∥ 2 ∣ F k ) ≤ C { ∥ Δ k ∥ 2 ν ˉ + ∥ Δ k ∥ 2 } , ν ˉ = min ( ν , 1 ) , (49) E ( ∥ q k + 1 ∥ ∣ F k ) ≤ C . (50) \mathbb{E}(\left\|q_{k+1}\right\|\mid\mathcal{F}_k)\le C. \tag*{(50)} E ( ∥ q k + 1 ∥ ∣ F k ) ≤ C . (50)
Proof. Let ϕ = Log x ⋆ \phi=\operatorname{Log}_{x_\star} ϕ = Log x ⋆ on U 1 U_1 U 1 and, for x ∈ U ‾ 0 x\in\overline{U}_0 x ∈ U 0 , define Φ x ( v ) = ϕ { R x ( v ) } \Phi_x(v)=\phi\{R_x(v)\} Φ x ( v ) = ϕ { R x ( v )} for ∥ v ∥ x ≤ r R \|v\|_x\le r_R ∥ v ∥ x ≤ r R . Choose a smooth cutoff χ \chi χ that is one on the ball of radius r R / 2 r_R/2 r R /2 and zero outside the ball of radius r R r_R r R , and set
Φ ~ x ( v ) = ϕ ( x ) + χ ( v ) { Φ x ( v ) − ϕ ( x ) } \widetilde{\Phi}_x(v)=\phi(x)+\chi(v)\{\Phi_x(v)-\phi(x)\} Φ x ( v ) = ϕ ( x ) + χ ( v ) { Φ x ( v ) − ϕ ( x )}
on the retraction tube, extended by ϕ ( x ) \phi(x) ϕ ( x ) outside the tube. Compactness and the C 2 C^2 C 2 assumption imply a uniform second order expansion
Φ ~ x ( − η g ) = ϕ ( x ) − η D ϕ x [ g ] + η 2 R ( x , g , η ) , ∥ R ( x , g , η ) ∥ ≤ C ∥ g ∥ x 2 . (51) \widetilde{\Phi}_x(-\eta g)=\phi(x)-\eta D\phi_x[g]+\eta^2R(x,g,\eta),\qquad\|R(x,g,\eta)\|\le C\|g\|_x^2. \tag*{(51)} Φ x ( − η g ) = ϕ ( x ) − η D ϕ x [ g ] + η 2 R ( x , g , η ) , ∥ R ( x , g , η ) ∥ ≤ C ∥ g ∥ x 2 . (51)
By the probability statement (C.2) in Lemma C.2, this truncated expansion agrees with the original recursion for every retained update with probability tending to one. It is therefore enough to analyze the truncated local recursion.
Write x = x k x=x_k x = x k . The conditional second moment bound in (12) and (C.7) give a term q k + 1 q_{k+1} q k + 1 satisfying (C.6). For the conditional mean, Assumption 3.2 and Lemma C.1 yield
D ϕ x [ grad F ( x ) ] = P x → x ⋆ grad F ( x ) + { D ϕ x − P x → x ⋆ } grad F ( x ) = H Δ k + r F ( x ) , ∥ r F ( x ) ∥ ≤ C { ∥ Δ k ∥ 1 + ρ + ∥ Δ k ∥ 2 } . (52) \begin{aligned}
D\phi_x[\operatorname{grad}F(x)]=P_{x\to x_\star}\operatorname{grad}F(x)+\{D\phi_x-P_{x\to x_\star}\}\operatorname{grad}F(x) \\
&=H\Delta_k+r_F(x),\qquad\|r_F(x)\|\le C\{\|\Delta_k\|^{1+\rho}+\|\Delta_k\|^2\}. \tag*{(52)}
\end{aligned} D ϕ x [ grad F ( x )] = P x → x ⋆ grad F ( x ) + { D ϕ x − P x → x ⋆ } grad F ( x ) = H Δ k + r F ( x ) , ∥ r F ( x ) ∥ ≤ C { ∥ Δ k ∥ 1 + ρ + ∥ Δ k ∥ 2 } . (52)
Expansion (C.8) also uses that grad F \operatorname{grad}F grad F is bounded on the local compact set.
For the centered noise, put
m k + 1 = − [ D ϕ x { ϵ k + 1 ( x ) } − ξ k + 1 ] . m_{k+1}=-[D\phi_x\{\epsilon_{k+1}(x)\}-\xi_{k+1}]. m k + 1 = − [ D ϕ x { ϵ k + 1 ( x )} − ξ k + 1 ] .
It is a martingale difference. By adding and subtracting P x → x ⋆ ϵ k + 1 ( x ) P_{x\to x_\star}\epsilon_{k+1}(x) P x → x ⋆ ϵ k + 1 ( x ) , using (13), Lemma C.1, and the conditional second moment bound, we obtain
E ( ∥ m k + 1 ∥ 2 ∣ F k ) ≤ C { ∥ Δ k ∥ 2 ν ˉ + ∥ Δ k ∥ 2 } , ν ˉ = min ( ν , 1 ) . \mathbb{E}(\|m_{k+1}\|^2\mid\mathcal{F}_k)\le C\{\|\Delta_k\|^{2\bar{\nu}}+\|\Delta_k\|^2\},\qquad\bar{\nu}=\min(\nu,1). E ( ∥ m k + 1 ∥ 2 ∣ F k ) ≤ C { ∥ Δ k ∥ 2 ν ˉ + ∥ Δ k ∥ 2 } , ν ˉ = min ( ν , 1 ) .
Finally set a k + 1 = − r F ( x k ) a_{k+1}=-r_F(x_k) a k + 1 = − r F ( x k ) , with its sign absorbed into the notation in (C.3), and collect the Taylor remainder in q k + 1 q_{k+1} q k + 1 . This proves (C.3)–(C.6). □
Burn-in-reset localized recursion
Let ϕ = Log x ⋆ \phi= \operatorname{Log}_{x_\star} ϕ = Log x ⋆ and K = ϕ ( U ‾ 0 ) \mathcal{K} = \phi(\overline{U}_0) K = ϕ ( U 0 ) . For δ ∈ K \delta\in\mathcal{K} δ ∈ K , write x ( δ ) = ϕ − 1 ( δ ) x(\delta) = \phi^{-1}(\delta) x ( δ ) = ϕ − 1 ( δ ) and let Φ ~ x \widetilde{\Phi}_x Φ x be the increment-truncated map in (C.7). Define
r F ( δ ) = D ϕ x ( δ ) [ grad F { x ( δ ) } ] − H δ , r_F(\delta) = D\phi_{x(\delta)}[\operatorname{grad} F\{x(\delta)\}] - H\delta, r F ( δ ) = D ϕ x ( δ ) [ grad F { x ( δ )}] − H δ ,
a k + 1 l o c ( δ ) = − 1 { δ ∈ K } r F ( δ ) , a^{\mathrm{loc}}_{k+1}(\delta) = -\mathbf{1}\{\delta\in\mathcal{K}\}r_F(\delta), a k + 1 loc ( δ ) = − 1 { δ ∈ K } r F ( δ ) ,
m k + 1 l o c ( δ ) = 1 { δ ∈ K } [ ξ k + 1 − D ϕ x ( δ ) { ϵ k + 1 ( x ( δ ) ) } ] , m^{\mathrm{loc}}_{k+1}(\delta) = \mathbf{1}\{\delta\in\mathcal{K}\}\left[\xi_{k+1} - D\phi_{x(\delta)}\{\epsilon_{k+1}(x(\delta))\}\right], m k + 1 loc ( δ ) = 1 { δ ∈ K } [ ξ k + 1 − D ϕ x ( δ ) { ϵ k + 1 ( x ( δ ))} ] ,
q k + 1 l o c ( δ ) = 1 { δ ∈ K } η k + 1 − 2 [ Φ ~ x ( δ ) { − η k + 1 G k + 1 ( x ( δ ) ) } − δ + η k + 1 D ϕ x ( δ ) { G k + 1 ( x ( δ ) ) } ] . q^{\mathrm{loc}}_{k+1}(\delta) = \mathbf{1}\{\delta\in\mathcal{K}\}\eta_{k+1}^{-2}\left[\widetilde{\Phi}_{x(\delta)}\{-\eta_{k+1}G_{k+1}(x(\delta))\} - \delta+ \eta_{k+1}D\phi_{x(\delta)}\{G_{k+1}(x(\delta))\}\right]. q k + 1 loc ( δ ) = 1 { δ ∈ K } η k + 1 − 2 [ Φ x ( δ ) { − η k + 1 G k + 1 ( x ( δ ))} − δ + η k + 1 D ϕ x ( δ ) { G k + 1 ( x ( δ ))} ] .
where every coefficient is set to zero when δ ∉ K \delta\notin\mathcal{K} δ ∈ / K . The state indicator is predictable. Assumption 3.3, applied at predictable random arguments, therefore makes m k + 1 l o c ( δ k ) m^{\mathrm{loc}}_{k+1}(\delta_k) m k + 1 loc ( δ k ) a martingale difference for every F k \mathcal{F}_k F k -measurable δ k \delta_k δ k . The bounds (C.4)–(C.5) hold globally with Δ k \Delta_k Δ k replaced by δ k \delta_k δ k . Moreover,
∥ q k + 1 l o c ( δ k ) ∥ ≤ C 1 { δ k ∈ K } ∥ G k + 1 { x ( δ k ) } ∥ 2 . \left\|q^{\mathrm{loc}}_{k+1}(\delta_k)\right\| \le C\mathbf{1}\{\delta_k \in\mathcal{K}\}\left\|G_{k+1}\{x(\delta_k)\}\right\|^2. q k + 1 loc ( δ k ) ≤ C 1 { δ k ∈ K } ∥ G k + 1 { x ( δ k )} ∥ 2 .
For increments inside the inner cutoff tube this is (C.7). Outside that tube, η k + 1 ∥ G ∥ > r R / 2 \eta_{k+1}\|G\| > r_R/2 η k + 1 ∥ G ∥ > r R /2 implies η k + 1 ∥ G ∥ ≤ ( 2 / r R ) η k + 1 2 ∥ G ∥ 2 \eta_{k+1}\|G\| \le(2/r_R)\eta_{k+1}^2\|G\|^2 η k + 1 ∥ G ∥ ≤ ( 2/ r R ) η k + 1 2 ∥ G ∥ 2 , which bounds both the omitted first order term and the truncated displacement. Hence E ( ∥ q k + 1 l o c ( δ k ) ∥ ∣ F k ) ≤ C \mathbb{E}(\|q^{\mathrm{loc}}_{k+1}(\delta_k)\| \mid\mathcal{F}_k) \le C E ( ∥ q k + 1 loc ( δ k ) ∥ ∣ F k ) ≤ C uses only (12) .
For each n n n , initialize
Δ b n , n l o c = { Log x ⋆ ( x b n ) , x b n ∈ U 0 , 0 , x b n ∉ U 0 . \Delta^{\mathrm{loc}}_{b_n,n} =
\begin{cases}
\operatorname{Log}_{x_\star}(x_{b_n}), & x_{b_n} \in U_0,\\
0, & x_{b_n} \notin U_0.
\end{cases} Δ b n , n loc = { Log x ⋆ ( x b n ) , 0 , x b n ∈ U 0 , x b n ∈ / U 0 .
and, for b n ≤ k < n b_n \le k < n b n ≤ k < n , define the globally valid auxiliary recursion
Δ k + 1 , n l o c = Δ k , n l o c − η k + 1 ( H Δ k , n l o c + ξ k + 1 ) + η k + 1 a k + 1 l o c ( Δ k , n l o c ) + η k + 1 m k + 1 l o c ( Δ k , n l o c ) + η k + 1 2 q k + 1 l o c ( Δ k , n l o c ) . (53) \begin{aligned}
\Delta^{\mathrm{loc}}_{k+1,n}
={}&\Delta^{\mathrm{loc}}_{k,n} - \eta_{k+1}(H\Delta^{\mathrm{loc}}_{k,n}+\xi_{k+1})\\
&+\eta_{k+1}a^{\mathrm{loc}}_{k+1}(\Delta^{\mathrm{loc}}_{k,n})
+\eta_{k+1}m^{\mathrm{loc}}_{k+1}(\Delta^{\mathrm{loc}}_{k,n})
+\eta_{k+1}^{2}q^{\mathrm{loc}}_{k+1}(\Delta^{\mathrm{loc}}_{k,n}). \tag*{(53)}
\end{aligned} Δ k + 1 , n loc = Δ k , n loc − η k + 1 ( H Δ k , n loc + ξ k + 1 ) + η k + 1 a k + 1 loc ( Δ k , n loc ) + η k + 1 m k + 1 loc ( Δ k , n loc ) + η k + 1 2 q k + 1 loc ( Δ k , n loc ) . (53)
Outside K \mathcal{K} K , this is the stable leading linear recursion driven by ξ k + 1 \xi_{k+1} ξ k + 1 . Assumption 3.4 imposes the uniform mean square bound on the full triangular array in (C.9).
Lemma C.4 (Localization agreement and transfer). Let
E n = { x k ∈ U 0 for all b n ≤ k ≤ n } ∩ { max b n ≤ k < n η k + 1 ∥ G k + 1 ( x k ) ∥ ≤ r R / 2 } . E_n = \left\{x_k \in U_0\ \text{for all } b_n \le k \le n\right\}\cap\left\{\max_{b_n\le k<n}\eta_{k+1}\left\|G_{k+1}(x_k)\right\|\le r_R/2\right\}. E n = { x k ∈ U 0 for all b n ≤ k ≤ n } ∩ { b n ≤ k < n max η k + 1 ∥ G k + 1 ( x k ) ∥ ≤ r R /2 } . Then P ( E n ) → 1 \mathbb{P}(E_n)\to1 P ( E n ) → 1 , and on E n E_n E n ,
Δ k , n l o c = Log x ⋆ ( x k ) , b n ≤ k ≤ n . \Delta^{\mathrm{loc}}_{k,n}=\operatorname{Log}_{x_\star}(x_k),\qquad b_n\le k\le n. Δ k , n loc = Log x ⋆ ( x k ) , b n ≤ k ≤ n . Consequently, any endpoint, L 2 L^2 L 2 -path, estimator, normalizer, or statistic built from the original and auxiliary retained paths and agreeing on E n E_n E n has the same weak limit whenever one of the two versions has a weak limit.
Proof. Assumption 3.1 and Lemma C.2 give P ( E n ) → 1 \mathbb{P}(E_n)\to1 P ( E n ) → 1 . On E n E_n E n , the state indicator equals one and the increment-truncated map equals the original retraction at every retained update. Equality follows by induction from the common burn-in state. For any two corresponding random elements X n X_n X n and X n l o c X_n^{\mathrm{loc}} X n loc in a metric space, P { d ( X n , X n l o c ) > 0 } ≤ P ( E n c ) → 0 \mathbb{P}\{d(X_n,X_n^{\mathrm{loc}})>0\}\le\mathbb{P}(E_n^c)\to0 P { d ( X n , X n loc ) > 0 } ≤ P ( E n c ) → 0 ; Slutsky’s theorem gives the transfer statement.
Lemma C.5 (Expansion of the anchor logarithm). Let a ∈ U 0 a \in U_0 a ∈ U 0 and define
Ψ a ( x ) = P a → x ⋆ Log a ( x ) ∈ T x ⋆ M . \Psi_a(x)=P_{a\to x_\star}\operatorname{Log}_a(x)\in T_{x_\star}\mathcal{M}. Ψ a ( x ) = P a → x ⋆ Log a ( x ) ∈ T x ⋆ M . There is a uniformly bounded invertible linear map J a : T x ⋆ M → T x ⋆ M J_a:T_{x_\star}\mathcal{M}\to T_{x_\star}\mathcal{M} J a : T x ⋆ M → T x ⋆ M such that
Ψ a ( x ) − Ψ a ( x ⋆ ) = J a Log x ⋆ ( x ) + R a ( x ) , ∥ R a ( x ) ∥ ≤ C dist ( x , x ⋆ ) 2 . (54) \Psi_a(x)-\Psi_a(x_\star)=J_a\operatorname{Log}_{x_\star}(x)+R_a(x),\qquad\lVert R_a(x)\rVert\le C\operatorname{dist}(x,x_\star)^2. \tag*{(54)} Ψ a ( x ) − Ψ a ( x ⋆ ) = J a Log x ⋆ ( x ) + R a ( x ) , ∥ R a ( x )∥ ≤ C dist ( x , x ⋆ ) 2 . (54) If a → x ⋆ a\to x_\star a → x ⋆ , then J a → I J_a\to I J a → I in operator norm.
Proof. For a a a and x x x in the compact strongly convex set, ( a , x ) ↦ Ψ a ( x ) (a,x)\mapsto\Psi_a(x) ( a , x ) ↦ Ψ a ( x ) is C 2 C^2 C 2 . Apply Taylor’s theorem to x ↦ Ψ a ( x ) x\mapsto\Psi_a(x) x ↦ Ψ a ( x ) at x ⋆ x_\star x ⋆ in normal coordinates. Its derivative is
J a = D ( Ψ a ∘ Exp x ⋆ ) 0 , J_a=D(\Psi_a\circ\operatorname{Exp}_{x_\star})_0, J a = D ( Ψ a ∘ Exp x ⋆ ) 0 , and the uniform bound on the second derivative gives the quadratic remainder in (C.10). The inverse function theorem shows that J a J_a J a is invertible after U 0 U_0 U 0 is chosen sufficiently small; compactness makes the inverse bound uniform. At a = x ⋆ a=x_\star a = x ⋆ , Ψ x ⋆ ( x ) = Log x ⋆ ( x ) \Psi_{x_\star}(x)=\operatorname{Log}_{x_\star}(x) Ψ x ⋆ ( x ) = Log x ⋆ ( x ) and hence J x ⋆ = I J_{x_\star}=I J x ⋆ = I . Continuity in a a a gives the last assertion.
Averaging remainder
We first isolate the averaging remainder for the globally defined auxiliary array. Within this subsection only, abbreviate Δ k , n l o c \Delta^{\mathrm{loc}}_{k,n} Δ k , n loc by Δ k \Delta_k Δ k and the coefficients in (C.9) by a k + 1 a_{k+1} a k + 1 , m k + 1 m_{k+1} m k + 1 , and q k + 1 q_{k+1} q k + 1 . All norms in this subsection are in T x ⋆ M T_{x_\star}\mathcal{M} T x ⋆ M .
Lemma C.6 (Abel decomposition). Under the recursion (C.9), for 1 ≤ m ≤ N 1\le m\le N 1 ≤ m ≤ N ,
H ∑ k = b b + m − 1 Δ k = − ∑ k = b b + m − 1 ξ k + 1 + R m , n , (55) H\sum_{k=b}^{b+m-1}\Delta_k=-\sum_{k=b}^{b+m-1}\xi_{k+1}+R_{m,n}, \tag*{(55)} H k = b ∑ b + m − 1 Δ k = − k = b ∑ b + m − 1 ξ k + 1 + R m , n , (55) where R m , n R_{m,n} R m , n is the sum of
R m , n ( 1 ) = η b + 1 − 1 Δ b − η b + m − 1 Δ b + m , (56) R^{(1)}_{m,n}=\eta_{b+1}^{-1}\Delta_b-\eta_{b+m}^{-1}\Delta_{b+m}, \tag*{(56)} R m , n ( 1 ) = η b + 1 − 1 Δ b − η b + m − 1 Δ b + m , (56) R m , n ( 2 ) = ∑ k = b + 1 b + m − 1 ( η k + 1 − 1 − η k − 1 ) Δ k , (57) R^{(2)}_{m,n}=\sum_{k=b+1}^{b+m-1}\left(\eta_{k+1}^{-1}-\eta_k^{-1}\right)\Delta_k, \tag*{(57)} R m , n ( 2 ) = k = b + 1 ∑ b + m − 1 ( η k + 1 − 1 − η k − 1 ) Δ k , (57) R^{(3)}_{m,n}=\sum_{k=b}^{b+m-1}a_{k+1},\qquad R^{(4)}_{m,n}=\sum_{k=b}^{b+m-1}m_{k+1},
{#eq:C.14}
R m , n ( 5 ) = ∑ k = b b + m − 1 η k + 1 q k + 1 . (58) R^{(5)}_{m,n}=\sum_{k=b}^{b+m-1}\eta_{k+1}q_{k+1}. \tag*{(58)} R m , n ( 5 ) = k = b ∑ b + m − 1 η k + 1 q k + 1 . (58) The groups in (C.12), (C.13), (C.14), and (C.15) exhaust R m , n R_{m,n} R m , n . A harmless shift of the first or last iterate changes (C.11) by o p ( N ) o_p(\sqrt{N}) o p ( N ) .
Proof. Divide (C.9) by η k + 1 \eta_{k+1} η k + 1 and rearrange:
H Δ k = η k + 1 − 1 ( Δ k − Δ k + 1 ) − ξ k + 1 + a k + 1 + m k + 1 + η k + 1 q k + 1 . H\Delta_k=\eta_{k+1}^{-1}(\Delta_k-\Delta_{k+1})-\xi_{k+1}+a_{k+1}+m_{k+1}+\eta_{k+1}q_{k+1}. H Δ k = η k + 1 − 1 ( Δ k − Δ k + 1 ) − ξ k + 1 + a k + 1 + m k + 1 + η k + 1 q k + 1 . Summation from k = b k=b k = b to b + m − 1 b+m-1 b + m − 1 and discrete integration by parts give (C.12)–(C.15).
Lemma C.7 (Negligibility of the Abel remainder). Under Assumptions 3.1–3.4,
1 N R N , n → p 0 (59) \frac{1}{\sqrt{N}}R_{N,n} \xrightarrow{p} 0 \tag*{(59)} N 1 R N , n p 0 (59) and
1 N 2 ∑ m = 1 N ∥ R m , n ∥ 2 → p 0. (60) \frac{1}{N^2}\sum_{m=1}^{N}\lVert R_{m,n}\rVert^2 \xrightarrow{p} 0. \tag*{(60)} N 2 1 m = 1 ∑ N ∥ R m , n ∥ 2 p 0. (60)
Proof. We treat the five terms separately.
For R m , n ( 1 ) R_{m,n}^{(1)} R m , n ( 1 ) , Assumption 3.4 gives
E ∥ η k − 1 Δ k ∥ 2 ≤ C η k − 1 ≤ C k β . \mathbb{E}\left\lVert\eta_k^{-1}\Delta_k\right\rVert^2 \le C\eta_k^{-1}\le Ck^\beta. E η k − 1 Δ k 2 ≤ C η k − 1 ≤ C k β .
Therefore
1 N E ∥ R N , n ( 1 ) ∥ 2 ≤ C b β + n β N ⟶ 0 , \frac{1}{N}\mathbb{E}\left\lVert R_{N,n}^{(1)}\right\rVert^2 \le C\frac{b^\beta+n^\beta}{N}\longrightarrow0, N 1 E R N , n ( 1 ) 2 ≤ C N b β + n β ⟶ 0 ,
because b / n → 0 b/n\to0 b / n → 0 and β < 1 \beta<1 β < 1 . Also,
1 N 2 ∑ m = 1 N E ∥ η b + m − 1 Δ b + m ∥ 2 ≤ C N 2 ∑ k = b + 1 n k β = O ( n β − 1 ) → 0. \frac{1}{N^2}\sum_{m=1}^{N}\mathbb{E}\left\lVert\eta_{b+m}^{-1}\Delta_{b+m}\right\rVert^2 \le\frac{C}{N^2}\sum_{k=b+1}^{n}k^\beta=O(n^{\beta-1})\to0. N 2 1 m = 1 ∑ N E η b + m − 1 Δ b + m 2 ≤ N 2 C k = b + 1 ∑ n k β = O ( n β − 1 ) → 0.
The initial boundary contributes N − 1 η b − 1 = O ( b β / N ) → 0 N^{-1}\eta_b^{-1}=O(b^\beta/N)\to0 N − 1 η b − 1 = O ( b β / N ) → 0 to the same display.
For R m , n ( 2 ) R_{m,n}^{(2)} R m , n ( 2 ) , the mean value theorem gives
0 ≤ η k + 1 − 1 − η k − 1 ≤ C k β − 1 . 0\le\eta_{k+1}^{-1}-\eta_k^{-1}\le Ck^{\beta-1}. 0 ≤ η k + 1 − 1 − η k − 1 ≤ C k β − 1 .
Minkowski’s inequality and (15) yield
{ E ∥ R m , n ( 2 ) ∥ 2 } 1 / 2 ≤ C ∑ k = b + 1 b + m − 1 k β − 1 k − β / 2 ≤ C ( b + m ) β / 2 . \left\{\mathbb{E}\left\lVert R_{m,n}^{(2)}\right\rVert^2\right\}^{1/2}
\le C\sum_{k=b+1}^{b+m-1}k^{\beta-1}k^{-\beta/2}
\le C(b+m)^{\beta/2}. { E R m , n ( 2 ) 2 } 1/2 ≤ C k = b + 1 ∑ b + m − 1 k β − 1 k − β /2 ≤ C ( b + m ) β /2 .
Consequently,
1 N E ∥ R N , n ( 2 ) ∥ 2 = O ( n β − 1 ) → 0 , 1 N 2 ∑ m = 1 N E ∥ R m , n ( 2 ) ∥ 2 = O ( n β − 1 ) → 0. \frac{1}{N}\mathbb{E}\left\lVert R_{N,n}^{(2)}\right\rVert^2=O(n^{\beta-1})\to0,
\qquad
\frac{1}{N^2}\sum_{m=1}^{N}\mathbb{E}\left\lVert R_{m,n}^{(2)}\right\rVert^2=O(n^{\beta-1})\to0. N 1 E R N , n ( 2 ) 2 = O ( n β − 1 ) → 0 , N 2 1 m = 1 ∑ N E R m , n ( 2 ) 2 = O ( n β − 1 ) → 0.
For R m , n ( 3 ) R_{m,n}^{(3)} R m , n ( 3 ) , (C.4), Jensen’s inequality, and local boundedness give
E ∥ a k + 1 ∥ ≤ C { ( E ∥ Δ k ∥ 2 ) ( 1 + ρ ) / 2 + E ∥ Δ k ∥ 2 } ≤ C { k − β ( 1 + ρ ) / 2 + k − β } . \mathbb{E}\lVert a_{k+1}\rVert
\le C\left\{\left(\mathbb{E}\lVert\Delta_k\rVert^2\right)^{(1+\rho)/2}+\mathbb{E}\lVert\Delta_k\rVert^2\right\}
\le C\left\{k^{-\beta(1+\rho)/2}+k^{-\beta}\right\}. E ∥ a k + 1 ∥ ≤ C { ( E ∥ Δ k ∥ 2 ) ( 1 + ρ ) /2 + E ∥ Δ k ∥ 2 } ≤ C { k − β ( 1 + ρ ) /2 + k − β } .
Hence
1 N E max m ≤ N ∥ R m , n ( 3 ) ∥ ≤ 1 N ∑ k = b n − 1 E ∥ a k + 1 ∥ = o ( 1 ) \frac{1}{\sqrt{N}}\mathbb{E}\max_{m\le N}\left\lVert R_{m,n}^{(3)}\right\rVert
\le\frac{1}{\sqrt{N}}\sum_{k=b}^{n-1}\mathbb{E}\lVert a_{k+1}\rVert=o(1) N 1 E m ≤ N max R m , n ( 3 ) ≤ N 1 k = b ∑ n − 1 E ∥ a k + 1 ∥ = o ( 1 )
by β ( 1 + ρ ) > 1 \beta(1+\rho)>1 β ( 1 + ρ ) > 1 and β > 1 / 2 \beta>1/2 β > 1/2 . Markov’s inequality proves uniform o p ( N ) o_p(\sqrt{N}) o p ( N ) control, which implies both (C.16) and (C.17) for this term. For R m , n ( 4 ) R^{(4)}_{m,n} R m , n ( 4 ) , Doob’s inequality and (C.5) imply
1 N E max m ≤ N ∥ R m , n ( 4 ) ∥ 2 ≤ C N ∑ k = b n − 1 E { ∥ Δ k ∥ 2 ν ˉ + ∥ Δ k ∥ 2 } ≤ C N ∑ k = b n − 1 { η ˉ k + η k } ⟶ 0. \frac{1}{N}\mathbb{E}\max_{m\leq N}\left\|R^{(4)}_{m,n}\right\|^2 \leq\frac{C}{N}\sum_{k=b}^{n-1}\mathbb{E}\left\{\|\Delta_k\|^{2\bar{\nu}}+\|\Delta_k\|^2\right\}
\leq\frac{C}{N}\sum_{k=b}^{n-1}\{\bar{\eta}_k+\eta_k\}\longrightarrow0. N 1 E m ≤ N max R m , n ( 4 ) 2 ≤ N C k = b ∑ n − 1 E { ∥ Δ k ∥ 2 ν ˉ + ∥ Δ k ∥ 2 } ≤ N C k = b ∑ n − 1 { η ˉ k + η k } ⟶ 0.
For ν ˉ ≤ 1 \bar{\nu}\leq1 ν ˉ ≤ 1 , the second line uses Jensen’s inequality; local boundedness covers any original ν > 1 \nu>1 ν > 1 . Thus this term is uniformly o p ( N ) o_p(\sqrt{N}) o p ( N ) .
Finally, (C.6) gives
1 N E max m ≤ N ∥ R m , n ( 5 ) ∥ ≤ C N ∑ k = b n − 1 η k + 1 = O ( n 1 / 2 − β ) ⟶ 0. \frac{1}{\sqrt{N}}\mathbb{E}\max_{m\leq N}\left\|R^{(5)}_{m,n}\right\|\leq\frac{C}{\sqrt{N}}\sum_{k=b}^{n-1}\eta_{k+1}=O(n^{1/2-\beta})\longrightarrow0. N 1 E m ≤ N max R m , n ( 5 ) ≤ N C k = b ∑ n − 1 η k + 1 = O ( n 1/2 − β ) ⟶ 0.
Combining the five estimates proves the lemma.
The same estimates quantify the auxiliary averaging error:
∥ R N , n ∥ N + { 1 N 2 ∑ m = 1 N ∥ R m , n ∥ 2 } 1 / 2 = O p ( n − ( 1 − β ) / 2 + n − { β ( 1 + ρ ) − 1 } / 2 + n 1 / 2 − β + n − ν ˉ β / 2 ) . \frac{\|R_{N,n}\|}{\sqrt{N}}+\left\{\frac{1}{N^2}\sum_{m=1}^{N}\|R_{m,n}\|^2\right\}^{1/2}
=O_p\left(n^{-(1-\beta)/2}+n^{-\{\beta(1+\rho)-1\}/2}+n^{1/2-\beta}+n^{-\bar{\nu}\beta/2}\right). N ∥ R N , n ∥ + { N 2 1 m = 1 ∑ N ∥ R m , n ∥ 2 } 1/2 = O p ( n − ( 1 − β ) /2 + n − { β ( 1 + ρ ) − 1 } /2 + n 1/2 − β + n − ν ˉ β /2 ) .
The first term comes from boundaries and changing gains, the second from the mean-field remainder, the third from quadratic and retraction terms, and the fourth from state-dependent noise. For ρ = 1 \rho=1 ρ = 1 , choosing β = 0.6 \beta=0.6 β = 0.6 gives a slowest bound n − 0.1 n^{-0.1} n − 0.1 when ν ˉ ≥ 1 / 3 \bar{\nu}\geq1/3 ν ˉ ≥ 1/3 ; taking β \beta β near one instead slows the boundary bound. These are stochastic-approximation remainder bounds, not coverage-error rates: the martingale approximation, localization probability, and inverse-normalizer tails require separate quantitative control.
Chart and bridge limits
Here Δ k \Delta_k Δ k again denotes the original normal-coordinate path defined in § 3.1 ; Δ k , n l o c \Delta^{\mathrm{loc}}_{k,n} Δ k , n loc continues to denote the auxiliary array.
Lemma C.8 (Chart partial sums). Let
v n ⋆ = Log a n ( x ⋆ ) , Y k , n = P a n → x ⋆ ( v k , n − v n ⋆ ) . v_n^\star=\operatorname{Log}_{a_n}(x_\star),\qquad Y_{k,n}=P_{a_n\to x_\star}(v_{k,n}-v_n^\star). v n ⋆ = Log a n ( x ⋆ ) , Y k , n = P a n → x ⋆ ( v k , n − v n ⋆ ) . Under the assumptions of Theorem 3.8 ,
1 N ∑ k = b + 1 b + ⌊ N r ⌋ { Y k , n − J a n Δ k } → p 0 (61) \frac{1}{\sqrt{N}}\sum_{k=b+1}^{b+\lfloor Nr\rfloor}\{Y_{k,n}-J_{a_n}\Delta_k\}\xrightarrow{p}0 \tag*{(61)} N 1 k = b + 1 ∑ b + ⌊ N r ⌋ { Y k , n − J a n Δ k } p 0 (61) uniformly in r ∈ [ 0 , 1 ] r\in[0,1] r ∈ [ 0 , 1 ] , and J a n → p I J_{a_n}\xrightarrow{p}I J a n p I .
Proof. On the agreement event E n E_n E n in Lemma C.4 , expansion (C.10) gives
∥ Y k , n − J a n Δ k ∥ ≤ C ∥ Δ k , n l o c ∥ 2 . \|Y_{k,n}-J_{a_n}\Delta_k\|\leq C\left\|\Delta^{\mathrm{loc}}_{k,n}\right\|^2. ∥ Y k , n − J a n Δ k ∥ ≤ C Δ k , n loc 2 .
Consequently, for every ϵ > 0 \epsilon> 0 ϵ > 0 ,
P { 1 N ∑ k = b + 1 n ∥ Y k , n − J a n Δ k ∥ > ϵ } ≤ P ( E n c ) + C ϵ N ∑ k = b + 1 n E ∥ Δ k , n l o c ∥ 2 ≤ P ( E n c ) + C ϵ N ∑ k = b + 1 n η k ⟶ 0. \mathbb{P}\left\{\frac{1}{\sqrt{N}}\sum_{k=b+1}^{n}\left\|Y_{k,n}-J_{a_n}\Delta_k\right\|>\epsilon\right\}
\leq\mathbb{P}(E_n^{\mathrm{c}})+\frac{C}{\epsilon\sqrt{N}}\sum_{k=b+1}^{n}\mathbb{E}\left\|\Delta_{k,n}^{\mathrm{loc}}\right\|^2
\leq\mathbb{P}(E_n^{\mathrm{c}})+\frac{C}{\epsilon\sqrt{N}}\sum_{k=b+1}^{n}\eta_k\longrightarrow0. P { N 1 k = b + 1 ∑ n ∥ Y k , n − J a n Δ k ∥ > ϵ } ≤ P ( E n c ) + ϵ N C k = b + 1 ∑ n E Δ k , n loc 2 ≤ P ( E n c ) + ϵ N C k = b + 1 ∑ n η k ⟶ 0.
This bound controls every partial sum and proves the uniform convergence in (C.18). Moreover,
P { dist ( a n , x ⋆ ) > ϵ } ≤ P ( E n c ) + ϵ − 2 E ∥ Δ b n , n l o c ∥ 2 ⟶ 0. \mathbb{P}\{\operatorname{dist}(a_n,x_\star)>\epsilon\}\leq\mathbb{P}(E_n^{\mathrm{c}})+\epsilon^{-2}\mathbb{E}\left\|\Delta_{b_n,n}^{\mathrm{loc}}\right\|^2\longrightarrow0. P { dist ( a n , x ⋆ ) > ϵ } ≤ P ( E n c ) + ϵ − 2 E Δ b n , n loc 2 ⟶ 0.
Thus a n → x ⋆ a_n\to x_\star a n → x ⋆ in probability, and the last part of Lemma C.5 gives J a n → I J_{a_n}\to I J a n → I .
Lemma C.9 (High-probability inverse-chart domain). There is an open neighborhood D \mathcal{D} D of ( x ⋆ , 0 ) (x_\star,0) ( x ⋆ , 0 ) in the tangent bundle on which ( a , v ) ↦ Exp a ( v ) (a,v)\mapsto\operatorname{Exp}_a(v) ( a , v ) ↦ Exp a ( v ) takes values in the common normal neighborhood U 1 U_1 U 1 , and
P { ( a n , v ˉ n ) ∈ D } ⟶ 1. \mathbb{P}\{(a_n,\bar{v}_n)\in\mathcal{D}\}\longrightarrow1. P {( a n , v ˉ n ) ∈ D } ⟶ 1. For any coordinate family χ a \chi_a χ a satisfying Proposition B.4(b), the analogous statement holds for a common domain of ( a , z ) ↦ χ a − 1 ( z ) (a,z)\mapsto\chi_a^{-1}(z) ( a , z ) ↦ χ a − 1 ( z ) , with z ˉ n = N n − 1 ∑ k = b n + 1 n χ a n ( x k ) \bar{z}_n=N_n^{-1}\sum_{k=b_n+1}^{n}\chi_{a_n}(x_k) z ˉ n = N n − 1 ∑ k = b n + 1 n χ a n ( x k ) .
Proof. The proof of Lemma C.8 gives a n → x ⋆ a_n\to x_\star a n → x ⋆ in probability. Combining the endpoint case of Lemma C.8 with Theorem 3.7 gives v ˉ n − v n ⋆ = O p ( N n − 1 / 2 ) \bar{v}_n-v_n^\star=O_p(N_n^{-1/2}) v ˉ n − v n ⋆ = O p ( N n − 1/2 ) . Smoothness of the logarithm on the common normal neighborhood gives v n ⋆ → 0 v_n^\star\to0 v n ⋆ → 0 in probability. Hence ( a n , v ˉ n ) → ( x ⋆ , 0 ) (a_n,\bar{v}_n)\to(x_\star,0) ( a n , v ˉ n ) → ( x ⋆ , 0 ) in probability. Local smoothness and the inverse function theorem provide an open neighborhood D \mathcal{D} D of that limit on which the exponential chart maps into U 1 U_1 U 1 , proving the first claim.
For the alternative chart, the uniform expansion (D.5), the same event-splitting argument as in Lemma C.8, and the endpoint partial sum limit give z ˉ n − χ a n ( x ⋆ ) = O p ( N n − 1 / 2 ) \bar{z}_n-\chi_{a_n}(x_\star)=O_p(N_n^{-1/2}) z ˉ n − χ a n ( x ⋆ ) = O p ( N n − 1/2 ) . Joint continuity gives χ a n ( x ⋆ ) → 0 \chi_{a_n}(x_\star)\to0 χ a n ( x ⋆ ) → 0 . The parameterized inverse function theorem then provides a common open inverse-chart domain containing ( a n , z ˉ n ) (a_n,\bar{z}_n) ( a n , z ˉ n ) with probability tending to one.
Lemma C.10 (Positive definiteness of the Brownian bridge Gram matrix). For fixed d d d , the matrix Q d Q_d Q d in (8) is positive definite almost surely.
Proof. Choose distinct points 0 < t 1 < ⋯ < t d < 1 0<t_1<\cdots<t_d<1 0 < t 1 < ⋯ < t d < 1 and form the random matrix M = ( ( B d ( t j ) ) i ) j , i = 1 d M=((B_d(t_j))_i)_{j,i=1}^d M = (( B d ( t j ) ) i ) j , i = 1 d . Its columns are independent, each with positive definite covariance ( min ( t j , t ℓ ) − t j t ℓ ) j , ℓ (\min(t_j,t_\ell)-t_jt_\ell)_{j,\ell} ( min ( t j , t ℓ ) − t j t ℓ ) j , ℓ . Hence vec ( M ) \operatorname{vec}(M) vec ( M ) is a nondegenerate Gaussian vector in R d 2 \mathbb{R}^{d^2} R d 2 , so M M M has a Lebesgue density. The polynomial det ( M ) \det(M) det ( M ) is not identically zero; therefore P { det ( M ) = 0 } = 0 \mathbb{P}\{\det(M)=0\}=0 P { det ( M ) = 0 } = 0 .
If Q d Q_d Q d were singular, there would be a nonzero u ∈ R d u\in\mathbb{R}^d u ∈ R d such that
0 = u T Q d u = ∫ 0 1 { u T B d ( r ) } 2 d r . 0=u^{\mathsf{T}}Q_du=\int_0^1\{u^{\mathsf{T}}B_d(r)\}^2\,\mathrm{d}r. 0 = u T Q d u = ∫ 0 1 { u T B d ( r ) } 2 d r .
Continuity of B d B_d B d would imply u T B d ( r ) = 0 u^{\mathsf{T}}B_d(r)=0 u T B d ( r ) = 0 for all r r r , and hence M u = 0 Mu=0 M u = 0 . Thus M M M would be singular. The preceding paragraph proves that this event has probability zero.
Proofs
Write b = b n b=b_n b = b n and N = N n N=N_n N = N n throughout this appendix.
Proof of Proposition 3.6
Proof of Proposition 3.6. We first verify the sufficient condition for the drift. Write ϕ = Log x ⋆ \phi=\operatorname{Log}_{x_\star} ϕ = Log x ⋆ , δ = Δ k , n l o c \delta=\Delta^{\mathrm{loc}}_{k,n} δ = Δ k , n loc , and η = η k + 1 \eta=\eta_{k+1} η = η k + 1 . For δ ∈ K = ϕ ( U ‾ 0 ) \delta\in\mathcal{K}=\phi(\overline{U}_0) δ ∈ K = ϕ ( U 0 ) , the coefficient construction in Lemma C.3 gives exactly
Δ k + 1 , n l o c = Φ ~ x { − η G k + 1 ( x ) } , x = ϕ − 1 ( δ ) . \Delta^{\mathrm{loc}}_{k+1,n}=\widetilde{\Phi}_x\{-\eta G_{k+1}(x)\},\qquad x=\phi^{-1}(\delta). Δ k + 1 , n loc = Φ x { − η G k + 1 ( x )} , x = ϕ − 1 ( δ ) .
The cutoff map has uniformly bounded first and second derivatives in its increment argument. In particular, its displacement from ϕ ( x ) \phi(x) ϕ ( x ) is bounded by C η ∥ G k + 1 ( x ) ∥ C\eta\lVert G_{k+1}(x)\rVert C η ∥ G k + 1 ( x )∥ for every increment, including those outside the retraction tube. For D = Δ k + 1 , n l o c − δ D=\Delta^{\mathrm{loc}}_{k+1,n}-\delta D = Δ k + 1 , n loc − δ , conditional second moments and (C.7) therefore imply
E ( ∥ D ∥ 2 ∣ F k ) ≤ C η 2 , E ( D ∣ F k ) = − η D ϕ x [ grad F ( x ) ] + r k , ∥ r k ∥ ≤ C η 2 . \mathbb{E}(\lVert D\rVert^2\mid\mathcal{F}_k)\le C\eta^2,\qquad\mathbb{E}(D\mid\mathcal{F}_k)=-\eta D\phi_x[\operatorname{grad}F(x)]+r_k,\qquad\lVert r_k\rVert\le C\eta^2. E (∥ D ∥ 2 ∣ F k ) ≤ C η 2 , E ( D ∣ F k ) = − η D ϕ x [ grad F ( x )] + r k , ∥ r k ∥ ≤ C η 2 .
The Gauss lemma identifies the radial derivative exactly:
⟨ δ , D ϕ x [ grad F ( x ) ] ⟩ = ⟨ δ , P x → x ⋆ grad F ( x ) ⟩ ≥ μ 0 ∥ δ ∥ 2 . \langle\delta,D\phi_x[\operatorname{grad}F(x)]\rangle=\langle\delta,P_{x\to x_\star}\operatorname{grad}F(x)\rangle\ge\mu_0\lVert\delta\rVert^2. ⟨ δ , D ϕ x [ grad F ( x )]⟩ = ⟨ δ , P x → x ⋆ grad F ( x )⟩ ≥ μ 0 ∥ δ ∥ 2 .
Expanding ∥ δ + D ∥ 2 \lVert\delta+D\rVert^2 ∥ δ + D ∥ 2 and using boundedness of K \mathcal{K} K gives E ( ∥ Δ k + 1 , n l o c ∥ 2 ∣ F k ) ≤ ( 1 − 2 μ 0 η ) ∥ δ ∥ 2 + C η 2 \mathbb{E}(\lVert\Delta^{\mathrm{loc}}_{k+1,n}\rVert^2\mid\mathcal{F}_k)\le(1-2\mu_0\eta)\lVert\delta\rVert^2+C\eta^2 E (∥ Δ k + 1 , n loc ∥ 2 ∣ F k ) ≤ ( 1 − 2 μ 0 η ) ∥ δ ∥ 2 + C η 2 . For δ ∉ K \delta\notin\mathcal{K} δ ∈ / K , the update is ( I − η H ) δ − η ξ k + 1 (I-\eta H)\delta-\eta\xi_{k+1} ( I − η H ) δ − η ξ k + 1 . Conditional centering and positive definiteness of H H H give the same form of bound with a possibly smaller positive drift constant, for all sufficiently small η \eta η . Since b n → ∞ b_n\to\infty b n → ∞ , these bounds establish (21) uniformly over the retained array for all sufficiently large n n n , without using (15) .
Let u k , n = E ∥ Δ k , n l o c ∥ 2 u_{k,n}=\mathbb{E}\lVert\Delta^{\mathrm{loc}}_{k,n}\rVert^2 u k , n = E ∥ Δ k , n loc ∥ 2 . Taking expectations in (21) gives
u k + 1 , n ≤ ( 1 − c 0 η k + 1 ) u k , n + C 0 η k + 1 2 . u_{k+1,n}\le(1-c_0\eta_{k+1})u_{k,n}+C_0\eta_{k+1}^2. u k + 1 , n ≤ ( 1 − c 0 η k + 1 ) u k , n + C 0 η k + 1 2 .
For η k = η 0 k − β \eta_k=\eta_0k^{-\beta} η k = η 0 k − β with β < 1 \beta<1 β < 1 ,
η k η k + 1 = ( 1 + 1 k ) β ≤ 1 + C k . \frac{\eta_k}{\eta_{k+1}}=\left(1+\frac{1}{k}\right)^\beta\le1+\frac{C}{k}. η k + 1 η k = ( 1 + k 1 ) β ≤ 1 + k C .
Because k − 1 = o ( η k + 1 ) k^{-1}=o(\eta_{k+1}) k − 1 = o ( η k + 1 ) , there is k 0 k_0 k 0 such that C / k ≤ c 0 η k + 1 / 2 C/k\le c_0\eta_{k+1}/2 C / k ≤ c 0 η k + 1 /2 for k ≥ k 0 k\ge k_0 k ≥ k 0 . Choose K ≥ C b K\ge C_b K ≥ C b so large that C 0 ≤ K c 0 / 4 C_0\le Kc_0/4 C 0 ≤ K c 0 /4 . If u k , n ≤ K η k u_{k,n}\le K\eta_k u k , n ≤ K η k , then
u k + 1 , n ≤ K η k ( 1 − c 0 η k + 1 ) + C 0 η k + 1 2 ≤ K η k + 1 ( 1 + C 0 2 η k + 1 ) ( 1 − c 0 η k + 1 ) + C 0 η k + 1 2 ≤ K η k + 1 − K c 0 4 η k + 1 2 + C 0 η k + 1 2 ≤ K η k + 1 , \begin{aligned}
u_{k+1,n}&\le K\eta_k(1-c_0\eta_{k+1})+C_0\eta_{k+1}^2\\
&\le K\eta_{k+1}\left(1+\frac{C_0}{2}\eta_{k+1}\right)(1-c_0\eta_{k+1})+C_0\eta_{k+1}^2\\
&\le K\eta_{k+1}-\frac{Kc_0}{4}\eta_{k+1}^2+C_0\eta_{k+1}^2\le K\eta_{k+1},
\end{aligned} u k + 1 , n ≤ K η k ( 1 − c 0 η k + 1 ) + C 0 η k + 1 2 ≤ K η k + 1 ( 1 + 2 C 0 η k + 1 ) ( 1 − c 0 η k + 1 ) + C 0 η k + 1 2 ≤ K η k + 1 − 4 K c 0 η k + 1 2 + C 0 η k + 1 2 ≤ K η k + 1 ,
where k 0 k_0 k 0 can be enlarged if necessary. Take n n n sufficiently large that b n ≥ k 0 b_n\ge k_0 b n ≥ k 0 . The uniform initial bound gives u b n , n ≤ K η b n u_{b_n,n}\le K\eta_{b_n} u b n , n ≤ K η b n , and induction from b n b_n b n to n n n proves (15) with constants independent of n n n .
Proof of Theorem 3.7
Proof of Theorem 3.7. In this proof, Δ k \Delta_k Δ k abbreviates the auxiliary array Δ k , n l o c \Delta^{\mathrm{loc}}_{k,n} Δ k , n loc ; the last step transfers the limit to the original path. By the martingale functional central limit theorem under Assumption 3.5,
1 N ∑ k = b b + ⌊ N r ⌋ − 1 ξ k + 1 ⇝ Σ 1 / 2 W d ( r ) (62) \frac{1}{\sqrt{N}}\sum_{k=b}^{b+\lfloor Nr\rfloor-1}\xi_{k+1}\rightsquigarrow\Sigma^{1/2}W_d(r) \tag*{(62)} N 1 k = b ∑ b + ⌊ N r ⌋ − 1 ξ k + 1 ⇝ Σ 1/2 W d ( r ) (62)
in D ( [ 0 , 1 ] ; R d ) D([0,1];\mathbb{R}^d) D ([ 0 , 1 ] ; R d ) . Since the limit is continuous, this also implies convergence in L 2 [ 0 , 1 ] L^2[0,1] L 2 [ 0 , 1 ] , jointly with the endpoint.
By Lemmas C.6 and C.7 , the difference between the retained Δ \Delta Δ partial sum path and − H − 1 -H^{-1} − H − 1 times the path in (D.1) converges to zero in L 2 L^2 L 2 , and the endpoint difference is o p ( 1 ) o_p(1) o p ( 1 ) . For the shift between sums starting at b b b and b + 1 b+1 b + 1 , the path difference at r r r is N − 1 / 2 ( Δ b + ⌊ N r ⌋ − Δ b ) N^{-1/2}(\Delta_{b+\lfloor Nr\rfloor}-\Delta_b) N − 1/2 ( Δ b + ⌊ N r ⌋ − Δ b ) . Therefore
∫ 0 1 ∥ Δ b + ⌊ N r ⌋ − Δ b N ∥ 2 d r ≤ 2 N ∥ Δ b ∥ 2 + 2 N 2 ∑ k = b n − 1 ∥ Δ k ∥ 2 ⟶ p 0 \int_0^1\left\|\frac{\Delta_{b+\lfloor Nr\rfloor}-\Delta_b}{\sqrt{N}}\right\|^2\,dr
\leq\frac{2}{N}\|\Delta_b\|^2+\frac{2}{N^2}\sum_{k=b}^{n-1}\|\Delta_k\|^2 \mathrel{\mathop{\longrightarrow}^{p}}0 ∫ 0 1 N Δ b + ⌊ N r ⌋ − Δ b 2 d r ≤ N 2 ∥ Δ b ∥ 2 + N 2 2 k = b ∑ n − 1 ∥ Δ k ∥ 2 ⟶ p 0
by (15) and Markov’s inequality. Separately, the endpoint shift is bounded by N − 1 / 2 ( ∥ Δ b ∥ + ∥ Δ n ∥ ) = o p ( 1 ) N^{-1/2}(\|\Delta_b\|+\|\Delta_n\|)=o_p(1) N − 1/2 ( ∥ Δ b ∥ + ∥ Δ n ∥ ) = o p ( 1 ) . Slutsky’s theorem in L 2 × R d L^2\times\mathbb{R}^d L 2 × R d proves the auxiliary version of (23). The agreement event in Lemma C.4 then transfers the joint L 2 L^2 L 2 -path and endpoint limit to the retained original recursion, proving (23).
Proof of Theorem 3.8
Proof of Theorem 3.8. Here Δ k \Delta_k Δ k denotes the original normal-coordinate path from § 3.1 . Write b = b n b=b_n b = b n and N = N n N=N_n N = N n . Parallel transport all vectors and operators from T a n M T_{a_n}\mathcal{M} T a n M to T x ⋆ M T_{x_\star}\mathcal{M} T x ⋆ M ; parallel transport is an isometry and does not change the statistic.
By Lemma C.8 and Theorem 3.7 ,
{ 1 N ∑ k = b + 1 b + ⌊ N r ⌋ ( v k , n − v n ⋆ ) , 1 N ∑ k = b + 1 n ( v k , n − v n ⋆ ) } ⇝ { − L W d ( r ) , − L W d ( 1 ) } , (63) \left\{\frac{1}{\sqrt{N}}\sum_{k=b+1}^{b+\lfloor Nr\rfloor}(v_{k,n}-v_n^\star),\frac{1}{\sqrt{N}}\sum_{k=b+1}^{n}(v_{k,n}-v_n^\star)\right\}\rightsquigarrow\{-LW_d(r),-LW_d(1)\}, \tag*{(63)} ⎩ ⎨ ⎧ N 1 k = b + 1 ∑ b + ⌊ N r ⌋ ( v k , n − v n ⋆ ) , N 1 k = b + 1 ∑ n ( v k , n − v n ⋆ ) ⎭ ⎬ ⎫ ⇝ { − L W d ( r ) , − L W d ( 1 )} , (63)
where L = H − 1 Σ 1 / 2 L=H^{-1}\Sigma^{1/2} L = H − 1 Σ 1/2 , and the equality is understood after transport. The sign agrees with Theorem 3.7 and cancels in the quadratic statistic.
Because constants cancel from the bridge, (D.2) gives, in L 2 [ 0 , 1 ] L^2[0,1] L 2 [ 0 , 1 ] ,
1 N C ⌊ N r ⌋ , n ⇝ − L B d ( r ) . (64) \frac{1}{\sqrt{N}}C_{\lfloor Nr\rfloor,n}\rightsquigarrow-LB_d(r). \tag*{(64)} N 1 C ⌊ N r ⌋ , n ⇝ − L B d ( r ) . (64)
To transfer (D.3) to the normalizer, note that the map f ↦ ∫ 0 1 f ( r ) f ( r ) T d r f\mapsto\int_0^1f(r)f(r)^{\mathsf{T}}\,dr f ↦ ∫ 0 1 f ( r ) f ( r ) T d r is continuous from L 2 ( [ 0 , 1 ] ; R d ) L^2([0,1];\mathbb{R}^d) L 2 ([ 0 , 1 ] ; R d ) to the space of symmetric matrices: indeed,
∥ ∫ ( f f T − g g T ) ∥ o p ≤ ( ∥ f ∥ L 2 + ∥ g ∥ L 2 ) ∥ f − g ∥ L 2 . \left\|\int(f f^{\mathsf{T}}-g g^{\mathsf{T}})\right\|_{\mathrm{op}}\leq(\|f\|_{L^2}+\|g\|_{L^2})\|f-g\|_{L^2}. ∫ ( f f T − g g T ) op ≤ ( ∥ f ∥ L 2 + ∥ g ∥ L 2 ) ∥ f − g ∥ L 2 .
Because C 0 , n = C N , n = 0 C_{0,n}=C_{N,n}=0 C 0 , n = C N , n = 0 , the discrete sum in (6) is exactly the integral of the corresponding step process. Thus
V ^ n ⇝ L Q d L T . \widehat{V}_n\rightsquigarrow LQ_dL^{\mathsf{T}}. V n ⇝ L Q d L T .
This proves part (b). If Σ \Sigma Σ is positive definite, then L L L is invertible; Lemma C.10 implies that the limiting normalizer is positive definite and hence that V ^ n \widehat{V}_n V n is invertible with probability tending to one.
For the point estimator, (D.2) gives
N P a n → x ⋆ ( v ˉ n − v n ⋆ ) ⇝ − L W d ( 1 ) . (65) \sqrt{N}\operatorname{P}_{a_n\to x_\star}(\bar v_n-v_n^\star)\rightsquigarrow-LW_d(1). \tag*{(65)} N P a n → x ⋆ ( v ˉ n − v n ⋆ ) ⇝ − L W d ( 1 ) . (65)
By Lemma C.9 , the inverse chart is evaluated on its common domain with probability tending to one. On that event, the inverse chart expansion from Lemma C.5 gives
Log x ⋆ { Exp a n ( v ˉ n ) } = J a n − 1 P a n → x ⋆ ( v ˉ n − v n ⋆ ) + O p ( ∥ v ˉ n − v n ⋆ ∥ 2 ) . \operatorname{Log}_{x_\star}\{\operatorname{Exp}_{a_n}(\bar v_n)\}=J_{a_n}^{-1}\operatorname{P}_{a_n\to x_\star}(\bar v_n-v_n^\star)+O_p(\lVert\bar v_n-v_n^\star\rVert^2). Log x ⋆ { Exp a n ( v ˉ n )} = J a n − 1 P a n → x ⋆ ( v ˉ n − v n ⋆ ) + O p (∥ v ˉ n − v n ⋆ ∥ 2 ) .
The squared term is O p ( N − 1 ) O_p(N^{-1}) O p ( N − 1 ) , and J a n → I J_{a_n}\to I J a n → I , so (24) follows.
At x = x ⋆ x=x_\star x = x ⋆ , the numerator in (7) , after multiplication by N \sqrt{N} N and transport, converges jointly with V ^ n \widehat V_n V n to − L W d ( 1 ) -LW_d(1) − L W d ( 1 ) . Therefore
T n ( x ⋆ ) ⇝ W d ( 1 ) T L T ( L Q d L T ) − 1 L W d ( 1 ) = W d ( 1 ) T Q d − 1 W d ( 1 ) = T d . \begin{aligned}
T_n(x_\star)\rightsquigarrow W_d(1)^{\mathsf T}L^{\mathsf T}(LQ_dL^{\mathsf T})^{-1}LW_d(1) \\
&= W_d(1)^{\mathsf T}Q_d^{-1}W_d(1)=T_d.
\end{aligned} T n ( x ⋆ ) ⇝ W d ( 1 ) T L T ( L Q d L T ) − 1 L W d ( 1 ) = W d ( 1 ) T Q d − 1 W d ( 1 ) = T d .
A Brownian bridge is independent of W d ( 1 ) W_d(1) W d ( 1 ) . Conditional on the positive definite Q d Q_d Q d , the final quadratic form has a continuous distribution. Hence the unconditional distribution is continuous, and (27) follows.
Proof of Corollary 3.9
Proof of Corollary 3.9. The intrinsic delta method applied to (24) gives (28) . By consistency and continuity, D n D_n D n converges, after the natural tangent identifications, to D h x ⋆ Dh_{x_\star} D h x ⋆ . The numerator and the transformed self-normalizer therefore have a joint limit of the form
− A h W d ( 1 ) , A h Q d A h T , -A_hW_d(1),\qquad A_hQ_dA_h^{\mathsf T}, − A h W d ( 1 ) , A h Q d A h T ,
where A h A_h A h has row rank q q q . Let C A = ( A h A h T ) 1 / 2 C_A=(A_hA_h^{\mathsf T})^{1/2} C A = ( A h A h T ) 1/2 and W ~ q = C A − 1 A h W d \widetilde W_q=C_A^{-1}A_hW_d W q = C A − 1 A h W d . Then W ~ q \widetilde W_q W q is a standard q q q -dimensional Brownian motion and
A h Q d A h T = C A { ∫ 0 1 ( W ~ q ( r ) − r W ~ q ( 1 ) ) ( W ~ q ( r ) − r W ~ q ( 1 ) ) T d r } C A . A_hQ_dA_h^{\mathsf T}=C_A\left\{\int_0^1(\widetilde W_q(r)-r\widetilde W_q(1))(\widetilde W_q(r)-r\widetilde W_q(1))^{\mathsf T}\,\mathrm{d}r\right\}C_A. A h Q d A h T = C A { ∫ 0 1 ( W q ( r ) − r W q ( 1 )) ( W q ( r ) − r W q ( 1 ) ) T d r } C A .
By Lemma C.10 , this limit is positive definite almost surely, so V ^ h , n \widehat V_{h,n} V h , n is invertible with probability tending to one. Cancelling C A C_A C A from the projected endpoint and bridge normalizer yields T q T_q T q . □
Proof of Corollary B.1. The convergence of V ^ n \widehat V_n V n in (25) implies ∥ V ^ n ∥ o p = O p ( 1 ) \lVert\widehat V_n\rVert_{\mathrm{op}}=O_p(1) ∥ V n ∥ op = O p ( 1 ) , while continuity of D h Dh D h and consistency of ( a n , v ˉ n ) (a_n,\bar v_n) ( a n , v ˉ n ) imply ∥ D n ∥ o p = O p ( 1 ) \lVert D_n\rVert_{\mathrm{op}}=O_p(1) ∥ D n ∥ op = O p ( 1 ) . Hence
∥ V ^ h , n a p p − V ^ h , n ∥ o p ≤ ∥ D ^ n − D n ∥ o p ∥ V ^ n ∥ o p { ∥ D ^ n ∥ o p + ∥ D n ∥ o p } = o p ( 1 ) . \left\lVert\widehat V_{h,n}^{\mathrm{app}}-\widehat V_{h,n}\right\rVert_{\mathrm{op}}\leq\left\lVert\widehat D_n-D_n\right\rVert_{\mathrm{op}}\lVert\widehat V_n\rVert_{\mathrm{op}}\{\lVert\widehat D_n\rVert_{\mathrm{op}}+\lVert D_n\rVert_{\mathrm{op}}\}=o_p(1). V h , n app − V h , n op ≤ D n − D n op ∥ V n ∥ op {∥ D n ∥ op + ∥ D n ∥ op } = o p ( 1 ) .
The projected endpoint and normalizer therefore have the same joint limit as in the proof of Corollary 3.9. That limit is positive definite almost surely under the row-rank condition, so continuity of matrix inversion and Slutsky’s theorem prove the first claim.
For the centered-difference statement, choose orthonormal coordinates in the fixed-dimensional space T a n M T_{a_n}\mathcal{M} T a n M and write g n ( v ) = h { Exp a n ( v ) } g_n(v)=h\{\operatorname{Exp}_{a_n}(v)\} g n ( v ) = h { Exp a n ( v )} . Componentwise Taylor expansion at v ˉ n \bar v_n v ˉ n gives
g n ( v ˉ n + s n e j ) − g n ( v ˉ n − s n e j ) 2 s n = D g n ( v ˉ n ) [ e j ] + O p ( s n 2 ) , \frac{g_n(\bar v_n+s_ne_j)-g_n(\bar v_n-s_ne_j)}{2s_n}=Dg_n(\bar v_n)[e_j]+O_p(s_n^2), 2 s n g n ( v ˉ n + s n e j ) − g n ( v ˉ n − s n e j ) = D g n ( v ˉ n ) [ e j ] + O p ( s n 2 ) ,
uniformly over the finitely many coordinate directions. The common-domain and uniform C 3 C^3 C 3 conditions make the remainder uniform, hence ∥ D ^ n − D n ∥ o p = O p ( s n 2 ) = o p ( 1 ) \lVert\widehat D_n-D_n\rVert_{\mathrm{op}}=O_p(s_n^2)=o_p(1) ∥ D n − D n ∥ op = O p ( s n 2 ) = o p ( 1 ) . This argument needs no comparison between s n s_n s n and N n − 1 / 2 N_n^{-1/2} N n − 1/2 . □
Proof of Proposition B.2. If Σ \Sigma Σ is positive definite, the joint endpoint and normalizer limit is
{ Z n , V ~ n , λ n } ⇝ { − L W d ( 1 ) , L Q d L T , 0 } , L = H − 1 Σ 1 / 2 . \{Z_n,\widetilde{V}_n,\lambda_n\}\rightsquigarrow\{-LW_d(1),LQ_dL^{\mathsf T},0\},\qquad L=H^{-1}\Sigma^{1/2}. { Z n , V n , λ n } ⇝ { − L W d ( 1 ) , L Q d L T , 0 } , L = H − 1 Σ 1/2 .
where Z n = N P a n → x ⋆ ( v ˉ n − v ⋆ n ) Z_n=\sqrt{N}P_{a_n\to x_\star}(\bar{v}_n-v_\star^n) Z n = N P a n → x ⋆ ( v ˉ n − v ⋆ n ) and V ~ n \widetilde{V}_n V n is the transported normalizer in Theorem 3.8 . Its limiting normalizer is positive definite almost surely. Joint Slutsky convergence, which does not require independence of the ridge, gives
{ Z n , V ~ n + λ n I } ⇝ { − L W d ( 1 ) , L Q d L T } . \{Z_n,\widetilde{V}_n+\lambda_n I\}\rightsquigarrow\{-LW_d(1),LQ_dL^{\mathsf T}\}. { Z n , V n + λ n I } ⇝ { − L W d ( 1 ) , L Q d L T } .
Matrix inversion is continuous on the SPD matrix cone, so the ridge does not change the limiting quadratic form. Under the row-rank condition of Corollary 3.9 , its proof gives the joint convergence of the functional numerator, projected normalizer, and ridge:
{ N { h ( x ^ n ) − h ( x ⋆ ) } , V ^ h , n , λ h , n } ⇝ { − A h W d ( 1 ) , A h Q d A h T , 0 } . \{\sqrt{N}\{h(\widehat{x}_n)-h(x_\star)\},\widehat{V}_{h,n},\lambda_{h,n}\}\rightsquigarrow\{-A_hW_d(1),A_hQ_dA_h^{\mathsf T},0\}. { N { h ( x n ) − h ( x ⋆ )} , V h , n , λ h , n } ⇝ { − A h W d ( 1 ) , A h Q d A h T , 0 } .
The limiting projected normalizer is positive definite almost surely, so the same Slutsky and continuous-mapping argument applies to V ^ h , n + λ h , n I q \widehat{V}_{h,n}+\lambda_{h,n}I_q V h , n + λ h , n I q and, by Corollary B.1 , to its derivative-approximated version.
Proof of Proposition B.4
Proof of Proposition B.4. For part (a), let O O O be an orthogonal change of basis in T a n M T_{a_n}\mathcal{M} T a n M . Then every v k , n v_{k,n} v k , n and C m , n C_{m,n} C m , n is replaced by O v k , n Ov_{k,n} O v k , n and O C m , n OC_{m,n} O C m , n , while V ~ n \widetilde{V}_n V n is replaced by O V ~ n O T O\widetilde{V}_nO^{\mathsf T} O V n O T . The quadratic form in (7) is therefore unchanged.
For part (b), transport the alternative coordinates to T x ⋆ M T_{x_\star}\mathcal{M} T x ⋆ M and write
Ξ a ( x ) = P a → x ⋆ χ a ( x ) . \Xi_a(x)=P_{a\to x_\star}\chi_a(x). Ξ a ( x ) = P a → x ⋆ χ a ( x ) .
On the event E n ∩ { a n ∈ A } E_n\cap\{a_n\in\mathcal{A}\} E n ∩ { a n ∈ A } , which has probability tending to one, every retained iterate lies in the common forward-chart domain. A uniform Taylor expansion at x ⋆ x_\star x ⋆ therefore gives
Ξ a ( x ) − Ξ a ( x ⋆ ) = K a Log x ⋆ ( x ) + R a X ( x ) , ∥ R a X ( x ) ∥ ≤ C dist ( x , x ⋆ ) 2 , (66) \Xi_a(x)-\Xi_a(x_\star)=K_a\operatorname{Log}_{x_\star}(x)+R_a^X(x),\qquad\lVert R_a^X(x)\rVert\le C\operatorname{dist}(x,x_\star)^2, \tag*{(66)} Ξ a ( x ) − Ξ a ( x ⋆ ) = K a Log x ⋆ ( x ) + R a X ( x ) , ∥ R a X ( x )∥ ≤ C dist ( x , x ⋆ ) 2 , (66)
where K a = D ( Ξ a ∘ Exp x ⋆ ) 0 K_a=D(\Xi_a\circ\operatorname{Exp}_{x_\star})_0 K a = D ( Ξ a ∘ Exp x ⋆ ) 0 and K a K_a K a and K a − 1 K_a^{-1} K a − 1 are uniformly bounded. Since a n → x ⋆ a_n\to x_\star a n → x ⋆ in probability, K a n → K ⋆ K_{a_n}\to K_\star K a n → K ⋆ for a nonsingular K ⋆ K_\star K ⋆ . The proof of Lemma C.8 applied to (D.5) shows that the alternative-coordinate endpoint and bridge limits are multiplied by K ⋆ K_\star K ⋆ . Thus the limiting numerator and normalizer are − K ⋆ L W d ( 1 ) -K_\star LW_d(1) − K ⋆ L W d ( 1 ) and K ⋆ L Q d L T K ⋆ T K_\star LQ_dL^{\mathsf T}K_\star^{\mathsf T} K ⋆ L Q d L T K ⋆ T . When Σ \Sigma Σ is positive definite, the nonsingular factor K ⋆ L K_\star L K ⋆ L cancels from the full quadratic form. By Lemma C.9 , the alternative inverse chart is evaluated on a common domain with probability tending to one. Writing z ˉ n = N − 1 ∑ k = b + 1 n χ a n ( x k ) \bar{z}_n=N^{-1}\sum_{k=b+1}^{n}\chi_{a_n}(x_k) z ˉ n = N − 1 ∑ k = b + 1 n χ a n ( x k ) , its Taylor expansion gives
N Log x ⋆ { χ a n − 1 ( z ˉ n ) } = K a n − 1 P a n → x ⋆ N { z ˉ n − χ a n ( x ⋆ ) } + o p ( 1 ) , \sqrt{N}\operatorname{Log}_{x_\star}\{\chi_{a_n}^{-1}(\bar{z}_n)\}=K_{a_n}^{-1}P_{a_n\to x_\star}\sqrt{N}\{\bar{z}_n-\chi_{a_n}(x_\star)\}+o_p(1), N Log x ⋆ { χ a n − 1 ( z ˉ n )} = K a n − 1 P a n → x ⋆ N { z ˉ n − χ a n ( x ⋆ )} + o p ( 1 ) ,
and hence the intrinsic limit (24), without requiring Σ \Sigma Σ to be nonsingular.
For a smooth functional, the derivative of h ∘ χ a n − 1 h\circ\chi_{a_n}^{-1} h ∘ χ a n − 1 at z ˉ n \bar{z}_n z ˉ n , after the natural tangent identifications, converges to D h x ⋆ K ⋆ − 1 Dh_{x_\star}K_\star^{-1} D h x ⋆ K ⋆ − 1 . Premultiplying the alternative-coordinate endpoint and bridge limits thus gives
− A h W d ( 1 ) , A h Q d A h T . -A_hW_d(1),\qquad A_hQ_dA_h^{\mathsf T}. − A h W d ( 1 ) , A h Q d A h T .
The row-rank argument in the proof of Corollary 3.9 yields the T q \mathcal{T}_q T q limit without inverting L L L . An operator-norm-consistent derivative approximation perturbs the tight alternative-coordinate normalizer by o p ( 1 ) o_p(1) o p ( 1 ) , exactly as in the proof of Corollary B.1 .
For part (c), in Euclidean space v k , n = x k − a n v_{k,n}=x_k-a_n v k , n = x k − a n and x ˉ n = N n − 1 ∑ k = b n + 1 n x k \bar{x}_n=N_n^{-1}\sum_{k=b_n+1}^{n}x_k x ˉ n = N n − 1 ∑ k = b n + 1 n x k . Moreover,
C m , n = ∑ j = 1 m x b n + j − m N n ∑ j = 1 N n x b n + j , C_{m,n}=\sum_{j=1}^{m}x_{b_n+j}-\frac{m}{N_n}\sum_{j=1}^{N_n}x_{b_n+j}, C m , n = j = 1 ∑ m x b n + j − N n m j = 1 ∑ N n x b n + j ,
so the anchor cancels exactly. Equations (6) and (7) are consequently the Euclidean random scaling normalizer and statistic after relabeling the retained observations.
Proof of Proposition B.5
Proof of Proposition B.5. Because H H H and Σ \Sigma Σ are positive definite, Ω = H − 1 Σ H − 1 \Omega=H^{-1}\Sigma H^{-1} Ω = H − 1 Σ H − 1 is positive definite. Equation (B.1, together with eventual self-adjoint positive definiteness of Ω ^ n \widehat{\Omega}_n Ω n , implies after transport that Ω ^ n − 1 ⟶ p Ω − 1 \widehat{\Omega}_n^{-1}\overset{p}{\longrightarrow}\Omega^{-1} Ω n − 1 ⟶ p Ω − 1 in operator norm. At x = x ⋆ x=x_\star x = x ⋆ , (D.4 and (B.1 imply, by Slutsky’s theorem,
N n ⟨ v ˉ n − v ⋆ , Ω ^ n − 1 ( v ˉ n − v ⋆ ) ⟩ ⇝ χ d 2 . N_n\left\langle\bar{v}_n-v^\star,\widehat{\Omega}_n^{-1}(\bar{v}_n-v^\star)\right\rangle\leadsto\chi_d^2. N n ⟨ v ˉ n − v ⋆ , Ω n − 1 ( v ˉ n − v ⋆ ) ⟩ ⇝ χ d 2 .
The self-normalized convergence is (26).
Ignoring the asymptotically inactive restriction to the common chart, the image of the Wald region under Log a n \operatorname{Log}_{a_n} Log a n is an ellipsoid with center v ˉ n \bar{v}_n v ˉ n , shape matrix Ω ^ n \widehat{\Omega}_n Ω n , and squared radius χ d , 1 − α 2 / N n \chi^2_{d,1-\alpha}/N_n χ d , 1 − α 2 / N n . Hence its tangent space volume equals
ω d ( χ d , 1 − α 2 N n ) d / 2 det ( Ω ^ n ) 1 / 2 . \omega_d\left(\frac{\chi^2_{d,1-\alpha}}{N_n}\right)^{d/2}\det(\widehat{\Omega}_n)^{1/2}. ω d ( N n χ d , 1 − α 2 ) d /2 det ( Ω n ) 1/2 .
Parallel transport preserves determinants in orthonormal frames, and Ω = L L T \Omega=LL^{\mathsf T} Ω = L L T with L = H − 1 Σ 1 / 2 L=H^{-1}\Sigma^{1/2} L = H − 1 Σ 1/2 . This proves (B.3).
The image of the self-normalized region is similarly an ellipsoid with shape matrix V ^ n \widehat{V}_n V n and squared radius c d , 1 − α / N n c_{d,1-\alpha}/N_n c d , 1 − α / N n . By (25) and the continuous mapping theorem,
det ( V ^ n ) 1 / 2 ⇝ det ( L Q d L T ) 1 / 2 = ∣ det L ∣ det ( Q d ) 1 / 2 , \det(\widehat{V}_n)^{1/2}\leadsto\det(LQ_dL^{\mathsf T})^{1/2}=|\det L|\det(Q_d)^{1/2}, det ( V n ) 1/2 ⇝ det ( L Q d L T ) 1/2 = ∣ det L ∣ det ( Q d ) 1/2 ,
which proves (B.4). The two limits contain the same model-dependent determinant but different pivotal factors, so no deterministic volume ordering follows.
Verification for the applications
Proof of Proposition B.6. Differentiation under the expectation gives conditional unbiasedness and the operators in (30). At the minimizer, E { grad ℓ ( x ⋆ ; Z ) } = 0 \mathbb{E}\{\operatorname{grad}\ell(x_\star;Z)\}=0 E { grad ℓ ( x ⋆ ; Z )} = 0 , so the second operator is the covariance of the stochastic gradient noise.
It remains to verify the local second moment condition. Let ϵ ( x ; Z ) = grad ℓ ( x ; Z ) − grad F ( x ) \epsilon(x;Z)=\operatorname{grad}\ell(x;Z)-\operatorname{grad}F(x) ϵ ( x ; Z ) = grad ℓ ( x ; Z ) − grad F ( x ) . By local noise continuity,
E ∥ ϵ ( x ; Z ) ∥ 2 ≤ 2 E ∥ P x → x ⋆ ϵ ( x ; Z ) − ϵ ( x ⋆ ; Z ) ∥ 2 + 2 E ∥ ϵ ( x ⋆ ; Z ) ∥ 2 ≤ C . \begin{aligned}
\mathbb{E}\|\epsilon(x;Z)\|^2\leq2\mathbb{E}\|P_{x\to x_\star}\epsilon(x;Z)-\epsilon(x_\star;Z)\|^2+2\mathbb{E}\|\epsilon(x_\star;Z)\|^2 \\
&\leq C.
\end{aligned} E ∥ ϵ ( x ; Z ) ∥ 2 ≤ 2 E ∥ P x → x ⋆ ϵ ( x ; Z ) − ϵ ( x ⋆ ; Z ) ∥ 2 + 2 E ∥ ϵ ( x ⋆ ; Z ) ∥ 2 ≤ C .
Thus the noise has a locally bounded second moment. The deterministic mean gradient is bounded on that neighborhood, and hence sup x E ∥ grad ℓ ( x ; Z ) ∥ 2 < ∞ \sup_x\mathbb{E}\|\operatorname{grad}\ell(x;Z)\|^2<\infty sup x E ∥ grad ℓ ( x ; Z ) ∥ 2 < ∞ . The i.i.d. sequence ξ k = grad ℓ ( x ⋆ ; Z k ) \xi_k=\operatorname{grad}\ell(x_\star;Z_k) ξ k = grad ℓ ( x ⋆ ; Z k ) satisfies the functional central limit theorem and Lindeberg condition under its finite second moment. The remaining assumptions are imposed in the proposition. Therefore Theorems 3.7 and 3.8 and Corollary 3.9 give (B.5) and the confidence procedures in Proposition B.6 .
Proof of Proposition B.7. On a compact set separated from the cut locus, the squared-distance function is smooth jointly in ( x , z ) (x,z) ( x , z ) and its derivatives are uniformly controlled by the envelope in Proposition B.7(a). The gradient identity (31) follows from the first variation formula. Thus the stochastic gradient at μ \mu μ is − Log μ ( Z ) -\operatorname{Log}_{\mu}(Z) − Log μ ( Z ) , its mean is zero, and its covariance is Σ μ \Sigma_{\mu} Σ μ . Differentiating the mean gradient gives H μ H_{\mu} H μ . The envelope assumptions verify the Hölder Hessian and local L 2 L^2 L 2 noise-continuity conditions.
On a Hadamard manifold, the logarithm is globally single valued and the Fréchet mean is unique under the finite squared-distance moment in Proposition B.7(b). For bounded support on the affine invariant SPD manifold, compactness of closed geodesic balls and smoothness of squared distance bound its derivatives uniformly on a compact neighborhood of μ \mu μ and the support. This implies the local Hessian and noise-continuity bounds. For unbounded support, the local mean-field and noise-continuity conditions remain assumptions in part (b). Once localization and the mean square rate are available, Theorem 3.8 gives (B.6) and the confidence region.
Proof of Proposition B.8. The Grassmann tangent space at [ U ] [U] [ U ] can be represented by matrices Δ \Delta Δ satisfying U T Δ = 0 U^{\mathsf T}\Delta=0 U T Δ = 0 . Projecting the Euclidean gradient − X X T U -XX^{\mathsf T}U − X X T U onto this tangent space gives (35).
Use the eigenbasis [ U ⋆ , U ⊥ ] [U_{\star},U_{\perp}] [ U ⋆ , U ⊥ ] and a tangent matrix K ∈ R ( p − r ) × r K\in\mathbb{R}^{(p-r)\times r} K ∈ R ( p − r ) × r . A geodesic or any second order local parameterization has expansion
U ( K ) = U ⋆ + U ⊥ K − 1 2 U ⋆ K T K + O ( ∥ K ∥ 3 ) . U(K)=U_{\star}+U_{\perp}K-\frac{1}{2}U_{\star}K^{\mathsf T}K+O(\lVert K\rVert^3). U ( K ) = U ⋆ + U ⊥ K − 2 1 U ⋆ K T K + O (∥ K ∥ 3 ) .
Substitution into (34) gives
F { [ U ( K ) ] } − F ( [ U ⋆ ] ) = 1 2 ∑ i = 1 p − r ∑ j = 1 r ( λ j − λ r + i ) K i j 2 + O ( ∥ K ∥ 3 ) . F\{[U(K)]\}-F([U_{\star}])=\frac{1}{2}\sum_{i=1}^{p-r}\sum_{j=1}^{r}(\lambda_j-\lambda_{r+i})K_{ij}^{2}+O(\lVert K\rVert^3). F {[ U ( K )]} − F ([ U ⋆ ]) = 2 1 i = 1 ∑ p − r j = 1 ∑ r ( λ j − λ r + i ) K ij 2 + O (∥ K ∥ 3 ) .
This proves (36); the eigengap makes the Hessian positive definite.
At U ⋆ U_{\star} U ⋆ ,
G ( [ U ⋆ ] , X ) = − U ⊥ X ⊥ X ∥ T . G([U_{\star}],X)=-U_{\perp}X_{\perp}X_{\parallel}^{\mathsf T}. G ([ U ⋆ ] , X ) = − U ⊥ X ⊥ X ∥ T .
Its expectation is zero because U ⊥ T C U ⋆ = 0 U_{\perp}^{\mathsf T}CU_{\star}=0 U ⊥ T C U ⋆ = 0 , and projecting by U ⊥ T U_{\perp}^{\mathsf T} U ⊥ T before vectorization gives (37). Moreover, ∥ G ( [ U ] , X ) ∥ ≤ ∥ X ∥ 2 \lVert G([U],X)\rVert\leq\lVert X\rVert^2 ∥ G ([ U ] , X )∥ ≤ ∥ X ∥ 2 , so E ∥ X ∥ 4 < ∞ \mathbb{E}\lVert X\rVert^4<\infty E ∥ X ∥ 4 < ∞ gives the gradient second moment. Smoothness of the Grassmann projection implies
∥ G ( [ U ] , X ) − P [ U ⋆ ] → [ U ] G ( [ U ⋆ ] , X ) ∥ ≤ C ∥ X ∥ 2 dist ( [ U ] , [ U ⋆ ] ) \lVert G([U],X)-P_{[U_{\star}]\to[U]}G([U_{\star}],X)\rVert\leq C\lVert X\rVert^2\operatorname{dist}([U],[U_{\star}]) ∥ G ([ U ] , X ) − P [ U ⋆ ] → [ U ] G ([ U ⋆ ] , X )∥ ≤ C ∥ X ∥ 2 dist ([ U ] , [ U ⋆ ])
locally. Squaring and taking expectations verifies (13) with ν = 1 \nu=1 ν = 1 . The deterministic objective is smooth on the compact Grassmann manifold, so its Hessian is locally Lipschitz. The general theorem now gives (B.7) and the self-normalized region.