AI Disclosure Statement. The authors utilized the colosseum harness [22 ] on top of Gemini 4 throughout the mathematical development of this work. Specifically, the vast majority of the technical proofs and mathematical derivations in this manuscript were generated and drafted by the model. The human authors independently established several key structural lemmas, improved the exposition, and carefully verified all mathematical steps. Furthermore, this work is accompanied by a fully functional, machine-checked formal proof in Lean 4. The human authors audited the formalization to confirm that all definitions and theorem statements faithfully and accurately encode the intended mathematical results. The machine-checked formal proof in Lean 4 is publicly available at GitHub repository. The authors take full responsibility for the accuracy, rigor, and conclusions of this work.
Introduction
Sketching is the workhorse of randomized linear algebra: we replace a large matrix by a small random image of it and do the expensive work on the image [32 , 18 , 40 , 26 ] . To sketch a matrix A ∈ R n × d A \in\mathbb{R}^{n \times d} A ∈ R n × d , we draw a tall random matrix Ω ∈ R n × k \Omega\in\mathbb{R}^{n \times k} Ω ∈ R n × k and form the product Ω T A \Omega^{\mathsf{T}} A Ω T A . To measure the cost of a sketching method, we need to consider two aspects. First is the embedding dimension k k k , which controls the size of (and therefore cost of working on) the image; when the problem is governed by an r r r -dimensional subspace, the natural goal is k = O ( r ) k = O(r) k = O ( r ) . Second, we need to account for the cost of computing Ω T A \Omega^{\mathsf{T}} A Ω T A . Indeed, if Ω \Omega Ω is a dense Gaussian matrix, then computing Ω T A \Omega^{\mathsf{T}} A Ω T A can easily become the computation bottleneck.
To resolve this, a very popular approach is to use sparse random matrices [11 , 25 , 21 MDM+23, 7 ] . If Ω \Omega Ω has at most ζ \zeta ζ nonzeros per row, then we can form the sketch Ω T A \Omega^{\mathsf{T}} A Ω T A in O ( ζ nnz ( A ) ) O(\zeta\operatorname{nnz}(A)) O ( ζ nnz ( A )) time. We call ζ \zeta ζ the row sparsity of Ω \Omega Ω . This paper studies a single question: how sparse can the test matrix be when k = O ( r ) k = O(r) k = O ( r ) while still being reliable for numerical algorithms?
Embeddings and Injections
Classically, randomized numerical linear algebra (RNLA) methods have used the Oblivious Subspace Embedding (OSE) condition to characterize when a sketching matrix is reliable [32 , 40 , 34 , 26 , 6 ] . We say that Ω ∈ R n × k \Omega\in\mathbb{R}^{n \times k} Ω ∈ R n × k is an OSE if, for every matrix Q ∈ R n × r Q \in\mathbb{R}^{n \times r} Q ∈ R n × r with orthonormal columns, we have
( 1 − ε ) I r ⪯ Q T Ω Ω T Q ⪯ ( 1 + ε ) I r (1-\varepsilon)I_r \preceq Q^{\mathsf{T}}\Omega\Omega^{\mathsf{T}}Q \preceq(1+\varepsilon)I_r ( 1 − ε ) I r ⪯ Q T Ω Ω T Q ⪯ ( 1 + ε ) I r
with high probability. Equivalently, the OSE guarantee ensures that vectors in the range of Q Q Q are neither stretched nor squished by large amount. A long line of work has shown that sparse sketching matrices with embedding dimension k = O ( r ) k = O(r) k = O ( r ) and constant dilation ε = 1 / 2 \varepsilon= 1/2 ε = 1/2 must have row sparsity ζ = Ω ( log r ) \zeta= \Omega(\log r) ζ = Ω ( log r ) , and that this rate is achieved by the SparseStack matrix (see below) [8 , 28 , 5 , 3 , 4 , 17 ] .
While the OSE definition is natural, and is guaranteed to hold for many sketching matrices, it is often also overkill. Several works in RNLA show that many downstream algorithms, like methods for low-rank approximation [18 , 40 , 27 , 26 , 6 ] and least-squares regression [30 , 13 , 6 ] —do not actually require a two-sided guarantee. Instead, it suffices for the sketch to satisfy a one-sided lower bound:
Q T Ω Ω T Q ⪰ α I r , or equivalently ∥ Ω T Q x ∥ 2 ≥ α ∥ x ∥ 2 for all x ∈ R r , (1) Q^{\mathsf{T}}\Omega\Omega^{\mathsf{T}}Q \succeq\alpha I_r,
\qquad\text{or equivalently} \qquad\lVert\Omega^{\mathsf{T}}Qx\rVert^2 \geq\alpha\lVert x\rVert^2 \quad\text{for all } x \in\mathbb{R}^r, \tag*{(1)} Q T Ω Ω T Q ⪰ α I r , or equivalently ∥ Ω T Q x ∥ 2 ≥ α ∥ x ∥ 2 for all x ∈ R r , (1)
where α > 0 \alpha> 0 α > 0 is called the injectivity parameter , all subject the normalization criteria E [ Ω Ω T ] = I \mathbb{E}[\Omega\Omega^{\mathsf{T}}] = I E [ Ω Ω T ] = I . While this weaker criteria was known or folklore in specific contexts, its systematic study was initiated by Camaño, Epperly, Meyer, and Tropp [6 ] , who introduced the term Oblivious Subspace Injection (OSI) . It will be helpful to think of the OSI condition in terms of spectra. Indeed, equation (1) holds if and only if
σ min ( Ω T Q ) 2 ≥ α \sigma_{\min}(\Omega^{\mathsf{T}}Q)^2 \geq\alpha σ m i n ( Ω T Q ) 2 ≥ α
with high constant probability.
The central question we want to understand is, at linear sketch dimension k = Θ ( r ) k = \Theta(r) k = Θ ( r ) , how small can we make the row sparsity ζ \zeta ζ while maintaining constant injectivity α \alpha α . Most specifically, can a sparse matrix be an OSI with sparsity ζ = o ( log r ) \zeta= o(\log r) ζ = o ( log r ) , proving that the OSEs formalism is unable to fully justify when sparse matrices are reliable numerical tools? We answer this in the affirmative.
The SparseStack sketch. To minimize both sketching time and storage, a popular family of structured random matrices in RNLA is CountSketch [11 , 2 , 37 ] and its multi-block generalization, SparseStack [38 , 28 , 25 , 29 , 8 , 6 ] . Throughout this paper, a SparseStack sketch matrix Ω ζ , b ∈ R n × k \Omega_{\zeta,b} \in\mathbb{R}^{n \times k} Ω ζ , b ∈ R n × k has n ≥ r n \ge r n ≥ r rows, row sparsity ζ \zeta ζ , and total embedding dimension k = ζ b = Θ ( r ) ≪ n k = \zeta b = \Theta(r) \ll n k = ζ b = Θ ( r ) ≪ n .
We define these matrices directly in terms of independent Rademacher signs and sparse coordinate selections. Let e 1 , … , e b ∈ R b e_1,\ldots,e_b \in\mathbb{R}^{b} e 1 , … , e b ∈ R b denote the standard basis vectors. A CountSketch matrix Φ ∈ R n × b \Phi\in\mathbb{R}^{n \times b} Φ ∈ R n × b has rows given by
Φ = [ ϱ 1 e s 1 T ⋮ ϱ n e s n T ] ∈ R n × b , where ϱ i ∼ Rademacher i . i . d . ; s i ∼ Uniform { 1 , … , b } i . i . d . (2) \Phi=
\begin{bmatrix}
\varrho_1 e_{s_1}^{\mathsf{T}}\\
\vdots\\
\varrho_n e_{s_n}^{\mathsf{T}}
\end{bmatrix}
\in\mathbb{R}^{n \times b},
\qquad
\text{where}\qquad
\varrho_i \sim\operatorname{Rademacher}\ \mathrm{i.i.d.};
\qquad
s_i \sim\operatorname{Uniform}\{1,\ldots,b\}\ \mathrm{i.i.d.} \tag*{(2)} Φ = ϱ 1 e s 1 T ⋮ ϱ n e s n T ∈ R n × b , where ϱ i ∼ Rademacher i.i.d. ; s i ∼ Uniform { 1 , … , b } i.i.d. (2)
That is, each row of Φ \Phi Φ contains a single non-zero entry ϱ i ∈ { − 1 , 1 } \varrho_i \in\{-1,1\} ϱ i ∈ { − 1 , 1 } placed at an independent, uniformly chosen column index s i ∈ [ b ] s_i \in[b] s i ∈ [ b ] .
A SparseStack matrix Ω ζ , b ∈ R n × k \Omega_{\zeta,b} \in\mathbb{R}^{n \times k} Ω ζ , b ∈ R n × k with row sparsity ζ \zeta ζ , block size b b b , and dimension k = ζ b k = \zeta b k = ζ b horizontally concatenates ζ \zeta ζ independent CountSketch matrices Φ 1 , … , Φ ζ ∼ Φ \Phi_1,\ldots,\Phi_\zeta\sim\Phi Φ 1 , … , Φ ζ ∼ Φ , scaled by ζ − 1 / 2 \zeta^{-1/2} ζ − 1/2 .
Ω ζ , b : = 1 ζ [ Φ 1 ⋯ Φ ζ ] = 1 ζ [ ϱ 1 , 1 e s 1 , 1 T ⋯ ϱ 1 , ζ e s 1 , ζ T ϱ 2 , 1 e s 2 , 1 T ⋯ ϱ 2 , ζ e s 2 , ζ T ⋮ ⋱ ⋮ ϱ n , 1 e s n , 1 T ⋯ ϱ n , ζ e s n , ζ T ] ∈ R n × k , (3) \Omega_{\zeta,b} := \frac{1}{\sqrt{\zeta}}\left[\Phi_1\ \cdots\ \Phi_\zeta\right]
= \frac{1}{\sqrt{\zeta}}
\begin{bmatrix}
\varrho_{1,1}e_{s_{1,1}}^{\mathsf{T}} & \cdots& \varrho_{1,\zeta}e_{s_{1,\zeta}}^{\mathsf{T}}\\
\varrho_{2,1}e_{s_{2,1}}^{\mathsf{T}} & \cdots& \varrho_{2,\zeta}e_{s_{2,\zeta}}^{\mathsf{T}}\\
\vdots& \ddots& \vdots\\
\varrho_{n,1}e_{s_{n,1}}^{\mathsf{T}} & \cdots& \varrho_{n,\zeta}e_{s_{n,\zeta}}^{\mathsf{T}}
\end{bmatrix}
\in\mathbb{R}^{n \times k}, \tag*{(3)} Ω ζ , b := ζ 1 [ Φ 1 ⋯ Φ ζ ] = ζ 1 ϱ 1 , 1 e s 1 , 1 T ϱ 2 , 1 e s 2 , 1 T ⋮ ϱ n , 1 e s n , 1 T ⋯ ⋯ ⋱ ⋯ ϱ 1 , ζ e s 1 , ζ T ϱ 2 , ζ e s 2 , ζ T ⋮ ϱ n , ζ e s n , ζ T ∈ R n × k , (3)
where { ϱ i t } \{\varrho_{it}\} { ϱ i t } are i.i.d. Rademacher random variables and { s i t } \{s_{it}\} { s i t } are i.i.d. uniform random indices on [ b ] [b] [ b ] . By construction, every row of Ω ζ , b \Omega_{\zeta,b} Ω ζ , b has exactly ζ \zeta ζ non-zero entries of magnitude ζ − 1 / 2 \zeta^{-1/2} ζ − 1/2 and unit Euclidean norm. Since distinct rows have mean-zero inner products, Ω ζ , b \Omega_{\zeta,b} Ω ζ , b is isotropic in expectation.
Proof methods for OSEs and the Θ ( log r ) \Theta(\log r) Θ ( log r ) barrier. In the two-sided ( 1 ± ε ) (1 \pm\varepsilon) ( 1 ± ε ) -OSE setting, SparseStack is the block-independent model within the OSNAP framework of Nelson and Nguyen [28 ] and Cohen [8 ] . Recent work by Heidary [17 ] established that dimension k = Θ ( r / ε 2 ) k = \Theta(r/\varepsilon^2) k = Θ ( r / ε 2 ) and row sparsity ζ = Θ ( ( log r ) / ε ) \zeta= \Theta((\log r)/\varepsilon) ζ = Θ (( log r ) / ε ) are both sufficient and optimal for a two-sided ( 1 ± ε ) (1 \pm\varepsilon) ( 1 ± ε ) -embedding.
Existing methods establish injectivity by viewing the SparseStack as an average of independent CountSketch blocks, giving
A T A = Q T Ω ζ , b Ω ζ , b T Q = 1 ζ ∑ t = 1 ζ Q T Φ t Φ t T Q . (4) A^{\mathsf{T}}A = Q^{\mathsf{T}}\Omega_{\zeta,b}\Omega_{\zeta,b}^{\mathsf{T}}Q = \frac{1}{\zeta}\sum_{t=1}^{\zeta} Q^{\mathsf{T}}\Phi_t\Phi_t^{\mathsf{T}}Q. \tag*{(4)} A T A = Q T Ω ζ , b Ω ζ , b T Q = ζ 1 t = 1 ∑ ζ Q T Φ t Φ t T Q . (4)
In particular, using matrix comparison theorems, Tropp [35 ] proved that row sparsity ζ = Θ ( log r ) \zeta= \Theta(\log r) ζ = Θ ( log r ) ensures σ min ( A ) 2 ≥ Ω ( 1 ) \sigma_{\min}(A)^2 \ge\Omega(1) σ m i n ( A ) 2 ≥ Ω ( 1 ) at linear sketch dimension k = Θ ( r ) k = \Theta(r) k = Θ ( r ) . The question of whether constant injectivity is achievable with row sparsity strictly below Θ ( log r ) \Theta(\log r) Θ ( log r ) was posed by Camaño et al. [6 ] and has remained open.
Techniques that analyze A T A A^{\mathsf{T}}A A T A directly as a sum of independent blocks hit a conceptual barrier at row sparsity Θ ( log r ) \Theta(\log r) Θ ( log r ) . Below this threshold, the largest singular value σ max ( A ) \sigma_{\max}(A) σ m a x ( A ) is no longer bounded by O ( 1 ) O(1) O ( 1 ) with high probability. It is easy to see this via a coupon-collector argument: hashing $r basis coordinates into b = Θ ( r / ζ ) b = \Theta(r/\zeta) b = Θ ( r / ζ ) buckets in a block inevitably creates overloaded buckets where Θ ( log ( r / ζ ) / log log ( r / ζ 2 ) ) = ω ( ζ ) \Theta(\log(r/\zeta)/\log\log(r/\zeta^{2})) = \omega(\zeta) Θ ( log ( r / ζ ) / log log ( r / ζ 2 )) = ω ( ζ ) coordinates collide, and a unit vector aligned with those colliding coordinates has its entries add up to give ∥ Ω ζ , b Q x ∥ = ω ( 1 ) \lVert\Omega_{\zeta,b}Qx\rVert= \omega(1) ∥ Ω ζ , b Q x ∥ = ω ( 1 ) .
Nonetheless, it has been experimentally shown that SparseStacks, as well as other sparse sketching matrices, are OSIs with favorable values of α \alpha α even when ζ = 4 \zeta= 4 ζ = 4 and k = 2 r k = 2r k = 2 r [MDM+23, CEMT25]. Motivated by these questions, Camaño et al. [6 ] conjectured that constant row sparsity ζ = O ( 1 ) \zeta= O(1) ζ = O ( 1 ) might suffice for an OSI at linear dimension k = Θ ( r ) k = \Theta(r) k = Θ ( r ) . However, Huang, Rudelson, and Tikhomirov [19 ] recently refuted this conjecture, showing that constant row sparsity fails for any OSI distribution. This leaves open the central question: What is the true optimal sparsity for an Oblivious Subspace Injection? Does injectivity inherently require Θ ( log r ) \Theta(\log r) Θ ( log r ) sparsity as in the two-sided OSE case, or does the relaxation to one-sided injectivity permit a strictly sparser sketch?
Main Results
In this paper, we resolve this question by establishing matching (up to log log r \log\log r log log r factors) upper and lower bounds on the row sparsity of OSI at linear embedding dimension, showing that the optimal threshold is Θ ~ ( log r ) \widetilde{\Theta}(\sqrt{\log r}) Θ ( log r ) .
Our first main result is an upper bound proving that SparseStack achieves constant injectivity at linear embedding dimension with row sparsity only O ( log r ) O(\sqrt{\log r}) O ( log r ) , well below the Θ ( log r ) \Theta(\log r) Θ ( log r ) barrier of two-sided embeddings.
Theorem 1.1. There exist universal constants C , K > 0 C, K > 0 C , K > 0 and α ∈ ( 0 , 1 ) \alpha\in(0,1) α ∈ ( 0 , 1 ) such that the following holds. For every n ≥ r ≥ 2 n \ge r \ge2 n ≥ r ≥ 2 , every fixed isometry Q ∈ R n × r Q \in\mathbb{R}^{n \times r} Q ∈ R n × r satisfying Q T Q = I r Q^{\mathsf{T}}Q = I_r Q T Q = I r , and integer sketch parameters satisfying
ζ = O ( log r ) with ζ ≥ C log ( e r ) , b = Θ ( r ζ ) , k = ζ b ≥ K r , \zeta= O(\sqrt{\log r}) \text{ with } \zeta\ge C\sqrt{\log(er)}, \qquad b = \Theta\left(\frac{r}{\zeta}\right), \qquad k = \zeta b \ge Kr, ζ = O ( log r ) with ζ ≥ C log ( er ) , b = Θ ( ζ r ) , k = ζ b ≥ K r , the sketched matrix A = Ω ζ , b T Q ∈ R k × r A = \Omega_{\zeta,b}^{\mathsf{T}}Q \in\mathbb{R}^{k \times r} A = Ω ζ , b T Q ∈ R k × r formed from the ζ \zeta ζ -block SparseStack sketch Ω ζ , b ∈ R n × k \Omega_{\zeta,b} \in\mathbb{R}^{n \times k} Ω ζ , b ∈ R n × k satisfies
P [ σ min ( A ) 2 ≥ α ] ≥ 19 20 . \mathbb{P}\left[\sigma_{\min}(A)^2 \ge\alpha\right] \ge\frac{19}{20}. P [ σ m i n ( A ) 2 ≥ α ] ≥ 20 19 . Moreover, E [ Ω ζ , b Ω ζ , b T ] = I n \mathbb{E}\left[\Omega_{\zeta,b}\Omega_{\zeta,b}^{\mathsf{T}}\right] = I_n E [ Ω ζ , b Ω ζ , b T ] = I n and the total sketch dimension is linear in r r r , namely k = ζ b = Θ ( r ) k = \zeta b = \Theta(r) k = ζ b = Θ ( r ) .
Our second main result is a general sparsity lower bound showing that this log r \sqrt{\log r} log r scale is essentially unavoidable. Recently, Huang, Rudelson, and Tikhomirov [19 ] , denoted by HRT, proved that constant row sparsity ζ = O ( 1 ) \zeta= O(1) ζ = O ( 1 ) cannot achieve an OSI by introducing three structural assumptions on the sketching matrix: uniform entry-magnitude bounds of order O ( ζ − 1 / 2 ) O(\zeta^{-1/2}) O ( ζ − 1/2 ) on all non-zero entries, deterministic row-norm and support-size regularity, and a bound on pairwise row-support overlaps. In this work, we eliminate the first two assumptions and retain only the overlap condition to establish the Ω ~ ( log r ) \widetilde{\Omega}(\sqrt{\log r}) Ω ( log r ) sparsity lower bound:
Definition 1.2 (Overlap condition). A matrix S ∈ R n × k S \in\mathbb{R}^{n \times k} S ∈ R n × k with rows s 1 , … , s n s_1,\ldots,s_n s 1 , … , s n satisfies the overlap condition if the number of pairs of distinct rows { u , v } ⊂ [ n ] \{u,v\} \subset[n] { u , v } ⊂ [ n ] sharing two or more non-zero coordinates satisfies ∑ 1 ≤ u < v ≤ n 1 { ∣ supp ( s u ) ∩ supp ( s v ) ∣ ≥ 2 } ≤ η n 2 k \sum_{1 \le u < v \le n} \mathbf{1}\{\lvert\operatorname{supp}(s_u)\cap\operatorname{supp}(s_v)\rvert\ge2\} \le\eta\frac{n^2}{k} ∑ 1 ≤ u < v ≤ n 1 {∣ supp ( s u ) ∩ supp ( s v )∣ ≥ 2 } ≤ η k n 2 for an overlap parameter η = o ( 1 ) \eta= o(1) η = o ( 1 ) . Specifically, we need η = O ( ( log log r / log r ) 3 ) \eta= O((\log\log r/\log r)^3) η = O (( log log r / log r ) 3 ) .
Here the o ( 1 ) o(1) o ( 1 ) bound on η \eta η is a function of the subspace dimension r r r : as we will show in the proof by requiring η ≤ ( C H ζ 6 ) − 1 \eta\le(C_H\zeta^6)^{-1} η ≤ ( C H ζ 6 ) − 1 for a constant C H > 0 C_H > 0 C H > 0 , under the target sparsity scale ζ = Θ ( log r / log log r ) \zeta= \Theta(\sqrt{\log r/\log\log r}) ζ = Θ ( log r / log log r ) our analysis ultimately sets η \eta η to be on the order of Θ ( ( log log r / log r ) 3 ) \Theta((\log\log r/\log r)^3) Θ (( log log r / log r ) 3 ) . Because this requirement is only a logarithmically decaying function of r r r , it is far more permissive than the polynomial decay exhibited by standard sparse sketching matrices. Specifically, in ensembles such as SparseStack and OSNAP [28 , 8 ] at linear dimension k = Θ ( r ) k = \Theta(r) k = Θ ( r ) , any two rows share a given column with probability ζ / k \zeta/k ζ / k and thus share two or more columns with probability at most ( ζ 2 ) ( ζ / k ) 2 = O ( ζ 4 / k 2 ) \binom{\zeta}{2}(\zeta/k)^2 = O(\zeta^4/k^2) ( 2 ζ ) ( ζ / k ) 2 = O ( ζ 4 / k 2 ) ; summing over all ( n 2 ) \binom{n}{2} ( 2 n ) row pairs yields an actual overlap parameter of order η = O ( ζ 4 / k ) = O ~ ( 1 / r ) ≪ ( log log r / log r ) 3 \eta= O(\zeta^4/k) = \widetilde{O}(1/r) \ll(\log\log r/\log r)^3 η = O ( ζ 4 / k ) = O ( 1/ r ) ≪ ( log log r / log r ) 3 , showing a polynomial-versus-polylogarithmic gap and confirming that the overlap condition encompasses all standard sparse sketching constructions.
Theorem 1.3. For sufficiently large r r r , let n ≥ Ω ( r log 3 r ) n \ge\Omega(r\log^3 r) n ≥ Ω ( r log 3 r ) , k ≤ K r k \le Kr k ≤ K r for a constant K ≥ 1 K \ge1 K ≥ 1 , and α ∈ ( 0 , 1 ) \alpha\in(0,1) α ∈ ( 0 , 1 ) . If an isotropic random sketching matrix Ω ∈ R n × k \Omega\in\mathbb{R}^{n \times k} Ω ∈ R n × k with row sparsity ζ \zeta ζ satisfies the overlap condition with the overlap parameter η = O ( ( log log r / log r ) 3 ) \eta= O((\log\log r/\log r)^3) η = O (( log log r / log r ) 3 ) with high probability, and is an ( r , α ) (r,\alpha) ( r , α ) -OSI, then the row sparsity must satisfy
ζ = Ω ( log r log log r ) \zeta= \Omega\left(\sqrt{\frac{\log r}{\log\log r}}\right) ζ = Ω ( log log r log r )
Separation of OSIs from OSEs. Together, Theorems 1.1 and 1.3 separate OSI from OSE. Nelson and Nguy~en [28 ] proved that for any n ≥ 100 r 2 n \ge100r^2 n ≥ 100 r 2 , any ( r , 1 ± ε ) (r,1 \pm\varepsilon) ( r , 1 ± ε ) -OSE at linear dimension k = O ( r ) k = O(r) k = O ( r ) requires row sparsity Ω ( log r / log log r ) \Omega(\log r/\log\log r) Ω ( log r / log log r ) . Theorem 1.1 demonstrates that SparseStack is an ( r , α ) (r,\alpha) ( r , α ) -OSI with row sparsity O ( log r ) O(\sqrt{\log r}) O ( log r ) , showing that one-sided injectivity requires strictly less sparsity than two-sided preservation:
ζ O S I = Θ ( log r ) ≪ Θ ( log r ) = ζ O S E . \zeta_{\mathrm{OSI}} = \Theta(\sqrt{\log r}) \ll\Theta(\log r) = \zeta_{\mathrm{OSE}}. ζ OSI = Θ ( log r ) ≪ Θ ( log r ) = ζ OSE .
Algorithmic consequences. Injectivity alone supports several algorithmic primitives in randomized numerical linear algebra. Following the reductions of Camaño et al. [6 ] :
Sketch-and-solve least-squares regression: Given an overdetermined system min x ∈ R r ∥ B x − u ∥ 2 \min_{x\in\mathbb{R}^r}\|Bx-u\|_2 min x ∈ R r ∥ B x − u ∥ 2 with B ∈ R n × r B \in\mathbb{R}^{n\times r} B ∈ R n × r of full column rank and u ∈ R n u \in\mathbb{R}^n u ∈ R n , an ( r , α ) (r,\alpha) ( r , α ) -OSI Ω ∈ R n × k \Omega\in\mathbb{R}^{n\times k} Ω ∈ R n × k produces an estimator x ^ ∈ argmin x ∥ Ω ⊤ ( B x − u ) ∥ 2 \hat{x} \in\operatorname*{argmin}_{x}\|\Omega^\top(Bx-u)\|_2 x ^ ∈ argmin x ∥ Ω ⊤ ( B x − u ) ∥ 2 that satisfies
∥ B x ^ − u ∥ 2 2 ≤ C α min x ∈ R r ∥ B x − u ∥ 2 2 \|B\hat{x}-u\|_2^2 \le\frac{C}{\alpha}\min_{x\in\mathbb{R}^r}\|Bx-u\|_2^2 ∥ B x ^ − u ∥ 2 2 ≤ α C x ∈ R r min ∥ B x − u ∥ 2 2
with probability at least 9 / 10 9/10 9/10 . By Theorem 1.1, computing the sketch Ω ⊤ B \Omega^\top B Ω ⊤ B requires only O ( nnz ( B ) log r ) O(\operatorname{nnz}(B)\sqrt{\log r}) O ( nnz ( B ) log r ) operations, improving upon the Ω ( nnz ( B ) log r ) \Omega(\operatorname{nnz}(B)\log r) Ω ( nnz ( B ) log r ) cost required by two-sided OSEs.
Low-rank approximation and randomized SVD: For a matrix M M M , computing randomized SVD and Nyström approximations [27 ] via an ( r , α ) (r,\alpha) ( r , α ) -OSI achieves constant-factor accuracy with probability at least 4 / 5 4/5 4/5 [6 Theorem 1.3] . Theorem 1.1 reduces the sketching cost to O ( nnz ( M ) log r ) O(\operatorname{nnz}(M)\sqrt{\log r}) O ( nnz ( M ) log r ) .
Union Bounds and Flat and Spiky Vectors.
Union Bound Approaches An alternative approach matrix concentration theorems is to establish point-wise lower bounds on ∥ A x ∥ \|Ax\| ∥ A x ∥ for each fixed x ∈ S r − 1 x \in S^{r-1} x ∈ S r − 1 , and then obtain the desired bound on σ min ( A ) \sigma_{\min}(A) σ m i n ( A ) through a union bound over S r − 1 S^{r-1} S r − 1 . Naive net ε \varepsilon ε -net arguments obviously fail here. To see this concretely, consider a 2-sparse unit vector such as Q x = ( e 1 + e 2 ) / 2 Qx=(e_1+e_2)/\sqrt{2} Q x = ( e 1 + e 2 ) / 2 . In any given CountSketch block t t t , its image Φ t T Q x = 1 2 ( ϱ 1 t e s 1 t + ϱ 2 t e s 2 t ) \Phi_t^{\mathsf{T}}Qx=\frac{1}{\sqrt{2}}(\varrho_{1t}e_{s_{1t}}+\varrho_{2t}e_{s_{2t}}) Φ t T Q x = 2 1 ( ϱ 1 t e s 1 t + ϱ 2 t e s 2 t ) is identically zero precisely when coordinates 1 and 2 select the same bucket (s 1 t = s 2 t s_{1t}=s_{2t} s 1 t = s 2 t , which occurs with probability 1 / b 1/b 1/ b ) and receive opposite signs (ϱ 1 t = − ϱ 2 t \varrho_{1t}=-\varrho_{2t} ϱ 1 t = − ϱ 2 t , which occurs with probability 1 / 2 1/2 1/2 ). This collision cancellation annihilates the vector in block t t t with probability 1 / ( 2 b ) 1/(2b) 1/ ( 2 b ) . Since the ζ \zeta ζ blocks are independent, the vector is completely canceled across the entire sketch with probability
P [ Ω ζ , b T Q x = 0 ] = ( 2 b ) − ζ = e − ζ log ( 2 b ) . (5) \mathbb{P}\left[\Omega_{\zeta,b}^{\mathsf{T}}Qx=0\right]=(2b)^{-\zeta}=e^{-\zeta\log(2b)}. \tag*{(5)} P [ Ω ζ , b T Q x = 0 ] = ( 2 b ) − ζ = e − ζ l o g ( 2 b ) . (5)
Since a naive ε \varepsilon ε -net over the unit sphere S r − 1 S^{r-1} S r − 1 contains e Θ ( r ) e^{\Theta(r)} e Θ ( r ) vectors, which would demand ζ log ( 2 b ) = Ω ( r ) \zeta\log(2b)=\Omega(r) ζ log ( 2 b ) = Ω ( r ) and force an impractical near-linear sparsity ζ = Ω ~ ( r ) \zeta=\widetilde{\Omega}(r) ζ = Ω ( r ) . Furthermore, even for vectors whose coordinates are spread out so that pointwise concentration holds with failure probability e − Ω ( r ) e^{-\Omega(r)} e − Ω ( r ) , extending a pointwise bound over an ε \varepsilon ε -net requires controlling the discretization error via the operator norm:
∥ Ω ζ , b T Q ( x − x ′ ) ∥ ≤ ∥ Ω ζ , b T Q ∥ ⋅ ∥ x − x ′ ∥ . (6) \left\|\Omega_{\zeta,b}^{\mathsf{T}}Q(x-x')\right\|\leq\left\|\Omega_{\zeta,b}^{\mathsf{T}}Q\right\|\cdot\left\|x-x'\right\|. \tag*{(6)} Ω ζ , b T Q ( x − x ′ ) ≤ Ω ζ , b T Q ⋅ ∥ x − x ′ ∥ . (6)
This step crucially relies on an O ( 1 ) O(1) O ( 1 ) upper bound on ∥ Ω ζ , b T Q ∥ \|\Omega_{\zeta,b}^{\mathsf{T}}Q\| ∥ Ω ζ , b T Q ∥ , which, as discussed above, due to the coupon-collector analysis, ceases to hold when ζ ≪ log r \zeta\ll\log r ζ ≪ log r .
Combinatorial Random Matrix Approaches The combinatorial random matrix literature has developed a more intricate toolbox for taking union bounds over S r − 1 S^{r-1} S r − 1 . The core of this approach is distinguishing between compressible vectors , which have mass concentrated on a small set of entries, and incompressible vectors , which have mass spread more evenly among the components. The key observation is that there are fewer compressible vectors: for example, in the extreme compressible case of 1-sparse vectors on S r − 1 S^{r-1} S r − 1 , there are only r r r such vectors. Thus even if the corresponding “bad” events have larger probabilities for the compressible vectors (eg. that event that x x x gets compressed by a random matrix), the union bound over such compressible x x x is smaller. For incompressible vectors, typically the “bad” events have much smaller probabilities.
The more levels of compression that are considered when breaking up the union bound, the stronger the results are, and these tools have been successful applied to proving bounds on the singular values of combinatorial random matrices [31 ] , sharp bounds on their singularity probabilities [9 , 33 , 20 ] , and for random matrices below the O ( log ( r ) / r ) O(\log(r)/r) O ( log ( r ) / r ) sparsity threshold, sharp bounds on their rank [10 , 12 , 15 ] .
In our proof, we will take a coarse approach that distinguishes spiky vectors from flat vectors (see Definition 2.1 ), which are similar to the notions of compressible and incompressible vectors in this literature. Ultimately though, one strength of our proof is that we do not require taking intricate union bounds at many levels of compressibility.
Comparison with Prior Work
Subspace embeddings (OSEs). The study of subspace embeddings was initiated by Sarłós [32 ] . Clarkson and Woodruff [11 ] showed that CountSketch yields an input-sparsity OSE with dimension k = O ( r 2 ) k = O(r^{2}) k = O ( r 2 ) . To achieve linear dimension k = O ( r ) k = O(r) k = O ( r ) , Nelson and Nguy˜ˆen [28 ] introduced the OSNAP framework and conjectured that row sparsity ζ = O ( log r ) \zeta= O(\log r) ζ = O ( log r ) suffices. Intermediate bounds were obtained by Cohen [8 ] (k = O ( r log r ) k = O(r\log r) k = O ( r log r ) , ζ = O ( log r ) \zeta= O(\log r) ζ = O ( log r ) ) and Chenakkod et al. [5 ] (k = O ( r ) k = O(r) k = O ( r ) , ζ = O ( log 4 r ) \zeta= O(\log^{4} r) ζ = O ( log 4 r ) ). Recently, Heidary [17 ] settled the conjecture, proving that SparseStack is an ( r , 1 ± ε ) (r, 1\pm\varepsilon) ( r , 1 ± ε ) -OSE with dimension k = O ( r / ε 2 ) k = O(r/\varepsilon^{2}) k = O ( r / ε 2 ) and sparsity ζ = O ( ε − 1 log ( r / δ ) ) \zeta= O(\varepsilon^{-1}\log(r/\delta)) ζ = O ( ε − 1 log ( r / δ )) . Nelson and Nguy˜ˆen [28 ] also showed that any linear-dimension OSE requires row sparsity Ω ( log r / log log r ) \Omega(\log r/\log\log r) Ω ( log r / log log r ) , confirming that Θ ( log r ) \Theta(\log r) Θ ( log r ) is essentially optimal for two-sided embeddings.
Table 1. Comparison of dimension and row sparsity parameters for SparseStack and general sketching distributions at fixed accuracy and constant failure probability. The OSE bounds require two-sided distortion 1 ± ε 1 \pm\varepsilon 1 ± ε , whereas the OSI bounds guarantee one-sided constant injectivity σ min 2 ≥ α \sigma_{\min}^{2} \ge\alpha σ m i n 2 ≥ α .
Oblivious Subspace Injections (OSIs). Camaño, Epperly, Meyer, and Tropp [6 ] introduced OSIs and showed that one-sided injectivity suffices for core RNLA tasks. Using comparison theorems for the minimum eigenvalue of random positive-semidefinite matrices [35 ] , they showed that SparseStack achieves injectivity with k = O ( r ) k = O(r) k = O ( r ) and ζ = O ( log r ) \zeta= O(\log r) ζ = O ( log r ) . Subsequent Gaussian universality results [35 ] sharpened these guarantees, but existing analyses inherently rely on operator-norm concentration around the mean, which restricts them to ζ = Ω ( log r ) \zeta= \Omega(\log r) ζ = Ω ( log r ) . In contrast, our Theorem 1.1 achieves injectivity at ζ = O ( log r ) \zeta= O(\sqrt{\log r}) ζ = O ( log r ) by decoupling the sketch and avoiding operator-norm concentration entirely.
Sparsity lower bounds. For two-sided embeddings, Nelson and Nguy˜ˆen [28 ] proved that any linear-dimension OSE requires row sparsity Ω ( log r / log log r ) \Omega(\log r/\log\log r) Ω ( log r / log log r ) . For one-sided injections, Camaño et al. [6 ] conjectured that constant sparsity ζ = O ( 1 ) \zeta= O(1) ζ = O ( 1 ) might suffice (Problem 5.5 in the Simons Institute open problem collection). Huang, Rudelson, and Tikhomirov [19 ] refuted this by proving that constant row sparsity fails for any OSI, using matching-rooted trees to construct cancellation vectors. Theorem 1.3 establishes that any isotropic distribution over row-sparse matrices requires row sparsity ζ = Ω ( log r / log log r ) \zeta= \Omega(\sqrt{\log r/\log\log r}) ζ = Ω ( log r / log log r ) , showing that the log r \sqrt{\log r} log r scale is fundamental to all OSIs.
Conventions and Notation
All vector spaces considered in this paper are real and finite-dimensional. For d ≥ 1 d \ge1 d ≥ 1 , we write S d − 1 = { x ∈ R d : ∥ x ∥ = 1 } S^{d-1} = \{x \in\mathbb{R}^{d} : \lVert x\rVert= 1\} S d − 1 = { x ∈ R d : ∥ x ∥ = 1 } for the Euclidean unit sphere and B 2 d = { x ∈ R d : ∥ x ∥ ≤ 1 } B_{2}^{d} = \{x \in\mathbb{R}^{d} : \lVert x\rVert\le1\} B 2 d = { x ∈ R d : ∥ x ∥ ≤ 1 } for the Euclidean unit ball in R d \mathbb{R}^{d} R d . Vector norms ∥ ⋅ ∥ \lVert\cdot\rVert ∥ ⋅ ∥ are Euclidean, matrix norms ∥ ⋅ ∥ \lVert\cdot\rVert ∥ ⋅ ∥ without subscripts denote the spectral (operator) norm, and ∥ ⋅ ∥ F \lVert\cdot\rVert_{F} ∥ ⋅ ∥ F denotes the Frobenius norm. For a vector z ∈ R n z \in\mathbb{R}^{n} z ∈ R n and a coordinate subset I ⊆ [ n ] I \subseteq[n] I ⊆ [ n ] (with complement I c = [ n ] ∖ I I^{c} = [n] \setminus I I c = [ n ] ∖ I ), we write z I ∈ R ∣ I ∣ z_{I} \in\mathbb{R}^{|I|} z I ∈ R ∣ I ∣ for the subvector of coordinates indexed by I I I . Similarly, for a matrix V ∈ R n × d V \in\mathbb{R}^{n \times d} V ∈ R n × d , we write V I ∈ R ∣ I ∣ × d V_{I} \in\mathbb{R}^{|I| \times d} V I ∈ R ∣ I ∣ × d for the row submatrix indexed by I I I .
For a ζ \zeta ζ -block SparseStack matrix Ω ζ , b = ζ − 1 / 2 [ Φ 1 ⋯ Φ ζ ] ∈ R n × k \Omega_{\zeta,b} = \zeta^{-1/2}[\Phi_{1}\ \cdots\ \Phi_{\zeta}] \in\mathbb{R}^{n \times k} Ω ζ , b = ζ − 1/2 [ Φ 1 ⋯ Φ ζ ] ∈ R n × k (k = ζ b k = \zeta b k = ζ b ) as in (3) , the normalization factor ζ − 1 / 2 \zeta^{-1/2} ζ − 1/2 is always determined by the number of blocks ζ \zeta ζ in the subscript. When viewing the transpose Ω ζ , b T ∈ R k × n \Omega_{\zeta,b}^{\mathsf{T}} \in\mathbb{R}^{k \times n} Ω ζ , b T ∈ R k × n (or an individual block Φ t T ∈ R b × n \Phi_{t}^{\mathsf{T}} \in\mathbb{R}^{b \times n} Φ t T ∈ R b × n ) as the sketching operator acting on data vectors in R n \mathbb{R}^{n} R n , its i i i -th column vector is the 1-sparse bucket vector ϕ i t = ϱ i t e s i t ∈ R b \phi_{it} = \varrho_{it}e_{s_{it}} \in\mathbb{R}^{b} ϕ i t = ϱ i t e s i t ∈ R b corresponding to the i i i -th coordinate of the input data. Ω ζ , b , I ∈ R ∣ I ∣ × k \Omega_{\zeta,b,I} \in\mathbb{R}^{|I| \times k} Ω ζ , b , I ∈ R ∣ I ∣ × k denotes the row submatrix of Ω ζ , b \Omega_{\zeta,b} Ω ζ , b indexed by I I I , so that Ω ζ , b T z = Ω ζ , b , I T z I + Ω ζ , b , I c T z I c ∈ R k \Omega_{\zeta,b}^{\mathsf{T}}z = \Omega_{\zeta,b,I}^{\mathsf{T}}z_{I} + \Omega_{\zeta,b,I^{c}}^{\mathsf{T}}z_{I^{c}} \in\mathbb{R}^{k} Ω ζ , b T z = Ω ζ , b , I T z I + Ω ζ , b , I c T z I c ∈ R k decomposes the sketch of z ∈ R n z \in\mathbb{R}^{n} z ∈ R n .
High-Level Overview
Upper Bound Overview
In this subsection, we provide an intuitive, high-level overview of the proof of Theorem 1.1, explaining why standard sketching arguments break down in the ζ = O ( log r ) \zeta= O(\sqrt{\log r}) ζ = O ( log r ) sparsity regime and how our proof overcomes these barriers. Because our proof splits the SparseStack into two equal halves, from this section onward we consider for notational convenience a 2 ζ 2\zeta 2 ζ -block SparseStack Ω 2 ζ , b ∈ R n × 2 k \Omega_{2\zeta,b} \in\mathbb{R}^{n \times2k} Ω 2 ζ , b ∈ R n × 2 k with row sparsity 2 ζ = O ( log r ) 2\zeta= O(\sqrt{\log r}) 2 ζ = O ( log r ) and total embedding dimension 2 k = 2 ζ b = Θ ( r ) 2k = 2\zeta b = \Theta(r) 2 k = 2 ζ b = Θ ( r ) , so that each half has ζ \zeta ζ blocks and dimension k = ζ b k = \zeta b k = ζ b .
Why standard arguments fail: flat versus spiky vectors. Our goal is to show that for a fixed r r r -dimensional subspace, the column space of an isometry Q ∈ R n × r Q \in\mathbb{R}^{n \times r} Q ∈ R n × r , the SparseStack sketch matrix Ω 2 ζ , b ∈ R n × 2 k \Omega_{2\zeta,b} \in\mathbb{R}^{n \times2k} Ω 2 ζ , b ∈ R n × 2 k with total embedding dimension 2 k = 2 ζ b = Θ ( r ) 2k = 2\zeta b = \Theta(r) 2 k = 2 ζ b = Θ ( r ) and row sparsity 2 ζ = O ( log r ) 2\zeta= O(\sqrt{\log r}) 2 ζ = O ( log r ) does not break injectivity on any unit vector y = Q x y = Qx y = Q x for x ∈ S r − 1 = { x ∈ R r : ∥ x ∥ = 1 } x \in S^{r-1} = \{x \in\mathbb{R}^{r} : \lVert x \rVert= 1\} x ∈ S r − 1 = { x ∈ R r : ∥ x ∥ = 1 } ; meaning that no unit vector in the subspace has its sketched norm ∥ Ω 2 ζ , b T y ∥ \lVert\Omega_{2\zeta,b}^{\mathsf{T}}y \rVert ∥ Ω 2 ζ , b T y ∥ shrink below a constant threshold α \sqrt{\alpha} α . To understand the main difficulty, we distinguish between spiky vectors, that have a large portion of their mass concentrated on Θ ( r / ζ ) \Theta(r/\zeta) Θ ( r / ζ ) coordinates, and flat vectors, which have norm well-distributed.
For a vector y ∈ R n y \in\mathbb{R}^{n} y ∈ R n , and an integer threshold m m m , define
d m ( y ) : = min I ⊆ [ n ] ∣ I ∣ ≤ m ∥ y I c ∥ , (7) d_{m}(y) := \min_{\substack{I \subseteq[n]\\ |I| \leq m}} \lVert y_{I^{c}} \rVert, \tag*{(7)} d m ( y ) := I ⊆ [ n ] ∣ I ∣ ≤ m min ∥ y I c ∥ , (7)
where I c : = [ n ] ∖ I I^{c} := [n] \setminus I I c := [ n ] ∖ I .
Definition 2.1 (Flat and Spiky Vectors). We say a vector y ∈ R n y \in\mathbb{R}^{n} y ∈ R n is ( m , ε ) (m,\varepsilon) ( m , ε ) -flat , if d m ( y ) ≥ ε d_{m}(y) \geq\varepsilon d m ( y ) ≥ ε . We say y y y is ( m , ε ) (m,\varepsilon) ( m , ε ) -spiky if $d_{m}(y) < \varepsilon.
When m m m and ε \varepsilon ε are clear from context, we will just call these flat and spiky vectors respectively. In our proofs, we will take m = Θ ( r / ζ ) m = \Theta(r/\zeta) m = Θ ( r / ζ ) .
As seen in (5) and (6) , these two regimes cannot be analyzed directly using a union bound in the 2 ζ = O ( log r ) 2\zeta= O(\sqrt{\log r}) 2 ζ = O ( log r ) sparsity regime. Here, our strategy is to split the 2 ζ 2\zeta 2 ζ -block stack into two independent halves of ζ = O ( log r ) \zeta= O(\sqrt{\log r}) ζ = O ( log r ) blocks each, inducing a first half X = ( Ω ζ , b ( 1 ) ) T Q ∈ R k × r X = (\Omega_{\zeta,b}^{(1)})^{\mathsf{T}}Q \in\mathbb{R}^{k \times r} X = ( Ω ζ , b ( 1 ) ) T Q ∈ R k × r and an independent second half Y = ( Ω ζ , b ( 2 ) ) T Q ∈ R k × r Y = (\Omega_{\zeta,b}^{(2)})^{\mathsf{T}}Q \in\mathbb{R}^{k \times r} Y = ( Ω ζ , b ( 2 ) ) T Q ∈ R k × r (k = ζ b k = \zeta b k = ζ b ), and to analyze flat vectors and spiky vectors through a coordinated two-stage process.
Preserving flat vectors norm.
We first show that the first half X = ( Ω ζ , b ( 1 ) ) T Q X = (\Omega_{\zeta,b}^{(1)})^{\mathsf{T}}Q X = ( Ω ζ , b ( 1 ) ) T Q simultaneously preserves the norm of all flat unit vectors y = Q x y = Qx y = Q x with high probability (Theorem 2.2). For a fixed x ∈ S r − 1 x \in S^{r-1} x ∈ S r − 1 with d m ( y ) ≥ ε d_m(y) \ge\varepsilon d m ( y ) ≥ ε , we decompose y = h + a z y = h + az y = h + a z into its top-m m m coordinates h h h and its tail a z az a z , where a = d m ( y ) ≥ ε a = d_m(y) \ge\varepsilon a = d m ( y ) ≥ ε and ∥ z ∥ = 1 \lVert z\rVert= 1 ∥ z ∥ = 1 . Since each coordinate of a z az a z is at most 1 / m 1/\sqrt{m} 1/ m , dividing by a ≥ ε a \ge\varepsilon a ≥ ε gives ∥ z ∥ ∞ 2 ≤ 1 m ε 2 \lVert z\rVert_{\infty}^{2} \le\frac{1}{m\varepsilon^{2}} ∥ z ∥ ∞ 2 ≤ m ε 2 1 . Because every coordinate of z z z satisfies ∣ z i ∣ ≤ 1 / m ε 2 |z_i| \le1/\sqrt{m\varepsilon^{2}} ∣ z i ∣ ≤ 1/ m ε 2 and its random signs are independent of h h h , its norm does not cancel across buckets, yielding in Lemma 3.2 for any fixed x x x ,
P [ ∥ ( Ω ζ , b ( 1 ) ) T Q x ∥ < ε 10 ] ≤ 2 exp ( − Ω ( ζ ⋅ min { b , m ε 2 } ) ) . \mathbb{P}\left[\left\lVert(\Omega_{\zeta,b}^{(1)})^{\mathsf{T}}Qx\right\rVert< \frac{\varepsilon}{10}\right] \le2\exp\left(-\Omega\left(\zeta\cdot\min\left\{b,m\varepsilon^{2}\right\}\right)\right). P [ ( Ω ζ , b ( 1 ) ) T Q x < 10 ε ] ≤ 2 exp ( − Ω ( ζ ⋅ min { b , m ε 2 } ) ) .
Choosing m = Θ ( r / ζ ) m = \Theta(r/\zeta) m = Θ ( r / ζ ) and k = ζ b = Θ ( r ) k = \zeta b = \Theta(r) k = ζ b = Θ ( r ) makes this failure probability for a fixed x x x at most e − Ω ( r ) e^{-\Omega(r)} e − Ω ( r ) .
To extend this bound from a fixed x x x to all flat unit vectors y = Q x y = Qx y = Q x (x ∈ S r − 1 x \in S^{r-1} x ∈ S r − 1 ) without bounding the operator norm ∥ X ∥ \lVert X\rVert ∥ X ∥ , we show that for any η ∈ ( 0 , 1 ) \eta\in(0,1) η ∈ ( 0 , 1 ) , one can find a finite point set N η \mathcal{N}_{\eta} N η of size ∣ N η ∣ ≤ ( 16 / η ) r |\mathcal{N}_{\eta}| \le(16/\eta)^r ∣ N η ∣ ≤ ( 16/ η ) r such that for every x ∈ S r − 1 x \in S^{r-1} x ∈ S r − 1 , there exists x ′ ∈ N η x' \in\mathcal{N}_{\eta} x ′ ∈ N η satisfying
∥ x − x ′ ∥ ≤ η and ∥ X ( x − x ′ ) ∥ 2 ≤ η 2 4 r ∥ X ∥ F 2 . \lVert x-x'\rVert\le\eta\qquad\text{and}\qquad\lVert X(x-x')\rVert^{2} \le\frac{\eta^{2}}{4r}\lVert X\rVert_{F}^{2}. ∥ x − x ′ ∥ ≤ η and ∥ X ( x − x ′ ) ∥ 2 ≤ 4 r η 2 ∥ X ∥ F 2 .
(Lemma 3.3). Since E [ ∥ X ∥ F 2 ] = r \mathbb{E}[\lVert X\rVert_{F}^{2}] = r E [∥ X ∥ F 2 ] = r , Markov’s inequality ensures ∥ X ∥ F 2 = O ( r ) \lVert X\rVert_{F}^{2} = O(r) ∥ X ∥ F 2 = O ( r ) with high constant probability, which bounds the approximation error ∥ X ( x − x ′ ) ∥ \lVert X(x-x')\rVert ∥ X ( x − x ′ )∥ by O ( η ) O(\eta) O ( η ) without using ∥ X ∥ \lVert X\rVert ∥ X ∥ . Taking a union bound over all ∣ N η ∣ ≤ ( 16 / η ) r |\mathcal{N}_{\eta}| \le(16/\eta)^r ∣ N η ∣ ≤ ( 16/ η ) r points where each fixed point of N η \mathcal{N}_{\eta} N η has failure probability e − Ω ( r ) e^{-\Omega(r)} e − Ω ( r ) yields the following theorem:
Theorem 2.2. Fix any constant ε ∈ ( 0 , 1 ) \varepsilon\in(0,1) ε ∈ ( 0 , 1 ) and failure parameter δ l o c ∈ ( 0 , 1 ) \delta_{\mathrm{loc}} \in(0,1) δ loc ∈ ( 0 , 1 ) . There exists a constant λ = Ω ( ε ) \lambda= \Omega(\varepsilon) λ = Ω ( ε ) depending only on ε \varepsilon ε and δ l o c \delta_{\mathrm{loc}} δ loc such that for every fixed isometry Q ∈ R n × r Q \in\mathbb{R}^{n \times r} Q ∈ R n × r with r r r sufficiently large, every 1 ≤ ζ ≤ r 1 \le\zeta\le r 1 ≤ ζ ≤ r , every k = ζ b = Ω ( r ) k = \zeta b = \Omega(r) k = ζ b = Ω ( r ) , and m = Θ ( r / ζ ) m = \Theta(r/\zeta) m = Θ ( r / ζ ) with sufficiently large implied constants, the sketch X = ( Ω ζ , b ( 1 ) ) T Q ∈ R k × r X = (\Omega_{\zeta,b}^{(1)})^{\mathsf{T}}Q \in\mathbb{R}^{k \times r} X = ( Ω ζ , b ( 1 ) ) T Q ∈ R k × r satisfies
P [ ∃ x ∈ S r − 1 : d m ( Q x ) ≥ ε , ∥ X x ∥ < λ ] ≤ δ l o c . \mathbb{P}\left[\exists x \in S^{r-1}:d_m(Qx)\ge\varepsilon,\ \lVert Xx\rVert<\lambda\right]\le\delta_{\mathrm{loc}}. P [ ∃ x ∈ S r − 1 : d m ( Q x ) ≥ ε , ∥ X x ∥ < λ ] ≤ δ loc .
Finding a common support for problematic spiky vectors.
By Theorem 2.2, we know that all flat vectors are already well-preserved by Ω ζ , b ( 1 ) \Omega_{\zeta,b}^{(1)} Ω ζ , b ( 1 ) , so we only need to consider spiky unit vectors y = Q x y = Qx y = Q x satisfying d m ( y ) < ε d_m(y) < \varepsilon d m ( y ) < ε . To preserve the norm of all spiky vectors in col ( Q ) \operatorname{col}(Q) col ( Q ) simultaneously, it is not enough that each individual spiky vector concentrates on some set of m m m coordinates, since we cannot afford a union bound over all ( n m ) \binom{n}{m} ( m n ) possible m m m -coordinate subsets. Instead, we show that all spiky unit vectors y = Q x ∈ col ( Q ) y = Qx \in\operatorname{col}(Q) y = Q x ∈ col ( Q ) satisfying ∥ X x ∥ < λ \lVert Xx\rVert< \lambda ∥ X x ∥ < λ must in fact share a single, common set I ⊆ [ n ] I \subseteq[n] I ⊆ [ n ] of O ( m ) O(m) O ( m ) coordinates where their norm concentrates, avoiding any union bound over candidate supports.
The basic idea is as follows. By maneuvering by Theorem 2.2, we are able to show that there exists some H = ε 2 I r + λ − 2 X T X H = \varepsilon^{2}I_r + \lambda^{-2}X^{\mathsf{T}}X H = ε 2 I r + λ − 2 X T X , such that for all x x x , we have
( d m ( Q x ) ) 2 ≤ x T H x . (8) (d_m(Qx))^{2}\le x^{\mathsf{T}}Hx. \tag*{(8)} ( d m ( Q x ) ) 2 ≤ x T H x . (8)
In other words, for each x x x , there exists a set I I I with ∣ I ∣ ≤ m |I| \le m ∣ I ∣ ≤ m such that ∥ Q I c x ∥ 2 − x T H x ≤ 0 \lVert Q_{I^c}x\rVert^2 - x^{\mathsf T}Hx \le0 ∥ Q I c x ∥ 2 − x T H x ≤ 0 .
We would like to leverage a minimax-style argument to show that there exists some I I I , such that for all x x x , we have ∥ Q I c x ∥ 2 − x T H x ≤ 0 \lVert Q_{I^c}x\rVert^2 - x^{\mathsf T}Hx \le0 ∥ Q I c x ∥ 2 − x T H x ≤ 0 . Unfortunately there are two obstacles.
The first obstacle is the minimization over I I I is not over a convex space.
The second obstacle is the objective function ∥ Q I c x ∥ 2 − x T H x \lVert Q_{I^c}x\rVert^2 - x^{\mathsf T}Hx ∥ Q I c x ∥ 2 − x T H x is not concave in x x x .
To resolve the second obstacle, we show that up to the loss of some constant factors, we can attain the guarantee ∥ Q I c x ∥ 2 − x T H x ≤ 0 \lVert Q_{I^c}x\rVert^2 - x^{\mathsf T}Hx \le0 ∥ Q I c x ∥ 2 − x T H x ≤ 0 simultaneously with the same I I I for a whole distribution of x ∼ N ( 0 , R ) x \sim\mathcal{N}(0,R) x ∼ N ( 0 , R ) . Formally, using Lemma 4.1 , we attain that
min ∣ I ∣ ≤ 4 m E x ∼ R x T Q I c T Q I c x − 4 3 tr ( R H ) ≤ 0. (9) \min_{|I|\le4m}\mathbb{E}_{x\sim R} x^{\mathsf T}Q_{I^c}^{\mathsf T}Q_{I^c}x-\frac{4}{3}\operatorname{tr}(RH)\le0. \tag*{(9)} ∣ I ∣ ≤ 4 m min E x ∼ R x T Q I c T Q I c x − 3 4 tr ( R H ) ≤ 0. (9)
Notice that the objective above in linear in R R R . Further, by relaxing from minimization over I I I to minimization over the convex set of fractional weight vector w ∈ [ 0 , 1 ] n w \in[0,1]^n w ∈ [ 0 , 1 ] n with ∑ w i ≤ 4 m \sum w_i \le4m ∑ w i ≤ 4 m .
We can now use minimax duality over R \mathcal{R} R and w w w , and ultimately round back that resulting w ∗ w^* w ∗ to find a single I I I with the desired property.
Formally we prove the following result.
Theorem 2.3. Let Q ∈ R n × r Q \in\mathbb{R}^{n\times r} Q ∈ R n × r be an isometry. Suppose a matrix X ∈ R k × r X \in\mathbb{R}^{k\times r} X ∈ R k × r satisfies
d m ( Q x ) ≥ ε , ∥ x ∥ = 1 ⟹ ∥ X x ∥ ≥ λ d_m(Qx) \ge\varepsilon,\quad\lVert x\rVert=1\quad\Longrightarrow\quad\lVert Xx\rVert\ge\lambda d m ( Q x ) ≥ ε , ∥ x ∥ = 1 ⟹ ∥ X x ∥ ≥ λ for some ε > 0 \varepsilon>0 ε > 0 and λ > 0 \lambda>0 λ > 0 . Then there exists a coordinate set I ⊆ [ n ] I\subseteq[n] I ⊆ [ n ] with ∣ I ∣ ≤ 8 m |I|\le8m ∣ I ∣ ≤ 8 m such that
Q I c T Q I c ⪯ 8 3 ε 2 I r + 8 3 λ 2 X T X . Q_{I^c}^{\mathsf T}Q_{I^c}\preceq\frac{8}{3}\varepsilon^2 I_r+\frac{8}{3\lambda^2}X^{\mathsf T}X. Q I c T Q I c ⪯ 3 8 ε 2 I r + 3 λ 2 8 X T X .
This theorem implies the following: since ∥ ( Q x ) I ∥ 2 + ∥ ( Q x ) I c ∥ 2 = ∥ Q x ∥ 2 = 1 \lVert(Qx)_I\rVert^2+\lVert(Qx)_{I^c}\rVert^2=\lVert Qx\rVert^2=1 ∥( Q x ) I ∥ 2 + ∥( Q x ) I c ∥ 2 = ∥ Q x ∥ 2 = 1 for every unit vector x ∈ S r − 1 x\in S^{r-1} x ∈ S r − 1 , we have
∥ ( Q x ) I ∥ 2 = 1 − ∥ ( Q x ) I c ∥ 2 ≥ 1 − 8 3 ε 2 − 8 3 λ 2 ∥ X x ∥ 2 . \lVert(Qx)_I\rVert^2=1-\lVert(Qx)_{I^c}\rVert^2\ge1-\frac{8}{3}\varepsilon^2-\frac{8}{3\lambda^2}\lVert Xx\rVert^2. ∥( Q x ) I ∥ 2 = 1 − ∥( Q x ) I c ∥ 2 ≥ 1 − 3 8 ε 2 − 3 λ 2 8 ∥ X x ∥ 2 .
In particular, choosing ε = 1 / 3 \varepsilon=1/3 ε = 1/3 shows that for every unit vector x x x whose sketch norm ∥ X x ∥ \lVert Xx\rVert ∥ X x ∥ is small, approximately at least 19 27 \frac{19}{27} 27 19 of the squared norm of Q x Qx Q x is guaranteed to reside on the single coordinate set I I I of size ∣ I ∣ ≤ 8 m = O ( r / ζ ) |I|\le8m=O(r/\zeta) ∣ I ∣ ≤ 8 m = O ( r / ζ ) .
Preserving the heavy coordinates in the second half.
Combining Theorem 2.2 and Theorem 2.3, the first half ( Ω ζ , b ( 1 ) ) T (\Omega_{\zeta,b}^{(1)})^{\mathsf T} ( Ω ζ , b ( 1 ) ) T with high probability preserves all flat vectors y = Q x y=Qx y = Q x and localizes the norm of all spiky vectors y = Q x y=Qx y = Q x on which injectivity fails under ( Ω ζ , b ( 1 ) ) T (\Omega_{\zeta,b}^{(1)})^{\mathsf T} ( Ω ζ , b ( 1 ) ) T (i.e., with small ∥ ( Ω ζ , b ( 1 ) ) T y ∥ \lVert(\Omega_{\zeta,b}^{(1)})^{\mathsf T}y\rVert ∥( Ω ζ , b ( 1 ) ) T y ∥ ) onto a single small coordinate set I ⊆ [ n ] I\subseteq[n] I ⊆ [ n ] of size N : = ∣ I ∣ ≤ 8 m = O ( r / ζ ) N:=|I|\le8m=O(r/\zeta) N := ∣ I ∣ ≤ 8 m = O ( r / ζ ) . Therefore, the remaining task is to show that whenever ∥ ( Ω ζ , b ( 1 ) ) T y ∥ \lVert(\Omega_{\zeta,b}^{(1)})^{\mathsf T}y\rVert ∥( Ω ζ , b ( 1 ) ) T y ∥ is small, the independent second half ( Ω ζ , b ( 2 ) ) T (\Omega_{\zeta,b}^{(2)})^{\mathsf T} ( Ω ζ , b ( 2 ) ) T preserves the norm of y y y , ensuring that injectivity is not broken on y y y under the full sketch.
Why is it easier to prove that Ω ζ , b ( 2 ) \Omega_{\zeta,b}^{(2)} Ω ζ , b ( 2 ) handles these spiky vectors even though Ω ζ , b ( 1 ) \Omega_{\zeta,b}^{(1)} Ω ζ , b ( 1 ) and Ω ζ , b ( 2 ) \Omega_{\zeta,b}^{(2)} Ω ζ , b ( 2 ) share the same distribution? The reason is that we have now restricted the spiky coordinates from all $n coordinates down to a single known set I I I of size N = O ( m ) N = O(m) N = O ( m ) . Using the shorthand Ω : = Ω ζ , b ( 2 ) \Omega:= \Omega_{\zeta,b}^{(2)} Ω := Ω ζ , b ( 2 ) , we can expand the squared sketch norm ∥ Ω ζ , b ( 2 ) y ∥ 2 \|\Omega_{\zeta,b}^{(2)}y\|^2 ∥ Ω ζ , b ( 2 ) y ∥ 2 as follows:
∥ ( Ω ζ , b ( 2 ) ) ⊤ y ∥ 2 = ∥ Ω I ⊤ y I + Ω I c ⊤ Q I c x ∥ 2 = ∥ Ω I ⊤ y I ∥ 2 + 2 y I ⊤ ( Ω I Ω I c ⊤ Q I c ) x + ∥ Ω I c ⊤ Q I c x ∥ 2 . (10) \begin{aligned}
\left\|(\Omega_{\zeta,b}^{(2)})^\top y\right\|^2
= \left\|\Omega_I^\top y_I+\Omega_{I^c}^\top Q_{I^c}x\right\|^2 \\
&= \left\|\Omega_I^\top y_I\right\|^2+2y_I^\top\left(\Omega_I\Omega_{I^c}^\top Q_{I^c}\right)x+\left\|\Omega_{I^c}^\top Q_{I^c}x\right\|^2. \tag*{(10)}
\end{aligned} ( Ω ζ , b ( 2 ) ) ⊤ y 2 = Ω I ⊤ y I + Ω I c ⊤ Q I c x 2 = Ω I ⊤ y I 2 + 2 y I ⊤ ( Ω I Ω I c ⊤ Q I c ) x + Ω I c ⊤ Q I c x 2 . (10)
Controlling the cross term of collisions between I I I and I c I^c I c . In § 5 , we will show that [a small modification of] this cross term in (10) is smaller than a sufficiently small constant. To do this, we will bound the operator norm of Ω I Ω I c ⊤ Q I c \Omega_I\Omega_{I^c}^\top Q_{I^c} Ω I Ω I c ⊤ Q I c using Lemma 2.4 below. The intuition behind Lemma 2.4 is that the matrix Ω I Ω I c ⊤ ∈ R ∣ I ∣ × ∣ I c ∣ \Omega_I\Omega_{I^c}^\top\in\mathbb{R}^{|I|\times|I^c|} Ω I Ω I c ⊤ ∈ R ∣ I ∣ × ∣ I c ∣ has entries which are non-zero with probability roughly Θ ( ζ 2 / r ) \Theta(\zeta^2/r) Θ ( ζ 2 / r ) . Indeed, ( Ω I Ω I c ⊤ ) i j (\Omega_I\Omega_{I^c}^\top)_{ij} ( Ω I Ω I c ⊤ ) ij is a dot-product between two independent, ζ \zeta ζ -sparse vectors in R k \mathbb{R}^k R k , so they have a non-zero dot product with probability roughly Θ ( ζ 2 / k ) = Θ ( ζ 2 / r ) \Theta(\zeta^2/k)=\Theta(\zeta^2/r) Θ ( ζ 2 / k ) = Θ ( ζ 2 / r ) . If ζ = Ω ( log ( r ) ) \zeta=\Omega(\sqrt{\log(r)}) ζ = Ω ( log ( r ) ) , and thus the non-zero probability is Ω ( log ( r ) / r ) \Omega(\log(r)/r) Ω ( log ( r ) / r ) , we can expect universal behavior from Ω I Ω I c ⊤ \Omega_I\Omega_{I^c}^\top Ω I Ω I c ⊤ . Leveraging a result of [17 ] for OSEs with entry non-zero probability Ω ( log ( r ) / r ) \Omega(\log(r)/r) Ω ( log ( r ) / r ) , we show that [a slight modification of] the cross-term matrix Ω I Ω I c ⊤ Q I c \Omega_I\Omega_{I^c}^\top Q_{I^c} Ω I Ω I c ⊤ Q I c has constant operator norm in expectation.
Of course, because the entries of Ω I Ω I c ⊤ \Omega_I\Omega_{I^c}^\top Ω I Ω I c ⊤ are not independent, proving this requires a careful coupling argument to a suitable matrix with i.i.d. entries. We leave the details of this argument to § 5 . We state the main result of the section here.
Lemma 2.4. There exists a universal constant C c g > 0 C_{\mathrm{cg}}>0 C cg > 0 such that the following holds. Let Ω ∈ R n × k \Omega\in\mathbb{R}^{n\times k} Ω ∈ R n × k (k = ζ b k=\zeta b k = ζ b ) be a SparseStack sketch with ζ ≥ 1 \zeta\ge1 ζ ≥ 1 blocks and b ≥ 2 ζ b\ge2\zeta b ≥ 2 ζ buckets per block. Let I , J ⊆ [ n ] I,J\subseteq[n] I , J ⊆ [ n ] be disjoint coordinate sets with ∣ I ∣ = N ≥ 1 |I|=N\ge1 ∣ I ∣ = N ≥ 1 , and let W ∈ R ∣ J ∣ × d W\in\mathbb{R}^{|J|\times d} W ∈ R ∣ J ∣ × d (d ≥ 1 d\ge1 d ≥ 1 ) be a fixed isometry. Let T ⊆ [ k ] T\subseteq[k] T ⊆ [ k ] be the set of rows in which Ω I \Omega_I Ω I has exactly one non-zero entry, and let Ω ~ ⊤ \widetilde{\Omega}^\top Ω ⊤ denote the restrictions of Ω ⊤ \Omega^\top Ω ⊤ to rows in T T T .
For every integer p ≥ max { 1 , log ( 2 d ) } p\ge\max\{1,\log(2d)\} p ≥ max { 1 , log ( 2 d )} , setting D p = d + 2 p + 1 D_p=d+2p+1 D p = d + 2 p + 1 , we have conditionally on Ω I \Omega_I Ω I :
E [ ∥ Ω ~ I Ω ~ J ⊤ W ∥ ∣ Ω I ] ≤ C c g ( N + D p k + p + 1 ζ 2 + N d ζ 2 b 3 ( ζ 3 ) ) . \mathbb{E}\left[\left\|\widetilde{\Omega}_I\widetilde{\Omega}_J^\top W\right\|\mid\Omega_I\right]\le C_{\mathrm{cg}}\left(\sqrt{\frac{N+D_p}{k}}+\sqrt{\frac{p+1}{\zeta^2}}+\sqrt{\frac{Nd}{\zeta^2b^3}\binom{\zeta}{3}}\right). E [ Ω I Ω J ⊤ W ∣ Ω I ] ≤ C cg ( k N + D p + ζ 2 p + 1 + ζ 2 b 3 N d ( 3 ζ ) ) . If N = 0 N=0 N = 0 or d = 0 d=0 d = 0 , then the expression is trivially 0.
Avoiding collisions among heavy coordinates in I I I . In § 6 , we show that the term ∥ Ω I ⊤ y I ∥ 2 \|\Omega_I^\top y_I\|^2 ∥ Ω I ⊤ y I ∥ 2 is large. Here we leverage the fact that N = ∣ I ∣ = O ( r / ζ ) N=|I|=O(r/\zeta) N = ∣ I ∣ = O ( r / ζ ) to show that the columns of Ω I ⊤ \Omega_I^\top Ω I ⊤ do not have many collisions, and thus most entries in y I y_I y I are directly copied to ζ \zeta ζ entries in Ω I ⊤ y I \Omega_I^\top y_I Ω I ⊤ y I (up to a sign change, and a factor of 1 / ζ 1/\sqrt{\zeta} 1/ ζ ).
Indeed, we can show that any coordinate i ∈ I i\in I i ∈ I lands in a bucket containing no other element of I I I with probability at least 1 − N / b ≥ 15 / 16 1-N/b\ge15/16 1 − N / b ≥ 15/16 . When a coordinate is isolated in its bucket, no other coordinate of I I I can cancel its contribution. If a constant fraction of the buckets that i i i lands in contain no other element of I I I , then a constant fraction of the norm of y I y_I y I will be represented in the output. We say in this case that the element i i i is good (see Definition 6.2).
For each i i i , it is easy to show using standard concentration tools that with probability 1 − e − Ω ( ζ ) 1-e^{-\Omega(\zeta)} 1 − e − Ω ( ζ ) , element i i i is good. Unfortunately, this is not large enough to guarantee via a union bound that all
Figure 1. Isolated buckets: a lone coordinate is copied to the output; colliding ones may cancel.
elements in I I I are good. Thus it is still possible that the mass of y I y_I y I concentrates on the elements that are not good, and that this mass gets canceled by other elements of I I I .
To bypass this union-bound barrier, we divide the ζ \zeta ζ blocks of the second half into two disjoint groups, Group 1 and Group 2, of ζ 0 = ζ / 2 = O ( log r ) \zeta_0 = \zeta/2 = O(\sqrt{\log r}) ζ 0 = ζ /2 = O ( log r ) blocks each:
Group 1: Let G ⊆ I G \subseteq I G ⊆ I be the good coordinates isolated in at least 11 12 ζ 0 \frac{11}{12}\zeta_0 12 11 ζ 0 blocks of Group 1, and let B = I ∖ G B = I \setminus G B = I ∖ G be the remaining bad coordinates. Each coordinate of I I I falls into B B B with probability e − Ω ( ζ ) e^{-\Omega(\zeta)} e − Ω ( ζ ) , so by Markov’s inequality the bad set shrinks to ∣ B ∣ ≤ N e − Ω ( ζ ) |B| \leq N e^{-\Omega(\zeta)} ∣ B ∣ ≤ N e − Ω ( ζ ) with high probability, while all coordinates in G G G are already preserved by Group 1.
Group 2: Conditioned on Group 1, the good coordinates G G G are already preserved by Group 1, so we can absorb G G G into the complement I c I^c I c , whose cancellation effect on B B B is negligible by the exact same argument from Lemma 2.4 as for I c I^c I c ! This is the decisive factor that makes a union bound possible in Group 2 even though it failed before. Thus, in Group 2, the surviving bad coordinates B B B only need to avoid colliding with each other. Since only ∣ B ∣ ≤ N e − Ω ( ζ ) |B| \leq N e^{-\Omega(\zeta)} ∣ B ∣ ≤ N e − Ω ( ζ ) bad coordinates remain across the b b b buckets, the per-block collision probability within B B B drops from O ( 1 ) O(1) O ( 1 ) to ∣ B ∣ / b ≤ e − Ω ( ζ ) / 16 |B|/b \leq e^{-\Omega(\zeta)}/16 ∣ B ∣/ b ≤ e − Ω ( ζ ) /16 . Thus, a coordinate i ∈ B i \in B i ∈ B fails to be isolated in 11 12 ζ 0 \frac{11}{12}\zeta_0 12 11 ζ 0 blocks of Group 2 with probability at most
2 ζ 0 ( e − Ω ( ζ ) 16 ) ζ 0 / 12 = e − Ω ( ζ 2 ) = e − Ω ( log r ) ≪ 1 r , 2^{\zeta_0}\left(\frac{e^{-\Omega(\zeta)}}{16}\right)^{\zeta_0/12}
= e^{-\Omega(\zeta^2)}
= e^{-\Omega(\log r)}
\ll\frac{1}{r}, 2 ζ 0 ( 16 e − Ω ( ζ ) ) ζ 0 /12 = e − Ω ( ζ 2 ) = e − Ω ( l o g r ) ≪ r 1 ,
which is now small enough to union bound over all coordinates in B B B .
Notice that the argument above bounding the cancellation from the complement only uses ∑ j = 1 n ∥ q j ∥ 2 = ∥ Q ∥ F 2 = r \sum_{j=1}^{n}\lVert q_j\rVert^2 = \lVert Q\rVert_F^2 = r ∑ j = 1 n ∥ q j ∥ 2 = ∥ Q ∥ F 2 = r and never required the target coordinate set to carry a constant fraction of the total norm. This is why the exact same argument applies to Group 2 even though B B B is not guaranteed to carry a constant fraction of the norm on its own. The condition that I I I carries a large fraction of the norm is only used at the very end when we sum the squared sketch norms from Group 1, which preserves ( Q x ) G (Qx)_G ( Q x ) G , and Group 2, which preserves ( Q x ) B (Qx)_B ( Q x ) B , to recover ∥ ( Q x ) G ∥ 2 + ∥ ( Q x ) B ∥ 2 = ∥ ( Q x ) I ∥ 2 \lVert(Qx)_G\rVert^2 + \lVert(Qx)_B\rVert^2 = \lVert(Qx)_I\rVert^2 ∥( Q x ) G ∥ 2 + ∥( Q x ) B ∥ 2 = ∥( Q x ) I ∥ 2 . This yields the following end-to-end guarantee for the second half of the sketch.
Theorem 2.5. Fix any constants t 0 ∈ ( 0 , 1 / 4 ) t_0 \in(0,1/4) t 0 ∈ ( 0 , 1/4 ) and δ r e p ∈ ( 0 , 1 ) \delta_{\mathrm{rep}} \in(0,1) δ rep ∈ ( 0 , 1 ) . There exists a constant β = Ω ( 1 ) \beta= \Omega(1) β = Ω ( 1 ) such that for every fixed isometry Q ∈ R n × r Q \in\mathbb{R}^{n \times r} Q ∈ R n × r with r ≥ 2 r \geq2 r ≥ 2 , every integer pair ζ = Ω ( log r ) \zeta= \Omega(\sqrt{\log r}) ζ = Ω ( log r ) and k = ζ b = Ω ( r ) k = \zeta b = \Omega(r) k = ζ b = Ω ( r ) , and every coordinate set I ⊆ [ n ] I \subseteq[n] I ⊆ [ n ] of size ∣ I ∣ ≤ min { r , b / 16 } |I| \leq\min\{r,b/16\} ∣ I ∣ ≤ min { r , b /16 } , the sketch Y = ( Ω ζ , b ( 2 ) ) T Q ∈ R k × r Y = (\Omega_{\zeta,b}^{(2)})^{\mathsf{T}}Q \in\mathbb{R}^{k \times r} Y = ( Ω ζ , b ( 2 ) ) T Q ∈ R k × r satisfies, with probability at least 1 − δ r e p 1-\delta_{\mathrm{rep}} 1 − δ rep ,
∥ Y x ∥ 2 ≥ β ∥ ( Q x ) I ∥ 2 − t 0 ∥ ( Q x ) I c ∥ 2 \lVert Yx\rVert^2 \geq\beta\lVert(Qx)_I\rVert^2 - t_0\lVert(Qx)_{I^c}\rVert^2 ∥ Y x ∥ 2 ≥ β ∥( Q x ) I ∥ 2 − t 0 ∥( Q x ) I c ∥ 2 for all x ∈ R r x \in\mathbb{R}^r x ∈ R r ,
or equivalently, Y T Y + ( β + t 0 ) Q I c T Q I c ⪰ β I r . \text{or equivalently,}\quad Y^{\mathsf{T}}Y + (\beta+t_0)Q_{I^c}^{\mathsf{T}}Q_{I^c} \succeq\beta I_r. or equivalently, Y T Y + ( β + t 0 ) Q I c T Q I c ⪰ β I r .
With Theorem 2.3 from the first half X X X and Theorem 2.5 from the second half Y Y Y in hand, the proof of Theorem 1.1 concludes in three lines: Theorem 2.5 gives Y T Y + ( β + t 0 ) Q I c T Q I c ⪰ β I r Y^{\mathsf{T}}Y+(\beta+t_0)Q_{I^c}^{\mathsf{T}}Q_{I^c}\succeq\beta I_r Y T Y + ( β + t 0 ) Q I c T Q I c ⪰ β I r , while Theorem 2.3 bounds the norm on I c I^c I c by Q I c T Q I c ⪯ u I r + v X T X Q_{I^c}^{\mathsf{T}}Q_{I^c}\preceq uI_r+vX^{\mathsf{T}}X Q I c T Q I c ⪯ u I r + v X T X with u = 8 3 ξ 2 < β β + t 0 u=\frac{8}{3}\xi^2<\frac{\beta}{\beta+t_0} u = 3 8 ξ 2 < β + t 0 β . Substituting the latter into the former immediately gives Y T Y + ( β + t 0 ) v X T X ⪰ ( β − ( β + t 0 ) u ) I r = Ω ( I r ) Y^{\mathsf{T}}Y+(\beta+t_0)vX^{\mathsf{T}}X\succeq(\beta-(\beta+t_0)u)I_r=\Omega(I_r) Y T Y + ( β + t 0 ) v X T X ⪰ ( β − ( β + t 0 ) u ) I r = Ω ( I r ) . Since the full sketch A = Ω ζ , b Q ∈ R 2 k × r A=\Omega_{\zeta,b}Q\in\mathbb{R}^{2k\times r} A = Ω ζ , b Q ∈ R 2 k × r is formed by stacking the two halves X X X and Y Y Y via A = 1 2 [ X Y ] A = \frac{1}{\sqrt{2}}\begin{bmatrix} X \\ Y \end{bmatrix} A = 2 1 [ X Y ] so that A T A = 1 2 ( X T X + Y T Y ) A^{\mathsf{T}}A = \frac{1}{2}(X^{\mathsf{T}}X + Y^{\mathsf{T}}Y) A T A = 2 1 ( X T X + Y T Y ) , this positive lower bound on a weighted sum of X T X X^{\mathsf{T}}X X T X and Y T Y Y^{\mathsf{T}}Y Y T Y immediately implies A T A ⪰ Ω ( I r ) A^{\mathsf{T}}A \succeq\Omega(I_r) A T A ⪰ Ω ( I r ) , completing the proof.
Remark 2.6. Because our analysis divides Ω ζ , b \Omega_{\zeta,b} Ω ζ , b into two halves, we are only able to show a constant lower bound on the smallest singular value of Ω ζ , b T Q \Omega_{\zeta,b}^{\mathsf{T}}Q Ω ζ , b T Q , and not a 1 − ε 1-\varepsilon 1 − ε lower bound. We believe this to be an artifact of the proof, and not a fundamental obstruction.
Why ζ = O ( log r ) \zeta= O(\sqrt{\log r}) ζ = O ( log r ) is the natural scale. With the full two-stage architecture in view, we can now see why ζ = O ( log r ) \zeta= O(\sqrt{\log r}) ζ = O ( log r ) is the natural scale for constant injectivity. In the first half, spiky vectors in col ( Q ) \operatorname{col}(Q) col ( Q ) are localized onto a single coordinate set I I I of size N = O ( r / ζ ) N = O(r/\zeta) N = O ( r / ζ ) . In the second half, dividing the ζ \zeta ζ blocks into Group 1 and Group 2 ensures that most coordinates of I I I avoid collisions in Group 1, leaving only ∣ B ∣ ≤ N e − Ω ( ζ ) |B| \leq N e^{-\Omega(\zeta)} ∣ B ∣ ≤ N e − Ω ( ζ ) un-isolated coordinates. In Group 2, because the number of surviving coordinates has shrunk to ∣ B ∣ ≤ N e − Ω ( ζ ) |B| \leq N e^{-\Omega(\zeta)} ∣ B ∣ ≤ N e − Ω ( ζ ) , their mutual collision probability per block drops from O ( 1 ) O(1) O ( 1 ) to ∣ B ∣ / b ≤ e − Ω ( ζ ) / 16 |B|/b \leq e^{-\Omega(\zeta)}/16 ∣ B ∣/ b ≤ e − Ω ( ζ ) /16 , suppressing the failure probability of Group 2 to e − Ω ( ζ 2 ) e^{-\Omega(\zeta^2)} e − Ω ( ζ 2 ) . Setting the row sparsity to ζ = O ( log r ) \zeta= O(\sqrt{\log r}) ζ = O ( log r ) (specifically ζ ≥ C log ( e r ) \zeta\geq C\sqrt{\log(er)} ζ ≥ C log ( er ) ) is precisely the scale where this failure probability becomes e − Ω ( log r ) ≪ 1 / r e^{-\Omega(\log r)} \ll1/r e − Ω ( l o g r ) ≪ 1/ r , allowing the union bound over all surviving coordinates in B B B to succeed.
Another lens to see this is the cross term argument of § 5 . Here we argued that the cross term, governing cancellations between I I I and I c I^c I c , was small because the operator norm of the matrix Ω ζ , b , I ( 2 ) ( Ω ζ , b , I c ( 2 ) ) T Q \Omega_{\zeta,b,I}^{(2)}(\Omega_{\zeta,b,I^c}^{(2)})^{\mathsf{T}}Q Ω ζ , b , I ( 2 ) ( Ω ζ , b , I c ( 2 ) ) T Q was bounded by a sufficiently small constant. This used the fact that the entries of Ω ζ , b , I ( 2 ) ( Ω ζ , b , I c ( 2 ) ) T \Omega_{\zeta,b,I}^{(2)}(\Omega_{\zeta,b,I^c}^{(2)})^{\mathsf{T}} Ω ζ , b , I ( 2 ) ( Ω ζ , b , I c ( 2 ) ) T were almost i.i.d., with non-zero probability of ζ 2 r \frac{\zeta^2}{r} r ζ 2 , which is above the universality threshold log ( r ) r \frac{\log(r)}{r} r l o g ( r ) whenever ζ = Ω ( log ( r ) ) \zeta= \Omega(\sqrt{\log(r)}) ζ = Ω ( log ( r ) ) .
Figure 2. Proof structure: X X X handles flat vectors, and the two groups of Y Y Y handle spiky ones.
Lower Bound Overview
Our starting point is the breakthrough framework of Huang, Rudelson, and Tikhomirov [19 ] , denoted by HRT, which demonstrated that constant row sparsity ζ = O ( 1 ) \zeta= O(1) ζ = O ( 1 ) cannot achieve an OSI. However, their analysis was capped at ζ ≤ ( log r ) 1 / 7 \zeta\le(\log r)^{1/7} ζ ≤ ( log r ) 1/7 and relied on an artificial entry-magnitude regularity assumption. In this work, we build on HRT’s high-level combinatorial architecture, but fundamentally overhaul the underlying linear-algebraic cancellation technique to achieve the tight ζ ≤ O ( log r / log log r ) \zeta\le O(\sqrt{\log r}/\log\log r) ζ ≤ O ( log r / log log r ) lower bound without entry-magnitude assumptions.
Preliminaries
Recall that an isotropic random sketching matrix Ω ∈ R n × k \Omega\in\mathbb{R}^{n \times k} Ω ∈ R n × k with k ≤ K r k \le Kr k ≤ K r for some constant K K K is an ( r , α ) (r,\alpha) ( r , α ) -OSI if P Ω [ σ min ( Ω ⊤ Q ) 2 ≥ α ] ≥ 19 / 20 \mathbb{P}_{\Omega}[\sigma_{\min}(\Omega^\top Q)^2 \ge\alpha] \ge19/20 P Ω [ σ m i n ( Ω ⊤ Q ) 2 ≥ α ] ≥ 19/20 for every matrix Q ∈ R n × r Q \in\mathbb{R}^{n \times r} Q ∈ R n × r . To show that Ω \Omega Ω fails this property, it suffices to find a single r r r -dimensional subspace Q Q Q where injectivity is broken under the random sketch with high probability. Even if Q Q Q is restricted to coordinate subspaces spanned by r r r standard basis vectors { e i } i ∈ R \{e_i\}_{i\in R} { e i } i ∈ R , where Q R = I R Q_R = I_R Q R = I R for an index subset R ⊂ [ n ] R \subset[n] R ⊂ [ n ] with ∣ R ∣ = r |R| = r ∣ R ∣ = r , showing that injectivity fails on one such candidate is already sufficient. Furthermore, if we select R ⊂ [ n ] R \subset[n] R ⊂ [ n ] of size r r r uniformly at random, showing that injectivity fails with high probability on average is also sufficient:
min R ∗ ∈ ( [ n ] r ) P Ω [ σ min ( Ω R ∗ ⊤ ) 2 ≥ α ] ≤ E R ∼ Unif ( ( [ n ] r ) ) [ P Ω [ σ min ( Ω R ⊤ ) 2 ≥ α ] ] = E Ω [ P R ∼ Unif ( ( [ n ] r ) ) [ σ min ( Ω R ⊤ ) 2 ≥ α ] ] . (11) \min_{R^* \in\binom{[n]}{r}} \mathbb{P}_{\Omega}\left[\sigma_{\min}(\Omega^\top_{R^*})^2 \ge\alpha\right] \le\mathbb{E}_{R \sim\operatorname{Unif}\left(\binom{[n]}{r}\right)}\left[\mathbb{P}_{\Omega}\left[\sigma_{\min}(\Omega^\top_R)^2 \ge\alpha\right]\right]
= \mathbb{E}_{\Omega}\left[\mathbb{P}_{R \sim\operatorname{Unif}\left(\binom{[n]}{r}\right)}\left[\sigma_{\min}(\Omega^\top_R)^2 \ge\alpha\right]\right]. \tag*{(11)} R ∗ ∈ ( r [ n ] ) min P Ω [ σ m i n ( Ω R ∗ ⊤ ) 2 ≥ α ] ≤ E R ∼ Unif ( ( r [ n ] ) ) [ P Ω [ σ m i n ( Ω R ⊤ ) 2 ≥ α ] ] = E Ω [ P R ∼ Unif ( ( r [ n ] ) ) [ σ m i n ( Ω R ⊤ ) 2 ≥ α ] ] . (11)
By isotropy, E Ω [ ∥ Ω ∥ F 2 ] = ∑ i = 1 n E [ ∥ s i ∥ 2 2 ] = n \mathbb{E}_{\Omega}[\|\Omega\|_F^2] = \sum_{i=1}^{n} \mathbb{E}[\|s_i\|_2^2] = n E Ω [ ∥Ω ∥ F 2 ] = ∑ i = 1 n E [ ∥ s i ∥ 2 2 ] = n . Hence, by Markov’s inequality, the bad event of a large Frobenius norm satisfies P Ω [ ∥ Ω ∥ F 2 > 10 n ] ≤ 1 / 10 \mathbb{P}_{\Omega}[\|\Omega\|_F^2 > 10n] \le1/10 P Ω [ ∥Ω ∥ F 2 > 10 n ] ≤ 1/10 , and by assumption Ω \Omega Ω also satisfies the overlap condition with high probability (say with failure probability at most 1 / 10 1/10 1/10 ). Therefore, we can reduce our objective to proving the following:
For any fixed matrix S ∈ R n × k S \in\mathbb{R}^{n \times k} S ∈ R n × k with row sparsity ζ \zeta ζ and ∥ S ∥ F 2 ≤ 10 n \|S\|_F^2 \le10n ∥ S ∥ F 2 ≤ 10 n satisfying the overlap condition, prove that a random coordinate subspace R ∼ Unif ( ( [ n ] r ) ) R \sim\operatorname{Unif}\left(\binom{[n]}{r}\right) R ∼ Unif ( ( r [ n ] ) ) satisfies P [ σ min ( S R ⊤ ) 2 ≥ α ] ≤ 1 / 4 \mathbb{P}[\sigma_{\min}(S_R^\top)^2 \ge\alpha] \le1/4 P [ σ m i n ( S R ⊤ ) 2 ≥ α ] ≤ 1/4 .
Once this statement is established, adding the bad event probability and the sketch success probability on a random subspace, the overall success probability on the subspace R ∗ R^* R ∗ is at most 9 / 20 < 19 / 20 9/20 < 19/20 9/20 < 19/20 , which refutes the injectivity and completes the reduction.
In the bipartite graph representation G = ( [ n ] , [ k ] , E ) G = ([n], [k], E) G = ([ n ] , [ k ] , E ) of an n × k n \times k n × k matrix S S S , an edge connects row i i i and column j j j whenever S i j ≠ 0 S_{ij} \ne0 S ij = 0 , and two distinct rows u , v ∈ [ n ] u,v \in[n] u , v ∈ [ n ] share two or more columns in their support if and only if they participate in a 4-cycle C 4 C_4 C 4 . The overlap condition bounds the number of such row pairs sharing two or more columns by η n 2 / k \eta n^2/k η n 2 / k with overlap parameter η ≤ ( C H ζ 6 ) − 1 \eta\le(C_H\zeta^6)^{-1} η ≤ ( C H ζ 6 ) − 1 , ensuring that 4-cycles are sparse across S S S . Under our contradiction hypothesis ζ ≤ c log r / log log r \zeta\le c\sqrt{\log r}/\log\log r ζ ≤ c log r / log log r , this threshold satisfies
1 C H ζ 6 ≥ 1 C H c 6 ( log log r log r ) 3 = Ω ( ( log log r log r ) 3 ) , \frac{1}{C_H\zeta^6} \ge\frac{1}{C_H c^6}\left(\frac{\log\log r}{\log r}\right)^3 = \Omega\left(\left(\frac{\log\log r}{\log r}\right)^3\right), C H ζ 6 1 ≥ C H c 6 1 ( log r log log r ) 3 = Ω ( ( log r log log r ) 3 ) ,
so the overlap condition allows an overlap parameter η = Ω ( ( log log r / log r ) 3 ) \eta= \Omega((\log\log r/\log r)^3) η = Ω (( log log r / log r ) 3 ) that decays with r r r . This regularity ensures that the density of 4-cycles across S S S is sufficiently small that subsampling an intermediate subset of n 0 = Θ ( r ζ 6 / α ) n_0 = \Theta(r\zeta^6/\alpha) n 0 = Θ ( r ζ 6 / α ) rows.
The Tree Cancellation Mechanism.
To understand the core intuition of the HRT framework, consider the simplest illustrative setting. Suppose that every row s i s_i s i satisfies ∥ s i ∥ 2 = 1 \lVert s_i\rVert_2 = 1 ∥ s i ∥ 2 = 1 , and all non-zero entries are identically ± 1 / ζ \pm1/\sqrt{\zeta} ± 1/ ζ . Furthermore, suppose that the locations of the non-zero entries are chosen uniformly at random for each row.
Breaking injectivity on a coordinate subspace simply means that among the selected rows of S S S , there exists a non-trivial linear combination whose resulting norm is tiny. If we single out one row from this combination, the remaining rows must collectively cancel all of its non-zero entries, while simultaneously leaving negligible norm across all other coordinates. To formalize this mechanism, suppose we pick an arbitrary row as a center row s i ∗ s_{i_*} s i ∗ and wish to cancel all ζ \zeta ζ of its non-zero entries. Then for each column j a ∈ supp ( s i ∗ ) j_a \in\operatorname{supp}(s_{i_*}) j a ∈ supp ( s i ∗ ) with a ∈ [ ζ ] a \in[\zeta] a ∈ [ ζ ] , one would naturally hope to bring another row s ℓ s_\ell s ℓ sharing the same column j a j_a j a in its support and take a difference such as s i ∗ − s ℓ s_{i_*} - s_\ell s i ∗ − s ℓ . However, doing so introduces an obstacle because row s ℓ s_\ell s ℓ possesses ζ − 1 \zeta- 1 ζ − 1 other non-zero entries on columns outside j a j_a j a . We refer to any non-zero coordinate outside the specific target column intended for cancellation as an outer entry or outer column . These additional outer non-zero entries generally carry substantial norm, so simply subtracting one row per column increases the overall norm.
Instead, suppose we select q q q auxiliary rows for each column. Averaging these q q q rows preserves a large entry on the target column, while on all outer columns, their mutually disjoint supports dilute the entries by a factor of 1 / q 1/q 1/ q . Using these averaged vectors to cancel each column of the center row leaves only tiny residuals across all other coordinates. Consequently, a collection of just 1 + ζ q 1 + \zeta q 1 + ζ q rows yields a linear combination that breaks injectivity.
Formally, we select q q q rows L a = { ℓ a , 1 , … , ℓ a , q } L_a = \{\ell_{a,1}, \ldots, \ell_{a,q}\} L a = { ℓ a , 1 , … , ℓ a , q } for each column j a ∈ supp ( s i ∗ ) j_a \in\operatorname{supp}(s_{i_*}) j a ∈ supp ( s i ∗ ) with a ∈ [ ζ ] a \in[\zeta] a ∈ [ ζ ] , and define the cancellation vector :
x : = e i ∗ − ∑ a = 1 ζ ∑ ℓ ∈ L a S i ∗ , j a q S ℓ , j a e ℓ ∈ R r . (12) x := e_{i_*} - \sum_{a=1}^{\zeta}\sum_{\ell\in L_a}\frac{S_{i_*,j_a}}{qS_{\ell,j_a}}e_\ell\in\mathbb{R}^r. \tag*{(12)} x := e i ∗ − a = 1 ∑ ζ ℓ ∈ L a ∑ q S ℓ , j a S i ∗ , j a e ℓ ∈ R r . (12)
While this cancellation vector is an element of R n \mathbb{R}^n R n , we treat x x x as an element of R r \mathbb{R}^r R r : its restriction to the sampled rows indexed by R R R yields a vector in R r \mathbb{R}^r R r .
For convenience in this example, let us assume that all q ζ q\zeta q ζ leaf rows have mutually non-overlapping supports outside the center row support supp ( s i ∗ ) \operatorname{supp}(s_{i_*}) supp ( s i ∗ ) . Under this disjointness assumption, the center coordinates cancel identically via ( S R T x ) j a = S i ∗ , j a − q ⋅ S i ∗ , j a q = 0 (S_R^\mathsf{T}x)_{j_a} = S_{i_*,j_a} - q \cdot\frac{S_{i_*,j_a}}{q} = 0 ( S R T x ) j a = S i ∗ , j a − q ⋅ q S i ∗ , j a = 0 . Moreover, on outer columns c ∉ supp ( s i ∗ ) c \notin\operatorname{supp}(s_{i_*}) c ∈ / supp ( s i ∗ ) , we obtain:
∑ c ∉ supp ( s i ∗ ) ∣ ( S R T x ) c ∣ 2 = ∑ a = 1 ζ ∑ ℓ ∈ L a ( 1 q ) 2 ∑ c ≠ j a ∣ S ℓ , c ∣ 2 ≤ ∑ a = 1 ζ q ⋅ 1 q 2 ⋅ ∥ s ℓ ∥ 2 2 = ζ q . (13) \sum_{c \notin\operatorname{supp}(s_{i_*})} \left|(S_R^\mathsf{T}x)_c\right|^2 = \sum_{a=1}^{\zeta}\sum_{\ell\in L_a}\left(\frac{1}{q}\right)^2\sum_{c \ne j_a}|S_{\ell,c}|^2 \le\sum_{a=1}^{\zeta}q \cdot\frac{1}{q^2}\cdot\lVert s_\ell\rVert_2^2 = \frac{\zeta}{q}. \tag*{(13)} c ∈ / supp ( s i ∗ ) ∑ ( S R T x ) c 2 = a = 1 ∑ ζ ℓ ∈ L a ∑ ( q 1 ) 2 c = j a ∑ ∣ S ℓ , c ∣ 2 ≤ a = 1 ∑ ζ q ⋅ q 2 1 ⋅ ∥ s ℓ ∥ 2 2 = q ζ . (13)
Because ∥ x ∥ 2 2 = 1 + ζ / q ≥ 1 \lVert x\rVert_2^2 = 1 + \zeta/q \ge1 ∥ x ∥ 2 2 = 1 + ζ / q ≥ 1 , setting q = 2 ζ / α q = 2\zeta/\alpha q = 2 ζ / α yields σ min ( S R T ) 2 ≤ ∥ S R T x ∥ 2 2 / ∥ x ∥ 2 2 < α \sigma_{\min}(S_R^\mathsf{T})^2 \le\lVert S_R^\mathsf{T}x\rVert_2^2/\lVert x\rVert_2^2 < \alpha σ m i n ( S R T ) 2 ≤ ∥ S R T x ∥ 2 2 / ∥ x ∥ 2 2 < α , breaking injectivity using a single collection of 1 + ζ q = Θ ( ζ 2 / α ) 1 + \zeta q = \Theta(\zeta^2/\alpha) 1 + ζ q = Θ ( ζ 2 / α ) rows.
Bipartite Graph Modeling. We can view this cancellation structure naturally in bipartite graphs. Let G = ( [ n ] , [ k ] , E ) G = ([n],[k],E) G = ([ n ] , [ k ] , E ) be the bipartite graph between the n n n rows and k k k columns, with an edge between row i i i and column j j j whenever S i j ≠ 0 S_{ij} \ne0 S ij = 0 . The cancellation pattern above is simply a rooted tree in G G G . This tree consists of a center row, ζ \zeta ζ branch columns connected to it, and q q q leaf rows attached to each branch column, with 1 + ζ q 1+\zeta q 1 + ζ q rows in total. To make the outer cancellation work, the leaf rows must have disjoint outer supports.
Now suppose we can find T T T such vertex-disjoint trees in G G G . If a random coordinate subspace Q Q Q contains every row of even one tree, injectivity fails on Q Q Q . If the random sample R R R captures each tree with probability at least p p p , the chance of missing all T T T trees is at most ( 1 − p ) T (1-p)^T ( 1 − p ) T , which we want to show is tiny. This is precisely where the lower bound on ζ \zeta ζ originates: smaller ζ \zeta ζ shrinks each tree to fewer rows, which makes the capture probability p p p much larger.
While this tree cancellation structure resembles that of HRT, it features a crucial difference. In HRT, leaves were only required to have disjoint outer entries within the same branch. In contrast, we also require leaf rows in different branches to have disjoint outer entries. This gives a much tighter bound on the outer residual, which permits choosing a smaller value of q q q . In turn, reducing q q q increases the capture probability p p p , directly improving the lower bound.
Relative Weights and Candidate Sets. In actual matrices, entry values need not be uniform. HRT assumed that all non-zero entries have magnitude O ( 1 / ζ ) O(1/\sqrt{\zeta}) O ( 1/ ζ ) to handle this issue, but our approach completely drops this assumption. The idea is to pick leaf rows whose entries on the supp ( s i ∗ ) \operatorname{supp}(s_{i^*}) supp ( s i ∗ ) are relatively large. Specifically, for row i i i and column j j j , we define the relative weight w i j = ∣ S i j ∣ 2 / ∥ s i ∥ 2 2 w_{ij}=|S_{ij}|^2/\|s_i\|_2^2 w ij = ∣ S ij ∣ 2 /∥ s i ∥ 2 2 . For each column j j j , we sort the rows containing j j j by their relative weights w i j w_{ij} w ij and place the top D D D rows into a leaf candidate set F j F_j F j . When D D D is small enough, then some rows never appear in any candidate set, and we choose center rows exclusively from these unassigned rows.
Assuming we can find leaves whose supports do not overlap outside the center support supp ( s i ∗ ) \operatorname{supp}(s_{i^*}) supp ( s i ∗ ) , defining the cancellation vector x x x as in (12) bounds the outer entries:
( S i ∗ , j q S ℓ , j ) 2 ∑ c ≠ j S ℓ , c 2 ≤ w i ∗ , j ∥ s i ∗ ∥ 2 2 q 2 w ℓ , j ∥ s ℓ ∥ 2 2 ∥ s ℓ ∥ 2 2 = w i ∗ , j w ℓ , j ∥ s i ∗ ∥ 2 2 q 2 ≤ ∥ s i ∗ ∥ 2 2 q 2 . (14) \left(\frac{S_{i^*,j}}{qS_{\ell,j}}\right)^2\sum_{c\ne j}S_{\ell,c}^2
\le\frac{w_{i^*,j}\|s_{i^*}\|_2^2}{q^2w_{\ell,j}\|s_\ell\|_2^2}\|s_\ell\|_2^2
= \frac{w_{i^*,j}}{w_{\ell,j}}\frac{\|s_{i^*}\|_2^2}{q^2}
\le\frac{\|s_{i^*}\|_2^2}{q^2}. \tag*{(14)} ( q S ℓ , j S i ∗ , j ) 2 c = j ∑ S ℓ , c 2 ≤ q 2 w ℓ , j ∥ s ℓ ∥ 2 2 w i ∗ , j ∥ s i ∗ ∥ 2 2 ∥ s ℓ ∥ 2 2 = w ℓ , j w i ∗ , j q 2 ∥ s i ∗ ∥ 2 2 ≤ q 2 ∥ s i ∗ ∥ 2 2 . (14)
Whenever the center row satisfies ∥ s i ∗ ∥ 2 2 ≤ M \|s_{i^*}\|_2^2\le M ∥ s i ∗ ∥ 2 2 ≤ M for a constant M M M , summing this over all q q q leaves on the ζ \zeta ζ branches gives a total squared norm on the outer columns of at most ζ M / q \zeta M/q ζ M / q . Choosing q = Θ ( ζ / α ) q=\Theta(\zeta/\alpha) q = Θ ( ζ / α ) then keeps this residual squared norm below α / 4 \alpha/4 α /4 .
Graph Preprocessing and Tree Searching.
A few technical details remain. We need two main conditions: first, the center row must have norm bounded by a constant; second, we must be able to pick ζ q \zeta q ζ q leaf rows from our candidate sets F j F_j F j such that their outer entries do not overlap. To handle these issues, we first prune the graph G G G . Because our fixed matrix S S S satisfies ∑ i = 1 n ∥ s i ∥ 2 2 = ∥ S ∥ F 2 ≤ 10 n \sum_{i=1}^{n}\|s_i\|_2^2=\|S\|_F^2\le10n ∑ i = 1 n ∥ s i ∥ 2 2 = ∥ S ∥ F 2 ≤ 10 n , at most 10 n / M 10n/M 10 n / M rows can have ∥ s i ∥ 2 2 > M \|s_i\|_2^2>M ∥ s i ∥ 2 2 > M . Setting M = 160 M=160 M = 160 and simply removing these heavy rows deterministically leaves at least 15 n / 16 15n/16 15 n /16 rows in S S S .
Also, we use the overlap condition here to get a good property for a graph G G G . In G G G , two rows sharing two or more columns correspond to a 4-cycle C 4 C_4 C 4 . Suppose we initially sample n 0 = Θ ( r ⋅ poly ( ζ ) ) n_0=\Theta(r\cdot\operatorname{poly}(\zeta)) n 0 = Θ ( r ⋅ poly ( ζ )) rows from S S S and build G G G on these n 0 n_0 n 0 rows. Under the overlap condition, the full matrix has at most η n 2 / k \eta n^2/k η n 2 / k row pairs sharing two or more columns, so when sampling n 0 n_0 n 0 rows, each pair is chosen with probability roughly ( n 0 / n ) 2 (n_0/n)^2 ( n 0 / n ) 2 , which bounds the expected number of 4-cycles in G G G by η n 2 k ⋅ n 0 2 n 2 = η n 0 k ⋅ n 0 = o ( n 0 ) \eta\frac{n^2}{k}\cdot\frac{n_0^2}{n^2}=\eta\frac{n_0}{k}\cdot n_0=o(n_0) η k n 2 ⋅ n 2 n 0 2 = η k n 0 ⋅ n 0 = o ( n 0 ) . Deleting the rows in these 4-cycles removes very few vertices, leaving a C 4 C_4 C 4 -free subgraph G 0 G_0 G 0 on at least 5 n 0 / 8 5n_0/8 5 n 0 /8 rows. On this clean graph G 0 G_0 G 0 , we can now form the leaf candidate sets F j F_j F j and find a tree of size 1 + ζ q 1+\zeta q 1 + ζ q as before. Because G 0 G_0 G 0 is C 4 C_4 C 4 -free, leaves selected in the same branch automatically cannot share any outer columns.
What remains is ensuring that leaves in different branches do not collide outside the center support supp ( s i ∗ ) \operatorname{supp}(s_{i^*}) supp ( s i ∗ ) . Consider a single tree first. We want to pick q q q rows from each candidate set F j a F_{j_a} F j a , but a row could potentially appear in multiple candidate sets, so we must examine whether we can always choose fresh leaf rows. When we pick leaves branch by branch, the first a − 1 a-1 a − 1 branches take up at most q ( ζ − 1 ) ( a − 1 ) ≤ q ( ζ − 1 ) 2 q(\zeta-1)(a-1) \le q(\zeta-1)^2 q ( ζ − 1 ) ( a − 1 ) ≤ q ( ζ − 1 ) 2 outer entries. Now we choose q q q rows of a a a -th branch. Because G 0 G_0 G 0 is C 4 C_4 C 4 -free, none of the already chosen leaf rows can have a non-zero entry at column j a j_a j a , as that would mean sharing two columns with the center row, which is a contradiction. Moreover, for each outer entries occupied by leaves from the preceding a − 1 a-1 a − 1 branches, the C 4 C_4 C 4 -free condition guarantees that at most one row in F j a F_{j_a} F j a can contain that column, because If there were two or more, they would share that column in addition to j a j_a j a . Therefore, as long as D : = ∣ F j a ∣ ≥ q + q ( ζ − 1 ) 2 = : L D := |F_{j_a}| \ge q+q(\zeta-1)^2 =: L D := ∣ F j a ∣ ≥ q + q ( ζ − 1 ) 2 =: L , at most q ( ζ − 1 ) 2 q(\zeta-1)^2 q ( ζ − 1 ) 2 rows can collide with previously occupied columns, allowing us to select q q q collision-free rows. This gives a single tree whose leaves are completely disjoint on all outer columns.
Now, how do we find T T T such disjoint trees? We first select the center rows. To prevent different trees from interfering with each other, we require their supports to be completely pairwise disjoint. To do so, we conceptually consider a conflict graph whose vertices are rows, with an edge between any two rows that share at least one column. Because G 0 G_0 G 0 is C 4 C_4 C 4 -free and each row has bounded degree, the average degree in this conflict graph is small. Turán’s theorem then yields an independent set of size T = Θ ( r / poly ( ζ ) ) T=\Theta(r/\operatorname{poly}(\zeta)) T = Θ ( r / poly ( ζ )) , providing T T T center rows with completely disjoint column supports.
What remains is to find pairwise disjoint subsets of size L L L from the candidate sets F j F_j F j corresponding to all T ζ T\zeta T ζ branch columns. To establish the existence of such disjoint subsets, we apply Hall’s Marriage Theorem. We define an auxiliary bipartite assignment graph as follows: the left vertices are the branch columns across all T T T centers, each cloned L L L times to represent the L L L required reservoir slots; the right vertices are the candidate rows in ⋃ j F j \bigcup_j F_j ⋃ j F j , where each row is connected to the slots of column j j j whenever it belongs to F j F_j F j . Then every left slot has degree D D D , while every right row has degree at most ζ L \zeta L ζ L . Therefore, we require at least D ≥ ζ L D \ge\zeta L D ≥ ζ L to obtain the desired assignment, and under this condition Hall’s condition holds: for any subset of left slots A A A , counting the edges between A A A and its neighborhood N ( A ) N(A) N ( A ) gives ∣ A ∣ D = e ( A , N ( A ) ) ≤ ∣ N ( A ) ∣ ζ L |A|D = e(A,N(A)) \le|N(A)|\zeta L ∣ A ∣ D = e ( A , N ( A )) ≤ ∣ N ( A ) ∣ ζ L , which yields ∣ N ( A ) ∣ ≥ D ζ L ∣ A ∣ ≥ ∣ A ∣ |N(A)| \ge\frac{D}{\zeta L}|A| \ge|A| ∣ N ( A ) ∣ ≥ ζ L D ∣ A ∣ ≥ ∣ A ∣ . This matching assigns mutually disjoint leaf candidate sets of size L L L to all branches of all T T T centers simultaneously. Then, as in the single tree case, it gives T T T pairwise vertex-disjoint witness trees T 1 , … , T T T_1,\ldots,T_T T 1 , … , T T , each having B t r e e = 1 + ζ q = Θ ( ζ 2 / α ) B_{\mathrm{tree}}=1+\zeta q=\Theta(\zeta^2/\alpha) B tree = 1 + ζ q = Θ ( ζ 2 / α ) rows.
Tree Capture Probability. When sampling r r r rows from these n 0 n_0 n 0 rows, each row of G 0 G_0 G 0 is selected with probability roughly r / n 0 = 1 / poly ( ζ ) r/n_0=1/\operatorname{poly}(\zeta) r / n 0 = 1/ poly ( ζ ) . Because each tree has size B t r e e = 1 + q ζ = Θ ( ζ 2 / α ) B_{\mathrm{tree}}=1+q\zeta=\Theta(\zeta^2/\alpha) B tree = 1 + q ζ = Θ ( ζ 2 / α ) , the sample captures any specific tree with probability
p ≥ ( 1 poly ( ζ ) ) B t r e e = exp ( − Θ ( ζ 2 α log ζ ) ) . (15) p \ge\left(\frac{1}{\operatorname{poly}(\zeta)}\right)^{B_{\mathrm{tree}}}
= \exp\left(-\Theta\left(\frac{\zeta^2}{\alpha}\log\zeta\right)\right). \tag*{(15)} p ≥ ( poly ( ζ ) 1 ) B tree = exp ( − Θ ( α ζ 2 log ζ ) ) . (15)
Because the T T T trees are mutually vertex-disjoint, the events of capturing different trees are essentially independent. Thus the probability of missing all T T T trees simultaneously is bounded by ( 1 − p ) T ≤ exp ( − T p ) (1-p)^T \le\exp(-Tp) ( 1 − p ) T ≤ exp ( − T p ) . To ensure that at least one tree is captured with high probability, we only require T p ≥ Ω ( 1 ) Tp \ge\Omega(1) T p ≥ Ω ( 1 ) , which is equivalent to log T ≳ ζ 2 α log ζ \log T \gtrsim\frac{\zeta^2}{\alpha}\log\zeta log T ≳ α ζ 2 log ζ by our parameter setting. Since we extracted T = Θ ( r / poly ( ζ ) ) T=\Theta(r/\operatorname{poly}(\zeta)) T = Θ ( r / poly ( ζ )) center rows, we have log T = log r − O ( log ζ ) \log T=\log r-O(\log\zeta) log T = log r − O ( log ζ ) . Substituting this into the balance condition yields
ζ 2 α log ζ ≲ log r ⟹ ζ ≤ O ( log r log log r ) . (16) \frac{\zeta^2}{\alpha}\log\zeta\lesssim\log r \Longrightarrow\zeta\le O\left(\sqrt{\frac{\log r}{\log\log r}}\right). \tag*{(16)} α ζ 2 log ζ ≲ log r ⟹ ζ ≤ O ( log log r log r ) . (16)
Whenever ζ \zeta ζ falls below this threshold, a random sample captures at least one witness tree with high probability. The corresponding cancellation vector from Equation (12) then breaks injectivity with σ min ( S R T ) 2 < α \sigma_{\min}(S_R^T)^2 < \alpha σ m i n ( S R T ) 2 < α , concluding the lower bound proof.
Organization. The remainder of the paper is organized as follows:
Section 3: Preserves all flat unit vectors under the first-half sketch (Theorem 2.2).
Section 4: Extracts a common support I I I capturing all spiky vectors (Theorem 2.3).
Section 5: Bounds cross-term cancellation between I I I and I c I^c I c via coupling.
Section 6: Preserves the coordinates on I I I using the second-half sketch (Theorem 2.5).
Section 7: Combines both sketch halves to complete the proof of Theorem 1.1.
Section 8: Preprocesses the sketching matrix to construct a C 4 C_4 C 4 -free bipartite graph G 0 G_0 G 0 .
Section 9: Extracts a forest of disjoint cancellation trees in G 0 G_0 G 0 (Theorem 9.7 ).
Section 10: Concludes the lower bound proof for Theorem 1.3.
Preserving Flat Vectors
Recall from § 2 that our overall strategy splits the 2 ζ 2\zeta 2 ζ -block sketch Ω 2 ζ , b ∈ R n × 2 k \Omega_{2\zeta,b} \in\mathbb{R}^{n\times2k} Ω 2 ζ , b ∈ R n × 2 k into two independent halves X = ( Ω ζ , b ( 1 ) ) ⊤ Q ∈ R k × r X = (\Omega_{\zeta,b}^{(1)})^\top Q \in\mathbb{R}^{k\times r} X = ( Ω ζ , b ( 1 ) ) ⊤ Q ∈ R k × r and Y = ( Ω ζ , b ( 2 ) ) ⊤ Q ∈ R k × r Y = (\Omega_{\zeta,b}^{(2)})^\top Q \in\mathbb{R}^{k\times r} Y = ( Ω ζ , b ( 2 ) ) ⊤ Q ∈ R k × r of ζ = O ( log r ) \zeta= O(\sqrt{\log r}) ζ = O ( log r ) blocks and embedding dimension k = ζ b k = \zeta b k = ζ b each, and begins by showing that the first half X X X simultaneously preserves the norm of all flat unit vectors y = Q x y = Qx y = Q x for x ∈ S r − 1 x \in S^{r-1} x ∈ S r − 1 . Our goal in this section is to prove Theorem 2.2, which states that ∥ X x ∥ ≥ λ = Ω ( ε ) \|Xx\| \ge\lambda= \Omega(\varepsilon) ∥ X x ∥ ≥ λ = Ω ( ε ) simultaneously for all flat unit vectors y = Q x y = Qx y = Q x with x ∈ S r − 1 x \in S^{r-1} x ∈ S r − 1 with high probability.
We first show that for any single fixed flat vector y = Q x y = Qx y = Q x satisfying d m ( y ) ≥ ε d_m(y) \ge\varepsilon d m ( y ) ≥ ε , the sketch preserves ∥ Ω ζ , b y ∥ = Ω ( ε ) \|\Omega_{\zeta,b}y\| = \Omega(\varepsilon) ∥ Ω ζ , b y ∥ = Ω ( ε ) with exponentially small failure probability (Lemma 3.2 ). To do so, we decompose y = h + a z y = h + az y = h + a z into the vector h h h supported on its m m m largest coordinates in magnitude and its tail a z az a z supported on the remaining coordinates, where a = d m ( y ) ≥ ε a = d_m(y) \ge\varepsilon a = d m ( y ) ≥ ε and ∥ z ∥ = 1 \|z\| = 1 ∥ z ∥ = 1 . Because every coordinate of the tail a z az a z is smaller in magnitude than each of the m m m coordinates in h h h , the unit tail z z z has bounded entries ∥ z ∥ ∞ 2 ≤ 1 m ε 2 = : D − 1 \|z\|_\infty^2 \le\frac{1}{m\varepsilon^2} =: D^{-1} ∥ z ∥ ∞ 2 ≤ m ε 2 1 =: D − 1 .
We establish ∥ Ω ζ , b y ∥ ≥ Ω ( ε ) \|\Omega_{\zeta,b}y\| \ge\Omega(\varepsilon) ∥ Ω ζ , b y ∥ ≥ Ω ( ε ) by revealing the random column indices { s i t } \{s_{it}\} { s i t } and the random signs { ϱ i t } \{\varrho_{it}\} { ϱ i t } in turn. First, in each block t ∈ [ ζ ] t \in[\zeta] t ∈ [ ζ ] , the column indices partition the unit squared norm ∑ i = 1 n z i 2 = 1 \sum_{i=1}^n z_i^2 = 1 ∑ i = 1 n z i 2 = 1 among the b b b buckets, so the total unnormalized tail weight summed across all ζ \zeta ζ blocks is ζ \zeta ζ . We call a bucket u u u is light if it receives at most v max : = 20 / D v_{\max} := 20/D v m a x := 20/ D of z z z 's squared norm (i.e., for a block t t t , ∑ i : s i t = u z i 2 ≤ 20 / D \sum_{i:s_{it}=u} z_i^2 \le20/D ∑ i : s i t = u z i 2 ≤ 20/ D ), and heavy otherwise.
Here, we show that with probability at least 1 − e − Ω ( ζ D ) 1-e^{-\Omega(\zeta D)} 1 − e − Ω ( ζ D ) , the vast majority of the tail weight (≥ 7 8 ζ \ge\frac{7}{8}\zeta ≥ 8 7 ζ ) is contained in light buckets. We can prove that only a small fraction of the total weight ζ \zeta ζ lands in heavy buckets in expectation. Then, we prove that for each light bucket u u u in block t t t , its unnormalized squared entry ( ∑ i : s i t = u ϱ i t y i ) 2 \left(\sum_{i:s_{it}=u}\varrho_{it}y_i\right)^2 ( ∑ i : s i t = u ϱ i t y i ) 2 retains at least Ω ( a 2 ∑ i : s i t = u z i 2 ) \Omega\left(a^2\sum_{i:s_{it}=u}z_i^2\right) Ω ( a 2 ∑ i : s i t = u z i 2 ) with constant probability p 0 ≥ 4 / 27 p_0 \ge4/27 p 0 ≥ 4/27 . Because each light bucket has small weight ≤ 20 / D \le20/D ≤ 20/ D , summing these independent contributions across all light buckets gives Ω ( a 2 ζ ) \Omega(a^2\zeta) Ω ( a 2 ζ ) with failure probability e − Ω ( ζ D ) e^{-\Omega(\zeta D)} e − Ω ( ζ D ) , yielding ∥ Ω ζ , b y ∥ ≥ Ω ( a ) ≥ Ω ( ε ) \|\Omega_{\zeta,b}y\| \ge\Omega(a) \ge\Omega(\varepsilon) ∥ Ω ζ , b y ∥ ≥ Ω ( a ) ≥ Ω ( ε ) .
Fact 3.1 (McDiarmid’s bounded-differences inequality [23 ] ). If f f f is a function of independent random variables Z 1 , … , Z N Z_1,\ldots,Z_N Z 1 , … , Z N such that changing Z i Z_i Z i changes the value of f f f by at most c i c_i c i , then for every t ≥ 0 t \ge0 t ≥ 0 ,
P [ ∣ f − E [ f ] ∣ ≥ t ] ≤ exp ( − 2 t 2 ∑ i = 1 N c i 2 ) . (17) \mathbb{P}\left[|f-\mathbb{E}[f]| \ge t\right] \le\exp\left(-\frac{2t^2}{\sum_{i=1}^{N}c_i^2}\right). \tag*{(17)} P [ ∣ f − E [ f ] ∣ ≥ t ] ≤ exp ( − ∑ i = 1 N c i 2 2 t 2 ) . (17)
Lemma 3.2. There exist universal constants c 0 , c s b > 0 c_0,c_{\mathrm{sb}} > 0 c 0 , c sb > 0 , such that for a given vector y = h + a z y=h+az y = h + a z , where h , z ∈ R n h,z \in\mathbb{R}^n h , z ∈ R n have disjoint supports, a > 0 a>0 a > 0 , ∥ z ∥ = 1 \|z\|=1 ∥ z ∥ = 1 , and
1 ≤ D ≤ b , ∥ z ∥ ∞ 2 ≤ D − 1 , 1 \le D \le b,\qquad\|z\|_{\infty}^2 \le D^{-1}, 1 ≤ D ≤ b , ∥ z ∥ ∞ 2 ≤ D − 1 , a SparseStack sketch Ω ζ , b ∈ R n × k \Omega_{\zeta,b} \in\mathbb{R}^{n\times k} Ω ζ , b ∈ R n × k with k = ζ b k=\zeta b k = ζ b satisfies
P [ ∥ Ω ζ , b y ∥ < c 0 a ] ≤ 2 e − c s b ζ D . \mathbb{P}\left[\|\Omega_{\zeta,b}y\| < c_0a\right] \le2e^{-c_{\mathrm{sb}}\zeta D}. P [ ∥ Ω ζ , b y ∥ < c 0 a ] ≤ 2 e − c sb ζ D .
Proof. Write p i = z i 2 p_i=z_i^2 p i = z i 2 , so that ∑ i = 1 n p i = 1 \sum_{i=1}^{n}p_i=1 ∑ i = 1 n p i = 1 and ∑ i = 1 n p i 2 ≤ max i p i ≤ 1 / D \sum_{i=1}^{n}p_i^2\le\max_i p_i\le1/D ∑ i = 1 n p i 2 ≤ max i p i ≤ 1/ D . For each block t ∈ [ ζ ] t\in[\zeta] t ∈ [ ζ ] and bucket u ∈ [ b ] u\in[b] u ∈ [ b ] , define the bucket tail weight W t u W_{tu} W t u and the excess weight L \mathcal{L} L by
W t u = ∑ i : s i t = u p i , L = ∑ t = 1 ζ ∑ u = 1 b max { W t u − 10 D , 0 } . W_{tu}=\sum_{i:s_{it}=u}p_i,\qquad\mathcal{L}=\sum_{t=1}^{\zeta}\sum_{u=1}^{b}\max\left\{W_{tu}-\frac{10}{D},0\right\}. W t u = i : s i t = u ∑ p i , L = t = 1 ∑ ζ u = 1 ∑ b max { W t u − D 10 , 0 } .
For a fixed block t t t , expanding W t u 2 W_{tu}^2 W t u 2 and using independence of the column indices { s i t } i = 1 n \{s_{it}\}_{i=1}^{n} { s i t } i = 1 n gives
E [ ∑ u = 1 b W t u 2 ] = ∑ i = 1 n p i 2 + 1 b ( 1 − ∑ i = 1 n p i 2 ) ≤ 1 D + 1 b ≤ 2 D . \mathbb{E}\left[\sum_{u=1}^{b}W_{tu}^2\right]=\sum_{i=1}^{n}p_i^2+\frac{1}{b}\left(1-\sum_{i=1}^{n}p_i^2\right)\le\frac{1}{D}+\frac{1}{b}\le\frac{2}{D}. E [ u = 1 ∑ b W t u 2 ] = i = 1 ∑ n p i 2 + b 1 ( 1 − i = 1 ∑ n p i 2 ) ≤ D 1 + b 1 ≤ D 2 .
Using the elementary pointwise inequality max { x − A , 0 } ≤ x 2 / ( 4 A ) \max\{x-A,0\}\le x^2/(4A) max { x − A , 0 } ≤ x 2 / ( 4 A ) for x ≥ 0 x\ge0 x ≥ 0 and A = 10 / D > 0 A=10/D>0 A = 10/ D > 0 , we obtain
E [ L ] ≤ D 40 ∑ t = 1 ζ E [ ∑ u = 1 b W t u 2 ] ≤ ζ 20 . \mathbb{E}[\mathcal{L}]\le\frac{D}{40}\sum_{t=1}^{\zeta}\mathbb{E}\left[\sum_{u=1}^{b}W_{tu}^2\right]\le\frac{\zeta}{20}. E [ L ] ≤ 40 D t = 1 ∑ ζ E [ u = 1 ∑ b W t u 2 ] ≤ 20 ζ .
Observe that changing a single column index s i t s_{it} s i t moves weight p i p_i p i from one bucket in block t t t to another, which changes L \mathcal{L} L by at most p i p_i p i . With the fact that ∑ t = 1 ζ ∑ i = 1 n p i 2 ≤ ζ / D \sum_{t=1}^{\zeta}\sum_{i=1}^{n}p_i^2\le\zeta/D ∑ t = 1 ζ ∑ i = 1 n p i 2 ≤ ζ / D , applying Fact 3.1 (17) gives
P [ L > ζ 16 ] ≤ exp ( − 2 ( 1 16 − 1 20 ) 2 ζ D ) ≤ e − c s b ζ D \mathbb{P}\left[\mathcal{L}>\frac{\zeta}{16}\right]\le\exp\left(-2\left(\frac{1}{16}-\frac{1}{20}\right)^2\zeta D\right)\le e^{-c_{\mathrm{sb}}\zeta D} P [ L > 16 ζ ] ≤ exp ( − 2 ( 16 1 − 20 1 ) 2 ζ D ) ≤ e − c sb ζ D
for a universal constant c s b > 0 c_{\mathrm{sb}}>0 c sb > 0 . On the event { L ≤ ζ / 16 } \{\mathcal{L}\le\zeta/16\} { L ≤ ζ /16 } , since W t u ≤ 2 max { W t u − 10 / D , 0 } W_{tu}\le2\max\{W_{tu}-10/D,0\} W t u ≤ 2 max { W t u − 10/ D , 0 } whenever W t u > 20 / D W_{tu}>20/D W t u > 20/ D , the heavy buckets with W t u > 20 / D W_{tu}>20/D W t u > 20/ D carry total tail weight at most 2 L ≤ ζ / 8 2\mathcal{L}\le\zeta/8 2 L ≤ ζ /8 . For light bucket j = ( t , u ) j=(t,u) j = ( t , u ) , let its tail weight be v j = W t u v_j=W_{tu} v j = W t u . Thus, if we retain only the light buckets ( t , u ) (t,u) ( t , u ) satisfying W t u ≤ 20 / D W_{tu}\le20/D W t u ≤ 20/ D , their total tail weight and maximum bucket weight v max v_{\max} v m a x satisfy
∑ j v j ≥ ζ − ζ 8 = 7 ζ 8 , v max ≤ 20 D . \sum_j v_j\ge\zeta-\frac{\zeta}{8}=\frac{7\zeta}{8},\qquad v_{\max}\le\frac{20}{D}. j ∑ v j ≥ ζ − 8 ζ = 8 7 ζ , v m a x ≤ D 20 .
Now condition on all column indices { s i t } \{s_{it}\} { s i t } on the event { L ≤ ζ / 16 } \{\mathcal{L} \le\zeta/16\} { L ≤ ζ /16 } and on all signs { ϱ i t } i ∈ supp ( h ) \{\varrho_{it}\}_{i\in\operatorname{supp}(h)} { ϱ i t } i ∈ supp ( h ) in the support of h h h . Because supp ( h ) ∩ supp ( z ) = ∅ \operatorname{supp}(h)\cap\operatorname{supp}(z)=\varnothing supp ( h ) ∩ supp ( z ) = ∅ , the signs { ϱ i t } i ∈ supp ( z ) \{\varrho_{it}\}_{i\in\operatorname{supp}(z)} { ϱ i t } i ∈ supp ( z ) remain independent and uniform on { − 1 , 1 } \{-1,1\} { − 1 , 1 } . For a light bucket j = ( t , u ) j=(t,u) j = ( t , u ) , its unnormalized output entry is h j + X 0 , j h_j+X_{0,j} h j + X 0 , j , where h j h_j h j is the deterministic contribution from h h h and X 0 , j = a ∑ i ∈ supp ( z ) : s i t = u ϱ i t z i X_{0,j}=a\sum_{i\in\operatorname{supp}(z):s_{it}=u}\varrho_{it}z_i X 0 , j = a ∑ i ∈ supp ( z ) : s i t = u ϱ i t z i is a Rademacher sum of variance a 2 v j a^2v_j a 2 v j . By fourth-moment expansion for Rademacher sums, E [ X 0 , j 2 ] = a 2 v j \mathbb{E}[X_{0,j}^2]=a^2v_j E [ X 0 , j 2 ] = a 2 v j and E [ X 0 , j 4 ] ≤ 3 a 4 v j 2 \mathbb{E}[X_{0,j}^4]\leq3a^4v_j^2 E [ X 0 , j 4 ] ≤ 3 a 4 v j 2 , so the non-negative random variable Z j = ( h j + X 0 , j ) 2 Z_j=(h_j+X_{0,j})^2 Z j = ( h j + X 0 , j ) 2 satisfies
E [ Z j ] = h j 2 + a 2 v j ≥ a 2 v j , E [ Z j 2 ] = h j 4 + 6 h j 2 a 2 v j + E [ X 0 , j 4 ] ≤ 3 ( h j 2 + a 2 v j ) 2 = 3 ( E [ Z j ] ) 2 . \mathbb{E}[Z_j]=h_j^2+a^2v_j\geq a^2v_j,\qquad\mathbb{E}[Z_j^2]=h_j^4+6h_j^2a^2v_j+\mathbb{E}[X_{0,j}^4]\leq3\left(h_j^2+a^2v_j\right)^2=3\left(\mathbb{E}[Z_j]\right)^2. E [ Z j ] = h j 2 + a 2 v j ≥ a 2 v j , E [ Z j 2 ] = h j 4 + 6 h j 2 a 2 v j + E [ X 0 , j 4 ] ≤ 3 ( h j 2 + a 2 v j ) 2 = 3 ( E [ Z j ] ) 2 .
Decomposing Z j = Z j 1 { Z j ≤ 1 3 E [ Z j ] } + Z j 1 { Z j ≥ 1 3 E [ Z j ] } Z_j=Z_j\mathbf{1}_{\{Z_j\leq\frac{1}{3}\mathbb{E}[Z_j]\}}+Z_j\mathbf{1}_{\{Z_j\geq\frac{1}{3}\mathbb{E}[Z_j]\}} Z j = Z j 1 { Z j ≤ 3 1 E [ Z j ]} + Z j 1 { Z j ≥ 3 1 E [ Z j ]} and applying Cauchy–Schwarz to the second term gives
E [ Z j ] ≤ 1 3 E [ Z j ] + E [ Z j 2 ] P [ Z j ≥ 1 3 E [ Z j ] ] , \mathbb{E}[Z_j]\leq\frac{1}{3}\mathbb{E}[Z_j]+\sqrt{\mathbb{E}\left[Z_j^2\right]}\sqrt{\mathbb{P}\left[Z_j\geq\frac{1}{3}\mathbb{E}[Z_j]\right]}, E [ Z j ] ≤ 3 1 E [ Z j ] + E [ Z j 2 ] P [ Z j ≥ 3 1 E [ Z j ] ] ,
so rearranging and using E [ Z j 2 ] ≤ 3 ( E [ Z j ] ) 2 \mathbb{E}[Z_j^2]\leq3(\mathbb{E}[Z_j])^2 E [ Z j 2 ] ≤ 3 ( E [ Z j ] ) 2 and 1 3 E [ Z j ] ≥ a 2 v j / 3 \frac{1}{3}\mathbb{E}[Z_j]\geq a^2v_j/3 3 1 E [ Z j ] ≥ a 2 v j /3 yields
P [ ( h j + X 0 , j ) 2 ≥ a 2 v j 3 ] ≥ P [ Z j ≥ 1 3 E [ Z j ] ] ≥ 4 9 ( E [ Z j ] ) 2 E [ Z j 2 ] ≥ 4 27 . \mathbb{P}\left[(h_j+X_{0,j})^2\geq\frac{a^2v_j}{3}\right]\geq\mathbb{P}\left[Z_j\geq\frac{1}{3}\mathbb{E}[Z_j]\right]\geq\frac{4}{9}\frac{(\mathbb{E}[Z_j])^2}{\mathbb{E}[Z_j^2]}\geq\frac{4}{27}. P [ ( h j + X 0 , j ) 2 ≥ 3 a 2 v j ] ≥ P [ Z j ≥ 3 1 E [ Z j ] ] ≥ 9 4 E [ Z j 2 ] ( E [ Z j ] ) 2 ≥ 27 4 .
Furthermore, since distinct buckets ( t , u ) (t,u) ( t , u ) within each block receive disjoint sets of z z z -coordinates and the signs across blocks are independent, the indicators B j = 1 { ( h j + X 0 , j ) 2 ≥ a 2 v j / 3 } B_j=\mathbf{1}_{\{(h_j+X_{0,j})^2\geq a^2v_j/3\}} B j = 1 {( h j + X 0 , j ) 2 ≥ a 2 v j /3 } are mutually independent across all light buckets j j j , with P [ B j = 1 ] ≥ p 0 : = 4 / 27 \mathbb{P}[B_j=1]\geq p_0:=4/27 P [ B j = 1 ] ≥ p 0 := 4/27 .
To establish that the sum of these independent contributions concentrates around its expectation, we bound the lower tail of the weighted sum ∑ j v j B j \sum_jv_jB_j ∑ j v j B j . For each light bucket j j j , normalize its weight by setting u j = v j / v max ∈ [ 0 , 1 ] u_j=v_j/v_{\max}\in[0,1] u j = v j / v m a x ∈ [ 0 , 1 ] . Since B j ∈ { 0 , 1 } B_j\in\{0,1\} B j ∈ { 0 , 1 } with P [ B j = 1 ∣ { s i t } , ϱ supp ( h ) ] ≥ p 0 \mathbb{P}[B_j=1\mid\{s_{it}\},\varrho_{\operatorname{supp}(h)}]\geq p_0 P [ B j = 1 ∣ { s i t } , ϱ supp ( h ) ] ≥ p 0 , we have:
E [ exp ( − u j B j ) ∣ { s i t } , ϱ supp ( h ) ] = 1 − P [ B j = 1 ] ( 1 − e − u j ) ≤ 1 − p 0 ( 1 − e − u j ) . \mathbb{E}\left[\exp(-u_jB_j)\mid\{s_{it}\},\varrho_{\operatorname{supp}(h)}\right]=1-\mathbb{P}[B_j=1](1-e^{-u_j})\leq1-p_0(1-e^{-u_j}). E [ exp ( − u j B j ) ∣ { s i t } , ϱ supp ( h ) ] = 1 − P [ B j = 1 ] ( 1 − e − u j ) ≤ 1 − p 0 ( 1 − e − u j ) .
Using the concavity of f ( u ) = 1 − e − u f(u)=1-e^{-u} f ( u ) = 1 − e − u , 1 − e − u ≥ ( 1 − e − 1 ) u 1-e^{-u}\geq(1-e^{-1})u 1 − e − u ≥ ( 1 − e − 1 ) u holds for all u ∈ [ 0 , 1 ] u\in[0,1] u ∈ [ 0 , 1 ] . Then we have:
E [ exp ( − u j B j ) ∣ { s i t } , ϱ supp ( h ) ] ≤ 1 − p 0 ( 1 − e − 1 ) u j ≤ exp ( − p 0 ( 1 − e − 1 ) u j ) . \mathbb{E}\left[\exp(-u_jB_j)\mid\{s_{it}\},\varrho_{\operatorname{supp}(h)}\right]\leq1-p_0(1-e^{-1})u_j\leq\exp\left(-p_0(1-e^{-1})u_j\right). E [ exp ( − u j B j ) ∣ { s i t } , ϱ supp ( h ) ] ≤ 1 − p 0 ( 1 − e − 1 ) u j ≤ exp ( − p 0 ( 1 − e − 1 ) u j ) .
By conditional mutual independence of the indicators { B j } \{B_j\} { B j } across all light buckets, the joint moment-generating function factorizes as
E [ exp ( − ∑ j v j B j v max ) | { s i t } , ϱ supp ( h ) ] = ∏ j E [ exp ( − u j B j ) ∣ { s i t } , ϱ supp ( h ) ] ≤ ∏ j exp ( − p 0 ( 1 − e − 1 ) u j ) = exp ( − 2 c 1 ∑ j v j v max ) , \begin{aligned}
\mathbb{E}\left[\exp\left(-\sum_j\frac{v_jB_j}{v_{\max}}\right)\,\middle|\,\{s_{it}\},\varrho_{\operatorname{supp}(h)}\right]=\prod_j\mathbb{E}\left[\exp(-u_jB_j)\mid\{s_{it}\},\varrho_{\operatorname{supp}(h)}\right] \\
&\leq\prod_j\exp\left(-p_0(1-e^{-1})u_j\right)=\exp\left(-2c_1\frac{\sum_jv_j}{v_{\max}}\right),
\end{aligned} E [ exp ( − j ∑ v m a x v j B j ) { s i t } , ϱ supp ( h ) ] = j ∏ E [ exp ( − u j B j ) ∣ { s i t } , ϱ supp ( h ) ] ≤ j ∏ exp ( − p 0 ( 1 − e − 1 ) u j ) = exp ( − 2 c 1 v m a x ∑ j v j ) ,
where we set c 1 = 1 2 p 0 ( 1 − e − 1 ) > 0 c_1=\frac{1}{2}p_0(1-e^{-1})>0 c 1 = 2 1 p 0 ( 1 − e − 1 ) > 0 . We now apply Markov’s inequality to the non-negative random variable exp ( − ∑ j v j B j / v max ) \exp(-\sum_j v_j B_j/v_{\max}) exp ( − ∑ j v j B j / v m a x ) . We obtain
P [ ∑ j v j B j < c 1 ∑ j v j | { s i t } , ϱ s u p p ( h ) ] ≤ E [ exp ( − ∑ j v j B j v max ) ∣ { s i t } , ϱ s u p p ( h ) ] exp ( − c 1 ∑ j v j v max ) ≤ exp ( − c 1 ∑ j v j v max ) . \begin{aligned}
\mathbb{P}\left[\sum_j v_j B_j < c_1\sum_j v_j \,\middle|\, \{s_{it}\},\varrho_{\mathrm{supp}(h)}\right] \leq\frac{\mathbb{E}\left[\left.\exp\left(-\sum_j \frac{v_jB_j}{v_{\max}}\right)\,\right|\,\{s_{it}\},\varrho_{\mathrm{supp}(h)}\right]}{\exp\left(-c_1\frac{\sum_jv_j}{v_{\max}}\right)} \\
&\leq\exp\left(-c_1\frac{\sum_jv_j}{v_{\max}}\right).
\end{aligned} P [ j ∑ v j B j < c 1 j ∑ v j { s i t } , ϱ supp ( h ) ] ≤ exp ( − c 1 v m a x ∑ j v j ) E [ exp ( − ∑ j v m a x v j B j ) { s i t } , ϱ supp ( h ) ] ≤ exp ( − c 1 v m a x ∑ j v j ) .
Recalling that ∑ j v j ≥ 7 8 ζ \sum_j v_j \geq\frac{7}{8}\zeta ∑ j v j ≥ 8 7 ζ and v max ≤ 20 D v_{\max} \leq\frac{20}{D} v m a x ≤ D 20 on the event { L ≤ ζ / 16 } \{\mathcal{L}\leq\zeta/16\} { L ≤ ζ /16 } , the exponent is bounded by
c 1 ∑ j v j v max ≥ c 1 7 ζ / 8 20 / D = 7 c 1 160 ζ D ≥ c s b ζ D , c_1\frac{\sum_jv_j}{v_{\max}} \geq c_1\frac{7\zeta/8}{20/D}=\frac{7c_1}{160}\zeta D\geq c_{\mathrm{sb}}\zeta D, c 1 v m a x ∑ j v j ≥ c 1 20/ D 7 ζ /8 = 160 7 c 1 ζ D ≥ c sb ζ D ,
with c s b : = min ( 1 3200 , 7 c 1 160 ) c_{\mathrm{sb}}:=\min\left(\frac{1}{3200},\frac{7c_1}{160}\right) c sb := min ( 3200 1 , 160 7 c 1 ) . Thus,
P [ ∑ j v j B j < c 1 ∑ j v j | { s i t } , ϱ s u p p ( h ) ] ≤ e − c s b ζ D . \mathbb{P}\left[\sum_jv_jB_j<c_1\sum_jv_j\,\middle|\,\{s_{it}\},\varrho_{\mathrm{supp}(h)}\right]\leq e^{-c_{\mathrm{sb}}\zeta D}. P [ j ∑ v j B j < c 1 j ∑ v j { s i t } , ϱ supp ( h ) ] ≤ e − c sb ζ D .
Outside this conditional failure event, we lower bound the squared Euclidean norm of Ω ζ , b ⊤ y \Omega_{\zeta,b}^{\top}y Ω ζ , b ⊤ y . By the normalization factor ζ − 1 / 2 \zeta^{-1/2} ζ − 1/2 on each block, the squared norm is the sum of squared bucket entries across all ζ \zeta ζ blocks. Retaining only the non-negative contributions from the light buckets and applying the lower bound ( h j + X 0 , j ) 2 ≥ a 2 v j 3 B j (h_j+X_{0,j})^2\geq\frac{a^2v_j}{3}B_j ( h j + X 0 , j ) 2 ≥ 3 a 2 v j B j gives
∥ Ω ζ , b ⊤ y ∥ 2 = 1 ζ ∑ t = 1 ζ ∑ u = 1 b ( h t u + X 0 , t u ) 2 ≥ 1 ζ ∑ j ∈ l i g h t ( h j + X 0 , j ) 2 ≥ 1 ζ ∑ j a 2 v j 3 B j ≥ a 2 3 ζ c 1 ∑ j v j ≥ a 2 3 ζ c 1 ( 7 8 ζ ) ≥ c 0 2 a 2 , \begin{aligned}
\left\|\Omega_{\zeta,b}^{\top}y\right\|^2=\frac{1}{\zeta}\sum_{t=1}^{\zeta}\sum_{u=1}^{b}(h_{tu}+X_{0,tu})^2\geq\frac{1}{\zeta}\sum_{j\in\mathrm{light}}(h_j+X_{0,j})^2\geq\frac{1}{\zeta}\sum_j\frac{a^2v_j}{3}B_j \\
&\geq\frac{a^2}{3\zeta}c_1\sum_jv_j\geq\frac{a^2}{3\zeta}c_1\left(\frac{7}{8}\zeta\right)\geq c_0^2a^2,
\end{aligned} Ω ζ , b ⊤ y 2 = ζ 1 t = 1 ∑ ζ u = 1 ∑ b ( h t u + X 0 , t u ) 2 ≥ ζ 1 j ∈ light ∑ ( h j + X 0 , j ) 2 ≥ ζ 1 j ∑ 3 a 2 v j B j ≥ 3 ζ a 2 c 1 j ∑ v j ≥ 3 ζ a 2 c 1 ( 8 7 ζ ) ≥ c 0 2 a 2 ,
where c 0 = c 1 / 4 > 0 c_0=\sqrt{c_1/4}>0 c 0 = c 1 /4 > 0 . Taking square roots yields ∥ Ω ζ , b ⊤ y ∥ ≥ c 0 a \|\Omega_{\zeta,b}^{\top}y\|\geq c_0a ∥ Ω ζ , b ⊤ y ∥ ≥ c 0 a .
Finally, by a union bound, the event ∥ Ω ζ , b ⊤ y ∥ < c 0 a \|\Omega_{\zeta,b}^{\top}y\|<c_0a ∥ Ω ζ , b ⊤ y ∥ < c 0 a can occur only if either the column indices fail to allocate enough weight to light buckets (L > ζ / 16 \mathcal{L}>\zeta/16 L > ζ /16 , which occurs with probability at most e − c s b ζ D e^{-c_{\mathrm{sb}}\zeta D} e − c sb ζ D ) or the conditional sign realization fails (∑ j v j B j < c 1 ∑ j v j \sum_jv_jB_j<c_1\sum_jv_j ∑ j v j B j < c 1 ∑ j v j , which occurs with conditional probability at most e − c s b ζ D e^{-c_{\mathrm{sb}}\zeta D} e − c sb ζ D ). The total failure probability is therefore at most
P [ ∥ Ω ζ , b ⊤ y ∥ < c 0 a ] ≤ e − c s b ζ D + e − c s b ζ D = 2 e − c s b ζ D , \mathbb{P}\left[\left\|\Omega_{\zeta,b}^{\top}y\right\|<c_0a\right]\leq e^{-c_{\mathrm{sb}}\zeta D}+e^{-c_{\mathrm{sb}}\zeta D}=2e^{-c_{\mathrm{sb}}\zeta D}, P [ Ω ζ , b ⊤ y < c 0 a ] ≤ e − c sb ζ D + e − c sb ζ D = 2 e − c sb ζ D ,
which completes the proof.
Having shown in Lemma 3.2 that Ω ζ , b ⊤ \Omega_{\zeta,b}^{\top} Ω ζ , b ⊤ preserves the norm of any single flat vector with failure probability e − Ω ( ζ D ) = e − Ω ( r ) e^{-\Omega(\zeta D)}=e^{-\Omega(r)} e − Ω ( ζ D ) = e − Ω ( r ) when m = Θ ( r / ζ ) m=\Theta(r/\zeta) m = Θ ( r / ζ ) and k = ζ b = Θ ( r ) k=\zeta b=\Theta(r) k = ζ b = Θ ( r ) , we now turn to the second step: extending this pointwise lower bound simultaneously to all flat unit vectors y = Q x y=Qx y = Q x (x ∈ S r − 1 x\in S^{r-1} x ∈ S r − 1 ).
Lemma 3.3 constructs a deterministic finite grid N η \mathcal{N}_{\eta} N η of size at most ( 16 / η ) r (16/\eta)^r ( 16/ η ) r such that for any matrix T T T with r r r columns and any unit vector x ∈ S r − 1 x\in S^{r-1} x ∈ S r − 1 , randomly rounding each coordinate of x x x in an unbiased manner to an adjacent multiple of h = η / r h=\eta/\sqrt{r} h = η / r guarantees the existence of a grid point y ∈ N η y\in\mathcal{N}_{\eta} y ∈ N η satisfying both ∥ x − y ∥ ≤ η \|x-y\|\leq\eta ∥ x − y ∥ ≤ η and ∥ T ( x − y ) ∥ 2 ≤ η 2 4 r ∥ T ∥ F 2 \|T(x-y)\|^2\leq\frac{\eta^2}{4r}\|T\|_F^2 ∥ T ( x − y ) ∥ 2 ≤ 4 r η 2 ∥ T ∥ F 2 .
Lemma 3.3. For every 0 < η < 1 0 < \eta< 1 0 < η < 1 , there exists a deterministic finite set N η ⊂ ( 1 + η ) B 2 r N_{\eta} \subset(1+\eta)B_{2}^{r} N η ⊂ ( 1 + η ) B 2 r of cardinality
∣ N η ∣ ≤ ( 16 η ) r . |N_{\eta}| \le\left(\frac{16}{\eta}\right)^{r}. ∣ N η ∣ ≤ ( η 16 ) r . such that, for every real matrix T T T with r r r columns and every unit vector x ∈ S r − 1 x \in S^{r-1} x ∈ S r − 1 , there is some y ∈ N η y \in N_{\eta} y ∈ N η satisfying
∥ x − y ∥ ≤ η , ∥ T ( x − y ) ∥ 2 2 ≤ η 2 4 r ∥ T ∥ F 2 . \lVert x-y\rVert\le\eta,\qquad\lVert T(x-y)\rVert_{2}^{2} \le\frac{\eta^{2}}{4r}\lVert T\rVert_{F}^{2}. ∥ x − y ∥ ≤ η , ∥ T ( x − y ) ∥ 2 2 ≤ 4 r η 2 ∥ T ∥ F 2 .
Proof. Set the grid spacing to be h = η / r h=\eta/\sqrt{r} h = η / r . Given any unit vector x ∈ S r − 1 x\in S^{r-1} x ∈ S r − 1 , we construct a random lattice point y ∈ h Z r y\in h\mathbb{Z}^{r} y ∈ h Z r by independently rounding each coordinate x k x_k x k for 1 ≤ k ≤ r 1\le k\le r 1 ≤ k ≤ r to one of its two neighboring multiples of h h h , choosing the rounding probabilities so that E [ y k ] = x k \mathbb{E}[y_k]=x_k E [ y k ] = x k . Since ∣ x k − y k ∣ ≤ h |x_k-y_k|\le h ∣ x k − y k ∣ ≤ h for every coordinate, every realization of y y y satisfies
∥ x − y ∥ ≤ r h 2 = η \lVert x-y\rVert\le\sqrt{r h^{2}}=\eta ∥ x − y ∥ ≤ r h 2 = η
and hence
∥ y ∥ ≤ 1 + η . \lVert y\rVert\le1+\eta. ∥ y ∥ ≤ 1 + η .
Moreover, the coordinate rounding errors x k − y k x_k-y_k x k − y k are independent, mean-zero, and supported in intervals of length h h h , so each coordinate error has variance E [ ( x k − y k ) 2 ] ≤ h 2 / 4 \mathbb{E}[(x_k-y_k)^2]\le h^2/4 E [( x k − y k ) 2 ] ≤ h 2 /4 . Consequently, for any deterministic matrix T T T with columns t 1 , … , t r t_1,\ldots,t_r t 1 , … , t r ,
E [ ∥ T ( x − y ) ∥ 2 ] = ∑ k = 1 r E [ ( x k − y k ) 2 ] ∥ t k ∥ 2 ≤ h 2 4 ∑ k = 1 r ∥ t k ∥ 2 = η 2 4 r ∥ T ∥ F 2 . \mathbb{E}\left[\lVert T(x-y)\rVert^2\right]
=\sum_{k=1}^{r}\mathbb{E}\left[(x_k-y_k)^2\right]\lVert t_k\rVert^2
\le\frac{h^2}{4}\sum_{k=1}^{r}\lVert t_k\rVert^2
=\frac{\eta^2}{4r}\lVert T\rVert_F^2. E [ ∥ T ( x − y ) ∥ 2 ] = k = 1 ∑ r E [ ( x k − y k ) 2 ] ∥ t k ∥ 2 ≤ 4 h 2 k = 1 ∑ r ∥ t k ∥ 2 = 4 r η 2 ∥ T ∥ F 2 .
By the probabilistic method, at least one realization y y y in the deterministic finite lattice set N η = h Z r ∩ ( 1 + η ) B 2 r N_{\eta}=h\mathbb{Z}^{r}\cap(1+\eta)B_{2}^{r} N η = h Z r ∩ ( 1 + η ) B 2 r achieves ∥ T ( x − y ) ∥ 2 ≤ η 2 4 r ∥ T ∥ F 2 \lVert T(x-y)\rVert^2\le\frac{\eta^2}{4r}\lVert T\rVert_F^2 ∥ T ( x − y ) ∥ 2 ≤ 4 r η 2 ∥ T ∥ F 2 .
To bound the cardinality ∣ N η ∣ |N_{\eta}| ∣ N η ∣ , note that the open hypercubes of side length h h h centered at the points of N η N_{\eta} N η are pairwise disjoint and have Euclidean diameter r h = η \sqrt{r}h=\eta r h = η , so they are all contained in the Euclidean ball ( 1 + 3 η / 2 ) B 2 r (1+3\eta/2)B_{2}^{r} ( 1 + 3 η /2 ) B 2 r . Using the standard formula for the volume of an r r r -dimensional Euclidean ball,
vol ( B 2 r ) = π r / 2 Γ ( r / 2 + 1 ) ≤ ( 2 π e r ) r / 2 < ( 5 r ) r . \operatorname{vol}(B_{2}^{r})=\frac{\pi^{r/2}}{\Gamma(r/2+1)}
\le\left(\frac{2\pi e}{r}\right)^{r/2}
<\left(\frac{5}{\sqrt{r}}\right)^{r}. vol ( B 2 r ) = Γ ( r /2 + 1 ) π r /2 ≤ ( r 2 π e ) r /2 < ( r 5 ) r .
By volume comparison,
∣ N η ∣ ≤ vol ( ( 1 + 3 η / 2 ) B 2 r ) h r < ( 5 ( 1 + 3 η / 2 ) η ) r ≤ ( 16 η ) r , |N_{\eta}|
\le\frac{\operatorname{vol}((1+3\eta/2)B_{2}^{r})}{h^{r}}
<\left(\frac{5(1+3\eta/2)}{\eta}\right)^{r}
\le\left(\frac{16}{\eta}\right)^{r}, ∣ N η ∣ ≤ h r vol (( 1 + 3 η /2 ) B 2 r ) < ( η 5 ( 1 + 3 η /2 ) ) r ≤ ( η 16 ) r ,
as claimed. □ \square □
We now combine Lemma 3.2 and Lemma 3.3 to establish Theorem 2.2.
Theorem 2.2. Fix any constant ε ∈ ( 0 , 1 ) \varepsilon\in(0,1) ε ∈ ( 0 , 1 ) and failure parameter δ l o c ∈ ( 0 , 1 ) \delta_{\mathrm{loc}}\in(0,1) δ loc ∈ ( 0 , 1 ) . There exists a constant λ = Ω ( ε ) \lambda=\Omega(\varepsilon) λ = Ω ( ε ) depending only on ε \varepsilon ε and δ l o c \delta_{\mathrm{loc}} δ loc such that for every fixed isometry Q ∈ R n × r Q\in\mathbb{R}^{n\times r} Q ∈ R n × r with r r r sufficiently large, every 1 ≤ ζ ≤ r 1\le\zeta\le r 1 ≤ ζ ≤ r , every k = ζ b = Ω ( r ) k=\zeta b=\Omega(r) k = ζ b = Ω ( r ) , and m = Θ ( r / ζ ) m=\Theta(r/\zeta) m = Θ ( r / ζ ) with sufficiently large implied constants, the sketch X = ( Ω ζ , b ( 1 ) ) T Q ∈ R k × r X=(\Omega_{\zeta,b}^{(1)})^{\mathsf{T}}Q\in\mathbb{R}^{k\times r} X = ( Ω ζ , b ( 1 ) ) T Q ∈ R k × r satisfies
P [ ∃ x ∈ S r − 1 : d m ( Q x ) ≥ ε , ∥ X x ∥ < λ ] ≤ δ l o c . \mathbb{P}\left[\exists x\in S^{r-1}:\ d_m(Qx)\ge\varepsilon,\ \lVert Xx\rVert<\lambda\right]\le\delta_{\mathrm{loc}}. P [ ∃ x ∈ S r − 1 : d m ( Q x ) ≥ ε , ∥ X x ∥ < λ ] ≤ δ loc .
Proof. Suppose k = ζ b ≥ C l o c r k=\zeta b\ge C_{\mathrm{loc}}r k = ζ b ≥ C loc r and m ≥ M r / ζ m\ge Mr/\zeta m ≥ M r / ζ for sufficiently large constants C l o c , M > 0 C_{\mathrm{loc}},M>0 C loc , M > 0 depending on ε \varepsilon ε and δ l o c \delta_{\mathrm{loc}} δ loc . Since each row of Ω ζ , b ( 1 ) \Omega_{\zeta,b}^{(1)} Ω ζ , b ( 1 ) has unit norm and E [ Ω ζ , b ( 1 ) ( Ω ζ , b ( 1 ) ) ⊤ ] = I n \mathbb{E}[\Omega_{\zeta,b}^{(1)}(\Omega_{\zeta,b}^{(1)})^{\top}]=I_n E [ Ω ζ , b ( 1 ) ( Ω ζ , b ( 1 ) ) ⊤ ] = I n , we have E [ ∥ X ∥ F 2 ] = tr ( Q ⊤ Q ) = r \mathbb{E}[\lVert X\rVert_F^2]=\operatorname{tr}(Q^{\top}Q)=r E [∥ X ∥ F 2 ] = tr ( Q ⊤ Q ) = r . By Markov’s inequality, with probability at least 1 − δ l o c / 2 1-\delta_{\mathrm{loc}}/2 1 − δ loc /2 ,
∥ X ∥ F 2 ≤ 2 δ l o c r . (18) \lVert X\rVert_F^2\le\frac{2}{\delta_{\mathrm{loc}}}r. \tag*{(18)} ∥ X ∥ F 2 ≤ δ loc 2 r . (18)
Choose the net resolution η = c ′ ε \eta=c'\varepsilon η = c ′ ε with a sufficiently small constant c ′ > 0 c'>0 c ′ > 0 depending on δ l o c \delta_{\mathrm{loc}} δ loc so that η 2 δ l o c ≤ 1 4 c 0 ε \frac{\eta}{\sqrt{2\delta_{\mathrm{loc}}}}\le\frac{1}{4}c_0\varepsilon 2 δ loc η ≤ 4 1 c 0 ε , and set λ = 1 2 c 0 ε \lambda=\frac{1}{2}c_0\varepsilon λ = 2 1 c 0 ε , where c 0 c_0 c 0 is the constant from Lemma 3.2 .
On the event (18) , suppose there exists a unit vector x ∈ S r − 1 x\in S^{r-1} x ∈ S r − 1 satisfying d m ( Q x ) ≥ ε d_m(Qx)\ge\varepsilon d m ( Q x ) ≥ ε and ∥ X x ∥ < λ \lVert Xx\rVert<\lambda ∥ X x ∥ < λ . Applying Lemma 3.3 to T = X T=X T = X , there exists a lattice point y ∈ N η y\in\mathcal{N}_{\eta} y ∈ N η such that ∥ x − y ∥ ≤ η \lVert x-y\rVert\le\eta ∥ x − y ∥ ≤ η and ∥ X ( x − y ) ∥ ≤ η 2 δ l o c ≤ 1 4 c 0 ε \lVert X(x-y)\rVert\le\frac{\eta}{\sqrt{2\delta_{\mathrm{loc}}}}\le\frac{1}{4}c_0\varepsilon ∥ X ( x − y )∥ ≤ 2 δ loc η ≤ 4 1 c 0 ε . Since Q Q Q is an isometry and d m d_m d m is 1-Lipschitz, this lattice point y y y satisfies
d m ( Q y ) ≥ d m ( Q x ) − ∥ Q ( x − y ) ∥ ≥ ε − η ≥ 7 8 ε , d_m(Qy)\ge d_m(Qx)-\lVert Q(x-y)\rVert\ge\varepsilon-\eta\ge\frac{7}{8}\varepsilon, d m ( Q y ) ≥ d m ( Q x ) − ∥ Q ( x − y )∥ ≥ ε − η ≥ 8 7 ε ,
and
∥ X y ∥ ≤ ∥ X x ∥ + ∥ X ( x − y ) ∥ < λ + 1 4 c 0 ε = 3 4 c 0 ε ≤ c 0 ( ε − η ) . \lVert Xy\rVert\le\lVert Xx\rVert+\lVert X(x-y)\rVert<\lambda+\frac{1}{4}c_0\varepsilon=\frac{3}{4}c_0\varepsilon\le c_0(\varepsilon-\eta). ∥ X y ∥ ≤ ∥ X x ∥ + ∥ X ( x − y )∥ < λ + 4 1 c 0 ε = 4 3 c 0 ε ≤ c 0 ( ε − η ) .
Fix any deterministic lattice point y ∈ N η y\in\mathcal{N}_{\eta} y ∈ N η satisfying d m ( Q y ) ≥ ε − η d_m(Qy)\ge\varepsilon-\eta d m ( Q y ) ≥ ε − η . Let I y ⊆ [ n ] I_y\subseteq[n] I y ⊆ [ n ] be the indices of the m m m largest coordinates of Q y Qy Q y in magnitude, and decompose Q y = h + a z Qy=h+az Q y = h + a z where h = ( Q y ) I y h=(Qy)_{I_y} h = ( Q y ) I y , a = ∥ ( Q y ) I y c ∥ = d m ( Q y ) ≥ ε − η a=\lVert(Qy)_{I_y^c}\rVert=d_m(Qy)\ge\varepsilon-\eta a = ∥( Q y ) I y c ∥ = d m ( Q y ) ≥ ε − η , and z = a − 1 ( Q y ) I y c z=a^{-1}(Qy)_{I_y^c} z = a − 1 ( Q y ) I y c has unit norm ∥ z ∥ = 1 \lVert z\rVert=1 ∥ z ∥ = 1 and support disjoint from h h h . Since every coordinate of ( Q y ) I y c (Qy)_{I_y^c} ( Q y ) I y c is bounded in magnitude by each of the m m m coordinates in I y I_y I y , we have
∥ z ∥ ∞ 2 ≤ ∥ Q y ∥ 2 m a 2 ≤ ( 1 + η ) 2 m ( ε − η ) 2 . \lVert z\rVert_\infty^2\le\frac{\lVert Qy\rVert^2}{ma^2}\le\frac{(1+\eta)^2}{m(\varepsilon-\eta)^2}. ∥ z ∥ ∞ 2 ≤ m a 2 ∥ Q y ∥ 2 ≤ m ( ε − η ) 2 ( 1 + η ) 2 .
Setting
D = min { b , m ( ε − η ) 2 ( 1 + η ) 2 } , D=\min\left\{b,m\frac{(\varepsilon-\eta)^2}{(1+\eta)^2}\right\}, D = min { b , m ( 1 + η ) 2 ( ε − η ) 2 } ,
we have 1 ≤ D ≤ b 1\le D\le b 1 ≤ D ≤ b and ∥ z ∥ ∞ 2 ≤ 1 / D \lVert z\rVert_\infty^2\le1/D ∥ z ∥ ∞ 2 ≤ 1/ D . Moreover, since ζ ≤ r \zeta\le r ζ ≤ r , m ≥ M r / ζ m\ge Mr/\zeta m ≥ M r / ζ , and b ζ ≥ C l o c r b\zeta\ge C_{\mathrm{loc}}r b ζ ≥ C loc r , choosing the constants M , C l o c > 0 M,C_{\mathrm{loc}}>0 M , C loc > 0 sufficiently large ensures that
ζ D = min { b ζ , m ζ ( ε − η ) 2 ( 1 + η ) 2 } ≥ min { C l o c , M ( ε − η ) 2 ( 1 + η ) 2 } r ≥ 2 log ( 16 / η ) c s b r . \zeta D=\min\left\{b\zeta,m\zeta\frac{(\varepsilon-\eta)^2}{(1+\eta)^2}\right\}\ge\min\left\{C_{\mathrm{loc}},M\frac{(\varepsilon-\eta)^2}{(1+\eta)^2}\right\}r\ge\frac{2\log(16/\eta)}{c_{\mathrm{sb}}}r. ζ D = min { b ζ , m ζ ( 1 + η ) 2 ( ε − η ) 2 } ≥ min { C loc , M ( 1 + η ) 2 ( ε − η ) 2 } r ≥ c sb 2 log ( 16/ η ) r .
Applying Lemma 3.2 to Q y = h + a z Qy=h+az Q y = h + a z shows that
P [ ∥ X y ∥ < 3 4 c 0 ε ] ≤ P [ ∥ ( Ω ζ , b ( 1 ) ) ⊤ Q y ∥ < c 0 a ] ≤ 2 e − c s b ζ D ≤ 2 ( η 16 ) 2 r . \mathbb{P}\left[\lVert Xy\rVert<\frac{3}{4}c_0\varepsilon\right]\le\mathbb{P}\left[\left\lVert(\Omega_{\zeta,b}^{(1)})^{\top}Qy\right\rVert<c_0a\right]\le2e^{-c_{\mathrm{sb}}\zeta D}\le2\left(\frac{\eta}{16}\right)^{2r}. P [ ∥ X y ∥ < 4 3 c 0 ε ] ≤ P [ ( Ω ζ , b ( 1 ) ) ⊤ Q y < c 0 a ] ≤ 2 e − c sb ζ D ≤ 2 ( 16 η ) 2 r .
Taking a union bound over all ∣ N η ∣ ≤ ( 16 / η ) r \lvert\mathcal{N}_{\eta}\rvert\le(16/\eta)^r ∣ N η ∣ ≤ ( 16/ η ) r deterministic lattice points y ∈ N η y\in\mathcal{N}_{\eta} y ∈ N η and adding the Frobenius failure probability δ l o c / 2 \delta_{\mathrm{loc}}/2 δ loc /2 from (18) , the total probability that such a unit vector x x x exists is at most
δ l o c 2 + ( 16 η ) r ⋅ 2 ( η 16 ) 2 r = δ l o c 2 + 2 ( η 16 ) r ≤ δ l o c \frac{\delta_{\mathrm{loc}}}{2}+\left(\frac{16}{\eta}\right)^r\cdot2\left(\frac{\eta}{16}\right)^{2r}
=\frac{\delta_{\mathrm{loc}}}{2}+2\left(\frac{\eta}{16}\right)^r\le\delta_{\mathrm{loc}} 2 δ loc + ( η 16 ) r ⋅ 2 ( 16 η ) 2 r = 2 δ loc + 2 ( 16 η ) r ≤ δ loc
for r r r sufficiently large. □ \square □
Extracting a Common Support for Spiky Vectors
In § 3 , we showed that with high probability the first half of the sketch ( Ω ζ , b ( 1 ) ) ⊤ ∈ R k × n (\Omega_{\zeta,b}^{(1)})^\top\in\mathbb{R}^{k\times n} ( Ω ζ , b ( 1 ) ) ⊤ ∈ R k × n preserves the norm of all flat unit vectors. Consequently, any unit vector y = Q x y=Qx y = Q x on which injectivity fails under the first half ∥ ( Ω ζ , b ( 1 ) ) ⊤ y ∥ < λ \|(\Omega_{\zeta,b}^{(1)})^\top y\|<\lambda ∥ ( Ω ζ , b ( 1 ) ) ⊤ y ∥ < λ must be spiky , satisfying d m ( y ) < ε d_m(y)<\varepsilon d m ( y ) < ε . Our goal in this section is to prove Theorem 2.3, showing that all such spiky vectors must concentrate their norm on a single, common set I ⊆ [ n ] I\subseteq[n] I ⊆ [ n ] of at most 8 m 8m 8 m coordinates.
Step 1: Expected tail norm of random linear combinations. Recall from § 2 that a random linear combination of vectors supported on mostly disjoint m m m -coordinate sets spreads its entries across their union and becomes flat. To make this precise, for a matrix V ∈ R n × r V\in\mathbb{R}^{n\times r} V ∈ R n × r and Gaussian coefficients x ∼ N ( 0 , R ) x\sim N(0,R) x ∼ N ( 0 , R ) with covariance R ⪰ 0 R\succeq0 R ⪰ 0 , the random linear combination g = V x ∈ R n g=Vx\in\mathbb{R}^n g = V x ∈ R n is a centered Gaussian vector whose i i i -th coordinate has marginal variance a i = E [ g i 2 ] a_i=\mathbb{E}[g_i^2] a i = E [ g i 2 ] .
Sorting these variances as a 1 ≥ a 2 ≥ ⋯ ≥ a n a_1\geq a_2\geq\cdots\geq a_n a 1 ≥ a 2 ≥ ⋯ ≥ a n , Lemma 4.1 shows that the expected squared tail norm E [ d m ( g ) 2 ] \mathbb{E}[d_m(g)^2] E [ d m ( g ) 2 ] after deleting the m m m largest coordinates of g g g is at least 3 4 \frac{3}{4} 4 3 of the total variance ∑ i > 4 m a i \sum_{i>4m}a_i ∑ i > 4 m a i outside the top 4 m 4m 4 m coordinates. In other words, whenever a random linear combination places noticeable variance outside its top 4 m 4m 4 m coordinates, the resulting vector is guaranteed to have large tail norm d m ( g ) d_m(g) d m ( g ) on average and thus behaves like a flat vector rather than a spiky one.
Lemma 4.1. Let g ∈ R n g\in\mathbb{R}^n g ∈ R n be a centered Gaussian vector with arbitrary coordinate correlations, and let its coordinate variances be sorted in non-increasing order as a 1 ≥ ⋯ ≥ a n a_1\geq\cdots\geq a_n a 1 ≥ ⋯ ≥ a n . If n > 4 m n>4m n > 4 m , then
E [ d m ( g ) 2 ] ≥ 3 4 ∑ i > 4 m a i . \mathbb{E}\left[d_m(g)^2\right]\geq\frac{3}{4}\sum_{i>4m}a_i. E [ d m ( g ) 2 ] ≥ 4 3 i > 4 m ∑ a i .
Proof. For every vector z ∈ R n z\in\mathbb{R}^n z ∈ R n and every truncation threshold T ≥ 0 T\geq0 T ≥ 0 , removing at most m m m coordinates from the sum of truncated squares ∑ i = 1 n min { z i 2 , T } \sum_{i=1}^n\min\{z_i^2,T\} ∑ i = 1 n min { z i 2 , T } reduces the sum by at most m T mT m T . Hence
d m ( z ) 2 = min ∣ I ∣ ≤ m ∑ i ∉ I z i 2 ≥ min ∣ I ∣ ≤ m ∑ i ∉ I min { z i 2 , T } ≥ ∑ i = 1 n min { z i 2 , T } − m T . (19) d_m(z)^2=\min_{|I|\leq m}\sum_{i\notin I}z_i^2\geq\min_{|I|\leq m}\sum_{i\notin I}\min\{z_i^2,T\}\geq\sum_{i=1}^n\min\{z_i^2,T\}-mT. \tag*{(19)} d m ( z ) 2 = ∣ I ∣ ≤ m min i ∈ / I ∑ z i 2 ≥ ∣ I ∣ ≤ m min i ∈ / I ∑ min { z i 2 , T } ≥ i = 1 ∑ n min { z i 2 , T } − m T . (19)
For a standard normal random variable G ∼ N ( 0 , 1 ) G\sim N(0,1) G ∼ N ( 0 , 1 ) , we have E [ G 2 ] = 1 \mathbb{E}[G^2]=1 E [ G 2 ] = 1 and E [ G 4 ] = 3 \mathbb{E}[G^4]=3 E [ G 4 ] = 3 . Applying the elementary inequality max { x − A , 0 } ≤ x 2 / ( 4 A ) \max\{x-A,0\}\leq x^2/(4A) max { x − A , 0 } ≤ x 2 / ( 4 A ) with A = 3 A=3 A = 3 and x = G 2 x=G^2 x = G 2 gives
E [ min { G 2 , 3 } ] = E [ G 2 ] − E [ max { G 2 − 3 , 0 } ] ≥ 1 − E [ G 4 ] 12 = 1 − 3 12 = 3 4 . \mathbb{E}\left[\min\{G^2,3\}\right]=\mathbb{E}\left[G^2\right]-\mathbb{E}\left[\max\{G^2-3,0\}\right]\geq1-\frac{\mathbb{E}\left[G^4\right]}{12}=1-\frac{3}{12}=\frac{3}{4}. E [ min { G 2 , 3 } ] = E [ G 2 ] − E [ max { G 2 − 3 , 0 } ] ≥ 1 − 12 E [ G 4 ] = 1 − 12 3 = 4 3 .
Now set j = 4 m j=4m j = 4 m and a = a j a=a_j a = a j . If a = 0 a=0 a = 0 , then a i = 0 a_i=0 a i = 0 for all i > 4 m i>4m i > 4 m and the lemma holds trivially. Otherwise, apply (19) to z = g z=g z = g with threshold T = 3 a > 0 T=3a>0 T = 3 a > 0 . Since each marginal is distributed as g i ∼ a i G g_i\sim\sqrt{a_i}G g i ∼ a i G , monotonicity of u ↦ min { u , 3 a } u\mapsto\min\{u,3a\} u ↦ min { u , 3 a } implies that for i ≤ j i\leq j i ≤ j , where a i ≥ a a_i\geq a a i ≥ a ,
E [ min { g i 2 , 3 a } ] ≥ E [ min { a G 2 , 3 a } ] = a E [ min { G 2 , 3 } ] ≥ 3 4 a , \mathbb{E}\left[\min\{g_i^2,3a\}\right]\geq\mathbb{E}\left[\min\{aG^2,3a\}\right]=a\mathbb{E}\left[\min\{G^2,3\}\right]\geq\frac{3}{4}a, E [ min { g i 2 , 3 a } ] ≥ E [ min { a G 2 , 3 a } ] = a E [ min { G 2 , 3 } ] ≥ 4 3 a ,
whereas for i > j i>j i > j , where a i ≤ a a_i\leq a a i ≤ a ,
E [ min { g i 2 , 3 a } ] ≥ E [ min { a i G 2 , 3 a i } ] = a i E [ min { G 2 , 3 } ] ≥ 3 4 a i . \mathbb{E}\left[\min\{g_i^2,3a\}\right]\geq\mathbb{E}\left[\min\{a_iG^2,3a_i\}\right]=a_i\mathbb{E}\left[\min\{G^2,3\}\right]\geq\frac{3}{4}a_i. E [ min { g i 2 , 3 a } ] ≥ E [ min { a i G 2 , 3 a i } ] = a i E [ min { G 2 , 3 } ] ≥ 4 3 a i .
Taking expectations in (19) and noting that 3 4 j a − m ( 3 a ) = 3 m a − 3 m a = 0 \frac{3}{4}ja-m(3a)=3ma-3ma=0 4 3 j a − m ( 3 a ) = 3 ma − 3 ma = 0 yields
E [ d m ( g ) 2 ] ≥ ∑ i = 1 j 3 4 a + ∑ i > 4 m 3 4 a i − 3 m a = 3 4 ∑ i > 4 m a i . \mathbb{E}\left[d_m(g)^2\right]\geq\sum_{i=1}^{j}\frac{3}{4}a+\sum_{i>4m}\frac{3}{4}a_i-3ma=\frac{3}{4}\sum_{i>4m}a_i. E [ d m ( g ) 2 ] ≥ i = 1 ∑ j 4 3 a + i > 4 m ∑ 4 3 a i − 3 ma = 4 3 i > 4 m ∑ a i .
:::
Step 2: Extracting a single common support via minimax duality. In our setting, every unit vector x ∈ S r − 1 x \in S^{r-1} x ∈ S r − 1 satisfies either d m ( Q x ) 2 < ε 2 d_m(Qx)^2 < \varepsilon^2 d m ( Q x ) 2 < ε 2 when ∥ X x ∥ < λ \lVert Xx\rVert< \lambda ∥ X x ∥ < λ , or d m ( Q x ) 2 ≤ ∥ Q x ∥ 2 = 1 ≤ λ − 2 ∥ X x ∥ 2 d_m(Qx)^2 \le\lVert Qx\rVert^2 = 1 \le\lambda^{-2}\lVert Xx\rVert^2 d m ( Q x ) 2 ≤ ∥ Q x ∥ 2 = 1 ≤ λ − 2 ∥ X x ∥ 2 when ∥ X x ∥ ≥ λ \lVert Xx\rVert\ge\lambda ∥ X x ∥ ≥ λ . Combining the two cases yields the quadratic upper bound
d m ( Q x ) 2 ≤ ε 2 ∥ x ∥ 2 + λ − 2 ∥ X x ∥ 2 = x T H x , where H = ε 2 I r + λ − 2 X T X . (20) d_m(Qx)^2 \le\varepsilon^2\lVert x\rVert^2 + \lambda^{-2}\lVert Xx\rVert^2 = x^\mathsf{T}Hx, \qquad\text{where } H = \varepsilon^2 I_r + \lambda^{-2}X^\mathsf{T}X. \tag*{(20)} d m ( Q x ) 2 ≤ ε 2 ∥ x ∥ 2 + λ − 2 ∥ X x ∥ 2 = x T H x , where H = ε 2 I r + λ − 2 X T X . (20)
More generally, whenever V ∈ R n × r V \in\mathbb{R}^{n \times r} V ∈ R n × r and H ⪰ 0 H \succeq0 H ⪰ 0 satisfy d m ( V x ) 2 ≤ x T H x d_m(Vx)^2 \le x^\mathsf{T}Hx d m ( V x ) 2 ≤ x T H x for all x ∈ R r x \in\mathbb{R}^r x ∈ R r , taking expectations over x ∼ N ( 0 , R ) x \sim\mathcal{N}(0,R) x ∼ N ( 0 , R ) and applying Lemma 4.1 implies that for every covariance R ⪰ 0 R \succeq0 R ⪰ 0 , there exists a set of 4 m 4m 4 m coordinates—potentially depending on R R R —outside of which the expected squared norm of V x Vx V x is at most 4 3 E [ x T H x ] \frac{4}{3}\mathbb{E}[x^\mathsf{T}Hx] 3 4 E [ x T H x ] .
The goal of Step 2 is to lift this distributional guarantee to a single, common coordinate set I I I that works for all x ∈ R r x \in\mathbb{R}^r x ∈ R r simultaneously. As outlined in Section 2, we relax the choice of a 4 m 4m 4 m -element subset to a fractional weight vector w ∈ [ 0 , 1 ] n w \in[0,1]^n w ∈ [ 0 , 1 ] n with ∑ i = 1 n w i ≤ 4 m \sum_{i=1}^n w_i \le4m ∑ i = 1 n w i ≤ 4 m , which forms a compact convex set W \mathcal{W} W . We will show that by minimax duality, there exists a single weight vector w ∈ W w \in\mathcal{W} w ∈ W that bounds the weighted tail norm for all R ⪰ 0 R \succeq0 R ⪰ 0 at once. Rounding these weights at w i ≥ 1 / 2 w_i \ge1/2 w i ≥ 1/2 then yields a single coordinate set I ⊆ [ n ] I \subseteq[n] I ⊆ [ n ] of size ∣ I ∣ ≤ 8 m |I| \le8m ∣ I ∣ ≤ 8 m satisfying ∥ ( V x ) I c ∥ 2 ≤ 8 3 x T H x \lVert(Vx)_{I^c}\rVert^2 \le\frac{8}{3}x^\mathsf{T}Hx ∥( V x ) I c ∥ 2 ≤ 3 8 x T H x for all x ∈ R r x \in\mathbb{R}^r x ∈ R r , or equivalently V I c T V I c ⪯ 8 3 H V_{I^c}^\mathsf{T}V_{I^c} \preceq\frac{8}{3}H V I c T V I c ⪯ 3 8 H . We formally state the result and the proof below.
Lemma 4.2. Let V ∈ R n × r V \in\mathbb{R}^{n \times r} V ∈ R n × r , let H ⪰ 0 H \succeq0 H ⪰ 0 be an r × r r \times r r × r positive-semidefinite matrix, and let m ≥ 1 m \ge1 m ≥ 1 be an integer. Suppose that d m ( V x ) 2 ≤ x T H x d_m(Vx)^2 \le x^\mathsf{T}Hx d m ( V x ) 2 ≤ x T H x for all x ∈ R r x \in\mathbb{R}^r x ∈ R r . Then there exists a coordinate subset I ⊆ [ n ] I \subseteq[n] I ⊆ [ n ] satisfying
∣ I ∣ ≤ 8 m , V I c T V I c ⪯ 8 3 H . |I| \le8m, \qquad V_{I^c}^\mathsf{T}V_{I^c} \preceq\frac{8}{3}H. ∣ I ∣ ≤ 8 m , V I c T V I c ⪯ 3 8 H .
Proof. The case r = 0 r=0 r = 0 is trivial, and if n ≤ 4 m n \le4m n ≤ 4 m we may simply take I = [ n ] I = [n] I = [ n ] . Assume n > 4 m n > 4m n > 4 m , put j = 4 m j = 4m j = 4 m , and write v i = V T e i ∈ R r v_i = V^\mathsf{T}e_i \in\mathbb{R}^r v i = V T e i ∈ R r for the i i i -th row of V V V .
For any positive-semidefinite matrix R ⪰ 0 R \succeq0 R ⪰ 0 with tr R = 1 \operatorname{tr} R = 1 tr R = 1 , let x ∼ N ( 0 , R ) x \sim\mathcal{N}(0,R) x ∼ N ( 0 , R ) . Then g = V x ∈ R n g = Vx \in\mathbb{R}^n g = V x ∈ R n is a centered Gaussian vector whose i i i -th coordinate has variance a i ( R ) = E [ ( v i T x ) 2 ] = v i T R v i a_i(R) = \mathbb{E}[(v_i^\mathsf{T}x)^2] = v_i^\mathsf{T}Rv_i a i ( R ) = E [( v i T x ) 2 ] = v i T R v i . By the hypothesis of the lemma, E [ d m ( V x ) 2 ] ≤ E [ x T H x ] = tr ( H R ) \mathbb{E}[d_m(Vx)^2] \le\mathbb{E}[x^\mathsf{T}Hx] = \operatorname{tr}(HR) E [ d m ( V x ) 2 ] ≤ E [ x T H x ] = tr ( H R ) . On the other hand, Lemma 4.1 shows that E [ d m ( V x ) 2 ] \mathbb{E}[d_m(Vx)^2] E [ d m ( V x ) 2 ] is at least 3 4 \frac{3}{4} 4 3 times the sum of all v i T R v i v_i^\mathsf{T}Rv_i v i T R v i excluding the 4 m 4m 4 m largest values:
min ∣ T ∣ ≤ 4 m ∑ i ∉ T v i T R v i ≤ 4 3 tr ( H R ) . (21) \min_{\lvert T\rvert\le4m}\sum_{i\notin T}v_i^\mathsf{T}Rv_i \le\frac{4}{3}\operatorname{tr}(HR). \tag*{(21)} ∣ T ∣ ≤ 4 m min i ∈ / T ∑ v i T R v i ≤ 3 4 tr ( H R ) . (21)
Define the compact convex weight polytope W \mathcal{W} W and the linear matrix map D ( w ) D(w) D ( w ) by
W = { w ∈ [ 0 , 1 ] n : ∑ i = 1 n w i ≤ 4 m } , D ( w ) = ∑ i = 1 n ( 1 − w i ) v i v i T . \mathcal{W} = \left\{w \in[0,1]^n : \sum_{i=1}^n w_i \le4m\right\}, \qquad D(w) = \sum_{i=1}^n(1-w_i)v_iv_i^\mathsf{T}. W = { w ∈ [ 0 , 1 ] n : i = 1 ∑ n w i ≤ 4 m } , D ( w ) = i = 1 ∑ n ( 1 − w i ) v i v i T .
For any fixed R ⪰ 0 R \succeq0 R ⪰ 0 , the linear functional w ↦ tr ( R D ( w ) ) = ∑ i = 1 n ( 1 − w i ) v i T R v i w \mapsto\operatorname{tr}(RD(w)) = \sum_{i=1}^n(1-w_i)v_i^\mathsf{T}Rv_i w ↦ tr ( R D ( w )) = ∑ i = 1 n ( 1 − w i ) v i T R v i is minimized over W \mathcal{W} W by placing unit weight w i = 1 w_i = 1 w i = 1 on the 4 m 4m 4 m indices with the largest values of v i T R v i v_i^\mathsf{T}Rv_i v i T R v i . Therefore, (21) is equivalent to the inequality
min w ∈ W tr ( R ( D ( w ) − 4 3 H ) ) ≤ 0 for all R ⪰ 0 with tr R = 1. (22) \min_{w\in\mathcal{W}}\operatorname{tr}\left(R\left(D(w)-\frac{4}{3}H\right)\right) \le0 \qquad\text{for all } R \succeq0 \text{ with } \operatorname{tr} R = 1. \tag*{(22)} w ∈ W min tr ( R ( D ( w ) − 3 4 H ) ) ≤ 0 for all R ⪰ 0 with tr R = 1. (22)
Claim 4.3. There exists a single weight vector w ∈ W w \in\mathcal{W} w ∈ W such that D ( w ) ⪯ 4 3 H D(w) \preceq\frac{4}{3}H D ( w ) ⪯ 3 4 H .
Proof. Let R = { R : R ⪰ 0 , tr ( R ) = 1 } \mathcal{R}=\{R:R\succeq0,\operatorname{tr}(R)=1\} R = { R : R ⪰ 0 , tr ( R ) = 1 } , such that both R \mathcal{R} R and W \mathcal{W} W are convex and compact. By a standard minimax theorem for compact convex sets and bilinear payoffs [14 ] , since the function tr ( R ( D ( w ) − 4 3 H ) ) \operatorname{tr}\left(R\left(D(w)-\frac{4}{3}H\right)\right) tr ( R ( D ( w ) − 3 4 H ) ) is bilinear in w w w and R R R , we have that
min w ∈ W max R ∈ R tr ( R ( D ( w ) − 4 3 H ) ) = max R ∈ R min w ∈ W tr ( R ( D ( w ) − 4 3 H ) ) ≤ 0. (23) \min_{w\in\mathcal{W}}\max_{R\in\mathcal{R}}\operatorname{tr}\left(R\left(D(w)-\frac{4}{3}H\right)\right)=\max_{R\in\mathcal{R}}\min_{w\in\mathcal{W}}\operatorname{tr}\left(R\left(D(w)-\frac{4}{3}H\right)\right)\leq0. \tag*{(23)} w ∈ W min R ∈ R max tr ( R ( D ( w ) − 3 4 H ) ) = R ∈ R max w ∈ W min tr ( R ( D ( w ) − 3 4 H ) ) ≤ 0. (23)
Now for any symmetric matrix A A A , we have that
max R ∈ R tr ( R A ) = λ max ( A ) . (24) \max_{R\in\mathcal{R}}\operatorname{tr}(RA)=\lambda_{\max}(A). \tag*{(24)} R ∈ R max tr ( R A ) = λ m a x ( A ) . (24)
Thus
min w ∈ W λ max ( D ( w ) − 4 3 H ) ≤ 0 , (25) \min_{w\in\mathcal{W}}\lambda_{\max}\left(D(w)-\frac{4}{3}H\right)\leq0, \tag*{(25)} w ∈ W min λ m a x ( D ( w ) − 3 4 H ) ≤ 0 , (25)
meaning that there exists some w ∈ W w\in\mathcal{W} w ∈ W such that D ( w ) ⪯ 4 3 H D(w)\preceq\frac{4}{3}H D ( w ) ⪯ 3 4 H . This proves the claim.
Now we round the w w w attained by Claim 4.3. Define the thresholded coordinate set I = { i ∈ [ n ] : w i ≥ 1 / 2 } I=\{i\in[n]:w_i\geq1/2\} I = { i ∈ [ n ] : w i ≥ 1/2 } . By Markov’s inequality on the weights, ∣ I ∣ ≤ 2 ∑ i = 1 n w i ≤ 2 j = 8 m |I|\leq2\sum_{i=1}^{n}w_i\leq2j=8m ∣ I ∣ ≤ 2 ∑ i = 1 n w i ≤ 2 j = 8 m , and for every i ∈ I c i\in I^{c} i ∈ I c we have 1 ≤ 2 ( 1 − w i ) 1\leq2(1-w_i) 1 ≤ 2 ( 1 − w i ) . Hence
V I c T V I c = ∑ i ∈ I c v i v i T ≤ 2 ∑ i = 1 n ( 1 − w i ) v i v i T = 2 D ( w ) ⪯ 8 3 H , V_{I^{c}}^{\mathsf{T}}V_{I^{c}}=\sum_{i\in I^{c}}v_iv_i^{\mathsf{T}}\leq2\sum_{i=1}^{n}(1-w_i)v_iv_i^{\mathsf{T}}=2D(w)\preceq\frac{8}{3}H, V I c T V I c = i ∈ I c ∑ v i v i T ≤ 2 i = 1 ∑ n ( 1 − w i ) v i v i T = 2 D ( w ) ⪯ 3 8 H ,
completing the proof.
Finally, we apply Lemma 4.2 with V = Q V=Q V = Q and H = ε 2 I r + λ − 2 X T X H=\varepsilon^{2}I_r+\lambda^{-2}X^{\mathsf{T}}X H = ε 2 I r + λ − 2 X T X to obtain Theorem 2.3. As discussed above, the matrix inequality Q I c T Q I c ⪯ 8 3 ε 2 I r + 8 3 λ 2 X T X Q_{I^{c}}^{\mathsf{T}}Q_{I^{c}}\preceq\frac{8}{3}\varepsilon^{2}I_r+\frac{8}{3\lambda^{2}}X^{\mathsf{T}}X Q I c T Q I c ⪯ 3 8 ε 2 I r + 3 λ 2 8 X T X states that every spiky unit vector y = Q x y=Qx y = Q x with small ∥ X x ∥ \lVert Xx\rVert ∥ X x ∥ concentrates at least 1 − 8 3 ε 2 1-\frac{8}{3}\varepsilon^{2} 1 − 3 8 ε 2 of its squared norm on the single coordinate set I I I of size ∣ I ∣ ≤ 8 m |I|\leq8m ∣ I ∣ ≤ 8 m .
Theorem 2.3. Let Q ∈ R n × r Q\in\mathbb{R}^{n\times r} Q ∈ R n × r be an isometry. Suppose a matrix X ∈ R k × r X\in\mathbb{R}^{k\times r} X ∈ R k × r satisfies
d m ( Q x ) ≥ ε , ∥ x ∥ = 1 ⟹ ∥ X x ∥ ≥ λ d_m(Qx)\geq\varepsilon,\qquad\lVert x\rVert=1\quad\Longrightarrow\quad\lVert Xx\rVert\geq\lambda d m ( Q x ) ≥ ε , ∥ x ∥ = 1 ⟹ ∥ X x ∥ ≥ λ for some ε > 0 \varepsilon>0 ε > 0 and λ > 0 \lambda>0 λ > 0 . Then there exists a coordinate set I ⊆ [ n ] I\subseteq[n] I ⊆ [ n ] with ∣ I ∣ ≤ 8 m |I|\leq8m ∣ I ∣ ≤ 8 m such that
Q I c T Q I c ⪯ 8 3 ε 2 I r + 8 3 λ 2 X T X . Q_{I^{c}}^{\mathsf{T}}Q_{I^{c}}\preceq\frac{8}{3}\varepsilon^{2}I_r+\frac{8}{3\lambda^{2}}X^{\mathsf{T}}X. Q I c T Q I c ⪯ 3 8 ε 2 I r + 3 λ 2 8 X T X .
Proof. By (20), the hypothesis implies d m ( Q x ) 2 ≤ x T ( ε 2 I r + λ − 2 X T X ) x d_m(Qx)^2\leq x^{\mathsf{T}}(\varepsilon^{2}I_r+\lambda^{-2}X^{\mathsf{T}}X)x d m ( Q x ) 2 ≤ x T ( ε 2 I r + λ − 2 X T X ) x for all x ∈ R r x\in\mathbb{R}^{r} x ∈ R r . Applying Lemma 4.2 with V = Q V=Q V = Q and H = ε 2 I r + λ − 2 X T X H=\varepsilon^{2}I_r+\lambda^{-2}X^{\mathsf{T}}X H = ε 2 I r + λ − 2 X T X guarantees the existence of I ⊆ [ n ] I\subseteq[n] I ⊆ [ n ] with ∣ I ∣ ≤ 8 m |I|\leq8m ∣ I ∣ ≤ 8 m satisfying Q I c T Q I c ⪯ 8 3 ε 2 I r + 8 3 λ 2 X T X Q_{I^{c}}^{\mathsf{T}}Q_{I^{c}}\preceq\frac{8}{3}\varepsilon^{2}I_r+\frac{8}{3\lambda^{2}}X^{\mathsf{T}}X Q I c T Q I c ⪯ 3 8 ε 2 I r + 3 λ 2 8 X T X .
Bounding Cancellation from the Complementary Coordinates
In Section 4, we proved that every spiky vector y ∈ span ( Q ) y \in\operatorname{span}(Q) y ∈ span ( Q ) whose norm under the first half ( Ω ζ , b ( 1 ) ) ⊤ (\Omega_{\zeta,b}^{(1)})^{\top} ( Ω ζ , b ( 1 ) ) ⊤ is small must concentrate its norm on a single common coordinate set I ⊆ [ n ] I \subseteq[n] I ⊆ [ n ] of size N = ∣ I ∣ ≤ 8 m = O ( r / ζ ) N = |I| \le8m = O(r/\zeta) N = ∣ I ∣ ≤ 8 m = O ( r / ζ ) . It remains to show that with high probability, any unit vector y ∈ span ( Q ) y \in\operatorname{span}(Q) y ∈ span ( Q ) with sufficiently large mass on the coordinates y I y_I y I , satisfies that ∥ ( Ω ζ , b ( 2 ) ) ⊤ y ∥ ≥ Ω ( 1 ) \|(\Omega_{\zeta,b}^{(2)})^{\top}y\| \ge\Omega(1) ∥ ( Ω ζ , b ( 2 ) ) ⊤ y ∥ ≥ Ω ( 1 ) .
Throughout this section, we will use the shorthand Ω ⊤ : = ( Ω ζ , b ( 2 ) ) ⊤ ∈ R k × n \Omega^{\top} := (\Omega_{\zeta,b}^{(2)})^{\top} \in\mathbb{R}^{k \times n} Ω ⊤ := ( Ω ζ , b ( 2 ) ) ⊤ ∈ R k × n . We will decompose y = Q x y = Qx y = Q x into its heavy coordinates h = ( Q x ) I h = (Qx)_I h = ( Q x ) I and complementary coordinates ( Q x ) I c = W f (Qx)_{I^c} = Wf ( Q x ) I c = W f (with W ∈ R ∣ I c ∣ × d W \in\mathbb{R}^{|I^c| \times d} W ∈ R ∣ I c ∣ × d an orthonormal basis for col ( Q I c ) \operatorname{col}(Q_{I^c}) col ( Q I c ) and ∥ f ∥ = ∥ ( Q x ) I c ∥ \|f\| = \|(Qx)_{I^c}\| ∥ f ∥ = ∥ ( Q x ) I c ∥ ).
With this notation, we can expand the squared sketch norm ∥ Ω ζ , b ( 2 ) y ∥ 2 \|\Omega_{\zeta,b}^{(2)}y\|^2 ∥ Ω ζ , b ( 2 ) y ∥ 2 as follows:
∥ Ω ζ , b ( 2 ) y ∥ 2 = ∥ Ω I ⊤ h + Ω I c ⊤ W f ∥ 2 = ∥ Ω I ⊤ h ∥ 2 + 2 h ⊤ ( Ω I Ω I c ⊤ W ) f + ∥ Ω I c ⊤ W f ∥ 2 . \|\Omega_{\zeta,b}^{(2)}y\|^2 = \|\Omega_I^{\top}h + \Omega_{I^c}^{\top}Wf\|^2 = \|\Omega_I^{\top}h\|^2 + 2h^{\top}(\Omega_I\Omega_{I^c}^{\top}W)f + \|\Omega_{I^c}^{\top}Wf\|^2 . ∥ Ω ζ , b ( 2 ) y ∥ 2 = ∥ Ω I ⊤ h + Ω I c ⊤ W f ∥ 2 = ∥ Ω I ⊤ h ∥ 2 + 2 h ⊤ ( Ω I Ω I c ⊤ W ) f + ∥ Ω I c ⊤ W f ∥ 2 .
For reasons that will become clear in § 5.2 , it will be easier to lower bound this norm over a subset of coordinates T ⊆ [ k ] T \subseteq[k] T ⊆ [ k ] , which correspond to rows of Ω I ⊤ \Omega_I^{\top} Ω I ⊤ in which there is exactly one non-zero entry. Let Ω ~ \widetilde{\Omega} Ω denote the restriction of Ω \Omega Ω to rows in T T T . Then we have
∥ Ω ⊤ y ∥ 2 ≥ ∥ Ω ~ ⊤ y ∥ 2 = ∥ Ω ~ I ⊤ h ∥ 2 + 2 h ⊤ ( Ω ~ I Ω ~ I c ⊤ W ) f + ∥ Ω ~ I c ⊤ W f ∥ 2 . \|\Omega^{\top}y\|^2 \ge\|\widetilde{\Omega}^{\top}y\|^2 = \|\widetilde{\Omega}_I^{\top}h\|^2 + 2h^{\top}(\widetilde{\Omega}_I\widetilde{\Omega}_{I^c}^{\top}W)f + \|\widetilde{\Omega}_{I^c}^{\top}Wf\|^2 . ∥ Ω ⊤ y ∥ 2 ≥ ∥ Ω ⊤ y ∥ 2 = ∥ Ω I ⊤ h ∥ 2 + 2 h ⊤ ( Ω I Ω I c ⊤ W ) f + ∥ Ω I c ⊤ W f ∥ 2 .
In § 6 , we will claim that there is not too much cancellation within I I I , and thus ∥ Ω ~ I ⊤ h ∥ 2 \|\widetilde{\Omega}_I^{\top}h\|^2 ∥ Ω I ⊤ h ∥ 2 is a sufficiently large constant.
In this section, we will show that in expectation, the cross term is sufficiently small, by bounding the operator norm ∥ Ω ~ I Ω ~ I c ⊤ W ∥ \|\widetilde{\Omega}_I\widetilde{\Omega}_{I^c}^{\top}W\| ∥ Ω I Ω I c ⊤ W ∥ . The following is the main result of Section 5.
Lemma 5.1. There exists a universal constant C c g > 0 C_{\mathrm{cg}} > 0 C cg > 0 such that the following holds. Let Ω ∈ R n × k \Omega\in\mathbb{R}^{n \times k} Ω ∈ R n × k (k = ζ b k = \zeta b k = ζ b ) be a SparseStack sketch with ζ ≥ 1 \zeta\ge1 ζ ≥ 1 blocks and b ≥ 2 ζ b \ge2\zeta b ≥ 2 ζ buckets per block. Let I , J ⊆ [ n ] I, J \subseteq[n] I , J ⊆ [ n ] be disjoint coordinate sets with ∣ I ∣ = N ≥ 1 |I| = N \ge1 ∣ I ∣ = N ≥ 1 , and let W ∈ R ∣ J ∣ × d W \in\mathbb{R}^{|J| \times d} W ∈ R ∣ J ∣ × d (d ≥ 1 d \ge1 d ≥ 1 ) be a fixed isometry. Let T ⊆ [ k ] T \subseteq[k] T ⊆ [ k ] be the set of rows in which Ω I ⊤ \Omega_I^{\top} Ω I ⊤ has exactly one non-zero entry, and let Ω ~ ⊤ \widetilde{\Omega}^{\top} Ω ⊤ denote the restrictions of Ω ⊤ \Omega^{\top} Ω ⊤ to rows in T T T .
For every integer p ≥ max { 1 , log ( 2 d ) } p \ge\max\{1, \log(2d)\} p ≥ max { 1 , log ( 2 d )} , setting D p = d + 2 p + 1 D_p = d + 2p + 1 D p = d + 2 p + 1 , we have conditionally on Ω I \Omega_I Ω I :
E [ ∥ Ω ~ I Ω ~ I c ⊤ W ∥ | Ω I ] ≤ C c g ( N + D p k + p + 1 ζ 2 + N d ζ 2 b 3 ( ζ 3 ) ) . \mathbb{E}\left[\left\|\widetilde{\Omega}_I\widetilde{\Omega}_{I^c}^{\top}W\right\| \,\middle|\, \Omega_I\right] \le C_{\mathrm{cg}}\left(\sqrt{\frac{N+D_p}{k}}+\frac{p+1}{\zeta^2}+\sqrt{\frac{Nd}{\zeta^2b^3}\left(\frac{\zeta}{3}\right)}\right). E [ Ω I Ω I c ⊤ W Ω I ] ≤ C cg ( k N + D p + ζ 2 p + 1 + ζ 2 b 3 N d ( 3 ζ ) ) . If N = 0 N = 0 N = 0 or d = 0 d = 0 d = 0 , then the expression is trivially 0.
Before proving this result, let us unpack its meaning. When applied with N = Θ ( r / ζ ) N = \Theta(r/\zeta) N = Θ ( r / ζ ) and d = Θ ( r ) d = \Theta(r) d = Θ ( r ) , ζ = O ( log ( r ) ) \zeta= O(\sqrt{\log(r)}) ζ = O ( log ( r ) ) , and k = Θ ( r ) k = \Theta(r) k = Θ ( r ) , this result says that E [ ∥ Ω ~ I Ω ~ I c ⊤ W ∥ | Ω ζ , b , I ] = O ( 1 ) \mathbb{E}\left[\left\|\widetilde{\Omega}_I\widetilde{\Omega}_{I^c}^{\top}W\right\| \,\middle|\, \Omega_{\zeta,b,I}\right] = O(1) E [ Ω I Ω I c ⊤ W Ω ζ , b , I ] = O ( 1 ) . As noted before in § 1.1 (see the discussion of “coupon-collector argument”), in this regime, the operator norm of Ω I \Omega_I Ω I or Ω I c \Omega_{I^c} Ω I c is likely to be as large as Θ ( log r / log log r ) \Theta(\sqrt{\log r}/\log\log r) Θ ( log r / log log r ) , and so it is crucial to our proof that this cross-term cancels out the super-constant norm of Ω I \Omega_I Ω I or Ω I c \Omega_{I^c} Ω I c alone.
The key idea in proving Lemma 2.4 is to compare Ω ~ I Ω ~ I c ⊤ W \widetilde{\Omega}_I\widetilde{\Omega}_{I^c}^{\top}W Ω I Ω I c ⊤ W to a matrix X W XW X W , where X X X has i.i.d. entries, with the same entry-wise marginal distribution as Ω ~ I Ω ~ I c ⊤ \widetilde{\Omega}_I\widetilde{\Omega}_{I^c}^{\top} Ω I Ω I c ⊤ . It is easy to check that each entry ( Ω ~ I Ω ~ I c ⊤ ) i , j (\widetilde{\Omega}_I\widetilde{\Omega}_{I^c}^{\top})_{i,j} ( Ω I Ω I c ⊤ ) i , j is non-zero with probability roughly ζ / b = Θ ( ζ 2 / r ) = Ω ( log r / r ) \zeta/b = \Theta(\zeta^2/r) = \Omega(\log r/r) ζ / b = Θ ( ζ 2 / r ) = Ω ( log r / r ) in the regime of our main theorem. To control the operator norm of this matrix X W XW X W , we are able to leverage the recent work of [17 ] , which shows that SparseStack is an OSE in this regime of Θ ( log r / r ) \Theta(\log r/r) Θ ( log r / r ) sparsity. We prove this i.i.d. result in § 5.1 .
In § 5.2 , we show how to use Jensen’s inequality to transfer this result from the i.i.d. matrix to \widetilde{\Omega}_{I}\widetilde{\Omega}_{I}^{\mathsf{T}}_{c}W .
Finally in § 5.3 , we give an additional estimate for high order moments of the norm of Ω ζ , b Q \Omega_{\zeta,b}Q Ω ζ , b Q , also based off of [17 ] . Note that this is only used in the high-sparsity regime ζ = Ω ( log r ) \zeta=\Omega(\log r) ζ = Ω ( log r ) , but enables our result to hold for the full spectrum of ζ \zeta ζ above the Θ ( log r ) \Theta(\sqrt{\log r}) Θ ( log r ) threshold.
Operator-norm estimate for matrices with independent entries
To bound the cross-term Ω ~ I Ω ~ I c T W \widetilde{\Omega}_{I}\widetilde{\Omega}_{I^{c}}^{\mathsf{T}}W Ω I Ω I c T W , we first analyze the simpler model where the X W XW X W , where X X X has independent entries. For a random matrix X ∈ R m × n X\in\mathbb{R}^{m\times n} X ∈ R m × n with independent centered entries and a fixed isometry W ∈ R n × d W\in\mathbb{R}^{n\times d} W ∈ R n × d , we control E [ ∥ X W ∥ 2 ] = E [ ∥ ( X W ) T ( X W ) ∥ ] \mathbb{E}[\lVert XW\rVert^{2}]=\mathbb{E}[\lVert(XW)^{\mathsf{T}}(XW)\rVert] E [∥ X W ∥ 2 ] = E [∥( X W ) T ( X W )∥] via decomposing it into its diagonal and off-diagonal parts:
( X W ) T ( X W ) = ∑ ℓ = 1 m ∑ j = 1 n X ℓ j 2 w j w j T + ∑ ℓ = 1 m ∑ j ≠ j ′ X ℓ j X ℓ j ′ w j w j ′ T . (XW)^{\mathsf{T}}(XW)=\sum_{\ell=1}^{m}\sum_{j=1}^{n}X_{\ell j}^{2}w_jw_j^{\mathsf{T}}+\sum_{\ell=1}^{m}\sum_{j\ne j'}X_{\ell j}X_{\ell j'}w_jw_{j'}^{\mathsf{T}}. ( X W ) T ( X W ) = ℓ = 1 ∑ m j = 1 ∑ n X ℓ j 2 w j w j T + ℓ = 1 ∑ m j = j ′ ∑ X ℓ j X ℓ j ′ w j w j ′ T .
This separation is essential because the two terms exhibit fundamentally different probabilistic behaviors:
Diagonal term (j = j ′ j=j' j = j ′ ): The entries X ℓ j 2 X_{\ell j}^{2} X ℓ j 2 are non-negative with non-zero mean. Centering yields a sum of independent positive-semidefinite rank-one random matrices with zero mean, which is naturally controlled by the Matrix Bernstein inequality (Fact 5.3).
Off-diagonal term (j ≠ j ′ j\ne j' j = j ′ ): This is a hollow symmetric matrix whose entries are mean-zero products of independent coordinates with no diagonal bias. Its operator norm concentrates sharply via high-order trace moments (Fact 5.2).
Fact 5.2 ([17 ] ). Let B ≥ 2 B\ge2 B ≥ 2 and a ≥ 1 a\ge1 a ≥ 1 be integers, put m = a B m=aB m = a B , and let X ∈ R m × n X\in\mathbb{R}^{m\times n} X ∈ R m × n have independent entries distributed as
P [ X ℓ j = 1 ] = P [ X ℓ j = − 1 ] = 1 2 B , P [ X ℓ j = 0 ] = 1 − 1 B . \mathbb{P}[X_{\ell j}=1]=\mathbb{P}[X_{\ell j}=-1]=\frac{1}{2B},\qquad\mathbb{P}[X_{\ell j}=0]=1-\frac{1}{B}. P [ X ℓ j = 1 ] = P [ X ℓ j = − 1 ] = 2 B 1 , P [ X ℓ j = 0 ] = 1 − B 1 . Let W ∈ R n × d W\in\mathbb{R}^{n\times d} W ∈ R n × d (d ≥ 1 d\ge1 d ≥ 1 ) be a fixed isometry satisfying W T W = I d W^{\mathsf{T}}W=I_d W T W = I d , with rows w j T w_j^{\mathsf{T}} w j T . For the off-diagonal matrix
F = 1 a ∑ ℓ = 1 m ∑ j ≠ j ′ X ℓ j X ℓ j ′ w j w j ′ T (26) F=\frac{1}{a}\sum_{\ell=1}^{m}\sum_{j\ne j'}X_{\ell j}X_{\ell j'}w_jw_{j'}^{\mathsf{T}} \tag*{(26)} F = a 1 ℓ = 1 ∑ m j = j ′ ∑ X ℓ j X ℓ j ′ w j w j ′ T (26) and every integer p ≥ 1 p\ge1 p ≥ 1 , set D p = d + 2 p + 1 D_p=d+2p+1 D p = d + 2 p + 1 . Then
E [ tr ( F 2 p ) ] ≤ d [ 3 ( 3 + 2 ) ( D p m + D p m + 2 p + 1 a ) ] 2 p . (27) \mathbb{E}\left[\operatorname{tr}(F^{2p})\right]\le d\left[3(3+\sqrt{2})\left(\sqrt{\frac{D_p}{m}}+\frac{D_p}{m}+\frac{2p+1}{a}\right)\right]^{2p}. \tag*{(27)} E [ tr ( F 2 p ) ] ≤ d [ 3 ( 3 + 2 ) ( m D p + m D p + a 2 p + 1 ) ] 2 p . (27)
Fact 5.3 ([34 ] ). If R j R_j R j are independent, centered, symmetric d × d d\times d d × d random matrices satisfying
∥ R j ∥ ≤ ℓ almost surely and ∑ j E [ R j 2 ] ⪯ v I d , \lVert R_j\rVert\le\ell\quad\text{almost surely}\qquad\text{and}\qquad\sum_j\mathbb{E}[R_j^2]\preceq vI_d, ∥ R j ∥ ≤ ℓ almost surely and j ∑ E [ R j 2 ] ⪯ v I d ,
then
E [ ∥ ∑ j R j ∥ ] ≤ 2 v log ( 2 d ) + ℓ 3 log ( 2 d ) . (28) \mathbb{E}\left[\left\|\sum_j R_j\right\|\right]\le\sqrt{2v\log(2d)}+\frac{\ell}{3}\log(2d). \tag*{(28)} E [ j ∑ R j ] ≤ 2 v log ( 2 d ) + 3 ℓ log ( 2 d ) . (28)
Combining Fact 5.2 for the off-diagonal sum and Fact 5.3 for the centered diagonal sum yields Lemma 5.4 . This lemma states that when an m × n m \times n m × n independent-entry matrix X X X with non-zero probability 1 / B 1/B 1/ B is applied to a d d d -dimensional isometry W W W , the expected squared operator norm E [ ∥ X W ∥ 2 ] \mathbb{E}[\|XW\|^2] E [ ∥ X W ∥ 2 ] is bounded by the expected column sparsity a = m / B a=m/B a = m / B , the dimension-to-sparsity ratio D p / B D_p/B D p / B , and the moment order p ≥ log ( 2 d ) p \ge\log(2d) p ≥ log ( 2 d ) .
Lemma 5.4. In the independent-entry model of Fact 5.2 , if p ≥ max { 1 , log ( 2 d ) } p \ge\max\{1,\log(2d)\} p ≥ max { 1 , log ( 2 d )} is an integer and D p = d + 2 p + 1 D_p=d+2p+1 D p = d + 2 p + 1 , then there is an absolute constant C i n d > 0 C_{\mathrm{ind}}>0 C ind > 0 such that
E [ ∥ X W ∥ 2 ] ≤ C i n d ( a + D p B + p ) . \mathbb{E}\left[\|XW\|^2\right]\le C_{\mathrm{ind}}\left(a+\frac{D_p}{B}+p\right). E [ ∥ X W ∥ 2 ] ≤ C ind ( a + B D p + p ) .
Proof. We separate a − 1 ( X W ) T ( X W ) a^{-1}(XW)^{\mathsf T}(XW) a − 1 ( X W ) T ( X W ) into its diagonal and off-diagonal parts:
a − 1 ( X W ) T ( X W ) = D + F , D = 1 a ∑ ℓ = 1 m ∑ j = 1 n X ℓ j 2 w j w j T , a^{-1}(XW)^{\mathsf T}(XW)=D+F,\qquad D=\frac{1}{a}\sum_{\ell=1}^{m}\sum_{j=1}^{n}X_{\ell j}^{2}w_jw_j^{\mathsf T}, a − 1 ( X W ) T ( X W ) = D + F , D = a 1 ℓ = 1 ∑ m j = 1 ∑ n X ℓ j 2 w j w j T ,
where F F F is defined in (26). Since E [ X ℓ j 2 ] = B − 1 \mathbb{E}[X_{\ell j}^{2}]=B^{-1} E [ X ℓ j 2 ] = B − 1 , we have E [ D ] = m a B ∑ j = 1 n w j w j T = I d \mathbb{E}[D]=\frac{m}{aB}\sum_{j=1}^{n}w_jw_j^{\mathsf T}=I_d E [ D ] = a B m ∑ j = 1 n w j w j T = I d . Writing D − I d = ∑ ℓ , j R ℓ j D-I_d=\sum_{\ell,j}R_{\ell j} D − I d = ∑ ℓ , j R ℓ j , the summands
R ℓ j = 1 a ( X ℓ j 2 − B − 1 ) w j w j T R_{\ell j}=\frac{1}{a}\left(X_{\ell j}^{2}-B^{-1}\right)w_jw_j^{\mathsf T} R ℓ j = a 1 ( X ℓ j 2 − B − 1 ) w j w j T
are independent, centered, symmetric d × d d\times d d × d matrices. Since ∥ w j ∥ 2 ≤ 1 \|w_j\|^2\le1 ∥ w j ∥ 2 ≤ 1 , each summand satisfies ∥ R ℓ j ∥ ≤ 1 / a \|R_{\ell j}\|\le1/a ∥ R ℓ j ∥ ≤ 1/ a , and the variance proxy satisfies
∑ ℓ , j E [ R ℓ j 2 ] ⪯ m B a 2 ∑ j = 1 n ∥ w j ∥ 2 w j w j T ⪯ 1 a I d . \sum_{\ell,j}\mathbb{E}\left[R_{\ell j}^{2}\right]\preceq\frac{m}{Ba^2}\sum_{j=1}^{n}\|w_j\|^2w_jw_j^{\mathsf T}\preceq\frac{1}{a}I_d. ℓ , j ∑ E [ R ℓ j 2 ] ⪯ B a 2 m j = 1 ∑ n ∥ w j ∥ 2 w j w j T ⪯ a 1 I d .
Applying Fact 5.3 with v = 1 / a v=1/a v = 1/ a , ℓ = 1 / a \ell=1/a ℓ = 1/ a , and log ( 2 d ) ≤ p \log(2d)\le p log ( 2 d ) ≤ p yields
E [ ∥ D ∥ ] ≤ 1 + E [ ∥ D − I d ∥ ] ≤ 1 + 2 p a + p 3 a . \mathbb{E}[\|D\|]\le1+\mathbb{E}[\|D-I_d\|]\le1+\sqrt{\frac{2p}{a}}+\frac{p}{3a}. E [ ∥ D ∥ ] ≤ 1 + E [ ∥ D − I d ∥ ] ≤ 1 + a 2 p + 3 a p .
For the off-diagonal part F F F , since ∥ F ∥ 2 p ≤ tr ( F 2 p ) \|F\|^{2p}\le\operatorname{tr}(F^{2p}) ∥ F ∥ 2 p ≤ tr ( F 2 p ) and d 1 / ( 2 p ) ≤ e < 2 d^{1/(2p)}\le\sqrt{e}<2 d 1/ ( 2 p ) ≤ e < 2 for p ≥ log ( 2 d ) p\ge\log(2d) p ≥ log ( 2 d ) and m = a B m=aB m = a B , Jensen’s inequality and Fact 5.2 give
E [ ∥ F ∥ ] ≤ ( E [ tr ( F 2 p ) ] ) 1 / ( 2 p ) ≤ O ( 1 ) ( D p a B + D p a B + 2 p + 1 a ) . \mathbb{E}[\|F\|]\le\left(\mathbb{E}\left[\operatorname{tr}(F^{2p})\right]\right)^{1/(2p)}
\le O(1)\left(\sqrt{\frac{D_p}{aB}}+\frac{D_p}{aB}+\frac{2p+1}{a}\right). E [ ∥ F ∥ ] ≤ ( E [ tr ( F 2 p ) ] ) 1/ ( 2 p ) ≤ O ( 1 ) ( a B D p + a B D p + a 2 p + 1 ) .
Multiplying the bound on E [ ∥ D ∥ ] + E [ ∥ F ∥ ] \mathbb{E}[\|D\|]+\mathbb{E}[\|F\|] E [ ∥ D ∥ ] + E [ ∥ F ∥ ] by a a a and applying the Cauchy–Schwarz inequalities 2 a p ≤ a + p / 2 \sqrt{2ap}\le a+p/2 2 a p ≤ a + p /2 and a D p / B ≤ 1 2 ( a + D p / B ) \sqrt{aD_p/B}\le\frac{1}{2}(a+D_p/B) a D p / B ≤ 2 1 ( a + D p / B ) yields
E [ ∥ X W ∥ 2 ] = a E [ ∥ D + F ∥ ] ≤ a ( E [ ∥ D ∥ ] + E [ ∥ F ∥ ] ) ≤ O ( a + D p B + p ) , \mathbb{E}\left[\|XW\|^2\right]=a\mathbb{E}[\|D+F\|]\le a\left(\mathbb{E}[\|D\|]+\mathbb{E}[\|F\|]\right)\le O\left(a+\frac{D_p}{B}+p\right), E [ ∥ X W ∥ 2 ] = a E [ ∥ D + F ∥ ] ≤ a ( E [ ∥ D ∥ ] + E [ ∥ F ∥ ] ) ≤ O ( a + B D p + p ) ,
completing the proof.
Coupling argument to bound the cross-term
Our goal is to reduce the analysis of the expected operator norm of Ω ~ I Ω ~ I c T W \widetilde{\Omega}_{I}\widetilde{\Omega}_{I^c}^{\mathsf{T}}W Ω I Ω I c T W to the independent-entry setting of Lemma 5.4. To do this, observe that we can write
( Ω ~ I Ω ~ I c T ) i j = 1 ζ ∑ t = 1 ζ α t i ϱ i t ξ j t [ s i t ] , (29) \left(\widetilde{\Omega}_{I}\widetilde{\Omega}_{I^c}^{\mathsf{T}}\right)_{ij}=\frac{1}{\zeta}\sum_{t=1}^{\zeta}\alpha_{ti}\varrho_{it}\xi_{jt}[s_{it}], \tag*{(29)} ( Ω I Ω I c T ) ij = ζ 1 t = 1 ∑ ζ α t i ϱ i t ξ j t [ s i t ] , (29)
where α t i \alpha_{ti} α t i is the indicator that coordinate i i i occupies a bucket in block t t t with no other element of I I I , and ξ j t = ϱ j t e u j t ∈ R b \xi_{jt}=\varrho_{jt}e_{u_{jt}}\in\mathbb{R}^{b} ξ j t = ϱ j t e u j t ∈ R b . (Recall that u ℓ t ∼ Uniform [ b ] u_{\ell t}\sim\operatorname{Uniform}[b] u ℓ t ∼ Uniform [ b ] , ϱ ℓ t ∈ { − 1 , 1 } \varrho_{\ell t}\in\{-1,1\} ϱ ℓ t ∈ { − 1 , 1 } uniform.) Note that conditional on Ω I \Omega_I Ω I (which fixes the α t i \alpha_{ti} α t i and the ξ i t \xi_{it} ξ i t ), even though the ξ j t \xi_{jt} ξ j t are independent for each j j j , the entries ( Ω ~ I Ω ~ I c T ) i j \left(\widetilde{\Omega}_{I}\widetilde{\Omega}_{I^c}^{\mathsf{T}}\right)_{ij} ( Ω I Ω I c T ) ij are not independent, since the entries of ξ j t \xi_{jt} ξ j t are anti-correlated.
However, if we replaced the ξ j t \xi_{jt} ξ j t with some random vectors η j t \eta_{jt} η j t , where each entry of η j t \eta_{jt} η j t was i.i.d. with distribution equal to the marginal distribution of ξ j t \xi_{jt} ξ j t , we would have that, conditional on Ω ζ , b , I \Omega_{\zeta,b,I} Ω ζ , b , I , the entries
1 ζ ∑ t = 1 ζ α i t ξ i t T η j t = 1 ζ ∑ t = 1 ζ α i t ϱ i t η j t [ s i t ] (30) \frac{1}{\zeta}\sum_{t=1}^{\zeta}\alpha_{it}\xi_{it}^{\mathsf{T}}\eta_{jt}=\frac{1}{\zeta}\sum_{t=1}^{\zeta}\alpha_{it}\varrho_{it}\eta_{jt}[s_{it}] \tag*{(30)} ζ 1 t = 1 ∑ ζ α i t ξ i t T η j t = ζ 1 t = 1 ∑ ζ α i t ϱ i t η j t [ s i t ] (30)
are independent. Indeed, each s i t s_{it} s i t , for which α i t = 1 \alpha_{it}=1 α i t = 1 , is distinct, and thus the values ( x t i j : = η j t [ s i t ] ) i ∈ I , j ∈ J (x_{tij}:=\eta_{jt}[s_{it}])_{i\in I,j\in J} ( x t ij := η j t [ s i t ] ) i ∈ I , j ∈ J are independent! Summing over t t t retains independence.
To bound the expected operator norm of Ω ~ I Ω ~ I c T W \widetilde{\Omega}_{I}\widetilde{\Omega}_{I^c}^{\mathsf{T}}W Ω I Ω I c T W , we will use Jensen’s argument to reduce to the independent entry case. To set this up, consider the following coupling of a ξ ∈ R b \xi\in\mathbb{R}^{b} ξ ∈ R b with an independent-entry vector η ∈ { − 1 , 0 , 1 } b \eta\in\{-1,0,1\}^{b} η ∈ { − 1 , 0 , 1 } b :
Draw η ∈ { − 1 , 0 , 1 } b \eta\in\{-1,0,1\}^{b} η ∈ { − 1 , 0 , 1 } b with independent entries: P [ η u = ± 1 ] = 1 2 b \mathbb{P}[\eta_u=\pm1]=\frac{1}{2b} P [ η u = ± 1 ] = 2 b 1 and P [ η u = 0 ] = 1 − 1 b \mathbb{P}[\eta_u=0]=1-\frac{1}{b} P [ η u = 0 ] = 1 − b 1 .
If η ≠ 0 \eta\ne0 η = 0 , pick u ∗ u^* u ∗ uniformly from supp ( η ) = { u ∈ [ b ] : η u ≠ 0 } \operatorname{supp}(\eta)=\{u\in[b]:\eta_u\ne0\} supp ( η ) = { u ∈ [ b ] : η u = 0 } and set ξ = η u ∗ e u ∗ \xi=\eta_{u^*}e_{u^*} ξ = η u ∗ e u ∗ ; if η = 0 \eta=0 η = 0 , pick ξ ∈ { ± e 1 , … , ± e b } \xi\in\{\pm e_1,\ldots,\pm e_b\} ξ ∈ { ± e 1 , … , ± e b } uniformly.
By symmetry, ξ \xi ξ has the exact marginal distribution of a CountSketch column. We will use the following claim.
Claim 5.5. In the coupling above, we have
E [ η ∣ ξ ] = c b − 1 ξ , where c b = [ 1 − ( 1 − 1 b ) b ] − 1 ∈ [ 1 , 1 1 − e − 1 ] ⊂ [ 1 , 2 ) . (31) \mathbb{E}[\eta\mid\xi]=c_b^{-1}\xi,\qquad\text{where }c_b=\left[1-\left(1-\frac{1}{b}\right)^b\right]^{-1}\in\left[1,\frac{1}{1-e^{-1}}\right]\subset[1,2). \tag*{(31)} E [ η ∣ ξ ] = c b − 1 ξ , where c b = [ 1 − ( 1 − b 1 ) b ] − 1 ∈ [ 1 , 1 − e − 1 1 ] ⊂ [ 1 , 2 ) . (31)
Proof. Conditioned on ξ = ϱ e u \xi=\varrho e_u ξ = ϱ e u , the coordinate η u \eta_u η u equals ϱ \varrho ϱ whenever η ≠ 0 \eta\ne0 η = 0 (which occurs with probability c b − 1 : = 1 − ( 1 − 1 / b ) b c_b^{-1}:=1-(1-1/b)^b c b − 1 := 1 − ( 1 − 1/ b ) b ), while all other coordinates u ′ ≠ u u'\ne u u ′ = u have conditional mean zero. □ \square □
Using the claim above and conditional Jensen’s inequality yields the following lemma.
Lemma 5.6. With all variables defined as in Lemma 2.4, and c b c_b c b as in Claim 5.5, we have
E [ ∥ Ω ~ I Ω ~ I c T W ∥ | Ω ζ , b , I ] ≤ c b E [ ∥ U W ∥ ] , (32) \mathbb{E}\left[\left\|\widetilde{\Omega}_{I}\widetilde{\Omega}_{I^c}^{\mathsf{T}}W\right\|\middle|\Omega_{\zeta,b,I}\right]\le c_b\mathbb{E}[\|UW\|], \tag*{(32)} E [ Ω I Ω I c T W Ω ζ , b , I ] ≤ c b E [ ∥ U W ∥ ] , (32)
where the N × ∣ J ∣ N\times|J| N × ∣ J ∣ matrix U U U has independent entries U i j = 1 ζ ∑ t = 1 ζ x t i j U_{ij}=\frac{1}{\zeta}\sum_{t=1}^{\zeta}x_{tij} U ij = ζ 1 ∑ t = 1 ζ x t ij , where all x t i j x_{tij} x t ij are independently drawn as
x t i j = { 1 with probability 1 2 b , − 1 with probability 1 2 b , 0 with probability 1 − 1 b . (33) x_{tij}=
\begin{cases}
1 & \text{with probability }\frac{1}{2b},\\
-1 & \text{with probability }\frac{1}{2b},\\
0 & \text{with probability }1-\frac{1}{b}.
\end{cases} \tag*{(33)} x t ij = ⎩ ⎨ ⎧ 1 − 1 0 with probability 2 b 1 , with probability 2 b 1 , with probability 1 − b 1 . (33)
Proof. Condition on Ω ζ , b , I \Omega_{\zeta,b,I} Ω ζ , b , I throughout the proof. First define the intermediate matrix U ~ \widetilde{U} U to be the matrix with entries U ~ i j = 1 ζ ∑ t = 1 ζ α i t ϱ i t η j t [ s i t ] \widetilde{U}_{ij}=\frac{1}{\zeta}\sum_{t=1}^{\zeta}\alpha_{it}\varrho_{it}\eta_{jt}[s_{it}] U ij = ζ 1 ∑ t = 1 ζ α i t ϱ i t η j t [ s i t ] , where each vector η j t \eta_{jt} η j t is coupled with ξ j t \xi_{jt} ξ j t as in Claim 5.5. Notice that each entry U ~ i j \widetilde{U}_{ij} U ij is independently drawn, but their distributions depend on the α i t \alpha_{it} α i t .
The operator norm is convex. So by Jensen’s inequality, additionally conditioning on the ξ j t \xi_{jt} ξ j t , and applying Claim 5.5, we have
c_b^{-1}\left\|\widetilde{\Omega}_I\widetilde{\Omega}_I^{\mathsf{T}}_cW\right\|
\leq\mathbb{E}_{\eta_{jt}}\left[\left\|\widetilde{U}W\right\|\mid\{\xi_{tj}\}\right]. \tag*{(34)}
Taking expectation over the ξ t j \xi_{tj} ξ t j on both sides, yields
\mathbb{E}\left[\left\|\widetilde{\Omega}_I\widetilde{\Omega}_I^{\mathsf{T}}_cW\right\|\right]
\leq c_b\mathbb{E}_{\eta_{jt},\xi_{jt}}\left[\left\|\widetilde{U}W\right\|\right]
=c_b\mathbb{E}_{\widetilde{U}}\left[\left\|\widetilde{U}W\right\|\right]. \tag*{(35)}
Now we will apply Jensen’s inequality again to reduce this to the matrix U U U from the claim which has i.i.d. entries. Observe that U ~ \widetilde{U} U has entries U ~ i j = 1 ζ ∑ t = 1 ζ x ~ t i j \widetilde{U}_{ij}=\frac{1}{\zeta}\sum_{t=1}^{\zeta}\widetilde{x}_{tij} U ij = ζ 1 ∑ t = 1 ζ x t ij , where
x ~ t i j = { α t i with probability 1 2 b − α t i with probability 1 2 b 0 with probability 1 − 1 b (36) \widetilde{x}_{tij}=
\begin{cases}
\alpha_{ti} & \text{with probability }\frac{1}{2b}\\
-\alpha_{ti} & \text{with probability }\frac{1}{2b}\\
0 & \text{with probability }1-\frac{1}{b}
\end{cases} \tag*{(36)} x t ij = ⎩ ⎨ ⎧ α t i − α t i 0 with probability 2 b 1 with probability 2 b 1 with probability 1 − b 1 (36)
Couple the variables x ~ t i j \widetilde{x}_{tij} x t ij and x t i j x_{tij} x t ij such that
x ~ t i j = α i t x t i j = { E [ x t i j ] α t i = 0 x t i j α t i = 1. (37) \widetilde{x}_{tij}=\alpha_{it}x_{tij}=
\begin{cases}
\mathbb{E}[x_{tij}] & \alpha_{ti}=0\\
x_{tij} & \alpha_{ti}=1.
\end{cases} \tag*{(37)} x t ij = α i t x t ij = { E [ x t ij ] x t ij α t i = 0 α t i = 1. (37)
Thus we have from Jensen’s inequality again that conditional on all the x t i j x_{tij} x t ij where α t i = 1 \alpha_{ti}=1 α t i = 1 ,
∥ U ~ W ∥ ≤ E x t i j : α t i = 0 [ ∥ U W ∥ ] . (38) \left\|\widetilde{U}W\right\|
\leq\mathbb{E}_{x_{tij}:\alpha_{ti}=0}\left[\left\|UW\right\|\right]. \tag*{(38)} U W ≤ E x t ij : α t i = 0 [ ∥ U W ∥ ] . (38)
Taking expectation over all the x t i j x_{tij} x t ij where α t i = 1 \alpha_{ti}=1 α t i = 1 yields
E U ~ [ ∥ U ~ W ∥ ] ≤ E U [ ∥ U W ∥ ] . (39) \mathbb{E}_{\widetilde{U}}\left[\left\|\widetilde{U}W\right\|\right]
\leq\mathbb{E}_{U}\left[\left\|UW\right\|\right]. \tag*{(39)} E U [ U W ] ≤ E U [ ∥ U W ∥ ] . (39)
Combining with (35) yields the lemma. □ \square □
We are now ready to prove Lemma 2.4, which we restate here for the reader’s convenience.
Lemma 5.7. There exists a universal constant C c g > 0 C_{\mathrm{cg}}>0 C cg > 0 such that the following holds. Let Ω ∈ R n × k \Omega\in\mathbb{R}^{n\times k} Ω ∈ R n × k (k = ζ b k=\zeta b k = ζ b ) be a SparseStack sketch with ζ ≥ 1 \zeta\geq1 ζ ≥ 1 blocks and b ≥ 2 ζ b\geq2\zeta b ≥ 2 ζ buckets per block. Let I , J ⊆ [ n ] I,J\subseteq[n] I , J ⊆ [ n ] be disjoint coordinate sets with ∣ I ∣ = N ≥ 1 |I|=N\geq1 ∣ I ∣ = N ≥ 1 , and let W ∈ R ∣ J ∣ × d W\in\mathbb{R}^{|J|\times d} W ∈ R ∣ J ∣ × d (d ≥ 1 d\geq1 d ≥ 1 ) be a fixed isometry. Let T ⊆ [ k ] T\subseteq[k] T ⊆ [ k ] be the set of rows in which Ω I \Omega_I Ω I has exactly one non-zero entry, and let Ω ~ T \widetilde{\Omega}^{\mathsf{T}} Ω T denote the restrictions of Ω T \Omega^{\mathsf{T}} Ω T to rows in T T T .
For every integer p ≥ max { 1 , log ( 2 d ) } p\geq\max\{1,\log(2d)\} p ≥ max { 1 , log ( 2 d )} , setting D p = d + 2 p + 1 D_p=d+2p+1 D p = d + 2 p + 1 , we have conditionally on Ω I \Omega_I Ω I :
E [ ∥ Ω ~ I Ω ~ J T W ∥ ∣ Ω I ] ≤ C c g ( N + D p k + p + 1 ζ 2 + N d ζ 2 b 3 ( ζ 3 ) ) . \mathbb{E}\left[\left\|\widetilde{\Omega}_I\widetilde{\Omega}_J^{\mathsf{T}}W\right\|\mid\Omega_I\right]
\leq C_{\mathrm{cg}}\left(\sqrt{\frac{N+D_p}{k}+\frac{p+1}{\zeta^2}}+\sqrt{\frac{Nd}{\zeta^2b^3}\left(\frac{\zeta}{3}\right)}\right). E [ Ω I Ω J T W ∣ Ω I ] ≤ C cg ( k N + D p + ζ 2 p + 1 + ζ 2 b 3 N d ( 3 ζ ) ) . If N = 0 N=0 N = 0 or d = 0 d=0 d = 0 , then the expression is trivially 0.
Proof. Fix Ω I \Omega_I Ω I . By Lemma 5.6,
E [ ∥ Ω ~ I Ω ~ I c ⊤ W ∥ ∣ Ω I ] ≤ c b E [ ∥ 1 ζ U W ∥ ] , where U i j = ∑ t = 1 ζ x t i j ( i ∈ I , j ∈ J ) , (40) \mathbb{E}\left[\left\|\widetilde{\Omega}_I\widetilde{\Omega}_{I^c}^{\top}W\right\| \mid\Omega_I\right] \le c_b\mathbb{E}\left[\left\|\frac{1}{\zeta}UW\right\|\right],
\qquad\text{where } U_{ij}=\sum_{t=1}^{\zeta}x_{tij}\quad(i\in I,\ j\in J), \tag*{(40)} E [ Ω I Ω I c ⊤ W ∣ Ω I ] ≤ c b E [ ζ 1 U W ] , where U ij = t = 1 ∑ ζ x t ij ( i ∈ I , j ∈ J ) , (40)
where all variables x t i j x_{tij} x t ij for t ∈ [ ζ ] t\in[\zeta] t ∈ [ ζ ] , i ∈ I i\in I i ∈ I , j ∈ J j\in J j ∈ J are mutually independent and distributed as { − 1 , 0 , 1 } \{-1,0,1\} { − 1 , 0 , 1 } with probabilities 1 2 b , 1 − 1 b , 1 2 b \frac{1}{2b},1-\frac{1}{b},\frac{1}{2b} 2 b 1 , 1 − b 1 , 2 b 1 . The entries U i j U_{ij} U ij of the N × ∣ J ∣ N\times|J| N × ∣ J ∣ matrix U U U are mutually independent and symmetric. For each pair ( i , j ) ∈ I × J (i,j)\in I\times J ( i , j ) ∈ I × J , let c i j = ∑ t = 1 ζ 1 { x t i j ≠ 0 } ∼ Bin ( ζ , 1 / b ) c_{ij}=\sum_{t=1}^{\zeta}\mathbf{1}_{\{x_{tij}\ne0\}}\sim\operatorname{Bin}(\zeta,1/b) c ij = ∑ t = 1 ζ 1 { x t ij = 0 } ∼ Bin ( ζ , 1/ b ) be the number of blocks in which ( i , j ) (i,j) ( i , j ) is active, and decompose U U U into low- and high-multiplicity parts:
U i j l o = U i j 1 { c i j ≤ 2 } , U i j h i = U i j 1 { c i j ≥ 3 } , U = U l o + U h i . U_{ij}^{\mathrm{lo}}=U_{ij}\mathbf{1}_{\{c_{ij}\le2\}},\qquad U_{ij}^{\mathrm{hi}}=U_{ij}\mathbf{1}_{\{c_{ij}\ge3\}},\qquad U=U^{\mathrm{lo}}+U^{\mathrm{hi}}. U ij lo = U ij 1 { c ij ≤ 2 } , U ij hi = U ij 1 { c ij ≥ 3 } , U = U lo + U hi .
High-multiplicity part U h i U^{\mathrm{hi}} U hi . The entries of U h i U^{\mathrm{hi}} U hi are mutually independent and centered, and conditional on c i j = c c_{ij}=c c ij = c , U i j U_{ij} U ij is a sum of c c c independent Rademacher signs so E [ U i j 2 ∣ c i j = c ] = c \mathbb{E}[U_{ij}^{2}\mid c_{ij}=c]=c E [ U ij 2 ∣ c ij = c ] = c . Using the elementary inequality c 1 { c ≥ 3 } ≤ 3 ( c 3 ) c\mathbf{1}_{\{c\ge3\}}\le3\binom{c}{3} c 1 { c ≥ 3 } ≤ 3 ( 3 c ) for integers c ≥ 0 c\ge0 c ≥ 0 and E [ ( c 11 3 ) ] = ( ζ 3 ) b − 3 \mathbb{E}\left[\binom{c_{11}}{3}\right]=\binom{\zeta}{3}b^{-3} E [ ( 3 c 11 ) ] = ( 3 ζ ) b − 3 , together with ∑ j ∈ J ∥ w j ∥ 2 = tr ( W ⊤ W ) = d \sum_{j\in J}\|w_j\|^{2}=\operatorname{tr}(W^{\top}W)=d ∑ j ∈ J ∥ w j ∥ 2 = tr ( W ⊤ W ) = d , we bound the expected squared Frobenius norm of U h i W / ζ U^{\mathrm{hi}}W/\zeta U hi W / ζ :
E [ ∥ 1 ζ U h i W ∥ F 2 ] = 1 ζ 2 ∑ i ∈ I ∑ j ∈ J E [ ( U i j h i ) 2 ] ∥ w j ∥ 2 = N d ζ 2 E [ c 11 1 { c 11 ≥ 3 } ] ≤ 3 N d 2 ζ b 3 ( ζ 3 ) . (41) \mathbb{E}\left[\left\|\frac{1}{\zeta}U^{\mathrm{hi}}W\right\|_{F}^{2}\right]
=\frac{1}{\zeta^{2}}\sum_{i\in I}\sum_{j\in J}\mathbb{E}\left[(U_{ij}^{\mathrm{hi}})^{2}\right]\|w_j\|^{2}
=\frac{Nd}{\zeta^{2}}\mathbb{E}\left[c_{11}\mathbf{1}_{\{c_{11}\ge3\}}\right]
\le\frac{3Nd}{2\zeta b^{3}}\binom{\zeta}{3}. \tag*{(41)} E [ ζ 1 U hi W F 2 ] = ζ 2 1 i ∈ I ∑ j ∈ J ∑ E [ ( U ij hi ) 2 ] ∥ w j ∥ 2 = ζ 2 N d E [ c 11 1 { c 11 ≥ 3 } ] ≤ 2 ζ b 3 3 N d ( 3 ζ ) . (41)
Low-multiplicity part U l o U^{\mathrm{lo}} U lo . Our goal will be to use Lemma 5.4 to control the operator norm of U l o W U^{\mathrm{lo}}W U lo W , but since the entries of U l o U^{\mathrm{lo}} U lo are distributed differently than those of 5.2, we will need to leverage convexity again to reduce to this case. Put B = b / ζ ≥ 2 B=b/\zeta\ge2 B = b / ζ ≥ 2 , using b ≥ 2 ζ b\ge2\zeta b ≥ 2 ζ . Each entry V = U i j l o V=U_{ij}^{\mathrm{lo}} V = U ij lo is symmetric, bounded in magnitude by ∣ V ∣ ≤ 2 |V|\le2 ∣ V ∣ ≤ 2 , and satisfies E [ ∣ V ∣ ] ≤ E [ ∑ t = 1 ζ ∣ x t i j ∣ ] = ζ / b = 1 / B \mathbb{E}[|V|]\le\mathbb{E}\left[\sum_{t=1}^{\zeta}|x_{tij}|\right]=\zeta/b=1/B E [ ∣ V ∣ ] ≤ E [ ∑ t = 1 ζ ∣ x t ij ∣ ] = ζ / b = 1/ B . Let η B ∈ { − 1 , 0 , 1 } \eta_B\in\{-1,0,1\} η B ∈ { − 1 , 0 , 1 } take values ± 1 \pm1 ± 1 with probability 1 / ( 2 B ) 1/(2B) 1/ ( 2 B ) each and 0 0 0 with probability 1 − 1 / B 1-1/B 1 − 1/ B . We seek to control the ∥ U l o W ∥ \|U^{\mathrm{lo}}W\| ∥ U lo W ∥ by replacing each copy of V V V with a copy of 2 η B 2\eta_B 2 η B .
Claim 5.8. For any convex function ϕ \phi ϕ , we have
E [ ϕ ( V ) ] ≤ E [ ϕ ( 2 η B ) ] . (42) \mathbb{E}[\phi(V)]\le\mathbb{E}[\phi(2\eta_B)]. \tag*{(42)} E [ ϕ ( V )] ≤ E [ ϕ ( 2 η B )] . (42)
Proof. Let ψ \psi ψ be the even part of ϕ \phi ϕ , namely 1 2 ( ϕ ( u ) + ϕ ( − u ) ) \frac{1}{2}(\phi(u)+\phi(-u)) 2 1 ( ϕ ( u ) + ϕ ( − u )) . Observe that ψ \psi ψ is convex because the average of two convex function is convex. Further, its convexity implies that it is non-decreasing on [ 0 , 2 ] [0,2] [ 0 , 2 ] , and satisfies for any symmetric X X X ,
E X [ ψ ( X ) ] = E X [ ϕ ( X ) ] , (43) \mathbb{E}_X[\psi(X)]=\mathbb{E}_X[\phi(X)], \tag*{(43)} E X [ ψ ( X )] = E X [ ϕ ( X )] , (43)
Interpolating linearly on [ 0 , 2 ] [0,2] [ 0 , 2 ] gives
E [ ψ ( V ) ] ≤ ( 1 − E [ ∣ V ∣ ] 2 ) ψ ( 0 ) + E [ ∣ V ∣ ] 2 ψ ( 2 ) (44) \mathbb{E}[\psi(V)]\le\left(1-\frac{\mathbb{E}[|V|]}{2}\right)\psi(0)+\frac{\mathbb{E}[|V|]}{2}\psi(2) \tag*{(44)} E [ ψ ( V )] ≤ ( 1 − 2 E [ ∣ V ∣ ] ) ψ ( 0 ) + 2 E [ ∣ V ∣ ] ψ ( 2 ) (44)
≤ ( 1 − 1 B ) ψ ( 0 ) + 1 B ψ ( 2 ) = E [ ψ ( 2 η B ) ] , (45) \le\left(1-\frac{1}{B}\right)\psi(0)+\frac{1}{B}\psi(2)=\mathbb{E}[\psi(2\eta_B)], \tag*{(45)} ≤ ( 1 − B 1 ) ψ ( 0 ) + B 1 ψ ( 2 ) = E [ ψ ( 2 η B )] , (45)
where the second line follows from the fact that ψ \psi ψ is non-decreasing on [ 0 , 2 ] [0,2] [ 0 , 2 ] and the bound E [ ∣ V ∣ ] ≤ 1 / B \mathbb{E}[|V|]\le1/B E [ ∣ V ∣ ] ≤ 1/ B . Finally, applying (43) twice with the symmetric distributions of V V V and 2 η B 2\eta_B 2 η B respectively, yields the desired claim.
:::
Applying Claim 5.8 entry by entry to the convex function U l o ↦ ∥ U l o W ∥ U^{\mathrm{lo}} \mapsto\lVert U^{\mathrm{lo}}W\rVert U lo ↦ ∥ U lo W ∥ yields
E [ ∥ U l o W ∥ ] ≤ 2 E [ ∥ X N W ∥ ] . \mathbb{E}\left[\left\lVert U^{\mathrm{lo}}W\right\rVert\right] \le2\mathbb{E}\left[\left\lVert X_NW\right\rVert\right]. E [ U lo W ] ≤ 2 E [ ∥ X N W ∥ ] .
where X N ∈ { − 1 , 0 , 1 } N × ∣ J ∣ X_N \in\{-1,0,1\}^{N\times|J|} X N ∈ { − 1 , 0 , 1 } N × ∣ J ∣ has independent entries with non-zero probability 1 / B 1/B 1/ B . Letting a = N / B + 1 ≥ 1 a=N/B+1\ge1 a = N / B + 1 ≥ 1 and m = a B ≥ N m=aB\ge N m = a B ≥ N , we may append m − N m-N m − N independent rows to X N X_N X N to form an m × ∣ J ∣ m\times|J| m × ∣ J ∣ matrix X m X_m X m , which can only increase the operator norm ∥ X N W ∥ ≤ ∥ X m W ∥ \lVert X_NW\rVert\le\lVert X_mW\rVert ∥ X N W ∥ ≤ ∥ X m W ∥ . Applying Lemma 5.4 to X m W X_mW X m W with B = b / ζ B=b/\zeta B = b / ζ gives
E [ ∥ 1 ζ U l o W ∥ ] ≤ 2 ζ ( E [ ∥ X m W ∥ 2 ] ) 1 / 2 ≤ O ( 1 ) ζ a + D p B + p ≤ O ( 1 ) N + D p k + p + 1 ζ 2 . (46) \mathbb{E}\left[\left\lVert\frac{1}{\zeta}U^{\mathrm{lo}}W\right\rVert\right]\le\frac{2}{\zeta}\left(\mathbb{E}\left[\left\lVert X_mW\right\rVert^2\right]\right)^{1/2}\le\frac{O(1)}{\zeta}\sqrt{a+\frac{D_p}{B}}+p\le O(1)\sqrt{\frac{N+D_p}{k}+\frac{p+1}{\zeta^2}}. \tag*{(46)} E [ ζ 1 U lo W ] ≤ ζ 2 ( E [ ∥ X m W ∥ 2 ] ) 1/2 ≤ ζ O ( 1 ) a + B D p + p ≤ O ( 1 ) k N + D p + ζ 2 p + 1 . (46)
Combining (40) , (41) , and (46) via the triangle inequality and Cauchy–Schwarz (E [ ∥ U h i W / ζ ∥ ] ≤ ( E [ ∥ U h i W / ζ ∥ F 2 ] ) 1 / 2 \mathbb{E}[\lVert U^{\mathrm{hi}}W/\zeta\rVert]\le(\mathbb{E}[\lVert U^{\mathrm{hi}}W/\zeta\rVert_{\mathcal{F}}^2])^{1/2} E [∥ U hi W / ζ ∥] ≤ ( E [∥ U hi W / ζ ∥ F 2 ] ) 1/2 ) completes the proof of Lemma 2.4.
High-moment bound for the full sketch
In this subsection, we establish a high-order (2 p 2p 2 p -th) moment bound for the full sketch G = U T Ω ζ , b Ω ζ , b T U G=U^{\mathsf T}\Omega_{\zeta,b}\Omega_{\zeta,b}^{\mathsf T}U G = U T Ω ζ , b Ω ζ , b T U of an arbitrary isometry U ∈ R n × D U\in\mathbb{R}^{n\times D} U ∈ R n × D . This bound is the main tool in proving Theorem 1.1 for the high-sparsity regime ζ = Ω ( log ( r ) ) \zeta=\Omega(\log(r)) ζ = Ω ( log ( r )) . In this regime, the sketch possesses enough independent blocks that it acts directly as an OSE on the entire subspace. The entrywise coupling (31) expresses the off-diagonal error G − I D G-I_D G − I D as the conditional expectation of the hollow matrix F F F , directly transferring the moment bound of Fact 5.2 to the SparseStack sketch.
Lemma 5.9. There exists an absolute constant C m o m > 0 C_{\mathrm{mom}}>0 C mom > 0 such that for any fixed isometry U ∈ R n × D U\in\mathbb{R}^{n\times D} U ∈ R n × D (D ≥ 1 D\ge1 D ≥ 1 ), any b ≥ 2 b\ge2 b ≥ 2 , ζ ≥ 1 \zeta\ge1 ζ ≥ 1 , and any integer p ≥ 1 p\ge1 p ≥ 1 , the matrix G = U T Ω ζ , b Ω ζ , b T U G=U^{\mathsf T}\Omega_{\zeta,b}\Omega_{\zeta,b}^{\mathsf T}U G = U T Ω ζ , b Ω ζ , b T U for a ζ \zeta ζ -block SparseStack Ω ζ , b ∈ R k × n \Omega_{\zeta,b}\in\mathbb{R}^{k\times n} Ω ζ , b ∈ R k × n (k = ζ b k=\zeta b k = ζ b ) satisfies
E [ ∥ G − I D ∥ 2 p ] ≤ D [ C m o m ( D + 2 p + 1 k + D + 2 p + 1 k + 2 p + 1 ζ ) ] 2 p . (47) \mathbb{E}\left[\lVert G-I_D\rVert^{2p}\right]\le D\left[C_{\mathrm{mom}}\left(\sqrt{\frac{D+2p+1}{k}}+\frac{D+2p+1}{k}+\frac{2p+1}{\zeta}\right)\right]^{2p}. \tag*{(47)} E [ ∥ G − I D ∥ 2 p ] ≤ D [ C mom ( k D + 2 p + 1 + k D + 2 p + 1 + ζ 2 p + 1 ) ] 2 p . (47) Proof. Since every row of Ω ζ , b \Omega_{\zeta,b} Ω ζ , b has deterministic unit Euclidean norm and U T U = I D U^{\mathsf T}U=I_D U T U = I D , the diagonal part of U T Ω ζ , b Ω ζ , b T U = 1 ζ ∑ t = 1 ζ U T Φ t Φ t T U U^{\mathsf T}\Omega_{\zeta,b}\Omega_{\zeta,b}^{\mathsf T}U=\frac{1}{\zeta}\sum_{t=1}^{\zeta}U^{\mathsf T}\Phi_t\Phi_t^{\mathsf T}U U T Ω ζ , b Ω ζ , b T U = ζ 1 ∑ t = 1 ζ U T Φ t Φ t T U is deterministically equal to ∑ i = 1 n u i u i T = I D \sum_{i=1}^{n}u_i u_i^{\mathsf T}=I_D ∑ i = 1 n u i u i T = I D . Thus G − I D G-I_D G − I D is a purely off-diagonal sum over distinct coordinates i ≠ i ′ i\ne i' i = i ′ :
G − I D = 1 ζ ∑ t = 1 ζ ∑ i ≠ i ′ ⟨ ξ t i , ξ t i ′ , u ⟩ u i u i ′ T . G-I_D=\frac{1}{\zeta}\sum_{t=1}^{\zeta}\sum_{i\ne i'}\langle\xi_{ti},\xi_{ti'},u\rangle u_i u_{i'}^{\mathsf T}. G − I D = ζ 1 t = 1 ∑ ζ i = i ′ ∑ ⟨ ξ t i , ξ t i ′ , u ⟩ u i u i ′ T . where ξ t i = ϱ t i e s t i ∈ R b \xi_{ti}=\varrho_{ti}e_{s_{ti}}\in\mathbb{R}^{b} ξ t i = ϱ t i e s t i ∈ R b is the i i i -th column vector of the sketching block Φ t T ∈ R b × n \Phi_t^{\mathsf T}\in\mathbb{R}^{b\times n} Φ t T ∈ R b × n . Applying the coupling (31) independently to each ( η t i , ξ t i ) (\eta_{ti},\xi_{ti}) ( η t i , ξ t i ) , independence across distinct coordinates i ≠ i ′ i\ne i' i = i ′ gives
G − I D = E [ c b 2 F ∣ { ξ t i } ] . G-I_D=\mathbb{E}\left[c_b^2F\mid\{\xi_{ti}\}\right]. G − I D = E [ c b 2 F ∣ { ξ t i } ] . where F F F is the off-diagonal matrix of Fact 5.2 with B = b B=b B = b , a = ζ a=\zeta a = ζ , and m = k = ζ b m=k=\zeta b m = k = ζ b . By conditional Jensen’s inequality for the convex function M ↦ ∥ M ∥ 2 p ≤ tr ( M 2 p ) M\mapsto\lVert M\rVert^{2p}\le\operatorname{tr}(M^{2p}) M ↦ ∥ M ∥ 2 p ≤ tr ( M 2 p ) , together with c b < 2 c_b<2 c b < 2 and Fact 5.2, we obtain (47) with C m o m = 12 ( 3 + 2 ) C_{\mathrm{mom}}=12(3+\sqrt{2}) C mom = 12 ( 3 + 2 ) .
Preserving the Heavy Coordinates
Our goal in this section is to prove Theorem 2.5, showing that the second-half sketch ( Ω ζ , b ( 2 ) ) T ∈ R k × n (\Omega^{(2)}_{\zeta,b})^{\mathsf{T}} \in\mathbb{R}^{k \times n} ( Ω ζ , b ( 2 ) ) T ∈ R k × n (k = ζ b k = \zeta b k = ζ b ) preserves the norm on the heavy coordinates y I y_I y I , for I ⊆ [ n ] I \subseteq[n] I ⊆ [ n ] of size ∣ I ∣ ≤ min { r , b / 16 } |I| \le\min\{r,b/16\} ∣ I ∣ ≤ min { r , b /16 } . For any y = Q x y = Qx y = Q x , decomposing y y y into its heavy coordinates h = ( Q x ) I h = (Qx)_I h = ( Q x ) I and complementary coordinates ( Q x ) I c = W f (Qx)_{I^c} = Wf ( Q x ) I c = W f (with W W W an orthonormal basis for col ( Q I c ) \operatorname{col}(Q_{I^c}) col ( Q I c ) and ∥ f ∥ = ∥ ( Q x ) I c ∥ \lVert f\rVert= \lVert(Qx)_{I^c}\rVert ∥ f ∥ = ∥( Q x ) I c ∥ ) splits the sketch as
( Ω ζ , b ( 2 ) ) T Q x = ( Ω ζ , b , I ( 2 ) ) T h + ( Ω ζ , b , I c ( 2 ) ) T W f , (48) (\Omega^{(2)}_{\zeta,b})^{\mathsf{T}}Qx = (\Omega^{(2)}_{\zeta,b,I})^{\mathsf{T}}h + (\Omega^{(2)}_{\zeta,b,I^c})^{\mathsf{T}}Wf, \tag*{(48)} ( Ω ζ , b ( 2 ) ) T Q x = ( Ω ζ , b , I ( 2 ) ) T h + ( Ω ζ , b , I c ( 2 ) ) T W f , (48)
Our main approach in showing that the norm of y I y_I y I is preserved by ( Ω ζ , b , I ( 2 ) ) T (\Omega^{(2)}_{\zeta,b,I})^{\mathsf{T}} ( Ω ζ , b , I ( 2 ) ) T will be finding rows in ( Ω ζ , b , I ( 2 ) ) T (\Omega^{(2)}_{\zeta,b,I})^{\mathsf{T}} ( Ω ζ , b , I ( 2 ) ) T which only have a single non-zero entry. To formalize this idea, we introduce the following definitions.
Definition 6.1 (Isolated bucket). For a matrix Ω T ∈ R k × N \Omega^{\mathsf{T}} \in\mathbb{R}^{k \times N} Ω T ∈ R k × N , we say that column i ∈ [ N ] i \in[N] i ∈ [ N ] lands in bucket r r r if Ω r i T ≠ 0 \Omega^{\mathsf{T}}_{ri} \ne0 Ω r i T = 0 . We say bucket r r r is isolated if the r r r th row of Ω T \Omega^{\mathsf{T}} Ω T has a single non-zero entry.
In Ω ζ , b , I ( 2 ) \Omega^{(2)}_{\zeta,b,I} Ω ζ , b , I ( 2 ) , if i ∈ I i \in I i ∈ I lands in an isolated bucket r r r , then ( ( Ω ζ , b , I ( 2 ) ) T y I ) r = ± y i ζ ((\Omega^{(2)}_{\zeta,b,I})^{\mathsf{T}}y_I)_r = \pm\frac{y_i}{\sqrt{\zeta}} (( Ω ζ , b , I ( 2 ) ) T y I ) r = ± ζ y i . Thus up to constants, the mass of coordinate y i y_i y i will be preserved by ( Ω ζ , b , I ( 2 ) ) T (\Omega^{(2)}_{\zeta,b,I})^{\mathsf{T}} ( Ω ζ , b , I ( 2 ) ) T whenever a constant fraction of the rows in buckets i i i lands in are isolated. We formalize this in the following definition.
Definition 6.2 (Good coordinate). For a matrix Ω T ∈ R k × N \Omega^{\mathsf{T}} \in\mathbb{R}^{k \times N} Ω T ∈ R k × N , we say that a coordinate i ∈ [ N ] i \in[N] i ∈ [ N ] is good if in at least a 11 12 \frac{11}{12} 12 11 fraction of the buckets i i i lands in are isolated.
Fix some i ∈ I i \in I i ∈ I . When b ≥ 16 ∣ I ∣ b \ge16|I| b ≥ 16∣ I ∣ , in each block Φ \Phi Φ of ( Ω ζ , b , I ( 2 ) ) T (\Omega^{(2)}_{\zeta,b,I})^{\mathsf{T}} ( Ω ζ , b , I ( 2 ) ) T , with probability at least 15 / 16 15/16 15/16 , i i i lands in an isolated bucket. Thus with probability 1 − e − Ω ( ζ ) 1-e^{-\Omega(\zeta)} 1 − e − Ω ( ζ ) , coordinate i i i is good.
At sparsity ζ = O ( log r ) \zeta= O(\sqrt{\log r}) ζ = O ( log r ) , however, the failure probability e − Ω ( ζ ) e^{-\Omega(\zeta)} e − Ω ( ζ ) is far too large for a union bound over all ∣ I ∣ ≈ r / ζ |I| \approx r/\zeta ∣ I ∣ ≈ r / ζ coordinates. We overcome this by splitting ( Ω ζ , b , I ( 2 ) ) T (\Omega^{(2)}_{\zeta,b,I})^{\mathsf{T}} ( Ω ζ , b , I ( 2 ) ) T into two independent groups of ζ 1 = ζ / 2 \zeta_1 = \zeta/2 ζ 1 = ζ /2 blocks. Our strategy is as follows:
Using the first group, we will argue that a 1 − e − Ω ( ζ ) 1-e^{-\Omega(\zeta)} 1 − e − Ω ( ζ ) fraction of the coordinates in I I I are good, and thus their norm is mostly preserved (Lemma 6.3).
For the remaining coordinates B ⊂ I B \subset I B ⊂ I which are not good for the first group of block, we will consider the second group of blocks, restricted to the columns in B B B . Since ∣ B ∣ ≲ r e − Ω ( ζ ) |B| \lesssim r e^{-\Omega(\zeta)} ∣ B ∣ ≲ r e − Ω ( ζ ) , on this new block, the coordinates in B B B will have fewer collisions, and thus, we will be able to show that with high probability, all elements of B B B are good in this submatrix. We show this in Lemma 6.4. We can then again use Lemma 6.3 to show that their norm is preserved.
Finally, we put together these estimates to prove Theorem 2.5.
The following lemma analyzes this single-group behavior for an arbitrary coordinate set S ⊆ [ n ] S \subseteq[n] S ⊆ [ n ] of size ∣ S ∣ ≤ r |S| \le r ∣ S ∣ ≤ r and complementary isometry W ∈ R ∣ S c ∣ × d W \in\mathbb{R}^{|S^c|\times d} W ∈ R ∣ S c ∣ × d .
Lemma 6.3. Fix any constants t , δ r e p > 0 t,\delta_{\mathrm{rep}} > 0 t , δ rep > 0 , a set S ⊆ [ n ] S \subseteq[n] S ⊆ [ n ] , and an isometry W ∈ R ∣ S c ∣ × d W \in\mathbb{R}^{|S^c|\times d} W ∈ R ∣ S c ∣ × d for d < 2 r d < 2r d < 2 r . Suppose Ω T ∈ R k × n \Omega^{\mathsf{T}} \in\mathbb{R}^{k \times n} Ω T ∈ R k × n is a SparseStack matrix for ζ = Ω ( log r ) \zeta= \Omega(\sqrt{\log r}) ζ = Ω ( log r ) and k = ζ b = Ω ( r ) k = \zeta b = \Omega(r) k = ζ b = Ω ( r ) . With probability at least 1 − δ r e p / 4 1-\delta_{\mathrm{rep}}/4 1 − δ rep /4 over Ω \Omega Ω , we have simultaneously for all h ∈ R ∣ S ∣ h \in\mathbb{R}^{|S|} h ∈ R ∣ S ∣ and f ∈ R d f \in\mathbb{R}^d f ∈ R d :
∥ Ω S T h + Ω S c T W f ∥ 2 + t ∥ f ∥ 2 ≥ ( 11 12 − t ) ∥ h G ∥ 2 − t ∥ h B ∥ 2 , (49) \left\lVert\Omega_S^{\mathsf{T}}h + \Omega_{S^c}^{\mathsf{T}}Wf \right\rVert^2 + t\lVert f\rVert^2 \ge\left(\frac{11}{12}-t\right)\lVert h_G\rVert^2 - t\lVert h_B\rVert^2, \tag*{(49)} Ω S T h + Ω S c T W f 2 + t ∥ f ∥ 2 ≥ ( 12 11 − t ) ∥ h G ∥ 2 − t ∥ h B ∥ 2 , (49) where G G G contains all good coordinates in S S S (for the matrix Ω ζ , b , S \Omega_{\zeta,b,S} Ω ζ , b , S ), and B = S ∖ G B = S \setminus G B = S ∖ G .
Proof. Set L = log ( e r ) ≥ 1 L=\log(er)\ge1 L = log ( er ) ≥ 1 . Let T ⊆ [ k ] T\subseteq[k] T ⊆ [ k ] be the set of all isolated buckets of Ω S T \Omega_S^{\mathsf T} Ω S T , and define Ω ~ S T \widetilde{\Omega}_S^{\mathsf T} Ω S T and Ω ~ S c T \widetilde{\Omega}_{S^c}^{\mathsf T} Ω S c T respectively be the restrictions of Ω S T \Omega_S^{\mathsf T} Ω S T and Ω S c T \Omega_{S^c}^{\mathsf T} Ω S c T to rows in T T T . We apply Lemma 2.4 with block count ζ \zeta ζ , ∣ S ∣ ≤ r |S|\le r ∣ S ∣ ≤ r , d ≤ 2 r d\le2r d ≤ 2 r , and p = ⌈ log ( 4 e r ) ⌉ = Θ ( L ) p=\lceil\log(4er)\rceil=\Theta(L) p = ⌈ log ( 4 er )⌉ = Θ ( L ) . Lemma 2.4 yields
E [ ∥ Ω ~ S Ω ~ S c T W ∥ | Ω S ] ≤ O ( r k + L ζ + r 2 ζ b 3 ) . \mathbb{E}\left[\left\|\widetilde{\Omega}_S\widetilde{\Omega}_{S^c}^{\mathsf T}W\right\|\middle|\Omega_S\right]\le O\left(\sqrt{\frac{r}{k}}+\frac{L}{\zeta}+\sqrt{\frac{r^2\zeta}{b^3}}\right). E [ Ω S Ω S c T W Ω S ] ≤ O ( k r + ζ L + b 3 r 2 ζ ) .
Choosing the implied constants in ζ = Ω ( L ) \zeta=\Omega(\sqrt{L}) ζ = Ω ( L ) and k = Ω ( r ) k=\Omega(r) k = Ω ( r ) sufficiently large relative to t , δ r e p t,\delta_{\mathrm{rep}} t , δ rep , Markov’s inequality ensures that conditionally on Ω S \Omega_S Ω S , the event { ∥ Ω ~ S Ω ~ S c T W ∥ ≤ t } \left\{\left\|\widetilde{\Omega}_S\widetilde{\Omega}_{S^c}^{\mathsf T}W\right\|\le t\right\} { Ω S Ω S c T W ≤ t } occurs with probability at least 1 − δ r e p / 4 1-\delta_{\mathrm{rep}}/4 1 − δ rep /4 .
Furthermore, because Ω ~ S T \widetilde{\Omega}_S^{\mathsf T} Ω S T retains only the isolated buckets, every row of Ω ~ S T \widetilde{\Omega}_S^{\mathsf T} Ω S T contains at most one non-zero entry, so distinct columns of Ω ~ S T \widetilde{\Omega}_S^{\mathsf T} Ω S T have pairwise disjoint supports. Hence, the columns are mutually orthogonal, which implies ∥ Ω ~ S T h ∥ 2 = ∑ i ∈ S a i ζ h i 2 \left\|\widetilde{\Omega}_S^{\mathsf T}h\right\|^2=\sum_{i\in S}\frac{a_i}{\zeta}h_i^2 Ω S T h 2 = ∑ i ∈ S ζ a i h i 2 , where a i a_i a i is the number of isolated buckets of Ω S T \Omega_S^{\mathsf T} Ω S T that i i i lands in. Thus by definition of good, a i ≥ 11 12 ζ a_i\ge\frac{11}{12}\zeta a i ≥ 12 11 ζ for i ∈ G i\in G i ∈ G .
On the event { ∥ Ω ~ S Ω ~ S c T W ∥ ≤ t } \left\{\left\|\widetilde{\Omega}_S\widetilde{\Omega}_{S^c}^{\mathsf T}W\right\|\le t\right\} { Ω S Ω S c T W ≤ t } , dropping coordinates outside T T T and completing the square with 2 ∥ h ∥ ∥ f ∥ ≤ ∥ h ∥ 2 + ∥ f ∥ 2 = ∥ h G ∥ 2 + ∥ h B ∥ 2 + ∥ f ∥ 2 2\|h\|\|f\|\le\|h\|^2+\|f\|^2=\|h_G\|^2+\|h_B\|^2+\|f\|^2 2∥ h ∥∥ f ∥ ≤ ∥ h ∥ 2 + ∥ f ∥ 2 = ∥ h G ∥ 2 + ∥ h B ∥ 2 + ∥ f ∥ 2 yields:
∥ Ω S T h + Ω S c T W f ∥ 2 + t ∥ f ∥ 2 ≥ ∥ Ω ~ S T h + Ω ~ S c T W f ∥ 2 + t ∥ f ∥ 2 = ∑ i ∈ S a i ζ h i 2 + 2 h T Ω ~ S Ω ~ S c T W f + ∥ Ω ~ S c T W f ∥ 2 + t ∥ f ∥ 2 ≥ 11 12 ∥ h G ∥ 2 − 2 t ∥ h ∥ ∥ f ∥ + t ∥ f ∥ 2 ≥ ( 11 12 − t ) ∥ h G ∥ 2 − t ∥ h B ∥ 2 , \begin{aligned}
\left\|\Omega_S^{\mathsf T}h+\Omega_{S^c}^{\mathsf T}Wf\right\|^2+t\|f\|^2
\ge\left\|\widetilde{\Omega}_S^{\mathsf T}h+\widetilde{\Omega}_{S^c}^{\mathsf T}Wf\right\|^2+t\|f\|^2 \\
&= \sum_{i\in S}\frac{a_i}{\zeta}h_i^2+2h^{\mathsf T}\widetilde{\Omega}_S\widetilde{\Omega}_{S^c}^{\mathsf T}Wf+\left\|\widetilde{\Omega}_{S^c}^{\mathsf T}Wf\right\|^2+t\|f\|^2 \\
&\ge\frac{11}{12}\|h_G\|^2-2t\|h\|\|f\|+t\|f\|^2 \\
&\ge\left(\frac{11}{12}-t\right)\|h_G\|^2-t\|h_B\|^2,
\end{aligned} Ω S T h + Ω S c T W f 2 + t ∥ f ∥ 2 ≥ Ω S T h + Ω S c T W f 2 + t ∥ f ∥ 2 = i ∈ S ∑ ζ a i h i 2 + 2 h T Ω S Ω S c T W f + Ω S c T W f 2 + t ∥ f ∥ 2 ≥ 12 11 ∥ h G ∥ 2 − 2 t ∥ h ∥∥ f ∥ + t ∥ f ∥ 2 ≥ ( 12 11 − t ) ∥ h G ∥ 2 − t ∥ h B ∥ 2 ,
completing the proof.
It remains to preserve the norm of the coordinates that fail to land in sufficiently many isolated buckets in the first group. Among the coordinates of I I I , the set of coordinates that fail to achieve isolation in at least 11 12 ζ 1 \frac{11}{12}\zeta_1 12 11 ζ 1 blocks of Group 1, denoted by B B B , has expected size E [ ∣ B ∣ ] ≤ ∣ I ∣ e − Ω ( ζ ) \mathbb{E}[|B|]\le|I|e^{-\Omega(\zeta)} E [ ∣ B ∣ ] ≤ ∣ I ∣ e − Ω ( ζ ) . Our goal is to ensure that their norms are fully preserved in the second group. While this was impossible for the full set I I I , the size of B B B is now small enough that a union bound over all coordinates in B B B becomes applicable. The following lemma formalizes this via a balls-into-bins analysis across the two groups.
Lemma 6.4. For each i ∈ I i\in I i ∈ I , let a i a_i a i be the number of blocks in Group 1 where i i i lands in a bucket containing no other element of I I I . For ζ = Ω ( log r ) \zeta=\Omega(\sqrt{\log r}) ζ = Ω ( log r ) and ∣ I ∣ ≤ min { r , b / 16 } |I|\le\min\{r,b/16\} ∣ I ∣ ≤ min { r , b /16 } , two independent ζ 1 \zeta_1 ζ 1 -block groups satisfy:
(i) In Group 1, the bad set B = { i ∈ I : a i < 11 12 ζ 1 } B=\{i\in I:a_i<\frac{11}{12}\zeta_1\} B = { i ∈ I : a i < 12 11 ζ 1 } satisfies ∣ B ∣ ≤ ∣ I ∣ e − Ω ( ζ ) |B|\le|I|e^{-\Omega(\zeta)} ∣ B ∣ ≤ ∣ I ∣ e − Ω ( ζ ) with probability at least 1 − δ r e p / 4 1-\delta_{\mathrm{rep}}/4 1 − δ rep /4 .
(ii) Conditionally on any such Group 1, with probability at least 1 − δ r e p / 4 1-\delta_{\mathrm{rep}}/4 1 − δ rep /4 over Group 2, every coordinate in B B B lands in a bucket containing no other element of B B B in at least 11 12 ζ 1 \frac{11}{12}\zeta_1 12 11 ζ 1 blocks of Group 2.
Proof. Set c 1 = 1 / 2304 c_1 = 1/2304 c 1 = 1/2304 . For part (i), since ∣ I ∣ ≤ b / 16 |I| \le b/16 ∣ I ∣ ≤ b /16 , in each block of Group 1 a coordinate i ∈ I i \in I i ∈ I avoids colliding with the other ∣ I ∣ − 1 |I|-1 ∣ I ∣ − 1 elements of I I I with probability at least 1 − ( ∣ I ∣ − 1 ) / b ≥ 15 / 16 1-(|I|-1)/b \ge15/16 1 − ( ∣ I ∣ − 1 ) / b ≥ 15/16 . Thus E [ a i ] ≥ 15 16 ζ 1 \mathbb{E}[a_i] \ge\frac{15}{16}\zeta_1 E [ a i ] ≥ 16 15 ζ 1 . Changing a single column index in block t t t alters a i a_i a i by at most 1. Applying McDiarmid’s inequality (Fact 3.1) gives
P [ i ∈ B ] = P [ a i < 11 12 ζ 1 ] ≤ exp ( − 2 ( 1 48 ) 2 ζ 1 ) = e − 2 c 1 ζ 1 . \mathbb{P}[i \in B] = \mathbb{P}\left[a_i < \frac{11}{12}\zeta_1\right] \le\exp\left(-2\left(\frac{1}{48}\right)^2\zeta_1\right) = e^{-2c_1\zeta_1}. P [ i ∈ B ] = P [ a i < 12 11 ζ 1 ] ≤ exp ( − 2 ( 48 1 ) 2 ζ 1 ) = e − 2 c 1 ζ 1 .
Thus E [ ∣ B ∣ ] ≤ ∣ I ∣ e − 2 c 1 ζ 1 \mathbb{E}[|B|] \le|I|e^{-2c_1\zeta_1} E [ ∣ B ∣ ] ≤ ∣ I ∣ e − 2 c 1 ζ 1 , and Markov’s inequality yields Pr [ ∣ B ∣ > ∣ I ∣ e − c 1 ζ 1 ] ≤ e − c 1 ζ 1 ≤ δ r e p / 4 \Pr[|B| > |I|e^{-c_1\zeta_1}] \le e^{-c_1\zeta_1} \le\delta_{\mathrm{rep}}/4 Pr [ ∣ B ∣ > ∣ I ∣ e − c 1 ζ 1 ] ≤ e − c 1 ζ 1 ≤ δ rep /4 for ζ = Ω ( log r ) \zeta= \Omega(\sqrt{\log r}) ζ = Ω ( log r ) with sufficiently large constant.
For part (ii), condition on any realization of Group 1 satisfying ∣ B ∣ ≤ ∣ I ∣ e − c 1 ζ 1 |B| \le|I|e^{-c_1\zeta_1} ∣ B ∣ ≤ ∣ I ∣ e − c 1 ζ 1 . For any fixed i ∈ B i \in B i ∈ B and any block t t t of Group 2, the probability that i i i collides with another element of B B B in block t t t is at most
∣ B ∣ b ≤ ∣ I ∣ b e − c 1 ζ 1 ≤ 1 16 e − c 1 ζ 1 . \frac{|B|}{b} \le\frac{|I|}{b}e^{-c_1\zeta_1} \le\frac{1}{16}e^{-c_1\zeta_1}. b ∣ B ∣ ≤ b ∣ I ∣ e − c 1 ζ 1 ≤ 16 1 e − c 1 ζ 1 .
Across the ζ 1 \zeta_1 ζ 1 independent blocks of Group 2, coordinate i ∈ B i \in B i ∈ B fails to achieve isolation from B ∖ { i } B \setminus\{i\} B ∖ { i } in at least 11 12 ζ 1 \frac{11}{12}\zeta_1 12 11 ζ 1 blocks only if at least u = ζ 1 / 12 u = \zeta_1/12 u = ζ 1 /12 collisions occur. Union-bounding over all ( ζ 1 u ) ≤ 2 ζ 1 \binom{\zeta_1}{u} \le2^{\zeta_1} ( u ζ 1 ) ≤ 2 ζ 1 subsets of blocks of size u u u , there is an absolute constant c 2 > 0 c_2 > 0 c 2 > 0 such that the probability of failing isolation in ≥ ζ 1 / 12 \ge\zeta_1/12 ≥ ζ 1 /12 blocks is at most
2 ζ 1 ( ∣ B ∣ b ) ζ 1 / 12 ≤ 2 ζ 1 ( e − c 1 ζ 1 16 ) ζ 1 / 12 ≤ e − c 2 ζ 1 2 . 2^{\zeta_1}\left(\frac{|B|}{b}\right)^{\zeta_1/12} \le2^{\zeta_1}\left(\frac{e^{-c_1\zeta_1}}{16}\right)^{\zeta_1/12} \le e^{-c_2\zeta_1^2}. 2 ζ 1 ( b ∣ B ∣ ) ζ 1 /12 ≤ 2 ζ 1 ( 16 e − c 1 ζ 1 ) ζ 1 /12 ≤ e − c 2 ζ 1 2 .
Taking a union bound over all ∣ B ∣ ≤ r |B| \le r ∣ B ∣ ≤ r coordinates in B B B gives
P [ ∃ i ∈ B isolated in < 11 12 ζ 1 blocks of Group 2 ∣ Group 1 ] ≤ r e − c 2 ζ 1 2 ≤ δ r e p 4 \mathbb{P}\left[\exists i \in B\text{ isolated in }<\frac{11}{12}\zeta_1\text{ blocks of Group 2}\mid\text{Group 1}\right] \le re^{-c_2\zeta_1^2} \le\frac{\delta_{\mathrm{rep}}}{4} P [ ∃ i ∈ B isolated in < 12 11 ζ 1 blocks of Group 2 ∣ Group 1 ] ≤ r e − c 2 ζ 1 2 ≤ 4 δ rep
for ζ = Ω ( log r ) \zeta= \Omega(\sqrt{\log r}) ζ = Ω ( log r ) with sufficiently large implied constant, completing the proof. \quad\Box
We are now ready to combine Lemmas 6.3 and 6.4 to prove Theorem 2.5.
Theorem 2.5. Fix any constants t 0 ∈ ( 0 , 1 / 4 ) t_0 \in(0,1/4) t 0 ∈ ( 0 , 1/4 ) and δ r e p ∈ ( 0 , 1 ) \delta_{\mathrm{rep}} \in(0,1) δ rep ∈ ( 0 , 1 ) . There exists a constant β = Ω ( 1 ) \beta= \Omega(1) β = Ω ( 1 ) such that for every fixed isometry Q ∈ R n × r Q \in\mathbb{R}^{n\times r} Q ∈ R n × r with r ≥ 2 r \ge2 r ≥ 2 , every integer pair ζ = Ω ( log r ) \zeta= \Omega(\sqrt{\log r}) ζ = Ω ( log r ) and k = ζ b = Ω ( r ) k = \zeta b = \Omega(r) k = ζ b = Ω ( r ) , and every coordinate set I ⊆ [ n ] I \subseteq[n] I ⊆ [ n ] of size ∣ I ∣ ≤ min { r , b / 16 } |I| \le\min\{r,b/16\} ∣ I ∣ ≤ min { r , b /16 } , the sketch Y = ( Ω ζ , b ( 2 ) ) ⊤ Q ∈ R k × r Y = (\Omega_{\zeta,b}^{(2)})^\top Q \in\mathbb{R}^{k\times r} Y = ( Ω ζ , b ( 2 ) ) ⊤ Q ∈ R k × r satisfies, with probability at least 1 − δ r e p 1-\delta_{\mathrm{rep}} 1 − δ rep ,
∥ Y x ∥ 2 ≥ β ∥ ( Q x ) I ∥ 2 − t 0 ∥ ( Q x ) I c ∥ 2 for all x ∈ R r , \|Yx\|^2 \ge\beta\|(Qx)_I\|^2 - t_0\|(Qx)_{I^c}\|^2 \qquad\text{for all }x \in\mathbb{R}^r, ∥ Y x ∥ 2 ≥ β ∥ ( Q x ) I ∥ 2 − t 0 ∥ ( Q x ) I c ∥ 2 for all x ∈ R r , or equivalently, Y ⊤ Y + ( β + t 0 ) Q I c ⊤ Q I c ⪰ β I r Y^\top Y + (\beta+t_0)Q_{I^c}^\top Q_{I^c} \succeq\beta I_r Y ⊤ Y + ( β + t 0 ) Q I c ⊤ Q I c ⪰ β I r .
Proof. Let I c = [ n ] ∖ I I^c = [n]\setminus I I c = [ n ] ∖ I , and let W ∈ R ∣ I c ∣ × d W \in\mathbb{R}^{|I^c|\times d} W ∈ R ∣ I c ∣ × d with d = rank ( Q I c ) ≤ r d = \operatorname{rank}(Q_{I^c}) \le r d = rank ( Q I c ) ≤ r be an orthonormal basis for col ( Q I c ) \operatorname{col}(Q_{I^c}) col ( Q I c ) , so that ( Q x ) I c = W f (Qx)_{I^c} = Wf ( Q x ) I c = W f where f ∈ R d f \in\mathbb{R}^d f ∈ R d satisfies ∥ f ∥ = ∥ ( Q x ) I c ∥ \|f\| = \|(Qx)_{I^c}\| ∥ f ∥ = ∥ ( Q x ) I c ∥ . Write h = ( Q x ) I ∈ R ∣ I ∣ h = (Qx)_I \in\mathbb{R}^{|I|} h = ( Q x ) I ∈ R ∣ I ∣ , so that ∥ h ∥ 2 + ∥ f ∥ 2 = ∥ Q x ∥ 2 = ∥ x ∥ 2 \|h\|^2 + \|f\|^2 = \|Qx\|^2 = \|x\|^2 ∥ h ∥ 2 + ∥ f ∥ 2 = ∥ Q x ∥ 2 = ∥ x ∥ 2 .
Partition the ζ \zeta ζ blocks of Ω ζ , b ( 2 ) \Omega_{\zeta,b}^{(2)} Ω ζ , b ( 2 ) into two disjoint groups, Group 1 and Group 2, of ζ 1 = ζ / 2 \zeta_1 = \zeta/2 ζ 1 = ζ /2 blocks, with corresponding sketching matrices Ω 1 , Ω 2 ∈ R n × k 1 \Omega_1,\Omega_2 \in\mathbb{R}^{n\times k_1} Ω 1 , Ω 2 ∈ R n × k 1 (k 1 = ζ 1 b = ζ 2 b k_1 = \zeta_1b = \frac{\zeta}{2}b k 1 = ζ 1 b = 2 ζ b ). Then
∥ Y x ∥ 2 = ζ 1 ζ ( ∥ Ω 1 , I ⊤ h + Ω 1 , I c ⊤ W f ∥ 2 + ∥ Ω 2 , I ⊤ h + Ω 2 , I c ⊤ W f ∥ 2 ) . \|Yx\|^2 = \frac{\zeta_1}{\zeta}\left(\left\|\Omega_{1,I}^\top h+\Omega_{1,I^c}^\top Wf\right\|^2+\left\|\Omega_{2,I}^\top h+\Omega_{2,I^c}^\top Wf\right\|^2\right). ∥ Y x ∥ 2 = ζ ζ 1 ( Ω 1 , I ⊤ h + Ω 1 , I c ⊤ W f 2 + Ω 2 , I ⊤ h + Ω 2 , I c ⊤ W f 2 ) .
In Group 1, we apply Lemma 6.3 with target set S = I S = I S = I and isometry W W W . By Lemma 6.4(i), Group 1 isolates a good set G ⊆ I G \subseteq I G ⊆ I , leaving a bad set B = I ∖ G B = I \setminus G B = I ∖ G of size ∣ B ∣ ≤ ∣ I ∣ e − c 1 ζ 1 |B| \le|I|e^{-c_1\zeta_1} ∣ B ∣ ≤ ∣ I ∣ e − c 1 ζ 1 outside a failure event of probability at most δ r e p / 4 \delta_{\mathrm{rep}}/4 δ rep /4 . By Lemma 6.3 with t = t 0 t = t_0 t = t 0 , outside another failure event of probability at most δ r e p / 4 \delta_{\mathrm{rep}}/4 δ rep /4 , for all x ∈ R r x \in\mathbb{R}^r x ∈ R r :
∥ Ω 1 , I h + Ω 1 , I c T W f ∥ 2 + t 0 ∥ f ∥ 2 ≥ ( 11 12 − t 0 ) ∥ h G ∥ 2 − t 0 ∥ h B ∥ 2 . (50) \left\|\Omega_{1,I}h+\Omega_{1,I^c}^{\mathsf{T}}Wf\right\|^2+t_0\|f\|^2\geq\left(\frac{11}{12}-t_0\right)\|h_G\|^2-t_0\|h_B\|^2. \tag*{(50)} Ω 1 , I h + Ω 1 , I c T W f 2 + t 0 ∥ f ∥ 2 ≥ ( 12 11 − t 0 ) ∥ h G ∥ 2 − t 0 ∥ h B ∥ 2 . (50)
Next, condition on Group 1 on the event where ∣ B ∣ ≤ ∣ I ∣ e − c 1 ζ 1 |B| \le|I|e^{-c_1\zeta_1} ∣ B ∣ ≤ ∣ I ∣ e − c 1 ζ 1 and (50) hold. In Group 2, we set B B B as the target set (S = B S = B S = B ) and absorb G G G into the complementary coordinates via the augmented isometry
W + = diag ( I G , W ) ∈ R ∣ G ∪ I c ∣ × ( ∣ G ∣ + d ) , W_+ = \operatorname{diag}(I_G,W)\in\mathbb{R}^{|G\cup I^c|\times(|G|+d)}, W + = diag ( I G , W ) ∈ R ∣ G ∪ I c ∣ × ( ∣ G ∣ + d ) ,
which has rank ∣ G ∣ + d ≤ ∣ I ∣ + d ≤ 2 r |G|+d\leq|I|+d\leq2r ∣ G ∣ + d ≤ ∣ I ∣ + d ≤ 2 r and is fixed conditionally on Group 1. The complementary vector becomes f + = ( h G , f ) f_+=(h_G,f) f + = ( h G , f ) , which satisfies ∥ f + ∥ 2 = ∥ h G ∥ 2 + ∥ f ∥ 2 \|f_+\|^2=\|h_G\|^2+\|f\|^2 ∥ f + ∥ 2 = ∥ h G ∥ 2 + ∥ f ∥ 2 . By Lemma 6.4(ii), every coordinate in B B B is isolated in Group 2 with conditional probability at least 1 − δ r e p / 4 1-\delta_{\mathrm{rep}}/4 1 − δ rep /4 , so the bad set in Group 2 is empty. Applying Lemma 6.3 to Group 2 with target set S = B S = B S = B , isometry W + W_+ W + , and t = t 0 t = t_0 t = t 0 shows that outside a conditional failure event of probability at most δ r e p / 4 \delta_{\mathrm{rep}}/4 δ rep /4 , for all x ∈ R r x \in\mathbb{R}^r x ∈ R r :
∥ Ω 2 , I h + Ω 2 , I c T W f ∥ 2 + t 0 ( ∥ h G ∥ 2 + ∥ f ∥ 2 ) ≥ ( 11 12 − t 0 ) ∥ h B ∥ 2 . (51) \left\|\Omega_{2,I}h+\Omega_{2,I^c}^{\mathsf{T}}Wf\right\|^2+t_0\left(\|h_G\|^2+\|f\|^2\right)\geq\left(\frac{11}{12}-t_0\right)\|h_B\|^2. \tag*{(51)} Ω 2 , I h + Ω 2 , I c T W f 2 + t 0 ( ∥ h G ∥ 2 + ∥ f ∥ 2 ) ≥ ( 12 11 − t 0 ) ∥ h B ∥ 2 . (51)
Adding (50) and (51) cancels − t 0 ∥ h G ∥ 2 -t_0\|h_G\|^2 − t 0 ∥ h G ∥ 2 and − t 0 ∥ h B ∥ 2 -t_0\|h_B\|^2 − t 0 ∥ h B ∥ 2 against the ( 11 12 − t 0 ) \left(\frac{11}{12}-t_0\right) ( 12 11 − t 0 ) terms, giving
∥ Ω 1 , I h + Ω 1 , I c T W f ∥ 2 + ∥ Ω 2 , I h + Ω 2 , I c T W f ∥ 2 + 2 t 0 ∥ f ∥ 2 ≥ ( 11 12 − 2 t 0 ) ( ∥ h G ∥ 2 + ∥ h B ∥ 2 ) = ( 11 12 − 2 t 0 ) ∥ h ∥ 2 . \begin{aligned}
\left\|\Omega_{1,I}h+\Omega_{1,I^c}^{\mathsf{T}}Wf\right\|^2+\left\|\Omega_{2,I}h+\Omega_{2,I^c}^{\mathsf{T}}Wf\right\|^2+2t_0\|f\|^2
&\geq\left(\frac{11}{12}-2t_0\right)\left(\|h_G\|^2+\|h_B\|^2\right) \\
&=\left(\frac{11}{12}-2t_0\right)\|h\|^2.
\end{aligned} Ω 1 , I h + Ω 1 , I c T W f 2 + Ω 2 , I h + Ω 2 , I c T W f 2 + 2 t 0 ∥ f ∥ 2 ≥ ( 12 11 − 2 t 0 ) ( ∥ h G ∥ 2 + ∥ h B ∥ 2 ) = ( 12 11 − 2 t 0 ) ∥ h ∥ 2 .
Therefore,
∥ Y x ∥ 2 + t 0 ∥ f ∥ 2 ≥ β ∥ h ∥ 2 \|Yx\|^2+t_0\|f\|^2\geq\beta\|h\|^2 ∥ Y x ∥ 2 + t 0 ∥ f ∥ 2 ≥ β ∥ h ∥ 2
with β = ( 11 24 − t 0 ) > 0 \beta=\left(\frac{11}{24}-t_0\right)>0 β = ( 24 11 − t 0 ) > 0 . Substituting ∥ h ∥ 2 = ∥ x ∥ 2 − ∥ f ∥ 2 \|h\|^2=\|x\|^2-\|f\|^2 ∥ h ∥ 2 = ∥ x ∥ 2 − ∥ f ∥ 2 and ∥ f ∥ 2 = ∥ ( Q x ) I c ∥ 2 \|f\|^2=\|(Qx)_{I^c}\|^2 ∥ f ∥ 2 = ∥ ( Q x ) I c ∥ 2 gives
∥ Y x ∥ 2 + ( β + t 0 ) ∥ ( Q x ) I c ∥ 2 ≥ β ∥ x ∥ 2 for all x ∈ R r . \|Yx\|^2+(\beta+t_0)\|(Qx)_{I^c}\|^2\geq\beta\|x\|^2\qquad\text{for all }x\in\mathbb{R}^r. ∥ Y x ∥ 2 + ( β + t 0 ) ∥ ( Q x ) I c ∥ 2 ≥ β ∥ x ∥ 2 for all x ∈ R r .
Union bound over the four failure events ensures that this holds with probability at least 1 − δ r e p 1-\delta_{\mathrm{rep}} 1 − δ rep . □ \square □
Proof of the main theorem
We are now ready to prove the main theorem.
Theorem 1.1. There exist universal constants C , K > 0 C,K>0 C , K > 0 and α ∈ ( 0 , 1 ) \alpha\in(0,1) α ∈ ( 0 , 1 ) such that the following holds. For every n ≥ r ≥ 2 n\geq r\geq2 n ≥ r ≥ 2 , every fixed isometry Q ∈ R n × r Q\in\mathbb{R}^{n\times r} Q ∈ R n × r satisfying Q T Q = I r Q^{\mathsf{T}}Q=I_r Q T Q = I r , and integer sketch parameters satisfying
ζ = O ( log r ) with ζ ≥ C log ( e r ) , b = Θ ( r ζ ) , k = ζ b ≥ K r , \zeta=O\left(\sqrt{\log r}\right)\text{ with }\zeta\geq C\sqrt{\log(er)},\qquad b=\Theta\left(\frac{r}{\zeta}\right),\qquad k=\zeta b\geq Kr, ζ = O ( log r ) with ζ ≥ C log ( er ) , b = Θ ( ζ r ) , k = ζ b ≥ K r ,
the sketched matrix A = Ω ζ , b ⊤ Q ∈ R k × r A = \Omega_{\zeta,b}^{\top}Q \in\mathbb{R}^{k\times r} A = Ω ζ , b ⊤ Q ∈ R k × r formed from the ζ \zeta ζ -block SparseStack sketch Ω ζ , b ∈ R n × k \Omega_{\zeta,b} \in\mathbb{R}^{n\times k} Ω ζ , b ∈ R n × k satisfies
P [ σ min ( A ) 2 ≥ α ] ≥ 19 20 . \mathbb{P}\left[\sigma_{\min}(A)^2 \ge\alpha\right] \ge\frac{19}{20}. P [ σ m i n ( A ) 2 ≥ α ] ≥ 20 19 .
Moreover, E [ Ω ζ , b Ω ζ , b ⊤ ] = I n \mathbb{E}\left[\Omega_{\zeta,b}\Omega_{\zeta,b}^{\top}\right] = I_n E [ Ω ζ , b Ω ζ , b ⊤ ] = I n and the total sketch dimension is linear in r r r , namely k = ζ b = Θ ( r ) k = \zeta b = \Theta(r) k = ζ b = Θ ( r ) .
Proof. We write the total number of blocks as 2 ζ 2\zeta 2 ζ and the total embedding dimension as 2 k = 2 ζ b ≥ K r 2k = 2\zeta b \ge Kr 2 k = 2 ζ b ≥ K r , so that the sketched matrix is A = Ω 2 ζ , b ⊤ Q ∈ R 2 k × r A = \Omega_{2\zeta,b}^{\top}Q \in\mathbb{R}^{2k\times r} A = Ω 2 ζ , b ⊤ Q ∈ R 2 k × r . Partition the 2 ζ 2\zeta 2 ζ blocks of Ω 2 ζ , b ∈ R n × 2 k \Omega_{2\zeta,b} \in\mathbb{R}^{n\times2k} Ω 2 ζ , b ∈ R n × 2 k into two independent halves: the first half Ω ζ , b ( 1 ) ∈ R n × k \Omega_{\zeta,b}^{(1)} \in\mathbb{R}^{n\times k} Ω ζ , b ( 1 ) ∈ R n × k and the second half Ω ζ , b ( 2 ) ∈ R n × k \Omega_{\zeta,b}^{(2)} \in\mathbb{R}^{n\times k} Ω ζ , b ( 2 ) ∈ R n × k (where k = ζ b k = \zeta b k = ζ b ). Let X = ( Ω ζ , b ( 1 ) ) ⊤ Q ∈ R k × r X = \left(\Omega_{\zeta,b}^{(1)}\right)^{\top}Q \in\mathbb{R}^{k\times r} X = ( Ω ζ , b ( 1 ) ) ⊤ Q ∈ R k × r and Y = ( Ω ζ , b ( 2 ) ) ⊤ Q ∈ R k × r Y = \left(\Omega_{\zeta,b}^{(2)}\right)^{\top}Q \in\mathbb{R}^{k\times r} Y = ( Ω ζ , b ( 2 ) ) ⊤ Q ∈ R k × r . Then A = 1 2 [ X Y ] ∈ R 2 k × r A = \frac{1}{\sqrt{2}}\begin{bmatrix}X\\Y\end{bmatrix} \in\mathbb{R}^{2k\times r} A = 2 1 [ X Y ] ∈ R 2 k × r .
We set the parameters as follows: fix t 0 = 11 48 t_0 = \frac{11}{48} t 0 = 48 11 in Theorem 2.5, so that β = 11 24 − t 0 = 11 48 \beta= \frac{11}{24} - t_0 = \frac{11}{48} β = 24 11 − t 0 = 48 11 and γ = β + t 0 = 11 24 \gamma= \beta+ t_0 = \frac{11}{24} γ = β + t 0 = 24 11 ; fix ϵ = 1 3 \epsilon= \frac{1}{3} ϵ = 3 1 in Theorem 2.2, so that u : = 8 3 ϵ 2 = 8 27 < 1 2 = β γ u := \frac{8}{3}\epsilon^2 = \frac{8}{27} < \frac{1}{2} = \frac{\beta}{\gamma} u := 3 8 ϵ 2 = 27 8 < 2 1 = γ β ; and let λ = Ω ( ϵ ) \lambda= \Omega(\epsilon) λ = Ω ( ϵ ) , C l o c , M > 0 C_{\mathrm{loc}}, M > 0 C loc , M > 0 be the constants from Theorem 2.2 for ϵ = 1 / 3 \epsilon= 1/3 ϵ = 1/3 and failure probability δ l o c = 1 40 \delta_{\mathrm{loc}} = \frac{1}{40} δ loc = 40 1 , setting v = 8 3 λ 2 v = \frac{8}{3\lambda^2} v = 3 λ 2 8 (so that γ v ≥ 1 \gamma v \ge1 γ v ≥ 1 ).
First, apply Theorem 2.2 to the first half X = ( Ω ζ , b ( 1 ) ) ⊤ Q ∈ R k × r X = \left(\Omega_{\zeta,b}^{(1)}\right)^{\top}Q \in\mathbb{R}^{k\times r} X = ( Ω ζ , b ( 1 ) ) ⊤ Q ∈ R k × r , which has k = ζ b ≥ 1 2 K r ≥ C l o c r k = \zeta b \ge\frac{1}{2}Kr \ge C_{\mathrm{loc}}r k = ζ b ≥ 2 1 K r ≥ C loc r for K K K sufficiently large. Outside an event of probability at most δ l o c = 1 40 \delta_{\mathrm{loc}} = \frac{1}{40} δ loc = 40 1 , Theorem 2.3 supplies a coordinate set I ⊆ [ n ] I \subseteq[n] I ⊆ [ n ] determined by X X X , with m = M r / ζ m = Mr/\zeta m = M r / ζ , such that
∣ I ∣ ≤ 8 m , Q I c ⊤ Q I c ⪯ u I r + v X ⊤ X , where I c = [ n ] ∖ I . (52) |I| \le8m,\qquad Q_{I^c}^{\top}Q_{I^c} \preceq uI_r + vX^{\top}X,\qquad\text{where } I^c = [n]\setminus I. \tag*{(52)} ∣ I ∣ ≤ 8 m , Q I c ⊤ Q I c ⪯ u I r + v X ⊤ X , where I c = [ n ] ∖ I . (52)
Since b = k / ζ ≥ K r / ( 2 ζ ) b = k/\zeta\ge Kr/(2\zeta) b = k / ζ ≥ K r / ( 2 ζ ) , choosing K ≥ 256 M K \ge256M K ≥ 256 M ensures that
∣ I ∣ ≤ 8 m = 8 M r ζ ≤ min { r , b 16 } , |I| \le8m = \frac{8Mr}{\zeta} \le\min\left\{r,\frac{b}{16}\right\}, ∣ I ∣ ≤ 8 m = ζ 8 M r ≤ min { r , 16 b } ,
where 8 M r ζ ≤ r \frac{8Mr}{\zeta} \le r ζ 8 M r ≤ r holds because ζ = Θ ( log r ) ≥ 8 M \zeta= \Theta(\sqrt{\log r}) \ge8M ζ = Θ ( log r ) ≥ 8 M for r r r sufficiently large. Furthermore, k ≥ 1 2 K r k \ge\frac{1}{2}Kr k ≥ 2 1 K r satisfies the embedding dimension requirement of Theorem 2.5 for K K K sufficiently large with failure probability δ r e p = 1 40 \delta_{\mathrm{rep}} = \frac{1}{40} δ rep = 40 1 .
Next, condition on the first half X X X and hence on the realized set I I I . Because the second half Y = ( Ω ζ , b ( 2 ) ) ⊤ Q Y = \left(\Omega_{\zeta,b}^{(2)}\right)^{\top}Q Y = ( Ω ζ , b ( 2 ) ) ⊤ Q is independent of X X X , Theorem 2.5 applies conditionally on X X X and shows that with conditional failure probability at most δ r e p = 1 40 \delta_{\mathrm{rep}} = \frac{1}{40} δ rep = 40 1 ,
Y ⊤ Y + γ Q I c ⊤ Q I c ⪰ β I r , (53) Y^{\top}Y + \gamma Q_{I^c}^{\top}Q_{I^c} \succeq\beta I_r, \tag*{(53)} Y ⊤ Y + γ Q I c ⊤ Q I c ⪰ β I r , (53)
where γ = β + t 0 \gamma= \beta+ t_0 γ = β + t 0 . Substituting (52) into (53) gives
Y ⊤ Y + γ v X ⊤ X ⪰ ( β − γ u ) I r . Y^{\top}Y + \gamma vX^{\top}X \succeq(\beta- \gamma u)I_r. Y ⊤ Y + γ v X ⊤ X ⪰ ( β − γ u ) I r .
Finally, since γ v ≥ 1 \gamma v \ge1 γ v ≥ 1 and β − γ u > 0 \beta- \gamma u > 0 β − γ u > 0 ,
A ⊤ A = 1 2 ( X ⊤ X + Y ⊤ Y ) ⪰ 1 2 γ v ( Y ⊤ Y + γ v X ⊤ X ) ⪰ 1 2 v ( β γ − u ) I r = : α I r , A^{\top}A = \frac{1}{2}\left(X^{\top}X + Y^{\top}Y\right) \succeq\frac{1}{2\gamma v}\left(Y^{\top}Y + \gamma vX^{\top}X\right) \succeq\frac{1}{2v}\left(\frac{\beta}{\gamma} - u\right)I_r =: \alpha I_r, A ⊤ A = 2 1 ( X ⊤ X + Y ⊤ Y ) ⪰ 2 γ v 1 ( Y ⊤ Y + γ v X ⊤ X ) ⪰ 2 v 1 ( γ β − u ) I r =: α I r ,
where α = 1 2 v ( β γ − u ) = 11 108 v > 0 \alpha= \frac{1}{2v}\left(\frac{\beta}{\gamma} - u\right) = \frac{11}{108v} > 0 α = 2 v 1 ( γ β − u ) = 108 v 11 > 0 is an absolute constant. The total failure probability is at most δ l o c + δ r e p = 1 40 + 1 40 = 1 20 \delta_{\mathrm{loc}} + \delta_{\mathrm{rep}} = \frac{1}{40} + \frac{1}{40} = \frac{1}{20} δ loc + δ rep = 40 1 + 40 1 = 20 1 , completing the proof of Theorem 1.1.
Bipartite Graph Preprocessing and the C 4 C_4 C 4 -Free Graph G 0 G_0 G 0
As we discussed above, we fix a deterministic matrix S ∈ R n × k S \in\mathbb{R}^{n \times k} S ∈ R n × k with row sparsity ζ \zeta ζ (meaning ∣ supp ( s i ) ∣ ≤ ζ |\operatorname{supp}(s_i)| \le\zeta ∣ supp ( s i ) ∣ ≤ ζ for all i ∈ [ n ] i \in[n] i ∈ [ n ] ) and ∥ S ∥ F 2 ≤ 10 n \|S\|_F^2 \le10n ∥ S ∥ F 2 ≤ 10 n satisfying the overlap condition, and evaluate its injectivity against a random coordinate subspace R ∼ Unif ( ( [ n ] r ) ) R \sim\operatorname{Unif}\left(\binom{[n]}{r}\right) R ∼ Unif ( ( r [ n ] ) ) . We represent S S S as a bipartite graph G = ( [ n ] , [ k ] , E ) G = ([n], [k], E) G = ([ n ] , [ k ] , E ) between the n n n rows and k k k columns, with an edge between row i i i and column j j j whenever S i j ≠ 0 S_{ij} \ne0 S ij = 0 . In this section, we establish the graph preprocessing pipeline that filters out pathological rows and constructs a large C 4 C_4 C 4 -free subgraph G 0 G_0 G 0 . All subsequent tree cancellation arguments will be executed directly on G 0 G_0 G 0 .
Row Preprocessing
We first clarify the independence structure of the preprocessing steps. Because the matrix S S S is fixed deterministically, properties such as row Euclidean norms and support sizes are fixed deterministic attributes of each row in S S S . Specifically, we classify the row set [ n ] [n] [ n ] into regular rows and three exceptional sets:
Heavy rows: I h e a v y = { i ∈ [ n ] : ∥ s i ∥ 2 2 > M } I_{\mathrm{heavy}} = \{i \in[n] : \|s_i\|_2^2 > M\} I heavy = { i ∈ [ n ] : ∥ s i ∥ 2 2 > M } , where M = 160 M = 160 M = 160 .
Small-norm rows: I s m a l l = { i ∈ [ n ] : ∥ s i ∥ 2 2 < α } I_{\mathrm{small}} = \{i \in[n] : \|s_i\|_2^2 < \alpha\} I small = { i ∈ [ n ] : ∥ s i ∥ 2 2 < α } .
One-sparse rows: I o n e = { i ∈ [ n ] : ∣ supp ( s i ) ∣ = 1 } I_{\mathrm{one}} = \{i \in[n] : |\operatorname{supp}(s_i)| = 1\} I one = { i ∈ [ n ] : ∣ supp ( s i ) ∣ = 1 } .
The following lemma shows that heavy rows constitute at most a 1 / 16 1/16 1/16 fraction of all rows.
Lemma 8.1. Let M = 160 M = 160 M = 160 . In any matrix S ∈ R n × k S \in\mathbb{R}^{n \times k} S ∈ R n × k with ∥ S ∥ F 2 ≤ 10 n \|S\|_F^2 \le10n ∥ S ∥ F 2 ≤ 10 n , the set of heavy rows satisfies ∣ I h e a v y ∣ ≤ n / 16 |I_{\mathrm{heavy}}| \le n/16 ∣ I heavy ∣ ≤ n /16 .
Proof. By definition of the Frobenius norm, we have
10 n ≥ ∥ S ∥ F 2 = ∑ i = 1 n ∥ s i ∥ 2 2 ≥ ∑ i ∈ I h e a v y ∥ s i ∥ 2 2 > M ∣ I h e a v y ∣ = 160 ∣ I h e a v y ∣ . 10n \ge\|S\|_F^2 = \sum_{i=1}^{n} \|s_i\|_2^2 \ge\sum_{i \in I_{\mathrm{heavy}}} \|s_i\|_2^2 > M|I_{\mathrm{heavy}}| = 160|I_{\mathrm{heavy}}|. 10 n ≥ ∥ S ∥ F 2 = i = 1 ∑ n ∥ s i ∥ 2 2 ≥ i ∈ I heavy ∑ ∥ s i ∥ 2 2 > M ∣ I heavy ∣ = 160∣ I heavy ∣.
We next show that if either small-norm rows or one-sparse rows are abundant in S S S , then injectivity breaks directly on a random coordinate subspace R R R without requiring tree structures.
Lemma 8.2. For sufficiently large r r r and n ≥ Ω ( r log 3 r ) n \ge\Omega(r \log^3 r) n ≥ Ω ( r log 3 r ) , in any matrix S ∈ R n × k S \in\mathbb{R}^{n \times k} S ∈ R n × k with k ≤ K r k \le Kr k ≤ K r , if either ∣ I s m a l l ∣ ≥ n / 16 |I_{\mathrm{small}}| \ge n/16 ∣ I small ∣ ≥ n /16 or ∣ I o n e ∣ ≥ n / 16 |I_{\mathrm{one}}| \ge n/16 ∣ I one ∣ ≥ n /16 , then a uniformly random coordinate subspace R ∼ Unif ( ( [ n ] r ) ) R \sim\operatorname{Unif}\left(\binom{[n]}{r}\right) R ∼ Unif ( ( r [ n ] ) ) satisfies
P R ∼ Unif ( ( [ n ] r ) ) [ σ min ( S R T ) 2 < α ] ≥ 9 10 . (54) \mathbb{P}_{R \sim\operatorname{Unif}\left(\binom{[n]}{r}\right)}\left[\sigma_{\min}\left(S_R^{\mathsf{T}}\right)^2 < \alpha\right] \ge\frac{9}{10}. \tag*{(54)} P R ∼ Unif ( ( r [ n ] ) ) [ σ m i n ( S R T ) 2 < α ] ≥ 10 9 . (54)
Proof. We consider each condition separately.
Case 1: ∣ I s m a l l ∣ ≥ n / 16 |I_{\mathrm{small}}| \ge n/16 ∣ I small ∣ ≥ n /16 . When drawing r r r indices uniformly without replacement from [ n ] [n] [ n ] , the probability of missing all elements of I s m a l l I_{\mathrm{small}} I small is at most
∏ t = 0 r − 1 ( 1 − ∣ I s m a l l ∣ n − t ) ≤ ( 1 − 1 16 ) r ≤ exp ( − r 16 ) ≤ 1 10 \prod_{t=0}^{r-1}\left(1-\frac{|I_{\mathrm{small}}|}{n-t}\right) \le\left(1-\frac{1}{16}\right)^r \le\exp\left(-\frac{r}{16}\right) \le\frac{1}{10} t = 0 ∏ r − 1 ( 1 − n − t ∣ I small ∣ ) ≤ ( 1 − 16 1 ) r ≤ exp ( − 16 r ) ≤ 10 1
for all r ≥ 37 r \ge37 r ≥ 37 . Whenever R R R contains an index i ∈ I s m a l l i \in I_{\mathrm{small}} i ∈ I small , the standard basis vector e i ∈ R r e_i \in\mathbb{R}^r e i ∈ R r satisfies ∥ S R e i ∥ 2 2 = ∥ s i ∥ 2 2 < α \lVert S_R e_i\rVert_2^2 = \lVert s_i\rVert_2^2 < \alpha ∥ S R e i ∥ 2 2 = ∥ s i ∥ 2 2 < α , which breaks injectivity with σ min ( S R ) 2 < α \sigma_{\min}(S_R)^2 < \alpha σ m i n ( S R ) 2 < α .
Case 2: ∣ I o n e ∣ ≥ n / 16 |I_{\mathrm{one}}| \ge n/16 ∣ I one ∣ ≥ n /16 . Partition I o n e = ⋃ j = 1 k J j I_{\mathrm{one}} = \bigcup_{j=1}^k J_j I one = ⋃ j = 1 k J j according to the unique non-zero column of each row, where J j = { i ∈ I o n e : supp ( s i ) = { j } } J_j = \{i \in I_{\mathrm{one}} : \operatorname{supp}(s_i) = \{j\}\} J j = { i ∈ I one : supp ( s i ) = { j }} and m j = ∣ J j ∣ m_j = |J_j| m j = ∣ J j ∣ . Call a column j ∈ [ k ] j \in[k] j ∈ [ k ] heavy if m j ≥ n 32 K r m_j \ge\frac{n}{32Kr} m j ≥ 32 K r n , and let U = ⋃ j : m j ≥ n / ( 32 K r ) J j U = \bigcup_{j:m_j\ge n/(32Kr)} J_j U = ⋃ j : m j ≥ n / ( 32 K r ) J j be the union of the heavy column buckets. Since k ≤ K r k \le Kr k ≤ K r , all light columns together contain fewer than k ⋅ n 32 K r ≤ n 32 k \cdot\frac{n}{32Kr} \le\frac{n}{32} k ⋅ 32 K r n ≤ 32 n rows, so ∣ U ∣ ≥ ∣ I o n e ∣ − n 32 ≥ n 32 |U| \ge|I_{\mathrm{one}}| - \frac{n}{32} \ge\frac{n}{32} ∣ U ∣ ≥ ∣ I one ∣ − 32 n ≥ 32 n .
We expose the r r r draws of R ∼ Unif ( [ n ] r ) R \sim\operatorname{Unif}\binom{[n]}{r} R ∼ Unif ( r [ n ] ) sequentially in two phases of r 1 = r / 2 r_1 = r/2 r 1 = r /2 and r 2 = r − r 1 = r / 2 r_2 = r-r_1 = r/2 r 2 = r − r 1 = r /2 draws. Since ∣ U ∣ ≥ n / 32 |U| \ge n/32 ∣ U ∣ ≥ n /32 and r 1 < n / 64 r_1 < n/64 r 1 < n /64 , each of the first r 1 r_1 r 1 sequential draws lands in U U U with conditional probability at least ( ∣ U ∣ − r 1 ) / n ≥ 1 / 64 (|U|-r_1)/n \ge1/64 ( ∣ U ∣ − r 1 ) / n ≥ 1/64 given the previous draws. Consequently, by Chernoff’s bound, at least M 1 ≥ r 1 / 128 M_1 \ge r_1/128 M 1 ≥ r 1 /128 draws land in U U U with probability at least 19 / 20 19/20 19/20 for r r r sufficiently large. If two of these draws already fall into the same bucket J j J_j J j , a collision has occurred; otherwise, the first phase hits at least M 1 ≥ r 1 / 128 M_1 \ge r_1/128 M 1 ≥ r 1 /128 distinct heavy buckets J j J_j J j , each of which retains at least m j − 1 ≥ n 64 K r m_j-1 \ge\frac{n}{64Kr} m j − 1 ≥ 64 K r n unselected rows. Consequently, in each of the r 2 ≥ r / 2 r_2 \ge r/2 r 2 ≥ r /2 draws of the second phase, conditioned on no collision having occurred yet, the next draw hits one of the already-visited heavy buckets with probability at least
M 1 ⋅ n 64 K r n ≥ r 1 128 ⋅ 1 64 K r ≥ 1 16384 K . \frac{M_1 \cdot\frac{n}{64Kr}}{n} \ge\frac{r_1}{128}\cdot\frac{1}{64Kr} \ge\frac{1}{16384K}. n M 1 ⋅ 64 K r n ≥ 128 r 1 ⋅ 64 K r 1 ≥ 16384 K 1 .
so the conditional probability of avoiding a collision across all r 2 r_2 r 2 draws of the second phase is at most
( 1 − 1 16384 K ) r / 2 ≤ exp ( − r 32768 K ) ≤ 1 20 \left(1-\frac{1}{16384K}\right)^{r/2} \le\exp\left(-\frac{r}{32768K}\right) \le\frac{1}{20} ( 1 − 16384 K 1 ) r /2 ≤ exp ( − 32768 K r ) ≤ 20 1
for r ≥ C 0 K r \ge C_0K r ≥ C 0 K . Thus, with probability at least 1 − 1 20 − 1 20 = 9 10 1-\frac{1}{20}-\frac{1}{20}=\frac{9}{10} 1 − 20 1 − 20 1 = 10 9 , R R R contains two distinct rows u , v ∈ J j ∗ u,v \in J_{j^*} u , v ∈ J j ∗ from the same column j ∗ ∈ [ k ] j^* \in[k] j ∗ ∈ [ k ] . Setting x u = S v , j ∗ x_u=S_{v,j^*} x u = S v , j ∗ , x v = − S u , j ∗ x_v=-S_{u,j^*} x v = − S u , j ∗ , and zero elsewhere produces a non-zero vector x ∈ R r x \in\mathbb{R}^r x ∈ R r with S R ⊤ x = 0 S_R^\top x=0 S R ⊤ x = 0 . Thus injectivity breaks with σ min ( S R ) 2 = 0 < α \sigma_{\min}(S_R)^2=0<\alpha σ m i n ( S R ) 2 = 0 < α , completing the proof.
By Lemma 8.1 and Lemma 8.2, we may assume that ∣ I h e a v y ∪ I s m a l l ∪ I o n e ∣ ≤ 3 n / 16 |I_{\mathrm{heavy}} \cup I_{\mathrm{small}} \cup I_{\mathrm{one}}| \le3n/16 ∣ I heavy ∪ I small ∪ I one ∣ ≤ 3 n /16 . We define the set of clean rows by
V c l e a n = [ n ] ∖ ( I h e a v y ∪ I s m a l l ∪ I o n e ) . (55) V_{\mathrm{clean}}=[n]\setminus(I_{\mathrm{heavy}}\cup I_{\mathrm{small}}\cup I_{\mathrm{one}}). \tag*{(55)} V clean = [ n ] ∖ ( I heavy ∪ I small ∪ I one ) . (55)
The set V c l e a n V_{\mathrm{clean}} V clean is a fixed deterministic subset of size at least 13 n / 16 13n/16 13 n /16 . Every row i ∈ V c l e a n i \in V_{\mathrm{clean}} i ∈ V clean deterministically satisfies α ≤ ∥ s i ∥ 2 2 ≤ M \alpha\le\lVert s_i\rVert_2^2 \le M α ≤ ∥ s i ∥ 2 2 ≤ M and 2 ≤ ∣ supp ( s i ) ∣ ≤ ζ 2 \le|\operatorname{supp}(s_i)| \le\zeta 2 ≤ ∣ supp ( s i ) ∣ ≤ ζ .
Construction of the C4-Free Subgraph G0
We now employ the overlap condition to eliminate 4-cycles. By the definition of the bipartite graph G G G , two rows sharing two or more columns in their support form a 4-cycle in G G G .
To construct the C 4 C_4 C 4 -free subgraph G 0 G_0 G 0 , we subsample a row set V 0 ⊂ [ n ] V_0 \subset[n] V 0 ⊂ [ n ] of size n 0 = C s a m p r ζ 6 / α n_0=C_{\mathrm{samp}}r\zeta^6/\alpha n 0 = C samp r ζ 6 / α uniformly at random from [ n ] [n] [ n ] . This intermediate subsampling step couples cleanly with the ultimate coordinate sample R R R : choosing V 0 ⊂ [ n ] V_0 \subset[n] V 0 ⊂ [ n ] of size n 0 n_0 n 0 uniformly and then drawing r r r rows R ⊂ V 0 R \subset V_0 R ⊂ V 0 uniformly from V 0 V_0 V 0 produces a marginal distribution identical to drawing R ∼ Unif ( [ n ] r ) R \sim\operatorname{Unif}\binom{[n]}{r} R ∼ Unif ( r [ n ] ) directly from [ n ] [n] [ n ] . To restrict to clean rows and eliminate all 4-cycles, we form the pruned row set V ( G 0 ) ⊆ V 0 ∩ V c l e a n V(G_0) \subseteq V_0 \cap V_{\mathrm{clean}} V ( G 0 ) ⊆ V 0 ∩ V clean by discarding the non-clean rows V 0 ∖ V c l e a n V_0 \setminus V_{\mathrm{clean}} V 0 ∖ V clean and removing one row from each pair { u , v } ⊂ V 0 ∩ V c l e a n \{u,v\} \subset V_0 \cap V_{\mathrm{clean}} { u , v } ⊂ V 0 ∩ V clean sharing two or more columns ( ∣ supp ( s u ) ∩ supp ( s v ) ∣ ≥ 2 ) (|\operatorname{supp}(s_u)\cap\operatorname{supp}(s_v)|\ge2) ( ∣ supp ( s u ) ∩ supp ( s v ) ∣ ≥ 2 ) , and define G 0 = ( V ( G 0 ) , [ k ] , E 0 ) G_0=(V(G_0),[k],E_0) G 0 = ( V ( G 0 ) , [ k ] , E 0 ) to be the bipartite subgraph of G G G induced by V ( G 0 ) V(G_0) V ( G 0 ) and [ k ] [k] [ k ] .
Lemma 8.3. For sufficiently large r r r and n ≥ Ω ( r log 3 r ) n \ge\Omega(r\log^{3} r) n ≥ Ω ( r log 3 r ) , let S ∈ R n × k S \in\mathbb{R}^{n \times k} S ∈ R n × k be a matrix with row sparsity ζ \zeta ζ satisfying the overlap condition and ∣ V c l e a n ∣ ≥ 13 n / 16 |V_{\mathrm{clean}}| \ge13n/16 ∣ V clean ∣ ≥ 13 n /16 . Let n 0 = C s a m p r ζ 6 / α n_0 = C_{\mathrm{samp}}r\zeta^6/\alpha n 0 = C samp r ζ 6 / α for a sufficiently large constant C s a m p > 0 C_{\mathrm{samp}} > 0 C samp > 0 , and let V 0 ⊂ [ n ] V_0 \subset[n] V 0 ⊂ [ n ] be subsampled uniformly at random among all subsets of [ n ] [n] [ n ] of size n 0 n_0 n 0 . Then with probability at least 7 / 8 7/8 7/8 , the set V 0 V_0 V 0 contains at least 3 n 0 / 4 3n_0/4 3 n 0 /4 clean rows and at most n 0 / 16 n_0/16 n 0 /16 pairs sharing two or more columns, and the resulting induced subgraph G 0 = ( V ( G 0 ) , [ k ] , E 0 ) G_0 = (V(G_0), [k], E_0) G 0 = ( V ( G 0 ) , [ k ] , E 0 ) satisfies:
The row set has size ∣ V ( G 0 ) ∣ ≥ 5 n 0 / 8 |V(G_0)| \ge5n_0/8 ∣ V ( G 0 ) ∣ ≥ 5 n 0 /8 .
The graph G 0 G_0 G 0 is completely C 4 C_4 C 4 -free: no two rows share more than one column.
Every row i ∈ V ( G 0 ) i \in V(G_0) i ∈ V ( G 0 ) satisfies α ≤ ∥ s i ∥ 2 2 ≤ M \alpha\le\lVert s_i\rVert_2^2 \le M α ≤ ∥ s i ∥ 2 2 ≤ M and 2 ≤ deg G 0 ( i ) ≤ ζ 2 \le\deg_{G_0}(i) \le\zeta 2 ≤ deg G 0 ( i ) ≤ ζ .
Proof. First, let N b a d = ∣ V 0 ∖ V c l e a n ∣ N_{\mathrm{bad}} = |V_0 \setminus V_{\mathrm{clean}}| N bad = ∣ V 0 ∖ V clean ∣ denote the number of non-clean rows in V 0 V_0 V 0 . Since ∣ [ n ] ∖ V c l e a n ∣ ≤ 3 n / 16 |[n] \setminus V_{\mathrm{clean}}| \le3n/16 ∣ [ n ] ∖ V clean ∣ ≤ 3 n /16 , when drawing n 0 n_0 n 0 rows uniformly without replacement from [ n ] [n] [ n ] , the hypergeometric random variable N b a d N_{\mathrm{bad}} N bad has mean E [ N b a d ] ≤ 3 n 0 / 16 \mathbb{E}[N_{\mathrm{bad}}] \le3n_0/16 E [ N bad ] ≤ 3 n 0 /16 and variance Var ( N b a d ) ≤ 3 n 0 / 16 \operatorname{Var}(N_{\mathrm{bad}}) \le3n_0/16 Var ( N bad ) ≤ 3 n 0 /16 . By Chebyshev’s inequality, for C s a m p C_{\mathrm{samp}} C samp sufficiently large we have
P [ N b a d > n 0 4 ] ≤ P [ N b a d − E [ N b a d ] > n 0 16 ] ≤ 3 n 0 / 16 ( n 0 / 16 ) 2 = 48 n 0 ≤ 1 16 . \mathbb{P}\left[N_{\mathrm{bad}} > \frac{n_0}{4}\right] \le\mathbb{P}\left[N_{\mathrm{bad}}-\mathbb{E}[N_{\mathrm{bad}}] > \frac{n_0}{16}\right] \le\frac{3n_0/16}{(n_0/16)^2} = \frac{48}{n_0} \le\frac{1}{16}. P [ N bad > 4 n 0 ] ≤ P [ N bad − E [ N bad ] > 16 n 0 ] ≤ ( n 0 /16 ) 2 3 n 0 /16 = n 0 48 ≤ 16 1 .
Next, let N c o l l N_{\mathrm{coll}} N coll denote the number of pairs { u , v } ⊂ V 0 \{u,v\} \subset V_0 { u , v } ⊂ V 0 such that ∣ supp ( s u ) ∩ supp ( s v ) ∣ ≥ 2 |\operatorname{supp}(s_u) \cap\operatorname{supp}(s_v)| \ge2 ∣ supp ( s u ) ∩ supp ( s v ) ∣ ≥ 2 . In the bipartite graph G G G , every such pair corresponds to a 4-cycle C 4 C_4 C 4 . When drawing n 0 n_0 n 0 rows uniformly without replacement from [ n ] [n] [ n ] , each pair of rows from [ n ] [n] [ n ] is chosen with probability
( n − 2 n 0 − 2 ) ( n n 0 ) = n 0 ( n 0 − 1 ) n ( n − 1 ) ≤ n 0 2 n 2 . \frac{\binom{n-2}{n_0-2}}{\binom{n}{n_0}} = \frac{n_0(n_0-1)}{n(n-1)} \le\frac{n_0^2}{n^2}. ( n 0 n ) ( n 0 − 2 n − 2 ) = n ( n − 1 ) n 0 ( n 0 − 1 ) ≤ n 2 n 0 2 .
Using the overlap condition, the expected number of such pairs in V 0 V_0 V 0 satisfies
E [ N c o l l ] ≤ n 0 2 n 2 ∑ 1 ≤ u < v ≤ n 1 { ∣ supp ( s u ) ∩ supp ( s v ) ∣ ≥ 2 } ≤ n 0 2 n 2 ( η n 2 k ) = η n 0 2 k . \mathbb{E}[N_{\mathrm{coll}}] \le\frac{n_0^2}{n^2} \sum_{1 \le u < v \le n} \mathbf{1}\{|\operatorname{supp}(s_u) \cap\operatorname{supp}(s_v)| \ge2\} \le\frac{n_0^2}{n^2}\left(\eta\frac{n^2}{k}\right) = \eta\frac{n_0^2}{k}. E [ N coll ] ≤ n 2 n 0 2 1 ≤ u < v ≤ n ∑ 1 { ∣ supp ( s u ) ∩ supp ( s v ) ∣ ≥ 2 } ≤ n 2 n 0 2 ( η k n 2 ) = η k n 0 2 .
Since k < r k < r k < r trivially implies σ min ( S R T ) = 0 \sigma_{\min}(S_R^{\mathsf{T}}) = 0 σ m i n ( S R T ) = 0 , we may assume k ≥ r k \ge r k ≥ r . Substituting η ≤ 1 / ( C H ζ 6 ) \eta\le1/(C_H\zeta^6) η ≤ 1/ ( C H ζ 6 ) and n 0 = C s a m p r ζ 6 / α n_0 = C_{\mathrm{samp}}r\zeta^6/\alpha n 0 = C samp r ζ 6 / α gives
E [ N c o l l ] ≤ 1 C H ζ 6 n 0 ⋅ ( C s a m p r ζ 6 / α ) r = C s a m p C H α n 0 . \mathbb{E}[N_{\mathrm{coll}}] \le\frac{1}{C_H\zeta^6}\frac{n_0 \cdot(C_{\mathrm{samp}}r\zeta^6/\alpha)}{r} = \frac{C_{\mathrm{samp}}}{C_H\alpha}n_0. E [ N coll ] ≤ C H ζ 6 1 r n 0 ⋅ ( C samp r ζ 6 / α ) = C H α C samp n 0 .
Choosing C H ≥ 256 C s a m p / α C_H \ge256C_{\mathrm{samp}}/\alpha C H ≥ 256 C samp / α ensures that E [ N c o l l ] ≤ n 0 / 256 \mathbb{E}[N_{\mathrm{coll}}] \le n_0/256 E [ N coll ] ≤ n 0 /256 . By Markov’s inequality, we have
P [ N c o l l > n 0 16 ] ≤ E [ N c o l l ] n 0 / 16 ≤ 16 256 = 1 16 . \mathbb{P}\left[N_{\mathrm{coll}} > \frac{n_0}{16}\right] \le\frac{\mathbb{E}[N_{\mathrm{coll}}]}{n_0/16} \le\frac{16}{256} = \frac{1}{16}. P [ N coll > 16 n 0 ] ≤ n 0 /16 E [ N coll ] ≤ 256 16 = 16 1 .
By a union bound, with probability at least 1 − 1 / 16 − 1 / 16 = 7 / 8 1 - 1/16 - 1/16 = 7/8 1 − 1/16 − 1/16 = 7/8 , we have N b a d ≤ n 0 / 4 N_{\mathrm{bad}} \le n_0/4 N bad ≤ n 0 /4 and N c o l l ≤ n 0 / 16 N_{\mathrm{coll}} \le n_0/16 N coll ≤ n 0 /16 . Discarding the N b a d N_{\mathrm{bad}} N bad non-clean rows and deleting one row from each such pair removes at most n 0 / 4 + n 0 / 16 = 5 n 0 / 16 n_0/4 + n_0/16 = 5n_0/16 n 0 /4 + n 0 /16 = 5 n 0 /16 vertices and eliminates every 4-cycle, leaving a C 4 C_4 C 4 -free subgraph G 0 G_0 G 0 with at least n 0 − 5 n 0 / 16 = 11 n 0 / 16 ≥ 5 n 0 / 8 n_0 - 5n_0/16 = 11n_0/16 \ge5n_0/8 n 0 − 5 n 0 /16 = 11 n 0 /16 ≥ 5 n 0 /8 rows. All remaining rows belong to V c l e a n V_{\mathrm{clean}} V clean and thus inherit the norm bounds α ≤ ∥ s i ∥ 2 2 ≤ M \alpha\le\lVert s_i\rVert_2^2 \le M α ≤ ∥ s i ∥ 2 2 ≤ M and support bounds 2 ≤ deg G 0 ( i ) ≤ ζ 2 \le\deg_{G_0}(i) \le\zeta 2 ≤ deg G 0 ( i ) ≤ ζ . ∎
Constructing a Forest of Disjoint Trees
Having constructed the C 4 C_4 C 4 -free subgraph G 0 G_0 G 0 , where every row satisfies α ≤ ∥ s i ∥ 2 2 ≤ M \alpha\le\|s_i\|_2^2 \le M α ≤ ∥ s i ∥ 2 2 ≤ M and 2 ≤ deg G 0 ( i ) ≤ ζ 2 \le\deg_{G_0}(i) \le\zeta 2 ≤ deg G 0 ( i ) ≤ ζ , our main goal in this section is to prove Theorem 9.7 , which constructs a forest of T = Θ ( r / ζ 5 ) T = \Theta(r/\zeta^5) T = Θ ( r / ζ 5 ) mutually vertex-disjoint witness trees on G 0 G_0 G 0 that breaks injectivity. Each tree in G 0 G_0 G 0 consists of a center row i ∗ ∈ V ( G 0 ) i^* \in V(G_0) i ∗ ∈ V ( G 0 ) , its branch columns supp ( s i ∗ ) ⊂ [ k ] \operatorname{supp}(s_{i^*}) \subset[k] supp ( s i ∗ ) ⊂ [ k ] , and q = 4 ζ M / α q = 4\zeta M/\alpha q = 4 ζ M / α leaf rows in V ( G 0 ) V(G_0) V ( G 0 ) attached to each branch column.
Disjoint Leaves for a Single Tree
Recall that for any row i ∈ V ( G 0 ) i \in V(G_0) i ∈ V ( G 0 ) and column j ∈ [ k ] j \in[k] j ∈ [ k ] with S i j ≠ 0 S_{ij} \ne0 S ij = 0 , the relative weight is defined by w i j = ∣ S i j ∣ 2 / ∥ s i ∥ 2 2 ∈ ( 0 , 1 ] w_{ij} = |S_{ij}|^2/\|s_i\|_2^2 \in(0,1] w ij = ∣ S ij ∣ 2 /∥ s i ∥ 2 2 ∈ ( 0 , 1 ] . For each column j ∈ [ k ] j \in[k] j ∈ [ k ] , we sort all rows in V ( G 0 ) V(G_0) V ( G 0 ) containing j j j by non-increasing relative weights and place the top D D D rows into a candidate set F j ⊂ V ( G 0 ) F_j \subset V(G_0) F j ⊂ V ( G 0 ) . Because center rows i ∗ i^* i ∗ will be chosen exclusively from rows outside ⋃ j F j \bigcup_j F_j ⋃ j F j , every branch column j ∈ supp ( s i ∗ ) j \in\operatorname{supp}(s_{i^*}) j ∈ supp ( s i ∗ ) is guaranteed to have full size ∣ F j ∣ = D |F_j| = D ∣ F j ∣ = D and satisfy w ℓ , j ≥ w i ∗ , j w_{\ell,j} \ge w_{i^*,j} w ℓ , j ≥ w i ∗ , j for all ℓ ∈ F j \ell\in F_j ℓ ∈ F j .
The following lemma bounds the squared norm on the outer columns for a center row and its leaf rows.
Lemma 9.1. For a matrix S ∈ R n × k S \in\mathbb{R}^{n \times k} S ∈ R n × k with row sparsity ζ \zeta ζ , let s i ∗ T s_{i^*}^{\mathsf{T}} s i ∗ T be a row in G 0 G_0 G 0 with branches supp ( s i ∗ ) = { j 1 , … , j d } \operatorname{supp}(s_{i^*}) = \{j_1,\ldots,j_d\} supp ( s i ∗ ) = { j 1 , … , j d } of size d ≤ ζ d \le\zeta d ≤ ζ . For each branch index a ∈ [ d ] a \in[d] a ∈ [ d ] , let L a ⊆ F j a ∖ { i ∗ } L_a \subseteq F_{j_a} \setminus\{i^*\} L a ⊆ F j a ∖ { i ∗ } be a subset of q q q leaf rows. Suppose that for every leaf ℓ ∈ L a \ell\in L_a ℓ ∈ L a , the relative weight satisfies w ℓ , j a ≥ w i ∗ , j a w_{\ell,j_a} \ge w_{i^*,j_a} w ℓ , j a ≥ w i ∗ , j a , and suppose that the outer supports supp ( s ℓ ) ∖ { j a } \operatorname{supp}(s_\ell) \setminus\{j_a\} supp ( s ℓ ) ∖ { j a } across all ℓ ∈ ⋃ a = 1 d L a \ell\in\bigcup_{a=1}^d L_a ℓ ∈ ⋃ a = 1 d L a are pairwise disjoint. Define the cancellation vector x ∈ R n x \in\mathbb{R}^n x ∈ R n by
x i ∗ = 1 , x ℓ = − S i ∗ , j a q S ℓ , j a for each a ∈ [ d ] and ℓ ∈ L a , x m = 0 otherwise . (56) x_{i^*}=1,\qquad x_\ell=-\frac{S_{i^*,j_a}}{qS_{\ell,j_a}}\ \text{for each }a\in[d]\text{ and }\ell\in L_a,\qquad x_m=0\ \text{otherwise}. \tag*{(56)} x i ∗ = 1 , x ℓ = − q S ℓ , j a S i ∗ , j a for each a ∈ [ d ] and ℓ ∈ L a , x m = 0 otherwise . (56) Then for every branch column j a ∈ supp ( s i ∗ ) j_a \in\operatorname{supp}(s_{i^*}) j a ∈ supp ( s i ∗ ) , we have ( S T x ) j a = 0 (S^{\mathsf{T}}x)_{j_a}=0 ( S T x ) j a = 0 . Moreover, the total squared norm on all outer columns satisfies
∑ c ∉ supp ( s i ∗ ) ∣ ( S T x ) c ∣ 2 ≤ ∥ s i ∗ ∥ 2 2 q . (57) \sum_{c\notin\operatorname{supp}(s_{i^*})}|(S^{\mathsf{T}}x)_c|^2\le\frac{\|s_{i^*}\|_2^2}{q}. \tag*{(57)} c ∈ / supp ( s i ∗ ) ∑ ∣ ( S T x ) c ∣ 2 ≤ q ∥ s i ∗ ∥ 2 2 . (57)
Proof. We first verify the exact cancellation on the active columns of the center row. For any fixed branch a ∈ [ d ] a \in[d] a ∈ [ d ] , the j a j_a j a -th coordinate of S T x S^{\mathsf{T}}x S T x receives contributions only from row i ∗ i^* i ∗ and the leaf rows in L a L_a L a , because no other leaf contains j a j_a j a in its support. Using the definition of x x x in Equation (56) , we compute
( S T x ) j a = S i ∗ , j a x i ∗ + ∑ ℓ ∈ L a S ℓ , j a x ℓ = S i ∗ , j a − ∑ ℓ ∈ L a S ℓ , j a ( S i ∗ , j a q S ℓ , j a ) = S i ∗ , j a − q ⋅ S i ∗ , j a q = 0. (S^{\mathsf{T}}x)_{j_a}=S_{i^*,j_a}x_{i^*}+\sum_{\ell\in L_a}S_{\ell,j_a}x_\ell=S_{i^*,j_a}-\sum_{\ell\in L_a}S_{\ell,j_a}\left(\frac{S_{i^*,j_a}}{qS_{\ell,j_a}}\right)=S_{i^*,j_a}-q\cdot\frac{S_{i^*,j_a}}{q}=0. ( S T x ) j a = S i ∗ , j a x i ∗ + ℓ ∈ L a ∑ S ℓ , j a x ℓ = S i ∗ , j a − ℓ ∈ L a ∑ S ℓ , j a ( q S ℓ , j a S i ∗ , j a ) = S i ∗ , j a − q ⋅ q S i ∗ , j a = 0.
Thus exact cancellation holds on every column in supp ( s i ∗ ) \operatorname{supp}(s_{i^*}) supp ( s i ∗ ) .
We next evaluate the outer coordinates. By assumption, the outer supports supp ( s ℓ ) ∖ { j a } \operatorname{supp}(s_\ell) \setminus\{j_a\} supp ( s ℓ ) ∖ { j a } are pairwise disjoint across all leaves ℓ ∈ ⋃ a = 1 d L a \ell\in\bigcup_{a=1}^d L_a ℓ ∈ ⋃ a = 1 d L a . Therefore, for every outer column c ∉ supp ( s i ∗ ) c \notin\operatorname{supp}(s_{i^*}) c ∈ / supp ( s i ∗ ) , at most one leaf row contributes to ( S T x ) c (S^{\mathsf{T}}x)_c ( S T x ) c . Thus we get:
∑ c ∉ supp ( s i ∗ ) ∣ ( S T x ) c ∣ 2 = ∑ a = 1 d ∑ ℓ ∈ L a ∑ c ≠ j a ∣ x ℓ S ℓ , c ∣ 2 = ∑ a = 1 d ∑ ℓ ∈ L a ∣ x ℓ ∣ 2 ∑ c ≠ j a ∣ S ℓ , c ∣ 2 . (58) \sum_{c\notin\operatorname{supp}(s_{i^*})}|(S^{\mathsf{T}}x)_c|^2
=\sum_{a=1}^d\sum_{\ell\in L_a}\sum_{c\ne j_a}|x_\ell S_{\ell,c}|^2
=\sum_{a=1}^d\sum_{\ell\in L_a}|x_\ell|^2\sum_{c\ne j_a}|S_{\ell,c}|^2. \tag*{(58)} c ∈ / supp ( s i ∗ ) ∑ ∣ ( S T x ) c ∣ 2 = a = 1 ∑ d ℓ ∈ L a ∑ c = j a ∑ ∣ x ℓ S ℓ , c ∣ 2 = a = 1 ∑ d ℓ ∈ L a ∑ ∣ x ℓ ∣ 2 c = j a ∑ ∣ S ℓ , c ∣ 2 . (58)
By the definition, ∑ c ≠ j a ∣ S ℓ , c ∣ 2 ≤ ∥ s ℓ ∥ 2 2 \sum_{c\ne j_a}|S_{\ell,c}|^2 \le\|s_\ell\|_2^2 ∑ c = j a ∣ S ℓ , c ∣ 2 ≤ ∥ s ℓ ∥ 2 2 . Substituting the value of x ℓ x_\ell x ℓ gives
∣ x ℓ ∣ 2 ∑ c ≠ j a ∣ S ℓ , c ∣ 2 ≤ ( S i ∗ , j a q S ℓ , j a ) 2 ∥ s ℓ ∥ 2 2 = ∣ S i ∗ , j a ∣ 2 q 2 ∣ S ℓ , j a ∣ 2 ∥ s ℓ ∥ 2 2 . |x_\ell|^2\sum_{c\ne j_a}|S_{\ell,c}|^2 \le\left(\frac{S_{i^*,j_a}}{qS_{\ell,j_a}}\right)^2\|s_\ell\|_2^2=\frac{|S_{i^*,j_a}|^2}{q^2|S_{\ell,j_a}|^2}\|s_\ell\|_2^2. ∣ x ℓ ∣ 2 c = j a ∑ ∣ S ℓ , c ∣ 2 ≤ ( q S ℓ , j a S i ∗ , j a ) 2 ∥ s ℓ ∥ 2 2 = q 2 ∣ S ℓ , j a ∣ 2 ∣ S i ∗ , j a ∣ 2 ∥ s ℓ ∥ 2 2 .
By the definition of the relative weights w i j w_{ij} w ij , we have ∣ S i ∗ , j a ∣ 2 = w i ∗ , j a ∥ s i ∗ ∥ 2 2 |S_{i^*,j_a}|^2=w_{i^*,j_a}\|s_{i^*}\|_2^2 ∣ S i ∗ , j a ∣ 2 = w i ∗ , j a ∥ s i ∗ ∥ 2 2 and ∣ S ℓ , j a ∣ 2 = w ℓ , j a ∥ s ℓ ∥ 2 2 |S_{\ell,j_a}|^2=w_{\ell,j_a}\|s_\ell\|_2^2 ∣ S ℓ , j a ∣ 2 = w ℓ , j a ∥ s ℓ ∥ 2 2 . Substituting these identities into the bound yields
∣ x ℓ ∣ 2 ∑ c ≠ j a ∣ S ℓ , c ∣ 2 ≤ w i ∗ , j a ∥ s i ∗ ∥ 2 2 q 2 w ℓ , j a ∥ s ℓ ∥ 2 2 ∥ s ℓ ∥ 2 2 = w i ∗ , j a w ℓ , j a ∥ s i ∗ ∥ 2 2 q 2 . |x_\ell|^2\sum_{c\ne j_a}|S_{\ell,c}|^2\le\frac{w_{i^*,j_a}\|s_{i^*}\|_2^2}{q^2w_{\ell,j_a}\|s_\ell\|_2^2}\|s_\ell\|_2^2=\frac{w_{i^*,j_a}}{w_{\ell,j_a}}\frac{\|s_{i^*}\|_2^2}{q^2}. ∣ x ℓ ∣ 2 c = j a ∑ ∣ S ℓ , c ∣ 2 ≤ q 2 w ℓ , j a ∥ s ℓ ∥ 2 2 w i ∗ , j a ∥ s i ∗ ∥ 2 2 ∥ s ℓ ∥ 2 2 = w ℓ , j a w i ∗ , j a q 2 ∥ s i ∗ ∥ 2 2 .
Since row ℓ ∈ F j a \ell\in F_{j_a} ℓ ∈ F j a was selected from the candidate set with w ℓ , j a ≥ w i ∗ , j a w_{\ell,j_a}\ge w_{i^*,j_a} w ℓ , j a ≥ w i ∗ , j a , the ratio satisfies w i ∗ , j a / w ℓ , j a ≤ 1 w_{i^*,j_a}/w_{\ell,j_a}\le1 w i ∗ , j a / w ℓ , j a ≤ 1 . Summing this over all q q q leaves in branch a a a gives
∑ ℓ ∈ L a ∣ x ℓ ∣ 2 ∑ c ≠ j a ∣ S ℓ , c ∣ 2 ≤ q ⋅ ∥ s i ∗ ∥ 2 2 q 2 = ∥ s i ∗ ∥ 2 2 q . \sum_{\ell\in L_a}|x_\ell|^2\sum_{c\ne j_a}|S_{\ell,c}|^2\le q\cdot\frac{\|s_{i^*}\|_2^2}{q^2}=\frac{\|s_{i^*}\|_2^2}{q}. ℓ ∈ L a ∑ ∣ x ℓ ∣ 2 c = j a ∑ ∣ S ℓ , c ∣ 2 ≤ q ⋅ q 2 ∥ s i ∗ ∥ 2 2 = q ∥ s i ∗ ∥ 2 2 .
Finally, summing over all d d d active branches and using d ≤ ζ d\le\zeta d ≤ ζ yields Equation (57) .
Therefore, when selecting leaf rows, we require their outer supports to be pairwise disjoint. We now show that whenever each leaf candidate set F j F_j F j has size at least L = q + q ( ζ − 1 ) 2 L=q+q(\zeta-1)^2 L = q + q ( ζ − 1 ) 2 , such a selection is always guaranteed on any C 4 C_4 C 4 -free graph.
Lemma 9.2. For a matrix S ∈ R n × k S\in\mathbb{R}^{n\times k} S ∈ R n × k with row sparsity ζ \zeta ζ , let s i ∗ ⊤ s_{i^*}^{\top} s i ∗ ⊤ be a center row in G 0 G_0 G 0 with branches supp ( s i ∗ ) = { j 1 , … , j d } \operatorname{supp}(s_{i^*})=\{j_1,\ldots,j_d\} supp ( s i ∗ ) = { j 1 , … , j d } of size d ≤ ζ d\le\zeta d ≤ ζ . For each branch a ∈ [ d ] a\in[d] a ∈ [ d ] , let F j a ⊆ V ( G 0 ) ∖ { i ∗ } F_{j_a}\subseteq V(G_0)\setminus\{i^*\} F j a ⊆ V ( G 0 ) ∖ { i ∗ } be a candidate set of size at least
L = q + q ( ζ − 1 ) 2 . (59) L=q+q(\zeta-1)^2. \tag*{(59)} L = q + q ( ζ − 1 ) 2 . (59) Then there exist subsets L a ⊆ F j a L_a\subseteq F_{j_a} L a ⊆ F j a of size q q q for all a ∈ [ d ] a\in[d] a ∈ [ d ] such that the outer supports supp ( s ℓ ) ∖ { j a } \operatorname{supp}(s_\ell)\setminus\{j_a\} supp ( s ℓ ) ∖ { j a } across all ℓ ∈ ⋃ a = 1 d L a \ell\in\bigcup_{a=1}^d L_a ℓ ∈ ⋃ a = 1 d L a are mutually disjoint.
Proof. We select the subsets L 1 , … , L d L_1,\ldots,L_d L 1 , … , L d sequentially for branches a = 1 , … , d a=1,\ldots,d a = 1 , … , d . Suppose we have already selected subsets L 1 , … , L a − 1 L_1,\ldots,L_{a-1} L 1 , … , L a − 1 satisfying the disjointness condition. The total number of outer coordinates occupied by these previously selected leaves is at most
∑ t = 1 a − 1 ∑ ℓ ∈ L t ∣ supp ( s ℓ ) ∖ { j t } ∣ ≤ ( a − 1 ) ⋅ q ( ζ − 1 ) ≤ q ( ζ − 1 ) 2 . \sum_{t=1}^{a-1}\sum_{\ell\in L_t}|\operatorname{supp}(s_\ell)\setminus\{j_t\}|\le(a-1)\cdot q(\zeta-1)\le q(\zeta-1)^2. t = 1 ∑ a − 1 ℓ ∈ L t ∑ ∣ supp ( s ℓ ) ∖ { j t } ∣ ≤ ( a − 1 ) ⋅ q ( ζ − 1 ) ≤ q ( ζ − 1 ) 2 .
We now analyze how many rows in F j a F_{j_a} F j a can conflict with previously selected leaves. First, no row in F j a F_{j_a} F j a can have been selected in any previous branch L t L_t L t for t < a t<a t < a . Indeed, if row ℓ \ell ℓ belonged to both F j a F_{j_a} F j a and L t L_t L t , it would contain both column j a j_a j a and column j t j_t j t . But the center row i ∗ i^* i ∗ also contains both j a j_a j a and j t j_t j t , which would create a 4-cycle between rows { i ∗ , ℓ } \{i^*,\ell\} { i ∗ , ℓ } and columns { j a , j t } \{j_a,j_t\} { j a , j t } , contradicting the fact that G 0 G_0 G 0 is C 4 C_4 C 4 -free.
Second, for each occupied column c ∈ ⋃ t = 1 a − 1 ⋃ ℓ ∈ L t ( supp ( s ℓ ) ∖ { j t } ) c\in\bigcup_{t=1}^{a-1}\bigcup_{\ell\in L_t}(\operatorname{supp}(s_\ell)\setminus\{j_t\}) c ∈ ⋃ t = 1 a − 1 ⋃ ℓ ∈ L t ( supp ( s ℓ ) ∖ { j t }) , there is at most one row in F j a F_{j_a} F j a containing column c c c . If there were two distinct rows ℓ 1 , ℓ 2 ∈ F j a \ell_1,\ell_2\in F_{j_a} ℓ 1 , ℓ 2 ∈ F j a containing column c c c , they would both contain column j a j_a j a and column c c c , forming a 4-cycle between rows { ℓ 1 , ℓ 2 } \{\ell_1,\ell_2\} { ℓ 1 , ℓ 2 } and columns { j a , c } \{j_a,c\} { j a , c } , which again contradicts C 4 C_4 C 4 -freeness. Consequently, each of the at most q ( ζ − 1 ) 2 q(\zeta-1)^2 q ( ζ − 1 ) 2 occupied columns in previously selected leaves eliminates at most one candidate row in F j a F_{j_a} F j a . Since ∣ F j a ∣ ≥ L = q + q ( ζ − 1 ) 2 |F_{j_a}| \ge L = q + q(\zeta-1)^2 ∣ F j a ∣ ≥ L = q + q ( ζ − 1 ) 2 , there remain at least
∣ F j a ∣ − q ( ζ − 1 ) 2 ≥ q |F_{j_a}| - q(\zeta-1)^2 \ge q ∣ F j a ∣ − q ( ζ − 1 ) 2 ≥ q
candidate rows in F j a F_{j_a} F j a whose outer supports are completely disjoint from all previously occupied columns. Furthermore, among these remaining rows, no two rows can share any outer column because G 0 G_0 G 0 is C 4 C_4 C 4 -free. Picking any q q q of these rows as L a L_a L a maintains the induction hypothesis and completes the proof.
Disjoint Multiple Trees
Having established how to construct a single tree whose leaves have pairwise disjoint outer supports, which by Lemma 9.1 guarantees the existence of a cancellation vector achieving small residual norm, we now discuss how to extract multiple such trees in G 0 G_0 G 0 that are mutually vertex-disjoint.
Before we start, we first cap the maximum column degree by splitting columns into virtual columns . Specifically, set the degree threshold Δ : = ζ n 0 / k \Delta:= \zeta n_0/k Δ := ζ n 0 / k . For each column j ∈ [ k ] j \in[k] j ∈ [ k ] with deg G 0 ( j ) > Δ \deg_{G_0}(j) > \Delta deg G 0 ( j ) > Δ , we partition the rows of V ( G 0 ) V(G_0) V ( G 0 ) containing j j j into disjoint groups of size at most Δ \Delta Δ , viewing each resulting group across all columns as a virtual column indexed by j ′ ∈ [ k ′ ] j' \in[k'] j ′ ∈ [ k ′ ] . Since ∑ j = 1 k deg G 0 ( j ) ≤ ζ ∣ V ( G 0 ) ∣ ≤ ζ n 0 \sum_{j=1}^{k}\deg_{G_0}(j) \le\zeta|V(G_0)| \le\zeta n_0 ∑ j = 1 k deg G 0 ( j ) ≤ ζ ∣ V ( G 0 ) ∣ ≤ ζ n 0 , splitting adds at most ∑ j = 1 k deg G 0 ( j ) / Δ ≤ k \sum_{j=1}^{k}\deg_{G_0}(j)/\Delta\le k ∑ j = 1 k deg G 0 ( j ) /Δ ≤ k groups, so the total number of virtual columns is bounded by
k ′ ≤ k + ζ n 0 Δ = 2 k ≤ 2 K r . k' \le k + \frac{\zeta n_0}{\Delta} = 2k \le2Kr. k ′ ≤ k + Δ ζ n 0 = 2 k ≤ 2 K r .
Since each row i ∈ V ( G 0 ) i \in V(G_0) i ∈ V ( G 0 ) has at most ζ \zeta ζ non-zero entries, each row i i i still belongs to at most ζ \zeta ζ virtual columns. Within each virtual column j ′ ∈ [ k ′ ] j' \in[k'] j ′ ∈ [ k ′ ] , we sort its rows by non-increasing relative weight w i j w_{ij} w ij on the underlying column j ∈ [ k ] j \in[k] j ∈ [ k ] and place the top D D D rows into its candidate set F j ′ ⊆ V ( G 0 ) F_{j'} \subseteq V(G_0) F j ′ ⊆ V ( G 0 ) .
To construct multiple vertex-disjoint trees, we first seek center rows whose supports (over the virtual columns) are all mutually disjoint. In principle, branch columns could overlap between different trees as long as the vertex sets of the center rows and leaf rows remain disjoint; however, having pairwise disjoint supports for the center rows allows us to draw the leaf rows of each tree from distinct candidate sets F j ′ F_{j'} F j ′ . Lemma 9.4 establishes how many such center rows with pairwise disjoint supports can be extracted. To prove it, we use the following independent-set version of Turán’s theorem, also established by Caro and Wei [36 , 1 , 39 ] .
Lemma 9.3. Let H = ( V H , E H ) H = (V_H, E_H) H = ( V H , E H ) be a finite undirected graph with average degree d a v g = 2 ∣ E H ∣ / ∣ V H ∣ d_{\mathrm{avg}} = 2|E_H|/|V_H| d avg = 2∣ E H ∣/∣ V H ∣ . Then the independence number of H H H satisfies
α ( H ) ≥ ∣ V H ∣ 1 + d a v g . (60) \alpha(H) \ge\frac{|V_H|}{1+d_{\mathrm{avg}}}. \tag*{(60)} α ( H ) ≥ 1 + d avg ∣ V H ∣ . (60)
Lemma 9.4. For a matrix S ∈ R n × k S \in\mathbb{R}^{n \times k} S ∈ R n × k with row sparsity ζ \zeta ζ , let U U U be the set of rows in G 0 G_0 G 0 that do not belong to any candidate set F j ′ F_{j'} F j ′ for j ′ ∈ [ k ′ ] j' \in[k'] j ′ ∈ [ k ′ ] . Let H = ( U , E H ) H = (U, E_H) H = ( U , E H ) be the conflict graph with an edge between rows u , v ∈ U u,v \in U u , v ∈ U whenever u u u and v v v share at least one virtual column in [ k ′ ] [k'] [ k ′ ] . Then there exists an independent set in H H H of size
T = Θ ( r ζ 5 ) , (61) T = \Theta\left(\frac{r}{\zeta^5}\right), \tag*{(61)} T = Θ ( ζ 5 r ) , (61) providing T T T center rows whose virtual-column supports are pairwise disjoint.
Proof. We first bound the size of U U U . Each virtual column j ′ ∈ [ k ′ ] j' \in[k'] j ′ ∈ [ k ′ ] places at most D D D rows into its candidate set F j ′ F_{j'} F j ′ . Since there are k ′ ≤ 2 k ≤ 2 K r k' \le2k \le2Kr k ′ ≤ 2 k ≤ 2 K r virtual columns, the total number of rows belonging to at least one candidate set is at most D k ′ ≤ 2 D K r Dk' \le2DKr D k ′ ≤ 2 D K r . Since ∣ V ( G 0 ) ∣ ≥ 5 n 0 / 8 |V(G_0)| \ge5n_0/8 ∣ V ( G 0 ) ∣ ≥ 5 n 0 /8 and n 0 = C s a m p r ζ 6 / α n_0 = C_{\mathrm{samp}}r\zeta^6/\alpha n 0 = C samp r ζ 6 / α , choosing C s a m p C_{\mathrm{samp}} C samp sufficiently large ensures that
∣ U ∣ ≥ ∣ V ( G 0 ) ∣ − 2 D K r ≥ n 0 2 . |U| \ge|V(G_0)| - 2DKr \ge\frac{n_0}{2}. ∣ U ∣ ≥ ∣ V ( G 0 ) ∣ − 2 D K r ≥ 2 n 0 .
We next bound the number of edges in the conflict graph H H H . Two rows u , v ∈ U u,v \in U u , v ∈ U share an edge if and only if there is some virtual column j ′ ∈ [ k ′ ] j' \in[k'] j ′ ∈ [ k ′ ] containing both u u u and v v v . For each virtual column j ′ ∈ [ k ′ ] j' \in[k'] j ′ ∈ [ k ′ ] , let d j ′ d_{j'} d j ′ denote its degree in U U U . The number of edges in H H H is bounded by
∣ E H ∣ ≤ ∑ j ′ = 1 k ′ ( d j ′ 2 ) = 1 2 ∑ j ′ = 1 k ′ d j ′ ( d j ′ − 1 ) . |E_H| \le\sum_{j'=1}^{k'} \binom{d_{j'}}{2} = \frac{1}{2}\sum_{j'=1}^{k'} d_{j'}(d_{j'}-1). ∣ E H ∣ ≤ j ′ = 1 ∑ k ′ ( 2 d j ′ ) = 2 1 j ′ = 1 ∑ k ′ d j ′ ( d j ′ − 1 ) .
By construction of the virtual columns, every virtual column j ′ ∈ [ k ′ ] j' \in[k'] j ′ ∈ [ k ′ ] has degree d j ′ ≤ Δ = ζ n 0 / k d_{j'} \le\Delta= \zeta n_0/k d j ′ ≤ Δ = ζ n 0 / k , and since each row in U U U belongs to at most ζ \zeta ζ virtual columns, the total sum of degrees satisfies ∑ j ′ = 1 k ′ d j ′ ≤ ζ ∣ U ∣ \sum_{j'=1}^{k'}d_{j'} \le\zeta|U| ∑ j ′ = 1 k ′ d j ′ ≤ ζ ∣ U ∣ . Therefore, the overlap sum satisfies
∑ j ′ = 1 k ′ d j ′ ( d j ′ − 1 ) ≤ Δ ∑ j ′ = 1 k ′ d j ′ ≤ ζ 2 n 0 ∣ U ∣ k . \sum_{j'=1}^{k'} d_{j'}(d_{j'}-1) \le\Delta\sum_{j'=1}^{k'}d_{j'} \le\frac{\zeta^2 n_0|U|}{k}. j ′ = 1 ∑ k ′ d j ′ ( d j ′ − 1 ) ≤ Δ j ′ = 1 ∑ k ′ d j ′ ≤ k ζ 2 n 0 ∣ U ∣ .
Consequently, using k ≥ r k \ge r k ≥ r (as k < r k<r k < r trivially implies σ min ( S R T ) = 0 \sigma_{\min}(S_R^{\mathsf T})=0 σ m i n ( S R T ) = 0 ), the average degree d a v g = 2 ∣ E H ∣ / ∣ U ∣ d_{\mathrm{avg}}=2|E_H|/|U| d avg = 2∣ E H ∣/∣ U ∣ in the conflict graph satisfies
d a v g ≤ O ( ζ 2 n 0 k ) ≤ O ( ζ 2 n 0 r ) = O ( ζ 8 ) . d_{\mathrm{avg}} \le O\left(\frac{\zeta^2n_0}{k}\right) \le O\left(\frac{\zeta^2n_0}{r}\right) = O(\zeta^8). d avg ≤ O ( k ζ 2 n 0 ) ≤ O ( r ζ 2 n 0 ) = O ( ζ 8 ) .
By Lemma 9.3 , the independence number of H H H satisfies
α ( H ) ≥ ∣ U ∣ 1 + d a v g ≥ n 0 / 2 1 + O ( ζ 8 ) = Θ ( r ζ 2 ) . \alpha(H) \ge\frac{|U|}{1+d_{\mathrm{avg}}} \ge\frac{n_0/2}{1+O(\zeta^8)} = \Theta\left(\frac{r}{\zeta^2}\right). α ( H ) ≥ 1 + d avg ∣ U ∣ ≥ 1 + O ( ζ 8 ) n 0 /2 = Θ ( ζ 2 r ) .
Selecting any T = Θ ( r / ζ 5 ) T=\Theta(r/\zeta^5) T = Θ ( r / ζ 5 ) rows from this independent set yields T T T center rows whose virtual-column supports are completely pairwise disjoint.
Having found center rows whose supports over the branch (virtual) columns are pairwise disjoint, we next select leaf rows from the corresponding candidate sets such that their outer supports are pairwise disjoint. In Lemma 9.2 , we showed that for a single tree, having candidate sets of size at least L L L guarantees such a leaf selection. Therefore, if we can extract mutually disjoint subsets of size L L L from the candidate sets F j ′ F_{j'} F j ′ for all active branches across the T T T trees, we can apply the single-tree selection procedure to each tree independently. In the following lemmas, we establish the existence of these disjoint reservoirs via Hall’s Marriage Theorem [16 ] .
Lemma 9.5. Let B = ( V L , V R , E B ) B=(V_L,V_R,E_B) B = ( V L , V R , E B ) be a finite bipartite graph. There exists a matching in B B B saturating every vertex in V L V_L V L if and only if for every subset A ⊆ V L A \subseteq V_L A ⊆ V L , the neighborhood N B ( A ) ⊆ V R N_B(A) \subseteq V_R N B ( A ) ⊆ V R satisfies the marriage condition
∣ N B ( A ) ∣ ≥ ∣ A ∣ . (62) |N_B(A)| \ge|A|. \tag*{(62)} ∣ N B ( A ) ∣ ≥ ∣ A ∣. (62)
Lemma 9.6. For a matrix S ∈ R n × k S \in\mathbb{R}^{n \times k} S ∈ R n × k with row sparsity ζ \zeta ζ , let J b r a n c h ⊆ [ k ′ ] J_{\mathrm{branch}} \subseteq[k'] J branch ⊆ [ k ′ ] be the set of branch virtual columns across the T T T center rows in G 0 G_0 G 0 , with ∣ J b r a n c h ∣ ≤ ζ T |J_{\mathrm{branch}}| \le\zeta T ∣ J branch ∣ ≤ ζ T . Suppose that for every j ′ ∈ J b r a n c h j' \in J_{\mathrm{branch}} j ′ ∈ J branch , the candidate set size satisfies
D ≥ ζ L . (63) D \ge\zeta L. \tag*{(63)} D ≥ ζ L . (63) Then there exist mutually disjoint subsets R j ′ ⊆ F j ′ R_{j'} \subseteq F_{j'} R j ′ ⊆ F j ′ of size ∣ R j ′ ∣ = L |R_{j'}| = L ∣ R j ′ ∣ = L for all j ′ ∈ J b r a n c h j' \in J_{\mathrm{branch}} j ′ ∈ J branch .
Proof. We construct an auxiliary bipartite assignment graph B = ( V L , V R , E B ) B = (V_L, V_R, E_B) B = ( V L , V R , E B ) as follows:
The left vertex set V L V_L V L consists of all branch virtual columns j ′ ∈ J b r a n c h j' \in J_{\mathrm{branch}} j ′ ∈ J branch , each cloned L L L times. That is, V L = J b r a n c h × [ L ] V_L = J_{\mathrm{branch}} \times[L] V L = J branch × [ L ] , so ∣ V L ∣ = L ∣ J b r a n c h ∣ |V_L| = L|J_{\mathrm{branch}}| ∣ V L ∣ = L ∣ J branch ∣ .
The right vertex set V R = ⋃ j ′ ∈ J b r a n c h F j ′ ⊂ V ( G 0 ) V_R = \bigcup_{j' \in J_{\mathrm{branch}}} F_{j'} \subset V(G_0) V R = ⋃ j ′ ∈ J branch F j ′ ⊂ V ( G 0 ) consists of all candidate rows.
An edge is placed between a left slot ( j ′ , t ) ∈ V L (j', t) \in V_L ( j ′ , t ) ∈ V L and a right row ℓ ∈ V R \ell\in V_R ℓ ∈ V R if and only if ℓ ∈ F j ′ \ell\in F_{j'} ℓ ∈ F j ′ .
Every left slot ( j ′ , t ) ∈ V L (j', t) \in V_L ( j ′ , t ) ∈ V L connects to every row in F j ′ F_{j'} F j ′ , so its degree is exactly deg B ( ( j ′ , t ) ) = ∣ F j ′ ∣ = D \deg_B((j', t)) = |F_{j'}| = D deg B (( j ′ , t )) = ∣ F j ′ ∣ = D . On the other hand, every right row ℓ ∈ V R \ell\in V_R ℓ ∈ V R has degree at most ζ \zeta ζ in G 0 G_0 G 0 , so it can belong to candidate sets F j ′ F_{j'} F j ′ for at most ζ \zeta ζ distinct virtual columns. Since each virtual column is cloned L L L times on the left, the degree of any right row ℓ ∈ V R \ell\in V_R ℓ ∈ V R is bounded by
deg B ( ℓ ) ≤ ζ L . \deg_B(\ell) \le\zeta L. deg B ( ℓ ) ≤ ζ L . We now verify Hall’s marriage condition. Let A ⊆ V L A \subseteq V_L A ⊆ V L be an arbitrary subset of left slots, and let N B ( A ) ⊆ V R N_B(A) \subseteq V_R N B ( A ) ⊆ V R denote its neighborhood in B B B . We count the number of edges e B ( A , N B ( A ) ) e_B(A, N_B(A)) e B ( A , N B ( A )) between A A A and N B ( A ) N_B(A) N B ( A ) from both sides. From the left side, every slot in A A A has degree D D D , so
e B ( A , N B ( A ) ) = ∣ A ∣ ⋅ D . e_B(A, N_B(A)) = |A| \cdot D. e B ( A , N B ( A )) = ∣ A ∣ ⋅ D . From the right side, every vertex in N B ( A ) N_B(A) N B ( A ) has degree at most ζ L \zeta L ζ L , so
e B ( A , N B ( A ) ) ≤ ∣ N B ( A ) ∣ ⋅ ( ζ L ) . e_B(A, N_B(A)) \le|N_B(A)| \cdot(\zeta L). e B ( A , N B ( A )) ≤ ∣ N B ( A ) ∣ ⋅ ( ζ L ) . Combining these two relations gives
∣ A ∣ ⋅ D ≤ ∣ N B ( A ) ∣ ⋅ ( ζ L ) ⟹ ∣ N B ( A ) ∣ ≥ D ζ L ∣ A ∣ . |A| \cdot D \le|N_B(A)| \cdot(\zeta L) \Longrightarrow|N_B(A)| \ge\frac{D}{\zeta L}|A|. ∣ A ∣ ⋅ D ≤ ∣ N B ( A ) ∣ ⋅ ( ζ L ) ⟹ ∣ N B ( A ) ∣ ≥ ζ L D ∣ A ∣. Since D ≥ ζ L D \ge\zeta L D ≥ ζ L by Equation (63) , we have ∣ N B ( A ) ∣ ≥ ∣ A ∣ |N_B(A)| \ge|A| ∣ N B ( A ) ∣ ≥ ∣ A ∣ for every subset A ⊆ V L A \subseteq V_L A ⊆ V L .
By Lemma 9.5, there exists a matching in B B B that saturates all vertices of V L V_L V L . This matching assigns each slot ( j ′ , t ) (j', t) ( j ′ , t ) to a distinct candidate row ℓ ∈ F j ′ \ell\in F_{j'} ℓ ∈ F j ′ . For each j ′ ∈ J b r a n c h j' \in J_{\mathrm{branch}} j ′ ∈ J branch , the set of L L L rows matched to its slots forms the reservoir R j ′ ⊆ F j ′ R_{j'} \subseteq F_{j'} R j ′ ⊆ F j ′ . Because the matching pairs each slot with a distinct right vertex, the reservoirs R j ′ R_{j'} R j ′ across all j ′ ∈ J b r a n c h j' \in J_{\mathrm{branch}} j ′ ∈ J branch are pairwise disjoint, completing the proof.
We now combine these combinatorial components to establish the existence of the full forest of witness trees.
Theorem 9.7. For a matrix S ∈ R n × k S \in\mathbb{R}^{n \times k} S ∈ R n × k with row sparsity ζ \zeta ζ , set q = 4 ζ M α q = \frac{4\zeta M}{\alpha} q = α 4 ζ M . In the C 4 C_4 C 4 -free subgraph G 0 G_0 G 0 , there exist T = Θ ( r / ζ 5 ) T = \Theta(r/\zeta^5) T = Θ ( r / ζ 5 ) mutually vertex-disjoint witness trees T 1 , … , T T T_1, \ldots, T_T T 1 , … , T T such that:
Each tree T s T_s T s consists of a center row i s ∗ i_s^* i s ∗ and at most ζ q \zeta q ζ q leaves, with total size
B t r e e = 1 + ζ q = Θ ( ζ 2 α ) . (64) B_{\mathrm{tree}} = 1 + \zeta q = \Theta\left(\frac{\zeta^2}{\alpha}\right). \tag*{(64)} B tree = 1 + ζ q = Θ ( α ζ 2 ) . (64)
The trees T 1 , … , T T T_1,\ldots,T_T T 1 , … , T T are mutually vertex-disjoint: V ( T s ) ∩ V ( T u ) = ∅ V(T_s) \cap V(T_u) = \varnothing V ( T s ) ∩ V ( T u ) = ∅ for all s ≠ u s \ne u s = u .
For each tree T s T_s T s , the cancellation vector x ( s ) ∈ R n x^{(s)} \in\mathbb{R}^n x ( s ) ∈ R n supported on V ( T s ) V(T_s) V ( T s ) satisfies
∥ x ( s ) ∥ 2 2 ≥ 1 and ∥ S T x ( s ) ∥ 2 2 ≤ α 4 . (65) \lVert x^{(s)} \rVert_2^2 \ge1 \quad\text{and}\quad\lVert S^{\mathsf{T}} x^{(s)} \rVert_2^2 \le\frac{\alpha}{4}. \tag*{(65)} ∥ x ( s ) ∥ 2 2 ≥ 1 and ∥ S T x ( s ) ∥ 2 2 ≤ 4 α . (65)
Proof. By Lemma 9.4 , we extract T = Θ ( r / ζ 5 ) T = \Theta(r/\zeta^5) T = Θ ( r / ζ 5 ) center rows i 1 ∗ , … , i T ∗ i_1^*,\ldots,i_T^* i 1 ∗ , … , i T ∗ whose virtual-column supports are pairwise disjoint and which do not belong to any candidate set F j ′ F_{j'} F j ′ . By Lemma 9.6, since D ≥ ζ L D \ge\zeta L D ≥ ζ L , there exist mutually disjoint leaf reservoirs R j ′ ⊆ F j ′ R_{j'} \subseteq F_{j'} R j ′ ⊆ F j ′ of size L = q + q ( ζ − 1 ) 2 L = q + q(\zeta- 1)^2 L = q + q ( ζ − 1 ) 2 for all active branch virtual columns.
For each center row i s ∗ i_s^* i s ∗ , its virtual columns correspond one-to-one with its branch columns supp ( s i s ∗ ) ⊆ [ k ] \operatorname{supp}(s_{i_s^*}) \subseteq[k] supp ( s i s ∗ ) ⊆ [ k ] , and we apply Lemma 9.2 within R j ′ R_{j'} R j ′ to select q q q leaves per branch column. Even if two different center rows have virtual columns originating from the same underlying column in [ k ] [k] [ k ] , their assigned reservoirs R j ′ R_{j'} R j ′ are mutually disjoint subsets of V ( G 0 ) V(G_0) V ( G 0 ) , and the center rows are distinct and do not belong to any candidate set, so the resulting trees T 1 , … , T T T_1,\ldots,T_T T 1 , … , T T are mutually vertex-disjoint: V ( T s ) ∩ V ( T u ) = ∅ V(T_s) \cap V(T_u) = \varnothing V ( T s ) ∩ V ( T u ) = ∅ .
Finally, for each tree T s T_s T s , let x ( s ) ∈ R n x^{(s)} \in\mathbb{R}^n x ( s ) ∈ R n be the cancellation vector defined in Lemma 9.1 on V ( T s ) V(T_s) V ( T s ) . Because x i s ∗ ( s ) = 1 x_{i_s^*}^{(s)} = 1 x i s ∗ ( s ) = 1 , we have ∥ x ( s ) ∥ 2 2 = 1 + ∑ a = 1 d s ∑ ℓ ∈ L a ( x ℓ ( s ) ) 2 ≥ 1 \lVert x^{(s)} \rVert_2^2 = 1 + \sum_{a=1}^{d_s} \sum_{\ell\in L_a} (x_\ell^{(s)})^2 \ge1 ∥ x ( s ) ∥ 2 2 = 1 + ∑ a = 1 d s ∑ ℓ ∈ L a ( x ℓ ( s ) ) 2 ≥ 1 . Moreover, Lemma 9.1 guarantees that ( S T x ( s ) ) j a = 0 (S^{\mathsf{T}}x^{(s)})_{j_a} = 0 ( S T x ( s ) ) j a = 0 for every branch column j a ∈ supp ( s i s ∗ ) j_a \in\operatorname{supp}(s_{i_s^*}) j a ∈ supp ( s i s ∗ ) , so the squared Euclidean norm of S T x ( s ) S^{\mathsf{T}}x^{(s)} S T x ( s ) is supported entirely on the outer columns. Since every row in G 0 G_0 G 0 satisfies ∥ s i s ∗ ∥ 2 2 ≤ M \lVert s_{i_s^*} \rVert_2^2 \le M ∥ s i s ∗ ∥ 2 2 ≤ M , applying Equation (57) and substituting q = 4 ζ M / α q = 4\zeta M/\alpha q = 4 ζ M / α gives
∥ S T x ( s ) ∥ 2 2 = ∑ c ∉ supp ( s i s ∗ ) ∣ ( S T x ( s ) ) c ∣ 2 ≤ ζ ∥ s i s ∗ ∥ 2 2 q ≤ ζ M q = α 4 , \lVert S^{\mathsf{T}}x^{(s)} \rVert_2^2
= \sum_{c \notin\operatorname{supp}(s_{i_s^*})} \lvert(S^{\mathsf{T}}x^{(s)})_c \rvert^2
\le\frac{\zeta\lVert s_{i_s^*} \rVert_2^2}{q}
\le\frac{\zeta M}{q}
= \frac{\alpha}{4}, ∥ S T x ( s ) ∥ 2 2 = c ∈ / supp ( s i s ∗ ) ∑ ∣( S T x ( s ) ) c ∣ 2 ≤ q ζ ∥ s i s ∗ ∥ 2 2 ≤ q ζ M = 4 α ,
which completes the proof.
Proof of the Lower Bound Theorem
In this section, we complete the proof of the sparsity lower bound. We first formalize the reduction principle from § 2 , which establishes that breaking injectivity on a random coordinate sample for a fixed matrix refutes OSI for any isotropic distribution.
Lemma 10.1. For sufficiently large r r r and n ≥ Ω ( r log 3 r ) n \ge\Omega(r \log^3 r) n ≥ Ω ( r log 3 r ) , let α ∈ ( 0 , 1 ) \alpha\in(0,1) α ∈ ( 0 , 1 ) . Suppose that for every fixed matrix S ∈ R n × k S \in\mathbb{R}^{n \times k} S ∈ R n × k with row sparsity ζ \zeta ζ and ∥ S ∥ F 2 ≤ 10 n \lVert S \rVert_F^2 \le10n ∥ S ∥ F 2 ≤ 10 n satisfying the overlap condition, a uniformly random coordinate index set R ∼ Unif ( ( [ n ] r ) ) R \sim\operatorname{Unif}\left(\binom{[n]}{r}\right) R ∼ Unif ( ( r [ n ] ) ) satisfies
P R ∼ Unif ( ( [ n ] r ) ) [ σ min ( S R T ) 2 ≥ α ] ≤ 1 4 . (66) \mathbb{P}_{R \sim\operatorname{Unif}\left(\binom{[n]}{r}\right)}
\left[\sigma_{\min}(S_R^{\mathsf{T}})^2 \ge\alpha\right] \le\frac{1}{4}. \tag*{(66)} P R ∼ Unif ( ( r [ n ] ) ) [ σ m i n ( S R T ) 2 ≥ α ] ≤ 4 1 . (66) Then no isotropic distribution over row-ζ \zeta ζ -sparse matrices in R n × k \mathbb{R}^{n \times k} R n × k satisfying the overlap condition with high probability can be an ( r , α ) (r,\alpha) ( r , α ) -OSI.
Proof. Suppose for contradiction that an isotropic distribution D \mathcal{D} D over row-ζ \zeta ζ -sparse matrices in R n × k \mathbb{R}^{n\times k} R n × k satisfies the overlap condition with probability at least 9 / 10 9/10 9/10 and is an ( r , α ) (r,\alpha) ( r , α ) -OSI. By definition, every fixed orthonormal matrix Q ∈ R n × r Q \in\mathbb{R}^{n\times r} Q ∈ R n × r satisfies
P Ω ∼ D [ σ min ( Ω T Q ) 2 ≥ α ] ≥ 19 20 . \mathbb{P}_{\Omega\sim\mathcal{D}}\left[\sigma_{\min}(\Omega^{\mathsf{T}}Q)^2 \ge\alpha\right] \ge\frac{19}{20}. P Ω ∼ D [ σ m i n ( Ω T Q ) 2 ≥ α ] ≥ 20 19 .
In particular, this guarantee must hold for every coordinate subspace Q R = I R Q_R=I_R Q R = I R where R ⊂ [ n ] R\subset[n] R ⊂ [ n ] has size r r r .
Here, the minimum success probability over coordinate subspaces is bounded by the average failure probability:
min R ∗ ∈ ( [ n ] r ) P Ω ∼ D [ σ min ( Ω R ∗ T ) 2 ≥ α ] ≤ E R ∼ Unif ( ( [ n ] r ) ) [ P Ω ∼ D [ σ min ( Ω R T ) 2 ≥ α ] ] = E Ω ∼ D [ P R ∼ Unif ( ( [ n ] r ) ) [ σ min ( Ω R T ) 2 ≥ α ] ] . \begin{aligned}
\min_{R^*\in\binom{[n]}{r}}\mathbb{P}_{\Omega\sim\mathcal{D}}\left[\sigma_{\min}(\Omega^{\mathsf{T}}_{R^*})^2\ge\alpha\right]
\le\mathbb{E}_{R\sim\operatorname{Unif}\left(\binom{[n]}{r}\right)}
\left[\mathbb{P}_{\Omega\sim\mathcal{D}}\left[\sigma_{\min}(\Omega^{\mathsf{T}}_R)^2\ge\alpha\right]\right] \\
&= \mathbb{E}_{\Omega\sim\mathcal{D}}\left[\mathbb{P}_{R\sim\operatorname{Unif}\left(\binom{[n]}{r}\right)}
\left[\sigma_{\min}(\Omega^{\mathsf{T}}_R)^2\ge\alpha\right]\right].
\end{aligned} R ∗ ∈ ( r [ n ] ) min P Ω ∼ D [ σ m i n ( Ω R ∗ T ) 2 ≥ α ] ≤ E R ∼ Unif ( ( r [ n ] ) ) [ P Ω ∼ D [ σ m i n ( Ω R T ) 2 ≥ α ] ] = E Ω ∼ D [ P R ∼ Unif ( ( r [ n ] ) ) [ σ m i n ( Ω R T ) 2 ≥ α ] ] .
Next, by the isotropy condition E [ Ω Ω T ] = I n \mathbb{E}[\Omega\Omega^{\mathsf{T}}]=I_n E [ Ω Ω T ] = I n , each row satisfies E [ ∥ s i ∥ 2 2 ] = 1 \mathbb{E}[\|s_i\|_2^2]=1 E [ ∥ s i ∥ 2 2 ] = 1 , so E [ ∥ Ω ∥ F 2 ] = n \mathbb{E}[\|\Omega\|_F^2]=n E [ ∥Ω ∥ F 2 ] = n . By Markov’s inequality, P [ ∥ Ω ∥ F 2 > 10 n ] ≤ 1 / 10 \mathbb{P}[\|\Omega\|_F^2>10n]\le1/10 P [ ∥Ω ∥ F 2 > 10 n ] ≤ 1/10 , and by hypothesis Ω \Omega Ω violates the overlap condition with probability at most 1 / 10 1/10 1/10 . Define the good event E \mathcal{E} E on which Ω \Omega Ω satisfies both ∥ Ω ∥ F 2 ≤ 10 n \|\Omega\|_F^2\le10n ∥Ω ∥ F 2 ≤ 10 n and the overlap condition, so that P [ E c ] ≤ 1 / 5 \mathbb{P}[\mathcal{E}^{c}]\le1/5 P [ E c ] ≤ 1/5 . Whenever Ω \Omega Ω satisfies E \mathcal{E} E , the hypothesis in Equation (66) applies to S = Ω S=\Omega S = Ω , giving P R [ σ min ( Ω R T ) 2 ≥ α ] ≤ 1 / 4 \mathbb{P}_R[\sigma_{\min}(\Omega^{\mathsf{T}}_R)^2\ge\alpha]\le1/4 P R [ σ m i n ( Ω R T ) 2 ≥ α ] ≤ 1/4 . Conditioning on E \mathcal{E} E yields
min R ∗ ∈ ( [ n ] r ) P Ω ∼ D [ σ min ( Ω R ∗ T ) 2 ≥ α ] ≤ E Ω ∼ D [ P R [ σ min ( Ω R T ) 2 ≥ α ] ] ≤ P [ E c ] + E Ω ∼ D [ P R [ σ min ( Ω R T ) 2 ≥ α ] ∣ E ] ≤ 1 5 + 1 4 = 9 20 . \begin{aligned}
\min_{R^*\in\binom{[n]}{r}}\mathbb{P}_{\Omega\sim\mathcal{D}}\left[\sigma_{\min}(\Omega^{\mathsf{T}}_{R^*})^2\ge\alpha\right]
\le\mathbb{E}_{\Omega\sim\mathcal{D}}\left[\mathbb{P}_R\left[\sigma_{\min}(\Omega^{\mathsf{T}}_R)^2\ge\alpha\right]\right] \\
&\le\mathbb{P}[\mathcal{E}^{c}]+\mathbb{E}_{\Omega\sim\mathcal{D}}\left[\mathbb{P}_R\left[\sigma_{\min}(\Omega^{\mathsf{T}}_R)^2\ge\alpha\right]\mid\mathcal{E}\right] \\
&\le\frac{1}{5}+\frac{1}{4}=\frac{9}{20}.
\end{aligned} R ∗ ∈ ( r [ n ] ) min P Ω ∼ D [ σ m i n ( Ω R ∗ T ) 2 ≥ α ] ≤ E Ω ∼ D [ P R [ σ m i n ( Ω R T ) 2 ≥ α ] ] ≤ P [ E c ] + E Ω ∼ D [ P R [ σ m i n ( Ω R T ) 2 ≥ α ] ∣ E ] ≤ 5 1 + 4 1 = 20 9 .
Therefore, there exists a specific coordinate subset R ∗ ⊂ [ n ] R^*\subset[n] R ∗ ⊂ [ n ] of size r r r such that P Ω [ σ min ( Ω R ∗ T ) 2 ≥ α ] ≤ 9 / 20 \mathbb{P}_{\Omega}[\sigma_{\min}(\Omega^{\mathsf{T}}_{R^*})^2\ge\alpha]\le9/20 P Ω [ σ m i n ( Ω R ∗ T ) 2 ≥ α ] ≤ 9/20 . This strictly contradicts the requirement that P Ω [ σ min ( Ω R ∗ T ) 2 ≥ α ] ≥ 19 / 20 \mathbb{P}_{\Omega}[\sigma_{\min}(\Omega^{\mathsf{T}}_{R^*})^2\ge\alpha]\ge19/20 P Ω [ σ m i n ( Ω R ∗ T ) 2 ≥ α ] ≥ 19/20 , completing the proof. □
Therefore, it now remains only to establish Equation (66) : showing that every sparse matrix satisfying the overlap condition has its injectivity broken on a random coordinate sample with high probability will complete the entire proof.
Theorem 1.3. For sufficiently large r r r , let n ≥ Ω ( r log 3 r ) n\ge\Omega(r\log^3 r) n ≥ Ω ( r log 3 r ) , k ≤ K r k\le Kr k ≤ K r for a constant K ≥ 1 K\ge1 K ≥ 1 , and α ∈ ( 0 , 1 ) \alpha\in(0,1) α ∈ ( 0 , 1 ) . If an isotropic random sketching matrix Ω ∈ R n × k \Omega\in\mathbb{R}^{n\times k} Ω ∈ R n × k with row sparsity ζ \zeta ζ satisfies the overlap condition with the overlap parameter η = O ( ( log log r log r ) 3 ) \eta=O\left(\left(\frac{\log\log r}{\log r}\right)^3\right) η = O ( ( l o g r l o g l o g r ) 3 ) with high probability, and is an ( r , α ) (r,\alpha) ( r , α ) -OSI, then the row sparsity must satisfy
ζ = Ω ( log r log log r ) . \zeta=\Omega\left(\sqrt{\frac{\log r}{\log\log r}}\right). ζ = Ω ( log log r log r ) .
Proof. Suppose for contradiction that the row sparsity satisfies
ζ ≤ c log r log log r (67) \zeta\le c\sqrt{\frac{\log r}{\log\log r}} \tag*{(67)} ζ ≤ c log log r log r (67)
for a sufficiently small constant c > 0 c > 0 c > 0 . By Lemma 10.1 , it suffices to show that for every fixed matrix S ∈ R n × k S \in\mathbb{R}^{n \times k} S ∈ R n × k with row sparsity ζ \zeta ζ and ∥ S ∥ F 2 ≤ 10 n \lVert S\rVert_{F}^{2} \le10n ∥ S ∥ F 2 ≤ 10 n satisfying the overlap condition, a random coordinate index set R ∼ Unif ( [ n ] r ) R \sim\operatorname{Unif}\binom{[n]}{r} R ∼ Unif ( r [ n ] ) satisfies P R [ σ min ( S R T ) 2 ≥ α ] ≤ 1 / 4 \mathbb{P}_{R}[\sigma_{\min}(S_{R}^{\mathsf{T}})^{2} \ge\alpha] \le1/4 P R [ σ m i n ( S R T ) 2 ≥ α ] ≤ 1/4 .
If ∣ V c l e a n ∣ < 13 n / 16 |V_{\mathrm{clean}}| < 13n/16 ∣ V clean ∣ < 13 n /16 , then Lemma 8.1 and Lemma 8.2 immediately imply P R [ σ min ( S R T ) 2 ≥ α ] ≤ 1 / 10 ≤ 1 / 4 \mathbb{P}_{R}[\sigma_{\min}(S_{R}^{\mathsf{T}})^{2} \ge\alpha] \le1/10 \le1/4 P R [ σ m i n ( S R T ) 2 ≥ α ] ≤ 1/10 ≤ 1/4 . Otherwise, assume ∣ V c l e a n ∣ ≥ 13 n / 16 |V_{\mathrm{clean}}| \ge13n/16 ∣ V clean ∣ ≥ 13 n /16 . We analyze the probability that a uniform random sample R ∼ Unif ( [ n ] r ) R \sim\operatorname{Unif}\binom{[n]}{r} R ∼ Unif ( r [ n ] ) captures at least one witness tree. Under the two-stage sampling coupling, we first draw an intermediate row set V 0 ⊂ [ n ] V_{0} \subset[n] V 0 ⊂ [ n ] of size n 0 = Θ ( r ζ 6 / α ) n_{0} = \Theta(r\zeta^{6}/\alpha) n 0 = Θ ( r ζ 6 / α ) uniformly at random from [ n ] [n] [ n ] , and then draw R ⊂ V 0 R \subset V_{0} R ⊂ V 0 of size r r r uniformly at random from V 0 V_{0} V 0 , whose marginal distribution is identical to R ∼ Unif ( [ n ] r ) R \sim\operatorname{Unif}\binom{[n]}{r} R ∼ Unif ( r [ n ] ) . By Lemma 8.3 and Theorem 9.7 , with probability at least 7 / 8 7/8 7/8 over the choice of V 0 V_{0} V 0 , the pruned C 4 C_{4} C 4 -free subgraph G 0 G_{0} G 0 on V ( G 0 ) ⊆ V 0 ∩ V c l e a n V(G_{0}) \subseteq V_{0} \cap V_{\mathrm{clean}} V ( G 0 ) ⊆ V 0 ∩ V clean satisfies ∣ V ( G 0 ) ∣ ≥ 5 n 0 / 8 |V(G_{0})| \ge5n_{0}/8 ∣ V ( G 0 ) ∣ ≥ 5 n 0 /8 and contains T = Θ ( r / ζ 5 ) T = \Theta(r/\zeta^{5}) T = Θ ( r / ζ 5 ) mutually vertex-disjoint witness trees T 1 , … , T T T_{1}, \ldots, T_{T} T 1 , … , T T , each of size B t r e e = Θ ( ζ 2 / α ) B_{\mathrm{tree}} = \Theta(\zeta^{2}/\alpha) B tree = Θ ( ζ 2 / α ) . Conditioned on such a realization of V 0 V_{0} V 0 , since V ( T s ) ⊆ V ( G 0 ) ⊆ V 0 V(T_{s}) \subseteq V(G_{0}) \subseteq V_{0} V ( T s ) ⊆ V ( G 0 ) ⊆ V 0 , drawing r r r rows uniformly without replacement from V 0 V_{0} V 0 includes each vertex of T s T_{s} T s with conditional probability at least ( r − B t r e e ) / n 0 ≥ r / ( 2 n 0 ) = Ω ( 1 / ζ 6 ) (r-B_{\mathrm{tree}})/n_{0} \ge r/(2n_{0}) = \Omega(1/\zeta^{6}) ( r − B tree ) / n 0 ≥ r / ( 2 n 0 ) = Ω ( 1/ ζ 6 ) . For any fixed witness tree T s T_{s} T s , the probability that all B t r e e B_{\mathrm{tree}} B tree of its vertices are captured by R R R satisfies
p = P [ V ( T s ) ⊆ R ∣ V 0 ] ≥ ( r 2 n 0 ) B t r e e = ( Ω ( 1 ζ 6 ) ) Θ ( ζ 2 / α ) = exp ( − O ( ζ 2 α log ζ ) ) . p = \mathbb{P}[V(T_{s}) \subseteq R \mid V_{0}] \ge\left(\frac{r}{2n_{0}}\right)^{B_{\mathrm{tree}}} = \left(\Omega\left(\frac{1}{\zeta^{6}}\right)\right)^{\Theta(\zeta^{2}/\alpha)} = \exp\left(-O\left(\frac{\zeta^{2}}{\alpha}\log\zeta\right)\right). p = P [ V ( T s ) ⊆ R ∣ V 0 ] ≥ ( 2 n 0 r ) B tree = ( Ω ( ζ 6 1 ) ) Θ ( ζ 2 / α ) = exp ( − O ( α ζ 2 log ζ ) ) .
Under the assumption in Equation (67) , choosing c c c sufficiently small ensures that O ( ζ 2 α log ζ ) ≤ 1 2 log r O(\frac{\zeta^{2}}{\alpha}\log\zeta) \le\frac{1}{2}\log r O ( α ζ 2 log ζ ) ≤ 2 1 log r , which yields p ≥ 1 / r p \ge1/\sqrt{r} p ≥ 1/ r .
Because the T T T trees are mutually vertex-disjoint, sampling without replacement exhibits negative correlation across the capture events { V ( T s ) ⊆ R } \{V(T_{s}) \subseteq R\} { V ( T s ) ⊆ R } . Consequently, the conditional probability that the random sample misses all T T T trees satisfies
P [ ⋂ s = 1 T { V ( T s ) ⊈ R } ∣ V 0 ] ≤ ( 1 − p ) T ≤ exp ( − T p ) . \mathbb{P}\left[\left.\bigcap_{s=1}^{T}\{V(T_{s}) \not\subseteq R\}\,\right|\,V_{0}\right] \le(1-p)^{T} \le\exp(-Tp). P [ s = 1 ⋂ T { V ( T s ) ⊆ R } V 0 ] ≤ ( 1 − p ) T ≤ exp ( − T p ) .
Since T = Θ ( r / ζ 5 ) T = \Theta(r/\zeta^{5}) T = Θ ( r / ζ 5 ) , we have T p ≥ Θ ( r 1 / 2 / ζ 5 ) ≥ r 1 / 4 ≥ 3 Tp \ge\Theta(r^{1/2}/\zeta^{5}) \ge r^{1/4} \ge3 T p ≥ Θ ( r 1/2 / ζ 5 ) ≥ r 1/4 ≥ 3 for sufficiently large r r r . Therefore, conditioned on V 0 V_{0} V 0 satisfying the event of Lemma 8.3 , with probability at least 1 − exp ( − 3 ) ≥ 9 / 10 1-\exp(-3) \ge9/10 1 − exp ( − 3 ) ≥ 9/10 the sample R R R completely captures at least one witness tree T s ∗ ⊆ R T_{s^{*}} \subseteq R T s ∗ ⊆ R .
On this capture event, all vertices of V ( T s ∗ ) V(T_{s^{*}}) V ( T s ∗ ) belong to R R R . Restricting the cancellation vector x ( s ∗ ) ∈ R n x^{(s^{*})} \in\mathbb{R}^{n} x ( s ∗ ) ∈ R n from Theorem 9.7 to the coordinates in R R R produces a non-zero vector z ∈ R r z \in\mathbb{R}^{r} z ∈ R r that breaks injectivity on R R R , satisfying ∥ z ∥ 2 = ∥ x ( s ∗ ) ∥ 2 ≥ 1 \lVert z\rVert_{2} = \lVert x^{(s^{*})}\rVert_{2} \ge1 ∥ z ∥ 2 = ∥ x ( s ∗ ) ∥ 2 ≥ 1 and
σ min ( S R T ) 2 ≤ ∥ S R T z ∥ 2 2 ∥ z ∥ 2 2 ≤ α 4 < α . \sigma_{\min}(S_{R}^{\mathsf{T}})^{2} \le\frac{\lVert S_{R}^{\mathsf{T}}z\rVert_{2}^{2}}{\lVert z\rVert_{2}^{2}} \le\frac{\alpha}{4} < \alpha. σ m i n ( S R T ) 2 ≤ ∥ z ∥ 2 2 ∥ S R T z ∥ 2 2 ≤ 4 α < α .
Combining the failure probability at most 1 / 8 1/8 1/8 of V 0 V_{0} V 0 in Lemma 8.3 and the conditional failure probability at most 1 / 10 1/10 1/10 of tree capture gives P R [ σ min ( S R T ) 2 ≥ α ] ≤ 1 / 8 + 1 / 10 = 9 / 40 ≤ 1 / 4 \mathbb{P}_{R}[\sigma_{\min}(S_{R}^{\mathsf{T}})^{2} \ge\alpha] \le1/8 + 1/10 = 9/40 \le1/4 P R [ σ m i n ( S R T ) 2 ≥ α ] ≤ 1/8 + 1/10 = 9/40 ≤ 1/4 . By Lemma 10.1 , no isotropic distribution over row-ζ \zeta ζ -sparse matrices satisfying the overlap condition with high probability can be an ( r , α ) (r,\alpha) ( r , α ) -OSI. □
Acknowledgments
MS’s work is supported by the NSF under award 2402314. RAM would like to acknowledge the DARPA DIAL and DARPA AIQ programs for providing partial support of this work. Kijun Shin and David Woodruff would like to thank partial support from a Simons Investigator Award and Office of Naval Research (ONR) grant number N000142112647.