Introduction
The Steiner tree problem asks for a minimum-weight network connecting a prescribed set of terminals, possibly through additional points. It is one of Karp’s original NP-complete problems [26 ] ; the best known polynomial-time approximation ratio is ln 4 + ε ≈ 1.39 \ln4+\varepsilon\approx1.39 ln 4 + ε ≈ 1.39 [6 ] , and no ratio better than 96 / 95 96/95 96/95 is achievable unless P = N P \mathrm{P}=\mathrm{NP} P = NP [13 ] . Steiner problems have also been studied in streaming [14 ] , dynamic [25 , 24 , 29 ] , online [25 , 23 , 1 , 3 , 4 ] , and sublinear-query models [16 , 31 ] .
An instance of the metric Steiner tree problem consists of a finite set V V V of n n n points, a metric w w w on V V V , and a set T ⊆ V T\subseteq V T ⊆ V of k k k terminals. A feasible solution is a tree in the complete graph on V V V whose vertex set contains T T T ; we write O P T \mathrm{OPT} OPT for the least weight of a feasible solution. In the distance-oracle model an algorithm knows V V V and T T T and may query any pair { u , v } \{u,v\} { u , v } to learn w ( u , v ) w(u,v) w ( u , v ) ; its cost is the number of queries. Let L L L be the weight of a minimum spanning tree (MST) of the metric restricted to T T T . Doubling a Steiner tree and shortcutting gives O P T ≤ L ≤ 2 O P T \mathrm{OPT}\leq L\leq2\mathrm{OPT} OPT ≤ L ≤ 2 OPT , and Czumaj and Sohler [19 ] showed that L L L can be estimated within a factor 1 + ε 1+\varepsilon 1 + ε using O ~ ( k ) \widetilde{O}(k) O ( k ) queries, so a ( 2 + ε ) (2+\varepsilon) ( 2 + ε ) -estimate needs only a nearly linear number of queries. The natural question that prior works have focused on is the cost of estimating O P T \mathrm{OPT} OPT within a factor strictly below 2 2 2 .
Chen, Khanna, and Tan [16 ] initiated this study and settled much of the landscape: estimating O P T \mathrm{OPT} OPT within a factor 5 / 3 − ε 5/3-\varepsilon 5/3 − ε requires Ω ε ( n 2 ) \Omega_\varepsilon(n^2) Ω ε ( n 2 ) queries, even when k = O ( 1 ) k=O(1) k = O ( 1 ) , and explicitly returning a tree of weight below 2 ⋅ O P T 2\cdot\mathrm{OPT} 2 ⋅ OPT requires Ω ( n k ) \Omega(nk) Ω ( nk ) queries. This explains why we can only aim for estimation algorithms (that do not return a tree). They also gave an estimator with a factor 2 − ε 0 2-\varepsilon_0 2 − ε 0 using O ~ ( n 12 / 7 + k n 6 / 7 ) = O ~ ( n 13 / 7 ) \widetilde{O}(n^{12/7}+kn^{6/7})=\widetilde{O}(n^{13/7}) O ( n 12/7 + k n 6/7 ) = O ( n 13/7 ) queries. Mahabadi, Roghani, Tarnawski, and Vakilian [31 ] improved this to O ~ ( n 5 / 3 ) \widetilde{O}(n^{5/3}) O ( n 5/3 ) using improved bounds for sublinear set cover. On the lower-bound side, [16 ] showed that Ω ~ ( n + k 6 / 5 ) \widetilde{\Omega}(n+k^{6/5}) Ω ( n + k 6/5 ) queries are needed, which is Ω ~ ( n 6 / 5 ) \widetilde{\Omega}(n^{6/5}) Ω ( n 6/5 ) when k = Θ ( n ) k=\Theta(n) k = Θ ( n ) . Mahabadi, Tarnawski, and Vakilian [33 ] then gave an improved lower bound of Ω ~ ( n 7 / 5 ) \widetilde{\Omega}(n^{7/5}) Ω ( n 7/5 ) . See also Table 1 .
Table 1. Bounds for metric Steiner tree in the distance-oracle model. The upper bounds below factor 2 hold for one particular (very small) constant ε 0 > 0 \varepsilon_0>0 ε 0 > 0 ; our lower bound holds for every fixed ε ∈ ( 0 , 1 ) \varepsilon\in(0,1) ε ∈ ( 0 , 1 ) and every fixed ε ′ > 0 \varepsilon'>0 ε ′ > 0 , and the instances have k = Θ ( n ) k=\Theta(n) k = Θ ( n ) .
This problem belongs to a broader line of work on estimating optimization problems with distance or adjacency queries, including minimum spanning tree [18 , 19 ] , maximal independent set [34 , 36 , 32 ] , matching [36 , 27 , 5 , 9 , 11 , 10 , 7 , 30 , 2 ] , Steiner forest [32 ] , and metric TSP [15 , 17 ] .
Our results
We determine the exponent of the query complexity of estimation below factor 2 2 2 . Following [16 , 31 ] , we define an estimate within factor 2 − η 2-\eta 2 − η to be a number C ^ \widehat{C} C with
O P T ≤ C ^ ≤ ( 2 − η ) O P T . (1) \mathrm{OPT}\leq\widehat{C}\leq(2-\eta)\mathrm{OPT}. \tag*{(1)} OPT ≤ C ≤ ( 2 − η ) OPT . (1)
Theorem 1.1 (Upper bound). There is an absolute constant η 0 > 0 \eta_0>0 η 0 > 0 and a randomized algorithm that, given an n n n -point instance, outputs with probability at least 2 / 3 2/3 2/3 a number C ^ \widehat{C} C satisfying (1) with η = η 0 \eta=\eta_0 η = η 0 . In the worst case it makes O ~ ( n 3 / 2 ) \widetilde{O}(n^{3/2}) O ( n 3/2 ) distance queries and uses O ~ ( n 3 / 2 ) \widetilde{O}(n^{3/2}) O ( n 3/2 ) time.
We also show that the exponent 3 / 2 3/2 3/2 is tight by giving an improved hardness for sublinear algorithms. See Corollary 1.3.
Theorem 1.2 (Lower bound). For every η ∈ ( 0 , 1 ) \eta\in(0,1) η ∈ ( 0 , 1 ) and every integer s ≥ 1 s\geq1 s ≥ 1 there are constants c η , s > 0 c_{\eta,s}>0 c η , s > 0 and n η , s n_{\eta,s} n η , s such that the following holds for every n ≥ n η , s n\geq n_{\eta,s} n ≥ n η , s . Every randomized algorithm that, on every n n n -point instance, outputs with probability at least 2 / 3 2/3 2/3 a number C ^ \widehat{C} C satisfying (1) makes at least c η , s n b s c_{\eta,s}n^{b_s} c η , s n b s queries in expectation on some instance, where
b s = 3 s + 7 2 s + 5 = 3 2 − 1 4 s + 10 . b_s=\frac{3s+7}{2s+5}=\frac{3}{2}-\frac{1}{4s+10}. b s = 2 s + 5 3 s + 7 = 2 3 − 4 s + 10 1 .
The hard instances have Θ η , s ( n ) \Theta_{\eta,s}(n) Θ η , s ( n ) terminals, and every distance between two distinct points lies in { Θ η , s ( n − 3 ) , 1 , 2 } \{\Theta_{\eta,s}(n^{-3}),1,2\} { Θ η , s ( n − 3 ) , 1 , 2 } .
Corollary 1.3. For every fixed η ∈ ( 0 , 1 ) \eta\in(0,1) η ∈ ( 0 , 1 ) and ε > 0 \varepsilon>0 ε > 0 , estimating the metric Steiner tree cost within a factor 2 − η 2-\eta 2 − η requires Ω η , ε ( n 3 / 2 − ε ) \Omega_{\eta,\varepsilon}(n^{3/2-\varepsilon}) Ω η , ε ( n 3/2 − ε ) queries. In particular, for the factor 2 − η 0 2-\eta_0 2 − η 0 of Theorem 1.1 the query complexity is n 3 / 2 ± o ( 1 ) n^{3/2\pm o(1)} n 3/2 ± o ( 1 ) .
The n 3 / 2 n^{3/2} n 3/2 query complexity is thus tight up to subpolynomial factors. The lower bound is stated for expected query complexity; in particular it also holds for worst-case complexity.
Lean formalization Both of our results are formalized in Lean. The formalization is available under https://github.com/vakiliana/sublinear-steiner-tree .
In the following § 1.2 and § 1.3 we give an intuitive overview of the techniques underlying our results.
Technical Overview of the Algorithm
In this section, we provide an overview of the techniques used to design our algorithm of Theorem 1.1 . We begin with the framework of Chen, Khanna, and Tan [16 ] and its implementation by Mahabadi, Roghani, Tarnawski, and Vakilian [31 ] , which reduces estimation below factor 2 to estimating a set cover quantity and estimates that quantity by estimating the size of a maximal matching in an auxiliary graph. We then describe the new idea that brings the cost of this set cover step from O ~ ( n 5 / 3 ) \widetilde{O}(n^{5/3}) O ( n 5/3 ) down to O ~ ( n 3 / 2 ) \widetilde{O}(n^{3/2}) O ( n 3/2 ) . As we will see, this idea alone does not yield an O ~ ( n 3 / 2 ) \widetilde{O}(n^{3/2}) O ( n 3/2 ) algorithm for Steiner tree, and we describe the remaining challenges and how we overcome them step by step. Throughout, n n n is the number of points, k ≤ n k\le n k ≤ n the number of terminals, and for simplicity we assume k = Θ ( n ) k=\Theta(n) k = Θ ( n ) , which is the hardest case. A nonterminal point is called a center .
Overview of the Previous Algorithms
Starting point: the terminal MST and the set cover reduction. Let L L L be the weight of a minimum spanning tree on the terminals. It is well known that O P T ≤ L ≤ 2 O P T \mathrm{OPT}\le L\le2\mathrm{OPT} OPT ≤ L ≤ 2 OPT , and L L L can be estimated within a factor 1 + ε 1+\varepsilon 1 + ε using O ~ ( k ) \widetilde{O}(k) O ( k ) queries [19 ] . Therefore the whole difficulty of beating factor 2 is to certify, with few queries, a discount D ≤ L − O P T D\le L-\mathrm{OPT} D ≤ L − OPT that is a constant fraction of L L L whenever such a discount exists. The algorithm then outputs L − D L-D L − D . The framework of [16 ] looks for two types of improvements over the MST. For the first type, fix a scale a a a and consider the clusters of terminals at this scale, i.e., the connected components of the graph on terminals in which two terminals are adjacent if their distance is less than a a a . The MST spends roughly a a a to connect each of these clusters to the rest. Now, if a center v v v is close to three or more clusters, we can instead connect all of them to v v v using short edges, which is cheaper. For the second type, we look for a small group of terminals (at most four) inside one cluster that can be connected through a single center more cheaply than the MST connects them. The main structural result of [16 ] is that these two types of improvements are the only ones that matter: if neither type yields a significant improvement at any scale, then L ≤ ( 2 − η ) OPT L \le(2-\eta)\operatorname{OPT} L ≤ ( 2 − η ) OPT , and it suffices to output an estimate of L L L .
The first type of improvement is naturally a set cover problem. Fix a scale a a a and let the universe U U U be the set of clusters at scale a a a . For each center v v v , let S v ⊆ U S_v \subseteq U S v ⊆ U be the clusters that have a terminal within distance 5 a / 8 5a/8 5 a /8 of v v v . A set S v S_v S v is useful only if ∣ S v ∣ ≥ 3 |S_v| \ge3 ∣ S v ∣ ≥ 3 , since connecting two clusters through a center is never cheaper than connecting them directly. We call such sets eligible (see Figure 1 (a)). Let F \mathcal{F} F be the collection of all sets S v S_v S v , and write SC ( U , F ) \operatorname{SC}(U,\mathcal{F}) SC ( U , F ) for the least number of eligible sets and singletons needed to cover U U U . The total improvement of the first type at scale a a a is captured by the saving
Figure 1. (a) The instance at one scale. The universe is the set of clusters x , p , q , r , u , z , w x,p,q,r,u,z,w x , p , q , r , u , z , w , and each center defines the set of clusters within distance 5 a / 8 5a/8 5 a /8 of it, namely S 1 = { x , p , q } S_1=\{x,p,q\} S 1 = { x , p , q } , S 2 = { x , r , u } S_2=\{x,r,u\} S 2 = { x , r , u } , and S 3 = { x , z , w } S_3=\{x,z,w\} S 3 = { x , z , w } . The grey center S 4 S_4 S 4 meets only two clusters and is ineligible. (b) The multigraph H H H , with one edge per eligible set containing both endpoints, so x x x has degree 6. A maximal matching (bold) satisfies ∣ M ∣ ≤ Φ ≤ 2 ∣ M ∣ |M|\leq\Phi\leq2|M| ∣ M ∣ ≤ Φ ≤ 2∣ M ∣ . (c) With cap b = 2 b=2 b = 2 , x x x keeps S 1 , S 2 S_1,S_2 S 1 , S 2 and forgets S 3 S_3 S 3 , so the dashed edges disappear. Only a full vertex can forget an edge, and the lists of unmatched vertices are disjoint, so at most 2 n / b 2n/b 2 n / b unmatched vertices are full.
Φ = ∣ U ∣ − SC ( U , F ) . \Phi= |U| - \operatorname{SC}(U,\mathcal{F}). Φ = ∣ U ∣ − SC ( U , F ) .
Intuitively, the MST has ∣ U ∣ − 1 |U|-1 ∣ U ∣ − 1 edges of length at least a a a . Once centers are allowed, only SC ( U , F ) − 1 \operatorname{SC}(U,\mathcal{F})-1 SC ( U , F ) − 1 of them are still needed, as the others can be replaced by short edges to centers. Thus Φ \Phi Φ counts the long MST edges at this scale that we save. [16 ] show that it suffices to estimate Φ \Phi Φ at every scale within a constant factor plus an additive ε ∣ U ∣ \varepsilon|U| ε ∣ U ∣ , using only membership queries “is cluster C C C in S v S_v S v ?”, each of which is implemented by a few distance queries. They give such an estimator with O ~ ( ∣ F ∣ 7 / 4 ) \widetilde{O}(|\mathcal{F}|^{7/4}) O ( ∣ F ∣ 7/4 ) membership queries, which leads to their O ~ ( n 13 / 7 ) \widetilde{O}(n^{13/7}) O ( n 13/7 ) bound.
Estimating the saving via maximal matching. Both [16 ] and [31 ] estimate the saving through a maximal matching in an auxiliary graph, and we follow the more efficient implementation of [31 ] . Consider the multigraph H H H whose vertices are the elements of U U U , with one edge between e e e and e ′ e' e ′ for every eligible set containing both (Figure 1 (b)). Let M \mathcal{M} M be any maximal matching of H H H . Then
∣ M ∣ ≤ Φ ≤ 2 ∣ M ∣ . |\mathcal{M}| \le\Phi\le2|\mathcal{M}|. ∣ M ∣ ≤ Φ ≤ 2∣ M ∣.
The left inequality holds because the ∣ M ∣ |\mathcal{M}| ∣ M ∣ sets labelling the matched edges cover 2 ∣ M ∣ 2|\mathcal{M}| 2∣ M ∣ elements. The right one holds because the unmatched vertices form an independent set, so no eligible set contains two of them, and each of them needs its own set in any cover. The size of a random greedy maximal matching can be estimated by the local algorithm of [N008, YYI09, Beh21]: sample a few uniformly random vertices and, for each, decide whether it is matched by exploring its incident edges in random rank order and recursing on the other endpoints. For a uniformly random starting vertex the exploration visits about d ˉ \bar{d} d ˉ vertices in expectation, where d ˉ \bar{d} d ˉ is the average degree of H H H . Each visited vertex costs about n n n membership queries, since one has to query all n n n sets to learn which of them contain the element.1 Thus, one local query costs roughly d ˉ ⋅ n \bar{d}\cdot n d ˉ ⋅ n , and
d ˉ ≤ ( number of sets containing an element ) × ( number of elements in a set ) . \bar{d} \le\bigl(\text{number of sets containing an element}\bigr) \times\bigl(\text{number of elements in a set}\bigr). d ˉ ≤ ( number of sets containing an element ) × ( number of elements in a set ) .
Everything is therefore about making both factors small, cheaply. [31 ] sparsify both. Sets with more than n x n^x n x elements are detected by testing each set against O ~ ( n 1 − x ) \widetilde{O}(n^{1-x}) O ( n 1 − x ) random elements, taken into the cover, and their elements deleted. Symmetrically, elements lying in more than n x n^x n x sets are detected by testing O ~ ( n 1 − x ) \widetilde{O}(n^{1-x}) O ( n 1 − x ) random sets against all elements, covered by those sets, and deleted. Each step costs O ~ ( n 2 − x ) \widetilde{O}(n^{2-x}) O ( n 2 − x ) queries and adds O ~ ( n 1 − x ) \widetilde{O}(n^{1-x}) O ( n 1 − x ) sets to the additive error. Then, we have d ˉ ≤ n 2 x \bar{d}\leq n^{2x} d ˉ ≤ n 2 x , one local query costs O ~ ( n 1 + 2 x ) \widetilde{O}(n^{1+2x}) O ( n 1 + 2 x ) , and balancing against the sparsification cost n 2 − x n^{2-x} n 2 − x gives x = 1 / 3 x=1/3 x = 1/3 , i.e. O ~ ( n 5 / 3 ) \widetilde{O}(n^{5/3}) O ( n 5/3 ) .
Improving the Set Cover Estimator: Capping Instead of Removing
Challenge (1): the exploration cost is a product of two thresholds. [31 ] bring both factors of d ˉ \bar{d} d ˉ down to n x n^x n x , and each of the two sparsification steps costs O ~ ( n 2 − x ) \widetilde{O}(n^{2-x}) O ( n 2 − x ) queries. The exploration cost n 1 + 2 x n^{1+2x} n 1 + 2 x grows with the product of the two thresholds, so at the balance point x = 1 / 3 x=1/3 x = 1/3 , lowering either threshold further costs more queries than it saves. To go below n 5 / 3 n^{5/3} n 5/3 we need to make one of the two factors small without paying for it. We show that this is possible for the frequency of elements: it can be replaced by a constant at no cost in queries and with a negligible loss in the estimate.
Cap the incidence lists. Instead of removing an element that lies in many sets, we keep it and let it forget most of its sets. Fix an arbitrary global order on the sets, say by index. For each element e e e let its list L ( e ) L(e) L ( e ) be the first b b b sets in this order that contain e e e , or all of them if there are fewer than b b b , where b b b is a constant fixed below. Define the truncated multigraph H b H_b H b on U U U by putting an edge between e e e and e ′ e' e ′ labelled S S S if and only if S S S is eligible and S ∈ L ( e ) ∩ L ( e ′ ) S\in L(e)\cap L(e') S ∈ L ( e ) ∩ L ( e ′ ) : both endpoints must remember S S S (Figure 1 (c)). Computing L ( e ) L(e) L ( e ) costs no more than the n n n queries we already spend when visiting e e e , since we scan the sets in order and stop after b b b hits. The degree of every vertex of H b H_b H b is at most b b b times the maximum set size, regardless of how many sets contain it, so the frequency factor in d ˉ \bar{d} d ˉ has been replaced by the constant b b b for free. The question is whether a maximal matching of H b H_b H b still estimates Φ \Phi Φ .
Why capping is safe. The lower bound ∣ M b ∣ ≤ Φ |\mathcal{M}_b| \le\Phi ∣ M b ∣ ≤ Φ is unaffected, as every edge of H b H_b H b is an edge of H H H . The danger is in the upper bound. Two vertices left unmatched by a maximal matching M b \mathcal{M}_b M b of H b H_b H b might share an eligible set that one of them forgot. Call a vertex full if its list was truncated, i.e. ∣ L ( e ) ∣ = b |L(e)| = b ∣ L ( e ) ∣ = b . Because every forgotten set belongs to a full vertex, the same argument applies without modification to the unmatched vertices that are not full: no eligible set contains two of them, and each requires its own set in any cover.
It remains to bound the number of full unmatched vertices, and here is the key observation: the lists of unmatched vertices are pairwise disjoint on eligible sets. Indeed, if an eligible set S S S belonged to the lists of two unmatched vertices e , e ′ e,e' e , e ′ , then e e ′ ee' e e ′ would be an edge of H b H_b H b with both endpoints unmatched, contradicting maximality. Consequently the lists of all unmatched vertices together contain at most 2 n 2n 2 n entries, since each eligible set appears in at most one of them and each ineligible set, having at most two elements, in at most two. As a full vertex contributes b b b of these entries, there are at most 2 n / b 2n/b 2 n / b full unmatched vertices. Writing I I I for the set of unmatched vertices we get S C ≥ ∣ I ∣ − 2 n / b \mathrm{SC} \ge|I| - 2n/b SC ≥ ∣ I ∣ − 2 n / b , and therefore
1 2 Φ − n b ≤ ∣ M b ∣ ≤ Φ . \frac{1}{2}\Phi- \frac{n}{b} \le|\mathcal{M}_b| \le\Phi. 2 1 Φ − b n ≤ ∣ M b ∣ ≤ Φ.
Choosing b = 100 n / ( ε ∣ U ∣ ) b = 100n/(\varepsilon|U|) b = 100 n / ( ε ∣ U ∣ ) , which is a constant when ∣ U ∣ = Θ ( n ) |U| = \Theta(n) ∣ U ∣ = Θ ( n ) , puts the loss n / b n/b n / b inside the additive error ε ∣ U ∣ \varepsilon|U| ε ∣ U ∣ .
The new balance. With the frequency factor gone, only the first sparsification step remains. If we remove sets with more than P P P elements, at cost O ~ ( n ⋅ n / P ) \widetilde{O}(n \cdot n/P) O ( n ⋅ n / P ) , the degree of H b H_b H b is at most b P bP b P and one local query costs O ~ ( b P ⋅ n ) \widetilde{O}(bP \cdot n) O ( b P ⋅ n ) . Balancing n 2 / P n^2/P n 2 / P against b P n bPn b P n gives P = n P = \sqrt{n} P = n and a total of O ~ ( n 3 / 2 ) \widetilde{O}(n^{3/2}) O ( n 3/2 ) .
Why the Improved Set Cover Estimator Alone Is Not Enough
Challenge (2): the improvement does not plug into the framework of [16 , 31 ] . Recall that the universe U U U at scale a a a is the set of clusters at this scale, and that the estimator must estimate Φ \Phi Φ up to an additive error of ε ∣ U ∣ \varepsilon|U| ε ∣ U ∣ . The budget is proportional to ∣ U ∣ |U| ∣ U ∣ rather than to n n n because the MST pays about a ∣ U ∣ a|U| a ∣ U ∣ at scale a a a , so an error of ε ∣ U ∣ \varepsilon|U| ε ∣ U ∣ in Φ \Phi Φ costs about ε a ∣ U ∣ \varepsilon a|U| ε a ∣ U ∣ in the discount, which sums to O ( ε L ) O(\varepsilon L) O ( ε L ) over the scales. So far we have implicitly assumed ∣ U ∣ = Θ ( n ) |U| = \Theta(n) ∣ U ∣ = Θ ( n ) , which is what made the cap b = 100 n / ( ε ∣ U ∣ ) b = 100n/(\varepsilon|U|) b = 100 n / ( ε ∣ U ∣ ) a constant, but at a given scale ∣ U ∣ |U| ∣ U ∣ can be much smaller. If ∣ U ∣ |U| ∣ U ∣ is very small, no estimator is needed, since we can compute the whole instance by brute force, i.e., query the distance from a few representative terminals of each cluster to every center, which costs O ~ ( n ) \widetilde{O}(n) O ( n ) queries per cluster, and then compute Φ \Phi Φ exactly. Within a budget of O ~ ( n 3 / 2 ) \widetilde{O}(n^{3/2}) O ( n 3/2 ) this handles ∣ U ∣ ≤ n |U| \le\sqrt{n} ∣ U ∣ ≤ n , so the estimator must handle every ∣ U ∣ |U| ∣ U ∣ between n \sqrt{n} n and n n n .
Take ∣ U ∣ = n |U| = \sqrt{n} ∣ U ∣ = n . Fitting the capping loss 2 n / b 2n/b 2 n / b into ε ∣ U ∣ \varepsilon|U| ε ∣ U ∣ forces b ≈ n / ( ε ∣ U ∣ ) = n / ε b \approx n/(\varepsilon|U|) = \sqrt{n}/\varepsilon b ≈ n / ( ε ∣ U ∣ ) = n / ε , which is no longer a constant, and keeping the exploration cost b P n bPn b P n at n 3 / 2 n^{3/2} n 3/2 then requires the large-set threshold to shrink to P ≈ ∣ U ∣ / n P \approx|U|/\sqrt{n} P ≈ ∣ U ∣/ n , so that ∣ U ∣ |U| ∣ U ∣ cancels in b P ≈ n / ε bP \approx\sqrt{n}/\varepsilon b P ≈ n / ε . Both parameters must thus be defined relative to ∣ U ∣ |U| ∣ U ∣ . This is fine in principle, as sets with more than ∣ U ∣ / n |U|/\sqrt{n} ∣ U ∣/ n elements are detected by testing every center against O ~ ( n ) \widetilde{O}(\sqrt{n}) O ( n ) uniformly random elements of U U U , but it requires sampling uniformly random clusters cheaply, which the representation of [16 , 31 ] does not allow. There a cluster is represented by a greedily built net of representative terminals, and the only way to get a uniformly random cluster is to pick a random terminal and run a breadth-first search over the terminals to find its cluster, which costs O ~ ( k ) \widetilde{O}(k) O ( k ) queries per attempt while only a ∣ U ∣ / k |U|/k ∣ U ∣/ k fraction of the attempts succeed. For this reason, [16 , 31 ] sample terminals instead, which is free, and call a set large if it contains more than P P P terminals. The number of large sets is then k / P k/P k / P , and since all of them are taken into the cover, the error budget forces P ≳ k / ( ε ∣ U ∣ ) P \gtrsim k/(\varepsilon|U|) P ≳ k / ( ε ∣ U ∣ ) . Together with b ≳ n / ( ε ∣ U ∣ ) b \gtrsim n/(\varepsilon|U|) b ≳ n / ( ε ∣ U ∣ ) this gives b P ≳ n k / ( ε 2 ∣ U ∣ 2 ) bP \gtrsim nk/(\varepsilon^2|U|^2) b P ≳ nk / ( ε 2 ∣ U ∣ 2 ) , in which ∣ U ∣ |U| ∣ U ∣ does not cancel, and balancing b P n bP n b P n against the brute-force cost n ∣ U ∣ n|U| n ∣ U ∣ brings us back to ∣ U ∣ ≈ n 2 / 3 |U| \approx n^{2/3} ∣ U ∣ ≈ n 2/3 and O ~ ( n 5 / 3 ) \widetilde{O}(n^{5/3}) O ( n 5/3 ) . The problem is a mismatch of units, as the capping loss is counted in clusters while set sizes are counted in terminals. To benefit from capping we therefore need a representation of the universe in which sampling a uniformly random element, estimating ∣ U ∣ |U| ∣ U ∣ , and measuring set sizes are all cheap and expressed in the same unit.
Measuring Everything in Sampled Points
A random sample of terminals as the unit. To overcome this issue, instead of constructing a greedy net, we represent the terminals at scale a a a using a random subset N N N . Specifically, we include each terminal u u u in N N N independently with probability roughly inversely proportional to the number of terminals in a small ball around u u u (up to a logarithmic factor). With high probability, every terminal is within distance ρ \rho ρ of a point of N N N , where ρ \rho ρ is a small constant fraction of a a a . Moreover, a ball with d d d terminals contributes only about log n \log n log n points to N N N in expectation, since each of them is kept with probability about log n / d \log n/d log n / d , and the terminals can be covered by O ~ ( L / a ) \widetilde{O}(L/a) O ( L / a ) such balls, as any ρ \rho ρ -separated set of terminals spans an MST with that many edges of length at least ρ \rho ρ and weight at most 2 L 2L 2 L . Hence ∣ N ∣ ≤ O ~ ( L / a ) |N| \le\widetilde{O}(L/a) ∣ N ∣ ≤ O ( L / a ) . Crucially, we do not construct N N N explicitly. Whenever the algorithm first visits a terminal u u u , it estimates the local density around u u u using a few random distance queries and decides on the fly whether u ∈ N u \in N u ∈ N . This allows us to sample a uniformly random point of N N N (and estimate ∣ N ∣ |N| ∣ N ∣ ) in O ~ ( k ) \widetilde{O}(k) O ( k ) queries by picking random terminals until one belongs to N N N . We then define the set cover instance at scale a a a over the connected components of N N N , where a center’s set consists of the components with a point of N N N within distance 5 a / 8 5a/8 5 a /8 of it. By setting both the large-set threshold P ≈ ∣ N ∣ / n P \approx|N|/\sqrt{n} P ≈ ∣ N ∣/ n and the cap b ≈ n / ( ε ∣ N ∣ ) b \approx n/(\varepsilon|N|) b ≈ n / ( ε ∣ N ∣ ) in terms of ∣ N ∣ |N| ∣ N ∣ , the dependency on ∣ N ∣ |N| ∣ N ∣ cancels in b P ≈ n / ε bP \approx\sqrt{n}/\varepsilon b P ≈ n / ε at every scale. The only trade-off is an additive error of ε ∣ N ∣ \varepsilon|N| ε ∣ N ∣ instead of ε ∣ U ∣ \varepsilon|U| ε ∣ U ∣ , which contributes at most ε a ∣ N ∣ ≤ O ~ ( ε L ) \varepsilon a|N| \le\widetilde{O}(\varepsilon L) ε a ∣ N ∣ ≤ O ( ε L ) to the discount per scale and is easily absorbed by setting ε = 1 / polylog ( n ) \varepsilon= 1/\operatorname{polylog}(n) ε = 1/ polylog ( n ) .
Challenge (3): detecting large sets using a single reusable sample. Because N N N is defined implicitly and sampling a single point of N N N takes O ~ ( k ) \widetilde{O}(k) O ( k ) queries, we cannot draw a fresh sample of O ~ ( n ) \widetilde{O}(\sqrt{n}) O ( n ) points for each of the n n n centers during the sparsification step as in [31 ] , which would cost O ~ ( n 5 / 2 ) \widetilde{O}(n^{5/2}) O ( n 5/2 ) queries. Instead, we draw a single sample of O ~ ( n ) \widetilde{O}(\sqrt{n}) O ( n ) points from N N N at the start of the scale in O ~ ( k n ) = O ~ ( n 3 / 2 ) \widetilde{O}(k\sqrt{n}) = \widetilde{O}(n^{3/2}) O ( k n ) = O ( n 3/2 ) queries and reuse it across all n n n centers, removing a sampled point whenever a selected center is within distance 5 a / 8 5a/8 5 a /8 of it. Although reusing a sample across adaptive steps creates dependencies in general, here each sampled point is deleted as soon as it contributes to selecting a center, and a stability argument shows that this reusable sample still accurately detects all large sets. Afterward, the local matching exploration proceeds as in [31 ] in O ~ ( b P n ) = O ~ ( n 3 / 2 ) \widetilde{O}(bPn) = \widetilde{O}(n^{3/2}) O ( b P n ) = O ( n 3/2 ) queries.
Finally, working with the components of N N N rather than the true terminal clusters introduces a few additional technicalities in certifying the discounts and adapting the structural theorem of [16 ] , which we handle in the formal analysis to obtain a ( 2 − Ω ( 1 ) ) (2-\Omega(1)) ( 2 − Ω ( 1 )) -approximation in O ~ ( n 3 / 2 ) \widetilde{O}(n^{3/2}) O ( n 3/2 ) queries and time.
Technical Overview of the Lower Bound
In this section, we give an overview of the lower bound construction in Theorem 1.2 . By Yao’s principle, it suffices to construct two distributions D Y \mathcal{D}_Y D Y and D N \mathcal{D}_N D N over instances such that, with high probability, the optimal cost under D N \mathcal{D}_N D N is more than 2 − η 2-\eta 2 − η times the optimal cost under D Y \mathcal{D}_Y D Y , and to show that every deterministic algorithm with few queries sees almost the same distribution of answers under both. We first describe the instances and explain why their costs differ. We then give a heuristic calculation that explains the exponent b s = ( 3 s + 7 ) / ( 2 s + 5 ) b_s = (3s + 7)/(2s + 5) b s = ( 3 s + 7 ) / ( 2 s + 5 ) which approaches 3 / 2 3/2 3/2 . Finally, we describe the obstacles to turning this calculation into a proof and how we overcome them, step by step. Throughout, η \eta η and s s s are fixed, d d d is a large constant, and we ignore constant factors.
The Hard Instances
Clusters, special terminals, and centers. As in [16 Section 7] , the instances hide special terminals among many more ordinary ones (Figure 4 ). There are A A A ordinary clusters , each consisting of R R R terminals at pairwise distance λ ≈ 0 \lambda\approx0 λ ≈ 0 , so there are K = A R K = AR K = A R ordinary terminals. There are m = d A m = dA m = d A special terminals , each at distance 2 from all other terminals. Finally, there are m R mR m R nonterminal centers . Every center c c c has a random neighbor set nb ( c ) \operatorname{nb}(c) nb ( c ) of special terminals. It is at distance 1 from its neighbors and at distance 2 from all other points. A uniformly random bijection decides which terminal plays which role, so the algorithm knows neither which terminals are special nor how the ordinary terminals are grouped into clusters. The centers dominate the number of points, so n = Θ ( m R ) n = \Theta(mR) n = Θ ( m R ) , while a random terminal is special with probability only about m / K = d / R m/K = d/R m / K = d / R . The ordinary clusters serve as camouflage. Each of them can be connected internally at negligible cost, so together they add only about 2 A = 2 m / d 2A = 2m/d 2 A = 2 m / d to the cost of a Steiner tree, but they make the special terminals rare and hence expensive to find.
Figure 4. The construction, schematically with d = 4 d = 4 d = 4 . Squares are terminals, circles are centers, and lines represent unit distances. Ordinary clusters are the same in both worlds. Fillers have at most two neighbors and give no saving. In YES the stars collectively cover almost all special terminals. In NO they are confined to the small set of positive-weight terminals (orange). The weights, not the geometry or the filler probabilities, distinguish the two distributions.
Structured vs. random centers. Earlier constructions [16 , 33 ] arrange the centers in public blocks and hide fixed gadgets in them. In [16 Section 7] , a center adjacent to two special terminals already reveals a star. The construction of [33 ] makes the view of any two terminals uninformative, so that detecting a star requires three of its terminals, which raises the exponent from 6 / 5 6/5 6/5 to 7 / 5 7/5 7/5 . Its generalization to α + 1 \alpha+ 1 α + 1 terminals gives the exponent ( 3 α + 1 ) / ( 2 α + 1 ) (3\alpha+ 1)/(2\alpha+ 1) ( 3 α + 1 ) / ( 2 α + 1 ) , but only for estimation within factor α / ( α − 1 ) \alpha/(\alpha- 1) α / ( α − 1 ) . We take a different route and replace the structure by randomness. As we explain next, this decouples the hiding from the cost gap: we obtain the exponent ( 3 α + 1 ) / ( 2 α + 1 ) (3\alpha+ 1)/(2\alpha+ 1) ( 3 α + 1 ) / ( 2 α + 1 ) with α = s + 2 \alpha= s + 2 α = s + 2 for every factor 2 − η 2 - \eta 2 − η .
Stars and the cost gap. In our construction, most centers are fillers : their neighbor set is empty, a single special terminal, or a pair of special terminals. Fillers are useless for a Steiner tree, since connecting two terminals through a center costs 1 + 1 = 2 1 + 1 = 2 1 + 1 = 2 , the same as connecting them directly. A rare center is a star : its neighbor set consists of d d d special terminals, which it connects at cost d d d instead of 2 ( d − 1 ) 2(d - 1) 2 ( d − 1 ) . Each center is a star with probability Θ ( 1 / R ) \Theta(1/R) Θ ( 1/ R ) , so that a special terminal lies in a constant number of stars on average, while the number of stars is only a small fraction of m m m because d d d is large. If the stars cover almost all special terminals, we can attach every covered special terminal to a star by a unit edge and then connect the few stars and the clusters by edges of weight 2, so OPT ≈ m \operatorname{OPT} \approx m OPT ≈ m . If the stars touch only a δ \delta δ fraction of the special terminals, for a small constant δ > 0 \delta> 0 δ > 0 , then almost every special terminal still needs an edge of weight 2, so OPT ≈ 2 m \operatorname{OPT} \approx2m OPT ≈ 2 m . Taking δ \delta δ and 1 / d 1/d 1/ d small compared with η \eta η makes the ratio larger than 2 − η 2 - \eta 2 − η .
The only difference is in random weights. The two distributions share all of the above and differ only in where the stars are placed. Every special terminal u u u receives an independent random weight W u ≥ 0 W_u \ge0 W u ≥ 0 with mean 1. Given the weights, each center independently becomes a star on a d d d -set H H H with probability proportional to ∏ u ∈ H W u \prod_{u \in H} W_u ∏ u ∈ H W u , while the probabilities of singleton and pair neighbor sets do not depend on the weights, and the empty neighbor set takes the remaining probability. In D y \mathcal{D}_y D y , every weight is at least 1 / 2 1/2 1/2 , so the stars are spread over all special terminals and cover almost all of them. In D n D_n D n , all but a δ \delta δ fraction of the weights are 0, so all stars lie among the few terminals of positive weight. As the mean weight is 1 in both cases, the expected number of stars is the same. In fact, equal means already make disjoint stars look the same. Suppose for a moment that s = 1 s=1 s = 1 and that every weight equals 1 in D y D_y D y . Given the weights, a fixed collection of stars occurs with probability proportional to ∏ u W u k u \prod_u W_u^{k_u} ∏ u W u k u , where k u k_u k u is the number of these stars that contain u u u . If the stars are pairwise disjoint, then every k u ≤ 1 k_u \le1 k u ≤ 1 , and since the weights are independent with mean 1, this product has expectation 1 in both worlds. If two of the stars share a terminal u u u , then k u = 2 k_u=2 k u = 2 , and E [ W u 2 ] \mathbb{E}[W_u^2] E [ W u 2 ] is 1 in D y D_y D y but at least 1 / δ 1/\delta 1/ δ in D n D_n D n , where the stars crowd onto few terminals.
We therefore choose the two weight distributions to have the same first s s s moments, a standard setup in statistical lower bounds [35 ] . For every fixed s s s this is compatible with the support conditions above: suitable distributions arise from the alternating signs of the coefficients in a classical interpolation identity (Lemmas 5.4 and 5.5 ). The product form of the star probabilities is what makes moment matching effective. Roughly speaking, an observation in which every special terminal belongs to at most s s s of the observed stars has the same probability in both worlds, so to tell them apart, the algorithm has to see some special terminal in s + 1 s+1 s + 1 stars. Note that the cost gap depends only on the supports of the two weight distributions, so increasing s s s improves the hiding without weakening the gap.
A Heuristic Count: Where the Exponent Comes From
We now estimate how many queries an algorithm needs in order to see a special terminal in s + 1 s+1 s + 1 stars. This calculation is not a proof, but it determines the choice of parameters.
Special terminals are expensive to find. Since the roles are random, an algorithm can single out a special terminal in essentially two ways. First, it can query a center against a terminal and receive the answer 1. A center has at most two filler neighbors, hidden among about K K K terminals, so a query against a terminal about which nothing is known yet is answered 1 with probability O ( 1 / K ) O(1/K) O ( 1/ K ) . Second, it can query a terminal against other terminals until it finds a cluster mate. This takes about K / R = A K/R=A K / R = A queries per terminal, and only a d / R d/R d / R fraction of the terminals have no cluster mate. Either way, the algorithm pays about K K K queries for each special terminal that it finds, so with q ≈ K R q \approx KR q ≈ K R queries it finds about R R R of them. Let Q Q Q be the set of special terminals found.
Stars hide behind pairs. Neither one nor two positive answers at a center reveal a star. A center has a pair filler with probability 1 / 4 1/4 1/4 but is a star only with probability Θ ( 1 / R ) \Theta(1/R) Θ ( 1/ R ) , so two positive answers at a center are almost always explained by a pair filler. To recognize a star, the algorithm needs a third positive answer at the same center. Testing the center against the terminals in Q Q Q provides one only if three members of the star lie in Q Q Q . Testing it against terminals that have not been found yet gives a positive answer with probability O ( 1 / K ) O(1/K) O ( 1/ K ) per query, and only if the center is a star in the first place. So this succeeds with probability only O ( 1 / ( K R ) ) O(1/(KR)) O ( 1/ ( K R )) per query, and the algorithm needs about K R KR K R queries to find even a single star in this way. Hence, with fewer queries, the algorithm essentially sees only the stars visible from Q Q Q , namely those with at least three members in Q Q Q .
Counting visible stars. Every special terminal lies in about R R R pair fillers, and the other member of such a pair is a uniformly random special terminal. Thus about R ∣ Q ∣ 2 / m ≈ R 3 / m R|Q|^2/m \approx R^3/m R ∣ Q ∣ 2 / m ≈ R 3 / m centers have two neighbors in Q Q Q . Each of them is a star with probability only O ( 1 / R ) O(1/R) O ( 1/ R ) , and a star has a third neighbor in Q Q Q with probability O ( ∣ Q ∣ / m ) O(|Q|/m) O ( ∣ Q ∣/ m ) , so about R 3 / m 2 R^3/m^2 R 3 / m 2 stars are visible. Moreover, a fixed special terminal u u u is a neighbor of only about R R R centers. Even if all of them have a second neighbor in Q Q Q , the terminal u u u lies in only about R ⋅ 1 R ⋅ R m = R m R \cdot\frac{1}{R} \cdot\frac{R}{m} = \frac{R}{m} R ⋅ R 1 ⋅ m R = m R visible stars. This can indeed happen, since an adaptive algorithm can put the other members of all about R R R pair fillers of u u u into Q Q Q . It does not contradict the total count: Q Q Q contains only about R 3 / m R^3/m R 3 / m filler pairs, so only about R 2 / m R^2/m R 2 / m terminals can have all their partners in Q Q Q at once, and R 2 m ⋅ R m = R 3 m 2 \frac{R^2}{m} \cdot\frac{R}{m} = \frac{R^3}{m^2} m R 2 ⋅ m R = m 2 R 3 . Each further visible star through the same terminal therefore costs a factor of about R / m R/m R / m , and the probability that some special terminal lies in s + 1 s+1 s + 1 visible stars is about
R 3 m 2 ( R m ) s = R s + 3 m s + 2 . \frac{R^3}{m^2}\left(\frac{R}{m}\right)^s = \frac{R^{s+3}}{m^{s+2}}. m 2 R 3 ( m R ) s = m s + 2 R s + 3 .
Balancing. The two worlds look alike as long as this probability is small, that is, as long as R R R is at most a small constant times m ( s + 2 ) / ( s + 3 ) m^{(s+2)/(s+3)} m ( s + 2 ) / ( s + 3 ) . Since n = Θ ( m R ) n = \Theta(mR) n = Θ ( m R ) , this allows m ≈ n ( s + 3 ) / ( 2 s + 5 ) m \approx n^{(s+3)/(2s+5)} m ≈ n ( s + 3 ) / ( 2 s + 5 ) and R ≈ n ( s + 2 ) / ( 2 s + 5 ) R \approx n^{(s+2)/(2s+5)} R ≈ n ( s + 2 ) / ( 2 s + 5 ) , and the number of queries is then q ≈ K R = Θ ( n R ) = Θ ( n ( 3 s + 7 ) / ( 2 s + 5 ) ) q \approx KR = \Theta(nR) = \Theta(n^{(3s+7)/(2s+5)}) q ≈ K R = Θ ( n R ) = Θ ( n ( 3 s + 7 ) / ( 2 s + 5 ) ) , which is the bound of Theorem 1.2 . As s s s grows, both R R R and m m m approach n \sqrt{n} n , and the bound approaches n 3 / 2 n^{3/2} n 3/2 . Heuristically, this is also where constructions of this type must stop: once R R R is comparable with m m m , an algorithm with about K R ≈ n 3 / 2 KR \approx n^{3/2} K R ≈ n 3/2 queries finds a constant fraction of all special terminals, and many stars become visible. This is consistent with Theorem 1.1 .
Hiding the Stars: Keys and Exposure
The heuristic count makes three simplifying assumptions: that answers carry no information about the weights unless they reveal a star, that an algorithm cannot find special terminals and stars faster than described, and that it suffices to count the stars that are actually seen. The first and the last are false as stated, the second needs a proof, and all three are complicated by adaptivity. We now describe how we deal with these issues.
Challenge (1): every answer leaks a little about the weights. In the metric, the weights affect every query between a center and a special terminal, not only those that reveal a star. For example, the first query { c , u } \{c,u\} { c , u } at a center c c c is answered 1 slightly more often when W u W_u W u is large, because a star containing u u u is then more likely. The bias is only O ( 1 / ( R m ) ) O(1/(Rm)) O ( 1/ ( R m )) on a probability of order 1 / m 1/m 1/ m , but the algorithm makes many such queries, and we have to show that the biases do not add up. To this end, we compare the metric with an auxiliary random process, the two-key experiment , which answers the queries of the algorithm adaptively (§ 6 ). In it, every center first draws a filler without using the weights. If the filler is a pair, its two members are the keys of the center, and the center may also hold a hidden potential star : a d d d -set that contains both keys and is drawn using the weights (Figure 5 ). The center answers like its filler until both keys have been queried at it, and only after this activation can its potential star affect the answers. Thus every answer at a center before its activation is independent of the weights, and a star stays locked until both of its keys have been found.
Figure 5. A center c c c in the two-key experiment, with d = 4 d=4 d = 4 . Its filler is the pair of keys (solid), and its potential star H c H_c H c consists of the keys and d − 2 d-2 d − 2 further special terminals (dashed). Until both keys have been queried at c c c , the center answers 1 exactly for the keys. Afterwards it also answers 1 for the other members of H c H_c H c , unless H c H_c H c contains a terminal that was answered 2 at c c c earlier.
Replacing the metric by the two-key experiment changes the distribution of the transcript by O ( q / ( m R 2 ) + q / ( m R 2 ) ) O(\sqrt{q/(mR^2)} + q/(mR^2)) O ( q / ( m R 2 ) + q / ( m R 2 )) in total variation distance. Before activation, the two processes answer each query with probabilities that differ by O ( 1 / ( R m ) ) O(1/(Rm)) O ( 1/ ( R m )) , around a base probability of order 1 / m 1/m 1/ m . So each query contributes O ( 1 / ( m R 2 ) ) O(1/(mR^2)) O ( 1/ ( m R 2 )) to the relative entropy between the transcript distributions, and Pinsker’s inequality turns the sum over the q q q queries into the first term. After activation, the two processes differ only in the probability that a center with two positive answers holds a star. This probability is O ( 1 / R ) O(1/R) O ( 1/ R ) in both, and the two values differ by O ( 1 / R 2 ) O(1/R^2) O ( 1/ R 2 ) . Since only O ( q / m ) O(q/m) O ( q / m ) centers are activated in expectation, this gives the second term. For q ≤ c 0 K R q \le c_0KR q ≤ c 0 K R with a small constant c 0 c_0 c 0 , both terms are small, since K R = Θ ( m R 2 ) KR = \Theta(mR^2) K R = Θ ( m R 2 ) . This is where the pair fillers are essential: they make two positive answers at a center an almost certain sign of a pair, so the first two contacts with a star look like contacts with a pair. The singleton fillers keep the probability of a positive answer of order 1 / m 1/m 1/ m even after many negative answers at the same center, which the relative entropy bound needs.
Challenge (2): bounding what an adaptive algorithm can expose. Next, we make the first two steps of the heuristic count rigorous for the two-key experiment (§ 7.1 ). For the analysis only, we maintain a set Q Q Q of exposed special terminals. A special terminal is noticed and added to Q Q Q when it first receives a positive answer, when it has been involved in about A A A queries, or when one of a few similar counters becomes large. Moreover, whenever a center has both keys and a third member of its potential star in Q Q Q , we add the entire potential star to Q Q Q . This closure only gives the algorithm more information. A query is lucky if it asks a center about an unexposed member of its potential star. We stop the analysis at the first lucky query, or when one of a few other unlikely events happens, for example when more than R R R terminals have been noticed. Before this stop, no special terminal outside Q Q Q has been involved in a query that met a filler or a potential star. Consequently, the answers depend only on the bijection, the fillers, and the potential stars visible from Q Q Q , namely those with both keys and a third member in Q Q Q (Figure 2 ).
Figure 2. Visibility in the two-key experiment, with d = 4 d = 4 d = 4 and with the ordinary terminals omitted. Squares are special terminals, circles are centers, and the shaded block is the exposed set Q Q Q . Solid lines join each center to its keys, and dashed lines join it to the other members of its potential star. The left star is visible, since both keys and a third member lie in Q Q Q . Closure then adds its last member (dotted) to Q Q Q . The right star is not visible, since only its keys lie in Q Q Q . It can affect an answer only through a lucky query against one of its unexposed members, or after these members are exposed for other reasons.
To show that the stop is unlikely, we use an exchange argument , similar to those in [16 , 33 ] . Consider a special terminal u u u outside Q Q Q . Every earlier query involving u u u was answered exactly as it would have been for most ordinary terminals, so swapping the labels of u u u and of such an ordinary terminal yields another configuration that is consistent with the transcript so far. Since the bijection is uniform, such swaps preserve probabilities, and counting them shows that, given the transcript, a terminal outside Q Q Q is special with probability O ( m / K ) O(m/K) O ( m / K ) and belongs to the filler of a given center with probability O ( 1 / K ) O(1/K) O ( 1/ K ) . This makes the cost of finding special terminals rigorous: q q q queries notice O ( q / K ) O(q/K) O ( q / K ) special terminals in expectation, so for q ≤ c 0 K R q \le c_0KR q ≤ c 0 K R , more than R R R terminals are noticed only with small probability. A lucky query also needs a center that holds a potential star. Here we use that a potential star can be erased without changing the transcript, as long as its center has given at most two positive answers. Hence conditioning on the transcript can rule out some potential stars, but never their absence, so a center holds a potential star with probability O ( 1 / R ) O(1/R) O ( 1/ R ) even given the transcript, and each query is lucky with probability O ( 1 / ( K R ) ) O(1/(KR)) O ( 1/ ( K R )) . Finally, the exchange argument needs a large supply of ordinary terminals to swap with. This supply lasts unless the algorithm finds the cluster mates of a constant fraction of all ordinary terminals, which takes about A K ≫ K R AK \gg KR A K ≫ K R queries.
Hiding the Weights: Moment Matching with Bounded Reuse
Challenge (3): the absence of a star is evidence too. By the above, until the first stop the transcript can be computed from the weight-independent data, namely the bijection and the fillers, together with the replies to star requests , each of which reveals all potential stars visible from the current set Q Q Q (§ 7.2 ). Here Q Q Q only grows and has size at most 2 R 2R 2 R . It therefore suffices to show that the replies have almost the same distribution in the two worlds. Given the weights, the probability of a sequence of replies is a polynomial in the weights, and since the weights are independent with matching moments, its average is the same in both worlds if every weight appears in it with degree at most s s s . A reply, however, reveals not only the visible stars but also the absence of all other visible stars, and absence carries information too. For instance, a terminal of weight 0 0 0 never lies in a star, so each center that has this terminal as a key and holds no visible star slightly favors D n \mathcal{D}_n D n . In the two-key experiment, the probability of such an absence involves the weights of all candidate supports at the center, and these factors accumulate over many centers. So the degree of a weight is not controlled by the number of stars that are actually revealed.
We therefore generate the potential stars differently in the analysis. For every center with a pair filler and every d d d -set H H H that contains its keys, a proposal is made independently with a small probability that does not depend on the weights, and a proposal is retained with probability p H ( W ) = ∏ u ∈ H ( W u / M ) p_H(W) = \prod_{u\in H}(W_u/M) p H ( W ) = ∏ u ∈ H ( W u / M ) , where M M M bounds the weights. The proposal probability is chosen so that each support is retained with exactly the same probability as in the two-key experiment. This proposal model differs from the two-key experiment only when a center has two or more retained supports, which happens with probability O ( 1 / R 2 ) O(1/R^2) O ( 1/ R 2 ) per center. As we will see, only O ( R 3 / m ) O(R^3/m) O ( R 3 / m ) centers have both keys in Q Q Q , so this changes the distribution of the replies by only O ( R / m ) O(R/m) O ( R / m ) . In the proposal model, a star request reveals all proposals in the requested region together with their retention bits, rejected proposals included. The probability of such a record is a weight-independent factor times a product of factors p H ( W ) p_H(W) p H ( W ) for retained proposals and 1 − p H ( W ) 1-p_H(W) 1 − p H ( W ) for rejected ones. The coin with probability p H ( W ) p_H(W) p H ( W ) is flipped only when H H H is proposed, so once the proposals are revealed, the supports that were not proposed do not depend on W W W . So the degree of W u W_u W u is at most the number of revealed proposals containing u u u , and expanding the products shows that every record in which no terminal lies in more than s s s revealed proposals has exactly the same probability in the two worlds. Thus the two worlds can be coupled until some terminal lies in s + 1 s+1 s + 1 revealed proposals.
Challenge (4): bounding reuse when Q Q Q is adaptive. It remains to show, following the heuristic count, that some terminal lies in s + 1 s+1 s + 1 revealed proposals only with small probability. The difficulty is that Q Q Q depends on the transcript, and an adaptive algorithm could try to steer Q Q Q toward sets that contain unusually many filler pairs. We handle this with a bound that holds uniformly: with high probability, simultaneously for all sets Q Q Q of special terminals, the number of pair fillers with both members in Q Q Q is O ( ∣ Q ∣ Q 2 / m + ∣ Q ∣ log m ) O(|Q|Q^2/m+|Q|\log m) O ( ∣ Q ∣ Q 2 / m + ∣ Q ∣ log m ) , and every special terminal lies in O ( R ) O(R) O ( R ) pair fillers. For a fixed set Q Q Q of size h h h , the first bound fails with probability m − Ω ( h ) m^{-\Omega(h)} m − Ω ( h ) by a Chernoff bound, which beats the at most m h m^h m h sets of that size in a union bound. Hence, up to lower-order terms, every star request, which has ∣ Q ∣ ≤ 2 R |Q|\leq2R ∣ Q ∣ ≤ 2 R , concerns a region of proposals of total proposal probability O ( R 3 / m 2 ) O(R^3/m^2) O ( R 3 / m 2 ) , in which the proposals containing any fixed terminal have total probability O ( R / m ) O(R/m) O ( R / m ) , as in the heuristic count.
Now let N u N_u N u be the number of revealed proposals containing u u u . Whether a proposal is made is independent of everything revealed before it, so each proposal containing u u u is revealed with probability equal to its proposal probability. If N u = k N_u=k N u = k at that moment, it increases the number of ( s + 1 ) (s+1) ( s + 1 ) -subsets of revealed proposals containing u u u by ( k s ) \binom{k}{s} ( s k ) . As the proposals containing u u u have total probability O ( R / m ) O(R/m) O ( R / m ) , this gives E [ ( N u j + 1 ) ] ≤ O ( R / m ) ⋅ E [ ( N u j ) ] \mathbb{E}\left[\binom{N_u}{j+1}\right]\leq O(R/m)\cdot\mathbb{E}\left[\binom{N_u}{j}\right] E [ ( j + 1 N u ) ] ≤ O ( R / m ) ⋅ E [ ( j N u ) ] for every j j j , and hence E [ N s + 1 ] ≤ ( O ( R / m ) ) s E [ N u ] \mathbb{E}[N_{s+1}] \leq(O(R/m))^{s}\mathbb{E}[N_u] E [ N s + 1 ] ≤ ( O ( R / m ) ) s E [ N u ] . Summing over u u u , some terminal lies in s + 1 s+1 s + 1 revealed proposals with probability O ( R 3 / m 2 ⋅ ( R / m ) s ) = O ( R s + 3 / m s + 2 ) O(R^3/m^2 \cdot(R/m)^s) = O(R^{s+3}/m^{s+2}) O ( R 3 / m 2 ⋅ ( R / m ) s ) = O ( R s + 3 / m s + 2 ) , as predicted. The same bound on the expected number of revealed proposals shows that closure, which adds at most d d d terminals per revealed star, adds more than R R R terminals to Q Q Q only with probability O ( R 2 / m 2 ) O(R^2/m^2) O ( R 2 / m 2 ) . So the exposed set indeed stays of size at most 2 R 2R 2 R .
Putting things together. Collecting the errors, for every deterministic algorithm with q q q queries, the total variation distance between its transcripts under D y \mathcal{D}_y D y and D n \mathcal{D}_n D n is at most
O ( q m R 2 + q K R + q A K + R s + 3 m s + 2 + R m + log m R ) + m − 10 . O\left(\sqrt{\frac{q}{mR^2}}+\frac{q}{KR}+\frac{q}{AK}+\frac{R^{s+3}}{m^{s+2}}+\frac{R}{m}+\frac{\log m}{R}\right)+m^{-10}. O ( m R 2 q + K R q + A K q + m s + 2 R s + 3 + m R + R log m ) + m − 10 .
The first term comes from the two-key experiment; the second from noticing too many special terminals and from lucky queries; the third from finding too many clusters; the fourth from the reuse of a weight; and the remaining ones from the proposal model, closure, and the uniform sparsity bound. Concretely, we take A = t s + 3 A=t^{s+3} A = t s + 3 and R = γ 0 t s + 2 R=\gamma_0t^{s+2} R = γ 0 t s + 2 , so that R R R is a small constant times m ( s + 2 ) / ( s + 3 ) m^{(s+2)/(s+3)} m ( s + 2 ) / ( s + 3 ) , and q = c 0 K R q=c_0KR q = c 0 K R for a small constant c 0 c_0 c 0 . Then n = Θ ( t 2 s + 5 ) n=\Theta(t^{2s+5}) n = Θ ( t 2 s + 5 ) and the distance is at most 1 / 10 1/10 1/10 , so by Le Cam’s two-point method, no algorithm with at most c 0 K R = Θ ( n ( 3 s + 7 ) / ( 2 s + 5 ) ) c_0KR=\Theta(n^{(3s+7)/(2s+5)}) c 0 K R = Θ ( n ( 3 s + 7 ) / ( 2 s + 5 ) ) queries can estimate OPT within a factor 2 − η 2-\eta 2 − η with probability 2 / 3 2/3 2/3 . Padding with extra nonterminals extends the bound to every n n n , and Markov’s inequality extends it to expected query complexity.
Organization
§ 2 collects notation and standard facts. § 3 and § 4 prove Theorem 1.1 : § 3 proves the structural theorem, and § 4 presents and analyzes the O ~ ( n 3 / 2 ) \widetilde{O}(n^{3/2}) O ( n 3/2 ) estimation algorithm. The lower bound is independent of these sections. § 5 to § 7 prove Theorem 1.2 : § 5 constructs the two hard distributions and proves the gap in their optimal costs, § 6 replaces the metric by a simpler random process called the two-key experiment, and § 7 analyzes this process and completes the proof. We conclude in § 8 .
Preliminaries
Instances and conventions. An instance is a triple ( V , T , w ) (V,T,w) ( V , T , w ) with ∣ V ∣ = n |V|=n ∣ V ∣ = n and ∣ T ∣ = k |T|=k ∣ T ∣ = k . We write L L L for the terminal MST weight and OPT for the Steiner tree cost. The notation O ~ ( ⋅ ) \widetilde{O}(\cdot) O ( ⋅ ) hides fixed powers of log ( n ) \log(n) log ( n ) .
Threshold graphs and contraction benefit. For X ⊆ V X\subseteq V X ⊆ V and a > 0 a>0 a > 0 let G a ( X ) G_a(X) G a ( X ) be the graph on X X X whose edges are the pairs at distance less than a a a , let comp ( G ) \operatorname{comp}(G) comp ( G ) be the set of connected components of a graph G G G , and let κ T ( a ) = ∣ comp ( G a ( T ) ) ∣ \kappa_T(a)=|\operatorname{comp}(G_a(T))| κ T ( a ) = ∣ comp ( G a ( T )) ∣ . Kruskal’s algorithm gives
L = ∫ 0 ∞ ( κ T ( a ) − 1 ) d a . (2) L=\int_0^\infty(\kappa_T(a)-1)\,da. \tag*{(2)} L = ∫ 0 ∞ ( κ T ( a ) − 1 ) d a . (2)
For a family F \mathcal{F} F of subsets of T T T , the contraction benefit B T ( F ) B_T(\mathcal{F}) B T ( F ) is the decrease of the MST weight when every set of F \mathcal{F} F is identified at zero cost; for one set Y Y Y we write B T ( Y ) B_T(Y) B T ( Y ) . For Y ⊆ T Y\subseteq T Y ⊆ T let c ( Y ) c(Y) c ( Y ) be the least weight of a tree spanning Y Y Y and at most one additional point of V V V . For ∣ Y ∣ ≤ 4 |Y|\leq4 ∣ Y ∣ ≤ 4 it is computed from the distances between Y Y Y and V V V in O ( n ) O(n) O ( n ) time.
Lemma 2.1 (Rank monotonicity and lifting). For every threshold a a a , the decrease of the number of components of G a ( T ) G_a(T) G a ( T ) caused by a set of zero links, given a fixed set of earlier zero links, does not increase when a a a grows or when more earlier links are added. The benefit is the integral of this decrease over a a a . Moreover, if networks of total weight C F C_{\mathcal{F}} C F connect each set of F \mathcal{F} F , then L − OPT ≥ B T ( F ) − C F L-\operatorname{OPT} \ge B_T(\mathcal{F})-C_{\mathcal{F}} L − OPT ≥ B T ( F ) − C F .
Proof. Apply (2) before and after adding the zero links. At each threshold their contribution is their rank in the graphic matroid of the complete graph on T T T contracted by the edges of G a ( T ) G_a(T) G a ( T ) and by the earlier links; contracting more edges cannot increase it. For the last claim lift an MST of the contracted metric to the original points and add the networks: this connects T T T at weight at most L − B T ( F ) + C F L-B_T(\mathcal{F})+C_{\mathcal{F}} L − B T ( F ) + C F .
Probability tools
All distributions that we compare are on finite sets. Let P P P and Q Q Q be distributions on a finite set Ω \Omega Ω . Their total variation distance is
d T V ( P , Q ) = max E ⊆ Ω ∣ P ( E ) − Q ( E ) ∣ = 1 2 ∑ ω ∈ Ω ∣ P ( ω ) − Q ( ω ) ∣ . d_{\mathrm{TV}}(P,Q)=\max_{E\subseteq\Omega}|P(E)-Q(E)|=\frac{1}{2}\sum_{\omega\in\Omega}|P(\omega)-Q(\omega)|. d TV ( P , Q ) = E ⊆ Ω max ∣ P ( E ) − Q ( E ) ∣ = 2 1 ω ∈ Ω ∑ ∣ P ( ω ) − Q ( ω ) ∣.
and it satisfies the triangle inequality. If Q ( ω ) > 0 Q(\omega)>0 Q ( ω ) > 0 whenever P ( ω ) > 0 P(\omega)>0 P ( ω ) > 0 , their relative entropy is
KL ( P ∥ Q ) = ∑ ω : P ( ω ) > 0 P ( ω ) log P ( ω ) Q ( ω ) . \operatorname{KL}(P\mathbin{\|}Q)=\sum_{\omega:P(\omega)>0}P(\omega)\log\frac{P(\omega)}{Q(\omega)}. KL ( P ∥ Q ) = ω : P ( ω ) > 0 ∑ P ( ω ) log Q ( ω ) P ( ω ) .
For x ∈ [ 0 , 1 ] x\in[0,1] x ∈ [ 0 , 1 ] , Ber ( x ) \operatorname{Ber}(x) Ber ( x ) is the distribution on { 0 , 1 } \{0,1\} { 0 , 1 } that gives 1 1 1 probability x x x . For random objects T T T and Z Z Z , we write L ( T ) \mathcal{L}(T) L ( T ) for the distribution of T T T , and L ( T ∣ Z = z ) \mathcal{L}(T\mid Z=z) L ( T ∣ Z = z ) for its conditional distribution given Z = z Z=z Z = z .
Lemma 2.2 (Standard facts). Let P P P and Q Q Q be distributions on a finite set.
(a) Coupling. If random objects X X X and Y Y Y on one probability space have distributions P P P and Q Q Q , then d T V ( P , Q ) ≤ P [ X ≠ Y ] d_{\mathrm{TV}}(P,Q)\le\mathbb{P}[X\ne Y] d TV ( P , Q ) ≤ P [ X = Y ] . Moreover, some such pair has P [ X ≠ Y ] = d T V ( P , Q ) \mathbb{P}[X\ne Y]=d_{\mathrm{TV}}(P,Q) P [ X = Y ] = d TV ( P , Q ) .
(b) Projection. If X ∼ P X\sim P X ∼ P and Y ∼ Q Y\sim Q Y ∼ Q , then d T V ( L ( ϕ ( X ) ) , L ( ϕ ( Y ) ) ) ≤ d T V ( P , Q ) d_{\mathrm{TV}}(\mathcal{L}(\phi(X)),\mathcal{L}(\phi(Y)))\le d_{\mathrm{TV}}(P,Q) d TV ( L ( ϕ ( X )) , L ( ϕ ( Y ))) ≤ d TV ( P , Q ) for every map ϕ \phi ϕ .
(c) Mixtures. If random objects Z Z Z and Z ′ Z' Z ′ have the same distribution, then for all random objects T T T and T ′ T' T ′ ,
d T V ( L ( T ) , L ( T ′ ) ) ≤ ∑ z P [ Z = z ] d T V ( L ( T ∣ Z = z ) , L ( T ′ ∣ Z ′ = z ) ) ≤ max z d T V ( L ( T ∣ Z = z ) , L ( T ′ ∣ Z ′ = z ) ) , \begin{aligned}
d_{\mathrm{TV}}(\mathcal{L}(T),\mathcal{L}(T')) &\le\sum_z\mathbb{P}[Z=z]d_{\mathrm{TV}}(\mathcal{L}(T\mid Z=z),\mathcal{L}(T'\mid Z'=z))\\
&\le\max_z d_{\mathrm{TV}}(\mathcal{L}(T\mid Z=z),\mathcal{L}(T'\mid Z'=z)),
\end{aligned} d TV ( L ( T ) , L ( T ′ )) ≤ z ∑ P [ Z = z ] d TV ( L ( T ∣ Z = z ) , L ( T ′ ∣ Z ′ = z )) ≤ z max d TV ( L ( T ∣ Z = z ) , L ( T ′ ∣ Z ′ = z )) , the sum and the maximum over the values z z z with P [ Z = z ] > 0 \mathbb{P}[Z=z]>0 P [ Z = z ] > 0 .
(d) Chain rule. Let P P P and Q Q Q be distributions of sequences z = ( z 1 , … , z q ) z=(z_1,\ldots,z_q) z = ( z 1 , … , z q ) such that Q ( z ) > 0 Q(z)>0 Q ( z ) > 0 whenever P ( z ) > 0 P(z)>0 P ( z ) > 0 . For t ≤ q t\le q t ≤ q , let P ( ⋅ ∣ z < t ) P(\mathord\cdot\mid z_{<t}) P ( ⋅ ∣ z < t ) and Q ( ⋅ ∣ z < t ) Q(\mathord\cdot\mid z_{<t}) Q ( ⋅ ∣ z < t ) be the conditional distributions of the t t t th entry given that the first t − 1 t-1 t − 1 entries are z 1 , … , z t − 1 z_1,\ldots,z_{t-1} z 1 , … , z t − 1 . Then
KL ( P ∥ Q ) = ∑ t = 1 q E z ∼ P [ KL ( P ( ⋅ ∣ z < t ) ∥ Q ( ⋅ ∣ z < t ) ) ] . \operatorname{KL}(P\mathbin{\|}Q)=\sum_{t=1}^{q}\mathbb{E}_{z\sim P}\left[\operatorname{KL}\left(P(\mathord\cdot\mid z_{<t})\mathbin{\|}Q(\mathord\cdot\mid z_{<t})\right)\right]. KL ( P ∥ Q ) = t = 1 ∑ q E z ∼ P [ KL ( P ( ⋅ ∣ z < t ) ∥ Q ( ⋅ ∣ z < t ) ) ] . (e) Pinsker’s inequality. d T V ( P , Q ) ≤ KL ( P ∥ Q ) / 2 d_{\mathrm{TV}}(P,Q)\le\sqrt{\operatorname{KL}(P\mathbin{\|}Q)/2} d TV ( P , Q ) ≤ KL ( P ∥ Q ) /2 .
(f) Bernoulli bound. For x ∈ [ 0 , 1 ] x \in[0,1] x ∈ [ 0 , 1 ] and y ∈ ( 0 , 1 ) y \in(0,1) y ∈ ( 0 , 1 ) , KL ( Ber ( x ) ∥ Ber ( y ) ) ≤ ( x − y ) 2 y ( 1 − y ) \operatorname{KL}(\operatorname{Ber}(x)\mathbin\Vert\operatorname{Ber}(y)) \le\frac{(x-y)^2}{y(1-y)} KL ( Ber ( x ) ∥ Ber ( y )) ≤ y ( 1 − y ) ( x − y ) 2 .
Proof. (a) For every event E E E , P ( E ) − Q ( E ) = P [ X ∈ E ] − P [ Y ∈ E ] ≤ P [ X ∈ E , Y ∉ E ] ≤ P [ X ≠ Y ] P(E)-Q(E)=\mathbb{P}[X\in E]-\mathbb{P}[Y\in E]\le\mathbb{P}[X\in E,Y\notin E]\le\mathbb{P}[X\ne Y] P ( E ) − Q ( E ) = P [ X ∈ E ] − P [ Y ∈ E ] ≤ P [ X ∈ E , Y ∈ / E ] ≤ P [ X = Y ] . For the second part, let r ( ω ) = min { P ( ω ) , Q ( ω ) } r(\omega)=\min\{P(\omega),Q(\omega)\} r ( ω ) = min { P ( ω ) , Q ( ω )} and ρ = ∑ ω r ( ω ) = 1 − d T V ( P , Q ) \rho=\sum_{\omega}r(\omega)=1-d_{\mathrm{TV}}(P,Q) ρ = ∑ ω r ( ω ) = 1 − d TV ( P , Q ) . With probability ρ \rho ρ , draw X = Y X=Y X = Y from the distribution r / ρ r/\rho r / ρ ; otherwise, draw X X X from ( P − r ) / ( 1 − ρ ) (P-r)/(1-\rho) ( P − r ) / ( 1 − ρ ) and Y Y Y from ( Q − r ) / ( 1 − ρ ) (Q-r)/(1-\rho) ( Q − r ) / ( 1 − ρ ) . The last two distributions have disjoint supports, so X ≠ Y X\ne Y X = Y exactly in the second case.
(b) Every event about ϕ ( X ) \phi(X) ϕ ( X ) is an event about X X X .
(c) For every event E E E ,
P [ T ∈ E ] − P [ T ′ ∈ E ] = ∑ z P [ Z = z ] ( P [ T ∈ E ∣ Z = z ] − P [ T ′ ∈ E ∣ Z ′ = z ] ) \mathbb{P}[T\in E]-\mathbb{P}[T'\in E]=\sum_z \mathbb{P}[Z=z]\left(\mathbb{P}[T\in E\mid Z=z]-\mathbb{P}[T'\in E\mid Z'=z]\right) P [ T ∈ E ] − P [ T ′ ∈ E ] = z ∑ P [ Z = z ] ( P [ T ∈ E ∣ Z = z ] − P [ T ′ ∈ E ∣ Z ′ = z ] )
and each bracket is at most d T V ( L ( T ∣ Z = z ) , L ( T ′ ∣ Z ′ = z ) ) d_{\mathrm{TV}}(\mathcal{L}(T\mid Z=z),\mathcal{L}(T'\mid Z'=z)) d TV ( L ( T ∣ Z = z ) , L ( T ′ ∣ Z ′ = z )) .
(d) The hypothesis gives Q ( z 1 , … , z t ) > 0 Q(z_1,\ldots,z_t)>0 Q ( z 1 , … , z t ) > 0 whenever P ( z 1 , … , z t ) > 0 P(z_1,\ldots,z_t)>0 P ( z 1 , … , z t ) > 0 , so all the conditional distributions are defined. Write P ( z ) P(z) P ( z ) and Q ( z ) Q(z) Q ( z ) as products of the conditional probabilities of the entries, take logarithms, and in the t t t th term sum out z t + 1 , … , z q z_{t+1},\ldots,z_q z t + 1 , … , z q .
(e) See [20 Lemma 11.6.1] .
(f) By log z ≤ z − 1 \log z\le z-1 log z ≤ z − 1 ,
KL ( Ber ( x ) ∥ Ber ( y ) ) ≤ x ( x y − 1 ) + ( 1 − x ) ( 1 − x 1 − y − 1 ) = x 2 y + ( 1 − x ) 2 1 − y − 1 = ( x − y ) 2 y ( 1 − y ) . \operatorname{KL}(\operatorname{Ber}(x)\mathbin\Vert\operatorname{Ber}(y))\le x\left(\frac{x}{y}-1\right)+(1-x)\left(\frac{1-x}{1-y}-1\right)=\frac{x^2}{y}+\frac{(1-x)^2}{1-y}-1=\frac{(x-y)^2}{y(1-y)}. KL ( Ber ( x ) ∥ Ber ( y )) ≤ x ( y x − 1 ) + ( 1 − x ) ( 1 − y 1 − x − 1 ) = y x 2 + 1 − y ( 1 − x ) 2 − 1 = y ( 1 − y ) ( x − y ) 2 .
Proposition 2.3 (Le Cam [12 ] ). If a fair coin decides whether an observation is drawn from P P P or from Q Q Q , every rule guessing the coin from the observation is correct with probability at most ( 1 + d T V ( P , Q ) ) / 2 (1+d_{\mathrm{TV}}(P,Q))/2 ( 1 + d TV ( P , Q )) /2 . This holds for transcripts of adaptive algorithms.
We use Pinsker’s inequality with the chain rule for relative entropy, and the bound
d T V ( P , Q ) ≤ KL ( P ∥ Q ) 2 , KL ( Ber ( x ) ∥ Ber ( y ) ) ≤ ( x − y ) 2 y ( 1 − y ) ( 0 < y < 1 ) , d_{\mathrm{TV}}(P,Q)\le\sqrt{\frac{\operatorname{KL}(P\mathbin\Vert Q)}{2}},\qquad\operatorname{KL}(\operatorname{Ber}(x)\mathbin\Vert\operatorname{Ber}(y))\le\frac{(x-y)^2}{y(1-y)}\quad(0<y<1), d TV ( P , Q ) ≤ 2 KL ( P ∥ Q ) , KL ( Ber ( x ) ∥ Ber ( y )) ≤ y ( 1 − y ) ( x − y ) 2 ( 0 < y < 1 ) ,
which follows from log z ≤ z − 1 \log z\le z-1 log z ≤ z − 1 . We use Chernoff, Hoeffding, and bounded-differences inequalities in their standard exponential forms; in particular a sum X X X of independent Bernoulli variables with mean μ \mu μ satisfies E e X ≤ e ( e − 1 ) μ \mathbb{E}e^X\le e^{(e-1)\mu} E e X ≤ e ( e − 1 ) μ . Finally, a nonnegative supermartingale ( M j ) (M_j) ( M j ) with M 0 = 1 M_0=1 M 0 = 1 satisfies P [ sup j M j ≥ 1 / β ] ≤ β \mathbb{P}[\sup_j M_j\ge1/\beta]\le\beta P [ sup j M j ≥ 1/ β ] ≤ β .
The Structural Theorem
Throughout § 3 and § 4 we fix the constants
e 0 = 10 − 4 , χ = 2 − e 0 , p = 2 χ − 2 , b 0 = p − 1 , ω = 101 100 , α 0 = δ 0 = 1 100 , θ 0 = 1 20 , σ 0 = 1 1000 . e_0=10^{-4},\qquad\chi=2-e_0,\qquad p=2\chi-2,\qquad b_0=p-1,\qquad\omega=\frac{101}{100},\qquad\alpha_0=\delta_0=\frac{1}{100},\qquad\theta_0=\frac{1}{20},\qquad\sigma_0=\frac{1}{1000}. e 0 = 1 0 − 4 , χ = 2 − e 0 , p = 2 χ − 2 , b 0 = p − 1 , ω = 100 101 , α 0 = δ 0 = 100 1 , θ 0 = 20 1 , σ 0 = 1000 1 .
Thus b 0 = 1 − 2 e 0 b_0=1-2e_0 b 0 = 1 − 2 e 0 , p = 2 − 2 e 0 p=2-2e_0 p = 2 − 2 e 0 , and b 0 < 1 < p < χ < 2 b_0<1<p<\chi<2 b 0 < 1 < p < χ < 2 .
Savings. Consider a finite universe U U U , a list of rows (subsets of U U U ), and a fixed set of rows declared eligible . The saving of this system is
max F ( ∣ ⋃ R ∈ F R ∣ − ∣ F ∣ ) , \max_{\mathcal{F}}\left(\left|\bigcup_{R\in\mathcal{F}}R\right|-|\mathcal{F}|\right), F max ( R ∈ F ⋃ R − ∣ F ∣ ) ,
the maximum being over sublists F \mathcal{F} F of eligible rows, including the empty one. Equivalently, it is ∣ U ∣ |U| ∣ U ∣ minus the least number of eligible rows and singletons covering U U U . Removing elements from U U U while keeping the eligibility of every row fixed decreases the saving by at most the number of removed elements.
Ideal statistics. Fix a ∗ > 0 a_* > 0 a ∗ > 0 and u ∈ [ 0 , 1 ) u \in[0,1) u ∈ [ 0 , 1 ) and put a i = a ∗ ω i + u a_i = a_*\omega^{i+u} a i = a ∗ ω i + u for i ∈ Z i \in\mathbb{Z} i ∈ Z . Let Q i = comp ( G a i ( T ) ) \mathcal{Q}_i = \operatorname{comp}(G_{a_i}(T)) Q i = comp ( G a i ( T )) be the exact components at scale i i i . The row of v ∈ V v \in V v ∈ V at scale i i i consists of the components C ∈ Q i C \in\mathcal{Q}_i C ∈ Q i containing a terminal within distance 3 a i / 5 3a_i/5 3 a i /5 of v v v , and it is eligible if it contains at least three components. Let Y i Y_i Y i be the saving of Q i \mathcal{Q}_i Q i with the rows of all points of V V V . For C ∈ Q i C \in\mathcal{Q}_i C ∈ Q i let
A i ( C ) = max Y ⊆ C , 2 ≤ ∣ Y ∣ ≤ 4 ( B T ( Y ) − c ( Y ) ) + , A_i(C) = \max_{\substack{Y \subseteq C,\\ 2 \le|Y| \le4}} \bigl(B_T(Y)-c(Y)\bigr)_+, A i ( C ) = Y ⊆ C , 2 ≤ ∣ Y ∣ ≤ 4 max ( B T ( Y ) − c ( Y ) ) + ,
and define the two scale-weighted statistics
Γ = ∑ i a i Y i , Λ = ∑ i a i ∣ { C ∈ Q i : A i ( C ) ≥ σ 0 a i } ∣ . (3) \Gamma= \sum_i a_iY_i,\qquad\Lambda= \sum_i a_i\left|\left\{C \in\mathcal{Q}_i : A_i(C) \ge\sigma_0a_i\right\}\right|. \tag*{(3)} Γ = i ∑ a i Y i , Λ = i ∑ a i ∣ { C ∈ Q i : A i ( C ) ≥ σ 0 a i } ∣ . (3)
Theorem 3.1 (Structural theorem). For every a ∗ > 0 a_* > 0 a ∗ > 0 and u ∈ [ 0 , 1 ) u \in[0,1) u ∈ [ 0 , 1 ) ,
L ≤ ( 2 − 89 ϵ 0 90 ) OPT + 1001 4000 Γ + ϵ 0 Λ . (4) L \le\left(2-\frac{89\epsilon_0}{90}\right)\operatorname{OPT}+\frac{1001}{4000}\Gamma+\epsilon_0\Lambda. \tag*{(4)} L ≤ ( 2 − 90 89 ϵ 0 ) OPT + 4000 1001 Γ + ϵ 0 Λ. (4) In particular,
Γ ≤ ϵ 0 L / 100 and Λ ≤ L / 10 ⟹ L ≤ ( 2 − ϵ 0 / 2 ) OPT . (5) \Gamma\le\epsilon_0L/100\quad\text{and}\quad\Lambda\le L/10\qquad\Longrightarrow\qquad L \le(2-\epsilon_0/2)\operatorname{OPT}. \tag*{(5)} Γ ≤ ϵ 0 L /100 and Λ ≤ L /10 ⟹ L ≤ ( 2 − ϵ 0 /2 ) OPT . (5) Both statements remain true if the sums in (3) are restricted to the indices i i i with d min / 2 < a i < 2 Δ T d_{\min}/2 < a_i < 2\Delta_T d m i n /2 < a i < 2 Δ T , where d min d_{\min} d m i n and Δ T \Delta_T Δ T are the smallest and the largest distance between two distinct terminals.
The second statement follows from the first and L ≤ 2 OPT L \le2\operatorname{OPT} L ≤ 2 OPT , since 89 90 − 1001 200000 − 1 5 > 1 2 \frac{89}{90}-\frac{1001}{200000}-\frac{1}{5} > \frac{1}{2} 90 89 − 200000 1001 − 5 1 > 2 1 . To prove the first we transform an optimal Steiner tree into a spanning tree on T T T by local steps, which are illustrated in Figure 3 . The process is a tool of the analysis and is not executed by any algorithm.
Figure 3. Steps in the proof of Theorem 3.1; squares are terminals and circles are nonterminals. (a) A cherry attached at v v v , with trunk z z z , arms s ≤ s ′ s \le s' s ≤ s ′ , primary port x x x , length r = z + s r = z+s r = z + s , and cost r + s ′ r+s' r + s ′ . (b) Promoting a fringe: in every cherry the longer arm (dashed) is replaced by the sibling edge (green), and the cherry center is then suppressed. (c) A short-core star: three limbs attached to a nonterminal core of weight at most θ 0 \theta_0 θ 0 times the shortest limb length; the shortest limb is kept, and the other two (dashed) are replaced by terminal edges to its primary port.
Reducing an optimal tree
Let K K K be an optimal Steiner tree. Cleanup repeatedly deletes nonterminal leaves and suppresses nonterminals of degree two, replacing a path x y z x\,y\,z x y z through such a y y y by the edge x z xz x z ; by the triangle inequality it does not increase the weight, and afterwards every nonterminal has degree at least three. Let E T ( K ) E_T(K) E T ( K ) be the edges of K K K joining two terminals and E S ( K ) E_S(K) E S ( K ) the remaining edges. Edges of E T ( K ) E_T(K) E T ( K ) are never removed, and the components of ( T , E T ( K ) ) (T,E_T(K)) ( T , E T ( K )) are called permanent groups ; they only merge during the process. We track the potential
V ( K ) = w ( E T ( K ) ) + χ w ( E S ( K ) ) , \mathcal{V}(K) = w(E_T(K)) + \chi w(E_S(K)), V ( K ) = w ( E T ( K )) + χ w ( E S ( K )) ,
which is initially at most χ O P T \chi\mathrm{OPT} χ OPT ; cleanup does not increase it.
Limbs. Let v v v be a nonterminal. A direct limb at v v v is a tree edge v x vx v x with x ∈ T x \in T x ∈ T ; its length and cost are r = W = w ( v , x ) r = W = w(v,x) r = W = w ( v , x ) , and x x x is its primary port. A cherry at v v v is a nonterminal y y y of degree three adjacent to v v v and to two terminals x , x ′ x,x' x , x ′ , where s = w ( y , x ) ≤ s ′ = w ( y , x ′ ) s = w(y,x) \le s' = w(y,x') s = w ( y , x ) ≤ s ′ = w ( y , x ′ ) ; its trunk is z = w ( v , y ) z = w(v,y) z = w ( v , y ) , its length r = z + s r = z+s r = z + s , its cost W = r + s ′ W = r+s' W = r + s ′ , its primary port x x x , and its ports x x x and x ′ x' x ′ . A limb is attached at v v v , and its region is the vertex set of the component of K K K minus the limb’s edge at v v v that does not contain v v v . A fan is a nonterminal v v v of degree three whose neighbors are a nonterminal v ′ v' v ′ , called its parent, and two limbs attached at v v v ; its region is the component of K − v v ′ K-vv' K − v v ′ containing v v v .
Good reconnections. The following steps are performed whenever the added terminal edges weigh at most χ \chi χ times the removed edges; each is followed by cleanup.
(G1) One edge: remove e ∈ E S ( K ) e \in E_S(K) e ∈ E S ( K ) and add a terminal edge joining the two components of K − e K-e K − e .
(G2) Cherry: remove the three edges at a cherry center y y y , add x x ′ xx' x x ′ , and add a terminal edge from the region of the cherry to its complement.
(G3) Fan: remove a fan v v v together with the edges at v v v and at its cherry centers; add the sibling edges of its cherries and the edge between its two primary ports, of total weight I I I , and one terminal edge from the region of the fan to its complement.
Each step keeps a tree, does not increase V \mathcal{V} V (removed edges have coefficient χ \chi χ , added ones coefficient 1 1 1 ), and strictly decreases ∣ E S ( K ) ∣ \lvert E_S(K)\rvert ∣ E S ( K )∣ .
Lemma 3.2 (Cut barriers). Suppose that no good reconnection applies.
(i) If x , x ′ ∈ T x,x' \in T x , x ′ ∈ T and e ∈ E S ( K ) e \in E_S(K) e ∈ E S ( K ) lies on the tree path between them, then w ( x , x ′ ) > χ w ( e ) w(x,x') > \chi w(e) w ( x , x ′ ) > χ w ( e ) . Consequently no component of G a ( T ) ∪ E T ( K ) G_a(T)\cup E_T(K) G a ( T ) ∪ E T ( K ) has terminals on both sides of an edge e ∈ E S ( K ) e \in E_S(K) e ∈ E S ( K ) with χ w ( e ) ≥ a \chi w(e) \ge a χ w ( e ) ≥ a .
(ii) In every cherry s ′ < s / ( χ − 1 ) s' < s/(\chi-1) s ′ < s / ( χ − 1 ) , and every limb has W ≤ ( 1 + 1 χ − 1 ) r W \le\left(1+\frac{1}{\chi-1}\right)r W ≤ ( 1 + χ − 1 1 ) r .
(iii) Every terminal pair with exactly one terminal in the region of a limb of length r r r is at distance greater than χ z + ( χ − 1 ) ( s + s ′ ) ≥ p r \chi z+(\chi-1)(s+s') \ge pr χz + ( χ − 1 ) ( s + s ′ ) ≥ p r , where z = r z=r z = r and s = s ′ = 0 s=s'=0 s = s ′ = 0 for a direct limb. Hence, for a ≤ p r a \le pr a ≤ p r , no component of G a ( T ) ∪ E T ( K ) G_a(T)\cup E_T(K) G a ( T ) ∪ E T ( K ) meets both the region and its complement.
Proof. (i) Otherwise (G1) applies to e e e and x x ′ xx' x x ′ . A path of G a ( T ) ∪ E T ( K ) G_a(T)\cup E_T(K) G a ( T ) ∪ E T ( K ) between the two sides of e e e has a step crossing the cut of e e e ; it is not a tree edge, so it is a pair at distance less than a ≤ χ w ( e ) a \le\chi w(e) a ≤ χ w ( e ) . (ii) Apply (i) to x , x ′ x,x' x , x ′ and e = y z e=yz e = y z : s + s ′ ≥ w ( x , x ′ ) > χ s ′ s+s' \ge w(x,x') > \chi s' s + s ′ ≥ w ( x , x ′ ) > χ s ′ . Then W = r + s ′ ≤ r + s / ( χ − 1 ) W=r+s' \le r+s/(\chi-1) W = r + s ′ ≤ r + s / ( χ − 1 ) . (iii) For a direct limb this is (i) with e = v x e=vx e = v x . For a cherry, a crossing pair f f f with w ( f ) ≤ w ( x , x ′ ) w(f) \le w(x,x') w ( f ) ≤ w ( x , x ′ ) would allow (G2). Hence w ( f ) > χ ( z + s + s ′ ) − ( s + s ′ ) = χ z + ( χ − 1 ) ( s + s ′ ) ≥ 2 ( χ − 1 ) ( z + s ) = p r w(f) > \chi(z+s+s')-(s+s')=\chi z+(\chi-1)(s+s') \ge2(\chi-1)(z+s)=pr w ( f ) > χ ( z + s + s ′ ) − ( s + s ′ ) = χz + ( χ − 1 ) ( s + s ′ ) ≥ 2 ( χ − 1 ) ( z + s ) = p r , using χ ≤ 2 \chi\le2 χ ≤ 2 and s ≤ s ′ s \le s' s ≤ s ′ . Terminal edges of K K K do not cross tree cuts. □ \square □
Recorded events and their charges
When no good reconnection applies, the process performs one of the following events, preferring a star, and then cleans up and exhausts good reconnections again.
Short-core stars. A short-core star consists of three limbs with pairwise disjoint regions, each attached outside the regions of the other two, whose attachment points are joined by a subtree of nonterminals (the core ) of weight at most θ 0 ℓ \theta_0\ell θ 0 ℓ , where ℓ \ell ℓ is the smallest of the three lengths. The event keeps a limb of length ℓ \ell ℓ and deletes the edges of the other two limbs; for each deleted cherry it adds the sibling edge x x ′ xx' x x ′ , and for each deleted limb it adds an edge from its primary port to the kept primary port. The charge is the total cost of the two deleted limbs. The event records its center , the attachment point of the kept limb, and the index i i i with a i ≤ p ℓ < ω a i a_i \le p\ell< \omega a_i a i ≤ p ℓ < ω a i .
Lemma 3.3 (Stars). A star increases V \mathcal{V} V by at most ( 21 10 − χ ) \left(\frac{21}{10}-\chi\right) ( 10 21 − χ ) times its charge and permanently joins its three primary ports. The total charge W S W_S W S of all stars satisfies W S ≤ 5 2 Γ W_S \le\frac{5}{2}\Gamma W S ≤ 2 5 Γ .
Proof. Let x 1 , x 2 , x 3 x_1,x_2,x_3 x 1 , x 2 , x 3 be the primary ports, r 1 , r 2 , r 3 r_1,r_2,r_3 r 1 , r 2 , r 3 the lengths, and h h h the kept limb, so r h = ℓ r_h=\ell r h = ℓ . For j ≠ h j\ne h j = h , Lemma 3.2 (iii) and the tree path give p r j < w ( x j , x h ) ≤ r j + θ 0 ℓ + ℓ pr_j<w(x_j,x_h)\le r_j+\theta_0\ell+\ell p r j < w ( x j , x h ) ≤ r j + θ 0 ℓ + ℓ , so r j < ( 1 + θ 0 ) ℓ / b 0 r_j<(1+\theta_0)\ell/b_0 r j < ( 1 + θ 0 ) ℓ / b 0 . Every primary port is within r j + θ 0 ℓ r_j+\theta_0\ell r j + θ 0 ℓ of the recorded center v h v_h v h , and since a i > p ℓ / ω a_i>p\ell/\omega a i > p ℓ / ω ,
w ( v h , x j ) a i < ω p ( 1 + θ 0 b 0 + θ 0 ) < 3 5 . \frac{w(v_h,x_j)}{a_i}<\frac{\omega}{p}\left(\frac{1+\theta_0}{b_0}+\theta_0\right)<\frac{3}{5}. a i w ( v h , x j ) < p ω ( b 0 1 + θ 0 + θ 0 ) < 5 3 .
As a i ≤ p ℓ ≤ p r j a_i\le p\ell\le pr_j a i ≤ p ℓ ≤ p r j for all j j j , Lemma 3.2 (iii) places the three primary ports in different components of G a i ( T ) ∪ E T ( K ) G_{a_i}(T)\cup E_T(K) G a i ( T ) ∪ E T ( K ) , hence in different exact components, and the row of v h v_h v h is eligible.
Fix i i i and consider the bipartite incidence graph Ψ \Psi Ψ between the stars recorded at scale i i i and the exact components Q i Q_i Q i , each star being adjacent to the components of its primary ports. When a star is recorded, all earlier stars have their ports joined by permanent edges, so every connected component of Ψ \Psi Ψ lies inside one component of G a i ( T ) ∪ E T ( K ) G_{a_i}(T)\cup E_T(K) G a i ( T ) ∪ E T ( K ) , and the new star meets three different ones. Hence Ψ \Psi Ψ is a forest. If it has f f f stars and c c c connected components, it touches u = 2 f + c u=2f+c u = 2 f + c exact components; the at most f f f rows of the recorded centers are eligible and cover them, so Y i ≥ u − f ≥ f Y_i\ge u-f\ge f Y i ≥ u − f ≥ f .
By Lemma 3.2 (ii) a deleted limb costs at most ( 1 + 1 χ − 1 ) ( 1 + θ 0 ) ℓ / b 0 \left(1+\frac{1}{\chi-1}\right)(1+\theta_0)\ell/b_0 ( 1 + χ − 1 1 ) ( 1 + θ 0 ) ℓ / b 0 , so the charge is below 4.21 ℓ < 5 2 a i 4.21\ell<\frac{5}{2}a_i 4.21 ℓ < 2 5 a i , and W S ≤ ∑ i 5 2 a i Y i = 5 2 Γ W_S\le\sum_i\frac{5}{2}a_iY_i=\frac{5}{2}\Gamma W S ≤ ∑ i 2 5 a i Y i = 2 5 Γ . For a deleted limb j j j the added edges weigh at most ( s j + s j ′ ) + ( r j + θ 0 ℓ + ℓ ) ≤ 2 W j + θ 0 ℓ (s_j+s'_j)+(r_j+\theta_0\ell+\ell)\le2W_j+\theta_0\ell ( s j + s j ′ ) + ( r j + θ 0 ℓ + ℓ ) ≤ 2 W j + θ 0 ℓ , using ℓ ≤ r j \ell\le r_j ℓ ≤ r j and s j ≤ s j ′ s_j\le s'_j s j ≤ s j ′ (for a direct limb s j = s j ′ = 0 s_j=s'_j=0 s j = s j ′ = 0 ). Since the charge is at least 2 ℓ 2\ell 2 ℓ , the added weight is at most ( 2 + θ 0 ) (2+\theta_0) ( 2 + θ 0 ) times the charge, and the removed edges have coefficient χ \chi χ . The added edges keep the tree connected and join the ports. □
Promotions. Root K K K at a terminal. A fan is a fringe if its region contains no nonterminal other than the fan and its cherry centers (terminal subtrees below its ports may be arbitrary). Let its limbs have lengths r 1 , r 2 r_1,r_2 r 1 , r 2 , put R f = r 1 + r 2 R_f=r_1+r_2 R f = r 1 + r 2 , and let z z z be the weight of its parent edge. If it has at least one cherry, its promotion replaces in each cherry the longer arm y x ′ yx' y x ′ by the sibling edge x x ′ xx' x x ′ and suppresses y y y . The charge is the sum of the longer arms s j ′ s'_j s j ′ of its cherries. The promotion is small if s j < α 0 r j s_j<\alpha_0r_j s j < α 0 r j for every cherry, and regular if it is not small and z ≥ δ 0 R f z\ge\delta_0R_f z ≥ δ 0 R f . Other fringes are not promoted.
Lemma 3.4 (Small promotions). A promotion increases V \mathcal{V} V by at most ( 2 − χ ) (2-\chi) ( 2 − χ ) times its charge. The total charge W B W_B W B of small promotions satisfies W B ≤ α 0 ( 1 − α 0 ) ( χ − 1 ) OPT < OPT / 90 W_B\le\frac{\alpha_0}{(1-\alpha_0)(\chi-1)}\operatorname{OPT}<\operatorname{OPT}/90 W B ≤ ( 1 − α 0 ) ( χ − 1 ) α 0 OPT < OPT /90 .
Proof. The new edge costs at most s + s ′ ≤ 2 s ′ s+s'\le2s' s + s ′ ≤ 2 s ′ . If s j < α 0 ( z j + s j ) s_j<\alpha_0(z_j+s_j) s j < α 0 ( z j + s j ) then s j < α 0 z j / ( 1 − α 0 ) s_j<\alpha_0z_j/(1-\alpha_0) s j < α 0 z j / ( 1 − α 0 ) , and Lemma 3.2 (ii) gives s j ′ < α 0 z j / ( ( 1 − α 0 ) ( χ − 1 ) ) s'_j<\alpha_0z_j/((1-\alpha_0)(\chi-1)) s j ′ < α 0 z j / (( 1 − α 0 ) ( χ − 1 )) . A small promotion destroys the nonterminal–nonterminal edges y y ′ yy' y y ′ of its cherries. The total weight of nonterminal–nonterminal edges never increases: such edges arise only by suppressing a nonterminal between two nonterminals, which replaces two of them by a lighter one. It is initially at most OPT. □
Lemma 3.5 (Regular promotions). Every regular promotion can be assigned a pair ( i , C ) (i,C) ( i , C ) with C ∈ Q i C \in Q_i C ∈ Q i and A i ( C ) ≥ σ 0 a i A_i(C) \ge\sigma_0a_i A i ( C ) ≥ σ 0 a i , such that its charge is less than a i a_i a i and no terminal set C C C is assigned twice. Hence the total charge W R W_R W R of regular promotions satisfies W R ≤ Λ W_R \le\Lambda W R ≤ Λ .
Proof. By Lemma 3.2 (iii) applied to each limb and the other primary port, b 0 r 1 < r 2 b_0r_1 < r_2 b 0 r 1 < r 2 and b 0 r 2 < r 1 b_0r_2 < r_1 b 0 r 2 < r 1 ; with ℓ = min { r 1 , r 2 } \ell= \min\{r_1,r_2\} ℓ = min { r 1 , r 2 } this gives R f < 2 ℓ / b 0 R_f < 2\ell/b_0 R f < 2 ℓ / b 0 . Let a = a i a = a_i a = a i be the least grid scale above R f / ( χ − 1 ) R_f/(\chi-1) R f / ( χ − 1 ) , so a ≤ ω R f / ( χ − 1 ) < 2.03 ℓ a \le\omega R_f/(\chi-1) < 2.03\ell a ≤ ω R f / ( χ − 1 ) < 2.03 ℓ .
All ports lie in one exact component C C C of G a ( T ) G_a(T) G a ( T ) . Ports of different limbs are at distance at most R f + ∑ j ( s j ′ − s j ) < R f / ( χ − 1 ) R_f+\sum_j(s'_j-s_j)<R_f/(\chi-1) R f + ∑ j ( s j ′ − s j ) < R f / ( χ − 1 ) , because s j ′ − s j < ( 1 χ − 1 − 1 ) s j s'_j-s_j<(\frac{1}{\chi-1}-1)s_j s j ′ − s j < ( χ − 1 1 − 1 ) s j . For a cherry j j j , Lemma 3.2 (iii) applied to x 3 − j x_{3-j} x 3 − j gives χ z j + ( χ − 1 ) ( s j + s j ′ ) < w ( x j , x 3 − j ) ≤ R f \chi z_j+(\chi-1)(s_j+s'_j)<w(x_j,x_{3-j})\le R_f χ z j + ( χ − 1 ) ( s j + s j ′ ) < w ( x j , x 3 − j ) ≤ R f , so w ( x j , x j ′ ) ≤ s j + s j ′ < R f / ( χ − 1 ) w(x_j,x'_j)\le s_j+s'_j<R_f/(\chi-1) w ( x j , x j ′ ) ≤ s j + s j ′ < R f / ( χ − 1 ) .
C C C lies in the region of the fringe. As (G3) does not apply, every terminal pair leaving the region is at distance greater than χ ( z + W 1 + W 2 ) − I \chi(z+W_1+W_2)-I χ ( z + W 1 + W 2 ) − I , where I ≤ R f + ∑ j ( s j + s j ′ ) I\le R_f+\sum_j(s_j+s'_j) I ≤ R f + ∑ j ( s j + s j ′ ) . Since ( χ − 1 ) s j ′ − s j ≥ − ( 2 − χ ) r j (\chi-1)s'_j-s_j\ge-(2-\chi)r_j ( χ − 1 ) s j ′ − s j ≥ − ( 2 − χ ) r j , this is at least χ z + ( 2 χ − 3 ) R f ≥ ( χ δ 0 + b 0 ) R f > ω R f / ( χ − 1 ) ≥ a \chi z+(2\chi-3)R_f\ge(\chi\delta_0+b_0)R_f>\omega R_f/(\chi-1)\ge a χz + ( 2 χ − 3 ) R f ≥ ( χ δ 0 + b 0 ) R f > ω R f / ( χ − 1 ) ≥ a .
Local evidence. Some cherry j j j has s j ≥ α 0 r j s_j\ge\alpha_0r_j s j ≥ α 0 r j ; let Y = { x j , x j ′ , x 3 − j } ⊆ C Y=\{x_j,x'_j,x_{3-j}\}\subseteq C Y = { x j , x j ′ , x 3 − j } ⊆ C . Treat the permanent edges as earlier zero links. For t ≤ p ℓ t\le p\ell t ≤ p ℓ , Lemma 3.2 (iii) separates x 3 − j x_{3-j} x 3 − j from the other two, and for t ≤ χ s j t\le\chi s_j t ≤ χ s j , Lemma 3.2 (i) for the edge y x j yx_j y x j separates x j x_j x j from x j ′ x'_j x j ′ . By Lemma 2.1 , B T ( Y ) ≥ p ℓ + min { χ s j , p ℓ } B_T(Y)\ge p\ell+\min\{\chi s_j,p\ell\} B T ( Y ) ≥ p ℓ + min { χ s j , p ℓ } . Connecting Y Y Y through y y y costs at most s j + s j ′ + z j + r 3 − j = R f + s j ′ s_j+s'_j+z_j+r_{3-j}=R_f+s'_j s j + s j ′ + z j + r 3 − j = R f + s j ′ . Using s j ′ < s j / ( χ − 1 ) s'_j<s_j/(\chi-1) s j ′ < s j / ( χ − 1 ) , s j < ℓ / b 0 s_j<\ell/b_0 s j < ℓ / b 0 , and R f < 2 ℓ / b 0 R_f<2\ell/b_0 R f < 2 ℓ / b 0 in both cases of the minimum,
B T ( Y ) − c ( Y ) ≥ ( χ − 1 χ − 1 ) s j − ( 2 b 0 − p + χ b 0 − p ) ℓ > 0.9997 α 0 ℓ − 0.0012 ℓ > 0.008 ℓ > σ 0 a . B_T(Y)-c(Y)\ge\left(\chi-\frac{1}{\chi-1}\right)s_j-\left(\frac{2}{b_0}-p+\frac{\chi}{b_0}-p\right)\ell>0.9997\alpha_0\ell-0.0012\ell>0.008\ell>\sigma_0a. B T ( Y ) − c ( Y ) ≥ ( χ − χ − 1 1 ) s j − ( b 0 2 − p + b 0 χ − p ) ℓ > 0.9997 α 0 ℓ − 0.0012 ℓ > 0.008 ℓ > σ 0 a . Thus A i ( C ) > σ 0 a i A_i(C)>\sigma_0a_i A i ( C ) > σ 0 a i , and the charge is ∑ j s j ′ < R f / ( χ − 1 ) < a i \sum_j s'_j<R_f/(\chi-1)<a_i ∑ j s j ′ < R f / ( χ − 1 ) < a i .
No repetition. Before the promotion the terminals of the region form at most four permanent groups, and afterwards at most two. Every later regular promotion has three ports in three different permanent groups (they are separated by the edges y x j yx_j y x j and by the limb regions), all in its assigned component. Since groups only merge, the terminal set C C C cannot be assigned again. Finally each promotion raises V \mathcal{V} V by at most ( 2 − χ ) (2-\chi) ( 2 − χ ) times its charge, as in Lemma 3.4 . ◊
The process terminates
Lemma 3.6. If E S ( K ) ≠ ∅ E_S(K)\ne\varnothing E S ( K ) = ∅ and no good reconnection applies , then a short-core star, a small promotion, or a regular promotion exists.
Proof. Contract every permanent group to a vertex and root the tree at a group. A core is a component of the subgraph induced by the nonterminals; choose a core K K K below which no other core lies. Every vertex of K K K has one parent edge, its other neighbors are its children, and its children outside K K K are groups whose subtrees contain no nonterminal; edges to them are direct limbs. The height of a vertex of K K K is the number of nonterminals on a longest downward path in K K K .
Three limbs attached at one vertex form a short-core star with a core of weight zero; this includes the parent edge if it leads to a group. Assume there is no star. Then a vertex of height one has exactly two group children and a nonterminal parent, so it is a cherry at its parent. A vertex of height two has exactly two children, both limbs and at least one a cherry, and a nonterminal parent: it is a fan, and by the choice of K K K a fringe. A root of height at most two would have a group parent, so K K K has a vertex of height three.
Suppose no fringe can be promoted, so each fringe has a non-small cherry and a parent edge shorter than δ 0 R f < d 0 ℓ \delta_0R_f<d_0\ell δ 0 R f < d 0 ℓ , where d 0 = 2 δ 0 / b 0 < 0.02001 d_0=2\delta_0/b_0<0.02001 d 0 = 2 δ 0 / b 0 < 0.02001 and ℓ \ell ℓ is its shortest limb length. Let v ′ v' v ′ have height three and a fringe child v v v with shortest limb length ℓ 1 \ell_1 ℓ 1 ; as v ′ v' v ′ has degree at least three, it has another child. If that child is a limb of length r r r , then Lemma 3.2 (iii) for the shortest limb of v v v gives r > ( b 0 − d 0 ) ℓ 1 r>(b_0-d_0)\ell_1 r > ( b 0 − d 0 ) ℓ 1 , and two limbs of v v v together with this limb form a star whose core, the edge v v ′ vv' v v ′ , @page 19/58 starts_mid=0 ends_mid=1
Lemma 3.5 (Regular promotions). Every regular promotion can be assigned a pair ( i , C ) (i,C) ( i , C ) with C ∈ Q i C \in Q_i C ∈ Q i and A i ( C ) ≥ σ 0 a i A_i(C) \ge\sigma_0a_i A i ( C ) ≥ σ 0 a i , such that its charge is less than a i a_i a i and no terminal set C C C is assigned twice. Hence the total charge W R W_R W R of regular promotions satisfies W R ≤ Λ W_R \le\Lambda W R ≤ Λ .
Proof. By Lemma 3.2 (iii) applied to each limb and the other primary port, b 0 r 1 < r 2 b_0r_1 < r_2 b 0 r 1 < r 2 and b 0 r 2 < r 1 b_0r_2 < r_1 b 0 r 2 < r 1 ; with ℓ = min { r 1 , r 2 } \ell= \min\{r_1,r_2\} ℓ = min { r 1 , r 2 } this gives R f < 2 ℓ / b 0 R_f < 2\ell/b_0 R f < 2 ℓ / b 0 . Let a = a i a = a_i a = a i be the least grid scale above R f / ( χ − 1 ) R_f/(\chi- 1) R f / ( χ − 1 ) , so a ≤ ω R f / ( χ − 1 ) < 2.03 ℓ a \le\omega R_f/(\chi- 1) < 2.03\ell a ≤ ω R f / ( χ − 1 ) < 2.03 ℓ .
All ports lie in one exact component C C C of G a ( T ) G_a(T) G a ( T ) . Ports of different limbs are at distance at most R f + ∑ j ( s j ′ − s j ) < R f / ( χ − 1 ) R_f+\sum_j(s'_j-s_j)<R_f/(\chi-1) R f + ∑ j ( s j ′ − s j ) < R f / ( χ − 1 ) , because s j ′ − s j < ( 1 χ − 1 − 1 ) s j s'_j-s_j<(\frac{1}{\chi-1}-1)s_j s j ′ − s j < ( χ − 1 1 − 1 ) s j . For a cherry j j j , Lemma 3.2 (iii) applied to x 3 − j x_{3-j} x 3 − j gives χ z j + ( χ − 1 ) ( s j + s j ′ ) < w ( x j , x 3 − j ) ≤ R f \chi z_j+(\chi-1)(s_j+s'_j)<w(x_j,x_{3-j})\le R_f χ z j + ( χ − 1 ) ( s j + s j ′ ) < w ( x j , x 3 − j ) ≤ R f , so w ( x j , x j ′ ) ≤ s j + s j ′ < R f / ( χ − 1 ) w(x_j,x'_j)\le s_j+s'_j<R_f/(\chi-1) w ( x j , x j ′ ) ≤ s j + s j ′ < R f / ( χ − 1 ) .
C C C lies in the region of the fringe. As (G3) does not apply, every terminal pair leaving the region is at distance greater than χ ( z + W 1 + W 2 ) − I \chi(z+W_1+W_2)-I χ ( z + W 1 + W 2 ) − I , where I ≤ R f + ∑ j ( s j + s j ′ ) I\le R_f+\sum_j(s_j+s'_j) I ≤ R f + ∑ j ( s j + s j ′ ) . Since ( χ − 1 ) s j ′ − s j ≥ − ( 2 − χ ) r j (\chi-1)s'_j-s_j\ge-(2-\chi)r_j ( χ − 1 ) s j ′ − s j ≥ − ( 2 − χ ) r j , this is at least χ z + ( 2 χ − 3 ) R f ≥ ( χ δ 0 + b 0 ) R f > ω R f / ( χ − 1 ) ≥ a \chi z+(2\chi-3)R_f\ge(\chi\delta_0+b_0)R_f>\omega R_f/(\chi-1)\ge a χz + ( 2 χ − 3 ) R f ≥ ( χ δ 0 + b 0 ) R f > ω R f / ( χ − 1 ) ≥ a .
Local evidence. Some cherry j j j has s j ≥ α 0 r j s_j\ge\alpha_0r_j s j ≥ α 0 r j ; let Y = { x j , x j ′ , x 3 − j } ⊆ C Y=\{x_j,x'_j,x_{3-j}\}\subseteq C Y = { x j , x j ′ , x 3 − j } ⊆ C . Treat the permanent edges as earlier zero links. For t ≤ p ℓ t\le p\ell t ≤ p ℓ , Lemma 3.2 (iii) separates x 3 − j x_{3-j} x 3 − j from the other two, and for t ≤ χ s j t\le\chi s_j t ≤ χ s j , Lemma 3.2 (i) for the edge y x j yx_j y x j separates x j x_j x j from x j ′ x'_j x j ′ . By Lemma 2.1 , B T ( Y ) ≥ p ℓ + min { χ s j , p ℓ } B_T(Y)\ge p\ell+\min\{\chi s_j,p\ell\} B T ( Y ) ≥ p ℓ + min { χ s j , p ℓ } . Connecting Y Y Y through y y y costs at most s j + s j ′ + z j + r 3 − j = R f + s j ′ s_j+s'_j+z_j+r_{3-j}=R_f+s'_j s j + s j ′ + z j + r 3 − j = R f + s j ′ . Using s j ′ < s j / ( χ − 1 ) s'_j<s_j/(\chi-1) s j ′ < s j / ( χ − 1 ) , s j < ℓ / b 0 s_j<\ell/b_0 s j < ℓ / b 0 , and R f < 2 ℓ / b 0 R_f<2\ell/b_0 R f < 2 ℓ / b 0 in both cases of the minimum,
B T ( Y ) − c ( Y ) ≥ ( χ − 1 χ − 1 ) s j − ( 2 b 0 − p + χ b 0 − p ) ℓ > 0.9997 α 0 ℓ − 0.0012 ℓ > 0.008 ℓ > σ 0 a . B_T(Y)-c(Y)\ge\left(\chi-\frac{1}{\chi-1}\right)s_j-\left(\frac{2}{b_0}-p+\frac{\chi}{b_0}-p\right)\ell>0.9997\alpha_0\ell-0.0012\ell>0.008\ell>\sigma_0a. B T ( Y ) − c ( Y ) ≥ ( χ − χ − 1 1 ) s j − ( b 0 2 − p + b 0 χ − p ) ℓ > 0.9997 α 0 ℓ − 0.0012 ℓ > 0.008 ℓ > σ 0 a . Thus A i ( C ) > σ 0 a i A_i(C)>\sigma_0a_i A i ( C ) > σ 0 a i , and the charge is ∑ j s j ′ < R f / ( χ − 1 ) < a i \sum_j s'_j<R_f/(\chi-1)<a_i ∑ j s j ′ < R f / ( χ − 1 ) < a i .
No repetition. Before the promotion the terminals of the region form at most four permanent groups, and afterwards at most two. Every later regular promotion has three ports in three different permanent groups (they are separated by the edges y x j yx_j y x j and by the limb regions), all in its assigned component. Since groups only merge, the terminal set C C C cannot be assigned again. Finally each promotion raises V \mathcal{V} V by at most ( 2 − χ ) (2-\chi) ( 2 − χ ) times its charge, as in Lemma 3.4 . □
The process terminates {#sec:3.3} Lemma 3.6. If E S ( K ) ≠ ∅ E_S(K)\ne\varnothing E S ( K ) = ∅ and no good reconnection applies , then a short-core star, a small promotion, or a regular promotion exists.
Proof. Contract every permanent group to a vertex and root the tree at a group. A core is a component of the subgraph induced by the nonterminals; choose a core K K K below which no other core lies. Every vertex of K K K has one parent edge, its other neighbors are its children, and its children outside K K K are groups whose subtrees contain no nonterminal; edges to them are direct limbs. The height of a vertex of K K K is the number of nonterminals on a longest downward path in K K K .
Three limbs attached at one vertex form a short-core star with a core of weight zero; this includes the parent edge if it leads to a group. Assume there is no star. Then a vertex of height one has exactly two group children and a nonterminal parent, so it is a cherry at its parent. A vertex of height two has exactly two children, both limbs and at least one a cherry, and a nonterminal parent: it is a fan, and by the choice of K K K a fringe. A root of height at most two would have a group parent, so K K K has a vertex of height three.
Suppose no fringe can be promoted, so each fringe has a non-small cherry and a parent edge shorter than δ 0 R f < d 0 ℓ \delta_0R_f<d_0\ell δ 0 R f < d 0 ℓ , where d 0 = 2 δ 0 / b 0 < 0.02001 d_0=2\delta_0/b_0<0.02001 d 0 = 2 δ 0 / b 0 < 0.02001 and ℓ \ell ℓ is its shortest limb length. Let v ′ v' v ′ have height three and a fringe child v v v with shortest limb length ℓ 1 \ell_1 ℓ 1 ; as v ′ v' v ′ has degree at least three, it has another child. If that child is a limb of length r r r , then Lemma 3.2 (iii) for the shortest limb of v v v gives r > ( b 0 − d 0 ) ℓ 1 r>(b_0-d_0)\ell_1 r > ( b 0 − d 0 ) ℓ 1 , and two limbs of v v v together with this limb form a star whose core, the edge v v ′ vv' v v ′ ,
weighs less than d 0 ℓ 1 < θ 0 min { r , ℓ 1 } d_0\ell_1 < \theta_0\min\{r,\ell_1\} d 0 ℓ 1 < θ 0 min { r , ℓ 1 } . Otherwise it is a fringe with shortest limb length ℓ 2 \ell_2 ℓ 2 ; we may assume ℓ 1 ≤ ℓ 2 \ell_1 \le\ell_2 ℓ 1 ≤ ℓ 2 . Then ( b 0 − d 0 ) ℓ 2 < ( 1 + d 0 ) ℓ 1 (b_0-d_0)\ell_2 < (1+d_0)\ell_1 ( b 0 − d 0 ) ℓ 2 < ( 1 + d 0 ) ℓ 1 , so ℓ 2 < 1.042 ℓ 1 \ell_2 < 1.042\ell_1 ℓ 2 < 1.042 ℓ 1 , the two parent edges weigh less than d 0 ( ℓ 1 + ℓ 2 ) < 0.041 ℓ 1 < θ 0 ℓ 1 d_0(\ell_1+\ell_2) < 0.041\ell_1 < \theta_0\ell_1 d 0 ( ℓ 1 + ℓ 2 ) < 0.041 ℓ 1 < θ 0 ℓ 1 , and two limbs of the first fringe with one limb of the second form a star.
Proof of Theorem 3.1. Every event strictly decreases ∣ E S ( K ) ∣ |E_S(K)| ∣ E S ( K ) ∣ , so by Lemma 3.6 the process ends with a spanning tree of T T T , whose weight is at least L L L and equals its potential. Good reconnections and cleanup do not increase V \mathcal{V} V , so by Lemmas 3.3 to 3.5,
L ≤ χ OPT + ( 21 10 − χ ) W S + ( 2 − χ ) ( W B + W R ) ≤ ( 2 − ϵ 0 ) OPT + 1001 4000 Γ + ϵ 0 ( OPT 90 + Λ ) , L \le\chi\operatorname{OPT}+\left(\frac{21}{10}-\chi\right)W_S+(2-\chi)(W_B+W_R) \le(2-\epsilon_0)\operatorname{OPT}+\frac{1001}{4000}\Gamma+\epsilon_0\left(\frac{\operatorname{OPT}}{90}+\Lambda\right), L ≤ χ OPT + ( 10 21 − χ ) W S + ( 2 − χ ) ( W B + W R ) ≤ ( 2 − ϵ 0 ) OPT + 4000 1001 Γ + ϵ 0 ( 90 OPT + Λ ) ,
which is (4) . Recorded star scales satisfy 5 6 d min ≤ a i ≤ Δ T \frac{5}{6}d_{\min}\le a_i\le\Delta_T 6 5 d m i n ≤ a i ≤ Δ T , because an eligible row contains two ports within 6 5 a i \frac{6}{5}a_i 5 6 a i of each other and the ports of a star are separated at scale a i a_i a i ; a regular scale exceeds d min d_{\min} d m i n , and its component is a proper subset of T T T , so it is below Δ T \Delta_T Δ T . Hence only indices in the stated range are used.
The O ~ ( n 3 / 2 ) \widetilde{O}(n^{3/2}) O ( n 3/2 ) Estimation Algorithm
We now prove Theorem 1.1 . After normalization (§ 4.1 ), the estimator computes at every scale a lower estimate of the saving on a random cover, and a lower estimate of the band statistic, and returns an upper estimate of L L L minus the larger of the two discounts of Lemmas 4.1 and 4.2 . Throughout, β \beta β denotes a failure parameter that is inverse-polylogarithmic in n n n , chosen in § 4.12 .
Normalization and sound statistics
Flooring. If k ≤ 1 k\le1 k ≤ 1 we return 0 0 0 , and if k = 2 k=2 k = 2 the terminal distance. Otherwise we query one terminal against T T T and let R m e t R_{\mathrm{met}} R met be the largest answer, so R m e t ≤ Δ T ≤ 2 R m e t R_{\mathrm{met}}\le\Delta_T\le2R_{\mathrm{met}} R met ≤ Δ T ≤ 2 R met and R m e t ≤ L 0 R_{\mathrm{met}}\le L_0 R met ≤ L 0 , where L 0 L_0 L 0 and OPT 0 \operatorname{OPT}_0 OPT 0 denote the original costs. For a target improvement η w ∈ ( 0 , 1 ) \eta_w\in(0,1) η w ∈ ( 0 , 1 ) put ζ = η w / 100 \zeta=\eta_w/100 ζ = η w /100 and λ f = ζ R m e t / k \lambda_f=\zeta R_{\mathrm{met}}/k λ f = ζ R met / k , and replace every distance between distinct points by max { w ( x , y ) , λ f } \max\{w(x,y),\lambda_f\} max { w ( x , y ) , λ f } . This is a metric, one query of it costs one original query, and terminal distances now lie in [ λ f , 2 R m e t ] [\lambda_f,2R_{\mathrm{met}}] [ λ f , 2 R met ] . A cleaned optimal tree has at most 2 k − 3 2k-3 2 k − 3 edges, so the new optimum satisfies OPT 0 ≤ OPT ≤ OPT 0 + 2 k λ f ≤ ( 1 + 4 ζ ) OPT 0 \operatorname{OPT}_0\le\operatorname{OPT}\le\operatorname{OPT}_0+2k\lambda_f\le(1+4\zeta)\operatorname{OPT}_0 OPT 0 ≤ OPT ≤ OPT 0 + 2 k λ f ≤ ( 1 + 4 ζ ) OPT 0 . Hence an estimate in [ OPT , ( 2 − η w ) OPT ] [\operatorname{OPT},(2-\eta_w)\operatorname{OPT}] [ OPT , ( 2 − η w ) OPT ] for the floored metric lies in [ OPT 0 , ( 2 − η w / 2 ) OPT 0 ] [\operatorname{OPT}_0,(2-\eta_w/2)\operatorname{OPT}_0] [ OPT 0 , ( 2 − η w /2 ) OPT 0 ] . From now on we work with the floored metric and write L , OPT L,\operatorname{OPT} L , OPT for its costs.
MST estimation. By the estimator of Czumaj and Sohler [19 ] , for a fixed accuracy ν > 0 \nu>0 ν > 0 we obtain with probability 1 − β ′ 1-\beta' 1 − β ′ values with
L / ( 1 + ν ) ≤ L ‾ ≤ L ≤ L ‾ ≤ ( 1 + ν ) L (6) L/(1+\nu)\le\underline{L}\le L\le\overline{L}\le(1+\nu)L \tag*{(6)} L / ( 1 + ν ) ≤ L ≤ L ≤ L ≤ ( 1 + ν ) L (6)
using O ~ ( k ν − O ( 1 ) log ( 1 / β ′ ) ) \widetilde{O}(k\nu^{-O(1)}\log(1/\beta')) O ( k ν − O ( 1 ) log ( 1/ β ′ )) queries and time.
The grid. Fix δ = 10 − 10 \delta=10^{-10} δ = 1 0 − 10 and τ = σ 0 / 100 \tau=\sigma_0/100 τ = σ 0 /100 . Put q 0 = ⌈ log ω 2 ⌉ + 1 q_0=\lceil\log_\omega2\rceil+1 q 0 = ⌈ log ω 2 ⌉ + 1 , I = ⌈ log ω ( 4 R m e t / λ f ) ⌉ + q 0 + 3 I=\lceil\log_\omega(4R_{\mathrm{met}}/\lambda_f)\rceil+q_0+3 I = ⌈ log ω ( 4 R met / λ f )⌉ + q 0 + 3 , and
a i = λ f Θ ω i − q 0 , 0 ≤ i < I , (7) a_i=\lambda_f\mathbin{\Theta}\omega^{i-q_0},\qquad0\le i<I, \tag*{(7)} a i = λ f Θ ω i − q 0 , 0 ≤ i < I , (7)
where Θ \Theta Θ has density 1 / ( Θ log ω ) 1/(\Theta\log\omega) 1/ ( Θ log ω ) on [ 1 , ω ) [1,\omega) [ 1 , ω ) , so that the shift is log-uniform (propose Θ \Theta Θ uniformly and accept it with probability 1 / Θ 1/\Theta 1/Θ ). The grid starts below λ f / 2 ≤ d min / 2 \lambda_f/2\le d_{\min}/2 λ f /2 ≤ d m i n /2 , ends above 4 R m e t ≥ 2 Δ T 4R_{\mathrm{met}}\ge2\Delta_T 4 R met ≥ 2 Δ T , and its largest scale is below 5 R m e t ≤ 5 L 5R_{\mathrm{met}}\le5L 5 R met ≤ 5 L . Hence I = O ( log ( k + 2 ) ) I=O(\log(k+2)) I = O ( log ( k + 2 )) , ∑ i a i ≤ 10 L / ( ω − 1 ) \sum_i a_i\le10L/(\omega-1) ∑ i a i ≤ 10 L / ( ω − 1 ) , and Theorem 3.1 applies to the finite sums over this grid.
The saving statistic on a cover
For each scale let N i ⊆ T N_i \subseteq T N i ⊆ T be a ρ i \rho_i ρ i -cover of T T T , that is, every terminal is within distance ρ i \rho_i ρ i of N i N_i N i , where 0 < ρ i ≤ δ a i 0 < \rho_i \le\delta a_i 0 < ρ i ≤ δ a i . Let C i = comp ( G a i ( N i ) ) C_i = \operatorname{comp}(G_{a_i}(N_i)) C i = comp ( G a i ( N i )) . The row of v ∈ V v \in V v ∈ V consists of the components containing a point of N i N_i N i within distance 5 a i / 8 5a_i/8 5 a i /8 of v v v ; it is eligible if it contains at least three components, and this eligibility is never recomputed after deletions. Let Φ i \Phi_i Φ i be the saving of C i C_i C i with these rows. Fix a map π i : T → N i \pi_i:T \to N_i π i : T → N i that moves points by at most ρ i \rho_i ρ i and fixes N i N_i N i , and let P i P_i P i be the partition of T T T into the preimages of the components of C i C_i C i . Writing ⪯ \preceq ⪯ for refinement,
comp ( G a i − 2 ρ i ( T ) ) ⪯ P i ⪯ Q i , (8) \operatorname{comp}(G_{a_i-2\rho_i}(T)) \preceq P_i \preceq Q_i, \tag*{(8)} comp ( G a i − 2 ρ i ( T )) ⪯ P i ⪯ Q i , (8)
because a pair at distance less than a i − 2 ρ i a_i-2\rho_i a i − 2 ρ i is mapped to a pair at distance less than a i a_i a i , while edges of G a i ( N i ) G_{a_i}(N_i) G a i ( N i ) and the attachments x π i ( x ) x\pi_i(x) x π i ( x ) are edges of G a i ( T ) G_{a_i}(T) G a i ( T ) .
Lemma 4.1 (The saving is sound). Let Q S Q_S Q S be a fixed integer with 5 4 ω − Q S < 1 − 2 δ \frac{5}{4}\omega^{-Q_S} < 1-2\delta 4 5 ω − Q S < 1 − 2 δ . Then ∑ i a i Φ i ≤ 32 Q S ( L − OPT ) \sum_i a_i\Phi_i \le32Q_S(L-\operatorname{OPT}) ∑ i a i Φ i ≤ 32 Q S ( L − OPT ) .
Proof. Fix a scale, write a = a i a=a_i a = a i and ρ = ρ i \rho=\rho_i ρ = ρ i , and take a list of eligible rows attaining Φ = Φ i \Phi=\Phi_i Φ = Φ i . In its incidence graph between rows and components, every connected component has saving at least two, since otherwise a single eligible row would do better. With u u u components, z z z rows, and c c c connected components, a spanning forest has f = u + z − c f=u+z-c f = u + z − c edges, and f − 2 z = Φ − c ≥ Φ / 2 f-2z=\Phi-c\ge\Phi/2 f − 2 z = Φ − c ≥ Φ/2 . Deleting a row of forest degree at most two does not decrease f − 2 z f-2z f − 2 z ; after all such deletions f ≥ 3 z f\ge3z f ≥ 3 z . For each forest edge pick a retained point of the component within 5 a / 8 5a/8 5 a /8 of the row’s center. The resulting stars cost at most 5 a f / 8 5af/8 5 a f /8 , and their identifications have rank f − z f-z f − z in P i P_i P i , hence by (8) at every threshold below a − 2 ρ a-2\rho a − 2 ρ . With r = f − 2 z ≥ z r=f-2z\ge z r = f − 2 z ≥ z their net benefit is at least
( a − 2 ρ ) ( f − z ) − 5 8 a f ≥ a [ ( 3 8 − 2 δ ) r − ( 1 4 + 2 δ ) z ] ≥ 1 16 a r ≥ 1 32 a Φ . (a-2\rho)(f-z)-\frac{5}{8}af \ge a\left[\left(\frac{3}{8}-2\delta\right)r-\left(\frac{1}{4}+2\delta\right)z\right] \ge\frac{1}{16}ar \ge\frac{1}{32}a\Phi. ( a − 2 ρ ) ( f − z ) − 8 5 a f ≥ a [ ( 8 3 − 2 δ ) r − ( 4 1 + 2 δ ) z ] ≥ 16 1 a r ≥ 32 1 a Φ.
Now process the scales of one residue class modulo Q S Q_S Q S in increasing order. An earlier star has diameter at most 5 4 ω − Q S a < a − 2 ρ \frac{5}{4}\omega^{-Q_S}a<a-2\rho 4 5 ω − Q S a < a − 2 ρ , so it lies inside one component of G a − 2 ρ ( T ) G_{a-2\rho}(T) G a − 2 ρ ( T ) ; therefore, at every threshold t ≤ a − 2 ρ t\le a-2\rho t ≤ a − 2 ρ , each component of G t ( T ) G_t(T) G t ( T ) together with all earlier links still lies in one part of P i P_i P i , and the marginal rank of the current stars is still f − z f-z f − z . Telescoping and Lemma 2.1 bound the sum over the class by 32 ( L − OPT ) 32(L-\operatorname{OPT}) 32 ( L − OPT ) ; summing the Q S Q_S Q S classes proves the claim. □
The band statistic
For C ∈ C i C\in C_i C ∈ C i and Y ⊆ C Y\subseteq C Y ⊆ C , let B C ( Y ) B_C(Y) B C ( Y ) be the integral over t ∈ [ τ a i , a i − 2 ρ i ] t\in[\tau a_i,a_i-2\rho_i] t ∈ [ τ a i , a i − 2 ρ i ] of the number of components of G t ( C ) G_t(C) G t ( C ) meeting Y Y Y , minus one, and define the band score
D i ( C ) = max { 0 , max Y ⊆ C , 2 ≤ ∣ Y ∣ ≤ 4 [ B C ( Y ) − c ( Y ) − 6 ρ i ] } . (9) D_i(C)=\max\left\{0,\max_{\substack{Y\subseteq C,\\2\le|Y|\le4}}\left[B_C(Y)-c(Y)-6\rho_i\right]\right\}. \tag*{(9)} D i ( C ) = max ⎩ ⎨ ⎧ 0 , Y ⊆ C , 2 ≤ ∣ Y ∣ ≤ 4 max [ B C ( Y ) − c ( Y ) − 6 ρ i ] ⎭ ⎬ ⎫ . (9)
If ∣ C ∣ ≤ B |C|\le B ∣ C ∣ ≤ B , it is computed from the distances between C C C and V V V in O ( n B 4 + poly ( B ) ) O(nB^4+\operatorname{poly}(B)) O ( n B 4 + poly ( B )) time, and 0 ≤ D i ( C ) ≤ 3 a i 0\le D_i(C)\le3a_i 0 ≤ D i ( C ) ≤ 3 a i . Let β T ( x , y ) \beta_T(x,y) β T ( x , y ) be the bottleneck distance between x x x and y y y in the complete graph on T T T . For x , y ∈ C x,y\in C x , y ∈ C , if β T ≤ β C ≤ β T + 2 ρ i \beta_T\le\beta_C\le\beta_T+2\rho_i β T ≤ β C ≤ β T + 2 ρ i : if β T ( x , y ) < a i − 2 ρ i \beta_T(x,y)<a_i-2\rho_i β T ( x , y ) < a i − 2 ρ i , project a bottleneck path to N i N_i N i ; otherwise use that C C C is connected in G a i ( N i ) G_{a_i}(N_i) G a i ( N i ) . The integrand above is determined by the at most three merge heights of Y Y Y in the bottleneck hierarchy, clipped to the band, so B C ( Y ) B_C(Y) B C ( Y ) exceeds the corresponding quantity for T T T by at most 6 ρ i 6\rho_i 6 ρ i .
Lemma 4.2 (The band statistic is sound). Let Q Q Q_Q Q Q be a fixed integer with ω − Q Q < τ / 2 \omega^{-Q_Q}<\tau/2 ω − Q Q < τ /2 . For all subfamilies F i ⊆ C i \mathcal F_i\subseteq C_i F i ⊆ C i , ∑ i ∑ C ∈ F i D i ( C ) ≤ Q Q ( L − OPT ) \sum_i\sum_{C\in\mathcal F_i}D_i(C)\le Q_Q(L-\operatorname{OPT}) ∑ i ∑ C ∈ F i D i ( C ) ≤ Q Q ( L − OPT ) .
Proof. Fix a residue class modulo Q Q Q_Q Q Q and process its scales in increasing order, identifying an attaining set Y C Y_C Y C for each C C C with D i ( C ) > 0 D_i(C)>0 D i ( C ) > 0 . At a threshold t t t in the band, sets in different components of C i C_i C i lie in different parts of comp ( G t ( T ) ) \operatorname{comp}(G_t(T)) comp ( G t ( T )) by (8) , so their ranks add. A set from an earlier scale a ′ ≤ ω − Q Q a i < τ a i a' \le\omega^{-Q_Q}a_i < \tau a_i a ′ ≤ ω − Q Q a i < τ a i lies in a component of G a ′ ( N i ′ ) G_{a'}(N_{i'}) G a ′ ( N i ′ ) , hence in one component of G t ( T ) G_t(T) G t ( T ) , so it does not change ranks in the current band. The marginal benefit of the current sets is therefore at least ∑ C ( B C ( Y C ) − 6 ρ i ) \sum_C(B_C(Y_C)-6\rho_i) ∑ C ( B C ( Y C ) − 6 ρ i ) , and Lemma 2.1 with the one-point networks of cost c ( Y C ) c(Y_C) c ( Y C ) bounds the sum over the class by L − OPT L-\operatorname{OPT} L − OPT .
Local random-greedy matching on multigraphs
Let H H H be a finite loopless multigraph on a vertex set U U U with m = ∣ U ∣ ≥ 1 m=|U|\ge1 m = ∣ U ∣ ≥ 1 and with p p p labeled edge copies, and put N H = m + p + 2 N_H=m+p+2 N H = m + p + 2 . A uniformly random order of the copies (their ranks) defines the random-greedy maximal matching, which scans the copies by increasing rank and adds a copy whenever both endpoints are free. We use the local oracles of [34 , 36 , 5 ] . For a copy e e e with endpoint u u u , the directed oracle EO ( e , u ) \operatorname{EO}(e,u) EO ( e , u ) decides whether u u u is still unmatched just before the rank of e e e : it scans the other copies f f f at u u u of smaller rank in increasing order, calls EO ( f , u f ) \operatorname{EO}(f,u_f) EO ( f , u f ) for their other endpoints u f u_f u f , and answers “no” at the first positive call. The vertex oracle VO ( v ) \operatorname{VO}(v) VO ( v ) scans the copies e = v u e=vu e = v u at v v v in increasing order and returns the first one with EO ( e , u ) = “yes” \operatorname{EO}(e,u)=“\text{yes}” EO ( e , u ) = “ yes ” , or “unmatched”. Answers are memoized within one call of VO \operatorname{VO} VO . A copy reported by VO ( v ) \operatorname{VO}(v) VO ( v ) is the matched copy of v v v .
Lemma 4.3 (Call bound). Fix H H H and a directed copy ( e , u ) (e,u) ( e , u ) , and let Q ( e , u ) Q(e,u) Q ( e , u ) be the number of invocations of EO ( e , u ) \operatorname{EO}(e,u) EO ( e , u ) , cached ones included, summed over separate calls VO ( v ) \operatorname{VO}(v) VO ( v ) for all v ∈ U v\in U v ∈ U . Over a uniformly random order, E Q ( e , u ) = O ( log N H ) \mathbb{E}\,Q(e,u)=O(\log N_H) E Q ( e , u ) = O ( log N H ) . Consequently, for a uniformly random root independent of the order, the expected number of oracle invocations is O ( ( 1 + p / m ) log N H ) O((1+p/m)\log N_H) O (( 1 + p / m ) log N H ) , and the expected number of invocations at a fixed vertex x x x is O ( ( 1 + d x ) log N H / m ) O((1+d_x)\log N_H/m) O (( 1 + d x ) log N H / m ) , where d x d_x d x is its degree.
Proof. The argument is that of Beh21, Section 3, adapted to parallel copies (see also MRTV25, Section 5). Within one call of VO \operatorname{VO} VO every directed state is evaluated once and invokes each lower-ranked state at most once, so Q ( e , u ) ≤ m ( p + 1 ) ≤ N H 2 Q(e,u)\le m(p+1)\le N_H^2 Q ( e , u ) ≤ m ( p + 1 ) ≤ N H 2 for every order.
The stack of an invocation is a walk w 0 , w 1 , … , w ℓ = u w_0,w_1,\ldots,w_\ell=u w 0 , w 1 , … , w ℓ = u along copies e 1 , … , e ℓ = e e_1,\ldots,e_\ell=e e 1 , … , e ℓ = e of decreasing rank. It is a simple path: at a first repeated vertex the closing copy has smaller rank than the copy used at the earlier visit, so it was tested there, and its answer “no” certifies a lower-ranked matched copy at its other endpoint, which would have stopped the later scan first. Let r ( f ) r(f) r ( f ) be the round in which f f f is removed by the parallel version of greedy matching (in each round the copies that are rank minima at both endpoints among the surviving copies are matched and their neighbors removed). Along a stack r r r does not increase, and it decreases strictly every two steps. Indeed, a matched queried copy recurses only to lower-ranked copies at its head, which were removed in earlier rounds. An unmatched queried copy e j e_j e j is removed in the round of its matched blocker, which lies at its head because the scan at its tail reached it; the next queried copy is either this blocker, which is matched, or a lower-ranked copy at the same vertex, removed in an earlier round. Parallel greedy matching on H H H is parallel greedy MIS on the line graph of H H H , a simple graph on the p p p copies; padding it with isolated vertices to N H N_H N H vertices, the theorem of Fischer and Noever [22 ] shows that, for a suitable constant C C C , all stacks have length at most D = C log N H D=C\log N_H D = C log N H except with probability N H − 3 N_H^{-3} N H − 3 .
For a stack P = ( e 1 , … , e ℓ ) P=(e_1,\ldots,e_\ell) P = ( e 1 , … , e ℓ ) ending at ( e , u ) (e,u) ( e , u ) under order π \pi π , let f ( π , P ) f(\pi,P) f ( π , P ) rotate the ranks along P P P : e j e_j e j receives the rank of e j + 1 e_{j+1} e j + 1 for j < ℓ j<\ell j < ℓ , and e ℓ e_\ell e ℓ the rank of e 1 e_1 e 1 . For a fixed image, the preimage stacks are nested suffixes of one another. Indeed, let two of them first diverge, reading backwards from ( e , u ) (e,u) ( e , u ) , at a common copy e ′ e' e ′ preceded by f ′ f' f ′ in the first stack, where α = π 1 ( e ′ ) ≤ π 2 ( e ′ ) \alpha=\pi_1(e')\le\pi_2(e') α = π 1 ( e ′ ) ≤ π 2 ( e ′ ) . By simplicity f ′ f' f ′ is not on the second stack, so π 2 ( f ′ ) = α \pi_2(f')=\alpha π 2 ( f ′ ) = α , and the two orders agree on all copies of rank below α \alpha α . If f ′ f' f ′ were unmatched under π 2 \pi_2 π 2 , its matched blocker of rank below α \alpha α would also block the first stack before f ′ f' f ′ or before e ′ e' e ′ ; so f ′ f' f ′ is matched under π 2 \pi_2 π 2 , and the second stack would have stopped at f ′ f' f ′ before reaching e ′ e' e ′ , a contradiction. Since P P P and its image determine π \pi π , every image has at most D D D preimages with short stacks. Counting the pairs ( π , P ) (\pi,P) ( π , P ) gives E Q ( e , u ) ≤ D + N H 2 ⋅ N − 3 \mathbb{E}\,Q(e,u)\le D+N_H^2\cdot N^{-3} E Q ( e , u ) ≤ D + N H 2 ⋅ N − 3 . The consequences follow by summing over directed copies and dividing by m m m .
Lazy ranks on padded candidate lists
In our application the incident copies of a vertex are not listed explicitly. Instead every vertex v v v has a fixed indexable list of candidate slots ; testing a slot returns null or an incident copy with a shared identifier. Every copy occurs in exactly one slot at each endpoint, and a successful test also locates the reverse slot. Preparing a vertex and testing a slot cost at most C ∗ C_* C ∗ , and the slot-cost bounds at v v v sum to W v W_v W v ; these bounds are fixed before the ranks are drawn.
Lemma 4.4 (Padded lists). There is an implementation of the vertex oracle with correct answers and an event E \mathcal{E} E of probability 1 − N ∗ − Ω ( 1 ) 1-N_*^{-\Omega(1)} 1 − N ∗ − Ω ( 1 ) , for a polynomial bound N ∗ N_* N ∗ on the number of slots, such that for a uniformly random root
E [ 1 E Cost ] ≤ O ~ ( C ∗ ( 1 + p m ) + 1 m ∑ v W v ) . (10) \mathbb{E}\left[1_{\mathcal{E}}\operatorname{Cost}\right]\le\widetilde{O}\left(C_*\left(1+\frac{p}{m}\right)+\frac{1}{m}\sum_v W_v\right). \tag*{(10)} E [ 1 E Cost ] ≤ O ( C ∗ ( 1 + m p ) + m 1 v ∑ W v ) . (10) Degrees need not be known. The bound holds separately for several nonnegative cost accounts.
Proof. Ranks are independent uniform numbers in [ 0 , 1 ) [0,1) [ 0 , 1 ) , exposed lazily on the common breakpoints 0 , 2 − P 0 , 2 1 − P 0 , … , 1 0,2^{-P_0},2^{1-P_0},\ldots,1 0 , 2 − P 0 , 2 1 − P 0 , … , 1 , where 2 − P 0 2^{-P_0} 2 − P 0 is below the reciprocal of N ∗ N_* N ∗ . Every vertex v v v maintains a boundary b v b_v b v , initially 0 0 0 , such that all incident ranks below b v b_v b v are known, and the following invariant holds after each step: every assigned rank is stored at both endpoints, and every unassigned copy u v uv uv has rank uniform on [ max { b u , b v } , 1 ) [\max\{b_u,b_v\},1) [ max { b u , b v } , 1 ) , independently. To extend v v v from [ a , b ) [a,b) [ a , b ) with a = b v a=b_v a = b v , note that every other endpoint has b u ≤ a b_u\le a b u ≤ a or b u ≥ b b_u\ge b b u ≥ b , because the breakpoints are common. In the first case the rank of an unassigned copy lies in [ a , b ) [a,b) [ a , b ) with probability p a , b = ( b − a ) / ( 1 − a ) p_{a,b}=(b-a)/(1-a) p a , b = ( b − a ) / ( 1 − a ) , independently; on success it receives a fresh uniform rank in [ a , b ) [a,b) [ a , b ) , which is stored at both endpoints. In the second case the rank is already at least b b b . We realize this by Bernoulli( p a , b ) (p_{a,b}) ( p a , b ) coins on all slot indices, skipping failures by geometric jumps, and testing only the selected slots; coins on null slots and on assigned copies are discarded. After the step we set b v = b b_v=b b v = b ; the invariant follows because distinct unassigned copies receive independent interval conditions. The oracles above only need the copies at v v v in rank order below a requested value, which this procedure provides.
For an opened interval, the selected slots cost O ( p a , b W v + C ∗ log N ∗ ) O(p_{a,b}W_v+C_*\log N_*) O ( p a , b W v + C ∗ log N ∗ ) with high probability. The number of incident ranks below x x x is binomial with mean x d v xd_v x d v , so if the last boundary of v v v is b v = 2 x b_v=2x b v = 2 x , then b v d v = O ( j v + log N ∗ ) b_vd_v=O(j_v+\log N_*) b v d v = O ( j v + log N ∗ ) , where j v j_v j v is the number of rank-ordered requests at v v v . Since the interval probabilities sum to at most 2 b v 2b_v 2 b v , the cost at v v v is O ~ ( ( W v / d v ) ( j v + 1 ) + C ∗ ) \widetilde{O}((W_v/d_v)(j_v+1)+C_*) O (( W v / d v ) ( j v + 1 ) + C ∗ ) on the joint high-probability event. By Lemma 4.3 the expected value of j v j_v j v for a random root is O ( ( 1 + d v ) log N H / m ) O((1+d_v)\log N_H/m) O (( 1 + d v ) log N H / m ) , under the original law of the ranks, and an isolated vertex is entered only as the root. Summing over v v v proves (10) .
Independent terminal covers and completeness
At scale a = a i a=a_i a = a i put ρ = δ a \rho=\delta a ρ = δ a , h = ⌈ log ( k + 2 ) ⌉ h=\lceil\log(k+2)\rceil h = ⌈ log ( k + 2 )⌉ , r c = ρ / h r_c=\rho/h r c = ρ / h , and ψ = ⌈ ℓ log ( 4 k Γ / β ) ⌉ \psi=\lceil\ell\log(4k\Gamma/\beta)\rceil ψ = ⌈ ℓ log ( 4 k Γ/ β )⌉ . For x ∈ T x\in T x ∈ T let d x d_x d x be the number of other terminals within distance r c r_c r c of x x x . Every terminal x x x is retained independently with probability q x = 1 − ( d x / ( d x + 1 ) ) ψ q_x=1-(d_x/(d_x+1))^\psi q x = 1 − ( d x / ( d x + 1 ) ) ψ , and one fixed terminal is always retained; N = N i N=N_i N = N i is the retained set and g = g i = ∣ N i ∣ g = g_i = |N_i| g = g i = ∣ N i ∣ . The bit of x x x is resolved on demand, without computing d x d_x d x : we draw uniform terminals and query their distance to x x x , retaining x x x if x x x itself is drawn before ψ \psi ψ other terminals within distance r c r_c r c , and rejecting it otherwise. The draws for different pairs of a terminal and a scale use private independent randomness, and resolved bits are cached.
Lemma 4.5 (Covers). Except with probability β \beta β , simultaneously at all scales, N i N_i N i is a ρ i \rho_i ρ i -cover of T T T and
g i ≤ H c + 4 ψ h L δ a i , H c = 2 + 2 ψ + log ( 2 I / β ) . (11) g_i \le H_c + \frac{4\psi hL}{\delta a_i}, \qquad H_c = 2 + 2\psi+ \log(2I/\beta). \tag*{(11)} g i ≤ H c + δ a i 4 ψ h L , H c = 2 + 2 ψ + log ( 2 I / β ) . (11) Moreover, for every adaptive procedure that does not observe unresolved draws, the total number W W W of draws and the number A A A of retained terminals it has found satisfy
P [ W > 2 k ( A + log ( 1 / β ) ) at some time ] ≤ β . (12) \mathbb{P}\left[W > 2k\left(A+\log(1/\beta)\right)\text{ at some time}\right] \le\beta. \tag*{(12)} P [ W > 2 k ( A + log ( 1/ β ) ) at some time ] ≤ β . (12) Proof. A draw hits its current target with probability 1 / k 1/k 1/ k , given the past, and every such hit reveals a new retained terminal. Hence exp ( W / ( 2 k ) − # hits ) \exp(W/(2k)-\#\text{hits}) exp ( W / ( 2 k ) − # hits ) is a nonnegative supermartingale, and the maximal inequality gives (12) .
For the cover property, let B j ( x ) B_j(x) B j ( x ) be the set of terminals within j j j hops of x x x in the graph joining terminals at distance at most r c r_c r c . Since ∣ B h ( x ) ∣ ≤ k |B_h(x)| \le k ∣ B h ( x ) ∣ ≤ k , some j < h j < h j < h has ∣ B j ( x ) ∣ / ∣ B j + 1 ( x ) ∣ ≥ k − 1 / h ≥ e − 1 |B_j(x)|/|B_{j+1}(x)| \ge k^{-1/h} \ge e^{-1} ∣ B j ( x ) ∣/∣ B j + 1 ( x ) ∣ ≥ k − 1/ h ≥ e − 1 . Every y ∈ B j ( x ) y \in B_j(x) y ∈ B j ( x ) has its closed neighborhood in B j + 1 ( x ) B_{j+1}(x) B j + 1 ( x ) , so ∑ y ∈ B h ( x ) 1 / ( d y + 1 ) ≥ e − 1 \sum_{y\in B_h(x)}1/(d_y+1) \ge e^{-1} ∑ y ∈ B h ( x ) 1/ ( d y + 1 ) ≥ e − 1 , and the probability that no terminal of B h ( x ) B_h(x) B h ( x ) is retained is at most exp ( − ψ / e ) ≤ β / ( 4 k I ) \exp(-\psi/e) \le\beta/(4kI) exp ( − ψ / e ) ≤ β / ( 4 k I ) . All of B h ( x ) B_h(x) B h ( x ) is within distance h r c = ρ hr_c = \rho h r c = ρ of x x x . For the size, q x ≤ ψ / ( d x + 1 ) q_x \le\psi/(d_x+1) q x ≤ ψ / ( d x + 1 ) , and ∑ x 1 / ( d x + 1 ) \sum_x 1/(d_x+1) ∑ x 1/ ( d x + 1 ) is the expected number of local minima of a random order in this graph; local minima are pairwise more than r c r_c r c apart, so a doubled MST tour bounds their number by 1 + 2 L / r c 1+2L/r_c 1 + 2 L / r c . Hence E g ≤ 1 + ψ ( 1 + 2 h L / ρ ) \mathbb{E}g \le1+\psi(1+2hL/\rho) E g ≤ 1 + ψ ( 1 + 2 h L / ρ ) , and the Bernoulli exponential moment gives g ≤ 2 E g + log ( 2 I / β ) g \le2\mathbb{E}g+\log(2I/\beta) g ≤ 2 E g + log ( 2 I / β ) except with probability β / ( 2 I ) \beta/(2I) β / ( 2 I ) . Union bounds finish the proof.
Corollary 4.6. Let E \mathcal{E} E be any event and suppose E [ 1 E A ] ≤ μ A \mathbb{E}[\mathbf{1}_{\mathcal{E}}A] \le\mu_A E [ 1 E A ] ≤ μ A . For β A , β W ∈ ( 0 , 1 ) \beta_A,\beta_W \in(0,1) β A , β W ∈ ( 0 , 1 ) , the event E \mathcal{E} E and W > 2 k ( μ A / β A + log ( 1 / β W ) ) W > 2k(\mu_A/\beta_A+\log(1/\beta_W)) W > 2 k ( μ A / β A + log ( 1/ β W )) occur together with probability at most β A + β W \beta_A+\beta_W β A + β W .
Proof. Apply Markov’s inequality to 1 E A \mathbf{1}_{\mathcal{E}}A 1 E A and (12) with β W \beta_W β W .
Uniform terminals accepted exactly when retained are independent uniform samples of N N N , and s s s of them cost O ~ ( k ( s + 1 ) ) \widetilde{O}(k(s+1)) O ( k ( s + 1 )) draws by (12) . Stopping after s = C ξ − 2 log ( 2 / β ) s=C\xi^{-2}\log(2/\beta) s = C ξ − 2 log ( 2/ β ) acceptances among t s t_s t s proposals and setting g ± = ( 1 ± ξ / 2 ) k s / t s g_{\pm}=(1\pm\xi/2)ks/t_s g ± = ( 1 ± ξ /2 ) k s / t s gives, with probability 1 − β 1-\beta 1 − β ,
( 1 − ξ ) g ≤ g − ≤ g ≤ g + ≤ ( 1 + ξ ) g . (13) (1-\xi)g \le g_- \le g \le g_+ \le(1+\xi)g. \tag*{(13)} ( 1 − ξ ) g ≤ g − ≤ g ≤ g + ≤ ( 1 + ξ ) g . (13)
On the event of Lemma 4.5 , using ∑ i a i ≤ 10 L / ( ω − 1 ) \sum_i a_i \le10L/(\omega-1) ∑ i a i ≤ 10 L / ( ω − 1 ) ,
∑ i a i g i ≤ A L , A = 1 + 10 H c ω − 1 + 4 ψ h I δ , (14) \sum_i a_i g_i \le\mathcal{A}L, \qquad\mathcal{A}=1+\frac{10H_c}{\omega-1}+\frac{4\psi hI}{\delta}, \tag*{(14)} i ∑ a i g i ≤ A L , A = 1 + ω − 1 10 H c + δ 4 ψ h I , (14)
and A \mathcal{A} A is polylogarithmic in n n n . We use the notation of § 4.1 for these covers, with ρ i = δ a i \rho_i=\delta a_i ρ i = δ a i .
Completeness of the saving
On the cover event,
Y i ≤ Φ i . (15) Y_i \le\Phi_i. \tag*{(15)} Y i ≤ Φ i . (15)
Indeed, if a center is within 3 a i / 5 3a_i/5 3 a i /5 of x x x in an exact component, then π i ( x ) \pi_i(x) π i ( x ) lies in the same exact component and within 3 a i / 5 + ρ i < 5 a i / 8 3a_i/5+\rho_i < 5a_i/8 3 a i /5 + ρ i < 5 a i /8 of the center; distinct exact components contain distinct representative components by (8) . Thus every eligible ideal row family yields an eligible family on C i C_i C i with at least the same union and the same number of rows.
Completeness of the band statistic. Let B Q = ⌈ 400 A ⌉ B_Q=\lceil400A\rceil B Q = ⌈ 400 A ⌉ and T Q = ∑ i ∑ C ∈ C i , ∣ C ∣ ≤ B Q D i ( C ) T_Q=\sum_i\sum_{C\in\mathcal{C}_i,\lvert C\rvert\le B_Q}D_i(C) T Q = ∑ i ∑ C ∈ C i , ∣ C ∣ ≤ B Q D i ( C ) .
Lemma 4.7 (Band completeness). On the cover event , T Q ≤ Q Q ( L − O P T ) T_Q\le Q_Q(L-\mathrm{OPT}) T Q ≤ Q Q ( L − OPT ) . Moreover, with probability at least 1 − 10 − 4 1-10^{-4} 1 − 1 0 − 4 over the shift of the grid,
Λ ≤ 2 T Q / σ 0 + L / 100. (16) \Lambda\le2T_Q/\sigma_0+L/100. \tag*{(16)} Λ ≤ 2 T Q / σ 0 + L /100. (16)
Proof. The first claim is Lemma 4.2 . Call an exact component persistent if it is a part of P i \mathcal{P}_i P i . By (8) , a component of Q i \mathcal{Q}_i Q i that is also a component of G a i − 2 ρ i ( T ) G_{a_i-2\rho_i}(T) G a i − 2 ρ i ( T ) is persistent, so the number of nonpersistent components is at most κ T ( a i − 2 ρ i ) − κ T ( a i ) \kappa_T(a_i-2\rho_i)-\kappa_T(a_i) κ T ( a i − 2 ρ i ) − κ T ( a i ) , the number of MST edges with weight in [ a i − 2 ρ i , a i ) [a_i-2\rho_i,a_i) [ a i − 2 ρ i , a i ) . Under the log-uniform shift an MST edge of weight w w w contributes expected scale weight
1 log ω ∫ w / ( 1 − 2 δ ) w 1 a d a = 2 δ w ( 1 − 2 δ ) log ω , \frac{1}{\log\omega}\int_{w/(1-2\delta)}^w \frac{1}{a}\,da=\frac{2\delta w}{(1-2\delta)\log\omega}, log ω 1 ∫ w / ( 1 − 2 δ ) w a 1 d a = ( 1 − 2 δ ) log ω 2 δ w ,
so by Markov’s inequality the scale-weighted number of nonpersistent components is at most L / 200 L/200 L /200 with probability at least 1 − 10 − 4 1-10^{-4} 1 − 1 0 − 4 . Persistent components with more than B Q B_Q B Q retained points have scale weight at most ∑ i a i g i / ( B Q + 1 ) ≤ L / 400 \sum_i a_i g_i/(B_Q+1)\le L/400 ∑ i a i g i / ( B Q + 1 ) ≤ L /400 by (14) .
Let C ∈ Q i C\in\mathcal{Q}_i C ∈ Q i be any other component with A i ( C ) ≥ σ 0 a i A_i(C)\ge\sigma_0a_i A i ( C ) ≥ σ 0 a i , so that C ∩ N i C\cap N_i C ∩ N i is a component of C i \mathcal{C}_i C i with at most B Q B_Q B Q points, and project an attaining set Y Y Y by π i \pi_i π i . The benefit decreases by at most 3 ρ i 3\rho_i 3 ρ i , the one-point connection cost increases by at most 8 ρ i 8\rho_i 8 ρ i , restricting to the band loses at most 3 τ a i + 6 ρ i 3\tau a_i+6\rho_i 3 τ a i + 6 ρ i , and the penalty is 6 ρ i 6\rho_i 6 ρ i . Hence D i ( C ∩ N i ) ≥ A i ( C ) − 23 ρ i − 3 τ a i ≥ σ 0 a i / 2 D_i(C\cap N_i)\ge A_i(C)-23\rho_i-3\tau a_i\ge\sigma_0a_i/2 D i ( C ∩ N i ) ≥ A i ( C ) − 23 ρ i − 3 τ a i ≥ σ 0 a i /2 ; the projected set has at least two points, since otherwise its benefit would be at most 3 ρ i < σ 0 a i 3\rho_i<\sigma_0a_i 3 ρ i < σ 0 a i . Summing proves (4.11).
Scores of retained components and the estimator
At each scale we shall compute Z i Z_i Z i with
0 ≤ Z i ≤ Φ i , Z i ≥ Φ i / 2 − ε g i , ε = ε 0 10 5 A . (17) 0\le Z_i\le\Phi_i,\qquad Z_i\ge\Phi_i/2-\varepsilon g_i,\qquad\varepsilon=\frac{\varepsilon_0}{10^5A}. \tag*{(17)} 0 ≤ Z i ≤ Φ i , Z i ≥ Φ i /2 − ε g i , ε = 1 0 5 A ε 0 . (17)
A breadth-first search in G a ( N ) G_a(N) G a ( N ) from a retained point either lists its component, if it has at most B B B points, or stops after B + 1 B+1 B + 1 points; each popped point is compared with all terminals, resolving retention only for those within distance a a a , so the search costs O ( k B ) O(kB) O ( k B ) queries and discloses at most B + 1 B+1 B + 1 retained points. Given two complete components within 5 a / 8 5a/8 5 a /8 of a center v v v , the original eligibility of the row of v v v is decided by scanning the terminals within 5 a / 8 5a/8 5 a /8 of v v v outside the two components until a retained one is found; it lies in a third component.
Let D D D be the union of the balls A v = { x ∈ T : w ( v , x ) ≤ 5 a / 8 } A_v=\{x\in T:w(v,x)\le5a/8\} A v = { x ∈ T : w ( v , x ) ≤ 5 a /8 } of at most s s s selected centers. Let C 0 C_0 C 0 be the components with at most B B B points that meet D D D , let U U U be the other components with at most B B B points, let b l a r g e b_{\mathrm{large}} b large be the number of larger components, and let Ψ U \Psi_U Ψ U be the saving of the system restricted to U U U , with the original eligibility.
Lemma 4.8 (Deletion and scores). We have
Φ ≤ b l a r g e + ∣ C 0 ∣ + Ψ U , ∣ C 0 ∣ − 2 s + Ψ U ≤ Φ . (18) \Phi\le b_{\mathrm{large}}+\lvert C_0\rvert+\Psi_U,\qquad\lvert C_0\rvert-2s+\Psi_U\le\Phi. \tag*{(18)} Φ ≤ b large + ∣ C 0 ∣ + Ψ U , ∣ C 0 ∣ − 2 s + Ψ U ≤ Φ. (18) Suppose a graph on U U U has expected random-greedy matching size μ \mu μ with Ψ U / 2 − Δ ≤ μ ≤ Ψ U \Psi_U/2-\Delta\le\mu\le\Psi_U Ψ U /2 − Δ ≤ μ ≤ Ψ U . Then a bounded score of a uniform retained point has mean ( ∣ C 0 ∣ + μ ) / g (\lvert C_0\rvert+\mu)/g (∣ C 0 ∣ + μ ) / g , and with O ( ε − 2 log ( 2 / β ) ) O(\varepsilon^{-2}\log(2/\beta)) O ( ε − 2 log ( 2/ β )) samples one obtains, with probability 1 − β 1-\beta 1 − β , a value Z Z Z with 0 ≤ Z ≤ Φ 0\le Z\le\Phi 0 ≤ Z ≤ Φ and
Z ≥ Φ / 2 − b l a r g e / 2 − Δ − 2 s − 3 ξ g . (19) Z\ge\Phi/2-b_{\mathrm{large}}/2-\Delta-2s-3\xi g. \tag*{(19)} Z ≥ Φ/2 − b large /2 − Δ − 2 s − 3 ξ g . (19)
Proof. The first inequality holds because removing elements while keeping eligibility fixed decreases the saving by at most their number. For the second, add the eligible selected rows to a family @page 25/58 starts_mid=0 ends_mid=1 Completeness of the band statistic. Let B Q = ⌈ 400 A ⌉ B_Q=\lceil400A\rceil B Q = ⌈ 400 A ⌉ and T Q = ∑ i ∑ C ∈ C i , ∣ C ∣ ≤ B Q D i ( C ) T_Q=\sum_i\sum_{C\in\mathcal{C}_i,\lvert C\rvert\le B_Q}D_i(C) T Q = ∑ i ∑ C ∈ C i , ∣ C ∣ ≤ B Q D i ( C ) .
Lemma 4.7 (Band completeness). On the cover event , T Q ≤ Q Q ( L − O P T ) T_Q\le Q_Q(L-\mathrm{OPT}) T Q ≤ Q Q ( L − OPT ) . Moreover, with probability at least 1 − 10 − 4 1-10^{-4} 1 − 1 0 − 4 over the shift of the grid,
Λ ≤ 2 T Q / σ 0 + L / 100. (20) \Lambda\le2T_Q/\sigma_0+L/100. \tag*{(20)} Λ ≤ 2 T Q / σ 0 + L /100. (20)
Proof. The first claim is Lemma 4.2 . Call an exact component persistent if it is a part of P i \mathcal{P}_i P i . By (8) , a component of Q i \mathcal{Q}_i Q i that is also a component of G a i − 2 ρ i ( T ) G_{a_i-2\rho_i}(T) G a i − 2 ρ i ( T ) is persistent, so the number of nonpersistent components is at most κ T ( a i − 2 ρ i ) − κ T ( a i ) \kappa_T(a_i-2\rho_i)-\kappa_T(a_i) κ T ( a i − 2 ρ i ) − κ T ( a i ) , the number of MST edges with weight in [ a i − 2 ρ i , a i ) [a_i-2\rho_i,a_i) [ a i − 2 ρ i , a i ) . Under the log-uniform shift an MST edge of weight w w w contributes expected scale weight
1 log ω ∫ w / ( 1 − 2 δ ) w 1 a d a = 2 δ w ( 1 − 2 δ ) log ω , \frac{1}{\log\omega}\int_{w/(1-2\delta)}^w \frac{1}{a}\,da=\frac{2\delta w}{(1-2\delta)\log\omega}, log ω 1 ∫ w / ( 1 − 2 δ ) w a 1 d a = ( 1 − 2 δ ) log ω 2 δ w ,
so by Markov’s inequality the scale-weighted number of nonpersistent components is at most L / 200 L/200 L /200 with probability at least 1 − 10 − 4 1-10^{-4} 1 − 1 0 − 4 . Persistent components with more than B Q B_Q B Q retained points have scale weight at most ∑ i a i g i / ( B Q + 1 ) ≤ L / 400 \sum_i a_i g_i/(B_Q+1)\le L/400 ∑ i a i g i / ( B Q + 1 ) ≤ L /400 by (14) .
Let C ∈ Q i C\in\mathcal{Q}_i C ∈ Q i be any other component with A i ( C ) ≥ σ 0 a i A_i(C)\ge\sigma_0a_i A i ( C ) ≥ σ 0 a i , so that C ∩ N i C\cap N_i C ∩ N i is a component of C i \mathcal{C}_i C i with at most B Q B_Q B Q points, and project an attaining set Y Y Y by π i \pi_i π i . The benefit decreases by at most 3 ρ i 3\rho_i 3 ρ i , the one-point connection cost increases by at most 8 ρ i 8\rho_i 8 ρ i , restricting to the band loses at most 3 τ a i + 6 ρ i 3\tau a_i+6\rho_i 3 τ a i + 6 ρ i , and the penalty is 6 ρ i 6\rho_i 6 ρ i . Hence D i ( C ∩ N i ) ≥ A i ( C ) − 23 ρ i − 3 τ a i ≥ σ 0 a i / 2 D_i(C\cap N_i)\ge A_i(C)-23\rho_i-3\tau a_i\ge\sigma_0a_i/2 D i ( C ∩ N i ) ≥ A i ( C ) − 23 ρ i − 3 τ a i ≥ σ 0 a i /2 ; the projected set has at least two points, since otherwise its benefit would be at most 3 ρ i < σ 0 a i 3\rho_i<\sigma_0a_i 3 ρ i < σ 0 a i . Summing proves (4.11).
Scores of retained components and the estimator
At each scale we shall compute Z i Z_i Z i with
0 ≤ Z i ≤ Φ i , Z i ≥ Φ i / 2 − ε g i , ε = ε 0 10 5 A . (21) 0\le Z_i\le\Phi_i,\qquad Z_i\ge\Phi_i/2-\varepsilon g_i,\qquad\varepsilon=\frac{\varepsilon_0}{10^5A}. \tag*{(21)} 0 ≤ Z i ≤ Φ i , Z i ≥ Φ i /2 − ε g i , ε = 1 0 5 A ε 0 . (21)
A breadth-first search in G a ( N ) G_a(N) G a ( N ) from a retained point either lists its component, if it has at most B B B points, or stops after B + 1 B+1 B + 1 points; each popped point is compared with all terminals, resolving retention only for those within distance a a a , so the search costs O ( k B ) O(kB) O ( k B ) queries and discloses at most B + 1 B+1 B + 1 retained points. Given two complete components within 5 a / 8 5a/8 5 a /8 of a center v v v , the original eligibility of the row of v v v is decided by scanning the terminals within 5 a / 8 5a/8 5 a /8 of v v v outside the two components until a retained one is found; it lies in a third component.
Let D D D be the union of the balls A v = { x ∈ T : w ( v , x ) ≤ 5 a / 8 } A_v=\{x\in T:w(v,x)\le5a/8\} A v = { x ∈ T : w ( v , x ) ≤ 5 a /8 } of at most s s s selected centers. Let C 0 C_0 C 0 be the components with at most B B B points that meet D D D , let U U U be the other components with at most B B B points, let b l a r g e b_{\mathrm{large}} b large be the number of larger components, and let Ψ U \Psi_U Ψ U be the saving of the system restricted to U U U , with the original eligibility.
Lemma 4.8 (Deletion and scores). We have
Φ ≤ b l a r g e + ∣ C 0 ∣ + Ψ U , ∣ C 0 ∣ − 2 s + Ψ U ≤ Φ . (22) \Phi\le b_{\mathrm{large}}+\lvert C_0\rvert+\Psi_U,\qquad\lvert C_0\rvert-2s+\Psi_U\le\Phi. \tag*{(22)} Φ ≤ b large + ∣ C 0 ∣ + Ψ U , ∣ C 0 ∣ − 2 s + Ψ U ≤ Φ. (22) Suppose a graph on U U U has expected random-greedy matching size μ \mu μ with Ψ U / 2 − Δ ≤ μ ≤ Ψ U \Psi_U/2-\Delta\le\mu\le\Psi_U Ψ U /2 − Δ ≤ μ ≤ Ψ U . Then a bounded score of a uniform retained point has mean ( ∣ C 0 ∣ + μ ) / g (\lvert C_0\rvert+\mu)/g (∣ C 0 ∣ + μ ) / g , and with O ( ε − 2 log ( 2 / β ) ) O(\varepsilon^{-2}\log(2/\beta)) O ( ε − 2 log ( 2/ β )) samples one obtains, with probability 1 − β 1-\beta 1 − β , a value Z Z Z with 0 ≤ Z ≤ Φ 0\le Z\le\Phi 0 ≤ Z ≤ Φ and
Z ≥ Φ / 2 − b l a r g e / 2 − Δ − 2 s − 3 ξ g . (23) Z\ge\Phi/2-b_{\mathrm{large}}/2-\Delta-2s-3\xi g. \tag*{(23)} Z ≥ Φ/2 − b large /2 − Δ − 2 s − 3 ξ g . (23)
Proof. The first inequality holds because removing elements while keeping eligibility fixed decreases the saving by at most their number. For the second, add the eligible selected rows to a family attaining Ψ U \Psi_U Ψ U . These rows do not meet U U U , and every component of C 0 C_0 C 0 lies in a selected row; an ineligible selected row leaves at most two of them uncovered. For the score, explore the component C C C of a uniform retained point. If it is large the score is 0; if C ∈ C 0 C \in C_0 C ∈ C 0 it is 1 / ∣ C ∣ 1/|C| 1/∣ C ∣ ; if C ∈ U C \in U C ∈ U , then with probability 1 / ∣ C ∣ 1/|C| 1/∣ C ∣ we run the matching oracle at C C C with fresh ranks and score half of the matched indicator, and otherwise score 0. Every C ∈ U C \in U C ∈ U is initiated with probability 1 / g 1/g 1/ g and the matched indicators sum to 2 μ 2\mu 2 μ in expectation, which gives the mean. Let ϑ ^ \hat{\vartheta} ϑ ^ be the empirical mean and Z = max { 0 , g − max { 0 , ϑ ^ − ξ } − 2 s } Z=\max\{0,g-\max\{0,\hat{\vartheta}-\xi\}-2s\} Z = max { 0 , g − max { 0 , ϑ ^ − ξ } − 2 s } . On the Hoeffding and calibration events, Z ≤ max { 0 , ∣ C 0 ∣ + μ − 2 s } ≤ Φ Z\leq\max\{0,|C_0|+\mu-2s\}\leq\Phi Z ≤ max { 0 , ∣ C 0 ∣ + μ − 2 s } ≤ Φ by (4.13), and Z ≥ ∣ C 0 ∣ + μ − 2 s − 3 ξ g Z\geq|C_0|+\mu-2s-3\xi g Z ≥ ∣ C 0 ∣ + μ − 2 s − 3 ξ g , which together with (4.13) gives (4.14).
The band estimate. With an independent sample, score a uniform retained point by D i ( C ) / ( 3 a i ∣ C ∣ ) D_i(C)/(3a_i|C|) D i ( C ) / ( 3 a i ∣ C ∣ ) if its component C C C has at most B Q B_Q B Q points, and 0 otherwise; the mean is ∑ ∣ C ∣ ≤ B Q D i ( C ) / ( 3 a i g i ) \sum_{|C|\leq B_Q}D_i(C)/(3a_i g_i) ∑ ∣ C ∣ ≤ B Q D i ( C ) / ( 3 a i g i ) . With ξ Q = σ 0 / ( 10 5 A ) \xi_Q=\sigma_0/(10^5A) ξ Q = σ 0 / ( 1 0 5 A ) , the estimate T ^ i = 3 a i g − max { 0 , ϑ ^ i − ξ Q } \widehat{T}_i=3a_i g-\max\{0,\hat{\vartheta}_i-\xi_Q\} T i = 3 a i g − max { 0 , ϑ ^ i − ξ Q } loses at most 9 ξ Q a i g i 9\xi_Qa_i g_i 9 ξ Q a i g i , so T ^ = ∑ i T ^ i \widehat{T}=\sum_i\widehat{T}_i T = ∑ i T i satisfies
0 ≤ T ^ ≤ T Q , T Q − T ^ ≤ σ 0 L / 1000. (24) 0\leq\widehat{T}\leq T_Q,\qquad T_Q-\widehat{T}\leq\sigma_0L/1000. \tag*{(24)} 0 ≤ T ≤ T Q , T Q − T ≤ σ 0 L /1000. (24)
Each sample costs O ( n B Q 4 ) O(nB_Q^4) O ( n B Q 4 ) time and O ( B Q ) O(B_Q) O ( B Q ) disclosures, so this costs O ~ ( n ) \widetilde{O}(n) O ( n ) in total.
The estimate and its guarantee
Let S = ∑ i a i Z i S=\sum_i a_iZ_i S = ∑ i a i Z i , let L ‾ \overline{L} L be as in (6) , and return
C ^ = L ‾ − max { S 32 Q S , T ^ Q Q } . (25) \widehat{C}=\overline{L}-\max\left\{\frac{S}{32Q_S},\frac{\widehat{T}}{Q_Q}\right\}. \tag*{(25)} C = L − max { 32 Q S S , Q Q T } . (25)
Proposition 4.9. Let c ∗ = min { e 0 / ( 32000 Q S ) , σ 0 / ( 1000 Q Q ) , e 0 / 8 , 1 / 10 } c_*=\min\{e_0/(32000Q_S),\sigma_0/(1000Q_Q),e_0/8,1/10\} c ∗ = min { e 0 / ( 32000 Q S ) , σ 0 / ( 1000 Q Q ) , e 0 /8 , 1/10 } and 0 < ν ≤ c ∗ / 10 0<\nu\leq c_*/10 0 < ν ≤ c ∗ /10 . On the events of Lemmas 4.5 and 4.7, of (6) , and of (4.12) and (24) , the output satisfies O P T ≤ C ^ ≤ ( 2 − c ∗ ) O P T \mathrm{OPT}\leq\widehat{C}\leq(2-c_*)\mathrm{OPT} OPT ≤ C ≤ ( 2 − c ∗ ) OPT for the floored metric.
Proof. By (4.12) and Lemma 4.1, S / ( 32 Q S ) ≤ L ‾ − O P T S/(32Q_S)\leq\overline{L}-\mathrm{OPT} S / ( 32 Q S ) ≤ L − OPT , and by (24) and Lemma 4.7, T ^ / Q Q ≤ L ‾ − O P T \widehat{T}/Q_Q\leq\overline{L}-\mathrm{OPT} T / Q Q ≤ L − OPT ; since L ‾ ≥ L \overline{L}\geq L L ≥ L , we get C ^ ≥ O P T \widehat{C}\geq\mathrm{OPT} C ≥ OPT . By (15) , (4.12), and (14) , Γ ≤ ∑ i a i ( 2 Z i + 2 ε g i ) ≤ 2 S + e 0 L / 1000 \Gamma\leq\sum_i a_i(2Z_i+2\varepsilon g_i)\leq2S+e_0L/1000 Γ ≤ ∑ i a i ( 2 Z i + 2 ε g i ) ≤ 2 S + e 0 L /1000 . If S ≤ e 0 L / 1000 S\leq e_0L/1000 S ≤ e 0 L /1000 and T ^ ≤ σ 0 L / 1000 \widehat{T}\leq\sigma_0L/1000 T ≤ σ 0 L /1000 , then Γ ≤ e 0 L / 250 \Gamma\leq e_0L/250 Γ ≤ e 0 L /250 and, by Lemma 4.7, Λ ≤ 4 L / 1000 + L / 100 < L / 10 \Lambda\leq4L/1000+L/100<L/10 Λ ≤ 4 L /1000 + L /100 < L /10 , so Theorem 3.1 gives L ≤ ( 2 − e 0 / 2 ) O P T L\leq(2-e_0/2)\mathrm{OPT} L ≤ ( 2 − e 0 /2 ) OPT and C ^ ≤ ( 1 + ν ) ( 2 − e 0 / 2 ) O P T ≤ ( 2 − c ∗ ) O P T \widehat{C}\leq(1+\nu)(2-e_0/2)\mathrm{OPT}\leq(2-c_*)\mathrm{OPT} C ≤ ( 1 + ν ) ( 2 − e 0 /2 ) OPT ≤ ( 2 − c ∗ ) OPT . Otherwise one discount is at least c ∗ L c_*L c ∗ L , and C ^ ≤ ( 1 + ν − c ∗ ) L ≤ ( 2 − c ∗ ) O P T \widehat{C}\leq(1+\nu-c_*)L\leq(2-c_*)\mathrm{OPT} C ≤ ( 1 + ν − c ∗ ) L ≤ ( 2 − c ∗ ) OPT since L ≤ 2 O P T L\leq2\mathrm{OPT} L ≤ 2 OPT .
With η w = c ∗ \eta_w=c_* η w = c ∗ , the flooring step turns this into the factor 2 − η 0 2-\eta_0 2 − η 0 with η 0 = c ∗ / 2 \eta_0=c_*/2 η 0 = c ∗ /2 for the original metric. It remains to compute the Z i Z_i Z i within the claimed budget.
Small covers. If g ≤ 100 t / ε g\leq100t/\varepsilon g ≤ 100 t / ε , where t = ⌈ n ⌉ t=\lceil\sqrt{n}\rceil t = ⌈ n ⌉ , we resolve all retention bits, read the distances within N N N and between N N N and all centers, and greedily accept an eligible row whenever it contains two components not yet claimed, claiming all its components. If u u u components are claimed by f f f rows then u ≥ 2 f u\geq2f u ≥ 2 f , the claimed family certifies saving u − f ≥ u / 2 u-f\geq u/2 u − f ≥ u /2 , and every eligible row contains at most one unclaimed component, so Φ ≤ u \Phi\leq u Φ ≤ u . Thus Z = u − f Z=u-f Z = u − f satisfies (4.12), at cost $\widetilde{O}((n+k)t\operatorname{poly}(1/\varepsilon))`.
A reusable sample for deleting large rows
The following lemma selects few rows that cover most of every large row, using a single sample whose records are reused across all rows. It is related to the sparse vector technique [28 ] and to the transfer from stability to generalization in adaptive data analysis [21 , 8 ] .
Lemma 4.10 (Reusable sample). Let R 1 , … , R N r R_1,\ldots,R_{N_r} R 1 , … , R N r be fixed rows in a universe of size g g g , let t ≥ 1 t \ge1 t ≥ 1 , and let 0 < β < 1 / 4 0 < \beta< 1/4 0 < β < 1/4 . A procedure that knows t t t but not g g g uses N s = O ( t log ( t / β ) log ( ( N r + 1 ) t / β ) ) N_s = O(t\log(t/\beta)\log((N_r+1)t/\beta)) N s = O ( t log ( t / β ) log (( N r + 1 ) t / β )) independent uniform samples of the universe and O ( N r N s ) O(N_rN_s) O ( N r N s ) membership tests, and selects at most t t t rows such that, except with probability β \beta β , every selected row contains at least g / t g/t g / t elements outside the previously selected rows, and in the end every row contains at most 4 g / t 4g/t 4 g / t elements outside the selected rows.
Proof. Keep the sample records as live records. Process the rows in order; if row R j R_j R j contains c c c live records, accept it with probability
p ( c ) = 1 1 + exp ( ( H − c ) / b ) , H = 2 N s / t , (26) p(c) = \frac{1}{1+\exp((H-c)/b)}, \qquad H = 2N_s/t, \tag*{(26)} p ( c ) = 1 + exp (( H − c ) / b ) 1 , H = 2 N s / t , (26)
and on acceptance delete its records. Stop after t t t acceptances. Only the acceptance bits are used later.
Stability. Compare samples that differ by one added record. Along a common transcript the record matters only up to the first accepted row containing it. At a row containing it, with q b = e 1 / b − 1 q_b=e^{1/b}-1 q b = e 1/ b − 1 , the acceptance probabilities p ′ p' p ′ and p p p with and without the record satisfy p ′ = ( 1 + q b ) p / ( 1 + q b p ) p'=(1+q_b)p/(1+q_bp) p ′ = ( 1 + q b ) p / ( 1 + q b p ) , so p ′ / p ≤ e 1 / b p'/p\le e^{1/b} p ′ / p ≤ e 1/ b and ( 1 − p ) / ( 1 − p ′ ) = 1 + q b p (1-p)/(1-p')=1+q_bp ( 1 − p ) / ( 1 − p ′ ) = 1 + q b p . In one direction there is at most one acceptance factor; in the other the rejection factors multiply to at most exp ( q b ∑ p ) \exp(q_b\sum p) exp ( q b ∑ p ) , where the sum of the predictable hazards p p p exceeds log ( 1 / δ ′ ) \log(1/\delta') log ( 1/ δ ′ ) only with probability δ ′ \delta' δ ′ because exp ( ∑ p ) ∏ ( 1 − p ) \exp(\sum p)\prod(1-p) exp ( ∑ p ) ∏ ( 1 − p ) stopped at the first relevant acceptance is a supermartingale. Hence the transcript is ( ε b , δ ′ ) (\varepsilon_b,\delta') ( ε b , δ ′ ) -stable with ε b = max { 1 / b , q b log ( 1 / δ ′ ) } \varepsilon_b=\max\{1/b,q_b\log(1/\delta')\} ε b = max { 1/ b , q b log ( 1/ δ ′ )} . A replacement is a removal followed by an addition, so under replacement the transcript is ( 2 ε b , ( 1 + e ε b ) δ ′ ) (2\varepsilon_b,(1+e^{\varepsilon_b})\delta') ( 2 ε b , ( 1 + e ε b ) δ ′ ) -stable.
Transfer. Let X 1 , … , X N s X_1,\ldots,X_{N_s} X 1 , … , X N s be i.i.d. with law ν \nu ν , let the transcript T T T be ( ε , δ 1 ) (\varepsilon,\delta_1) ( ε , δ 1 ) -stable under replacement, and let ν T = N s − 1 ∑ l L ( X l ∣ T ) \nu_T=N_s^{-1}\sum_l\mathcal{L}(X_l\mid T) ν T = N s − 1 ∑ l L ( X l ∣ T ) . Except on transcripts of probability 2 δ 1 / ζ 2\delta_1/\zeta 2 δ 1 / ζ , simultaneously for all f : U → [ 0 , 1 ] f:\mathcal{U}\to[0,1] f : U → [ 0 , 1 ] ,
ν T ( f ) ≤ e ε ν ( f ) + ζ , ν ( f ) ≤ e ε ν T ( f ) + ζ . (27) \nu_T(f)\le e^\varepsilon\nu(f)+\zeta,\qquad\nu(f)\le e^\varepsilon\nu_T(f)+\zeta. \tag*{(27)} ν T ( f ) ≤ e ε ν ( f ) + ζ , ν ( f ) ≤ e ε ν T ( f ) + ζ . (27)
To see this, replace a uniformly random coordinate X I X_I X I by a fresh draw. The joint law of X I X_I X I and the new transcript is ν ⊗ L ( T ) \nu\otimes\mathcal{L}(T) ν ⊗ L ( T ) , and stability compares it with the joint law of X I X_I X I and the old transcript in both directions with additive error δ 1 \delta_1 δ 1 ; Markov's inequality over transcripts gives (4.18).
Margins. Put ρ = 1 / t \rho=1/t ρ = 1/ t , ε = log ( 4 / 3 ) \varepsilon=\log(4/3) ε = log ( 4/3 ) , ζ = ρ / 16 \zeta=\rho/16 ζ = ρ /16 , δ 1 = β ρ / 128 \delta_1=\beta\rho/128 δ 1 = β ρ /128 , and β 0 = β ρ / 64 \beta_0=\beta\rho/64 β 0 = β ρ /64 , and choose b = Θ ( log ( t / β ) ) b=\Theta(\log(t/\beta)) b = Θ ( log ( t / β )) so that the stability parameters hold. If N s ≥ C t b log ( ( N r + 1 ) t / β ) N_s\ge Ct b\log((N_r+1)t/\beta) N s ≥ C t b log (( N r + 1 ) t / β ) , then except with probability β 0 \beta_0 β 0 every accepted row has empirical fresh fraction at least 7 ρ / 4 7\rho/4 7 ρ /4 and every rejected row at most 9 ρ / 4 9\rho/4 9 ρ /4 , since a decision beyond these margins has probability at most exp ( − N s / ( 4 t b ) ) \exp(-N_s/(4tb)) exp ( − N s / ( 4 t b )) . Except on transcripts of probability β 0 / ζ \beta_0/\zeta β 0 / ζ this event has posterior probability at least 1 − ζ 1-\zeta 1 − ζ . The fresh part of a row is determined by the row and the earlier acceptance bits, so its posterior mean is at least ( 1 − ζ ) 7 ρ / 4 (1-\zeta)7\rho/4 ( 1 − ζ ) 7 ρ /4 on acceptance and at most 9 ρ / 4 + ζ 9\rho/4+\zeta 9 ρ /4 + ζ on rejection. By (4.18), the true fresh fractions are at least 303 256 ρ > ρ \frac{303}{256}\rho>\rho 256 303 ρ > ρ and at most 151 48 ρ < 4 ρ \frac{151}{48}\rho<4\rho 48 151 ρ < 4 ρ , respectively. The total exceptional probability is at most ( β 0 + 2 δ 1 ) / ζ = β / 2 (\beta_0+2\delta_1)/\zeta=\beta/2 ( β 0 + 2 δ 1 ) / ζ = β /2 . Accepted fresh sets are disjoint, so after t t t acceptances the universe is covered; otherwise all rows were tested, and later deletions only shrink them. □ \square □
The rule (4.17) can be evaluated exactly with integer arithmetic by choosing b = 1 / ( t log z ∗ ) b=1/(t\log z_*) b = 1/ ( t log z ∗ ) for z ∗ = 1 + 1 / q ∗ z_*=1+1/q_* z ∗ = 1 + 1/ q ∗ with an integer q ∗ = Θ ( t log ( 2 t / β ) ) q_*=\Theta(t\log(2t/\beta)) q ∗ = Θ ( t log ( 2 t / β )) , so that p ( c ) = 1 / ( 1 + z ∗ N s − t c ) p(c)=1/(1+z_*^{N_s-tc}) p ( c ) = 1/ ( 1 + z ∗ N s − t c ) . The sample records must not influence the order of the rows or any other test; in our application all decisions are taken before the selected balls are scanned.
Mutual incidence capping and proof of thm:upper
Fix a scale with g > 50 t / ε g > 50t/\varepsilon g > 50 t / ε , and let ξ = ε / 100 \xi= \varepsilon/100 ξ = ε /100 and B = ⌈ 10 / ε ⌉ B = \lceil10/\varepsilon\rceil B = ⌈ 10/ ε ⌉ . Apply Lemma 4.10 to the rows A v ∩ N A_v \cap N A v ∩ N of all centers, with uniform retained samples, and let D D D be the union of the selected balls A v A_v A v on T T T . This costs O ~ ( n t ) \widetilde{O}(nt) O ( n t ) queries for the pool and O ( k t ) O(kt) O ( k t ) for the balls, and on the pool event
∣ A v ∩ ( N ∖ D ) ∣ ≤ P : = 4 g / t for every center v . (28) \lvert A_v \cap(N \setminus D)\rvert\le P := 4g/t \qquad\text{for every center } v. \tag*{(28)} ∣ A v ∩ ( N ∖ D )∣ ≤ P := 4 g / t for every center v . (28)
Define C 0 C_0 C 0 and U U U as in Section 4.4, with s ≤ t s \le t s ≤ t . For C ∈ U C \in U C ∈ U let L ( C ) L(C) L ( C ) be the first b = ⌈ 100 n / ( ε g − ) ⌉ b = \lceil100n/(\varepsilon g_-)\rceil b = ⌈ 100 n / ( ε g − )⌉ centers, in a fixed public order, within distance 5 a / 8 5a/8 5 a /8 of C C C (all of them if there are fewer); it is found in O ( n B ) O(nB) O ( n B ) time. The multigraph on U U U has a port edge ( v , min { x , x ′ } , max { x , x ′ } ) (v,\min\{x,x'\},\max\{x,x'\}) ( v , min { x , x ′ } , max { x , x ′ }) between C C C and C ′ C' C ′ whenever v v v belongs to L ( C ) ∩ L ( C ′ ) L(C) \cap L(C') L ( C ) ∩ L ( C ′ ) , the row of v v v is eligible, and x ∈ C x \in C x ∈ C , x ′ ∈ C ′ x' \in C' x ′ ∈ C ′ are retained points within 5 a / 8 5a/8 5 a /8 of v v v .
Lemma 4.11 (Mutual capping). Every independent set I ⊆ U I \subseteq U I ⊆ U of this multigraph satisfies Ψ U ≤ ∣ U ∣ − ∣ I ∣ + 2 n / b \Psi_U \le|U| - |I| + 2n/b Ψ U ≤ ∣ U ∣ − ∣ I ∣ + 2 n / b . Consequently every maximal matching, and hence the expected random-greedy matching size μ \mu μ , satisfies
Ψ U / 2 − n / b ≤ μ ≤ Ψ U . (29) \Psi_U/2 - n/b \le\mu\le\Psi_U . \tag*{(29)} Ψ U /2 − n / b ≤ μ ≤ Ψ U . (29)
Proof. An eligible center lies in the lists of at most one member of I I I , since otherwise those two members would be adjacent, and an ineligible center is within 5 a / 8 5a/8 5 a /8 of at most two components. Hence ∑ C ∈ I ∣ L ( C ) ∣ ≤ 2 n \sum_{C\in I} |L(C)| \le2n ∑ C ∈ I ∣ L ( C ) ∣ ≤ 2 n , and at most 2 n / b 2n/b 2 n / b members of I I I have a full list. Each other member kept all its incident centers, so an eligible row covers at most one of them, and every cover of U U U by eligible rows and singletons has at least ∣ I ∣ − 2 n / b |I| - 2n/b ∣ I ∣ − 2 n / b elements. For a maximal matching of size z z z , the unmatched vertices form an independent set, so Ψ U ≤ 2 z + 2 n / b \Psi_U \le2z + 2n/b Ψ U ≤ 2 z + 2 n / b ; conversely, the centers of the matched edges cover 2 z 2z 2 z components with at most z z z eligible rows, so z ≤ Ψ U z \le\Psi_U z ≤ Ψ U . □
The candidate slots of C ∈ U C \in U C ∈ U are the triples ( v , x , y ) (v,x,y) ( v , x , y ) with v ∈ L ( C ) v \in L(C) v ∈ L ( C ) , x ∈ C x \in C x ∈ C within 5 a / 8 5a/8 5 a /8 of v v v , and y ∈ T y \in T y ∈ T ; there are at most k b B kbB k b B of them. A slot is tested by checking y ∉ D y \notin D y ∈ / D and w ( v , y ) ≤ 5 a / 8 w(v,y) \le5a/8 w ( v , y ) ≤ 5 a /8 , then resolving the retention of y y y , exploring its component C ′ C' C ′ , rejecting it if it is large, meets D D D , or equals C C C , preparing L ( C ′ ) L(C') L ( C ′ ) , checking v ∈ L ( C ′ ) v \in L(C') v ∈ L ( C ′ ) , and testing the eligibility of v v v . This recognizes exactly the port edges, and the reverse slot is ( v , y , x ) (v,y,x) ( v , y , x ) . By (28) , at most b B P bBP b B P slots of C C C reach a retained y y y ; each costs O ( n B ) O(nB) O ( n B ) time and O ( B ) O(B) O ( B ) disclosures, and the others cost O ( 1 ) O(1) O ( 1 ) and disclose nothing. Thus, in the notation of Lemma 4.4 , for the account of ordinary work and for the account of disclosures respectively,
d C ≤ b B P , W C = O ( k b B + n B 2 b 2 P ) , C ∗ = O ( n B ) ; W C ′ = O ( B 2 b P ) , C ∗ ′ = O ( B ) . (30) d_C \le bBP,\qquad W_C = O(kbB+nB^2b^2P),\qquad C_* = O(nB);
\qquad W'_C = O(B^2bP),\qquad C'_* = O(B). \tag*{(30)} d C ≤ b B P , W C = O ( k b B + n B 2 b 2 P ) , C ∗ = O ( n B ) ; W C ′ = O ( B 2 b P ) , C ∗ ′ = O ( B ) . (30)
Using the score of Lemma 4.8 with Δ = n / b \Delta= n/b Δ = n / b from (29) , and b l a r g e ≤ g / ( B + 1 ) b_{\mathrm{large}} \le g/(B+1) b large ≤ g / ( B + 1 ) ,
Z ≥ Φ / 2 − ε g 20 − ε g 100 − ε g 25 − 3 ε g 100 ≥ Φ / 2 − ε g , Z \ge\Phi/2 - \frac{\varepsilon g}{20} - \frac{\varepsilon g}{100} - \frac{\varepsilon g}{25} - \frac{3\varepsilon g}{100} \ge\Phi/2 - \varepsilon g, Z ≥ Φ/2 − 20 ε g − 100 ε g − 25 ε g − 100 3 ε g ≥ Φ/2 − ε g ,
so (4.12) holds. Since g > 50 t / ε g > 50t/\varepsilon g > 50 t / ε we have b P = O ( n / ( ε t ) ) bP = O(n/(\varepsilon t)) b P = O ( n / ( εt )) and k b = O ( n 2 / t ) kb = O(n^2/t) k b = O ( n 2 / t ) , so by Lemma 4.4 a matching call from a uniform root costs O ~ ( ( n + n 2 / t ) poly ( 1 / ε ) ) \widetilde{O}((n+n^2/t)\operatorname{poly}(1/\varepsilon)) O (( n + n 2 / t ) poly ( 1/ ε )) time and O ( ( 1 + n / t ) poly ( 1 / ε ) ) O((1+n/t)\operatorname{poly}(1/\varepsilon)) O (( 1 + n / t ) poly ( 1/ ε )) disclosures in expectation on its high-probability event. Only O ( ε − 2 log ( 1 / β ) ) O(\varepsilon^{-2}\log(1/\beta)) O ( ε − 2 log ( 1/ β )) calls are made per scale. Together with the pool, the masks, and the explorations, the work at one scale is
O ~ ( ( n t + n 2 / t ) poly ( 1 / ε ) ) . (31) \widetilde{O}\bigl((nt+n^2/t)\operatorname{poly}(1/\varepsilon)\bigr). \tag*{(31)} O ( ( n t + n 2 / t ) poly ( 1/ ε ) ) . (31)
and the expected number μ A \mu_A μ A of disclosures on the joint good event is O ~ ( ( t + n / t ) poly ( 1 / ε ) ) \widetilde{O}((t+n/t)\operatorname{poly}(1/\varepsilon)) O (( t + n / t ) poly ( 1/ ε )) . By Corollary 4.6 the number of draws is O ~ ( k ( t + n / t ) poly ( 1 / ε ) ) \widetilde{O}(k(t+n/t)\operatorname{poly}(1/\varepsilon)) O ( k ( t + n / t ) poly ( 1/ ε )) except with small probability.
Proof of Theorem 1.1 . Run the estimator (25) of Section 4.4 with t = ⌈ n ⌉ t=\lceil\sqrt{n}\rceil t = ⌈ n ⌉ , using the small-cover routine when g + ≤ 100 t / ε g_{+}\le100t/\varepsilon g + ≤ 100 t / ε and the routine of this subsection otherwise. Choose the per-scale failure probabilities inverse-polylogarithmic so that the cover, calibration, pool, score, MST, and rank-implementation failures sum to less than 1 / 12 1/12 1/12 ; the persistence failure of Lemma 4.7 is below 10 − 4 10^{-4} 1 0 − 4 . Bound the expected work and disclosures of the uncapped algorithm on these events, and impose a global cap of a large constant times these expectations; by Markov’s inequality and Corollary 4.6 the cap is hit with probability less than 1 / 12 1/12 1/12 , and a cap hit is treated as failure. The total failure probability is below 1 / 3 1/3 1/3 , and on success Proposition 4.9 and flooring give the factor 2 − η 0 2-\eta_{0} 2 − η 0 . As ε \varepsilon ε , A A A , B B B , and B Q B_{Q} B Q are polylogarithmic and there are O ( log n ) O(\log n) O ( log n ) scales, (31) gives O ~ ( n 3 / 2 ) \widetilde{O}(n^{3/2}) O ( n 3/2 ) queries and time.
The Lower-Bound Construction
In the rest of the paper, we prove our lower bound, Theorem 1.2 . Recall that it states that for every η ∈ ( 0 , 1 ) \eta\in(0,1) η ∈ ( 0 , 1 ) and every integer s ≥ 1 s\ge1 s ≥ 1 , estimating OPT within a factor 2 − η 2-\eta 2 − η requires Ω ( n b s ) \Omega(n^{b_s}) Ω ( n b s ) queries, where b s = ( 3 s + 7 ) / ( 2 s + 5 ) b_s=(3s+7)/(2s+5) b s = ( 3 s + 7 ) / ( 2 s + 5 ) tends to 3 / 2 3/2 3/2 as s s s grows. As outlined in § 1.3 , we construct two distributions D y \mathcal{D}_y D y and D n \mathcal{D}_n D n over instances such that, with high probability, the optimal cost under D n \mathcal{D}_n D n is more than 2 − η 2-\eta 2 − η times the optimal cost under D y \mathcal{D}_y D y , and we show that every algorithm with few queries sees almost the same distribution of answers under both. This section constructs the two distributions and proves the cost gap. § 6 replaces the metric by a simpler random process, the two-key experiment, and § 7 analyzes this process and proves Theorem 1.2 . This part is independent of the upper bound, and its parameters are defined afresh.
Throughout, fix η ∈ ( 0 , 1 ) \eta\in(0,1) η ∈ ( 0 , 1 ) and an integer s ≥ 1 s\ge1 s ≥ 1 . Here 2 − η 2-\eta 2 − η is the approximation factor, and s s s is the number of moments that the two weight distributions will share. Larger s s s gives a stronger lower bound. All constants below may depend on η \eta η , s s s , but not on the input size or the algorithm.
Each instance has just three types of points: ordinary terminals in tight clusters, special terminals hidden among them, and nonterminal centers. The centers are sampled independently given weights on the special terminals. There is no need to partition the special terminals into groups, to order centers within rows, or to select a hidden block.
Clusters, special terminals, and centers
Choose constants in the following order:
δ = η 100 , H 0 = 64 log 4 δ , d ≥ max { 4 , 2 + 8 H 0 δ } . (32) \delta=\frac{\eta}{100},\qquad H_{0}=64\log\frac{4}{\delta},\qquad d\ge\max\left\{4,\frac{2+8H_{0}}{\delta}\right\}. \tag*{(32)} δ = 100 η , H 0 = 64 log δ 4 , d ≥ max { 4 , δ 2 + 8 H 0 } . (32)
where d d d is an integer. Let t t t be a growing integer, and set
A = t s + 3 , R = ⌈ γ 0 t s + 2 ⌉ , K = A R , m = d A , λ = K − 3 . (33) A=t^{s+3},\qquad R=\lceil\gamma_{0}t^{s+2}\rceil,\qquad K=AR,\qquad m=dA,\qquad\lambda=K^{-3}. \tag*{(33)} A = t s + 3 , R = ⌈ γ 0 t s + 2 ⌉ , K = A R , m = d A , λ = K − 3 . (33)
The small constant γ 0 > 0 \gamma_{0}>0 γ 0 > 0 will be fixed in § 7.3 .
The point set V V V contains three types of points. The terminal set T T T consists of the K K K ordinary terminals , grouped into A A A ordinary clusters of R R R terminals each, and the m m m special terminals . The remaining m R mR m R points are nonterminals, which we call centers. Since m R = d A R = d K mR=dAR=dK m R = d A R = d K ,
n = ∣ V ∣ = K + m + m R = ( d + 1 ) K + m , ∣ T ∣ = K + m = Θ ( n ) . (34) n=|V|=K+m+mR=(d+1)K+m,\qquad|T|=K+m=\Theta(n). \tag*{(34)} n = ∣ V ∣ = K + m + m R = ( d + 1 ) K + m , ∣ T ∣ = K + m = Θ ( n ) . (34)
The point set V V V consists of K + m K+m K + m terminals and m R mR m R centers. The algorithm knows V V V and T T T , so it can tell the centers from the terminals, but it does not know which terminals are special or which ordinary terminals share a cluster. To hide these roles, take a uniformly random bijection π \pi π from T T T to the positions
[ m ] ⊔ ( [ A ] × [ R ] ) . [m] \mathbin{\sqcup} ([A] \times[R]). [ m ] ⊔ ([ A ] × [ R ]) .
Write S = π − 1 ( [ m ] ) S = \pi^{-1}([m]) S = π − 1 ([ m ]) for the special terminals and G a = π − 1 ( { a } × [ R ] ) G_a = \pi^{-1}(\{a\} \times[R]) G a = π − 1 ({ a } × [ R ]) for ordinary cluster a a a . We identify a special terminal with its position in [ m ] [m] [ m ] when no confusion can arise.
The weights. We use two finitely supported distributions ν y , ν n \nu_y,\nu_n ν y , ν n on a fixed interval [ 0 , M ] [0,M] [ 0 , M ] satisfying
E ν y [ W j ] = E ν n [ W j ] ( 0 ≤ j ≤ s ) , E [ W ] = 1 , (35) \mathbb{E}_{\nu_y}[W^j] = \mathbb{E}_{\nu_n}[W^j]\quad(0 \le j \le s), \qquad\mathbb{E}[W] = 1, \tag*{(35)} E ν y [ W j ] = E ν n [ W j ] ( 0 ≤ j ≤ s ) , E [ W ] = 1 , (35)
and
P ν y [ W ≥ 1 / 2 ] = 1 , P ν n [ W > 0 ] ≤ δ . (36) \mathbb{P}_{\nu_y}[W \ge1/2] = 1, \qquad\mathbb{P}_{\nu_n}[W > 0] \le\delta. \tag*{(36)} P ν y [ W ≥ 1/2 ] = 1 , P ν n [ W > 0 ] ≤ δ . (36)
Here W W W denotes a single weight drawn from the distribution in the subscript, and the condition E [ W ] = 1 \mathbb{E}[W] = 1 E [ W ] = 1 holds for both distributions. Lemma 5.5 below constructs such distributions. Independently of π \pi π , give the special positions independent weights W 1 , … , W m W_1,\ldots,W_m W 1 , … , W m , using ν y \nu_y ν y in YES and ν n \nu_n ν n in NO.
The neighbor sets. Given the weights, sample independently for each center c c c a neighbor set nb ( c ) ⊆ [ m ] \operatorname{nb}(c) \subseteq[m] nb ( c ) ⊆ [ m ] . For a d d d -set H H H , put
a W ( H ) = H 0 d R ( m d ) ∏ u ∈ H W u , β = 1 4 ( m 2 ) . a_W(H) = \frac{H_0}{dR\binom{m}{d}} \prod_{u\in H} W_u,\qquad\beta= \frac{1}{4\binom{m}{2}}. a W ( H ) = d R ( d m ) H 0 u ∈ H ∏ W u , β = 4 ( 2 m ) 1 .
The distribution of nb ( c ) \operatorname{nb}(c) nb ( c ) is
{#eq:5.6}
A center with d d d neighbors is a star , and the others are fillers . The probabilities are valid for all weights once R R R is sufficiently large, since ∑ H a W ( H ) ≤ H 0 M d / ( d R ) ≤ 1 / 2 \sum_H a_W(H) \le H_0M^d/(dR) \le1/2 ∑ H a W ( H ) ≤ H 0 M d / ( d R ) ≤ 1/2 .
The singleton and pair probabilities do not depend on the weights. The pair fillers hide the first two contacts with a star. Singleton fillers keep this background present even after many negative answers. The empty set absorbs the remaining mass. In particular, we do not normalize star probabilities by a random sum of weights: their product form will be essential for moment matching.
The metric. For distinct points x , y x,y x , y , define
w ( x , y ) = { λ , x , y ∈ G a for some ordinary cluster G a , 1 , x = c is a center , y ∈ S , π ( y ) ∈ nb ( c ) , or conversely , 2 , otherwise , (37) w(x,y) =
\begin{cases}
\lambda, & x,y \in G_a \text{ for some ordinary cluster } G_a,\\
1, & x=c \text{ is a center},\ y \in S,\ \pi(y) \in\operatorname{nb}(c),\ \text{or conversely},\\
2, & \text{otherwise},
\end{cases} \tag*{(37)} w ( x , y ) = ⎩ ⎨ ⎧ λ , 1 , 2 , x , y ∈ G a for some ordinary cluster G a , x = c is a center , y ∈ S , π ( y ) ∈ nb ( c ) , or conversely , otherwise , (37)
and put w ( x , x ) = 0 w(x,x)=0 w ( x , x ) = 0 . Let D y , D n D_y,D_n D y , D n be the resulting distributions. Their only difference is the weight distribution. All distances are fixed before the algorithm starts.
Lemma 5.1. Every instance in the support of D Y \mathcal{D}_{Y} D Y or D N \mathcal{D}_{N} D N is a metric.
Proof. Only the triangle inequality needs checking. As every distance is at most 2, a violation needs two distances whose sum is less than 2. Two λ \lambda λ -edges lie within one ordinary cluster, where the third distance is also λ \lambda λ . A λ \lambda λ -edge and a unit edge cannot share an endpoint: the former has only ordinary endpoints, and the latter has none.
The gap in the optimal costs
Contracting the ordinary clusters costs λ ( K − A ) \lambda(K-A) λ ( K − A ) and leaves κ = A + m \kappa= A + m κ = A + m terminal pieces : one for each cluster and one for each special terminal. For a set X X X of centers, let H X H_X H X be the bipartite graph on these pieces and X X X whose edges are the unit distances, and let r ( X ) r(X) r ( X ) be the size of a maximum forest in H X H_X H X . The quantity r ( X ) − 2 ∣ X ∣ r(X)-2|X| r ( X ) − 2∣ X ∣ is the profit from using X X X .
Lemma 5.2 (Profit identity). For every constructed instance,
OPT = λ ( K − A ) + 2 ( κ − 1 ) − max X ( r ( X ) − 2 ∣ X ∣ ) . (38) \operatorname{OPT} = \lambda(K-A) + 2(\kappa-1) - \max_{X}\bigl(r(X)-2|X|\bigr). \tag*{(38)} OPT = λ ( K − A ) + 2 ( κ − 1 ) − X max ( r ( X ) − 2∣ X ∣ ) . (38) The maximum may be restricted to sets of stars.
Proof. For fixed X X X , take the λ \lambda λ -spanning trees within clusters and a maximum forest of H X H_X H X . The remaining κ + ∣ X ∣ − r ( X ) \kappa+|X|-r(X) κ + ∣ X ∣ − r ( X ) components can be joined by edges of weight 2: an edge of weight 1 between different components would contradict the maximality of the forest. This gives a tree of weight λ ( K − A ) + 2 ( κ − 1 ) − ( r ( X ) − 2 ∣ X ∣ ) \lambda(K-A)+2(\kappa-1)-(r(X)-2|X|) λ ( K − A ) + 2 ( κ − 1 ) − ( r ( X ) − 2∣ X ∣ ) .
Conversely, a feasible tree using X X X has K + m + ∣ X ∣ − 1 K+m+|X|-1 K + m + ∣ X ∣ − 1 edges. At most K − A K-A K − A are λ \lambda λ -edges, and at most r ( X ) r(X) r ( X ) are unit edges. Writing all other edges as weight 2 gives the same expression as a lower bound. Finally, deleting a filler from a maximum forest removes at most two edges and saves the penalty 2 for that center. It cannot decrease the profit.
Lemma 5.3 (Cost gap). Define
U m = λ ( K − A ) + ( 1 + η / 50 ) m − 2 , V m = λ ( K − A ) + ( 2 − η / 50 ) m − 2. U_m = \lambda(K-A) + (1+\eta/50)m - 2,\qquad V_m = \lambda(K-A) + (2-\eta/50)m - 2. U m = λ ( K − A ) + ( 1 + η /50 ) m − 2 , V m = λ ( K − A ) + ( 2 − η /50 ) m − 2.
With probability 1 − e − Ω ( m ) 1-e^{-\Omega(m)} 1 − e − Ω ( m ) , a YES instance has OPT ≤ U m \operatorname{OPT}\le U_m OPT ≤ U m . With the same probability, a NO instance has OPT ≥ V m \operatorname{OPT}\ge V_m OPT ≥ V m . Moreover,
V m − ( 2 − η ) U m = 47 η + η 2 50 m + ( 1 − η ) ( 2 − λ ( K − A ) ) > 0. (39) V_m-(2-\eta)U_m=\frac{47\eta+\eta^2}{50}m+(1-\eta)(2-\lambda(K-A))>0. \tag*{(39)} V m − ( 2 − η ) U m = 50 47 η + η 2 m + ( 1 − η ) ( 2 − λ ( K − A )) > 0. (39)
Proof. The NO case. Let S + = { u : W u > 0 } S_+=\{u:W_u>0\} S + = { u : W u > 0 } . Every star lies in S + S_+ S + , and ∣ S + ∣ ≤ 2 δ m |S_+|\le2\delta m ∣ S + ∣ ≤ 2 δ m except with probability e − Ω ( m ) e^{-\Omega(m)} e − Ω ( m ) . A component of an incidence forest with v v v special terminals and x x x centers contributes v − x − 1 v-x-1 v − x − 1 to its profit. Summing over components shows that stars have total profit at most ∣ S + ∣ |S_+| ∣ S + ∣ . By Lemma 5.2 ,
OPT ≥ λ ( K − A ) + 2 ( A + m − 1 ) − 2 δ m ≥ V m . \operatorname{OPT}\ge\lambda(K-A)+2(A+m-1)-2\delta m\ge V_m. OPT ≥ λ ( K − A ) + 2 ( A + m − 1 ) − 2 δ m ≥ V m .
The YES case. There are enough stars to cover almost all special terminals, but few enough that their center penalties are negligible. To see this, for 1 ≤ h ≤ m 1\le h\le m 1 ≤ h ≤ m let
E h = ( m h ) − 1 ∑ ∣ H ∣ = h H ⊆ [ m ] ∏ v ∈ H W v E_h=\binom{m}{h}^{-1}\sum_{\substack{|H|=h\\H\subseteq[m]}}\prod_{v\in H}W_v E h = ( h m ) − 1 ∣ H ∣ = h H ⊆ [ m ] ∑ v ∈ H ∏ W v
be the average weight product over all h h h -subsets of [ m ] [m] [ m ] , and for u ∈ [ m ] u\in[m] u ∈ [ m ] let
E d − 1 ( − u ) = ( m − 1 d − 1 ) − 1 ∑ ∣ H ∣ = d − 1 H ⊆ [ m ] u ∉ H ∏ v ∈ H W v E_{d-1}^{(-u)}=\binom{m-1}{d-1}^{-1}\sum_{\substack{|H|=d-1\\H\subseteq[m]\\u\notin H}}\prod_{v\in H}W_v E d − 1 ( − u ) = ( d − 1 m − 1 ) − 1 ∣ H ∣ = d − 1 H ⊆ [ m ] u ∈ / H ∑ v ∈ H ∏ W v
be the same average over the ( d − 1 ) (d-1) ( d − 1 ) -subsets that avoid u u u . With probability 1 − e − Ω ( m ) 1-e^{-\Omega(m)} 1 − e − Ω ( m ) , E d E_d E d and all E d − 1 ( − u ) E_{d-1}^{(-u)} E d − 1 ( − u ) lie in [ 1 / 2 , 2 ] [1/2,2] [ 1/2 , 2 ] . Indeed, let W ‾ \overline{W} W be the mean of all weights, and W ‾ ( − u ) \overline{W}^{(-u)} W ( − u ) the mean of the weights other than u u u . By Hoeffding’s inequality, ∣ W ‾ − 1 ∣ ≤ 1 / ( 8 d ) |\overline{W}-1|\le1/(8d) ∣ W − 1∣ ≤ 1/ ( 8 d ) except with probability e − Ω ( m ) e^{-\Omega(m)} e − Ω ( m ) , and deterministically ∣ W ‾ ( − u ) − W ‾ ∣ ≤ M / ( m − 1 ) |\overline{W}^{(-u)}-\overline{W}|\le M/(m-1) ∣ W ( − u ) − W ∣ ≤ M / ( m − 1 ) for every u u u . An average of products over the h h h -subsets of n n n numbers in [ 0 , M ] [0,M] [ 0 , M ] differs from the h h h -th power of their mean by at most ( h 2 ) M h / n \binom{h}{2}M^h/n ( 2 h ) M h / n , since a uniformly random h h h -tuple of indices repeats an index with probability at most ( h 2 ) / n \binom{h}{2}/n ( 2 h ) / n . So, for large m m m , E d E_d E d and every E d − 1 ( − u ) E_{d-1}^{(-u)} E d − 1 ( − u ) are within 1 / 4 1/4 1/4 of a power x h x^h x h with h ∈ { d − 1 , d } h\in\{d-1,d\} h ∈ { d − 1 , d } and ∣ x − 1 ∣ ≤ 1 / ( 4 d ) |x-1|\le1/(4d) ∣ x − 1∣ ≤ 1/ ( 4 d ) . Such powers lie in [ 3 / 4 , e 1 / 4 ] [3/4,e^{1/4}] [ 3/4 , e 1/4 ] , hence the averages lie in [ 1 / 2 , 2 ] [1/2,2] [ 1/2 , 2 ] .
Condition on such weights. Each of the m R mR m R centers is a star with probability ∑ ∣ H ∣ = d a W ( H ) = H 0 E d / ( d R ) \sum_{|H|=d}aW(H)=H_0E_d/(dR) ∑ ∣ H ∣ = d aW ( H ) = H 0 E d / ( d R ) , independently. So the number Z Z Z of stars is binomial with mean H 0 m E d / d ∈ [ H 0 m / ( 2 d ) , 2 H 0 m / d ] H_0mE_d/d\in[H_0m/(2d),2H_0m/d] H 0 m E d / d ∈ [ H 0 m / ( 2 d ) , 2 H 0 m / d ] , and by a Chernoff bound, except with probability e − Ω ( m ) e^{-\Omega(m)} e − Ω ( m ) ,
H 0 m 4 d ≤ Z ≤ 4 H 0 m d . \frac{H_0m}{4d}\le Z\le\frac{4H_0m}{d}. 4 d H 0 m ≤ Z ≤ d 4 H 0 m .
Given Z Z Z , the star supports are independent, and each is a d d d -set H H H chosen with probability proportional to ∏ v ∈ H W v \prod_{v\in H}W_v ∏ v ∈ H W v . The d d d -sets containing u u u are the sets { u } ∪ H ′ \{u\}\cup H' { u } ∪ H ′ with ∣ H ′ ∣ = d − 1 |H'|=d-1 ∣ H ′ ∣ = d − 1 and u ∉ H ′ u\notin H' u ∈ / H ′ , and ( m − 1 d − 1 ) / ( m d ) = d / m \binom{m-1}{d-1}/\binom{m}{d}=d/m ( d − 1 m − 1 ) / ( d m ) = d / m . So each support contains a fixed terminal u u u with probability
d m W u E d − 1 ( − u ) E d ≥ d 8 m , \frac{d}{m}\frac{W_uE_{d-1}^{(-u)}}{E_d}\ge\frac{d}{8m}, m d E d W u E d − 1 ( − u ) ≥ 8 m d ,
where the inequality uses W u ≥ 1 / 2 W_u\ge1/2 W u ≥ 1/2 (YES), E d − 1 ( − u ) ≥ 1 / 2 E_{d-1}^{(-u)}\ge1/2 E d − 1 ( − u ) ≥ 1/2 and E d ≤ 2 E_d\le2 E d ≤ 2 . Let U U U be the number of special terminals that lie in no star support. Since Z ≥ H 0 m / ( 4 d ) Z\ge H_0m/(4d) Z ≥ H 0 m / ( 4 d ) , each u u u is uncovered with probability at most ( 1 − d / ( 8 m ) ) Z ≤ e − H 0 / 32 (1-d/(8m))^Z \le e^{-H_0/32} ( 1 − d / ( 8 m ) ) Z ≤ e − H 0 /32 . So E [ U ] ≤ m e − H 0 / 32 < δ m / 2 \mathbb{E}[U] \le me^{-H_0/32} < \delta m/2 E [ U ] ≤ m e − H 0 /32 < δ m /2 , by the choice of H 0 H_0 H 0 . Given Z = z Z=z Z = z , U U U is a function of the z z z independent supports, and replacing one support changes U U U by at most d d d . Since z ≤ 4 H 0 m / d z \le4H_0m/d z ≤ 4 H 0 m / d , the bounded-differences inequality gives
P [ U ≥ E [ U ] + δ m 2 ] ≤ exp ( − 2 ( δ m / 2 ) 2 z d 2 ) ≤ exp ( − δ 2 m 8 H 0 d ) = e − Ω ( m ) . \mathbb{P}\left[U \ge\mathbb{E}[U] + \frac{\delta m}{2}\right] \le\exp\left(-\frac{2(\delta m/2)^2}{zd^2}\right) \le\exp\left(-\frac{\delta^2m}{8H_0d}\right) = e^{-\Omega(m)}. P [ U ≥ E [ U ] + 2 δ m ] ≤ exp ( − z d 2 2 ( δ m /2 ) 2 ) ≤ exp ( − 8 H 0 d δ 2 m ) = e − Ω ( m ) .
So at most δ m \delta m δ m terminals are uncovered, except with probability e − Ω ( m ) e^{-\Omega(m)} e − Ω ( m ) .
Take X X X to be all stars. For each covered terminal, pick one unit edge to a star containing it. These edges form a forest in H X H_X H X : every special terminal has degree one in it, so it has no cycle. Hence r ( X ) ≥ ( 1 − δ ) m r(X) \ge(1-\delta)m r ( X ) ≥ ( 1 − δ ) m . Since ∣ X ∣ = Z ≤ 4 H 0 m / d |X|=Z \le4H_0m/d ∣ X ∣ = Z ≤ 4 H 0 m / d , we get r ( X ) − 2 ∣ X ∣ ≥ ( 1 − δ ) m − 8 H 0 m / d r(X)-2|X| \ge(1-\delta)m-8H_0m/d r ( X ) − 2∣ X ∣ ≥ ( 1 − δ ) m − 8 H 0 m / d .
Lemma 5.2 , with κ = A + m \kappa=A+m κ = A + m and A = m / d A=m/d A = m / d , gives
OPT ≤ λ ( K − A ) + 2 ( A + m − 1 ) − ( ( 1 − δ ) m − 8 H 0 m d ) = λ ( K − A ) + ( 1 + δ + 2 + 8 H 0 d ) m − 2. \operatorname{OPT} \le\lambda(K-A)+2(A+m-1)-\left((1-\delta)m-\frac{8H_0m}{d}\right)=\lambda(K-A)+\left(1+\delta+\frac{2+8H_0}{d}\right)m-2. OPT ≤ λ ( K − A ) + 2 ( A + m − 1 ) − ( ( 1 − δ ) m − d 8 H 0 m ) = λ ( K − A ) + ( 1 + δ + d 2 + 8 H 0 ) m − 2.
By (32) , d ≥ ( 2 + 8 H 0 ) / δ d \ge(2+8H_0)/\delta d ≥ ( 2 + 8 H 0 ) / δ , so ( 2 + 8 H 0 ) / d ≤ δ (2+8H_0)/d \le\delta ( 2 + 8 H 0 ) / d ≤ δ , and the coefficient of m m m is at most 1 + 2 δ = 1 + η / 50 1+2\delta=1+\eta/50 1 + 2 δ = 1 + η /50 . Hence OPT ≤ U m \operatorname{OPT} \le U_m OPT ≤ U m . Finally, we verify (39) . Write ℓ = λ ( K − A ) \ell=\lambda(K-A) ℓ = λ ( K − A ) . Then
( 2 − η ) U m = ( 2 − η ) ℓ + ( 2 − η ) ( 1 + η 50 ) m − 2 ( 2 − η ) , ( 2 − η ) ( 1 + η 50 ) = 2 − 48 η 50 − η 2 50 . (2-\eta)U_m=(2-\eta)\ell+(2-\eta)\left(1+\frac{\eta}{50}\right)m-2(2-\eta), \qquad(2-\eta)\left(1+\frac{\eta}{50}\right)=2-\frac{48\eta}{50}-\frac{\eta^2}{50}. ( 2 − η ) U m = ( 2 − η ) ℓ + ( 2 − η ) ( 1 + 50 η ) m − 2 ( 2 − η ) , ( 2 − η ) ( 1 + 50 η ) = 2 − 50 48 η − 50 η 2 .
Subtracting this from V m = ℓ + ( 2 − η / 50 ) m − 2 V_m=\ell+(2-\eta/50)m-2 V m = ℓ + ( 2 − η /50 ) m − 2 , term by term,
V m − ( 2 − η ) U m = 47 η + η 2 50 m + ( 1 − η ) ( 2 − ℓ ) . V_m-(2-\eta)U_m=\frac{47\eta+\eta^2}{50}m+(1-\eta)(2-\ell). V m − ( 2 − η ) U m = 50 47 η + η 2 m + ( 1 − η ) ( 2 − ℓ ) .
Both terms are positive. For the first, η > 0 \eta>0 η > 0 and m > 0 m>0 m > 0 . For the second, 1 − η > 0 1-\eta>0 1 − η > 0 , and ℓ < λ K = K − 2 ≤ 1 \ell<\lambda K=K^{-2}\le1 ℓ < λ K = K − 2 ≤ 1 , so 2 − ℓ > 1 2-\ell>1 2 − ℓ > 1 .
Constructing the two weight distributions
The cost argument needed only the support conditions (36) and the common mean 1 1 1 . The indistinguishability argument will use the additional matching moments. These requirements are compatible for every fixed s s s . We use the following classical identity from Lagrange interpolation. It says that divided differences of order n n n vanish on polynomials of degree less than n n n . We include its short proof for completeness.
Lemma 5.4 (A classical interpolation identity). Let x 0 < x 1 < ⋯ < x n x_0<x_1<\cdots<x_n x 0 < x 1 < ⋯ < x n be real numbers, and let w i = ∏ j ≠ i ( x i − x j ) − 1 w_i=\prod_{j\ne i}(x_i-x_j)^{-1} w i = ∏ j = i ( x i − x j ) − 1 for 0 ≤ i ≤ n 0\le i\le n 0 ≤ i ≤ n . Then
∑ i = 0 n w i x i r = 0 ( 0 ≤ r ≤ n − 1 ) . \sum_{i=0}^{n}w_i x_i^r=0 \qquad(0\le r\le n-1). i = 0 ∑ n w i x i r = 0 ( 0 ≤ r ≤ n − 1 ) .
Proof. For values y 0 , … , y n y_0,\ldots,y_n y 0 , … , y n , the Lagrange interpolation polynomial p ( x ) = ∑ i y i L i ( x ) p(x)=\sum_i y_iL_i(x) p ( x ) = ∑ i y i L i ( x ) , with L i ( x ) = ∏ j ≠ i x − x j x i − x j L_i(x)=\prod_{j\ne i}\frac{x-x_j}{x_i-x_j} L i ( x ) = ∏ j = i x i − x j x − x j , has degree at most n n n , and p ( x k ) = y k p(x_k)=y_k p ( x k ) = y k for every k k k , since L i ( x k ) L_i(x_k) L i ( x k ) is 1 1 1 for k = i k=i k = i and 0 0 0 for k ≠ i k\ne i k = i . It is the only such polynomial: the difference of two of them has degree at most n n n and vanishes at the n + 1 n+1 n + 1 points x k x_k x k , so it is zero. The numerator of L i L_i L i is a product of n n n factors x − x j x-x_j x − x j , so the coefficient of x n x^n x n in L i L_i L i is w i w_i w i , and in p p p it is ∑ i w i y i \sum_i w_i y_i ∑ i w i y i . Now take y i = x i r y_i=x_i^r y i = x i r with r ≤ n − 1 r\le n-1 r ≤ n − 1 . By uniqueness, p ( x ) = x r p(x)=x^r p ( x ) = x r , whose coefficient of x n x^n x n is 0 0 0 .
Lemma 5.5 (Moment-matched weights). For every integer s ≥ 1 s\ge1 s ≥ 1 and δ > 0 \delta>0 δ > 0 , there are finitely supported distributions on [ 0 , M ] [0,M] [ 0 , M ] , for some M = M ( s , δ ) M=M(s,\delta) M = M ( s , δ ) , satisfying (35) and (36) .
Proof. Take an integer B ≥ 3 B \ge3 B ≥ 3 and the s + 2 s+2 s + 2 nodes
x 0 = 0 < x 1 = 1 < x 2 = B < x 3 = B 2 < ⋯ < x s + 1 = B s , x_0=0 < x_1=1 < x_2=B < x_3=B^2 < \cdots< x_{s+1}=B^s, x 0 = 0 < x 1 = 1 < x 2 = B < x 3 = B 2 < ⋯ < x s + 1 = B s ,
that is, x j + 1 = B j x_{j+1}=B^j x j + 1 = B j for 1 ≤ j ≤ s 1 \le j \le s 1 ≤ j ≤ s , and let w i = ∏ j ≠ i ( x i − x j ) − 1 w_i=\prod_{j\ne i}(x_i-x_j)^{-1} w i = ∏ j = i ( x i − x j ) − 1 . By Lemma 5.4 with n = s + 1 n=s+1 n = s + 1 , ∑ i w i x i r = 0 \sum_i w_i x_i^r=0 ∑ i w i x i r = 0 for 0 ≤ r ≤ s 0 \le r \le s 0 ≤ r ≤ s . In ∏ j ≠ i ( x i − x j ) \prod_{j\ne i}(x_i-x_j) ∏ j = i ( x i − x j ) , exactly the s + 1 − i s+1-i s + 1 − i factors with j > i j>i j > i are negative, so the signs of the w i w_i w i alternate.
The two distributions. Let a i = ∣ w i ∣ a_i=|w_i| a i = ∣ w i ∣ . Since the signs alternate, the identity says
∑ i e v e n a i x i r = ∑ i o d d a i x i r ( 0 ≤ r ≤ s ) . \sum_{i\ \mathrm{even}} a_i x_i^r=\sum_{i\ \mathrm{odd}} a_i x_i^r \qquad(0 \le r \le s). i even ∑ a i x i r = i odd ∑ a i x i r ( 0 ≤ r ≤ s ) .
For r = 0 r=0 r = 0 , call the common value T 0 T_0 T 0 . The NO distribution puts mass a i / T 0 a_i/T_0 a i / T 0 on x i x_i x i for each even i i i , and the YES distribution puts mass a i / T 0 a_i/T_0 a i / T 0 on x i x_i x i for each odd i i i . Both have total mass 1, and their moments of every order r ≤ s r \le s r ≤ s agree. Let μ 0 \mu_0 μ 0 be their common mean. Dividing every node by μ 0 \mu_0 μ 0 multiplies the r r r -th moment of both distributions by μ 0 − r \mu_0^{-r} μ 0 − r , so the moments still agree, and the mean becomes 1. This gives (5.4).
Sizes of the masses. The distances from x 0 = 0 x_0=0 x 0 = 0 to the other nodes are 1 , B , … , B s 1,B,\ldots,B^s 1 , B , … , B s , so
a 0 = B − s ( s + 1 ) / 2 . a_0=B^{-s(s+1)/2}. a 0 = B − s ( s + 1 ) /2 .
The distances from x 1 = 1 x_1=1 x 1 = 1 are 1 1 1 and B h − 1 = B h ( 1 − B − h ) B^h-1=B^h(1-B^{-h}) B h − 1 = B h ( 1 − B − h ) for 1 ≤ h ≤ s 1 \le h \le s 1 ≤ h ≤ s , so
a 1 a 0 = ∏ h = 1 s ( 1 − B − h ) − 1 ∈ [ 1 , 1 + 1 B − 2 ] , \frac{a_1}{a_0}=\prod_{h=1}^{s}(1-B^{-h})^{-1}\in\left[1,1+\frac{1}{B-2}\right], a 0 a 1 = h = 1 ∏ s ( 1 − B − h ) − 1 ∈ [ 1 , 1 + B − 2 1 ] ,
using ∏ h ( 1 − B − h ) ≥ 1 − ∑ h ≥ 1 B − h = 1 − 1 B − 1 \prod_h(1-B^{-h})\ge1-\sum_{h\ge1}B^{-h}=1-\frac{1}{B-1} ∏ h ( 1 − B − h ) ≥ 1 − ∑ h ≥ 1 B − h = 1 − B − 1 1 . For 1 ≤ j ≤ s 1 \le j \le s 1 ≤ j ≤ s , the distances from x j + 1 = B j x_{j+1}=B^j x j + 1 = B j are B j B^j B j (to 0), B j ( 1 − B − j ) B^j(1-B^{-j}) B j ( 1 − B − j ) (to 1), B j ( 1 − B h − j ) B^j(1-B^{h-j}) B j ( 1 − B h − j ) (to B h B^h B h with h < j h<j h < j ), and B h ( 1 − B j − h ) B^h(1-B^{j-h}) B h ( 1 − B j − h ) (to B h B^h B h with h > j h>j h > j ). Their powers of B B B multiply to B s ( s + 1 ) / 2 + j ( j + 1 ) / 2 B^{s(s+1)/2+j(j+1)/2} B s ( s + 1 ) /2 + j ( j + 1 ) /2 , and the remaining s s s factors are each at least 1 / 2 1/2 1/2 . Hence
a j + 1 a 0 ≤ 2 s B − j ( j + 1 ) / 2 ( 1 ≤ j ≤ s ) . \frac{a_{j+1}}{a_0}\le2^sB^{-j(j+1)/2}\qquad(1\le j\le s). a 0 a j + 1 ≤ 2 s B − j ( j + 1 ) /2 ( 1 ≤ j ≤ s ) .
The supports. The even indices other than 0 are i = j + 1 i=j+1 i = j + 1 with j ≥ 1 j\ge1 j ≥ 1 , so 1 ≤ T 0 / a 0 ≤ 1 + s 2 s / B 1\le T_0/a_0\le1+s2^s/B 1 ≤ T 0 / a 0 ≤ 1 + s 2 s / B . On the YES side, the odd indices other than 1 are i = j + 1 i=j+1 i = j + 1 with j ≥ 2 j\ge2 j ≥ 2 , and for them a j + 1 x j + 1 / a 0 ≤ 2 s B − j ( j − 1 ) / 2 ≤ 2 s / B a_{j+1}x_{j+1}/a_0\le2^sB^{-j(j-1)/2}\le2^s/B a j + 1 x j + 1 / a 0 ≤ 2 s B − j ( j − 1 ) /2 ≤ 2 s / B . Hence
μ 0 = ∑ i o d d a i x i T 0 = a 1 / a 0 + O s ( B − 1 ) 1 + O s ( B − 1 ) = 1 + O s ( B − 1 ) . \mu_0=\frac{\sum_{i\ \mathrm{odd}}a_ix_i}{T_0}
=\frac{a_1/a_0+O_s(B^{-1})}{1+O_s(B^{-1})}
=1+O_s(B^{-1}). μ 0 = T 0 ∑ i odd a i x i = 1 + O s ( B − 1 ) a 1 / a 0 + O s ( B − 1 ) = 1 + O s ( B − 1 ) .
Now take B B B large, depending on s s s and δ \delta δ . The smallest YES weight is x 1 / μ 0 = 1 / μ 0 ≥ 1 / 2 x_1/\mu_0=1/\mu_0\ge1/2 x 1 / μ 0 = 1/ μ 0 ≥ 1/2 . Rescaling does not move mass off 0, so the NO mass away from zero is ∑ i ≥ 2 e v e n a i / T 0 ≤ s 2 s / B ≤ δ \sum_{i\ge2\ \mathrm{even}}a_i/T_0\le s2^s/B\le\delta ∑ i ≥ 2 even a i / T 0 ≤ s 2 s / B ≤ δ . Finally, the largest node is B s B^s B s , so both distributions live on [ 0 , M ] [0,M] [ 0 , M ] with M = B s / μ 0 M=B^s/\mu_0 M = B s / μ 0 . This gives (5.5). □
Moment matching is a standard tool in statistical lower bounds [35 ] . Here its role is to hide which terminals can participate in stars, while leaving the fraction of such terminals very different in the two worlds.
Replacing the Metric by a Surrogate
Sections 6 and 7 show that an algorithm making few queries cannot tell D Y \mathcal{D}_Y D Y from D N \mathcal{D}_N D N : what it sees has almost the same distribution in both cases. The two input distributions differ only in the weights, and the weights matter only through the stars. In the metric, however, the weights have a small effect on every query between a center and a special terminal. For example, the first such query { c , u } \{c,u\} { c , u } at a center is answered 1 slightly more often when W u W_u W u is large, because a star containing u u u is then more likely. In this section we remove these small effects. We replace each metric distribution D ∙ \mathcal{D}_{\bullet} D ∙ by a simpler random process G ∙ \mathcal{G}_{\bullet} G ∙ that answers the algorithm’s queries adaptively. This process is called the two-key experiment . In it, the answers at a center do not depend on the weights until that center has answered 1 twice. Lemma 6.1 shows that this replacement changes what the algorithm sees only slightly. § 7 then compares G v \mathcal{G}_v G v with G n \mathcal{G}_n G n .
Transcripts. Fix a deterministic algorithm A \mathcal{A} A making at most q q q queries, with no repeated pair. A query is a pair { x , y } \{x,y\} { x , y } of points, and its answer is w ( x , y ) w(x,y) w ( x , y ) . The transcript of a run of A \mathcal{A} A is the sequence ( ( p 1 , z 1 ) , … , ( p k , z k ) ) ((p_1,z_1),\ldots,(p_k,z_k)) (( p 1 , z 1 ) , … , ( p k , z k )) of its queries p i p_i p i and their answers z i z_i z i , in the order in which they are asked. A prefix is the sequence of its first i i i entries, for some 0 ≤ i ≤ k 0 \le i \le k 0 ≤ i ≤ k . We write τ j \tau_j τ j for the prefix of the first j − 1 j-1 j − 1 entries, the prefix before query j j j . The next pair asked by A \mathcal{A} A is determined by the preceding answers, and its output is determined by its transcript. For a distribution D \mathcal{D} D , write trans ( A , D ) \operatorname{trans}(\mathcal{A},\mathcal{D}) trans ( A , D ) for the distribution of the transcript when the input is drawn from D \mathcal{D} D .
Experiments. An experiment is a randomized procedure that answers the queries of A \mathcal{A} A one at a time. Unlike an input, it is not a fixed metric: its answers may depend on the order in which the queries are asked. We run A \mathcal{A} A against an experiment G \mathcal{G} G exactly as against an input. This defines the transcript and its distribution trans ( A , G ) \operatorname{trans}(\mathcal{A},\mathcal{G}) trans ( A , G ) . Experiments are used only in the analysis, and the hard inputs remain the metrics of § 5 .
The two-key experiment
In the two-key experiment, each center c c c draws a filler F c F_c F c , a random set of at most two special terminals, without using the weights. If F c F_c F c is a pair, its two members are the keys of c c c , and c c c may also hold a hidden potential star H c H_c H c : a d d d -set that contains both keys and is drawn using the weights. The center answers like its filler until both keys have been queried at c c c . Only then can its star affect the answers (Figure 5 ). Thus each star stays locked until both of its keys have been queried at its center. Formally, the experiment is defined as follows.
Keep the same random bijection and weights. Independently for every center c c c , sample a weight-independent filler F c F_c F c with
P [ F c = ∅ ] = 1 2 , P [ F c = { u } ] = 1 4 m , P [ F c = { u , v } ] = β . \mathbb{P}[F_c=\varnothing]=\frac{1}{2},\qquad\mathbb{P}[F_c=\{u\}]=\frac{1}{4m},\qquad\mathbb{P}[F_c=\{u,v\}]=\beta. P [ F c = ∅ ] = 2 1 , P [ F c = { u }] = 4 m 1 , P [ F c = { u , v }] = β .
On singletons and pairs these are the probabilities of (5.6). Unlike (5.6), the filler puts probability 1 2 \frac{1}{2} 2 1 on ∅ \varnothing ∅ instead of 1 2 − ∑ H a W ( H ) \frac{1}{2}-\sum_H a_W(H) 2 1 − ∑ H a W ( H ) , and it does not depend on the weights. If F c = e F_c=e F c = e is a pair, its two members are the keys of c c c , and we sample a potential star H c H_c H c by
P [ H c = H ∣ F c = e , W ] = a W ( H ) β for every d -set H ⊇ e , (40) \mathbb{P}[H_c=H\mid F_c=e,W]=\frac{a_W(H)}{\beta}\qquad\text{for every }d\text{-set }H\supseteq e, \tag*{(40)} P [ H c = H ∣ F c = e , W ] = β a W ( H ) for every d -set H ⊇ e , (40)
and leave it absent with the remaining probability. For other fillers H c H_c H c is always absent. We represent absence by H c = ∅ H_c=\varnothing H c = ∅ . Potential stars are independent given the weights and fillers.
Until both keys have been queried at c c c , answer a query { c , u } \{c,u\} { c , u } by 1 exactly when u ∈ F c u\in F_c u ∈ F c . The query of the second key is still answered by this rule (so it is answered 1) and then activates c c c . Let B c B_c B c be the special terminals queried at c c c before activation that are not keys. They were all answered 2. After activation, answer by 1 exactly when
u ∈ F c or ( H c exists , H c ∩ B c = ∅ , u ∈ H c ) . u\in F_c\quad\text{or}\quad(H_c\text{ exists},\ H_c\cap B_c=\varnothing,\ u\in H_c). u ∈ F c or ( H c exists , H c ∩ B c = ∅ , u ∈ H c ) .
A potential star meeting B c B_c B c is discarded : it never affects answers, but we keep it as a hidden object. This avoids contradicting an earlier negative answer. All queries other than center–special-terminal queries use (37) . Denote the two experiments by G y , G n \mathcal{G}_y,\mathcal{G}_n G y , G n . In particular, the answers at c c c depend on the weights only after c c c is activated, that is, after c c c has answered 1 to both of its keys.
The following bound both validates (40) for large R R R and records its useful sparsity. For any prescribed set P P P of p ≤ d p\le d p ≤ d special positions,
∑ H ⊇ P a W ( H ) ≤ H 0 M d d R ( m − p d − p ) ( m d ) ≤ H 0 M d d p − 1 R m p = O ( 1 R m p ) . (41) \sum_{H\supseteq P}a_W(H)\le\frac{H_0M^d}{dR}\frac{\binom{m-p}{d-p}}{\binom{m}{d}}\le\frac{H_0M^d d^{p-1}}{Rm^p}=O\left(\frac{1}{Rm^p}\right). \tag*{(41)} H ⊇ P ∑ a W ( H ) ≤ d R H 0 M d ( d m ) ( d − p m − p ) ≤ R m p H 0 M d d p − 1 = O ( R m p 1 ) . (41)
Indeed, every weight is at most M M M , so each term is at most H 0 M d / ( d R ( m d ) ) H_0M^d/(dR\binom{m}{d}) H 0 M d / ( d R ( d m ) ) , there are ( m − p d − p ) \binom{m-p}{d-p} ( d − p m − p ) sets H ⊇ P H\supseteq P H ⊇ P , and ( m − p d − p ) / ( m d ) = ∏ i < p d − i m − i ≤ ( d / m ) p \binom{m-p}{d-p}/\binom{m}{d}=\prod_{i<p}\frac{d-i}{m-i}\le(d/m)^p ( d − p m − p ) / ( d m ) = ∏ i < p m − i d − i ≤ ( d / m ) p . In particular, for every pair e e e and every u ∉ e u\notin e u ∈ / e ,
P [ H c exists ∣ F c = e , W ] ≤ B 0 R , P [ u ∈ H c ∣ F c = e , W ] ≤ B 1 R m , (42) \mathbb{P}[H_c\text{ exists}\mid F_c=e,W]\le\frac{B_0}{R},\qquad\mathbb{P}[u\in H_c\mid F_c=e,W]\le\frac{B_1}{Rm}, \tag*{(42)} P [ H c exists ∣ F c = e , W ] ≤ R B 0 , P [ u ∈ H c ∣ F c = e , W ] ≤ R m B 1 , (42)
where B 0 = 2 ( d − 1 ) H 0 M d B_0=2(d-1)H_0M^d B 0 = 2 ( d − 1 ) H 0 M d and B 1 = 2 ( d − 2 ) B 0 B_1=2(d-2)B_0 B 1 = 2 ( d − 2 ) B 0 . To see this, divide (41) by β = 1 / ( 2 m ( m − 1 ) ) \beta=1/(2m(m-1)) β = 1/ ( 2 m ( m − 1 )) : for P = e P=e P = e it gives B 0 / R B_0/R B 0 / R , and for P = e ∪ { u } P=e\cup\{u\} P = e ∪ { u } it gives ( d − 2 ) B 0 / ( R ( m − 2 ) ) ≤ B 1 / ( R m ) (d-2)B_0/(R(m-2))\le B_1/(Rm) ( d − 2 ) B 0 / ( R ( m − 2 )) ≤ B 1 / ( R m ) , since m ≥ 4 m\ge4 m ≥ 4 . These estimates hold for every realization of the weights. We henceforth take R ≥ 2 B 0 R\ge2B_0 R ≥ 2 B 0 . Then the probabilities in (40) add up to at most B 0 / R ≤ 1 / 2 B_0/R\le1/2 B 0 / R ≤ 1/2 , so (40) is a valid distribution, and H c H_c H c is absent with probability at least 1 / 2 1/2 1/2 .
Comparison with the metric
Lemma 6.1 (Comparison with the metric). For either ∙ ∈ { y , n } \bullet\in\{y,n\} ∙ ∈ { y , n } ,
d T V ( trans ( A , D ∙ ) , trans ( A , G ∙ ) ) ≤ C ( q m R 2 + q m R 2 ) . (43) d_{\mathrm{TV}}\bigl(\operatorname{trans}(\mathcal{A},\mathcal{D}_{\bullet}),\operatorname{trans}(\mathcal{A},\mathcal{G}_{\bullet})\bigr)\le C\left(\sqrt{\frac{q}{mR^2}}+\frac{q}{mR^2}\right). \tag*{(43)} d TV ( trans ( A , D ∙ ) , trans ( A , G ∙ ) ) ≤ C ( m R 2 q + m R 2 q ) . (43)
The rest of this subsection proves Lemma 6.1. Fix ∙ \bullet ∙ , and let C 0 = d H 0 M d C_0=dH_0M^d C 0 = d H 0 M d , so that B 0 ≤ 2 C 0 B_0\le2C_0 B 0 ≤ 2 C 0 . We assume m ≥ 8 m\ge8 m ≥ 8 and R ≥ 4 C 0 R\ge4C_0 R ≥ 4 C 0 (so that R ≥ 2 B 0 R\ge2B_0 R ≥ 2 B 0 ). Both hold for all large t t t , since C 0 C_0 C 0 depends only on η \eta η and s s s . Plan. We use a third experiment, the hybrid Hyb, defined below. At each center it answers like G ∙ \mathcal{G}_{\bullet} G ∙ until the center has answered 1 twice, and like D ∙ \mathcal{D}_{\bullet} D ∙ afterwards. For fixed π \pi π and W W W (see the reductions below), we show that
d T V ( trans ( A , D ∙ ) , trans ( A , H y b ) ) ≤ 32 C 0 q m R 2 , (44) d_{\mathrm{TV}}(\operatorname{trans}(\mathcal{A},\mathcal{D}_{\bullet}),\operatorname{trans}(\mathcal{A},\mathrm{Hyb})) \le32C_{0}\sqrt{\frac{q}{mR^{2}}}, \tag*{(44)} d TV ( trans ( A , D ∙ ) , trans ( A , Hyb )) ≤ 32 C 0 m R 2 q , (44)
d T V ( trans ( A , H y b ) , trans ( A , G ∙ ) ) ≤ 16 C 0 2 q m R 2 . (45) d_{\mathrm{TV}}(\operatorname{trans}(\mathcal{A},\mathrm{Hyb}),\operatorname{trans}(\mathcal{A},\mathcal{G}_{\bullet})) \le\frac{16C_{0}^{2}q}{mR^{2}}. \tag*{(45)} d TV ( trans ( A , Hyb ) , trans ( A , G ∙ )) ≤ m R 2 16 C 0 2 q . (45)
The triangle inequality then gives (43) . The two bounds need different arguments. Before a center’s second answer 1, D ∙ \mathcal{D}_{\bullet} D ∙ and Hyb answer each query with slightly different probabilities, and we add up these small differences using relative entropy. The hybrid and G ∙ \mathcal{G}_{\bullet} G ∙ differ only in the set from which a center answers after its second answer 1. These sets differ with probability O ( 1 / R 2 ) O(1/R^{2}) O ( 1/ R 2 ) at each center, and in expectation only O ( q / m ) O(q/m) O ( q / m ) centers receive a second answer 1. Here we use a coupling.
Reductions. First, we may assume that every transcript has exactly q q q entries: if A \mathcal{A} A stops after k < q k<q k < q queries, we append q − k q-k q − k copies of a symbol ⊥ \perp ⊥ . The padded transcript and the original one determine each other, so by Lemma 2.2 (b) this changes none of the distances in (43) –(45) .
Second, we may fix π \pi π and W W W . The pair ( π , W ) (\pi,W) ( π , W ) has the same distribution under D ∙ \mathcal{D}_{\bullet} D ∙ and G ∙ \mathcal{G}_{\bullet} G ∙ , so by Lemma 2.2 (c) it suffices to prove (43) conditionally on every value of ( π , W ) (\pi,W) ( π , W ) , with a bound that does not depend on this value. From now on, π \pi π and W W W are fixed, all probabilities are conditional on them, and the hybrid is defined in this setting.
Third, by (37) , the fixed π \pi π determines the answer to every query between two terminals (λ \lambda λ or 2), between two centers (2), and between a center and an ordinary terminal (2), in D ∙ \mathcal{D}_{\bullet} D ∙ , Hyb and G ∙ \mathcal{G}_{\bullet} G ∙ alike. Only a query { c , u } \{c,u\} { c , u } between a center c c c and a special terminal u u u has a random answer, 1 or 2.
Answer rules and local histories. Since A \mathcal{A} A is deterministic, the query that follows a prefix τ \tau τ is determined by τ \tau τ , and we call it the next query . The answer rule of an experiment gives, for every prefix τ \tau τ of positive probability, the probability of each answer to the next query given τ \tau τ . The probability of a transcript is the product of the probabilities of its answers given the preceding prefixes, so the answer rule determines the transcript distribution.
For a prefix τ \tau τ and a center c c c , the local history of c c c in τ \tau τ is the pair ( P , B ) (P,B) ( P , B ) , where P P P is the set of special terminals u u u such that { c , u } \{c,u\} { c , u } was answered 1 in τ \tau τ , and B B B is the set of those answered 2. If ∣ P ∣ ≥ 2 |P|\ge2 ∣ P ∣ ≥ 2 , let e e e be the set of the first two terminals answered 1 at c c c , and B ∘ B^{\circ} B ∘ the set of terminals answered 2 at c c c before the second 1. Thus e ⊆ P e\subseteq P e ⊆ P and B ∘ ⊆ B B^{\circ}\subseteq B B ∘ ⊆ B , and e e e and B ∘ B^{\circ} B ∘ do not change after the second answer 1 at $c.
Three functions. We use two distributions on subsets N ⊆ [ m ] N\subseteq[m] N ⊆ [ m ] :
the distribution Φ \Phi Φ of a filler: Φ ( ∅ ) = 1 2 \Phi(\varnothing)=\frac{1}{2} Φ ( ∅ ) = 2 1 , Φ ( { v } ) = 1 4 m \Phi(\{v\})=\frac{1}{4m} Φ ({ v }) = 4 m 1 for each v v v , Φ ( { v , v ′ } ) = β \Phi(\{v,v'\})=\beta Φ ({ v , v ′ }) = β for each pair, and Φ ( N ) = 0 \Phi(N)=0 Φ ( N ) = 0 for every other set N N N ;
the distribution μ W \mu_W μ W of a neighbor set, given by (5.6): equal to Φ \Phi Φ on singletons and pairs, μ W ( H ) = a W ( H ) \mu_W(H)=a_W(H) μ W ( H ) = a W ( H ) for each d d d -set H H H , μ W ( ∅ ) = 1 2 − σ \mu_W(\varnothing)=\frac{1}{2}-\sigma μ W ( ∅ ) = 2 1 − σ , where σ = ∑ ∣ H ∣ = d a W ( H ) \sigma=\sum_{|H|=d}a_W(H) σ = ∑ ∣ H ∣ = d a W ( H ) , and μ W ( N ) = 0 \mu_W(N)=0 μ W ( N ) = 0 for every other set N N N .
For a distribution ψ \psi ψ on subsets of [ m ] [m] [ m ] and disjoint P , B ⊆ [ m ] P,B\subseteq[m] P , B ⊆ [ m ] , let
ψ ( P , B ) = ∑ N ⊇ P , N ∩ B = ∅ ψ ( N ) \psi(P,B)=\sum_{\substack{N\supseteq P,\,N\cap B=\varnothing}}\psi(N) ψ ( P , B ) = N ⊇ P , N ∩ B = ∅ ∑ ψ ( N )
be the probability that a set drawn from ψ \psi ψ contains P P P and avoids B B B . If ( P , B ) (P,B) ( P , B ) is the local history of a center whose neighbor set is drawn from ψ \psi ψ , then ψ ( P , B ) \psi(P,B) ψ ( P , B ) is the probability that this center gives exactly the recorded answers. Let
f ( P , B ) = Φ ( P , B ) , g ( P , B ) = μ W ( P , B ) , s ( P , B ) = ∑ ∣ H ∣ = d , H ⊇ P H ∩ B = ∅ a W ( H ) . f(P,B)=\Phi(P,B),\qquad g(P,B)=\mu_W(P,B),\qquad s(P,B)=\sum_{\substack{|H|=d,\ H\supseteq P\\ H\cap B=\varnothing}}a_W(H). f ( P , B ) = Φ ( P , B ) , g ( P , B ) = μ W ( P , B ) , s ( P , B ) = ∣ H ∣ = d , H ⊇ P H ∩ B = ∅ ∑ a W ( H ) .
The next claim says that g g g is f f f plus a small correction.
Claim 1. Let P , B ⊆ [ m ] P,B\subseteq[m] P , B ⊆ [ m ] be disjoint, let p = ∣ P ∣ p=|P| p = ∣ P ∣ and b = ∣ B ∣ b=|B| b = ∣ B ∣ , and let u , v ∉ B u,v\notin B u , v ∈ / B be distinct.
(i) g ( P , B ) = f ( P , B ) + s ( P , B ) − σ ⋅ 1 [ P = ∅ ] g(P,B)=f(P,B)+s(P,B)-\sigma\cdot\mathbf{1}[P=\varnothing] g ( P , B ) = f ( P , B ) + s ( P , B ) − σ ⋅ 1 [ P = ∅ ] .
(ii) f ( ∅ , B ) = 1 2 + m − b 4 m + β ( m − b 2 ) ∈ [ 1 2 , 1 ] f(\varnothing,B)=\frac{1}{2}+\frac{m-b}{4m}+\beta\binom{m-b}{2}\in\left[\frac{1}{2},1\right] f ( ∅ , B ) = 2 1 + 4 m m − b + β ( 2 m − b ) ∈ [ 2 1 , 1 ] , f ( { v } , B ) = 1 4 m + β ( m − b − 1 ) ∈ [ 1 4 m , 3 4 m ] f(\{v\},B)=\frac{1}{4m}+\beta(m-b-1)\in\left[\frac{1}{4m},\frac{3}{4m}\right] f ({ v } , B ) = 4 m 1 + β ( m − b − 1 ) ∈ [ 4 m 1 , 4 m 3 ] , and f ( { v , u } , B ) = β f(\{v,u\},B)=\beta f ({ v , u } , B ) = β .
(iii) If p ≤ 2 p\leq2 p ≤ 2 , then 0 ≤ s ( P , B ) ≤ C 0 / ( R m p ) 0\leq s(P,B)\leq C_0/(Rm^p) 0 ≤ s ( P , B ) ≤ C 0 / ( R m p ) . Moreover, 0 ≤ s ( ∅ , B ) ≤ σ ≤ C 0 / R ≤ 1 4 0\leq s(\varnothing,B)\leq\sigma\leq C_0/R\leq\frac{1}{4} 0 ≤ s ( ∅ , B ) ≤ σ ≤ C 0 / R ≤ 4 1 .
Proof of Claim 1 . (i) The two distributions agree on singletons and pairs. On ∅ \varnothing ∅ , μ W \mu_W μ W has σ \sigma σ less than Φ \Phi Φ , and ∅ \varnothing ∅ contains P P P only if P = ∅ P=\varnothing P = ∅ . Finally, μ W \mu_W μ W also gives probability to the d d d -sets, and those that contain P P P and avoid B B B have total probability s ( P , B ) s(P,B) s ( P , B ) .
(ii) A set of positive Φ \Phi Φ -probability that avoids B B B is ∅ \varnothing ∅ , one of the m − b m-b m − b singletons outside B B B , or one of the ( m − b 2 ) \binom{m-b}{2} ( 2 m − b ) pairs outside B B B . Such a set that contains v v v is { v } \{v\} { v } or one of the m − b − 1 m-b-1 m − b − 1 pairs { v , w } \{v,w\} { v , w } with w ∉ B ∪ { v } w\notin B\cup\{v\} w ∈ / B ∪ { v } , and the only such set that contains both v v v and u u u is { v , u } \{v,u\} { v , u } . The ranges follow from β ( m − 1 ) = 1 2 m \beta(m-1)=\frac{1}{2m} β ( m − 1 ) = 2 m 1 and β ( m 2 ) = 1 4 \beta\binom{m}{2}=\frac{1}{4} β ( 2 m ) = 4 1 .
(iii) By (41) , s ( P , B ) ≤ ∑ H ⊇ P a W ( H ) ≤ H 0 M d d p − 1 / ( R m p ) ≤ C 0 / ( R m p ) s(P,B)\leq\sum_{H\supseteq P}a_W(H)\leq H_0M^d d^{p-1}/(Rm^p)\leq C_0/(Rm^p) s ( P , B ) ≤ ∑ H ⊇ P a W ( H ) ≤ H 0 M d d p − 1 / ( R m p ) ≤ C 0 / ( R m p ) for p ≤ 2 p\leq2 p ≤ 2 . Also s ( ∅ , B ) ≤ s ( ∅ , ∅ ) = σ s(\varnothing,B)\leq s(\varnothing,\varnothing)=\sigma s ( ∅ , B ) ≤ s ( ∅ , ∅ ) = σ , the case p = 0 p=0 p = 0 gives σ ≤ C 0 / R \sigma\leq C_0/R σ ≤ C 0 / R , and C 0 / R ≤ 1 4 C_0/R\leq\frac{1}{4} C 0 / R ≤ 4 1 since R ≥ 4 C 0 R\geq4C_0 R ≥ 4 C 0 . ◁
The metric and the experiment. For a pair e e e and a set B ∘ B^\circ B ∘ disjoint from e e e , let H \mathcal{H} H be the family of d d d -sets H ⊇ e H\supseteq e H ⊇ e with H ∩ B ∘ = ∅ H\cap B^\circ=\varnothing H ∩ B ∘ = ∅ , and let a = ∑ H ∈ H a W ( H ) a=\sum_{H\in\mathcal{H}}a_W(H) a = ∑ H ∈ H a W ( H ) . Define two distributions on subsets of [ m ] [m] [ m ] , both supported on { e } ∪ H \{e\}\cup\mathcal{H} { e } ∪ H :
ψ h y b ( e ) = β β + a , ψ h y b ( H ) = a W ( H ) β + a ; ψ e x p ( e ) = 1 − a β , ψ e x p ( H ) = a W ( H ) β ( H ∈ H ) . \psi^{\mathrm{hyb}}(e)=\frac{\beta}{\beta+a},\qquad\psi^{\mathrm{hyb}}(H)=\frac{a_W(H)}{\beta+a};
\qquad
\psi^{\mathrm{exp}}(e)=1-\frac{a}{\beta},\qquad\psi^{\mathrm{exp}}(H)=\frac{a_W(H)}{\beta}\quad(H\in\mathcal{H}). ψ hyb ( e ) = β + a β , ψ hyb ( H ) = β + a a W ( H ) ; ψ exp ( e ) = 1 − β a , ψ exp ( H ) = β a W ( H ) ( H ∈ H ) .
By (42) , a ≤ B 0 β / R ≤ β / 2 a\leq B_0\beta/R\leq\beta/2 a ≤ B 0 β / R ≤ β /2 , so both are distributions. Both describe the set from which a center c c c answers after its second answer 1, given its answers until then. In D ∙ \mathcal{D}_\bullet D ∙ this set is nb ( c ) \operatorname{nb}(c) nb ( c ) , whose conditional distribution is ψ h y b \psi^{\mathrm{hyb}} ψ hyb (Claim 3 below), and G ∙ \mathcal{G}_\bullet G ∙ draws it from ψ e x p \psi^{\mathrm{exp}} ψ exp (Claim 2 ).
From now on, let τ \tau τ be a prefix of positive probability in the experiment considered, let { c , u } \{c,u\} { c , u } be the next query, with u u u special, and let ( P , B ) (P,B) ( P , B ) be the local history of c c c in τ \tau τ . Then u ∉ P ∪ B u\notin P\cup B u ∈ / P ∪ B , since no pair is queried twice. If ∣ P ∣ ≥ 2 |P|\geq2 ∣ P ∣ ≥ 2 , we use ψ h y b \psi^{\mathrm{hyb}} ψ hyb and ψ e x p \psi^{\mathrm{exp}} ψ exp with the e e e and B ∘ B^\circ B ∘ of c c c . The next claim writes the answer rules of D ∙ \mathcal{D}_\bullet D ∙ and G ∙ \mathcal{G}_\bullet G ∙ in the same form: a ratio of probabilities of two local histories.
Claim 2. Given τ \tau τ , the probability that the answer to { c , u } \{c,u\} { c , u } is 1 is:
(i) in D ∙ \mathcal{D}_\bullet D ∙ : x = g ( P ∪ { u } , B ) g ( P , B ) x=\frac{g(P\cup\{u\},B)}{g(P,B)} x = g ( P , B ) g ( P ∪ { u } , B ) ;
(ii) in G ∙ \mathcal{G}_\bullet G ∙ : x 0 = f ( P ∪ { u } , B ) f ( P , B ) x_0=\frac{f(P\cup\{u\},B)}{f(P,B)} x 0 = f ( P , B ) f ( P ∪ { u } , B ) if ∣ P ∣ ≤ 1 |P|\leq1 ∣ P ∣ ≤ 1 , and ψ e x p ( P ∪ { u } , B ) ψ e x p ( P , B ) \frac{\psi^{\mathrm{exp}}(P\cup\{u\},B)}{\psi^{\mathrm{exp}}(P,B)} ψ exp ( P , B ) ψ exp ( P ∪ { u } , B ) if ∣ P ∣ ≥ 2 |P|\geq2 ∣ P ∣ ≥ 2 .
Proof of Claim 2 . Since π \pi π and W W W are fixed, the random data of different centers are independent: the neighbor sets in D ∙ \mathcal{D}_\bullet D ∙ , and the pairs ( F c ′ , H c ) (F'_c,H_c) ( F c ′ , H c ) in G ∙ \mathcal{G}_\bullet G ∙ . As A \mathcal{A} A is deterministic and all other answers are fixed, τ \tau τ occurs exactly when the data of every center produce the answers recorded at that center. Hence, given τ \tau τ , the data of c c c have their prior distribution conditioned on producing the answers recorded at c c c , and the probability of the answer 1 depends only on these answers.
(i) In D ∙ \mathbf{D}_{\bullet} D ∙ , the center c c c answers 1 exactly for the members of nb ( c ) \operatorname{nb}(c) nb ( c ) , which has distribution μ W \mu_W μ W . So the probability that the answer is 1 is the probability that nb ( c ) \operatorname{nb}(c) nb ( c ) contains P ∪ { u } P \cup\{u\} P ∪ { u } and avoids B B B , divided by the probability that nb ( c ) \operatorname{nb}(c) nb ( c ) contains P P P and avoids B B B . This is x x x .
(ii) In G ∙ \mathbf{G}_{\bullet} G ∙ , the center c c c is activated exactly at its second answer 1, and up to and including that query, it answers 1 exactly for the members of F c F_c F c , which has distribution Φ \Phi Φ . If ∣ P ∣ ≤ 1 |P| \le1 ∣ P ∣ ≤ 1 , then c c c is not activated, and the query { c , u } \{c,u\} { c , u } is also answered by membership in F c F_c F c , even if it activates c c c . As in (i), this gives x 0 x_0 x 0 .
If ∣ P ∣ ≥ 2 |P| \ge2 ∣ P ∣ ≥ 2 , the answers at c c c before its second 1 are produced exactly when F c ⊇ e F_c \supseteq e F c ⊇ e and F c ∩ B ∘ = ∅ F_c \cap B^\circ= \varnothing F c ∩ B ∘ = ∅ , that is, when F c = e F_c=e F c = e (as ∣ F c ∣ ≤ 2 |F_c| \le2 ∣ F c ∣ ≤ 2 ). Then c c c was activated with B c = B ∘ B_c=B^\circ B c = B ∘ , and afterwards it answers 1 exactly for the members of the set N c N_c N c that equals H c H_c H c if H c H_c H c exists and avoids B ∘ B^\circ B ∘ , and equals e e e otherwise. Given F c = e F_c=e F c = e , (40) shows that N c N_c N c has distribution ψ e x p \psi^{\mathrm{exp}} ψ exp . Every set in its support contains e e e and avoids B ∘ B^\circ B ∘ , so the answers after activation are produced exactly when N c N_c N c contains P P P and avoids B B B . Hence the probability that the answer is 1 is Φ ( e ) ψ e x p ( P ∪ { u } , B ) \Phi(e)\psi^{\mathrm{exp}}(P \cup\{u\},B) Φ ( e ) ψ exp ( P ∪ { u } , B ) divided by Φ ( e ) ψ e x p ( P , B ) \Phi(e)\psi^{\mathrm{exp}}(P,B) Φ ( e ) ψ exp ( P , B ) .
The hybrid. Let Hyb be the experiment whose answer rule gives every query with a fixed answer that answer, and, in the situation above,
P H y b [ the answer is 1 ∣ τ ] = { x 0 = f ( P ∪ { u } , B ) f ( P , B ) , ∣ P ∣ ≤ 1 , ψ h y b ( P ∪ { u } , B ) ψ h y b ( P , B ) , ∣ P ∣ ≥ 2. \mathbb{P}_{\mathrm{Hyb}}[\text{the answer is 1}\mid\tau]=
\begin{cases}
x_0=\dfrac{f(P\cup\{u\},B)}{f(P,B)}, & |P|\le1,\\
\dfrac{\psi^{\mathrm{hyb}}(P\cup\{u\},B)}{\psi^{\mathrm{hyb}}(P,B)}, & |P|\ge2.
\end{cases} P Hyb [ the answer is 1 ∣ τ ] = ⎩ ⎨ ⎧ x 0 = f ( P , B ) f ( P ∪ { u } , B ) , ψ hyb ( P , B ) ψ hyb ( P ∪ { u } , B ) , ∣ P ∣ ≤ 1 , ∣ P ∣ ≥ 2.
The next claim shows that, at a center that has answered 1 at least twice, Hyb answers like D ∙ \mathbf{D}_{\bullet} D ∙ .
Claim 3. If ∣ P ∣ ≥ 2 |P| \ge2 ∣ P ∣ ≥ 2 , then ψ h y b ( P ∪ { u } , B ) / ψ h y b ( P , B ) = x \psi^{\mathrm{hyb}}(P\cup\{u\},B)/\psi^{\mathrm{hyb}}(P,B)=x ψ hyb ( P ∪ { u } , B ) / ψ hyb ( P , B ) = x .
Proof of Claim 3 . Under μ W \mu_W μ W , the set e e e has probability β \beta β , each H ∈ H H\in\mathcal{H} H ∈ H has probability a W ( H ) a_W(H) a W ( H ) , and every other set that contains e e e and avoids B ∘ B^\circ B ∘ has probability 0. These probabilities add up to β + a \beta+a β + a , so ψ h y b \psi^{\mathrm{hyb}} ψ hyb is μ W \mu_W μ W conditioned on containing e e e and avoiding B ∘ B^\circ B ∘ . If P ′ ⊇ e P'\supseteq e P ′ ⊇ e and B ′ ⊇ B ∘ B'\supseteq B^\circ B ′ ⊇ B ∘ , then every set that contains P ′ P' P ′ and avoids B ′ B' B ′ also contains e e e and avoids B ∘ B^\circ B ∘ , so g ( P ′ , B ′ ) = ( β + a ) ψ h y b ( P ′ , B ′ ) g(P',B')=(\beta+a)\psi^{\mathrm{hyb}}(P',B') g ( P ′ , B ′ ) = ( β + a ) ψ hyb ( P ′ , B ′ ) . Apply this to ( P ∪ { u } , B ) (P\cup\{u\},B) ( P ∪ { u } , B ) and to ( P , B ) (P,B) ( P , B ) , and divide.
In summary, the probability that a query to a center c c c is answered 1 is given by the following table.
Table 2.
Thus D ∙ \mathbf{D}_{\bullet} D ∙ and Hyb differ only at queries to centers that have answered 1 at most once, and Hyb and G ∙ \mathbf{G}_{\bullet} G ∙ differ only at queries to centers that have answered 1 at least twice.
Proof of (44) . The next claim says that before a center’s second answer 1, the metric and the hybrid answer 1 with almost the same probability, which is of order 1 / m 1/m 1/ m .
Claim 4. If ∣ P ∣ ≤ 1 |P| \le1 ∣ P ∣ ≤ 1 , then
1 4 m ≤ x 0 ≤ 2 m − 1 ≤ 1 2 and ∣ x − x 0 ∣ ≤ 16 C 0 R m . \frac{1}{4m} \le x_0 \le\frac{2}{m-1} \le\frac{1}{2}
\qquad\text{and}\qquad
|x-x_0| \le\frac{16C_0}{Rm}. 4 m 1 ≤ x 0 ≤ m − 1 2 ≤ 2 1 and ∣ x − x 0 ∣ ≤ R m 16 C 0 .
Proof of Claim 4 . Let F = f ( P , B ) F=f(P,B) F = f ( P , B ) , F ′ = f ( P ∪ { u } , B ) F'=f(P\cup\{u\},B) F ′ = f ( P ∪ { u } , B ) , Δ = g ( P , B ) − F \Delta=g(P,B)-F Δ = g ( P , B ) − F and Δ ′ = g ( P ∪ { u } , B ) − F ′ \Delta'=g(P\cup\{u\},B)-F' Δ ′ = g ( P ∪ { u } , B ) − F ′ . Then x 0 = F ′ / F x_0=F'/F x 0 = F ′ / F and x = ( F ′ + Δ ′ ) / ( F + Δ ) x=(F'+\Delta')/(F+\Delta) x = ( F ′ + Δ ′ ) / ( F + Δ ) , so
x − x 0 = F ( F ′ + Δ ′ ) − F ′ ( F + Δ ) F ( F + Δ ) = Δ ′ − x 0 Δ F + Δ , hence ∣ x − x 0 ∣ ≤ ∣ Δ ′ ∣ + x 0 ∣ Δ ∣ F + Δ . x-x_0=\frac{F(F'+\Delta')-F'(F+\Delta)}{F(F+\Delta)}
=\frac{\Delta'-x_0\Delta}{F+\Delta},
\qquad\text{hence}\qquad
|x-x_0|\le\frac{|\Delta'|+x_0|\Delta|}{F+\Delta}. x − x 0 = F ( F + Δ ) F ( F ′ + Δ ′ ) − F ′ ( F + Δ ) = F + Δ Δ ′ − x 0 Δ , hence ∣ x − x 0 ∣ ≤ F + Δ ∣ Δ ′ ∣ + x 0 ∣Δ∣ .
We use Claim 1 throughout.
Case P = ∅ P=\varnothing P = ∅ . Here F ∈ [ 1 2 , 1 ] F\in[\frac{1}{2},1] F ∈ [ 2 1 , 1 ] and F ′ = f ( { u } , B ) ∈ [ 1 4 m , 3 4 m ] F'=f(\{u\},B)\in[\frac{1}{4m},\frac{3}{4m}] F ′ = f ({ u } , B ) ∈ [ 4 m 1 , 4 m 3 ] , so x 0 ∈ [ 1 4 m , 3 2 m ] x_0\in[\frac{1}{4m},\frac{3}{2m}] x 0 ∈ [ 4 m 1 , 2 m 3 ] . Moreover,
Δ = s ( ∅ , B ) − σ ∈ [ − C 0 R , 0 ] , Δ ′ = s ( { u } , B ) ∈ [ 0 , C 0 R m ] . \Delta=s(\varnothing,B)-\sigma\in\left[-\frac{C_0}{R},0\right],
\qquad
\Delta'=s(\{u\},B)\in\left[0,\frac{C_0}{Rm}\right]. Δ = s ( ∅ , B ) − σ ∈ [ − R C 0 , 0 ] , Δ ′ = s ({ u } , B ) ∈ [ 0 , R m C 0 ] .
Hence F + Δ ≥ 1 2 − 1 4 = 1 4 F+\Delta\ge\frac{1}{2}-\frac{1}{4}=\frac{1}{4} F + Δ ≥ 2 1 − 4 1 = 4 1 , and
∣ x − x 0 ∣ ≤ 4 ( C 0 R m + 3 2 m ⋅ C 0 R ) = 10 C 0 R m . |x-x_0|\le4\left(\frac{C_0}{Rm}+\frac{3}{2m}\cdot\frac{C_0}{R}\right)
=\frac{10C_0}{Rm}. ∣ x − x 0 ∣ ≤ 4 ( R m C 0 + 2 m 3 ⋅ R C 0 ) = R m 10 C 0 .
Case P = { v } P=\{v\} P = { v } . Here F = f ( { v } , B ) ∈ [ 1 4 m , 3 4 m ] F=f(\{v\},B)\in[\frac{1}{4m},\frac{3}{4m}] F = f ({ v } , B ) ∈ [ 4 m 1 , 4 m 3 ] and F ′ = β = 1 2 m ( m − 1 ) F'=\beta=\frac{1}{2m(m-1)} F ′ = β = 2 m ( m − 1 ) 1 , so x 0 = β / F ∈ [ 2 3 ( m − 1 ) , 2 m − 1 ] x_0=\beta/F\in[\frac{2}{3(m-1)},\frac{2}{m-1}] x 0 = β / F ∈ [ 3 ( m − 1 ) 2 , m − 1 2 ] . Moreover,
Δ = s ( { v } , B ) ∈ [ 0 , C 0 R m ] , Δ ′ = s ( { v , u } , B ) ∈ [ 0 , C 0 R m 2 ] . \Delta=s(\{v\},B)\in\left[0,\frac{C_0}{Rm}\right],
\qquad
\Delta'=s(\{v,u\},B)\in\left[0,\frac{C_0}{Rm^2}\right]. Δ = s ({ v } , B ) ∈ [ 0 , R m C 0 ] , Δ ′ = s ({ v , u } , B ) ∈ [ 0 , R m 2 C 0 ] .
Hence F + Δ ≥ F ≥ 1 4 m F+\Delta\ge F\ge\frac{1}{4m} F + Δ ≥ F ≥ 4 m 1 , and, as x 0 ≤ 2 m − 1 ≤ 3 m x_0\le\frac{2}{m-1}\le\frac{3}{m} x 0 ≤ m − 1 2 ≤ m 3 ,
∣ x − x 0 ∣ ≤ 4 m ( C 0 R m 2 + 3 m ⋅ C 0 R m ) = 16 C 0 R m . |x-x_0|\le4m\left(\frac{C_0}{Rm^2}+\frac{3}{m}\cdot\frac{C_0}{Rm}\right)
=\frac{16C_0}{Rm}. ∣ x − x 0 ∣ ≤ 4 m ( R m 2 C 0 + m 3 ⋅ R m C 0 ) = R m 16 C 0 .
In both cases 1 4 m ≤ x 0 ≤ 2 m − 1 \frac{1}{4m}\le x_0\le\frac{2}{m-1} 4 m 1 ≤ x 0 ≤ m − 1 2 , since 2 3 ( m − 1 ) ≥ 1 4 m \frac{2}{3(m-1)}\ge\frac{1}{4m} 3 ( m − 1 ) 2 ≥ 4 m 1 and 3 2 m ≤ 2 m − 1 \frac{3}{2m}\le\frac{2}{m-1} 2 m 3 ≤ m − 1 2 . Finally, 2 m − 1 ≤ 1 2 \frac{2}{m-1}\le\frac{1}{2} m − 1 2 ≤ 2 1 as m ≥ 8 m\ge8 m ≥ 8 . ◁
Now apply the chain rule, Lemma 2.2 (d), to the transcript distributions of D ∙ \mathcal{D}_{\bullet} D ∙ and Hyb. Its hypothesis holds: at a query to a center that has answered 1 at most once, Hyb gives both answers positive probability, as 0 < x 0 < 1 0<x_0<1 0 < x 0 < 1 , and at every other step the two answer rules are equal. Consider the term of step t t t , for a prefix drawn from D ∙ \mathcal{D}_{\bullet} D ∙ .
If the next query is to a center that has answered 1 at most once, the two conditional distributions of the answer are Ber ( x ) \operatorname{Ber}(x) Ber ( x ) and Ber ( x 0 ) \operatorname{Ber}(x_0) Ber ( x 0 ) , where the outcome 1 stands for the answer 1. By Claim 4 , x 0 ( 1 − x 0 ) ≥ 1 4 m ⋅ 1 2 = 1 8 m x_0(1-x_0)\ge\frac{1}{4m}\cdot\frac{1}{2}=\frac{1}{8m} x 0 ( 1 − x 0 ) ≥ 4 m 1 ⋅ 2 1 = 8 m 1 , and by Lemma 2.2 (f) and Claim 4 ,
KL ( Ber ( x ) ∥ Ber ( x 0 ) ) ≤ ( x − x 0 ) 2 x 0 ( 1 − x 0 ) ≤ ( 16 C 0 R m ) 2 ⋅ 8 m = 2048 C 0 2 R 2 m . \operatorname{KL}(\operatorname{Ber}(x)\mathbin\Vert\operatorname{Ber}(x_0))
\le\frac{(x-x_0)^2}{x_0(1-x_0)}
\le\left(\frac{16C_0}{Rm}\right)^2\cdot8m
=\frac{2048C_0^2}{R^2m}. KL ( Ber ( x ) ∥ Ber ( x 0 )) ≤ x 0 ( 1 − x 0 ) ( x − x 0 ) 2 ≤ ( R m 16 C 0 ) 2 ⋅ 8 m = R 2 m 2048 C 0 2 .
So each of the q q q terms is at most 2048 C 0 2 / ( R 2 m ) 2048C_0^2/(R^2m) 2048 C 0 2 / ( R 2 m ) , and the relative entropy between the two transcript distributions is at most 2048 C 0 2 q / ( R 2 m ) 2048C_0^2q/(R^2m) 2048 C 0 2 q / ( R 2 m ) . By Pinsker’s inequality, Lemma 2.2 (e), the total variation distance is at most 1024 C 0 2 q / ( R 2 m ) = 32 C 0 q / ( m R ) \sqrt{1024C_0^2q/(R^2m)}=32C_0\sqrt{q}/(mR) 1024 C 0 2 q / ( R 2 m ) = 32 C 0 q / ( m R ) . This proves (44) .
Proof of (45) . The hybrid and G ∙ \mathcal{G}_\bullet G ∙ differ only in the distribution of the set from which a center answers after its second answer 1. The next claim shows that these two distributions are very close.
Claim 5. For every pair e e e and every set B ∘ B^{\circ} B ∘ disjoint from e e e ,
d T V ( ψ h y b , ψ e x p ) = a 2 β ( β + a ) ≤ ( B 0 R ) 2 . d_{\mathrm{TV}}(\psi^{\mathrm{hyb}},\psi^{\mathrm{exp}})=\frac{a^2}{\beta(\beta+a)}\leq\left(\frac{B_0}{R}\right)^2. d TV ( ψ hyb , ψ exp ) = β ( β + a ) a 2 ≤ ( R B 0 ) 2 .
Proof of Claim 5 . For H ∈ H H\in\mathcal{H} H ∈ H ,
ψ e x p ( H ) − ψ h y b ( H ) = a W ( H ) ( 1 β − 1 β + a ) = a 2 W ( H ) β ( β + a ) ≥ 0 , \psi^{\mathrm{exp}}(H)-\psi^{\mathrm{hyb}}(H)=aW(H)\left(\frac{1}{\beta}-\frac{1}{\beta+a}\right)=\frac{a^2W(H)}{\beta(\beta+a)}\geq0, ψ exp ( H ) − ψ hyb ( H ) = aW ( H ) ( β 1 − β + a 1 ) = β ( β + a ) a 2 W ( H ) ≥ 0 ,
and these differences add up to a 2 / ( β ( β + a ) ) a^2/(\beta(\beta+a)) a 2 / ( β ( β + a )) . For e e e ,
ψ h y b ( e ) − ψ e x p ( e ) = β β + a − β − a β = β 2 − ( β − a ) ( β + a ) β ( β + a ) = a 2 β ( β + a ) . \psi^{\mathrm{hyb}}(e)-\psi^{\mathrm{exp}}(e)=\frac{\beta}{\beta+a}-\frac{\beta-a}{\beta}=\frac{\beta^2-(\beta-a)(\beta+a)}{\beta(\beta+a)}=\frac{a^2}{\beta(\beta+a)}. ψ hyb ( e ) − ψ exp ( e ) = β + a β − β β − a = β ( β + a ) β 2 − ( β − a ) ( β + a ) = β ( β + a ) a 2 .
The total variation distance is half the sum of the absolute differences, which gives the equality. Finally, a 2 / ( β ( β + a ) ) ≤ ( a / β ) 2 a^2/(\beta(\beta+a))\leq(a/\beta)^2 a 2 / ( β ( β + a )) ≤ ( a / β ) 2 , and a ≤ B 0 β / R a\leq B_0\beta/R a ≤ B 0 β / R by (42) . ◁
To use Claim 5 , we run Hyb and G ∙ \mathcal{G}_\bullet G ∙ in a form in which each center draws, at its second answer 1, the whole set from which it answers afterwards. This is the principle of deferred decisions.
Claim 6. Let ψ \psi ψ be ψ h y b \psi^{\mathrm{hyb}} ψ hyb or ψ e x p \psi^{\mathrm{exp}} ψ exp . The following procedure has the answer rule of Hyb, respectively of G ∙ \mathcal{G}_\bullet G ∙ . A query with a fixed answer gets that answer. A query to a center that has answered 1 at most once is answered 1 with probability x 0 x_0 x 0 . When a center c c c gives its second answer 1, it draws a set N c N_c N c from ψ \psi ψ , with the e e e and B ∘ B^{\circ} B ∘ of c c c , independently of everything else. At every later query { c , u } \{c,u\} { c , u } , it answers 1 exactly when u ∈ N c u\in N_c u ∈ N c .
Proof of Claim 6 . At a query to a center that has answered 1 at most once, the procedure follows the rule by construction. Now consider a query { c , u } \{c,u\} { c , u } after c c c has drawn N c N_c N c , and let ( P , B ) (P,B) ( P , B ) be the local history of c c c . Every set N N N in the support of ψ \psi ψ contains e e e and avoids B ∘ B^{\circ} B ∘ , so the answers given at c c c since the draw agree with N c = N N_c=N N c = N exactly when N ⊇ P N\supseteq P N ⊇ P and N ∩ B = ∅ N\cap B=\varnothing N ∩ B = ∅ . Since N c N_c N c was drawn independently of everything else, the probability of the prefix together with the event N c = N N_c=N N c = N is a factor that does not depend on N N N , times ψ ( N ) 1 [ N ⊇ P , N ∩ B = ∅ ] \psi(N)\mathbf{1}[N\supseteq P,N\cap B=\varnothing] ψ ( N ) 1 [ N ⊇ P , N ∩ B = ∅ ] . So, given the prefix, N c = N N_c=N N c = N with probability ψ ( N ) 1 [ N ⊇ P , N ∩ B = ∅ ] / ψ ( P , B ) \psi(N)\mathbf{1}[N\supseteq P,N\cap B=\varnothing]/\psi(P,B) ψ ( N ) 1 [ N ⊇ P , N ∩ B = ∅ ] / ψ ( P , B ) , and the answer is 1 with probability ψ ( P ∪ { u } , B ) / ψ ( P , B ) \psi(P\cup\{u\},B)/\psi(P,B) ψ ( P ∪ { u } , B ) / ψ ( P , B ) . This is the rule of Hyb for ψ = ψ h y b \psi=\psi^{\mathrm{hyb}} ψ = ψ hyb , and the rule of G ∙ \mathcal{G}_\bullet G ∙ for ψ = ψ e x p \psi=\psi^{\mathrm{exp}} ψ = ψ exp , by Claim 2 . ◁
We now run the two experiments together. Let U 1 , … , U q , V 1 , … , V q U_1,\ldots,U_q,V_1,\ldots,V_q U 1 , … , U q , V 1 , … , V q be independent and uniform in [ 0 , 1 ] [0,1] [ 0 , 1 ] . For all e e e and B ∘ B^{\circ} B ∘ , Lemma 2.2 (a) gives a pair of random sets with distributions ψ h y b \psi^{\mathrm{hyb}} ψ hyb and ψ e x p \psi^{\mathrm{exp}} ψ exp that differ with probability d T V ( ψ h y b , ψ e x p ) d_{\mathrm{TV}}(\psi^{\mathrm{hyb}},\psi^{\mathrm{exp}}) d TV ( ψ hyb , ψ exp ) . Fix a map that turns a uniform number into such a pair. Run Hyb and G ∙ \mathcal{G}_\bullet G ∙ side by side, each in the form of Claim 6 . At step t t t , each run looks only at its own past, and:
a query with a fixed answer gets that answer;
a query to a center that has answered 1 at most once is answered 1 exactly when U t < x 0 U_t<x_0 U t < x 0 , with x 0 x_0 x 0 computed from the run’s own past;
if this answer gives the center its second 1, the run feeds V t V_t V t into the map for its own e e e and B ∘ B^{\circ} B ∘ . The hybrid run keeps the first set of the resulting pair as N c N_c N c , and the experiment run keeps the second;
Each run uses U t U_t U t and V t V_t V t only through its own past, exactly as it would alone. So, by Claim 6 , the hybrid run has the transcript distribution of Hyb, and the experiment run that of G ∙ \mathcal{G}_{\bullet} G ∙ .
Suppose that the two transcripts agree before step t t t . Then the two runs ask the same query, have the same local histories, and compute the same x 0 x_0 x 0 , e e e and B ∘ B^\circ B ∘ . So they give the same answer to a query with a fixed answer and to a query to a center that has answered 1 at most once, since they use the same U t U_t U t . At a query to a center c c c that has answered 1 at least twice, they answer by membership in their own sets N c N_c N c . These two sets were obtained from the same V s V_s V s and the same e e e and B ∘ B^\circ B ∘ , at the step s s s where c c c got its second answer 1. If they are equal, the answers agree.
Call a step an activation if it gives some center its second answer 1. Let E s E_s E s be the event that the runs agree before step s s s , that step s s s is an activation, and that the two sets obtained from V s V_s V s differ. By the previous paragraph, the transcripts differ only if some E s E_s E s occurs. The first two conditions are decided by U 1 , … , U s U_1,\ldots,U_s U 1 , … , U s and V 1 , … , V s − 1 V_1,\ldots,V_{s-1} V 1 , … , V s − 1 , and V s V_s V s is independent of these. So, given the first two conditions, the two sets differ with probability at most ( B 0 / R ) 2 (B_0/R)^2 ( B 0 / R ) 2 by Claim 5 , and
P [ E s ] ≤ ( B 0 R ) 2 P [ step s is an activation in the hybrid run ] . \mathbb{P}[E_s] \leq\left(\frac{B_0}{R}\right)^2\mathbb{P}[\text{step }s\text{ is an activation in the hybrid run}]. P [ E s ] ≤ ( R B 0 ) 2 P [ step s is an activation in the hybrid run ] .
Step s s s can be an activation only if it is a query to a center that has answered 1 exactly once, and then it is one with probability x 0 ≤ 2 / ( m − 1 ) x_0 \leq2/(m-1) x 0 ≤ 2/ ( m − 1 ) given the past, by Claim 4 . So the expected number of activations in the hybrid run is at most 2 q / ( m − 1 ) ≤ 4 q / m 2q/(m-1) \leq4q/m 2 q / ( m − 1 ) ≤ 4 q / m . By Lemma 2.2 (a) and B 0 ≤ 2 C 0 B_0 \leq2C_0 B 0 ≤ 2 C 0 ,
d T V ( trans ( A , Hyb ) , trans ( A , G ∙ ) ) ≤ P [ the transcripts differ ] ≤ ∑ s = 1 q P [ E s ] ≤ ( B 0 R ) 2 4 q m ≤ 16 C 0 2 q m R 2 . d_{\mathrm{TV}}(\operatorname{trans}(\mathcal{A},\operatorname{Hyb}),\operatorname{trans}(\mathcal{A},\mathcal{G}_{\bullet})) \leq\mathbb{P}[\text{the transcripts differ}] \leq\sum_{s=1}^{q}\mathbb{P}[E_s] \leq\left(\frac{B_0}{R}\right)^2\frac{4q}{m} \leq\frac{16C_0^2q}{mR^2}. d TV ( trans ( A , Hyb ) , trans ( A , G ∙ )) ≤ P [ the transcripts differ ] ≤ s = 1 ∑ q P [ E s ] ≤ ( R B 0 ) 2 m 4 q ≤ m R 2 16 C 0 2 q .
This proves (6.6).
Proof of Lemma 6.1 . For fixed π \pi π and W W W , (6.5), (6.6) and the triangle inequality give
d T V ( trans ( A , D ∙ ) , trans ( A , G ∙ ) ) ≤ 32 C 0 q m R 2 + 16 C 0 2 q m R 2 ≤ C ( q m R 2 + q m R 2 ) . d_{\mathrm{TV}}(\operatorname{trans}(\mathcal{A},\mathcal{D}_{\bullet}),\operatorname{trans}(\mathcal{A},\mathcal{G}_{\bullet})) \leq32C_0\sqrt{\frac{q}{mR^2}}+\frac{16C_0^2q}{mR^2} \leq C\left(\sqrt{\frac{q}{mR^2}}+\frac{q}{mR^2}\right). d TV ( trans ( A , D ∙ ) , trans ( A , G ∙ )) ≤ 32 C 0 m R 2 q + m R 2 16 C 0 2 q ≤ C ( m R 2 q + m R 2 q ) .
with C = max { 32 C 0 , 16 C 0 2 } C=\max\{32C_0,16C_0^2\} C = max { 32 C 0 , 16 C 0 2 } . This bound does not depend on π \pi π and W W W , so by the reductions it holds for random π \pi π and W W W and for the original transcripts.
Erasing an unseen star
From now on, an answer 1 to a query between a center and a special terminal is called positive . The next lemma concerns G ∙ \mathcal{G}_{\bullet} G ∙ alone. Its hidden data are π \pi π , W W W , the fillers, and the potential stars. Together with A \mathcal{A} A , they determine the transcript. Fix a center c c c , and let Z Z Z be all hidden data other than H c H_c H c . Deleting the potential star of c c c means making H c H_c H c absent while keeping Z Z Z .
Lemma 6.2 (Erasing an unseen star ). Let τ \tau τ be a prefix in which c c c has at most two positive answers.
(a) If the transcript begins with τ \tau τ , then it still begins with τ \tau τ after the potential star of c c c is deleted.
(b) For every value z z z of Z Z Z such that the event that Z = z Z=z Z = z and the transcript begins with τ \tau τ has positive probability,
P [ H c is present ∣ Z = z , the transcript begins with τ ] ≤ B 0 R . \mathbb{P}[H_c\text{ is present}\mid Z=z,\ \text{the transcript begins with }\tau] \leq\frac{B_0}{R}. P [ H c is present ∣ Z = z , the transcript begins with τ ] ≤ R B 0 .
Proof. (a) Compare the runs with and without H c H_c H c , and suppose that they agree before some query of τ \tau τ . Then A \mathcal{A} A asks the same query in both. Only the answer to a query { c , u } \{c,u\} { c , u } with u u u special can depend on H c H_c H c . Up to and including its activating query, c c c answers by membership in F c F_c F c , and its activation time and B c B_c B c depend only on F c F_c F c and on the queries at c c c , which agree. After activation, the star can only change an answer 2 2 2 into 1 1 1 , for some u ∉ F c u \notin F_c u ∈ / F c . Activation happens only after both members of F c F_c F c were answered 1 1 1 at c c c , so such an answer would be a third positive answer at c c c in τ \tau τ , which is excluded. Hence the two answers agree, and by induction the two runs agree on all of τ \tau τ .
(b) If F c F_c F c is not a pair (this is part of z z z ), then H c H_c H c is absent. So let F c = e F_c=e F c = e . Potential stars are independent given the weights and the fillers, so given Z = z Z=z Z = z , the star H c H_c H c still has the distribution (6.1), and it is present with probability p ≤ B 0 / R p \le B_0/R p ≤ B 0 / R by (6.3). Given Z = z Z=z Z = z , the value of H c H_c H c determines whether the transcript begins with τ \tau τ . Some value does, so by (a) absence does. Let y ≤ p y \le p y ≤ p be the probability, given Z = z Z=z Z = z , that H c H_c H c is present and the transcript begins with τ \tau τ . Then the probability in (b) is
y ( 1 − p ) + y ≤ p ( 1 − p ) + p = p ≤ B 0 R . \frac{y}{(1-p)+y} \le\frac{p}{(1-p)+p}=p\le\frac{B_0}{R}. ( 1 − p ) + y y ≤ ( 1 − p ) + p p = p ≤ R B 0 .
as y ↦ y / ( ( 1 − p ) + y ) y \mapsto y/((1-p)+y) y ↦ y / (( 1 − p ) + y ) is increasing. In words, conditioning on τ \tau τ may rule out some potential stars, but never their absence. □ \square □
Since (b) holds for every value of Z Z Z , the same bound holds given that the transcript begins with τ \tau τ together with any part of the other hidden data.
Lower-Bound Analysis
We now compare the two-key experiments G y \mathcal{G}_y G y and G n \mathcal{G}_n G n and prove Theorem 1.2 . § 7.1 shows that few queries single out only a few special terminals. § 7.2 shows that the potential stars visible from these terminals reveal little about the weights. § 7.3 combines these bounds with Lemma 6.1 .
What few queries can reveal
We work in either two-key experiment. The aim is to show that few queries can single out only a few special terminals. Several components of this subsection, such as the bookkeeping of noticed terminals and the exchange arguments of Lemmas 7.3 and 7.4 , are similar to those in earlier works [16 , 33 ] . The new ingredients handle potential stars that have not yet affected any answer.
Noticed and exposed terminals
Put θ = 1 / 16 \theta=1/16 θ = 1/16 and D = ⌊ θ A ⌋ D=\lfloor\theta A\rfloor D = ⌊ θ A ⌋ . Recall that τ j \tau_j τ j is the prefix before query j j j , and let deg j ( u ) \operatorname{deg}_j(u) deg j ( u ) count earlier queries involving u u u . An ordinary terminal is marked once it receives a λ \lambda λ -answer. For an ordinary cluster G G G , let e j ( G ) e_j(G) e j ( G ) be the number of terminals outside G G G queried against its members before query j j j . For a special terminal u u u , define
b j F ( u ) = # { x ordinary : { x , c } was queried before j for some c with u ∈ F c } , b_j^F(u)=\#\{x\text{ ordinary}:\{x,c\}\text{ was queried before }j\text{ for some }c\text{ with }u\in F_c\}, b j F ( u ) = # { x ordinary : { x , c } was queried before j for some c with u ∈ F c } ,
b j H ( u ) = # { x ordinary : { x , c } was queried before j for some c with u ∈ H c } . b_j^H(u)=\#\{x\text{ ordinary}:\{x,c\}\text{ was queried before }j\text{ for some }c\text{ with }u\in H_c\}. b j H ( u ) = # { x ordinary : { x , c } was queried before j for some c with u ∈ H c } .
The second counter includes discarded potential stars. These counters measure the ordinary labels that cannot safely exchange roles with u u u . Two deterministic budgets will be useful:
∑ G e q + 1 ( G ) ≤ 2 q , ∑ u ∈ S b q + 1 F ( u ) ≤ 2 q . (46) \sum_G e_{q+1}(G) \le2q,\qquad\sum_{u\in S} b^F_{q+1}(u) \le2q. \tag*{(46)} G ∑ e q + 1 ( G ) ≤ 2 q , u ∈ S ∑ b q + 1 F ( u ) ≤ 2 q . (46)
The latter uses ∣ F c ∣ ≤ 2 |F_c|\le2 ∣ F c ∣ ≤ 2 .
Definition 7.1 (Noticing and closure). Maintain sets N ⊆ Q ⊆ S \mathcal{N}\subseteq Q\subseteq S N ⊆ Q ⊆ S , initially empty. A terminal u ∈ S ∖ Q u\in S\setminus Q u ∈ S ∖ Q is noticed and added to both sets when it first receives answer 1, its query count reaches D D D , or its b F b^F b F counter reaches θ K \theta K θ K . After adding newly noticed terminals in a fixed order, close Q Q Q under the rule
F c ⊆ Q , ∣ F c ∣ = 2 , ∣ H c ∩ Q ∣ ≥ 3 ⟹ H c ⊆ Q . (47) F_c\subseteq Q,\quad|F_c|=2,\quad|H_c\cap Q|\ge3 \quad\Longrightarrow\quad H_c\subseteq Q. \tag*{(47)} F c ⊆ Q , ∣ F c ∣ = 2 , ∣ H c ∩ Q ∣ ≥ 3 ⟹ H c ⊆ Q . (47) Repeat the rule until it makes no further addition. A terminal added by closure is exposed , but is not subsequently counted as a new noticed terminal.
A query { c , u } \{c,u\} { c , u } is lucky if, just before it, u ∈ S ∖ Q u\in S\setminus Q u ∈ S ∖ Q and u ∈ H c u\in H_c u ∈ H c . This definition includes discarded stars and does not require the answer to be 1.
The sets are analytical objects, not information given to A \mathcal{A} A . Closure deliberately exposes an entire potential star as soon as three of its members, including its keys, are available. This only gives the analysis more information.
Stop the analysis at the first query j ∗ j_* j ∗ where any of the following occurs:
(i) at least θ K \theta K θ K ordinary terminals have been marked (cluster stop );
(ii) some u ∈ S ∖ Q u\in S\setminus Q u ∈ S ∖ Q has b H ( u ) ≥ θ K b^H(u)\ge\theta K b H ( u ) ≥ θ K (star-counter stop );
(iii) the query is lucky;
(iv) ∣ N ∣ > R |\mathcal{N}|>R ∣ N ∣ > R (notice overflow );
(v) ∣ Q ∣ > 2 R |Q|>2R ∣ Q ∣ > 2 R (closure overflow ).
Luck is tested against the pre-query set Q Q Q , before updating any counters or sets, and it has priority if several stops occur together. For the other stops use a fixed tie order in which closure overflow comes last. Set j ∗ = q + 1 j_*=q+1 j ∗ = q + 1 if none occurs.
Call a special terminal quiet if no earlier query { c , u } \{c,u\} { c , u } had u ∈ F c ∪ H c u\in F_c\cup H_c u ∈ F c ∪ H c .
Lemma 7.2 (Before the first stop). At the start of every query j ≤ j ∗ j\le j_* j ≤ j ∗ : every special terminal outside Q Q Q is quiet, both keys of every active center lie in Q Q Q , and every center with at least three positive answers has its potential star entirely in Q Q Q .
Proof. A query to a filler member returns 1 and notices it unless it is already in Q Q Q . A query to a potential-star member outside Q Q Q is lucky. Thus terminals still outside Q Q Q are quiet. Activation requires two positive key answers, so both keys are in Q Q Q . A third positive answer requires a non-key star member. Without a lucky query that member was already in Q Q Q , and (47) then adds the whole star.
The exchange argument
Condition on a prefix τ j \tau_j τ j , and let Ω j \Omega_j Ω j be its consistent configurations: the values of the hidden data for which the transcript agrees with τ j \tau_j τ j . To simplify notation, we slightly abuse notation and also write τ j \tau_j τ j for this event. Thus P [ τ j ] = P [ Ω j ] \mathbb{P}[\tau_j]=\mathbb{P}[\Omega_j] P [ τ j ] = P [ Ω j ] , and P [ E ∣ τ j ] \mathbb{P}[E\mid\tau_j] P [ E ∣ τ j ] is the probability of E E E given that the transcript begins with τ j \tau_j τ j . We use the following elementary counting principle, which also applies to nonuniform configurations. If every configuration in an event E ⊆ Ω j E \subseteq\Omega_j E ⊆ Ω j has at least F F F probability-preserving moves into Ω j \Omega_j Ω j , and each target receives at most B B B such moves, then
P [ E ∣ τ j ] ≤ B / F . (48) \mathbb{P}[E \mid\tau_j] \le B/F. \tag*{(48)} P [ E ∣ τ j ] ≤ B / F . (48)
Indeed, summing the probability mass of all moves gives F P [ E ] ≤ B P [ Ω j ] F\mathbb{P}[E] \le B\mathbb{P}[\Omega_j] F P [ E ] ≤ B P [ Ω j ] .
Every move below transposes two terminal labels in π \pi π , keeping all data attached to positions fixed. The labels exchange their roles, clusters, weights, and memberships in fillers and potential stars. Since π \pi π is uniform and independent of that data, each move preserves probability. Only consistency with the prefix and the number of inverse moves need checking.
Lemma 7.3 (Clusters are hard to find). The probability of a cluster stop is O ( q / ( A K ) ) O(q/(AK)) O ( q / ( A K )) .
Proof. Count newly marked terminals up to and including the query that causes the cluster stop. Terminals already having query count at least D D D contribute at most 2 q / D = O ( q / A ) 2q/D = O(q/A) 2 q / D = O ( q / A ) marks. Clusters with e j ( G ) ≥ θ K e_j(G) \ge\theta K e j ( G ) ≥ θ K contribute at most 2 q R / ( θ K ) = O ( q / A ) 2qR/(\theta K) = O(q/A) 2 q R / ( θ K ) = O ( q / A ) marks, by (46) .
For every other new mark, fix the queried pair { u , v } \{u,v\} { u , v } and consider configurations where u u u is unmarked, deg j ( u ) < D \deg_j(u) < D deg j ( u ) < D , and u , v u,v u , v lie in a cluster G G G with e j ( G ) < θ K e_j(G) < \theta K e j ( G ) < θ K . Transpose u u u with an ordinary terminal x ∉ G x \notin G x ∈ / G , excluding x x x if it is marked, if it was queried against G G G , or if its cluster contains a terminal previously queried against u u u . Before the cluster stop at most $2\theta K + DR + R \le K/2$ labels are excluded, for large A A A . No old λ \lambda λ -answer is lost, no new one is created, and all center answers involving either ordinary terminal remain 2. Thus at least K / 2 K/2 K /2 moves preserve the prefix. In a target, x x x lies in the cluster of v v v , so there are at most R R R inverse moves. By (48) , the conditional probability of such a mark is at most $2R/K = 2/A$.
Summing over the 2 q 2q 2 q query endpoints gives O ( q / A ) O(q/A) O ( q / A ) expected marks. A cluster stop needs θ K \theta K θ K marks, so Markov’s inequality proves the claim.
For a terminal u u u , let E j ( u ) E_j(u) E j ( u ) be the event that no cluster stop has occurred, u u u is special and quiet, and
deg j ( u ) < D , b j F ( u ) < θ K , b j H ( u ) < θ K . \deg_j(u) < D,\qquad b_j^F(u) < \theta K,\qquad b_j^H(u) < \theta K. deg j ( u ) < D , b j F ( u ) < θ K , b j H ( u ) < θ K .
Before the first stop this event holds for every special terminal outside Q Q Q .
Lemma 7.4 (Special terminals are hard to recognize). For every terminal u u u , center c c c , and prefix,
P [ E j ( u ) ∣ τ j ] ≤ 4 m 3 K , P [ E j ( u ) , u ∈ F c ∣ τ j ] ≤ 8 3 K . (49) \mathbb{P}[E_j(u) \mid\tau_j] \le\frac{4m}{3K},\qquad
\mathbb{P}[E_j(u),u \in F_c \mid\tau_j] \le\frac{8}{3K}. \tag*{(49)} P [ E j ( u ) ∣ τ j ] ≤ 3 K 4 m , P [ E j ( u ) , u ∈ F c ∣ τ j ] ≤ 3 K 8 . (49) The same statements hold under the distribution in which the potential star of one fixed center is deleted and all other data keep their original distribution.
Proof. Transpose u u u with an ordinary label x x x . Exclude marked x x x , labels whose cluster contains a terminal queried against u u u , and labels counted by b j F ( u ) b_j^F(u) b j F ( u ) or b j H ( u ) b_j^H(u) b j H ( u ) . At most 3 θ K + D R ≤ K / 4 3\theta K + DR \le K/4 3 θ K + D R ≤ K /4 labels are excluded, leaving 3 K / 4 3K/4 3 K /4 moves.
All these moves preserve the prefix. Terminal–terminal answers remain 2, by the first two exclusions. By quietness, earlier center queries at u u u met neither a filler nor a potential star. By the counter exclusions the same remains true of earlier center queries at x x x after it takes u u u ’s position. Consequently neither positive answers nor activation or discard decisions change.
In a target, the moved label x x x is special, giving at most m m m inverse moves. For the second event it also belongs to F c F_c F c , giving at most two. Apply (48) . Deleting one fixed potential star commutes with the same transpositions, so the argument also applies when the star is deleted.
Most stops are unlikely
Lemma 7.5 (Other stops are unlikely). The probability that the first stop is anything other than closure overflow is at most
C ( q A K + q K R ) . (50) C\left(\frac{q}{AK}+\frac{q}{KR}\right). \tag*{(50)} C ( A K q + K R q ) . (50)
Proof. The cluster term is Lemma 7.3 . We bound the other three types of stop along prefixes with no earlier stop. All conditional estimates below condition only on the observable prefix τ j \tau_j τ j . Having no earlier stop is part of the event being bounded, not additional conditioning on hidden information.
Notices. On a query that is not lucky, a new answer-1 notice must come from u ∈ F c u \in F_c u ∈ F c , so Lemma 7.4 bounds its probability by O ( 1 / K ) O(1/K) O ( 1/ K ) . There are at most 2 q / D 2q/D 2 q / D crossings of query count D D D , and the crossing times and labels are determined by the transcript before the query. Each yields a new special notice with probability O ( m / K ) O(m/K) O ( m / K ) , contributing O ( q m / ( D K ) ) = O ( q / K ) O(qm/(DK)) = O(q/K) O ( q m / ( D K )) = O ( q / K ) in expectation. Finally, (46) permits at most 2 q / ( θ K ) 2q/(\theta K) 2 q / ( θ K ) notices from the b F b^F b F threshold. Only terminals outside Q Q Q are counted, so vertices already added by closure do not enter this accounting. The expected number of such direct notices is O ( q / K ) O(q/K) O ( q / K ) . Markov’s inequality at R R R bounds a first notice overflow by O ( q / ( K R ) ) O(q/(KR)) O ( q / ( K R )) .
Star counters. An increment of b H ( u ) b^H(u) b H ( u ) for u ∉ Q u \notin Q u ∈ / Q comes from an ordinary-terminal query at a center with a potential star. By Lemma 7.2 , that center has at most two positive answers: otherwise its entire potential star is already in Q Q Q . By Lemma 6.2, the conditional probability that such a center has a potential star is O ( 1 / R ) O(1/R) O ( 1/ R ) . One query increments at most d d d star counters. The expected total number of relevant increments, including a possible stopping query, is therefore O ( q / R ) O(q/R) O ( q / R ) . Reaching θ K \theta K θ K gives probability O ( q / ( K R ) ) O(q/(KR)) O ( q / ( K R )) .
Lucky queries. Let L j L_j L j be the event that query j j j is lucky and no stop occurred before it. Fix a prefix τ = τ j \tau=\tau_j τ = τ j of positive probability and its next pair { c , u } \{c,u\} { c , u } . Only a pair between a center c c c and a terminal u u u can be lucky. We show that P [ L j ∣ τ ] = O ( 1 / ( K R ) ) \mathbb{P}[L_j\mid\tau]=O(1/(KR)) P [ L j ∣ τ ] = O ( 1/ ( K R )) .
Write a configuration as ( Z , h ) (Z,h) ( Z , h ) , where h h h is the value of H c H_c H c (h = ∅ h=\varnothing h = ∅ if H c H_c H c is absent) and Z Z Z consists of all other hidden data. The star H c H_c H c is drawn by (40) , which uses only F c F_c F c and W W W , and these are part of Z Z Z . So P [ Z , h ] = P [ Z ] ρ Z ( h ) \mathbb{P}[Z,h]=\mathbb{P}[Z]\rho_Z(h) P [ Z , h ] = P [ Z ] ρ Z ( h ) with ρ Z ( h ) = P [ H c = h ∣ F c , W ] \rho_Z(h)=\mathbb{P}[H_c=h\mid F_c,W] ρ Z ( h ) = P [ H c = h ∣ F c , W ] , and by (42) ,
∑ h ≠ ∅ ρ Z ( h ) ≤ B 0 R , ∑ h ∋ u ρ Z ( h ) ≤ B 1 R m if u ∉ F c . (51) \sum_{h\ne\varnothing}\rho_Z(h)\le\frac{B_0}{R},\qquad
\sum_{h\ni u}\rho_Z(h)\le\frac{B_1}{Rm}\quad\text{if }u\notin F_c. \tag*{(51)} h = ∅ ∑ ρ Z ( h ) ≤ R B 0 , h ∋ u ∑ ρ Z ( h ) ≤ R m B 1 if u ∈ / F c . (51)
and both sums are 0 0 0 if F c F_c F c is not a pair. Let P 0 c \mathbb{P}_0^c P 0 c be the distribution of ( Z , ∅ ) (Z,\varnothing) ( Z , ∅ ) when ( Z , H c ) (Z,H_c) ( Z , H c ) is drawn as usual. It deletes the potential star of c c c and keeps all other data, so it gives probability P [ Z ] \mathbb{P}[Z] P [ Z ] to ( Z , ∅ ) (Z,\varnothing) ( Z , ∅ ) . This is deletion, not conditioning on absence. The experiment itself produces ( Z , ∅ ) (Z,\varnothing) ( Z , ∅ ) with probability P [ Z ] ρ Z ( ∅ ) ≥ ( 1 − B 0 / R ) P [ Z ] \mathbb{P}[Z]\rho_Z(\varnothing)\ge(1-B_0/R)\mathbb{P}[Z] P [ Z ] ρ Z ( ∅ ) ≥ ( 1 − B 0 / R ) P [ Z ] . Keeping only the configurations of τ \tau τ in which H c H_c H c is absent, we get
P [ τ ] ≥ ∑ Z : ( Z , ∅ ) ∈ τ P [ Z ] ρ Z ( ∅ ) ≥ ( 1 − B 0 R ) ∑ Z : ( Z , ∅ ) ∈ τ P [ Z ] = ( 1 − B 0 R ) P 0 c [ τ ] . (52) \mathbb{P}[\tau]\ge
\sum_{Z:(Z,\varnothing)\in\tau}\mathbb{P}[Z]\rho_Z(\varnothing)
\ge\left(1-\frac{B_0}{R}\right)
\sum_{Z:(Z,\varnothing)\in\tau}\mathbb{P}[Z]
=\left(1-\frac{B_0}{R}\right)\mathbb{P}_0^c[\tau]. \tag*{(52)} P [ τ ] ≥ Z : ( Z , ∅ ) ∈ τ ∑ P [ Z ] ρ Z ( ∅ ) ≥ ( 1 − R B 0 ) Z : ( Z , ∅ ) ∈ τ ∑ P [ Z ] = ( 1 − R B 0 ) P 0 c [ τ ] . (52)
Now let ( Z , h ) ∈ τ ∩ L j (Z,h)\in\tau\cap L_j ( Z , h ) ∈ τ ∩ L j . Luck means that u ∈ h u\in h u ∈ h and u ∈ S ∖ Q u\in S\setminus Q u ∈ S ∖ Q . Since no stop occurred before query j j j , the event E j ( u ) E_j(u) E j ( u ) holds, and c c c has at most two positive answers: otherwise Lemma 7.2 would give H c ⊆ Q H_c\subseteq Q H c ⊆ Q , contradicting u ∈ H c ∖ Q u\in H_c\setminus Q u ∈ H c ∖ Q . Hence, by Lemma 6.2(a), ( Z , ∅ ) (Z,\varnothing) ( Z , ∅ ) is still in τ \tau τ . It is also in E j ( u ) E_j(u) E j ( u ) , because deleting H c H_c H c only removes memberships in a potential star. Whether u ∈ F c u\in F_c u ∈ F c is decided by Z Z Z . Grouping the configurations ( Z , h ) (Z,h) ( Z , h ) by Z Z Z and using (51) , we get
P [ τ , L j , u ∈ F c ] ≤ ∑ Z : ( Z , ∅ ) ∈ τ ∩ E j ( u ) u ∈ F c P [ Z ] ∑ h ≠ ∅ ρ Z ( h ) ≤ B 0 R P 0 c [ τ , E j ( u ) , u ∈ F c ] . \mathbb{P}[\tau,L_j,u\in F_c]\le\sum_{\substack{Z:(Z,\varnothing)\in\tau\cap E_j(u)\\u\in F_c}}\mathbb{P}[Z]\sum_{h\ne\varnothing}\rho_Z(h)\le\frac{B_0}{R}P_0^c[\tau,E_j(u),u\in F_c]. P [ τ , L j , u ∈ F c ] ≤ Z : ( Z , ∅ ) ∈ τ ∩ E j ( u ) u ∈ F c ∑ P [ Z ] h = ∅ ∑ ρ Z ( h ) ≤ R B 0 P 0 c [ τ , E j ( u ) , u ∈ F c ] .
and in the same way, with the second bound in (51) ,
P [ τ , L j , u ∉ F c ] ≤ B 1 R m P 0 c [ τ , E j ( u ) ] . \mathbb{P}[\tau,L_j,u\notin F_c]\le\frac{B_1}{Rm}P_0^c[\tau,E_j(u)]. P [ τ , L j , u ∈ / F c ] ≤ R m B 1 P 0 c [ τ , E j ( u )] .
By Lemma 7.4 under P 0 c P_0^c P 0 c , the right-hand sides are at most B 0 R ⋅ 8 3 K P 0 c [ τ ] \frac{B_0}{R}\cdot\frac{8}{3K}P_0^c[\tau] R B 0 ⋅ 3 K 8 P 0 c [ τ ] and B 1 R m ⋅ 4 m 3 K P 0 c [ τ ] \frac{B_1}{Rm}\cdot\frac{4m}{3K}P_0^c[\tau] R m B 1 ⋅ 3 K 4 m P 0 c [ τ ] . Adding them and using (52) with R ≥ 2 B 0 R\ge2B_0 R ≥ 2 B 0 gives
P [ L j ∣ τ ] ≤ 2 ( 8 B 0 + 4 B 1 ) 3 K R = O ( 1 K R ) . \mathbb{P}[L_j\mid\tau]\le\frac{2(8B_0+4B_1)}{3KR}=O\left(\frac{1}{KR}\right). P [ L j ∣ τ ] ≤ 3 K R 2 ( 8 B 0 + 4 B 1 ) = O ( K R 1 ) .
Averaging over τ \tau τ gives P [ L j ] = O ( 1 / ( K R ) ) \mathbb{P}[L_j]=O(1/(KR)) P [ L j ] = O ( 1/ ( K R )) , and summing over the q q q queries shows that the first stop is lucky with probability O ( q / ( K R ) ) O(q/(KR)) O ( q / ( K R )) .
The coupling: bounded reuse of weights
Lemma 7.5 shows that stops other than closure overflow are unlikely. In this subsection we show that closure overflow is unlikely too, and we compare the two experiments up to their first stop. The two experiments differ only in the weights, and the weights enter only through the potential stars. Before the first stop, the answers depend on the potential stars only through the stars visible from the exposed set Q Q Q : those with at least three members in Q Q Q , including both keys. We show that the visible stars have almost the same distribution in both experiments, in four steps.
§ 7.2.1 computes the transcript, up to the first stop, from weight-independent data and the stars visible from Q Q Q .
§ 7.2.2 shows that, with high probability, every small set of special terminals contains few filler pairs. This must hold for all sets at once, since Q Q Q depends on the run.
§ 7.2.3 proves a general lemma. Suppose that each possible support is first proposed with a probability that does not depend on the weights, and that each proposed support is then retained with probability equal to the product of its normalized weights. Then the two weight distributions give exactly the same distribution of what is revealed until some terminal lies in s + 1 s+1 s + 1 revealed proposals, and this is unlikely.
§ 7.2.4 applies this lemma to the visible stars and bounds the probability of closure overflow.
Simulating the experiment with star requests
We first reduce the comparison of the two experiments to a comparison of the potential stars visible from the exposed set. We call the bijection π \pi π together with all fillers F c F_c F c the common data , and denote it by Γ \Gamma Γ . They have the same distribution in both worlds and are independent of the weights. For Q ⊆ [ m ] Q\subseteq[m] Q ⊆ [ m ] , a potential star H c H_c H c is visible from Q Q Q if
∣ F c ∣ = 2 , F c ⊆ Q , ∣ H c ∩ Q ∣ ≥ 3. |F_c|=2,\qquad F_c\subseteq Q,\qquad|H_c\cap Q|\ge3. ∣ F c ∣ = 2 , F c ⊆ Q , ∣ H c ∩ Q ∣ ≥ 3.
A star request for Q Q Q returns exactly the existing potential stars visible from Q Q Q . Its reply includes the center labels and entire supports, even for discarded stars. An empty reply also counts as an answer. The star record is the sequence of requests and replies. Recall from § 7.1.1 that j ∗ j_\ast j ∗ is the first query at which a stop occurs (a cluster stop, a star-counter stop, a lucky query, notice overflow, or closure overflow), and j ∗ = q + 1 j_\ast=q+1 j ∗ = q + 1 if no stop occurs.
Lemma 7.6 (Simulation). There is a deterministic procedure, the simulator, that uses Γ \Gamma Γ and star requests and whose answers agree with the two-key experiment through every query before j ∗ j_\ast j ∗ . Its requests are for increasing sets Q Q Q of size at most 2 R 2R 2 R , each determined by Γ \Gamma Γ and earlier replies.
Proof. The simulator maintains the transcript, activation states, the sets B c B_c B c , the counters deg \deg deg , b F b^F b F , and N , Q \mathcal{N}, Q N , Q . It adds notices using the simulated answers and uses star requests to implement closure. If a notice or closure update would cross either size cap, it halts, and its remaining transcript is completed by any fixed answer rule, without further requests. Otherwise it repeats closure to completion.
For an inactive center it answers from F c F_c F c . For an active center it also uses every revealed potential star disjoint from B c B_c B c . Inductively these answers agree before j ∗ j_\ast j ∗ . Indeed, both keys of an active center lie in Q Q Q . If a queried non-key u ∈ Q u\in Q u ∈ Q belongs to H c H_c H c , then three star members lie in Q Q Q , and a request has revealed it. If u ∉ Q u\notin Q u ∈ / Q , the query is not lucky, so u ∉ H c u\notin H_c u ∈ / H c and the answer is 2. All other queries are determined by Γ \Gamma Γ alone. The same induction identifies the maintained Q Q Q with the exposed set. It also covers query j ∗ j_\ast j ∗ itself unless that query is lucky: the answer to it and the resulting updates of N \mathcal{N} N and Q Q Q agree as well. □ \square □
The simulator is only a device for comparing distributions. In particular, it need not know when an unobservable star-counter stop or lucky query occurs. We charge these events using Lemma 7.5 .
Uniform sparsity of the filler pairs
We view each pair filler F c = { u , v } F_c=\{u,v\} F c = { u , v } as an edge between u u u and v v v , labeled by c c c . Since several centers may have the same pair, these edges form a multigraph on [ m ] [m] [ m ] . For Q ⊆ [ m ] Q\subseteq[m] Q ⊆ [ m ] , let g ( Q ) = # { c : ∣ F c ∣ = 2 , F c ⊆ Q } g(Q)=\#\{c: |F_c|=2,\ F_c\subseteq Q\} g ( Q ) = # { c : ∣ F c ∣ = 2 , F c ⊆ Q } be the number of these edges inside Q Q Q .
Lemma 7.7 (Filler sparsity). If R ≥ 40 log m R\ge40\log m R ≥ 40 log m , then with probability at least 1 − m − 10 1-m^{-10} 1 − m − 10 every terminal belongs to at most 40 R 40R 40 R pair fillers and, simultaneously for every Q ⊆ [ m ] Q\subseteq[m] Q ⊆ [ m ] ,
g ( Q ) ≤ 40 ( R ∣ Q ∣ 2 m + ∣ Q ∣ log m ) . (53) g(Q) \le40\left(\frac{R|Q|^2}{m}+|Q|\log m\right). \tag*{(53)} g ( Q ) ≤ 40 ( m R ∣ Q ∣ 2 + ∣ Q ∣ log m ) . (53)
Proof. We use the following tail bound, whose proof is the standard derivation of the Chernoff bound. If X X X is a sum of independent Bernoulli variables and E [ X ] ≤ μ \mathbb{E}[X]\le\mu E [ X ] ≤ μ , then E [ e X ] ≤ e ( e − 1 ) μ \mathbb{E}[e^X]\le e^{(e-1)\mu} E [ e X ] ≤ e ( e − 1 ) μ (§ 2.1 ), and Markov’s inequality applied to e X e^X e X gives
P [ X ≥ t ] ≤ e ( e − 1 ) μ − t ≤ e − t / 2 for every t ≥ 4 μ . (54) \mathbb{P}[X\ge t]\le e^{(e-1)\mu-t}\le e^{-t/2}
\qquad\text{for every }t\ge4\mu. \tag*{(54)} P [ X ≥ t ] ≤ e ( e − 1 ) μ − t ≤ e − t /2 for every t ≥ 4 μ . (54)
There are m R mR m R centers, their fillers are independent, and each filler equals a given pair with probability β = 1 / ( 2 m ( m − 1 ) ) \beta=1/(2m(m-1)) β = 1/ ( 2 m ( m − 1 )) .
A fixed terminal lies in m − 1 m-1 m − 1 pairs. So the number of pair fillers containing it is a sum of m R mR m R independent Bernoulli variables, each with mean ( m − 1 ) β = 1 / ( 2 m ) (m-1)\beta=1/(2m) ( m − 1 ) β = 1/ ( 2 m ) , so its expectation is R / 2 R/2 R /2 . By (54) with t = 40 R t=40R t = 40 R , it is at least 40 R 40R 40 R with probability at most e − 20 R ≤ m − 20 e^{-20R}\le m^{-20} e − 20 R ≤ m − 20 , as R ≥ 40 log m R\ge40\log m R ≥ 40 log m . A union bound over the m m m terminals proves the degree statement, except with probability m − 19 m^{-19} m − 19 . Now fix Q Q Q of size h ≥ 1 h \ge1 h ≥ 1 . Then g ( Q ) g(Q) g ( Q ) is a sum of m R mR m R independent Bernoulli variables, each with mean ( h / 2 ) β (h/2)\beta ( h /2 ) β , so E [ g ( Q ) ] = R h ( h − 1 ) / ( 4 ( m − 1 ) ) ≤ R h 2 / ( 2 m ) \mathbb{E}[g(Q)] = Rh(h-1)/(4(m-1)) \le Rh^2/(2m) E [ g ( Q )] = R h ( h − 1 ) / ( 4 ( m − 1 )) ≤ R h 2 / ( 2 m ) . Apply (54) with t = 40 ( R h 2 / m + h log m ) ≥ 4 E [ g ( Q ) ] t = 40(Rh^2/m + h\log m) \ge4\mathbb{E}[g(Q)] t = 40 ( R h 2 / m + h log m ) ≥ 4 E [ g ( Q )] : the bound (53) fails with probability at most e − 20 h log m = m − 20 h e^{-20h\log m} = m^{-20h} e − 20 h l o g m = m − 20 h . There are at most m h m^h m h sets of size h h h , so (53) fails for some Q Q Q with probability at most ∑ h ≥ 1 m − 19 h ≤ 2 m − 19 \sum_{h\ge1}m^{-19h} \le2m^{-19} ∑ h ≥ 1 m − 19 h ≤ 2 m − 19 . Altogether, the failure probability is at most 3 m − 19 ≤ m − 10 3m^{-19} \le m^{-10} 3 m − 19 ≤ m − 10 .
The uniformity is essential: Q Q Q depends on the transcript and is not an independently sampled set. We call the common data Γ \Gamma Γ good if every terminal belongs to at most 40 R 40R 40 R pair fillers and (53) holds for every Q ⊆ [ m ] Q \subseteq[m] Q ⊆ [ m ] . By Lemma 7.7 , Γ \Gamma Γ is good with probability at least 1 − m − 10 1-m^{-10} 1 − m − 10 .
A discrete moment-matching lemma
The goal of this subsection is a general lemma behind the moment matching. We generate random supports in two stages: each possible support is first proposed with a probability that does not depend on the weights, and a proposed support is then retained with probability equal to the product of its normalized weights. A protocol may observe the proposals adaptively, the rejected ones included. We first show (Claim 7 ) that a fixed pattern of retained and rejected supports has the same probability in both worlds, on average over the weights, as long as no terminal lies in more than s s s of these supports. Lemma 7.8 extends this to adaptive protocols and shows that some terminal lies in s + 1 s+1 s + 1 revealed proposals only with small probability. § 7.2.4 applies the lemma to the potential stars visible from the exposed set.
For a support H ⊆ [ m ] H \subseteq[m] H ⊆ [ m ] , write p H ( W ) = ∏ u ∈ H ( W u / M ) ∈ [ 0 , 1 ] p_H(W) = \prod_{u\in H}(W_u/M) \in[0,1] p H ( W ) = ∏ u ∈ H ( W u / M ) ∈ [ 0 , 1 ] .
Claim 7 (Moment identity). Let H 1 , … , H ℓ H_1,\ldots,H_\ell H 1 , … , H ℓ be supports such that every terminal lies in at most s s s of them. Then for all b 1 , … , b ℓ ∈ { 0 , 1 } b_1,\ldots,b_\ell\in\{0,1\} b 1 , … , b ℓ ∈ { 0 , 1 } , the expectation
E [ ∏ i : b i = 1 p H i ( W ) ∏ i : b i = 0 ( 1 − p H i ( W ) ) ] (55) \mathbb{E}\left[\prod_{i:b_i=1}p_{H_i}(W)\prod_{i:b_i=0}(1-p_{H_i}(W))\right] \tag*{(55)} E [ i : b i = 1 ∏ p H i ( W ) i : b i = 0 ∏ ( 1 − p H i ( W )) ] (55) has the same value under the two weight distributions.
Proof. Let I = { i : b i = 1 } I=\{i:b_i=1\} I = { i : b i = 1 } . Expanding each factor 1 − p H i ( W ) 1-p_{H_i}(W) 1 − p H i ( W ) gives
∏ i ∈ I p H i ( W ) ∏ i ∉ I ( 1 − p H i ( W ) ) = ∑ J ⊆ [ ℓ ] ∖ I ( − 1 ) ∣ J ∣ ∏ i ∈ I ∪ J p H i ( W ) = ∑ J ⊆ [ ℓ ] ∖ I ( − 1 ) ∣ J ∣ ∏ u ∈ [ m ] ( W u M ) k u ( J ) . \prod_{i\in I}p_{H_i}(W)\prod_{i\notin I}(1-p_{H_i}(W))
= \sum_{J\subseteq[\ell]\setminus I}(-1)^{|J|}\prod_{i\in I\cup J}p_{H_i}(W)
= \sum_{J\subseteq[\ell]\setminus I}(-1)^{|J|}\prod_{u\in[m]}\left(\frac{W_u}{M}\right)^{k_u(J)}. i ∈ I ∏ p H i ( W ) i ∈ / I ∏ ( 1 − p H i ( W )) = J ⊆ [ ℓ ] ∖ I ∑ ( − 1 ) ∣ J ∣ i ∈ I ∪ J ∏ p H i ( W ) = J ⊆ [ ℓ ] ∖ I ∑ ( − 1 ) ∣ J ∣ u ∈ [ m ] ∏ ( M W u ) k u ( J ) . where k u ( J ) = # { i ∈ I ∪ J : u ∈ H i } ≤ s k_u(J)=\#\{i\in I\cup J:u\in H_i\}\le s k u ( J ) = # { i ∈ I ∪ J : u ∈ H i } ≤ s . As the weights are independent, the term for J J J has expectation ∏ u E [ ( W u / M ) k u ( J ) ] \prod_{u}\mathbb{E}[(W_u/M)^{k_u(J)}] ∏ u E [( W u / M ) k u ( J ) ] , and by (35) each factor is the same under the two distributions.
Claim 7 concerns a fixed collection of supports. We now describe the random two-stage process that generates the supports, and how an adaptive protocol observes it. The protocol is only an analysis device. It answers no distance queries and only sees which supports are proposed and which are retained. In § 7.2.4 , it will be the simulator of Lemma 7.6 , whose star requests ask for the supports visible from the exposed set. There the potential stars are drawn by this two-stage process, so a center may have several retained supports, and Lemma 7.9 shows that this is close to the two-key experiment, in which every center has at most one potential star.
Proposals. Fix a finite set of types . A type x x x is an abstract label that comes with a support H x ⊆ [ m ] H_x \subseteq[m] H x ⊆ [ m ] of size at most d d d and a proposal probability λ x ∈ [ 0 , 1 ] \lambda_x \in[0,1] λ x ∈ [ 0 , 1 ] . Different types may have the same support and the same proposal probability. Independently of the weights, and independently across types, each type x x x is proposed with probability λ x \lambda_x λ x . A proposed type is called a proposal . Conditional on the weights, each proposal x x x is then retained independently with probability p H x ( W ) p_{H_x}(W) p H x ( W ) , and rejected otherwise. In § 7.2.4 , the types will be the pairs ( c , H ) (c,H) ( c , H ) of a center c c c and a possible support H H H of its potential star. The factors for b i = 0 b_i=0 b i = 0 in (55) account for rejected proposals: rejection also depends on the weights.
Adaptive requests. A region is a set of types. A protocol observes the proposals through requests . It requests regions D 1 ⊆ D 2 ⊆ ⋯ D_1 \subseteq D_2 \subseteq\cdots D 1 ⊆ D 2 ⊆ ⋯ one by one, and the answer to the request for D k D_k D k reveals which types in D k ∖ D k − 1 D_k \setminus D_{k-1} D k ∖ D k − 1 are proposed, together with the retention bits of these proposals (where D 0 = ∅ D_0=\varnothing D 0 = ∅ ). Each region is chosen using only the earlier answers and fixed side information, which is independent of the weights and of the proposal and retention bits. The record is the sequence of requests and answers, and the revealed proposals are the proposals in the requested regions.
Lemma 7.8 (Adaptive Bernoulli proposals). Suppose that every region D D D requested by the protocol satisfies
∑ x ∈ D λ x ≤ L ∗ , and ∑ x ∈ D u ∈ H x λ x ≤ G ∗ for every u . (56) \sum_{x\in D}\lambda_x \le L_*, \qquad\text{and} \qquad\sum_{\substack{x\in D\\u\in H_x}}\lambda_x \le G_* \quad\text{for every }u. \tag*{(56)} x ∈ D ∑ λ x ≤ L ∗ , and x ∈ D u ∈ H x ∑ λ x ≤ G ∗ for every u . (56) Then the total variation distance between the distributions of the record in the two worlds is at most d L ∗ G ∗ s dL_*G_*^s d L ∗ G ∗ s . In either world, the expected number of revealed proposals is at most L ∗ L_* L ∗ .
Proof. Fix the side information, that is, the data that the protocol knows in advance (in § 7.2.4 , these are the common data Γ \Gamma Γ ). We may assume that the protocol reads the types of each newly requested region one at a time, in a fixed order: for a type x x x it first reads the proposal bit a x a_x a x (a x = 1 a_x=1 a x = 1 if x x x is proposed) and then, if a x = 1 a_x=1 a x = 1 , the retention bit b x b_x b x (b x = 1 b_x=1 b x = 1 if x x x is retained). Which types are read, and in which order, depends only on the bits read so far. For a terminal u u u , let N u N_u N u be the number of proposed types read so far whose support contains u u u .
Step 1: equal distributions until some weight is used s + 1 s+1 s + 1 times. Stop reading just before the retention bit of the first proposed type x x x whose proposal makes N u = s + 1 N_u=s+1 N u = s + 1 for some u ∈ H x u\in H_x u ∈ H x . Consider a possible stopped record r r r . Let X r X_r X r be the set of types whose proposal bits are read in r r r , and X r + ⊆ X r X_r^+ \subseteq X_r X r + ⊆ X r the set of types whose retention bits are read in r r r . Both sets are determined by r r r , and
P [ r ] = ∏ x ∈ X r λ x a x ( 1 − λ x ) 1 − a x ⋅ E [ ∏ x ∈ X r + p H x ( W ) b x ( 1 − p H x ( W ) ) 1 − b x ] . \mathbb{P}[r]=\prod_{x\in X_r}\lambda_x^{a_x}(1-\lambda_x)^{1-a_x}\cdot\mathbb{E}\left[\prod_{x\in X_r^+}p_{H_x}(W)^{b_x}(1-p_{H_x}(W))^{1-b_x}\right]. P [ r ] = x ∈ X r ∏ λ x a x ( 1 − λ x ) 1 − a x ⋅ E x ∈ X r + ∏ p H x ( W ) b x ( 1 − p H x ( W ) ) 1 − b x .
The first factor does not depend on the weights. By the choice of the stopping time, every terminal lies in at most s s s of the supports H x H_x H x with x ∈ X r + x\in X_r^+ x ∈ X r + , so by Claim 7 the second factor is the same under both weight distributions. Hence the stopped record has the same distribution in the two worlds. Couple the two runs so that their stopped records coincide, and continue them independently. If the stopping rule never applies, the stopped record is the full record. So the full records can differ only if, in the full run, N u ≥ s + 1 N_u\ge s+1 N u ≥ s + 1 at the end for some u u u . By Lemma 2.2(a), the total variation distance is at most the probability of this event.
Step 2: some weight is rarely used s + 1 s+1 s + 1 times. Consider the full run, without stopping. For a read type x x x , let N u − ( x ) N_u^-(x) N u − ( x ) be the value of N u N_u N u just before x x x is read. When N u N_u N u increases from k k k to k + 1 k+1 k + 1 , the value of ( N u j + 1 ) \binom{N_u}{j+1} ( j + 1 N u ) increases by ( k j ) \binom{k}{j} ( j k ) , by Pascal’s rule. Hence, at the end of the run, for every integer j ≥ 0 , j \ge0, j ≥ 0 ,
( N u j + 1 ) = ∑ x r e a d u ∈ H x a x ( N u − ( x ) j ) \binom{N_u}{j+1}=\sum_{\substack{x\ \mathrm{read}\\u\in H_x}}a_x\binom{N_u^-(x)}{j} ( j + 1 N u ) = x read u ∈ H x ∑ a x ( j N u − ( x ) )
Just before x x x is read, a x a_x a x is a Ber ( λ x ) \operatorname{Ber}(\lambda_x) Ber ( λ x ) variable independent of everything read so far, and in particular of whether x x x is read and of N u − ( x ) N_u^-(x) N u − ( x ) . Taking expectations, we get
E [ ( N u j + 1 ) ] = E [ ∑ x r e a d u ∈ H x λ x ( N u − ( x ) j ) ] ≤ G ∗ E [ ( N u j ) ] . \mathbb{E}\left[\binom{N_u}{j+1}\right]=\mathbb{E}\left[\sum_{\substack{x\ \mathrm{read}\\u\in H_x}}\lambda_x\binom{N_u^-(x)}{j}\right]\le G_*\mathbb{E}\left[\binom{N_u}{j}\right]. E [ ( j + 1 N u ) ] = E x read u ∈ H x ∑ λ x ( j N u − ( x ) ) ≤ G ∗ E [ ( j N u ) ] .
The inequality uses N u − ( x ) ≤ N u N_u^-(x)\le N_u N u − ( x ) ≤ N u and the fact that the read types form the last requested region, so by (56) their values λ x \lambda_x λ x with u ∈ H x u\in H_x u ∈ H x sum to at most G ∗ G_* G ∗ . Applying this for j = s , s − 1 , … , 1 j=s,s-1,\ldots,1 j = s , s − 1 , … , 1 gives E [ ( N u s + 1 ) ] ≤ G ∗ s E [ N u ] \mathbb{E}\left[\binom{N_u}{s+1}\right]\le G_*^s\mathbb{E}[N_u] E [ ( s + 1 N u ) ] ≤ G ∗ s E [ N u ] . Let Y Y Y be the number of proposed types read, that is, of revealed proposals. Then ∑ u N u ≤ d Y \sum_uN_u\le dY ∑ u N u ≤ d Y , and the same argument with all read types in place of those containing u u u gives E [ Y ] = E [ ∑ x r e a d λ x ] ≤ L ∗ \mathbb{E}[Y]=\mathbb{E}[\sum_{x\ \mathrm{read}}\lambda_x]\le L_* E [ Y ] = E [ ∑ x read λ x ] ≤ L ∗ , which is the last statement of the lemma. Therefore
P [ ∃ u : N u ≥ s + 1 ] ≤ ∑ u E [ ( N u s + 1 ) ] ≤ G ∗ s ∑ u E [ N u ] ≤ d L ∗ G ∗ s . \mathbb{P}[\exists u:N_u\ge s+1]\le\sum_u\mathbb{E}\left[\binom{N_u}{s+1}\right]\le G_*^s\sum_u\mathbb{E}[N_u]\le dL_*G_*^s. P [ ∃ u : N u ≥ s + 1 ] ≤ u ∑ E [ ( s + 1 N u ) ] ≤ G ∗ s u ∑ E [ N u ] ≤ d L ∗ G ∗ s .
Applying the lemma to visible stars
Condition on good common data Γ \Gamma Γ . For each center with pair filler and each d d d -set H ⊇ F c H\supseteq F_c H ⊇ F c , introduce the type x = ( c , H ) x=(c,H) x = ( c , H ) , with support H H H and proposal probability
λ ∗ = H 0 M d d R β ( m d ) . (57) \lambda_*=\frac{H_0M^d}{dR\beta\binom{m}{d}}. \tag*{(57)} λ ∗ = d R β ( d m ) H 0 M d . (57)
For large parameters λ ∗ < 1 \lambda_*<1 λ ∗ < 1 . Retaining a proposal with probability p H ( W ) p_H(W) p H ( W ) gives probability
r H : = λ ∗ p H ( W ) = a W ( H ) β , r_H:=\lambda_*p_H(W)=\frac{a_W(H)}{\beta}, r H := λ ∗ p H ( W ) = β a W ( H ) ,
exactly the mass of H H H in (40) . The types whose supports are visible from Q Q Q form the region
D ( Q ) = { ( c , H ) : ∣ F c ∣ = 2 , F c ⊆ Q , ∣ H ∩ Q ∣ ≥ 3 } . \mathcal{D}(Q)=\{(c,H):|F_c|=2,\ F_c\subseteq Q,\ |H\cap Q|\ge3\}. D ( Q ) = {( c , H ) : ∣ F c ∣ = 2 , F c ⊆ Q , ∣ H ∩ Q ∣ ≥ 3 } .
These regions grow with Q Q Q .
Budgets. Assume R ≥ 40 log m R\ge40\log m R ≥ 40 log m and 2 R ≤ m 2R\le m 2 R ≤ m , and put
B ∗ = 160 ( R 3 m + R log m ) , L ∗ = 2 B 1 B ∗ m , G ∗ = 240 B 1 R m , (58) B_*=160\left(\frac{R^3}{m}+R\log m\right),\qquad L_*=\frac{2B_1B_*}{m},\qquad G_*=\frac{240B_1R}{m}, \tag*{(58)} B ∗ = 160 ( m R 3 + R log m ) , L ∗ = m 2 B 1 B ∗ , G ∗ = m 240 B 1 R , (58)
with B 1 B_1 B 1 as in (42) . We show that, for every Q Q Q with ∣ Q ∣ ≤ 2 R |Q|\le2R ∣ Q ∣ ≤ 2 R , at most B ∗ B_* B ∗ centers have their filler pair in Q Q Q , and the region D ( Q ) \mathcal{D}(Q) D ( Q ) satisfies (56) with these L ∗ L_* L ∗ and G ∗ G_* G ∗ .
The first claim is (53) with ∣ Q ∣ ≤ 2 R |Q|\le2R ∣ Q ∣ ≤ 2 R . For the others, fix a pair e e e and a terminal u ∉ e u\notin e u ∈ / e . There are ( m − 3 d − 3 ) \binom{m-3}{d-3} ( d − 3 m − 3 ) sets H ⊇ e ∪ { u } H\supseteq e\cup\{u\} H ⊇ e ∪ { u } of size d d d , and, as in the derivation of (42) ,
λ ∗ ( m − 3 d − 3 ) = H 0 M d d R β ( m − 3 d − 3 ) ( m d ) = ( d − 2 ) B 0 R ( m − 2 ) ≤ B 1 R m . (59) \lambda_*\binom{m-3}{d-3}=\frac{H_0M^d}{dR\beta}\frac{\binom{m-3}{d-3}}{\binom{m}{d}}=\frac{(d-2)B_0}{R(m-2)}\le\frac{B_1}{Rm}. \tag*{(59)} λ ∗ ( d − 3 m − 3 ) = d R β H 0 M d ( d m ) ( d − 3 m − 3 ) = R ( m − 2 ) ( d − 2 ) B 0 ≤ R m B 1 . (59)
Consider a center c c c with F c ⊆ Q F_c \subseteq Q F c ⊆ Q . Every type ( c , H ) (c,H) ( c , H ) in D ( Q ) \mathcal{D}(Q) D ( Q ) has a member of Q ∖ F c Q \setminus F_c Q ∖ F c in H H H . So, by (59) , the types of c c c in D ( Q ) \mathcal{D}(Q) D ( Q ) have total proposal probability at most ∣ Q ∣ B 1 / ( R m ) ≤ 2 B 1 / m |Q|B_1/(Rm) \le2B_1/m ∣ Q ∣ B 1 / ( R m ) ≤ 2 B 1 / m . Summing over the at most B ∗ B_* B ∗ such centers gives L ∗ L_* L ∗ . Now fix a terminal u u u . By Lemma 7.7 , at most 40 R 40R 40 R centers have u ∈ F c u \in F_c u ∈ F c , and they contribute at most 40 R ⋅ 2 B 1 / m 40R \cdot2B_1/m 40 R ⋅ 2 B 1 / m . Every other center with F c ⊆ Q F_c \subseteq Q F c ⊆ Q contributes at most B 1 / ( R m ) B_1/(Rm) B 1 / ( R m ) , by (59) with e = F c e=F_c e = F c . Together, these centers contribute at most B ∗ B 1 / ( R m ) ≤ 160 B 1 R / m B_*B_1/(Rm) \le160B_1R/m B ∗ B 1 / ( R m ) ≤ 160 B 1 R / m , since B ∗ / R ≤ 160 ( R / 2 + R / 40 ) ≤ 160 R B_*/R \le160(R/2+R/40) \le160R B ∗ / R ≤ 160 ( R /2 + R /40 ) ≤ 160 R . The total is at most G ∗ G_* G ∗ .
Two models for the stars. We call the way the two-key experiment draws potential stars, (40) , the star model : given its keys and the weights, a center has potential star H H H with probability r H r_H r H and none with the remaining probability, so it has at most one potential star. In the proposal model , every type ( c , H ) (c,H) ( c , H ) is proposed and retained independently, so H H H is retained at c c c with probability r H r_H r H , and a center may have several retained supports. The simulator then treats each retained support as a potential star of c c c .
Lemma 7.9 (Visible-star coupling). Given good common data, the distributions of the star record in the two worlds differ by at most
2 B 0 2 B ∗ R 2 + d L ∗ G ∗ . (60) \frac{2B_0^2B_*}{R^2}+dL_*G_*. \tag*{(60)} R 2 2 B 0 2 B ∗ + d L ∗ G ∗ . (60) In either world, the probability that closure overflow is the first stop is at most
B 0 2 B ∗ R 2 + d L ∗ R . (61) \frac{B_0^2B_*}{R^2}+\frac{dL_*}{R}. \tag*{(61)} R 2 B 0 2 B ∗ + R d L ∗ . (61)
Proof. Fix one world. For a center c c c with a pair filler, let v c = ∑ H ⊇ F c r H v_c=\sum_{H\supseteq F_c}r_H v c = ∑ H ⊇ F c r H . Then v c ≤ B 0 / R v_c\le B_0/R v c ≤ B 0 / R by (42) .
One center. Given the weights, the set of supports at c c c has distributions at total variation distance at most v c 2 v_c^2 v c 2 in the two models. Indeed, the star model gives probability r H r_H r H to each singleton { H } \{H\} { H } and 1 − v c 1-v_c 1 − v c to the empty set. The proposal model gives no less probability to the empty set, since ∏ H ( 1 − r H ) ≥ 1 − v c \prod_H(1-r_H)\ge1-v_c ∏ H ( 1 − r H ) ≥ 1 − v c , and no more to any singleton. Its other outcomes have at least two supports. So the distance is the total singleton deficit
∑ H r H ( 1 − ∏ H ′ ≠ H ( 1 − r H ′ ) ) ≤ ∑ H r H ∑ H ′ ≠ H r H ′ ≤ v c 2 . \sum_H r_H\left(1-\prod_{H'\ne H}(1-r_{H'})\right)\le\sum_H r_H\sum_{H'\ne H}r_{H'}\le v_c^2. H ∑ r H 1 − H ′ = H ∏ ( 1 − r H ′ ) ≤ H ∑ r H H ′ = H ∑ r H ′ ≤ v c 2 .
From the star model to the proposal model. Call a center opened at the first request whose set contains its filler pair, and order the opened centers by this time, breaking ties by a fixed order. Before c c c is opened, no request has involved a support at c c c , so the supports at c c c can be sampled when c c c is opened. Every opened center has its filler pair in the last requested set, which has size at most 2 R 2R 2 R , so at most B ∗ B_* B ∗ centers are opened. For 0 ≤ k ≤ B ∗ 0\le k\le B_* 0 ≤ k ≤ B ∗ , let M k M_k M k be the run in which the first k k k opened centers use the proposal model and all other centers use the star model. Thus M 0 M_0 M 0 uses the star model, and M B ∗ M_{B_*} M B ∗ has the same star record as the proposal model. The runs M k − 1 M_{k-1} M k − 1 and M k M_k M k agree until the k k k th center is opened. At that moment, given the weights and the record so far, the supports at that center still have their original distribution in either model, since different centers are independent given the weights. Couple them as in the previous paragraph, and use the same randomness for everything else. Then the two star records differ with probability at most B 0 2 / R 2 B_0^2/R^2 B 0 2 / R 2 . By Lemma 2.2 (a) and the triangle inequality, the star records of the two models are within total variation distance B 0 2 B ∗ / R 2 B_0^2B_*/R^2 B 0 2 B ∗ / R 2 .
Comparing the worlds. In the proposal model, a star request for Q Q Q amounts to requesting the region D ( Q ) \mathcal{D}(Q) D ( Q ) , where ∣ Q ∣ ≤ 2 R |Q|\le2R ∣ Q ∣ ≤ 2 R and Q Q Q is chosen using only the common data and earlier replies, so the common data are the side information of Lemma 7.8 . Give the simulator the stronger record of all proposals in the requested regions and their retention bits. It uses only the retained supports. By the budgets and Lemma 7.8 , the distributions of this stronger record in the two worlds are within d L ∗ G ∗ s dL_*G_*^s d L ∗ G ∗ s , and hence so are those of the star record, which is a function of it (Lemma 2.2 (b)). Adding the error B 0 2 B ∗ / R 2 B_0^2B_*/R^2 B 0 2 B ∗ / R 2 in each world proves (60) .
Closure overflow. Suppose that closure overflow is the first stop. Since closure overflow comes last in the tie order, at most R R R terminals are noticed at that query, so closure has added more than R R R terminals to Q Q Q . Each closure step adds at most d d d terminals, all from one star visible from the current Q Q Q , which has size at most $2R$. By the proof of Lemma 7.6 , the simulator performs the same closure steps, so its star record reveals more than R / d R/d R / d stars. In the proposal model, revealed supports are revealed proposals, whose expected number is at most $L_* by {xref}Lemma 7.8. By Markov’s inequality, more than $R/d$ of them are revealed with probability at most $dL_*/R$. Adding the error $B_0^2B_*/R^2$ between the two models proves {xref}(7.16)`.
Proof of Theorem 1.2
We assemble the argument before choosing the constants in (33) . The resulting bound displays the two main costs: finding hidden special terminals, and reusing a weight beyond the matched moment order.
Lemma 7.10 (The two transcripts are close). Suppose C log m ≤ R C\log m \le R C log m ≤ R , 2 R ≤ m 2R \le m 2 R ≤ m , R log m ≤ m R\log m \le m R log m ≤ m , and q ≤ K R q \le KR q ≤ K R . For every deterministic algorithm making at most q q q queries,
d T V ( trans ( A , D y ) , trans ( A , D n ) ) ≤ C ( q m R 2 + q K R + q A K + R s + 3 m s + 2 + R m + log m R ) + m − 10 . (62) d_{\mathrm{TV}}\bigl(\operatorname{trans}(\mathcal{A},\mathcal{D}_y),\operatorname{trans}(\mathcal{A},\mathcal{D}_n)\bigr)
\le C\left(\sqrt{\frac{q}{mR^2}}+\frac{q}{KR}+\frac{q}{AK}+\frac{R^{s+3}}{m^{s+2}}+\frac{R}{m}+\frac{\log m}{R}\right)+m^{-10}. \tag*{(62)} d TV ( trans ( A , D y ) , trans ( A , D n ) ) ≤ C ( m R 2 q + K R q + A K q + m s + 2 R s + 3 + m R + R log m ) + m − 10 . (62)
Proof. For ∙ ∈ { y , n } \bullet\in\{\mathrm{y},\mathrm{n}\} ∙ ∈ { y , n } , write T ∙ = trans ( A , D ∙ ) T_\bullet=\operatorname{trans}(\mathcal{A},\mathcal{D}_\bullet) T ∙ = trans ( A , D ∙ ) for the transcript distribution in the metric and E ∙ = trans ( A , G ∙ ) E_\bullet=\operatorname{trans}(\mathcal{A},\mathcal{G}_\bullet) E ∙ = trans ( A , G ∙ ) for that in the two-key experiment. By the triangle inequality,
d T V ( T y , T n ) ≤ d T V ( T y , E y ) + d T V ( E y , E n ) + d T V ( E n , T n ) . (63) d_{\mathrm{TV}}(T_y,T_n)\le d_{\mathrm{TV}}(T_y,E_y)+d_{\mathrm{TV}}(E_y,E_n)+d_{\mathrm{TV}}(E_n,T_n). \tag*{(63)} d TV ( T y , T n ) ≤ d TV ( T y , E y ) + d TV ( E y , E n ) + d TV ( E n , T n ) . (63)
By Lemma 6.1 , the first and last terms are at most 2 C q / ( m R 2 ) 2C\sqrt{q/(mR^2)} 2 C q / ( m R 2 ) each, since q / ( m R 2 ) = q / ( d K R ) ≤ 1 q/(mR^2)=q/(dKR)\le1 q / ( m R 2 ) = q / ( d K R ) ≤ 1 .
For the middle term, the common data Γ \Gamma Γ are good except with probability at most m − 10 m^{-10} m − 10 (Lemma 7.7 ). Since Γ \Gamma Γ has the same distribution in both experiments, Lemma 2.2 (c) bounds d T V ( E y , E n ) d_{\mathrm{TV}}(E_y,E_n) d TV ( E y , E n ) by the average over Γ \Gamma Γ of the distance between the conditional distributions given Γ \Gamma Γ . Bad Γ \Gamma Γ contribute at most m − 10 m^{-10} m − 10 to this average. Now fix good Γ \Gamma Γ . In each experiment, the output of the simulator is a function of the star record, and by Lemma 7.6 it equals the transcript unless a stop occurs. So, by Lemma 2.2 (a,b) and Lemma 7.9 , the conditional distance given Γ \Gamma Γ is at most
P y [ a stop occurs ∣ Γ ] + P n [ a stop occurs ∣ Γ ] + 2 B 0 2 B ∗ R 2 + d L ∗ G ∗ s . \mathbb{P}_y[\text{a stop occurs}\mid\Gamma]+\mathbb{P}_n[\text{a stop occurs}\mid\Gamma]+\frac{2B_0^2B_*}{R^2}+dL_*G_*^s. P y [ a stop occurs ∣ Γ ] + P n [ a stop occurs ∣ Γ ] + R 2 2 B 0 2 B ∗ + d L ∗ G ∗ s .
If a stop occurs, then either the first stop is not closure overflow, or it is closure overflow. After averaging over Γ \Gamma Γ , the first case has probability at most C ( q / ( A K ) + q / ( K R ) ) C(q/(AK)+q/(KR)) C ( q / ( A K ) + q / ( K R )) in each experiment, by Lemma 7.5 . The second case has probability at most B 0 2 B ∗ / R 2 + d L ∗ / R B_0^2B_*/R^2+dL_*/R B 0 2 B ∗ / R 2 + d L ∗ / R , by Lemma 7.9 . Altogether,
d T V ( E y , E n ) ≤ m − 10 + 2 C ( q A K + q K R ) + 4 B 0 2 B ∗ R 2 + d L ∗ G ∗ s + 2 d L ∗ R . (64) d_{\mathrm{TV}}(E_y,E_n)\le m^{-10}+2C\left(\frac{q}{AK}+\frac{q}{KR}\right)+\frac{4B_0^2B_*}{R^2}+dL_*G_*^s+\frac{2dL_*}{R}. \tag*{(64)} d TV ( E y , E n ) ≤ m − 10 + 2 C ( A K q + K R q ) + R 2 4 B 0 2 B ∗ + d L ∗ G ∗ s + R 2 d L ∗ . (64)
By (58) , the last three terms satisfy
B ∗ R 2 = O ( R m + log m R ) , \frac{B_*}{R^2}=O\left(\frac{R}{m}+\frac{\log m}{R}\right), R 2 B ∗ = O ( m R + R log m ) ,
d L ∗ G ∗ s = O ( R s + 3 m s + 2 + ( R m ) s + 1 log m ) , dL_*G_*^s=O\left(\frac{R^{s+3}}{m^{s+2}}+\left(\frac{R}{m}\right)^{s+1}\log m\right), d L ∗ G ∗ s = O ( m s + 2 R s + 3 + ( m R ) s + 1 log m ) ,
d L ∗ R = O ( R 2 m 2 + log m m ) . \frac{dL_*}{R}=O\left(\frac{R^2}{m^2}+\frac{\log m}{m}\right). R d L ∗ = O ( m 2 R 2 + m log m ) .
As s ≥ 1 s\ge1 s ≥ 1 , R log m ≤ m R\log m\le m R log m ≤ m , and log m ≤ R \log m\le R log m ≤ R , the secondary term in the second line and both terms in the third are O ( R / m ) O(R/m) O ( R / m ) . Inserting these bounds and (64) into (63) gives (62) .
Choosing the parameters. Recall A = t s + 3 A=t^{s+3} A = t s + 3 , R = ⌊ γ 0 t s + 2 ⌋ R=\lfloor\gamma_0t^{s+2}\rfloor R = ⌊ γ 0 t s + 2 ⌋ , K = A R K=AR K = A R , and m = d A m=dA m = d A . Take q = ⌊ c 0 K R ⌋ q=\lfloor c_0KR\rfloor q = ⌊ c 0 K R ⌋ for a small c 0 ∈ ( 0 , 1 ) c_0\in(0,1) c 0 ∈ ( 0 , 1 ) . Then
q m R 2 ≤ c 0 d , q K R ≤ c 0 , q A K ≤ c 0 R A = o ( 1 ) , R s + 3 m s + 2 ≤ γ 0 s + 3 d s + 2 . \frac{q}{mR^2}\le\frac{c_0}{d},\qquad
\frac{q}{KR}\le c_0,\qquad
\frac{q}{AK}\le\frac{c_0R}{A}=o(1),\qquad
\frac{R^{s+3}}{m^{s+2}}\le\frac{\gamma_0^{s+3}}{d^{s+2}}. m R 2 q ≤ d c 0 , K R q ≤ c 0 , A K q ≤ A c 0 R = o ( 1 ) , m s + 2 R s + 3 ≤ d s + 2 γ 0 s + 3 .
The remaining terms of (62) tend to zero, and all its side conditions hold for large t t t . Choose γ 0 \gamma_0 γ 0 and then c 0 c_0 c 0 small enough that the transcript distance is at most 1 / 10 1/10 1/10 for all large t t t . The sizes satisfy
n = Θ ( t 2 s + 5 ) , K R = Θ ( t 3 s + 7 ) = Θ ( n ( 3 s + 7 ) / ( 2 s + 5 ) ) . (65) n=\Theta(t^{2s+5}),\qquad KR=\Theta(t^{3s+7})=\Theta\left(n^{(3s+7)/(2s+5)}\right). \tag*{(65)} n = Θ ( t 2 s + 5 ) , K R = Θ ( t 3 s + 7 ) = Θ ( n ( 3 s + 7 ) / ( 2 s + 5 ) ) . (65)
Proof of Theorem 1.2 . Draw the input from μ = 1 2 D y + 1 2 D n \mu=\frac{1}{2}D_y+\frac{1}{2}D_n μ = 2 1 D y + 2 1 D n . By Proposition 2.3 and Lemma 7.10 , any deterministic algorithm with at most q q q queries guesses its world correctly with probability at most ( 1 + 1 / 10 ) / 2 = 11 / 20 (1+1/10)/2=11/20 ( 1 + 1/10 ) /2 = 11/20 .
Turn an estimator into a guess by answering YES precisely when C ^ ≤ ( 2 − η ) U m \widehat{C}\le(2-\eta)U_m C ≤ ( 2 − η ) U m . Whenever the estimator is correct and the cost bound of Lemma 5.3 holds, this guess is correct: in YES, C ^ ≤ ( 2 − η ) O P T ≤ ( 2 − η ) U m \widehat{C}\le(2-\eta)\mathrm{OPT}\le(2-\eta)U_m C ≤ ( 2 − η ) OPT ≤ ( 2 − η ) U m , whereas in NO, C ^ ≥ O P T ≥ V m > ( 2 − η ) U m \widehat{C}\ge\mathrm{OPT}\ge V_m>(2-\eta)U_m C ≥ OPT ≥ V m > ( 2 − η ) U m . Thus any such estimator succeeds under μ \mu μ with probability at most 11 / 20 + e − Ω ( m ) < 17 / 30 11/20+e^{-\Omega(m)}<17/30 11/20 + e − Ω ( m ) < 17/30 for large t t t . Averaging over random bits gives the same bound for randomized algorithms with at most q q q queries.
All input sizes. Let n t n_t n t be the size in (34) . Since n t + 1 / n t → 1 n_{t+1}/n_t\to1 n t + 1 / n t → 1 , for any sufficiently large n n n choose the largest t t t with n t ≤ n n_t\le n n t ≤ n and add n − n t n-n_t n − n t publicly named nonterminals at distance 2 2 2 from all other points. The result is a metric and the optimum is unchanged. Indeed, a new nonterminal of degree j ≥ 2 j\ge2 j ≥ 2 can be removed by joining its former neighbors with a path of j − 1 j-1 j − 1 edges of weight at most 2 2 2 , while a new leaf can simply be deleted. Queries involving these points have fixed answers and give no information. The lower bound therefore applies to every sufficiently large n n n , with n = Θ ( n t ) n=\Theta(n_t) n = Θ ( n t ) , Θ ( n ) \Theta(n) Θ ( n ) terminals, and λ = K − 3 = Θ ( n − 3 ) \lambda=K^{-3}=\Theta(n^{-3}) λ = K − 3 = Θ ( n − 3 ) .
Expected query complexity. Suppose an algorithm succeeds with probability at least 2 / 3 2/3 2/3 on every instance and uses at most q / 10 q/10 q /10 queries in expectation on every instance. Truncating it after q q q queries loses at most 1 / 10 1/10 1/10 success probability, by Markov’s inequality. Its success under μ \mu μ would then be at least 2 / 3 − 1 / 10 = 17 / 30 2/3-1/10=17/30 2/3 − 1/10 = 17/30 , contradicting the preceding bound. Some instance therefore requires more than q / 10 = Ω ( n b s ) q/10=\Omega(n^{b_s}) q /10 = Ω ( n b s ) expected queries, with b s b_s b s as in (65) . Finally, b s = 3 / 2 − 1 / ( 4 s + 10 ) b_s=3/2-1/(4s+10) b s = 3/2 − 1/ ( 4 s + 10 ) tends to 3 / 2 3/2 3/2 , proving Corollary 1.3.
Conclusion and AI Disclosure
In this work we showed that the query complexity of better-than-2 estimation of metric Steiner tree cost is Θ ( n 3 / 2 ) \Theta(n^{3/2}) Θ ( n 3/2 ) , up to subpolynomial factors. The most interesting follow-up question is be to understand the frontier of query complexity versus estimability for smaller approximation ratios. Specifically, [16 ] showed that estimation within a factor better than 5 / 3 5/3 5/3 is impossible in sublinear time, whereas the algorithms given in [16 , 31 ] and this paper give ( 2 − η ) (2-\eta) ( 2 − η ) -approximation for a very small constant η \eta η . Is it still possible to get sublinear query complexity while achieving a significantly better approximation ratio, say 1.9?
AI Disclosure. The algorithm and the lower bound were discovered using a custom agentic harness developed by the authors, which used GPT 6 Astra, Claude Opus 5, Gemini 3.8 Flash, Kimi K3, and MAI-Thinking-1. The main ideas behind both results can be largely attributed to GPT 6 Astra. AI tools were also used for proofreading and polishing. The authors assume all responsibility for the paper’s content and its correctness. GPT 6.1 Sol (in GitHub Copilot) was used to develop the Lean formalization.