This research was supported in part by the Office of Naval Research under award numbers N00014-24-1-2628 and N00014-24-1-2666.
Mathematics Subject Classification. 65K05, 90C25, 90C30, 93D10, 93D20.
Introduction
We consider unconstrained minimization of a smooth convex function,
f ⋆ : = inf x ∈ R d f ( x ) , f ∈ S 0 , L , (1) f_\star:= \inf_{x\in\mathbb{R}^{d}} f(x), \qquad f \in\mathcal{S}_{0,L}, \tag*{(1)} f ⋆ := x ∈ R d inf f ( x ) , f ∈ S 0 , L , (1)
where S 0 , L \mathcal{S}_{0,L} S 0 , L denotes the class of continuously differentiable convex functions f : R d → R f : \mathbb{R}^{d} \to\mathbb{R} f : R d → R with L L L -Lipschitz gradients for some L > 0 L > 0 L > 0 . We study both this class and its strongly convex subclass S μ , L ⊆ S 0 , L \mathcal{S}_{\mu,L} \subseteq\mathcal{S}_{0,L} S μ , L ⊆ S 0 , L , with most of our analysis devoted to the subclass S μ , L \mathcal{S}_{\mu,L} S μ , L . More precisely, for 0 < μ ≤ L 0 < \mu\le L 0 < μ ≤ L , S μ , L \mathcal{S}_{\mu,L} S μ , L consists of functions satisfying
μ 2 ∥ x − y ∥ 2 ≤ f ( x ) − f ( y ) − ⟨ ∇ f ( y ) , x − y ⟩ ≤ L 2 ∥ x − y ∥ 2 , x , y ∈ R d , (2) \frac{\mu}{2}\lVert x-y\rVert^{2} \le f(x)-f(y)-\langle\nabla f(y),x-y\rangle\le\frac{L}{2}\lVert x-y\rVert^{2}, \qquad x,y\in\mathbb{R}^{d}, \tag*{(2)} 2 μ ∥ x − y ∥ 2 ≤ f ( x ) − f ( y ) − ⟨ ∇ f ( y ) , x − y ⟩ ≤ 2 L ∥ x − y ∥ 2 , x , y ∈ R d , (2)
where we use ∥ ⋅ ∥ \lVert\cdot\rVert ∥ ⋅ ∥ for the Euclidean norm. Setting μ = 0 \mu= 0 μ = 0 in (2) gives the smooth convex class S 0 , L \mathcal{S}_{0,L} S 0 , L introduced above. For this class, we assume that the minimum is attained, i.e., there exists x ⋆ ∈ R d x_\star\in\mathbb{R}^{d} x ⋆ ∈ R d such that f ⋆ = f ( x ⋆ ) f_\star= f(x_\star) f ⋆ = f ( x ⋆ ) . For functions in S μ , L \mathcal{S}_{\mu,L} S μ , L , the minimizer is unique, and we write κ : = L / μ \kappa:= L/\mu κ := L / μ for the condition number of the function f f f .
In the classical first-order oracle model for iteratively solving (1) , where an exact gradient ∇ f ( y k ) \nabla f(y_k) ∇ f ( y k ) at the current query point y k y_k y k is available without any delay at each iteration k k k , the optimal worst-case iteration complexity for the class S μ , L \mathcal{S}_{\mu,L} S μ , L of L L L -smooth, μ \mu μ -strongly convex functions scales as Θ ( κ log ( 1 / ϵ ) ) \Theta(\sqrt{\kappa}\log(1/\epsilon)) Θ ( κ log ( 1/ ϵ )) [39 ] , where ϵ > 0 \epsilon> 0 ϵ > 0 is the target objective-value error. For the class S 0 , L \mathcal{S}_{0,L} S 0 , L of L L L -smooth convex functions, the corresponding complexity is Θ ( L R 2 / ϵ ) \Theta(\sqrt{LR^2/\epsilon}) Θ ( L R 2 / ϵ ) [39 ] , where R R R bounds the initial distance to a minimizer. These lower bounds are established through hard instances in sufficiently high dimensions. The underlying finite-iteration lower bounds hold for every κ > 1 \kappa> 1 κ > 1 in the strongly convex case; the commonly stated Θ ( κ log ( 1 / ϵ ) ) \Theta(\sqrt{\kappa}\log(1/\epsilon)) Θ ( κ log ( 1/ ϵ )) complexity scaling is understood for condition numbers bounded away from 1 and sufficiently small target accuracy ϵ \epsilon ϵ . In terms of achievability, gradient descent (GD) does not attain these bounds in general: its worst-case iteration complexity is O ( κ log ( 1 / ϵ ) ) O(\kappa\log(1/\epsilon)) O ( κ log ( 1/ ϵ )) in the strongly convex case and O ( 1 / ϵ ) O(1/\epsilon) O ( 1/ ϵ ) in the convex case. On the other hand, Nesterov’s accelerated gradient method (NAG) [38 ] achieves these accelerated rates with constant step-size and momentum parameters for functions in S μ , L \mathcal{S}_{\mu,L} S μ , L , and with appropriately varying momentum parameters for functions in S 0 , L \mathcal{S}_{0,L} S 0 , L [39 ] . Polyak’s heavy-ball method (HB) [41 ] also achieves the accelerated κ \sqrt{\kappa} κ dependence on strongly convex quadratics, although the same guarantee does not extend to arbitrary smooth strongly convex objectives under the classical HB parameter choices [34 ] . These optimal rates of NAG and HB provide a natural benchmark for characterizing the performance of first-order methods under settings that deviate from the classical first-order oracle model.
In this paper, we study a fixed-delay variation of the classical first-order oracle model for solving (1) , with a focus on non-idling methods. Specifically, for a fixed integer τ ≥ 1 \tau\ge1 τ ≥ 1 , the oracle returns the exact gradient ∇ f ( y k − τ ) \nabla f(y_{k-\tau}) ∇ f ( y k − τ ) at iteration k k k , where y k − τ y_{k-\tau} y k − τ is the point queried at iteration k − τ k-\tau k − τ [3 , 51 CCS24] . We call a method non-idling if it uses this arriving gradient to perform an update at every iteration. An idling (waiting) method instead postpones its next update until the gradient at its current query point is available. This non-idling, fixed-delay setting is motivated by asynchronous and distributed optimization. For example, a worker in such a system may evaluate a gradient at one iterate while updates from other workers change the shared iterate. When the gradient arrives, it therefore corresponds to an earlier iterate and is stale [7 , 52 , 46 , 13 ] . Although delays in such systems can vary across workers and iterations, we consider exact gradients and a common fixed delay to isolate the effect of staleness. Analyzing this setting can help develop tools for studying more general asynchronous distributed methods, including those with time-varying delays and stochastic gradient errors. We are particularly interested in whether non-idling methods can attain the optimal iteration-complexity scaling in this fixed-delay model.
We use the worst-case lower bounds of Arjevani, Shamir, and Srebro [3 Theorem 4] as the target iteration-complexity scale for non-idling methods. These bounds hold for a deterministic fixed-delay first-order span model , which permits both waiting and non-idling methods. At iteration k k k , the span model requires the new iterate to be a linear combination of gradients from queries made no later than iteration k − τ k-\tau k − τ . The precise span condition is stated later in Theorem 2.1 . For sufficiently large dimension and under the same accuracy and conditioning qualifications as above, their results imply the following iteration-complexity lower bounds for this span model
Ω ( ( τ + 1 ) κ log 1 ϵ ) for f ∈ S μ , L , Ω ( ( τ + 1 ) L R 2 ϵ ) for f ∈ S 0 , L . (3) \begin{aligned}
\Omega\left((\tau+1)\sqrt{\kappa}\log\frac{1}{\epsilon}\right)&\quad\text{for } f\in\mathcal{S}_{\mu,L},\\
\Omega\left((\tau+1)\sqrt{\frac{LR^2}{\epsilon}}\right)&\quad\text{for } f\in\mathcal{S}_{0,L}. \tag*{(3)}
\end{aligned} Ω ( ( τ + 1 ) κ log ϵ 1 ) Ω ( ( τ + 1 ) ϵ L R 2 ) for f ∈ S μ , L , for f ∈ S 0 , L . (3)
The key point is that, compared with the undelayed accelerated rates, these lower bounds introduce an additional factor of τ + 1 \tau+1 τ + 1 . This delay penalty occurs even for quadratic objectives.
In terms of achievability in this fixed-delay, non-idling setting, Arjevani, Shamir, and Srebro [3 ] analyze delayed gradient descent (GD),
x k + 1 = x k − α ∇ f ( x k − τ ) , (4) x_{k+1}=x_k-\alpha\nabla f(x_{k-\tau}), \tag*{(4)} x k + 1 = x k − α ∇ f ( x k − τ ) , (4)
which processes a τ \tau τ -stale gradient at every iteration. On smooth strongly convex quadratics, their guarantees for delayed GD have complexity of order ( τ + 1 ) κ log ( 1 / ϵ ) (\tau+1)\kappa\log(1/\epsilon) ( τ + 1 ) κ log ( 1/ ϵ ) , and on smooth convex quadratics the iteration complexity is of order ( τ + 1 ) L R 2 / ϵ (\tau+1)LR^2/\epsilon ( τ + 1 ) L R 2 / ϵ [3 ] . Hence delayed GD matches the linear dependence on delay in the lower bounds, but, as in the undelayed setting, it does not achieve the accelerated dependence on κ \kappa κ in the strongly convex case or ϵ \epsilon ϵ in the convex case. In particular, they leave open whether an accelerated gradient method can attain these rates while updating with a stale gradient at every iteration. This raises two related questions : can the classical HB and NAG recurrences retain acceleration under delay with suitably chosen parameters while remaining non-idling, and, if not, can a more general non-idling momentum recurrence attain these lower complexity bounds?
To answer these questions, we study a generalized momentum method (GMM) with constant parameters under a fixed delay in the non-idling setting. With x k x_k x k denoting the main iterate and y k y_k y k the gradient query point, the delayed GMM updates take the following form:
y k = x k + ν ( x k − x k − 1 ) , x k + 1 = x k − α ∇ f ( y k − τ ) + β ( x k − x k − 1 ) , k ≥ 0. (5) \begin{aligned}
y_k&=x_k+\nu(x_k-x_{k-1}),\\
x_{k+1}&=x_k-\alpha\nabla f(y_{k-\tau})+\beta(x_k-x_{k-1}),\qquad k\ge0. \tag*{(5)}
\end{aligned} y k x k + 1 = x k + ν ( x k − x k − 1 ) , = x k − α ∇ f ( y k − τ ) + β ( x k − x k − 1 ) , k ≥ 0. (5)
Here α > 0 \alpha>0 α > 0 is the stepsize, β ≥ 0 \beta\ge0 β ≥ 0 is the momentum coefficient, and ν ≥ 0 \nu\ge0 ν ≥ 0 is the gradient-query extrapolation coefficient. The parameters ( α , β , ν ) (\alpha,\beta,\nu) ( α , β , ν ) remain fixed across iterations, although their values may depend on the smoothness and strong-convexity constants and the delay. The initial values x − τ − 1 , … , x 0 x_{-\tau-1},\ldots,x_0 x − τ − 1 , … , x 0 determine the query points needed to start the iteration. We use the terms delayed HB and delayed NAG for the specializations ν = 0 \nu=0 ν = 0 and ν = β \nu=\beta ν = β , respectively, in (5) . These two specializations are the classical HB and NAG updates using the arriving τ \tau τ -delayed gradient at every iteration. Delayed GMM allows ν \nu ν to be chosen independently of β \beta β , rather than requiring ν = 0 \nu=0 ν = 0 or ν = β \nu=\beta ν = β . We use this freedom to adapt the extrapolation to the delay parameter τ \tau τ .
The main results in this paper identify a sharp qualitative difference between the delayed HB and NAG specializations and the general delayed GMM family. Under every fixed positive delay τ ≥ 1 \tau\geq1 τ ≥ 1 , delayed HB with constant parameters fails to retain κ \sqrt{\kappa} κ -acceleration on strongly convex quadratics; the same is true of delayed NAG, even when gradients are delayed by only one iteration (τ = 1 \tau=1 τ = 1 ), for every choice of constant parameters in the standard positive-momentum regime that ensures convergence on strongly convex quadratics. In contrast, we give an explicit choice of parameters, with ν \nu ν proportional to the delay, for which delayed GMM achieves κ \sqrt{\kappa} κ -acceleration on the entire class S μ , L \mathcal{S}_{\mu,L} S μ , L for every fixed delay τ ≥ 1 \tau\geq1 τ ≥ 1 and for all sufficiently large κ \kappa κ . We call this method delay-resistant acceleration with momentum (DREAM). We also extend this approach to the smooth convex class S 0 , L \mathcal{S}_{0,L} S 0 , L by using DREAM as an inner solver in an accelerated proximal-point construction. For sufficiently small target error, the resulting iteration complexity matches the delay and accuracy dependence of the convex lower bound in (3) up to logarithmic factors.
Main results and contributions
We now state the main contributions of this paper more precisely. But first, we clarify their relation to the lower bounds in (3) . These bounds apply to both waiting and non-idling methods, whereas our focus is on non-idling methods. Indeed, a waiting method can match the order of these lower bounds by running NAG, making an update only after the gradient at its current query point has been returned, and remaining idle for the intervening τ \tau τ iterations [3 ] . Thus one accelerated update is performed every τ + 1 \tau+1 τ + 1 iterations. This construction establishes attainability of the lower-bound scaling, but it does not exploit gradients that arrive while the algorithm is waiting. In an asynchronous distributed architecture, forcing every update to wait for a fresh gradient can sacrifice the benefits of pipelining or require waiting for the slowest worker. The question we address is therefore whether non-idling methods can achieve acceleration while using the arriving τ \tau τ -stale gradient at every iteration .
We now turn to our main results, presented in § 3 –§ 7 . In what follows, we also use the class Q μ , L ⊂ S μ , L \mathcal{Q}_{\mu,L}\subset\mathcal{S}_{\mu,L} Q μ , L ⊂ S μ , L of quadratic objectives whose symmetric Hessians have spectra contained in [ μ , L ] [\mu,L] [ μ , L ] . Throughout these results, τ \tau τ is a fixed positive integer.
Limitations of delayed HB and NAG. In § 3 , we first characterize the convergence of delayed GMM on Q μ , L \mathcal{Q}_{\mu,L} Q μ , L . For fixed parameters ( α , β , ν ) (\alpha,\beta,\nu) ( α , β , ν ) , the delayed GMM iterates converge to the minimizer x ⋆ x_\star x ⋆ for every f ∈ Q μ , L f\in\mathcal{Q}_{\mu,L} f ∈ Q μ , L and every initial history if and only if
ρ ( α , β , ν ) : = sup λ ∈ [ μ , L ] max { ∣ z ∣ : z ∈ C , P τ , α λ ( z ) = 0 } < 1 , \rho(\alpha,\beta,\nu):=\sup_{\lambda\in[\mu,L]}\max\{|z|:z\in\mathbb{C},P_{\tau,\alpha\lambda}(z)=0\}<1, ρ ( α , β , ν ) := λ ∈ [ μ , L ] sup max { ∣ z ∣ : z ∈ C , P τ , α λ ( z ) = 0 } < 1 ,
where the characteristic polynomial of the scalar delayed GMM error recurrence along a Hessian eigendirection with eigenvalue λ \lambda λ is given by
P τ , α λ ( z ) : = z τ ( z − 1 ) ( z − β ) + α λ ( ( 1 + ν ) z − ν ) . (6) P_{\tau,\alpha\lambda}(z):=z^\tau(z-1)(z-\beta)+\alpha\lambda\bigl((1+\nu)z-\nu\bigr). \tag*{(6)} P τ , α λ ( z ) := z τ ( z − 1 ) ( z − β ) + α λ ( ( 1 + ν ) z − ν ) . (6)
When ρ ( α , β , ν ) ∈ ( 0 , 1 ) \rho(\alpha,\beta,\nu)\in(0,1) ρ ( α , β , ν ) ∈ ( 0 , 1 ) , the iterates satisfy
∥ x k − x ⋆ ∥ ≤ c k ρ ( α , β , ν ) k , k ≥ 0 , \lVert x_k-x_\star\rVert\leq c_k\rho(\alpha,\beta,\nu)^k,\qquad k\geq0, ∥ x k − x ⋆ ∥ ≤ c k ρ ( α , β , ν ) k , k ≥ 0 ,
for some non-negative sequence { c k } k ≥ 0 \{c_k\}_{k\geq0} { c k } k ≥ 0 that grows at most polynomially in k k k . We call such a parameter choice Schur stable , as it implies that the polynomial P τ , α λ P_{\tau,\alpha}^{\lambda} P τ , α λ is Schur stable , i.e., all the roots lie strictly inside the unit circle for all λ ∈ [ μ , L ] \lambda\in[\mu,L] λ ∈ [ μ , L ] . Thus, ρ ( α , β , ν ) \rho(\alpha,\beta,\nu) ρ ( α , β , ν ) characterizes the worst-case asymptotic convergence factor over Q μ , L Q_{\mu,L} Q μ , L . In particular, for every ρ ^ \widehat{\rho} ρ satisfying ρ ( α , β , ν ) < ρ ^ < 1 \rho(\alpha,\beta,\nu)<\widehat{\rho}<1 ρ ( α , β , ν ) < ρ < 1 , the iterates admit a geometric bound of the form ∥ x k − x ⋆ ∥ ≤ C ρ ^ ρ ^ k \lVert x_k-x_\star\rVert\leq C_{\widehat{\rho}}\widehat{\rho}^{k} ∥ x k − x ⋆ ∥ ≤ C ρ ρ k for some constant C ρ ^ > 0 C_{\widehat{\rho}}>0 C ρ > 0 independent of k k k .
For the HB and NAG specializations of delayed GMM, let ρ H B ∗ ( τ , κ ) \rho^\ast_{\mathrm{HB}}(\tau,\kappa) ρ HB ∗ ( τ , κ ) and ρ N A G ∗ ( τ , κ ) \rho^\ast_{\mathrm{NAG}}(\tau,\kappa) ρ NAG ∗ ( τ , κ ) denote the infima of ρ ( α , β , 0 ) \rho(\alpha,\beta,0) ρ ( α , β , 0 ) and ρ ( α , β , β ) \rho(\alpha,\beta,\beta) ρ ( α , β , β ) , respectively, over Schur-stable choices with α > 0 \alpha>0 α > 0 and 0 < β < 1 0<\beta<1 0 < β < 1 .3 Theorems 3.17 and 3.12 establish that, for every fixed τ ≥ 1 \tau\geq1 τ ≥ 1 , as κ → ∞ \kappa\to\infty κ → ∞ ,
1 − ρ H B ∗ ( τ , κ ) = O τ ( κ − 1 ) , τ ≥ 1 , 1 − ρ N A G ∗ ( τ , κ ) ≤ ( 2 + o ( 1 ) ) κ − 2 / 3 , τ = 1 , 1 − ρ N A G ∗ ( τ , κ ) = O τ ( κ − 1 ) , τ ≥ 2. (7) \begin{aligned}
1-\rho^\ast_{\mathrm{HB}}(\tau,\kappa)&=\mathcal{O}_{\tau}(\kappa^{-1}), && \tau\geq1,\\
1-\rho^\ast_{\mathrm{NAG}}(\tau,\kappa)&\leq(2+o(1))\kappa^{-2/3}, && \tau=1,\\
1-\rho^\ast_{\mathrm{NAG}}(\tau,\kappa)&=\mathcal{O}_{\tau}(\kappa^{-1}), && \tau\geq2. \tag*{(7)}
\end{aligned} 1 − ρ HB ∗ ( τ , κ ) 1 − ρ NAG ∗ ( τ , κ ) 1 − ρ NAG ∗ ( τ , κ ) = O τ ( κ − 1 ) , ≤ ( 2 + o ( 1 )) κ − 2/3 , = O τ ( κ − 1 ) , τ ≥ 1 , τ = 1 , τ ≥ 2. (7)
Here and throughout, o ( 1 ) o(1) o ( 1 ) is with respect to κ → ∞ \kappa\to\infty κ → ∞ , while O τ ( ⋅ ) \mathcal{O}_{\tau}(\cdot) O τ ( ⋅ ) denotes big-O notation with constants that may depend on τ \tau τ but are independent of κ \kappa κ . Thus, even with parameters adapted to the delay, neither delayed HB nor delayed NAG retains the accelerated κ \sqrt{\kappa} κ dependence under any fixed positive delay. For the single-delay case τ = 1 \tau=1 τ = 1 , Lemma 3.13 also establishes that, for every κ ≥ 64 \kappa\geq64 κ ≥ 64 ,
1 − ρ N A G ∗ ( 1 , κ ) ≥ 7 22 κ − 2 / 3 . 1-\rho^\ast_{\mathrm{NAG}}(1,\kappa)\geq\frac{7}{22}\kappa^{-2/3}. 1 − ρ NAG ∗ ( 1 , κ ) ≥ 22 7 κ − 2/3 .
Together with the upper bound above, this shows that the optimal spectral gap for delayed NAG with τ = 1 \tau=1 τ = 1 has order Θ ( κ − 2 / 3 ) \Theta(\kappa^{-2/3}) Θ ( κ − 2/3 ) . Thus, for τ = 1 \tau=1 τ = 1 , delayed NAG improves on the κ − 1 \kappa^{-1} κ − 1 gap of delayed HB, but neither method attains the accelerated κ − 1 / 2 \kappa^{-1/2} κ − 1/2 gap. In summary, our first contribution is a characterization of delayed GMM convergence on quadratics through its characteristic polynomial. We then use this characterization to derive stability and spectral-gap bounds showing that neither delayed HB nor delayed NAG retains acceleration under any fixed positive delay.
Acceleration and optimal parameters for delayed GMM on quadratics. Our next contribution, established in § 3 , shows that allowing the extrapolation parameter ν \nu ν to be chosen independently of β \beta β enables delayed GMM to achieve acceleration on Q μ , L Q_{\mu,L} Q μ , L . Specifically, for the one-step delay τ = 1 \tau=1 τ = 1 , Proposition 3.19 gives an explicit delayed GMM tuning with ρ ( α , β , ν ) = 1 − 1 / ( 2 κ ) \rho(\alpha,\beta,\nu)=1-1/(2\sqrt{\kappa}) ρ ( α , β , ν ) = 1 − 1/ ( 2 κ ) for every κ ≥ 1 \kappa\geq1 κ ≥ 1 . Moreover, Proposition 3.20 shows that the coefficient 1 / 2 1/2 1/2 in the spectral gap 1 − ρ ( α , β , ν ) = 1 / ( 2 κ ) 1-\rho(\alpha,\beta,\nu)=1/(2\sqrt{\kappa}) 1 − ρ ( α , β , ν ) = 1/ ( 2 κ ) is asymptotically sharp as κ → ∞ \kappa\to\infty κ → ∞ among Schur-stable, constant-parameter delayed GMM tunings with β , ν ≥ 0 \beta,\nu\geq0 β , ν ≥ 0 .
To obtain accelerated tunings for every fixed delay τ ≥ 1 \tau\geq1 τ ≥ 1 ,4 we next consider the parameterization
α = δ L L , β = 1 − 2 δ L κ , \alpha=\frac{\delta_L}{L},\qquad\beta=1-\frac{2\sqrt{\delta_L}}{\sqrt{\kappa}}, α = L δ L , β = 1 − κ 2 δ L ,
with δ L > 0 \delta_L>0 δ L > 0 and ν \nu ν to be chosen. Theorem 3.22 then shows that, on Q μ , L Q_{\mu,L} Q μ , L , this parameterization satisfies ρ ( α , β , ν ) ≤ 1 − γ / κ \rho(\alpha,\beta,\nu)\leq1-\gamma/\sqrt{\kappa} ρ ( α , β , ν ) ≤ 1 − γ / κ for every 0 < γ < δ L 0<\gamma<\sqrt{\delta_L} 0 < γ < δ L and all sufficiently large κ \kappa κ , provided that the limiting characteristic polynomial as κ → ∞ \kappa\to\infty κ → ∞
P ∞ ( z ; δ , ν ) : = z τ ( z − 1 ) 2 + δ [ ( 1 + ν ) z − ν ] P_{\infty}(z;\delta,\nu):=z^{\tau}(z-1)^2+\delta[(1+\nu)z-\nu] P ∞ ( z ; δ , ν ) := z τ ( z − 1 ) 2 + δ [( 1 + ν ) z − ν ]
is Schur stable for every 0 < δ ≤ δ L 0 < \delta\le\delta_L 0 < δ ≤ δ L . The theorem also shows that ν > τ \nu> \tau ν > τ is necessary for this stability condition.
Since Theorem 3.22 allows every 0 < γ < δ L 0 < \gamma< \sqrt{\delta_L} 0 < γ < δ L , increasing the admissible value of δ L \delta_L δ L improves the guaranteed convergence rate. We therefore choose δ L \delta_L δ L and ν \nu ν to make δ L \delta_L δ L as large as the stated stability condition permits. Let δ τ c r i t \delta_\tau^{\mathrm{crit}} δ τ crit denote the supremum of admissible values of δ L \delta_L δ L over ν > τ \nu> \tau ν > τ . Corollary B.4 characterizes this threshold exactly and identifies the corresponding optimal extrapolation parameter ν τ c r i t \nu_\tau^{\mathrm{crit}} ν τ crit . In particular, every 0 < γ < δ τ c r i t 0 < \gamma< \sqrt{\delta_\tau^{\mathrm{crit}}} 0 < γ < δ τ crit is achievable for sufficiently large κ \kappa κ by choosing ν = ν τ c r i t \nu= \nu_\tau^{\mathrm{crit}} ν = ν τ crit and γ 2 < δ L < δ τ c r i t \gamma^2 < \delta_L < \delta_\tau^{\mathrm{crit}} γ 2 < δ L < δ τ crit . Thus, δ τ c r i t \sqrt{\delta_\tau^{\mathrm{crit}}} δ τ crit is the supremum of the rate coefficients γ \gamma γ guaranteed by Theorem 3.22 when δ L \delta_L δ L and ν \nu ν are varied. These results provide explicit accelerated tunings of delayed GMM on Q μ , L Q_{\mu,L} Q μ , L , with an asymptotically sharp spectral-gap coefficient for τ = 1 \tau= 1 τ = 1 and, for each fixed τ ≥ 1 \tau\ge1 τ ≥ 1 , an exact characterization of the supremum of rate coefficients guaranteed by Theorem 3.22 within the displayed parameterization. We also consider a simple explicit tuning. Writing M : = 2 τ + 1 M := 2\tau+ 1 M := 2 τ + 1 , we take δ L = 1 / M 2 \delta_L = 1/M^2 δ L = 1/ M 2 and ν = M − 1 2 \nu= M - \frac{1}{2} ν = M − 2 1 . Theorem 3.25 then guarantees an accelerated rate constant γ = 1 / ( 2 M ) \gamma= 1/(2M) γ = 1/ ( 2 M ) on Q μ , L Q_{\mu,L} Q μ , L . Moreover, Proposition B.5 in the appendix shows that, as τ → ∞ \tau\to\infty τ → ∞ ,
δ τ c r i t 1 / ( 2 M ) ⟶ 4 a ⋆ cos a ⋆ < 3 , \frac{\sqrt{\delta_\tau^{\mathrm{crit}}}}{1/(2M)} \longrightarrow4a_\star\sqrt{\cos a_\star} < 3, 1/ ( 2 M ) δ τ crit ⟶ 4 a ⋆ cos a ⋆ < 3 ,
where a ⋆ ∈ ( 0 , π / 2 ) a_\star\in(0,\pi/2) a ⋆ ∈ ( 0 , π /2 ) is the unique solution of a ⋆ tan a ⋆ = 2 a_\star\tan a_\star= 2 a ⋆ tan a ⋆ = 2 . Thus, for sufficiently large delays, this simple explicit tuning retains more than one third of the best rate coefficient guaranteed by Theorem 3.22. We adopt this tuning for DREAM because the same parameters also guarantee accelerated convergence on S μ , L S_{\mu,L} S μ , L , as discussed next.
Accelerated convergence of DREAM on smooth strongly convex objectives. Our next contribution, presented in § 4 –§ 6 , is an accelerated convergence guarantee for DREAM on the full class S μ , L S_{\mu,L} S μ , L . Recall that DREAM uses the explicit parameter choice:
α = 1 M 2 L , β = 1 − 2 M κ , ν = M − 1 2 = 2 τ + 1 2 . (8) \alpha= \frac{1}{M^2L}, \qquad\beta= 1 - \frac{2}{M\sqrt{\kappa}}, \qquad\nu= M - \frac{1}{2} = 2\tau+ \frac{1}{2}. \tag*{(8)} α = M 2 L 1 , β = 1 − M κ 2 , ν = M − 2 1 = 2 τ + 2 1 . (8)
Proving convergence over the broader function class S μ , L S_{\mu,L} S μ , L for DREAM requires a different proof approach from that used for quadratics. Theorem 4.1 , stated in § 4 and proved in § 5 , shows that, for every fixed integer τ ≥ 1 \tau\ge1 τ ≥ 1 and every c ∈ ( 0 , 2 / 3 ) c \in(0,2/3) c ∈ ( 0 , 2/3 ) , there exists κ 0 ( τ , c ) \kappa_0(\tau,c) κ 0 ( τ , c ) such that, for κ ≥ κ 0 ( τ , c ) \kappa\ge\kappa_0(\tau,c) κ ≥ κ 0 ( τ , c ) , the DREAM iterates for every f ∈ S μ , L f \in S_{\mu,L} f ∈ S μ , L and every choice of initial iterates x − τ − 1 , … , x 0 ∈ R d x_{-\tau-1},\ldots,x_0 \in\mathbb{R}^d x − τ − 1 , … , x 0 ∈ R d satisfy
∥ x k − x ⋆ ∥ ≤ C ρ k , ρ = 1 − c M κ , k ≥ 0 , (9) \lVert x_k - x_\star\rVert\le C\rho^k, \qquad\rho= 1 - \frac{c}{M\sqrt{\kappa}}, \qquad k \ge0, \tag*{(9)} ∥ x k − x ⋆ ∥ ≤ C ρ k , ρ = 1 − M κ c , k ≥ 0 , (9)
where C C C may depend on the initial history and problem parameters but is independent of k k k . Note that this bound does not imply that the distance to the minimizer decreases at every iteration. Next, for the constant-history initialization x − τ − 1 = ⋯ = x 0 = x i n i t x_{-\tau-1} = \cdots= x_0 = x_{\mathrm{init}} x − τ − 1 = ⋯ = x 0 = x init , Theorem 6.1 in § 6 gives polynomial bounds on the threshold κ 0 ( τ , c ) \kappa_0(\tau,c) κ 0 ( τ , c ) and the prefactor C C C in (9) :
κ 0 ( τ , c ) = O c ( M 8 ) , C = O c ( M 5 κ 2 ) ∥ x i n i t − x ⋆ ∥ , (10) \kappa_0(\tau,c) = \mathcal{O}_c(M^8), \qquad C = \mathcal{O}_c(M^5\kappa^2)\lVert x_{\mathrm{init}} - x_\star\rVert, \tag*{(10)} κ 0 ( τ , c ) = O c ( M 8 ) , C = O c ( M 5 κ 2 ) ∥ x init − x ⋆ ∥ , (10)
where O c ( ⋅ ) \mathcal{O}_c(\cdot) O c ( ⋅ ) hides constants depending only on c c c . Consequently, for R : = ∥ x i n i t − x ⋆ ∥ > 0 R := \lVert x_{\mathrm{init}} - x_\star\rVert> 0 R := ∥ x init − x ⋆ ∥ > 0 and 0 < ϵ < μ R 2 / 4 0 < \epsilon< \mu R^2/4 0 < ϵ < μ R 2 /4 , smoothness implies that the accuracy f ( x k ) − f ⋆ ≤ ϵ f(x_k) - f_\star\le\epsilon f ( x k ) − f ⋆ ≤ ϵ is attained within
O c ( M κ [ log μ R 2 ϵ + log M + log κ ] ) (11) \mathcal{O}_c\left(M\sqrt{\kappa}\left[\log\frac{\mu R^2}{\epsilon} + \log M + \log\kappa\right]\right) \tag*{(11)} O c ( M κ [ log ϵ μ R 2 + log M + log κ ] ) (11)
updates. Thus, DREAM achieves accelerated convergence on S μ , L \mathcal{S}_{\mu,L} S μ , L without idle iterations. In particular, its iteration complexity matches the strongly convex lower bound in (3) up to the displayed logarithmic factors.
Extending DREAM to smooth convex optimization. Our next contribution, presented in § 7 , is an accelerated method for smooth convex optimization that uses DREAM to approximately solve a sequence of strongly convex regularized problems, with the outer iterations following the Catalyst accelerated proximal-point framework [33 ] . Specifically, let f ∈ S 0 , L f \in\mathcal{S}_{0,L} f ∈ S 0 , L have a nonempty minimizer set. Fix x ⋆ ∈ arg min f x_{\star} \in\arg\min f x ⋆ ∈ arg min f and write R ⋆ : = ∥ x 0 − x ⋆ ∥ R_{\star} := \lVert x_0 - x_{\star}\rVert R ⋆ := ∥ x 0 − x ⋆ ∥ . At each outer iteration s ≥ 1 s \ge1 s ≥ 1 , we run DREAM for a prescribed number of steps to obtain an approximate minimizer x s x_s x s of
H s ( x ) = f ( x ) + λ 2 ∥ x − v s − 1 ∥ 2 H_s(x) = f(x) + \frac{\lambda}{2}\lVert x-v_{s-1}\rVert^2 H s ( x ) = f ( x ) + 2 λ ∥ x − v s − 1 ∥ 2
for a regularization parameter λ > 0 \lambda> 0 λ > 0 . Here v s − 1 v_{s-1} v s − 1 is obtained by extrapolating the preceding outer iterates, as specified later in § 7 . Before each run of DREAM, its starting point and all past iterates required by the delayed recurrence are set to v s − 1 v_{s-1} v s − 1 . Since H s H_s H s is λ \lambda λ -strongly convex and ( L + λ ) (L+\lambda) ( L + λ ) -smooth, its condition number is χ λ = 1 + L / λ \chi_{\lambda} = 1 + L/\lambda χ λ = 1 + L / λ . Set κ ˉ 0 : = max { 2 , κ 0 ( τ , c ) } \bar{\kappa}_0 := \max\{2,\kappa_0(\tau,c)\} κ ˉ 0 := max { 2 , κ 0 ( τ , c )} and choose λ = L / ( κ ˉ 0 − 1 ) \lambda= L/(\bar{\kappa}_0-1) λ = L / ( κ ˉ 0 − 1 ) . Then χ λ = κ ˉ 0 ≥ κ 0 ( τ , c ) \chi_{\lambda} = \bar{\kappa}_0 \ge\kappa_0(\tau,c) χ λ = κ ˉ 0 ≥ κ 0 ( τ , c ) , so DREAM’s strongly convex convergence guarantee in Theorem 6.1 applies to minimizing each regularized objective H s H_s H s . Consequently, Theorem 7.2 and Corollary 7.3 show that, for every ϵ > 0 \epsilon> 0 ϵ > 0 , the method produces an outer iterate x s x_s x s satisfying f ( x s ) − f ⋆ ≤ ϵ f(x_s)-f_{\star} \le\epsilon f ( x s ) − f ⋆ ≤ ϵ using a total of
O ~ c ( M [ κ ˉ 0 + L R ⋆ 2 ϵ ] ) (12) \widetilde{\mathcal{O}}_c\left(M\left[\sqrt{\bar{\kappa}_0}+\sqrt{\frac{LR_{\star}^2}{\epsilon}}\right]\right) \tag*{(12)} O c ( M [ κ ˉ 0 + ϵ L R ⋆ 2 ] ) (12)
inner delayed-gradient updates. Here O ~ c \widetilde{\mathcal{O}}_c O c denotes big-O notation with logarithmic factors omitted and a constant that may depend on c c c . The M κ ˉ 0 M\sqrt{\bar{\kappa}_0} M κ ˉ 0 term in this expression accounts for the cost of the first run of DREAM, and the prescribed inner iteration counts do not require prior knowledge of R ⋆ R_{\star} R ⋆ . When ϵ ≤ λ R ⋆ 2 \epsilon\le\lambda R_{\star}^2 ϵ ≤ λ R ⋆ 2 , the first term is bounded by a constant multiple of the second term. Recalling that M = 2 τ + 1 M = 2\tau+ 1 M = 2 τ + 1 , the bound therefore simplifies to
O ~ c ( ( τ + 1 ) L R ⋆ 2 ϵ ) \widetilde{\mathcal{O}}_c\left((\tau+1)\sqrt{\frac{LR_{\star}^2}{\epsilon}}\right) O c ( ( τ + 1 ) ϵ L R ⋆ 2 )
Thus, when ϵ ≤ λ R ⋆ 2 \epsilon\le\lambda R_{\star}^2 ϵ ≤ λ R ⋆ 2 , the accelerated proximal-point method with DREAM as its inner solver matches the delay and accuracy dependence of the convex lower bound in (3) , up to logarithmic factors.
An analytic proof technique valid for every fixed delay. Our convergence guarantees for general f ∈ S μ , L f \in\mathcal{S}_{\mu,L} f ∈ S μ , L do not follow from quadratic stability analysis alone: the gradient mapping is nonlinear, and stability on quadratics does not necessarily imply convergence at the same rate over the full function class. To overcome this difficulty, we represent (5) as a feedback interconnection between a linear delayed system and the nonlinear gradient mapping. Our analysis relies on finite-horizon, rate-weighted Zames–Falb integral quadratic constraints (IQCs), which exploit relations among gradients evaluated at different iterates. Unlike pointwise inequalities derived from smoothness and strong convexity, these constraints capture additional relations between gradients along the trajectory. We then separate the gradient mapping into a linear part, with slope ( μ + L ) / 2 (\mu+L)/2 ( μ + L ) /2 , and a nonlinear remainder. This transformation allows us to establish stability of the rate-weighted linear dynamics and derive a frequency-domain condition that, through the Kalman–Yakubovich–Popov lemma, yields a quadratic Lyapunov certificate for the desired convergence rate. For our delay-dependent parameterization, we show that, as κ → ∞ \kappa\to\infty κ → ∞ , verification of this condition reduces to proving positivity of an explicit trigonometric polynomial. To extend the result to sufficiently large finite κ \kappa κ , we establish quantitative estimates separately for frequencies near zero, where the limiting approximation requires special care, and for frequencies away from zero. This yields explicit convergence guarantees and the polynomial bounds in (10) , without requiring semidefinite programming or numerical search.
The value of this argument is constructive: it identifies the parameters in (8) analytically , and certifies an accelerated rate for the full nonlinear function class at every fixed delay, without a delay-by-delay semidefinite optimization, numerical frequency search, or empirical parameter fitting. The main contribution of this work is an explicitly parameterized, constant-parameter momentum method that achieves accelerated convergence using stale gradients without idling, together with a fully analytic convergence certificate valid for every fixed positive delay and sufficiently large condition number. Our analysis also provides explicit polynomial bounds on the condition-number threshold and the transient prefactor, under constant-history initialization.
Related work
Delayed gradients and asynchronous first-order methods. Choi, Choi, and Seok [CCS24] establish non-ergodic linear-convergence estimates for delayed gradient descent under smooth strong convexity and extend their analysis to objectives satisfying the Polyak–Łojasiewicz condition, including stochastic, time-varying delays. In the stochastic setting, Stich and Karimireddy [51 ] establish nonasymptotic bounds for delayed stochastic gradient descent on smooth quasi-convex and nonconvex objectives; delays affect the deterministic optimization term but not the leading stochastic term, and accelerated methods are not considered. Fang, Huang, and Lin [17 ] introduce momentum compensation for asynchronous first-order methods with bounded, potentially time-varying delays. By modifying Nesterov’s extrapolation point to account for momentum accumulated during the delay, their accelerated method attains accelerated dependence on the condition number [17 Corollary 1] , but its iteration complexity depends quadratically on the delay bound. Ajang et al. [2 ] propose a modified Nesterov method for asynchronous pipeline-parallel optimization and establish sublinear convergence under fixed gradient delays. Pond, Zhao, and Hale [43 ] study distributed generalized momentum methods allowing arbitrarily long computation and communication delays, obtaining convergence guarantees under additional structural assumptions. These settings and guarantees differ from the fixed-delay, constant-parameter recurrence studied here. Devolder, Glineur, and Nesterov [15 , 14 ] develop inexact first-order oracle frameworks encompassing gradients evaluated at shifted points. Although delayed gradients can be modeled in this way, the resulting errors depend on the iterate history, and standard inexact-oracle bounds do not directly establish accelerated convergence. Related asynchronous acceleration results have also been established under different oracle and computational models. Hannah, Feng, and Yin [24 ] develop an accelerated asynchronous randomized block coordinate descent method with optimal complexity under suitable restrictions on the delay. Hendrikx, Bach, and Massoulie [23 ] establish accelerated convergence for decentralized optimization with local updates, with complexity depending on communication delays and network parameters. These results demonstrate that acceleration under asynchronous execution is possible, but their decentralized oracle models differ from the deterministic, fixed-delay full-gradient model studied here.
Foundations of IQCs and exponential-rate certificates. Zames and Falb [57 ] introduce multipliers for monotone and slope-restricted nonlinearities, while Megretski and Rantzer [36 ] develop the integral-quadratic-constraint (IQC) framework for robust stability analysis. Related foundations include Willems’ dissipativity theory [56 ] and Rantzer’s frequency-domain formulation of the Kalman–Yakubovich–Popov lemma [44 ] ; Pfifer and Seiler [42 ] develop IQCs for delayed systems. Lessard, Recht, and Packard [34 ] introduce a systematic IQC approach to analyze first-order optimization algorithms in a unified manner, including gradient descent, heavy-ball, and Nesterov’s method, while Hu and Lessard [25 ] provide a complementary dissipativity analysis of acceleration. Boczar, Lessard, and Recht [6 ] develop rate-weighted IQCs, including Zames–Falb multipliers for slope-restricted nonlinearities, to certify prescribed exponential convergence rates through frequency-domain inequalities that can be checked numerically using linear matrix inequalities.
Zames–Falb analysis and undelayed momentum methods. Generalized momentum parameterizations separating extrapolation from inertia have also been studied without delays [54 Gür, 9 ] . Cyrus, Hu, Van Scoy, and Lessard [10 ] employ a rate-weighted Zames–Falb IQC to analyze robust momentum without delays and derive an analytical tuning that trades convergence speed against robustness to gradient errors. Freeman [18 ] extends exponential-rate Zames–Falb analysis to noncausal multipliers for feedback interconnections of linear time-invariant systems and static nonlinearities, yielding less conservative convergence-rate certificates. He also establishes the validity of these multipliers for vector-valued gradient mappings by exploiting cyclic monotonicity. Zhang, Seiler, and Carrasco [58 ] show that nonlinear feedback systems can converge more slowly than their linear counterparts and develop a convex search over noncausal Zames–Falb multipliers supported on finitely many lags to certify exponential convergence rates. Van Scoy and Lessard [55 ] develop an undelayed generalized momentum method for twice continuously differentiable smooth strongly convex objectives. Using an off-by-one Zames–Falb multiplier, they establish global convergence and characterize the asymptotic rate through linearization at the minimizer, obtaining a bound of the form ∥ x k − x ⋆ ∥ ≤ c ( k ) ρ k \lVert x_k-x_\star\rVert\le c(k)\rho^k ∥ x k − x ⋆ ∥ ≤ c ( k ) ρ k with a subexponentially growing c ( k ) c(k) c ( k ) . Their leading-order iteration-complexity comparison neglects the contribution of log c ( k ) \log c(k) log c ( k ) as k → ∞ k \to\infty k → ∞ .
IQC-based numerical certificates with delayed gradients. Michalowsky, Scherer, and Ebenbauer [37 ] develop an IQC-based framework for analyzing and synthesizing gradient algorithms with certified convergence rates and robustness guarantees, using rate-weighted Zames–Falb multipliers and exploiting structural properties of the objective. Scherer and Ebenbauer [48 ] develop a general convex synthesis framework based on rate-weighted Zames–Falb IQCs and the Youla parameterization, in which algorithm parameters and convergence certificates are obtained through semidefinite programming. Their framework accommodates arbitrary fixed gradient delays, as illustrated by numerical synthesis and convergence-rate certification for one- and two-step delays; Scherer, Ebenbauer, and Holicki [49 ] provide a tutorial treatment, including delayed gradient information. These works establish rigorous and general IQC-based frameworks for analyzing and synthesizing optimization algorithms. However, their application to delayed gradients relies on numerical semidefinite optimization to obtain algorithm parameters and certify convergence rates for prescribed delays, with numerical illustrations for one- and two-step delays. Although the resulting certificates imply global linear convergence with finite, certificate-dependent prefactors, these numerical results do not provide closed-form delay-dependent parameter choices, explicit bounds on the dependence of convergence rates and transient prefactors on the delay and condition number, or corresponding quantitative iteration-complexity guarantees. In contrast, we derive closed-form, delay-dependent parameters for a generalized momentum method and analytically verify a delay- and rate-dependent Zames–Falb certificate for every fixed positive delay and sufficiently large condition number, without numerical optimization or semidefinite feasibility searches. Our convergence guarantees hold over the full class of smooth strongly convex objectives and include a quantitative condition-number threshold and explicit polynomial bounds on the transient prefactor under constant-history initialization. These results yield near-optimal iteration complexity in the delay and condition number, up to logarithmic factors. We further extend our guarantees to general smooth convex objectives through a regularized proximal-point construction, obtaining near-optimal complexity in the high-accuracy regime while explicitly accounting for initialization and transient costs.
Organization
Section § 2 sets out the delayed first-order model and notation. Section § 3 studies the characteristic polynomials and proves the limitations of classical momentum for strongly convex quadratics. Section § 4 considers the general strongly convex setting beyond quadratics. In particular, Sections § 4.1 and § 5 develop the control-theoretic certificate and prove convergence on S μ , L S_{\mu,L} S μ , L ; Section § 6 quantifies its threshold and transient factor. Section § 7 extends the construction to smooth convex objectives, followed by experiments and concluding remarks. Appendix § 12 examines parameter design in the quadratic setting.
Notation
For nonempty S ⊂ C S \subset\mathbb{C} S ⊂ C , set dist ( w , S ) = inf ζ ∈ S ∣ w − ζ ∣ \operatorname{dist}(w,S)=\inf_{\zeta\in S}|w-\zeta| dist ( w , S ) = inf ζ ∈ S ∣ w − ζ ∣ ; 1 { E } \mathbf{1}_{\{E\}} 1 { E } denotes the indicator of a statement E E E . For a square matrix A A A , its spectrum σ ( A ) \sigma(A) σ ( A ) is the set of its eigenvalues, and its spectral radius is ρ ( A ) = max λ ∈ σ ( A ) ∣ λ ∣ \rho(A)=\max_{\lambda\in\sigma(A)}|\lambda| ρ ( A ) = max λ ∈ σ ( A ) ∣ λ ∣ . For a nonconstant polynomial P P P , we similarly define ρ ( P ) = max { ∣ λ ∣ : P ( λ ) = 0 } \rho(P)=\max\{|\lambda|:P(\lambda)=0\} ρ ( P ) = max { ∣ λ ∣ : P ( λ ) = 0 } ; equivalently, ρ ( P ) = ρ ( C P ) \rho(P)=\rho(C_P) ρ ( P ) = ρ ( C P ) , where C P C_P C P is its companion matrix. A square matrix or polynomial is Schur stable if its spectral radius is strictly less than one. A polynomial is Hurwitz stable if all its roots have strictly negative real parts.
Consider the following discrete-time dynamics with state s k ∈ C n s_k\in\mathbb{C}^{n} s k ∈ C n , input u k ∈ C m u_k\in\mathbb{C}^{m} u k ∈ C m , and output y k ∈ C p y_k\in\mathbb{C}^{p} y k ∈ C p , described by
s k + 1 = A s k + B u k , y k = C s k + D u k . s_{k+1}=As_k+Bu_k,\qquad y_k=Cs_k+Du_k. s k + 1 = A s k + B u k , y k = C s k + D u k .
where A A A , B B B , C C C , D D D are fixed matrices. For a two-sided sequence ( x k ) k ∈ Z (x_k)_{k\in\mathbb{Z}} ( x k ) k ∈ Z , its bilateral z-transform is X ( z ) = ∑ k ∈ Z x k z − k X(z)=\sum_{k\in\mathbb{Z}}x_kz^{-k} X ( z ) = ∑ k ∈ Z x k z − k , wherever the series converges. The zero-state response of the system is obtained by setting s 0 = 0 s_0=0 s 0 = 0 and applying an input with u k = 0 u_k=0 u k = 0 for k < 0 k<0 k < 0 . Thus, for k ≥ 1 k\ge1 k ≥ 1 , the state is determined entirely by the input: s k = ∑ j = 0 k − 1 A k − 1 − j B u j s_k=\sum_{j=0}^{k-1}A^{k-1-j}Bu_j s k = ∑ j = 0 k − 1 A k − 1 − j B u j . Extending the resulting state and output sequences by zero to negative indices, their z z z -transforms satisfy
Y ( z ) = G ( z ) U ( z ) , G ( z ) = C ( z I − A ) − 1 B + D . Y(z)=G(z)U(z),\qquad G(z)=C(zI-A)^{-1}B+D. Y ( z ) = G ( z ) U ( z ) , G ( z ) = C ( z I − A ) − 1 B + D .
where the relevant transforms converge and z I − A zI-A z I − A is invertible. The rational matrix G G G is called the transfer matrix , or the transfer function in the scalar case when m = p = 1 m=p=1 m = p = 1 . On the unit circle, the discrete-time Fourier transform is X ( e i θ ) = ∑ k ∈ Z x k e − i k θ X(e^{\mathrm{i}\theta})=\sum_{k\in\mathbb{Z}}x_ke^{-\mathrm{i}k\theta} X ( e i θ ) = ∑ k ∈ Z x k e − i k θ for θ ∈ [ 0 , 2 π ) \theta\in[0,2\pi) θ ∈ [ 0 , 2 π ) , interpreted pointwise when the series converges and in the L 2 L^2 L 2 sense for square-summable sequences.
For a rational transfer matrix F ( z ) F(z) F ( z ) , a pole is a finite complex number at which at least one entry of F F F has a singularity after cancellation of common polynomial factors. The transfer matrix is stable if all its poles lie in the open unit disk D : = { z ∈ C : ∣ z ∣ < 1 } \mathbb{D}:=\{z\in\mathbb{C}:|z|<1\} D := { z ∈ C : ∣ z ∣ < 1 } . It is proper if the entrywise limit F ( ∞ ) : = lim z → ∞ F ( z ) F(\infty):=\lim_{z\to\infty}F(z) F ( ∞ ) := lim z → ∞ F ( z ) exists and is finite. Equivalently, each nonzero entry of F F F , expressed as a ratio of coprime polynomials in z z z , has numerator degree no greater than denominator degree.
For a rational transfer matrix F F F , its para-Hermitian adjoint is F ∼ ( z ) : = F ( 1 / z ) ∗ F^{\sim}(z) := F(1/z)^* F ∼ ( z ) := F ( 1/ z ) ∗ , where ∗ * ∗ denotes conjugate transpose; thus F ∼ ( e i θ ) = F ( e i θ ) ∗ F^{\sim}(e^{\mathrm{i}\theta}) = F(e^{\mathrm{i}\theta})^* F ∼ ( e i θ ) = F ( e i θ ) ∗ . We use standard terminology from discrete-time linear systems theory; for further background on state-space representations, z z z -transforms, Fourier transforms, transfer matrices, and stability, see Hinrichsen and Pritchard [27 Secs. 2.2–2.3 and Appendix A.3] .
Preliminaries
The problem class and the algorithm
Smooth (strongly) convex functions. Let 0 < μ ≤ L 0 < \mu\le L 0 < μ ≤ L and set κ = L / μ \kappa= L/\mu κ = L / μ . Let S μ , L \mathcal{S}_{\mu,L} S μ , L denote the class of μ \mu μ -strongly convex, continuously differentiable functions f : R d → R f : \mathbb{R}^d \to\mathbb{R} f : R d → R that are L L L -smooth, i.e., ∇ f \nabla f ∇ f is L L L -Lipschitz. These are precisely the functions satisfying, for all x , y ∈ R d x,y \in\mathbb{R}^d x , y ∈ R d ,
μ L μ + L ∥ x − y ∥ 2 + 1 μ + L ∥ ∇ f ( x ) − ∇ f ( y ) ∥ 2 ≤ ⟨ ∇ f ( x ) − ∇ f ( y ) , x − y ⟩ ; (13) \frac{\mu L}{\mu+ L}\lVert x-y\rVert^2 + \frac{1}{\mu+ L}\lVert\nabla f(x)-\nabla f(y)\rVert^2 \le\langle\nabla f(x)-\nabla f(y),x-y\rangle; \tag*{(13)} μ + L μL ∥ x − y ∥ 2 + μ + L 1 ∥ ∇ f ( x ) − ∇ f ( y ) ∥ 2 ≤ ⟨ ∇ f ( x ) − ∇ f ( y ) , x − y ⟩ ; (13)
see, e.g., [39 Theorem 2.1.12] . Since μ > 0 \mu> 0 μ > 0 , f f f admits a unique minimizer x ∗ x_* x ∗ , i.e., the global minimum f ∗ : = inf x ∈ R d f ( x ) f_* := \inf_{x\in\mathbb{R}^d} f(x) f ∗ := inf x ∈ R d f ( x ) is achieved at x ∗ x_* x ∗ . The subclass of S μ , L \mathcal{S}_{\mu,L} S μ , L consisting of quadratic functions is denoted by Q μ , L \mathcal{Q}_{\mu,L} Q μ , L .
We will also consider the class of convex functions, i.e., μ = 0 \mu= 0 μ = 0 , that are L L L -smooth, which consist of convex functions with an L L L -Lipschitz gradient [39 ] . We will denote this class by F L \mathcal{F}_L F L . Clearly, Q μ , L ⊂ S μ , L ⊂ F L \mathcal{Q}_{\mu,L} \subset\mathcal{S}_{\mu,L} \subset\mathcal{F}_L Q μ , L ⊂ S μ , L ⊂ F L .
Generalized momentum methods. Given f ∈ S μ , L f \in\mathcal{S}_{\mu,L} f ∈ S μ , L , we consider the generalized momentum methods (GMM) for solving the unconstrained minimization problem min x ∈ R d f ( x ) \min_{x\in\mathbb{R}^d} f(x) min x ∈ R d f ( x ) . When subject to time-delayed gradients, GMM admits the update
x k + 1 = x k − α ( ∇ f ( y k − τ ) + w k ) + β ( x k − x k − 1 ) , y k = x k + ν ( x k − x k − 1 ) , (14) x_{k+1} = x_k - \alpha\bigl(\nabla f(y_{k-\tau}) + w_k\bigr) + \beta(x_k-x_{k-1}), \qquad y_k = x_k + \nu(x_k-x_{k-1}), \tag*{(14)} x k + 1 = x k − α ( ∇ f ( y k − τ ) + w k ) + β ( x k − x k − 1 ) , y k = x k + ν ( x k − x k − 1 ) , (14)
starting from the initialization x 0 , x − 1 , … , x − 1 − τ ∈ R d x_0,x_{-1},\ldots,x_{-1-\tau} \in\mathbb{R}^d x 0 , x − 1 , … , x − 1 − τ ∈ R d where τ ∈ Z + \tau\in\mathbb{Z}_+ τ ∈ Z + is the time delay in the gradients and w k ∈ R d w_k \in\mathbb{R}^d w k ∈ R d is the additive gradient error at step k k k . For simplicity, we take x 0 = x − 1 = ⋯ = x − 1 − τ x_0 = x_{-1} = \cdots= x_{-1-\tau} x 0 = x − 1 = ⋯ = x − 1 − τ . GMM admits three parameters α \alpha α , β \beta β and ν \nu ν : The parameter α > 0 \alpha> 0 α > 0 is the stepsize, and β , ν ≥ 0 \beta,\nu\ge0 β , ν ≥ 0 are the momentum parameters . In the absence of delays (when τ = 0 \tau= 0 τ = 0 ), these methods have been previously studied in the literature with and without gradient errors [Gür, VSFL18]. GMM encompasses several well-known momentum-based first-order algorithms as special cases in the absence of time-delays and gradient errors. For the setting τ = 0 \tau= 0 τ = 0 , when ν = 0 \nu= 0 ν = 0 , GMM reduces to the Polyak’s heavy-ball (HB) method; whereas when ν = β \nu= \beta ν = β , it recovers Nesterov’s accelerated gradient (NAG). In addition, triple momentum method (TMM) is obtained by the specific parameter choice α = 1 + ρ ˉ L \alpha= \frac{1+\bar{\rho}}{L} α = L 1 + ρ ˉ , β = ρ ˉ 2 2 − ρ ˉ \beta= \frac{\bar{\rho}^2}{2-\bar{\rho}} β = 2 − ρ ˉ ρ ˉ 2 , ν = ρ ˉ 2 ( 1 + ρ ˉ ) ( 2 − ρ ˉ ) \nu= \frac{\bar{\rho}^2}{(1+\bar{\rho})(2-\bar{\rho})} ν = ( 1 + ρ ˉ ) ( 2 − ρ ˉ ) ρ ˉ 2 , where ρ ˉ = 1 − 1 / κ \bar{\rho} = 1 - 1/\sqrt{\kappa} ρ ˉ = 1 − 1/ κ [54 ] . In the absence of momentum (when β = ν = 0 \beta= \nu= 0 β = ν = 0 ), GMM recovers the gradient descent method.
GMM as a dynamical system. We next write GMM as a dynamical system. Notice that if we introduce the vector
ξ k = [ x k x k − 1 ⋯ x k − τ − 1 ] ∈ R ( τ + 2 ) d , \xi_k =
\begin{bmatrix}
x_k \\
x_{k-1} \\
\cdots\\
x_{k-\tau-1}
\end{bmatrix}
\in\mathbb{R}^{(\tau+2)d}, ξ k = x k x k − 1 ⋯ x k − τ − 1 ∈ R ( τ + 2 ) d ,
which concatenates the iterates from time k − τ − 1 k-\tau-1 k − τ − 1 to time k k k , then noisy time-delayed GMM dynamics in (14) can be rewritten as a discrete-time dynamical system
ξ k + 1 = A ξ k + B ∇ f ( C ξ k ) , (15) \xi_{k+1}=A\xi_k+B\nabla f(C\xi_k), \tag*{(15)} ξ k + 1 = A ξ k + B ∇ f ( C ξ k ) , (15)
where the matrices A A A , B B B and C C C are defined as
A = A ~ ⊗ I d , B = B ~ ⊗ I d , C = C ~ ⊗ I d , (16) A=\widetilde{A}\otimes I_d,\qquad B=\widetilde{B}\otimes I_d,\qquad C=\widetilde{C}\otimes I_d, \tag*{(16)} A = A ⊗ I d , B = B ⊗ I d , C = C ⊗ I d , (16)
with
B ~ = [ − α 0 ⋯ 0 ] T ∈ R ( τ + 2 ) × 1 , C ~ = [ 0 1 × τ 1 + ν − ν ] ∈ R 1 × ( τ + 2 ) , \widetilde{B}=\begin{bmatrix}-\alpha&0&\cdots&0\end{bmatrix}^{\mathsf{T}}\in\mathbb{R}^{(\tau+2)\times1},\qquad\widetilde{C}=\begin{bmatrix}0_{1\times\tau}&1+\nu&-\nu\end{bmatrix}\in\mathbb{R}^{1\times(\tau+2)}, B = [ − α 0 ⋯ 0 ] T ∈ R ( τ + 2 ) × 1 , C = [ 0 1 × τ 1 + ν − ν ] ∈ R 1 × ( τ + 2 ) ,
and
A ~ = [ 1 + β − β 0 ⋯ 0 0 1 0 0 ⋯ 0 0 0 1 0 ⋯ 0 0 ⋮ ⋱ ⋱ ⋱ ⋮ ⋮ 0 ⋯ 0 1 0 0 0 ⋯ 0 0 1 0 ] ∈ R ( τ + 2 ) × ( τ + 2 ) for τ > 0 , \widetilde{A}=
\begin{bmatrix}
1+\beta&-\beta&0&\cdots&0&0\\
1&0&0&\cdots&0&0\\
0&1&0&\cdots&0&0\\
\vdots&\ddots&\ddots&\ddots&\vdots&\vdots\\
0&\cdots&0&1&0&0\\
0&\cdots&0&0&1&0
\end{bmatrix}\in\mathbb{R}^{(\tau+2)\times(\tau+2)}
\qquad\text{for }\tau>0, A = 1 + β 1 0 ⋮ 0 0 − β 0 1 ⋱ ⋯ ⋯ 0 0 0 ⋱ 0 0 ⋯ ⋯ ⋯ ⋱ 1 0 0 0 0 ⋮ 0 1 0 0 0 ⋮ 0 0 ∈ R ( τ + 2 ) × ( τ + 2 ) for τ > 0 ,
whereas
A ~ = [ 1 + β − β 1 0 ] ∈ R 2 × 2 for τ = 0. \widetilde{A}=
\begin{bmatrix}
1+\beta&-\beta\\
1&0
\end{bmatrix}\in\mathbb{R}^{2\times2}
\qquad\text{for }\tau=0. A = [ 1 + β 1 − β 0 ] ∈ R 2 × 2 for τ = 0.
Linear convergence rate. For f ∈ S μ , L f\in\mathcal{S}_{\mu,L} f ∈ S μ , L , we say that GMM converges (globally) linearly with rate ρ \rho ρ if for every initialization x 0 x_0 x 0 , it holds that ∥ x k − x ⋆ ∥ ≤ a k ρ k ∥ x 0 − x ⋆ ∥ \lVert x_k-x_\star\rVert\leq a_k\rho^k\lVert x_0-x_\star\rVert ∥ x k − x ⋆ ∥ ≤ a k ρ k ∥ x 0 − x ⋆ ∥ for some ρ ∈ [ 0 , 1 ) \rho\in[0,1) ρ ∈ [ 0 , 1 ) and scalar coefficients a k a_k a k having at most polynomial growth, i.e., there exist real scalars c 1 , c 2 ≥ 0 c_1,c_2\geq0 c 1 , c 2 ≥ 0 (that does not depend on the initialization x 0 x_0 x 0 ) such that 0 ≤ a k ≤ c 1 ( 1 + k ) c 2 0\leq a_k\leq c_1(1+k)^{c_2} 0 ≤ a k ≤ c 1 ( 1 + k ) c 2 for every k ≥ 0 k\geq0 k ≥ 0 . By L L L -smoothness, we have f ( x k ) − f ( x ⋆ ) ≤ L 2 ∥ x k − x ⋆ ∥ 2 f(x_k)-f(x_\star)\leq\frac{L}{2}\lVert x_k-x_\star\rVert^2 f ( x k ) − f ( x ⋆ ) ≤ 2 L ∥ x k − x ⋆ ∥ 2 so that if GMM converges linearly with rate ρ \rho ρ then f ( x k ) − f ( x ⋆ ) ≤ L a k 2 ρ 2 k ∥ x 0 − x ⋆ ∥ 2 f(x_k)-f(x_\star)\leq La_k^2\rho^{2k}\lVert x_0-x_\star\rVert^2 f ( x k ) − f ( x ⋆ ) ≤ L a k 2 ρ 2 k ∥ x 0 − x ⋆ ∥ 2 .
Lower bounds
Theorem 2.1 (Lower bound under delay; [3 Theorem 4] ). Fix a delay τ ≥ 0 \tau\geq0 τ ≥ 0 and consider any first-order method whose iterates x 0 , x 1 , … x_0,x_1,\ldots x 0 , x 1 , … are generated from delayed gradients, in the sense that
x 0 = ⋯ = x τ = 0 , x k + 1 ∈ span { ∇ F ( x 0 ) , … , ∇ F ( x k − τ ) } for all k ≥ τ . (17) x_0=\cdots=x_\tau=0,\qquad x_{k+1}\in\operatorname{span}\{\nabla F(x_0),\ldots,\nabla F(x_{k-\tau})\}\qquad\text{for all }k\geq\tau. \tag*{(17)} x 0 = ⋯ = x τ = 0 , x k + 1 ∈ span { ∇ F ( x 0 ) , … , ∇ F ( x k − τ )} for all k ≥ τ . (17) Then for every κ > 1 \kappa>1 κ > 1 , k ≥ τ + 1 k\geq\tau+1 k ≥ τ + 1 and every sufficiently large dimension d d d :
(Strongly convex case.) There exists F ∈ S μ , L F\in\mathcal{S}_{\mu,L} F ∈ S μ , L that is quadratic, with minimizer x ⋆ x_\star x ⋆ , such that
F ( x k ) − F ( x ⋆ ) ≥ μ 4 exp ( − 5 k ( κ − 1 ) ( τ + 1 ) ) ∥ x 0 − x ⋆ ∥ 2 . (18) F(x_k)-F(x_\star)\geq\frac{\mu}{4}\exp\left(-\frac{5k}{(\sqrt{\kappa}-1)(\tau+1)}\right)\lVert x_0-x_\star\rVert^2. \tag*{(18)} F ( x k ) − F ( x ⋆ ) ≥ 4 μ exp ( − ( κ − 1 ) ( τ + 1 ) 5 k ) ∥ x 0 − x ⋆ ∥ 2 . (18) (Convex case.) There exists an L L L -smooth convex quadratic F : R d → R F:\mathbb{R}^d\to\mathbb{R} F : R d → R , with minimizer x ⋆ x_\star x ⋆ , such that
F ( x k ) − F ( x ⋆ ) ≥ L ( τ + 1 ) 2 ∥ x 0 − x ⋆ ∥ 2 45 k 2 . (19) F(x_k)-F(x_\star)\geq\frac{L(\tau+1)^2\lVert x_0-x_\star\rVert^2}{45k^2}. \tag*{(19)} F ( x k ) − F ( x ⋆ ) ≥ 45 k 2 L ( τ + 1 ) 2 ∥ x 0 − x ⋆ ∥ 2 . (19)
One can interpret Theorem (2.1) as an obstruction to delay-free acceleration: no first-order method of the form (17) can retain the undelayed accelerated contraction factor once a delay is present. We make this precise below.
Corollary 2.2 (The contraction factor must degrade with the delay). Fix κ = L / μ > 1 \kappa=L/\mu>1 κ = L / μ > 1 and a delay τ ≥ 1 \tau\ge1 τ ≥ 1 , and consider a first-order method satisfying (17) in every dimension. Suppose there exist constants C 0 > 0 C_0>0 C 0 > 0 and c ∈ ( 0 , κ ] c\in(0,\sqrt{\kappa}] c ∈ ( 0 , κ ] , depending only on μ , L , τ \mu,L,\tau μ , L , τ and the chosen method, such that for every dimension d d d , every F : R d → R F:\mathbb{R}^d\to\mathbb{R} F : R d → R in S μ , L \mathcal{S}_{\mu,L} S μ , L , and every k ≥ 0 k\ge0 k ≥ 0 ,
F ( x k ) − F ( x ⋆ ) ≤ C 0 ρ 2 k ∥ x 0 − x ⋆ ∥ 2 , ρ = 1 − c κ ∈ [ 0 , 1 ) . (20) F(x_k)-F(x_\star)\le C_0\rho^{2k}\lVert x_0-x_\star\rVert^2,\qquad\rho=1-\frac{c}{\sqrt{\kappa}}\in[0,1). \tag*{(20)} F ( x k ) − F ( x ⋆ ) ≤ C 0 ρ 2 k ∥ x 0 − x ⋆ ∥ 2 , ρ = 1 − κ c ∈ [ 0 , 1 ) . (20) Then, necessarily it holds that
c ≤ 5 κ 2 ( κ − 1 ) ( τ + 1 ) . (21) c\le\frac{5\sqrt{\kappa}}{2(\sqrt{\kappa}-1)(\tau+1)}. \tag*{(21)} c ≤ 2 ( κ − 1 ) ( τ + 1 ) 5 κ . (21) In particular, for κ ≥ 2 \kappa\ge2 κ ≥ 2 , we have c ≤ c 0 / τ c\le c_0/\tau c ≤ c 0 / τ for the universal constant c 0 = 5 2 / ( 2 − 1 ) > 0 c_0=5\sqrt{2}/(\sqrt{2}-1)>0 c 0 = 5 2 / ( 2 − 1 ) > 0 . Hence, c c c cannot be chosen independently of τ \tau τ : any constant rate of the form ρ = 1 − c / κ \rho=1-c/\sqrt{\kappa} ρ = 1 − c / κ with c c c bounded away from 0 uniformly in τ \tau τ is unattainable.
Proof. By the upper bound (20) and the lower bound (18) ,
μ 4 exp ( − 5 k ( κ − 1 ) ( τ + 1 ) ) ∥ x 0 − x ⋆ ∥ 2 ≤ F ( x k ) − F ( x ⋆ ) ≤ C 0 ρ 2 k ∥ x 0 − x ⋆ ∥ 2 , \frac{\mu}{4}\exp\left(-\frac{5k}{(\sqrt{\kappa}-1)(\tau+1)}\right)\lVert x_0-x_\star\rVert^2\le F(x_k)-F(x_\star)\le C_0\rho^{2k}\lVert x_0-x_\star\rVert^2, 4 μ exp ( − ( κ − 1 ) ( τ + 1 ) 5 k ) ∥ x 0 − x ⋆ ∥ 2 ≤ F ( x k ) − F ( x ⋆ ) ≤ C 0 ρ 2 k ∥ x 0 − x ⋆ ∥ 2 ,
for every k ≥ 0 k\ge0 k ≥ 0 . Since ρ 2 k = ( 1 − c κ ) 2 k ≤ e − c κ 2 k \rho^{2k}=\left(1-\frac{c}{\sqrt{\kappa}}\right)^{2k}\le e^{-\frac{c}{\sqrt{\kappa}}2k} ρ 2 k = ( 1 − κ c ) 2 k ≤ e − κ c 2 k , comparing the exponents, this is only possible if
− 5 k ( κ − 1 ) ( τ + 1 ) ≥ − c κ 2 k , -\frac{5k}{(\sqrt{\kappa}-1)(\tau+1)}\ge-\frac{c}{\sqrt{\kappa}}2k, − ( κ − 1 ) ( τ + 1 ) 5 k ≥ − κ c 2 k ,
which implies c ≤ 5 κ 2 ( κ − 1 ) ( τ + 1 ) c\le\frac{5\sqrt{\kappa}}{2(\sqrt{\kappa}-1)(\tau+1)} c ≤ 2 ( κ − 1 ) ( τ + 1 ) 5 κ . For κ ≥ 2 \kappa\ge2 κ ≥ 2 , we get c ≤ 5 2 2 ( 2 − 1 ) ( τ + 1 ) = O ( 1 / τ ) c\le\frac{5\sqrt{2}}{2(\sqrt{2}-1)(\tau+1)}=O(1/\tau) c ≤ 2 ( 2 − 1 ) ( τ + 1 ) 5 2 = O ( 1/ τ ) as claimed. □
The quadratic case: necessity, exact constants, and optimality
Throughout this section, we assume that there are no gradient errors, i.e., w k = 0 w_k=0 w k = 0 for all k ≥ 0 k\ge0 k ≥ 0 . Consider a strongly convex quadratic objective f ∈ Q μ , L f\in\mathcal{Q}_{\mu,L} f ∈ Q μ , L . Then, f f f admits the representation
f ( x ) = 1 2 x T Q x + b T x + c , (22) f(x)=\frac{1}{2}x^\mathsf{T}Qx+b^\mathsf{T}x+c, \tag*{(22)} f ( x ) = 2 1 x T Q x + b T x + c , (22)
where Q Q Q is a symmetric positive-definite d × d d\times d d × d matrix, b ∈ R d b\in\mathbb{R}^d b ∈ R d and c ∈ R c\in\mathbb{R} c ∈ R . Consider the eigenvalue decomposition Q = U Λ U T Q=U\Lambda U^\mathsf{T} Q = U Λ U T where U U U is orthogonal satisfying U U T = U T U = I d UU^\mathsf{T}=U^\mathsf{T}U=I_d U U T = U T U = I d and Λ \Lambda Λ is diagonal where λ i : = Λ i i \lambda_i:=\Lambda_{ii} λ i := Λ ii are the eigenvalues of Q Q Q in non-decreasing order, i.e.,
μ ≤ λ 1 ≤ λ 2 ≤ ⋯ ≤ λ d ≤ L . \mu\le\lambda_1\le\lambda_2\le\cdots\le\lambda_d\le L. μ ≤ λ 1 ≤ λ 2 ≤ ⋯ ≤ λ d ≤ L .
Note that for quadratic f f f , we have ∇ f ( x ) = Q ( x − x ⋆ ) \nabla f(x)=Q(x-x_\star) ∇ f ( x ) = Q ( x − x ⋆ ) so that the GMM system in (15) becomes
ξ k + 1 − ξ ⋆ = A Q ( ξ k − ξ ⋆ ) with A Q : = A + B Q C , (23) \xi_{k+1}-\xi_\star=A_Q(\xi_k-\xi_\star)\quad\text{with}\quad A_Q:=A+BQC, \tag*{(23)} ξ k + 1 − ξ ⋆ = A Q ( ξ k − ξ ⋆ ) with A Q := A + B QC , (23)
where we used A ξ ⋆ = ξ ⋆ A\xi_\star=\xi_\star A ξ ⋆ = ξ ⋆ and C ξ ⋆ = x ⋆ C\xi_\star=x_\star C ξ ⋆ = x ⋆ and the vector
ξ ⋆ : = [ x ⋆ T x ⋆ T ⋯ x ⋆ T ] T ∈ R ( τ + 2 ) d , \xi_\star:=\begin{bmatrix}x_\star^\mathsf{T}&x_\star^\mathsf{T}&\cdots&x_\star^\mathsf{T}\end{bmatrix}^\mathsf{T}\in\mathbb{R}^{(\tau+2)d}, ξ ⋆ := [ x ⋆ T x ⋆ T ⋯ x ⋆ T ] T ∈ R ( τ + 2 ) d ,
concatenates τ + 2 \tau+2 τ + 2 copies of the unique optimum x ⋆ x_\star x ⋆ . By the recursion (23) , ξ k − ξ ⋆ = A Q k ( ξ 0 − ξ ⋆ ) \xi_k-\xi_\star=A_Q^k(\xi_0-\xi_\star) ξ k − ξ ⋆ = A Q k ( ξ 0 − ξ ⋆ ) for any k ≥ 0 k\ge0 k ≥ 0 ; therefore, the linear convergence rate ρ \rho ρ is given by the spectral radius of the iteration matrix A Q A_Q A Q . The following result shows that the convergence rate can be characterized by the roots of polynomials.
Theorem 3.1. Consider time-delayed GMM with delay parameter τ ≥ 0 \tau\ge0 τ ≥ 0 when f f f is a quadratic of the form (22) . For any given λ > 0 \lambda> 0 λ > 0 , consider the following degree ( τ + 2 ) (\tau+ 2) ( τ + 2 ) -polynomial
P τ , δ ( z ) = z τ ( z − 1 ) ( z − β ) + δ [ ( 1 + ν ) z − ν ] , δ : = α λ . (24) P_{\tau,\delta}(z) = z^\tau(z-1)(z-\beta) + \delta[(1+\nu)z-\nu], \qquad\delta:= \alpha\lambda. \tag*{(24)} P τ , δ ( z ) = z τ ( z − 1 ) ( z − β ) + δ [( 1 + ν ) z − ν ] , δ := α λ . (24) Then, z ∈ σ ( A Q ) z \in\sigma(A_Q) z ∈ σ ( A Q ) if and only if P τ , δ i ( z ) = 0 P_{\tau,\delta_i}(z) = 0 P τ , δ i ( z ) = 0 with δ i = α λ i \delta_i = \alpha\lambda_i δ i = α λ i and λ i ∈ σ ( Q ) \lambda_i \in\sigma(Q) λ i ∈ σ ( Q ) for some 1 ≤ i ≤ d 1 \le i \le d 1 ≤ i ≤ d ; therefore, we have
ρ ( A Q ) = max 1 ≤ i ≤ d max { ∣ z ∣ : P τ , δ i ( z ) = 0 , δ i = α λ i } . \rho(A_Q) = \max_{1 \le i \le d} \max\{|z| : P_{\tau,\delta_i}(z) = 0,\quad\delta_i = \alpha\lambda_i\}. ρ ( A Q ) = 1 ≤ i ≤ d max max { ∣ z ∣ : P τ , δ i ( z ) = 0 , δ i = α λ i } . In particular, if the GMM parameters ( α , β , ν ) ∈ S G M M : = { ( α , β , ν ) : ρ ( A Q ) < 1 } (\alpha,\beta,\nu) \in\mathcal{S}_{GMM}:=\{(\alpha,\beta,\nu):\rho(A_Q)<1\} ( α , β , ν ) ∈ S GM M := {( α , β , ν ) : ρ ( A Q ) < 1 } , then GMM will converge linearly with rate ρ = ρ ( A Q ) \rho=\rho(A_Q) ρ = ρ ( A Q ) .
Proof. In the absence of gradient errors, the dynamics is linear. It is well-known from the standard stability theory of linear difference equations that we have linear convergence for GMM if and only if the spectral radius ρ ( A Q ) < 1 \rho(A_Q)<1 ρ ( A Q ) < 1 [53 ] and in this case the convergence rate will be ρ = ρ ( A Q ) \rho=\rho(A_Q) ρ = ρ ( A Q ) . In the rest of the proof, we will derive the claimed formula for ρ ( A Q ) \rho(A_Q) ρ ( A Q ) . Using the definition of B B B and C C C , we first note that the matrix A Q = A + B Q C A_Q=A+BQC A Q = A + B QC admits a block representation of the form A Q = [ M i j + N i j ( Q ) ] i , j = 1 τ + 2 A_Q=[M_{ij}+N_{ij}(Q)]_{i,j=1}^{\tau+2} A Q = [ M ij + N ij ( Q ) ] i , j = 1 τ + 2 where each M i j M_{ij} M ij and N i j ( Q ) N_{ij}(Q) N ij ( Q ) is a d × d d\times d d × d block given by
M i j = { ( 1 + β ) I d , ( i , j ) = ( 1 , 1 ) , − β I d , ( i , j ) = ( 1 , 2 ) , I d , i = j + 1 , 0 d , otherwise , and N i j ( Q ) = { − α ( 1 + ν ) Q , ( i , j ) = ( 1 , τ + 1 ) , α ν Q , ( i , j ) = ( 1 , τ + 2 ) , 0 d , otherwise . (25) M_{ij} =
\begin{cases}
(1+\beta)I_d, & (i,j)=(1,1),\\
-\beta I_d, & (i,j)=(1,2),\\
I_d, & i=j+1,\\
0_d, & \text{otherwise},
\end{cases}
\qquad\text{and}\qquad
N_{ij}(Q) =
\begin{cases}
-\alpha(1+\nu)Q, & (i,j)=(1,\tau+1),\\
\alpha\nu Q, & (i,j)=(1,\tau+2),\\
0_d, & \text{otherwise}.
\end{cases} \tag*{(25)} M ij = ⎩ ⎨ ⎧ ( 1 + β ) I d , − β I d , I d , 0 d , ( i , j ) = ( 1 , 1 ) , ( i , j ) = ( 1 , 2 ) , i = j + 1 , otherwise , and N ij ( Q ) = ⎩ ⎨ ⎧ − α ( 1 + ν ) Q , α ν Q , 0 d , ( i , j ) = ( 1 , τ + 1 ) , ( i , j ) = ( 1 , τ + 2 ) , otherwise . (25)
The matrix N i j ( Q ) N_{ij}(Q) N ij ( Q ) is a function of the input matrix Q Q Q . For example, for τ ≥ 3 \tau\ge3 τ ≥ 3 , we have
A Q = [ ( 1 + β ) I d − β I d 0 d ⋯ 0 d − α ( 1 + ν ) Q α ν Q I d 0 d 0 d ⋯ 0 d 0 d 0 d I d 0 d ⋯ 0 d 0 d 0 d 0 d I d ⋯ 0 d 0 d ⋮ ⋮ ⋮ ⋱ ⋮ ⋮ 0 d 0 d 0 d ⋯ I d 0 d 0 d 0 d 0 d ⋯ 0 d I d 0 d ] ∈ R ( τ + 2 ) d × ( τ + 2 ) d . A_Q =
\begin{bmatrix}
(1+\beta)I_d & -\beta I_d & 0_d & \cdots& 0_d & -\alpha(1+\nu)Q & \alpha\nu Q\\
I_d & 0_d & 0_d & \cdots& 0_d & 0_d\\
0_d & I_d & 0_d & \cdots& 0_d & 0_d\\
0_d & 0_d & I_d & \cdots& 0_d & 0_d\\
\vdots& \vdots& \vdots& \ddots& \vdots& \vdots\\
0_d & 0_d & 0_d & \cdots& I_d & 0_d\\
0_d & 0_d & 0_d & \cdots& 0_d & I_d & 0_d
\end{bmatrix}
\in\mathbb{R}^{(\tau+2)d\times(\tau+2)d}. A Q = ( 1 + β ) I d I d 0 d 0 d ⋮ 0 d 0 d − β I d 0 d I d 0 d ⋮ 0 d 0 d 0 d 0 d 0 d I d ⋮ 0 d 0 d ⋯ ⋯ ⋯ ⋯ ⋱ ⋯ ⋯ 0 d 0 d 0 d 0 d ⋮ I d 0 d − α ( 1 + ν ) Q 0 d 0 d 0 d ⋮ 0 d I d α ν Q 0 d ∈ R ( τ + 2 ) d × ( τ + 2 ) d .
In the following, we will argue that A Q A_Q A Q is similar to a block diagonal matrix with ( τ + 2 ) × ( τ + 2 ) (\tau+2)\times(\tau+2) ( τ + 2 ) × ( τ + 2 ) diagonal blocks. Since Q Q Q admits the eigenvalue decomposition Q = U Λ U T Q=U\Lambda U^{\mathsf{T}} Q = U Λ U T , using the fact that U U T = I d UU^{\mathsf{T}}=I_d U U T = I d , if we define
V = diag ( U , U , … , U ) V=\operatorname{diag}(U,U,\ldots,U) V = diag ( U , U , … , U )
then V − 1 = V T V^{-1}=V^{\mathsf{T}} V − 1 = V T and
A Q = V A Λ V T where A Λ = [ M i j + N i j ( Λ ) ] i , j = 1 τ + 2 . (26) A_Q=VA_\Lambda V^{\mathsf{T}} \qquad\text{where}\qquad A_\Lambda=[M_{ij}+N_{ij}(\Lambda)]_{i,j=1}^{\tau+2}. \tag*{(26)} A Q = V A Λ V T where A Λ = [ M ij + N ij ( Λ ) ] i , j = 1 τ + 2 . (26)
is a block matrix with diagonal blocks—here, N i j ( Λ ) N_{ij}(\Lambda) N ij ( Λ ) is defined as in (25) , replacing Q Q Q with Λ \Lambda Λ . Let P ∈ R ( τ + 2 ) d × ( τ + 2 ) d P\in\mathbb{R}^{(\tau+2)d\times(\tau+2)d} P ∈ R ( τ + 2 ) d × ( τ + 2 ) d be the permutation matrix that reorders the elements of ξ k = [ x k T x k − 1 T ⋯ x k − τ − 1 T ] T \xi_k=[x_k^{\mathsf{T}}\ x_{k-1}^{\mathsf{T}}\ \cdots\ x_{k-\tau-1}^{\mathsf{T}}]^{\mathsf{T}} ξ k = [ x k T x k − 1 T ⋯ x k − τ − 1 T ] T in increasing coordinates, i.e., P P P is the permutation matrix such that given any ξ k \xi_k ξ k , P ξ k P\xi_k P ξ k is formed according to
P ξ k = [ ξ k [ 1 ] ξ k [ 2 ] ⋮ ξ k [ d ] ] with ξ k [ i ] = [ ( x k ) i ( x k − 1 ) i ⋮ ( x k − τ − 1 ) i ] , ∀ i = 1 , … , d . P\xi_k=
\begin{bmatrix}
\xi_k^{[1]}\\
\xi_k^{[2]}\\
\vdots\\
\xi_k^{[d]}
\end{bmatrix}
\qquad\text{with}\qquad
\xi_k^{[i]}=
\begin{bmatrix}
(x_k)_i\\
(x_{k-1})_i\\
\vdots\\
(x_{k-\tau-1})_i
\end{bmatrix},
\quad\forall i=1,\ldots,d. P ξ k = ξ k [ 1 ] ξ k [ 2 ] ⋮ ξ k [ d ] with ξ k [ i ] = ( x k ) i ( x k − 1 ) i ⋮ ( x k − τ − 1 ) i , ∀ i = 1 , … , d .
where ( x k ) i (x_k)_i ( x k ) i denotes the i i i -th (element) coordinate of x k x_k x k . Note that such a matrix P P P is independent of the iteration counter k k k . Thus, it is straightforward to check
P A Λ P T = blkdiag ( A ~ ( λ 1 ) , A ~ ( λ 2 ) , … , A ~ ( λ d ) ) (27) P A_{\Lambda} P^{\mathsf{T}} = \operatorname{blkdiag}\bigl(\widetilde{A}^{(\lambda_1)},\widetilde{A}^{(\lambda_2)},\ldots,\widetilde{A}^{(\lambda_d)}\bigr) \tag*{(27)} P A Λ P T = blkdiag ( A ( λ 1 ) , A ( λ 2 ) , … , A ( λ d ) ) (27)
with
A ~ ( λ ) = [ M ~ i j + N ~ i j ( λ ) ] i , j = 1 τ + 2 ∈ R ( τ + 2 ) × ( τ + 2 ) , \widetilde{A}^{(\lambda)} = \left[\widetilde{M}_{ij}+\widetilde{N}_{ij}(\lambda)\right]_{i,j=1}^{\tau+2} \in\mathbb{R}^{(\tau+2)\times(\tau+2)}, A ( λ ) = [ M ij + N ij ( λ ) ] i , j = 1 τ + 2 ∈ R ( τ + 2 ) × ( τ + 2 ) ,
where each M ~ i j \widetilde{M}_{ij} M ij and N ~ i j ( λ ) \widetilde{N}_{ij}(\lambda) N ij ( λ ) is a scalar given by
M ~ i j = { 1 + β , ( i , j ) = ( 1 , 1 ) , − β , ( i , j ) = ( 1 , 2 ) , 1 , i = j + 1 , 0 , otherwise , and N ~ i j ( λ ) = { − α ( 1 + ν ) λ , ( i , j ) = ( 1 , τ + 1 ) , α ν λ , ( i , j ) = ( 1 , τ + 2 ) , 0 , otherwise . \widetilde{M}_{ij} =
\begin{cases}
1+\beta, & (i,j)=(1,1),\\
-\beta, & (i,j)=(1,2),\\
1, & i=j+1,\\
0, & \text{otherwise},
\end{cases}
\qquad\text{and}\qquad
\widetilde{N}_{ij}(\lambda) =
\begin{cases}
-\alpha(1+\nu)\lambda, & (i,j)=(1,\tau+1),\\
\alpha\nu\lambda, & (i,j)=(1,\tau+2),\\
0, & \text{otherwise}.
\end{cases} M ij = ⎩ ⎨ ⎧ 1 + β , − β , 1 , 0 , ( i , j ) = ( 1 , 1 ) , ( i , j ) = ( 1 , 2 ) , i = j + 1 , otherwise , and N ij ( λ ) = ⎩ ⎨ ⎧ − α ( 1 + ν ) λ , α ν λ , 0 , ( i , j ) = ( 1 , τ + 1 ) , ( i , j ) = ( 1 , τ + 2 ) , otherwise .
Hence, from (27) , (26) we conclude that for S : = V P T S := VP^{\mathsf{T}} S := V P T , we have
A Q = S blkdiag ( A ~ ( λ 1 ) , A ~ ( λ 2 ) , … , A ~ ( λ d ) ) S − 1 , A_Q = S\operatorname{blkdiag}\bigl(\widetilde{A}^{(\lambda_1)},\widetilde{A}^{(\lambda_2)},\ldots,\widetilde{A}^{(\lambda_d)}\bigr)S^{-1}, A Q = S blkdiag ( A ( λ 1 ) , A ( λ 2 ) , … , A ( λ d ) ) S − 1 ,
which follows from S − 1 = S T S^{-1}=S^{\mathsf{T}} S − 1 = S T due to P T = P − 1 P^{\mathsf{T}}=P^{-1} P T = P − 1 and V − 1 = V T V^{-1}=V^{\mathsf{T}} V − 1 = V T . Then, using a similarity transformation implies that the characteristic polynomial of A Q A_Q A Q satisfies
p τ ( z ) : = det ( z I − A Q ) = det ( z I − blkdiag ( A ~ ( λ 1 ) , A ~ ( λ 2 ) , … , A ~ ( λ d ) ) ) = ∏ i = 1 d p τ , λ i ( z ) p_{\tau}(z) := \det(zI-A_Q)=\det\bigl(zI-\operatorname{blkdiag}(\widetilde{A}^{(\lambda_1)},\widetilde{A}^{(\lambda_2)},\ldots,\widetilde{A}^{(\lambda_d)})\bigr)=\prod_{i=1}^{d}p_{\tau,\lambda_i}(z) p τ ( z ) := det ( z I − A Q ) = det ( z I − blkdiag ( A ( λ 1 ) , A ( λ 2 ) , … , A ( λ d ) ) ) = i = 1 ∏ d p τ , λ i ( z )
where
p τ , λ i ( z ) : = det ( z I − A ~ ( λ i ) ) , ∀ i = 1 , … , d . (28) p_{\tau,\lambda_i}(z) := \det\bigl(zI-\widetilde{A}^{(\lambda_i)}\bigr), \qquad\forall\ i=1,\ldots,d. \tag*{(28)} p τ , λ i ( z ) := det ( z I − A ( λ i ) ) , ∀ i = 1 , … , d . (28)
It is straightforward to see that the matrix A ~ ( λ ) \widetilde{A}^{(\lambda)} A ( λ ) is similar to a companion matrix and admits the characteristic polynomial
p τ , λ ( z ) = z τ + 2 + ∑ j = 0 τ + 1 a j ( λ ) z j , p_{\tau,\lambda}(z)=z^{\tau+2}+\sum_{j=0}^{\tau+1}a_j(\lambda)z^j, p τ , λ ( z ) = z τ + 2 + j = 0 ∑ τ + 1 a j ( λ ) z j ,
with a τ + 2 − j ( λ ) : = − ( M ~ 1 j + N ~ 1 j ( λ ) ) a_{\tau+2-j}(\lambda):=-(\widetilde{M}_{1j}+\widetilde{N}_{1j}(\lambda)) a τ + 2 − j ( λ ) := − ( M 1 j + N 1 j ( λ )) for 1 ≤ j ≤ τ + 2 1\leq j\leq\tau+2 1 ≤ j ≤ τ + 2 . Equivalently,
a τ + 1 ( λ ) = { − 1 − β if τ ≥ 2 , − 1 − β if τ = 1 , − 1 − β + α ( 1 + ν ) λ if τ = 0 , a τ ( λ ) = { β if τ ≥ 2 , β + α λ ( 1 + ν ) if τ = 1 , β − α ν λ if τ = 0. (29) a_{\tau+1}(\lambda)=
\begin{cases}
-1-\beta& \text{if }\tau\geq2,\\
-1-\beta& \text{if }\tau=1,\\
-1-\beta+\alpha(1+\nu)\lambda& \text{if }\tau=0,
\end{cases}
\qquad
a_{\tau}(\lambda)=
\begin{cases}
\beta& \text{if }\tau\geq2,\\
\beta+\alpha\lambda(1+\nu) & \text{if }\tau=1,\\
\beta-\alpha\nu\lambda& \text{if }\tau=0.
\end{cases} \tag*{(29)} a τ + 1 ( λ ) = ⎩ ⎨ ⎧ − 1 − β − 1 − β − 1 − β + α ( 1 + ν ) λ if τ ≥ 2 , if τ = 1 , if τ = 0 , a τ ( λ ) = ⎩ ⎨ ⎧ β β + α λ ( 1 + ν ) β − α ν λ if τ ≥ 2 , if τ = 1 , if τ = 0. (29)
a 1 ( λ ) = { α ( 1 + ν ) λ if τ ≥ 2 , β + α ( 1 + ν ) λ if τ = 1 , − 1 − β + α ( 1 + ν ) λ if τ = 0 , a 0 ( λ ) = { − α ν λ if τ ≥ 2 , − α ν λ if τ = 1 , β − α ν λ if τ = 0. (30) a_1(\lambda)=
\begin{cases}
\alpha(1+\nu)\lambda& \text{if }\tau\geq2,\\
\beta+\alpha(1+\nu)\lambda& \text{if }\tau=1,\\
-1-\beta+\alpha(1+\nu)\lambda& \text{if }\tau=0,
\end{cases}
\qquad
a_0(\lambda)=
\begin{cases}
-\alpha\nu\lambda& \text{if }\tau\geq2,\\
-\alpha\nu\lambda& \text{if }\tau=1,\\
\beta-\alpha\nu\lambda& \text{if }\tau=0.
\end{cases} \tag*{(30)} a 1 ( λ ) = ⎩ ⎨ ⎧ α ( 1 + ν ) λ β + α ( 1 + ν ) λ − 1 − β + α ( 1 + ν ) λ if τ ≥ 2 , if τ = 1 , if τ = 0 , a 0 ( λ ) = ⎩ ⎨ ⎧ − α ν λ − α ν λ β − α ν λ if τ ≥ 2 , if τ = 1 , if τ = 0. (30)
and a n ( λ ) = 0 a_n(\lambda)=0 a n ( λ ) = 0 if τ ≥ 3 \tau\geq3 τ ≥ 3 and n ∈ [ 2 , τ − 1 ] n\in[2,\tau-1] n ∈ [ 2 , τ − 1 ] . We also notice p τ , λ i ( z ) = P τ , δ i ( z ) p_{\tau,\lambda_i}(z)=P_{\tau,\delta_i}(z) p τ , λ i ( z ) = P τ , δ i ( z ) where the latter is a more compact representation. Thus, z z z is an eigenvalue of A Q A_Q A Q if and only if p τ ( z ) = ∏ i = 1 d p τ , λ i ( z ) = ∏ i = 1 d P τ , δ i ( z ) = 0 p_{\tau}(z)=\prod_{i=1}^{d}p_{\tau,\lambda_i}(z)=\prod_{i=1}^{d}P_{\tau,\delta_i}(z)=0 p τ ( z ) = ∏ i = 1 d p τ , λ i ( z ) = ∏ i = 1 d P τ , δ i ( z ) = 0 , which implies that
ρ ( A Q ) = max i ρ ( A ~ ( λ i ) ) = max 1 ≤ i ≤ d max { ∣ z ∣ : P τ , δ i ( z ) = 0 } . \rho(A_Q)=\max_i \rho\bigl(\widetilde{A}^{(\lambda_i)}\bigr)=\max_{1\leq i\leq d}\max\{|z|:P_{\tau,\delta_i}(z)=0\}. ρ ( A Q ) = i max ρ ( A ( λ i ) ) = 1 ≤ i ≤ d max max { ∣ z ∣ : P τ , δ i ( z ) = 0 } .
This concludes the proof.
Let Q = U Λ U ⊤ Q = U\Lambda U^{\top} Q = U Λ U ⊤ be the eigenvalue decomposition of Q Q Q , with orthonormal eigenvectors u 1 , … , u d u_1,\ldots,u_d u 1 , … , u d (the columns of U U U ) and Q u i = λ i u i Qu_i = \lambda_i u_i Q u i = λ i u i . Let e k : = x k − x ⋆ e_k := x_k - x_\star e k := x k − x ⋆ denote the error and let
ϵ k ( i ) : = u i ⊤ e k , i = 1 , … , d , \epsilon_k^{(i)} := u_i^{\top}e_k,\qquad i=1,\ldots,d, ϵ k ( i ) := u i ⊤ e k , i = 1 , … , d ,
denote its i i i -th coordinate in the eigenvector basis, so that e k = ∑ i = 1 d ϵ k ( i ) u i e_k = \sum_{i=1}^d \epsilon_k^{(i)}u_i e k = ∑ i = 1 d ϵ k ( i ) u i . Two facts make these coordinates convenient. First, since ∇ f ( y ) = Q ( y − x ⋆ ) \nabla f(y) = Q(y-x_\star) ∇ f ( y ) = Q ( y − x ⋆ ) and u i ⊤ Q = λ i u i ⊤ u_i^{\top}Q = \lambda_i u_i^{\top} u i ⊤ Q = λ i u i ⊤ ,
u i ⊤ ∇ f ( y ) = λ i u i ⊤ ( y − x ⋆ ) ∀ y ∈ R d , u_i^{\top}\nabla f(y) = \lambda_i u_i^{\top}(y-x_\star)\qquad\forall\, y\in\mathbb{R}^d, u i ⊤ ∇ f ( y ) = λ i u i ⊤ ( y − x ⋆ ) ∀ y ∈ R d ,
i.e., in the i i i -th coordinate of the gradient ∇ f ( x k ) \nabla f(x_k) ∇ f ( x k ) under this basis is equal to the error ϵ k ( i ) \epsilon_k^{(i)} ϵ k ( i ) multiplied by the curvature λ i \lambda_i λ i . Second, y k = x k + ν ( x k − x k − 1 ) y_k = x_k + \nu(x_k-x_{k-1}) y k = x k + ν ( x k − x k − 1 ) in (14) implies that y k − τ − x ⋆ = ( 1 + ν ) e k − τ − ν e k − τ − 1 y_{k-\tau}-x_\star= (1+\nu)e_{k-\tau}-\nu e_{k-\tau-1} y k − τ − x ⋆ = ( 1 + ν ) e k − τ − ν e k − τ − 1 ; taking the inner product of both sides of the first equality in (14) with u i u_i u i therefore gives, for each i i i separately,
ϵ k + 1 ( i ) = ( 1 + β ) ϵ k ( i ) − β ϵ k − 1 ( i ) − α λ i [ ( 1 + ν ) ϵ k − τ ( i ) − ν ϵ k − τ − 1 ( i ) ] . (31) \epsilon_{k+1}^{(i)} = (1+\beta)\epsilon_k^{(i)}-\beta\epsilon_{k-1}^{(i)}-\alpha\lambda_i\left[(1+\nu)\epsilon_{k-\tau}^{(i)}-\nu\epsilon_{k-\tau-1}^{(i)}\right]. \tag*{(31)} ϵ k + 1 ( i ) = ( 1 + β ) ϵ k ( i ) − β ϵ k − 1 ( i ) − α λ i [ ( 1 + ν ) ϵ k − τ ( i ) − ν ϵ k − τ − 1 ( i ) ] . (31)
Thus, the d d d -dimensional error recursion splits into d d d independent scalar recursions, one for each eigenvalue of Q Q Q ; this is the block-diagonalization in the proof of Theorem 3.1 . Substituting ϵ k ( i ) = z k \epsilon_k^{(i)} = z^k ϵ k ( i ) = z k into (31) shows that its characteristic polynomial is P τ , δ i P_{\tau,\delta_i} P τ , δ i with δ i = α λ i \delta_i=\alpha\lambda_i δ i = α λ i . In (31) , the step size α \alpha α and the curvature λ i \lambda_i λ i appear only through their product δ i = α λ i \delta_i=\alpha\lambda_i δ i = α λ i , which is the factor by which the recursion scales the delayed, extrapolated error before subtracting it from the current iterate. We call δ = α λ \delta=\alpha\lambda δ = α λ the gain associated with the eigenvalue λ \lambda λ . The name comes from control theory, where the update is viewed as a feedback loop (the iterate determines the point at which the gradient is evaluated, and the gradient is fed back to correct the iterate) and the factor multiplying the fed-back signal is called the loop gain. In optimization terms, δ i \delta_i δ i is the effective, dimensionless step size along the eigendirection u i u_i u i : for gradient descent (β = ν = 0 \beta=\nu=0 β = ν = 0 , τ = 0 \tau=0 τ = 0 ), (31) reduces to ϵ k + 1 ( i ) = ( 1 − δ i ) ϵ k ( i ) \epsilon_{k+1}^{(i)}=(1-\delta_i)\epsilon_k^{(i)} ϵ k + 1 ( i ) = ( 1 − δ i ) ϵ k ( i ) . Since P τ , δ P_{\tau,\delta} P τ , δ depends on α \alpha α and λ \lambda λ only through δ \delta δ , the stability and convergence properties of GMM on quadratics are functions of the gains δ ∈ [ α μ , α L ] \delta\in[\alpha\mu,\alpha L] δ ∈ [ α μ , α L ] together with β \beta β and ν \nu ν .
Before turning to general smooth strongly convex objectives, we analyze the quadratic case, where the augmented error dynamics are linear and the worst-case convergence rate is governed exactly by the roots of the scalar characteristic polynomial P τ , δ ( z ) P_{\tau,\delta}(z) P τ , δ ( z ) . This exactly-solvable case is where the two sharpest claims of the paper are established –which we summarize below in the next two paragraphs as well.
First, it shows that the classical one-parameter momentum families are insufficient. The Schur-stable positive-momentum Nesterov (ν = β \nu=\beta ν = β ) and heavy-ball (ν = 0 \nu=0 ν = 0 ) tunings provably cannot retain κ \sqrt{\kappa} κ -acceleration under any fixed positive delay (Theorems 3.12 and 3.17): a fixed delay confines heavy ball to spectral gap at most O τ ( κ − 1 ) \mathcal{O}_{\tau}(\kappa^{-1}) O τ ( κ − 1 ) and confines Nesterov to gap at most O ( κ − 2 / 3 ) \mathcal{O}(\kappa^{-2/3}) O ( κ − 2/3 ) at τ = 1 \tau=1 τ = 1 and O τ ( κ − 1 ) \mathcal{O}_{\tau}(\kappa^{-1}) O τ ( κ − 1 ) for τ ≥ 2 \tau\ge2 τ ≥ 2 . The additional extrapolation freedom provided by the GMM framework therefore plays a substantive role in the convergence behavior, rather than merely amounting to a cosmetic reparametrization.
Second, it shows that the delay-matched method has the right asymptotic scale. With the very parameters of Theorem 6.1 , i.e., α = 1 / ( M 2 L ) \alpha=1/(M^2L) α = 1/ ( M 2 L ) , β = 1 − 2 q / M \beta=1-2q/M β = 1 − 2 q / M , and ν = M − 1 2 \nu=M-\frac{1}{2} ν = M − 2 1 with M = 2 τ + 1 M=2\tau+1 M = 2 τ + 1 , an elementary root-location argument certifies acceleration on every quadratic for each fixed positive delay and sufficiently large κ \kappa κ (Theorem 3.25 ); a separately optimized one-delay tuning sharpens this at τ = 1 \tau=1 τ = 1 to the exact worst-case spectral radius 1 − 1 2 κ − 1 / 2 1-\frac{1}{2}\kappa^{-1/2} 1 − 2 1 κ − 1/2 (Proposition 3.19 ). In particular, for each fixed τ ≥ 1 \tau\ge1 τ ≥ 1 and κ \kappa κ sufficiently large, Theorem 3.25 gives ρ ≤ 1 − c q u a d / ( τ κ ) \rho\le1-c_{\mathrm{quad}}/(\tau\sqrt{\kappa}) ρ ≤ 1 − c quad / ( τ κ ) for a universal constant c q u a d > 0 c_{\mathrm{quad}}>0 c quad > 0 , matching the dependence on τ \tau τ and κ \kappa κ indicated by Corollary 2.2 .
The main theorem then carries the positive half of this picture—acceleration with these parameters—from quadratics to the entire class S μ , L \mathcal{S}_{\mu,L} S μ , L , where root-location is unavailable and the analytic certificate in Lemma 5.2 (§ 5 ) is required. Throughout the present section f f f is a strongly convex quadratic, i.e., f ∈ Q μ , L f \in\mathcal{Q}_{\mu,L} f ∈ Q μ , L , the quadratics in S μ , L \mathcal{S}_{\mu,L} S μ , L whose Hessian spectrum lies in [ μ , L ] [\mu,L] [ μ , L ] ; the scalar characteristic polynomial defined in (24) governs the quadratic dynamics.
NAG and HB cannot achieve acceleration with κ \sqrt{\kappa} κ scaling
Fix an integer delay τ ≥ 1 \tau\ge1 τ ≥ 1 and a strongly convex quadratic f ∈ Q μ , L f \in\mathcal{Q}_{\mu,L} f ∈ Q μ , L with condition number κ = L / μ ≥ 1 \kappa= L/\mu\ge1 κ = L / μ ≥ 1 . For each eigenvalue λ ∈ [ μ , L ] \lambda\in[\mu,L] λ ∈ [ μ , L ] of the Hessian, recall the scalar characteristic polynomial
P τ , δ ( z ) = z τ ( z − 1 ) ( z − β ) + δ [ ( 1 + ν ) z − ν ] , δ : = α λ . (32) P_{\tau,\delta}(z) = z^\tau(z-1)(z-\beta) + \delta\bigl[(1+\nu)z-\nu\bigr], \qquad\delta:= \alpha\lambda. \tag*{(32)} P τ , δ ( z ) = z τ ( z − 1 ) ( z − β ) + δ [ ( 1 + ν ) z − ν ] , δ := α λ . (32)
defined earlier in (24) . Nesterov is the one-parameter subfamily ν = β \nu=\beta ν = β , for which the characteristic polynomial P τ , δ ( z ) P_{\tau,\delta}(z) P τ , δ ( z ) reduces to
P τ , δ N A G ( z ) = z τ ( z − 1 ) ( z − β ) + δ [ ( 1 + β ) z − β ] . (33) P^{\mathrm{NAG}}_{\tau,\delta}(z) = z^\tau(z-1)(z-\beta) + \delta\bigl[(1+\beta)z-\beta\bigr]. \tag*{(33)} P τ , δ NAG ( z ) = z τ ( z − 1 ) ( z − β ) + δ [ ( 1 + β ) z − β ] . (33)
Terminology. At δ = 0 \delta=0 δ = 0 , the characteristic polynomial P τ , δ P_{\tau,\delta} P τ , δ has τ \tau τ roots at the origin, together with the roots z = β z=\beta z = β and z = 1 z=1 z = 1 . Hence, when δ \delta δ is sufficiently small and β \beta β is close to 1, τ \tau τ roots remain close to the origin while two roots remain close to 1. The modes associated with the roots near the origin decay rapidly and are nonbinding for the asymptotic convergence rate in the regime considered. We therefore refer to the roots of P τ , δ P_{\tau,\delta} P τ , δ that remain close to 1 as the slow iteration modes , and to those near the origin as the fast iteration modes , or simply the nonbinding roots .
The Hessian spectral interval has endpoints μ \mu μ and L L L . Since the characteristic polynomial depends on a Hessian eigenvalue λ ∈ σ ( Q ) \lambda\in\sigma(Q) λ ∈ σ ( Q ) only through
δ = α λ , \delta=\alpha\lambda, δ = α λ ,
we write
s : = 1 − β ∈ ( 0 , 1 ) , Δ : = α L , δ μ : = α μ = Δ κ . (34) s := 1-\beta\in(0,1), \qquad\Delta:= \alpha L, \qquad\delta_\mu:= \alpha\mu= \frac{\Delta}{\kappa}. \tag*{(34)} s := 1 − β ∈ ( 0 , 1 ) , Δ := α L , δ μ := α μ = κ Δ . (34)
The quantity δ = α λ \delta=\alpha\lambda δ = α λ can be viewed as the effective step size along the Hessian eigendirection associated with λ \lambda λ . Thus, as λ \lambda λ ranges over [ μ , L ] [\mu,L] [ μ , L ] , δ ∈ [ δ μ , Δ ] \delta\in[\delta_\mu,\Delta] δ ∈ [ δ μ , Δ ] , with the low-curvature endpoint λ = μ \lambda=\mu λ = μ corresponding to the smallest effective step size δ μ \delta_\mu δ μ , and the high-curvature endpoint λ = L \lambda=L λ = L corresponding to the largest effective step size Δ \Delta Δ . We use the terms low-curvature endpoint and high-curvature endpoint for these two Hessian directions, reserving the terminology slow and fast iteration modes for the roots of P τ , δ P_{\tau,\delta} P τ , δ .
Proof strategy. The two spectral endpoints play complementary roles in the lower-bound argument. The low-curvature endpoint is used to obtain a lower bound on the overall spectral radius. Indeed, since
ρ = sup λ ∈ [ μ , L ] ρ ( P τ , α λ ) , \rho= \sup_{\lambda\in[\mu,L]} \rho(P_{\tau,\alpha\lambda}), ρ = λ ∈ [ μ , L ] sup ρ ( P τ , α λ ) ,
we have ρ ≥ ρ ( P τ , δ μ ) \rho\ge\rho(P_{\tau,\delta_\mu}) ρ ≥ ρ ( P τ , δ μ ) . Because δ μ = Δ / κ \delta_\mu=\Delta/\kappa δ μ = Δ/ κ is small when κ \kappa κ is large, the slow iteration modes of P τ , δ μ P_{\tau,\delta_\mu} P τ , δ μ remain close to 1. Locating these roots yields the lower bounds on the overall spectral radius established in Lemmas 3.8 , 3.9 , and 3.14 . Importantly, the argument does not require the worst-case spectral radius to be attained at λ = μ \lambda=\mu λ = μ ; the lower endpoint is used only to produce a valid lower bound.
The high-curvature endpoint is used instead to derive necessary stability conditions. Any choice of parameters that is Schur stable for every λ ∈ [ μ , L ] \lambda\in[\mu,L] λ ∈ [ μ , L ] must, in particular, make P τ , Δ P_{\tau,\Delta} P τ , Δ Schur stable. Evaluating the stability conditions at λ = L \lambda= L λ = L therefore yields necessary restrictions on the largest effective step size Δ = α L \Delta= \alpha L Δ = α L ; these are the bounds obtained in Propositions 3.6 and 3.16.
The lower-bound theorems combine these two endpoint arguments. For heavy ball at every fixed delay, and for Nesterov when τ ≥ 2 \tau\ge2 τ ≥ 2 , the high-curvature stability conditions force Δ \Delta Δ to be sufficiently small relative to s = 1 − β s = 1 - \beta s = 1 − β . Consequently, δ μ = Δ s \delta_\mu= \frac{\Delta}{s} δ μ = s Δ is too small to move the slow iteration modes of P τ , δ μ P_{\tau,\delta_\mu} P τ , δ μ sufficiently far from 1 1 1 , leading to the O τ ( κ − 1 ) O_\tau(\kappa^{-1}) O τ ( κ − 1 ) spectral-gap bound. For Nesterov with τ = 1 \tau= 1 τ = 1 , the high-curvature stability restriction is weaker, of the form Δ = O ( s ) \Delta= O(\sqrt{s}) Δ = O ( s ) , and balancing this restriction with the low-curvature root bounds instead gives the O ( κ − 2 / 3 ) O(\kappa^{-2/3}) O ( κ − 2/3 ) spectral-gap barrier in Theorem 3.12 .
All statements below about heavy ball and Nesterov—including the quantifier “every tuning” in the theorems and their summaries—are for the standard positive-momentum regime β ∈ ( 0 , 1 ) \beta\in(0,1) β ∈ ( 0 , 1 ) , equivalently s ∈ ( 0 , 1 ) s \in(0,1) s ∈ ( 0 , 1 ) . Within a given one-parameter family—Nesterov (ν = β = 1 − s \nu= \beta= 1 - s ν = β = 1 − s ) or heavy ball (ν = 0 \nu= 0 ν = 0 , treated in § 3.2.4 )—a tuning is fixed by the pair ( α , s ) (\alpha,s) ( α , s ) . For either algorithm family, f a m ∈ { N A G , H B } \mathrm{fam} \in\{\mathrm{NAG},\mathrm{HB}\} fam ∈ { NAG , HB } , on problems with condition number κ \kappa κ and gradient delay τ \tau τ , its worst-case spectral radius and the family’s best certifiable rate in view of Theorem 3.1 are
ρ ( α , β ) : = sup λ ∈ [ μ , L ] max { ∣ z ∣ : P τ , α λ ( z ) = 0 } , ρ f a m ⋆ = ρ f a m ⋆ ( τ , κ ) : = inf { ρ ( α , β ) : α > 0 , β ∈ ( 0 , 1 ) } . \rho(\alpha,\beta) := \sup_{\lambda\in[\mu,L]} \max\{|z| : P_{\tau,\alpha\lambda}(z) = 0\}, \qquad\rho^\star_{\mathrm{fam}} = \rho^\star_{\mathrm{fam}}(\tau,\kappa) := \inf\{\rho(\alpha,\beta) : \alpha> 0,\ \beta\in(0,1)\}. ρ ( α , β ) := λ ∈ [ μ , L ] sup max { ∣ z ∣ : P τ , α λ ( z ) = 0 } , ρ fam ⋆ = ρ fam ⋆ ( τ , κ ) := inf { ρ ( α , β ) : α > 0 , β ∈ ( 0 , 1 )} .
A tuning is Schur stable iff ρ = ρ ( α , β ) < 1 \rho= \rho(\alpha,\beta) < 1 ρ = ρ ( α , β ) < 1 . The family accelerates at delay τ \tau τ iff 1 − ρ f a m ⋆ = Θ ( κ − 1 / 2 ) 1-\rho^\star_{\mathrm{fam}} = \Theta(\kappa^{-1/2}) 1 − ρ fam ⋆ = Θ ( κ − 1/2 ) . In the theorem below, we will show 1 − ρ N A G ⋆ 1-\rho^\star_{\mathrm{NAG}} 1 − ρ NAG ⋆ never achieves this dependence on κ \kappa κ . Before we present this result, we provide some lemmas about stability in the next section.
Note that for HB, we have ν = 0 \nu= 0 ν = 0 ; so that the polynomial P τ , δ P_{\tau,\delta} P τ , δ given in (24) simplifies to
P τ , δ H B ( z ) = z K τ , δ H B ( z ) , where K τ , δ H B ( z ) : = z τ − 1 ( z − 1 ) ( z − β ) + δ . (35) P^{\mathrm{HB}}_{\tau,\delta}(z) = zK^{\mathrm{HB}}_{\tau,\delta}(z), \qquad\text{where } K^{\mathrm{HB}}_{\tau,\delta}(z) := z^{\tau-1}(z-1)(z-\beta) + \delta. \tag*{(35)} P τ , δ HB ( z ) = z K τ , δ HB ( z ) , where K τ , δ HB ( z ) := z τ − 1 ( z − 1 ) ( z − β ) + δ . (35)
Three stability lemmas
The analysis rests on locating the roots of (33) relative to the unit circle. The classical tool for this purpose is the Schur–Cohn criterion [47 , 12 , 35 , 28 ] . The following lemma recalls a standard result: the Schur stability of a polynomial P P P is equivalent to the Hurwitz stability of its Cayley transform P ~ \widetilde{P} P .
Lemma 3.2 (Cayley transformation). Let P P P be a real polynomial of degree n n n with P ( − 1 ) ≠ 0 P(-1) \ne0 P ( − 1 ) = 0 , and define its Cayley transform as the polynomial
P ~ ( w ) : = ( 1 − w ) n P ( 1 + w 1 − w ) . \widetilde{P}(w) := (1-w)^n P\left(\frac{1+w}{1-w}\right). P ( w ) := ( 1 − w ) n P ( 1 − w 1 + w ) . Then P ~ \widetilde{P} P is a real polynomial of degree n n n , and P P P is Schur stable (all roots in ∣ z ∣ < 1 |z| < 1 ∣ z ∣ < 1 ) if and only if P ~ \widetilde{P} P is Hurwitz (all roots in Re w < 0 \operatorname{Re} w < 0 Re w < 0 ).
Proof. The Möbius map w ↦ z = 1 + w 1 − w w \mapsto z = \frac{1+w}{1-w} w ↦ z = 1 − w 1 + w is a bijection of the Riemann sphere with inverse z ↦ w = z − 1 z + 1 z \mapsto w = \frac{z-1}{z+1} z ↦ w = z + 1 z − 1 ; it sends the open left half-plane { w : Re w < 0 } \{w : \operatorname{Re} w < 0\} { w : Re w < 0 } onto the open unit disk { z : ∣ z ∣ < 1 } \{z : |z| < 1\} { z : ∣ z ∣ < 1 } , and also the extended imaginary axis onto the unit circle, and w = 1 ↦ z = ∞ w = 1 \mapsto z = \infty w = 1 ↦ z = ∞ , w = − 1 ↦ z = 0 w = -1 \mapsto z = 0 w = − 1 ↦ z = 0 . Clearing denominators, P ~ ( w ) = ( 1 − w ) n P ( 1 + w 1 − w ) \widetilde{P}(w) = (1-w)^nP\left(\frac{1+w}{1-w}\right) P ( w ) = ( 1 − w ) n P ( 1 − w 1 + w ) is a polynomial of degree at most n n n ; its leading coefficient is P ~ \widetilde{P} P ’s coefficient of w n w^n w n , which equals ( − 1 ) n P ( − 1 ) ≠ 0 (-1)^nP(-1) \ne0 ( − 1 ) n P ( − 1 ) = 0 —equivalently, as w → ∞ w \to\infty w → ∞ the transformed argument tends to z = − 1 z = -1 z = − 1 , so this leading coefficient can vanish only if P ( − 1 ) = 0 P(-1) = 0 P ( − 1 ) = 0 ; hence deg P ~ = n \deg\widetilde{P} = n deg P = n . Each finite root z 0 ≠ − 1 z_0 \ne-1 z 0 = − 1 of P P P corresponds to the root w 0 = z 0 − 1 z 0 + 1 w_0 = \frac{z_0-1}{z_0+1} w 0 = z 0 + 1 z 0 − 1 of P ~ \widetilde{P} P , with ∣ z 0 ∣ < 1 ⟺ Re w 0 < 0 |z_0| < 1 \Longleftrightarrow\operatorname{Re} w_0 < 0 ∣ z 0 ∣ < 1 ⟺ Re w 0 < 0 . As the map is a bijection and degrees agree, P P P has all roots in the disk iff P ~ \widetilde{P} P has all roots in the left half-plane.
When τ = 1 \tau= 1 τ = 1 , the characteristic polynomial P τ , δ P_{\tau,\delta} P τ , δ is a cubic polynomial. The following lemma characterizes Schur stability of cubic polynomials and will be used in our analysis.
Lemma 3.3 (cubic Schur–Cohn conditions for stability). Let P ( z ) = z 3 + a 2 z 2 + a 1 z + a 0 P(z) = z^3 + a_2z^2 + a_1z + a_0 P ( z ) = z 3 + a 2 z 2 + a 1 z + a 0 be real with P ( − 1 ) ≠ 0 P(-1) \ne0 P ( − 1 ) = 0 . P P P is Schur stable if and only if the four Jury conditions hold:
(i) P ( 1 ) > 0 , (ii) P ( − 1 ) < 0 , (iii) ∣ a 0 ∣ < 1 , (iv) ∣ a 0 a 2 − a 1 ∣ < 1 − a 0 2 . (36) \begin{aligned}
\text{(i)}\;&P(1) > 0, \qquad\text{(ii)}\; P(-1) < 0, \qquad\text{(iii)}\; |a_0| < 1, \qquad\text{(iv)}\; |a_0a_2-a_1| < 1-a_0^2. \tag*{(36)}
\end{aligned} (i) P ( 1 ) > 0 , (ii) P ( − 1 ) < 0 , (iii) ∣ a 0 ∣ < 1 , (iv) ∣ a 0 a 2 − a 1 ∣ < 1 − a 0 2 . (36)
Proof. The Schur–Cohn stability conditions for cubics are classical. In the notation P ( z ) = z 3 + a 2 z 2 + a 1 z + a 0 P(z) = z^3 + a_2z^2 + a_1z + a_0 P ( z ) = z 3 + a 2 z 2 + a 1 z + a 0 , they reduce to [20 Theorem A]
(I) P ( 1 ) > 0 , (II) P ( − 1 ) < 0 , (III) ∣ a 0 ∣ < 1 , (IV) − ( a 0 a 2 − a 1 ) < 1 − a 0 2 . (37) \text{(I)}\; P(1) > 0, \qquad\text{(II)}\; P(-1) < 0, \qquad\text{(III)}\; |a_0| < 1, \qquad\text{(IV)}\; -(a_0a_2-a_1) < 1-a_0^2. \tag*{(37)} (I) P ( 1 ) > 0 , (II) P ( − 1 ) < 0 , (III) ∣ a 0 ∣ < 1 , (IV) − ( a 0 a 2 − a 1 ) < 1 − a 0 2 . (37)
Clearly (I)–(III) coincide with (i)–(iii). It remains to show that, under these three conditions, (IV) is equivalent to (iv). Set
J : = a 0 a 2 − a 1 , r : = a 2 + a 0 . J := a_0a_2-a_1, \qquad r := a_2+a_0. J := a 0 a 2 − a 1 , r := a 2 + a 0 .
Condition (IV) gives the lower bound J > − ( 1 − a 0 2 ) J > -(1-a_0^2) J > − ( 1 − a 0 2 ) . We show that the upper bound J < 1 − a 0 2 J < 1-a_0^2 J < 1 − a 0 2 follows from (I)–(III). Indeed,
P ( 1 ) = 1 + a 1 + r > 0 , P ( − 1 ) = − 1 − a 1 + r < 0 ; P(1) = 1+a_1+r > 0, \qquad P(-1) = -1-a_1+r < 0; P ( 1 ) = 1 + a 1 + r > 0 , P ( − 1 ) = − 1 − a 1 + r < 0 ;
hence, − 1 − a 1 < r < 1 + a 1 -1-a_1 < r < 1+a_1 − 1 − a 1 < r < 1 + a 1 . Thus 1 + a 1 > 0 1+a_1 > 0 1 + a 1 > 0 and ∣ r ∣ < 1 + a 1 |r| < 1+a_1 ∣ r ∣ < 1 + a 1 . Since ∣ a 0 ∣ < 1 |a_0| < 1 ∣ a 0 ∣ < 1 , it follows that ∣ a 0 r ∣ < 1 + a 1 |a_0r| < 1+a_1 ∣ a 0 r ∣ < 1 + a 1 , and in particular a 0 r < 1 + a 1 a_0r < 1+a_1 a 0 r < 1 + a 1 . Now J = a 0 a 2 − a 1 = a 0 ( r − a 0 ) − a 1 = a 0 r − a 0 2 − a 1 J = a_0a_2-a_1 = a_0(r-a_0)-a_1 = a_0r-a_0^2-a_1 J = a 0 a 2 − a 1 = a 0 ( r − a 0 ) − a 1 = a 0 r − a 0 2 − a 1 , which implies that
J ≤ ∣ a 0 r ∣ − a 0 2 − a 1 < ( 1 + a 1 ) − a 0 2 − a 1 = 1 − a 0 2 . J \le|a_0r|-a_0^2-a_1 < (1+a_1)-a_0^2-a_1 = 1-a_0^2. J ≤ ∣ a 0 r ∣ − a 0 2 − a 1 < ( 1 + a 1 ) − a 0 2 − a 1 = 1 − a 0 2 .
Therefore, under (I)–(III), condition (IV) is equivalent to − ( 1 − a 0 2 ) < J < 1 − a 0 2 -(1-a_0^2) < J < 1-a_0^2 − ( 1 − a 0 2 ) < J < 1 − a 0 2 . Since ∣ a 0 ∣ < 1 |a_0| < 1 ∣ a 0 ∣ < 1 , we have 1 − a 0 2 > 0 1-a_0^2 > 0 1 − a 0 2 > 0 , and this double inequality is exactly ∣ J ∣ < 1 − a 0 2 |J| < 1-a_0^2 ∣ J ∣ < 1 − a 0 2 . Substituting J = a 0 a 2 − a 1 J = a_0a_2-a_1 J = a 0 a 2 − a 1 gives ∣ a 0 a 2 − a 1 ∣ < 1 − a 0 2 |a_0a_2-a_1| < 1-a_0^2 ∣ a 0 a 2 − a 1 ∣ < 1 − a 0 2 , which is condition (iv). The proof is complete.
When τ = 2 \tau= 2 τ = 2 , the characteristic polynomial P τ , δ P_{\tau,\delta} P τ , δ is a quartic polynomial. The following result characterizes Hurwitz stability of quartic polynomials.
Lemma 3.4 (stability for quartic polynomials). A real quartic A 4 w 4 + A 3 w 3 + A 2 w 2 + A 1 w + A 0 A_4w^4 + A_3w^3 + A_2w^2 + A_1w + A_0 A 4 w 4 + A 3 w 3 + A 2 w 2 + A 1 w + A 0 with A i > 0 A_i > 0 A i > 0 for i = 0 , 1 , 2 , 3 , 4 i = 0,1,2,3,4 i = 0 , 1 , 2 , 3 , 4 is Hurwitz if and only if its third Hurwitz determinant is positive:
H 3 : = A 3 A 2 A 1 − A 1 2 A 4 − A 0 A 3 2 > 0. (38) H_3 := A_3A_2A_1-A_1^2A_4-A_0A_3^2 > 0. \tag*{(38)} H 3 := A 3 A 2 A 1 − A 1 2 A 4 − A 0 A 3 2 > 0. (38)
Proof. The Hurwitz determinants of the quartic are H 1 = A 3 H_1 = A_3 H 1 = A 3 , H 2 = A 3 A 2 − A 4 A 1 H_2 = A_3A_2-A_4A_1 H 2 = A 3 A 2 − A 4 A 1 , H 3 = A 1 H 2 − A 3 2 A 0 = A 3 A 2 A 1 − A 1 2 A 4 − A 0 A 3 2 H_3 = A_1H_2-A_3^2A_0 = A_3A_2A_1-A_1^2A_4-A_0A_3^2 H 3 = A 1 H 2 − A 3 2 A 0 = A 3 A 2 A 1 − A 1 2 A 4 − A 0 A 3 2 , and H 4 = A 0 H 3 H_4 = A_0H_3 H 4 = A 0 H 3 . By the Liénard–Chipart criterion [31 , 4 ] once all coefficients are positive, it suffices to test the odd-indexed Hurwitz determinants H 1 H_1 H 1 , H 3 H_3 H 3 ; since A 3 = H 1 > 0 A_3 = H_1 > 0 A 3 = H 1 > 0 is given, Hurwitz stability is equivalent to H 3 > 0 H_3 > 0 H 3 > 0 .
Before deriving stability restrictions for delayed NAG, we observe that the condition β < 1 \beta< 1 β < 1 is not an additional assumption in the positive-momentum regime. Indeed, as the following lemma shows, Schur stability itself requires β < 1 \beta< 1 β < 1 for every positive delay.
Lemma 3.5 (Necessary momentum bound for delayed NAG). Let τ ≥ 1 \tau\ge1 τ ≥ 1 be an integer, δ > 0 \delta> 0 δ > 0 , and β ∈ ( 0 , ∞ ) \beta\in(0,\infty) β ∈ ( 0 , ∞ ) . Consider the delayed NAG characteristic polynomial
P τ , δ ( z ) = z τ ( z − 1 ) ( z − β ) + δ [ ( 1 + β ) z − β ] . P_{\tau,\delta}(z) = z^\tau(z-1)(z-\beta) + \delta\left[(1+\beta)z-\beta\right]. P τ , δ ( z ) = z τ ( z − 1 ) ( z − β ) + δ [ ( 1 + β ) z − β ] . If P τ , δ P_{\tau,\delta} P τ , δ is Schur stable, then β < 1 \beta< 1 β < 1 .
Proof. Suppose, for contradiction, that β ≥ 1 \beta\ge1 β ≥ 1 . Since τ ≥ 1 \tau\ge1 τ ≥ 1 , the polynomial P τ , δ P_{\tau,\delta} P τ , δ is monic and has constant coefficient − δ β -\delta\beta − δ β . Schur stability therefore implies
0 < δ β < 1 ; hence, 0 < δ < 1 β ≤ 1. 0 < \delta\beta< 1; \qquad\text{hence,} \qquad0 < \delta< \frac{1}{\beta} \le1. 0 < δ β < 1 ; hence, 0 < δ < β 1 ≤ 1. We consider two cases.
Case 1: τ = 1 \tau= 1 τ = 1 . The characteristic polynomial is
P 1 , δ ( z ) = z 3 − ( 1 + β ) z 2 + [ β + δ ( 1 + β ) ] z − δ β . P_{1,\delta}(z) = z^3 - (1+\beta)z^2 + [\beta+\delta(1+\beta)]z-\delta\beta. P 1 , δ ( z ) = z 3 − ( 1 + β ) z 2 + [ β + δ ( 1 + β )] z − δ β . Writing P 1 , δ ( z ) = z 3 + a 2 z 2 + a 1 z + a 0 P_{1,\delta}(z)=z^3+a_2z^2+a_1z+a_0 P 1 , δ ( z ) = z 3 + a 2 z 2 + a 1 z + a 0 , we have
a 2 = − ( 1 + β ) , a 1 = β + δ ( 1 + β ) , a 0 = − δ β . a_2=-(1+\beta), \qquad a_1=\beta+\delta(1+\beta), \qquad a_0=-\delta\beta. a 2 = − ( 1 + β ) , a 1 = β + δ ( 1 + β ) , a 0 = − δ β . By Lemma 3.3 , Schur stability requires ∣ a 0 a 2 − a 1 ∣ < 1 − a 0 2 |a_0a_2-a_1|<1-a_0^2 ∣ a 0 a 2 − a 1 ∣ < 1 − a 0 2 . In particular, since a 0 a 2 − a 1 < 0 a_0a_2-a_1<0 a 0 a 2 − a 1 < 0 , this means
a 0 a 2 − a 1 > − ( 1 − a 0 2 ) . a_0a_2-a_1>-(1-a_0^2). a 0 a 2 − a 1 > − ( 1 − a 0 2 ) . Substituting the coefficients gives
1 − β + δ ( β 2 − 1 ) − δ 2 β 2 > 0. (39) 1-\beta+\delta(\beta^2-1)-\delta^2\beta^2>0. \tag*{(39)} 1 − β + δ ( β 2 − 1 ) − δ 2 β 2 > 0. (39) However, setting t = δ β ∈ ( 0 , 1 ) t=\delta\beta\in(0,1) t = δ β ∈ ( 0 , 1 ) , we obtain the identity
1 − β + δ ( β 2 − 1 ) − δ 2 β 2 = − t 2 + ( β − 1 ) ( β − t ) ( 1 − t ) β < 0 , 1-\beta+\delta(\beta^2-1)-\delta^2\beta^2
=
-\frac{t^2+(\beta-1)(\beta-t)(1-t)}{\beta}<0, 1 − β + δ ( β 2 − 1 ) − δ 2 β 2 = − β t 2 + ( β − 1 ) ( β − t ) ( 1 − t ) < 0 , where the strict inequality follows from β ≥ 1 \beta\ge1 β ≥ 1 and 0 < t < 1 0<t<1 0 < t < 1 . This contradicts (39) .
Case 2: τ ≥ 2 \tau\ge2 τ ≥ 2 . Consider the Cayley transform
P ~ ( w ) = ( 1 − w ) τ + 2 P τ , δ ( 1 + w 1 − w ) . \widetilde{P}(w)=(1-w)^{\tau+2}P_{\tau,\delta}\left(\frac{1+w}{1-w}\right). P ( w ) = ( 1 − w ) τ + 2 P τ , δ ( 1 − w 1 + w ) . By Lemma 3.2 , Schur stability of P τ , δ P_{\tau,\delta} P τ , δ implies that P ~ \widetilde{P} P is Hurwitz stable. Moreover, P ~ ( 0 ) = P τ , δ ( 1 ) = δ > 0 \widetilde{P}(0)=P_{\tau,\delta}(1)=\delta>0 P ( 0 ) = P τ , δ ( 1 ) = δ > 0 . Since a real Hurwitz polynomial with positive constant coefficient has strictly positive coefficients, necessarily P ~ ′ ( 0 ) > 0 \widetilde{P}'(0)>0 P ′ ( 0 ) > 0 . On the other hand, direct differentiation gives
P ~ ′ ( 0 ) = − ( τ + 2 ) P τ , δ ( 1 ) + 2 P τ , δ ′ ( 1 ) = 2 ( 1 − β ) + δ ( 2 β − τ ) . \widetilde{P}'(0)=-(\tau+2)P_{\tau,\delta}(1)+2P_{\tau,\delta}'(1)=2(1-\beta)+\delta(2\beta-\tau). P ′ ( 0 ) = − ( τ + 2 ) P τ , δ ( 1 ) + 2 P τ , δ ′ ( 1 ) = 2 ( 1 − β ) + δ ( 2 β − τ ) . Using τ ≥ 2 \tau\ge2 τ ≥ 2 , β ≥ 1 \beta\ge1 β ≥ 1 , and 0 < δ < 1 0<\delta<1 0 < δ < 1 , we obtain P ~ ′ ( 0 ) ≤ 2 ( 1 − β ) + 2 δ ( β − 1 ) = 2 ( β − 1 ) ( δ − 1 ) ≤ 0 \widetilde{P}'(0)\le2(1-\beta)+2\delta(\beta-1)=2(\beta-1)(\delta-1)\le0 P ′ ( 0 ) ≤ 2 ( 1 − β ) + 2 δ ( β − 1 ) = 2 ( β − 1 ) ( δ − 1 ) ≤ 0 , contradicting P ~ ′ ( 0 ) > 0 \widetilde{P}'(0)>0 P ′ ( 0 ) > 0 . Thus β ≥ 1 \beta\ge1 β ≥ 1 is impossible in either case, and consequently β < 1 \beta<1 β < 1 . ∎
Fast-mode necessary conditions for NAG
At the fast mode λ = L \lambda= L λ = L , we have δ = α L = Δ \delta= \alpha L = \Delta δ = α L = Δ . Schur stability of P τ , Δ P_{\tau,\Delta} P τ , Δ caps the admissible step. The following proposition provides a bound on Δ \Delta Δ for Schur stability.
Proposition 3.6 (fast-mode caps). Let 0 < s < 1 2 0 < s < \frac{1}{2} 0 < s < 2 1 (equivalently β > 1 2 \beta> \frac{1}{2} β > 2 1 ) and suppose the Nesterov polynomial P τ , Δ N A G P_{\tau,\Delta}^{\mathrm{NAG}} P τ , Δ NAG of (33) is Schur stable. Then:
(a) ( τ = 1 ) (\tau= 1) ( τ = 1 ) Δ 2 β 2 < s \Delta^2\beta^2 < s Δ 2 β 2 < s ; hence, Δ < s / β ≤ 2 s \Delta< \sqrt{s}/\beta\le2\sqrt{s} Δ < s / β ≤ 2 s ;
(b) ( τ = 2 ) (\tau= 2) ( τ = 2 ) Δ < s \Delta< s Δ < s ;
(c) ( τ ≥ 3 ) (\tau\ge3) ( τ ≥ 3 ) Δ < 2 s τ − 2 + 2 s ≤ 2 s τ − 2 \Delta< \frac{2s}{\tau- 2 + 2s} \le\frac{2s}{\tau- 2} Δ < τ − 2 + 2 s 2 s ≤ τ − 2 2 s .
Proof. For simplicity, throughout the proof we write P τ , Δ P_{\tau,\Delta} P τ , Δ for the degree-( τ + 2 ) (\tau+ 2) ( τ + 2 ) Nesterov polynomial P τ , Δ N A G P_{\tau,\Delta}^{\mathrm{NAG}} P τ , Δ NAG , suppressing the superscript. Thus, τ = 1 \tau= 1 τ = 1 gives a cubic and τ = 2 \tau= 2 τ = 2 a quartic.
(a) The cubic. For τ = 1 \tau= 1 τ = 1 , P 1 , Δ ( z ) = z 3 − ( 1 + β ) z 2 + [ β + Δ ( 1 + β ) ] z − Δ β P_{1,\Delta}(z) = z^3 - (1 + \beta)z^2 + [\beta+ \Delta(1 + \beta)]z - \Delta\beta P 1 , Δ ( z ) = z 3 − ( 1 + β ) z 2 + [ β + Δ ( 1 + β )] z − Δ β , i.e., a 2 = − ( 1 + β ) a_2 = -(1 + \beta) a 2 = − ( 1 + β ) , a 1 = β + Δ ( 1 + β ) a_1 = \beta+ \Delta(1 + \beta) a 1 = β + Δ ( 1 + β ) , a 0 = − Δ β a_0 = -\Delta\beta a 0 = − Δ β . Since Schur stability is assumed, all four Jury conditions hold. The first two are P ( 1 ) = Δ > 0 P(1) = \Delta> 0 P ( 1 ) = Δ > 0 and P ( − 1 ) = − 2 − 2 β − Δ ( 1 + 2 β ) < 0 P(-1) = -2 - 2\beta- \Delta(1 + 2\beta) < 0 P ( − 1 ) = − 2 − 2 β − Δ ( 1 + 2 β ) < 0 , the third one implies Δ β < 1 \Delta\beta< 1 Δ β < 1 , while the step restriction we need comes from condition (iv). Compute
a 0 a 2 − a 1 = Δ β ( 1 + β ) − β − Δ ( 1 + β ) = − β − Δ ( 1 + β ) s , a_0a_2 - a_1 = \Delta\beta(1 + \beta) - \beta- \Delta(1 + \beta) = -\beta- \Delta(1 + \beta)s, a 0 a 2 − a 1 = Δ β ( 1 + β ) − β − Δ ( 1 + β ) = − β − Δ ( 1 + β ) s ,
so ∣ a 0 a 2 − a 1 ∣ = β + Δ ( 1 + β ) s |a_0a_2 - a_1| = \beta+ \Delta(1 + \beta)s ∣ a 0 a 2 − a 1 ∣ = β + Δ ( 1 + β ) s , and (iv) reads
β + Δ ( 1 + β ) s < 1 − Δ 2 β 2 ⟺ s − Δ ( 1 + β ) s − Δ 2 β 2 > 0. \beta+ \Delta(1 + \beta)s < 1 - \Delta^2\beta^2 \Longleftrightarrow s - \Delta(1 + \beta)s - \Delta^2\beta^2 > 0. β + Δ ( 1 + β ) s < 1 − Δ 2 β 2 ⟺ s − Δ ( 1 + β ) s − Δ 2 β 2 > 0.
Since the left side is positive, Δ 2 β 2 < s ( 1 − Δ ( 1 + β ) ) ≤ s \Delta^2\beta^2 < s(1 - \Delta(1 + \beta)) \le s Δ 2 β 2 < s ( 1 − Δ ( 1 + β )) ≤ s , i.e. Δ < s / β ≤ 2 s \Delta< \sqrt{s}/\beta\le2\sqrt{s} Δ < s / β ≤ 2 s using β > 1 2 \beta> \frac{1}{2} β > 2 1 .
(b) The quartic. For τ = 2 \tau= 2 τ = 2 , P 2 , Δ ( z ) = z 4 + ( s − 2 ) z 3 + ( 1 − s ) z 2 + Δ ( 2 − s ) z − Δ ( 1 − s ) P_{2,\Delta}(z) = z^4 + (s - 2)z^3 + (1 - s)z^2 + \Delta(2 - s)z - \Delta(1 - s) P 2 , Δ ( z ) = z 4 + ( s − 2 ) z 3 + ( 1 − s ) z 2 + Δ ( 2 − s ) z − Δ ( 1 − s ) . By Lemma 3.2 , P 2 , Δ P_{2,\Delta} P 2 , Δ is Schur iff its Cayley transform P ~ ( w ) = ( 1 − w ) 4 P 2 , Δ ( 1 + w 1 − w ) = A 4 w 4 + A 3 w 3 + A 2 w 2 + A 1 w + A 0 \widetilde{P}(w) = (1 - w)^4P_{2,\Delta}\left(\frac{1+w}{1-w}\right) = A_4w^4 + A_3w^3 + A_2w^2 + A_1w + A_0 P ( w ) = ( 1 − w ) 4 P 2 , Δ ( 1 − w 1 + w ) = A 4 w 4 + A 3 w 3 + A 2 w 2 + A 1 w + A 0 is Hurwitz, with
A 4 = 4 − 2 s − 3 Δ + 2 Δ s , A 3 = 8 − 2 s + 8 Δ − 6 Δ s , A 2 = 4 + 2 s − 6 Δ + 6 Δ s , A 1 = 2 s ( 1 − Δ ) , A 0 = Δ . (40) \begin{aligned}
A_4 &= 4 - 2s - 3\Delta+ 2\Delta s, & A_3 &= 8 - 2s + 8\Delta- 6\Delta s,\\
A_2 &= 4 + 2s - 6\Delta+ 6\Delta s, & A_1 &= 2s(1 - \Delta), & A_0 &= \Delta. \tag*{(40)}
\end{aligned} A 4 A 2 = 4 − 2 s − 3Δ + 2Δ s , = 4 + 2 s − 6Δ + 6Δ s , A 3 A 1 = 8 − 2 s + 8Δ − 6Δ s , = 2 s ( 1 − Δ ) , A 0 = Δ. (40)
Since P 2 , Δ P_{2,\Delta} P 2 , Δ is Schur stable, Lemma 3.2 implies that its Cayley transform P ~ \widetilde{P} P is Hurwitz. A real Hurwitz polynomial necessarily has coefficients of the same sign.5 Since A 0 = Δ > 0 A_0 = \Delta> 0 A 0 = Δ > 0 , all coefficients of P ~ \widetilde{P} P are positive. Therefore Lemma 3.4 applies: Schur stability is equivalent to H 3 > 0 H_3 > 0 H 3 > 0 . Expanding H 3 H_3 H 3 defined in (38) , and substituting (40) , we obtain
H 3 = 64 [ s − g ( s ) Δ − ( 2 − s ) Δ 2 − ( 1 − s ) 3 Δ 3 ] , (41) H_3 = 64\left[s - g(s)\Delta- (2 - s)\Delta^2 - (1 - s)^3\Delta^3\right], \tag*{(41)} H 3 = 64 [ s − g ( s ) Δ − ( 2 − s ) Δ 2 − ( 1 − s ) 3 Δ 3 ] , (41)
where g ( s ) : = 1 + s ( 1 − s ) 2 g(s) := 1 + s(1 - s)^2 g ( s ) := 1 + s ( 1 − s ) 2 . Using g ( s ) ≥ 1 g(s) \ge1 g ( s ) ≥ 1 on [ 0 , 1 ] [0,1] [ 0 , 1 ] , Δ > 0 \Delta> 0 Δ > 0 and s ∈ ( 0 , 1 ) s \in(0,1) s ∈ ( 0 , 1 ) ; H 3 > 0 H_3 > 0 H 3 > 0 implies that
s > g ( s ) Δ + ( 2 − s ) Δ 2 + ( 1 − s ) 3 Δ 3 ≥ g ( s ) Δ ≥ Δ , s > g(s)\Delta+ (2 - s)\Delta^2 + (1 - s)^3\Delta^3 \ge g(s)\Delta\ge\Delta, s > g ( s ) Δ + ( 2 − s ) Δ 2 + ( 1 − s ) 3 Δ 3 ≥ g ( s ) Δ ≥ Δ ,
i.e., Δ < s \Delta< s Δ < s .
(c) General τ \tau τ . Write P ~ ( w ) = ( 1 − w ) τ + 2 P τ , Δ ( 1 + w 1 − w ) \widetilde{P}(w) = (1 - w)^{\tau+2}P_{\tau,\Delta}\left(\frac{1+w}{1-w}\right) P ( w ) = ( 1 − w ) τ + 2 P τ , Δ ( 1 − w 1 + w ) . Then P ~ ( 0 ) = P τ , Δ ( 1 ) = Δ > 0 \widetilde{P}(0) = P_{\tau,\Delta}(1) = \Delta> 0 P ( 0 ) = P τ , Δ ( 1 ) = Δ > 0 , and
P ~ ′ ( 0 ) : = d P ~ ( w ) d w ∣ w = 0 = − ( τ + 2 ) P τ , Δ ( 1 ) + 2 P τ , Δ ′ ( 1 ) = − ( τ + 2 ) Δ + 2 [ s + Δ ( 1 + β ) ] \widetilde{P}'(0) := \left.\frac{d\widetilde{P}(w)}{dw}\right|_{w=0} = -(\tau+ 2)P_{\tau,\Delta}(1) + 2P'_{\tau,\Delta}(1) = -(\tau+ 2)\Delta+ 2\left[s + \Delta(1 + \beta)\right] P ′ ( 0 ) := d w d P ( w ) w = 0 = − ( τ + 2 ) P τ , Δ ( 1 ) + 2 P τ , Δ ′ ( 1 ) = − ( τ + 2 ) Δ + 2 [ s + Δ ( 1 + β ) ]
= 2 s + Δ ( 2 β − τ ) = 2 s − Δ ( τ − 2 + 2 s ) , = 2s + \Delta(2\beta-\tau) = 2s-\Delta(\tau-2+2s), = 2 s + Δ ( 2 β − τ ) = 2 s − Δ ( τ − 2 + 2 s ) ,
where we used P τ , Δ ′ ( 1 ) = s + Δ ( 1 + β ) P'_{\tau,\Delta}(1)=s+\Delta(1+\beta) P τ , Δ ′ ( 1 ) = s + Δ ( 1 + β ) and β = 1 − s \beta=1-s β = 1 − s . Since a Hurwitz polynomial has all coefficients of the same sign with its constant term, and P ~ ( 0 ) = Δ > 0 \widetilde{P}(0)=\Delta>0 P ( 0 ) = Δ > 0 , the coefficient of w 1 w^1 w 1 , which equals P ~ ′ ( 0 ) \widetilde{P}'(0) P ′ ( 0 ) is also positive, implying that
\Delta< \frac{2s}{\tau-2+2s} \le\frac{2s}{\tau-2}\quad\text{for}\footnote{Note that for $\tau=2$, the same analysis using this inequality gives the crude bound $\Delta<1$; therefore, for $\tau=2$, a more precise analysis based on the stability of quartic polynomials was given in part (b) of the proof.}
\tau\ge3, \tag*{(42)}
which completes the proof. □
Remark 3.7. The three regimes use three different facets of the same stability machinery. For τ ≥ 3 \tau\ge3 τ ≥ 3 , the argument relies only on the signs of the coefficients of the Cayley transform P ~ \widetilde{P} P . This analysis, however, is too loose for τ = 2 \tau=2 τ = 2 and does not apply to τ = 1 \tau=1 τ = 1 . Accordingly, the case τ = 1 \tau=1 τ = 1 is handled using the cubic Schur–Cohn condition (iv) in Lemma 3.3 , whereas the case τ = 2 \tau=2 τ = 2 is handled using the quartic Liénard–Chipart determinant (41) .
Slow-mode bounds for NAG and HB
At the slow mode λ = μ \lambda=\mu λ = μ , we have δ = δ μ = Δ / κ \delta=\delta_\mu=\Delta/\kappa δ = δ μ = Δ/ κ small. We next provide two lower bounds on ρ \rho ρ for NAG based on the location of the slow modes.
Lemma 3.8 (Lower bound on ρ \rho ρ for NAG). Fix τ ≥ 1 \tau\ge1 τ ≥ 1 and suppose 0 < δ μ ≤ s 2 / 8 0<\delta_\mu\le s^2/8 0 < δ μ ≤ s 2 /8 and 2 τ δ μ / s ≤ 1 / 3 2\tau\delta_\mu/s\le1/3 2 τ δ μ / s ≤ 1/3 . Then P τ , δ μ N A G P^{\mathrm{NAG}}_{\tau,\delta_\mu} P τ , δ μ NAG has a real root in ( 1 − 2 δ μ / s , 1 ) (1-2\delta_\mu/s,1) ( 1 − 2 δ μ / s , 1 ) ; hence, ρ > 1 − 2 δ μ / s \rho>1-2\delta_\mu/s ρ > 1 − 2 δ μ / s .
Proof. Put z 0 = 1 − 2 δ μ / s z_0=1-2\delta_\mu/s z 0 = 1 − 2 δ μ / s . From δ μ ≤ s 2 / 8 \delta_\mu\le s^2/8 δ μ ≤ s 2 /8 , we get 2 δ μ / s ≤ s / 4 2\delta_\mu/s\le s/4 2 δ μ / s ≤ s /4 ; therefore, z 0 ∈ [ 1 − s / 4 , 1 ) ⊂ ( β , 1 ) z_0\in[1-s/4,1)\subset(\beta,1) z 0 ∈ [ 1 − s /4 , 1 ) ⊂ ( β , 1 ) , z 0 − β = s − 2 δ μ / s ≥ 3 s / 4 > 0 z_0-\beta=s-2\delta_\mu/s\ge3s/4>0 z 0 − β = s − 2 δ μ / s ≥ 3 s /4 > 0 , and z 0 > 0 z_0>0 z 0 > 0 . Now P τ , δ μ N A G ( 1 ) = δ μ > 0 P^{\mathrm{NAG}}_{\tau,\delta_\mu}(1)=\delta_\mu>0 P τ , δ μ NAG ( 1 ) = δ μ > 0 ; moreover, using z 0 − 1 = − 2 δ μ / s z_0-1=-2\delta_\mu/s z 0 − 1 = − 2 δ μ / s and ( 1 + β ) z 0 − β < ( 1 + β ) − β = 1 (1+\beta)z_0-\beta<(1+\beta)-\beta=1 ( 1 + β ) z 0 − β < ( 1 + β ) − β = 1 , we get
P τ , δ μ N A G ( z 0 ) = z 0 τ ( z 0 − 1 ) ( z 0 − β ) + δ μ [ ( 1 + β ) z 0 − β ] < − z 0 τ 2 δ μ s 3 s 4 + δ μ = δ μ ( 1 − 3 2 z 0 τ ) . P^{\mathrm{NAG}}_{\tau,\delta_\mu}(z_0)=z_0^\tau(z_0-1)(z_0-\beta)+\delta_\mu[(1+\beta)z_0-\beta]<-z_0^\tau\frac{2\delta_\mu}{s}\frac{3s}{4}+\delta_\mu=\delta_\mu\left(1-\frac{3}{2}z_0^\tau\right). P τ , δ μ NAG ( z 0 ) = z 0 τ ( z 0 − 1 ) ( z 0 − β ) + δ μ [( 1 + β ) z 0 − β ] < − z 0 τ s 2 δ μ 4 3 s + δ μ = δ μ ( 1 − 2 3 z 0 τ ) .
Since 2 δ μ / s ≤ s / 4 ≤ 1 2\delta_\mu/s\le s/4\le1 2 δ μ / s ≤ s /4 ≤ 1 , by Bernoulli inequality, z 0 τ = ( 1 − 2 δ μ / s ) τ ≥ 1 − 2 τ δ μ / s ≥ 1 − 1 / 3 = 2 / 3 z_0^\tau=(1-2\delta_\mu/s)^\tau\ge1-2\tau\delta_\mu/s\ge1-1/3=2/3 z 0 τ = ( 1 − 2 δ μ / s ) τ ≥ 1 − 2 τ δ μ / s ≥ 1 − 1/3 = 2/3 , where in the last inequality we used our assumption on δ μ \delta_\mu δ μ . Therefore, 1 − 3 2 z 0 τ ≤ 0 1-\frac{3}{2}z_0^\tau\le0 1 − 2 3 z 0 τ ≤ 0 , i.e., P τ , δ μ N A G ( z 0 ) < 0 < P τ , δ μ N A G ( 1 ) P^{\mathrm{NAG}}_{\tau,\delta_\mu}(z_0)<0<P^{\mathrm{NAG}}_{\tau,\delta_\mu}(1) P τ , δ μ NAG ( z 0 ) < 0 < P τ , δ μ NAG ( 1 ) . The intermediate value theorem yields a real root in ( z 0 , 1 ) (z_0,1) ( z 0 , 1 ) ; hence, Theorem 3.1 implies that ρ > z 0 \rho>z_0 ρ > z 0 . □
Lemma 3.9 (Lower bound on the spectral radius for τ = 1 \tau=1 τ = 1 ). There exists δ 0 > 0 \delta_0>0 δ 0 > 0 such that the following holds for 0 < δ μ ≤ δ 0 0<\delta_\mu\le\delta_0 0 < δ μ ≤ δ 0 : For τ = 1 \tau=1 τ = 1 and 0 < s ≤ 1 / 2 0<s\le1/2 0 < s ≤ 1/2 , the two roots z 1 , z 2 z_1,z_2 z 1 , z 2 of P 1 , δ μ N A G P^{\mathrm{NAG}}_{1,\delta_\mu} P 1 , δ μ NAG other than the one near 0 0 0 are either a positive real pair or a complex-conjugate pair, with product
z 1 z 2 = 1 − s + O ( δ μ 2 ) , (43) z_1z_2=1-s+O(\delta_\mu^2), \tag*{(43)} z 1 z 2 = 1 − s + O ( δ μ 2 ) , (43) where the implied constant is uniform over 0 < s ≤ 1 / 2 0<s\le1/2 0 < s ≤ 1/2 and 0 < δ μ ≤ δ 0 0<\delta_\mu\le\delta_0 0 < δ μ ≤ δ 0 ; that is, it is independent of both s s s and δ μ \delta_\mu δ μ . Consequently ρ ≥ ∣ z 1 z 2 ∣ = 1 − s + O ( δ μ 2 ) \rho\ge\sqrt{|z_1z_2|}=\sqrt{1-s}+O(\delta_\mu^2) ρ ≥ ∣ z 1 z 2 ∣ = 1 − s + O ( δ μ 2 ) , and 1 − ρ ≤ 1 − 1 − s + O ( δ μ 2 ) = s 1 + 1 − s + O ( δ μ 2 ) 1-\rho\le1-\sqrt{1-s}+O(\delta_\mu^2)=\frac{s}{1+\sqrt{1-s}}+O(\delta_\mu^2) 1 − ρ ≤ 1 − 1 − s + O ( δ μ 2 ) = 1 + 1 − s s + O ( δ μ 2 )
Proof. Substituting z = 1 + ξ z=1+\xi z = 1 + ξ into (33) with τ = 1 \tau=1 τ = 1 , δ = δ μ \delta=\delta_\mu δ = δ μ and β = 1 − s \beta=1-s β = 1 − s gives
P 1 , δ μ N A G ( 1 + ξ ) = Q ( ξ , δ μ , s ) : = ξ 3 + ( 1 + s ) ξ 2 + [ s + δ μ ( 1 + β ) ] ξ + δ μ , (44) P^{\mathrm{NAG}}_{1,\delta_\mu}(1+\xi)=Q(\xi,\delta_\mu,s):=\xi^3+(1+s)\xi^2+[s+\delta_\mu(1+\beta)]\xi+\delta_\mu, \tag*{(44)} P 1 , δ μ NAG ( 1 + ξ ) = Q ( ξ , δ μ , s ) := ξ 3 + ( 1 + s ) ξ 2 + [ s + δ μ ( 1 + β )] ξ + δ μ , (44)
which we regard as a polynomial—hence a C ∞ C^\infty C ∞ function—jointly in the real variables ( ξ , δ μ , s ) (\xi,\delta_\mu,s) ( ξ , δ μ , s ) , and we also note that the momentum coefficient β = β ( s ) \beta=\beta(s) β = β ( s ) is itself polynomial in s s s , so after substituting β = 1 − s \beta=1-s β = 1 − s , Q Q Q remains a polynomial in ( ξ , δ μ , s ) (\xi,\delta_\mu,s) ( ξ , δ μ , s ) . Denote by ξ 1 , ξ 2 , ξ 3 \xi_1,\xi_2,\xi_3 ξ 1 , ξ 2 , ξ 3 the three roots of Q ( ⋅ , δ μ , s ) Q(\cdot,\delta_\mu,s) Q ( ⋅ , δ μ , s ) ; by Vieta’s formulas,
ξ 1 + ξ 2 + ξ 3 = − ( 1 + s ) , ξ 1 ξ 2 ξ 3 = − δ μ . (45) \xi_1+\xi_2+\xi_3=-(1+s),\qquad\xi_1\xi_2\xi_3=-\delta_\mu. \tag*{(45)} ξ 1 + ξ 2 + ξ 3 = − ( 1 + s ) , ξ 1 ξ 2 ξ 3 = − δ μ . (45)
At the unperturbed value δ μ = 0 \delta_\mu=0 δ μ = 0 ,
Q ( ξ , 0 , s ) = ξ 3 + ( 1 + s ) ξ 2 + s ξ = ξ ( ξ + 1 ) ( ξ + s ) , (46) Q(\xi,0,s)=\xi^3+(1+s)\xi^2+s\xi=\xi(\xi+1)(\xi+s), \tag*{(46)} Q ( ξ , 0 , s ) = ξ 3 + ( 1 + s ) ξ 2 + s ξ = ξ ( ξ + 1 ) ( ξ + s ) , (46)
so the root corresponding to z = 0 z=0 z = 0 is ξ 3 0 = − 1 \xi_3^0=-1 ξ 3 0 = − 1 , while the remaining two sit at ξ 1 0 = 0 \xi_1^0=0 ξ 1 0 = 0 and ξ 2 0 = − s \xi_2^0=-s ξ 2 0 = − s .
Step 1: a uniformly isolated, smooth root near ξ 3 0 = − 1 \xi_3^0=-1 ξ 3 0 = − 1 . We first show that, for all small δ μ > 0 \delta_\mu>0 δ μ > 0 and all s ∈ [ 0 , 1 2 ] s\in[0,\frac{1}{2}] s ∈ [ 0 , 2 1 ] , the polynomial Q ( ⋅ , δ μ , s ) Q(\cdot,\delta_\mu,s) Q ( ⋅ , δ μ , s ) has exactly one root in the disk D : = { ξ ∈ C : ∣ ξ + 1 ∣ < 1 4 } \mathbb{D}:=\{\xi\in\mathbb{C}:|\xi+1|<\frac{1}{4}\} D := { ξ ∈ C : ∣ ξ + 1∣ < 4 1 } centered at the unperturbed root ξ 3 0 = − 1 \xi_3^0=-1 ξ 3 0 = − 1 , and that this root is real, simple, and a C ∞ C^\infty C ∞ function of ( δ μ , s ) (\delta_\mu,s) ( δ μ , s ) .
Fix s ∈ [ 0 , 1 2 ] s\in[0,\frac{1}{2}] s ∈ [ 0 , 2 1 ] and view ξ ↦ Q ( ξ , δ μ , s ) \xi\mapsto Q(\xi,\delta_\mu,s) ξ ↦ Q ( ξ , δ μ , s ) as a polynomial on C \mathbb{C} C . On the circle ∂ D = { ξ ∈ C : ∣ ξ − ξ 3 0 ∣ = 1 4 } = { ξ ∈ C : ∣ ξ + 1 ∣ = 1 4 } \partial\mathbb{D}=\{\xi\in\mathbb{C}:|\xi-\xi_3^0|=\frac{1}{4}\}=\{\xi\in\mathbb{C}:|\xi+1|=\frac{1}{4}\} ∂ D = { ξ ∈ C : ∣ ξ − ξ 3 0 ∣ = 4 1 } = { ξ ∈ C : ∣ ξ + 1∣ = 4 1 } , we compare Q Q Q with its unperturbed value Q ( ⋅ , 0 , s ) Q(\cdot,0,s) Q ( ⋅ , 0 , s ) from (46) . Their difference is
Q ( ξ , δ μ , s ) − Q ( ξ , 0 , s ) = δ μ [ ( 1 + β ) ξ + 1 ] , Q(\xi,\delta_\mu,s)-Q(\xi,0,s)=\delta_\mu[(1+\beta)\xi+1], Q ( ξ , δ μ , s ) − Q ( ξ , 0 , s ) = δ μ [( 1 + β ) ξ + 1 ] ,
and s ∈ [ 0 , 1 2 ] s\in[0,\frac{1}{2}] s ∈ [ 0 , 2 1 ] implies that β = 1 − s ∈ [ 1 2 , 1 ] \beta=1-s\in[\frac{1}{2},1] β = 1 − s ∈ [ 2 1 , 1 ] , i.e., 1 + β ≤ 2 1+\beta\le2 1 + β ≤ 2 ; thus, every ξ ∈ ∂ D \xi\in\partial\mathbb{D} ξ ∈ ∂ D (for which ∣ ξ ∣ ≤ 5 4 |\xi|\le\frac{5}{4} ∣ ξ ∣ ≤ 4 5 ) satisfies
∣ Q ( ξ , δ μ , s ) − Q ( ξ , 0 , s ) ∣ ≤ ∣ δ μ ∣ ( 2 ⋅ 5 4 + 1 ) = 7 2 ∣ δ μ ∣ ≤ 7 2 δ 0 . (47) |Q(\xi,\delta_\mu,s)-Q(\xi,0,s)|\le|\delta_\mu|\left(2\cdot\frac{5}{4}+1\right)=\frac{7}{2}|\delta_\mu|\le\frac{7}{2}\delta_0. \tag*{(47)} ∣ Q ( ξ , δ μ , s ) − Q ( ξ , 0 , s ) ∣ ≤ ∣ δ μ ∣ ( 2 ⋅ 4 5 + 1 ) = 2 7 ∣ δ μ ∣ ≤ 2 7 δ 0 . (47)
On the same circle, the distances from ξ \xi ξ to the other two unperturbed roots, ξ 1 0 \xi_1^0 ξ 1 0 and ξ 2 0 \xi_2^0 ξ 2 0 , are bounded below uniformly in s s s : ∣ ξ − ξ 1 0 ∣ = ∣ ξ ∣ ≥ 1 − 1 4 = 3 4 |\xi-\xi_1^0|=|\xi|\ge1-\frac{1}{4}=\frac{3}{4} ∣ ξ − ξ 1 0 ∣ = ∣ ξ ∣ ≥ 1 − 4 1 = 4 3 and ∣ ξ − ξ 2 0 ∣ = ∣ ξ + s ∣ ≥ ( 1 − s ) − 1 4 ≥ 1 4 |\xi-\xi_2^0|=|\xi+s|\ge(1-s)-\frac{1}{4}\ge\frac{1}{4} ∣ ξ − ξ 2 0 ∣ = ∣ ξ + s ∣ ≥ ( 1 − s ) − 4 1 ≥ 4 1 ; hence, (46) implies that
∣ Q ( ξ , 0 , s ) ∣ = ∣ ξ ∣ ∣ ξ + 1 ∣ ∣ ξ + s ∣ ≥ 3 4 ⋅ 1 4 ⋅ 1 4 = 3 64 , ∀ ξ ∈ ∂ D . (48) |Q(\xi,0,s)|=|\xi||\xi+1||\xi+s|\ge\frac{3}{4}\cdot\frac{1}{4}\cdot\frac{1}{4}=\frac{3}{64},\qquad\forall\,\xi\in\partial\mathbb{D}. \tag*{(48)} ∣ Q ( ξ , 0 , s ) ∣ = ∣ ξ ∣∣ ξ + 1∣∣ ξ + s ∣ ≥ 4 3 ⋅ 4 1 ⋅ 4 1 = 64 3 , ∀ ξ ∈ ∂ D . (48)
Fix δ 0 ∈ ( 0 , 3 224 ) \delta_0\in(0,\frac{3}{224}) δ 0 ∈ ( 0 , 224 3 ) , then we have 7 2 δ μ < 3 64 \frac{7}{2}\delta_\mu<\frac{3}{64} 2 7 δ μ < 64 3 for 0 ≤ δ μ ≤ δ 0 0\le\delta_\mu\le\delta_0 0 ≤ δ μ ≤ δ 0 , so (47) –(48) give the strict inequality ∣ Q ( ξ , δ μ , s ) − Q ( ξ , 0 , s ) ∣ < ∣ Q ( ξ , 0 , s ) ∣ |Q(\xi,\delta_\mu,s)-Q(\xi,0,s)|<|Q(\xi,0,s)| ∣ Q ( ξ , δ μ , s ) − Q ( ξ , 0 , s ) ∣ < ∣ Q ( ξ , 0 , s ) ∣ on ∂ D \partial\mathbb{D} ∂ D . By Rouché’s theorem, Q ( ⋅ , δ μ , s ) Q(\cdot,\delta_\mu,s) Q ( ⋅ , δ μ , s ) and Q ( ⋅ , 0 , s ) Q(\cdot,0,s) Q ( ⋅ , 0 , s ) have the same number of zeros in D \mathbb{D} D , counted with multiplicity; as ξ 3 0 = − 1 \xi_3^0=-1 ξ 3 0 = − 1 is the only zero of Q ( ⋅ , 0 , s ) Q(\cdot,0,s) Q ( ⋅ , 0 , s ) in D \mathbb{D} D , which is also simple, Q ( ⋅ , δ μ , s ) Q(\cdot,\delta_\mu,s) Q ( ⋅ , δ μ , s ) has exactly one zero in D \mathbb{D} D as well, and it is simple. Let ξ 3 ( δ μ , s ) \xi_3(\delta_\mu,s) ξ 3 ( δ μ , s ) denote that zero as a function of δ μ \delta_\mu δ μ and s s s . Because Q Q Q has real coefficients and D \mathbb{D} D is symmetric about the real axis, the conjugate ξ 3 ‾ \overline{\xi_3} ξ 3 is also a zero in D \mathbb{D} D ; uniqueness of the zero in D \mathbb{D} D implies that ξ 3 = ξ 3 ‾ \xi_3=\overline{\xi_3} ξ 3 = ξ 3 , i.e., ξ 3 \xi_3 ξ 3 is real. All the estimates above are strict and uniform in s ∈ [ 0 , 1 2 ] s\in[0,\frac{1}{2}] s ∈ [ 0 , 2 1 ] , and depend continuously on ( δ μ , s ) (\delta_\mu,s) ( δ μ , s ) ; hence, they persist on an open neighbourhood Ω ⊃ { ( δ μ , s ) ∈ R 2 : ( δ μ , s ) ∈ [ 0 , δ 0 ] × [ 0 , 1 2 ] } \Omega\supset\{(\delta_\mu,s)\in\mathbb{R}^2:(\delta_\mu,s)\in[0,\delta_0]\times[0,\frac{1}{2}]\} Ω ⊃ {( δ μ , s ) ∈ R 2 : ( δ μ , s ) ∈ [ 0 , δ 0 ] × [ 0 , 2 1 ]} , on which Simplicity means ∂ ξ Q ( ξ , δ μ , s ) ∣ ξ = ξ 3 ≠ 0 \partial_\xi Q(\xi,\delta_\mu,s)|_{\xi=\xi_3}\ne0 ∂ ξ Q ( ξ , δ μ , s ) ∣ ξ = ξ 3 = 0 throughout Ω \Omega Ω , so the implicit function theorem [30 Section 3] applies at every point of Ω \Omega Ω : near each point the unique root in D \mathbb{D} D is given by a C ∞ C^\infty C ∞ function, and by uniqueness these local representations all coincide with ξ 3 \xi_3 ξ 3 . Therefore
ξ 3 : Ω → D ∩ R is of class C ∞ , ξ 3 ( 0 , s ) = − 1. \xi_3:\Omega\to\mathbb{D}\cap\mathbb{R}\quad\text{is of class }C^\infty,\qquad\xi_3(0,s)=-1. ξ 3 : Ω → D ∩ R is of class C ∞ , ξ 3 ( 0 , s ) = − 1.
In particular ξ 3 \xi_3 ξ 3 stays within 1 4 \frac{1}{4} 4 1 of − 1 -1 − 1 , so
ξ 3 ( δ μ , s ) ∈ [ − 5 4 , − 3 4 ] , whence ∣ ξ 3 ∣ ≥ 3 4 > 0 on Ω . (49) \xi_3(\delta_\mu,s)\in\left[-\frac{5}{4},-\frac{3}{4}\right],\qquad\text{whence }|\xi_3|\ge\frac{3}{4}>0\text{ on }\Omega. \tag*{(49)} ξ 3 ( δ μ , s ) ∈ [ − 4 5 , − 4 3 ] , whence ∣ ξ 3 ∣ ≥ 4 3 > 0 on Ω. (49)
In the rest, we adopt the notation Q ξ ( ξ ~ , δ , s ) : = ∂ ξ Q ( ξ , δ , s ) ∣ ξ = ξ ~ Q_{\xi}(\widetilde{\xi},\delta,s):=\left.\partial_{\xi}Q(\xi,\delta,s)\right|_{\xi=\widetilde{\xi}} Q ξ ( ξ , δ , s ) := ∂ ξ Q ( ξ , δ , s ) ∣ ξ = ξ and Q δ ( ξ , δ ~ , s ) : = ∂ δ Q ( ξ , δ , s ) ∣ δ = δ ~ Q_{\delta}(\xi,\widetilde{\delta},s):=\left.\partial_{\delta}Q(\xi,\delta,s)\right|_{\delta=\widetilde{\delta}} Q δ ( ξ , δ , s ) := ∂ δ Q ( ξ , δ , s ) ∣ δ = δ .
Step 2: first-order calibration. Recall that ξ 3 ( 0 , s ) = ξ 3 0 = − 1 \xi_{3}(0,s)=\xi_{3}^{0}=-1 ξ 3 ( 0 , s ) = ξ 3 0 = − 1 for s ∈ [ 0 , 1 2 ] s\in[0,\frac{1}{2}] s ∈ [ 0 , 2 1 ] ; hence, Q ξ ( − 1 , 0 , s ) = 3 − 2 ( 1 + s ) + s = 1 − s Q_{\xi}(-1,0,s)=3-2(1+s)+s=1-s Q ξ ( − 1 , 0 , s ) = 3 − 2 ( 1 + s ) + s = 1 − s and Q δ μ ( − 1 , 0 , s ) = − ( 1 + β ) + 1 = − β Q_{\delta_{\mu}}(-1,0,s)=-(1+\beta)+1=-\beta Q δ μ ( − 1 , 0 , s ) = − ( 1 + β ) + 1 = − β . Thus, the implicit function theorem implies that
∂ δ μ ξ 3 ( 0 , s ) = − Q δ μ ( − 1 , 0 , s ) Q ξ ( − 1 , 0 , s ) = β 1 − s = 1 , (50) \partial_{\delta_{\mu}}\xi_{3}(0,s)=-\frac{Q_{\delta_{\mu}}(-1,0,s)}{Q_{\xi}(-1,0,s)}=\frac{\beta}{1-s}=1, \tag*{(50)} ∂ δ μ ξ 3 ( 0 , s ) = − Q ξ ( − 1 , 0 , s ) Q δ μ ( − 1 , 0 , s ) = 1 − s β = 1 , (50)
where the last equality is due to the definition of s : = 1 − β s:=1-\beta s := 1 − β . Since ∂ δ μ 2 ξ 3 ( ⋅ , ⋅ ) \partial_{\delta_{\mu}}^{2}\xi_{3}(\cdot,\cdot) ∂ δ μ 2 ξ 3 ( ⋅ , ⋅ ) is continuous, hence bounded, on the compact set [ 0 , δ 0 ] × [ 0 , 1 2 ] [0,\delta_{0}]\times[0,\frac{1}{2}] [ 0 , δ 0 ] × [ 0 , 2 1 ] , Taylor’s theorem in δ μ \delta_{\mu} δ μ yields
ξ ~ 3 ( δ μ , s ) = − 1 + δ μ + O ( δ μ 2 ) uniformly in s ∈ [ 0 , 1 2 ] . (51) \widetilde{\xi}_{3}(\delta_{\mu},s)=-1+\delta_{\mu}+O(\delta_{\mu}^{2})\quad\text{uniformly in }s\in[0,\tfrac{1}{2}]. \tag*{(51)} ξ 3 ( δ μ , s ) = − 1 + δ μ + O ( δ μ 2 ) uniformly in s ∈ [ 0 , 2 1 ] . (51)
Step 3: the product z 1 z 2 z_{1}z_{2} z 1 z 2 and its C 2 C^{2} C 2 dependence on $s. Let z 3 = 1 + ξ 3 z_{3}=1+\xi_{3} z 3 = 1 + ξ 3 be the root of P 1 , δ μ P_{1,\delta_{\mu}} P 1 , δ μ near $0$, and z 1 = 1 + ξ 1 , z 2 = 1 + ξ 2 z_{1}=1+\xi_{1},z_{2}=1+\xi_{2} z 1 = 1 + ξ 1 , z 2 = 1 + ξ 2 be the other two roots of P τ , δ N A G ( z ) P_{\tau,\delta}^{\mathrm{NAG}}(z) P τ , δ NAG ( z ) defined in (33) . Eliminating ξ 3 \xi_{3} ξ 3 from (45) ,
ξ 1 + ξ 2 = − ( 1 + s ) − ξ 3 , ξ 1 ξ 2 = − δ μ ξ 3 , \xi_{1}+\xi_{2}=-(1+s)-\xi_{3},\qquad\xi_{1}\xi_{2}=-\frac{\delta_{\mu}}{\xi_{3}}, ξ 1 + ξ 2 = − ( 1 + s ) − ξ 3 , ξ 1 ξ 2 = − ξ 3 δ μ ,
where the quotient is well-defined as the denominator ξ 3 ( δ μ , s ) ≠ 0 \xi_{3}(\delta_{\mu},s)\ne0 ξ 3 ( δ μ , s ) = 0 on Ω \Omega Ω by (49) . Hence
g ( s , δ μ ) : = z 1 z 2 = ( 1 + ξ 1 ) ( 1 + ξ 2 ) = 1 + ξ 1 + ξ 2 + ξ 1 ξ 2 = − s − ξ 3 ( δ μ , s ) − δ μ ξ 3 ( δ μ , s ) . (52) g(s,\delta_{\mu}):=z_{1}z_{2}=(1+\xi_{1})(1+\xi_{2})=1+\xi_{1}+\xi_{2}+\xi_{1}\xi_{2}=-s-\xi_{3}(\delta_{\mu},s)-\frac{\delta_{\mu}}{\xi_{3}(\delta_{\mu},s)}. \tag*{(52)} g ( s , δ μ ) := z 1 z 2 = ( 1 + ξ 1 ) ( 1 + ξ 2 ) = 1 + ξ 1 + ξ 2 + ξ 1 ξ 2 = − s − ξ 3 ( δ μ , s ) − ξ 3 ( δ μ , s ) δ μ . (52)
Since ξ 3 \xi_{3} ξ 3 belongs to C ∞ C^{\infty} C ∞ and is bounded away from 0 0 0 on Ω \Omega Ω , we can conclude that g g g also belongs to the class C ∞ C^{\infty} C ∞ jointly in ( s , δ μ ) (s,\delta_{\mu}) ( s , δ μ ) on Ω \Omega Ω . Define R ( s , δ μ ) : = g ( s , δ μ ) − ( 1 − s ) R(s,\delta_{\mu}):=g(s,\delta_{\mu})-(1-s) R ( s , δ μ ) := g ( s , δ μ ) − ( 1 − s ) for ( δ μ , s ) ∈ Ω (\delta_{\mu},s)\in\Omega ( δ μ , s ) ∈ Ω . Since ξ 3 ( 0 , s ) = − 1 \xi_{3}(0,s)=-1 ξ 3 ( 0 , s ) = − 1 , (52) implies that
g ( s , 0 ) = − s − ( − 1 ) − 0 = 1 − s , i.e., R ( s , 0 ) ≡ 0 ∀ s ∈ [ 0 , 1 2 ] . g(s,0)=-s-(-1)-0=1-s,\qquad\text{i.e.,}\quad R(s,0)\equiv0\quad\forall s\in[0,\tfrac{1}{2}]. g ( s , 0 ) = − s − ( − 1 ) − 0 = 1 − s , i.e., R ( s , 0 ) ≡ 0 ∀ s ∈ [ 0 , 2 1 ] .
Differentiating (52) in δ μ \delta_{\mu} δ μ and using ∂ δ μ ξ 3 ( 0 , s ) = 1 \partial_{\delta_{\mu}}\xi_{3}(0,s)=1 ∂ δ μ ξ 3 ( 0 , s ) = 1 from (50) , we get
∂ δ μ g ( s , 0 ) = − ∂ δ μ ξ 3 ( 0 , s ) − 1 ξ 3 ( 0 , s ) = − 1 − 1 − 1 = 0 , i.e., ∂ δ μ R ( s , 0 ) = 0. \partial_{\delta_{\mu}}g(s,0)=-\partial_{\delta_{\mu}}\xi_{3}(0,s)-\frac{1}{\xi_{3}(0,s)}=-1-\frac{1}{-1}=0,\qquad\text{i.e.,}\quad\partial_{\delta_{\mu}}R(s,0)=0. ∂ δ μ g ( s , 0 ) = − ∂ δ μ ξ 3 ( 0 , s ) − ξ 3 ( 0 , s ) 1 = − 1 − − 1 1 = 0 , i.e., ∂ δ μ R ( s , 0 ) = 0.
Thus R ( ⋅ , δ μ ) R(\cdot,\delta_{\mu}) R ( ⋅ , δ μ ) vanishes to second order in δ μ \delta_{\mu} δ μ , uniformly in s s s . Taylor’s theorem with integral remainder gives, for all ( s , δ μ ) ∈ [ 0 , 1 2 ] × [ 0 , δ 0 ] (s,\delta_{\mu})\in[0,\frac{1}{2}]\times[0,\delta_{0}] ( s , δ μ ) ∈ [ 0 , 2 1 ] × [ 0 , δ 0 ] , that
R ( s , δ μ ) = ( ∫ 0 1 ( 1 − t ) ∂ δ μ 2 R ( s , t δ μ ) d t ) δ μ 2 . (53) R(s,\delta_{\mu})=\left(\int_{0}^{1}(1-t)\partial_{\delta_{\mu}}^{2}R(s,t\delta_{\mu})\,dt\right)\delta_{\mu}^{2}. \tag*{(53)} R ( s , δ μ ) = ( ∫ 0 1 ( 1 − t ) ∂ δ μ 2 R ( s , t δ μ ) d t ) δ μ 2 . (53)
Since ∂ δ μ 2 R = ∂ δ μ 2 g \partial_{\delta_{\mu}}^{2}R=\partial_{\delta_{\mu}}^{2}g ∂ δ μ 2 R = ∂ δ μ 2 g is C ∞ C^{\infty} C ∞ , its s s s -derivatives of orders 0 , 1 , 2 0,1,2 0 , 1 , 2 are continuous, hence bounded, on the compact set [ 0 , 1 2 ] × [ 0 , δ 0 ] [0,\frac{1}{2}]\times[0,\delta_{0}] [ 0 , 2 1 ] × [ 0 , δ 0 ] ; differentiating (53) twice under the integral sign is therefore justified, and shows that s ↦ r ( s , δ μ ) s\mapsto r(s,\delta_{\mu}) s ↦ r ( s , δ μ ) is of class C 2 C^{2} C 2 (indeed C ∞ C^{\infty} C ∞ ) with
sup 0 ≤ s ≤ 1 / 2 0 ≤ δ μ ≤ δ 0 ( ∣ r ∣ + ∣ ∂ s r ∣ + ∣ ∂ s 2 r ∣ ) < ∞ . (54) \sup_{\substack{0\le s\le1/2\\0\le\delta_{\mu}\le\delta_{0}}}\left(|r|+|\partial_{s}r|+|\partial_{s}^{2}r|\right)<\infty. \tag*{(54)} 0 ≤ s ≤ 1/2 0 ≤ δ μ ≤ δ 0 sup ( ∣ r ∣ + ∣ ∂ s r ∣ + ∣ ∂ s 2 r ∣ ) < ∞. (54)
Combining (53) with R ( s , δ μ ) : = g ( s , δ μ ) − ( 1 − s ) R(s,\delta_{\mu}):=g(s,\delta_{\mu})-(1-s) R ( s , δ μ ) := g ( s , δ μ ) − ( 1 − s ) and g ( s , δ μ ) = z 1 z 2 g(s,\delta_{\mu})=z_{1}z_{2} g ( s , δ μ ) = z 1 z 2 for ( δ μ , s ) ∈ Ω (\delta_{\mu},s)\in\Omega ( δ μ , s ) ∈ Ω , we get
z 1 z 2 = 1 − s + r ( s , δ μ ) δ μ 2 . (55) z_{1}z_{2}=1-s+r(s,\delta_{\mu})\delta_{\mu}^{2}. \tag*{(55)} z 1 z 2 = 1 − s + r ( s , δ μ ) δ μ 2 . (55)
which establishes (43) ; moreover, we also see that the O ( δ μ 2 ) O(\delta_\mu^2) O ( δ μ 2 ) term in (43) has the explicit form r ( s , δ μ ) δ μ 2 r(s,\delta_\mu)\delta_\mu^2 r ( s , δ μ ) δ μ 2 such that s ↦ r ( s , δ μ ) s\mapsto r(s,\delta_\mu) s ↦ r ( s , δ μ ) is of class C 2 C^2 C 2 and (54) holds. Therefore, in addition to (43) , we can also conclude that ∂ ∂ s ( z 1 z 2 ) = − 1 + O ( δ μ 2 ) \frac{\partial}{\partial s}(z_1z_2)=-1+O(\delta_\mu^2) ∂ s ∂ ( z 1 z 2 ) = − 1 + O ( δ μ 2 ) and ∂ 2 ∂ s 2 ( z 1 z 2 ) = O ( δ μ 2 ) \frac{\partial^2}{\partial s^2}(z_1z_2)=O(\delta_\mu^2) ∂ s 2 ∂ 2 ( z 1 z 2 ) = O ( δ μ 2 ) uniformly in s s s .
Step 4: the spectral-radius bound. After shrinking δ 0 \delta_0 δ 0 if necessary so that δ 0 2 sup s ∣ r ∣ < 1 4 \delta_0^2\sup_s|r|<\frac14 δ 0 2 sup s ∣ r ∣ < 4 1 , the lower bound 1 − s ≥ 1 2 1-s\ge\frac12 1 − s ≥ 2 1 and (55) give
z 1 z 2 = 1 − s + r δ μ 2 ≥ 1 2 − 1 4 = 1 4 > 0. (56) z_1z_2=1-s+r\delta_\mu^2\ge\frac12-\frac14=\frac14>0. \tag*{(56)} z 1 z 2 = 1 − s + r δ μ 2 ≥ 2 1 − 4 1 = 4 1 > 0. (56)
Moreover, by (51) , z 1 + z 2 = 2 + ( ξ 1 + ξ 2 ) = 2 − s − δ μ + O ( δ μ 2 ) ≥ 3 2 − δ μ + O ( δ μ 2 ) > 0 z_1+z_2=2+(\xi_1+\xi_2)=2-s-\delta_\mu+O(\delta_\mu^2)\ge\frac32-\delta_\mu+O(\delta_\mu^2)>0 z 1 + z 2 = 2 + ( ξ 1 + ξ 2 ) = 2 − s − δ μ + O ( δ μ 2 ) ≥ 2 3 − δ μ + O ( δ μ 2 ) > 0 for δ 0 \delta_0 δ 0 small. Since P 1 , δ μ P_{1,\delta_\mu} P 1 , δ μ has real coefficients, the pair ( z 1 , z 2 ) (z_1,z_2) ( z 1 , z 2 ) is either real or complex-conjugate; in the real case, (56) together with z 1 + z 2 > 0 z_1+z_2>0 z 1 + z 2 > 0 forces both to be positive. This proves the dichotomy in the statement. In either case,
ρ ≥ max { ∣ z 1 ∣ , ∣ z 2 ∣ } ≥ ∣ z 1 z 2 ∣ = 1 − s + r δ μ 2 . \rho\ge\max\{|z_1|,|z_2|\}\ge\sqrt{|z_1z_2|}=\sqrt{1-s+r\delta_\mu^2}. ρ ≥ max { ∣ z 1 ∣ , ∣ z 2 ∣ } ≥ ∣ z 1 z 2 ∣ = 1 − s + r δ μ 2 .
Because 1 − s + r δ μ 2 ≥ 1 4 1-s+r\delta_\mu^2\ge\frac14 1 − s + r δ μ 2 ≥ 4 1 by (56) , the elementary identity a + h − a = h / ( a + h + a ) \sqrt{a+h}-\sqrt{a}=h/(\sqrt{a+h}+\sqrt{a}) a + h − a = h / ( a + h + a ) with a = 1 − s a=1-s a = 1 − s and h = r δ μ 2 h=r\delta_\mu^2 h = r δ μ 2 gives
1 − s + r δ μ 2 = 1 − s + r ~ ( s , δ μ ) δ μ 2 , r ~ : = r 1 − s + r δ μ 2 + 1 − s . (57) \sqrt{1-s+r\delta_\mu^2}=\sqrt{1-s}+\widetilde r(s,\delta_\mu)\delta_\mu^2,\qquad\widetilde r:=\frac{r}{\sqrt{1-s+r\delta_\mu^2}+\sqrt{1-s}}. \tag*{(57)} 1 − s + r δ μ 2 = 1 − s + r ( s , δ μ ) δ μ 2 , r := 1 − s + r δ μ 2 + 1 − s r . (57)
The denominator in (57) is bounded below by 2 1 − s ≥ 2 2\sqrt{1-s}\ge\sqrt{2} 2 1 − s ≥ 2 and is C 2 C^2 C 2 in s s s , while the numerator r r r obeys (54) ; hence r ~ \widetilde r r is C 2 C^2 C 2 in s s s with r ~ , ∂ s r ~ , ∂ s 2 r ~ \widetilde r,\partial_s\widetilde r,\partial_s^2\widetilde r r , ∂ s r , ∂ s 2 r bounded uniformly in ( s , δ μ ) (s,\delta_\mu) ( s , δ μ ) . Therefore ρ ≥ 1 − s + O ( δ μ 2 ) \rho\ge\sqrt{1-s}+O(\delta_\mu^2) ρ ≥ 1 − s + O ( δ μ 2 ) , and
1 − ρ ≤ 1 − 1 − s + O ( δ μ 2 ) = s 1 + 1 − s + O ( δ μ 2 ) , 1-\rho\le1-\sqrt{1-s}+O(\delta_\mu^2)=\frac{s}{1+\sqrt{1-s}}+O(\delta_\mu^2), 1 − ρ ≤ 1 − 1 − s + O ( δ μ 2 ) = 1 + 1 − s s + O ( δ μ 2 ) ,
where we use the elementary identity 1 − 1 − s = s 1 + 1 − s 1-\sqrt{1-s}=\frac{s}{1+\sqrt{1-s}} 1 − 1 − s = 1 + 1 − s s . Note that the O ( δ μ 2 ) O(\delta_\mu^2) O ( δ μ 2 ) remainder inherits the same C 2 C^2 C 2 -in-s s s dependence from (57) . This completes the proof. □ \square □
Remark 3.10. The proof of Theorem 3.12 uses only the uniform O ( δ μ 2 ) O(\delta_\mu^2) O ( δ μ 2 ) control in (43) ; no sign information about the remainder is needed.
Lemma 3.11 (The regime s = O ( 1 / κ ) s=O(1/\kappa) s = O ( 1/ κ ) ). Fix τ ≥ 1 \tau\ge1 τ ≥ 1 and B > 0 B>0 B > 0 , and define
C : = B + B + 2. C:=B+\sqrt{B}+2. C := B + B + 2. which depends only on B B B , not on τ \tau τ . Then, there is a threshold κ 0 = κ 0 ( τ , B ) \kappa_0=\kappa_0(\tau,B) κ 0 = κ 0 ( τ , B ) such that, for every κ ≥ κ 0 \kappa\ge\kappa_0 κ ≥ κ 0 and every pair ( s , δ μ ) (s,\delta_\mu) ( s , δ μ ) satisfying
0 < s ≤ B κ , 0 ≤ δ μ ≤ B κ 2 , β = 1 − s , 0<s\le\frac{B}{\kappa},\qquad0\le\delta_\mu\le\frac{B}{\kappa^2},\qquad\beta=1-s, 0 < s ≤ κ B , 0 ≤ δ μ ≤ κ 2 B , β = 1 − s , the delayed Nesterov polynomial P τ , δ N A G P_{\tau,\delta}^{\mathrm{NAG}} P τ , δ NAG , defined in (33) , has two roots6 z 1 , z 2 z_1,z_2 z 1 , z 2 for δ = δ μ \delta=\delta_\mu δ = δ μ such that
max { ∣ z 1 − 1 ∣ , ∣ z 2 − 1 ∣ } ≤ C κ . \max\{|z_1-1|,|z_2-1|\}\le\frac{C}{\kappa}. max { ∣ z 1 − 1∣ , ∣ z 2 − 1∣ } ≤ κ C . Consequently its spectral radius satisfies ρ ≥ 1 − C κ − 1 \rho\ge1-C\kappa^{-1} ρ ≥ 1 − C κ − 1 ; equivalently, 1 − ρ ≤ C κ − 1 1-\rho\le C\kappa^{-1} 1 − ρ ≤ C κ − 1 . The same conclusion holds for the delayed heavy-ball polynomial P τ , δ H B P_{\tau,\delta}^{\mathrm{HB}} P τ , δ HB , defined in (35) , for δ = δ μ \delta=\delta_\mu δ = δ μ : its spectral radius likewise satisfies ρ ≥ 1 − C κ − 1 \rho\ge1-C\kappa^{-1} ρ ≥ 1 − C κ − 1 , with the same constant C C C .
Proof. Given B > 0 B > 0 B > 0 , let ϵ : = κ − 1 \epsilon:= \kappa^{-1} ϵ := κ − 1 , and we set the two small parameters, s s s and δ μ \delta_\mu δ μ , and β \beta β as follows:
s = σ ϵ , δ μ = η ϵ 2 , β = 1 − σ ϵ , for σ ∈ ( 0 , B ] , η ∈ [ 0 , B ] . s = \sigma\epsilon,\qquad\delta_\mu= \eta\epsilon^2,\qquad\beta= 1-\sigma\epsilon,\qquad\text{for }\sigma\in(0,B],\ \eta\in[0,B]. s = σ ϵ , δ μ = η ϵ 2 , β = 1 − σ ϵ , for σ ∈ ( 0 , B ] , η ∈ [ 0 , B ] .
We locate the roots near z = 1 z = 1 z = 1 through the substitution z = 1 + ϵ w z = 1+\epsilon w z = 1 + ϵ w for some w ∈ C w \in\mathbb{C} w ∈ C . Since
z − 1 = ϵ w , z − β = ϵ ( w + σ ) , ( 1 + β ) z − β = 1 + ( 2 − σ ϵ ) ϵ w , z-1 = \epsilon w,\qquad z-\beta= \epsilon(w+\sigma),\qquad(1+\beta)z-\beta= 1+(2-\sigma\epsilon)\epsilon w, z − 1 = ϵ w , z − β = ϵ ( w + σ ) , ( 1 + β ) z − β = 1 + ( 2 − σ ϵ ) ϵ w ,
both characteristic polynomials factor through the same power of ϵ \epsilon ϵ :
P τ , δ μ N A G ( 1 + ϵ w ) = ϵ 2 G ϵ ( w ) , G ϵ ( w ) : = ( 1 + ϵ w ) τ w ( w + σ ) + η [ 1 + ( 2 − σ ϵ ) ϵ w ] , P_{\tau,\delta_\mu}^{\mathrm{NAG}}(1+\epsilon w) = \epsilon^2G_\epsilon(w),\qquad G_\epsilon(w) := (1+\epsilon w)^\tau w(w+\sigma)+\eta\left[1+(2-\sigma\epsilon)\epsilon w\right], P τ , δ μ NAG ( 1 + ϵ w ) = ϵ 2 G ϵ ( w ) , G ϵ ( w ) := ( 1 + ϵ w ) τ w ( w + σ ) + η [ 1 + ( 2 − σ ϵ ) ϵ w ] ,
K τ , δ μ H B ( 1 + ϵ w ) = ϵ 2 G ~ ϵ ( w ) , G ~ ϵ ( w ) : = ( 1 + ϵ w ) τ − 1 w ( w + σ ) + η , K_{\tau,\delta_\mu}^{\mathrm{HB}}(1+\epsilon w) = \epsilon^2\widetilde{G}_\epsilon(w),\qquad\widetilde{G}_\epsilon(w) := (1+\epsilon w)^{\tau-1}w(w+\sigma)+\eta, K τ , δ μ HB ( 1 + ϵ w ) = ϵ 2 G ϵ ( w ) , G ϵ ( w ) := ( 1 + ϵ w ) τ − 1 w ( w + σ ) + η ,
using P τ , δ μ H B = z K τ , δ μ H B P_{\tau,\delta_\mu}^{\mathrm{HB}} = zK_{\tau,\delta_\mu}^{\mathrm{HB}} P τ , δ μ HB = z K τ , δ μ HB , where K τ , δ μ H B K_{\tau,\delta_\mu}^{\mathrm{HB}} K τ , δ μ HB is defined in (35) . As ϵ → 0 \epsilon\to0 ϵ → 0 both G ϵ G_\epsilon G ϵ and G ~ ϵ \widetilde{G}_\epsilon G ϵ reduce to the same ϵ \epsilon ϵ -independent quadratic
g ( w ) : = w ( w + σ ) + η . g(w) := w(w+\sigma)+\eta. g ( w ) := w ( w + σ ) + η .
Step 1: a circle enclosing the roots of g g g . If g ( w ) = 0 g(w) = 0 g ( w ) = 0 , then ∣ w ∣ 2 ≤ σ ∣ w ∣ + η |w|^2 \le\sigma|w|+\eta ∣ w ∣ 2 ≤ σ ∣ w ∣ + η , so ∣ w ∣ ≤ 1 2 ( σ + σ 2 + 4 η ) ≤ σ + η ≤ B + B |w| \le\frac{1}{2}\left(\sigma+\sqrt{\sigma^2+4\eta}\right) \le\sigma+\sqrt{\eta} \le B+\sqrt{B} ∣ w ∣ ≤ 2 1 ( σ + σ 2 + 4 η ) ≤ σ + η ≤ B + B . Hence, with R : = C = B + B + 2 R := C = B+\sqrt{B}+2 R := C = B + B + 2 , both roots of g g g lie in { w : ∣ w ∣ < R − 1 } \{w: |w| < R-1\} { w : ∣ w ∣ < R − 1 } .
Moreover, on the circle { w : ∣ w ∣ = R } \{w: |w| = R\} { w : ∣ w ∣ = R } , using ∣ w + σ ∣ ≥ R − σ ≥ R − B |w+\sigma| \ge R-\sigma\ge R-B ∣ w + σ ∣ ≥ R − σ ≥ R − B and η ≤ B \eta\le B η ≤ B ,
∣ g ( w ) ∣ ≥ ∣ w ∣ ∣ w + σ ∣ − η ≥ R ( R − B ) − B = : m R . |g(w)| \ge|w||w+\sigma|-\eta\ge R(R-B)-B =: m_R. ∣ g ( w ) ∣ ≥ ∣ w ∣∣ w + σ ∣ − η ≥ R ( R − B ) − B =: m R .
Since R − B = B + 2 ≥ 2 R-B = \sqrt{B}+2 \ge2 R − B = B + 2 ≥ 2 , we have m R ≥ 2 R − B = B + 2 B + 4 > 0 m_R \ge2R-B = B+2\sqrt{B}+4 > 0 m R ≥ 2 R − B = B + 2 B + 4 > 0 .
Step 2: a uniform O ( ϵ ) O(\epsilon) O ( ϵ ) perturbation bound on ∣ w ∣ = R |w| = R ∣ w ∣ = R . For ϵ ≤ 1 / R \epsilon\le1/R ϵ ≤ 1/ R we have ∣ ϵ w ∣ ≤ 1 |\epsilon w| \le1 ∣ ϵ w ∣ ≤ 1 on { w : ∣ w ∣ = R } \{w: |w| = R\} { w : ∣ w ∣ = R } , so for every integer p ≥ 0 p \ge0 p ≥ 0 ,
∣ ( 1 + ϵ w ) p − 1 ∣ ≤ ∑ k = 1 p ( p k ) ∣ ϵ w ∣ k ≤ ( 2 p − 1 ) ϵ R . \left|(1+\epsilon w)^p-1\right| \le\sum_{k=1}^{p}\binom{p}{k}|\epsilon w|^k \le(2^p-1)\epsilon R. ∣ ( 1 + ϵ w ) p − 1 ∣ ≤ k = 1 ∑ p ( k p ) ∣ ϵ w ∣ k ≤ ( 2 p − 1 ) ϵ R .
Since ∣ w ( w + σ ) ∣ ≤ R ( R + B ) |w(w+\sigma)| \le R(R+B) ∣ w ( w + σ ) ∣ ≤ R ( R + B ) and ∣ 2 − σ ϵ ∣ ≤ 2 |2-\sigma\epsilon| \le2 ∣2 − σ ϵ ∣ ≤ 2 , for w w w such that ∣ w ∣ = R |w| = R ∣ w ∣ = R , we get
∣ G ϵ ( w ) − g ( w ) ∣ ≤ ( 2 τ − 1 ) ϵ R ⋅ R ( R + B ) + 2 B ϵ R ≤ c 1 ϵ , (58) |G_\epsilon(w)-g(w)| \le(2^\tau-1)\epsilon R\cdot R(R+B)+2B\epsilon R \le c_1\epsilon, \tag*{(58)} ∣ G ϵ ( w ) − g ( w ) ∣ ≤ ( 2 τ − 1 ) ϵ R ⋅ R ( R + B ) + 2 B ϵ R ≤ c 1 ϵ , (58)
∣ G ~ ϵ ( w ) − g ( w ) ∣ ≤ ( 2 τ − 1 − 1 ) ϵ R ⋅ R ( R + B ) ≤ c 1 ϵ , (59) |\widetilde{G}_\epsilon(w)-g(w)| \le(2^{\tau-1}-1)\epsilon R\cdot R(R+B) \le c_1\epsilon, \tag*{(59)} ∣ G ϵ ( w ) − g ( w ) ∣ ≤ ( 2 τ − 1 − 1 ) ϵ R ⋅ R ( R + B ) ≤ c 1 ϵ , (59)
where c 1 : = ( 2 τ − 1 ) R 2 ( R + B ) + 2 B R c_1 := (2^\tau-1)R^2(R+B)+2BR c 1 := ( 2 τ − 1 ) R 2 ( R + B ) + 2 B R depends only on ( τ , B ) (\tau,B) ( τ , B ) .
Step 3: Rouché’s theorem and conclusion. Set κ 0 : = max { R , 1 + c 1 / m R } \kappa_0 := \max\{R,1+c_1/m_R\} κ 0 := max { R , 1 + c 1 / m R } . For κ ≥ κ 0 \kappa\ge\kappa_0 κ ≥ κ 0 , we have ϵ ≤ 1 / R \epsilon\le1/R ϵ ≤ 1/ R , and using the definition ϵ = 1 / κ \epsilon= 1/\kappa ϵ = 1/ κ , we have c 1 ϵ < m R c_1\epsilon< m_R c 1 ϵ < m R . Hence, on { w : ∣ w ∣ = R } \{w: |w| = R\} { w : ∣ w ∣ = R } , we have
∣ G ϵ ( w ) − g ( w ) ∣ < m R ≤ ∣ g ( w ) ∣ , ∣ G ~ ϵ ( w ) − g ( w ) ∣ < m R ≤ ∣ g ( w ) ∣ . |G_\epsilon(w)-g(w)| < m_R \le|g(w)|,\qquad|\widetilde{G}_\epsilon(w)-g(w)| < m_R \le|g(w)|. ∣ G ϵ ( w ) − g ( w ) ∣ < m R ≤ ∣ g ( w ) ∣ , ∣ G ϵ ( w ) − g ( w ) ∣ < m R ≤ ∣ g ( w ) ∣.
By Rouché’s theorem each of G ϵ , G ~ ϵ G_\epsilon,\widetilde{G}_\epsilon G ϵ , G ϵ has the same number of zeros in { w : ∣ w ∣ < R } \{w: |w| < R\} { w : ∣ w ∣ < R } as g g g , counted with multiplicity. Since g g g has two zeros in { w : ∣ w ∣ < R } \{w: |w| < R\} { w : ∣ w ∣ < R } , let w 1 , w 2 w_1,w_2 w 1 , w 2 be the zeros of G ϵ G_\epsilon G ϵ (resp. G ~ ϵ \widetilde{G}_\epsilon G ϵ ) in { w : ∣ w ∣ < R } \{w: |w| < R\} { w : ∣ w ∣ < R } ; hence, z i : = 1 + ϵ w i z_i := 1+\epsilon w_i z i := 1 + ϵ w i for i = 1 , 2 i=1,2 i = 1 , 2 are roots of P τ , δ μ N A G P_{\tau,\delta_\mu}^{\mathrm{NAG}} P τ , δ μ NAG (resp. P τ , δ μ H B P_{\tau,\delta_\mu}^{\mathrm{HB}} P τ , δ μ HB ), and
∣ z i − 1 ∣ = ϵ ∣ w i ∣ < ϵ R = R κ = C κ , i = 1 , 2. |z_i-1| = \epsilon|w_i| < \epsilon R = \frac{R}{\kappa} = \frac{C}{\kappa},\qquad i=1,2. ∣ z i − 1∣ = ϵ ∣ w i ∣ < ϵ R = κ R = κ C , i = 1 , 2.
In particular ∣ z i ∣ ≥ 1 − C / κ |z_i| \ge1-C/\kappa ∣ z i ∣ ≥ 1 − C / κ , so the spectral radius satisfies ρ ≥ max i ∣ z i ∣ ≥ 1 − C κ − 1 \rho\ge\max_i |z_i| \ge1-C\kappa^{-1} ρ ≥ max i ∣ z i ∣ ≥ 1 − C κ − 1 .
Nesterov cannot κ \sqrt{\kappa} κ -accelerate under delay
Theorem 3.12 (Nesterov cannot achieve κ \sqrt{\kappa} κ -acceleration under delay). For the delayed Nesterov method ( ν = β ) (\nu=\beta) ( ν = β ) on Q μ , L Q_{\mu,L} Q μ , L , the best certifiable rate ρ N A G ∗ ( τ , κ ) \rho_{\mathrm{NAG}}^*(\tau,\kappa) ρ NAG ∗ ( τ , κ ) over Schur-stable tunings satisfies
1 − ρ N A G ∗ ≤ { ( 2 + o ( 1 ) ) κ − 2 / 3 , τ = 1 , O ( κ − 1 ) , τ = 2 , O τ ( κ − 1 ) , τ ≥ 3. (60) 1-\rho_{\mathrm{NAG}}^* \le\begin{cases}
(2+o(1))\kappa^{-2/3}, & \tau=1,\\
O(\kappa^{-1}), & \tau=2,\\
O_\tau(\kappa^{-1}), & \tau\ge3.
\end{cases} \tag*{(60)} 1 − ρ NAG ∗ ≤ ⎩ ⎨ ⎧ ( 2 + o ( 1 )) κ − 2/3 , O ( κ − 1 ) , O τ ( κ − 1 ) , τ = 1 , τ = 2 , τ ≥ 3. (60) for every fixed delay τ ∈ Z + \tau\in\mathbb{Z}_+ τ ∈ Z + and all sufficiently large κ \kappa κ . In particular,
1 − ρ N A G ∗ = o ( κ − 1 / 2 ) , 1-\rho_{\mathrm{NAG}}^*=o(\kappa^{-1/2}), 1 − ρ NAG ∗ = o ( κ − 1/2 ) , i.e., a fixed delay prevents the classical Nesterov choice ν = β \nu=\beta ν = β from retaining κ \sqrt{\kappa} κ -acceleration.
Proof. Fix a Schur-stable Nesterov tuning and write s = 1 − β ∈ ( 0 , 1 ) s=1-\beta\in(0,1) s = 1 − β ∈ ( 0 , 1 ) , Δ = α L \Delta=\alpha L Δ = α L , and δ μ = Δ / κ \delta_\mu=\Delta/\kappa δ μ = Δ/ κ . We prove an upper bound on 1 − ρ 1-\rho 1 − ρ for every such tuning; taking the supremum over tunings is the same as bounding 1 − ρ N A G ∗ 1-\rho_{\mathrm{NAG}}^* 1 − ρ NAG ∗ .
We first dispose of the small-momentum regime β ≤ 1 / 2 \beta\le1/2 β ≤ 1/2 which corresponds to s ≥ 1 2 s\ge\frac{1}{2} s ≥ 2 1 . In this case, as shown below, Δ = O τ ( 1 ) \Delta=O_\tau(1) Δ = O τ ( 1 ) for every Schur-stable tuning. Indeed, the fast-mode polynomial P τ , Δ N A G P_{\tau,\Delta}^{\mathrm{NAG}} P τ , Δ NAG is monic of degree n = τ + 2 n=\tau+2 n = τ + 2 and all its roots lie in the unit disk. Write P τ , Δ N A G ( z ) = ∏ i = 1 n ( z − z i ) = z n + ∑ j = 0 n − 1 a j z j P_{\tau,\Delta}^{\mathrm{NAG}}(z)=\prod_{i=1}^n(z-z_i)=z^n+\sum_{j=0}^{n-1}a_jz^j P τ , Δ NAG ( z ) = ∏ i = 1 n ( z − z i ) = z n + ∑ j = 0 n − 1 a j z j , with roots ∣ z i ∣ < 1 |z_i|<1 ∣ z i ∣ < 1 . By Vieta formulas, a n − k = ( − 1 ) k ∑ ∣ I ∣ = k ∏ i ∈ I z i a_{n-k}=(-1)^k\sum_{|I|=k}\prod_{i\in I}z_i a n − k = ( − 1 ) k ∑ ∣ I ∣ = k ∏ i ∈ I z i . Since ∣ z i ∣ < 1 |z_i|<1 ∣ z i ∣ < 1 for i = 1 , … , n i=1,\ldots,n i = 1 , … , n , we have ∣ a n − k ∣ < ( n k ) |a_{n-k}|<\binom{n}{k} ∣ a n − k ∣ < ( k n ) . In particular, as n = τ + 2 n=\tau+2 n = τ + 2 , for k = τ + 1 k=\tau+1 k = τ + 1 we obtain that the coefficient a 1 a_1 a 1 (which is the coefficient of z z z in P τ , Δ N A G P_{\tau,\Delta}^{\mathrm{NAG}} P τ , Δ NAG ) satisfies ∣ a 1 ∣ < τ + 2 |a_1|<\tau+2 ∣ a 1 ∣ < τ + 2 . If τ = 1 \tau=1 τ = 1 , we have the coefficient 0 < a 1 = β + Δ ( 1 + β ) < 3 0<a_1=\beta+\Delta(1+\beta)<3 0 < a 1 = β + Δ ( 1 + β ) < 3 ; therefore, Δ = O ( 1 ) \Delta=O(1) Δ = O ( 1 ) . If τ ≥ 2 \tau\ge2 τ ≥ 2 , then 0 < a 1 = Δ ( 1 + β ) ≤ τ + 2 0<a_1=\Delta(1+\beta)\le\tau+2 0 < a 1 = Δ ( 1 + β ) ≤ τ + 2 ; hence, we again have Δ = O τ ( 1 ) \Delta=O_\tau(1) Δ = O τ ( 1 ) . Consequently, using (34) ,
δ μ = Δ κ = O τ ( κ − 1 ) . \delta_\mu=\frac{\Delta}{\kappa}=O_\tau(\kappa^{-1}). δ μ = κ Δ = O τ ( κ − 1 ) .
For all sufficiently large κ \kappa κ , Lemma 3.8 applies to the slow mode and yields
1 − ρ < 2 δ μ s ≤ 4 δ μ = O τ ( κ − 1 ) , 1-\rho<\frac{2\delta_\mu}{s}\le4\delta_\mu=O_\tau(\kappa^{-1}), 1 − ρ < s 2 δ μ ≤ 4 δ μ = O τ ( κ − 1 ) ,
because s ≥ 1 2 s\ge\frac{1}{2} s ≥ 2 1 . Thus, tunings with s ≥ 1 2 s\ge\frac{1}{2} s ≥ 2 1 are nonaccelerated. It remains to consider the setting with 0 < s < 1 2 0<s<\frac{1}{2} 0 < s < 2 1 , for which Proposition 3.6 applies.
Case τ ≥ 3 \tau\ge3 τ ≥ 3 . By Proposition 3.6 (c), due to Schur stability, we have
Δ < 2 s τ − 2 + 2 s ≤ 2 s τ − 2 . \Delta<\frac{2s}{\tau-2+2s}\le\frac{2s}{\tau-2}. Δ < τ − 2 + 2 s 2 s ≤ τ − 2 2 s .
First suppose that
s ≥ 16 ( τ − 2 ) κ . s\ge\frac{16}{(\tau-2)\kappa}. s ≥ ( τ − 2 ) κ 16 .
Then
δ μ = Δ κ < 2 s ( τ − 2 ) κ ≤ s 2 8 and 2 τ δ μ s < 4 τ ( τ − 2 ) κ , \delta_\mu=\frac{\Delta}{\kappa}<\frac{2s}{(\tau-2)\kappa}\le\frac{s^2}{8}
\qquad\text{and}\qquad
\frac{2\tau\delta_\mu}{s}<\frac{4\tau}{(\tau-2)\kappa}, δ μ = κ Δ < ( τ − 2 ) κ 2 s ≤ 8 s 2 and s 2 τ δ μ < ( τ − 2 ) κ 4 τ ,
where 4 τ ( τ − 2 ) κ ≤ 1 / 3 \frac{4\tau}{(\tau-2)\kappa}\le1/3 ( τ − 2 ) κ 4 τ ≤ 1/3 for all sufficiently large κ \kappa κ at fixed τ \tau τ . Thus, Lemma 3.8 applies to the setting we consider here, i.e.,
1 − ρ < 2 δ μ s = 2 Δ κ s < 4 ( τ − 2 ) κ . (61) 1-\rho<\frac{2\delta_\mu}{s}=\frac{2\Delta}{\kappa s}<\frac{4}{(\tau-2)\kappa}. \tag*{(61)} 1 − ρ < s 2 δ μ = κ s 2Δ < ( τ − 2 ) κ 4 . (61)
Next, we consider the complementary scenario for s s s , i.e., s < 16 ( τ − 2 ) κ s < \frac{16}{(\tau-2)\kappa} s < ( τ − 2 ) κ 16 . Then s = O τ ( κ − 1 ) s = O_{\tau}(\kappa^{-1}) s = O τ ( κ − 1 ) , and the same fast-mode cap gives
δ μ = Δ κ < 2 s ( τ − 2 ) κ = O τ ( κ − 2 ) . \delta_{\mu} = \frac{\Delta}{\kappa} < \frac{2s}{(\tau-2)\kappa} = O_{\tau}(\kappa^{-2}). δ μ = κ Δ < ( τ − 2 ) κ 2 s = O τ ( κ − 2 ) .
Thus, the hypotheses of Lemma 3.11 hold, and it shows that 1 − ρ ≤ C κ − 1 1-\rho\le C\kappa^{-1} 1 − ρ ≤ C κ − 1 for some C > 0 C>0 C > 0 . From this inequality and (61) , we conclude every Schur-stable tuning with τ ≥ 3 \tau\ge3 τ ≥ 3 satisfies 1 − ρ = O τ ( κ − 1 ) 1-\rho= O_{\tau}(\kappa^{-1}) 1 − ρ = O τ ( κ − 1 ) for any 0 < s < 1 / 2 0<s<1/2 0 < s < 1/2 .
Case τ = 2 \tau=2 τ = 2 . By Proposition 3.6 (b), due to Schur stability, we have Δ < s \Delta<s Δ < s . First, suppose that s ≥ 16 / κ s\ge16/\kappa s ≥ 16/ κ . Then,
δ μ = Δ κ < s κ ≤ s 2 16 ≤ s 2 8 and 2 τ δ μ s = 4 δ μ s < 4 κ . \delta_{\mu} = \frac{\Delta}{\kappa} < \frac{s}{\kappa} \le\frac{s^2}{16} \le\frac{s^2}{8}
\qquad\text{and}\qquad
\frac{2\tau\delta_{\mu}}{s} = \frac{4\delta_{\mu}}{s} < \frac{4}{\kappa}. δ μ = κ Δ < κ s ≤ 16 s 2 ≤ 8 s 2 and s 2 τ δ μ = s 4 δ μ < κ 4 .
where 4 κ ≤ 1 / 3 \frac{4}{\kappa}\le1/3 κ 4 ≤ 1/3 for all sufficiently large κ \kappa κ . Lemma 3.8 therefore gives 1 − ρ ≤ 2 δ μ s = 2 Δ κ s < 2 κ 1-\rho\le\frac{2\delta_{\mu}}{s} = \frac{2\Delta}{\kappa s} < \frac{2}{\kappa} 1 − ρ ≤ s 2 δ μ = κ s 2Δ < κ 2 . On the other hand, if one has s < 16 κ s<\frac{16}{\kappa} s < κ 16 , then s = O ( κ − 1 ) s=O(\kappa^{-1}) s = O ( κ − 1 ) and δ μ = Δ κ < s κ = O ( κ − 2 ) \delta_{\mu}=\frac{\Delta}{\kappa}<\frac{s}{\kappa}=O(\kappa^{-2}) δ μ = κ Δ < κ s = O ( κ − 2 ) ; hence, for this scenario Lemma 3.11 implies that 1 − ρ ≤ C κ − 1 1-\rho\le C\kappa^{-1} 1 − ρ ≤ C κ − 1 for some C > 0 C>0 C > 0 . Thus, we can conclude that every Schur-stable tuning with τ = 2 \tau=2 τ = 2 satisfies 1 − ρ = O ( κ − 1 ) 1-\rho=O(\kappa^{-1}) 1 − ρ = O ( κ − 1 ) .
Case τ = 1 \tau=1 τ = 1 (the κ − 2 / 3 \kappa^{-2/3} κ − 2/3 barrier). By Proposition 3.6 (a), every Schur-stable tuning with 0 < s < 1 2 0<s<\frac{1}{2} 0 < s < 2 1 satisfies
Δ < s β ≤ 2 s , δ μ = Δ κ < s β κ ≤ 2 s κ . (62) \Delta< \frac{\sqrt{s}}{\beta} \le2\sqrt{s},
\qquad
\delta_{\mu} = \frac{\Delta}{\kappa} < \frac{\sqrt{s}}{\beta\kappa} \le\frac{2\sqrt{s}}{\kappa}. \tag*{(62)} Δ < β s ≤ 2 s , δ μ = κ Δ < β κ s ≤ κ 2 s . (62)
For sufficiently large κ \kappa κ , let S κ ∈ ( 0 , 1 2 ) S_{\kappa}\in(0,\frac{1}{2}) S κ ∈ ( 0 , 2 1 ) be a solution of
( 1 − S κ ) κ S κ 3 / 2 = 8. (63) (1-S_{\kappa})\kappa S_{\kappa}^{3/2}=8. \tag*{(63)} ( 1 − S κ ) κ S κ 3/2 = 8. (63)
Note that for all sufficiently large κ \kappa κ , the solution S κ S_{\kappa} S κ is unique because s ↦ ( 1 − s ) s 3 / 2 s\mapsto(1-s)s^{3/2} s ↦ ( 1 − s ) s 3/2 is strictly increasing on ( 0 , 1 2 ) (0,\frac{1}{2}) ( 0 , 2 1 ) ; moreover, as κ → ∞ \kappa\to\infty κ → ∞ , it holds that
S κ = ( 8 / κ ) 2 / 3 ( 1 + o ( 1 ) ) = 4 κ − 2 / 3 ( 1 + o ( 1 ) ) , (64) S_{\kappa}=(8/\kappa)^{2/3}(1+o(1))=4\kappa^{-2/3}(1+o(1)), \tag*{(64)} S κ = ( 8/ κ ) 2/3 ( 1 + o ( 1 )) = 4 κ − 2/3 ( 1 + o ( 1 )) , (64)
where o ( 1 ) o(1) o ( 1 ) means a quantity tending to zero as κ → ∞ \kappa\to\infty κ → ∞ . Note that by (62) , every Schur-stable tuning satisfies Δ < Δ max : = s / β \Delta<\Delta_{\max}:=\sqrt{s}/\beta Δ < Δ m a x := s / β . Next, we will consider the cases s ≥ S κ s\ge S_{\kappa} s ≥ S κ and s < S κ s<S_{\kappa} s < S κ separately.
Case s ≥ S κ s\ge S_{\kappa} s ≥ S κ : For s s s such that 1 2 ≥ s ≥ S κ \frac{1}{2}\ge s\ge S_{\kappa} 2 1 ≥ s ≥ S κ , we can argue that δ μ < Δ max / κ ≤ s 2 / 8 \delta_{\mu}<\Delta_{\max}/\kappa\le s^2/8 δ μ < Δ m a x / κ ≤ s 2 /8 . Indeed, for s = S κ s=S_{\kappa} s = S κ , we have exactly Δ max / κ = S κ 2 / 8 \Delta_{\max}/\kappa=S_{\kappa}^2/8 Δ m a x / κ = S κ 2 /8 . Therefore, using the fact that ( 1 − s ) s 3 / 2 (1-s)s^{3/2} ( 1 − s ) s 3/2 is increasing on ( 0 , 1 2 ) (0,\frac{1}{2}) ( 0 , 2 1 ) , we get
δ μ ≤ Δ max ( 1 − s ) κ = s ( 1 − s ) β κ ≤ s 2 8 . \delta_{\mu}\le\frac{\Delta_{\max}}{(1-s)\kappa}=\frac{\sqrt{s}}{(1-s)\beta\kappa}\le\frac{s^2}{8}. δ μ ≤ ( 1 − s ) κ Δ m a x = ( 1 − s ) β κ s ≤ 8 s 2 .
Moreover, we also have 2 δ μ / s ≤ s / 4 ≤ 1 / 3 2\delta_{\mu}/s\le s/4\le1/3 2 δ μ / s ≤ s /4 ≤ 1/3 as s < 1 / 2 s<1/2 s < 1/2 . Thus, Lemma 3.8 is applicable here, and together with Δ < Δ max \Delta<\Delta_{\max} Δ < Δ m a x , Lemma 3.8 implies that
1 − ρ < 2 δ μ s = 2 Δ κ s ≤ G κ ( s ) : = 2 ( 1 − s ) s κ . 1-\rho<\frac{2\delta_{\mu}}{s}=\frac{2\Delta}{\kappa s}\le G_{\kappa}(s):=\frac{2}{(1-s)\sqrt{s}\kappa}. 1 − ρ < s 2 δ μ = κ s 2Δ ≤ G κ ( s ) := ( 1 − s ) s κ 2 .
The function G κ ( s ) G_{\kappa}(s) G κ ( s ) is decreasing on ( 0 , 1 / 3 ) (0,1/3) ( 0 , 1/3 ) and increasing on ( 1 / 3 , 1 / 2 ] (1/3,1/2] ( 1/3 , 1/2 ] . Hence,
sup s ∈ [ S κ , 1 / 2 ] G κ ( s ) = max { G κ ( S κ ) , G κ ( 1 / 2 ) } . \sup_{s\in[S_{\kappa},1/2]}G_{\kappa}(s)=\max\{G_{\kappa}(S_{\kappa}),G_{\kappa}(1/2)\}. s ∈ [ S κ , 1/2 ] sup G κ ( s ) = max { G κ ( S κ ) , G κ ( 1/2 )} .
From (63) , we have G κ ( S κ ) = S κ 4 = ( 1 + o ( 1 ) ) κ − 2 / 3 G_{\kappa}(S_{\kappa}) = \frac{S_{\kappa}}{4} = (1+o(1))\kappa^{-2/3} G κ ( S κ ) = 4 S κ = ( 1 + o ( 1 )) κ − 2/3 , whereas G κ ( 1 / 2 ) = 4 2 κ = o ( κ − 2 / 3 ) G_{\kappa}(1/2) = \frac{4\sqrt{2}}{\kappa} = o(\kappa^{-2/3}) G κ ( 1/2 ) = κ 4 2 = o ( κ − 2/3 ) . Therefore, for all sufficiently large κ \kappa κ ,
1 − ρ ≤ sup s ∈ [ S κ , 1 / 2 ] G κ ( s ) = ( 1 + o ( 1 ) ) κ − 2 / 3 . 1-\rho\le\sup_{s\in[S_{\kappa},1/2]}G_{\kappa}(s)=(1+o(1))\kappa^{-2/3}. 1 − ρ ≤ s ∈ [ S κ , 1/2 ] sup G κ ( s ) = ( 1 + o ( 1 )) κ − 2/3 .
Case s < S κ s<S_{\kappa} s < S κ : Since every admissible tuning satisfies Δ < Δ max \Delta<\Delta_{\max} Δ < Δ m a x , for 0 < s < S κ 0<s<S_{\kappa} 0 < s < S κ , we have
δ μ = Δ κ < Δ max κ = s ( 1 − s ) κ ≤ 2 S κ κ = O ( κ − 4 / 3 ) , (65) \delta_{\mu}=\frac{\Delta}{\kappa}<\frac{\Delta_{\max}}{\kappa}=\frac{\sqrt{s}}{(1-s)\kappa}\le\frac{2\sqrt{S_{\kappa}}}{\kappa}=O(\kappa^{-4/3}), \tag*{(65)} δ μ = κ Δ < κ Δ m a x = ( 1 − s ) κ s ≤ κ 2 S κ = O ( κ − 4/3 ) , (65)
where we used (62) . Thus, for all sufficiently large κ \kappa κ , we have δ μ ≤ δ 0 \delta_{\mu}\le\delta_{0} δ μ ≤ δ 0 uniformly for 0 < s < S κ 0<s<S_{\kappa} 0 < s < S κ and over all admissible tunings. Lemma 3.9 therefore gives
1 − ρ ≤ s 1 + 1 − s + O ( δ μ 2 ) , 1-\rho\le\frac{s}{1+\sqrt{1-s}}+O(\delta_{\mu}^{2}), 1 − ρ ≤ 1 + 1 − s s + O ( δ μ 2 ) ,
where the O ( δ μ 2 ) O(\delta_{\mu}^{2}) O ( δ μ 2 ) term is uniform in s s s and over all admissible tunings. The first term is increasing in s s s , so over s < S κ s<S_{\kappa} s < S κ ,
s 1 + 1 − s ≤ S κ 1 + 1 − S κ = S κ 2 ( 1 + o ( 1 ) ) . \frac{s}{1+\sqrt{1-s}}\le\frac{S_{\kappa}}{1+\sqrt{1-S_{\kappa}}}=\frac{S_{\kappa}}{2}(1+o(1)). 1 + 1 − s s ≤ 1 + 1 − S κ S κ = 2 S κ ( 1 + o ( 1 )) .
Moreover, (65) implies that δ μ 2 = O ( κ − 8 / 3 ) = o ( κ − 2 / 3 ) \delta_{\mu}^{2}=O(\kappa^{-8/3})=o(\kappa^{-2/3}) δ μ 2 = O ( κ − 8/3 ) = o ( κ − 2/3 ) . Thus,
1 − ρ ≤ S κ 2 ( 1 + o ( 1 ) ) = 2 ( 1 + o ( 1 ) ) κ − 2 / 3 . 1-\rho\le\frac{S_{\kappa}}{2}(1+o(1))=2(1+o(1))\kappa^{-2/3}. 1 − ρ ≤ 2 S κ ( 1 + o ( 1 )) = 2 ( 1 + o ( 1 )) κ − 2/3 .
Combining the two subcases, every Schur-stable tuning with τ = 1 \tau=1 τ = 1 satisfies
1 − ρ ≤ ( 2 + o ( 1 ) ) κ − 2 / 3 . 1-\rho\le(2+o(1))\kappa^{-2/3}. 1 − ρ ≤ ( 2 + o ( 1 )) κ − 2/3 .
Finally, combining the three delay cases (τ = 1 \tau=1 τ = 1 , τ = 2 \tau=2 τ = 2 , τ ≥ 3 \tau\ge3 τ ≥ 3 ) and then optimizing over Schur-stable tunings gives (60) . In every fixed-delay case, 1 − ρ N A G ⋆ = o ( κ − 1 / 2 ) 1-\rho_{\mathrm{NAG}}^{\star}=o(\kappa^{-1/2}) 1 − ρ NAG ⋆ = o ( κ − 1/2 ) , so delayed Nesterov does not retain κ \sqrt{\kappa} κ -acceleration.
The previous result provides an upper bound for the spectral gap 1 − ρ N A G ⋆ ≤ ( 2 + o ( 1 ) ) κ − 2 / 3 1-\rho_{\mathrm{NAG}}^{\star}\le(2+o(1))\kappa^{-2/3} 1 − ρ NAG ⋆ ≤ ( 2 + o ( 1 )) κ − 2/3 for τ = 1 \tau=1 τ = 1 , a natural question is whether this bound is tight or equivalently whether an Θ ( κ − 2 / 3 ) \Theta(\kappa^{-2/3}) Θ ( κ − 2/3 ) spectral gap is achievable by NAG. The following result shows that this is indeed the case. This shows that NAG is still faster than gradient descent for delay τ = 1 \tau=1 τ = 1 when κ \kappa κ is sufficiently large but it is still not accelerated in the sense that it cannot achieve the spectral gap of Θ ( κ − 1 / 2 ) \Theta(\kappa^{-1/2}) Θ ( κ − 1/2 ) .
Lemma 3.13 (Matching Θ ( κ − 2 / 3 ) \Theta(\kappa^{-2/3}) Θ ( κ − 2/3 ) spectral gap for NAG when τ = 1 \tau=1 τ = 1 ). Let τ = 1 \tau=1 τ = 1 and κ ≥ 64 \kappa\ge64 κ ≥ 64 . Choose
s = κ − 2 / 3 , β = 1 − s , Δ = α L = s 2 = 1 2 κ − 1 / 3 . (66) s=\kappa^{-2/3},\qquad\beta=1-s,\qquad\Delta=\alpha L=\frac{\sqrt{s}}{2}=\frac{1}{2}\kappa^{-1/3}. \tag*{(66)} s = κ − 2/3 , β = 1 − s , Δ = α L = 2 s = 2 1 κ − 1/3 . (66) Then δ μ = Δ / κ = s 2 / 2 \delta_{\mu}=\Delta/\kappa=s^2/2 δ μ = Δ/ κ = s 2 /2 , and for every δ ∈ [ δ μ , Δ ] \delta\in[\delta_{\mu},\Delta] δ ∈ [ δ μ , Δ ] , every root z z z of P 1 , δ N A G P^{\mathrm{NAG}}_{1,\delta} P 1 , δ NAG satisfies
∣ z ∣ ≤ 1 − 7 22 s = 1 − 7 22 κ − 2 / 3 . (67) |z|\le1-\frac{7}{22}s=1-\frac{7}{22}\kappa^{-2/3}. \tag*{(67)} ∣ z ∣ ≤ 1 − 22 7 s = 1 − 22 7 κ − 2/3 . (67) Consequently, the tuning (66) is Schur stable and
ρ ( α , β ) ≤ 1 − 7 22 κ − 2 / 3 , 1 − ρ N A G ⋆ ( 1 , κ ) ≥ 7 22 κ − 2 / 3 . (68) \rho(\alpha,\beta)\le1-\frac{7}{22}\kappa^{-2/3},\qquad1-\rho_{\mathrm{NAG}}^{\star}(1,\kappa)\ge\frac{7}{22}\kappa^{-2/3}. \tag*{(68)} ρ ( α , β ) ≤ 1 − 22 7 κ − 2/3 , 1 − ρ NAG ⋆ ( 1 , κ ) ≥ 22 7 κ − 2/3 . (68) In particular, together with Theorem 3.12 ,
1 − ρ N A G ⋆ ( 1 , κ ) = Θ ( κ − 2 / 3 ) as κ → ∞ . 1-\rho_{\mathrm{NAG}}^{\star}(1,\kappa)=\Theta(\kappa^{-2/3})\qquad\text{as }\kappa\to\infty. 1 − ρ NAG ⋆ ( 1 , κ ) = Θ ( κ − 2/3 ) as κ → ∞.
Proof. Fix δ ∈ [ δ μ , Δ ] \delta\in[\delta_\mu,\Delta] δ ∈ [ δ μ , Δ ] . Since κ ≥ 64 \kappa\ge64 κ ≥ 64 , we have s ≤ 1 / 16 s \le1/16 s ≤ 1/16 , and therefore
β = 1 − s ≥ 15 16 , Δ = s 2 ≤ 1 8 . \beta= 1-s \ge\frac{15}{16}, \qquad\Delta= \frac{\sqrt{s}}{2} \le\frac{1}{8}. β = 1 − s ≥ 16 15 , Δ = 2 s ≤ 8 1 .
For τ = 1 \tau=1 τ = 1 , the NAG characteristic polynomial (33) is
P 1 , δ N A G ( z ) = z ( z − 1 ) ( z − β ) + δ [ ( 1 + β ) z − β ] . P^{\mathrm{NAG}}_{1,\delta}(z)=z(z-1)(z-\beta)+\delta\left[(1+\beta)z-\beta\right]. P 1 , δ NAG ( z ) = z ( z − 1 ) ( z − β ) + δ [ ( 1 + β ) z − β ] .
Evaluating it at 0 0 0 and at δ \delta δ gives
P 1 , δ N A G ( 0 ) = − δ β < 0 , P 1 , δ N A G ( δ ) = δ 3 > 0. P^{\mathrm{NAG}}_{1,\delta}(0)=-\delta\beta<0,\qquad P^{\mathrm{NAG}}_{1,\delta}(\delta)=\delta^3>0. P 1 , δ NAG ( 0 ) = − δ β < 0 , P 1 , δ NAG ( δ ) = δ 3 > 0.
Hence, by the intermediate value theorem, there exists a real root x δ ∈ ( 0 , δ ) x_\delta\in(0,\delta) x δ ∈ ( 0 , δ ) . In particular, x δ < δ ≤ Δ ≤ 1 / 8 x_\delta<\delta\le\Delta\le1/8 x δ < δ ≤ Δ ≤ 1/8 . Set
d δ : = β − ( 1 + β ) x δ = 1 − s − ( 2 − s ) x δ . d_\delta:=\beta-(1+\beta)x_\delta=1-s-(2-s)x_\delta. d δ := β − ( 1 + β ) x δ = 1 − s − ( 2 − s ) x δ .
Using s ≤ 1 / 16 s\le1/16 s ≤ 1/16 and x δ < 1 / 8 x_\delta<1/8 x δ < 1/8 ,
d δ ≥ 1 − 1 16 − 2 ⋅ 1 8 = 11 16 > 1 2 . (69) d_\delta\ge1-\frac{1}{16}-2\cdot\frac{1}{8}=\frac{11}{16}>\frac{1}{2}. \tag*{(69)} d δ ≥ 1 − 16 1 − 2 ⋅ 8 1 = 16 11 > 2 1 . (69)
Since P 1 , δ N A G ( x δ ) = 0 P^{\mathrm{NAG}}_{1,\delta}(x_\delta)=0 P 1 , δ NAG ( x δ ) = 0 ,
x δ ( 1 − x δ ) ( β − x δ ) = δ [ β − ( 1 + β ) x δ ] = δ d δ . (70) x_\delta(1-x_\delta)(\beta-x_\delta)=\delta\left[\beta-(1+\beta)x_\delta\right]=\delta d_\delta. \tag*{(70)} x δ ( 1 − x δ ) ( β − x δ ) = δ [ β − ( 1 + β ) x δ ] = δ d δ . (70)
Moreover, expanding the product ( 1 − x δ ) ( β − x δ ) = β − ( 1 + β ) x δ + x δ 2 = d δ + x δ 2 (1-x_\delta)(\beta-x_\delta)=\beta-(1+\beta)x_\delta+x_\delta^2=d_\delta+x_\delta^2 ( 1 − x δ ) ( β − x δ ) = β − ( 1 + β ) x δ + x δ 2 = d δ + x δ 2 , and inserting this equality back into (70) gives
δ = x δ ( 1 + x δ 2 d δ ) . (71) \delta=x_\delta\left(1+\frac{x_\delta^2}{d_\delta}\right). \tag*{(71)} δ = x δ ( 1 + d δ x δ 2 ) . (71)
Because d δ > 1 / 2 d_\delta>1/2 d δ > 1/2 and x δ < δ x_\delta<\delta x δ < δ , (71) yields δ < x δ ( 1 + 2 δ 2 ) \delta<x_\delta(1+2\delta^2) δ < x δ ( 1 + 2 δ 2 ) , and hence, using δ ≤ Δ = s / 2 \delta\le\Delta=\sqrt{s}/2 δ ≤ Δ = s /2 and δ ≥ δ μ = s 2 / 2 \delta\ge\delta_\mu=s^2/2 δ ≥ δ μ = s 2 /2 ,
x δ > δ 1 + 2 δ 2 ≥ δ μ 1 + 2 Δ 2 = s 2 2 + s . (72) x_\delta>\frac{\delta}{1+2\delta^2}\ge\frac{\delta_\mu}{1+2\Delta^2}=\frac{s^2}{2+s}. \tag*{(72)} x δ > 1 + 2 δ 2 δ ≥ 1 + 2 Δ 2 δ μ = 2 + s s 2 . (72)
We now factor out this real root. Since
P 1 , δ N A G ( z ) = z 3 − ( 1 + β ) z 2 + [ β + δ ( 1 + β ) ] z − δ β , P^{\mathrm{NAG}}_{1,\delta}(z)=z^3-(1+\beta)z^2+\left[\beta+\delta(1+\beta)\right]z-\delta\beta, P 1 , δ NAG ( z ) = z 3 − ( 1 + β ) z 2 + [ β + δ ( 1 + β ) ] z − δ β ,
Vieta’s formulas give the sum and product of the remaining two roots as 1 + β − x δ 1+\beta-x_\delta 1 + β − x δ and δ β / x δ \delta\beta/x_\delta δ β / x δ , respectively; hence we have
P 1 , δ N A G ( z ) = ( z − x δ ) [ z 2 − ( 1 + β − x δ ) z + δ β x δ ] . (73) P^{\mathrm{NAG}}_{1,\delta}(z)=(z-x_\delta)\left[z^2-(1+\beta-x_\delta)z+\frac{\delta\beta}{x_\delta}\right]. \tag*{(73)} P 1 , δ NAG ( z ) = ( z − x δ ) [ z 2 − ( 1 + β − x δ ) z + x δ δ β ] . (73)
We next show that the quadratic factor has a complex-conjugate pair of roots. By (71) , its constant term is
δ β x δ = 1 − s + ( 1 − s ) x δ 2 d δ . (74) \frac{\delta\beta}{x_\delta}=1-s+\frac{(1-s)x_\delta^2}{d_\delta}. \tag*{(74)} x δ δ β = 1 − s + d δ ( 1 − s ) x δ 2 . (74)
Therefore its discriminant equals
( 2 − s − x δ ) 2 − 4 ( 1 − s + ( 1 − s ) x δ 2 d δ ) = ( s + x δ ) 2 − 4 x δ − 4 ( 1 − s ) x δ 2 d δ < ( s + x δ ) 2 − 4 x δ . (75) (2-s-x_\delta)^2-4\left(1-s+\frac{(1-s)x_\delta^2}{d_\delta}\right)=(s+x_\delta)^2-4x_\delta-\frac{4(1-s)x_\delta^2}{d_\delta}<(s+x_\delta)^2-4x_\delta. \tag*{(75)} ( 2 − s − x δ ) 2 − 4 ( 1 − s + d δ ( 1 − s ) x δ 2 ) = ( s + x δ ) 2 − 4 x δ − d δ 4 ( 1 − s ) x δ 2 < ( s + x δ ) 2 − 4 x δ . (75)
The two zeros, as a function of x x x , of ϕ ( x ) : = ( s + x ) 2 − 4 x \phi(x) := (s+x)^2-4x ϕ ( x ) := ( s + x ) 2 − 4 x are
x − = ( 1 − 1 − s ) 2 , x + = ( 1 + 1 − s ) 2 . x_-=(1-\sqrt{1-s})^2,\qquad x_+=(1+\sqrt{1-s})^2. x − = ( 1 − 1 − s ) 2 , x + = ( 1 + 1 − s ) 2 .
Since 1 − s ≥ 1 − s \sqrt{1-s}\ge1-s 1 − s ≥ 1 − s ,
x − = s 2 ( 1 + 1 − s ) 2 ≤ s 2 ( 2 − s ) 2 . x_-=\frac{s^2}{(1+\sqrt{1-s})^2}\le\frac{s^2}{(2-s)^2}. x − = ( 1 + 1 − s ) 2 s 2 ≤ ( 2 − s ) 2 s 2 .
For 0 < s ≤ 1 / 16 0<s\le1/16 0 < s ≤ 1/16 , we have ( 2 − s ) 2 − ( 2 + s ) = 2 − 5 s + s 2 > 0 (2-s)^2-(2+s)=2-5s+s^2>0 ( 2 − s ) 2 − ( 2 + s ) = 2 − 5 s + s 2 > 0 , so x − < s 2 2 + s < x δ x_-<\frac{s^2}{2+s}<x_\delta x − < 2 + s s 2 < x δ by (72) . On the other hand, x δ < 1 / 8 < 1 < x + x_\delta<1/8<1<x_+ x δ < 1/8 < 1 < x + . Therefore, x δ ∈ ( x − , x + ) x_\delta\in(x_-,x_+) x δ ∈ ( x − , x + ) . Moreover, the leading coefficient of the quadratic ϕ ( x ) \phi(x) ϕ ( x ) is positive; therefore, ϕ ( x ) < 0 \phi(x)<0 ϕ ( x ) < 0 for x ∈ ( x − , x + ) x\in(x_-,x_+) x ∈ ( x − , x + ) . Thus, ϕ ( x δ ) = ( s + x δ ) 2 − 4 x δ < 0 \phi(x_\delta)=(s+x_\delta)^2-4x_\delta<0 ϕ ( x δ ) = ( s + x δ ) 2 − 4 x δ < 0 , and (75) shows that the quadratic factor in (73) indeed has a complex-conjugate pair. Let z 1 , z 2 z_1,z_2 z 1 , z 2 denote this pair. By (74) , Vieta’s formula applied to this factor and (69) give
∣ z 1 ∣ 2 = ∣ z 2 ∣ 2 = z 1 z 2 = 1 − s + ( 1 − s ) x δ 2 d δ < 1 − s + Δ 2 11 / 16 = 1 − 7 11 s . |z_1|^2=|z_2|^2=z_1z_2=1-s+\frac{(1-s)x_\delta^2}{d_\delta}<1-s+\frac{\Delta^2}{11/16}=1-\frac{7}{11}s. ∣ z 1 ∣ 2 = ∣ z 2 ∣ 2 = z 1 z 2 = 1 − s + d δ ( 1 − s ) x δ 2 < 1 − s + 11/16 Δ 2 = 1 − 11 7 s .
where we used 1 − s < 1 1-s<1 1 − s < 1 and x δ < δ ≤ Δ = s / 2 x_\delta<\delta\le\Delta=\sqrt{s}/2 x δ < δ ≤ Δ = s /2 . Using 1 − u ≤ 1 − u / 2 \sqrt{1-u}\le1-u/2 1 − u ≤ 1 − u /2 for u ∈ [ 0 , 1 ] u\in[0,1] u ∈ [ 0 , 1 ] ,
∣ z 1 ∣ = ∣ z 2 ∣ < 1 − 7 11 s ≤ 1 − 7 22 s . |z_1|=|z_2|<\sqrt{1-\frac{7}{11}s}\le1-\frac{7}{22}s. ∣ z 1 ∣ = ∣ z 2 ∣ < 1 − 11 7 s ≤ 1 − 22 7 s .
The remaining root satisfies x δ < 1 / 8 < 1 − 7 s / 22 x_\delta<1/8<1-7s/22 x δ < 1/8 < 1 − 7 s /22 for s ≤ 1 / 16 s\le1/16 s ≤ 1/16 . All the preceding estimates are uniform over δ ∈ [ δ μ , Δ ] \delta\in[\delta_\mu,\Delta] δ ∈ [ δ μ , Δ ] , since they use only the bounds δ ≤ Δ \delta\le\Delta δ ≤ Δ , x δ < δ ≤ Δ x_\delta<\delta\le\Delta x δ < δ ≤ Δ , and d δ ≥ 11 / 16 d_\delta\ge11/16 d δ ≥ 11/16 . Hence, every root of P 1 , δ N A G P^{\mathrm{NAG}}_{1,\delta} P 1 , δ NAG satisfies (67) , uniformly over δ ∈ [ δ μ , Δ ] \delta\in[\delta_\mu,\Delta] δ ∈ [ δ μ , Δ ] . Since [ δ μ , Δ ] [\delta_\mu,\Delta] [ δ μ , Δ ] is precisely the range of δ = α λ \delta=\alpha\lambda δ = α λ for λ ∈ [ μ , L ] \lambda\in[\mu,L] λ ∈ [ μ , L ] , the definition of ρ ( α , β ) \rho(\alpha,\beta) ρ ( α , β ) gives the first inequality in (68) . Taking the infimum over all Schur-stable NAG tunings then yields 1 − ρ N A G ⋆ ( 1 , κ ) ≥ 7 22 κ − 2 / 3 1-\rho^\star_{\mathrm{NAG}}(1,\kappa)\ge\frac{7}{22}\kappa^{-2/3} 1 − ρ NAG ⋆ ( 1 , κ ) ≥ 22 7 κ − 2/3 . Combining this bound with Theorem 3.12 proves 1 − ρ N A G ⋆ ( 1 , κ ) = Θ ( κ − 2 / 3 ) 1-\rho^\star_{\mathrm{NAG}}(1,\kappa)=\Theta(\kappa^{-2/3}) 1 − ρ NAG ⋆ ( 1 , κ ) = Θ ( κ − 2/3 ) .
Heavy ball is non-accelerated under delay
We now treat the heavy-ball (HB) method, i.e., GMM with ν = 0 \nu=0 ν = 0 , which also belongs to the family with the characteristic polynomial in (24) . The conclusion is even simpler than for the Nesterov’s method; indeed, we show that for every fixed delay, every Schur-stable heavy-ball tuning has spectral gap at most of order κ − 1 \kappa^{-1} κ − 1 . Thus, the heavy-ball method also fails to retain the κ \sqrt{\kappa} κ accelerated scaling.
The factorization, and the contrast with Polyak. Recall from (35) that the characteristic polynomial of the delayed HB admits the factorization
P τ , δ H B ( z ) = z τ ( z − 1 ) ( z − β ) + δ z = z K τ , δ H B ( z ) , where K τ , δ H B ( z ) = z τ − 1 ( z − 1 ) ( z − β ) + δ . (76) P^{\mathrm{HB}}_{\tau,\delta}(z)=z^\tau(z-1)(z-\beta)+\delta z=zK^{\mathrm{HB}}_{\tau,\delta}(z),\qquad\text{where}\qquad K^{\mathrm{HB}}_{\tau,\delta}(z)=z^{\tau-1}(z-1)(z-\beta)+\delta. \tag*{(76)} P τ , δ HB ( z ) = z τ ( z − 1 ) ( z − β ) + δ z = z K τ , δ HB ( z ) , where K τ , δ HB ( z ) = z τ − 1 ( z − 1 ) ( z − β ) + δ . (76)
The root z = 0 z=0 z = 0 is irrelevant for the convergence rate; the nontrivial rate is governed by the roots of K τ , δ H B K^{\mathrm{HB}}_{\tau,\delta} K τ , δ HB as δ \delta δ ranges over [ δ μ , Δ ] [\delta_\mu,\Delta] [ δ μ , Δ ] .
Now consider the undelayed case τ = 0 \tau=0 τ = 0 , where P 0 , δ H B ( z ) = z 2 − ( 1 + β − δ ) z + β P^{\mathrm{HB}}_{0,\delta}(z)=z^2-(1+\beta-\delta)z+\beta P 0 , δ HB ( z ) = z 2 − ( 1 + β − δ ) z + β ; thus, the product of the two roots is the constant β > 0 \beta>0 β > 0 . When the pair is complex, the modulus of each root is exactly β \sqrt{\beta} β , independent of δ \delta δ ; hence, the Polyak choice keeps both root moduli equal to β \sqrt{\beta} β throughout the spectral interval and gives the accelerated quadratic rate. On the other hand, under positive delay, already at τ = 1 \tau=1 τ = 1 , the product of the nontrivial pair becomes β + δ \beta+\delta β + δ , which varies across the spectrum as λ \lambda λ varies in [ μ , L ] [\mu,L] [ μ , L ] .
Lemma 3.14 (Lower bound on ρ \rho ρ for HB). Fix τ ≥ 1 \tau\ge1 τ ≥ 1 . Suppose 0 < δ μ ≤ s 2 8 0 < \delta_\mu\le\frac{s^2}{8} 0 < δ μ ≤ 8 s 2 and 2 ( τ − 1 ) δ μ s ≤ 1 3 \frac{2(\tau-1)\delta_\mu}{s} \le\frac{1}{3} s 2 ( τ − 1 ) δ μ ≤ 3 1 . Then K τ , δ μ H B K^{\mathrm{HB}}_{\tau,\delta_\mu} K τ , δ μ HB has a real root in [ 1 − 2 δ μ s , 1 ) \left[1-\frac{2\delta_\mu}{s},1\right) [ 1 − s 2 δ μ , 1 ) . Consequently ρ ≥ 1 − 2 δ μ s \rho\ge1-\frac{2\delta_\mu}{s} ρ ≥ 1 − s 2 δ μ , and hence 1 − ρ ≤ 2 δ μ s 1-\rho\le\frac{2\delta_\mu}{s} 1 − ρ ≤ s 2 δ μ . *
Proof. Set z 0 : = 1 − 2 δ μ s z_0 := 1-\frac{2\delta_\mu}{s} z 0 := 1 − s 2 δ μ . The condition δ μ ≤ s 2 / 8 \delta_\mu\le s^2/8 δ μ ≤ s 2 /8 gives 2 δ μ s ≤ s 4 \frac{2\delta_\mu}{s} \le\frac{s}{4} s 2 δ μ ≤ 4 s ; hence,
z 0 ∈ [ 1 − s / 4 , 1 ) ⊂ ( β , 1 ) , z 0 − β = s − 2 δ μ s ≥ 3 s 4 . z_0 \in[1-s/4,1) \subset(\beta,1), \qquad z_0-\beta=s-\frac{2\delta_\mu}{s}\ge\frac{3s}{4}. z 0 ∈ [ 1 − s /4 , 1 ) ⊂ ( β , 1 ) , z 0 − β = s − s 2 δ μ ≥ 4 3 s . Moreover K τ , δ μ H B ( 1 ) = δ μ > 0 K^{\mathrm{HB}}_{\tau,\delta_\mu}(1)=\delta_\mu>0 K τ , δ μ HB ( 1 ) = δ μ > 0 . Therefore, at z 0 z_0 z 0 , we have
K τ , δ μ H B ( z 0 ) = z 0 τ − 1 ( z 0 − 1 ) ( z 0 − β ) + δ μ ≤ − z 0 τ − 1 2 δ μ s 3 s 4 + δ μ = δ μ ( 1 − 3 2 z 0 τ − 1 ) . K^{\mathrm{HB}}_{\tau,\delta_\mu}(z_0)=z_0^{\tau-1}(z_0-1)(z_0-\beta)+\delta_\mu\le-z_0^{\tau-1}\frac{2\delta_\mu}{s}\frac{3s}{4}+\delta_\mu=\delta_\mu\left(1-\frac{3}{2}z_0^{\tau-1}\right). K τ , δ μ HB ( z 0 ) = z 0 τ − 1 ( z 0 − 1 ) ( z 0 − β ) + δ μ ≤ − z 0 τ − 1 s 2 δ μ 4 3 s + δ μ = δ μ ( 1 − 2 3 z 0 τ − 1 ) . Note that the first hypothesis implies 2 δ μ s ≤ s 4 < 1 4 < 1 \frac{2\delta_\mu}{s}\le\frac{s}{4}<\frac{1}{4}<1 s 2 δ μ ≤ 4 s < 4 1 < 1 , where we have used 0 < s < 1 0<s<1 0 < s < 1 . Thus, for τ ≥ 2 \tau\ge2 τ ≥ 2 , Bernoulli’s inequality gives z 0 τ − 1 = ( 1 − 2 δ μ s ) τ − 1 ≥ 1 − ( τ − 1 ) 2 δ μ s ≥ 2 3 z_0^{\tau-1}=\left(1-\frac{2\delta_\mu}{s}\right)^{\tau-1}\ge1-(\tau-1)\frac{2\delta_\mu}{s}\ge\frac{2}{3} z 0 τ − 1 = ( 1 − s 2 δ μ ) τ − 1 ≥ 1 − ( τ − 1 ) s 2 δ μ ≥ 3 2 , where the last inequality follows from the second hypothesis. For τ = 1 \tau=1 τ = 1 , the same bound holds trivially since z 0 τ − 1 = 1 z_0^{\tau-1}=1 z 0 τ − 1 = 1 . Therefore, K τ , δ μ H B ( z 0 ) ≤ δ μ ( 1 − 3 2 z 0 τ − 1 ) ≤ 0 < K τ , δ μ ( 1 ) K^{\mathrm{HB}}_{\tau,\delta_\mu}(z_0)\le\delta_\mu\left(1-\frac{3}{2}z_0^{\tau-1}\right)\le0<K_{\tau,\delta_\mu}(1) K τ , δ μ HB ( z 0 ) ≤ δ μ ( 1 − 2 3 z 0 τ − 1 ) ≤ 0 < K τ , δ μ ( 1 ) . By the intermediate value theorem, K τ , δ μ K_{\tau,\delta_\mu} K τ , δ μ has a real root in [ z 0 , 1 ) [z_0,1) [ z 0 , 1 ) , and the displayed spectral-radius bound follows.
At τ = 1 \tau=1 τ = 1 , K 1 , δ H B ( z ) K^{\mathrm{HB}}_{1,\delta}(z) K 1 , δ HB ( z ) in (76) is a quadratic, i.e.,
K 1 , δ H B ( z ) = z 2 − ( 1 + β ) z + ( β + δ ) . (77) K^{\mathrm{HB}}_{1,\delta}(z)=z^2-(1+\beta)z+(\beta+\delta). \tag*{(77)} K 1 , δ HB ( z ) = z 2 − ( 1 + β ) z + ( β + δ ) . (77)
The following proposition gives the explicit bound 1 − ρ < 2 / κ 1-\rho<2/\kappa 1 − ρ < 2/ κ for every Schur-stable heavy-ball tuning with τ = 1 \tau=1 τ = 1 .
Proposition 3.15 (τ = 1 \tau=1 τ = 1 for heavy ball). For every Schur-stable heavy-ball tuning with τ = 1 \tau=1 τ = 1 , 1 − ρ < 2 κ 1-\rho<\frac{2}{\kappa} 1 − ρ < κ 2 . Consequently, 1 − ρ H B ⋆ ( 1 , κ ) < 2 κ 1-\rho^\star_{\mathrm{HB}}(1,\kappa)<\frac{2}{\kappa} 1 − ρ HB ⋆ ( 1 , κ ) < κ 2 .
Proof. Let z 1 , z 2 z_1,z_2 z 1 , z 2 be the two roots of (77) . We will use three elementary inequalities. First, since z 1 + z 2 = 1 + β = 2 − s z_1+z_2=1+\beta=2-s z 1 + z 2 = 1 + β = 2 − s , the triangle inequality gives
ρ ≥ max { ∣ z 1 ∣ , ∣ z 2 ∣ } ≥ ∣ z 1 + z 2 ∣ 2 = 1 − s 2 . (78) \rho\ge\max\{|z_1|,|z_2|\}\ge\frac{|z_1+z_2|}{2}=1-\frac{s}{2}. \tag*{(78)} ρ ≥ max { ∣ z 1 ∣ , ∣ z 2 ∣ } ≥ 2 ∣ z 1 + z 2 ∣ = 1 − 2 s . (78)
Thus, 1 − ρ ≤ s 2 1-\rho\le\frac{s}{2} 1 − ρ ≤ 2 s . Second, on the fast mode δ = Δ \delta=\Delta δ = Δ , the root product is z 1 z 2 = β + Δ = 1 − ( s − Δ ) z_1z_2=\beta+\Delta=1-(s-\Delta) z 1 z 2 = β + Δ = 1 − ( s − Δ ) . Schur stability implies that β + Δ < 1 \beta+\Delta<1 β + Δ < 1 ; hence, Δ < s \Delta<s Δ < s . Since max { ∣ z 1 ∣ , ∣ z 2 ∣ } ≥ ∣ z 1 z 2 ∣ = β + Δ \max\{|z_1|,|z_2|\}\ge\sqrt{|z_1z_2|}=\sqrt{\beta+\Delta} max { ∣ z 1 ∣ , ∣ z 2 ∣ } ≥ ∣ z 1 z 2 ∣ = β + Δ , we obtain
1 − ρ ≤ 1 − 1 − ( s − Δ ) ≤ s − Δ . (79) 1-\rho\le1-\sqrt{1-(s-\Delta)}\le s-\Delta. \tag*{(79)} 1 − ρ ≤ 1 − 1 − ( s − Δ ) ≤ s − Δ. (79)
Third, on the slow mode δ = δ μ \delta=\delta_\mu δ = δ μ , if δ μ ≤ s 2 / 4 \delta_\mu\le s^2/4 δ μ ≤ s 2 /4 , the two roots are real and the larger one is
z + = 1 2 [ ( 2 − s ) + s 2 − 4 δ μ ] . z_+=\frac{1}{2}\left[(2-s)+\sqrt{s^2-4\delta_\mu}\right]. z + = 2 1 [ ( 2 − s ) + s 2 − 4 δ μ ] .
Using 1 − u ≥ 1 − u \sqrt{1-u}\ge1-u 1 − u ≥ 1 − u for u ∈ [ 0 , 1 ] u\in[0,1] u ∈ [ 0 , 1 ] gives s 2 − 4 δ μ ≥ s − 4 δ μ s \sqrt{s^2-4\delta_\mu}\ge s-\frac{4\delta_\mu}{s} s 2 − 4 δ μ ≥ s − s 4 δ μ ; hence,
1 − ρ ≤ 2 δ μ s = 2 Δ κ s for δ μ ≤ s 2 / 4. (80) 1-\rho\le\frac{2\delta_\mu}{s}=\frac{2\Delta}{\kappa s}\quad\text{for }\delta_\mu\le s^2/4. \tag*{(80)} 1 − ρ ≤ s 2 δ μ = κ s 2Δ for δ μ ≤ s 2 /4. (80)
We now combine these bounds. If s < 2 / κ s<2/\kappa s < 2/ κ , then (78) gives 1 − ρ < 1 κ < 2 κ 1-\rho<\frac{1}{\kappa}<\frac{2}{\kappa} 1 − ρ < κ 1 < κ 2 , which leads to the desired result. Now suppose s ≥ 2 / κ s\ge2/\kappa s ≥ 2/ κ . If δ μ > s 2 / 4 \delta_\mu>s^2/4 δ μ > s 2 /4 , then Δ > κ s 2 / 4 \Delta>\kappa s^2/4 Δ > κ s 2 /4 , and (79) gives 1 − ρ ≤ s − Δ < s − κ s 2 4 1-\rho\le s-\Delta<s-\frac{\kappa s^2}{4} 1 − ρ ≤ s − Δ < s − 4 κ s 2 . The function s ↦ s − κ s 2 / 4 s \mapsto s-\kappa s^2/4 s ↦ s − κ s 2 /4 is decreasing on [ 2 / κ , ∞ ) [2/\kappa,\infty) [ 2/ κ , ∞ ) ; therefore, for s = 2 / κ s=2/\kappa s = 2/ κ it achieves its maximum value on [ 2 / κ , ∞ ) [2/\kappa,\infty) [ 2/ κ , ∞ ) , namely 1 / κ 1/\kappa 1/ κ . Finally, it remains to consider the scenario with s ≥ 2 / κ s \ge2/\kappa s ≥ 2/ κ and δ μ ≤ s 2 / 4 \delta_\mu\le s^2/4 δ μ ≤ s 2 /4 . For this setting, both (79) and (80) apply, and we get
1 − ρ ≤ min { 2 Δ κ s , s − Δ } . 1-\rho\le\min\left\{\frac{2\Delta}{\kappa s},s-\Delta\right\}. 1 − ρ ≤ min { κ s 2Δ , s − Δ } .
The maximum of the right-hand side over 0 ≤ Δ ≤ s 0 \le\Delta\le s 0 ≤ Δ ≤ s occurs when the two affine functions balance, i.e. at Δ ⋆ = κ s 2 / ( 2 + κ s ) ∈ [ 0 , s ] \Delta_\star=\kappa s^2/(2+\kappa s)\in[0,s] Δ ⋆ = κ s 2 / ( 2 + κ s ) ∈ [ 0 , s ] , and the corresponding value is 2 s 2 + κ s < 2 κ \frac{2s}{2+\kappa s}<\frac{2}{\kappa} 2 + κ s 2 s < κ 2 . Thus, for all cases 1 − ρ < 2 / κ 1-\rho<2/\kappa 1 − ρ < 2/ κ .
The following result bounds the fast mode Δ \Delta Δ for any τ ≥ 1 \tau\ge1 τ ≥ 1 and shows that β < 1 \beta<1 β < 1 is necessary for Schur stability.
Proposition 3.16 (fast-mode cap, heavy ball). Fix τ ≥ 1 \tau\ge1 τ ≥ 1 . Let Δ > 0 \Delta>0 Δ > 0 and β > 0 \beta>0 β > 0 . If P τ , Δ H B P_{\tau,\Delta}^{\mathrm{HB}} P τ , Δ HB is Schur stable, then Δ < 2 s τ \Delta<\frac{2s}{\tau} Δ < τ 2 s where s = 1 − β s=1-\beta s = 1 − β . In particular, Schur stability necessarily implies β < 1 \beta<1 β < 1 .
Proof. Consider the Cayley transform
P ~ τ , Δ H B ( w ) : = ( 1 − w ) τ + 2 P τ , Δ H B ( 1 + w 1 − w ) . \widetilde{P}_{\tau,\Delta}^{\mathrm{HB}}(w):=(1-w)^{\tau+2}P_{\tau,\Delta}^{\mathrm{HB}}\left(\frac{1+w}{1-w}\right). P τ , Δ HB ( w ) := ( 1 − w ) τ + 2 P τ , Δ HB ( 1 − w 1 + w ) .
Since P τ , Δ H B P_{\tau,\Delta}^{\mathrm{HB}} P τ , Δ HB is Schur stable, Lemma 3.2 implies that P ~ τ , Δ H B \widetilde{P}_{\tau,\Delta}^{\mathrm{HB}} P τ , Δ HB is Hurwitz. Also P ~ τ , Δ H B ( 0 ) = P τ , Δ H B ( 1 ) = Δ > 0 \widetilde{P}_{\tau,\Delta}^{\mathrm{HB}}(0)=P_{\tau,\Delta}^{\mathrm{HB}}(1)=\Delta>0 P τ , Δ HB ( 0 ) = P τ , Δ HB ( 1 ) = Δ > 0 . A real Hurwitz polynomial has all coefficients of the same sign, so the coefficient of w w w in P ~ τ , Δ H B \widetilde{P}_{\tau,\Delta}^{\mathrm{HB}} P τ , Δ HB , which we denote by a ~ 1 \widetilde{a}_1 a 1 , must be positive. By differentiating P ~ τ , Δ H B \widetilde{P}_{\tau,\Delta}^{\mathrm{HB}} P τ , Δ HB at w = 0 w=0 w = 0 , we can compute
a ~ 1 = − ( τ + 2 ) P τ , Δ H B ( 1 ) + 2 P τ , Δ H B ′ ( 1 ) . \widetilde{a}_1=-(\tau+2)P_{\tau,\Delta}^{\mathrm{HB}}(1)+2P_{\tau,\Delta}^{\mathrm{HB}\prime}(1). a 1 = − ( τ + 2 ) P τ , Δ HB ( 1 ) + 2 P τ , Δ HB ′ ( 1 ) .
where P τ , Δ H B ( 1 ) = Δ P_{\tau,\Delta}^{\mathrm{HB}}(1)=\Delta P τ , Δ HB ( 1 ) = Δ and P τ , Δ H B ′ ( 1 ) = s + Δ P_{\tau,\Delta}^{\mathrm{HB}\prime}(1)=s+\Delta P τ , Δ HB ′ ( 1 ) = s + Δ ; therefore, a ~ 1 = − ( τ + 2 ) Δ + 2 ( s + Δ ) = 2 s − τ Δ \widetilde{a}_1=-(\tau+2)\Delta+2(s+\Delta)=2s-\tau\Delta a 1 = − ( τ + 2 ) Δ + 2 ( s + Δ ) = 2 s − τ Δ . Thus, a ~ 1 > 0 \widetilde{a}_1>0 a 1 > 0 leads to the desired bound 2 s − τ Δ > 0 2s-\tau\Delta>0 2 s − τ Δ > 0 . Finally, this implies β = 1 − s < 1 \beta=1-s<1 β = 1 − s < 1 as Δ > 0 \Delta>0 Δ > 0 .
Theorem 3.17 (Heavy ball is non-accelerated under any fixed positive delay). For the delayed heavy-ball method on Q μ , L Q_{\mu,L} Q μ , L and any fixed τ ≥ 1 \tau\ge1 τ ≥ 1 , every Schur-stable tuning satisfies 1 − ρ = O τ ( κ − 1 ) 1-\rho=O_\tau(\kappa^{-1}) 1 − ρ = O τ ( κ − 1 ) , as κ → ∞ \kappa\to\infty κ → ∞ . Consequently,
1 − ρ H B ⋆ ( τ , κ ) = O τ ( κ − 1 ) = o ( κ − 1 / 2 ) , 1-\rho_{\mathrm{HB}}^\star(\tau,\kappa)=O_\tau(\kappa^{-1})=o(\kappa^{-1/2}), 1 − ρ HB ⋆ ( τ , κ ) = O τ ( κ − 1 ) = o ( κ − 1/2 ) , i.e., the delayed heavy ball never retains the accelerated κ − 1 / 2 \kappa^{-1/2} κ − 1/2 scaling.
Proof. Fix a Schur-stable heavy-ball tuning and write s = 1 − β ∈ ( 0 , 1 ) s=1-\beta\in(0,1) s = 1 − β ∈ ( 0 , 1 ) , Δ = α L \Delta=\alpha L Δ = α L , and δ μ = Δ / κ \delta_\mu=\Delta/\kappa δ μ = Δ/ κ . First suppose that s ≥ 16 τ κ s\ge\frac{16}{\tau\kappa} s ≥ τ κ 16 . By Proposition 3.16 , Δ < 2 s τ \Delta<\frac{2s}{\tau} Δ < τ 2 s , and δ μ = Δ κ < 2 s τ κ \delta_\mu=\frac{\Delta}{\kappa}<\frac{2s}{\tau\kappa} δ μ = κ Δ < τ κ 2 s . By the lower bound on s s s , we get δ μ < 2 s τ κ ≤ s 2 8 \delta_\mu<\frac{2s}{\tau\kappa}\le\frac{s^2}{8} δ μ < τ κ 2 s ≤ 8 s 2 ; moreover, we also have
2 ( τ − 1 ) δ μ s ≤ 4 ( τ − 1 ) τ κ , \frac{2(\tau-1)\delta_\mu}{s}\le\frac{4(\tau-1)}{\tau\kappa}, s 2 ( τ − 1 ) δ μ ≤ τ κ 4 ( τ − 1 ) ,
which is at most 1 / 3 1/3 1/3 for all sufficiently large κ \kappa κ at fixed τ \tau τ ; therefore, Lemma 3.14 applies here and it yields
1 − ρ ≤ 2 δ μ s = 2 Δ κ s < 4 τ κ . 1-\rho\le\frac{2\delta_\mu}{s}=\frac{2\Delta}{\kappa s}<\frac{4}{\tau\kappa}. 1 − ρ ≤ s 2 δ μ = κ s 2Δ < τ κ 4 .
Next, consider the remaining scenario, i.e., s < 16 τ κ s<\frac{16}{\tau\kappa} s < τ κ 16 . Then s = O τ ( κ − 1 ) s=O_\tau(\kappa^{-1}) s = O τ ( κ − 1 ) , and the same fast-mode cap gives
δ μ = Δ κ < 2 s τ κ = O τ ( κ − 2 ) . \delta_\mu=\frac{\Delta}{\kappa}<\frac{2s}{\tau\kappa}=O_\tau(\kappa^{-2}). δ μ = κ Δ < τ κ 2 s = O τ ( κ − 2 ) .
Thus, for a constant B τ B_{\tau} B τ , depending only on the fixed delay, the hypotheses of Lemma 3.11 hold for the heavy-ball slow factor, and it implies that 1 − ρ ≤ C κ − 1 1-\rho\le C\kappa^{-1} 1 − ρ ≤ C κ − 1 . Combining the two regimes proves 1 − ρ = O τ ( κ − 1 ) 1-\rho=O_{\tau}(\kappa^{-1}) 1 − ρ = O τ ( κ − 1 ) for every Schur-stable heavy-ball tuning. Taking the infimum over Schur-stable tunings gives the stated bound on 1 − ρ H B ⋆ 1-\rho_{\mathrm{HB}}^{\star} 1 − ρ HB ⋆ .
GMM achieves the accelerated rate with κ \sqrt{\kappa} κ scaling when τ = 1 \tau=1 τ = 1
For a one-step delay (τ = 1 \tau=1 τ = 1 ), the next proposition gives explicit GMM parameters whose worst-case spectral radius is exactly 1 − 1 / ( 2 κ ) 1-1/(2\sqrt{\kappa}) 1 − 1/ ( 2 κ ) . Theorem 3.25 later gives a different parameter choice that achieves acceleration for every fixed positive delay, provided κ \kappa κ is sufficiently large.
Throughout this and the following subsection we write δ L : = α L \delta_L:=\alpha L δ L := α L for the fast-mode gain to emphasize its dependence on L L L ; this quantity was denoted as Δ \Delta Δ in the lower-bound analysis above.
Proposition 3.19 (an accelerated delayed GMM for τ = 1 \tau=1 τ = 1 ). For τ = 1 \tau=1 τ = 1 , let q : = κ − 1 / 2 q:=\kappa^{-1/2} q := κ − 1/2 and ρ : = 1 − q 2 \rho:=1-\frac{q}{2} ρ := 1 − 2 q . With the following constant parameters,
α = 1 4 L ( 1 − q 2 2 ) , β = 1 − q + q 2 2 , ν = 2 ( 1 − q + q 2 4 ) 1 − q 2 2 , \alpha=\frac{1}{4L}\left(1-\frac{q^2}{2}\right),\qquad\beta=1-q+\frac{q^2}{2},\qquad\nu=\frac{2\left(1-q+\frac{q^2}{4}\right)}{1-\frac{q^2}{2}}, α = 4 L 1 ( 1 − 2 q 2 ) , β = 1 − q + 2 q 2 , ν = 1 − 2 q 2 2 ( 1 − q + 4 q 2 ) , the delayed GMM is Schur stable on every quadratic with spectrum in [ μ , L ] [\mu,L] [ μ , L ] , and its worst-case spectral radius can be computed exactly as follows:
sup λ ∈ [ μ , L ] max { ∣ z ∣ : P 1 , α λ ( z ) = 0 } = ρ = 1 − 1 2 κ = 1 − q 2 . \sup_{\lambda\in[\mu,L]}\max\{|z|:P_{1,\alpha\lambda}(z)=0\}=\rho=1-\frac{1}{2\sqrt{\kappa}}=1-\frac{q}{2}. λ ∈ [ μ , L ] sup max { ∣ z ∣ : P 1 , α λ ( z ) = 0 } = ρ = 1 − 2 κ 1 = 1 − 2 q .
Proof. Write δ L = α L = 1 4 ( 1 − q 2 2 ) \delta_L=\alpha L=\frac{1}{4}\left(1-\frac{q^2}{2}\right) δ L = α L = 4 1 ( 1 − 2 q 2 ) and t = λ / L ∈ [ q 2 , 1 ] t=\lambda/L\in[q^2,1] t = λ / L ∈ [ q 2 , 1 ] ; hence, δ = t δ L \delta=t\delta_L δ = t δ L where we used the definition δ : = α λ \delta:=\alpha\lambda δ := α λ for λ ∈ [ μ , L ] \lambda\in[\mu,L] λ ∈ [ μ , L ] . After some computations,
P 1 , t δ L ( z ) = z 3 − ( 2 − q + q 2 2 ) z 2 + ( 1 − q + q 2 2 + t ( 3 4 − q 2 ) ) z − t ( 1 2 − q 2 + q 2 8 ) . (81) P_{1,t\delta_L}(z)=z^3-\left(2-q+\frac{q^2}{2}\right)z^2+\left(1-q+\frac{q^2}{2}+t\left(\frac{3}{4}-\frac{q}{2}\right)\right)z-t\left(\frac{1}{2}-\frac{q}{2}+\frac{q^2}{8}\right). \tag*{(81)} P 1 , t δ L ( z ) = z 3 − ( 2 − q + 2 q 2 ) z 2 + ( 1 − q + 2 q 2 + t ( 4 3 − 2 q ) ) z − t ( 2 1 − 2 q + 8 q 2 ) . (81)
If q = 1 q=1 q = 1 , then the interval t ∈ [ q 2 , 1 ] t\in[q^2,1] t ∈ [ q 2 , 1 ] degenerates to t = 1 t=1 t = 1 and P 1 , δ L ( z ) = ( z − 1 2 ) 3 P_{1,\delta_L}(z)=(z-\frac{1}{2})^3 P 1 , δ L ( z ) = ( z − 2 1 ) 3 ; thus, the claim holds. Next, assume 0 < q < 1 0<q<1 0 < q < 1 . At the endpoints one has the exact factorizations P 1 , q 2 δ L ( z ) = ( z − ρ ) 2 ( z − q 2 2 ) P_{1,q^2\delta_L}(z)=(z-\rho)^2\left(z-\frac{q^2}{2}\right) P 1 , q 2 δ L ( z ) = ( z − ρ ) 2 ( z − 2 q 2 ) and P 1 , δ L ( z ) = ( z − 1 2 ) ( z 2 − 2 a z + ρ 2 ) P_{1,\delta_L}(z)=\left(z-\frac{1}{2}\right)(z^2-2az+\rho^2) P 1 , δ L ( z ) = ( z − 2 1 ) ( z 2 − 2 a z + ρ 2 ) with a = 3 4 − q 2 + q 2 4 a=\frac{3}{4}-\frac{q}{2}+\frac{q^2}{4} a = 4 3 − 2 q + 4 q 2 —note that for t = 1 t=1 t = 1 , since ρ − a = 1 − q 2 4 > 0 \rho-a=\frac{1-q^2}{4}>0 ρ − a = 4 1 − q 2 > 0 , the quadratic factor in P 1 , δ L P_{1,\delta_L} P 1 , δ L has complex roots of modulus exactly ρ \rho ρ , and 1 2 < ρ \frac{1}{2}<\rho 2 1 < ρ . Hence, all the roots z z z of P 1 , t δ L ( z ) P_{1,t\delta_L}(z) P 1 , t δ L ( z ) satisfy ∣ z ∣ ≤ ρ |z|\le\rho ∣ z ∣ ≤ ρ at the endpoints t = q 2 t=q^2 t = q 2 or t = 1 t=1 t = 1 . For interior t ∈ ( q 2 , 1 ) t\in(q^2,1) t ∈ ( q 2 , 1 ) , set z = ρ y z=\rho y z = ρ y and apply the Schur-Cohn conditions for stability to p ^ t ( y ) = ρ − 3 P 1 , t δ L ( ρ y ) \hat p_t(y)=\rho^{-3}P_{1,t\delta_L}(\rho y) p ^ t ( y ) = ρ − 3 P 1 , t δ L ( ρ y ) . More specifically, with the cubic normalized as y 3 + a 2 y 2 + a 1 y + a 0 y^3+a_2y^2+a_1y+a_0 y 3 + a 2 y 2 + a 1 y + a 0 , Lemma 3.3 requires the positivity of p ^ t ( 1 ) \hat p_t(1) p ^ t ( 1 ) , − p ^ t ( − 1 ) -\hat p_t(-1) − p ^ t ( − 1 ) , 1 − a 0 2 1-a_0^2 1 − a 0 2 , and the two margins 1 − a 0 2 ± ( a 0 a 2 − a 1 ) 1-a_0^2\pm(a_0a_2-a_1) 1 − a 0 2 ± ( a 0 a 2 − a 1 ) . After some straightforward computations, we get
p ^ t ( 1 ) = ( 1 − q ) ( t − q 2 ) ( 2 − q ) 2 , − p ^ t ( − 1 ) = 16 ( 1 − q ) + 7 q 2 − q 3 + ( 5 − 3 q ) t ( 2 − q ) 2 , 1 − a 0 2 = ( 2 − q − t ) ( 2 − q + t ) ( 2 − q ) 2 , (82) \hat p_t(1)=\frac{(1-q)(t-q^2)}{(2-q)^2},\qquad-\hat p_t(-1)=\frac{16(1-q)+7q^2-q^3+(5-3q)t}{(2-q)^2},\qquad1-a_0^2=\frac{(2-q-t)(2-q+t)}{(2-q)^2}, \tag*{(82)} p ^ t ( 1 ) = ( 2 − q ) 2 ( 1 − q ) ( t − q 2 ) , − p ^ t ( − 1 ) = ( 2 − q ) 2 16 ( 1 − q ) + 7 q 2 − q 3 + ( 5 − 3 q ) t , 1 − a 0 2 = ( 2 − q ) 2 ( 2 − q − t ) ( 2 − q + t ) , (82)
1 − a 0 2 + ( a 0 a 2 − a 1 ) = ( t − q 2 ) ( 1 − t ) ( 2 − q ) 2 , 1 − a 0 2 − ( a 0 a 2 − a 1 ) = 8 ( 1 − q ) + 3 q 2 − q 2 t − t − t 2 ( 2 − q ) 2 . (83) 1-a_0^2+(a_0a_2-a_1)=\frac{(t-q^2)(1-t)}{(2-q)^2},\qquad1-a_0^2-(a_0a_2-a_1)=\frac{8(1-q)+3q^2-q^2t-t-t^2}{(2-q)^2}. \tag*{(83)} 1 − a 0 2 + ( a 0 a 2 − a 1 ) = ( 2 − q ) 2 ( t − q 2 ) ( 1 − t ) , 1 − a 0 2 − ( a 0 a 2 − a 1 ) = ( 2 − q ) 2 8 ( 1 − q ) + 3 q 2 − q 2 t − t − t 2 . (83)
All denominators are positive in (82) and (83) . The numerators in (82) and the first one in (83) are all strictly positive for 0 < q < 1 0<q<1 0 < q < 1 and q 2 < t < 1 q^2<t<1 q 2 < t < 1 . On the other hand, for the second numerator in (83) , define
N ( t ) : = 8 ( 1 − q ) + 3 q 2 − q 2 t − t − t 2 . N(t):=8(1-q)+3q^2-q^2t-t-t^2. N ( t ) := 8 ( 1 − q ) + 3 q 2 − q 2 t − t − t 2 .
Then N ′ ( t ) = − q 2 − 1 − 2 t < 0 N'(t)=-q^2-1-2t<0 N ′ ( t ) = − q 2 − 1 − 2 t < 0 , so N N N is decreasing in t t t . Since t < 1 t<1 t < 1 ,
N ( t ) > N ( 1 ) = 8 ( 1 − q ) + 3 q 2 − q 2 − 1 − 1 = 2 ( 1 − q ) ( 3 − q ) > 0. N(t)>N(1)=8(1-q)+3q^2-q^2-1-1=2(1-q)(3-q)>0. N ( t ) > N ( 1 ) = 8 ( 1 − q ) + 3 q 2 − q 2 − 1 − 1 = 2 ( 1 − q ) ( 3 − q ) > 0.
Thus, all Schur–Cohn inequalities hold, which implies that all roots of p ^ t \hat{p}_t p ^ t lie in the open unit disk. Equivalently, all roots of P 1 , t δ L P_{1,t\delta_L} P 1 , t δ L satisfy ∣ z ∣ < ρ |z|<\rho ∣ z ∣ < ρ for q 2 < t < 1 q^2<t<1 q 2 < t < 1 , while at each endpoint at least one root satisfies ∣ z ∣ = ρ |z|=\rho ∣ z ∣ = ρ . This completes the proof.
Proposition 3.19 gives 1 − ρ = q / 2 1-\rho=q/2 1 − ρ = q /2 , where q = κ − 1 / 2 q=\kappa^{-1/2} q = κ − 1/2 . Thus, its normalized spectral gap ( 1 − ρ ) / q (1-\rho)/q ( 1 − ρ ) / q is exactly 1 / 2 1/2 1/2 . The next result shows that this leading constant cannot be improved and identifies the parameter behavior required to attain it. Before presenting this result, we introduce some notation. Let L > 0 L>0 L > 0 be fixed and τ = 1 \tau=1 τ = 1 . For each q q q , consider GMM parameters α q > 0 \alpha_q>0 α q > 0 , β q ≥ 0 \beta_q\geq0 β q ≥ 0 and ν q ≥ 0 \nu_q\geq0 ν q ≥ 0 , which may depend arbitrarily on q q q , and define the normalized stepsize
Δ q : = α q L . \Delta_q:=\alpha_qL. Δ q := α q L .
For each normalized curvature
t : = λ L ∈ [ q 2 , 1 ] , t:=\frac{\lambda}{L}\in[q^2,1], t := L λ ∈ [ q 2 , 1 ] ,
let
P q , t ( z ) : = z ( z − 1 ) ( z − β q ) + t Δ q ( ( 1 + ν q ) z − ν q ) . (84) P_{q,t}(z):=z(z-1)(z-\beta_q)+t\Delta_q\bigl((1+\nu_q)z-\nu_q\bigr). \tag*{(84)} P q , t ( z ) := z ( z − 1 ) ( z − β q ) + t Δ q ( ( 1 + ν q ) z − ν q ) . (84)
denote the corresponding characteristic polynomial. define its worst-case spectral radius by
ρ ( q ; θ q ) : = sup t ∈ [ q 2 , 1 ] max { ∣ z ∣ : P q , t ( z ) = 0 } , \rho(q;\theta_q):=\sup_{t\in[q^2,1]}\max\{|z|:P_{q,t}(z)=0\}, ρ ( q ; θ q ) := t ∈ [ q 2 , 1 ] sup max { ∣ z ∣ : P q , t ( z ) = 0 } ,
where θ q \theta_q θ q is held fixed and the supremum is taken only over the curvature parameter t t t .
Proposition 3.20 (Sharp asymptotic optimality for τ = 1 \tau=1 τ = 1 ). Let L > 0 L>0 L > 0 be fixed, τ = 1 \tau=1 τ = 1 and let θ q = ( Δ q , β q , ν q ) \theta_q=(\Delta_q,\beta_q,\nu_q) θ q = ( Δ q , β q , ν q ) be any family of parameters with Δ q > 0 \Delta_q>0 Δ q > 0 , β q ≥ 0 \beta_q\geq0 β q ≥ 0 , and ν q ≥ 0 \nu_q\geq0 ν q ≥ 0 , such that P q , t ( ⋅ ) P_{q,t}(\cdot) P q , t ( ⋅ ) defined in (84) is Schur stable for every t ∈ [ q 2 , 1 ] t\in[q^2,1] t ∈ [ q 2 , 1 ] . Then
lim sup q ↓ 0 G ( q ; θ q ) ≤ 1 2 . \limsup_{q\downarrow0}G(q;\theta_q)\leq\frac{1}{2}. q ↓ 0 lim sup G ( q ; θ q ) ≤ 2 1 . Hence the tuning of Proposition 3.19, for which ρ ( q ; θ q ) = 1 − q 2 \rho(q;\theta_q)=1-\frac{q}{2} ρ ( q ; θ q ) = 1 − 2 q , is asymptotically optimal. Moreover, any asymptotically optimal family satisfying
G ( q ; θ q ) → 1 2 as q ↓ 0 G(q;\theta_q)\to\frac{1}{2}\qquad\text{as }q\downarrow0 G ( q ; θ q ) → 2 1 as q ↓ 0 must obey
Δ q = 1 4 + o ( 1 ) , β q = 1 − q + o ( q ) , ν q = 2 + o ( 1 ) . \Delta_q=\frac{1}{4}+o(1),\qquad\beta_q=1-q+o(q),\qquad\nu_q=2+o(1). Δ q = 4 1 + o ( 1 ) , β q = 1 − q + o ( q ) , ν q = 2 + o ( 1 ) .
Proof. For each n ≥ 1 n \ge1 n ≥ 1 , define
s n : = sup 0 < q < 1 / n G ( q ; θ q ) . s_n := \sup_{0<q<1/n} G(q;\theta_q). s n := 0 < q < 1/ n sup G ( q ; θ q ) .
Then
γ : = lim sup q ↓ 0 G ( q ; θ q ) = lim n → ∞ s n ∈ [ 0 , + ∞ ] . \gamma:= \limsup_{q\downarrow0} G(q;\theta_q) = \lim_{n\to\infty} s_n \in[0,+\infty]. γ := q ↓ 0 lim sup G ( q ; θ q ) = n → ∞ lim s n ∈ [ 0 , + ∞ ] .
For each n n n , choose q n ∈ ( 0 , 1 / n ) q_n \in(0,1/n) q n ∈ ( 0 , 1/ n ) such that G ( q n ; θ q n ) ≥ min { s n , n } − 1 n G(q_n;\theta_{q_n}) \ge\min\{s_n,n\} - \frac{1}{n} G ( q n ; θ q n ) ≥ min { s n , n } − n 1 . Such a choice is possible by the definition of the supremum. Since 0 < q n < 1 / n 0<q_n<1/n 0 < q n < 1/ n , we have q n → 0 q_n \to0 q n → 0 . Moreover,
G n : = G ( q n ; θ q n ) ⟶ γ , G_n := G(q_n;\theta_{q_n}) \longrightarrow\gamma, G n := G ( q n ; θ q n ) ⟶ γ ,
where the case ( γ = + ∞ ) (\gamma=+\infty) ( γ = + ∞ ) is understood in the extended-real sense. For notational convenience, define
ρ n : = ρ ( q n ; θ q n ) , β n : = β q n , Δ n : = Δ q n , ν n : = ν q n , and ω n : = Δ q n ν q n . \rho_n := \rho(q_n;\theta_{q_n}), \qquad\beta_n := \beta_{q_n}, \qquad\Delta_n := \Delta_{q_n}, \qquad\nu_n := \nu_{q_n}, \qquad\text{and} \qquad\omega_n := \Delta_{q_n}\nu_{q_n}. ρ n := ρ ( q n ; θ q n ) , β n := β q n , Δ n := Δ q n , ν n := ν q n , and ω n := Δ q n ν q n .
By the definition of G n G_n G n , 1 − ρ n = q n G n 1-\rho_n=q_nG_n 1 − ρ n = q n G n . The proof proceeds in several steps.
Step 1: Boundedness of parameters. We first show that the parameter sequences { β n } \{\beta_n\} { β n } , { Δ n } \{\Delta_n\} { Δ n } and { ω n } \{\omega_n\} { ω n } are bounded. At the upper endpoint t = λ / L = 1 t=\lambda/L=1 t = λ / L = 1 (when λ = L \lambda=L λ = L ), define
P ‾ k ( z ) : = P q n , 1 ( z ) . \overline{P}_k(z) := P_{q_n,1}(z). P k ( z ) := P q n , 1 ( z ) .
Using ω n = Δ n ν n \omega_n=\Delta_n\nu_n ω n = Δ n ν n , this polynomial can be written as
P ‾ k ( z ) = z 3 − ( 1 + β n ) z 2 + ( β n + Δ n + ω n ) z − ω n . \overline{P}_k(z)=z^3-(1+\beta_n)z^2+(\beta_n+\Delta_n+\omega_n)z-\omega_n. P k ( z ) = z 3 − ( 1 + β n ) z 2 + ( β n + Δ n + ω n ) z − ω n .
Let its roots be ζ 1 , n \zeta_{1,n} ζ 1 , n , ζ 2 , n \zeta_{2,n} ζ 2 , n and ζ 3 , n \zeta_{3,n} ζ 3 , n . Because ρ n \rho_n ρ n is the worst-case spectral radius over t ∈ [ q n 2 , 1 ] t\in[q_n^2,1] t ∈ [ q n 2 , 1 ] , we have
∣ ζ j , n ∣ ≤ ρ n < 1 , j = 1 , 2 , 3. |\zeta_{j,n}| \le\rho_n < 1, \qquad j=1,2,3. ∣ ζ j , n ∣ ≤ ρ n < 1 , j = 1 , 2 , 3.
By Vieta’s formulas, ζ 1 , n + ζ 2 , n + ζ 3 , n = 1 + β n \zeta_{1,n}+\zeta_{2,n}+\zeta_{3,n}=1+\beta_n ζ 1 , n + ζ 2 , n + ζ 3 , n = 1 + β n . Since β n ≥ 0 \beta_n\ge0 β n ≥ 0 ,
1 + β n = ∣ ζ 1 , n + ζ 2 , n + ζ 3 , n ∣ ≤ ∑ j = 1 3 ∣ ζ j , n ∣ ≤ 3 ρ n < 3. 1+\beta_n=|\zeta_{1,n}+\zeta_{2,n}+\zeta_{3,n}|
\le\sum_{j=1}^{3}|\zeta_{j,n}|
\le3\rho_n<3. 1 + β n = ∣ ζ 1 , n + ζ 2 , n + ζ 3 , n ∣ ≤ j = 1 ∑ 3 ∣ ζ j , n ∣ ≤ 3 ρ n < 3.
Therefore, 0 ≤ β n < 2 0\le\beta_n<2 0 ≤ β n < 2 . On the other hand, the product-of-roots identity gives ζ 1 , n ζ 2 , n ζ 3 , n = ω n \zeta_{1,n}\zeta_{2,n}\zeta_{3,n}=\omega_n ζ 1 , n ζ 2 , n ζ 3 , n = ω n . Since ω n = Δ n ν n ≥ 0 \omega_n=\Delta_n\nu_n\ge0 ω n = Δ n ν n ≥ 0 ,
0 ≤ ω n = ∣ ζ 1 , n ζ 2 , n ζ 3 , n ∣ ≤ ρ n 3 < 1. 0\le\omega_n=|\zeta_{1,n}\zeta_{2,n}\zeta_{3,n}|
\le\rho_n^3<1. 0 ≤ ω n = ∣ ζ 1 , n ζ 2 , n ζ 3 , n ∣ ≤ ρ n 3 < 1.
Finally, P n , L ( 1 ) = Δ n P_{n,L}(1)=\Delta_n P n , L ( 1 ) = Δ n . Factoring the polynomial in terms of its roots therefore gives
Δ n = ∣ ( 1 − ζ 1 , n ) ( 1 − ζ 2 , n ) ( 1 − ζ 3 , n ) ∣ . \Delta_n=|(1-\zeta_{1,n})(1-\zeta_{2,n})(1-\zeta_{3,n})|. Δ n = ∣ ( 1 − ζ 1 , n ) ( 1 − ζ 2 , n ) ( 1 − ζ 3 , n ) ∣.
Hence
0 < Δ n ≤ ∏ j = 1 3 ( 1 + ∣ ζ j , n ∣ ) ≤ ( 1 + ρ n ) 3 < 8. 0<\Delta_n\le\prod_{j=1}^{3}(1+|\zeta_{j,n}|)
\le(1+\rho_n)^3<8. 0 < Δ n ≤ j = 1 ∏ 3 ( 1 + ∣ ζ j , n ∣ ) ≤ ( 1 + ρ n ) 3 < 8.
We have thus proved the uniform bounds 0 ≤ β n < 2 0\le\beta_n<2 0 ≤ β n < 2 , 0 < Δ n < 8 0<\Delta_n<8 0 < Δ n < 8 and 0 ≤ ω n < 1 0\le\omega_n<1 0 ≤ ω n < 1 . By the Bolzano–Weierstrass theorem applied to the bounded sequence { ( β n , Δ n , ω n ) } n ≥ 1 ⊂ R 3 \{(\beta_n,\Delta_n,\omega_n)\}_{n\ge1}\subset\mathbb{R}^3 {( β n , Δ n , ω n ) } n ≥ 1 ⊂ R 3 , there exist a strictly increasing sequence of indices n 1 < n 2 < ⋯ n_1<n_2<\cdots n 1 < n 2 < ⋯ and limits β ˉ , Δ ˉ , ω ˉ ∈ R \bar{\beta},\bar{\Delta},\bar{\omega}\in\mathbb{R} β ˉ , Δ ˉ , ω ˉ ∈ R such that
β n k ⟶ β ˉ , Δ n k ⟶ Δ ˉ , ω n k ⟶ ω ˉ as k → ∞ . \beta_{n_k}\longrightarrow\bar{\beta}, \qquad
\Delta_{n_k}\longrightarrow\bar{\Delta}, \qquad
\omega_{n_k}\longrightarrow\bar{\omega}
\qquad\text{as }k\to\infty. β n k ⟶ β ˉ , Δ n k ⟶ Δ ˉ , ω n k ⟶ ω ˉ as k → ∞.
Since q n → 0 q_n \to0 q n → 0 and G n → γ G_n \to\gamma G n → γ , the corresponding subsequences also satisfy q n k → 0 q_{n_k} \to0 q n k → 0 and G n k → γ G_{n_k} \to\gamma G n k → γ .
*Step 2: Finer analysis of the limit β ˉ \bar{\beta} β ˉ . * We next analyze possible values of the limit β ˉ \bar{\beta} β ˉ and its consequences on the γ \gamma γ value. To do this, we consider t = q n k 2 t=q_{n_k}^2 t = q n k 2 and analyze the lower endpoint of the interval for t t t . Define
P k ( z ) : = P q n k , q n k 2 ( z ) . \mathcal{P}_k(z) := P_{q_{n_k},q_{n_k}^2}(z). P k ( z ) := P q n k , q n k 2 ( z ) .
Then
P k ( z ) = z ( z − 1 ) ( z − β n k ) + q n k 2 [ Δ n k + ( Δ n k + ω n k ) ( z − 1 ) ] . (85) \mathcal{P}_k(z)=z(z-1)(z-\beta_{n_k})+q_{n_k}^2\left[\Delta_{n_k}+(\Delta_{n_k}+\omega_{n_k})(z-1)\right]. \tag*{(85)} P k ( z ) = z ( z − 1 ) ( z − β n k ) + q n k 2 [ Δ n k + ( Δ n k + ω n k ) ( z − 1 ) ] . (85)
Because q n k → 0 q_{n_k} \to0 q n k → 0 , β n k → β ˉ \beta_{n_k} \to\bar{\beta} β n k → β ˉ , and the sequences { Δ n k } \{\Delta_{n_k}\} { Δ n k } and { ω n k } \{\omega_{n_k}\} { ω n k } are bounded, we have the coefficient-wise convergence
P k ( z ) ⟶ P ∞ ( z ) : = z ( z − 1 ) ( z − β ˉ ) . \mathcal{P}_k(z) \longrightarrow\mathcal{P}_{\infty}(z):=z(z-1)(z-\bar{\beta}). P k ( z ) ⟶ P ∞ ( z ) := z ( z − 1 ) ( z − β ˉ ) .
Since β ˉ \bar{\beta} β ˉ is a root of the limiting polynomial z ( z − 1 ) ( z − β ˉ ) z(z-1)(z-\bar{\beta}) z ( z − 1 ) ( z − β ˉ ) , continuity of polynomial roots implies that there exist roots z n k z_{n_k} z n k of P k ( z ) \mathcal{P}_k(z) P k ( z ) such that z n k ⟶ β ˉ z_{n_k}\longrightarrow\bar{\beta} z n k ⟶ β ˉ . Because every root of P k ( z ) \mathcal{P}_k(z) P k ( z ) satisfies
∣ z n k ∣ ≤ ρ n k < 1 , |z_{n_k}|\leq\rho_{n_k}<1, ∣ z n k ∣ ≤ ρ n k < 1 ,
taking limits gives ∣ β ˉ ∣ ≤ 1 |\bar{\beta}|\leq1 ∣ β ˉ ∣ ≤ 1 . Since β ˉ ≥ 0 \bar{\beta}\geq0 β ˉ ≥ 0 , it follows that 0 ≤ β ˉ ≤ 1 0\leq\bar{\beta}\leq1 0 ≤ β ˉ ≤ 1 . There are two cases:
Case (i): Suppose first that β ˉ < 1 \bar{\beta}<1 β ˉ < 1 . Then z = 1 z=1 z = 1 is a simple root of the limiting polynomial P ∞ ( z ) = z ( z − 1 ) ( z − β ˉ ) \mathcal{P}_{\infty}(z)=z(z-1)(z-\bar{\beta}) P ∞ ( z ) = z ( z − 1 ) ( z − β ˉ ) . By continuity of roots, there exists a root z n k z_{n_k} z n k of P k ( z ) \mathcal{P}_k(z) P k ( z ) such that z n k ⟶ 1 z_{n_k}\longrightarrow1 z n k ⟶ 1 . Define
δ n k : = 1 − z n k . \delta_{n_k}:=1-z_{n_k}. δ n k := 1 − z n k .
Since P k ( z n k ) = 0 \mathcal{P}_k(z_{n_k})=0 P k ( z n k ) = 0 , the identity
P k ( z ) = z ( z − 1 ) ( z − β n k ) + q n k 2 [ Δ n k + ( Δ n k + ω n k ) ( z − 1 ) ] \mathcal{P}_k(z)=z(z-1)(z-\beta_{n_k})+q_{n_k}^2\left[\Delta_{n_k}+(\Delta_{n_k}+\omega_{n_k})(z-1)\right] P k ( z ) = z ( z − 1 ) ( z − β n k ) + q n k 2 [ Δ n k + ( Δ n k + ω n k ) ( z − 1 ) ]
gives
[ z n k ( z n k − β n k ) + q n k 2 ( Δ n k + ω n k ) ] δ n k = q n k 2 Δ n k . \left[z_{n_k}(z_{n_k}-\beta_{n_k})+q_{n_k}^2(\Delta_{n_k}+\omega_{n_k})\right]\delta_{n_k}=q_{n_k}^2\Delta_{n_k}. [ z n k ( z n k − β n k ) + q n k 2 ( Δ n k + ω n k ) ] δ n k = q n k 2 Δ n k .
Moreover,
z n k ( z n k − β n k ) + q n k 2 ( Δ n k + ω n k ) ⟶ 1 − β ˉ > 0. z_{n_k}(z_{n_k}-\beta_{n_k})+q_{n_k}^2(\Delta_{n_k}+\omega_{n_k})\longrightarrow1-\bar{\beta}>0. z n k ( z n k − β n k ) + q n k 2 ( Δ n k + ω n k ) ⟶ 1 − β ˉ > 0.
Therefore, the bracketed factor is bounded away from zero for all sufficiently large k k k . Since { Δ n k } \{\Delta_{n_k}\} { Δ n k } is bounded, it follows that
∣ δ n k ∣ = O ( q n k 2 ) . |\delta_{n_k}|=O(q_{n_k}^2). ∣ δ n k ∣ = O ( q n k 2 ) .
Hence,
∣ z n k ∣ = ∣ 1 − δ n k ∣ ≥ 1 − ∣ δ n k ∣ = 1 − O ( q n k 2 ) . |z_{n_k}|=|1-\delta_{n_k}|\geq1-|\delta_{n_k}|=1-O(q_{n_k}^2). ∣ z n k ∣ = ∣1 − δ n k ∣ ≥ 1 − ∣ δ n k ∣ = 1 − O ( q n k 2 ) .
Consequently,
0 < G n k = 1 − ρ n k q n k ≤ 1 − ∣ z n k ∣ q n k = O ( q n k ) ⟶ 0. 0<G_{n_k}=\frac{1-\rho_{n_k}}{q_{n_k}}\leq\frac{1-|z_{n_k}|}{q_{n_k}}=O(q_{n_k})\longrightarrow0. 0 < G n k = q n k 1 − ρ n k ≤ q n k 1 − ∣ z n k ∣ = O ( q n k ) ⟶ 0.
Since G n k → γ G_{n_k}\to\gamma G n k → γ , this implies γ = 0 \gamma=0 γ = 0 . Thus, if γ = 0 \gamma=0 γ = 0 , the desired upper bound is immediate.
Case (ii): It remains to consider the case β ˉ = 1 \bar{\beta}=1 β ˉ = 1 . We will argue that in this case γ ≤ 1 / 2 \gamma\leq1/2 γ ≤ 1/2 . Assuming β = 1 \beta=1 β = 1 , the polynomials P k ( z ) \mathcal{P}_k(z) P k ( z ) satisfy
P k ( z ) ⟶ P ∞ ( z ) = z ( z − 1 ) 2 . \mathcal{P}_k(z)\longrightarrow\mathcal{P}_{\infty}(z)=z(z-1)^2. P k ( z ) ⟶ P ∞ ( z ) = z ( z − 1 ) 2 .
We may consequently label their roots as r 0 , n k r_{0,n_k} r 0 , n k , r 1 , n k r_{1,n_k} r 1 , n k and r 2 , n k r_{2,n_k} r 2 , n k so that
r 0 , n k ⟶ 0 , r 1 , n k ⟶ 1 , r 2 , n k ⟶ 1. r_{0,n_k} \longrightarrow0,\qquad r_{1,n_k} \longrightarrow1,\qquad r_{2,n_k} \longrightarrow1. r 0 , n k ⟶ 0 , r 1 , n k ⟶ 1 , r 2 , n k ⟶ 1.
Evaluating P ‾ k ( z ) \overline{P}_k(z) P k ( z ) at z = 1 z=1 z = 1 gives
q n k 2 Δ n k = P ‾ k ( 1 ) = ( 1 − r 0 , n k ) ( 1 − r 1 , n k ) ( 1 − r 2 , n k ) . q_{n_k}^{2}\Delta_{n_k}=\overline{P}_k(1)=(1-r_{0,n_k})(1-r_{1,n_k})(1-r_{2,n_k}). q n k 2 Δ n k = P k ( 1 ) = ( 1 − r 0 , n k ) ( 1 − r 1 , n k ) ( 1 − r 2 , n k ) .
Taking absolute values and using ∣ r j , n k ∣ ≤ ρ n k |r_{j,n_k}|\leq\rho_{n_k} ∣ r j , n k ∣ ≤ ρ n k , we obtain
q n k 2 Δ n k = ∣ 1 − r 0 , n k ∣ ∣ 1 − r 1 , n k ∣ ∣ 1 − r 2 , n k ∣ ≥ ∣ 1 − r 0 , n k ∣ ( 1 − ρ n k ) 2 = ∣ 1 − r 0 , n k ∣ q n k 2 G n k 2 . q_{n_k}^{2}\Delta_{n_k}=|1-r_{0,n_k}||1-r_{1,n_k}||1-r_{2,n_k}|\geq|1-r_{0,n_k}|(1-\rho_{n_k})^{2}=|1-r_{0,n_k}|q_{n_k}^{2}G_{n_k}^{2}. q n k 2 Δ n k = ∣1 − r 0 , n k ∣∣1 − r 1 , n k ∣∣1 − r 2 , n k ∣ ≥ ∣1 − r 0 , n k ∣ ( 1 − ρ n k ) 2 = ∣1 − r 0 , n k ∣ q n k 2 G n k 2 .
After dividing by q n k 2 q_{n_k}^{2} q n k 2 , we obtain
Δ n k ≥ ∣ 1 − r 0 , n k ∣ G n k 2 . (86) \Delta_{n_k}\geq|1-r_{0,n_k}|G_{n_k}^{2}. \tag*{(86)} Δ n k ≥ ∣1 − r 0 , n k ∣ G n k 2 . (86)
Since { Δ n k } \{\Delta_{n_k}\} { Δ n k } is bounded and ∣ 1 − r 0 , n k ∣ ⟶ 1 |1-r_{0,n_k}|\longrightarrow1 ∣1 − r 0 , n k ∣ ⟶ 1 , the sequence { G n k } \{G_{n_k}\} { G n k } is bounded. Hence γ < + ∞ \gamma<+\infty γ < + ∞ . We may now pass to the limit in the preceding inequality (86) to obtain
Δ ‾ ≥ γ 2 . (87) \overline{\Delta}\geq\gamma^{2}. \tag*{(87)} Δ ≥ γ 2 . (87)
We next return to the t = 1 t=1 t = 1 endpoint associated. Write
P ‾ k ( z ) = z 3 + a 2 , n k z 2 + a 1 , n k z + a 0 , n k , \overline{P}_k(z)=z^{3}+a_{2,n_k}z^{2}+a_{1,n_k}z+a_{0,n_k}, P k ( z ) = z 3 + a 2 , n k z 2 + a 1 , n k z + a 0 , n k ,
where
a 2 , n k = − ( 1 + β n k ) , a 1 , n k = β n k + Δ n k + ω n k , a 0 , n k = − ω n k . a_{2,n_k}=-(1+\beta_{n_k}),\qquad a_{1,n_k}=\beta_{n_k}+\Delta_{n_k}+\omega_{n_k},\qquad a_{0,n_k}=-\omega_{n_k}. a 2 , n k = − ( 1 + β n k ) , a 1 , n k = β n k + Δ n k + ω n k , a 0 , n k = − ω n k .
Because P ‾ k \overline{P}_k P k is Schur stable, the fourth cubic Schur–Cohn condition from Lemma 3.3 gives
∣ a 0 , n k a 2 , n k − a 1 , n k ∣ < 1 − a 0 , n k 2 . |a_{0,n_k}a_{2,n_k}-a_{1,n_k}|<1-a_{0,n_k}^{2}. ∣ a 0 , n k a 2 , n k − a 1 , n k ∣ < 1 − a 0 , n k 2 .
With a direct calculation, this is equivalent to ∣ β n k ( ω n k − 1 ) − Δ n k ∣ < 1 − ω n k 2 |\beta_{n_k}(\omega_{n_k}-1)-\Delta_{n_k}|<1-\omega_{n_k}^{2} ∣ β n k ( ω n k − 1 ) − Δ n k ∣ < 1 − ω n k 2 . Passing to the limit as k → ∞ k\to\infty k → ∞ , and using β ‾ = 1 \overline{\beta}=1 β = 1 , gives
∣ ω ‾ − 1 − Δ ‾ ∣ ≤ 1 − ω ‾ 2 . (88) |\overline{\omega}-1-\overline{\Delta}|\leq1-\overline{\omega}^{2}. \tag*{(88)} ∣ ω − 1 − Δ ∣ ≤ 1 − ω 2 . (88)
This bound and (87) imply
0 ≤ ω ‾ ≤ 1 , Δ ‾ ≥ 0. 0\leq\overline{\omega}\leq1,\qquad\overline{\Delta}\geq0. 0 ≤ ω ≤ 1 , Δ ≥ 0.
Thus, ω ‾ − 1 − Δ ‾ ≤ 0 \overline{\omega}-1-\overline{\Delta}\leq0 ω − 1 − Δ ≤ 0 , and hence ∣ ω ‾ − 1 − Δ ‾ ∣ = 1 + Δ ‾ − ω ‾ |\overline{\omega}-1-\overline{\Delta}|=1+\overline{\Delta}-\overline{\omega} ∣ ω − 1 − Δ ∣ = 1 + Δ − ω . It follows from (88) that 1 + Δ ‾ − ω ‾ ≤ 1 − ω ‾ 2 1+\overline{\Delta}-\overline{\omega}\leq1-\overline{\omega}^{2} 1 + Δ − ω ≤ 1 − ω 2 , or equivalently,
Δ ‾ ≤ ω ‾ − ω ‾ 2 = ω ‾ ( 1 − ω ‾ ) ≤ 1 4 . \overline{\Delta}\leq\overline{\omega}-\overline{\omega}^{2}=\overline{\omega}(1-\overline{\omega})\leq\frac{1}{4}. Δ ≤ ω − ω 2 = ω ( 1 − ω ) ≤ 4 1 .
Combining this with the other bound obtained in (87) gives
γ 2 ≤ Δ ‾ ≤ 1 4 and therefore γ = lim sup q ↓ 0 G ( q ; θ q ) ≤ 1 2 . \gamma^{2}\leq\overline{\Delta}\leq\frac{1}{4}\qquad\text{and therefore}\qquad\gamma=\limsup_{q\downarrow0}G(q;\theta_q)\leq\frac{1}{2}. γ 2 ≤ Δ ≤ 4 1 and therefore γ = q ↓ 0 lim sup G ( q ; θ q ) ≤ 2 1 .
Step 3: Achieving the optimal γ = 1 / 2 \gamma=1/2 γ = 1/2 . Proposition 3.19 provides a particular family of tunings for which ρ ( q ; θ q ) = 1 − q 2 \rho(q;\theta_q)=1-\frac{q}{2} ρ ( q ; θ q ) = 1 − 2 q . For this family, G ( q ; θ q ) = 1 2 G(q;\theta_q)=\frac{1}{2} G ( q ; θ q ) = 2 1 for every q q q , proving that the constant 1 / 2 1/2 1/2 is sharp.
We next argue that achieving γ = 1 / 2 \gamma= 1/2 γ = 1/2 implies that β q , Δ q , ω q \beta_q,\Delta_q,\omega_q β q , Δ q , ω q goes to a particular limit. Suppose that G ( q ; θ q ) ⟶ 1 2 G(q;\theta_q)\longrightarrow\frac{1}{2} G ( q ; θ q ) ⟶ 2 1 as q ↓ 0 q\downarrow0 q ↓ 0 , and define
ω q : = Δ q ν q . \omega_q := \Delta_q\nu_q. ω q := Δ q ν q .
Let { q ^ m } m ≥ 1 \{\widehat{q}_m\}_{m\ge1} { q m } m ≥ 1 be an arbitrary sequence satisfying q ^ m ⟶ 0 \widehat{q}_m\longrightarrow0 q m ⟶ 0 . The large-curvature endpoint estimates established above show that the sequence { ( β q ^ m , Δ q ^ m , ω q ^ m ) } m ≥ 1 \{(\beta_{\widehat{q}_m},\Delta_{\widehat{q}_m},\omega_{\widehat{q}_m})\}_{m\ge1} {( β q m , Δ q m , ω q m ) } m ≥ 1 is bounded in R 3 \mathbb{R}^3 R 3 . Therefore, there exists a strictly increasing sequence of indices m 1 < m 2 < ⋯ m_1 < m_2 < \cdots m 1 < m 2 < ⋯ and limits β ~ , Δ ~ , ω ~ \widetilde{\beta},\widetilde{\Delta},\widetilde{\omega} β , Δ , ω such that
β q ^ m j ⟶ β ~ , Δ q ^ m j ⟶ Δ ~ , ω q ^ m j ⟶ ω ~ . \beta_{\widehat{q}_{m_j}}\longrightarrow\widetilde{\beta},\qquad\Delta_{\widehat{q}_{m_j}}\longrightarrow\widetilde{\Delta},\qquad\omega_{\widehat{q}_{m_j}}\longrightarrow\widetilde{\omega}. β q m j ⟶ β , Δ q m j ⟶ Δ , ω q m j ⟶ ω .
Since G ( q ^ m j ; θ q ^ m j ) ⟶ 1 2 G(\widehat{q}_{m_j};\theta_{\widehat{q}_{m_j}})\longrightarrow\frac{1}{2} G ( q m j ; θ q m j ) ⟶ 2 1 , the endpoint argument proved above applies to this subsequence with γ = 1 / 2 \gamma=1/2 γ = 1/2 . It gives
β ~ = 1 , \widetilde{\beta}=1, β = 1 ,
and
γ 2 = 1 4 ≤ Δ ~ ≤ ω ~ ( 1 − ω ~ ) ≤ 1 4 . \gamma^2=\frac{1}{4}\le\widetilde{\Delta}\le\widetilde{\omega}(1-\widetilde{\omega})\le\frac{1}{4}. γ 2 = 4 1 ≤ Δ ≤ ω ( 1 − ω ) ≤ 4 1 .
Therefore,
Δ ~ = 1 4 , ω ~ = 1 2 , β ~ = 1. \widetilde{\Delta}=\frac{1}{4},\qquad\widetilde{\omega}=\frac{1}{2},\qquad\widetilde{\beta}=1. Δ = 4 1 , ω = 2 1 , β = 1.
Since the original sequence { q ^ m } \{\widehat{q}_m\} { q m } was arbitrary, the preceding argument shows that every cluster point of
( β q , Δ q , ω q ) (\beta_q,\Delta_q,\omega_q) ( β q , Δ q , ω q )
as q ↓ 0 q\downarrow0 q ↓ 0 equals ( 1 , 1 4 , 1 2 ) (1,\frac{1}{4},\frac{1}{2}) ( 1 , 4 1 , 2 1 ) . Together with the boundedness established in Step 1, this implies
β q ⟶ 1 , Δ q ⟶ 1 4 , ω q ⟶ 1 2 . \beta_q\longrightarrow1,\qquad\Delta_q\longrightarrow\frac{1}{4},\qquad\omega_q\longrightarrow\frac{1}{2}. β q ⟶ 1 , Δ q ⟶ 4 1 , ω q ⟶ 2 1 .
Since ω q = Δ q ν q \omega_q=\Delta_q\nu_q ω q = Δ q ν q and Δ q ⟶ 1 4 > 0 \Delta_q\longrightarrow\frac{1}{4}>0 Δ q ⟶ 4 1 > 0 , we obtain ν q = ω q Δ q ⟶ 2 \nu_q=\frac{\omega_q}{\Delta_q}\longrightarrow2 ν q = Δ q ω q ⟶ 2 .
Step 4: Taylor expansion of β q \beta_q β q in q q q . It remains to determine the first-order behavior of β q \beta_q β q . At the lower endpoint associated with t = q 2 t=q^2 t = q 2 ,
P q , q 2 ( z ) = z ( z − 1 ) ( z − β ) + q 2 Δ q ( ( 1 + ν q ) z − ν q ) ⟶ z ( z − 1 ) 2 , P_{q,q^2}(z)=z(z-1)(z-\beta)+q^2\Delta_q((1+\nu_q)z-\nu_q)\longrightarrow z(z-1)^2, P q , q 2 ( z ) = z ( z − 1 ) ( z − β ) + q 2 Δ q (( 1 + ν q ) z − ν q ) ⟶ z ( z − 1 ) 2 ,
coefficientwise as q ↓ 0 q\downarrow0 q ↓ 0 . Hence, for all sufficiently small q q q , its roots may be labeled as r 0 , q r_{0,q} r 0 , q , r 1 , q r_{1,q} r 1 , q and r 2 , q r_{2,q} r 2 , q so that
r 0 , q ⟶ 0 , r 1 , q ⟶ 1 , r 2 , q ⟶ 1. r_{0,q}\longrightarrow0,\qquad r_{1,q}\longrightarrow1,\qquad r_{2,q}\longrightarrow1. r 0 , q ⟶ 0 , r 1 , q ⟶ 1 , r 2 , q ⟶ 1.
For i = 1 , 2 i=1,2 i = 1 , 2 , define
u i , q : = 1 − r i , q q , u_{i,q}:=\frac{1-r_{i,q}}{q}, u i , q := q 1 − r i , q ,
so that r i , q = 1 − q u i , q r_{i,q}=1-qu_{i,q} r i , q = 1 − q u i , q . Evaluating P q , q 2 P_{q,q^2} P q , q 2 at z = 1 z=1 z = 1 gives
q 2 Δ q = ( 1 − r 0 , q ) ( 1 − r 1 , q ) ( 1 − r 2 , q ) . q^2\Delta_q=(1-r_{0,q})(1-r_{1,q})(1-r_{2,q}). q 2 Δ q = ( 1 − r 0 , q ) ( 1 − r 1 , q ) ( 1 − r 2 , q ) .
Since 1 − r i , q = q u i , q 1-r_{i,q}=qu_{i,q} 1 − r i , q = q u i , q for i = 1 , 2 i=1,2 i = 1 , 2 , we obtain Δ q = ( 1 − r 0 , q ) u 1 , q u 2 , q \Delta_q=(1-r_{0,q})u_{1,q}u_{2,q} Δ q = ( 1 − r 0 , q ) u 1 , q u 2 , q . Using Δ q ⟶ 1 4 \Delta_q\longrightarrow\frac{1}{4} Δ q ⟶ 4 1 and r 0 , q ⟶ 0 r_{0,q}\longrightarrow0 r 0 , q ⟶ 0 , we conclude that
u 1 , q u 2 , q ⟶ 1 4 . (89) u_{1,q}u_{2,q}\longrightarrow\frac{1}{4}. \tag*{(89)} u 1 , q u 2 , q ⟶ 4 1 . (89)
Moreover, by assumption, ρ ( q ; θ q ) = 1 − q G ( q ; θ q ) = 1 − q 2 + o ( q ) \rho(q;\theta_q)=1-qG(q;\theta_q)=1-\frac{q}{2}+o(q) ρ ( q ; θ q ) = 1 − q G ( q ; θ q ) = 1 − 2 q + o ( q ) . Therefore, for i = 1 , 2 i=1,2 i = 1 , 2 ,
∣ 1 − q u i , q ∣ = ∣ r i , q ∣ ≤ ρ ( q ; θ q ) = 1 − q 2 + o ( q ) . |1-qu_{i,q}|=|r_{i,q}|\leq\rho(q;\theta_q)=1-\frac{q}{2}+o(q). ∣1 − q u i , q ∣ = ∣ r i , q ∣ ≤ ρ ( q ; θ q ) = 1 − 2 q + o ( q ) .
Squaring gives
1 − 2 q Re ( u i , q ) + q 2 ∣ u i , q ∣ 2 ≤ 1 − q + o ( q ) . 1-2q\operatorname{Re}(u_{i,q})+q^2|u_{i,q}|^2\leq1-q+o(q). 1 − 2 q Re ( u i , q ) + q 2 ∣ u i , q ∣ 2 ≤ 1 − q + o ( q ) .
After subtracting 1 1 1 and dividing by q > 0 q>0 q > 0 , we obtain 2 Re ( u i , q ) − q ∣ u i , q ∣ 2 ≥ 1 + o ( 1 ) 2\operatorname{Re}(u_{i,q})-q|u_{i,q}|^2\geq1+o(1) 2 Re ( u i , q ) − q ∣ u i , q ∣ 2 ≥ 1 + o ( 1 ) . In particular,
Re ( u i , q ) ≥ 1 2 + o ( 1 ) , i = 1 , 2. (90) \operatorname{Re}(u_{i,q})\geq\frac{1}{2}+o(1),\qquad i=1,2. \tag*{(90)} Re ( u i , q ) ≥ 2 1 + o ( 1 ) , i = 1 , 2. (90)
Because P q , q 2 P_{q,q^2} P q , q 2 has real coefficients and r 0 , q r_{0,q} r 0 , q is the unique root near zero, the roots r 1 , q r_{1,q} r 1 , q and r 2 , q r_{2,q} r 2 , q are either both real or form a complex-conjugate pair. We now show that in either case
u 1 , q ⟶ 1 2 and u 2 , q ⟶ 1 2 as q ↓ 0. u_{1,q}\longrightarrow\frac{1}{2}\quad\text{and}\quad u_{2,q}\longrightarrow\frac{1}{2}\quad\text{as }q\downarrow0. u 1 , q ⟶ 2 1 and u 2 , q ⟶ 2 1 as q ↓ 0.
Suppose otherwise. Then there exist ϵ > 0 \epsilon>0 ϵ > 0 and a sequence q n ↓ 0 q_n\downarrow0 q n ↓ 0 such that
max i = 1 , 2 ∣ u i , q n − 1 2 ∣ ≥ ϵ for all n . \max_{i=1,2}\left|u_{i,q_n}-\frac{1}{2}\right|\geq\epsilon\qquad\text{for all }n. i = 1 , 2 max u i , q n − 2 1 ≥ ϵ for all n .
Passing to a subsequence, we may assume that r 1 , q n , r 2 , q n r_{1,q_n},r_{2,q_n} r 1 , q n , r 2 , q n are either both real for every n n n , or form a complex-conjugate pair for every n n n . In the complex-conjugate case, from (89) ,
u 2 , q n = u 1 , q n ‾ , ∣ u 1 , q n ∣ 2 = u 1 , q n u 2 , q n ⟶ 1 4 . u_{2,q_n}=\overline{u_{1,q_n}},\qquad|u_{1,q_n}|^2=u_{1,q_n}u_{2,q_n}\longrightarrow\frac{1}{4}. u 2 , q n = u 1 , q n , ∣ u 1 , q n ∣ 2 = u 1 , q n u 2 , q n ⟶ 4 1 .
Hence ∣ u 1 , q n ∣ → 1 / 2 |u_{1,q_n}|\to1/2 ∣ u 1 , q n ∣ → 1/2 . Since Re ( u 1 , q n ) ≥ 1 2 + o ( 1 ) \operatorname{Re}(u_{1,q_n})\geq\frac{1}{2}+o(1) Re ( u 1 , q n ) ≥ 2 1 + o ( 1 ) by (90) and Re ( u 1 , q n ) ≤ ∣ u 1 , q n ∣ \operatorname{Re}(u_{1,q_n})\leq|u_{1,q_n}| Re ( u 1 , q n ) ≤ ∣ u 1 , q n ∣ , we obtain Re ( u 1 , q n ) → 1 / 2 \operatorname{Re}(u_{1,q_n})\to1/2 Re ( u 1 , q n ) → 1/2 , and therefore Im ( u 1 , q n ) → 0 \operatorname{Im}(u_{1,q_n})\to0 Im ( u 1 , q n ) → 0 . Thus u 1 , q n , u 2 , q n → 1 / 2 u_{1,q_n},u_{2,q_n}\to1/2 u 1 , q n , u 2 , q n → 1/2 . In the real-root case, from (90) and (89) , we have
u i , q n ≥ 1 2 + o ( 1 ) , i = 1 , 2 , and u 1 , q n u 2 , q n ⟶ 1 4 . u_{i,q_n}\geq\frac{1}{2}+o(1),\qquad i=1,2,\qquad\text{and}\qquad u_{1,q_n}u_{2,q_n}\longrightarrow\frac{1}{4}. u i , q n ≥ 2 1 + o ( 1 ) , i = 1 , 2 , and u 1 , q n u 2 , q n ⟶ 4 1 .
These relations force u 1 , q n , u 2 , q n → 1 / 2 u_{1,q_n},u_{2,q_n}\to1/2 u 1 , q n , u 2 , q n → 1/2 . Hence, regardless of roots being complex or real, we have
u 1 , q , u 2 , q ⟶ 1 2 and u 1 , q + u 2 , q ⟶ 1. (91) u_{1,q},u_{2,q}\longrightarrow\frac{1}{2}\qquad\text{and}\qquad u_{1,q}+u_{2,q}\longrightarrow1. \tag*{(91)} u 1 , q , u 2 , q ⟶ 2 1 and u 1 , q + u 2 , q ⟶ 1. (91)
By Vieta’s formula for the sum of the roots of P q , q 2 P_{q,q^2} P q , q 2 , r 0 , q + r 1 , q + r 2 , q = 1 + β q r_{0,q}+r_{1,q}+r_{2,q}=1+\beta_q r 0 , q + r 1 , q + r 2 , q = 1 + β q . Substituting r 1 , q = 1 − q u 1 , q r_{1,q}=1-qu_{1,q} r 1 , q = 1 − q u 1 , q , and r 2 , q = 1 − q u 2 , q r_{2,q}=1-qu_{2,q} r 2 , q = 1 − q u 2 , q , gives
1 − β q q = u 1 , q + u 2 , q − r 0 , q q . \frac{1-\beta_q}{q}=u_{1,q}+u_{2,q}-\frac{r_{0,q}}{q}. q 1 − β q = u 1 , q + u 2 , q − q r 0 , q .
On the other hand, the product-of-roots identity applied to the polynomial P q , q 2 P_{q,q^2} P q , q 2 gives r 0 , q r 1 , q r 2 , q = q 2 ω q r_{0,q}r_{1,q}r_{2,q}=q^2\omega_q r 0 , q r 1 , q r 2 , q = q 2 ω q . Since r 1 , q r 2 , q → 1 r_{1,q}r_{2,q}\to1 r 1 , q r 2 , q → 1 and ω q \omega_q ω q is bounded, it follows that r 0 , q = q 2 ω q r 1 , q r 2 , q = O ( q 2 ) r_{0,q}=\frac{q^2\omega_q}{r_{1,q}r_{2,q}}=O(q^2) r 0 , q = r 1 , q r 2 , q q 2 ω q = O ( q 2 ) . Therefore, r 0 , q q → 0 \frac{r_{0,q}}{q}\to0 q r 0 , q → 0 . Consequently, from (91) , we obtain 1 − β q q → 1 \frac{1-\beta_q}{q}\to1 q 1 − β q → 1 , which is equivalent to β q = 1 − q + o ( q ) \beta_q=1-q+o(q) β q = 1 − q + o ( q ) . Finally, since Δ q = α q L \Delta_q=\alpha_qL Δ q = α q L , we conclude that
α q L = 1 4 + o ( 1 ) , β q = 1 − q + o ( q ) , ν q = 2 + o ( 1 ) , \alpha_qL=\frac{1}{4}+o(1),\qquad\beta_q=1-q+o(q),\qquad\nu_q=2+o(1), α q L = 4 1 + o ( 1 ) , β q = 1 − q + o ( q ) , ν q = 2 + o ( 1 ) ,
which completes the proof.
Acceleration with GMM for arbitrary delays τ ≥ 1 \tau\ge1 τ ≥ 1
Recall that
P τ , δ ( z ) = z τ ( z − 1 ) ( z − β ) + δ [ ( 1 + ν ) z − ν ] . (92) P_{\tau,\delta}(z)=z^\tau(z-1)(z-\beta)+\delta[(1+\nu)z-\nu]. \tag*{(92)} P τ , δ ( z ) = z τ ( z − 1 ) ( z − β ) + δ [( 1 + ν ) z − ν ] . (92)
The worst-case spectral radius for the quadratic setting is
ρ τ , κ ( α , β , ν ) : = sup t ∈ [ q 2 , 1 ] max { ∣ z ∣ : P τ , t δ L ( z ) = 0 } . (93) \rho_{\tau,\kappa}(\alpha,\beta,\nu):=\sup_{t\in[q^2,1]}\max\{|z|:P_{\tau,t\delta_L}(z)=0\}. \tag*{(93)} ρ τ , κ ( α , β , ν ) := t ∈ [ q 2 , 1 ] sup max { ∣ z ∣ : P τ , t δ L ( z ) = 0 } . (93)
Acceleration means that, for fixed τ ∈ Z + \tau\in\mathbb{Z}_+ τ ∈ Z + ,
ρ τ , κ ≤ 1 − c ( τ ) κ − 1 / 2 \rho_{\tau,\kappa}\le1-c(\tau)\kappa^{-1/2} ρ τ , κ ≤ 1 − c ( τ ) κ − 1/2
holds for all sufficiently large κ > 0 \kappa>0 κ > 0 with c ( τ ) > 0 c(\tau)>0 c ( τ ) > 0 independent of κ \kappa κ . For δ = 0 \delta=0 δ = 0 and β \beta β near 1 1 1 , P τ , δ P_{\tau,\delta} P τ , δ has τ \tau τ roots near 0 0 0 and two roots near 1 1 1 ; therefore, the worst-case spectral radius is governed by the roots near 1 1 1 . As before, we refer to the roots of P τ , δ ( z ) P_{\tau,\delta}(z) P τ , δ ( z ) that stay close to 1 1 1 as the slow modes , and those near zero as the fast modes , or nonbinding roots .
Slow-mode asymptotics
The following result shows that the slow mode admits an expansion as a function of q : = κ − 1 / 2 q:=\kappa^{-1/2} q := κ − 1/2 , where the delay τ \tau τ does not affect the first-order terms, but it affects only the higher order terms.
Lemma 3.21 (Slow-mode asymptotics). Fix τ ≥ 1 \tau\ge1 τ ≥ 1 , L L L , δ L > 0 \delta_L>0 δ L > 0 , and ν ≥ 0 \nu\ge0 ν ≥ 0 , and let { μ n } \{\mu_n\} { μ n } be a sequence such that μ n ↘ 0 \mu_n\searrow0 μ n ↘ 0 . For κ n : = L / μ n \kappa_n:=L/\mu_n κ n := L / μ n , let q n : = κ n − 1 / 2 q_n:=\kappa_n^{-1/2} q n := κ n − 1/2 , and we set
a : = δ L , β n : = 1 − 2 a q n . a:=\sqrt{\delta_L},\qquad\beta_n:=1-2a q_n. a := δ L , β n := 1 − 2 a q n . Let t n = q n 2 ℓ n t_n=q_n^2\ell_n t n = q n 2 ℓ n for some ℓ n \ell_n ℓ n such that 1 ≤ ℓ n ≤ B 1\le\ell_n\le B 1 ≤ ℓ n ≤ B for some fixed B < ∞ B<\infty B < ∞ . Consider the characteristic polynomial of GMM with δ = t n δ L \delta=t_n\delta_L δ = t n δ L and β n = 1 − 2 δ L q n \beta_n=1-2\sqrt{\delta_L}q_n β n = 1 − 2 δ L q n for a fixed ν \nu ν , i.e.
P n ( z ) : = P τ , t n δ L ( z ) = z τ ( z − 1 ) ( z − β n ) + t n δ L [ ( 1 + ν ) z − ν ] . (94) P_n(z):=P_{\tau,t_n\delta_L}(z)=z^\tau(z-1)(z-\beta_n)+t_n\delta_L[(1+\nu)z-\nu]. \tag*{(94)} P n ( z ) := P τ , t n δ L ( z ) = z τ ( z − 1 ) ( z − β n ) + t n δ L [( 1 + ν ) z − ν ] . (94) Then, for all sufficiently large n n n , P n P_n P n has exactly two roots z n ± z_n^\pm z n ± converging to 1 1 1 . Indeed, for y n ± y_n^\pm y n ± such that z n ± = 1 + q n y n ± z_n^\pm=1+q_n y_n^\pm z n ± = 1 + q n y n ± , one has
dist ( y n ± , { − a + i a ℓ n − 1 , − a − i a ℓ n − 1 } ) = O B ( q n 1 / 2 ) , (95) \operatorname{dist}\left(y_n^\pm,\{-a+ia\sqrt{\ell_n-1},-a-ia\sqrt{\ell_n-1}\}\right)=O_B(q_n^{1/2}), \tag*{(95)} dist ( y n ± , { − a + ia ℓ n − 1 , − a − ia ℓ n − 1 } ) = O B ( q n 1/2 ) , (95) where the O ( ⋅ ) O(\cdot) O ( ⋅ ) constants are also allowed to depend on the fixed parameters τ \tau τ , δ L \delta_L δ L and ν \nu ν . Consequently,
∣ z n ± ∣ = 1 − a q n + O B ( q n 3 / 2 ) = 1 − δ L q n + o ( q n ) . (96) |z_n^\pm|=1-aq_n+O_B(q_n^{3/2})=1-\sqrt{\delta_L}\,q_n+o(q_n). \tag*{(96)} ∣ z n ± ∣ = 1 − a q n + O B ( q n 3/2 ) = 1 − δ L q n + o ( q n ) . (96) The remaining τ \tau τ roots of P n P_n P n converge to 0 0 0 and are nonbinding for the spectral radius.
Proof. Set z = 1 + q n w z=1+q_nw z = 1 + q n w . Then, we have
z − 1 = q n w , z − β n = q n ( w + 2 a ) . z-1=q_nw,\qquad z-\beta_n=q_n(w+2a). z − 1 = q n w , z − β n = q n ( w + 2 a ) .
Since t n = q n 2 ℓ n t_n=q_n^2\ell_n t n = q n 2 ℓ n and δ L = a 2 \delta_L=a^2 δ L = a 2 , we also have
t n δ L [ ( 1 + ν ) z − ν ] = q n 2 ℓ n a 2 [ 1 + ( 1 + ν ) q n w ] . t_n\delta_L[(1+\nu)z-\nu]=q_n^2\ell_n a^2[1+(1+\nu)q_nw]. t n δ L [( 1 + ν ) z − ν ] = q n 2 ℓ n a 2 [ 1 + ( 1 + ν ) q n w ] .
After division by q n 2 q_n^2 q n 2 , the rescaled polynomial is
F n ( w ) : = q n − 2 P n ( 1 + q n w ) = ( 1 + q n w ) τ w ( w + 2 a ) + ℓ n a 2 [ 1 + ( 1 + ν ) q n w ] . (97) F_n(w) := q_n^{-2}P_n(1+q_nw) = (1+q_nw)^\tau w(w+2a) + \ell_n a^2[1+(1+\nu)q_nw]. \tag*{(97)} F n ( w ) := q n − 2 P n ( 1 + q n w ) = ( 1 + q n w ) τ w ( w + 2 a ) + ℓ n a 2 [ 1 + ( 1 + ν ) q n w ] . (97)
Compare F n F_n F n with the moving quadratic
Q n ( w ) : = w ( w + 2 a ) + ℓ n a 2 = ( w + a ) 2 + a 2 ( ℓ n − 1 ) . (98) Q_n(w) := w(w+2a)+\ell_n a^2 = (w+a)^2+a^2(\ell_n-1). \tag*{(98)} Q n ( w ) := w ( w + 2 a ) + ℓ n a 2 = ( w + a ) 2 + a 2 ( ℓ n − 1 ) . (98)
The roots of Q n Q_n Q n are
r n ± = − a ± i a ℓ n − 1 . (99) r_n^\pm=-a\pm ia\sqrt{\ell_n-1}. \tag*{(99)} r n ± = − a ± ia ℓ n − 1 . (99)
Since 1 ≤ ℓ n ≤ B 1\leq\ell_n\leq B 1 ≤ ℓ n ≤ B , the roots r n ± r_n^\pm r n ± remain in a fixed bounded subset of the w w w -plane; indeed, ∣ r n ± ∣ = a ℓ n ≤ a B \lvert r_n^\pm\rvert=a\sqrt{\ell_n}\leq a\sqrt{B} ∣ r n ± ∣ = a ℓ n ≤ a B . Choose R = 3 a B R=3a\sqrt{B} R = 3 a B so that ∣ r n ± ∣ < R / 2 \lvert r_n^\pm\rvert<R/2 ∣ r n ± ∣ < R /2 for every n n n . On the circle { w : ∣ w ∣ = R } \{w:\lvert w\rvert=R\} { w : ∣ w ∣ = R } , the quadratics Q n Q_n Q n are bounded away from zero uniformly in n n n , i.e., ∣ Q n ( w ) ∣ = ∣ w − r n + ∣ ∣ w − r n − ∣ ≥ m R > 0 \lvert Q_n(w)\rvert=\lvert w-r_n^+\rvert\lvert w-r_n^-\rvert\geq m_R>0 ∣ Q n ( w )∣ = ∣ w − r n + ∣ ∣ w − r n − ∣ ≥ m R > 0 for m R : = ( R − a B ) 2 m_R:=(R-a\sqrt{B})^2 m R := ( R − a B ) 2 . Moreover, for ∣ w ∣ ≤ R \lvert w\rvert\leq R ∣ w ∣ ≤ R , uniformly in w w w , we have
F n ( w ) − Q n ( w ) = ( ( 1 + q n w ) τ − 1 ) w ( w + 2 a ) + ℓ n a 2 ( 1 + ν ) q n w = O B ( q n ) , F_n(w)-Q_n(w)=\bigl((1+q_nw)^\tau-1\bigr)w(w+2a)+\ell_na^2(1+\nu)q_nw=O_B(q_n), F n ( w ) − Q n ( w ) = ( ( 1 + q n w ) τ − 1 ) w ( w + 2 a ) + ℓ n a 2 ( 1 + ν ) q n w = O B ( q n ) ,
in the last equality we use the fact that R = 3 a B R=3a\sqrt{B} R = 3 a B . Note that here the constants hidden by O ( ⋅ ) O(\cdot) O ( ⋅ ) are allowed to depend on the fixed parameters τ \tau τ , δ L \delta_L δ L and ν \nu ν . Therefore, ∣ F n − Q n ∣ < m R \lvert F_n-Q_n\rvert<m_R ∣ F n − Q n ∣ < m R on ∣ w ∣ = R \lvert w\rvert=R ∣ w ∣ = R for all large n n n as q n ↘ 0 q_n\searrow0 q n ↘ 0 as n → ∞ n\to\infty n → ∞ . By Rouche’s theorem, F n F_n F n has exactly two roots in { w : ∣ w ∣ < R } \{w:\lvert w\rvert<R\} { w : ∣ w ∣ < R } . These two roots correspond exactly to the two roots of P n P_n P n in the disk ∣ z − 1 ∣ < q n R \lvert z-1\rvert<q_nR ∣ z − 1 ∣ < q n R ; in particular, they converge to 1 1 1 in the original z z z -variable. Denote them by z n ± = 1 + q n y n ± z_n^\pm=1+q_ny_n^\pm z n ± = 1 + q n y n ± .
For each such bounded root y n ± y_n^\pm y n ± , we have F n ( y n ± ) = 0 F_n(y_n^\pm)=0 F n ( y n ± ) = 0 . Therefore, using the uniform estimate above, we get
∣ Q n ( y n ± ) ∣ = ∣ Q n ( y n ± ) − F n ( y n ± ) ∣ = O B ( q n ) . \lvert Q_n(y_n^\pm)\rvert=\lvert Q_n(y_n^\pm)-F_n(y_n^\pm)\rvert=O_B(q_n). ∣ Q n ( y n ± )∣ = ∣ Q n ( y n ± ) − F n ( y n ± )∣ = O B ( q n ) .
Since Q n ( w ) = ( w − r n + ) ( w − r n − ) Q_n(w)=(w-r_n^+)(w-r_n^-) Q n ( w ) = ( w − r n + ) ( w − r n − ) , we get
∣ Q n ( y n ± ) ∣ = ∣ y n ± − r n + ∣ ∣ y n ± − r n − ∣ = O B ( q n ) . \lvert Q_n(y_n^\pm)\rvert=\lvert y_n^\pm-r_n^+\rvert\lvert y_n^\pm-r_n^-\rvert=O_B(q_n). ∣ Q n ( y n ± )∣ = ∣ y n ± − r n + ∣ ∣ y n ± − r n − ∣ = O B ( q n ) .
Moreover, since dist ( y n ± , { r n + , r n − } ) 2 ≤ ∣ y n ± − r n + ∣ ∣ y n ± − r n − ∣ \operatorname{dist}(y_n^\pm,\{r_n^+,r_n^-\})^2\leq\lvert y_n^\pm-r_n^+\rvert\lvert y_n^\pm-r_n^-\rvert dist ( y n ± , { r n + , r n − } ) 2 ≤ ∣ y n ± − r n + ∣ ∣ y n ± − r n − ∣ , we have
dist ( y n ± , { r n + , r n − } ) = O B ( q n 1 / 2 ) , \operatorname{dist}(y_n^\pm,\{r_n^+,r_n^-\})=O_B(q_n^{1/2}), dist ( y n ± , { r n + , r n − }) = O B ( q n 1/2 ) ,
which proves (95) .
Note that both roots r n ± r_n^\pm r n ± have real part − a -a − a ; hence,
Re y n ± = − a + O B ( q n 1 / 2 ) , y n ± = O B ( 1 ) . \operatorname{Re}y_n^\pm=-a+O_B(q_n^{1/2}),\qquad y_n^\pm=O_B(1). Re y n ± = − a + O B ( q n 1/2 ) , y n ± = O B ( 1 ) .
Using z n ± = 1 + q n y n ± z_n^\pm=1+q_ny_n^\pm z n ± = 1 + q n y n ± , we obtain
∣ z n ± ∣ 2 = ∣ 1 + q n y n ± ∣ 2 = 1 + 2 q n Re y n ± + q n 2 ∣ y n ± ∣ 2 = 1 − 2 a q n + O B ( q n 3 / 2 ) . \lvert z_n^\pm\rvert^2=\lvert1+q_ny_n^\pm\rvert^2=1+2q_n\operatorname{Re}y_n^\pm+q_n^2\lvert y_n^\pm\rvert^2=1-2aq_n+O_B(q_n^{3/2}). ∣ z n ± ∣ 2 = ∣ 1 + q n y n ± ∣ 2 = 1 + 2 q n Re y n ± + q n 2 ∣ y n ± ∣ 2 = 1 − 2 a q n + O B ( q n 3/2 ) .
Taking square roots and using 1 + u = 1 + 1 2 u + O ( u 2 ) \sqrt{1+u}=1+\frac{1}{2}u+O(u^2) 1 + u = 1 + 2 1 u + O ( u 2 ) as u → 0 u\to0 u → 0 gives
∣ z n ± ∣ = 1 − a q n + O B ( q n 3 / 2 ) , \lvert z_n^\pm\rvert=1-aq_n+O_B(q_n^{3/2}), ∣ z n ± ∣ = 1 − a q n + O B ( q n 3/2 ) ,
which is (96) .
It remains to identify the nonbinding roots. Since q n → 0 q_n\to0 q n → 0 , β n → 1 \beta_n\to1 β n → 1 , and t n = q n 2 ℓ n → 0 t_n=q_n^2\ell_n\to0 t n = q n 2 ℓ n → 0 , the coefficientwise limit of P n P_n P n is z τ ( z − 1 ) 2 z^\tau(z-1)^2 z τ ( z − 1 ) 2 . By continuity of roots as a multiset, exactly τ \tau τ roots of P n P_n P n converge to 0 0 0 ; thus, they are non-binding for the spectral radius, while the remaining two converge to 1 1 1 . This accounts for all τ + 2 \tau+2 τ + 2 roots and completes the proof.
If we choose δ L = c ~ / ( 1 + τ ) 2 \delta_L=\tilde{c}/(1+\tau)^2 δ L = c ~ / ( 1 + τ ) 2 , where c ~ > 0 \tilde{c}>0 c ~ > 0 is fixed as q = 1 / κ → 0 q=1/\sqrt{\kappa}\to0 q = 1/ κ → 0 , then the previous lemma (Lemma 3.21) suggests that at t = q 2 t=q^2 t = q 2 (for ℓ n = 1 \ell_n=1 ℓ n = 1 , ∀ n \forall n ∀ n ) the near 1-roots (slow modes) demonstrate an accelerated behavior, i.e., we have
ρ ( P τ , q 2 δ L ) = 1 − δ L q + o ( q ) = 1 − Θ ( 1 ( 1 + τ ) κ ) as q → 0 for fixed τ . \rho(P_{\tau,q^2\delta_L})=1-\sqrt{\delta_L}\,q+o(q)=1-\Theta\left(\frac{1}{(1+\tau)\sqrt{\kappa}}\right)\quad\text{as }q\to0\text{ for fixed }\tau. ρ ( P τ , q 2 δ L ) = 1 − δ L q + o ( q ) = 1 − Θ ( ( 1 + τ ) κ 1 ) as q → 0 for fixed τ .
This is consistent with the accelerated GMM construction for τ = 1 \tau=1 τ = 1 in Proposition 3.19 . There, however, the choice is ν = 2 + O ( q ) ≥ 1 \nu=2+O(q)\ge1 ν = 2 + O ( q ) ≥ 1 for τ = 1 \tau=1 τ = 1 , which is larger than the Nesterov-type choice ν = β < 1 \nu=\beta<1 ν = β < 1 . This suggests that larger values of ν \nu ν can be beneficial, at least for τ = 1 \tau=1 τ = 1 . Motivated by this observation, we consider the following choice of parameters:
α = δ L L , β = 1 − 2 δ L q , ν = ν τ , (100) \alpha=\frac{\delta_L}{L},\qquad\beta=1-2\sqrt{\delta_L}\,q,\qquad\nu=\nu_\tau, \tag*{(100)} α = L δ L , β = 1 − 2 δ L q , ν = ν τ , (100)
where δ L \delta_L δ L is independent of κ \kappa κ and ν τ \nu_\tau ν τ is chosen as a function of τ \tau τ . The closed-form choice in Theorem 3.25 is ν τ = 2 τ + 1 2 \nu_\tau=2\tau+\frac{1}{2} ν τ = 2 τ + 2 1 , where we show that under this ansatz, the slow modes yield a rate gap of order δ L q \sqrt{\delta_L}\,q δ L q for fixed τ ≥ 1 \tau\ge1 τ ≥ 1 . The remaining task is to stabilize the fast and intermediate modes introduced by the delay.
The choices of α \alpha α and β \beta β in (100) are also motivated by the parameter choice for the classical Nesterov acceleration. In the delay-free case τ = 0 \tau=0 τ = 0 , NAG with stepsize α = 1 / L \alpha=1/L α = 1/ L has the standard choice of parameters for strongly convex functions β = 1 − q 1 + q = 1 − 2 q + O ( q 2 ) \beta=\frac{1-q}{1+q}=1-2q+O(q^2) β = 1 + q 1 − q = 1 − 2 q + O ( q 2 ) , see, e.g., [39 (2.2.22)] . In addition, for a stepsize
α = δ L L , 0 < δ L ≤ 1 , \alpha=\frac{\delta_L}{L},\qquad0<\delta_L\le1, α = L δ L , 0 < δ L ≤ 1 ,
the corresponding optimal momentum parameter (that leads to the fastest linear rate) for delay-free NAG on quadratic objectives is known to be
β = 1 − α μ 1 + α μ = 1 − δ L q 1 + δ L q = 1 − 2 δ L q + O ( q 2 ) , \beta=\frac{1-\sqrt{\alpha\mu}}{1+\sqrt{\alpha\mu}}=\frac{1-\sqrt{\delta_L}\,q}{1+\sqrt{\delta_L}\,q}=1-2\sqrt{\delta_L}\,q+O(q^2), β = 1 + α μ 1 − α μ = 1 + δ L q 1 − δ L q = 1 − 2 δ L q + O ( q 2 ) ,
[1 Lemma 2.1] , [Gür]. Thus, the choices of α \alpha α and β \beta β in (100) agree with the delay-free NAG scaling to leading order in q q q . The extrapolation parameter ν τ \nu_\tau ν τ is then chosen separately to stabilize the effects of the delay.
Asymptotics of spectral radius
Letting q ↓ 0 q\downarrow0 q ↓ 0 sends β → 1 \beta\to1 β → 1 . The limiting characteristic polynomial is
P ∞ ( z ; δ , ν ) : = z τ ( z − 1 ) 2 + δ [ ( 1 + ν ) z − ν ] . (101) P_\infty(z;\delta,\nu):=z^\tau(z-1)^2+\delta[(1+\nu)z-\nu]. \tag*{(101)} P ∞ ( z ; δ , ν ) := z τ ( z − 1 ) 2 + δ [( 1 + ν ) z − ν ] . (101)
The roots of the characteristic polynomial converge to those of P ∞ P_\infty P ∞ as q → 0 q\to0 q → 0 . The next result shows that if P ∞ P_\infty P ∞ is strictly Schur stable for every 0 < δ ≤ δ L 0<\delta\le\delta_L 0 < δ ≤ δ L , then the spectral radius of GMM exhibits accelerated behavior: specifically, it is bounded above by 1 − c / κ 1-c/\sqrt{\kappa} 1 − c / κ for some constant c > 0 c>0 c > 0 , provided κ = 1 / q 2 \kappa=1/q^2 κ = 1/ q 2 is sufficiently large and the parameters α \alpha α and β \beta β are chosen according to (100) .
Theorem 3.22 (Asymptotics of spectral radius). Fix τ ≥ 1 \tau\ge1 τ ≥ 1 . Suppose δ L > 0 \delta_L>0 δ L > 0 and ν ≥ 0 \nu\ge0 ν ≥ 0 ensure that
P ∞ ( ⋅ ; δ , ν ) is strictly Schur stable for every 0 < δ ≤ δ L . (102) P_\infty(\cdot;\delta,\nu)\text{ is strictly Schur stable for every }0<\delta\le\delta_L. \tag*{(102)} P ∞ ( ⋅ ; δ , ν ) is strictly Schur stable for every 0 < δ ≤ δ L . (102)
Choose α = δ L / L \alpha= \delta_L/L α = δ L / L , β = 1 − 2 δ L q \beta= 1 - 2\sqrt{\delta_L}\,q β = 1 − 2 δ L q , and ν \nu ν as given. Then, we necessarily have ν > τ \nu> \tau ν > τ , and for every c < δ L c < \sqrt{\delta_L} c < δ L , there exists q 0 > 0 q_0 > 0 q 0 > 0 such that
ρ τ , κ ( α , β , ν ) ≤ 1 − c q , (103) \rho_{\tau,\kappa}(\alpha,\beta,\nu) \le1-cq, \tag*{(103)} ρ τ , κ ( α , β , ν ) ≤ 1 − c q , (103)
whenever 0 < q ≤ q 0 0 < q \le q_0 0 < q ≤ q 0 , where ρ τ , κ ( α , β , ν ) \rho_{\tau,\kappa}(\alpha,\beta,\nu) ρ τ , κ ( α , β , ν ) is defined in (93) .
Proof. We first show that the hypothesis in (102) implies that ν > τ \nu> \tau ν > τ . The only point requiring justification is the expansion of the roots that bifurcate from the double root z = 1 z = 1 z = 1 of P ∞ ( ⋅ ; 0 , ν ) P_\infty(\mathord{\cdot};0,\nu) P ∞ ( ⋅ ; 0 , ν ) . Let ϵ = δ \epsilon= \sqrt{\delta} ϵ = δ and z = 1 + ϵ w z = 1 + \epsilon w z = 1 + ϵ w , then
P ∞ ( 1 + ϵ w ; ϵ 2 , ν ) = ϵ 2 G ( w , ϵ ) , (104) P_\infty(1+\epsilon w;\epsilon^2,\nu)=\epsilon^2G(w,\epsilon), \tag*{(104)} P ∞ ( 1 + ϵ w ; ϵ 2 , ν ) = ϵ 2 G ( w , ϵ ) , (104)
where
G ( w , ϵ ) : = ( 1 + ϵ w ) τ w 2 + 1 + ( 1 + ν ) ϵ w . (105) G(w,\epsilon) := (1+\epsilon w)^\tau w^2 + 1 + (1+\nu)\epsilon w. \tag*{(105)} G ( w , ϵ ) := ( 1 + ϵ w ) τ w 2 + 1 + ( 1 + ν ) ϵ w . (105)
The function G G G is analytic in a neighborhood of each point ( ± i , 0 ) (\pm i,0) ( ± i , 0 ) , and
G ( w , 0 ) = w 2 + 1. G(w,0)=w^2+1. G ( w , 0 ) = w 2 + 1.
Since G ( ± i , 0 ) = 0 G(\pm i,0)=0 G ( ± i , 0 ) = 0 and ∂ w G ( ± i , 0 ) = ± 2 i ≠ 0 \partial_wG(\pm i,0)=\pm2i \ne0 ∂ w G ( ± i , 0 ) = ± 2 i = 0 , the analytic implicit function theorem [30 ] gives two analytic branches
w ± ( ϵ ) = ± i + a ± ϵ + b ± ϵ 2 + c ± ϵ 3 + O ( ϵ 4 ) , (106) w_\pm(\epsilon)=\pm i+a_\pm\epsilon+b_\pm\epsilon^2+c_\pm\epsilon^3+O(\epsilon^4), \tag*{(106)} w ± ( ϵ ) = ± i + a ± ϵ + b ± ϵ 2 + c ± ϵ 3 + O ( ϵ 4 ) , (106)
which satisfies G ( w ± ( ϵ ) , ϵ ) = 0 G(w_\pm(\epsilon),\epsilon)=0 G ( w ± ( ϵ ) , ϵ ) = 0 ; thus, the corresponding roots of P ∞ P_\infty P ∞ admit convergent expansions7 in powers of ϵ = δ \epsilon=\sqrt{\delta} ϵ = δ . Substituting (106) into G ( w ± ( ϵ ) , ϵ ) = 0 G(w_\pm(\epsilon),\epsilon)=0 G ( w ± ( ϵ ) , ϵ ) = 0 , and using the fact that the coefficient of ϵ \epsilon ϵ is 0 leads to
2 ( ± i ) a ± + τ ( ± i ) 3 + ( 1 + ν ) ( ± i ) = 0. 2(\pm i)a_\pm+\tau(\pm i)^3+(1+\nu)(\pm i)=0. 2 ( ± i ) a ± + τ ( ± i ) 3 + ( 1 + ν ) ( ± i ) = 0.
Since ( ± i ) 2 = − 1 (\pm i)^2=-1 ( ± i ) 2 = − 1 , this yields
a ± = τ − 1 − ν 2 . (107) a_\pm=\frac{\tau-1-\nu}{2}. \tag*{(107)} a ± = 2 τ − 1 − ν . (107)
Consequently, z ± = 1 + ϵ ( ± i + τ − 1 − ν 2 ϵ + O ( ϵ 2 ) ) = 1 ± i ϵ + τ − 1 − ν 2 ϵ 2 + O ( ϵ 3 ) z_\pm=1+\epsilon\left(\pm i+\frac{\tau-1-\nu}{2}\epsilon+O(\epsilon^2)\right)=1\pm i\epsilon+\frac{\tau-1-\nu}{2}\epsilon^2+O(\epsilon^3) z ± = 1 + ϵ ( ± i + 2 τ − 1 − ν ϵ + O ( ϵ 2 ) ) = 1 ± i ϵ + 2 τ − 1 − ν ϵ 2 + O ( ϵ 3 ) ; hence,
z ± ( δ ) = 1 + τ − 1 − ν 2 δ ± i δ + O ( δ 3 / 2 ) , (108) z_\pm(\delta)=1+\frac{\tau-1-\nu}{2}\delta\pm i\sqrt{\delta}+O(\delta^{3/2}), \tag*{(108)} z ± ( δ ) = 1 + 2 τ − 1 − ν δ ± i δ + O ( δ 3/2 ) , (108)
which implies that
∣ z ± ( δ ) ∣ 2 = 1 + δ ( τ − ν ) + O ( δ 3 / 2 ) . (109) |z_\pm(\delta)|^2=1+\delta(\tau-\nu)+O(\delta^{3/2}). \tag*{(109)} ∣ z ± ( δ ) ∣ 2 = 1 + δ ( τ − ν ) + O ( δ 3/2 ) . (109)
Thus, ν < τ \nu< \tau ν < τ is impossible under (102) . The borderline case ν = τ \nu=\tau ν = τ is also impossible. We can see this by continuing the same analytic expansion two orders further, which gives
∣ z ± ( δ ) ∣ 2 = 1 + τ ( 2 τ 2 + 3 τ + 1 ) 6 δ 2 + O ( δ 5 / 2 ) (110) |z_\pm(\delta)|^2=1+\frac{\tau(2\tau^2+3\tau+1)}{6}\delta^2+O(\delta^{5/2}) \tag*{(110)} ∣ z ± ( δ ) ∣ 2 = 1 + 6 τ ( 2 τ 2 + 3 τ + 1 ) δ 2 + O ( δ 5/2 ) (110)
when ν = τ \nu=\tau ν = τ . For τ > 0 \tau>0 τ > 0 , the coefficient of δ 2 \delta^2 δ 2 in (110) is positive, contradicting strict Schur stability for all sufficiently small positive δ \delta δ . If τ = 0 \tau=0 τ = 0 and ν = 0 \nu=0 ν = 0 , then P ∞ ( z ; δ , 0 ) = z 2 + ( δ − 2 ) z + 1 P_\infty(z;\delta,0)=z^2+(\delta-2)z+1 P ∞ ( z ; δ , 0 ) = z 2 + ( δ − 2 ) z + 1 , whose two roots have product one; for 0 < δ < 4 0<\delta<4 0 < δ < 4 , they are non-real conjugates; hence, they lie on the unit circle – this again contradicts strict Schur stability. Thus, we can conclude that
ν > τ . (111) \nu>\tau. \tag*{(111)} ν > τ . (111)
We now prove the asserted uniform bound. For the sake of contradiction, suppose that there exist c < δ L c < \sqrt{\delta_L} c < δ L , a sequence q n ↓ 0 q_n \downarrow0 q n ↓ 0 , numbers t n ∈ [ q n 2 , 1 ] t_n \in[q_n^2,1] t n ∈ [ q n 2 , 1 ] , and roots z n z_n z n of P τ , t n δ L P_{\tau,t_n\delta_L} P τ , t n δ L such that
∣ z n ∣ > 1 − c q n . (112) |z_n| > 1-cq_n. \tag*{(112)} ∣ z n ∣ > 1 − c q n . (112)
with t n = q n 2 ℓ n t_n=q_n^2\ell_n t n = q n 2 ℓ n for some 1 ≤ ℓ n ≤ q n − 2 1\leq\ell_n\leq q_n^{-2} 1 ≤ ℓ n ≤ q n − 2 , where
P τ , t n δ L ( z ) = z τ ( z − 1 ) ( z − 1 + 2 δ L q n ) + t n δ L ( ( 1 + ν ) z − ν ) P_{\tau,t_n\delta_L}(z)=z^\tau(z-1)(z-1+2\sqrt{\delta_L}q_n)+t_n\delta_L((1+\nu)z-\nu) P τ , t n δ L ( z ) = z τ ( z − 1 ) ( z − 1 + 2 δ L q n ) + t n δ L (( 1 + ν ) z − ν )
with our choice of α \alpha α and β \beta β . To simplify the notation, we denote these polynomials by P t n , q n P_{t_n,q_n} P t n , q n in the rest of the proof. Passing to a subsequence, we may assume that
t n → t 0 ∈ [ 0 , 1 ] , z n → z 0 , t_n\to t_0\in[0,1],\qquad z_n\to z_0, t n → t 0 ∈ [ 0 , 1 ] , z n → z 0 ,
which is possible since the roots are uniformly bounded as the polynomials P t n , q n P_{t_n,q_n} P t n , q n are monic of fixed degree and have uniformly bounded coefficients. From Eq. (112) , we have ∣ z 0 ∣ ≥ 1 |z_0|\geq1 ∣ z 0 ∣ ≥ 1 .
Since q n → 0 q_n\to0 q n → 0 and t n → t 0 ∈ [ 0 , 1 ] t_n\to t_0\in[0,1] t n → t 0 ∈ [ 0 , 1 ] , P t n , q n ⟶ P ∞ ( ⋅ ; t 0 δ L , ν ) P_{t_n,q_n}\longrightarrow P_\infty(\mathord{\cdot};t_0\delta_L,\nu) P t n , q n ⟶ P ∞ ( ⋅ ; t 0 δ L , ν ) coefficientwise. Hence, z 0 z_0 z 0 is a root of P ∞ ( ⋅ ; t 0 δ L , ν ) P_\infty(\mathord{\cdot};t_0\delta_L,\nu) P ∞ ( ⋅ ; t 0 δ L , ν ) . If t 0 > 0 t_0>0 t 0 > 0 , then 0 < t 0 δ L ≤ δ L 0<t_0\delta_L\leq\delta_L 0 < t 0 δ L ≤ δ L , and the hypothesis (102) implies that ∣ z 0 ∣ < 1 |z_0|<1 ∣ z 0 ∣ < 1 , which is a contradiction. Therefore, we must have
t n → 0. (113) t_n\to0. \tag*{(113)} t n → 0. (113)
From (101) , the coefficientwise limit is now z τ ( z − 1 ) 2 z^\tau(z-1)^2 z τ ( z − 1 ) 2 . Since ∣ z 0 ∣ ≥ 1 |z_0|\geq1 ∣ z 0 ∣ ≥ 1 , it follows that z 0 = 1 z_0=1 z 0 = 1 . Write
ξ n : = z n − 1 , s n : = 2 δ L q n . (114) \xi_n:=z_n-1,\qquad s_n:=2\sqrt{\delta_L}\,q_n. \tag*{(114)} ξ n := z n − 1 , s n := 2 δ L q n . (114)
The root equation P t n , q n ( z n ) = 0 P_{t_n,q_n}(z_n)=0 P t n , q n ( z n ) = 0 is exactly
( 1 + ξ n ) τ ξ n ( ξ n + s n ) + t n δ L ( 1 + ( 1 + ν ) ξ n ) = 0. (115) (1+\xi_n)^\tau\xi_n(\xi_n+s_n)+t_n\delta_L(1+(1+\nu)\xi_n)=0. \tag*{(115)} ( 1 + ξ n ) τ ξ n ( ξ n + s n ) + t n δ L ( 1 + ( 1 + ν ) ξ n ) = 0. (115)
Since ξ n → 0 \xi_n\to0 ξ n → 0 , (115) implies that there exists a constant C > 0 C>0 C > 0 , independent of n n n , such that, for all sufficiently large n n n , ∣ ξ n ∣ ∣ ξ n + s n ∣ ≤ C t n |\xi_n||\xi_n+s_n|\leq Ct_n ∣ ξ n ∣∣ ξ n + s n ∣ ≤ C t n ; hence, we also have ∣ ξ n ∣ 2 ≤ C t n + s n ∣ ξ n ∣ |\xi_n|^2\leq Ct_n+s_n|\xi_n| ∣ ξ n ∣ 2 ≤ C t n + s n ∣ ξ n ∣ . Consequently,
∣ ξ n ∣ = O ( s n + t n ) . (116) |\xi_n|=O(s_n+\sqrt{t_n}). \tag*{(116)} ∣ ξ n ∣ = O ( s n + t n ) . (116)
Expanding ( 1 + ξ n ) τ = 1 + τ ξ n + O ( ∣ ξ n ∣ 2 ) (1+\xi_n)^\tau=1+\tau\xi_n+O(|\xi_n|^2) ( 1 + ξ n ) τ = 1 + τ ξ n + O ( ∣ ξ n ∣ 2 ) in (115) , we obtain the local quadratic relation
ξ n 2 + [ s n + t n δ L ( 1 + ν ) ] ξ n + t n δ L = O ( ∣ ξ n ∣ 3 + s n ∣ ξ n ∣ 2 ) . (117) \xi_n^2+\bigl[s_n+t_n\delta_L(1+\nu)\bigr]\xi_n+t_n\delta_L
=O\bigl(|\xi_n|^3+s_n|\xi_n|^2\bigr). \tag*{(117)} ξ n 2 + [ s n + t n δ L ( 1 + ν ) ] ξ n + t n δ L = O ( ∣ ξ n ∣ 3 + s n ∣ ξ n ∣ 2 ) . (117)
The remaining analysis depends only on the size of the ratio t n / q n 2 t_n/q_n^2 t n / q n 2 . Since t n ≥ q n 2 t_n\geq q_n^2 t n ≥ q n 2 , this ratio lies in [ 1 , ∞ ) [1,\infty) [ 1 , ∞ ) . After passing to a subsequence, there are exactly two possibilities: either t n / q n 2 → ∞ t_n/q_n^2\to\infty t n / q n 2 → ∞ , or the ratio is bounded, in which case it has a further subsequential limit ℓ ∈ [ 1 , ∞ ) \ell\in[1,\infty) ℓ ∈ [ 1 , ∞ ) . We treat the unbounded alternative first; the bounded alternative is the final case below.
Suppose that t n / q n 2 → ∞ t_n/q_n^2\to\infty t n / q n 2 → ∞ . Then
s n t n = 2 δ L q n t n = 2 δ L ( q n 2 t n ) 1 / 2 ⟶ 0. \frac{s_n}{\sqrt{t_n}}
=2\sqrt{\delta_L}\frac{q_n}{\sqrt{t_n}}
=2\sqrt{\delta_L}\left(\frac{q_n^2}{t_n}\right)^{1/2}
\longrightarrow0. t n s n = 2 δ L t n q n = 2 δ L ( t n q n 2 ) 1/2 ⟶ 0.
Thus s n = o ( t n ) s_n=o(\sqrt{t_n}) s n = o ( t n ) , and (116) gives ∣ ξ n ∣ = O ( t n ) |\xi_n|=O(\sqrt{t_n}) ∣ ξ n ∣ = O ( t n ) . Define
η n : = ξ n t n . (118) \eta_n:=\frac{\xi_n}{\sqrt{t_n}}. \tag*{(118)} η n := t n ξ n . (118)
The preceding estimate shows that { η n } ⊂ C \{\eta_n\} \subset\mathbb{C} { η n } ⊂ C sequence is bounded; hence, after passing to a further subsequence we may assume that η n → η 0 \eta_n \to\eta_0 η n → η 0 . To identify η 0 \eta_0 η 0 , substitute ξ n = t n η n \xi_n = \sqrt{t_n}\eta_n ξ n = t n η n into the exact equation (115) and divide by t n t_n t n . This gives the exact scaled equation
( 1 + t n η n ) τ ( η n + s n t n ) + δ L ( 1 + ( 1 + ν ) t n η n ) = 0. (119) (1+\sqrt{t_n}\eta_n)^\tau\left(\eta_n+\frac{s_n}{\sqrt{t_n}}\right)+\delta_L\left(1+(1+\nu)\sqrt{t_n}\eta_n\right)=0. \tag*{(119)} ( 1 + t n η n ) τ ( η n + t n s n ) + δ L ( 1 + ( 1 + ν ) t n η n ) = 0. (119)
Since t n → 0 t_n \to0 t n → 0 , s n / t n → 0 s_n/\sqrt{t_n} \to0 s n / t n → 0 , and { η n } \{\eta_n\} { η n } is bounded converging to η 0 \eta_0 η 0 , letting n → ∞ n \to\infty n → ∞ in (118) yields η 0 2 + δ L = 0 \eta_0^2+\delta_L=0 η 0 2 + δ L = 0 ; hence,
η 0 = ± i δ L . (120) \eta_0=\pm i\sqrt{\delta_L}. \tag*{(120)} η 0 = ± i δ L . (120)
We next expand (118) one order further. Choose R > 0 R>0 R > 0 such that ∣ η n ∣ ≤ R |\eta_n|\le R ∣ η n ∣ ≤ R for all sufficiently large n n n . Since η n → η 0 \eta_n\to\eta_0 η n → η 0 and ∣ η 0 ∣ = δ L |\eta_0|=\sqrt{\delta_L} ∣ η 0 ∣ = δ L , we may take, for example, R = 2 δ L R=2\sqrt{\delta_L} R = 2 δ L for all sufficiently large n n n . Taylor’s formula on the fixed disk ∣ η ∣ ≤ R |\eta|\le R ∣ η ∣ ≤ R gives
( 1 + t n η ) τ = 1 + τ t n η + O R ( t n ) , (1+\sqrt{t_n}\eta)^\tau=1+\tau\sqrt{t_n}\eta+O_R(t_n), ( 1 + t n η ) τ = 1 + τ t n η + O R ( t n ) ,
where the constant in O R ( t n ) O_R(t_n) O R ( t n ) is independent of n n n and of ∣ η ∣ ≤ R |\eta|\le R ∣ η ∣ ≤ R . Applying this with η = η n \eta=\eta_n η = η n in (118), and using s n / t n → 0 s_n/\sqrt{t_n}\to0 s n / t n → 0 , yields
0 = η n 2 + δ L + s n t n η n + t n ( τ η n 3 + δ L ( 1 + ν ) η n ) + o ( s n t n + t n ) . (121) 0=\eta_n^2+\delta_L+\frac{s_n}{\sqrt{t_n}}\eta_n+\sqrt{t_n}\left(\tau\eta_n^3+\delta_L(1+\nu)\eta_n\right)+o\left(\frac{s_n}{\sqrt{t_n}}+\sqrt{t_n}\right). \tag*{(121)} 0 = η n 2 + δ L + t n s n η n + t n ( τ η n 3 + δ L ( 1 + ν ) η n ) + o ( t n s n + t n ) . (121)
Here and below, the little-o o o estimates are understood along the present subsequence, for which η n → η 0 \eta_n\to\eta_0 η n → η 0 . The point of the preceding R R R -argument is that the Taylor remainder is controlled uniformly on a fixed disk containing all sufficiently large η n \eta_n η n ; hence, the implicit constant does not depend on n n n . Write η n : = η 0 + h n \eta_n:=\eta_0+h_n η n := η 0 + h n , where η 0 \eta_0 η 0 is as in (120) , and set
m n : = s n t n + t n . m_n:=\frac{s_n}{\sqrt{t_n}}+\sqrt{t_n}. m n := t n s n + t n .
Since s n / t n → 0 s_n/\sqrt{t_n}\to0 s n / t n → 0 and t n → 0 t_n\to0 t n → 0 , we have m n → 0 m_n\to0 m n → 0 . From (121) , using that { η n } \{\eta_n\} { η n } is bounded, we obtain η n 2 + δ L = O ( m n ) \eta_n^2+\delta_L=O(m_n) η n 2 + δ L = O ( m n ) . Since η 0 2 = − δ L \eta_0^2=-\delta_L η 0 2 = − δ L , it follows that
η n 2 + δ L = ( η n − η 0 ) ( η n + η 0 ) = h n ( 2 η 0 + h n ) . \eta_n^2+\delta_L=(\eta_n-\eta_0)(\eta_n+\eta_0)=h_n(2\eta_0+h_n). η n 2 + δ L = ( η n − η 0 ) ( η n + η 0 ) = h n ( 2 η 0 + h n ) .
Moreover, h n → 0 h_n\to0 h n → 0 and η 0 = ± i δ L ≠ 0 \eta_0=\pm i\sqrt{\delta_L}\ne0 η 0 = ± i δ L = 0 , so ∣ 2 η 0 + h n ∣ |2\eta_0+h_n| ∣2 η 0 + h n ∣ is bounded away from zero for all sufficiently large n n n . Consequently, h n = O ( m n ) h_n=O(m_n) h n = O ( m n ) ; hence h n 2 = O ( m n 2 ) = o ( m n ) h_n^2=O(m_n^2)=o(m_n) h n 2 = O ( m n 2 ) = o ( m n ) .
We can now expand (121) around the simple root η 0 \eta_0 η 0 . Using η n = η 0 + h n \eta_n=\eta_0+h_n η n = η 0 + h n , we have
η n 2 + δ L = 2 η 0 h n + o ( m n ) , \eta_n^2+\delta_L=2\eta_0h_n+o(m_n), η n 2 + δ L = 2 η 0 h n + o ( m n ) ,
and, since h n = O ( m n ) h_n=O(m_n) h n = O ( m n ) , we also have s n t n η n = s n t n η 0 + o ( m n ) \frac{s_n}{\sqrt{t_n}}\eta_n=\frac{s_n}{\sqrt{t_n}}\eta_0+o(m_n) t n s n η n = t n s n η 0 + o ( m n ) . Similarly,
t n ( τ η n 3 + δ L ( 1 + ν ) η n ) = t n ( τ η 0 3 + δ L ( 1 + ν ) η 0 ) + o ( m n ) . \sqrt{t_n}\left(\tau\eta_n^3+\delta_L(1+\nu)\eta_n\right)=\sqrt{t_n}\left(\tau\eta_0^3+\delta_L(1+\nu)\eta_0\right)+o(m_n). t n ( τ η n 3 + δ L ( 1 + ν ) η n ) = t n ( τ η 0 3 + δ L ( 1 + ν ) η 0 ) + o ( m n ) .
Since η 0 2 = − δ L \eta_0^2=-\delta_L η 0 2 = − δ L , we have
τ η 0 3 + δ L ( 1 + ν ) η 0 = δ L ( 1 + ν − τ ) η 0 . \tau\eta_0^3+\delta_L(1+\nu)\eta_0=\delta_L(1+\nu-\tau)\eta_0. τ η 0 3 + δ L ( 1 + ν ) η 0 = δ L ( 1 + ν − τ ) η 0 .
Therefore, (121) gives
2 η 0 h n + s n t n η 0 + t n δ L ( 1 + ν − τ ) η 0 = o ( m n ) . 2\eta_0h_n+\frac{s_n}{\sqrt{t_n}}\eta_0+\sqrt{t_n}\delta_L(1+\nu-\tau)\eta_0=o(m_n). 2 η 0 h n + t n s n η 0 + t n δ L ( 1 + ν − τ ) η 0 = o ( m n ) .
Dividing by 2 η 0 ≠ 0 2\eta_0 \ne0 2 η 0 = 0 , we conclude that
h n = − 1 2 ( s n t n + t n δ L ( 1 + ν − τ ) ) + o ( s n t n + t n ) . (122) h_n=-\frac{1}{2}\left(\frac{s_n}{\sqrt{t_n}}+\sqrt{t_n}\,\delta_L(1+\nu-\tau)\right)+o\left(\frac{s_n}{\sqrt{t_n}}+\sqrt{t_n}\right). \tag*{(122)} h n = − 2 1 ( t n s n + t n δ L ( 1 + ν − τ ) ) + o ( t n s n + t n ) . (122)
Based on the expansion (122) and using ξ n = η n t n \xi_n=\eta_n\sqrt{t_n} ξ n = η n t n , every near-one root in this regime satisfies
ξ n = ± i t n δ L − s n 2 − t n δ L ( 1 + ν − τ ) 2 + o ( q n + t n ) , (123) \xi_n=\pm i\sqrt{t_n\delta_L}-\frac{s_n}{2}-\frac{t_n\delta_L(1+\nu-\tau)}{2}+o(q_n+t_n), \tag*{(123)} ξ n = ± i t n δ L − 2 s n − 2 t n δ L ( 1 + ν − τ ) + o ( q n + t n ) , (123)
with the sign determined by the branch. It follows that
2 Re ξ n = − s n − t n δ L ( 1 + ν − τ ) + o ( q n + t n ) , 2\operatorname{Re}\xi_n=-s_n-t_n\delta_L(1+\nu-\tau)+o(q_n+t_n), 2 Re ξ n = − s n − t n δ L ( 1 + ν − τ ) + o ( q n + t n ) ,
and
∣ ξ n ∣ 2 = t n δ L + o ( q n + t n ) . |\xi_n|^2=t_n\delta_L+o(q_n+t_n). ∣ ξ n ∣ 2 = t n δ L + o ( q n + t n ) .
Thus
∣ 1 + ξ n ∣ 2 = 1 + 2 Re ξ n + ∣ ξ n ∣ 2 = 1 − s n + t n δ L ( τ − ν ) + o ( q n + t n ) . (124) \begin{aligned}
|1+\xi_n|^2&=1+2\operatorname{Re}\xi_n+|\xi_n|^2\\
&=1-s_n+t_n\delta_L(\tau-\nu)+o(q_n+t_n). \tag*{(124)}
\end{aligned} ∣1 + ξ n ∣ 2 = 1 + 2 Re ξ n + ∣ ξ n ∣ 2 = 1 − s n + t n δ L ( τ − ν ) + o ( q n + t n ) . (124)
By (111) , the coefficient δ L ( τ − ν ) \delta_L(\tau-\nu) δ L ( τ − ν ) is negative. If t n = O ( q n ) t_n=O(q_n) t n = O ( q n ) , then (124) yields
∣ 1 + ξ n ∣ ≤ 1 − δ L q n + o ( q n ) , |1+\xi_n|\le1-\sqrt{\delta_L}\,q_n+o(q_n), ∣1 + ξ n ∣ ≤ 1 − δ L q n + o ( q n ) ,
contradicting (112) because c < δ L c<\sqrt{\delta_L} c < δ L . If t n / q n → ∞ t_n/q_n\to\infty t n / q n → ∞ , then the negative term t n δ L ( τ − ν ) t_n\delta_L(\tau-\nu) t n δ L ( τ − ν ) dominates the q n q_n q n -scale in (124) , and (112) is again impossible.
It remains only to consider the bounded alternative for t n / q n 2 t_n/q_n^2 t n / q n 2 . This is the only remaining possibility because, as observed above, every subsequence of the ratios either has a further subsequence tending to ∞ \infty ∞ or a bounded further subsequence. Passing to a subsequence in the bounded case, we may assume that
t n q n 2 → ℓ ∈ [ 1 , ∞ ) . \frac{t_n}{q_n^2}\to\ell\in[1,\infty). q n 2 t n → ℓ ∈ [ 1 , ∞ ) .
The estimate (116) gives ξ n = O ( q n ) \xi_n=O(q_n) ξ n = O ( q n ) . Thus z n = 1 + ξ n z_n=1+\xi_n z n = 1 + ξ n is one of the two near-one roots covered by Lemma 3.21. Hence
∣ z n ∣ = 1 − δ L q n + o ( q n ) , |z_n|=1-\sqrt{\delta_L}\,q_n+o(q_n), ∣ z n ∣ = 1 − δ L q n + o ( q n ) ,
which contradicts (112) , again because c < δ L c<\sqrt{\delta_L} c < δ L .
The contradiction proves that, for every c < δ L c<\sqrt{\delta_L} c < δ L , there exists q 0 > 0 q_0>0 q 0 > 0 such that every root of P t , q P_{t,q} P t , q , uniformly for t ∈ [ q 2 , 1 ] t\in[q^2,1] t ∈ [ q 2 , 1 ] , lies in ∣ z ∣ ≤ 1 − c q |z|\le1-cq ∣ z ∣ ≤ 1 − c q whenever 0 < q ≤ q 0 0<q\le q_0 0 < q ≤ q 0 . By the definition of ρ τ , κ \rho_{\tau,\kappa} ρ τ , κ , this is precisely (103) . □
Stability analysis via the Cayley transform
We reuse the Cayley transform of Lemma 3.2 ,
z = 1 + w 1 − w , (125) z=\frac{1+w}{1-w}, \tag*{(125)} z = 1 − w 1 + w , (125)
which sends Re w < 0 \operatorname{Re}w<0 Re w < 0 to ∣ z ∣ < 1 |z|<1 ∣ z ∣ < 1 . Multiplying by ( 1 − w ) τ + 2 (1-w)^{\tau+2} ( 1 − w ) τ + 2 , the transformed polynomial becomes
H τ , δ , ν ( w ) : = ( 1 − w ) τ + 2 P ∞ ( 1 + w 1 − w ; δ , ν ) = 4 w 2 ( 1 + w ) τ + δ [ 1 + ( 1 + 2 ν ) w ] ( 1 − w ) τ + 1 . (126) \mathcal{H}_{\tau,\delta,\nu}(w):=(1-w)^{\tau+2}P_\infty\left(\frac{1+w}{1-w};\delta,\nu\right)=4w^2(1+w)^\tau+\delta[1+(1+2\nu)w](1-w)^{\tau+1}. \tag*{(126)} H τ , δ , ν ( w ) := ( 1 − w ) τ + 2 P ∞ ( 1 − w 1 + w ; δ , ν ) = 4 w 2 ( 1 + w ) τ + δ [ 1 + ( 1 + 2 ν ) w ] ( 1 − w ) τ + 1 . (126)
Based on Lemma 3.2 , P ∞ P_\infty P ∞ is Schur stable if and only if H τ , δ , ν \mathcal{H}_{\tau,\delta,\nu} H τ , δ , ν is Hurwitz stable provided that P ∞ ( − 1 ; δ , ν ) ≠ 0 P_\infty(-1;\delta,\nu)\ne0 P ∞ ( − 1 ; δ , ν ) = 0 . Then, as a corollary to the previous theorem, we get the following result.
Corollary 3.23. If ν > τ \nu> \tau ν > τ , then H τ , δ , ν \mathcal{H}_{\tau,\delta,\nu} H τ , δ , ν is Hurwitz stable for all sufficiently small δ > 0 \delta> 0 δ > 0 .
Proof. The proof of Theorem 3.22 shows that for ν > τ \nu> \tau ν > τ , the polynomial P ∞ ( z ; δ , ν ) P_{\infty}(z;\delta,\nu) P ∞ ( z ; δ , ν ) is Schur stable for δ \delta δ small, more specifically this follows from the expansion (109) . The remaining τ \tau τ roots stay near 0 0 0 ; hence, they are inside the unit disk for sufficiently small δ > 0 \delta> 0 δ > 0 , by continuity. Note that P ∞ ( − 1 ; 0 , ν ) = 4 ( − 1 ) τ ≠ 0 P_{\infty}(-1;0,\nu)=4(-1)^{\tau}\ne0 P ∞ ( − 1 ; 0 , ν ) = 4 ( − 1 ) τ = 0 so that P ∞ ( − 1 ; δ , ν ) ≠ 0 P_{\infty}(-1;\delta,\nu)\ne0 P ∞ ( − 1 ; δ , ν ) = 0 for δ \delta δ around 0 0 0 . Then, P ∞ P_{\infty} P ∞ is Schur stable if and only if H τ , δ , ν \mathcal{H}_{\tau,\delta,\nu} H τ , δ , ν is Hurwitz stable by Lemma 3.2; and we conclude.
Lemma 3.24 (explicit no-imaginary-axis interval). Consider the following parameter choice
ν = M − 1 2 with M : = 2 τ + 1. \nu=M-\frac{1}{2}\qquad\text{with}\qquad M:=2\tau+1. ν = M − 2 1 with M := 2 τ + 1. Then H τ , δ , ν \mathcal{H}_{\tau,\delta,\nu} H τ , δ , ν has no root on the imaginary axis for every
0 < δ ≤ M − 2 . (127) 0<\delta\le M^{-2}. \tag*{(127)} 0 < δ ≤ M − 2 . (127) Proof. Fix an arbitrary δ ∈ ( 0 , M − 2 ] \delta\in(0,M^{-2}] δ ∈ ( 0 , M − 2 ] . Suppose, for contradiction, that H τ , δ , ν \mathcal{H}_{\tau,\delta,\nu} H τ , δ , ν has a root on the imaginary axis. Since we have
H τ , δ , ν ( 0 ) = δ > 0 , \mathcal{H}_{\tau,\delta,\nu}(0)=\delta>0, H τ , δ , ν ( 0 ) = δ > 0 , this root is non-zero. Moreover, H τ , δ , ν \mathcal{H}_{\tau,\delta,\nu} H τ , δ , ν has real coefficients, so its non-real roots occur in conjugate pairs. Hence, without the loss of generality we can assume that w = i ω w=i\omega w = iω , ω > 0 \omega>0 ω > 0 , is a root. Put u = arctan ω ∈ ( 0 , π / 2 ) u=\arctan\omega\in(0,\pi/2) u = arctan ω ∈ ( 0 , π /2 ) and a : = 1 + 2 ν = 2 M a:=1+2\nu=2M a := 1 + 2 ν = 2 M . From (126) ,
δ ( 1 + i a ω ) ( 1 − i ω ) τ + 1 = 4 ω 2 ( 1 + i ω ) τ . (128) \delta(1+ia\omega)(1-i\omega)^{\tau+1}=4\omega^2(1+i\omega)^{\tau}. \tag*{(128)} δ ( 1 + iaω ) ( 1 − iω ) τ + 1 = 4 ω 2 ( 1 + iω ) τ . (128) Taking arguments in (128) requires keeping track of the branch. Since u = arctan ω u=\arctan\omega u = arctan ω , we have
arg ( 1 − i ω ) = − u , arg ( 1 + i ω ) = u . \arg(1-i\omega)=-u,\qquad\arg(1+i\omega)=u. arg ( 1 − iω ) = − u , arg ( 1 + iω ) = u . Hence, equating the arguments of both sides in (128) ,
arg ( 1 + i a ω ) − ( τ + 1 ) u ≡ τ u ( m o d 2 π ) , \arg(1+ia\omega)-(\tau+1)u\equiv\tau u\pmod{2\pi}, arg ( 1 + iaω ) − ( τ + 1 ) u ≡ τ u ( mod 2 π ) , or equivalently
arg ( 1 + i a ω ) ≡ ( 2 τ + 1 ) u = M u ( m o d 2 π ) . \arg(1+ia\omega)\equiv(2\tau+1)u=Mu\pmod{2\pi}. arg ( 1 + iaω ) ≡ ( 2 τ + 1 ) u = M u ( mod 2 π ) . The principal argument on the left is
arg ( 1 + i a ω ) = arctan ( a ω ) = arctan ( 2 M tan u ) ∈ ( 0 , π / 2 ) . \arg(1+ia\omega)=\arctan(a\omega)=\arctan(2M\tan u)\in(0,\pi/2). arg ( 1 + iaω ) = arctan ( aω ) = arctan ( 2 M tan u ) ∈ ( 0 , π /2 ) . If we put y : = M u y:=Mu y := M u , then y y y must be congruent modulo 2 π 2\pi 2 π to an angle in ( 0 , π / 2 ) (0,\pi/2) ( 0 , π /2 ) . Therefore there is an integer j ≥ 0 j\ge0 j ≥ 0 such that
y ∈ ( 2 j π , 2 j π + π / 2 ) , tan y = tan ( arg ( 1 + i a ω ) ) = a ω = 2 M tan u = 2 M tan ( y / M ) . (129) y\in(2j\pi,2j\pi+\pi/2),\qquad\tan y=\tan(\arg(1+ia\omega))=a\omega=2M\tan u=2M\tan(y/M). \tag*{(129)} y ∈ ( 2 j π , 2 j π + π /2 ) , tan y = tan ( arg ( 1 + iaω )) = aω = 2 M tan u = 2 M tan ( y / M ) . (129) Taking moduli of both sides in (128) , noting ∣ 1 ± i ω ∣ = ( 1 + ω 2 ) 1 / 2 |1\pm i\omega|=(1+\omega^2)^{1/2} ∣1 ± iω ∣ = ( 1 + ω 2 ) 1/2 , we have also
δ ( 1 + a 2 ω 2 ) 1 / 2 ( 1 + ω 2 ) ( τ + 1 ) / 2 = 4 ω 2 ( 1 + ω 2 ) τ / 2 . \delta(1+a^2\omega^2)^{1/2}(1+\omega^2)^{(\tau+1)/2}=4\omega^2(1+\omega^2)^{\tau/2}. δ ( 1 + a 2 ω 2 ) 1/2 ( 1 + ω 2 ) ( τ + 1 ) /2 = 4 ω 2 ( 1 + ω 2 ) τ /2 . Reorganizing this and using a = 2 M a=2M a = 2 M , we get
δ = 4 ω 2 1 + 4 M 2 ω 2 1 + ω 2 . (130) \delta=\frac{4\omega^2}{\sqrt{1+4M^2\omega^2}\sqrt{1+\omega^2}}. \tag*{(130)} δ = 1 + 4 M 2 ω 2 1 + ω 2 4 ω 2 . (130)
Now substitute ω = tan u \omega= \tan u ω = tan u . Since u ∈ ( 0 , π / 2 ) u \in(0,\pi/2) u ∈ ( 0 , π /2 ) ,
1 + ω 2 = 1 + tan 2 u = sec u = 1 cos u . \sqrt{1+\omega^2}=\sqrt{1+\tan^2 u}=\sec u=\frac{1}{\cos u}. 1 + ω 2 = 1 + tan 2 u = sec u = cos u 1 .
Thus,
δ = 4 tan 2 u ⋅ cos ( u ) 1 + 4 M 2 tan 2 u = 4 sin 2 u cos 2 u + 4 M 2 sin 2 u . \delta=\frac{4\tan^2 u\cdot\cos(u)}{\sqrt{1+4M^2\tan^2 u}}=\frac{4\sin^2 u}{\sqrt{\cos^2 u+4M^2\sin^2 u}}. δ = 1 + 4 M 2 tan 2 u 4 tan 2 u ⋅ cos ( u ) = cos 2 u + 4 M 2 sin 2 u 4 sin 2 u .
From the argument calculation, with y = M u y=Mu y = M u ,
tan y = 2 M tan u , y ∈ ( 2 j π , 2 j π + π / 2 ) . \tan y=2M\tan u,\qquad y\in(2j\pi,2j\pi+\pi/2). tan y = 2 M tan u , y ∈ ( 2 j π , 2 j π + π /2 ) .
Since y y y is congruent modulo 2 π 2\pi 2 π to an angle in ( 0 , π / 2 ) (0,\pi/2) ( 0 , π /2 ) , cos y > 0 \cos y>0 cos y > 0 . Therefore,
cos y = 1 1 + tan 2 y = 1 1 + 4 M 2 tan 2 u . \cos y=\frac{1}{\sqrt{1+\tan^2 y}}=\frac{1}{\sqrt{1+4M^2\tan^2 u}}. cos y = 1 + tan 2 y 1 = 1 + 4 M 2 tan 2 u 1 .
Using this in the previous expression for δ \delta δ gives
δ = 4 tan 2 u ⋅ cos u ⋅ cos y = 4 sin 2 u cos y cos u . \delta=4\tan^2 u\cdot\cos u\cdot\cos y=4\sin^2 u\frac{\cos y}{\cos u}. δ = 4 tan 2 u ⋅ cos u ⋅ cos y = 4 sin 2 u cos u cos y .
Finally, y = M u y=Mu y = M u , we get
δ = 4 sin 2 u cos ( M u ) cos u = 4 sin 2 ( y / M ) cos y cos ( y / M ) . (131) \delta=4\sin^2 u\frac{\cos(Mu)}{\cos u}=4\sin^2(y/M)\frac{\cos y}{\cos(y/M)}. \tag*{(131)} δ = 4 sin 2 u cos u cos ( M u ) = 4 sin 2 ( y / M ) cos ( y / M ) cos y . (131)
We next show that any y y y satisfying (129) , with δ \delta δ defined by (131) , necessarily satisfies δ > M − 2 \delta>M^{-2} δ > M − 2 . There are two cases:
Case 1: j = 0 j=0 j = 0 . Then 0 < y < π / 2 0<y<\pi/2 0 < y < π /2 , and 0 < y / M ≤ π / 6 0<y/M\leq\pi/6 0 < y / M ≤ π /6 . Since tan x ≥ x \tan x\geq x tan x ≥ x , (129) gives tan y ≥ 2 y \tan y\geq2y tan y ≥ 2 y . The function tan y / y \tan y/y tan y / y is increasing, and tan ( 11 / 10 ) < 2 ( 11 / 10 ) \tan(11/10)<2(11/10) tan ( 11/10 ) < 2 ( 11/10 ) , so
y > 11 10 . y>\frac{11}{10}. y > 10 11 .
On [ 0 , π / 6 ] [0,\pi/6] [ 0 , π /6 ] , tan x ≤ ( 4 / 3 ) x \tan x\leq(4/3)x tan x ≤ ( 4/3 ) x ; hence tan y ≤ ( 8 / 3 ) y \tan y\leq(8/3)y tan y ≤ ( 8/3 ) y . Since tan ( 13 / 10 ) > ( 8 / 3 ) ( 13 / 10 ) \tan(13/10)>(8/3)(13/10) tan ( 13/10 ) > ( 8/3 ) ( 13/10 ) ,
y < 13 10 . y<\frac{13}{10}. y < 10 13 .
Thus cos y > cos ( 13 / 10 ) > 1 / 4 \cos y>\cos(13/10)>1/4 cos y > cos ( 13/10 ) > 1/4 . By concavity of sine on [ 0 , π / 2 ] [0,\pi/2] [ 0 , π /2 ] ,
sin x ≥ 3 π x , 0 ≤ x ≤ π / 6. \sin x\geq\frac{3}{\pi}x,\qquad0\leq x\leq\pi/6. sin x ≥ π 3 x , 0 ≤ x ≤ π /6.
Therefore,
δ ≥ 4 ( 3 y π M ) 2 cos y > 4 ( 3 π 11 10 M ) 2 1 4 = 1089 100 π 2 1 M 2 > 1 M 2 . \delta\geq4\left(\frac{3y}{\pi M}\right)^2\cos y>4\left(\frac{3}{\pi}\frac{11}{10M}\right)^2\frac{1}{4}=\frac{1089}{100\pi^2}\frac{1}{M^2}>\frac{1}{M^2}. δ ≥ 4 ( π M 3 y ) 2 cos y > 4 ( π 3 10 M 11 ) 2 4 1 = 100 π 2 1089 M 2 1 > M 2 1 .
Case 2: j ≥ 1 j\geq1 j ≥ 1 . This case is vacuous for τ = 1 \tau=1 τ = 1 , because then M = 3 M=3 M = 3 and u < π / 2 u<\pi/2 u < π /2 imply y = M u < 3 π / 2 < 2 π y=Mu<3\pi/2<2\pi y = M u < 3 π /2 < 2 π . Thus, j ≥ 1 j\geq1 j ≥ 1 implies M > 4 M>4 M > 4 . Since y ≥ 2 π y\geq2\pi y ≥ 2 π , we have u = y / M ≥ 2 π / M u=y/M\geq2\pi/M u = y / M ≥ 2 π / M . Using the concavity of sin ( ⋅ ) \sin(\cdot) sin ( ⋅ ) in this region gives
sin u ≥ 2 π u ≥ 4 M . (132) \sin u\geq\frac{2}{\pi}u\geq\frac{4}{M}. \tag*{(132)} sin u ≥ π 2 u ≥ M 4 . (132)
Using tan y = 2 M tan u \tan y=2M\tan u tan y = 2 M tan u in (131) is equivalent to the expression
δ = 4 sin 2 u cos 2 u + 4 M 2 sin 2 u . (133) \delta=\frac{4\sin^2u}{\sqrt{\cos^2u+4M^2\sin^2u}}. \tag*{(133)} δ = cos 2 u + 4 M 2 sin 2 u 4 sin 2 u . (133)
Indeed, cos y = [ 1 + 4 M 2 tan 2 u ] − 1 / 2 \cos y=[1+4M^2\tan^2u]^{-1/2} cos y = [ 1 + 4 M 2 tan 2 u ] − 1/2 and cos y > 0 \cos y>0 cos y > 0 . The right side of (133) is increasing in s = sin u ∈ ( 0 , 1 ) s=\sin u\in(0,1) s = sin u ∈ ( 0 , 1 ) . Combining this monotonicity with (132) ,
δ ≥ 4 ( 4 / M ) 2 1 + ( 4 M 2 − 1 ) ( 4 / M ) 2 = 64 M 2 65 − 16 / M 2 > 1 M 2 . \delta\geq\frac{4(4/M)^2}{\sqrt{1+(4M^2-1)(4/M)^2}}=\frac{64}{M^2\sqrt{65-16/M^2}}>\frac{1}{M^2}. δ ≥ 1 + ( 4 M 2 − 1 ) ( 4/ M ) 2 4 ( 4/ M ) 2 = M 2 65 − 16/ M 2 64 > M 2 1 .
Both cases contradict δ ≤ M − 2 \delta\leq M^{-2} δ ≤ M − 2 , so no imaginary-axis root exists when 0 < δ ≤ M − 2 0<\delta\leq M^{-2} 0 < δ ≤ M − 2 .
Theorem 3.25 (closed-form accelerated family for every fixed delay). Fix any integer τ ≥ 1 \tau\geq1 τ ≥ 1 , with M = 2 τ + 1 M=2\tau+1 M = 2 τ + 1 , set
α = 1 M 2 L , β = 1 − 2 M q , ν = M − 1 2 = 2 τ + 1 2 . (134) \alpha=\frac{1}{M^2L},\qquad\beta=1-\frac{2}{M}q,\qquad\nu=M-\frac{1}{2}=2\tau+\frac{1}{2}. \tag*{(134)} α = M 2 L 1 , β = 1 − M 2 q , ν = M − 2 1 = 2 τ + 2 1 . (134) Then delayed GMM on quadratics satisfies, for every fixed τ \tau τ and every 0 < c < 1 / M 0<c<1/M 0 < c < 1/ M , the accelerated bound
ρ τ , κ ( α , β , ν ) ≤ 1 − c q = 1 − c κ − 1 / 2 for all κ ≥ κ 0 a l l ( τ , c ) , (135) \rho_{\tau,\kappa}(\alpha,\beta,\nu)\leq1-cq=1-c\kappa^{-1/2}\qquad\text{for all }\kappa\geq\kappa^{\mathrm{all}}_0(\tau,c), \tag*{(135)} ρ τ , κ ( α , β , ν ) ≤ 1 − c q = 1 − c κ − 1/2 for all κ ≥ κ 0 all ( τ , c ) , (135) where κ 0 a l l ( τ , c ) < ∞ \kappa^{\mathrm{all}}_0(\tau,c)<\infty κ 0 all ( τ , c ) < ∞ is a finite threshold depending only on τ \tau τ and c c c . In particular, taking c = 1 2 M c=\frac{1}{2M} c = 2 M 1 gives ρ τ , κ ≤ 1 − 1 2 ( 2 τ + 1 ) κ − 1 / 2 \rho_{\tau,\kappa}\leq1-\frac{1}{2(2\tau+1)}\kappa^{-1/2} ρ τ , κ ≤ 1 − 2 ( 2 τ + 1 ) 1 κ − 1/2 .
Proof. By Corollary 3.23, since ν = M − 1 / 2 > τ \nu=M-1/2>\tau ν = M − 1/2 > τ , the Hurwitz image H τ , δ , ν \mathcal{H}_{\tau,\delta,\nu} H τ , δ , ν is stable for all sufficiently small δ > 0 \delta>0 δ > 0 ; fix such δ ∗ \delta_* δ ∗ satisfying δ ∗ ∈ ( 0 , M − 2 ] \delta_*\in(0,M^{-2}] δ ∗ ∈ ( 0 , M − 2 ] . The leading coefficient of H τ , δ , ν \mathcal{H}_{\tau,\delta,\nu} H τ , δ , ν is
4 + ( − 1 ) τ + 1 δ ( 1 + 2 ν ) = 4 + ( − 1 ) τ + 1 2 M δ , 4+(-1)^{\tau+1}\delta(1+2\nu)=4+(-1)^{\tau+1}2M\delta, 4 + ( − 1 ) τ + 1 δ ( 1 + 2 ν ) = 4 + ( − 1 ) τ + 1 2 M δ ,
and for 0 ≤ δ ≤ M − 2 0\leq\delta\leq M^{-2} 0 ≤ δ ≤ M − 2 we have 2 M δ ≤ 2 / M ≤ 2 / 3 < 4 2M\delta\leq2/M\leq2/3<4 2 M δ ≤ 2/ M ≤ 2/3 < 4 , so it never vanishes: the degree is constant and no root escapes to infinity on [ δ ∗ , M − 2 ] [\delta_*,M^{-2}] [ δ ∗ , M − 2 ] . By Lemma 3.24 no root lies on the imaginary axis for any δ ∈ ( 0 , M − 2 ] \delta\in(0,M^{-2}] δ ∈ ( 0 , M − 2 ] . Hence the number of right-half-plane roots of H τ , δ , ν \mathcal{H}_{\tau,\delta,\nu} H τ , δ , ν is constant as δ \delta δ runs over ( 0 , M − 2 ] (0,M^{-2}] ( 0 , M − 2 ] (a root count can change only by a root crossing the axis or escaping to infinity, both excluded), and equals its value at δ ∗ \delta_* δ ∗ , namely 0 0 0 . Thus, H τ , δ , ν \mathcal{H}_{\tau,\delta,\nu} H τ , δ , ν is Hurwitz, i.e., P ∞ ( ⋅ ; δ , ν ) P_\infty(\cdot;\delta,\nu) P ∞ ( ⋅ ; δ , ν ) is Schur stable, for every 0 < δ ≤ M − 2 0<\delta\leq M^{-2} 0 < δ ≤ M − 2 . Therefore, for every 0 < c < 1 / M 0<c<1/M 0 < c < 1/ M , Theorem 3.22 with δ L = M − 2 \delta_L=M^{-2} δ L = M − 2 (so δ L = 1 / M \sqrt{\delta_L}=1/M δ L = 1/ M ) then yields the bound in (135) for all q ≤ q 0 ( τ , c ) q\leq q_0(\tau,c) q ≤ q 0 ( τ , c ) , i.e., for all κ ≥ κ 0 a l l ( τ , c ) : = q 0 ( τ , c ) − 2 \kappa\geq\kappa^{\mathrm{all}}_0(\tau,c):=q_0(\tau,c)^{-2} κ ≥ κ 0 all ( τ , c ) := q 0 ( τ , c ) − 2 .
The closed-form parameter choice in (134) is deliberately conservative: it is designed so that Lemma 3.24 can exclude every unit-circle crossing of the roots by hand, at the cost of a rate constant c < 1 / M c<1/M c < 1/ M that need not be optimal. For ν > τ \nu>\tau ν > τ , let δ L ( ν ) \delta_L(\nu) δ L ( ν ) denote the first gain at which P ∞ ( ⋅ ; δ , ν ) P_\infty(\cdot;\delta,\nu) P ∞ ( ⋅ ; δ , ν ) has a unit-circle root, as defined in appendix (302) . We write δ ˉ : = α L \bar{\delta}:=\alpha L δ ˉ := α L for the chosen gain, denoted by δ L \delta_L δ L in Theorem 3.22, and take 0 < δ ˉ < δ L ( ν ) 0<\bar{\delta}<\delta_L(\nu) 0 < δ ˉ < δ L ( ν ) . After fixing α = δ ˉ / L \alpha=\bar{\delta}/L α = δ ˉ / L and β = 1 − 2 δ ˉ q \beta=1-2\sqrt{\bar{\delta}}q β = 1 − 2 δ ˉ q within GMM, it is natural to ask which extrapolation (momentum) parameter ν \nu ν is best, and what the corresponding best certifiable rate ρ ≤ 1 − c / κ \rho\leq1-c/\sqrt{\kappa} ρ ≤ 1 − c / κ is. Since Theorem 3.22 delivers every 0 < c < δ L ( ν ) 0<c<\sqrt{\delta_L(\nu)} 0 < c < δ L ( ν ) whenever ν > τ \nu>\tau ν > τ , by choosing c 2 < δ ˉ < δ L ( ν ) c^2<\bar{\delta}<\delta_L(\nu) c 2 < δ ˉ < δ L ( ν ) , this is the design problem max ν > τ δ L ( ν ) \max_{\nu>\tau}\delta_L(\nu) max ν > τ δ L ( ν ) , which Supplementary Section B solves exactly (Corollary B.4). The maximum first-crossing gain is attained by an explicit matched extrapolation ν τ c r i t \nu_\tau^{\mathrm{crit}} ν τ crit of Definition B.3 in the appendix, with supremum of certified rate constants c τ c r i t = δ τ c r i t c_\tau^{\mathrm{crit}}=\sqrt{\delta_\tau^{\mathrm{crit}}} c τ crit = δ τ crit obeying the large-delay asymptotics in Proposition B.5 in the appendix
c τ c r i t = a ∗ cos a ∗ τ ( 1 + o ( 1 ) ) ≈ 0.7415 τ , ν τ c r i t = 2 a ∗ 2 τ ( 1 + o ( 1 ) ) ≈ 1.725 τ , c_\tau^{\mathrm{crit}}=\frac{a_*\sqrt{\cos a_*}}{\tau}(1+o(1))\approx\frac{0.7415}{\tau},\qquad
\nu_\tau^{\mathrm{crit}}=\frac{2}{a_*^2}\tau(1+o(1))\approx1.725\,\tau, c τ crit = τ a ∗ cos a ∗ ( 1 + o ( 1 )) ≈ τ 0.7415 , ν τ crit = a ∗ 2 2 τ ( 1 + o ( 1 )) ≈ 1.725 τ ,
where a ⋆ ≈ 1.0768 a_{\star} \approx1.0768 a ⋆ ≈ 1.0768 is the unique solution of the equation a tan a = 2 a \tan a = 2 a tan a = 2 on the interval ( 0 , π / 2 ) (0,\pi/2) ( 0 , π /2 ) . Two features of this optimum inform the parameters used throughout the paper: acceleration survives every fixed delay but its constant is asymptotically proportional to 1 / τ 1/\tau 1/ τ , and the optimal extrapolation grows linearly in the delay — a look-ahead the classical one-parameter families ( ν = 0 , ν ≈ 1 ) (\nu= 0, \nu\approx1) ( ν = 0 , ν ≈ 1 ) cannot express, which is precisely why they forfeit κ \sqrt{\kappa} κ -acceleration (Theorems 3.12 and 3.17 ).
The conservative choice ν = M − 1 2 ∼ 2 τ \nu= M - \frac{1}{2} \sim2\tau ν = M − 2 1 ∼ 2 τ of Theorem 3.25 carries this same scaling and differs from the optimum only in constants. Indeed, it yields ρ ≤ 1 − c / κ \rho\le1 - c/\sqrt{\kappa} ρ ≤ 1 − c / κ with c = 1 2 ( 2 τ + 1 ) ≈ 1 / ( 4 τ ) c = \frac{1}{2(2\tau+1)} \approx1/(4\tau) c = 2 ( 2 τ + 1 ) 1 ≈ 1/ ( 4 τ ) for large τ \tau τ , so c τ c r i t / c → a ⋆ cos a ⋆ / ( 1 / 4 ) < 3 c_{\tau}^{\mathrm{crit}}/c \to a_{\star}\sqrt{\cos a_{\star}}/(1/4) < 3 c τ crit / c → a ⋆ cos a ⋆ / ( 1/4 ) < 3 as τ → ∞ \tau\to\infty τ → ∞ ; the certified gap constant is thus asymptotically within a factor of three of this supremum. We adopt this choice throughout because it extends to the strongly convex case without requiring modification to the parameter choice, furnishing a single parameter setting valid for both quadratic and smooth strongly convex objectives.
Strongly convex smooth case: Acceleration with delays
We start with presenting a theorem that shows that the same parameter choice (studied in the quadratic case) also achieves acceleration in the strongly convex case, i.e., it achieves a convergence rate ρ = 1 − Θ ( 1 ( 2 τ + 1 ) κ ) \rho= 1 - \Theta\left(\frac{1}{(2\tau+1)\sqrt{\kappa}}\right) ρ = 1 − Θ ( ( 2 τ + 1 ) κ 1 ) .
Theorem 4.1 (arbitrary fixed-delay acceleration). Fix any integer τ ≥ 1 \tau\ge1 τ ≥ 1 , write M = 2 τ + 1 M = 2\tau+ 1 M = 2 τ + 1 , and fix any constant c ∈ ( 0 , 2 / 3 ) c \in(0,2/3) c ∈ ( 0 , 2/3 ) . There is a threshold κ 0 = κ 0 ( τ , c ) \kappa_0 = \kappa_0(\tau,c) κ 0 = κ 0 ( τ , c ) such that for every κ ≥ κ 0 \kappa\ge\kappa_0 κ ≥ κ 0 , every f ∈ S μ , L f \in\mathcal{S}_{\mu,L} f ∈ S μ , L and every finite initial history x − τ − 1 , … , x 0 x_{-\tau-1},\ldots,x_0 x − τ − 1 , … , x 0 , delayed GMM (14) with parameters
α = 1 ( 2 τ + 1 ) 2 L = 1 M 2 L , β = 1 − 1 ( 2 τ + 1 ) κ = 1 − 2 M κ , ν = 2 τ + 1 2 = M − 1 2 \alpha= \frac{1}{(2\tau+1)^2L} = \frac{1}{M^2L}, \qquad\beta= 1 - \frac{1}{(2\tau+1)\sqrt{\kappa}} = 1 - \frac{2}{M\sqrt{\kappa}}, \qquad\nu= 2\tau+ \frac{1}{2} = M - \frac{1}{2} α = ( 2 τ + 1 ) 2 L 1 = M 2 L 1 , β = 1 − ( 2 τ + 1 ) κ 1 = 1 − M κ 2 , ν = 2 τ + 2 1 = M − 2 1 satisfies
∥ x k − x ⋆ ∥ ≤ C ( 1 − c ( 2 τ + 1 ) κ ) k , k ≥ 0 , (136) \lVert x_k - x_{\star} \rVert\le C\left(1-\frac{c}{(2\tau+1)\sqrt{\kappa}}\right)^k, \qquad k \ge0, \tag*{(136)} ∥ x k − x ⋆ ∥ ≤ C ( 1 − ( 2 τ + 1 ) κ c ) k , k ≥ 0 , (136) for some finite constant C > 0 C > 0 C > 0 that may depend on the fixed parameters and the initial history relative to x ⋆ x_{\star} x ⋆ , but not on the iteration counter k ≥ 0 k \ge0 k ≥ 0 .
Before we establish the proof of Theorem 4.1 in § 5 , in the following, we provide two building blocks that the proof rests on.
Building Blocks for the Proof of Theorem 4.1
The first building block is the transfer function associated with the linear part of delayed GMM (§ 4.1.1 ). The key structural observation is that GMM can be represented as the interconnection of a linear plant (linear dynamical system) and a nonlinear gradient mapping. The linear plant captures the momentum averaging and the extrapolation from x k x_k x k to y k y_k y k , while the nonlinear component maps y k y_k y k to the gradient input ∇ f ( y k ) \nabla f(y_k) ∇ f ( y k ) . Thus, all nonlinearity in the GMM dynamics enters through the gradient mapping. The second building block is a hard integral quadratic constraint (IQC), valid on every finite horizon, satisfied by the gradient mapping of every f ∈ S μ , L f \in\mathcal{S}_{\mu,L} f ∈ S μ , L (§ 4.1.2 ). The role of the IQC is to encode this nonlinear relation through a quadratic inequality implied by smoothness and strong convexity. For the purposes of the present analysis, the hard IQC may therefore be viewed as a finite-horizon quadratic consequence of these assumptions, one that remains valid after summation over any finite number of iterations.
Standing normalization and parameters. The whole certificate is carried out under a fixed normalization and a fixed choice of parameters, which we record once here and use throughout the remainder of the paper. After scaling f f f by 1 / μ 1/\mu 1/ μ we work with
μ = 1 , L = q − 2 , q = κ − 1 / 2 ∈ ( 0 , 1 ] , (137) \mu= 1,\qquad L = q^{-2},\qquad q = \kappa^{-1/2} \in(0,1], \tag*{(137)} μ = 1 , L = q − 2 , q = κ − 1/2 ∈ ( 0 , 1 ] , (137)
and, for a fixed c ∈ ( 0 , 2 / 3 ) c \in(0,2/3) c ∈ ( 0 , 2/3 ) ,
α = q 2 M 2 , β = 1 − 2 q M , ν = M − 1 2 , ρ = 1 − c q M . (138) \alpha= \frac{q^2}{M^2},\qquad\beta= 1-\frac{2q}{M},\qquad\nu= M-\frac{1}{2},\qquad\rho= 1-\frac{cq}{M}. \tag*{(138)} α = M 2 q 2 , β = 1 − M 2 q , ν = M − 2 1 , ρ = 1 − M c q . (138)
Here ρ \rho ρ is the contraction rate to be certified, and we work in the large condition number regime which corresponds to the parameter q = κ − 1 / 2 q = \kappa^{-1/2} q = κ − 1/2 being small and around zero.
The transfer function of the linear plant
We analyze delayed GMM by viewing the method as a feedback interconnection between a linear time-invariant subsystem8 and the nonlinear gradient mapping. To expose this structure, we break the feedback relation u k = ∇ f ( y k ) u_k = \nabla f(y_k) u k = ∇ f ( y k ) and temporarily regard u = ( u k ) u = (u_k) u = ( u k ) as an exogenous input. The resulting state recursion is
x k + 1 = ( 1 + β ) x k − β x k − 1 − α u k − τ . (139) x_{k+1} = (1+\beta)x_k - \beta x_{k-1} - \alpha u_{k-\tau}. \tag*{(139)} x k + 1 = ( 1 + β ) x k − β x k − 1 − α u k − τ . (139)
The linear subsystem therefore consists of the delay, the momentum recursion, and the extrapolation from x k x_k x k to the evaluation point y k y_k y k , while the gradient map ∇ f \nabla f ∇ f is the only nonlinear component in this dynamical system. This feedback (Lur’e) decomposition is illustrated in Figure 1 . In particular, the factor z − τ z^{-\tau} z − τ represents the delay that maps the current gradient sequence u k u_k u k to the stale signal u k − τ u_{k-\tau} u k − τ entering the momentum recursion.
Figure 1. Feedback (Lur’e) decomposition of delayed GMM. Breaking the feedback relation u k = ∇ f ( y k ) u_k = \nabla f(y_k) u k = ∇ f ( y k ) separates the method into the interconnection of (i) the linear plant g τ , q ( z ) g_{\tau,q}(z) g τ , q ( z ) , which incorporates the delay, momentum recursion, and extrapolation, and (ii) the static nonlinear gradient map ∇ f \nabla f ∇ f .
Taking the bilateral z z z -transform of (139) , with zero initial conditions (x k = 0 x_k=0 x k = 0 for all k ≤ 0 k\le0 k ≤ 0 and u k = 0 u_k=0 u k = 0 for k < 0 k<0 k < 0 ) for the input-output calculation, gives
( z − ( 1 + β ) + β z − 1 ) X ( z ) = − α z − τ U ( z ) . (140) \left(z-(1+\beta)+\beta z^{-1}\right)X(z)=-\alpha z^{-\tau}U(z). \tag*{(140)} ( z − ( 1 + β ) + β z − 1 ) X ( z ) = − α z − τ U ( z ) . (140)
where X ( z ) X(z) X ( z ) and U ( z ) U(z) U ( z ) are the z z z -transforms of { x k } \{x_k\} { x k } and { u k } \{u_k\} { u k } . Multiplying by z z z and factoring the quadratic denominator yields
( z − 1 ) ( z − β ) X ( z ) = − α z 1 − τ U ( z ) , X ( z ) = − α z 1 − τ ( z − 1 ) ( z − β ) U ( z ) . (141) (z-1)(z-\beta)X(z)=-\alpha z^{1-\tau}U(z), \qquad X(z)=-\alpha\frac{z^{1-\tau}}{(z-1)(z-\beta)}U(z). \tag*{(141)} ( z − 1 ) ( z − β ) X ( z ) = − α z 1 − τ U ( z ) , X ( z ) = − α ( z − 1 ) ( z − β ) z 1 − τ U ( z ) . (141)
The evaluation point is the extrapolated point
y k = x k + ν ( x k − x k − 1 ) = ( 1 + ν ) x k − ν x k − 1 . (142) y_k=x_k+\nu(x_k-x_{k-1})=(1+\nu)x_k-\nu x_{k-1}. \tag*{(142)} y k = x k + ν ( x k − x k − 1 ) = ( 1 + ν ) x k − ν x k − 1 . (142)
so in the z z z -domain
Y ( z ) = ( 1 + ν − ν z − 1 ) X ( z ) = ( 1 + ν ) z − ν z X ( z ) . (143) Y(z)=(1+\nu-\nu z^{-1})X(z)=\frac{(1+\nu)z-\nu}{z}X(z). \tag*{(143)} Y ( z ) = ( 1 + ν − ν z − 1 ) X ( z ) = z ( 1 + ν ) z − ν X ( z ) . (143)
Combining (141) and (143) gives
Y ( z ) = − α z − τ ( ( 1 + ν ) z − ν ) ( z − 1 ) ( z − β ) U ( z ) . (144) Y(z)=-\alpha\frac{z^{-\tau}((1+\nu)z-\nu)}{(z-1)(z-\beta)}U(z). \tag*{(144)} Y ( z ) = − α ( z − 1 ) ( z − β ) z − τ (( 1 + ν ) z − ν ) U ( z ) . (144)
Finally, substituting the normalized parameters α = q 2 / M 2 \alpha=q^2/M^2 α = q 2 / M 2 and ν = M − 1 2 \nu=M-\frac{1}{2} ν = M − 2 1 (so that 1 + ν = M + 1 2 1+\nu=M+\frac{1}{2} 1 + ν = M + 2 1 ) gives the scalar transfer function from the gradient signal u k = ∇ f ( y k ) u_k=\nabla f(y_k) u k = ∇ f ( y k ) to the evaluation point y k y_k y k :
Y ( z ) = g τ , q ( z ) U ( z ) with g τ , q ( z ) : = − q 2 M 2 z − τ ( ( M + 1 2 ) z − ( M − 1 2 ) ) ( z − 1 ) ( z − β ) . (145) Y(z)=g_{\tau,q}(z)U(z)\quad\text{with}\quad g_{\tau,q}(z):=-\frac{q^2}{M^2}\frac{z^{-\tau}\left(\left(M+\frac{1}{2}\right)z-\left(M-\frac{1}{2}\right)\right)}{(z-1)(z-\beta)}. \tag*{(145)} Y ( z ) = g τ , q ( z ) U ( z ) with g τ , q ( z ) := − M 2 q 2 ( z − 1 ) ( z − β ) z − τ ( ( M + 2 1 ) z − ( M − 2 1 ) ) . (145)
where each factor in g τ , q ( z ) g_{\tau,q}(z) g τ , q ( z ) has a direct interpretation: z − τ z^{-\tau} z − τ represents τ \tau τ -step delay of u k u_k u k , the factor ( ( M + 1 2 ) z − ( M − 1 2 ) ) \left(\left(M+\frac{1}{2}\right)z-\left(M-\frac{1}{2}\right)\right) ( ( M + 2 1 ) z − ( M − 2 1 ) ) is the extrapolation from x k x_k x k to y k y_k y k , and ( z − 1 ) ( z − β ) (z-1)(z-\beta) ( z − 1 ) ( z − β ) is the denominator induced due to momentum recursion. We also introduce the transfer matrix
G τ , q ( z ) : = g τ , q ( z ) I d ∈ C d × d with Y ( z ) = g τ , q ( z ) U ( z ) = G τ , q ( z ) U ( z ) . (146) G_{\tau,q}(z):=g_{\tau,q}(z)I_d\in\mathbb{C}^{d\times d}\quad\text{with}\quad Y(z)=g_{\tau,q}(z)U(z)=G_{\tau,q}(z)U(z). \tag*{(146)} G τ , q ( z ) := g τ , q ( z ) I d ∈ C d × d with Y ( z ) = g τ , q ( z ) U ( z ) = G τ , q ( z ) U ( z ) . (146)
Integral quadratic constraints (IQCs) associated to the gradient map f f f
In § 4.1.1 , we isolate and study the linear part of delayed GMM by treating the gradient values u k u_k u k as an external input, and derive the corresponding transfer function in Eq. (145) . However, in the GMM algorithm, u k u_k u k is not an arbitrary input but is generated by the non-linear gradient map u k = ∇ f ( y k ) u_k=\nabla f(y_k) u k = ∇ f ( y k ) . In this subsection, we restore this feedback relation and characterize the constraints that the gradient map imposes on the input-output pairs ( y k , u k ) (y_k,u_k) ( y k , u k ) .
Two features will be important in what follows. First, the inequalities hold on every finite horizon; this will later allow us to sum them directly within a finite-horizon Lyapunov estimate. Second, in the rest of the paper, without loss of generality, we translate the minimizer to the origin, so ∇ f ( 0 ) = 0 \nabla f(0)=0 ∇ f ( 0 ) = 0 . This can clearly be obtained by simply replacing x k x_k x k by x k − x ⋆ x_k-x_\star x k − x ⋆ and y k y_k y k by y k − x ⋆ y_k-x_\star y k − x ⋆ in the subsequent analysis. This normalization will simplify our analysis.
For any k ∈ Z + k\in\mathbb{Z}_+ k ∈ Z + , given a graph pair u k = ∇ f ( y k ) u_k=\nabla f(y_k) u k = ∇ f ( y k ) , define
r k : = u k − μ y k , s k : = L y k − u k . (147) r_k:=u_k-\mu y_k,\qquad s_k:=Ly_k-u_k. \tag*{(147)} r k := u k − μ y k , s k := L y k − u k . (147)
By (13) , for all indices i , j ∈ Z + i,j\in\mathbb{Z}_{+} i , j ∈ Z + ,
⟨ r i − r j , s i − s j ⟩ ≥ 0 , ⟨ r i , s i ⟩ ≥ 0. (148) \langle r_i-r_j,s_i-s_j\rangle\ge0,\qquad\langle r_i,s_i\rangle\ge0. \tag*{(148)} ⟨ r i − r j , s i − s j ⟩ ≥ 0 , ⟨ r i , s i ⟩ ≥ 0. (148)
The second inequality is obtained from the first by setting one ( r , s ) (r,s) ( r , s ) pair to ( 0 , 0 ) (0,0) ( 0 , 0 ) corresponding to the minimizer, i.e., x ⋆ = 0 x_\star=0 x ⋆ = 0 with ∇ f ( x ⋆ ) = 0 \nabla f(x_\star)=0 ∇ f ( x ⋆ ) = 0 . We shall use the following equivalent convex-potential form: Put γ : = L − μ ≥ 0 \gamma:=L-\mu\ge0 γ := L − μ ≥ 0 . Throughout this section we assume μ < L \mu<L μ < L , so that γ : = L − μ > 0 \gamma:=L-\mu>0 γ := L − μ > 0 . If L = μ L=\mu L = μ , then r k = s k = 0 r_k=s_k=0 r k = s k = 0 for all k k k and every inequality below holds trivially. We call ( y , u ) ∈ R d × R d (y,u)\in\mathbb{R}^d\times\mathbb{R}^d ( y , u ) ∈ R d × R d a point–gradient pair of f f f if u = ∇ f ( y ) u=\nabla f(y) u = ∇ f ( y ) . If γ > 0 \gamma>0 γ > 0 , define
F ( y ) : = f ( y ) − μ 2 ∥ y ∥ 2 . F(y):=f(y)-\frac{\mu}{2}\lVert y\rVert^2. F ( y ) := f ( y ) − 2 μ ∥ y ∥ 2 .
Then F F F is convex and γ \gamma γ -smooth; hence, its Fenchel conjugate F ∗ F^\ast F ∗ is 1 / γ 1/\gamma 1/ γ -strongly convex, which implies that ψ ( ⋅ ) \psi(\cdot) ψ ( ⋅ ) is merely convex, where
ψ ( r ) : = γ F ∗ ( r ) − 1 2 ∥ r ∥ 2 − γ F ∗ ( 0 ) . (149) \psi(r):=\gamma F^\ast(r)-\frac{1}{2}\lVert r\rVert^2-\gamma F^\ast(0). \tag*{(149)} ψ ( r ) := γ F ∗ ( r ) − 2 1 ∥ r ∥ 2 − γ F ∗ ( 0 ) . (149)
Note that F ∗ ( 0 ) = − inf y F ( y ) = − F ( 0 ) F^\ast(0)=-\inf_y F(y)=-F(0) F ∗ ( 0 ) = − inf y F ( y ) = − F ( 0 ) . Indeed, F F F is convex and ∇ F ( 0 ) = 0 \nabla F(0)=0 ∇ F ( 0 ) = 0 , so zero is a minimizer of F F F , where we used the fact that ∇ f ( 0 ) = 0 \nabla f(0)=0 ∇ f ( 0 ) = 0 . As we discuss next, this construction allows us to express s k s_k s k as a subgradient of ψ \psi ψ at r k r_k r k . Indeed, since r = ∇ F ( y ) r=\nabla F(y) r = ∇ F ( y ) implies y ∈ ∂ F ∗ ( r ) y\in\partial F^\ast(r) y ∈ ∂ F ∗ ( r ) , we have
s = γ y − r ∈ ∂ ψ ( r ) . (150) s=\gamma y-r\in\partial\psi(r). \tag*{(150)} s = γ y − r ∈ ∂ ψ ( r ) . (150)
Moreover 0 ∈ ∂ ψ ( 0 ) 0\in\partial\psi(0) 0 ∈ ∂ ψ ( 0 ) and ψ ( 0 ) = 0 \psi(0)=0 ψ ( 0 ) = 0 ; hence, convexity of ψ \psi ψ gives
ψ ( r ) ≥ 0 for all r . (151) \psi(r)\ge0\qquad\text{for all }r. \tag*{(151)} ψ ( r ) ≥ 0 for all r . (151)
The next lemma shows s k s_k s k is a subgradient of ψ \psi ψ evaluated at r k r_k r k . Given a finite trajectory { ( y k , u k ) } k = 0 N \{(y_k,u_k)\}_{k=0}^N {( y k , u k ) } k = 0 N , this lemma also extends the sequence by padding it with ( 0 , 0 ) (0,0) ( 0 , 0 ) for all ( k < 0 ) (k<0) ( k < 0 ) and ( k > N ) (k>N) ( k > N ) . This convention simplifies the notation in the subsequent analysis by allowing us to avoid treating the boundary terms for k < 0 k<0 k < 0 and k > N k>N k > N separately.
Lemma 4.2 (padding a finite trajectory with the minimizer). Let f ∈ S μ , L f\in S_{\mu,L} f ∈ S μ , L with ∇ f ( 0 ) = 0 \nabla f(0)=0 ∇ f ( 0 ) = 0 , let ( y k , u k ) = ( y k , ∇ f ( y k ) ) (y_k,u_k)=(y_k,\nabla f(y_k)) ( y k , u k ) = ( y k , ∇ f ( y k )) for 0 ≤ k ≤ N 0\le k\le N 0 ≤ k ≤ N , and set ( y k , u k ) : = ( 0 , 0 ) (y_k,u_k):=(0,0) ( y k , u k ) := ( 0 , 0 ) for k ∈ Z ∖ { 0 , … , N } k\in\mathbb{Z}\setminus\{0,\ldots,N\} k ∈ Z ∖ { 0 , … , N } . Define r k : = u k − μ y k r_k:=u_k-\mu y_k r k := u k − μ y k and s k : = L y k − u k s_k:=Ly_k-u_k s k := L y k − u k for k ∈ Z k\in\mathbb{Z} k ∈ Z . Then:
(a) s k ∈ ∂ ψ ( r k ) s_k\in\partial\psi(r_k) s k ∈ ∂ ψ ( r k ) and ψ ( r k ) ≥ 0 \psi(r_k)\ge0 ψ ( r k ) ≥ 0 for every k ∈ Z k\in\mathbb{Z} k ∈ Z ; consequently,
⟨ r i − r j , s i − s j ⟩ ≥ 0 and ⟨ r i , s i ⟩ ≥ 0 for all i , j ∈ Z . \langle r_i-r_j,s_i-s_j\rangle\ge0\qquad\text{and}\qquad\langle r_i,s_i\rangle\ge0\qquad\text{for all }i,j\in\mathbb{Z}. ⟨ r i − r j , s i − s j ⟩ ≥ 0 and ⟨ r i , s i ⟩ ≥ 0 for all i , j ∈ Z . (b) r k = s k = 0 r_k=s_k=0 r k = s k = 0 for k ∉ { 0 , … , N } k\notin\{0,\ldots,N\} k ∈ / { 0 , … , N } ; hence every sum ∑ k ∈ Z ⟨ r k , s k − j ⟩ \sum_{k\in\mathbb{Z}}\langle r_k,s_{k-j}\rangle ∑ k ∈ Z ⟨ r k , s k − j ⟩ has at most N + 1 N+1 N + 1 nonzero terms and, for every j ∈ Z j\in\mathbb{Z} j ∈ Z ,
∑ k ∈ Z ⟨ r k , s k − j ⟩ = ∑ k ∈ Z ⟨ r k + j , s k ⟩ . \sum_{k\in\mathbb{Z}}\langle r_k,s_{k-j}\rangle=\sum_{k\in\mathbb{Z}}\langle r_{k+j},s_k\rangle. k ∈ Z ∑ ⟨ r k , s k − j ⟩ = k ∈ Z ∑ ⟨ r k + j , s k ⟩ .
Proof. Since ∇ f ( 0 ) = 0 \nabla f(0)=0 ∇ f ( 0 ) = 0 , the pair ( 0 , 0 ) (0,0) ( 0 , 0 ) is a point–gradient pair of f f f , so every pair of the extended sequence is of the form ( y , ∇ f ( y ) ) (y,\nabla f(y)) ( y , ∇ f ( y )) for some y ∈ R d y\in\mathbb{R}^d y ∈ R d (y = y k y=y_k y = y k for 0 ≤ k ≤ N 0\le k\le N 0 ≤ k ≤ N and y = 0 y=0 y = 0 otherwise). Hence (150) and (151) give s k ∈ ∂ ψ ( r k ) s_k\in\partial\psi(r_k) s k ∈ ∂ ψ ( r k ) and ψ ( r k ) ≥ 0 \psi(r_k)\ge0 ψ ( r k ) ≥ 0 for every k ∈ Z k\in\mathbb{Z} k ∈ Z . Monotonicity of the subdifferential of the convex function ψ \psi ψ yields ⟨ r i − r j , s i − s j ⟩ ≥ 0 \langle r_i-r_j,s_i-s_j\rangle\ge0 ⟨ r i − r j , s i − s j ⟩ ≥ 0 , and choosing j ∉ { 0 , … , N } j\notin\{0,\ldots,N\} j ∈ / { 0 , … , N } , for which ( r j , s j ) = ( 0 , 0 ) (r_j,s_j)=(0,0) ( r j , s j ) = ( 0 , 0 ) , yields ⟨ r i , s i ⟩ ≥ 0 \langle r_i,s_i\rangle\ge0 ⟨ r i , s i ⟩ ≥ 0 ; this proves (a). For (b), y k = u k = 0 y_k=u_k=0 y k = u k = 0 gives r k = s k = 0 r_k=s_k=0 r k = s k = 0 for k ∉ { 0 , … , N } k\notin\{0,\ldots,N\} k ∈ / { 0 , … , N } . Hence, for every j ∈ Z j\in\mathbb{Z} j ∈ Z , the sum ∑ k ∈ Z ⟨ r k , s k − j ⟩ \sum_{k\in\mathbb{Z}}\langle r_k,s_{k-j}\rangle ∑ k ∈ Z ⟨ r k , s k − j ⟩ has at most N + 1 N+1 N + 1 nonzero terms and is therefore finite. The change of variables ℓ = k − j \ell=k-j ℓ = k − j then gives ∑ k ∈ Z ⟨ r k , s k − j ⟩ = ∑ ℓ ∈ Z ⟨ r ℓ + j , s ℓ ⟩ \sum_{k\in\mathbb{Z}}\langle r_k,s_{k-j}\rangle=\sum_{\ell\in\mathbb{Z}}\langle r_{\ell+j},s_\ell\rangle ∑ k ∈ Z ⟨ r k , s k − j ⟩ = ∑ ℓ ∈ Z ⟨ r ℓ + j , s ℓ ⟩ , which proves (b). ∎
Consider any finite cycle of indices k 0 , … , k n , k n + 1 k_0,\ldots,k_n,k_{n+1} k 0 , … , k n , k n + 1 where k n + 1 = k 0 k_{n+1}=k_0 k n + 1 = k 0 . Based on the previous lemma, if we apply the subgradient inequality for the convex function ψ \psi ψ , we get
ψ ( r k i + 1 ) − ψ ( r k i ) ≥ ⟨ s k i , r k i + 1 − r k i ⟩ , ∀ i = 0 , … , n , \psi(r_{k_{i+1}})-\psi(r_{k_i}) \ge\langle s_{k_i},r_{k_{i+1}}-r_{k_i}\rangle,\qquad\forall i=0,\ldots,n, ψ ( r k i + 1 ) − ψ ( r k i ) ≥ ⟨ s k i , r k i + 1 − r k i ⟩ , ∀ i = 0 , … , n ,
i.e., it holds successively around the cycle. Summing over i ∈ { 0 , … , n } i\in\{0,\ldots,n\} i ∈ { 0 , … , n } , we obtain that for every finite cycle of indices k 0 , … , k n , k n + 1 = k 0 k_0,\ldots,k_n,k_{n+1}=k_0 k 0 , … , k n , k n + 1 = k 0 , we have
∑ i = 0 n ⟨ s k i , r k i + 1 − r k i ⟩ ≤ 0. \sum_{i=0}^{n}\langle s_{k_i},r_{k_{i+1}}-r_{k_i}\rangle\le0. i = 0 ∑ n ⟨ s k i , r k i + 1 − r k i ⟩ ≤ 0.
This is referred to as the cyclic monotonicity property in the literature. By Rockafellar [45 ] , pairs ( r k , s k ) (r_k,s_k) ( r k , s k ) that satisfy such a property arise from the subdifferential of a convex function, here ψ \psi ψ of (149) .
The inequalities below involve finite sums of centered iterate–gradient pairs along a finite trajectory. Quadratic constraints of this form, which remain valid on every finite horizon, are referred to as hard integral quadratic constraints (hard IQCs). This finite-horizon validity distinguishes hard IQCs from soft IQCs, which are required to hold only over the entire infinite-horizon signal; see, e.g., [36 , 34 ] .
We use Zames–Falb IQCs to capture relations between gradients evaluated at different iterates. The multiplier family originates with Zames and Falb [57 ] , while the general IQC framework was developed by Megretski and Rantzer [36 ] . For smooth strongly convex gradient maps, Lessard, Recht, and Packard [34 Lemma 9] establish a causal finite-horizon (hard) Zames–Falb IQC. We recall this constraint in our notation and provide a self-contained proof. In this setting, the IQC is used as a tool for analyzing the stability of the feedback interconnection between the linear GMM dynamics and the nonlinear gradient map. Rather than modeling the nonlinear block exactly, the IQC specifies a quadratic constraint that every admissible gradient map must satisfy, so stability can be established uniformly over the entire class of nonlinearities consistent with the assumption on f f f . The frequency-domain formulation is useful because the linear dynamics were already represented in § 4.1.1 by the transfer function g τ , q ( z ) g_{\tau,q}(z) g τ , q ( z ) . Using Parseval’s identity, the time-domain IQC can be converted into an equivalent frequency-domain quadratic inequality, allowing the linear transfer function and the non-linear constraint to be combined directly in a single stability analysis. We now present a Zames–Falb IQC for the nonlinearity ∇ f \nabla f ∇ f .
Lemma 4.3 (hard Zames–Falb IQC). Let h 1 , … , h J h_1,\ldots,h_J h 1 , … , h J be a nonnegative sequence with ∑ j = 1 J h j ≤ 1 \sum_{j=1}^{J}h_j\le1 ∑ j = 1 J h j ≤ 1 , and set h : C → C h:\mathbb{C}\to\mathbb{C} h : C → C such that h ( z ) : = ∑ j = 1 J h j z − j h(z):=\sum_{j=1}^{J}h_jz^{-j} h ( z ) := ∑ j = 1 J h j z − j . Assume the minimizer has been translated to the origin, so that ∇ f ( 0 ) = 0 \nabla f(0)=0 ∇ f ( 0 ) = 0 . Fix any horizon N ≥ 0 N\ge0 N ≥ 0 and any finite trajectory of point–gradient pairs { ( y k , u k ) } k = 0 N \{(y_k,u_k)\}_{k=0}^{N} {( y k , u k ) } k = 0 N with u k = ∇ f ( y k ) u_k=\nabla f(y_k) u k = ∇ f ( y k ) , and zero-extend it by ( y k , u k ) = ( 0 , 0 ) (y_k,u_k)=(0,0) ( y k , u k ) = ( 0 , 0 ) for k ∉ { 0 , … , N } k\notin\{0,\ldots,N\} k ∈ / { 0 , … , N } , so that the slope-shifted sequences r k = u k − μ y k r_k=u_k-\mu y_k r k = u k − μ y k and s k = L y k − u k s_k=Ly_k-u_k s k = L y k − u k have finite support and all sums below are finite. Then, for this horizon,
2 ∑ k ⟨ r k , s k ⟩ − 2 ∑ j = 1 J h j ∑ k ⟨ r k , s k − j ⟩ ≥ 0 , (152) 2\sum_k\langle r_k,s_k\rangle-2\sum_{j=1}^{J}h_j\sum_k\langle r_k,s_{k-j}\rangle\ge0, \tag*{(152)} 2 k ∑ ⟨ r k , s k ⟩ − 2 j = 1 ∑ J h j k ∑ ⟨ r k , s k − j ⟩ ≥ 0 , (152) where each ∑ k \sum_k ∑ k runs over Z \mathbb{Z} Z but has at most N + 1 N+1 N + 1 nonzero terms. Equivalently, by Parseval,
1 2 π ∫ 0 2 π 2 Re ( [ ( 1 − h ( e i θ ) ) R ( e i θ ) ] ∗ S ( e i θ ) ) d θ ≥ 0 , (153) \frac{1}{2\pi}\int_{0}^{2\pi}2\operatorname{Re}\left([(1-h(e^{i\theta}))R(e^{i\theta})]^*S(e^{i\theta})\right)\,\mathrm{d}\theta\ge0, \tag*{(153)} 2 π 1 ∫ 0 2 π 2 Re ( [( 1 − h ( e i θ )) R ( e i θ ) ] ∗ S ( e i θ ) ) d θ ≥ 0 , (153) where R , S R,S R , S are the discrete-time Fourier transforms of the zero-extended sequences r , s r,s r , s , i.e., R ( e i θ ) : = ∑ k ∈ Z r k e − i k θ R(e^{i\theta}):=\sum_{k\in\mathbb{Z}}r_ke^{-ik\theta} R ( e i θ ) := ∑ k ∈ Z r k e − ik θ and S ( e i θ ) : = ∑ k ∈ Z s k e − i k θ S(e^{i\theta}):=\sum_{k\in\mathbb{Z}}s_ke^{-ik\theta} S ( e i θ ) := ∑ k ∈ Z s k e − ik θ . Since (152) holds for every horizon N N N separately, with
no requirement on the trajectory beyond N N N , the constraint is hard: it may be applied to any finite truncation without a tail term.
Finally, substituting r k = u k − μ y k r_k = u_k - \mu y_k r k = u k − μ y k and s k = L y k − u k s_k = Ly_k - u_k s k = L y k − u k expresses the integrand of (153) , at each frequency θ ∈ [ 0 , 2 π ) \theta\in[0,2\pi) θ ∈ [ 0 , 2 π ) , as a Hermitian form in the original signals:
2 Re [ ( 1 − h ( e i θ ) ) R ( e i θ ) ∗ S ( e i θ ) ] = [ Y ( e i θ ) U ( e i θ ) ] ∗ Π μ , L , d ( h ( e i θ ) ) [ Y ( e i θ ) U ( e i θ ) ] . 2\operatorname{Re}\left[(1-h(e^{i\theta}))R(e^{i\theta})^*S(e^{i\theta})\right]
=
\begin{bmatrix} Y(e^{i\theta}) \\ U(e^{i\theta}) \end{bmatrix}^{*}
\Pi_{\mu,L,d}(h(e^{i\theta}))
\begin{bmatrix} Y(e^{i\theta}) \\ U(e^{i\theta}) \end{bmatrix}. 2 Re [ ( 1 − h ( e i θ )) R ( e i θ ) ∗ S ( e i θ ) ] = [ Y ( e i θ ) U ( e i θ ) ] ∗ Π μ , L , d ( h ( e i θ )) [ Y ( e i θ ) U ( e i θ ) ] .
where Y , U ∈ C d Y,U \in\mathbb{C}^{d} Y , U ∈ C d are the Fourier transforms of y , u y,u y , u , and writing the scalar h = h ( e i θ ) h=h(e^{i\theta}) h = h ( e i θ ) for brevity,
Π μ , L , d ( h ) : = Π μ , L ( h ) ⊗ I d , where Π μ , L ( h ) : = [ − μ L ( 2 − h − h ˉ ) L ( 1 − h ˉ ) + μ ( 1 − h ) L ( 1 − h ) + μ ( 1 − h ˉ ) − ( 2 − h − h ˉ ) ] . (154) \Pi_{\mu,L,d}(h):=\Pi_{\mu,L}(h)\otimes I_d,\qquad\text{where}\qquad
\Pi_{\mu,L}(h):=
\begin{bmatrix}
-\mu L(2-h-\bar h) & L(1-\bar h)+\mu(1-h)\\
L(1-h)+\mu(1-\bar h) & -(2-h-\bar h)
\end{bmatrix}. \tag*{(154)} Π μ , L , d ( h ) := Π μ , L ( h ) ⊗ I d , where Π μ , L ( h ) := [ − μL ( 2 − h − h ˉ ) L ( 1 − h ) + μ ( 1 − h ˉ ) L ( 1 − h ˉ ) + μ ( 1 − h ) − ( 2 − h − h ˉ ) ] . (154)
Proof. All sums are finite by the zero extension (Lemma 4.2 ). For a fixed lag j ≥ 1 j\ge1 j ≥ 1 , the subgradient inequality for s k ∈ ∂ ψ ( r k ) s_k\in\partial\psi(r_k) s k ∈ ∂ ψ ( r k ) gives
ψ ( r k + j ) ≥ ψ ( r k ) + ⟨ s k , r k + j − r k ⟩ . \psi(r_{k+j})\ge\psi(r_k)+\langle s_k,r_{k+j}-r_k\rangle. ψ ( r k + j ) ≥ ψ ( r k ) + ⟨ s k , r k + j − r k ⟩ .
which implies that
⟨ s k , r k − r k + j ⟩ ≥ ψ ( r k ) − ψ ( r k + j ) . \langle s_k,r_k-r_{k+j}\rangle\ge\psi(r_k)-\psi(r_{k+j}). ⟨ s k , r k − r k + j ⟩ ≥ ψ ( r k ) − ψ ( r k + j ) .
Summing this inequality over all k ∈ Z k\in\mathbb{Z} k ∈ Z , the right-hand side telescopes to ∑ k ψ ( r k ) − ∑ k ψ ( r k + j ) = 0 \sum_k\psi(r_k)-\sum_k\psi(r_{k+j})=0 ∑ k ψ ( r k ) − ∑ k ψ ( r k + j ) = 0 since k ↦ k + j k\mapsto k+j k ↦ k + j is a bijection of Z \mathbb{Z} Z and { r k } \{r_k\} { r k } has finite support; thus, we get
∑ k ⟨ s k , r k − r k + j ⟩ ≥ 0. (155) \sum_k\langle s_k,r_k-r_{k+j}\rangle\ge0. \tag*{(155)} k ∑ ⟨ s k , r k − r k + j ⟩ ≥ 0. (155)
Since
∑ k ⟨ s k , r k + j ⟩ = ∑ k ⟨ r k , s k − j ⟩ , \sum_k\langle s_k,r_{k+j}\rangle=\sum_k\langle r_k,s_{k-j}\rangle, k ∑ ⟨ s k , r k + j ⟩ = k ∑ ⟨ r k , s k − j ⟩ ,
by Lemma 4.2 , the inequality (155) implies that
A − B j ≥ 0 , A : = ∑ k ⟨ r k , s k ⟩ , B j : = ∑ k ⟨ r k , s k − j ⟩ . A-B_j\ge0,\qquad A:=\sum_k\langle r_k,s_k\rangle,\qquad B_j:=\sum_k\langle r_k,s_{k-j}\rangle. A − B j ≥ 0 , A := k ∑ ⟨ r k , s k ⟩ , B j := k ∑ ⟨ r k , s k − j ⟩ .
Moreover, by (148) , we also have A ≥ 0 A\ge0 A ≥ 0 . Therefore, with H : = ∑ j = 1 J h j ≤ 1 H:=\sum_{j=1}^{J}h_j\le1 H := ∑ j = 1 J h j ≤ 1 , it follows that
A − ∑ j = 1 J h j B j = ( 1 − H ) A + ∑ j = 1 J h j ( A − B j ) ≥ 0 , A-\sum_{j=1}^{J}h_jB_j=(1-H)A+\sum_{j=1}^{J}h_j(A-B_j)\ge0, A − j = 1 ∑ J h j B j = ( 1 − H ) A + j = 1 ∑ J h j ( A − B j ) ≥ 0 ,
which yields (152) after multiplying it by 2.
Parseval’s identity and the time-shift property of the discrete-time Fourier transform give
∑ k ⟨ r k , s k ⟩ = 1 2 π ∫ 0 2 π R ( e i θ ) ∗ S ( e i θ ) d θ , and ∑ k ⟨ r k , s k − j ⟩ = 1 2 π ∫ 0 2 π e − i j θ R ( e i θ ) ∗ S ( e i θ ) d θ . \sum_k\langle r_k,s_k\rangle=\frac{1}{2\pi}\int_0^{2\pi}R(e^{i\theta})^*S(e^{i\theta})\,d\theta,\qquad
\text{and}\qquad
\sum_k\langle r_k,s_{k-j}\rangle=\frac{1}{2\pi}\int_0^{2\pi}e^{-ij\theta}R(e^{i\theta})^*S(e^{i\theta})\,d\theta. k ∑ ⟨ r k , s k ⟩ = 2 π 1 ∫ 0 2 π R ( e i θ ) ∗ S ( e i θ ) d θ , and k ∑ ⟨ r k , s k − j ⟩ = 2 π 1 ∫ 0 2 π e − ij θ R ( e i θ ) ∗ S ( e i θ ) d θ .
The time-domain expression is real, so after taking the real part of the integral we obtain (153) . Because the zero-extended sequences are supported on { 0 , … , N } \{0,\ldots,N\} { 0 , … , N } , their discrete-time Fourier transforms are the finite trigonometric polynomials:
Y ( e i θ ) = ∑ k = 0 N y k e − i k θ , U ( e i θ ) = ∑ k = 0 N u k e − i k θ , Y(e^{i\theta})=\sum_{k=0}^{N}y_ke^{-ik\theta},\qquad U(e^{i\theta})=\sum_{k=0}^{N}u_ke^{-ik\theta}, Y ( e i θ ) = k = 0 ∑ N y k e − ik θ , U ( e i θ ) = k = 0 ∑ N u k e − ik θ ,
R ( e i θ ) = ∑ k = 0 N r k e − i k θ , S ( e i θ ) = ∑ k = 0 N s k e − i k θ , R(e^{i\theta})=\sum_{k=0}^{N}r_k e^{-ik\theta},\qquad S(e^{i\theta})=\sum_{k=0}^{N}s_k e^{-ik\theta}, R ( e i θ ) = k = 0 ∑ N r k e − ik θ , S ( e i θ ) = k = 0 ∑ N s k e − ik θ ,
with r k = u k − μ y k r_k=u_k-\mu y_k r k = u k − μ y k and s k = L y k − u k s_k=Ly_k-u_k s k = L y k − u k ; by linearity, R = U − μ Y R=U-\mu Y R = U − μ Y and S = L Y − U S=LY-U S = L Y − U . Then, we can express the integrand of (153) , at each frequency θ ∈ [ 0 , 2 π ) \theta\in[0,2\pi) θ ∈ [ 0 , 2 π ) :
2 Re [ ( 1 − h ( e i θ ) ) R ( e i θ ) ∗ S ( e i θ ) ] = [ Y ( e i θ ) U ( e i θ ) ] ∗ Π μ , L , d ( h ( e i θ ) ) [ Y ( e i θ ) U ( e i θ ) ] , (156) 2\operatorname{Re}\left[(1-h(e^{i\theta}))R(e^{i\theta})^*S(e^{i\theta})\right]
=
\begin{bmatrix}Y(e^{i\theta})\\U(e^{i\theta})\end{bmatrix}^{*}
\Pi_{\mu,L,d}(h(e^{i\theta}))
\begin{bmatrix}Y(e^{i\theta})\\U(e^{i\theta})\end{bmatrix}, \tag*{(156)} 2 Re [ ( 1 − h ( e i θ )) R ( e i θ ) ∗ S ( e i θ ) ] = [ Y ( e i θ ) U ( e i θ ) ] ∗ Π μ , L , d ( h ( e i θ )) [ Y ( e i θ ) U ( e i θ ) ] , (156)
which is the claimed Hermitian form. This completes the proof.
The Hermitian matrix-valued function Π μ , L , d ( h ( e i θ ) ) \Pi_{\mu,L,d}(h(e^{i\theta})) Π μ , L , d ( h ( e i θ )) is called a multiplier : a Hermitian weight assigned to each frequency θ ∈ [ 0 , 2 π ) \theta\in[0,2\pi) θ ∈ [ 0 , 2 π ) , in the sense of the integral-quadratic-constraint framework [36 ] . The specific multiplier used here, generated by the nonnegative averaging weight h h h , is a Zames–Falb multiplier [57 , 11 ] . Its scalar matrix representation (154) coincides with the standard Zames–Falb form; see [6 Proposition 14, Eq. (6)] .
To certify an exponential convergence rate, we use rate-weighted Zames–Falb IQCs. Boczar, Lessard, and Recht Boczar, Lessard, and Recht [6 Theorem 16] develop exponential-rate Zames–Falb multipliers, while Lessard, Recht, and Packard Lessard, Recht, and Packard [34 Lemma 10] establish a hard rate-weighted IQC for the one-step delay case. The following lemma establishes a rate-weighted IQC over a finite time horizon for nonnegative weights assigned to finitely many past iterations. We subsequently specialize these weights to the preceding τ \tau τ iterations. The argument of the preceding lemma requires ∑ j = 1 J h j ≤ 1 \sum_{j=1}^{J}h_j\le1 ∑ j = 1 J h j ≤ 1 . Such a property will still hold after replacing the lag-j j j coefficient h j h_j h j by the effective coefficient h ^ j : = ρ j h j \hat{h}_j:=\rho^j h_j h ^ j := ρ j h j for j = 1 , … , J j=1,\ldots,J j = 1 , … , J . Indeed, if h j ≥ 0 h_j\ge0 h j ≥ 0 for all j j j , 0 < ρ ≤ 1 0<\rho\le1 0 < ρ ≤ 1 , and ∑ j = 1 J h j ≤ 1 \sum_{j=1}^{J}h_j\le1 ∑ j = 1 J h j ≤ 1 , then h ^ j ≥ 0 \hat{h}_j\ge0 h ^ j ≥ 0 for all j j j and ∑ j = 1 J h ^ j = ∑ j = 1 J ρ j h j ≤ ∑ j = 1 J h j ≤ 1 \sum_{j=1}^{J}\hat{h}_j=\sum_{j=1}^{J}\rho^j h_j\le\sum_{j=1}^{J}h_j\le1 ∑ j = 1 J h ^ j = ∑ j = 1 J ρ j h j ≤ ∑ j = 1 J h j ≤ 1 .
Thus the weighted coefficients satisfy the same admissibility conditions as the original coefficients. The following argument shows that a similar IQC also holds for the rate-weighted sequences.
Lemma 4.4 (rate-weighted hard Zames–Falb IQC). Let 0 < ρ ≤ 1 0<\rho\le1 0 < ρ ≤ 1 . Given the finite trajectory of point–gradient pairs ( y k , u k ) = ( y k , ∇ f ( y k ) ) (y_k,u_k)=(y_k,\nabla f(y_k)) ( y k , u k ) = ( y k , ∇ f ( y k )) , 0 ≤ k ≤ N 0\le k\le N 0 ≤ k ≤ N , extend it to all integers by setting ( y k , u k ) = ( 0 , 0 ) (y_k,u_k)=(0,0) ( y k , u k ) = ( 0 , 0 ) for k ∉ { 0 , … , N } k\notin\{0,\ldots,N\} k ∈ / { 0 , … , N } , and put y ^ k = ρ − k y k \widehat{y}_k=\rho^{-k}y_k y k = ρ − k y k , u ^ k = ρ − k u k \widehat{u}_k=\rho^{-k}u_k u k = ρ − k u k . Let
h ρ ( z ) = ∑ j = 1 J ρ j h j z − j , h j ≥ 0 , ∑ j = 1 J h j ≤ 1. (157) h_\rho(z)=\sum_{j=1}^{J}\rho^j h_j z^{-j},\qquad h_j\ge0,\qquad\sum_{j=1}^{J}h_j\le1. \tag*{(157)} h ρ ( z ) = j = 1 ∑ J ρ j h j z − j , h j ≥ 0 , j = 1 ∑ J h j ≤ 1. (157) Then the weighted, zero-extended pair ( y ^ , u ^ ) (\widehat{y},\widehat{u}) ( y , u ) satisfies the hard IQC generated by Π μ , L , d ( h ρ ) \Pi_{\mu,L,d}(h_\rho) Π μ , L , d ( h ρ ) , with Π μ , L , d ( ⋅ ) \Pi_{\mu,L,d}(\cdot) Π μ , L , d ( ⋅ ) as in (154) ; writing Y ^ , U ^ \widehat{Y},\widehat{U} Y , U for the Fourier transforms of y ^ , u ^ \widehat{y},\widehat{u} y , u ,
1 2 π ∫ 0 2 π [ Y ^ ( e i θ ) U ^ ( e i θ ) ] ∗ Π μ , L , d ( h ρ ( e i θ ) ) [ Y ^ ( e i θ ) U ^ ( e i θ ) ] d θ ≥ 0. \frac{1}{2\pi}\int_{0}^{2\pi}
\begin{bmatrix}\widehat{Y}(e^{i\theta})\\\widehat{U}(e^{i\theta})\end{bmatrix}^{*}
\Pi_{\mu,L,d}(h_\rho(e^{i\theta}))
\begin{bmatrix}\widehat{Y}(e^{i\theta})\\\widehat{U}(e^{i\theta})\end{bmatrix}
\,d\theta\ge0. 2 π 1 ∫ 0 2 π [ Y ( e i θ ) U ( e i θ ) ] ∗ Π μ , L , d ( h ρ ( e i θ )) [ Y ( e i θ ) U ( e i θ ) ] d θ ≥ 0. In particular , taking J = τ J=\tau J = τ and h j = 1 / τ h_j=1/\tau h j = 1/ τ gives h ρ = h ρ , τ h_\rho=h_{\rho,\tau} h ρ = h ρ , τ , where
h ρ , τ ( z ) = 1 τ ∑ j = 1 τ ρ j z − j . h_{\rho,\tau}(z)=\frac{1}{\tau}\sum_{j=1}^{\tau}\rho^j z^{-j}. h ρ , τ ( z ) = τ 1 j = 1 ∑ τ ρ j z − j . Proof. Set r ^ k = ρ − k r k \widehat{r}_k=\rho^{-k}r_k r k = ρ − k r k and s ^ k = ρ − k s k \widehat{s}_k=\rho^{-k}s_k s k = ρ − k s k . By Parseval, it suffices to prove the time-domain inequality
2 ∑ k ⟨ r ^ k , s ^ k ⟩ − 2 ∑ j = 1 J ρ j h j ∑ k ⟨ r ^ k , s ^ k − j ⟩ ≥ 0. (158) 2\sum_k\langle\widehat{r}_k,\widehat{s}_k\rangle-2\sum_{j=1}^{J}\rho^j h_j\sum_k\langle\widehat{r}_k,\widehat{s}_{k-j}\rangle\ge0. \tag*{(158)} 2 k ∑ ⟨ r k , s k ⟩ − 2 j = 1 ∑ J ρ j h j k ∑ ⟨ r k , s k − j ⟩ ≥ 0. (158)
For a fixed lag j j j define λ k = ρ − 2 k \lambda_k=\rho^{-2k} λ k = ρ − 2 k for k ≥ 0 k\ge0 k ≥ 0 and λ k = 0 \lambda_k=0 λ k = 0 for k < 0 k<0 k < 0 . The zero extension makes all sums finite. As in the proof of Lemma 4.3 , the subgradient inequality gives
⟨ s k , r k − r k + j ⟩ ≥ ψ ( r k ) − ψ ( r k + j ) . \langle s_k,r_k-r_{k+j}\rangle\ge\psi(r_k)-\psi(r_{k+j}). ⟨ s k , r k − r k + j ⟩ ≥ ψ ( r k ) − ψ ( r k + j ) .
Multiplying by λ k \lambda_k λ k and summing over k ≥ 0 k\ge0 k ≥ 0 yields
∑ k ≥ 0 λ k ⟨ s k , r k − r k + j ⟩ ≥ ∑ ℓ ≥ 0 ( λ ℓ − λ ℓ − j ) ψ ( r ℓ ) ≥ 0 , \sum_{k\ge0}\lambda_k\langle s_k,r_k-r_{k+j}\rangle\ge\sum_{\ell\ge0}(\lambda_\ell-\lambda_{\ell-j})\psi(r_\ell)\ge0, k ≥ 0 ∑ λ k ⟨ s k , r k − r k + j ⟩ ≥ ℓ ≥ 0 ∑ ( λ ℓ − λ ℓ − j ) ψ ( r ℓ ) ≥ 0 ,
where λ ℓ − j = 0 \lambda_{\ell-j}=0 λ ℓ − j = 0 for ℓ < j \ell<j ℓ < j and, for ℓ ≥ j \ell\ge j ℓ ≥ j , λ ℓ − λ ℓ − j = ρ − 2 ℓ ( 1 − ρ 2 j ) ≥ 0 \lambda_\ell-\lambda_{\ell-j}=\rho^{-2\ell}(1-\rho^{2j})\ge0 λ ℓ − λ ℓ − j = ρ − 2 ℓ ( 1 − ρ 2 j ) ≥ 0 . Hence
∑ k ≥ 0 λ k ⟨ s k , r k ⟩ ≥ ∑ k ≥ 0 λ k ⟨ s k , r k + j ⟩ . (159) \sum_{k\ge0}\lambda_k\langle s_k,r_k\rangle\ge\sum_{k\ge0}\lambda_k\langle s_k,r_{k+j}\rangle. \tag*{(159)} k ≥ 0 ∑ λ k ⟨ s k , r k ⟩ ≥ k ≥ 0 ∑ λ k ⟨ s k , r k + j ⟩ . (159)
On the other hand, similar to ∑ k ≥ 0 ⟨ r k , s k − j ⟩ = ∑ k ≥ 0 ⟨ r k + j , s k ⟩ \sum_{k\ge0}\langle r_k,s_{k-j}\rangle=\sum_{k\ge0}\langle r_{k+j},s_k\rangle ∑ k ≥ 0 ⟨ r k , s k − j ⟩ = ∑ k ≥ 0 ⟨ r k + j , s k ⟩ , we also have
ρ j ∑ k ⟨ r ^ k , s ^ k − j ⟩ = ∑ k ≥ 0 λ k ⟨ r k + j , s k ⟩ . \rho^j\sum_k\langle\widehat r_k,\widehat s_{k-j}\rangle=\sum_{k\ge0}\lambda_k\langle r_{k+j},s_k\rangle. ρ j k ∑ ⟨ r k , s k − j ⟩ = k ≥ 0 ∑ λ k ⟨ r k + j , s k ⟩ .
In addition,
∑ k ⟨ r ^ k , s ^ k ⟩ = ∑ k ≥ 0 λ k ⟨ r k , s k ⟩ . \sum_k\langle\widehat r_k,\widehat s_k\rangle=\sum_{k\ge0}\lambda_k\langle r_k,s_k\rangle. k ∑ ⟨ r k , s k ⟩ = k ≥ 0 ∑ λ k ⟨ r k , s k ⟩ .
Then, by (159) , each lag j j j satisfies
∑ k ⟨ r ^ k , s ^ k ⟩ − ρ j ∑ k ⟨ r ^ k , s ^ k − j ⟩ ≥ 0. \sum_k\langle\widehat r_k,\widehat s_k\rangle-\rho^j\sum_k\langle\widehat r_k,\widehat s_{k-j}\rangle\ge0. k ∑ ⟨ r k , s k ⟩ − ρ j k ∑ ⟨ r k , s k − j ⟩ ≥ 0.
Multiplying this inequality by the nonnegative weight 2 h j 2h_j 2 h j and summing over j = 1 , … , J j=1,\ldots,J j = 1 , … , J , then using ∑ j = 1 J h j ≤ 1 \sum_{j=1}^{J}h_j\le1 ∑ j = 1 J h j ≤ 1 together with ∑ k ⟨ r ^ k , s ^ k ⟩ ≥ 0 \sum_k\langle\widehat r_k,\widehat s_k\rangle\ge0 ∑ k ⟨ r k , s k ⟩ ≥ 0 , yields
2 ∑ k ⟨ r ^ k , s ^ k ⟩ − 2 ∑ j = 1 J ρ j h j ∑ k ⟨ r ^ k , s ^ k − j ⟩ ≥ 0 , 2\sum_k\langle\widehat r_k,\widehat s_k\rangle-2\sum_{j=1}^{J}\rho^jh_j\sum_k\langle\widehat r_k,\widehat s_{k-j}\rangle\ge0, 2 k ∑ ⟨ r k , s k ⟩ − 2 j = 1 ∑ J ρ j h j k ∑ ⟨ r k , s k − j ⟩ ≥ 0 ,
which yields (157). The Fourier conversion is the same as in Lemma 4.3 , now applied to the weighted sequences. If Y ^ \widehat Y Y and U ^ \widehat U U are the Fourier transforms of y ^ \widehat y y and u ^ \widehat u u , then (157) is equivalent to
1 2 π ∫ 0 2 π [ Y ^ U ^ ] ∗ Π μ , L , d ( h ρ ( e i θ ) ) [ Y ^ U ^ ] d θ ≥ 0. \frac{1}{2\pi}\int_{0}^{2\pi}
\begin{bmatrix}\widehat Y\\\widehat U\end{bmatrix}^{*}
\Pi_{\mu,L,d}(h_\rho(e^{\mathrm{i}\theta}))
\begin{bmatrix}\widehat Y\\\widehat U\end{bmatrix}
\,\mathrm{d}\theta\ge0. 2 π 1 ∫ 0 2 π [ Y U ] ∗ Π μ , L , d ( h ρ ( e i θ )) [ Y U ] d θ ≥ 0.
This is the desired hard IQC for the rate-weighted sequences. Finally, h ρ h_\rho h ρ becomes h ρ , τ h_{\rho,\tau} h ρ , τ for J = τ J=\tau J = τ and h j = 1 / τ h_j=1/\tau h j = 1/ τ .
Proving Theorem 4.1: Accelerated Rates for DREAM
In this section, we provide a linear-rate analysis of DREAM and provide a proof of Theorem 4.1 where we will build on the IQC techniques and the transfer function of the linear plant developed in § 4.1 . Our analysis begins by decomposing the gradient as
∇ f ( x ) = m x + v ( x ) , i.e., v ( x ) : = ∇ f ( x ) − m x , \nabla f(x)=mx+v(x),\qquad\text{i.e.,}\quad v(x):=\nabla f(x)-mx, ∇ f ( x ) = m x + v ( x ) , i.e., v ( x ) := ∇ f ( x ) − m x ,
where m = 1 2 ( 1 + q − 2 ) m = \frac{1}{2}(1+q^{-2}) m = 2 1 ( 1 + q − 2 ) is the midpoint of the slope interval [ 1 , q − 2 ] [1,q^{-2}] [ 1 , q − 2 ] , and v v v is the residual deviation from this constant slope. Recall that without loss of generality we take μ = 1 \mu=1 μ = 1 and ∇ f ( 0 ) = 0 \nabla f(0)=0 ∇ f ( 0 ) = 0 in our analysis.
Absorbing the linear term m x mx m x into the GMM dynamics yields a linear recursion, which we call the midpoint plant . This recursion is obtained by setting v ≡ 0 v \equiv0 v ≡ 0 . To certify contraction at a target rate ρ \rho ρ , we pass to rate-weighted coordinates, so that rate-ρ \rho ρ convergence of the original iteration becomes stability of the weighted system. Our objectives are twofold: (i) establish that the rate-weighted midpoint plant is Schur stable; and (ii) control the residual nonlinearity v v v via a Zames–Falb IQC and show that, whenever the corresponding frequency-domain inequality holds, the full rate-weighted feedback system is stable. Then, stability of the rate-weighted system implies linear convergence of the original GMM iteration at rate ρ \rho ρ .
Definition 5.1 (Midpoint plant). Let m = 1 2 ( 1 + q − 2 ) m = \frac{1}{2}(1+q^{-2}) m = 2 1 ( 1 + q − 2 ) denote the midpoint of the slope interval [ μ , L ] = [ 1 , q − 2 ] [\mu,L]=[1,q^{-2}] [ μ , L ] = [ 1 , q − 2 ] . The midpoint plant is the linear time-invariant recursion obtained from delayed GMM (14) by replacing the gradient with the constant slope m m m , that is, by setting u k = m y k u_k = m y_k u k = m y k :
x k + 1 = ( 1 + β ) x k − β x k − 1 − α m y k − τ , y k = ( 1 + ν ) x k − ν x k − 1 . (160) x_{k+1}=(1+\beta)x_k-\beta x_{k-1}-\alpha m y_{k-\tau}, \qquad y_k=(1+\nu)x_k-\nu x_{k-1}. \tag*{(160)} x k + 1 = ( 1 + β ) x k − β x k − 1 − α m y k − τ , y k = ( 1 + ν ) x k − ν x k − 1 . (160)
This is exactly delayed GMM run on a quadratic whose curvature equals the midpoint m m m . As we discussed in § 3 , its stability is governed by the characteristic polynomial P τ , δ m P_{\tau,\delta_m} P τ , δ m with gain δ m : = α m \delta_m := \alpha m δ m := α m , i.e.,
P τ , δ m ( z ) = z τ ( z − 1 ) ( z − β ) + δ m [ ( 1 + ν ) z − ν ] . (161) P_{\tau,\delta_m}(z)=z^\tau(z-1)(z-\beta)+\delta_m\left[(1+\nu)z-\nu\right]. \tag*{(161)} P τ , δ m ( z ) = z τ ( z − 1 ) ( z − β ) + δ m [ ( 1 + ν ) z − ν ] . (161)
Before presenting the proof of Theorem 4.1 , we first discuss sufficient conditions that would imply a given target convergence rate ρ \rho ρ .
A Convergence Rate Lemma: Sufficient Conditions for Linear Convergence
Recall from Figure 1 that before closing the gradient feedback loop, the linear part of (14) maps the gradient sequence ( u k ) (u_k) ( u k ) to the query-point sequence ( y k ) (y_k) ( y k ) . Under the z z z -transform this map can be expressed as a single rational function, the transfer function g τ , q g_{\tau,q} g τ , q . Indeed, writing Y ( z ) , U ( z ) Y(z), U(z) Y ( z ) , U ( z ) for the transforms of ( y k ) , ( u k ) (y_k),(u_k) ( y k ) , ( u k ) , under zero initial conditions ( x k = 0 (x_k=0 ( x k = 0 for k ≤ 0 k \le0 k ≤ 0 and u k = 0 u_k=0 u k = 0 for k < 0 ) k<0) k < 0 ) , one has Y = g τ , q U Y=g_{\tau,q}U Y = g τ , q U .
To certify contraction at rate ρ \rho ρ we pass to the rate-weighted coordinates ( ⋅ ^ ) k = ρ − k ( ⋅ ) k (\hat{\cdot})_k=\rho^{-k}(\cdot)_k ( ⋅ ^ ) k = ρ − k ( ⋅ ) k , in which a sequence decaying like ρ k \rho^k ρ k becomes bounded. Multiplying a sequence by ρ − k \rho^{-k} ρ − k evaluates its transform at ρ z \rho z ρ z , so in these coordinates the linear part has transfer function g ρ ( z ) : = g τ , q ( ρ z ) g_\rho(z):=g_{\tau,q}(\rho z) g ρ ( z ) := g τ , q ( ρ z ) , i.e. under the same zero initial conditions, Y ^ = g ρ U ^ \hat{Y}=g_\rho\hat{U} Y ^ = g ρ U ^ . The transfer-function form of the midpoint plant itself—the closed loop of G G G with the constant gain m m m —appears in Lemma 5.5 below, where its poles are identified with the roots of P τ , δ m P_{\tau,\delta_m} P τ , δ m . The following lemma formalizes this reduction: if the rate-weighted midpoint plant is Schur and the associated strict frequency-domain inequality holds, then delayed GMM converges linearly at rate ρ \rho ρ . In the next subsection, we will provide a proof of this lemma.
Lemma 5.2 (Convergence Rate Lemma). Consider DREAM, namely delayed GMM (14) with the parameters of Theorem 4.1 . Let ρ = 1 − c q / M ∈ ( 0 , 1 ) \rho=1-cq/M\in(0,1) ρ = 1 − c q / M ∈ ( 0 , 1 ) with c ∈ ( 0 , 2 3 ) ∗ a s i n ∗ x r e f ‘ ( 137 ) ‘ ∗ b e t h e t a r g e t r a t e a n d l e t ∗ c\in(0,\frac{2}{3}) *as in* {xref}`(137)` *be the target rate and let* c ∈ ( 0 , 3 2 ) ∗ a s in ∗ x r e f ‘ ( 137 ) ‘ ∗ b e t h e t a r g e t r a t e an d l e t ∗ m=\frac{1}{2}(1+q^{-2}) be the midpoint of the slope interval [ μ , L ] = [ 1 , q − 2 ] [\mu,L]=[1,q^{-2}] [ μ , L ] = [ 1 , q − 2 ] . Assume 0 < q < M / ( 2 M + 1 ) 0<q<M/(2M+1) 0 < q < M / ( 2 M + 1 ) . Suppose the following two conditions hold.
(H1) Midpoint-pole hypothesis (stability of the linear surrogate). Every pole of the midpoint plant of Definition 5.1 —equivalently, every root of the midpoint characteristic polynomial P τ , δ m P_{\tau,\delta_m} P τ , δ m with
δ m = α m \delta_m=\alpha_m δ m = α m —lies in the open disk { ∣ z ∣ < ρ } \{|z|<\rho\} { ∣ z ∣ < ρ } . Equivalently, after the rate weighting ( ⋅ ) ^ k = ρ − k ( ⋅ ) k \widehat{(\cdot)}_k=\rho^{-k}(\cdot)_k ( ⋅ ) k = ρ − k ( ⋅ ) k , the midpoint plant is Schur.
(H2) Strict frequency-domain inequality (FDI). With z = ρ e i θ z=\rho e^{\mathrm{i}\theta} z = ρ e i θ , the Hermitian form
Φ τ , q ( θ ) : = [ g τ , q ( z ) 1 ] ∗ Π 1 , q − 2 ( h ρ , τ ( e i θ ) ) [ g τ , q ( z ) 1 ] < 0 for every θ ∈ [ 0 , 2 π ) . (162) \Phi_{\tau,q}(\theta):=
\begin{bmatrix}
g_{\tau,q}(z)\\
1
\end{bmatrix}^{*}
\Pi_{1,q^{-2}}\left(h_{\rho,\tau}(e^{\mathrm{i}\theta})\right)
\begin{bmatrix}
g_{\tau,q}(z)\\
1
\end{bmatrix}
<0\quad\text{for every }\theta\in[0,2\pi). \tag*{(162)} Φ τ , q ( θ ) := [ g τ , q ( z ) 1 ] ∗ Π 1 , q − 2 ( h ρ , τ ( e i θ ) ) [ g τ , q ( z ) 1 ] < 0 for every θ ∈ [ 0 , 2 π ) . (162)
Here g τ , q g_{\tau,q} g τ , q is the transfer function (145) of the linear part of (14) , and Π 1 , q − 2 ( h ρ , τ ) \Pi_{1,q^{-2}}(h_{\rho,\tau}) Π 1 , q − 2 ( h ρ , τ ) is the Zames–Falb multiplier (154) evaluated at the rate-weighted weight h ρ , τ h_{\rho,\tau} h ρ , τ of Lemma 4.4 .
Then the iterates of delayed GMM converge linearly with rate ρ \rho ρ , i.e., there exists a finite constant C > 0 C>0 C > 0 such that
∥ x k − x ⋆ ∥ ≤ C ρ k , k ≥ 0. \lVert x_k-x_{\star}\rVert\le C\rho^k,\qquad k\ge0. ∥ x k − x ⋆ ∥ ≤ C ρ k , k ≥ 0.
Moreover, the rate is uniform over the function class S 1 , q − 2 \mathcal{S}_{1,q^{-2}} S 1 , q − 2 . Specifically, for fixed algorithmic parameters, the same value of ρ \rho ρ applies to every objective f ∈ S 1 , q − 2 f\in\mathcal{S}_{1,q^{-2}} f ∈ S 1 , q − 2 with minimizer x ⋆ x_{\star} x ⋆ and every finite initial history of the recursion, with C C C depending only on the fixed algorithmic parameters and on the initial history relative to x ⋆ x_{\star} x ⋆ .
Proof of the Convergence Rate Lemma
This subsection proves the convergence rate lemma (Lemma 5.2 ). The proof is an energy argument, assembled from a small number of supporting lemmas, which we state here at the point of use; their proofs are routine computations and are collected in Appendix § 11 so as not to interrupt the argument. The proof will assume the hypotheses (H1) and (H2) hold; these hypotheses themselves are verified afterwards, in § 5.3.1 –§ 5.3.3 .
Treating u k u_k u k as an exogenous input to the linear dynamics in (14) , the corresponding input-output map from u k u_k u k to y k y_k y k has transfer function g τ , q g_{\tau,q} g τ , q ; that is, Y ( z ) = g τ , q ( z ) U ( z ) Y(z)=g_{\tau,q}(z)U(z) Y ( z ) = g τ , q ( z ) U ( z ) under zero initial conditions; see (145) . If we introduce the rate-weighted sequences y ^ k = ρ − k y k \widehat{y}_k=\rho^{-k}y_k y k = ρ − k y k and u ^ k = ρ − k u k \widehat{u}_k=\rho^{-k}u_k u k = ρ − k u k , the following lemma, using the basic properties of z z z -transforms, shows that the z z z -transforms Y ^ ( z ) \widehat{Y}(z) Y ( z ) and U ^ ( z ) \widehat{U}(z) U ( z ) of the weighted sequences are related through the rate-normalized transfer function g τ , q ( ρ z ) g_{\tau,q}(\rho z) g τ , q ( ρ z ) .
Lemma 5.3 (rate-weighted plant). Introduce the rate-weighted signals y ^ k = ρ − k y k \widehat{y}_k=\rho^{-k}y_k y k = ρ − k y k and u ^ k = ρ − k u k \widehat{u}_k=\rho^{-k}u_k u k = ρ − k u k , and let Y ^ , U ^ \widehat{Y},\widehat{U} Y , U denote their z z z -transforms. We have Y ^ ( z ) = g ρ ( z ) U ^ ( z ) \widehat{Y}(z)=g_{\rho}(z)\widehat{U}(z) Y ( z ) = g ρ ( z ) U ( z ) under zero initial conditions, where
g ρ ( z ) = g τ , q ( ρ z ) . (163) g_{\rho}(z)=g_{\tau,q}(\rho z). \tag*{(163)} g ρ ( z ) = g τ , q ( ρ z ) . (163)
The corresponding transfer matrix is G ρ ( z ) = g ρ ( z ) I d G_{\rho}(z)=g_{\rho}(z)I_d G ρ ( z ) = g ρ ( z ) I d ; thus Y ^ = G ρ U ^ \widehat{Y}=G_{\rho}\widehat{U} Y = G ρ U . The scalar transfer function g ρ ( z ) g_{\rho}(z) g ρ ( z ) equals g τ , q ( ρ z ) g_{\tau,q}(\rho z) g τ , q ( ρ z ) evaluated at ρ z \rho z ρ z , i.e., rate weighting divides every pole of g τ , q g_{\tau,q} g τ , q by ρ \rho ρ .
As in Figure 1 , we think of GMM as an interconnection of a linear system and a gradient nonlinearity; and we call the linear part the open-loop system, whereas the whole GMM dynamics is referred to as the close-loop system with the gradients added. Because rate weighting divides every pole of g τ , q g_{\tau,q} g τ , q by ρ \rho ρ , the poles of g ρ g_{\rho} g ρ may lie outside the unit circle; equivalently, the rate-weighted characteristic polynomial need not be Schur. This is an obstacle, as we need stability of the open-loop plant for our arguments based on IQC techniques to work. To overcome this issue, we split the gradient into its midpoint slope and a remainder, ∇ f ( v k ) = m v k + v k \nabla f(v_k)=m v_k+v_k ∇ f ( v k ) = m v k + v k , i.e., v k = v ( y k ) v_k=v(y_k) v k = v ( y k ) , and pass to the rate-weighted sequences y ^ k = ρ − k y k \widehat{y}_k=\rho^{-k}y_k y k = ρ − k y k and v ^ k = ρ − k v k \widehat{v}_k=\rho^{-k}v_k v k = ρ − k v k . We now describe the resulting weighted midpoint-transformed dynamics and show that they are Schur precisely when hypothesis (H1) holds. In plain terms: after the change of variables, the delayed recursion is an ordinary linear system with finitely many states, and we will show that its stability can be characterized exactly as in hypothesis (H1) .
Lemma 5.4 (weighted midpoint dynamics). Let ρ > 0 \rho> 0 ρ > 0 , and define the weighted variables
x ^ k = ρ − k x k , y ^ k = ρ − k y k , v ^ k = ρ − k v k \hat{x}_k = \rho^{-k}x_k,\qquad\hat{y}_k = \rho^{-k}y_k,\qquad\hat{v}_k = \rho^{-k}v_k x ^ k = ρ − k x k , y ^ k = ρ − k y k , v ^ k = ρ − k v k under zero initial conditions. Suppose the unweighted linear dynamics are
x k + 1 = ( 1 + β ) x k − β x k − 1 − α u k − τ , y k = ( 1 + ν ) x k − ν x k − 1 , x_{k+1} = (1+\beta)x_k - \beta x_{k-1} - \alpha u_{k-\tau},\qquad y_k = (1+\nu)x_k - \nu x_{k-1}, x k + 1 = ( 1 + β ) x k − β x k − 1 − α u k − τ , y k = ( 1 + ν ) x k − ν x k − 1 , where u k = m y k + v k u_k = my_k + v_k u k = m y k + v k . Then the weighted midpoint-transformed dynamics satisfy
x ^ k + 1 = 1 + β ρ x ^ k − β ρ 2 x ^ k − 1 − α m ( 1 + ν ) ρ τ + 1 x ^ k − τ + α m ν ρ τ + 2 x ^ k − τ − 1 − α ρ τ + 1 v ^ k − τ , (164) \hat{x}_{k+1} = \frac{1+\beta}{\rho}\hat{x}_k - \frac{\beta}{\rho^2}\hat{x}_{k-1} - \frac{\alpha m(1+\nu)}{\rho^{\tau+1}}\hat{x}_{k-\tau} + \frac{\alpha m\nu}{\rho^{\tau+2}}\hat{x}_{k-\tau-1} - \frac{\alpha}{\rho^{\tau+1}}\hat{v}_{k-\tau}, \tag*{(164)} x ^ k + 1 = ρ 1 + β x ^ k − ρ 2 β x ^ k − 1 − ρ τ + 1 α m ( 1 + ν ) x ^ k − τ + ρ τ + 2 α m ν x ^ k − τ − 1 − ρ τ + 1 α v ^ k − τ , (164) y ^ k = ( 1 + ν ) x ^ k − ν ρ x ^ k − 1 . (165) \hat{y}_k = (1+\nu)\hat{x}_k - \frac{\nu}{\rho}\hat{x}_{k-1}. \tag*{(165)} y ^ k = ( 1 + ν ) x ^ k − ρ ν x ^ k − 1 . (165)
The following lemma says that stability of the weighted midpoint-transformed dynamics provided in the previous lemma is equivalent to the hypothesis (H1) .
Lemma 5.5 (stability of the weighted-midpoint transformed dynamics). Recall from Lemma 5.3 that g ρ g_\rho g ρ is the transfer function from the rate-weighted gradient input to the rate-weighted query point, i.e., Y ^ = g ρ U ^ \hat{Y}=g_\rho\hat{U} Y ^ = g ρ U ^ . Write g ρ = N / D g_\rho=N/D g ρ = N / D as a ratio of polynomials, where N ( z ) = − α [ ( 1 + ν ) ρ z − ν ] N(z)=-\alpha[(1+\nu)\rho z-\nu] N ( z ) = − α [( 1 + ν ) ρ z − ν ] and D ( z ) = ( ρ z ) τ ( ρ z − 1 ) ( ρ z − β ) D(z)=(\rho z)^\tau(\rho z-1)(\rho z-\beta) D ( z ) = ( ρ z ) τ ( ρ z − 1 ) ( ρ z − β ) . We next consider the midpoint change of variables u ^ = m y ^ + v ^ \hat{u}=m\hat{y}+\hat{v} u ^ = m y ^ + v ^ , with residual input v ^ k = u ^ k − m y ^ k \hat{v}_k=\hat{u}_k-m\hat{y}_k v ^ k = u ^ k − m y ^ k . Eliminating u ^ \hat{u} u ^ gives the transformed plant g ~ ρ \tilde{g}_\rho g ~ ρ , i.e., the transfer function from the residual input v ^ \hat{v} v ^ to the query point y ^ \hat{y} y ^ is
Y ^ = g ~ ρ V ^ , g ~ ρ = g ρ 1 − m g ρ = N D − m N . (166) \hat{Y}=\tilde{g}_\rho\hat{V},\qquad\tilde{g}_\rho=\frac{g_\rho}{1-mg_\rho}=\frac{N}{D-mN}. \tag*{(166)} Y ^ = g ~ ρ V ^ , g ~ ρ = 1 − m g ρ g ρ = D − m N N . (166) The denominator is the rate-scaled midpoint characteristic polynomial: D − m N = P τ , δ m ( ρ ⋅ ) D-mN=P_{\tau,\delta_m}(\rho\cdot) D − m N = P τ , δ m ( ρ ⋅ ) with δ m = α m \delta_m=\alpha m δ m = α m , so its roots are the points r / ρ r/\rho r / ρ with P τ , δ m ( r ) = 0 P_{\tau,\delta_m}(r)=0 P τ , δ m ( r ) = 0 . Assume 0 < q < M / ( 2 M + 1 ) 0<q<M/(2M+1) 0 < q < M / ( 2 M + 1 ) (with M = 2 τ + 1 M=2\tau+1 M = 2 τ + 1 , β = 1 − 2 q / M \beta=1-2q/M β = 1 − 2 q / M ). Then the numerator N N N and denominator D − m N D-mN D − m N are coprime, so g ~ ρ \tilde{g}_\rho g ~ ρ has no pole-zero cancellation and its poles are exactly the roots of D − m N D-mN D − m N , i.e., the points r / ρ r/\rho r / ρ with P τ , δ m ( r ) = 0 P_{\tau,\delta_m}(r)=0 P τ , δ m ( r ) = 0 . Moreover, letting A A A be the state matrix of the finite-dimensional realization (164) –(165) with state
ξ k = [ x ^ k x ^ k − 1 ⋯ x ^ k − τ − 1 v ^ k − 1 ⋯ v ^ k − τ ] T ∈ R ( 2 τ + 2 ) d , \xi_k=\left[\hat{x}_k\ \ \hat{x}_{k-1}\ \ \cdots\ \ \hat{x}_{k-\tau-1}\ \ \hat{v}_{k-1}\ \ \cdots\ \ \hat{v}_{k-\tau}\right]^{\mathsf{T}}\in\mathbb{R}^{(2\tau+2)d}, ξ k = [ x ^ k x ^ k − 1 ⋯ x ^ k − τ − 1 v ^ k − 1 ⋯ v ^ k − τ ] T ∈ R ( 2 τ + 2 ) d , the matrix A A A has the spectrum σ ( A ) = P ( g ~ ρ ) ∪ { 0 } \sigma(A)=\mathcal{P}(\tilde{g}_\rho)\cup\{0\} σ ( A ) = P ( g ~ ρ ) ∪ { 0 } , where P ( g ~ ρ ) \mathcal{P}(\tilde{g}_\rho) P ( g ~ ρ ) denotes the poles of g ~ ρ \tilde{g}_\rho g ~ ρ . Consequently g ~ ρ \tilde{g}_\rho g ~ ρ is Schur—and, equivalently, A A A is Schur—if and only if every root of P τ , δ m P_{\tau,\delta_m} P τ , δ m lies in { ∣ z ∣ < ρ } \{|z|<\rho\} { ∣ z ∣ < ρ } , that is, if and only if the midpoint-pole hypothesis (H1) holds.
In the next lemma, we transport the strict FDI (H2) from the original plant to the Schur midpoint plant; this transport is a congruence of the underlying quadratic form, an operation we first recall. A congruence is the way a Hermitian quadratic form changes under a linear change of variables: if a form is written x ∗ Π x x^*\Pi x x ∗ Π x for a Hermitian matrix Π \Pi Π and one substitutes x = T y x=Ty x = T y with T T T invertible, then
x ∗ Π x = ( T y ) ∗ Π ( T y ) = y ∗ ( T ∗ Π T ) y , x^*\Pi x=(Ty)^*\Pi(Ty)=y^*(T^*\Pi T)y, x ∗ Π x = ( T y ) ∗ Π ( T y ) = y ∗ ( T ∗ Π T ) y ,
so in the new variable the form has the congruent matrix T ∗ Π T T^*\Pi T T ∗ Π T , and the value of the form—hence its sign—is unchanged. Here the change of variables is the midpoint substitution u ~ = m y ~ + v ~ \tilde{u}=m\tilde{y}+\tilde{v} u ~ = m y ~ + v ~ , which relates the old IQC variables ( y ~ , u ~ ) (\tilde{y},\tilde{u}) ( y ~ , u ~ ) and the new variables ( y ~ , v ~ ) (\tilde{y},\tilde{v}) ( y ~ , v ~ ) by
[ y ^ u ^ ] = T m , d [ y ^ v ^ ] , T m , d : = T m ⊗ I d where T m = [ 1 0 m 1 ] . \begin{bmatrix}\hat{y}\\ \hat{u}\end{bmatrix}=T_{m,d}\begin{bmatrix}\hat{y}\\ \hat{v}\end{bmatrix},\qquad T_{m,d}:=T_m\otimes I_d\quad\text{where}\quad T_m=\begin{bmatrix}1&0\\m&1\end{bmatrix}. [ y ^ u ^ ] = T m , d [ y ^ v ^ ] , T m , d := T m ⊗ I d where T m = [ 1 m 0 1 ] .
and the same relation holds for the Fourier transforms. Let Π ( z ) : = Π 1 , q − 2 ( h ρ , τ ( z ) ) \Pi(z):=\Pi_{1,q-2}(h_{\rho,\tau}(z)) Π ( z ) := Π 1 , q − 2 ( h ρ , τ ( z )) denote the 2 × 2 2\times2 2 × 2 multiplier from (154) and define
Π ~ ( z ) : = T m ∗ Π ( z ) T m , Π d ( z ) : = Π ( z ) ⊗ I d , \widetilde{\Pi}(z):=T_m^*\Pi(z)T_m,\qquad\Pi_d(z):=\Pi(z)\otimes I_d, Π ( z ) := T m ∗ Π ( z ) T m , Π d ( z ) := Π ( z ) ⊗ I d ,
Π ~ d ( z ) : = Π ~ ( z ) ⊗ I d = T m , d ∗ Π d ( z ) T m , d . (167) \widetilde{\Pi}_d(z):=\widetilde{\Pi}(z)\otimes I_d=T_{m,d}^*\Pi_d(z)T_{m,d}. \tag*{(167)} Π d ( z ) := Π ( z ) ⊗ I d = T m , d ∗ Π d ( z ) T m , d . (167)
On the unit circle, the vector-signal quadratic forms obey the exact identity
[ Y ^ U ^ ] ∗ Π d [ Y ^ U ^ ] = [ Y ^ V ^ ] ∗ Π ~ d [ Y ^ V ^ ] . \begin{bmatrix}\widehat{Y}\\\widehat{U}\end{bmatrix}^*\Pi_d\begin{bmatrix}\widehat{Y}\\\widehat{U}\end{bmatrix}=\begin{bmatrix}\widehat{Y}\\\widehat{V}\end{bmatrix}^*\widetilde{\Pi}_d\begin{bmatrix}\widehat{Y}\\\widehat{V}\end{bmatrix}. [ Y U ] ∗ Π d [ Y U ] = [ Y V ] ∗ Π d [ Y V ] .
We have the scalar transfer functions g = g ρ ( e i θ ) g=g_\rho(e^{i\theta}) g = g ρ ( e i θ ) and g ~ = g ~ ρ ( e i θ ) = g / ( 1 − m g ) \widetilde{g}=\widetilde{g}_\rho(e^{i\theta})=g/(1-mg) g = g ρ ( e i θ ) = g / ( 1 − m g ) . Whenever g g g is finite and 1 − m g ≠ 0 1-mg\ne0 1 − m g = 0 ,
T m [ g ~ 1 ] = 1 1 − m g [ g 1 ] . T_m\begin{bmatrix}\widetilde{g}\\1\end{bmatrix}=\frac{1}{1-mg}\begin{bmatrix}g\\1\end{bmatrix}. T m [ g 1 ] = 1 − m g 1 [ g 1 ] .
Consequently, with both multipliers evaluated at e i θ e^{i\theta} e i θ ,
\begin{bmatrix}\widetilde{g}\\1\end{bmatrix}^*\widetilde{\Pi}\begin{bmatrix}\widetilde{g}\\1\end{bmatrix}=\frac{1}{\lvert1-mg\rvert^2}\begin{bmatrix}g\\1\end{bmatrix}^*\Pi\begin{bmatrix}g\\1\end{bmatrix}. \tag*{(168)}\end{bmatrix}
Here, 1 − m g 1-mg 1 − m g is a scalar. For the actual d d d -dimensional plant, the Kronecker product gives
\begin{bmatrix}\widetilde{G}_\rho\\I_d\end{bmatrix}^*\widetilde{\Pi}_d\begin{bmatrix}\widetilde{G}_\rho\\I_d\end{bmatrix}=\left(\begin{bmatrix}\widetilde{g}_\rho\\1\end{bmatrix}^*\widetilde{\Pi}\begin{bmatrix}\widetilde{g}_\rho\\1\end{bmatrix}\right)I_d, \tag*{(169)}\end{bmatrix}
where G ~ ρ : = g ~ ρ I d \widetilde{G}_\rho:=\widetilde{g}_\rho I_d G ρ := g ρ I d . Thus a scalar margin − η -\eta − η becomes the matrix margin − η I d -\eta I_d − η I d , without a dimension-dependent factor in the following sense: If the right hand side of (168) is less than a scalar − η -\eta − η ; this leads to an upper bound − η I d -\eta I_d − η I d for the right-hand side (in the sense of matrix inequalities in the Loewner ordering) in (169) .
Lemma 5.6 (FDI equivalence). Consider DREAM, namely delayed GMM (14) with the parameters of Theorem 4.1 where 0 < ρ < 1 0<\rho<1 0 < ρ < 1 and ρ ≠ β \rho\ne\beta ρ = β . Under the midpoint-pole hypothesis (H1 ), g ρ g_\rho g ρ is finite on the unit circle and 1 − m g ρ ( e i θ ) ≠ 0 1-mg_\rho(e^{i\theta})\ne0 1 − m g ρ ( e i θ ) = 0 for every θ ∈ [ 0 , 2 π ) \theta\in[0,2\pi) θ ∈ [ 0 , 2 π ) . In addition, the scalar transformed plant g ~ ρ = g ρ / ( 1 − m g ρ ) \widetilde{g}_\rho=g_\rho/(1-mg_\rho) g ρ = g ρ / ( 1 − m g ρ ) is well defined on the unit circle and the scalar factor ∣ 1 − m g ∣ 2 \lvert1-mg\rvert^2 ∣ 1 − m g ∣ 2 defined in (168) with g = g ρ g=g_\rho g = g ρ is finite and strictly positive. Consequently the strict FDI in (162) is equivalent to a strict FDI for g ~ ρ \widetilde{g}_\rho g ρ with multiplier Π ~ \widetilde{\Pi} Π , and hence, by (169) to the lifted matrix FDI for G ~ ρ \widetilde{G}_\rho G ρ with multiplier Π ~ d \widetilde{\Pi}_d Π d .
We next show that a strict FDI yields a positive-definite quadratic function V V V of a suitably augmented state that satisfies the strict dissipation inequality (173) below and thereby controls the iterates. This is a standard step in the IQC analysis of optimization algorithms [34 ] in its dissipativity form [50 , 26 , 25 ] ; the lemma below specializes it to our multipliers and systems and identifies the one-step quadratic quantity σ k \sigma_k σ k explicitly. In the language of dissipativity theory [56 ] , V V V is a storage function , the energy stored in the system, and − σ k -\sigma_k − σ k the supply rate , a prescribed quadratic function of the input v ^ k \hat{v}_k v ^ k and output y ^ k \hat{y}_k y ^ k (and of their recent past, through the multiplier memory) representing the energy delivered to the system at step k k k ; the inequality says that the stored energy can grow in one step by at most the energy delivered, minus a strict margin. Thus, unlike a Lyapunov function, V V V need not decrease at every step; what keeps it bounded is the hard IQC of Lemma 4.4 , by which the gradient nonlinearity delivers no net energy over any finite horizon, ∑ k = 0 N σ k ≥ 0 \sum_{k=0}^{N}\sigma_k \ge0 ∑ k = 0 N σ k ≥ 0 , so that summing (173) bounds V V V and hence the iterates.
Lemma 5.7 (Existence of quadratic storage functions). Let G ( z ) ∈ C d × d \mathcal{G}(z) \in\mathbb{C}^{d \times d} G ( z ) ∈ C d × d be a proper rational transfer matrix from the rate-weighted residual input v ^ k \hat{v}_k v ^ k to the rate-weighted query point y ^ k \hat{y}_k y ^ k , both with d d d coordinates, with realization
ξ k + 1 = A ξ k + B v ^ k , y ^ k = C ξ k + D v ^ k , (170) \begin{aligned}
\xi_{k+1} &= A\xi_k + B\hat{v}_k, & \hat{y}_k &= C\xi_k + D\hat{v}_k, \tag*{(170)}
\end{aligned} ξ k + 1 = A ξ k + B v ^ k , y ^ k = C ξ k + D v ^ k , (170) where ξ k ∈ C n \xi_k \in\mathbb{C}^n ξ k ∈ C n , A ∈ C n × n A \in\mathbb{C}^{n \times n} A ∈ C n × n is Schur, B ∈ C n × d B \in\mathbb{C}^{n \times d} B ∈ C n × d , C ∈ C d × n C \in\mathbb{C}^{d \times n} C ∈ C d × n , and D ∈ C d × d D \in\mathbb{C}^{d \times d} D ∈ C d × d . Thus G ( z ) = C ( z I n − A ) − 1 B + D \mathcal{G}(z) = C(zI_n-A)^{-1}B+D G ( z ) = C ( z I n − A ) − 1 B + D . Let
h ( z ) = ∑ j = 1 J c j z − j , for some given c j ≥ 0 , such that s h : = ∑ j = 1 J c j < 1 , h(z) = \sum_{j=1}^{J} c_j z^{-j}, \qquad\text{for some given } c_j \ge0, \qquad\text{such that } s_h := \sum_{j=1}^{J} c_j < 1, h ( z ) = j = 1 ∑ J c j z − j , for some given c j ≥ 0 , such that s h := j = 1 ∑ J c j < 1 , be a finite-memory linear filter defined through its transfer function, so that when it acts on the sequence s ^ k \hat{s}_k s ^ k , it produces the output sequence ( h s ^ ) k : = ∑ j = 1 J c j s ^ k − j (h\hat{s})_k := \sum_{j=1}^{J} c_j \hat{s}_{k-j} ( h s ^ ) k := ∑ j = 1 J c j s ^ k − j . Fix a > 0 a>0 a > 0 and define
r ^ k : = a y ^ k + v ^ k , s ^ k : = a y ^ k − v ^ k , \hat{r}_k := a\hat{y}_k+\hat{v}_k, \qquad\hat{s}_k := a\hat{y}_k-\hat{v}_k, r ^ k := a y ^ k + v ^ k , s ^ k := a y ^ k − v ^ k , together with
σ k : = 2 Re ⟨ r ^ k , s ^ k ⟩ − 2 ∑ j = 1 J c j Re ⟨ r ^ k , s ^ k − j ⟩ . (171) \sigma_k := 2\operatorname{Re}\langle\hat{r}_k,\hat{s}_k\rangle- 2\sum_{j=1}^{J} c_j \operatorname{Re}\langle\hat{r}_k,\hat{s}_{k-j}\rangle. \tag*{(171)} σ k := 2 Re ⟨ r ^ k , s ^ k ⟩ − 2 j = 1 ∑ J c j Re ⟨ r ^ k , s ^ k − j ⟩ . (171) Define
Ψ ( z ) : = [ a ( 2 − h ( z ) ) h ( z ) a h ( z ) ( 2 − h ( z ) ) ] , J 0 : = [ 1 0 0 − 1 ] , \Psi(z) := \begin{bmatrix} a(2-h(z)) & h(z) \\ ah(z) & (2-h(z)) \end{bmatrix}, \qquad J_0 := \begin{bmatrix} 1 & 0 \\ 0 & -1 \end{bmatrix}, Ψ ( z ) := [ a ( 2 − h ( z )) ah ( z ) h ( z ) ( 2 − h ( z )) ] , J 0 := [ 1 0 0 − 1 ] , and
Ψ d ( z ) : = Ψ ( z ) ⊗ I d , J 0 , d : = J 0 ⊗ I d , \Psi_d(z) := \Psi(z) \otimes I_d, \qquad J_{0,d} := J_0 \otimes I_d, Ψ d ( z ) := Ψ ( z ) ⊗ I d , J 0 , d := J 0 ⊗ I d , Π ~ ( z ) : = 1 2 Ψ ~ ( z ) J 0 Ψ ( z ) , Π ~ d ( z ) : = Π ~ ( z ) ⊗ I d = 1 2 Ψ ~ d ( z ) J 0 , d Ψ d ( z ) . \widetilde{\Pi}(z) := \frac{1}{2}\widetilde{\Psi}(z)J_0\Psi(z), \qquad\widetilde{\Pi}_d(z) := \widetilde{\Pi}(z)\otimes I_d = \frac{1}{2}\widetilde{\Psi}_d(z)J_{0,d}\Psi_d(z). Π ( z ) := 2 1 Ψ ( z ) J 0 Ψ ( z ) , Π d ( z ) := Π ( z ) ⊗ I d = 2 1 Ψ d ( z ) J 0 , d Ψ d ( z ) . Assume that, for some η > 0 \eta>0 η > 0 ,
\begin{bmatrix} \mathcal{G}(e^{i\theta}) \\ I_d \end{bmatrix}^{*}
\widetilde{\Pi}_d(e^{i\theta})
\begin{bmatrix} \mathcal{G}(e^{i\theta}) \\ I_d \end{bmatrix}
\preceq-\eta I_d, \qquad0 \le\theta< 2\pi. \tag*{(172)}\end{bmatrix}
Augment the plant state with the finite memory of the filter:
Ξ k : = [ ξ k q k ] , q k : = [ s ^ k − 1 s ^ k − 2 ⋮ s ^ k − J ] . \Xi_k := \begin{bmatrix} \xi_k \\ q_k \end{bmatrix}, \qquad q_k := \begin{bmatrix} \hat{s}_{k-1} \\ \hat{s}_{k-2} \\ \vdots\\ \hat{s}_{k-J} \end{bmatrix}. Ξ k := [ ξ k q k ] , q k := s ^ k − 1 s ^ k − 2 ⋮ s ^ k − J . Here q k ∈ C J d q_k \in\mathbb{C}^{Jd} q k ∈ C J d and Ξ k ∈ C n a \Xi_k \in\mathbb{C}^{n_a} Ξ k ∈ C n a , where n a : = n + J d n_a := n+Jd n a := n + J d . Thus ξ k \xi_k ξ k is the state of G \mathcal{G} G , while q k q_k q k stores the past values s ^ k − 1 , s ^ k − 2 , … , s ^ k − J \hat{s}_{k-1},\hat{s}_{k-2},\ldots,\hat{s}_{k-J} s ^ k − 1 , s ^ k − 2 , … , s ^ k − J needed to evaluate ( h s ^ ) k (h\hat{s})_k ( h s ^ ) k at time k k k . Then, there exist a Hermitian matrix X ∈ C n a × n a X \in\mathbb{C}^{n_a \times n_a} X ∈ C n a × n a , X ≻ 0 X \succ0 X ≻ 0 and ϵ > 0 \epsilon>0 ϵ > 0 such that, with V ( Ξ ) : = Ξ ∗ X Ξ V(\Xi):=\Xi^{*}X\Xi V ( Ξ ) := Ξ ∗ X Ξ , every trajectory of the augmented system satisfies
V ( Ξ k + 1 ) − V ( Ξ k ) + σ k ≤ − ϵ ( ∥ Ξ k ∥ 2 + ∥ v ^ k ∥ 2 ) . (173) V(\Xi_{k+1})-V(\Xi_k)+\sigma_k \le-\epsilon\left(\|\Xi_k\|^2+\|\hat{v}_k\|^2\right). \tag*{(173)} V ( Ξ k + 1 ) − V ( Ξ k ) + σ k ≤ − ϵ ( ∥ Ξ k ∥ 2 + ∥ v ^ k ∥ 2 ) . (173)
We are now ready to prove Lemma 5.2 by assembling these technical results.
Proof of Lemma 5.2 (Convergence Rate Lemma)
Proof. As in § 4.1.2 , translate the minimizer to the origin, so that x ⋆ = 0 x_\star= 0 x ⋆ = 0 and ∇ f ( 0 ) = 0 \nabla f(0) = 0 ∇ f ( 0 ) = 0 . Without loss of generality, we take μ = 1 \mu= 1 μ = 1 , write L = q − 2 L = q^{-2} L = q − 2 and set
m = μ + L 2 = 1 + q − 2 2 , a = L − μ 2 = q − 2 − 1 2 . m = \frac{\mu+ L}{2} = \frac{1 + q^{-2}}{2}, \qquad a = \frac{L - \mu}{2} = \frac{q^{-2} - 1}{2}. m = 2 μ + L = 2 1 + q − 2 , a = 2 L − μ = 2 q − 2 − 1 .
We show that the rate-weighted state is uniformly bounded. This is sufficient because x k = ρ k x ^ k x_k = \rho^k\widehat{x}_k x k = ρ k x k . The proof consists of three main steps: 1) rate-weighting and analysis of the resulting weighted midpoint dynamics; 2) construction of a quadratic storage function; and 3) application of hard IQCs.
Step 1: Rate weighting and midpoint transformation. Define
x ^ k = ρ − k x k , y ^ k = ρ − k y k , u ^ k = ρ − k u k , v ^ k : = u ^ k − m y ^ k . \widehat{x}_k = \rho^{-k}x_k,\qquad\widehat{y}_k = \rho^{-k}y_k,\qquad\widehat{u}_k = \rho^{-k}u_k,\qquad\widehat{v}_k := \widehat{u}_k - m\widehat{y}_k. x k = ρ − k x k , y k = ρ − k y k , u k = ρ − k u k , v k := u k − m y k .
By Lemma 5.3, the transfer function from u ^ \widehat{u} u to y ^ \widehat{y} y is g ρ ( z ) = g τ , q ( ρ z ) g_\rho(z) = g_{\tau,q}(\rho z) g ρ ( z ) = g τ , q ( ρ z ) , i.e., Y ^ = g ρ U ^ \widehat{Y} = g_\rho\widehat{U} Y = g ρ U . Moreover, according to Lemma 5.5 , using the midpoint change of variables u ^ = m y ^ + v ^ \widehat{u} = m\widehat{y} + \widehat{v} u = m y + v , (166) gives
Y ^ = G ~ ρ ( z ) V ^ , G ~ ρ ( z ) = g ~ ρ ( z ) I d with g ~ ρ ( z ) = g ρ ( z ) 1 − m g ρ ( z ) . \widehat{Y} = \widetilde{G}_\rho(z)\widehat{V}, \qquad\widetilde{G}_\rho(z) = \widetilde{g}_\rho(z)I_d \quad\text{with} \quad\widetilde{g}_\rho(z) = \frac{g_\rho(z)}{1 - m g_\rho(z)}. Y = G ρ ( z ) V , G ρ ( z ) = g ρ ( z ) I d with g ρ ( z ) = 1 − m g ρ ( z ) g ρ ( z ) .
Hypothesis (H1) and Lemma 5.5 imply that the dynamics (164) –(165) admit the transfer matrix G ~ ρ \widetilde{G}_\rho G ρ with the state vector ξ k \xi_k ξ k described in Lemma 5.5 , and by Lemma 5.5 , the state matrix A A A is Schur-stable. Moreover, by Lemma 5.6 , hypothesis (H2) gives a strict scalar FDI for g ~ ρ \widetilde{g}_\rho g ρ with the 2 × 2 2 \times2 2 × 2 multiplier
Π ~ = T m ∗ Π 1 , q − 2 ( h ρ , τ ) T m , T m = [ 1 0 m 1 ] . \widetilde{\Pi} = T_m^* \Pi_{1,q^{-2}}(h_{\rho,\tau})T_m,\qquad T_m = \begin{bmatrix} 1 & 0 \\ m & 1 \end{bmatrix}. Π = T m ∗ Π 1 , q − 2 ( h ρ , τ ) T m , T m = [ 1 m 0 1 ] .
The scalar form is continuous and strictly negative on the unit circle; compactness gives a uniform margin η > 0 \eta> 0 η > 0 . By (169) ,
[ G ~ ρ ( e i θ ) I d ] ∗ Π ~ d ( e i θ ) [ G ~ ρ ( e i θ ) I d ] ⪯ − η I d . \begin{bmatrix} \widetilde{G}_\rho(e^{i\theta}) \\ I_d \end{bmatrix}^* \widetilde{\Pi}_d(e^{i\theta}) \begin{bmatrix} \widetilde{G}_\rho(e^{i\theta}) \\ I_d \end{bmatrix} \preceq-\eta I_d. [ G ρ ( e i θ ) I d ] ∗ Π d ( e i θ ) [ G ρ ( e i θ ) I d ] ⪯ − η I d .
This is the matrix FDI required by Lemma 5.7 with G = G ~ ρ \mathcal{G} = \widetilde{G}_\rho G = G ρ .
Step 2: Dissipation inequality. For
h ρ , τ ( z ) = 1 τ ∑ j = 1 τ ρ j z − j , c j : = ρ j τ , h_{\rho,\tau}(z) = \frac{1}{\tau}\sum_{j=1}^{\tau}\rho^j z^{-j}, \qquad c_j := \frac{\rho^j}{\tau}, h ρ , τ ( z ) = τ 1 j = 1 ∑ τ ρ j z − j , c j := τ ρ j ,
we have c j ≥ 0 c_j \ge0 c j ≥ 0 and ∑ j = 1 τ c j < 1 \sum_{j=1}^{\tau} c_j < 1 ∑ j = 1 τ c j < 1 . Introduce the slope-shifted weighted signals
r ^ k : = u ^ k − y ^ k , s ^ k : = L y ^ k − u ^ k . \widehat{r}_k := \widehat{u}_k - \widehat{y}_k,\qquad\widehat{s}_k := L\widehat{y}_k - \widehat{u}_k. r k := u k − y k , s k := L y k − u k .
Because u ^ = m y ^ + v ^ \widehat{u} = m\widehat{y} + \widehat{v} u = m y + v and m − 1 = q − 2 − m = a m - 1 = q^{-2} - m = a m − 1 = q − 2 − m = a (since m m m is the midpoint), these satisfy
r ^ k = a y ^ k + v ^ k , s ^ k = a y ^ k − v ^ k . \widehat{r}_k = a\widehat{y}_k + \widehat{v}_k,\qquad\widehat{s}_k = a\widehat{y}_k - \widehat{v}_k. r k = a y k + v k , s k = a y k − v k .
The state of the Schur plant G ~ ρ \widetilde{G}_\rho G ρ is the delay state from Lemma 5.5 ,
ξ k = [ x ^ k x ^ k − 1 ⋯ x ^ k − τ − 1 v ^ k − 1 ⋯ v ^ k − τ ] T ∈ R ( 2 τ + 2 ) d . \xi_k = \left[\widehat{x}_k\ \ \widehat{x}_{k-1}\ \ \cdots\ \ \widehat{x}_{k-\tau-1}\ \ \widehat{v}_{k-1}\ \ \cdots\ \ \widehat{v}_{k-\tau}\right]^{\mathsf{T}} \in\mathbb{R}^{(2\tau+2)d}. ξ k = [ x k x k − 1 ⋯ x k − τ − 1 v k − 1 ⋯ v k − τ ] T ∈ R ( 2 τ + 2 ) d .
To realize the filter h ρ , τ h_{\rho,\tau} h ρ , τ in state space, adjoin the multiplier memory
q k : = [ s ^ k − 1 s ^ k − 2 ⋯ s ^ k − τ ] T ∈ R τ d , q_k := \begin{bmatrix} \widehat{s}_{k-1} & \widehat{s}_{k-2} & \cdots& \widehat{s}_{k-\tau} \end{bmatrix}^{\mathsf{T}} \in\mathbb{R}^{\tau d}, q k := [ s k − 1 s k − 2 ⋯ s k − τ ] T ∈ R τ d ,
and define the full plant-and-multiplier state
Ξ k : = [ ξ k q k ] = [ x ^ k ⋯ x ^ k − τ − 1 v ^ k − 1 ⋯ v ^ k − τ s ^ k − 1 ⋯ s ^ k − τ ] T ∈ R ( 3 τ + 2 ) d . (174) \Xi_k := \begin{bmatrix} \xi_k \\ q_k \end{bmatrix} = \begin{bmatrix} \widehat{x}_k & \cdots& \widehat{x}_{k-\tau-1} & \widehat{v}_{k-1} & \cdots& \widehat{v}_{k-\tau} & \widehat{s}_{k-1} & \cdots& \widehat{s}_{k-\tau} \end{bmatrix}^{\mathsf{T}} \in\mathbb{R}^{(3\tau+2)d}. \tag*{(174)} Ξ k := [ ξ k q k ] = [ x k ⋯ x k − τ − 1 v k − 1 ⋯ v k − τ s k − 1 ⋯ s k − τ ] T ∈ R ( 3 τ + 2 ) d . (174)
Thus ξ k \xi_k ξ k is the state of the midpoint-transformed plant itself, while Ξ k \Xi_k Ξ k is obtained from ξ k \xi_k ξ k by adjoining exactly the τ \tau τ vector-valued memory slots required by the FIR multiplier. In particular, x ^ k \widehat{x}_k x k is a d d d -dimensional block of both ξ k \xi_k ξ k and Ξ k \Xi_k Ξ k . The two histories should not be conflated: ξ 0 \xi_0 ξ 0 contains the prescribed algorithmic prehistory, including the actual delayed residual inputs v ^ − 1 , … , v ^ − τ \widehat{v}_{-1},\ldots,\widehat{v}_{-\tau} v − 1 , … , v − τ , whereas q k q_k q k is an auxiliary filter state. We initialize this multiplier memory at zero, q 0 = 0 q_0=0 q 0 = 0 . Accordingly, whenever a negative-indexed s ^ k − j \widehat{s}_{k-j} s k − j appears through the multiplier realization below, it is understood that it is set to zero; this does not alter the algorithmic prehistory stored in ξ 0 \xi_0 ξ 0 .
For this augmented realization, define
σ k : = 2 ⟨ r ^ k , s ^ k ⟩ − 2 τ ∑ j = 1 τ ρ j ⟨ r ^ k , s ^ k − j ⟩ , (175) \sigma_k := 2\langle\widehat{r}_k,\widehat{s}_k\rangle- \frac{2}{\tau}\sum_{j=1}^{\tau}\rho^j\langle\widehat{r}_k,\widehat{s}_{k-j}\rangle, \tag*{(175)} σ k := 2 ⟨ r k , s k ⟩ − τ 2 j = 1 ∑ τ ρ j ⟨ r k , s k − j ⟩ , (175)
where s ^ k − j = 0 \widehat{s}_{k-j}=0 s k − j = 0 whenever k − j < 0 k-j<0 k − j < 0 , consistently with q 0 = 0 q_0=0 q 0 = 0 . With J = τ J=\tau J = τ and c j = ρ j / τ c_j=\rho^j/\tau c j = ρ j / τ , the preceding identities put the transformed FDI exactly in the notation of Lemma 5.7 . Indeed, using m = ( q − 2 + 1 ) / 2 m=(q^{-2}+1)/2 m = ( q − 2 + 1 ) /2 , a = ( q − 2 − 1 ) / 2 a=(q^{-2}-1)/2 a = ( q − 2 − 1 ) /2 , it is straightforward to verify that
Π ~ d = T m , d T Π 1 , q − 2 , d ( h ) T m , d = 1 2 Ψ ~ d ∗ ⋆ J 0 , d Ψ d , \widetilde{\Pi}_d=T_{m,d}^{\mathsf{T}}\Pi_{1,q^{-2},d}(h)T_{m,d}=\frac{1}{2}\widetilde{\Psi}_d^{*}\star J_{0,d}\Psi_d, Π d = T m , d T Π 1 , q − 2 , d ( h ) T m , d = 2 1 Ψ d ∗ ⋆ J 0 , d Ψ d ,
where Ψ d \Psi_d Ψ d and J 0 , d J_{0,d} J 0 , d is as in Lemma 5.7 . Therefore, Lemma 5.7 is directly applicable and there exist a Hermitian matrix X ≻ 0 X\succ0 X ≻ 0 and a constant ϵ > 0 \epsilon>0 ϵ > 0 , depending only on the fixed algorithmic and certificate parameters, such that, with V ( Ξ ) : = Ξ ∗ X Ξ V(\Xi):=\Xi^{*}X\Xi V ( Ξ ) := Ξ ∗ X Ξ ,
V ( Ξ k + 1 ) − V ( Ξ k ) + σ k ≤ − ϵ ( ∥ Ξ k ∥ 2 + ∥ v ^ k ∥ 2 ) ( k ≥ 0 ) . (176) V(\Xi_{k+1})-V(\Xi_k)+\sigma_k\leq-\epsilon\left(\lVert\Xi_k\rVert^2+\lVert\widehat{v}_k\rVert^2\right)\qquad(k\geq0). \tag*{(176)} V ( Ξ k + 1 ) − V ( Ξ k ) + σ k ≤ − ϵ ( ∥ Ξ k ∥ 2 + ∥ v k ∥ 2 ) ( k ≥ 0 ) . (176)
Step 3: Finite-horizon use of the hard IQC. Fix N ≥ 0 N\geq0 N ≥ 0 and sum (176) from k = 0 k=0 k = 0 to N N N :
V ( Ξ N + 1 ) − V ( Ξ 0 ) + ∑ k = 0 N σ k ≤ − ϵ ∑ k = 0 N ( ∥ Ξ k ∥ 2 + ∥ v ^ k ∥ 2 ) . (177) V(\Xi_{N+1})-V(\Xi_0)+\sum_{k=0}^{N}\sigma_k\leq-\epsilon\sum_{k=0}^{N}\left(\lVert\Xi_k\rVert^2+\lVert\widehat{v}_k\rVert^2\right). \tag*{(177)} V ( Ξ N + 1 ) − V ( Ξ 0 ) + k = 0 ∑ N σ k ≤ − ϵ k = 0 ∑ N ( ∥ Ξ k ∥ 2 + ∥ v k ∥ 2 ) . (177)
Next, we verify that the sums ∑ k = 0 N σ k \sum_{k=0}^{N}\sigma_k ∑ k = 0 N σ k on the left are non-negative. Take the actual graph points ( y k , u k ) = ( y k , ∇ f ( y k ) ) (y_k,u_k)=(y_k,\nabla f(y_k)) ( y k , u k ) = ( y k , ∇ f ( y k )) for 0 ≤ k ≤ N 0\leq k\leq N 0 ≤ k ≤ N and pad this finite segment with the minimizer ( 0 , 0 ) (0,0) ( 0 , 0 ) outside { 0 , … , N } \{0,\ldots,N\} { 0 , … , N } , as in Lemma 4.2 . After rate weighting, let r ^ ( N ) \widehat{r}^{(N)} r ( N ) and s ^ ( N ) \widehat{s}^{(N)} s ( N ) denote the corresponding slope-shifted sequences. Lemma 4.4 , with J = τ J=\tau J = τ and h j = 1 / τ h_j=1/\tau h j = 1/ τ , gives
2 ∑ k ∈ Z ⟨ r ^ k ( N ) , s ^ k ( N ) ⟩ − 2 τ ∑ j = 1 τ ρ j ∑ k ∈ Z ⟨ r ^ k ( N ) , s ^ k − j ( N ) ⟩ ≥ 0. 2\sum_{k\in\mathbb{Z}}\langle\widehat{r}^{(N)}_k,\widehat{s}^{(N)}_k\rangle-\frac{2}{\tau}\sum_{j=1}^{\tau}\rho^j\sum_{k\in\mathbb{Z}}\langle\widehat{r}^{(N)}_k,\widehat{s}^{(N)}_{k-j}\rangle\geq0. 2 k ∈ Z ∑ ⟨ r k ( N ) , s k ( N ) ⟩ − τ 2 j = 1 ∑ τ ρ j k ∈ Z ∑ ⟨ r k ( N ) , s k − j ( N ) ⟩ ≥ 0.
For 0 ≤ k ≤ N 0\leq k\leq N 0 ≤ k ≤ N , the padded signals agree with the actual weighted trajectory; for k < 0 k<0 k < 0 or k > N k>N k > N , r ^ k ( N ) = 0 \widehat{r}^{(N)}_k=0 r k ( N ) = 0 . Consequently the all-integer quadratic form above is exactly the horizon sum in (175) .
(the terms with k − j < 0 k-j<0 k − j < 0 vanish by the zero padding, exactly as imposed by the zero multiplier memory). Thus, we have
∑ k = 0 N σ k ≥ 0. (178) \sum_{k=0}^{N}\sigma_k \ge0. \tag*{(178)} k = 0 ∑ N σ k ≥ 0. (178)
Since X ≻ 0 X \succ0 X ≻ 0 , we also have V ( Ξ N + 1 ) ≥ 0 V(\Xi_{N+1}) \ge0 V ( Ξ N + 1 ) ≥ 0 . Combining (177) and (178) therefore yields
∑ k = 0 N ( ∥ Ξ k ∥ 2 + ∥ v ^ k ∥ 2 ) ≤ ϵ − 1 V ( Ξ 0 ) , N ≥ 0. (179) \sum_{k=0}^{N}\left(\lVert\Xi_k\rVert^2+\lVert\widehat{v}_k\rVert^2\right)\le\epsilon^{-1}V(\Xi_0),\qquad N\ge0. \tag*{(179)} k = 0 ∑ N ( ∥ Ξ k ∥ 2 + ∥ v k ∥ 2 ) ≤ ϵ − 1 V ( Ξ 0 ) , N ≥ 0. (179)
In particular, for every k ≥ 0 k\ge0 k ≥ 0 , ∥ Ξ k ∥ ≤ V ( Ξ 0 ) / ϵ \lVert\Xi_k\rVert\le\sqrt{V(\Xi_0)}/\epsilon ∥ Ξ k ∥ ≤ V ( Ξ 0 ) / ϵ . The vector x ^ k \widehat{x}_k x k is a d d d -dimensional block of the plant state; hence,
∥ x ^ k ∥ ≤ V ( Ξ 0 ) / ϵ ( k ≥ 0 ) . \lVert\widehat{x}_k\rVert\le\sqrt{V(\Xi_0)}/\epsilon\qquad(k\ge0). ∥ x k ∥ ≤ V ( Ξ 0 ) / ϵ ( k ≥ 0 ) .
Therefore, in the translated coordinates, we get
∥ x k ∥ ≤ C ρ k , C : = V ( Ξ 0 ) / ϵ . \lVert x_k\rVert\le C\rho^k,\qquad C:=\sqrt{V(\Xi_0)}/\epsilon. ∥ x k ∥ ≤ C ρ k , C := V ( Ξ 0 ) / ϵ .
The certificate ( X , ϵ ) (X,\epsilon) ( X , ϵ ) is independent of the particular f ∈ S 1 , q − 2 f\in\mathcal{S}_{1,q^{-2}} f ∈ S 1 , q − 2 . It remains only to note that the initial augmented state is uniformly controlled by the prescribed finite history. Indeed, ∇ f ( 0 ) = 0 \nabla f(0)=0 ∇ f ( 0 ) = 0 and q − 2 q^{-2} q − 2 -smoothness imply, for every initial-history index j j j ,
∥ v j ∥ = ∥ ∇ f ( y j ) − m y j ∥ ≤ ( q − 2 + m ) ∥ y j ∥ , \lVert v_j\rVert=\lVert\nabla f(y_j)-my_j\rVert\le(q^{-2}+m)\lVert y_j\rVert, ∥ v j ∥ = ∥ ∇ f ( y j ) − m y j ∥ ≤ ( q − 2 + m ) ∥ y j ∥ ,
while y j = ( 1 + ν ) x j − ν x j − 1 y_j=(1+\nu)x_j-\nu x_{j-1} y j = ( 1 + ν ) x j − ν x j − 1 . Moreover, for j ≤ 0 j\le0 j ≤ 0 one has ρ − j ≤ 1 \rho^{-j}\le1 ρ − j ≤ 1 , so the corresponding weighted history variables are no larger in norm than their unweighted counterparts; the multiplier memory is initialized at zero. Consequently there is a constant K K K , depending only on the fixed algorithmic and certificate parameters, such that
V ( Ξ 0 ) ≤ λ max ( X ) ∥ Ξ 0 ∥ 2 ≤ K ∑ j = − τ − 1 0 ∥ x j ∥ 2 . V(\Xi_0)\le\lambda_{\max}(X)\lVert\Xi_0\rVert^2\le K\sum_{j=-\tau-1}^{0}\lVert x_j\rVert^2. V ( Ξ 0 ) ≤ λ m a x ( X ) ∥ Ξ 0 ∥ 2 ≤ K j = − τ − 1 ∑ 0 ∥ x j ∥ 2 .
Thus, C > 0 C>0 C > 0 may be chosen as a finite function of the initial history, uniformly over f ∈ S 1 , q − 2 f\in\mathcal{S}_{1,q^{-2}} f ∈ S 1 , q − 2 . Undoing the translation of the minimizer gives ∥ x k − x ⋆ ∥ ≤ C ρ k \lVert x_k-x_\star\rVert\le C\rho^k ∥ x k − x ⋆ ∥ ≤ C ρ k , as claimed. □ \square □
Proof of Theorem 4.1
The convergence rate lemma presented in § 5.1 –§ 5.2 assumes (H1) and (H2) and establishes Theorem 4.1 for a constant C > 0 C>0 C > 0 and convergence rate ρ ∈ ( 0 , 1 ) \rho\in(0,1) ρ ∈ ( 0 , 1 ) . It therefore suffices to verify that (H1) and (H2) hold for our choice of GMM parameters and for the rate
ρ = 1 − c q M , c ∈ ( 0 , 2 / 3 ) . \rho=1-\frac{cq}{M},\qquad c\in(0,2/3). ρ = 1 − M c q , c ∈ ( 0 , 2/3 ) .
The verification of (H1) is carried out in § 5.3.1 , while that of (H2) is carried out in § 5.3.2 –§ 5.3.3 , thereby completing the proof of Theorem 4.1 .
Verification of hypothesis (H1)
We now verify hypothesis (H1) of the Convergence Rate Lemma (Lemma 5.2 ), namely that the midpoint plant obtained after replacing the gradient by the constant slope m m m is Schur, with all poles contained in { ∣ z ∣ < ρ } \{|z| < \rho\} { ∣ z ∣ < ρ } .
Replacing the gradient with the midpoint slope m = 1 + q − 2 2 m = \frac{1+q^{-2}}{2} m = 2 1 + q − 2 turns the delayed recursion into the quadratic iteration corresponding to Hessian eigenvalue λ = m \lambda= m λ = m . Hence, the midpoint characteristic polynomial in (161) is exactly the quadratic characteristic polynomial in (24) , evaluated at the single gain
δ = α m = 1 + q 2 2 M 2 , \delta= \alpha m = \frac{1+q^2}{2M^2}, δ = α m = 2 M 2 1 + q 2 ,
namely
z τ ( z − 1 ) ( z − β ) + α m [ ( 1 + ν ) z − ν ] = P τ , α m ( z ) . z^\tau(z-1)(z-\beta)+\alpha m\left[(1+\nu)z-\nu\right]=P_{\tau,\alpha m}(z). z τ ( z − 1 ) ( z − β ) + α m [ ( 1 + ν ) z − ν ] = P τ , α m ( z ) .
As q ↓ 0 q \downarrow0 q ↓ 0 , we have δ ↓ 1 2 M 2 \delta\downarrow\frac{1}{2M^2} δ ↓ 2 M 2 1 and this polynomial converges to P ∞ ( ⋅ ; 1 2 M 2 , M − 1 2 ) P_\infty\left(\cdot;\frac{1}{2M^2},M-\frac{1}{2}\right) P ∞ ( ⋅ ; 2 M 2 1 , M − 2 1 ) , whose Schur stability follows from the quadratic analysis of § 3 . Under the Cayley transform (Lemma 3.2 ), Schur stability of P ∞ P_\infty P ∞ is equivalent to Hurwitz stability of its transformed polynomial. Corollary 3.23 establishes this for sufficiently small positive gain, while Lemma 3.24 excludes crossings through the imaginary axis on 0 < δ ≤ M − 2 0<\delta\leq M^{-2} 0 < δ ≤ M − 2 . Since
1 2 M 2 ∈ ( 0 , M − 2 ] , \frac{1}{2M^2}\in(0,M^{-2}], 2 M 2 1 ∈ ( 0 , M − 2 ] ,
the limiting midpoint polynomial is strictly Schur. By continuity of the roots, there is a fixed r ∗ < 1 r_\ast<1 r ∗ < 1 that bounds their moduli for all sufficiently small q > 0 q>0 q > 0 . Since ρ = 1 − c q / M → 1 \rho=1-cq/M\to1 ρ = 1 − c q / M → 1 , we also have r ∗ < ρ r_\ast<\rho r ∗ < ρ for sufficiently small q q q . Thus, the qualitative stability required in hypothesis (H1) is inherited directly from the quadratic analysis when q q q is sufficiently small (or equivalently when κ \kappa κ is sufficiently large).
Verification of hypothesis (H2), part I: exact factorization of the FDI
By Lemma 5.2 and the verification of H1 in § 5.3.1 , it remains only to verify (H2) which says that the FDI in (162) is strictly satisfied at every frequency. The obstacle is now purely one of form. As written, Φ τ , q \Phi_{\tau,q} Φ τ , q that appears in (162) is a 2 × 2 2\times2 2 × 2 Hermitian form involving the transfer function g τ , q g_{\tau,q} g τ , q and the multiplier h ρ , τ h_{\rho,\tau} h ρ , τ , and its sign cannot be read off directly. This section removes that obstacle by reducing the form to a single scalar trigonometric polynomial. The reduction proceeds in three steps: first, a scalar factorization of the Hermitian form; second, a telescoping identity that reveals the endpoint factor ( z − 1 ) (z-1) ( z − 1 ) hidden in the multiplier; and third, the combination of these two identities into a rational representation whose numerator, N τ , q N_{\tau,q} N τ , q , is the quantity whose sign will be analyzed throughout the remainder of the paper.
Lemma 5.8 (scalar factorization). With h = h ρ , τ ( e i θ ) h=h_{\rho,\tau}(e^{i\theta}) h = h ρ , τ ( e i θ ) , g = g τ , q ( ρ e i θ ) g=g_{\tau,q}(\rho e^{i\theta}) g = g τ , q ( ρ e i θ ) , L = q − 2 L=q^{-2} L = q − 2 , and with g ‾ \overline{g} g denoting the complex conjugate of g g g , we have
Φ τ , q ( θ ) = 2 Re [ ( 1 − h ) ( 1 − g ‾ ) ( L g − 1 ) ] . (180) \Phi_{\tau,q}(\theta)=2\operatorname{Re}\left[(1-h)(1-\overline{g})(Lg-1)\right]. \tag*{(180)} Φ τ , q ( θ ) = 2 Re [ ( 1 − h ) ( 1 − g ) ( Lg − 1 ) ] . (180) Proof. Define
Then 1 − h ‾ = W ‾ 1-\overline{h}=\overline{W} 1 − h = W and
2 − h − h ‾ = ( 1 − h ) + ( 1 − h ‾ ) = W + W ‾ . 2-h-\overline{h}=(1-h)+(1-\overline{h})=W+\overline{W}. 2 − h − h = ( 1 − h ) + ( 1 − h ) = W + W .
Using μ = 1 \mu= 1 μ = 1 and L = q − 2 L = q^{-2} L = q − 2 in (154) , the multiplier can therefore be written as
Π 1 , L ( h ) = [ − L ( W + W ˉ ) L W ˉ + W L W + W ˉ − ( W + W ˉ ) ] . \Pi_{1,L}(h)=\begin{bmatrix}-L(W+\bar{W}) & L\bar{W}+W \\ LW+\bar{W} & -(W+\bar{W})\end{bmatrix}. Π 1 , L ( h ) = [ − L ( W + W ˉ ) L W + W ˉ L W ˉ + W − ( W + W ˉ ) ] .
Substituting this matrix into the scalar quadratic form in (162) gives
Φ τ , q ( θ ) = [ g ˉ 1 ] [ − L ( W + W ˉ ) L W ˉ + W L W + W ˉ − ( W + W ˉ ) ] [ g 1 ] = − L ( W + W ˉ ) ∣ g ∣ 2 − ( W + W ˉ ) + g ˉ ( L W ˉ + W ) + ( L W + W ˉ ) g . \begin{aligned}
\Phi_{\tau,q}(\theta)&=\begin{bmatrix}\bar{g} & 1\end{bmatrix}\begin{bmatrix}-L(W+\bar{W}) & L\bar{W}+W \\ LW+\bar{W} & -(W+\bar{W})\end{bmatrix}\begin{bmatrix}g \\ 1\end{bmatrix}\\
&=-L(W+\bar{W})|g|^2-(W+\bar{W})+\bar{g}(L\bar{W}+W)+(LW+\bar{W})g.
\end{aligned} Φ τ , q ( θ ) = [ g ˉ 1 ] [ − L ( W + W ˉ ) L W + W ˉ L W ˉ + W − ( W + W ˉ ) ] [ g 1 ] = − L ( W + W ˉ ) ∣ g ∣ 2 − ( W + W ˉ ) + g ˉ ( L W ˉ + W ) + ( L W + W ˉ ) g .
The two off-diagonal terms are conjugates of each other, because L L L is real:
g ˉ ( L W ˉ + W ) = ( L W + W ˉ ) g ‾ . \bar{g}(L\bar{W}+W)=\overline{(LW+\bar{W})g}. g ˉ ( L W ˉ + W ) = ( L W + W ˉ ) g .
Hence,
Φ τ , q ( θ ) = − ( W + W ˉ ) ( L ∣ g ∣ 2 + 1 ) + 2 Re [ ( L W + W ˉ ) g ] . (181) \Phi_{\tau,q}(\theta)=-(W+\bar{W})(L|g|^2+1)+2\operatorname{Re}\left[(LW+\bar{W})g\right]. \tag*{(181)} Φ τ , q ( θ ) = − ( W + W ˉ ) ( L ∣ g ∣ 2 + 1 ) + 2 Re [ ( L W + W ˉ ) g ] . (181)
We now collect the terms proportional to L L L , and from (181) , these terms are
− L ( W + W ˉ ) ∣ g ∣ 2 + 2 L Re ( W g ) . -L(W+\bar{W})|g|^2+2L\operatorname{Re}(Wg). − L ( W + W ˉ ) ∣ g ∣ 2 + 2 L Re ( W g ) .
Since
Re ( W ∣ g ∣ 2 ) = ∣ g ∣ 2 Re ( W ) = 1 2 ( W + W ˉ ) ∣ g ∣ 2 , \operatorname{Re}(W|g|^2)=|g|^2\operatorname{Re}(W)=\frac{1}{2}(W+\bar{W})|g|^2, Re ( W ∣ g ∣ 2 ) = ∣ g ∣ 2 Re ( W ) = 2 1 ( W + W ˉ ) ∣ g ∣ 2 ,
we have
− L ( W + W ˉ ) ∣ g ∣ 2 + 2 L Re ( W g ) = 2 L Re ( W g ) − 2 L Re ( W ∣ g ∣ 2 ) = 2 L Re [ W g − W g g ˉ ] = 2 L Re [ W g ( 1 − g ˉ ) ] . \begin{aligned}
-L(W+\bar{W})|g|^2+2L\operatorname{Re}(Wg)&=2L\operatorname{Re}(Wg)-2L\operatorname{Re}(W|g|^2)\\
&=2L\operatorname{Re}[Wg-Wg\bar{g}]\\
&=2L\operatorname{Re}[Wg(1-\bar{g})].
\end{aligned} − L ( W + W ˉ ) ∣ g ∣ 2 + 2 L Re ( W g ) = 2 L Re ( W g ) − 2 L Re ( W ∣ g ∣ 2 ) = 2 L Re [ W g − W g g ˉ ] = 2 L Re [ W g ( 1 − g ˉ )] .
It remains to collect the terms independent of L L L . These are
− ( W + W ˉ ) + 2 Re ( W ˉ g ) . -(W+\bar{W})+2\operatorname{Re}(\bar{W}g). − ( W + W ˉ ) + 2 Re ( W ˉ g ) .
Since Re ( W ˉ g ) = Re ( W g ˉ ) \operatorname{Re}(\bar{W}g)=\operatorname{Re}(W\bar{g}) Re ( W ˉ g ) = Re ( W g ˉ ) ,
− ( W + W ˉ ) + 2 Re ( W ˉ g ) = − 2 Re ( W ) + 2 Re ( W g ˉ ) = − 2 Re [ W ( 1 − g ˉ ) ] . \begin{aligned}
-(W+\bar{W})+2\operatorname{Re}(\bar{W}g)&=-2\operatorname{Re}(W)+2\operatorname{Re}(W\bar{g})\\
&=-2\operatorname{Re}[W(1-\bar{g})].
\end{aligned} − ( W + W ˉ ) + 2 Re ( W ˉ g ) = − 2 Re ( W ) + 2 Re ( W g ˉ ) = − 2 Re [ W ( 1 − g ˉ )] .
Combining the L L L -dependent and L L L -independent parts gives
Φ τ , q ( θ ) = 2 L Re [ W g ( 1 − g ˉ ) ] − 2 Re [ W ( 1 − g ˉ ) ] = 2 Re [ W ( 1 − g ˉ ) ( L g − 1 ) ] . \begin{aligned}
\Phi_{\tau,q}(\theta)&=2L\operatorname{Re}[Wg(1-\bar{g})]-2\operatorname{Re}[W(1-\bar{g})]\\
&=2\operatorname{Re}[W(1-\bar{g})(Lg-1)].
\end{aligned} Φ τ , q ( θ ) = 2 L Re [ W g ( 1 − g ˉ )] − 2 Re [ W ( 1 − g ˉ )] = 2 Re [ W ( 1 − g ˉ ) ( Lg − 1 )] .
Finally, substituting W = 1 − h W=1-h W = 1 − h yields
Φ τ , q ( θ ) = 2 Re [ ( 1 − h ) ( 1 − g ˉ ) ( L g − 1 ) ] , \Phi_{\tau,q}(\theta)=2\operatorname{Re}[(1-h)(1-\bar{g})(Lg-1)], Φ τ , q ( θ ) = 2 Re [( 1 − h ) ( 1 − g ˉ ) ( Lg − 1 )] ,
which is (180) .
The factorization is the point at which all matrix notation collapses to a scalar expression. For the remainder of this section we fix notation for the recurring factors. Write, throughout this section,
z = ρ e i θ , D 1 = z − 1 , D 2 = z − β , P = ( M + 1 2 ) z − ( M − 1 2 ) , α = q 2 M 2 . (182) z = \rho e^{i\theta}, \qquad D_1 = z - 1, \qquad D_2 = z - \beta, \qquad P = \left(M + \frac{1}{2}\right)z - \left(M - \frac{1}{2}\right), \qquad\alpha= \frac{q^2}{M^2}. \tag*{(182)} z = ρ e i θ , D 1 = z − 1 , D 2 = z − β , P = ( M + 2 1 ) z − ( M − 2 1 ) , α = M 2 q 2 . (182)
We also set
h ~ ( w ) = 1 τ ∑ k = 0 τ − 1 ( τ − k ) w k for τ ≥ 1. (183) \widetilde{h}(w) = \frac{1}{\tau}\sum_{k=0}^{\tau-1}(\tau-k)w^k \qquad\text{for } \tau\ge1. \tag*{(183)} h ( w ) = τ 1 k = 0 ∑ τ − 1 ( τ − k ) w k for τ ≥ 1. (183)
The next result shows how W = 1 − h W = 1 - h W = 1 − h can be factored in terms of D ‾ 1 \overline{D}_1 D 1 and h ~ ( z ‾ ) \widetilde{h}(\overline{z}) h ( z ) , for h = h ρ , τ h = h_{\rho,\tau} h = h ρ , τ .
Lemma 5.9 (Factorization of W = 1 − h W = 1 - h W = 1 − h ). For h = h ρ , τ h = h_{\rho,\tau} h = h ρ , τ and z = ρ e i θ z = \rho e^{i\theta} z = ρ e i θ , it holds that
W = 1 − h ρ , τ ( e i θ ) = − D ‾ 1 h ~ ( z ‾ ) . (184) W = 1 - h_{\rho,\tau}(e^{i\theta}) = -\overline{D}_1\widetilde{h}(\overline{z}). \tag*{(184)} W = 1 − h ρ , τ ( e i θ ) = − D 1 h ( z ) . (184)
Proof. Recall from (182) that z = ρ e i θ z = \rho e^{i\theta} z = ρ e i θ . Hence,
w : = z ‾ = ρ e − i θ , w j = ρ j e − i j θ . w := \overline{z} = \rho e^{-i\theta}, \qquad w^j = \rho^j e^{-ij\theta}. w := z = ρ e − i θ , w j = ρ j e − ij θ .
Therefore, the term h ρ , τ ( e i θ ) h_{\rho,\tau}(e^{i\theta}) h ρ , τ ( e i θ ) can be written as a polynomial in the variable w w w :
h ρ , τ ( e i θ ) = 1 τ ∑ j = 1 τ ρ j e − i j θ = 1 τ ∑ j = 1 τ w j . h_{\rho,\tau}(e^{i\theta}) = \frac{1}{\tau}\sum_{j=1}^{\tau}\rho^j e^{-ij\theta} = \frac{1}{\tau}\sum_{j=1}^{\tau}w^j. h ρ , τ ( e i θ ) = τ 1 j = 1 ∑ τ ρ j e − ij θ = τ 1 j = 1 ∑ τ w j .
It follows that
1 − h ρ , τ ( e i θ ) = 1 − 1 τ ∑ j = 1 τ w j = 1 τ ∑ j = 1 τ ( 1 − w j ) . 1 - h_{\rho,\tau}(e^{i\theta}) = 1 - \frac{1}{\tau}\sum_{j=1}^{\tau}w^j = \frac{1}{\tau}\sum_{j=1}^{\tau}(1-w^j). 1 − h ρ , τ ( e i θ ) = 1 − τ 1 j = 1 ∑ τ w j = τ 1 j = 1 ∑ τ ( 1 − w j ) .
For each j ≥ 1 j \ge1 j ≥ 1 we use the finite geometric identity
1 − w j = ( 1 − w ) ( 1 + w + ⋯ + w j − 1 ) = ( 1 − w ) ∑ k = 0 j − 1 w k . 1-w^j = (1-w)(1+w+\cdots+w^{j-1}) = (1-w)\sum_{k=0}^{j-1}w^k. 1 − w j = ( 1 − w ) ( 1 + w + ⋯ + w j − 1 ) = ( 1 − w ) k = 0 ∑ j − 1 w k .
Substituting this identity into the previous display gives
1 − h ρ , τ ( e i θ ) = 1 − w τ ∑ j = 1 τ ∑ k = 0 j − 1 w k . 1-h_{\rho,\tau}(e^{i\theta}) = \frac{1-w}{\tau}\sum_{j=1}^{\tau}\sum_{k=0}^{j-1}w^k. 1 − h ρ , τ ( e i θ ) = τ 1 − w j = 1 ∑ τ k = 0 ∑ j − 1 w k .
We now interchange the order of summation. For a fixed power w k w^k w k with 0 ≤ k ≤ τ − 1 0 \le k \le\tau-1 0 ≤ k ≤ τ − 1 , the index j j j must satisfy j ≥ k + 1 j \ge k+1 j ≥ k + 1 and j ≤ τ j \le\tau j ≤ τ ; hence, this power appears exactly τ − k \tau-k τ − k times. Therefore,
∑ j = 1 τ ∑ k = 0 j − 1 w k = ∑ k = 0 τ − 1 ( τ − k ) w k . \sum_{j=1}^{\tau}\sum_{k=0}^{j-1}w^k = \sum_{k=0}^{\tau-1}(\tau-k)w^k. j = 1 ∑ τ k = 0 ∑ j − 1 w k = k = 0 ∑ τ − 1 ( τ − k ) w k .
Using the definition (183) of h ~ \widetilde{h} h , we obtain
1 − h ρ , τ ( e i θ ) = ( 1 − w ) 1 τ ∑ k = 0 τ − 1 ( τ − k ) w k = ( 1 − w ) h ~ ( w ) . 1-h_{\rho,\tau}(e^{i\theta}) = (1-w)\frac{1}{\tau}\sum_{k=0}^{\tau-1}(\tau-k)w^k = (1-w)\widetilde{h}(w). 1 − h ρ , τ ( e i θ ) = ( 1 − w ) τ 1 k = 0 ∑ τ − 1 ( τ − k ) w k = ( 1 − w ) h ( w ) .
Finally, since D 1 = z − 1 D_1=z-1 D 1 = z − 1 , we have
D ˉ 1 = z ˉ − 1 = w − 1 , 1 − w = − D ˉ 1 . \bar{D}_1=\bar{z}-1=w-1,\qquad1-w=-\bar{D}_1. D ˉ 1 = z ˉ − 1 = w − 1 , 1 − w = − D ˉ 1 .
Evaluating the preceding factorization at w = z ˉ w=\bar{z} w = z ˉ gives
W : = 1 − h ρ , τ ( e i θ ) = ( 1 − z ˉ ) h ~ ( z ˉ ) = − D ˉ 1 h ~ ( z ˉ ) , W:=1-h_{\rho,\tau}(e^{i\theta})=(1-\bar{z})\tilde{h}(\bar{z})=-\bar{D}_1\tilde{h}(\bar{z}), W := 1 − h ρ , τ ( e i θ ) = ( 1 − z ˉ ) h ~ ( z ˉ ) = − D ˉ 1 h ~ ( z ˉ ) ,
which proves (184) .
Building on the previous two lemmas, the next lemma shows that Φ τ , q ( θ ) \Phi_{\tau,q}(\theta) Φ τ , q ( θ ) can be expressed as the ratio of a real even trigonometric polynomial N τ , q ( θ ) N_{\tau,q}(\theta) N τ , q ( θ ) and a strictly positive scalar term for every angle θ \theta θ . As such, this lemma implies that the question about the negativity of Φ τ , q ( θ ) \Phi_{\tau,q}(\theta) Φ τ , q ( θ ) reduces to the negativity of the polynomial N τ , q ( θ ) N_{\tau,q}(\theta) N τ , q ( θ ) .
Proposition 5.10 (Representation of Φ τ , q \Phi_{\tau,q} Φ τ , q as a ratio). For every q ∈ ( 0 , 1 ] q\in(0,1] q ∈ ( 0 , 1 ] and θ ∈ [ 0 , 2 π ) \theta\in[0,2\pi) θ ∈ [ 0 , 2 π ) ,
Φ τ , q ( θ ) = N τ , q ( θ ) M 2 ∣ D 1 ∣ 2 ∣ D 2 ∣ 2 , N τ , q ( θ ) : = 2 Re ( D ˉ 1 S ) , (185) \Phi_{\tau,q}(\theta)=\frac{N_{\tau,q}(\theta)}{M^2|D_1|^2|D_2|^2},\qquad N_{\tau,q}(\theta):=2\operatorname{Re}(\bar{D}_1S), \tag*{(185)} Φ τ , q ( θ ) = M 2 ∣ D 1 ∣ 2 ∣ D 2 ∣ 2 N τ , q ( θ ) , N τ , q ( θ ) := 2 Re ( D ˉ 1 S ) , (185) where, with z = ρ e i θ z=\rho e^{i\theta} z = ρ e i θ and h ~ = h ~ ( z ˉ ) \tilde{h}=\tilde{h}(\bar{z}) h ~ = h ~ ( z ˉ ) ,
S = D ˉ 1 h ~ z − τ P D ˉ 2 + M 2 ∣ D 1 ∣ 2 ∣ h ~ ∣ 2 ∣ D 2 ∣ 2 + α h ~ ∣ P ∣ 2 ρ − 2 τ + α M 2 h ~ z − τ P ˉ D 1 D 2 , (186) S=\bar{D}_1\tilde{h}z^{-\tau}P\bar{D}_2+M^2|D_1|^2|\tilde{h}|^2|D_2|^2+\alpha\tilde{h}|P|^2\rho^{-2\tau}+\alpha M^2\tilde{h}z^{-\tau}\bar{P}D_1D_2, \tag*{(186)} S = D ˉ 1 h ~ z − τ P D ˉ 2 + M 2 ∣ D 1 ∣ 2 ∣ h ~ ∣ 2 ∣ D 2 ∣ 2 + α h ~ ∣ P ∣ 2 ρ − 2 τ + α M 2 h ~ z − τ P ˉ D 1 D 2 , (186) *and α = q 2 / M 2 \alpha=q^2/M^2 α = q 2 / M 2 . The numerator N τ , q N_{\tau,q} N τ , q is an even real trigonometric polynomial of degree at most 2 τ + 2 2\tau+2 2 τ + 2 , and the denominator in (185) is strictly positive for all θ \theta θ when q > 0 q>0 q > 0 and 0 < c < 2 0<c<2 0 < c < 2 .
Proof. We start from the scalar factorization in Lemma 5.8. With W = 1 − h ρ , τ ( e i θ ) W=1-h_{\rho,\tau}(e^{i\theta}) W = 1 − h ρ , τ ( e i θ ) , it reads
Φ τ , q ( θ ) = 2 Re [ W ( 1 − g ˉ ) ( L g − 1 ) ] . \Phi_{\tau,q}(\theta)=2\operatorname{Re}\left[W(1-\bar{g})(Lg-1)\right]. Φ τ , q ( θ ) = 2 Re [ W ( 1 − g ˉ ) ( Lg − 1 ) ] .
Expanding the product gives four terms:
Φ τ , q ( θ ) = 2 Re { W L g − W − W g ˉ L g + W g ˉ } . (187) \Phi_{\tau,q}(\theta)=2\operatorname{Re}\{WLg-W-W\bar{g}Lg+W\bar{g}\}. \tag*{(187)} Φ τ , q ( θ ) = 2 Re { W Lg − W − W g ˉ Lg + W g ˉ } . (187)
We now rewrite each of these four terms using the notation in (182) . Since L = q − 2 L=q^{-2} L = q − 2 and α = q 2 / M 2 \alpha=q^2/M^2 α = q 2 / M 2 , using the definition of g g g given in (145) , we obtain
L g = − z − τ P M 2 D 1 D 2 , g ˉ = − α z ˉ − τ P ˉ D ˉ 1 D ˉ 2 . (188) Lg=-\frac{z^{-\tau}P}{M^2D_1D_2},\qquad\bar{g}=-\alpha\frac{\bar{z}^{-\tau}\bar{P}}{\bar{D}_1\bar{D}_2}. \tag*{(188)} Lg = − M 2 D 1 D 2 z − τ P , g ˉ = − α D ˉ 1 D ˉ 2 z ˉ − τ P ˉ . (188)
Moreover, Lemma 5.9 gives
W = − D ˉ 1 h ~ , h ~ = h ~ ( z ˉ ) . (189) W=-\bar{D}_1\tilde{h},\qquad\tilde{h}=\tilde{h}(\bar{z}). \tag*{(189)} W = − D ˉ 1 h ~ , h ~ = h ~ ( z ˉ ) . (189)
Substituting (188) –(189) into (187) , we place the four summands over the common complex denominator
M 2 D 1 ∣ D 2 ∣ 2 = M 2 D 1 D 2 D ˉ 2 . M^2D_1|D_2|^2=M^2D_1D_2\bar{D}_2. M 2 D 1 ∣ D 2 ∣ 2 = M 2 D 1 D 2 D ˉ 2 .
The four contributions are as follows: First,
W L g = ( − D ˉ 1 h ~ ) ( − z − τ P M 2 D 1 D 2 ) = D ˉ 1 h ~ z − τ P D ˉ 2 M 2 D 1 ∣ D 2 ∣ 2 . WLg=(-\bar{D}_1\tilde{h})\left(-\frac{z^{-\tau}P}{M^2D_1D_2}\right)=\frac{\bar{D}_1\tilde{h}z^{-\tau}P\bar{D}_2}{M^2D_1|D_2|^2}. W Lg = ( − D ˉ 1 h ~ ) ( − M 2 D 1 D 2 z − τ P ) = M 2 D 1 ∣ D 2 ∣ 2 D ˉ 1 h ~ z − τ P D ˉ 2 .
Second,
− W = D ‾ 1 h ~ = M 2 D 1 D ‾ 1 ∣ h ~ ∣ ∣ D 2 ∣ 2 M 2 D 1 ∣ D 2 ∣ 2 = M 2 ∣ D 1 ∣ 2 ∣ h ~ ∣ ∣ D 2 ∣ 2 M 2 D 1 ∣ D 2 ∣ 2 . -W=\overline{D}_1\widetilde{h}=\frac{M^2D_1\overline{D}_1|\widetilde{h}| |D_2|^2}{M^2D_1|D_2|^2}=\frac{M^2|D_1|^2|\widetilde{h}| |D_2|^2}{M^2D_1|D_2|^2}. − W = D 1 h = M 2 D 1 ∣ D 2 ∣ 2 M 2 D 1 D 1 ∣ h ∣∣ D 2 ∣ 2 = M 2 D 1 ∣ D 2 ∣ 2 M 2 ∣ D 1 ∣ 2 ∣ h ∣∣ D 2 ∣ 2 .
Third, using z − τ ( z ‾ ) − τ = ∣ z ∣ − 2 τ = ρ − 2 τ z^{-\tau}(\overline{z})^{-\tau}=|z|^{-2\tau}=\rho^{-2\tau} z − τ ( z ) − τ = ∣ z ∣ − 2 τ = ρ − 2 τ ,
− W g ‾ L g = − ( − D ‾ 1 h ~ ) ( − α ( z ‾ ) − τ P ‾ D ‾ 1 D ‾ 2 ) ( z − τ P M 2 D 1 D 2 ) = α h ~ ∣ P ∣ 2 ρ − 2 τ M 2 D 1 ∣ D 2 ∣ 2 . -W\overline{g}Lg=-(-\overline{D}_1\widetilde{h})\left(-\alpha\frac{(\overline{z})^{-\tau}\overline{P}}{\overline{D}_1\overline{D}_2}\right)\left(\frac{z^{-\tau}P}{M^2D_1D_2}\right)=\frac{\alpha\widetilde{h}|P|^2\rho^{-2\tau}}{M^2D_1|D_2|^2}. − W g Lg = − ( − D 1 h ) ( − α D 1 D 2 ( z ) − τ P ) ( M 2 D 1 D 2 z − τ P ) = M 2 D 1 ∣ D 2 ∣ 2 α h ∣ P ∣ 2 ρ − 2 τ .
Finally,
W g ‾ = ( − D ‾ 1 h ~ ) ( − α ( z ‾ ) − τ P ‾ D ‾ 1 D ‾ 2 ) = α h ~ z − τ P ‾ D ‾ 2 = α M 2 h ~ z − τ P ‾ D 1 D 2 M 2 D 1 ∣ D 2 ∣ 2 . W\overline{g}=(-\overline{D}_1\widetilde{h})\left(-\alpha\frac{(\overline{z})^{-\tau}\overline{P}}{\overline{D}_1\overline{D}_2}\right)=\frac{\alpha\widetilde{h}z^{-\tau}\overline{P}}{\overline{D}_2}=\frac{\alpha M^2\widetilde{h}z^{-\tau}\overline{P}D_1D_2}{M^2D_1|D_2|^2}. W g = ( − D 1 h ) ( − α D 1 D 2 ( z ) − τ P ) = D 2 α h z − τ P = M 2 D 1 ∣ D 2 ∣ 2 α M 2 h z − τ P D 1 D 2 .
This calculation also explains the sign of the last term: both factors in W g ‾ W\overline{g} W g carry a minus sign; hence, the two minus signs cancel. Adding the four displayed fractions yields
W L g − W − W g ‾ L g + W g ‾ = S M 2 D 1 ∣ D 2 ∣ 2 . (190) WLg-W-W\overline{g}Lg+W\overline{g}=\frac{S}{M^2D_1|D_2|^2}. \tag*{(190)} W Lg − W − W g Lg + W g = M 2 D 1 ∣ D 2 ∣ 2 S . (190)
where S S S is exactly the expression in (186) . It remains only to convert the denominator in (190) into a real positive denominator. Since
1 D 1 = D ‾ 1 ∣ D 1 ∣ 2 , \frac{1}{D_1}=\frac{\overline{D}_1}{|D_1|^2}, D 1 1 = ∣ D 1 ∣ 2 D 1 ,
we have
S M 2 D 1 ∣ D 2 ∣ 2 = D ‾ 1 S M 2 ∣ D 1 ∣ 2 ∣ D 2 ∣ 2 . \frac{S}{M^2D_1|D_2|^2}=\frac{\overline{D}_1S}{M^2|D_1|^2|D_2|^2}. M 2 D 1 ∣ D 2 ∣ 2 S = M 2 ∣ D 1 ∣ 2 ∣ D 2 ∣ 2 D 1 S .
Taking twice the real part gives
Φ τ , q ( θ ) = 2 Re ( D ‾ 1 S ) M 2 ∣ D 1 ∣ 2 ∣ D 2 ∣ 2 . \Phi_{\tau,q}(\theta)=\frac{2\operatorname{Re}(\overline{D}_1S)}{M^2|D_1|^2|D_2|^2}. Φ τ , q ( θ ) = M 2 ∣ D 1 ∣ 2 ∣ D 2 ∣ 2 2 Re ( D 1 S ) .
which proves (185) with N τ , q ( θ ) = 2 Re ( D ‾ 1 S ) N_{\tau,q}(\theta)=2\operatorname{Re}(\overline{D}_1S) N τ , q ( θ ) = 2 Re ( D 1 S ) .
We next record why this numerator is a finite trigonometric polynomial of the claimed degree. The quantities D 1 , D 2 , P D_1,D_2,P D 1 , D 2 , P are affine polynomials in z = ρ e i θ z=\rho e^{i\theta} z = ρ e i θ , while their conjugates are affine polynomials in z ‾ = ρ e − i θ \overline{z}=\rho e^{-i\theta} z = ρ e − i θ . The factor h ~ ( z ‾ ) \widetilde{h}(\overline{z}) h ( z ) is a polynomial of degree τ − 1 \tau-1 τ − 1 in z ‾ \overline{z} z . Hence, every term in D ‾ 1 S \overline{D}_1S D 1 S is a finite linear combination of monomials z a z ‾ b z^a\overline{z}^b z a z b , possibly also multiplied by z − τ z^{-\tau} z − τ or z ‾ − τ \overline{z}^{-\tau} z − τ . Therefore, after writing z = ρ e i θ z=\rho e^{i\theta} z = ρ e i θ and z ‾ = ρ e − i θ \overline{z}=\rho e^{-i\theta} z = ρ e − i θ , each monomial is a constant times e i ℓ θ e^{i\ell\theta} e i ℓ θ for some integer ℓ \ell ℓ . Inspecting the largest possible powers in the four terms of D ‾ 1 S \overline{D}_1S D 1 S shows that ∣ ℓ ∣ ≤ 2 τ + 2 |\ell|\leq2\tau+2 ∣ ℓ ∣ ≤ 2 τ + 2 . The worst negative exponent can occur in the first term, where the factors z − τ z^{-\tau} z − τ , h ~ ( z ‾ ) \widetilde{h}(\overline{z}) h ( z ) , and the three affine conjugate factors contribute down to − 2 τ − 2 -2\tau-2 − 2 τ − 2 ; the largest positive exponent is smaller, at most τ + 2 \tau+2 τ + 2 , and the remaining two terms lie within these bounds. Thus, N τ , q N_{\tau,q} N τ , q is a real trigonometric polynomial of degree at most 2 τ + 2 2\tau+2 2 τ + 2 .
The numerator is real by definition, because it is twice a real part. It is even because the underlying transfer functions have real coefficients. More explicitly,
h ρ , τ ( e − i θ ) = h ρ , τ ( e i θ ) ‾ , g τ , q ( ρ e − i θ ) = g τ , q ( ρ e i θ ) ‾ . h_{\rho,\tau}(e^{-i\theta})=\overline{h_{\rho,\tau}(e^{i\theta})},\qquad g_{\tau,q}(\rho e^{-i\theta})=\overline{g_{\tau,q}(\rho e^{i\theta})}. h ρ , τ ( e − i θ ) = h ρ , τ ( e i θ ) , g τ , q ( ρ e − i θ ) = g τ , q ( ρ e i θ ) .
Substituting these identities into the scalar factorization shows that Φ τ , q ( − θ ) = Φ τ , q ( θ ) \Phi_{\tau,q}(-\theta)=\Phi_{\tau,q}(\theta) Φ τ , q ( − θ ) = Φ τ , q ( θ ) . The denominator M 2 ∣ D 1 ∣ 2 ∣ D 2 ∣ 2 M^2|D_1|^2|D_2|^2 M 2 ∣ D 1 ∣ 2 ∣ D 2 ∣ 2 is also even in θ \theta θ ; therefore, N τ , q N_{\tau,q} N τ , q is even.
Finally we prove strict positivity of the denominator. In the parameter range used here, q > 0 q > 0 q > 0 and 0 < c < 2 0 < c < 2 0 < c < 2 , which implies that
ρ = 1 − c q M < 1. \rho= 1 - \frac{cq}{M} < 1. ρ = 1 − M c q < 1.
Thus, D 1 = z − 1 D_1 = z - 1 D 1 = z − 1 cannot vanish, since ∣ z ∣ = ρ < 1 |z| = \rho< 1 ∣ z ∣ = ρ < 1 . For D 2 D_2 D 2 , note that
D 2 = 0 ⟹ z = ρ e i θ = β = 1 − 2 q M . D_2 = 0 \quad\Longrightarrow\quad z = \rho e^{i\theta} = \beta= 1 - \frac{2q}{M}. D 2 = 0 ⟹ z = ρ e i θ = β = 1 − M 2 q .
Since τ ≥ 1 \tau\ge1 τ ≥ 1 , we have M = 2 τ + 1 ≥ 3 M = 2\tau+ 1 \ge3 M = 2 τ + 1 ≥ 3 ; together with q ∈ ( 0 , 1 ] q \in(0, 1] q ∈ ( 0 , 1 ] , this gives
β = 1 − 2 q M ≥ 1 − 2 3 > 0. \beta= 1 - \frac{2q}{M} \ge1 - \frac{2}{3} > 0. β = 1 − M 2 q ≥ 1 − 3 2 > 0.
Thus, β \beta β is real and positive, and D 2 = 0 D_2 = 0 D 2 = 0 would imply that θ = 0 \theta= 0 θ = 0 and ρ = β \rho= \beta ρ = β . On the other hand, ρ = β \rho= \beta ρ = β is equivalent to
1 − c q M = 1 − 2 q M , 1 - \frac{cq}{M} = 1 - \frac{2q}{M}, 1 − M c q = 1 − M 2 q ,
which gives c = 2 c = 2 c = 2 , contradicting c < 2 c < 2 c < 2 . Hence, D 2 ≠ 0 D_2 \ne0 D 2 = 0 as well, and therefore
M 2 ∣ D 1 ∣ 2 ∣ D 2 ∣ 2 > 0 , ∀ θ ∈ [ 0 , 2 π ) . M^2 |D_1|^2 |D_2|^2 > 0, \qquad\forall\theta\in[0, 2\pi). M 2 ∣ D 1 ∣ 2 ∣ D 2 ∣ 2 > 0 , ∀ θ ∈ [ 0 , 2 π ) .
Verification of hypothesis (H2), part II: the all-delay positivity certificate
Given the factorization (185) , to verify (H2), it suffices to show that the numerator N τ , q < 0 N_{\tau,q} < 0 N τ , q < 0 for all τ \tau τ as long as q q q is sufficiently small (when κ ≥ κ 0 ( τ , c ) \kappa\ge\kappa_0(\tau,c) κ ≥ κ 0 ( τ , c ) or equivalently when q < q 0 : = κ 0 ( τ , c ) − 1 / 2 q < q_0 := \kappa_0(\tau,c)^{-1/2} q < q 0 := κ 0 ( τ , c ) − 1/2 ). One complication is that the numerator N τ , q N_{\tau,q} N τ , q depends on both the delay τ \tau τ and the condition parameter q q q .
We first remove the q q q -dependence by passing to the limit q ↓ 0 q \downarrow0 q ↓ 0 , in which case the FDI (N τ , q < 0 N_{\tau,q} < 0 N τ , q < 0 ) is equivalent to requiring that a single explicit polynomial r τ ( cos θ ) r_\tau(\cos\theta) r τ ( cos θ ) be positive for all θ \theta θ . We then exploit the structure of this polynomial to establish this positivity and use a continuity argument in q q q to conclude that N τ , q < 0 N_{\tau,q} < 0 N τ , q < 0 for all sufficiently small q q q . Although this argument is asymptotic in q q q and does not identify the threshold q 0 q_0 q 0 below which the conclusion holds, in § 6 , specifically Theorem 6.1 , we provide a more refined argument that yields an explicit admissible bound for q q q and characterizes its dependence on τ \tau τ and c c c .
For fixed θ ∈ ( 0 , π ] \theta\in(0,\pi] θ ∈ ( 0 , π ] and q ↓ 0 q \downarrow0 q ↓ 0 , we have ρ = 1 − c q / M → 1 \rho= 1 - cq/M \to1 ρ = 1 − c q / M → 1 , β → 1 \beta\to1 β → 1 as well as
h ρ , τ ( e i θ ) → H τ ( θ ) : = 1 τ ∑ j = 1 τ e − i j θ . (191) h_{\rho,\tau}(e^{i\theta}) \to H_\tau(\theta) := \frac{1}{\tau}\sum_{j=1}^{\tau} e^{-ij\theta}. \tag*{(191)} h ρ , τ ( e i θ ) → H τ ( θ ) := τ 1 j = 1 ∑ τ e − ij θ . (191)
Moreover, by the definition in (145) , we have g τ , q ( ρ e i θ ) → 0 g_{\tau,q}(\rho e^{i\theta}) \to0 g τ , q ( ρ e i θ ) → 0 and q − 2 g τ , q ( ρ e i θ ) → A τ , 0 ( θ ) q^{-2}g_{\tau,q}(\rho e^{i\theta}) \to A_{\tau,0}(\theta) q − 2 g τ , q ( ρ e i θ ) → A τ , 0 ( θ ) , where
A τ , 0 ( θ ) = − e − i τ θ ( ( M + 1 2 ) e i θ − ( M − 1 2 ) ) M 2 ( e i θ − 1 ) 2 . (192) A_{\tau,0}(\theta) = -\frac{e^{-i\tau\theta}\left(\left(M+\frac{1}{2}\right)e^{i\theta}-\left(M-\frac{1}{2}\right)\right)}{M^2(e^{i\theta}-1)^2}. \tag*{(192)} A τ , 0 ( θ ) = − M 2 ( e i θ − 1 ) 2 e − i τ θ ( ( M + 2 1 ) e i θ − ( M − 2 1 ) ) . (192)
Thus, by Lemma 5.8 , we have
Φ τ , 0 ( θ ) : = lim q ↓ 0 Φ τ , q ( θ ) = 2 Re [ ( 1 − H τ ( θ ) ) ( A τ , 0 ( θ ) − 1 ) ] . (193) \Phi_{\tau,0}(\theta) := \lim_{q\downarrow0}\Phi_{\tau,q}(\theta) = 2\operatorname{Re}\left[(1-H_\tau(\theta))(A_{\tau,0}(\theta)-1)\right]. \tag*{(193)} Φ τ , 0 ( θ ) := q ↓ 0 lim Φ τ , q ( θ ) = 2 Re [ ( 1 − H τ ( θ )) ( A τ , 0 ( θ ) − 1 ) ] . (193)
The endpoint θ = 0 \theta= 0 θ = 0 is defined by continuity. In particular, the following calculation makes this continuous endpoint value explicit. Although A τ , 0 ( θ ) A_{\tau,0}(\theta) A τ , 0 ( θ ) itself has a second-order pole at the origin, the pole is cancelled in the real part of the product in (193) ; the resulting endpoint magnitude is Θ ( 1 ) \Theta(1) Θ ( 1 ) as τ → ∞ \tau\to\infty τ → ∞ .
Lemma 5.11 (endpoint value of the limiting FDI). The continuous extension of Φ τ , 0 \Phi_{\tau,0} Φ τ , 0 to θ = 0 \theta=0 θ = 0 satisfies
Φ τ , 0 ( 0 ) = lim θ → 0 Φ τ , 0 ( θ ) = − τ + 1 3 M . (194) \Phi_{\tau,0}(0)=\lim_{\theta\to0}\Phi_{\tau,0}(\theta)=-\frac{\tau+1}{3M}. \tag*{(194)} Φ τ , 0 ( 0 ) = θ → 0 lim Φ τ , 0 ( θ ) = − 3 M τ + 1 . (194) Proof. We expand the two factors in (193) . First, recall that H τ ( θ ) = 1 τ ∑ j = 1 τ e − i j θ H_{\tau}(\theta)=\frac{1}{\tau}\sum_{j=1}^{\tau}e^{-ij\theta} H τ ( θ ) = τ 1 ∑ j = 1 τ e − ij θ . Using e − i j θ = 1 − i j θ − j 2 θ 2 / 2 + O ( θ 3 ) e^{-ij\theta}=1-ij\theta-j^{2}\theta^{2}/2+O(\theta^{3}) e − ij θ = 1 − ij θ − j 2 θ 2 /2 + O ( θ 3 ) and the sums
1 τ ∑ j = 1 τ j = τ + 1 2 , 1 τ ∑ j = 1 τ j 2 = ( τ + 1 ) ( 2 τ + 1 ) 6 = ( τ + 1 ) M 6 , \frac{1}{\tau}\sum_{j=1}^{\tau}j=\frac{\tau+1}{2},\qquad\frac{1}{\tau}\sum_{j=1}^{\tau}j^{2}=\frac{(\tau+1)(2\tau+1)}{6}=\frac{(\tau+1)M}{6}, τ 1 j = 1 ∑ τ j = 2 τ + 1 , τ 1 j = 1 ∑ τ j 2 = 6 ( τ + 1 ) ( 2 τ + 1 ) = 6 ( τ + 1 ) M , we obtain
1 − H τ ( θ ) = i τ + 1 2 θ + ( τ + 1 ) M 12 θ 2 + O ( θ 3 ) . (195) 1-H_{\tau}(\theta)=i\frac{\tau+1}{2}\theta+\frac{(\tau+1)M}{12}\theta^{2}+O(\theta^{3}). \tag*{(195)} 1 − H τ ( θ ) = i 2 τ + 1 θ + 12 ( τ + 1 ) M θ 2 + O ( θ 3 ) . (195) Next consider A τ , 0 A_{\tau,0} A τ , 0 . The numerator in (192) has the expansion
e − i τ θ ( ( M + 1 2 ) e i θ − ( M − 1 2 ) ) e^{-i\tau\theta}\left(\left(M+\frac{1}{2}\right)e^{i\theta}-\left(M-\frac{1}{2}\right)\right) e − i τ θ ( ( M + 2 1 ) e i θ − ( M − 2 1 ) ) = 1 + i ( M + 1 2 − τ ) θ + O ( θ 2 ) = 1 + i ( τ + 3 2 ) θ + O ( θ 2 ) , =1+i\left(M+\frac{1}{2}-\tau\right)\theta+O(\theta^{2})=1+i\left(\tau+\frac{3}{2}\right)\theta+O(\theta^{2}), = 1 + i ( M + 2 1 − τ ) θ + O ( θ 2 ) = 1 + i ( τ + 2 3 ) θ + O ( θ 2 ) , where M = 2 τ + 1 M=2\tau+1 M = 2 τ + 1 was used in the last equality. Also,
( e i θ − 1 ) 2 = − θ 2 − i θ 3 + O ( θ 4 ) , 1 ( e i θ − 1 ) 2 = − 1 θ 2 ( 1 − i θ + O ( θ 2 ) ) . \left(e^{i\theta}-1\right)^{2}=-\theta^{2}-i\theta^{3}+O(\theta^{4}),\qquad\frac{1}{\left(e^{i\theta}-1\right)^{2}}=-\frac{1}{\theta^{2}}\left(1-i\theta+O(\theta^{2})\right). ( e i θ − 1 ) 2 = − θ 2 − i θ 3 + O ( θ 4 ) , ( e i θ − 1 ) 2 1 = − θ 2 1 ( 1 − i θ + O ( θ 2 ) ) . Substituting these expansions into (192) gives
A τ , 0 ( θ ) = 1 M 2 θ 2 ( 1 + i ( τ + 3 2 ) θ + O ( θ 2 ) ) ( 1 − i θ + O ( θ 2 ) ) A_{\tau,0}(\theta)=\frac{1}{M^{2}\theta^{2}}\left(1+i\left(\tau+\frac{3}{2}\right)\theta+O(\theta^{2})\right)\left(1-i\theta+O(\theta^{2})\right) A τ , 0 ( θ ) = M 2 θ 2 1 ( 1 + i ( τ + 2 3 ) θ + O ( θ 2 ) ) ( 1 − i θ + O ( θ 2 ) ) = 1 M 2 θ 2 + i 2 M θ + O ( 1 ) . (196) =\frac{1}{M^{2}\theta^{2}}+\frac{i}{2M\theta}+O(1). \tag*{(196)} = M 2 θ 2 1 + 2 M θ i + O ( 1 ) . (196) The subtraction of 1 1 1 in A τ , 0 ( θ ) − 1 A_{\tau,0}(\theta)-1 A τ , 0 ( θ ) − 1 only changes the bounded O ( 1 ) O(1) O ( 1 ) term. Combining (195) and (196) , we find
( 1 − H τ ( θ ) ) ( A τ , 0 ( θ ) − 1 ) = ( i τ + 1 2 θ + ( τ + 1 ) M 12 θ 2 + O ( θ 3 ) ) ( 1 M 2 θ 2 + i 2 M θ + O ( 1 ) ) = i ( τ + 1 ) 2 M 2 θ − τ + 1 4 M + τ + 1 12 M + O ( θ ) = i ( τ + 1 ) 2 M 2 θ − τ + 1 6 M + O ( θ ) . \begin{aligned}
(1-H_{\tau}(\theta))(A_{\tau,0}(\theta)-1)
&=\left(i\frac{\tau+1}{2}\theta+\frac{(\tau+1)M}{12}\theta^{2}+O(\theta^{3})\right)\left(\frac{1}{M^{2}\theta^{2}}+\frac{i}{2M\theta}+O(1)\right)\\
&=\frac{i(\tau+1)}{2M^{2}\theta}-\frac{\tau+1}{4M}+\frac{\tau+1}{12M}+O(\theta)\\
&=\frac{i(\tau+1)}{2M^{2}\theta}-\frac{\tau+1}{6M}+O(\theta).
\end{aligned} ( 1 − H τ ( θ )) ( A τ , 0 ( θ ) − 1 ) = ( i 2 τ + 1 θ + 12 ( τ + 1 ) M θ 2 + O ( θ 3 ) ) ( M 2 θ 2 1 + 2 M θ i + O ( 1 ) ) = 2 M 2 θ i ( τ + 1 ) − 4 M τ + 1 + 12 M τ + 1 + O ( θ ) = 2 M 2 θ i ( τ + 1 ) − 6 M τ + 1 + O ( θ ) . Although ( 1 − H τ ( θ ) ) ( A τ , 0 ( θ ) − 1 ) (1-H_{\tau}(\theta))(A_{\tau,0}(\theta)-1) ( 1 − H τ ( θ )) ( A τ , 0 ( θ ) − 1 ) has a purely imaginary O ( θ − 1 ) O(\theta^{-1}) O ( θ − 1 ) singularity, its real part has the finite limit − τ + 1 6 M -\frac{\tau+1}{6M} − 6 M τ + 1 . Therefore,
Φ τ , 0 ( 0 ) : = lim θ → 0 Φ τ , 0 ( θ ) = − τ + 1 3 M . \Phi_{\tau,0}(0):=\lim_{\theta\to0}\Phi_{\tau,0}(\theta)=-\frac{\tau+1}{3M}. Φ τ , 0 ( 0 ) := θ → 0 lim Φ τ , 0 ( θ ) = − 3 M τ + 1 . This is the value assigned to Φ τ , 0 ( 0 ) \Phi_{\tau,0}(0) Φ τ , 0 ( 0 ) in the continuous extension.
We now express the limiting function Φ τ , 0 ( θ ) \Phi_{\tau,0}(\theta) Φ τ , 0 ( θ ) in terms of a polynomial r τ r_{\tau} r τ .
Lemma 5.12 (explicit limiting polynomial). For every τ ≥ 1 \tau\ge1 τ ≥ 1 there is a real polynomial r τ r_\tau r τ of degree 2 τ 2\tau 2 τ such that
Φ τ , 0 ( θ ) = − r τ ( cos θ ) 4 M 2 , 0 ≤ θ ≤ 2 π . (197) \Phi_{\tau,0}(\theta)=-\frac{r_\tau(\cos\theta)}{4M^2},\qquad0\le\theta\le2\pi. \tag*{(197)} Φ τ , 0 ( θ ) = − 4 M 2 r τ ( cos θ ) , 0 ≤ θ ≤ 2 π . (197) It is given explicitly by
r τ ( cos θ ) = c 0 + 2 ∑ m = 1 τ c m cos ( m θ ) + 2 ∑ m = τ + 1 2 τ c m cos ( m θ ) , r_\tau(\cos\theta)=c_0+2\sum_{m=1}^{\tau}c_m\cos(m\theta)+2\sum_{m=\tau+1}^{2\tau}c_m\cos(m\theta), r τ ( cos θ ) = c 0 + 2 m = 1 ∑ τ c m cos ( m θ ) + 2 m = τ + 1 ∑ 2 τ c m cos ( m θ ) , c 0 = 8 M 2 − 2 ( τ + 1 ) , c_0=8M^2-2(\tau+1), c 0 = 8 M 2 − 2 ( τ + 1 ) , c m = − 4 M 2 τ − 2 ( τ + 1 ) , 1 ≤ m ≤ τ , c_m=-\frac{4M^2}{\tau}-2(\tau+1),\qquad1\le m\le\tau, c m = − τ 4 M 2 − 2 ( τ + 1 ) , 1 ≤ m ≤ τ , c m = 2 τ ( M 2 − m 2 ) , τ + 1 ≤ m ≤ 2 τ . (198) c_m=\frac{2}{\tau}(M^2-m^2),\qquad\tau+1\le m\le2\tau. \tag*{(198)} c m = τ 2 ( M 2 − m 2 ) , τ + 1 ≤ m ≤ 2 τ . (198) Moreover we have r τ ( 1 ) > 0 r_\tau(1)>0 r τ ( 1 ) > 0 and r τ ( − 1 ) > 0 r_\tau(-1)>0 r τ ( − 1 ) > 0 where
r τ ( 1 ) = 2 M ( M + 1 ) 3 , r τ ( − 1 ) = { 4 M ( 2 M − 1 ) , if τ is even , 4 M ( 2 M + 1 ) ( 1 + 1 / τ ) , if τ is odd . (199) r_\tau(1)=\frac{2M(M+1)}{3},\qquad
r_\tau(-1)=
\begin{cases}
4M(2M-1), & \text{if }\tau\text{ is even},\\
4M(2M+1)(1+1/\tau), & \text{if }\tau\text{ is odd}.
\end{cases} \tag*{(199)} r τ ( 1 ) = 3 2 M ( M + 1 ) , r τ ( − 1 ) = { 4 M ( 2 M − 1 ) , 4 M ( 2 M + 1 ) ( 1 + 1/ τ ) , if τ is even , if τ is odd . (199)
Proof. Since H τ ( − θ ) ‾ = H τ ( θ ) ‾ \overline{H_\tau(-\theta)}=\overline{H_\tau(\theta)} H τ ( − θ ) = H τ ( θ ) and A τ , 0 ( − θ ) ‾ = A τ , 0 ( θ ) ‾ \overline{A_{\tau,0}(-\theta)}=\overline{A_{\tau,0}(\theta)} A τ , 0 ( − θ ) = A τ , 0 ( θ ) , the real-valued function Φ τ , 0 \Phi_{\tau,0} Φ τ , 0 is even and 2 π 2\pi 2 π -periodic. Moreover, r τ ( cos θ ) r_\tau(\cos\theta) r τ ( cos θ ) is even and 2 π 2\pi 2 π -periodic. Hence, it suffices to establish (197) for 0 ≤ θ ≤ π 0\le\theta\le\pi 0 ≤ θ ≤ π .
Let z = e i θ z=e^{i\theta} z = e i θ with 0 < θ ≤ π 0<\theta\le\pi 0 < θ ≤ π . Taking ρ = 1 \rho=1 ρ = 1 within Lemma 5.9 , leads to
W = 1 − H τ = ( 1 − z − 1 ) E ( z − 1 ) , E ( z − 1 ) = 1 τ ∑ k = 0 τ − 1 ( τ − k ) z − k . (200) W=1-H_\tau=(1-z^{-1})E(z^{-1}),\qquad E(z^{-1})=\frac{1}{\tau}\sum_{k=0}^{\tau-1}(\tau-k)z^{-k}. \tag*{(200)} W = 1 − H τ = ( 1 − z − 1 ) E ( z − 1 ) , E ( z − 1 ) = τ 1 k = 0 ∑ τ − 1 ( τ − k ) z − k . (200)
where H τ H_\tau H τ is as in (191) . Put a = M + 1 2 a=M+\frac{1}{2} a = M + 2 1 and b = M − 1 2 b=M-\frac{1}{2} b = M − 2 1 . Then
A τ , 0 = − z − τ ( a z − b ) M 2 ( z − 1 ) 2 , A_{\tau,0}=-\frac{z^{-\tau}(az-b)}{M^2(z-1)^2}, A τ , 0 = − M 2 ( z − 1 ) 2 z − τ ( a z − b ) ,
and therefore
W A τ , 0 = − E ( z − 1 ) z − τ − 1 ( a z − b ) M 2 ( z − 1 ) . (201) WA_{\tau,0}=-\frac{E(z^{-1})z^{-\tau-1}(az-b)}{M^2(z-1)}. \tag*{(201)} W A τ , 0 = − M 2 ( z − 1 ) E ( z − 1 ) z − τ − 1 ( a z − b ) . (201)
Adding the conjugate expression, i.e. substituting z − 1 z^{-1} z − 1 for z z z , gives
W A τ , 0 + W A τ , 0 ‾ = 1 τ M 2 ∑ k = 0 τ − 1 ( τ − k ) a ( z n − z − ( n − 1 ) ) − b ( z n + 1 − z − n ) z − 1 , (202) WA_{\tau,0}+\overline{WA_{\tau,0}}=\frac{1}{\tau M^2}\sum_{k=0}^{\tau-1}(\tau-k)\frac{a(z^n-z^{-(n-1)})-b(z^{n+1}-z^{-n})}{z-1}, \tag*{(202)} W A τ , 0 + W A τ , 0 = τ M 2 1 k = 0 ∑ τ − 1 ( τ − k ) z − 1 a ( z n − z − ( n − 1 ) ) − b ( z n + 1 − z − n ) , (202)
where n = τ + 1 + k n=\tau+1+k n = τ + 1 + k . Since
z n − z − ( n − 1 ) z − 1 = ∑ r = − ( n − 1 ) n − 1 z r , z n + 1 − z − n z − 1 = ∑ r = − n n z r , (203) \frac{z^n-z^{-(n-1)}}{z-1}=\sum_{r=-(n-1)}^{n-1}z^r,\qquad
\frac{z^{n+1}-z^{-n}}{z-1}=\sum_{r=-n}^{n}z^r, \tag*{(203)} z − 1 z n − z − ( n − 1 ) = r = − ( n − 1 ) ∑ n − 1 z r , z − 1 z n + 1 − z − n = r = − n ∑ n z r , (203)
we obtain
W A τ , 0 + W A τ , 0 ‾ = 1 τ M 2 ∑ k = 0 τ − 1 ( τ − k ) ( ∑ ∣ r ∣ ≤ n − 1 z r − b ( z n + z − n ) ) . (204) WA_{\tau,0}+\overline{WA_{\tau,0}}=\frac{1}{\tau M^2}\sum_{k=0}^{\tau-1}(\tau-k)\left(\sum_{\lvert r\rvert\le n-1}z^r-b(z^n+z^{-n})\right). \tag*{(204)} W A τ , 0 + W A τ , 0 = τ M 2 1 k = 0 ∑ τ − 1 ( τ − k ) ∣ r ∣ ≤ n − 1 ∑ z r − b ( z n + z − n ) . (204)
On the other hand,
W + W ‾ = 2 − 1 τ ∑ j = 1 τ ( z j + z − j ) . (205) W+\overline{W}=2-\frac{1}{\tau}\sum_{j=1}^{\tau}(z^j+z^{-j}). \tag*{(205)} W + W = 2 − τ 1 j = 1 ∑ τ ( z j + z − j ) . (205)
We observe from (204) and (205) that both W + W ‾ W+\overline{W} W + W and W A τ , 0 + W A τ , 0 ‾ WA_{\tau,0}+\overline{WA_{\tau,0}} W A τ , 0 + W A τ , 0 are symmetric Laurent polynomials in z z z . Hence,
− 4 M 2 Φ τ , 0 = 4 M 2 [ ( W + W ‾ ) − ( W A τ , 0 + W A τ , 0 ‾ ) ] , -4M^2\Phi_{\tau,0}=4M^2\left[(W+\overline{W})-(WA_{\tau,0}+\overline{WA_{\tau,0}})\right], − 4 M 2 Φ τ , 0 = 4 M 2 [ ( W + W ) − ( W A τ , 0 + W A τ , 0 ) ] ,
is also a symmetric Laurent polynomial. We now collect the coefficients of each pair z j z^j z j and z − j z^{-j} z − j together with the constant term. The symmetry ensures that the coefficients of z j z^j z j and z − j z^{-j} z − j coincide, so the expression can be written in the cosine basis z j + z − j = 2 cos ( j θ ) z^j+z^{-j}=2\cos(j\theta) z j + z − j = 2 cos ( j θ ) . Substituting the coefficients obtained from (205) and (204) then yields the representation given in (198) .
It remains to establish the endpoint values in (199) , corresponding to θ = 0 \theta=0 θ = 0 and θ = π \theta=\pi θ = π . At θ = 0 \theta=0 θ = 0 , Lemma 5.11 gives Φ τ , 0 ( 0 ) = − ( τ + 1 ) / ( 3 M ) \Phi_{\tau,0}(0)=-(\tau+1)/(3M) Φ τ , 0 ( 0 ) = − ( τ + 1 ) / ( 3 M ) so that we get r τ ( 1 ) = − 4 M 2 Φ τ , 0 ( 0 ) = 2 M ( M + 1 ) 3 r_\tau(1)=-4M^2\Phi_{\tau,0}(0)=\frac{2M(M+1)}{3} r τ ( 1 ) = − 4 M 2 Φ τ , 0 ( 0 ) = 3 2 M ( M + 1 ) where we used M = 2 τ + 1 M=2\tau+1 M = 2 τ + 1 . It remains to establish the formula for r τ ( − 1 ) r_\tau(-1) r τ ( − 1 ) . Recall from (193) that
Φ τ , 0 ( θ ) = 2 Re [ ( 1 − H τ ( θ ) ) ( A τ , 0 ( θ ) − 1 ) ] . (206) \Phi_{\tau,0}(\theta)=2\operatorname{Re}\left[(1-H_\tau(\theta))(A_{\tau,0}(\theta)-1)\right]. \tag*{(206)} Φ τ , 0 ( θ ) = 2 Re [ ( 1 − H τ ( θ )) ( A τ , 0 ( θ ) − 1 ) ] . (206)
where H τ ( θ ) : = 1 τ ∑ j = 1 τ e − i j θ H_\tau(\theta):=\frac{1}{\tau}\sum_{j=1}^{\tau}e^{-ij\theta} H τ ( θ ) := τ 1 ∑ j = 1 τ e − ij θ and A τ , 0 ( θ ) A_{\tau,0}(\theta) A τ , 0 ( θ ) is as in (192) . At θ = π \theta=\pi θ = π , one has A τ , 0 ( π ) = ( − 1 ) τ / ( 2 M ) A_{\tau,0}(\pi)=(-1)^\tau/(2M) A τ , 0 ( π ) = ( − 1 ) τ / ( 2 M ) . There are two cases depending on whether τ \tau τ is even or odd: If τ \tau τ is even, then H τ ( π ) = 1 τ ∑ j = 1 τ ( − 1 ) j = 0 H_\tau(\pi)=\frac{1}{\tau}\sum_{j=1}^{\tau}(-1)^j=0 H τ ( π ) = τ 1 ∑ j = 1 τ ( − 1 ) j = 0 , so 1 − H τ ( π ) = 1 > 0 1-H_\tau(\pi)=1>0 1 − H τ ( π ) = 1 > 0 and A τ , 0 ( π ) − 1 = 1 / ( 2 M ) − 1 < 0 A_{\tau,0}(\pi)-1=1/(2M)-1<0 A τ , 0 ( π ) − 1 = 1/ ( 2 M ) − 1 < 0 . Thus, by (206) , Φ τ , 0 ( π ) = 1 / M − 2 < 0 \Phi_{\tau,0}(\pi)=1/M-2<0 Φ τ , 0 ( π ) = 1/ M − 2 < 0 , equivalently r τ ( − 1 ) = 4 M ( 2 M − 1 ) > 0 r_\tau(-1)=4M(2M-1)>0 r τ ( − 1 ) = 4 M ( 2 M − 1 ) > 0 . On the other hand, if τ \tau τ is odd, then H τ ( π ) = − 1 / τ H_\tau(\pi)=-1/\tau H τ ( π ) = − 1/ τ , so 1 − H τ ( π ) = 1 + 1 / τ > 0 1-H_\tau(\pi)=1+1/\tau>0 1 − H τ ( π ) = 1 + 1/ τ > 0 and A τ , 0 ( π ) − 1 = − 1 / ( 2 M ) − 1 < 0 A_{\tau,0}(\pi)-1=-1/(2M)-1<0 A τ , 0 ( π ) − 1 = − 1/ ( 2 M ) − 1 < 0 . Thus, again Φ τ , 0 ( π ) = − 2 ( 1 + 1 / τ ) ( 1 / ( 2 M ) + 1 ) < 0 \Phi_{\tau,0}(\pi)=-2(1+1/\tau)(1/(2M)+1)<0 Φ τ , 0 ( π ) = − 2 ( 1 + 1/ τ ) ( 1/ ( 2 M ) + 1 ) < 0 , equivalently r τ ( − 1 ) = ( 1 + 1 / τ ) 4 M ( 2 M + 1 ) > 0 r_\tau(-1)=(1+1/\tau)4M(2M+1)>0 r τ ( − 1 ) = ( 1 + 1/ τ ) 4 M ( 2 M + 1 ) > 0 . Using M = 2 τ + 1 M=2\tau+1 M = 2 τ + 1 , this proves the formula for r τ ( − 1 ) r_\tau(-1) r τ ( − 1 ) and we conclude.
In Figure 2 , we plot the function Φ τ , 0 \Phi_{\tau,0} Φ τ , 0 which arises in the q ↓ 0 q\downarrow0 q ↓ 0 limit for various values of τ \tau τ . We observe that this function stays strictly negative, and that it becomes more oscillatory as τ \tau τ increases. The fact that the oscillations increase is expected, because r τ r_\tau r τ , that represents Φ τ , 0 \Phi_{\tau,0} Φ τ , 0 up to a constant by Lemma 5.12 , is a degree-2 τ 2\tau 2 τ polynomial and its degree increases when τ \tau τ increases.
Figure 2. The function Φ τ , 0 ( θ ) \Phi_{\tau,0}(\theta) Φ τ , 0 ( θ ) becomes increasingly oscillatory as τ \tau τ grows while remaining strictly negative.
In the next proposition, we also prove rigorously that Φ τ , 0 < 0 \Phi_{\tau,0}<0 Φ τ , 0 < 0 for every θ \theta θ as suggested by Figure 2 . The main idea is to show that the even trigonometric polynomial r τ ( cos θ ) r_\tau(\cos\theta) r τ ( cos θ ) is strictly positive for every θ \theta θ , and then invoke Lemma 5.12 . By evenness and periodicity, it suffices to verify positivity for θ ∈ [ 0 , π ] \theta\in[0,\pi] θ ∈ [ 0 , π ] . Because these polynomials exhibit highly oscillatory behavior, we control their sign uniformly over all delays by splitting [ 0 , π ] [0,\pi] [ 0 , π ] into two regions: a region away from the origin, where a Dirichlet-kernel bound applies, and a region near the origin, where we use Taylor series approximations and exploit the cosine-series representation (198) . In particular, differentiation with respect to θ \theta θ introduces powers of the frequencies, so the Taylor coefficients are combinations of the moments ∑ m c m m 2 k \sum_m c_m m^{2k} ∑ m c m m 2 k . Since c m c_m c m is constant in m m m for 1 ≤ m ≤ τ 1\le m\le\tau 1 ≤ m ≤ τ and quadratic in m m m for τ + 1 ≤ m ≤ 2 τ \tau+1\le m\le2\tau τ + 1 ≤ m ≤ 2 τ , these moments reduce to power sums ∑ m = 1 τ m p \sum_{m=1}^{\tau}m^p ∑ m = 1 τ m p , with p ≤ 8 p\le8 p ≤ 8 in the estimates below. Faulhaber’s formulas then provide explicit expressions for these sums in terms of τ \tau τ which in return allows us to control the sign of r τ r_\tau r τ .
Proposition 5.13 (all-delay positivity of the limiting polynomial). For every integer τ ≥ 1 \tau\ge1 τ ≥ 1 ,
r τ ( x ) > 0 for − 1 ≤ x ≤ 1. (207) r_\tau(x)>0 \quad\text{for } -1\le x\le1. \tag*{(207)} r τ ( x ) > 0 for − 1 ≤ x ≤ 1. (207) Consequently, Φ τ , 0 ( θ ) < 0 \Phi_{\tau,0}(\theta)<0 Φ τ , 0 ( θ ) < 0 for all 0 ≤ θ ≤ 2 π 0\le\theta\le2\pi 0 ≤ θ ≤ 2 π .
Proof. The idea is to split [ 0 , π ] [0,\pi] [ 0 , π ] into an outer and an inner region around the origin and bound r τ r_\tau r τ below on each. In the outer region θ ≥ 3 / ( 2 τ ) \theta\ge3/(2\tau) θ ≥ 3/ ( 2 τ ) we control the cosine sum ∑ k = 1 n cos ( k θ ) \sum_{k=1}^{n}\cos(k\theta) ∑ k = 1 n cos ( k θ ) by the Dirichlet kernel and compare it to an explicit threshold T n T_n T n ; in the inner region we use sharp Taylor envelopes for 1 − cos ( θ ) 1-\cos(\theta) 1 − cos ( θ ) together with Faulhaber’s power-sum formulas. Put n = τ n=\tau n = τ . It is enough to prove positivity for x = cos θ x=\cos\theta x = cos θ , 0 ≤ θ ≤ π 0\le\theta\le\pi 0 ≤ θ ≤ π . From (198) , write
r n ( cos θ ) = c 0 − 2 a ∑ k = 1 n cos ( k θ ) + 2 ∑ ℓ = 1 n d ℓ cos ( ( n + ℓ ) θ ) , (208) r_n(\cos\theta)=c_0-2a\sum_{k=1}^{n}\cos(k\theta)+2\sum_{\ell=1}^{n}d_\ell\cos((n+\ell)\theta), \tag*{(208)} r n ( cos θ ) = c 0 − 2 a k = 1 ∑ n cos ( k θ ) + 2 ℓ = 1 ∑ n d ℓ cos (( n + ℓ ) θ ) , (208)
where
c 0 = 32 n 2 + 30 n + 6 , a = 18 n 2 + 18 n + 4 n , d ℓ = 6 n 2 + 8 n + 2 − 4 n ℓ − 2 ℓ 2 n . (209) c_0=32n^2+30n+6,\qquad a=\frac{18n^2+18n+4}{n},\qquad d_\ell=\frac{6n^2+8n+2-4n\ell-2\ell^2}{n}. \tag*{(209)} c 0 = 32 n 2 + 30 n + 6 , a = n 18 n 2 + 18 n + 4 , d ℓ = n 6 n 2 + 8 n + 2 − 4 n ℓ − 2 ℓ 2 . (209)
The sequence d ℓ d_\ell d ℓ is positive: it is decreasing in ℓ \ell ℓ , and d n = 8 + 2 / n > 0 d_n=8+2/n>0 d n = 8 + 2/ n > 0 . Also
D n : = ∑ ℓ = 1 n d ℓ = 5 ( n + 1 ) ( 2 n + 1 ) 3 , T n : = c 0 − 2 D n 2 a = n ( 19 n 2 + 15 n + 2 ) 3 ( 3 n + 1 ) ( 3 n + 2 ) . (210) D_n:=\sum_{\ell=1}^{n}d_\ell=\frac{5(n+1)(2n+1)}{3},\qquad T_n:=\frac{c_0-2D_n}{2a}=\frac{n(19n^2+15n+2)}{3(3n+1)(3n+2)}. \tag*{(210)} D n := ℓ = 1 ∑ n d ℓ = 3 5 ( n + 1 ) ( 2 n + 1 ) , T n := 2 a c 0 − 2 D n = 3 ( 3 n + 1 ) ( 3 n + 2 ) n ( 19 n 2 + 15 n + 2 ) . (210)
We split the interval into two regions.
Outer region: θ ≥ 3 / ( 2 n ) \theta\ge3/(2n) θ ≥ 3/ ( 2 n ) . Let C n ( θ ) = ∑ k = 1 n cos ( k θ ) C_n(\theta)=\sum_{k=1}^{n}\cos(k\theta) C n ( θ ) = ∑ k = 1 n cos ( k θ ) . The Dirichlet formula gives
C n ( θ ) = sin ( ( n + 1 2 ) θ ) 2 sin ( θ / 2 ) − 1 2 ≤ 1 2 sin ( θ / 2 ) − 1 2 . (211) C_n(\theta)=\frac{\sin((n+\frac{1}{2})\theta)}{2\sin(\theta/2)}-\frac{1}{2}\le\frac{1}{2\sin(\theta/2)}-\frac{1}{2}. \tag*{(211)} C n ( θ ) = 2 sin ( θ /2 ) sin (( n + 2 1 ) θ ) − 2 1 ≤ 2 sin ( θ /2 ) 1 − 2 1 . (211)
Since
3 4 n ≤ θ 2 ≤ π 2 , \frac{3}{4n}\le\frac{\theta}{2}\le\frac{\pi}{2}, 4 n 3 ≤ 2 θ ≤ 2 π ,
and sin t \sin t sin t is increasing on [ 0 , π / 2 ] [0,\pi/2] [ 0 , π /2 ] , we have
sin ( θ 2 ) ≥ sin ( 3 4 n ) ≥ 3 4 n − 1 6 ( 3 4 n ) 3 = 3 ( 32 n 2 − 3 ) 128 n 3 . (212) \sin\left(\frac{\theta}{2}\right)\ge\sin\left(\frac{3}{4n}\right)\ge\frac{3}{4n}-\frac{1}{6}\left(\frac{3}{4n}\right)^3=\frac{3(32n^2-3)}{128n^3}. \tag*{(212)} sin ( 2 θ ) ≥ sin ( 4 n 3 ) ≥ 4 n 3 − 6 1 ( 4 n 3 ) 3 = 128 n 3 3 ( 32 n 2 − 3 ) . (212)
where we used sin t ≥ t − t 3 / 6 \sin t \ge t - t^3/6 sin t ≥ t − t 3 /6 for t ≥ 0 t \ge0 t ≥ 0 . Therefore,
C n ( θ ) ≤ C ‾ n ( θ ) : = 64 n 3 3 ( 32 n 2 − 3 ) − 1 2 . (213) C_n(\theta) \le\overline{C}_n(\theta) := \frac{64n^3}{3(32n^2-3)}-\frac{1}{2}. \tag*{(213)} C n ( θ ) ≤ C n ( θ ) := 3 ( 32 n 2 − 3 ) 64 n 3 − 2 1 . (213)
This upper bound is strictly smaller than T n T_n T n . Indeed,
Δ ‾ n : = T n − C ‾ n ( θ ) = 64 n 5 + 672 n 4 + 622 n 3 + 21 n 2 − 93 n − 18 6 ( 3 n + 1 ) ( 3 n + 2 ) ( 32 n 2 − 3 ) (214) \overline{\Delta}_n := T_n-\overline{C}_n(\theta)=\frac{64n^5+672n^4+622n^3+21n^2-93n-18}{6(3n+1)(3n+2)(32n^2-3)} \tag*{(214)} Δ n := T n − C n ( θ ) = 6 ( 3 n + 1 ) ( 3 n + 2 ) ( 32 n 2 − 3 ) 64 n 5 + 672 n 4 + 622 n 3 + 21 n 2 − 93 n − 18 (214)
= 64 m 5 + 992 m 4 + 3950 m 3 + 6559 m 2 + 4823 m + 1268 6 ( 3 n + 1 ) ( 3 n + 2 ) ( 32 n 2 − 3 ) > 0 , (215) =\frac{64m^5+992m^4+3950m^3+6559m^2+4823m+1268}{6(3n+1)(3n+2)(32n^2-3)}>0, \tag*{(215)} = 6 ( 3 n + 1 ) ( 3 n + 2 ) ( 32 n 2 − 3 ) 64 m 5 + 992 m 4 + 3950 m 3 + 6559 m 2 + 4823 m + 1268 > 0 , (215)
where m = n − 1 ≥ 0 m=n-1\ge0 m = n − 1 ≥ 0 . Therefore, using d ℓ > 0 d_\ell>0 d ℓ > 0 and cos ( ( n + ℓ ) θ ) ≥ − 1 \cos((n+\ell)\theta)\ge-1 cos (( n + ℓ ) θ ) ≥ − 1 ,
r n ( cos θ ) ≥ c 0 − 2 a C ‾ n ( θ ) − 2 D n ≥ 2 a Δ ‾ n > c 0 − 2 a T n − 2 D n = 0. (216) r_n(\cos\theta)\ge c_0-2a\overline{C}_n(\theta)-2D_n\ge2a\overline{\Delta}_n>c_0-2aT_n-2D_n=0. \tag*{(216)} r n ( cos θ ) ≥ c 0 − 2 a C n ( θ ) − 2 D n ≥ 2 a Δ n > c 0 − 2 a T n − 2 D n = 0. (216)
Inner region: 0 ≤ θ ≤ 3 / ( 2 n ) 0\le\theta\le3/(2n) 0 ≤ θ ≤ 3/ ( 2 n ) . We use two elementary Taylor inequalities, valid for all u ≥ 0 u\ge0 u ≥ 0 :
1 − cos u ≥ u 2 2 − u 4 24 , cos u − 1 ≥ − u 2 2 + u 4 24 − u 6 720 . (217) 1-\cos u\ge\frac{u^2}{2}-\frac{u^4}{24},\qquad\cos u-1\ge-\frac{u^2}{2}+\frac{u^4}{24}-\frac{u^6}{720}. \tag*{(217)} 1 − cos u ≥ 2 u 2 − 24 u 4 , cos u − 1 ≥ − 2 u 2 + 24 u 4 − 720 u 6 . (217)
For completeness we recall the proof. The difference of both sides in the first inequality e 1 ( u ) : = 1 − cos u − ( u 2 / 2 − u 4 / 24 ) e_1(u):=1-\cos u-(u^2/2-u^4/24) e 1 ( u ) := 1 − cos u − ( u 2 /2 − u 4 /24 ) has fourth derivative 1 − cos u ≥ 0 1-\cos u\ge0 1 − cos u ≥ 0 and its derivatives of orders 0 , … , 3 0,\ldots,3 0 , … , 3 vanish at zero. Similarly, e 2 ( u ) : = cos u − 1 − ( − u 2 / 2 + u 4 / 24 − u 6 / 720 ) e_2(u):=\cos u-1-(-u^2/2+u^4/24-u^6/720) e 2 ( u ) := cos u − 1 − ( − u 2 /2 + u 4 /24 − u 6 /720 ) has sixth derivative 1 − cos u ≥ 0 1-\cos u\ge0 1 − cos u ≥ 0 and its derivatives of orders 0 , … , 5 0,\ldots,5 0 , … , 5 vanish at zero. Repeated integration gives (217) .
Using (217) in
r n ( cos θ ) = r n ( 1 ) + 2 a ∑ k = 1 n ( 1 − cos ( k θ ) ) + 2 ∑ ℓ = 1 n d ℓ ( cos ( ( n + ℓ ) θ ) − 1 ) r_n(\cos\theta)=r_n(1)+2a\sum_{k=1}^{n}(1-\cos(k\theta))+2\sum_{\ell=1}^{n}d_\ell(\cos((n+\ell)\theta)-1) r n ( cos θ ) = r n ( 1 ) + 2 a k = 1 ∑ n ( 1 − cos ( k θ )) + 2 ℓ = 1 ∑ n d ℓ ( cos (( n + ℓ ) θ ) − 1 )
gives
r n ( cos θ ) ≥ r n ( 1 ) + R 2 θ 2 + R 4 θ 4 − R 6 θ 6 , (218) r_n(\cos\theta)\ge r_n(1)+R_2\theta^2+R_4\theta^4-R_6\theta^6, \tag*{(218)} r n ( cos θ ) ≥ r n ( 1 ) + R 2 θ 2 + R 4 θ 4 − R 6 θ 6 , (218)
where the coefficients before simplification are
R 2 = a ∑ k = 1 n k 2 − ∑ ℓ = 1 n d ℓ ( n + ℓ ) 2 , (219) R_2=a\sum_{k=1}^{n}k^2-\sum_{\ell=1}^{n}d_\ell(n+\ell)^2, \tag*{(219)} R 2 = a k = 1 ∑ n k 2 − ℓ = 1 ∑ n d ℓ ( n + ℓ ) 2 , (219)
R 4 = − a 12 ∑ k = 1 n k 4 + 1 12 ∑ ℓ = 1 n d ℓ ( n + ℓ ) 4 , (220) R_4=-\frac{a}{12}\sum_{k=1}^{n}k^4+\frac{1}{12}\sum_{\ell=1}^{n}d_\ell(n+\ell)^4, \tag*{(220)} R 4 = − 12 a k = 1 ∑ n k 4 + 12 1 ℓ = 1 ∑ n d ℓ ( n + ℓ ) 4 , (220)
R 6 = 1 360 ∑ ℓ = 1 n d ℓ ( n + ℓ ) 6 . (221) R_6=\frac{1}{360}\sum_{\ell=1}^{n}d_\ell(n+\ell)^6. \tag*{(221)} R 6 = 360 1 ℓ = 1 ∑ n d ℓ ( n + ℓ ) 6 . (221)
Since d ℓ d_\ell d ℓ is quadratic in ℓ \ell ℓ , the last identity uses power sums up to degree eight. Applying Faulhaber’s formula :
∑ j = 1 n j p = 1 p + 1 ∑ r = 0 p ( p + 1 r ) B r + n p + 1 − r , 0 ≤ p ≤ 8 , \sum_{j=1}^{n}j^p=\frac{1}{p+1}\sum_{r=0}^{p}\binom{p+1}{r}B_r^+n^{p+1-r},\qquad0\le p\le8, j = 1 ∑ n j p = p + 1 1 r = 0 ∑ p ( r p + 1 ) B r + n p + 1 − r , 0 ≤ p ≤ 8 ,
in the Bernoulli convention determined by
t 1 − e − t = ∑ r = 0 ∞ B r + t r r ! , \frac{t}{1-e^{-t}}=\sum_{r=0}^{\infty}B_r^+\frac{t^r}{r!}, 1 − e − t t = r = 0 ∑ ∞ B r + r ! t r ,
so that B 1 + = 1 / 2 B_1^+ = 1/2 B 1 + = 1/2 [19 Sec. 6.5, Eq. (6.78)] , gives
r n ( 1 ) = 4 ( n + 1 ) ( 2 n + 1 ) 3 , (222) r_n(1) = \frac{4(n+1)(2n+1)}{3}, \tag*{(222)} r n ( 1 ) = 3 4 ( n + 1 ) ( 2 n + 1 ) , (222)
R 2 = − 2 ( n − 1 ) ( n + 1 ) ( n + 2 ) ( 2 n + 1 ) 15 , (223) R_2 = -\frac{2(n-1)(n+1)(n+2)(2n+1)}{15}, \tag*{(223)} R 2 = − 15 2 ( n − 1 ) ( n + 1 ) ( n + 2 ) ( 2 n + 1 ) , (223)
R 4 = ( n + 1 ) ( 2 n + 1 ) ( 510 n 4 + 1020 n 3 + 539 n 2 + 29 n + 2 ) 1260 , (224) R_4 = \frac{(n+1)(2n+1)(510n^4+1020n^3+539n^2+29n+2)}{1260}, \tag*{(224)} R 4 = 1260 ( n + 1 ) ( 2 n + 1 ) ( 510 n 4 + 1020 n 3 + 539 n 2 + 29 n + 2 ) , (224)
R 6 = ( n + 1 ) ( 2 n + 1 ) ( 4975 n 6 + 14925 n 5 + 13390 n 4 + 1905 n 3 − 1583 n 2 − 48 n + 36 ) 113400 . (225) R_6 = \frac{(n+1)(2n+1)(4975n^6+14925n^5+13390n^4+1905n^3-1583n^2-48n+36)}{113400}. \tag*{(225)} R 6 = 113400 ( n + 1 ) ( 2 n + 1 ) ( 4975 n 6 + 14925 n 5 + 13390 n 4 + 1905 n 3 − 1583 n 2 − 48 n + 36 ) . (225)
These identities are ordinary polynomial identities in n n n : substituting (209) into (219) –(221) , replacing ∑ ℓ = 1 n ℓ j \sum_{\ell=1}^{n}\ell^j ∑ ℓ = 1 n ℓ j by Faulhaber’s formula for j ≤ 8 j \le8 j ≤ 8 , and clearing the positive common denominator reproduces the four displayed numerators. Equivalently, a purely finite check is obtained by clearing denominators and comparing coefficients of the resulting polynomials in n n n . Thus, no inequality is being used in (222) –(225) ; the inequalities enter only in the sign checks that follow. The coefficient R 2 R_2 R 2 is nonpositive. The quartic and sixth-order terms satisfy, for all n ≥ 1 n \ge1 n ≥ 1 ,
R 4 − 9 4 n 2 R 6 = ( n + 1 ) ( 2 n + 1 ) 50400 n 2 ( 15425 n 6 + 25875 n 5 + 8170 n 4 (226) R_4-\frac{9}{4n^2}R_6=\frac{(n+1)(2n+1)}{50400n^2}(15425n^6+25875n^5+8170n^4 \tag*{(226)} R 4 − 4 n 2 9 R 6 = 50400 n 2 ( n + 1 ) ( 2 n + 1 ) ( 15425 n 6 + 25875 n 5 + 8170 n 4 (226)
− 745 n 3 + 1663 n 2 + 48 n − 36 ) > 0. (227) -745n^3+1663n^2+48n-36)>0. \tag*{(227)} − 745 n 3 + 1663 n 2 + 48 n − 36 ) > 0. (227)
The positivity is immediate after substituting n = m + 1 n=m+1 n = m + 1 ; the polynomial in parentheses becomes
15425 m 6 + 118425 m 5 + 368920 m 4 + 599185 m 3 + 538573 m 2 + 255744 m + 50400. 15425m^6+118425m^5+368920m^4+599185m^3+538573m^2+255744m+50400. 15425 m 6 + 118425 m 5 + 368920 m 4 + 599185 m 3 + 538573 m 2 + 255744 m + 50400.
Moreover, R 6 > 0 R_6>0 R 6 > 0 by (221) , since d ℓ > 0 d_\ell>0 d ℓ > 0 for 1 ≤ ℓ ≤ n 1\le\ell\le n 1 ≤ ℓ ≤ n . Since θ 2 ≤ 9 / ( 4 n 2 ) \theta^2\le9/(4n^2) θ 2 ≤ 9/ ( 4 n 2 ) in the inner region, (226) yields
R 4 − R 6 θ 2 ≥ R 4 − 9 4 n 2 R 6 > 0. R_4-R_6\theta^2\ge R_4-\frac{9}{4n^2}R_6>0. R 4 − R 6 θ 2 ≥ R 4 − 4 n 2 9 R 6 > 0.
Consequently,
R 4 θ 4 − R 6 θ 6 = θ 4 ( R 4 − R 6 θ 2 ) ≥ 0. R_4\theta^4-R_6\theta^6=\theta^4(R_4-R_6\theta^2)\ge0. R 4 θ 4 − R 6 θ 6 = θ 4 ( R 4 − R 6 θ 2 ) ≥ 0.
Furthermore, since R 2 ≤ 0 R_2\le0 R 2 ≤ 0 and θ 2 ≤ 9 / ( 4 n 2 ) \theta^2\le9/(4n^2) θ 2 ≤ 9/ ( 4 n 2 ) ,
R 2 θ 2 ≥ 9 4 n 2 R 2 . R_2\theta^2\ge\frac{9}{4n^2}R_2. R 2 θ 2 ≥ 4 n 2 9 R 2 .
Hence,
r n ( cos θ ) ≥ r n ( 1 ) + R 2 θ 2 ≥ r n ( 1 ) + 9 4 n 2 R 2 (228) r_n(\cos\theta)\ge r_n(1)+R_2\theta^2\ge r_n(1)+\frac{9}{4n^2}R_2 \tag*{(228)} r n ( cos θ ) ≥ r n ( 1 ) + R 2 θ 2 ≥ r n ( 1 ) + 4 n 2 9 R 2 (228)
= ( n + 1 ) ( 2 n + 1 ) ( 31 n 2 − 9 n + 18 ) 30 n 2 > 0. (229) =\frac{(n+1)(2n+1)(31n^2-9n+18)}{30n^2}>0. \tag*{(229)} = 30 n 2 ( n + 1 ) ( 2 n + 1 ) ( 31 n 2 − 9 n + 18 ) > 0. (229)
The inner and outer regions cover [ 0 , π ] [0,\pi] [ 0 , π ] . Thus r τ > 0 r_\tau>0 r τ > 0 on [ − 1 , 1 ] [-1,1] [ − 1 , 1 ] . The final assertion follows from (197) .
The previous result establishes the limiting FDI, Φ τ , 0 < 0 \Phi_{\tau,0}<0 Φ τ , 0 < 0 , obtained in the limit q ↓ 0 q\downarrow0 q ↓ 0 . In the following corollary, we first show that this strict negativity persists on each fixed frequency interval bounded away from zero when q > 0 q>0 q > 0 is sufficiently small. We then treat the neighborhood of zero separately using a joint expansion in q q q and θ \theta θ .
Corollary 5.14 (outer limiting negativity). For every θ ⋆ ∈ ( 0 , π ] \theta_\star\in(0,\pi] θ ⋆ ∈ ( 0 , π ] , there exist constants m 1 > 0 m_1>0 m 1 > 0 and q 1 > 0 q_1>0 q 1 > 0 , depending on ( τ , c , θ ⋆ ) (\tau,c,\theta_\star) ( τ , c , θ ⋆ ) , such that
N τ , q ( θ ) ≤ − m 1 < 0 , θ ∈ [ θ ⋆ , π ] , 0 < q ≤ q 1 . (230) N_{\tau,q}(\theta)\le-m_1<0,\qquad\theta\in[\theta_\star,\pi],\qquad0<q\le q_1. \tag*{(230)} N τ , q ( θ ) ≤ − m 1 < 0 , θ ∈ [ θ ⋆ , π ] , 0 < q ≤ q 1 . (230)
Proof. By Proposition 5.13 and (197) , Φ τ , 0 ( θ ) < 0 \Phi_{\tau,0}(\theta)<0 Φ τ , 0 ( θ ) < 0 on [ 0 , π ] [0,\pi] [ 0 , π ] . For fixed θ ⋆ > 0 \theta_\star>0 θ ⋆ > 0 , the denominators in the rational representation of Φ τ , q \Phi_{\tau,q} Φ τ , q are uniformly bounded away from zero on [ θ ⋆ , π ] [\theta_\star,\pi] [ θ ⋆ , π ] for all sufficiently small q q q . Consequently, Φ τ , q → Φ τ , 0 \Phi_{\tau,q}\to\Phi_{\tau,0} Φ τ , q → Φ τ , 0 uniformly on [ θ ⋆ , π ] [\theta_\star,\pi] [ θ ⋆ , π ] as q ↓ 0 q\downarrow0 q ↓ 0 . Since Φ τ , 0 \Phi_{\tau,0} Φ τ , 0 is strictly negative on this compact interval, there exist η > 0 \eta>0 η > 0 and q 1 > 0 q_1>0 q 1 > 0 such that
Φ τ , q ( θ ) ≤ − η , θ ∈ [ θ ⋆ , π ] , 0 < q ≤ q 1 . \Phi_{\tau,q}(\theta)\le-\eta,\qquad\theta\in[\theta_\star,\pi],\qquad0<q\le q_1. Φ τ , q ( θ ) ≤ − η , θ ∈ [ θ ⋆ , π ] , 0 < q ≤ q 1 .
The denominator in (185) is positive and uniformly bounded below on the same interval. Multiplying by this denominator therefore yields (230) , after possibly decreasing q 1 q_1 q 1 . □ \square □
Corollary 5.14 establishes the required negativity uniformly on every frequency interval bounded away from the origin. It therefore remains to control the low-frequency regime, which is singular because the limiting denominator vanishes at θ = 0 \theta=0 θ = 0 . This requires a joint analysis in q q q and θ \theta θ .
For 0 ≤ θ ≤ 1 / 2 0\le\theta\le1/2 0 ≤ θ ≤ 1/2 , a Taylor expansion of N τ , q N_{\tau,q} N τ , q at θ = 0 \theta=0 θ = 0 yields
N τ , q ( θ ) = a 0 ( q ) + a 2 ( q ) θ 2 + a 4 ( q ) θ 4 + R 6 ( θ , q ) , ∣ R 6 ( θ , q ) ∣ ≤ C 6 θ 6 , (231) N_{\tau,q}(\theta)=a_0(q)+a_2(q)\theta^2+a_4(q)\theta^4+R_6(\theta,q),\qquad|R_6(\theta,q)|\le C_6\theta^6, \tag*{(231)} N τ , q ( θ ) = a 0 ( q ) + a 2 ( q ) θ 2 + a 4 ( q ) θ 4 + R 6 ( θ , q ) , ∣ R 6 ( θ , q ) ∣ ≤ C 6 θ 6 , (231)
where C 6 < ∞ C_6<\infty C 6 < ∞ can be chosen uniformly for all sufficiently small q ≥ 0 q\ge0 q ≥ 0 . For fixed τ \tau τ and c c c , the numerator in (185) –(186) is a finite trigonometric polynomial with coefficients analytic in q q q near zero, since ρ ( 0 ) = 1 \rho(0)=1 ρ ( 0 ) = 1 .9 Its sixth derivative with respect to θ \theta θ is therefore continuous and uniformly bounded for 0 ≤ θ ≤ 1 / 2 0\le\theta\le1/2 0 ≤ θ ≤ 1/2 and sufficiently small q ≥ 0 q\ge0 q ≥ 0 . This gives the stated remainder bound.
Direct, albeit lengthy, expansion of the same numerator in q q q and θ \theta θ gives the following leading coefficients. The intermediate algebra is lengthy and is therefore omitted here for brevity.
a 0 ( q ) = − B q 3 + O ( q 4 ) , a 2 ( q ) = − λ q + O ( q 2 ) , a 4 ( q ) = − A 0 + O ( q ) , (232) a_0(q)=-Bq^3+O(q^4),\qquad a_2(q)=-\lambda q+O(q^2),\qquad a_4(q)=-A_0+O(q), \tag*{(232)} a 0 ( q ) = − B q 3 + O ( q 4 ) , a 2 ( q ) = − λ q + O ( q 2 ) , a 4 ( q ) = − A 0 + O ( q ) , (232)
where
B = ( τ + 1 ) c ( 1 − c ) 2 M 3 , λ = ( τ + 1 ) ( 2 − 3 c ) M , A 0 = M ( τ + 1 ) 3 . (233) B=\frac{(\tau+1)c(1-c)^2}{M^3},\qquad\lambda=\frac{(\tau+1)(2-3c)}{M},\qquad A_0=\frac{M(\tau+1)}{3}. \tag*{(233)} B = M 3 ( τ + 1 ) c ( 1 − c ) 2 , λ = M ( τ + 1 ) ( 2 − 3 c ) , A 0 = 3 M ( τ + 1 ) . (233)
In particular, at q = 0 q=0 q = 0 , (185) and Lemma 5.11 give a 4 ( 0 ) = M 2 Φ τ , 0 ( 0 ) = − M ( τ + 1 ) / 3 a_4(0)=M^2\Phi_{\tau,0}(0)=-M(\tau+1)/3 a 4 ( 0 ) = M 2 Φ τ , 0 ( 0 ) = − M ( τ + 1 ) /3 .
Since c ∈ ( 0 , 2 / 3 ) c\in(0,2/3) c ∈ ( 0 , 2/3 ) , all three constants are strictly positive. Hence, for some q ∗ > 0 q_\ast>0 q ∗ > 0 ,
a 0 ( q ) ≤ − B 2 q 3 , a 2 ( q ) ≤ − λ 2 q , a 4 ( q ) ≤ − A 0 2 , 0 < q ≤ q ∗ . a_0(q)\le-\frac{B}{2}q^3,\qquad a_2(q)\le-\frac{\lambda}{2}q,\qquad a_4(q)\le-\frac{A_0}{2},\qquad0<q\le q_\ast. a 0 ( q ) ≤ − 2 B q 3 , a 2 ( q ) ≤ − 2 λ q , a 4 ( q ) ≤ − 2 A 0 , 0 < q ≤ q ∗ .
Choose 0 < θ ⋆ ≤ 1 / 2 0<\theta_\star\le1/2 0 < θ ⋆ ≤ 1/2 such that
C 6 θ ⋆ 2 ≤ A 0 4 . C_6\theta_\star^2\le\frac{A_0}{4}. C 6 θ ⋆ 2 ≤ 4 A 0 .
Then, for 0 ≤ θ ≤ θ ⋆ 0\le\theta\le\theta_\star 0 ≤ θ ≤ θ ⋆ and 0 < q ≤ q ∗ 0<q\le q_\ast 0 < q ≤ q ∗ ,
N τ , q ( θ ) ≤ − B 2 q 3 − λ 2 q θ 2 − A 0 4 θ 4 < 0. (234) N_{\tau,q}(\theta)\le-\frac{B}{2}q^3-\frac{\lambda}{2}q\theta^2-\frac{A_0}{4}\theta^4<0. \tag*{(234)} N τ , q ( θ ) ≤ − 2 B q 3 − 2 λ q θ 2 − 4 A 0 θ 4 < 0. (234)
Thus N τ , q N_{\tau,q} N τ , q is strictly negative in a neighborhood of the origin. On the complementary interval θ ∈ [ θ ⋆ , π ] \theta\in[\theta_\star,\pi] θ ∈ [ θ ⋆ , π ] , Corollary 5.14 yields strict negativity for all sufficiently small q q q . Taking the minimum of the corresponding admissible q q q -bounds therefore gives q f d i ( τ , c ) > 0 q_{\mathrm{fdi}}(\tau,c)>0 q fdi ( τ , c ) > 0 such that
Φ τ , q ( θ ) < 0 , θ ∈ [ 0 , 2 π ) , 0 < q ≤ q f d i ( τ , c ) . (235) \Phi_{\tau,q}(\theta)<0,\qquad\theta\in[0,2\pi),\qquad0<q\le q_{\mathrm{fdi}}(\tau,c). \tag*{(235)} Φ τ , q ( θ ) < 0 , θ ∈ [ 0 , 2 π ) , 0 < q ≤ q fdi ( τ , c ) . (235)
Hence hypothesis (H2) holds for all sufficiently small q > 0 q>0 q > 0 .
Iteration Complexity with Explicit Constants
While Theorem 4.1 establishes accelerated rates, it does not provide explicit values for the prefactor C C C in the error bounds or for the threshold κ 0 \kappa_0 κ 0 . Consequently, it does not yield a fully explicit characterization of the iteration complexity. The following result shows that κ 0 \kappa_0 κ 0 depends polynomially on τ \tau τ , while C C C depends polynomially on τ \tau τ and κ \kappa κ , and provides explicit admissible formulas for both.
Theorem 6.1 (Polynomial bounds on the threshold and prefactor). In the setting of Theorem 4.1 , the threshold κ 0 ( τ , c ) \kappa_0(\tau,c) κ 0 ( τ , c ) can be chosen to depend polynomially on M = 2 τ + 1 M=2\tau+1 M = 2 τ + 1 , while the prefactor C C C depends polynomially on M M M and κ \kappa κ . More precisely, under the constant-history initialization
x − τ − 1 = ⋯ = x 0 = x i n i t , x_{-\tau-1}=\cdots=x_0=x_{\mathrm{init}}, x − τ − 1 = ⋯ = x 0 = x init , one may choose
κ 0 ( τ , c ) = O c ( M 8 ) , C = O c ( M 5 κ 2 ) ∥ x i n i t − x ⋆ ∥ . (236) \kappa_0(\tau,c)=\mathcal{O}_c(M^8),\qquad C=\mathcal{O}_c(M^5\kappa^2)\lVert x_{\mathrm{init}}-x_\star\rVert. \tag*{(236)} κ 0 ( τ , c ) = O c ( M 8 ) , C = O c ( M 5 κ 2 ) ∥ x init − x ⋆ ∥ . (236)
Proof. The result follows by keeping quantitative track of the constants in the asymptotic analysis underlying Theorem 4.1 . We give the main estimates and indicate how they enter the argument; the omitted calculations consist of explicit, but lengthy, bounds on the coefficients appearing in the stability and frequency-domain estimates.
Work in the normalization (137) –(138) , translate the minimizer to the origin, and write
q = κ − 1 / 2 , M = 2 τ + 1 , ρ = 1 − c q M . q=\kappa^{-1/2},\qquad M=2\tau+1,\qquad\rho=1-\frac{cq}{M}. q = κ − 1/2 , M = 2 τ + 1 , ρ = 1 − M c q .
We first quantify the two hypotheses (H1 ) and (H2 ) used in the proof of Theorem 4.1 .
For the midpoint-stability condition, a quantitative perturbation argument for the roots of the midpoint polynomial P τ , q P_{\tau,q} P τ , q shows that, whenever 0 < q ≤ q m i d ( τ , c ) 0<q\le q_{\mathrm{mid}}(\tau,c) 0 < q ≤ q mid ( τ , c ) , where
q m i d ( τ , c ) : = min { q ˉ , 1 80 M 2 } , q ˉ : = M 2 M + 1 , q_{\mathrm{mid}}(\tau,c):=\min\left\{\bar q,\frac{1}{80M^2}\right\},\qquad\bar q:=\frac{M}{2M+1}, q mid ( τ , c ) := min { q ˉ , 80 M 2 1 } , q ˉ := 2 M + 1 M ,
every root of P τ , q P_{\tau,q} P τ , q lies in the disk { z ∈ C : ∣ z ∣ < ρ } \{z\in\mathbb{C}:|z|<\rho\} { z ∈ C : ∣ z ∣ < ρ } . Thus (H1 ) holds. In particular, q m i d = Θ ( M − 2 ) q_{\mathrm{mid}}=\Theta(M^{-2}) q mid = Θ ( M − 2 ) .
We next quantify the frequency-domain condition. After some lengthy but trivial computations, one can show that (235) holds for q f d i = Θ c ( M − 4 ) q_{\mathrm{fdi}}=\Theta_c(M^{-4}) q fdi = Θ c ( M − 4 ) , such that, for every 0 < q ≤ q f d i 0<q\le q_{\mathrm{fdi}} 0 < q ≤ q fdi ,
− Φ τ , q ( θ ) ≥ η c Φ , 0 ≤ θ ≤ π , -\Phi_{\tau,q}(\theta)\ge\eta_c^\Phi,\qquad0\le\theta\le\pi, − Φ τ , q ( θ ) ≥ η c Φ , 0 ≤ θ ≤ π ,
where
η c Φ : = min { c ( 1 − c ) 2 72 , 1 54 π 4 } > 0. \eta_c^\Phi:=\min\left\{\frac{c(1-c)^2}{72},\frac{1}{54\pi^4}\right\}>0. η c Φ := min { 72 c ( 1 − c ) 2 , 54 π 4 1 } > 0.
Hence (H2 ) holds uniformly over the full frequency range.
Consequently, with
q 0 ( τ , c ) : = min { q m i d ( τ , c ) , q f d i ( τ , c ) } , q_0(\tau,c):=\min\{q_{\mathrm{mid}}(\tau,c),q_{\mathrm{fdi}}(\tau,c)\}, q 0 ( τ , c ) := min { q mid ( τ , c ) , q fdi ( τ , c )} ,
both (H1 ) and (H2 ) hold whenever 0 < q ≤ q 0 ( τ , c ) 0<q\le q_0(\tau,c) 0 < q ≤ q 0 ( τ , c ) . Since the frequency-domain restriction is the more stringent one asymptotically,
q 0 ( τ , c ) = Θ c ( M − 4 ) . q_0(\tau,c)=\Theta_c(M^{-4}). q 0 ( τ , c ) = Θ c ( M − 4 ) .
To control the transient prefactor, recall the normalization (137) , under which the strong-convexity and smoothness constants become 1 1 1 and κ = L / μ \kappa=L/\mu κ = L / μ , respectively. Define m q : = 1 + κ 2 m_q:=\frac{1+\kappa}{2} m q := 2 1 + κ , ω q : = κ − 1 2 \omega_q:=\frac{\kappa-1}{2} ω q := 2 κ − 1 , the midpoint and half-width of the interval [ 1 , κ ] [1,\kappa] [ 1 , κ ] . We separate the normalized gradient into a linear component with slope m q m_q m q and a remainder by setting v ^ k : = u ^ k − m q y ^ k \widehat{v}_k := \widehat{u}_k - m_q\widehat{y}_k v k := u k − m q y k . Thus, v ^ k \widehat{v}_k v k measures the deviation of the normalized gradient from its midpoint-slope linear approximation. Smoothness and strong convexity imply ∥ v ^ k ∥ ≤ ω q ∥ y ^ k ∥ \|\widehat{v}_k\| \leq\omega_q\|\widehat{y}_k\| ∥ v k ∥ ≤ ω q ∥ y k ∥ . We therefore introduce the rescaled remainder
w ^ k : = v ^ k ω q , ∥ w ^ k ∥ ≤ ∥ y ^ k ∥ . \widehat{w}_k := \frac{\widehat{v}_k}{\omega_q}, \qquad\|\widehat{w}_k\| \leq\|\widehat{y}_k\|. w k := ω q v k , ∥ w k ∥ ≤ ∥ y k ∥.
This rescaling allows us to work with a bound on the nonlinear remainder that is independent of κ \kappa κ , while retaining the condition-number dependence explicitly in the linear relation u ^ k = m q y ^ k + ω q w ^ k \widehat{u}_k = m_q\widehat{y}_k + \omega_q\widehat{w}_k u k = m q y k + ω q w k . This separation facilitates sharper estimates of the transient prefactor.
The uniform FDI margin and midpoint-stability estimate then yield quantitative gain bounds for the rate-weighted recursion driven by w ^ \widehat{w} w . Tracking the factors of ω q \omega_q ω q when these normalized bounds are converted back to the original midpoint variables, and combining them with the constant-history bound on the initial augmented state, gives
∥ x k − x ⋆ ∥ ≤ C c l n o r m ρ k , ∀ k ≥ 0 , with C c l n o r m = O c ( M 5 κ 2 ∥ x i n i t − x ⋆ ∥ ) . \|x_k-x_\star\| \leq C_{\mathrm{cl}}^{\mathrm{norm}}\rho^k,\qquad\forall k\geq0,\quad\text{with}\quad C_{\mathrm{cl}}^{\mathrm{norm}}=\mathcal{O}_c(M^5\kappa^2\|x_{\mathrm{init}}-x_\star\|). ∥ x k − x ⋆ ∥ ≤ C cl norm ρ k , ∀ k ≥ 0 , with C cl norm = O c ( M 5 κ 2 ∥ x init − x ⋆ ∥ ) .
Finally, since κ = q − 2 \kappa=q^{-2} κ = q − 2 , the condition q ≤ q 0 ( τ , c ) q\leq q_0(\tau,c) q ≤ q 0 ( τ , c ) is equivalent to κ ≥ κ 0 ( τ , c ) : = ( q 0 ( τ , c ) ) − 2 \kappa\geq\kappa_0(\tau,c):=(q_0(\tau,c))^{-2} κ ≥ κ 0 ( τ , c ) := ( q 0 ( τ , c ) ) − 2 . Therefore, κ 0 ( τ , c ) = O c ( M 8 ) \kappa_0(\tau,c)=\mathcal{O}_c(M^8) κ 0 ( τ , c ) = O c ( M 8 ) . Undoing the normalization gives
α = q 2 M 2 μ = 1 M 2 L , β = 1 − 2 q M = 1 − 2 M κ , ρ = 1 − c M κ . (237) \alpha=\frac{q^2}{M^2\mu}=\frac{1}{M^2L},\qquad\beta=1-\frac{2q}{M}=1-\frac{2}{M\sqrt{\kappa}},\qquad\rho=1-\frac{c}{M\sqrt{\kappa}}. \tag*{(237)} α = M 2 μ q 2 = M 2 L 1 , β = 1 − M 2 q = 1 − M κ 2 , ρ = 1 − M κ c . (237)
Thus the claimed rate, threshold, and prefactor bounds follow. □
Remark 6.2 (iteration complexity and lower-bound comparison). Since ρ = 1 − c / ( M κ ) \rho=1-c/(M\sqrt{\kappa}) ρ = 1 − c / ( M κ ) and − log ( 1 − t ) ≥ t -\log(1-t)\geq t − log ( 1 − t ) ≥ t for 0 < t < 1 0<t<1 0 < t < 1 , we have
k ≥ M κ c log C c l n o r m ϵ ⟹ ∥ x k − x ⋆ ∥ ≤ ϵ . (238) k\geq\frac{M\sqrt{\kappa}}{c}\log\frac{C_{\mathrm{cl}}^{\mathrm{norm}}}{\epsilon}\quad\Longrightarrow\quad\|x_k-x_\star\|\leq\epsilon. \tag*{(238)} k ≥ c M κ log ϵ C cl norm ⟹ ∥ x k − x ⋆ ∥ ≤ ϵ . (238)
Under constant-history initialization,
log C c l n o r m = O c ( log M + log κ + log ( 1 + ∥ x i n i t − x ⋆ ∥ ) ) . \log C_{\mathrm{cl}}^{\mathrm{norm}}=\mathcal{O}_c(\log M+\log\kappa+\log(1+\|x_{\mathrm{init}}-x_\star\|)). log C cl norm = O c ( log M + log κ + log ( 1 + ∥ x init − x ⋆ ∥ )) .
Thus the leading accuracy-dependent term is O ~ ( τ κ log 1 ϵ ) \widetilde{\mathcal{O}}(\tau\sqrt{\kappa}\log\frac{1}{\epsilon}) O ( τ κ log ϵ 1 ) , where O ~ \widetilde{\mathcal{O}} O hides some logarithmic overhead from the explicit prefactor. This matches, up to that logarithmic overhead, the delayed first-order span-model lower-bound scale of Arjevani, Shamir, and Srebro on quadratic instances (Theorem 2.1).
The contraction rate of Theorem 6.1 , together with the polynomial prefactor bound (236) , pins down the iteration complexity of delayed GMM and shows it to be optimal, up to logarithmic factors, on the class for which a matching lower bound is known. We summarize these observations in the following corollary.
Corollary 6.3 (iteration complexity and near-optimality). Fix τ ≥ 1 \tau\geq1 τ ≥ 1 and c ∈ ( 0 , 2 / 3 ) c\in(0,2/3) c ∈ ( 0 , 2/3 ) , and let κ ≥ κ 0 ( τ , c ) \kappa\geq\kappa_0(\tau,c) κ ≥ κ 0 ( τ , c ) as in Theorem 6.1 . For every f ∈ S μ , L f\in\mathcal{S}_{\mu,L} f ∈ S μ , L and every accuracy ϵ > 0 \epsilon>0 ϵ > 0 , delayed GMM with the parameters of Theorem 6.1 and constant-history initialization x − τ − 1 = ⋯ = x 0 = x i n i t x_{-\tau-1}=\cdots=x_0=x_{\mathrm{init}} x − τ − 1 = ⋯ = x 0 = x init attains ∥ x k − x ⋆ ∥ ≤ ϵ \|x_k-x_\star\|\leq\epsilon ∥ x k − x ⋆ ∥ ≤ ϵ after
k ≥ ( 2 τ + 1 ) κ c log C ϵ (239) k\geq\frac{(2\tau+1)\sqrt{\kappa}}{c}\log\frac{C}{\epsilon} \tag*{(239)} k ≥ c ( 2 τ + 1 ) κ log ϵ C (239)
iterations, where C C C is the prefactor of Theorem 6.1 . The prefactor enters only inside the logarithm: by the constant-history bound (236) , log C = O c ( log ( 2 τ + 1 ) + log κ + log ∥ x i n i t − x ⋆ ∥ ) \log C = O_{\mathrm{c}}\bigl(\log(2\tau+1)+\log\kappa+\log\lVert x_{\mathrm{init}}-x_\star\rVert\bigr) log C = O c ( log ( 2 τ + 1 ) + log κ + log ∥ x init − x ⋆ ∥ ) , so the cost beyond ( 2 τ + 1 ) κ c log 1 ϵ \frac{(2\tau+1)\sqrt{\kappa}}{c}\log\frac{1}{\epsilon} c ( 2 τ + 1 ) κ log ϵ 1 is logarithmic in τ \tau τ and κ \kappa κ . This meets the delayed first-order span-model lower bound Ω ( τ κ log 1 ϵ ) \Omega\bigl(\tau\sqrt{\kappa}\log\frac{1}{\epsilon}\bigr) Ω ( τ κ log ϵ 1 ) of Arjevani, Shamir, and Srebro on quadratic instances (Theorem 2.1) up to a factor logarithmic in τ \tau τ (equivalently M = 2 τ + 1 M=2\tau+1 M = 2 τ + 1 ) and κ \kappa κ .
Acceleration under delay for smooth convex objectives
Our strongly convex analysis can be extended to smooth convex objectives that are not strongly convex. Let S 0 , L S_{0,L} S 0 , L denote the class of functions that are L L L -smooth and convex (but not necessarily strongly convex) on R d \mathbb{R}^{d} R d . Let f ∈ S 0 , L f\in S_{0,L} f ∈ S 0 , L , assume that arg min x f ( x ) \operatorname{arg\,min}_{x}f(x) arg min x f ( x ) is nonempty, and fix an arbitrary x ⋆ ∈ arg min x f ( x ) x_\star\in\operatorname{arg\,min}_{x}f(x) x ⋆ ∈ arg min x f ( x ) . Write
R ⋆ : = ∥ x 0 − x ⋆ ∥ . R_\star:=\lVert x_0-x_\star\rVert. R ⋆ := ∥ x 0 − x ⋆ ∥ .
The quantity R ⋆ R_\star R ⋆ will appear only in the convergence bound, and it is not assumed to be known to the algorithm. We consider the Catalyst framework [32 ] and state the standard relative-error accelerated proximal-point estimate.
Lemma 7.1 (relative-error accelerated proximal point). Let f ∈ S 0 , L f\in S_{0,L} f ∈ S 0 , L and λ > 0 \lambda>0 λ > 0 . Given some arbitrary v 0 v_0 v 0 , set x 0 = v 0 x_0=v_0 x 0 = v 0 and a 0 = 1 a_0=1 a 0 = 1 . For each s ≥ 1 s\geq1 s ≥ 1 , define
H s ( x ) : = f ( x ) + λ 2 ∥ x − v s − 1 ∥ 2 , H s ⋆ : = min x H s ( x ) , (240) H_s(x):=f(x)+\frac{\lambda}{2}\lVert x-v_{s-1}\rVert^2,\qquad H_s^\star:=\min_x H_s(x), \tag*{(240)} H s ( x ) := f ( x ) + 2 λ ∥ x − v s − 1 ∥ 2 , H s ⋆ := x min H s ( x ) , (240) and suppose that x s x_s x s satisfies the relative accuracy condition
H s ( x s ) − H s ⋆ ≤ δ s λ 2 ∥ x s − v s − 1 ∥ 2 , δ s : = 1 ( s + 1 ) 2 . (241) H_s(x_s)-H_s^\star\leq\frac{\delta_s\lambda}{2}\lVert x_s-v_{s-1}\rVert^2,\qquad\delta_s:=\frac{1}{(s+1)^2}. \tag*{(241)} H s ( x s ) − H s ⋆ ≤ 2 δ s λ ∥ x s − v s − 1 ∥ 2 , δ s := ( s + 1 ) 2 1 . (241) Choose a s ∈ ( 0 , 1 ) a_s\in(0,1) a s ∈ ( 0 , 1 ) as the positive solution of
a s 2 = ( 1 − a s ) a s − 1 2 , (242) a_s^2=(1-a_s)a_{s-1}^2, \tag*{(242)} a s 2 = ( 1 − a s ) a s − 1 2 , (242) and set
θ s : = a s − 1 ( 1 − a s − 1 ) a s − 1 2 + a s , v s : = x s + θ s ( x s − x s − 1 ) . (243) \theta_s:=\frac{a_{s-1}(1-a_{s-1})}{a_{s-1}^2+a_s},\qquad v_s:=x_s+\theta_s(x_s-x_{s-1}). \tag*{(243)} θ s := a s − 1 2 + a s a s − 1 ( 1 − a s − 1 ) , v s := x s + θ s ( x s − x s − 1 ) . (243) Then, for every s ≥ 1 s\geq1 s ≥ 1 ,
f ( x s ) − f ( x ⋆ ) ≤ 4 λ R ⋆ 2 ( s + 1 ) 2 . (244) f(x_s)-f(x_\star)\leq\frac{4\lambda R_\star^2}{(s+1)^2}. \tag*{(244)} f ( x s ) − f ( x ⋆ ) ≤ ( s + 1 ) 2 4 λ R ⋆ 2 . (244)
Proof. The result is a direct specialization of Theorem 7 and Proposition 9 of Lin, Mairal, and Harchaoui [33 ] . We verify the correspondence and constants for completeness.
In the notation of [33 ] , the proximal regularization parameter denoted by κ \kappa κ corresponds to λ \lambda λ here. Since the outer objective is convex, we take μ = 0 \mu=0 μ = 0 . Algorithm 2 of [33 ] therefore applies with the algorithm parameters a 0 = 1 a_0=1 a 0 = 1 , α s = a s \alpha_s=a_s α s = a s and q = μ + λ μ + λ = 0 q=\frac{\mu+\lambda}{\mu+\lambda}=0 q = μ + λ μ + λ = 0 , and equations (10)–(11) of Algorithm 2 reduce exactly to (240)–(241). Likewise, their relative stopping criterion (C2) coincides with (241) .
Theorem 7 of [33 ] yields
f ( x s ) − f ( x ⋆ ) ≤ A s − 1 ∏ j = 1 s ( 1 − δ j ) λ 2 ∥ x 0 − x ⋆ ∥ 2 , (245) f(x_s)-f(x_\star)\leq\frac{A_{s-1}}{\prod_{j=1}^{s}(1-\delta_j)}\frac{\lambda}{2}\lVert x_0-x_\star\rVert^2, \tag*{(245)} f ( x s ) − f ( x ⋆ ) ≤ ∏ j = 1 s ( 1 − δ j ) A s − 1 2 λ ∥ x 0 − x ⋆ ∥ 2 , (245)
where A s : = ∏ j = 1 s ( 1 − a j ) A_s := \prod_{j=1}^{s}(1-a_j) A s := ∏ j = 1 s ( 1 − a j ) , and A 0 : = 1 A_0 := 1 A 0 := 1 . By [33 Lemma 4] ,
A s ≤ 4 ( 2 + s γ 0 / λ ) 2 . A_s \le\frac{4}{\left(2+s\sqrt{\gamma_0/\lambda}\right)^2}. A s ≤ ( 2 + s γ 0 / λ ) 2 4 .
where γ 0 : = ( λ + μ ) a 0 ( a 0 − q ) \gamma_0 := (\lambda+\mu)a_0(a_0-q) γ 0 := ( λ + μ ) a 0 ( a 0 − q ) . Since μ = 0 \mu=0 μ = 0 , q = 0 q=0 q = 0 , and a 0 = 1 a_0=1 a 0 = 1 , we obtain γ 0 = λ \gamma_0=\lambda γ 0 = λ , and hence A s ≤ 4 / ( s + 2 ) 2 A_s \le4/(s+2)^2 A s ≤ 4/ ( s + 2 ) 2 . For the choice δ j = 1 ( j + 1 ) 2 \delta_j=\frac{1}{(j+1)^2} δ j = ( j + 1 ) 2 1 , the product in the denominator telescopes:
∏ j = 1 s ( 1 − δ j ) = ∏ j = 1 s j j + 1 ⋅ j + 2 j + 1 = s + 2 2 ( s + 1 ) ≥ 1 2 . \prod_{j=1}^{s}(1-\delta_j)=\prod_{j=1}^{s}\frac{j}{j+1}\cdot\frac{j+2}{j+1}=\frac{s+2}{2(s+1)}\ge\frac{1}{2}. j = 1 ∏ s ( 1 − δ j ) = j = 1 ∏ s j + 1 j ⋅ j + 1 j + 2 = 2 ( s + 1 ) s + 2 ≥ 2 1 .
Substituting these estimates into (245) , we conclude.
We now instantiate each proximal subproblem with delayed GMM. Fix τ ≥ 1 \tau\ge1 τ ≥ 1 and c ∈ ( 0 , 2 / 3 ) c\in(0,2/3) c ∈ ( 0 , 2/3 ) , and recall M : = 2 τ + 1 M:=2\tau+1 M := 2 τ + 1 . For a regularization parameter λ > 0 \lambda>0 λ > 0 , define the condition number of the proximal subproblems (condition number of the regularized function H s H_s H s ) by
χ λ : = L + λ λ = 1 + L λ . (246) \chi_\lambda:=\frac{L+\lambda}{\lambda}=1+\frac{L}{\lambda}. \tag*{(246)} χ λ := λ L + λ = 1 + λ L . (246)
Assume χ λ ≥ κ 0 ( τ , c ) \chi_\lambda\ge\kappa_0(\tau,c) χ λ ≥ κ 0 ( τ , c ) , where κ 0 ( τ , c ) \kappa_0(\tau,c) κ 0 ( τ , c ) is the threshold of Theorem 6.1. By Theorem 6.1, under constant-history initialization there is a dimensionless prefactor B τ , c ( χ λ ) B_{\tau,c}(\chi_\lambda) B τ , c ( χ λ ) such that delayed GMM applied to any λ \lambda λ -strongly convex and ( L + λ ) (L+\lambda) ( L + λ ) -smooth objective satisfies
∥ z t − z ⋆ ∥ ≤ B τ , c ( χ λ ) ( 1 − c M χ λ ) t ∥ z 0 − z ⋆ ∥ . (247) \lVert z_t-z_\star\rVert\le B_{\tau,c}(\chi_\lambda)\left(1-\frac{c}{M\sqrt{\chi_\lambda}}\right)^t\lVert z_0-z_\star\rVert. \tag*{(247)} ∥ z t − z ⋆ ∥ ≤ B τ , c ( χ λ ) ( 1 − M χ λ c ) t ∥ z 0 − z ⋆ ∥ . (247)
Moreover, Theorem 6.1 gives
B τ , c ( χ λ ) = O c ( M 5 χ λ 2 ) . (248) B_{\tau,c}(\chi_\lambda)=\mathcal{O}_c(M^5\chi_\lambda^2). \tag*{(248)} B τ , c ( χ λ ) = O c ( M 5 χ λ 2 ) . (248)
At outer iteration s s s , initialize the entire inner delayed history at the current proximal center v s − 1 v_{s-1} v s − 1 and apply delayed GMM to H s H_s H s in (240) with parameters
α i n = 1 M 2 ( L + λ ) , β i n = 1 − 2 M χ λ , ν i n = M − 1 2 . (249) \alpha_{\mathrm{in}}=\frac{1}{M^2(L+\lambda)},\qquad\beta_{\mathrm{in}}=1-\frac{2}{M\sqrt{\chi_\lambda}},\qquad\nu_{\mathrm{in}}=M-\frac{1}{2}. \tag*{(249)} α in = M 2 ( L + λ ) 1 , β in = 1 − M χ λ 2 , ν in = M − 2 1 . (249)
Thus, an inner delayed gradient evaluation at a query point u u u uses
∇ H s ( u ) = ∇ f ( u ) + λ ( u − v s − 1 ) . (250) \nabla H_s(u)=\nabla f(u)+\lambda(u-v_{s-1}). \tag*{(250)} ∇ H s ( u ) = ∇ f ( u ) + λ ( u − v s − 1 ) . (250)
More, specifically DREAM as an inner solver for the subproblem (240) consists of the updates
y j = z j + ν i n ( z j − z j − 1 ) followed by z j + 1 = z j − α i n ∇ H s ( y j − τ ) + β i n ( z j − z j − 1 ) , y_j=z_j+\nu_{\mathrm{in}}(z_j-z_{j-1})\quad\text{followed by}\quad z_{j+1}=z_j-\alpha_{\mathrm{in}}\nabla H_s(y_{j-\tau})+\beta_{\mathrm{in}}(z_j-z_{j-1}), y j = z j + ν in ( z j − z j − 1 ) followed by z j + 1 = z j − α in ∇ H s ( y j − τ ) + β in ( z j − z j − 1 ) ,
with initialization z − τ − 1 = ⋯ = z − 1 = z 0 = v s − 1 z_{-\tau-1}=\cdots=z_{-1}=z_0=v_{s-1} z − τ − 1 = ⋯ = z − 1 = z 0 = v s − 1 .
Theorem 7.2 (accelerated delayed rate for smooth convex objectives). Fix τ ≥ 1 \tau\ge1 τ ≥ 1 and c ∈ ( 0 , 2 / 3 ) c\in(0,2/3) c ∈ ( 0 , 2/3 ) , and choose λ > 0 \lambda>0 λ > 0 such that
χ λ = 1 + L λ ≥ κ 0 ( τ , c ) . \chi_\lambda=1+\frac{L}{\lambda}\ge\kappa_0(\tau,c). χ λ = 1 + λ L ≥ κ 0 ( τ , c ) . For each outer iteration s ≥ 1 s\ge1 s ≥ 1 , run delayed GMM on H s H_s H s with the parameters (249) , with constant inner history initialized at v s − 1 v_{s-1} v s − 1 , for
T s : = ⌊ M χ λ 2 c log ( 4 χ λ B τ , c ( χ λ ) 2 ( s + 1 ) 2 ) ⌋ (251) T_s:=\left\lfloor\frac{M\sqrt{\chi_\lambda}}{2c}\log\left(4\chi_\lambda B_{\tau,c}(\chi_\lambda)^2(s+1)^2\right)\right\rfloor \tag*{(251)} T s := ⌊ 2 c M χ λ log ( 4 χ λ B τ , c ( χ λ ) 2 ( s + 1 ) 2 ) ⌋ (251)
inner iterations, and denote the resulting point by x s x_s x s . Update a s a_s a s and v s v_s v s according to (242) –(243) . Then the algorithm does not require any a priori bound on R ⋆ = ∥ x 0 − x ⋆ ∥ R_\star= \lVert x_0 - x_\star\rVert R ⋆ = ∥ x 0 − x ⋆ ∥ , and for every s ≥ 1 s \ge1 s ≥ 1 ,
f ( x s ) − f ( x ⋆ ) ≤ 4 λ R ⋆ 2 ( s + 1 ) 2 . (252) f(x_s) - f(x_\star) \le\frac{4\lambda R_\star^2}{(s+1)^2}. \tag*{(252)} f ( x s ) − f ( x ⋆ ) ≤ ( s + 1 ) 2 4 λ R ⋆ 2 . (252)
Consequently, if
K ϵ : = max { 1 , ⌈ 2 λ R ⋆ 2 ϵ ⌉ } , (253) K_\epsilon:= \max\left\{1,\left\lceil2\sqrt{\frac{\lambda R_\star^2}{\epsilon}}\right\rceil\right\}, \tag*{(253)} K ϵ := max { 1 , ⌈ 2 ϵ λ R ⋆ 2 ⌉ } , (253)
then f ( x K ϵ ) − f ( x ⋆ ) ≤ ϵ f(x_{K_\epsilon}) - f(x_\star) \le\epsilon f ( x K ϵ ) − f ( x ⋆ ) ≤ ϵ . The total number N ϵ : = ∑ s = 1 K ϵ T s N_\epsilon:= \sum_{s=1}^{K_\epsilon} T_s N ϵ := ∑ s = 1 K ϵ T s of delayed-GMM inner iterations satisfies
N ϵ ≤ K ϵ + M χ λ 2 c K ϵ [ log ( 4 χ λ B τ , c ( χ λ ) 2 ) + 2 log ( K ϵ + 1 ) ] . (254) N_\epsilon\le K_\epsilon+ \frac{M\sqrt{\chi_\lambda}}{2c}K_\epsilon\left[\log\left(4\chi_\lambda B_{\tau,c}(\chi_\lambda)^2\right) + 2\log(K_\epsilon+1)\right]. \tag*{(254)} N ϵ ≤ K ϵ + 2 c M χ λ K ϵ [ log ( 4 χ λ B τ , c ( χ λ ) 2 ) + 2 log ( K ϵ + 1 ) ] . (254)
Since
K ϵ ≤ 1 + 2 λ R ⋆ 2 ϵ and λ ⋅ χ λ = L + λ , K_\epsilon\le1 + 2\sqrt{\frac{\lambda R_\star^2}{\epsilon}}
\qquad\text{and}\qquad
\lambda\cdot\chi_\lambda= L+\lambda, K ϵ ≤ 1 + 2 ϵ λ R ⋆ 2 and λ ⋅ χ λ = L + λ ,
it follows that
N ϵ = O ~ ( ( 2 τ + 1 ) [ χ λ + ( L + λ ) R ⋆ 2 ϵ ] ) , (255) N_\epsilon= \widetilde{O}\left((2\tau+1)\left[\sqrt{\chi_\lambda}+\sqrt{\frac{(L+\lambda)R_\star^2}{\epsilon}}\right]\right), \tag*{(255)} N ϵ = O ( ( 2 τ + 1 ) [ χ λ + ϵ ( L + λ ) R ⋆ 2 ] ) , (255)
where O ~ \widetilde{O} O hides logarithmic factors in c c c , τ \tau τ , χ λ \chi_\lambda χ λ , and λ R ⋆ 2 / ϵ \lambda R_\star^2/\epsilon λ R ⋆ 2 / ϵ .
Proof. Fix an outer iteration s s s and let
p s : = arg min x H s ( x ) . p_s := \mathop{\arg\min}_{x} H_s(x). p s := arg min x H s ( x ) .
Since H s H_s H s is λ \lambda λ -strongly convex and ( L + λ ) (L+\lambda) ( L + λ ) -smooth, its condition number is χ λ \chi_\lambda χ λ . Theorem 6.1 , applied to the inner run initialized at z 0 = v s − 1 z_0=v_{s-1} z 0 = v s − 1 with constant initialization (with z − τ − 1 = ⋯ = z − 1 = z 0 = v s − 1 z_{-\tau-1}=\cdots=z_{-1}=z_0=v_{s-1} z − τ − 1 = ⋯ = z − 1 = z 0 = v s − 1 ) gives
∥ z t − p s ∥ ≤ B τ , c ( χ λ ) ρ λ t ∥ v s − 1 − p s ∥ , ρ λ : = 1 − c M χ λ . (256) \lVert z_t-p_s\rVert\le B_{\tau,c}(\chi_\lambda)\rho_\lambda^t\lVert v_{s-1}-p_s\rVert,\qquad
\rho_\lambda:=1-\frac{c}{M\sqrt{\chi_\lambda}}. \tag*{(256)} ∥ z t − p s ∥ ≤ B τ , c ( χ λ ) ρ λ t ∥ v s − 1 − p s ∥ , ρ λ := 1 − M χ λ c . (256)
By ( L + λ ) (L+\lambda) ( L + λ ) -smoothness and ∇ H s ( p s ) = 0 \nabla H_s(p_s)=0 ∇ H s ( p s ) = 0 ,
H s ( z t ) − H s ⋆ ≤ L + λ 2 ∥ z t − p s ∥ 2 ≤ λ χ λ 2 B τ , c ( χ λ ) 2 ρ λ 2 t ∥ v s − 1 − p s ∥ 2 . (257) H_s(z_t)-H^\star_s\le\frac{L+\lambda}{2}\lVert z_t-p_s\rVert^2\le\frac{\lambda\chi_\lambda}{2}B_{\tau,c}(\chi_\lambda)^2\rho_\lambda^{2t}\lVert v_{s-1}-p_s\rVert^2. \tag*{(257)} H s ( z t ) − H s ⋆ ≤ 2 L + λ ∥ z t − p s ∥ 2 ≤ 2 λ χ λ B τ , c ( χ λ ) 2 ρ λ 2 t ∥ v s − 1 − p s ∥ 2 . (257)
Since 1 − u ≤ e − u 1-u\le e^{-u} 1 − u ≤ e − u for u ∈ ( 0 , 1 ) u\in(0,1) u ∈ ( 0 , 1 ) , the choice (251) implies B τ , c ( χ λ ) 2 ρ λ 2 T s ≤ δ s / ( 4 χ λ ) B_{\tau,c}(\chi_\lambda)^2\rho_\lambda^{2T_s}\le\delta_s/(4\chi_\lambda) B τ , c ( χ λ ) 2 ρ λ 2 T s ≤ δ s / ( 4 χ λ ) with δ s = 1 / ( s + 1 ) 2 \delta_s=1/(s+1)^2 δ s = 1/ ( s + 1 ) 2 . Hence,
H s ( x s ) − H s ⋆ ≤ δ s λ 8 ∥ v s − 1 − p s ∥ 2 , (258) H_s(x_s)-H^\star_s\le\frac{\delta_s\lambda}{8}\lVert v_{s-1}-p_s\rVert^2, \tag*{(258)} H s ( x s ) − H s ⋆ ≤ 8 δ s λ ∥ v s − 1 − p s ∥ 2 , (258)
where x s = z T s x_s=z_{T_s} x s = z T s . Strong convexity also gives
∥ x s − p s ∥ 2 ≤ 2 λ ( H s ( x s ) − H s ⋆ ) ≤ δ s 4 ∥ v s − 1 − p s ∥ 2 . \lVert x_s-p_s\rVert^2\le\frac{2}{\lambda}\left(H_s(x_s)-H^\star_s\right)\le\frac{\delta_s}{4}\lVert v_{s-1}-p_s\rVert^2. ∥ x s − p s ∥ 2 ≤ λ 2 ( H s ( x s ) − H s ⋆ ) ≤ 4 δ s ∥ v s − 1 − p s ∥ 2 .
Because δ s ≤ 1 \delta_s\le1 δ s ≤ 1 ,
∥ x s − v s − 1 ∥ ≥ ∥ v s − 1 − p s ∥ − ∥ x s − p s ∥ ≥ 1 2 ∥ v s − 1 − p s ∥ . \lVert x_s-v_{s-1}\rVert\ge\lVert v_{s-1}-p_s\rVert-\lVert x_s-p_s\rVert\ge\frac{1}{2}\lVert v_{s-1}-p_s\rVert. ∥ x s − v s − 1 ∥ ≥ ∥ v s − 1 − p s ∥ − ∥ x s − p s ∥ ≥ 2 1 ∥ v s − 1 − p s ∥ .
Combining this inequality with (258) yields
H s ( x s ) − H s ⋆ ≤ δ s λ 2 ∥ x s − v s − 1 ∥ 2 . H_s(x_s)-H^\star_s \le\frac{\delta_s\lambda}{2}\lVert x_s-v_{s-1}\rVert^2. H s ( x s ) − H s ⋆ ≤ 2 δ s λ ∥ x s − v s − 1 ∥ 2 .
Thus the relative accuracy condition (241) holds at every outer iteration. Lemma 7.1 therefore gives (252) . The choice (253) then implies f ( x K ϵ ) − f ( x ⋆ ) ≤ ϵ f(x_{K_\epsilon})-f(x_\star)\le\epsilon f ( x K ϵ ) − f ( x ⋆ ) ≤ ϵ .
It remains to count inner iterations. Summing (251) and using s + 1 ≤ K ϵ + 1 s+1\le K_\epsilon+1 s + 1 ≤ K ϵ + 1 gives
N ϵ ≤ K ϵ + M χ λ 2 c ∑ s = 1 K ϵ [ log ( 4 χ λ B τ , c ( χ λ ) ) + 2 log ( s + 1 ) ] ≤ K ϵ + M χ λ 2 c K ϵ [ log ( 4 χ λ B τ , c ( χ λ ) 2 ) + 2 log ( K ϵ + 1 ) ] , (259) \begin{aligned}
N_\epsilon\le K_\epsilon+\frac{M\sqrt{\chi_\lambda}}{2c}\sum_{s=1}^{K_\epsilon}\left[\log(4\chi_\lambda B_{\tau,c}(\chi_\lambda))+2\log(s+1)\right] \\
&\le K_\epsilon+\frac{M\sqrt{\chi_\lambda}}{2c}K_\epsilon\left[\log\left(4\chi_\lambda B_{\tau,c}(\chi_\lambda)^2\right)+2\log(K_\epsilon+1)\right], \tag*{(259)}
\end{aligned} N ϵ ≤ K ϵ + 2 c M χ λ s = 1 ∑ K ϵ [ log ( 4 χ λ B τ , c ( χ λ )) + 2 log ( s + 1 ) ] ≤ K ϵ + 2 c M χ λ K ϵ [ log ( 4 χ λ B τ , c ( χ λ ) 2 ) + 2 log ( K ϵ + 1 ) ] , (259)
which is (252). Since
χ λ ⋅ λ = L + λ , \sqrt{\chi_\lambda}\cdot\sqrt{\lambda}=\sqrt{L+\lambda}, χ λ ⋅ λ = L + λ ,
substituting (253) gives (255) .
Corollary 7.3 (Iteration complexity and near-optimality). Fix τ ≥ 1 \tau\ge1 τ ≥ 1 and c ∈ ( 0 , 2 / 3 ) c\in(0,2/3) c ∈ ( 0 , 2/3 ) , and let f ∈ S 0 , L f\in S_{0,L} f ∈ S 0 , L . Define
κ ˉ 0 ( τ , c ) : = max { 2 , κ 0 ( τ , c ) } , λ : = L κ ˉ 0 ( τ , c ) − 1 . (260) \bar{\kappa}_0(\tau,c):=\max\{2,\kappa_0(\tau,c)\},\qquad\lambda:=\frac{L}{\bar{\kappa}_0(\tau,c)-1}. \tag*{(260)} κ ˉ 0 ( τ , c ) := max { 2 , κ 0 ( τ , c )} , λ := κ ˉ 0 ( τ , c ) − 1 L . (260) Then χ λ = κ ˉ 0 ( τ , c ) \chi_\lambda=\bar{\kappa}_0(\tau,c) χ λ = κ ˉ 0 ( τ , c ) and L + λ ≤ 2 L L+\lambda\le2L L + λ ≤ 2 L . Hence, by (255) , for every target accuracy ϵ > 0 \epsilon>0 ϵ > 0 , the proximal-point-type method of Theorem 7.2, with delayed GMM applied to the inner problems, produces an outer iterate x K ϵ x_{K_\epsilon} x K ϵ satisfying
f ( x K ϵ ) − f ( x ⋆ ) ≤ ϵ f(x_{K_\epsilon})-f(x_\star)\le\epsilon f ( x K ϵ ) − f ( x ⋆ ) ≤ ϵ using a total of
N ϵ = O ~ ( τ [ κ ˉ 0 ( τ , c ) + L R ⋆ 2 ϵ ] ) (261) N_\epsilon=\widetilde{O}\left(\tau\left[\sqrt{\bar{\kappa}_0(\tau,c)}+\sqrt{\frac{LR_\star^2}{\epsilon}}\right]\right) \tag*{(261)} N ϵ = O ( τ [ κ ˉ 0 ( τ , c ) + ϵ L R ⋆ 2 ] ) (261) delayed-GMM inner iterations. Moreover, in the high-accuracy regime
ϵ ≤ λ R ⋆ 2 = L R ⋆ 2 κ ˉ 0 ( τ , c ) − 1 , (262) \epsilon\le\lambda R_\star^2=\frac{LR_\star^2}{\bar{\kappa}_0(\tau,c)-1}, \tag*{(262)} ϵ ≤ λ R ⋆ 2 = κ ˉ 0 ( τ , c ) − 1 L R ⋆ 2 , (262) the first term in (261) is absorbed by the second, and the complexity simplifies to
N ϵ = O ~ ( τ L R ⋆ 2 ϵ ) . (263) N_\epsilon=\widetilde{O}\left(\tau\sqrt{\frac{LR_\star^2}{\epsilon}}\right). \tag*{(263)} N ϵ = O ( τ ϵ L R ⋆ 2 ) . (263) Thus, in this regime, the iteration complexity matches, up to logarithmic factors, the delayed first-order span-model lower-bound scale
Ω ( τ L R ⋆ 2 ϵ ) \Omega\left(\tau\sqrt{\frac{LR_\star^2}{\epsilon}}\right) Ω ( τ ϵ L R ⋆ 2 ) of Arjevani, Shamir, and Srebro on quadratic instances; see Theorem 2.1.
Proof. By the choice of parameters (260) ,
χ λ = κ ˉ 0 ( τ , c ) , L + λ = L κ ˉ 0 ( τ , c ) κ ˉ 0 ( τ , c ) − 1 ≤ 2 L , \chi_\lambda=\bar{\kappa}_0(\tau,c),\qquad L+\lambda=L\frac{\bar{\kappa}_0(\tau,c)}{\bar{\kappa}_0(\tau,c)-1}\le2L, χ λ = κ ˉ 0 ( τ , c ) , L + λ = L κ ˉ 0 ( τ , c ) − 1 κ ˉ 0 ( τ , c ) ≤ 2 L ,
where the last inequality uses κ ˉ 0 ( τ , c ) ≥ 2 \bar{\kappa}_0(\tau,c)\ge2 κ ˉ 0 ( τ , c ) ≥ 2 . Substituting these relations, together with M = 2 τ + 1 M=2\tau+1 M = 2 τ + 1 , into the general complexity bound (255) gives
N ϵ = O ~ ( τ [ κ ˉ 0 ( τ , c ) + L R ⋆ 2 ϵ ] ) , N_\epsilon=\widetilde{O}\left(\tau\left[\sqrt{\bar{\kappa}_0(\tau,c)}+\sqrt{\frac{LR_\star^2}{\epsilon}}\right]\right), N ϵ = O ( τ [ κ ˉ 0 ( τ , c ) + ϵ L R ⋆ 2 ] ) ,
which is (261) . The polynomial dependence of the inner prefactor B τ , c B_{\tau,c} B τ , c in (248) contributes only logarithmic factors to this bound.
Now suppose that ϵ \epsilon ϵ satisfies (262) . Then L R ⋆ 2 / ϵ ≥ κ ˉ 0 ( τ , c ) − 1 LR_\star^2/\epsilon\ge\bar{\kappa}_0(\tau,c)-1 L R ⋆ 2 / ϵ ≥ κ ˉ 0 ( τ , c ) − 1 . Since κ ˉ 0 ( τ , c ) ≥ 2 \bar{\kappa}_0(\tau,c)\ge2 κ ˉ 0 ( τ , c ) ≥ 2 , we have
κ ˉ 0 ( τ , c ) ≤ 2 ( κ ˉ 0 ( τ , c ) − 1 ) ≤ 2 L R ⋆ 2 ϵ , \bar{\kappa}_0(\tau,c)\le2\bigl(\bar{\kappa}_0(\tau,c)-1\bigr)\le\frac{2LR_\star^2}{\epsilon}, κ ˉ 0 ( τ , c ) ≤ 2 ( κ ˉ 0 ( τ , c ) − 1 ) ≤ ϵ 2 L R ⋆ 2 ,
and therefore
κ ˉ 0 ( τ , c ) ≤ 2 L R ⋆ 2 ϵ . \sqrt{\bar{\kappa}_0(\tau,c)}\le\sqrt{2}\sqrt{\frac{LR_\star^2}{\epsilon}}. κ ˉ 0 ( τ , c ) ≤ 2 ϵ L R ⋆ 2 .
Thus the first term in (261) is absorbed by the second, yielding (263) . Comparing this with the lower bound of Theorem 2.1 gives the claimed near-optimality up to logarithmic factors. □
Numerical Experiments
We use numerical experiments to test three consequences of the preceding analysis. First, on strongly convex quadratics, we verify the predicted loss of κ \sqrt{\kappa} κ -acceleration for classical heavy-ball (HB) and Nesterov accelerated gradient (NAG) under delayed gradients. Second, we examine whether the delay-matched generalized momentum method, referred to as DREAM, preserves the κ − 1 / 2 \kappa^{-1/2} κ − 1/2 spectral-gap scaling predicted by the theory. Third, we test whether the same behavior is visible on a nonlinear smooth strongly convex objective and whether the proposed parameters remain effective when the design delay is only a conservative upper bound. In all comparisons, the classical methods are numerically tuned, whereas DREAM uses the prescribed closed-form parameters; thus, the numerical protocol favors the baselines rather than the proposed method.
Quadratic objectives and spectral-radius diagnostics
For x ∈ R d x\in\mathbb{R}^d x ∈ R d , we consider the quadratic objective f q ( x ) = 1 2 x T Q x f_q(x)=\frac{1}{2}x^\mathsf{T}Qx f q ( x ) = 2 1 x T Q x where Q ≻ 0 Q\succ0 Q ≻ 0 . This centered form entails no loss of generality: any strongly convex quadratic can be written as f ( x ) = 1 2 ( x − x ⋆ ) T Q ( x − x ⋆ ) + f ⋆ f(x)=\frac{1}{2}(x-x_\star)^\mathsf{T}Q(x-x_\star)+f_\star f ( x ) = 2 1 ( x − x ⋆ ) T Q ( x − x ⋆ ) + f ⋆ , where x ⋆ x_\star x ⋆ is its minimizer and f ⋆ = f ( x ⋆ ) f_\star=f(x_\star) f ⋆ = f ( x ⋆ ) , and the methods we consider are invariant under translations, so shifting the minimizer to the origin moves the iterates by x ⋆ x_\star x ⋆ without altering the optimization dynamics. For x ∈ R d x\in\mathbb{R}^d x ∈ R d , consider
f q ( x ) = 1 2 x T Q x , Q ≻ 0 , σ ( Q ) ⊆ [ μ , L ] , κ = L / μ . (264) f_q(x)=\frac{1}{2}x^\mathsf{T}Qx,\qquad Q\succ0,\qquad\sigma(Q)\subseteq[\mu,L],\qquad\kappa=L/\mu. \tag*{(264)} f q ( x ) = 2 1 x T Q x , Q ≻ 0 , σ ( Q ) ⊆ [ μ , L ] , κ = L / μ . (264)
For a curvature mode λ ∈ [ μ , L ] \lambda\in[\mu,L] λ ∈ [ μ , L ] , the delayed GMM recursion is governed by the characteristic polynomial
P τ , δ ( z ) = z τ ( z − 1 ) ( z − β ) + δ ( ( 1 + ν ) z − ν ) , δ = α λ . (265) P_{\tau,\delta}(z)=z^\tau(z-1)(z-\beta)+\delta\bigl((1+\nu)z-\nu\bigr),\qquad\delta=\alpha\lambda. \tag*{(265)} P τ , δ ( z ) = z τ ( z − 1 ) ( z − β ) + δ ( ( 1 + ν ) z − ν ) , δ = α λ . (265)
Given a finite curvature grid Λ ⊂ [ μ , L ] \Lambda\subset[\mu, L] Λ ⊂ [ μ , L ] , we evaluate the empirical worst-case spectral radius
ρ ^ ( θ ; κ , τ ) = max λ ∈ Λ max { ∣ z ∣ : P τ , α λ ( z ) = 0 } , (266) \widehat{\rho}(\theta;\kappa,\tau)=\max_{\lambda\in\Lambda}\max\{|z|:P_{\tau,\alpha\lambda}(z)=0\}, \tag*{(266)} ρ ( θ ; κ , τ ) = λ ∈ Λ max max { ∣ z ∣ : P τ , α λ ( z ) = 0 } , (266)
where θ = α \theta=\alpha θ = α for gradient descent (GD), θ = ( α , β ) \theta=(\alpha,\beta) θ = ( α , β ) for HB and NAG, with ν = 0 \nu=0 ν = 0 for HB and ν = β \nu=\beta ν = β for NAG. For these three methods we report the smallest stable value found by the parameter search,
ρ ^ ∗ ( κ , τ ) = min θ : ρ ^ ( θ ; κ , τ ) < 1 ρ ^ ( θ ; κ , τ ) . (267) \widehat{\rho}^{*}(\kappa,\tau)=\min_{\theta:\widehat{\rho}(\theta;\kappa,\tau)<1}\widehat{\rho}(\theta;\kappa,\tau). \tag*{(267)} ρ ∗ ( κ , τ ) = θ : ρ ( θ ; κ , τ ) < 1 min ρ ( θ ; κ , τ ) . (267)
For DREAM, no numerical minimization is performed; we evaluate the spectral radius at the prescribed delay-matched parameters.
To distinguish accelerated from non-accelerated scaling, suppose 1 − ρ = Θ ( C κ − p ) 1-\rho=\Theta(C\kappa^{-p}) 1 − ρ = Θ ( C κ − p ) . Since − log ρ = 1 − ρ + O ( 1 − ρ ) 2 -\log\rho=1-\rho+O(1-\rho)^2 − log ρ = 1 − ρ + O ( 1 − ρ ) 2 as ρ → 1 \rho\to1 ρ → 1 , a least-squares fit of log ( − log ρ ) \log(-\log\rho) log ( − log ρ ) against log κ \log\kappa log κ has slope approximately − p -p − p . Thus, a slope close to − 1 / 2 -1/2 − 1/2 indicates accelerated κ \sqrt{\kappa} κ -dependence, whereas a slope approximately − 1 -1 − 1 indicates the non-accelerated κ \kappa κ -dependence.
Delay destroys classical momentum acceleration
We first optimize GD, HB, and NAG separately for each delay. We set μ = 1 \mu= 1 μ = 1 , use 33 logarithmically spaced condition numbers in [ 10 2 , 10 6 ] [10^2, 10^6] [ 1 0 2 , 1 0 6 ] , and consider τ ∈ { 0 , 1 , … , 5 } \tau\in\{0,1,\ldots,5\} τ ∈ { 0 , 1 , … , 5 } . For each parameter choice, we approximate the worst-case spectral radius of P τ , δ ( z ) P_{\tau,\delta}(z) P τ , δ ( z ) , with δ = α λ \delta= \alpha\lambda δ = α λ , over 10 equally spaced values of λ ∈ [ μ , L ] \lambda\in[\mu,L] λ ∈ [ μ , L ] , including the endpoints. We then minimize this worst-case radius over the candidate algorithm parameters using a grid search. For GD, we search over N α = 500 N_\alpha= 500 N α = 500 equally spaced stepsizes in [ 10 − 12 , α h i ] [10^{-12},\alpha_{\mathrm{hi}}] [ 1 0 − 12 , α hi ] , where α h i \alpha_{\mathrm{hi}} α hi is determined by starting from α 0 = 0.01 / L \alpha_0 = 0.01/L α 0 = 0.01/ L , halving until stability is achieved, and then doubling until instability occurs. The stepsize yielding the smallest worst-case spectral radius is selected. For HB and NAG, β \beta β is selected through an initial grid search over 25 equally spaced values of s = log ( 1 − β ) s = \log(1-\beta) s = log ( 1 − β ) in [ log ( 10 − 7 ) , 0 ] [\log(10^{-7}),0] [ log ( 1 0 − 7 ) , 0 ] . For each candidate β \beta β , we repeat the stepsize search described above using N α = 150 N_\alpha= 150 N α = 150 stepsizes and select the ( α , β ) (\alpha,\beta) ( α , β ) pair with the smallest worst-case spectral radius. We then perform 2 successive refinements, each evaluating 25 candidate values in a smaller interval [ s b e s t − Δ , s b e s t + Δ ] ∩ [ log ( 10 − 7 ) , 0 ] [s_{\mathrm{best}}-\Delta,s_{\mathrm{best}}+\Delta]\cap[\log(10^{-7}),0] [ s best − Δ , s best + Δ ] ∩ [ log ( 1 0 − 7 ) , 0 ] , where s b e s t s_{\mathrm{best}} s best corresponds to the best parameter pair found and Δ \Delta Δ is the previous grid spacing. The α \alpha α search is repeated for each candidate β \beta β in both refinements, and the best stable ( α , β ) (\alpha,\beta) ( α , β ) pair across all 3 rounds is retained.
Figure 3 shows a sharp change in scaling once delay is introduced. At τ = 0 \tau= 0 τ = 0 , HB and NAG have fitted slopes − 0.500 -0.500 − 0.500 and − 0.505 -0.505 − 0.505 , respectively, while GD has slope − 1.000 -1.000 − 1.000 , reproducing the standard accelerated/non-accelerated dichotomy. With one-step delay, HB already has slope − 1.000 -1.000 − 1.000 , while NAG has the intermediate slope − 0.659 -0.659 − 0.659 . For τ = 2 \tau= 2 τ = 2 , the NAG slope degrades to − 0.976 -0.976 − 0.976 , and for τ ≥ 3 \tau\ge3 τ ≥ 3 all three methods are essentially indistinguishable at the level of the condition-number exponent. These computations are consistent with Theorem 3.12 and Theorem 3.17 , which show that classical momentum cannot retain the κ \sqrt{\kappa} κ -accelerated rate in the presence of any fixed positive delay, even in the case that its free parameters are optimally tuned. Moreover, Lemma 3.13 establishes a matching non-accelerated Θ ( κ − 2 / 3 ) \Theta(\kappa^{-2/3}) Θ ( κ − 2/3 ) rate for NAG when τ = 1 \tau= 1 τ = 1 . The numerical results closely track this theoretical prediction, providing empirical confirmation of the κ − 2 / 3 \kappa^{-2/3} κ − 2/3 scaling.
Figure 3. Empirical worst-case contraction factor for GD, HB, and NAG after numerical parameter tuning. The vertical axis is log ( − log ρ ∗ ) \log(-\log\rho^{*}) log ( − log ρ ∗ ) , so the fitted slope estimates the condition-number exponent of the spectral gap. HB loses the − 1 / 2 -1/2 − 1/2 accelerated slope after one step of delay; NAG degrades more gradually but approaches the non-accelerated slope − 1 -1 − 1 as the delay increases.
Convergence rate comparison for one-step delay
For the one-step delay (τ = 1 \tau= 1 τ = 1 ), we also provide a direct numerical comparison of the convergence rates for GD, HB and NAG. We set τ = 1 \tau= 1 τ = 1 and κ = 10 \kappa= 10 κ = 10 . HB, and NAG use their numerically optimized parameters, obtained using 10 logarithmically spaced values of λ ∈ [ μ , L ] \lambda\in[\mu, L] λ ∈ [ μ , L ] , whereas DREAM uses the closed-form parameterization from Proposition 3.19 specialized to τ = 1 \tau= 1 τ = 1 . Figure 4 plots the roots of the characteristic polynomial P 1 , δ ( z ) P_{1,\delta}(z) P 1 , δ ( z ) at the empirically worst-case curvature mode λ ⋆ ∈ [ μ , L ] \lambda_{\star} \in[\mu, L] λ ⋆ ∈ [ μ , L ] , defined as the mode that maximizes the spectral radius of P 1 , α λ ( z ) P_{1,\alpha\lambda}(z) P 1 , α λ ( z ) over λ ∈ [ μ , L ] \lambda\in[\mu, L] λ ∈ [ μ , L ] . The dashed black circle is the unit circle, and each dash-dotted circle has radius equal to the corresponding worst-case contraction factor.
Figure 4. Pole locations for τ = 1 \tau= 1 τ = 1 and κ = 10 \kappa= 10 κ = 10 . The method-specific dash-dotted circles have radii equal to the corresponding worst-case spectral radii. The proposed method DREAM has the fastest convergence.
For the three momentum methods shown in the figure, the estimated rates are
ρ ^ H B ⋆ ≈ 0.909 , ρ ^ N A G ⋆ ≈ 0.887 , ρ ^ D R E A M ⋆ ≈ 0.842. (268) \widehat{\rho}_{\mathrm{HB}}^{\star} \approx0.909,\qquad\widehat{\rho}_{\mathrm{NAG}}^{\star} \approx0.887,\qquad\widehat{\rho}_{\mathrm{DREAM}}^{\star} \approx0.842. \tag*{(268)} ρ HB ⋆ ≈ 0.909 , ρ NAG ⋆ ≈ 0.887 , ρ DREAM ⋆ ≈ 0.842. (268)
For DREAM, the last value agrees with the exact one-step delay formula ρ D R E A M = 1 − 1 2 κ \rho_{\mathrm{DREAM}} = 1 - \frac{1}{2\sqrt{\kappa}} ρ DREAM = 1 − 2 κ 1 , at κ = 10 \kappa= 10 κ = 10 . Thus, the dominant DREAM root lies visibly farther inside the unit disk than the dominant roots of HB and NAG; even if we choose the best parameters for HB and NAG numerically. As predicted by our theory, NAG is faster than HB but DREAM is the fastest in terms of the (worst-case) convergence rate. We observe a similar behavior for larger choices of κ \kappa κ values as well.
Accelerated behavior of DREAM at larger delays
We next test whether the accelerated exponent persists well beyond the one-step setting. We consider τ ∈ { 1 , 5 , 10 , 100 } \tau\in\{1,5,10,100\} τ ∈ { 1 , 5 , 10 , 100 } and κ ∈ [ 10 2 , 10 6 ] \kappa\in[10^{2}, 10^{6}] κ ∈ [ 1 0 2 , 1 0 6 ] . GD, HB, and NAG are again tuned numerically, using 50 logarithmically spaced values of λ ∈ [ μ , L ] \lambda\in[\mu, L] λ ∈ [ μ , L ] , while DREAM is evaluated without tuning at
M = 2 τ + 1 , α = 1 M 2 L , β = 1 − 2 M κ , ν = M − 1 2 , (269) M = 2\tau+ 1,\qquad\alpha= \frac{1}{M^{2}L},\qquad\beta= 1 - \frac{2}{M\sqrt{\kappa}},\qquad\nu= M - \frac{1}{2}, \tag*{(269)} M = 2 τ + 1 , α = M 2 L 1 , β = 1 − M κ 2 , ν = M − 2 1 , (269)
based on the parameters of Theorem 3.25 . For consistency across delays, we use the general parameterization (269) for all τ ∈ { 1 , 5 , 10 , 100 } \tau\in\{1,5,10,100\} τ ∈ { 1 , 5 , 10 , 100 } , including τ = 1 \tau= 1 τ = 1 . This differs from the sharper one-step parameterization of Proposition 3.19 , used in the pole comparison of Figure 4 .
Figure 5 shows the resulting rate diagnostic, and Table 1 summarizes the fitted slopes displayed in the figure. The classical methods approach the non-accelerated value − 1 -1 − 1 ; NAG retains an intermediate exponent at τ = 1 \tau= 1 τ = 1 but is effectively non-accelerated by τ = 5 \tau= 5 τ = 5 . In contrast, the DREAM slope remains between − 0.48 -0.48 − 0.48 and − 0.49 -0.49 − 0.49 from τ = 1 \tau= 1 τ = 1 through τ = 100 \tau= 100 τ = 100 . Increasing the delay therefore changes the multiplicative constant in the spectral gap, as predicted by the 1 / ( 2 τ + 1 ) 1/(2\tau+ 1) 1/ ( 2 τ + 1 ) factor, but does not change its observed κ − 1 / 2 \kappa^{-1/2} κ − 1/2 dependence.
Table 1. Fitted slopes of log ( − log ρ ) \log(-\log\rho) log ( − log ρ ) versus log κ \log\kappa log κ in the larger-delay quadratic experiment. Values near − 1 / 2 -1/2 − 1/2 indicate accelerated scaling; values near − 1 -1 − 1 indicate non-accelerated scaling.
Figure 5. Condition-number scaling on quadratic objectives for delays up to τ = 100 \tau= 100 τ = 100 . GD, HB, and NAG are numerically tuned; DREAM uses the fixed parameters in (269) . The proposed method retains a slope close to − 1 / 2 -1/2 − 1/2 for every tested delay, while the classical methods approach slope − 1 -1 − 1 .
Strongly convex setting: ℓ 2 \ell_2 ℓ 2 -regularized logistic regression
We next test whether the quadratic picture persists for a nonlinear smooth strongly convex problem. We use ℓ 2 \ell_2 ℓ 2 -regularized logistic regression,
f ( x ) = 1 m ∑ i = 1 m log ( 1 + exp ( − y i a i T x ) ) + μ r 2 ∥ x ∥ 2 , y i ∈ { − 1 , 1 } . (270) f(x) = \frac{1}{m}\sum_{i=1}^{m} \log\left(1+\exp\left(-y_i a_i^{\mathsf{T}}x\right)\right) + \frac{\mu_r}{2}\lVert x\rVert^2,\qquad y_i \in\{-1,1\}. \tag*{(270)} f ( x ) = m 1 i = 1 ∑ m log ( 1 + exp ( − y i a i T x ) ) + 2 μ r ∥ x ∥ 2 , y i ∈ { − 1 , 1 } . (270)
Writing A ∈ R m × d A \in\mathbb{R}^{m\times d} A ∈ R m × d for the feature matrix, the Hessian satisfies
μ r I ⪯ ∇ 2 f ( x ) ⪯ 1 4 m A T A + μ r I ; (271) \mu_r I \preceq\nabla^2 f(x) \preceq\frac{1}{4m}A^{\mathsf{T}}A+\mu_r I; \tag*{(271)} μ r I ⪯ ∇ 2 f ( x ) ⪯ 4 m 1 A T A + μ r I ; (271)
hence, a valid global smoothness and strong-convexity pair is
μ g l o b a l = μ r , L g l o b a l = λ max ( A T A 4 m ) + μ r . (272) \mu_{\mathrm{global}}=\mu_r,\qquad L_{\mathrm{global}}=\lambda_{\max}\left(\frac{A^{\mathsf{T}}A}{4m}\right)+\mu_r. \tag*{(272)} μ global = μ r , L global = λ m a x ( 4 m A T A ) + μ r . (272)
To vary conditioning in a controlled way, we use m = 1000 m = 1000 m = 1000 , d = 10 d = 10 d = 10 , and μ r = 0.1 \mu_r = 0.1 μ r = 0.1 , and construct the feature matrix as
A = 4 m U diag ( c g ~ ) V T + y 1 d T , (273) A = \sqrt{4m}\,U \operatorname{diag}(\sqrt{c\tilde{\mathbf{g}}})V^{\mathsf{T}} + y\mathbf{1}_d^{\mathsf{T}}, \tag*{(273)} A = 4 m U diag ( c g ~ ) V T + y 1 d T , (273)
where U ∈ R m × d U \in\mathbb{R}^{m \times d} U ∈ R m × d has orthonormal columns and V ∈ R d × d V \in\mathbb{R}^{d \times d} V ∈ R d × d is orthogonal, both obtained from the QR factorizations of random matrices with i.i.d. standard normal entries. of dimensions m × d m \times d m × d and d × d d \times d d × d , respectively. Here, y ∈ { − 1 , 1 } m y \in\{-1,1\}^{m} y ∈ { − 1 , 1 } m is the vector of class labels, and 1 d ∈ R d \mathbf{1}_d \in\mathbb{R}^{d} 1 d ∈ R d denotes the vector of ones. The class labels are assigned deterministically, with y i = + 1 y_i = +1 y i = + 1 for i = 1 , … , m / 2 i = 1,\ldots,m/2 i = 1 , … , m /2 and y i = − 1 y_i = -1 y i = − 1 for i = m / 2 + 1 , … , m i = m/2 + 1,\ldots,m i = m /2 + 1 , … , m . The class-dependent offset term y 1 d T y\mathbf{1}_d^{\mathsf{T}} y 1 d T introduces a separation between the two classes. Since this additional term changes the spectrum of A T A / ( 4 m ) A^{\mathsf{T}}A/(4m) A T A / ( 4 m ) , the entries of the vector g ~ \tilde{\mathbf{g}} g ~ are scaled by the constant c c c to achieve the prescribed global condition-number bound.
To set the condition number of the objective to a desired value κ t a r g e t > 1 \kappa_{\mathrm{target}} > 1 κ target > 1 , we initialize g ~ ∈ R d \tilde{\mathbf{g}} \in\mathbb{R}^{d} g ~ ∈ R d with logarithmically spaced entries g ~ j = μ r ( κ t a r g e t − 1 ) ( j − 1 ) / ( d − 1 ) \tilde{g}_j = \mu_r(\kappa_{\mathrm{target}} - 1)^{(j-1)/(d-1)} g ~ j = μ r ( κ target − 1 ) ( j − 1 ) / ( d − 1 ) , for j = 1 , … , d j = 1,\ldots,d j = 1 , … , d , and set g = c g ~ \mathbf{g} = c\tilde{\mathbf{g}} g = c g ~ , where c > 0 c > 0 c > 0 is determined numerically, to a relative tolerance of 10 − 8 10^{-8} 1 0 − 8 , so that
λ max ( A ( c ) T A ( c ) 4 m ) = μ r ( κ t a r g e t − 1 ) . \lambda_{\max}\left(\frac{A(c)^{\mathsf{T}}A(c)}{4m}\right) = \mu_r(\kappa_{\mathrm{target}} - 1). λ m a x ( 4 m A ( c ) T A ( c ) ) = μ r ( κ target − 1 ) .
The numerical rescaling accounts for the class-dependent mean term, which changes the eigenvalues of A T A / ( 4 m ) A^{\mathsf{T}}A/(4m) A T A / ( 4 m ) from those prescribed by g \mathbf{g} g alone. Consequently, L g l o b a l = κ t a r g e t μ r L_{\mathrm{global}} = \kappa_{\mathrm{target}}\mu_r L global = κ target μ r , and the global condition-number bound used to choose the algorithm parameters is κ g l o b a l : = L g l o b a l / μ g l o b a l = \kappa_{\mathrm{global}} := L_{\mathrm{global}}/\mu_{\mathrm{global}} = κ global := L global / μ global = κ t a r g e t \kappa_{\mathrm{target}} κ target .
Given ϵ > 0 \epsilon> 0 ϵ > 0 , for trajectory-based comparisons we define
r k = ∥ x k − x ⋆ ∥ ∥ x 0 − x ⋆ ∥ , N ϵ : = min { k : r k ≤ ϵ } . (274) r_k = \frac{\lVert x_k - x^\star\rVert}{\lVert x_0 - x^\star\rVert}, \qquad N_\epsilon:= \min\{k : r_k \le\epsilon\}. \tag*{(274)} r k = ∥ x 0 − x ⋆ ∥ ∥ x k − x ⋆ ∥ , N ϵ := min { k : r k ≤ ϵ } . (274)
where x ⋆ x^\star x ⋆ denotes the minimizer of the objective function. We compute a numerical approximation to x ⋆ x^\star x ⋆ using L-BFGS-B followed by damped Newton refinement.
At fixed ϵ \epsilon ϵ , accelerated scaling corresponds to N ϵ = O ( κ ) N_\epsilon= O(\sqrt{\kappa}) N ϵ = O ( κ ) , whereas non-accelerated scaling corresponds to N ϵ = O ( κ ) N_\epsilon= O(\kappa) N ϵ = O ( κ ) . We use ϵ = 10 − 8 \epsilon= 10^{-8} ϵ = 1 0 − 8 and delays τ ∈ { 0 , 1 , 2 , 3 , 5 , 10 } \tau\in\{0,1,2,3,5,10\} τ ∈ { 0 , 1 , 2 , 3 , 5 , 10 } . For each instance and delay, GD, HB, and NAG are tuned by evaluating a grid of candidate parameters and selecting those with the smallest empirically estimated contraction factor, obtained by fitting the decay of log ∥ x k − x ⋆ ∥ \log\lVert x_k - x^\star\rVert log ∥ x k − x ⋆ ∥ against the iteration index k k k . To reduce computational cost, large candidate grids are evaluated in two stages: candidates are first ranked using short trajectories covering 12% of the iteration budget, after which the 48 highest-ranked candidates are evaluated over the full horizon. This tuning criterion reflects the geometric convergence rates studied in our theoretical analysis, while the resulting iteration counts N ϵ N_\epsilon N ϵ measure the corresponding finite-accuracy performance. DREAM is not tuned and uses (269) with ( L , κ ) = ( L g l o b a l , κ g l o b a l ) (L,\kappa) = (L_{\mathrm{global}},\kappa_{\mathrm{global}}) ( L , κ ) = ( L global , κ global ) .
Figure 6 displays the first-hit iteration counts. With no delay, HB and NAG follow the κ \sqrt{\kappa} κ reference trend while GD follows the κ \kappa κ trend. Under delay, HB and then NAG move toward the non-accelerated reference; by τ = 2 \tau= 2 τ = 2 –3 3 3 , their iteration counts scale similarly to GD over the tested range. DREAM continues to track the κ \sqrt{\kappa} κ guide for every tested delay. The experiment therefore reproduces on a nonlinear objective the same qualitative separation observed on quadratics: delay changes the constant for the delay-matched method but not the observed condition-number exponent.
Figure 6. First-hit stopping time N ε N_{\varepsilon} N ε for controlled-condition-number regularized logistic regression with ε = 10 − 8 \varepsilon= 10^{-8} ε = 1 0 − 8 . Dotted reference lines show O ( κ ) \mathcal{O}(\kappa) O ( κ ) and O ( κ ) \mathcal{O}(\sqrt{\kappa}) O ( κ ) scaling. Classical momentum schemes, i.e., HB and NAG, progressively approaches the non-accelerated trend as the delay grows, whereas DREAM remains close to the accelerated reference over the tested range.
Robustness to conservative delay estimates
The theoretical parameterization uses the true delay τ \tau τ . In practice, one may only know an upper bound. To assess this mismatch empirically, let τ ^ \hat{\tau} τ ^ denote the delay used to choose the parameters while the recursion itself uses the true delay τ \tau τ . We compare the matched choice τ ^ = τ \hat{\tau} = \tau τ ^ = τ with the conservative fixed choice τ ^ = 10 \hat{\tau} = 10 τ ^ = 10 for
τ ∈ { 0 , 1 , 2 , 3 , 5 , 10 } . (275) \tau\in\{0,1,2,3,5,10\}. \tag*{(275)} τ ∈ { 0 , 1 , 2 , 3 , 5 , 10 } . (275)
In each case, the parameters are obtained from (269) after replacing τ \tau τ by τ ^ \hat{\tau} τ ^ . Thus, only the parameter design uses the conservative delay; the gradient in the recursion remains delayed by the actual τ \tau τ . We test the two designs on controlled-condition-number least-squares and regularized logistic-regression instances and use the stopping rule (274) with ϵ = 10 − 6 \epsilon= 10^{-6} ϵ = 1 0 − 6 . Figure 7 and Figure 8 show that designing for τ ^ = 10 \hat{\tau} = 10 τ ^ = 10 increases the iteration count when the actual delay is smaller, but the log-log slope with respect to κ \kappa κ remains close to the accelerated value. The fitted exponents for the conservative design are approximately 0.49 for least squares and 0.52 for logistic regression. When τ = 10 \tau= 10 τ = 10 , the matched and conservative designs coincide, as they should.
Figure 7. Robustness to delay overestimation on controlled-condition-number least squares. The matched design uses τ ^ = τ \hat{\tau}=\tau τ ^ = τ , while the conservative design uses τ ^ = 10 \hat{\tau}=10 τ ^ = 10 for every actual delay. Both exhibit approximately κ \sqrt{\kappa} κ scaling over the tested range; conservatism primarily increases the iteration-count prefactor. A smaller N ε N_{\varepsilon} N ε indicates that fewer iterations are required to reach ε \varepsilon ε -accuracy and therefore corresponds to better performance.
Figure 8. Robustness to delay overestimation on controlled-condition-number regularized logistic regression. The matched and conservative designs have nearly parallel log-log trends, and they coincide when τ = τ ^ = 10 \tau= \hat{\tau} = 10 τ = τ ^ = 10 . A smaller N ϵ N_{\epsilon} N ϵ indicates that fewer iterations are required to reach ϵ \epsilon ϵ -accuracy and therefore corresponds to better performance.
This final experiment is intended as a robustness check rather than a theoretical guarantee. Our analysis assumes that the delay is known and establishes convergence for the corresponding matched-delay parameterization. The numerical results indicate that the accelerated O ( κ ) O(\sqrt{\kappa}) O ( κ ) scaling persists even when only an upper bound on the actual delay is available. Over the tested range, delay overestimation appears to primarily increase the multiplicative prefactor C C C , while leaving the observed O ( κ ) O(\sqrt{\kappa}) O ( κ ) dependence on the condition number unchanged.
Conclusion
We studied acceleration of first-order methods under fixed gradient delays. Our results show that classical momentum schemes are fundamentally sensitive to stale gradient information: neither heavy-ball nor Nesterov’s accelerated gradient method can, in general, be retuned to retain the accelerated condition-number dependence under a fixed positive delay. In contrast, by decoupling extrapolation from momentum and matching the extrapolation scale to the delay, DREAM retains the accelerated κ \sqrt{\kappa} κ dependence while updating the iterate at every iteration using the available delayed gradient. For smooth strongly convex objectives, the resulting iteration complexity has the near-optimal scaling O ~ ( τ κ log ( 1 / ε ) ) \widetilde{O}(\tau\sqrt{\kappa}\log(1/\varepsilon)) O ( τ κ log ( 1/ ε )) , and a regularization-and-restart construction extends the approach to smooth convex objectives. Our analysis also provides an analytic, all-delay IQC certificate that avoids numerical semidefinite programs or frequency-domain searches. The numerical experiments support the predicted separation between classical and delay-matched momentum and indicate that the accelerated scaling can be robust to conservative estimates of the delay.
Several directions remain open. Our theory assumes a fixed deterministic delay known to the algorithm, whereas practical asynchronous systems may exhibit time-varying, heterogeneous, or stochastic delays. Extending delay-matched momentum and the accompanying analytic certificates to these settings is a natural next step. It would also be interesting to study stochastic gradients and nonconvex objectives; more broadly, our results suggest that the adverse effect of delayed information need not preclude acceleration: appropriately adapting the algorithmic dynamics to the structure of the delay can preserve accelerated condition-number dependence even when fresh gradients are unavailable.
Acknowledgements
This work was supported in part by the Office of Naval Research under Grant N00014-24-1-2628 and N00014-24-1-2666. The authors used generative artificial intelligence (AI) tools to assist with language editing, improving the exposition, drafting and revising portions of the mathematical proof, developing and refining code for numerical experiments, and refining constants in certain estimates. All AI-assisted material was carefully reviewed, corrected where necessary, and further developed by the authors. The authors assume full responsibility for the mathematical correctness and all other content of the final manuscript.
Proofs of the supporting lemmas
We prove the supporting lemmas of § 5.2 in the order they were stated. The proofs are independent of one another and of the main argument. Throughout, fix the rate-weighted signals
x ^ k = ρ − k x k , y ^ k = ρ − k y k , u ^ k = ρ − k u k , v ^ k = u ^ k − m y ^ k . \widehat{x}_k = \rho^{-k}x_k,\qquad\widehat{y}_k = \rho^{-k}y_k,\qquad\widehat{u}_k = \rho^{-k}u_k,\qquad\widehat{v}_k = \widehat{u}_k - m\widehat{y}_k. x k = ρ − k x k , y k = ρ − k y k , u k = ρ − k u k , v k = u k − m y k .
Proof of Lemma 5.3
If Y ( z ) = g τ , q ( z ) U ( z ) Y(z) = g_{\tau,q}(z)U(z) Y ( z ) = g τ , q ( z ) U ( z ) denotes the unweighted zero-state input-output relation, then rate weighting corresponds to evaluating the old transform at ρ z \rho z ρ z . Namely,
Y ^ ( z ) = ∑ k ≥ 0 ρ − k y k z − k = ∑ k ≥ 0 y k ( ρ z ) − k = Y ( ρ z ) , U ^ ( z ) = U ( ρ z ) . \widehat{Y}(z) = \sum_{k\ge0}\rho^{-k}y_k z^{-k} = \sum_{k\ge0}y_k(\rho z)^{-k} = Y(\rho z),\qquad\widehat{U}(z) = U(\rho z). Y ( z ) = k ≥ 0 ∑ ρ − k y k z − k = k ≥ 0 ∑ y k ( ρ z ) − k = Y ( ρ z ) , U ( z ) = U ( ρ z ) .
Thus
Y ^ ( z ) = Y ( ρ z ) = g τ , q ( ρ z ) U ( ρ z ) = g τ , q ( ρ z ) U ^ ( z ) . \widehat{Y}(z) = Y(\rho z) = g_{\tau,q}(\rho z)U(\rho z) = g_{\tau,q}(\rho z)\widehat{U}(z). Y ( z ) = Y ( ρ z ) = g τ , q ( ρ z ) U ( ρ z ) = g τ , q ( ρ z ) U ( z ) .
Consequently the transfer from u ^ \widehat{u} u to y ^ \widehat{y} y is
g ρ ( z ) : = g τ , q ( ρ z ) , G ρ ( z ) : = g ρ ( z ) I d . g_\rho(z) := g_{\tau,q}(\rho z),\qquad G_\rho(z) := g_\rho(z)I_d. g ρ ( z ) := g τ , q ( ρ z ) , G ρ ( z ) := g ρ ( z ) I d .
Equivalently, rate weighting divides every open-loop pole by ρ \rho ρ : a pole p p p of g τ , q g_{\tau,q} g τ , q becomes a pole p / ρ p/\rho p / ρ of g τ , q ( ρ z ) g_{\tau,q}(\rho z) g τ , q ( ρ z ) .
Proof of Lemma 5.4
The midpoint change of variables gives
u k − τ = m y k − τ + v k − τ . u_{k-\tau} = my_{k-\tau} + v_{k-\tau}. u k − τ = m y k − τ + v k − τ .
Using the formula for y y y at the delayed time k − τ k-\tau k − τ ,
y k − τ = ( 1 + ν ) x k − τ − ν x k − τ − 1 , y_{k-\tau} = (1+\nu)x_{k-\tau} - \nu x_{k-\tau-1}, y k − τ = ( 1 + ν ) x k − τ − ν x k − τ − 1 ,
the state recursion becomes
x k + 1 = ( 1 + β ) x k − β x k − 1 − α m y k − τ − α v k − τ = ( 1 + β ) x k − β x k − 1 − α m ( 1 + ν ) x k − τ + α m ν x k − τ − 1 − α v k − τ . \begin{aligned}
x_{k+1} &= (1+\beta)x_k - \beta x_{k-1} - \alpha m y_{k-\tau} - \alpha v_{k-\tau}\\
&= (1+\beta)x_k - \beta x_{k-1} - \alpha m(1+\nu)x_{k-\tau} + \alpha m\nu x_{k-\tau-1} - \alpha v_{k-\tau}.
\end{aligned} x k + 1 = ( 1 + β ) x k − β x k − 1 − α m y k − τ − α v k − τ = ( 1 + β ) x k − β x k − 1 − α m ( 1 + ν ) x k − τ + α m ν x k − τ − 1 − α v k − τ .
Multiply this identity by ρ − ( k + 1 ) \rho^{-(k+1)} ρ − ( k + 1 ) . Since x j = ρ j x ^ j x_j = \rho^j\widehat{x}_j x j = ρ j x j and v j = ρ j v ^ j v_j = \rho^j\widehat{v}_j v j = ρ j v j , the terms are rescaled as
ρ − ( k + 1 ) x k = ρ − 1 x ^ k , ρ − ( k + 1 ) x k − 1 = ρ − 2 x ^ k − 1 , ρ − ( k + 1 ) x k − τ = ρ − ( τ + 1 ) x ^ k − τ , ρ − ( k + 1 ) x k − τ − 1 = ρ − ( τ + 2 ) x ^ k − τ − 1 , ρ − ( k + 1 ) v k − τ = ρ − ( τ + 1 ) v ^ k − τ . \begin{aligned}
\rho^{-(k+1)}x_k &= \rho^{-1}\widehat{x}_k,\qquad&\rho^{-(k+1)}x_{k-1} &= \rho^{-2}\widehat{x}_{k-1},\\
\rho^{-(k+1)}x_{k-\tau} &= \rho^{-(\tau+1)}\widehat{x}_{k-\tau},\qquad&\rho^{-(k+1)}x_{k-\tau-1} &= \rho^{-(\tau+2)}\widehat{x}_{k-\tau-1},\\
\rho^{-(k+1)}v_{k-\tau} &= \rho^{-(\tau+1)}\widehat{v}_{k-\tau}.
\end{aligned} ρ − ( k + 1 ) x k ρ − ( k + 1 ) x k − τ ρ − ( k + 1 ) v k − τ = ρ − 1 x k , = ρ − ( τ + 1 ) x k − τ , = ρ − ( τ + 1 ) v k − τ . ρ − ( k + 1 ) x k − 1 ρ − ( k + 1 ) x k − τ − 1 = ρ − 2 x k − 1 , = ρ − ( τ + 2 ) x k − τ − 1 ,
Substitution gives the first identity in (164) . For the output, multiply y k = ( 1 + ν ) x k − ν x k − 1 y_k = (1+\nu)x_k - \nu x_{k-1} y k = ( 1 + ν ) x k − ν x k − 1 by ρ − k \rho^{-k} ρ − k . Then
ρ − k x k = x ^ k , ρ − k x k − 1 = ρ − 1 x ^ k − 1 , \rho^{-k}x_k = \widehat{x}_k,\qquad\rho^{-k}x_{k-1} = \rho^{-1}\widehat{x}_{k-1}, ρ − k x k = x k , ρ − k x k − 1 = ρ − 1 x k − 1 ,
which gives (165) .
Proof of Lemma 5.5
Throughout, α = q 2 / M 2 \alpha=q^2/M^2 α = q 2 / M 2 , ν = M − 1 2 \nu=M-\frac{1}{2} ν = M − 2 1 , m = 1 2 ( 1 + q − 2 ) m=\frac{1}{2}(1+q^{-2}) m = 2 1 ( 1 + q − 2 ) , and δ m = α m \delta_m=\alpha m δ m = α m ; recall α , m , ν > 0 \alpha,m,\nu>0 α , m , ν > 0 and τ ≥ 1 \tau\geq1 τ ≥ 1 . Step 1: loop elimination and the denominator. Clearing the delay factor from the open-loop transfer gives
g τ , q ( z ) = N 0 ( z ) D 0 ( z ) , N 0 ( z ) = − α [ ( 1 + ν ) z − ν ] , D 0 ( z ) = z τ ( z − 1 ) ( z − β ) , g_{\tau,q}(z)=\frac{N_0(z)}{D_0(z)},\qquad N_0(z)=-\alpha\left[(1+\nu)z-\nu\right],\qquad D_0(z)=z^\tau(z-1)(z-\beta), g τ , q ( z ) = D 0 ( z ) N 0 ( z ) , N 0 ( z ) = − α [ ( 1 + ν ) z − ν ] , D 0 ( z ) = z τ ( z − 1 ) ( z − β ) ,
so the rate-weighted plant is g ρ ( z ) = g τ , q ( ρ z ) = N ( z ) / D ( z ) g_\rho(z)=g_{\tau,q}(\rho z)=N(z)/D(z) g ρ ( z ) = g τ , q ( ρ z ) = N ( z ) / D ( z ) with N ( z ) = N 0 ( ρ z ) N(z)=N_0(\rho z) N ( z ) = N 0 ( ρ z ) and D ( z ) = D 0 ( ρ z ) D(z)=D_0(\rho z) D ( z ) = D 0 ( ρ z ) . Substituting U ^ = m Y ^ + V ^ \widehat{U} =m\widehat{Y}+\widehat{V} U = m Y + V into Y ^ = g ρ U ^ \widehat{Y}=g_\rho\widehat{U} Y = g ρ U and collecting the Y ^ \widehat{Y} Y terms yields ( 1 − m g ρ ) Y ^ = g ρ V ^ (1-mg_\rho)\widehat{Y}=g_\rho\widehat{V} ( 1 − m g ρ ) Y = g ρ V . Note
1 − m g ρ = D − m N D , 1-mg_\rho=\frac{D-mN}{D}, 1 − m g ρ = D D − m N ,
D ( z ) − m N ( z ) = ( ρ z ) τ ( ρ z − 1 ) ( ρ z − β ) + δ m [ ( 1 + ν ) ρ z − ν ] = P τ , δ m ( ρ z ) . \begin{aligned}
D(z)-mN(z)&=(\rho z)^\tau(\rho z-1)(\rho z-\beta)+\delta_m\left[(1+\nu)\rho z-\nu\right]\\
&=P_{\tau,\delta_m}(\rho z).
\end{aligned} D ( z ) − m N ( z ) = ( ρ z ) τ ( ρ z − 1 ) ( ρ z − β ) + δ m [ ( 1 + ν ) ρ z − ν ] = P τ , δ m ( ρ z ) .
This is a non-zero polynomial of degree τ + 2 \tau+2 τ + 2 . Noting that the function 1 − m g ρ 1-mg_\rho 1 − m g ρ is not identically zero, we obtain
Y ^ = g ~ ρ V ^ , g ~ ρ = g ρ 1 − m g ρ = N D − m N . \widehat{Y}=\widetilde{g}_\rho\widehat{V},\qquad\widetilde{g}_\rho=\frac{g_\rho}{1-mg_\rho}=\frac{N}{D-mN}. Y = g ρ V , g ρ = 1 − m g ρ g ρ = D − m N N .
with denominator the rate-scaled midpoint characteristic polynomial P τ , δ m ( ρ ⋅ ) P_{\tau,\delta_m}(\rho\cdot) P τ , δ m ( ρ ⋅ ) . In particular the roots of D − m N D-mN D − m N are { r / ρ : P τ , δ m ( r ) = 0 } \{r/\rho:P_{\tau,\delta_m}(r)=0\} { r / ρ : P τ , δ m ( r ) = 0 } .
Step 2: no pole–zero cancellation. The numerator N 0 N_0 N 0 has the single finite zero
z ∗ = ν 1 + ν = M − 1 2 M + 1 2 = 2 M − 1 2 M + 1 ∈ ( 0 , 1 ) , z_\ast=\frac{\nu}{1+\nu}=\frac{M-\frac{1}{2}}{M+\frac{1}{2}}=\frac{2M-1}{2M+1}\in(0,1), z ∗ = 1 + ν ν = M + 2 1 M − 2 1 = 2 M + 1 2 M − 1 ∈ ( 0 , 1 ) ,
while the roots of D 0 D_0 D 0 are 0 0 0 , 1 1 1 , and β \beta β . Clearly z ∗ ≠ 0 z_\ast\neq0 z ∗ = 0 and z ∗ ≠ 1 z_\ast\neq1 z ∗ = 1 . The equality z ∗ = β = 1 − 2 q M z_\ast=\beta=1-\frac{2q}{M} z ∗ = β = 1 − M 2 q would force 2 q M = 1 − 2 M − 1 2 M + 1 = 2 2 M + 1 \frac{2q}{M}=1-\frac{2M-1}{2M+1}=\frac{2}{2M+1} M 2 q = 1 − 2 M + 1 2 M − 1 = 2 M + 1 2 , i.e. q = M 2 M + 1 q=\frac{M}{2M+1} q = 2 M + 1 M , which the standing hypothesis q < M 2 M + 1 q<\frac{M}{2M+1} q < 2 M + 1 M excludes. Hence gcd ( N 0 , D 0 ) = 1 \gcd(N_0,D_0)=1 g cd( N 0 , D 0 ) = 1 , and rescaling z ↦ ρ z z\mapsto\rho z z ↦ ρ z preserves coprimeness, so gcd ( N , D ) = 1 \gcd(N,D)=1 g cd( N , D ) = 1 . Finally, any common zero z 0 z_0 z 0 of N N N and D − m N D-mN D − m N would satisfy N ( z 0 ) = 0 N(z_0)=0 N ( z 0 ) = 0 and D ( z 0 ) = ( D ( z 0 ) − m N ( z 0 ) ) + m N ( z 0 ) = 0 D(z_0)=(D(z_0)-mN(z_0))+mN(z_0)=0 D ( z 0 ) = ( D ( z 0 ) − m N ( z 0 )) + m N ( z 0 ) = 0 , contradicting gcd ( N , D ) = 1 \gcd(N,D)=1 g cd( N , D ) = 1 ; therefore
gcd ( N , D − m N ) = 1. \gcd(N,D-mN)=1. g cd( N , D − m N ) = 1.
Thus g ~ ρ = N / ( D − m N ) \widetilde{g}_\rho=N/(D-mN) g ρ = N / ( D − m N ) is in lowest terms, and its poles are exactly the roots of D − m N = P τ , δ m ( ρ ⋅ ) D-mN=P_{\tau,\delta_m}(\rho\cdot) D − m N = P τ , δ m ( ρ ⋅ ) . Since deg N = 1 < τ + 2 = deg ( D − m N ) \deg N=1<\tau+2=\deg(D-mN) deg N = 1 < τ + 2 = deg ( D − m N ) , the transfer g ~ ρ \widetilde{g}_\rho g ρ is strictly proper and has no pole at infinity.
Step 3: spectrum of the realization. Collecting the terms of (164) by lag, the homogeneous x ^ \widehat{x} x -recursion reads
x ^ k + 1 = ∑ j = 0 τ + 1 a j x ^ k − j (276) \widehat{x}_{k+1}=\sum_{j=0}^{\tau+1}a_j\widehat{x}_{k-j} \tag*{(276)} x k + 1 = j = 0 ∑ τ + 1 a j x k − j (276)
with
a j = 1 + β ρ 1 { j = 0 } − β ρ 2 1 { j = 1 } − α m ( 1 + ν ) ρ τ + 1 1 { j = τ } + α m ν ρ τ + 2 1 { j = τ + 1 } , 0 ≤ j ≤ τ + 1. a_j=\frac{1+\beta}{\rho}\mathbf{1}_{\{j=0\}}-\frac{\beta}{\rho^2}\mathbf{1}_{\{j=1\}}-\frac{\alpha m(1+\nu)}{\rho^{\tau+1}}\mathbf{1}_{\{j=\tau\}}+\frac{\alpha m\nu}{\rho^{\tau+2}}\mathbf{1}_{\{j=\tau+1\}},\qquad0\leq j\leq\tau+1. a j = ρ 1 + β 1 { j = 0 } − ρ 2 β 1 { j = 1 } − ρ τ + 1 α m ( 1 + ν ) 1 { j = τ } + ρ τ + 2 α m ν 1 { j = τ + 1 } , 0 ≤ j ≤ τ + 1.
Note that, for τ = 1 \tau=1 τ = 1 , there are two terms that contribute to a 1 a_1 a 1 . Let F ∈ R ( τ + 2 ) × ( τ + 2 ) F\in\mathbb{R}^{(\tau+2)\times(\tau+2)} F ∈ R ( τ + 2 ) × ( τ + 2 ) be the scalar companion matrix of the recursion x ^ k + 1 = ∑ j = 0 τ + 1 a j x ^ k − j \hat{x}_{k+1}=\sum_{j=0}^{\tau+1}a_j\hat{x}_{k-j} x ^ k + 1 = ∑ j = 0 τ + 1 a j x ^ k − j , i.e., the matrix
F : = [ a 0 a 1 a 2 ⋯ a τ a τ + 1 1 0 0 ⋯ 0 0 0 1 0 ⋯ 0 0 ⋮ ⋱ ⋱ ⋱ ⋮ ⋮ 0 ⋯ 0 1 0 0 0 ⋯ 0 0 1 0 ] . F:=\begin{bmatrix}
a_0 & a_1 & a_2 & \cdots& a_\tau& a_{\tau+1}\\
1 & 0 & 0 & \cdots& 0 & 0\\
0 & 1 & 0 & \cdots& 0 & 0\\
\vdots& \ddots& \ddots& \ddots& \vdots& \vdots\\
0 & \cdots& 0 & 1 & 0 & 0\\
0 & \cdots& 0 & 0 & 1 & 0
\end{bmatrix}. F := a 0 1 0 ⋮ 0 0 a 1 0 1 ⋱ ⋯ ⋯ a 2 0 0 ⋱ 0 0 ⋯ ⋯ ⋯ ⋱ 1 0 a τ 0 0 ⋮ 0 1 a τ + 1 0 0 ⋮ 0 0 .
whose first row implements the recursion and whose subdiagonal of ones shifts the stored history. Expanding along the first row gives det ( z I − F ) = z τ + 2 − ∑ j = 0 τ + 1 a j z τ + 1 − j \det(zI-F)=z^{\tau+2}-\sum_{j=0}^{\tau+1}a_jz^{\tau+1-j} det ( z I − F ) = z τ + 2 − ∑ j = 0 τ + 1 a j z τ + 1 − j . Let E ∈ R ( τ + 2 ) × τ E\in\mathbb{R}^{(\tau+2)\times\tau} E ∈ R ( τ + 2 ) × τ have its only nonzero entry E 1 , τ = − α / ρ τ + 1 E_{1,\tau}=-\alpha/\rho^{\tau+1} E 1 , τ = − α / ρ τ + 1 .
Splitting the state as ξ k = ( X k T , W k T ) T \xi_k=(X_k^{\mathsf T},W_k^{\mathsf T})^{\mathsf T} ξ k = ( X k T , W k T ) T with X k = ( x ^ k , … , x ^ k − τ − 1 ) T X_k=(\hat{x}_k,\ldots,\hat{x}_{k-\tau-1})^{\mathsf T} X k = ( x ^ k , … , x ^ k − τ − 1 ) T and W k = ( v ^ k − 1 , … , v ^ k − τ ) T W_k=(\hat{v}_{k-1},\ldots,\hat{v}_{k-\tau})^{\mathsf T} W k = ( v ^ k − 1 , … , v ^ k − τ ) T , the state matrix is block upper triangular, and given by
A = A 0 ⊗ I d , A 0 = [ F E 0 J ] . A=A_0\otimes I_d,\qquad A_0=\begin{bmatrix}F&E\\0&J\end{bmatrix}. A = A 0 ⊗ I d , A 0 = [ F 0 E J ] .
where E E E records the delayed input v ^ k − τ \hat{v}_{k-\tau} v ^ k − τ in (165) , J J J is the τ × τ \tau\times\tau τ × τ down-shift with
W k + 1 = J W k + e 1 v ^ k , J : = [ 0 0 ⋯ 0 0 1 0 ⋯ 0 0 0 1 ⋱ ⋮ ⋮ ⋮ ⋱ ⋱ 0 0 0 ⋯ 0 1 0 ] ∈ R τ × τ . W_{k+1}=JW_k+e_1\hat{v}_k,\qquad J:=\begin{bmatrix}
0&0&\cdots&0&0\\
1&0&\cdots&0&0\\
0&1&\ddots&\vdots&\vdots\\
\vdots&\ddots&\ddots&0&0\\
0&\cdots&0&1&0
\end{bmatrix}\in\mathbb{R}^{\tau\times\tau}. W k + 1 = J W k + e 1 v ^ k , J := 0 1 0 ⋮ 0 0 0 1 ⋱ ⋯ ⋯ ⋯ ⋱ ⋱ 0 0 0 ⋮ 0 1 0 0 ⋮ 0 0 ∈ R τ × τ .
where e 1 ∈ R τ e_1\in\mathbb{R}^{\tau} e 1 ∈ R τ is the first basis vector. For the matrix A A A , the lower-left block vanishes because the stored inputs evolve independently of the x ^ \hat{x} x ^ -states. The matrix J J J is nilpotent with det ( z I τ + 2 − J ) = z τ \det(zI_{\tau+2}-J)=z^\tau det ( z I τ + 2 − J ) = z τ , so
det ( z I 2 τ + 2 − A 0 ) = z τ det ( z I τ + 2 − F ) , det ( z I ( 2 τ + 2 ) d − A ) = [ det ( z I 2 τ + 2 − A 0 ) ] d . \det(zI_{2\tau+2}-A_0)=z^\tau\det(zI_{\tau+2}-F),\qquad\det(zI_{(2\tau+2)d}-A)=\left[\det(zI_{2\tau+2}-A_0)\right]^d. det ( z I 2 τ + 2 − A 0 ) = z τ det ( z I τ + 2 − F ) , det ( z I ( 2 τ + 2 ) d − A ) = [ det ( z I 2 τ + 2 − A 0 ) ] d .
We conclude that the spectrum σ ( A ) = σ ( A 0 ) = σ ( F ) ∪ { 0 } \sigma(A)=\sigma(A_0)=\sigma(F)\cup\{0\} σ ( A ) = σ ( A 0 ) = σ ( F ) ∪ { 0 } .
For the companion block, det ( z I − F ) = z τ + 2 − ∑ j = 0 τ + 1 a j z τ + 1 − j \det(zI-F)=z^{\tau+2}-\sum_{j=0}^{\tau+1}a_jz^{\tau+1-j} det ( z I − F ) = z τ + 2 − ∑ j = 0 τ + 1 a j z τ + 1 − j . Multiplying by ρ τ + 2 \rho^{\tau+2} ρ τ + 2 and inserting the coefficients above gives
ρ τ + 2 det ( z I − F ) = ρ τ + 2 z τ + 2 − ( 1 + β ) ρ τ + 1 z τ + 1 + β ρ τ z τ + α m ( 1 + ν ) ρ z − α m ν = P τ , δ m ( ρ z ) , \rho^{\tau+2}\det(zI-F)=\rho^{\tau+2}z^{\tau+2}-(1+\beta)\rho^{\tau+1}z^{\tau+1}+\beta\rho^\tau z^\tau+\alpha m(1+\nu)\rho z-\alpha m\nu=P_{\tau,\delta_m}(\rho z), ρ τ + 2 det ( z I − F ) = ρ τ + 2 z τ + 2 − ( 1 + β ) ρ τ + 1 z τ + 1 + β ρ τ z τ + α m ( 1 + ν ) ρ z − α m ν = P τ , δ m ( ρ z ) ,
by expanding ( ρ z ) τ ( ρ z − 1 ) ( ρ z − β ) + δ m [ ( 1 + ν ) ρ z − ν ] (\rho z)^\tau(\rho z-1)(\rho z-\beta)+\delta_m[(1+\nu)\rho z-\nu] ( ρ z ) τ ( ρ z − 1 ) ( ρ z − β ) + δ m [( 1 + ν ) ρ z − ν ] . Thus det ( z I − F ) \det(zI-F) det ( z I − F ) equals D − m N D-mN D − m N up to the nonzero factor ρ − ( τ + 2 ) \rho^{-(\tau+2)} ρ − ( τ + 2 ) , and by Step 2 its roots are exactly the poles of g ~ ρ \widetilde{g}_\rho g ρ ; that is, σ ( F ) = P ( G ~ ) \sigma(F)=\mathcal{P}(\widetilde{G}) σ ( F ) = P ( G ) . Moreover P τ , δ m ( 0 ) = − δ m ν ≠ 0 P_{\tau,\delta_m}(0)=-\delta_m\nu\ne0 P τ , δ m ( 0 ) = − δ m ν = 0 , so 0 ∉ spec ( F ) 0\notin\operatorname{spec}(F) 0 ∈ / spec ( F ) and the union is disjoint:
σ ( A ) = σ ( A 0 ) = P ( G ~ ) ∪ { 0 } , \sigma(A)=\sigma(A_0)=\mathcal{P}(\widetilde{G})\cup\{0\}, σ ( A ) = σ ( A 0 ) = P ( G ) ∪ { 0 } ,
as sets. Each algebraic multiplicity in A 0 A_0 A 0 is multiplied by d d d in A A A .
Step 4: the Schur equivalences. Since 0 0 0 lies in the open unit disk, A A A is Schur if and only if P ( g ~ ρ ) ⊂ { ∣ z ∣ < 1 } \mathcal{P}(\widetilde{g}_\rho)\subset\{|z|<1\} P ( g ρ ) ⊂ { ∣ z ∣ < 1 } , i.e. if and only if g ~ ρ \widetilde{g}_\rho g ρ is Schur. By Steps 1–2 the poles of G ~ \widetilde{G} G are { r / ρ : P τ , δ m ( r ) = 0 } \{r/\rho:P_{\tau,\delta_m}(r)=0\} { r / ρ : P τ , δ m ( r ) = 0 } , and ∣ r / ρ ∣ < 1 ⟺ ∣ r ∣ < ρ |r/\rho|<1\Longleftrightarrow|r|<\rho ∣ r / ρ ∣ < 1 ⟺ ∣ r ∣ < ρ . Hence G ~ \widetilde{G} G (equivalently A A A ) is Schur if and only if every root of the midpoint characteristic polynomial P τ , δ m P_{\tau,\delta_m} P τ , δ m lies in { ∣ z ∣ < ρ } \{|z|<\rho\} { ∣ z ∣ < ρ } , which is the midpoint-pole hypothesis (H1).
Proof of Lemma 5.6
By the identity (168) , it suffices to show that 1 − m g ρ ( e i θ ) ≠ 0 1-mg_{\rho}(e^{\mathrm{i}\theta}) \ne0 1 − m g ρ ( e i θ ) = 0 for all θ \theta θ ; multiplication by the resulting positive scalar ∣ 1 − m g ρ ∣ − 2 \lvert1-mg_{\rho}\rvert^{-2} ∣ 1 − m g ρ ∣ − 2 cannot change the sign of the quadratic form, and the equivalence of the two strict FDIs follows.
First, g ρ g_{\rho} g ρ is finite on the unit circle: with g τ , q = N 0 / D 0 g_{\tau,q}=N_{0}/D_{0} g τ , q = N 0 / D 0 in lowest terms, the poles of g ρ ( z ) = g τ , q ( ρ z ) g_{\rho}(z)=g_{\tau,q}(\rho z) g ρ ( z ) = g τ , q ( ρ z ) lie at z ∈ { 0 , 1 / ρ , β / ρ } z \in\{0,1/\rho,\beta/\rho\} z ∈ { 0 , 1/ ρ , β / ρ } , none of modulus one, since 0 < β = 1 − 2 q / M < 1 − c q / M = ρ < 1 0<\beta=1-2q/M<1-cq/M=\rho<1 0 < β = 1 − 2 q / M < 1 − c q / M = ρ < 1 and ρ ≠ β \rho\ne\beta ρ = β . Next, from
1 − m g ρ ( z ) = D 0 ( ρ z ) − m N 0 ( ρ z ) D 0 ( ρ z ) = P m ( ρ z ) D 0 ( ρ z ) , P m : = D 0 − m N 0 , 1-mg_{\rho}(z)=\frac{D_{0}(\rho z)-mN_{0}(\rho z)}{D_{0}(\rho z)}=\frac{P_{m}(\rho z)}{D_{0}(\rho z)},\qquad P_{m}:=D_{0}-mN_{0}, 1 − m g ρ ( z ) = D 0 ( ρ z ) D 0 ( ρ z ) − m N 0 ( ρ z ) = D 0 ( ρ z ) P m ( ρ z ) , P m := D 0 − m N 0 ,
a zero 1 − m g ρ ( e i θ ) = 0 1-mg_{\rho}(e^{\mathrm{i}\theta})=0 1 − m g ρ ( e i θ ) = 0 would force P m ( ρ e i θ ) = 0 P_{m}(\rho e^{\mathrm{i}\theta})=0 P m ( ρ e i θ ) = 0 , i.e. a root of the midpoint characteristic polynomial of modulus exactly ρ \rho ρ . This contradicts (H1), which places every root of P m P_{m} P m in { ∣ z ∣ < ρ } \{\lvert z\rvert<\rho\} {∣ z ∣ < ρ } . Hence 1 − m g ρ ( e i θ ) ≠ 0 1-mg_{\rho}(e^{\mathrm{i}\theta})\ne0 1 − m g ρ ( e i θ ) = 0 for all θ ∈ [ 0 , 2 π ) \theta\in[0,2\pi) θ ∈ [ 0 , 2 π ) , and the claim follows.
Proof of Lemma 5.7. The proof has five steps. The main point is to convert the indefinite supply into the standard bounded-real supply by a causal, stably invertible change of input. Step 1: Exact factorization of the supply. Set w : = 1 − h w:=1-h w := 1 − h and define
e + , k : = r ^ k + ( w s ^ ) k , e − , k : = r ^ k − ( w s ^ ) k . e_{+,k}:=\hat r_k+(w\hat s)_k,\qquad e_{-,k}:=\hat r_k-(w\hat s)_k. e + , k := r ^ k + ( w s ^ ) k , e − , k := r ^ k − ( w s ^ ) k .
Since
( w s ^ ) k = s ^ k − ( h s ^ ) k = s ^ k − ∑ j = 1 J c j s ^ k − j , (w\hat s)_k=\hat s_k-(h\hat s)_k=\hat s_k-\sum_{j=1}^{J}c_j\hat s_{k-j}, ( w s ^ ) k = s ^ k − ( h s ^ ) k = s ^ k − j = 1 ∑ J c j s ^ k − j ,
the polarization identity
∥ p + q ∥ 2 − ∥ p − q ∥ 2 = 4 Re ⟨ p , q ⟩ \lVert p+q\rVert^2-\lVert p-q\rVert^2=4\operatorname{Re}\langle p,q\rangle ∥ p + q ∥ 2 − ∥ p − q ∥ 2 = 4 Re ⟨ p , q ⟩
yields, for every k k k ,
1 2 ( ∥ e + , k ∥ 2 − ∥ e − , k ∥ 2 ) = 2 Re ⟨ r ^ k , ( w s ^ ) k ⟩ = 2 Re ⟨ r ^ k , s ^ k ⟩ − 2 ∑ j = 1 J c j Re ⟨ r ^ k , s ^ k − j ⟩ = σ k . (277) \begin{aligned}
\frac{1}{2}\left(\lVert e_{+,k}\rVert^2-\lVert e_{-,k}\rVert^2\right)
&=2\operatorname{Re}\langle\hat r_k,(w\hat s)_k\rangle\\
&=2\operatorname{Re}\langle\hat r_k,\hat s_k\rangle-2\sum_{j=1}^{J}c_j\operatorname{Re}\langle\hat r_k,\hat s_{k-j}\rangle\\
&=\sigma_k. \tag*{(277)}
\end{aligned} 2 1 ( ∥ e + , k ∥ 2 − ∥ e − , k ∥ 2 ) = 2 Re ⟨ r ^ k , ( w s ^ ) k ⟩ = 2 Re ⟨ r ^ k , s ^ k ⟩ − 2 j = 1 ∑ J c j Re ⟨ r ^ k , s ^ k − j ⟩ = σ k . (277)
Thus the original supply is exactly the difference-of-squares supply associated with ( e + , e − ) (e_+,e_-) ( e + , e − ) . No summation in time, boundary term, or zero-initial-condition assumption is involved in (277) .
Using r ^ = a y ^ + v ^ \hat r=a\hat y+\hat v r ^ = a y ^ + v ^ and s ^ = a y ^ − v ^ \hat s=a\hat y-\hat v s ^ = a y ^ − v ^ gives
e + = r ^ + s ^ − h s ^ = a ( 2 − h ) y ^ + h v ^ , e − = r ^ − s ^ + h s ^ = a h y ^ + ( 2 − h ) v ^ . \begin{aligned}
e_+&=\hat r+\hat s-h\hat s=a(2-h)\hat y+h\hat v,\\
e_-&=\hat r-\hat s+h\hat s=ah\hat y+(2-h)\hat v.
\end{aligned} e + e − = r ^ + s ^ − h s ^ = a ( 2 − h ) y ^ + h v ^ , = r ^ − s ^ + h s ^ = ah y ^ + ( 2 − h ) v ^ .
Hence
\begin{bmatrix}e_+\\e_-\end{bmatrix}
=\Psi_d\begin{bmatrix}\hat y\\\hat v\end{bmatrix}. \tag*{(278)}\end{bmatrix}
For zero initial state, y ^ = G v ^ \hat y=\mathcal{G}\hat v y ^ = G v ^ , and therefore the transfer matrices from v ^ \hat v v ^ to e + e_+ e + and e − e_- e − are
H + : = a ( 2 − h ) G + h I d , H − : = a h G + ( 2 − h ) I d . (279) H_+:=a(2-h)\mathcal{G}+hI_d,\qquad H_-:=ah\mathcal{G}+(2-h)I_d. \tag*{(279)} H + := a ( 2 − h ) G + h I d , H − := ah G + ( 2 − h ) I d . (279)
On the unit circle, (278) implies
[ G I d ] ∗ Π ~ d [ G I d ] = 1 2 [ H + H − ] ∗ J 0 , d [ H + H − ] = 1 2 ( H + ∗ H + − H − ∗ H − ) . \begin{bmatrix} \mathcal{G} \\ I_d \end{bmatrix}^{*}
\widetilde{\Pi}_d
\begin{bmatrix} \mathcal{G} \\ I_d \end{bmatrix}
=
\frac{1}{2}
\begin{bmatrix} H_{+} \\ H_{-} \end{bmatrix}^{*}
J_{0,d}
\begin{bmatrix} H_{+} \\ H_{-} \end{bmatrix}
=
\frac{1}{2}\left(H_{+}^{*}H_{+}-H_{-}^{*}H_{-}\right). [ G I d ] ∗ Π d [ G I d ] = 2 1 [ H + H − ] ∗ J 0 , d [ H + H − ] = 2 1 ( H + ∗ H + − H − ∗ H − ) .
Consequently, (172) is equivalent to
H + ( e i θ ) ∗ H + ( e i θ ) − H − ( e i θ ) ∗ H − ( e i θ ) ⪯ − 2 η I d (280) H_{+}(e^{i\theta})^{*}H_{+}(e^{i\theta})-H_{-}(e^{i\theta})^{*}H_{-}(e^{i\theta})\preceq-2\eta I_d \tag*{(280)} H + ( e i θ ) ∗ H + ( e i θ ) − H − ( e i θ ) ∗ H − ( e i θ ) ⪯ − 2 η I d (280)
for every θ \theta θ .
Step 2: Stable invertibility of H − H_{-} H − . We show that H − − 1 H_{-}^{-1} H − − 1 exists as a stable, proper rational transfer matrix. Since H − H_{-} H − is itself stable and proper, it suffices to establish that H − ( z ) H_{-}(z) H − ( z ) is nonsingular for every ∣ z ∣ ≥ 1 |z|\ge1 ∣ z ∣ ≥ 1 , including at the infinity limit. For t ∈ [ 0 , 1 ] t\in[0,1] t ∈ [ 0 , 1 ] , define the homotopy
H + ( t ) : = t a ( 2 − h ) G + h I d , H − ( t ) : = t a h G + ( 2 − h ) I d , H_{+}^{(t)}:=ta(2-h)\mathcal{G}+hI_d,\qquad H_{-}^{(t)}:=tah\mathcal{G}+(2-h)I_d, H + ( t ) := t a ( 2 − h ) G + h I d , H − ( t ) := t ah G + ( 2 − h ) I d ,
and, for fixed θ \theta θ ,
Φ t ( θ ) : = [ t G ( e i θ ) I d ] ∗ Π ~ d ( e i θ ) [ t G ( e i θ ) I d ] . \Phi_t(\theta):=
\begin{bmatrix}
t\mathcal{G}(e^{i\theta})\\
I_d
\end{bmatrix}^{*}
\widetilde{\Pi}_d(e^{i\theta})
\begin{bmatrix}
t\mathcal{G}(e^{i\theta})\\
I_d
\end{bmatrix}. Φ t ( θ ) := [ t G ( e i θ ) I d ] ∗ Π d ( e i θ ) [ t G ( e i θ ) I d ] .
Partition Π ~ d = [ ( Π ~ d ) i j ] i , j = 1 2 \widetilde{\Pi}_d=[(\widetilde{\Pi}_d)_{ij}]_{i,j=1}^{2} Π d = [( Π d ) ij ] i , j = 1 2 into d × d d\times d d × d blocks. Directly from Π ~ d = 1 2 Ψ d ∼ J 0 , d Ψ d \widetilde{\Pi}_d=\frac{1}{2}\Psi_d^{\sim}J_{0,d}\Psi_d Π d = 2 1 Ψ d ∼ J 0 , d Ψ d ,
( Π ~ d ) 11 = 2 a 2 Re ( 1 − h ) I d , ( Π ~ d ) 22 = − 2 Re ( 1 − h ) I d , (281) (\widetilde{\Pi}_d)_{11}=2a^2\operatorname{Re}(1-h)I_d,\qquad
(\widetilde{\Pi}_d)_{22}=-2\operatorname{Re}(1-h)I_d, \tag*{(281)} ( Π d ) 11 = 2 a 2 Re ( 1 − h ) I d , ( Π d ) 22 = − 2 Re ( 1 − h ) I d , (281)
on the unit circle. Moreover,
Re ( 1 − h ( e i θ ) ) ≥ 1 − ∣ h ( e i θ ) ∣ ≥ 1 − s h > 0. \operatorname{Re}(1-h(e^{i\theta}))\ge1-|h(e^{i\theta})|\ge1-s_h>0. Re ( 1 − h ( e i θ )) ≥ 1 − ∣ h ( e i θ ) ∣ ≥ 1 − s h > 0.
Thus ( Π ~ d ) 11 ( e i θ ) ≻ 0 (\widetilde{\Pi}_d)_{11}(e^{i\theta})\succ0 ( Π d ) 11 ( e i θ ) ≻ 0 and ( Π ~ d ) 22 ( e i θ ) ≺ 0 (\widetilde{\Pi}_d)_{22}(e^{i\theta})\prec0 ( Π d ) 22 ( e i θ ) ≺ 0 for every θ \theta θ . For any fixed u ∈ C d u\in\mathbb{C}^{d} u ∈ C d , the scalar function
f θ , u ( t ) : = u ∗ Φ t ( θ ) u f_{\theta,u}(t):=u^{*}\Phi_t(\theta)u f θ , u ( t ) := u ∗ Φ t ( θ ) u
is a real quadratic polynomial whose t 2 t^2 t 2 -coefficient is u ∗ G ∗ ( Π ~ d ) 11 G u ≥ 0 u^{*}\mathcal{G}^{*}(\widetilde{\Pi}_d)_{11}\mathcal{G}u\ge0 u ∗ G ∗ ( Π d ) 11 G u ≥ 0 . Hence f θ , u f_{\theta,u} f θ , u is convex. Since
Φ 0 ( θ ) = ( Π ~ d ) 22 ( e i θ ) ≺ 0 , Φ 1 ( θ ) ⪯ − η I d ≺ 0 , \Phi_0(\theta)=(\widetilde{\Pi}_d)_{22}(e^{i\theta})\prec0,\qquad
\Phi_1(\theta)\preceq-\eta I_d\prec0, Φ 0 ( θ ) = ( Π d ) 22 ( e i θ ) ≺ 0 , Φ 1 ( θ ) ⪯ − η I d ≺ 0 ,
convexity gives, for 0 ≤ t ≤ 1 0\le t\le1 0 ≤ t ≤ 1 ,
u ∗ Φ t ( θ ) u ≤ ( 1 − t ) u ∗ Φ 0 ( θ ) u + t u ∗ Φ 1 ( θ ) u < 0 u^{*}\Phi_t(\theta)u\le(1-t)u^{*}\Phi_0(\theta)u+tu^{*}\Phi_1(\theta)u<0 u ∗ Φ t ( θ ) u ≤ ( 1 − t ) u ∗ Φ 0 ( θ ) u + t u ∗ Φ 1 ( θ ) u < 0
whenever u ≠ 0 u\ne0 u = 0 . Therefore
Φ t ( θ ) ≺ 0 , t ∈ [ 0 , 1 ] , 0 ≤ θ < 2 π . (282) \Phi_t(\theta)\prec0,\qquad t\in[0,1],\quad0\le\theta<2\pi. \tag*{(282)} Φ t ( θ ) ≺ 0 , t ∈ [ 0 , 1 ] , 0 ≤ θ < 2 π . (282)
As in Step 1,
Φ t ( θ ) = 1 2 ( H + ( t ) ( e i θ ) ∗ H + ( t ) ( e i θ ) − H − ( t ) ( e i θ ) ∗ H − ( t ) ( e i θ ) ) . (283) \Phi_t(\theta)=\frac{1}{2}\left(H_{+}^{(t)}(e^{i\theta})^{*}H_{+}^{(t)}(e^{i\theta})-H_{-}^{(t)}(e^{i\theta})^{*}H_{-}^{(t)}(e^{i\theta})\right). \tag*{(283)} Φ t ( θ ) = 2 1 ( H + ( t ) ( e i θ ) ∗ H + ( t ) ( e i θ ) − H − ( t ) ( e i θ ) ∗ H − ( t ) ( e i θ ) ) . (283)
It follows that H − ( t ) ( e i θ ) H_{-}^{(t)}(e^{i\theta}) H − ( t ) ( e i θ ) is nonsingular for every ( t , θ ) (t,\theta) ( t , θ ) . Indeed, if H − ( t ) ( e i θ ) u = 0 H_{-}^{(t)}(e^{i\theta})u=0 H − ( t ) ( e i θ ) u = 0 with u ≠ 0 u\ne0 u = 0 , then (283) would imply
u ∗ Φ t ( θ ) u = 1 2 ∥ H + ( t ) ( e i θ ) u ∥ 2 ≥ 0 , u^{*}\Phi_t(\theta)u=\frac{1}{2}\left\|H_{+}^{(t)}(e^{i\theta})u\right\|^{2}\ge0, u ∗ Φ t ( θ ) u = 2 1 H + ( t ) ( e i θ ) u 2 ≥ 0 ,
contrary to (282) .
We now exclude zeros in the whole exterior disc, not merely on its boundary. Because G \mathcal{G} G is stable and proper, it is analytic on a neighborhood of { z : ∣ z ∣ ≥ 1 } ∪ { ∞ } \{z: |z| \ge1\} \cup\{\infty\} { z : ∣ z ∣ ≥ 1 } ∪ { ∞ } . Introduce λ = z − 1 \lambda= z^{-1} λ = z − 1 and define
H ^ − ( t ) ( λ ) : = { H − ( t ) ( λ − 1 ) , λ ≠ 0 , 2 I d , λ = 0. \widehat{H}_{-}^{(t)}(\lambda) := \begin{cases}
H^{(t)}_{-}(\lambda^{-1}), & \lambda\ne0,\\
2I_d, & \lambda= 0.
\end{cases} H − ( t ) ( λ ) := { H − ( t ) ( λ − 1 ) , 2 I d , λ = 0 , λ = 0.
The value at λ = 0 \lambda= 0 λ = 0 is the removable limit because h ( ∞ ) = 0 h(\infty) = 0 h ( ∞ ) = 0 ; hence H ^ − ( t ) \widehat{H}_{-}^{(t)} H − ( t ) is analytic on a neighborhood of the closed unit disc D ‾ \overline{\mathbb{D}} D , uniformly in t ∈ [ 0 , 1 ] t \in[0,1] t ∈ [ 0 , 1 ] . Set
d t ( λ ) : = det H ^ − ( t ) ( λ ) . d_t(\lambda) := \det\widehat{H}_{-}^{(t)}(\lambda). d t ( λ ) := det H − ( t ) ( λ ) .
By the preceding paragraph, d t ( λ ) ≠ 0 d_t(\lambda) \ne0 d t ( λ ) = 0 for ∣ λ ∣ = 1 |\lambda| = 1 ∣ λ ∣ = 1 , for every t t t . Therefore the winding number
N ( t ) : = 1 2 π i ∫ ∣ λ ∣ = 1 d t ′ ( λ ) d t ( λ ) d λ N(t) := \frac{1}{2\pi i}\int_{|\lambda|=1}\frac{d_t'(\lambda)}{d_t(\lambda)}\,d\lambda N ( t ) := 2 π i 1 ∫ ∣ λ ∣ = 1 d t ( λ ) d t ′ ( λ ) d λ
is well-defined. By the argument principle, N ( t ) N(t) N ( t ) is the number of zeros of d t d_t d t in D \mathbb{D} D , counted with multiplicity. Since ( t , λ ) ↦ d t ( λ ) (t,\lambda) \mapsto d_t(\lambda) ( t , λ ) ↦ d t ( λ ) is continuous and never vanishes on [ 0 , 1 ] × ∂ D [0,1] \times\partial\mathbb{D} [ 0 , 1 ] × ∂ D , the integer-valued function N ( t ) N(t) N ( t ) is constant.
At t = 0 t = 0 t = 0 ,
H ^ − ( 0 ) ( λ ) = ( 2 − ∑ j = 1 J c j λ j ) I d . \widehat{H}_{-}^{(0)}(\lambda) = \left(2-\sum_{j=1}^{J}c_j\lambda^j\right)I_d. H − ( 0 ) ( λ ) = ( 2 − j = 1 ∑ J c j λ j ) I d .
For ∣ λ ∣ ≤ 1 |\lambda| \le1 ∣ λ ∣ ≤ 1 ,
∣ ∑ j = 1 J c j λ j ∣ ≤ s h < 1 , \left|\sum_{j=1}^{J}c_j\lambda^j\right| \le s_h < 1, j = 1 ∑ J c j λ j ≤ s h < 1 ,
so H ^ − ( 0 ) ( λ ) \widehat{H}_{-}^{(0)}(\lambda) H − ( 0 ) ( λ ) is nonsingular throughout D ‾ \overline{\mathbb{D}} D . Hence N ( 0 ) = 0 N(0) = 0 N ( 0 ) = 0 , and therefore N ( t ) = 0 N(t) = 0 N ( t ) = 0 for all t t t . Together with nonsingularity on ∂ D \partial\mathbb{D} ∂ D and the identity H ^ − ( t ) ( 0 ) = 2 I d \widehat{H}_{-}^{(t)}(0) = 2I_d H − ( t ) ( 0 ) = 2 I d , this shows that
det H − ( t ) ( z ) ≠ 0 for every ∣ z ∣ ≥ 1 , \det H_{-}^{(t)}(z) \ne0 \qquad\text{for every } |z| \ge1, det H − ( t ) ( z ) = 0 for every ∣ z ∣ ≥ 1 ,
and that there is no zero at infinity. In particular,
det H − ( z ) ≠ 0 ( ∣ z ∣ ≥ 1 ) , H − ( ∞ ) = 2 I d . (284) \det H_{-}(z) \ne0 \quad(|z| \ge1), \qquad H_{-}(\infty) = 2I_d. \tag*{(284)} det H − ( z ) = 0 ( ∣ z ∣ ≥ 1 ) , H − ( ∞ ) = 2 I d . (284)
Since H − H_{-} H − is stable and proper, these conditions imply that H − − 1 H_{-}^{-1} H − − 1 is also stable and proper.
*Step 3: Augmented realization and the change of input v ^ ↦ e − \widehat{v} \mapsto e_{-} v ↦ e − . We use the Schur realization ( A , B , C , D ) (A,B,C,D) ( A , B , C , D ) of G \mathcal{G} G specified in (170) :
ξ ^ k + 1 = A ξ ^ k + B v ^ k , y ^ k = C ξ ^ k + D v ^ k , (285) \begin{aligned}
\widehat{\xi}_{k+1} &= A\widehat{\xi}_k + B\widehat{v}_k, \\
\widehat{y}_k &= C\widehat{\xi}_k + D\widehat{v}_k, \tag*{(285)}
\end{aligned} ξ k + 1 y k = A ξ k + B v k , = C ξ k + D v k , (285)
with A A A Schur. Since
s ^ k = a C ξ ^ k + ( a D − I d ) v ^ k , ( h s ^ ) k = ∑ j = 1 J c j s ^ k − j , \widehat{s}_k = aC\widehat{\xi}_k + (aD-I_d)\widehat{v}_k, \qquad(h\widehat{s})_k = \sum_{j=1}^{J}c_j\widehat{s}_{k-j}, s k = a C ξ k + ( a D − I d ) v k , ( h s ) k = j = 1 ∑ J c j s k − j ,
augment ξ k \xi_k ξ k by the filter states
q k : = [ s ^ k − 1 s ^ k − 2 ⋮ s ^ k − J ] , Ξ k : = [ ξ k q k ] . q_k := \begin{bmatrix} \hat{s}_{k-1} \\ \hat{s}_{k-2} \\ \vdots\\ \hat{s}_{k-J} \end{bmatrix}, \qquad\Xi_k := \begin{bmatrix} \xi_k \\ q_k \end{bmatrix}. q k := s ^ k − 1 s ^ k − 2 ⋮ s ^ k − J , Ξ k := [ ξ k q k ] .
Let
L h : = [ c 1 I d c 2 I ⋯ c J I d ] , S : = [ 0 0 ⋯ 0 I d 0 ⋯ 0 0 I d ⋱ ⋮ ⋮ ⋱ ⋱ 0 ] , E : = [ I d 0 ⋮ 0 ] . L_h := \begin{bmatrix} c_1 I_d & c_2 I & \cdots& c_J I_d \end{bmatrix}, \qquad S := \begin{bmatrix} 0 & 0 & \cdots& 0 \\ I_d & 0 & \cdots& 0 \\ 0 & I_d & \ddots& \vdots\\ \vdots& \ddots& \ddots& 0 \end{bmatrix}, \qquad E := \begin{bmatrix} I_d \\ 0 \\ \vdots\\ 0 \end{bmatrix}. L h := [ c 1 I d c 2 I ⋯ c J I d ] , S := 0 I d 0 ⋮ 0 0 I d ⋱ ⋯ ⋯ ⋱ ⋱ 0 0 ⋮ 0 , E := I d 0 ⋮ 0 .
Here L h ∈ C d × J d L_h \in\mathbb{C}^{d \times Jd} L h ∈ C d × J d , S ∈ C J d × J d S \in\mathbb{C}^{Jd \times Jd} S ∈ C J d × J d , E ∈ C J d × d E \in\mathbb{C}^{Jd \times d} E ∈ C J d × d , and each displayed zero block is d × d d \times d d × d . Then ( h s ^ ) k = L h q k (h\hat{s})_k = L_hq_k ( h s ^ ) k = L h q k and
q k + 1 = S q k + E s ^ k = S q k + E a C ξ k + E ( a D − I d ) v ^ k . q_{k+1} = Sq_k + E\hat{s}_k = Sq_k + EaC\xi_k + E(aD-I_d)\hat{v}_k. q k + 1 = S q k + E s ^ k = S q k + E a C ξ k + E ( a D − I d ) v ^ k .
Thus
Ξ k + 1 = A a Ξ k + B a v ^ k , (286) \Xi_{k+1} = A_a\Xi_k + B_a\hat{v}_k, \tag*{(286)} Ξ k + 1 = A a Ξ k + B a v ^ k , (286)
where
A a = [ A 0 E a C S ] , B a = [ B E ( a D − I d ) ] . (287) A_a = \begin{bmatrix} A & 0 \\ EaC & S \end{bmatrix}, \qquad B_a = \begin{bmatrix} B \\ E(aD-I_d) \end{bmatrix}. \tag*{(287)} A a = [ A E a C 0 S ] , B a = [ B E ( a D − I d ) ] . (287)
The matrices A a A_a A a and B a B_a B a have dimensions n a × n a n_a \times n_a n a × n a and n a × d n_a \times d n a × d , respectively. Because A a A_a A a is block lower triangular with diagonal blocks A A A and the nilpotent matrix S S S , it is Schur.
From Step 1,
e + = 2 a y ^ − ( h s ^ ) , e − = 2 v ^ + ( h s ^ ) . e_+ = 2a\hat{y} - (h\hat{s}), \qquad e_- = 2\hat{v} + (h\hat{s}). e + = 2 a y ^ − ( h s ^ ) , e − = 2 v ^ + ( h s ^ ) .
Therefore
e + , k = C + Ξ k + D + v ^ k , e − , k = C − Ξ k + D − v ^ k , (288) e_{+,k} = C_+\Xi_k + D_+\hat{v}_k, \qquad e_{-,k} = C_-\Xi_k + D_-\hat{v}_k, \tag*{(288)} e + , k = C + Ξ k + D + v ^ k , e − , k = C − Ξ k + D − v ^ k , (288)
with
C + = [ 2 a C − L h ] , D + = 2 a D , C_+ = \begin{bmatrix} 2aC & -L_h \end{bmatrix}, \qquad D_+ = 2aD, C + = [ 2 a C − L h ] , D + = 2 a D ,
C − = [ 0 L h ] , D − = 2 I d . (289) C_- = \begin{bmatrix} 0 & L_h \end{bmatrix}, \qquad D_- = 2I_d. \tag*{(289)} C − = [ 0 L h ] , D − = 2 I d . (289)
Here C ± ∈ C d × n a C_{\pm} \in\mathbb{C}^{d \times n_a} C ± ∈ C d × n a and D ± ∈ C d × d D_{\pm} \in\mathbb{C}^{d \times d} D ± ∈ C d × d . In particular, D − D_- D − is invertible. Hence, pointwise in time,
v ^ k = D − − 1 ( e − , k − C − Ξ k ) . (290) \hat{v}_k = D_-^{-1}(e_{-,k} - C_-\Xi_k). \tag*{(290)} v ^ k = D − − 1 ( e − , k − C − Ξ k ) . (290)
Thus replacing v ^ k \hat{v}_k v ^ k by e − , k e_{-,k} e − , k is an invertible static change of input coordinates on the augmented state space. Substitution of (290) into (286) and (288) yields the equivalent realization
Ξ ˉ k + 1 = A ˉ Ξ ˉ k + B ˉ e − , k , e + , k = C ˉ Ξ ˉ k + D ˉ e − , k , (291) \bar{\Xi}_{k+1} = \bar{A}\bar{\Xi}_k + \bar{B}e_{-,k}, \qquad e_{+,k} = \bar{C}\bar{\Xi}_k + \bar{D}e_{-,k}, \tag*{(291)} Ξ ˉ k + 1 = A ˉ Ξ ˉ k + B ˉ e − , k , e + , k = C ˉ Ξ ˉ k + D ˉ e − , k , (291)
where
A ˉ = A a − B a D − − 1 C − , B ˉ = B a D − − 1 , \bar{A} = A_a - B_aD_-^{-1}C_-, \qquad\bar{B} = B_aD_-^{-1}, A ˉ = A a − B a D − − 1 C − , B ˉ = B a D − − 1 ,
C ˉ = C + − D + D − − 1 C − , D ˉ = D + D − − 1 . (292) \bar{C} = C_+ - D_+D_-^{-1}C_-, \qquad\bar{D} = D_+D_-^{-1}. \tag*{(292)} C ˉ = C + − D + D − − 1 C − , D ˉ = D + D − − 1 . (292)
Step 4: Stability of the transformed realization and strict bounded realness. For zero initial state,
e + = H + v ^ , e − = H − v ^ . e_+ = H_+\hat{v}, \qquad e_- = H_-\hat{v}. e + = H + v ^ , e − = H − v ^ .
By (284) , H − − 1 H_-^{-1} H − − 1 is stable and proper. Hence the transfer from the new input e − e_- e − to the output e + e_+ e + is
K = H + H − − 1 . (293) K = H_+H_-^{-1}. \tag*{(293)} K = H + H − − 1 . (293)
The realization in (291) has precisely this transfer.
We next verify that the state matrix is Schur stable. This does not follow from stability of the transfer matrix alone, since some eigenvalues of the state matrix may not appear as poles of the transfer matrix.10 For ∣ z ∣ ≥ 1 |z| \ge1 ∣ z ∣ ≥ 1 , z I n a − A a zI_{n_a}-A_a z I n a − A a is invertible where n a = n + J d n_a=n+Jd n a = n + J d . Using (292) , the determinant identity and Sylvester’s determinant theorem give
det ( z I n a − A ˉ ) = det ( z I n a − A a ) det ( I d + D − 1 C − ( z I n a − A a ) − 1 B a ) = det ( z I n a − A a ) det ( D − 1 H − ( z ) ) . (294) \begin{aligned}
\det(zI_{n_a}-\bar{A})&=\det(zI_{n_a}-A_a)\det\left(I_d+D^{-1}C_-(zI_{n_a}-A_a)^{-1}B_a\right)\\
&=\det(zI_{n_a}-A_a)\det\left(D^{-1}H_-(z)\right). \tag*{(294)}
\end{aligned} det ( z I n a − A ˉ ) = det ( z I n a − A a ) det ( I d + D − 1 C − ( z I n a − A a ) − 1 B a ) = det ( z I n a − A a ) det ( D − 1 H − ( z ) ) . (294)
The first factor in (294) is nonzero because A a A_a A a is Schur; the second is nonzero by (284) . Hence det ( z I n a − A ˉ ) ≠ 0 \det(zI_{n_a}-\bar{A})\ne0 det ( z I n a − A ˉ ) = 0 for every ∣ z ∣ ≥ 1 |z|\ge1 ∣ z ∣ ≥ 1 , so A ˉ \bar{A} A ˉ is Schur.
Finally, rewrite (280) as
H − ∗ H − − H + ∗ H + ⪰ 2 η I d H_-^*H_- - H_+^*H_+ \succeq2\eta I_d H − ∗ H − − H + ∗ H + ⪰ 2 η I d
on the unit circle and pre- and post-multiply by H − − ∗ H_-^{-*} H − −∗ and H − − 1 H_-^{-1} H − − 1 . Using (293) ,
I d − K ( e i θ ) ∗ K ( e i θ ) ⪰ 2 η H − ( e i θ ) − ∗ H − ( e i θ ) − 1 ≻ 0. (295) I_d-K(e^{i\theta})^*K(e^{i\theta})\succeq2\eta H_-(e^{i\theta})^{-*}H_-(e^{i\theta})^{-1}\succ0. \tag*{(295)} I d − K ( e i θ ) ∗ K ( e i θ ) ⪰ 2 η H − ( e i θ ) −∗ H − ( e i θ ) − 1 ≻ 0. (295)
Since the unit circle is compact and H − H_- H − is continuous and nonsingular there, (295) has a uniform positive margin. In particular,
∥ K ∥ H ∞ : = sup θ ∥ K ( e i θ ) ∥ < 1. \|K\|_{H^\infty}:=\sup_\theta\|K(e^{i\theta})\|<1. ∥ K ∥ H ∞ := θ sup ∥ K ( e i θ ) ∥ < 1.
We invoke the Kalman–Yakubovich–Popov (KYP) lemma in the form of the strict discrete-time bounded-real lemma [16 Theorem 2.2(b)] : if A ˉ \bar{A} A ˉ is Schur and K ( z ) = C ˉ ( z I − A ˉ ) − 1 B ˉ + D ˉ K(z)=\bar{C}(zI-\bar{A})^{-1}\bar{B}+\bar{D} K ( z ) = C ˉ ( z I − A ˉ ) − 1 B ˉ + D ˉ satisfies ∥ K ∥ H ∞ < 1 \|K\|_{H^\infty}<1 ∥ K ∥ H ∞ < 1 , then there exists a Hermitian matrix X 0 X_0 X 0 such that
R 0 : = B ˉ ∗ X 0 B ˉ + D ˉ ∗ D ˉ − I d ≺ 0 (296) R_0:=\bar{B}^*X_0\bar{B}+\bar{D}^*\bar{D}-I_d\prec0 \tag*{(296)} R 0 := B ˉ ∗ X 0 B ˉ + D ˉ ∗ D ˉ − I d ≺ 0 (296)
and
A ˉ ∗ X 0 A ˉ − X 0 + C ˉ ∗ C ˉ − ( A ˉ ∗ X 0 B ˉ + C ˉ ∗ D ˉ ) R 0 − 1 ( B ˉ ∗ X 0 A ˉ + D ˉ ∗ C ˉ ) ≺ 0. (297) \bar{A}^*X_0\bar{A}-X_0+\bar{C}^*\bar{C}-(\bar{A}^*X_0\bar{B}+\bar{C}^*\bar{D})R_0^{-1}(\bar{B}^*X_0\bar{A}+\bar{D}^*\bar{C})\prec0. \tag*{(297)} A ˉ ∗ X 0 A ˉ − X 0 + C ˉ ∗ C ˉ − ( A ˉ ∗ X 0 B ˉ + C ˉ ∗ D ˉ ) R 0 − 1 ( B ˉ ∗ X 0 A ˉ + D ˉ ∗ C ˉ ) ≺ 0. (297)
The left-hand side of (297) is the Schur complement of the block R 0 R_0 R 0 in the Hermitian matrix
M 0 : = [ A ˉ ∗ X 0 A ˉ − X 0 + C ˉ ∗ C ˉ A ˉ ∗ X 0 B ˉ + C ˉ ∗ D ˉ B ˉ ∗ X 0 A ˉ + D ˉ ∗ C ˉ R 0 ] . (298) \mathcal{M}_0:=\begin{bmatrix}
\bar{A}^*X_0\bar{A}-X_0+\bar{C}^*\bar{C} & \bar{A}^*X_0\bar{B}+\bar{C}^*\bar{D}\\
\bar{B}^*X_0\bar{A}+\bar{D}^*\bar{C} & R_0
\end{bmatrix}. \tag*{(298)} M 0 := [ A ˉ ∗ X 0 A ˉ − X 0 + C ˉ ∗ C ˉ B ˉ ∗ X 0 A ˉ + D ˉ ∗ C ˉ A ˉ ∗ X 0 B ˉ + C ˉ ∗ D ˉ R 0 ] . (298)
and a Hermitian matrix with a negative definite block R 0 R_0 R 0 is negative definite if and only if the Schur complement of R 0 R_0 R 0 is negative definite. Hence (296) –(297) are equivalent to M 0 ≺ 0 \mathcal{M}_0\prec0 M 0 ≺ 0 .
Step 5: Positivity of the storage and return to the original input. We have necessarily X 0 ≻ 0 X_0 \succ0 X 0 ≻ 0 . To see this, note that Q : = X 0 − A ˉ ∗ X 0 A ˉ ≻ 0 Q := X_0-\bar{A}^{*}X_0\bar{A}\succ0 Q := X 0 − A ˉ ∗ X 0 A ˉ ≻ 0 by the top-left block of (298) and A ˉ \bar{A} A ˉ is Schur, therefore by the definition of Q Q Q and using the identity X 0 = Q + A ˉ ∗ X 0 A ˉ X_0=Q+\bar{A}^{*}X_0\bar{A} X 0 = Q + A ˉ ∗ X 0 A ˉ recursively
X 0 = Q + A ˉ ∗ X 0 A ˉ = Q + A ˉ ∗ Q A ˉ + ( A ˉ ∗ ) 2 X 0 A ˉ 2 = ⋯ = ∑ k = 0 N − 1 ( A ˉ ∗ ) k Q A ˉ k + ( A ˉ ∗ ) N X 0 A ˉ N . X_0=Q+\bar{A}^{*}X_0\bar{A}=Q+\bar{A}^{*}Q\bar{A}+(\bar{A}^{*})^2X_0\bar{A}^2=\cdots=\sum_{k=0}^{N-1}(\bar{A}^{*})^kQ\bar{A}^k+(\bar{A}^{*})^NX_0\bar{A}^N. X 0 = Q + A ˉ ∗ X 0 A ˉ = Q + A ˉ ∗ Q A ˉ + ( A ˉ ∗ ) 2 X 0 A ˉ 2 = ⋯ = k = 0 ∑ N − 1 ( A ˉ ∗ ) k Q A ˉ k + ( A ˉ ∗ ) N X 0 A ˉ N .
Since A ˉ \bar{A} A ˉ is Schur, A ˉ N → 0 \bar{A}^N\to0 A ˉ N → 0 as N → ∞ N\to\infty N → ∞ and we get X 0 = ∑ k ≥ 0 ( A ˉ ∗ ) k Q A ˉ k X_0=\sum_{k\geq0}(\bar{A}^{*})^kQ\bar{A}^k X 0 = ∑ k ≥ 0 ( A ˉ ∗ ) k Q A ˉ k , and every summand is positive semidefinite with the k = 0 k=0 k = 0 summand equal to Q Q Q ; hence X 0 ⪰ Q ≻ 0 X_0\succeq Q\succ0 X 0 ⪰ Q ≻ 0 . This is the KYP lemma for the frequency-domain inequality K ( e i θ ) ∗ K ( e i θ ) − I d ≺ 0 K(e^{i\theta})^{*}K(e^{i\theta})-I_d\prec0 K ( e i θ ) ∗ K ( e i θ ) − I d ≺ 0 , i.e., for the quadratic form ∥ e + ∥ 2 − ∥ e − ∥ 2 \lVert e_+\rVert^2-\lVert e_-\rVert^2 ∥ e + ∥ 2 − ∥ e − ∥ 2 in the output–input pair of K K K .11 We apply it with state Ξ k = ( ξ k , q k ) \Xi_k=(\xi_k,q_k) Ξ k = ( ξ k , q k ) , input e − , k e_{-,k} e − , k , output e + , k e_{+,k} e + , k , and ( A ˉ , B ˉ , C ˉ , D ˉ ) (\bar{A},\bar{B},\bar{C},\bar{D}) ( A ˉ , B ˉ , C ˉ , D ˉ ) as in (292) , so that K = H + H − − 1 K=H_+H_-^{-1} K = H + H − − 1 by (293) ; its hypotheses are exactly the conclusions of Step 4, namely (294) and (295) . Since the supply of Step 1 is σ k = 1 2 ( ∥ e + , k ∥ 2 − ∥ e − , k ∥ 2 ) \sigma_k=\frac{1}{2}(\lVert e_{+,k}\rVert^2-\lVert e_{-,k}\rVert^2) σ k = 2 1 (∥ e + , k ∥ 2 − ∥ e − , k ∥ 2 ) , we set X : = 1 2 X 0 X:=\frac{1}{2}X_0 X := 2 1 X 0 ; halving (298) gives
M X : = [ A ˉ ∗ X A ˉ − X + 1 2 C ˉ ∗ C ˉ A ˉ ∗ X B ˉ + 1 2 C ˉ ∗ D ˉ B ˉ ∗ X A ˉ + 1 2 D ˉ ∗ C ˉ B ˉ ∗ X B ˉ + 1 2 ( D ˉ ∗ D ˉ − I d ) ] ≺ 0. (299) \mathcal{M}_X:=\begin{bmatrix}
\bar{A}^{*}X\bar{A}-X+\frac{1}{2}\bar{C}^{*}\bar{C} & \bar{A}^{*}X\bar{B}+\frac{1}{2}\bar{C}^{*}\bar{D}\\
\bar{B}^{*}X\bar{A}+\frac{1}{2}\bar{D}^{*}\bar{C} & \bar{B}^{*}X\bar{B}+\frac{1}{2}(\bar{D}^{*}\bar{D}-I_d)
\end{bmatrix}\prec0. \tag*{(299)} M X := [ A ˉ ∗ X A ˉ − X + 2 1 C ˉ ∗ C ˉ B ˉ ∗ X A ˉ + 2 1 D ˉ ∗ C ˉ A ˉ ∗ X B ˉ + 2 1 C ˉ ∗ D ˉ B ˉ ∗ X B ˉ + 2 1 ( D ˉ ∗ D ˉ − I d ) ] ≺ 0. (299)
Since (299) is a strict finite-dimensional inequality, there exists δ > 0 \delta>0 δ > 0 such that M X ⪯ − δ I n a + d \mathcal{M}_X\preceq-\delta I_{n_a+d} M X ⪯ − δ I n a + d . Using this inequality and evaluating this quadratic form at w k : = col ( Ξ k , e − , k ) w_k:=\operatorname{col}(\Xi_k,e_{-,k}) w k := col ( Ξ k , e − , k ) gives, along every trajectory of (291) ,
w k ∗ M X w k = Ξ k ∗ ( A ˉ ∗ X A ˉ − X ) Ξ k + 2 Re Ξ k ∗ A ˉ ∗ X B ˉ e − , k + e − , k ∗ B ˉ ∗ X B ˉ e − , k + 1 2 ( Ξ k ∗ C ˉ ∗ C ˉ Ξ k + 2 Re Ξ k ∗ C ˉ ∗ D ˉ e − , k + e − , k ∗ D ˉ ∗ D ˉ e − , k ) − 1 2 ∥ e − , k ∥ 2 = ( A ˉ Ξ k + B ˉ e − , k ) ∗ X ( A ˉ Ξ k + B ˉ e − , k ) − Ξ k ∗ X Ξ k + 1 2 ∥ C ˉ Ξ k + D ˉ e − , k ∥ 2 − 1 2 ∥ e − , k ∥ 2 = V ( Ξ k + 1 ) − V ( Ξ k ) + 1 2 ( ∥ e + , k ∥ 2 − ∥ e − , k ∥ 2 ) ≤ − δ ∥ w k ∥ 2 . \begin{aligned}
w_k^{*}\mathcal{M}_Xw_k
&=\Xi_k^{*}(\bar{A}^{*}X\bar{A}-X)\Xi_k+2\operatorname{Re}\Xi_k^{*}\bar{A}^{*}X\bar{B}e_{-,k}+e_{-,k}^{*}\bar{B}^{*}X\bar{B}e_{-,k}\\
&\quad+\frac{1}{2}\left(\Xi_k^{*}\bar{C}^{*}\bar{C}\Xi_k+2\operatorname{Re}\Xi_k^{*}\bar{C}^{*}\bar{D}e_{-,k}+e_{-,k}^{*}\bar{D}^{*}\bar{D}e_{-,k}\right)-\frac{1}{2}\lVert e_{-,k}\rVert^2\\
&=(\bar{A}\Xi_k+\bar{B}e_{-,k})^{*}X(\bar{A}\Xi_k+\bar{B}e_{-,k})-\Xi_k^{*}X\Xi_k+\frac{1}{2}\lVert\bar{C}\Xi_k+\bar{D}e_{-,k}\rVert^2-\frac{1}{2}\lVert e_{-,k}\rVert^2\\
&=V(\Xi_{k+1})-V(\Xi_k)+\frac{1}{2}(\lVert e_{+,k}\rVert^2-\lVert e_{-,k}\rVert^2)\leq-\delta\lVert w_k\rVert^2.
\end{aligned} w k ∗ M X w k = Ξ k ∗ ( A ˉ ∗ X A ˉ − X ) Ξ k + 2 Re Ξ k ∗ A ˉ ∗ X B ˉ e − , k + e − , k ∗ B ˉ ∗ X B ˉ e − , k + 2 1 ( Ξ k ∗ C ˉ ∗ C ˉ Ξ k + 2 Re Ξ k ∗ C ˉ ∗ D ˉ e − , k + e − , k ∗ D ˉ ∗ D ˉ e − , k ) − 2 1 ∥ e − , k ∥ 2 = ( A ˉ Ξ k + B ˉ e − , k ) ∗ X ( A ˉ Ξ k + B ˉ e − , k ) − Ξ k ∗ X Ξ k + 2 1 ∥ C ˉ Ξ k + D ˉ e − , k ∥ 2 − 2 1 ∥ e − , k ∥ 2 = V ( Ξ k + 1 ) − V ( Ξ k ) + 2 1 (∥ e + , k ∥ 2 − ∥ e − , k ∥ 2 ) ≤ − δ ∥ w k ∥ 2 .
Since ∥ w k ∥ 2 = ∥ Ξ k ∥ 2 + ∥ e − , k ∥ 2 \lVert w_k\rVert^2=\lVert\Xi_k\rVert^2+\lVert e_{-,k}\rVert^2 ∥ w k ∥ 2 = ∥ Ξ k ∥ 2 + ∥ e − , k ∥ 2 , this leads to
V ( Ξ k + 1 ) − V ( Ξ k ) + 1 2 ( ∥ e + , k ∥ 2 − ∥ e − , k ∥ 2 ) ≤ − δ ( ∥ Ξ k ∥ 2 + ∥ e − , k ∥ 2 ) . (300) V(\Xi_{k+1})-V(\Xi_k)+\frac{1}{2}(\lVert e_{+,k}\rVert^2-\lVert e_{-,k}\rVert^2)\leq-\delta(\lVert\Xi_k\rVert^2+\lVert e_{-,k}\rVert^2). \tag*{(300)} V ( Ξ k + 1 ) − V ( Ξ k ) + 2 1 (∥ e + , k ∥ 2 − ∥ e − , k ∥ 2 ) ≤ − δ (∥ Ξ k ∥ 2 + ∥ e − , k ∥ 2 ) . (300)
where V ( Ξ ) = Ξ ∗ X Ξ V(\Xi)=\Xi^{*}X\Xi V ( Ξ ) = Ξ ∗ X Ξ .
It remains only to express the strict dissipation bound in terms of the original input v ^ k \hat{v}_k v ^ k . From (288) , e − , k = C − Ξ k + D − v ^ k e_{-,k}=C_-\Xi_k+D_-\hat{v}_k e − , k = C − Ξ k + D − v ^ k , D − = 2 I d D_-=2I_d D − = 2 I d . Hence
[ Ξ k e − , k ] = T [ Ξ k v ^ k ] , T : = [ I n a 0 C − D − ] ∈ C ( n a + d ) × ( n a + d ) . \begin{bmatrix}\Xi_k\\e_{-,k}\end{bmatrix}
=T\begin{bmatrix}\Xi_k\\\hat{v}_k\end{bmatrix},
\qquad
T:=\begin{bmatrix}I_{n_a}&0\\C_-&D_-\end{bmatrix}\in\mathbb{C}^{(n_a+d)\times(n_a+d)}. [ Ξ k e − , k ] = T [ Ξ k v ^ k ] , T := [ I n a C − 0 D − ] ∈ C ( n a + d ) × ( n a + d ) .
The block triangular matrix T T T is invertible because D − D_- D − is invertible. Therefore, with γ : = σ min ( T ) 2 > 0 \gamma:=\sigma_{\min}(T)^2>0 γ := σ m i n ( T ) 2 > 0 ,
∥ Ξ k ∥ 2 + ∥ e − , k ∥ 2 ≥ γ ( ∥ Ξ k ∥ 2 + ∥ v ^ k ∥ 2 ) (301) \lVert\Xi_k\rVert^2+\lVert e_{-,k}\rVert^2\geq\gamma(\lVert\Xi_k\rVert^2+\lVert\hat{v}_k\rVert^2) \tag*{(301)} ∥ Ξ k ∥ 2 + ∥ e − , k ∥ 2 ≥ γ (∥ Ξ k ∥ 2 + ∥ v ^ k ∥ 2 ) (301)
for all ( Ξ k , v ^ k ) (\Xi_k,\hat{v}_k) ( Ξ k , v ^ k ) .
Combining the exact identity (277) , the dissipation inequality (300) , and (301) , we conclude that
V ( Ξ k + 1 ) − V ( Ξ k ) + σ k ≤ − δ γ ( ∥ Ξ k ∥ 2 + ∥ v ^ k ∥ 2 ) . V(\Xi_{k+1})-V(\Xi_k)+\sigma_k\leq-\delta\gamma(\lVert\Xi_k\rVert^2+\lVert\hat{v}_k\rVert^2). V ( Ξ k + 1 ) − V ( Ξ k ) + σ k ≤ − δ γ (∥ Ξ k ∥ 2 + ∥ v ^ k ∥ 2 ) .
Thus (173) holds with ϵ : = δ γ > 0 \epsilon:=\delta\gamma>0 ϵ := δ γ > 0 .
Optimal parameter design for GMM on quadratics
We now determine exactly the supremum of the rate constants certified by Theorem 3.22 over all admissible extrapolation parameters ν \nu ν , and we prove that this optimal constant degrades exactly like 1 / τ 1/\tau 1/ τ as the delay grows.
Theorem 3.22 reduces the design of an accelerated delayed method on quadratics to a single scalar question. It shows that whenever the limiting polynomial P ∞ ( ⋅ ; δ , ν ) P_{\infty}(\cdot;\delta,\nu) P ∞ ( ⋅ ; δ , ν ) is strictly Schur for every gain 0 < δ ≤ δ L 0<\delta\le\delta_L 0 < δ ≤ δ L , the method run with α = δ L / L \alpha=\delta_L/L α = δ L / L and β = 1 − 2 δ L q \beta=1-2\sqrt{\delta_L}\,q β = 1 − 2 δ L q attains the accelerated rate ρ τ , κ ≤ 1 − c q \rho_{\tau,\kappa}\le1-cq ρ τ , κ ≤ 1 − c q for every 0 < c < δ L 0<c<\sqrt{\delta_L} 0 < c < δ L and all sufficiently large κ \kappa κ . Thus, for a given admissible gain interval, the supremum of the rate constants certified by Theorem 3.22 is δ L \sqrt{\delta_L} δ L . Since the extrapolation parameter ν \nu ν remains free, it is natural to choose ν \nu ν so as to maximize the length of this stability interval. To make this precise, fix an integer τ ≥ 1 \tau\ge1 τ ≥ 1 and for ν > τ \nu>\tau ν > τ let
δ L ( ν ) : = inf { δ > 0 : P ∞ ( ⋅ ; δ , ν ) has a root on the unit circle } (302) \delta_L(\nu):=\inf\{\delta>0:P_{\infty}(\cdot;\delta,\nu)\text{ has a root on the unit circle}\} \tag*{(302)} δ L ( ν ) := inf { δ > 0 : P ∞ ( ⋅ ; δ , ν ) has a root on the unit circle } (302)
be the first positive gain at which a root lies on the unit circle.
By Corollary 3.23 and Lemma B.2 below, δ L ( ν ) > 0 \delta_L(\nu)>0 δ L ( ν ) > 0 for every ν > τ \nu>\tau ν > τ , and then P ∞ ( ⋅ ; δ , ν ) P_{\infty}(\cdot;\delta,\nu) P ∞ ( ⋅ ; δ , ν ) is strictly Schur for all 0 < δ < δ L ( ν ) 0<\delta<\delta_L(\nu) 0 < δ < δ L ( ν ) . Hence, every δ ^ L < δ L ( ν ) \widehat{\delta}_L<\delta_L(\nu) δ L < δ L ( ν ) is admissible in Theorem 3.22. Letting δ ^ L ↑ δ L ( ν ) \widehat{\delta}_L\uparrow\delta_L(\nu) δ L ↑ δ L ( ν ) shows that the supremum of the certifiable rate constants for a fixed ν \nu ν is δ L ( ν ) \sqrt{\delta_L(\nu)} δ L ( ν ) . The design problem is therefore
max ν > τ δ L ( ν ) , (303) \max_{\nu>\tau}\delta_L(\nu), \tag*{(303)} ν > τ max δ L ( ν ) , (303)
whose square root is the supremum of rate constants certifiable by Theorem 3.22. The difficulty in (303) is that a root z = r e i θ z=re^{i\theta} z = r e i θ of P ∞ P_{\infty} P ∞ may reach the unit circle at any frequency θ ∈ [ 0 , 2 π ) \theta\in[0,2\pi) θ ∈ [ 0 , 2 π ) , so δ L ( ν ) \delta_L(\nu) δ L ( ν ) is a priori governed by an infinite family of competing crossing conditions. We resolve this difficulty completely below: for every admissible ν \nu ν , strict stability is first lost at a single explicitly characterized low-frequency crossing (Lemma B.2 ), and consequently (303) is solved exactly (Corollary B.4 ).
The stability boundary
At a loss of strict stability, a root of P ∞ P_{\infty} P ∞ lies on the unit circle, say z = e i θ z=e^{i\theta} z = e i θ . Since P ∞ P_{\infty} P ∞ has real coefficients, its complex roots occur in conjugate pairs, so e i θ e^{i\theta} e i θ and e − i θ e^{-i\theta} e − i θ cross together and we may take θ ∈ ( 0 , π ) \theta\in(0,\pi) θ ∈ ( 0 , π ) . From (101) , a unit-circle root satisfies
e i τ θ ( e i θ − 1 ) 2 + δ [ ( 1 + ν ) e i θ − ν ] = 0. (304) e^{i\tau\theta}(e^{i\theta}-1)^2+\delta\left[(1+\nu)e^{i\theta}-\nu\right]=0. \tag*{(304)} e i τ θ ( e i θ − 1 ) 2 + δ [ ( 1 + ν ) e i θ − ν ] = 0. (304)
This is one complex equation—two real equations—in the two real unknowns ( δ , ν ) (\delta,\nu) ( δ , ν ) ; as θ \theta θ ranges over ( 0 , π ) (0,\pi) ( 0 , π ) it traces the stability boundary in the ( ν , δ ) (\nu,\delta) ( ν , δ ) plane. The next proposition parametrizes it explicitly.
Proposition B.1 (the stability boundary). Fix an integer τ ≥ 1 \tau\ge1 τ ≥ 1 and write u : = θ / 2 u:=\theta/2 u := θ /2 and M = 2 τ + 1 M=2\tau+1 M = 2 τ + 1 . For θ ∈ ( 0 , π ) \theta\in(0,\pi) θ ∈ ( 0 , π ) , the equation (304) admits a solution ( δ , ν ) ∈ ( 0 , ∞ ) × R (\delta,\nu)\in(0,\infty)\times\mathbb{R} ( δ , ν ) ∈ ( 0 , ∞ ) × R if and only if cos ( M u ) > 0 \cos(Mu)>0 cos ( M u ) > 0 . When this condition holds, the solution is unique and is given by
δ = D τ ( u ) : = 4 sin 2 u cos ( M u ) cos u , ν = N τ ( u ) : = sin ( 2 τ u ) 2 sin u cos ( M u ) . (305) \delta=D_\tau(u):=\frac{4\sin^2u\cos(Mu)}{\cos u},\qquad\nu=N_\tau(u):=\frac{\sin(2\tau u)}{2\sin u\cos(Mu)}. \tag*{(305)} δ = D τ ( u ) := cos u 4 sin 2 u cos ( M u ) , ν = N τ ( u ) := 2 sin u cos ( M u ) sin ( 2 τ u ) . (305) Both D τ D_\tau D τ and N τ N_\tau N τ are positive on the interval
I τ : = ( 0 , π 2 M ) , (306) \mathcal{I}_\tau:=\left(0,\frac{\pi}{2M}\right), \tag*{(306)} I τ := ( 0 , 2 M π ) , (306) and D τ D_\tau D τ vanishes at its two endpoints u = 0 u=0 u = 0 and u = π / ( 2 M ) u=\pi/(2M) u = π / ( 2 M ) .
Proof. Using ( e i θ − 1 ) 2 = − 4 sin 2 u e i θ (e^{i\theta}-1)^2=-4\sin^2u\,e^{i\theta} ( e i θ − 1 ) 2 = − 4 sin 2 u e i θ and θ = 2 u \theta=2u θ = 2 u , equation (304) is equivalent to
δ [ ( 1 + ν ) e 2 i u − ν ] = 4 sin 2 u e i ( 2 τ + 2 ) u . \delta\left[(1+\nu)e^{2iu}-\nu\right]=4\sin^2u\,e^{i(2\tau+2)u}. δ [ ( 1 + ν ) e 2 i u − ν ] = 4 sin 2 u e i ( 2 τ + 2 ) u .
Factor e i u e^{iu} e i u from each side. On the left, ( 1 + ν ) e 2 i u − ν = e i u [ cos u + i ( 1 + 2 ν ) sin u ] (1+\nu)e^{2iu}-\nu=e^{iu}\left[\cos u+i(1+2\nu)\sin u\right] ( 1 + ν ) e 2 i u − ν = e i u [ cos u + i ( 1 + 2 ν ) sin u ] ; on the right, e i ( 2 τ + 2 ) u = e i u e i M u e^{i(2\tau+2)u}=e^{iu}e^{iMu} e i ( 2 τ + 2 ) u = e i u e i M u because 2 τ + 2 = M + 1 2\tau+2=M+1 2 τ + 2 = M + 1 . Cancelling e i u e^{iu} e i u leaves
δ [ cos u + i ( 1 + 2 ν ) sin u ] = 4 sin 2 u [ cos ( M u ) + i sin ( M u ) ] . \delta\left[\cos u+i(1+2\nu)\sin u\right]=4\sin^2u\left[\cos(Mu)+i\sin(Mu)\right]. δ [ cos u + i ( 1 + 2 ν ) sin u ] = 4 sin 2 u [ cos ( M u ) + i sin ( M u ) ] .
Equating real and imaginary parts, with δ \delta δ and ν \nu ν real, gives
δ cos u = 4 sin 2 u cos ( M u ) , δ ( 1 + 2 ν ) sin u = 4 sin 2 u sin ( M u ) . (307) \delta\cos u=4\sin^2u\cos(Mu),\qquad\delta(1+2\nu)\sin u=4\sin^2u\sin(Mu). \tag*{(307)} δ cos u = 4 sin 2 u cos ( M u ) , δ ( 1 + 2 ν ) sin u = 4 sin 2 u sin ( M u ) . (307)
Because u ∈ ( 0 , π / 2 ) u\in(0,\pi/2) u ∈ ( 0 , π /2 ) , we have cos u > 0 \cos u>0 cos u > 0 and sin u > 0 \sin u>0 sin u > 0 . Thus any solution with δ > 0 \delta>0 δ > 0 must satisfy cos ( M u ) > 0 \cos(Mu)>0 cos ( M u ) > 0 . In particular, the denominator defining N τ ( u ) N_\tau(u) N τ ( u ) is nonzero. The first equation in (307) then uniquely determines δ = D τ ( u ) \delta=D_\tau(u) δ = D τ ( u ) . Dividing the second by the first and using the angle-difference identity cos u sin ( M u ) − sin u cos ( M u ) = sin ( ( M − 1 ) u ) = sin ( 2 τ u ) \cos u\sin(Mu)-\sin u\cos(Mu)=\sin((M-1)u)=\sin(2\tau u) cos u sin ( M u ) − sin u cos ( M u ) = sin (( M − 1 ) u ) = sin ( 2 τ u ) ,
1 + 2 ν = cos u sin ( M u ) sin u cos ( M u ) , hence 2 ν = sin ( 2 τ u ) sin u cos ( M u ) , (308) 1+2\nu=\frac{\cos u\sin(Mu)}{\sin u\cos(Mu)},
\qquad\text{hence}\qquad
2\nu=\frac{\sin(2\tau u)}{\sin u\cos(Mu)}, \tag*{(308)} 1 + 2 ν = sin u cos ( M u ) cos u sin ( M u ) , hence 2 ν = sin u cos ( M u ) sin ( 2 τ u ) , (308)
which is ν = N τ ( u ) \nu=N_\tau(u) ν = N τ ( u ) . Conversely, ( δ , ν ) = ( D τ ( u ) , N τ ( u ) ) (\delta,\nu)=(D_\tau(u),N_\tau(u)) ( δ , ν ) = ( D τ ( u ) , N τ ( u )) solves (307) and hence (304) . On I τ I_\tau I τ one has M u ∈ ( 0 , π / 2 ) Mu\in(0,\pi/2) M u ∈ ( 0 , π /2 ) , so cos ( M u ) > 0 \cos(Mu)>0 cos ( M u ) > 0 and cos u > 0 \cos u>0 cos u > 0 , whence D τ ( u ) > 0 D_\tau(u)>0 D τ ( u ) > 0 ; and 2 τ u < 2 τ ⋅ π 2 M = τ π M < π 2 2\tau u<2\tau\cdot\frac{\pi}{2M}=\frac{\tau\pi}{M}<\frac{\pi}{2} 2 τ u < 2 τ ⋅ 2 M π = M τ π < 2 π , so sin ( 2 τ u ) > 0 \sin(2\tau u)>0 sin ( 2 τ u ) > 0 and N τ ( u ) > 0 N_\tau(u)>0 N τ ( u ) > 0 . Finally sin u \sin u sin u vanishes at u = 0 u=0 u = 0 and cos ( M u ) \cos(Mu) cos ( M u ) at u = π / ( 2 M ) u=\pi/(2M) u = π / ( 2 M ) , so D τ = 0 D_\tau=0 D τ = 0 at both endpoints.
Stability loss through unit-circle crossings
Fix ν ∈ ( τ , ∞ ) \nu\in(\tau,\infty) ν ∈ ( τ , ∞ ) . As the gain δ \delta δ increases from zero, strict Schur stability persists until a root of P ∞ ( ⋅ ; δ , ν ) P_\infty(\mathord{\cdot};\delta,\nu) P ∞ ( ⋅ ; δ , ν ) first reaches the unit circle. Thus, by (302) , δ L ( ν ) \delta_L(\nu) δ L ( ν ) is the smallest positive gain associated with a unit-circle crossing; moreover, δ L ( ν ) > 0 \delta_L(\nu)>0 δ L ( ν ) > 0 by the expansion (109) .
The crossing equations (307) substantially restrict the possible crossing frequencies. Since u ∈ ( 0 , π / 2 ) u\in(0,\pi/2) u ∈ ( 0 , π /2 ) and δ > 0 \delta>0 δ > 0 , the first equation in (307) implies cos ( M u ) > 0 \cos(Mu)>0 cos ( M u ) > 0 . Moreover, ν > τ ≥ 1 \nu>\tau\geq1 ν > τ ≥ 1 implies 1 + 2 ν > 0 1+2\nu>0 1 + 2 ν > 0 , so the second equation gives sin ( M u ) > 0 \sin(Mu)>0 sin ( M u ) > 0 . Hence every positive-gain crossing relevant to the design problem (303) satisfies
M u ∈ ( 2 k π , 2 k π + π 2 ) , k ≥ 0. Mu\in\left(2k\pi,2k\pi+\frac{\pi}{2}\right),\qquad k\geq0. M u ∈ ( 2 k π , 2 k π + 2 π ) , k ≥ 0.
The band corresponding to k = 0 k=0 k = 0 is precisely I τ I_\tau I τ in (306) , the interval of lowest crossing frequencies. Borrowing the main-lobe terminology for oscillatory kernels [22 ] , we refer to I τ I_\tau I τ as the first lobe . It is the band adjacent to the marginal frequency u = 0 u=0 u = 0 , where at zero gain the polynomial z τ ( z − 1 ) 2 z^\tau(z-1)^2 z τ ( z − 1 ) 2 has a double root at z = 1 z=1 z = 1 . The following lemma shows that, among all positive-gain unit-circle crossings for a fixed ν > τ \nu>\tau ν > τ , the first-lobe crossing has the smallest gain. Consequently, δ L ( ν ) \delta_L(\nu) δ L ( ν ) is attained on I τ I_\tau I τ .
Lemma B.2 (all-frequency exclusion and first-lobe optimality). Fix τ ≥ 1 \tau\geq1 τ ≥ 1 , M = 2 τ + 1 M=2\tau+1 M = 2 τ + 1 , and ν > τ \nu>\tau ν > τ ; write A : = 1 + 2 ν A:=1+2\nu A := 1 + 2 ν , so that A > M A>M A > M . Then the following hold.
(a) (Crossing conditions.) Every unit-circle root of P ∞ ( ⋅ ; δ , ν ) P_\infty(\mathord{\cdot};\delta,\nu) P ∞ ( ⋅ ; δ , ν ) with δ > 0 \delta>0 δ > 0 and θ ∈ ( 0 , π ) \theta\in(0,\pi) θ ∈ ( 0 , π ) has u = θ / 2 u=\theta/2 u = θ /2 satisfying
tan ( M u ) = A tan u , M u ∈ ( 2 k π , 2 k π + π 2 ) for some k ≥ 0 , δ = D τ ( u ) . (309) \tan(Mu)=A\tan u,\qquad Mu\in\left(2k\pi,2k\pi+\frac{\pi}{2}\right)\text{ for some }k\geq0,\qquad\delta=D_\tau(u). \tag*{(309)} tan ( M u ) = A tan u , M u ∈ ( 2 k π , 2 k π + 2 π ) for some k ≥ 0 , δ = D τ ( u ) . (309)
(b) (Monotonicity of the gain along the boundary.) Along the crossing set { u : N τ ( u ) = ν } \{u : N_{\tau}(u) = \nu\} { u : N τ ( u ) = ν } one has D τ ( u ) = Φ A ( tan u ) D_{\tau}(u) = \Phi_{A}(\tan u) D τ ( u ) = Φ A ( tan u ) , where
Φ A ( t ) : = 4 t 2 ( 1 + t 2 ) ( 1 + A 2 t 2 ) (310) \Phi_{A}(t) := \frac{4t^{2}}{\sqrt{(1+t^{2})(1+A^{2}t^{2})}} \tag*{(310)} Φ A ( t ) := ( 1 + t 2 ) ( 1 + A 2 t 2 ) 4 t 2 (310)
is strictly increasing on ( 0 , ∞ ) (0,\infty) ( 0 , ∞ ) . Consequently, D τ ( u ) D_{\tau}(u) D τ ( u ) is strictly increasing with u u u along the crossing set.
(c) (A unique lowest crossing.) The map R ( u ) : = tan ( M u ) / tan u R(u) := \tan(Mu)/\tan u R ( u ) := tan ( M u ) / tan u is a strictly increasing bijection of I τ I_{\tau} I τ onto ( M , ∞ ) (M,\infty) ( M , ∞ ) , with 1 + 2 N τ ( u ) = R ( u ) 1 + 2N_{\tau}(u) = R(u) 1 + 2 N τ ( u ) = R ( u ) ; hence N τ N_{\tau} N τ is a strictly increasing bijection I τ → ( τ , ∞ ) I_{\tau} \to(\tau,\infty) I τ → ( τ , ∞ ) . The crossing set therefore contains exactly one point u 0 ( ν ) = N τ − 1 ( ν ) u_{0}(\nu) = N_{\tau}^{-1}(\nu) u 0 ( ν ) = N τ − 1 ( ν ) in I τ I_{\tau} I τ , and every other positive-gain crossing has u > 2 π / M > π / ( 2 M ) > u 0 ( ν ) u > 2\pi/M > \pi/(2M) > u_{0}(\nu) u > 2 π / M > π / ( 2 M ) > u 0 ( ν ) .
(d) (Endpoints do not preempt.) The endpoint z = + 1 z = +1 z = + 1 is never a root at positive gain, and the endpoint z = − 1 z = -1 z = − 1 (which can occur only for even τ \tau τ ) occurs at δ = 4 / A \delta= 4/A δ = 4/ A , which exceeds δ L ( ν ) \delta_{L}(\nu) δ L ( ν ) . In particular, strict stability is first lost at the first-lobe frequency u 0 ( ν ) u_{0}(\nu) u 0 ( ν ) , and
δ L ( ν ) = D τ ( u 0 ( ν ) ) = D τ ( N τ − 1 ( ν ) ) . (311) \delta_{L}(\nu) = D_{\tau}(u_{0}(\nu)) = D_{\tau}(N_{\tau}^{-1}(\nu)). \tag*{(311)} δ L ( ν ) = D τ ( u 0 ( ν )) = D τ ( N τ − 1 ( ν )) . (311)
Proof. (a) At a positive-gain unit-circle root, u ∈ ( 0 , π 2 ) u \in(0,\frac{\pi}{2}) u ∈ ( 0 , 2 π ) and (307) holds with δ > 0 \delta> 0 δ > 0 . Since sin u > 0 \sin u > 0 sin u > 0 and sin 2 u > 0 \sin^{2}u > 0 sin 2 u > 0 , the first equation forces cos ( M u ) > 0 \cos(Mu) > 0 cos ( M u ) > 0 and the second forces sin ( M u ) > 0 \sin(Mu) > 0 sin ( M u ) > 0 , placing M u Mu M u in a first-quadrant band. Dividing the second equation in (307) by the first gives tan ( M u ) = ( 1 + 2 ν ) tan u = A tan u \tan(Mu) = (1 + 2\nu)\tan u = A\tan u tan ( M u ) = ( 1 + 2 ν ) tan u = A tan u , and the first equation is δ = D τ ( u ) \delta= D_{\tau}(u) δ = D τ ( u ) .
(b) Fix a crossing u u u and put t = tan u > 0 t = \tan u > 0 t = tan u > 0 . Since cos ( M u ) > 0 \cos(Mu) > 0 cos ( M u ) > 0 and tan ( M u ) = A t \tan(Mu) = At tan ( M u ) = A t , we have cos ( M u ) = ( 1 + A 2 t 2 ) − 1 / 2 \cos(Mu) = (1 + A^{2}t^{2})^{-1/2} cos ( M u ) = ( 1 + A 2 t 2 ) − 1/2 . Using sin 2 u = t 2 cos 2 u \sin^{2}u = t^{2}\cos^{2}u sin 2 u = t 2 cos 2 u and cos u = ( 1 + t 2 ) − 1 / 2 \cos u = (1 + t^{2})^{-1/2} cos u = ( 1 + t 2 ) − 1/2 ,
D τ ( u ) = 4 sin 2 u cos ( M u ) cos u = 4 t 2 cos u cos ( M u ) = 4 t 2 ( 1 + t 2 ) ( 1 + A 2 t 2 ) = Φ A ( t ) . D_{\tau}(u) = \frac{4\sin^{2}u\cos(Mu)}{\cos u} = 4t^{2}\cos u\cos(Mu) = \frac{4t^{2}}{\sqrt{(1+t^{2})(1+A^{2}t^{2})}} = \Phi_{A}(t). D τ ( u ) = cos u 4 sin 2 u cos ( M u ) = 4 t 2 cos u cos ( M u ) = ( 1 + t 2 ) ( 1 + A 2 t 2 ) 4 t 2 = Φ A ( t ) .
Differentiating log Φ A \log\Phi_{A} log Φ A ,
t d d t log Φ A ( t ) = 2 − t 2 1 + t 2 − A 2 t 2 1 + A 2 t 2 > 2 − 1 − 1 = 0 , t\frac{d}{dt}\log\Phi_{A}(t) = 2 - \frac{t^{2}}{1+t^{2}} - \frac{A^{2}t^{2}}{1+A^{2}t^{2}} > 2 - 1 - 1 = 0, t d t d log Φ A ( t ) = 2 − 1 + t 2 t 2 − 1 + A 2 t 2 A 2 t 2 > 2 − 1 − 1 = 0 ,
because each subtracted fraction lies in ( 0 , 1 ) (0,1) ( 0 , 1 ) . Hence Φ A \Phi_{A} Φ A is strictly increasing on ( 0 , ∞ ) (0,\infty) ( 0 , ∞ ) , and since t = tan u t = \tan u t = tan u is strictly increasing on ( 0 , π 2 ) (0,\frac{\pi}{2}) ( 0 , 2 π ) , so is u ↦ D τ ( u ) u \mapsto D_{\tau}(u) u ↦ D τ ( u ) along the crossing set.
(c) By (308) , 1 + 2 N τ ( u ) = R ( u ) 1 + 2N_{\tau}(u) = R(u) 1 + 2 N τ ( u ) = R ( u ) . On I τ I_{\tau} I τ both tan ( M u ) > 0 \tan(Mu) > 0 tan ( M u ) > 0 and tan u > 0 \tan u > 0 tan u > 0 , so R > 0 R > 0 R > 0 ; moreover, R ( 0 + ) = M R(0+) = M R ( 0 + ) = M (since tan ( M u ) / tan u → M \tan(Mu)/\tan u \to M tan ( M u ) / tan u → M ) and R ( u ) → + ∞ R(u) \to+\infty R ( u ) → + ∞ as u → ( π / 2 M ) − u \to(\pi/2M)^{-} u → ( π /2 M ) − (since tan ( M u ) → + ∞ \tan(Mu) \to+\infty tan ( M u ) → + ∞ ). For monotonicity,
( log R ) ′ ( u ) = 2 M sin ( 2 M u ) − 2 sin ( 2 u ) , (\log R)'(u) = \frac{2M}{\sin(2Mu)} - \frac{2}{\sin(2u)}, ( log R ) ′ ( u ) = sin ( 2 M u ) 2 M − sin ( 2 u ) 2 ,
which is positive iff M sin ( 2 u ) > sin ( 2 M u ) M\sin(2u) > \sin(2Mu) M sin ( 2 u ) > sin ( 2 M u ) . Setting v = 2 u ∈ ( 0 , π M ) v = 2u \in(0,\frac{\pi}{M}) v = 2 u ∈ ( 0 , M π ) and h ( v ) : = M sin v − sin ( M v ) h(v) := M\sin v - \sin(Mv) h ( v ) := M sin v − sin ( M v ) , one has h ( 0 ) = 0 h(0) = 0 h ( 0 ) = 0 and h ′ ( v ) = M ( cos v − cos ( M v ) ) > 0 h'(v) = M(\cos v - \cos(Mv)) > 0 h ′ ( v ) = M ( cos v − cos ( M v )) > 0 , because 0 < v < M v < π 0 < v < Mv < \pi 0 < v < M v < π and cos \cos cos is strictly decreasing on ( 0 , π ) (0,\pi) ( 0 , π ) . Thus h > 0 h > 0 h > 0 on ( 0 , π M ) (0,\frac{\pi}{M}) ( 0 , M π ) and R R R is strictly increasing on I τ I_{\tau} I τ . Consequently R : I τ → ( M , ∞ ) R : I_{\tau} \to(M,\infty) R : I τ → ( M , ∞ ) and N τ = ( R − 1 ) / 2 : I τ → ( τ , ∞ ) N_{\tau} = (R - 1)/2 : I_{\tau} \to(\tau,\infty) N τ = ( R − 1 ) /2 : I τ → ( τ , ∞ ) are strictly increasing bijections. Since A > M A > M A > M , the equation R ( u ) = A R(u) = A R ( u ) = A has a unique solution u 0 ( ν ) = N τ − 1 ( ν ) ∈ I τ u_{0}(\nu) = N_{\tau}^{-1}(\nu) \in I_{\tau} u 0 ( ν ) = N τ − 1 ( ν ) ∈ I τ . Any positive-gain crossing outside I τ I_{\tau} I τ lies in a band k ≥ 1 k \ge1 k ≥ 1 , hence has M u > 2 π Mu > 2\pi M u > 2 π , i.e. u > 2 π / M u > 2\pi/M u > 2 π / M , which exceeds π / ( 2 M ) \pi/(2M) π / ( 2 M ) and therefore u 0 ( ν ) u_{0}(\nu) u 0 ( ν ) . (d) From (101) , P ∞ ( 1 ; δ , ν ) = δ > 0 P_\infty(1;\delta,\nu)=\delta>0 P ∞ ( 1 ; δ , ν ) = δ > 0 , so z = + 1 z=+1 z = + 1 is never a root at positive gain. Likewise P ∞ ( − 1 ; δ , ν ) = 4 ( − 1 ) τ − δ A P_\infty(-1;\delta,\nu)=4(-1)^\tau-\delta A P ∞ ( − 1 ; δ , ν ) = 4 ( − 1 ) τ − δ A , which vanishes only at δ = 4 ( − 1 ) τ / A \delta=4(-1)^\tau/A δ = 4 ( − 1 ) τ / A : negative (hence impossible) for odd τ \tau τ , and equal to 4 / A 4/A 4/ A for even τ \tau τ . Combining (305) and (308) at the first-lobe crossing u 0 ( ν ) u_0(\nu) u 0 ( ν ) gives
A D τ ( u 0 ) = D τ ( u 0 ) ( 1 + 2 N τ ( u 0 ) ) = 4 sin u 0 sin ( M u 0 ) < 4 , A D_\tau(u_0)=D_\tau(u_0)(1+2N_\tau(u_0))=4\sin u_0\sin(Mu_0)<4, A D τ ( u 0 ) = D τ ( u 0 ) ( 1 + 2 N τ ( u 0 )) = 4 sin u 0 sin ( M u 0 ) < 4 ,
because u 0 ∈ ( 0 , π / 2 ) u_0\in(0,\pi/2) u 0 ∈ ( 0 , π /2 ) and M u 0 ∈ ( 0 , π / 2 ) Mu_0\in(0,\pi/2) M u 0 ∈ ( 0 , π /2 ) . Hence, D τ ( u 0 ) < 4 A D_\tau(u_0)<\frac{4}{A} D τ ( u 0 ) < A 4 , which implies that the z = − 1 z=-1 z = − 1 crossing does not preempt the first-lobe crossing.
Finally, assembling the parts: by (a) every complex (non-real) positive-gain unit-circle root corresponds to a crossing u u u with gain D τ ( u ) D_\tau(u) D τ ( u ) , by (c) the smallest such u u u is u 0 ( ν ) u_0(\nu) u 0 ( ν ) , and by (b) D τ D_\tau D τ is increasing along the crossing set, so the smallest positive crossing gain is D τ ( u 0 ( ν ) ) D_\tau(u_0(\nu)) D τ ( u 0 ( ν )) . By (d) the endpoint z = − 1 z=-1 z = − 1 does not undercut it. Since P ∞ P_\infty P ∞ is strictly Schur for all small δ > 0 \delta>0 δ > 0 (Corollary 3.23 ) and no root touches the circle for 0 < δ < D τ ( u 0 ( ν ) ) 0<\delta<D_\tau(u_0(\nu)) 0 < δ < D τ ( u 0 ( ν )) , the first stability loss occurs exactly at δ = D τ ( u 0 ( ν ) ) \delta=D_\tau(u_0(\nu)) δ = D τ ( u 0 ( ν )) , which is (311) .
The optimal gain and matched extrapolation. Lemma B.2 identifies δ L ( ν ) \delta_L(\nu) δ L ( ν ) with the value of D τ D_\tau D τ at the unique first-lobe crossing N τ − 1 ( ν ) N_\tau^{-1}(\nu) N τ − 1 ( ν ) . Maximizing over ν \nu ν therefore amounts to maximizing D τ D_\tau D τ over I τ \mathcal{I}_\tau I τ , which we now carry out.
Definition B.3 (critical gain and critical ν \nu ν ). Let u τ ⋆ ∈ I τ u_\tau^\star\in\mathcal{I}_\tau u τ ⋆ ∈ I τ maximize D τ D_\tau D τ . Differentiating log D τ \log D_\tau log D τ , by first-order optimality conditions, u τ ⋆ u_\tau^\star u τ ⋆ solves the equation
D τ ′ ( u ) D τ ( u ) = 2 cot u + tan u − M tan ( M u ) = 0. (312) \frac{D_\tau'(u)}{D_\tau(u)}=2\cot u+\tan u-M\tan(Mu)=0. \tag*{(312)} D τ ( u ) D τ ′ ( u ) = 2 cot u + tan u − M tan ( M u ) = 0. (312) The left-hand side is strictly decreasing on I τ \mathcal{I}_\tau I τ : its derivative is − 2 csc 2 u + sec 2 u − M 2 sec 2 ( M u ) -2\csc^2u+\sec^2u-M^2\sec^2(Mu) − 2 csc 2 u + sec 2 u − M 2 sec 2 ( M u ) , which is negative there because u < π / ( 2 M ) ≤ π / 6 u<\pi/(2M)\le\pi/6 u < π / ( 2 M ) ≤ π /6 gives − 2 csc 2 u + sec 2 u = − 1 − 2 cot 2 u + tan 2 u < 0 -2\csc^2u+\sec^2u=-1-2\cot^2u+\tan^2u<0 − 2 csc 2 u + sec 2 u = − 1 − 2 cot 2 u + tan 2 u < 0 , while − M 2 sec 2 ( M u ) < 0 -M^2\sec^2(Mu)<0 − M 2 sec 2 ( M u ) < 0 . Since the left-hand side runs from + ∞ +\infty + ∞ as u → 0 + u\to0^+ u → 0 + to − ∞ -\infty − ∞ as u → ( π / 2 M ) − u\to(\pi/2M)^- u → ( π /2 M ) − , the maximizer u τ ⋆ u_\tau^\star u τ ⋆ is the unique root of (312) in I τ \mathcal{I}_\tau I τ . We set
δ τ c r i t : = D τ ( u τ ⋆ ) , ν τ c r i t : = N τ ( u τ ⋆ ) , c τ c r i t : = δ τ c r i t . (313) \delta_\tau^{\mathrm{crit}}:=D_\tau(u_\tau^\star),\qquad\nu_\tau^{\mathrm{crit}}:=N_\tau(u_\tau^\star),\qquad c_\tau^{\mathrm{crit}}:=\sqrt{\delta_\tau^{\mathrm{crit}}}. \tag*{(313)} δ τ crit := D τ ( u τ ⋆ ) , ν τ crit := N τ ( u τ ⋆ ) , c τ crit := δ τ crit . (313)
Corollary B.4 (exact solution of the design problem). For every τ ≥ 1 \tau\ge1 τ ≥ 1 , the design problem (303) is solved by ν = ν τ c r i t \nu=\nu_\tau^{\mathrm{crit}} ν = ν τ crit , with optimal value
max ν > τ δ L ( ν ) = max u ∈ I τ D τ ( u ) = D τ ( u τ ⋆ ) = δ τ c r i t . (314) \max_{\nu>\tau}\delta_L(\nu)=\max_{u\in\mathcal{I}_\tau}D_\tau(u)=D_\tau(u_\tau^\star)=\delta_\tau^{\mathrm{crit}}. \tag*{(314)} ν > τ max δ L ( ν ) = u ∈ I τ max D τ ( u ) = D τ ( u τ ⋆ ) = δ τ crit . (314) Consequently, c τ c r i t = δ τ c r i t c_\tau^{\mathrm{crit}}=\sqrt{\delta_\tau^{\mathrm{crit}}} c τ crit = δ τ crit is the supremum of rate constants certifiable by Theorem 3.22 over the entire admissible family { ν > τ } \{\nu>\tau\} { ν > τ } . More precisely, for every fixed τ \tau τ and every 0 < c < c τ c r i t 0<c<c_\tau^{\mathrm{crit}} 0 < c < c τ crit , choose an operating gain δ ˉ = α L \bar{\delta}=\alpha L δ ˉ = α L satisfying c 2 < δ ˉ < δ τ c r i t c^2<\bar{\delta}<\delta_\tau^{\mathrm{crit}} c 2 < δ ˉ < δ τ crit . Then, with
ν = ν τ c r i t , α = δ ˉ L , β = 1 − 2 δ ˉ q , \nu=\nu_\tau^{\mathrm{crit}},\qquad\alpha=\frac{\bar{\delta}}{L},\qquad\beta=1-2\sqrt{\bar{\delta}}\,q, ν = ν τ crit , α = L δ ˉ , β = 1 − 2 δ ˉ q , Theorem 3.22 yields ρ τ , κ ≤ 1 − c q \rho_{\tau,\kappa}\le1-cq ρ τ , κ ≤ 1 − c q for all sufficiently large κ \kappa κ . Moreover, by (314) , no admissible ν > τ \nu>\tau ν > τ permits a larger stability threshold than δ τ c r i t \delta_\tau^{\mathrm{crit}} δ τ crit ; hence, no larger rate constant than c τ c r i t c_\tau^{\mathrm{crit}} c τ crit can be certified by Theorem 3.22.
Proof. By Lemma B.2 , δ L ( ν ) = D τ ( N T − 1 ( ν ) ) \delta_L(\nu) = D_\tau(N_T^{-1}(\nu)) δ L ( ν ) = D τ ( N T − 1 ( ν )) for every ν > τ \nu> \tau ν > τ , and N T − 1 N_T^{-1} N T − 1 is a bijection of ( τ , ∞ ) (\tau,\infty) ( τ , ∞ ) onto I τ I_\tau I τ . Hence, as ν \nu ν ranges over ( τ , ∞ ) (\tau,\infty) ( τ , ∞ ) , the argument N T − 1 ( ν ) N_T^{-1}(\nu) N T − 1 ( ν ) ranges over all of I τ I_\tau I τ , so
max ν > τ δ L ( ν ) = max u ∈ I τ D τ ( u ) = D τ ( u τ ∗ ) = δ τ c r i t , \max_{\nu>\tau}\delta_L(\nu)=\max_{u\in I_\tau}D_\tau(u)=D_\tau(u_\tau^*)=\delta_\tau^{\mathrm{crit}}, ν > τ max δ L ( ν ) = u ∈ I τ max D τ ( u ) = D τ ( u τ ∗ ) = δ τ crit ,
the maximum being attained at the interior stationary point u τ ∗ u_\tau^* u τ ∗ of Definition B.3 , i.e. at ν = N τ ( u τ ∗ ) = ν τ c r i t \nu=N_\tau(u_\tau^*)=\nu_\tau^{\mathrm{crit}} ν = N τ ( u τ ∗ ) = ν τ crit . Since δ ˉ < δ τ c r i t \bar{\delta}<\delta_\tau^{\mathrm{crit}} δ ˉ < δ τ crit , Lemma B.2 gives strict Schur stability for every 0 < δ ≤ δ ˉ 0<\delta\le\bar{\delta} 0 < δ ≤ δ ˉ . The rate statement is then Theorem 3.22 applied with δ L = δ ˉ \delta_L=\bar{\delta} δ L = δ ˉ (c 2 < δ L < δ τ c r i t c^2<\delta_L<\delta_\tau^{\mathrm{crit}} c 2 < δ L < δ τ crit ) and this ν \nu ν , and the supremum claim for c τ c r i t = δ τ c r i t c_\tau^{\mathrm{crit}}=\sqrt{\delta_\tau^{\mathrm{crit}}} c τ crit = δ τ crit follows because ⋅ \sqrt{\mathord{\cdot}} ⋅ is increasing and δ τ c r i t \delta_\tau^{\mathrm{crit}} δ τ crit is the maximum of δ L ( ν ) \delta_L(\nu) δ L ( ν ) .
Two remarks delimit the scope of Corollary B.4 . First, it is an optimality statement within the family and ansatz of Theorem 3.22: it identifies the supremum of certifiable rate constants from the quadratic limiting polynomial P ∞ P_\infty P ∞ under the parametrization α = δ ˉ / L \alpha=\bar{\delta}/L α = δ ˉ / L , β = 1 − 2 δ ˉ q \beta=1-2\sqrt{\bar{\delta}}q β = 1 − 2 δ ˉ q , with ν \nu ν free and 0 < δ ˉ < δ L ( ν ) 0<\bar{\delta}<\delta_L(\nu) 0 < δ ˉ < δ L ( ν ) . It is not a lower bound over arbitrary delayed first-order methods; that role is played by the span-model bound of Theorem 2.1 . Second, this design constant governs the quadratic case and the verification of hypothesis (H1) at the midpoint gain (§ 5.3.1 ). It is distinct from the restriction c < 2 / 3 c<2/3 c < 2/3 in the main theorem, which arises in the low-frequency analysis of the rate-weighted Zames–Falb certificate used to verify hypothesis (H2) . Specifically, the coefficient λ = ( τ + 1 ) ( 2 − 3 c ) / M \lambda=(\tau+1)(2-3c)/M λ = ( τ + 1 ) ( 2 − 3 c ) / M in (233) is positive for c < 2 / 3 c<2/3 c < 2/3 , as required for the strict negativity estimate (234) .
For the rigorous all-class result we do not need the exact optimizer: Theorem 3.25 adopts the slightly conservative closed-form choice ν = M − 1 2 \nu=M-\frac{1}{2} ν = M − 2 1 and δ L = M − 2 \delta_L=M^{-2} δ L = M − 2 , for which Lemma 3.24 excludes every unit-circle crossing on 0 < δ ≤ M − 2 0<\delta\le M^{-2} 0 < δ ≤ M − 2 and yields the rate constant c < 1 / M c<1/M c < 1/ M . As computed in the next subsection, this conservative choice has the same delay scaling as the optimal one—ν τ c r i t = Θ ( τ ) \nu_\tau^{\mathrm{crit}}=\Theta(\tau) ν τ crit = Θ ( τ ) and c τ c r i t = Θ ( 1 / τ ) c_\tau^{\mathrm{crit}}=\Theta(1/\tau) c τ crit = Θ ( 1/ τ ) , matching ν = M − 1 2 ∼ 2 τ \nu=M-\frac{1}{2}\sim2\tau ν = M − 2 1 ∼ 2 τ and c < 1 / M ∼ 1 / ( 2 τ ) c<1/M\sim1/(2\tau) c < 1/ M ∼ 1/ ( 2 τ ) —so the two agree in order and differ only in constants.
Consequently, c τ c r i t = δ τ c r i t c_\tau^{\mathrm{crit}}=\sqrt{\delta_\tau^{\mathrm{crit}}} c τ crit = δ τ crit is the supremum of the rate constants certifiable by Theorem 3.22 over the admissible family { ν > τ } \{\nu>\tau\} { ν > τ } . Indeed, for every fixed τ \tau τ and every 0 < c < c τ c r i t 0<c<c_\tau^{\mathrm{crit}} 0 < c < c τ crit , choose
c 2 < δ ˉ < δ τ c r i t . c^2<\bar{\delta}<\delta_\tau^{\mathrm{crit}}. c 2 < δ ˉ < δ τ crit .
Then the choice ν = ν τ c r i t \nu=\nu_\tau^{\mathrm{crit}} ν = ν τ crit , α = δ ˉ / L \alpha=\bar{\delta}/L α = δ ˉ / L , and β = 1 − 2 δ ˉ q \beta=1-2\sqrt{\bar{\delta}}q β = 1 − 2 δ ˉ q yields ρ τ , κ ≤ 1 − c q \rho_{\tau,\kappa}\le1-cq ρ τ , κ ≤ 1 − c q for all sufficiently large κ \kappa κ . No admissible ν \nu ν yields a larger first-crossing threshold than δ τ c r i t \delta_\tau^{\mathrm{crit}} δ τ crit .
The delay scaling law: acceleration degrades as 1 / τ 1/\tau 1/ τ
Having solved the design problem exactly, we now quantify how the optimal constants behave as the delay grows. The interval I τ = ( 0 , π / ( 2 M ) ) I_\tau=(0,\pi/(2M)) I τ = ( 0 , π / ( 2 M )) shrinks like 1 / M 1/M 1/ M , so the natural variable is the rescaled frequency a : = M u ∈ ( 0 , π / 2 ) a:=Mu\in(0,\pi/2) a := M u ∈ ( 0 , π /2 ) , on which the maximizer u τ ∗ u_\tau^* u τ ∗ concentrates.
Proposition B.5 (delay scaling of the critical constants). As τ → ∞ \tau\to\infty τ → ∞ ,
δ τ c r i t = a ∗ 2 cos a ∗ τ 2 ( 1 + o ( 1 ) ) , c τ c r i t = a ∗ cos a ∗ τ ( 1 + o ( 1 ) ) , ν τ c r i t = 2 a ∗ 2 τ ( 1 + o ( 1 ) ) , (315) \delta_\tau^{\mathrm{crit}}=\frac{a_*^2\cos a_*}{\tau^2}(1+o(1)),\qquad c_\tau^{\mathrm{crit}}=\frac{a_*\sqrt{\cos a_*}}{\tau}(1+o(1)),\qquad\nu_\tau^{\mathrm{crit}}=\frac{2}{a_*^2}\tau(1+o(1)), \tag*{(315)} δ τ crit = τ 2 a ∗ 2 cos a ∗ ( 1 + o ( 1 )) , c τ crit = τ a ∗ cos a ∗ ( 1 + o ( 1 )) , ν τ crit = a ∗ 2 2 τ ( 1 + o ( 1 )) , (315) where a ∗ ∈ ( 0 , π 2 ) a_*\in(0,\frac{\pi}{2}) a ∗ ∈ ( 0 , 2 π ) is the unique solution of
a ∗ tan a ∗ = 2 , (316) a_*\tan a_*=2, \tag*{(316)} a ∗ tan a ∗ = 2 , (316) Numerically, a ∗ ≈ 1.0768 a_*\approx1.0768 a ∗ ≈ 1.0768 , a ∗ cos a ∗ ≈ 0.7415 a_*\sqrt{\cos a_*}\approx0.7415 a ∗ cos a ∗ ≈ 0.7415 and 2 / a ∗ 2 ≈ 1.7246 2/a_*^2\approx1.7246 2/ a ∗ 2 ≈ 1.7246 .
Proof. Substituting u = a / M u=a/M u = a / M into (305) and using sin ( a / M ) = a / M + O ( M − 3 ) \sin(a/M)=a/M+O(M^{-3}) sin ( a / M ) = a / M + O ( M − 3 ) and cos ( a / M ) = 1 + O ( M − 2 ) \cos(a/M)=1+O(M^{-2}) cos ( a / M ) = 1 + O ( M − 2 ) ,
M 2 D τ ( a M ) = 4 M 2 sin 2 a M cos a cos ( a / M ) = 4 a 2 cos a + O ( M − 2 ) , (317) M^2D_\tau\left(\frac{a}{M}\right)=4M^2\sin^2\frac{a}{M}\frac{\cos a}{\cos(a/M)}=4a^2\cos a+O(M^{-2}), \tag*{(317)} M 2 D τ ( M a ) = 4 M 2 sin 2 M a cos ( a / M ) cos a = 4 a 2 cos a + O ( M − 2 ) , (317)
uniformly for a a a in compact subsets of ( 0 , π 2 ) (0,\frac{\pi}{2}) ( 0 , 2 π ) . For the extrapolation, 2 τ / M = 1 − 1 / M 2\tau/M=1-1/M 2 τ / M = 1 − 1/ M , so sin ( 2 τ u ) = sin ( a − a / M ) = sin a − a cos a M + O ( M − 2 ) \sin(2\tau u)=\sin(a-a/M)=\sin a-\frac{a\cos a}{M}+O(M^{-2}) sin ( 2 τ u ) = sin ( a − a / M ) = sin a − M a c o s a + O ( M − 2 ) and 2 sin ( a / M ) cos a = 2 a cos a M ( 1 + O ( M − 2 ) ) 2\sin(a/M)\cos a=\frac{2a\cos a}{M}(1+O(M^{-2})) 2 sin ( a / M ) cos a = M 2 a c o s a ( 1 + O ( M − 2 )) ; hence
N τ ( a M ) = sin ( 2 τ a / M ) 2 sin ( a / M ) cos a = M 2 a tan a + O ( 1 ) , (318) N_\tau\left(\frac{a}{M}\right)=\frac{\sin(2\tau a/M)}{2\sin(a/M)\cos a}=\frac{M}{2a}\tan a+O(1), \tag*{(318)} N τ ( M a ) = 2 sin ( a / M ) cos a sin ( 2 τ a / M ) = 2 a M tan a + O ( 1 ) , (318)
again uniformly on compact subsets of ( 0 , π 2 ) (0,\frac{\pi}{2}) ( 0 , 2 π ) .
We first locate the maximizer. Write f M ( a ) : = M 2 D τ ( a / M ) f_M(a):=M^2D_\tau(a/M) f M ( a ) := M 2 D τ ( a / M ) and f ∞ ( a ) : = 4 a 2 cos a f_\infty(a):=4a^2\cos a f ∞ ( a ) := 4 a 2 cos a , so that (317) states f M → f ∞ f_M\to f_\infty f M → f ∞ uniformly on compact subsets of ( 0 , π 2 ) (0,\frac{\pi}{2}) ( 0 , 2 π ) . The limit profile f ∞ f_\infty f ∞ vanishes as a → 0 + a\to0^+ a → 0 + and as a → ( π 2 ) − a\to(\frac{\pi}{2})^- a → ( 2 π ) − and is positive between, so it attains its maximum at an interior point; its derivative 4 a ( 2 cos a − a sin a ) 4a(2\cos a-a\sin a) 4 a ( 2 cos a − a sin a ) vanishes on ( 0 , π 2 ) (0,\frac{\pi}{2}) ( 0 , 2 π ) exactly where a tan a = 2 a\tan a=2 a tan a = 2 , i.e. at a = a ⋆ a=a_\star a = a ⋆ , which is therefore the unique maximizer of f ∞ f_\infty f ∞ .
We claim that the maximizer a τ ⋆ : = M u τ ⋆ a_\tau^\star:=Mu_\tau^\star a τ ⋆ := M u τ ⋆ of f M f_M f M converges to a ⋆ a_\star a ⋆ . Because ( 0 , π 2 ) (0,\frac{\pi}{2}) ( 0 , 2 π ) is open, uniform convergence on compact subsets does not by itself preclude a τ ⋆ a_\tau^\star a τ ⋆ from approaching an endpoint. We therefore first establish uniform endpoint control. For every a ∈ ( 0 , π 2 ) a\in(0,\frac{\pi}{2}) a ∈ ( 0 , 2 π ) and M ≥ 3 M\ge3 M ≥ 3 ,
0 ≤ f M ( a ) = 4 M 2 sin 2 ( a / M ) cos a cos ( a / M ) ≤ 8 3 a 2 cos a , 0\le f_M(a)=4M^2\sin^2(a/M)\frac{\cos a}{\cos(a/M)}\le\frac{8}{\sqrt{3}}a^2\cos a, 0 ≤ f M ( a ) = 4 M 2 sin 2 ( a / M ) cos ( a / M ) cos a ≤ 3 8 a 2 cos a ,
where we used sin ( a / M ) ≤ a / M \sin(a/M)\le a/M sin ( a / M ) ≤ a / M and cos ( a / M ) ≥ cos ( π / 6 ) = 3 / 2 \cos(a/M)\ge\cos(\pi/6)=\sqrt{3}/2 cos ( a / M ) ≥ cos ( π /6 ) = 3 /2 . Since a 2 cos a a^2\cos a a 2 cos a tends to zero as a → 0 + a\to0^+ a → 0 + and as a → ( π 2 ) − a\to(\frac{\pi}{2})^- a → ( 2 π ) − , the profiles f M f_M f M are uniformly small near both endpoints.
Fix η ∈ ( 0 , π 2 ) \eta\in(0,\frac{\pi}{2}) η ∈ ( 0 , 2 π ) small enough that a ⋆ ∈ K : = [ η , π 2 − η ] a_\star\in K:=[\eta,\frac{\pi}{2}-\eta] a ⋆ ∈ K := [ η , 2 π − η ] , and set V : = f ∞ ( a ⋆ ) > 0 V:=f_\infty(a_\star)>0 V := f ∞ ( a ⋆ ) > 0 . Shrinking η \eta η if necessary, the preceding bound and the corresponding endpoint behavior of f ∞ f_\infty f ∞ give
f M ( a ) < V 3 and f ∞ ( a ) < V 3 , on ( 0 , η ] ∪ [ π 2 − η , π 2 ) f_M(a)<\frac{V}{3}\quad\text{and}\quad f_\infty(a)<\frac{V}{3},\quad\text{on}\quad(0,\eta]\cup\left[\frac{\pi}{2}-\eta,\frac{\pi}{2}\right) f M ( a ) < 3 V and f ∞ ( a ) < 3 V , on ( 0 , η ] ∪ [ 2 π − η , 2 π )
for all sufficiently large M M M . On the other hand, by the uniform convergence in (317) on K K K , f M ( a ⋆ ) → f ∞ ( a ⋆ ) = V f_M(a_\star)\to f_\infty(a_\star)=V f M ( a ⋆ ) → f ∞ ( a ⋆ ) = V , and hence f M ( a ⋆ ) > 2 V / 3 f_M(a_\star)>2V/3 f M ( a ⋆ ) > 2 V /3 for all sufficiently large M M M . Therefore no maximizer of f M f_M f M can lie in the endpoint strips. Thus, a τ ⋆ ∈ K a_\tau^\star\in K a τ ⋆ ∈ K for all sufficiently large M M M .
Confined to K K K , the maximizer now converges by the standard argument. Uniform convergence on K K K gives max K f M − max K f ∞ ≤ sup K ∣ f M − f ∞ ∣ → 0 \max_K f_M-\max_K f_\infty\le\sup_K|f_M-f_\infty|\to0 max K f M − max K f ∞ ≤ sup K ∣ f M − f ∞ ∣ → 0 , so f M ( a τ ⋆ ) = max K f M → max K f ∞ = f ∞ ( a ⋆ ) f_M(a_\tau^\star)=\max_K f_M\to\max_K f_\infty=f_\infty(a_\star) f M ( a τ ⋆ ) = max K f M → max K f ∞ = f ∞ ( a ⋆ ) . If a τ j ⋆ → a ′ a_{\tau_j}^\star\to a' a τ j ⋆ → a ′ along a subsequence, then uniform convergence of f M j f_{M_j} f M j to f ∞ f_\infty f ∞ on K K K together with the continuity of f ∞ f_\infty f ∞ yields f M j ( a τ j ⋆ ) → f ∞ ( a ′ ) f_{M_j}(a_{\tau_j}^\star)\to f_\infty(a') f M j ( a τ j ⋆ ) → f ∞ ( a ′ ) ; comparing with f M j ( a τ j ⋆ ) = max K f M j → max K f ∞ = f ∞ ( a ⋆ ) f_{M_j}(a_{\tau_j}^\star)=\max_K f_{M_j}\to\max_K f_\infty=f_\infty(a_\star) f M j ( a τ j ⋆ ) = max K f M j → max K f ∞ = f ∞ ( a ⋆ ) gives f ∞ ( a ′ ) = f ∞ ( a ⋆ ) f_\infty(a')=f_\infty(a_\star) f ∞ ( a ′ ) = f ∞ ( a ⋆ ) , so a ′ a' a ′ maximizes f ∞ f_\infty f ∞ on K K K . This forces a ′ = a ⋆ a'=a_\star a ′ = a ⋆ . As every subsequential limit equals a ⋆ a_\star a ⋆ , we conclude a τ ⋆ → a ⋆ a_\tau^\star\to a_\star a τ ⋆ → a ⋆ . Therefore
max u ∈ I τ D τ ( u ) = 1 M 2 f M ( a τ ⋆ ) = 4 M 2 ( a ⋆ 2 cos a ⋆ ) ( 1 + o ( 1 ) ) . \max_{u\in I_\tau}D_\tau(u)=\frac{1}{M^2}f_M(a_\tau^\star)=\frac{4}{M^2}(a_\star^2\cos a_\star)(1+o(1)). u ∈ I τ max D τ ( u ) = M 2 1 f M ( a τ ⋆ ) = M 2 4 ( a ⋆ 2 cos a ⋆ ) ( 1 + o ( 1 )) .
This is δ τ c r i t \delta_\tau^{\mathrm{crit}} δ τ crit ; since M = 2 τ + 1 = 2 τ ( 1 + o ( 1 ) ) M=2\tau+1=2\tau(1+o(1)) M = 2 τ + 1 = 2 τ ( 1 + o ( 1 )) ,
δ τ c r i t = 4 a ⋆ 2 cos a ⋆ M 2 ( 1 + o ( 1 ) ) = a ⋆ 2 cos a ⋆ τ 2 ( 1 + o ( 1 ) ) , \delta_\tau^{\mathrm{crit}}=\frac{4a_\star^2\cos a_\star}{M^2}(1+o(1))=\frac{a_\star^2\cos a_\star}{\tau^2}(1+o(1)), δ τ crit = M 2 4 a ⋆ 2 cos a ⋆ ( 1 + o ( 1 )) = τ 2 a ⋆ 2 cos a ⋆ ( 1 + o ( 1 )) ,
and c τ c r i t = δ τ c r i t c_\tau^{\mathrm{crit}}=\sqrt{\delta_\tau^{\mathrm{crit}}} c τ crit = δ τ crit gives the stated expression. Finally, evaluating (318) at a τ ⋆ → a ⋆ a_\tau^\star\to a_\star a τ ⋆ → a ⋆ and using tan a ⋆ = 2 / a ⋆ \tan a_\star=2/a_\star tan a ⋆ = 2/ a ⋆ from (316) ,
ν τ c r i t = N τ ( u τ ⋆ ) = M 2 a ⋆ tan a ⋆ ( 1 + o ( 1 ) ) = M a ⋆ 2 ( 1 + o ( 1 ) ) = 2 a ⋆ 2 τ ( 1 + o ( 1 ) ) , \nu_\tau^{\mathrm{crit}}=N_\tau(u_\tau^\star)=\frac{M}{2a_\star}\tan a_\star(1+o(1))=\frac{M}{a_\star^2}(1+o(1))=\frac{2}{a_\star^2}\tau(1+o(1)), ν τ crit = N τ ( u τ ⋆ ) = 2 a ⋆ M tan a ⋆ ( 1 + o ( 1 )) = a ⋆ 2 M ( 1 + o ( 1 )) = a ⋆ 2 2 τ ( 1 + o ( 1 )) ,
again with M = 2 τ ( 1 + o ( 1 ) ) M=2\tau(1+o(1)) M = 2 τ ( 1 + o ( 1 )) . The numerical values follow from a ⋆ ≈ 1.0768739863 a_\star\approx1.0768739863 a ⋆ ≈ 1.0768739863 .