Introduction
This paper develops a hybrid convex–nonconvex augmented Lagrangian (AL) variant of HALLaR of [44 ] , namely HALLaR+, which can solve SDPs, in general standard form, i.e.,
min X ∈ S n { ⟨ C , X ⟩ : A X = b , X ⪰ 0 } (1) \min_{X \in\mathbb{S}^{n}} \{\langle C, X\rangle: \mathcal{A}X=b,\ X \succeq0\} \tag*{(1)} X ∈ S n min {⟨ C , X ⟩ : A X = b , X ⪰ 0 } (1)
where R m \mathbb{R}^{m} R m is the m m m -dimensional Euclidean space equipped with the standard inner product, S n \mathbb{S}^{n} S n is the space of n × n n \times n n × n real symmetric matrices equipped with the Frobenius inner product, ⪰ 0 \succeq0 ⪰ 0 denotes the Löwner partial order on S n \mathbb{S}^{n} S n , so that X ⪰ 0 X \succeq0 X ⪰ 0 means X X X is positive semidefinite (PSD), C ∈ S n C \in\mathbb{S}^{n} C ∈ S n , b ∈ R m b \in\mathbb{R}^{m} b ∈ R m , and A : S n → R m \mathcal{A} : \mathbb{S}^{n} \to\mathbb{R}^{m} A : S n → R m is a linear operator with adjoint A ∗ : R m → S n \mathcal{A}^{*} : \mathbb{R}^{m} \to\mathbb{S}^{n} A ∗ : R m → S n . In contrast, HALLaR can only solve SDPs where one of the constraints in (1) is a bounded trace constraint, i.e., tr ( X ) ≤ τ \operatorname{tr}(X) \le\tau tr ( X ) ≤ τ for some τ > 0 \tau> 0 τ > 0 . Even though a trace constraint can always be added to (1) with a constant τ \tau τ that preserves at least one optimal solution, choosing such a τ \tau τ in practice is challenging: a value that is too small may exclude all optimal solutions, whereas one that is too large may substantially increase the computational effort.
Approach and relation to HALLaR. HALLaR+ is an inexact AL method that generates sequences { Z t } \{Z_t\} { Z t } and { p t } \{p_t\} { p t } according to the recursion
Z t ≈ arg min X ∈ S n { L β t − 1 ( X ; p t − 1 ) : = ⟨ C , X ⟩ + ⟨ p t − 1 , b − A X ⟩ + β t − 1 2 ∥ b − A X ∥ 2 : X ⪰ 0 } , (2) Z_t \approx\underset{X \in\mathbb{S}^{n}}{\operatorname{arg\,min}}\left\{\mathcal{L}_{\beta_{t-1}}(X;p_{t-1}) := \langle C,X\rangle+ \langle p_{t-1},b-\mathcal{A}X\rangle+ \frac{\beta_{t-1}}{2}\lVert b-\mathcal{A}X\rVert^{2} : X \succeq0\right\}, \tag*{(2)} Z t ≈ X ∈ S n arg min { L β t − 1 ( X ; p t − 1 ) := ⟨ C , X ⟩ + ⟨ p t − 1 , b − A X ⟩ + 2 β t − 1 ∥ b − A X ∥ 2 : X ⪰ 0 } , (2)
p t = p t − 1 + β t − 1 ( b − A Z t ) , β t = γ β t − 1 , (3) p_t = p_{t-1}+\beta_{t-1}(b-\mathcal{A}Z_t),\qquad\beta_t=\gamma\beta_{t-1}, \tag*{(3)} p t = p t − 1 + β t − 1 ( b − A Z t ) , β t = γ β t − 1 , (3)
with γ = 2 \gamma= 2 γ = 2 . HALLaR differs from the above updates (2a) and (2b) in two aspects. First, as HALLAR solves SDPs over the set { X ∈ S n : A ( X ) = b , tr ( X ) ≤ τ , X ⪰ 0 } \{X \in\mathbb{S}^{n}: \mathcal{A}(X) = b,\ \operatorname{tr}(X) \le\tau,\ X \succeq0\} { X ∈ S n : A ( X ) = b , tr ( X ) ≤ τ , X ⪰ 0 } , its AL subproblems are obtained by dualizing A ( X ) = b \mathcal{A}(X) = b A ( X ) = b , while keeping tr ( X ) ≤ τ \operatorname{tr}(X) \le\tau tr ( X ) ≤ τ and X ⪰ 0 X \succeq0 X ⪰ 0 as constraints. Hence, in contrast to (2a), their feasible regions are bounded. Second, the theoretical analysis of HALLaR assumes γ = 1 \gamma= 1 γ = 1 , and hence that the penalty parameters are kept constant.
Both HALLaR and HALLaR+ use Frank–Wolfe (FW), or conditional-gradient, variants to approximately solve their AL subproblem (2a), initialized at Z t − 1 Z_{t-1} Z t − 1 . However, because the AL subproblems arising in HALLaR+ have an unbounded feasible region, the FW variant used to solve them relies on substantially different techniques from the one used by HALLaR. Both FW variants generate matrices in factored form, i.e., X = U U T X = UU^{\mathsf{T}} X = U U T , where the number of columns r r r of U U U varies during the algorithm but is usually much smaller than n n n . This low-rank representation reduces memory footprint from O ( n 2 ) O(n^2) O ( n 2 ) to O ( n r ) O(nr) O ( n r ) and exploits sparsity in the SDP data [8 , 9 , 44 , 2 ] . Both X X X and U U U are viewed as iterates in our discussion, with U U U referred to as a factor of X X X . For a matrix V V V , col ( V ) \operatorname{col}(V) col ( V ) denotes its number of columns. Starting from a low-rank factor U ~ ∈ R n × r \widetilde{U} \in\mathbb{R}^{n \times r} U ∈ R n × r , an iteration of both FW variants consists of the following two basic steps:
find a factor U ∈ R n × r U \in\mathbb{R}^{n \times r} U ∈ R n × r such that col ( U ) ≤ col ( U ~ ) \operatorname{col}(U) \le\operatorname{col}(\widetilde{U}) col ( U ) ≤ col ( U ) and L β t − 1 ( U U T ; p t − 1 ) ≤ L β t − 1 ( U ~ U ~ T ; p t − 1 ) \mathcal{L}_{\beta_{t-1}}(UU^{\mathsf{T}};p_{t-1}) \le\mathcal{L}_{\beta_{t-1}}(\widetilde{U}\widetilde{U}^{\mathsf{T}};p_{t-1}) L β t − 1 ( U U T ; p t − 1 ) ≤ L β t − 1 ( U U T ; p t − 1 ) ;
without forming the associated full matrix, check whether U U T UU^{\mathsf{T}} U U T satisfies the prescribed approximate optimality conditions for (2a). If so, terminate and set Z t = U U T Z_t = UU^{\mathsf{T}} Z t = U U T ; otherwise, perform an FW-type step based at U U T UU^{\mathsf{T}} U U T to escape from this spurious solution U U U to obtain a subsequent low-rank factor U ~ + \widetilde{U}^{+} U + satisfying col ( U ~ + ) ≤ col ( U ) + 1 \operatorname{col}(\widetilde{U}^{+}) \le\operatorname{col}(U)+1 col ( U + ) ≤ col ( U ) + 1 , and hence col ( U ~ + ) ≤ col ( U ~ ) + 1 \operatorname{col}(\widetilde{U}^{+}) \le\operatorname{col}(\widetilde{U})+1 col ( U + ) ≤ col ( U ) + 1 .
We now discuss (i) in greater detail. The choice U = U ~ U = \widetilde{U} U = U trivially satisfies step i) and has been used by methods such as CGAL and T-CGAL [66 , 67 ] (see also the methods in [68 , 16 , 52 ] for similar limited memory or reduced storage variants). Such methods can be viewed as essentially pure Frank-Wolfe methods, but have the disadvantage of producing iterates with high number of columns, since steps (i)–(ii) must generally be repeated many times. On the other hand, to implement the second part of (i), HALLaR and HALLaR+ both use the nonlinear solver ADAP-AIPP, initialized at U ~ \widetilde{U} U , to find a near-stationary point U U U of min V ∈ R n × r L β t − 1 ( V V T ; p t − 1 ) \min_{V \in\mathbb{R}^{n \times r}} \mathcal{L}_{\beta_{t-1}}(VV^{\mathsf{T}};p_{t-1}) min V ∈ R n × r L β t − 1 ( V V T ; p t − 1 ) satisfying the descent condition in step i). Since the objective of this unconstrained subproblem is nonconvex and has only a locally Lipschitz continuous gradient, establishing the required complexity guarantees for ADAP-AIPP in this setting requires a more delicate analysis than the one used in HALLaR. The additional difficulty arises from the fact that, whereas the complexity of HALLaR depends on the diameter of the bounded feasible set, the complexity of HALLaR+ depends on a uniform bound on the diameters of all generated sublevel sets, some of which depend on both β t − 1 \beta_{t-1} β t − 1 and p t − 1 p_{t-1} p t − 1 .
We next discuss (ii). Unlike HALLaR, which uses a computable FW gap to estimate whether the primal gap of min X ⪰ 0 { L β t − 1 ( X ; p t − 1 ) : X ⪰ 0 , tr ( X ) ≤ 1 } \min_{X \succeq0}\{\mathcal{L}_{\beta_{t-1}}(X;p_{t-1}): X \succeq0,\ \operatorname{tr}(X) \le1\} min X ⪰ 0 { L β t − 1 ( X ; p t − 1 ) : X ⪰ 0 , tr ( X ) ≤ 1 } at X = U U T X = UU^{\mathsf{T}} X = U U T is small, HALLaR+ uses approximate optimality conditions tailored to the unbounded subproblem (2). Specifically, with S = ∇ X L β t − 1 ( U U T ; p t − 1 ) S = \nabla_X\mathcal{L}_{\beta_{t-1}}(UU^{\mathsf{T}};p_{t-1}) S = ∇ X L β t − 1 ( U U T ; p t − 1 ) , it checks whether λ min ( S ) \lambda_{\min}(S) λ m i n ( S ) is nearly nonnegative and ⟨ S , U U T ⟩ \langle S, UU^{\mathsf{T}}\rangle ⟨ S , U U T ⟩ is small. If either condition fails, HALLaR+ finds a rank-one matrix y ∗ ( y ∗ ) T y^*(y^*)^{\mathsf{T}} y ∗ ( y ∗ ) T by solving min X ⪰ 0 { ⟨ S , X ⟩ : tr ( X ) ≤ 1 } \min_{X \succeq0}\{\langle S,X\rangle:\operatorname{tr}(X) \le1\} min X ⪰ 0 {⟨ S , X ⟩ : tr ( X ) ≤ 1 } and forms the next iterate as a conic combination of this matrix and the current iterate, with the two coefficients of the conic combination chosen by a two-dimensional line search.
This approach for solving each AL subproblem is closely related to those of Laue [35 ] and Duchi et al. [17 , 36 ] , which develop FW variants for the generic static problem of minimizing a convex function ϕ ( ⋅ ) \phi(\cdot) ϕ ( ⋅ ) over S + n \mathbb{S}^{n}_{+} S + n . The two-dimensional line search for the conic coefficients used in our FW variant is motivated by the scheme of Laue. However, whereas Laue’s convergence analysis is based on a primal-gap criterion that is not directly checkable in practice, our method uses the aforementioned tractable termination criteria. Duchi et al., on the other hand, propose a different unbounded FW variant based on two successive one-dimensional line-searches rather than a single two-dimensional line search. Like our FW variant, their method terminates based on checkable approximate optimality conditions. While closely related to these methods, our FW variant requires a more involved analysis because it must account for the changing AL subproblems in (2) and the dynamics of their associated sublevel sets, a difficulty absent from the static settings considered by Laue and Duchi et al.
Our results and computational impact. Assuming that a primal–dual KKT pair exists and that the set of optimal solutions of (1) is nonempty and bounded, it is shown that HALLaR+ obtains an approximate primal–dual solution of (1) and its dual with provable computational complexity bounds expressed in terms of parameters associated with the SDP instance and user-specified tolerances. In particular, if the tolerances on primal feasibility, dual feasibility, and complementarity are all equal to ϵ ∈ ( 0 , 1 ] \epsilon\in(0,1] ϵ ∈ ( 0 , 1 ] , then HALLaR+ performs O ( log ( 1 / ϵ ) ) O(\log(1/\epsilon)) O ( log ( 1/ ϵ )) AL iterations, O ( ϵ − 2 ) O(\epsilon^{-2}) O ( ϵ − 2 ) FW iterations/minimum eigenvector computations, and O ( ϵ − 3 ) O(\epsilon^{-3}) O ( ϵ − 3 ) ADAP-AIPP iterations. Our GPU implementation of HALLaR+ is faster than cuHALLaR [2 ] on 12 of 14 Hamming and 24 of 28 GSET maximum stable set instances, as well as all 26 matrix completion instances. On a single NVIDIA H200, it reaches 10 − 5 10^{-5} 1 0 − 5 relative precision in about 44 seconds on a Hamming SDP (n ≈ 8.4 × 10 6 n \approx8.4 \times10^{6} n ≈ 8.4 × 1 0 6 , m ≈ 9.7 × 10 7 m \approx9.7 \times10^{7} m ≈ 9.7 × 1 0 7 ) and 31 seconds on a matrix completion SDP (n = 2 × 10 6 n = 2 \times10^{6} n = 2 × 1 0 6 , m ≈ 2.6 × 10 8 m \approx2.6 \times10^{8} m ≈ 2.6 × 1 0 8 ). On the other hand, cuLoRADS [23 ] fails on the Hamming instance and takes about 4,100 seconds on the matrix completion instance. At 10 − 4 10^{-4} 1 0 − 4 relative precision, cuHALLaR+ solves 80 of 81 QAPLIB relaxations, compared with 56 for CSDP [5 ] , 17 for HALLaR, and one for cuLoRADS. On the Mittelmann benchmarks [43 ] , cuHALLaR+ solves 27 instances within one hour, cuLoRADS solves 15, and CARDAL [37 ] solves 14. The final cuHALLaR+ iterate has rank two on every Hamming instance, at most r + 1 r + 1 r + 1 on every matrix completion instance with ground-truth rank r r r , and 465 on the largest random quadratic assignment instance.
Related works. Interior-point methods are widely used to solve small and medium-scale SDPs to high accuracy. Their applicability to large-scale instances is limited, however, because each Newton-type iteration typically requires solving a large linear system, often through the factorization of a large and potentially dense matrix [5 , 3 , 45 , 59 ] . We therefore focus below on methods specifically designed for large-scale SDPs.
SDPNAL+ [69 , 65 ] is an augmented Lagrangian (AL) method that approximately solves each AL subproblem using a semismooth Newton–CG method. Spectral bundle methods, and more generally bundle methods, have also been developed for large-scale SDPs (see, for example, [25 , 10 , 24 , 14 , 46 , 4 ] ). Another substantial line of work applies the alternating direction method of multipliers (ADMM) and related operator-splitting methods to large-scale SDPs (e.g., [47 , 20 , 13 , 40 , 70 , 33 , 12 , 63 , 48 ] ). Since methods based on nonconvex low-rank (LR) factorizations are most closely related to the approach developed in this paper, we discuss them in greater detail below. We refer the reader to the survey [41 ] for a broader overview of methods for solving large-scale SDPs.
The nonconvex LR approach introduced in [8 , 9 ] underlies many successful algorithms for solving important classes of SDPs. In particular, the SDPLR method developed in these works applies an AL method to the smooth nonconvex reformulation
min U { C ∙ U U T : A ( U U T ) = b , ∥ U ∥ F ≤ 1 , U ∈ R n × r } (4) \min_{U}\ \left\{ C \bullet UU^{T} \ :\ A(UU^{T}) = b,\quad\lVert U\rVert_{F} \le1,\quad U \in\mathbb{R}^{n \times r} \right\} \tag*{(4)} U min { C ∙ U U T : A ( U U T ) = b , ∥ U ∥ F ≤ 1 , U ∈ R n × r } (4)
and approximately solves its AL subproblems using a limited-memory BFGS method. Although SDPLR is designed to handle equality constraints, it can be modified to accommodate inequality constraints (see, for example, [34 ] ).
Several methods solve ( P r ) (P_r) ( P r ) without employing an AL framework. For MaxCut SDPs, several specialized algorithms solve ( P r ) (P_r) ( P r ) using optimization techniques that preserve feasibility throughout their execution (e.g., [7 , 27 , 18 , 42 , 32 ] ). Riemannian optimization methods have also been employed for structured classes of SDPs for which the feasible set of ( P r ) (P_r) ( P r ) is a smooth manifold (e.g., [51 , 32 , 30 , 42 ] ). Related feasibility-preserving methods based on manifold-optimization techniques have been developed for more challenging classes of SDPs in which the feasible sets of ( P r ) (P_r) ( P r ) are algebraic varieties [55 , 56 , 57 ] . Alternative feasibility-preserving strategies have also been explored in [50 ] .
More recently, several low-rank SDP solvers have combined low-rank factorization with augmented-Lagrangian-type schemes [31 , 6 ] . Like SDPLR, LoRADS and its GPU counterpart cuLoRADS [23 , 22 ] use the factorization X = U U T X = UU^{T} X = U U T in their first phase and apply an inexact AL method to ( P r ) (P_r) ( P r ) . In their second phase, these methods adopt the bilinear factorization X = U V T X = UV^{T} X = U V T , penalize the additional constraint U = V U = V U = V , and apply an alternating-minimization scheme to the resulting augmented Lagrangian function. The ALORA method [15 ] combines gradient descent with a spectral block Frank–Wolfe method to approximately minimize its nonconvex AL subproblems. CARDAL [37 ] combines rank-adaptive AL iterations with matrix-free L-BFGS and negative-curvature corrections, provides slack-based approximate optimality conditions, and supports distributed GPU execution. SDPF+ [58 ] combines a feasible low-rank method with an AL phase using a dynamically weighted quadratic penalty. Other low-rank methods focus on specially structured SDPs whose factorized nonconvex AL subproblems take the form arg min U ∈ M L β t − 1 ( U U T ; p t − 1 ) \mathop{\operatorname{arg\,min}}_{U \in\mathcal{M}} \mathcal{L}_{\beta_{t-1}}(UU^{T}; p_{t-1}) arg min U ∈ M L β t − 1 ( U U T ; p t − 1 ) , where M \mathcal{M} M is a Riemannian manifold. Such subproblems are solved using either a Riemannian trust-region method [60 , 61 ] or a Riemannian semismooth Newton method [62 ] . For SDP-RL and polyhedral-SDP relaxations, respectively, RiNNAL+ and RiNNAL-POP [29 , 28 ] combine low-rank AL iterations with projected-gradient steps in the original matrix variable to adjust the rank and escape spurious factor-space local minima.
Structure of the paper. This paper is organized into five sections and two appendices. Section 2 introduces the pair of primal-dual SDP problems and develops a general AL framework for solving them. Section 3 presents X-AL, an instance of this framework whose iterates are generated in the S + n \mathbb{S}_{+}^{n} S + n space, and establishes its main complexity result for solving the SDP pair. Section 4 introduces HALLaR+, a computationally and memory-efficient variant of X-AL implemented in the U U U -space. Theorem 4.1, the main theorem of the paper, establishes the complexity bounds for HALLaR+. Section 5 reports computational experiments comparing HALLaR+ with several existing solvers on a large collection of SDPs. Appendix A provides technical convex-analysis results used throughout the paper and develops the relaxed unbounded Frank–Wolfe (RUFW) framework used by X-AL. Finally, Appendix B presents the convergence analyses of an adaptive version of FISTA and an inexact proximal point method, both implemented in the U-space and used as subroutines within HAllaR+.
Basic definitions and notation
Let E \mathbb{E} E be a finite-dimensional inner-product space. We denote its inner product by ⟨ X , Y ⟩ \langle X,Y\rangle ⟨ X , Y ⟩ and the associated norm by ∥ X ∥ : = ⟨ X , X ⟩ \|X\| := \sqrt{\langle X,X\rangle} ∥ X ∥ := ⟨ X , X ⟩ . When E = S n \mathbb{E} = \mathbb{S}^{n} E = S n , we use the trace inner product ⟨ X , Y ⟩ : = tr ( X Y ) \langle X,Y\rangle:= \operatorname{tr}(XY) ⟨ X , Y ⟩ := tr ( X Y ) , whose associated norm is the Frobenius norm. We use the notation
log 2 + ( t ) : = max { 0 , log 2 t } , log 1 + ( t ) : = max { 1 , log t } , \log_{2}^{+}(t) := \max\{0,\log_{2}t\}, \qquad\log_{1}^{+}(t) := \max\{1,\log t\}, log 2 + ( t ) := max { 0 , log 2 t } , log 1 + ( t ) := max { 1 , log t } ,
and write O 1 ( Θ ) O_{1}(\Theta) O 1 ( Θ ) as shorthand for O ( 1 + Θ ) O(1+\Theta) O ( 1 + Θ ) . The operator norm of A \mathcal{A} A is ∥ A ∥ : = max { ∥ A X ∥ : ∥ X ∥ ≤ 1 } \|\mathcal{A}\| := \max\{\|\mathcal{A}X\| : \|X\| \le1\} ∥ A ∥ := max { ∥ A X ∥ : ∥ X ∥ ≤ 1 } .
We first recall some standard definitions from convex analysis. For a set Ω ⊆ E \Omega\subseteq\mathbb{E} Ω ⊆ E , its indicator function is denoted by δ Ω \delta_{\Omega} δ Ω , where δ Ω ( X ) = 0 \delta_{\Omega}(X)=0 δ Ω ( X ) = 0 if X ∈ Ω X\in\Omega X ∈ Ω and δ Ω ( X ) = ∞ \delta_{\Omega}(X)=\infty δ Ω ( X ) = ∞ otherwise. If Ω \Omega Ω is closed and convex, its normal cone at X ∈ Ω X\in\Omega X ∈ Ω is
N Ω ( X ) : = { V : ⟨ V , U − X ⟩ ≤ 0 , ∀ U ∈ Ω } . (5) N_{\Omega}(X) := \{V : \langle V,U-X\rangle\le0,\ \forall U\in\Omega\}. \tag*{(5)} N Ω ( X ) := { V : ⟨ V , U − X ⟩ ≤ 0 , ∀ U ∈ Ω } . (5)
For a proper closed convex function f : E → R ∪ { ∞ } f:\mathbb{E}\to\mathbb{R}\cup\{\infty\} f : E → R ∪ { ∞ } , a point X ∈ dom f X\in\operatorname{dom}f X ∈ dom f , and a scalar ε ≥ 0 \varepsilon\ge0 ε ≥ 0 , the ε \varepsilon ε -subdifferential of f f f at X X X is
∂ ε f ( X ) : = { V : f ( U ) ≥ f ( X ) + ⟨ V , U − X ⟩ − ε , ∀ U ∈ E } . \partial_{\varepsilon}f(X) := \{V : f(U) \ge f(X)+\langle V,U-X\rangle-\varepsilon,\ \forall U\in\mathbb{E}\}. ∂ ε f ( X ) := { V : f ( U ) ≥ f ( X ) + ⟨ V , U − X ⟩ − ε , ∀ U ∈ E } .
For a nonempty closed convex set Ω ⊆ E \Omega\subseteq\mathbb{E} Ω ⊆ E , the Euclidean projection of X ∈ E X\in\mathbb{E} X ∈ E onto Ω \Omega Ω is
proj Ω ( X ) : = arg min Y ∈ Ω ∥ X − Y ∥ (6) \operatorname{proj}_{\Omega}(X) := \mathop{\arg\min}_{Y\in\Omega}\|X-Y\| \tag*{(6)} proj Ω ( X ) := arg min Y ∈ Ω ∥ X − Y ∥ (6)
and the distance from a point q ∈ E q\in\mathbb{E} q ∈ E to Ω \Omega Ω is dist ( q , Ω ) : = inf { ∥ q − s ∥ : s ∈ Ω } \operatorname{dist}(q,\Omega):=\inf\{\|q-s\|:s\in\Omega\} dist ( q , Ω ) := inf { ∥ q − s ∥ : s ∈ Ω } .
Given a set Ω ⊆ E \Omega\subseteq\mathbb{E} Ω ⊆ E , a point X 0 ∈ Ω X_{0}\in\Omega X 0 ∈ Ω , and a function f : Ω → R f:\Omega\to\mathbb{R} f : Ω → R , we define the sublevel set of f f f at X 0 X_{0} X 0 and its radius by
S ( X 0 ; f ) : = { X ′ ∈ Ω : f ( X ′ ) ≤ f ( X 0 ) } , (7) S(X_{0};f) := \{X' \in\Omega: f(X') \le f(X_{0})\}, \tag*{(7)} S ( X 0 ; f ) := { X ′ ∈ Ω : f ( X ′ ) ≤ f ( X 0 )} , (7)
D ( X 0 ; f ) : = sup { ∥ X ′ ′ ∥ : X ′ ′ ∈ S ( X 0 ; f ) } . (8) D(X_{0};f) := \sup\{\|X''\| : X'' \in S(X_{0};f)\}. \tag*{(8)} D ( X 0 ; f ) := sup { ∥ X ′′ ∥ : X ′′ ∈ S ( X 0 ; f )} . (8)
We next introduce notation related to a differentiable function g : E → R g:\mathbb{E}\to\mathbb{R} g : E → R . Its affine approximation at z ˉ ∈ E \bar{z}\in\mathbb{E} z ˉ ∈ E is
ℓ g ( z ; z ˉ ) : = g ( z ˉ ) + ⟨ ∇ g ( z ˉ ) , z − z ˉ ⟩ ∀ z ∈ E . (9) \ell_{g}(z;\bar{z}) := g(\bar{z})+\langle\nabla g(\bar{z}),z-\bar{z}\rangle\qquad\forall z\in\mathbb{E}. \tag*{(9)} ℓ g ( z ; z ˉ ) := g ( z ˉ ) + ⟨ ∇ g ( z ˉ ) , z − z ˉ ⟩ ∀ z ∈ E . (9)
Function g g g is said to have a locally Lipschitz continuous gradient if, for every U ∈ E U\in\mathbb{E} U ∈ E , there exist constants R = R U > 0 R=R_{U}>0 R = R U > 0 and L ˉ U > 0 \bar{L}_{U}>0 L ˉ U > 0 such that
∥ ∇ g ( U ′ ) − ∇ g ( U ′ ′ ) ∥ ≤ L ˉ U ∥ U ′ − U ′ ′ ∥ , ∀ U ′ , U ′ ′ ∈ U + B ( R ) (10) \|\nabla g(U')-\nabla g(U'')\| \le\bar{L}_{U}\|U'-U''\|,\qquad\forall U',U''\in U+B(R) \tag*{(10)} ∥∇ g ( U ′ ) − ∇ g ( U ′′ ) ∥ ≤ L ˉ U ∥ U ′ − U ′′ ∥ , ∀ U ′ , U ′′ ∈ U + B ( R ) (10)
where B ( R ) : = { X ∈ E : ∥ X ∥ ≤ R } B(R):=\{X\in\mathbb{E}:\|X\|\le R\} B ( R ) := { X ∈ E : ∥ X ∥ ≤ R } . The Lipschitz constant of ∇ g ( ⋅ ) \nabla g(\cdot) ∇ g ( ⋅ ) and the maximum of ∥ ∇ g ( ⋅ ) ∥ \|\nabla g(\cdot)\| ∥∇ g ( ⋅ ) ∥ both relative to the ball B ( R ) B(R) B ( R ) are denoted, respectively, by
[ LIP ] ( R ; g ) : = inf { L > 0 : ∥ ∇ g ( X ) − ∇ g ( X ′ ) ∥ ≤ L ∥ X − X ′ ∥ ∀ X , X ′ ∈ B ( R ) } , (11) [\operatorname{LIP}](R;g) := \inf\{L>0:\|\nabla g(X)-\nabla g(X')\|\le L\|X-X'\|\ \ \forall X,X'\in B(R)\}, \tag*{(11)} [ LIP ] ( R ; g ) := inf { L > 0 : ∥∇ g ( X ) − ∇ g ( X ′ ) ∥ ≤ L ∥ X − X ′ ∥ ∀ X , X ′ ∈ B ( R )} , (11)
∇ ( R ; g ) : = sup { ∥ ∇ g ( U ) ∥ : U ∈ B ( R ) } . (12) \nabla(R;g) := \sup\{\|\nabla g(U)\|:U\in B(R)\}. \tag*{(12)} ∇ ( R ; g ) := sup { ∥∇ g ( U ) ∥ : U ∈ B ( R )} . (12)
The SDP problem and an inexact AL framework
This section develops an inexact augmented Lagrangian (AL) framework for computing approximate primal-dual solutions of an SDP. The framework is independent of the particular method used to solve the AL subproblems. At each outer iteration, the inner solver need only return a PSD matrix satisfying prescribed approximate first-order optimality conditions for the current AL subproblem. The multiplier is then updated, and the penalty parameter is increased whenever the primal residual remains above the target tolerance. This separation allows the outer-iteration complexity of the AL framework to be analyzed independently of the specific inner solver used to satisfy the required inexactness conditions.
The section is organized as follows. Subsection 2.1 introduces the primal-dual SDP pair, discusses its assumptions, and specifies the approximate primal-dual solution concept targeted by the framework. Subsection 2.2 formally presents the inexact AL framework. Finally, Subsection 2.3 proves Theorem 2.4, which gives the main outer-iteration complexity bound for the framework.
The SDP problem and assumptions
We consider the primal-dual pair of SDPs
min X ∈ S + n { ⟨ C , X ⟩ : A X = b } (13) \min_{X \in\mathbb{S}_{+}^{n}} \{\langle C,X\rangle: AX=b\} \tag*{(13)} X ∈ S + n min {⟨ C , X ⟩ : A X = b } (13)
and
max { ⟨ b , p ⟩ : C − A ∗ p ⪰ 0 , p ∈ R m } . (14) \max\{\langle b,p\rangle: C-A^{*}p \succeq0,\ p \in\mathbb{R}^{m}\}. \tag*{(14)} max {⟨ b , p ⟩ : C − A ∗ p ⪰ 0 , p ∈ R m } . (14)
In contrast to the formulation in [44 ] , which assumes a bounded feasible region, problem 11 is posed over the full PSD cone S + n \mathbb{S}_{+}^{n} S + n , without a constraint of the form tr ( X ) ≤ τ \operatorname{tr}(X) \leq\tau tr ( X ) ≤ τ . In the absence of such a bound, the analysis instead relies on properties of the SDP solution set.
We now state the assumptions that are made on the SDP data.
Assumption 2.1. The set of optimal solutions of (13) is nonempty and bounded.
Assumption 2.2. There exists an optimal primal dual pair ( X ∗ , p ∗ ) ∈ S + n × R m (X_{*},p_{*}) \in\mathbb{S}_{+}^{n} \times\mathbb{R}^{m} ( X ∗ , p ∗ ) ∈ S + n × R m satisfying
0 ∈ C − A ∗ p ∗ + N S + n ( X ∗ ) , A X ∗ = b . (15) 0 \in C-A^{*}p_{*}+N_{\mathbb{S}_{+}^{n}}(X_{*}), \qquad AX_{*}=b. \tag*{(15)} 0 ∈ C − A ∗ p ∗ + N S + n ( X ∗ ) , A X ∗ = b . (15)
The definition of the normal cone in (5) , applied with Ω = S + n \Omega=\mathbb{S}_{+}^{n} Ω = S + n and X = X ∗ X=X_{*} X = X ∗ implies that (15) is equivalent to
A X ∗ = b , X ∗ ⪰ 0 , C − A ∗ p ∗ ⪰ 0 , ⟨ C − A ∗ p ∗ , X ∗ ⟩ = 0. (16) AX_{*}=b,\qquad X_{*}\succeq0,\qquad C-A^{*}p_{*}\succeq0,\qquad\langle C-A^{*}p_{*},X_{*}\rangle=0. \tag*{(16)} A X ∗ = b , X ∗ ⪰ 0 , C − A ∗ p ∗ ⪰ 0 , ⟨ C − A ∗ p ∗ , X ∗ ⟩ = 0. (16)
Relation (16) expresses primal feasibility, dual feasibility, and complementarity. In particular, ( X ∗ , p ∗ ) (X_{*},p_{*}) ( X ∗ , p ∗ ) is a primal-dual KKT pair (13) –(14) , and the primal and dual problems have zero duality gap at this pair.
The AL framework will terminate when approximate counterparts of the three nontrivial conditions in (16) have been obtained. The next definition precisely characterizes our notion of inexactness.
Definition 2.3. For tolerances ( ε p , ε c , ε d ) > 0 (\varepsilon_{p},\varepsilon_{c},\varepsilon_{d})>0 ( ε p , ε c , ε d ) > 0 , a pair ( Z ˉ , p ˉ ) ∈ S + n × R m (\bar{Z},\bar{p})\in\mathbb{S}_{+}^{n}\times\mathbb{R}^{m} ( Z ˉ , p ˉ ) ∈ S + n × R m is called an ( ε p , ε c , ε d ) (\varepsilon_{p},\varepsilon_{c},\varepsilon_{d}) ( ε p , ε c , ε d ) -solution of (13) –(14) if
∥ A ( Z ˉ ) − b ∥ ≤ ε p , λ min ( S ˉ ) ≥ − ε d , ⟨ S ˉ , Z ˉ ⟩ ≤ ε c , (17) \lVert A(\bar{Z})-b\rVert\leq\varepsilon_{p},\qquad\lambda_{\min}(\bar{S})\geq-\varepsilon_{d},\qquad\langle\bar{S},\bar{Z}\rangle\leq\varepsilon_{c}, \tag*{(17)} ∥ A ( Z ˉ ) − b ∥ ≤ ε p , λ m i n ( S ˉ ) ≥ − ε d , ⟨ S ˉ , Z ˉ ⟩ ≤ ε c , (17) where S ˉ : = S ( p ˉ ) \bar{S}:=S(\bar{p}) S ˉ := S ( p ˉ ) and
S ( p ) : = C − A ∗ p . (18) S(p):=C-A^{*}p. \tag*{(18)} S ( p ) := C − A ∗ p . (18)
Definition 2.3 is a direct relaxation of the exact KKT conditions. Indeed, setting ε p = ε c = ε d = 0 \varepsilon_{p}=\varepsilon_{c}=\varepsilon_{d}=0 ε p = ε c = ε d = 0 in (17) recovers (16) .
AL Framework
This subsection presents the AL framework for approximately solving the primal-dual SDP pair (13) –(14) . For ease of analysis and presentation, the AL framework is presented in the X X X -space, meaning that all iterates are full n × n n \times n n × n positive semidefinite matrices. Our ultimate goal is to later present an algorithm that never works with such dense matrices X X X and instead stores only low-rank factors U U U , which typically have only a few columns. This memory-efficient low-rank algorithm, HALLAr + r+ r + , is presented in Section 4 and can be viewed as a variant of the AL framework.
The AL framework is based on approximately solving a sequence of augmented Lagrangian subproblems (2). For convenience, we define the shorthand
r ( X ) : = b − A X ∀ X ∈ S n . (19) r(X):=b-AX \qquad\forall X\in\mathbb{S}^{n}. \tag*{(19)} r ( X ) := b − A X ∀ X ∈ S n . (19)
Hence, for a multiplier p ∈ R m p\in\mathbb{R}^{m} p ∈ R m and a penalty parameter β ∈ R + \beta\in\mathbb{R}_{+} β ∈ R + , the augmented Lagrangian function is defined as
L β ( X ; p ) : = ⟨ C , X ⟩ + ⟨ p , r ( X ) ⟩ + β 2 ∥ r ( X ) ∥ 2 . (20) \mathcal{L}_{\beta}(X;p):=\langle C,X\rangle+\langle p,r(X)\rangle+\frac{\beta}{2}\lVert r(X)\rVert^{2}. \tag*{(20)} L β ( X ; p ) := ⟨ C , X ⟩ + ⟨ p , r ( X )⟩ + 2 β ∥ r ( X ) ∥ 2 . (20)
Moreover, during the t t t -th iteration of the AL framework, the augmented Lagrangian subproblem (2) can be written as
min X ∈ S + n L β t − 1 ( X ; p t − 1 ) = min X ∈ S + n { ⟨ C , X ⟩ + ⟨ p t − 1 , r ( X ) ⟩ + β t − 1 2 ∥ r ( X ) ∥ 2 } . \min_{X\in\mathbb{S}_{+}^{n}}\mathcal{L}_{\beta_{t-1}}(X;p_{t-1})
=
\min_{X\in\mathbb{S}_{+}^{n}}
\left\{
\langle C,X\rangle+\langle p_{t-1},r(X)\rangle+\frac{\beta_{t-1}}{2}\lVert r(X)\rVert^{2}
\right\}. X ∈ S + n min L β t − 1 ( X ; p t − 1 ) = X ∈ S + n min { ⟨ C , X ⟩ + ⟨ p t − 1 , r ( X )⟩ + 2 β t − 1 ∥ r ( X ) ∥ 2 } .
Since the AL objective in (2) is convex and differentiable and the feasible region is the closed convex cone S + n \mathbb{S}_{+}^{n} S + n , its first-order optimality condition is
0 ∈ ∇ X L β t − 1 ( X ; p t − 1 ) + N S + n ( X ) . (21) 0\in\nabla_{X}\mathcal{L}_{\beta_{t-1}}(X;p_{t-1})+N_{\mathbb{S}_{+}^{n}}(X). \tag*{(21)} 0 ∈ ∇ X L β t − 1 ( X ; p t − 1 ) + N S + n ( X ) . (21)
By the normal-cone characterization in (5) , condition (19) is equivalent to
∇ X L β t − 1 ( X ; p t − 1 ) ⪰ 0 , ⟨ ∇ X L β t − 1 ( X ; p t − 1 ) , X ⟩ = 0. (22) \nabla_X \mathcal{L}_{\beta_{t-1}}(X;p_{t-1}) \succeq0,\qquad\langle\nabla_X \mathcal{L}_{\beta_{t-1}}(X;p_{t-1}),X\rangle= 0. \tag*{(22)} ∇ X L β t − 1 ( X ; p t − 1 ) ⪰ 0 , ⟨ ∇ X L β t − 1 ( X ; p t − 1 ) , X ⟩ = 0. (22)
The AL framework in this subsection approximately solves each AL subproblem based on stopping criteria that are direct relaxations of the optimality conditions in (22) (see (23) below). This differs from the HALLAr and CGAL methods in [44 , 66 ] , which assume a compact feasible region and approximately solve their AL subproblems by performing a prescribed number of Frank–Wolfe iterations or by terminating when the Frank–Wolfe gap is sufficiently small. These approaches are not directly applicable to (2a), whose feasible region is unbounded. The AL framework is now presented.
AL Framework
Input: Initial pair ( Z 0 , p 0 ) ∈ S + n × R m (Z_0,p_0) \in\mathbb{S}_{+}^{n} \times\mathbb{R}^{m} ( Z 0 , p 0 ) ∈ S + n × R m , tolerance triple ( ε p , ε c , ε d ) ∈ R + + 3 (\varepsilon_p,\varepsilon_c,\varepsilon_d) \in\mathbb{R}_{++}^{3} ( ε p , ε c , ε d ) ∈ R ++ 3 , and initial penalty parameter β 0 ∈ R + + \beta_0 \in\mathbb{R}_{++} β 0 ∈ R ++ .
Output: Pair ( Z ‾ , p ‾ ) (\overline{Z},\overline{p}) ( Z , p ) satisfying (15).
Set t ← 1 t \leftarrow1 t ← 1 .
Find Z t ∈ S + n Z_t \in\mathbb{S}_{+}^{n} Z t ∈ S + n , which together with S t = C − A ∗ [ p t − 1 + β t − 1 r ( Z t ) ] S_t = C - \mathcal{A}^{*}[p_{t-1} + \beta_{t-1}r(Z_t)] S t = C − A ∗ [ p t − 1 + β t − 1 r ( Z t )] , satisfies
λ min ( S t ) ≥ − ε d , ∣ ⟨ S t , Z t ⟩ ∣ ≤ ε c . (23) \lambda_{\min}(S_t) \geq-\varepsilon_d,\qquad\lvert\langle S_t,Z_t\rangle\rvert\leq\varepsilon_c. \tag*{(23)} λ m i n ( S t ) ≥ − ε d , ∣⟨ S t , Z t ⟩∣ ≤ ε c . (23)
Set p t = p t − 1 + β t − 1 r ( Z t ) p_t = p_{t-1} + \beta_{t-1}r(Z_t) p t = p t − 1 + β t − 1 r ( Z t ) .
If ∥ r ( Z t ) ∥ ≤ ε p \lVert r(Z_t)\rVert\leq\varepsilon_p ∥ r ( Z t )∥ ≤ ε p , then stop and output ( Z ‾ , p ‾ ) ← ( Z t , p t ) (\overline{Z},\overline{p}) \leftarrow(Z_t,p_t) ( Z , p ) ← ( Z t , p t ) .
set β t ← 2 β t − 1 \beta_t \leftarrow2\beta_{t-1} β t ← 2 β t − 1 , t ← t + 1 t \leftarrow t+1 t ← t + 1 , and go to Step 1.
First, Step 1 computes Z t ∈ S + n Z_t \in\mathbb{S}_{+}^{n} Z t ∈ S + n satisfying an inexact optimality condition for the current AL subproblem. Indeed, the identity
∇ X L β ( X ; p ) = C − A ∗ [ p + β r ( X ) ] (24) \nabla_X \mathcal{L}_{\beta}(X;p) = C - \mathcal{A}^{*}[p+\beta r(X)] \tag*{(24)} ∇ X L β ( X ; p ) = C − A ∗ [ p + β r ( X )] (24)
implies that ∇ X L β t − 1 ( Z t ; p t − 1 ) = S t \nabla_X \mathcal{L}_{\beta_{t-1}}(Z_t;p_{t-1}) = S_t ∇ X L β t − 1 ( Z t ; p t − 1 ) = S t . Thus, the conditions in (22) are a special case of (23) with ε c = ε d = 0 \varepsilon_c=\varepsilon_d=0 ε c = ε d = 0 The AL framework requires only that (23) hold and does not prescribe how such a point is computed. A Frank–Wolfe-type method for computing such a point is developed in later sections.
Second, Step 2 performs the standard AL multiplier update, which, together with the definition of S ( ⋅ ) S(\cdot) S ( ⋅ ) in (16), implies that S t = S ( p t ) S_t=S(p_t) S t = S ( p t ) . The two conditions in (23) then ensure that ( Z t , p t ) (Z_t,p_t) ( Z t , p t ) satisfies the approximate dual-feasibility and complementarity requirements in (15). Third, because Z t ∈ S + n Z_t \in\mathbb{S}_{+}^{n} Z t ∈ S + n by construction, Step 3 only needs to check primal feasibility. If ∥ r ( Z t ) ∥ ≤ ε p \lVert r(Z_t)\rVert\leq\varepsilon_p ∥ r ( Z t )∥ ≤ ε p , then ( Z t , p t ) (Z_t,p_t) ( Z t , p t ) satisfies all the conditions in (15) in view of (23) . In this case ( Z t , p t ) (Z_t,p_t) ( Z t , p t ) is returned, otherwise, if ∥ r ( Z t ) ∥ > ε p \lVert r(Z_t)\rVert> \varepsilon_p ∥ r ( Z t )∥ > ε p , Step 4 doubles the penalty parameter and proceeds to the next iteration. Consequently, for every t ≥ 1 t \geq1 t ≥ 1 , the penalty parameters satisfy
β t − 1 = 2 t − 1 β 0 . (25) \beta_{t-1}=2^{t-1}\beta_0. \tag*{(25)} β t − 1 = 2 t − 1 β 0 . (25)
For the analysis, the same formula defines the penalty parameter at the terminal index.
Theorem 2.4 gives the main outer-iteration complexity bound for the AL framework. Its proof is presented in Subsection 2.3.
Theorem 2.4. Let ε : = ( ε p , ε c , ε d ) > 0 \varepsilon:= (\varepsilon_p,\varepsilon_c,\varepsilon_d)>0 ε := ( ε p , ε c , ε d ) > 0 be given and define
C 1 = C 1 ( ε ) : = ε c + ε d ε p , τ ∗ : = max { 1 , 2 tr ( X ∗ ) } , β ‾ ( ε ) : = β 0 + 1 ε p [ 4 ( ∥ p 0 ∥ + ∥ p ∗ ∥ ) + 16 C 1 τ ∗ ] . (26) C_1=C_1(\varepsilon):=\frac{\varepsilon_c+\varepsilon_d}{\varepsilon_p},\qquad\tau_*:=\max\{1,2\operatorname{tr}(X_*)\},\qquad\overline{\beta}(\varepsilon):=\beta_0+\frac{1}{\varepsilon_p}\left[4(\lVert p_0\rVert+\lVert p_*\rVert)+16C_1\tau_*\right]. \tag*{(26)} C 1 = C 1 ( ε ) := ε p ε c + ε d , τ ∗ := max { 1 , 2 tr ( X ∗ )} , β ( ε ) := β 0 + ε p 1 [ 4 (∥ p 0 ∥ + ∥ p ∗ ∥) + 16 C 1 τ ∗ ] . (26) Then the following statements about the AL framework hold:
i) it terminates in at most T T T iterations, where
T = T ε : = 1 + ⌈ log 2 + ( 2 ( ∥ p 0 ∥ + ∥ p ∗ ∥ ) + 8 C 1 τ ∗ β 0 ε p ) ⌉ ; (27) T=T_{\varepsilon}:=1+\left\lceil\log_2^{+}\left(\frac{2(\lVert p_0\rVert+\lVert p_*\rVert)+8C_1\tau_*}{\beta_0\varepsilon_p}\right)\right\rceil; \tag*{(27)} T = T ε := 1 + ⌈ log 2 + ( β 0 ε p 2 (∥ p 0 ∥ + ∥ p ∗ ∥) + 8 C 1 τ ∗ ) ⌉ ; (27)
ii) its last generated penalty parameter β ˉ \bar{\beta} β ˉ is bounded by
β ˉ ≤ β ˉ ( ϵ ) (28) \bar{\beta} \le\bar{\beta}(\epsilon) \tag*{(28)} β ˉ ≤ β ˉ ( ϵ ) (28)
where the bound β ˉ ( ϵ ) \bar{\beta}(\epsilon) β ˉ ( ϵ ) is defined in (26) ;
iii) its output pair ( Z ˉ , p ˉ ) (\bar{Z},\bar{p}) ( Z ˉ , p ˉ ) is an ( ϵ p , ϵ c , ϵ d ) (\epsilon_p,\epsilon_c,\epsilon_d) ( ϵ p , ϵ c , ϵ d ) -solution of (13) –(14) , and every multiplier p t p_t p t generated by the AL framework satisfies
∥ p t ∥ ≤ ∥ p ∗ ∥ + ∥ p 0 − p ∗ ∥ 2 + 4 ϵ p C 1 β ˉ ( ϵ ) τ ∗ . (29) \lVert p_t\rVert\le\lVert p_*\rVert+ \sqrt{\lVert p_0-p_*\rVert^2+4\epsilon_p C_1\bar{\beta}(\epsilon)\tau_*}. \tag*{(29)} ∥ p t ∥ ≤ ∥ p ∗ ∥ + ∥ p 0 − p ∗ ∥ 2 + 4 ϵ p C 1 β ˉ ( ϵ ) τ ∗ . (29)
Proof of Theorem 2.4
Besides Assumption 2.2, the proof below uses only relation (23) and Steps 2–4 of the AL framework, and hence it does not depend on how Step 1 is implemented. For every t ≥ 1 t\ge1 t ≥ 1 , define the gap quantity
ϑ t ∗ : = max U ∈ Δ τ ∗ ⟨ S t , Z t − U ⟩ , (30) \vartheta_t^* := \max_{U\in\Delta_{\tau_*}}\langle S_t,Z_t-U\rangle, \tag*{(30)} ϑ t ∗ := U ∈ Δ τ ∗ max ⟨ S t , Z t − U ⟩ , (30)
where Δ τ \Delta_\tau Δ τ is as in (102) and τ ∗ \tau_* τ ∗ is as in (26) . We first show that the inner product between the dual error p t − p ∗ p_t-p_* p t − p ∗ and the primal residual r ( Z t ) r(Z_t) r ( Z t ) is controlled by ϑ t ∗ \vartheta_t^* ϑ t ∗ . This relation will be instrumental in bounding the sequence of primal residuals ∥ r ( Z t ) ∥ \lVert r(Z_t)\rVert ∥ r ( Z t )∥ .
Lemma 2.5. For every t ≥ 1 t\ge1 t ≥ 1 , the iterate pair ( Z t , p t ) (Z_t,p_t) ( Z t , p t ) satisfies
⟨ p t − p ∗ , r ( Z t ) ⟩ ≤ ϑ t ∗ , (31) \langle p_t-p_*,r(Z_t)\rangle\le\vartheta_t^*, \tag*{(31)} ⟨ p t − p ∗ , r ( Z t )⟩ ≤ ϑ t ∗ , (31)
where ϑ t ∗ \vartheta_t^* ϑ t ∗ is as in (30) .
Proof. Using (30) , the inclusion X ∗ ∈ Δ τ ∗ X_*\in\Delta_{\tau_*} X ∗ ∈ Δ τ ∗ , and the fact that the identity p t = p t − 1 + β t − 1 r ( Z t ) p_t=p_{t-1}+\beta_{t-1}r(Z_t) p t = p t − 1 + β t − 1 r ( Z t ) and (24) imply that S t = C − A ∗ p t S_t=C-\mathcal{A}^*p_t S t = C − A ∗ p t , we have
ϑ t ∗ = max U ∈ Δ τ ∗ ⟨ S t , Z t − U ⟩ ≥ ⟨ S t , Z t − X ∗ ⟩ = ⟨ C − A ∗ p t , Z t − X ∗ ⟩ = ⟨ A ∗ ( p ∗ − p t ) , Z t − X ∗ ⟩ + ⟨ C − A ∗ p ∗ , Z t − X ∗ ⟩ . (32) \begin{aligned}
\vartheta_t^* = \max_{U\in\Delta_{\tau_*}}\langle S_t,Z_t-U\rangle\ge\langle S_t,Z_t-X_*\rangle= \langle C-\mathcal{A}^*p_t,Z_t-X_*\rangle\\
&= \langle\mathcal{A}^*(p_*-p_t),Z_t-X_*\rangle+\langle C-\mathcal{A}^*p_*,Z_t-X_*\rangle. \tag*{(32)}
\end{aligned} ϑ t ∗ = U ∈ Δ τ ∗ max ⟨ S t , Z t − U ⟩ ≥ ⟨ S t , Z t − X ∗ ⟩ = ⟨ C − A ∗ p t , Z t − X ∗ ⟩ = ⟨ A ∗ ( p ∗ − p t ) , Z t − X ∗ ⟩ + ⟨ C − A ∗ p ∗ , Z t − X ∗ ⟩ . (32)
Using (15) and the definition in (19) , the first term in (32) satisfies
⟨ A ∗ ( p ∗ − p t ) , Z t − X ∗ ⟩ = ⟨ p ∗ − p t , A ( Z t − X ∗ ) ⟩ = ⟨ p ∗ − p t , A Z t − b ⟩ = ⟨ p t − p ∗ , b − A Z t ⟩ = ⟨ p t − p ∗ , r ( Z t ) ⟩ . (33) \begin{aligned}
\langle\mathcal{A}^*(p_*-p_t),Z_t-X_*\rangle=\langle p_*-p_t,\mathcal{A}(Z_t-X_*)\rangle=\langle p_*-p_t,\mathcal{A}Z_t-b\rangle\\
&=\langle p_t-p_*,b-\mathcal{A}Z_t\rangle=\langle p_t-p_*,r(Z_t)\rangle. \tag*{(33)}
\end{aligned} ⟨ A ∗ ( p ∗ − p t ) , Z t − X ∗ ⟩ = ⟨ p ∗ − p t , A ( Z t − X ∗ )⟩ = ⟨ p ∗ − p t , A Z t − b ⟩ = ⟨ p t − p ∗ , b − A Z t ⟩ = ⟨ p t − p ∗ , r ( Z t )⟩ . (33)
It is then easy to see from (15) that C − A ∗ p ∗ ∈ − N S + n ( X ∗ ) C-\mathcal{A}^*p_*\in-N_{\mathbb{S}^n_+}(X_*) C − A ∗ p ∗ ∈ − N S + n ( X ∗ ) . This inclusion, the fact that Z t ∈ S + n Z_t\in\mathbb{S}^n_+ Z t ∈ S + n , and the definition of normal cone in (5) with u = Z t u=Z_t u = Z t , v = − ( C − A ∗ p ∗ ) v=-(C-\mathcal{A}^*p_*) v = − ( C − A ∗ p ∗ ) , Ω = S + n \Omega=\mathbb{S}^n_+ Ω = S + n , and x = X ∗ x=X_* x = X ∗ imply that
⟨ C − A ∗ p ∗ , Z t − X ∗ ⟩ ≥ 0. (34) \langle C-\mathcal{A}^*p_*,Z_t-X_*\rangle\ge0. \tag*{(34)} ⟨ C − A ∗ p ∗ , Z t − X ∗ ⟩ ≥ 0. (34)
Relation (31) then follows immediately from relations (32) , (33) , and (34) .
The following lemma bounds the quantity ϑ t ∗ \vartheta_t^* ϑ t ∗ by a fixed quantity that does not vary with the iteration index t t t .
Lemma 2.6. For every t ≥ 1 t\ge1 t ≥ 1 , we have
ϑ t ∗ ≤ ϵ c + ϵ d τ ∗ , (35) \vartheta_t^*\le\epsilon_c+\epsilon_d\tau_*, \tag*{(35)} ϑ t ∗ ≤ ϵ c + ϵ d τ ∗ , (35)
where ϑ t ∗ \vartheta_t^* ϑ t ∗ is as in (30) .
Proof. Definition (30) and Proposition A.2(a), applied with ( X , G , τ ) = ( Z t , S t , τ ∗ ) (X,G,\tau)=(Z_t,S_t,\tau_*) ( X , G , τ ) = ( Z t , S t , τ ∗ ) , imply that
ϑ t ∗ ≤ ∣ ⟨ S t , Z t ⟩ ∣ + max { λ max ( − S t ) , 0 } τ ∗ ≤ ϵ c + ϵ d τ ∗ , \begin{aligned}
\vartheta_t^*\le\left|\langle S_t,Z_t\rangle\right|+\max\{\lambda_{\max}(-S_t),0\}\tau_* \\
&\le\epsilon_c+\epsilon_d\tau_*,
\end{aligned} ϑ t ∗ ≤ ∣ ⟨ S t , Z t ⟩ ∣ + max { λ m a x ( − S t ) , 0 } τ ∗ ≤ ϵ c + ϵ d τ ∗ ,
where the last inequality follows from (23) and the identity λ max ( − S t ) = − λ min ( S t ) \lambda_{\max}(-S_t)=-\lambda_{\min}(S_t) λ m a x ( − S t ) = − λ m i n ( S t ) .
The next lemma establishes key bounds on the sequence of Lagrange multipliers and primal feasibility residuals.
Lemma 2.7. For every generated iteration index t ≥ 1 t \ge1 t ≥ 1 , we have
∥ p t − p ∗ ∥ 2 + β t − 1 2 ∥ r ( Z t ) ∥ 2 ≤ ∥ p 0 − p ∗ ∥ 2 + 4 ε p C 1 β t − 1 τ ∗ . (36) \lVert p_t-p_*\rVert^2+\beta_{t-1}^2\lVert r(Z_t)\rVert^2\le\lVert p_0-p_*\rVert^2+4\varepsilon_p C_1\beta_{t-1}\tau_*. \tag*{(36)} ∥ p t − p ∗ ∥ 2 + β t − 1 2 ∥ r ( Z t ) ∥ 2 ≤ ∥ p 0 − p ∗ ∥ 2 + 4 ε p C 1 β t − 1 τ ∗ . (36) Proof. The multiplier update in Step 2 implies
∥ p t − p ∗ ∥ 2 + β t − 1 2 ∥ r ( Z t ) ∥ 2 = ∥ p t − 1 − p ∗ ∥ 2 + 2 β t − 1 ⟨ p t − p ∗ , r ( Z t ) ⟩ ≤ ∥ p t − 1 − p ∗ ∥ 2 + 2 ε p C 1 β t − 1 τ ∗ . (37) \begin{aligned}
\lVert p_t-p_*\rVert^2+\beta_{t-1}^2\lVert r(Z_t)\rVert^2
&=\lVert p_{t-1}-p_*\rVert^2+2\beta_{t-1}\langle p_t-p_*,r(Z_t)\rangle\\
&\le\lVert p_{t-1}-p_*\rVert^2+2\varepsilon_p C_1\beta_{t-1}\tau_*. \tag*{(37)}
\end{aligned} ∥ p t − p ∗ ∥ 2 + β t − 1 2 ∥ r ( Z t ) ∥ 2 = ∥ p t − 1 − p ∗ ∥ 2 + 2 β t − 1 ⟨ p t − p ∗ , r ( Z t )⟩ ≤ ∥ p t − 1 − p ∗ ∥ 2 + 2 ε p C 1 β t − 1 τ ∗ . (37) where the inequality follows from (29), (33), and (24). For t = 1 t=1 t = 1 , relation (36) follows directly from (35). For t ≥ 2 t\ge2 t ≥ 2 , applying (36) at index t − 1 t-1 t − 1 to the right-hand side of (35) and using β t − 2 = β t − 1 / 2 \beta_{t-2}=\beta_{t-1}/2 β t − 2 = β t − 1 /2 proves the induction step.
Proof of Theorem 2.4. For convenience, we define
Ψ ∗ = Ψ ∗ ( ε ) : = 2 ( ∥ p 0 ∥ + ∥ p ∗ ∥ ) + 8 C 1 τ ∗ . (38) \Psi_*=\Psi_*(\varepsilon):=2(\lVert p_0\rVert+\lVert p_*\rVert)+8C_1\tau_*. \tag*{(38)} Ψ ∗ = Ψ ∗ ( ε ) := 2 (∥ p 0 ∥ + ∥ p ∗ ∥) + 8 C 1 τ ∗ . (38)
Using the fact that y ≤ 2 ⌈ log + y ⌉ ≤ 1 + 2 y y\le2^{\lceil\log^+ y\rceil}\le1+2y y ≤ 2 ⌈ l o g + y ⌉ ≤ 1 + 2 y for every y > 0 y>0 y > 0 with y = Ψ ∗ / ( β 0 ε p ) y=\Psi_*/(\beta_0\varepsilon_p) y = Ψ ∗ / ( β 0 ε p ) , the definition of T T T in (25), relation (23), and the definition of β ˉ ( ε ) \bar{\beta}(\varepsilon) β ˉ ( ε ) in (24), we obtain
β ˉ ( ε ) = β 0 + 2 Ψ ∗ ε p ≥ β 0 2 T − 1 = β T − 1 ≥ Ψ ∗ ε p ≥ 1 ε p max { 2 ( ∥ p 0 ∥ + ∥ p ∗ ∥ ) , 8 C 1 τ ∗ } . (39) \bar{\beta}(\varepsilon)=\beta_0+\frac{2\Psi_*}{\varepsilon_p}\ge\beta_0 2^{T-1}=\beta_{T-1}\ge\frac{\Psi_*}{\varepsilon_p}\ge\frac{1}{\varepsilon_p}\max\{2(\lVert p_0\rVert+\lVert p_*\rVert),8C_1\tau_*\}. \tag*{(39)} β ˉ ( ε ) = β 0 + ε p 2 Ψ ∗ ≥ β 0 2 T − 1 = β T − 1 ≥ ε p Ψ ∗ ≥ ε p 1 max { 2 (∥ p 0 ∥ + ∥ p ∗ ∥) , 8 C 1 τ ∗ } . (39)
Suppose now, for the sake of contradiction, that the AL framework has not terminated at the end of iteration T T T , i.e., that ∥ r ( Z T ) ∥ > ε p \lVert r(Z_T)\rVert>\varepsilon_p ∥ r ( Z T )∥ > ε p in Step 3. Then, dropping the nonnegative term ∥ p T − p ∗ ∥ 2 \lVert p_T-p_*\rVert^2 ∥ p T − p ∗ ∥ 2 from the left-hand side of (34) with t = T t=T t = T and dividing the resulting inequality by β T − 1 2 ε p 2 \beta_{T-1}^2\varepsilon_p^2 β T − 1 2 ε p 2 , we obtain
1 < ∥ r ( Z T ) ∥ 2 ε p 2 ≤ ( 34 ) ∥ p 0 − p ∗ ∥ 2 ε p 2 β T − 1 2 + 4 C 1 τ ∗ ε p β T − 1 ≤ ( ∥ p 0 ∥ + ∥ p ∗ ∥ ) 2 ε p 2 β T − 1 2 + 4 C 1 τ ∗ ε p β T − 1 ≤ ( 37 ) 1 4 + 1 2 = 3 4 , (40) 1<\frac{\lVert r(Z_T)\rVert^2}{\varepsilon_p^2}\mathrel{\overset{(34)}{\le}}\frac{\lVert p_0-p_*\rVert^2}{\varepsilon_p^2\beta_{T-1}^2}+\frac{4C_1\tau_*}{\varepsilon_p\beta_{T-1}}\le\frac{(\lVert p_0\rVert+\lVert p_*\rVert)^2}{\varepsilon_p^2\beta_{T-1}^2}+\frac{4C_1\tau_*}{\varepsilon_p\beta_{T-1}}\mathrel{\overset{(37)}{\le}}\frac14+\frac12=\frac34, \tag*{(40)} 1 < ε p 2 ∥ r ( Z T ) ∥ 2 ≤ ( 34 ) ε p 2 β T − 1 2 ∥ p 0 − p ∗ ∥ 2 + ε p β T − 1 4 C 1 τ ∗ ≤ ε p 2 β T − 1 2 (∥ p 0 ∥ + ∥ p ∗ ∥ ) 2 + ε p β T − 1 4 C 1 τ ∗ ≤ ( 37 ) 4 1 + 2 1 = 4 3 , (40)
where the second inequality uses the triangle inequality and the last one bounds the two fractions by means of the first and second terms in the maximum in (37), respectively. This contradiction shows that the AL framework performs at most T T T outer iterations.
Let t ˉ ≤ T \bar{t}\le T t ˉ ≤ T be the terminal outer index, so that ( Z ˉ , p ˉ ) = ( Z t ˉ , p t ˉ ) (\bar{Z},\bar{p})=(Z_{\bar{t}},p_{\bar{t}}) ( Z ˉ , p ˉ ) = ( Z t ˉ , p t ˉ ) and β ˉ = β t ˉ − 1 \bar{\beta}=\beta_{\bar{t}-1} β ˉ = β t ˉ − 1 . Since { β t } \{\beta_t\} { β t } is nondecreasing by relation (23), we have β ˉ = β t ˉ − 1 ≤ β T − 1 \bar{\beta}=\beta_{\bar{t}-1}\le\beta_{T-1} β ˉ = β t ˉ − 1 ≤ β T − 1 , and hence the upper bound in (37) yields (26). Moreover, it is easy to see that (22) implies that S ˉ = C − A ∗ p ˉ = S t ˉ \bar{S}=C-\mathcal{A}^*\bar{p}=S_{\bar{t}} S ˉ = C − A ∗ p ˉ = S t ˉ , and thus the two inequalities in (21) with t = t ˉ t=\bar{t} t = t ˉ together with the stopping test in Step 3 give the three relations in (15). Hence, ( Z ˉ , p ˉ ) (\bar{Z},\bar{p}) ( Z ˉ , p ˉ ) is an ( ε p , ε c , ε d ) (\varepsilon_p,\varepsilon_c,\varepsilon_d) ( ε p , ε c , ε d ) -solution of (11)–(12).
Finally, let t ≤ t ˉ t\le\bar{t} t ≤ t ˉ be any generated iteration index. Dropping the nonnegative term β t − 1 2 ∥ r ( Z t ) ∥ 2 \beta_{t-1}^2\lVert r(Z_t)\rVert^2 β t − 1 2 ∥ r ( Z t ) ∥ 2 from the left-hand side of (34) and using the facts that β t − 1 ≤ β t ˉ − 1 = β ˉ \beta_{t-1}\le\beta_{\bar{t}-1}=\bar{\beta} β t − 1 ≤ β t ˉ − 1 = β ˉ and β ˉ ≤ β ˉ ( ε ) \bar{\beta}\le\bar{\beta}(\varepsilon) β ˉ ≤ β ˉ ( ε ) , we conclude that
∥ p t − p ∗ ∥ 2 ≤ ( 34 ) ∥ p 0 − p ∗ ∥ 2 + 4 ε p C 1 β t − 1 τ ∗ ≤ ( 26 ) ∥ p 0 − p ∗ ∥ 2 + 4 ε p C 1 β ˉ ( ε ) τ ∗ . \lVert p_t-p_*\rVert^2\mathrel{\overset{(34)}{\le}}\lVert p_0-p_*\rVert^2+4\varepsilon_p C_1\beta_{t-1}\tau_*\mathrel{\overset{(26)}{\le}}\lVert p_0-p_*\rVert^2+4\varepsilon_p C_1\bar{\beta}(\varepsilon)\tau_*. ∥ p t − p ∗ ∥ 2 ≤ ( 34 ) ∥ p 0 − p ∗ ∥ 2 + 4 ε p C 1 β t − 1 τ ∗ ≤ ( 26 ) ∥ p 0 − p ∗ ∥ 2 + 4 ε p C 1 β ˉ ( ε ) τ ∗ .
Relation (27) now follows by taking square roots in the above inequality and using the triangle inequality $\lVert p_t\rVert\le\lVert p_\rVert+\lVert p_t-p_ \rVert.
X-AL Method
This section presents a concrete method for approximately solving the linearly constrained SDP (11) under Assumption 2.1, and establishes its total inner-iteration complexity. The method is a special instance of the AL framework of Section 2 in which the Relaxed Unbounded Frank–Wolfe routine of Appendix A.2 implements Step 1. A feature of the resulting bound is that the number of inner iterations performed during an outer iteration does not depend on how suboptimal the warm start Z t − 1 Z_{t-1} Z t − 1 is for the current AL subproblem.
X-AL Method and Main Complexity Result
We now make Step 1 of the AL framework clear while keeping the description entirely in the X X X -space. At outer iteration t ≥ 1 t\ge1 t ≥ 1 , the X X X -space implementation of Step 1 applies the relaxed unbounded Frank–Wolfe framework (RUFW) of Appendix A.2 directly to the AL subproblem
min { L β t − 1 ( X ; p t − 1 ) : X ∈ S + n } , (41) \min\{\mathcal{L}_{\beta_{t-1}}(X;p_{t-1}):X\in\mathbb{S}_+^n\}, \tag*{(41)} min { L β t − 1 ( X ; p t − 1 ) : X ∈ S + n } , (41)
with initial point Z t − 1 Z_{t-1} Z t − 1 and tolerance pair ( ε d , ε c ) (\varepsilon_d,\varepsilon_c) ( ε d , ε c ) . The output returned by this call is denoted by the matrix Z t Z_t Z t . We are now ready to present the X-Space AL Method.
X-AL Method
Input: Initial pair ( Z 0 , p 0 ) ∈ S + n × R m (Z_0,p_0)\in\mathbb{S}^{n}_{+}\times\mathbb{R}^{m} ( Z 0 , p 0 ) ∈ S + n × R m , tolerances ( ε p , ε c , ε d ) ∈ R + + 3 (\varepsilon_p,\varepsilon_c,\varepsilon_d)\in\mathbb{R}^{3}_{++} ( ε p , ε c , ε d ) ∈ R ++ 3 , and initial penalty parameter β 0 ∈ R + + \beta_0\in\mathbb{R}_{++} β 0 ∈ R ++ . Output: Pair ( Z ˉ , p ˉ ) (\bar{Z},\bar{p}) ( Z ˉ , p ˉ ) satisfying (15).
Set t ← 1 t\leftarrow1 t ← 1 .
Perform the following X-space relaxed unbounded Frank–Wolfe (RUFW) call:
1.a) Set X ~ ← Z t − 1 \widetilde{X}\leftarrow Z_{t-1} X ← Z t − 1 .
1.b) Compute X ∈ S + n X\in\mathbb{S}^{n}_{+} X ∈ S + n such that
L β t − 1 ( X ; p t − 1 ) ≤ L β t − 1 ( X ~ ; p t − 1 ) (42) \mathcal{L}_{\beta_{t-1}}(X;p_{t-1})\leq\mathcal{L}_{\beta_{t-1}}(\widetilde{X};p_{t-1}) \tag*{(42)} L β t − 1 ( X ; p t − 1 ) ≤ L β t − 1 ( X ; p t − 1 ) (42)
and set
S ← ∇ X L β t − 1 ( X ; p t − 1 ) . S\leftarrow\nabla_X\mathcal{L}_{\beta_{t-1}}(X;p_{t-1}). S ← ∇ X L β t − 1 ( X ; p t − 1 ) .
1.c) Compute a minimum eigenpair ( λ min ( S ) , v min ( S ) ) (\lambda_{\min}(S),v_{\min}(S)) ( λ m i n ( S ) , v m i n ( S )) of S S S with ∥ v min ( S ) ∥ = 1 \lVert v_{\min}(S)\rVert=1 ∥ v m i n ( S )∥ = 1 .
1.d) If
λ min ( S ) ≥ − ε d , ∣ ⟨ S , X ⟩ ∣ ≤ ε c , (43) \lambda_{\min}(S)\geq-\varepsilon_d,\qquad\lvert\langle S,X\rangle\rvert\leq\varepsilon_c, \tag*{(43)} λ m i n ( S ) ≥ − ε d , ∣⟨ S , X ⟩∣ ≤ ε c , (43)
then set Z t ← X Z_t\leftarrow X Z t ← X and go to Step 2.
1.e) Set
y F : = { v min ( S ) , λ min ( S ) < 0 , 0 , λ min ( S ) ≥ 0 , X F : = y F ( y F ) T (44) y^F:=\begin{cases}
v_{\min}(S), & \lambda_{\min}(S)<0,\\
0, & \lambda_{\min}(S)\geq0,
\end{cases}
\qquad X^F:=y^F(y^F)^\mathsf{T} \tag*{(44)} y F := { v m i n ( S ) , 0 , λ m i n ( S ) < 0 , λ m i n ( S ) ≥ 0 , X F := y F ( y F ) T (44)
and compute
( α , ω ) ∈ arg min { L β t − 1 ( α ′ X + ω ′ X F ; p t − 1 ) : α ′ ≥ 0 , ω ′ ≥ 0 } . (45) (\alpha,\omega)\in\mathop{\operatorname{arg\,min}}\left\{\mathcal{L}_{\beta_{t-1}}(\alpha'X+\omega'X^F;p_{t-1}):\alpha'\geq0,\ \omega'\geq0\right\}. \tag*{(45)} ( α , ω ) ∈ arg min { L β t − 1 ( α ′ X + ω ′ X F ; p t − 1 ) : α ′ ≥ 0 , ω ′ ≥ 0 } . (45)
1.f) Set
X ~ ← α X + ω X F , (46) \widetilde{X}\leftarrow\alpha X+\omega X^F, \tag*{(46)} X ← α X + ω X F , (46)
and return to Step 1.b).
Set
p t ← p t − 1 + β t − 1 r ( Z t ) , S t ← C − A ∗ p t . (47) p_t\leftarrow p_{t-1}+\beta_{t-1}r(Z_t),\qquad S_t\leftarrow C-\mathcal{A}^*p_t. \tag*{(47)} p t ← p t − 1 + β t − 1 r ( Z t ) , S t ← C − A ∗ p t . (47)
If ∥ r ( Z t ) ∥ ≤ ε p \lVert r(Z_t)\rVert\leq\varepsilon_p ∥ r ( Z t )∥ ≤ ε p , then set ( Z ˉ , p ˉ ) ← ( Z t , p t ) (\bar{Z},\bar{p})\leftarrow(Z_t,p_t) ( Z ˉ , p ˉ ) ← ( Z t , p t ) , output ( Z ˉ , p ˉ ) (\bar{Z},\bar{p}) ( Z ˉ , p ˉ ) , and stop .
Set
β t ← 2 β t − 1 , t ← t + 1 , (48) \beta_t\leftarrow2\beta_{t-1},\qquad t\leftarrow t+1, \tag*{(48)} β t ← 2 β t − 1 , t ← t + 1 , (48)
and return to Step 1.
Step 1 approximately solves the current AL subproblem by means of an X-space relaxed unbounded Frank–Wolfe (RUFW) procedure described in Appendix § 7 .§ 2 . At the beginning of the t t t -th outer iteration, Step 1.a) initializes the RUFW iterate at Z t − 1 Z_{t-1} Z t − 1 . Given the current point X ~ \widetilde{X} X , Step 1.b) computes a point X ∈ S + n X\in\mathbb{S}^{n}_{+} X ∈ S + n whose AL value is no larger than that of X ~ \widetilde{X} X , and evaluates the corresponding gradient
S = ∇ X L β t − 1 ( X ; p t − 1 ) . S=\nabla_X\mathcal{L}_{\beta_{t-1}}(X;p_{t-1}). S = ∇ X L β t − 1 ( X ; p t − 1 ) .
Step 1.c) then computes a minimum eigenpair of S S S , and Step 1.e) forms the rank-one matrix X F = y F ( y F ) T X^F=y^F(y^F)^\mathsf{T} X F = y F ( y F ) T associated with a direction of negative curvature of the linearized objective. Step 1.d) tests whether X X X satisfies the given approximate optimality conditions (43) . If so, the inner loop terminates and Z t Z_t Z t is set equal to X X X . Otherwise, Step 1.e) minimizes the AL objective over nonnegative combinations of X X X and X F X^F X F , and Step 1.f) uses the resulting point as the next RUFW iterate. The inner loop in Step 1 exits at Step 1.d) precisely when the termination criterion (41) holds, at which point the method sets Z t = X Z_t = X Z t = X and proceeds to Step 2. Hence, upon exiting Step 1, S = ∇ X L β t − 1 ( Z t ; p t − 1 ) S = \nabla_X \mathcal{L}_{\beta_{t-1}}(Z_t; p_{t-1}) S = ∇ X L β t − 1 ( Z t ; p t − 1 ) .
Step 2 then performs the standard AL multiplier update. Combining the fact that relations (22) and (45) imply that
∇ X L β t − 1 ( Z t ; p t − 1 ) = C − A ∗ [ p t − 1 + β t − 1 r ( Z t ) ] = C − A ∗ p t = S t . (49) \nabla_X \mathcal{L}_{\beta_{t-1}}(Z_t; p_{t-1}) = C - \mathcal{A}^*[p_{t-1} + \beta_{t-1}r(Z_t)] = C - \mathcal{A}^*p_t = S_t. \tag*{(49)} ∇ X L β t − 1 ( Z t ; p t − 1 ) = C − A ∗ [ p t − 1 + β t − 1 r ( Z t )] = C − A ∗ p t = S t . (49)
with the stopping conditions in (41) shows that the pair ( Z t , p t ) (Z_t,p_t) ( Z t , p t ) produced after Step 2 always satisfies (21). Hence, Step 1 implements the approximate AL-subproblem solve required in Step 1 of the AL framework presented in Subsection 2.2. The remaining steps carry out the corresponding operations of that framework. Step 2 updates the multiplier and defines S t S_t S t , Step 3 applies the primal-feasibility stopping test, and Step 4 increases the penalty parameter before the next outer iteration.
The following result provides an iteration-complexity bound for the X-AL method and will be proved in the next subsection.
Theorem 3.1. Set ε : = ( ε p , ε c , ε d ) \varepsilon:= (\varepsilon_p,\varepsilon_c,\varepsilon_d) ε := ( ε p , ε c , ε d ) . Let C 1 = C 1 ( ε ) C_1 = C_1(\varepsilon) C 1 = C 1 ( ε ) and τ ∗ \tau_* τ ∗ be as in (24), and let T = T ε T = T_\varepsilon T = T ε and β ˉ ( ε ) \bar{\beta}(\varepsilon) β ˉ ( ε ) be as in (25) and (24), respectively. Define κ ˉ \bar{\kappa} κ ˉ and τ ˉ \bar{\tau} τ ˉ by
κ ˉ : = L β 0 ( Z 0 ; p 0 ) + 9 ( ∥ p 0 ∥ 2 + ∥ p ∗ ∥ 2 ) β 0 + ( 6 T + 4 ) ε p C 1 τ ∗ . (50) \bar{\kappa} := \mathcal{L}_{\beta_0}(Z_0; p_0) + \frac{9\left(\lVert p_0\rVert^2 + \lVert p_*\rVert^2\right)}{\beta_0} + (6T + 4)\varepsilon_p C_1\tau_*. \tag*{(50)} κ ˉ := L β 0 ( Z 0 ; p 0 ) + β 0 9 ( ∥ p 0 ∥ 2 + ∥ p ∗ ∥ 2 ) + ( 6 T + 4 ) ε p C 1 τ ∗ . (50) τ ˉ : = max { 1 , 2 sup { tr ( Z ) : Z ∈ S + n , ⟨ C , Z ⟩ + β 0 4 ∥ r ( Z ) ∥ 2 ≤ κ ˉ } } . (51) \bar{\tau} := \max\left\{1, 2\sup\left\{\operatorname{tr}(Z) : Z \in\mathbb{S}_+^n, \langle C,Z\rangle+ \frac{\beta_0}{4}\lVert r(Z)\rVert^2 \le\bar{\kappa}\right\}\right\}. \tag*{(51)} τ ˉ := max { 1 , 2 sup { tr ( Z ) : Z ∈ S + n , ⟨ C , Z ⟩ + 4 β 0 ∥ r ( Z ) ∥ 2 ≤ κ ˉ } } . (51) Then, the X-AL method returns a pair ( Z ˉ , p ˉ ) (\bar{Z},\bar{p}) ( Z ˉ , p ˉ ) satisfying (15) in at most T T T outer iterations and at most
2 T + 24 τ ˉ 2 ∥ A ∥ 2 β ˉ ( ε ) ( 1 ε c + 2 ε d ) (52) 2T + 24\bar{\tau}^2\lVert\mathcal{A}\rVert^2\bar{\beta}(\varepsilon)\left(\frac{1}{\varepsilon_c} + \frac{2}{\varepsilon_d}\right) \tag*{(52)} 2 T + 24 τ ˉ 2 ∥ A ∥ 2 β ˉ ( ε ) ( ε c 1 + ε d 2 ) (52) inner iterations.
We now make two remarks about Theorem 3.1. First, the bound in (50) does not involve the objective gaps L β t − 1 ( Z t − 1 ; p t − 1 ) − min X ∈ S + n L β t − 1 ( X ; p t − 1 ) \mathcal{L}_{\beta_{t-1}}(Z_{t-1};p_{t-1}) - \min_{X\in\mathbb{S}_+^n}\mathcal{L}_{\beta_{t-1}}(X;p_{t-1}) L β t − 1 ( Z t − 1 ; p t − 1 ) − min X ∈ S + n L β t − 1 ( X ; p t − 1 ) of the warm starts Z t − 1 Z_{t-1} Z t − 1 for the AL subproblems. This is a consequence of Proposition A.3(b), whose bound is independent of the initial objective gap because the first conic search of every RUFW call contains the full Frank–Wolfe step to the linear-oracle point over the trace ball Δ τ t \Delta_{\tau_t} Δ τ t (see Lemma A.7(a)). The initial pair ( Z 0 , p 0 ) (Z_0,p_0) ( Z 0 , p 0 ) enters (50) only through T T T , β ˉ ( ε ) \bar{\beta}(\varepsilon) β ˉ ( ε ) , and the radius τ ˉ \bar{\tau} τ ˉ . Second, suppose that ε p = ε c = ε d = ε ∈ ( 0 , 1 ] \varepsilon_p = \varepsilon_c = \varepsilon_d = \varepsilon\in(0,1] ε p = ε c = ε d = ε ∈ ( 0 , 1 ] . Then C 1 = 2 C_1 = 2 C 1 = 2 , β ˉ ( ε ) = O ( 1 / ε ) \bar{\beta}(\varepsilon) = O(1/\varepsilon) β ˉ ( ε ) = O ( 1/ ε ) , and T = O ( log ( 1 / ε ) ) T = O(\log(1/\varepsilon)) T = O ( log ( 1/ ε )) . Moreover, since ε T \varepsilon T εT is bounded for ε ∈ ( 0 , 1 ] \varepsilon\in(0,1] ε ∈ ( 0 , 1 ] , the scalar κ ˉ \bar{\kappa} κ ˉ in (48) is bounded above by a constant that does not depend on ε \varepsilon ε , and hence so is τ ˉ \bar{\tau} τ ˉ in view of (49) and Proposition A.1. Thus, for fixed problem data, initial pair, and β 0 \beta_0 β 0 , the bound (50) is O ( ε − 2 ) O(\varepsilon^{-2}) O ( ε − 2 ) .
Proof of Theorem 3.1
The proof of Theorem 3.1 relies on the following lemma.
Lemma 3.2. For every iteration index t t t generated by the X-space AL method, define
τ t : = max { 1 , 2 sup { tr ( X ) : X ∈ S ( Z t − 1 ; L β t − 1 ( ⋅ ; p t − 1 ) ) } } . (53) \tau_t := \max\left\{1, 2\sup\left\{\operatorname{tr}(X) : X \in\mathcal{S}(Z_{t-1};\mathcal{L}_{\beta_{t-1}}(\cdot;p_{t-1}))\right\}\right\}. \tag*{(53)} τ t := max { 1 , 2 sup { tr ( X ) : X ∈ S ( Z t − 1 ; L β t − 1 ( ⋅ ; p t − 1 )) } } . (53) Then the following statements about the t t t -th outer iteration hold:
(a) the quantity τ t \tau_t τ t is finite;
(b) the RUFW call in Step 1 terminates in at most
⌈ 2 + 12 τ t 2 β t − 1 ∥ A ∥ 2 ( 1 ε c + 2 ε d ) ⌉ (54) \left\lceil2 + 12\tau_t^2\beta_{t-1}\lVert\mathcal{A}\rVert^2\left(\frac{1}{\varepsilon_c} + \frac{2}{\varepsilon_d}\right)\right\rceil \tag*{(54)} ⌈ 2 + 12 τ t 2 β t − 1 ∥ A ∥ 2 ( ε c 1 + ε d 2 ) ⌉ (54) inner iterations with a pair ( Z t , p t ) (Z_t,p_t) ( Z t , p t ) satisfying (21) and the relation
L β t − 1 ( Z t ; p t − 1 ) ≤ L β t − 1 ( Z t − 1 ; p t − 1 ) . (55) \mathcal{L}_{\beta_{t-1}}(Z_t;p_{t-1}) \le\mathcal{L}_{\beta_{t-1}}(Z_{t-1};p_{t-1}). \tag*{(55)} L β t − 1 ( Z t ; p t − 1 ) ≤ L β t − 1 ( Z t − 1 ; p t − 1 ) . (55)
Proof. (a) Fix a generated iteration index t t t . Definition (5) implies
S ( Z t − 1 ; L β t − 1 ( ⋅ ; p t − 1 ) ) = { X ∈ S + n : L β t − 1 ( X ; p t − 1 ) ≤ L β t − 1 ( Z t − 1 ; p t − 1 ) } . \mathcal{S}(Z_{t-1};\mathcal{L}_{\beta_{t-1}}(\mathord{\cdot};p_{t-1}))=\left\{X\in\mathbb{S}_{+}^{n}:\mathcal{L}_{\beta_{t-1}}(X;p_{t-1})\leq\mathcal{L}_{\beta_{t-1}}(Z_{t-1};p_{t-1})\right\}. S ( Z t − 1 ; L β t − 1 ( ⋅ ; p t − 1 )) = { X ∈ S + n : L β t − 1 ( X ; p t − 1 ) ≤ L β t − 1 ( Z t − 1 ; p t − 1 ) } .
Proposition A.1, applied with ( q , β , κ ) = ( p t − 1 , β t − 1 , L β t − 1 ( Z t − 1 ; p t − 1 ) ) (q,\beta,\kappa)=(p_{t-1},\beta_{t-1},\mathcal{L}_{\beta_{t-1}}(Z_{t-1};p_{t-1})) ( q , β , κ ) = ( p t − 1 , β t − 1 , L β t − 1 ( Z t − 1 ; p t − 1 )) , shows that this set is bounded. It is nonempty because it contains Z t − 1 Z_{t-1} Z t − 1 . Hence, τ t < ∞ \tau_t<\infty τ t < ∞ by (51).
(b) Relations (18) and (22) verify RUFW assumption (A1) with L = β t − 1 ∥ A ∥ 2 L=\beta_{t-1}\lVert A\rVert^2 L = β t − 1 ∥ A ∥ 2 , while part (a) verifies assumption (A2). Apply Proposition A.3 to the objective L β t − 1 ( ⋅ ; p t − 1 ) \mathcal{L}_{\beta_{t-1}}(\mathord{\cdot};p_{t-1}) L β t − 1 ( ⋅ ; p t − 1 ) with initial point Z t − 1 Z_{t-1} Z t − 1 , Lipschitz constant β t − 1 ∥ A ∥ 2 \beta_{t-1}\lVert A\rVert^2 β t − 1 ∥ A ∥ 2 , and tolerance pair ( ϵ , δ ) = ( ϵ d , ϵ c ) (\epsilon,\delta)=(\epsilon_d,\epsilon_c) ( ϵ , δ ) = ( ϵ d , ϵ c ) . Indeed, it follows from (116), (5), and (51) that τ ˉ = τ t \bar{\tau}=\tau_t τ ˉ = τ t , and hence the bound in (117) becomes the bound in (52). It then follows from these conclusions, Proposition A.3(a)-(b), the stopping criteria in (41), and relation (47) that RUFW terminates with a pair ( Z t , p t ) (Z_t,p_t) ( Z t , p t ) satisfying (21) and (53) in at most (52) inner iterations.
The remainder of this subsection is dedicated to bounding the local radii τ t \tau_t τ t uniformly. For every κ ∈ R \kappa\in\mathbb{R} κ ∈ R , define
P β 0 ( κ ) : = { U ∈ S + n : ⟨ C , U ⟩ + β 0 4 ∥ A U − b ∥ 2 ≤ κ } . (56) \mathcal{P}_{\beta_0}(\kappa):=\left\{U\in\mathbb{S}_{+}^{n}:\langle C,U\rangle+\frac{\beta_0}{4}\lVert AU-b\rVert^2\leq\kappa\right\}. \tag*{(56)} P β 0 ( κ ) := { U ∈ S + n : ⟨ C , U ⟩ + 4 β 0 ∥ A U − b ∥ 2 ≤ κ } . (56)
We first observe that Proposition A.1, applied with ( q , β ) = ( 0 , β 0 / 2 ) (q,\beta)=(0,\beta_0/2) ( q , β ) = ( 0 , β 0 /2 ) , implies that P β 0 ( κ ) \mathcal{P}_{\beta_0}(\kappa) P β 0 ( κ ) is bounded for every κ ∈ R \kappa\in\mathbb{R} κ ∈ R . Our strategy for showing that the local sublevel sets are uniformly bounded is to find κ ˉ ∈ R \bar{\kappa}\in\mathbb{R} κ ˉ ∈ R , independent of t t t , such that
S ( Z t − 1 ; L β t − 1 ( ⋅ ; p t − 1 ) ) ⊆ P β 0 ( κ ˉ ) \mathcal{S}(Z_{t-1};\mathcal{L}_{\beta_{t-1}}(\mathord{\cdot};p_{t-1}))\subseteq\mathcal{P}_{\beta_0}(\bar{\kappa}) S ( Z t − 1 ; L β t − 1 ( ⋅ ; p t − 1 )) ⊆ P β 0 ( κ ˉ )
for every generated iteration index t t t .
Lemma 3.3. For every iteration index t ≥ 1 t\geq1 t ≥ 1 , we have:
∥ p t ∥ 2 β t ≤ 3 β 0 ( ∥ p 0 ∥ 2 + ∥ p ∗ ∥ 2 ) + 4 ϵ p C 1 τ ∗ , (57) \frac{\lVert p_t\rVert^2}{\beta_t}\leq\frac{3}{\beta_0}\left(\lVert p_0\rVert^2+\lVert p_*\rVert^2\right)+4\epsilon_pC_1\tau_*, \tag*{(57)} β t ∥ p t ∥ 2 ≤ β 0 3 ( ∥ p 0 ∥ 2 + ∥ p ∗ ∥ 2 ) + 4 ϵ p C 1 τ ∗ , (57) L β t ( Z t ; p t ) ≤ L β 0 ( Z 0 ; p 0 ) + 3 ∥ p 0 − p ∗ ∥ 2 β 0 + 6 t ϵ p C 1 τ ∗ . (58) \mathcal{L}_{\beta_t}(Z_t;p_t)\leq\mathcal{L}_{\beta_0}(Z_0;p_0)+\frac{3\lVert p_0-p_*\rVert^2}{\beta_0}+6t\epsilon_pC_1\tau_*. \tag*{(58)} L β t ( Z t ; p t ) ≤ L β 0 ( Z 0 ; p 0 ) + β 0 3 ∥ p 0 − p ∗ ∥ 2 + 6 t ϵ p C 1 τ ∗ . (58) where C 1 C_1 C 1 and τ ∗ \tau_* τ ∗ are defined in (24).
Proof. The triangle inequality, (34), and β t = 2 β t − 1 \beta_t=2\beta_{t-1} β t = 2 β t − 1 imply
∥ p t ∥ 2 β t ≤ ∥ p ∗ ∥ 2 + ∥ p t − p ∗ ∥ 2 β t − 1 ≤ ∥ p ∗ ∥ 2 + ∥ p 0 − p ∗ ∥ 2 β t − 1 + 4 ϵ p C 1 τ ∗ ≤ 3 ( ∥ p 0 ∥ 2 + ∥ p ∗ ∥ 2 ) β 0 + 4 ϵ p C 1 τ ∗ , \begin{aligned}
\frac{\lVert p_t\rVert^2}{\beta_t}\leq\frac{\lVert p_*\rVert^2+\lVert p_t-p_*\rVert^2}{\beta_{t-1}} \\
&\leq\frac{\lVert p_*\rVert^2+\lVert p_0-p_*\rVert^2}{\beta_{t-1}}+4\epsilon_pC_1\tau_* \\
&\leq\frac{3(\lVert p_0\rVert^2+\lVert p_*\rVert^2)}{\beta_0}+4\epsilon_pC_1\tau_*,
\end{aligned} β t ∥ p t ∥ 2 ≤ β t − 1 ∥ p ∗ ∥ 2 + ∥ p t − p ∗ ∥ 2 ≤ β t − 1 ∥ p ∗ ∥ 2 + ∥ p 0 − p ∗ ∥ 2 + 4 ϵ p C 1 τ ∗ ≤ β 0 3 (∥ p 0 ∥ 2 + ∥ p ∗ ∥ 2 ) + 4 ϵ p C 1 τ ∗ ,
where the last inequality uses β t − 1 ≥ β 0 \beta_{t-1}\geq\beta_0 β t − 1 ≥ β 0 and ∥ p 0 − p ∗ ∥ 2 ≤ 2 ( ∥ p 0 ∥ 2 + ∥ p ∗ ∥ 2 ) \lVert p_0-p_*\rVert^2\leq2(\lVert p_0\rVert^2+\lVert p_*\rVert^2) ∥ p 0 − p ∗ ∥ 2 ≤ 2 (∥ p 0 ∥ 2 + ∥ p ∗ ∥ 2 ) . This proves (55).
Relations (53), (18), (45), and (46) also imply
L β t ( Z t ; p t ) ≤ L β t − 1 ( Z t − 1 ; p t − 1 ) + 3 β t − 1 2 ∥ r ( Z t ) ∥ 2 ≤ L β t − 1 ( Z t − 1 ; p t − 1 ) + 3 ∥ p 0 − p ∗ ∥ 2 2 β t − 1 + 6 ϵ p C 1 τ ∗ . (59) \begin{aligned}
\mathcal{L}_{\beta_t}(Z_t;p_t)\leq\mathcal{L}_{\beta_{t-1}}(Z_{t-1};p_{t-1})+\frac{3\beta_{t-1}}{2}\lVert r(Z_t)\rVert^2 \\
&\leq\mathcal{L}_{\beta_{t-1}}(Z_{t-1};p_{t-1})+\frac{3\lVert p_0-p_*\rVert^2}{2\beta_{t-1}}+6\epsilon_pC_1\tau_*. \tag*{(59)}
\end{aligned} L β t ( Z t ; p t ) ≤ L β t − 1 ( Z t − 1 ; p t − 1 ) + 2 3 β t − 1 ∥ r ( Z t ) ∥ 2 ≤ L β t − 1 ( Z t − 1 ; p t − 1 ) + 2 β t − 1 3 ∥ p 0 − p ∗ ∥ 2 + 6 ϵ p C 1 τ ∗ . (59)
Induction using (57) and (23) yields
L β t ( Z t ; p t ) ≤ L β 0 ( Z 0 ; p 0 ) + 3 ∥ p 0 − p ∗ ∥ 2 β 0 ( 1 − 2 − t ) + 6 t ϵ p C 1 τ ∗ , \mathcal{L}_{\beta_t}(Z_t;p_t)\leq\mathcal{L}_{\beta_0}(Z_0;p_0)+\frac{3\lVert p_0-p_*\rVert^2}{\beta_0}\left(1-2^{-t}\right)+6t\epsilon_pC_1\tau_*, L β t ( Z t ; p t ) ≤ L β 0 ( Z 0 ; p 0 ) + β 0 3 ∥ p 0 − p ∗ ∥ 2 ( 1 − 2 − t ) + 6 t ϵ p C 1 τ ∗ ,
which implies (56).
Lemma 3.4. For every t ≥ 0 t\geq0 t ≥ 0 , we have
S ( Z t ; L β t ( ⋅ ; p t ) ) ⊆ P β 0 ( κ ˉ ) , (60) \mathcal{S}(Z_t;\mathcal{L}_{\beta_t}(\mathord{\cdot};p_t))\subseteq\mathcal{P}_{\beta_0}(\bar{\kappa}), \tag*{(60)} S ( Z t ; L β t ( ⋅ ; p t )) ⊆ P β 0 ( κ ˉ ) , (60) where S ( ⋅ ; ⋅ ) \mathcal{S}(\mathord{\cdot};\mathord{\cdot}) S ( ⋅ ; ⋅ ) is as in (5), and κ ˉ \bar{\kappa} κ ˉ is as in (48).
Proof. Assume that U ∈ S ( Z t ; L β t ( ⋅ ; p t ) ) U \in S(Z_t;\mathcal{L}_{\beta_t}(\cdot;p_t)) U ∈ S ( Z t ; L β t ( ⋅ ; p t )) , which clearly implies that L β t ( U ; p t ) ≤ L β t ( Z t ; p t ) \mathcal{L}_{\beta_t}(U;p_t) \le\mathcal{L}_{\beta_t}(Z_t;p_t) L β t ( U ; p t ) ≤ L β t ( Z t ; p t ) , in view of the definition of S ( ⋅ ; ⋅ ) S(\cdot;\cdot) S ( ⋅ ; ⋅ ) in (7) . This observation together with (20) , the fact that β t ≥ β 0 \beta_t \ge\beta_0 β t ≥ β 0 , and Young’s inequality, then imply that
⟨ C , U ⟩ + β 0 4 ∥ r ( U ) ∥ 2 ≤ ⟨ C , U ⟩ + β t 4 ∥ r ( U ) ∥ 2 = ( 18 ) L β t ( U ; p t ) − ⟨ p t , r ( U ) ⟩ − β t 4 ∥ r ( U ) ∥ 2 ≤ L β t ( Z t ; p t ) − ⟨ p t , r ( U ) ⟩ − β t 4 ∥ r ( U ) ∥ 2 ≤ L β t ( Z t ; p t ) + ∥ p t ∥ 2 β t ≤ κ ˉ , (61) \begin{aligned}
\langle C,U\rangle+\frac{\beta_0}{4}\lVert r(U)\rVert^2\leq\langle C,U\rangle+\frac{\beta_t}{4}\lVert r(U)\rVert^2\overset{(18)}{=}\mathcal{L}_{\beta_t}(U;p_t)-\langle p_t,r(U)\rangle-\frac{\beta_t}{4}\lVert r(U)\rVert^2 \\
&\leq\mathcal{L}_{\beta_t}(Z_t;p_t)-\langle p_t,r(U)\rangle-\frac{\beta_t}{4}\lVert r(U)\rVert^2\leq\mathcal{L}_{\beta_t}(Z_t;p_t)+\frac{\lVert p_t\rVert^2}{\beta_t}\leq\bar{\kappa}, \tag*{(61)}
\end{aligned} ⟨ C , U ⟩ + 4 β 0 ∥ r ( U ) ∥ 2 ≤ ⟨ C , U ⟩ + 4 β t ∥ r ( U ) ∥ 2 = ( 18 ) L β t ( U ; p t ) − ⟨ p t , r ( U )⟩ − 4 β t ∥ r ( U ) ∥ 2 ≤ L β t ( Z t ; p t ) − ⟨ p t , r ( U )⟩ − 4 β t ∥ r ( U ) ∥ 2 ≤ L β t ( Z t ; p t ) + β t ∥ p t ∥ 2 ≤ κ ˉ , (61)
where the last inequality is due to (57) , (58) , and the definition of κ ˉ \bar{\kappa} κ ˉ in (50) . The conclusion of the lemma now follows from (56) .
Proof of Theorem 3.1. Lemma 3.2 (b) implies that X-AL is an instance of the AL framework. Hence, Theorem 2.4 implies that it terminates at an outer iteration index t ˉ ≤ T \bar{t}\leq T t ˉ ≤ T with a pair satisfying (17) . For every 1 ≤ t ≤ t ˉ 1\leq t\leq\bar{t} 1 ≤ t ≤ t ˉ , Lemma 3.4 at index t − 1 t-1 t − 1 and definitions (53) and (51) imply τ t ≤ τ ˉ \tau_t\leq\bar{\tau} τ t ≤ τ ˉ . Thus, induction using (54) and β t − 1 = 2 β t − 2 \beta_{t-1}=2\beta_{t-2} β t − 1 = 2 β t − 2 shows that the first t t t outer iterations perform at most
2 t + 12 τ ˉ 2 ∥ A ∥ 2 ( 1 ε c + 2 ε d ) ( 2 β t − 1 − β 0 ) (62) 2t+12\bar{\tau}^{2}\lVert A\rVert^2\left(\frac{1}{\varepsilon_c}+\frac{2}{\varepsilon_d}\right)(2\beta_{t-1}-\beta_0) \tag*{(62)} 2 t + 12 τ ˉ 2 ∥ A ∥ 2 ( ε c 1 + ε d 2 ) ( 2 β t − 1 − β 0 ) (62)
inner iterations. Applying (62) with t = t ˉ t=\bar{t} t = t ˉ and using t ˉ ≤ T \bar{t}\leq T t ˉ ≤ T and (28) yields (52) .
The following lemma bounds the total decrease of the AL objectives over the inner solves. It will be used to bound the ADAP-AIPP work of HALLaR+.
Lemma 3.5. Suppose that the iterates generated by the AL framework satisfy (55) for every t ≥ 1 t\geq1 t ≥ 1 . Then the total decrease of the AL objectives over all inner solves is at most
κ ˉ + ∥ b ∥ ∥ p ∗ ∥ , (63) \bar{\kappa}+\lVert b\rVert\lVert p_*\rVert, \tag*{(63)} κ ˉ + ∥ b ∥ ∥ p ∗ ∥ , (63) where κ ˉ \bar{\kappa} κ ˉ is as in (50) .
Proof. Let t ≥ 1 t\geq1 t ≥ 1 be an iteration index. The multiplier update, penalty doubling, and (20) imply
L β t ( Z t ; p t ) + 3 ∥ p t − p ∗ ∥ 2 2 β t = L β t − 1 ( Z t ; p t − 1 ) + 3 ∥ p t − 1 − p ∗ ∥ 2 2 β t − 1 + 3 ⟨ p t − p ∗ , r ( Z t ) ⟩ − 3 ∥ p t − p ∗ ∥ 2 4 β t − 1 ≤ L β t − 1 ( Z t ; p t − 1 ) + 3 ∥ p t − 1 − p ∗ ∥ 2 2 β t − 1 + 3 ε p C 1 τ ∗ . (64) \begin{aligned}
\mathcal{L}_{\beta_t}(Z_t;p_t)+\frac{3\lVert p_t-p_*\rVert^2}{2\beta_t}
=\mathcal{L}_{\beta_{t-1}}(Z_t;p_{t-1})+\frac{3\lVert p_{t-1}-p_*\rVert^2}{2\beta_{t-1}}
+3\langle p_t-p_*,r(Z_t)\rangle-\frac{3\lVert p_t-p_*\rVert^2}{4\beta_{t-1}} \\
&\leq\mathcal{L}_{\beta_{t-1}}(Z_t;p_{t-1})+\frac{3\lVert p_{t-1}-p_*\rVert^2}{2\beta_{t-1}}+3\varepsilon_p C_1\tau_*. \tag*{(64)}
\end{aligned} L β t ( Z t ; p t ) + 2 β t 3 ∥ p t − p ∗ ∥ 2 = L β t − 1 ( Z t ; p t − 1 ) + 2 β t − 1 3 ∥ p t − 1 − p ∗ ∥ 2 + 3 ⟨ p t − p ∗ , r ( Z t )⟩ − 4 β t − 1 3 ∥ p t − p ∗ ∥ 2 ≤ L β t − 1 ( Z t ; p t − 1 ) + 2 β t − 1 3 ∥ p t − 1 − p ∗ ∥ 2 + 3 ε p C 1 τ ∗ . (64)
The last inequality follows from (31) , (35) , and (26) . Moreover, (16) and completion of the square imply
L β t ( Z t ; p t ) + 3 ∥ p t − p ∗ ∥ 2 2 β t ≥ ⟨ b , p ∗ ⟩ + ∥ p t − p ∗ ∥ 2 β t ≥ ⟨ b , p ∗ ⟩ . (65) \mathcal{L}_{\beta_t}(Z_t;p_t)+\frac{3\lVert p_t-p_*\rVert^2}{2\beta_t}\geq\langle b,p_*\rangle+\frac{\lVert p_t-p_*\rVert^2}{\beta_t}\geq\langle b,p_*\rangle. \tag*{(65)} L β t ( Z t ; p t ) + 2 β t 3 ∥ p t − p ∗ ∥ 2 ≥ ⟨ b , p ∗ ⟩ + β t ∥ p t − p ∗ ∥ 2 ≥ ⟨ b , p ∗ ⟩ . (65)
Applying (64) successively over the outer iterations and using (65) and t ˉ ≤ T \bar{t}\leq T t ˉ ≤ T bounds the total decrease over the inner solves by
L β 0 ( Z 0 ; p 0 ) + 3 ∥ p 0 − p ∗ ∥ 2 2 β 0 + 3 T ε p C 1 τ ∗ − ⟨ b , p ∗ ⟩ ≤ ( 48 ) κ ˉ + ∥ b ∥ ∥ p ∗ ∥ , \mathcal{L}_{\beta_0}(Z_0;p_0)+\frac{3\lVert p_0-p_*\rVert^2}{2\beta_0}+3T\varepsilon_p C_1\tau_*-\langle b,p_*\rangle\overset{(48)}{\leq}\bar{\kappa}+\lVert b\rVert\lVert p_*\rVert, L β 0 ( Z 0 ; p 0 ) + 2 β 0 3 ∥ p 0 − p ∗ ∥ 2 + 3 T ε p C 1 τ ∗ − ⟨ b , p ∗ ⟩ ≤ ( 48 ) κ ˉ + ∥ b ∥ ∥ p ∗ ∥ ,
where the last inequality uses ∥ p 0 − p ∗ ∥ 2 ≤ 2 ( ∥ p 0 ∥ 2 + ∥ p ∗ ∥ 2 ) \lVert p_0-p_*\rVert^2\leq2(\lVert p_0\rVert^2+\lVert p_*\rVert^2) ∥ p 0 − p ∗ ∥ 2 ≤ 2 (∥ p 0 ∥ 2 + ∥ p ∗ ∥ 2 ) and Cauchy–Schwarz.
HALLaR+ in the U U U -Space
This section presents HALLaR+, the factorized implementation of the X-AL method of § 3 . It contains two subsections. Subsection 4.1 states the HALLaR+ method and presents its main complexity result, namely Theorem 4.1 . Subsection 4.2 is devoted to the proof of Theorem 4.1 .
Description of HALLaR+ and Main Theorem
We start by stating the HALLaR+ method.
HALLaR+ Method
Input: Initial pair ( Y 0 , p 0 ) ∈ R n × r 0 × R m (Y_0,p_0) \in\mathbb{R}^{n \times r_0} \times\mathbb{R}^m ( Y 0 , p 0 ) ∈ R n × r 0 × R m for some r 0 ≥ 1 r_0 \ge1 r 0 ≥ 1 , tolerance triple ( ε p , ε c , ε d ) ∈ R + + 3 (\varepsilon_p,\varepsilon_c,\varepsilon_d) \in\mathbb{R}_{++}^3 ( ε p , ε c , ε d ) ∈ R ++ 3 , initial penalty parameter β 0 ∈ R + + \beta_0 \in\mathbb{R}_{++} β 0 ∈ R ++ , and hybrid low-rank (HLR) parameters η 0 > 0 \eta_0 > 0 η 0 > 0 , M 0 > 1 / 2 M_0 > 1/2 M 0 > 1/2 , and ρ ˉ > 0 \bar{\rho} > 0 ρ ˉ > 0 .
Output: Pair ( Y ˉ , p ˉ ) (\bar{Y},\bar{p}) ( Y ˉ , p ˉ ) such that ( Y ˉ Y ˉ T , p ˉ ) (\bar{Y}\bar{Y}^{\mathsf{T}},\bar{p}) ( Y ˉ Y ˉ T , p ˉ ) satisfies (15).
Set t ← 1 t \leftarrow1 t ← 1 .
Perform the following U U U -space HLR (U-HLR) call to approximately minimize t t t -th AL subproblem:
1.a) Set ( U ~ 0 , r ^ 1 ) ← ( Y t − 1 , r t − 1 ) (\widetilde{U}_0,\hat{r}_1) \leftarrow(Y_{t-1},r_{t-1}) ( U 0 , r ^ 1 ) ← ( Y t − 1 , r t − 1 ) and k ← 1 k \leftarrow1 k ← 1 .
1.b) Call ADAP-AIPP in Appendix B.2 with input
W 0 = U ~ k − 1 , g ( U ) = L β t − 1 ( U U T ; p t − 1 ) , Ω = R n × r ^ k , η 0 = η 0 , M 0 = M 0 , ρ ˉ = ρ ˉ (66) W_0=\widetilde{U}_{k-1},\quad g(U)=\mathcal{L}_{\beta_{t-1}}(UU^{\mathsf{T}};p_{t-1}),\quad\Omega=\mathbb{R}^{n\times\hat{r}_k},\quad\eta_0=\eta_0,\quad M_0=M_0,\quad\bar{\rho}=\bar{\rho} \tag*{(66)} W 0 = U k − 1 , g ( U ) = L β t − 1 ( U U T ; p t − 1 ) , Ω = R n × r ^ k , η 0 = η 0 , M 0 = M 0 , ρ ˉ = ρ ˉ (66)
to obtain output pair ( U k , R k ) ∈ R n × r ^ k × R n × r ^ k (U_k,R_k) \in\mathbb{R}^{n\times\hat{r}_k}\times\mathbb{R}^{n\times\hat{r}_k} ( U k , R k ) ∈ R n × r ^ k × R n × r ^ k and set
S k ← C − A ∗ [ p t − 1 + β t − 1 r ( U k U k T ) ] . S_k \leftarrow C-\mathcal{A}^{*}[p_{t-1}+\beta_{t-1}r(U_kU_k^{\mathsf{T}})]. S k ← C − A ∗ [ p t − 1 + β t − 1 r ( U k U k T )] .
1.c) Compute a minimum eigenpair ( λ min ( S k ) , v min ( S k ) ) (\lambda_{\min}(S_k),v_{\min}(S_k)) ( λ m i n ( S k ) , v m i n ( S k )) of S k S_k S k with ∥ v min ( S k ) ∥ = 1 \lVert v_{\min}(S_k)\rVert=1 ∥ v m i n ( S k )∥ = 1 , and set
y k F : = { v min ( S k ) , λ min ( S k ) < 0 , 0 , otherwise . (67) y_k^F :=
\begin{cases}
v_{\min}(S_k), & \lambda_{\min}(S_k)<0,\\
0, & \text{otherwise}.
\end{cases} \tag*{(67)} y k F := { v m i n ( S k ) , 0 , λ m i n ( S k ) < 0 , otherwise . (67)
1.d) If
λ min ( S k ) ≥ − ε d , ∣ ⟨ S k U k , U k ⟩ ∣ ≤ ε c , (68) \lambda_{\min}(S_k)\ge-\varepsilon_d,\qquad\lvert\langle S_kU_k,U_k\rangle\rvert\le\varepsilon_c, \tag*{(68)} λ m i n ( S k ) ≥ − ε d , ∣⟨ S k U k , U k ⟩∣ ≤ ε c , (68)
then set ( Y t , r t ) ← ( U k , r ^ k ) (Y_t,r_t)\leftarrow(U_k,\hat{r}_k) ( Y t , r t ) ← ( U k , r ^ k ) and go to Step 2 .
1.e) Compute
( α k , ω k ) ∈ arg min { L β t − 1 ( α U k U k T + ω y k F ( y k F ) T ; p t − 1 ) : α ≥ 0 , ω ≥ 0 } (69) (\alpha_k,\omega_k)\in\operatorname*{arg\,min}\left\{\mathcal{L}_{\beta_{t-1}}\left(\alpha U_kU_k^{\mathsf{T}}+\omega y_k^F(y_k^F)^{\mathsf{T}};p_{t-1}\right):\alpha\ge0,\ \omega\ge0\right\} \tag*{(69)} ( α k , ω k ) ∈ arg min { L β t − 1 ( α U k U k T + ω y k F ( y k F ) T ; p t − 1 ) : α ≥ 0 , ω ≥ 0 } (69)
and set
( U ~ k , r ^ k + 1 ) : = { ( ω k y k F , 1 ) , α k = 0 , ( [ α k U k , ω k y k F ] , r ^ k + 1 ) , α k > 0. (70) (\widetilde{U}_k,\hat{r}_{k+1}):=
\begin{cases}
\left(\sqrt{\omega_k}y_k^F,1\right), & \alpha_k=0,\\
\left([\sqrt{\alpha_k}U_k,\sqrt{\omega_k}y_k^F],\hat{r}_k+1\right), & \alpha_k>0.
\end{cases} \tag*{(70)} ( U k , r ^ k + 1 ) := { ( ω k y k F , 1 ) , ( [ α k U k , ω k y k F ] , r ^ k + 1 ) , α k = 0 , α k > 0. (70)
1.f) Set k ← k + 1 k\leftarrow k+1 k ← k + 1 and return to Step 1.b).
Set
p t ← p t − 1 + β t − 1 r ( Y t Y t T ) . (71) p_t\leftarrow p_{t-1}+\beta_{t-1}r(Y_tY_t^{\mathsf{T}}). \tag*{(71)} p t ← p t − 1 + β t − 1 r ( Y t Y t T ) . (71)
If ∥ r ( Y t Y t T ) ∥ ≤ ε p \lVert r(Y_tY_t^{\mathsf{T}})\rVert\le\varepsilon_p ∥ r ( Y t Y t T )∥ ≤ ε p , then output ( Y ˉ , p ˉ ) ← ( Y t , p t ) (\bar{Y},\bar{p})\leftarrow(Y_t,p_t) ( Y ˉ , p ˉ ) ← ( Y t , p t ) and stop.
Set
β t ← 2 β t − 1 , t ← t + 1 , (72) \beta_t\leftarrow2\beta_{t-1},\qquad t\leftarrow t+1, \tag*{(72)} β t ← 2 β t − 1 , t ← t + 1 , (72)
and return to Step 1.
Theorem 4.1. Set ε : = ( ε p , ε c , ε d ) \varepsilon:=(\varepsilon_p,\varepsilon_c,\varepsilon_d) ε := ( ε p , ε c , ε d ) . Let C 1 C_1 C 1 , τ ∗ \tau_* τ ∗ , and β ˉ ( ε ) \bar{\beta}(\varepsilon) β ˉ ( ε ) be as in (24), let T T T be as in (25), and let κ ˉ \bar{\kappa} κ ˉ and τ ˉ \bar{\tau} τ ˉ be as in (48) and (49), respectively. Define
C ˉ ( ε ) : = 24 β ˉ ( ε ) ∥ A ∥ 2 ( 3 2 τ ˉ + 1 ) 2 + 8 [ ∥ C ∥ + ∥ A ∥ ( ∥ p ∗ ∥ + ∥ p 0 − p ∗ ∥ 2 + 4 ε p C 1 β ˉ ( ε ) τ ∗ + β ˉ ( ε ) ∥ b ∥ ) ] . (73) \bar{C}(\varepsilon):=24\bar{\beta}(\varepsilon)\lVert\mathcal{A}\rVert^2(3\sqrt{2}\bar{\tau}+1)^2+8\left[\lVert C\rVert+\lVert\mathcal{A}\rVert\left(\lVert p_*\rVert+\sqrt{\lVert p_0-p_*\rVert^2+4\varepsilon_p C_1\bar{\beta}(\varepsilon)\tau_*}+\bar{\beta}(\varepsilon)\lVert b\rVert\right)\right]. \tag*{(73)} C ˉ ( ε ) := 24 β ˉ ( ε ) ∥ A ∥ 2 ( 3 2 τ ˉ + 1 ) 2 + 8 [ ∥ C ∥ + ∥ A ∥ ( ∥ p ∗ ∥ + ∥ p 0 − p ∗ ∥ 2 + 4 ε p C 1 β ˉ ( ε ) τ ∗ + β ˉ ( ε ) ∥ b ∥ ) ] . (73) The following statements about HALLaR+ hold:
(a) it returns a pair ( Y ˉ , p ˉ ) (\bar{Y},\bar{p}) ( Y ˉ , p ˉ ) such that ( Y ˉ Y ˉ ⊤ , p ˉ ) (\bar{Y}\bar{Y}^{\top},\bar{p}) ( Y ˉ Y ˉ ⊤ , p ˉ ) is an ( ε p , ε c , ε d ) (\varepsilon_p,\varepsilon_c,\varepsilon_d) ( ε p , ε c , ε d ) -solution of (11)–(12) in at most T T T outer iterations and at most
2 T + 24 τ ˉ 2 ∥ A ∥ 2 β ˉ ( ε ) ( 1 ε c + 2 ε d ) (74) 2T+24\bar{\tau}^{2}\lVert A\rVert^{2}\bar{\beta}(\varepsilon)\left(\frac{1}{\varepsilon_c}+\frac{2}{\varepsilon_d}\right) \tag*{(74)} 2 T + 24 τ ˉ 2 ∥ A ∥ 2 β ˉ ( ε ) ( ε c 1 + ε d 2 ) (74)
inner iterations/minimum eigenvector (MEV) computations;
(b) the total number of ADAP-AIPP iterations/ADAP-FISTA calls it performs in its Step 1.b) is at most
( 1 + ⌈ log 2 + ( η 0 C ˉ ( ε ) ) ⌉ ) [ 2 T + 24 τ ˉ 2 ∥ A ∥ 2 β ˉ ( ε ) ( 1 ε c + 2 ε d ) ] + C σ max { η 0 − 1 , C ˉ ( ε ) } ρ ˉ 2 ( κ ˉ + ∥ b ∥ ∥ p ∗ ∥ ) , (75) \left(1+\left\lceil\log_{2}^{+}\left(\eta_{0}\bar{C}(\varepsilon)\right)\right\rceil\right)\left[2T+24\bar{\tau}^{2}\lVert A\rVert^{2}\bar{\beta}(\varepsilon)\left(\frac{1}{\varepsilon_c}+\frac{2}{\varepsilon_d}\right)\right]+\frac{C_{\sigma}\max\{\eta_{0}^{-1},\bar{C}(\varepsilon)\}}{\bar{\rho}^{2}}(\bar{\kappa}+\lVert b\rVert\lVert p_{*}\rVert), \tag*{(75)} ( 1 + ⌈ log 2 + ( η 0 C ˉ ( ε ) ) ⌉ ) [ 2 T + 24 τ ˉ 2 ∥ A ∥ 2 β ˉ ( ε ) ( ε c 1 + ε d 2 ) ] + ρ ˉ 2 C σ max { η 0 − 1 , C ˉ ( ε )} ( κ ˉ + ∥ b ∥ ∥ p ∗ ∥) , (75)
where η 0 \eta_{0} η 0 and ρ ˉ \bar{\rho} ρ ˉ are inputs to HALLaR+ and C σ C_{\sigma} C σ is as in (175).
Proof of Theorem 4.1
We first state a key proposition, without proof, which will be instrumental in proving Theorem 4.1. Due to the length of its proof, we defer the proof to Subsection 4.2.1.
Proposition 4.2. Consider the t t t -th iteration of HALLaR+ and let
C ( τ t , β t − 1 ) : = 24 β t − 1 ∥ A ∥ 2 ( 3 2 τ t + 1 ) 2 + 8 ∥ C − A ∗ ( p t − 1 + β t − 1 b ) ∥ (76) C(\tau_t,\beta_{t-1}):=24\beta_{t-1}\lVert A\rVert^{2}(3\sqrt{2\tau_t}+1)^{2}+8\lVert C-A^{*}(p_{t-1}+\beta_{t-1}b)\rVert \tag*{(76)} C ( τ t , β t − 1 ) := 24 β t − 1 ∥ A ∥ 2 ( 3 2 τ t + 1 ) 2 + 8 ∥ C − A ∗ ( p t − 1 + β t − 1 b )∥ (76) where τ t \tau_t τ t is as in (51) with Z t − 1 : = Y t − 1 Y t − 1 ⊤ Z_{t-1}:=Y_{t-1}Y_{t-1}^{\top} Z t − 1 := Y t − 1 Y t − 1 ⊤ . The following statements about the U-HLR call in Step 1 hold:
(a) it performs at most
⌈ 2 + 12 τ t 2 β t − 1 ∥ A ∥ 2 ( 1 ε c + 2 ε d ) ⌉ (77) \left\lceil2+12\tau_t^{2}\beta_{t-1}\lVert A\rVert^{2}\left(\frac{1}{\varepsilon_c}+\frac{2}{\varepsilon_d}\right)\right\rceil \tag*{(77)} ⌈ 2 + 12 τ t 2 β t − 1 ∥ A ∥ 2 ( ε c 1 + ε d 2 ) ⌉ (77) iterations/minimum eigenvector (MEV) computations to find a pair ( Y t , r t ) (Y_t,r_t) ( Y t , r t ) satisfying
λ min ( ∇ L β t − 1 ( Y t Y t ⊤ ; p t − 1 ) ) ≥ − ε d , ∣ ⟨ ∇ L β t − 1 ( Y t Y t ⊤ ; p t − 1 ) Y t , Y t ⟩ ∣ ≤ ε c , (78) \lambda_{\min}\left(\nabla L_{\beta_{t-1}}(Y_tY_t^{\top};p_{t-1})\right)\geq-\varepsilon_d,\qquad\left\lvert\left\langle\nabla L_{\beta_{t-1}}(Y_tY_t^{\top};p_{t-1})Y_t,Y_t\right\rangle\right\rvert\leq\varepsilon_c, \tag*{(78)} λ m i n ( ∇ L β t − 1 ( Y t Y t ⊤ ; p t − 1 ) ) ≥ − ε d , ⟨ ∇ L β t − 1 ( Y t Y t ⊤ ; p t − 1 ) Y t , Y t ⟩ ≤ ε c , (78) L β t − 1 ( Y t Y t ⊤ ; p t − 1 ) ≤ L β t − 1 ( Y t − 1 Y t − 1 ⊤ ; p t − 1 ) , (79) L_{\beta_{t-1}}(Y_tY_t^{\top};p_{t-1})\leq L_{\beta_{t-1}}(Y_{t-1}Y_{t-1}^{\top};p_{t-1}), \tag*{(79)} L β t − 1 ( Y t Y t ⊤ ; p t − 1 ) ≤ L β t − 1 ( Y t − 1 Y t − 1 ⊤ ; p t − 1 ) , (79) where ( ε c , ε d ) (\varepsilon_c,\varepsilon_d) ( ε c , ε d ) are input tolerances;
(b) the total number of ADAP-AIPP iterations/ADAP-FISTA calls it performs is at most
( 1 + ⌈ log 2 + ( η 0 C ( τ t , β t − 1 ) ) ⌉ ) ( 2 + 12 τ t 2 β t − 1 ∥ A ∥ 2 ( 1 ε c + 2 ε d ) ) + C σ max { η 0 − 1 , C ( τ t , β t − 1 ) } ρ ˉ 2 [ L β t − 1 ( Y t − 1 Y t − 1 ⊤ ; p t − 1 ) − L β t − 1 ( Y t Y t ⊤ ; p t − 1 ) ] \left(1+\left\lceil\log_{2}^{+}\left(\eta_{0}C(\tau_t,\beta_{t-1})\right)\right\rceil\right)\left(2+12\tau_t^{2}\beta_{t-1}\lVert A\rVert^{2}\left(\frac{1}{\varepsilon_c}+\frac{2}{\varepsilon_d}\right)\right)
+\frac{C_{\sigma}\max\{\eta_{0}^{-1},C(\tau_t,\beta_{t-1})\}}{\bar{\rho}^{2}}\left[L_{\beta_{t-1}}(Y_{t-1}Y_{t-1}^{\top};p_{t-1})-L_{\beta_{t-1}}(Y_tY_t^{\top};p_{t-1})\right] ( 1 + ⌈ log 2 + ( η 0 C ( τ t , β t − 1 ) ) ⌉ ) ( 2 + 12 τ t 2 β t − 1 ∥ A ∥ 2 ( ε c 1 + ε d 2 ) ) + ρ ˉ 2 C σ max { η 0 − 1 , C ( τ t , β t − 1 )} [ L β t − 1 ( Y t − 1 Y t − 1 ⊤ ; p t − 1 ) − L β t − 1 ( Y t Y t ⊤ ; p t − 1 ) ] where η 0 \eta_{0} η 0 and ρ ˉ \bar{\rho} ρ ˉ are inputs to HALLaR+ and C σ C_{\sigma} C σ and C ( τ t , β t − 1 ) C(\tau_t,\beta_{t-1}) C ( τ t , β t − 1 ) are as in (175) and (74), respectively.
We are now ready to prove Theorem 4.1.
Proof of Theorem 4.1. For every generated iteration index t ≥ 0 t\geq0 t ≥ 0 , set Z t : = Y t Y t ⊤ Z_t:=Y_tY_t^{\top} Z t := Y t Y t ⊤ .
(a) Let t ≥ 1 t\geq1 t ≥ 1 be a generated iteration index. Proposition 4.2(a) implies that the U-HLR call in Step 1 terminates with a matrix Y t Y_t Y t satisfying (76) and (77). Moreover, relations (22) and (69) imply that
∇ X L β t − 1 ( Z t ; p t − 1 ) = C − A ∗ [ p t − 1 + β t − 1 r ( Z t ) ] = C − A ∗ p t = S t , \nabla_XL_{\beta_{t-1}}(Z_t;p_{t-1})=C-A^{*}[p_{t-1}+\beta_{t-1}r(Z_t)]=C-A^{*}p_t=S_t, ∇ X L β t − 1 ( Z t ; p t − 1 ) = C − A ∗ [ p t − 1 + β t − 1 r ( Z t )] = C − A ∗ p t = S t ,
and the identity Z t = Y t Y t ⊤ Z_t=Y_tY_t^{\top} Z t = Y t Y t ⊤ implies that ⟨ S t Y t , Y t ⟩ = ⟨ S t , Z t ⟩ \langle S_tY_t,Y_t\rangle=\langle S_t,Z_t\rangle ⟨ S t Y t , Y t ⟩ = ⟨ S t , Z t ⟩ . Hence, (76) implies that ( Z t , p t ) (Z_t,p_t) ( Z t , p t ) satisfies (21), and (77) is precisely (53). Thus, Step 1 of HALLaR+ implements Step 1 of the AL framework, and Steps 2–4 of HALLaR+ coincide with Steps 2–4 of the AL framework with Z t = Y t Y t ⊤ Z_t=Y_tY_t^{\top} Z t = Y t Y t ⊤ . It then follows from Theorem 2.4 that HALLaR+ terminates at an outer iteration index t ˉ ≤ T \bar{t}\leq T t ˉ ≤ T with a pair ( Y ˉ , p ˉ ) (\bar{Y},\bar{p}) ( Y ˉ , p ˉ ) such that ( Y ˉ Y ˉ ⊤ , p ˉ ) (\bar{Y}\bar{Y}^{\top},\bar{p}) ( Y ˉ Y ˉ ⊤ , p ˉ ) is an ( ε p , ε c , ε d ) (\varepsilon_p,\varepsilon_c,\varepsilon_d) ( ε p , ε c , ε d ) -solution of (11)–(12), that β ˉ = β t ˉ − 1 ≤ β ˉ ( ε ) \bar{\beta}=\beta_{\bar{t}-1}\leq\bar{\beta}(\varepsilon) β ˉ = β t ˉ − 1 ≤ β ˉ ( ε ) , and that every multiplier p t p_t p t with t ≥ 1 t\geq1 t ≥ 1 satisfies (27).
The proofs of Lemmas 3.3 and 3.4 use only (21), (53), and the outer updates. Hence, they apply with Z t = Y t Y t ⊤ Z_t=Y_tY_t^{\top} Z t = Y t Y t ⊤ and imply τ t ≤ τ ˉ \tau_t\leq\bar{\tau} τ t ≤ τ ˉ . Applying the induction used in (60) to (75) and using t ˉ ≤ T \bar{t}\leq T t ˉ ≤ T and (26) yields (72). Since every inner iteration performs one MEV computation, this proves (a). (b) For every 1 ≤ t ≤ t ˉ 1 \le t \le\bar{t} 1 ≤ t ≤ t ˉ , the triangle inequality, (29) , and (28) imply
∥ C − A ∗ ( p t − 1 + β t − 1 b ) ∥ ≤ ∥ C ∥ + ∥ A ∥ ( ∥ p t − 1 ∥ + β t − 1 ∥ b ∥ ) ≤ ∥ C ∥ + ∥ A ∥ ( ∥ p ∗ ∥ + ∥ p 0 − p ∗ ∥ 2 + 4 ε p C 1 β ˉ ( ε ) τ ∗ + β ˉ ( ε ) ∥ b ∥ ) . \begin{aligned}
\left\|C-A^{*}(p_{t-1}+\beta_{t-1}b)\right\| \le\|C\|+\|A\|\left(\|p_{t-1}\|+\beta_{t-1}\|b\|\right) \\
&\le\|C\|+\|A\|\left(\|p_{*}\|+\sqrt{\|p_{0}-p_{*}\|^{2}+4\varepsilon_{p}C_{1}\bar{\beta}(\varepsilon)\tau_{*}}+\bar{\beta}(\varepsilon)\|b\|\right).
\end{aligned} ∥ C − A ∗ ( p t − 1 + β t − 1 b ) ∥ ≤ ∥ C ∥ + ∥ A ∥ ( ∥ p t − 1 ∥ + β t − 1 ∥ b ∥ ) ≤ ∥ C ∥ + ∥ A ∥ ( ∥ p ∗ ∥ + ∥ p 0 − p ∗ ∥ 2 + 4 ε p C 1 β ˉ ( ε ) τ ∗ + β ˉ ( ε ) ∥ b ∥ ) .
where, for t = 1 t=1 t = 1 , the last inequality uses ∥ p 0 ∥ ≤ ∥ p ∗ ∥ + ∥ p 0 − p ∗ ∥ \|p_{0}\|\le\|p_{*}\|+\|p_{0}-p_{*}\| ∥ p 0 ∥ ≤ ∥ p ∗ ∥ + ∥ p 0 − p ∗ ∥ . Together with τ t ≤ τ ˉ \tau_{t}\le\bar{\tau} τ t ≤ τ ˉ and definitions (73) and (76) , this implies
C ( τ t , β t − 1 ) ≤ C ˉ ( ε ) . (80) \mathcal{C}(\tau_{t},\beta_{t-1})\le\bar{\mathcal{C}}(\varepsilon). \tag*{(80)} C ( τ t , β t − 1 ) ≤ C ˉ ( ε ) . (80)
Lemma 3.5, applied with Z t = Y t Y t T Z_{t}=Y_{t}Y_{t}^{\mathsf{T}} Z t = Y t Y t T , bounds the total AL decrease by (63) . Combining this bound with Proposition 4.2 (b), (80) , and part (a) yields (75) .
Proof of Proposition 4.2
This subsection is devoted to proving Proposition 4.2 , which establishes the iteration complexity of the U-HLR call performed in Step 1 of the t t t -th iteration of HAllaR+. Accordingly, throughout this subsection, we fix an arbitrary iteration index t ≥ 1 t\ge1 t ≥ 1 and consider just the t t t -th iteration of HAllaR+. We now define several quantities which arise in the technical complexity analysis of the U-HLR call. Let
ϕ t ( ⋅ ) : = L β t − 1 ( ⋅ ; p t − 1 ) , g t ( U ) : = ϕ t ( U U T ) . (81) \phi_{t}(\cdot):=\mathcal{L}_{\beta_{t-1}}(\cdot;p_{t-1}),\qquad g_{t}(U):=\phi_{t}(UU^{\mathsf{T}}). \tag*{(81)} ϕ t ( ⋅ ) := L β t − 1 ( ⋅ ; p t − 1 ) , g t ( U ) := ϕ t ( U U T ) . (81)
For every iteration index k ≥ 1 k\ge1 k ≥ 1 of the U-HLR call in Step 1, define
D ~ k : = 6 D ( U ~ k − 1 ; g t ) + 1 , Lip ~ k : = [ LIP ] ( D ~ k ; g t ) , (82) \widetilde{D}_{k}:=6D(\widetilde{U}_{k-1};g_{t})+1,\qquad\widetilde{\operatorname{Lip}}_{k}:=[\operatorname{LIP}](\widetilde{D}_{k};g_{t}), \tag*{(82)} D k := 6 D ( U k − 1 ; g t ) + 1 , Lip k := [ LIP ] ( D k ; g t ) , (82)
M ~ k : = 24 β t − 1 ∥ A ∥ 2 D ~ k 2 + 8 ∥ C − A ∗ ( p t − 1 + β t − 1 b ) ∥ (83) \widetilde{M}_{k}:=24\beta_{t-1}\|A\|^{2}\widetilde{D}_{k}^{2}+8\left\|C-A^{*}(p_{t-1}+\beta_{t-1}b)\right\| \tag*{(83)} M k := 24 β t − 1 ∥ A ∥ 2 D k 2 + 8 ∥ C − A ∗ ( p t − 1 + β t − 1 b ) ∥ (83)
where D ( ⋅ ; ⋅ ) D(\cdot;\cdot) D ( ⋅ ; ⋅ ) is as in (8) , and [ LIP ] ( ⋅ ; ⋅ ) [\operatorname{LIP}](\cdot;\cdot) [ LIP ] ( ⋅ ; ⋅ ) is as in (11) .
We now start off with a lemma that shows that the call to ADAP-AIPP in Step 1.b) is made with inputs that satisfy assumptions (C1), (C2), and (C4) in Appendix § 8 .§ 2 . It is easy to see that assumption (C3) is always satisfied.
Lemma 4.3. Let k ≥ 1 k\ge1 k ≥ 1 and consider the k k k -th iteration of the U-HLR call in Step 1. The input of the ADAP-AIPP call made in Step 1.b) of U-HLR satisfies
(a) the function g t ( U ) = ϕ t ( U U T ) g_{t}(U)=\phi_{t}(UU^{\mathsf{T}}) g t ( U ) = ϕ t ( U U T ) is differentiable with locally Lipschitz continuous gradient. Moreover,
Lip ~ k ≤ 6 β t − 1 ∥ A ∥ 2 D ~ k 2 + 2 ∥ C − A ∗ ( p t − 1 + β t − 1 b ) ∥ ; (84) \widetilde{\operatorname{Lip}}_{k}\le6\beta_{t-1}\|A\|^{2}\widetilde{D}_{k}^{2}+2\left\|C-A^{*}(p_{t-1}+\beta_{t-1}b)\right\|; \tag*{(84)} Lip k ≤ 6 β t − 1 ∥ A ∥ 2 D k 2 + 2 ∥ C − A ∗ ( p t − 1 + β t − 1 b ) ∥ ; (84) (b) g t ( U ) g_{t}(U) g t ( U ) is bounded below on R n × r ^ k \mathbb{R}^{n\times\widehat{r}_{k}} R n × r k ;
(c) the sublevel set S ( U ~ k − 1 ; g t ) S(\widetilde{U}_{k-1};g_{t}) S ( U k − 1 ; g t ) has finite diameter D ( U ~ k − 1 ; g t ) D(\widetilde{U}_{k-1};g_{t}) D ( U k − 1 ; g t ) , where S ( ⋅ ; ⋅ ) S(\cdot;\cdot) S ( ⋅ ; ⋅ ) and D ( ⋅ ; ⋅ ) D(\cdot;\cdot) D ( ⋅ ; ⋅ ) are as in (7) and (8) , respectively.
Proof. (a) It is easy to see that g t ( U ) = ϕ t ( U U T ) g_{t}(U)=\phi_{t}(UU^{\mathsf{T}}) g t ( U ) = ϕ t ( U U T ) is differentiable. To show g t ( U ) g_{t}(U) g t ( U ) has locally Lipschitz continuous gradient, let R > 0 R>0 R > 0 be an arbitrary scalar and B [ R ] B[R] B [ R ] be a ball centered at 0 with radius R R R . It suffices to show that g t ( U ) g_{t}(U) g t ( U ) has Lipschitz continuous gradient on B [ R ] B[R] B [ R ] . Thus, let U U U and V V V be in B [ R ] B[R] B [ R ] . First, it is easy to see that
∥ U U T − V V T ∥ = ∥ U ( U − V ) T − ( V − U ) V T ∥ ≤ ∥ U ∥ ∥ U − V ∥ + ∥ V ∥ ∥ U − V ∥ ≤ 2 R ∥ U − V ∥ . (85) \begin{aligned}
\left\|UU^{\mathsf{T}}-VV^{\mathsf{T}}\right\|=\left\|U(U-V)^{\mathsf{T}}-(V-U)V^{\mathsf{T}}\right\| \\
&\le\|U\|\|U-V\|+\|V\|\|U-V\|\le2R\|U-V\|. \tag*{(85)}
\end{aligned} U U T − V V T = U ( U − V ) T − ( V − U ) V T ≤ ∥ U ∥∥ U − V ∥ + ∥ V ∥∥ U − V ∥ ≤ 2 R ∥ U − V ∥. (85)
Moreover, it follows from the fact that ϕ t ( ⋅ ) \phi_{t}(\cdot) ϕ t ( ⋅ ) is ( β t − 1 ∥ A ∥ 2 ) (\beta_{t-1}\|A\|^{2}) ( β t − 1 ∥ A ∥ 2 ) -smooth that
∥ ∇ ϕ t ( V V T ) ∥ = ∥ ∇ ϕ t ( V V T ) − ∇ ϕ t ( 0 ) + ∇ ϕ t ( 0 ) ∥ ≤ β t − 1 ∥ A ∥ 2 ∥ V V T ∥ + ∥ ∇ ϕ t ( 0 ) ∥ ≤ β t − 1 ∥ A ∥ 2 R 2 + ∥ ∇ ϕ t ( 0 ) ∥ . (86) \begin{aligned}
\|\nabla\phi_{t}(VV^{\mathsf{T}})\|=\|\nabla\phi_{t}(VV^{\mathsf{T}})-\nabla\phi_{t}(0)+\nabla\phi_{t}(0)\|
\le\beta_{t-1}\|A\|^{2}\|VV^{\mathsf{T}}\|+\|\nabla\phi_{t}(0)\| \\
&\le\beta_{t-1}\|A\|^{2}R^{2}+\|\nabla\phi_{t}(0)\|. \tag*{(86)}
\end{aligned} ∥∇ ϕ t ( V V T ) ∥ = ∥∇ ϕ t ( V V T ) − ∇ ϕ t ( 0 ) + ∇ ϕ t ( 0 ) ∥ ≤ β t − 1 ∥ A ∥ 2 ∥ V V T ∥ + ∥∇ ϕ t ( 0 ) ∥ ≤ β t − 1 ∥ A ∥ 2 R 2 + ∥∇ ϕ t ( 0 ) ∥. (86)
It then follows from relations (85) and (86) , the fact that ϕ t \phi_{t} ϕ t is ( β t − 1 ∥ A ∥ 2 ) (\beta_{t-1}\|A\|^{2}) ( β t − 1 ∥ A ∥ 2 ) -smooth, and the fact that ∇ g t ( U ) = 2 ∇ ϕ t ( U U T ) U \nabla g_{t}(U)=2\nabla\phi_{t}(UU^{\mathsf{T}})U ∇ g t ( U ) = 2∇ ϕ t ( U U T ) U that
∥ ∇ g t ( U ) − ∇ g t ( V ) ∥ ≤ ∥ 2 ∇ ϕ t ( U U T ) U − 2 ∇ ϕ t ( V V T ) U + 2 ∇ ϕ t ( V V T ) U − 2 ∇ ϕ t ( V V T ) V ∥ \|\nabla g_{t}(U)-\nabla g_{t}(V)\|\le\|2\nabla\phi_{t}(UU^{\mathsf{T}})U-2\nabla\phi_{t}(VV^{\mathsf{T}})U+2\nabla\phi_{t}(VV^{\mathsf{T}})U-2\nabla\phi_{t}(VV^{\mathsf{T}})V\| ∥∇ g t ( U ) − ∇ g t ( V ) ∥ ≤ ∥2∇ ϕ t ( U U T ) U − 2∇ ϕ t ( V V T ) U + 2∇ ϕ t ( V V T ) U − 2∇ ϕ t ( V V T ) V ∥
∥ ∇ g t ( U ) − ∇ g t ( V ) ∥ ≤ 2 ∥ U ∥ ∥ ∇ ϕ t ( U U ⊤ ) − ∇ ϕ t ( V V ⊤ ) ∥ + 2 ∥ ∇ ϕ t ( V V ⊤ ) ∥ ∥ U − V ∥ ≤ 2 β t − 1 ∥ A ∥ 2 ∥ U ∥ ∥ U U ⊤ − V V ⊤ ∥ + 2 ∥ ∇ ϕ t ( V V ⊤ ) ∥ ∥ U − V ∥ ≤ (83),(84) ≤ 4 β t − 1 ∥ A ∥ 2 R 2 ∥ U − V ∥ + 2 ( β t − 1 ∥ A ∥ 2 R 2 + ∥ ∇ ϕ t ( 0 ) ∥ ) ∥ U − V ∥ = ( 6 β t − 1 ∥ A ∥ 2 R 2 + 2 ∥ ∇ ϕ t ( 0 ) ∥ ) ∥ U − V ∥ . (87) \begin{aligned}
\|\nabla g_t(U)-\nabla g_t(V)\| &\le2\|U\|\|\nabla\phi_t(UU^\top)-\nabla\phi_t(VV^\top)\|+2\|\nabla\phi_t(VV^\top)\|\|U-V\|\\
&\le2\beta_{t-1}\|A\|^2\|U\|\|UU^\top-VV^\top\|+2\|\nabla\phi_t(VV^\top)\|\|U-V\|\\
&\mathrel{\phantom{\le}}{\scriptstyle\text{(83),(84)}}\le4\beta_{t-1}\|A\|^2R^2\|U-V\|+2(\beta_{t-1}\|A\|^2R^2+\|\nabla\phi_t(0)\|)\|U-V\|\\
&=(6\beta_{t-1}\|A\|^2R^2+2\|\nabla\phi_t(0)\|)\|U-V\|. \tag*{(87)}
\end{aligned} ∥∇ g t ( U ) − ∇ g t ( V ) ∥ ≤ 2∥ U ∥∥∇ ϕ t ( U U ⊤ ) − ∇ ϕ t ( V V ⊤ ) ∥ + 2∥∇ ϕ t ( V V ⊤ ) ∥∥ U − V ∥ ≤ 2 β t − 1 ∥ A ∥ 2 ∥ U ∥∥ U U ⊤ − V V ⊤ ∥ + 2∥∇ ϕ t ( V V ⊤ ) ∥∥ U − V ∥ ≤ (83),(84) ≤ 4 β t − 1 ∥ A ∥ 2 R 2 ∥ U − V ∥ + 2 ( β t − 1 ∥ A ∥ 2 R 2 + ∥∇ ϕ t ( 0 ) ∥ ) ∥ U − V ∥ = ( 6 β t − 1 ∥ A ∥ 2 R 2 + 2∥∇ ϕ t ( 0 ) ∥ ) ∥ U − V ∥. (87)
Hence, it follows that ∇ g t ( U ) \nabla g_t(U) ∇ g t ( U ) is ( 6 β t − 1 ∥ A ∥ 2 R 2 + 2 ∥ ∇ ϕ t ( 0 ) ∥ ) (6\beta_{t-1}\|A\|^2R^2+2\|\nabla\phi_t(0)\|) ( 6 β t − 1 ∥ A ∥ 2 R 2 + 2∥∇ ϕ t ( 0 ) ∥ ) -Lipschitz continuous on B [ R ] B[R] B [ R ] . Since R R R was arbitrary, it follows that ∇ g t ( U ) \nabla g_t(U) ∇ g t ( U ) is locally Lipschitz continuous. Moreover, it follows from the definitions of ( Lip ) g k (\operatorname{Lip})_{g_k} ( Lip ) g k and [ LIP ] ( ⋅ ; ⋅ ) [\operatorname{LIP}](\cdot;\cdot) [ LIP ] ( ⋅ ; ⋅ ) in (80) and (9), respectively, and replacing R R R in (85) with D ~ k \widetilde{D}_k D k that ( Lip ) g k ≤ 6 β t − 1 ∥ A ∥ 2 D ~ k 2 + 2 ∥ ∇ ϕ t ( 0 ) ∥ (\operatorname{Lip})_{g_k}\le6\beta_{t-1}\|A\|^2\widetilde{D}_k^2+2\|\nabla\phi_t(0)\| ( Lip ) g k ≤ 6 β t − 1 ∥ A ∥ 2 D k 2 + 2∥∇ ϕ t ( 0 ) ∥ . The bound in (82) then follows from this conclusion and the fact that the definition of ϕ t \phi_t ϕ t in (79) implies that ∇ ϕ t ( 0 ) = C − A ∗ ( p t − 1 + β t − 1 b ) \nabla\phi_t(0)=C-A^*(p_{t-1}+\beta_{t-1}b) ∇ ϕ t ( 0 ) = C − A ∗ ( p t − 1 + β t − 1 b ) .
(b) It is easy to see that
inf U ∈ R n × r ^ k g t ( U ) = inf U ∈ R n × r ^ k ϕ t ( U U ⊤ ) = inf X ⪰ 0 , rank ( X ) ≤ r ^ k ϕ t ( X ) ≥ inf X ⪰ 0 ϕ t ( X ) > − ∞ . (88) \inf_{U\in\mathbb{R}^{n\times\widehat r_k}}g_t(U)=\inf_{U\in\mathbb{R}^{n\times\widehat r_k}}\phi_t(UU^\top)=\inf_{\substack{X\succeq0,\\\operatorname{rank}(X)\le\widehat r_k}}\phi_t(X)\ge\inf_{X\succeq0}\phi_t(X)>-\infty. \tag*{(88)} U ∈ R n × r k inf g t ( U ) = U ∈ R n × r k inf ϕ t ( U U ⊤ ) = X ⪰ 0 , rank ( X ) ≤ r k inf ϕ t ( X ) ≥ X ⪰ 0 inf ϕ t ( X ) > − ∞. (88)
Hence, it follows from (86) that g t g_t g t is bounded below on R n × r ^ k \mathbb{R}^{n\times\widehat r_k} R n × r k .
(c) Let U ∈ S ( U ~ k − 1 ; g t ) U\in S(\widetilde{U}_{k-1};g_t) U ∈ S ( U k − 1 ; g t ) . It follows by the definition of S ( ⋅ ; ⋅ ) S(\cdot;\cdot) S ( ⋅ ; ⋅ ) in (5) that g t ( U ) ≤ g t ( U ~ k − 1 ) g_t(U)\le g_t(\widetilde{U}_{k-1}) g t ( U ) ≤ g t ( U k − 1 ) , which by the definition of g t g_t g t in (79) implies that ϕ t ( U U ⊤ ) ≤ ϕ t ( U ~ k − 1 U ~ k − 1 ⊤ ) \phi_t(UU^\top)\le\phi_t(\widetilde{U}_{k-1}\widetilde{U}_{k-1}^\top) ϕ t ( U U ⊤ ) ≤ ϕ t ( U k − 1 U k − 1 ⊤ ) . Thus, U U ⊤ ∈ S ( U ~ k − 1 U ~ k − 1 ⊤ ; ϕ t ) UU^\top\in S(\widetilde{U}_{k-1}\widetilde{U}_{k-1}^\top;\phi_t) U U ⊤ ∈ S ( U k − 1 U k − 1 ⊤ ; ϕ t ) . Now observe that it follows from Proposition A.1 and the definition of ϕ t \phi_t ϕ t in (79) that the sublevel sets of ϕ t \phi_t ϕ t are bounded. Hence S ( U ~ k − 1 U ~ k − 1 ⊤ ; ϕ t ) S(\widetilde{U}_{k-1}\widetilde{U}_{k-1}^\top;\phi_t) S ( U k − 1 U k − 1 ⊤ ; ϕ t ) is bounded. Thus it follows that ∥ U U ⊤ ∥ ≤ M \|UU^\top\|\le M ∥ U U ⊤ ∥ ≤ M for some scalar M > 0 M>0 M > 0 . Now observe from the Cauchy-Schwarz inequality that
∥ U ∥ 2 = ⟨ U , U ⟩ = ⟨ I n , U U ⊤ ⟩ ≤ n ∥ U U ⊤ ∥ ≤ n M . \|U\|^2=\langle U,U\rangle=\langle I_n,UU^\top\rangle\le\sqrt{n}\|UU^\top\|\le\sqrt{n}M. ∥ U ∥ 2 = ⟨ U , U ⟩ = ⟨ I n , U U ⊤ ⟩ ≤ n ∥ U U ⊤ ∥ ≤ n M .
Thus, we have ∥ U ∥ ≤ n 1 / 4 M \|U\|\le n^{1/4}\sqrt{M} ∥ U ∥ ≤ n 1/4 M . Since U U U was arbitrary, it follows that S ( U ~ k − 1 ; g t ) S(\widetilde{U}_{k-1};g_t) S ( U k − 1 ; g t ) is bounded with finite diameter D ( U ~ k − 1 ; g t ) D(\widetilde{U}_{k-1};g_t) D ( U k − 1 ; g t ) .
The following lemma establishes properties of the ADAP-AIPP call made in Step 1.b).
Lemma 4.4. Consider the k k k -th iteration of the U-HLR call in Step 1. The following statements about the ADAP-AIPP call in Step 1.b) of U-HLR hold:
(a) it terminates with a point U k ∈ R n × r ^ k U_k\in\mathbb{R}^{n\times\widehat r_k} U k ∈ R n × r k satisfying
R k = 2 ∇ ϕ t ( U k U k ⊤ ) U k , ∥ R k ∥ ≤ ρ ˉ (89) R_k=2\nabla\phi_t(U_kU_k^\top)U_k,\qquad\|R_k\|\le\bar{\rho} \tag*{(89)} R k = 2∇ ϕ t ( U k U k ⊤ ) U k , ∥ R k ∥ ≤ ρ ˉ (89) ϕ t ( U k U k ⊤ ) ≤ ϕ t ( U ~ k − 1 U ~ k − 1 ⊤ ) ; (90) \phi_t(U_kU_k^\top)\le\phi_t(\widetilde{U}_{k-1}\widetilde{U}_{k-1}^\top); \tag*{(90)} ϕ t ( U k U k ⊤ ) ≤ ϕ t ( U k − 1 U k − 1 ⊤ ) ; (90) (b) its total number of ADAP-AIPP iterations/ADAP-FISTA calls performed within the ADAP-AIPP call is bounded by
1 + C σ max { η 0 − 1 , M ~ k } ρ ˉ 2 [ ϕ t ( U ~ k − 1 U ~ k − 1 ⊤ ) − ϕ t ( U k U k ⊤ ) ] + ⌈ log 2 + ( η 0 M ~ k ) ⌉ , 1+\frac{C_\sigma\max\{\eta_0^{-1},\widetilde{M}_k\}}{\bar{\rho}^{\,2}}\left[\phi_t(\widetilde{U}_{k-1}\widetilde{U}_{k-1}^\top)-\phi_t(U_kU_k^\top)\right]+\left\lceil\log_2^+(\eta_0\widetilde{M}_k)\right\rceil, 1 + ρ ˉ 2 C σ max { η 0 − 1 , M k } [ ϕ t ( U k − 1 U k − 1 ⊤ ) − ϕ t ( U k U k ⊤ ) ] + ⌈ log 2 + ( η 0 M k ) ⌉ , where C σ C_\sigma C σ is as in (175), M ~ k \widetilde{M}_k M k is as in (81), and η 0 \eta_0 η 0 and ρ ˉ \bar{\rho} ρ ˉ are inputs to ADAP-AIPP.
Proof. (a) Let k ≥ 1 k\ge1 k ≥ 1 and consider the k k k -th iteration of the U-HLR call performed in Step 1. By Lemma 4.3(a)–(c), and because Step 1.b) calls ADAP-AIPP with Ω = R n × r ^ k \Omega=\mathbb{R}^{n\times\widehat r_k} Ω = R n × r k , a closed convex set with easy projection, the inputs of this call satisfy assumptions (C1)–(C4) of Appendix B.2. It then follows from Proposition B.13(a), the fact that the call to ADAP-AIPP is made with initial point W 0 = U ~ k − 1 W_0=\widetilde{U}_{k-1} W 0 = U k − 1 and function g t ( U ) = ϕ t ( U U ⊤ ) g_t(U)=\phi_t(UU^\top) g t ( U ) = ϕ t ( U U ⊤ ) as in (79), and the fact that ADAP-AIPP terminates with ( W ‾ , R ‾ ) = ( U k , R k ) (\overline{W},\overline{R})=(U_k,R_k) ( W , R ) = ( U k , R k ) that the pair ( U k , R k ) (U_k,R_k) ( U k , R k ) satisfies
R k ∈ ∇ g t ( U k ) + N R n × r ^ k ( U k ) , ∥ R k ∥ ≤ ρ ˉ (91) R_k\in\nabla g_t(U_k)+\mathcal{N}_{\mathbb{R}^{n\times\widehat r_k}}(U_k),\qquad\|R_k\|\le\bar{\rho} \tag*{(91)} R k ∈ ∇ g t ( U k ) + N R n × r k ( U k ) , ∥ R k ∥ ≤ ρ ˉ (91)
ϕ t ( U k U k ⊤ ) ≤ ϕ t ( U ~ k − 1 U ~ k − 1 ⊤ ) . (92) \phi_t(U_kU_k^\top)\le\phi_t(\widetilde{U}_{k-1}\widetilde{U}_{k-1}^\top). \tag*{(92)} ϕ t ( U k U k ⊤ ) ≤ ϕ t ( U k − 1 U k − 1 ⊤ ) . (92)
Relations (87) and (88) then follow immediately from (89), (90), and the facts that ∇ g t ( U k ) = 2 ∇ ϕ t ( U k U k ⊤ ) U k \nabla g_t(U_k)=2\nabla\phi_t(U_kU_k^\top)U_k ∇ g t ( U k ) = 2∇ ϕ t ( U k U k ⊤ ) U k and N R n × r ^ k ( U k ) = { 0 } \mathcal{N}_{\mathbb{R}^{n\times\widehat r_k}}(U_k)=\{0\} N R n × r k ( U k ) = { 0 } .
(b) Let k ≥ 1 k\ge1 k ≥ 1 and consider the k k k -th iteration of the U-HLR call performed in Step 1 of HALLaR+. It follows from Lemma 4.3 that the input of the ADAP-AIPP call satisfies assumptions (C1)–(C4) in Appendix B.2. It then follows from this observation, Proposition B.13(b), and the definition of ( Lip ) k (\operatorname{Lip})_k ( Lip ) k in (82) that the total number of ADAP-AIPP iterations/ADAP-FISTA calls performed within the ADAP-AIPP call is upper bounded by
1 + C σ max { η 0 − 1 , 4 ( Lip ) k } ρ ˉ 2 [ ϕ t ( U ~ k − 1 U ~ k − 1 T ) − ϕ t ( U k U k T ) ] + [ log 2 + ( 4 η 0 ( Lip ) k ) ] . 1+\frac{C_{\sigma}\max\left\{\eta_{0}^{-1},4(\operatorname{Lip})_{k}\right\}}{\bar{\rho}^{2}}\left[\phi_t\left(\widetilde{U}_{k-1}\widetilde{U}_{k-1}^{\mathsf{T}}\right)-\phi_t\left(U_kU_k^{\mathsf{T}}\right)\right]+\left[\log_{2}^{+}\left(4\eta_{0}(\operatorname{Lip})_{k}\right)\right]. 1 + ρ ˉ 2 C σ max { η 0 − 1 , 4 ( Lip ) k } [ ϕ t ( U k − 1 U k − 1 T ) − ϕ t ( U k U k T ) ] + [ log 2 + ( 4 η 0 ( Lip ) k ) ] .
The complexity bound in part (b) then follows immediately from this bound and the fact that the bound on ( Lip ) k (\operatorname{Lip})_k ( Lip ) k in Lemma 4.3(a) and the definition of M ~ k \widetilde{M}_k M k in (83) imply that 4 ( Lip ) k ≤ M ~ k 4(\operatorname{Lip})_k\leq\widetilde{M}_k 4 ( Lip ) k ≤ M k .
Lemma 4.5
Lemma 4.5. The following statements about the U-HLR call performed in Step 1 hold:
(a) it is an instance of the RUFW framework of Appendix A.2 applied to
min X ∈ S + n ϕ t ( X ) (93) \min_{X\in\mathbb{S}_{+}^{n}}\phi_t(X) \tag*{(93)} X ∈ S + n min ϕ t ( X ) (93)
with initial point X ~ 0 = Y t − 1 Y t − 1 T \widetilde{X}_0=Y_{t-1}Y_{t-1}^{\mathsf{T}} X 0 = Y t − 1 Y t − 1 T and tolerance pair ( ϵ , δ ) = ( ϵ d , ϵ c ) (\epsilon,\delta)=(\epsilon_d,\epsilon_c) ( ϵ , δ ) = ( ϵ d , ϵ c ) , where ϕ t \phi_t ϕ t is as in (81) ;
(b) for every iteration index k ≥ 1 k\geq1 k ≥ 1 at which Step 1.e) is performed, it holds that
ϕ t ( U ~ k U ~ k T ) ≤ ϕ t ( U k U k T ) ≤ ϕ t ( U ~ k − 1 U ~ k − 1 T ) . (94) \phi_t\left(\widetilde{U}_k\widetilde{U}_k^{\mathsf{T}}\right)\leq\phi_t\left(U_kU_k^{\mathsf{T}}\right)\leq\phi_t\left(\widetilde{U}_{k-1}\widetilde{U}_{k-1}^{\mathsf{T}}\right). \tag*{(94)} ϕ t ( U k U k T ) ≤ ϕ t ( U k U k T ) ≤ ϕ t ( U k − 1 U k − 1 T ) . (94)
Hence, it follows that for every iteration index k ≥ 1 k\geq1 k ≥ 1 , ϕ t ( U ~ k − 1 U ~ k − 1 T ) ≤ ϕ t ( U ~ 0 U ~ 0 T ) \phi_t\left(\widetilde{U}_{k-1}\widetilde{U}_{k-1}^{\mathsf{T}}\right)\leq\phi_t\left(\widetilde{U}_0\widetilde{U}_0^{\mathsf{T}}\right) ϕ t ( U k − 1 U k − 1 T ) ≤ ϕ t ( U 0 U 0 T ) .
Proof. (a) Let k ≥ 1 k\geq1 k ≥ 1 be a generated iteration index of the U-HLR call and associate with the U-HLR iterates the matrices
X k : = U k U k T , X k F : = y k F ( y k F ) T . (95) X_k:=U_kU_k^{\mathsf{T}},\qquad X_k^{F}:=y_k^{F}(y_k^{F})^{\mathsf{T}}. \tag*{(95)} X k := U k U k T , X k F := y k F ( y k F ) T . (95)
Set X ~ 0 : = U ~ 0 U ~ 0 T = Y t − 1 Y t − 1 T \widetilde{X}_0:=\widetilde{U}_0\widetilde{U}_0^{\mathsf{T}}=Y_{t-1}Y_{t-1}^{\mathsf{T}} X 0 := U 0 U 0 T = Y t − 1 Y t − 1 T , where the last identity is due to Step 1.a), and whenever Step 1.e) is executed, also define X ~ k : = U ~ k U ~ k T \widetilde{X}_k:=\widetilde{U}_k\widetilde{U}_k^{\mathsf{T}} X k := U k U k T . Relation (90) gives ϕ t ( X k ) ≤ ϕ t ( X ~ k − 1 ) \phi_t(X_k)\leq\phi_t(\widetilde{X}_{k-1}) ϕ t ( X k ) ≤ ϕ t ( X k − 1 ) , which is (113) . Moreover, S k = C − A ∗ ( p t − 1 + β t − 1 r ( X k ) ) = ∇ X ϕ t ( X k ) S_k=C-\mathcal{A}^{*}(p_{t-1}+\beta_{t-1}r(X_k))=\nabla_X\phi_t(X_k) S k = C − A ∗ ( p t − 1 + β t − 1 r ( X k )) = ∇ X ϕ t ( X k ) and ⟨ S k U k , U k ⟩ = ⟨ S k , X k ⟩ \langle S_kU_k,U_k\rangle=\langle S_k,X_k\rangle ⟨ S k U k , U k ⟩ = ⟨ S k , X k ⟩ . Thus, Steps 1.c)–1.d) of the U-HLR call coincide with Steps 2 and 3 of the RUFW framework with G k = S k G_k=S_k G k = S k and tolerance pair ( ϵ , δ ) = ( ϵ d , ϵ c ) (\epsilon,\delta)=(\epsilon_d,\epsilon_c) ( ϵ , δ ) = ( ϵ d , ϵ c ) . Relation (69) is the RUFW conic search (116) , and direct multiplication in (70) yields
X ~ k = α k X k + ω k X k F , \widetilde{X}_k=\alpha_kX_k+\omega_kX_k^{F}, X k = α k X k + ω k X k F ,
x r e f ‘ ( 115 ) ‘ {xref}`(115)` x r e f ‘ ( 115 ) ‘
which is (117) . Hence the U-HLR call is an instance of the RUFW framework applied to ϕ t \phi_t ϕ t with initial point X ~ 0 = Y t − 1 Y t − 1 T \widetilde{X}_0=Y_{t-1}Y_{t-1}^{\mathsf{T}} X 0 = Y t − 1 Y t − 1 T and tolerance pair ( ϵ , δ ) = ( ϵ d , ϵ c ) (\epsilon,\delta)=(\epsilon_d,\epsilon_c) ( ϵ , δ ) = ( ϵ d , ϵ c ) .
(b) By part (a), Lemma A.4 applies to the iterates X k = U k U k T X_k=U_kU_k^{\mathsf{T}} X k = U k U k T , X k F = y k F ( y k F ) T X_k^{F}=y_k^{F}(y_k^{F})^{\mathsf{T}} X k F = y k F ( y k F ) T , and X ~ k = U ~ k U ~ k T \widetilde{X}_k=\widetilde{U}_k\widetilde{U}_k^{\mathsf{T}} X k = U k U k T . Thus, (120) gives ϕ t ( U ~ k U ~ k T ) ≤ ϕ t ( U k U k T ) ≤ ϕ t ( U ~ k − 1 U ~ k − 1 T ) \phi_t(\widetilde{U}_k\widetilde{U}_k^{\mathsf{T}})\leq\phi_t(U_kU_k^{\mathsf{T}})\leq\phi_t(\widetilde{U}_{k-1}\widetilde{U}_{k-1}^{\mathsf{T}}) ϕ t ( U k U k T ) ≤ ϕ t ( U k U k T ) ≤ ϕ t ( U k − 1 U k − 1 T ) , which is (94) . Applying these relations recursively implies the last conclusion of the lemma.
Lemma 4.6
Lemma 4.6. For any iteration index k ≥ 1 k\geq1 k ≥ 1 of the U-HLR call in Step 1, it holds that
M ~ k ≤ C ( τ t , β t − 1 ) (96) \widetilde{M}_k\leq C(\tau_t,\beta_{t-1}) \tag*{(96)} M k ≤ C ( τ t , β t − 1 ) (96)
where C ( τ t , β t − 1 ) C(\tau_t,\beta_{t-1}) C ( τ t , β t − 1 ) and M ~ k \widetilde{M}_k M k are as in (76) and (83) , respectively.
Proof. Let k ≥ 1 k\geq1 k ≥ 1 be an iteration index of the U-HLR call in Step 1. We claim that
D ~ k ≤ 3 2 τ t + 1 (97) \widetilde{D}_k\leq3\sqrt{2\tau_t}+1 \tag*{(97)} D k ≤ 3 2 τ t + 1 (97)
where τ t \tau_t τ t and D ~ k \widetilde{D}_k D k are as in (53) and (82) , respectively. Hence, consider U ∈ S ( U ~ k − 1 ; g t ) U\in\mathcal{S}(\widetilde{U}_{k-1};g_t) U ∈ S ( U k − 1 ; g t ) where S ( ⋅ ; ⋅ ) \mathcal{S}(\cdot;\cdot) S ( ⋅ ; ⋅ ) is as in (7) . It then follows from this inclusion, the definition of g t g_t g t in (81) , the last conclusion of Lemma 4.5(b), and the fact that U ~ 0 = Y t − 1 \widetilde{U}_0=Y_{t-1} U 0 = Y t − 1 in Step 1.a) that
ϕ t ( U U T ) ≤ ϕ t ( U ~ k − 1 U ~ k − 1 T ) ≤ ϕ t ( U ~ 0 U ~ 0 T ) = ϕ t ( Y t − 1 Y t − 1 T ) \phi_t(UU^{\mathsf{T}})\leq\phi_t(\widetilde{U}_{k-1}\widetilde{U}_{k-1}^{\mathsf{T}})\leq\phi_t(\widetilde{U}_0\widetilde{U}_0^{\mathsf{T}})=\phi_t(Y_{t-1}Y_{t-1}^{\mathsf{T}}) ϕ t ( U U T ) ≤ ϕ t ( U k − 1 U k − 1 T ) ≤ ϕ t ( U 0 U 0 T ) = ϕ t ( Y t − 1 Y t − 1 T )
which implies that U U T ∈ S ( Z t − 1 ; ϕ t ) UU^{\mathsf{T}}\in\mathcal{S}(Z_{t-1};\phi_t) U U T ∈ S ( Z t − 1 ; ϕ t ) with Z t − 1 : = Y t − 1 Y t − 1 T Z_{t-1}:=Y_{t-1}Y_{t-1}^{\mathsf{T}} Z t − 1 := Y t − 1 Y t − 1 T . It then follows from this inclusion and the definition of τ t \tau_t τ t in (53) that ∥ U ∥ 2 = tr ( U U T ) ≤ τ t / 2 \lVert U\rVert^2=\operatorname{tr}(UU^{\mathsf{T}})\leq\tau_t/2 ∥ U ∥ 2 = tr ( U U T ) ≤ τ t /2 . This bound together with the definition of D ~ k \widetilde{D}_k D k in (82) gives
D ~ k ≤ 6 τ t / 2 + 1 = 3 2 τ t + 1 \widetilde{D}_k\leq6\sqrt{\tau_t/2}+1=3\sqrt{2\tau_t}+1 D k ≤ 6 τ t /2 + 1 = 3 2 τ t + 1
which proves (97) . It then follows from (97) and the definitions of C ( τ t , β t − 1 ) C(\tau_t,\beta_{t-1}) C ( τ t , β t − 1 ) and M ~ k \widetilde{M}_k M k in (76) and (83) , respectively, that relation (96) must hold.
We are now ready to prove Proposition 4.2.
Proof of Proposition 4.2. (a) It follows from Lemma 4.5(a) that the U-HLR call in Step 1 is an instance of the RUFW framework applied to ϕ t \phi_t ϕ t with initial point X ~ 0 = Y t − 1 Y t − 1 T = Z t − 1 \widetilde{X}_0 = Y_{t-1}Y_{t-1}^{\mathsf{T}} = Z_{t-1} X 0 = Y t − 1 Y t − 1 T = Z t − 1 and tolerance pair ( ϵ , δ ) = ( ϵ d , ϵ c ) (\epsilon,\delta) = (\epsilon_d,\epsilon_c) ( ϵ , δ ) = ( ϵ d , ϵ c ) . Relations (20) and (24) imply that ϕ t \phi_t ϕ t satisfies RUFW assumption (A1) with L = β t − 1 ∥ A ∥ 2 L = \beta_{t-1}\lVert\mathcal{A}\rVert^2 L = β t − 1 ∥ A ∥ 2 , while Proposition A.1 implies that ϕ t \phi_t ϕ t satisfies RUFW assumption (A2). Moreover, the definitions of τ ˉ \bar{\tau} τ ˉ in (118) and τ t \tau_t τ t in (53) , together with (7) , imply that τ ˉ = τ t \bar{\tau} = \tau_t τ ˉ = τ t . It then follows from Proposition A.3(b) with L = β t − 1 ∥ A ∥ 2 L = \beta_{t-1}\lVert\mathcal{A}\rVert^2 L = β t − 1 ∥ A ∥ 2 and τ ˉ = τ t \bar{\tau} = \tau_t τ ˉ = τ t that the U-HLR call terminates in at most (77) iterations with an iterate X k = U k U k T X_k = U_kU_k^{\mathsf{T}} X k = U k U k T satisfying (111) and (112) with ϕ = ϕ t \phi= \phi_t ϕ = ϕ t and X ~ 0 = Y t − 1 Y t − 1 T \widetilde{X}_0 = Y_{t-1}Y_{t-1}^{\mathsf{T}} X 0 = Y t − 1 Y t − 1 T . Since Step 1.d) sets Y t = U k Y_t = U_k Y t = U k , and since S k = ∇ ϕ t ( U k U k T ) S_k = \nabla\phi_t(U_kU_k^{\mathsf{T}}) S k = ∇ ϕ t ( U k U k T ) and ⟨ S k U k , U k ⟩ = ⟨ S k , U k U k T ⟩ \langle S_kU_k,U_k\rangle= \langle S_k,U_kU_k^{\mathsf{T}}\rangle ⟨ S k U k , U k ⟩ = ⟨ S k , U k U k T ⟩ , relations (111) and (112) are precisely (78) and (79) . Statement (a) now follows from the fact that every iteration of the U-HLR call performs exactly one MEV computation in Step 1.c).
(b) Let k ≥ 1 k \ge1 k ≥ 1 be an iteration index of the U-HLR call in Step 1. It follows from the complexity bound in Lemma 4.4(b) and the bound on M ~ k \widetilde{M}_k M k in (96) that the ADAP-AIPP call in Step 1.b) of the k k k -th iteration of U-HLR performs at most
1 + C σ max { η 0 − 1 , C ( τ t , β t − 1 ) } ρ ˉ 2 [ ϕ t ( U ~ k − 1 U ~ k − 1 T ) − ϕ t ( U k U k T ) ] + [ log 2 + ( η 0 C ( τ t , β t − 1 ) ) ] (98) 1 + \frac{C_\sigma\max\{\eta_0^{-1},C(\tau_t,\beta_{t-1})\}}{\bar{\rho}^2}\left[\phi_t(\widetilde{U}_{k-1}\widetilde{U}_{k-1}^{\mathsf{T}})-\phi_t(U_kU_k^{\mathsf{T}})\right] + \left[\log_2^+(\eta_0C(\tau_t,\beta_{t-1}))\right] \tag*{(98)} 1 + ρ ˉ 2 C σ max { η 0 − 1 , C ( τ t , β t − 1 )} [ ϕ t ( U k − 1 U k − 1 T ) − ϕ t ( U k U k T ) ] + [ log 2 + ( η 0 C ( τ t , β t − 1 )) ] (98)
iterations/ADAP-FISTA calls.
Let k ^ \hat{k} k ^ denote the final iteration index of the U-HLR call. Relation (94) implies that the total decrease produced by its ADAP-AIPP calls is at most
ϕ t ( Y t − 1 Y t − 1 T ) − ϕ t ( Y t Y t T ) . \phi_t(Y_{t-1}Y_{t-1}^{\mathsf{T}})-\phi_t(Y_tY_t^{\mathsf{T}}). ϕ t ( Y t − 1 Y t − 1 T ) − ϕ t ( Y t Y t T ) .
Combining this bound with (98) , (77) , and the definition of ϕ t \phi_t ϕ t in (81) proves (b).
Numerical Results
In this section we report computational results for cuHALLaR+, together with comparisons with existing SDP solvers. The problem classes are maximum stable set, matrix completion, quadratic assignment problem (QAP), and general SDPs from the Mittelmann benchmark collection [43 ] . We first describe the implementations and the comparison criteria.
Implementation details The GPU implementation cuHALLaR+ uses C/CUDA. It is compiled for the NVIDIA H200 GPU architecture with compute capability 9.0 using CMake 3.30.2, GCC 12.3.0, and NVIDIA nvcc 12.6.68 (CUDA Toolkit 12.6.1), and linked against OpenBLAS 0.3.23. The GPU experiments use one NVIDIA H200 with 142 GB of GPU memory on a PACE node equipped with an Intel Xeon Platinum 8469C CPU.
For maximum stable set, cuHALLaR+ uses hybrid sparse low-rank (HSLR) input [1 ] . All cuHALLaR baselines are the on-the-fly results reported in [2 ] , using the settings reported there. These implementations, as well as the QAP implementations of cuHALLaR+ and HALLaR, evaluate C C C , A ( U U T ) \mathcal{A}(UU^{\mathsf{T}}) A ( U U T ) , and A ∗ ( p ) U \mathcal{A}^*(p)U A ∗ ( p ) U directly from the problem data without assembling the SDP coefficient matrices. cuLoRADS, CARDAL, and CSDP use SDPA input [64 ] . All solvers use this format for the Mittelmann collection.
Competing methods and comparison criteria The GPU experiments compare cuHALLaR+ with cuHALLaR, cuLoRADS v1.0.1 [23 ] and the single-GPU implementation of CARDAL v0.1.0 [37 ] .
Given positive tolerances ( ϵ p , ϵ g , ϵ d ) (\epsilon_p,\epsilon_g,\epsilon_d) ( ϵ p , ϵ g , ϵ d ) , cuHALLaR+ stops at ( X ˉ , p ˉ ) (\bar{X},\bar{p}) ( X ˉ , p ˉ ) , with X ˉ ⪰ 0 \bar{X} \succeq0 X ˉ ⪰ 0 and S ˉ : = C − A ∗ p ˉ \bar{S} := C-\mathcal{A}^*\bar{p} S ˉ := C − A ∗ p ˉ , when
∥ A X ˉ − b ∥ 2 1 + ∥ b ∥ 2 ≤ ϵ p , ∣ ⟨ C , X ˉ ⟩ − ⟨ b , p ˉ ⟩ ∣ 1 + ∣ ⟨ C , X ˉ ⟩ ∣ + ∣ ⟨ b , p ˉ ⟩ ∣ ≤ ϵ g , max { 0 , − λ ^ min ( S ˉ ) } 1 + ∣ ⟨ C , X ˉ ⟩ ∣ + ∣ ⟨ b , p ˉ ⟩ ∣ ≤ ϵ d , (99) \frac{\lVert\mathcal{A}\bar{X}-b\rVert_2}{1+\lVert b\rVert_2}\le\epsilon_p,\qquad
\frac{\left|\langle C,\bar{X}\rangle-\langle b,\bar{p}\rangle\right|}{1+\left|\langle C,\bar{X}\rangle\right|+\left|\langle b,\bar{p}\rangle\right|}\le\epsilon_g,\qquad
\frac{\max\{0,-\hat{\lambda}_{\min}(\bar{S})\}}{1+\left|\langle C,\bar{X}\rangle\right|+\left|\langle b,\bar{p}\rangle\right|}\le\epsilon_d, \tag*{(99)} 1 + ∥ b ∥ 2 ∥ A X ˉ − b ∥ 2 ≤ ϵ p , 1 + ⟨ C , X ˉ ⟩ + ∣ ⟨ b , p ˉ ⟩ ∣ ⟨ C , X ˉ ⟩ − ⟨ b , p ˉ ⟩ ≤ ϵ g , 1 + ⟨ C , X ˉ ⟩ + ∣ ⟨ b , p ˉ ⟩ ∣ max { 0 , − λ ^ m i n ( S ˉ )} ≤ ϵ d , (99)
where λ ^ min \hat{\lambda}_{\min} λ ^ m i n is the computed smallest eigenvalue. We set ϵ p = ϵ g = ϵ d = ϵ \epsilon_p=\epsilon_g=\epsilon_d=\epsilon ϵ p = ϵ g = ϵ d = ϵ . HALLaR and cuHALLaR use the trace-constrained stopping tests of [44 ] and [2 ] , respectively. The comparison target ϵ \epsilon ϵ is 10 − 4 10^{-4} 1 0 − 4 for QAP and 10 − 5 10^{-5} 1 0 − 5 for every other problem class. Solver-specific stopping tests and phase settings are retained. The time limit is one hour for the Mittelmann SDPs, six hours for the QAPLIB instances, and eight hours for the random QAP instances.
Times are in seconds, and an entry time/rank also reports the final rank. The cuLoRADS v1.0.1 times include its dual-infeasibility evaluations, which cuLoRADS performs on the CPU after each phase and reports separately from its solver time, whereas the Mittelmann benchmark site5 reports only the solver time. The dual residual is part of the stopping test (97), and the cuHALLaR+ times include every evaluation of it. Therefore, we include the corresponding cuLoRADS dual evaluation time within its final solve time. We use TL to denote an instance that hit the time-limit and ** to denote a crash or an instance too large for the solver. An entry */N denotes an instance with reported maximum relative error N N N .
Maximum stable set
Given a graph G = ( [ n ] , E ) G = ([n], E) G = ([ n ] , E ) , the maximum stable set problem consists of finding a subset of vertices of largest cardinality such that no two vertices are connected by an edge. Lovász [39 ] introduced a constant, the ϑ \vartheta ϑ -function, which upper bounds the value of the maximum stable set. The ϑ \vartheta ϑ -function is the value of the SDP
max { e e T ∙ X : X i j = 0 , i j ∈ E , tr X = 1 , X ⪰ 0 , X ∈ S n ( R ) } \max\ \{ee^{\mathsf{T}} \mathbin{\bullet} X : X_{ij} = 0,\ ij \in E,\ \operatorname{tr} X = 1,\ X \succeq0,\ X \in\mathcal{S}^{n}(\mathbb{R})\} max { e e T ∙ X : X ij = 0 , ij ∈ E , tr X = 1 , X ⪰ 0 , X ∈ S n ( R )}
where e = ( 1 , 1 , … , 1 ) ∈ R n e = (1, 1, \ldots, 1) \in\mathbb{R}^{n} e = ( 1 , 1 , … , 1 ) ∈ R n is the all ones vector. It was shown in [21 ] that the ϑ \vartheta ϑ -function agrees exactly with the stable set number for perfect graphs.
Table 1 compares cuHALLaR+ with cuHALLaR from [2 ] , cuLoRADS, and CARDAL on the Hamming instances. Both cuHALLaR+ and cuHALLaR solve all 14 Hamming instances and return rank-two solutions throughout. cuHALLaR+ is faster on 12 instances; cuHALLaR is faster on H 18 , 2 H_{18,2} H 18 , 2 and H 19 , 2 H_{19,2} H 19 , 2 . cuLoRADS and CARDAL report solves on the first five instances, and cuHALLaR+ is faster than both on each of these. CARDAL is faster than cuLoRADS only on H 10 , 2 H_{10,2} H 10 , 2 . On the largest graph, with over eight million vertices, cuHALLaR+ takes 44.13 seconds, compared with 83.905 seconds for cuHALLaR.
Table 1. Runtimes for Hamming stable-set SDPs.
Table 2 reports the GSET results. The cuHALLaR runs from [2 ] use trace bound τ = 1 \tau= 1 τ = 1 . Ratios are shown only for cuLoRADS runs that pass the accuracy check.
Table 2. Runtimes for GSET stable-set SDPs.
Both cuHALLaR+ and cuHALLaR solve all 28 GSET instances, compared with 26 for CARDAL and 19 for cuLoRADS. cuHALLaR+ is faster than cuHALLaR on 24 instances. On the largest graph, G81, cuHALLaR+ returns a rank-two solution in 48.91 seconds, compared with a rank-three solution in 64.20 seconds for cuHALLaR.
Matrix completion
Consider the problem of retrieving a low rank matrix M ∈ R n 1 × n 2 M \in\mathbb{R}^{n_1 \times n_2} M ∈ R n 1 × n 2 , where n 1 ≤ n 2 n_1 \le n_2 n 1 ≤ n 2 , by observing a subset of its entries: M i j M_{ij} M ij , i j ∈ I ij \in\mathcal{I} ij ∈ I . A standard approach to tackle this problem is by considering the nuclear norm relaxation:
min Y { ∥ Y ∥ ∗ : Y i j = M i j , ∀ i j ∈ I , Y ∈ R n 1 × n 2 } \min_Y \left\{ \lVert Y \rVert_* \ : \ Y_{ij} = M_{ij},\ \forall ij \in\mathcal{I},\quad Y \in\mathbb{R}^{n_1 \times n_2} \right\} Y min { ∥ Y ∥ ∗ : Y ij = M ij , ∀ ij ∈ I , Y ∈ R n 1 × n 2 }
The above problem can be rephrased as the following SDP:
min X { 1 2 tr ( X ) : X = ( W 1 Y Y T W 2 ) ⪰ 0 , Y i , j = M i , j ∀ i j ∈ I , X ∈ S n 1 + n 2 ( R ) } \min_X \left\{ \frac{1}{2}\operatorname{tr}(X) \ : \ X = \begin{pmatrix} W_1 & Y \\ Y^{\mathsf{T}} & W_2 \end{pmatrix} \succeq0,\quad Y_{i,j} = M_{i,j}\ \forall ij \in\mathcal{I},\quad X \in\mathbb{S}^{n_1+n_2}(\mathbb{R}) \right\} X min { 2 1 tr ( X ) : X = ( W 1 Y T Y W 2 ) ⪰ 0 , Y i , j = M i , j ∀ ij ∈ I , X ∈ S n 1 + n 2 ( R ) }
The nuclear norm relaxation was introduced in [19 ] . It was shown in [11 ] it provably completes the matrix when m = ∣ I ∣ m = \lvert\mathcal{I}\rvert m = ∣ I ∣ is sufficiently large and the indices of the observations are independent and uniform.
Matrix completion instances are generated randomly using the protocol in [2 ] : given r ≤ n 1 ≤ n 2 r \le n_1 \le n_2 r ≤ n 1 ≤ n 2 , the hidden solution matrix is M = M 1 M 2 T M = M_1M_2^{\mathsf{T}} M = M 1 M 2 T , where M 1 ∈ R n 1 × r M_1 \in\mathbb{R}^{n_1 \times r} M 1 ∈ R n 1 × r and M 2 ∈ R n 2 × r M_2 \in\mathbb{R}^{n_2 \times r} M 2 ∈ R n 2 × r have independent standard Gaussian entries. Then m = ⌈ r 2 ( n 1 + n 2 ) log ( n 1 + n 2 ) ⌉ m = \lceil r^2(n_1+n_2)\log(n_1+n_2)\rceil m = ⌈ r 2 ( n 1 + n 2 ) log ( n 1 + n 2 )⌉ entries are sampled uniformly at random; the oversampling ratio is r log ( n 1 + n 2 ) r\log(n_1+n_2) r log ( n 1 + n 2 ) .
In Table 3 , both cuHALLaR+ and cuHALLaR solve all 26 instances, while cuLoRADS and CARDAL solve 25 and 18, respectively. cuHALLaR+ is faster than cuHALLaR on all 26 instances and faster than cuLoRADS and CARDAL on every instance for which they report a solve. The final ranks of both cuHALLaR+ and cuHALLaR agree with the planted rank on 24 instances and exceed it by one on the remaining two. On the largest instance, with over 261 million observations, cuHALLaR+ takes 30.82 seconds, compared with 53.3 seconds for cuHALLaR and 4,124.44 seconds for cuLoRADS.
Table 3. Runtimes for matrix-completion SDPs. The cuHALLaR results are reproduced from [2 Table 2] , using tolerance 10 − 5 10^{-5} 1 0 − 5 and a three-hour time limit.
Quadratic assignment problem
Given matrices A , B ∈ S ν A, B \in\mathbb{S}^{\nu} A , B ∈ S ν , the quadratic assignment problem (QAP) consists of minimizing tr ( A P B P T ) \operatorname{tr}(APBP^{\mathsf{T}}) tr ( A P B P T ) over the ν × ν \nu\times\nu ν × ν permutation matrices P P P . With column-wise vectorization, the identity
tr ( A P B P T ) = vec ( P ) T ( B ⊗ A ) vec ( P ) \operatorname{tr}(APBP^{\mathsf{T}}) = \operatorname{vec}(P)^{\mathsf{T}}(B \otimes A)\operatorname{vec}(P) tr ( A P B P T ) = vec ( P ) T ( B ⊗ A ) vec ( P )
motivates the following SDP relaxation:
min X { ( B ⊗ A ) ∙ X : tr 1 X = I ν , tr 2 X = I ν , e ν 2 e ν 2 T ∙ X = ν 2 , X ⪰ 0 , X ∈ S ν 2 ( R ) } . (100) \min_{X}\left\{(B\otimes A)\bullet X:\quad\operatorname{tr}_1 X=I_{\nu},\quad\operatorname{tr}_2 X=I_{\nu},\quad e_{\nu^2}e_{\nu^2}^{\mathsf{T}}\bullet X=\nu^2,\quad X\succeq0,\quad X\in\mathbb{S}^{\nu^2}(\mathbb{R})\right\}. \tag*{(100)} X min { ( B ⊗ A ) ∙ X : tr 1 X = I ν , tr 2 X = I ν , e ν 2 e ν 2 T ∙ X = ν 2 , X ⪰ 0 , X ∈ S ν 2 ( R ) } . (100)
Here e ν 2 ∈ R ν 2 e_{\nu^2}\in\mathbb{R}^{\nu^2} e ν 2 ∈ R ν 2 is the all-ones vector. The index ( i , j ) (i,j) ( i , j ) denotes the entry i + ( j − 1 ) ν i+(j-1)\nu i + ( j − 1 ) ν of a vectorized ν × ν \nu\times\nu ν × ν matrix, and the partial traces are defined by
[ tr 1 X ] i j = ∑ ℓ = 1 ν X ( i , ℓ ) , ( j , ℓ ) , [ tr 2 X ] i j = ∑ ℓ = 1 ν X ( ℓ , i ) , ( ℓ , j ) . [\operatorname{tr}_1 X]_{ij}=\sum_{\ell=1}^{\nu}X_{(i,\ell),(j,\ell)},\qquad[\operatorname{tr}_2 X]_{ij}=\sum_{\ell=1}^{\nu}X_{(\ell,i),(\ell,j)}. [ tr 1 X ] ij = ℓ = 1 ∑ ν X ( i , ℓ ) , ( j , ℓ ) , [ tr 2 X ] ij = ℓ = 1 ∑ ν X ( ℓ , i ) , ( ℓ , j ) .
For every permutation matrix P P P , the rank-one matrix X = vec ( P ) vec ( P ) T X=\operatorname{vec}(P)\operatorname{vec}(P)^{\mathsf{T}} X = vec ( P ) vec ( P ) T is feasible for (100) and has the same objective value. Thus, (100) is a relaxation of QAP and is equivalent to that in [44 Section 4.6] , up to the ordering of the Kronecker factors. Its matrix dimension is n = ν 2 n=\nu^2 n = ν 2 . For ν ≥ 2 \nu\ge2 ν ≥ 2 , we omit the redundant final diagonal equation of tr 2 X = I ν \operatorname{tr}_2X=I_{\nu} tr 2 X = I ν , leaving m = ν ( ν + 1 ) m=\nu(\nu+1) m = ν ( ν + 1 ) independent scalar constraints. The partial-trace constraints imply tr X = ν \operatorname{tr}X=\nu tr X = ν , but HALLaR+ does not require this trace bound as an input. HALLaR uses the trace bound τ = ν \tau=\nu τ = ν .
For nonsymmetric QAPLIB data, the objective matrix is replaced by 0.5 ( B ⊗ A + B T ⊗ A T ) 0.5(B\otimes A+B^{\mathsf{T}}\otimes A^{\mathsf{T}}) 0.5 ( B ⊗ A + B T ⊗ A T ) . This symmetrization preserves the quadratic form and leaves the constraints in (100) unchanged.
Table 4 covers 32 QAPLIB instances, including the 11 tai instances in [44 Table 10] . All methods use a six-hour time limit.
Table 4. QAPLIB SDP results.
cuHALLaR+ starts from U 0 = e ν 2 / ν U_0=e_{\nu^2}/\sqrt{\nu} U 0 = e ν 2 / ν and the zero multiplier. The objective is scaled by 1 / ( ∥ A ∥ F ∥ B ∥ F ) 1/(\lVert A\rVert_F\lVert B\rVert_F) 1/ (∥ A ∥ F ∥ B ∥ F ) , and the same scaling is used in the stopping test (99) . For all instances after the first 11, CSDP receives the independent equality constraints and the redundant constraint tr X + s = ν \operatorname{tr}X+s=\nu tr X + s = ν , s ≥ 0 s\ge0 s ≥ 0 .
CSDP reaches the target accuracy on 23 of the 32 instances, including the nine smallest tai instances. HALLaR reaches the target on seven instances. The final errors reported by cuLoRADS exceed the target on all 32 instances.
In Table 5 , CSDP reaches the target on 15 of 18 instances, while HALLaR does so on four. cuLoRADS finishes above the target on all 18 instances. On the two largest instances, CSDP takes about 10,400 seconds on esc64a and has no completed result for esc128.
Table 5. Remaining QAPLIB instances of [44 Table 11] , part 1 of 2.
CSDP reaches the target on 18 of the 31 instances in Table 6, while cuLoRADS does so only on sko64. CSDP misses the accuracy target on all six sko100 instances. On tho150, the largest instance in this group, cuHALLaR+ reaches the time limit.
Table 7 reports results at the 11 random QAP dimensions in [44 Table 12] . For our runs, A A A and B B B are symmetric with zero diagonal; each strict upper-triangular entry is obtained by rounding an independent uniform draw on [ 0 , 10 ] [0,10] [ 0 , 10 ] . We generate one instance per size using Julia 1.10.1 and seed 0, and solve (100) with a tolerance of 10 − 4 10^{-4} 1 0 − 4 and an eight-hour solver limit. HALLaR and cuHALLaR+ use the same Gaussian unit-norm initial factor.
Table 7. Random QAP SDP results.
HALLaR, CSDP, and cuLoRADS record no solves at the target accuracy on the random instances. At ν = 400 \nu=400 ν = 400 , the SDP has dimension 160,000 and the reported HALLaR error is 2.5 × 10 − 3 2.5\times10^{-3} 2.5 × 1 0 − 3 . For cuLoRADS, the ν = 200 \nu=200 ν = 200 instance with n = 40 , 000 n=40,000 n = 40 , 000 exhausts GPU memory on the H200 during solver initialization.
General SDP benchmark
Table 8 reports results for the same 27 Mittelmann SDP instances [43 ] considered in [23 Table 7] . CARDAL solves 14 of the 27 instances and reaches the one-hour limit on the remaining 13. cuLoRADS has reported runtimes on 15 instances.
Table 8. Runtimes for Mittelmann SDPs.
Concluding remarks
This paper develops HALLar+, a low-rank inexact augmented Lagrangian method for SDPs with possibly unbounded feasible regions. A key practical advantage is that the method does not require a user-specified trace bound τ \tau τ . Choosing such a bound can be difficult: a value that is too small may exclude all optimal solutions, whereas an overly conservative value may substantially increase the computational effort. Under the stated assumptions, we establish a uniform bound on the diameters of the relevant augmented Lagrangian sublevel sets and use it to derive iteration-complexity guarantees for obtaining approximate primal–dual solutions.
The numerical experiments demonstrate strong computational performance on several classes of SDPs, with substantial speedups over existing solvers on many instances. On the Mittelmann benchmarks, cuHALLaR+ achieves speedups of up to approximately 4.09 × 4.09\times 4.09 × over cuLoRADS and 112.63 × 112.63\times 112.63 × over CARDAL among instances successfully solved by the respective methods. It also solves the largest random QAP relaxation tested, with matrix dimension 160,000, in approximately 68 minutes.
Future work includes extending HALLar+ to SDPs with inequality constraints and multiple positive semidefinite blocks, as well as to doubly nonnegative programs. For the latter, an important challenge is to handle the entrywise nonnegativity constraints while preserving the memory advantages of the low-rank factorization.
X-Space and Convex-Analysis Machinery
This appendix collects the technical convex-analysis facts and describes the relaxed unbounded Frank–Wolfe method used in our X X X -space analysis.
Technical Convex-Analysis Facts
This subsection collects the convex-analysis facts used in the X-AL analysis of § 3 . The next proposition establishes the boundedness property used for the AL sublevel sets in the X-AL analysis.
Proposition A.1. Assume that Assumption 2.1 holds. For every q ∈ R m q \in\mathbb{R}^{m} q ∈ R m , β > 0 \beta> 0 β > 0 , and κ ∈ R \kappa\in\mathbb{R} κ ∈ R , the set
{ X ∈ S + n : L β ( X ; q ) ≤ κ } (101) \left\{X \in\mathbb{S}_{+}^{n} : \mathcal{L}_{\beta}(X;q) \le\kappa\right\} \tag*{(101)} { X ∈ S + n : L β ( X ; q ) ≤ κ } (101) is bounded.
Proof. The proof can be found in [49 Theorem 8.4, p. 64] (see also [26 Proposition A.2.2.3, p. 40] ).
For every τ > 0 \tau> 0 τ > 0 , let
Δ τ : = { X ∈ S + n : tr ( X ) ≤ τ } . (102) \Delta_{\tau} := \left\{X \in\mathbb{S}_{+}^{n} : \operatorname{tr}(X) \le\tau\right\}. \tag*{(102)} Δ τ := { X ∈ S + n : tr ( X ) ≤ τ } . (102)
For every G ∈ S n G \in\mathbb{S}^{n} G ∈ S n , the spectral decomposition of G G G gives
min X ∈ Δ τ ⟨ G , X ⟩ = τ min { λ min ( G ) , 0 } . (103) \min_{X \in\Delta_{\tau}} \langle G,X\rangle= \tau\min\left\{\lambda_{\min}(G),0\right\}. \tag*{(103)} X ∈ Δ τ min ⟨ G , X ⟩ = τ min { λ m i n ( G ) , 0 } . (103)
Proposition A.2. Let τ > 0 \tau> 0 τ > 0 , X ∈ S + n X \in\mathbb{S}_{+}^{n} X ∈ S + n , and G ∈ S n G \in\mathbb{S}^{n} G ∈ S n , and define
ϵ τ ( X ; G ) : = max Y ∈ Δ τ ⟨ G , X − Y ⟩ . (104) \epsilon_{\tau}(X;G) := \max_{Y \in\Delta_{\tau}} \langle G,X-Y\rangle. \tag*{(104)} ϵ τ ( X ; G ) := Y ∈ Δ τ max ⟨ G , X − Y ⟩ . (104) Then the following statements hold:
(a) We have
ϵ τ ( X ; G ) ≤ ∣ ⟨ G , X ⟩ ∣ + τ max { λ max ( − G ) , 0 } . (105) \epsilon_{\tau}(X;G) \le\left|\langle G,X\rangle\right|+\tau\max\left\{\lambda_{\max}(-G),0\right\}. \tag*{(105)} ϵ τ ( X ; G ) ≤ ∣ ⟨ G , X ⟩ ∣ + τ max { λ m a x ( − G ) , 0 } . (105) (b) If X ∈ Δ τ / 2 X \in\Delta_{\tau/2} X ∈ Δ τ /2 , then
3 ϵ τ ( X ; G ) ≥ ∣ ⟨ G , X ⟩ ∣ + τ max { λ max ( − G ) , 0 } . (106) 3\epsilon_{\tau}(X;G) \ge\left|\langle G,X\rangle\right|+\tau\max\left\{\lambda_{\max}(-G),0\right\}. \tag*{(106)} 3 ϵ τ ( X ; G ) ≥ ∣ ⟨ G , X ⟩ ∣ + τ max { λ m a x ( − G ) , 0 } . (106)
Proof. a) Relations (101) and (102) imply that
ϵ τ ( X ; G ) = ⟨ G , X ⟩ − τ min { λ min ( G ) , 0 } = ⟨ G , X ⟩ + τ max { λ max ( − G ) , 0 } . (107) \epsilon_{\tau}(X;G)=\langle G,X\rangle-\tau\min\left\{\lambda_{\min}(G),0\right\}=\langle G,X\rangle+\tau\max\left\{\lambda_{\max}(-G),0\right\}. \tag*{(107)} ϵ τ ( X ; G ) = ⟨ G , X ⟩ − τ min { λ m i n ( G ) , 0 } = ⟨ G , X ⟩ + τ max { λ m a x ( − G ) , 0 } . (107)
and hence that (103) holds.
b) Let X ∈ Δ τ / 2 X \in\Delta_{\tau/2} X ∈ Δ τ /2 be given. Since 0 0 0 and 2 X 2X 2 X are both in Δ τ \Delta_{\tau} Δ τ due to the definition of Δ τ \Delta_{\tau} Δ τ in (100), it follows from (102) with Y = 0 Y=0 Y = 0 and Y = 2 X Y=2X Y = 2 X that
ϵ τ ( X ; G ) ≥ max { ⟨ G , X − 0 ⟩ , ⟨ G , X − 2 X ⟩ } = ∣ ⟨ G , X ⟩ ∣ . (108) \epsilon_{\tau}(X;G) \ge\max\left\{\langle G,X-0\rangle,\langle G,X-2X\rangle\right\}=\left|\langle G,X\rangle\right|. \tag*{(108)} ϵ τ ( X ; G ) ≥ max { ⟨ G , X − 0 ⟩ , ⟨ G , X − 2 X ⟩ } = ∣ ⟨ G , X ⟩ ∣ . (108)
Combining (105) and (106), we obtain
τ max { λ max ( − G ) , 0 } = ϵ τ ( X ; G ) − ⟨ G , X ⟩ ≤ ϵ τ ( X ; G ) + ∣ ⟨ G , X ⟩ ∣ ≤ 2 ϵ τ ( X ; G ) . \tau\max\left\{\lambda_{\max}(-G),0\right\}=\epsilon_{\tau}(X;G)-\langle G,X\rangle\le\epsilon_{\tau}(X;G)+\left|\langle G,X\rangle\right|\le2\epsilon_{\tau}(X;G). τ max { λ m a x ( − G ) , 0 } = ϵ τ ( X ; G ) − ⟨ G , X ⟩ ≤ ϵ τ ( X ; G ) + ∣ ⟨ G , X ⟩ ∣ ≤ 2 ϵ τ ( X ; G ) .
Adding this inequality to (106) proves (104).
X-Space Relaxed Unbounded Frank–Wolfe Material
Given an initial point X ~ 0 ∈ S + n \widetilde{X}_{0} \in\mathbb{S}_{+}^{n} X 0 ∈ S + n , this subsection presents the Relaxed Unbounded Frank-Wolfe (RUFW) framework for solving
ϕ ∗ : = min { ϕ ( X ) : X ∈ S + n } , (109) \phi_{*}:=\min\left\{\phi(X):X\in\mathbb{S}_{+}^{n}\right\}, \tag*{(109)} ϕ ∗ := min { ϕ ( X ) : X ∈ S + n } , (109)
where the function ϕ : S + n → R \phi:\mathbb{S}_{+}^{n}\to\mathbb{R} ϕ : S + n → R is assumed to satisfy the following conditions:
(A1) ϕ \phi ϕ is convex and differentiable on S + n \mathbb{S}_{+}^{n} S + n , and its gradient is L L L -Lipschitz continuous for some L > 0 L>0 L > 0 , i.e.,
∥ ∇ ϕ ( X ′ ) − ∇ ϕ ( X ) ∥ ≤ L ∥ X ′ − X ∥ ∀ X , X ′ ∈ S + n ; \left\|\nabla\phi(X')-\nabla\phi(X)\right\|\le L\left\|X'-X\right\|\qquad\forall X,X'\in\mathbb{S}_{+}^{n}; ∥ ∇ ϕ ( X ′ ) − ∇ ϕ ( X ) ∥ ≤ L ∥ X ′ − X ∥ ∀ X , X ′ ∈ S + n ;
(A2) the sublevel set S ( X ~ 0 ; ϕ ) S(\widetilde{X}_0;\phi) S ( X 0 ; ϕ ) is bounded, where S ( ⋅ ; ⋅ ) S(\cdot;\cdot) S ( ⋅ ; ⋅ ) is defined as in (5) with Ω = S + n \Omega= \mathbb{S}_{+}^{n} Ω = S + n .
Assumption (A1) implies that
ϕ ( X ′ ) ≤ ϕ ( X ) + ⟨ ∇ ϕ ( X ) , X ′ − X ⟩ + L 2 ∥ X ′ − X ∥ 2 ∀ X , X ′ ∈ S + n . (110) \phi(X') \leq\phi(X) + \langle\nabla\phi(X),X'-X\rangle+ \frac{L}{2}\lVert X'-X\rVert^{2} \qquad\forall\ X,X'\in\mathbb{S}_{+}^{n}. \tag*{(110)} ϕ ( X ′ ) ≤ ϕ ( X ) + ⟨ ∇ ϕ ( X ) , X ′ − X ⟩ + 2 L ∥ X ′ − X ∥ 2 ∀ X , X ′ ∈ S + n . (110)
Here, the inner product is the Frobenius one and the norm is the one induced by the Frobenius inner product, i.e., ⟨ U , V ⟩ = tr ( U T V ) \langle U,V\rangle= \operatorname{tr}(U^{T}V) ⟨ U , V ⟩ = tr ( U T V ) and ∥ U ∥ 2 = tr ( U T U ) \lVert U\rVert^{2} = \operatorname{tr}(U^{T}U) ∥ U ∥ 2 = tr ( U T U ) .
Under (A1) -(A2) , it is also easy to see the minimum in (107) is attained since (A1) –(A2) imply that ϕ \phi ϕ is a continuous coercive function. Likewise, these assumptions imply that step 4 in the RUFW framework below is well-defined. We now describe the type of solution that RUFW aims to find.
Problem: Given a function ϕ \phi ϕ and initial point X ~ 0 ∈ S + n \widetilde{X}_0\in\mathbb{S}_{+}^{n} X 0 ∈ S + n satisfying (A1) –(A2) and a tolerance pair ( ϵ , δ ) ∈ R + + 2 (\epsilon,\delta)\in\mathbb{R}_{++}^{2} ( ϵ , δ ) ∈ R ++ 2 , RUFW aims to find a point X ‾ ∈ S + n \overline{X}\in\mathbb{S}_{+}^{n} X ∈ S + n satisfying
λ min ( ∇ ϕ ( X ‾ ) ) ≥ − ϵ , ∣ ⟨ ∇ ϕ ( X ‾ ) , X ‾ ⟩ ∣ ≤ δ ; (111) \lambda_{\min}\left(\nabla\phi(\overline{X})\right)\geq-\epsilon,\qquad\left\lvert\left\langle\nabla\phi(\overline{X}),\overline{X}\right\rangle\right\rvert\leq\delta; \tag*{(111)} λ m i n ( ∇ ϕ ( X ) ) ≥ − ϵ , ⟨ ∇ ϕ ( X ) , X ⟩ ≤ δ ; (111)
ϕ ( X ‾ ) ≤ ϕ ( X ~ 0 ) . (112) \phi(\overline{X})\leq\phi(\widetilde{X}_0). \tag*{(112)} ϕ ( X ) ≤ ϕ ( X 0 ) . (112)
The formal description of the relaxed unbounded Frank–Wolfe (RUFW) framework is given below.
Relaxed Unbounded Frank–Wolfe (RUFW) Framework
Input: tolerance pair ( ϵ , δ ) ∈ R + + 2 (\epsilon,\delta)\in\mathbb{R}_{++}^{2} ( ϵ , δ ) ∈ R ++ 2 , function ϕ : S + n → R \phi:\mathbb{S}_{+}^{n}\to\mathbb{R} ϕ : S + n → R , and initial point X ~ 0 ∈ S + n \widetilde{X}_0\in\mathbb{S}_{+}^{n} X 0 ∈ S + n .
Step 0. Set k ← 1 k\leftarrow1 k ← 1 .
Step 1. Choose X k ∈ S + n X_k\in\mathbb{S}_{+}^{n} X k ∈ S + n such that
ϕ ( X k ) ≤ ϕ ( X ~ k − 1 ) . (113) \phi(X_k)\leq\phi(\widetilde{X}_{k-1}). \tag*{(113)} ϕ ( X k ) ≤ ϕ ( X k − 1 ) . (113)
Step 2. Set G k ← ∇ ϕ ( X k ) G_k\leftarrow\nabla\phi(X_k) G k ← ∇ ϕ ( X k ) . Compute a minimum eigenpair ( λ k min ( G k ) , v k min ( G k ) ) (\lambda_k^{\min}(G_k),v_k^{\min}(G_k)) ( λ k m i n ( G k ) , v k m i n ( G k )) of G k G_k G k with ∥ v k min ( G k ) ∥ = 1 \lVert v_k^{\min}(G_k)\rVert=1 ∥ v k m i n ( G k )∥ = 1 , and set
y k F : = { v k min ( G k ) , if λ k min ( G k ) < 0 , 0 , if λ k min ( G k ) ≥ 0 , X k F : = y k F ( y k F ) T ; (114) y_k^{F}:=
\begin{cases}
v_k^{\min}(G_k), & \text{if }\lambda_k^{\min}(G_k)<0,\\
0, & \text{if }\lambda_k^{\min}(G_k)\geq0,
\end{cases}
\qquad
X_k^{F}:=y_k^{F}(y_k^{F})^{\mathsf{T}}; \tag*{(114)} y k F := { v k m i n ( G k ) , 0 , if λ k m i n ( G k ) < 0 , if λ k m i n ( G k ) ≥ 0 , X k F := y k F ( y k F ) T ; (114)
Step 3. If
λ k min ( G k ) ≥ − ϵ , ∣ ⟨ G k , X k ⟩ ∣ ≤ δ , (115) \lambda_k^{\min}(G_k)\geq-\epsilon,\qquad\lvert\langle G_k,X_k\rangle\rvert\leq\delta, \tag*{(115)} λ k m i n ( G k ) ≥ − ϵ , ∣⟨ G k , X k ⟩∣ ≤ δ , (115)
then set X ‾ ← X k \overline{X}\leftarrow X_k X ← X k , output X ‾ \overline{X} X , and stop .
Step 4. Compute
( α k , ω k ) ∈ arg min { ϕ ( α X k + ω X k F ) : α ≥ 0 , ω ≥ 0 } (116) (\alpha_k,\omega_k)\in\mathop{\arg\min}\{\phi(\alpha X_k+\omega X_k^{F}):\alpha\geq0,\ \omega\geq0\} \tag*{(116)} ( α k , ω k ) ∈ arg min { ϕ ( α X k + ω X k F ) : α ≥ 0 , ω ≥ 0 } (116)
and set
X ~ k : = α k X k + ω k X k F ; (117) \widetilde{X}_k:=\alpha_kX_k+\omega_kX_k^{F}; \tag*{(117)} X k := α k X k + ω k X k F ; (117)
Step 5. Set k ← k + 1 k\leftarrow k+1 k ← k + 1 and return to Step 1.
Step 1 allows any choice of X k ∈ S + n X_k\in\mathbb{S}_{+}^{n} X k ∈ S + n whose objective value does not exceed that of the current iterate X ~ k − 1 \widetilde{X}_{k-1} X k − 1 . Step 2 evaluates the gradient G k = ∇ ϕ ( X k ) G_k=\nabla\phi(X_k) G k = ∇ ϕ ( X k ) and computes a minimum eigenpair of G k G_k G k to construct X k F X_k^{F} X k F and check the stopping conditions (113) in Step 3. If these conditions are not satisfied, Step 4 determines the coefficients ( α k , ω k ) (\alpha_k,\omega_k) ( α k , ω k ) by solving the two-dimensional optimization problem (114), following an approach similar to that of Laue [35 ] . The next iterate X ~ k \widetilde{X}_k X k is then formed as the conic combination of X k X_k X k and X k F X_k^{F} X k F given in (115). Since ( α , ω ) = ( 1 , 0 ) (\alpha,\omega)=(1,0) ( α , ω ) = ( 1 , 0 ) is feasible in (114), Step 4 does not increase the objective value. Together with (111), this ensures that ϕ ( X k ) ≤ ϕ ( X ~ 0 ) \phi(X_k)\leq\phi(\widetilde{X}_0) ϕ ( X k ) ≤ ϕ ( X 0 ) for every k k k . Thus, whenever RUFW terminates in Step 3, its output X ‾ \overline{X} X satisfies both (109) and (110).
The following proposition states the main convergence result of the RUFW framework. Its proof is given in the following subsubsection. In contrast to classical Frank–Wolfe complexity bounds, the bound in (117) below does not depend on the initial objective gap ϕ ( X ~ 0 ) − ϕ ∗ \phi(\widetilde{X}_0)-\phi_* ϕ ( X 0 ) − ϕ ∗ . The initial point X ~ 0 \widetilde{X}_0 X 0 enters the bound only through the radius r ˉ \bar{r} r ˉ of its sublevel set.
Proposition A.3. Assume that ( A 1 ) (A1) ( A 1 ) –( A 2 ) (A2) ( A 2 ) hold and define
τ ˉ ≔ max { 1 , 2 sup { tr ( X ) : X ∈ S + n , ϕ ( X ) ≤ ϕ ( X ~ 0 ) } } , ϵ k ( τ ˉ ) ≔ max X ∈ Δ τ ˉ ⟨ G k , X k − X ⟩ . (118) \bar{\tau} \coloneqq\max\left\{1, 2\sup\left\{\operatorname{tr}(X): X \in\mathbb{S}_{+}^{n},\ \phi(X) \le\phi(\widetilde{X}_{0})\right\}\right\}, \qquad\epsilon_{k}(\bar{\tau}) \coloneqq\max_{X\in\Delta_{\bar{\tau}}} \langle G_{k}, X_{k}-X\rangle. \tag*{(118)} τ ˉ : = max { 1 , 2 sup { tr ( X ) : X ∈ S + n , ϕ ( X ) ≤ ϕ ( X 0 ) } } , ϵ k ( τ ˉ ) : = X ∈ Δ τ ˉ max ⟨ G k , X k − X ⟩ . (118) Then, the following statements hold for RUFW:
(a) for every iteration index k ≥ 1 k \ge1 k ≥ 1 , it holds that ϕ ( X k ) ≤ ϕ ( X ~ 0 ) \phi(X_{k}) \le\phi(\widetilde{X}_{0}) ϕ ( X k ) ≤ ϕ ( X 0 ) ;
(b) RUFW terminates with a point X ‾ \overline{X} X satisfying (109) and (110) in at most
⌈ 2 + 12 τ ˉ 2 L ( 1 δ + 2 ϵ ) ⌉ (119) \left\lceil2+12\bar{\tau}^{2}L\left(\frac{1}{\delta}+\frac{2}{\epsilon}\right)\right\rceil \tag*{(119)} ⌈ 2 + 12 τ ˉ 2 L ( δ 1 + ϵ 2 ) ⌉ (119) iterations.
Proof of Proposition A.3
This subsubsection proves Proposition A.3. Recall the trace-bounded spectrahedron Δ τ \Delta_{\tau} Δ τ and its linear-oracle value from (100) and (101), respectively. Let X ∗ X_{*} X ∗ be a minimizer of (107). Since ϕ ( X ∗ ) ≤ ϕ ( X ~ 0 ) \phi(X_{*}) \le\phi(\widetilde{X}_{0}) ϕ ( X ∗ ) ≤ ϕ ( X 0 ) , it follows from (A2) and (116) that X ∗ ∈ Δ τ ˉ / 2 ⊆ Δ τ ˉ X_{*} \in\Delta_{\bar{\tau}/2} \subseteq\Delta_{\bar{\tau}} X ∗ ∈ Δ τ ˉ /2 ⊆ Δ τ ˉ . We now begin with a lemma describing key properties of the RUFW iterates.
Lemma A.4. For every iteration index k ≥ 1 k \ge1 k ≥ 1 , the following statements hold:
(a) it holds
ϕ ( X ~ k ) ≤ ϕ ( X k ) ≤ ϕ ( X ~ k − 1 ) , tr ( X k ) ≤ τ ˉ / 2 , tr ( X ~ k ) ≤ τ ˉ / 2 ; (120) \phi(\widetilde{X}_{k}) \le\phi(X_{k}) \le\phi(\widetilde{X}_{k-1}), \qquad\operatorname{tr}(X_{k}) \le\bar{\tau}/2,\qquad\operatorname{tr}(\widetilde{X}_{k}) \le\bar{\tau}/2; \tag*{(120)} ϕ ( X k ) ≤ ϕ ( X k ) ≤ ϕ ( X k − 1 ) , tr ( X k ) ≤ τ ˉ /2 , tr ( X k ) ≤ τ ˉ /2 ; (120) (b) the FW gap ϵ k ( τ ˉ ) \epsilon_{k}(\bar{\tau}) ϵ k ( τ ˉ ) satisfies
∣ ⟨ G k , X k ⟩ ∣ ≤ ϵ k ( τ ˉ ) , λ k min ( G k ) ≥ − 2 ϵ k ( τ ˉ ) τ ˉ . (121) \left|\langle G_{k},X_{k}\rangle\right| \le\epsilon_{k}(\bar{\tau}), \qquad\lambda_{k}^{\min}(G_{k}) \ge-\frac{2\epsilon_{k}(\bar{\tau})}{\bar{\tau}}. \tag*{(121)} ∣ ⟨ G k , X k ⟩ ∣ ≤ ϵ k ( τ ˉ ) , λ k m i n ( G k ) ≥ − τ ˉ 2 ϵ k ( τ ˉ ) . (121)
Proof. (a) Relation (111) gives ϕ ( X k ) ≤ ϕ ( X ~ k − 1 ) \phi(X_{k}) \le\phi(\widetilde{X}_{k-1}) ϕ ( X k ) ≤ ϕ ( X k − 1 ) . Moreover, the choice of ( α k , ω k ) (\alpha_{k},\omega_{k}) ( α k , ω k ) in (114) and the update (115) give ϕ ( X ~ k ) ≤ ϕ ( X k ) \phi(\widetilde{X}_{k}) \le\phi(X_{k}) ϕ ( X k ) ≤ ϕ ( X k ) . Applying these inequalities recursively yields
ϕ ( X k ) ≤ ϕ ( X ~ 0 ) , ϕ ( X ~ k ) ≤ ϕ ( X ~ 0 ) . \phi(X_{k}) \le\phi(\widetilde{X}_{0}), \qquad\phi(\widetilde{X}_{k}) \le\phi(\widetilde{X}_{0}). ϕ ( X k ) ≤ ϕ ( X 0 ) , ϕ ( X k ) ≤ ϕ ( X 0 ) .
The trace bounds now follow from the definition of τ ˉ \bar{\tau} τ ˉ in (116).
(b) By part (a) and Step 1, X k ∈ Δ τ ˉ / 2 X_{k} \in\Delta_{\bar{\tau}/2} X k ∈ Δ τ ˉ /2 . Moreover, the definition of the FW gap and (102) imply that
ϵ k ( τ ˉ ) = max X ∈ Δ τ ˉ ⟨ G k , X k − X ⟩ = ϵ τ ˉ ( X k ; G k ) . \epsilon_{k}(\bar{\tau}) = \max_{X\in\Delta_{\bar{\tau}}}\langle G_{k},X_{k}-X\rangle= \epsilon_{\bar{\tau}}(X_{k};G_{k}). ϵ k ( τ ˉ ) = X ∈ Δ τ ˉ max ⟨ G k , X k − X ⟩ = ϵ τ ˉ ( X k ; G k ) .
Hence, the argument proving Proposition A.2(b), applied with ( τ , X , G ) = ( τ ˉ , X k , G k ) (\tau,X,G)=(\bar{\tau},X_{k},G_{k}) ( τ , X , G ) = ( τ ˉ , X k , G k ) , and the identity λ k min ( G k ) = − λ max ( − G k ) \lambda_{k}^{\min}(G_{k})=-\lambda_{\max}(-G_{k}) λ k m i n ( G k ) = − λ m a x ( − G k ) yield (119). □
The following lemma characterizes key properties of ϵ k ( τ ˉ ) \epsilon_{k}(\bar{\tau}) ϵ k ( τ ˉ ) and X k F X_{k}^{F} X k F .
Lemma A.5. For every iteration index k ≥ 1 k \ge1 k ≥ 1 , the following relations hold:
τ ˉ X k F ∈ arg min X ∈ Δ τ ˉ ⟨ G k , X ⟩ , ϵ k ( τ ˉ ) = ⟨ G k , X k − τ ˉ X k F ⟩ , (122) \bar{\tau}X_{k}^{F} \in\arg\min_{X\in\Delta_{\bar{\tau}}}\langle G_{k},X\rangle,\qquad\epsilon_{k}(\bar{\tau})=\langle G_{k},X_{k}-\bar{\tau}X_{k}^{F}\rangle, \tag*{(122)} τ ˉ X k F ∈ arg X ∈ Δ τ ˉ min ⟨ G k , X ⟩ , ϵ k ( τ ˉ ) = ⟨ G k , X k − τ ˉ X k F ⟩ , (122) ϵ k ( τ ˉ ) ≥ ϕ ( X k ) − ϕ ∗ . (123) \epsilon_{k}(\bar{\tau}) \ge\phi(X_{k})-\phi_{*}. \tag*{(123)} ϵ k ( τ ˉ ) ≥ ϕ ( X k ) − ϕ ∗ . (123) Proof. It is easy to see from (112), the unit eigenvector convention, and (101) with τ = 1 \tau=1 τ = 1 that X k F ∈ arg min X ∈ Δ 1 ⟨ G k , X ⟩ X_{k}^{F}\in\arg\min_{X\in\Delta_{1}}\langle G_{k},X\rangle X k F ∈ arg min X ∈ Δ 1 ⟨ G k , X ⟩ : if λ k min < 0 \lambda_{k}^{\min}<0 λ k m i n < 0 , then X k F = v k min ( G k ) ( v k min ( G k ) ) ⊤ X_{k}^{F}=v_{k}^{\min}(G_{k})(v_{k}^{\min}(G_{k}))^{\top} X k F = v k m i n ( G k ) ( v k m i n ( G k ) ) ⊤ attains λ k min ( G k ) \lambda_{k}^{\min}(G_{k}) λ k m i n ( G k ) , while if λ k min ( G k ) ≥ 0 \lambda_{k}^{\min}(G_{k})\ge0 λ k m i n ( G k ) ≥ 0 , then X k F = 0 X_{k}^{F}=0 X k F = 0 attains the zero minimum. Scaling gives the first relation in (120). The second relation in (120) follows immediately from the first relation and the definition of ϵ k ( τ ˉ ) \epsilon_{k}(\bar{\tau}) ϵ k ( τ ˉ ) in (116). Finally, convexity, X ∗ ∈ Δ τ ˉ X_{*}\in\Delta_{\bar{\tau}} X ∗ ∈ Δ τ ˉ , and (116) give
ϕ ∗ = ϕ ( X ∗ ) ≥ ϕ ( X k ) + ⟨ G k , X ∗ − X k ⟩ = ϕ ( X k ) − ⟨ G k , X k − X ∗ ⟩ ≥ ϕ ( X k ) − ϵ k ( τ ˉ ) , \phi_{*}=\phi(X_{*})\ge\phi(X_{k})+\langle G_{k},X_{*}-X_{k}\rangle=\phi(X_{k})-\langle G_{k},X_{k}-X_{*}\rangle\ge\phi(X_{k})-\epsilon_{k}(\bar{\tau}), ϕ ∗ = ϕ ( X ∗ ) ≥ ϕ ( X k ) + ⟨ G k , X ∗ − X k ⟩ = ϕ ( X k ) − ⟨ G k , X k − X ∗ ⟩ ≥ ϕ ( X k ) − ϵ k ( τ ˉ ) , which is (121). □
The following proposition gives the one-step estimate on which the analysis is based. It uses the fact that, for every α ∈ [ 0 , 1 ] \alpha\in[0,1] α ∈ [ 0 , 1 ] , the conic search in Step 4 includes the point ( 1 − α ) X k + α τ ˉ X k F (1-\alpha)X_{k}+\alpha\bar{\tau}X_{k}^{F} ( 1 − α ) X k + α τ ˉ X k F .
Proposition A.6. For every iteration index k ≥ 1 k \ge1 k ≥ 1 and every α ∈ [ 0 , 1 ] \alpha\in[0,1] α ∈ [ 0 , 1 ] , we have
ϕ ( X ~ k ) ≤ ϕ ( X k ) − α ϵ k ( τ ˉ ) + 3 τ ˉ 2 L 2 α 2 . (124) \phi(\widetilde{X}_k) \le\phi(X_k) - \alpha\epsilon_k(\bar{\tau}) + \frac{3\bar{\tau}^{2}L}{2}\alpha^2. \tag*{(124)} ϕ ( X k ) ≤ ϕ ( X k ) − α ϵ k ( τ ˉ ) + 2 3 τ ˉ 2 L α 2 . (124) where ϵ k ( τ ˉ ) \epsilon_k(\bar{\tau}) ϵ k ( τ ˉ ) is as in (116).
Proof. Let α ∈ [ 0 , 1 ] \alpha\in[0,1] α ∈ [ 0 , 1 ] be given. Since the pair ( 1 − α , α τ ˉ ) (1-\alpha,\alpha\bar{\tau}) ( 1 − α , α τ ˉ ) is feasible in (114), relations (114)–(115) imply ϕ ( X ~ k ) ≤ ϕ ( ( 1 − α ) X k + α τ ˉ X k F ) = ϕ ( X k + α ( τ ˉ X k F − X k ) ) \phi(\widetilde{X}_k) \le\phi((1-\alpha)X_k+\alpha\bar{\tau}X_k^F)=\phi(X_k+\alpha(\bar{\tau}X_k^F-X_k)) ϕ ( X k ) ≤ ϕ (( 1 − α ) X k + α τ ˉ X k F ) = ϕ ( X k + α ( τ ˉ X k F − X k )) . Observe also that the first relation in (120) and (118) imply that
∥ τ ˉ X k F − X k ∥ ≤ ∥ τ ˉ X k F ∥ + ∥ X k ∥ ≤ 3 τ ˉ 2 . (125) \lVert\bar{\tau}X_k^F-X_k\rVert\le\lVert\bar{\tau}X_k^F\rVert+\lVert X_k\rVert\le\frac{3\bar{\tau}}{2}. \tag*{(125)} ∥ τ ˉ X k F − X k ∥ ≤ ∥ τ ˉ X k F ∥ + ∥ X k ∥ ≤ 2 3 τ ˉ . (125) It then follows from the above observations, (123), G k = ∇ ϕ ( X k ) G_k=\nabla\phi(X_k) G k = ∇ ϕ ( X k ) , and (108) with X ′ = X k + α ( τ ˉ X k F − X k ) ∈ S + n X'=X_k+\alpha(\bar{\tau}X_k^F-X_k)\in S_+^n X ′ = X k + α ( τ ˉ X k F − X k ) ∈ S + n that
ϕ ( X ~ k ) ≤ ϕ ( X k + α ( τ ˉ X k F − X k ) ) ≤ ϕ ( X k ) + α ⟨ ∇ ϕ ( X k ) , τ ˉ X k F − X k ⟩ + L α 2 2 ∥ τ ˉ X k F − X k ∥ 2 ≤ ϕ ( X k ) + α ⟨ G k , τ ˉ X k F − X k ⟩ + 9 L α 2 τ ˉ 2 8 = ϕ ( X k ) − α ϵ k ( τ ˉ ) + 9 L α 2 τ ˉ 2 8 . \begin{aligned}
\phi(\widetilde{X}_k) &\le\phi(X_k+\alpha(\bar{\tau}X_k^F-X_k)) \le\phi(X_k)+\alpha\langle\nabla\phi(X_k),\bar{\tau}X_k^F-X_k\rangle+\frac{L\alpha^2}{2}\lVert\bar{\tau}X_k^F-X_k\rVert^2 \\
&\le\phi(X_k)+\alpha\langle G_k,\bar{\tau}X_k^F-X_k\rangle+\frac{9L\alpha^2\bar{\tau}^2}{8}=\phi(X_k)-\alpha\epsilon_k(\bar{\tau})+\frac{9L\alpha^2\bar{\tau}^2}{8}.
\end{aligned} ϕ ( X k ) ≤ ϕ ( X k + α ( τ ˉ X k F − X k )) ≤ ϕ ( X k ) + α ⟨ ∇ ϕ ( X k ) , τ ˉ X k F − X k ⟩ + 2 L α 2 ∥ τ ˉ X k F − X k ∥ 2 ≤ ϕ ( X k ) + α ⟨ G k , τ ˉ X k F − X k ⟩ + 8 9 L α 2 τ ˉ 2 = ϕ ( X k ) − α ϵ k ( τ ˉ ) + 8 9 L α 2 τ ˉ 2 . Relation (122) then follows immediately from the above inequalities and the fact that 9 / 8 ≤ 3 / 2 9/8\le3/2 9/8 ≤ 3/2 .
The next lemma establishes O ( 1 / k ) O(1/k) O ( 1/ k ) bounds on the objective gap and the minimum Frank–Wolfe gap in (121), both independent of the initial objective gap ϕ ( X ~ 0 ) − ϕ ∗ \phi(\widetilde{X}_0)-\phi_* ϕ ( X 0 ) − ϕ ∗ . By the two relations in (119), the latter bound also controls the quantities appearing in the stopping conditions (113).
Lemma A.7. Let k ≥ 1 k\ge1 k ≥ 1 be an iteration index such that Step 4 is performed at iterations 1 , … , k 1,\ldots,k 1 , … , k . Then the following statements hold:
(a) we have
ϕ ( X ~ k ) − ϕ ∗ ≤ 6 τ ˉ 2 L k + 3 ; (126) \phi(\widetilde{X}_k)-\phi_* \le\frac{6\bar{\tau}^2L}{k+3}; \tag*{(126)} ϕ ( X k ) − ϕ ∗ ≤ k + 3 6 τ ˉ 2 L ; (126) (b) if k ≥ 2 k\ge2 k ≥ 2 , then
min 1 ≤ j ≤ k ϵ j ( τ ˉ ) ≤ 12 τ ˉ 2 L k . (127) \min_{1\le j\le k}\epsilon_j(\bar{\tau})\le\frac{12\bar{\tau}^2L}{k}. \tag*{(127)} 1 ≤ j ≤ k min ϵ j ( τ ˉ ) ≤ k 12 τ ˉ 2 L . (127) Proof. (a) Let j ∈ { 1 , … , k } j\in\{1,\ldots,k\} j ∈ { 1 , … , k } and α ∈ [ 0 , 1 ] \alpha\in[0,1] α ∈ [ 0 , 1 ] be given. Using relations (122) and (121), the fact that 1 − α ≥ 0 1-\alpha\ge0 1 − α ≥ 0 , and the second inequality in (118), we obtain
ϕ ( X ~ j ) − ϕ ∗ ≤ ϕ ( X j ) − ϕ ∗ − α ϵ j ( τ ˉ ) + 3 τ ˉ 2 L 2 α 2 ≤ ( 1 − α ) [ ϕ ( X j ) − ϕ ∗ ] + 3 τ ˉ 2 L 2 α 2 ≤ ( 1 − α ) [ ϕ ( X ~ j − 1 ) − ϕ ∗ ] + 3 τ ˉ 2 L 2 α 2 . (128) \begin{aligned}
\phi(\widetilde{X}_j)-\phi_* &\le\phi(X_j)-\phi_*-\alpha\epsilon_j(\bar{\tau})+\frac{3\bar{\tau}^2L}{2}\alpha^2 \\
&\le(1-\alpha)[\phi(X_j)-\phi_*]+\frac{3\bar{\tau}^2L}{2}\alpha^2 \\
&\le(1-\alpha)[\phi(\widetilde{X}_{j-1})-\phi_*]+\frac{3\bar{\tau}^2L}{2}\alpha^2. \tag*{(128)}
\end{aligned} ϕ ( X j ) − ϕ ∗ ≤ ϕ ( X j ) − ϕ ∗ − α ϵ j ( τ ˉ ) + 2 3 τ ˉ 2 L α 2 ≤ ( 1 − α ) [ ϕ ( X j ) − ϕ ∗ ] + 2 3 τ ˉ 2 L α 2 ≤ ( 1 − α ) [ ϕ ( X j − 1 ) − ϕ ∗ ] + 2 3 τ ˉ 2 L α 2 . (128) We now prove (124) by induction. Relation (126) with j = 1 j=1 j = 1 and α = 1 \alpha=1 α = 1 gives
ϕ ( X ~ 1 ) − ϕ ∗ ≤ 3 τ ˉ 2 L 2 = 6 τ ˉ 2 L 1 + 3 , \phi(\widetilde{X}_1)-\phi_* \le\frac{3\bar{\tau}^2L}{2}=\frac{6\bar{\tau}^2L}{1+3}, ϕ ( X 1 ) − ϕ ∗ ≤ 2 3 τ ˉ 2 L = 1 + 3 6 τ ˉ 2 L , which is (124) with k = 1 k=1 k = 1 . Observe that this bound does not depend on X ~ 0 \widetilde{X}_0 X 0 since the coefficient 1 − α 1-\alpha 1 − α multiplying ϕ ( X ~ 0 ) − ϕ ∗ \phi(\widetilde{X}_0)-\phi_* ϕ ( X 0 ) − ϕ ∗ in (126) vanishes. Assume now that 2 ≤ j ≤ k 2\le j\le k 2 ≤ j ≤ k and that ϕ ( X ~ j − 1 ) − ϕ ∗ ≤ 6 τ ˉ 2 L / ( j + 2 ) \phi(\widetilde{X}_{j-1})-\phi_*\le6\bar{\tau}^2L/(j+2) ϕ ( X j − 1 ) − ϕ ∗ ≤ 6 τ ˉ 2 L / ( j + 2 ) . Relation (126) with α = 2 / ( j + 2 ) ∈ [ 0 , 1 ] \alpha=2/(j+2)\in[0,1] α = 2/ ( j + 2 ) ∈ [ 0 , 1 ] and this assumption then imply that
ϕ ( X ~ j ) − ϕ ∗ ≤ j j + 2 ⋅ 6 τ ˉ 2 L j + 2 + 3 τ ˉ 2 L 2 ⋅ 4 ( j + 2 ) 2 = 6 τ ˉ 2 L ( j + 1 ) ( j + 2 ) 2 ≤ 6 τ ˉ 2 L j + 3 , \phi(\widetilde{X}_j)-\phi_* \le\frac{j}{j+2}\cdot\frac{6\bar{\tau}^2L}{j+2}+\frac{3\bar{\tau}^2L}{2}\cdot\frac{4}{(j+2)^2}=\frac{6\bar{\tau}^2L(j+1)}{(j+2)^2}\le\frac{6\bar{\tau}^2L}{j+3}, ϕ ( X j ) − ϕ ∗ ≤ j + 2 j ⋅ j + 2 6 τ ˉ 2 L + 2 3 τ ˉ 2 L ⋅ ( j + 2 ) 2 4 = ( j + 2 ) 2 6 τ ˉ 2 L ( j + 1 ) ≤ j + 3 6 τ ˉ 2 L , where the last inequality is due to the fact that ( j + 1 ) ( j + 3 ) ≤ ( j + 2 ) 2 (j+1)(j+3)\le(j+2)^2 ( j + 1 ) ( j + 3 ) ≤ ( j + 2 ) 2 . Relation (124) then follows by induction.
(b) Let k ≥ 2 k\ge2 k ≥ 2 be given and note that ⌊ k / 2 ⌋ ≥ 1 \lfloor k/2\rfloor\ge1 ⌊ k /2 ⌋ ≥ 1 . Let α ∈ ( 0 , 1 ] \alpha\in(0,1] α ∈ ( 0 , 1 ] and j ∈ { 1 , … , k } j\in\{1,\ldots,k\} j ∈ { 1 , … , k } be given. Relation (122) and the second inequality in (118) imply that
α ϵ j ( τ ˉ ) ≤ ϕ ( X j ) − ϕ ( X ~ j ) + 3 τ ˉ 2 L 2 α 2 ≤ ϕ ( X ~ j − 1 ) − ϕ ( X ~ j ) + 3 τ ˉ 2 L 2 α 2 . (129) \alpha\epsilon_j(\bar{\tau})\le\phi(X_j)-\phi(\widetilde{X}_j)+\frac{3\bar{\tau}^2L}{2}\alpha^2\le\phi(\widetilde{X}_{j-1})-\phi(\widetilde{X}_j)+\frac{3\bar{\tau}^2L}{2}\alpha^2. \tag*{(129)} α ϵ j ( τ ˉ ) ≤ ϕ ( X j ) − ϕ ( X j ) + 2 3 τ ˉ 2 L α 2 ≤ ϕ ( X j − 1 ) − ϕ ( X j ) + 2 3 τ ˉ 2 L α 2 . (129)
Summing (129) over j = ⌊ k / 2 ⌋ + 1 , … , k j=\lfloor k/2\rfloor+1,\ldots,k j = ⌊ k /2 ⌋ + 1 , … , k , a range with ⌈ k / 2 ⌉ \lceil k/2\rceil ⌈ k /2 ⌉ indices, and using the fact that ϕ ( X ~ k ) ≥ ϕ ∗ \phi(\widetilde{X}_k)\geq\phi_\ast ϕ ( X k ) ≥ ϕ ∗ and (126) with k k k replaced by ⌊ k / 2 ⌋ \lfloor k/2\rfloor ⌊ k /2 ⌋ , we obtain
α ⌈ k 2 ⌉ min 1 ≤ j ≤ k ϵ j ( τ ˉ ) ≤ α ∑ j = ⌊ k / 2 ⌋ + 1 k ϵ j ( τ ˉ ) ≤ ( 127 ) ϕ ( X ~ ⌊ k / 2 ⌋ ) − ϕ ( X ~ k ) + ⌈ k 2 ⌉ 3 τ ˉ 2 L 2 α 2 \alpha\left\lceil\frac{k}{2}\right\rceil\min_{1\leq j\leq k}\epsilon_j(\bar{\tau})\leq\alpha\sum_{j=\lfloor k/2\rfloor+1}^{k}\epsilon_j(\bar{\tau})\mathrel{\overset{(127)}{\leq}}\phi(\widetilde{X}_{\lfloor k/2\rfloor})-\phi(\widetilde{X}_k)+\left\lceil\frac{k}{2}\right\rceil\frac{3\bar{\tau}^{2}L}{2}\alpha^2 α ⌈ 2 k ⌉ 1 ≤ j ≤ k min ϵ j ( τ ˉ ) ≤ α j = ⌊ k /2 ⌋ + 1 ∑ k ϵ j ( τ ˉ ) ≤ ( 127 ) ϕ ( X ⌊ k /2 ⌋ ) − ϕ ( X k ) + ⌈ 2 k ⌉ 2 3 τ ˉ 2 L α 2
≤ ( 124 ) 6 τ ˉ 2 L ⌊ k / 2 ⌋ + 3 + ⌈ k 2 ⌉ 3 τ ˉ 2 L 2 α 2 . \mathrel{\overset{(124)}{\leq}}\frac{6\bar{\tau}^{2}L}{\lfloor k/2\rfloor+3}+\left\lceil\frac{k}{2}\right\rceil\frac{3\bar{\tau}^{2}L}{2}\alpha^2. ≤ ( 124 ) ⌊ k /2 ⌋ + 3 6 τ ˉ 2 L + ⌈ 2 k ⌉ 2 3 τ ˉ 2 L α 2 .
Taking α = 2 / ( ⌊ k / 2 ⌋ + 3 ) ∈ ( 0 , 1 ] \alpha=2/(\lfloor k/2\rfloor+3)\in(0,1] α = 2/ (⌊ k /2 ⌋ + 3 ) ∈ ( 0 , 1 ] in the above inequality and dividing the result by α ⌈ k / 2 ⌉ \alpha\lceil k/2\rceil α ⌈ k /2 ⌉ , we conclude that
min 1 ≤ j ≤ k ϵ j ( τ ˉ ) ≤ 3 τ ˉ 2 L ⌊ k / 2 ⌋ + 3 τ ˉ 2 L ⌊ k / 2 ⌋ + 3 ≤ 6 τ ˉ 2 L k + 6 τ ˉ 2 L k = 12 τ ˉ 2 L k . \min_{1\leq j\leq k}\epsilon_j(\bar{\tau})\leq\frac{3\bar{\tau}^{2}L}{\lfloor k/2\rfloor}+\frac{3\bar{\tau}^{2}L}{\lfloor k/2\rfloor+3}\leq\frac{6\bar{\tau}^{2}L}{k}+\frac{6\bar{\tau}^{2}L}{k}=\frac{12\bar{\tau}^{2}L}{k}. 1 ≤ j ≤ k min ϵ j ( τ ˉ ) ≤ ⌊ k /2 ⌋ 3 τ ˉ 2 L + ⌊ k /2 ⌋ + 3 3 τ ˉ 2 L ≤ k 6 τ ˉ 2 L + k 6 τ ˉ 2 L = k 12 τ ˉ 2 L .
where the last inequality is due to the fact that both ⌈ k / 2 ⌉ \lceil k/2\rceil ⌈ k /2 ⌉ and ⌊ k / 2 ⌋ + 3 \lfloor k/2\rfloor+3 ⌊ k /2 ⌋ + 3 are bounded below by k / 2 k/2 k /2 .
We are now ready to prove Proposition A.3.
Proof of Proposition A.3. (a) Applying the first relation in (120) recursively implies that ϕ ( X k ) ≤ ϕ ( X ~ 0 ) \phi(X_k)\leq\phi(\widetilde{X}_0) ϕ ( X k ) ≤ ϕ ( X 0 ) for every iteration index k ≥ 1 k\geq1 k ≥ 1 .
(b) Let k k k denote the integer in (119) . Then k ≥ 2 k\geq2 k ≥ 2 and k > 12 τ ˉ 2 L ( 1 / δ + 2 / ϵ ) k>12\bar{\tau}^{2}L(1/\delta+2/\epsilon) k > 12 τ ˉ 2 L ( 1/ δ + 2/ ϵ ) , and hence
12 τ ˉ 2 L k < ( 1 δ + 2 ϵ ) − 1 ≤ min { δ , ϵ 2 } . (130) \frac{12\bar{\tau}^{2}L}{k}<\left(\frac{1}{\delta}+\frac{2}{\epsilon}\right)^{-1}\leq\min\left\{\delta,\frac{\epsilon}{2}\right\}. \tag*{(130)} k 12 τ ˉ 2 L < ( δ 1 + ϵ 2 ) − 1 ≤ min { δ , 2 ϵ } . (130)
Suppose now for contradiction, that RUFW does not terminate at any iteration index j ≤ k j\leq k j ≤ k , so that it performs Step 4 at iterations 1 , … , k 1,\ldots,k 1 , … , k and the stopping test (115) fails at every iteration index j ≤ k j\leq k j ≤ k . Both relations in (121) , the fact that τ ˉ ≥ 1 \bar{\tau}\geq1 τ ˉ ≥ 1 , and (130) imply that there exists an iteration index i ∈ { 1 , … , k } i\in\{1,\ldots,k\} i ∈ { 1 , … , k } such that
∣ ⟨ G i , X i ⟩ ∣ ≤ ( 119 ) ϵ i ( τ ˉ ) ≤ ( 125 ) 12 τ ˉ 2 L k < ( 128 ) δ , \lvert\langle G_i,X_i\rangle\rvert\mathrel{\overset{(119)}{\leq}}\epsilon_i(\bar{\tau})\mathrel{\overset{(125)}{\leq}}\frac{12\bar{\tau}^{2}L}{k}\mathrel{\overset{(128)}{<}}\delta, ∣⟨ G i , X i ⟩∣ ≤ ( 119 ) ϵ i ( τ ˉ ) ≤ ( 125 ) k 12 τ ˉ 2 L < ( 128 ) δ ,
λ i min ( G i ) ≥ ( 119 ) − 2 ϵ i ( τ ˉ ) τ ˉ ≥ ( 125 ) − 24 τ ˉ L k ≥ − 2 ⋅ 12 τ ˉ 2 L k > ( 128 ) − ϵ . \lambda_i^{\min}(G_i)\mathrel{\overset{(119)}{\geq}}\frac{-2\epsilon_i(\bar{\tau})}{\bar{\tau}}\mathrel{\overset{(125)}{\geq}}-\frac{24\bar{\tau}L}{k}\mathrel{\geq}-2\cdot\frac{12\bar{\tau}^{2}L}{k}\mathrel{\overset{(128)}{>}}-\epsilon. λ i m i n ( G i ) ≥ ( 119 ) τ ˉ − 2 ϵ i ( τ ˉ ) ≥ ( 125 ) − k 24 τ ˉ L ≥ − 2 ⋅ k 12 τ ˉ 2 L > ( 128 ) − ϵ .
Thus, the stopping test (115) holds at the iteration index i ≤ k i\leq k i ≤ k , which contradicts the assumption that RUFW does not terminate at any iteration index j ≤ k j\leq k j ≤ k . Hence, RUFW terminates at some iteration index j ≤ k j\leq k j ≤ k with output X ˉ = X j \bar{X}=X_j X ˉ = X j , which satisfies (109) in view of the stopping test (115) and satisfies (110) in view of part (a).
U-Space and Factorized Machinery
This appendix collects the material used to implement the descent step (111) of the RUFW framework of Appendix A.2 in the factorized U U U -space. It is divided into two parts. Subsection B.1 presents the ADAP-FISTA method while Subsection B.2 presents the ADAP-AIPP method, whose subproblems are solved by ADAP-FISTA.
ADAP-FISTA
Let E \mathrm{E} E be a finite-dimensional inner product space and let Ω ⊆ E \Omega\subseteq\mathrm{E} Ω ⊆ E be a nonempty closed convex set. This section presents the ADAP-FISTA method for the problem
min { ψ ( U ) : U ∈ Ω } (131) \min\{\psi(U):U\in\Omega\} \tag*{(131)} min { ψ ( U ) : U ∈ Ω } (131)
where ψ \psi ψ and Ω \Omega Ω are assumed to satisfy the following assumptions:
(B1) ψ \psi ψ is a real-valued function that is differentiable on E \mathrm{E} E and whose gradient is locally Lipschitz continuous in the sense of (10) ;
(B2) Ω \Omega Ω is closed, possibly unbounded, convex set that admits an easily computable projection;
(B3) a point X 0 ∈ Ω X_0\in\Omega X 0 ∈ Ω is given such that the sublevel set S ( X 0 ; ψ ) S(X_0;\psi) S ( X 0 ; ψ ) has finite diameter D ( X 0 ; ψ ) D(X_0;\psi) D ( X 0 ; ψ ) , where S ( ⋅ ; ⋅ ) S(\cdot;\cdot) S ( ⋅ ; ⋅ ) and D ( ⋅ ; ⋅ ) D(\cdot;\cdot) D ( ⋅ ; ⋅ ) are as in (5) and (6), respectively.
Problem: Given ( ψ , Ω , X 0 ) (\psi,\Omega,X_0) ( ψ , Ω , X 0 ) satisfying (B1)–(B3), the goal of ADAP-FISTA is to find a pair ( Y ‾ , V ‾ ) ∈ Ω × E (\overline{Y},\overline{V}) \in\Omega\times\mathbb{E} ( Y , V ) ∈ Ω × E such that
V ‾ ∈ ∇ ψ ( Y ‾ ) + N Ω ( Y ‾ ) , ∥ V ‾ ∥ ≤ σ ∥ Y ‾ − X 0 ∥ , ψ ( Y ‾ ) ≤ ψ ( X 0 ) . (132) \overline{V} \in\nabla\psi(\overline{Y}) + N_{\Omega}(\overline{Y}), \qquad\|\overline{V}\| \le\sigma\|\overline{Y}-X_0\|, \qquad\psi(\overline{Y}) \le\psi(X_0). \tag*{(132)} V ∈ ∇ ψ ( Y ) + N Ω ( Y ) , ∥ V ∥ ≤ σ ∥ Y − X 0 ∥ , ψ ( Y ) ≤ ψ ( X 0 ) . (132)
The formal description of ADAP-FISTA is now given below.
ADAP-FISTA Method
Universal Parameters: σ > 0 \sigma> 0 σ > 0 and χ ∈ ( 0 , 1 ) \chi\in(0,1) χ ∈ ( 0 , 1 ) .
Input: initial point Y ‾ 0 ∈ Ω \overline{Y}_0 \in\Omega Y 0 ∈ Ω , pair μ > 0 \mu> 0 μ > 0 , L 0 > μ L_0 > \mu L 0 > μ , and a function ψ \psi ψ and a set Ω \Omega Ω as in (B1)–(B3)
set Y 0 = X 0 = Y ‾ 0 Y_0 = X_0 = \overline{Y}_0 Y 0 = X 0 = Y 0 , A 0 = 0 A_0 = 0 A 0 = 0 , τ 0 = 1 \tau_0 = 1 τ 0 = 1 , and i = 1 i = 1 i = 1 ;
set L = L i − 1 L = L_{i-1} L = L i − 1 ;
compute
a = τ i − 1 + τ i − 1 2 + 4 τ i − 1 A i − 1 ( L − μ ) 2 ( L − μ ) , X ~ = A i − 1 Y i − 1 + a X i − 1 A i − 1 + a , (133) a = \frac{\tau_{i-1}+\sqrt{\tau_{i-1}^{2}+4\tau_{i-1}A_{i-1}(L-\mu)}}{2(L-\mu)}, \qquad\widetilde{X} = \frac{A_{i-1}Y_{i-1}+aX_{i-1}}{A_{i-1}+a}, \tag*{(133)} a = 2 ( L − μ ) τ i − 1 + τ i − 1 2 + 4 τ i − 1 A i − 1 ( L − μ ) , X = A i − 1 + a A i − 1 Y i − 1 + a X i − 1 , (133)
Y : = arg min U ∈ Ω { ℓ ψ ( U ; X ~ ) + L 2 ∥ U − X ~ ∥ 2 } . (134) Y := \mathop{\arg\min}_{U\in\Omega}\left\{\ell_{\psi}(U;\widetilde{X})+\frac{L}{2}\|U-\widetilde{X}\|^{2}\right\}. \tag*{(134)} Y := arg min U ∈ Ω { ℓ ψ ( U ; X ) + 2 L ∥ U − X ∥ 2 } . (134)
If the inequality
ℓ ψ ( Y ; X ~ ) + ( 1 − χ ) L 2 ∥ Y − X ~ ∥ 2 ≥ ψ ( Y ) (135) \ell_{\psi}(Y;\widetilde{X})+\frac{(1-\chi)L}{2}\|Y-\widetilde{X}\|^{2}\ge\psi(Y) \tag*{(135)} ℓ ψ ( Y ; X ) + 2 ( 1 − χ ) L ∥ Y − X ∥ 2 ≥ ψ ( Y ) (135)
holds then set
a i − 1 = a , X ~ i − 1 = X ~ , Y ~ i = Y , L i = L (136) a_{i-1}=a,\qquad\widetilde{X}_{i-1}=\widetilde{X},\qquad\widetilde{Y}_i=Y,\qquad L_i=L \tag*{(136)} a i − 1 = a , X i − 1 = X , Y i = Y , L i = L (136)
and go to step 3; else set L ← 2 L L \leftarrow2L L ← 2 L and repeat step 2;
compute
A i = A i − 1 + a i − 1 , τ i = τ i − 1 + a i − 1 μ , (137) A_i=A_{i-1}+a_{i-1},\qquad\tau_i=\tau_{i-1}+a_{i-1}\mu, \tag*{(137)} A i = A i − 1 + a i − 1 , τ i = τ i − 1 + a i − 1 μ , (137)
X i = 1 τ i [ μ a i − 1 Y i + τ i − 1 X i − 1 − a i − 1 ( L i − μ ) ( X ~ i − 1 − Y i ) ] ; X_i=\frac{1}{\tau_i}\left[\mu a_{i-1}Y_i+\tau_{i-1}X_{i-1}-a_{i-1}(L_i-\mu)(\widetilde{X}_{i-1}-Y_i)\right]; X i = τ i 1 [ μ a i − 1 Y i + τ i − 1 X i − 1 − a i − 1 ( L i − μ ) ( X i − 1 − Y i ) ] ;
if the following three inequalities hold
∥ Y i − Y ‾ 0 ∥ 2 ≥ χ A i L i ∥ Y i − X ~ i − 1 ∥ 2 , ψ ( Y i ) ≤ ψ ( Y ‾ 0 ) (138) \|Y_i-\overline{Y}_0\|^2\ge\chi A_iL_i\|Y_i-\widetilde{X}_{i-1}\|^2,\qquad\psi(Y_i)\le\psi(\overline{Y}_0) \tag*{(138)} ∥ Y i − Y 0 ∥ 2 ≥ χ A i L i ∥ Y i − X i − 1 ∥ 2 , ψ ( Y i ) ≤ ψ ( Y 0 ) (138)
∥ Y i − X i ∥ ≤ ∥ Y i − Y ‾ 0 ∥ (139) \|Y_i-X_i\|\le\|Y_i-\overline{Y}_0\| \tag*{(139)} ∥ Y i − X i ∥ ≤ ∥ Y i − Y 0 ∥ (139)
hold, then go to step 5; otherwise, stop with failure ;
compute
V i = ∇ ψ ( Y i ) − ∇ ψ ( X ~ i − 1 ) + L i ( X ~ i − 1 − Y i ) ; (140) V_i=\nabla\psi(Y_i)-\nabla\psi(\widetilde{X}_{i-1})+L_i(\widetilde{X}_{i-1}-Y_i); \tag*{(140)} V i = ∇ ψ ( Y i ) − ∇ ψ ( X i − 1 ) + L i ( X i − 1 − Y i ) ; (140)
if the inequality
∥ V i ∥ ≤ σ ∥ Y i − Y ‾ 0 ∥ (141) \|V_i\|\le\sigma\|Y_i-\overline{Y}_0\| \tag*{(141)} ∥ V i ∥ ≤ σ ∥ Y i − Y 0 ∥ (141)
holds then stop with success and output ( Y ‾ , V ‾ , L ‾ ) : = ( Y i , V i , L i ) (\overline{Y},\overline{V},\overline{L}):=(Y_i,V_i,L_i) ( Y , V , L ) := ( Y i , V i , L i ) ; otherwise, i ← i + 1 i\leftarrow i+1 i ← i + 1 and go to step 1.
It is easy to see that the update for Y i Y_i Y i in (148) is equivalent to projecting the point X ~ i − 1 − ( 1 / L i ) ∇ ψ ( X ~ i − 1 ) \widetilde{X}_{i-1}-(1/L_i)\nabla\psi(\widetilde{X}_{i-1}) X i − 1 − ( 1/ L i ) ∇ ψ ( X i − 1 ) onto the set Ω \Omega Ω . ADAP-FISTA takes a parameter μ > 0 \mu>0 μ > 0 as input, even when we do not know in advance whether ψ \psi ψ is μ \mu μ -strongly convex. The method therefore checks the inequalities in (136) and (137) at each iteration. If any of these inequalities is violated, the method stops with failure. When ψ \psi ψ is μ \mu μ -strongly convex, these inequalities always hold, and ADAP-FISTA returns a triple satisfying (139) within a logarithmic number of iterations. To accommodate objectives with locally Lipschitz continuous gradients, we use different inequalities from those in earlier adaptive variants of FISTA [38 , 54 ] .
Before stating the main complexity result of ADAP-FISTA, we introduce the quantities which appear in the complexity result:
D 0 : = 6 D ( X 0 ; ψ ) + 1 , G 0 : = ∇ ( D 0 / 2 ; ψ ) , ( lip ) 0 : = [ LIP ] ( D 0 ; ψ ) (142) D_0:=6D(X_0;\psi)+1,\qquad G_0:=\nabla(D_0/2;\psi),\qquad(\operatorname{lip})_0:=[\operatorname{LIP}](D_0;\psi) \tag*{(142)} D 0 := 6 D ( X 0 ; ψ ) + 1 , G 0 := ∇ ( D 0 /2 ; ψ ) , ( lip ) 0 := [ LIP ] ( D 0 ; ψ ) (142)
where D ( ⋅ ; ⋅ ) D(\cdot;\cdot) D ( ⋅ ; ⋅ ) , ∇ ( ⋅ ; ⋅ ) \nabla(\cdot;\cdot) ∇ ( ⋅ ; ⋅ ) , and [LIP] are as in (8) , (12) , and (11) , respectively. The quantities above are finite. Indeed, (B3) implies that D 0 D_0 D 0 is finite. Since ∇ ψ \nabla\psi ∇ ψ is continuous, it follows that G 0 < ∞ G_0 < \infty G 0 < ∞ . Since ∇ ψ \nabla\psi ∇ ψ is locally Lipschitz, a finite-cover argument on the compact ball B [ D 0 ] B[D_0] B [ D 0 ] yields that the Lipschitz constant ( lip ) 0 (\operatorname{lip})_0 ( lip ) 0 is finite. The main proposition which states ADAP-FISTA’s main complexity result is now given. Its proof is given in the next two subsections.
Proposition B.1. Let
N 0 : = max { L 0 , G 0 , ( lip ) 0 } . (143) N_0 := \max\{L_0,G_0,(\operatorname{lip})_0\}. \tag*{(143)} N 0 := max { L 0 , G 0 , ( lip ) 0 } . (143) where L 0 L_0 L 0 is input to ADAP-FISTA and G 0 G_0 G 0 and ( lip ) 0 (\operatorname{lip})_0 ( lip ) 0 are as in (142) . The following statements about ADAP-FISTA hold:
(a) it always stops (with either success or failure) in at most
O 1 ( N 0 μ log 2 + ( N 0 ) + log 2 + ( N 0 L 0 ) ) \mathcal{O}_1\left(\sqrt{\frac{N_0}{\mu}}\log_2^+(N_0)+\log_2^+\left(\frac{N_0}{L_0}\right)\right) O 1 ( μ N 0 log 2 + ( N 0 ) + log 2 + ( L 0 N 0 ) ) iterations/resolvent evaluations;
(b) if it stops successfully, then the pair ( Y ‾ , V ‾ ) ∈ Ω × E (\overline{Y},\overline{V}) \in\Omega\times\mathbb{E} ( Y , V ) ∈ Ω × E output by ADAP-FISTA satisfies the goal (132) , i.e.,
V ‾ ∈ ∇ ψ ( Y ‾ ) + N Ω ( Y ‾ ) , ∥ V ‾ ∥ ≤ σ ∥ Y ‾ − Y ‾ 0 ∥ , (144) \overline{V} \in\nabla\psi(\overline{Y})+N_{\Omega}(\overline{Y}),\qquad\|\overline{V}\|\leq\sigma\|\overline{Y}-\overline{Y}_0\|, \tag*{(144)} V ∈ ∇ ψ ( Y ) + N Ω ( Y ) , ∥ V ∥ ≤ σ ∥ Y − Y 0 ∥ , (144) ψ ( Y ‾ ) ≤ ψ ( Y ‾ 0 ) , (145) \psi(\overline{Y})\leq\psi(\overline{Y}_0), \tag*{(145)} ψ ( Y ) ≤ ψ ( Y 0 ) , (145) and the Lipschitz estimate L ‾ \overline{L} L that it outputs satisfies
L 0 ≤ L ‾ ≤ max { L 0 , 12 G 0 , 2 ( lip ) 0 1 − χ } ; (146) L_0\leq\overline{L}\leq\max\left\{L_0,12G_0,\frac{2(\operatorname{lip})_0}{1-\chi}\right\}; \tag*{(146)} L 0 ≤ L ≤ max { L 0 , 12 G 0 , 1 − χ 2 ( lip ) 0 } ; (146) (c) if ψ \psi ψ is μ \mu μ -strongly convex on B [ D 0 ] B[D_0] B [ D 0 ] , then ADAP-FISTA always terminates with success.
We now make one brief remark about the above proposition. In general, the assumption in (c) that ψ \psi ψ is μ \mu μ -strongly convex on B [ D 0 ] B[D_0] B [ D 0 ] is not directly verifiable since we do not know D 0 D_0 D 0 . However, we will see in the next section that it will play an important role in guaranteeing that an inner loop of our other method ADAP-AIPP always terminates.
Proof of Proposition B.1(a)-(b)
This subsection is dedicated to proving Proposition B.1(a)-(b). We first define several quantities which arise in our technical analysis:
ζ : = ( lip ) 0 + max { L 0 , 12 G 0 , 2 ( lip ) 0 1 − χ } , Q : = 4 μ max { L 0 , 12 G 0 , 2 ( lip ) 0 1 − χ } . (147) \zeta:= (\operatorname{lip})_0+\max\left\{L_0,12G_0,\frac{2(\operatorname{lip})_0}{1-\chi}\right\},\qquad Q:=\sqrt{\frac{4}{\mu}\max\left\{L_0,12G_0,\frac{2(\operatorname{lip})_0}{1-\chi}\right\}}. \tag*{(147)} ζ := ( lip ) 0 + max { L 0 , 12 G 0 , 1 − χ 2 ( lip ) 0 } , Q := μ 4 max { L 0 , 12 G 0 , 1 − χ 2 ( lip ) 0 } . (147)
Also note that since ∇ ψ \nabla\psi ∇ ψ is ( lip ) 0 (\operatorname{lip})_0 ( lip ) 0 -Lipschitz on B [ D 0 ] B[D_0] B [ D 0 ] it follows from that
ψ ( X ′ ′ ) ≤ ψ ( X ′ ) + ⟨ ∇ ψ ( X ′ ) , X ′ ′ − X ′ ⟩ + ( lip ) 0 2 ∥ X ′ ′ − X ′ ∥ 2 ∀ X ′ , X ′ ′ ∈ B [ D 0 ] . (148) \psi(X^{\prime\prime})\leq\psi(X^\prime)+\langle\nabla\psi(X^\prime),X^{\prime\prime}-X^\prime\rangle+\frac{(\operatorname{lip})_0}{2}\|X^{\prime\prime}-X^\prime\|^2\qquad\forall X^\prime,X^{\prime\prime}\in B[D_0]. \tag*{(148)} ψ ( X ′′ ) ≤ ψ ( X ′ ) + ⟨ ∇ ψ ( X ′ ) , X ′′ − X ′ ⟩ + 2 ( lip ) 0 ∥ X ′′ − X ′ ∥ 2 ∀ X ′ , X ′′ ∈ B [ D 0 ] . (148)
We start off with a key lemma which characterizes properties of the output of the loop in step 2 of ADAP-FISTA.
Lemma B.2. Let i ≥ 1 i\geq1 i ≥ 1 be an iteration index of ADAP-FISTA. If the line-search in the loop in step 2 terminates, then the following statements then hold:
(a)
a i − 1 = τ i − 1 + τ i − 1 2 + 4 τ i − 1 A i − 1 ( L i − μ ) 2 ( L i − μ ) , X ~ i − 1 = A i − 1 Y i − 1 + a i − 1 X i − 1 A i − 1 + a i − 1 , (149) a_{i-1}=\frac{\tau_{i-1}+\sqrt{\tau_{i-1}^2+4\tau_{i-1}A_{i-1}(L_i-\mu)}}{2(L_i-\mu)},\qquad\widetilde{X}_{i-1}=\frac{A_{i-1}Y_{i-1}+a_{i-1}X_{i-1}}{A_{i-1}+a_{i-1}}, \tag*{(149)} a i − 1 = 2 ( L i − μ ) τ i − 1 + τ i − 1 2 + 4 τ i − 1 A i − 1 ( L i − μ ) , X i − 1 = A i − 1 + a i − 1 A i − 1 Y i − 1 + a i − 1 X i − 1 , (149) Y i : = arg min U ∈ Ω { ℓ ψ ( U ; X ~ i − 1 ) + L i 2 ∥ U − X ~ i − 1 ∥ 2 } ; (150) Y_i:=\mathop{\arg\min}_{U\in\Omega}\left\{\ell_\psi(U;\widetilde{X}_{i-1})+\frac{L_i}{2}\|U-\widetilde{X}_{i-1}\|^2\right\}; \tag*{(150)} Y i := arg min U ∈ Ω { ℓ ψ ( U ; X i − 1 ) + 2 L i ∥ U − X i − 1 ∥ 2 } ; (150)
(b) it also holds that
τ i − 1 = 1 + A i − 1 μ , τ i − 1 A i a i − 1 2 = L i − μ (151) \tau_{i-1}=1+A_{i-1}\mu,\qquad\frac{\tau_{i-1}A_i}{a_{i-1}^{2}}=L_i-\mu \tag*{(151)} τ i − 1 = 1 + A i − 1 μ , a i − 1 2 τ i − 1 A i = L i − μ (151)
A i L i ≥ max { i 2 4 , ( 1 + Q − 1 ) 2 ( i − 1 ) } . (152) A_iL_i\ge\max\left\{\frac{i^2}{4},\left(1+Q^{-1}\right)^{2(i-1)}\right\}. \tag*{(152)} A i L i ≥ max { 4 i 2 , ( 1 + Q − 1 ) 2 ( i − 1 ) } . (152)
Proof. (a) The proof of (a) is immediate from relations (133) and (134) and the way a i − 1 a_{i-1} a i − 1 , X ~ i − 1 \widetilde{X}_{i-1} X i − 1 , Y i Y_i Y i , and L i L_i L i are set in (136) at the end of the loop in step 2 of ADAP-FISTA. (b) The first equality in (151) follows directly from both of the relations in (137) . The second equality in (151) follows immediately from the definition of a i − 1 a_{i-1} a i − 1 in (149) and the first relation in (137) . The proof of the lower bound on A i L i A_iL_i A i L i in (152) follows from a near identical argument to the proof of Lemma A.4 in [53 ] .
We now start off with a key technical lemma which is instrumental for showing that the line-search performed in step 2 of ADAP-FISTA always terminates.
Lemma B.3. Let θ > 0 \theta>0 θ > 0 , D 0 D_0 D 0 be as in (142) , and X ^ ∈ B [ D 0 / 2 ] \widehat{X}\in B[D_0/2] X ∈ B [ D 0 /2 ] . For L ^ > 0 \widehat{L}>0 L > 0 , define
Y L ^ + : = arg min U ∈ Ω { ℓ ψ ( U ; X ^ ) + L ^ 2 ∥ U − X ^ ∥ 2 } . (153) Y_{\widehat{L}}^{+}:=\mathop{\arg\min}_{U\in\Omega}\left\{\ell_{\psi}(U;\widehat{X})+\frac{\widehat{L}}{2}\lVert U-\widehat{X}\rVert^{2}\right\}. \tag*{(153)} Y L + := arg min U ∈ Ω { ℓ ψ ( U ; X ) + 2 L ∥ U − X ∥ 2 } . (153) If
L ^ ≥ max { 6 ∥ ∇ ψ ( X ^ ) ∥ , ( lip ) 0 θ } ≥ max { 6 ∥ ∇ ψ ( X ^ ) ∥ D 0 , ( lip ) 0 θ } , (154) \widehat{L}\ge\max\left\{6\lVert\nabla\psi(\widehat{X})\rVert,\frac{(\operatorname{lip})_0}{\theta}\right\}\ge\max\left\{\frac{6\lVert\nabla\psi(\widehat{X})\rVert}{D_0},\frac{(\operatorname{lip})_0}{\theta}\right\}, \tag*{(154)} L ≥ max { 6 ∥ ∇ ψ ( X )∥ , θ ( lip ) 0 } ≥ max { D 0 6 ∥ ∇ ψ ( X )∥ , θ ( lip ) 0 } , (154) then Y L ^ + ∈ B [ D 0 ] Y_{\widehat{L}}^{+}\in B[D_0] Y L + ∈ B [ D 0 ] and
ℓ ψ ( Y L ^ + ; X ^ ) + 0.5 L ^ ∥ Y L ^ + − X ^ ∥ 2 ≥ ψ ( Y L ^ + ) . (155) \ell_{\psi}(Y_{\widehat{L}}^{+};\widehat{X})+0.5\widehat{L}\lVert Y_{\widehat{L}}^{+}-\widehat{X}\rVert^{2}\ge\psi(Y_{\widehat{L}}^{+}). \tag*{(155)} ℓ ψ ( Y L + ; X ) + 0.5 L ∥ Y L + − X ∥ 2 ≥ ψ ( Y L + ) . (155) Proof. It follows from the fact that X 0 ∈ S ( X 0 ; ψ ) X_0\in\mathcal{S}(X_0;\psi) X 0 ∈ S ( X 0 ; ψ ) , (B3), and the definition of D 0 D_0 D 0 in (142) that ∥ X 0 ∥ ≤ D 0 / 6 \lVert X_0\rVert\le D_0/6 ∥ X 0 ∥ ≤ D 0 /6 . It is easy to see from this fact, (153) , the nonexpansivity of the projection operator defined in (6) , the assumption on L ^ \widehat{L} L in (152), the fact that D 0 ≥ 1 D_0\ge1 D 0 ≥ 1 , and the fact that X ^ ∈ B [ D 0 / 2 ] \widehat{X}\in B[D_0/2] X ∈ B [ D 0 /2 ] that:
∥ Y L ^ + ∥ ≤ ∥ proj Ω ( X ^ − 1 L ^ ∇ ψ ( X ^ ) ) − proj Ω ( X 0 ) + proj Ω ( X 0 ) ∥ ≤ ∥ X ^ − 1 L ^ ∇ ψ ( X ^ ) − X 0 ∥ + ∥ X 0 ∥ ≤ ∥ X ^ − X 0 ∥ + 1 L ^ ∥ ∇ ψ ( X ^ ) ∥ + ∥ X 0 ∥ ≤ 4 D 0 6 + D 0 6 + D 0 6 = D 0 . (156) \begin{aligned}
\lVert Y_{\widehat{L}}^{+}\rVert
&\le\left\lVert\operatorname{proj}_{\Omega}\left(\widehat{X}-\frac{1}{\widehat{L}}\nabla\psi(\widehat{X})\right)-\operatorname{proj}_{\Omega}(X_0)+\operatorname{proj}_{\Omega}(X_0)\right\rVert\\
&\le\left\lVert\widehat{X}-\frac{1}{\widehat{L}}\nabla\psi(\widehat{X})-X_0\right\rVert+\lVert X_0\rVert\\
&\le\lVert\widehat{X}-X_0\rVert+\frac{1}{\widehat{L}}\lVert\nabla\psi(\widehat{X})\rVert+\lVert X_0\rVert\\
&\le\frac{4D_0}{6}+\frac{D_0}{6}+\frac{D_0}{6}=D_0. \tag*{(156)}
\end{aligned} ∥ Y L + ∥ ≤ proj Ω ( X − L 1 ∇ ψ ( X ) ) − proj Ω ( X 0 ) + proj Ω ( X 0 ) ≤ X − L 1 ∇ ψ ( X ) − X 0 + ∥ X 0 ∥ ≤ ∥ X − X 0 ∥ + L 1 ∥ ∇ ψ ( X )∥ + ∥ X 0 ∥ ≤ 6 4 D 0 + 6 D 0 + 6 D 0 = D 0 . (156) Hence, Y L ^ + ∈ B [ D 0 ] Y_{\widehat{L}}^{+}\in B[D_0] Y L + ∈ B [ D 0 ] . Since X ^ \widehat{X} X and Y L ^ + Y_{\widehat{L}}^{+} Y L + belong to B ( D 0 ) B(D_0) B ( D 0 ) , it follows from (148) with X ′ ′ = Y L ^ + X^{\prime\prime}=Y_{\widehat{L}}^{+} X ′′ = Y L + and X ′ = X ^ X^\prime=\widehat{X} X ′ = X that
ψ ( Y L ^ + ) ≤ ℓ ψ ( Y L ^ + ; X ^ ) + ( lip ) 0 2 ∥ Y L ^ + − X ^ ∥ 2 \psi(Y_{\widehat{L}}^{+})\le\ell_{\psi}(Y_{\widehat{L}}^{+};\widehat{X})+\frac{(\operatorname{lip})_0}{2}\lVert Y_{\widehat{L}}^{+}-\widehat{X}\rVert^{2} ψ ( Y L + ) ≤ ℓ ψ ( Y L + ; X ) + 2 ( lip ) 0 ∥ Y L + − X ∥ 2 which in view of the assumption on L ^ \widehat{L} L in (152) implies the second conclusion of the lemma. □
The following lemma shows that the line-search in step 2 of ADAP-FISTA always terminates and provides a lower and upper bound on the sequence of Lipschitz estimates { L i } \{L_i\} { L i } .
Lemma B.4. For any iteration index i ≥ 1 i\ge1 i ≥ 1 generated by ADAP-FISTA, the line-search in step 2 always terminates, the iterates X i − 1 X_{i-1} X i − 1 , Y i − 1 Y_{i-1} Y i − 1 , and X ~ i − 1 \widetilde{X}_{i-1} X i − 1 all lie in B [ D 0 / 2 ] B[D_0/2] B [ D 0 /2 ] , and
L 0 ≤ L i ≤ max { L 0 , 12 G 0 , 2 ( lip ) 0 1 − χ } . (157) L_0\le L_i\le\max\left\{L_0,12G_0,\frac{2(\operatorname{lip})_0}{1-\chi}\right\}. \tag*{(157)} L 0 ≤ L i ≤ max { L 0 , 12 G 0 , 1 − χ 2 ( lip ) 0 } . (157)
Proof. We proceed by induction. We do not include the argument for the i = 1 i=1 i = 1 case since it follows from the same argument as below and by using the fact that X 0 = Y 0 ∈ B [ D 0 / 6 ] X_0=Y_0\in B[D_0/6] X 0 = Y 0 ∈ B [ D 0 /6 ] .
Assume then that the statement holds for an iteration index i ≥ 1 i\ge1 i ≥ 1 generated by ADAP-FISTA. We show that these facts hold also for the ( i + 1 ) (i+1) ( i + 1 ) -st iteration index. Since the method generates the ( i + 1 ) (i+1) ( i + 1 ) -st iteration index, the inequalities checked in step 4 of the i i i -th iteration are satisfied, i.e.,
ψ ( Y i ) ≤ ψ ( X 0 ) , ∥ Y i − X i ∥ ≤ ∥ Y i − X 0 ∥ . (158) \psi(Y_i)\le\psi(X_0),\qquad\lVert Y_i-X_i\rVert\le\lVert Y_i-X_0\rVert. \tag*{(158)} ψ ( Y i ) ≤ ψ ( X 0 ) , ∥ Y i − X i ∥ ≤ ∥ Y i − X 0 ∥ . (158)
Hence, Y i ∈ S ( X 0 ; ψ ) Y_i \in\mathcal{S}(X_0;\psi) Y i ∈ S ( X 0 ; ψ ) and by (140), ∥ Y i ∥ ≤ D 0 / 6 \lVert Y_i\rVert\le D_0/6 ∥ Y i ∥ ≤ D 0 /6 . Using this fact, (155), and ∥ X 0 ∥ ≤ D 0 / 6 \lVert X_0\rVert\le D_0/6 ∥ X 0 ∥ ≤ D 0 /6 , we obtain
∥ X i ∥ ≤ ∥ Y i ∥ + ∥ Y i − X i ∥ ≤ ( 155 ) ∥ Y i ∥ + ∥ Y i − X 0 ∥ ≤ 2 ∥ Y i ∥ + ∥ X 0 ∥ ≤ D 0 2 . (159) \lVert X_i\rVert\le\lVert Y_i\rVert+ \lVert Y_i-X_i\rVert\overset{(155)}{\le} \lVert Y_i\rVert+ \lVert Y_i-X_0\rVert\le2\lVert Y_i\rVert+ \lVert X_0\rVert\le\frac{D_0}{2}. \tag*{(159)} ∥ X i ∥ ≤ ∥ Y i ∥ + ∥ Y i − X i ∥ ≤ ( 155 ) ∥ Y i ∥ + ∥ Y i − X 0 ∥ ≤ 2 ∥ Y i ∥ + ∥ X 0 ∥ ≤ 2 D 0 . (159)
Thus, X i , Y i ∈ B [ D 0 / 2 ] X_i,Y_i \in B[D_0/2] X i , Y i ∈ B [ D 0 /2 ] , and every tentative point X ~ \widetilde{X} X computed in (131) of the ( i + 1 ) (i+1) ( i + 1 ) -st iteration also lies in B [ D 0 / 2 ] B[D_0/2] B [ D 0 /2 ] .
This fact, the way Y Y Y is computed in (132), the last conclusion in Lemma B.3, and the check in (133) imply that if the Lipschitz estimate L L L generated in step 2 of ADAP-FISTA becomes large enough, then the line-search in step 2 of the ( i + 1 ) (i+1) ( i + 1 ) -st iteration must terminate. This fact and (134) imply that a i a_i a i , X ~ i \widetilde{X}_i X i , Y i + 1 Y_{i+1} Y i + 1 , and L i + 1 L_{i+1} L i + 1 are all generated and that X ~ i \widetilde{X}_i X i lies in B [ D 0 / 2 ] B[D_0/2] B [ D 0 /2 ] . We now show that the inequalities in (154) hold for L i + 1 L_{i+1} L i + 1 . It is immediate from the way that L L L is updated in step 2 and the way L i + 1 L_{i+1} L i + 1 is set in (134) that L 0 ≤ L i + 1 L_0 \le L_{i+1} L 0 ≤ L i + 1 . We now show by contradiction that the second relation in (154) holds for L i + 1 L_{i+1} L i + 1 . Suppose then that the line search in step 2 of ADAP-FISTA’s ( i + 1 ) (i+1) ( i + 1 ) -st iteration terminates with L i + 1 L_{i+1} L i + 1 that satisfies
L i + 1 > max { L 0 , 12 G 0 , 2 ( lip ) 0 1 − χ } . (160) L_{i+1}>\max\left\{L_0,12G_0,\frac{2(\operatorname{lip})_0}{1-\chi}\right\}. \tag*{(160)} L i + 1 > max { L 0 , 12 G 0 , 1 − χ 2 ( lip ) 0 } . (160)
It then follows that (133) was not satisfied with L = L i + 1 / 2 L=L_{i+1}/2 L = L i + 1 /2 and the X ~ \widetilde{X} X and Y Y Y in (131) and (132) corresponding to this L L L . It then follows from (157), the fact that X ~ ∈ B [ D 0 / 2 ] \widetilde{X}\in B[D_0/2] X ∈ B [ D 0 /2 ] , and the definition of G 0 G_0 G 0 in (140) that
L = L i + 1 2 > max { 6 G 0 , ( lip ) 0 1 − χ } ≥ ( 140 ) max { 6 ∥ ∇ ψ ( X ~ ) ∥ , ( lip ) 0 1 − χ } . L=\frac{L_{i+1}}{2}>\max\left\{6G_0,\frac{(\operatorname{lip})_0}{1-\chi}\right\}\overset{(140)}{\ge}\max\left\{6\lVert\nabla\psi(\widetilde{X})\rVert,\frac{(\operatorname{lip})_0}{1-\chi}\right\}. L = 2 L i + 1 > max { 6 G 0 , 1 − χ ( lip ) 0 } ≥ ( 140 ) max { 6 ∥ ∇ ψ ( X )∥ , 1 − χ ( lip ) 0 } .
Hence, it follows from this inclusion, the update for Y Y Y in (132), and Lemma B.3 with θ = ( 1 − χ ) \theta=(1-\chi) θ = ( 1 − χ ) , X ^ = X ~ \widehat{X}=\widetilde{X} X = X , L ^ = L \widehat{L}=L L = L , and Y L ^ + = Y Y_{\widehat{L}}^+=Y Y L + = Y that Y ∈ B [ D 0 ] Y\in B[D_0] Y ∈ B [ D 0 ] and
ℓ ψ ( Y ; X ~ ) + ( 1 − χ ) L 2 ∥ Y − X ~ ∥ 2 ≥ ψ ( Y ) . (161) \ell_\psi(Y;\widetilde{X})+\frac{(1-\chi)L}{2}\lVert Y-\widetilde{X}\rVert^2\ge\psi(Y). \tag*{(161)} ℓ ψ ( Y ; X ) + 2 ( 1 − χ ) L ∥ Y − X ∥ 2 ≥ ψ ( Y ) . (161)
which contradicts that (133) was not satisfied with L = L i + 1 / 2 L=L_{i+1}/2 L = L i + 1 /2 , X ~ \widetilde{X} X , and Y Y Y .
Lemma B.5. For any iteration index i ≥ 1 i\ge1 i ≥ 1 where ADAP-FISTA does not fail in its step 4, it holds that
Y i ∈ B [ D 0 ] (162) Y_i\in B[D_0] \tag*{(162)} Y i ∈ B [ D 0 ] (162) V i ∈ ∇ ψ ( Y i ) + N Ω ( Y i ) , ∥ V i ∥ ≤ ζ ∥ Y i − X ~ i − 1 ∥ . (163) V_i\in\nabla\psi(Y_i)+N_\Omega(Y_i),\qquad\lVert V_i\rVert\le\zeta\lVert Y_i-\widetilde{X}_{i-1}\rVert. \tag*{(163)} V i ∈ ∇ ψ ( Y i ) + N Ω ( Y i ) , ∥ V i ∥ ≤ ζ ∥ Y i − X i − 1 ∥ . (163) Proof. Let i ≥ 1 i\ge1 i ≥ 1 be an iteration index where ADAP-FISTA does not fail in its step 4. It follows that the second relation in (136) must hold. This observation and relation (148) imply that Y i ∈ S ( X 0 ; ψ ) Y_i\in\mathcal{S}(X_0;\psi) Y i ∈ S ( X 0 ; ψ ) . It then follows from this fact and (140) that Y i ∈ B [ D 0 / 6 ] ⊆ B [ D 0 ] Y_i\in B[D_0/6]\subseteq B[D_0] Y i ∈ B [ D 0 /6 ] ⊆ B [ D 0 ] , which shows (159).
It follows from the fact that ADAP-FISTA did not fail in its step 4 that it reaches step 5 and V i V_i V i in (138) is computed. It follows from (148) and its associated optimality condition that 0 ∈ ∇ ψ ( X ~ i − 1 ) + N Ω ( Y i ) − L i ( X ~ i − 1 − Y i ) 0\in\nabla\psi(\widetilde{X}_{i-1})+N_\Omega(Y_i)-L_i(\widetilde{X}_{i-1}-Y_i) 0 ∈ ∇ ψ ( X i − 1 ) + N Ω ( Y i ) − L i ( X i − 1 − Y i ) , which in view of the definition of V i V_i V i in (138) implies the inclusion in (160).
Lemma B.4 and (159) imply that X ~ i − 1 \widetilde{X}_{i-1} X i − 1 and Y i Y_i Y i both lie in B [ D 0 ] B[D_0] B [ D 0 ] . It follows from these observations, triangle inequality, the fact that ∇ ψ \nabla\psi ∇ ψ is ( lip ) 0 (\operatorname{lip})_0 ( lip ) 0 -continuous, the bound (154) on L i L_i L i , and the definition of ζ \zeta ζ in (145) that
∥ V i ∥ ≤ ( 138 ) ∥ ∇ ψ ( Y i ) − ∇ ψ ( X ~ i − 1 ) ∥ + L i ∥ X ~ i − 1 − Y i ∥ \lVert V_i\rVert\overset{(138)}{\le}\lVert\nabla\psi(Y_i)-\nabla\psi(\widetilde{X}_{i-1})\rVert+L_i\lVert\widetilde{X}_{i-1}-Y_i\rVert ∥ V i ∥ ≤ ( 138 ) ∥ ∇ ψ ( Y i ) − ∇ ψ ( X i − 1 )∥ + L i ∥ X i − 1 − Y i ∥ ≤ ( lip ) 0 ∥ Y i − X ~ i − 1 ∥ + L i ∥ Y i − X ~ i − 1 ∥ ≤ ( 154 ) ζ ∥ Y i − X ~ i − 1 ∥ \le(\operatorname{lip})_0\lVert Y_i-\widetilde{X}_{i-1}\rVert+L_i\lVert Y_i-\widetilde{X}_{i-1}\rVert\overset{(154)}{\le}\zeta\lVert Y_i-\widetilde{X}_{i-1}\rVert ≤ ( lip ) 0 ∥ Y i − X i − 1 ∥ + L i ∥ Y i − X i − 1 ∥ ≤ ( 154 ) ζ ∥ Y i − X i − 1 ∥ which immediately implies that the inequality in (160) holds.
The following proposition states that ADAP-FISTA terminates with success or failure in a logarithmic number of iterations.
Proposition B.6. ADAP-FISTA always stops (with either success or failure) and does so by performing at most
⌈ ( 1 + Q ) log 2 + ( ζ 2 χ σ 2 ) + 1 ⌉ + ⌈ log 2 + ( max { 6 G 0 L 0 , ( lip ) 0 ( 1 − χ ) L 0 } ) ⌉ (164) \left\lceil(1+Q)\log_2^+\left(\frac{\zeta^2}{\chi\sigma^2}\right)+1\right\rceil+\left\lceil\log_2^+\left(\max\left\{\frac{6G_0}{L_0},\frac{(\operatorname{lip})_0}{(1-\chi)L_0}\right\}\right)\right\rceil \tag*{(164)} ⌈ ( 1 + Q ) log 2 + ( χ σ 2 ζ 2 ) + 1 ⌉ + ⌈ log 2 + ( max { L 0 6 G 0 , ( 1 − χ ) L 0 ( lip ) 0 } ) ⌉ (164) iterations/resolvent evaluations, where D 0 D_0 D 0 , G 0 G_0 G 0 , and ( lip ) 0 (\operatorname{lip})_0 ( lip ) 0 are as in (140) and ζ \zeta ζ and Q Q Q are as in (145)
Proof. Let l l l denote the first quantity in (164) . Using this definition and the inequality log ( 1 + α ) ≥ α / ( 1 + α ) \log(1+\alpha) \ge\alpha/(1+\alpha) log ( 1 + α ) ≥ α / ( 1 + α ) for any α ≥ 0 \alpha\ge0 α ≥ 0 , it is easy to verify that
( 1 + Q − 1 ) 2 ( l − 1 ) ≥ ζ 2 χ σ 2 . (165) (1+Q^{-1})^{2(l-1)} \ge\frac{\zeta^2}{\chi\sigma^2}. \tag*{(165)} ( 1 + Q − 1 ) 2 ( l − 1 ) ≥ χ σ 2 ζ 2 . (165)
We claim that ADAP-FISTA terminates with success or failure in at most l l l iterations. Indeed, it suffices to show that if ADAP-FISTA has not stopped with success or failure before the l l l -th iteration and does not fail during the l l l -th iteration, then it must stop successfully at the l l l -th iteration. So, assume that ADAP-FISTA has not stopped with failure up to (and including) the l l l -th iteration. In view of step 4 of ADAP-FISTA, it follows that (138) holds with i = l i=l i = l .
This observation, inequality in (163) with i = l i=l i = l , the first inequality in (138) with i = l i=l i = l , inequality (152) with i = l i=l i = l , and (165) then imply that
∥ Y l − Y ‾ 0 ∥ 2 ≥ ( 136 ) χ A l L l ∥ Y l − X ~ l − 1 ∥ 2 ≥ ( 160 ) χ ζ 2 A l L l ∥ V l ∥ 2 ≥ ( 150 ) χ ζ 2 ( 1 + Q − 1 ) 2 ( l − 1 ) ∥ V l ∥ 2 ≥ ( 162 ) 1 σ 2 ∥ V l ∥ 2 , (166) \left\lVert Y_l-\overline{Y}_0\right\rVert^2 \mathrel{\overset{(136)}{\ge}} \chi A_lL_l\left\lVert Y_l-\widetilde{X}_{l-1}\right\rVert^2 \mathrel{\overset{(160)}{\ge}} \frac{\chi}{\zeta^2}A_lL_l\left\lVert V_l\right\rVert^2 \mathrel{\overset{(150)}{\ge}} \frac{\chi}{\zeta^2}(1+Q^{-1})^{2(l-1)}\left\lVert V_l\right\rVert^2 \mathrel{\overset{(162)}{\ge}} \frac{1}{\sigma^2}\left\lVert V_l\right\rVert^2, \tag*{(166)} Y l − Y 0 2 ≥ ( 136 ) χ A l L l Y l − X l − 1 2 ≥ ( 160 ) ζ 2 χ A l L l ∥ V l ∥ 2 ≥ ( 150 ) ζ 2 χ ( 1 + Q − 1 ) 2 ( l − 1 ) ∥ V l ∥ 2 ≥ ( 162 ) σ 2 1 ∥ V l ∥ 2 , (166)
and hence that (141) is satisfied. In view of Step 5 of ADAP-FISTA, the method must successfully stop at the end of the l l l -th iteration. We have thus shown that the above claim holds. Moreover, in view of (157) , it follows that the second term in (164) is a bound on the total number of times L i L_i L i is multiplied by 2 and step 2 is repeated. Since exactly one resolvent evaluation occurs every time step 2 is executed, the desired conclusion follows.
Proof of Proposition B.1(a)-(b). (a) It follows from the definition of ζ \zeta ζ in (147) that O ( log + 2 ( ζ 2 ) ) = O ( log + 2 ( ζ ) ) = O ( log + 2 ( max { L 0 , ( lip ) 0 , G 0 } ) ) \mathcal{O}(\log_+^2(\zeta^2))=\mathcal{O}(\log_+^2(\zeta))=\mathcal{O}(\log_+^2(\max\{L_0,(\operatorname{lip})_0,G_0\})) O ( log + 2 ( ζ 2 )) = O ( log + 2 ( ζ )) = O ( log + 2 ( max { L 0 , ( lip ) 0 , G 0 })) . The result immediately follows from this observation, Proposition B.6, and the definitions of Q Q Q and N 0 N_0 N 0 in (147) and (143) , respectively.
(b) It follows immediately from the termination criterion (141) in step 5 of ADAP-FISTA and the inclusion in (163) that (144) must hold if ADAP-FISTA stops successfully.
To see (145) , first observe that since ADAP-FISTA stops successfully, the second relation in (138) held in ADAP-FISTA’s last iteration. Hence, this observation together with relation (150) imply that Y ∈ Ω Y\in\Omega Y ∈ Ω and ψ ( Y ‾ ) ≤ ψ ( Y ‾ 0 ) \psi(\overline{Y})\le\psi(\overline{Y}_0) ψ ( Y ) ≤ ψ ( Y 0 ) . The bound (146) follows immediately from the fact that relation (157) holds for all iteration indices i ≥ 1 i\ge1 i ≥ 1 generated by ADAP-FISTA.
Proof of Proposition B.1(c)
This subsection proves Proposition B.1(c). Throughout, we assume that ψ \psi ψ is μ \mu μ -strongly convex on B [ D 0 ] B[D_0] B [ D 0 ] . The following lemma proves key facts of the i i i -th iteration of ADAP-FISTA.
Lemma B.7. For every iteration i ≥ 1 i\ge1 i ≥ 1 of ADAP-FISTA, the following statements hold:
(a) for every X ∈ Ω ∩ B [ D 0 ] X\in\Omega\cap B[D_0] X ∈ Ω ∩ B [ D 0 ] , we have
( A i − 1 [ ψ ( Y i − 1 ) − ψ ( X ) ] + τ i − 1 2 ∥ X − X i − 1 ∥ 2 ) − ( A i [ ψ ( Y i ) − ψ ( X ) ] + τ i 2 ∥ X − X i ∥ 2 ) ≥ χ A i L i 2 ∥ Y i − X ~ i − 1 ∥ 2 . \left(A_{i-1}[\psi(Y_{i-1})-\psi(X)]+\frac{\tau_{i-1}}{2}\left\lVert X-X_{i-1}\right\rVert^2\right)-\left(A_i[\psi(Y_i)-\psi(X)]+\frac{\tau_i}{2}\left\lVert X-X_i\right\rVert^2\right)\ge\frac{\chi A_iL_i}{2}\left\lVert Y_i-\widetilde{X}_{i-1}\right\rVert^2. ( A i − 1 [ ψ ( Y i − 1 ) − ψ ( X )] + 2 τ i − 1 ∥ X − X i − 1 ∥ 2 ) − ( A i [ ψ ( Y i ) − ψ ( X )] + 2 τ i ∥ X − X i ∥ 2 ) ≥ 2 χ A i L i Y i − X i − 1 2 . (b) for every X ∈ Ω ∩ B [ D 0 ] X\in\Omega\cap B[D_0] X ∈ Ω ∩ B [ D 0 ] , we have
A i [ ψ ( Y i ) − ψ ( X ) ] + τ i 2 ∥ X − X i ∥ 2 ≤ 1 2 ∥ X − Y ‾ 0 ∥ 2 − χ 2 ∑ j = 1 i A j L j ∥ Y j − X ~ j − 1 ∥ 2 . (167) A_i[\psi(Y_i)-\psi(X)]+\frac{\tau_i}{2}\left\lVert X-X_i\right\rVert^2\le\frac{1}{2}\left\lVert X-\overline{Y}_0\right\rVert^2-\frac{\chi}{2}\sum_{j=1}^{i}A_jL_j\left\lVert Y_j-\widetilde{X}_{j-1}\right\rVert^2. \tag*{(167)} A i [ ψ ( Y i ) − ψ ( X )] + 2 τ i ∥ X − X i ∥ 2 ≤ 2 1 X − Y 0 2 − 2 χ j = 1 ∑ i A j L j Y j − X j − 1 2 . (167) (c) the method does not fail at iteration i i i .
Proof. (a) For every iteration i ≥ 1 i\ge1 i ≥ 1 of ADAP-FISTA and X ∈ Ω X\in\Omega X ∈ Ω , define
γ i ( X ) : = ℓ ψ ( Y i , X ~ i − 1 ) + ⟨ ( L i − μ ) ( X ~ i − 1 − Y i ) , X − Y i ⟩ + μ 2 ∥ Y i − X ~ i − 1 ∥ 2 + μ 2 ∥ X − Y i ∥ 2 . (168) \gamma_i(X):=\ell_\psi(Y_i,\widetilde{X}_{i-1})+\left\langle(L_i-\mu)(\widetilde{X}_{i-1}-Y_i),X-Y_i\right\rangle+\frac{\mu}{2}\left\lVert Y_i-\widetilde{X}_{i-1}\right\rVert^2+\frac{\mu}{2}\left\lVert X-Y_i\right\rVert^2. \tag*{(168)} γ i ( X ) := ℓ ψ ( Y i , X i − 1 ) + ⟨ ( L i − μ ) ( X i − 1 − Y i ) , X − Y i ⟩ + 2 μ Y i − X i − 1 2 + 2 μ ∥ X − Y i ∥ 2 . (168)
It is now easy to see from similar arguments as in Lemma A.7 of [53 ] that for every iteration i ≥ 1 i\ge1 i ≥ 1 and X ∈ Ω X\in\Omega X ∈ Ω , we have
A i γ i ( Y i − 1 ) + a i − 1 γ i ( X ) + τ i − 1 2 ∥ X i − 1 − X ∥ 2 − τ i 2 ∥ X i − X ∥ 2 ≥ A i ψ ( Y i ) + χ A i L i 2 ∥ Y i − X ~ i − 1 ∥ 2 . (169) A_i\gamma_i(Y_{i-1})+a_{i-1}\gamma_i(X)+\frac{\tau_{i-1}}{2}\left\lVert X_{i-1}-X\right\rVert^2-\frac{\tau_i}{2}\left\lVert X_i-X\right\rVert^2
\ge A_i\psi(Y_i)+\frac{\chi A_iL_i}{2}\left\lVert Y_i-\widetilde{X}_{i-1}\right\rVert^2. \tag*{(169)} A i γ i ( Y i − 1 ) + a i − 1 γ i ( X ) + 2 τ i − 1 ∥ X i − 1 − X ∥ 2 − 2 τ i ∥ X i − X ∥ 2 ≥ A i ψ ( Y i ) + 2 χ A i L i Y i − X i − 1 2 . (169)
To get an inequality that telescopes, we now show that γ i ( X ) ≤ ψ ( X ) \gamma_i(X) \le\psi(X) γ i ( X ) ≤ ψ ( X ) for all X ∈ Ω ∩ B [ D 0 ] X \in\Omega\cap B[D_0] X ∈ Ω ∩ B [ D 0 ] . To show the claim for every iteration i ≥ 1 i \ge1 i ≥ 1 , define
γ ~ i ( X ) : = ℓ ψ ( X ; X ~ i − 1 ) + μ 2 ∥ X − X ~ i − 1 ∥ 2 . (170) \widetilde{\gamma}_i(X) := \ell_\psi(X;\widetilde{X}_{i-1}) + \frac{\mu}{2}\lVert X-\widetilde{X}_{i-1}\rVert^2. \tag*{(170)} γ i ( X ) := ℓ ψ ( X ; X i − 1 ) + 2 μ ∥ X − X i − 1 ∥ 2 . (170)
Now observe that Lemma B.4 implies that X ~ i − 1 ∈ B [ D 0 / 2 ] ⊆ B [ D 0 ] \widetilde{X}_{i-1} \in B[D_0/2] \subseteq B[D_0] X i − 1 ∈ B [ D 0 /2 ] ⊆ B [ D 0 ] . It follows from this observation and the assumption that ψ \psi ψ is μ \mu μ -strongly convex on B [ D 0 ] B[D_0] B [ D 0 ] that γ ~ i ( X ) ≤ ψ ( X ) \widetilde{\gamma}_i(X) \le\psi(X) γ i ( X ) ≤ ψ ( X ) for all X ∈ B [ D 0 ] X \in B[D_0] X ∈ B [ D 0 ] . The rest of the argument showing that γ i ( X ) ≤ ψ ( X ) \gamma_i(X) \le\psi(X) γ i ( X ) ≤ ψ ( X ) for all X ∈ Ω ∩ B ( D 0 ) X \in\Omega\cap B(D_0) X ∈ Ω ∩ B ( D 0 ) is identical to the one in Lemma A.8 in [53 ] .
Lemma B.4 imply that Y i − 1 ∈ B [ D 0 / 2 ] ∩ Ω ⊆ B [ D 0 ] ∩ Ω Y_{i-1} \in B[D_0/2] \cap\Omega\subseteq B[D_0] \cap\Omega Y i − 1 ∈ B [ D 0 /2 ] ∩ Ω ⊆ B [ D 0 ] ∩ Ω . This observation, the fact that γ i ( X ) ≤ ψ ( X ) \gamma_i(X) \le\psi(X) γ i ( X ) ≤ ψ ( X ) for all X ∈ B [ D 0 ] ∩ Ω X \in B[D_0] \cap\Omega X ∈ B [ D 0 ] ∩ Ω , and (169) imply that
A i − 1 ψ ( Y i − 1 ) + a i − 1 ψ ( X ) + τ i − 1 2 ∥ X i − 1 − X ∥ 2 − τ i 2 ∥ X i − X ∥ 2 ≥ A i ψ ( Y i ) + χ A i L i 2 ∥ Y i − X ~ i − 1 ∥ 2 . (171) A_{i-1}\psi(Y_{i-1})+a_{i-1}\psi(X)+\frac{\tau_{i-1}}{2}\lVert X_{i-1}-X\rVert^2-\frac{\tau_i}{2}\lVert X_i-X\rVert^2
\ge A_i\psi(Y_i)+\frac{\chi A_iL_i}{2}\lVert Y_i-\widetilde{X}_{i-1}\rVert^2. \tag*{(171)} A i − 1 ψ ( Y i − 1 ) + a i − 1 ψ ( X ) + 2 τ i − 1 ∥ X i − 1 − X ∥ 2 − 2 τ i ∥ X i − X ∥ 2 ≥ A i ψ ( Y i ) + 2 χ A i L i ∥ Y i − X i − 1 ∥ 2 . (171)
holds for all X ∈ B [ D 0 ] ∩ Ω X \in B[D_0] \cap\Omega X ∈ B [ D 0 ] ∩ Ω . The relation in part (a) then immediately follows from subtracting A i ψ ( X ) A_i\psi(X) A i ψ ( X ) from both sides of (171) and using the fact that (137) implies A i = A i − 1 + a i − 1 A_i=A_{i-1}+a_{i-1} A i = A i − 1 + a i − 1 .
(b) It follows from summing the inequality of in part (a) from 1 1 1 to i i i , and using the facts that X 0 = Y ˉ 0 X_0=\bar{Y}_0 X 0 = Y ˉ 0 , A 0 = 0 A_0=0 A 0 = 0 and τ 0 = 1 \tau_0=1 τ 0 = 1 that
A i [ ψ ( Y i ) − ψ ( X ) ] + τ i 2 ∥ X − X i ∥ 2 ≤ 1 2 ∥ X − Y ˉ 0 ∥ 2 − χ 2 ∑ j = 1 i A j L j ∥ Y j − X ~ j − 1 ∥ 2 . A_i[\psi(Y_i)-\psi(X)]+\frac{\tau_i}{2}\lVert X-X_i\rVert^2\le\frac{1}{2}\lVert X-\bar{Y}_0\rVert^2-\frac{\chi}{2}\sum_{j=1}^{i}A_jL_j\lVert Y_j-\widetilde{X}_{j-1}\rVert^2. A i [ ψ ( Y i ) − ψ ( X )] + 2 τ i ∥ X − X i ∥ 2 ≤ 2 1 ∥ X − Y ˉ 0 ∥ 2 − 2 χ j = 1 ∑ i A j L j ∥ Y j − X j − 1 ∥ 2 .
which immediately implies that (167) holds.
(c) To show that ADAP-FISTA does not fail at its iteration i i i , we show that all three inequalities checked in (138) and (139) in its step 4 must hold. We first show that ψ ( Y i ) ≤ ψ ( Y ˉ 0 ) \psi(Y_i) \le\psi(\bar{Y}_0) ψ ( Y i ) ≤ ψ ( Y ˉ 0 ) . First observe that it is easy to see that 0 ∈ B [ D 0 / 6 ] ∩ Ω ⊆ B [ D 0 ] ∩ Ω 0 \in B[D_0/6] \cap\Omega\subseteq B[D_0] \cap\Omega 0 ∈ B [ D 0 /6 ] ∩ Ω ⊆ B [ D 0 ] ∩ Ω in view of the definition of D 0 D_0 D 0 in (142) . Thus, using (167) with X = Y ˉ 0 X=\bar{Y}_0 X = Y ˉ 0 and the facts that the sequences τ i \tau_i τ i , A i A_i A i , and L i L_i L i are nonnegative, we see that
A i [ ψ ( Y i ) − ψ ( X 0 ) ] ≤ 1 2 ∥ 0 − Y ˉ 0 ∥ 2 = 0 A_i[\psi(Y_i)-\psi(X_0)]\le\frac{1}{2}\lVert0-\bar{Y}_0\rVert^2=0 A i [ ψ ( Y i ) − ψ ( X 0 )] ≤ 2 1 ∥ 0 − Y ˉ 0 ∥ 2 = 0
Hence, diving both sides by A i A_i A i , we see that ψ ( Y i ) ≤ ψ ( Y ˉ 0 ) \psi(Y_i)\le\psi(\bar{Y}_0) ψ ( Y i ) ≤ ψ ( Y ˉ 0 ) . To conclude the proof, we now show that
∥ Y i − X i ∥ ≤ ∥ Y i − X 0 ∥ , ∥ Y i − X 0 ∥ 2 ≥ χ A i L i ∥ Y i − X ~ i − 1 ∥ 2 . (172) \lVert Y_i-X_i\rVert\le\lVert Y_i-X_0\rVert,\qquad\lVert Y_i-X_0\rVert^2\ge\chi A_iL_i\lVert Y_i-\widetilde{X}_{i-1}\rVert^2. \tag*{(172)} ∥ Y i − X i ∥ ≤ ∥ Y i − X 0 ∥ , ∥ Y i − X 0 ∥ 2 ≥ χ A i L i ∥ Y i − X i − 1 ∥ 2 . (172)
It follows from above conclusion, (150) , and the definition of D 0 D_0 D 0 in (142) that Y i ∈ B [ D 0 ] ∩ Ω Y_i\in B[D_0]\cap\Omega Y i ∈ B [ D 0 ] ∩ Ω . Hence, using (167) with X = Y i X=Y_i X = Y i and the fact that τ i ≥ 1 \tau_i\ge1 τ i ≥ 1 , we see that ∥ Y i − X i ∥ ≤ ∥ Y i − Y ˉ 0 ∥ \lVert Y_i-X_i\rVert\le\lVert Y_i-\bar{Y}_0\rVert ∥ Y i − X i ∥ ≤ ∥ Y i − Y ˉ 0 ∥ . Likewise also using (167) with X = Y i X=Y_i X = Y i but keeping the last term on the right-hand side of (167) , we see that ∥ Y i − Y ˉ 0 ∥ 2 ≥ χ A i L i ∥ Y i − X ~ i − 1 ∥ 2 \lVert Y_i-\bar{Y}_0\rVert^2\ge\chi A_iL_i\lVert Y_i-\widetilde{X}_{i-1}\rVert^2 ∥ Y i − Y ˉ 0 ∥ 2 ≥ χ A i L i ∥ Y i − X i − 1 ∥ 2 . Hence, the three inequalities checked in (138) and (139) must hold at iteration i i i , which implies that ADAP-FISTA does not fail in its i i i -ith iteration.
We are now ready to prove Proposition B.1(c).
Proof of Proposition B.1(c). It follows from Lemma B.7 (c) that, for any iteration i ≥ 1 i\ge1 i ≥ 1 , ADAP-FISTA does not terminate with failure at step 4 when applied to a function ψ \psi ψ that is μ \mu μ -strongly convex on B [ D 0 ] B[D_0] B [ D 0 ] and satisfies assumptions (B1)–(B3). Hence, it follows from Proposition B.1(a)–(b) that ADAP-FISTA must always terminate successfully and return a pair ( Y ˉ , V ˉ ) ∈ Ω × E (\bar{Y},\bar{V})\in\Omega\times\mathbb{E} ( Y ˉ , V ˉ ) ∈ Ω × E satisfying (144) and (145) and a Lipschitz estimate L ˉ \bar{L} L ˉ satisfying (146) .
ADAP-AIPP Method and Complexity
Let E \mathbb{E} E be a finite-dimensional inner product space. This section considers
g ⋆ : = min { g ( U ) : U ∈ Ω } , g_\star:=\min\{g(U):U\in\Omega\}, g ⋆ := min { g ( U ) : U ∈ Ω } ,
under the following assumptions:
(C1) g g g is a real-valued function that is differentiable on E \mathbb{E} E and whose gradient is locally Lipschitz continuous in the sense of (8); (C2) g g g is bounded below on Ω \Omega Ω ;
(C3) Ω \Omega Ω is closed, possibly unbounded, convex set that admits an easily computable projection;
(C4) a point W 0 ∈ Ω W_0 \in\Omega W 0 ∈ Ω is given such that the sublevel set S ( W 0 ; g ) S(W_0;g) S ( W 0 ; g ) has finite diameter D ( W 0 ; g ) D(W_0;g) D ( W 0 ; g ) , where S ( ⋅ ; ⋅ ) S(\cdot;\cdot) S ( ⋅ ; ⋅ ) and D ( ⋅ ; ⋅ ) D(\cdot;\cdot) D ( ⋅ ; ⋅ ) are as in (5) and (6), respectively.
Problem: Given ( g , Ω , W 0 ) (g,\Omega,W_0) ( g , Ω , W 0 ) satisfying (C1)-(C4) and a tolerance ρ ˉ ∈ R + + \bar{\rho} \in\mathbb{R}_{++} ρ ˉ ∈ R ++ , ADAP-AIPP seeks ( W , R ) ∈ Ω × E (W,R) \in\Omega\times\mathbb{E} ( W , R ) ∈ Ω × E such that
R ∈ ∇ g ( W ) + N Ω ( W ) , ∥ R ∥ ≤ ρ ˉ . R \in\nabla g(W) + N_{\Omega}(W), \qquad\lVert R\rVert\le\bar{\rho}. R ∈ ∇ g ( W ) + N Ω ( W ) , ∥ R ∥ ≤ ρ ˉ .
At its j j j -th iteration, calls ADAP-FISTA several times until it successfully finds an approximate stationary point of
min U ∈ Ω { ψ η ( U ) : = η g ( U ) + 1 2 ∥ U − W j − 1 ∥ 2 } (173) \min_{U\in\Omega}\left\{\psi_{\eta}(U):=\eta g(U)+\frac{1}{2}\lVert U-W_{j-1}\rVert^2\right\} \tag*{(173)} U ∈ Ω min { ψ η ( U ) := η g ( U ) + 2 1 ∥ U − W j − 1 ∥ 2 } (173)
where η > 0 \eta>0 η > 0 is a prox stepsize that is halved after an unsuccessful ADAP-FISTA call.
ADAP-AIPP Method
Universal Parameters (used by ADAP-FISTA): σ ∈ ( 0 , 1 / 2 ) \sigma\in(0,1/2) σ ∈ ( 0 , 1/2 ) , χ ∈ ( 0 , 1 ) \chi\in(0,1) χ ∈ ( 0 , 1 ) .
Input: an initial point W 0 ∈ Ω W_0 \in\Omega W 0 ∈ Ω , a function g g g satisfying (C1)-(C4), a set Ω \Omega Ω , prox stepsize η 0 > 0 \eta_0>0 η 0 > 0 , an initial Lipschitz estimate M 0 M_0 M 0 , and tolerance ρ ˉ > 0 \bar{\rho}>0 ρ ˉ > 0 .
Step 0. Set j ← 1 j \gets1 j ← 1 and η ← η 0 \eta\gets\eta_0 η ← η 0 .
Step 1. Call ADAP-FISTA with input
Y ˉ 0 = W j − 1 , μ = 1 2 , L 0 = M j − 1 , ψ ( ⋅ ) = η g ( ⋅ ) + 1 2 ∥ ⋅ − W j − 1 ∥ 2 , Ω = Ω . (174) \bar{Y}_0=W_{j-1}, \qquad\mu=\frac{1}{2}, \qquad L_0=M_{j-1}, \qquad\psi(\cdot)=\eta g(\cdot)+\frac{1}{2}\lVert\cdot-W_{j-1}\rVert^2,\qquad\Omega=\Omega. \tag*{(174)} Y ˉ 0 = W j − 1 , μ = 2 1 , L 0 = M j − 1 , ψ ( ⋅ ) = η g ( ⋅ ) + 2 1 ∥ ⋅ − W j − 1 ∥ 2 , Ω = Ω. (174)
Step 2. If the above ADAP-FISTA call terminates successfully, then let ( W j , V j , M j ) (W_j,V_j,M_j) ( W j , V j , M j ) be its output and set
η j ← η , R j ← V j + W j − 1 − W j η j ; (175) \eta_j\gets\eta,\qquad R_j\gets\frac{V_j+W_{j-1}-W_j}{\eta_j}; \tag*{(175)} η j ← η , R j ← η j V j + W j − 1 − W j ; (175)
else, set η ← η / 2 \eta\gets\eta/2 η ← η /2 and return to Step 1.
Step 3. If ∥ R j ∥ ≤ ρ ˉ \lVert R_j\rVert\le\bar{\rho} ∥ R j ∥ ≤ ρ ˉ , then output ( W ˉ , R ˉ ) ← ( W j , R j ) (\bar{W},\bar{R})\gets(W_j,R_j) ( W ˉ , R ˉ ) ← ( W j , R j ) and stop; else, set j ← j + 1 j\gets j+1 j ← j + 1 and return to Step 1.
We now define several quantities which arise in ADAP-AIPP’s technical analysis and complexity result.
Let
D j : = 6 D ( W j − 1 ; g ) + 1 , D 0 : = D 1 (176) \mathcal{D}_j:=6D(W_{j-1};g)+1,\qquad\mathcal{D}_0:=\mathcal{D}_1 \tag*{(176)} D j := 6 D ( W j − 1 ; g ) + 1 , D 0 := D 1 (176)
where j ≥ 1 j\ge1 j ≥ 1 is an iteration index and D ( ⋅ ; ⋅ ) D(\cdot;\cdot) D ( ⋅ ; ⋅ ) is as in (6). For every j ≥ 0 j\ge0 j ≥ 0 , we define
∇ j : = ∇ ( D j / 2 ; g ) , ( Lip ) j : = [ LIP ] ( D j ; g ) (177) \nabla_j:=\nabla(\mathcal{D}_j/2;g),\qquad(\operatorname{Lip})_j:=[\operatorname{LIP}](\mathcal{D}_j;g) \tag*{(177)} ∇ j := ∇ ( D j /2 ; g ) , ( Lip ) j := [ LIP ] ( D j ; g ) (177)
where ∇ ( ⋅ ; ⋅ ) \nabla(\cdot;\cdot) ∇ ( ⋅ ; ⋅ ) , and [ LIP ] ( ⋅ ; ⋅ ) [\operatorname{LIP}](\cdot;\cdot) [ LIP ] ( ⋅ ; ⋅ ) are as in (10) and (9), respectively. We also let
C σ = 2 ( 1 − σ ) 2 ( 1 − 2 σ ) 2 (178) C_{\sigma}=\frac{2(1-\sigma)^2}{(1-2\sigma)^2} \tag*{(178)} C σ = ( 1 − 2 σ ) 2 2 ( 1 − σ ) 2 (178)
where σ ∈ ( 0 , 1 / 2 ) \sigma\in(0,1/2) σ ∈ ( 0 , 1/2 ) is an universal input to ADAP-AIPP.
The next lemma states useful facts about the sublevel sets of the prox function in (170).
Lemma B.8. Let j ≥ 1 j\ge1 j ≥ 1 be an iteration index generated by ADAP-AIPP and consider ψ η ( ⋅ ) \psi_{\eta}(\cdot) ψ η ( ⋅ ) as in (170), i.e., ψ η ( ⋅ ) : = η g ( ⋅ ) + 0.5 ∥ ⋅ − W j − 1 ∥ 2 \psi_{\eta}(\cdot):=\eta g(\cdot)+0.5\lVert\cdot-W_{j-1}\rVert^2 ψ η ( ⋅ ) := η g ( ⋅ ) + 0.5 ∥ ⋅ − W j − 1 ∥ 2 where η > 0 \eta>0 η > 0 is a prox stepsize. The following facts then hold:
(a) the diameter D ( W j − 1 ; ψ η ) ≤ D ( W j − 1 ; g ) D(W_{j-1};\psi_{\eta})\le D(W_{j-1};g) D ( W j − 1 ; ψ η ) ≤ D ( W j − 1 ; g ) ;
(b) ∇ ψ η ( ⋅ ) \nabla\psi_{\eta}(\cdot) ∇ ψ η ( ⋅ ) is 1 + η ( Lip ) j 1+\eta(\operatorname{Lip})_j 1 + η ( Lip ) j Lipschitz continuous on B [ D j ] B[\mathcal{D}_j] B [ D j ] ;
(c) it holds that
sup { ∥ ∇ ψ η ( U ) ∥ : U ∈ B [ D j / 2 ] } ≤ η ∇ j + D j . (179) \sup\{\lVert\nabla\psi_{\eta}(U)\rVert:U\in B[\mathcal{D}_j/2]\}\le\eta\nabla_j+\mathcal{D}_j. \tag*{(179)} sup {∥ ∇ ψ η ( U )∥ : U ∈ B [ D j /2 ]} ≤ η ∇ j + D j . (179)
Proof. (a) Let U ′ ∈ S ( W j − 1 ; ψ η ) U' \in S(W_{j-1};\psi_\eta) U ′ ∈ S ( W j − 1 ; ψ η ) . Hence, it follows from the definition of S ( ⋅ ; ⋅ ) S(\cdot;\cdot) S ( ⋅ ; ⋅ ) in (7) that ψ η ( U ′ ) ≤ ψ η ( W j − 1 ) \psi_\eta(U') \le\psi_\eta(W_{j-1}) ψ η ( U ′ ) ≤ ψ η ( W j − 1 ) . It then follows from the definition of ψ η ( ⋅ ) \psi_\eta(\cdot) ψ η ( ⋅ ) that η g ( U ′ ) ≤ η g ( U ′ ) + 0.5 ∥ U ′ − W j − 1 ∥ 2 ≤ η g ( W j − 1 ) \eta g(U') \le\eta g(U') + 0.5\lVert U' - W_{j-1}\rVert^2 \le\eta g(W_{j-1}) η g ( U ′ ) ≤ η g ( U ′ ) + 0.5 ∥ U ′ − W j − 1 ∥ 2 ≤ η g ( W j − 1 ) , which implies that U ′ ∈ S ( W j − 1 ; g ) U' \in S(W_{j-1};g) U ′ ∈ S ( W j − 1 ; g ) . Thus, it follows that S ( W j − 1 ; ψ η ) ⊆ S ( W j − 1 ; g ) S(W_{j-1};\psi_\eta) \subseteq S(W_{j-1};g) S ( W j − 1 ; ψ η ) ⊆ S ( W j − 1 ; g ) which implies that D ( W j − 1 ; ψ η ) ≤ D ( W j − 1 ; g ) D(W_{j-1};\psi_\eta) \le D(W_{j-1};g) D ( W j − 1 ; ψ η ) ≤ D ( W j − 1 ; g ) in view of the definition of D ( W j − 1 ; g ) D(W_{j-1};g) D ( W j − 1 ; g ) in (8) .
(b) It is easy to see that
∇ ψ η ( U ) = η ∇ g ( U ) + ( U − W j − 1 ) . (180) \nabla\psi_\eta(U) = \eta\nabla g(U) + (U - W_{j-1}). \tag*{(180)} ∇ ψ η ( U ) = η ∇ g ( U ) + ( U − W j − 1 ) . (180)
The result then follows from (177) and the fact that ∇ g \nabla g ∇ g is ( Lip ) j (\operatorname{Lip})_j ( Lip ) j -Lipschitz continuous on B [ D j ] B[\mathcal{D}_j] B [ D j ] .
(c) It is immediate from (177) and the definition of ∇ j \nabla_j ∇ j in (174) that
sup { ∥ ∇ ψ η ( U ) ∥ : U ∈ B [ D j / 2 ] } ≤ η ∇ j + sup U ∈ B [ D j / 2 ] ∥ U ∥ + ∥ W j − 1 ∥ (181) \sup\{\lVert\nabla\psi_\eta(U)\rVert: U \in B[\mathcal{D}_j/2]\} \le\eta\nabla_j + \sup_{U\in B[\mathcal{D}_j/2]}\lVert U\rVert+ \lVert W_{j-1}\rVert \tag*{(181)} sup {∥ ∇ ψ η ( U )∥ : U ∈ B [ D j /2 ]} ≤ η ∇ j + U ∈ B [ D j /2 ] sup ∥ U ∥ + ∥ W j − 1 ∥ (181)
≤ η ∇ j + D j 2 + D j 6 ≤ η ∇ j + D j (182) \le\eta\nabla_j + \frac{\mathcal{D}_j}{2} + \frac{\mathcal{D}_j}{6} \le\eta\nabla_j + \mathcal{D}_j \tag*{(182)} ≤ η ∇ j + 2 D j + 6 D j ≤ η ∇ j + D j (182)
where the second inequality follows from the definition of D j \mathcal{D}_j D j in (173) and the fact that W j − 1 ∈ S ( W j − 1 ; g ) W_{j-1} \in S(W_{j-1};g) W j − 1 ∈ S ( W j − 1 ; g ) .
Lemma B.9
Lemma B.9. Let j ≥ 1 j \ge1 j ≥ 1 be an iteration index of ADAP-AIPP , and define
Θ j : = max { M j − 1 , η 0 ∇ j + D j , 1 + η 0 ( Lip ) j } . (183) \Theta_j := \max\{M_{j-1},\eta_0\nabla_j+\mathcal{D}_j,1+\eta_0(\operatorname{Lip})_j\}. \tag*{(183)} Θ j := max { M j − 1 , η 0 ∇ j + D j , 1 + η 0 ( Lip ) j } . (183)
Then the following statements hold:
(a) the loop in steps 1 and 2 of the j j j -th iteration is repeated at most a logarithmic number of times before terminating with a triple ( W j , V j , M j ) ∈ Ω × E × R + + (W_j,V_j,M_j) \in\Omega\times E \times\mathbb{R}_{++} ( W j , V j , M j ) ∈ Ω × E × R ++ and η j \eta_j η j satisfying
η 0 ≥ η j ≥ min { η 0 , 1 / ( 4 ( Lip ) j ) } , R j ∈ ∇ g ( W j ) + N Ω ( W j ) , (184) \eta_0 \ge\eta_j \ge\min\{\eta_0,1/(4(\operatorname{Lip})_j)\},\qquad R_j \in\nabla g(W_j)+N_\Omega(W_j), \tag*{(184)} η 0 ≥ η j ≥ min { η 0 , 1/ ( 4 ( Lip ) j )} , R j ∈ ∇ g ( W j ) + N Ω ( W j ) , (184)
∥ V j ∥ ≤ σ ∥ W j − W j − 1 ∥ , η j ( g ( W j − 1 ) − g ( W j ) ) ≥ 1 2 ∥ W j − W j − 1 ∥ 2 , (185) \lVert V_j\rVert\le\sigma\lVert W_j-W_{j-1}\rVert,\qquad\eta_j\bigl(g(W_{j-1})-g(W_j)\bigr)\ge\frac{1}{2}\lVert W_j-W_{j-1}\rVert^2, \tag*{(185)} ∥ V j ∥ ≤ σ ∥ W j − W j − 1 ∥ , η j ( g ( W j − 1 ) − g ( W j ) ) ≥ 2 1 ∥ W j − W j − 1 ∥ 2 , (185)
and
M j − 1 ≤ M j ≤ max { M j − 1 , 12 ( η j ∇ j + D j ) , 2 ( 1 + η j ( Lip ) j ) 1 − χ } ; (186) M_{j-1}\le M_j\le\max\left\{M_{j-1},12(\eta_j\nabla_j+\mathcal{D}_j),\frac{2(1+\eta_j(\operatorname{Lip})_j)}{1-\chi}\right\}; \tag*{(186)} M j − 1 ≤ M j ≤ max { M j − 1 , 12 ( η j ∇ j + D j ) , 1 − χ 2 ( 1 + η j ( Lip ) j ) } ; (186)
(b) every ADAP-FISTA call made in step 1 within this loop terminates with either success or failure, in at most
O 1 ( Θ j log 2 + ( Θ j ) + log 2 + ( Θ j M j − 1 ) ) (187) \mathcal{O}_1\left(\sqrt{\Theta_j}\log_2^+(\Theta_j)+\log_2^+\left(\frac{\Theta_j}{M_{j-1}}\right)\right) \tag*{(187)} O 1 ( Θ j log 2 + ( Θ j ) + log 2 + ( M j − 1 Θ j ) ) (187)
ACG iterations/resolvent evaluations;
(c) the inequality holds:
g ( W j − 1 ) − g ( W j ) ≥ η j C σ ∥ R j ∥ 2 . (188) g(W_{j-1})-g(W_j)\ge\frac{\eta_j}{C_\sigma}\lVert R_j\rVert^2. \tag*{(188)} g ( W j − 1 ) − g ( W j ) ≥ C σ η j ∥ R j ∥ 2 . (188)
where C σ C_\sigma C σ is as in (175).
Proof. (a) Let j ≥ 1 j \ge1 j ≥ 1 be an iteration index of ADAP-AIPP and consider its j j j -th iteration. It follows from step 1 and Lemma B.8 (b) that ADAP-AIPP calls ADAP-FISTA with a function ψ ( ⋅ ) = η g ( ⋅ ) + 1 2 ∥ ⋅ − W j − 1 ∥ 2 \psi(\cdot)=\eta g(\cdot)+\frac{1}{2}\lVert\cdot-W_{j-1}\rVert^2 ψ ( ⋅ ) = η g ( ⋅ ) + 2 1 ∥ ⋅ − W j − 1 ∥ 2 whose gradient is 1 + η ( Lip ) j 1+\eta(\operatorname{Lip})_j 1 + η ( Lip ) j Lipschitz continuous on B [ D j ] B[\mathcal{D}_j] B [ D j ] . Thus, this function is ( 1 − η ( Lip ) j ) (1-\eta(\operatorname{Lip})_j) ( 1 − η ( Lip ) j ) -strongly convex on B [ D j ] B[\mathcal{D}_j] B [ D j ] . Thus, it follows from this observation, the fact that ADAP-FISTA is called with μ = 1 / 2 \mu=1/2 μ = 1/2 , and Proposition B.1 (c) that if ADAP-FISTA is called with ψ ( ⋅ ) = η g ( ⋅ ) + 1 2 ∥ ⋅ − W j − 1 ∥ 2 \psi(\cdot)=\eta g(\cdot)+\frac{1}{2}\lVert\cdot-W_{j-1}\rVert^2 ψ ( ⋅ ) = η g ( ⋅ ) + 2 1 ∥ ⋅ − W j − 1 ∥ 2 where η ≤ 1 / ( 2 ( Lip ) j ) \eta\le1/(2(\operatorname{Lip})_j) η ≤ 1/ ( 2 ( Lip ) j ) , then the call to ADAP-FISTA must terminate successfully. The first conclusion of part (a) and the first relation in (181) then immediately follow from this observation, the fact that η \eta η is halved after every unsuccessful ADAP-FISTA call, and the way η j \eta_j η j is set in (172). It then follows from Proposition B.1 (b), the fact that ADAP-AIPP calls ADAP-FISTA with input as in (171), and the way η j \eta_j η j is set in (172) that ADAP-FISTA terminates with a triple ( W j , V j , M j ) (W_j,V_j,M_j) ( W j , V j , M j ) and η j \eta_j η j satisfying
V j ∈ η j ∇ g ( W j ) + ( W j − W j − 1 ) + N Ω ( W j ) , ∥ V j ∥ ≤ σ ∥ W j − W j − 1 ∥ (189) V_j\in\eta_j\nabla g(W_j)+(W_j-W_{j-1})+N_\Omega(W_j),\qquad\lVert V_j\rVert\le\sigma\lVert W_j-W_{j-1}\rVert \tag*{(189)} V j ∈ η j ∇ g ( W j ) + ( W j − W j − 1 ) + N Ω ( W j ) , ∥ V j ∥ ≤ σ ∥ W j − W j − 1 ∥ (189)
η j ( g ( W j − 1 ) − g ( W j ) ) ≥ 1 2 ∥ W j − W j − 1 ∥ 2 (190) \eta_j\bigl(g(W_{j-1})-g(W_j)\bigr)\ge\frac{1}{2}\lVert W_j-W_{j-1}\rVert^2 \tag*{(190)} η j ( g ( W j − 1 ) − g ( W j ) ) ≥ 2 1 ∥ W j − W j − 1 ∥ 2 (190)
which immediately implies both relations in (182). The second relation in (181) is immediate from the inclusion in (186) and the definition of R j R_j R j in (172). Relation (183) follows from the fact that ADAP-FISTA is called with input as in (171), relation (144), Lemma B.8(a)-(c), and the way that η j \eta_j η j is set in (172).
(b) Let j ≥ 1 j \ge1 j ≥ 1 and consider ψ η : = η g ( ⋅ ) + 0.5 ∥ ⋅ − W j − 1 ∥ 2 \psi_\eta:= \eta g(\cdot) + 0.5\lVert\cdot- W_{j-1}\rVert^2 ψ η := η g ( ⋅ ) + 0.5 ∥ ⋅ − W j − 1 ∥ 2 where η > 0 \eta> 0 η > 0 . It follows from Lemma B.8(a) and the definition of D j D_j D j in (173) that D ( W j − 1 ; ψ η ) ≤ D ( W j − 1 ; g ) ≤ D j D(W_{j-1};\psi_\eta) \le D(W_{j-1};g) \le D_j D ( W j − 1 ; ψ η ) ≤ D ( W j − 1 ; g ) ≤ D j . This relation together with (176) then imply that
sup { ∥ ∇ ψ η ( U ) ∥ : U ∈ B [ D ( W j − 1 ; ψ η ) / 2 ] } ≤ sup { ∥ ∇ ψ η ( U ) ∥ : U ∈ B [ D j / 2 ] } ≤ η ∇ j + D j . \sup\{\lVert\nabla\psi_\eta(U)\rVert: U \in B[D(W_{j-1};\psi_\eta)/2]\} \le\sup\{\lVert\nabla\psi_\eta(U)\rVert: U \in B[D_j/2]\} \le\eta\nabla_j + D_j. sup {∥ ∇ ψ η ( U )∥ : U ∈ B [ D ( W j − 1 ; ψ η ) /2 ]} ≤ sup {∥ ∇ ψ η ( U )∥ : U ∈ B [ D j /2 ]} ≤ η ∇ j + D j .
It then follows from these conclusions, Proposition B.1(a), the fact that ADAP-FISTA is called with input as in (171), the fact that the way η \eta η is updated in step 2 implies that η ≤ η 0 \eta\le\eta_0 η ≤ η 0 for all η \eta η generated by ADAP-AIPP, and the definition of Θ j \Theta_j Θ j in (180) that every ADAP-FISTA call within the loop in steps 1 and 2 performs at most (184) ACG iterations/resolvent evaluations.
(c) Let σ ~ = σ / ( 1 − σ ) \tilde{\sigma} = \sigma/(1-\sigma) σ ~ = σ / ( 1 − σ ) where σ ∈ ( 0 , 1 / 2 ) \sigma\in(0,1/2) σ ∈ ( 0 , 1/2 ) is an universal input to ADAP-AIPP. It follows from the first relation in (182) and the definition of R j R_j R j in (172) that ∥ V j ∥ ≤ σ ∥ W j − W j − 1 ∥ ≤ σ ∥ V j ∥ + σ ∥ η j R j ∥ \lVert V_j\rVert\le\sigma\lVert W_j-W_{j-1}\rVert\le\sigma\lVert V_j\rVert+ \sigma\lVert\eta_j R_j\rVert ∥ V j ∥ ≤ σ ∥ W j − W j − 1 ∥ ≤ σ ∥ V j ∥ + σ ∥ η j R j ∥ . This relation together with the definition of σ ~ \tilde{\sigma} σ ~ then imply that
∥ V j η j ∥ ≤ σ ~ ∥ R j ∥ . (191) \left\lVert\frac{V_j}{\eta_j}\right\rVert\le\tilde{\sigma}\lVert R_j\rVert. \tag*{(191)} η j V j ≤ σ ~ ∥ R j ∥ . (191)
It then follows from the definition of R j R_j R j in (172), relation (188), and multiplying both sides of the second inequality in (182) by 2 / η j 2 2/\eta_j^2 2/ η j 2 that
2 η j 2 [ g ( W j − 1 ) − g ( W j ) ] ≥ ( 182 ) ∥ W j − 1 − W j η j ∥ 2 = ( 172 ) ∥ R j − V j η j ∥ 2 ≥ ( 188 ) min ∥ z ∥ ≤ σ ~ ∥ R j ∥ ∥ R j − z ∥ 2 = min 0 ≤ t ≤ σ ~ ∥ R j ∥ ( ∥ R j ∥ − t ) 2 = ( 1 − σ ~ ) 2 ∥ R j ∥ 2 . (192) \frac{2}{\eta_j^2}[g(W_{j-1})-g(W_j)] \mathrel{\geq}^{(182)} \left\lVert\frac{W_{j-1}-W_j}{\eta_j}\right\rVert^2 \mathrel{=}^{(172)} \left\lVert R_j-\frac{V_j}{\eta_j}\right\rVert^2
\mathrel{\geq}^{(188)} \min_{\lVert z\rVert\leq\tilde{\sigma}\lVert R_j\rVert}\lVert R_j-z\rVert^2
= \min_{0\leq t\leq\tilde{\sigma}\lVert R_j\rVert}(\lVert R_j\rVert-t)^2
= (1-\tilde{\sigma})^2\lVert R_j\rVert^2. \tag*{(192)} η j 2 2 [ g ( W j − 1 ) − g ( W j )] ≥ ( 182 ) η j W j − 1 − W j 2 = ( 172 ) R j − η j V j 2 ≥ ( 188 ) ∥ z ∥ ≤ σ ~ ∥ R j ∥ min ∥ R j − z ∥ 2 = 0 ≤ t ≤ σ ~ ∥ R j ∥ min (∥ R j ∥ − t ) 2 = ( 1 − σ ~ ) 2 ∥ R j ∥ 2 . (192)
Relation (185) then follows from multiplying both sides of (189) by η j / 2 \eta_j/2 η j /2 , the fact that σ ~ = σ / ( 1 − σ ) \tilde{\sigma}=\sigma/(1-\sigma) σ ~ = σ / ( 1 − σ ) , and the definition of C σ C_\sigma C σ in (175).
Lemma B.10. The sequence { D j } \{D_j\} { D j } is nonincreasing, i.e., for any iteration index j ≥ 1 j \ge1 j ≥ 1 , it holds that
D j ≤ D j − 1 ≤ D 0 (193) D_j \le D_{j-1} \le D_0 \tag*{(193)} D j ≤ D j − 1 ≤ D 0 (193) where recall D j D_j D j is as in (173).
Proof. Let j ≥ 1 j \ge1 j ≥ 1 be an iteration index of ADAP-AIPP and let U ′ ∈ S ( W j ; g ) U' \in S(W_j;g) U ′ ∈ S ( W j ; g ) where S ( ⋅ ; ⋅ ) S(\cdot;\cdot) S ( ⋅ ; ⋅ ) is as in (5). It follows from this inclusion, (185), and the facts that η j \eta_j η j , C σ C_\sigma C σ , and ∥ R j ∥ 2 \lVert R_j\rVert^2 ∥ R j ∥ 2 are all nonnegative quantities that
g ( U ′ ) ≤ g ( W j ) ≤ ( 185 ) g ( W j − 1 ) − η j C σ ∥ R j ∥ 2 ≤ ( 175 ) g ( W j − 1 ) . g(U') \le g(W_j) \mathrel{\leq}^{(185)} g(W_{j-1})-\frac{\eta_j}{C_\sigma}\lVert R_j\rVert^2 \mathrel{\leq}^{(175)} g(W_{j-1}). g ( U ′ ) ≤ g ( W j ) ≤ ( 185 ) g ( W j − 1 ) − C σ η j ∥ R j ∥ 2 ≤ ( 175 ) g ( W j − 1 ) .
Hence, it follows from the above relation that U ′ ∈ S ( W j − 1 ; g ) U' \in S(W_{j-1};g) U ′ ∈ S ( W j − 1 ; g ) and thus S ( W j ; g ) ⊆ S ( W j − 1 ; g ) S(W_j;g) \subseteq S(W_{j-1};g) S ( W j ; g ) ⊆ S ( W j − 1 ; g ) . It then follows from the definition of D ( ⋅ ; ⋅ ) D(\cdot;\cdot) D ( ⋅ ; ⋅ ) in (6) that D ( W j ; g ) ≤ D ( W j − 1 ; g ) D(W_j;g) \le D(W_{j-1};g) D ( W j ; g ) ≤ D ( W j − 1 ; g ) . The first inequality in (190) then follows from this relation and the definition of D j D_j D j in (173). The second inequality in (190) then follows from the first inequality in (190), a simple induction argument, and the fact that D 1 = D 0 D_1=D_0 D 1 = D 0 . □
Lemma B.11. For every iteration index j ≥ 1 j \ge1 j ≥ 1 , it holds that
M 0 ≤ M j ≤ max { M 0 , 12 ( η 0 ∇ 0 + D 0 ) , 2 ( 1 + η 0 ( Lip ) 0 ) 1 − χ } . (194) M_0 \le M_j \le\max\left\{M_0,12(\eta_0\nabla_0+D_0),\frac{2(1+\eta_0(\operatorname{Lip})_0)}{1-\chi}\right\}. \tag*{(194)} M 0 ≤ M j ≤ max { M 0 , 12 ( η 0 ∇ 0 + D 0 ) , 1 − χ 2 ( 1 + η 0 ( Lip ) 0 ) } . (194)
Proof. The first inequality in (191) is immediate from the first inequality in (183) together with a simple induction argument. To show the second inequality in (191), we also proceed by induction. We do not include the argument for the j = 1 j=1 j = 1 case since it follows from a similar argument as below.
Suppose then that the second inequality in (191) holds for an iteration index j ≥ 1 j \ge1 j ≥ 1 generated by ADAP-AIPP. We show that it also must hold for the ( j + 1 ) (j+1) ( j + 1 ) -st iteration index. It follows from the second inequality in (183) with j = j + 1 j = j + 1 j = j + 1 , the bounds on D j + 1 \mathcal{D}_{j+1} D j + 1 and η j + 1 \eta_{j+1} η j + 1 in (193) and (184) , respectively, and the definitions of ∇ j \nabla_j ∇ j and ( Lip ) j (\operatorname{Lip})_j ( Lip ) j in (177) that
M j + 1 ≤ ( 183 ) max { M j , 12 ( η j + 1 ∇ j + 1 + D j + 1 ) , 2 ( 1 + η j + 1 ( Lip ) j + 1 ) 1 − χ } ≤ ( 181 ) max { M j , 12 ( η 0 ∇ j + 1 + D j + 1 ) , 2 ( 1 + η 0 ( Lip ) j + 1 ) 1 − χ } ≤ ( 190 ) max { M j , 12 ( η 0 ∇ 0 + D 0 ) , 2 ( 1 + η 0 ( Lip ) 0 ) 1 − χ } . (195) \begin{aligned}
M_{j+1} &\mathrel{\leq}^{(183)} \max\left\{M_j,12(\eta_{j+1}\nabla_{j+1}+\mathcal{D}_{j+1}),\frac{2(1+\eta_{j+1}(\operatorname{Lip})_{j+1})}{1-\chi}\right\}\\
&\mathrel{\leq}^{(181)} \max\left\{M_j,12(\eta_0\nabla_{j+1}+\mathcal{D}_{j+1}),\frac{2(1+\eta_0(\operatorname{Lip})_{j+1})}{1-\chi}\right\}\\
&\mathrel{\leq}^{(190)} \max\left\{M_j,12(\eta_0\nabla_0+\mathcal{D}_0),\frac{2(1+\eta_0(\operatorname{Lip})_0)}{1-\chi}\right\}. \tag*{(195)}
\end{aligned} M j + 1 ≤ ( 183 ) max { M j , 12 ( η j + 1 ∇ j + 1 + D j + 1 ) , 1 − χ 2 ( 1 + η j + 1 ( Lip ) j + 1 ) } ≤ ( 181 ) max { M j , 12 ( η 0 ∇ j + 1 + D j + 1 ) , 1 − χ 2 ( 1 + η 0 ( Lip ) j + 1 ) } ≤ ( 190 ) max { M j , 12 ( η 0 ∇ 0 + D 0 ) , 1 − χ 2 ( 1 + η 0 ( Lip ) 0 ) } . (195)
The fact that (194) holds for all iteration indices j ≥ 1 j \geq1 j ≥ 1 then follows immediately from (195) and the induction hypothesis.
The following lemma uniformly bounds the quantity Θ j \Theta_j Θ j in (183) . This is useful for uniformly bounding the number of gradient evaluations made during each ADAP-FISTA call in step 1.
Lemma B.12. Let
Θ ˉ : = max { M 0 , η 0 ∇ 0 + D 0 , 1 + η 0 ( Lip ) 0 } . (196) \bar{\Theta}:=\max\left\{M_0,\eta_0\nabla_0+\mathcal{D}_0,1+\eta_0(\operatorname{Lip})_0\right\}. \tag*{(196)} Θ ˉ := max { M 0 , η 0 ∇ 0 + D 0 , 1 + η 0 ( Lip ) 0 } . (196) For every iteration index j ≥ 1 j \geq1 j ≥ 1 , it holds that Θ j = O ( Θ ˉ ) \Theta_j=O(\bar{\Theta}) Θ j = O ( Θ ˉ ) .
Proof. Let j ≥ 1 j \geq1 j ≥ 1 be an iteration index generated by ADAP-FISTA. It follows from (193) and the definitions of ( Lip ) j (\operatorname{Lip})_j ( Lip ) j and ∇ j \nabla_j ∇ j in (177) that D j ≤ D 0 \mathcal{D}_j \leq\mathcal{D}_0 D j ≤ D 0 , ( Lip ) j ≤ ( Lip ) 0 (\operatorname{Lip})_j \leq(\operatorname{Lip})_0 ( Lip ) j ≤ ( Lip ) 0 , and ∇ j ≤ ∇ 0 \nabla_j \leq\nabla_0 ∇ j ≤ ∇ 0 . It is then immediate from this bound, the bound on M j M_j M j in (194) , the fact that χ ∈ ( 0 , 1 ) \chi\in(0,1) χ ∈ ( 0 , 1 ) , and (196) that
Θ j = ( 180 ) max { M j − 1 , η 0 ∇ j + D j , 1 + η 0 ( Lip ) j } \Theta_j \mathrel{=}^{(180)} \max\left\{M_{j-1},\eta_0\nabla_j+\mathcal{D}_j,1+\eta_0(\operatorname{Lip})_j\right\} Θ j = ( 180 ) max { M j − 1 , η 0 ∇ j + D j , 1 + η 0 ( Lip ) j }
≤ ( 190 ) max { M j − 1 , η 0 ∇ 0 + D 0 , 1 + η 0 ( Lip ) 0 } (197) \mathrel{\leq}^{(190)} \max\left\{M_{j-1},\eta_0\nabla_0+\mathcal{D}_0,1+\eta_0(\operatorname{Lip})_0\right\} \tag*{(197)} ≤ ( 190 ) max { M j − 1 , η 0 ∇ 0 + D 0 , 1 + η 0 ( Lip ) 0 } (197)
≤ ( 191 ) max { M 0 , 12 ( η 0 ∇ 0 + D 0 ) , 2 ( 1 + η 0 ( Lip ) 0 ) 1 − χ } = ( 193 ) O ( Θ ˉ ) . (198) \mathrel{\leq}^{(191)} \max\left\{M_0,12(\eta_0\nabla_0+\mathcal{D}_0),\frac{2(1+\eta_0(\operatorname{Lip})_0)}{1-\chi}\right\}\mathrel{=}^{(193)}O(\bar{\Theta}). \tag*{(198)} ≤ ( 191 ) max { M 0 , 12 ( η 0 ∇ 0 + D 0 ) , 1 − χ 2 ( 1 + η 0 ( Lip ) 0 ) } = ( 193 ) O ( Θ ˉ ) . (198)
which immediately implies the result of the lemma.
The following proposition now states the main complexity results of ADAP-AIPP.
Proposition B.13. The following statements about ADAP-AIPP hold:
(a) it terminates with a pair ( W ˉ , R ˉ ) ∈ Ω × E (\bar{W},\bar{R})\in\Omega\times\mathsf{E} ( W ˉ , R ˉ ) ∈ Ω × E satisfying
R ˉ ∈ ∇ g ( W ˉ ) + N Ω ( W ˉ ) , ∥ R ˉ ∥ ≤ ρ ˉ (199) \bar{R}\in\nabla g(\bar{W})+N_{\Omega}(\bar{W}),\qquad\lVert\bar{R}\rVert\leq\bar{\rho} \tag*{(199)} R ˉ ∈ ∇ g ( W ˉ ) + N Ω ( W ˉ ) , ∥ R ˉ ∥ ≤ ρ ˉ (199) g ( W ˉ ) ≤ g ( W 0 ) ; (200) g(\bar{W})\leq g(W_0); \tag*{(200)} g ( W ˉ ) ≤ g ( W 0 ) ; (200) (b) if ℓ \ell ℓ denotes its final iteration index, then its total number of ADAP-AIPP iterations/ADAP-FISTA calls is bounded by
1 + C σ max { η 0 − 1 , 4 ( Lip ) 0 } ρ ˉ 2 [ g ( W 0 ) − g ( W ℓ ) ] + [ log 2 + ( 4 η 0 ( Lip ) 0 ) ] (201) 1+\frac{C_{\sigma}\max\left\{\eta_0^{-1},4(\operatorname{Lip})_0\right\}}{\bar{\rho}^{2}}\left[g(W_0)-g(W_{\ell})\right]+\left[\log_2^{+}(4\eta_0(\operatorname{Lip})_0)\right] \tag*{(201)} 1 + ρ ˉ 2 C σ max { η 0 − 1 , 4 ( Lip ) 0 } [ g ( W 0 ) − g ( W ℓ ) ] + [ log 2 + ( 4 η 0 ( Lip ) 0 ) ] (201) where ( Lip ) 0 (\operatorname{Lip})_0 ( Lip ) 0 and C σ C_{\sigma} C σ are as in (177) and (178) , respectively, and η 0 \eta_0 η 0 , ρ ˉ \bar{\rho} ρ ˉ , and W 0 W_0 W 0 are inputs to ADAP-AIPP.
Proof. (a) We first show that ADAP-AIPP must terminate and that its final iteration index ℓ ≥ 1 \ell\geq1 ℓ ≥ 1 satisfies
ℓ ≤ 1 + C σ max { η 0 − 1 , 4 ( Lip ) 0 } ρ ˉ 2 [ g ( W 0 ) − g ( W ℓ ) ] . (202) \ell\leq1+\frac{C_{\sigma}\max\left\{\eta_0^{-1},4(\operatorname{Lip})_0\right\}}{\bar{\rho}^{2}}\left[g(W_0)-g(W_{\ell})\right]. \tag*{(202)} ℓ ≤ 1 + ρ ˉ 2 C σ max { η 0 − 1 , 4 ( Lip ) 0 } [ g ( W 0 ) − g ( W ℓ ) ] . (202)
Suppose by contradiction that ADAP-AIPP generates some index j ˉ \bar{j} j ˉ such that
j ˉ > 1 + C σ max { η 0 − 1 , 4 ( Lip ) 0 } ρ ˉ 2 [ g ( W 0 ) − g ( W j ˉ ) ] . (203) \bar{j}>1+\frac{C_{\sigma}\max\left\{\eta_0^{-1},4(\operatorname{Lip})_0\right\}}{\bar{\rho}^{2}}\left[g(W_0)-g(W_{\bar{j}})\right]. \tag*{(203)} j ˉ > 1 + ρ ˉ 2 C σ max { η 0 − 1 , 4 ( Lip ) 0 } [ g ( W 0 ) − g ( W j ˉ ) ] . (203)
As ADAP-AIPP generates j ˉ \bar{j} j ˉ as an iteration index, it does not terminate at the ( j ˉ − 1 ) (\bar{j}-1) ( j ˉ − 1 ) -th iteration. In view of step 3 of ADAP-AIPP, this implies that ∥ R j ∥ > ρ ˉ \lVert R_j\rVert> \bar{\rho} ∥ R j ∥ > ρ ˉ for every j = 1 , … , j ˉ − 1 j=1,\ldots,\bar{j}-1 j = 1 , … , j ˉ − 1 . Using this conclusion, summing inequality (188) from 1 to j ˉ \bar{j} j ˉ , and using the lower bound on η j \eta_j η j in (184) , we conclude that
( j ˉ − 1 ) ρ ˉ 2 ≤ ∑ j = 1 j ˉ − 1 ∥ R j ∥ 2 ≤ ∑ j = 1 j ˉ ∥ R j ∥ 2 ≤ ( 185 ) C σ ∑ j = 1 j ˉ g ( W j − 1 ) − g ( W j ) η j ≤ ( 181 ) C σ [ min { η 0 , 1 / ( 4 ( Lip ) 0 ) } ] [ g ( W 0 ) − g ( W j ˉ ) ] (\bar{j}-1)\bar{\rho}^{2} \le\sum_{j=1}^{\bar{j}-1}\lVert R_j\rVert^{2} \le\sum_{j=1}^{\bar{j}}\lVert R_j\rVert^{2} \mathrel{\overset{(185)}{\le}} C_{\sigma}\sum_{j=1}^{\bar{j}}\frac{g(W_{j-1})-g(W_j)}{\eta_j} \mathrel{\overset{(181)}{\le}} \frac{C_{\sigma}}{\left[\min\{\eta_0,1/(4(\operatorname{Lip})_0)\}\right]}[g(W_0)-g(W_{\bar{j}})] ( j ˉ − 1 ) ρ ˉ 2 ≤ j = 1 ∑ j ˉ − 1 ∥ R j ∥ 2 ≤ j = 1 ∑ j ˉ ∥ R j ∥ 2 ≤ ( 185 ) C σ j = 1 ∑ j ˉ η j g ( W j − 1 ) − g ( W j ) ≤ ( 181 ) [ min { η 0 , 1/ ( 4 ( Lip ) 0 )} ] C σ [ g ( W 0 ) − g ( W j ˉ )]
which contradicts (203) . Hence, ADAP-AIPP only generates iteration indices j ≥ 1 j\ge1 j ≥ 1 satisfying
j ≤ 1 + C σ max { η 0 − 1 , 4 ( Lip ) 0 } ρ ˉ 2 [ g ( W 0 ) − g ( W j ) ] . (204) j\le1+\frac{C_{\sigma}\max\{\eta_0^{-1},4(\operatorname{Lip})_0\}}{\bar{\rho}^{2}}[g(W_0)-g(W_j)]. \tag*{(204)} j ≤ 1 + ρ ˉ 2 C σ max { η 0 − 1 , 4 ( Lip ) 0 } [ g ( W 0 ) − g ( W j )] . (204)
In particular, since g ( W j ) ≥ g ∗ g(W_j)\ge g_* g ( W j ) ≥ g ∗ for every generated index j j j , we have
j ≤ 1 + C σ max { η 0 − 1 , 4 ( Lip ) 0 } ρ ˉ 2 [ g ( W 0 ) − g ∗ ] . j\le1+\frac{C_{\sigma}\max\{\eta_0^{-1},4(\operatorname{Lip})_0\}}{\bar{\rho}^{2}}[g(W_0)-g_*]. j ≤ 1 + ρ ˉ 2 C σ max { η 0 − 1 , 4 ( Lip ) 0 } [ g ( W 0 ) − g ∗ ] .
Thus ADAP-AIPP cannot generate infinitely many iteration indices, and hence it must terminate. Applying (204) with j = ℓ j=\ell j = ℓ , where ℓ \ell ℓ is ADAP-AIPP’s final iteration index, gives (202) .
The fact that ADAP-AIPP’s output ( W ‾ , R ‾ ) (\overline{W},\overline{R}) ( W , R ) satisfies (199) is immediate from ADAP-AIPP’s termination criterion in step 3 and the fact that the inclusion in (184) holds for every generated iteration index j ≥ 1 j\ge1 j ≥ 1 . Moreover, it follows from (188) and a simple induction argument that g ( W j ) ≤ g ( W 0 ) g(W_j)\le g(W_0) g ( W j ) ≤ g ( W 0 ) for all iteration indices j ≥ 1 j\ge1 j ≥ 1 , which implies that output ( W ‾ , R ‾ ) (\overline{W},\overline{R}) ( W , R ) satisfies (200) .
(b) The bound on the total number of ADAP-FISTA calls in (201) is immediate from the bound on ADAP-AIPP’s final iteration index ℓ \ell ℓ in (202) , Lemma B.9(a), and the fact that (193) implies that 4 ( Lip ) j ≤ 4 ( Lip ) 0 4(\operatorname{Lip})_j\le4(\operatorname{Lip})_0 4 ( Lip ) j ≤ 4 ( Lip ) 0 . □
Remark: Each ADAP-FISTA call performs at most
O ( Θ ˉ log + 2 ( Θ ˉ ) + log + 2 ( Θ ˉ ) ) . (205) \mathcal{O}\left(\sqrt{\bar{\Theta}}\log_{+}^{2}(\bar{\Theta})+\log_{+}^{2}(\bar{\Theta})\right). \tag*{(205)} O ( Θ ˉ log + 2 ( Θ ˉ ) + log + 2 ( Θ ˉ ) ) . (205)
gradient evaluations and projections onto Ω \Omega Ω where Θ ˉ \bar{\Theta} Θ ˉ is as in (196) . ▪
Proof. The result follows immediately from the complexity bound in (187) fact that Lemma B.12 implies that Θ j = O ( Θ ˉ ) \Theta_j=\mathcal{O}(\bar{\Theta}) Θ j = O ( Θ ˉ ) , and the first inequality in (194) . □
References