Introduction
Linear and convex quadratic programs [1 , 2 ] are fundamental optimization problems used in many fields, including operations research, economics, robotics, machine learning, and scientific computing [3 , 4 , 5 , 6 , 7 , 8 ] . These problems can be expressed as
min x ∈ R n { 1 2 ⟨ x , Q x ⟩ + ⟨ c , x ⟩ } subject to A x ≤ b , (1) \min_{x\in\mathbb{R}^{n}} \left\{ \frac{1}{2}\langle x,Qx\rangle+ \langle c,x\rangle\right\}\ \text{subject to } Ax\le b, \tag*{(1)} x ∈ R n min { 2 1 ⟨ x , Q x ⟩ + ⟨ c , x ⟩ } subject to A x ≤ b , (1)
where Q ∈ R n × n Q\in\mathbb{R}^{n\times n} Q ∈ R n × n is symmetric positive semidefinite (SPSD), A ∈ R m × n A\in\mathbb{R}^{m\times n} A ∈ R m × n , c ∈ R n c\in\mathbb{R}^{n} c ∈ R n , and b ∈ R m b\in\mathbb{R}^{m} b ∈ R m . Problem (1) is a linear program when Q = 0 Q=0 Q = 0 and a quadratic program otherwise. The study of algorithms for these problems began in the 1940s and 1950s, and much of modern optimization theory originates in it [9 , 10 , 11 ] . In current applications, the dimensions m m m and n n n are often large, and researchers have developed many new methods to solve problem (1) at that scale [12 , 13 ] .
We consider here the continuous-time optimization of problem (1) via a differential inclusion [14 , 15 ] . Let C = { x ∈ R n : A x ≤ b } C=\{x\in\mathbb{R}^{n}:Ax\le b\} C = { x ∈ R n : A x ≤ b } and assume that C C C is nonempty. The (nonsmooth) gradient flow of problem (1) is the differential inclusion
x ˙ ( τ ) ∈ − Q x ( τ ) − c − N C ( x ( τ ) ) , x ( 0 ) = x 0 ∈ C , (2) \dot{x}(\tau)\in-Qx(\tau)-c-N_C(x(\tau)),\qquad x(0)=x_0\in C, \tag*{(2)} x ˙ ( τ ) ∈ − Q x ( τ ) − c − N C ( x ( τ )) , x ( 0 ) = x 0 ∈ C , (2)
where N C ( x ( τ ) ) N_C(x(\tau)) N C ( x ( τ )) denotes the normal cone of C C C at x ( τ ) x(\tau) x ( τ ) (see Table 1 in § 2 ). At each time τ \tau τ , the velocity x ˙ ( τ ) \dot{x}(\tau) x ˙ ( τ ) is an element of the nonempty set on the right-hand side of (2). This differential inclusion has a unique solution, which converges to an optimal solution of problem (1) whenever one exists [14 , 15 ] .
Table 1. Some notation and definitions used in this work.
Solutions of differential inclusions, however, have rarely been computed exactly, except in simple examples [16 , 17 ] and for certain linear complementarity systems describing mechanical systems and electrical networks [18 ] . Most research has instead focused on the theory of differential inclusions and on the numerical approximation of their solutions in, e.g., nonsmooth analysis, control theory, and optimization [14 , 15 , 19 , 20 , 21 , 22 , 23 ] . Exact solutions have recently been computed for the differential inclusions of basis pursuit and the lasso [5 , 24 ] by Tendero et al. [25 ] and Langlois and Darbon [26 ] . Both papers used these solutions to analyze the dynamics and to derive algorithms that are exact (numerically up to machine precision) and competitive with established methods, such as the simplex method and coordinate descent.
Motivated by these results, we solve exactly the differential inclusion (2) characterizing linear and convex quadratic programs. We use the minimal selection principle (§ 2.2 ), which states that at each time, the solution moves along the element of minimal norm in the set on the right-hand side of (2). Our analysis yields two algorithms. Algorithm 1 computes the solution exactly when Q = t I n × n Q=tI_{n\times n} Q = t I n × n with t ≥ 0 t\ge0 t ≥ 0 , that is, for linear programs (t = 0 t=0 t = 0 ) and projections onto polyhedra (t > 0 t>0 t > 0 ), and terminates after finitely many iterations. Algorithm 2, which we call minimal selection descent (MSD), discretizes (2) by the explicit Euler method and applies to every SPSD Q Q Q .
Contributions of this paper: (i) We compute the solution of the differential inclusion (2) exactly. Each piece of the solution is an explicit function of time given by one nonnegative least-squares problem, and each breakpoint is the first zero of an explicit analytic function. (ii) We prove that the solution has finitely many pieces for every SPSD Q Q Q , and its final piece yields either an optimal solution of problem (1) or a certificate of unboundedness. When Q = t I n × n Q=tI_{n\times n} Q = t I n × n with t ≥ 0 t\ge0 t ≥ 0 , every piece lies on a straight line with a closed-form breakpoint and no active set occurs twice, so Algorithm 1 terminates after finitely many iterations. (iii) For general SPSD Q Q Q , each step of MSD preserves feasibility and solves one nonnegative least-squares problem over the active constraints.
Algorithm 1: Computing an optimal solution of problem (1) with Q = t I n × n Q=tI_{n\times n} Q = t I n × n , or certifying unboundedness.
Input: A matrix A ∈ R m × n A\in\mathbb{R}^{m\times n} A ∈ R m × n , vectors b ∈ R m b\in\mathbb{R}^{m} b ∈ R m , c ∈ R n c\in\mathbb{R}^{n} c ∈ R n , a scalar t ≥ 0 t\ge0 t ≥ 0 , and a feasible initial point x ( 0 ) ∈ C x^{(0)}\in C x ( 0 ) ∈ C .
Output: An optimal solution x s x^{s} x s of (1) with dual variable p s p^{s} p s , or a direction d s d^{s} d s certifying unboundedness (A d s ≤ 0 Ad^{s}\le0 A d s ≤ 0 and ⟨ c , d s ⟩ < 0 \langle c,d^{s}\rangle<0 ⟨ c , d s ⟩ < 0 ).
for k = 0 , 1 , 2 , … k=0,1,2,\ldots k = 0 , 1 , 2 , … do
Compute E ( k ) = { i ∈ { 1 , … , m } : [ A x ( k ) ] i = b i } \mathcal{E}^{(k)}=\{i\in\{1,\ldots,m\}:[Ax^{(k)}]_{i}=b_{i}\} E ( k ) = { i ∈ { 1 , … , m } : [ A x ( k ) ] i = b i } ;
Compute p ( k ) ∈ arg min p i ≥ 0 , i ∈ E ( k ) p i = 0 , i ∉ E ( k ) 1 2 ∥ A T p + t x ( k ) + c ∥ 2 2 p^{(k)}\in\underset{\substack{p_i\ge0,\ i\in\mathcal{E}^{(k)}\\p_i=0,\ i\notin\mathcal{E}^{(k)}}}{\operatorname{arg\,min}}\frac{1}{2}\left\|A^{\mathsf{T}}p+tx^{(k)}+c\right\|_{2}^{2} p ( k ) ∈ p i ≥ 0 , i ∈ E ( k ) p i = 0 , i ∈ / E ( k ) arg min 2 1 A T p + t x ( k ) + c 2 2 and set d ( k ) = − A T p ( k ) − t x ( k ) − c d^{(k)}=-A^{\mathsf{T}}p^{(k)}-tx^{(k)}-c d ( k ) = − A T p ( k ) − t x ( k ) − c ;
Compute the stepsize Δ ( k ) = min i ∉ E ( k ) b i − [ A x ( k ) ] i max ( 0 , [ A d ( k ) ] i ) \Delta^{(k)}=\min_{i\notin\mathcal{E}^{(k)}}\frac{b_i-[Ax^{(k)}]_i}{\max(0,[Ad^{(k)}]_i)} Δ ( k ) = min i ∈ / E ( k ) m a x ( 0 , [ A d ( k ) ] i ) b i − [ A x ( k ) ] i .
if t > 0 t>0 t > 0 and t Δ ( k ) ≥ 1 t\Delta^{(k)}\ge1 t Δ ( k ) ≥ 1 then
Set x s = x ( k ) + 1 t d ( k ) x^{s}=x^{(k)}+\frac{1}{t}d^{(k)} x s = x ( k ) + t 1 d ( k ) and p s = p ( k ) p^{s}=p^{(k)} p s = p ( k ) ;
return ( x s , p s ) (x^{s},p^{s}) ( x s , p s ) ; // Optimal
else if t = 0 t=0 t = 0 and Δ ( k ) = + ∞ \Delta^{(k)}=+\infty Δ ( k ) = + ∞ then
if ∥ d ( k ) ∥ 2 = 0 \|d^{(k)}\|_{2}=0 ∥ d ( k ) ∥ 2 = 0 then return ( x ( k ) , p ( k ) ) (x^{(k)},p^{(k)}) ( x ( k ) , p ( k ) ) , an optimal pair;
else return d s = d ( k ) d^{s}=d^{(k)} d s = d ( k ) , certifying unboundedness;
Update x ( k + 1 ) = x ( k ) + Δ ( k ) d ( k ) x^{(k+1)}=x^{(k)}+\Delta^{(k)}d^{(k)} x ( k + 1 ) = x ( k ) + Δ ( k ) d ( k ) ;
end
Algorithm 2: Minimal selection descent for problem (1) with SPSD matrix Q Q Q .
Input: A matrix A ∈ R m × n A\in\mathbb{R}^{m\times n} A ∈ R m × n , an SPSD matrix Q ∈ R n × n Q\in\mathbb{R}^{n\times n} Q ∈ R n × n , vectors b ∈ R m b\in\mathbb{R}^{m} b ∈ R m , c ∈ R n c\in\mathbb{R}^{n} c ∈ R n , a feasible initial point x ( 0 ) ∈ C x^{(0)}\in C x ( 0 ) ∈ C , and a tolerance ε ≥ 0 \varepsilon\ge0 ε ≥ 0 .
Output: An approximate primal optimal solution x s x^{s} x s of (1) with dual variable p s p^{s} p s , or a direction d s d^{s} d s certifying unboundedness (A d s ≤ 0 Ad^{s}\le0 A d s ≤ 0 , Q d s = 0 Qd^{s}=0 Q d s = 0 , and ⟨ c , d s ⟩ < 0 \langle c,d^{s}\rangle<0 ⟨ c , d s ⟩ < 0 ).
for k = 0 , 1 , 2 , … k=0,1,2,\ldots k = 0 , 1 , 2 , … do
Compute E ( k ) = { i ∈ { 1 , … , m } : [ A x ( k ) ] i = b i } \mathcal{E}^{(k)}=\{i\in\{1,\ldots,m\}:[Ax^{(k)}]_{i}=b_{i}\} E ( k ) = { i ∈ { 1 , … , m } : [ A x ( k ) ] i = b i } ;
Compute d ( k ) = − ( A T p ( k ) + Q x ( k ) + c ) d^{(k)}=-(A^{\mathsf{T}}p^{(k)}+Qx^{(k)}+c) d ( k ) = − ( A T p ( k ) + Q x ( k ) + c ) , where p ( k ) ∈ arg min p i ≥ 0 , i ∈ E ( k ) p i = 0 , i ∉ E ( k ) 1 2 ∥ A T p + Q x ( k ) + c ∥ 2 2 p^{(k)}\in\underset{\substack{p_i\ge0,\ i\in\mathcal{E}^{(k)}\\p_i=0,\ i\notin\mathcal{E}^{(k)}}}{\operatorname{arg\,min}}\frac{1}{2}\left\|A^{\mathsf{T}}p+Qx^{(k)}+c\right\|_{2}^{2} p ( k ) ∈ p i ≥ 0 , i ∈ E ( k ) p i = 0 , i ∈ / E ( k ) arg min 2 1 A T p + Q x ( k ) + c 2 2 ;
if ∥ d ( k ) ∥ 2 ≤ ε \|d^{(k)}\|_{2}\le\varepsilon ∥ d ( k ) ∥ 2 ≤ ε then return ( x s , p s ) = ( x ( k ) , p ( k ) ) (x^{s},p^{s})=(x^{(k)},p^{(k)}) ( x s , p s ) = ( x ( k ) , p ( k ) ) // Converged;
Compute the stepsize Δ ( k ) = min ( min i ∉ E ( k ) { b i − [ A x ( k ) ] i max ( 0 , [ A d ( k ) ] i ) } , ∥ d ( k ) ∥ 2 2 ⟨ d ( k ) , Q d ( k ) ⟩ ) \Delta^{(k)}=\min\left(\min_{i\notin\mathcal{E}^{(k)}}\left\{\frac{b_i-[Ax^{(k)}]_i}{\max(0,[Ad^{(k)}]_i)}\right\},\frac{\|d^{(k)}\|_{2}^{2}}{\langle d^{(k)},Qd^{(k)}\rangle}\right) Δ ( k ) = min ( min i ∈ / E ( k ) { m a x ( 0 , [ A d ( k ) ] i ) b i − [ A x ( k ) ] i } , ⟨ d ( k ) , Q d ( k ) ⟩ ∥ d ( k ) ∥ 2 2 ) ;
if Δ ( k ) = + ∞ \Delta^{(k)}=+\infty Δ ( k ) = + ∞ then return d s = d ( k ) d^{s}=d^{(k)} d s = d ( k ) // Unbounded;
Update x ( k + 1 ) = x ( k ) + Δ ( k ) d ( k ) x^{(k+1)}=x^{(k)}+\Delta^{(k)}d^{(k)} x ( k + 1 ) = x ( k ) + Δ ( k ) d ( k ) ;
end
We prove that the iterates converge to an optimal solution when one exists and otherwise certify unboundedness or drive the objective to − ∞ -\infty − ∞ . When strict complementarity holds at the limit, the iterates also identify the optimal active set after finitely many steps and then converge at a linear rate. (iv) We apply our algorithms to a positivity-preserving limiter for a discontinuous Galerkin scheme and to optimal control problems constrained by the Poisson and heat equations. In the limiter, Algorithm 1 conserves the total mass much more accurately than the splitting method of Liu et al. [8 ] . On the control problems, MSD reaches the optimal value to machine precision and is competitive in time with commercial and open-source solvers.
Related work: Reviews of algorithms for linear and convex quadratic programs include [11 , 12 , 13 ] . Closest to our approach are the steepest descent methods for linear programs, which move from one feasible point to another along steepest descent directions over the faces of the feasible polyhedron [27 , 28 , 29 , 30 , 31 ] . These methods either assumed that the data are nondegenerate or approximated the steepest descent direction. When Q = 0 Q = 0 Q = 0 , the pieces of the trajectory of (2) are also rays along steepest descent directions. We compute these directions exactly, and our analysis extends to convex quadratic programs. Like primal active-set methods for quadratic programs [32 Section 16.5] , our algorithms keep the iterates feasible and hold a set of constraints active on each step. Active-set methods, however, step toward the minimizer of the objective on the face of the current set, which requires solving a linear system involving Q Q Q , and they change the set one constraint at a time. Our algorithms move along the minimal selection, compute it from one nonnegative least-squares problem problem in A A A , and can add or remove several constraints at once.
By the minimal selection principle, the differential inclusion (2) can be written equivalently as a projected dynamical system [19 , 20 ] , whose velocity is the projection of − Q x − c -Qx - c − Q x − c onto the tangent cone of C C C (§ 2.3 ). It can also be written as a complementarity system, which couples a differential equation with complementarity conditions between the multipliers of the constraints and their slacks. The three formulations have the same unique solution [33 ] . After a constant state is appended, the complementarity formulation is a linear complementarity system, a class that also models mechanical systems with unilateral constraints and electrical networks with ideal diodes [18 ] . Heemels et al. [18 ] construct the solutions of such systems piece by piece, but for (2) their assumptions require all rows of A A A to be linearly independent, and they leave open whether the pieces can accumulate. Our analysis requires linear independence only for the active constraints and proves that the pieces are finitely many.
Organization of this paper: We begin in § 2 by detailing our setup and discussing the minimal selection principle for differential inclusions, which casts the problem of computing an optimal solution to (1) (or certifying unboundedness) into that of computing the trajectory of an ordinary differential equation. In § 3 , we characterize the instantaneous and local dynamics of the solution to the differential inclusions associated with problem (1). In § 4 , we analyze the global solution to the differential inclusions and investigate its asymptotic limit, showing in particular that the global solution and its asymptotic limit can be computed from finitely many pieces and nonnegative least-squares problems. § 5 analyzes the explicit Euler discretization of the differential inclusion associated with problem (1) for a general SPSD Q Q Q , and shows that the iterates converge to an optimal solution or certify unboundedness, and that they identify the optimal active set and converge at a linear rate when the multipliers at the limit are positive. We present in § 6 numerical experiments to test our algorithms on challenging scientific computing problems, including preserving positivity for discontinuous Galerkin methods and PDE-constrained optimal control problems.
Preliminaries
We review here the first-order optimality conditions of problem (1) and present our differential inclusions approach. Table 1 below summarizes some notation and definitions. We refer the reader to Appendix § 10 for more mathematical background.
Setup
We assume throughout that problem (1) is feasible, i.e., there is x ∈ R n x \in\mathbb{R}^{n} x ∈ R n such that A x ≤ b Ax \le b A x ≤ b . For convenience, we use the convention 0 / 0 = 0 0/0=0 0/0 = 0 and, for a > 0 a>0 a > 0 , a / 0 = + ∞ a/0=+\infty a /0 = + ∞ . Let C = { x ∈ R n : A x ≤ b } C=\{x \in\mathbb{R}^{n}: Ax \le b\} C = { x ∈ R n : A x ≤ b } , let χ C \chi_C χ C denote the characteristic function of C C C , let f ( x ) = 1 2 ⟨ x , Q x ⟩ + ⟨ c , x ⟩ f(x)=\frac{1}{2}\langle x,Qx\rangle+\langle c,x\rangle f ( x ) = 2 1 ⟨ x , Q x ⟩ + ⟨ c , x ⟩ , and let V ( x ) = f ( x ) + χ C ( x ) V(x)=f(x)+\chi_C(x) V ( x ) = f ( x ) + χ C ( x ) denote the objective function in (1).
The subdifferential of V V V at a point x 0 ∈ C x_0 \in C x 0 ∈ C is described in terms of the normal cone N C ( x 0 ) N_C(x_0) N C ( x 0 ) (Table 1 ). Subdifferential calculus [34 Exercise 10.22(a),(b)] gives
∂ x V ( x 0 ) = ∇ x f ( x 0 ) + N C ( x 0 ) . \partial_x V(x_0)=\nabla_x f(x_0)+N_C(x_0). ∂ x V ( x 0 ) = ∇ x f ( x 0 ) + N C ( x 0 ) .
Since C C C is a polyhedron, the rows of A A A corresponding to the constraints active at x 0 x_0 x 0 generate N C ( x 0 ) N_C(x_0) N C ( x 0 ) [16 Example 5.2.6b] . We define correspondingly the equicorrelation set of x 0 x_0 x 0 as
E ( x 0 ) : = { i ∈ [ m ] : [ A x 0 ] i = b i } . \mathcal{E}(x_0):=\{i\in[m]:[Ax_0]_i=b_i\}. E ( x 0 ) := { i ∈ [ m ] : [ A x 0 ] i = b i } .
Hence, explicitly,
∂ x V ( x 0 ) = { ∇ x f ( x 0 ) + A T p : p E ( x 0 ) ≥ 0 , p E c ( x 0 ) = 0 } . (3) \partial_x V(x_0)=\left\{\nabla_x f(x_0)+A^{\mathsf{T}}p:p_{\mathcal{E}(x_0)}\ge0,\ p_{\mathcal{E}^{c}(x_0)}=0\right\}. \tag*{(3)} ∂ x V ( x 0 ) = { ∇ x f ( x 0 ) + A T p : p E ( x 0 ) ≥ 0 , p E c ( x 0 ) = 0 } . (3)
A point x s ∈ C x^s \in C x s ∈ C is an optimal solution of problem (1) if and only if 0 ∈ ∂ x V ( x s ) 0\in\partial_x V(x^s) 0 ∈ ∂ x V ( x s ) [34 Theorem 10.1] . Equivalently, there is p s ∈ R m p^s\in\mathbb{R}^{m} p s ∈ R m with p E ( x s ) s ≥ 0 p^s_{\mathcal{E}(x^s)}\ge0 p E ( x s ) s ≥ 0 and p E c ( x s ) s = 0 p^s_{\mathcal{E}^{c}(x^s)}=0 p E c ( x s ) s = 0 such that
∇ x f ( x s ) + A T p s = 0. \nabla_x f(x^s)+A^{\mathsf{T}}p^s=0. ∇ x f ( x s ) + A T p s = 0.
If Q Q Q is SPD, then V V V is strongly convex and problem (1) has a unique optimal solution. If Q Q Q is merely SPSD, problem (1) may have no optimal solution, or several.
Differential inclusions, the minimal selection principle, and the slow system
To certify if problem (1) has an optimal solution, and output one when it does, we propose to compute the trajectory of the differential inclusion
x ˙ ( τ ) ∈ − ∂ x V ( x ( τ ) ) , x ( 0 ) = x 0 ∈ C . (4) \dot{x}(\tau) \in-\partial_x V(x(\tau)), \quad x(0)=x_0 \in C. \tag*{(4)} x ˙ ( τ ) ∈ − ∂ x V ( x ( τ )) , x ( 0 ) = x 0 ∈ C . (4)
A differential inclusion replaces the right-hand side of an ordinary differential equation with an appropriate set-valued mapping. When the mapping is the subdifferential of a proper, lower semicontinuous, convex function (as in (4)), the inclusion is called a gradient inclusion . The next theorem states that the solution of (4) exists, is unique, and converges to a minimizer of V V V when one exists. The theorem also states the minimal selection principle , namely that at every τ ≥ 0 \tau\ge0 τ ≥ 0 , the solution evolves along the element of minimal norm in − ∂ x V ( x ( τ ) ) -\partial_x V(x(\tau)) − ∂ x V ( x ( τ )) .
Theorem 2.1. The differential inclusion (4) has a unique global solution x : [ 0 , + ∞ ) → C x\colon[0,+\infty) \to C x : [ 0 , + ∞ ) → C that is globally Lipschitz continuous and satisfies the following properties:
(i) The right derivative τ ↦ x ˙ ( τ ) \tau\mapsto\dot{x}(\tau) τ ↦ x ˙ ( τ ) is right-continuous and τ ↦ ∥ x ˙ ( τ ) ∥ 2 \tau\mapsto\lVert\dot{x}(\tau)\rVert_2 τ ↦ ∥ x ˙ ( τ ) ∥ 2 is nonincreasing.
(ii) If x ↦ V ( x ) x \mapsto V(x) x ↦ V ( x ) achieves its minimum at some point, then τ ↦ x ( τ ) \tau\mapsto x(\tau) τ ↦ x ( τ ) converges to such a point:
lim τ → + ∞ V ( x ( τ ) ) = min x ∈ R n V ( x ) and lim τ → + ∞ x ( τ ) ∈ arg min x ∈ R n V ( x ) . \lim_{\tau\to+\infty} V(x(\tau))=\min_{x\in\mathbb{R}^n}V(x)
\quad\text{and}\quad
\lim_{\tau\to+\infty}x(\tau)\in\arg\min_{x\in\mathbb{R}^n}V(x). τ → + ∞ lim V ( x ( τ )) = x ∈ R n min V ( x ) and τ → + ∞ lim x ( τ ) ∈ arg x ∈ R n min V ( x ) . (iii) (The minimal selection principle. ) The function τ ↦ x ( τ ) \tau\mapsto x(\tau) τ ↦ x ( τ ) solves the initial value problem
x ˙ ( τ ) = − proj ∂ x V ( x ( τ ) ) ( 0 ) , x ( 0 ) = x 0 ∈ C (5) \dot{x}(\tau)=-\operatorname{proj}_{\partial_x V(x(\tau))}(0), \quad x(0)=x_0\in C \tag*{(5)} x ˙ ( τ ) = − proj ∂ x V ( x ( τ )) ( 0 ) , x ( 0 ) = x 0 ∈ C (5) at every τ ≥ 0 \tau\ge0 τ ≥ 0 , the derivative on the left being the right derivative of part (i).
Proof. See [14, Theorem 1 (p. 147); Theorem 1 (p. 159)] and [15 Theorem 3.1, p. 54] .
Geometric interpretation of the minimal selection principle
Following [14 ] , we call (5) the slow system and its solution the slow solution . At any x ∈ C x \in C x ∈ C , the normal cone N C ( x ) N_C(x) N C ( x ) and the tangent cone T C ( x ) T_C(x) T C ( x ) (Table 1) are mutually polar closed convex cones, so by Moreau’s decomposition theorem [35 ] , every z ∈ R n z \in\mathbb{R}^n z ∈ R n decomposes as
z = proj N C ( x ) ( z ) + proj T C ( x ) ( z ) , ⟨ proj N C ( x ) ( z ) , proj T C ( x ) ( z ) ⟩ = 0. (6) z=\operatorname{proj}_{N_C(x)}(z)+\operatorname{proj}_{T_C(x)}(z), \quad
\langle\operatorname{proj}_{N_C(x)}(z),\operatorname{proj}_{T_C(x)}(z)\rangle=0. \tag*{(6)} z = proj N C ( x ) ( z ) + proj T C ( x ) ( z ) , ⟨ proj N C ( x ) ( z ) , proj T C ( x ) ( z )⟩ = 0. (6)
Since ∂ x V ( x ) = ∇ x f ( x ) + N C ( x ) \partial_xV(x)=\nabla_xf(x)+N_C(x) ∂ x V ( x ) = ∇ x f ( x ) + N C ( x ) , we have proj ∂ x V ( x ) ( 0 ) = ∇ x f ( x ) + proj N C ( x ) ( − ∇ x f ( x ) ) \operatorname{proj}_{\partial_xV(x)}(0)=\nabla_xf(x)+\operatorname{proj}_{N_C(x)}(-\nabla_xf(x)) proj ∂ x V ( x ) ( 0 ) = ∇ x f ( x ) + proj N C ( x ) ( − ∇ x f ( x )) , and applying (6) with z = − ∇ x f ( x ) z=-\nabla_xf(x) z = − ∇ x f ( x ) gives − proj ∂ x V ( x ) ( 0 ) = proj T C ( x ) ( − ∇ x f ( x ) ) -\operatorname{proj}_{\partial_xV(x)}(0)=\operatorname{proj}_{T_C(x)}(-\nabla_xf(x)) − proj ∂ x V ( x ) ( 0 ) = proj T C ( x ) ( − ∇ x f ( x )) . The slow system (5) then becomes the projected dynamical system [19 ]
x ˙ ( τ ) = d ( x ( τ ) ) , x ( 0 ) = x 0 ∈ C , where d ( x ) : = proj T C ( x ) ( − ∇ x f ( x ) ) . \dot{x}(\tau)=d(x(\tau)), \quad x(0)=x_0\in C, \quad\text{where}\quad d(x):=\operatorname{proj}_{T_C(x)}(-\nabla_xf(x)). x ˙ ( τ ) = d ( x ( τ )) , x ( 0 ) = x 0 ∈ C , where d ( x ) := proj T C ( x ) ( − ∇ x f ( x )) .
We call d ( x ) d(x) d ( x ) the direction at x ∈ C x \in C x ∈ C . The slow solution therefore evolves along the projection of the negative gradient − ∇ x f -\nabla_xf − ∇ x f onto the tangent cone of C C C .
Computing the direction via nonnegative least squares
Integrating the slow system (5) requires evaluating the direction d ( x ( τ ) ) \mathbf{d}(x(\tau)) d ( x ( τ )) for all τ ≥ 0 \tau\ge0 τ ≥ 0 . At a single point x 0 ∈ C x_0 \in C x 0 ∈ C , we evaluate d ( x 0 ) \mathbf{d}(x_0) d ( x 0 ) via nonnegative least squares (NNLS). Write n ( x 0 ) : = proj N C ( x 0 ) ( − ∇ x f ( x 0 ) ) \mathbf{n}(x_0) := \operatorname{proj}_{N_C(x_0)}(-\nabla_x f(x_0)) n ( x 0 ) := proj N C ( x 0 ) ( − ∇ x f ( x 0 )) . By formula (3) , we have n ( x 0 ) = A T p ( x 0 ) \mathbf{n}(x_0) = A^{\mathsf{T}}\mathbf{p}(x_0) n ( x 0 ) = A T p ( x 0 ) , where
p ( x 0 ) ∈ arg min p ∈ R m { 1 2 ∥ A T p + ∇ x f ( x 0 ) ∥ 2 2 : p E ( x 0 ) ≥ 0 , p E C ( x 0 ) = 0 } . (7) \mathbf{p}(x_0) \in\underset{\mathbf{p}\in\mathbb{R}^m}{\operatorname{arg\,min}}\left\{\frac{1}{2}\left\|A^{\mathsf{T}}\mathbf{p}+\nabla_x f(x_0)\right\|_2^2:\ \mathbf{p}_{E(x_0)}\ge0,\ \mathbf{p}_{E^C(x_0)}=0\right\}. \tag*{(7)} p ( x 0 ) ∈ p ∈ R m arg min { 2 1 A T p + ∇ x f ( x 0 ) 2 2 : p E ( x 0 ) ≥ 0 , p E C ( x 0 ) = 0 } . (7)
Moreau’s decomposition theorem (6) then gives
d ( x 0 ) = − A T p ( x 0 ) − ∇ x f ( x 0 ) . (8) \mathbf{d}(x_0)=-A^{\mathsf{T}}\mathbf{p}(x_0)-\nabla_x f(x_0). \tag*{(8)} d ( x 0 ) = − A T p ( x 0 ) − ∇ x f ( x 0 ) . (8)
The vector p ( x 0 ) \mathbf{p}(x_0) p ( x 0 ) may not always be unique, but the projections d ( x 0 ) \mathbf{d}(x_0) d ( x 0 ) and n ( x 0 ) \mathbf{n}(x_0) n ( x 0 ) are. The optimal NNLS solution p ( x 0 ) \mathbf{p}(x_0) p ( x 0 ) admits an explicit form in terms of the active set
J ( p ( x 0 ) ) : = { i ∈ [ m ] : [ p ( x 0 ) ] i > 0 } . (9) J(\mathbf{p}(x_0)):=\{i\in[m]:[\mathbf{p}(x_0)]_i>0\}. \tag*{(9)} J ( p ( x 0 )) := { i ∈ [ m ] : [ p ( x 0 ) ] i > 0 } . (9)
We write J ( x 0 ) ≡ J ( p ( x 0 ) ) J(x_0)\equiv J(\mathbf{p}(x_0)) J ( x 0 ) ≡ J ( p ( x 0 )) when p ( x 0 ) \mathbf{p}(x_0) p ( x 0 ) is unambiguous. By Carathéodory’s theorem [36 Corollary 17.1.2] , we can choose p ( x 0 ) \mathbf{p}(x_0) p ( x 0 ) so that
the rows of A J ( x 0 ) are linearly independent. \text{the rows of }A_{J(x_0)}\text{ are linearly independent.} the rows of A J ( x 0 ) are linearly independent.
Since p J ( x 0 ) ( x 0 ) \mathbf{p}_{J(x_0)}(x_0) p J ( x 0 ) ( x 0 ) is strictly positive, it is an unconstrained minimizer of q ↦ 1 2 ∥ A J ( x 0 ) T q + ∇ x f ( x 0 ) ∥ 2 2 \mathbf{q}\mapsto\frac{1}{2}\left\|A_{J(x_0)}^{\mathsf{T}}\mathbf{q}+\nabla_x f(x_0)\right\|_2^2 q ↦ 2 1 A J ( x 0 ) T q + ∇ x f ( x 0 ) 2 2 by convexity. The first-order optimality conditions give
p J ( x 0 ) ( x 0 ) = − ( A J ( x 0 ) A J ( x 0 ) T ) − 1 A J ( x 0 ) ∇ x f ( x 0 ) . (10) \mathbf{p}_{J(x_0)}(x_0)=-\left(A_{J(x_0)}A_{J(x_0)}^{\mathsf{T}}\right)^{-1}A_{J(x_0)}\nabla_x f(x_0). \tag*{(10)} p J ( x 0 ) ( x 0 ) = − ( A J ( x 0 ) A J ( x 0 ) T ) − 1 A J ( x 0 ) ∇ x f ( x 0 ) . (10)
Let
P J ( x 0 ) ⊥ : = I n × n − A J ( x 0 ) T ( A J ( x 0 ) A J ( x 0 ) T ) − 1 A J ( x 0 ) (11) P_{J(x_0)}^\perp:=I_{n\times n}-A_{J(x_0)}^{\mathsf{T}}\left(A_{J(x_0)}A_{J(x_0)}^{\mathsf{T}}\right)^{-1}A_{J(x_0)} \tag*{(11)} P J ( x 0 ) ⊥ := I n × n − A J ( x 0 ) T ( A J ( x 0 ) A J ( x 0 ) T ) − 1 A J ( x 0 ) (11)
denote the orthogonal projection onto ker ( A J ( x 0 ) ) \ker(A_{J(x_0)}) ker ( A J ( x 0 ) ) . Since p ( x 0 ) \mathbf{p}(x_0) p ( x 0 ) vanishes outside J ( x 0 ) J(x_0) J ( x 0 ) , substituting (10) into (8) gives
d ( x 0 ) = − P J ( x 0 ) ⊥ ∇ x f ( x 0 ) . (12) \mathbf{d}(x_0)=-P_{J(x_0)}^\perp\nabla_x f(x_0). \tag*{(12)} d ( x 0 ) = − P J ( x 0 ) ⊥ ∇ x f ( x 0 ) . (12)
When J ( x 0 ) = ∅ J(x_0)=\varnothing J ( x 0 ) = ∅ , we read (10) as p ( x 0 ) = 0 \mathbf{p}(x_0)=\mathbf{0} p ( x 0 ) = 0 and (11) as P J ( x 0 ) ⊥ = I n × n P_{J(x_0)}^\perp=I_{n\times n} P J ( x 0 ) ⊥ = I n × n .
Remark 2.1. Specialized algorithms for solving NNLS problems include active-set algorithms terminating in finite time, such as the classical Lawson–Hanson algorithm [37 ] and its generalization, Meyer’s algorithm [38 ] . See also* [39 ] for a review of other cone projection algorithms.
Karush–Kuhn–Tucker and strict complementarity conditions
The solution p ( x 0 ) \mathbf{p}(x_0) p ( x 0 ) of the NNLS problem (7) and the direction d ( x 0 ) \mathbf{d}(x_0) d ( x 0 ) satisfy the Karush–Kuhn–Tucker (KKT) conditions of that problem and two of their consequences:
[ A d ( x 0 ) ] E ( x 0 ) ≤ 0 , (13) [A\mathbf{d}(x_0)]_{E(x_0)}\le0, \tag*{(13)} [ A d ( x 0 ) ] E ( x 0 ) ≤ 0 , (13)
[ p ( x 0 ) ] i [ A d ( x 0 ) ] i = 0 for i = 1 , … , m , (14) [\mathbf{p}(x_0)]_i[A\mathbf{d}(x_0)]_i=0\text{ for }i=1,\ldots,m, \tag*{(14)} [ p ( x 0 ) ] i [ A d ( x 0 ) ] i = 0 for i = 1 , … , m , (14)
[ A d ( x 0 ) ] i = 0 for every i ∈ J ( x 0 ) , (15) [A\mathbf{d}(x_0)]_i=0\text{ for every }i\in J(x_0), \tag*{(15)} [ A d ( x 0 ) ] i = 0 for every i ∈ J ( x 0 ) , (15)
∥ d ( x 0 ) ∥ 2 2 + ⟨ ∇ x f ( x 0 ) , d ( x 0 ) ⟩ = 0. (16) \|\mathbf{d}(x_0)\|_2^2+\langle\nabla_x f(x_0),\mathbf{d}(x_0)\rangle=0. \tag*{(16)} ∥ d ( x 0 ) ∥ 2 2 + ⟨ ∇ x f ( x 0 ) , d ( x 0 )⟩ = 0. (16)
Conditions (13a)–(13d) all follow from Moreau’s decomposition (6). Indeed, condition (13a) states that d ( x 0 ) ∈ T C ( x 0 ) d(x_0) \in T_C(x_0) d ( x 0 ) ∈ T C ( x 0 ) . For condition (13b), the orthogonality in (6) gives
⟨ A d ( x 0 ) , p ( x 0 ) ⟩ = ⟨ d ( x 0 ) , A ⊤ p ( x 0 ) ⟩ = ⟨ d ( x 0 ) , n ( x 0 ) ⟩ = 0 , \langle A d(x_0),p(x_0)\rangle=\langle d(x_0),A^\top p(x_0)\rangle=\langle d(x_0),n(x_0)\rangle=0, ⟨ A d ( x 0 ) , p ( x 0 )⟩ = ⟨ d ( x 0 ) , A ⊤ p ( x 0 )⟩ = ⟨ d ( x 0 ) , n ( x 0 )⟩ = 0 ,
and every term of ⟨ A d ( x 0 ) , p ( x 0 ) ⟩ \langle A d(x_0),p(x_0)\rangle ⟨ A d ( x 0 ) , p ( x 0 )⟩ vanishes because p E ( x 0 ) ( x 0 ) ≥ 0 p_{\mathcal E(x_0)}(x_0) \ge0 p E ( x 0 ) ( x 0 ) ≥ 0 , p E C ( x 0 ) ( x 0 ) = 0 p_{\mathcal E^C(x_0)}(x_0)=0 p E C ( x 0 ) ( x 0 ) = 0 , and (13a) holds. Condition (13c) follows from (13b) because [ p ( x 0 ) ] i > 0 [p(x_0)]_i>0 [ p ( x 0 ) ] i > 0 for every i ∈ J ( x 0 ) i \in J(x_0) i ∈ J ( x 0 ) . Condition (13d) follows from (8), which gives ⟨ ∇ x f ( x 0 ) , d ( x 0 ) ⟩ = ⟨ − d ( x 0 ) − n ( x 0 ) , d ( x 0 ) ⟩ = − ∥ d ( x 0 ) ∥ 2 2 \langle\nabla_x f(x_0),d(x_0)\rangle=\langle-d(x_0)-n(x_0),d(x_0)\rangle=-\lVert d(x_0)\rVert_2^2 ⟨ ∇ x f ( x 0 ) , d ( x 0 )⟩ = ⟨ − d ( x 0 ) − n ( x 0 ) , d ( x 0 )⟩ = − ∥ d ( x 0 ) ∥ 2 2 . Problem (7) is convex with linear constraints, so conditions (13a)–(13b) are also sufficient. Every feasible p p p whose residual − A ⊤ p − ∇ x f ( x 0 ) -A^\top p-\nabla_x f(x_0) − A ⊤ p − ∇ x f ( x 0 ) satisfies (13a)–(13b) in place of d ( x 0 ) d(x_0) d ( x 0 ) solves (7).
Remark 2.2. The KKT complementarity condition (13b) over the equicorrelation set E ( x 0 ) \mathcal E(x_0) E ( x 0 ) need not be strict. To ease the exposition, the results of Sections 3.1–3.4, except Proposition 3.1, assume the strict complementarity condition
[ A d ( x 0 ) ] i < 0 for every i ∈ E ( x 0 ) ∖ J ( x 0 ) . [Ad(x_0)]_i<0 \quad\text{for every } i\in\mathcal E(x_0)\setminus J(x_0). [ A d ( x 0 ) ] i < 0 for every i ∈ E ( x 0 ) ∖ J ( x 0 ) .
(SCC)
Section 3.5 extends these results to the case where (SCC) fails.
Local solution of the slow system
In this section, we compute the local solution τ ↦ x ( τ ) \tau\mapsto x(\tau) τ ↦ x ( τ ) of the slow system (5) started at x 0 ∈ C x_0\in C x 0 ∈ C and show that τ ↦ d ( x ( τ ) ) \tau\mapsto d(x(\tau)) τ ↦ d ( x ( τ )) may jump at a breakpoint time τ ∗ ( x 0 ) > 0 \tau_*(x_0)>0 τ ∗ ( x 0 ) > 0 . Section 3.1 analyzes the instantaneous dynamics, Section 3.2 integrates the system on [ 0 , τ ∗ ( x 0 ) ) [0,\tau_*(x_0)) [ 0 , τ ∗ ( x 0 )) , and Section 3.3 extends the solution to τ ∗ ( x 0 ) \tau_*(x_0) τ ∗ ( x 0 ) whenever it is finite. Section 3.4 computes the breakpoint time in practice, and Section 3.5 extends these results to the case where (SCC) fails.
Instantaneous dynamics of the slow system
Descent properties
How does the slow system (5) evolve instantaneously? We first show that the objective function V V V decreases along the ray Δ ↦ x 0 + Δ d ( x 0 ) \Delta\mapsto x_0+\Delta d(x_0) Δ ↦ x 0 + Δ d ( x 0 ) for small Δ > 0 \Delta>0 Δ > 0 . The decrease requires that x 0 + Δ d ( x 0 ) x_0+\Delta d(x_0) x 0 + Δ d ( x 0 ) stay feasible, that is, A ( x 0 + Δ d ( x 0 ) ) ≤ b A(x_0+\Delta d(x_0))\le b A ( x 0 + Δ d ( x 0 )) ≤ b . Since [ A d ( x 0 ) ] E ( x 0 ) ≤ 0 [Ad(x_0)]_{\mathcal E(x_0)}\le0 [ A d ( x 0 ) ] E ( x 0 ) ≤ 0 by (13a) and [ A x 0 ] E C ( x 0 ) < b E C ( x 0 ) [Ax_0]_{\mathcal E^C(x_0)}<b_{\mathcal E^C(x_0)} [ A x 0 ] E C ( x 0 ) < b E C ( x 0 ) by definition, the largest Δ \Delta Δ for which this inequality holds is
Δ f e a s ( x 0 ) : = min i ∈ E C ( x 0 ) b i − [ A x 0 ] i max ( 0 , [ A d ( x 0 ) ] i ) . (17) \Delta_{\mathrm{feas}}(x_0):=\min_{i\in\mathcal E^C(x_0)}\frac{b_i-[Ax_0]_i}{\max(0,[Ad(x_0)]_i)}. \tag*{(17)} Δ feas ( x 0 ) := i ∈ E C ( x 0 ) min max ( 0 , [ A d ( x 0 ) ] i ) b i − [ A x 0 ] i . (17)
which equals + ∞ +\infty + ∞ if E C ( x 0 ) = ∅ \mathcal E^C(x_0)=\varnothing E C ( x 0 ) = ∅ or [ A d ( x 0 ) ] E C ( x 0 ) ≤ 0 [Ad(x_0)]_{\mathcal E^C(x_0)}\le0 [ A d ( x 0 ) ] E C ( x 0 ) ≤ 0 . In any case, x 0 + Δ d ( x 0 ) ∈ C x_0+\Delta d(x_0)\in C x 0 + Δ d ( x 0 ) ∈ C for every Δ ∈ [ 0 , Δ f e a s ( x 0 ) ] \Delta\in[0,\Delta_{\mathrm{feas}}(x_0)] Δ ∈ [ 0 , Δ feas ( x 0 )] , and condition (13d) gives
V ( x 0 + Δ d ( x 0 ) ) = V ( x 0 ) − Δ ∥ d ( x 0 ) ∥ 2 2 + Δ 2 2 ⟨ d ( x 0 ) , Q d ( x 0 ) ⟩ . (18) V(x_0+\Delta d(x_0))=V(x_0)-\Delta\lVert d(x_0)\rVert_2^2+\frac{\Delta^2}{2}\langle d(x_0),Qd(x_0)\rangle. \tag*{(18)} V ( x 0 + Δ d ( x 0 )) = V ( x 0 ) − Δ ∥ d ( x 0 ) ∥ 2 2 + 2 Δ 2 ⟨ d ( x 0 ) , Q d ( x 0 )⟩ . (18)
Formula (15) shows that d ( x 0 ) d(x_0) d ( x 0 ) is a descent direction.
Proposition 3.1. Let x 0 ∈ C x_0\in C x 0 ∈ C . If d ( x 0 ) ≠ 0 d(x_0)\ne0 d ( x 0 ) = 0 , then V ( x 0 + Δ d ( x 0 ) ) < V ( x 0 ) V(x_0+\Delta d(x_0))<V(x_0) V ( x 0 + Δ d ( x 0 )) < V ( x 0 ) for every Δ ∈ ( 0 , Δ + ( x 0 ) ) \Delta\in(0,\Delta_+(x_0)) Δ ∈ ( 0 , Δ + ( x 0 )) , where
Δ + ( x 0 ) : = min ( Δ f e a s ( x 0 ) , 2 ∥ d ( x 0 ) ∥ 2 2 ⟨ d ( x 0 ) , Q d ( x 0 ) ⟩ ) . (19) \Delta_+(x_0):=\min\left(\Delta_{\mathrm{feas}}(x_0),\frac{2\lVert d(x_0)\rVert_2^2}{\langle d(x_0),Qd(x_0)\rangle}\right). \tag*{(19)} Δ + ( x 0 ) := min ( Δ feas ( x 0 ) , ⟨ d ( x 0 ) , Q d ( x 0 )⟩ 2 ∥ d ( x 0 ) ∥ 2 2 ) . (19)
Proof. Suppose d ( x 0 ) ≠ 0 d(x_0) \ne0 d ( x 0 ) = 0 . If ⟨ d ( x 0 ) , Q d ( x 0 ) ⟩ = 0 \langle d(x_0), Qd(x_0)\rangle= 0 ⟨ d ( x 0 ) , Q d ( x 0 )⟩ = 0 , then the right-hand side of (18) decreases with Δ \Delta Δ until Δ = Δ f e a s ( x 0 ) \Delta= \Delta_{\mathrm{feas}}(x_0) Δ = Δ feas ( x 0 ) . If ⟨ d ( x 0 ) , Q d ( x 0 ) ⟩ > 0 \langle d(x_0), Qd(x_0)\rangle> 0 ⟨ d ( x 0 ) , Q d ( x 0 )⟩ > 0 , the difference V ( x 0 + Δ d ( x 0 ) ) − V ( x 0 ) V(x_0+\Delta d(x_0))-V(x_0) V ( x 0 + Δ d ( x 0 )) − V ( x 0 ) stays negative for Δ \Delta Δ between 0 0 0 and the smaller of Δ f e a s ( x 0 ) \Delta_{\mathrm{feas}}(x_0) Δ feas ( x 0 ) and 2 ∥ d ( x 0 ) ∥ 2 2 / ⟨ d ( x 0 ) , Q d ( x 0 ) ⟩ 2\lVert d(x_0)\rVert_2^2/\langle d(x_0),Qd(x_0)\rangle 2 ∥ d ( x 0 ) ∥ 2 2 / ⟨ d ( x 0 ) , Q d ( x 0 )⟩ . In either case, V ( x 0 + Δ d ( x 0 ) ) < V ( x 0 ) V(x_0+\Delta d(x_0))<V(x_0) V ( x 0 + Δ d ( x 0 )) < V ( x 0 ) whenever Δ ∈ ( 0 , Δ + ( x 0 ) ) \Delta\in(0,\Delta_+(x_0)) Δ ∈ ( 0 , Δ + ( x 0 )) .
The evolution rule
Proposition 3.1 shows that V V V decreases along the ray Δ ↦ x 0 + Δ d ( x 0 ) \Delta\mapsto x_0+\Delta d(x_0) Δ ↦ x 0 + Δ d ( x 0 ) . The ray stays in C C C for every Δ ∈ [ 0 , Δ f e a s ( x 0 ) ] \Delta\in[0,\Delta_{\mathrm{feas}}(x_0)] Δ ∈ [ 0 , Δ feas ( x 0 )] , so the direction d ( x 0 + Δ d ( x 0 ) ) d(x_0+\Delta d(x_0)) d ( x 0 + Δ d ( x 0 )) is well-defined there. We prove next that this direction is a linear function of d ( x 0 ) d(x_0) d ( x 0 ) for small Δ > 0 \Delta>0 Δ > 0 , a result we call the evolution rule . We first show that the active set does not change along the ray, that is, J ( x 0 + Δ d ( x 0 ) ) = J ( x 0 ) J(x_0+\Delta d(x_0))=J(x_0) J ( x 0 + Δ d ( x 0 )) = J ( x 0 ) , as long as Δ \Delta Δ stays below Δ f e a s ( x 0 ) \Delta_{\mathrm{feas}}(x_0) Δ feas ( x 0 ) and
Δ J ( x 0 ) : = min i ∈ J ( x 0 ) [ p ( x 0 ) ] i max ( 0 , [ ( A J ( x 0 ) A J ( x 0 ) T ) − 1 A J ( x 0 ) Q d ( x 0 ) ] i ) . (20) \Delta_J(x_0) := \min_{i\in J(x_0)} \frac{[p(x_0)]_i}{\max\left(0,\left[(A_{J(x_0)}A_{J(x_0)}^{\mathsf T})^{-1}A_{J(x_0)}Qd(x_0)\right]_i\right)}. \tag*{(20)} Δ J ( x 0 ) := i ∈ J ( x 0 ) min max ( 0 , [ ( A J ( x 0 ) A J ( x 0 ) T ) − 1 A J ( x 0 ) Q d ( x 0 ) ] i ) [ p ( x 0 ) ] i . (20)
We set Δ J ( x 0 ) = + ∞ \Delta_J(x_0)=+\infty Δ J ( x 0 ) = + ∞ when J ( x 0 ) = ∅ J(x_0)=\varnothing J ( x 0 ) = ∅ .
Lemma 3.1. Let x 0 ∈ C x_0\in C x 0 ∈ C , choose p ( x 0 ) p(x_0) p ( x 0 ) so that (LI) holds, and assume that the strict complementarity condition (SCC) holds. Then J ( x 0 + Δ d ( x 0 ) ) = J ( x 0 ) J(x_0+\Delta d(x_0))=J(x_0) J ( x 0 + Δ d ( x 0 )) = J ( x 0 ) for every Δ ∈ [ 0 , Δ ∗ ( x 0 ) ) \Delta\in[0,\Delta_*(x_0)) Δ ∈ [ 0 , Δ ∗ ( x 0 )) , where Δ ∗ ( x 0 ) : = min ( Δ f e a s ( x 0 ) , Δ J ( x 0 ) ) \Delta_*(x_0):=\min(\Delta_{\mathrm{feas}}(x_0),\Delta_J(x_0)) Δ ∗ ( x 0 ) := min ( Δ feas ( x 0 ) , Δ J ( x 0 )) .
Proof. Let Δ ∈ ( 0 , Δ ∗ ( x 0 ) ) \Delta\in(0,\Delta_*(x_0)) Δ ∈ ( 0 , Δ ∗ ( x 0 )) . Since Δ < Δ f e a s ( x 0 ) \Delta<\Delta_{\mathrm{feas}}(x_0) Δ < Δ feas ( x 0 ) , the point x 0 + Δ d ( x 0 ) x_0+\Delta d(x_0) x 0 + Δ d ( x 0 ) lies in C C C and the direction d ( x 0 + Δ d ( x 0 ) ) d(x_0+\Delta d(x_0)) d ( x 0 + Δ d ( x 0 )) is well-defined. We first show that E ( x 0 + Δ d ( x 0 ) ) = J ( x 0 ) E(x_0+\Delta d(x_0))=J(x_0) E ( x 0 + Δ d ( x 0 )) = J ( x 0 ) . On the one hand, J ( x 0 ) ⊂ E ( x 0 ) J(x_0)\subset E(x_0) J ( x 0 ) ⊂ E ( x 0 ) because p E C ( x 0 ) ( x 0 ) = 0 p_{E^C(x_0)}(x_0)=0 p E C ( x 0 ) ( x 0 ) = 0 , so condition (13c) gives
[ A ( x 0 + Δ d ( x 0 ) ) ] J ( x 0 ) = [ A x 0 ] J ( x 0 ) + Δ [ A d ( x 0 ) ] J ( x 0 ) = b J ( x 0 ) , [A(x_0+\Delta d(x_0))]_{J(x_0)}=[Ax_0]_{J(x_0)}+\Delta[Ad(x_0)]_{J(x_0)}=b_{J(x_0)}, [ A ( x 0 + Δ d ( x 0 )) ] J ( x 0 ) = [ A x 0 ] J ( x 0 ) + Δ [ A d ( x 0 ) ] J ( x 0 ) = b J ( x 0 ) ,
that is, J ( x 0 ) ⊂ E ( x 0 + Δ d ( x 0 ) ) J(x_0)\subset E(x_0+\Delta d(x_0)) J ( x 0 ) ⊂ E ( x 0 + Δ d ( x 0 )) . On the other hand, (SCC) implies
[ A ( x 0 + Δ d ( x 0 ) ) ] E ( x 0 ) ∖ J ( x 0 ) = [ A x 0 ] E ( x 0 ) ∖ J ( x 0 ) + Δ [ A d ( x 0 ) ] E ( x 0 ) ∖ J ( x 0 ) < b E ( x 0 ) ∖ J ( x 0 ) , [A(x_0+\Delta d(x_0))]_{E(x_0)\setminus J(x_0)}=[Ax_0]_{E(x_0)\setminus J(x_0)}+\Delta[Ad(x_0)]_{E(x_0)\setminus J(x_0)}<b_{E(x_0)\setminus J(x_0)}, [ A ( x 0 + Δ d ( x 0 )) ] E ( x 0 ) ∖ J ( x 0 ) = [ A x 0 ] E ( x 0 ) ∖ J ( x 0 ) + Δ [ A d ( x 0 ) ] E ( x 0 ) ∖ J ( x 0 ) < b E ( x 0 ) ∖ J ( x 0 ) ,
while every i ∈ E C ( x 0 ) i\in E^C(x_0) i ∈ E C ( x 0 ) satisfies [ A ( x 0 + Δ d ( x 0 ) ) ] i < b i [A(x_0+\Delta d(x_0))]_i<b_i [ A ( x 0 + Δ d ( x 0 )) ] i < b i by the definition (14) of Δ f e a s ( x 0 ) , t h a t i s , \Delta_{\mathrm{feas}}(x_0), that is, Δ feas ( x 0 ) , t ha t i s , E(x_0+\Delta d(x_0))\subset J(x_0)$. Combining, we find E ( x 0 + Δ d ( x 0 ) ) = J ( x 0 ) E(x_0+\Delta d(x_0))=J(x_0) E ( x 0 + Δ d ( x 0 )) = J ( x 0 ) .
Now consider the NNLS problem (7) at x 0 + Δ d ( x 0 ) x_0+\Delta d(x_0) x 0 + Δ d ( x 0 ) ,
min p E ( x 0 + Δ d ( x 0 ) ) ≥ 0 p E C ( x 0 + Δ d ( x 0 ) ) = 0 1 2 ∥ A T p + Q ( x 0 + Δ d ( x 0 ) ) + c ∥ 2 2 , \min_{\substack{p_{E(x_0+\Delta d(x_0))}\ge0\\p_{E^C(x_0+\Delta d(x_0))}=0}} \frac{1}{2}\left\lVert A^{\mathsf T}p+Q(x_0+\Delta d(x_0))+c\right\rVert_2^2, p E ( x 0 + Δ d ( x 0 )) ≥ 0 p E C ( x 0 + Δ d ( x 0 )) = 0 min 2 1 A T p + Q ( x 0 + Δ d ( x 0 )) + c 2 2 ,
whose solutions give the direction d ( x 0 + Δ d ( x 0 ) ) d(x_0+\Delta d(x_0)) d ( x 0 + Δ d ( x 0 )) via formula (8). Let p L S ( x 0 + Δ d ( x 0 ) ) p^{\mathrm{LS}}(x_0+\Delta d(x_0)) p LS ( x 0 + Δ d ( x 0 )) minimize the same objective over the vectors supported on J ( x 0 ) J(x_0) J ( x 0 ) without the sign constraints. This minimizer is unique because the rows of A J ( x 0 ) A_{J(x_0)} A J ( x 0 ) are linearly independent. Moreover,
p J ( x 0 ) L S ( x 0 + Δ d ( x 0 ) ) = − ( A J ( x 0 ) A J ( x 0 ) T ) − 1 A J ( x 0 ) ( Q ( x 0 + Δ d ( x 0 ) ) + c ) = − ( A J ( x 0 ) A J ( x 0 ) T ) − 1 A J ( x 0 ) ( Q x 0 + c ) − Δ ( A J ( x 0 ) A J ( x 0 ) T ) − 1 A J ( x 0 ) Q d ( x 0 ) = p J ( x 0 ) ( x 0 ) − Δ ( A J ( x 0 ) A J ( x 0 ) T ) − 1 A J ( x 0 ) Q d ( x 0 ) , \begin{aligned}
p^{\mathrm{LS}}_{J(x_0)}(x_0+\Delta d(x_0))=-(A_{J(x_0)}A_{J(x_0)}^{\mathsf T})^{-1}A_{J(x_0)}(Q(x_0+\Delta d(x_0))+c) \\
&=-(A_{J(x_0)}A_{J(x_0)}^{\mathsf T})^{-1}A_{J(x_0)}(Qx_0+c)-\Delta(A_{J(x_0)}A_{J(x_0)}^{\mathsf T})^{-1}A_{J(x_0)}Qd(x_0) \\
&=p_{J(x_0)}(x_0)-\Delta(A_{J(x_0)}A_{J(x_0)}^{\mathsf T})^{-1}A_{J(x_0)}Qd(x_0),
\end{aligned} p J ( x 0 ) LS ( x 0 + Δ d ( x 0 )) = − ( A J ( x 0 ) A J ( x 0 ) T ) − 1 A J ( x 0 ) ( Q ( x 0 + Δ d ( x 0 )) + c ) = − ( A J ( x 0 ) A J ( x 0 ) T ) − 1 A J ( x 0 ) ( Q x 0 + c ) − Δ ( A J ( x 0 ) A J ( x 0 ) T ) − 1 A J ( x 0 ) Q d ( x 0 ) = p J ( x 0 ) ( x 0 ) − Δ ( A J ( x 0 ) A J ( x 0 ) T ) − 1 A J ( x 0 ) Q d ( x 0 ) ,
where we used formula (10) to obtain the last equality. Each entry of p J ( x 0 ) L S ( x 0 + Δ d ( x 0 ) ) p^{\mathrm{LS}}_{J(x_0)}(x_0+\Delta d(x_0)) p J ( x 0 ) LS ( x 0 + Δ d ( x 0 )) is affine in Δ \Delta Δ and positive at Δ = 0 \Delta=0 Δ = 0 , and it stays positive for every Δ < Δ J ( x 0 ) \Delta<\Delta_J(x_0) Δ < Δ J ( x 0 ) by (17). Since E ( x 0 + Δ d ( x 0 ) ) = J ( x 0 ) E(x_0+\Delta d(x_0))=J(x_0) E ( x 0 + Δ d ( x 0 )) = J ( x 0 ) , the vector p L S ( x 0 + Δ d ( x 0 ) ) p^{\mathrm{LS}}(x_0+\Delta d(x_0)) p LS ( x 0 + Δ d ( x 0 )) is therefore feasible for the NNLS problem at x 0 + Δ d ( x 0 ) x_0+\Delta d(x_0) x 0 + Δ d ( x 0 ) . It minimizes the objective without the sign constraints, so it solves that problem, and it is the only solution because the rows of A J ( x 0 ) A_{J(x_0)} A J ( x 0 ) are linearly independent. We conclude that J ( x 0 + Δ d ( x 0 ) ) = J ( x 0 ) J(x_0+\Delta d(x_0))=J(x_0) J ( x 0 + Δ d ( x 0 )) = J ( x 0 ) for every Δ ∈ ( 0 , Δ ∗ ( x 0 ) ) \Delta\in(0,\Delta_*(x_0)) Δ ∈ ( 0 , Δ ∗ ( x 0 )) , and trivially at Δ = 0 \Delta=0 Δ = 0 .
By Lemma 3.1, the active set stays equal to J ( x 0 ) J(x_0) J ( x 0 ) for every Δ ∈ ( 0 , Δ ∗ ( x 0 ) ) \Delta\in(0,\Delta_*(x_0)) Δ ∈ ( 0 , Δ ∗ ( x 0 )) , so the point x 0 + Δ d ( x 0 ) x_0+\Delta d(x_0) x 0 + Δ d ( x 0 ) stays in the relative interior of the face of C C C identified by J ( x 0 ) J(x_0) J ( x 0 ) . Formula (12) therefore holds along the ray with the same projection P J ( x 0 ) ⊥ P^\perp_{J(x_0)} P J ( x 0 ) ⊥ , and the evolution rule follows.
Proposition 3.2 (Evolution rule). Under the hypotheses of Lemma 3.1, we have
d ( x 0 + Δ d ( x 0 ) ) = ( I n × n − Δ P J ( x 0 ) ⊥ Q ) d ( x 0 ) for every Δ ∈ [ 0 , Δ ∗ ( x 0 ) ) . (21) d(x_0+\Delta d(x_0))=\left(I_{n\times n}-\Delta P^\perp_{J(x_0)}Q\right)d(x_0)\quad\text{for every }\Delta\in[0,\Delta_*(x_0)). \tag*{(21)} d ( x 0 + Δ d ( x 0 )) = ( I n × n − Δ P J ( x 0 ) ⊥ Q ) d ( x 0 ) for every Δ ∈ [ 0 , Δ ∗ ( x 0 )) . (21)
Proof. By Lemma 3.1, formula (12) applied at x 0 + Δ d ( x 0 ) x_0+\Delta d(x_0) x 0 + Δ d ( x 0 ) yields
d ( x 0 + Δ d ( x 0 ) ) = − P J ( x 0 + Δ d ( x 0 ) ) ⊥ ( Q ( x 0 + Δ d ( x 0 ) ) + c ) = − P J ( x 0 ) ⊥ ( Q ( x 0 + Δ d ( x 0 ) ) + c ) = − P J ( x 0 ) ⊥ ( Q x 0 + c ) − Δ P J ( x 0 ) ⊥ Q d ( x 0 ) = d ( x 0 ) − Δ P J ( x 0 ) ⊥ Q d ( x 0 ) . \begin{aligned}
d(x_0+\Delta d(x_0))&=-P^\perp_{J(x_0+\Delta d(x_0))}\left(Q(x_0+\Delta d(x_0))+c\right)\\
&=-P^\perp_{J(x_0)}\left(Q(x_0+\Delta d(x_0))+c\right)\\
&=-P^\perp_{J(x_0)}(Qx_0+c)-\Delta P^\perp_{J(x_0)}Qd(x_0)\\
&=d(x_0)-\Delta P^\perp_{J(x_0)}Qd(x_0).
\end{aligned} d ( x 0 + Δ d ( x 0 )) = − P J ( x 0 + Δ d ( x 0 )) ⊥ ( Q ( x 0 + Δ d ( x 0 )) + c ) = − P J ( x 0 ) ⊥ ( Q ( x 0 + Δ d ( x 0 )) + c ) = − P J ( x 0 ) ⊥ ( Q x 0 + c ) − Δ P J ( x 0 ) ⊥ Q d ( x 0 ) = d ( x 0 ) − Δ P J ( x 0 ) ⊥ Q d ( x 0 ) .
Rearranging yields (21) .
Local dynamics of the slow system
Proposition 3.2 describes how the direction d ( x 0 + Δ d ( x 0 ) ) d(x_0+\Delta d(x_0)) d ( x 0 + Δ d ( x 0 )) changes with Δ \Delta Δ . In particular, taking the limit Δ → 0 + \Delta\to0^+ Δ → 0 + in (21) yields
lim Δ → 0 + d ( x 0 + Δ d ( x 0 ) ) − d ( x 0 ) Δ = − P J ( x 0 ) ⊥ Q d ( x 0 ) . \lim_{\Delta\to0^+}\frac{d(x_0+\Delta d(x_0))-d(x_0)}{\Delta}=-P^\perp_{J(x_0)}Qd(x_0). Δ → 0 + lim Δ d ( x 0 + Δ d ( x 0 )) − d ( x 0 ) = − P J ( x 0 ) ⊥ Q d ( x 0 ) .
This limit suggests that d ( x ( τ ) ) = e − P J ( x 0 ) ⊥ Q τ d ( x 0 ) d(x(\tau))=e^{-P^\perp_{J(x_0)}Q\tau}d(x_0) d ( x ( τ )) = e − P J ( x 0 ) ⊥ Q τ d ( x 0 ) for small τ > 0 \tau>0 τ > 0 . We now verify this formula under the strict complementarity condition (SCC). Since x ˙ ( τ ) = d ( x ( τ ) ) \dot{x}(\tau)=d(x(\tau)) x ˙ ( τ ) = d ( x ( τ )) , integrating this formula gives the candidate solution x 0 + M ( τ ) d ( x 0 ) x_0+M(\tau)d(x_0) x 0 + M ( τ ) d ( x 0 ) , where
M ( τ ) : = ∫ 0 τ e − P J ( x 0 ) ⊥ Q s d s . (22) M(\tau):=\int_0^\tau e^{-P^\perp_{J(x_0)}Qs}\,ds. \tag*{(22)} M ( τ ) := ∫ 0 τ e − P J ( x 0 ) ⊥ Q s d s . (22)
The constraints in J ( x 0 ) J(x_0) J ( x 0 ) stay active along the candidate solution. By the identity A J ( x 0 ) P J ( x 0 ) ⊥ = 0 A_{J(x_0)}P^\perp_{J(x_0)}=0 A J ( x 0 ) P J ( x 0 ) ⊥ = 0 , every term of order k ≥ 1 k\ge1 k ≥ 1 in the series of A J ( x 0 ) e − P J ( x 0 ) ⊥ Q s A_{J(x_0)}e^{-P^\perp_{J(x_0)}Qs} A J ( x 0 ) e − P J ( x 0 ) ⊥ Q s vanishes, so condition (15) gives
A J ( x 0 ) M ( τ ) d ( x 0 ) = τ A J ( x 0 ) d ( x 0 ) = 0 for every τ ≥ 0. (23) A_{J(x_0)}M(\tau)d(x_0)=\tau A_{J(x_0)}d(x_0)=0\quad\text{for every }\tau\ge0. \tag*{(23)} A J ( x 0 ) M ( τ ) d ( x 0 ) = τ A J ( x 0 ) d ( x 0 ) = 0 for every τ ≥ 0. (23)
Formula (10) , evaluated at the candidate solution with the active set J ( x 0 ) J(x_0) J ( x 0 ) , gives the multipliers
w ( τ ) : = p J ( x 0 ) ( x 0 ) − ( A J ( x 0 ) A J ( x 0 ) T ) − 1 A J ( x 0 ) Q M ( τ ) d ( x 0 ) . (24) w(\tau):=p_{J(x_0)}(x_0)-\left(A_{J(x_0)}A_{J(x_0)}^{\mathsf{T}}\right)^{-1}A_{J(x_0)}QM(\tau)d(x_0). \tag*{(24)} w ( τ ) := p J ( x 0 ) ( x 0 ) − ( A J ( x 0 ) A J ( x 0 ) T ) − 1 A J ( x 0 ) QM ( τ ) d ( x 0 ) . (24)
The candidate solution stays in C C C , with equicorrelation set J ( x 0 ) J(x_0) J ( x 0 ) and nonnegative multipliers w ( τ ) w(\tau) w ( τ ) , until a constraint outside J ( x 0 ) J(x_0) J ( x 0 ) becomes active or a multiplier w i ( τ ) w_i(\tau) w i ( τ ) turns negative. We therefore define the feasibility time , active-set time , and breakpoint time
τ f e a s ( x 0 ) : = inf { τ > 0 : [ A ( x 0 + M ( τ ) d ( x 0 ) ) ] i = b i for some i ∈ J C ( x 0 ) } . (25) \tau_{\mathrm{feas}}(x_0):=\inf\left\{\tau>0:\left[A\left(x_0+M(\tau)d(x_0)\right)\right]_i=b_i\text{ for some }i\in J^C(x_0)\right\}. \tag*{(25)} τ feas ( x 0 ) := inf { τ > 0 : [ A ( x 0 + M ( τ ) d ( x 0 ) ) ] i = b i for some i ∈ J C ( x 0 ) } . (25)
τ J ( x 0 ) : = inf { τ > 0 : w i ( τ ) < 0 for some i ∈ J ( x 0 ) } . (26) \tau_J(x_0):=\inf\left\{\tau>0:w_i(\tau)<0\text{ for some }i\in J(x_0)\right\}. \tag*{(26)} τ J ( x 0 ) := inf { τ > 0 : w i ( τ ) < 0 for some i ∈ J ( x 0 ) } . (26)
τ ∗ ( x 0 ) : = min ( τ f e a s ( x 0 ) , τ J ( x 0 ) ) . (27) \tau_*(x_0):=\min\left(\tau_{\mathrm{feas}}(x_0),\tau_J(x_0)\right). \tag*{(27)} τ ∗ ( x 0 ) := min ( τ feas ( x 0 ) , τ J ( x 0 ) ) . (27)
The next theorem shows that τ ∗ ( x 0 ) > 0 \tau_*(x_0)>0 τ ∗ ( x 0 ) > 0 and that, before this time, the candidate solution is the slow solution, with the constraints outside J ( x 0 ) J(x_0) J ( x 0 ) inactive and the multipliers w ( τ ) w(\tau) w ( τ ) nonnegative.
Theorem 3.1. Let x 0 ∈ C x_0 \in C x 0 ∈ C , choose p ( x 0 ) p(x_0) p ( x 0 ) so that (LI) holds, and assume that (SCC) holds. Then τ ∗ ( x 0 ) > 0 \tau_*(x_0)>0 τ ∗ ( x 0 ) > 0 and the slow solution solves the initial value problem
x ˙ ( τ ) = e − P J ( x 0 ) ⊥ Q τ d ( x 0 ) , x ( 0 ) = x 0 (28) \dot{x}(\tau)=e^{-P^\perp_{J(x_0)}Q\tau}d(x_0),\qquad x(0)=x_0 \tag*{(28)} x ˙ ( τ ) = e − P J ( x 0 ) ⊥ Q τ d ( x 0 ) , x ( 0 ) = x 0 (28) for τ ∈ [ 0 , τ ∗ ( x 0 ) ) \tau\in[0,\tau_*(x_0)) τ ∈ [ 0 , τ ∗ ( x 0 )) . In particular,
x ( τ ) = x 0 + M ( τ ) d ( x 0 ) for τ ∈ [ 0 , τ ∗ ( x 0 ) ) . (29) x(\tau)=x_0+M(\tau)d(x_0)\quad\text{for }\tau\in[0,\tau_*(x_0)). \tag*{(29)} x ( τ ) = x 0 + M ( τ ) d ( x 0 ) for τ ∈ [ 0 , τ ∗ ( x 0 )) . (29) Moreover, for every τ ∈ ( 0 , τ ∗ ( x 0 ) ) \tau\in(0,\tau_*(x_0)) τ ∈ ( 0 , τ ∗ ( x 0 )) , we have E ( x ( τ ) ) = J ( x 0 ) E(x(\tau))=J(x_0) E ( x ( τ )) = J ( x 0 ) and the Moreau decomposition
− Q x ( τ ) − c = d ( x ( τ ) ) + A J ( x 0 ) T w ( τ ) , w ( τ ) ≥ 0 , (30) -Qx(\tau)-c=d(x(\tau))+A^\mathsf{T}_{J(x_0)}w(\tau),\qquad w(\tau)\geq0, \tag*{(30)} − Q x ( τ ) − c = d ( x ( τ )) + A J ( x 0 ) T w ( τ ) , w ( τ ) ≥ 0 , (30) where d ( x ( τ ) ) ∈ T C ( x ( τ ) ) d(x(\tau))\in T_C(x(\tau)) d ( x ( τ )) ∈ T C ( x ( τ )) and A J ( x 0 ) T w ( τ ) ∈ N C ( x ( τ ) ) A^\mathsf{T}_{J(x_0)}w(\tau)\in N_C(x(\tau)) A J ( x 0 ) T w ( τ ) ∈ N C ( x ( τ )) .
Proof. Let x ~ ( τ ) : = x 0 + M ( τ ) d ( x 0 ) \widetilde{x}(\tau):=x_0+M(\tau)d(x_0) x ( τ ) := x 0 + M ( τ ) d ( x 0 ) . Then x ~ ( 0 ) = x 0 \widetilde{x}(0)=x_0 x ( 0 ) = x 0 . By the fundamental theorem of calculus,
x ~ ˙ ( τ ) = M ˙ ( τ ) d ( x 0 ) = e − P J ( x 0 ) ⊥ Q τ d ( x 0 ) . \dot{\widetilde{x}}(\tau)=\dot{M}(\tau)d(x_0)=e^{-P^\perp_{J(x_0)}Q\tau}d(x_0). x ˙ ( τ ) = M ˙ ( τ ) d ( x 0 ) = e − P J ( x 0 ) ⊥ Q τ d ( x 0 ) .
In the next three steps, we prove that τ ∗ ( x 0 ) > 0 \tau_*(x_0)>0 τ ∗ ( x 0 ) > 0 , then that x ~ ( τ ) ∈ C \widetilde{x}(\tau)\in C x ( τ ) ∈ C , E ( x ~ ( τ ) ) = J ( x 0 ) E(\widetilde{x}(\tau))=J(x_0) E ( x ( τ )) = J ( x 0 ) , and d ( x ~ ( τ ) ) = − P J ( x 0 ) ⊥ ( Q x ~ ( τ ) + c ) d(\widetilde{x}(\tau))=-P^\perp_{J(x_0)}(Q\widetilde{x}(\tau)+c) d ( x ( τ )) = − P J ( x 0 ) ⊥ ( Q x ( τ ) + c ) for τ ∈ ( 0 , τ ∗ ( x 0 ) ) \tau\in(0,\tau_*(x_0)) τ ∈ ( 0 , τ ∗ ( x 0 )) , and finally that τ ↦ x ~ ( τ ) \tau\mapsto\widetilde{x}(\tau) τ ↦ x ( τ ) is the slow solution.
Step 1. Let h ( τ ) : = A x ~ ( τ ) − b h(\tau):=A\widetilde{x}(\tau)-b h ( τ ) := A x ( τ ) − b . Then h ( 0 ) = A x 0 − b h(0)=Ax_0-b h ( 0 ) = A x 0 − b and h ˙ ( 0 ) = A d ( x 0 ) \dot{h}(0)=Ad(x_0) h ˙ ( 0 ) = A d ( x 0 ) . By definition of E ( x 0 ) E(x_0) E ( x 0 ) and by (SCC),
h E c ( x 0 ) ( 0 ) < 0 , h E ( x 0 ) ∖ J ( x 0 ) ( 0 ) = 0 , h ˙ E ( x 0 ) ∖ J ( x 0 ) ( 0 ) = [ A d ( x 0 ) ] E ( x 0 ) ∖ J ( x 0 ) < 0. h_{E^c(x_0)}(0)<0,\qquad h_{E(x_0)\setminus J(x_0)}(0)=0,\qquad\dot{h}_{E(x_0)\setminus J(x_0)}(0)=[Ad(x_0)]_{E(x_0)\setminus J(x_0)}<0. h E c ( x 0 ) ( 0 ) < 0 , h E ( x 0 ) ∖ J ( x 0 ) ( 0 ) = 0 , h ˙ E ( x 0 ) ∖ J ( x 0 ) ( 0 ) = [ A d ( x 0 ) ] E ( x 0 ) ∖ J ( x 0 ) < 0.
Thus h J c ( x 0 ) ( τ ) < 0 h_{J^c(x_0)}(\tau)<0 h J c ( x 0 ) ( τ ) < 0 for sufficiently small τ > 0 \tau>0 τ > 0 , giving τ f e a s ( x 0 ) > 0 \tau_{\mathrm{feas}}(x_0)>0 τ feas ( x 0 ) > 0 . Next, the function τ ↦ w ( τ ) \tau\mapsto w(\tau) τ ↦ w ( τ ) defined in (21) is continuous with w ( 0 ) = p J ( x 0 ) ( x 0 ) > 0 w(0)=p_{J(x_0)}(x_0)>0 w ( 0 ) = p J ( x 0 ) ( x 0 ) > 0 . Hence w ( τ ) > 0 w(\tau)>0 w ( τ ) > 0 for sufficiently small τ > 0 \tau>0 τ > 0 , giving τ J ( x 0 ) > 0 \tau_J(x_0)>0 τ J ( x 0 ) > 0 . We thus find τ ∗ ( x 0 ) > 0 \tau_*(x_0)>0 τ ∗ ( x 0 ) > 0 .
Step 2a. We first show that [ A x ~ ( τ ) ] J ( x 0 ) = b J ( x 0 ) [A\widetilde{x}(\tau)]_{J(x_0)}=b_{J(x_0)} [ A x ( τ ) ] J ( x 0 ) = b J ( x 0 ) for every τ ≥ 0 \tau\geq0 τ ≥ 0 . Since J ( x 0 ) ⊂ E ( x 0 ) J(x_0)\subset E(x_0) J ( x 0 ) ⊂ E ( x 0 ) , we have A J ( x 0 ) x 0 = b J ( x 0 ) A_{J(x_0)}x_0=b_{J(x_0)} A J ( x 0 ) x 0 = b J ( x 0 ) , and (20) gives
A J ( x 0 ) x ~ ( τ ) − b J ( x 0 ) = A J ( x 0 ) ( x 0 + M ( τ ) d ( x 0 ) ) − b J ( x 0 ) = 0 , τ ≥ 0. A_{J(x_0)}\widetilde{x}(\tau)-b_{J(x_0)}
=A_{J(x_0)}\left(x_0+M(\tau)d(x_0)\right)-b_{J(x_0)}=0,\qquad\tau\geq0. A J ( x 0 ) x ( τ ) − b J ( x 0 ) = A J ( x 0 ) ( x 0 + M ( τ ) d ( x 0 ) ) − b J ( x 0 ) = 0 , τ ≥ 0.
This identity shows J ( x 0 ) ⊂ E ( x ~ ( τ ) ) J(x_0)\subset E(\widetilde{x}(\tau)) J ( x 0 ) ⊂ E ( x ( τ )) whenever x ~ ( τ ) ∈ C \widetilde{x}(\tau)\in C x ( τ ) ∈ C .
Step 2b. Conversely, h J c ( x 0 ) h_{J^c(x_0)} h J c ( x 0 ) is continuous and negative near zero by Step 1, and none of its components vanishes on ( 0 , τ f e a s ( x 0 ) ) (0,\tau_{\mathrm{feas}}(x_0)) ( 0 , τ feas ( x 0 )) by definition (22a). By the intermediate value theorem, each of its components is negative on ( 0 , τ f e a s ( x 0 ) ) (0,\tau_{\mathrm{feas}}(x_0)) ( 0 , τ feas ( x 0 )) . Now let τ ∈ ( 0 , τ ∗ ( x 0 ) ) \tau\in(0,\tau_*(x_0)) τ ∈ ( 0 , τ ∗ ( x 0 )) . Since τ ∗ ( x 0 ) ≤ τ f e a s ( x 0 ) \tau_*(x_0)\leq\tau_{\mathrm{feas}}(x_0) τ ∗ ( x 0 ) ≤ τ feas ( x 0 ) , we have h J c ( x 0 ) ( τ ) < 0 h_{J^c(x_0)}(\tau)<0 h J c ( x 0 ) ( τ ) < 0 , so Step 2a shows that x ~ ( τ ) ∈ C \widetilde{x}(\tau)\in C x ( τ ) ∈ C and E ( x ~ ( τ ) ) ⊂ J ( x 0 ) E(\widetilde{x}(\tau))\subset J(x_0) E ( x ( τ )) ⊂ J ( x 0 ) . We conclude that E ( x ~ ( τ ) ) = J ( x 0 ) E(\widetilde{x}(\tau))=J(x_0) E ( x ( τ )) = J ( x 0 ) for all τ ∈ ( 0 , τ ∗ ( x 0 ) ) \tau\in(0,\tau_*(x_0)) τ ∈ ( 0 , τ ∗ ( x 0 )) .
Step 2c. We show that d ( x ~ ( τ ) ) = − P J ( x 0 ) ⊥ ( Q x ~ ( τ ) + c ) d(\widetilde{x}(\tau))=-P^\perp_{J(x_0)}(Q\widetilde{x}(\tau)+c) d ( x ( τ )) = − P J ( x 0 ) ⊥ ( Q x ( τ ) + c ) for all τ ∈ [ 0 , τ ∗ ( x 0 ) ) \tau\in[0,\tau_*(x_0)) τ ∈ [ 0 , τ ∗ ( x 0 )) . At τ = 0 \tau=0 τ = 0 , this is formula (12), so let τ ∈ ( 0 , τ ∗ ( x 0 ) ) \tau\in(0,\tau_*(x_0)) τ ∈ ( 0 , τ ∗ ( x 0 )) . Splitting − Q x ~ ( τ ) − c -Q\widetilde{x}(\tau)-c − Q x ( τ ) − c along I n × n = P J ( x 0 ) ⊥ + ( I n × n − P J ( x 0 ) ⊥ ) I_{n\times n}=P^\perp_{J(x_0)}+(I_{n\times n}-P^\perp_{J(x_0)}) I n × n = P J ( x 0 ) ⊥ + ( I n × n − P J ( x 0 ) ⊥ ) and expanding x ~ ( τ ) = x 0 + M ( τ ) d ( x 0 ) \widetilde{x}(\tau)=x_0+M(\tau)d(x_0) x ( τ ) = x 0 + M ( τ ) d ( x 0 ) with formulas (10) and (21) gives the decomposition
− Q x ~ ( τ ) − c = − P J ( x 0 ) ⊥ ( Q x ~ ( τ ) + c ) + A J ( x 0 ) T w ( τ ) . -Q\widetilde{x}(\tau)-c=-P^\perp_{J(x_0)}(Q\widetilde{x}(\tau)+c)+A^\mathsf{T}_{J(x_0)}w(\tau). − Q x ( τ ) − c = − P J ( x 0 ) ⊥ ( Q x ( τ ) + c ) + A J ( x 0 ) T w ( τ ) .
Since E ( x ~ ( τ ) ) = J ( x 0 ) E(\widetilde{x}(\tau))=J(x_0) E ( x ( τ )) = J ( x 0 ) and w ( τ ) ≥ 0 w(\tau)\geq0 w ( τ ) ≥ 0 by definition (22b) of τ J ( x 0 ) \tau_J(x_0) τ J ( x 0 ) , the second term on the right-hand side is a nonnegative combination of the rows active at x ~ ( τ ) \widetilde{x}(\tau) x ( τ ) , so it belongs to N C ( x ~ ( τ ) ) N_C(\widetilde{x}(\tau)) N C ( x ( τ )) . The first term lies in ker A J ( x 0 ) \ker A_{J(x_0)} ker A J ( x 0 ) because A J ( x 0 ) P J ( x 0 ) ⊥ = 0 A_{J(x_0)}P^\perp_{J(x_0)}=0 A J ( x 0 ) P J ( x 0 ) ⊥ = 0 , so it belongs to T C ( x ~ ( τ ) ) T_C(\widetilde{x}(\tau)) T C ( x ( τ )) and is orthogonal to the second term. By Moreau’s decomposition theorem (6), the first term is therefore proj T C ( x ~ ( τ ) ) ( − Q x ~ ( τ ) − c ) = d ( x ~ ( τ ) ) \operatorname{proj}_{T_C(\widetilde{x}(\tau))}(-Q\widetilde{x}(\tau)-c)=d(\widetilde{x}(\tau)) proj T C ( x ( τ )) ( − Q x ( τ ) − c ) = d ( x ( τ )) , as claimed.
Step 3. We now verify that τ ↦ x ~ ( τ ) \tau\mapsto\widetilde{x}(\tau) τ ↦ x ( τ ) solves the slow system (5). Step 2c and formula (12) give, for all τ ∈ [ 0 , τ ∗ ( x 0 ) ) \tau\in[0,\tau_{*}(x_{0})) τ ∈ [ 0 , τ ∗ ( x 0 )) ,
d ( x ~ ( τ ) ) = − P J ( x 0 ) ⊥ ( Q x 0 + c ) − P J ( x 0 ) ⊥ Q M ( τ ) d ( x 0 ) = ( I n × n − P J ( x 0 ) ⊥ Q M ( τ ) ) d ( x 0 ) = e − P J ( x 0 ) ⊥ Q τ d ( x 0 ) = x ~ ˙ ( τ ) , \begin{aligned}
d(\widetilde{x}(\tau))=-P_{J(x_{0})}^{\perp}(Qx_{0}+c)-P_{J(x_{0})}^{\perp}QM(\tau)d(x_{0}) \\
&=\left(I_{n\times n}-P_{J(x_{0})}^{\perp}QM(\tau)\right)d(x_{0})=e^{-P_{J(x_{0})}^{\perp}Q\tau}d(x_{0})=\dot{\widetilde{x}}(\tau),
\end{aligned} d ( x ( τ )) = − P J ( x 0 ) ⊥ ( Q x 0 + c ) − P J ( x 0 ) ⊥ QM ( τ ) d ( x 0 ) = ( I n × n − P J ( x 0 ) ⊥ QM ( τ ) ) d ( x 0 ) = e − P J ( x 0 ) ⊥ Q τ d ( x 0 ) = x ˙ ( τ ) ,
where the third equality follows from the fundamental theorem of calculus. Hence the candidate solution τ ↦ x ~ ( τ ) \tau\mapsto\widetilde{x}(\tau) τ ↦ x ( τ ) solves the slow system on [ 0 , τ ∗ ( x 0 ) ) [0,\tau_{*}(x_{0})) [ 0 , τ ∗ ( x 0 )) . Both τ ↦ x ( τ ) \tau\mapsto x(\tau) τ ↦ x ( τ ) and τ ↦ x ~ ( τ ) \tau\mapsto\widetilde{x}(\tau) τ ↦ x ( τ ) therefore solve the differential inclusion (4) on [ 0 , τ ∗ ( x 0 ) ) [0,\tau_{*}(x_{0})) [ 0 , τ ∗ ( x 0 )) and agree at τ = 0 \tau=0 τ = 0 . The locally Lipschitz function τ ↦ ∥ x ( τ ) − x ~ ( τ ) ∥ 2 2 \tau\mapsto\|x(\tau)-\widetilde{x}(\tau)\|_{2}^{2} τ ↦ ∥ x ( τ ) − x ( τ ) ∥ 2 2 is differentiable at almost every τ \tau τ , with derivative 2 ⟨ x ˙ ( τ ) − x ~ ˙ ( τ ) , x ( τ ) − x ~ ( τ ) ⟩ ≤ 0 2\langle\dot{x}(\tau)-\dot{\widetilde{x}}(\tau),x(\tau)-\widetilde{x}(\tau)\rangle\leq0 2 ⟨ x ˙ ( τ ) − x ˙ ( τ ) , x ( τ ) − x ( τ )⟩ ≤ 0 because ∂ x V \partial_{x}V ∂ x V is monotone. This nonnegative function vanishes at τ = 0 \tau=0 τ = 0 , so it vanishes identically because it is nonincreasing, and x ( τ ) = x ~ ( τ ) = x 0 + M ( τ ) d ( x 0 ) x(\tau)=\widetilde{x}(\tau)=x_{0}+M(\tau)d(x_{0}) x ( τ ) = x ( τ ) = x 0 + M ( τ ) d ( x 0 ) on [ 0 , τ ∗ ( x 0 ) ) [0,\tau_{*}(x_{0})) [ 0 , τ ∗ ( x 0 )) . The identity E ( x ( τ ) ) = J ( x 0 ) \mathcal{E}(x(\tau))=J(x_{0}) E ( x ( τ )) = J ( x 0 ) and formula (25) then follow from Steps 2b and 2c.
Continuation of the local dynamics
The feasibility and active-set times τ f e a s ( x 0 ) \tau_{\mathrm{feas}}(x_{0}) τ feas ( x 0 ) and τ J ( x 0 ) \tau_{J}(x_{0}) τ J ( x 0 ) , when finite, indicate when the equicorrelation set grows or the active set may shrink. Since τ ↦ x ( τ ) \tau\mapsto x(\tau) τ ↦ x ( τ ) is continuous by Theorem 2.1, formula (24) extends to τ ∗ ( x 0 ) \tau_{*}(x_{0}) τ ∗ ( x 0 ) whenever τ ∗ ( x 0 ) < + ∞ \tau_{*}(x_{0})<+\infty τ ∗ ( x 0 ) < + ∞ . The right derivative τ ↦ x ˙ ( τ ) = d ( x ( τ ) ) \tau\mapsto\dot{x}(\tau)=d(x(\tau)) τ ↦ x ˙ ( τ ) = d ( x ( τ )) , however, need not be continuous at τ ∗ ( x 0 ) \tau_{*}(x_{0}) τ ∗ ( x 0 ) , as the next result demonstrates.
Proposition 3.3. Let x 0 ∈ C x_{0} \in C x 0 ∈ C , choose p ( x 0 ) p(x_{0}) p ( x 0 ) so that (LI) holds, assume that (SCC) holds, and suppose τ ∗ ( x 0 ) < + ∞ \tau_{*}(x_{0})<+\infty τ ∗ ( x 0 ) < + ∞ . If [ A e − P J ( x 0 ) ⊥ Q τ ∗ ( x 0 ) d ( x 0 ) ] i > 0 \left[Ae^{-P_{J(x_{0})}^{\perp}Q\tau_{*}(x_{0})}d(x_{0})\right]_{i}>0 [ A e − P J ( x 0 ) ⊥ Q τ ∗ ( x 0 ) d ( x 0 ) ] i > 0 for some i ∈ E ( x ( τ ∗ ( x 0 ) ) ) i \in\mathcal{E}(x(\tau_{*}(x_{0}))) i ∈ E ( x ( τ ∗ ( x 0 ))) , then the right derivative τ ↦ x ˙ ( τ ) = d ( x ( τ ) ) \tau\mapsto\dot{x}(\tau)=d(x(\tau)) τ ↦ x ˙ ( τ ) = d ( x ( τ )) is discontinuous at τ ∗ ( x 0 ) \tau_{*}(x_{0}) τ ∗ ( x 0 ) , and
∥ d ( x ( τ ∗ ( x 0 ) ) ) ∥ 2 < ∥ e − P J ( x 0 ) ⊥ Q τ ∗ ( x 0 ) d ( x 0 ) ∥ 2 , (31) \|d(x(\tau_{*}(x_{0})))\|_{2}<\left\|e^{-P_{J(x_{0})}^{\perp}Q\tau_{*}(x_{0})}d(x_{0})\right\|_{2}, \tag*{(31)} ∥ d ( x ( τ ∗ ( x 0 ))) ∥ 2 < e − P J ( x 0 ) ⊥ Q τ ∗ ( x 0 ) d ( x 0 ) 2 , (31) where the right-hand side is the norm of the left limit of d ( x ( τ ) ) d(x(\tau)) d ( x ( τ )) as τ → τ ∗ ( x 0 ) − \tau\to\tau_{*}(x_{0})^{-} τ → τ ∗ ( x 0 ) − . Otherwise, the right derivative is continuous.
Proof. Let τ ∗ = τ ∗ ( x 0 ) \tau_{*}=\tau_{*}(x_{0}) τ ∗ = τ ∗ ( x 0 ) , x ∗ = x ( τ ∗ ) x_{*}=x(\tau_{*}) x ∗ = x ( τ ∗ ) , and E ∗ = E ( x ∗ ) \mathcal{E}_{*}=\mathcal{E}(x_{*}) E ∗ = E ( x ∗ ) , for short.
Step 1. By Theorem 3.1, E ( x ( τ ) ) = J ( x 0 ) \mathcal{E}(x(\tau))=J(x_{0}) E ( x ( τ )) = J ( x 0 ) for all τ ∈ ( 0 , τ ∗ ) \tau\in(0,\tau_{*}) τ ∈ ( 0 , τ ∗ ) . Let
I + : = { i ∈ J C ( x 0 ) : [ A x ∗ ] i = b i } . I^{+}:=\left\{i\in J^{C}(x_{0}):[Ax_{*}]_{i}=b_{i}\right\}. I + := { i ∈ J C ( x 0 ) : [ A x ∗ ] i = b i } .
so that E ∗ = J ( x 0 ) ∪ I + \mathcal{E}_{*}=J(x_{0})\cup I^{+} E ∗ = J ( x 0 ) ∪ I + . Indeed, x ∗ = x 0 + M ( τ ∗ ) d ( x 0 ) x_{*}=x_{0}+M(\tau_{*})d(x_{0}) x ∗ = x 0 + M ( τ ∗ ) d ( x 0 ) by continuity, so A J ( x 0 ) x 0 = b J ( x 0 ) A_{J(x_{0})}x_{0}=b_{J(x_{0})} A J ( x 0 ) x 0 = b J ( x 0 ) and (20) give A J ( x 0 ) x ∗ = b J ( x 0 ) A_{J(x_{0})}x_{*}=b_{J(x_{0})} A J ( x 0 ) x ∗ = b J ( x 0 ) . Define p ^ \hat{p} p ^ by p ^ J ( x 0 ) : = w ( τ ∗ ) \hat{p}_{J(x_{0})}:=w(\tau_{*}) p ^ J ( x 0 ) := w ( τ ∗ ) and p ^ J C ( x 0 ) : = 0 \hat{p}_{J^{C}(x_{0})}:=0 p ^ J C ( x 0 ) := 0 . Since w ( τ ) ≥ 0 w(\tau)\geq0 w ( τ ) ≥ 0 for τ ∈ ( 0 , τ ∗ ) \tau\in(0,\tau_{*}) τ ∈ ( 0 , τ ∗ ) by (25) and w w w is continuous, p ^ ≥ 0 \hat{p}\geq0 p ^ ≥ 0 . Because p ^ \hat{p} p ^ is supported on J ( x 0 ) ⊂ E ∗ J(x_{0})\subset\mathcal{E}_{*} J ( x 0 ) ⊂ E ∗ , it is a feasible solution of the NNLS problem (7) at x ∗ x_{*} x ∗ . Letting τ → τ ∗ − \tau\to\tau_{*}^{-} τ → τ ∗ − in (25) and (23) gives
− A T p ^ − Q x ∗ − c = ( I n × n − P J ( x 0 ) ⊥ Q M ( τ ∗ ) ) d ( x 0 ) = e − P J ( x 0 ) ⊥ Q τ ∗ d ( x 0 ) . (32) -A^{\mathsf{T}}\hat{p}-Qx_{*}-c=\left(I_{n\times n}-P_{J(x_{0})}^{\perp}QM(\tau_{*})\right)d(x_{0})=e^{-P_{J(x_{0})}^{\perp}Q\tau_{*}}d(x_{0}). \tag*{(32)} − A T p ^ − Q x ∗ − c = ( I n × n − P J ( x 0 ) ⊥ QM ( τ ∗ ) ) d ( x 0 ) = e − P J ( x 0 ) ⊥ Q τ ∗ d ( x 0 ) . (32)
Step 2. We now determine when p ^ \hat{p} p ^ is an optimal solution of the NNLS problem (7) at x ∗ x_{*} x ∗ . By Section 2.5, p ^ \hat{p} p ^ is optimal if and only if the KKT conditions (13a)–(13b) hold for its residual − A T p ^ − Q x ∗ − c -A^{\mathsf{T}}\hat{p}-Qx_{*}-c − A T p ^ − Q x ∗ − c . On the one hand, the sign condition (13a) reads
A E ∗ ( − A T p ^ − Q x ∗ − c ) = A E ∗ e − P J ( x 0 ) ⊥ Q τ ∗ d ( x 0 ) ≤ 0. (33) A_{\mathcal{E}_{*}}\left(-A^{\mathsf{T}}\hat{p}-Qx_{*}-c\right)=A_{\mathcal{E}_{*}}e^{-P_{J(x_{0})}^{\perp}Q\tau_{*}}d(x_{0})\leq0. \tag*{(33)} A E ∗ ( − A T p ^ − Q x ∗ − c ) = A E ∗ e − P J ( x 0 ) ⊥ Q τ ∗ d ( x 0 ) ≤ 0. (33)
with equality over J ( x 0 ) ⊂ E ∗ J(x_0) \subset\mathcal{E}_* J ( x 0 ) ⊂ E ∗ because A J ( x 0 ) e − P J ( x 0 ) ⊥ Q s d ( x 0 ) = 0 A_{J(x_0)}e^{-P^\perp_{J(x_0)}Q s}d(x_0)=0 A J ( x 0 ) e − P J ( x 0 ) ⊥ Q s d ( x 0 ) = 0 for all s ≥ 0 s\geq0 s ≥ 0 , as differentiating (20) shows. Since p ^ \hat p p ^ is supported on J ( x 0 ) J(x_0) J ( x 0 ) , this equality settles the complementarity condition (13b), so p ^ \hat p p ^ is optimal if and only if (28) holds. On the other hand, for every i ∈ I + i\in I^+ i ∈ I + , the function τ ↦ [ A ( x 0 + M ( τ ) d ( x 0 ) ) − b ] i \tau\mapsto[A(x_0+M(\tau)d(x_0))-b]_i τ ↦ [ A ( x 0 + M ( τ ) d ( x 0 )) − b ] i is negative on ( 0 , τ ∗ ) (0,\tau_*) ( 0 , τ ∗ ) by Step 1 and vanishes at τ ∗ \tau_* τ ∗ . Thus the derivative of the function at τ ∗ \tau_* τ ∗ is nonnegative, yielding
d d τ ∣ τ = τ ∗ [ A ( x 0 + M ( τ ) d ( x 0 ) ) ] i = [ A e − P J ( x 0 ) ⊥ Q τ ∗ d ( x 0 ) ] i ≥ 0. \left.\frac{d}{d\tau}\right|_{\tau=\tau_*} [A(x_0+M(\tau)d(x_0))]_i=[Ae^{-P^\perp_{J(x_0)}Q\tau_*}d(x_0)]_i\geq0. d τ d τ = τ ∗ [ A ( x 0 + M ( τ ) d ( x 0 )) ] i = [ A e − P J ( x 0 ) ⊥ Q τ ∗ d ( x 0 ) ] i ≥ 0.
Since [ A e − P J ( x 0 ) ⊥ Q τ ∗ d ( x 0 ) ] J ( x 0 ) = 0 [Ae^{-P^\perp_{J(x_0)}Q\tau_*}d(x_0)]_{J(x_0)}=0 [ A e − P J ( x 0 ) ⊥ Q τ ∗ d ( x 0 ) ] J ( x 0 ) = 0 and E ∗ = J ( x 0 ) ∪ I + \mathcal{E}_*=J(x_0)\cup I^+ E ∗ = J ( x 0 ) ∪ I + , condition (28) holds if and only if [ A e − P J ( x 0 ) ⊥ Q τ ∗ d ( x 0 ) ] I + = 0 [Ae^{-P^\perp_{J(x_0)}Q\tau_*}d(x_0)]_{I^+}=0 [ A e − P J ( x 0 ) ⊥ Q τ ∗ d ( x 0 ) ] I + = 0 . Therefore, p ^ \hat p p ^ is an optimal NNLS solution at x ∗ x_* x ∗ if and only if the hypothesis of the proposition fails.
Step 3a. If the hypothesis of the proposition fails, then Step 2 shows p ^ \hat p p ^ is an optimal NNLS solution. In that case,
x ˙ ( τ ∗ ) = d ( x ∗ ) = − ( A ⊤ p ^ + Q x ∗ + c ) = e − P J ( x 0 ) ⊥ Q τ ∗ d ( x 0 ) , \dot{x}(\tau_*)=d(x_*)=-\left(A^\top\hat p+Qx_*+c\right)=e^{-P^\perp_{J(x_0)}Q\tau_*}d(x_0), x ˙ ( τ ∗ ) = d ( x ∗ ) = − ( A ⊤ p ^ + Q x ∗ + c ) = e − P J ( x 0 ) ⊥ Q τ ∗ d ( x 0 ) ,
where the last equality is (27). By (23), the right-hand side is also the left limit of x ˙ ( τ ) \dot{x}(\tau) x ˙ ( τ ) as τ → τ ∗ − \tau\to\tau_*^- τ → τ ∗ − . Hence τ ↦ x ˙ ( τ ) \tau\mapsto\dot{x}(\tau) τ ↦ x ˙ ( τ ) is continuous at τ ∗ \tau_* τ ∗ .
Step 3b. If the hypothesis of the proposition holds, then Step 2 shows that p ^ \hat p p ^ is not an optimal NNLS solution. Since p ^ \hat p p ^ is feasible but not optimal, its objective value exceeds the optimal value 1 2 ∥ d ( x ∗ ) ∥ 2 2 \frac{1}{2}\|d(x_*)\|_2^2 2 1 ∥ d ( x ∗ ) ∥ 2 2 , and (27) yields ∥ d ( x ∗ ) ∥ 2 < ∥ A ⊤ p ^ + Q x ∗ + c ∥ 2 = ∥ e − P J ( x 0 ) ⊥ Q τ ∗ d ( x 0 ) ∥ 2 \|d(x_*)\|_2<\|A^\top\hat p+Qx_*+c\|_2=\|e^{-P^\perp_{J(x_0)}Q\tau_*}d(x_0)\|_2 ∥ d ( x ∗ ) ∥ 2 < ∥ A ⊤ p ^ + Q x ∗ + c ∥ 2 = ∥ e − P J ( x 0 ) ⊥ Q τ ∗ d ( x 0 ) ∥ 2 . This gives (26). By (23), the right-hand side of (26) is the norm of the left limit of x ˙ ( τ ) \dot{x}(\tau) x ˙ ( τ ) as τ → τ ∗ − \tau\to\tau_*^- τ → τ ∗ − , so this left limit differs from x ˙ ( τ ∗ ) = d ( x ∗ ) \dot{x}(\tau_*)=d(x_*) x ˙ ( τ ∗ ) = d ( x ∗ ) and τ ↦ x ˙ ( τ ) \tau\mapsto\dot{x}(\tau) τ ↦ x ˙ ( τ ) is discontinuous at τ ∗ . \tau_*. τ ∗ .
Computing the breakpoint time
Theorem 3.1 and Proposition 3.3 describe the slow solution up to and at the breakpoint time τ ∗ ( x 0 ) \tau_*(x_0) τ ∗ ( x 0 ) . Computing this time requires finding the first zero of the function
τ ↦ [ A ( x 0 + M ( τ ) d ( x 0 ) ) − b ] i , i ∈ J c ( x 0 ) , \tau\mapsto[A(x_0+M(\tau)d(x_0))-b]_i,\qquad i\in J^c(x_0), τ ↦ [ A ( x 0 + M ( τ ) d ( x 0 )) − b ] i , i ∈ J c ( x 0 ) ,
and the first time that a component of w ( τ ) w(\tau) w ( τ ) in (21) turns negative, if any exist. Condition (13c) gives d ( x 0 ) ∈ ker ( A J ) d(x_0)\in\ker(A_J) d ( x 0 ) ∈ ker ( A J ) (where we write J : = J ( x 0 ) J:=J(x_0) J := J ( x 0 ) for short). This kernel is the range of P J ⊥ P_J^\perp P J ⊥ and is invariant under P J ⊥ Q P_J^\perp Q P J ⊥ Q , which agrees on it with the SPSD matrix P J ⊥ Q P J ⊥ P_J^\perp QP_J^\perp P J ⊥ Q P J ⊥ . Expanding d ( x 0 ) d(x_0) d ( x 0 ) in an orthonormal basis of ker ( A J ) \ker(A_J) ker ( A J ) made of eigenvectors of P J ⊥ Q P J ⊥ P_J^\perp QP_J^\perp P J ⊥ Q P J ⊥ , with eigenvalues λ k ≥ 0 \lambda_k\geq0 λ k ≥ 0 , and integrating as in (19) shows that each component of M ( τ ) d ( x 0 ) M(\tau)d(x_0) M ( τ ) d ( x 0 ) is a linear combination of terms of the form
1 − e − λ k τ λ k (for λ k > 0 ) and τ (for λ k = 0 ) . \frac{1-e^{-\lambda_k\tau}}{\lambda_k}\ \text{(for }\lambda_k>0\text{) and }\tau\ \text{(for }\lambda_k=0\text{)}. λ k 1 − e − λ k τ (for λ k > 0 ) and τ (for λ k = 0 ) .
Thus the equations for τ f e a s ( x 0 ) \tau_{\mathrm{feas}}(x_0) τ feas ( x 0 ) and τ J ( x 0 ) \tau_J(x_0) τ J ( x 0 ) generally have no closed-form solution. These equations, however, have closed-form solutions when d ( x 0 ) d(x_0) d ( x 0 ) is an eigenvector of P J ⊥ Q P_J^\perp Q P J ⊥ Q , since each then involves a single term. The direction d ( x 0 ) d(x_0) d ( x 0 ) is such an eigenvector when Q = t I n × n Q=tI_{n\times n} Q = t I n × n with t ≥ 0 t\geq0 t ≥ 0 , because d ( x 0 ) ∈ ker ( A J ) d(x_0)\in\ker(A_J) d ( x 0 ) ∈ ker ( A J ) gives P J ⊥ Q d ( x 0 ) = t d ( x 0 ) P_J^\perp Qd(x_0)=td(x_0) P J ⊥ Q d ( x 0 ) = t d ( x 0 ) . Thus e − P J ⊥ Q τ d ( x 0 ) = e − t τ d ( x 0 ) e^{-P_J^\perp Q\tau}d(x_0)=e^{-t\tau}d(x_0) e − P J ⊥ Q τ d ( x 0 ) = e − t τ d ( x 0 ) and the trajectory of the slow solution on [ 0 , τ ∗ ( x 0 ) ) [0,\tau_*(x_0)) [ 0 , τ ∗ ( x 0 )) is
x ( τ ) = x 0 + Δ ( τ ) d ( x 0 ) , with Δ ( τ ) : = { τ if t = 0 , ( 1 − e − t τ ) / t if t > 0. (34) x(\tau)=x_0+\Delta(\tau)d(x_0),\qquad\text{with }\Delta(\tau):=
\begin{cases}
\tau& \text{if }t=0,\\
(1-e^{-t\tau})/t & \text{if }t>0.
\end{cases} \tag*{(34)} x ( τ ) = x 0 + Δ ( τ ) d ( x 0 ) , with Δ ( τ ) := { τ ( 1 − e − t τ ) / t if t = 0 , if t > 0. (34)
For the active-set time τ J ( x 0 ) \tau_J(x_0) τ J ( x 0 ) , we use M ( τ ) d ( x 0 ) = Δ ( τ ) d ( x 0 ) M(\tau)d(x_0)=\Delta(\tau)d(x_0) M ( τ ) d ( x 0 ) = Δ ( τ ) d ( x 0 ) and d ( x 0 ) ∈ ker ( A J ) d(x_0)\in\ker(A_J) d ( x 0 ) ∈ ker ( A J ) to get
w ( τ ) = p J ( x 0 ) − t Δ ( τ ) ( A J A J T ) − 1 A J d ( x 0 ) = p J ( x 0 ) > 0 w(\tau)=p_J(x_0)-t\Delta(\tau)\left(A_JA_J^{\mathsf{T}}\right)^{-1}A_Jd(x_0)=p_J(x_0)>0 w ( τ ) = p J ( x 0 ) − t Δ ( τ ) ( A J A J T ) − 1 A J d ( x 0 ) = p J ( x 0 ) > 0
for all τ > 0 \tau>0 τ > 0 . Hence we find τ J ( x 0 ) = + ∞ \tau_J(x_0)=+\infty τ J ( x 0 ) = + ∞ . For the feasibility time τ f e a s ( x 0 ) \tau_{\mathrm{feas}}(x_0) τ feas ( x 0 ) , each constraint i ∈ E ( x 0 ) ∖ J i\in\mathcal{E}(x_0)\setminus J i ∈ E ( x 0 ) ∖ J satisfies [ A ( x 0 + Δ ( τ ) d ( x 0 ) ) − b ] i = Δ ( τ ) [ A d ( x 0 ) ] i < 0 [A(x_0+\Delta(\tau)d(x_0))-b]_i=\Delta(\tau)[Ad(x_0)]_i<0 [ A ( x 0 + Δ ( τ ) d ( x 0 )) − b ] i = Δ ( τ ) [ A d ( x 0 ) ] i < 0 for τ > 0 \tau>0 τ > 0 by (SCC). By (17) and (25) , τ f e a s ( x 0 ) \tau_{\mathrm{feas}}(x_0) τ feas ( x 0 ) is therefore the solution of Δ ( τ ) = Δ f e a s ( x 0 ) \Delta(\tau)=\Delta_{\mathrm{feas}}(x_0) Δ ( τ ) = Δ feas ( x 0 ) , if one exists. Since τ ↦ Δ ( τ ) \tau\mapsto\Delta(\tau) τ ↦ Δ ( τ ) is strictly increasing with Δ ( 0 ) = 0 \Delta(0)=0 Δ ( 0 ) = 0 and, for t > 0 t>0 t > 0 , Δ ( τ ) → 1 / t \Delta(\tau)\to1/t Δ ( τ ) → 1/ t as τ → + ∞ \tau\to+\infty τ → + ∞ , the feasibility time is finite if and only if Δ f e a s ( x 0 ) < 1 / t \Delta_{\mathrm{feas}}(x_0)<1/t Δ feas ( x 0 ) < 1/ t . When this holds,
τ f e a s ( x 0 ) = { Δ f e a s ( x 0 ) if t = 0 , − log ( 1 − t Δ f e a s ( x 0 ) ) / t if t > 0 , (35) \tau_{\mathrm{feas}}(x_0)=
\begin{cases}
\Delta_{\mathrm{feas}}(x_0) & \text{if } t=0,\\
-\log(1-t\Delta_{\mathrm{feas}}(x_0))/t & \text{if } t>0,
\end{cases} \tag*{(35)} τ feas ( x 0 ) = { Δ feas ( x 0 ) − log ( 1 − t Δ feas ( x 0 )) / t if t = 0 , if t > 0 , (35)
and τ ∗ ( x 0 ) = τ f e a s ( x 0 ) \tau_*(x_0)=\tau_{\mathrm{feas}}(x_0) τ ∗ ( x 0 ) = τ feas ( x 0 ) . If Δ f e a s ( x 0 ) ≥ 1 / t \Delta_{\mathrm{feas}}(x_0)\ge1/t Δ feas ( x 0 ) ≥ 1/ t , then τ f e a s ( x 0 ) = + ∞ \tau_{\mathrm{feas}}(x_0)=+\infty τ feas ( x 0 ) = + ∞ , so τ ∗ ( x 0 ) = + ∞ \tau_*(x_0)=+\infty τ ∗ ( x 0 ) = + ∞ and (34) holds for all τ ≥ 0 \tau\ge0 τ ≥ 0 . For t > 0 t>0 t > 0 , x ( τ ) → x 0 + d ( x 0 ) / t x(\tau)\to x_0+d(x_0)/t x ( τ ) → x 0 + d ( x 0 ) / t as τ → + ∞ \tau\to+\infty τ → + ∞ . For t = 0 t=0 t = 0 and d ( x 0 ) ≠ 0 d(x_0)\ne0 d ( x 0 ) = 0 , formula (15) gives V ( x ( τ ) ) = V ( x 0 ) − τ ∥ d ( x 0 ) ∥ 2 → − ∞ V(x(\tau))=V(x_0)-\tau\lVert d(x_0)\rVert^2\to-\infty V ( x ( τ )) = V ( x 0 ) − τ ∥ d ( x 0 ) ∥ 2 → − ∞ .
Removing strict complementarity
All results so far except Proposition 3.1 assume the strict complementarity condition (SCC). When (SCC) fails, the grazing set
G ( x 0 ) : = { i ∈ E ( x 0 ) ∖ J ( x 0 ) : [ A d ( x 0 ) ] i = 0 } (36) G(x_0):=\{i\in\mathcal{E}(x_0)\setminus J(x_0):[Ad(x_0)]_i=0\} \tag*{(36)} G ( x 0 ) := { i ∈ E ( x 0 ) ∖ J ( x 0 ) : [ A d ( x 0 ) ] i = 0 } (36)
is nonempty. A grazing constraint stays active on the ray x 0 + Δ d ( x 0 ) x_0+\Delta d(x_0) x 0 + Δ d ( x 0 ) . Along the slow solution, it may join the active set, become inactive, or stay active with a zero multiplier. The direction d ( x 0 ) d(x_0) d ( x 0 ) on its own does not determine which of these occurs. The rate at which the direction d ( x ( τ ) ) d(x(\tau)) d ( x ( τ )) changes at τ = 0 \tau=0 τ = 0 is the projection of − Q d ( x 0 ) -Qd(x_0) − Q d ( x 0 ) onto the cone { s ∈ R n : [ A s ] J ( x 0 ) = 0 , [ A s ] G ( x 0 ) ≤ 0 } \{s\in\mathbb{R}^n:[As]_{J(x_0)}=0,\ [As]_{G(x_0)}\le0\} { s ∈ R n : [ A s ] J ( x 0 ) = 0 , [ A s ] G ( x 0 ) ≤ 0 } . A grazing constraint joins the active set if some multiplier of this projection is positive on it, and becomes inactive if the projection strictly decreases its value. Otherwise, the higher derivatives of d ( x ( τ ) ) d(x(\tau)) d ( x ( τ )) at τ = 0 \tau=0 τ = 0 determine which case occurs.
The next theorem, whose proof is highly technical and relegated to Appendix B.1, extends the results of § 3.2 –§ 3.3 to the case where (SCC) fails. Complementarity (14) holds for every optimal NNLS solution, so the set J ( x 0 ) ∪ G ( x 0 ) = { i ∈ E ( x 0 ) : [ A d ( x 0 ) ] i = 0 } J(x_0)\cup G(x_0)=\{i\in\mathcal{E}(x_0):[Ad(x_0)]_i=0\} J ( x 0 ) ∪ G ( x 0 ) = { i ∈ E ( x 0 ) : [ A d ( x 0 ) ] i = 0 } does not depend on the choice of p ( x 0 ) p(x_0) p ( x 0 ) . For an index set K ⊆ [ m ] K\subseteq[m] K ⊆ [ m ] , let P K P_K P K denote the orthogonal projection onto ker A K \ker A_K ker A K , given by (11) with K K K in place of J ( x 0 ) J(x_0) J ( x 0 ) when the rows of A K A_K A K are linearly independent. For a set K K K with J ( x 0 ) ⊆ K ⊆ J ( x 0 ) ∪ G ( x 0 ) J(x_0)\subseteq K\subseteq J(x_0)\cup G(x_0) J ( x 0 ) ⊆ K ⊆ J ( x 0 ) ∪ G ( x 0 ) , we write M K M_K M K and τ f e a s K ( x 0 ) \tau_{\mathrm{feas}}^K(x_0) τ feas K ( x 0 ) for (22) and (25) with K K K in place of J ( x 0 ) J(x_0) J ( x 0 ) . Here, τ f e a s K ( x 0 ) \tau_{\mathrm{feas}}^K(x_0) τ feas K ( x 0 ) ignores every constraint whose value vanishes identically along the curve x 0 + M K ( τ ) d ( x 0 ) x_0+M_K(\tau)d(x_0) x 0 + M K ( τ ) d ( x 0 ) . Let
τ J K ( x 0 ) : = inf { τ > 0 : y ∈ C and − Q y − c − e − P K ⊥ Q τ d ( x 0 ) ∉ N C ( y ) , \tau_J^K(x_0):=\inf\left\{\tau>0:y\in C\ \text{and}\ -Qy-c-e^{-P_K^\perp Q\tau}d(x_0)\notin N_C(y),
\right. τ J K ( x 0 ) := inf { τ > 0 : y ∈ C and − Q y − c − e − P K ⊥ Q τ d ( x 0 ) ∈ / N C ( y ) ,
where y = x 0 + M K ( τ ) d ( x 0 ) } (37) \left.
\text{where }y=x_0+M_K(\tau)d(x_0)\right\} \tag*{(37)} where y = x 0 + M K ( τ ) d ( x 0 ) } (37)
be the first time at which the normal part along the curve leaves the normal cone, and let τ ∗ K ( x 0 ) : = min ( τ f e a s K ( x 0 ) , τ J K ( x 0 ) ) \tau_*^K(x_0):=\min(\tau_{\mathrm{feas}}^K(x_0),\tau_J^K(x_0)) τ ∗ K ( x 0 ) := min ( τ feas K ( x 0 ) , τ J K ( x 0 )) . Unlike (26) , definition (37) does not have multipliers because the rows of A K A_K A K can be linearly dependent.
Theorem 3.2. Let x 0 ∈ C x_0\in C x 0 ∈ C and choose p ( x 0 ) p(x_0) p ( x 0 ) so that (LI) holds. There exists a set K K K with J ( x 0 ) ⊆ K ⊆ J ( x 0 ) ∪ G ( x 0 ) J(x_0)\subseteq K\subseteq J(x_0)\cup G(x_0) J ( x 0 ) ⊆ K ⊆ J ( x 0 ) ∪ G ( x 0 ) such that the following hold.
(i) The breakpoint time τ ∗ K ( x 0 ) \tau_*^K(x_0) τ ∗ K ( x 0 ) is positive, and the slow solution satisfies
x ( τ ) = x 0 + M K ( τ ) d ( x 0 ) , x ˙ ( τ ) = e − P K ⊥ Q τ d ( x 0 ) on [ 0 , τ ∗ K ( x 0 ) ) . (38) x(\tau)=x_0+M_K(\tau)d(x_0),\qquad\dot{x}(\tau)=e^{-P_K^\perp Q\tau}d(x_0)\quad\text{on }[0,\tau_*^K(x_0)). \tag*{(38)} x ( τ ) = x 0 + M K ( τ ) d ( x 0 ) , x ˙ ( τ ) = e − P K ⊥ Q τ d ( x 0 ) on [ 0 , τ ∗ K ( x 0 )) . (38)
Moreover, for all sufficiently small τ > 0 \tau>0 τ > 0 , there exists w ∈ R ∣ K ∣ \mathbf{w}\in\mathbb{R}^{|K|} w ∈ R ∣ K ∣ with positive entries such that
− Q x ( τ ) − c = d ( x ( τ ) ) + A K T w . (39) -Qx(\tau)-c=d(x(\tau))+A_K^{\mathsf{T}}\mathbf{w}. \tag*{(39)} − Q x ( τ ) − c = d ( x ( τ )) + A K T w . (39)
(ii) Suppose τ ∗ K ( x 0 ) < + ∞ \tau_*^K(x_0)<+\infty τ ∗ K ( x 0 ) < + ∞ . If [ A e − P K ⊥ Q τ ∗ K ( x 0 ) d ( x 0 ) ] i > 0 [Ae^{-P_K^\perp Q\tau_*^K(x_0)}d(x_0)]_i>0 [ A e − P K ⊥ Q τ ∗ K ( x 0 ) d ( x 0 ) ] i > 0 for some i ∈ E ( x ( τ ∗ K ( x 0 ) ) ) i\in\mathcal{E}(x(\tau_*^K(x_0))) i ∈ E ( x ( τ ∗ K ( x 0 ))) , then the right derivative τ ↦ x ˙ ( τ ) \tau\mapsto\dot{x}(\tau) τ ↦ x ˙ ( τ ) is discontinuous at τ ∗ K ( x 0 ) \tau_*^K(x_0) τ ∗ K ( x 0 ) , and
∥ d ( x ( τ ∗ K ( x 0 ) ) ) ∥ 2 < ∥ e − P K ⊥ Q τ ∗ K ( x 0 ) d ( x 0 ) ∥ 2 , (40) \left\lVert d(x(\tau_*^K(x_0)))\right\rVert_2<\left\lVert e^{-P_K^\perp Q\tau_*^K(x_0)}d(x_0)\right\rVert_2, \tag*{(40)} d ( x ( τ ∗ K ( x 0 ))) 2 < e − P K ⊥ Q τ ∗ K ( x 0 ) d ( x 0 ) 2 , (40)
where the right-hand side is the norm of the left limit of x ˙ ( τ ) \dot{x}(\tau) x ˙ ( τ ) as τ → τ ∗ K ( x 0 ) − \tau\to\tau_*^K(x_0)^- τ → τ ∗ K ( x 0 ) − . Otherwise, the right derivative is continuous.
(iii) If the rows of A J ( x 0 ) ∪ G ( x 0 ) A_{J(x_0)\cup G(x_0)} A J ( x 0 ) ∪ G ( x 0 ) are linearly independent, then J ( x ( τ ) ) = K J(x(\tau))=K J ( x ( τ )) = K for all sufficiently small τ > 0 \tau>0 τ > 0 , and K K K is the only set with J ( x 0 ) ⊆ K ⊆ J ( x 0 ) ∪ G ( x 0 ) J(x_0)\subseteq K\subseteq J(x_0)\cup G(x_0) J ( x 0 ) ⊆ K ⊆ J ( x 0 ) ∪ G ( x 0 ) for which part (i) holds.
We denote by J + ( x 0 ) J^+(x_0) J + ( x 0 ) the set K K K constructed in Appendix B.1, and write M ∗ M_* M ∗ , τ f e a s + ( x 0 ) \tau_{\mathrm{feas}}^+(x_0) τ feas + ( x 0 ) , τ J + ( x 0 ) \tau_J^+(x_0) τ J + ( x 0 ) , and τ ∗ + ( x 0 ) \tau_*^+(x_0) τ ∗ + ( x 0 ) for the objects built from it. By part (iii), K K K is unique when the rows of A J ( x 0 ) ∪ G ( x 0 ) A_{J(x_0)\cup G(x_0)} A J ( x 0 ) ∪ G ( x 0 ) are linearly independent. When they are linearly dependent, K K K need not be unique, but the slow solution is. Under (SCC), G ( x 0 ) = ∅ G(x_0)=\varnothing G ( x 0 ) = ∅ , so J + ( x 0 ) = J ( x 0 ) J^+(x_0)=J(x_0) J + ( x 0 ) = J ( x 0 ) . For τ ∈ ( 0 , τ f e a s + ( x 0 ) ) \tau\in(0,\tau_{\mathrm{feas}}^+(x_0)) τ ∈ ( 0 , τ feas + ( x 0 )) , the arguments of Steps 2b and 2c of the proof of Theorem 3.1 give E ( y ) = J ( x 0 ) \mathcal{E}(y)=J(x_0) E ( y ) = J ( x 0 ) and the normal part A J ( x 0 ) T w ( τ ) A_{J(x_0)}^{\mathsf{T}}w(\tau) A J ( x 0 ) T w ( τ ) , so (32) reduces to the sign test in (22b). Hence τ ∗ + ( x 0 ) = τ ∗ ( x 0 ) \tau_*^+(x_0)=\tau_*(x_0) τ ∗ + ( x 0 ) = τ ∗ ( x 0 ) , and Theorem 3.2 reduces to Theorem 3.1 and Proposition 3.3.
Global solution of the slow system
Global dynamics of the slow system
Section 3 solved the slow system (5) explicitly on an interval [ 0 , τ ∗ + ( x 0 ) ) [0,\tau_*^+(x_0)) [ 0 , τ ∗ + ( x 0 )) . When τ ∗ + ( x 0 ) \tau_*^+(x_0) τ ∗ + ( x 0 ) is finite, continuity extends the solution to τ = τ ∗ + ( x 0 ) \tau=\tau_*^+(x_0) τ = τ ∗ + ( x 0 ) . In this case, since x ( τ ∗ + ( x 0 ) ) ∈ C x(\tau_*^+(x_0))\in C x ( τ ∗ + ( x 0 )) ∈ C and the slow system is autonomous, Theorem 3.2 applies again at the point x ( τ ∗ + ( x 0 ) ) x(\tau_*^+(x_0)) x ( τ ∗ + ( x 0 )) and adds a second piece.
We now repeat these arguments to construct the global solution. Letting x ( 0 ) = x 0 ∈ C x^{(0)}=x_0\in C x ( 0 ) = x 0 ∈ C , τ ( 0 ) = 0 \tau^{(0)}=0 τ ( 0 ) = 0 , and k = 0 k=0 k = 0 , we iterate, with p ( k ) p^{(k)} p ( k ) chosen so that (LI) holds at x ( k ) x^{(k)} x ( k ) ,
p ( k ) ∈ arg min p ∈ E ( x ( k ) ) p E ( x ( k ) ) c = 0 1 2 ∥ A T p + Q x ( k ) + c ∥ 2 2 , d ( k ) = − A T p ( k ) − Q x ( k ) − c , J ∗ ( k ) = J + ( x ( k ) ) , τ ∗ ( k ) = τ ∗ + ( x ( k ) ) , x ( k + 1 ) = x ( k ) + M J ∗ ( k ) ( τ ∗ ( k ) ) d ( k ) , τ ( k + 1 ) = τ ( k ) + τ ∗ ( k ) . (41) \begin{aligned}
p^{(k)}&\in\underset{\substack{p\in\mathcal{E}(x^{(k)})\\p_{\mathcal{E}(x^{(k)})^c}=0}}{\arg\min}\ \frac{1}{2}\left\lVert A^{\mathsf{T}}p+Qx^{(k)}+c\right\rVert_2^2,\qquad d^{(k)}=-A^{\mathsf{T}}p^{(k)}-Qx^{(k)}-c,\\
J_*^{(k)}&=J^+(x^{(k)}),\qquad\tau_*^{(k)}=\tau_*^+(x^{(k)}),\\
x^{(k+1)}&=x^{(k)}+M_{J_*^{(k)}}(\tau_*^{(k)})d^{(k)},\qquad\tau^{(k+1)}=\tau^{(k)}+\tau_*^{(k)}. \tag*{(41)}
\end{aligned} p ( k ) J ∗ ( k ) x ( k + 1 ) ∈ p ∈ E ( x ( k ) ) p E ( x ( k ) ) c = 0 arg min 2 1 A T p + Q x ( k ) + c 2 2 , d ( k ) = − A T p ( k ) − Q x ( k ) − c , = J + ( x ( k ) ) , τ ∗ ( k ) = τ ∗ + ( x ( k ) ) , = x ( k ) + M J ∗ ( k ) ( τ ∗ ( k ) ) d ( k ) , τ ( k + 1 ) = τ ( k ) + τ ∗ ( k ) . (41)
The iteration stops at the first k k k with τ ∗ ( k ) = + ∞ \tau_*^{(k)}=+\infty τ ∗ ( k ) = + ∞ . The slow solution is then
x ( τ ) = x ( k ) + M J ∗ ( k ) ( τ − τ ( k ) ) d ( k ) over τ ∈ [ τ ( k ) , τ ( k + 1 ) ) . (42) x(\tau)=x^{(k)}+M_{J_*^{(k)}}(\tau-\tau^{(k)})d^{(k)}\quad\text{over }\tau\in[\tau^{(k)},\tau^{(k+1)}). \tag*{(42)} x ( τ ) = x ( k ) + M J ∗ ( k ) ( τ − τ ( k ) ) d ( k ) over τ ∈ [ τ ( k ) , τ ( k + 1 ) ) . (42)
We call the times τ ( k ) \tau^{(k)} τ ( k ) with k ≥ 1 k\ge1 k ≥ 1 the breakpoints of the slow solution, the interval [ τ ( k ) , τ ( k + 1 ) ) [\tau^{(k)},\tau^{(k+1)}) [ τ ( k ) , τ ( k + 1 ) ) its k k k -th piece , and J ∗ ( k ) J_*^{(k)} J ∗ ( k ) the active set of that piece. Formula (37) extends the slow solution to the interval [ 0 , τ ( k ) ) [0,\tau^{(k)}) [ 0 , τ ( k ) ) for every k ∈ N k\in\mathbb{N} k ∈ N with τ ( k ) < + ∞ \tau^{(k)}<+\infty τ ( k ) < + ∞ .
Finite termination of the slow system
What happens to the global dynamics as k → + ∞ k \to+\infty k → + ∞ ? We show below that the iteration (41) terminates after finitely many steps, i.e., there exists K ∈ N K \in\mathbb{N} K ∈ N such that τ ∗ ( K ) = + ∞ \tau_*^{(K)}=+\infty τ ∗ ( K ) = + ∞ . Moreover, the final piece either yields an optimal solution of problem (1) or certifies that the problem is unbounded.
Theorem 4.1. Let x 0 ∈ C \boldsymbol{x}_0 \in C x 0 ∈ C and set x ( 0 ) = x 0 \boldsymbol{x}^{(0)}=\boldsymbol{x}_0 x ( 0 ) = x 0 . Then there exists K ∈ N K \in\mathbb{N} K ∈ N such that τ ∗ ( K ) = + ∞ \tau_*^{(K)}=+\infty τ ∗ ( K ) = + ∞ and
d s : = lim τ → + ∞ d ( x ( τ ) ) = proj ker Q ∩ ker A J ∗ ( K ) ( d ( K ) ) . \boldsymbol{d}^{s} := \lim_{\tau\to+\infty} \boldsymbol{d}(\boldsymbol{x}(\tau)) = \operatorname{proj}_{\ker Q \cap\ker A_{J_*^{(K)}}}\left(\boldsymbol{d}^{(K)}\right). d s := τ → + ∞ lim d ( x ( τ )) = proj k e r Q ∩ k e r A J ∗ ( K ) ( d ( K ) ) . Moreover:
(i) If d s = 0 \boldsymbol{d}^{s}=0 d s = 0 , then lim τ → + ∞ x ( τ ) = x ( K ) + ( P J ∗ ( K ) ⊥ Q P J ∗ ( K ) ⊥ ) † d ( K ) \lim_{\tau\to+\infty}\boldsymbol{x}(\tau)=\boldsymbol{x}^{(K)}+\left(P_{J_*^{(K)}}^\perp QP_{J_*^{(K)}}^\perp\right)^\dagger\boldsymbol{d}^{(K)} lim τ → + ∞ x ( τ ) = x ( K ) + ( P J ∗ ( K ) ⊥ Q P J ∗ ( K ) ⊥ ) † d ( K ) is an optimal solution of problem (1), where the dagger denotes the pseudo-inverse.
(ii) If d s ≠ 0 \boldsymbol{d}^{s}\ne0 d s = 0 , then A d s ≤ 0 A\boldsymbol{d}^{s}\le0 A d s ≤ 0 , Q d s = 0 Q\boldsymbol{d}^{s}=0 Q d s = 0 , and ⟨ c , d s ⟩ = − ∥ d s ∥ 2 2 < 0 \langle\boldsymbol{c},\boldsymbol{d}^{s}\rangle=-\|\boldsymbol{d}^{s}\|_2^2<0 ⟨ c , d s ⟩ = − ∥ d s ∥ 2 2 < 0 . In particular, problem (1) is unbounded below along d s \boldsymbol{d}^{s} d s .
Proof. The idea of the proof is that the constraint values [ A x ( τ ) ] i − b i [A\boldsymbol{x}(\tau)]_i-b_i [ A x ( τ ) ] i − b i and the multipliers that decide when a constraint leaves the active set eventually keep a fixed sign. If there were infinitely many breakpoints, the trajectory would approach a motion with constant velocity, and its deviation from this motion would be a sum of fixed vectors with coefficients of decreasing size. The constraint values and the multipliers are linear in this deviation, so the first term that contributes to one of them determines its sign unless it vanishes identically. No constraint would then enter or leave the active set after some breakpoint, leading to a contradiction. The formulas for d s \boldsymbol{d}^{s} d s and the limit in (i) follow from the final piece (42) . The proof is highly technical and is relegated to Appendix § 11 .§ 2 . □
Remark 4.1 (The symmetric positive definite case). If Q Q Q is S P D SPD S P D , then Q d s = 0 Q\boldsymbol{d}^{s}=0 Q d s = 0 forces d s = 0 \boldsymbol{d}^{s}=0 d s = 0 , so x ( τ ) \boldsymbol{x}(\tau) x ( τ ) converges to the optimal solution x s \boldsymbol{x}^{s} x s , unique by strict convexity.
An exact algorithm for linear programs and projections onto polyhedra
We consider here the special case where Q = t I n × n Q=tI_{n\times n} Q = t I n × n , corresponding to linear programs (t = 0 t=0 t = 0 ) and projections onto a polyhedron (t > 0 t>0 t > 0 ). In this case, the slow solution can be computed exactly and no active set is visited more than once. Each piece of the slow solution is then a ray, as in § 3.4 , and the following iteration computes these rays. Let x ( 0 ) = x 0 ∈ C \boldsymbol{x}^{(0)}=\boldsymbol{x}_0\in C x ( 0 ) = x 0 ∈ C . For k = 0 , 1 , 2 , … k=0,1,2,\ldots k = 0 , 1 , 2 , … , choose the solution p ( k ) = p ( x ( k ) ) \boldsymbol{p}^{(k)}=\boldsymbol{p}(\boldsymbol{x}^{(k)}) p ( k ) = p ( x ( k ) ) of the NNLS problem (7) so that (LI) holds at x ( k ) \boldsymbol{x}^{(k)} x ( k ) . Let d ( k ) = d ( x ( k ) ) \boldsymbol{d}^{(k)}=\boldsymbol{d}(\boldsymbol{x}^{(k)}) d ( k ) = d ( x ( k ) ) and let Δ ( k ) = Δ f e a s ( x ( k ) ) \Delta^{(k)}=\Delta_{\mathrm{feas}}(\boldsymbol{x}^{(k)}) Δ ( k ) = Δ feas ( x ( k ) ) be given by (14), and set
x ( k + 1 ) = x ( k ) + Δ ( k ) d ( k ) \boldsymbol{x}^{(k+1)}=\boldsymbol{x}^{(k)}+\Delta^{(k)}\boldsymbol{d}^{(k)} x ( k + 1 ) = x ( k ) + Δ ( k ) d ( k )
as long as Δ ( k ) < 1 / t \Delta^{(k)}<1/t Δ ( k ) < 1/ t . {#eq:38}
We write K K K for the first index with Δ ( K ) ≥ 1 / t \Delta^{(K)}\ge1/t Δ ( K ) ≥ 1/ t , which we prove exists.
Theorem 4.2. Let Q = t I n × n Q=tI_{n\times n} Q = t I n × n with t ≥ 0 t\ge0 t ≥ 0 , τ ( 0 ) = 0 \tau^{(0)}=0 τ ( 0 ) = 0 , and let x ( 0 ) , x ( 1 ) , … \boldsymbol{x}^{(0)},\boldsymbol{x}^{(1)},\ldots x ( 0 ) , x ( 1 ) , … be the iterates of (38).
(i) The slow solution is
x ( τ ) = x ( k ) + { ( τ − τ ( k ) ) d ( k ) if t = 0 , ( 1 − e − t ( τ − τ ( k ) ) ) d ( k ) / t if t > 0 , for τ ∈ [ τ ( k ) , τ ( k + 1 ) ) and k = 0 , … , K . \boldsymbol{x}(\tau)=\boldsymbol{x}^{(k)}+
\begin{cases}
(\tau-\tau^{(k)})\boldsymbol{d}^{(k)} & \text{if }t=0,\\
(1-e^{-t(\tau-\tau^{(k)})})\boldsymbol{d}^{(k)}/t & \text{if }t>0,
\end{cases}
\qquad\text{for }\tau\in[\tau^{(k)},\tau^{(k+1)})\text{ and }k=0,\ldots,K. x ( τ ) = x ( k ) + { ( τ − τ ( k ) ) d ( k ) ( 1 − e − t ( τ − τ ( k ) ) ) d ( k ) / t if t = 0 , if t > 0 , for τ ∈ [ τ ( k ) , τ ( k + 1 ) ) and k = 0 , … , K . For k < K k<K k < K , the breakpoint is τ ( k + 1 ) = τ ( k ) + Δ ( k ) \tau^{(k+1)}=\tau^{(k)}+\Delta^{(k)} τ ( k + 1 ) = τ ( k ) + Δ ( k ) if t = 0 t=0 t = 0 and τ ( k + 1 ) = τ ( k ) − log ( 1 − t Δ ( k ) ) / t \tau^{(k+1)}=\tau^{(k)}-\log(1-t\Delta^{(k)})/t τ ( k + 1 ) = τ ( k ) − log ( 1 − t Δ ( k ) ) / t if t > 0 t>0 t > 0 . The final piece is [ τ ( K ) , + ∞ ) [\tau^{(K)},+\infty) [ τ ( K ) , + ∞ ) .
(ii) The norm ∥ A T p ( k ) ∥ 2 \lVert A^{\mathsf T}p^{(k)}\rVert_2 ∥ A T p ( k ) ∥ 2 depends on J ( x ( k ) ) J(x^{(k)}) J ( x ( k ) ) alone and increases strictly with k k k . Hence no active set occurs twice, the iteration (38) stops, and K ≤ 2 m − 1 K \le2^m-1 K ≤ 2 m − 1 .
(iii) If t > 0 t>0 t > 0 , then x s : = x ( K ) + 1 t d ( K ) x^s := x^{(K)}+\frac{1}{t}d^{(K)} x s := x ( K ) + t 1 d ( K ) is the unique optimal solution of problem (1) . If t = 0 t=0 t = 0 and d ( K ) = 0 d^{(K)}=0 d ( K ) = 0 , then x s : = x ( K ) x^s:=x^{(K)} x s := x ( K ) is an optimal solution. In both cases, letting p s : = p ( K ) p^s:=p^{(K)} p s := p ( K ) , the pair ( x s , p s ) (x^s,p^s) ( x s , p s ) satisfies the KKT conditions. If t = 0 t=0 t = 0 and d ( K ) ≠ 0 d^{(K)}\ne0 d ( K ) = 0 , then the ray x ( K ) + Δ d ( K ) x^{(K)}+\Delta d^{(K)} x ( K ) + Δ d ( K ) , Δ ≥ 0 \Delta\ge0 Δ ≥ 0 , is feasible and V ( x ( K ) + Δ d ( K ) ) → − ∞ V(x^{(K)}+\Delta d^{(K)})\to-\infty V ( x ( K ) + Δ d ( K ) ) → − ∞ as Δ → + ∞ \Delta\to+\infty Δ → + ∞ .
Proof. Throughout, we write J ( k ) : = J ( x ( k ) ) J^{(k)}:=J(x^{(k)}) J ( k ) := J ( x ( k ) ) and n ( k ) n^{(k)} n ( k ) for the normal component of the antigradient, so that − ∇ x f ( x ( k ) ) = d ( k ) + n ( k ) -\nabla_x f(x^{(k)})=d^{(k)}+n^{(k)} − ∇ x f ( x ( k ) ) = d ( k ) + n ( k ) with ⟨ d ( k ) , n ( k ) ⟩ = 0 \langle d^{(k)},n^{(k)}\rangle=0 ⟨ d ( k ) , n ( k ) ⟩ = 0 by (6) and (14) .
Step 1. Fix k ≥ 0 k\ge0 k ≥ 0 , suppose x ( τ ( k ) ) = x ( k ) x(\tau^{(k)})=x^{(k)} x ( τ ( k ) ) = x ( k ) . For 0 < Δ < min ( Δ ( k ) , 1 / t ) 0<\Delta<\min(\Delta^{(k)},1/t) 0 < Δ < min ( Δ ( k ) , 1/ t ) , where 1 / t 1/t 1/ t reads as + ∞ +\infty + ∞ when t = 0 t=0 t = 0 , and put x Δ : = x ( k ) + Δ d ( k ) x_\Delta:=x^{(k)}+\Delta d^{(k)} x Δ := x ( k ) + Δ d ( k ) . Every constraint i i i with [ p ( k ) ] i > 0 [p^{(k)}]_i>0 [ p ( k ) ] i > 0 lies in E ( x ( k ) ) \mathcal{E}(x^{(k)}) E ( x ( k ) ) and satisfies [ A d ( k ) ] i = 0 [Ad^{(k)}]_i=0 [ A d ( k ) ] i = 0 by (14) , so it is active at x Δ x_\Delta x Δ for every Δ ≥ 0 \Delta\ge0 Δ ≥ 0 and n ( k ) ∈ N C ( x Δ ) n^{(k)}\in N_C(x_\Delta) n ( k ) ∈ N C ( x Δ ) . A constraint i ∈ E ( x ( k ) ) i\in\mathcal{E}(x^{(k)}) i ∈ E ( x ( k ) ) with [ A d ( k ) ] i < 0 [Ad^{(k)}]_i<0 [ A d ( k ) ] i < 0 is inactive at x Δ x_\Delta x Δ , and so is every i ∉ E ( x ( k ) ) i\notin\mathcal{E}(x^{(k)}) i ∈ / E ( x ( k ) ) by the definition (17) of Δ ( k ) \Delta^{(k)} Δ ( k ) . Every constraint active at x Δ x_\Delta x Δ therefore satisfies [ A d ( k ) ] i = 0 [Ad^{(k)}]_i=0 [ A d ( k ) ] i = 0 , and ( 1 − t Δ ) d ( k ) ∈ T C ( x Δ ) (1-t\Delta)d^{(k)}\in T_C(x_\Delta) ( 1 − t Δ ) d ( k ) ∈ T C ( x Δ ) . Since ∇ x f \nabla_x f ∇ x f is affine with slope t I n × n tI_{n\times n} t I n × n , we have − ∇ x f ( x Δ ) = ( 1 − t Δ ) d ( k ) + n ( k ) -\nabla_x f(x_\Delta)=(1-t\Delta)d^{(k)}+n^{(k)} − ∇ x f ( x Δ ) = ( 1 − t Δ ) d ( k ) + n ( k ) , and the two terms are orthogonal. Moreau’s decomposition theorem (6) gives d ( x Δ ) = ( 1 − t Δ ) d ( k ) d(x_\Delta)=(1-t\Delta)d^{(k)} d ( x Δ ) = ( 1 − t Δ ) d ( k ) .
The function τ ↦ Δ ( τ ) \tau\mapsto\Delta(\tau) τ ↦ Δ ( τ ) of (34) satisfies Δ ˙ ( τ ) = e − t τ = 1 − t Δ ( τ ) \dot{\Delta}(\tau)=e^{-t\tau}=1-t\Delta(\tau) Δ ˙ ( τ ) = e − t τ = 1 − t Δ ( τ ) and increases strictly and continuously from 0 0 0 to 1 / t 1/t 1/ t , with 1 / t = + ∞ 1/t=+\infty 1/ t = + ∞ when t = 0 t=0 t = 0 . The curve x ~ ( τ ) : = x ( k ) + Δ ( τ − τ ( k ) ) d ( k ) \widetilde{x}(\tau):=x^{(k)}+\Delta(\tau-\tau^{(k)})d^{(k)} x ( τ ) := x ( k ) + Δ ( τ − τ ( k ) ) d ( k ) therefore solves the slow system (5) as long as Δ ( τ − τ ( k ) ) < Δ ( k ) \Delta(\tau-\tau^{(k)})<\Delta^{(k)} Δ ( τ − τ ( k ) ) < Δ ( k ) . Since the slow system is autonomous, the monotonicity argument at the end of the proof of Theorem 3.1 shows that x ~ \widetilde{x} x is the slow solution there. If Δ ( k ) ≥ 1 / t \Delta^{(k)}\ge1/t Δ ( k ) ≥ 1/ t , then k = K k=K k = K and x = x ~ x=\widetilde{x} x = x on [ τ ( K ) , + ∞ ) [\tau^{(K)},+\infty) [ τ ( K ) , + ∞ ) . If Δ ( k ) < 1 / t \Delta^{(k)}<1/t Δ ( k ) < 1/ t , then Δ ( τ − τ ( k ) ) \Delta(\tau-\tau^{(k)}) Δ ( τ − τ ( k ) ) reaches Δ ( k ) \Delta^{(k)} Δ ( k ) at τ = τ ( k + 1 ) \tau=\tau^{(k+1)} τ = τ ( k + 1 ) by (35) , and the continuity of x x x gives x ( τ ( k + 1 ) ) = x ( k + 1 ) x(\tau^{(k+1)})=x^{(k+1)} x ( τ ( k + 1 ) ) = x ( k + 1 ) . In this case Δ ( k ) \Delta^{(k)} Δ ( k ) is finite, so the least quotient in (17) is attained at a constraint i i i with [ A d ( k ) ] i > 0 [Ad^{(k)}]_i>0 [ A d ( k ) ] i > 0 , which is active at x ( k + 1 ) x^{(k+1)} x ( k + 1 ) . Induction on k k k proves (i).
Step 2. Let k k k satisfy Δ ( k ) < 1 / t \Delta^{(k)}<1/t Δ ( k ) < 1/ t . We first show that ∥ n ( k + 1 ) ∥ 2 > ∥ n ( k ) ∥ 2 \lVert n^{(k+1)}\rVert_2>\lVert n^{(k)}\rVert_2 ∥ n ( k + 1 ) ∥ 2 > ∥ n ( k ) ∥ 2 . Since ∇ x f \nabla_x f ∇ x f is affine with slope t I n × n tI_{n\times n} t I n × n ,
− ∇ x f ( x ( k + 1 ) ) = − ∇ x f ( x ( k ) ) − t Δ ( k ) d ( k ) = ( 1 − t Δ ( k ) ) d ( k ) + n ( k ) . -\nabla_x f(x^{(k+1)})=-\nabla_x f(x^{(k)})-t\Delta^{(k)}d^{(k)}=(1-t\Delta^{(k)})d^{(k)}+n^{(k)}. − ∇ x f ( x ( k + 1 ) ) = − ∇ x f ( x ( k ) ) − t Δ ( k ) d ( k ) = ( 1 − t Δ ( k ) ) d ( k ) + n ( k ) .
Put v : = ( 1 − t Δ ( k ) ) d ( k ) v:=(1-t\Delta^{(k)})d^{(k)} v := ( 1 − t Δ ( k ) ) d ( k ) and K : = { s ∈ R n : [ A s ] i ≤ 0 for every i with [ p ( k ) ] i > 0 } \mathcal{K}:=\{s\in\mathbb{R}^n:[As]_i\le0\text{ for every }i\text{ with }[p^{(k)}]_i>0\} K := { s ∈ R n : [ A s ] i ≤ 0 for every i with [ p ( k ) ] i > 0 } . The vector v v v lies in K \mathcal{K} K by (14) , the vector n ( k ) n^{(k)} n ( k ) is a nonnegative combination of the rows that define K \mathcal{K} K and so satisfies ⟨ n ( k ) , s ⟩ ≤ 0 \langle n^{(k)},s\rangle\le0 ⟨ n ( k ) , s ⟩ ≤ 0 for every s ∈ K s\in\mathcal{K} s ∈ K , and ⟨ v , n ( k ) ⟩ = 0 \langle v,n^{(k)}\rangle=0 ⟨ v , n ( k ) ⟩ = 0 . Since − ∇ x f ( x ( k + 1 ) ) − v = n ( k ) -\nabla_x f(x^{(k+1)})-v=n^{(k)} − ∇ x f ( x ( k + 1 ) ) − v = n ( k ) , these three properties and the characterization (5) give v = proj K ( − ∇ x f ( x ( k + 1 ) ) ) v=\operatorname{proj}_{\mathcal{K}}(-\nabla_x f(x^{(k+1)})) v = proj K ( − ∇ x f ( x ( k + 1 ) )) . The constraints with [ p ( k ) ] i > 0 [p^{(k)}]_i>0 [ p ( k ) ] i > 0 are active at x ( k + 1 ) x^{(k+1)} x ( k + 1 ) by Step 1, so T C ( x ( k + 1 ) ) ⊆ K T_C(x^{(k+1)})\subseteq\mathcal{K} T C ( x ( k + 1 ) ) ⊆ K . Step 1 also gives a constraint i i i active at x ( k + 1 ) x^{(k+1)} x ( k + 1 ) with [ A d ( k ) ] i > 0 [Ad^{(k)}]_i>0 [ A d ( k ) ] i > 0 . Since 1 − t Δ ( k ) > 0 1-t\Delta^{(k)}>0 1 − t Δ ( k ) > 0 , this constraint satisfies [ A v ] i > 0 [Av]_i>0 [ A v ] i > 0 , so v ∉ T C ( x ( k + 1 ) ) v\notin T_C(x^{(k+1)}) v ∈ / T C ( x ( k + 1 ) ) and v ≠ d ( k + 1 ) v\ne d^{(k+1)} v = d ( k + 1 ) . Since d ( k + 1 ) ∈ T C ( x ( k + 1 ) ) ⊆ K d^{(k+1)}\in T_C(x^{(k+1)})\subseteq\mathcal{K} d ( k + 1 ) ∈ T C ( x ( k + 1 ) ) ⊆ K and v v v is the unique point of K \mathcal{K} K nearest to − ∇ x f ( x ( k + 1 ) ) -\nabla_x f(x^{(k+1)}) − ∇ x f ( x ( k + 1 ) ) , we get ∥ n ( k + 1 ) ∥ 2 = ∥ − ∇ x f ( x ( k + 1 ) ) − d ( k + 1 ) ∥ 2 > ∥ − ∇ x f ( x ( k + 1 ) ) − v ∥ 2 = ∥ n ( k ) ∥ 2 \lVert n^{(k+1)}\rVert_2=\lVert-\nabla_x f(x^{(k+1)})-d^{(k+1)}\rVert_2>\lVert-\nabla_x f(x^{(k+1)})-v\rVert_2=\lVert n^{(k)}\rVert_2 ∥ n ( k + 1 ) ∥ 2 = ∥ − ∇ x f ( x ( k + 1 ) ) − d ( k + 1 ) ∥ 2 > ∥ − ∇ x f ( x ( k + 1 ) ) − v ∥ 2 = ∥ n ( k ) ∥ 2 .
Since A J ( k ) x ( k ) = b J ( k ) A_{J^{(k)}}x^{(k)}=b_{J^{(k)}} A J ( k ) x ( k ) = b J ( k ) , formula (10) with ∇ x f ( x ( k ) ) = t x ( k ) + c \nabla_x f(x^{(k)})=tx^{(k)}+c ∇ x f ( x ( k ) ) = t x ( k ) + c gives
p J ( k ) ( k ) = − ( A J ( k ) A J ( k ) T ) − 1 ( t b J ( k ) + A J ( k ) c ) . p_{J^{(k)}}^{(k)}=-\left(A_{J^{(k)}}A_{J^{(k)}}^{\mathsf T}\right)^{-1}\left(tb_{J^{(k)}}+A_{J^{(k)}}c\right). p J ( k ) ( k ) = − ( A J ( k ) A J ( k ) T ) − 1 ( t b J ( k ) + A J ( k ) c ) .
Hence n ( k ) = − A J ( k ) T p J ( k ) ( k ) n^{(k)}=-A_{J^{(k)}}^{\mathsf T}p_{J^{(k)}}^{(k)} n ( k ) = − A J ( k ) T p J ( k ) ( k ) depends on J ( k ) J^{(k)} J ( k ) alone, with n ( k ) = 0 n^{(k)}=0 n ( k ) = 0 when J ( k ) = ∅ J^{(k)}=\varnothing J ( k ) = ∅ . Since ∥ n ( 0 ) ∥ 2 < ∥ n ( 1 ) ∥ 2 < ⋯ \lVert n^{(0)}\rVert_2<\lVert n^{(1)}\rVert_2<\cdots ∥ n ( 0 ) ∥ 2 < ∥ n ( 1 ) ∥ 2 < ⋯ , the active sets J ( 0 ) , J ( 1 ) , … J^{(0)},J^{(1)},\ldots J ( 0 ) , J ( 1 ) , … are distinct subsets of [ m ] [m] [ m ] . There are 2 m 2^m 2 m such subsets, so the iteration stops and K + 1 ≤ 2 m K+1\le2^m K + 1 ≤ 2 m .
Step 3. First, let t > 0 t > 0 t > 0 . Since 1 / t ≤ Δ ( K ) 1/t \le\Delta^{(K)} 1/ t ≤ Δ ( K ) , the point x s = x ( K ) + 1 t d ( K ) x^s = x^{(K)} + \frac{1}{t}d^{(K)} x s = x ( K ) + t 1 d ( K ) lies in C C C by the definition (17) of Δ ( K ) \Delta^{(K)} Δ ( K ) . Since d ( K ) = − A T p ( K ) − t x ( K ) − c d^{(K)} = -A^\mathsf{T}p^{(K)} - tx^{(K)} - c d ( K ) = − A T p ( K ) − t x ( K ) − c , we have
∇ x f ( x s ) = t x s + c = t x ( K ) + d ( K ) + c = − A T p ( K ) . \nabla_x f(x^s) = tx^s + c = tx^{(K)} + d^{(K)} + c = -A^\mathsf{T}p^{(K)}. ∇ x f ( x s ) = t x s + c = t x ( K ) + d ( K ) + c = − A T p ( K ) .
where p ( K ) ≥ 0 p^{(K)} \ge0 p ( K ) ≥ 0 is supported on J ( K ) J^{(K)} J ( K ) , whose constraints are active at x s x^s x s because [ A x ( K ) ] J ( K ) = b J ( K ) [Ax^{(K)}]_{J^{(K)}} = b_{J^{(K)}} [ A x ( K ) ] J ( K ) = b J ( K ) and [ A d ( K ) ] J ( K ) = 0 [Ad^{(K)}]_{J^{(K)}} = 0 [ A d ( K ) ] J ( K ) = 0 by (13c). The pair ( x s , p ( K ) ) (x^s,p^{(K)}) ( x s , p ( K ) ) therefore satisfies the KKT conditions, so x s x^s x s is an optimal solution, unique by strict convexity, since t > 0 t > 0 t > 0 . Next, let t = 0 t = 0 t = 0 . If d ( K ) = 0 d^{(K)} = 0 d ( K ) = 0 , then ∇ x f ( x ( K ) ) = c = − A T p ( K ) \nabla_x f(x^{(K)}) = c = -A^\mathsf{T}p^{(K)} ∇ x f ( x ( K ) ) = c = − A T p ( K ) , where p ( K ) ≥ 0 p^{(K)} \ge0 p ( K ) ≥ 0 is supported on J ( K ) ⊆ E ( x ( K ) ) J^{(K)} \subseteq\mathcal{E}(x^{(K)}) J ( K ) ⊆ E ( x ( K ) ) , so the pair ( x ( K ) , p ( K ) ) (x^{(K)},p^{(K)}) ( x ( K ) , p ( K ) ) satisfies the KKT conditions and x ( K ) x^{(K)} x ( K ) is an optimal solution. If d ( K ) ≠ 0 d^{(K)} \ne0 d ( K ) = 0 , then Δ ( K ) = + ∞ \Delta^{(K)} = +\infty Δ ( K ) = + ∞ , so [ A d ( K ) ] i ≤ 0 [Ad^{(K)}]_i \le0 [ A d ( K ) ] i ≤ 0 for every i ∉ E ( x ( K ) ) i \notin\mathcal{E}(x^{(K)}) i ∈ / E ( x ( K ) ) by (17) , and (13a) gives the same over E ( x ( K ) ) \mathcal{E}(x^{(K)}) E ( x ( K ) ) . Hence A d ( K ) ≤ 0 Ad^{(K)} \le0 A d ( K ) ≤ 0 , and the ray x ( K ) + Δ d ( K ) x^{(K)} + \Delta d^{(K)} x ( K ) + Δ d ( K ) , Δ ≥ 0 \Delta\ge0 Δ ≥ 0 , is feasible. Formula (18) with Q = 0 Q = 0 Q = 0 then gives V ( x ( K ) + Δ d ( K ) ) = V ( x ( K ) ) − Δ ∥ d ( K ) ∥ 2 2 → − ∞ V(x^{(K)} + \Delta d^{(K)}) = V(x^{(K)}) - \Delta\lVert d^{(K)} \rVert_2^2 \to-\infty V ( x ( K ) + Δ d ( K ) ) = V ( x ( K ) ) − Δ ∥ d ( K ) ∥ 2 2 → − ∞ as Δ → + ∞ \Delta\to+\infty Δ → + ∞ .
Theorem 4.2 yields Algorithm 1. The algorithm computes the iterates x ( 0 ) , … , x ( K ) x^{(0)},\ldots,x^{(K)} x ( 0 ) , … , x ( K ) of (38) in at most 2 m 2^m 2 m iterations. At termination, the algorithm returns an optimal solution to problem (1) along with its corresponding dual solution or certifies that problem (1) is unbounded.
Minimal selection descent via the explicit Euler method
For SPSD Q Q Q , each piece of the slow solution requires the matrix exponential e − P J ⊥ Q τ e^{-P^\perp_JQ\tau} e − P J ⊥ Q τ , its integral M ( τ ) M(\tau) M ( τ ) , and the first zero of the constraint values and multipliers along it (see § 3.4 ). These computations are generally expensive unless Q = t I n × n Q = tI_{n \times n} Q = t I n × n , in which case Algorithm 1 can be used. To overcome this issue, we propose to perform gradient descent on the slow system (5) with a stepsize that keeps the iterates feasible and maximizes the descent. This leads to minimal selection descent (MSD), Algorithm 2. Unlike projected gradient descent (PGD), which steps along − ( Q x ( k ) + c ) -(Qx^{(k)} + c) − ( Q x ( k ) + c ) and then projects onto C C C , MSD projects − ( Q x ( k ) + c ) -(Qx^{(k)} + c) − ( Q x ( k ) + c ) onto the tangent cone T C ( x ( k ) ) T_C(x^{(k)}) T C ( x ( k ) ) and then steps along the direction d ( k ) d^{(k)} d ( k ) . This order is cheaper for a general polyhedron C C C . Indeed, the projection onto C C C is problem (1) with Q = I n × n Q = I_{n \times n} Q = I n × n , so PGD solves a problem of the same class at every step. In contrast, the projection onto T C ( x ( k ) ) T_C(x^{(k)}) T C ( x ( k ) ) is the NNLS problem (7) on the constraints active at x ( k ) x^{(k)} x ( k ) , for which efficient algorithms are available (see Remark 2.1).
Description of the MSD algorithm
Let Q ≠ 0 Q \ne0 Q = 0 be SPSD.1 At every iteration, the MSD algorithm seeks to maintain feasibility while minimizing the convex parabola
V ( x ( k ) + Δ d ( k ) ) = V ( x ( k ) ) − Δ ∥ d ( k ) ∥ 2 2 + Δ 2 2 ⟨ d ( k ) , Q d ( k ) ⟩ . V(x^{(k)} + \Delta d^{(k)}) = V(x^{(k)}) - \Delta\lVert d^{(k)} \rVert_2^2 + \frac{\Delta^2}{2}\langle d^{(k)}, Qd^{(k)}\rangle. V ( x ( k ) + Δ d ( k ) ) = V ( x ( k ) ) − Δ ∥ d ( k ) ∥ 2 2 + 2 Δ 2 ⟨ d ( k ) , Q d ( k ) ⟩ .
The parabola is minimized at Δ ∗ ( k ) : = ∥ d ( k ) ∥ 2 2 / ⟨ d ( k ) , Q d ( k ) ⟩ \Delta_*^{(k)} := \lVert d^{(k)} \rVert_2^2 / \langle d^{(k)}, Qd^{(k)}\rangle Δ ∗ ( k ) := ∥ d ( k ) ∥ 2 2 / ⟨ d ( k ) , Q d ( k ) ⟩ (equal to + ∞ +\infty + ∞ when d ( k ) ∈ ker Q d^{(k)} \in\operatorname{ker} Q d ( k ) ∈ ker Q ), so we set Δ ( k ) = min ( Δ f e a s ( x ( k ) ) , Δ ∗ ( k ) ) \Delta^{(k)} = \min(\Delta_{\mathrm{feas}}(x^{(k)}), \Delta_*^{(k)}) Δ ( k ) = min ( Δ feas ( x ( k ) ) , Δ ∗ ( k ) ) to ensure A x ( k ) ≤ b Ax^{(k)} \le b A x ( k ) ≤ b for every k ∈ N k \in\mathbb{N} k ∈ N and maximize the descent step. The size of the decrease depends on the largest eigenvalue of Q Q Q .
Lemma 5.1. Let λ max \lambda_{\max} λ m a x denote the largest eigenvalue of Q Q Q . If d ( k ) ≠ 0 d^{(k)} \ne0 d ( k ) = 0 and Δ ( k ) < + ∞ \Delta^{(k)} < +\infty Δ ( k ) < + ∞ , then
V ( x ( k + 1 ) ) ≤ V ( x ( k ) ) − min ( Δ f e a s ( x ( k ) ) , 1 / λ max ) ∥ d ( k ) ∥ 2 2 2 . (43) V(x^{(k+1)}) \le V(x^{(k)}) - \min\left(\Delta_{\mathrm{feas}}(x^{(k)}), 1/\lambda_{\max}\right)\frac{\lVert d^{(k)} \rVert_2^2}{2}. \tag*{(43)} V ( x ( k + 1 ) ) ≤ V ( x ( k ) ) − min ( Δ feas ( x ( k ) ) , 1/ λ m a x ) 2 ∥ d ( k ) ∥ 2 2 . (43)
Proof. If Δ ( k ) = Δ ∗ ( k ) \Delta^{(k)}=\Delta_*^{(k)} Δ ( k ) = Δ ∗ ( k ) , the parabola gives V ( x ( k + 1 ) ) = V ( x ( k ) ) − ∥ d ( k ) ∥ 2 4 2 ⟨ d ( k ) , Q d ( k ) ⟩ ≤ V ( x ( k ) ) − ∥ d ( k ) ∥ 2 2 2 λ max V(x^{(k+1)})=V(x^{(k)})-\frac{\lVert d^{(k)}\rVert_2^4}{2\langle d^{(k)},Qd^{(k)}\rangle}\le V(x^{(k)})-\frac{\lVert d^{(k)}\rVert_2^2}{2\lambda_{\max}} V ( x ( k + 1 ) ) = V ( x ( k ) ) − 2 ⟨ d ( k ) , Q d ( k ) ⟩ ∥ d ( k ) ∥ 2 4 ≤ V ( x ( k ) ) − 2 λ m a x ∥ d ( k ) ∥ 2 2 , since ⟨ d ( k ) , Q d ( k ) ⟩ ≤ λ max ∥ d ( k ) ∥ 2 2 \langle d^{(k)},Qd^{(k)}\rangle\le\lambda_{\max}\lVert d^{(k)}\rVert_2^2 ⟨ d ( k ) , Q d ( k ) ⟩ ≤ λ m a x ∥ d ( k ) ∥ 2 2 . If Δ ( k ) = Δ f e a s ( x ( k ) ) ≤ Δ ∗ ( k ) \Delta^{(k)}=\Delta_{\mathrm{feas}}(x^{(k)})\le\Delta_*^{(k)} Δ ( k ) = Δ feas ( x ( k ) ) ≤ Δ ∗ ( k ) , then Δ ( k ) ⟨ d ( k ) , Q d ( k ) ⟩ ≤ ∥ d ( k ) ∥ 2 2 \Delta^{(k)}\langle d^{(k)},Qd^{(k)}\rangle\le\lVert d^{(k)}\rVert_2^2 Δ ( k ) ⟨ d ( k ) , Q d ( k ) ⟩ ≤ ∥ d ( k ) ∥ 2 2 , and the parabola gives V ( x ( k + 1 ) ) ≤ V ( x ( k ) ) − Δ f e a s ( x ( k ) ) ∥ d ( k ) ∥ 2 2 / 2 V(x^{(k+1)})\le V(x^{(k)})-\Delta_{\mathrm{feas}}(x^{(k)})\lVert d^{(k)}\rVert_2^2/2 V ( x ( k + 1 ) ) ≤ V ( x ( k ) ) − Δ feas ( x ( k ) ) ∥ d ( k ) ∥ 2 2 /2 . Inequality (39) follows.
When Δ ( k ) = + ∞ \Delta^{(k)}=+\infty Δ ( k ) = + ∞ , the ray x ( k ) + Δ d ( k ) x^{(k)}+\Delta d^{(k)} x ( k ) + Δ d ( k ) , Δ ≥ 0 \Delta\ge0 Δ ≥ 0 , is feasible and V V V decreases without bound along it, so the algorithm stops and returns d ( k ) d^{(k)} d ( k ) . The next proposition confirms this and shows that the iteration detects every unbounded problem, either through that exit or through V ( x ( k ) ) → − ∞ V(x^{(k)})\to-\infty V ( x ( k ) ) → − ∞ . Theorem 4.1(ii) gives the same certificate for the slow solution.
Proposition 5.1. Let x ( 0 ) ∈ C x^{(0)}\in C x ( 0 ) ∈ C and let x ( k ) x^{(k)} x ( k ) denote the k k k -th iterate of Algorithm 2 with ε = 0 \varepsilon=0 ε = 0 and every p ( k ) p^{(k)} p ( k ) chosen so that (LI) holds at x ( k ) x^{(k)} x ( k ) . If problem (1) is unbounded below, then either V ( x ( k ) ) → − ∞ V(x^{(k)})\to-\infty V ( x ( k ) ) → − ∞ or some iterate has d ( K ) ≠ 0 d^{(K)}\ne0 d ( K ) = 0 , Δ ( K ) = + ∞ \Delta^{(K)}=+\infty Δ ( K ) = + ∞ , and
A d ( K ) ≤ 0 , Q d ( K ) = 0 , ⟨ c , d ( K ) ⟩ = − ∥ d ( K ) ∥ 2 2 < 0 , (44) Ad^{(K)}\le0,\qquad Qd^{(K)}=0,\qquad\langle c,d^{(K)}\rangle=-\lVert d^{(K)}\rVert_2^2<0, \tag*{(44)} A d ( K ) ≤ 0 , Q d ( K ) = 0 , ⟨ c , d ( K ) ⟩ = − ∥ d ( K ) ∥ 2 2 < 0 , (44) so V ( x ( K ) + Δ d ( K ) ) → − ∞ V(x^{(K)}+\Delta d^{(K)})\to-\infty V ( x ( K ) + Δ d ( K ) ) → − ∞ as Δ → + ∞ \Delta\to+\infty Δ → + ∞ along the feasible ray. If problem (1) admits an optimal solution, then no iterate with d ( k ) ≠ 0 d^{(k)}\ne0 d ( k ) = 0 has $\Delta^{(k)}=+\infty.
Proof. See Appendix B.3.3.
Remark 5.1 (Detecting unboundedness in practice). The exit on Δ ( k ) = + ∞ \Delta^{(k)}=+\infty Δ ( k ) = + ∞ does not always occur on unbounded problems. Let Q = diag ( 1 , 0 ) Q=\operatorname{diag}(1,0) Q = diag ( 1 , 0 ) , c = ( 0 , − 1 ) c=(0,-1) c = ( 0 , − 1 ) , and C = { x ∈ R 2 : − 1 ≤ x 1 ≤ 1 } C=\{x\in\mathbb{R}^2:-1\le x_1\le1\} C = { x ∈ R 2 : − 1 ≤ x 1 ≤ 1 } , so that the problem is unbounded below along ( 0 , 1 ) (0,1) ( 0 , 1 ) . From x ( 0 ) = ( 1 , 0 ) x^{(0)}=(1,0) x ( 0 ) = ( 1 , 0 ) , the iterates yield x ( k ) = ( ( − 1 ) k , 2 k ) x^{(k)}=((-1)^k,2k) x ( k ) = (( − 1 ) k , 2 k ) and Δ ( k ) = 2 \Delta^{(k)}=2 Δ ( k ) = 2 , so that V ( x ( k ) ) → − ∞ V(x^{(k)})\to-\infty V ( x ( k ) ) → − ∞ . In practice one must therefore also monitor V V V .
Convergence of the MSD algorithm
We now assume that problem (1) admits an optimal solution. Let V s V^s V s denote its optimal value. By Lemma 5.1 and Proposition 5.1, every step of Algorithm 2 is well-defined and decreases V V V . We show next that the iterates converge to an optimal solution. In particular, when the multipliers at the limit are positive, the iterates identify its active set after finitely many steps and then converge at a linear rate.
Theorem 5.1. Suppose problem (1) admits an optimal solution, let x ( 0 ) ∈ C x^{(0)}\in C x ( 0 ) ∈ C , and let x ( k ) x^{(k)} x ( k ) denote the k k k -th iterate of Algorithm 2 with ε = 0 \varepsilon=0 ε = 0 and every p ( k ) p^{(k)} p ( k ) chosen so that (LI) holds at x ( k ) x^{(k)} x ( k ) .
(i) Either d ( K ) = 0 d^{(K)}=0 d ( K ) = 0 for some K ∈ N K\in\mathbb{N} K ∈ N , in which case x ( K ) x^{(K)} x ( K ) is an optimal solution and we set x s : = x ( K ) x^s:=x^{(K)} x s := x ( K ) , or the iterates converge to an optimal solution x s x^s x s .
(ii) Assume J ( x s ) = E ( x s ) J(x^s)=\mathcal{E}(x^s) J ( x s ) = E ( x s ) , i.e., the dual solution p s p^s p s is positive on every constraint active at x s x^s x s . Then there is K ′ ∈ N K'\in\mathbb{N} K ′ ∈ N such that every k ≥ K ′ k\ge K' k ≥ K ′ at which the iteration has not stopped satisfies
E ( x ( k ) ) = J ( x ( k ) ) = E ( x s ) and Δ ( k ) = Δ ∗ ( k ) . (45) \mathcal{E}(x^{(k)})=J(x^{(k)})=\mathcal{E}(x^s)\quad\text{and}\quad\Delta^{(k)}=\Delta_*^{(k)}. \tag*{(45)} E ( x ( k ) ) = J ( x ( k ) ) = E ( x s ) and Δ ( k ) = Δ ∗ ( k ) . (45) Let S ˉ : = P E ( x s ) ⊥ Q P E ( x s ) ⊥ \bar{S}:=P_{\mathcal{E}(x^s)}^\perp QP_{\mathcal{E}(x^s)}^\perp S ˉ := P E ( x s ) ⊥ Q P E ( x s ) ⊥ and, when S ˉ ≠ 0 \bar{S}\ne0 S ˉ = 0 , let κ ˉ \bar{\kappa} κ ˉ denote the ratio of its largest to its smallest positive eigenvalue. If S ˉ = 0 \bar{S}=0 S ˉ = 0 , then d ( K ′ ) = 0 d^{(K')}=0 d ( K ′ ) = 0 . Otherwise, every such k k k satisfies
V ( x ( k + 1 ) ) − V s ≤ ( κ ˉ − 1 κ ˉ + 1 ) 2 ( V ( x ( k ) ) − V s ) . (46) V(x^{(k+1)})-V^s\le\left(\frac{\bar{\kappa}-1}{\bar{\kappa}+1}\right)^2\left(V(x^{(k)})-V^s\right). \tag*{(46)} V ( x ( k + 1 ) ) − V s ≤ ( κ ˉ + 1 κ ˉ − 1 ) 2 ( V ( x ( k ) ) − V s ) . (46)
Proof. The idea is to show that there are no blocked tails, that is, infinitely many consecutive blocked steps of summable length along which ∥ d ( k ) ∥ 2 \lVert d^{(k)}\rVert_{2} ∥ d ( k ) ∥ 2 stays bounded away from zero. Without a blocked tail, Lemma 5.1 and a polyhedral error bound give V ( x ( k ) ) ↓ V s V(x^{(k)}) \downarrow V^{s} V ( x ( k ) ) ↓ V s , and a finite-length argument gives convergence of the iterates. For part (ii), the condition J ( x s ) = E ( x s ) J(x^{s}) = \mathcal{E}(x^{s}) J ( x s ) = E ( x s ) keeps the multipliers positive near x s x^{s} x s , so once an iterate reaches the face of x s x^{s} x s , every later step is a steepest descent step along that face, and the Kantorovich inequality gives (42). The proof is technical and is relegated to Appendix B.3. □ \square □
Numerical experiments and applications
In this section, we test Algorithms 1 and 2 on challenging scientific computing problems. Section 6.1 applies Algorithm 1 to a positivity-preserving limiter for discontinuous Galerkin schemes, and Section 6.2 applies Algorithm 2 to high-dimensional optimal control problems with bounds on the control, one for the Poisson equation in 3D and one for the 2D, time-dependent heat equation. All experiments were carried out in MATLAB R2025b on an Apple M3 processor with 16 GB of RAM, with versions of MOSEK [40 ] , Gurobi [41 ] , and HiGHS [42 ] downloaded on 23 May 2026. We measure times with tic and toc. MOSEK runs the primal and dual feasibility with relative gap tolerances 10 − 10 10^{-10} 1 0 − 10 , Gurobi uses the barrier convergence tolerance 10 − 14 10^{-14} 1 0 − 14 with feasibility and optimality tolerances 10 − 9 10^{-9} 1 0 − 9 , and HiGHS uses optimality and primal and dual feasibility tolerances 10 − 10 10^{-10} 1 0 − 10 . Gurobi runs with its default method, which is the barrier method for convex quadratic programs, and HiGHS with its interior-point solver. All our code will be available on GitHub.2
Positivity-preserving limiters for finite element methods
We apply Algorithm 1 to a positivity-preserving limiter for numerical schemes [8 , 43 ] . We consider here the limiter of Liu et al. [8 ] , who enforce positivity in a semi-implicit discontinuous Galerkin (DG) scheme for Fokker–Planck equations, without losing conservation or high-order accuracy, by a two-stage postprocessing applied after each time step. Stage 1 corrects the cell averages of the DG solution by solving a convex quadratic program. Stage 2 then applies the Zhang–Shu scaling limiter [44 , 45 ] in each cell, which removes negative point values without changing the cell averages. Stage 1 is the problem we solve with Algorithm 1.
On a uniform mesh of n n n square cells, let f ˉ ∈ R n \bar{f} \in\mathbb{R}^{n} f ˉ ∈ R n hold the cell averages of the DG solution and let 1 ∈ R n \mathbf{1} \in\mathbb{R}^{n} 1 ∈ R n denote the vector of ones. Stage 1 solves [8 Lemma 1]
min x ∈ R n 1 2 ∥ x − f ˉ ∥ 2 2 subject to ⟨ 1 , x ⟩ = ⟨ 1 , f ˉ ⟩ and x ≥ ℓ 1 , (47) \min_{x\in\mathbb{R}^{n}} \frac{1}{2}\lVert x-\bar{f}\rVert_{2}^{2}\ \text{subject to }\langle\mathbf{1},x\rangle=\langle\mathbf{1},\bar{f}\rangle\ \text{and }x\geq\ell\mathbf{1}, \tag*{(47)} x ∈ R n min 2 1 ∥ x − f ˉ ∥ 2 2 subject to ⟨ 1 , x ⟩ = ⟨ 1 , f ˉ ⟩ and x ≥ ℓ 1 , (47)
where ℓ > 0 \ell>0 ℓ > 0 is the lower bound. The equality constraint conserves the total mass, and its coefficients are equal because the cells have equal areas. We pose problem (43) on the hyperplane ⟨ 1 , x ⟩ = ⟨ 1 , f ˉ ⟩ \langle\mathbf{1},x\rangle=\langle\mathbf{1},\bar{f}\rangle ⟨ 1 , x ⟩ = ⟨ 1 , f ˉ ⟩ . Problem (43) is then problem (1) with Q = I n × n Q=I_{n\times n} Q = I n × n and c = − f ˉ c=-\bar{f} c = − f ˉ , the case t = 1 t=1 t = 1 of Section 4.3.
We start Algorithm 1 from the feasible point x ( 0 ) = ℓ 1 + s ( max ( f ˉ , ℓ 1 ) − ℓ 1 ) x^{(0)}=\ell\mathbf{1}+s(\max(\bar{f},\ell\mathbf{1})-\ell\mathbf{1}) x ( 0 ) = ℓ 1 + s ( max ( f ˉ , ℓ 1 ) − ℓ 1 ) with s = ( ⟨ 1 , f ˉ ⟩ − n ℓ ) / ( ⟨ 1 , max ( f ˉ , ℓ 1 ) ⟩ − n ℓ ) s=(\langle\mathbf{1},\bar{f}\rangle-n\ell)/(\langle\mathbf{1},\max(\bar{f},\ell\mathbf{1})\rangle-n\ell) s = (⟨ 1 , f ˉ ⟩ − n ℓ ) / (⟨ 1 , max ( f ˉ , ℓ 1 )⟩ − n ℓ ) . For problem (43), the NNLS problem (7) at each iteration reduces to one scalar equation. From x ( 0 ) x^{(0)} x ( 0 ) , the coordinates above the bound reach it in increasing order of their cell averages, so our implementation computes the breakpoints after sorting these coordinates by cell average.
We compare Algorithm 1 with DR, the relaxed Douglas–Rachford splitting method of [8 ] , Algorithm DR, run with the parameters of [8 Remark 2] , which come from [46 ] . DR stops when h ∥ y ( k + 1 ) − y ( k ) ∥ 2 < ϵ t o l h\lVert y^{(k+1)}-y^{(k)}\rVert_2<\epsilon_{\mathrm{tol}} h ∥ y ( k + 1 ) − y ( k ) ∥ 2 < ϵ tol , where y ( k ) y^{(k)} y ( k ) is its auxiliary variable and h h h is the side length of a cell [8 Algorithm DR and Remark 3] . We do not test the interior-point solvers of § 6.2 on problem (47) because they cannot use the clipped cell averages as a starting point, which Algorithm 1 and DR do, and because problem (47) has up to 4.2 ⋅ 10 6 4.2\cdot10^6 4.2 ⋅ 1 0 6 variables at every time step. Neither our implementation of Algorithm 1 nor that of DR was tuned for speed.
A synthetic benchmark
Liu et al. [8, Section 4.6] time the DR algorithm on problem (47) with the synthetic data of [47 Appendix A] . On this data, f ˉ \bar{f} f ˉ holds the values of the function f f f on [ 0 , 1 ] 2 [0,1]^2 [ 0 , 1 ] 2
f ( x , y ) = { − 0.25 if ∣ x − 0.25 ∣ ≤ δ / 4 or ∣ x − 0.75 ∣ ≤ δ / 4 , cos 8 ( 2 π x ) + 10 − 13 otherwise, (48) f(x,y)=\begin{cases}
-0.25 & \text{if } \lvert x-0.25\rvert\le\delta/4\text{ or }\lvert x-0.75\rvert\le\delta/4,\\
\cos^8(2\pi x)+10^{-13} & \text{otherwise,}
\end{cases} \tag*{(48)} f ( x , y ) = { − 0.25 cos 8 ( 2 π x ) + 1 0 − 13 if ∣ x − 0.25 ∣ ≤ δ /4 or ∣ x − 0.75 ∣ ≤ δ /4 , otherwise, (48)
at the points of the grid { 0 , h , … , 1 − h } 2 \{0,h,\ldots,1-h\}^2 { 0 , h , … , 1 − h } 2 , so that n = h − 2 n=h^{-2} n = h − 2 . We take l = 10 − 13 l=10^{-13} l = 1 0 − 13 , ϵ t o l = 10 − 13 \epsilon_{\mathrm{tol}}=10^{-13} ϵ tol = 1 0 − 13 , and h = 2 − 7 , 2 − 8 , … , 2 − 11 h=2^{-7},2^{-8},\ldots,2^{-11} h = 2 − 7 , 2 − 8 , … , 2 − 11 from [8 Section 4 and Table 3] , and we pose problem (47) on all n n n points. Neither paper states the grid points or δ \delta δ . We take the grid above and δ = 0.05 \delta=0.05 δ = 0.05 , the fraction of [ 0 , 1 ] 2 [0,1]^2 [ 0 , 1 ] 2 that the two bands cover, which makes between 4.69 % 4.69\% 4.69% and 5.47 % 5.47\% 5.47% of the entries of f ˉ \bar{f} f ˉ negative, close to the 5 % 5\% 5% of Liu et al. We solve each instance one hundred times for each solver and we report the mean time of one solve. The KKT conditions of problem (47) give its minimizer as x r e f = max ( f ˉ + ν ∗ 1 , l 1 ) x_{\mathrm{ref}}=\max(\bar{f}+\nu^*\mathbf{1},l\mathbf{1}) x ref = max ( f ˉ + ν ∗ 1 , l 1 ) , where ν ∗ ∈ R \nu^*\in\mathbb{R} ν ∗ ∈ R solves ⟨ 1 , max ( f ˉ + ν 1 , l 1 ) ⟩ = ⟨ 1 , f ˉ ⟩ \langle\mathbf{1},\max(\bar{f}+\nu\mathbf{1},l\mathbf{1})\rangle=\langle\mathbf{1},\bar{f}\rangle ⟨ 1 , max ( f ˉ + ν 1 , l 1 )⟩ = ⟨ 1 , f ˉ ⟩ . We compute ν ∗ \nu^* ν ∗ by bisection and measure the accuracy of each solver against x r e f x_{\mathrm{ref}} x ref .
Table 2 reports the results. On every instance, both solvers return points that satisfy the lower bound, lie within 10 − 13 10^{-13} 1 0 − 13 of x r e f x_{\mathrm{ref}} x ref in the maximum norm, and conserve the total mass to within 4 ⋅ 10 − 8 4\cdot10^{-8} 4 ⋅ 1 0 − 8 . Algorithm 1 is faster than DR on four of the five grids, by at most a factor of 1.4, and slower by less than 3 % 3\% 3% for h = 2 − 8 h=2^{-8} h = 2 − 8 .
Table 2. Iterations of one solve, mean time of one solve of problem (47) over one hundred solves, along with the error ∥ x − x r e f ∥ ∞ \lVert x-x_{\mathrm{ref}}\rVert_\infty ∥ x − x ref ∥ ∞ and conservation residual ∣ ⟨ 1 , x ⟩ − ⟨ 1 , f ˉ ⟩ ∣ \lvert\langle\mathbf{1},x\rangle-\langle\mathbf{1},\bar{f}\rangle\rvert ∣⟨ 1 , x ⟩ − ⟨ 1 , f ˉ ⟩∣ .
A discontinuous Galerkin scheme for a Fokker–Planck equation
We now solve problem (47) at every time step of a DG computation, in which Algorithm 1 and DR each postprocess their own DG solution. Stage 1 runs at every time step, up to 30 400 steps per computation, so this test measures how the conservation error of each solver builds up over a long computation. We reproduce the accuracy test of [8 Section 4.1] , which solves the linear Fokker–Planck equation
∂ t f = Δ f + ∇ ⋅ ( f v ) in [ 1 , 20 ] × Ω , Ω = [ − 10 , 10 ] 2 , (49) \partial_t f=\Delta f+\nabla\cdot(fv)\text{ in }[1,20]\times\Omega,\qquad\Omega=[-10,10]^2, \tag*{(49)} ∂ t f = Δ f + ∇ ⋅ ( f v ) in [ 1 , 20 ] × Ω , Ω = [ − 10 , 10 ] 2 , (49)
with exact solution
f ( τ , v ) = 1 2 π ( 1 − e − 2 τ ) exp ( − ∥ v ∥ 2 2 2 ( 1 − e − 2 τ ) ) . (50) f(\tau,\mathbf{v}) = \frac{1}{2\pi(1-e^{-2\tau})}\exp\left(-\frac{\lVert\mathbf{v}\rVert_2^2}{2(1-e^{-2\tau})}\right). \tag*{(50)} f ( τ , v ) = 2 π ( 1 − e − 2 τ ) 1 exp ( − 2 ( 1 − e − 2 τ ) ∥ v ∥ 2 2 ) . (50)
We use the semi-implicit DG method of [8 Section 3.1] with polynomials of degree k = 2 k=2 k = 2 and k = 3 k=3 k = 3 on uniform grids of 64 2 64^2 6 4 2 , 128 2 128^2 12 8 2 , and 256 2 256^2 25 6 2 square cells. It treats the diffusion term implicitly, with the non-symmetric interior penalty form and penalty σ / h \sigma/h σ / h , σ = 1 \sigma=1 σ = 1 , and the convection term explicitly, with a Lax–Friedrichs flux. After each time step, the two-stage postprocessing solves problem (43) with l = 10 − 13 l=10^{-13} l = 1 0 − 13 for the cell averages and then applies the Zhang–Shu scaling limiter. We use the pairs ( h , Δ τ ) (h,\Delta\tau) ( h , Δ τ ) of [8 Table 1] with the same h h h in the penalty as in the stopping rule of DR, 0 for the tolerance in the Zhang–Shu limiter, Δ τ = 19 / 119 \Delta\tau=19/119 Δ τ = 19/119 for k = 3 k=3 k = 3 on the grid of 64 2 64^2 6 4 2 cells, and problem (43) posed on all cells. The basis on each cell is the tensor-product Legendre basis, orthonormal on the reference cell [ − 1 2 , 1 2 ] 2 [-\frac{1}{2},\frac{1}{2}]^2 [ − 2 1 , 2 1 ] 2 , and every integral uses the tensor-product Gauss rule with k + 1 k+1 k + 1 points per direction. The initial data is the L 2 L^2 L 2 projection of (50) at τ = 1 \tau=1 τ = 1 , followed by the Zhang–Shu limiter. On the boundary faces, the scheme has no interior penalty terms, as in [8 Section 3.1] . We set the convective flux to zero, as the conservation of mass in [8 Section 3.1] requires. The scheme thus imposes the zero-flux condition ( ∇ f + f v ) ⋅ n = 0 (\nabla f+f\mathbf{v})\cdot\mathbf{n}=0 ( ∇ f + f v ) ⋅ n = 0 weakly, where n \mathbf{n} n is the outer unit normal to Ω \Omega Ω . Liu et al. [8 Section 4.1] instead impose boundary data from the exact solution (50) . For τ ∈ [ 1 , 20 ] \tau\in[1,20] τ ∈ [ 1 , 20 ] , this solution is at most 3.1 ⋅ 10 − 23 3.1\cdot10^{-23} 3.1 ⋅ 1 0 − 23 on the boundary of Ω \Omega Ω , and its flux ( ∇ f + f v ) ⋅ n (\nabla f+f\mathbf{v})\cdot\mathbf{n} ( ∇ f + f v ) ⋅ n there is at most 2.3 ⋅ 10 − 24 2.3\cdot10^{-24} 2.3 ⋅ 1 0 − 24 in absolute value.
Table 3: DG computation of § 6.1.2 with stage 1 solved by Algorithm 1 and DR: drift of the total mass and time of stage 1. The time of stage 1 is summed over the time steps, where the time of each step is the mean over five calls.
Both solvers keep every cell average at or above l l l . With Algorithm 1, the errors L h 2 L_h^2 L h 2 and L h ∞ L_h^\infty L h ∞ at τ = 20 \tau=20 τ = 20 agree with [8 Table 1] in all four printed digits for both degrees and all three grids, and the convergence rates computed from them agree with those of [8 Table 1] to within 10 − 3 10^{-3} 1 0 − 3 . With DR, the errors also agree in all four digits, except for k = 3 k=3 k = 3 on the grid of 256 2 256^2 25 6 2 cells, where DR gives 7.556 ⋅ 10 − 8 7.556\cdot10^{-8} 7.556 ⋅ 1 0 − 8 and 8.083 ⋅ 10 − 8 8.083\cdot10^{-8} 8.083 ⋅ 1 0 − 8 against 7.555 ⋅ 10 − 8 7.555\cdot10^{-8} 7.555 ⋅ 1 0 − 8 and 8.079 ⋅ 10 − 8 8.079\cdot10^{-8} 8.079 ⋅ 1 0 − 8 .
Algorithm 1 keeps the drift max j ∣ ⟨ f h j , 1 ⟩ − ⟨ f h 0 , 1 ⟩ ∣ \max_j|\langle f_h^j,1\rangle-\langle f_h^0,1\rangle| max j ∣ ⟨ f h j , 1 ⟩ − ⟨ f h 0 , 1 ⟩ ∣ of the total mass over the time steps j j j below 2.2 ⋅ 10 − 12 2.2\cdot10^{-12} 2.2 ⋅ 1 0 − 12 in every computation, and DR below 2.4 ⋅ 10 − 10 2.4\cdot10^{-10} 2.4 ⋅ 1 0 − 10 (Table 3 ). Across the six computations, the drift of DR is 96 to 3800 times that of Algorithm 1. The exact solution (50) has total mass 1 on R 2 \mathbb{R}^2 R 2 , which sets the scale of the drift. This drift comes from stage 1, because DR returns max ( g ( k + 1 ) , 11 ) \max(\mathbf{g}^{(k+1)},1\mathbf{1}) max ( g ( k + 1 ) , 1 1 ) [8 Algorithm DR] , which satisfies the lower bound exactly and the conservation constraint only approximately. Stage 1 is faster with DR than with Algorithm 1, by factors of 1.8, 7.8, and 18.1 for k = 2 k = 2 k = 2 on the three grids and of 1.9, 7.7, and 16.3 for k = 3 k = 3 k = 3 . In § 6.1.1 , DR takes 17 iterations per solve. Here, after the first time step, DR takes one iteration per step on the grids of 128 2 128^2 12 8 2 and 256 2 256^2 25 6 2 cells, and Algorithm 1 takes two or three. On the grids of 64 2 64^2 6 4 2 cells, DR takes at most 22 iterations per step and Algorithm 1 at most 63. Our nonoptimized implementation of Algorithm 1, despite being slower than DR, conserves the total mass 96 to 3800 times more accurately than DR.
Table 3. DG computation of Section 6.1.2 with stage 1 solved by Algorithm 1 and DR: drift of the total mass and time of stage 1. The time of stage 1 is summed over the time steps, where the time of each step is the mean over five calls.
PDE-constrained optimization problems
Pearson and Gondzio [7 ] solve a challenging PDE-constrained optimization problem (PDECO) with bounds on the control. Other PDECOs with different degrees of complexity have also been considered in [48 , 49 , 50 , 51 , 52 , 53 ] . Here, we consider two of the examples from [7 ] , in which we seek a state y y y and a control u u u on a domain Ω \Omega Ω that minimize
1 2 ∫ Ω ( y − y d ) 2 + β 2 ∫ Ω u 2 subject to L y = u in Ω , y = 0 on ∂ Ω , u a ≤ u ≤ u b , (51) \frac{1}{2}\int_{\Omega}(y-y_d)^2+\frac{\beta}{2}\int_{\Omega}u^2 \quad\text{subject to}\quad\mathcal{L}y=u\text{ in }\Omega,\quad y=0\text{ on }\partial\Omega,\quad u_a\leq u\leq u_b, \tag*{(51)} 2 1 ∫ Ω ( y − y d ) 2 + 2 β ∫ Ω u 2 subject to L y = u in Ω , y = 0 on ∂ Ω , u a ≤ u ≤ u b , (51)
where L \mathcal{L} L is a differential operator, y d y_d y d is the desired state, β > 0 \beta> 0 β > 0 weights the control, and u a < u b u_a < u_b u a < u b are the bounds. In § 6.2.1 , L = − Δ \mathcal{L} = -\Delta L = − Δ on a three-dimensional domain. In § 6.2.2 , L \mathcal{L} L is the heat operator on a two-dimensional domain, and the integrals in (51) run over space and time. We run each solver five times on each instance, after one discarded call, and report the mean time of one solve. This time covers all the work the solver needs to return the state and the control, and excludes the accuracy measures below.
We discretize problem (51) before we optimize, as [7 ] do, with Q 1 Q1 Q 1 finite elements for the state, the control, and the adjoint, and consistent mass matrices [7 Sections 4.1 and 5] . With y \boldsymbol{y} y and u ∈ R N \boldsymbol{u}\in\mathbb{R}^{N} u ∈ R N the vectors of nodal values and y d \boldsymbol{y}_d y d those of the desired state, the discrete problem is
min y , u ∈ R N 1 2 ⟨ y − y d , M y ( y − y d ) ⟩ + β 2 ⟨ u , M u u ⟩ subject to K y = B u and u a 1 ≤ u ≤ u b 1 , (52) \min_{\boldsymbol{y},\boldsymbol{u}\in\mathbb{R}^{N}}\frac{1}{2}\langle\boldsymbol{y}-\boldsymbol{y}_d,M_y(\boldsymbol{y}-\boldsymbol{y}_d)\rangle+\frac{\beta}{2}\langle\boldsymbol{u},M_u\boldsymbol{u}\rangle\quad\text{subject to}\quad K\boldsymbol{y}=B\boldsymbol{u}\quad\text{and}\quad u_a\mathbf{1}\leq\boldsymbol{u}\leq u_b\mathbf{1}, \tag*{(52)} y , u ∈ R N min 2 1 ⟨ y − y d , M y ( y − y d )⟩ + 2 β ⟨ u , M u u ⟩ subject to K y = B u and u a 1 ≤ u ≤ u b 1 , (52)
where K K K discretizes L \mathcal{L} L together with the conditions on y y y and is invertible, B u B\boldsymbol{u} B u discretizes the right-hand side u u u of the state equation, and the mass matrices M y M_y M y and M u M_u M u discretize the two integrals in (51) and are SPD. Since the control acts on all of Ω \Omega Ω , the matrix B B B is built from the mass matrix of Ω \Omega Ω . MOSEK, Gurobi, and HiGHS solve problem (52) in the 2 N 2N 2 N variables x = ( y , u ) \boldsymbol{x}=(\boldsymbol{y},\boldsymbol{u}) x = ( y , u ) , with N N N equality constraints and 2 N 2N 2 N bounds. Before each call, we divide the objective by ∥ M y y d ∥ ∞ \lVert M_y\boldsymbol{y}_d\rVert_{\infty} ∥ M y y d ∥ ∞ , which leaves the minimizer unchanged. We score the point each solver returns, whatever status it reports.
Algorithm 2 solves a smaller, equivalent problem. Since K K K is invertible, the state equation gives y = G u \boldsymbol{y}=G\boldsymbol{u} y = G u with G = K − 1 B G=K^{-1}B G = K − 1 B , and eliminating y \boldsymbol{y} y from (52) leaves
min u ∈ R N 1 2 ⟨ u , Q r e d u ⟩ + ⟨ c r e d , u ⟩ subject to u a 1 ≤ u ≤ u b 1 , (53) \min_{\boldsymbol{u}\in\mathbb{R}^{N}}\frac{1}{2}\langle\boldsymbol{u},Q_{\mathrm{red}}\boldsymbol{u}\rangle+\langle\boldsymbol{c}_{\mathrm{red}},\boldsymbol{u}\rangle\quad\text{subject to}\quad u_a\mathbf{1}\leq\boldsymbol{u}\leq u_b\mathbf{1}, \tag*{(53)} u ∈ R N min 2 1 ⟨ u , Q red u ⟩ + ⟨ c red , u ⟩ subject to u a 1 ≤ u ≤ u b 1 , (53)
where Q r e d = G T M y G + β M u Q_{\mathrm{red}}=G^{T}M_yG+\beta M_u Q red = G T M y G + β M u and c r e d = − G T M y y d \boldsymbol{c}_{\mathrm{red}}=-G^{T}M_y\boldsymbol{y}_d c red = − G T M y y d . Problem (53) is problem (1) with Q = Q r e d Q=Q_{\mathrm{red}} Q = Q red , c = c r e d \boldsymbol{c}=\boldsymbol{c}_{\mathrm{red}} c = c red , A = [ I , − I ] A=[I,-I] A = [ I , − I ] , and b = ( u b 1 ; − u a 1 ) \boldsymbol{b}=(u_b\mathbf{1};-u_a\mathbf{1}) b = ( u b 1 ; − u a 1 ) . The matrix Q r e d Q_{\mathrm{red}} Q red is SPD because M u M_u M u is and β > 0 \beta>0 β > 0 . Since the rows of A A A are signed unit vectors and u a < u b u_a<u_b u a < u b , at most one bound on each coordinate u i u_i u i is active at a feasible point. The rows of every active set are therefore linearly independent, and Theorem 5.1 applies to problem (53) . We never form the dense matrix Q r e d Q_{\mathrm{red}} Q red . Each iteration of Algorithm 2 needs one product Q r e d d ( k ) Q_{\mathrm{red}}d^{(k)} Q red d ( k ) , which gives the exact step length Δ ∗ ( k ) \Delta_*^{(k)} Δ ∗ ( k ) and, since u ( k + 1 ) = u ( k ) + Δ ∗ ( k ) d ( k ) \boldsymbol{u}^{(k+1)}=\boldsymbol{u}^{(k)}+\Delta_*^{(k)}d^{(k)} u ( k + 1 ) = u ( k ) + Δ ∗ ( k ) d ( k ) , also updates the gradient. We apply Q r e d Q_{\mathrm{red}} Q red as an operator, at the cost given in each subsection below. On a box, the NNLS problem (7) splits into N N N scalar problems with closed-form solutions. We start from the midpoint u ( 0 ) = 1 2 ( u a + u b ) 1 u^{(0)} = \frac{1}{2}(u_a + u_b)\mathbf{1} u ( 0 ) = 2 1 ( u a + u b ) 1 and stop when ∥ d ( k ) ∥ 2 ≤ ε \lVert d^{(k)} \rVert_2 \le\varepsilon ∥ d ( k ) ∥ 2 ≤ ε with ε = 10 − 8 \varepsilon= 10^{-8} ε = 1 0 − 8 . Before solving, we divide Q r e d Q_{\mathrm{red}} Q red and c r e d c_{\mathrm{red}} c red by ∥ c r e d ∥ ∞ \lVert c_{\mathrm{red}}\rVert_\infty ∥ c red ∥ ∞ , which leaves the minimizer unchanged and makes ε \varepsilon ε relative to the data.
We measure accuracy in two ways. First, we build a reference solution. We fix the controls in the final active set of Algorithm 2 at their bounds, solve for the other controls in (53) by conjugate gradients to relative residual 10 − 14 10^{-14} 1 0 − 14 , and accept the result only if every free control lies strictly inside its bounds and the bound multipliers it implies are nonnegative. For every solver, we report the gap V ( u ) − V ( u r e f ) V(u) - V(u_{\mathrm{ref}}) V ( u ) − V ( u ref ) , where V V V is the objective of (53) . Second, we compute the optimality residuals following [54 ] . Specifically, we write problem (52) as the minimization of 1 2 ⟨ x , Q x ⟩ + ⟨ c , x ⟩ \frac{1}{2}\langle x,Qx\rangle+ \langle c,x\rangle 2 1 ⟨ x , Q x ⟩ + ⟨ c , x ⟩ subject to E x = 0 Ex = 0 E x = 0 and A x ≤ b Ax \le b A x ≤ b , where Q = diag ( M y , β M u ) Q = \operatorname{diag}(M_y,\beta M_u) Q = diag ( M y , β M u ) , c = ( − M y y d ; 0 ) c = (-M_y y_d;0) c = ( − M y y d ; 0 ) , E = [ K , − B ] E = [K,-B] E = [ K , − B ] , A = [ 0 , I ; 0 , − I ] A = [0,I;0,-I] A = [ 0 , I ; 0 , − I ] , and b = ( u b 1 ; − u a 1 ) b = (u_b\mathbf{1};-u_a\mathbf{1}) b = ( u b 1 ; − u a 1 ) , and let λ \lambda λ and p ≥ 0 p \ge0 p ≥ 0 be the multipliers of the two constraints. The primal residual, the dual residual, and the duality gap of a point ( x , λ , p ) (x,\lambda,p) ( x , λ , p ) are
r p = max ( ∥ E x ∥ ∞ , ∥ max ( 0 , A x − b ) ∥ ∞ ) , (54) r_p = \max(\lVert Ex\rVert_\infty,\lVert\max(0,Ax-b)\rVert_\infty), \tag*{(54)} r p = max (∥ E x ∥ ∞ , ∥ max ( 0 , A x − b ) ∥ ∞ ) , (54)
r d = ∥ Q x + c + E T λ + A T p ∥ ∞ , (55) r_d = \lVert Qx + c + E^{\mathsf{T}}\lambda+ A^{\mathsf{T}}p\rVert_\infty, \tag*{(55)} r d = ∥ Q x + c + E T λ + A T p ∥ ∞ , (55)
r g = ∣ ⟨ x , Q x ⟩ + ⟨ c , x ⟩ + ⟨ b , p ⟩ ∣ . (56) r_g = \lvert\langle x,Qx\rangle+ \langle c,x\rangle+ \langle b,p\rangle\rvert. \tag*{(56)} r g = ∣⟨ x , Q x ⟩ + ⟨ c , x ⟩ + ⟨ b , p ⟩∣ . (56)
Since the data of (52) carry a factor β h 3 \beta h^3 β h 3 in § 6.2.1 and β Δ τ h 2 \beta\Delta\tau h^2 β Δ τ h 2 in § 6.2.2 , we divide each residual by the size of the terms that compose it. Finally, each solver is scored from its own multipliers, and Algorithm 2 also from its adjoint state.
Control of the Poisson equation in 3D
The first example, from [7 Section 5] , is problem (51) with L = − Δ \mathcal{L} = -\Delta L = − Δ on Ω = [ 0 , 1 ] 3 \Omega= [0,1]^3 Ω = [ 0 , 1 ] 3 and desired state y d ( x ) = exp ( − 64 ∥ x − ( 1 / 2 , 1 / 2 , 1 / 2 ) ∥ 2 2 ) y_d(x) = \exp(-64\lVert x-(1/2,1/2,1/2)\rVert_2^2) y d ( x ) = exp ( − 64 ∥ x − ( 1/2 , 1/2 , 1/2 ) ∥ 2 2 ) . We impose y = 0 y = 0 y = 0 on ∂ Ω \partial\Omega ∂ Ω and follow the conventions of IFISS [55 ] . The grid is uniform with spacing h h h and 1 / h − 1 1/h - 1 1/ h − 1 interior nodes per direction, so the state and the control each have N = ( 1 / h − 1 ) 3 N = (1/h - 1)^3 N = ( 1/ h − 1 ) 3 degrees of freedom, and y d y_d y d holds the nodal values of y d y_d y d . The matrix K K K is the Q1 stiffness matrix of the negative Laplacian, and B = M y = M u = M B = M_y = M_u = M B = M y = M u = M is the consistent Q1 mass matrix. We use the pairs ( β , u a , u b ) = ( 10 − 4 , 0 , 20 ) (\beta,u_a,u_b) = (10^{-4},0,20) ( β , u a , u b ) = ( 1 0 − 4 , 0 , 20 ) and ( 10 − 2 , 0 , 1 ) (10^{-2},0,1) ( 1 0 − 2 , 0 , 1 ) and the meshes h = 2 − 2 , … , 2 − 5 h = 2^{-2},\ldots,2^{-5} h = 2 − 2 , … , 2 − 5 of [7 Table 6] .
We factor K K K once per mesh by a sparse Cholesky factorization under a nested-dissection ordering. One application of Q r e d = M K − 1 M K − 1 M + β M Q_{\mathrm{red}} = MK^{-1}MK^{-1}M + \beta M Q red = M K − 1 M K − 1 M + β M then costs four products with M M M and two solves with K K K , each a pair of triangular solves.
Table 4 reports the times and Table 5 compares the accuracy. Algorithm 2 is the fastest solver on every mesh of both pairs. Over both pairs and all four meshes, the objective gap of Algorithm 2 stays at or below 5.0 ⋅ 10 − 18 5.0\cdot10^{-18} 5.0 ⋅ 1 0 − 18 , smaller than those of HiGHS and MOSEK. Its largest relative residual, 2.8 ⋅ 10 − 9 2.8\cdot10^{-9} 2.8 ⋅ 1 0 − 9 , lies between those of HiGHS and Gurobi and that of MOSEK. Thus Algorithm 2 performs well and with high accuracy compared to highly optimized quadratic programming solvers.
Table 4. Poisson control of Section 6.2.1. Mean time in seconds of one solve of problem (48), over five solves, for both pairs ( β , u a , u b ) (\beta, u_a, u_b) ( β , u a , u b ) , and in parentheses the time divided by that of the fastest solver on the same mesh.
Table 5. Poisson control of Section 6.2.1. Largest objective gap V ( u ) − V ( u r e f ) V(u) - V(u_{\mathrm{ref}}) V ( u ) − V ( u ref ) and relative residual max ( r p , r d , r g ) \max(r_p, r_d, r_g) max ( r p , r d , r g ) each solver returns.
Control of the time-dependent heat equation in 2D
The second example is the time-dependent heat control problem of [7 Sections 3.3 and 5] , in which y τ − Δ y = u y_\tau-\Delta y=u y τ − Δ y = u holds on Ω × ( 0 , T ] \Omega\times(0,T] Ω × ( 0 , T ] and the integrals in (51) run over Ω × ( 0 , T ] \Omega\times(0,T] Ω × ( 0 , T ] . Pearson and Gondzio state neither the domain nor the initial and boundary data for this example. We take Ω = [ 0 , 1 ] 2 \Omega= [0,1]^2 Ω = [ 0 , 1 ] 2 , T = 1 T = 1 T = 1 , y = 0 y = 0 y = 0 on ∂ Ω \partial\Omega ∂ Ω , and y = 0 y = 0 y = 0 at τ = 0 \tau= 0 τ = 0 , as in the heat control examples of [56 , 57 ] , and the desired state y d ( x , τ ) = 1 2 sin ( 2 π x 1 x 2 ) y_d(x,\tau) = \frac{1}{2}\sin(2\pi x_1x_2) y d ( x , τ ) = 2 1 sin ( 2 π x 1 x 2 ) , the two-dimensional form of the desired state of [56, Section 6.2]. This desired state changes sign on Ω \Omega Ω , so both bounds can be active.
We discretize in space as in Section 6.2.1 and in time by the backward Euler method with step Δ τ = 0.02 \Delta\tau= 0.02 Δ τ = 0.02 , so T / Δ τ = 50 T/\Delta\tau= 50 T /Δ τ = 50 steps reach T T T , and we apply the trapezoidal rule to the integrals in time [7, Section 3.3]. With the time levels stacked, the state and the control each have N = 50 ( 1 / h − 1 ) 2 N = 50(1/h - 1)^2 N = 50 ( 1/ h − 1 ) 2 degrees of freedom. The matrix K \mathbf{K} K is block lower bidiagonal, with one backward Euler step on the diagonal and the negated spatial mass matrix below it. The matrix B \mathbf{B} B is Δ τ \Delta\tau Δ τ times the block diagonal spatial mass matrix, and M y = M u M_y = M_u M y = M u carries the trapezoidal weights. We use the pairs ( β , u a , u b ) = ( 10 − 2 , 0 , 1 ) (\beta, u_a, u_b) = (10^{-2}, 0, 1) ( β , u a , u b ) = ( 1 0 − 2 , 0 , 1 ) and ( 10 − 4 , 0 , 30 ) (10^{-4}, 0, 30) ( 1 0 − 4 , 0 , 30 ) of [7, Table 7] and the meshes h = 2 − 2 , … , 2 − 6 h = 2^{-2}, \ldots, 2^{-6} h = 2 − 2 , … , 2 − 6 .
On a uniform grid, the one-dimensional Q 1 Q1 Q 1 stiffness and mass matrices share the eigenvectors of the discrete sine transform, so the two-dimensional sine basis diagonalizes the spatial stiffness, the spatial mass, and one backward Euler step. In that basis, K − 1 \mathbf{K}^{-1} K − 1 and K − T \mathbf{K}^{-\mathsf{T}} K − T reduce to one scalar recurrence per spatial mode, forward and backward in time, and M y M_y M y and M u M_u M u are diagonal. One product with Q r e d Q_{\mathrm{red}} Q red then costs two sine transforms of every time level, which we compute as dense matrix products.
Table 6 reports the times. For β = 10 − 2 \beta= 10^{-2} β = 1 0 − 2 , Algorithm 2 is the fastest solver on every mesh. For β = 10 − 4 \beta= 10^{-4} β = 1 0 − 4 , it is the fastest on the two finest meshes, and Gurobi or HiGHS is faster on the three coarsest, by a factor of at most 2.0. From h = 2 − 5 h = 2^{-5} h = 2 − 5 to h = 2 − 6 h = 2^{-6} h = 2 − 6 , N N N grows by a factor of 4.13 and the time of every solver by a factor of 19 to 25. Over both pairs and all five meshes, the objective gap of Algorithm 2 stays at or below 9.6 ⋅ 10 − 16 9.6 \cdot10^{-16} 9.6 ⋅ 1 0 − 16 , again smaller than those of HiGHS and MOSEK. Its largest relative residual, 2.6 ⋅ 10 − 10 2.6 \cdot10^{-10} 2.6 ⋅ 1 0 − 10 , is also smaller than theirs (Table 7). MOSEK loses accuracy as the mesh is refined. For β = 10 − 4 \beta= 10^{-4} β = 1 0 − 4 on the finest mesh, its gap is 8.9 ⋅ 10 − 8 8.9 \cdot10^{-8} 8.9 ⋅ 1 0 − 8 , its largest relative residual 2.4 ⋅ 10 − 4 2.4 \cdot10^{-4} 2.4 ⋅ 1 0 − 4 , and its control differs from the reference by 1.6 ⋅ 10 − 1 1.6 \cdot10^{-1} 1.6 ⋅ 1 0 − 1 in the infinity norm, against a bound u ≤ 30 u \le30 u ≤ 30 . HiGHS reports an optimal status on every mesh, yet its control for β = 10 − 4 \beta= 10^{-4} β = 1 0 − 4 at h = 2 − 4 h = 2^{-4} h = 2 − 4 differs from the reference by 7.3 ⋅ 10 − 4 7.3 \cdot10^{-4} 7.3 ⋅ 1 0 − 4 in the infinity norm. Again, Algorithm 2 performs well and with high accuracy compared to highly optimized quadratic programming solvers.
Conclusion and future work
In this work, we solved the gradient flow of linear and convex quadratic programs exactly. By the minimal selection principle, the slow solution from every feasible point is piecewise explicit, and one
Table 6. Heat control of Section 6.2.2 with Δ τ = 0.02 \Delta\tau= 0.02 Δ τ = 0.02 , with columns as in Table 4. The table has no Gurobi entry for h = 2 − 6 h = 2^{-6} h = 2 − 6 because Gurobi did not converge within four hours for that problem.
Table 7. Heat control of Section 6.2.2. Largest objective gap V ( u ) − V ( u r e f ) V(u) - V(u_{\mathrm{ref}}) V ( u ) − V ( u ref ) and relative residual max ( r p , r d , r g ) \max(r_p, r_d, r_g) max ( r p , r d , r g ) each solver returns.
NNLS problem gives the initial direction of each piece. A piece ends when a constraint becomes active or a multiplier changes sign, and the norm of the velocity decreases strictly at every discontinuity. When strict complementarity fails, a recursion of cone projections selects the active set of the next piece. We proved that the slow solution has finitely many pieces for every SPSD matrix Q Q Q and that its final piece yields an optimal solution or a certificate of unboundedness. For Q = t I n × n Q = tI_{n \times n} Q = t I n × n , every piece is a ray with a closed-form breakpoint and no active set occurs twice, which gives the exact Algorithm 1. For general SPSD Q Q Q , we analyzed minimal selection descent (Algorithm 2), the explicit Euler discretization of the slow system, whose steps each solve one NNLS problem over the active constraints. Its iterates converge to an optimal solution when one exists and otherwise certify unboundedness or drive the objective to − ∞ -\infty − ∞ . When the multipliers at the limit are positive, the iterates identify the optimal active set after finitely many steps and then converge at a linear rate. In our experiments, Algorithm 1 conserved the total mass in a positivity-preserving limiter more accurately than a state-of-the-art Douglas–Rachford splitting method, and minimal selection descent reached the optimal value of PDE-constrained control problems to machine precision in times competitive with commercial and open-source solvers.
Several directions remain open. First, we expect that our solution method can be adapted to other differential inclusions: second-order gradient inclusions, including the ODE associated with Nesterov’s accelerated gradient method [58 ] ; inclusions arising from saddle-point problems and from mirror descent; and some gradient inclusions posed in Hilbert spaces. Second, we expect that minimal selection descent can be extended to general convex optimization problems, and our convergence analysis to other discretizations of the slow system. Third, we expect that differential inclusions can yield solution methods for second-order cone programs. Fourth, we expect that implementations of our algorithms on graphics processing units (GPUs) can scale to large linear and quadratic programs and be benchmarked against existing solvers at that scale. Finally, the experiments of Section 6 suggest applications to positivity-preserving variational schemes for PDEs and to PDE-constrained optimization. Some PDE-constrained optimization problems can be posed as second-order cone programs, and their continuous formulations lead to the gradient inclusions in Hilbert spaces mentioned above.
Conflicts of interest
The authors declare they have no conflicts of interest.
Declaration of generative AI and AI-assisted technologies in the writing process
During the preparation of this work, the authors used Claude Code (Anthropic) to assist with proof development, numerical implementation, and drafting. The authors reviewed and edited the content as needed and take full responsibility for the content of the publication.
Mathematical background
We list here some definitions and results from convex and functional analysis used in this work. For comprehensive references, see [14 , 16 , 34 , 36 ] .
Definition A.1 (Convex sets). A set C ⊂ R n C \subset\mathbb{R}^{n} C ⊂ R n is convex if for every pair ( x 1 , x 2 ) ∈ C × C (x_{1},x_{2}) \in C \times C ( x 1 , x 2 ) ∈ C × C and every scalar λ ∈ ( 0 , 1 ) \lambda\in(0,1) λ ∈ ( 0 , 1 ) , the vector λ x 1 + ( 1 − λ ) x 2 \lambda x_{1} + (1-\lambda)x_{2} λ x 1 + ( 1 − λ ) x 2 is contained in C C C .
Definition A.2 (Closed convex polyhedra). A set C ⊂ R n C \subset\mathbb{R}^{n} C ⊂ R n is a closed convex polyhedron if it is the intersection of a finite number of half-spaces:
C = { x ∈ R n : ⟨ u i , x ⟩ ≤ r i for every i ∈ [ m ] } . C = \left\{x \in\mathbb{R}^{n} : \langle u_{i}, x \rangle\leq r_{i}\text{ for every }i \in[m]\right\}. C = { x ∈ R n : ⟨ u i , x ⟩ ≤ r i for every i ∈ [ m ] } . with { u 1 , … , u m } ⊂ R n \{u_{1},\ldots,u_{m}\} \subset\mathbb{R}^{n} { u 1 , … , u m } ⊂ R n and { r 1 , … , r m } ⊂ R \{r_{1},\ldots,r_{m}\} \subset\mathbb{R} { r 1 , … , r m } ⊂ R .
Definition A.3 (Convex cones). A set K ⊂ R n K \subset\mathbb{R}^{n} K ⊂ R n is a cone if 0 ∈ K 0 \in K 0 ∈ K and λ x ∈ K \lambda x \in K λ x ∈ K for all x ∈ K x \in K x ∈ K and λ > 0 \lambda> 0 λ > 0 . A cone K K K is convex if it contains the point ∑ i = 1 m η i x i \sum_{i=1}^{m}\eta_{i}x_{i} ∑ i = 1 m η i x i whenever x i ∈ K x_{i} \in K x i ∈ K and η i ≥ 0 \eta_{i} \geq0 η i ≥ 0 for every i ∈ [ m ] i \in[m] i ∈ [ m ] .
Definition A.4 (Polyhedral cones). The conical hull of some finite set { x 1 , … , x m } ⊂ R n \{x_{1},\ldots,x_{m}\} \subset\mathbb{R}^{n} { x 1 , … , x m } ⊂ R n is
cone { x 1 , … , x m } : = { ∑ i = 1 m η i x i : η i ≥ 0 for every i ∈ [ m ] } . \operatorname{cone}\{x_{1},\ldots,x_{m}\} := \left\{\sum_{i=1}^{m}\eta_{i}x_{i} : \eta_{i} \geq0\text{ for every }i \in[m]\right\}. cone { x 1 , … , x m } := { i = 1 ∑ m η i x i : η i ≥ 0 for every i ∈ [ m ] } . A cone is polyhedral if K = cone { x 1 , … , x m } K = \operatorname{cone}\{x_{1},\ldots,x_{m}\} K = cone { x 1 , … , x m } for some finite set { x 1 , … , x m } ⊂ R n \{x_{1},\ldots,x_{m}\} \subset\mathbb{R}^{n} { x 1 , … , x m } ⊂ R n .
Definition A.5 (Proper functions). A function f f f defined over R n \mathbb{R}^{n} R n is proper if its domain dom f : = { x ∈ R n : f ( x ) < + ∞ } \operatorname{dom} f := \{x \in\mathbb{R}^{n} : f(x) < +\infty\} dom f := { x ∈ R n : f ( x ) < + ∞ } is nonempty and f ( x ) > − ∞ f(x) > -\infty f ( x ) > − ∞ for every x ∈ dom f x \in\operatorname{dom} f x ∈ dom f .
Definition A.6 (Lower semicontinuity). A proper function f : R n → R ∪ { + ∞ } f : \mathbb{R}^{n} \to\mathbb{R} \cup\{+\infty\} f : R n → R ∪ { + ∞ } is lower semicontinuous at x ∈ dom f x \in\operatorname{dom} f x ∈ dom f if for every sequence { x k } k = 1 + ∞ ⊂ R n \{x_{k}\}_{k=1}^{+\infty} \subset\mathbb{R}^{n} { x k } k = 1 + ∞ ⊂ R n converging to x x x , lim inf k → + ∞ f ( x k ) ≥ f ( x ) \liminf_{k \to+\infty} f(x_{k}) \geq f(x) lim inf k → + ∞ f ( x k ) ≥ f ( x ) .
We say f f f is lower semicontinuous if it is lower semicontinuous at every x ∈ dom f x \in\operatorname{dom} f x ∈ dom f .
Definition A.7 (Convex functions and the space Γ 0 ( R n ) \Gamma_{0}(\mathbb{R}^{n}) Γ 0 ( R n ) ). A proper function f : R n → R ∪ { + ∞ } f : \mathbb{R}^{n} \to\mathbb{R} \cup\{+\infty\} f : R n → R ∪ { + ∞ } is convex if dom f \operatorname{dom} f dom f is convex and if for every pair ( x 1 , x 2 ) ∈ dom f × dom f (x_{1},x_{2}) \in\operatorname{dom} f \times\operatorname{dom} f ( x 1 , x 2 ) ∈ dom f × dom f and every λ ∈ [ 0 , 1 ] \lambda\in[0,1] λ ∈ [ 0 , 1 ] ,
f ( λ x 1 + ( 1 − λ ) x 2 ) ≤ λ f ( x 1 ) + ( 1 − λ ) f ( x 2 ) . f(\lambda x_{1} + (1-\lambda)x_{2}) \leq\lambda f(x_{1}) + (1-\lambda)f(x_{2}). f ( λ x 1 + ( 1 − λ ) x 2 ) ≤ λ f ( x 1 ) + ( 1 − λ ) f ( x 2 ) . It is strictly convex if the inequality above is strict whenever x 1 ≠ x 2 x_{1} \ne x_{2} x 1 = x 2 and λ ∈ ( 0 , 1 ) \lambda\in(0,1) λ ∈ ( 0 , 1 ) , and it is t t t -strongly convex with t > 0 t > 0 t > 0 if for every pair ( x 1 , x 2 ) ∈ dom f × dom f (x_{1},x_{2}) \in\operatorname{dom} f \times\operatorname{dom} f ( x 1 , x 2 ) ∈ dom f × dom f and every λ ∈ [ 0 , 1 ] \lambda\in[0,1] λ ∈ [ 0 , 1 ] ,
f ( λ x 1 + ( 1 − λ ) x 2 ) ≤ λ f ( x 1 ) + ( 1 − λ ) f ( x 2 ) − t 2 λ ( 1 − λ ) ∥ x 1 − x 2 ∥ 2 2 . f(\lambda x_{1} + (1-\lambda)x_{2}) \leq\lambda f(x_{1}) + (1-\lambda)f(x_{2}) - \frac{t}{2}\lambda(1-\lambda)\lVert x_{1}-x_{2}\rVert_{2}^{2}. f ( λ x 1 + ( 1 − λ ) x 2 ) ≤ λ f ( x 1 ) + ( 1 − λ ) f ( x 2 ) − 2 t λ ( 1 − λ ) ∥ x 1 − x 2 ∥ 2 2 . The space of proper, lower semicontinuous, and convex functions over R n \mathbb{R}^{n} R n is denoted by Γ 0 ( R n ) \Gamma_{0}(\mathbb{R}^{n}) Γ 0 ( R n ) .
Definition A.8 (Subdifferentiability). A function f ∈ Γ 0 ( R n ) f \in\Gamma_0(\mathbb{R}^n) f ∈ Γ 0 ( R n ) is subdifferentiable at x ∈ R n x \in\mathbb{R}^n x ∈ R n if there exists p ∈ R n p \in\mathbb{R}^n p ∈ R n such that
f ( y ) ≥ f ( x ) + ⟨ p , y − x ⟩ for every y ∈ dom f . (57) f(y) \ge f(x) + \langle p, y - x \rangle\quad\text{for every } y \in\operatorname{dom} f. \tag*{(57)} f ( y ) ≥ f ( x ) + ⟨ p , y − x ⟩ for every y ∈ dom f . (57) In this case, p p p is called a subgradient of the function f f f at x x x . The set of subgradients at x ∈ R n x \in\mathbb{R}^n x ∈ R n is called the subdifferential of f f f at x x x and is denoted by ∂ f ( x ) \partial f(x) ∂ f ( x ) . When nonempty, ∂ f ( x ) \partial f(x) ∂ f ( x ) is a closed convex set. We note that f f f has a unique subgradient at x x x if and only if f f f is differentiable at x x x and ∂ f ( x ) = { ∇ f ( x ) } \partial f(x) = \{\nabla f(x)\} ∂ f ( x ) = { ∇ f ( x )} [36 Theorem 25.1] . Finally, subdifferentials satisfy the following calculus rules.
(i) Let f 1 , f 2 ∈ Γ 0 ( R n ) f_1, f_2 \in\Gamma_0(\mathbb{R}^n) f 1 , f 2 ∈ Γ 0 ( R n ) and assume ri dom f 1 ∩ ri dom f 2 ≠ ∅ \operatorname{ri}\operatorname{dom} f_1 \cap\operatorname{ri}\operatorname{dom} f_2 \ne\varnothing ri dom f 1 ∩ ri dom f 2 = ∅ . Then f 1 + f 2 ∈ Γ 0 ( R n ) f_1 + f_2 \in\Gamma_0(\mathbb{R}^n) f 1 + f 2 ∈ Γ 0 ( R n ) and ∂ ( f 1 + f 2 ) ( x ) = ∂ f 1 ( x ) + ∂ f 2 ( x ) \partial(f_1 + f_2)(x) = \partial f_1(x) + \partial f_2(x) ∂ ( f 1 + f 2 ) ( x ) = ∂ f 1 ( x ) + ∂ f 2 ( x ) for every x ∈ dom f 1 ∩ dom f 2 x \in\operatorname{dom} f_1 \cap\operatorname{dom} f_2 x ∈ dom f 1 ∩ dom f 2 .
(ii) Let f ∈ Γ 0 ( R n ) f \in\Gamma_0(\mathbb{R}^n) f ∈ Γ 0 ( R n ) , g ∈ Γ 0 ( R m ) g \in\Gamma_0(\mathbb{R}^m) g ∈ Γ 0 ( R m ) and assume Im ( A ) ∩ ri dom g ≠ ∅ \operatorname{Im}(A) \cap\operatorname{ri}\operatorname{dom} g \ne\varnothing Im ( A ) ∩ ri dom g = ∅ . Then the function x ↦ g ( A x ) x \mapsto g(Ax) x ↦ g ( A x ) is in Γ 0 ( R n ) \Gamma_0(\mathbb{R}^n) Γ 0 ( R n ) and ∂ ( g ∘ A ) ( x ) = A T ∂ g ( A x ) \partial(g \circ A)(x) = A^{\mathsf{T}}\partial g(Ax) ∂ ( g ∘ A ) ( x ) = A T ∂ g ( A x ) for every x ∈ R n x \in\mathbb{R}^n x ∈ R n satisfying A x ∈ dom g Ax \in\operatorname{dom} g A x ∈ dom g .
(iii) Let f ∈ Γ 0 ( R n ) f \in\Gamma_0(\mathbb{R}^n) f ∈ Γ 0 ( R n ) , g ∈ Γ 0 ( R m ) g \in\Gamma_0(\mathbb{R}^m) g ∈ Γ 0 ( R m ) and assume ri dom f ∩ ri dom ( g ∘ A ) ≠ ∅ \operatorname{ri}\operatorname{dom} f \cap\operatorname{ri}\operatorname{dom}(g \circ A) \ne\varnothing ri dom f ∩ ri dom ( g ∘ A ) = ∅ and Im ( A ) ∩ ri dom g ≠ ∅ \operatorname{Im}(A) \cap\operatorname{ri}\operatorname{dom} g \ne\varnothing Im ( A ) ∩ ri dom g = ∅ . Then ∂ ( f + g ∘ A ) ( x ) = ∂ f ( x ) + A T ∂ g ( A x ) \partial(f + g \circ A)(x) = \partial f(x) + A^{\mathsf{T}}\partial g(Ax) ∂ ( f + g ∘ A ) ( x ) = ∂ f ( x ) + A T ∂ g ( A x ) for every x ∈ dom f ∩ dom ( g ∘ A ) x \in\operatorname{dom} f \cap\operatorname{dom}(g \circ A) x ∈ dom f ∩ dom ( g ∘ A ) .
See [36 Theorem 23.8] for the proof of (i) and [36 Theorem 23.9] for the proof of (ii). The proof of (iii) follows by applying (i) with f 1 = f f_1 = f f 1 = f and f 2 = g ∘ A f_2 = g \circ A f 2 = g ∘ A , then (ii).
Definition A.9 (Monotone and maximal monotone mappings). Let F F F denote a set-valued mapping from R n \mathbb{R}^n R n to R n \mathbb{R}^n R n with graph { ( x , p ) ∈ R n × R n : p ∈ F ( x ) } \{(x,p) \in\mathbb{R}^n \times\mathbb{R}^n : p \in F(x)\} {( x , p ) ∈ R n × R n : p ∈ F ( x )} . We say F F F is monotone if
⟨ p 1 − p 2 , x 1 − x 2 ⟩ ≥ 0 (58) \langle p_1 - p_2, x_1 - x_2 \rangle\ge0 \tag*{(58)} ⟨ p 1 − p 2 , x 1 − x 2 ⟩ ≥ 0 (58) for every x 1 , x 2 ∈ R n x_1, x_2 \in\mathbb{R}^n x 1 , x 2 ∈ R n and every p 1 ∈ F ( x 1 ) p_1 \in F(x_1) p 1 ∈ F ( x 1 ) , p 2 ∈ F ( x 2 ) p_2 \in F(x_2) p 2 ∈ F ( x 2 ) . It is maximal if no other set-valued mapping strictly contains the graph of F F F . If f ∈ Γ 0 ( R n ) f \in\Gamma_0(\mathbb{R}^n) f ∈ Γ 0 ( R n ) , then its subdifferential is a maximal monotone mapping [34 Theorem 12.17] .
Definition A.10 (Characteristic functions). The characteristic function of a subset C ⊂ R n C \subset\mathbb{R}^n C ⊂ R n is defined as
χ C ( x ) : = { 0 , if x ∈ C , + ∞ otherwise . \chi_C(x) :=
\begin{cases}
0, & \text{if } x \in C,\\
+\infty& \text{otherwise}.
\end{cases} χ C ( x ) := { 0 , + ∞ if x ∈ C , otherwise . We have χ C ∈ Γ 0 ( R n ) \chi_C \in\Gamma_0(\mathbb{R}^n) χ C ∈ Γ 0 ( R n ) if and only if C C C is closed and convex.
Definition A.11 (Euclidean projection). Let C ⊂ R n C \subset\mathbb{R}^n C ⊂ R n denote a closed convex subset and let x ∈ R n x \in\mathbb{R}^n x ∈ R n . The projection of x x x onto C C C is defined as proj C ( x ) : = arg min y ∈ C ∥ x − y ∥ 2 \operatorname{proj}_C(x) := \operatorname*{arg\,min}_{y \in C} \lVert x - y \rVert^2 proj C ( x ) := arg min y ∈ C ∥ x − y ∥ 2 . The projection of x x x onto C C C exists, is unique, and satisfies the characterization [16 Section III, 3.1] :
⟨ x − proj C ( x ) , y − proj C ( x ) ⟩ ≤ 0 for every y ∈ C . (59) \langle x - \operatorname{proj}_C(x), y - \operatorname{proj}_C(x) \rangle\le0 \quad\text{for every } y \in C. \tag*{(59)} ⟨ x − proj C ( x ) , y − proj C ( x )⟩ ≤ 0 for every y ∈ C . (59)
Definition A.12 (Normal and tangent cones). Let C ⊂ R n C \subset\mathbb{R}^n C ⊂ R n denote a closed convex set. A vector s ∈ R n s \in\mathbb{R}^n s ∈ R n is said to be normal to C C C at x ∈ C x \in C x ∈ C if
⟨ s , y − x ⟩ ≤ 0 for every y ∈ C . \langle s, y - x \rangle\le0 \quad\text{for every } y \in C. ⟨ s , y − x ⟩ ≤ 0 for every y ∈ C . This set of directions is called the normal cone of C C C at x x x and is denoted by N C ( x ) N_C(x) N C ( x ) . The tangent cone of C C C at x x x is the polar of the normal cone of C C C at x x x :
T C ( x ) : = { d ∈ R n : ⟨ s , d ⟩ ≤ 0 for every s ∈ N C ( x ) } . T_C(x) := \{d \in\mathbb{R}^n : \langle s, d \rangle\le0 \text{ for every } s \in N_C(x)\}. T C ( x ) := { d ∈ R n : ⟨ s , d ⟩ ≤ 0 for every s ∈ N C ( x )} .
Technical proofs
Proof of Theorem 3.2
We prove part (i) in Steps 1–3, part (ii) in Step 4, and part (iii) in Step 5. Throughout, we write d : = d ( x 0 ) \boldsymbol{d}:=\boldsymbol{d}(x_0) d := d ( x 0 ) , p : = p ( x 0 ) \boldsymbol{p}:=\boldsymbol{p}(x_0) p := p ( x 0 ) , J : = J ( x 0 ) J:=J(x_0) J := J ( x 0 ) , and G : = G ( x 0 ) G:=G(x_0) G := G ( x 0 ) , where p \boldsymbol{p} p is chosen so that (LI) holds. Note that [ A d ] J ∪ G = 0 [A\boldsymbol{d}]_{J\cup G}=0 [ A d ] J ∪ G = 0 , by (13c) on J J J and (31) on G G G .
Step 1. For a set K K K with J ⊆ K ⊆ J ∪ G J\subseteq K\subseteq J\cup G J ⊆ K ⊆ J ∪ G , let y K ( τ ) : = x 0 + M K ( τ ) d \boldsymbol{y}_K(\tau):=x_0+M_K(\tau)\boldsymbol{d} y K ( τ ) := x 0 + M K ( τ ) d denote the candidate built from K K K , with M K M_K M K as in § 3.5 . We construct one such set K K K whose candidate is feasible on a right neighborhood of 0. Two identities hold. First, the computation behind (23) , with K K K in place of J ( x 0 ) J(x_0) J ( x 0 ) , gives A K M K ( τ ) d = 0 A_KM_K(\tau)\boldsymbol{d}=0 A K M K ( τ ) d = 0 , so the constraints of K K K remain active along y K \boldsymbol{y}_K y K . Second, y ˙ K ( τ ) = − P K ⊥ ( Q y K ( τ ) + c ) \dot{\boldsymbol{y}}_K(\tau)=-P_K^\perp(Q\boldsymbol{y}_K(\tau)+\boldsymbol{c}) y ˙ K ( τ ) = − P K ⊥ ( Q y K ( τ ) + c ) for every τ ≥ 0 \tau\geq0 τ ≥ 0 . Indeed, differentiating (22) gives y ˙ K ( τ ) = e − P K ⊥ Q τ d \dot{\boldsymbol{y}}_K(\tau)=e^{-P_K^\perp Q\tau}\boldsymbol{d} y ˙ K ( τ ) = e − P K ⊥ Q τ d , while integrating the derivative of s ↦ e − P K ⊥ Q s s\mapsto e^{-P_K^\perp Qs} s ↦ e − P K ⊥ Q s over [ 0 , τ ] [0,\tau] [ 0 , τ ] gives e − P K ⊥ Q τ − I n × n = − P K ⊥ Q M K ( τ ) e^{-P_K^\perp Q\tau}-I_{n\times n}=-P_K^\perp QM_K(\tau) e − P K ⊥ Q τ − I n × n = − P K ⊥ Q M K ( τ ) , so that y ˙ K ( τ ) = d − P K ⊥ Q M K ( τ ) d \dot{\boldsymbol{y}}_K(\tau)=\boldsymbol{d}-P_K^\perp QM_K(\tau)\boldsymbol{d} y ˙ K ( τ ) = d − P K ⊥ Q M K ( τ ) d . The right-hand side − P K ⊥ ( Q y K ( τ ) + c ) -P_K^\perp(Q\boldsymbol{y}_K(\tau)+\boldsymbol{c}) − P K ⊥ ( Q y K ( τ ) + c ) of the claimed identity expands to − P K ⊥ ( Q x 0 + c ) − P K ⊥ Q M K ( τ ) d -P_K^\perp(Qx_0+\boldsymbol{c})-P_K^\perp QM_K(\tau)\boldsymbol{d} − P K ⊥ ( Q x 0 + c ) − P K ⊥ Q M K ( τ ) d , and − P K ⊥ ( Q x 0 + c ) = d -P_K^\perp(Qx_0+\boldsymbol{c})=\boldsymbol{d} − P K ⊥ ( Q x 0 + c ) = d because d ∈ ker A K \boldsymbol{d}\in\ker A_K d ∈ ker A K and − ( Q x 0 + c ) − d = A J T p J ∈ Im ( A K T ) = ( ker A K ) ⊥ -(Qx_0+\boldsymbol{c})-\boldsymbol{d}=A_J^\mathsf{T}\boldsymbol{p}_J\in\operatorname{Im}(A_K^\mathsf{T})=(\ker A_K)^\perp − ( Q x 0 + c ) − d = A J T p J ∈ Im ( A K T ) = ( ker A K ) ⊥ . We call
n K ( τ ) : = − Q y K ( τ ) − c − y ˙ K ( τ ) = − ( I n × n − P K ⊥ ) ( Q y K ( τ ) + c ) (60) \boldsymbol{n}_K(\tau):=-Q\boldsymbol{y}_K(\tau)-\boldsymbol{c}-\dot{\boldsymbol{y}}_K(\tau)=-(I_{n\times n}-P_K^\perp)(Q\boldsymbol{y}_K(\tau)+\boldsymbol{c}) \tag*{(60)} n K ( τ ) := − Q y K ( τ ) − c − y ˙ K ( τ ) = − ( I n × n − P K ⊥ ) ( Q y K ( τ ) + c ) (60)
the normal part along y K \boldsymbol{y}_K y K , which in particular lies in Im ( A K T ) \operatorname{Im}(A_K^\mathsf{T}) Im ( A K T ) .
Suppose that y K ( τ ) ∈ C \boldsymbol{y}_K(\tau)\in C y K ( τ ) ∈ C and n K ( τ ) ∈ N C ( y K ( τ ) ) \boldsymbol{n}_K(\tau)\in N_C(\boldsymbol{y}_K(\tau)) n K ( τ ) ∈ N C ( y K ( τ )) for every τ \tau τ in an interval [ 0 , τ ′ ) [0,\tau') [ 0 , τ ′ ) . Then y ˙ K ( τ ) = − ∇ x f ( y K ( τ ) ) − n K ( τ ) ∈ − ∂ x V ( y K ( τ ) ) \dot{\boldsymbol{y}}_K(\tau)=-\nabla_xf(\boldsymbol{y}_K(\tau))-\boldsymbol{n}_K(\tau)\in-\partial_xV(\boldsymbol{y}_K(\tau)) y ˙ K ( τ ) = − ∇ x f ( y K ( τ )) − n K ( τ ) ∈ − ∂ x V ( y K ( τ )) , so the candidate solves (4) on [ 0 , τ ′ ) [0,\tau') [ 0 , τ ′ ) with y K ( 0 ) = x 0 \boldsymbol{y}_K(0)=x_0 y K ( 0 ) = x 0 . It therefore coincides there with the slow solution by the monotonicity argument in Step 3 of the proof of Theorem 3.1. Constraints inactive at x 0 x_0 x 0 and constraints i ∈ E ( x 0 ) i\in\mathcal{E}(x_0) i ∈ E ( x 0 ) with [ A d ] i < 0 [A\boldsymbol{d}]_i<0 [ A d ] i < 0 stay inactive for all sufficiently small τ > 0 \tau>0 τ > 0 along any candidate. Near τ = 0 \tau=0 τ = 0 , the two conditions therefore reduce to the signs of the constraint values ν i ( τ ) : = [ A y K ( τ ) ] i − b i \nu_i(\tau):=[A\boldsymbol{y}_K(\tau)]_i-b_i ν i ( τ ) := [ A y K ( τ ) ] i − b i for i ∈ G \ K i\in G\backslash K i ∈ G \ K and to the membership of n K ( τ ) \boldsymbol{n}_K(\tau) n K ( τ ) in N C ( y K ( τ ) ) N_C(\boldsymbol{y}_K(\tau)) N C ( y K ( τ )) . This membership holds whenever n K ( τ ) \boldsymbol{n}_K(\tau) n K ( τ ) is a nonnegative combination of the rows of A K A_K A K , since the constraints of K K K are active along y K \boldsymbol{y}_K y K . Steps 2 and 3 below build K K K iteratively and determine these signs near τ = 0 \tau=0 τ = 0 , respectively.
Step 2. A recursion selects K K K and produces vectors d r \boldsymbol{d}_r d r , starting from d 0 : = d \boldsymbol{d}_0:=\boldsymbol{d} d 0 := d , a running set L 1 : = J L_1:=J L 1 := J , and an undecided set U 1 : = G U_1:=G U 1 := G . At stage r = 1 , 2 , … r=1,2,\ldots r = 1 , 2 , … , let z r \boldsymbol{z}_r z r denote the projection of − Q d r − 1 -Q\boldsymbol{d}_{r-1} − Q d r − 1 onto the cone K r : = { s ∈ ker A L r : [ A s ] U r ≤ 0 } \mathcal{K}_r:=\{\boldsymbol{s}\in\ker A_{L_r}:[A\boldsymbol{s}]_{U_r}\leq0\} K r := { s ∈ ker A L r : [ A s ] U r ≤ 0 } . Its polar cone is the normal cone of the polyhedron K r \mathcal{K}_r K r at 0, where every constraint of K r \mathcal{K}_r K r is active, so it is generated by the vectors ± A i T \pm A_i^\mathsf{T} ± A i T for i ∈ L r i\in L_r i ∈ L r and A i T A_i^\mathsf{T} A i T for i ∈ U r i\in U_r i ∈ U r [16 Example 5.2.6b] . Moreau’s decomposition theorem, applied to K r \mathcal{K}_r K r and its polar cone as in (6) , therefore gives − Q d r − 1 = z r + A T η -Q\boldsymbol{d}_{r-1}=\boldsymbol{z}_r+A^\mathsf{T}\boldsymbol{\eta} − Q d r − 1 = z r + A T η for some η ∈ R m \boldsymbol{\eta}\in\mathbb{R}^m η ∈ R m supported on L r ∪ U r L_r\cup U_r L r ∪ U r , with η U r ≥ 0 \boldsymbol{\eta}_{U_r}\geq0 η U r ≥ 0 and ⟨ z r , A T η ⟩ = 0 \langle\boldsymbol{z}_r,A^\mathsf{T}\boldsymbol{\eta}\rangle=0 ⟨ z r , A T η ⟩ = 0 . Let H r H_r H r denote the set of such vectors η \boldsymbol{\eta} η , the multipliers of stage r r r . When the rows of A L r ∪ U r A_{L_r\cup U_r} A L r ∪ U r are linearly dependent, H r H_r H r may contain more than one vector. Every η ∈ H r \boldsymbol{\eta}\in H_r η ∈ H r satisfies the complementarity condition
η i [ A z r ] i = 0 for every i ∈ U r . (61) \eta_i[A\boldsymbol{z}_r]_i=0 \qquad\text{for every } i\in U_r. \tag*{(61)} η i [ A z r ] i = 0 for every i ∈ U r . (61)
Indeed, [ A z r ] L r = 0 [A\boldsymbol{z}_r]_{L_r}=0 [ A z r ] L r = 0 , so 0 = ⟨ z r , A T η ⟩ = ∑ i ∈ U r η i [ A z r ] i 0=\langle\boldsymbol{z}_r,A^\mathsf{T}\boldsymbol{\eta}\rangle=\sum_{i\in U_r}\eta_i[A\boldsymbol{z}_r]_i 0 = ⟨ z r , A T η ⟩ = ∑ i ∈ U r η i [ A z r ] i , and every term of this sum is nonpositive. The stage commits U r + : = { i ∈ U r : η i > 0 for some η ∈ H r } U_r^+:=\{i\in U_r:\eta_i>0\text{ for some }\boldsymbol{\eta}\in H_r\} U r + := { i ∈ U r : η i > 0 for some η ∈ H r } , drops D r : = { i ∈ U r : [ A z r ] i < 0 } D_r:=\{i\in U_r:[A\boldsymbol{z}_r]_i<0\} D r := { i ∈ U r : [ A z r ] i < 0 } , and sets L r + 1 : = L r ∪ U r + L_{r+1}:=L_r\cup U_r^+ L r + 1 := L r ∪ U r + , U r + 1 : = U r \ ( U r + ∪ D r ) U_{r+1}:=U_r\backslash(U_r^+\cup D_r) U r + 1 := U r \ ( U r + ∪ D r ) , and d r : = z r \boldsymbol{d}_r:=\boldsymbol{z}_r d r := z r . The sets U r + U_r^+ U r + and D r D_r D r are disjoint by (61) . The set H r H_r H r is convex, so averaging one vector of H r H_r H r for each index of U r + U_r^+ U r + gives a vector η ( r ) ∈ H r \boldsymbol{\eta}^{(r)}\in H_r η ( r ) ∈ H r with η i ( r ) > 0 \eta_i^{(r)}>0 η i ( r ) > 0 for every i ∈ U r + i\in U_r^+ i ∈ U r + ; we fix such a vector. For every i ∈ U r + 1 i\in U_{r+1} i ∈ U r + 1 , the definitions of U r + U_r^+ U r + and D r D_r D r give
η i = 0 for every η ∈ H r and [ A z r ] i = 0. (62) \eta_i=0\text{ for every }\boldsymbol{\eta}\in H_r
\qquad\text{and}\qquad
[A\boldsymbol{z}_r]_i=0. \tag*{(62)} η i = 0 for every η ∈ H r and [ A z r ] i = 0. (62)
A stage that commits or drops removes at least one index from the undecided set, so at most ∣ G ∣ |G| ∣ G ∣ stages do. Let s ˉ \bar{s} s ˉ denote the last such stage, with s ˉ : = 0 \bar{s}:=0 s ˉ := 0 if there is none. The running set stabilizes at K : = L s ˉ + 1 K:=L_{\bar{s}+1} K := L s ˉ + 1 , with J ⊆ K ⊆ J ∪ G J\subseteq K\subseteq J\cup G J ⊆ K ⊆ J ∪ G , and the undecided set at R : = U s ˉ + 1 R:=U_{\bar{s}+1} R := U s ˉ + 1 . We write J ∗ + ( x 0 ) J_*^+(x_0) J ∗ + ( x 0 ) , τ f e a s + ( x 0 ) \tau_{\mathrm{feas}}^+(x_0) τ feas + ( x 0 ) , and τ ∗ + ( x 0 ) \tau_*^+(x_0) τ ∗ + ( x 0 ) for K K K and the objects built from it, as in § 3.5 .
The first stage commits the grazing constraints at which some multiplier of the projection is positive, drops those with [ A z 1 ] i < 0 [A\mathbf{z}_1]_i<0 [ A z 1 ] i < 0 , and leaves the others undecided. Step 3 shows that, along the candidate, a constraint committed at stage r r r carries a weight of order τ r \tau^r τ r in the normal part, and that the value of a constraint dropped at stage r r r is of order τ r + 1 \tau^{r+1} τ r + 1 . For an undecided constraint, both leading terms vanish, and the next stage computes higher-order terms. The next lemma shows that the vectors d r \mathbf{d}_r d r depend on the final set K K K alone.
Lemma B.1 (Final-set identity). For every r ≥ 1 r\geq1 r ≥ 1 , every η ∈ H r \eta\in H_r η ∈ H r is supported on L r + 1 ⊆ K L_{r+1}\subseteq K L r + 1 ⊆ K , and z r ∈ ker A K \mathbf{z}_r\in\ker A_K z r ∈ ker A K . Hence − Q d r − 1 = z r + A K T η K -\mathbf{Q}\mathbf{d}_{r-1}=\mathbf{z}_r+A_K^{\mathsf T}\eta_K − Q d r − 1 = z r + A K T η K for every η ∈ H r \eta\in H_r η ∈ H r , and d r = z r = − P K ⊥ Q d r − 1 = ( − P K ⊥ Q ) r d \mathbf{d}_r=\mathbf{z}_r=-P_K^\perp\mathbf{Q}\mathbf{d}_{r-1}=(-P_K^\perp\mathbf{Q})^r\mathbf{d} d r = z r = − P K ⊥ Q d r − 1 = ( − P K ⊥ Q ) r d .
Proof. Let η ∈ H r \eta\in H_r η ∈ H r . It is supported on L r ∪ U r L_r\cup U_r L r ∪ U r . On D r D_r D r , complementarity (55) gives η i = 0 \eta_i=0 η i = 0 , and on U r + 1 U_{r+1} U r + 1 , (56) gives η i = 0 \eta_i=0 η i = 0 . Hence η \eta η is supported on L r ∪ U r + = L r + 1 ⊆ K L_r\cup U_r^+=L_{r+1}\subseteq K L r ∪ U r + = L r + 1 ⊆ K . Every index of K K K lies in L r L_r L r , in U r + U_r^+ U r + , or in U r + 1 U_{r+1} U r + 1 , because an index committed after stage r r r is undecided at stage r + 1 r+1 r + 1 . The definition of K r K_r K r gives [ A z r ] L r = 0 [A\mathbf{z}_r]_{L_r}=0 [ A z r ] L r = 0 . Complementarity (55) for η ( r ) \eta^{(r)} η ( r ) gives [ A z r ] i = 0 [A\mathbf{z}_r]_i=0 [ A z r ] i = 0 for i ∈ U r + i\in U_r^+ i ∈ U r + , and (56) gives the same for i ∈ U r + 1 i\in U_{r+1} i ∈ U r + 1 . Hence z r ∈ ker A K \mathbf{z}_r\in\ker A_K z r ∈ ker A K , while A T η = A K T η K ∈ Im ( A K T ) = ( ker A K ) ⊥ A^{\mathsf T}\eta=A_K^{\mathsf T}\eta_K\in\operatorname{Im}(A_K^{\mathsf T})=(\ker A_K)^\perp A T η = A K T η K ∈ Im ( A K T ) = ( ker A K ) ⊥ . Uniqueness of the orthogonal decomposition of − Q d r − 1 -\mathbf{Q}\mathbf{d}_{r-1} − Q d r − 1 along ker A K ⊕ Im ( A K T ) \ker A_K\oplus\operatorname{Im}(A_K^{\mathsf T}) ker A K ⊕ Im ( A K T ) gives z r = − P K ⊥ Q d r − 1 \mathbf{z}_r=-P_K^\perp\mathbf{Q}\mathbf{d}_{r-1} z r = − P K ⊥ Q d r − 1 . Since d r = z r \mathbf{d}_r=\mathbf{z}_r d r = z r and d 0 = d \mathbf{d}_0=\mathbf{d} d 0 = d , induction on r r r gives the last claim.
Step 3. Consider the functions of Step 1 along y K \mathbf{y}_K y K . By Lemma B.1, y ˙ K ( τ ) = e − P K ⊥ Q τ d = ∑ r ≥ 0 ( τ r / r ! ) d r \dot{\mathbf{y}}_K(\tau)=e^{-P_K^\perp\mathbf{Q}\tau}\mathbf{d}=\sum_{r\geq0}(\tau^r/r!)\mathbf{d}_r y ˙ K ( τ ) = e − P K ⊥ Q τ d = ∑ r ≥ 0 ( τ r / r !) d r . Every constraint value and the normal part n K \mathbf{n}_K n K are therefore entire functions of τ \tau τ .
Constraint values. Let i ∈ G i\in G i ∈ G be dropped at stage r r r . Then [ A d 0 ] i = [ A d ] i = 0 [A\mathbf{d}_0]_i=[A\mathbf{d}]_i=0 [ A d 0 ] i = [ A d ] i = 0 , while [ A d j ] i = [ A z j ] i = 0 [A\mathbf{d}_j]_i=[A\mathbf{z}_j]_i=0 [ A d j ] i = [ A z j ] i = 0 for 1 ≤ j ≤ r − 1 1\leq j\leq r-1 1 ≤ j ≤ r − 1 by (56), since i ∈ U j + 1 i\in U_{j+1} i ∈ U j + 1 , and [ A d r ] i = [ A z r ] i < 0 [A\mathbf{d}_r]_i=[A\mathbf{z}_r]_i<0 [ A d r ] i = [ A z r ] i < 0 . Hence ν i ( τ ) = ( τ r + 1 / ( r + 1 ) ! ) [ A z r ] i + O ( τ r + 2 ) \nu_i(\tau)=(\tau^{r+1}/(r+1)!)[A\mathbf{z}_r]_i+O(\tau^{r+2}) ν i ( τ ) = ( τ r + 1 / ( r + 1 )!) [ A z r ] i + O ( τ r + 2 ) is negative for all sufficiently small τ > 0 \tau>0 τ > 0 . The constraints of R R R are never decided and form the riding set . Every i ∈ R i\in R i ∈ R has [ A d r ] i = 0 [A\mathbf{d}_r]_i=0 [ A d r ] i = 0 for every r ≥ 0 r\geq0 r ≥ 0 by (56) and i ∈ G i\in G i ∈ G . With Step 1, this shows that R R R is the set of constraints outside K K K that τ f e a s K ( x 0 ) \tau_{\mathrm{feas}}^K(x_0) τ feas K ( x 0 ) ignores. Every constraint outside K ∪ R K\cup R K ∪ R has a value that is negative for all sufficiently small τ > 0 \tau>0 τ > 0 and, by the definition of τ f e a s K ( x 0 ) \tau_{\mathrm{feas}}^K(x_0) τ feas K ( x 0 ) , has no zero in ( 0 , τ f e a s K ( x 0 ) ) (0,\tau_{\mathrm{feas}}^K(x_0)) ( 0 , τ feas K ( x 0 )) . Hence τ f e a s K ( x 0 ) > 0 \tau_{\mathrm{feas}}^K(x_0)>0 τ feas K ( x 0 ) > 0 and
y K ( τ ) ∈ C and E ( y K ( τ ) ) = K ∪ R for every τ ∈ ( 0 , τ f e a s K ( x 0 ) ) . (63) \mathbf{y}_K(\tau)\in\mathcal{C}\quad\text{and}\quad\mathcal{E}(\mathbf{y}_K(\tau))=K\cup R\quad\text{for every }\tau\in(0,\tau_{\mathrm{feas}}^K(x_0)). \tag*{(63)} y K ( τ ) ∈ C and E ( y K ( τ )) = K ∪ R for every τ ∈ ( 0 , τ feas K ( x 0 )) . (63)
Normal part. We have ( d r y K / d τ r ) ( 0 ) = d r − 1 (d^r\mathbf{y}_K/d\tau^r)(0)=\mathbf{d}_{r-1} ( d r y K / d τ r ) ( 0 ) = d r − 1 for r ≥ 1 r\geq1 r ≥ 1 . Evaluating the normal part (54) and its derivatives at τ = 0 \tau=0 τ = 0 gives
n K ( 0 ) = − ( I n × n − P K ⊥ ) ( Q x 0 + c ) = A J T p J = A K T p K , \mathbf{n}_K(0)=-(I_{n\times n}-P_K^\perp)(\mathbf{Q}\mathbf{x}_0+\mathbf{c})=A_J^{\mathsf T}\mathbf{p}_J=A_K^{\mathsf T}\mathbf{p}_K, n K ( 0 ) = − ( I n × n − P K ⊥ ) ( Q x 0 + c ) = A J T p J = A K T p K ,
d r n K d τ r ( 0 ) = − ( I n × n − P K ⊥ ) Q d r − 1 = A K T η K ( r ) ( r ≥ 1 ) , \frac{d^r\mathbf{n}_K}{d\tau^r}(0)=-(I_{n\times n}-P_K^\perp)\mathbf{Q}\mathbf{d}_{r-1}=A_K^{\mathsf T}\eta_K^{(r)}\quad(r\geq1), d τ r d r n K ( 0 ) = − ( I n × n − P K ⊥ ) Q d r − 1 = A K T η K ( r ) ( r ≥ 1 ) ,
where the first line uses − ( Q x 0 + c ) = d + A J T p J -(\mathbf{Q}\mathbf{x}_0+\mathbf{c})=\mathbf{d}+A_J^{\mathsf T}\mathbf{p}_J − ( Q x 0 + c ) = d + A J T p J with d ∈ ker A K \mathbf{d}\in\ker A_K d ∈ ker A K , and the second line uses Lemma B.1 with z r ∈ ker A K \mathbf{z}_r\in\ker A_K z r ∈ ker A K . Let λ max \lambda_{\max} λ m a x denote the largest eigenvalue of Q \mathbf{Q} Q . Since
∥ ( I n × n − P K ⊥ ) Q ∥ 2 ≤ λ max and ∥ P K ⊥ Q ∥ 2 ≤ λ max , \left\|(I_{n\times n}-P_K^\perp)\mathbf{Q}\right\|_2\leq\lambda_{\max}\quad\text{and}\quad\left\|P_K^\perp\mathbf{Q}\right\|_2\leq\lambda_{\max}, ( I n × n − P K ⊥ ) Q 2 ≤ λ m a x and P K ⊥ Q 2 ≤ λ m a x ,
Lemma B.1 also gives ∥ ( d r n K / d τ r ) ( 0 ) ∥ 2 ≤ λ max ∥ d r − 1 ∥ 2 ≤ λ max r ∥ d ∥ 2 \left\|\left(\mathrm{d}^{r}n_K/\mathrm{d}\tau^{r}\right)(0)\right\|_2 \le\lambda_{\max}\left\|d_{r-1}\right\|_2 \le\lambda_{\max}^{r}\left\|d\right\|_2 ∥ ( d r n K / d τ r ) ( 0 ) ∥ 2 ≤ λ m a x ∥ d r − 1 ∥ 2 ≤ λ m a x r ∥ d ∥ 2 . Let
w ( τ ) : = p K + ∑ r = 1 s ˉ τ r r ! η K ( r ) + ( A K ⊤ ) † ρ ( τ ) , ρ ( τ ) : = n K ( τ ) − A K ⊤ ( p K + ∑ r = 1 s ˉ τ r r ! η K ( r ) ) , (64) w(\tau) := p_K + \sum_{r=1}^{\bar{s}} \frac{\tau^r}{r!}\eta_K^{(r)} + (A_K^\top)^\dagger\rho(\tau), \qquad\rho(\tau) := n_K(\tau) - A_K^\top\left(p_K + \sum_{r=1}^{\bar{s}} \frac{\tau^r}{r!}\eta_K^{(r)}\right), \tag*{(64)} w ( τ ) := p K + r = 1 ∑ s ˉ r ! τ r η K ( r ) + ( A K ⊤ ) † ρ ( τ ) , ρ ( τ ) := n K ( τ ) − A K ⊤ ( p K + r = 1 ∑ s ˉ r ! τ r η K ( r ) ) , (64)
and the dagger denotes the pseudo-inverse. The vector ρ ( τ ) = ∑ r > s ˉ ( τ r / r ! ) ( d r n K / d τ r ) ( 0 ) \rho(\tau) = \sum_{r>\bar{s}}(\tau^r/r!)(\mathrm{d}^r n_K/\mathrm{d}\tau^r)(0) ρ ( τ ) = ∑ r > s ˉ ( τ r / r !) ( d r n K / d τ r ) ( 0 ) lies in Im ( A K ⊤ ) \operatorname{Im}(A_K^\top) Im ( A K ⊤ ) and satisfies ∥ ρ ( τ ) ∥ 2 ≤ ∥ d ∥ 2 ∑ r > s ˉ ( λ max τ ) r / r ! = O ( τ s ˉ + 1 ) \|\rho(\tau)\|_2 \le\|d\|_2 \sum_{r>\bar{s}}(\lambda_{\max}\tau)^r/r! = O(\tau^{\bar{s}+1}) ∥ ρ ( τ ) ∥ 2 ≤ ∥ d ∥ 2 ∑ r > s ˉ ( λ m a x τ ) r / r ! = O ( τ s ˉ + 1 ) . Since A K ⊤ ( A K ⊤ ) † A_K^\top(A_K^\top)^\dagger A K ⊤ ( A K ⊤ ) † is the orthogonal projection onto Im ( A K ⊤ ) \operatorname{Im}(A_K^\top) Im ( A K ⊤ ) , we have A K ⊤ w ( τ ) = n K ( τ ) A_K^\top w(\tau) = n_K(\tau) A K ⊤ w ( τ ) = n K ( τ ) .
We claim that w ( τ ) w(\tau) w ( τ ) has positive entries for all sufficiently small τ > 0 \tau> 0 τ > 0 . A constraint i ∈ J i \in J i ∈ J has w i ( 0 ) = p i > 0 w_i(0) = p_i > 0 w i ( 0 ) = p i > 0 . Let i i i be committed at stage r ≤ s ˉ r \le\bar{s} r ≤ s ˉ . Then p i = 0 p_i = 0 p i = 0 because i ∉ J i \notin J i ∈ / J , and η i ( j ) = 0 \eta_i^{(j)} = 0 η i ( j ) = 0 for 1 ≤ j ≤ r − 1 1 \le j \le r-1 1 ≤ j ≤ r − 1 by (62) , because i ∈ U j + 1 i \in U_{j+1} i ∈ U j + 1 . Since η i ( r ) > 0 \eta_i^{(r)} > 0 η i ( r ) > 0 , we obtain w i ( τ ) = ( τ r / r ! ) η i ( r ) + O ( τ r + 1 ) w_i(\tau) = (\tau^r/r!)\eta_i^{(r)} + O(\tau^{r+1}) w i ( τ ) = ( τ r / r !) η i ( r ) + O ( τ r + 1 ) . Every constraint of K K K lies in J J J or is committed at a stage r ≤ s ˉ r \le\bar{s} r ≤ s ˉ , which proves the claim. The constraints of K K K are active along y K y_K y K , so n K ( τ ) = A K ⊤ w ( τ ) n_K(\tau) = A_K^\top w(\tau) n K ( τ ) = A K ⊤ w ( τ ) lies in N C ( y K ( τ ) ) N_C(y_K(\tau)) N C ( y K ( τ )) for all sufficiently small τ > 0 \tau> 0 τ > 0 . With (63) , the definition (37) then gives τ J K ( x 0 ) > 0 \tau_J^K(x_0) > 0 τ J K ( x 0 ) > 0 .
Conclusion. Both times are positive, so τ ∗ K ( x 0 ) > 0 \tau_*^K(x_0) > 0 τ ∗ K ( x 0 ) > 0 . Let τ ∈ [ 0 , τ ∗ K ( x 0 ) ) \tau\in[0,\tau_*^K(x_0)) τ ∈ [ 0 , τ ∗ K ( x 0 )) . By (63) , y K ( τ ) ∈ C y_K(\tau) \in C y K ( τ ) ∈ C , and then (37) gives n K ( τ ) ∈ N C ( y K ( τ ) ) n_K(\tau) \in N_C(y_K(\tau)) n K ( τ ) ∈ N C ( y K ( τ )) . Step 1 shows that y K y_K y K is the slow solution on [ 0 , τ ∗ K ( x 0 ) ) [0,\tau_*^K(x_0)) [ 0 , τ ∗ K ( x 0 )) , which proves (38) . Since d ( x ( τ ) ) = x ˙ ( τ ) d(x(\tau)) = \dot{x}(\tau) d ( x ( τ )) = x ˙ ( τ ) by Theorem 2.1 (iii), the identity n K ( τ ) = A K ⊤ w ( τ ) n_K(\tau) = A_K^\top w(\tau) n K ( τ ) = A K ⊤ w ( τ ) is (39) for all sufficiently small τ > 0 \tau> 0 τ > 0 . This proves part (i).
Step 4. We prove part (ii). Write τ ∗ : = τ ∗ K ( x 0 ) < + ∞ \tau_* := \tau_*^K(x_0) < +\infty τ ∗ := τ ∗ K ( x 0 ) < + ∞ , x ∗ : = x ( τ ∗ ) = y K ( τ ∗ ) x_* := x(\tau_*) = y_K(\tau_*) x ∗ := x ( τ ∗ ) = y K ( τ ∗ ) , and v − : = − P K ⊥ Q τ ∗ d = y ˙ K ( τ ∗ ) v^- := -P_K^\perp Q_{\tau_*}d = \dot{y}_K(\tau_*) v − := − P K ⊥ Q τ ∗ d = y ˙ K ( τ ∗ ) , the left limit of x ˙ ( τ ) \dot{x}(\tau) x ˙ ( τ ) as τ → τ ∗ \tau\to\tau_* τ → τ ∗ . By the conclusion of Step 3 and (63) , n K ( τ ) n_K(\tau) n K ( τ ) lies in the closed cone cone { A i ⊤ : i ∈ K ∪ R } \operatorname{cone}\{A_i^\top: i \in K \cup R\} cone { A i ⊤ : i ∈ K ∪ R } for every τ ∈ ( 0 , τ ∗ ) \tau\in(0,\tau_*) τ ∈ ( 0 , τ ∗ ) , hence also at τ ∗ \tau_* τ ∗ . Some p ^ ≥ 0 \hat{p} \ge0 p ^ ≥ 0 supported on K ∪ R K \cup R K ∪ R therefore satisfies A ⊤ p ^ = n K ( τ ∗ ) A^\top\hat{p} = n_K(\tau_*) A ⊤ p ^ = n K ( τ ∗ ) . The constraints of K ∪ R K \cup R K ∪ R are active at x ∗ x_* x ∗ by continuity, so p ^ \hat{p} p ^ is feasible for the NNLS problem (7) at x ∗ x_* x ∗ , and its residual is − A ⊤ p ^ − Q x ∗ − c = v − -A^\top\hat{p}-Qx_*-c = v^- − A ⊤ p ^ − Q x ∗ − c = v − . Let I + I^+ I + denote the constraints outside K ∪ R K \cup R K ∪ R active at x ∗ x_* x ∗ , so that E ( x ∗ ) = K ∪ R ∪ I + \mathcal{E}(x_*) = K \cup R \cup I^+ E ( x ∗ ) = K ∪ R ∪ I + . In the KKT test of Step 2 of the proof of Proposition 3.3 , the sign condition holds with equality over K K K , because the first identity of Step 1 differentiates to [ A v − ] K = 0 [Av^-]_K = 0 [ A v − ] K = 0 , and over R R R , because the values there vanish identically. This equality settles complementarity, since p ^ \hat{p} p ^ is supported on K ∪ R K \cup R K ∪ R . The values of the constraints of I + I^+ I + are negative on ( 0 , τ ∗ ) (0,\tau_*) ( 0 , τ ∗ ) and vanish at τ ∗ \tau_* τ ∗ , so [ A v − ] I + ≥ 0 [Av^-]_{I^+} \ge0 [ A v − ] I + ≥ 0 . Hence p ^ \hat{p} p ^ is an optimal NNLS solution at x ∗ x_* x ∗ if and only if [ A v − ] I + = 0 [Av^-]_{I^+} = 0 [ A v − ] I + = 0 , that is, if and only if the hypothesis of part (ii) fails. Steps 3a and 3b of that proof then apply, and part (ii) follows.
Step 5. We prove part (iii). Suppose that the rows of A J ∪ G A_{J\cup G} A J ∪ G are linearly independent, and let K ′ K' K ′ be a set with J ⊂ K ′ ⊂ J ∪ G J \subset K' \subset J \cup G J ⊂ K ′ ⊂ J ∪ G for which part (i) holds. By part (i), x ( τ ) = y K ′ ( τ ) x(\tau) = y_{K'}(\tau) x ( τ ) = y K ′ ( τ ) for τ ∈ [ 0 , τ ∗ K ′ ( x 0 ) ) \tau\in[0,\tau_*^{K'}(x_0)) τ ∈ [ 0 , τ ∗ K ′ ( x 0 )) , so the constraints of K ′ K' K ′ are active at x ( τ ) x(\tau) x ( τ ) . By Step 1, the constraints outside J ∪ G J \cup G J ∪ G are inactive at x ( τ ) x(\tau) x ( τ ) for all sufficiently small τ > 0 \tau> 0 τ > 0 . For such τ \tau τ , the rows of A E ( x ( τ ) ) A_{\mathcal{E}(x(\tau))} A E ( x ( τ )) are linearly independent, and part (i) gives w w w with positive entries such that − Q x ( τ ) − c − d ( x ( τ ) ) = A K ′ ⊤ w -Qx(\tau)-c-d(x(\tau)) = A_{K'}^\top w − Q x ( τ ) − c − d ( x ( τ )) = A K ′ ⊤ w . The vector p ′ p' p ′ equal to w w w on K ′ K' K ′ and to zero elsewhere is therefore feasible for the NNLS problem (7) at x ( τ ) x(\tau) x ( τ ) , and its residual d ( x ( τ ) ) d(x(\tau)) d ( x ( τ )) attains the optimal value, so p ′ p' p ′ is an optimal NNLS solution. Every optimal NNLS solution p ′ ′ p'' p ′′ satisfies A ⊤ p ′ ′ = n ( x ( τ ) ) = A ⊤ p ′ A^\top p'' = n(x(\tau)) = A^\top p' A ⊤ p ′′ = n ( x ( τ )) = A ⊤ p ′ and is supported on E ( x ( τ ) ) \mathcal{E}(x(\tau)) E ( x ( τ )) , whose rows are linearly independent, so p ′ ′ = p ′ p'' = p' p ′′ = p ′ . Hence J ( x ( τ ) ) = K ′ J(x(\tau)) = K' J ( x ( τ )) = K ′ for all sufficiently small τ > 0 \tau> 0 τ > 0 . The same holds for K ′ = K K'=K K ′ = K , so K ′ = K K'=K K ′ = K .
Proof of Theorem 4.1
Throughout, let τ ↦ x ( τ ) \tau\mapsto x(\tau) τ ↦ x ( τ ) denote the slow solution with x ( 0 ) = x ( 0 ) x(0) = x^{(0)} x ( 0 ) = x ( 0 ) , given piecewise by (42) , and let λ max \lambda_{\max} λ m a x denote the largest eigenvalue of Q Q Q . Theorem 4.1 claims the breakpoints are finitely many, the direction d ( x ( τ ) ) \mathbf{d}(x(\tau)) d ( x ( τ )) converges to a limit d s \mathbf{d}^{s} d s , and x ( τ ) x(\tau) x ( τ ) converges to an optimal solution if d s = 0 \mathbf{d}^{s}=0 d s = 0 , while the problem is unbounded below otherwise. The main difficulty is the first claim. Theorem 3.2 gives every piece a positive length but no lower bound on it, so infinitely many breakpoints could accumulate at a finite time or tend to infinity. We exclude both cases by contradiction. We first collect tools in Appendix B.2.1, among them three lemmas on trajectories whose pieces accumulate at a finite time. Appendix B.2.2 shows that every breakpoint is a zero of an affine event value , and that no breakpoint occurs once every event value either vanishes identically or has no zero. Appendix B.2.3 identifies the limit d s \mathbf{d}^{s} d s and proves the last claim of the theorem. Appendix B.2.4 excludes accumulation at a finite time by expanding the deviation of the trajectory from an affine path in powers of the remaining time. Appendix B.2.5 excludes accumulation at infinity by expanding the deviation of the shifted trajectory x ( τ ) − τ d s x(\tau)-\tau\mathbf{d}^{s} x ( τ ) − τ d s from its limit along eigenvectors. Appendix B.2.6 assembles the proof and derives the two formulas from the final piece.
Given J ⊆ [ m ] J\subseteq[m] J ⊆ [ m ] , let P J ⊥ P_{J}^{\perp} P J ⊥ denote the orthogonal projection onto ker A J \ker A_{J} ker A J and define
S J : = P J ⊥ Q P J ⊥ . (65) S_{J}:=P_{J}^{\perp}QP_{J}^{\perp}. \tag*{(65)} S J := P J ⊥ Q P J ⊥ . (65)
The matrix S J S_{J} S J is SPSD, with ∥ S J ∥ 2 ≤ λ max \lVert S_{J}\rVert_{2}\leq\lambda_{\max} ∥ S J ∥ 2 ≤ λ m a x and Im ( S J ) ⊆ ker A J \operatorname{Im}(S_{J})\subseteq\ker A_{J} Im ( S J ) ⊆ ker A J , and S J z = P J ⊥ Q z S_{J}z=P_{J}^{\perp}Qz S J z = P J ⊥ Q z for every z ∈ ker A J z\in\ker A_{J} z ∈ ker A J . When the rows of A J A_{J} A J are linearly independent, we write
p J ( x ) : = − ( A J A J T ) − 1 A J ∇ x f ( x ) (66) p_{J}(x):=-\left(A_{J}A_{J}^{\mathsf{T}}\right)^{-1}A_{J}\nabla_{x}f(x) \tag*{(66)} p J ( x ) := − ( A J A J T ) − 1 A J ∇ x f ( x ) (66)
for the least-squares multiplier of (10) , so that − ∇ x f ( x ) = − P J ⊥ ∇ x f ( x ) + A J T p J ( x ) -\nabla_{x}f(x)=-P_{J}^{\perp}\nabla_{x}f(x)+A_{J}^{\mathsf{T}}p_{J}(x) − ∇ x f ( x ) = − P J ⊥ ∇ x f ( x ) + A J T p J ( x ) . The next lemma extends formula (12) to cones larger than the tangent cone.
Lemma B.2. Let J J J be an index set with linearly independent rows, let K : = { s ∈ R n : A J s ≤ 0 } \mathcal{K}:=\{s\in\mathbb{R}^{n}:A_{J}s\leq0\} K := { s ∈ R n : A J s ≤ 0 } , and let x ∈ R n x\in\mathbb{R}^{n} x ∈ R n satisfy p J ( x ) ≥ 0 p_{J}(x)\geq0 p J ( x ) ≥ 0 . Then − P J ⊥ ∇ x f ( x ) = proj K ( − ∇ x f ( x ) ) -P_{J}^{\perp}\nabla_{x}f(x)=\operatorname{proj}_{\mathcal{K}}(-\nabla_{x}f(x)) − P J ⊥ ∇ x f ( x ) = proj K ( − ∇ x f ( x )) . In particular, d ( x ) = proj K ( − ∇ x f ( x ) ) d(x)=\operatorname{proj}_{\mathcal{K}}(-\nabla_{x}f(x)) d ( x ) = proj K ( − ∇ x f ( x )) for every x ∈ C x\in C x ∈ C and J = J ( x ) J=J(x) J = J ( x ) , with p ( x ) p(x) p ( x ) chosen so that (LI) holds.
Proof. The vector − P J ⊥ ∇ x f ( x ) -P_{J}^{\perp}\nabla_{x}f(x) − P J ⊥ ∇ x f ( x ) lies in ker A J ⊆ K \ker A_{J}\subseteq\mathcal{K} ker A J ⊆ K , the residual A J T p J ( x ) A_{J}^{\mathsf{T}}p_{J}(x) A J T p J ( x ) is a nonnegative combination of the rows of J J J and so lies in the polar cone of K \mathcal{K} K , and the two are orthogonal. These three properties characterize the projection onto K \mathcal{K} K . For J = J ( x ) J=J(x) J = J ( x ) , formula (10) shows that p J ( x ) p_{J}(x) p J ( x ) is the restriction of p ( x ) p(x) p ( x ) to J J J , which is positive, and formula (12) gives d ( x ) = − P J ⊥ ∇ x f ( x ) d(x)=-P_{J}^{\perp}\nabla_{x}f(x) d ( x ) = − P J ⊥ ∇ x f ( x ) .
The next two lemmas reduce membership in a cone generated by possibly dependent rows to sign conditions on the maps (60) and recall a basic inequality about closed convex cones.
Lemma B.3. Let E ⊆ [ m ] E\subseteq[m] E ⊆ [ m ] and v ∈ Im ( A E T ) v\in\operatorname{Im}(A_{E}^{\mathsf{T}}) v ∈ Im ( A E T ) . Then v ∈ cone { A i T : i ∈ E } v\in\operatorname{cone}\{A_{i}^{\mathsf{T}}:i\in E\} v ∈ cone { A i T : i ∈ E } if and only if some B ⊆ E B\subseteq E B ⊆ E such that the rows of A B A_{B} A B form a basis of the row space of A E A_{E} A E satisfies ( A B A B T ) − 1 A B v ≥ 0 \left(A_{B}A_{B}^{\mathsf{T}}\right)^{-1}A_{B}v\geq0 ( A B A B T ) − 1 A B v ≥ 0 .
Proof. Suppose that v v v lies in the cone. By Carathéodory’s theorem [36 Corollary 17.1.2] , v = A L T u v=A_{L}^{\mathsf{T}}u v = A L T u for some L ⊆ E L\subseteq E L ⊆ E with linearly independent rows and some u ≥ 0 u\geq0 u ≥ 0 . Extend L L L to a set B ⊆ E B\subseteq E B ⊆ E whose rows form a basis of the row space of A E A_{E} A E , and extend u u u by zero to B B B . Then v = A B T u v=A_{B}^{\mathsf{T}}u v = A B T u , and multiplying by ( A B A B T ) − 1 A B \left(A_{B}A_{B}^{\mathsf{T}}\right)^{-1}A_{B} ( A B A B T ) − 1 A B gives ( A B A B T ) − 1 A B v = u ≥ 0 \left(A_{B}A_{B}^{\mathsf{T}}\right)^{-1}A_{B}v=u\geq0 ( A B A B T ) − 1 A B v = u ≥ 0 . Conversely, Im ( A B T ) = Im ( A E T ) \operatorname{Im}(A_{B}^{\mathsf{T}})=\operatorname{Im}(A_{E}^{\mathsf{T}}) Im ( A B T ) = Im ( A E T ) contains v v v , and A B T ( A B A B T ) − 1 A B A_{B}^{\mathsf{T}}\left(A_{B}A_{B}^{\mathsf{T}}\right)^{-1}A_{B} A B T ( A B A B T ) − 1 A B is the orthogonal projection onto it. Hence v = A B T ( A B A B T ) − 1 A B v v=A_{B}^{\mathsf{T}}\left(A_{B}A_{B}^{\mathsf{T}}\right)^{-1}A_{B}v v = A B T ( A B A B T ) − 1 A B v , a nonnegative combination of rows of A E A_{E} A E .
Lemma B.4. Let K 2 ⊆ K 1 K_2 \subseteq K_1 K 2 ⊆ K 1 be closed convex cones, let z ∈ R n z \in\mathbb{R}^n z ∈ R n , and let z r : = proj K r ( z ) z_r := \operatorname{proj}_{K_r}(z) z r := proj K r ( z ) for r = 1 , 2 r = 1,2 r = 1 , 2 . Then ∥ z 1 − z 2 ∥ 2 2 ≤ ∥ z 1 ∥ 2 2 − ∥ z 2 ∥ 2 2 \lVert z_1-z_2\rVert_2^2 \le\lVert z_1\rVert_2^2-\lVert z_2\rVert_2^2 ∥ z 1 − z 2 ∥ 2 2 ≤ ∥ z 1 ∥ 2 2 − ∥ z 2 ∥ 2 2 .
Proof. Expanding the squares shows that the claim is equivalent to ⟨ z 1 − z 2 , z 2 ⟩ ≥ 0 \langle z_1-z_2,z_2\rangle\ge0 ⟨ z 1 − z 2 , z 2 ⟩ ≥ 0 . The projection onto K 2 K_2 K 2 satisfies ⟨ z − z 2 , z 2 ⟩ = 0 \langle z-z_2,z_2\rangle=0 ⟨ z − z 2 , z 2 ⟩ = 0 , so ⟨ z 1 − z 2 , z 2 ⟩ = ⟨ z 1 − z , z 2 ⟩ \langle z_1-z_2,z_2\rangle=\langle z_1-z,z_2\rangle ⟨ z 1 − z 2 , z 2 ⟩ = ⟨ z 1 − z , z 2 ⟩ , and the projection onto K 1 K_1 K 1 satisfies ⟨ z − z 1 , s ⟩ ≤ 0 \langle z-z_1,s\rangle\le0 ⟨ z − z 1 , s ⟩ ≤ 0 for every s ∈ K 1 s\in K_1 s ∈ K 1 , in particular for s = z 2 s=z_2 s = z 2 .
The next three lemmas prove control results about a trajectory whose pieces accumulate at a finite time, which will be applied in Appendix § 11 .§ 2.4 to the slow solution. Let τ ˉ < + ∞ \bar{\tau}<+\infty τ ˉ < + ∞ , let τ ( 0 ) < τ ( 1 ) < ⋯ \tau^{(0)}<\tau^{(1)}<\cdots τ ( 0 ) < τ ( 1 ) < ⋯ increase to τ ˉ \bar{\tau} τ ˉ , and call the intervals [ τ ( k ) , τ ( k + 1 ) ) [\tau^{(k)},\tau^{(k+1)}) [ τ ( k ) , τ ( k + 1 ) ) pieces . Let J \mathcal{J} J be a finite family of index sets, and let J ( ⋅ ) J(\cdot) J ( ⋅ ) be constant on each piece with values in J \mathcal{J} J . A piece on which J ( ⋅ ) = J J(\cdot)=J J ( ⋅ ) = J is a J J J -piece. Every J ∈ J J\in\mathcal{J} J ∈ J is assumed to occur on infinitely many pieces. We write σ ( τ ) : = τ ˉ − τ \sigma(\tau):=\bar{\tau}-\tau σ ( τ ) := τ ˉ − τ for the remaining time.
Lemma B.5. In this setting, let s s s be positive and nonincreasing on [ τ ( 0 ) , τ ˉ ) [\tau^{(0)},\bar{\tau}) [ τ ( 0 ) , τ ˉ ) , and put Λ ( τ ) : = ∫ τ τ ˉ s ( ρ ) d ρ \Lambda(\tau):=\int_\tau^{\bar{\tau}}s(\rho)\,d\rho Λ ( τ ) := ∫ τ τ ˉ s ( ρ ) d ρ , so that Λ ≤ σ s \Lambda\le\sigma s Λ ≤ σ s . Let h ∈ R n h\in\mathbb{R}^n h ∈ R n , and for J ∈ J J\in\mathcal{J} J ∈ J let u J : = P J ⊥ h u_J:=P_J^\perp h u J := P J ⊥ h be the drift of J J J and q J : = h − u J ∈ Im ( A J ⊤ ) q_J:=h-u_J\in\operatorname{Im}(A_J^\top) q J := h − u J ∈ Im ( A J ⊤ ) ; let u max : = max J ∈ J ∥ u J ∥ 2 u_{\max}:=\max_{J\in\mathcal{J}}\lVert u_J\rVert_2 u m a x := max J ∈ J ∥ u J ∥ 2 . Let ω \omega ω be bounded and measurable on [ τ ( 0 ) , τ ˉ ) [\tau^{(0)},\bar{\tau}) [ τ ( 0 ) , τ ˉ ) and satisfy, for every τ \tau τ there,
ω ( τ ) = − ∫ τ τ ˉ ( s ( ρ ) u J ( ρ ) − S J ( ρ ) ω ( ρ ) ) d ρ and A J ( τ ) ω ( τ ) = 0. (67) \omega(\tau)=-\int_\tau^{\bar{\tau}}\left(s(\rho)u_{J(\rho)}-S_{J(\rho)}\omega(\rho)\right)\,d\rho
\qquad\text{and}\qquad
A_{J(\tau)}\omega(\tau)=0. \tag*{(67)} ω ( τ ) = − ∫ τ τ ˉ ( s ( ρ ) u J ( ρ ) − S J ( ρ ) ω ( ρ ) ) d ρ and A J ( τ ) ω ( τ ) = 0. (67) Then:
(i) ∥ ω ( τ ) ∥ 2 ≤ 2 u max Λ ( τ ) \lVert\omega(\tau)\rVert_2\le2u_{\max}\Lambda(\tau) ∥ ω ( τ ) ∥ 2 ≤ 2 u m a x Λ ( τ ) whenever λ max σ ( τ ) ≤ 1 2 \lambda_{\max}\sigma(\tau)\le\frac{1}{2} λ m a x σ ( τ ) ≤ 2 1 ;
(ii) *the drifts u J u_J u J , J ∈ J J\in\mathcal{J} J ∈ J , all equal one vector, which lies in ker A J \ker A_J ker A J for every J ∈ J J\in\mathcal{J} J ∈ J .
Proof. Let Ω ( τ ) : = sup ρ ∈ [ τ , τ ˉ ) ∥ ω ( ρ ) ∥ 2 \Omega(\tau):=\sup_{\rho\in[\tau,\bar{\tau})}\lVert\omega(\rho)\rVert_2 Ω ( τ ) := sup ρ ∈ [ τ , τ ˉ ) ∥ ω ( ρ ) ∥ 2 , finite because ω \omega ω is bounded. For ρ ≥ τ \rho\ge\tau ρ ≥ τ , the first identity of (61) gives
∥ ω ( ρ ) ∥ 2 ≤ u max Λ ( ρ ) + λ max σ ( ρ ) Ω ( ρ ) ≤ u max Λ ( τ ) + 1 2 Ω ( τ ) , \lVert\omega(\rho)\rVert_2\le u_{\max}\Lambda(\rho)+\lambda_{\max}\sigma(\rho)\Omega(\rho)\le u_{\max}\Lambda(\tau)+\frac{1}{2}\Omega(\tau), ∥ ω ( ρ ) ∥ 2 ≤ u m a x Λ ( ρ ) + λ m a x σ ( ρ ) Ω ( ρ ) ≤ u m a x Λ ( τ ) + 2 1 Ω ( τ ) ,
because Λ \Lambda Λ and σ \sigma σ are nonincreasing and λ max σ ( τ ) ≤ 1 2 \lambda_{\max}\sigma(\tau)\le\frac{1}{2} λ m a x σ ( τ ) ≤ 2 1 . Taking the supremum over ρ \rho ρ gives Ω ( τ ) ≤ u max Λ ( τ ) + 1 2 Ω ( τ ) \Omega(\tau)\le u_{\max}\Lambda(\tau)+\frac{1}{2}\Omega(\tau) Ω ( τ ) ≤ u m a x Λ ( τ ) + 2 1 Ω ( τ ) . This proves (i).
Now, choose J ~ ∈ J \widetilde{J}\in\mathcal{J} J ∈ J minimizing ∥ u J ∥ 2 \lVert u_J\rVert_2 ∥ u J ∥ 2 and write u : = u J ~ u:=u_{\widetilde{J}} u := u J , q : = q J ~ q:=q_{\widetilde{J}} q := q J . For J ′ ∈ J J'\in\mathcal{J} J ′ ∈ J , since P J ′ ⊥ P_{J'}^\perp P J ′ ⊥ is symmetric and idempotent,
q ⊤ u J ′ = ⟨ u J ′ , u J ′ − u ⟩ = 1 2 ∥ u J ′ − u ∥ 2 2 + 1 2 ( ∥ u J ′ ∥ 2 2 − ∥ u ∥ 2 2 ) ≥ 1 2 ∥ u J ′ − u ∥ 2 2 , (68) q^\top u_{J'}=\langle u_{J'},u_{J'}-u\rangle=\frac{1}{2}\lVert u_{J'}-u\rVert_2^2+\frac{1}{2}\left(\lVert u_{J'}\rVert_2^2-\lVert u\rVert_2^2\right)\ge\frac{1}{2}\lVert u_{J'}-u\rVert_2^2, \tag*{(68)} q ⊤ u J ′ = ⟨ u J ′ , u J ′ − u ⟩ = 2 1 ∥ u J ′ − u ∥ 2 2 + 2 1 ( ∥ u J ′ ∥ 2 2 − ∥ u ∥ 2 2 ) ≥ 2 1 ∥ u J ′ − u ∥ 2 2 , (68)
and if u J ′ = u u_{J'}=u u J ′ = u , then u ∈ Im ( P J ′ ⊥ ) u\in\operatorname{Im}(P_{J'}^\perp) u ∈ Im ( P J ′ ⊥ ) and P J ′ ⊥ q = u J ′ − P J ′ ⊥ u = 0 P_{J'}^\perp q=u_{J'}-P_{J'}^\perp u=0 P J ′ ⊥ q = u J ′ − P J ′ ⊥ u = 0 , hence
q ⊤ S J ′ = ( P J ′ ⊥ q ) ⊤ Q P J ′ ⊥ = 0 ⊤ . (69) q^\top S_{J'}=(P_{J'}^\perp q)^\top QP_{J'}^\perp=0^\top. \tag*{(69)} q ⊤ S J ′ = ( P J ′ ⊥ q ) ⊤ Q P J ′ ⊥ = 0 ⊤ . (69)
Suppose that some J ′ ∈ J J'\in\mathcal{J} J ′ ∈ J has u J ′ ≠ u u_{J'}\ne u u J ′ = u , so that β = 1 2 min { ∥ u J ′ − u ∥ 2 2 : u J ′ ≠ u } > 0 \beta=\frac{1}{2}\min\{\lVert u_{J'}-u\rVert_2^2:u_{J'}\ne u\}>0 β = 2 1 min {∥ u J ′ − u ∥ 2 2 : u J ′ = u } > 0 , the family J \mathcal{J} J being finite. Let τ ( k ) \tau^{(k)} τ ( k ) be the left endpoint of a J ~ \widetilde{J} J -piece with λ max σ ( τ ( k ) ) ≤ 1 2 \lambda_{\max}\sigma(\tau^{(k)})\le\frac{1}{2} λ m a x σ ( τ ( k ) ) ≤ 2 1 , and let Θ \Theta Θ be the set of times ρ ≥ τ ( k ) \rho\ge\tau^{(k)} ρ ≥ τ ( k ) with u J ( ρ ) ≠ u u_{J(\rho)}\ne u u J ( ρ ) = u . Then q ⊤ ω ( τ ( k ) ) = 0 q^\top\omega(\tau^{(k)})=0 q ⊤ ω ( τ ( k ) ) = 0 , by the second identity of (61) and q ∈ Im ( A J ~ ⊤ ) q\in\operatorname{Im}(A_{\widetilde{J}}^\top) q ∈ Im ( A J ⊤ ) . We test the first identity at τ = τ ( k ) \tau=\tau^{(k)} τ = τ ( k ) with q q q . A time ρ ∉ Θ \rho\notin\Theta ρ ∈ / Θ contributes nothing, by (62) and (63). At a time ρ ∈ Θ \rho\in\Theta ρ ∈ Θ , the first term of the integrand contributes at least β s ( ρ ) \beta s(\rho) β s ( ρ ) , by (62), and the second at most 2 λ max u max ∥ q ∥ 2 σ ( τ ( k ) ) s ( ρ ) 2\lambda_{\max}u_{\max}\lVert q\rVert_2\sigma(\tau^{(k)})s(\rho) 2 λ m a x u m a x ∥ q ∥ 2 σ ( τ ( k ) ) s ( ρ ) in absolute value, by (i) and Λ ( ρ ) ≤ σ ( ρ ) s ( ρ ) ≤ σ ( τ ( k ) ) s ( ρ ) \Lambda(\rho)\le\sigma(\rho)s(\rho)\le\sigma(\tau^{(k)})s(\rho) Λ ( ρ ) ≤ σ ( ρ ) s ( ρ ) ≤ σ ( τ ( k ) ) s ( ρ ) . Hence
( β − 2 λ max u max ∥ q ∥ 2 σ ( τ ( k ) ) ) ∫ Θ s ( ρ ) d ρ ≤ 0. \left(\beta-2\lambda_{\max}u_{\max}\lVert q\rVert_2\sigma(\tau^{(k)})\right)\int_\Theta s(\rho)\,d\rho\le0. ( β − 2 λ m a x u m a x ∥ q ∥ 2 σ ( τ ( k ) ) ) ∫ Θ s ( ρ ) d ρ ≤ 0.
Since J ~ \widetilde{J} J occurs on infinitely many pieces and σ ( τ ( k ) ) → 0 \sigma(\tau^{(k)})\to0 σ ( τ ( k ) ) → 0 , some such τ ( k ) \tau^{(k)} τ ( k ) makes the factor in parentheses positive, and then the integral vanishes. The set Θ \Theta Θ is a union of pieces, each of positive length, and s > 0 s>0 s > 0 , so Θ \Theta Θ is empty. But every J ′ ∈ J J'\in\mathcal{J} J ′ ∈ J has pieces after τ ( k ) \tau^{(k)} τ ( k ) , a contradiction. Hence u J ′ = u u_{J'}=u u J ′ = u for every J ′ ∈ J J'\in\mathcal{J} J ′ ∈ J , and u = P J ~ ⊥ h ∈ ker A J ′ u=P_{\widetilde{J}}^\perp h\in\ker A_{J'} u = P J ⊥ h ∈ ker A J ′ .
Lemma B.6. Let v ˉ ∈ ker A J \bar{v} \in\ker A_J v ˉ ∈ ker A J for every J ∈ J J \in\mathcal{J} J ∈ J , and let ζ \zeta ζ be bounded and measurable and satisfy (61) with s = σ s=\sigma s = σ and h = Q v ˉ h=Q\bar{v} h = Q v ˉ . Then there exist vectors u [ 0 ] : = v ˉ , u [ 1 ] , u [ 2 ] , … \mathbf{u}^{[0]}:=\bar{v},\mathbf{u}^{[1]},\mathbf{u}^{[2]},\ldots u [ 0 ] := v ˉ , u [ 1 ] , u [ 2 ] , … such that S J u [ j − 1 ] = u [ j ] ∈ ker A J S_J\mathbf{u}^{[j-1]}=\mathbf{u}^{[j]}\in\ker A_J S J u [ j − 1 ] = u [ j ] ∈ ker A J for every J ∈ J J\in\mathcal{J} J ∈ J and every j ≥ 1 j\geq1 j ≥ 1 , and such that, for every r ≥ 0 r\geq0 r ≥ 0 , the function
ω r : = ζ + ∑ j = 1 r σ j + 1 ( j + 1 ) ! u [ j ] \omega_r:=\zeta+\sum_{j=1}^{r}\frac{\sigma^{j+1}}{(j+1)!}\mathbf{u}^{[j]} ω r := ζ + j = 1 ∑ r ( j + 1 )! σ j + 1 u [ j ] satisfies
∥ ω r ( τ ) ∥ 2 ≤ 2 ∥ u [ r + 1 ] ∥ 2 σ ( τ ) r + 2 ( r + 2 ) ! whenever λ max σ ( τ ) ≤ 1 2 . (70) \|\omega_r(\tau)\|_2\leq2\|\mathbf{u}^{[r+1]}\|_2\frac{\sigma(\tau)^{r+2}}{(r+2)!}\qquad\text{whenever }\lambda_{\max}\sigma(\tau)\leq\frac{1}{2}. \tag*{(70)} ∥ ω r ( τ ) ∥ 2 ≤ 2∥ u [ r + 1 ] ∥ 2 ( r + 2 )! σ ( τ ) r + 2 whenever λ m a x σ ( τ ) ≤ 2 1 . (70)
Proof. We show by induction on r ≥ 0 r\geq0 r ≥ 0 that u [ 1 ] , … , u [ r ] \mathbf{u}^{[1]},\ldots,\mathbf{u}^{[r]} u [ 1 ] , … , u [ r ] exist with the stated relations and that ω r \omega_r ω r satisfies the hypotheses of Lemma B.5 with s = σ r + 1 / ( r + 1 ) ! s=\sigma^{r+1}/(r+1)! s = σ r + 1 / ( r + 1 )! and h = Q u [ r ] h=Q\mathbf{u}^{[r]} h = Q u [ r ] , whose drifts are P J Q u [ r ] = S J u [ r ] P_JQ\mathbf{u}^{[r]}=S_J\mathbf{u}^{[r]} P J Q u [ r ] = S J u [ r ] because u [ r ] ∈ ker A J \mathbf{u}^{[r]}\in\ker A_J u [ r ] ∈ ker A J . Part (ii) of that lemma then supplies the common vector u [ r + 1 ] : = S J u [ r ] ∈ ker A J \mathbf{u}^{[r+1]}:=S_J\mathbf{u}^{[r]}\in\ker A_J u [ r + 1 ] := S J u [ r ] ∈ ker A J , and part (i), with Λ = σ r + 2 / ( r + 2 ) ! \Lambda=\sigma^{r+2}/(r+2)! Λ = σ r + 2 / ( r + 2 )! and u max = ∥ u [ r + 1 ] ∥ 2 u_{\max}=\|\mathbf{u}^{[r+1]}\|_2 u m a x = ∥ u [ r + 1 ] ∥ 2 , gives (64).
For r = 0 r=0 r = 0 , the function ω 0 = ζ \omega_0=\zeta ω 0 = ζ satisfies these assumptions. Let r ≥ 1 r\geq1 r ≥ 1 . The function ω r = ω r − 1 + σ r + 1 ( r + 1 ) ! u [ r ] \omega_r=\omega_{r-1}+\frac{\sigma^{r+1}}{(r+1)!}\mathbf{u}^{[r]} ω r = ω r − 1 + ( r + 1 )! σ r + 1 u [ r ] is bounded, and A J ( τ ) ω r ( τ ) = 0 A_{J(\tau)}\omega_r(\tau)=0 A J ( τ ) ω r ( τ ) = 0 because u [ r ] ∈ ker A J ( τ ) \mathbf{u}^{[r]}\in\ker A_{J(\tau)} u [ r ] ∈ ker A J ( τ ) . Since
σ ( τ ) r + 1 ( r + 1 ) ! = ∫ τ τ ˉ σ ( ρ ) r r ! d ρ , \frac{\sigma(\tau)^{r+1}}{(r+1)!}=\int_{\tau}^{\bar{\tau}}\frac{\sigma(\rho)^r}{r!}\,d\rho, ( r + 1 )! σ ( τ ) r + 1 = ∫ τ τ ˉ r ! σ ( ρ ) r d ρ ,
the identity (61) for ω r − 1 \omega_{r-1} ω r − 1 , whose drifts are u [ r ] \mathbf{u}^{[r]} u [ r ] , gives
ω r ( τ ) = ∫ τ τ ˉ S J ( ρ ) ω r − 1 ( ρ ) d ρ = − ∫ τ τ ˉ ( σ ( ρ ) r + 1 ( r + 1 ) ! S J ( ρ ) u [ r ] − S J ( ρ ) ω r ( ρ ) ) d ρ , \omega_r(\tau)=\int_{\tau}^{\bar{\tau}}S_{J(\rho)}\omega_{r-1}(\rho)\,d\rho
=-\int_{\tau}^{\bar{\tau}}\left(\frac{\sigma(\rho)^{r+1}}{(r+1)!}S_{J(\rho)}\mathbf{u}^{[r]}-S_{J(\rho)}\omega_r(\rho)\right)\,d\rho, ω r ( τ ) = ∫ τ τ ˉ S J ( ρ ) ω r − 1 ( ρ ) d ρ = − ∫ τ τ ˉ ( ( r + 1 )! σ ( ρ ) r + 1 S J ( ρ ) u [ r ] − S J ( ρ ) ω r ( ρ ) ) d ρ ,
which is (61) for ω r \omega_r ω r with s = σ r + 1 / ( r + 1 ) ! s=\sigma^{r+1}/(r+1)! s = σ r + 1 / ( r + 1 )! and the drifts S J u [ r ] S_J\mathbf{u}^{[r]} S J u [ r ] .
:::
Lemma B.7. In the setting of Lemma B.6, every a ∈ R n \mathbf{a}\in\mathbb{R}^n a ∈ R n admits a time τ a < τ ˉ \tau_a<\bar{\tau} τ a < τ ˉ such that τ ↦ ⟨ a , ζ ( τ ) ⟩ \tau\mapsto\langle\mathbf{a},\zeta(\tau)\rangle τ ↦ ⟨ a , ζ ( τ )⟩ either vanishes at every τ ∈ [ τ a , τ ˉ ) \tau\in[\tau_a,\bar{\tau}) τ ∈ [ τ a , τ ˉ ) or at none.
Proof. Let u [ j ] \mathbf{u}^{[j]} u [ j ] and ω r \omega_r ω r be as in Lemma B.6, put α j : = ⟨ a , u [ j ] ⟩ \alpha_j:=\langle\mathbf{a},\mathbf{u}^{[j]}\rangle α j := ⟨ a , u [ j ] ⟩ for j ≥ 1 j\geq1 j ≥ 1 , and let τ ′ ≥ τ ( 0 ) \tau'\geq\tau^{(0)} τ ′ ≥ τ ( 0 ) satisfy λ max σ ( τ ′ ) ≤ 1 2 \lambda_{\max}\sigma(\tau')\leq\frac{1}{2} λ m a x σ ( τ ′ ) ≤ 2 1 , so that (64) holds on [ τ ′ , τ ˉ ) [\tau',\bar{\tau}) [ τ ′ , τ ˉ ) . Note that ∥ u [ r + 1 ] ∥ 2 ≤ λ max r + 1 ∥ v ˉ ∥ 2 \|\mathbf{u}^{[r+1]}\|_2\leq\lambda_{\max}^{r+1}\|\bar{v}\|_2 ∥ u [ r + 1 ] ∥ 2 ≤ λ m a x r + 1 ∥ v ˉ ∥ 2 .
If every α j \alpha_j α j vanishes, then ⟨ a , ζ ⟩ = ⟨ a , ω r ⟩ \langle\mathbf{a},\zeta\rangle=\langle\mathbf{a},\omega_r\rangle ⟨ a , ζ ⟩ = ⟨ a , ω r ⟩ for every r r r , and (64) gives
∣ ⟨ a , ζ ( τ ) ⟩ ∣ ≤ 2 ∥ a ∥ 2 ∥ v ˉ ∥ 2 σ ( τ ) ( λ max σ ( τ ) ) r + 1 ≤ 2 − r ∥ a ∥ 2 ∥ v ˉ ∥ 2 σ ( τ ) |\langle\mathbf{a},\zeta(\tau)\rangle|\leq2\|\mathbf{a}\|_2\|\bar{v}\|_2\sigma(\tau)(\lambda_{\max}\sigma(\tau))^{r+1}\leq2^{-r}\|\mathbf{a}\|_2\|\bar{v}\|_2\sigma(\tau) ∣ ⟨ a , ζ ( τ )⟩ ∣ ≤ 2∥ a ∥ 2 ∥ v ˉ ∥ 2 σ ( τ ) ( λ m a x σ ( τ ) ) r + 1 ≤ 2 − r ∥ a ∥ 2 ∥ v ˉ ∥ 2 σ ( τ ) for every τ ∈ [ τ ′ , τ ˉ ) \tau\in[\tau',\bar{\tau}) τ ∈ [ τ ′ , τ ˉ ) . Letting r → ∞ r\to\infty r → ∞ makes ⟨ a , ζ ⟩ \langle\mathbf{a},\zeta\rangle ⟨ a , ζ ⟩ vanish there, and we take τ a = τ ′ \tau_a=\tau' τ a = τ ′ . Otherwise, let r r r be minimal with α r ≠ 0 \alpha_r\neq0 α r = 0 . Then ⟨ a , ζ ⟩ = ⟨ a , ω r ⟩ − α r σ r + 1 / ( r + 1 ) ! \langle\mathbf{a},\zeta\rangle=\langle\mathbf{a},\omega_r\rangle-\alpha_r\sigma^{r+1}/(r+1)! ⟨ a , ζ ⟩ = ⟨ a , ω r ⟩ − α r σ r + 1 / ( r + 1 )! , and (64) gives
∣ ⟨ a , ζ ( τ ) ⟩ ∣ ≥ ( ∣ α r ∣ − 2 ∥ a ∥ 2 ∥ u [ r + 1 ] ∥ 2 σ ( τ ) ) σ ( τ ) r + 1 ( r + 1 ) ! , |\langle\mathbf{a},\zeta(\tau)\rangle|\geq\left(|\alpha_r|-2\|\mathbf{a}\|_2\|\mathbf{u}^{[r+1]}\|_2\sigma(\tau)\right)\frac{\sigma(\tau)^{r+1}}{(r+1)!}, ∣ ⟨ a , ζ ( τ )⟩ ∣ ≥ ( ∣ α r ∣ − 2∥ a ∥ 2 ∥ u [ r + 1 ] ∥ 2 σ ( τ ) ) ( r + 1 )! σ ( τ ) r + 1 , which is positive once 2 ∥ a ∥ 2 ∥ u [ r + 1 ] ∥ 2 σ ( τ ) < ∣ α r ∣ 2\|\mathbf{a}\|_2\|\mathbf{u}^{[r+1]}\|_2\sigma(\tau)<|\alpha_r| 2∥ a ∥ 2 ∥ u [ r + 1 ] ∥ 2 σ ( τ ) < ∣ α r ∣ , that is, on some interval [ τ a , τ ˉ ) [\tau_a,\bar{\tau}) [ τ a , τ ˉ ) with τ a ≥ τ ′ \tau_a\geq\tau' τ a ≥ τ ′ .
Pieces and events
We now turn to the breakpoints of the slow solution and first describe how its pieces end. As before, a piece of the slow solution with active set J J J is a J J J -piece. For the k k k -th piece, let R ( k ) R^{(k)} R ( k ) denote the riding set of Appendix B.1 at x ( k ) x^{(k)} x ( k ) , and let N ( k ) : = cone { A i ⊤ : i ∈ J ∗ ( k ) ∪ R ( k ) } N^{(k)}:=\operatorname{cone}\{A_i^\top:i\in J_*^{(k)}\cup R^{(k)}\} N ( k ) := cone { A i ⊤ : i ∈ J ∗ ( k ) ∪ R ( k ) } , which contains − ∇ x f ( x ( τ ) ) − d ( x ( τ ) ) -\nabla_x f(x(\tau))-d(x(\tau)) − ∇ x f ( x ( τ )) − d ( x ( τ )) on the open piece by Lemma B.8(i).
Lemma B.8 (Dynamics on a piece). Consider the k k k -th piece and let J = J ∗ ( k ) J=J_*^{(k)} J = J ∗ ( k ) .
(i) For every τ ∈ [ τ ( k ) , τ ( k + 1 ) ) \tau\in[\tau^{(k)},\tau^{(k+1)}) τ ∈ [ τ ( k ) , τ ( k + 1 ) ) , the constraints of J J J are active at x ( τ ) x(\tau) x ( τ ) , and
d ( x ( τ ) ) = − P J ⊥ ∇ x f ( x ( τ ) ) ∈ ker A J . (71) d(x(\tau))=-P_J^\perp\nabla_x f(x(\tau))\in\ker A_J. \tag*{(71)} d ( x ( τ )) = − P J ⊥ ∇ x f ( x ( τ )) ∈ ker A J . (71)
For every τ ∈ ( τ ( k ) , τ ( k + 1 ) ) \tau\in(\tau^{(k)},\tau^{(k+1)}) τ ∈ ( τ ( k ) , τ ( k + 1 ) ) , we have E ( x ( τ ) ) = J ∪ R ( k ) \mathcal{E}(x(\tau))=J\cup R^{(k)} E ( x ( τ )) = J ∪ R ( k ) , A J ∪ R ( k ) d ( x ( τ ) ) = 0 A_{J\cup R^{(k)}}d(x(\tau))=0 A J ∪ R ( k ) d ( x ( τ )) = 0 , and − ∇ x f ( x ( τ ) ) − d ( x ( τ ) ) ∈ N ( k ) -\nabla_x f(x(\tau))-d(x(\tau))\in N^{(k)} − ∇ x f ( x ( τ )) − d ( x ( τ )) ∈ N ( k ) . If τ ( k + 1 ) < + ∞ \tau^{(k+1)}<+\infty τ ( k + 1 ) < + ∞ , then the constraints of J ∪ R ( k ) J\cup R^{(k)} J ∪ R ( k ) are active at x ( k + 1 ) x^{(k+1)} x ( k + 1 ) , and d ( x ( τ ) ) → − P J ⊥ ∇ x f ( x ( k + 1 ) ) d(x(\tau))\to-P_J^\perp\nabla_xf(x^{(k+1)}) d ( x ( τ )) → − P J ⊥ ∇ x f ( x ( k + 1 ) ) as τ ↑ τ ( k + 1 ) \tau\uparrow\tau^{(k+1)} τ ↑ τ ( k + 1 ) . This limit lies in ker A J ∪ R ( k ) \ker A_{J\cup R^{(k)}} ker A J ∪ R ( k ) , and − ( I n × n − P J ⊥ ) ∇ x f ( x ( k + 1 ) ) ∈ N ( k ) -(I_{n\times n}-P_J^\perp)\nabla_xf(x^{(k+1)})\in N^{(k)} − ( I n × n − P J ⊥ ) ∇ x f ( x ( k + 1 ) ) ∈ N ( k ) .
(ii) If τ ( k + 1 ) < + ∞ \tau^{(k+1)}<+\infty τ ( k + 1 ) < + ∞ , then at least one of two events occurs at τ ( k + 1 ) \tau^{(k+1)} τ ( k + 1 ) . In an enter event, some i ∉ J i\notin J i ∈ / J satisfies [ A x ( τ ) ] i < b i [Ax(\tau)]_i<b_i [ A x ( τ ) ] i < b i for every τ ∈ ( τ ( k ) , τ ( k + 1 ) ) \tau\in(\tau^{(k)},\tau^{(k+1)}) τ ∈ ( τ ( k ) , τ ( k + 1 ) ) and [ A x ( k + 1 ) ] i = b i [Ax^{(k+1)}]_i=b_i [ A x ( k + 1 ) ] i = b i . In a leave event, some B ⊆ J ∪ R ( k ) B\subseteq J\cup R^{(k)} B ⊆ J ∪ R ( k ) such that the rows of A B A_B A B form a basis of the row space of A J ∪ R ( k ) A_{J\cup R^{(k)}} A J ∪ R ( k ) , and some i ∈ B i\in B i ∈ B , are such that τ ↦ [ p B ( x ( τ ) ) ] i \tau\mapsto[p_B(x(\tau))]_i τ ↦ [ p B ( x ( τ )) ] i does not vanish identically on the piece and [ p B ( x ( k + 1 ) ) ] i = 0 [p_B(x^{(k+1)})]_i=0 [ p B ( x ( k + 1 ) ) ] i = 0 .
Proof. Measure time from τ ( k ) \tau^{(k)} τ ( k ) , write d : = d ( k ) \mathbf{d}:=d^{(k)} d := d ( k ) , R : = R ( k ) R:=R^{(k)} R := R ( k ) , and y : = y J \mathbf{y}:=\mathbf{y}_J y := y J for the candidate of Appendix B.1 at x ( k ) x^{(k)} x ( k ) , and let n J \mathbf{n}_J n J be its normal part (60) .
The slow system is autonomous, so Theorem 3.2(i) applies at x ( k ) x^{(k)} x ( k ) with K = J K=J K = J , and x ( τ ) = y ( τ ) x(\tau)=y(\tau) x ( τ ) = y ( τ ) on the piece. Step 1 of Appendix B.1 shows that the constraints of J J J stay active and that y ˙ ( τ ) = − P J ⊥ ∇ x f ( y ( τ ) ) \dot{y}(\tau)=-P_J^\perp\nabla_xf(y(\tau)) y ˙ ( τ ) = − P J ⊥ ∇ x f ( y ( τ )) , which lies in ker A J \ker A_J ker A J . Since d ( x ( τ ) ) = x ˙ ( τ ) d(x(\tau))=\dot{x}(\tau) d ( x ( τ )) = x ˙ ( τ ) by Theorem 2.1 (iii), this gives (71) . By (63) , E ( x ( τ ) ) = J ∪ R \mathcal{E}(x(\tau))=J\cup R E ( x ( τ )) = J ∪ R on the open piece. The values of the constraints of R R R vanish identically along y y y , so A J ∪ R y ˙ ( τ ) = 0 A_{J\cup R}\dot{y}(\tau)=0 A J ∪ R y ˙ ( τ ) = 0 for every τ ≥ 0 \tau\geq0 τ ≥ 0 . By Moreau’s decomposition (6) and (3) , − ∇ x f ( x ( τ ) ) − d ( x ( τ ) ) -\nabla_xf(x(\tau))-d(x(\tau)) − ∇ x f ( x ( τ )) − d ( x ( τ )) lies in N C ( x ( τ ) ) N_C(x(\tau)) N C ( x ( τ )) , which equals N ( k ) N^{(k)} N ( k ) on the open piece. The claims at τ ( k + 1 ) \tau^{(k+1)} τ ( k + 1 ) follow by continuity, because x ↦ P J ⊥ ∇ x f ( x ) x\mapsto P_J^\perp\nabla_xf(x) x ↦ P J ⊥ ∇ x f ( x ) is affine and N ( k ) N^{(k)} N ( k ) is closed. This proves (i).
Suppose that τ ( k + 1 ) < + ∞ \tau^{(k+1)}<+\infty τ ( k + 1 ) < + ∞ , and abbreviate τ f e a s : = τ f e a s + ( x ( k ) ) \tau_{\mathrm{feas}}:=\tau_{\mathrm{feas}}^+(x^{(k)}) τ feas := τ feas + ( x ( k ) ) and τ J : = τ J + ( x ( k ) ) \tau_J:=\tau_J^+(x^{(k)}) τ J := τ J + ( x ( k ) ) , so that τ ∗ ( k ) = min ( τ f e a s , τ J ) \tau_*^{(k)}=\min(\tau_{\mathrm{feas}},\tau_J) τ ∗ ( k ) = min ( τ feas , τ J ) is finite, and positive by Theorem 3.2(i). Suppose first that τ ∗ ( k ) = τ f e a s \tau_*^{(k)}=\tau_{\mathrm{feas}} τ ∗ ( k ) = τ feas . The infimum (25) runs over the constraints i ∉ J i\notin J i ∈ / J whose values do not vanish identically on the piece (§ 3.5 ). The positive zeros of each such value form a set that is closed in ( 0 , + ∞ ) (0,+\infty) ( 0 , + ∞ ) , with infimum at least τ f e a s > 0 \tau_{\mathrm{feas}}>0 τ feas > 0 , so some i ∉ J i\notin J i ∈ / J has a zero at τ f e a s \tau_{\mathrm{feas}} τ feas and none in ( 0 , τ f e a s ) (0,\tau_{\mathrm{feas}}) ( 0 , τ feas ) . Its value is nonpositive because x ( τ ) ∈ C x(\tau)\in C x ( τ ) ∈ C , hence negative on ( 0 , τ f e a s ) (0,\tau_{\mathrm{feas}}) ( 0 , τ feas ) : an enter event occurs.
Suppose now that τ J < τ f e a s \tau_J<\tau_{\mathrm{feas}} τ J < τ feas . For every τ ∈ ( 0 , τ f e a s ) \tau\in(0,\tau_{\mathrm{feas}}) τ ∈ ( 0 , τ feas ) , (63) gives y ( τ ) ∈ C y(\tau)\in C y ( τ ) ∈ C and N C ( y ( τ ) ) = N ( k ) N_C(y(\tau))=N^{(k)} N C ( y ( τ )) = N ( k ) , and the definition (37) gives n J ( τ ) ∈ N ( k ) n_J(\tau)\in N^{(k)} n J ( τ ) ∈ N ( k ) for τ ∈ ( 0 , τ J ) \tau\in(0,\tau_J) τ ∈ ( 0 , τ J ) . The cone N ( k ) N^{(k)} N ( k ) is closed, so n J ( τ J ) ∈ N ( k ) n_J(\tau_J)\in N^{(k)} n J ( τ J ) ∈ N ( k ) . Hence τ J \tau_J τ J does not belong to the set in (37) , and there are times in ( τ J , τ f e a s ) (\tau_J,\tau_{\mathrm{feas}}) ( τ J , τ feas ) arbitrarily close to τ J \tau_J τ J at which n J ∉ N ( k ) n_J\notin N^{(k)} n J ∈ / N ( k ) . Let B ⊆ J ∪ R B\subseteq J\cup R B ⊆ J ∪ R be such that the rows of A B A_B A B form a basis of the row space of A J ∪ R A_{J\cup R} A J ∪ R . Since A B y ˙ ( τ ) = 0 A_B\dot{y}(\tau)=0 A B y ˙ ( τ ) = 0 and n J ( τ ) ∈ Im ( A J T ) ⊆ Im ( A B T ) n_J(\tau)\in\operatorname{Im}(A_J^\mathsf{T})\subseteq\operatorname{Im}(A_B^\mathsf{T}) n J ( τ ) ∈ Im ( A J T ) ⊆ Im ( A B T ) for every τ ≥ 0 \tau\geq0 τ ≥ 0 , the definitions (60) and (66) give
n J ( τ ) = A B T ( A B A B T ) − 1 A B n J ( τ ) = A B T p B ( y ( τ ) ) for every τ ≥ 0. (72) n_J(\tau)=A_B^\mathsf{T}(A_BA_B^\mathsf{T})^{-1}A_Bn_J(\tau)=A_B^\mathsf{T}p_B(y(\tau))\qquad\text{for every }\tau\geq0. \tag*{(72)} n J ( τ ) = A B T ( A B A B T ) − 1 A B n J ( τ ) = A B T p B ( y ( τ )) for every τ ≥ 0. (72)
By Lemma B.3 and n J ( τ J ) ∈ N ( k ) n_J(\tau_J)\in N^{(k)} n J ( τ J ) ∈ N ( k ) , some such B B B satisfies p B ( y ( τ J ) ) ≥ 0 p_B(y(\tau_J))\geq0 p B ( y ( τ J )) ≥ 0 . The function τ ↦ p B ( y ( τ ) ) \tau\mapsto p_B(y(\tau)) τ ↦ p B ( y ( τ )) is entire, because p B p_B p B is affine and y y y is entire. Suppose that every i ∈ B i\in B i ∈ B with [ p B ( y ( τ J ) ) ] i = 0 [p_B(y(\tau_J))]_i=0 [ p B ( y ( τ J )) ] i = 0 had [ p B ( y ( ⋅ ) ) ] i ≡ 0 [p_B(y(\cdot))]_i\equiv0 [ p B ( y ( ⋅ )) ] i ≡ 0 . The positive entries stay positive near τ J \tau_J τ J by continuity, so p B ( y ( τ ) ) ≥ 0 p_B(y(\tau))\geq0 p B ( y ( τ )) ≥ 0 for every τ \tau τ near τ J \tau_J τ J , and (72) would give n J ( τ ) ∈ N ( k ) n_J(\tau)\in N^{(k)} n J ( τ ) ∈ N ( k ) for every τ > τ J \tau>\tau_J τ > τ J near τ J \tau_J τ J , a contradiction. Hence some i ∈ B i\in B i ∈ B has [ p B ( y ( τ J ) ) ] i = 0 [p_B(y(\tau_J))]_i=0 [ p B ( y ( τ J )) ] i = 0 and [ p B ( y ( ⋅ ) ) ] i ≢ 0 [p_B(y(\cdot))]_i\not\equiv0 [ p B ( y ( ⋅ )) ] i ≡ 0 . An entire function that does not vanish identically does not vanish identically on any interval. Since x = y x=y x = y on the piece and x ( k + 1 ) = y ( τ J ) x^{(k+1)}=y(\tau_J) x ( k + 1 ) = y ( τ J ) , a leave event of ( B , i ) (B,i) ( B , i ) occurs. This proves (ii). □
Suppose there are infinitely many breakpoints. At each breakpoint τ ( k + 1 ) \tau^{(k+1)} τ ( k + 1 ) , we fix one event of Lemma B.8 (ii), either an enter event of a constraint i ∉ J ∗ ( k ) i\notin J_*^{(k)} i ∈ / J ∗ ( k ) or a leave event, which we label by the pair ( B , i ) (B,i) ( B , i ) of its set B B B and index i i i in Lemma B.8 (ii). The event value of an enter event of i i i is [ A x ] i − b i [Ax]_i-b_i [ A x ] i − b i , and the event value of a leave event of ( B , i ) (B,i) ( B , i ) is [ p B ( x ) ] i [p_B(x)]_i [ p B ( x ) ] i . Both are affine in x x x , the second by (66) , and vanish at x ( k + 1 ) x^{(k+1)} x ( k + 1 ) . An active set, a constraint of an enter event, or a pair of a leave event is recurring if it occurs on infinitely many pieces. To study late breakpoints, we discard the finitely many pieces with a nonrecurring object and call the remaining pieces the tail . Every object on the tail is recurring, even after we discard finitely many further pieces. Let Φ \Phi Φ be the finite set of gradients of recurring event values. Every breakpoint of the tail is therefore a zero of a recurring event value. The next lemma excludes late breakpoints when each recurring event value eventually vanishes identically or has no zero.
Lemma B.9. Suppose there are infinitely many breakpoints. Let τ ↦ D ( τ ) \tau\mapsto D(\tau) τ ↦ D ( τ ) satisfy ϕ ( x ( τ ) ) = ⟨ a , D ( τ ) ⟩ \phi(x(\tau)) = \langle a,D(\tau)\rangle ϕ ( x ( τ )) = ⟨ a , D ( τ )⟩ on the tail at every τ \tau τ , for every recurring event value ϕ \phi ϕ and its gradient a ∈ Φ a \in\Phi a ∈ Φ . Suppose that every a ∈ Φ a \in\Phi a ∈ Φ admits a time τ a \tau_a τ a on the tail after which τ ↦ ⟨ a , D ( τ ) ⟩ \tau\mapsto\langle a,D(\tau)\rangle τ ↦ ⟨ a , D ( τ )⟩ either vanishes identically or has no zero. Then no piece of the tail starts after the largest of the times τ a \tau_a τ a .
Proof. Let τ ′ \tau' τ ′ be the largest of the times τ a \tau_a τ a . Consider a piece [ τ ( k ) , τ ( k + 1 ) ) [\tau^{(k)},\tau^{(k+1)}) [ τ ( k ) , τ ( k + 1 ) ) of the tail with τ ( k ) > τ ′ \tau^{(k)} > \tau' τ ( k ) > τ ′ , with the event fixed at τ ( k + 1 ) \tau^{(k+1)} τ ( k + 1 ) . This event is recurring, so its gradient a a a lies in Φ \Phi Φ . At an enter event, Lemma B.8(ii) makes ⟨ a , D ⟩ \langle a,D\rangle ⟨ a , D ⟩ negative on ( τ ( k ) , τ ( k + 1 ) ) (\tau^{(k)},\tau^{(k+1)}) ( τ ( k ) , τ ( k + 1 ) ) and zero at τ ( k + 1 ) \tau^{(k+1)} τ ( k + 1 ) . At a leave event, it makes ⟨ a , D ⟩ \langle a,D\rangle ⟨ a , D ⟩ zero at τ ( k + 1 ) \tau^{(k+1)} τ ( k + 1 ) without vanishing identically on the piece. Both cases contradict the dichotomy, because the closed piece lies after τ a \tau_a τ a .
The limit direction
We now show that the limit of d ( x ( τ ) ) d(x(\tau)) d ( x ( τ )) in Theorem 4.1 is the projection d s d^s d s defined by
R : = { v ∈ R n : A v ≤ 0 , Q v = 0 } , d s : = proj R ( − c ) , I 0 : = { i ∈ [ m ] : [ A d s ] i = 0 } . (73) \mathcal{R} := \{v \in\mathbb{R}^n : Av \le0,\ Qv = 0\}, \qquad d^s := \operatorname{proj}_{\mathcal{R}}(-c), \qquad I_0 := \{i \in[m] : [Ad^s]_i = 0\}. \tag*{(73)} R := { v ∈ R n : A v ≤ 0 , Q v = 0 } , d s := proj R ( − c ) , I 0 := { i ∈ [ m ] : [ A d s ] i = 0 } . (73)
We also prove the alternative between optimality and unboundedness in that theorem. Problem (1) is bounded below if and only if no direction d d d satisfies A d ≤ 0 Ad \le0 A d ≤ 0 , Q d = 0 Qd = 0 Q d = 0 , and ⟨ c , d ⟩ < 0 \langle c,d\rangle< 0 ⟨ c , d ⟩ < 0 , and it then attains its infimum [59 ] . Since R \mathcal{R} R is a closed convex cone, Moreau’s decomposition theorem gives
⟨ c + d s , d s ⟩ = 0 and ⟨ c + d s , r ⟩ ≥ 0 for every r ∈ R . (74) \langle c+d^s,d^s\rangle= 0 \qquad\text{and} \qquad\langle c+d^s,r\rangle\ge0 \text{ for every } r \in\mathcal{R}. \tag*{(74)} ⟨ c + d s , d s ⟩ = 0 and ⟨ c + d s , r ⟩ ≥ 0 for every r ∈ R . (74)
We also let C 0 : = { y ∈ R n : [ A y ] I 0 ≤ b I 0 } C_0 := \{y \in\mathbb{R}^n : [Ay]_{I_0} \le b_{I_0}\} C 0 := { y ∈ R n : [ A y ] I 0 ≤ b I 0 } , f 0 ( y ) : = f ( y ) + ⟨ d s , y ⟩ f_0(y) := f(y) + \langle d^s,y\rangle f 0 ( y ) := f ( y ) + ⟨ d s , y ⟩ , and V 0 : = f 0 + χ C 0 V_0 := f_0 + \chi_{C_0} V 0 := f 0 + χ C 0 , so that V 0 V_0 V 0 is the objective function of problem (1) with the data A I 0 A_{I_0} A I 0 , b I 0 b_{I_0} b I 0 , and c + d s c+d^s c + d s .
Lemma B.10. The following hold:
(i) The direction d s d^s d s satisfies A d s ≤ 0 Ad^s \le0 A d s ≤ 0 , Q d s = 0 Qd^s = 0 Q d s = 0 , and ⟨ c , d s ⟩ = − ∥ d s ∥ 2 2 \langle c,d^s\rangle= -\lVert d^s\rVert_2^2 ⟨ c , d s ⟩ = − ∥ d s ∥ 2 2 . If d s ≠ 0 d^s \ne0 d s = 0 , then for every x ∈ C x \in C x ∈ C the ray x + Δ d s x+\Delta d^s x + Δ d s , Δ ≥ 0 \Delta\ge0 Δ ≥ 0 , is feasible and V ( x + Δ d s ) = V ( x ) − Δ ∥ d s ∥ 2 2 → − ∞ V(x+\Delta d^s) = V(x)-\Delta\lVert d^s\rVert_2^2 \to-\infty V ( x + Δ d s ) = V ( x ) − Δ ∥ d s ∥ 2 2 → − ∞ as Δ → + ∞ \Delta\to+\infty Δ → + ∞ .
(ii) Every x ∈ C x \in C x ∈ C satisfies ∥ d ( x ) − d s ∥ 2 2 ≤ ∥ d ( x ) ∥ 2 2 − ∥ d s ∥ 2 2 \lVert d(x)-d^s\rVert_2^2 \le\lVert d(x)\rVert_2^2-\lVert d^s\rVert_2^2 ∥ d ( x ) − d s ∥ 2 2 ≤ ∥ d ( x ) ∥ 2 2 − ∥ d s ∥ 2 2 .
(iii) The function f 0 f_0 f 0 attains its minimum over C 0 C_0 C 0 .
Proof. The first two relations hold because d s ∈ R d^s \in\mathcal{R} d s ∈ R , and the third is the first relation in (68). The ray is feasible because A d s ≤ 0 Ad^s \le0 A d s ≤ 0 , and the formula for V ( x + Δ d s ) V(x+\Delta d^s) V ( x + Δ d s ) follows from Q d s = 0 Qd^s = 0 Q d s = 0 and the third relation. This proves (i).
Let x ∈ C x \in C x ∈ C . Every r ∈ R r \in\mathcal{R} r ∈ R satisfies ⟨ Q x , r ⟩ = ⟨ x , Q r ⟩ = 0 \langle Qx,r\rangle= \langle x,Qr\rangle= 0 ⟨ Q x , r ⟩ = ⟨ x , Q r ⟩ = 0 , so ∥ − ∇ x f ( x ) − r ∥ 2 2 \lVert-\nabla_x f(x)-r\rVert_2^2 ∥ − ∇ x f ( x ) − r ∥ 2 2 and ∥ − c − r ∥ 2 2 \lVert-c-r\rVert_2^2 ∥ − c − r ∥ 2 2 differ by a constant independent of r r r , and therefore proj R ( − ∇ x f ( x ) ) = d s \operatorname{proj}_{\mathcal{R}}(-\nabla_x f(x)) = d^s proj R ( − ∇ x f ( x )) = d s . Moreover R ⊆ T C ( x ) = { s ∈ R n : [ A s ] E ( x ) ≤ 0 } \mathcal{R} \subseteq T_C(x) = \{s \in\mathbb{R}^n : [As]_{\mathcal{E}(x)} \le0\} R ⊆ T C ( x ) = { s ∈ R n : [ A s ] E ( x ) ≤ 0 } . Lemma B.4, applied to R ⊆ T C ( x ) \mathcal{R} \subseteq T_C(x) R ⊆ T C ( x ) and the vector − ∇ x f ( x ) -\nabla_x f(x) − ∇ x f ( x ) , gives (ii).
The function f 0 f_0 f 0 is a convex quadratic with linear term c + d s c+d^s c + d s , and C ⊂ C 0 C \subset C_0 C ⊂ C 0 is nonempty, so the criterion above, applied to problem (1) with the data A I 0 A_{I_0} A I 0 , b I 0 b_{I_0} b I 0 , and c + d s c+d^s c + d s , reduces the claim to showing that ⟨ c + d s , r ⟩ ≥ 0 \langle c+d^s,r\rangle\ge0 ⟨ c + d s , r ⟩ ≥ 0 for every r r r with [ A r ] I 0 ≤ 0 [Ar]_{I_0} \le0 [ A r ] I 0 ≤ 0 and Q r = 0 Qr = 0 Q r = 0 . For such r r r and for Δ ≥ 0 \Delta\ge0 Δ ≥ 0 large enough, r + Δ d s ∈ R r+\Delta d^s \in\mathcal{R} r + Δ d s ∈ R , because [ A d s ] i < 0 [Ad^s]_i < 0 [ A d s ] i < 0 for every i ∉ I 0 i \notin I_0 i ∈ / I 0 . Both relations in (68) then give ⟨ c + d s , r ⟩ = ⟨ c + d s , r + Δ d s ⟩ ≥ 0 \langle c+d^s,r\rangle= \langle c+d^s,r+\Delta d^s\rangle\ge0 ⟨ c + d s , r ⟩ = ⟨ c + d s , r + Δ d s ⟩ ≥ 0 . This proves (iii).
Proposition B.1. The following hold:
(i) The direction satisfies d ( x ( τ ) ) → d s \mathbf{d}(\mathbf{x}(\tau)) \to\mathbf{d}^{s} d ( x ( τ )) → d s as τ → + ∞ \tau\to+\infty τ → + ∞ .
(ii) There exists τ 0 ≥ 0 \tau_{0} \ge0 τ 0 ≥ 0 such that E ( x ( τ ) ) ⊆ I 0 \mathcal{E}(\mathbf{x}(\tau)) \subseteq I_{0} E ( x ( τ )) ⊆ I 0 for every τ ≥ τ 0 \tau\ge\tau_{0} τ ≥ τ 0 .
(iii) The function τ ↦ y ( τ 0 + τ ) \tau\mapsto\mathbf{y}(\tau_{0}+\tau) τ ↦ y ( τ 0 + τ ) , where y ( τ ) : = x ( τ ) − τ d s \mathbf{y}(\tau) := \mathbf{x}(\tau)-\tau\mathbf{d}^{s} y ( τ ) := x ( τ ) − τ d s , is the solution of the differential inclusion (4) for V 0 V_{0} V 0 with initial value y ( τ 0 ) \mathbf{y}(\tau_{0}) y ( τ 0 ) . In particular, τ ↦ ∥ d ( x ( τ ) ) − d s ∥ 2 \tau\mapsto\lVert\mathbf{d}(\mathbf{x}(\tau))-\mathbf{d}^{s}\rVert_{2} τ ↦ ∥ d ( x ( τ )) − d s ∥ 2 is nonincreasing on [ τ 0 , + ∞ ) [\tau_{0},+\infty) [ τ 0 , + ∞ ) . The function y \mathbf{y} y converges as τ → + ∞ \tau\to+\infty τ → + ∞ to a point y ˉ \bar{\mathbf{y}} y ˉ that minimizes f 0 f_{0} f 0 over C 0 C_{0} C 0 . If d s = 0 \mathbf{d}^{s}=0 d s = 0 , then y ˉ = lim τ → + ∞ x ( τ ) \bar{\mathbf{y}}=\lim_{\tau\to+\infty}\mathbf{x}(\tau) y ˉ = lim τ → + ∞ x ( τ ) is an optimal solution of problem (1).
Proof. The function τ ↦ f 0 ( x ( τ ) ) \tau\mapsto f_{0}(\mathbf{x}(\tau)) τ ↦ f 0 ( x ( τ )) is Lipschitz continuous on bounded intervals, and at almost every τ \tau τ its derivative is ⟨ ∇ x f ( x ( τ ) ) + d s , d ( x ( τ ) ) ⟩ = − ∥ d ( x ( τ ) ) ∥ 2 2 + ⟨ d s , d ( x ( τ ) ) ⟩ \langle\nabla_{\mathbf{x}}f(\mathbf{x}(\tau))+\mathbf{d}^{s},\mathbf{d}(\mathbf{x}(\tau))\rangle=-\lVert\mathbf{d}(\mathbf{x}(\tau))\rVert_{2}^{2}+\langle\mathbf{d}^{s},\mathbf{d}(\mathbf{x}(\tau))\rangle ⟨ ∇ x f ( x ( τ )) + d s , d ( x ( τ ))⟩ = − ∥ d ( x ( τ )) ∥ 2 2 + ⟨ d s , d ( x ( τ ))⟩ , by (13d). Expanding the square in Lemma B.10 (ii) gives ⟨ d ( x ) , d s ⟩ ≥ ∥ d s ∥ 2 2 \langle\mathbf{d}(\mathbf{x}),\mathbf{d}^{s}\rangle\ge\lVert\mathbf{d}^{s}\rVert_{2}^{2} ⟨ d ( x ) , d s ⟩ ≥ ∥ d s ∥ 2 2 for every x ∈ C \mathbf{x}\in C x ∈ C , hence
∥ d ( x ) − d s ∥ 2 2 = ∥ d ( x ) ∥ 2 2 − 2 ⟨ d ( x ) , d s ⟩ + ∥ d s ∥ 2 2 ≤ ∥ d ( x ) ∥ 2 2 − ⟨ d ( x ) , d s ⟩ . \lVert\mathbf{d}(\mathbf{x})-\mathbf{d}^{s}\rVert_{2}^{2}=\lVert\mathbf{d}(\mathbf{x})\rVert_{2}^{2}-2\langle\mathbf{d}(\mathbf{x}),\mathbf{d}^{s}\rangle+\lVert\mathbf{d}^{s}\rVert_{2}^{2}\le\lVert\mathbf{d}(\mathbf{x})\rVert_{2}^{2}-\langle\mathbf{d}(\mathbf{x}),\mathbf{d}^{s}\rangle. ∥ d ( x ) − d s ∥ 2 2 = ∥ d ( x ) ∥ 2 2 − 2 ⟨ d ( x ) , d s ⟩ + ∥ d s ∥ 2 2 ≤ ∥ d ( x ) ∥ 2 2 − ⟨ d ( x ) , d s ⟩ .
Integrating, and using x ( τ ) ∈ C ⊂ C 0 \mathbf{x}(\tau)\in C\subset C_{0} x ( τ ) ∈ C ⊂ C 0 with Lemma B.10 (iii), we obtain for every $\tau\ge 0
∫ 0 τ ∥ d ( x ( ρ ) ) − d s ∥ 2 2 d ρ ≤ f 0 ( x ( 0 ) ) − f 0 ( x ( τ ) ) ≤ f 0 ( x ( 0 ) ) − min C 0 f 0 . \int_{0}^{\tau}\lVert\mathbf{d}(\mathbf{x}(\rho))-\mathbf{d}^{s}\rVert_{2}^{2}\,d\rho\le f_{0}(\mathbf{x}^{(0)})-f_{0}(\mathbf{x}(\tau))\le f_{0}(\mathbf{x}^{(0)})-\min_{C_{0}}f_{0}. ∫ 0 τ ∥ d ( x ( ρ )) − d s ∥ 2 2 d ρ ≤ f 0 ( x ( 0 ) ) − f 0 ( x ( τ )) ≤ f 0 ( x ( 0 ) ) − C 0 min f 0 .
so ∫ 0 + ∞ ∥ d ( x ( ρ ) ) − d s ∥ 2 2 d ρ \int_{0}^{+\infty}\lVert\mathbf{d}(\mathbf{x}(\rho))-\mathbf{d}^{s}\rVert_{2}^{2}\,d\rho ∫ 0 + ∞ ∥ d ( x ( ρ )) − d s ∥ 2 2 d ρ is finite. By Theorem 2.1 (i) and (iii), ∥ d ( x ( τ ) ) ∥ 2 2 \lVert\mathbf{d}(\mathbf{x}(\tau))\rVert_{2}^{2} ∥ d ( x ( τ )) ∥ 2 2 decreases to a limit ν \nu ν , and ν ≥ ∥ d s ∥ 2 2 \nu\ge\lVert\mathbf{d}^{s}\rVert_{2}^{2} ν ≥ ∥ d s ∥ 2 2 by Lemma B.10 (ii). Since ∥ d ( x ( τ ) ) − d s ∥ 2 2 ≥ ν − ∥ d s ∥ 2 2 \lVert\mathbf{d}(\mathbf{x}(\tau))-\mathbf{d}^{s}\rVert_{2}^{2}\ge\nu-\lVert\mathbf{d}^{s}\rVert_{2}^{2} ∥ d ( x ( τ )) − d s ∥ 2 2 ≥ ν − ∥ d s ∥ 2 2 for every τ \tau τ and the integral is finite, ν = ∥ d s ∥ 2 2 \nu=\lVert\mathbf{d}^{s}\rVert_{2}^{2} ν = ∥ d s ∥ 2 2 . Lemma B.10 (ii) at x = x ( τ ) \mathbf{x}=\mathbf{x}(\tau) x = x ( τ ) then gives ∥ d ( x ( τ ) ) − d s ∥ 2 2 ≤ ∥ d ( x ( τ ) ) ∥ 2 2 − ∥ d s ∥ 2 2 → 0 \lVert\mathbf{d}(\mathbf{x}(\tau))-\mathbf{d}^{s}\rVert_{2}^{2}\le\lVert\mathbf{d}(\mathbf{x}(\tau))\rVert_{2}^{2}-\lVert\mathbf{d}^{s}\rVert_{2}^{2}\to0 ∥ d ( x ( τ )) − d s ∥ 2 2 ≤ ∥ d ( x ( τ )) ∥ 2 2 − ∥ d s ∥ 2 2 → 0 . This proves (i).
Let i ∉ I 0 i\notin I_{0} i ∈ / I 0 . By (i), [ A d ( x ( τ ) ) ] i → [ A d s ] i < 0 [A\mathbf{d}(\mathbf{x}(\tau))]_{i}\to[A\mathbf{d}^{s}]_{i}<0 [ A d ( x ( τ )) ] i → [ A d s ] i < 0 , so [ A x ( τ ) ] i − b i = [ A x ( 0 ) ] i − b i + ∫ 0 τ [ A d ( x ( ρ ) ) ] i d ρ → − ∞ [A\mathbf{x}(\tau)]_{i}-b_{i}=[A\mathbf{x}^{(0)}]_{i}-b_{i}+\int_{0}^{\tau}[A\mathbf{d}(\mathbf{x}(\rho))]_{i}\,d\rho\to-\infty [ A x ( τ ) ] i − b i = [ A x ( 0 ) ] i − b i + ∫ 0 τ [ A d ( x ( ρ )) ] i d ρ → − ∞ . The constraints outside I 0 I_{0} I 0 are finitely many, so all of them are inactive at x ( τ ) \mathbf{x}(\tau) x ( τ ) from some time τ 0 \tau_{0} τ 0 on. This proves (ii).
Let τ ≥ τ 0 \tau\ge\tau_{0} τ ≥ τ 0 . Since [ A y ( τ ) ] I 0 = [ A x ( τ ) ] I 0 [A\mathbf{y}(\tau)]_{I_{0}}=[A\mathbf{x}(\tau)]_{I_{0}} [ A y ( τ ) ] I 0 = [ A x ( τ ) ] I 0 , we have y ( τ ) ∈ C 0 \mathbf{y}(\tau)\in C_{0} y ( τ ) ∈ C 0 , and by (ii) the constraints of C 0 C_{0} C 0 active at y ( τ ) \mathbf{y}(\tau) y ( τ ) are those of E ( x ( τ ) ) \mathcal{E}(\mathbf{x}(\tau)) E ( x ( τ )) , so N C 0 ( y ( τ ) ) = N C ( x ( τ ) ) N_{C_{0}}(\mathbf{y}(\tau))=N_{C}(\mathbf{x}(\tau)) N C 0 ( y ( τ )) = N C ( x ( τ )) , by [16 Example 5.2.6b] . Since Q d s = 0 Q\mathbf{d}^{s}=0 Q d s = 0 , we have ∇ f 0 ( y ( τ ) ) = ∇ x f ( x ( τ ) ) + d s \nabla f_{0}(\mathbf{y}(\tau))=\nabla_{\mathbf{x}}f(\mathbf{x}(\tau))+\mathbf{d}^{s} ∇ f 0 ( y ( τ )) = ∇ x f ( x ( τ )) + d s . With n ( x ) = − ∇ x f ( x ) − d ( x ) ∈ N C ( x ) \mathbf{n}(\mathbf{x})=-\nabla_{\mathbf{x}}f(\mathbf{x})-\mathbf{d}(\mathbf{x})\in N_{C}(\mathbf{x}) n ( x ) = − ∇ x f ( x ) − d ( x ) ∈ N C ( x ) as in § 2.4 , the right derivative of y \mathbf{y} y is therefore
y ˙ ( τ ) = d ( x ( τ ) ) − d s = − ∇ f 0 ( y ( τ ) ) − n ( x ( τ ) ) ∈ − ∂ V 0 ( y ( τ ) ) . \dot{\mathbf{y}}(\tau)=\mathbf{d}(\mathbf{x}(\tau))-\mathbf{d}^{s}=-\nabla f_{0}(\mathbf{y}(\tau))-\mathbf{n}(\mathbf{x}(\tau))\in-\partial V_{0}(\mathbf{y}(\tau)). y ˙ ( τ ) = d ( x ( τ )) − d s = − ∇ f 0 ( y ( τ )) − n ( x ( τ )) ∈ − ∂ V 0 ( y ( τ )) .
The function τ ↦ y ( τ 0 + τ ) \tau\mapsto\mathbf{y}(\tau_{0}+\tau) τ ↦ y ( τ 0 + τ ) is Lipschitz continuous, so it solves (4) for V 0 V_{0} V 0 with initial value y ( τ 0 ) ∈ C 0 \mathbf{y}(\tau_{0})\in C_{0} y ( τ 0 ) ∈ C 0 . It is the slow solution of that problem. Indeed, two Lipschitz continuous solutions with the same initial value coincide because the monotonicity of ∂ V 0 \partial V_{0} ∂ V 0 makes the squared distance between them nonincreasing. Theorem 2.1 (i) then makes the norm of its right derivative d ( x ( τ ) ) − d s \mathbf{d}(\mathbf{x}(\tau))-\mathbf{d}^{s} d ( x ( τ )) − d s nonincreasing. Since V 0 V_{0} V 0 attains its minimum by Lemma B.10 (iii), Theorem 2.1 (ii) shows that y ( τ ) \mathbf{y}(\tau) y ( τ ) converges to a minimizer y ˉ \bar{\mathbf{y}} y ˉ of V 0 V_{0} V 0 . If d s = 0 \mathbf{d}^{s}=0 d s = 0 , then I 0 = [ m ] I_{0}=[m] I 0 = [ m ] , C 0 = C C_{0}=C C 0 = C , V 0 = V V_{0}=V V 0 = V , and y = x \mathbf{y}=\mathbf{x} y = x , which gives the last claim.
No accumulation of breakpoints at a finite time
Proposition B.2. The breakpoints of the slow solution do not accumulate at a finite time, in the sense that if there are infinitely many, then τ ( k ) → + ∞ \tau^{(k)}\to+\infty τ ( k ) → + ∞ .
Suppose, for contradiction, that there are infinitely many breakpoints and that lim k → + ∞ τ ( k ) = τ ˉ < + ∞ \lim_{k\to+\infty}\tau^{(k)}=\bar{\tau}<+\infty lim k → + ∞ τ ( k ) = τ ˉ < + ∞ . We fix the events and the tail as in Appendix § 11 .§ 2.2 and write σ ( τ ) : = τ ˉ − τ \sigma(\tau):=\bar{\tau}-\tau σ ( τ ) := τ ˉ − τ .
Lemma B.11. In this setting, the following holds.
(i) The limit x ˉ : = lim τ ↑ τ ˉ x ( τ ) \bar{x} := \lim_{\tau\uparrow\bar{\tau}} x(\tau) x ˉ := lim τ ↑ τ ˉ x ( τ ) exists. Every recurring active set J J J satisfies A J x ˉ = b J A_J\bar{x} = b_J A J x ˉ = b J and, with d ˉ J : = − P J ⊥ ∇ x f ( x ˉ ) \bar{d}_J := -P_J^\perp\nabla_x f(\bar{x}) d ˉ J := − P J ⊥ ∇ x f ( x ˉ ) ,
d ( x ( τ ) ) = d ˉ J − S J ( x ( τ ) − x ˉ ) on every J -piece of the tail . (75) d(x(\tau)) = \bar{d}_J - S_J(x(\tau)-\bar{x}) \quad\text{on every } J\text{-piece of the tail}. \tag*{(75)} d ( x ( τ )) = d ˉ J − S J ( x ( τ ) − x ˉ ) on every J -piece of the tail . (75) (ii) The vectors d ˉ J \bar{d}_J d ˉ J of the recurring active sets coincide. Their common value v ˉ \bar{v} v ˉ satisfies d ( x ( τ ) ) → v ˉ d(x(\tau)) \to\bar{v} d ( x ( τ )) → v ˉ as τ ↑ τ ˉ \tau\uparrow\bar{\tau} τ ↑ τ ˉ and v ˉ ∈ ker A J \bar{v} \in\ker A_J v ˉ ∈ ker A J for every recurring J J J .
Proof. The slow solution is Lipschitz continuous by Theorem 2.1 , so x ( τ ) x(\tau) x ( τ ) has a limit x ˉ \bar{x} x ˉ as τ ↑ τ ˉ \tau\uparrow\bar{\tau} τ ↑ τ ˉ , which lies in the closed set C C C . Let J J J be recurring. Its constraints are active on the J J J -pieces by Lemma B.8 (i), and these pieces contain times arbitrarily close to τ ˉ \bar{\tau} τ ˉ , so A J x ˉ = b J A_J\bar{x} = b_J A J x ˉ = b J and x ( τ ) − x ˉ ∈ ker A J x(\tau)-\bar{x} \in\ker A_J x ( τ ) − x ˉ ∈ ker A J on every J J J -piece. Since ∇ x f \nabla_x f ∇ x f is affine with slope Q Q Q , formula (65) gives d ( x ( τ ) ) = d ˉ J − P J ⊥ Q ( x ( τ ) − x ˉ ) = d ˉ J − S J ( x ( τ ) − x ˉ ) d(x(\tau)) = \bar{d}_J - P_J^\perp Q(x(\tau)-\bar{x}) = \bar{d}_J - S_J(x(\tau)-\bar{x}) d ( x ( τ )) = d ˉ J − P J ⊥ Q ( x ( τ ) − x ˉ ) = d ˉ J − S J ( x ( τ ) − x ˉ ) there. This proves (i).
Let τ ( k + 1 ) \tau^{(k+1)} τ ( k + 1 ) be a breakpoint of the tail, let J = J ( k ) J = J^{(k)} J = J ( k ) , and let d − : = − P J ⊥ ∇ x f ( x ( k + 1 ) ) d^- := -P_J^\perp\nabla_x f(x^{(k+1)}) d − := − P J ⊥ ∇ x f ( x ( k + 1 ) ) be the limit of d ( x ( τ ) ) d(x(\tau)) d ( x ( τ )) as τ ↑ τ ( k + 1 ) \tau\uparrow\tau^{(k+1)} τ ↑ τ ( k + 1 ) . By Lemma B.8 (i), n − : = − ∇ x f ( x ( k + 1 ) ) − d − n^- := -\nabla_x f(x^{(k+1)}) - d^- n − := − ∇ x f ( x ( k + 1 ) ) − d − lies in N ( k ) N^{(k)} N ( k ) , and A J ∪ R ( k ) d − = 0 A_{J\cup R^{(k)}}d^- = 0 A J ∪ R ( k ) d − = 0 . By Lemma B.3 , some B ⊆ J ∪ R ( k ) B \subseteq J \cup R^{(k)} B ⊆ J ∪ R ( k ) whose rows form a basis of the row space of A J ∪ R ( k ) A_{J\cup R^{(k)}} A J ∪ R ( k ) satisfies ( A B A B ⊤ ) − 1 A B n − ≥ 0 (A_BA_B^\top)^{-1}A_Bn^- \ge0 ( A B A B ⊤ ) − 1 A B n − ≥ 0 . Since A B d − = 0 A_Bd^- = 0 A B d − = 0 , this vector is p B ( x ( k + 1 ) ) p_B(x^{(k+1)}) p B ( x ( k + 1 ) ) , and d − = − P B ⊥ ∇ x f ( x ( k + 1 ) ) d^- = -P_B^\perp\nabla_x f(x^{(k+1)}) d − = − P B ⊥ ∇ x f ( x ( k + 1 ) ) , because d − ∈ ker A B d^- \in\ker A_B d − ∈ ker A B and n − ∈ Im ( A B ⊤ ) n^- \in\operatorname{Im}(A_B^\top) n − ∈ Im ( A B ⊤ ) . Lemma B.2 with B B B for J J J then gives d − = proj K ( − ∇ x f ( x ( k + 1 ) ) ) d^- = \operatorname{proj}_{K}(-\nabla_x f(x^{(k+1)})) d − = proj K ( − ∇ x f ( x ( k + 1 ) )) for K : = { s ∈ R n : A B s ≤ 0 } K := \{s \in\mathbb{R}^n : A_Bs \le0\} K := { s ∈ R n : A B s ≤ 0 } . The constraints of B B B are active at x ( k + 1 ) x^{(k+1)} x ( k + 1 ) , so T C ( x ( k + 1 ) ) ⊆ K T_C(x^{(k+1)}) \subseteq K T C ( x ( k + 1 ) ) ⊆ K , and Lemma B.4 gives
∥ d ( k + 1 ) − d − ∥ 2 2 ≤ ∥ d − ∥ 2 2 − ∥ d ( k + 1 ) ∥ 2 2 ≤ ∥ d ( k ) ∥ 2 2 − ∥ d ( k + 1 ) ∥ 2 2 . \left\|d^{(k+1)}-d^-\right\|_2^2 \le\left\|d^-\right\|_2^2-\left\|d^{(k+1)}\right\|_2^2 \le\left\|d^{(k)}\right\|_2^2-\left\|d^{(k+1)}\right\|_2^2. d ( k + 1 ) − d − 2 2 ≤ d − 2 2 − d ( k + 1 ) 2 2 ≤ d ( k ) 2 2 − d ( k + 1 ) 2 2 .
where the second inequality holds because ∥ d − ∥ 2 = lim τ ↑ τ ( k + 1 ) ∥ d ( x ( τ ) ) ∥ 2 ≤ ∥ d ( k ) ∥ 2 \left\|d^-\right\|_2 = \lim_{\tau\uparrow\tau^{(k+1)}} \left\|d(x(\tau))\right\|_2 \le\left\|d^{(k)}\right\|_2 ∥ d − ∥ 2 = lim τ ↑ τ ( k + 1 ) ∥ d ( x ( τ )) ∥ 2 ≤ d ( k ) 2 , the function τ ↦ ∥ x ˙ ( τ ) ∥ 2 = ∥ d ( x ( τ ) ) ∥ 2 \tau\mapsto\|\dot{x}(\tau)\|_2 = \|d(x(\tau))\|_2 τ ↦ ∥ x ˙ ( τ ) ∥ 2 = ∥ d ( x ( τ )) ∥ 2 being nonincreasing by Theorem 2.1 (i) and (iii). Similarly, the sequence ∥ d ( k ) ∥ 2 \left\|d^{(k)}\right\|_2 d ( k ) 2 converges, so the right-hand side tends to 0 as k → ∞ k \to\infty k → ∞ . By (69) and continuity,
d − = d ˉ J ( k ) − S J ( k ) ( x ( k + 1 ) − x ˉ ) and d ( k + 1 ) = d ˉ J ( k + 1 ) − S J ( k + 1 ) ( x ( k + 1 ) − x ˉ ) , d^- = \bar{d}_{J^{(k)}} - S_{J^{(k)}}(x^{(k+1)}-\bar{x}) \quad\text{and} \quad d^{(k+1)} = \bar{d}_{J^{(k+1)}} - S_{J^{(k+1)}}(x^{(k+1)}-\bar{x}), d − = d ˉ J ( k ) − S J ( k ) ( x ( k + 1 ) − x ˉ ) and d ( k + 1 ) = d ˉ J ( k + 1 ) − S J ( k + 1 ) ( x ( k + 1 ) − x ˉ ) ,
so
∥ d ˉ J ( k + 1 ) − d ˉ J ( k ) ∥ 2 ≤ ∥ d ( k + 1 ) − d − ∥ 2 + 2 λ max ∥ x ( k + 1 ) − x ˉ ∥ 2 → 0. \left\|\bar{d}_{J^{(k+1)}}-\bar{d}_{J^{(k)}}\right\|_2 \le\left\|d^{(k+1)}-d^-\right\|_2 + 2\lambda_{\max}\left\|x^{(k+1)}-\bar{x}\right\|_2 \to0. d ˉ J ( k + 1 ) − d ˉ J ( k ) 2 ≤ d ( k + 1 ) − d − 2 + 2 λ m a x x ( k + 1 ) − x ˉ 2 → 0.
The vectors d ˉ J \bar{d}_J d ˉ J of the recurring active sets are finitely many, so d ˉ J ( k + 1 ) = d ˉ J ( k ) \bar{d}_{J^{(k+1)}} = \bar{d}_{J^{(k)}} d ˉ J ( k + 1 ) = d ˉ J ( k ) for every large k k k : either they all coincide, or the bound eventually falls below their least pairwise distance. Hence d ˉ J ( k ) \bar{d}_{J^{(k)}} d ˉ J ( k ) converges to some constant v ˉ \bar{v} v ˉ . Every recurring active set occurs at arbitrarily large k k k , so its vector d ˉ J \bar{d}_J d ˉ J equals v ˉ \bar{v} v ˉ . Then (69) gives d ( x ( τ ) ) → v ˉ d(x(\tau)) \to\bar{v} d ( x ( τ )) → v ˉ as τ ↑ τ ˉ \tau\uparrow\bar{\tau} τ ↑ τ ˉ , and the directions on the J J J -pieces lie in the closed subspace ker A J \ker A_J ker A J , so v ˉ ∈ ker A J \bar{v} \in\ker A_J v ˉ ∈ ker A J . □
We measure the deviation of the trajectory from the affine path τ ↦ x ˉ − σ ( τ ) v ˉ \tau\mapsto\bar{x}-\sigma(\tau)\bar{v} τ ↦ x ˉ − σ ( τ ) v ˉ via
ζ ( τ ) : = x ( τ ) − x ˉ + σ ( τ ) v ˉ . (76) \zeta(\tau) := x(\tau)-\bar{x}+\sigma(\tau)\bar{v}. \tag*{(76)} ζ ( τ ) := x ( τ ) − x ˉ + σ ( τ ) v ˉ . (76)
The slow solution is Lipschitz continuous with x ˙ = d ( x ) \dot{x}=d(x) x ˙ = d ( x ) , so x ( τ ) − x ˉ = − ∫ τ τ ˉ d ( x ( ρ ) ) d ρ x(\tau)-\bar{x}=-\int_\tau^{\bar{\tau}}d(x(\rho))\,d\rho x ( τ ) − x ˉ = − ∫ τ τ ˉ d ( x ( ρ )) d ρ and
ζ ( τ ) = − ∫ τ τ ˉ ( d ( x ( ρ ) ) − v ˉ ) d ρ , ∥ ζ ( τ ) ∥ 2 ≤ σ ( τ ) sup ρ ∈ [ τ , τ ˉ ) ∥ d ( x ( ρ ) ) − v ˉ ∥ 2 . (77) \zeta(\tau)=-\int_\tau^{\bar{\tau}}(d(x(\rho))-\bar{v})\,d\rho,\qquad\|\zeta(\tau)\|_2 \le\sigma(\tau)\sup_{\rho\in[\tau,\bar{\tau})}\|d(x(\rho))-\bar{v}\|_2. \tag*{(77)} ζ ( τ ) = − ∫ τ τ ˉ ( d ( x ( ρ )) − v ˉ ) d ρ , ∥ ζ ( τ ) ∥ 2 ≤ σ ( τ ) ρ ∈ [ τ , τ ˉ ) sup ∥ d ( x ( ρ )) − v ˉ ∥ 2 . (77)
By Lemma B.11 (ii), the supremum tends to 0 as τ ↑ τ ˉ \tau\uparrow\bar{\tau} τ ↑ τ ˉ .
Lemma B.12. In this setting, the following holds.
(i) On every J J J -piece of the tail, ζ ˙ = σ S J v ˉ − S J ζ \dot{\zeta}=\sigma S_J\bar{v}-S_J\zeta ζ ˙ = σ S J v ˉ − S J ζ , and A J ζ = 0 A_J\zeta=0 A J ζ = 0 on the piece and at both of its endpoints.
(ii) Let ϕ ( x ) = ϕ ( x ˉ ) + ⟨ a , x − x ˉ ⟩ \phi(x)=\phi(\bar{x})+\langle a,x-\bar{x}\rangle ϕ ( x ) = ϕ ( x ˉ ) + ⟨ a , x − x ˉ ⟩ be an affine function that vanishes at x ( k ) x^{(k)} x ( k ) for infinitely many k k k . Then ϕ ( x ˉ ) = 0 \phi(\bar{x})=0 ϕ ( x ˉ ) = 0 , ⟨ a , v ˉ ⟩ = 0 \langle a,\bar{v}\rangle=0 ⟨ a , v ˉ ⟩ = 0 , and ϕ ( x ( τ ) ) = ⟨ a , ζ ( τ ) ⟩ \phi(x(\tau))=\langle a,\zeta(\tau)\rangle ϕ ( x ( τ )) = ⟨ a , ζ ( τ )⟩ for every τ \tau τ on the tail.
Proof. By Lemma B.11, A J ( x ( τ ) − x ˉ ) = 0 A_J(x(\tau)-\bar{x})=0 A J ( x ( τ ) − x ˉ ) = 0 on the piece and at its endpoints, and A J v ˉ = 0 A_J\bar{v}=0 A J v ˉ = 0 , which gives the second claim. For the first, (69) with d ˉ J = v ˉ \bar{d}_J=\bar{v} d ˉ J = v ˉ and (70) give x ˙ = v ˉ − S J ( ζ − σ v ˉ ) = v ˉ + σ S J v ˉ − S J ζ \dot{x}=\bar{v}-S_J(\zeta-\sigma\bar{v})=\bar{v}+\sigma S_J\bar{v}-S_J\zeta x ˙ = v ˉ − S J ( ζ − σ v ˉ ) = v ˉ + σ S J v ˉ − S J ζ , and ζ ˙ = x ˙ − v ˉ \dot{\zeta}=\dot{x}-\bar{v} ζ ˙ = x ˙ − v ˉ because σ ˙ = − 1 \dot{\sigma}=-1 σ ˙ = − 1 . This proves (i).
The iterates x ( k ) x^{(k)} x ( k ) converge to x ˉ \bar{x} x ˉ , so ϕ ( x ˉ ) = 0 \phi(\bar{x})=0 ϕ ( x ˉ ) = 0 by continuity. At the iterates where ϕ \phi ϕ vanishes, (70) gives 0 = ⟨ a , x ( k ) − x ˉ ⟩ = − σ ( τ ( k ) ) ⟨ a , v ˉ ⟩ + ⟨ a , ζ ( τ ( k ) ) ⟩ 0=\langle a,x^{(k)}-\bar{x}\rangle=-\sigma(\tau^{(k)})\langle a,\bar{v}\rangle+\langle a,\zeta(\tau^{(k)})\rangle 0 = ⟨ a , x ( k ) − x ˉ ⟩ = − σ ( τ ( k ) ) ⟨ a , v ˉ ⟩ + ⟨ a , ζ ( τ ( k ) )⟩ . Dividing by σ ( τ ( k ) ) \sigma(\tau^{(k)}) σ ( τ ( k ) ) and letting k → ∞ k\to\infty k → ∞ gives ⟨ a , v ˉ ⟩ = 0 \langle a,\bar{v}\rangle=0 ⟨ a , v ˉ ⟩ = 0 , by (71). Hence ϕ ( x ( τ ) ) = ⟨ a , ζ ( τ ) − σ ( τ ) v ˉ ⟩ = ⟨ a , ζ ( τ ) ⟩ \phi(x(\tau))=\langle a,\zeta(\tau)-\sigma(\tau)\bar{v}\rangle=\langle a,\zeta(\tau)\rangle ϕ ( x ( τ )) = ⟨ a , ζ ( τ ) − σ ( τ ) v ˉ ⟩ = ⟨ a , ζ ( τ )⟩ . This proves (ii).
Proof of Proposition B.2. We apply the lemmas of Appendix B.2.1 to the pieces of the tail, relabeled from 0, with the recurring active sets as the family J \mathcal{J} J . The hypotheses of Lemma B.6 hold for ζ \zeta ζ and v ˉ \bar{v} v ˉ . Indeed, the limit velocity lies in ker A J \ker A_J ker A J for every recurring J J J by Lemma B.11(ii), the deviation is continuous and bounded by (71), and Lemma B.12(i) gives A J ( τ ) ζ ( τ ) = 0 A_{J(\tau)}\zeta(\tau)=0 A J ( τ ) ζ ( τ ) = 0 together with ζ ˙ = σ S J v ˉ − S J ζ \dot{\zeta}=\sigma S_J\bar{v}-S_J\zeta ζ ˙ = σ S J v ˉ − S J ζ on every J J J -piece. The integrand of that equation is bounded and [ τ , τ ′ ] [\tau,\tau'] [ τ , τ ′ ] meets finitely many pieces for τ < τ ′ < τ ˉ \tau<\tau'<\bar{\tau} τ < τ ′ < τ ˉ , so ζ ( τ ) − ζ ( τ ′ ) = − ∫ τ τ ′ ζ ˙ ( ρ ) d ρ \zeta(\tau)-\zeta(\tau')=-\int_{\tau}^{\tau'}\dot{\zeta}(\rho)\,d\rho ζ ( τ ) − ζ ( τ ′ ) = − ∫ τ τ ′ ζ ˙ ( ρ ) d ρ ; letting τ ′ ↑ τ ˉ \tau'\uparrow\bar{\tau} τ ′ ↑ τ ˉ , with ζ ( τ ′ ) → 0 \zeta(\tau')\to0 ζ ( τ ′ ) → 0 by (71), gives (61) with s = σ s=\sigma s = σ and h = Q v ˉ h=Q\bar{v} h = Q v ˉ , whose drifts are P J ⊥ Q v ˉ = S J v ˉ P_J^\perp Q\bar{v}=S_J\bar{v} P J ⊥ Q v ˉ = S J v ˉ . Lemmas B.6 and B.7 therefore apply to ζ \zeta ζ . Every recurring event value vanishes at the iterates of its own events, so Lemma B.12(ii) gives ϕ ( x ( τ ) ) = ⟨ a , ζ ( τ ) ⟩ \phi(x(\tau))=\langle a,\zeta(\tau)\rangle ϕ ( x ( τ )) = ⟨ a , ζ ( τ )⟩ at every τ \tau τ on the tail, for every recurring event value ϕ \phi ϕ and its gradient a ∈ Φ a\in\Phi a ∈ Φ . Finally, by Lemma B.7, every a ∈ Φ a\in\Phi a ∈ Φ admits a time τ a < τ ˉ \tau_a<\bar{\tau} τ a < τ ˉ such that τ ↦ ⟨ a , ζ ( τ ) ⟩ \tau\mapsto\langle a,\zeta(\tau)\rangle τ ↦ ⟨ a , ζ ( τ )⟩ either vanishes on all of [ τ a , τ ˉ ) [\tau_a,\bar{\tau}) [ τ a , τ ˉ ) or has no zero there. By Lemma B.9 with D = ζ D=\zeta D = ζ , no piece of the tail starts after the largest of these times, which contradicts τ ( k ) ↑ τ ˉ \tau^{(k)}\uparrow\bar{\tau} τ ( k ) ↑ τ ˉ .
No accumulation of breakpoints at infinity
Proposition B.3. The slow solution has finitely many breakpoints.
We prove this below. Write y ( τ ) = x ( τ ) − τ d s y(\tau)=x(\tau)-\tau d^s y ( τ ) = x ( τ ) − τ d s , with τ 0 \tau_0 τ 0 and y ˉ \bar{y} y ˉ as in Proposition B.1, and let
e ( τ ) : = y ( τ ) − y ˉ , e(\tau):=y(\tau)-\bar{y}, e ( τ ) := y ( τ ) − y ˉ ,
so that e ( τ ) → 0 e(\tau)\to0 e ( τ ) → 0 as τ → + ∞ \tau\to+\infty τ → + ∞ , and the right derivative of e e e is e ˙ ( τ ) = d ( x ( τ ) ) − d s \dot{e}(\tau)=d(x(\tau))-d^s e ˙ ( τ ) = d ( x ( τ )) − d s .
Lemma B.13. Let J J J be an index set whose J J J -pieces contain arbitrarily large times. Then, at every time τ \tau τ of every J J J -piece,
e ( τ ) ∈ ker A J and e ˙ ( τ ) = − P J ⊥ Q e ( τ ) = − S J e ( τ ) . e(\tau)\in\ker A_J
\qquad\text{and}\qquad
\dot{e}(\tau)=-P_J^\perp Qe(\tau)=-S_Je(\tau). e ( τ ) ∈ ker A J and e ˙ ( τ ) = − P J ⊥ Q e ( τ ) = − S J e ( τ ) .
Proof. By Lemma B.8(i) and Proposition B.1(ii), at the times τ ≥ τ 0 \tau\ge\tau_0 τ ≥ τ 0 of the J J J -pieces we have J ⊆ E ( x ( τ ) ) ⊆ I 0 J\subseteq\mathcal{E}(x(\tau))\subseteq I_0 J ⊆ E ( x ( τ )) ⊆ I 0 and [ A x ( τ ) ] J = b J [Ax(\tau)]_J=b_J [ A x ( τ ) ] J = b J . Since J ⊆ I 0 J\subseteq I_0 J ⊆ I 0 , we have [ A d s ] J = 0 [Ad^s]_J=0 [ A d s ] J = 0 , so [ A y ( τ ) ] J = [ A x ( τ ) ] J = b J [Ay(\tau)]_J=[Ax(\tau)]_J=b_J [ A y ( τ ) ] J = [ A x ( τ ) ] J = b J at every time of every J J J -piece. Letting τ → + ∞ \tau\to+\infty τ → + ∞ gives [ A y ˉ ] J = b J [A\bar{y}]_J=b_J [ A y ˉ ] J = b J , hence e ( τ ) ∈ ker A J e(\tau)\in\ker A_J e ( τ ) ∈ ker A J there.
Since Q d s = 0 Qd^s=0 Q d s = 0 , we have ∇ x f ( x ( τ ) ) = ∇ x f ( y ( τ ) ) \nabla_x f(x(\tau))=\nabla_x f(y(\tau)) ∇ x f ( x ( τ )) = ∇ x f ( y ( τ )) and ∇ x f ( y ˉ ) \nabla_x f(\bar{y}) ∇ x f ( y ˉ ) as τ → + ∞ \tau\to+\infty τ → + ∞ . Along the J J J -pieces, d ( x ( τ ) ) = − P J ⊥ ∇ x f ( y ( τ ) ) d(x(\tau))=-P_J^\perp\nabla_x f(y(\tau)) d ( x ( τ )) = − P J ⊥ ∇ x f ( y ( τ )) , by (65), converges to − P J ⊥ ∇ x f ( y ˉ ) -P_J^\perp\nabla_x f(\bar{y}) − P J ⊥ ∇ x f ( y ˉ ) , and to d s d^s d s by Proposition B.1(i), so d s = − P J ⊥ ∇ x f ( y ˉ ) d^s=-P_J^\perp\nabla_x f(\bar{y}) d s = − P J ⊥ ∇ x f ( y ˉ ) . Hence, at every time τ \tau τ of every J J J -piece, e ˙ ( τ ) = − P J ⊥ ( ∇ x f ( y ( τ ) ) − ∇ x f ( y ˉ ) ) = − P J ⊥ Q e ( τ ) \dot{e}(\tau)=-P_J^\perp(\nabla_x f(y(\tau))-\nabla_x f(\bar{y}))=-P_J^\perp Qe(\tau) e ˙ ( τ ) = − P J ⊥ ( ∇ x f ( y ( τ )) − ∇ x f ( y ˉ )) = − P J ⊥ Q e ( τ ) , and P J ⊥ Q e ( τ ) = S J e ( τ ) P_J^\perp Qe(\tau)=S_Je(\tau) P J ⊥ Q e ( τ ) = S J e ( τ ) because e ( τ ) ∈ ker A J e(\tau)\in\ker A_J e ( τ ) ∈ ker A J .
Suppose there are infinitely many breakpoints, so that τ ( k ) → + ∞ \tau^{(k)}\to+\infty τ ( k ) → + ∞ by Proposition B.2. We fix the events and the tail as in Appendix B.2.2, discarding further pieces so that every piece of the tail starts after τ 0 \tau_0 τ 0 . Every recurring active set then satisfies the hypothesis of Lemma B.13, and τ ↦ ∥ e ( τ ) ∥ 2 \tau\mapsto\lVert e(\tau)\rVert_2 τ ↦ ∥ e ( τ ) ∥ 2 is nonincreasing on the tail by Proposition B.1(iii).
Lemma B.14. Let W ⊆ R n W \subseteq\mathbb{R}^n W ⊆ R n be a subspace, let H ∈ R n × n \mathbf{H} \in\mathbb{R}^{n\times n} H ∈ R n × n be symmetric, and for recurring J J J let P J W P_J^W P J W denote the orthogonal projection onto W ∩ ker A J W \cap\ker A_J W ∩ ker A J . Let τ ↦ ξ ( τ ) \tau\mapsto\xi(\tau) τ ↦ ξ ( τ ) be defined and continuous on the tail, continuously differentiable on the interior of each piece and from the right at its left endpoint, and nonzero at every time of the tail. Suppose that on every J J J -piece of the tail
ξ ( τ ) ∈ W ∩ ker A J and ξ ˙ = − P J W H ξ , (78) \xi(\tau) \in W \cap\ker A_J \qquad\text{and} \qquad\dot{\xi}=-P_J^W\mathbf{H}\xi, \tag*{(78)} ξ ( τ ) ∈ W ∩ ker A J and ξ ˙ = − P J W H ξ , (78) and that at every breakpoint τ ( k ) \tau^{(k)} τ ( k ) of the tail
∥ ξ ˙ ( τ ( k ) ) ∥ 2 ≤ lim τ ↑ τ ( k ) ∥ ξ ˙ ( τ ) ∥ 2 . (79) \left\|\dot{\xi}\left(\tau^{(k)}\right)\right\|_2 \leq\lim_{\tau\mathrel{\uparrow} \tau^{(k)}}\left\|\dot{\xi}(\tau)\right\|_2. \tag*{(79)} ξ ˙ ( τ ( k ) ) 2 ≤ τ ↑ τ ( k ) lim ξ ˙ ( τ ) 2 . (79) Then:
The unit vector θ ( τ ) : = ξ ( τ ) / ∥ ξ ( τ ) ∥ 2 \boldsymbol{\theta}(\tau):=\xi(\tau)/\|\xi(\tau)\|_2 θ ( τ ) := ξ ( τ ) /∥ ξ ( τ ) ∥ 2 converges as τ → + ∞ \tau\to+\infty τ → + ∞ to a unit vector θ ˉ \bar{\boldsymbol{\theta}} θ ˉ , and there exists μ ∈ R \mu\in\mathbb{R} μ ∈ R such that θ ˉ ∈ W ∩ ker A J \bar{\boldsymbol{\theta}}\in W\cap\ker A_J θ ˉ ∈ W ∩ ker A J and P J W H θ ˉ = μ θ ˉ P_J^W\mathbf{H}\bar{\boldsymbol{\theta}}=\mu\bar{\boldsymbol{\theta}} P J W H θ ˉ = μ θ ˉ for every recurring J J J .
From some time on, γ ( τ ) : = ⟨ ξ ( τ ) , θ ˉ ⟩ \gamma(\tau):=\langle\xi(\tau),\bar{\boldsymbol{\theta}}\rangle γ ( τ ) := ⟨ ξ ( τ ) , θ ˉ ⟩ is positive, and from that time the function ξ + : = ξ / γ − θ ˉ \xi^+:=\xi/\gamma-\bar{\boldsymbol{\theta}} ξ + := ξ / γ − θ ˉ is continuous, continuously differentiable on each piece, takes its values in W + : = { s ∈ W : ⟨ s , θ ˉ ⟩ = 0 } W^+:=\{s\in W:\langle s,\bar{\boldsymbol{\theta}}\rangle=0\} W + := { s ∈ W : ⟨ s , θ ˉ ⟩ = 0 } , tends to 0 \mathbf{0} 0 , and satisfies (78) and (79) with W + W^+ W + and H − μ I n × n \mathbf{H}-\mu I_{n\times n} H − μ I n × n for W W W and H \mathbf{H} H .
Proof. For recurring J J J , let H J \mathbf{H}_J H J denote the restriction of P J W H P_J^W\mathbf{H} P J W H to W ∩ ker A J W\cap\ker A_J W ∩ ker A J . It is a symmetric linear map from that subspace into itself, with ∥ H J ∥ 2 ≤ ∥ H ∥ 2 \|\mathbf{H}_J\|_2\leq\|\mathbf{H}\|_2 ∥ H J ∥ 2 ≤ ∥ H ∥ 2 and ⟨ s , H J s ⟩ = ⟨ s , H s ⟩ \langle s,\mathbf{H}_Js\rangle=\langle s,\mathbf{H}s\rangle ⟨ s , H J s ⟩ = ⟨ s , H s ⟩ there. Let ϱ : = ⟨ θ , H θ ⟩ \varrho:=\langle\boldsymbol{\theta},\mathbf{H}\boldsymbol{\theta}\rangle ϱ := ⟨ θ , H θ ⟩ and ψ : = ∥ θ ˙ ∥ 2 2 \psi:=\|\dot{\boldsymbol{\theta}}\|_2^2 ψ := ∥ θ ˙ ∥ 2 2 , the dot denoting a right derivative at the left endpoint of a piece. On each J J J -piece, (78) is a linear differential equation with constant coefficients, so ξ \xi ξ is real analytic there, and so are θ \boldsymbol{\theta} θ , ϱ \varrho ϱ , and ψ \psi ψ , because ξ \xi ξ does not vanish.
Step 1. We show that ϱ \varrho ϱ converges to some μ ∈ R \mu\in\mathbb{R} μ ∈ R and that ψ ( τ ) → 0 \psi(\tau)\to0 ψ ( τ ) → 0 as τ → + ∞ \tau\to+\infty τ → + ∞ . On a J J J -piece, (78) gives ⟨ ξ , ξ ˙ ⟩ = − ⟨ ξ , H ξ ⟩ \langle\xi,\dot{\xi}\rangle=-\langle\xi,\mathbf{H}\xi\rangle ⟨ ξ , ξ ˙ ⟩ = − ⟨ ξ , H ξ ⟩ , so d d τ ∥ ξ ∥ 2 = − ϱ ∥ ξ ∥ 2 \frac{d}{d\tau}\|\xi\|_2=-\varrho\|\xi\|_2 d τ d ∥ ξ ∥ 2 = − ϱ ∥ ξ ∥ 2 and the vector of W ∩ ker A J W\cap\ker A_J W ∩ ker A J
θ ˙ = − H J θ + ϱ θ (80) \dot{\boldsymbol{\theta}}=-\mathbf{H}_J\boldsymbol{\theta}+\varrho\boldsymbol{\theta} \tag*{(80)} θ ˙ = − H J θ + ϱ θ (80)
is orthogonal to θ \boldsymbol{\theta} θ . Hence ϱ ˙ = 2 ⟨ H θ , θ ˙ ⟩ = 2 ⟨ H J θ , θ ˙ ⟩ = 2 ⟨ ϱ θ − θ ˙ , θ ˙ ⟩ = − 2 ψ \dot{\varrho}=2\langle\mathbf{H}\boldsymbol{\theta},\dot{\boldsymbol{\theta}}\rangle=2\langle\mathbf{H}_J\boldsymbol{\theta},\dot{\boldsymbol{\theta}}\rangle=2\langle\varrho\boldsymbol{\theta}-\dot{\boldsymbol{\theta}},\dot{\boldsymbol{\theta}}\rangle=-2\psi ϱ ˙ = 2 ⟨ H θ , θ ˙ ⟩ = 2 ⟨ H J θ , θ ˙ ⟩ = 2 ⟨ ϱ θ − θ ˙ , θ ˙ ⟩ = − 2 ψ , and
ψ = ∥ H J θ ∥ 2 2 − ϱ 2 = ∥ ξ ˙ ∥ 2 2 ∥ ξ ∥ 2 2 − ϱ 2 . (81) \psi=\|\mathbf{H}_J\boldsymbol{\theta}\|_2^2-\varrho^2=\frac{\|\dot{\xi}\|_2^2}{\|\xi\|_2^2}-\varrho^2. \tag*{(81)} ψ = ∥ H J θ ∥ 2 2 − ϱ 2 = ∥ ξ ∥ 2 2 ∥ ξ ˙ ∥ 2 2 − ϱ 2 . (81)
Differentiating (80) gives θ ¨ = − H J θ ˙ + ϱ ˙ θ + ϱ θ ˙ \ddot{\boldsymbol{\theta}}=-\mathbf{H}_J\dot{\boldsymbol{\theta}}+\dot{\varrho}\boldsymbol{\theta}+\varrho\dot{\boldsymbol{\theta}} θ ¨ = − H J θ ˙ + ϱ ˙ θ + ϱ θ ˙ , so ψ ˙ = 2 ⟨ θ ˙ , θ ¨ ⟩ = − 2 ⟨ θ ˙ , H J θ ˙ ⟩ + 2 ϱ ψ ≤ 4 ∥ H ∥ 2 ψ \dot{\psi}=2\langle\dot{\boldsymbol{\theta}},\ddot{\boldsymbol{\theta}}\rangle=-2\langle\dot{\boldsymbol{\theta}},\mathbf{H}_J\dot{\boldsymbol{\theta}}\rangle+2\varrho\psi\leq4\|\mathbf{H}\|_2\psi ψ ˙ = 2 ⟨ θ ˙ , θ ¨ ⟩ = − 2 ⟨ θ ˙ , H J θ ˙ ⟩ + 2 ϱ ψ ≤ 4∥ H ∥ 2 ψ on the piece. At a breakpoint, ξ \xi ξ and ϱ \varrho ϱ are continuous and ∥ ξ ˙ ∥ 2 \|\dot{\xi}\|_2 ∥ ξ ˙ ∥ 2 does not increase, by (79) , so ψ \psi ψ does not increase, by (81) . Hence τ ↦ e − 4 ∥ H ∥ 2 τ ψ ( τ ) \tau\mapsto e^{-4\|\mathbf{H}\|_2\tau}\psi(\tau) τ ↦ e − 4∥ H ∥ 2 τ ψ ( τ ) is nonincreasing on the tail. The function ϱ \varrho ϱ is continuous, nonincreasing, and bounded below by − ∥ H ∥ 2 -\|\mathbf{H}\|_2 − ∥ H ∥ 2 , so it converges to some μ ∈ R \mu\in\mathbb{R} μ ∈ R , and
∫ τ 1 + ∞ ψ ( ρ ) d ρ = 1 2 ( ϱ ( τ 1 ) − μ ) < + ∞ \int_{\tau_1}^{+\infty}\psi(\rho)\,d\rho=\frac{1}{2}\bigl(\varrho(\tau_1)-\mu\bigr)<+\infty ∫ τ 1 + ∞ ψ ( ρ ) d ρ = 2 1 ( ϱ ( τ 1 ) − μ ) < + ∞
for every τ 1 \tau_1 τ 1 on the tail. For ρ ∈ [ τ − 1 , τ ] \rho\in[\tau-1,\tau] ρ ∈ [ τ − 1 , τ ] we have ψ ( ρ ) ≥ e − 4 ∥ H ∥ 2 ψ ( τ ) \psi(\rho)\geq e^{-4\|\mathbf{H}\|_2}\psi(\tau) ψ ( ρ ) ≥ e − 4∥ H ∥ 2 ψ ( τ ) , so ψ ( τ ) ≤ e 4 ∥ H ∥ 2 ∫ τ − 1 τ ψ ( ρ ) d ρ → 0 \psi(\tau)\leq e^{4\|\mathbf{H}\|_2}\int_{\tau-1}^{\tau}\psi(\rho)\,d\rho\to0 ψ ( τ ) ≤ e 4∥ H ∥ 2 ∫ τ − 1 τ ψ ( ρ ) d ρ → 0 .
Step 2. We show that 0 ≤ ϱ − μ ≤ κ 2 ψ 0\leq\varrho-\mu\leq\kappa^2\psi 0 ≤ ϱ − μ ≤ κ 2 ψ on [ τ 1 , + ∞ ) [\tau_1,+\infty) [ τ 1 , + ∞ ) , for some κ > 0 \kappa>0 κ > 0 and some time τ 1 \tau_1 τ 1 on the tail. Let J J J be recurring. By (80) , ∥ θ ˙ ∥ 2 2 = ∥ ( H J − ϱ ) θ ∥ 2 2 \|\dot{\boldsymbol{\theta}}\|_2^2=\|(\mathbf{H}_J-\varrho)\boldsymbol{\theta}\|_2^2 ∥ θ ˙ ∥ 2 2 = ∥ ( H J − ϱ ) θ ∥ 2 2 is at least the distance from ϱ \varrho ϱ to the spectrum of H J \mathbf{H}_J H J . Since ψ → 0 \psi\to0 ψ → 0 , ϱ → μ \varrho\to\mu ϱ → μ , and the J J J -pieces contain arbitrarily large times, μ \mu μ is an eigenvalue of H J \mathbf{H}_J H J . There are finitely many recurring active sets, so we can fix δ > 0 \delta>0 δ > 0 smaller than the distance from μ \mu μ to the other eigenvalues of every H J \mathbf{H}_J H J with J J J recurring. If μ \mu μ is the only eigenvalue of every such H J H_J H J , we set δ : = 1 \delta:= 1 δ := 1 . Let τ 1 \tau_1 τ 1 be a time on the tail after which ϱ − μ ≤ δ / 2 \varrho-\mu\le\delta/2 ϱ − μ ≤ δ /2 , consider a time τ ≥ τ 1 \tau\ge\tau_1 τ ≥ τ 1 of a J J J -piece, and let θ μ \theta_\mu θ μ be the orthogonal projection of θ ( τ ) \theta(\tau) θ ( τ ) onto the μ \mu μ -eigenspace of H J H_J H J . The other eigenvalues of H J H_J H J lie at distance at least δ / 2 \delta/2 δ /2 from ϱ \varrho ϱ , so expanding θ \theta θ in an orthonormal eigenbasis of H J H_J H J gives
ψ = ∥ ( H J − ϱ ) θ ∥ 2 2 ≥ δ 2 4 ∥ θ − θ μ ∥ 2 2 , ϱ − μ = ⟨ θ − θ μ , ( H J − μ ) ( θ − θ μ ) ⟩ ≤ 2 ∥ H ∥ 2 ∥ θ − θ μ ∥ 2 . \psi=\lVert(H_J-\varrho)\theta\rVert_2^2\ge\frac{\delta^2}{4}\lVert\theta-\theta_\mu\rVert_2^2,\qquad\varrho-\mu=\langle\theta-\theta_\mu,(H_J-\mu)(\theta-\theta_\mu)\rangle\le2\lVert H\rVert_2\lVert\theta-\theta_\mu\rVert_2. ψ = ∥( H J − ϱ ) θ ∥ 2 2 ≥ 4 δ 2 ∥ θ − θ μ ∥ 2 2 , ϱ − μ = ⟨ θ − θ μ , ( H J − μ ) ( θ − θ μ )⟩ ≤ 2 ∥ H ∥ 2 ∥ θ − θ μ ∥ 2 .
The claim holds with κ 2 : = 8 ( 1 + ∥ H ∥ 2 ) / δ 2 \kappa^2:=8(1+\lVert H\rVert_2)/\delta^2 κ 2 := 8 ( 1 + ∥ H ∥ 2 ) / δ 2 , and ϱ ≥ μ \varrho\ge\mu ϱ ≥ μ because ϱ \varrho ϱ is nonincreasing with limit μ \mu μ .
Step 3. We prove (i). If ϱ ( τ 2 ) = μ \varrho(\tau_2)=\mu ϱ ( τ 2 ) = μ at some time τ 2 ≥ τ 1 \tau_2\ge\tau_1 τ 2 ≥ τ 1 , then ϱ = μ \varrho=\mu ϱ = μ on [ τ 2 , + ∞ ) [\tau_2,+\infty) [ τ 2 , + ∞ ) , so ψ = − ϱ ˙ / 2 \psi=-\dot{\varrho}/2 ψ = − ϱ ˙ /2 vanishes there and θ \theta θ is constant. Otherwise ϱ > μ \varrho>\mu ϱ > μ on the tail, and on each piece after τ 1 \tau_1 τ 1 , Step 2 gives d d τ ϱ − μ = − ψ / ϱ − μ ≤ − ψ / κ = − ∥ θ ˙ ∥ 2 / κ \frac{d}{d\tau}\sqrt{\varrho-\mu}=-\psi/\sqrt{\varrho-\mu}\le-\sqrt{\psi}/\kappa=-\lVert\dot{\theta}\rVert_2/\kappa d τ d ϱ − μ = − ψ / ϱ − μ ≤ − ψ / κ = − ∥ θ ˙ ∥ 2 / κ . The function ϱ − μ \sqrt{\varrho-\mu} ϱ − μ is continuous, so integrating gives ∫ τ 1 + ∞ ∥ θ ˙ ( ρ ) ∥ 2 d ρ ≤ κ ϱ ( τ 1 ) − μ \int_{\tau_1}^{+\infty}\lVert\dot{\theta}(\rho)\rVert_2\,d\rho\le\kappa\sqrt{\varrho(\tau_1)-\mu} ∫ τ 1 + ∞ ∥ θ ˙ ( ρ ) ∥ 2 d ρ ≤ κ ϱ ( τ 1 ) − μ . In both cases, θ ( τ ) \theta(\tau) θ ( τ ) converges to a unit vector θ ˉ \bar{\theta} θ ˉ . For recurring J J J , the directions on the J J J -pieces lie in the closed subspace W ∩ ker A J W\cap\ker A_J W ∩ ker A J , so θ ˉ \bar{\theta} θ ˉ does, and along the J J J -pieces ( H J − ϱ ) θ = − θ ˙ → 0 (H_J-\varrho)\theta=-\dot{\theta}\to0 ( H J − ϱ ) θ = − θ ˙ → 0 by Step 1. Hence H J θ ˉ = μ θ ˉ H_J\bar{\theta}=\mu\bar{\theta} H J θ ˉ = μ θ ˉ , that is, P J W H θ ˉ = μ θ ˉ P_J^W H\bar{\theta}=\mu\bar{\theta} P J W H θ ˉ = μ θ ˉ .
Step 4. We prove (ii). Since ⟨ θ , θ ˉ ⟩ → 1 \langle\theta,\bar{\theta}\rangle\to1 ⟨ θ , θ ˉ ⟩ → 1 , the function γ = ∥ ξ ∥ 2 ⟨ θ , θ ˉ ⟩ \gamma=\lVert\xi\rVert_2\langle\theta,\bar{\theta}\rangle γ = ∥ ξ ∥ 2 ⟨ θ , θ ˉ ⟩ is positive from some time on. From that time on, ξ + = θ / ⟨ θ , θ ˉ ⟩ − θ ˉ \xi^+=\theta/\langle\theta,\bar{\theta}\rangle-\bar{\theta} ξ + = θ / ⟨ θ , θ ˉ ⟩ − θ ˉ is continuous, continuously differentiable on each piece, orthogonal to θ ˉ \bar{\theta} θ ˉ , and tends to 0 0 0 , and on every J J J -piece it lies in W + ∩ ker A J W^+\cap\ker A_J W + ∩ ker A J , since ξ \xi ξ and θ ˉ \bar{\theta} θ ˉ lie in W ∩ ker A J W\cap\ker A_J W ∩ ker A J . Consider a J J J -piece and write P : = P J W P:=P_J^W P := P J W . Both P P P and H H H are symmetric and P s = s Ps=s P s = s for s ∈ { ξ , ξ + , θ ˉ } s\in\{\xi,\xi^+,\bar{\theta}\} s ∈ { ξ , ξ + , θ ˉ } , so (i) gives
⟨ P H s , θ ˉ ⟩ = ⟨ s , P H θ ˉ ⟩ = μ ⟨ s , θ ˉ ⟩ for s ∈ { ξ , ξ + } . (82) \langle PHs,\bar{\theta}\rangle=\langle s,PH\bar{\theta}\rangle=\mu\langle s,\bar{\theta}\rangle\qquad\text{for }s\in\{\xi,\xi^+\}. \tag*{(82)} ⟨ P H s , θ ˉ ⟩ = ⟨ s , P H θ ˉ ⟩ = μ ⟨ s , θ ˉ ⟩ for s ∈ { ξ , ξ + } . (82)
With s = ξ s=\xi s = ξ , this identity and (78) give γ ˙ = − μ γ \dot{\gamma}=-\mu\gamma γ ˙ = − μ γ , and then
ξ ˙ + = ( ξ ˙ − γ ˙ ξ ) / γ = − P H ( θ ˉ + ξ + ) + μ ( θ ˉ + ξ + ) = − ( P H − μ ) ξ + . \dot{\xi}^+=(\dot{\xi}-\dot{\gamma}\xi)/\gamma=-PH(\bar{\theta}+\xi^+)+\mu(\bar{\theta}+\xi^+)=-(PH-\mu)\xi^+. ξ ˙ + = ( ξ ˙ − γ ˙ ξ ) / γ = − P H ( θ ˉ + ξ + ) + μ ( θ ˉ + ξ + ) = − ( P H − μ ) ξ + .
With s = ξ + s=\xi^+ s = ξ + , identity (76) shows that P H ξ + PH\xi^+ P H ξ + is orthogonal to θ ˉ \bar{\theta} θ ˉ , so it lies in W + ∩ ker A J W^+\cap\ker A_J W + ∩ ker A J . Since this subspace lies in W ∩ ker A J W\cap\ker A_J W ∩ ker A J , we have P J W + = P J W + P P_J^{W^+}=P_J^{W^+}P P J W + = P J W + P , and therefore P J W + H ξ + = P J W + P H ξ + = P H ξ + P_J^{W^+}H\xi^+=P_J^{W^+}PH\xi^+=PH\xi^+ P J W + H ξ + = P J W + P H ξ + = P H ξ + . Hence ξ ˙ + = − P J W + ( H − μ I n × n ) ξ + \dot{\xi}^+=-P_J^{W^+}(H-\mu I_{n\times n})\xi^+ ξ ˙ + = − P J W + ( H − μ I n × n ) ξ + , which is (78) for ξ + \xi^+ ξ + .
It remains to prove (79) for ξ + \xi^+ ξ + . On a piece, ξ ˙ = γ ˙ ( θ ˉ + ξ + ) + γ ξ ˙ + = γ ( ξ ˙ + − μ ( θ ˉ + ξ + ) ) \dot{\xi}=\dot{\gamma}(\bar{\theta}+\xi^+)+\gamma\dot{\xi}^+=\gamma(\dot{\xi}^+-\mu(\bar{\theta}+\xi^+)) ξ ˙ = γ ˙ ( θ ˉ + ξ + ) + γ ξ ˙ + = γ ( ξ ˙ + − μ ( θ ˉ + ξ + )) , where ξ ˙ + \dot{\xi}^+ ξ ˙ + is orthogonal to θ ˉ \bar{\theta} θ ˉ and ⟨ ξ ˙ + , ξ + ⟩ = − ⟨ ξ + , H ξ + ⟩ + μ ∥ ξ + ∥ 2 2 \langle\dot{\xi}^+,\xi^+\rangle=-\langle\xi^+,H\xi^+\rangle+\mu\lVert\xi^+\rVert_2^2 ⟨ ξ ˙ + , ξ + ⟩ = − ⟨ ξ + , H ξ + ⟩ + μ ∥ ξ + ∥ 2 2 . Therefore
∥ ξ ˙ ∥ 2 2 = γ 2 ( ∥ ξ ˙ + ∥ 2 2 + 2 μ ⟨ ξ + , H ξ + ⟩ − 2 μ 2 ∥ ξ + ∥ 2 2 + μ 2 ∥ θ ˉ + ξ + ∥ 2 2 ) . \lVert\dot{\xi}\rVert_2^2=\gamma^2\left(\lVert\dot{\xi}^+\rVert_2^2+2\mu\langle\xi^+,H\xi^+\rangle-2\mu^2\lVert\xi^+\rVert_2^2+\mu^2\lVert\bar{\theta}+\xi^+\rVert_2^2\right). ∥ ξ ˙ ∥ 2 2 = γ 2 ( ∥ ξ ˙ + ∥ 2 2 + 2 μ ⟨ ξ + , H ξ + ⟩ − 2 μ 2 ∥ ξ + ∥ 2 2 + μ 2 ∥ θ ˉ + ξ + ∥ 2 2 ) .
At a breakpoint, γ \gamma γ and ξ + \xi^+ ξ + are continuous, so every term on the right-hand side except ∥ ξ ˙ + ∥ 2 2 \lVert\dot{\xi}^+\rVert_2^2 ∥ ξ ˙ + ∥ 2 2 is continuous, and ∥ ξ + ∥ 2 \lVert\xi^+\rVert_2 ∥ ξ + ∥ 2 does not increase because ∥ ξ ˙ ∥ 2 \lVert\dot{\xi}\rVert_2 ∥ ξ ˙ ∥ 2 does not.
□ \square □
Proof of Proposition B.3. We now derive a contradiction in three steps.
Step 1. We show that there exist an integer r r r with 0 ≤ r ≤ n 0\le r\le n 0 ≤ r ≤ n , unit vectors θ ˉ 1 , … , θ ˉ r \bar{\theta}_1,\ldots,\bar{\theta}_r θ ˉ 1 , … , θ ˉ r , a time τ ′ \tau' τ ′ on the tail, and functions γ 0 , … , γ r − 1 \gamma_0,\ldots,\gamma_{r-1} γ 0 , … , γ r − 1 , positive on [ τ ′ , + ∞ ) [\tau',+\infty) [ τ ′ , + ∞ ) , such that
e ( τ ) = ∑ j = 1 r γ 0 ( τ ) ⋯ γ j − 1 ( τ ) θ ˉ j for every τ ≥ τ ′ , γ j ( τ ) ⟶ 0 as τ → + ∞ , for 1 ≤ j < r . (83) e(\tau)=\sum_{j=1}^{r}\gamma_0(\tau)\cdots\gamma_{j-1}(\tau)\bar{\theta}_j\qquad\text{for every }\tau\ge\tau',\qquad\gamma_j(\tau)\longrightarrow0\quad\text{as }\tau\to+\infty,\quad\text{for }1\le j<r. \tag*{(83)} e ( τ ) = j = 1 ∑ r γ 0 ( τ ) ⋯ γ j − 1 ( τ ) θ ˉ j for every τ ≥ τ ′ , γ j ( τ ) ⟶ 0 as τ → + ∞ , for 1 ≤ j < r . (83)
For r = 0 r=0 r = 0 the sum is empty and e e e vanishes on [ τ ′ , + ∞ ) [\tau',+\infty) [ τ ′ , + ∞ ) . Let ξ \xi ξ satisfy (78) . If ξ \xi ξ vanishes at a time τ 1 \tau_1 τ 1 , then it vanishes on [ τ 1 , + ∞ ) [\tau_1,+\infty) [ τ 1 , + ∞ ) . Indeed, equation (78) is linear on each piece, so ξ \xi ξ vanishes on the rest of the piece containing τ 1 \tau_1 τ 1 and, by continuity, at the start of the next piece. Hence ξ \xi ξ either vanishes from some time on or vanishes at no time.
Set ξ 0 : = e \xi_0:=e ξ 0 := e and W 0 : = R n W_0:=\mathbb{R}^n W 0 := R n . Then P W 0 ⊥ = P J ⊥ P_{W_0}^{\perp}=P_J^{\perp} P W 0 ⊥ = P J ⊥ , so ξ 0 \xi_0 ξ 0 satisfies (78) with H = Q H=Q H = Q by Lemma B.13, and it satisfies (79) because ∥ e ˙ ∥ 2 \lVert\dot e\rVert_2 ∥ e ˙ ∥ 2 is nonincreasing on the tail. If ξ 0 \xi_0 ξ 0 vanishes from some time on, the claim holds with r = 0 r=0 r = 0 . Otherwise Lemma B.14 gives a unit vector θ ˉ 1 \bar\theta_1 θ ˉ 1 , a number μ 1 \mu_1 μ 1 , a function γ 0 \gamma_0 γ 0 , positive from some time on, and the function ξ 1 : = ξ 0 + \xi_1:=\xi_0^+ ξ 1 := ξ 0 + , with values in the ( n − 1 ) (n-1) ( n − 1 ) -dimensional subspace W 1 W_1 W 1 orthogonal to θ ˉ 1 \bar\theta_1 θ ˉ 1 , satisfying (78) and (79) with W 1 W_1 W 1 and Q − μ 1 I n × n Q-\mu_1 I_{n\times n} Q − μ 1 I n × n . Discarding the pieces that start before γ 0 \gamma_0 γ 0 becomes positive, we repeat the argument with ξ 1 \xi_1 ξ 1 . After j j j repetitions,
e = γ 0 ( θ ˉ 1 + γ 1 ( θ ˉ 2 + ⋯ + γ j − 1 ( θ ˉ j + ξ j ) ) ) e=\gamma_0\left(\bar\theta_1+\gamma_1\left(\bar\theta_2+\cdots+\gamma_{j-1}\left(\bar\theta_j+\xi_j\right)\right)\right) e = γ 0 ( θ ˉ 1 + γ 1 ( θ ˉ 2 + ⋯ + γ j − 1 ( θ ˉ j + ξ j ) ) )
with ξ j \xi_j ξ j taking its values in a subspace W j W_j W j of dimension n − j n-j n − j . Since W n = { 0 } W_n=\{0\} W n = { 0 } , some ξ r \xi_r ξ r with r ≤ n r\le n r ≤ n vanishes from a time τ ′ \tau' τ ′ on, and (83) holds, because γ j = ⟨ ξ j , θ ˉ j + 1 ⟩ → 0 \gamma_j=\langle\xi_j,\bar\theta_{j+1}\rangle\to0 γ j = ⟨ ξ j , θ ˉ j + 1 ⟩ → 0 , as ξ j → 0 \xi_j\to0 ξ j → 0 for j ≥ 1 j\ge1 j ≥ 1 by Lemma B.14(ii).
Step 2. We show that ϕ ( x ( τ ) ) = ⟨ a , e ( τ ) ⟩ \phi(x(\tau))=\langle a,e(\tau)\rangle ϕ ( x ( τ )) = ⟨ a , e ( τ )⟩ at every τ \tau τ on the tail, for every recurring event value ϕ \phi ϕ and its gradient a ∈ Φ a\in\Phi a ∈ Φ . Let i i i be a constraint that recurs in enter events. At the iterates of these events, i ∈ E ( x ( k + 1 ) ) ⊆ I 0 i\in\mathcal{E}(x^{(k+1)})\subseteq I_0 i ∈ E ( x ( k + 1 ) ) ⊆ I 0 by Proposition B.1(ii), so [ A d s ] i = 0 [A d^s]_i=0 [ A d s ] i = 0 and [ A x ( τ ) ] i − b i = [ A y ( τ ) ] i − b i [A x(\tau)]_i-b_i=[A y(\tau)]_i-b_i [ A x ( τ ) ] i − b i = [ A y ( τ ) ] i − b i for every τ \tau τ . This affine function of y ( τ ) y(\tau) y ( τ ) vanishes at the iterates of the enter events of i i i , whose times tend to infinity, so it vanishes at y ˉ \bar y y ˉ and equals ⟨ a , e ( τ ) ⟩ \langle a,e(\tau)\rangle ⟨ a , e ( τ )⟩ . Now let ( B , i ) (B,i) ( B , i ) be a recurring pair of a leave event. Since Q d s = 0 Qd^s=0 Q d s = 0 , formula (66) gives [ p B ( x ( τ ) ) ] i = [ p B ( y ( τ ) ) ] i [p_B(x(\tau))]_i=[p_B(y(\tau))]_i [ p B ( x ( τ )) ] i = [ p B ( y ( τ )) ] i , an affine function of y ( τ ) y(\tau) y ( τ ) that vanishes at the iterates of the leave events of ( B , i ) (B,i) ( B , i ) , hence at y ˉ \bar y y ˉ , and equals ⟨ a , e ( τ ) ⟩ \langle a,e(\tau)\rangle ⟨ a , e ( τ )⟩ .
Step 3. Fix a ∈ Φ a\in\Phi a ∈ Φ and put α j : = ⟨ a , θ ˉ j ⟩ \alpha_j:=\langle a,\bar\theta_j\rangle α j := ⟨ a , θ ˉ j ⟩ . By (83) , ⟨ a , e ( τ ) ⟩ = ∑ j = 1 r γ 0 ( τ ) ⋯ γ j − 1 ( τ ) α j \langle a,e(\tau)\rangle=\sum_{j=1}^{r}\gamma_0(\tau)\cdots\gamma_{j-1}(\tau)\alpha_j ⟨ a , e ( τ )⟩ = ∑ j = 1 r γ 0 ( τ ) ⋯ γ j − 1 ( τ ) α j for τ ≥ τ ′ \tau\ge\tau' τ ≥ τ ′ . If every α j \alpha_j α j vanishes, then ⟨ a , e ⟩ \langle a,e\rangle ⟨ a , e ⟩ vanishes identically on [ τ ′ , + ∞ ) [\tau',+\infty) [ τ ′ , + ∞ ) , and we put τ a : = τ ′ \tau_a:=\tau' τ a := τ ′ . Otherwise let j j j be minimal with α j ≠ 0 \alpha_j\ne0 α j = 0 . Then
⟨ a , e ( τ ) ⟩ = γ 0 ( τ ) ⋯ γ j − 1 ( τ ) ( α j + ∑ l = j + 1 r γ j ( τ ) ⋯ γ l − 1 ( τ ) α l ) \langle a,e(\tau)\rangle=\gamma_0(\tau)\cdots\gamma_{j-1}(\tau)\left(\alpha_j+\sum_{l=j+1}^{r}\gamma_j(\tau)\cdots\gamma_{l-1}(\tau)\alpha_l\right) ⟨ a , e ( τ )⟩ = γ 0 ( τ ) ⋯ γ j − 1 ( τ ) α j + l = j + 1 ∑ r γ j ( τ ) ⋯ γ l − 1 ( τ ) α l
and γ j ⋯ γ l − 1 → 0 \gamma_j\cdots\gamma_{l-1}\to0 γ j ⋯ γ l − 1 → 0 for l > j l>j l > j , so ⟨ a , e ⟩ \langle a,e\rangle ⟨ a , e ⟩ has no zero on some interval [ τ a , + ∞ ) [\tau_a,+\infty) [ τ a , + ∞ ) . By Lemma B.9 with D = e D=e D = e , no piece of the tail starts after the largest of the times τ a \tau_a τ a , a ∈ Φ a\in\Phi a ∈ Φ , which contradicts τ ( k ) → + ∞ \tau^{(k)}\to+\infty τ ( k ) → + ∞ .
Conclusion of the proof
Proof of Theorem 4.1. By Proposition B.3, the breakpoints are finitely many, so the final piece is [ τ ( K ) , + ∞ ) [\tau^{(K)},+\infty) [ τ ( K ) , + ∞ ) and τ ( K + 1 ) = τ ( K ) + τ ∗ ( K ) = + ∞ \tau^{(K+1)}=\tau^{(K)}+\tau_*^{(K)}=+\infty τ ( K + 1 ) = τ ( K ) + τ ∗ ( K ) = + ∞ . By Proposition B.1(i), d ( x ( τ ) ) d(x(\tau)) d ( x ( τ )) converges to the vector d s d^s d s of (67). By (65), d ( K ) ∈ ker A J ∗ ( K ) d^{(K)}\in\ker A_{J_*^{(K)}} d ( K ) ∈ ker A J ∗ ( K ) , and by (37), x ( τ ) = x ( K ) + M J ∗ ( K ) ( τ − τ ( K ) ) d ( K ) x(\tau)=x^{(K)}+M_{J_*^{(K)}}(\tau-\tau^{(K)})d^{(K)} x ( τ ) = x ( K ) + M J ∗ ( K ) ( τ − τ ( K ) ) d ( K ) on the final piece, where M ˙ J ∗ ( K ) ( τ ) = e − P J ∗ ( K ) ⊥ Q τ \dot M_{J_*^{(K)}}(\tau)=e^{-P_{J_*^{(K)}}^{\perp}Q\tau} M ˙ J ∗ ( K ) ( τ ) = e − P J ∗ ( K ) ⊥ Q τ by (19). The matrices P J ∗ ( K ) ⊥ Q P_{J_*^{(K)}}^{\perp}Q P J ∗ ( K ) ⊥ Q and S J ∗ ( K ) S_{J_*^{(K)}} S J ∗ ( K ) agree on ker A J ∗ ( K ) \ker A_{J_*^{(K)}} ker A J ∗ ( K ) and map it into itself. A vector v ∈ ker A J ∗ ( K ) v\in\ker A_{J_*^{(K)}} v ∈ ker A J ∗ ( K ) satisfies S J ∗ ( K ) v = 0 S_{J_*^{(K)}}v=0 S J ∗ ( K ) v = 0 if and only if ⟨ v , Q v ⟩ = 0 \langle v,Qv\rangle=0 ⟨ v , Q v ⟩ = 0 , that is, Q v = 0 Qv=0 Q v = 0 , so ker A J ∗ ( K ) \ker A_{J_*^{(K)}} ker A J ∗ ( K ) is the orthogonal sum of ker Q ∩ ker A J ∗ ( K ) \ker Q\cap\ker A_{J_*^{(K)}} ker Q ∩ ker A J ∗ ( K ) and Im ( S J ∗ ( K ) ) \operatorname{Im}(S_{J_*^{(K)}}) Im ( S J ∗ ( K ) ) , on which S J ∗ ( K ) S_{J_*^{(K)}} S J ∗ ( K ) is positive definite. Then
d ( x ( τ ) ) = e − S J ∗ ( K ) ( τ − τ ( K ) ) d ( K ) ⟶ proj ker Q ∩ ker A J ∗ ( K ) ( d ( K ) ) , (84) d(x(\tau))=e^{-S_{J_*^{(K)}}(\tau-\tau^{(K)})}d^{(K)}\longrightarrow\operatorname{proj}_{\ker Q\cap\ker A_{J_*^{(K)}}}\left(d^{(K)}\right), \tag*{(84)} d ( x ( τ )) = e − S J ∗ ( K ) ( τ − τ ( K ) ) d ( K ) ⟶ proj k e r Q ∩ k e r A J ∗ ( K ) ( d ( K ) ) , (84)
so d s d^{s} d s is this projection. Finally, if d s = 0 d^{s}=0 d s = 0 , then d ( K ) ∈ Im ( S J ∗ ( K ) ) d^{(K)} \in\operatorname{Im}(S_{J_{*}^{(K)}}) d ( K ) ∈ Im ( S J ∗ ( K ) ) and so
M J ∗ ( K ) ( τ ) d ( K ) = ( S J ∗ ( K ) ) † ( I n × n − e − S J ∗ ( K ) τ ) d ( K ) . M_{J_{*}^{(K)}}(\tau)d^{(K)}=(S_{J_{*}^{(K)}})^{\dagger}\left(I_{n\times n}-e^{-S_{J_{*}^{(K)}}\tau}\right)d^{(K)}. M J ∗ ( K ) ( τ ) d ( K ) = ( S J ∗ ( K ) ) † ( I n × n − e − S J ∗ ( K ) τ ) d ( K ) .
Hence x ( τ ) x(\tau) x ( τ ) converges to x ( K ) + ( S J ∗ ( K ) ) † d ( K ) x^{(K)}+(S_{J_{*}^{(K)}})^{\dagger}d^{(K)} x ( K ) + ( S J ∗ ( K ) ) † d ( K ) , an optimal solution by Proposition B.1(iii). If d s ≠ 0 d^{s}\ne0 d s = 0 , then Lemma B.10 (i) gives (ii).
Proofs for Section 5
This appendix proves Proposition 5.1 and Theorem 5.1. Appendix § 11 .§ 3.1 collects the objects the proofs need beyond Section 5, Appendix § 11 .§ 3.2 excludes the one obstruction to convergence, Appendix § 11 .§ 3.3 proves the proposition, and Appendix § 11 .§ 3.4 proves the theorem.
Throughout, Q Q Q is SPSD with largest eigenvalue λ max \lambda_{\max} λ m a x . We write x ( k ) x^{(k)} x ( k ) , d ( k ) d^{(k)} d ( k ) , Δ ( k ) \Delta^{(k)} Δ ( k ) for the iterates, directions, and steps of Algorithm 2, run with tolerance ε = 0 \varepsilon=0 ε = 0 , and we abbreviate E ( k ) : = E ( x ( k ) ) \mathcal{E}^{(k)}:=\mathcal{E}(x^{(k)}) E ( k ) := E ( x ( k ) ) , J ( k ) : = J ( x ( k ) ) J^{(k)}:=J(x^{(k)}) J ( k ) := J ( x ( k ) ) . We choose every NNLS solution p ( x ) p(x) p ( x ) below so that (LI) holds.
Setup
Recall the compressed Hessian S J S_J S J of (59) and its properties, collected in Appendix § 11 .§ 2.1 . It is positive semidefinite on ker A J \ker A_J ker A J . The direction d ( x ) d(x) d ( x ) vanishes exactly at the optimal solutions, since d ( x ) = 0 d(x)=0 d ( x ) = 0 if and only if 0 ∈ ∂ x V ( x ) 0\in\partial_xV(x) 0 ∈ ∂ x V ( x ) (§ 2.1 ). We may therefore assume d ( k ) ≠ 0 d^{(k)}\ne0 d ( k ) = 0 , since Algorithm 2 stops at d ( k ) = 0 d^{(k)}=0 d ( k ) = 0 , its test being ∥ d ( k ) ∥ 2 2 ≤ ε = 0 \lVert d^{(k)}\rVert_2^2\le\varepsilon=0 ∥ d ( k ) ∥ 2 2 ≤ ε = 0 . Likewise Δ ( k ) < + ∞ \Delta^{(k)}<+\infty Δ ( k ) < + ∞ at every step of an infinite iteration, since Δ ( k ) = + ∞ \Delta^{(k)}=+\infty Δ ( k ) = + ∞ stops it.
We call step k k k blocked if Δ ( k ) = Δ f e a s ( x ( k ) ) < Δ ∗ ( k ) \Delta^{(k)}=\Delta_{\mathrm{feas}}(x^{(k)})<\Delta^{(k)}_{*} Δ ( k ) = Δ feas ( x ( k ) ) < Δ ∗ ( k ) , and full otherwise. A blocked step ends on a constraint: some i k ∉ E ( k ) i_k\notin\mathcal{E}^{(k)} i k ∈ / E ( k ) attains the minimum in (14), with
[ A d ( k ) ] i k > 0 , [ A x ( k ) ] i k < b i k , [ A x ( k + 1 ) ] i k = b i k , (85) [Ad^{(k)}]_{i_k}>0,\qquad[Ax^{(k)}]_{i_k}<b_{i_k},\qquad[Ax^{(k+1)}]_{i_k}=b_{i_k}, \tag*{(85)} [ A d ( k ) ] i k > 0 , [ A x ( k ) ] i k < b i k , [ A x ( k + 1 ) ] i k = b i k , (85)
and we call i k i_k i k the blocking wall of step k k k . The first relation holds because Δ f e a s ( x ( k ) ) < Δ ∗ ( k ) ≤ + ∞ \Delta_{\mathrm{feas}}(x^{(k)})<\Delta^{(k)}_{*}\le+\infty Δ feas ( x ( k ) ) < Δ ∗ ( k ) ≤ + ∞ , so the minimizing quotient in (14) is finite and positive. Every step has Δ ( k ) > 0 \Delta^{(k)}>0 Δ ( k ) > 0 . Indeed, on a blocked step, the first two relations of (78) give Δ f e a s ( x ( k ) ) > 0 \Delta_{\mathrm{feas}}(x^{(k)})>0 Δ feas ( x ( k ) ) > 0 , while on a full step Δ ( k ) = Δ ∗ ( k ) = ∥ d ( k ) ∥ 2 2 / ⟨ d ( k ) , Q d ( k ) ⟩ > 0 \Delta^{(k)}=\Delta^{(k)}_{*}=\lVert d^{(k)}\rVert_2^2/\langle d^{(k)},Qd^{(k)}\rangle>0 Δ ( k ) = Δ ∗ ( k ) = ∥ d ( k ) ∥ 2 2 / ⟨ d ( k ) , Q d ( k ) ⟩ > 0 because d ( k ) ≠ 0 d^{(k)}\ne0 d ( k ) = 0 . The constraints of J ( k ) J^{(k)} J ( k ) are active at x ( k ) x^{(k)} x ( k ) because J ( k ) ⊆ E ( k ) J^{(k)}\subseteq\mathcal{E}^{(k)} J ( k ) ⊆ E ( k ) , and stay active at x ( k + 1 ) x^{(k+1)} x ( k + 1 ) because [ A d ( k ) ] J ( k ) = 0 [Ad^{(k)}]_{J^{(k)}}=0 [ A d ( k ) ] J ( k ) = 0 by (13c). Hence J ( k ) ⊆ E ( k + 1 ) J^{(k)}\subseteq\mathcal{E}^{(k+1)} J ( k ) ⊆ E ( k + 1 ) .
Definition B.1 (Blocked tail). A blocked tail is an infinite sequence of consecutive iterates of Algorithm 2, indexed by k ≥ 0 k\ge0 k ≥ 0 after relabeling, such that every step is blocked, inf k ∥ d ( k ) ∥ 2 > 0 \inf_k\lVert d^{(k)}\rVert_2>0 inf k ∥ d ( k ) ∥ 2 > 0 , and ∑ k Δ ( k ) < + ∞ \sum_k\Delta^{(k)}<+\infty ∑ k Δ ( k ) < + ∞ .
Lemma B.15. Suppose the iteration is infinite, that ∥ d ( k ) ∥ 2 ≥ θ > 0 \lVert d^{(k)}\rVert_2\ge\theta>0 ∥ d ( k ) ∥ 2 ≥ θ > 0 for every k k k , and that inf k V ( x ( k ) ) > − ∞ \inf_k V(x^{(k)})>-\infty inf k V ( x ( k ) ) > − ∞ . Then only finitely many steps have Δ ( k ) ≥ 1 / λ max \Delta^{(k)}\ge1/\lambda_{\max} Δ ( k ) ≥ 1/ λ m a x , and the steps after the last of them, or all steps if there is none, form a blocked tail.
Proof. Since Δ ( k ) ≤ Δ f e a s ( x ( k ) ) \Delta^{(k)}\le\Delta_{\mathrm{feas}}(x^{(k)}) Δ ( k ) ≤ Δ feas ( x ( k ) ) , the bound (39) gives
V ( x ( k ) ) − V ( x ( k + 1 ) ) ≥ 1 2 min ( Δ ( k ) , 1 / λ max ) θ 2 ≥ 0. V(x^{(k)})-V(x^{(k+1)})\ge\frac{1}{2}\min\left(\Delta^{(k)},1/\lambda_{\max}\right)\theta^2\ge0. V ( x ( k ) ) − V ( x ( k + 1 ) ) ≥ 2 1 min ( Δ ( k ) , 1/ λ m a x ) θ 2 ≥ 0.
Summing over k k k bounds 1 2 θ 2 ∑ k min ( Δ ( k ) , 1 / λ max ) \frac{1}{2}\theta^2\sum_k\min\left(\Delta^{(k)},1/\lambda_{\max}\right) 2 1 θ 2 ∑ k min ( Δ ( k ) , 1/ λ m a x ) by V ( x ( 0 ) ) − inf k V ( x ( k ) ) V(x^{(0)})-\inf_kV(x^{(k)}) V ( x ( 0 ) ) − inf k V ( x ( k ) ) . Hence finitely many steps have Δ ( k ) ≥ 1 / λ max \Delta^{(k)}\ge1/\lambda_{\max} Δ ( k ) ≥ 1/ λ m a x , and the others have summable length. A full step has Δ ( k ) = Δ ∗ ( k ) ≥ 1 / λ max \Delta^{(k)}=\Delta^{(k)}_{*}\ge1/\lambda_{\max} Δ ( k ) = Δ ∗ ( k ) ≥ 1/ λ m a x , the quotient being finite because Δ ( k ) < + ∞ \Delta^{(k)}<+\infty Δ ( k ) < + ∞ , so the steps after the last one with Δ ( k ) ≥ 1 / λ max \Delta^{(k)}\ge1/\lambda_{\max} Δ ( k ) ≥ 1/ λ m a x are blocked, summable, and have ∥ d ( k ) ∥ 2 ≥ θ \lVert d^{(k)}\rVert_2\ge\theta ∥ d ( k ) ∥ 2 ≥ θ . ∎
The blocked tail
Theorem 5.1 rests on excluding a single pattern.
Lemma B.16. Every step of Algorithm 2 satisfies ∥ d ( k + 1 ) ∥ 2 ≤ ( 1 + λ max Δ ( k ) ) ∥ d ( k ) ∥ 2 \lVert d^{(k+1)}\rVert_2 \leq(1+\lambda_{\max}\Delta^{(k)})\lVert d^{(k)}\rVert_2 ∥ d ( k + 1 ) ∥ 2 ≤ ( 1 + λ m a x Δ ( k ) ) ∥ d ( k ) ∥ 2 . More precisely, with d ~ : = proj T C ( x ( k + 1 ) ) ( − ∇ x f ( x ( k ) ) ) \tilde d := \operatorname{proj}_{T_C(x^{(k+1)})}(-\nabla_x f(x^{(k)})) d ~ := proj T C ( x ( k + 1 ) ) ( − ∇ x f ( x ( k ) )) , we have that
∥ d ~ − d ( k ) ∥ 2 2 ≤ ∥ d ( k ) ∥ 2 2 − ∥ d ~ ∥ 2 2 and ∥ d ( k + 1 ) − d ~ ∥ 2 ≤ λ max Δ ( k ) ∥ d ( k ) ∥ 2 . \lVert\tilde d-d^{(k)}\rVert_2^2 \leq\lVert d^{(k)}\rVert_2^2-\lVert\tilde d\rVert_2^2 \quad\text{and} \quad\lVert d^{(k+1)}-\tilde d\rVert_2 \leq\lambda_{\max}\Delta^{(k)}\lVert d^{(k)}\rVert_2. ∥ d ~ − d ( k ) ∥ 2 2 ≤ ∥ d ( k ) ∥ 2 2 − ∥ d ~ ∥ 2 2 and ∥ d ( k + 1 ) − d ~ ∥ 2 ≤ λ m a x Δ ( k ) ∥ d ( k ) ∥ 2 .
Proof. On every step, full or blocked, J ( k ) ⊆ E ( k + 1 ) J^{(k)} \subseteq E^{(k+1)} J ( k ) ⊆ E ( k + 1 ) as shown in Appendix B.3.1, so T : = T C ( x ( k + 1 ) ) ⊆ K : = { s : A J ( k ) s ≤ 0 } T := T_C(x^{(k+1)}) \subseteq K := \{s : A_{J^{(k)}}s \leq0\} T := T C ( x ( k + 1 ) ) ⊆ K := { s : A J ( k ) s ≤ 0 } , and Lemma B.2 gives d ( k ) = proj K ( − ∇ x f ( x ( k ) ) ) d^{(k)}=\operatorname{proj}_K(-\nabla_x f(x^{(k)})) d ( k ) = proj K ( − ∇ x f ( x ( k ) )) . Set d ~ : = proj T ( − ∇ x f ( x ( k ) ) ) \tilde d:=\operatorname{proj}_T(-\nabla_x f(x^{(k)})) d ~ := proj T ( − ∇ x f ( x ( k ) )) . Lemma B.4 applied to T ⊆ K T\subseteq K T ⊆ K gives the first estimate, hence ∥ d ~ ∥ 2 ≤ ∥ d ( k ) ∥ 2 \lVert\tilde d\rVert_2\leq\lVert d^{(k)}\rVert_2 ∥ d ~ ∥ 2 ≤ ∥ d ( k ) ∥ 2 , and the 1-Lipschitz property of proj T \operatorname{proj}_T proj T gives ∥ d ( k + 1 ) − d ~ ∥ 2 ≤ ∥ Q ( x ( k + 1 ) − x ( k ) ) ∥ 2 ≤ λ max Δ ( k ) ∥ d ( k ) ∥ 2 \lVert d^{(k+1)}-\tilde d\rVert_2\leq\lVert Q(x^{(k+1)}-x^{(k)})\rVert_2\leq\lambda_{\max}\Delta^{(k)}\lVert d^{(k)}\rVert_2 ∥ d ( k + 1 ) − d ~ ∥ 2 ≤ ∥ Q ( x ( k + 1 ) − x ( k ) ) ∥ 2 ≤ λ m a x Δ ( k ) ∥ d ( k ) ∥ 2 . Adding the two bounds proves the claim.
Along a blocked tail, iterating Lemma B.16 with 1 + t ≤ e t 1+t\leq e^t 1 + t ≤ e t gives
sup l ≥ 0 ∥ d ( l ) ∥ 2 ≤ ∥ d ( 0 ) ∥ 2 exp ( λ max ∑ l ≥ 0 Δ ( l ) ) = : B 0 < + ∞ , (86) \sup_{l\geq0}\lVert d^{(l)}\rVert_2\leq\lVert d^{(0)}\rVert_2\exp\left(\lambda_{\max}\sum_{l\geq0}\Delta^{(l)}\right)=:B_0<+\infty, \tag*{(86)} l ≥ 0 sup ∥ d ( l ) ∥ 2 ≤ ∥ d ( 0 ) ∥ 2 exp ( λ m a x l ≥ 0 ∑ Δ ( l ) ) =: B 0 < + ∞ , (86)
the sum being finite by Definition B.1, and d ( 0 ) d^{(0)} d ( 0 ) being the first direction of the tail. Hence ∥ x ( k + 1 ) − x ( k ) ∥ 2 ≤ B 0 Δ ( k ) \lVert x^{(k+1)}-x^{(k)}\rVert_2\leq B_0\Delta^{(k)} ∥ x ( k + 1 ) − x ( k ) ∥ 2 ≤ B 0 Δ ( k ) , so the iterates are Cauchy and converge to some x ˉ ∈ C \bar x\in C x ˉ ∈ C .
Theorem B.1. Algorithm 2, run with tolerance ε = 0 \varepsilon=0 ε = 0 , has no blocked tails.
We prove this below. Fix a blocked tail with limit x ˉ \bar x x ˉ , step lengths Δ ( k ) \Delta^{(k)} Δ ( k ) , blocking walls i k i_k i k , and active sets J ( k ) J^{(k)} J ( k ) . Both index collections are finite, so we may discard finitely many further steps until every occurring active set and every blocking wall repeats infinitely often. We call these recurring .
The limit velocity. For recurring J J J , the map x ↦ − P J ⊥ ∇ x f ( x ) x\mapsto-P_J^\perp\nabla_x f(x) x ↦ − P J ⊥ ∇ x f ( x ) is affine, so the directions along the steps with J ( k ) = J J^{(k)}=J J ( k ) = J converge:
d ( k ) = − P J ⊥ ∇ x f ( x ( k ) ) ⟶ d ˉ J : = − P J ⊥ ∇ x f ( x ˉ ) , ∥ d ( k ) − d ˉ J ∥ 2 ≤ λ max ∥ x ( k ) − x ˉ ∥ 2 (87) d^{(k)}=-P_J^\perp\nabla_x f(x^{(k)})\longrightarrow\bar d_J:=-P_J^\perp\nabla_x f(\bar x),\qquad\lVert d^{(k)}-\bar d_J\rVert_2\leq\lambda_{\max}\lVert x^{(k)}-\bar x\rVert_2 \tag*{(87)} d ( k ) = − P J ⊥ ∇ x f ( x ( k ) ) ⟶ d ˉ J := − P J ⊥ ∇ x f ( x ˉ ) , ∥ d ( k ) − d ˉ J ∥ 2 ≤ λ m a x ∥ x ( k ) − x ˉ ∥ 2 (87)
for every k k k with J ( k ) = J J^{(k)}=J J ( k ) = J . The next two results show that these limits coincide.
Proposition B.4. Let a blocked tail be given as above. Its directions converge to a single limit v ˉ \bar v v ˉ with ∥ v ˉ ∥ 2 > 0 \lVert\bar v\rVert_2>0 ∥ v ˉ ∥ 2 > 0 . Moreover d ˉ J = v ˉ \bar d_J=\bar v d ˉ J = v ˉ and v ˉ ∈ ker A J \bar v\in\ker A_J v ˉ ∈ ker A J for every recurring active set J J J .
Proof. We proceed in four steps.
Step 1. Set d ~ : = proj T C ( x ( k + 1 ) ) ( − ∇ x f ( x ( k ) ) ) \tilde d:=\operatorname{proj}_{T_C(x^{(k+1)})}(-\nabla_x f(x^{(k)})) d ~ := proj T C ( x ( k + 1 ) ) ( − ∇ x f ( x ( k ) )) . Lemma B.16 gives
∥ d ~ − d ( k ) ∥ 2 2 ≤ ∥ d ( k ) ∥ 2 2 − ∥ d ~ ∥ 2 2 and ∥ d ( k + 1 ) − d ~ ∥ 2 ≤ λ max Δ ( k ) ∥ d ( k ) ∥ 2 ≤ λ max B 0 Δ ( k ) , (88) \lVert\tilde d-d^{(k)}\rVert_2^2\leq\lVert d^{(k)}\rVert_2^2-\lVert\tilde d\rVert_2^2\quad\text{and}\quad\lVert d^{(k+1)}-\tilde d\rVert_2\leq\lambda_{\max}\Delta^{(k)}\lVert d^{(k)}\rVert_2\leq\lambda_{\max}B_0\Delta^{(k)}, \tag*{(88)} ∥ d ~ − d ( k ) ∥ 2 2 ≤ ∥ d ( k ) ∥ 2 2 − ∥ d ~ ∥ 2 2 and ∥ d ( k + 1 ) − d ~ ∥ 2 ≤ λ m a x Δ ( k ) ∥ d ( k ) ∥ 2 ≤ λ m a x B 0 Δ ( k ) , (88)
the last inequality by (79). Hence ∥ d ( k + 1 ) ∥ 2 ≤ ∥ d ( k ) ∥ 2 + λ max B 0 Δ ( k ) \lVert d^{(k+1)}\rVert_2\leq\lVert d^{(k)}\rVert_2+\lambda_{\max}B_0\Delta^{(k)} ∥ d ( k + 1 ) ∥ 2 ≤ ∥ d ( k ) ∥ 2 + λ m a x B 0 Δ ( k ) . Since ∑ k Δ ( k ) < + ∞ \sum_k\Delta^{(k)}<+\infty ∑ k Δ ( k ) < + ∞ , the sequence ∥ d ( k ) ∥ 2 + λ max B 0 ∑ l ≥ k Δ ( l ) \lVert d^{(k)}\rVert_2+\lambda_{\max}B_0\sum_{l\geq k}\Delta^{(l)} ∥ d ( k ) ∥ 2 + λ m a x B 0 ∑ l ≥ k Δ ( l ) is nonincreasing and bounded below, so lim k ∥ d ( k ) ∥ 2 \lim_k\lVert d^{(k)}\rVert_2 lim k ∥ d ( k ) ∥ 2 exists and is at least $\inf_k\lVert d^{(k)}\rVert_2>0. Step 2. From ∥ d ~ ∥ 2 ≥ ∥ d ( k + 1 ) ∥ 2 − λ max B 0 Δ ( k ) \lVert\tilde d\rVert_2 \ge\lVert d^{(k+1)}\rVert_2-\lambda_{\max}B_0\Delta^{(k)} ∥ d ~ ∥ 2 ≥ ∥ d ( k + 1 ) ∥ 2 − λ m a x B 0 Δ ( k ) and (88) , the case ∥ d ( k + 1 ) ∥ 2 < λ max B 0 Δ ( k ) \lVert d^{(k+1)}\rVert_2<\lambda_{\max}B_0\Delta^{(k)} ∥ d ( k + 1 ) ∥ 2 < λ m a x B 0 Δ ( k ) being trivial,
∥ d ~ − d ( k ) ∥ 2 2 ≤ ∥ d ( k ) ∥ 2 2 − ∥ d ( k + 1 ) ∥ 2 2 + 2 λ max B 0 2 Δ ( k ) + ( λ max B 0 Δ ( k ) ) 2 , \lVert\tilde d-d^{(k)}\rVert_2^2 \le\lVert d^{(k)}\rVert_2^2-\lVert d^{(k+1)}\rVert_2^2+2\lambda_{\max}B_0^2\Delta^{(k)}+\left(\lambda_{\max}B_0\Delta^{(k)}\right)^2, ∥ d ~ − d ( k ) ∥ 2 2 ≤ ∥ d ( k ) ∥ 2 2 − ∥ d ( k + 1 ) ∥ 2 2 + 2 λ m a x B 0 2 Δ ( k ) + ( λ m a x B 0 Δ ( k ) ) 2 ,
and the right-hand side tends to 0 0 0 because ∥ d ( k ) ∥ 2 \lVert d^{(k)}\rVert_2 ∥ d ( k ) ∥ 2 converges and Δ ( k ) → 0 \Delta^{(k)}\to0 Δ ( k ) → 0 . The triangle inequality now gives ∥ d ( k + 1 ) − d ( k ) ∥ 2 → 0 \lVert d^{(k+1)}-d^{(k)}\rVert_2\to0 ∥ d ( k + 1 ) − d ( k ) ∥ 2 → 0 .
Step 3. By (87) , ∥ d ( k ) − d ˉ J ( k ) ∥ 2 ≤ λ max ∥ x ( k ) − x ˉ ∥ 2 → 0 \lVert d^{(k)}-\bar d_{J(k)}\rVert_2\le\lambda_{\max}\lVert x^{(k)}-\bar x\rVert_2\to0 ∥ d ( k ) − d ˉ J ( k ) ∥ 2 ≤ λ m a x ∥ x ( k ) − x ˉ ∥ 2 → 0 uniformly in the active set. The finitely many values { d ˉ J : J recurring } \{\bar d_J:J\text{ recurring}\} { d ˉ J : J recurring } lie at least 2 ρ > 0 2\rho>0 2 ρ > 0 apart, where ρ = 1 \rho=1 ρ = 1 if only one value occurs. For k k k large,
∥ d ˉ J ( k + 1 ) − d ˉ J ( k ) ∥ 2 ≤ ∥ d ( k + 1 ) − d ( k ) ∥ 2 + λ max ∥ x ( k ) − x ˉ ∥ 2 + λ max ∥ x ( k + 1 ) − x ˉ ∥ 2 < 2 ρ , \lVert\bar d_{J(k+1)}-\bar d_{J(k)}\rVert_2\le\lVert d^{(k+1)}-d^{(k)}\rVert_2+\lambda_{\max}\lVert x^{(k)}-\bar x\rVert_2+\lambda_{\max}\lVert x^{(k+1)}-\bar x\rVert_2<2\rho, ∥ d ˉ J ( k + 1 ) − d ˉ J ( k ) ∥ 2 ≤ ∥ d ( k + 1 ) − d ( k ) ∥ 2 + λ m a x ∥ x ( k ) − x ˉ ∥ 2 + λ m a x ∥ x ( k + 1 ) − x ˉ ∥ 2 < 2 ρ ,
which forces d ˉ J ( k + 1 ) = d ˉ J ( k ) \bar d_{J(k+1)}=\bar d_{J(k)} d ˉ J ( k + 1 ) = d ˉ J ( k ) . Hence d ˉ J ( k ) \bar d_{J(k)} d ˉ J ( k ) is eventually constant, equal to some v ˉ \bar v v ˉ . Every recurring active set attains it, so d ˉ J = v ˉ \bar d_J=\bar v d ˉ J = v ˉ for all recurring J J J , and d ( k ) → v ˉ d^{(k)}\to\bar v d ( k ) → v ˉ with ∥ v ˉ ∥ 2 > 0 \lVert\bar v\rVert_2>0 ∥ v ˉ ∥ 2 > 0 .
Step 4. For recurring J J J , the directions with J ( k ) = J J^{(k)}=J J ( k ) = J lie in the closed subspace ker A J \ker A_J ker A J and converge to v ˉ \bar v v ˉ , so v ˉ ∈ ker A J \bar v\in\ker A_J v ˉ ∈ ker A J .
Exact reduction
We place the blocked tail in coordinates adapted to its limit, namely
σ k : = ∑ l ≥ k Δ ( l ) , ζ ( k ) : = x ( k ) − x ˉ + σ k v ˉ , (89) \sigma_k:=\sum_{l\ge k}\Delta^{(l)},\qquad\zeta^{(k)}:=x^{(k)}-\bar x+\sigma_k\bar v, \tag*{(89)} σ k := l ≥ k ∑ Δ ( l ) , ζ ( k ) := x ( k ) − x ˉ + σ k v ˉ , (89)
so σ k ↓ 0 \sigma_k\downarrow0 σ k ↓ 0 , x ( k ) − x ˉ = − ∑ l ≥ k Δ ( l ) d ( l ) x^{(k)}-\bar x=-\sum_{l\ge k}\Delta^{(l)}d^{(l)} x ( k ) − x ˉ = − ∑ l ≥ k Δ ( l ) d ( l ) by telescoping, and ζ ( k ) = − ∑ l ≥ k Δ ( l ) ( d ( l ) − v ˉ ) → 0 \zeta^{(k)}=-\sum_{l\ge k}\Delta^{(l)}(d^{(l)}-\bar v)\to0 ζ ( k ) = − ∑ l ≥ k Δ ( l ) ( d ( l ) − v ˉ ) → 0 . Define
g : = Q v ˉ , u J : = P J ⊥ g = S J v ˉ (90) g:=Q\bar v,\qquad u_J:=P_J^\perp g=S_J\bar v \tag*{(90)} g := Q v ˉ , u J := P J ⊥ g = S J v ˉ (90)
for recurring J J J , where S J v ˉ = P J ⊥ Q v ˉ S_J\bar v=P_J^\perp Q\bar v S J v ˉ = P J ⊥ Q v ˉ because v ˉ ∈ ker A J \bar v\in\ker A_J v ˉ ∈ ker A J by Proposition B.4.
Lemma B.17 (Recurring walls). Every recurring blocking wall i i i satisfies:
(i) A i x ˉ = b i A_i\bar x=b_i A i x ˉ = b i and A i v ˉ = 0 A_i\bar v=0 A i v ˉ = 0 ;
(ii) A i ζ ( k ) = A i ( x ( k ) − x ˉ ) ≤ 0 A_i\zeta^{(k)}=A_i(x^{(k)}-\bar x)\le0 A i ζ ( k ) = A i ( x ( k ) − x ˉ ) ≤ 0 for every k k k on the tail, with strict inequality whenever i i i is the blocking wall of step k k k , and A i ζ ( k + 1 ) = 0 A_i\zeta^{(k+1)}=0 A i ζ ( k + 1 ) = 0 for every such k k k .
Proof. By (89) , ζ ( k ) = − ∑ l ≥ k Δ ( l ) ( d ( l ) − v ˉ ) \zeta^{(k)}=-\sum_{l\ge k}\Delta^{(l)}(d^{(l)}-\bar v) ζ ( k ) = − ∑ l ≥ k Δ ( l ) ( d ( l ) − v ˉ ) , so ∥ ζ ( k ) ∥ 2 ≤ σ k sup l ≥ k ∥ d ( l ) − v ˉ ∥ 2 = o ( σ k ) \lVert\zeta^{(k)}\rVert_2\le\sigma_k\sup_{l\ge k}\lVert d^{(l)}-\bar v\rVert_2=o(\sigma_k) ∥ ζ ( k ) ∥ 2 ≤ σ k sup l ≥ k ∥ d ( l ) − v ˉ ∥ 2 = o ( σ k ) by Proposition B.4. At the steps k k k blocked by i i i , A i x ( k + 1 ) = b i A_ix^{(k+1)}=b_i A i x ( k + 1 ) = b i by (85) ; these iterates converge to x ˉ \bar x x ˉ , so A i x ˉ = b i A_i\bar x=b_i A i x ˉ = b i , and then 0 = A i ( x ( k + 1 ) − x ˉ ) = − σ k + 1 A i v ˉ + A i ζ ( k + 1 ) 0=A_i(x^{(k+1)}-\bar x)=-\sigma_{k+1}A_i\bar v+A_i\zeta^{(k+1)} 0 = A i ( x ( k + 1 ) − x ˉ ) = − σ k + 1 A i v ˉ + A i ζ ( k + 1 ) . Dividing by σ k + 1 \sigma_{k+1} σ k + 1 and letting k → ∞ k\to\infty k → ∞ along the hits gives A i v ˉ = 0 A_i\bar v=0 A i v ˉ = 0 . This proves (i).
Feasibility gives A i ( x ( k ) − x ˉ ) = A i x ( k ) − b i ≤ 0 A_i(x^{(k)}-\bar x)=A_ix^{(k)}-b_i\le0 A i ( x ( k ) − x ˉ ) = A i x ( k ) − b i ≤ 0 , and (i) gives A i ζ ( k ) = A i ( x ( k ) − x ˉ ) A_i\zeta^{(k)}=A_i(x^{(k)}-\bar x) A i ζ ( k ) = A i ( x ( k ) − x ˉ ) . The inequality is strict at the blocking steps of i i i by (85) , where A i ζ ( k + 1 ) = A i ( x ( k + 1 ) − x ˉ ) = 0 A_i\zeta^{(k+1)}=A_i(x^{(k+1)}-\bar x)=0 A i ζ ( k + 1 ) = A i ( x ( k + 1 ) − x ˉ ) = 0 .
Lemma B.18. On the blocked tail, relabeled as above so that every J ( k ) J^{(k)} J ( k ) is recurring, for every k k k :
(i) A J ( k ) ζ ( k ) = A J ( k ) ζ ( k + 1 ) = 0 A_{J^{(k)}}\zeta^{(k)}=A_{J^{(k)}}\zeta^{(k+1)}=0 A J ( k ) ζ ( k ) = A J ( k ) ζ ( k + 1 ) = 0 ;
(ii) ζ ( k + 1 ) = ζ ( k ) + Δ ( k ) σ k u J ( k ) − Δ ( k ) S J ( k ) ζ ( k ) \zeta^{(k+1)}=\zeta^{(k)}+\Delta^{(k)}\sigma_k u_{J^{(k)}}-\Delta^{(k)}S_{J^{(k)}}\zeta^{(k)} ζ ( k + 1 ) = ζ ( k ) + Δ ( k ) σ k u J ( k ) − Δ ( k ) S J ( k ) ζ ( k ) . Proof. The constraints of J ( k ) J^{(k)} J ( k ) are active at x ( k ) x^{(k)} x ( k ) and x ( k + 1 ) x^{(k+1)} x ( k + 1 ) , as shown after (85) . Since J ( k ) J^{(k)} J ( k ) recurs, they are active at infinitely many iterates converging to x ˉ \bar{x} x ˉ , so A J ( k ) x ˉ = b J ( k ) A_{J^{(k)}}\bar{x}=b_{J^{(k)}} A J ( k ) x ˉ = b J ( k ) and A J ( k ) ( x ( k ) − x ˉ ) = A J ( k ) ( x ( k + 1 ) − x ˉ ) = 0 A_{J^{(k)}}(x^{(k)}-\bar{x})=A_{J^{(k)}}(x^{(k+1)}-\bar{x})=0 A J ( k ) ( x ( k ) − x ˉ ) = A J ( k ) ( x ( k + 1 ) − x ˉ ) = 0 . Proposition B.4 gives A J ( k ) v ˉ = 0 A_{J^{(k)}}\bar{v}=0 A J ( k ) v ˉ = 0 . Adding σ k \sigma_k σ k and σ k + 1 \sigma_{k+1} σ k + 1 gives (i).
By Proposition B.4, − P J ( k ) ⊥ ∇ x f ( x ˉ ) = d J ( k ) ( k ) = v ˉ -P_{J^{(k)}}^{\perp}\nabla_x f(\bar{x})=d_{J^{(k)}}^{(k)}=\bar{v} − P J ( k ) ⊥ ∇ x f ( x ˉ ) = d J ( k ) ( k ) = v ˉ . Hence,
d ( k ) = − P J ( k ) ⊥ ∇ x f ( x ( k ) ) = v ˉ − P J ( k ) ⊥ Q ( x ( k ) − x ˉ ) = v ˉ + σ k u J ( k ) − S J ( k ) ζ ( k ) . d^{(k)}=-P_{J^{(k)}}^{\perp}\nabla_x f(x^{(k)})=\bar{v}-P_{J^{(k)}}^{\perp}Q(x^{(k)}-\bar{x})=\bar{v}+\sigma_k u_{J^{(k)}}-S_{J^{(k)}}\zeta^{(k)}. d ( k ) = − P J ( k ) ⊥ ∇ x f ( x ( k ) ) = v ˉ − P J ( k ) ⊥ Q ( x ( k ) − x ˉ ) = v ˉ + σ k u J ( k ) − S J ( k ) ζ ( k ) .
where the last step uses P J ( k ) ⊥ Q ζ ( k ) = S J ( k ) ζ ( k ) P_{J^{(k)}}^{\perp}Q\zeta^{(k)}=S_{J^{(k)}}\zeta^{(k)} P J ( k ) ⊥ Q ζ ( k ) = S J ( k ) ζ ( k ) , valid because ζ ( k ) ∈ ker A J ( k ) \zeta^{(k)}\in\ker A_{J^{(k)}} ζ ( k ) ∈ ker A J ( k ) by (i). Then
ζ ( k + 1 ) = x ( k ) − x ˉ + Δ ( k ) d ( k ) + σ k + 1 v ˉ = ζ ( k ) + ( σ k + 1 − σ k + Δ ( k ) ) v ˉ + Δ ( k ) σ k u J ( k ) − Δ ( k ) S J ( k ) ζ ( k ) , \zeta^{(k+1)}=x^{(k)}-\bar{x}+\Delta^{(k)}d^{(k)}+\sigma_{k+1}\bar{v}=\zeta^{(k)}+(\sigma_{k+1}-\sigma_k+\Delta^{(k)})\bar{v}+\Delta^{(k)}\sigma_k u_{J^{(k)}}-\Delta^{(k)}S_{J^{(k)}}\zeta^{(k)}, ζ ( k + 1 ) = x ( k ) − x ˉ + Δ ( k ) d ( k ) + σ k + 1 v ˉ = ζ ( k ) + ( σ k + 1 − σ k + Δ ( k ) ) v ˉ + Δ ( k ) σ k u J ( k ) − Δ ( k ) S J ( k ) ζ ( k ) ,
and σ k + 1 − σ k + Δ ( k ) = 0 \sigma_{k+1}-\sigma_k+\Delta^{(k)}=0 σ k + 1 − σ k + Δ ( k ) = 0 .
Exclusion. Summing the recursion of Lemma B.18(ii) from k k k to k ′ − 1 k'-1 k ′ − 1 and letting k ′ → ∞ k'\to\infty k ′ → ∞ gives
ζ ( k ) = − ∑ l ≥ k Δ ( l ) ( σ l u J ( l ) − S J ( l ) ζ ( l ) ) , (91) \zeta^{(k)}=-\sum_{l\geq k}\Delta^{(l)}\left(\sigma_lu_{J^{(l)}}-S_{J^{(l)}}\zeta^{(l)}\right), \tag*{(91)} ζ ( k ) = − l ≥ k ∑ Δ ( l ) ( σ l u J ( l ) − S J ( l ) ζ ( l ) ) , (91)
the series converging absolutely because ∥ ζ ( l ) ∥ 2 \|\zeta^{(l)}\|_2 ∥ ζ ( l ) ∥ 2 is bounded, ∥ S J ∥ 2 ≤ λ max \|S_J\|_2\leq\lambda_{\max} ∥ S J ∥ 2 ≤ λ m a x , and σ k < + ∞ \sigma_k<+\infty σ k < + ∞ . This identity is (61) with sums over steps in place of integrals, and its weights are the iterated tail sums σ k [ 0 ] : = 1 \sigma_k^{[0]}:=1 σ k [ 0 ] := 1 and σ k [ j + 1 ] : = ∑ l ≥ k Δ ( l ) σ l [ j ] \sigma_k^{[j+1]}:=\sum_{l\geq k}\Delta^{(l)}\sigma_l^{[j]} σ k [ j + 1 ] := ∑ l ≥ k Δ ( l ) σ l [ j ] , so that σ k [ 1 ] = σ k \sigma_k^{[1]}=\sigma_k σ k [ 1 ] = σ k . They are positive and nonincreasing in k k k , and they satisfy σ k [ j + 1 ] ≤ σ k σ k [ j ] \sigma_k^{[j+1]}\leq\sigma_k\sigma_k^{[j]} σ k [ j + 1 ] ≤ σ k σ k [ j ] , hence σ k [ j ] ≤ σ k j \sigma_k^{[j]}\leq\sigma_k^j σ k [ j ] ≤ σ k j .
Lemma B.19. On the blocked tail, relabeled so that every J ( k ) J^{(k)} J ( k ) is recurring:
(i) There exist vectors u [ 0 ] : = v ˉ , u [ 1 ] , u [ 2 ] , … u^{[0]}:=\bar{v},u^{[1]},u^{[2]},\ldots u [ 0 ] := v ˉ , u [ 1 ] , u [ 2 ] , … such that S J u [ j − 1 ] = u [ j ] ∈ ker A J S_Ju^{[j-1]}=u^{[j]}\in\ker A_J S J u [ j − 1 ] = u [ j ] ∈ ker A J for every recurring J J J and every j ≥ 1 j\geq1 j ≥ 1 , and such that, for every r ≥ 0 r\geq0 r ≥ 0 , the sequence ω r ( k ) : = ζ ( k ) + ∑ j = 1 r σ k [ j + 1 ] u [ j ] \omega_r^{(k)}:=\zeta^{(k)}+\sum_{j=1}^{r}\sigma_k^{[j+1]}u^{[j]} ω r ( k ) := ζ ( k ) + ∑ j = 1 r σ k [ j + 1 ] u [ j ] satisfies ∥ ω r ( k ) ∥ 2 ≤ 2 ∥ u [ r + 1 ] ∥ 2 σ k [ r + 2 ] \|\omega_r^{(k)}\|_2\leq2\|u^{[r+1]}\|_2\sigma_k^{[r+2]} ∥ ω r ( k ) ∥ 2 ≤ 2∥ u [ r + 1 ] ∥ 2 σ k [ r + 2 ] whenever λ max σ k ≤ 1 2 \lambda_{\max}\sigma_k\leq\frac12 λ m a x σ k ≤ 2 1 ;
(ii) Every a ∈ R n a\in\mathbb{R}^n a ∈ R n has an index k a k_a k a such that ⟨ a , ζ ( k ) ⟩ \langle a,\zeta^{(k)}\rangle ⟨ a , ζ ( k ) ⟩ is either all zero or all nonzero for k ≥ k a k\geq k_a k ≥ k a .
Proof. The proofs of Lemmas B.5, B.6, and B.7 carry over to (84), with sums over l ≥ k l\geq k l ≥ k in place of integrals over [ τ , τ ˉ ) [\tau,\bar{\tau}) [ τ , τ ˉ ) , Δ ( l ) \Delta^{(l)} Δ ( l ) in place of d ρ d\rho d ρ , σ k \sigma_k σ k in place of σ ( τ ) \sigma(\tau) σ ( τ ) , and σ k [ j ] \sigma_k^{[j]} σ k [ j ] in place of σ j / j ! \sigma^j/j! σ j / j ! . They use the weights only for the properties above and the recursion that defines them, and of the steps only Δ ( l ) > 0 \Delta^{(l)}>0 Δ ( l ) > 0 , in place of the positive length of the pieces. The hypotheses hold by Proposition B.4, (82), and Lemma B.18(i). □
Proof of Theorem B.1. Suppose a blocked tail exists and adopt the relabeling above. Fix a recurring blocking wall i i i and put φ ( k ) : = A i ζ ( k ) = A i x ( k ) − b i \varphi^{(k)}:=A_i\zeta^{(k)}=A_ix^{(k)}-b_i φ ( k ) := A i ζ ( k ) = A i x ( k ) − b i , by Lemma B.17(ii), so that φ ( k + 1 ) = 0 \varphi^{(k+1)}=0 φ ( k + 1 ) = 0 whenever i i i blocks step k k k . By Lemma B.19(ii) with a = A i T a=A_i^{\mathsf T} a = A i T , either φ \varphi φ vanishes at every iterate beyond some index, or it vanishes at none of them. In the first case i ∈ E ( k ) i\in\mathcal{E}^{(k)} i ∈ E ( k ) at every such iterate, whereas (78) requires i ∉ E ( k ) i\notin\mathcal{E}^{(k)} i ∈ / E ( k ) at every step that i i i blocks. In the second case φ ( k + 1 ) ≠ 0 \varphi^{(k+1)}\ne0 φ ( k + 1 ) = 0 at every such k k k , so no such step is blocked by i i i . Either way i i i blocks finitely many steps. The recurring walls are finitely many, so from some step on no step is blocked by a recurring wall, although every late step is blocked by one. No blocked tail exists. □
Proof of Proposition 5.1
When problem (1) admits an optimal solution, we write X s X^s X s for its optimal solution set and V s V^s V s for its optimal value. The proofs of this appendix rest on one observation. At a feasible x x x with active set J = J ( x ) J=J(x) J = J ( x ) and multipliers p J = p J ( x ) ≥ 0 \mathbf{p}_J=\mathbf{p}_J(x)\ge0 p J = p J ( x ) ≥ 0 , the relations below hold, the last one by Moreau’s decomposition (6):
A x ≤ b , A J x = b J , p J ≥ 0 , Q x + A J T p J = − c − d ( x ) . (92) A x\le b,\qquad A_Jx=b_J,\qquad\mathbf{p}_J\ge0,\qquad Qx+A_J^{\mathsf{T}}\mathbf{p}_J=-c-d(x). \tag*{(92)} A x ≤ b , A J x = b J , p J ≥ 0 , Q x + A J T p J = − c − d ( x ) . (92)
These are the KKT conditions of problem (1) with one inexact equation, whose residual is exactly − d ( x ) -d(x) − d ( x ) . Accordingly, for an index set J ⊆ [ m ] J\subseteq[m] J ⊆ [ m ] , let Υ J \Upsilon_J Υ J denote the solution set, possibly empty, of the exact system, in the variables ( y , s ) ∈ R n × R ∣ J ∣ (y,s)\in\mathbb{R}^n\times\mathbb{R}^{|J|} ( y , s ) ∈ R n × R ∣ J ∣ ,
Υ J : = { ( y , s ) : A y ≤ b , A J y = b J , s ≥ 0 , Q y + A J T s = − c } , \Upsilon_J:=\{(y,s):Ay\le b,\ A_Jy=b_J,\ s\ge0,\ Qy+A_J^{\mathsf{T}}s=-c\}, Υ J := {( y , s ) : A y ≤ b , A J y = b J , s ≥ 0 , Q y + A J T s = − c } ,
and let D J D_J D J be the set of stationarity residuals compatible with the other three relations of Υ J \Upsilon_J Υ J ,
D J : = { Q y + A J T s + c : A y ≤ b , A J y = b J , s ≥ 0 } ⊆ R n . D_J:=\{Qy+A_J^{\mathsf{T}}s+c:Ay\le b,\ A_Jy=b_J,\ s\ge0\}\subseteq\mathbb{R}^n. D J := { Q y + A J T s + c : A y ≤ b , A J y = b J , s ≥ 0 } ⊆ R n .
We use four facts. First, the set D J D_J D J is the image of a polyhedron under an affine map, hence a polyhedron, hence closed: a polyhedron is finitely generated and an affine image of a finitely generated set is finitely generated, so [36 Theorem 19.1] applies twice. Second, the set Υ J \Upsilon_J Υ J is nonempty if and only if 0 ∈ D J 0\in D_J 0 ∈ D J . Third, every feasible x x x satisfies − d ( x ) ∈ D J ( x ) -d(x)\in D_{J(x)} − d ( x ) ∈ D J ( x ) , by (92) . Finally, every ( y ^ , s ^ ) ∈ Υ J (\hat{y},\hat{s})\in\Upsilon_J ( y ^ , s ^ ) ∈ Υ J makes y ^ \hat{y} y ^ an optimal solution of problem (1), since y ^ \hat{y} y ^ is feasible and convexity of f f f gives f ( y ) − f ( y ^ ) ≥ ⟨ ∇ x f ( y ^ ) , y − y ^ ⟩ = − ⟨ s ^ , A J y − b J ⟩ ≥ 0 f(y)-f(\hat{y})\ge\langle\nabla_xf(\hat{y}),y-\hat{y}\rangle=-\langle\hat{s},A_Jy-b_J\rangle\ge0 f ( y ) − f ( y ^ ) ≥ ⟨ ∇ x f ( y ^ ) , y − y ^ ⟩ = − ⟨ s ^ , A J y − b J ⟩ ≥ 0 at every feasible y y y .
Lemma B.20. Suppose problem (1) admits an optimal solution. There exist ϵ 0 > 0 \epsilon_0>0 ϵ 0 > 0 and κ 0 > 0 \kappa_0>0 κ 0 > 0 , depending only on A A A , b b b , Q Q Q , and c c c , such that every x ∈ C x\in C x ∈ C with ∥ d ( x ) ∥ 2 < ϵ 0 \lVert d(x)\rVert_2<\epsilon_0 ∥ d ( x ) ∥ 2 < ϵ 0 satisfies
V ( x ) − V s ≤ κ 0 ∥ d ( x ) ∥ 2 2 . (93) V(x)-V^s\le\kappa_0\lVert d(x)\rVert_2^2. \tag*{(93)} V ( x ) − V s ≤ κ 0 ∥ d ( x ) ∥ 2 2 . (93)
Proof. Define ϵ 0 : = min { 1 } ∪ { dist ( 0 , D J ) : J with 0 ∉ D J } \epsilon_0:=\min\{1\}\cup\{\operatorname{dist}(0,D_J):J\text{ with }0\notin D_J\} ϵ 0 := min { 1 } ∪ { dist ( 0 , D J ) : J with 0 ∈ / D J } with dist ( 0 , ∅ ) : = + ∞ \operatorname{dist}(0,\varnothing):=+\infty dist ( 0 , ∅ ) := + ∞ . The minimum runs over 1 1 1 and finitely many distances to closed sets that avoid the origin, so ϵ 0 ∈ ( 0 , 1 ] \epsilon_0\in(0,1] ϵ 0 ∈ ( 0 , 1 ] . For each J J J with Υ J ≠ ∅ \Upsilon_J\ne\varnothing Υ J = ∅ , let h J h_J h J denote the constant of the error bound [60 ] for the linear system defining Υ J \Upsilon_J Υ J , which depends only on the coefficient matrices, not on the right-hand sides, and set
κ 0 : = λ max 2 max { h J 2 : Υ J ≠ ∅ } , \kappa_0:=\frac{\lambda_{\max}}{2}\max\{h_J^2:\Upsilon_J\ne\varnothing\}, κ 0 := 2 λ m a x max { h J 2 : Υ J = ∅ } ,
a maximum over a nonempty finite set since every optimal x x x satisfies ( x , p J ( x ) ( x ) ) ∈ Υ J ( x ) (x,p_{J(x)}(x))\in\Upsilon_{J(x)} ( x , p J ( x ) ( x )) ∈ Υ J ( x ) , by (92) with d ( x ) = 0 d(x)=0 d ( x ) = 0 . If it vanishes, replace κ 0 \kappa_0 κ 0 by any positive number. Now let x x x be feasible with ∥ d ( x ) ∥ 2 < ϵ 0 \lVert d(x)\rVert_2<\epsilon_0 ∥ d ( x ) ∥ 2 < ϵ 0 and put J : = J ( x ) J:=J(x) J := J ( x ) . Since − d ( x ) ∈ D J -d(x)\in D_J − d ( x ) ∈ D J , the case 0 ∉ D J 0\notin D_J 0 ∈ / D J would give ∥ d ( x ) ∥ 2 ≥ dist ( 0 , D J ) ≥ ϵ 0 \lVert d(x)\rVert_2\ge\operatorname{dist}(0,D_J)\ge\epsilon_0 ∥ d ( x ) ∥ 2 ≥ dist ( 0 , D J ) ≥ ϵ 0 , a contradiction. Hence Υ J ≠ ∅ \Upsilon_J\ne\varnothing Υ J = ∅ . The point ( x , p J ( x ) ) (x,p_J(x)) ( x , p J ( x )) satisfies every relation defining Υ J \Upsilon_J Υ J exactly except the stationarity equation, whose violation is − d ( x ) -d(x) − d ( x ) , so the error bound yields ( y ^ , s ^ ) ∈ Υ J (\hat{y},\hat{s})\in\Upsilon_J ( y ^ , s ^ ) ∈ Υ J with ∥ x − y ^ ∥ 2 ≤ h J ∥ d ( x ) ∥ 2 \lVert x-\hat{y}\rVert_2\le h_J\lVert d(x)\rVert_2 ∥ x − y ^ ∥ 2 ≤ h J ∥ d ( x ) ∥ 2 . Finally, since V ( y ^ ) = V s V(\hat{y})=V^s V ( y ^ ) = V s by the fourth fact above, expanding V V V at y ^ \hat{y} y ^ and using Q y ^ + c = − A J T s ^ Q\hat{y}+c=-A_J^{\mathsf{T}}\hat{s} Q y ^ + c = − A J T s ^ and A J x = A J y ^ = b J A_Jx=A_J\hat{y}=b_J A J x = A J y ^ = b J ,
V ( x ) − V s = − ⟨ s ^ , A J ( x − y ^ ) ⟩ + 1 2 ⟨ x − y ^ , Q ( x − y ^ ) ⟩ ≤ λ max 2 h J 2 ∥ d ( x ) ∥ 2 2 ≤ κ 0 ∥ d ( x ) ∥ 2 2 , V(x)-V^s=-\langle\hat{s},A_J(x-\hat{y})\rangle+\frac{1}{2}\langle x-\hat{y},Q(x-\hat{y})\rangle\le\frac{\lambda_{\max}}{2}h_J^2\lVert d(x)\rVert_2^2\le\kappa_0\lVert d(x)\rVert_2^2, V ( x ) − V s = − ⟨ s ^ , A J ( x − y ^ )⟩ + 2 1 ⟨ x − y ^ , Q ( x − y ^ )⟩ ≤ 2 λ m a x h J 2 ∥ d ( x ) ∥ 2 2 ≤ κ 0 ∥ d ( x ) ∥ 2 2 ,
where the first term vanishes.
Proof of Proposition 5.1. Suppose first that some iterate satisfies d ( K ) ≠ 0 d^{(K)}\ne0 d ( K ) = 0 and Δ ( K ) = + ∞ \Delta^{(K)}=+\infty Δ ( K ) = + ∞ , and write x = x ( K ) x=x^{(K)} x = x ( K ) , d = d ( K ) d=d^{(K)} d = d ( K ) , J = J ( x ) J=J(x) J = J ( x ) . Then Δ ∗ ( K ) = + ∞ \Delta_*^{(K)}=+\infty Δ ∗ ( K ) = + ∞ gives ⟨ d , Q d ⟩ = 0 \langle d,Qd\rangle=0 ⟨ d , Q d ⟩ = 0 , hence d ∈ ker Q d\in\ker Q d ∈ ker Q , since ⟨ d , Q d ⟩ = ∥ Q 1 / 2 d ∥ 2 2 \langle d,Qd\rangle=\lVert Q^{1/2}d\rVert_2^2 ⟨ d , Q d ⟩ = ∥ Q 1/2 d ∥ 2 2 for SPSD Q Q Q . Moreover, A J d = 0 A_Jd=0 A J d = 0 by the KKT condition (15) , [ A d ] i ≤ 0 [Ad]_i\le0 [ A d ] i ≤ 0 for i ∈ E ( x ) ∖ J i \in\mathcal{E}(x) \setminus J i ∈ E ( x ) ∖ J because d ∈ T C ( x ) d \in T_C(x) d ∈ T C ( x ) , and [ A d ] i ≤ 0 [Ad]_i \le0 [ A d ] i ≤ 0 for i ∉ E ( x ) i \notin\mathcal{E}(x) i ∈ / E ( x ) because Δ f e a s ( x ) = + ∞ \Delta_{\mathrm{feas}}(x) = +\infty Δ feas ( x ) = + ∞ in (14). Hence A d ≤ 0 Ad \le0 A d ≤ 0 . Finally, the optimality condition (13d) gives ⟨ c , d ⟩ = ⟨ Q x + c , d ⟩ − ⟨ x , Q d ⟩ = − ∥ d ∥ 2 2 \langle c,d\rangle= \langle Qx+c,d\rangle-\langle x,Qd\rangle= -\|d\|_2^2 ⟨ c , d ⟩ = ⟨ Q x + c , d ⟩ − ⟨ x , Q d ⟩ = − ∥ d ∥ 2 2 . The ray x + Δ d x+\Delta d x + Δ d is therefore feasible, and V ( x + Δ d ) = V ( x ) − Δ ∥ d ∥ 2 2 → − ∞ V(x+\Delta d) = V(x)-\Delta\|d\|_2^2 \to-\infty V ( x + Δ d ) = V ( x ) − Δ∥ d ∥ 2 2 → − ∞ , which proves (40) and unboundedness along the ray. In particular, no such iterate exists when problem (1) is bounded below, and V ( x ( k ) ) ≥ V s V(x^{(k)}) \ge V^s V ( x ( k ) ) ≥ V s holds there because the iterates are feasible, which proves the final claim.
Now suppose the problem is unbounded below and no iterate has Δ ( k ) = + ∞ \Delta^{(k)}=+\infty Δ ( k ) = + ∞ . No optimal solution exists, so Υ J = ∅ \Upsilon_J=\varnothing Υ J = ∅ for every J J J , every closed set D J D_J D J avoids the origin, and θ 0 : = min J dist ( 0 , D J ) > 0 \theta_0:=\min_J\operatorname{dist}(0,D_J)>0 θ 0 := min J dist ( 0 , D J ) > 0 (with dist ( 0 , ∅ ) = + ∞ \operatorname{dist}(0,\varnothing)=+\infty dist ( 0 , ∅ ) = + ∞ ). Since − d ( k ) ∈ D J ( k ) -d^{(k)}\in D_{J^{(k)}} − d ( k ) ∈ D J ( k ) , every iterate satisfies ∥ d ( k ) ∥ 2 ≥ θ 0 \|d^{(k)}\|_2\ge\theta_0 ∥ d ( k ) ∥ 2 ≥ θ 0 , so the iteration is infinite. If V ( x ( k ) ) V(x^{(k)}) V ( x ( k ) ) decreased to some V ∞ > − ∞ V_\infty>-\infty V ∞ > − ∞ , Lemma B.15 would produce a blocked tail, which Theorem B.1 excludes. Hence V ( x ( k ) ) → − ∞ V(x^{(k)})\to-\infty V ( x ( k ) ) → − ∞ .
Proof of Theorem 5.1
Assume throughout that problem (1) has an optimal solution. By the first part of the proof of Proposition 5.1, no iterate has Δ ( k ) = + ∞ \Delta^{(k)}=+\infty Δ ( k ) = + ∞ , and every step with d ( k ) ∈ ker Q d^{(k)}\in\ker Q d ( k ) ∈ ker Q is blocked. If V ( x ( k ) ) V(x^{(k)}) V ( x ( k ) ) decreased to some V ∞ > V s V_\infty>V^s V ∞ > V s , then every k k k would have ∥ d ( k ) ∥ 2 ≥ θ : = min ( ϵ 0 , ( V ∞ − V s ) / κ 0 ) > 0 \|d^{(k)}\|_2\ge\theta:=\min(\epsilon_0,\sqrt{(V_\infty-V^s)/\kappa_0})>0 ∥ d ( k ) ∥ 2 ≥ θ := min ( ϵ 0 , ( V ∞ − V s ) / κ 0 ) > 0 : either ∥ d ( k ) ∥ 2 ≥ ϵ 0 \|d^{(k)}\|_2\ge\epsilon_0 ∥ d ( k ) ∥ 2 ≥ ϵ 0 , or (86) gives κ 0 ∥ d ( k ) ∥ 2 2 ≥ V ( x ( k ) ) − V s ≥ V ∞ − V s \kappa_0\|d^{(k)}\|_2^2\ge V(x^{(k)})-V^s\ge V_\infty-V^s κ 0 ∥ d ( k ) ∥ 2 2 ≥ V ( x ( k ) ) − V s ≥ V ∞ − V s . Lemma B.15 would then yield a blocked tail, which Theorem B.1 excludes. Hence V ( x ( k ) ) ↓ V s V(x^{(k)})\downarrow V^s V ( x ( k ) ) ↓ V s .
Lemma B.21. Every step satisfies ∥ x ( k + 1 ) − x ( k ) ∥ 2 ≤ 2 ( V ( x ( k ) ) − V ( x ( k + 1 ) ) ) / ∥ d ( k ) ∥ 2 \|x^{(k+1)}-x^{(k)}\|_2\le2(V(x^{(k)})-V(x^{(k+1)}))/\|d^{(k)}\|_2 ∥ x ( k + 1 ) − x ( k ) ∥ 2 ≤ 2 ( V ( x ( k ) ) − V ( x ( k + 1 ) )) /∥ d ( k ) ∥ 2 . Moreover, if the values V ( x ( k ) ) V(x^{(k)}) V ( x ( k ) ) decrease to V s V^s V s , then every k k k with V ( x ( k ) ) − V s ≤ κ 0 ϵ 0 2 V(x^{(k)})-V^s\le\kappa_0\epsilon_0^2 V ( x ( k ) ) − V s ≤ κ 0 ϵ 0 2 satisfies
∑ l ≥ k ∥ x ( l + 1 ) − x ( l ) ∥ 2 ≤ 4 κ 0 V ( x ( k ) ) − V s , (94) \sum_{l\ge k}\|x^{(l+1)}-x^{(l)}\|_2 \le4\sqrt{\kappa_0}\sqrt{V(x^{(k)})-V^s}, \tag*{(94)} l ≥ k ∑ ∥ x ( l + 1 ) − x ( l ) ∥ 2 ≤ 4 κ 0 V ( x ( k ) ) − V s , (94)
and the iterates converge to a point x s ∈ X s x^s\in X^s x s ∈ X s .
Proof. On a full step, evaluating (15) at Δ ∗ ( k ) \Delta_*^{(k)} Δ ∗ ( k ) gives V ( x ( k ) ) − V ( x ( k + 1 ) ) = 1 2 Δ ( k ) ∥ d ( k ) ∥ 2 2 V(x^{(k)})-V(x^{(k+1)})=\frac{1}{2}\Delta^{(k)}\|d^{(k)}\|_2^2 V ( x ( k ) ) − V ( x ( k + 1 ) ) = 2 1 Δ ( k ) ∥ d ( k ) ∥ 2 2 . On a blocked step, the proof of Lemma 5.1 gives V ( x ( k ) ) − V ( x ( k + 1 ) ) ≥ 1 2 Δ ( k ) ∥ d ( k ) ∥ 2 2 V(x^{(k)})-V(x^{(k+1)})\ge\frac{1}{2}\Delta^{(k)}\|d^{(k)}\|_2^2 V ( x ( k ) ) − V ( x ( k + 1 ) ) ≥ 2 1 Δ ( k ) ∥ d ( k ) ∥ 2 2 . Thus, since ∥ x ( k + 1 ) − x ( k ) ∥ 2 = Δ ( k ) ∥ d ( k ) ∥ 2 \|x^{(k+1)}-x^{(k)}\|_2=\Delta^{(k)}\|d^{(k)}\|_2 ∥ x ( k + 1 ) − x ( k ) ∥ 2 = Δ ( k ) ∥ d ( k ) ∥ 2 , dividing both by ∥ d ( k ) ∥ 2 / 2 \|d^{(k)}\|_2/2 ∥ d ( k ) ∥ 2 /2 gives the first claim. The numbers a k : = V ( x ( k ) ) − V s a_k:=V(x^{(k)})-V^s a k := V ( x ( k ) ) − V s are nonnegative and nonincreasing, and a k > 0 a_k>0 a k > 0 whenever step k k k is taken, since x ( k ) x^{(k)} x ( k ) is then not optimal. Let a k ≤ κ 0 ϵ 0 2 a_k\le\kappa_0\epsilon_0^2 a k ≤ κ 0 ϵ 0 2 , which holds for some k k k since a k → 0 a_k\to0 a k → 0 , and let l ≥ k l\ge k l ≥ k , so a l ≤ κ 0 ϵ 0 2 a_l\le\kappa_0\epsilon_0^2 a l ≤ κ 0 ϵ 0 2 . Then ∥ d ( l ) ∥ 2 ≥ a l / κ 0 \|d^{(l)}\|_2\ge\sqrt{a_l/\kappa_0} ∥ d ( l ) ∥ 2 ≥ a l / κ 0 , by (86) when ∥ d ( l ) ∥ 2 < ϵ 0 \|d^{(l)}\|_2<\epsilon_0 ∥ d ( l ) ∥ 2 < ϵ 0 , and because ∥ d ( l ) ∥ 2 ≥ ϵ 0 ≥ a l / κ 0 \|d^{(l)}\|_2\ge\epsilon_0\ge\sqrt{a_l/\kappa_0} ∥ d ( l ) ∥ 2 ≥ ϵ 0 ≥ a l / κ 0 otherwise. Hence
∥ x ( l + 1 ) − x ( l ) ∥ 2 ≤ 2 κ 0 a l − a l + 1 a l ≤ 4 κ 0 ( a l − a l + 1 ) , \|x^{(l+1)}-x^{(l)}\|_2\le2\sqrt{\kappa_0}\frac{a_l-a_{l+1}}{\sqrt{a_l}}\le4\sqrt{\kappa_0}(\sqrt{a_l}-\sqrt{a_{l+1}}), ∥ x ( l + 1 ) − x ( l ) ∥ 2 ≤ 2 κ 0 a l a l − a l + 1 ≤ 4 κ 0 ( a l − a l + 1 ) ,
using ( a l − a l + 1 ) / a l = ( a l − a l + 1 ) ( a l + a l + 1 ) / a l ≤ 2 ( a l − a l + 1 ) (a_l-a_{l+1})/\sqrt{a_l}=(\sqrt{a_l}-\sqrt{a_{l+1}})(\sqrt{a_l}+\sqrt{a_{l+1}})/\sqrt{a_l}\le2(\sqrt{a_l}-\sqrt{a_{l+1}}) ( a l − a l + 1 ) / a l = ( a l − a l + 1 ) ( a l + a l + 1 ) / a l ≤ 2 ( a l − a l + 1 ) , since 0 ≤ a l + 1 ≤ a l 0\le a_{l+1}\le a_l 0 ≤ a l + 1 ≤ a l , and the sum over l ≥ k l\ge k l ≥ k telescopes to (87). The iterates are then Cauchy. Their limit x s x^s x s lies in the closed set C C C , and V ( x s ) = lim k V ( x ( k ) ) = V s V(x^s)=\lim_kV(x^{(k)})=V^s V ( x s ) = lim k V ( x ( k ) ) = V s by continuity, so x s ∈ X s x^s\in X^s x s ∈ X s .
Part (i) of Theorem 5.1 follows. If d ( K ) = 0 d^{(K)}=0 d ( K ) = 0 for some K K K , then x ( K ) ∈ X s x^{(K)}\in X^s x ( K ) ∈ X s and x s = x ( K ) x^s=x^{(K)} x s = x ( K ) . Otherwise V ( x ( k ) ) ↓ V s V(x^{(k)})\downarrow V^s V ( x ( k ) ) ↓ V s as shown above and Lemma B.21 gives x ( k ) → x s ∈ X s x^{(k)}\to x^s\in X^s x ( k ) → x s ∈ X s .
For part (ii), set J s : = J ( x s ) = E ( x s ) J^s:=J(x^s)=\mathcal{E}(x^s) J s := J ( x s ) = E ( x s ) . By (LI), the vectors { A i T : i ∈ J s } \{A_i^\mathsf{T}:i\in J^s\} { A i T : i ∈ J s } are linearly independent. Hence the dual solution p s p^s p s is unique, and [ p s ] i > 0 [p^s]_i>0 [ p s ] i > 0 for every i ∈ J s i\in J^s i ∈ J s because J s J^s J s is the support of p s p^s p s . Write s 0 : = min i ∉ J s ( b i − [ A x s ] i ) > 0 s_0:=\min_{i\notin J^s}(b_i-[Ax^s]_i)>0 s 0 := min i ∈ / J s ( b i − [ A x s ] i ) > 0 for the least inactive slack at x s x^s x s . Let S ˉ : = S J s \bar{S}:=S_{J^s} S ˉ := S J s and R ˉ : = Im ( S ˉ ) \bar{R}:=\operatorname{Im}(\bar{S}) R ˉ := Im ( S ˉ ) . When R ˉ ≠ { 0 } \bar{R}\ne\{0\} R ˉ = { 0 } , let λ min ≤ λ max \lambda_{\min}\le\lambda_{\max} λ m i n ≤ λ m a x denote the smallest and largest positive eigenvalues of S ˉ \bar{S} S ˉ , so that κ ˉ = λ max / λ min \bar{\kappa}=\lambda_{\max}/\lambda_{\min} κ ˉ = λ m a x / λ m i n . The next lemma shows that, near x s x^s x s , the direction is linear on the face of x s x^s x s and bounded away from zero elsewhere.
Lemma B.22. There exist ρ 0 , δ 0 > 0 \rho_0,\delta_0>0 ρ 0 , δ 0 > 0 such that every feasible x x x with ∥ x − x s ∥ 2 ≤ ρ 0 \lVert x-x^s\rVert_2\leq\rho_0 ∥ x − x s ∥ 2 ≤ ρ 0 satisfies:
(i) E ( x ) ⊆ J s \mathcal{E}(x)\subseteq J^s E ( x ) ⊆ J s ;
(ii) if E ( x ) ≠ J s \mathcal{E}(x)\neq J^s E ( x ) = J s , then ∥ d ( x ) ∥ 2 ≥ δ 0 \lVert d(x)\rVert_2\geq\delta_0 ∥ d ( x ) ∥ 2 ≥ δ 0 ;
(iii) if E ( x ) = J s \mathcal{E}(x)=J^s E ( x ) = J s , then J ( x ) = J s J(x)=J^s J ( x ) = J s and d ( x ) = S ˉ ( x s − x ) ∈ R ˉ d(x)=\bar{S}(x^s-x)\in\bar{R} d ( x ) = S ˉ ( x s − x ) ∈ R ˉ , so that ∥ d ( x ) ∥ 2 ≤ λ max ∥ x − x s ∥ 2 \lVert d(x)\rVert_2\leq\lambda_{\max}\lVert x-x^s\rVert_2 ∥ d ( x ) ∥ 2 ≤ λ m a x ∥ x − x s ∥ 2 .
Proof. For i ∉ J s i\notin J^s i ∈ / J s the slack satisfies b i − [ A x ] i ≥ s 0 − ∥ A ∥ 2 ∥ x − x s ∥ 2 ≥ s 0 / 2 b_i-[Ax]_i\geq s_0-\lVert A\rVert_2\lVert x-x^s\rVert_2\geq s_0/2 b i − [ A x ] i ≥ s 0 − ∥ A ∥ 2 ∥ x − x s ∥ 2 ≥ s 0 /2 once ρ 0 ≤ s 0 / ( 2 ∥ A ∥ 2 ) \rho_0\leq s_0/(2\lVert A\rVert_2) ρ 0 ≤ s 0 / ( 2 ∥ A ∥ 2 ) . This proves (i). For (ii), by (6), ∥ d ( x ) ∥ 2 = dist ( − ∇ x f ( x ) , N C ( x ) ) \lVert d(x)\rVert_2=\operatorname{dist}(-\nabla_xf(x),N_C(x)) ∥ d ( x ) ∥ 2 = dist ( − ∇ x f ( x ) , N C ( x )) . By (i), E ( x ) \mathcal{E}(x) E ( x ) is a proper subset of J s J^s J s . We claim that dist ( − ∇ x f ( x s ) , N C ( x ) ) > 0 \operatorname{dist}(-\nabla_xf(x^s),N_C(x))>0 dist ( − ∇ x f ( x s ) , N C ( x )) > 0 . Otherwise − ∇ x f ( x s ) ∈ N C ( x ) -\nabla_xf(x^s)\in N_C(x) − ∇ x f ( x s ) ∈ N C ( x ) has a representation supported on E ( x ) \mathcal{E}(x) E ( x ) , which, compared with p s p^s p s over the linearly independent { A i T : i ∈ J s } \{A_i^\mathsf{T}:i\in J^s\} { A i T : i ∈ J s } , forces [ p s ] i = 0 [p^s]_i=0 [ p s ] i = 0 for i ∈ J s ∖ E ( x ) ≠ ∅ i\in J^s\setminus\mathcal{E}(x)\neq\varnothing i ∈ J s ∖ E ( x ) = ∅ and contradicts the hypothesis. The cone N C ( x ) N_C(x) N C ( x ) depends only on E ( x ) \mathcal{E}(x) E ( x ) , which takes finitely many values, so δ 0 : = 1 2 min { dist ( − ∇ x f ( x s ) , N C ( y ) ) : y ∈ C , E ( y ) ⊊ J s } \delta_0:=\frac{1}{2}\min\{\operatorname{dist}(-\nabla_xf(x^s),N_C(y)):y\in C,\mathcal{E}(y)\subsetneq J^s\} δ 0 := 2 1 min { dist ( − ∇ x f ( x s ) , N C ( y )) : y ∈ C , E ( y ) ⊊ J s } is positive. Hence
∥ d ( x ) ∥ 2 ≥ dist ( − ∇ x f ( x s ) , N C ( x ) ) − ∥ ∇ x f ( x ) − ∇ x f ( x s ) ∥ 2 ≥ 2 δ 0 − λ max ρ 0 , \lVert d(x)\rVert_2\geq\operatorname{dist}(-\nabla_xf(x^s),N_C(x))-\lVert\nabla_xf(x)-\nabla_xf(x^s)\rVert_2\geq2\delta_0-\lambda_{\max}\rho_0, ∥ d ( x ) ∥ 2 ≥ dist ( − ∇ x f ( x s ) , N C ( x )) − ∥ ∇ x f ( x ) − ∇ x f ( x s ) ∥ 2 ≥ 2 δ 0 − λ m a x ρ 0 , and ρ 0 ≤ δ 0 / λ max \rho_0\leq\delta_0/\lambda_{\max} ρ 0 ≤ δ 0 / λ m a x gives (ii).
The least-squares multiplier p J s ( x ) p_{J^s}(x) p J s ( x ) of (60) is affine in x x x with p J s ( x s ) = p s > 0 p_{J^s}(x^s)=p^s>0 p J s ( x s ) = p s > 0 , so shrinking ρ 0 \rho_0 ρ 0 keeps p J s ( x ) > 0 p_{J^s}(x)>0 p J s ( x ) > 0 on the ball. Since E ( x ) = J s \mathcal{E}(x)=J^s E ( x ) = J s , Lemma B.2 with K = T C ( x ) \mathcal{K}=T_C(x) K = T C ( x ) gives d ( x ) = − P J s ⊥ ∇ x f ( x ) d(x)=-P_{J^s}^{\perp}\nabla_xf(x) d ( x ) = − P J s ⊥ ∇ x f ( x ) . Every NNLS solution at x x x is supported in E ( x ) = J s \mathcal{E}(x)=J^s E ( x ) = J s , whose rows are linearly independent, so p ( x ) = p J s ( x ) p(x)=p_{J^s}(x) p ( x ) = p J s ( x ) and J ( x ) = J s J(x)=J^s J ( x ) = J s . Finally P J s ⊥ ∇ x f ( x s ) = − d ( x s ) = 0 P_{J^s}^{\perp}\nabla_xf(x^s)=-d(x^s)=0 P J s ⊥ ∇ x f ( x s ) = − d ( x s ) = 0 , since x s x^s x s is optimal, so d ( x ) = − P J s ⊥ Q ( x − x s ) = S ˉ ( x s − x ) d(x)=-P_{J^s}^{\perp}Q(x-x^s)=\bar{S}(x^s-x) d ( x ) = − P J s ⊥ Q ( x − x s ) = S ˉ ( x s − x ) , since x − x s ∈ ker A J s x-x^s\in\ker A_{J^s} x − x s ∈ ker A J s . This proves (iii).
On the affine set F : = { x : A J s x = b J s } \mathcal{F}:=\{x:A_{J^s}x=b_{J^s}\} F := { x : A J s x = b J s } , write e : = x − x s ∈ ker A J s e:=x-x^s\in\ker A_{J^s} e := x − x s ∈ ker A J s and e R ˉ e_{\bar{R}} e R ˉ for its component in R ˉ \bar{R} R ˉ . Since ∇ x f ( x s ) = − A J s T p s \nabla_xf(x^s)=-A_{J^s}^{\mathsf{T}}p^s ∇ x f ( x s ) = − A J s T p s is orthogonal to ker A J s \ker A_{J^s} ker A J s , every feasible x ∈ F x\in\mathcal{F} x ∈ F satisfies V ( x ) − V s = 1 2 ⟨ e , Q e ⟩ = 1 2 ⟨ e R ˉ , S ˉ e R ˉ ⟩ V(x)-V^s=\frac{1}{2}\langle e,Qe\rangle=\frac{1}{2}\langle e_{\bar{R}},\bar{S}e_{\bar{R}}\rangle V ( x ) − V s = 2 1 ⟨ e , Q e ⟩ = 2 1 ⟨ e R ˉ , S ˉ e R ˉ ⟩ , hence
λ ˉ min 2 ∥ e R ˉ ∥ 2 ≤ V ( x ) − V s ≤ λ ˉ max 2 ∥ e R ˉ ∥ 2 when R ˉ ≠ { 0 } . (95) \frac{\bar{\lambda}_{\min}}{2}\lVert e_{\bar{R}}\rVert^2\leq V(x)-V^s\leq\frac{\bar{\lambda}_{\max}}{2}\lVert e_{\bar{R}}\rVert^2\qquad\text{when }\bar{R}\neq\{0\}. \tag*{(95)} 2 λ ˉ m i n ∥ e R ˉ ∥ 2 ≤ V ( x ) − V s ≤ 2 λ ˉ m a x ∥ e R ˉ ∥ 2 when R ˉ = { 0 } . (95)
Lemma B.23. There exists ρ 1 ∈ ( 0 , ρ 0 ] \rho_1\in(0,\rho_0] ρ 1 ∈ ( 0 , ρ 0 ] such that if some iterate satisfies ∥ x ( k ) − x s ∥ 2 ≤ ρ 1 \lVert x^{(k)}-x^s\rVert_2\leq\rho_1 ∥ x ( k ) − x s ∥ 2 ≤ ρ 1 and E ( k ) = J s \mathcal{E}^{(k)}=J^s E ( k ) = J s , then every k ′ ≥ k k'\geq k k ′ ≥ k satisfies E ( k ′ ) = J ( k ′ ) = J s \mathcal{E}^{(k')}=J^{(k')}=J^s E ( k ′ ) = J ( k ′ ) = J s and ∥ x ( k ′ ) − x s ∥ 2 ≤ ρ 0 \lVert x^{(k')}-x^s\rVert_2\leq\rho_0 ∥ x ( k ′ ) − x s ∥ 2 ≤ ρ 0 , every step k ′ ≥ k k'\geq k k ′ ≥ k is full, and (42) holds for every k ′ ≥ k k'\geq k k ′ ≥ k at which the iteration has not stopped.
Proof. If R ˉ = { 0 } \bar{R}=\{0\} R ˉ = { 0 } , Lemma B.22(iii) gives d ( k ) = 0 d^{(k)}=0 d ( k ) = 0 , so the iteration stops at x ( k ) x^{(k)} x ( k ) and the claims hold with ρ 1 : = ρ 0 \rho_1:=\rho_0 ρ 1 := ρ 0 . Otherwise, choose ρ 1 : = min ( ρ 0 / κ ˉ , s 0 λ ˉ min / ( 4 ∥ A ∥ 2 λ max κ ˉ ) ) \rho_1:=\min(\rho_0/\sqrt{\bar{\kappa}},s_0\bar{\lambda}_{\min}/(4\lVert A\rVert_2\lambda_{\max}\sqrt{\bar{\kappa}})) ρ 1 := min ( ρ 0 / κ ˉ , s 0 λ ˉ m i n / ( 4 ∥ A ∥ 2 λ m a x κ ˉ )) and write e ( k ′ ) : = x ( k ′ ) − x s e^{(k')}:=x^{(k')}-x^s e ( k ′ ) := x ( k ′ ) − x s . We show by induction that every iterate from x ( k ) x^{(k)} x ( k ) onward lies in F ∩ { x : ∥ x − x s ∥ 2 ≤ κ ˉ ρ 1 } \mathcal{F}\cap\{x:\lVert x-x^s\rVert_2\leq\sqrt{\bar{\kappa}}\rho_1\} F ∩ { x : ∥ x − x s ∥ 2 ≤ κ ˉ ρ 1 } with E = J = J s \mathcal{E}=J=J^s E = J = J s , that its component orthogonal to R ˉ \bar{R} R ˉ equals that of e ( k ) e^{(k)} e ( k ) , and that it takes a full step. Given the hypothesis at k ′ k' k ′ , Lemma B.22(iii) applies at x ( k ′ ) x^{(k')} x ( k ′ ) , since κ ˉ ρ 1 ≤ ρ 0 \sqrt{\bar{\kappa}}\rho_1\leq\rho_0 κ ˉ ρ 1 ≤ ρ 0 , and gives J ( k ′ ) = J s J^{(k')}=J^s J ( k ′ ) = J s , d ( k ′ ) = − S ˉ e ( k ′ ) ∈ R ˉ d^{(k')}=-\bar{S}e^{(k')}\in\bar{R} d ( k ′ ) = − S ˉ e ( k ′ ) ∈ R ˉ , and
∥ d ( k ′ ) ∥ 2 ≤ λ max ∥ x ( k ′ ) − x s ∥ 2 ≤ λ max κ ˉ ρ 1 ≤ s 0 λ ˉ min 4 ∥ A ∥ 2 . \lVert d^{(k')}\rVert_2\leq\lambda_{\max}\lVert x^{(k')}-x^s\rVert_2\leq\lambda_{\max}\sqrt{\bar{\kappa}}\rho_1\leq\frac{s_0\bar{\lambda}_{\min}}{4\lVert A\rVert_2}. ∥ d ( k ′ ) ∥ 2 ≤ λ m a x ∥ x ( k ′ ) − x s ∥ 2 ≤ λ m a x κ ˉ ρ 1 ≤ 4 ∥ A ∥ 2 s 0 λ ˉ m i n . Since d ( k ′ ) ∈ R ˉ d^{(k')}\in\bar{R} d ( k ′ ) ∈ R ˉ , we have ⟨ d ( k ′ ) , Q d ( k ′ ) ⟩ = ⟨ d ( k ′ ) , S ˉ d ( k ′ ) ⟩ ≥ λ ˉ min ∥ d ( k ′ ) ∥ 2 2 \langle d^{(k')},Qd^{(k')}\rangle=\langle d^{(k')},\bar{S}d^{(k')}\rangle\geq\bar{\lambda}_{\min}\lVert d^{(k')}\rVert_2^2 ⟨ d ( k ′ ) , Q d ( k ′ ) ⟩ = ⟨ d ( k ′ ) , S ˉ d ( k ′ ) ⟩ ≥ λ ˉ m i n ∥ d ( k ′ ) ∥ 2 2 , so Δ ∗ ( k ′ ) ≤ 1 / λ ˉ min \Delta_*^{(k')}\leq1/\bar{\lambda}_{\min} Δ ∗ ( k ′ ) ≤ 1/ λ ˉ m i n . Every constraint outside J s J^s J s has slack at least s 0 / 2 s_0/2 s 0 /2 at x ( k ′ ) x^{(k')} x ( k ′ ) by the proof of Lemma B.22(i), so
Δ f e a s ( x ( k ′ ) ) ≥ s 0 / 2 ∥ A ∥ 2 ∥ d ( k ′ ) ∥ 2 ≥ 2 λ ˉ min > Δ ∗ ( k ′ ) . \Delta_{\mathrm{feas}}(x^{(k')})\geq\frac{s_0/2}{\lVert A\rVert_2\lVert d^{(k')}\rVert_2}\geq\frac{2}{\bar{\lambda}_{\min}}>\Delta_*^{(k')}. Δ feas ( x ( k ′ ) ) ≥ ∥ A ∥ 2 ∥ d ( k ′ ) ∥ 2 s 0 /2 ≥ λ ˉ m i n 2 > Δ ∗ ( k ′ ) .
and the step is full. It moves along R ˉ ⊆ ker A J s \bar{R} \subseteq\ker A_{J^{s}} R ˉ ⊆ ker A J s , so x ( k ′ + 1 ) ∈ F x^{(k'+1)} \in F x ( k ′ + 1 ) ∈ F , where the constraints of J s J^{s} J s are active and every other is strictly inactive because Δ ( k ′ ) < Δ f e a s ( x ( k ′ ) ) \Delta^{(k')} < \Delta_{\mathrm{feas}}(x^{(k')}) Δ ( k ′ ) < Δ feas ( x ( k ′ ) ) , giving E ( k ′ + 1 ) = J s \mathcal{E}^{(k'+1)} = J^{s} E ( k ′ + 1 ) = J s ; and the component of e ( k ′ + 1 ) e^{(k'+1)} e ( k ′ + 1 ) orthogonal to R ˉ \bar{R} R ˉ equals that of e ( k ′ ) e^{(k')} e ( k ′ ) , hence that of e ( k ) e^{(k)} e ( k ) . Since V V V is nonincreasing, (95) gives ∥ e R ˉ ( k ′ + 1 ) ∥ 2 2 ≤ κ ˉ ∥ e R ˉ ( k ′ ) ∥ 2 2 \|e_{\bar{R}}^{(k'+1)}\|_{2}^{2} \le\bar{\kappa}\|e_{\bar{R}}^{(k')}\|_{2}^{2} ∥ e R ˉ ( k ′ + 1 ) ∥ 2 2 ≤ κ ˉ ∥ e R ˉ ( k ′ ) ∥ 2 2 , so ∥ e ( k ′ + 1 ) ∥ 2 2 ≤ κ ˉ ∥ e R ˉ ( k ) ∥ 2 2 + ∥ e ( k ) − e R ˉ ( k ) ∥ 2 2 ≤ κ ˉ ∥ e ( k ) ∥ 2 2 ≤ κ ˉ ρ 1 2 \|e^{(k'+1)}\|_{2}^{2} \le\bar{\kappa}\|e_{\bar{R}}^{(k)}\|_{2}^{2}+\|e^{(k)}-e_{\bar{R}}^{(k)}\|_{2}^{2} \le\bar{\kappa}\|e^{(k)}\|_{2}^{2} \le\bar{\kappa}\rho_{1}^{2} ∥ e ( k ′ + 1 ) ∥ 2 2 ≤ κ ˉ ∥ e R ˉ ( k ) ∥ 2 2 + ∥ e ( k ) − e R ˉ ( k ) ∥ 2 2 ≤ κ ˉ ∥ e ( k ) ∥ 2 2 ≤ κ ˉ ρ 1 2 , closing the induction. On this tail, e R ˉ ( k ′ + 1 ) = e R ˉ ( k ′ ) − Δ ( k ′ ) S ˉ e R ˉ ( k ′ ) e_{\bar{R}}^{(k'+1)}=e_{\bar{R}}^{(k')}-\Delta^{(k')}\bar{S}e_{\bar{R}}^{(k')} e R ˉ ( k ′ + 1 ) = e R ˉ ( k ′ ) − Δ ( k ′ ) S ˉ e R ˉ ( k ′ ) with Δ ( k ′ ) = ∥ S ˉ e R ˉ ( k ′ ) ∥ 2 2 ⟨ S ˉ e R ˉ ( k ′ ) , S ˉ 2 e R ˉ ( k ′ ) ⟩ \Delta^{(k')}=\frac{\|\bar{S}e_{\bar{R}}^{(k')}\|_{2}^{2}}{\langle\bar{S}e_{\bar{R}}^{(k')},\bar{S}^{2}e_{\bar{R}}^{(k')}\rangle} Δ ( k ′ ) = ⟨ S ˉ e R ˉ ( k ′ ) , S ˉ 2 e R ˉ ( k ′ ) ⟩ ∥ S ˉ e R ˉ ( k ′ ) ∥ 2 2 , which is steepest descent with exact line search for the quadratic e ↦ 1 2 ⟨ e , S ˉ e ⟩ e\mapsto\frac{1}{2}\langle e,\bar{S}e\rangle e ↦ 2 1 ⟨ e , S ˉ e ⟩ on R ˉ \bar{R} R ˉ , where S ˉ \bar{S} S ˉ is positive definite with spectrum in [ λ ˉ min , λ ˉ max ] [\bar{\lambda}_{\min},\bar{\lambda}_{\max}] [ λ ˉ m i n , λ ˉ m a x ] . The Kantorovich bound [32 Theorem 3.3] , stated there for the norm e ↦ ⟨ e , S ˉ e ⟩ 1 / 2 e\mapsto\langle e,\bar{S}e\rangle^{1/2} e ↦ ⟨ e , S ˉ e ⟩ 1/2 , gives (46) , because V ( x ) − V s = 1 2 ⟨ e R ˉ , S ˉ e R ˉ ⟩ V(x)-V^{s}=\frac{1}{2}\langle e_{\bar{R}},\bar{S}e_{\bar{R}}\rangle V ( x ) − V s = 2 1 ⟨ e R ˉ , S ˉ e R ˉ ⟩ at every feasible x ∈ F x\in F x ∈ F .
Proposition B.5. Under the hypotheses of Theorem 5.1(ii), some iterate satisfies E ( k ) = J s \mathcal{E}^{(k)}=J^{s} E ( k ) = J s and ∥ x ( k ) − x s ∥ 2 ≤ ρ 1 \|x^{(k)}-x^{s}\|_{2}\le\rho_{1} ∥ x ( k ) − x s ∥ 2 ≤ ρ 1 .
Proof. If d ( K ) = 0 d^{(K)}=0 d ( K ) = 0 for some K K K , then x ( K ) = x s x^{(K)}=x^{s} x ( K ) = x s and the claim holds at k = K k=K k = K , so assume the iteration is infinite. By part (i), all late iterates have ∥ x ( k ) − x s ∥ 2 ≤ ρ 1 \|x^{(k)}-x^{s}\|_{2}\le\rho_{1} ∥ x ( k ) − x s ∥ 2 ≤ ρ 1 . If none of them had E ( k ) = J s \mathcal{E}^{(k)}=J^{s} E ( k ) = J s , then E ( k ) ⊊ J s \mathcal{E}^{(k)}\subsetneq J^{s} E ( k ) ⊊ J s by Lemma B.22 (i), and Lemma B.22 (ii) would give ∥ d ( k ) ∥ 2 ≥ δ 0 \|d^{(k)}\|_{2}\ge\delta_{0} ∥ d ( k ) ∥ 2 ≥ δ 0 for every late k k k . Since V ≥ V s V\ge V^{s} V ≥ V s , Lemma B.15 applied to the late run would then yield a blocked tail, contradicting Theorem B.1.
Let K ′ K' K ′ be the index of Proposition B.5. Lemma B.23 gives E ( k ) = J ( k ) = J s \mathcal{E}^{(k)}=J^{(k)}=J^{s} E ( k ) = J ( k ) = J s , a full step, and (46) at every k ≥ K ′ k\ge K' k ≥ K ′ at which the iteration has not stopped, which is (45) together with the rate. If S ˉ = 0 \bar{S}=0 S ˉ = 0 , then Lemma B.22 (iii) gives d ( K ′ ) = 0 d^{(K')}=0 d ( K ′ ) = 0 . □