betaFeedback

WorksMathematics

Inequality-Constrained Optimization via Sliding Mode Control

Abstract

We present COSMIC (Constrained Optimization via Sliding Mode for Inequality Constraints), a continuous-time solver for inequality-constrained optimization. COSMIC couples primal gradient-flow dynamics with multiplier dynamics derived from sliding mode control laws on a manifold defined by the smoothed Fischer-Burmeister function. We prove that COSMIC's trajectories reach this manifold in finite time from any primal initial point, after which the dynamics reduces to the gradient flow of a logarithmic barrier function. This yields a control-theoretic interpretation of barrier methods while allowing infeasible initialization. For strongly convex problems, we further establish exponential convergence to the minimizer. To address the numerical stiffness that arises in discretization, we develop a modified discrete-time algorithm. We validate COSMIC across various machine learning applications, demonstrating its ability to solve optimization problems accurately while scaling effectively to high-dimensional settings.

Introduction

We consider optimization problems of the form

min⁡x∈Rnf(x)s.t.g(x)≤0m,(1)\begin{aligned} \min_{x\in\mathbb{R}^{n}}\quad& f(x)\\ \text{s.t.}\quad& g(x) \le0_m, \tag*{(1)} \end{aligned}

where f:Rn→Rf:\mathbb{R}^{n}\to\mathbb{R} and g:Rn→Rmg:\mathbb{R}^{n}\to\mathbb{R}^{m} are assumed to be continuously differentiable throughout the paper. Inequality-constrained optimization problems, which subsume equality constraints, are prominent in various control, robotics, and machine learning applications, where the inequality constraints typically encode safety requirements or system and resource limitations. Classical approaches to solving (1) include projection-based methods, primal-dual algorithms, and penalty methods. The system-theoretic view of algorithms offers an alternative framework for analyzing and designing optimization algorithms through the lens of control theory [12]. This perspective has recently provided a control-theoretic interpretation of saddle-point flows, where the primal variables act as the plant of a dynamical system, and the Lagrange multipliers act as PID controllers that enforce constraint satisfaction [8, 6]. While PID controllers offer asymptotic convergence in these methods, sliding mode control (SMC) offers finite-time convergence and robustness to bounded uncertainty [18]. Recent work applies SMC for equality-constrained problems [11] with finite-time convergence. However, to date there is no first-order SMC-based solver for inequality-constrained problems, with convergence guarantees.

To fill this gap, we propose an SMC-based solver for (1). The resulting continuous-time optimization algorithm is computationally efficient and enjoys finite-time convergence guarantees. Moreover, it provides a control-theoretic interpretation of penalty methods, which, to the best of our knowledge, has not been established in the literature.

Related Work. There has been extensive work in viewing optimization algorithms as continuous time closed-loop systems with feedback controllers [16, 1, 4]. This perspective builds on the classical primal–dual (Arrow–Hurwicz–Uzawa) gradient flows [3], in which the multipliers act as an integral controller on the constraint violation. Chen and Ren [9] utilize nonlinear sign-based control to design projected gradient flows for linear-equality constrained convex problems. Within this line of work, [8, 6 b] set the multipliers to be PID controllers on the primal flow. Moreover, Zhang et al. [23] apply feedback linearization to inequality-constrained problems and relate the resulting algorithm to sequential quadratic programming. Closest to our work, Diana et al. [11] designs the dual dynamics of equality constrained problems as SMC controllers and Zenati and Youcef-Toumi [22] use discrete-time SMC to obtain a second-order solver for equality and inequality constrained problems, choosing a sliding manifold that encodes the KKT conditions and using the Fischer–Burmeister (FB) function.

Contributions. We introduce the Constrained Optimization via Sliding Mode for Inequality Constraints (COSMIC), a continuous-time flow for solving (1). Under COSMIC, the primal variables follow a gradient descent on the Lagrangian, while the multipliers are driven by SMC laws whose sliding manifold is the zero set of the smoothed FB function of the constraints and multipliers. On this manifold, primal and dual feasibility hold strictly and complementary slackness holds up to a perturbation ϵ>0\epsilon> 0. At equilibrium, the primal dynamics enforce stationarity, and the equilibria satisfy a perturbed KKT system, which reduces to the exact one as ϵ→0\epsilon\to0.

Our main results establish finite-time convergence to the sliding manifold and, for strongly convex problems, exponential convergence to the minimizer of (1). We also show that, once on the manifold, COSMIC reduces to the gradient flow of a logarithmic barrier method. This yields a control-theoretic interpretation of barrier methods and generalizes them, in the sense that our solver does not require a feasible initialization. To address the numerical difficulties that arise when discretizing COSMIC, we propose a modified discrete-time implementation. Finally, through training a support vector machine (SVM) on the Iris Setosa-vs-Versicolor dataset, and image completion with an input convex neural network (ICNN), we highlight the computational efficiency and accuracy of COSMIC relative to other nonlinear control-based optimization solvers. The code to reproduce the experiments is available at https://github.com/sudo-Ra-ONE/COSMIC.

Outline. Section 2 presents the mathematical preliminaries and is followed by the problem formulation and design of COSMIC, in Section 3. Section 4 formalizes the analysis and the numerical implementation of our solver, and Section 5 benchmarks it on various problems. Section 6 provides a summary and directions for future work. We provide a visual outline in Figure 1.

Visual Outline of the Paper

Figure 1. Visual Outline of the Paper

Mathematical Preliminaries

Notation. We denote by R\mathbb{R} and R>0\mathbb{R}_{>0} the real and positive real numbers, respectively, by ∥⋅∥\lVert\cdot\rVert the Euclidean norm on Rn\mathbb{R}^{n}, and by 0n\mathbf{0}_{n} and 1n\mathbf{1}_{n} the all-zeros and all-ones vectors in Rn\mathbb{R}^{n}. We let ⊘\oslash be the per-component division between two vectors. The class of kk-times continuously differentiable functions is CkC^{k}. For a differentiable f:Rn→Rf:\mathbb{R}^{n}\to\mathbb{R}, ∇f(x)\nabla f(x) is its gradient. Given h:Rn→Rmh:\mathbb{R}^{n}\to\mathbb{R}^{m} continuously differentiable, Jh(x)∈Rm×nJ_{h}(x)\in\mathbb{R}^{m\times n} denotes its Jacobian matrix. Given ρ,L>0\rho,L>0, a map f:Rn→Rf:\mathbb{R}^{n}\to\mathbb{R} is (i) ρ\rho-strongly convex if x↦f(x)−ρ2∥x∥2x\mapsto f(x)-\frac{\rho}{2}\lVert x\rVert^{2} is convex; (ii) LL-smooth if it is differentiable and ∇f\nabla f is LL-Lipschitz. The sign function is defined by sign⁡(x):=1\operatorname{sign}(x):=1 if x>0x>0, sign⁡(x):=−1\operatorname{sign}(x):=-1 if x<0x<0, and sign⁡(0):=0\operatorname{sign}(0):=0, and it is applied component-wise to vectors.

Optimality Conditions. Let F:={x∈Rn∣g(x)≤0m}\mathcal{F}:=\{x\in\mathbb{R}^{n}\mid g(x)\leq\mathbf{0}_{m}\} denote the feasible set of (1), f⋆:=inf⁡x∈Ff(x)f^{\star}:=\inf_{x\in\mathcal{F}}f(x) its optimal value, with the convention inf⁡∅=+∞\inf\varnothing=+\infty, and A(x):={i∈{1,…,m}∣gi(x)=0}\mathcal{A}(x):=\{i\in\{1,\ldots,m\}\mid g_{i}(x)=0\} denote the active set at x∈Fx\in\mathcal{F}. We say that linear independence constraint qualification (LICQ) holds at x∈Fx\in\mathcal{F} if the gradients {∇gi(x)}i∈A(x)\{\nabla g_{i}(x)\}_{i\in\mathcal{A}(x)} are linearly independent. A pair (x⋆,μ⋆)∈Rn+m(x^{\star},\mu^{\star})\in\mathbb{R}^{n+m} satisfies the Karush–Kuhn–Tucker (KKT) conditions if

∇f(x⋆)+Jg(x⋆)Tμ⋆=0n,(2)\nabla f(x^{\star})+J_{g}(x^{\star})^{\mathsf{T}}\mu^{\star}=\mathbf{0}_{n}, \tag*{(2)}
g(x⋆)≤0m,μ⋆≥0m,μi⋆gi(x⋆)=0,∀i=1,…,m.(3)g(x^{\star})\leq\mathbf{0}_{m},\quad\mu^{\star}\geq\mathbf{0}_{m},\quad\mu_{i}^{\star}g_{i}(x^{\star})=0,\quad\forall i=1,\ldots,m. \tag*{(3)}

For ϵ≥0\epsilon\geq0, let ϕϵ:R×R→R\phi_{\epsilon}:\mathbb{R}\times\mathbb{R}\to\mathbb{R}, ϕϵ(a,b):=a+b−a2+b2+ϵ2\phi_{\epsilon}(a,b):=a+b-\sqrt{a^{2}+b^{2}+\epsilon^{2}}. The Fischer–Burmeister (FB) function (Fischer, [14]) is ϕ0\phi_{0}, and ϕϵ\phi_{\epsilon}, ϵ>0\epsilon>0, is its smoothed version (Kanzow, [17]). For ϵ>0\epsilon>0, ϕϵ∈C1\phi_{\epsilon}\in C^{1} and ϕϵ(a,b)=0\phi_{\epsilon}(a,b)=0 if and only if a,b>0a,b>0 and ab=ϵ2/2ab=\epsilon^{2}/2. Hence, with a:=−gi(x)a:=-g_{i}(x) and b:=μib:=\mu_{i}, ϕϵ\phi_{\epsilon} encodes a perturbed version of the conditions in (3), which are exact for ϵ=0\epsilon=0.

Problem Formulation

We treat the primal dynamics of (1) as the plant and design the Lagrange multipliers as SMC-based feedback controllers that enforce the KKT conditions. SMC drives the system states to a predefined invariant, sliding manifold (Khalil, [18]), in two phases: a reaching phase, and a sliding phase, during which they slide toward the equilibrium. We next make this construction precise.

Consider the Lagrangian associated with (1), that is, the map L:Rn×Rm→RL:\mathbb{R}^{n}\times\mathbb{R}^{m}\to\mathbb{R} defined by L(x,μ)=f(x)+μTg(x)L(x,\mu)=f(x)+\mu^{\mathsf{T}}g(x), where μ∈R≥0m\mu\in\mathbb{R}_{\geq0}^{m} is the Lagrange multiplier of the constraints. Treating μ\mu as a control input to the gradient flow of LL with respect to xx yields the system

x˙=−∇xL(x,μ)=−∇f(x)−Jg(x)Tμ,(4)\dot{x}=-\nabla_{x}L(x,\mu)=-\nabla f(x)-J_{g}(x)^{\mathsf{T}}\mu, \tag*{(4)}

with output y=g(x)y=g(x). We design the input μ\mu via SMC so that the trajectories of (4) reach, and remain on, a manifold on which an ϵ\epsilon-perturbed KKT conditions hold. Fix ϵ>0\epsilon>0. With a slight abuse of notation, we consider ϕϵ:Rn×Rm→Rm\phi_{\epsilon}: \mathbb{R}^{n} \times\mathbb{R}^{m} \to\mathbb{R}^{m}, defined by ϕϵ(x,μ):=(ϕϵ(−g1(x),μ1),…,ϕϵ(−gm(x),μm))\phi_{\epsilon}(x,\mu) := \left(\phi_{\epsilon}(-g_{1}(x),\mu_{1}),\ldots,\phi_{\epsilon}(-g_{m}(x),\mu_{m})\right). Let the sliding manifold be given by S(x,μ)={(x,μ)∈Rn+m∣ϕϵ(x,μ)=0m}S(x,\mu) = \{(x,\mu) \in\mathbb{R}^{n+m} \mid\phi_{\epsilon}(x,\mu) = 0_{m}\}, where Si=ϕϵ,i(x,μ)S_{i} = \phi_{\epsilon,i}(x,\mu) for i∈{1,…,m}i \in\{1,\ldots,m\}, and consider the following sliding law

S˙i=−KSi−βsign⁡(Si),(5)\dot{S}_{i} = -KS_{i} - \beta\operatorname{sign}(S_{i}), \tag*{(5)}

where K>0K > 0 and β≥0\beta\ge0. Differentiating SS yields

S˙=ϕ˙ϵ=Dϵ(x,μ)μ˙−Gϵ(x,μ)Jgx˙,(6)\dot{S} = \dot{\phi}_{\epsilon} = D_{\epsilon}(x,\mu)\dot{\mu} - G_{\epsilon}(x,\mu)J_{g}\dot{x}, \tag*{(6)}

where Dϵ(x,μ),Gϵ(x,μ)∈Rm×mD_{\epsilon}(x,\mu),G_{\epsilon}(x,\mu) \in\mathbb{R}^{m \times m} are diagonal with entries Di=1−μigi(x)2+μi2+ϵ2D_{i} = 1 - \frac{\mu_{i}}{\sqrt{g_{i}(x)^{2}+\mu_{i}^{2}+\epsilon^{2}}} and Gi=1+gi(x)gi(x)2+μi2+ϵ2G_{i} = \frac{1+ \frac{g_{i}(x)}{\sqrt{g_{i}(x)^{2}+\mu_{i}^{2}+\epsilon^{2}}}}{ }. Since Di,Gi∈]0,2[D_{i},G_{i} \in]0,2[ for all ii and all (x,μ)(x,\mu), Dϵ(x,μ)D_{\epsilon}(x,\mu) is invertible. Combining (4) and (6) yields the closed loop dynamics

x˙=−∇f(x)−JgT(x)μ,μ˙=Dϵ−1(x,μ)(−Kϕϵ(x,μ)−βsign⁡(ϕϵ(x,μ))+Gϵ(x,μ)Jg(x)x˙),(7)\begin{aligned} \dot{x} &= -\nabla f(x) - J_{g}^{\mathsf{T}}(x)\mu,\\ \dot{\mu} &= D_{\epsilon}^{-1}(x,\mu)\left(-K\phi_{\epsilon}(x,\mu)-\beta\operatorname{sign}(\phi_{\epsilon}(x,\mu))+G_{\epsilon}(x,\mu)J_{g}(x)\dot{x}\right), \tag*{(7)} \end{aligned}

which we refer to as COSMIC. Because (7) is discontinuous, we consider solutions of (7) in the sense of Filippov [10]. Since f,g∈C1f,g \in C^{1} and Dϵ−1D_{\epsilon}^{-1} is continuous, the right-hand side of (7) is measurable and locally essentially bounded. Then, for every (x0,μ0)∈Rn+m(x_{0},\mu_{0}) \in\mathbb{R}^{n+m}, a Filippov solution of (7) exists [10 Proposition 3] on a maximal interval [0,Tmax⁡)[0,T_{\max}), with Tmax⁡∈]0,∞]T_{\max} \in]0,\infty].

Theorem 3.1 (Equilibria of (7) and KKT points of (1)). Consider the COSMIC dynamics (7) with K,ϵ>0K,\epsilon> 0, and β≥0\beta\ge0. Let Ωϵ\Omega_{\epsilon} denote the set of equilibria of (7). The following statements hold.

(i) (Perturbed KKT characterization) For every ϵ>0\epsilon> 0, (x^,μ^)∈Ωϵ(\hat{x},\hat{\mu}) \in\Omega_{\epsilon} if and only if ∇f(x^)+Jg(x^)Tμ^=0n\nabla f(\hat{x}) + J_{g}(\hat{x})^{\mathsf{T}}\hat{\mu} = 0_{n} and ϕϵ(x^,μ^)=0m\phi_{\epsilon}(\hat{x},\hat{\mu}) = 0_{m}.

(ii) (Limiting KKT points) Let (xk,μk)∈Ωϵk(x_{k},\mu_{k}) \in\Omega_{\epsilon_{k}} and ϵk→0\epsilon_{k} \to0. Then, every limit point (xˉ,μˉ)(\bar{x},\bar{\mu}) of {(xk,μk)}\{(x_{k},\mu_{k})\} satisfies the KKT conditions of (1).

(iii) (Multiplier convergence under LICQ) Let (xk,μk)∈Ωϵk(x_{k},\mu_{k}) \in\Omega_{\epsilon_{k}}, ϵk→0\epsilon_{k} \to0, and let xˉ\bar{x} be a limit point of {xk}\{x_{k}\} at which LICQ holds. Then, along every subsequence such that xk→xˉx_{k} \to\bar{x}, {μk}\{\mu_{k}\} converges to the unique μˉ\bar{\mu} such that (xˉ,μˉ)(\bar{x},\bar{\mu}) satisfies the KKT conditions of (1).

(iv) (Boundedness and suboptimality in the convex case) If ff and gig_{i}, i∈{1,…,m}i \in\{1,\ldots,m\}, are convex and the set of minimizers of (1) is nonempty and compact, then, for every ϵˉ>0\bar{\epsilon} > 0, the set of all equilibria obtained for ϵ∈]0,ϵˉ]\epsilon\in]0,\bar{\epsilon}], i.e., ⋃ϵ∈]0,ϵˉ]Ωϵ\bigcup_{\epsilon\in]0,\bar{\epsilon}]} \Omega_{\epsilon}, is bounded, and

0≤f(x)−f⋆≤mϵ22for all (x,μ)∈Ωϵ.(8)0 \le f(x)-f^{\star} \le\frac{m\epsilon^{2}}{2}\quad\text{for all }(x,\mu)\in\Omega_{\epsilon}. \tag*{(8)}

Consequently, any sequence {(xk,μk)}\{(x_{k},\mu_{k})\} as in (ii) is bounded and has at least one limit point satisfying the KKT conditions of (1).

Proof Item (i). Fix ϵ>0\epsilon> 0. Since Dϵ(x,μ)D_{\epsilon}(x,\mu) is invertible, (x^,μ^)∈Ωϵ(\hat{x},\hat{\mu}) \in\Omega_{\epsilon} if and only if ∇f(x^)+Jg(x^)Tμ^=0n\nabla f(\hat{x}) + J_{g}(\hat{x})^{\mathsf{T}}\hat{\mu} = 0_{n} and, using x^˙=0n\dot{\hat{x}} = 0_{n}, Kϕϵ(x^,μ^)+βsign⁡(ϕϵ(x^,μ^))=0mK\phi_{\epsilon}(\hat{x},\hat{\mu}) + \beta\operatorname{sign}(\phi_{\epsilon}(\hat{x},\hat{\mu})) = 0_{m}. Since K>0K > 0 and β≥0\beta\ge0, by definition of sign⁡(⋅)\operatorname{sign}(\cdot), the last equality holds if and only if ϕϵ(x^,μ^)=0m\phi_{\epsilon}(\hat{x},\hat{\mu}) = 0_m, which in turn is equivalent to g(x^)<0mg(\hat{x}) < 0_m, μ^>0m\hat{\mu} > 0_m, and μ^igi(x^)=−ϵ2/2\hat{\mu}_i g_i(\hat{x}) = -\epsilon^2/2 for all ii.

Item (ii). Let (xˉ,μˉ)(\bar{x},\bar{\mu}) be a limit point of {(xk,μk)}\{(x_k,\mu_k)\}. With a slight abuse of notation, we denote by {(xk,μk)}\{(x_k,\mu_k)\} also a subsequence converging to (xˉ,μˉ)(\bar{x},\bar{\mu}). Since (xk,μk)∈Ωϵk(x_k,\mu_k) \in\Omega_{\epsilon_k}, item (i) implies ∇f(xk)+Jg(xk)Tμk=0n\nabla f(x_k) + J_g(x_k)^{\mathsf{T}}\mu_k = 0_n, μk,igi(xk)=−ϵk2/2\mu_{k,i}g_i(x_k) = -\epsilon_k^2/2, g(xk)<0mg(x_k) < 0_m, and μk>0m\mu_k > 0_m, for all ii and kk. By continuity of ∇f\nabla f, JgJ_g, gg, passing to the limit yields (2) at (xˉ,μˉ)(\bar{x},\bar{\mu}).

Item (iii). Let (xk,μk)∈Ωϵk(x_k,\mu_k) \in\Omega_{\epsilon_k}, ϵk→0\epsilon_k \to0, and let xˉ\bar{x} be a limit point of {xk}\{x_k\} at which LICQ holds. Item (i) and continuity of gg implies g(xˉ)≤0mg(\bar{x}) \le0_m. Define the inactive constraints I:={1,…,m}∖A\mathcal{I} := \{1,\ldots,m\}\setminus\mathcal{A}, and let JA(x)J_{\mathcal{A}}(x) and JI(x)J_{\mathcal{I}}(x) be the rows of Jg(x)J_g(x) indexed by A\mathcal{A} and I\mathcal{I}, respectively, and use the same indexing also for μ\mu. By stationarity at (xk,μk)(x_k,\mu_k) we have

∇f(xk)+JA(xk)Tμk,A+JI(xk)Tμk,I=0n.(9)\nabla f(x_k) + J_{\mathcal{A}}(x_k)^{\mathsf{T}}\mu_{k,\mathcal{A}} + J_{\mathcal{I}}(x_k)^{\mathsf{T}}\mu_{k,\mathcal{I}} = 0_n. \tag*{(9)}

By LICQ, JA(x)JA(x)TJ_{\mathcal{A}}(x)J_{\mathcal{A}}(x)^{\mathsf{T}} is positive definite at xˉ\bar{x} and, by continuity, at some xkx_k for kk large enough. Multiplying both sides of (9) by JA(xk)J_{\mathcal{A}}(x_k) yields μk,A=−(JA(xk)JA(xk)T)−1JA(xk)(∇f(xk)+JI(xk)Tμk,I)\mu_{k,\mathcal{A}} = -\left(J_{\mathcal{A}}(x_k)J_{\mathcal{A}}(x_k)^{\mathsf{T}}\right)^{-1}J_{\mathcal{A}}(x_k)\left(\nabla f(x_k) + J_{\mathcal{I}}(x_k)^{\mathsf{T}}\mu_{k,\mathcal{I}}\right). Moreover, the inactive constraints’ multipliers vanish in the limit as k→∞k \to\infty, i.e., as xk→xˉx_k \to\bar{x} and ϵk→0\epsilon_k \to0 jointly. To see this, let i∈Ii \in\mathcal{I}, so that gi(xˉ)<0g_i(\bar{x}) < 0. By continuity, gi(xk)≤gi(xˉ)/2<0g_i(x_k) \le g_i(\bar{x})/2 < 0, for kk large enough, and by item (i), μk,igi(xk)=−ϵk2/2\mu_{k,i}g_i(x_k) = -\epsilon_k^2/2 and μk,i>0\mu_{k,i} > 0. Then, we have 0<μk,i=ϵk2−2gi(xk)≤ϵk2−gi(xˉ)→00 < \mu_{k,i} = \frac{\epsilon_k^2}{-2g_i(x_k)} \le\frac{\epsilon_k^2}{-g_i(\bar{x})} \to0 as k→∞k \to\infty, resulting in μˉI=0\bar{\mu}_{\mathcal{I}} = 0, and, by continuity of ∇f\nabla f and JgJ_g, μk,A→μˉA:=−(JA(xˉ)JA(xˉ)T)−1JA(xˉ)∇f(xˉ)\mu_{k,\mathcal{A}} \to\bar{\mu}_{\mathcal{A}} := -\left(J_{\mathcal{A}}(\bar{x})J_{\mathcal{A}}(\bar{x})^{\mathsf{T}}\right)^{-1}J_{\mathcal{A}}(\bar{x})\nabla f(\bar{x}). Hence μk→μˉ\mu_k \to\bar{\mu}, with μˉ=(μˉA,μˉI)\bar{\mu} = (\bar{\mu}_{\mathcal{A}},\bar{\mu}_{\mathcal{I}}). Finally, passing to the limit in (9) yields ∇f(xˉ)+Jg(xˉ)Tμˉ=0n\nabla f(\bar{x}) + J_g(\bar{x})^{\mathsf{T}}\bar{\mu} = 0_n. Moreover, μˉ≥0m\bar{\mu} \ge0_m is a limit of positive vectors with g(xˉ)≤0mg(\bar{x}) \le0_m, and μˉigi(xˉ)=0\bar{\mu}_i g_i(\bar{x}) = 0 for all ii, since gi(xˉ)=0g_i(\bar{x}) = 0 for i∈Ai \in\mathcal{A} and μˉi=0\bar{\mu}_i = 0 for i∈Ii \in\mathcal{I}.

Item (iv). Let ϵ>0\epsilon> 0 and (x,μ)∈Ωϵ(x,\mu) \in\Omega_{\epsilon}. By item (i), g(x)<0mg(x) < 0_m, μ>0m\mu> 0_m, μigi(x)=−ϵ2/2\mu_i g_i(x) = -\epsilon^2/2 for all ii. Hence xx is feasible and f(x)≥f⋆f(x) \ge f^\star. Since μ>0m\mu> 0_m and ff, gig_i are convex, z↦L(z,μ)z \mapsto L(z,\mu) is convex and, by stationarity, ∇xL(x,μ)=0n\nabla_x L(x,\mu) = 0_n. Hence, xx minimizes L(⋅,μ)L(\cdot,\mu) and, for all z∈Rnz \in\mathbb{R}^n,

f(z)+∑i=1mμigi(z)=L(z,μ)≥L(x,μ)=f(x)−mϵ22.(10)f(z) + \sum_{i=1}^{m}\mu_i g_i(z) = L(z,\mu) \ge L(x,\mu) = f(x) - \frac{m\epsilon^2}{2}. \tag*{(10)}

Choosing z=x⋆z = x^\star as a minimizer, and using μigi(x⋆)≤0\mu_i g_i(x^\star) \le0 yields (8). Now let ϵˉ>0\bar{\epsilon} > 0 and (x,μ)∈Ωϵ(x,\mu) \in\Omega_{\epsilon} with ϵ∈]0,ϵˉ]\epsilon\in]0,\bar{\epsilon}]. Then x∈{x∈F∣f(x)≤f⋆+mϵˉ2/2}x \in\{x \in\mathcal{F} \mid f(x) \le f^\star+ m\bar{\epsilon}^2/2\}, which is bounded as a sublevel set of a convex problem whose set of minimizers is nonempty and compact [21 Corollary 8.7.1]. Hence, xx is bounded uniformly in ϵ∈]0,ϵˉ]\epsilon\in]0,\bar{\epsilon}]. Moreover, fixing any (xs,μs)∈⋃ϵ∈]0,ϵˉ]Ωϵ(x_s,\mu_s) \in\bigcup_{\epsilon\in]0,\bar{\epsilon}]} \Omega_{\epsilon}, so that g(xs)<0mg(x_s) < 0_m by item (i), and choosing z=xsz = x_s, inequality (10) and f(x)≥f⋆f(x) \ge f^\star yield ∑i=1mμi(−gi(xs))≤f(xs)−f⋆+mϵˉ22\sum_{i=1}^{m}\mu_i\left(-g_i(x_s)\right) \le f(x_s) - f^\star+ \frac{m\bar{\epsilon}^2}{2}. Since −gi(xs)>0-g_i(x_s) > 0 and μi>0\mu_i > 0 for all ii, each term of the sum is nonnegative, so 0<μi≤f(xs)−f⋆+mϵˉ2/2−gi(xs)0 < \mu_i \le\frac{f(x_s)-f^\star+m\bar{\epsilon}^2/2}{-g_i(x_s)}, i=1,…,mi=1,\ldots,m, uniformly in ϵ∈]0,ϵˉ]\epsilon\in]0,\bar{\epsilon}]. Therefore, ⋃ϵ∈]0,ϵˉ]Ωϵ\bigcup_{\epsilon\in]0,\bar{\epsilon}]} \Omega_{\epsilon} is bounded. Finally, let (xk,μk)∈Ωϵk(x_k,\mu_k) \in\Omega_{\epsilon_k} with ϵk→0\epsilon_k \to0, and let ϵˉ:=sup⁡kϵk\bar{\epsilon} := \sup_k \epsilon_k, which is finite since {ϵk}\{\epsilon_k\} converges. Then ϵk∈]0,ϵˉ]\epsilon_k \in]0,\bar{\epsilon}] for all kk, so {(xk,μk)}⊂⋃ϵ∈]0,ϵˉ]Ωϵ\{(x_k,\mu_k)\} \subset\bigcup_{\epsilon\in]0,\bar{\epsilon}]} \Omega_{\epsilon}, which is bounded by the argument above. Then, by the Bolzano–Weierstrass theorem, {(xk,μk)}\{(x_k,\mu_k)\} admits a limit point, which satisfies the KKT conditions of (1) by item (ii). ■

Theorem 3.1.(i) implies that Ωϵ\Omega_{\epsilon} does not depend on the sliding mode parameters K>0K > 0 and β≥0\beta\ge0. Moreover, items (i)–(iii) assume neither feasibility of (1) nor existence of a minimizer, they characterize the equilibria of (7) whenever these exist. Item (i) yields g(x)<0mg(x) < 0_m for every (x,μ)∈Ωϵ(x,\mu) \in\Omega_{\epsilon}, so Ωϵ=∅\Omega_{\epsilon} = \varnothing if (1) has no strictly feasible point, e.g., if it is infeasible. If ff and gig_i are convex, inequality (10), which only uses convexity, implies that Ωϵ=∅\Omega_{\epsilon} = \varnothing if f⋆=−∞f^\star= -\infty. Statement (ii) requires no constraint qualification, since it assumes that {(xk,μk)}\{(x_k,\mu_k)\} has a limit point, which in particular requires {μk}\{\mu_k\} to have a bounded subsequence. On the other hand, item (iii) requires only that {xk}\{x_k\} has a limit point and shows that LICQ at xˉ\bar{x} guarantees convergence of {μk}\{\mu_k\} to the (unique) KKT multiplier at xˉ\bar{x}. Without constraint qualification, the sequence {μk}\{\mu_k\} may diverge even if {xk}\{x_k\} converges to a minimizer. Consider, for example, min⁡x∈R{x:−x3≤0}\min_{x\in\mathbb{R}}\{x:-x^3\leq0\}, whose minimizer x⋆=0x^\star=0 violates LICQ, since ∇g(0)=0\nabla g(0)=0, and admits no KKT multiplier. The equilibrium of (7) is (xϵ,μϵ)=(3ϵ2/2,4/(27ϵ4))(x_\epsilon,\mu_\epsilon)=\left(3\epsilon^2/2,4/(27\epsilon^4)\right), so xϵ→x⋆x_\epsilon\to x^\star while μϵ→∞\mu_\epsilon\to\infty as ϵ→0\epsilon\to0. Finally, item (iv) specializes the results to convex problems and states that every equilibrium of (7) is strictly feasible and mϵ2/2m\epsilon^2/2-suboptimal for any fixed ϵ>0\epsilon>0, rather than only in the limit ϵ→0\epsilon\to0. In particular, choosing ϵ≤2δ/m\epsilon\leq\sqrt{2\delta/m} guarantees f(x)−f⋆≤δf(x)-f^\star\leq\delta for a prescribed accuracy δ>0\delta>0. Moreover, under the assumptions of item (iv) the boundedness of {μk}\{\mu_k\} implicitly assumed in item (ii) holds automatically and the existence of a limit point satisfying the KKT conditions is guaranteed.

Convergence Analysis

First, we show that trajectories of COSMIC reach the sliding manifold SS in finite time.

Lemma 4.1 (Finite-time reaching of the sliding manifold). Let β>0\beta>0, and let (x(t),μ(t))(x(t),\mu(t)) be a Filippov solution of (7) on [0,Tmax⁡)[0,T_{\max}). Define ϕi,0:=ϕϵ,i(x(0),μ(0))\phi_{i,0}:=\phi_{\epsilon,i}(x(0),\mu(0)), for i∈{1,…,m}i\in\{1,\ldots,m\}, let tr:=max⁡iK∣ϕi,0∣+βKβt_r:=\max_i \frac{K|\phi_{i,0}|+\beta}{K\beta} and assume that Tmax⁡>trT_{\max}>t_r. Then ϕϵ(x(t),μ(t))=0m\phi_\epsilon(x(t),\mu(t))=0_m for all t∈[tr,Tmax⁡)t\in[t_r,T_{\max}).

Proof. Since ϕϵ,i(x(t),μ(t))∈C1\phi_{\epsilon,i}(x(t),\mu(t))\in C^1 and (x(t),μ(t))(x(t),\mu(t)) is absolutely continuous, ϕi(t):=ϕϵ,i(x(t),μ(t))\phi_i(t):=\phi_{\epsilon,i}(x(t),\mu(t)) is absolutely continuous and ϕ˙i=Dϵ,iμ˙i−Gϵ,i∇gi(x)⊤x˙\dot{\phi}_i=D_{\epsilon,i}\dot{\mu}_i-G_{\epsilon,i}\nabla g_i(x)^\top\dot{x} satisfies the sliding law (5) with μ˙\dot{\mu} in (7) that is, we have ϕ˙i=−Kϕi−βsign⁡(ϕi)\dot{\phi}_i=-K\phi_i-\beta\operatorname{sign}(\phi_i), for almost every t∈[0,Tmax⁡)t\in[0,T_{\max}). In particular, this implies that each ϕi\phi_i evolves independently of xx and of ϕj\phi_j, for every j≠ij\ne i. If ϕi,0>0\phi_{i,0}>0, then, as long as ϕi(t)>0\phi_i(t)>0, we have ϕ˙i=−Kϕi−β\dot{\phi}_i=-K\phi_i-\beta, whose solution is ϕi(t)=(ϕi,0+βK)e−Kt−βK\phi_i(t)=\left(\phi_{i,0}+\frac{\beta}{K}\right)e^{-Kt}-\frac{\beta}{K}. Thus, ϕi\phi_i vanishes at ti:=1Kln⁡(Kϕi,0+ββ)t_i:=\frac{1}{K}\ln\left(\frac{K\phi_{i,0}+\beta}{\beta}\right). By symmetry, we have that if ϕi,0<0\phi_{i,0}<0, the reaching time is ti:=1Kln⁡(−Kϕi,0+ββ)t_i:=\frac{1}{K}\ln\left(\frac{-K\phi_{i,0}+\beta}{\beta}\right), and if ϕi,0=0\phi_{i,0}=0, then ti=0t_i=0. Moreover, {ϕi=0}\{\phi_i=0\} is forward invariant, since ϕ˙i≤−β<0\dot{\phi}_i\leq-\beta<0 whenever ϕi>0\phi_i>0 and ϕ˙i≥β>0\dot{\phi}_i\geq\beta>0 whenever ϕi<0\phi_i<0. Thus, ϕi(t)=0\phi_i(t)=0 for all t∈[ti,Tmax⁡)t\in[t_i,T_{\max}), and the statement follows by setting tr=max⁡itit_r=\max_i t_i.

The reaching time trt_r in Lemma 4.1 depends only on KK, β\beta and the initial conditions via ∣ϕi,0∣|\phi_{i,0}|. Moreover, trt_r is finite because β>0\beta>0. For β=0\beta=0, ϕϵ(x(t),μ(t))\phi_\epsilon(x(t),\mu(t)) converges to 0m0_m only exponentially. The finite-time convergence of COSMIC also serve as a feasibility certificate.

Corollary 1 (Feasibility certificate). In the setting of Lemma 4.1, if the trajectory of COSMIC starting from some (x0,μ0)(x_0,\mu_0) escapes to infinity at T<trT<t_r, then (1) admits no strictly feasible point.

To establish convergence rates we introduce the following assumption.

Assumption 1 (Problem structure). Given f:Rn→Rf:\mathbb{R}^n\to\mathbb{R} and g:Rn→Rmg:\mathbb{R}^n\to\mathbb{R}^m, assume

  1. the function ff is LfL_f-smooth and the functions gig_i are convex with LgL_g-Lipschitz gradient, for all i∈{1,…,m}i\in\{1,\ldots,m\};

  1. there exists xs∈Rnx_s\in\mathbb{R}^n such that g(xs)<0mg(x_s)<0_m (Slater condition).

Lemma 4.1 holds on [0,Tmax⁡)[0,T_{\max}), but it does not guarantee that Tmax⁡>trT_{\max}>t_r, let alone that Tmax⁡=∞T_{\max}=\infty. Since the primal dynamics is gradient-like, the solution of (7) can escape to infinity in finite time only if the dual dynamics does. The next result provides a sufficient condition to preclude this.

Lemma 4.2 (Global existence). Let (x(t),μ(t))(x(t),\mu(t)) be a Filippov solution of (7) with ϕϵ(x(0),μ(0))≤0m\phi_{\epsilon}(x(0),\mu(0))\leq0_m. Under Assumption 1, Tmax⁡=+∞T_{\max}=+\infty.

Proof. We provide a sketch of the proof and give the full proof in Appendix § 9. Let ϕi(t):=ϕϵ,i(x(t),μ(t))\phi_i(t):=\phi_{\epsilon,i}(x(t),\mu(t)). Since ϕi,0≤0\phi_{i,0}\leq0, by the proof of Lemma 4.1, ϕi(t)∈[ϕi,0,0]\phi_i(t)\in[\phi_{i,0},0] for all tt, and μi\mu_i can be written as an explicit function of xx and ϕi(t)\phi_i(t), which diverges only if xx approaches the boundary of {x:gi(x)<−ϕi(t)}\{x:g_i(x)<-\phi_i(t)\}. Let xsx_s be a Slater point and μi+:=max⁡{μi,0}\mu_i^+:=\max\{\mu_i,0\}. On any bounded time interval (i) by convexity of gig_i and Slater’s condition, ∥x−xs∥\lVert x-x_s\rVert and ∫∑iμi+ dt\int\sum_i\mu_i^+\,\mathrm{d}t are bounded and (ii) the function V(t,x):=f(x)+∑i=1m(ϕigi−ϕi2+ϵ22log⁡(−ϕi(t)−gi(x)))V(t,x):=f(x)+\sum_{i=1}^{m}\left(\phi_i g_i-\frac{\phi_i^2+\epsilon^2}{2}\log(-\phi_i(t)-g_i(x))\right), grows at most by a constant times ∫∑iμi+ dt\int\sum_i\mu_i^+\,\mathrm{d}t, and hence stays bounded. Since V→+∞V\to+\infty as xx approaches the boundary of {x:gi(x)<−ϕi(t)}\{x:g_i(x)<-\phi_i(t)\}, xx stays away from that boundary. Therefore, (x,μ)(x,\mu) remains bounded on every bounded time interval, and Tmax⁡=+∞T_{\max}=+\infty.

Note that μ(0)≤0m\mu(0)\leq0_m implies ϕϵ(x(0),μ(0))<0m\phi_{\epsilon}(x(0),\mu(0))<0_m, so Lemma 4.2 holds whenever μ(0)≤0m\mu(0)\leq0_m. The lemma implies that COSMIC can be initialized at any x(0)∈Rnx(0)\in\mathbb{R}^n together with, for example μ(0)=0m\mu(0)=0_m. Hence, no projection onto the feasible set is required prior to running the algorithm.

Theorem 4.1 (Convergence of COSMIC). Under Assumption 1, let ff be ρ\rho-strongly convex, let β>0\beta>0, and let (x(t),μ(t))(x(t),\mu(t)) be a Filippov solution of (7) with ϕϵ(x(0),μ(0))≤0m\phi_{\epsilon}(x(0),\mu(0))\leq0_m. Then, COSMIC converges to (xϵ∗,μϵ∗)(x_\epsilon^*,\mu_\epsilon^*). In particular,

∥x(t)−xϵ∗∥≤e−ρ(t−tr)∥x(tr)−xϵ∗∥for all t≥tr,\lVert x(t)-x_\epsilon^*\rVert\leq e^{-\rho(t-t_r)}\lVert x(t_r)-x_\epsilon^*\rVert\qquad\text{for all }t\geq t_r,

with trt_r as in Lemma 4.1, and μ(t)→μϵ∗\mu(t)\to\mu_\epsilon^* as t→∞t\to\infty.

Proof. By Lemma 4.2, Tmax⁡=+∞T_{\max}=+\infty. Next, by Lemma 4.1, ϕϵ(x(t),μ(t))=0m ∀t≥tr\phi_{\epsilon}(x(t),\mu(t))=0_m\ \forall t\geq t_r, which, by Theorem 3.1.(i) implies g(x(t))<0mg(x(t))<0_m, μ(t)>0m\mu(t)>0_m and μi(t)gi(x(t))=−ϵ2/2\mu_i(t)g_i(x(t))=-\epsilon^2/2, ∀i\forall i, and ∀t≥tr\forall t\geq t_r. Substituting μi=−ϵ2/(2gi(x))\mu_i=-\epsilon^2/(2g_i(x)) into (7) yields x˙=−∇f(x)+ϵ22JgT1m⊘g(x)=−∇Bϵ(x)\dot{x}=-\nabla f(x)+\frac{\epsilon^2}{2}J_g^\mathsf{T}1_m\oslash g(x)=-\nabla B_\epsilon(x), where Bϵ(x)=f(x)−ϵ22∑i=1mlog⁡(−gi(x))B_\epsilon(x)=f(x)-\frac{\epsilon^2}{2}\sum_{i=1}^{m}\log(-g_i(x)). That is, xx follows the gradient flow of the logarithmic barrier function of Fiacco–McCormick (Fiacco and McCormick, [13]) defined on the feasible set F\mathcal{F}. By assumption, Bϵ(x)B_\epsilon(x) is ρ\rho-strongly convex on the nonempty set {x∈Rn∣g(x)<0m}\{x\in\mathbb{R}^n\mid g(x)<0_m\}, and its sublevel sets are compact. Then, BϵB_\epsilon has a unique minimizer xϵ∗x_\epsilon^* and ∇Bϵ(xϵ∗)=0n\nabla B_\epsilon(x_\epsilon^*)=0_n. Along the solution of (7), ddt∥x−xϵ∗∥2≤−2ρ∥x−xϵ∗∥2\frac{\mathrm{d}}{\mathrm{d}t}\lVert x-x_\epsilon^*\rVert^2\leq-2\rho\lVert x-x_\epsilon^*\rVert^2, hence ∥x(t)−xϵ∗∥≤e−ρ(t−tr)∥x(tr)−xϵ∗∥\lVert x(t)-x_\epsilon^*\rVert\leq e^{-\rho(t-t_r)}\lVert x(t_r)-x_\epsilon^*\rVert. Finally, since g(xϵ∗)<0mg(x_\epsilon^*)<0_m, continuity of gg gives μ(t)=−ϵ221m⊘g(x(t))→μϵ∗\mu(t)=-\frac{\epsilon^2}{2}1_m\oslash g(x(t))\to\mu_\epsilon^*.

Under the assumptions of Theorem 4.1, problem (1) has a unique minimizer and satisfies Slater’s condition. By Theorem 3.1.(iv), from any x(0)∈Rnx(0)\in\mathbb{R}^n and, e.g., μ(0)=0m\mu(0)=0_m, COSMIC reaches the strictly feasible region in finite time and converges exponentially to xϵ∗x_\epsilon^* with f(xϵ∗)−f∗≤mϵ2/2f(x_\epsilon^*)-f^*\leq m\epsilon^2/2.

Relation with barrier methods

Logarithmic barrier methods (see, e.g., Boyd and Vandenbergh, [5], Ch. 11) approximate (1) by the family of problems

min⁡x∈Rnf(x)−1p∑i=1mlog⁡(−gi(x)),p>0,(11)\min_{x\in\mathbb{R}^n} f(x)-\frac{1}{p}\sum_{i=1}^{m}\log\left(-g_i(x)\right),\qquad p>0, \tag*{(11)}

whose solutions x⋆x^\star define the central path. Setting λp,i⋆:=−1/(pgi(x⋆))\lambda^\star_{p,i}:=-1/(pg_i(x^\star)), the optimality conditions of (11) can be written as the perturbed KKT system

∇f(x)+Jg(x)Tλ=0n,g(x)<0m,λ>0m,λigi(x)=−1p,i∈{1,…,m},(12)\nabla f(x) + J_g(x)^{\mathsf T}\lambda= 0_n,\quad g(x) < 0_m,\quad\lambda> 0_m,\quad\lambda_i g_i(x) = -\frac{1}{p},\quad i \in\{1,\ldots,m\}, \tag*{(12)}

and, in the convex case, f(x⋆)−f⋆≤m/pf(x^\star)-f^\star\le m/p (Boyd and Vandenberghe, [5]). By Theorem 3.1.(i), COSMIC’s equilibria are the solutions of (12) with p=2/ϵ2p=2/\epsilon^2. In the convex case, the xx-components of Ωϵ\Omega_\epsilon are the minimizers of (11) with p=2/ϵ2p=2/\epsilon^2, and the bound in Theorem 3.1.(iv) recovers the classical duality gap m/pm/p. This provides a control-theoretic interpretation of the central path. Specifically, its points are the equilibria of (7). However, the barrier function in (11) is defined on the strict interior of the feasible set, so classical barrier methods must be initialized at a strictly feasible point, which in general requires solving an auxiliary phase (Boyd and Vandenberghe, [5]). In COSMIC, instead, μ\mu is a state of the dynamics driven by the SMC law (5), and strict feasibility is attained in finite time, once the trajectories reach the sliding manifold S\mathcal{S}.

Algorithm Design

Algorithm 1 is one step of the Euler discretized equivalents of (7) with two additional modifications. First, on the sliding manifold, Dϵ,i=1−μi/μi2+ϵi2D_{\epsilon,i}=1-\mu_i/\sqrt{\mu_i^2+\epsilon_i^2} for active constraints, so Dϵ−1D_\epsilon^{-1}, though well defined, becomes ill-conditioned as ϵ\epsilon decreases. We therefore replace Dϵ,i−1D_{\epsilon,i}^{-1} by (Dϵ,i2+δD2)−1/2≤1/δD(D_{\epsilon,i}^2+\delta_D^2)^{-1/2}\le1/\delta_D, with δD>0\delta_D>0. Second, to reduce chattering, we use the boundary-layer approach for SMCs and replace sign⁡(⋅)\operatorname{sign}(\cdot) by the component-wise saturation function (Khalil, [18]). Theorem 3.1 suggests decreasing ϵ\epsilon to approach a KKT point. We implement this via a vectorized ϵ\epsilon-homotopy that can be seen in Algorithm 2, where ϵi\epsilon_i is divided by rϵ>1r_\epsilon>1, with a lower bound ϵmin⁡>0\epsilon_{\min}>0 set to machine precision (in our case ϵmin⁡=10−16\epsilon_{\min}=10^{-16}). Specifically, ϵi\epsilon_i decays when ∣φϵ,i∣<cφϵi|\varphi_{\epsilon,i}|<c_\varphi\epsilon_i with cφ∈]0,1[c_\varphi\in]0,1[, as in path-following interior-point methods. Using a separate ϵi\epsilon_i per constraint lets ϵi\epsilon_i decay freely for inactive constraints, without being limited by the active constraints, for which a small ϵi\epsilon_i causes numerical stiffness. Unless stated otherwise, we use cφ=0.85c_\varphi=0.85 and rϵ=1.05r_\epsilon=1.05, which are values that we find to work well in practice. Since DϵD_\epsilon is diagonal, its inversion is computationally inexpensive and is implemented as a component-wise division of vectors.

Algorithm 1 COSMIC STEP() function
Input: x_k, μ_k, ε_k; gains K, β > 0; step size dt; saturation width ε_sat; damping floor δ_D
x_{k+1} ← x_k − dt(∇f(x_k) + J_g(x_k)^T μ_k)
s_k ← √(g(x_k)^2 + μ_k^2 + (ε_k)^2), φ_k ← μ_k − g(x_k) − s_k, d_k ← 1 − μ_k ⊙ s_k
D_{ε,k} ← diag(√(d_k^2 + δ_D^2))^{-1}, G_k ← diag(1 + g(x_k) ⊙ s_k)
μ_{k+1} ← μ_k − dt D_{ε,k}[Kφ_k + β sat(φ_k, ε_sat) + G_k J_g(x_k)(∇f(x_k) + J_g(x_k)^T μ_k)]
return x_{k+1}, μ_{k+1}, φ_k

Numerical Examples

We now validate COSMIC on SVM optimization for the Iris Setosa-vs-Versicolor dataset, and on image completion via ICNNs. We compare COSMIC to Feedback linearization (FL) Zhang et al.

Algorithm 2 COSMIC with smooth conditioning floor and safeguarded ϵ\epsilon-homotopy

Input: x0x_0; μ0\mu_0; ϵmin⁡\epsilon_{\min}; initial base smoothing ϵ0\epsilon_0; horizon NN; stationarity, feasibility, and complementarity tolerances (τs,τf,τc)(\tau_s,\tau_f,\tau_c); conditioning margin cDc_D; shrink safeguard cϕc_\phi; homotopy rate rϵr_\epsilon

while ∥∇f(xk)+Jg(xk)⊤μk∥>τs\left\lVert\nabla f(x_k)+J_g(x_k)^\top\mu_k\right\rVert>\tau_s, max⁡(g(xk),0)>τf\max\left(g(x_k),0\right)>\tau_f, max⁡i∣μk,igi(xk)∣>τc\max_i\left|\mu_{k,i}g_i(x_k)\right|>\tau_c, and k<Nk<N do

xk+1,μk+1,ϕk←STEP⁡(xk,μk,ϵk)x_{k+1},\mu_{k+1},\phi_k\leftarrow\operatorname{STEP}(x_k,\mu_k,\epsilon_k) (Algorithm 1)

m←1[∣ϕk,i∣<cϕϵk,i, i=1,…,m]m\leftarrow\mathbf{1}\left[\left|\phi_{k,i}\right|<c_\phi\epsilon_{k,i},\ i=1,\ldots,m\right], ϵshrunk←max⁡(ϵk/rϵ,ϵmin⁡)\epsilon_{\mathrm{shrunk}}\leftarrow\max\left(\epsilon_k/r_\epsilon,\epsilon_{\min}\right), ϵk+1←m⊙ϵshrunk+(1−m)⊙ϵk\epsilon_{k+1}\leftarrow m\odot\epsilon_{\mathrm{shrunk}}+(1-m)\odot\epsilon_k

rs←∥∇f(xk+1)+Jg(xk+1)⊤μk+1∥r_s\leftarrow\left\lVert\nabla f(x_{k+1})+J_g(x_{k+1})^\top\mu_{k+1}\right\rVert, rf←max⁡(g(xk+1),0)r_f\leftarrow\max\left(g(x_{k+1}),0\right), rc←max⁡i∣μk+1,igi(xk+1)∣r_c\leftarrow\max_i\left|\mu_{k+1,i}g_i(x_{k+1})\right|

end

return xN, μNx_N,\ \mu_N

(2025), Adam (Kingma and Ba, [19]) within an Augmented Lagrangian method (Adam-AL), and Adam applied to a penalty function (Adam-PEN) (Lu et al., 2021).

Remark 2 We re-implement the FL algorithm from Zhang et al. [23] in JAX with just-in-time compilation to speed up simulations. The algorithmic structure is unchanged, including the Gurobi quadratic program (QP) solver for the multipliers, which increases FL’s computational cost.

Iris Dataset SVM. We consider a support vector machine (SVM) benchmark on real data (Fisher, [15]). Training a soft-margin SVM is equivalent to min⁡w,b,ξ12∥w∥2+12∥ξ∥2\min_{w,b,\xi}\frac{1}{2}\lVert w\rVert^2+\frac{1}{2}\lVert\xi\rVert^2 subject to 1−yi(w⊤ai+b)−ξi≤01-y_i(w^\top a_i+b)-\xi_i\leq0, i=1,…,Nsi=1,\ldots,N_s, where ai∈Rda_i\in\mathbb{R}^d are the samples with labels yi∈{−1,+1}y_i\in\{-1,+1\}, and NsN_s is the number of samples. We use the full dataset (Ns=150N_s=150) with d=4d=4, and train a soft-margin SVM to separate the Virginica species from the other two. The resulting formulation has n=155n=155 optimization variables, and the optimal solution for both problems is computed using SLSQP and then refined by Newton’s method on the active-set KKT system. Each method is run from 10 different initial conditions, with components sampled independently and uniformly from [−1,1][-1,1]. Since, with the tested parameters, the second-order KKT-SMC method fails to converge from any initial condition, we defer its comparison with COSMIC to Appendix § 10.§ 1, on a hard-margin SVM problem with a smaller dataset. Figure 2 plots the log of the error of FL, Adam-AL, Adam-PEN, and COSMIC with respect to CPU time. COSMIC significantly outperforms the other methods, reaching the highest accuracy in the shortest CPU time. Although FL reaches a final accuracy of order 10−910^{-9}, its per-iteration cost is high, averaging 6.14 ms versus 0.042 ms for COSMIC, 0.059 ms for Adam-PEN, and 0.066 ms for Adam-AL.

Figure 2. Optimization of soft-margin SVM for Iris Dataset. The solid line is the mean error over the 10 runs, and the shaded band corresponds to their minimum and maximum.

Image Completion using ICNN We consider a higher-dimensional problem: completing the right half of a $64\times64 face image from its left half with a learned convex energy given by a partially input convex neural network (PICNN) (Amos et al., [2]). The completed half (n=2048 pixels) is obtained by minimizing the PICNN output subject to m=4096m = 4096 linear inequality constraints that enforce the valid pixel range [0,1][0, 1]. Figure 3 compares COSMIC with FL, Adam-AL and Adam-PEN, and with projected gradient descent with momentum (PGD-MOM), the inference method of [2]. All methods are run for 3000 iterations on the same 3 test images and 2 initial infeasible points per image drawn uniformly from [−0.5,1.5]n[-0.5, 1.5]^n. The CPU time is measured per iteration on the compiled solvers, excluding metric logging; for FL it includes the QP solve in Gurobi. The hyperparameters for PGD-MOM are adopted from Amos et al. [2], while those of the other methods are tuned separately via a parameter sweep for best performance, and listed in Table 2 in Appendix § 10.§ 2. Figure 3 shows that COSMIC reaches the highest accuracy in the shortest CPU time, and has better and comparable performance to adaptive momentum based methods. Although While FL achieves high accuracy, the QP solved at each iteration results in an average iteration time of approximately 24 ms, considerably higher than the 4.8–5.5 ms of the other methods, with a total runtime exceeding 80 s for 3000 iterations. Adam-AL, Adam-PEN, and PGD-MOM converge more slowly and do not reach the accuracy of COSMIC within the same budget.

Figure 3. Error convergence with respect to (a) iterations and (b) CPU time. The solid line shows the mean error over the 3 runs, and the shaded band spans their minimum and maximum. For visualization purposes, the CPU-time range is limited to 40s.

MethodHyperparameters
COSMICdt=0.3dt=0.3, K=5K=5, β=0.1\beta=0.1, ϵsat=0.1\epsilon_{\mathrm{sat}}=0.1
FLη=0.5\eta=0.5, K=6K=6
Adam-ALα0=10−2\alpha_0=10^{-2}, σ0=10\sigma_0=10, T=100T=100, γσ=1.03\gamma_\sigma=1.03, γα=0.97\gamma_\alpha=0.97
Adam-PENα0=10−2\alpha_0=10^{-2}, σ0=10\sigma_0=10, T=100T=100, γσ=1.03\gamma_\sigma=1.03, γα=0.97\gamma_\alpha=0.97

Table 2. Selected hyperparameters for image completion using ICNNs.

Conclusion

We introduce COSMIC (7), a continuous-time solver for inequality-constrained optimization in which the Lagrange multiplier dynamics are designed via SMC. Its trajectories reach, in finite time, the sliding manifold where the smoothed FB complementarity condition are satisfied. On this manifold, COSMIC is the gradient flow of a log-barrier function, which yields a control-theoretic interpretation of barrier methods. We show that the equilibria of (7) are exactly the points satisfying stationarity and the smoothed FB condition, and that, as the smoothing parameter ϵ\epsilon decays, every limit point of a sequence of equilibria is a KKT point of the original problem. For convex problems satisfying Slater’s condition, the suboptimality is bounded by mϵ2/2m\epsilon^2/2. For strongly convex costs, from any initial primal point, the trajectories become strictly feasible in finite time and then converge exponentially to the minimizer of the associated barrier problem. Experiments on SVM classification and image completion via an ICNN mapping confirm that COSMIC solves these problems accurately and scales up without significant increase in computational time per-iteration. Future directions include the extension of the convergence analysis beyond strong convexity, analysis of discrete-time implementations, and extensions to accommodate equality constraints.

Generative AI Use

Generative AI tools were used for code generation, editing, and proof checking. The authors reviewed and verified all AI-assisted content and take responsibility for the final content of the paper.

Acknowledgments

This work was supported as a part of NCCR Automation, a National Centre of Competence in Research, funded by the Swiss National Science Foundation (grant number 51NF40 225155).

Extended proof

Proof [Proof of Lemma 4.2] The right-hand side of (7) is measurable and locally essentially bounded. Hence a Filippov solution exists from every initial condition ([10 Proposition 3]), and, a maximal Filippov solution does not exist in finite time if and only if it leaves every compact set. Let (x(t),μ(t))(x(t),\mu(t)) be a maximal Filippov solution with ϕϵ(x(0),μ(0))≤0m\phi_{\epsilon}(x(0),\mu(0)) \le0_m, defined on [0,Tmax⁡)[0,T_{\max}), and assume, for the sake of contradiction, that Tmax⁡<∞T_{\max} < \infty. In what follows, c>0c > 0 denotes a generic constant which may depend on Tmax⁡T_{\max}, the initial condition, and the problem data. The value of cc may change from one occurrence to the next, even within the same line, but we keep the same symbol for simplicity of notation.

Since ϕi,0≤0\phi_{i,0} \le0, the proof of Lemma 4.1 gives ϕi(t):=ϕϵ,i(x(t),μ(t))∈[ϕi,0,0]\phi_i(t) := \phi_{\epsilon,i}(x(t),\mu(t)) \in[\phi_{i,0},0] and 0≤ϕ˙i≤K∣ϕi,0∣+β0 \le\dot{\phi}_i \le K|\phi_{i,0}|+\beta for almost every t∈[0,Tmax⁡)t \in[0,T_{\max}). Moreover, di:=−ϕi−gi(x)=gi2(x)+μi2+ϵ2−μid_i := -\phi_i-g_i(x)=\sqrt{g_i^2(x)+\mu_i^2+\epsilon^2}-\mu_i, where di>0d_i>0. Expanding (di+μi)2=gi2(x)+μi2+ϵ2(d_i+\mu_i)^2=g_i^2(x)+\mu_i^2+\epsilon^2 and substituting gi2(x)=ϕi2+2ϕidi+di2g_i^2(x)=\phi_i^2+2\phi_i d_i+d_i^2 yields

μi=ϕi+ϕi2+ϵ22di,equivalently,(μi−ϕi)di=ϕi2+ϵ22.(13)\mu_i=\phi_i+\frac{\phi_i^2+\epsilon^2}{2d_i},\qquad\text{equivalently,}\qquad(\mu_i-\phi_i)d_i=\frac{\phi_i^2+\epsilon^2}{2}. \tag*{(13)}

Therefore, μi>ϕi≥ϕi,0\mu_i>\phi_i\ge\phi_{i,0}, and μi\mu_i can diverge only if di→0d_i\to0 for some ii. It thus suffices to show that xx is bounded and that each did_i is bounded away from zero. Next, let xsx_s be a Slater point and γ:=min⁡j(−gj(xs))>0\gamma:= \min_j(-g_j(x_s))>0. Then, by (7)

ddt12∥x−xs∥2=−∇f(x)T(x−xs)−∑i=1mϕi∇gi(x)T(x−xs)−∑i=1m(μi−ϕi)∇gi(x)T(x−xs).\frac{d}{dt}\frac{1}{2}\lVert x-x_s\rVert^2=-\nabla f(x)^{\mathsf{T}}(x-x_s)-\sum_{i=1}^{m}\phi_i\nabla g_i(x)^{\mathsf{T}}(x-x_s)-\sum_{i=1}^{m}(\mu_i-\phi_i)\nabla g_i(x)^{\mathsf{T}}(x-x_s).

By the Cauchy–Schwarz inequality and since ∇f\nabla f is LfL_f-Lipschitz, it holds −∇f(x)T(x−xs)≤∥∇f(x)∥∥x−xs∥≤(∥∇f(xs)∥+Lf∥x−xs∥)∥x−xs∥≤c(1+∥x−xs∥2)-\nabla f(x)^{\mathsf{T}}(x-x_s)\le\lVert\nabla f(x)\rVert\lVert x-x_s\rVert\le(\lVert\nabla f(x_s)\rVert+L_f\lVert x-x_s\rVert)\lVert x-x_s\rVert\le c(1+\lVert x-x_s\rVert^2). Moreover, since ∣ϕi∣≤∣ϕi,0∣|\phi_i|\le|\phi_{i,0}| and ∇gi\nabla g_i is LgL_g-Lipschitz, the second sum is bounded by c(1+∥x−xs∥2)c(1+\lVert x-x_s\rVert^2). Since μi−ϕi>0\mu_i-\phi_i>0 and gi(x)=−ϕi−di≥−dig_i(x)=-\phi_i-d_i\ge-d_i, convexity of gig_i and (13) yield (μi−ϕi)∇gi(x)T(x−xs)≥(μi−ϕi)(gi(x)−gi(xs))≥γ(μi−ϕi)−c(\mu_i-\phi_i)\nabla g_i(x)^{\mathsf{T}(x-x_s)}\ge(\mu_i-\phi_i)(g_i(x)-g_i(x_s))\ge\gamma(\mu_i-\phi_i)-c. Thus,

ddt12∥x−xs∥2+γ∑i=1m(μi−ϕi)≤c(1+∥x−xs∥2).\frac{d}{dt}\frac{1}{2}\lVert x-x_s\rVert^2+\gamma\sum_{i=1}^{m}(\mu_i-\phi_i)\le c(1+\lVert x-x_s\rVert^2).

where cc subsumes the scaling constants. Moreover, since Tmax⁡<∞T_{\max}<\infty, Grönwall’s inequality gives ∥x(t)−xs∥≤c\lVert x(t)-x_s\rVert\le c and ∫0t∑i=1m(μi−ϕi)dτ≤c\int_0^t\sum_{i=1}^{m}(\mu_i-\phi_i)d\tau\le c on [0,Tmax⁡)[0,T_{\max}).

By (13), x˙=−∇xV(t,x)\dot{x}=-\nabla_xV(t,x), where

V(t,x):=f(x)+∑i=1m(ϕi(t)gi(x)−ϕi(t)2+ϵ22log⁡(−ϕi(t)−gi(x))).V(t,x):=f(x)+\sum_{i=1}^{m}\left(\phi_i(t)g_i(x)-\frac{\phi_i(t)^2+\epsilon^2}{2}\log\left(-\phi_i(t)-g_i(x)\right)\right).

Along the solution, and by using x˙=−∇xV(t,x)\dot{x}=-\nabla_xV(t,x), di=−ϕi−gi(x)d_i=-\phi_i-g_i(x), and (12), we have

ddtV(t,x(t))=∇xV(t,x)⊤x˙+∑i=1m∂V∂ϕi(t,x)ϕ˙i=−∥∇xV(t,x)∥2+∑i=1mϕ˙i(gi(x)−ϕilog⁡di+ϕi2+ϵ22di)≤∑i=1mϕ˙i(gi(x)−ϕilog⁡di+μi−ϕi)≤∑i=1mϕ˙i(c+μi−ϕi)≤c(1+∑i=1m(μi−ϕi)),\begin{aligned} \frac{d}{dt}V(t,x(t))&=\nabla_xV(t,x)^\top\dot{x}+\sum_{i=1}^{m}\frac{\partial V}{\partial\phi_i}(t,x)\dot{\phi}_i\\ &=-\lVert\nabla_xV(t,x)\rVert^2+\sum_{i=1}^{m}\dot{\phi}_i\left(g_i(x)-\phi_i\log d_i+\frac{\phi_i^2+\epsilon^2}{2d_i}\right)\\ &\leq\sum_{i=1}^{m}\dot{\phi}_i\left(g_i(x)-\phi_i\log d_i+\mu_i-\phi_i\right)\\ &\leq\sum_{i=1}^{m}\dot{\phi}_i\left(c+\mu_i-\phi_i\right)\leq c\left(1+\sum_{i=1}^{m}(\mu_i-\phi_i)\right), \end{aligned}

where we used that xx is bounded, so that ∣gi(x)∣≤c|g_i(x)|\leq c and di≤∣ϕi,0∣+∣gi(x)∣≤cd_i\leq|\phi_{i,0}|+|g_i(x)|\leq c, that −ϕilog⁡di≤∣ϕi,0∣max⁡{0,log⁡c}≤c-\phi_i\log d_i\leq|\phi_{i,0}|\max\{0,\log c\}\leq c since −ϕi≥0-\phi_i\geq0, and that μi−ϕi>0\mu_i-\phi_i>0 and 0≤ϕ˙i≤c0\leq\dot{\phi}_i\leq c. Integrating gives V(t,x(t))≤cV(t,x(t))\leq c. On the other hand, since xx is bounded and dj≤cd_j\leq c for all jj, all terms of VV except the ii-th logarithm are bounded below, and the latter is at least ϵ22∣log⁡di∣\frac{\epsilon^2}{2}|\log d_i| whenever di≤1d_i\leq1. Hence, di(t)≥e−cd_i(t)\geq e^{-c} on [0,Tmax⁡)[0,T_{\max}).

By (12), ϕi,0≤μi(t)≤c\phi_{i,0}\leq\mu_i(t)\leq c, so (x,μ)(x,\mu) remains in a compact set on [0,Tmax⁡)[0,T_{\max}), contradicting Tmax⁡<∞T_{\max}<\infty. Hence, Tmax⁡=+∞T_{\max}=+\infty.

Numerical Experiments

We provide implementation details and additional results for the experiments in § 5. Besides FL and KKT-SMC, we compare COSMIC with two standard first-order approaches to constrained learning based on Adam, namely Adam with an augmented Lagrangian formulation, and Adam applied to a quadratic penalty formulation. For completeness, we review these methods.

Adam with Penalty Method (Adam-PEN). Adam is run on the quadratic penalty function

Fσ(x)=f(x)+σ2∥max⁡(0,g(x))∥2,F_\sigma(x)=f(x)+\frac{\sigma}{2}\lVert\max(0,g(x))\rVert^2,

where σ>0\sigma>0 is an increasing penalty parameter capped at σmax⁡\sigma_{\max}.

Adam with an augmented Lagrangian (Adam-AL). As a first-order baseline, we apply the method of multipliers to the Powell–Hestenes–Rockafellar augmented Lagrangian

Lσ(x,λ)=f(x)+12σ∑i=1m(max⁡{0,λi+σgi(x)}2−λi2).(14)\mathcal{L}_\sigma(x,\lambda)=f(x)+\frac{1}{2\sigma}\sum_{i=1}^{m}\left(\max\{0,\lambda_i+\sigma g_i(x)\}^2-\lambda_i^2\right). \tag*{(14)}

Each outer iteration runs TT Adam steps on xx (with λ\lambda and σ\sigma fixed), then updates

λ←max⁡{0,λ+σg(x)},σ←min⁡{γσσ,σmax⁡},α←max⁡{γαα,αmin⁡}.(15)\lambda\leftarrow\max\{0,\lambda+\sigma g(x)\},\qquad\sigma\leftarrow\min\{\gamma_\sigma\sigma,\sigma_{\max}\},\qquad\alpha\leftarrow\max\{\gamma_\alpha\alpha,\alpha_{\min}\}. \tag*{(15)}

where α\alpha is the Adam step size, γσ≥1\gamma_\sigma\geq1, γα∈]0,1]\gamma_\alpha\in]0,1], and the Adam moments are reset after every multiplier update. The effective multiplier max⁡{0,λ+σg(x)}\max\{0,\lambda+\sigma g(x)\} is used when evaluating the KKT residual.

Hard-Margin SVM Optimization

To compare COSMIC with KKT-SMC, we consider the hard-margin SVM problem which has less variables. In this case, training an SVM is equivalent to solving

min⁡w,b12∥w∥22s.t.1−yi(wTai+b)≤0,i=1,…,Ns,\begin{aligned} \min_{w,b}\quad&\frac{1}{2}\lVert w\rVert_{2}^{2}\\ \text{s.t.}\quad&1-y_i\left(w^{\mathsf{T}}a_i+b\right)\leq0,\quad i=1,\ldots,N_s, \end{aligned}

where ai∈Rda_i\in\mathbb{R}^{d} are the samples, yi∈{−1,+1}y_i\in\{-1,+1\} their labels, and NsN_s is the number of samples. We use the linearly separable Iris Setosa-vs-Versicolor dataset with d=4d=4 features, which gives n=d+1=5n=d+1=5 optimization variables and Ns=100N_s=100 linear inequality constraints corresponding to only two species of the Iris flower: Setosa and Veriscolor. We aim to show the performance of COSMIC on this lower dimensional problem and also consider its sensitivity to cϕc_{\phi} and rϵr_{\epsilon}, introduced in Algorithm 2. All methods are run from the same 10 initial conditions, whose components are sampled uniformly at random from [−1,1][-1,1], with a fixed budget of N=20000N=20000 iterations. Figure 4 shows the error with respect to the optimal solution as a function of the CPU time for COSMIC, FL, and KKT-SMC, with the best-performing set of hyperparameters. The tested hyperparameters are listed in Table 1. For COSMIC, we test several values of KK, which give comparable performance. For the problem formulation considered here, the performance of COSMIC is relatively insensitive to the specific values of cϕc_{\phi} and rϵr_{\epsilon}. This is illustrated in Figure 4, where we vary cϕc_{\phi} over [0.8,0.95][0.8,0.95] and rϵr_{\epsilon} over [1.01,1.2][1.01,1.2].

Figure 4. Optimization of Hard-Margin SVM for Iris Dataset

MethodHyperparameters
COSMICdt=0.0035dt = 0.0035, K∈{50,60}K \in\{50, 60\}, β=0.1\beta= 0.1, ϵsat=0.01\epsilon_{\mathrm{sat}} = 0.01, ϵ0=10−10\epsilon_0 = 10^{-10}
FLη=0.1\eta= 0.1, K∈{5,10,15}K \in\{5, 10, 15\}
KKT-SMCη=0.8\eta= 0.8, qx=qc=1q_x = q_c = 1, ϵx=ϵc=0.01\epsilon_x = \epsilon_c = 0.01, ϕ=0.1\phi= 0.1, σc=10−15\sigma_c = 10^{-15}

Table 1. Tested hyperparameters for SVM optimization.

Since the problem is low-dimensional, the second-order KKT-SMC reaches the neighborhood of the minimum within a few iterations. However, its accuracy is limited by the non-vanishing smoothing parameter of the FB function. COSMIC requires more iterations, but attains a higher accuracy than KKT-SMC and has a lower average cost per iteration (0.045 ms versus 0.105 ms for KKT-SMC). FL is slower and less accurate than both sliding-mode-based methods.

Image Completion using Input Convex Neural Networks

Architecture. Let x∈R64×32x \in\mathbb{R}^{64 \times32} denote the observed left half of a face and y∈Rny \in\mathbb{R}^{n} the right half to be generated. The energy Fθ(x,⋅):Rn→RF_\theta(x,\cdot):\mathbb{R}^{n}\to\mathbb{R} is a convolutional PICNN with an input path uiu_i acting on xx and an output path ziz_i acting on yy,

u0=x,ui+1=ReLU⁡(W~iui+b~i),i=0,…,3,z1=sp⁡(W0y(y∘(W0yux+b0y))+W0ux+b0),zi+1=sp⁡(Wiz(zi∘[Wizuui+biz]+)+Wiyy+Wiuui+bi),i=1,2,3,Fθ(x,y)=W4z(z4∘[W4zuu4+b4z]+)+W4yy+W4uu4+b4+ℓ2∥y∥22,(16)\begin{aligned} u_0 &= x, \qquad u_{i+1} = \operatorname{ReLU}(\widetilde{W}_i u_i + \widetilde{b}_i), \qquad i=0,\ldots,3,\\ z_1 &= \operatorname{sp}\left(W_0^y\left(y \circ(W_0^{yu}x+b_0^y)\right)+W_0^u x+b_0\right),\\ z_{i+1} &= \operatorname{sp}\left(W_i^z\left(z_i \circ[W_i^{zu}u_i+b_i^z]_+\right)+W_i^y y+W_i^u u_i+b_i\right), \qquad i=1,2,3,\\ F_\theta(x,y) &= W_4^z\left(z_4 \circ[W_4^{zu}u_4+b_4^z]_+\right)+W_4^y y+W_4^u u_4+b_4+\frac{\ell}{2}\lVert y\rVert_2^2, \tag*{(16)} \end{aligned}

where sp⁡(s)=log⁡(1+es)\operatorname{sp}(s)=\log(1+e^s) is the softplus function, applied element-wise, [⋅]+[\cdot]_+ is the positive part, ∘\circ is the element-wise product, and θ\theta collects all weights and biases. We choose the network as in [2]. Because sp⁡\operatorname{sp} is convex and non-decreasing, the weights WizW_i^z acting on the hidden states are element-wise non-negative, and the gates [Wizuui+biz]+[W_i^{zu}u_i+b_i^z]_+ are non-negative and depend on xx only, every ziz_i is convex in yy. Hence, Fθ(x,y)F_\theta(x,y) is convex and smooth in yy for every xx, although not necessarily convex in xx, and the quadratic term makes it ρ\rho-strongly convex with ρ=0.05\rho=0.05, so (18) has a unique minimizer.

Training. The parameters θ\theta are trained with a denoising score matching loss, such that the true right half yy is the minimizer of Fθ(x,⋅)F_\theta(x,\cdot): for a noisy right half y~=y+σϵ\widetilde{y}=y+\sigma\epsilon, ϵ∼N(0,I)\epsilon\sim\mathcal{N}(0,I), σ∼U(0.02,0.25)\sigma\sim\mathcal{U}(0.02,0.25),

min⁡θ E(x,y)∼D, y~[∥∇yFθ(x,y~)−c(y~−y)∥22]s.t.Wiz≥0,i=1,…,4,(17)\min_{\theta}\ \mathbb{E}_{(x,y)\sim\mathcal{D},\,\widetilde{y}}\left[\left\lVert\nabla_y F_\theta(x,\widetilde{y})-c(\widetilde{y}-y)\right\rVert_2^2\right]\quad\text{s.t.}\quad W_i^z\geq0,\quad i=1,\ldots,4, \tag*{(17)}

with c=0.25c=0.25, i.e., Fθ(x,⋅)F_\theta(x,\cdot) is fitted to the convex bowl c2∥⋅−y∥22\frac{c}{2}\lVert\cdot-y\rVert_2^2 around the true completion. Problem (17) is solved with Adam (batch size 16, learning rate 10−310^{-3} with cosine decay to 10−410^{-4}, 2000 steps), where the weights WizW_i^z are projected onto the non-negative orthant after every step, as in [2]. We use the Olivetti faces (400 images of 40 subjects, 64×6464\times64 pixels in [0,1][0,1]) with 36 subjects for training, and 4 for testing, and random horizontal flips of the full image as data augmentation.

Problem. Given the left half xx of a test face, the completed right half is the minimiser of the learned energy subject to the pixel range,

min⁡y∈RnFθ(x,y)s.t.0n≤y≤1n.(18)\begin{aligned} \min_{y\in\mathbb{R}^{n}}\quad&F_\theta(x,y)\\ \text{s.t.}\quad&0_n\leq y\leq1_n. \tag*{(18)} \end{aligned}

Writing the constraints as g(y)=Ay−b≤0g(y)=Ay-b\le0 with

A=[I−I]∈Rm×n,b=[1n0n],m=2n=4096,(19)A=\begin{bmatrix} I \\ -I \end{bmatrix}\in\mathbb{R}^{m\times n},\qquad b=\begin{bmatrix} \mathbf{1}_n \\ \mathbf{0}_n \end{bmatrix},\qquad m=2n=4096, \tag*{(19)}

problem (18) becomes the strongly convex program min⁡yFθ(x,y)\min_y F_\theta(x,y) s.t. Ay≤bAy\le b. Since the constraints are affine, the Jacobian Jg(y)=AJ_g(y)=A is constant and the Hessian of the Lagrangian L(y,μ)=Fθ(x,y)+μT(Ay−b)L(y,\mu)=F_\theta(x,y)+\mu^\mathsf{T}(Ay-b) equals ∇y2Fθ(x,y)\nabla_y^2F_\theta(x,y). The unique primal–dual solution (y⋆,μ⋆)(y^\star,\mu^\star) is characterized by the KKT conditions

∇yFθ(x,y⋆)+ATμ⋆=0n,Ay⋆−b≤0m,μ⋆≥0m,μi⋆(Ay⋆−b)i=0,∀i.(20)\nabla_yF_\theta(x,y^\star)+A^\mathsf{T}\mu^\star=0_n,\qquad Ay^\star-b\le0_m,\qquad\mu^\star\ge0_m,\qquad\mu_i^\star(Ay^\star-b)_i=0,\quad\forall i. \tag*{(20)}

Reference solution and metrics. The reference y⋆y^\star is computed in two stages. L-BFGS-B first locates the solution and its active set A\mathcal{A}. The pixels within A\mathcal{A} are fixed at their bounds, while the remaining free pixels are refined by Newton’s method.

We present the corresponding convergence of the multiplier to its optimal value in Figure 5.

Figure 5. Multiplier error convergence with respect to (a) iterations and (b) CPU time. The solid line shows the mean error over the 3 runs, and the shaded band spans their minimum and maximum. For visualization purposes, the CPU-time range is intentionally limited to 40s.

We also showcase a sample of the resulting completed faces below.

Completed Faces from Olivetti Dataset using COSMIC, FL, Adam-AL, Adam-PEN, and PGD-MOM

Figure 6. Completed Faces from Olivetti Dataset using COSMIC, FL, Adam-AL, Adam-PEN, and PGD-MOM

References

  1. [1]A. Allibhoy and J. Cortés. Control-barrier-function-based design of gradient flows for constrained nonlinear programming. IEEE Transactions on Automatic Control, 69(6):3499–3514, 2023. doi: 10.1109/TAC.2023.3306492.DOI
  2. [2]B. Amos, L. Xu, and J. Z. Kolter. Input convex neural networks. In International Conference on Machine Learning, pages 146–155. PMLR, 2017.arxiv.org/abs/1609.07152
  3. [3]K. J. Arrow, L. Hurwicz, and H. Uzawa, editors. Studies in Linear and Nonlinear Programming. Stanford University Press, 1958.
  4. [4]G. Bianchin, J. Cortés, J. I. Poveda, and E. Dall’Anese. Time-varying optimization of lti systems via projected primal-dual gradient flows. IEEE Transactions on Control of Network Systems, 9 (1):474–486, 2021. doi: 10.1109/tcns.2021.3112762.DOI
  5. [5]S. Boyd and L. Vandenberghe. Convex Optimization. Cambridge University Press, 2004. ISBN 0521833787.
  6. [6]V. Centorrino, R. Hoteit, E. C. Balta, and J. Lygeros. SPPID: Saddle-point PID for constrained optimization. arXiv preprint arXiv:2609.31086, 2026a.arxiv.org/abs/2609.31086
  7. [8]V. Cerone, S. M. Fosson, S. Pirrera, and D. Regruto. A new framework for constrained optimization via feedback control of Lagrange multipliers. IEEE Transactions on Automatic Control, pages 1–16, 2025. ISSN 2334-3303. doi: 10.1109/tac.2025.3568651.DOI
  8. [9]F. Chen and W. Ren. Sign projected gradient flow: A continuous-time approach to convex optimization with linear equality constraints. Automatica, 120:109156, 2020. doi: 10.1016/j.automatica.2020.109156.DOI
  9. [10]J. Cortes. Discontinuous dynamical systems. IEEE Control Systems Magazine, 28(3):36–73, 2008. doi: 10.1109/mcs.2008.919306.DOI
  10. [11]B. Diana, S. Pandey, S. Kamal, and T. N. Dinh. Finite and fixed-time feedback-based continuous-time optimization. Automatica, 183:112569, 2026. doi: 10.1016/j.automatica.2025.112569.DOI
  11. [12]F. Dörfler, Z. He, G. Belgioioso, S. Bolognani, J. Lygeros, and M. Muehlebach. Toward a systems theory of algorithms. IEEE Control Systems Letters, 8:1198–1210, 2024. ISSN 2475-1456. doi: 10.1109/lcsys.2024.3406943.
  12. [13]A. V. Fiacco and G. P. McCormick. Nonlinear programming: sequential unconstrained minimization techniques. SIAM, 1990.
  13. [14]A. Fischer. A special Newton-type optimization method. Optimization, 24(3-4):269–284, January 1992. ISSN 1029-4945. doi: 10.1080/02331939208843795.DOI
  14. [15]R. A. Fisher. Iris dataset, 1936.
  15. [16]A. Hauswirth, Z. He, S. Bolognani, G. Hug, and F. Dörfler. Optimization algorithms as robust feedback controllers. Annual Reviews in Control, 57:100941, 2024. doi: 10.1016/j.arcontrol.2024.100941.DOI
  16. [17]C. Kanzow. Some noninterior continuation methods for linear complementarity problems. SIAM Journal on Matrix Analysis and Applications, 17(4):851–868, October 1996. ISSN 1095-7162. doi: 10.1137/S0895479894273134.DOI
  17. [18]H. K. Khalil. Nonlinear Systems. Pearson Education. Prentice Hall, 2002. ISBN 9780130673893.
  18. [19]D. P. Kingma and J. Ba. Adam: A method for stochastic optimization. arXiv preprint ARXIV:1412.6980, 2014.arxiv.org/abs/1412.6980
  19. [21]R. T. Rockafellar. Convex analysis, 1970.
  20. [22]A. Zenati and K. Youcef-Toumi. KKT Residual Sliding Manifolds: A Projection-Free Structured Finite-Time Solver for Nonlinear Constrained Optimisation, 2026.arxiv.org/abs/2610.15545
  21. [23]R. Zhang, A. Raghunathan, J. Shamma, and N. Li. Constrained optimization from a control perspective via feedback linearization. In Advances in Neural Information Processing Systems 38, pages 62328–62364, 2025. doi: 10.52202/085713-1873.DOI

Paper details

vlnc:W_sgjc5hds
Version
v1 · latest
Published
Fresh — added to the Hub within the last 30 days
Added
License
CC BY 4.0
Usage

views · 0

downloads · 0

A view counts once per browser per day, a download each time the PDF is requested.

  • Mathematics

Generative AI tools

View statement
Looking for related papers…
Contents