WorksMathematics
Inequality-Constrained Optimization via Sliding Mode Control
Abstract
We present COSMIC (Constrained Optimization via Sliding Mode for Inequality Constraints), a continuous-time solver for inequality-constrained optimization. COSMIC couples primal gradient-flow dynamics with multiplier dynamics derived from sliding mode control laws on a manifold defined by the smoothed Fischer-Burmeister function. We prove that COSMIC's trajectories reach this manifold in finite time from any primal initial point, after which the dynamics reduces to the gradient flow of a logarithmic barrier function. This yields a control-theoretic interpretation of barrier methods while allowing infeasible initialization. For strongly convex problems, we further establish exponential convergence to the minimizer. To address the numerical stiffness that arises in discretization, we develop a modified discrete-time algorithm. We validate COSMIC across various machine learning applications, demonstrating its ability to solve optimization problems accurately while scaling effectively to high-dimensional settings.
Introduction
We consider optimization problems of the form
where and are assumed to be continuously differentiable throughout the paper. Inequality-constrained optimization problems, which subsume equality constraints, are prominent in various control, robotics, and machine learning applications, where the inequality constraints typically encode safety requirements or system and resource limitations. Classical approaches to solving (1) include projection-based methods, primal-dual algorithms, and penalty methods. The system-theoretic view of algorithms offers an alternative framework for analyzing and designing optimization algorithms through the lens of control theory [12]. This perspective has recently provided a control-theoretic interpretation of saddle-point flows, where the primal variables act as the plant of a dynamical system, and the Lagrange multipliers act as PID controllers that enforce constraint satisfaction [8, 6]. While PID controllers offer asymptotic convergence in these methods, sliding mode control (SMC) offers finite-time convergence and robustness to bounded uncertainty [18]. Recent work applies SMC for equality-constrained problems [11] with finite-time convergence. However, to date there is no first-order SMC-based solver for inequality-constrained problems, with convergence guarantees.
To fill this gap, we propose an SMC-based solver for (1). The resulting continuous-time optimization algorithm is computationally efficient and enjoys finite-time convergence guarantees. Moreover, it provides a control-theoretic interpretation of penalty methods, which, to the best of our knowledge, has not been established in the literature.
Related Work. There has been extensive work in viewing optimization algorithms as continuous time closed-loop systems with feedback controllers [16, 1, 4]. This perspective builds on the classical primal–dual (Arrow–Hurwicz–Uzawa) gradient flows [3], in which the multipliers act as an integral controller on the constraint violation. Chen and Ren [9] utilize nonlinear sign-based control to design projected gradient flows for linear-equality constrained convex problems. Within this line of work, [8, 6 b] set the multipliers to be PID controllers on the primal flow. Moreover, Zhang et al. [23] apply feedback linearization to inequality-constrained problems and relate the resulting algorithm to sequential quadratic programming. Closest to our work, Diana et al. [11] designs the dual dynamics of equality constrained problems as SMC controllers and Zenati and Youcef-Toumi [22] use discrete-time SMC to obtain a second-order solver for equality and inequality constrained problems, choosing a sliding manifold that encodes the KKT conditions and using the Fischer–Burmeister (FB) function.
Contributions. We introduce the Constrained Optimization via Sliding Mode for Inequality Constraints (COSMIC), a continuous-time flow for solving (1). Under COSMIC, the primal variables follow a gradient descent on the Lagrangian, while the multipliers are driven by SMC laws whose sliding manifold is the zero set of the smoothed FB function of the constraints and multipliers. On this manifold, primal and dual feasibility hold strictly and complementary slackness holds up to a perturbation . At equilibrium, the primal dynamics enforce stationarity, and the equilibria satisfy a perturbed KKT system, which reduces to the exact one as .
Our main results establish finite-time convergence to the sliding manifold and, for strongly convex problems, exponential convergence to the minimizer of (1). We also show that, once on the manifold, COSMIC reduces to the gradient flow of a logarithmic barrier method. This yields a control-theoretic interpretation of barrier methods and generalizes them, in the sense that our solver does not require a feasible initialization. To address the numerical difficulties that arise when discretizing COSMIC, we propose a modified discrete-time implementation. Finally, through training a support vector machine (SVM) on the Iris Setosa-vs-Versicolor dataset, and image completion with an input convex neural network (ICNN), we highlight the computational efficiency and accuracy of COSMIC relative to other nonlinear control-based optimization solvers. The code to reproduce the experiments is available at https://github.com/sudo-Ra-ONE/COSMIC.
Outline. Section 2 presents the mathematical preliminaries and is followed by the problem formulation and design of COSMIC, in Section 3. Section 4 formalizes the analysis and the numerical implementation of our solver, and Section 5 benchmarks it on various problems. Section 6 provides a summary and directions for future work. We provide a visual outline in Figure 1.

Figure 1. Visual Outline of the Paper
Mathematical Preliminaries
Notation. We denote by and the real and positive real numbers, respectively, by the Euclidean norm on , and by and the all-zeros and all-ones vectors in . We let be the per-component division between two vectors. The class of -times continuously differentiable functions is . For a differentiable , is its gradient. Given continuously differentiable, denotes its Jacobian matrix. Given , a map is (i) -strongly convex if is convex; (ii) -smooth if it is differentiable and is -Lipschitz. The sign function is defined by if , if , and , and it is applied component-wise to vectors.
Optimality Conditions. Let denote the feasible set of (1), its optimal value, with the convention , and denote the active set at . We say that linear independence constraint qualification (LICQ) holds at if the gradients are linearly independent. A pair satisfies the Karush–Kuhn–Tucker (KKT) conditions if
For , let , . The Fischer–Burmeister (FB) function (Fischer, [14]) is , and , , is its smoothed version (Kanzow, [17]). For , and if and only if and . Hence, with and , encodes a perturbed version of the conditions in (3), which are exact for .
Problem Formulation
We treat the primal dynamics of (1) as the plant and design the Lagrange multipliers as SMC-based feedback controllers that enforce the KKT conditions. SMC drives the system states to a predefined invariant, sliding manifold (Khalil, [18]), in two phases: a reaching phase, and a sliding phase, during which they slide toward the equilibrium. We next make this construction precise.
Consider the Lagrangian associated with (1), that is, the map defined by , where is the Lagrange multiplier of the constraints. Treating as a control input to the gradient flow of with respect to yields the system
with output . We design the input via SMC so that the trajectories of (4) reach, and remain on, a manifold on which an -perturbed KKT conditions hold. Fix . With a slight abuse of notation, we consider , defined by . Let the sliding manifold be given by , where for , and consider the following sliding law
where and . Differentiating yields
where are diagonal with entries and . Since for all and all , is invertible. Combining (4) and (6) yields the closed loop dynamics
which we refer to as COSMIC. Because (7) is discontinuous, we consider solutions of (7) in the sense of Filippov [10]. Since and is continuous, the right-hand side of (7) is measurable and locally essentially bounded. Then, for every , a Filippov solution of (7) exists [10 Proposition 3] on a maximal interval , with .
Theorem 3.1 (Equilibria of (7) and KKT points of (1)). Consider the COSMIC dynamics (7) with , and . Let denote the set of equilibria of (7). The following statements hold.
(i) (Perturbed KKT characterization) For every , if and only if and .
(ii) (Limiting KKT points) Let and . Then, every limit point of satisfies the KKT conditions of (1).
(iii) (Multiplier convergence under LICQ) Let , , and let be a limit point of at which LICQ holds. Then, along every subsequence such that , converges to the unique such that satisfies the KKT conditions of (1).
(iv) (Boundedness and suboptimality in the convex case) If and , , are convex and the set of minimizers of (1) is nonempty and compact, then, for every , the set of all equilibria obtained for , i.e., , is bounded, and
Consequently, any sequence as in (ii) is bounded and has at least one limit point satisfying the KKT conditions of (1).
Proof Item (i). Fix . Since is invertible, if and only if and, using , . Since and , by definition of , the last equality holds if and only if , which in turn is equivalent to , , and for all .
Item (ii). Let be a limit point of . With a slight abuse of notation, we denote by also a subsequence converging to . Since , item (i) implies , , , and , for all and . By continuity of , , , passing to the limit yields (2) at .
Item (iii). Let , , and let be a limit point of at which LICQ holds. Item (i) and continuity of implies . Define the inactive constraints , and let and be the rows of indexed by and , respectively, and use the same indexing also for . By stationarity at we have
By LICQ, is positive definite at and, by continuity, at some for large enough. Multiplying both sides of (9) by yields . Moreover, the inactive constraints’ multipliers vanish in the limit as , i.e., as and jointly. To see this, let , so that . By continuity, , for large enough, and by item (i), and . Then, we have as , resulting in , and, by continuity of and , . Hence , with . Finally, passing to the limit in (9) yields . Moreover, is a limit of positive vectors with , and for all , since for and for .
Item (iv). Let and . By item (i), , , for all . Hence is feasible and . Since and , are convex, is convex and, by stationarity, . Hence, minimizes and, for all ,
Choosing as a minimizer, and using yields (8). Now let and with . Then , which is bounded as a sublevel set of a convex problem whose set of minimizers is nonempty and compact [21 Corollary 8.7.1]. Hence, is bounded uniformly in . Moreover, fixing any , so that by item (i), and choosing , inequality (10) and yield . Since and for all , each term of the sum is nonnegative, so , , uniformly in . Therefore, is bounded. Finally, let with , and let , which is finite since converges. Then for all , so , which is bounded by the argument above. Then, by the Bolzano–Weierstrass theorem, admits a limit point, which satisfies the KKT conditions of (1) by item (ii). ■
Theorem 3.1.(i) implies that does not depend on the sliding mode parameters and . Moreover, items (i)–(iii) assume neither feasibility of (1) nor existence of a minimizer, they characterize the equilibria of (7) whenever these exist. Item (i) yields for every , so if (1) has no strictly feasible point, e.g., if it is infeasible. If and are convex, inequality (10), which only uses convexity, implies that if . Statement (ii) requires no constraint qualification, since it assumes that has a limit point, which in particular requires to have a bounded subsequence. On the other hand, item (iii) requires only that has a limit point and shows that LICQ at guarantees convergence of to the (unique) KKT multiplier at . Without constraint qualification, the sequence may diverge even if converges to a minimizer. Consider, for example, , whose minimizer violates LICQ, since , and admits no KKT multiplier. The equilibrium of (7) is , so while as . Finally, item (iv) specializes the results to convex problems and states that every equilibrium of (7) is strictly feasible and -suboptimal for any fixed , rather than only in the limit . In particular, choosing guarantees for a prescribed accuracy . Moreover, under the assumptions of item (iv) the boundedness of implicitly assumed in item (ii) holds automatically and the existence of a limit point satisfying the KKT conditions is guaranteed.
Convergence Analysis
First, we show that trajectories of COSMIC reach the sliding manifold in finite time.
Lemma 4.1 (Finite-time reaching of the sliding manifold). Let , and let be a Filippov solution of (7) on . Define , for , let and assume that . Then for all .
Proof. Since and is absolutely continuous, is absolutely continuous and satisfies the sliding law (5) with in (7) that is, we have , for almost every . In particular, this implies that each evolves independently of and of , for every . If , then, as long as , we have , whose solution is . Thus, vanishes at . By symmetry, we have that if , the reaching time is , and if , then . Moreover, is forward invariant, since whenever and whenever . Thus, for all , and the statement follows by setting .
The reaching time in Lemma 4.1 depends only on , and the initial conditions via . Moreover, is finite because . For , converges to only exponentially. The finite-time convergence of COSMIC also serve as a feasibility certificate.
Corollary 1 (Feasibility certificate). In the setting of Lemma 4.1, if the trajectory of COSMIC starting from some escapes to infinity at , then (1) admits no strictly feasible point.
To establish convergence rates we introduce the following assumption.
Assumption 1 (Problem structure). Given and , assume
the function is -smooth and the functions are convex with -Lipschitz gradient, for all ;
there exists such that (Slater condition).
Lemma 4.1 holds on , but it does not guarantee that , let alone that . Since the primal dynamics is gradient-like, the solution of (7) can escape to infinity in finite time only if the dual dynamics does. The next result provides a sufficient condition to preclude this.
Lemma 4.2 (Global existence). Let be a Filippov solution of (7) with . Under Assumption 1, .
Proof. We provide a sketch of the proof and give the full proof in Appendix § 9. Let . Since , by the proof of Lemma 4.1, for all , and can be written as an explicit function of and , which diverges only if approaches the boundary of . Let be a Slater point and . On any bounded time interval (i) by convexity of and Slater’s condition, and are bounded and (ii) the function , grows at most by a constant times , and hence stays bounded. Since as approaches the boundary of , stays away from that boundary. Therefore, remains bounded on every bounded time interval, and .
Note that implies , so Lemma 4.2 holds whenever . The lemma implies that COSMIC can be initialized at any together with, for example . Hence, no projection onto the feasible set is required prior to running the algorithm.
Theorem 4.1 (Convergence of COSMIC). Under Assumption 1, let be -strongly convex, let , and let be a Filippov solution of (7) with . Then, COSMIC converges to . In particular,
with as in Lemma 4.1, and as .
Proof. By Lemma 4.2, . Next, by Lemma 4.1, , which, by Theorem 3.1.(i) implies , and , , and . Substituting into (7) yields , where . That is, follows the gradient flow of the logarithmic barrier function of Fiacco–McCormick (Fiacco and McCormick, [13]) defined on the feasible set . By assumption, is -strongly convex on the nonempty set , and its sublevel sets are compact. Then, has a unique minimizer and . Along the solution of (7), , hence . Finally, since , continuity of gives .
Under the assumptions of Theorem 4.1, problem (1) has a unique minimizer and satisfies Slater’s condition. By Theorem 3.1.(iv), from any and, e.g., , COSMIC reaches the strictly feasible region in finite time and converges exponentially to with .
Relation with barrier methods
Logarithmic barrier methods (see, e.g., Boyd and Vandenbergh, [5], Ch. 11) approximate (1) by the family of problems
whose solutions define the central path. Setting , the optimality conditions of (11) can be written as the perturbed KKT system
and, in the convex case, (Boyd and Vandenberghe, [5]). By Theorem 3.1.(i), COSMIC’s equilibria are the solutions of (12) with . In the convex case, the -components of are the minimizers of (11) with , and the bound in Theorem 3.1.(iv) recovers the classical duality gap . This provides a control-theoretic interpretation of the central path. Specifically, its points are the equilibria of (7). However, the barrier function in (11) is defined on the strict interior of the feasible set, so classical barrier methods must be initialized at a strictly feasible point, which in general requires solving an auxiliary phase (Boyd and Vandenberghe, [5]). In COSMIC, instead, is a state of the dynamics driven by the SMC law (5), and strict feasibility is attained in finite time, once the trajectories reach the sliding manifold .
Algorithm Design
Algorithm 1 is one step of the Euler discretized equivalents of (7) with two additional modifications. First, on the sliding manifold, for active constraints, so , though well defined, becomes ill-conditioned as decreases. We therefore replace by , with . Second, to reduce chattering, we use the boundary-layer approach for SMCs and replace by the component-wise saturation function (Khalil, [18]). Theorem 3.1 suggests decreasing to approach a KKT point. We implement this via a vectorized -homotopy that can be seen in Algorithm 2, where is divided by , with a lower bound set to machine precision (in our case ). Specifically, decays when with , as in path-following interior-point methods. Using a separate per constraint lets decay freely for inactive constraints, without being limited by the active constraints, for which a small causes numerical stiffness. Unless stated otherwise, we use and , which are values that we find to work well in practice. Since is diagonal, its inversion is computationally inexpensive and is implemented as a component-wise division of vectors.
Algorithm 1 COSMIC STEP() function
Input: x_k, μ_k, ε_k; gains K, β > 0; step size dt; saturation width ε_sat; damping floor δ_D
x_{k+1} ← x_k − dt(∇f(x_k) + J_g(x_k)^T μ_k)
s_k ← √(g(x_k)^2 + μ_k^2 + (ε_k)^2), φ_k ← μ_k − g(x_k) − s_k, d_k ← 1 − μ_k ⊙ s_k
D_{ε,k} ← diag(√(d_k^2 + δ_D^2))^{-1}, G_k ← diag(1 + g(x_k) ⊙ s_k)
μ_{k+1} ← μ_k − dt D_{ε,k}[Kφ_k + β sat(φ_k, ε_sat) + G_k J_g(x_k)(∇f(x_k) + J_g(x_k)^T μ_k)]
return x_{k+1}, μ_{k+1}, φ_k
Numerical Examples
We now validate COSMIC on SVM optimization for the Iris Setosa-vs-Versicolor dataset, and on image completion via ICNNs. We compare COSMIC to Feedback linearization (FL) Zhang et al.
Algorithm 2 COSMIC with smooth conditioning floor and safeguarded -homotopy
Input: ; ; ; initial base smoothing ; horizon ; stationarity, feasibility, and complementarity tolerances ; conditioning margin ; shrink safeguard ; homotopy rate
while , , , and do
(Algorithm 1)
, ,
, ,
end
return
(2025), Adam (Kingma and Ba, [19]) within an Augmented Lagrangian method (Adam-AL), and Adam applied to a penalty function (Adam-PEN) (Lu et al., 2021).
Remark 2 We re-implement the FL algorithm from Zhang et al. [23] in JAX with just-in-time compilation to speed up simulations. The algorithmic structure is unchanged, including the Gurobi quadratic program (QP) solver for the multipliers, which increases FL’s computational cost.
Iris Dataset SVM. We consider a support vector machine (SVM) benchmark on real data (Fisher, [15]). Training a soft-margin SVM is equivalent to subject to , , where are the samples with labels , and is the number of samples. We use the full dataset () with , and train a soft-margin SVM to separate the Virginica species from the other two. The resulting formulation has optimization variables, and the optimal solution for both problems is computed using SLSQP and then refined by Newton’s method on the active-set KKT system. Each method is run from 10 different initial conditions, with components sampled independently and uniformly from . Since, with the tested parameters, the second-order KKT-SMC method fails to converge from any initial condition, we defer its comparison with COSMIC to Appendix § 10.§ 1, on a hard-margin SVM problem with a smaller dataset. Figure 2 plots the log of the error of FL, Adam-AL, Adam-PEN, and COSMIC with respect to CPU time. COSMIC significantly outperforms the other methods, reaching the highest accuracy in the shortest CPU time. Although FL reaches a final accuracy of order , its per-iteration cost is high, averaging 6.14 ms versus 0.042 ms for COSMIC, 0.059 ms for Adam-PEN, and 0.066 ms for Adam-AL.
Figure 2. Optimization of soft-margin SVM for Iris Dataset. The solid line is the mean error over the 10 runs, and the shaded band corresponds to their minimum and maximum.
Image Completion using ICNN We consider a higher-dimensional problem: completing the right half of a $64\times64 face image from its left half with a learned convex energy given by a partially input convex neural network (PICNN) (Amos et al., [2]). The completed half (n=2048 pixels) is obtained by minimizing the PICNN output subject to linear inequality constraints that enforce the valid pixel range . Figure 3 compares COSMIC with FL, Adam-AL and Adam-PEN, and with projected gradient descent with momentum (PGD-MOM), the inference method of [2]. All methods are run for 3000 iterations on the same 3 test images and 2 initial infeasible points per image drawn uniformly from . The CPU time is measured per iteration on the compiled solvers, excluding metric logging; for FL it includes the QP solve in Gurobi. The hyperparameters for PGD-MOM are adopted from Amos et al. [2], while those of the other methods are tuned separately via a parameter sweep for best performance, and listed in Table 2 in Appendix § 10.§ 2. Figure 3 shows that COSMIC reaches the highest accuracy in the shortest CPU time, and has better and comparable performance to adaptive momentum based methods. Although While FL achieves high accuracy, the QP solved at each iteration results in an average iteration time of approximately 24 ms, considerably higher than the 4.8–5.5 ms of the other methods, with a total runtime exceeding 80 s for 3000 iterations. Adam-AL, Adam-PEN, and PGD-MOM converge more slowly and do not reach the accuracy of COSMIC within the same budget.
Figure 3. Error convergence with respect to (a) iterations and (b) CPU time. The solid line shows the mean error over the 3 runs, and the shaded band spans their minimum and maximum. For visualization purposes, the CPU-time range is limited to 40s.
| Method | Hyperparameters |
| COSMIC | , , , |
| FL | , |
| Adam-AL | , , , , |
| Adam-PEN | , , , , |
Table 2. Selected hyperparameters for image completion using ICNNs.
Conclusion
We introduce COSMIC (7), a continuous-time solver for inequality-constrained optimization in which the Lagrange multiplier dynamics are designed via SMC. Its trajectories reach, in finite time, the sliding manifold where the smoothed FB complementarity condition are satisfied. On this manifold, COSMIC is the gradient flow of a log-barrier function, which yields a control-theoretic interpretation of barrier methods. We show that the equilibria of (7) are exactly the points satisfying stationarity and the smoothed FB condition, and that, as the smoothing parameter decays, every limit point of a sequence of equilibria is a KKT point of the original problem. For convex problems satisfying Slater’s condition, the suboptimality is bounded by . For strongly convex costs, from any initial primal point, the trajectories become strictly feasible in finite time and then converge exponentially to the minimizer of the associated barrier problem. Experiments on SVM classification and image completion via an ICNN mapping confirm that COSMIC solves these problems accurately and scales up without significant increase in computational time per-iteration. Future directions include the extension of the convergence analysis beyond strong convexity, analysis of discrete-time implementations, and extensions to accommodate equality constraints.
Generative AI Use
Generative AI tools were used for code generation, editing, and proof checking. The authors reviewed and verified all AI-assisted content and take responsibility for the final content of the paper.
Acknowledgments
This work was supported as a part of NCCR Automation, a National Centre of Competence in Research, funded by the Swiss National Science Foundation (grant number 51NF40 225155).
Extended proof
Proof [Proof of Lemma 4.2] The right-hand side of (7) is measurable and locally essentially bounded. Hence a Filippov solution exists from every initial condition ([10 Proposition 3]), and, a maximal Filippov solution does not exist in finite time if and only if it leaves every compact set. Let be a maximal Filippov solution with , defined on , and assume, for the sake of contradiction, that . In what follows, denotes a generic constant which may depend on , the initial condition, and the problem data. The value of may change from one occurrence to the next, even within the same line, but we keep the same symbol for simplicity of notation.
Since , the proof of Lemma 4.1 gives and for almost every . Moreover, , where . Expanding and substituting yields
Therefore, , and can diverge only if for some . It thus suffices to show that is bounded and that each is bounded away from zero. Next, let be a Slater point and . Then, by (7)
By the Cauchy–Schwarz inequality and since is -Lipschitz, it holds . Moreover, since and is -Lipschitz, the second sum is bounded by . Since and , convexity of and (13) yield . Thus,
where subsumes the scaling constants. Moreover, since , Grönwall’s inequality gives and on .
By (13), , where
Along the solution, and by using , , and (12), we have
where we used that is bounded, so that and , that since , and that and . Integrating gives . On the other hand, since is bounded and for all , all terms of except the -th logarithm are bounded below, and the latter is at least whenever . Hence, on .
By (12), , so remains in a compact set on , contradicting . Hence, .
Numerical Experiments
We provide implementation details and additional results for the experiments in § 5. Besides FL and KKT-SMC, we compare COSMIC with two standard first-order approaches to constrained learning based on Adam, namely Adam with an augmented Lagrangian formulation, and Adam applied to a quadratic penalty formulation. For completeness, we review these methods.
Adam with Penalty Method (Adam-PEN). Adam is run on the quadratic penalty function
where is an increasing penalty parameter capped at .
Adam with an augmented Lagrangian (Adam-AL). As a first-order baseline, we apply the method of multipliers to the Powell–Hestenes–Rockafellar augmented Lagrangian
Each outer iteration runs Adam steps on (with and fixed), then updates
where is the Adam step size, , , and the Adam moments are reset after every multiplier update. The effective multiplier is used when evaluating the KKT residual.
Hard-Margin SVM Optimization
To compare COSMIC with KKT-SMC, we consider the hard-margin SVM problem which has less variables. In this case, training an SVM is equivalent to solving
where are the samples, their labels, and is the number of samples. We use the linearly separable Iris Setosa-vs-Versicolor dataset with features, which gives optimization variables and linear inequality constraints corresponding to only two species of the Iris flower: Setosa and Veriscolor. We aim to show the performance of COSMIC on this lower dimensional problem and also consider its sensitivity to and , introduced in Algorithm 2. All methods are run from the same 10 initial conditions, whose components are sampled uniformly at random from , with a fixed budget of iterations. Figure 4 shows the error with respect to the optimal solution as a function of the CPU time for COSMIC, FL, and KKT-SMC, with the best-performing set of hyperparameters. The tested hyperparameters are listed in Table 1. For COSMIC, we test several values of , which give comparable performance. For the problem formulation considered here, the performance of COSMIC is relatively insensitive to the specific values of and . This is illustrated in Figure 4, where we vary over and over .
Figure 4. Optimization of Hard-Margin SVM for Iris Dataset
| Method | Hyperparameters |
| COSMIC | , , , , |
| FL | , |
| KKT-SMC | , , , , |
Table 1. Tested hyperparameters for SVM optimization.
Since the problem is low-dimensional, the second-order KKT-SMC reaches the neighborhood of the minimum within a few iterations. However, its accuracy is limited by the non-vanishing smoothing parameter of the FB function. COSMIC requires more iterations, but attains a higher accuracy than KKT-SMC and has a lower average cost per iteration (0.045 ms versus 0.105 ms for KKT-SMC). FL is slower and less accurate than both sliding-mode-based methods.
Image Completion using Input Convex Neural Networks
Architecture. Let denote the observed left half of a face and the right half to be generated. The energy is a convolutional PICNN with an input path acting on and an output path acting on ,
where is the softplus function, applied element-wise, is the positive part, is the element-wise product, and collects all weights and biases. We choose the network as in [2]. Because is convex and non-decreasing, the weights acting on the hidden states are element-wise non-negative, and the gates are non-negative and depend on only, every is convex in . Hence, is convex and smooth in for every , although not necessarily convex in , and the quadratic term makes it -strongly convex with , so (18) has a unique minimizer.
Training. The parameters are trained with a denoising score matching loss, such that the true right half is the minimizer of : for a noisy right half , , ,
with , i.e., is fitted to the convex bowl around the true completion. Problem (17) is solved with Adam (batch size 16, learning rate with cosine decay to , 2000 steps), where the weights are projected onto the non-negative orthant after every step, as in [2]. We use the Olivetti faces (400 images of 40 subjects, pixels in ) with 36 subjects for training, and 4 for testing, and random horizontal flips of the full image as data augmentation.
Problem. Given the left half of a test face, the completed right half is the minimiser of the learned energy subject to the pixel range,
Writing the constraints as with
problem (18) becomes the strongly convex program s.t. . Since the constraints are affine, the Jacobian is constant and the Hessian of the Lagrangian equals . The unique primal–dual solution is characterized by the KKT conditions
Reference solution and metrics. The reference is computed in two stages. L-BFGS-B first locates the solution and its active set . The pixels within are fixed at their bounds, while the remaining free pixels are refined by Newton’s method.
We present the corresponding convergence of the multiplier to its optimal value in Figure 5.
Figure 5. Multiplier error convergence with respect to (a) iterations and (b) CPU time. The solid line shows the mean error over the 3 runs, and the shaded band spans their minimum and maximum. For visualization purposes, the CPU-time range is intentionally limited to 40s.
We also showcase a sample of the resulting completed faces below.

Figure 6. Completed Faces from Olivetti Dataset using COSMIC, FL, Adam-AL, Adam-PEN, and PGD-MOM
References
- [1]A. Allibhoy and J. Cortés. Control-barrier-function-based design of gradient flows for constrained nonlinear programming. IEEE Transactions on Automatic Control, 69(6):3499–3514, 2023. doi: 10.1109/TAC.2023.3306492.DOI
- [2]B. Amos, L. Xu, and J. Z. Kolter. Input convex neural networks. In International Conference on Machine Learning, pages 146–155. PMLR, 2017.arxiv.org/abs/1609.07152
- [3]K. J. Arrow, L. Hurwicz, and H. Uzawa, editors. Studies in Linear and Nonlinear Programming. Stanford University Press, 1958.
- [4]G. Bianchin, J. Cortés, J. I. Poveda, and E. Dall’Anese. Time-varying optimization of lti systems via projected primal-dual gradient flows. IEEE Transactions on Control of Network Systems, 9 (1):474–486, 2021. doi: 10.1109/tcns.2021.3112762.DOI
- [5]S. Boyd and L. Vandenberghe. Convex Optimization. Cambridge University Press, 2004. ISBN 0521833787.
- [6]V. Centorrino, R. Hoteit, E. C. Balta, and J. Lygeros. SPPID: Saddle-point PID for constrained optimization. arXiv preprint arXiv:2609.31086, 2026a.arxiv.org/abs/2609.31086
- [8]V. Cerone, S. M. Fosson, S. Pirrera, and D. Regruto. A new framework for constrained optimization via feedback control of Lagrange multipliers. IEEE Transactions on Automatic Control, pages 1–16, 2025. ISSN 2334-3303. doi: 10.1109/tac.2025.3568651.DOI
- [9]F. Chen and W. Ren. Sign projected gradient flow: A continuous-time approach to convex optimization with linear equality constraints. Automatica, 120:109156, 2020. doi: 10.1016/j.automatica.2020.109156.DOI
- [10]J. Cortes. Discontinuous dynamical systems. IEEE Control Systems Magazine, 28(3):36–73, 2008. doi: 10.1109/mcs.2008.919306.DOI
- [11]B. Diana, S. Pandey, S. Kamal, and T. N. Dinh. Finite and fixed-time feedback-based continuous-time optimization. Automatica, 183:112569, 2026. doi: 10.1016/j.automatica.2025.112569.DOI
- [12]F. Dörfler, Z. He, G. Belgioioso, S. Bolognani, J. Lygeros, and M. Muehlebach. Toward a systems theory of algorithms. IEEE Control Systems Letters, 8:1198–1210, 2024. ISSN 2475-1456. doi: 10.1109/lcsys.2024.3406943.
- [13]A. V. Fiacco and G. P. McCormick. Nonlinear programming: sequential unconstrained minimization techniques. SIAM, 1990.
- [14]A. Fischer. A special Newton-type optimization method. Optimization, 24(3-4):269–284, January 1992. ISSN 1029-4945. doi: 10.1080/02331939208843795.DOI
- [15]R. A. Fisher. Iris dataset, 1936.
- [16]A. Hauswirth, Z. He, S. Bolognani, G. Hug, and F. Dörfler. Optimization algorithms as robust feedback controllers. Annual Reviews in Control, 57:100941, 2024. doi: 10.1016/j.arcontrol.2024.100941.DOI
- [17]C. Kanzow. Some noninterior continuation methods for linear complementarity problems. SIAM Journal on Matrix Analysis and Applications, 17(4):851–868, October 1996. ISSN 1095-7162. doi: 10.1137/S0895479894273134.DOI
- [18]H. K. Khalil. Nonlinear Systems. Pearson Education. Prentice Hall, 2002. ISBN 9780130673893.
- [19]D. P. Kingma and J. Ba. Adam: A method for stochastic optimization. arXiv preprint ARXIV:1412.6980, 2014.arxiv.org/abs/1412.6980
- [21]R. T. Rockafellar. Convex analysis, 1970.
- [22]A. Zenati and K. Youcef-Toumi. KKT Residual Sliding Manifolds: A Projection-Free Structured Finite-Time Solver for Nonlinear Constrained Optimisation, 2026.arxiv.org/abs/2610.15545
- [23]R. Zhang, A. Raghunathan, J. Shamma, and N. Li. Constrained optimization from a control perspective via feedback linearization. In Advances in Neural Information Processing Systems 38, pages 62328–62364, 2025. doi: 10.52202/085713-1873.DOI



