WorksEngineering
Learning monotone game-theoretic linear MPC
Abstract
We propose a method to learn a game-theoretic model predictive control (GT-MPC) law by imitation from input and output data of a running multi-agent system. By interpreting the recorded inputs as the first moves of the equilibria of a game played by the agents, we learn the GT-MPC weights under hard input constraints and soft output constraints, without requiring the agents to reveal their objectives. We also impose that the resulting generalized Nash equilibrium (GNE) problem is strongly monotone, which ensures the existence and uniqueness of the equilibrium and, hence, that the learned GT-MPC law is well defined and can be evaluated efficiently online by variational GNE-seeking algorithms for linear-quadratic games. Moreover, strong monotonicity implies that the equilibrium is almost everywhere differentiable with respect to the references and the weights, which enables automatic differentiation for gradient-based optimization. In addition, we promote closed-loop stability of the unconstrained equilibrium feedback by incorporating a penalty that involves a jointly learned quadratic Lyapunov function. Learning a GT-MPC formulation enables us to explain the agents' objectives, reproduce their behavior in simulation in new scenarios, optimize their set-points, design mechanisms, and enforce additional constraints. A numerical example illustrates the approach.
Introduction
In game-theoretic model predictive control (GT-MPC), several agents act on a common dynamical system, each optimizing a personal objective that depends on the other agents’ moves under local and shared constraints, and applying the first move of the resulting generalized Nash equilibrium (GNE) at every time step [26, 18, 19, 8, 17, 13]. Each GT-MPC problem is a finite-horizon dynamic game; see [16, 21, 23] for solution algorithms and multi-vehicle applications. Modeling the agents’ costs is often difficult: the agents may be unwilling to disclose their objectives and strategies for privacy reasons, and in many applications the plant is even operated without explicit objectives to minimize, e.g., by standard decentralized control loops or even manually. A natural goal is then to learn, from the observed actions only, a GT-MPC law that reproduces them, so that the agents do not have to reveal their strategies, and that adds the main benefit of MPC, namely, handling input and output constraints. Moreover, once a game representation of agents’ behavior is available, it can be used for higher-level tasks, such as optimization of the set-point signals that enter the game at each step. It also allows reproducing the agents’ behavior in closed-loop simulation, e.g., on a digital twin of the plant, to evaluate what-if scenarios that have not been observed yet.
In this paper we reach this goal by imitation: we learn a game whose equilibria, given reference signals and estimated states, reproduce the agents’ actions. We restrict our attention to strongly monotone linear-quadratic (LQ) games. Roughly speaking, monotonicity plays for games the role that convexity plays for optimization: if the joint feasible set is nonempty, closed, and convex, a strongly monotone game has a unique variational GNE for every value of the parameter (here, the state estimate, reference, and previous input) [15]. So the learned control law is well defined and, as LQ-GNE problems can be solved efficiently [3, 7, 5], easily computable. Moreover, the variational GNE of a strongly monotone LQ game is a path-differentiable function of the parameters, whose derivatives can be computed by automatic differentiation [10], which makes gradient-based learning and equilibrium design well founded.
Unlike [9] and [1], who fit the agents’ objectives by minimizing the residuals of the variational inequality and of the KKT conditions at observed equilibria, respectively, we impose strong monotonicity of the learned game by construction, as also done in [22] through an LMI. Moreover, in [9, 1, 22] the number of unknowns grows linearly with the number of samples, as the dual variables of each sample are optimized. Finally, in our setting only the first move of each equilibrium is observed, not the entire sequence of planned moves that the methods in [9, 1, 22] would require to fit a GT-MPC formulation. Learning was also combined with GT-MPC in [20] for two autonomous vehicles: the GNEs of a known constrained dynamic game are computed offline from random initial conditions, and a neural network trained to predict their reward outcome is used as the terminal cost of each agent’s MPC problem. Here, instead, we learn the unknown costs from closed-loop data, do not assume decoupled dynamics, and solve the learned LQ game online.
We define a class of GT-MPC problems with soft shared output constraints based on a linear state-space model realization learned from the available input / output data. Then, we learn agent-specific weights and slack penalties by a bilevel problem whose inner problems are the LQ-GNE problems of the game at each datapoint, in which strong monotonicity holds by construction and closed-loop stability of the unconstrained Nash equilibrium feedback is enforced by a penalty involving a jointly learned quadratic Lyapunov function, so that the problem can be solved by L-BFGS-B. We illustrate in a numerical example that we can reconstruct an underlying game played by the agents very precisely and shape closed-loop behavior by reference optimization.
Section 2 presents the problem setting, Section 3 recalls monotone LQ games, Section 3.1 characterizes the monotonicity of the GT-MPC game, Section 3.2 presents the bilevel learning problem and its solution algorithm, Section 3.3 closed-loop stability, Section 4 a numerical example, and Section 5 conclusions. The code is available at https://github.com/bemporad/gtmpc_learn.
Notation. We write for the identity, for the Kronecker product, and for the symmetric part of a square matrix . For symmetric matrices, means that is positive semidefinite. We write for the Euclidean norm and for the Frobenius norm. For a vector whose entries are associated with the agents, collects the entries of agent and those of all the other agents. For a matrix acting on such vectors, is the block that maps into the entries of agent . We use for sample steps and for prediction steps. For a vector , . For a differentiable vector- or matrix-valued function , denotes its differential. For a matrix , a vector , and a set of indices , is the submatrix of the rows of indexed by , and the subvector of indexed by .
Data collection and problem setup
We consider a multi-agent process with agents, whose behavior we want to imitate, in which agent applies the input at sample step . We write , with , and for the measured outputs. We record piecewise-constant references , inputs , and outputs in experiments, each of length , where the input and output measurements can be noisy.
Our goal is to learn a GT-MPC controller from the collected data, as described in the sequel, which imitates the behavior of the agents in tracking their respective references while respecting the following additional input and output constraints:
where is the vector of input increments and , , are polyhedral sets. For simplicity, we assume that , , , with possibly infinite entries in , , , and . We denote by , , , and the entries of , , , and associated with agent . As in standard MPC practice, constraints on the inputs and their increments are enforced as hard constraints, while output constraints are treated as soft constraints.
Our aim is to learn a GT-MPC controller of the following form, generalizing the one suggested in [5 Section 5.2]:
where is the estimate at sample step of the state vector associated with the prediction model , with partitioned by agent according to the input allocation, with , and , , with for , represents the sequence of input increments decided by agent , is a slack variable for softening the shared output constraints, and , , and are, respectively, the prediction, control, and constraint horizons. Note that, in general, the agent-specific weights and consider the entire vectors of tracking-errors and input-increments, respectively.
Problem (2) can be recast as a parametric LQ-GNE problem with parameter , , and decision vector , where collects the input increments of all agents and . By letting denote the variational GNE of (2) corresponding to , the MPC action at sample step is given by the first input increment of the optimal sequence, i.e., , where selects the first increments from .
As we only have input/output data available, we first need to identify a linear discrete-time model , together with the initial state of each experiment, by solving the standard output-error minimization problem
where is a small regularization function, as described in [4] and the associated jax-sysid package. Then, we run a Kalman filter based on the resulting model, whose noise covariances are tuning parameters, to get state estimates , starting from , given the corresponding inputs and outputs in experiment .
The obtained datasets are used for learning a parametric linear-quadratic GNE problem of the form (2) by fitting the output and input-increment weights , , and the weights on slack variables , to best replicate the observed behavior of the expert controller, i.e., we wish that
where is the vector of input increments of the variational GNE of (2) for , i.e., we interpret the measured moves of the agents as the first moves of the GNE of a game of the form (2) played by the agents at step . Once the game is learned, the GT-MPC law applies the first move of the GNE, at which all the agents play best responses to each other. We also require that the resulting game be strongly monotone, so that its variational GNE, and hence the left-hand side of (3), is unique whenever (2) is feasible.1
Learning monotone GT-MPC games
In a parametric LQ game with constant agents’ Hessian matrices and affine dependence on the parameter in the linear cost terms, agent minimizes over the cost
where , is symmetric, and is affine in . Agent ’s problem is convex when . The pseudogradient collects the agents’ gradients with respect to their own decision variables, in the same ordering as , so that , with and collecting the vectors , where . The game is monotone with constant [15, 24] if
for all , , which here is the same as
When the game is strongly monotone. With a shared constraint , where is nonempty, closed, and convex, the variational GNEs are the solutions of the variational inequality defined by on [14]. Since is continuous and strongly monotone, this variational inequality has a unique solution [15], so the game has a unique variational GNE.
Pseudogradient structure
Let denote the input sequence for the full horizon, which is related to the increments of the agents as , , where is the vector of ones, has entries if and otherwise, and selects . Then, the explicit form of the predicted outputs is the following affine function of :
with and or, equivalently, , where and .
By substituting the predictions in (2), the cost of agent is an LQ cost of the form (4), with , for , and , where
for all , with , . So is linear in . Only the block row of enters , i.e., agent ’s own weight and the cross weights , where selects ; the other blocks of multiply terms that do not depend on , and do not affect the game.
The row block of and depends only on agent ’s weights and . For a fixed model , both and are linear in the weight matrices. The slack enters agent ’s problem only through the terms and the shared output constraints. Note that the diagonal block is positive definite when and . Agent ’s problem (2) is then a strictly convex QP in , and also in when .
The pseudogradient of the GT-MPC game is , with , , and , whose Jacobian with respect to is , and . So, by (5), the game is strongly monotone in , with constant , if and only if
For a fixed model, the first condition is an LMI in , and the second one is a simple bound on each quadratic slack penalty.
Proposition 1. Suppose that (7) holds with , that , and that the bounds on the increments contain 0. Then, for every , the GT-MPC game has a unique variational GNE .
Proof. The shared feasible set is closed and convex, and it is nonempty, since with large enough slacks is feasible. By (7), is continuous and strongly monotone with constant on this set, so the variational inequality that it defines on the set has a unique solution [15], which is the unique variational GNE.
Learning problem
Our goal now is to set up a learning problem that identifies all the weights in (2) such that the resulting game is strongly monotone, i.e., satisfies (7). Since the pseudogradient matrix of the GT-MPC game is constrained to the structure (6), we cannot directly exploit the form that characterizes every strongly monotone LQ game, as was recently shown in [6 Lemma 4.4], where is upper triangular and is block strictly upper triangular. Instead, let us parametrize the agent-specific weights with variables , , and , , where , are the cross weights on input increments, and the own weights are
where is small and fixed, while is the pseudogradient matrix (6) obtained with . By (6), adding to each provides for every . The remaining block does not affect the game; after learning we set it to , the smallest block for which . The slack penalties and are scalars and are kept directly as optimization variables. Then, we learn the agent-specific weights and the slack penalties by solving
where is the vector that collects all the learnable parameters of the game, is the measured input at sample step , and is the first move of the variational GNE
of the game (2) with the weights at the parameter . The scalar weights the regularization of the weights , , and .
Due to the GNE terms , the optimization problem in (9) is a bilevel problem, whose lower-level problems are the GNE problems of the game at the parameters . By Proposition 1, is uniquely defined for all and admissible , as the measured inputs satisfy the input constraints. For compactness of notation, in (9) we drop the superscript and number all the sample steps of all experiments consecutively as . We describe below how we solve this bound-constrained bilevel nonlinear least-squares problem with L-BFGS-B [12], including the initial guess and the computation of the GNEs and of their derivatives with respect to .
Initialization. We initialize the components of as
where is a scale factor, is small, , , and . Then (6) gives the symmetric matrix for every , where , so that at the initial point. With the slack penalties fixed to and , we choose by minimizing the imitation loss, i.e., the first term of the objective in (9), over a logarithmic grid of values of , which requires solving LQ-GNE problems for each grid value.
Computing the GNEs and their derivatives. As L-BFGS-B is a gradient-based method, we need to differentiate the LQ-GNEs with respect to efficiently. Recall that the pseudogradient of the game is , with and , and let collect the shared constraints of (2), where also depends on . As by strong monotonicity (5) matrix is invertible, by [7 Lemma 5.1] the variational GNE is , , where solves the QP
with and . Although and its derivatives could be obtained directly by solving (10) with a differentiable QP solver [2, 25], we take a more efficient approach here. Let be the value of the parameters at the -th iteration of the learning algorithm. We compute by a tailored LQ-GNE solver (such as DR-DAQP [3]) and identify the set of constraints with positive multipliers (we drop the dependence on from now on for simplicity of notation). Then, we write , where are the rows of of the active constraints. For a fixed active set, this is a linear system , with and , whose coefficients depend smoothly on through and . If the rows of are linearly independent, the multipliers are positive, and the inactive constraints are strictly satisfied, the active set does not change for small perturbations of . Then , , and hence are smooth functions of around , with , obtained by differentiating , and hence . In our numerical experiments, we implement the map from to the imitation loss through these linear systems in JAX [11].
Closed-loop stability enforcement
Strong monotonicity guarantees that the GT-MPC law is well defined, but not that the closed loop is asymptotically stable. Without constraints, the variational GNE is , which is linear in , so that the vector of first input increments of all agents is . In closed loop with the identified model and , the state evolves as , for some matrix , where . Under the assumption that the observed multi-agent system generating the training data is asymptotically stable, we can enforce asymptotic stability of the unconstrained closed loop a priori by adding to the objective of (9) the following penalty
where weights the penalty relative to the imitation loss, is a desired contraction rate, and the matrix , upper triangular with positive diagonal, is learned together with . Since is equivalent to with , a value of below 1 certifies the quadratic Lyapunov function of the unconstrained closed loop. Conversely, by [4 Lemma 1] applied to , for every matrix with spectral radius smaller than there exists such an , given by the Cholesky factor of the solution of a Lyapunov equation, for which , so the penalty can be zeroed for every game whose closed-loop matrix has spectral radius smaller than . As (11) is a penalty, we check that at the solution and, otherwise, increase and solve the problem again starting from the current solution. By the separation principle, stability of also implies stability of the nominal closed loop with the Kalman filter when constraints are not active, although it does not guarantee global closed-loop stability under constraints, nor robustness to the mismatch between the identified model and the actual process.
Numerical example
Generating system. We consider a randomly generated open-loop stable process with inputs, outputs, states, and agents, each commanding one input. The constraints are , , and we want to enforce . The agents operate the process by playing a GT-MPC game of the form (2) with , , the true model and state, the soft output constraints with the slack penalties and , and the weights , , , and , where is the th column of and . The pseudogradient matrix of this game is not symmetric, , so the game is not a potential game that can be represented by a single centralized MPC cost. It is also strongly monotone, with . All three inputs and outputs are dynamically coupled by the system dynamics.
Data collection. We record experiments of 200 steps in which the agents operate according to the underlying (unknown) GT-MPC strategy. Each experiment starts from the equilibrium of a reference uniformly distributed in , a range in which the corresponding equilibrium input always satisfies the input constraints, and the references of all the agents are piecewise constant, with new values uniformly distributed in every 15 steps. The output noise is uniformly distributed in .
Learning. Using jax-sysid, we first identify a model with states from these closed-loop data in 2.3 s.2 On 3 further experiments generated in the same way, the open-loop simulation of the identified model, from the initial states estimated by jax-sysid, achieves best fit rates of 98.7%, 98.6%, and 98.7% on the outputs , where collects the measured values of output , the simulated ones, and is the mean of .
For the GT-MPC structure to learn, we use , , , , and different values of between 1 and . We initialize the bilevel problem with , , and , searching over the grid for initializing the solution of (9), which we solve by the L-BFGS-B solver of SciPy with the stability penalty (11), , and . The Kalman filter uses the process noise covariance , the measurement noise covariance , and the initial covariance .
Controller. At each time step , the GT-MPC law estimates by a Kalman filter based on the identified model, sets , and computes the variational GNE of (2), which exists and is unique by Proposition 1, by the DR-DAQP solver [3] through the nashopt package [5].
Results. We validate the controllers on 10 runs of 80 steps, each with a set-point change from to , starting from the equilibrium of . Each component of is uniformly distributed in , and , with uniformly distributed in . The Kalman filter of the GT-MPC law starts from the equilibrium state of the identified model for . Table 1 reports, for different values of , the training time, the monotonicity constant of the learned game, and the closed-loop imitation errors
| Agents | Learned, | |||
| Time [s] | ||||
Table 1. Numerical example. Left: true weights used by the agents and weights learned with . Right: learning time, monotonicity constant , and imitation errors for .
where and are the measured outputs and the inputs of the closed loops with the GT-MPC law and with the agents in run . The average CPU time per step taken by the solver to evaluate the GT-MPC action ranges between 21 and 96 s depending on the control horizon . With closed-loop stability enforced, the GT-MPC law learned by (9) imitates the agents with imitation errors below 0.071 for all , and the smallest for , which is the control horizon indeed used by the agents generating the training data.
Figure 1 shows a run of the GT-MPC law learned by (9) with . The GT-MPC law produces closed-loop trajectories that are almost indistinguishable from those of the agents. As shown in Table 1 (left), for the learned weights are close to those of the agents up to a common scale factor , which leaves the GNEs unchanged, and the off-diagonal entries of and the cross weights are small, below and in magnitude, respectively. With the stability penalty, the unconstrained closed loops are asymptotically stable for all (spectral radius between 0.72 and 0.85).
Figure 1. Agents (black) and GT-MPC law learned with by (9) (blue, dashed), tracking the references (dotted), agents reacting to the reference generator (12) (green), and generated references (orange). Top: outputs and bounds (dashed). Bottom: inputs.
Set-point optimization
The learned game can be used for higher-level tasks without requiring the agents to reveal their objectives. As an example, a reference generator placed on top of the agents, who keep playing their own GT-MPC game, sends them at each step the reference
where is the reference of Figure 1 to track, is the output predicted by the identified model when the agents apply the GT-MPC action of the game learned with at , reduces the importance of tracking and , and prevents excessive deviations of from . We solve (12) by a Gauss-Newton method, using the derivatives of the GNE of § 3.2. As shown in Figure 1 (green) for and , the generator largely removes the dip and the bump of caused by the coupling with the other loops (see red circles), at the price of slower transients of and .
Conclusions
We have proposed a method for learning strongly monotone GT-MPC laws from input/output data of multiple agents, and illustrated the use of the learned game for explaining the agents’ objectives and for optimizing the references given to the agents. Future research will address other higher-level uses of the learned game, such as mechanism design, and extensions to state-dependent weights and nonlinear systems.
Generative AI Use
Generative AI tools were used to help write and proofread the paper and for generating the Python code used in the reported example.
Acknowledgments
This work was funded by the European Union (ERC Advanced Research Grant COMPACT, No. 101141351). Views and opinions expressed are however those of the authors only and do not necessarily reflect those of the European Union or the European Research Council. Neither the European Union nor the granting authority can be held responsible for them. The work was also supported by the Deutsche Forschungsgemeinschaft (DFG, German Research Foundation), project number 528033031.
References
- [1]S. Allen, S. A. Gabriel, and J. P. Dickerson. Using inverse optimization to learn cost functions in generalized Nash games. Computers & Operations Research, 142:105721, 2022.arxiv.org/abs/2102.12415
- [2]B. Amos and J. Z. Kolter. OptNet: Differentiable optimization as a layer in neural networks. In Proceedings of the 34th International Conference on Machine Learning, pages 136–145, 2017.arxiv.org/abs/1703.00443
- [3]D. Arnström, E. Benenati, and G. Belgioioso. DR–DAQP: A hybrid operator splitting and active-set solver for affine variational inequalities. arXiv preprint arXiv:2604.02531, 2026.arxiv.org/abs/2604.02531
- [4]A. Bemporad. An L-BFGS-B approach for linear and nonlinear system identification under ℓ₁ and group-lasso regularization. IEEE Transactions on Automatic Control, 70(7):4857–4864, 2025.arxiv.org/abs/2403.03827
- [5]A. Bemporad. NashOpt: A Python library for computing generalized Nash equilibria. arXiv preprint arXiv:2512.23636, 2025.arxiv.org/abs/2512.23636
- [6]A. Bemporad and T. Tatarenko. Learning parametric monotone games. arXiv preprint arXiv:2609.02494, 2026.arxiv.org/abs/2609.02494
- [7]A. Bemporad and T. Tatarenko. Solving monotone linear-quadratic generalized Nash equilibrium problems via quadratic programming. arXiv preprint arXiv:2608.07336, 2026.arxiv.org/abs/2608.07336
- [8]E. Benenati and S. Grammatico. Linear-quadratic dynamic games as receding-horizon variational inequalities. IEEE Transactions on Automatic Control, 71(4):2404–2417, 2026.arxiv.org/abs/2408.15703
- [9]D. Bertsimas, V. Gupta, and I.C. Paschalidis. Data-driven estimation in equilibrium using inverse optimization. Mathematical Programming, 153(2):595–633, 2015.
- [10]J. Bolte, E. Pauwels, and A. Silveti-Falls. Differentiating nonsmooth solutions to parametric monotone inclusion problems. SIAM Journal on Optimization, 34(1):71–97, 2024.arxiv.org/abs/2212.07844
- [11]J. Bradbury, R. Frostig, P. Hawkins, M.J. Johnson, C. Leary, D. Maclaurin, G. Necula, A. Paszke, J. VanderPlas, S. Wanderman-Milne, and Q. Zhang. JAX: composable transformations of Python+NumPy programs. http://github.com/google/jax, 2018.
- [12]R.H. Byrd, P. Lu, J. Nocedal, and C. Zhu. A limited memory algorithm for bound constrained optimization. SIAM Journal on Scientific Computing, 16(5):1190–1208, 1995.
- [13]E. Dimou, F. Fabiani, and A. Bemporad. Receding horizon games with stability guarantees. In 65th IEEE Conference on Decision and Control, 2026.
- [14]F. Facchinei and C. Kanzow. Generalized Nash equilibrium problems. 4OR, 5(3):173–210, 2007.
- [15]F. Facchinei and J.-S. Pang. Finite-Dimensional Variational Inequalities and Complementarity Problems. Springer, New York, 2003.
- [16]D. Fridovich-Keil, E. Ratner, L. Peters, A.D. Dragan, and C.J. Tomlin. Efficient iterative linear-quadratic approximations for nonlinear multi-player general-sum differential games. In Proc. IEEE International Conference on Robotics and Automation, pages 1475–1481, Paris, France, 2020.
- [17]S. Hall, G. Belgioioso, F. Dörfler, and D. Liao-McPherson. Stability certificates for receding-horizon games. IEEE Transactions on Automatic Control, 71(5):3511–3518, 2026.arxiv.org/abs/2404.12165
- [18]S. Hall, G. Belgioioso, D. Liao-McPherson, and F. Dörfler. Receding horizon games with coupling constraints for demand-side management. In 61st IEEE Conference on Decision and Control, pages 3795–3800, 2022.DOI
- [19]S. Hall and A. Bemporad. Solving multiparametric generalized Nash equilibrium problems and explicit game-theoretic model predictive control. arXiv preprint arXiv:2512.05505, 2025. Code available at https://github.com/bemporad/nash_mpqp.arxiv.org/abs/2512.05505
- [20]H. Kim, E.L. Zhu, C.S. Lim, and F. Borrelli. Learning two-agent motion planning strategies from generalized nash equilibrium for model predictive control. In Proc. 7th Annual Learning for Dynamics and Control Conference, volume 283 of Proceedings of Machine Learning Research, pages 112–123, 2025.arxiv.org/abs/2411.13983
- [21]S. Le Cleac’h, M. Schwager, and Z. Manchester. ALGAMES: A fast solver for constrained dynamic games. In Robotics: Science and Systems XVI, 2020.
- [22]A. Mallick, R. Rahimi Baghbadorani, P. Mohajerin Esfahani, and S. Grammatico. Learning in inverse games: Tractable training with probabilistic guarantees. arXiv preprint arXiv:2610.03481, 2026.arxiv.org/abs/2610.03481
- [23]M. Pustilnik and F. Borrelli. Generalized Nash equilibrium solutions in dynamic games with shared constraints. arXiv preprint arXiv:2502.19569, 2025.arxiv.org/abs/2502.19569
- [24]G. Scutari, D. P. Palomar, F. Facchinei, and J.-S. Pang. Convex optimization, game theory, and variational inequality theory. IEEE Signal Processing Magazine, 27(3):35–49, 2010.DOI
- [25]K. Tracy and Z. Manchester. On the differentiability of the primal-dual interior-point method. arXiv preprint arXiv:2406.11749, 2024.arxiv.org/abs/2406.11749
- [26]M. Wang, Z. Wang, J. Talbot, J. C. Gerdes, and M. Schwager. Game-theoretic planning for self-driving cars in multivehicle competitive scenarios. IEEE Transactions on Robotics, 37(4):1313–1325, 2021.
